From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1792571993; b=VWy4+sA2G42q6ttrM81vrl9S/iEkA/m41BwQ6I6EpwFTsSwjJM3Y9i/ghSMugIqYcdHd 2QGCbx5vrMn2JtLxFypCx9cqmCowxgFQaY9kw9eTWPDxGj1+uUPRDHjucAeIkN/5Zwk+l GM+ThNZtQ39Lu8onNLtajX0CWkBihQipV+gXk0oY/EFAax3COFPs5GvpXqi1tS9xzzqph hUUGtOA3sCr84VxnSmeZy4BZSlKZ3S8Fial1HmxuuqeUZDcF5urN3XXysBta8wLgBvLvl yoGhkUYydtUI9vgDZ+gfZjDw6LrBJ2mXUl6lf4fHiIP4q6DC5UdBe5wdliYTXwHiGsw== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1792571993; h=Received:DKIM-Signature:From:To:Cc:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:Content-Type: Content-Transfer-Encoding; bh=JY1mqR0n6Yfs3COqGT2YRXKLKbUnlNs6WzUFwuL4RGY=; b=EHV8EhMuhhMGdKjKVaZo4HdyPm1ruCRwOaSG/oMqim8Y08ccz+4E8uRSpqBG50keMXfc xbpHzGtFMiRWIe7/RRvOcAncVvFJOirPgZO8lfED3MMDgO9JdlxWeNm+yNO4Xn6R7qY8J 7M1VexMi3qoyj7v6/djz7OhG5pTCaEaeD5J85B75J/u1usPedtQr+45m1AOkpgyE/NPo1 EWrTXKukm8kSWkoWzi7nMcF1Ku77UMkcw46bIGGadtPGpBHmj1fY8EqJ1hE3lXU5KR0Ig yA9mhkwrwdaUE6PSjHVvcc11aIrouuyl9582E8T0CUWHsvmqA12dHHE8yFquskLjOXA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=WEcJbnQ8; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 1B7BF606A335; Wed, 21 Oct 2026 08:39:44 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=WEcJbnQ8 Received: by smtp.kernel.org (Postfix) with ESMTPSA id 5E14B1F000CF; Wed, 21 Oct 2026 08:39:18 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=korg; t=1792571984; bh=JY1mqR0n6Yfs3COqGT2YRXKLKbUnlNs6WzUFwuL4RGY=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=WEcJbnQ8VDBE1xMQQ9qk7VKGXEbgEs0cLiAsr//5ygsSyrBjz2O0xoZwejUMfycgy 4cvFpcEFPhT2jIaYkQeiXus8UgIzVcWu7+ZYlAj1b3/9ifsx5VbU4iE/FrT2glq45y LS9q8HCVQEWtfQZezdxq6EU9JcWS9yg42fsQ/wqU= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, torvalds@linux-foundation.org, stable@vger.kernel.org Cc: lwn@lwn.net, jslaby@suse.cz, Greg Kroah-Hartman Subject: Re: Linux 6.6.160 Date: Wed, 21 Oct 2026 10:38:14 +0200 Message-ID: <2026102114-hassle-illusive-cac6@gregkh> X-Mailer: git-send-email 2.56.0 In-Reply-To: <2026102114-remodeler-resort-f770@gregkh> References: <2026102114-remodeler-resort-f770@gregkh> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit diff --git a/CREDITS b/CREDITS index f33a33fd2371..3e1cf544c441 100644 --- a/CREDITS +++ b/CREDITS @@ -67,6 +67,11 @@ D: dosfs, LILO, some fd features, ATM, various other hacks here and there S: Buenos Aires S: Argentina +NTFS FILESYSTEM +N: Anton Altaparmakov +E: anton@tuxera.com +D: NTFS filesystem + N: Tim Alpaerts E: tim_alpaerts@toyota-motor-europe.com D: 802.2 class II logical link control layer, diff --git a/Documentation/ABI/testing/sysfs-fs-erofs b/Documentation/ABI/testing/sysfs-fs-erofs index 284224d1b56f..2997379bf804 100644 --- a/Documentation/ABI/testing/sysfs-fs-erofs +++ b/Documentation/ABI/testing/sysfs-fs-erofs @@ -5,7 +5,7 @@ Description: Shows all enabled kernel features. Supported features: zero_padding, compr_cfgs, big_pcluster, chunked_file, device_table, compr_head2, sb_chksum, ztailpacking, - dedupe, fragments. + dedupe, fragments, xattr_prefixes. What: /sys/fs/erofs//sync_decompress Date: November 2021 diff --git a/Documentation/admin-guide/cgroup-v1/memory.rst b/Documentation/admin-guide/cgroup-v1/memory.rst index ff456871bf4b..0eb71ca04687 100644 --- a/Documentation/admin-guide/cgroup-v1/memory.rst +++ b/Documentation/admin-guide/cgroup-v1/memory.rst @@ -47,7 +47,6 @@ Features: - pages are linked to per-memcg LRU exclusively, and there is no global LRU. - optionally, memory+swap usage can be accounted and limited. - hierarchical accounting - - soft limit - moving (recharging) account at moving a task is selectable. - usage threshold notifier - memory pressure notifier @@ -76,8 +75,9 @@ Brief summary of control files. memory.memsw.failcnt show the number of memory+Swap hits limits memory.max_usage_in_bytes show max memory usage recorded memory.memsw.max_usage_in_bytes show max memory+Swap usage recorded - memory.soft_limit_in_bytes set/show soft limit of memory usage - This knob is not available on CONFIG_PREEMPT_RT systems. + memory.soft_limit_in_bytes This knob is deprecated and has no effect. + Writes are ignored and reads always + return the maximum value. memory.stat show various statistics memory.use_hierarchy set/show hierarchical account enabled This knob is deprecated and shouldn't be @@ -332,9 +332,6 @@ memory.kmem.usage_in_bytes, or in a separate counter when it makes sense. The main "kmem" counter is fed into the main counter, so kmem charges will also be visible from the user counter. -Currently no soft limit is implemented for kernel memory. It is future work -to trigger slab reclaim when those limits are reached. - 2.7.1 Current Kernel Memory resources accounted ----------------------------------------------- @@ -695,42 +692,10 @@ For compatibility reasons writing 1 to memory.use_hierarchy will always pass:: 7. Soft limits ============== -Soft limits allow for greater sharing of memory. The idea behind soft limits -is to allow control groups to use as much of the memory as needed, provided - -a. There is no memory contention -b. They do not exceed their hard limit - -When the system detects memory contention or low memory, control groups -are pushed back to their soft limits. If the soft limit of each control -group is very high, they are pushed back as much as possible to make -sure that one control group does not starve the others of memory. - -Please note that soft limits is a best-effort feature; it comes with -no guarantees, but it does its best to make sure that when memory is -heavily contended for, memory is allocated based on the soft limit -hints/setup. Currently soft limit based reclaim is set up such that -it gets invoked from balance_pgdat (kswapd). - -7.1 Interface -------------- - -Soft limits can be setup by using the following commands (in this example we -assume a soft limit of 256 MiB):: - - # echo 256M > memory.soft_limit_in_bytes - -If we want to change this to 1G, we can at any time use:: - - # echo 1G > memory.soft_limit_in_bytes +Writing to memory.soft_limit_in_bytes has no effect and reading it will +always return the maximum value. -.. note:: - Soft limits take effect over a long period of time, since they involve - reclaiming memory for balancing between memory cgroups - -.. note:: - It is recommended to set the soft limit always below the hard limit, - otherwise the hard limit will take precedence. +Use memory.low and memory.min in cgroup v2 instead. .. _cgroup-v1-memory-move-charges: diff --git a/Documentation/arch/arm64/silicon-errata.rst b/Documentation/arch/arm64/silicon-errata.rst index b5b46b6ab277..c40e6271072f 100644 --- a/Documentation/arch/arm64/silicon-errata.rst +++ b/Documentation/arch/arm64/silicon-errata.rst @@ -161,6 +161,8 @@ stable kernels. +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-A725 | #3456106 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ +| ARM | Cortex-A725 | #3821522 | ARM64_ERRATUM_3821522 | ++----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X1 | #3324344 | ARM64_ERRATUM_3194386 | +----------------+-----------------+-----------------+-----------------------------+ | ARM | Cortex-X1 | #4193791 | ARM64_ERRATUM_4118414 | diff --git a/Documentation/filesystems/ntfs.rst b/Documentation/filesystems/ntfs.rst deleted file mode 100644 index 5bb093a26485..000000000000 --- a/Documentation/filesystems/ntfs.rst +++ /dev/null @@ -1,466 +0,0 @@ -.. SPDX-License-Identifier: GPL-2.0 - -================================ -The Linux NTFS filesystem driver -================================ - - -.. Table of contents - - - Overview - - Web site - - Features - - Supported mount options - - Known bugs and (mis-)features - - Using NTFS volume and stripe sets - - The Device-Mapper driver - - The Software RAID / MD driver - - Limitations when using the MD driver - - -Overview -======== - -Linux-NTFS comes with a number of user-space programs known as ntfsprogs. -These include mkntfs, a full-featured ntfs filesystem format utility, -ntfsundelete used for recovering files that were unintentionally deleted -from an NTFS volume and ntfsresize which is used to resize an NTFS partition. -See the web site for more information. - -To mount an NTFS 1.2/3.x (Windows NT4/2000/XP/2003) volume, use the file -system type 'ntfs'. The driver currently supports read-only mode (with no -fault-tolerance, encryption or journalling) and very limited, but safe, write -support. - -For fault tolerance and raid support (i.e. volume and stripe sets), you can -use the kernel's Software RAID / MD driver. See section "Using Software RAID -with NTFS" for details. - - -Web site -======== - -There is plenty of additional information on the linux-ntfs web site -at http://www.linux-ntfs.org/ - -The web site has a lot of additional information, such as a comprehensive -FAQ, documentation on the NTFS on-disk format, information on the Linux-NTFS -userspace utilities, etc. - - -Features -======== - -- This is a complete rewrite of the NTFS driver that used to be in the 2.4 and - earlier kernels. This new driver implements NTFS read support and is - functionally equivalent to the old ntfs driver and it also implements limited - write support. The biggest limitation at present is that files/directories - cannot be created or deleted. See below for the list of write features that - are so far supported. Another limitation is that writing to compressed files - is not implemented at all. Also, neither read nor write access to encrypted - files is so far implemented. -- The new driver has full support for sparse files on NTFS 3.x volumes which - the old driver isn't happy with. -- The new driver supports execution of binaries due to mmap() now being - supported. -- The new driver supports loopback mounting of files on NTFS which is used by - some Linux distributions to enable the user to run Linux from an NTFS - partition by creating a large file while in Windows and then loopback - mounting the file while in Linux and creating a Linux filesystem on it that - is used to install Linux on it. -- A comparison of the two drivers using:: - - time find . -type f -exec md5sum "{}" \; - - run three times in sequence with each driver (after a reboot) on a 1.4GiB - NTFS partition, showed the new driver to be 20% faster in total time elapsed - (from 9:43 minutes on average down to 7:53). The time spent in user space - was unchanged but the time spent in the kernel was decreased by a factor of - 2.5 (from 85 CPU seconds down to 33). -- The driver does not support short file names in general. For backwards - compatibility, we implement access to files using their short file names if - they exist. The driver will not create short file names however, and a - rename will discard any existing short file name. -- The new driver supports exporting of mounted NTFS volumes via NFS. -- The new driver supports async io (aio). -- The new driver supports fsync(2), fdatasync(2), and msync(2). -- The new driver supports readv(2) and writev(2). -- The new driver supports access time updates (including mtime and ctime). -- The new driver supports truncate(2) and open(2) with O_TRUNC. But at present - only very limited support for highly fragmented files, i.e. ones which have - their data attribute split across multiple extents, is included. Another - limitation is that at present truncate(2) will never create sparse files, - since to mark a file sparse we need to modify the directory entry for the - file and we do not implement directory modifications yet. -- The new driver supports write(2) which can both overwrite existing data and - extend the file size so that you can write beyond the existing data. Also, - writing into sparse regions is supported and the holes are filled in with - clusters. But at present only limited support for highly fragmented files, - i.e. ones which have their data attribute split across multiple extents, is - included. Another limitation is that write(2) will never create sparse - files, since to mark a file sparse we need to modify the directory entry for - the file and we do not implement directory modifications yet. - -Supported mount options -======================= - -In addition to the generic mount options described by the manual page for the -mount command (man 8 mount, also see man 5 fstab), the NTFS driver supports the -following mount options: - -======================= ======================================================= -iocharset=name Deprecated option. Still supported but please use - nls=name in the future. See description for nls=name. - -nls=name Character set to use when returning file names. - Unlike VFAT, NTFS suppresses names that contain - unconvertible characters. Note that most character - sets contain insufficient characters to represent all - possible Unicode characters that can exist on NTFS. - To be sure you are not missing any files, you are - advised to use nls=utf8 which is capable of - representing all Unicode characters. - -utf8= Option no longer supported. Currently mapped to - nls=utf8 but please use nls=utf8 in the future and - make sure utf8 is compiled either as module or into - the kernel. See description for nls=name. - -uid= -gid= -umask= Provide default owner, group, and access mode mask. - These options work as documented in mount(8). By - default, the files/directories are owned by root and - he/she has read and write permissions, as well as - browse permission for directories. No one else has any - access permissions. I.e. the mode on all files is by - default rw------- and for directories rwx------, a - consequence of the default fmask=0177 and dmask=0077. - Using a umask of zero will grant all permissions to - everyone, i.e. all files and directories will have mode - rwxrwxrwx. - -fmask= -dmask= Instead of specifying umask which applies both to - files and directories, fmask applies only to files and - dmask only to directories. - -sloppy= If sloppy is specified, ignore unknown mount options. - Otherwise the default behaviour is to abort mount if - any unknown options are found. - -show_sys_files= If show_sys_files is specified, show the system files - in directory listings. Otherwise the default behaviour - is to hide the system files. - Note that even when show_sys_files is specified, "$MFT" - will not be visible due to bugs/mis-features in glibc. - Further, note that irrespective of show_sys_files, all - files are accessible by name, i.e. you can always do - "ls -l \$UpCase" for example to specifically show the - system file containing the Unicode upcase table. - -case_sensitive= If case_sensitive is specified, treat all file names as - case sensitive and create file names in the POSIX - namespace. Otherwise the default behaviour is to treat - file names as case insensitive and to create file names - in the WIN32/LONG name space. Note, the Linux NTFS - driver will never create short file names and will - remove them on rename/delete of the corresponding long - file name. - Note that files remain accessible via their short file - name, if it exists. If case_sensitive, you will need - to provide the correct case of the short file name. - -disable_sparse= If disable_sparse is specified, creation of sparse - regions, i.e. holes, inside files is disabled for the - volume (for the duration of this mount only). By - default, creation of sparse regions is enabled, which - is consistent with the behaviour of traditional Unix - filesystems. - -errors=opt What to do when critical filesystem errors are found. - Following values can be used for "opt": - - ======== ========================================= - continue DEFAULT, try to clean-up as much as - possible, e.g. marking a corrupt inode as - bad so it is no longer accessed, and then - continue. - recover At present only supported is recovery of - the boot sector from the backup copy. - If read-only mount, the recovery is done - in memory only and not written to disk. - ======== ========================================= - - Note that the options are additive, i.e. specifying:: - - errors=continue,errors=recover - - means the driver will attempt to recover and if that - fails it will clean-up as much as possible and - continue. - -mft_zone_multiplier= Set the MFT zone multiplier for the volume (this - setting is not persistent across mounts and can be - changed from mount to mount but cannot be changed on - remount). Values of 1 to 4 are allowed, 1 being the - default. The MFT zone multiplier determines how much - space is reserved for the MFT on the volume. If all - other space is used up, then the MFT zone will be - shrunk dynamically, so this has no impact on the - amount of free space. However, it can have an impact - on performance by affecting fragmentation of the MFT. - In general use the default. If you have a lot of small - files then use a higher value. The values have the - following meaning: - - ===== ================================= - Value MFT zone size (% of volume size) - ===== ================================= - 1 12.5% - 2 25% - 3 37.5% - 4 50% - ===== ================================= - - Note this option is irrelevant for read-only mounts. -======================= ======================================================= - - -Known bugs and (mis-)features -============================= - -- The link count on each directory inode entry is set to 1, due to Linux not - supporting directory hard links. This may well confuse some user space - applications, since the directory names will have the same inode numbers. - This also speeds up ntfs_read_inode() immensely. And we haven't found any - problems with this approach so far. If you find a problem with this, please - let us know. - - -Please send bug reports/comments/feedback/abuse to the Linux-NTFS development -list at sourceforge: linux-ntfs-dev@lists.sourceforge.net - - -Using NTFS volume and stripe sets -================================= - -For support of volume and stripe sets, you can either use the kernel's -Device-Mapper driver or the kernel's Software RAID / MD driver. The former is -the recommended one to use for linear raid. But the latter is required for -raid level 5. For striping and mirroring, either driver should work fine. - - -The Device-Mapper driver ------------------------- - -You will need to create a table of the components of the volume/stripe set and -how they fit together and load this into the kernel using the dmsetup utility -(see man 8 dmsetup). - -Linear volume sets, i.e. linear raid, has been tested and works fine. Even -though untested, there is no reason why stripe sets, i.e. raid level 0, and -mirrors, i.e. raid level 1 should not work, too. Stripes with parity, i.e. -raid level 5, unfortunately cannot work yet because the current version of the -Device-Mapper driver does not support raid level 5. You may be able to use the -Software RAID / MD driver for raid level 5, see the next section for details. - -To create the table describing your volume you will need to know each of its -components and their sizes in sectors, i.e. multiples of 512-byte blocks. - -For NT4 fault tolerant volumes you can obtain the sizes using fdisk. So for -example if one of your partitions is /dev/hda2 you would do:: - - $ fdisk -ul /dev/hda - - Disk /dev/hda: 81.9 GB, 81964302336 bytes - 255 heads, 63 sectors/track, 9964 cylinders, total 160086528 sectors - Units = sectors of 1 * 512 = 512 bytes - - Device Boot Start End Blocks Id System - /dev/hda1 * 63 4209029 2104483+ 83 Linux - /dev/hda2 4209030 37768814 16779892+ 86 NTFS - /dev/hda3 37768815 46170809 4200997+ 83 Linux - -And you would know that /dev/hda2 has a size of 37768814 - 4209030 + 1 = -33559785 sectors. - -For Win2k and later dynamic disks, you can for example use the ldminfo utility -which is part of the Linux LDM tools (the latest version at the time of -writing is linux-ldm-0.0.8.tar.bz2). You can download it from: - - http://www.linux-ntfs.org/ - -Simply extract the downloaded archive (tar xvjf linux-ldm-0.0.8.tar.bz2), go -into it (cd linux-ldm-0.0.8) and change to the test directory (cd test). You -will find the precompiled (i386) ldminfo utility there. NOTE: You will not be -able to compile this yourself easily so use the binary version! - -Then you would use ldminfo in dump mode to obtain the necessary information:: - - $ ./ldminfo --dump /dev/hda - -This would dump the LDM database found on /dev/hda which describes all of your -dynamic disks and all the volumes on them. At the bottom you will see the -VOLUME DEFINITIONS section which is all you really need. You may need to look -further above to determine which of the disks in the volume definitions is -which device in Linux. Hint: Run ldminfo on each of your dynamic disks and -look at the Disk Id close to the top of the output for each (the PRIVATE HEADER -section). You can then find these Disk Ids in the VBLK DATABASE section in the - components where you will get the LDM Name for the disk that is found in -the VOLUME DEFINITIONS section. - -Note you will also need to enable the LDM driver in the Linux kernel. If your -distribution did not enable it, you will need to recompile the kernel with it -enabled. This will create the LDM partitions on each device at boot time. You -would then use those devices (for /dev/hda they would be /dev/hda1, 2, 3, etc) -in the Device-Mapper table. - -You can also bypass using the LDM driver by using the main device (e.g. -/dev/hda) and then using the offsets of the LDM partitions into this device as -the "Start sector of device" when creating the table. Once again ldminfo would -give you the correct information to do this. - -Assuming you know all your devices and their sizes things are easy. - -For a linear raid the table would look like this (note all values are in -512-byte sectors):: - - # Offset into Size of this Raid type Device Start sector - # volume device of device - 0 1028161 linear /dev/hda1 0 - 1028161 3903762 linear /dev/hdb2 0 - 4931923 2103211 linear /dev/hdc1 0 - -For a striped volume, i.e. raid level 0, you will need to know the chunk size -you used when creating the volume. Windows uses 64kiB as the default, so it -will probably be this unless you changes the defaults when creating the array. - -For a raid level 0 the table would look like this (note all values are in -512-byte sectors):: - - # Offset Size Raid Number Chunk 1st Start 2nd Start - # into of the type of size Device in Device in - # volume volume stripes device device - 0 2056320 striped 2 128 /dev/hda1 0 /dev/hdb1 0 - -If there are more than two devices, just add each of them to the end of the -line. - -Finally, for a mirrored volume, i.e. raid level 1, the table would look like -this (note all values are in 512-byte sectors):: - - # Ofs Size Raid Log Number Region Should Number Source Start Target Start - # in of the type type of log size sync? of Device in Device in - # vol volume params mirrors Device Device - 0 2056320 mirror core 2 16 nosync 2 /dev/hda1 0 /dev/hdb1 0 - -If you are mirroring to multiple devices you can specify further targets at the -end of the line. - -Note the "Should sync?" parameter "nosync" means that the two mirrors are -already in sync which will be the case on a clean shutdown of Windows. If the -mirrors are not clean, you can specify the "sync" option instead of "nosync" -and the Device-Mapper driver will then copy the entirety of the "Source Device" -to the "Target Device" or if you specified multiple target devices to all of -them. - -Once you have your table, save it in a file somewhere (e.g. /etc/ntfsvolume1), -and hand it over to dmsetup to work with, like so:: - - $ dmsetup create myvolume1 /etc/ntfsvolume1 - -You can obviously replace "myvolume1" with whatever name you like. - -If it all worked, you will now have the device /dev/device-mapper/myvolume1 -which you can then just use as an argument to the mount command as usual to -mount the ntfs volume. For example:: - - $ mount -t ntfs -o ro /dev/device-mapper/myvolume1 /mnt/myvol1 - -(You need to create the directory /mnt/myvol1 first and of course you can use -anything you like instead of /mnt/myvol1 as long as it is an existing -directory.) - -It is advisable to do the mount read-only to see if the volume has been setup -correctly to avoid the possibility of causing damage to the data on the ntfs -volume. - - -The Software RAID / MD driver ------------------------------ - -An alternative to using the Device-Mapper driver is to use the kernel's -Software RAID / MD driver. For which you need to set up your /etc/raidtab -appropriately (see man 5 raidtab). - -Linear volume sets, i.e. linear raid, as well as stripe sets, i.e. raid level -0, have been tested and work fine (though see section "Limitations when using -the MD driver with NTFS volumes" especially if you want to use linear raid). -Even though untested, there is no reason why mirrors, i.e. raid level 1, and -stripes with parity, i.e. raid level 5, should not work, too. - -You have to use the "persistent-superblock 0" option for each raid-disk in the -NTFS volume/stripe you are configuring in /etc/raidtab as the persistent -superblock used by the MD driver would damage the NTFS volume. - -Windows by default uses a stripe chunk size of 64k, so you probably want the -"chunk-size 64k" option for each raid-disk, too. - -For example, if you have a stripe set consisting of two partitions /dev/hda5 -and /dev/hdb1 your /etc/raidtab would look like this:: - - raiddev /dev/md0 - raid-level 0 - nr-raid-disks 2 - nr-spare-disks 0 - persistent-superblock 0 - chunk-size 64k - device /dev/hda5 - raid-disk 0 - device /dev/hdb1 - raid-disk 1 - -For linear raid, just change the raid-level above to "raid-level linear", for -mirrors, change it to "raid-level 1", and for stripe sets with parity, change -it to "raid-level 5". - -Note for stripe sets with parity you will also need to tell the MD driver -which parity algorithm to use by specifying the option "parity-algorithm -which", where you need to replace "which" with the name of the algorithm to -use (see man 5 raidtab for available algorithms) and you will have to try the -different available algorithms until you find one that works. Make sure you -are working read-only when playing with this as you may damage your data -otherwise. If you find which algorithm works please let us know (email the -linux-ntfs developers list linux-ntfs-dev@lists.sourceforge.net or drop in on -IRC in channel #ntfs on the irc.freenode.net network) so we can update this -documentation. - -Once the raidtab is setup, run for example raid0run -a to start all devices or -raid0run /dev/md0 to start a particular md device, in this case /dev/md0. - -Then just use the mount command as usual to mount the ntfs volume using for -example:: - - mount -t ntfs -o ro /dev/md0 /mnt/myntfsvolume - -It is advisable to do the mount read-only to see if the md volume has been -setup correctly to avoid the possibility of causing damage to the data on the -ntfs volume. - - -Limitations when using the Software RAID / MD driver ------------------------------------------------------ - -Using the md driver will not work properly if any of your NTFS partitions have -an odd number of sectors. This is especially important for linear raid as all -data after the first partition with an odd number of sectors will be offset by -one or more sectors so if you mount such a partition with write support you -will cause massive damage to the data on the volume which will only become -apparent when you try to use the volume again under Windows. - -So when using linear raid, make sure that all your partitions have an even -number of sectors BEFORE attempting to use it. You have been warned! - -Even better is to simply use the Device-Mapper for linear raid and then you do -not have this problem with odd numbers of sectors. diff --git a/Documentation/netlink/specs/rt_link.yaml b/Documentation/netlink/specs/rt_link.yaml index a8a1466adf17..9bf17d27db0a 100644 --- a/Documentation/netlink/specs/rt_link.yaml +++ b/Documentation/netlink/specs/rt_link.yaml @@ -767,6 +767,8 @@ attribute-sets: - name: txqlen type: u32 + checks: + max: 32767 - name: map type: binary diff --git a/Documentation/networking/strparser.rst b/Documentation/networking/strparser.rst index 7f623d1db72a..29144a9c4e83 100644 --- a/Documentation/networking/strparser.rst +++ b/Documentation/networking/strparser.rst @@ -77,7 +77,7 @@ Functions strp_done is called to release any resources held by the stream parser instance. This must be called after the stream processor - has been stopped. + has been stopped. The socket lock must not be held. :: diff --git a/Documentation/virt/kvm/api.rst b/Documentation/virt/kvm/api.rst index 21a7578142a1..0f7177c7653b 100644 --- a/Documentation/virt/kvm/api.rst +++ b/Documentation/virt/kvm/api.rst @@ -1363,6 +1363,12 @@ Instead, an abort (data abort if the cause of the page-table update was a load or a store, instruction abort if it was an instruction fetch) is injected in the guest. +S390: +^^^^^ + +Returns -EINVAL if the VM has the KVM_VM_S390_UCONTROL flag set. +Returns -EINVAL if called on a protected VM. + 4.36 KVM_SET_TSS_ADDR --------------------- diff --git a/MAINTAINERS b/MAINTAINERS index 8b1ee5b11eca..48f89f4638fc 100644 --- a/MAINTAINERS +++ b/MAINTAINERS @@ -5280,6 +5280,8 @@ S: Maintained T: git git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup.git F: Documentation/admin-guide/cgroup-v1/cpusets.rst F: include/linux/cpuset.h +F: kernel/cgroup/cpuset-internal.h +F: kernel/cgroup/cpuset-v1.c F: kernel/cgroup/cpuset.c F: tools/testing/selftests/cgroup/test_cpuset.c F: tools/testing/selftests/cgroup/test_cpuset_prs.sh @@ -15202,16 +15204,6 @@ W: https://github.com/davejiang/linux/wiki T: git https://github.com/davejiang/linux.git F: drivers/ntb/hw/intel/ -NTFS FILESYSTEM -M: Anton Altaparmakov -R: Namjae Jeon -L: linux-ntfs-dev@lists.sourceforge.net -S: Supported -W: http://www.tuxera.com/ -T: git git://git.kernel.org/pub/scm/linux/kernel/git/aia21/ntfs.git -F: Documentation/filesystems/ntfs.rst -F: fs/ntfs/ - NTFS3 FILESYSTEM M: Konstantin Komarov L: ntfs3@lists.linux.dev diff --git a/Makefile b/Makefile index 59b8cecf471f..0b151a07ee79 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 VERSION = 6 PATCHLEVEL = 6 -SUBLEVEL = 159 +SUBLEVEL = 160 EXTRAVERSION = NAME = Pinguïn Aangedreven @@ -754,12 +754,6 @@ endif # KBUILD_EXTMOD # Defaults to vmlinux, but the arch makefile usually adds further targets all: vmlinux -CFLAGS_GCOV := -fprofile-arcs -ftest-coverage -ifdef CONFIG_CC_IS_GCC -CFLAGS_GCOV += -fno-tree-loop-im -endif -export CFLAGS_GCOV - # The arch Makefiles can override CC_FLAGS_FTRACE. We may also append it later. ifdef CONFIG_FUNCTION_TRACER CC_FLAGS_FTRACE := -pg @@ -1007,6 +1001,27 @@ endif # Ensure compilers do not transform certain loops into calls to wcslen() KBUILD_CFLAGS += -fno-builtin-wcslen +CFLAGS_GCOV := -fprofile-arcs -ftest-coverage +ifdef CONFIG_CC_IS_GCC +CFLAGS_GCOV += -fno-tree-loop-im +# Use atomic counter updates to avoid concurrent-access crashes in GCOV. +# Only enable if -fprofile-update=prefer-atomic does not introduce new +# undefined symbols (e.g. libatomic calls that the kernel cannot link). +CFLAGS_GCOV += $(call try-run,\ + echo 'long long x; void f(void){x++;}' | \ + $(CC) $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) -w -fprofile-arcs \ + -ftest-coverage -x c - -c -o "$$TMP.base" && \ + echo 'long long x; void f(void){x++;}' | \ + $(CC) $(KBUILD_CPPFLAGS) $(KBUILD_CFLAGS) -w -fprofile-arcs \ + -ftest-coverage -fprofile-update=prefer-atomic \ + -x c - -c -o "$$TMP" && \ + $(NM) "$$TMP.base" | grep ' U ' > "$$TMP.ubase" || true ; \ + $(NM) "$$TMP" | grep ' U ' > "$$TMP.utest" || true ; \ + cmp -s "$$TMP.ubase" "$$TMP.utest",\ + -fprofile-update=prefer-atomic) +endif +export CFLAGS_GCOV + # change __FILE__ to the relative path from the srctree KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=) diff --git a/arch/alpha/include/asm/pgtable.h b/arch/alpha/include/asm/pgtable.h index 02e8817a8921..8b620551304e 100644 --- a/arch/alpha/include/asm/pgtable.h +++ b/arch/alpha/include/asm/pgtable.h @@ -17,6 +17,7 @@ #include /* For TASK_SIZE */ #include #include +#include struct mm_struct; struct vm_area_struct; @@ -213,6 +214,9 @@ extern inline void pud_set(pud_t * pudp, pmd_t * pmdp) { pud_val(*pudp) = _PAGE_TABLE | ((((unsigned long) pmdp) - PAGE_OFFSET) << (32-PAGE_SHIFT)); } +extern void migrate_flush_tlb_page(struct vm_area_struct *vma, + unsigned long addr); + extern inline unsigned long pmd_page_vaddr(pmd_t pmd) { @@ -232,7 +236,7 @@ extern inline int pte_none(pte_t pte) { return !pte_val(pte); } extern inline int pte_present(pte_t pte) { return pte_val(pte) & _PAGE_VALID; } extern inline void pte_clear(struct mm_struct *mm, unsigned long addr, pte_t *ptep) { - pte_val(*ptep) = 0; + WRITE_ONCE(pte_val(*ptep), 0); } extern inline int pmd_none(pmd_t pmd) { return !pmd_val(pmd); } @@ -294,6 +298,33 @@ extern inline pte_t * pte_offset_kernel(pmd_t * dir, unsigned long address) extern pgd_t swapper_pg_dir[1024]; +#ifdef CONFIG_COMPACTION +#define __HAVE_ARCH_PTEP_GET_AND_CLEAR + +static inline pte_t ptep_get_and_clear(struct mm_struct *mm, + unsigned long address, + pte_t *ptep) +{ + pte_t pte = READ_ONCE(*ptep); + + pte_clear(mm, address, ptep); + return pte; +} + +#define __HAVE_ARCH_PTEP_CLEAR_FLUSH + +static inline pte_t ptep_clear_flush(struct vm_area_struct *vma, + unsigned long addr, pte_t *ptep) +{ + struct mm_struct *mm = vma->vm_mm; + pte_t pte = ptep_get_and_clear(mm, addr, ptep); + + page_table_check_pte_clear(mm, pte); + migrate_flush_tlb_page(vma, addr); + return pte; +} + +#endif /* * The Alpha doesn't have any external MMU info: the kernel page * tables contain all the necessary information. diff --git a/arch/alpha/include/asm/tlbflush.h b/arch/alpha/include/asm/tlbflush.h index 94dc37cf873a..e78de2456ffa 100644 --- a/arch/alpha/include/asm/tlbflush.h +++ b/arch/alpha/include/asm/tlbflush.h @@ -91,7 +91,9 @@ flush_tlb_other(struct mm_struct *mm) unsigned long *mmc = &mm->context[smp_processor_id()]; /* Check it's not zero first to avoid cacheline ping pong when possible. */ - if (*mmc) *mmc = 0; + + if (READ_ONCE(*mmc)) + WRITE_ONCE(*mmc, 0); } #ifndef CONFIG_SMP diff --git a/arch/alpha/lib/Makefile b/arch/alpha/lib/Makefile index 1cc74f7b50ef..6a779b9018fd 100644 --- a/arch/alpha/lib/Makefile +++ b/arch/alpha/lib/Makefile @@ -4,7 +4,6 @@ # asflags-y := $(KBUILD_CFLAGS) -ccflags-y := -Werror # Many of these routines have implementations tuned for ev6. # Choose them iff we're targeting ev6 specifically. diff --git a/arch/alpha/mm/Makefile b/arch/alpha/mm/Makefile index bd770302eb82..2d05664058f6 100644 --- a/arch/alpha/mm/Makefile +++ b/arch/alpha/mm/Makefile @@ -3,6 +3,4 @@ # Makefile for the linux alpha-specific parts of the memory manager. # -ccflags-y := -Werror - -obj-y := init.o fault.o +obj-y := init.o fault.o tlbflush.o diff --git a/arch/alpha/mm/tlbflush.c b/arch/alpha/mm/tlbflush.c new file mode 100644 index 000000000000..ccbc317b9a34 --- /dev/null +++ b/arch/alpha/mm/tlbflush.c @@ -0,0 +1,112 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * Alpha TLB shootdown helpers + * + * Copyright (C) 2025 Magnus Lindholm + * + * Alpha-specific TLB flush helpers that cannot be expressed purely + * as inline functions. + * + * These helpers provide combined MM context handling (ASN rollover) + * and immediate TLB invalidation for page migration and memory + * compaction paths, where lazy shootdowns are insufficient. + */ + +#include +#include +#include +#include +#include +#include + +#define asn_locked() (cpu_data[smp_processor_id()].asn_lock) + +/* + * Migration/compaction helper: combine mm context (ASN) handling with an + * immediate per-page TLB invalidate and (for exec) an instruction barrier. + * + * This mirrors the SMP combined IPI handler semantics, but runs locally on UP. + */ +#ifndef CONFIG_SMP +void migrate_flush_tlb_page(struct vm_area_struct *vma, + unsigned long addr) +{ + struct mm_struct *mm = vma->vm_mm; + int tbi_type = (vma->vm_flags & VM_EXEC) ? 3 : 2; + + /* + * First do the mm-context side: + * If we're currently running this mm, reload a fresh context ASN. + * Otherwise, mark context invalid. + * + * On UP, this is mostly about matching the SMP semantics and ensuring + * exec/i-cache tagging assumptions hold when compaction migrates pages. + */ + if (mm == current->active_mm) + flush_tlb_current(mm); + else + flush_tlb_other(mm); + + /* + * Then do the immediate translation kill for this VA. + * For exec mappings, order instruction fetch after invalidation. + */ + tbi(tbi_type, addr); +} + +#else +struct tlb_mm_and_addr { + struct mm_struct *mm; + unsigned long addr; + int tbi_type; /* 2 = DTB, 3 = ITB+DTB */ +}; + +static void ipi_flush_mm_and_page(void *x) +{ + struct tlb_mm_and_addr *d = x; + + /* Part 1: mm context side (Alpha uses ASN/context as a key mechanism). */ + if (d->mm == current->active_mm && !asn_locked()) + __load_new_mm_context(d->mm); + else + flush_tlb_other(d->mm); + + /* Part 2: immediate per-VA invalidation on this CPU. */ + tbi(d->tbi_type, d->addr); +} + +void migrate_flush_tlb_page(struct vm_area_struct *vma, unsigned long addr) +{ + struct mm_struct *mm = vma->vm_mm; + struct tlb_mm_and_addr d = { + .mm = mm, + .addr = addr, + .tbi_type = (vma->vm_flags & VM_EXEC) ? 3 : 2, + }; + + /* + * One synchronous rendezvous: every CPU runs ipi_flush_mm_and_page(). + * This is the "combined" version of flush_tlb_mm + per-page invalidate. + */ + preempt_disable(); + on_each_cpu(ipi_flush_mm_and_page, &d, 1); + + /* + * mimic flush_tlb_mm()'s mm_users<=1 optimization. + */ + if (atomic_read(&mm->mm_users) <= 1) { + + int cpu, this_cpu; + this_cpu = smp_processor_id(); + + for (cpu = 0; cpu < NR_CPUS; cpu++) { + if (!cpu_online(cpu) || cpu == this_cpu) + continue; + if (READ_ONCE(mm->context[cpu])) + WRITE_ONCE(mm->context[cpu], 0); + } + } + preempt_enable(); +} + +#endif diff --git a/arch/arm/mm/dump.c b/arch/arm/mm/dump.c index a9381095ab36..cd032522d902 100644 --- a/arch/arm/mm/dump.c +++ b/arch/arm/mm/dump.c @@ -349,12 +349,12 @@ static void walk_pmd(struct pg_state *st, pud_t *pud, unsigned long start) for (i = 0; i < PTRS_PER_PMD; i++, pmd++) { addr = start + i * PMD_SIZE; domain = get_domain_name(pmd); - if (pmd_none(*pmd) || pmd_large(*pmd) || !pmd_present(*pmd)) + if (pmd_none(*pmd) || pmd_leaf(*pmd) || !pmd_present(*pmd)) note_page(st, addr, 4, pmd_val(*pmd), domain); else walk_pte(st, pmd, addr, domain); - if (SECTION_SIZE < PMD_SIZE && pmd_large(pmd[1])) { + if (SECTION_SIZE < PMD_SIZE && pmd_leaf(pmd[1])) { addr += SECTION_SIZE; pmd++; domain = get_domain_name(pmd); diff --git a/arch/arm/net/bpf_jit_32.c b/arch/arm/net/bpf_jit_32.c index 2292245cfe7a..8adc45ad1b2a 100644 --- a/arch/arm/net/bpf_jit_32.c +++ b/arch/arm/net/bpf_jit_32.c @@ -1988,28 +1988,21 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) /* If building the body of the JITed code fails somehow, * we fall back to the interpretation. */ - if (build_body(&ctx) < 0) { - image_ptr = NULL; - bpf_jit_binary_free(header); - prog = orig_prog; - goto out_imms; - } + if (build_body(&ctx) < 0) + goto out_free; build_epilogue(&ctx); /* 3.) Extra pass to validate JITed Code */ - if (validate_code(&ctx)) { - image_ptr = NULL; - bpf_jit_binary_free(header); - prog = orig_prog; - goto out_imms; - } + if (validate_code(&ctx)) + goto out_free; flush_icache_range((u32)header, (u32)(ctx.target + ctx.idx)); if (bpf_jit_enable > 1) /* there are 2 passes here */ bpf_jit_dump(prog->len, image_size, 2, ctx.target); - bpf_jit_binary_lock_ro(header); + if (bpf_jit_binary_lock_ro(header)) + goto out_free; prog->bpf_func = (void *)ctx.target; prog->jited = 1; prog->jited_len = image_size; @@ -2026,5 +2019,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_jit_prog_release_other(prog, prog == orig_prog ? tmp : orig_prog); return prog; + +out_free: + image_ptr = NULL; + bpf_jit_binary_free(header); + prog = orig_prog; + goto out_imms; } diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 723e73042c1a..fc29ac6bad57 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -1004,9 +1004,17 @@ config ARM64_ERRATUM_1902691 If unsure, say Y. +config ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT + bool + # Targeting errata affecting AMEVCNTR01, the AMU constant counter. + # On affected CPUs the counter may not increment at the expected rate, + # making it unreliable. As a result, AMU users should treat it as + # unavailable rather than relying on misleading counter values. + config ARM64_ERRATUM_2457168 bool "Cortex-A510: 2457168: workaround for AMEVCNTR01 incrementing incorrectly" depends on ARM64_AMU_EXTN + select ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT default y help This option adds the workaround for ARM Cortex-A510 erratum 2457168. @@ -1037,6 +1045,28 @@ config ARM64_ERRATUM_2645198 If unsure, say Y. +config ARM64_ERRATUM_3821522 + bool "Cortex-A725: 3821522: workaround for possible CNT_CYCLES increment error due to WFE/WFI" + depends on ARM64_AMU_EXTN + select ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT + default y + help + This option adds the workaround for ARM Cortex-A725 erratum 3821522. + + On affected A725 cores, the CNT_CYCLES event may incur a significant + increment error when entering and subsequently exiting WFx. + As a result, the CNT_CYCLES may diverge from the system counter + frequency at which it is expected to increment. + This renders the AMU counter AMEVCNTR01, that implements CNT_CYCLES, + being unreliable and unsuitable for use. + + Since there is no hardware workaround, reads of the affected CNT_CYCLES + counter return 0 in the relevant paths. This causes users of the counter + to treat it as unavailable and is functionally equivalent to firmware + disabling the affected counter. + + If unsure, say Y. + config ARM64_WORKAROUND_SPECULATIVE_UNPRIV_LOAD bool diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile index d95b3d6b471a..9cd82752acf3 100644 --- a/arch/arm64/kernel/Makefile +++ b/arch/arm64/kernel/Makefile @@ -25,6 +25,9 @@ KASAN_SANITIZE_stacktrace.o := n KCOV_INSTRUMENT_entry-common.o := n KCOV_INSTRUMENT_idle.o := n +# Softirq stack switching can run outside interrupt context. +KCOV_INSTRUMENT_irq.o := n + # Object file lists. obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ entry-common.o entry-fpsimd.o process.o ptrace.o \ diff --git a/arch/arm64/kernel/cpu_errata.c b/arch/arm64/kernel/cpu_errata.c index 03a438279909..871e440f386a 100644 --- a/arch/arm64/kernel/cpu_errata.c +++ b/arch/arm64/kernel/cpu_errata.c @@ -275,6 +275,20 @@ static const struct arm64_cpu_capabilities arm64_repeat_tlbi_list[] = { }; #endif +#ifdef CONFIG_ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT +static const struct midr_range workaround_amu_constcnt_list[] = { +#ifdef CONFIG_ARM64_ERRATUM_2457168 + /* Cortex-A510 r0p0-r1p1 */ + MIDR_RANGE(MIDR_CORTEX_A510, 0, 0, 1, 1), +#endif +#ifdef CONFIG_ARM64_ERRATUM_3821522 + /* Cortex-A725 r0p0 - r0p2 */ + MIDR_RANGE(MIDR_CORTEX_A725, 0, 0, 0, 2), +#endif + {} +}; +#endif /* CONFIG_ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT */ + #ifdef CONFIG_CAVIUM_ERRATUM_23154 static const struct midr_range cavium_erratum_23154_cpus[] = { MIDR_ALL_VERSIONS(MIDR_THUNDERX), @@ -766,14 +780,12 @@ const struct arm64_cpu_capabilities arm64_errata[] = { ERRATA_MIDR_REV_RANGE(MIDR_CORTEX_A510, 0, 0, 2) }, #endif -#ifdef CONFIG_ARM64_ERRATUM_2457168 +#ifdef CONFIG_ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT { - .desc = "ARM erratum 2457168", - .capability = ARM64_WORKAROUND_2457168, + .desc = "Broken AMU AMEVCNTR01 (const counter)", + .capability = ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT, .type = ARM64_CPUCAP_WEAK_LOCAL_CPU_FEATURE, - - /* Cortex-A510 r0p0-r1p1 */ - CAP_MIDR_RANGE(MIDR_CORTEX_A510, 0, 0, 1, 1) + CAP_MIDR_RANGE_LIST(workaround_amu_constcnt_list) }, #endif #ifdef CONFIG_ARM64_ERRATUM_2038923 diff --git a/arch/arm64/kernel/cpufeature.c b/arch/arm64/kernel/cpufeature.c index 5db25bfb854f..b85d6058a347 100644 --- a/arch/arm64/kernel/cpufeature.c +++ b/arch/arm64/kernel/cpufeature.c @@ -1950,7 +1950,7 @@ static void cpu_amu_enable(struct arm64_cpu_capabilities const *cap) cpumask_set_cpu(smp_processor_id(), &amu_cpus); /* 0 reference values signal broken/disabled counters */ - if (!this_cpu_has_cap(ARM64_WORKAROUND_2457168)) + if (!this_cpu_has_cap(ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT)) update_freq_counters_refs(); } } diff --git a/arch/arm64/kernel/ptrace.c b/arch/arm64/kernel/ptrace.c index c30f5a70bd18..69854a0a6301 100644 --- a/arch/arm64/kernel/ptrace.c +++ b/arch/arm64/kernel/ptrace.c @@ -559,6 +559,42 @@ static int gpr_get(struct task_struct *target, return membuf_write(&to, uregs, sizeof(*uregs)); } +static void update_syscall_orig_x0_after_ptrace(struct task_struct *target) +{ + struct pt_regs *regs = task_pt_regs(target); + struct kernel_siginfo *info = target->last_siginfo; + + /* + * Skip the update for NO_SYSCALL (set either by the user or the + * tracer), as regs[0] holds the return value (see the comment in + * el0_svc_common()) and can be unwound using syscall_rollback(). + */ + if (regs->syscallno == NO_SYSCALL) + return; + + /* We should only be called when target is in a ptrace stop */ + if (WARN_ON_ONCE(!info)) + return; + + /* + * For compat tasks, orig_r0 is provided directly through GPR index + * 17. + */ + if (is_compat_thread(task_thread_info(target))) + return; + + /* + * Don't update orig_x0 for a syscall-exit-stop, as x0 now contains the + * return value of the system call. + */ + if ((info->si_code & ~0x80) == SIGTRAP && + target->ptrace_message == PTRACE_EVENTMSG_SYSCALL_EXIT) { + return; + } + + regs->orig_x0 = regs->regs[0]; +} + static int gpr_set(struct task_struct *target, const struct user_regset *regset, unsigned int pos, unsigned int count, const void *kbuf, const void __user *ubuf) @@ -574,6 +610,14 @@ static int gpr_set(struct task_struct *target, const struct user_regset *regset, return -EINVAL; task_pt_regs(target)->user_regs = newregs; + + /* + * Keep orig_x0 authoritative so that seccomp (via + * syscall_get_arguments()), audit and the restart path all see the same + * first argument the syscall is dispatched with, even if it has been + * updated by a tracer. + */ + update_syscall_orig_x0_after_ptrace(target); return 0; } @@ -717,6 +761,12 @@ static int system_call_set(struct task_struct *target, return ret; task_pt_regs(target)->syscallno = syscallno; + + /* + * Re-sync orig_x0 in case the syscall number has been changed + * from NO_SYSCALL. + */ + update_syscall_orig_x0_after_ptrace(target); return ret; } diff --git a/arch/arm64/kernel/signal.c b/arch/arm64/kernel/signal.c index 032afb40dfa7..506d4b9f725c 100644 --- a/arch/arm64/kernel/signal.c +++ b/arch/arm64/kernel/signal.c @@ -286,6 +286,7 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user) unsigned int vl, vq; struct user_fpsimd_state fpsimd; u16 user_vl, flags; + bool fpsimd_only; bool sm; if (user->sve_size < sizeof(*user->sve)) @@ -296,19 +297,33 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user) if (err) return err; + fpsimd_only = (user->sve_size == sizeof(*user->sve)); sm = flags & SVE_SIG_FLAG_SM; + if (sm) { if (!system_supports_sme()) return -EINVAL; + /* + * Streaming SVE state is always preserved with an SVE payload. + * Only accept streaming state which has an SVE payload. + */ + if (fpsimd_only) + return -EINVAL; + vl = task_get_sme_vl(current); } else { /* - * A SME only system use SVE for streaming mode so can - * have a SVE formatted context with a zero VL and no - * payload data. + * Non-streaming SVE state may be preserved without an SVE + * payload, in which case all state is saved in the FPSIMD + * context. + * + * On SME-only systems, non-streaming (FPSIMD-only) state is + * always preserved without an SVE payload, and with VL==0. On + * such systems, only accept non-streaming state without an SVE + * payload. */ - if (!system_supports_sve() && !system_supports_sme()) + if (!system_supports_sve() && !fpsimd_only) return -EINVAL; vl = task_get_sve_vl(current); @@ -317,16 +332,7 @@ static int restore_sve_fpsimd_context(struct user_ctxs *user) if (user_vl != vl) return -EINVAL; - /* - * Non-streaming SVE state may be preserved without an SVE payload, in - * which case the SVE context only has a header with VL==0, and all - * state can be restored from the FPSIMD context. - * - * Streaming SVE state is always preserved with an SVE payload. For - * consistency and robustness, reject restoring streaming SVE state - * without an SVE payload. - */ - if (!sm && user->sve_size == sizeof(*user->sve)) + if (fpsimd_only) return restore_fpsimd_context(user); vq = sve_vq_from_vl(vl); diff --git a/arch/arm64/kernel/topology.c b/arch/arm64/kernel/topology.c index 1a2c72f3e7f8..24a8b61a4f46 100644 --- a/arch/arm64/kernel/topology.c +++ b/arch/arm64/kernel/topology.c @@ -268,12 +268,13 @@ static void cpu_read_corecnt(void *val) static void cpu_read_constcnt(void *val) { /* - * Return 0 if the current CPU is affected by erratum 2457168. A value - * of 0 is also returned if the current CPU does not support AMUs or if - * the counter is disabled. A return value of 0 at counter read is - * properly handled as an error case by the users of the counter. + * Return 0 if the current CPU is affected by a HW erratum. + * A value of 0 is also returned if the current CPU does not + * support AMUs or if the counter is disabled. A return + * value of 0 at counter read is properly handled as an error + * case by the users of the counter. */ - *(u64 *)val = this_cpu_has_cap(ARM64_WORKAROUND_2457168) ? + *(u64 *)val = this_cpu_has_cap(ARM64_WORKAROUND_BROKEN_AMU_CONSTCNT) ? 0UL : read_constcnt(); } diff --git a/arch/arm64/kvm/mmu.c b/arch/arm64/kvm/mmu.c index 482280fe22d7..47f76a78ba1f 100644 --- a/arch/arm64/kvm/mmu.c +++ b/arch/arm64/kvm/mmu.c @@ -1329,32 +1329,11 @@ transparent_hugepage_adjust(struct kvm *kvm, struct kvm_memory_slot *memslot, return PAGE_SIZE; } -static int get_vma_page_shift(struct vm_area_struct *vma, unsigned long hva) +static int get_vma_page_shift(struct vm_area_struct *vma) { - unsigned long pa; - if (is_vm_hugetlb_page(vma) && !(vma->vm_flags & VM_PFNMAP)) return huge_page_shift(hstate_vma(vma)); - if (!(vma->vm_flags & VM_PFNMAP)) - return PAGE_SHIFT; - - VM_BUG_ON(is_vm_hugetlb_page(vma)); - - pa = (vma->vm_pgoff << PAGE_SHIFT) + (hva - vma->vm_start); - -#ifndef __PAGETABLE_PMD_FOLDED - if ((hva & (PUD_SIZE - 1)) == (pa & (PUD_SIZE - 1)) && - ALIGN_DOWN(hva, PUD_SIZE) >= vma->vm_start && - ALIGN(hva, PUD_SIZE) <= vma->vm_end) - return PUD_SHIFT; -#endif - - if ((hva & (PMD_SIZE - 1)) == (pa & (PMD_SIZE - 1)) && - ALIGN_DOWN(hva, PMD_SIZE) >= vma->vm_start && - ALIGN(hva, PMD_SIZE) <= vma->vm_end) - return PMD_SHIFT; - return PAGE_SHIFT; } @@ -1456,7 +1435,7 @@ static int user_mem_abort(struct kvm_vcpu *vcpu, phys_addr_t fault_ipa, force_pte = true; vma_shift = PAGE_SHIFT; } else { - vma_shift = get_vma_page_shift(vma, hva); + vma_shift = get_vma_page_shift(vma); } switch (vma_shift) { diff --git a/arch/arm64/kvm/vgic/vgic-its.c b/arch/arm64/kvm/vgic/vgic-its.c index e6c5e18664cd..efc556b19682 100644 --- a/arch/arm64/kvm/vgic/vgic-its.c +++ b/arch/arm64/kvm/vgic/vgic-its.c @@ -2256,6 +2256,14 @@ static int vgic_its_save_ite(struct vgic_its *its, struct its_device *dev, u32 next_offset; u64 val; + /* + * MAPC with V=0 keeps the ITEs mapped but drops their collection, + * and with it the ICID. Save a zeroed entry, which the restore path + * reads back as invalid. + */ + if (!ite->collection) + return vgic_its_write_entry_lock(its, gpa, 0ULL, ite); + next_offset = compute_next_eventid_offset(&dev->itt_head, ite); val = ((u64)next_offset << KVM_ITS_ITE_NEXT_SHIFT) | ((u64)ite->irq->intid << KVM_ITS_ITE_PINTID_SHIFT) | diff --git a/arch/arm64/kvm/vgic/vgic.c b/arch/arm64/kvm/vgic/vgic.c index 6f78baa72fb8..9a6e2eab8427 100644 --- a/arch/arm64/kvm/vgic/vgic.c +++ b/arch/arm64/kvm/vgic/vgic.c @@ -160,6 +160,7 @@ void vgic_flush_pending_lpis(struct kvm_vcpu *vcpu) list_for_each_entry_safe(irq, tmp, &vgic_cpu->ap_list_head, ap_list) { if (irq->intid >= VGIC_MIN_LPI) { raw_spin_lock(&irq->irq_lock); + irq->pending_latch = false; list_del(&irq->ap_list); irq->vcpu = NULL; raw_spin_unlock(&irq->irq_lock); @@ -678,7 +679,11 @@ static void vgic_prune_ap_list(struct kvm_vcpu *vcpu) continue; } - /* This interrupt looks like it has to be migrated. */ + /* + * This interrupt looks like it has to be migrated, + * make sure it is kept alive while locks are dropped. + */ + vgic_get_irq_kref(irq); raw_spin_unlock(&irq->irq_lock); raw_spin_unlock(&vgic_cpu->ap_list_lock); @@ -723,6 +728,8 @@ static void vgic_prune_ap_list(struct kvm_vcpu *vcpu) raw_spin_unlock(&vcpuB->arch.vgic_cpu.ap_list_lock); raw_spin_unlock(&vcpuA->arch.vgic_cpu.ap_list_lock); + vgic_put_irq(vcpu->kvm, irq); + if (target_vcpu_needs_kick) { kvm_make_request(KVM_REQ_IRQ_PENDING, target_vcpu); kvm_vcpu_kick(target_vcpu); diff --git a/arch/arm64/net/bpf_jit_comp.c b/arch/arm64/net/bpf_jit_comp.c index 9fd0b658e460..f562060a50c4 100644 --- a/arch/arm64/net/bpf_jit_comp.c +++ b/arch/arm64/net/bpf_jit_comp.c @@ -1648,7 +1648,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) extable_offset = round_up(prog_size + PLT_TARGET_SIZE, extable_align); image_size = extable_offset + extable_size; header = bpf_jit_binary_alloc(image_size, &image_ptr, - sizeof(u32), jit_fill_hole); + sizeof(u64), jit_fill_hole); if (header == NULL) { prog = orig_prog; goto out_off; @@ -1697,7 +1697,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog->jited_len = 0; goto out_off; } - bpf_jit_binary_lock_ro(header); + if (bpf_jit_binary_lock_ro(header)) { + bpf_jit_binary_free(header); + prog->bpf_func = NULL; + prog->jited = 0; + prog->jited_len = 0; + goto out_off; + } } else { jit_data->ctx = ctx; jit_data->image = image_ptr; @@ -2358,3 +2364,8 @@ int bpf_arch_text_poke(void *ip, enum bpf_text_poke_type poke_type, return ret; } + +bool bpf_jit_supports_ptr_xchg(void) +{ + return true; +} diff --git a/arch/arm64/tools/cpucaps b/arch/arm64/tools/cpucaps index c251ef3caae5..1d3e0e178514 100644 --- a/arch/arm64/tools/cpucaps +++ b/arch/arm64/tools/cpucaps @@ -81,7 +81,6 @@ WORKAROUND_1902691 WORKAROUND_2038923 WORKAROUND_2064142 WORKAROUND_2077057 -WORKAROUND_2457168 WORKAROUND_2645198 WORKAROUND_2658417 WORKAROUND_AMPERE_AC03_CPU_38 @@ -101,3 +100,4 @@ WORKAROUND_REPEAT_TLBI WORKAROUND_SPECULATIVE_AT WORKAROUND_SPECULATIVE_SSBS WORKAROUND_SPECULATIVE_UNPRIV_LOAD +WORKAROUND_BROKEN_AMU_CONSTCNT diff --git a/arch/loongarch/kernel/Makefile b/arch/loongarch/kernel/Makefile index 4fcc168f0732..3c808c680370 100644 --- a/arch/loongarch/kernel/Makefile +++ b/arch/loongarch/kernel/Makefile @@ -57,7 +57,7 @@ obj-$(CONFIG_MAGIC_SYSRQ) += sysrq.o obj-$(CONFIG_RELOCATABLE) += relocate.o -obj-$(CONFIG_KEXEC) += machine_kexec.o relocate_kernel.o +obj-$(CONFIG_KEXEC_CORE) += machine_kexec.o relocate_kernel.o obj-$(CONFIG_CRASH_DUMP) += crash_dump.o obj-$(CONFIG_UNWINDER_GUESS) += unwind_guess.o diff --git a/arch/loongarch/kernel/machine_kexec.c b/arch/loongarch/kernel/machine_kexec.c index ce4e132ac137..7c93e9d1f020 100644 --- a/arch/loongarch/kernel/machine_kexec.c +++ b/arch/loongarch/kernel/machine_kexec.c @@ -62,6 +62,7 @@ static void kexec_image_info(const struct kimage *kimage) int machine_kexec_prepare(struct kimage *kimage) { int i; + char head[8]; char *bootloader = "kexec"; void *cmdline_ptr = (void *)KEXEC_CMDLINE_ADDR; @@ -72,7 +73,9 @@ int machine_kexec_prepare(struct kimage *kimage) /* Find the command line */ for (i = 0; i < kimage->nr_segments; i++) { - if (!strncmp(bootloader, (char __user *)kimage->segment[i].buf, strlen(bootloader))) { + if (copy_from_user(head, kimage->segment[i].buf, strlen(bootloader))) + continue; + if (!strncmp(bootloader, head, strlen(bootloader))) { if (!copy_from_user(cmdline_ptr, kimage->segment[i].buf, COMMAND_LINE_SIZE)) kimage->arch.cmdline_ptr = (unsigned long)cmdline_ptr; break; diff --git a/arch/loongarch/kernel/setup.c b/arch/loongarch/kernel/setup.c index 729b0660b61c..55dda1eed069 100644 --- a/arch/loongarch/kernel/setup.c +++ b/arch/loongarch/kernel/setup.c @@ -627,6 +627,7 @@ void __init setup_arch(char **cmdline_p) memblock_init(); pagetable_init(); bootcmdline_init(cmdline_p); + jump_label_init(); /* Initialise the static keys for early params */ parse_early_param(); reserve_initrd_mem(); diff --git a/arch/loongarch/net/bpf_jit.c b/arch/loongarch/net/bpf_jit.c index 54661cffd654..e3220af7d26a 100644 --- a/arch/loongarch/net/bpf_jit.c +++ b/arch/loongarch/net/bpf_jit.c @@ -205,17 +205,13 @@ bool bpf_jit_supports_kfunc_call(void) static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn) { - int off, tc_ninsn = 0; + int off, jmp_offset; u8 tcc = tail_call_reg(ctx); u8 a1 = LOONGARCH_GPR_A1; u8 a2 = LOONGARCH_GPR_A2; u8 t1 = LOONGARCH_GPR_T1; u8 t2 = LOONGARCH_GPR_T2; u8 t3 = LOONGARCH_GPR_T3; - const int idx0 = ctx->idx; - -#define cur_offset (ctx->idx - idx0) -#define jmp_offset (tc_ninsn - (cur_offset)) /* * a0: &ctx @@ -225,12 +221,12 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn) * if (index >= array->map.max_entries) * goto out; */ - tc_ninsn = insn ? ctx->offset[insn+1] - ctx->offset[insn] : ctx->offset[0]; emit_zext_32(ctx, a2, true); off = offsetof(struct bpf_array, map.max_entries); emit_insn(ctx, ldwu, t1, a1, off); /* bgeu $a2, $t1, jmp_offset */ + jmp_offset = ctx->image ? (ctx->offset[insn + 1] - ctx->idx) : 0; if (emit_tailcall_jmp(ctx, BPF_JGE, a2, t1, jmp_offset) < 0) goto toofar; @@ -239,6 +235,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn) * goto out; */ emit_insn(ctx, addid, REG_TCC, tcc, -1); + jmp_offset = ctx->image ? (ctx->offset[insn + 1] - ctx->idx) : 0; if (emit_tailcall_jmp(ctx, BPF_JSLT, REG_TCC, LOONGARCH_GPR_ZERO, jmp_offset) < 0) goto toofar; @@ -251,6 +248,7 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn) off = offsetof(struct bpf_array, ptrs); emit_insn(ctx, ldd, t2, t2, off); /* beq $t2, $zero, jmp_offset */ + jmp_offset = ctx->image ? (ctx->offset[insn + 1] - ctx->idx) : 0; if (emit_tailcall_jmp(ctx, BPF_JEQ, t2, LOONGARCH_GPR_ZERO, jmp_offset) < 0) goto toofar; @@ -264,8 +262,6 @@ static int emit_bpf_tail_call(struct jit_ctx *ctx, int insn) toofar: pr_info_once("tail_call: jump too far\n"); return -1; -#undef cur_offset -#undef jmp_offset } static void emit_atomic(const struct bpf_insn *insn, struct jit_ctx *ctx) @@ -1222,16 +1218,19 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) flush_icache_range((unsigned long)header, (unsigned long)(ctx.image + ctx.idx)); if (!prog->is_func || extra_pass) { + int err; + if (extra_pass && ctx.idx != jit_data->ctx.idx) { pr_err_once("multi-func JIT bug %d != %d\n", ctx.idx, jit_data->ctx.idx); - bpf_jit_binary_free(header); - prog->bpf_func = NULL; - prog->jited = 0; - prog->jited_len = 0; - goto out_offset; + goto out_free; + } + err = bpf_jit_binary_lock_ro(header); + if (err) { + pr_err_once("bpf_jit_binary_lock_ro() returned %d\n", + err); + goto out_free; } - bpf_jit_binary_lock_ro(header); } else { jit_data->ctx = ctx; jit_data->image = image_ptr; @@ -1261,6 +1260,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) return prog; + +out_free: + bpf_jit_binary_free(header); + prog->bpf_func = NULL; + prog->jited = 0; + prog->jited_len = 0; + goto out_offset; } /* Indicate the JIT backend supports mixing bpf2bpf and tailcalls. */ diff --git a/arch/mips/Kbuild b/arch/mips/Kbuild index af2967bffb73..e2d623621a00 100644 --- a/arch/mips/Kbuild +++ b/arch/mips/Kbuild @@ -1,10 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 -# Fail on warnings - also for files referenced in subdirs -# -Werror can be disabled for specific files using: -# CFLAGS_ := -Wno-error -ifeq ($(W),) -subdir-ccflags-y := -Werror -endif # platform specific definitions include $(srctree)/arch/mips/Kbuild.platforms diff --git a/arch/mips/include/asm/uasm.h b/arch/mips/include/asm/uasm.h index 296bcf31abb5..b43bfd445252 100644 --- a/arch/mips/include/asm/uasm.h +++ b/arch/mips/include/asm/uasm.h @@ -193,9 +193,7 @@ struct uasm_label { void uasm_build_label(struct uasm_label **lab, u32 *addr, int lid); -#ifdef CONFIG_64BIT int uasm_in_compat_space_p(long addr); -#endif int uasm_rel_hi(long val); int uasm_rel_lo(long val); void UASM_i_LA_mostly(u32 **buf, unsigned int rs, long addr); diff --git a/arch/mips/kernel/ftrace.c b/arch/mips/kernel/ftrace.c index f39e85fd58fa..b15615b28569 100644 --- a/arch/mips/kernel/ftrace.c +++ b/arch/mips/kernel/ftrace.c @@ -54,10 +54,20 @@ static inline void ftrace_dyn_arch_init_insns(void) u32 *buf; unsigned int v1; - /* la v1, _mcount */ - v1 = 3; - buf = (u32 *)&insn_la_mcount[0]; - UASM_i_LA(&buf, v1, MCOUNT_ADDR); + /* If we are not in compat space, the number of generated + * instructions will exceed the maximum expected limit of 2. + * To prevent buffer overflow, we avoid generating them. + * insn_la_mcount will not be used later in ftrace_make_call. + */ + if (uasm_in_compat_space_p(MCOUNT_ADDR)) { + /* la v1, _mcount */ + v1 = 3; + buf = (u32 *)&insn_la_mcount[0]; + UASM_i_LA(&buf, v1, MCOUNT_ADDR); + } else { + pr_warn("ftrace: mcount address beyond 32 bits is not supported (%lX)\n", + MCOUNT_ADDR); + } /* jal (ftrace_caller + 8), jump over the first two instruction */ buf = (u32 *)&insn_jal_ftrace_caller; @@ -189,6 +199,13 @@ int ftrace_make_call(struct dyn_ftrace *rec, unsigned long addr) unsigned int new; unsigned long ip = rec->ip; + /* When the code to patch does not belong to the kernel code + * space, we must use insn_la_mcount. However, if MCOUNT_ADDR + * is not in compat space, insn_la_mcount is not usable. + */ + if (!core_kernel_text(ip) && !uasm_in_compat_space_p(MCOUNT_ADDR)) + return -EFAULT; + new = core_kernel_text(ip) ? insn_jal_ftrace_caller : insn_la_mcount[0]; #ifdef CONFIG_64BIT diff --git a/arch/mips/net/bpf_jit_comp.c b/arch/mips/net/bpf_jit_comp.c index a40d926b6513..e355dfca4400 100644 --- a/arch/mips/net/bpf_jit_comp.c +++ b/arch/mips/net/bpf_jit_comp.c @@ -1012,7 +1012,8 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_prog_fill_jited_linfo(prog, &ctx.descriptors[1]); /* Set as read-only exec and flush instruction cache */ - bpf_jit_binary_lock_ro(header); + if (bpf_jit_binary_lock_ro(header)) + goto out_err; flush_icache_range((unsigned long)header, (unsigned long)&ctx.target[ctx.jit_index]); diff --git a/arch/mips/sgi-ip22/ip22-platform.c b/arch/mips/sgi-ip22/ip22-platform.c index 0b2002e02a47..3a53690b4b33 100644 --- a/arch/mips/sgi-ip22/ip22-platform.c +++ b/arch/mips/sgi-ip22/ip22-platform.c @@ -221,3 +221,35 @@ static int __init sgi_ds1286_devinit(void) } device_initcall(sgi_ds1286_devinit); + +#define SGI_ZILOG_BASE (HPC3_CHIP0_BASE + \ + offsetof(struct hpc3_regs, pbus_extregs[6]) + \ + offsetof(struct sgioc_regs, uart)) + +static struct resource sgi_zilog_resources[] = { + { + .start = SGI_ZILOG_BASE, + .end = SGI_ZILOG_BASE + 15, + .flags = IORESOURCE_MEM + }, + { + .start = SGI_SERIAL_IRQ, + .end = SGI_SERIAL_IRQ, + .flags = IORESOURCE_IRQ + } +}; + +static struct platform_device zilog_device = { + .name = "ip22zilog", + .id = 0, + .num_resources = ARRAY_SIZE(sgi_zilog_resources), + .resource = sgi_zilog_resources, +}; + + +static int __init sgi_zilog_devinit(void) +{ + return platform_device_register(&zilog_device); +} + +device_initcall(sgi_zilog_devinit); diff --git a/arch/parisc/net/bpf_jit_core.c b/arch/parisc/net/bpf_jit_core.c index 7b9cb3cda27e..06cbcd6fe87b 100644 --- a/arch/parisc/net/bpf_jit_core.c +++ b/arch/parisc/net/bpf_jit_core.c @@ -167,7 +167,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_flush_icache(jit_data->header, ctx->insns + ctx->ninsns); if (!prog->is_func || extra_pass) { - bpf_jit_binary_lock_ro(jit_data->header); + if (bpf_jit_binary_lock_ro(jit_data->header)) { + bpf_jit_binary_free(jit_data->header); + prog->bpf_func = NULL; + prog->jited = 0; + prog->jited_len = 0; + goto out_offset; + } prologue_len = ctx->epilogue_offset - ctx->body_len; for (i = 0; i < prog->len; i++) ctx->offset[i] += prologue_len; diff --git a/arch/powerpc/include/asm/book3s/64/pgtable.h b/arch/powerpc/include/asm/book3s/64/pgtable.h index 8a6e6b6daa90..224205eeeff1 100644 --- a/arch/powerpc/include/asm/book3s/64/pgtable.h +++ b/arch/powerpc/include/asm/book3s/64/pgtable.h @@ -1328,6 +1328,22 @@ extern int pudp_test_and_clear_young(struct vm_area_struct *vma, static inline pmd_t pmdp_huge_get_and_clear(struct mm_struct *mm, unsigned long addr, pmd_t *pmdp) { + pmd_t old_pmd; + + /* + * Non-present PMDs can be migration entries or device-private THP + * entries. This can happen at 2 places: + * - When the address space is being unmapped zap_huge_pmd(), and we + * encounter non-present pmds. + * - migrate_vma_collect_huge_pmd() could calls this during migration + * of device-private pmd entries. + */ + if (!pmd_present(*pmdp)) { + old_pmd = READ_ONCE(*pmdp); + pmd_clear(pmdp); + return old_pmd; + } + if (radix_enabled()) return radix__pmdp_huge_get_and_clear(mm, addr, pmdp); return hash__pmdp_huge_get_and_clear(mm, addr, pmdp); diff --git a/arch/powerpc/include/asm/iommu.h b/arch/powerpc/include/asm/iommu.h index 026695943550..744cc5fc22d3 100644 --- a/arch/powerpc/include/asm/iommu.h +++ b/arch/powerpc/include/asm/iommu.h @@ -156,6 +156,9 @@ extern int iommu_tce_table_put(struct iommu_table *tbl); extern struct iommu_table *iommu_init_table(struct iommu_table *tbl, int nid, unsigned long res_start, unsigned long res_end); bool iommu_table_in_use(struct iommu_table *tbl); +extern void iommu_table_reserve_pages(struct iommu_table *tbl, + unsigned long res_start, unsigned long res_end); +extern void iommu_table_clear(struct iommu_table *tbl); #define IOMMU_TABLE_GROUP_MAX_TABLES 2 @@ -218,7 +221,6 @@ extern long iommu_tce_xchg_no_kill(struct mm_struct *mm, extern void iommu_tce_kill(struct iommu_table *tbl, unsigned long entry, unsigned long pages); -extern struct iommu_table_group_ops spapr_tce_table_group_ops; #else static inline void iommu_register_group(struct iommu_table_group *table_group, int pci_domain_number, diff --git a/arch/powerpc/include/asm/xive.h b/arch/powerpc/include/asm/xive.h index 92930b0b5d0e..8ac7c1d3564f 100644 --- a/arch/powerpc/include/asm/xive.h +++ b/arch/powerpc/include/asm/xive.h @@ -91,7 +91,7 @@ static inline bool xive_enabled(void) { return __xive_enabled; } bool xive_spapr_init(void); bool xive_native_init(void); -void xive_smp_probe(void); +int xive_smp_probe(void); int xive_smp_prepare_cpu(unsigned int cpu); void xive_smp_setup_cpu(void); void xive_smp_disable_cpu(void); @@ -154,7 +154,7 @@ static inline bool xive_enabled(void) { return false; } static inline bool xive_spapr_init(void) { return false; } static inline bool xive_native_init(void) { return false; } -static inline void xive_smp_probe(void) { } +static inline int xive_smp_probe(void) { return -EINVAL; } static inline int xive_smp_prepare_cpu(unsigned int cpu) { return -EINVAL; } static inline void xive_smp_setup_cpu(void) { } static inline void xive_smp_disable_cpu(void) { } diff --git a/arch/powerpc/kernel/iommu.c b/arch/powerpc/kernel/iommu.c index 49f01f6b9346..9227ba3814f5 100644 --- a/arch/powerpc/kernel/iommu.c +++ b/arch/powerpc/kernel/iommu.c @@ -642,7 +642,7 @@ void ppc_iommu_unmap_sg(struct iommu_table *tbl, struct scatterlist *sglist, tbl->it_ops->flush(tbl); } -static void iommu_table_clear(struct iommu_table *tbl) +void iommu_table_clear(struct iommu_table *tbl) { /* * In case of firmware assisted dump system goes through clean @@ -683,10 +683,10 @@ static void iommu_table_clear(struct iommu_table *tbl) #endif } -static void iommu_table_reserve_pages(struct iommu_table *tbl, +void iommu_table_reserve_pages(struct iommu_table *tbl, unsigned long res_start, unsigned long res_end) { - int i; + unsigned long i; WARN_ON_ONCE(res_end < res_start); /* @@ -1101,59 +1101,6 @@ void iommu_tce_kill(struct iommu_table *tbl, } EXPORT_SYMBOL_GPL(iommu_tce_kill); -#if defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV) -static int iommu_take_ownership(struct iommu_table *tbl) -{ - unsigned long flags, i, sz = (tbl->it_size + 7) >> 3; - int ret = 0; - - /* - * VFIO does not control TCE entries allocation and the guest - * can write new TCEs on top of existing ones so iommu_tce_build() - * must be able to release old pages. This functionality - * requires exchange() callback defined so if it is not - * implemented, we disallow taking ownership over the table. - */ - if (!tbl->it_ops->xchg_no_kill) - return -EINVAL; - - spin_lock_irqsave(&tbl->large_pool.lock, flags); - for (i = 0; i < tbl->nr_pools; i++) - spin_lock_nest_lock(&tbl->pools[i].lock, &tbl->large_pool.lock); - - if (iommu_table_in_use(tbl)) { - pr_err("iommu_tce: it_map is not empty"); - ret = -EBUSY; - } else { - memset(tbl->it_map, 0xff, sz); - } - - for (i = 0; i < tbl->nr_pools; i++) - spin_unlock(&tbl->pools[i].lock); - spin_unlock_irqrestore(&tbl->large_pool.lock, flags); - - return ret; -} - -static void iommu_release_ownership(struct iommu_table *tbl) -{ - unsigned long flags, i, sz = (tbl->it_size + 7) >> 3; - - spin_lock_irqsave(&tbl->large_pool.lock, flags); - for (i = 0; i < tbl->nr_pools; i++) - spin_lock_nest_lock(&tbl->pools[i].lock, &tbl->large_pool.lock); - - memset(tbl->it_map, 0, sz); - - iommu_table_reserve_pages(tbl, tbl->it_reserved_start, - tbl->it_reserved_end); - - for (i = 0; i < tbl->nr_pools; i++) - spin_unlock(&tbl->pools[i].lock); - spin_unlock_irqrestore(&tbl->large_pool.lock, flags); -} -#endif - int iommu_add_device(struct iommu_table_group *table_group, struct device *dev) { /* @@ -1185,98 +1132,6 @@ int iommu_add_device(struct iommu_table_group *table_group, struct device *dev) EXPORT_SYMBOL_GPL(iommu_add_device); #if defined(CONFIG_PPC_PSERIES) || defined(CONFIG_PPC_POWERNV) -/* - * A simple iommu_table_group_ops which only allows reusing the existing - * iommu_table. This handles VFIO for POWER7 or the nested KVM. - * The ops does not allow creating windows and only allows reusing the existing - * one if it matches table_group->tce32_start/tce32_size/page_shift. - */ -static unsigned long spapr_tce_get_table_size(__u32 page_shift, - __u64 window_size, __u32 levels) -{ - unsigned long size; - - if (levels > 1) - return ~0U; - size = window_size >> (page_shift - 3); - return size; -} - -static long spapr_tce_create_table(struct iommu_table_group *table_group, int num, - __u32 page_shift, __u64 window_size, __u32 levels, - struct iommu_table **ptbl) -{ - struct iommu_table *tbl = table_group->tables[0]; - - if (num > 0) - return -EPERM; - - if (tbl->it_page_shift != page_shift || - tbl->it_size != (window_size >> page_shift) || - tbl->it_indirect_levels != levels - 1) - return -EINVAL; - - *ptbl = iommu_tce_table_get(tbl); - return 0; -} - -static long spapr_tce_set_window(struct iommu_table_group *table_group, - int num, struct iommu_table *tbl) -{ - return tbl == table_group->tables[num] ? 0 : -EPERM; -} - -static long spapr_tce_unset_window(struct iommu_table_group *table_group, int num) -{ - return 0; -} - -static long spapr_tce_take_ownership(struct iommu_table_group *table_group) -{ - int i, j, rc = 0; - - for (i = 0; i < IOMMU_TABLE_GROUP_MAX_TABLES; ++i) { - struct iommu_table *tbl = table_group->tables[i]; - - if (!tbl || !tbl->it_map) - continue; - - rc = iommu_take_ownership(tbl); - if (!rc) - continue; - - for (j = 0; j < i; ++j) - iommu_release_ownership(table_group->tables[j]); - return rc; - } - return 0; -} - -static void spapr_tce_release_ownership(struct iommu_table_group *table_group) -{ - int i; - - for (i = 0; i < IOMMU_TABLE_GROUP_MAX_TABLES; ++i) { - struct iommu_table *tbl = table_group->tables[i]; - - if (!tbl) - continue; - - iommu_table_clear(tbl); - if (tbl->it_map) - iommu_release_ownership(tbl); - } -} - -struct iommu_table_group_ops spapr_tce_table_group_ops = { - .get_table_size = spapr_tce_get_table_size, - .create_table = spapr_tce_create_table, - .set_window = spapr_tce_set_window, - .unset_window = spapr_tce_unset_window, - .take_ownership = spapr_tce_take_ownership, - .release_ownership = spapr_tce_release_ownership, -}; - /* * A simple iommu_ops to allow less cruft in generic VFIO code. */ diff --git a/arch/powerpc/kexec/ranges.c b/arch/powerpc/kexec/ranges.c index fb3e12f15214..ef025eea27e8 100644 --- a/arch/powerpc/kexec/ranges.c +++ b/arch/powerpc/kexec/ranges.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include @@ -101,19 +102,16 @@ static void __merge_memory_ranges(struct crash_mem *mem_rngs) struct range *ranges; int i, idx; - if (!mem_rngs) + if (!mem_rngs || mem_rngs->nr_ranges <= 1) return; idx = 0; - ranges = &(mem_rngs->ranges[0]); + ranges = mem_rngs->ranges; for (i = 1; i < mem_rngs->nr_ranges; i++) { - if (ranges[i].start <= (ranges[i-1].end + 1)) - ranges[idx].end = ranges[i].end; + if (ranges[i].start <= (ranges[idx].end + 1)) + ranges[idx].end = max(ranges[idx].end, ranges[i].end); else { idx++; - if (i == idx) - continue; - ranges[idx] = ranges[i]; } } diff --git a/arch/powerpc/mm/book3s64/pgtable.c b/arch/powerpc/mm/book3s64/pgtable.c index 5b4cbb25d9cf..5973b9d3b591 100644 --- a/arch/powerpc/mm/book3s64/pgtable.c +++ b/arch/powerpc/mm/book3s64/pgtable.c @@ -113,7 +113,7 @@ void set_pmd_at(struct mm_struct *mm, unsigned long addr, WARN_ON(pte_hw_valid(pmd_pte(*pmdp)) && !pte_protnone(pmd_pte(*pmdp))); assert_spin_locked(pmd_lockptr(mm, pmdp)); - WARN_ON(!(pmd_large(pmd))); + WARN_ON(!(pmd_leaf(pmd))); #endif trace_hugepage_set_pmd(addr, pmd_val(pmd)); return set_pte_at(mm, addr, pmdp_ptep(pmdp), pmd_pte(pmd)); @@ -180,16 +180,21 @@ pmd_t pmdp_huge_get_and_clear_full(struct vm_area_struct *vma, unsigned long addr, pmd_t *pmdp, int full) { pmd_t pmd; + bool was_present = pmd_present(*pmdp); + VM_BUG_ON(addr & ~HPAGE_PMD_MASK); - VM_BUG_ON((pmd_present(*pmdp) && !pmd_trans_huge(*pmdp) && - !pmd_devmap(*pmdp)) || !pmd_present(*pmdp)); + VM_BUG_ON(was_present && !pmd_trans_huge(*pmdp) && !pmd_devmap(*pmdp)); + /* + * Check pmdp_huge_get_and_clear() for non-present pmd case. + */ pmd = pmdp_huge_get_and_clear(vma->vm_mm, addr, pmdp); /* * if it not a fullmm flush, then we can possibly end up converting * this PMD pte entry to a regular level 0 PTE by a parallel page fault. - * Make sure we flush the tlb in this case. + * Make sure we flush the tlb in this case. TLB flush not needed for + * non-present case. */ - if (!full) + if (was_present && !full) flush_pmd_tlb_range(vma, addr, addr + HPAGE_PMD_SIZE); return pmd; } diff --git a/arch/powerpc/mm/book3s64/radix_pgtable.c b/arch/powerpc/mm/book3s64/radix_pgtable.c index ff643d8c29b3..68da05ae45aa 100644 --- a/arch/powerpc/mm/book3s64/radix_pgtable.c +++ b/arch/powerpc/mm/book3s64/radix_pgtable.c @@ -1001,7 +1001,7 @@ bool vmemmap_can_optimize(struct vmem_altmap *altmap, struct dev_pagemap *pgmap) int __meminit vmemmap_check_pmd(pmd_t *pmdp, int node, unsigned long addr, unsigned long next) { - int large = pmd_large(*pmdp); + int large = pmd_leaf(*pmdp); if (large) vmemmap_verify(pmdp_ptep(pmdp), node, addr, next); diff --git a/arch/powerpc/mm/pgtable-frag.c b/arch/powerpc/mm/pgtable-frag.c index 8c31802f97e8..01600cac89dc 100644 --- a/arch/powerpc/mm/pgtable-frag.c +++ b/arch/powerpc/mm/pgtable-frag.c @@ -25,6 +25,7 @@ void pte_frag_destroy(void *pte_frag) count = ((unsigned long)pte_frag & ~PAGE_MASK) >> PTE_FRAG_SIZE_SHIFT; /* We allow PTE_FRAG_NR fragments from a PTE page */ if (atomic_sub_and_test(PTE_FRAG_NR - count, &ptdesc->pt_frag_refcount)) { + folio_clear_active(ptdesc_folio(ptdesc)); pagetable_pte_dtor(ptdesc); pagetable_free(ptdesc); } diff --git a/arch/powerpc/mm/pgtable_64.c b/arch/powerpc/mm/pgtable_64.c index 5ac1fd30341b..0456119c9df9 100644 --- a/arch/powerpc/mm/pgtable_64.c +++ b/arch/powerpc/mm/pgtable_64.c @@ -132,7 +132,7 @@ struct page *pmd_page(pmd_t pmd) * enabled so these checks can't be used. */ if (!IS_ENABLED(CONFIG_HAVE_ARCH_HUGE_VMAP)) - VM_WARN_ON(!(pmd_large(pmd) || pmd_huge(pmd))); + VM_WARN_ON(!(pmd_leaf(pmd) || pmd_huge(pmd))); return pte_page(pmd_pte(pmd)); } return virt_to_page(pmd_page_vaddr(pmd)); diff --git a/arch/powerpc/net/bpf_jit_comp.c b/arch/powerpc/net/bpf_jit_comp.c index 37043dfc1add..877e7ff38785 100644 --- a/arch/powerpc/net/bpf_jit_comp.c +++ b/arch/powerpc/net/bpf_jit_comp.c @@ -39,7 +39,7 @@ int bpf_jit_emit_exit_insn(u32 *image, struct codegen_context *ctx, int tmp_reg, return 0; } -struct powerpc64_jit_data { +struct powerpc_jit_data { struct bpf_binary_header *header; u32 *addrs; u8 *image; @@ -59,7 +59,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) u8 *image = NULL; u32 *code_base; u32 *addrs; - struct powerpc64_jit_data *jit_data; + struct powerpc_jit_data *jit_data; struct codegen_context cgctx; int pass; int flen; @@ -205,7 +205,13 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) bpf_flush_icache(bpf_hdr, (u8 *)bpf_hdr + bpf_hdr->size); if (!fp->is_func || extra_pass) { - bpf_jit_binary_lock_ro(bpf_hdr); + if (bpf_jit_binary_lock_ro(bpf_hdr)) { + bpf_jit_binary_free(bpf_hdr); + fp->bpf_func = NULL; + fp->jited = 0; + fp->jited_len = 0; + goto out_addrs; + } bpf_prog_fill_jited_linfo(fp, addrs); out_addrs: kfree(addrs); diff --git a/arch/powerpc/perf/core-book3s.c b/arch/powerpc/perf/core-book3s.c index 4bb84dc4393f..fafb0343afd5 100644 --- a/arch/powerpc/perf/core-book3s.c +++ b/arch/powerpc/perf/core-book3s.c @@ -132,7 +132,10 @@ static unsigned long ebb_switch_in(bool ebb, struct cpu_hw_events *cpuhw) static inline void power_pmu_bhrb_enable(struct perf_event *event) {} static inline void power_pmu_bhrb_disable(struct perf_event *event) {} -static void power_pmu_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) {} +static void power_pmu_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in) +{ +} static inline void power_pmu_bhrb_read(struct perf_event *event, struct cpu_hw_events *cpuhw) {} static void pmao_restore_workaround(bool ebb) { } #endif /* CONFIG_PPC32 */ @@ -451,7 +454,8 @@ static void power_pmu_bhrb_disable(struct perf_event *event) /* Called from ctxsw to prevent one process's branch entries to * mingle with the other process's entries during context switch. */ -static void power_pmu_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) +static void power_pmu_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in) { if (!ppmu->bhrb_nr) return; diff --git a/arch/powerpc/platforms/powernv/smp.c b/arch/powerpc/platforms/powernv/smp.c index 9e1a25398f98..4887f9176aa4 100644 --- a/arch/powerpc/platforms/powernv/smp.c +++ b/arch/powerpc/platforms/powernv/smp.c @@ -329,10 +329,12 @@ static void pnv_cause_ipi(int cpu) static void __init pnv_smp_probe(void) { - if (xive_enabled()) - xive_smp_probe(); - else + if (xive_enabled()) { + if (xive_smp_probe() < 0) + return; + } else { xics_smp_probe(); + } if (cpu_has_feature(CPU_FTR_DBELL)) { ic_cause_ipi = smp_ops->cause_ipi; diff --git a/arch/powerpc/platforms/pseries/iommu.c b/arch/powerpc/platforms/pseries/iommu.c index 54fd2a0d6604..6a45e3e86f79 100644 --- a/arch/powerpc/platforms/pseries/iommu.c +++ b/arch/powerpc/platforms/pseries/iommu.c @@ -54,6 +54,57 @@ enum { DDW_EXT_QUERY_OUT_SIZE = 2 }; +static int iommu_take_ownership(struct iommu_table *tbl) +{ + unsigned long flags, i, sz = (tbl->it_size + 7) >> 3; + int ret = 0; + + /* + * VFIO does not control TCE entries allocation and the guest + * can write new TCEs on top of existing ones so iommu_tce_build() + * must be able to release old pages. This functionality + * requires exchange() callback defined so if it is not + * implemented, we disallow taking ownership over the table. + */ + if (!tbl->it_ops->xchg_no_kill) + return -EINVAL; + + spin_lock_irqsave(&tbl->large_pool.lock, flags); + for (i = 0; i < tbl->nr_pools; i++) + spin_lock_nest_lock(&tbl->pools[i].lock, &tbl->large_pool.lock); + + if (iommu_table_in_use(tbl)) { + pr_err("iommu_tce: it_map is not empty"); + ret = -EBUSY; + } else { + memset(tbl->it_map, 0xff, sz); + } + + for (i = 0; i < tbl->nr_pools; i++) + spin_unlock(&tbl->pools[i].lock); + spin_unlock_irqrestore(&tbl->large_pool.lock, flags); + + return ret; +} + +static void iommu_release_ownership(struct iommu_table *tbl) +{ + unsigned long flags, i, sz = (tbl->it_size + 7) >> 3; + + spin_lock_irqsave(&tbl->large_pool.lock, flags); + for (i = 0; i < tbl->nr_pools; i++) + spin_lock_nest_lock(&tbl->pools[i].lock, &tbl->large_pool.lock); + + memset(tbl->it_map, 0, sz); + + iommu_table_reserve_pages(tbl, tbl->it_reserved_start, + tbl->it_reserved_end); + + for (i = 0; i < tbl->nr_pools; i++) + spin_unlock(&tbl->pools[i].lock); + spin_unlock_irqrestore(&tbl->large_pool.lock, flags); +} + static struct iommu_table *iommu_pseries_alloc_table(int node) { struct iommu_table *tbl; @@ -67,6 +118,8 @@ static struct iommu_table *iommu_pseries_alloc_table(int node) return tbl; } +static struct iommu_table_group_ops spapr_tce_table_group_ops; + static struct iommu_table_group *iommu_pseries_alloc_group(int node) { struct iommu_table_group *table_group; @@ -1505,7 +1558,8 @@ static bool enable_ddw(struct pci_dev *dev, struct device_node *pdn) iommu_table_setparms_common(newtbl, pci->phb->bus->number, create.liobn, win_addr, 1UL << len, page_shift, NULL, &iommu_table_lpar_multi_ops); - iommu_init_table(newtbl, pci->phb->node, start, end); + iommu_init_table(newtbl, pci->phb->node, + start >> page_shift, end >> page_shift); pci->table_group->tables[1] = newtbl; @@ -1648,6 +1702,97 @@ static bool iommu_bypass_supported_pSeriesLP(struct pci_dev *pdev, u64 dma_mask) return false; } +/* + * A simple iommu_table_group_ops which only allows reusing the existing + * iommu_table. This handles VFIO for POWER7 or the nested KVM. + * The ops does not allow creating windows and only allows reusing the existing + * one if it matches table_group->tce32_start/tce32_size/page_shift. + */ +static unsigned long spapr_tce_get_table_size(__u32 page_shift, + __u64 window_size, __u32 levels) +{ + unsigned long size; + + if (levels > 1) + return ~0U; + size = window_size >> (page_shift - 3); + return size; +} + +static long spapr_tce_create_table(struct iommu_table_group *table_group, int num, + __u32 page_shift, __u64 window_size, __u32 levels, + struct iommu_table **ptbl) +{ + struct iommu_table *tbl = table_group->tables[0]; + + if (num > 0) + return -EPERM; + + if (tbl->it_page_shift != page_shift || + tbl->it_size != (window_size >> page_shift) || + tbl->it_indirect_levels != levels - 1) + return -EINVAL; + + *ptbl = iommu_tce_table_get(tbl); + return 0; +} + +static long spapr_tce_set_window(struct iommu_table_group *table_group, + int num, struct iommu_table *tbl) +{ + return tbl == table_group->tables[num] ? 0 : -EPERM; +} + +static long spapr_tce_unset_window(struct iommu_table_group *table_group, int num) +{ + return 0; +} + +static long spapr_tce_take_ownership(struct iommu_table_group *table_group) +{ + int i, j, rc = 0; + + for (i = 0; i < IOMMU_TABLE_GROUP_MAX_TABLES; ++i) { + struct iommu_table *tbl = table_group->tables[i]; + + if (!tbl || !tbl->it_map) + continue; + + rc = iommu_take_ownership(tbl); + if (!rc) + continue; + + for (j = 0; j < i; ++j) + iommu_release_ownership(table_group->tables[j]); + return rc; + } + return 0; +} + +static void spapr_tce_release_ownership(struct iommu_table_group *table_group) +{ + int i; + + for (i = 0; i < IOMMU_TABLE_GROUP_MAX_TABLES; ++i) { + struct iommu_table *tbl = table_group->tables[i]; + + if (!tbl) + continue; + + if (tbl->it_map) + iommu_release_ownership(tbl); + } +} + +static struct iommu_table_group_ops spapr_tce_table_group_ops = { + .get_table_size = spapr_tce_get_table_size, + .create_table = spapr_tce_create_table, + .set_window = spapr_tce_set_window, + .unset_window = spapr_tce_unset_window, + .take_ownership = spapr_tce_take_ownership, + .release_ownership = spapr_tce_release_ownership, +}; + static int iommu_mem_notifier(struct notifier_block *nb, unsigned long action, void *data) { diff --git a/arch/powerpc/platforms/pseries/smp.c b/arch/powerpc/platforms/pseries/smp.c index c597711ef20a..0d9b0b901807 100644 --- a/arch/powerpc/platforms/pseries/smp.c +++ b/arch/powerpc/platforms/pseries/smp.c @@ -194,10 +194,12 @@ static int pseries_cause_nmi_ipi(int cpu) static __init void pSeries_smp_probe(void) { - if (xive_enabled()) - xive_smp_probe(); - else + if (xive_enabled()) { + if (xive_smp_probe() < 0) + return; + } else { xics_smp_probe(); + } /* No doorbell facility, must use the interrupt controller for IPIs */ if (!cpu_has_feature(CPU_FTR_DBELL)) diff --git a/arch/powerpc/sysdev/xive/common.c b/arch/powerpc/sysdev/xive/common.c index a289cb97c1d7..a5da09c1895f 100644 --- a/arch/powerpc/sysdev/xive/common.c +++ b/arch/powerpc/sysdev/xive/common.c @@ -1270,15 +1270,19 @@ noinstr static void xive_cleanup_cpu_ipi(unsigned int cpu, struct xive_cpu *xc) xive_ops->put_ipi(cpu, xc); } -void __init xive_smp_probe(void) +int __init xive_smp_probe(void) { + int ret; + smp_ops->cause_ipi = xive_cause_ipi; /* Register the IPI */ - xive_init_ipis(); + ret = xive_init_ipis(); + if (ret < 0) + return ret; /* Allocate and setup IPI for the boot CPU */ - xive_setup_cpu_ipi(smp_processor_id()); + return xive_setup_cpu_ipi(smp_processor_id()); } #endif /* CONFIG_SMP */ diff --git a/arch/riscv/include/asm/kvm_host.h b/arch/riscv/include/asm/kvm_host.h index d2d7f9b4f7c0..ba434eab2f80 100644 --- a/arch/riscv/include/asm/kvm_host.h +++ b/arch/riscv/include/asm/kvm_host.h @@ -164,6 +164,10 @@ struct kvm_vcpu_csr { unsigned long scounteren; }; +struct kvm_vcpu_config { + u64 henvcfg; +}; + struct kvm_vcpu_arch { /* VCPU ran at least once */ bool ran_atleast_once; @@ -245,6 +249,9 @@ struct kvm_vcpu_arch { /* Performance monitoring context */ struct kvm_pmu pmu_context; + + /* 'static' configurations which are set only once */ + struct kvm_vcpu_config cfg; }; static inline void kvm_arch_sync_events(struct kvm *kvm) {} diff --git a/arch/riscv/include/asm/processor.h b/arch/riscv/include/asm/processor.h index 4747277983ad..f80361a1b458 100644 --- a/arch/riscv/include/asm/processor.h +++ b/arch/riscv/include/asm/processor.h @@ -58,6 +58,7 @@ struct thread_struct { struct __riscv_d_ext_state fstate; unsigned long bad_cause; unsigned long vstate_ctrl; + unsigned long status; struct __riscv_v_ext_state vstate; }; diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-offsets.c index e4589457e608..5eed44f90323 100644 --- a/arch/riscv/kernel/asm-offsets.c +++ b/arch/riscv/kernel/asm-offsets.c @@ -33,6 +33,7 @@ void asm_offsets(void) OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]); OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]); OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]); + OFFSET(TASK_THREAD_STATUS, task_struct, thread.status); OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags); OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_count); OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); @@ -336,6 +337,10 @@ void asm_offsets(void) offsetof(struct task_struct, thread.s[11]) - offsetof(struct task_struct, thread.ra) ); + DEFINE(TASK_THREAD_STATUS_RA, + offsetof(struct task_struct, thread.status) + - offsetof(struct task_struct, thread.ra) + ); DEFINE(TASK_THREAD_F0_F0, offsetof(struct task_struct, thread.fstate.f[0]) diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S index 11b2b1a3897d..b9d321fb1f75 100644 --- a/arch/riscv/kernel/entry.S +++ b/arch/riscv/kernel/entry.S @@ -256,7 +256,16 @@ SYM_FUNC_START(__switch_to) REG_S s9, TASK_THREAD_S9_RA(a3) REG_S s10, TASK_THREAD_S10_RA(a3) REG_S s11, TASK_THREAD_S11_RA(a3) + + /* save the user space access flag */ + li s0, SR_SUM + csrr s1, CSR_STATUS + and s1, s1, s0 + REG_S s1, TASK_THREAD_STATUS_RA(a3) + /* Restore context from next->thread */ + REG_L s0, TASK_THREAD_STATUS_RA(a4) + csrs CSR_STATUS, s0 REG_L ra, TASK_THREAD_RA_RA(a4) REG_L sp, TASK_THREAD_SP_RA(a4) REG_L s0, TASK_THREAD_S0_RA(a4) diff --git a/arch/riscv/kvm/aia_imsic.c b/arch/riscv/kvm/aia_imsic.c index c1585444f856..9e8785e67b6d 100644 --- a/arch/riscv/kvm/aia_imsic.c +++ b/arch/riscv/kvm/aia_imsic.c @@ -895,8 +895,15 @@ int kvm_riscv_aia_imsic_rw_attr(struct kvm *kvm, unsigned long type, if (!vcpu) return -ENODEV; - isel = KVM_DEV_RISCV_AIA_IMSIC_GET_ISEL(type); + if (mutex_lock_killable(&vcpu->mutex)) + return -EINTR; + imsic = vcpu->arch.aia_context.imsic_state; + if (!imsic) { + rc = -ENODEV; + goto out_unlock; + } + isel = KVM_DEV_RISCV_AIA_IMSIC_GET_ISEL(type); read_lock_irqsave(&imsic->vsfile_lock, flags); @@ -919,6 +926,8 @@ int kvm_riscv_aia_imsic_rw_attr(struct kvm *kvm, unsigned long type, rc = imsic_vsfile_rw(vsfile_hgei, vsfile_cpu, imsic->nr_eix, isel, write, val); +out_unlock: + mutex_unlock(&vcpu->mutex); return rc; } @@ -936,8 +945,11 @@ int kvm_riscv_aia_imsic_has_attr(struct kvm *kvm, unsigned long type) if (!vcpu) return -ENODEV; - isel = KVM_DEV_RISCV_AIA_IMSIC_GET_ISEL(type); imsic = vcpu->arch.aia_context.imsic_state; + if (!imsic) + return -ENODEV; + + isel = KVM_DEV_RISCV_AIA_IMSIC_GET_ISEL(type); return imsic_mrif_isel_check(imsic->nr_eix, isel); } diff --git a/arch/riscv/kvm/main.c b/arch/riscv/kvm/main.c index 48ae0d4b3932..4c03be303875 100644 --- a/arch/riscv/kvm/main.c +++ b/arch/riscv/kvm/main.c @@ -131,8 +131,8 @@ module_init(riscv_kvm_init); static void __exit riscv_kvm_exit(void) { - kvm_riscv_aia_exit(); - kvm_exit(); + + kvm_riscv_aia_exit(); } module_exit(riscv_kvm_exit); diff --git a/arch/riscv/kvm/vcpu.c b/arch/riscv/kvm/vcpu.c index e2c66bf31911..5ea1a704f145 100644 --- a/arch/riscv/kvm/vcpu.c +++ b/arch/riscv/kvm/vcpu.c @@ -521,31 +521,28 @@ int kvm_arch_vcpu_ioctl_set_guest_debug(struct kvm_vcpu *vcpu, return -EINVAL; } -static void kvm_riscv_vcpu_update_config(const unsigned long *isa) +static void kvm_riscv_vcpu_setup_config(struct kvm_vcpu *vcpu) { - u64 henvcfg = 0; + const unsigned long *isa = vcpu->arch.isa; + struct kvm_vcpu_config *cfg = &vcpu->arch.cfg; if (riscv_isa_extension_available(isa, SVPBMT)) - henvcfg |= ENVCFG_PBMTE; + cfg->henvcfg |= ENVCFG_PBMTE; if (riscv_isa_extension_available(isa, SSTC)) - henvcfg |= ENVCFG_STCE; + cfg->henvcfg |= ENVCFG_STCE; if (riscv_isa_extension_available(isa, ZICBOM)) - henvcfg |= (ENVCFG_CBIE | ENVCFG_CBCFE); + cfg->henvcfg |= (ENVCFG_CBIE | ENVCFG_CBCFE); if (riscv_isa_extension_available(isa, ZICBOZ)) - henvcfg |= ENVCFG_CBZE; - - csr_write(CSR_HENVCFG, henvcfg); -#ifdef CONFIG_32BIT - csr_write(CSR_HENVCFGH, henvcfg >> 32); -#endif + cfg->henvcfg |= ENVCFG_CBZE; } void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) { struct kvm_vcpu_csr *csr = &vcpu->arch.guest_csr; + struct kvm_vcpu_config *cfg = &vcpu->arch.cfg; csr_write(CSR_VSSTATUS, csr->vsstatus); csr_write(CSR_VSIE, csr->vsie); @@ -556,8 +553,9 @@ void kvm_arch_vcpu_load(struct kvm_vcpu *vcpu, int cpu) csr_write(CSR_VSTVAL, csr->vstval); csr_write(CSR_HVIP, csr->hvip); csr_write(CSR_VSATP, csr->vsatp); - - kvm_riscv_vcpu_update_config(vcpu->arch.isa); + csr_write(CSR_HENVCFG, cfg->henvcfg); + if (IS_ENABLED(CONFIG_32BIT)) + csr_write(CSR_HENVCFGH, cfg->henvcfg >> 32); kvm_riscv_gstage_update_hgatp(vcpu); @@ -677,6 +675,9 @@ int kvm_arch_vcpu_ioctl_run(struct kvm_vcpu *vcpu) struct kvm_cpu_trap trap; struct kvm_run *run = vcpu->run; + if (!vcpu->arch.ran_atleast_once) + kvm_riscv_vcpu_setup_config(vcpu); + /* Mark this VCPU ran at least once */ vcpu->arch.ran_atleast_once = true; diff --git a/arch/riscv/kvm/vcpu_vector.c b/arch/riscv/kvm/vcpu_vector.c index b430cbb69521..3c99400c024b 100644 --- a/arch/riscv/kvm/vcpu_vector.c +++ b/arch/riscv/kvm/vcpu_vector.c @@ -78,8 +78,11 @@ int kvm_riscv_vcpu_alloc_vector_context(struct kvm_vcpu *vcpu, return -ENOMEM; vcpu->arch.host_context.vector.datap = kzalloc(riscv_v_vsize, GFP_KERNEL); - if (!vcpu->arch.host_context.vector.datap) + if (!vcpu->arch.host_context.vector.datap) { + kfree(vcpu->arch.guest_context.vector.datap); + vcpu->arch.guest_context.vector.datap = NULL; return -ENOMEM; + } return 0; } diff --git a/arch/riscv/kvm/vmid.c b/arch/riscv/kvm/vmid.c index ddc98714ce8e..38f30f5a608c 100644 --- a/arch/riscv/kvm/vmid.c +++ b/arch/riscv/kvm/vmid.c @@ -26,7 +26,7 @@ void __init kvm_riscv_gstage_vmid_detect(void) /* Figure-out number of VMID bits in HW */ old = csr_read(CSR_HGATP); - csr_write(CSR_HGATP, old | HGATP_VMID); + csr_write(CSR_HGATP, (kvm_riscv_gstage_mode() << HGATP_MODE_SHIFT) | HGATP_VMID); vmid_bits = csr_read(CSR_HGATP); vmid_bits = (vmid_bits & HGATP_VMID) >> HGATP_VMID_SHIFT; vmid_bits = fls_long(vmid_bits); diff --git a/arch/riscv/net/bpf_jit_core.c b/arch/riscv/net/bpf_jit_core.c index 8d8c5eb2e8da..e439cb0bc5b0 100644 --- a/arch/riscv/net/bpf_jit_core.c +++ b/arch/riscv/net/bpf_jit_core.c @@ -176,8 +176,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) prog->jited_len = prog_size; if (!prog->is_func || extra_pass) { - if (WARN_ON(bpf_jit_binary_pack_finalize(prog, jit_data->ro_header, - jit_data->header))) { + if (WARN_ON(bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header))) { /* ro_header has been freed */ jit_data->ro_header = NULL; prog = orig_prog; @@ -269,7 +268,8 @@ void bpf_jit_free(struct bpf_prog *prog) * before freeing it. */ if (jit_data) { - bpf_jit_binary_pack_finalize(prog, jit_data->ro_header, jit_data->header); + bpf_jit_binary_pack_finalize(jit_data->ro_header, jit_data->header); + kfree(jit_data->ctx.offset); kfree(jit_data); } hdr = bpf_jit_binary_pack_hdr(prog); diff --git a/arch/s390/boot/vmem.c b/arch/s390/boot/vmem.c index 14e1a73ffcfe..978ee19bd8fc 100644 --- a/arch/s390/boot/vmem.c +++ b/arch/s390/boot/vmem.c @@ -327,7 +327,7 @@ static void pgtable_pmd_populate(pud_t *pud, unsigned long addr, unsigned long e } pte = boot_pte_alloc(); pmd_populate(&init_mm, pmd, pte); - } else if (pmd_large(*pmd)) { + } else if (pmd_leaf(*pmd)) { continue; } pgtable_pte_populate(pmd, addr, next, mode); diff --git a/arch/s390/include/asm/nmi.h b/arch/s390/include/asm/nmi.h index 227466ce9e41..da5a98183cb6 100644 --- a/arch/s390/include/asm/nmi.h +++ b/arch/s390/include/asm/nmi.h @@ -22,6 +22,7 @@ #define MCCK_CODE_SYSTEM_DAMAGE BIT(63) #define MCCK_CODE_EXT_DAMAGE BIT(63 - 5) #define MCCK_CODE_CP BIT(63 - 9) +#define MCCK_CODE_CK BIT(63 - 11) #define MCCK_CODE_STG_ERROR BIT(63 - 16) #define MCCK_CODE_STG_KEY_ERROR BIT(63 - 18) #define MCCK_CODE_STG_DEGRAD BIT(63 - 19) @@ -33,6 +34,8 @@ #define MCCK_CODE_FC_VALID BIT(63 - 43) #define MCCK_CODE_CPU_TIMER_VALID BIT(63 - 46) +#define MCCK_CODE_NO_GUEST (MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE | MCCK_CODE_CK) + #ifndef __ASSEMBLY__ union mci { diff --git a/arch/s390/include/asm/pci_insn.h b/arch/s390/include/asm/pci_insn.h index e5f57cfe1d45..5b2ed051f432 100644 --- a/arch/s390/include/asm/pci_insn.h +++ b/arch/s390/include/asm/pci_insn.h @@ -16,11 +16,11 @@ #define ZPCI_PCI_ST_FUNC_NOT_AVAIL 40 #define ZPCI_PCI_ST_ALREADY_IN_RQ_STATE 44 -/* Load/Store return codes */ -#define ZPCI_PCI_LS_OK 0 -#define ZPCI_PCI_LS_ERR 1 -#define ZPCI_PCI_LS_BUSY 2 -#define ZPCI_PCI_LS_INVAL_HANDLE 3 +/* PCI instruction condition codes */ +#define ZPCI_CC_OK 0 +#define ZPCI_CC_ERR 1 +#define ZPCI_CC_BUSY 2 +#define ZPCI_CC_INVAL_HANDLE 3 /* Load/Store address space identifiers */ #define ZPCI_PCIAS_MEMIO_0 0 diff --git a/arch/s390/include/asm/pgtable.h b/arch/s390/include/asm/pgtable.h index 2cc9d7bb1b2a..9c09370195d2 100644 --- a/arch/s390/include/asm/pgtable.h +++ b/arch/s390/include/asm/pgtable.h @@ -730,7 +730,7 @@ static inline int pmd_large(pmd_t pmd) static inline int pmd_bad(pmd_t pmd) { - if ((pmd_val(pmd) & _SEGMENT_ENTRY_TYPE_MASK) > 0 || pmd_large(pmd)) + if ((pmd_val(pmd) & _SEGMENT_ENTRY_TYPE_MASK) > 0 || pmd_leaf(pmd)) return 1; return (pmd_val(pmd) & ~_SEGMENT_ENTRY_BITS) != 0; } @@ -828,8 +828,8 @@ static inline int pte_protnone(pte_t pte) static inline int pmd_protnone(pmd_t pmd) { - /* pmd_large(pmd) implies pmd_present(pmd) */ - return pmd_large(pmd) && !(pmd_val(pmd) & _SEGMENT_ENTRY_READ); + /* pmd_leaf(pmd) implies pmd_present(pmd) */ + return pmd_leaf(pmd) && !(pmd_val(pmd) & _SEGMENT_ENTRY_READ); } #endif @@ -1389,7 +1389,7 @@ static inline unsigned long pmd_deref(pmd_t pmd) unsigned long origin_mask; origin_mask = _SEGMENT_ENTRY_ORIGIN; - if (pmd_large(pmd)) + if (pmd_leaf(pmd)) origin_mask = _SEGMENT_ENTRY_ORIGIN_LARGE; return (unsigned long)__va(pmd_val(pmd) & origin_mask); } diff --git a/arch/s390/kernel/nmi.c b/arch/s390/kernel/nmi.c index 38ec0487521c..bc07f6c11b33 100644 --- a/arch/s390/kernel/nmi.c +++ b/arch/s390/kernel/nmi.c @@ -347,8 +347,7 @@ static void notrace s390_backup_mcck_info(struct pt_regs *regs) sie_page = container_of(sie_block, struct sie_page, sie_block); mcck_backup = &sie_page->mcck_info; - mcck_backup->mcic = S390_lowcore.mcck_interruption_code & - ~(MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE); + mcck_backup->mcic = S390_lowcore.mcck_interruption_code & ~MCCK_CODE_NO_GUEST; mcck_backup->ext_damage_code = S390_lowcore.external_damage_code; mcck_backup->failing_storage_address = S390_lowcore.failing_storage_address; @@ -361,8 +360,6 @@ NOKPROBE_SYMBOL(s390_backup_mcck_info); #define ED_STP_ISLAND 6 /* External damage STP island check */ #define ED_STP_SYNC 7 /* External damage STP sync check */ -#define MCCK_CODE_NO_GUEST (MCCK_CODE_CP | MCCK_CODE_EXT_DAMAGE) - /* * machine check handler. */ diff --git a/arch/s390/kernel/perf_pai_crypto.c b/arch/s390/kernel/perf_pai_crypto.c index 044fba8332b2..b24ee5fbf958 100644 --- a/arch/s390/kernel/perf_pai_crypto.c +++ b/arch/s390/kernel/perf_pai_crypto.c @@ -378,7 +378,8 @@ static int paicrypt_push_sample(void) /* Called on schedule-in and schedule-out. No access to event structure, * but for sampling only event CRYPTO_ALL is allowed. */ -static void paicrypt_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) +static void paicrypt_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in) { /* We started with a clean page on event installation. So read out * results on schedule_out and if page was dirty, clear values. diff --git a/arch/s390/kernel/perf_pai_ext.c b/arch/s390/kernel/perf_pai_ext.c index 089bd3104b39..3b6f18675ab4 100644 --- a/arch/s390/kernel/perf_pai_ext.c +++ b/arch/s390/kernel/perf_pai_ext.c @@ -467,7 +467,8 @@ static int paiext_push_sample(void) /* Called on schedule-in and schedule-out. No access to event structure, * but for sampling only event NNPA_ALL is allowed. */ -static void paiext_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) +static void paiext_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in) { /* We started with a clean page on event installation. So read out * results on schedule_out and if page was dirty, clear values. diff --git a/arch/s390/kvm/interrupt.c b/arch/s390/kvm/interrupt.c index 4b3892584f5b..fe31bf352ce0 100644 --- a/arch/s390/kvm/interrupt.c +++ b/arch/s390/kvm/interrupt.c @@ -2862,9 +2862,7 @@ static int set_adapter_int(struct kvm_kernel_irq_routing_entry *e, void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu, struct mcck_volatile_info *mcck_info) { - struct kvm_s390_interrupt_info inti; struct kvm_s390_irq irq = {}; - struct kvm_s390_mchk_info *mchk; union mci mci; __u64 cr14 = 0; /* upper bits are not used */ int rc; @@ -2877,20 +2875,14 @@ void kvm_s390_reinject_machine_check(struct kvm_vcpu *vcpu, if (mci.w) cr14 |= CR14_WARNING_SUBMASK; - mchk = mci.ck ? &inti.mchk : &irq.u.mchk; - mchk->cr14 = cr14; - mchk->mcic = mcck_info->mcic; - mchk->ext_damage_code = mcck_info->ext_damage_code; - mchk->failing_storage_address = mcck_info->failing_storage_address; - if (mci.ck) { - /* Inject the floating machine check */ - inti.type = KVM_S390_MCHK; - rc = __inject_vm(vcpu->kvm, &inti); - } else { - /* Inject the machine check to specified vcpu */ - irq.type = KVM_S390_MCHK; - rc = kvm_s390_inject_vcpu(vcpu, &irq); - } + irq.u.mchk.cr14 = cr14; + irq.u.mchk.mcic = mcck_info->mcic; + irq.u.mchk.ext_damage_code = mcck_info->ext_damage_code; + irq.u.mchk.failing_storage_address = mcck_info->failing_storage_address; + + /* Inject the machine check to specified vcpu */ + irq.type = KVM_S390_MCHK; + rc = kvm_s390_inject_vcpu(vcpu, &irq); WARN_ON_ONCE(rc); } diff --git a/arch/s390/kvm/kvm-s390.c b/arch/s390/kvm/kvm-s390.c index cf075a893422..c0d8c59bb2cb 100644 --- a/arch/s390/kvm/kvm-s390.c +++ b/arch/s390/kvm/kvm-s390.c @@ -4597,8 +4597,8 @@ long kvm_arch_fault_in_page(struct kvm_vcpu *vcpu, gpa_t gpa, int writable) static void __kvm_inject_pfault_token(struct kvm_vcpu *vcpu, bool start_token, unsigned long token) { - struct kvm_s390_interrupt inti; - struct kvm_s390_irq irq; + struct kvm_s390_interrupt inti = {}; + struct kvm_s390_irq irq = {}; if (start_token) { irq.u.ext.ext_params2 = token; @@ -5769,6 +5769,9 @@ int kvm_arch_prepare_memory_region(struct kvm *kvm, { gpa_t size; + if (kvm_is_ucontrol(kvm)) + return -EINVAL; + /* When we are protected, we should not change the memory slots */ if (kvm_s390_pv_get_handle(kvm)) return -EINVAL; diff --git a/arch/s390/mm/fault.c b/arch/s390/mm/fault.c index 1a231181a413..9bafb1dbfafd 100644 --- a/arch/s390/mm/fault.c +++ b/arch/s390/mm/fault.c @@ -632,6 +632,8 @@ void do_secure_storage_access(struct pt_regs *regs) } fallthrough; case USER_FAULT: + if (faulthandler_disabled()) + return do_fault_error(regs, VM_FAULT_BADMAP); mm = current->mm; mmap_read_lock(mm); vma = find_vma(mm, addr); @@ -654,10 +656,17 @@ void do_secure_storage_access(struct pt_regs *regs) page = phys_to_page(addr); if (unlikely(!try_get_page(page))) break; - rc = arch_make_page_accessible(page); + rc = uv_convert_from_secure(page_to_phys(page)); + if (!rc) + clear_bit(PG_arch_1, &page->flags); put_page(page); + /* + * There are some valid fixup types for kernel + * accesses to donated secure memory. zeropad is one + * of them. + */ if (rc) - BUG(); + do_no_context(regs, VM_FAULT_BADACCESS); break; default: do_fault_error(regs, VM_FAULT_BADMAP); diff --git a/arch/s390/mm/gmap.c b/arch/s390/mm/gmap.c index 1a656db09c9f..b72facdb49f6 100644 --- a/arch/s390/mm/gmap.c +++ b/arch/s390/mm/gmap.c @@ -601,7 +601,7 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) pmd = pmd_offset(pud, vmaddr); VM_BUG_ON(pmd_none(*pmd)); /* Are we allowed to use huge pages? */ - if (pmd_large(*pmd) && !gmap->mm->context.allow_gmap_hpage_1m) + if (pmd_leaf(*pmd) && !gmap->mm->context.allow_gmap_hpage_1m) return -EFAULT; /* Link gmap segment table entry location to page table. */ rc = radix_tree_preload(GFP_KERNEL_ACCOUNT); @@ -613,7 +613,7 @@ int __gmap_link(struct gmap *gmap, unsigned long gaddr, unsigned long vmaddr) rc = radix_tree_insert(&gmap->host_to_guest, vmaddr >> PMD_SHIFT, table); if (!rc) { - if (pmd_large(*pmd)) { + if (pmd_leaf(*pmd)) { *table = (pmd_val(*pmd) & _SEGMENT_ENTRY_HARDWARE_BITS_LARGE) | _SEGMENT_ENTRY_GMAP_UC; @@ -943,7 +943,7 @@ static inline pmd_t *gmap_pmd_op_walk(struct gmap *gmap, unsigned long gaddr) } /* 4k page table entries are locked via the pte (pte_alloc_map_lock). */ - if (!pmd_large(*pmdp)) + if (!pmd_leaf(*pmdp)) spin_unlock(&gmap->guest_table_lock); return pmdp; } @@ -955,7 +955,7 @@ static inline pmd_t *gmap_pmd_op_walk(struct gmap *gmap, unsigned long gaddr) */ static inline void gmap_pmd_op_end(struct gmap *gmap, pmd_t *pmdp) { - if (pmd_large(*pmdp)) + if (pmd_leaf(*pmdp)) spin_unlock(&gmap->guest_table_lock); } @@ -1066,7 +1066,7 @@ static int gmap_protect_range(struct gmap *gmap, unsigned long gaddr, rc = -EAGAIN; pmdp = gmap_pmd_op_walk(gmap, gaddr); if (pmdp) { - if (!pmd_large(*pmdp)) { + if (!pmd_leaf(*pmdp)) { rc = gmap_protect_pte(gmap, gaddr, pmdp, prot, bits); if (!rc) { @@ -2498,7 +2498,7 @@ void gmap_sync_dirty_log_pmd(struct gmap *gmap, unsigned long bitmap[4], if (!pmdp) return; - if (pmd_large(*pmdp)) { + if (pmd_leaf(*pmdp)) { if (gmap_test_and_clear_dirty_pmd(gmap, pmdp, gaddr)) bitmap_fill(bitmap, _PAGE_ENTRIES); } else { diff --git a/arch/s390/mm/hugetlbpage.c b/arch/s390/mm/hugetlbpage.c index 5ebd730f1ffb..a74e1d9788d3 100644 --- a/arch/s390/mm/hugetlbpage.c +++ b/arch/s390/mm/hugetlbpage.c @@ -235,7 +235,7 @@ pte_t *huge_pte_offset(struct mm_struct *mm, int pmd_huge(pmd_t pmd) { - return pmd_large(pmd); + return pmd_leaf(pmd); } int pud_huge(pud_t pud) diff --git a/arch/s390/mm/pageattr.c b/arch/s390/mm/pageattr.c index 44271835c97e..ec8d3594cfc5 100644 --- a/arch/s390/mm/pageattr.c +++ b/arch/s390/mm/pageattr.c @@ -185,7 +185,7 @@ static int walk_pmd_level(pud_t *pudp, unsigned long addr, unsigned long end, if (pmd_none(*pmdp)) return -EINVAL; next = pmd_addr_end(addr, end); - if (pmd_large(*pmdp)) { + if (pmd_leaf(*pmdp)) { need_split = !!(flags & SET_MEMORY_4K); need_split |= !!(addr & ~PMD_MASK); need_split |= !!(addr + PMD_SIZE > next); diff --git a/arch/s390/mm/pgtable.c b/arch/s390/mm/pgtable.c index 1fb435b3913c..2319716ea03c 100644 --- a/arch/s390/mm/pgtable.c +++ b/arch/s390/mm/pgtable.c @@ -836,7 +836,7 @@ int set_guest_storage_key(struct mm_struct *mm, unsigned long addr, return key ? -EFAULT : 0; } - if (pmd_large(*pmdp)) { + if (pmd_leaf(*pmdp)) { paddr = pmd_val(*pmdp) & HPAGE_MASK; paddr |= addr & ~HPAGE_MASK; /* @@ -947,7 +947,7 @@ int reset_guest_reference_bit(struct mm_struct *mm, unsigned long addr) return 0; } - if (pmd_large(*pmdp)) { + if (pmd_leaf(*pmdp)) { paddr = pmd_val(*pmdp) & HPAGE_MASK; paddr |= addr & ~HPAGE_MASK; cc = page_reset_referenced(paddr); @@ -1011,7 +1011,7 @@ int get_guest_storage_key(struct mm_struct *mm, unsigned long addr, return 0; } - if (pmd_large(*pmdp)) { + if (pmd_leaf(*pmdp)) { paddr = pmd_val(*pmdp) & HPAGE_MASK; paddr |= addr & ~HPAGE_MASK; *key = page_get_storage_key(paddr); diff --git a/arch/s390/mm/vmem.c b/arch/s390/mm/vmem.c index 2d3f65da56ee..68df48d745bf 100644 --- a/arch/s390/mm/vmem.c +++ b/arch/s390/mm/vmem.c @@ -229,7 +229,7 @@ static int __ref modify_pmd_table(pud_t *pud, unsigned long addr, if (!add) { if (pmd_none(*pmd)) continue; - if (pmd_large(*pmd)) { + if (pmd_leaf(*pmd)) { if (IS_ALIGNED(addr, PMD_SIZE) && IS_ALIGNED(next, PMD_SIZE)) { if (!direct) @@ -274,7 +274,7 @@ static int __ref modify_pmd_table(pud_t *pud, unsigned long addr, if (!pte) goto out; pmd_populate(&init_mm, pmd, pte); - } else if (pmd_large(*pmd)) { + } else if (pmd_leaf(*pmd)) { if (!direct) vmemmap_use_sub_pmd(addr, next); continue; @@ -597,7 +597,7 @@ pte_t *vmem_get_alloc_pte(unsigned long addr, bool alloc) if (!pte) goto out; pmd_populate(&init_mm, pmd, pte); - } else if (WARN_ON_ONCE(pmd_large(*pmd))) { + } else if (WARN_ON_ONCE(pmd_leaf(*pmd))) { goto out; } ptep = pte_offset_kernel(pmd, addr); diff --git a/arch/s390/net/bpf_jit_comp.c b/arch/s390/net/bpf_jit_comp.c index 491789420d7c..eba2b42ff329 100644 --- a/arch/s390/net/bpf_jit_comp.c +++ b/arch/s390/net/bpf_jit_comp.c @@ -2015,7 +2015,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *fp) print_fn_code(jit.prg_buf, jit.size_prg); } if (!fp->is_func || extra_pass) { - bpf_jit_binary_lock_ro(header); + if (bpf_jit_binary_lock_ro(header)) { + bpf_jit_binary_free(header); + fp = orig_fp; + goto free_addrs; + } } else { jit_data->header = header; jit_data->ctx = jit; diff --git a/arch/sparc/kernel/Makefile b/arch/sparc/kernel/Makefile index 0984bb6f0f17..58ea4ef9b622 100644 --- a/arch/sparc/kernel/Makefile +++ b/arch/sparc/kernel/Makefile @@ -5,7 +5,6 @@ # asflags-y := -ansi -ccflags-y := -Werror # Undefine sparc when processing vmlinux.lds - it is used # And teach CPP we are doing $(BITS) builds (for this case) diff --git a/arch/sparc/lib/Makefile b/arch/sparc/lib/Makefile index 063556fe2cb1..59669ebddd4e 100644 --- a/arch/sparc/lib/Makefile +++ b/arch/sparc/lib/Makefile @@ -3,7 +3,6 @@ # asflags-y := -ansi -DST_DIV0=0x02 -ccflags-y := -Werror lib-$(CONFIG_SPARC32) += ashrdi3.o lib-$(CONFIG_SPARC32) += memcpy.o memset.o diff --git a/arch/sparc/mm/Makefile b/arch/sparc/mm/Makefile index 871354aa3c00..809d993f6d88 100644 --- a/arch/sparc/mm/Makefile +++ b/arch/sparc/mm/Makefile @@ -3,7 +3,6 @@ # asflags-y := -ansi -ccflags-y := -Werror obj-$(CONFIG_SPARC64) += ultra.o tlb.o tsb.o obj-y += fault_$(BITS).o diff --git a/arch/sparc/mm/init_64.c b/arch/sparc/mm/init_64.c index d7db4e737218..1ca9054d9b97 100644 --- a/arch/sparc/mm/init_64.c +++ b/arch/sparc/mm/init_64.c @@ -1672,7 +1672,7 @@ bool kern_addr_valid(unsigned long addr) if (pmd_none(*pmd)) return false; - if (pmd_large(*pmd)) + if (pmd_leaf(*pmd)) return pfn_valid(pmd_pfn(*pmd)); pte = pte_offset_kernel(pmd, addr); @@ -2968,7 +2968,7 @@ void update_mmu_cache_pmd(struct vm_area_struct *vma, unsigned long addr, struct mm_struct *mm; pmd_t entry = *pmd; - if (!pmd_large(entry) || !pmd_young(entry)) + if (!pmd_leaf(entry) || !pmd_young(entry)) return; pte = pmd_val(entry); diff --git a/arch/sparc/net/bpf_jit_comp_64.c b/arch/sparc/net/bpf_jit_comp_64.c index fa0759bfe498..73bf0aea8baf 100644 --- a/arch/sparc/net/bpf_jit_comp_64.c +++ b/arch/sparc/net/bpf_jit_comp_64.c @@ -1602,7 +1602,11 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) bpf_flush_icache(header, (u8 *)header + header->size); if (!prog->is_func || extra_pass) { - bpf_jit_binary_lock_ro(header); + if (bpf_jit_binary_lock_ro(header)) { + bpf_jit_binary_free(header); + prog = orig_prog; + goto out_off; + } } else { jit_data->ctx = ctx; jit_data->image = image_ptr; diff --git a/arch/sparc/prom/Makefile b/arch/sparc/prom/Makefile index 397b79af77f7..a1adc75d8055 100644 --- a/arch/sparc/prom/Makefile +++ b/arch/sparc/prom/Makefile @@ -3,7 +3,6 @@ # Linux. # asflags := -ansi -ccflags := -Werror lib-y := bootstr_$(BITS).o lib-y += init_$(BITS).o diff --git a/arch/um/kernel/skas/syscall.c b/arch/um/kernel/skas/syscall.c index 9ee19e566da3..b09e85279d2b 100644 --- a/arch/um/kernel/skas/syscall.c +++ b/arch/um/kernel/skas/syscall.c @@ -12,23 +12,13 @@ #include #include #include +#include void handle_syscall(struct uml_pt_regs *r) { struct pt_regs *regs = container_of(r, struct pt_regs, regs); int syscall; - /* - * If we have infinite CPU resources, then make every syscall also a - * preemption point, since we don't have any other preemption in this - * case, and kernel threads would basically never run until userspace - * went to sleep, even if said userspace interacts with the kernel in - * various ways. - */ - if (time_travel_mode == TT_MODE_INFCPU || - time_travel_mode == TT_MODE_EXTERNAL) - schedule(); - /* Initialize the syscall number and default return value. */ UPT_SYSCALL_NR(r) = PT_SYSCALL_NR(r->gp); PT_REGS_SET_SYSCALL_RETURN(regs, -ENOSYS); @@ -41,9 +31,25 @@ void handle_syscall(struct uml_pt_regs *r) goto out; syscall = UPT_SYSCALL_NR(r); - if (syscall >= 0 && syscall < __NR_syscalls) - PT_REGS_SET_SYSCALL_RETURN(regs, - EXECUTE_SYSCALL(syscall, regs)); + if (syscall >= 0 && syscall < __NR_syscalls) { + unsigned long ret = EXECUTE_SYSCALL(syscall, regs); + + PT_REGS_SET_SYSCALL_RETURN(regs, ret); + + /* + * An error value here can be some form of -ERESTARTSYS + * and then we'd just loop. Make any error syscalls take + * some time, so that it won't just loop if something is + * not ready, and hopefully other things will make some + * progress. + */ + if (IS_ERR_VALUE(ret) && + (time_travel_mode == TT_MODE_INFCPU || + time_travel_mode == TT_MODE_EXTERNAL)) { + um_udelay(1); + schedule(); + } + } out: syscall_trace_leave(regs); diff --git a/arch/x86/boot/compressed/ident_map_64.c b/arch/x86/boot/compressed/ident_map_64.c index d040080d7edb..71c6e2fdcec7 100644 --- a/arch/x86/boot/compressed/ident_map_64.c +++ b/arch/x86/boot/compressed/ident_map_64.c @@ -284,7 +284,7 @@ static int set_clr_page_flags(struct x86_mapping_info *info, pudp = pud_offset(p4dp, address); pmdp = pmd_offset(pudp, address); - if (pmd_large(*pmdp)) + if (pmd_leaf(*pmdp)) ptep = split_large_pmd(info, pmdp, address); else ptep = pte_offset_kernel(pmdp, address); diff --git a/arch/x86/crypto/Kconfig b/arch/x86/crypto/Kconfig index 9bbfd01cfa2f..83f0a4229dba 100644 --- a/arch/x86/crypto/Kconfig +++ b/arch/x86/crypto/Kconfig @@ -358,7 +358,7 @@ config CRYPTO_CHACHA20_X86_64 - AVX-512VL (Advanced Vector Extensions-512VL) config CRYPTO_AEGIS128_AESNI_SSE2 - tristate "AEAD ciphers: AEGIS-128 (AES-NI/SSE2)" + tristate "AEAD ciphers: AEGIS-128 (AES-NI/SSE4.1)" depends on X86 && 64BIT select CRYPTO_AEAD select CRYPTO_SIMD @@ -367,7 +367,7 @@ config CRYPTO_AEGIS128_AESNI_SSE2 Architecture: x86_64 using: - AES-NI (AES New Instructions) - - SSE2 (Streaming SIMD Extensions 2) + - SSE4.1 (Streaming SIMD Extensions 4.1) config CRYPTO_NHPOLY1305_SSE2 tristate "Hash functions: NHPoly1305 (SSE2)" diff --git a/arch/x86/crypto/aegis128-aesni-asm.S b/arch/x86/crypto/aegis128-aesni-asm.S index 2de859173940..6520d0ed9814 100644 --- a/arch/x86/crypto/aegis128-aesni-asm.S +++ b/arch/x86/crypto/aegis128-aesni-asm.S @@ -1,13 +1,12 @@ /* SPDX-License-Identifier: GPL-2.0-only */ /* - * AES-NI + SSE2 implementation of AEGIS-128 + * AES-NI + SSE4.1 implementation of AEGIS-128 * * Copyright (c) 2017-2018 Ondrej Mosnacek * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. */ #include -#include #include #define STATE0 %xmm0 @@ -20,11 +19,6 @@ #define T0 %xmm6 #define T1 %xmm7 -#define STATEP %rdi -#define LEN %esi -#define SRC %rdx -#define DST %rcx - .section .rodata.cst16.aegis128_const, "aM", @progbits, 32 .align 16 .Laegis128_const_0: @@ -73,6 +67,8 @@ * %r9 */ SYM_FUNC_START_LOCAL(__load_partial) + .set LEN, %ecx + .set SRC, %rsi xor %r9d, %r9d pxor MSG, MSG @@ -139,6 +135,8 @@ SYM_FUNC_END(__load_partial) * %r10 */ SYM_FUNC_START_LOCAL(__store_partial) + .set LEN, %ecx + .set DST, %rdx mov LEN, %r8d mov DST, %r9 @@ -185,16 +183,21 @@ SYM_FUNC_START_LOCAL(__store_partial) SYM_FUNC_END(__store_partial) /* - * void crypto_aegis128_aesni_init(void *state, const void *key, const void *iv); + * void aegis128_aesni_init(struct aegis_state *state, + * const struct aegis_block *key, + * const u8 iv[AEGIS128_NONCE_SIZE]); */ -SYM_FUNC_START(crypto_aegis128_aesni_init) +SYM_FUNC_START(aegis128_aesni_init) + .set STATEP, %rdi + .set KEYP, %rsi + .set IVP, %rdx FRAME_BEGIN /* load IV: */ - movdqu (%rdx), T1 + movdqu (IVP), T1 /* load key: */ - movdqa (%rsi), KEY + movdqa (KEYP), KEY pxor KEY, T1 movdqa T1, STATE0 movdqa KEY, STATE3 @@ -227,13 +230,16 @@ SYM_FUNC_START(crypto_aegis128_aesni_init) FRAME_END RET -SYM_FUNC_END(crypto_aegis128_aesni_init) +SYM_FUNC_END(aegis128_aesni_init) /* - * void crypto_aegis128_aesni_ad(void *state, unsigned int length, - * const void *data); + * void aegis128_aesni_ad(struct aegis_state *state, const u8 *data, + * unsigned int len); */ -SYM_FUNC_START(crypto_aegis128_aesni_ad) +SYM_FUNC_START(aegis128_aesni_ad) + .set STATEP, %rdi + .set SRC, %rsi + .set LEN, %edx FRAME_BEGIN cmp $0x10, LEN @@ -379,7 +385,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_ad) .Lad_out: FRAME_END RET -SYM_FUNC_END(crypto_aegis128_aesni_ad) +SYM_FUNC_END(aegis128_aesni_ad) .macro encrypt_block a s0 s1 s2 s3 s4 i movdq\a (\i * 0x10)(SRC), MSG @@ -400,10 +406,14 @@ SYM_FUNC_END(crypto_aegis128_aesni_ad) .endm /* - * void crypto_aegis128_aesni_enc(void *state, unsigned int length, - * const void *src, void *dst); + * void aegis128_aesni_enc(struct aegis_state *state, const u8 *src, u8 *dst, + * unsigned int len); */ -SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc) +SYM_FUNC_START(aegis128_aesni_enc) + .set STATEP, %rdi + .set SRC, %rsi + .set DST, %rdx + .set LEN, %ecx FRAME_BEGIN cmp $0x10, LEN @@ -494,13 +504,17 @@ SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc) .Lenc_out: FRAME_END RET -SYM_FUNC_END(crypto_aegis128_aesni_enc) +SYM_FUNC_END(aegis128_aesni_enc) /* - * void crypto_aegis128_aesni_enc_tail(void *state, unsigned int length, - * const void *src, void *dst); + * void aegis128_aesni_enc_tail(struct aegis_state *state, const u8 *src, + * u8 *dst, unsigned int len); */ -SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc_tail) +SYM_FUNC_START(aegis128_aesni_enc_tail) + .set STATEP, %rdi + .set SRC, %rsi + .set DST, %rdx + .set LEN, %ecx FRAME_BEGIN /* load the state: */ @@ -534,7 +548,7 @@ SYM_TYPED_FUNC_START(crypto_aegis128_aesni_enc_tail) FRAME_END RET -SYM_FUNC_END(crypto_aegis128_aesni_enc_tail) +SYM_FUNC_END(aegis128_aesni_enc_tail) .macro decrypt_block a s0 s1 s2 s3 s4 i movdq\a (\i * 0x10)(SRC), MSG @@ -554,10 +568,14 @@ SYM_FUNC_END(crypto_aegis128_aesni_enc_tail) .endm /* - * void crypto_aegis128_aesni_dec(void *state, unsigned int length, - * const void *src, void *dst); + * void aegis128_aesni_dec(struct aegis_state *state, const u8 *src, u8 *dst, + * unsigned int len); */ -SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec) +SYM_FUNC_START(aegis128_aesni_dec) + .set STATEP, %rdi + .set SRC, %rsi + .set DST, %rdx + .set LEN, %ecx FRAME_BEGIN cmp $0x10, LEN @@ -648,13 +666,17 @@ SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec) .Ldec_out: FRAME_END RET -SYM_FUNC_END(crypto_aegis128_aesni_dec) +SYM_FUNC_END(aegis128_aesni_dec) /* - * void crypto_aegis128_aesni_dec_tail(void *state, unsigned int length, - * const void *src, void *dst); + * void aegis128_aesni_dec_tail(struct aegis_state *state, const u8 *src, + * u8 *dst, unsigned int len); */ -SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec_tail) +SYM_FUNC_START(aegis128_aesni_dec_tail) + .set STATEP, %rdi + .set SRC, %rsi + .set DST, %rdx + .set LEN, %ecx FRAME_BEGIN /* load the state: */ @@ -698,14 +720,18 @@ SYM_TYPED_FUNC_START(crypto_aegis128_aesni_dec_tail) FRAME_END RET -SYM_FUNC_END(crypto_aegis128_aesni_dec_tail) +SYM_FUNC_END(aegis128_aesni_dec_tail) /* - * void crypto_aegis128_aesni_final(void *state, void *tag_xor, - * unsigned int assoclen, - * unsigned int cryptlen); + * void aegis128_aesni_final(struct aegis_state *state, + * struct aegis_block *tag_xor, + * unsigned int assoclen, unsigned int cryptlen); */ -SYM_FUNC_START(crypto_aegis128_aesni_final) +SYM_FUNC_START(aegis128_aesni_final) + .set STATEP, %rdi + .set TAG_XOR, %rsi + .set ASSOCLEN, %edx + .set CRYPTLEN, %ecx FRAME_BEGIN /* load the state: */ @@ -716,10 +742,8 @@ SYM_FUNC_START(crypto_aegis128_aesni_final) movdqu 0x40(STATEP), STATE4 /* prepare length block: */ - movd %edx, MSG - movd %ecx, T0 - pslldq $8, T0 - pxor T0, MSG + movd ASSOCLEN, MSG + pinsrd $2, CRYPTLEN, MSG psllq $3, MSG /* multiply by 8 (to get bit count) */ pxor STATE3, MSG @@ -734,7 +758,7 @@ SYM_FUNC_START(crypto_aegis128_aesni_final) aegis128_update; pxor MSG, STATE3 /* xor tag: */ - movdqu (%rsi), MSG + movdqu (TAG_XOR), MSG pxor STATE0, MSG pxor STATE1, MSG @@ -742,8 +766,8 @@ SYM_FUNC_START(crypto_aegis128_aesni_final) pxor STATE3, MSG pxor STATE4, MSG - movdqu MSG, (%rsi) + movdqu MSG, (TAG_XOR) FRAME_END RET -SYM_FUNC_END(crypto_aegis128_aesni_final) +SYM_FUNC_END(aegis128_aesni_final) diff --git a/arch/x86/crypto/aegis128-aesni-glue.c b/arch/x86/crypto/aegis128-aesni-glue.c index 4623189000d8..f5bbf274a54e 100644 --- a/arch/x86/crypto/aegis128-aesni-glue.c +++ b/arch/x86/crypto/aegis128-aesni-glue.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-or-later /* * The AEGIS-128 Authenticated-Encryption Algorithm - * Glue for AES-NI + SSE2 implementation + * Glue for AES-NI + SSE4.1 implementation * * Copyright (c) 2017-2018 Ondrej Mosnacek * Copyright (C) 2017-2018 Red Hat, Inc. All rights reserved. @@ -23,27 +23,6 @@ #define AEGIS128_MIN_AUTH_SIZE 8 #define AEGIS128_MAX_AUTH_SIZE 16 -asmlinkage void crypto_aegis128_aesni_init(void *state, void *key, void *iv); - -asmlinkage void crypto_aegis128_aesni_ad( - void *state, unsigned int length, const void *data); - -asmlinkage void crypto_aegis128_aesni_enc( - void *state, unsigned int length, const void *src, void *dst); - -asmlinkage void crypto_aegis128_aesni_dec( - void *state, unsigned int length, const void *src, void *dst); - -asmlinkage void crypto_aegis128_aesni_enc_tail( - void *state, unsigned int length, const void *src, void *dst); - -asmlinkage void crypto_aegis128_aesni_dec_tail( - void *state, unsigned int length, const void *src, void *dst); - -asmlinkage void crypto_aegis128_aesni_final( - void *state, void *tag_xor, unsigned int cryptlen, - unsigned int assoclen); - struct aegis_block { u8 bytes[AEGIS128_BLOCK_SIZE] __aligned(AEGIS128_BLOCK_ALIGN); }; @@ -56,15 +35,31 @@ struct aegis_ctx { struct aegis_block key; }; -struct aegis_crypt_ops { - int (*skcipher_walk_init)(struct skcipher_walk *walk, - struct aead_request *req, bool atomic); +asmlinkage void aegis128_aesni_init(struct aegis_state *state, + const struct aegis_block *key, + const u8 iv[AEGIS128_NONCE_SIZE]); - void (*crypt_blocks)(void *state, unsigned int length, const void *src, - void *dst); - void (*crypt_tail)(void *state, unsigned int length, const void *src, - void *dst); -}; +asmlinkage void aegis128_aesni_ad(struct aegis_state *state, const u8 *data, + unsigned int len); + +asmlinkage void aegis128_aesni_enc(struct aegis_state *state, const u8 *src, + u8 *dst, unsigned int len); + +asmlinkage void aegis128_aesni_dec(struct aegis_state *state, const u8 *src, + u8 *dst, unsigned int len); + +asmlinkage void aegis128_aesni_enc_tail(struct aegis_state *state, + const u8 *src, u8 *dst, + unsigned int len); + +asmlinkage void aegis128_aesni_dec_tail(struct aegis_state *state, + const u8 *src, u8 *dst, + unsigned int len); + +asmlinkage void aegis128_aesni_final(struct aegis_state *state, + struct aegis_block *tag_xor, + unsigned int assoclen, + unsigned int cryptlen); static void crypto_aegis128_aesni_process_ad( struct aegis_state *state, struct scatterlist *sg_src, @@ -85,15 +80,14 @@ static void crypto_aegis128_aesni_process_ad( if (pos > 0) { unsigned int fill = AEGIS128_BLOCK_SIZE - pos; memcpy(buf.bytes + pos, src, fill); - crypto_aegis128_aesni_ad(state, - AEGIS128_BLOCK_SIZE, - buf.bytes); + aegis128_aesni_ad(state, buf.bytes, + AEGIS128_BLOCK_SIZE); pos = 0; left -= fill; src += fill; } - crypto_aegis128_aesni_ad(state, left, src); + aegis128_aesni_ad(state, src, left); src += left & ~(AEGIS128_BLOCK_SIZE - 1); left &= AEGIS128_BLOCK_SIZE - 1; @@ -110,26 +104,47 @@ static void crypto_aegis128_aesni_process_ad( if (pos > 0) { memset(buf.bytes + pos, 0, AEGIS128_BLOCK_SIZE - pos); - crypto_aegis128_aesni_ad(state, AEGIS128_BLOCK_SIZE, buf.bytes); + aegis128_aesni_ad(state, buf.bytes, AEGIS128_BLOCK_SIZE); } } -static void crypto_aegis128_aesni_process_crypt( - struct aegis_state *state, struct skcipher_walk *walk, - const struct aegis_crypt_ops *ops) +static __always_inline int +crypto_aegis128_aesni_process_crypt(struct aegis_state *state, + struct skcipher_walk *walk, bool enc) { + int err = 0; + while (walk->nbytes >= AEGIS128_BLOCK_SIZE) { - ops->crypt_blocks(state, - round_down(walk->nbytes, AEGIS128_BLOCK_SIZE), - walk->src.virt.addr, walk->dst.virt.addr); - skcipher_walk_done(walk, walk->nbytes % AEGIS128_BLOCK_SIZE); + if (enc) + aegis128_aesni_enc(state, walk->src.virt.addr, + walk->dst.virt.addr, + round_down(walk->nbytes, + AEGIS128_BLOCK_SIZE)); + else + aegis128_aesni_dec(state, walk->src.virt.addr, + walk->dst.virt.addr, + round_down(walk->nbytes, + AEGIS128_BLOCK_SIZE)); + kernel_fpu_end(); + err = skcipher_walk_done(walk, + walk->nbytes % AEGIS128_BLOCK_SIZE); + kernel_fpu_begin(); } if (walk->nbytes) { - ops->crypt_tail(state, walk->nbytes, walk->src.virt.addr, - walk->dst.virt.addr); - skcipher_walk_done(walk, 0); + if (enc) + aegis128_aesni_enc_tail(state, walk->src.virt.addr, + walk->dst.virt.addr, + walk->nbytes); + else + aegis128_aesni_dec_tail(state, walk->src.virt.addr, + walk->dst.virt.addr, + walk->nbytes); + kernel_fpu_end(); + err = skcipher_walk_done(walk, 0); + kernel_fpu_begin(); } + return err; } static struct aegis_ctx *crypto_aegis128_aesni_ctx(struct crypto_aead *aead) @@ -162,42 +177,46 @@ static int crypto_aegis128_aesni_setauthsize(struct crypto_aead *tfm, return 0; } -static void crypto_aegis128_aesni_crypt(struct aead_request *req, - struct aegis_block *tag_xor, - unsigned int cryptlen, - const struct aegis_crypt_ops *ops) +static __always_inline int +crypto_aegis128_aesni_crypt(struct aead_request *req, + struct aegis_block *tag_xor, + unsigned int cryptlen, bool enc) { struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct aegis_ctx *ctx = crypto_aegis128_aesni_ctx(tfm); struct skcipher_walk walk; struct aegis_state state; + int err; - ops->skcipher_walk_init(&walk, req, true); + if (enc) + err = skcipher_walk_aead_encrypt(&walk, req, false); + else + err = skcipher_walk_aead_decrypt(&walk, req, false); + if (err) + return err; kernel_fpu_begin(); - crypto_aegis128_aesni_init(&state, ctx->key.bytes, req->iv); + aegis128_aesni_init(&state, &ctx->key, req->iv); crypto_aegis128_aesni_process_ad(&state, req->src, req->assoclen); - crypto_aegis128_aesni_process_crypt(&state, &walk, ops); - crypto_aegis128_aesni_final(&state, tag_xor, req->assoclen, cryptlen); - + err = crypto_aegis128_aesni_process_crypt(&state, &walk, enc); + if (err == 0) + aegis128_aesni_final(&state, tag_xor, req->assoclen, cryptlen); kernel_fpu_end(); + return err; } static int crypto_aegis128_aesni_encrypt(struct aead_request *req) { - static const struct aegis_crypt_ops OPS = { - .skcipher_walk_init = skcipher_walk_aead_encrypt, - .crypt_blocks = crypto_aegis128_aesni_enc, - .crypt_tail = crypto_aegis128_aesni_enc_tail, - }; - struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct aegis_block tag = {}; unsigned int authsize = crypto_aead_authsize(tfm); unsigned int cryptlen = req->cryptlen; + int err; - crypto_aegis128_aesni_crypt(req, &tag, cryptlen, &OPS); + err = crypto_aegis128_aesni_crypt(req, &tag, cryptlen, true); + if (err) + return err; scatterwalk_map_and_copy(tag.bytes, req->dst, req->assoclen + cryptlen, authsize, 1); @@ -208,21 +227,18 @@ static int crypto_aegis128_aesni_decrypt(struct aead_request *req) { static const struct aegis_block zeros = {}; - static const struct aegis_crypt_ops OPS = { - .skcipher_walk_init = skcipher_walk_aead_decrypt, - .crypt_blocks = crypto_aegis128_aesni_dec, - .crypt_tail = crypto_aegis128_aesni_dec_tail, - }; - struct crypto_aead *tfm = crypto_aead_reqtfm(req); struct aegis_block tag; unsigned int authsize = crypto_aead_authsize(tfm); unsigned int cryptlen = req->cryptlen - authsize; + int err; scatterwalk_map_and_copy(tag.bytes, req->src, req->assoclen + cryptlen, authsize, 0); - crypto_aegis128_aesni_crypt(req, &tag, cryptlen, &OPS); + err = crypto_aegis128_aesni_crypt(req, &tag, cryptlen, false); + if (err) + return err; return crypto_memneq(tag.bytes, zeros.bytes, authsize) ? -EBADMSG : 0; } @@ -267,7 +283,7 @@ static struct simd_aead_alg *simd_alg; static int __init crypto_aegis128_aesni_module_init(void) { - if (!boot_cpu_has(X86_FEATURE_XMM2) || + if (!boot_cpu_has(X86_FEATURE_XMM4_1) || !boot_cpu_has(X86_FEATURE_AES) || !cpu_has_xfeatures(XFEATURE_MASK_SSE, NULL)) return -ENODEV; @@ -286,6 +302,6 @@ module_exit(crypto_aegis128_aesni_module_exit); MODULE_LICENSE("GPL"); MODULE_AUTHOR("Ondrej Mosnacek "); -MODULE_DESCRIPTION("AEGIS-128 AEAD algorithm -- AESNI+SSE2 implementation"); +MODULE_DESCRIPTION("AEGIS-128 AEAD algorithm -- AESNI+SSE4.1 implementation"); MODULE_ALIAS_CRYPTO("aegis128"); MODULE_ALIAS_CRYPTO("aegis128-aesni"); diff --git a/arch/x86/events/amd/brs.c b/arch/x86/events/amd/brs.c index 01fcd43522bc..d752242ecc0d 100644 --- a/arch/x86/events/amd/brs.c +++ b/arch/x86/events/amd/brs.c @@ -381,7 +381,8 @@ static void amd_brs_poison_buffer(void) * On ctxswin, sched_in = true, called after the PMU has started * On ctxswout, sched_in = false, called before the PMU is stopped */ -void amd_pmu_brs_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) +void amd_pmu_brs_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in) { struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); diff --git a/arch/x86/events/amd/core.c b/arch/x86/events/amd/core.c index 24f356e561e2..b0126fbb39d1 100644 --- a/arch/x86/events/amd/core.c +++ b/arch/x86/events/amd/core.c @@ -733,13 +733,11 @@ static void amd_pmu_enable_event(struct perf_event *event) x86_pmu_enable_event(event); } -static void amd_pmu_enable_all(int added) +static void __amd_pmu_enable_all(void) { struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); int idx; - amd_brs_enable_all(); - for (idx = 0; idx < x86_pmu.num_counters; idx++) { /* only activate events which are marked as active */ if (!test_bit(idx, cpuc->active_mask)) @@ -754,6 +752,12 @@ static void amd_pmu_enable_all(int added) } } +static void amd_pmu_enable_all(int added) +{ + amd_brs_enable_all(); + __amd_pmu_enable_all(); +} + static void amd_pmu_v2_enable_event(struct perf_event *event) { struct hw_perf_event *hwc = &event->hw; @@ -1505,7 +1509,7 @@ static inline void amd_pmu_reload_virt(void) * set global enable bits once again */ amd_pmu_v2_disable_all(); - amd_pmu_enable_all(0); + __amd_pmu_enable_all(); amd_pmu_v2_enable_all(0); return; } diff --git a/arch/x86/events/amd/lbr.c b/arch/x86/events/amd/lbr.c index 0788c11febf6..0c1e7f0a613c 100644 --- a/arch/x86/events/amd/lbr.c +++ b/arch/x86/events/amd/lbr.c @@ -376,7 +376,8 @@ void amd_pmu_lbr_del(struct perf_event *event) perf_sched_cb_dec(event->pmu); } -void amd_pmu_lbr_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) +void amd_pmu_lbr_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in) { struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); diff --git a/arch/x86/events/core.c b/arch/x86/events/core.c index 1519ab15a544..497a1c20277f 100644 --- a/arch/x86/events/core.c +++ b/arch/x86/events/core.c @@ -2626,9 +2626,10 @@ static const struct attribute_group *x86_pmu_attr_groups[] = { NULL, }; -static void x86_pmu_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) +static void x86_pmu_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in) { - static_call_cond(x86_pmu_sched_task)(pmu_ctx, sched_in); + static_call_cond(x86_pmu_sched_task)(pmu_ctx, task, sched_in); } static void x86_pmu_swap_task_ctx(struct perf_event_pmu_context *prev_epc, diff --git a/arch/x86/events/intel/core.c b/arch/x86/events/intel/core.c index db5ad012dd40..9a87489887b8 100644 --- a/arch/x86/events/intel/core.c +++ b/arch/x86/events/intel/core.c @@ -2526,6 +2526,27 @@ static void intel_pmu_del_event(struct perf_event *event) intel_pmu_pebs_del(event); } +int __intel_pmu_quiesce(void) +{ + struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); + int pmu_enabled = cpuc->enabled; + + cpuc->enabled = 0; + if (pmu_enabled) + intel_pmu_disable_all(); + + return pmu_enabled; +} + +void __intel_pmu_resume(int pmu_enabled) +{ + struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); + + cpuc->enabled = pmu_enabled; + if (pmu_enabled) + intel_pmu_enable_all(0); +} + static int icl_set_topdown_event_period(struct perf_event *event) { struct hw_perf_event *hwc = &event->hw; @@ -2724,24 +2745,19 @@ static void intel_pmu_read_event(struct perf_event *event) { if (event->hw.flags & (PERF_X86_EVENT_AUTO_RELOAD | PERF_X86_EVENT_TOPDOWN)) { struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); - bool pmu_enabled = cpuc->enabled; + int pmu_enabled; /* Only need to call update_topdown_event() once for group read. */ if (is_metric_event(event) && (cpuc->txn_flags & PERF_PMU_TXN_READ)) return; - cpuc->enabled = 0; - if (pmu_enabled) - intel_pmu_disable_all(); + pmu_enabled = __intel_pmu_quiesce(); if (is_topdown_count(event)) static_call(intel_pmu_update_topdown_event)(event); else intel_pmu_drain_pebs_buffer(); - - cpuc->enabled = pmu_enabled; - if (pmu_enabled) - intel_pmu_enable_all(0); + __intel_pmu_resume(pmu_enabled); return; } @@ -4963,10 +4979,10 @@ static void intel_pmu_cpu_dead(int cpu) } static void intel_pmu_sched_task(struct perf_event_pmu_context *pmu_ctx, - bool sched_in) + struct task_struct *task, bool sched_in) { intel_pmu_pebs_sched_task(pmu_ctx, sched_in); - intel_pmu_lbr_sched_task(pmu_ctx, sched_in); + intel_pmu_lbr_sched_task(pmu_ctx, task, sched_in); } static void intel_pmu_swap_task_ctx(struct perf_event_pmu_context *prev_epc, @@ -6185,12 +6201,6 @@ __init int intel_pmu_init(void) x86_add_quirk(intel_arch_events_quirk); /* Install first, so it runs last */ - if (version >= 5) { - x86_pmu.intel_cap.anythread_deprecated = edx.split.anythread_deprecated; - if (x86_pmu.intel_cap.anythread_deprecated) - pr_cont(" AnyThread deprecated, "); - } - /* * Install the hw-cache-events table: */ @@ -6396,7 +6406,7 @@ __init int intel_pmu_init(void) x86_pmu.flags |= PMU_FL_INSTR_LATENCY; intel_pmu_pebs_data_source_grt(); - x86_pmu.pebs_latency_data = adl_latency_data_small; + x86_pmu.pebs_latency_data = grt_latency_data; x86_pmu.get_event_constraints = tnt_get_event_constraints; x86_pmu.limit_period = spr_limit_period; td_attr = tnt_events_attrs; @@ -6816,7 +6826,7 @@ __init int intel_pmu_init(void) x86_pmu.flags |= PMU_FL_NO_HT_SHARING; x86_pmu.flags |= PMU_FL_INSTR_LATENCY; x86_pmu.lbr_pt_coexist = true; - x86_pmu.pebs_latency_data = adl_latency_data_small; + x86_pmu.pebs_latency_data = grt_latency_data; x86_pmu.num_topdown_events = 8; static_call_update(intel_pmu_update_topdown_event, &adl_update_topdown_event); @@ -6981,8 +6991,10 @@ __init int intel_pmu_init(void) (u64)fixed_mask); /* AnyThread may be deprecated on arch perfmon v5 or later */ - if (x86_pmu.intel_cap.anythread_deprecated) + if (version >= 5 && edx.split.anythread_deprecated) { x86_pmu.format_attrs = intel_arch_formats_attr; + pr_cont("AnyThread deprecated, "); + } intel_pmu_check_event_constraints(x86_pmu.event_constraints, x86_pmu.num_counters, diff --git a/arch/x86/events/intel/ds.c b/arch/x86/events/intel/ds.c index 0c8c0e50262f..5f1867468b7c 100644 --- a/arch/x86/events/intel/ds.c +++ b/arch/x86/events/intel/ds.c @@ -256,12 +256,14 @@ static inline void pebs_set_tlb_lock(u64 *val, bool tlb, bool lock) } /* Retrieve the latency data for e-core of ADL */ -static u64 __adl_latency_data_small(struct perf_event *event, u64 status, - u8 dse, bool tlb, bool lock, bool blk) +static u64 __grt_latency_data(struct perf_event *event, u64 status, + u8 dse, bool tlb, bool lock, bool blk) { + union perf_mem_data_src src; u64 val; - WARN_ON_ONCE(hybrid_pmu(event->pmu)->cpu_type == hybrid_big); + WARN_ON_ONCE(is_hybrid() && + hybrid_pmu(event->pmu)->cpu_type == hybrid_big); dse &= PERF_PEBS_DATA_SOURCE_MASK; val = hybrid_var(event->pmu, pebs_data_source)[dse]; @@ -273,18 +275,27 @@ static u64 __adl_latency_data_small(struct perf_event *event, u64 status, else val |= P(BLK, NA); - return val; + src.val = val; + + if (event->hw.flags & + (PERF_X86_EVENT_PEBS_LDLAT | PERF_X86_EVENT_PEBS_LD_HSW)) + src.mem_op = P(OP, LOAD); + if (event->hw.flags & + (PERF_X86_EVENT_PEBS_STLAT | PERF_X86_EVENT_PEBS_ST_HSW)) + src.mem_op = P(OP, STORE); + + return src.val; } -u64 adl_latency_data_small(struct perf_event *event, u64 status) +u64 grt_latency_data(struct perf_event *event, u64 status) { union intel_x86_pebs_dse dse; dse.val = status; - return __adl_latency_data_small(event, status, dse.ld_dse, - dse.ld_locked, dse.ld_stlb_miss, - dse.ld_data_blk); + return __grt_latency_data(event, status, dse.ld_dse, + dse.ld_locked, dse.ld_stlb_miss, + dse.ld_data_blk); } /* Retrieve the latency data for e-core of MTL */ @@ -294,9 +305,9 @@ u64 mtl_latency_data_small(struct perf_event *event, u64 status) dse.val = status; - return __adl_latency_data_small(event, status, dse.mtl_dse, - dse.mtl_stlb_miss, dse.mtl_locked, - dse.mtl_fwd_blk); + return __grt_latency_data(event, status, dse.mtl_dse, + dse.mtl_stlb_miss, dse.mtl_locked, + dse.mtl_fwd_blk); } static u64 load_latency_data(struct perf_event *event, u64 status) @@ -845,8 +856,11 @@ int intel_pmu_drain_bts_buffer(void) void intel_pmu_drain_pebs_buffer(void) { + struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); struct perf_sample_data data; + WARN_ON_ONCE(cpuc->enabled); + static_call(x86_pmu_drain_pebs)(NULL, &data); } @@ -891,8 +905,8 @@ struct event_constraint intel_glm_pebs_event_constraints[] = { struct event_constraint intel_grt_pebs_event_constraints[] = { /* Allow all events as PEBS with no flags */ - INTEL_HYBRID_LAT_CONSTRAINT(0x5d0, 0x3), - INTEL_HYBRID_LAT_CONSTRAINT(0x6d0, 0xf), + INTEL_HYBRID_LDLAT_CONSTRAINT(0x5d0, 0x3), + INTEL_HYBRID_STLAT_CONSTRAINT(0x6d0, 0x3f), EVENT_CONSTRAINT_END }; @@ -1327,8 +1341,11 @@ static void intel_pmu_pebs_via_pt_enable(struct perf_event *event) static inline void intel_pmu_drain_large_pebs(struct cpu_hw_events *cpuc) { if (cpuc->n_pebs == cpuc->n_large_pebs && - cpuc->n_pebs != cpuc->n_pebs_via_pt) + cpuc->n_pebs != cpuc->n_pebs_via_pt) { + int enabled = __intel_pmu_quiesce(); intel_pmu_drain_pebs_buffer(); + __intel_pmu_resume(enabled); + } } void intel_pmu_pebs_enable(struct perf_event *event) diff --git a/arch/x86/events/intel/lbr.c b/arch/x86/events/intel/lbr.c index 3e904ba94278..dc4e0804b900 100644 --- a/arch/x86/events/intel/lbr.c +++ b/arch/x86/events/intel/lbr.c @@ -538,7 +538,8 @@ void intel_pmu_lbr_swap_task_ctx(struct perf_event_pmu_context *prev_epc, task_context_opt(next_ctx_data)->lbr_callstack_users); } -void intel_pmu_lbr_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) +void intel_pmu_lbr_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in) { struct cpu_hw_events *cpuc = this_cpu_ptr(&cpu_hw_events); void *task_ctx; diff --git a/arch/x86/events/intel/pt.c b/arch/x86/events/intel/pt.c index 7ee8dc80a359..57ed4e052167 100644 --- a/arch/x86/events/intel/pt.c +++ b/arch/x86/events/intel/pt.c @@ -1506,12 +1506,14 @@ void intel_pt_interrupt(void) perf_aux_output_end(&pt->handle, local_xchg(&buf->data_size, 0)); - if (!event->hw.state) { + event->hw.state |= PERF_HES_UPTODATE; + + if (!(event->hw.state & PERF_HES_STOPPED)) { int ret; buf = perf_aux_output_begin(&pt->handle, event); if (!buf) { - event->hw.state = PERF_HES_STOPPED; + event->hw.state |= PERF_HES_STOPPED; return; } @@ -1525,6 +1527,8 @@ void intel_pt_interrupt(void) pt_config_buffer(buf); pt_config_start(event); + + event->hw.state &= ~PERF_HES_UPTODATE; } } @@ -1574,6 +1578,19 @@ static void pt_event_start(struct perf_event *event, int mode) struct pt *pt = this_cpu_ptr(&pt_ctx); struct pt_buffer *buf; + /* + * Re-start subsequent to a call to pt_event_stop() without the + * PERF_EF_UPDATE flag. Absence of PERF_HES_UPTODATE indicates that + * perf_aux_output_begin() has already been called. This path can + * come about only in snapshot/overwrite mode - see pt_event_stop(). + */ + if (!(hwc->state & PERF_HES_UPTODATE)) { + hwc->state &= ~PERF_HES_STOPPED; + WRITE_ONCE(pt->handle_nmi, 1); + pt_config_start(event); + return; + } + buf = perf_aux_output_begin(&pt->handle, event); if (!buf) goto fail_stop; @@ -1585,7 +1602,7 @@ static void pt_event_start(struct perf_event *event, int mode) } WRITE_ONCE(pt->handle_nmi, 1); - hwc->state = 0; + hwc->state &= ~(PERF_HES_STOPPED | PERF_HES_UPTODATE); pt_config_buffer(buf); pt_config(event); @@ -1595,12 +1612,13 @@ static void pt_event_start(struct perf_event *event, int mode) fail_end_stop: perf_aux_output_end(&pt->handle, 0); fail_stop: - hwc->state = PERF_HES_STOPPED; + hwc->state |= PERF_HES_STOPPED | PERF_HES_UPTODATE; } static void pt_event_stop(struct perf_event *event, int mode) { struct pt *pt = this_cpu_ptr(&pt_ctx); + struct pt_buffer *buf; /* * Protect against the PMI racing with disabling wrmsr, @@ -1611,17 +1629,24 @@ static void pt_event_stop(struct perf_event *event, int mode) pt_config_stop(event); - if (event->hw.state == PERF_HES_STOPPED) - return; + event->hw.state |= PERF_HES_STOPPED; - event->hw.state = PERF_HES_STOPPED; - - if (mode & PERF_EF_UPDATE) { - struct pt_buffer *buf = perf_get_aux(&pt->handle); + if (event->hw.state & PERF_HES_UPTODATE) + return; - if (!buf) - return; + buf = perf_get_aux(&pt->handle); + if (!buf) + return; + /* + * When not in snapshot/overwrite mode, there is a possibility that the + * buffer has run out of space. The accounting for that is handled by + * the update, so always update in that case. Snapshot/overwrite mode is + * treated differently to allow for pt_event_snapshot_aux() which can + * still get called if the AUX-sampling event is not stopped until after + * PT is stopped. + */ + if ((mode & PERF_EF_UPDATE) || !buf->snapshot) { if (WARN_ON_ONCE(pt->handle.event != event)) return; @@ -1636,6 +1661,7 @@ static void pt_event_stop(struct perf_event *event, int mode) local_xchg(&buf->data_size, buf->nr_pages << PAGE_SHIFT); perf_aux_output_end(&pt->handle, local_xchg(&buf->data_size, 0)); + event->hw.state |= PERF_HES_UPTODATE; } } @@ -1697,13 +1723,15 @@ static int pt_event_add(struct perf_event *event, int mode) if (pt->handle.event) goto fail; + event->hw.state |= PERF_HES_UPTODATE; + if (mode & PERF_EF_START) { pt_event_start(event, 0); ret = -EINVAL; - if (hwc->state == PERF_HES_STOPPED) + if (hwc->state & PERF_HES_STOPPED) goto fail; } else { - hwc->state = PERF_HES_STOPPED; + hwc->state |= PERF_HES_STOPPED; } ret = 0; diff --git a/arch/x86/events/perf_event.h b/arch/x86/events/perf_event.h index 4ca15be45cb8..73fe3c65f5f5 100644 --- a/arch/x86/events/perf_event.h +++ b/arch/x86/events/perf_event.h @@ -470,6 +470,14 @@ struct cpu_hw_events { __EVENT_CONSTRAINT(c, n, INTEL_ARCH_EVENT_MASK|X86_ALL_EVENT_FLAGS, \ HWEIGHT(n), 0, PERF_X86_EVENT_PEBS_LAT_HYBRID) +#define INTEL_HYBRID_LDLAT_CONSTRAINT(c, n) \ + __EVENT_CONSTRAINT(c, n, INTEL_ARCH_EVENT_MASK|X86_ALL_EVENT_FLAGS, \ + HWEIGHT(n), 0, PERF_X86_EVENT_PEBS_LAT_HYBRID|PERF_X86_EVENT_PEBS_LD_HSW) + +#define INTEL_HYBRID_STLAT_CONSTRAINT(c, n) \ + __EVENT_CONSTRAINT(c, n, INTEL_ARCH_EVENT_MASK|X86_ALL_EVENT_FLAGS, \ + HWEIGHT(n), 0, PERF_X86_EVENT_PEBS_LAT_HYBRID|PERF_X86_EVENT_PEBS_ST_HSW) + /* Event constraint, but match on all event flags too. */ #define INTEL_FLAGS_EVENT_CONSTRAINT(c, n) \ EVENT_CONSTRAINT(c, n, ARCH_PERFMON_EVENTSEL_EVENT|X86_ALL_EVENT_FLAGS) @@ -609,7 +617,7 @@ union perf_capabilities { u64 perf_metrics:1; u64 pebs_output_pt_available:1; u64 pebs_timing_info:1; - u64 anythread_deprecated:1; + u64 __reserved:1; }; u64 capabilities; }; @@ -816,7 +824,7 @@ struct x86_pmu { void (*check_microcode)(void); void (*sched_task)(struct perf_event_pmu_context *pmu_ctx, - bool sched_in); + struct task_struct *task, bool sched_in); /* * Intel Arch Perfmon v2+ @@ -1310,7 +1318,8 @@ void amd_pmu_lbr_reset(void); void amd_pmu_lbr_read(void); void amd_pmu_lbr_add(struct perf_event *event); void amd_pmu_lbr_del(struct perf_event *event); -void amd_pmu_lbr_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in); +void amd_pmu_lbr_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in); void amd_pmu_lbr_enable_all(void); void amd_pmu_lbr_disable_all(void); int amd_pmu_lbr_hw_config(struct perf_event *event); @@ -1351,7 +1360,8 @@ static inline void amd_pmu_brs_del(struct perf_event *event) perf_sched_cb_dec(event->pmu); } -void amd_pmu_brs_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in); +void amd_pmu_brs_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in); #else static inline int amd_brs_init(void) { @@ -1376,7 +1386,8 @@ static inline void amd_pmu_brs_del(struct perf_event *event) { } -static inline void amd_pmu_brs_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in) +static inline void amd_pmu_brs_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in) { } @@ -1467,6 +1478,9 @@ static __always_inline void __intel_pmu_lbr_disable(void) wrmsrl(MSR_IA32_DEBUGCTLMSR, debugctl); } +extern int __intel_pmu_quiesce(void); +extern void __intel_pmu_resume(int pmu_enabled); + int intel_pmu_save_and_restart(struct perf_event *event); struct event_constraint * @@ -1499,7 +1513,7 @@ void intel_pmu_disable_bts(void); int intel_pmu_drain_bts_buffer(void); -u64 adl_latency_data_small(struct perf_event *event, u64 status); +u64 grt_latency_data(struct perf_event *event, u64 status); u64 mtl_latency_data_small(struct perf_event *event, u64 status); @@ -1558,7 +1572,8 @@ void intel_ds_init(void); void intel_pmu_lbr_swap_task_ctx(struct perf_event_pmu_context *prev_epc, struct perf_event_pmu_context *next_epc); -void intel_pmu_lbr_sched_task(struct perf_event_pmu_context *pmu_ctx, bool sched_in); +void intel_pmu_lbr_sched_task(struct perf_event_pmu_context *pmu_ctx, + struct task_struct *task, bool sched_in); u64 lbr_from_signext_quirk_wr(u64 val); diff --git a/arch/x86/include/asm/uaccess_64.h b/arch/x86/include/asm/uaccess_64.h index f2c02e4469cc..68417f5db5dc 100644 --- a/arch/x86/include/asm/uaccess_64.h +++ b/arch/x86/include/asm/uaccess_64.h @@ -133,7 +133,8 @@ raw_copy_to_user(void __user *dst, const void *src, unsigned long size) return copy_user_generic((__force void *)dst, src, size); } -extern long __copy_user_nocache(void *dst, const void __user *src, unsigned size); +#define copy_to_nontemporal copy_to_nontemporal +extern size_t copy_to_nontemporal(void *dst, const void *src, size_t size); extern long __copy_user_flushcache(void *dst, const void __user *src, unsigned size); static inline int @@ -143,7 +144,7 @@ __copy_from_user_inatomic_nocache(void *dst, const void __user *src, long ret; kasan_check_write(dst, size); stac(); - ret = __copy_user_nocache(dst, src, size); + ret = copy_to_nontemporal(dst, (__force const void *)src, size); clac(); return ret; } diff --git a/arch/x86/kernel/cpu/amd.c b/arch/x86/kernel/cpu/amd.c index a2313fa8a072..955da4c2385f 100644 --- a/arch/x86/kernel/cpu/amd.c +++ b/arch/x86/kernel/cpu/amd.c @@ -1196,6 +1196,17 @@ static void init_amd_zen3(struct cpuinfo_x86 *c) static void init_amd_zen4(struct cpuinfo_x86 *c) { init_amd_zen_common(); + + /* + * These Zen4 SoCs advertise support for virtualized VMLOAD/VMSAVE + * in some BIOS versions but they can lead to random host reboots. + */ + switch (c->x86_model) { + case 0x18 ... 0x1f: + case 0x60 ... 0x7f: + clear_cpu_cap(c, X86_FEATURE_V_VMSAVE_VMLOAD); + break; + } } static void init_amd_zen5(struct cpuinfo_x86 *c) diff --git a/arch/x86/kernel/cpu/resctrl/rdtgroup.c b/arch/x86/kernel/cpu/resctrl/rdtgroup.c index 1bc8b98e175c..02394f49a7c9 100644 --- a/arch/x86/kernel/cpu/resctrl/rdtgroup.c +++ b/arch/x86/kernel/cpu/resctrl/rdtgroup.c @@ -2315,15 +2315,24 @@ static void rdtgroup_kn_get(struct rdtgroup *rdtgrp, struct kernfs_node *kn) static void rdtgroup_kn_put(struct rdtgroup *rdtgrp, struct kernfs_node *kn) { - if (atomic_dec_and_test(&rdtgrp->waitcount) && - (rdtgrp->flags & RDT_DELETED)) { + bool needs_free; + + if (!atomic_dec_and_mutex_lock(&rdtgrp->waitcount, &rdtgroup_mutex)) { + kernfs_unbreak_active_protection(kn); + return; + } + + needs_free = rdtgrp->flags & RDT_DELETED; + + mutex_unlock(&rdtgroup_mutex); + + kernfs_unbreak_active_protection(kn); + + if (needs_free) { if (rdtgrp->mode == RDT_MODE_PSEUDO_LOCKSETUP || rdtgrp->mode == RDT_MODE_PSEUDO_LOCKED) rdtgroup_pseudo_lock_remove(rdtgrp); - kernfs_unbreak_active_protection(kn); rdtgroup_remove(rdtgrp); - } else { - kernfs_unbreak_active_protection(kn); } } diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index d29cb592e703..09902887ab38 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -2410,7 +2410,7 @@ static u16 kvm_hvcall_signal_event(struct kvm_vcpu *vcpu, struct kvm_hv_hcall *h if (!eventfd) return HV_STATUS_INVALID_PORT_ID; - eventfd_signal(eventfd, 1); + eventfd_signal(eventfd); return HV_STATUS_SUCCESS; } diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index 50a8c5701084..b243cb405808 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2358,6 +2358,12 @@ static void shadow_walk_init_using_root(struct kvm_shadow_walk_iterator *iterato { iterator->addr = addr; iterator->shadow_addr = root; + + if (WARN_ON_ONCE(!VALID_PAGE(root)) || kvm_mmu_is_dummy_root(root)) { + iterator->level = 0; + return; + } + iterator->level = vcpu->arch.mmu->root_role.level; if (iterator->level >= PT64_ROOT_4LEVEL && @@ -3129,7 +3135,7 @@ static int host_pfn_mapping_level(struct kvm *kvm, gfn_t gfn, if (pmd_none(pmd) || !pmd_present(pmd)) goto out; - if (pmd_large(pmd)) + if (pmd_leaf(pmd)) level = PG_LEVEL_2M; out: diff --git a/arch/x86/kvm/svm/hyperv.h b/arch/x86/kvm/svm/hyperv.h index 02f4784b5d44..1970f09fce9f 100644 --- a/arch/x86/kvm/svm/hyperv.h +++ b/arch/x86/kvm/svm/hyperv.h @@ -40,6 +40,13 @@ static inline bool nested_svm_l2_tlb_flush_enabled(struct kvm_vcpu *vcpu) return hv_vcpu->vp_assist_page.nested_control.features.directhypercall; } +static inline bool nested_svm_is_l2_tlb_flush_hcall(struct kvm_vcpu *vcpu) +{ + return guest_hv_cpuid_has_l2_tlb_flush(vcpu) && + nested_svm_l2_tlb_flush_enabled(vcpu) && + kvm_hv_is_tlb_flush_hcall(vcpu); +} + void svm_hv_inject_synthetic_vmexit_post_tlb_flush(struct kvm_vcpu *vcpu); #endif /* __ARCH_X86_KVM_SVM_HYPERV_H__ */ diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index 2be600876e0c..2e4db6602db1 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -626,6 +626,7 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm, struct vmcb *vmcb12 } else { svm_copy_lbrs(vmcb02, vmcb01); } + vmcb_mark_dirty(vmcb02, VMCB_LBR); svm_update_lbrv(&svm->vcpu); } @@ -729,24 +730,32 @@ static void nested_vmcb02_prepare_control(struct vcpu_svm *svm, vmcb02->control.event_inj_err = svm->nested.ctl.event_inj_err; /* - * next_rip is consumed on VMRUN as the return address pushed on the + * NextRIP is consumed on VMRUN as the return address pushed on the * stack for injected soft exceptions/interrupts. If nrips is exposed - * to L1, take it verbatim from vmcb12. If nrips is supported in - * hardware but not exposed to L1, stuff the actual L2 RIP to emulate - * what a nrips=0 CPU would do (L1 is responsible for advancing RIP - * prior to injecting the event). + * to L1, take it verbatim from vmcb12. + * + * If nrips is supported in hardware but not exposed to L1, stuff the + * actual L2 RIP to emulate what a nrips=0 CPU would do (L1 is + * responsible for advancing RIP prior to injecting the event). This is + * only the case for the first L2 run after VMRUN. After that (e.g. + * during save/restore), NextRIP is updated by the CPU and/or KVM, and + * the value of the L2 RIP from vmcb12 should not be used. */ - if (guest_can_use(vcpu, X86_FEATURE_NRIPS)) - vmcb02->control.next_rip = svm->nested.ctl.next_rip; - else if (boot_cpu_has(X86_FEATURE_NRIPS)) - vmcb02->control.next_rip = vmcb12_rip; + if (boot_cpu_has(X86_FEATURE_NRIPS)) { + if (guest_can_use(vcpu, X86_FEATURE_NRIPS) || + !svm->nested.nested_run_pending) + vmcb02->control.next_rip = svm->nested.ctl.next_rip; + else + vmcb02->control.next_rip = vmcb12_rip; + } svm->nmi_l1_to_l2 = is_evtinj_nmi(vmcb02->control.event_inj); if (is_evtinj_soft(vmcb02->control.event_inj)) { svm->soft_int_injected = true; svm->soft_int_csbase = vmcb12_csbase; svm->soft_int_old_rip = vmcb12_rip; - if (guest_can_use(vcpu, X86_FEATURE_NRIPS)) + if (guest_can_use(vcpu, X86_FEATURE_NRIPS) || + !svm->nested.nested_run_pending) svm->soft_int_next_rip = svm->nested.ctl.next_rip; else svm->soft_int_next_rip = vmcb12_rip; @@ -964,6 +973,15 @@ void svm_copy_vmrun_state(struct vmcb_save_area *to_save, to_save->rsp = from_save->rsp; to_save->rip = from_save->rip; to_save->cpl = 0; + + if (kvm_cpu_cap_has(X86_FEATURE_LBRV)) { + to_save->dbgctl = from_save->dbgctl; + to_save->br_from = from_save->br_from; + to_save->br_to = from_save->br_to; + to_save->last_excp_from = from_save->last_excp_from; + to_save->last_excp_to = from_save->last_excp_to; + to_save->dbgctl &= ~DEBUGCTL_RESERVED_BITS; + } } void svm_copy_vmloadsave_state(struct vmcb *to_vmcb, struct vmcb *from_vmcb) @@ -1080,10 +1098,12 @@ int nested_svm_vmexit(struct vcpu_svm *svm) kvm_make_request(KVM_REQ_EVENT, &svm->vcpu); if (unlikely(guest_can_use(vcpu, X86_FEATURE_LBRV) && - (svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK))) + (svm->nested.ctl.virt_ext & LBR_CTL_ENABLE_MASK))) { svm_copy_lbrs(vmcb12, vmcb02); - else + } else { svm_copy_lbrs(vmcb01, vmcb02); + vmcb_mark_dirty(vmcb01, VMCB_LBR); + } svm_update_lbrv(vcpu); @@ -1149,9 +1169,8 @@ int nested_svm_vmexit(struct vcpu_svm *svm) nested_svm_uninit_mmu_context(vcpu); - rc = nested_svm_load_cr3(vcpu, vmcb01->save.cr3, false, true); - if (rc) - return 1; + if (nested_svm_load_cr3(vcpu, vmcb01->save.cr3, false, true)) + kvm_make_request(KVM_REQ_TRIPLE_FAULT, vcpu); /* * Drop what we picked up for L2 via svm_complete_interrupts() so it @@ -1538,9 +1557,7 @@ int nested_svm_exit_special(struct vcpu_svm *svm) } case SVM_EXIT_VMMCALL: /* Hyper-V L2 TLB flush hypercall is handled by L0 */ - if (guest_hv_cpuid_has_l2_tlb_flush(vcpu) && - nested_svm_l2_tlb_flush_enabled(vcpu) && - kvm_hv_is_tlb_flush_hcall(vcpu)) + if (nested_svm_is_l2_tlb_flush_hcall(vcpu)) return NESTED_EXIT_HOST; break; default: diff --git a/arch/x86/kvm/svm/sev.c b/arch/x86/kvm/svm/sev.c index 0ba286bd50c2..75bd73f205ab 100644 --- a/arch/x86/kvm/svm/sev.c +++ b/arch/x86/kvm/svm/sev.c @@ -401,7 +401,7 @@ static int sev_launch_start(struct kvm *kvm, struct kvm_sev_cmd *argp) static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, unsigned long ulen, unsigned long *n, - int write) + unsigned int flags) { struct kvm_sev_info *sev = &to_kvm_svm(kvm)->sev_info; unsigned long npages, size; @@ -445,7 +445,7 @@ static struct page **sev_pin_memory(struct kvm *kvm, unsigned long uaddr, return ERR_PTR(-ENOMEM); /* Pin the user virtual address. */ - npinned = pin_user_pages_fast(uaddr, npages, write ? FOLL_WRITE : 0, pages); + npinned = pin_user_pages_fast(uaddr, npages, flags, pages); if (npinned != npages) { pr_err("SEV: Failure locking %lu pages.\n", npages); ret = -ENOMEM; @@ -533,7 +533,7 @@ static int sev_launch_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) vaddr_end = vaddr + size; /* Lock the user memory. */ - inpages = sev_pin_memory(kvm, vaddr, size, &npages, 1); + inpages = sev_pin_memory(kvm, vaddr, size, &npages, FOLL_WRITE); if (IS_ERR(inpages)) return PTR_ERR(inpages); @@ -985,7 +985,7 @@ static int sev_dbg_crypt(struct kvm *kvm, struct kvm_sev_cmd *argp, bool dec) if (IS_ERR(src_p)) return PTR_ERR(src_p); - dst_p = sev_pin_memory(kvm, dst_vaddr & PAGE_MASK, PAGE_SIZE, &n, 1); + dst_p = sev_pin_memory(kvm, dst_vaddr & PAGE_MASK, PAGE_SIZE, &n, FOLL_WRITE); if (IS_ERR(dst_p)) { sev_unpin_memory(kvm, src_p, n); return PTR_ERR(dst_p); @@ -1052,7 +1052,7 @@ static int sev_launch_secret(struct kvm *kvm, struct kvm_sev_cmd *argp) if (copy_from_user(¶ms, (void __user *)(uintptr_t)argp->data, sizeof(params))) return -EFAULT; - pages = sev_pin_memory(kvm, params.guest_uaddr, params.guest_len, &n, 1); + pages = sev_pin_memory(kvm, params.guest_uaddr, params.guest_len, &n, FOLL_WRITE); if (IS_ERR(pages)) return PTR_ERR(pages); @@ -1534,7 +1534,7 @@ static int sev_receive_update_data(struct kvm *kvm, struct kvm_sev_cmd *argp) /* Pin guest memory */ guest_page = sev_pin_memory(kvm, params.guest_uaddr & PAGE_MASK, - PAGE_SIZE, &n, 1); + PAGE_SIZE, &n, FOLL_WRITE); if (IS_ERR(guest_page)) { ret = PTR_ERR(guest_page); goto e_free_trans; @@ -1984,6 +1984,8 @@ int sev_mem_enc_register_region(struct kvm *kvm, struct enc_region *region; int ret = 0; + guard(mutex)(&kvm->lock); + if (!sev_guest(kvm)) return -ENOTTY; @@ -1998,11 +2000,10 @@ int sev_mem_enc_register_region(struct kvm *kvm, if (!region) return -ENOMEM; - mutex_lock(&kvm->lock); - region->pages = sev_pin_memory(kvm, range->addr, range->size, ®ion->npages, 1); + region->pages = sev_pin_memory(kvm, range->addr, range->size, ®ion->npages, + FOLL_WRITE | FOLL_LONGTERM); if (IS_ERR(region->pages)) { ret = PTR_ERR(region->pages); - mutex_unlock(&kvm->lock); goto e_free; } @@ -2020,8 +2021,6 @@ int sev_mem_enc_register_region(struct kvm *kvm, region->size = range->size; list_add_tail(®ion->list, &sev->regions_list); - mutex_unlock(&kvm->lock); - return ret; e_free: @@ -2988,6 +2987,11 @@ int sev_handle_vmgexit(struct kvm_vcpu *vcpu) control->exit_info_1, control->exit_info_2); ret = -EINVAL; break; + case SVM_EXIT_IOIO: + if (!((control->exit_info_1 & SVM_IOIO_SIZE_MASK) >> SVM_IOIO_SIZE_SHIFT)) + return 1; + + fallthrough; default: ret = svm_invoke_exit_handler(vcpu, exit_code); } @@ -3008,6 +3012,9 @@ int sev_es_string_io(struct vcpu_svm *svm, int size, unsigned int port, int in) if (unlikely(check_mul_overflow(count, size, &bytes))) return -EINVAL; + if (!bytes) + return 1; + r = setup_vmgexit_scratch(svm, in, bytes); if (r) return r; diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index 62c7e5753021..832886bec53d 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -49,6 +49,7 @@ #include "svm.h" #include "svm_ops.h" +#include "hyperv.h" #include "kvm_onhyperv.h" #include "svm_onhyperv.h" @@ -825,17 +826,7 @@ static bool msr_write_intercepted(struct kvm_vcpu *vcpu, u32 msr) u32 offset; u32 *msrpm; - /* - * For non-nested case: - * If the L01 MSR bitmap does not intercept the MSR, then we need to - * save it. - * - * For nested case: - * If the L02 MSR bitmap does not intercept the MSR, then we need to - * save it. - */ - msrpm = is_guest_mode(vcpu) ? to_svm(vcpu)->nested.msrpm: - to_svm(vcpu)->msrpm; + msrpm = __va(__sme_clr(to_svm(vcpu)->vmcb->control.msrpm_base_pa)); offset = svm_msrpm_offset(msr); bit_write = 2 * (msr & 0x0f) + 1; @@ -1013,8 +1004,6 @@ void svm_copy_lbrs(struct vmcb *to_vmcb, struct vmcb *from_vmcb) to_vmcb->save.br_to = from_vmcb->save.br_to; to_vmcb->save.last_excp_from = from_vmcb->save.last_excp_from; to_vmcb->save.last_excp_to = from_vmcb->save.last_excp_to; - - vmcb_mark_dirty(to_vmcb, VMCB_LBR); } static void svm_recalc_lbr_msr_intercepts(struct kvm_vcpu *vcpu) @@ -2914,19 +2903,19 @@ static int svm_get_msr(struct kvm_vcpu *vcpu, struct msr_data *msr_info) msr_info->data = svm->tsc_aux; break; case MSR_IA32_DEBUGCTLMSR: - msr_info->data = svm->vmcb->save.dbgctl; + msr_info->data = lbrv ? svm->vmcb->save.dbgctl : 0; break; case MSR_IA32_LASTBRANCHFROMIP: - msr_info->data = svm->vmcb->save.br_from; + msr_info->data = lbrv ? svm->vmcb->save.br_from : 0; break; case MSR_IA32_LASTBRANCHTOIP: - msr_info->data = svm->vmcb->save.br_to; + msr_info->data = lbrv ? svm->vmcb->save.br_to : 0; break; case MSR_IA32_LASTINTFROMIP: - msr_info->data = svm->vmcb->save.last_excp_from; + msr_info->data = lbrv ? svm->vmcb->save.last_excp_from : 0; break; case MSR_IA32_LASTINTTOIP: - msr_info->data = svm->vmcb->save.last_excp_to; + msr_info->data = lbrv ? svm->vmcb->save.last_excp_to : 0; break; case MSR_VM_HSAVE_PA: msr_info->data = svm->nested.hsave_msr; @@ -3202,6 +3191,38 @@ static int svm_set_msr(struct kvm_vcpu *vcpu, struct msr_data *msr) vmcb_mark_dirty(svm->vmcb, VMCB_LBR); svm_update_lbrv(vcpu); break; + case MSR_IA32_LASTBRANCHFROMIP: + if (!lbrv) + return KVM_MSR_RET_UNSUPPORTED; + if (!msr->host_initiated) + return 1; + svm->vmcb->save.br_from = data; + vmcb_mark_dirty(svm->vmcb, VMCB_LBR); + break; + case MSR_IA32_LASTBRANCHTOIP: + if (!lbrv) + return KVM_MSR_RET_UNSUPPORTED; + if (!msr->host_initiated) + return 1; + svm->vmcb->save.br_to = data; + vmcb_mark_dirty(svm->vmcb, VMCB_LBR); + break; + case MSR_IA32_LASTINTFROMIP: + if (!lbrv) + return KVM_MSR_RET_UNSUPPORTED; + if (!msr->host_initiated) + return 1; + svm->vmcb->save.last_excp_from = data; + vmcb_mark_dirty(svm->vmcb, VMCB_LBR); + break; + case MSR_IA32_LASTINTTOIP: + if (!lbrv) + return KVM_MSR_RET_UNSUPPORTED; + if (!msr->host_initiated) + return 1; + svm->vmcb->save.last_excp_to = data; + vmcb_mark_dirty(svm->vmcb, VMCB_LBR); + break; case MSR_VM_HSAVE_PA: /* * Old kernels did not validate the value written to @@ -3312,6 +3333,22 @@ static int invpcid_interception(struct kvm_vcpu *vcpu) return kvm_handle_invpcid(vcpu, type, gva); } +static int vmmcall_interception(struct kvm_vcpu *vcpu) +{ + /* + * Inject a #UD if L2 is active and the VMMCALL isn't a Hyper-V TLB + * hypercall, as VMMCALL #UDs if it's not intercepted, and this path is + * reachable if and only if L1 doesn't want to intercept VMMCALL or has + * enabled L0 (KVM) handling of Hyper-V L2 TLB flush hypercalls. + */ + if (is_guest_mode(vcpu) && !nested_svm_is_l2_tlb_flush_hcall(vcpu)) { + kvm_queue_exception(vcpu, UD_VECTOR); + return 1; + } + + return kvm_emulate_hypercall(vcpu); +} + static int (*const svm_exit_handlers[])(struct kvm_vcpu *vcpu) = { [SVM_EXIT_READ_CR0] = cr_interception, [SVM_EXIT_READ_CR3] = cr_interception, @@ -3362,7 +3399,7 @@ static int (*const svm_exit_handlers[])(struct kvm_vcpu *vcpu) = { [SVM_EXIT_TASK_SWITCH] = task_switch_interception, [SVM_EXIT_SHUTDOWN] = shutdown_interception, [SVM_EXIT_VMRUN] = vmrun_interception, - [SVM_EXIT_VMMCALL] = kvm_emulate_hypercall, + [SVM_EXIT_VMMCALL] = vmmcall_interception, [SVM_EXIT_VMLOAD] = vmload_interception, [SVM_EXIT_VMSAVE] = vmsave_interception, [SVM_EXIT_STGI] = stgi_interception, diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index a130e2d70c48..2d9cfdfa7806 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -3571,6 +3571,11 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, if (!from_vmentry) return NVMX_VMENTRY_VMEXIT; + kvm_vcpu_unmap(vcpu, &vmx->nested.apic_access_page_map, false); + kvm_vcpu_unmap(vcpu, &vmx->nested.virtual_apic_map, true); + kvm_vcpu_unmap(vcpu, &vmx->nested.pi_desc_map, true); + vmx->nested.pi_desc = NULL; + load_vmcs12_host_state(vcpu, vmcs12); vmcs12->vm_exit_reason = exit_reason.full; if (enable_shadow_vmcs || evmptr_is_valid(vmx->nested.hv_evmcs_vmptr)) @@ -5850,6 +5855,7 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) } operand; u16 vpid02; int r, gpr_index; + int cpu; if (!(vmx->nested.msrs.secondary_ctls_high & SECONDARY_EXEC_ENABLE_VPID) || @@ -5891,6 +5897,10 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) * and never explicitly flush vpid01. INVVPID targets a VPID, not a * VMCS, and so whether or not the current vmcs12 has VPID enabled is * irrelevant (and there may not be a loaded vmcs12). + * + * If vmcs02 was last loaded on a different pCPU, then defer the flush + * by invalidating the nested VPID tracking to ensure that KVM performs + * the invalidation on the correct pCPU. */ vpid02 = nested_get_vpid02(vcpu); switch (type) { @@ -5899,17 +5909,32 @@ static int handle_invvpid(struct kvm_vcpu *vcpu) is_noncanonical_address(operand.gla, vcpu)) return nested_vmx_fail(vcpu, VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); - vpid_sync_vcpu_addr(vpid02, operand.gla); + cpu = get_cpu(); + if (cpu != vmx->nested.vmcs02.cpu) + vmx->nested.last_vpid = 0; + else + vpid_sync_vcpu_addr(vpid02, operand.gla); + put_cpu(); break; case VMX_VPID_EXTENT_SINGLE_CONTEXT: case VMX_VPID_EXTENT_SINGLE_NON_GLOBAL: if (!operand.vpid) return nested_vmx_fail(vcpu, VMXERR_INVALID_OPERAND_TO_INVEPT_INVVPID); - vpid_sync_context(vpid02); + cpu = get_cpu(); + if (cpu != vmx->nested.vmcs02.cpu) + vmx->nested.last_vpid = 0; + else + vpid_sync_context(vpid02); + put_cpu(); break; case VMX_VPID_EXTENT_ALL_CONTEXT: - vpid_sync_context(vpid02); + cpu = get_cpu(); + if (cpu != vmx->nested.vmcs02.cpu) + vmx->nested.last_vpid = 0; + else + vpid_sync_context(vpid02); + put_cpu(); break; default: WARN_ON_ONCE(1); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index 313fce77e058..cb659d04d07e 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -1473,6 +1473,9 @@ static const u32 msrs_to_save_base[] = { MSR_IA32_UMWAIT_CONTROL, MSR_IA32_XFD, MSR_IA32_XFD_ERR, + MSR_IA32_DEBUGCTLMSR, + MSR_IA32_LASTBRANCHFROMIP, MSR_IA32_LASTBRANCHTOIP, + MSR_IA32_LASTINTFROMIP, MSR_IA32_LASTINTTOIP, }; static const u32 msrs_to_save_pmu[] = { @@ -1914,6 +1917,16 @@ static int kvm_set_msr_ignored_check(struct kvm_vcpu *vcpu, { int ret = __kvm_set_msr(vcpu, index, data, host_initiated); + if (ret == KVM_MSR_RET_UNSUPPORTED && host_initiated && !data) { + switch (index) { + case MSR_IA32_LASTBRANCHFROMIP: + case MSR_IA32_LASTBRANCHTOIP: + case MSR_IA32_LASTINTFROMIP: + case MSR_IA32_LASTINTTOIP: + return 0; + } + } + if (ret == KVM_MSR_RET_UNSUPPORTED) if (kvm_msr_ignored_check(index, data, true)) ret = 0; diff --git a/arch/x86/kvm/x86.h b/arch/x86/kvm/x86.h index 1222e5b3d558..35680fd07577 100644 --- a/arch/x86/kvm/x86.h +++ b/arch/x86/kvm/x86.h @@ -218,7 +218,7 @@ static inline bool is_noncanonical_address(u64 la, struct kvm_vcpu *vcpu) static inline void vcpu_cache_mmio_info(struct kvm_vcpu *vcpu, gva_t gva, gfn_t gfn, unsigned access) { - u64 gen = kvm_memslots(vcpu->kvm)->generation; + u64 gen = kvm_vcpu_memslots(vcpu)->generation; if (unlikely(gen & KVM_MEMSLOT_GEN_UPDATE_IN_PROGRESS)) return; @@ -235,7 +235,7 @@ static inline void vcpu_cache_mmio_info(struct kvm_vcpu *vcpu, static inline bool vcpu_match_mmio_gen(struct kvm_vcpu *vcpu) { - return vcpu->arch.mmio_gen == kvm_memslots(vcpu->kvm)->generation; + return vcpu->arch.mmio_gen == kvm_vcpu_memslots(vcpu)->generation; } /* diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index 94c7fee7a225..c5373a42be7f 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -2054,7 +2054,7 @@ static bool kvm_xen_hcall_evtchn_send(struct kvm_vcpu *vcpu, u64 param, u64 *r) if (ret < 0 && ret != -ENOTCONN) return false; } else { - eventfd_signal(evtchnfd->deliver.eventfd.ctx, 1); + eventfd_signal(evtchnfd->deliver.eventfd.ctx); } *r = 0; diff --git a/arch/x86/lib/copy_user_uncached_64.S b/arch/x86/lib/copy_user_uncached_64.S index 5c5f38d32672..b37a6f084c4c 100644 --- a/arch/x86/lib/copy_user_uncached_64.S +++ b/arch/x86/lib/copy_user_uncached_64.S @@ -26,7 +26,7 @@ * Output: * rax uncopied bytes or 0 if successful. */ -SYM_FUNC_START(__copy_user_nocache) +SYM_FUNC_START(copy_to_nontemporal) /* If destination is not 7-byte aligned, we'll have to align it */ testb $7,%dil jne .Lalign @@ -238,5 +238,5 @@ _ASM_EXTABLE_UA(95b, .Ldone) _ASM_EXTABLE_UA(52b, .Ldone0) _ASM_EXTABLE_UA(53b, .Ldone0) -SYM_FUNC_END(__copy_user_nocache) -EXPORT_SYMBOL(__copy_user_nocache) +SYM_FUNC_END(copy_to_nontemporal) +EXPORT_SYMBOL(copy_to_nontemporal) diff --git a/arch/x86/lib/usercopy_64.c b/arch/x86/lib/usercopy_64.c index e9251b89a9e9..1f1fb2ab1496 100644 --- a/arch/x86/lib/usercopy_64.c +++ b/arch/x86/lib/usercopy_64.c @@ -49,11 +49,11 @@ long __copy_user_flushcache(void *dst, const void __user *src, unsigned size) long rc; stac(); - rc = __copy_user_nocache(dst, src, size); + rc = copy_to_nontemporal(dst, (__force const void *)src, size); clac(); /* - * __copy_user_nocache() uses non-temporal stores for the bulk + * copy_to_nontemporal() uses non-temporal stores for the bulk * of the transfer, but we need to manually flush if the * transfer is unaligned. A cached memory copy is used when * destination or size is not naturally aligned. That is: diff --git a/arch/x86/mm/fault.c b/arch/x86/mm/fault.c index 4890cafea7b6..b620033fdcea 100644 --- a/arch/x86/mm/fault.c +++ b/arch/x86/mm/fault.c @@ -250,7 +250,7 @@ static noinline int vmalloc_fault(unsigned long address) if (!pmd_k) return -1; - if (pmd_large(*pmd_k)) + if (pmd_leaf(*pmd_k)) return 0; pte_k = pte_offset_kernel(pmd_k, address); @@ -319,7 +319,7 @@ static void dump_pagetable(unsigned long address) * And let's rather not kmap-atomic the pte, just in case * it's allocated already: */ - if (!low_pfn(pmd_pfn(*pmd)) || !pmd_present(*pmd) || pmd_large(*pmd)) + if (!low_pfn(pmd_pfn(*pmd)) || !pmd_present(*pmd) || pmd_leaf(*pmd)) goto out; pte = pte_offset_kernel(pmd, address); @@ -384,7 +384,7 @@ static void dump_pagetable(unsigned long address) goto bad; pr_cont("PMD %lx ", pmd_val(*pmd)); - if (!pmd_present(*pmd) || pmd_large(*pmd)) + if (!pmd_present(*pmd) || pmd_leaf(*pmd)) goto out; pte = pte_offset_kernel(pmd, address); @@ -1011,7 +1011,7 @@ spurious_kernel_fault(unsigned long error_code, unsigned long address) if (!pmd_present(*pmd)) return 0; - if (pmd_large(*pmd)) + if (pmd_leaf(*pmd)) return spurious_kernel_fault_check(error_code, (pte_t *) pmd); pte = pte_offset_kernel(pmd, address); diff --git a/arch/x86/mm/init_32.c b/arch/x86/mm/init_32.c index b63403d7179d..bb6785472f0c 100644 --- a/arch/x86/mm/init_32.c +++ b/arch/x86/mm/init_32.c @@ -463,7 +463,7 @@ void __init native_pagetable_init(void) break; /* should not be large page here */ - if (pmd_large(*pmd)) { + if (pmd_leaf(*pmd)) { pr_warn("try to clear pte for ram above max_low_pfn: pfn: %lx pmd: %p pmd phys: %lx, but pmd is big page and is not using pte !\n", pfn, pmd, __pa(pmd)); BUG_ON(1); diff --git a/arch/x86/mm/init_64.c b/arch/x86/mm/init_64.c index cf080c96b7dd..e49bf9b4370a 100644 --- a/arch/x86/mm/init_64.c +++ b/arch/x86/mm/init_64.c @@ -548,7 +548,7 @@ phys_pmd_init(pmd_t *pmd_page, unsigned long paddr, unsigned long paddr_end, } if (!pmd_none(*pmd)) { - if (!pmd_large(*pmd)) { + if (!pmd_leaf(*pmd)) { spin_lock(&init_mm.page_table_lock); pte = (pte_t *)pmd_page_vaddr(*pmd); paddr_last = phys_pte_init(pte, paddr, @@ -1145,7 +1145,7 @@ remove_pmd_table(pmd_t *pmd_start, unsigned long addr, unsigned long end, if (!pmd_present(*pmd)) continue; - if (pmd_large(*pmd)) { + if (pmd_leaf(*pmd)) { if (IS_ALIGNED(addr, PMD_SIZE) && IS_ALIGNED(next, PMD_SIZE)) { if (!direct) @@ -1553,9 +1553,9 @@ void __meminit vmemmap_set_pmd(pmd_t *pmd, void *p, int node, int __meminit vmemmap_check_pmd(pmd_t *pmd, int node, unsigned long addr, unsigned long next) { - int large = pmd_large(*pmd); + int large = pmd_leaf(*pmd); - if (pmd_large(*pmd)) { + if (pmd_leaf(*pmd)) { vmemmap_verify((pte_t *)pmd, node, addr, next); vmemmap_use_sub_pmd(addr, next); } diff --git a/arch/x86/mm/kasan_init_64.c b/arch/x86/mm/kasan_init_64.c index fcf508c52bdc..9dddf19a5571 100644 --- a/arch/x86/mm/kasan_init_64.c +++ b/arch/x86/mm/kasan_init_64.c @@ -95,7 +95,7 @@ static void __init kasan_populate_pud(pud_t *pud, unsigned long addr, pmd = pmd_offset(pud, addr); do { next = pmd_addr_end(addr, end); - if (!pmd_large(*pmd)) + if (!pmd_leaf(*pmd)) kasan_populate_pmd(pmd, addr, next, nid); } while (pmd++, addr = next, addr != end); } diff --git a/arch/x86/mm/mem_encrypt_identity.c b/arch/x86/mm/mem_encrypt_identity.c index 075899e3fc8a..b922b9fea6b6 100644 --- a/arch/x86/mm/mem_encrypt_identity.c +++ b/arch/x86/mm/mem_encrypt_identity.c @@ -157,7 +157,7 @@ static void __head sme_populate_pgd_large(struct sme_populate_pgd_data *ppd) return; pmd = pmd_offset(pud, ppd->vaddr); - if (pmd_large(*pmd)) + if (pmd_leaf(*pmd)) return; set_pmd(pmd, __pmd(ppd->paddr | ppd->pmd_flags)); @@ -181,7 +181,7 @@ static void __head sme_populate_pgd(struct sme_populate_pgd_data *ppd) set_pmd(pmd, __pmd(PMD_FLAGS | __pa(pte))); } - if (pmd_large(*pmd)) + if (pmd_leaf(*pmd)) return; pte = pte_offset_kernel(pmd, ppd->vaddr); diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 525794f1eefb..8838033aa450 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -709,7 +709,7 @@ pte_t *lookup_address_in_pgd_attr(pgd_t *pgd, unsigned long address, return NULL; *level = PG_LEVEL_2M; - if (pmd_large(*pmd) || !pmd_present(*pmd)) + if (pmd_leaf(*pmd) || !pmd_present(*pmd)) return (pte_t *)pmd; *nx |= pmd_flags(*pmd) & _PAGE_NX; @@ -1267,7 +1267,7 @@ static void unmap_pmd_range(pud_t *pud, unsigned long start, unsigned long end) * Try to unmap in 2M chunks. */ while (end - start >= PMD_SIZE) { - if (pmd_large(*pmd)) + if (pmd_leaf(*pmd)) pmd_clear(pmd); else __unmap_pmd_range(pud, pmd, start, start + PMD_SIZE); diff --git a/arch/x86/mm/pgtable.c b/arch/x86/mm/pgtable.c index 98aa303ad054..8bb739e1c2d0 100644 --- a/arch/x86/mm/pgtable.c +++ b/arch/x86/mm/pgtable.c @@ -791,7 +791,7 @@ int pud_clear_huge(pud_t *pud) */ int pmd_clear_huge(pmd_t *pmd) { - if (pmd_large(*pmd)) { + if (pmd_leaf(*pmd)) { pmd_clear(pmd); return 1; } diff --git a/arch/x86/mm/pti.c b/arch/x86/mm/pti.c index 916247c1d2a3..3dbd8b56f364 100644 --- a/arch/x86/mm/pti.c +++ b/arch/x86/mm/pti.c @@ -252,7 +252,7 @@ static pte_t *pti_user_pagetable_walk_pte(unsigned long address, bool late_text) return NULL; /* Large PMD mapping found */ - if (pmd_large(*pmd)) { + if (pmd_leaf(*pmd)) { /* Clear the PMD if we hit a large mapping from the first round */ if (late_text) { set_pmd(pmd, __pmd(0)); @@ -346,7 +346,7 @@ pti_clone_pgtable(unsigned long start, unsigned long end, continue; } - if (pmd_large(*pmd) || level == PTI_CLONE_PMD) { + if (pmd_leaf(*pmd) || level == PTI_CLONE_PMD) { target_pmd = pti_user_pagetable_walk_pmd(addr); if (WARN_ON(!target_pmd)) return; diff --git a/arch/x86/net/bpf_jit_comp.c b/arch/x86/net/bpf_jit_comp.c index 8028a5a4ab64..8fc949efb89f 100644 --- a/arch/x86/net/bpf_jit_comp.c +++ b/arch/x86/net/bpf_jit_comp.c @@ -2467,10 +2467,8 @@ int arch_prepare_bpf_trampoline(struct bpf_tramp_image *im, void *image, void *i u8 *prog; bool save_ret; - /* extra registers for struct arguments */ for (i = 0; i < m->nr_args; i++) - if (m->arg_flags[i] & BTF_FMODEL_STRUCT_ARG) - nr_regs += (m->arg_size[i] + 7) / 8 - 1; + nr_regs += (m->arg_size[i] + 7) / 8 - 1; /* x86-64 supports up to MAX_BPF_FUNC_ARGS arguments. 1-6 * are passed through regs, the remains are through stack. @@ -2951,7 +2949,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) * * Both cases are serious bugs and justify WARN_ON. */ - if (WARN_ON(bpf_jit_binary_pack_finalize(prog, header, rw_header))) { + if (WARN_ON(bpf_jit_binary_pack_finalize(header, rw_header))) { /* header has been freed */ header = NULL; goto out_image; @@ -3018,7 +3016,7 @@ void bpf_jit_free(struct bpf_prog *prog) * before freeing it. */ if (jit_data) { - bpf_jit_binary_pack_finalize(prog, jit_data->header, + bpf_jit_binary_pack_finalize(jit_data->header, jit_data->rw_header); kvfree(jit_data->addrs); kfree(jit_data); diff --git a/arch/x86/net/bpf_jit_comp32.c b/arch/x86/net/bpf_jit_comp32.c index 429a89c5468b..f2fc8c38629b 100644 --- a/arch/x86/net/bpf_jit_comp32.c +++ b/arch/x86/net/bpf_jit_comp32.c @@ -2600,8 +2600,7 @@ struct bpf_prog *bpf_int_jit_compile(struct bpf_prog *prog) if (bpf_jit_enable > 1) bpf_jit_dump(prog->len, proglen, pass + 1, image); - if (image) { - bpf_jit_binary_lock_ro(header); + if (image && !bpf_jit_binary_lock_ro(header)) { prog->bpf_func = (void *)image; prog->jited = 1; prog->jited_len = proglen; diff --git a/arch/x86/power/hibernate.c b/arch/x86/power/hibernate.c index d8af46e67750..a7ec9b8e2f6b 100644 --- a/arch/x86/power/hibernate.c +++ b/arch/x86/power/hibernate.c @@ -175,7 +175,7 @@ int relocate_restore_code(void) goto out; } pmd = pmd_offset(pud, relocated_restore_code); - if (pmd_large(*pmd)) { + if (pmd_leaf(*pmd)) { set_pmd(pmd, __pmd(pmd_val(*pmd) & ~_PAGE_NX)); goto out; } diff --git a/arch/x86/xen/mmu_pv.c b/arch/x86/xen/mmu_pv.c index 23f30ca52816..c6fe533ceffc 100644 --- a/arch/x86/xen/mmu_pv.c +++ b/arch/x86/xen/mmu_pv.c @@ -1112,7 +1112,7 @@ static void __init xen_cleanmfnmap_pmd(pmd_t *pmd, bool unpin) pte_t *pte_tbl; int i; - if (pmd_large(*pmd)) { + if (pmd_leaf(*pmd)) { pa = pmd_val(*pmd) & PHYSICAL_PAGE_MASK; xen_free_ro_pages(pa, PMD_SIZE); return; @@ -1924,7 +1924,7 @@ static phys_addr_t __init xen_early_virt_to_phys(unsigned long vaddr) if (!pmd_present(pmd)) return 0; pa = pmd_val(pmd) & PTE_PFN_MASK; - if (pmd_large(pmd)) + if (pmd_leaf(pmd)) return pa + (vaddr & ~PMD_MASK); pte = native_make_pte(xen_read_phys_ulong(pa + pte_index(vaddr) * diff --git a/block/bfq-cgroup.c b/block/bfq-cgroup.c index cb2a381193d1..8434581f0a70 100644 --- a/block/bfq-cgroup.c +++ b/block/bfq-cgroup.c @@ -548,14 +548,21 @@ static void bfq_pd_init(struct blkg_policy_data *pd) bfqg->rq_pos_tree = RB_ROOT; } -static void bfq_pd_free(struct blkg_policy_data *pd) +static void bfqg_release(struct rcu_head *rcu) { + struct blkg_policy_data *pd = + container_of(rcu, struct blkg_policy_data, rcu_head); struct bfq_group *bfqg = pd_to_bfqg(pd); bfqg_stats_exit(&bfqg->stats); bfqg_put(bfqg); } +static void bfq_pd_free(struct blkg_policy_data *pd) +{ + call_rcu(&pd->rcu_head, bfqg_release); +} + static void bfq_pd_reset_stats(struct blkg_policy_data *pd) { struct bfq_group *bfqg = pd_to_bfqg(pd); diff --git a/block/blk-cgroup.c b/block/blk-cgroup.c index 9b1034fe8c87..83bbc0c5448f 100644 --- a/block/blk-cgroup.c +++ b/block/blk-cgroup.c @@ -1574,6 +1574,8 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol) if (queue_is_mq(q)) blk_mq_freeze_queue(q); + + mutex_lock(&q->blkcg_mutex); retry: spin_lock_irq(&q->queue_lock); @@ -1638,6 +1640,7 @@ int blkcg_activate_policy(struct gendisk *disk, const struct blkcg_policy *pol) spin_unlock_irq(&q->queue_lock); out: + mutex_unlock(&q->blkcg_mutex); if (queue_is_mq(q)) blk_mq_unfreeze_queue(q); if (pinned_blkg) @@ -2194,7 +2197,7 @@ void blk_cgroup_bio_start(struct bio *bio) } u64_stats_update_end_irqrestore(&bis->sync, flags); - cgroup_rstat_updated(blkcg->css.cgroup, cpu); + __cgroup_rstat_updated(blkcg->css.cgroup, cpu); put_cpu(); } diff --git a/block/blk-cgroup.h b/block/blk-cgroup.h index 5b0bdc268ade..59ea7cc24f35 100644 --- a/block/blk-cgroup.h +++ b/block/blk-cgroup.h @@ -137,6 +137,8 @@ struct blkg_policy_data { struct blkcg_gq *blkg; int plid; bool online; + + struct rcu_head rcu_head; }; /* diff --git a/block/blk-iocost.c b/block/blk-iocost.c index 7291f59986b8..d96113c421be 100644 --- a/block/blk-iocost.c +++ b/block/blk-iocost.c @@ -3019,9 +3019,20 @@ static void ioc_pd_init(struct blkg_policy_data *pd) spin_unlock_irqrestore(&ioc->lock, flags); } +static void iocg_release(struct rcu_head *rcu) +{ + struct blkg_policy_data *pd = + container_of(rcu, struct blkg_policy_data, rcu_head); + struct ioc_gq *iocg = pd_to_iocg(pd); + + free_percpu(iocg->pcpu_stat); + kfree(iocg); +} + static void ioc_pd_free(struct blkg_policy_data *pd) { struct ioc_gq *iocg = pd_to_iocg(pd); + struct blkcg_gq *blkg = pd_to_blkg(pd); struct ioc *ioc = iocg->ioc; unsigned long flags; @@ -3043,8 +3054,14 @@ static void ioc_pd_free(struct blkg_policy_data *pd) hrtimer_cancel(&iocg->waitq_timer); } - free_percpu(iocg->pcpu_stat); - kfree(iocg); + + /* off ->active_iocgs and timer gone, so nothing can re-arm the delay */ + iocg->delay = 0; + iocg->indelay_since = 0; + if (blkg) + blkcg_clear_delay(blkg); + + call_rcu(&pd->rcu_head, iocg_release); } static void ioc_pd_stat(struct blkg_policy_data *pd, struct seq_file *s) diff --git a/block/blk-iolatency.c b/block/blk-iolatency.c index c1a6aba1d59e..e50fc4e4240f 100644 --- a/block/blk-iolatency.c +++ b/block/blk-iolatency.c @@ -1031,13 +1031,30 @@ static void iolatency_pd_offline(struct blkg_policy_data *pd) iolatency_clear_scaling(blkg); } -static void iolatency_pd_free(struct blkg_policy_data *pd) +static void iolat_release(struct rcu_head *rcu) { + struct blkg_policy_data *pd = + container_of(rcu, struct blkg_policy_data, rcu_head); struct iolatency_grp *iolat = pd_to_lat(pd); + free_percpu(iolat->stats); kfree(iolat); } +static void iolatency_pd_free(struct blkg_policy_data *pd) +{ + struct blkcg_gq *blkg = pd_to_blkg(pd); + + /* + * Groups throttled as collateral have min_lat_nsec == 0, so + * iolatency_pd_offline() leaves their delay set. Drop it here, where + * no in-flight bio can re-arm it via check_scale_change(). + */ + if (blkg) + blkcg_clear_delay(blkg); + call_rcu(&pd->rcu_head, iolat_release); +} + static struct cftype iolatency_files[] = { { .name = "latency", diff --git a/block/blk-throttle.c b/block/blk-throttle.c index 16f5766620a4..ed6fa1e7b862 100644 --- a/block/blk-throttle.c +++ b/block/blk-throttle.c @@ -486,16 +486,25 @@ static void throtl_pd_offline(struct blkg_policy_data *pd) throtl_upgrade_state(tg->td); } -static void throtl_pd_free(struct blkg_policy_data *pd) +static void tg_release(struct rcu_head *rcu) { + struct blkg_policy_data *pd = + container_of(rcu, struct blkg_policy_data, rcu_head); struct throtl_grp *tg = pd_to_tg(pd); - del_timer_sync(&tg->service_queue.pending_timer); blkg_rwstat_exit(&tg->stat_bytes); blkg_rwstat_exit(&tg->stat_ios); kfree(tg); } +static void throtl_pd_free(struct blkg_policy_data *pd) +{ + struct throtl_grp *tg = pd_to_tg(pd); + + del_timer_sync(&tg->service_queue.pending_timer); + call_rcu(&pd->rcu_head, tg_release); +} + static struct throtl_grp * throtl_rb_first(struct throtl_service_queue *parent_sq) { diff --git a/crypto/af_alg.c b/crypto/af_alg.c index e7fea668473d..e6e94b34e0af 100644 --- a/crypto/af_alg.c +++ b/crypto/af_alg.c @@ -1207,6 +1207,8 @@ int af_alg_get_rsgl(struct sock *sk, struct msghdr *msg, int flags, seglen = min_t(size_t, (maxsize - len), msg_data_left(msg)); + /* Never pin more pages than the remaining RX accounting budget. */ + seglen = min_t(size_t, seglen, af_alg_rcvbuf(sk)); if (list_empty(&areq->rsgl_list)) { rsgl = &areq->first_rsgl; diff --git a/crypto/algif_skcipher.c b/crypto/algif_skcipher.c index b12df4544d0b..8729ef268d9d 100644 --- a/crypto/algif_skcipher.c +++ b/crypto/algif_skcipher.c @@ -82,8 +82,14 @@ static int _skcipher_recvmsg(struct socket *sock, struct msghdr *msg, * If more buffers are to be expected to be processed, process only * full block size buffers. */ - if (ctx->more || len < ctx->used) + if (ctx->more || len < ctx->used) { + if (len < bs) { + err = -EINVAL; + goto free; + } + len -= len % bs; + } /* * Create a per request TX SGL for this request which tracks the diff --git a/crypto/drbg.c b/crypto/drbg.c index 211757f3f972..8d6dff6f8c9d 100644 --- a/crypto/drbg.c +++ b/crypto/drbg.c @@ -2001,7 +2001,6 @@ static int drbg_kcapi_seed(struct crypto_rng *tfm, */ static inline int __init drbg_healthcheck_sanity(void) { - int len = 0; #define OUTBUFLEN 16 unsigned char buf[OUTBUFLEN]; struct drbg_state *drbg = NULL; @@ -2043,12 +2042,12 @@ static inline int __init drbg_healthcheck_sanity(void) max_addtllen = drbg_max_addtl(drbg); max_request_bytes = drbg_max_request_bytes(drbg); drbg_string_fill(&addtl, buf, max_addtllen + 1); - /* overflow addtllen with additonal info string */ - len = drbg_generate(drbg, buf, OUTBUFLEN, &addtl); - BUG_ON(0 < len); + /* overflow addtllen with additional info string */ + ret = drbg_generate(drbg, buf, OUTBUFLEN, &addtl); + BUG_ON(ret == 0); /* overflow max_bits */ - len = drbg_generate(drbg, buf, (max_request_bytes + 1), NULL); - BUG_ON(0 < len); + ret = drbg_generate(drbg, buf, max_request_bytes + 1, NULL); + BUG_ON(ret == 0); /* overflow max addtllen with personalization string */ ret = drbg_seed(drbg, &addtl, false); diff --git a/drivers/accel/habanalabs/common/device.c b/drivers/accel/habanalabs/common/device.c index ebef56478e18..d8e28c6c849e 100644 --- a/drivers/accel/habanalabs/common/device.c +++ b/drivers/accel/habanalabs/common/device.c @@ -1966,7 +1966,7 @@ static void hl_notifier_event_send(struct hl_notifier_event *notifier_event, u64 notifier_event->events_mask |= event_mask; if (notifier_event->eventfd) - eventfd_signal(notifier_event->eventfd, 1); + eventfd_signal(notifier_event->eventfd); mutex_unlock(¬ifier_event->lock); } diff --git a/drivers/accel/habanalabs/common/memory.c b/drivers/accel/habanalabs/common/memory.c index 33a4246dd735..16516e007873 100644 --- a/drivers/accel/habanalabs/common/memory.c +++ b/drivers/accel/habanalabs/common/memory.c @@ -1806,9 +1806,6 @@ static void hl_release_dmabuf(struct dma_buf *dmabuf) struct hl_dmabuf_priv *hl_dmabuf = dmabuf->priv; struct hl_ctx *ctx; - if (!hl_dmabuf) - return; - ctx = hl_dmabuf->ctx; if (hl_dmabuf->memhash_hnode) @@ -1836,7 +1833,12 @@ static int export_dmabuf(struct hl_ctx *ctx, { DEFINE_DMA_BUF_EXPORT_INFO(exp_info); struct hl_device *hdev = ctx->hdev; - int rc, fd; + CLASS(get_unused_fd, fd)(flags); + + if (fd < 0) { + dev_err(hdev->dev, "failed to get a file descriptor for a dma-buf, %d\n", fd); + return fd; + } exp_info.ops = &habanalabs_dmabuf_ops; exp_info.size = total_size; @@ -1849,13 +1851,6 @@ static int export_dmabuf(struct hl_ctx *ctx, return PTR_ERR(hl_dmabuf->dmabuf); } - fd = dma_buf_fd(hl_dmabuf->dmabuf, flags); - if (fd < 0) { - dev_err(hdev->dev, "failed to get a file descriptor for a dma-buf, %d\n", fd); - rc = fd; - goto err_dma_buf_put; - } - hl_dmabuf->ctx = ctx; hl_ctx_get(hl_dmabuf->ctx); atomic_inc(&ctx->hdev->dmabuf_export_cnt); @@ -1867,13 +1862,10 @@ static int export_dmabuf(struct hl_ctx *ctx, get_file(ctx->hpriv->filp); *dmabuf_fd = fd; + fd_install(fd, hl_dmabuf->dmabuf->file); + fd = -EBADF; return 0; - -err_dma_buf_put: - hl_dmabuf->dmabuf->priv = NULL; - dma_buf_put(hl_dmabuf->dmabuf); - return rc; } static int validate_export_params_common(struct hl_device *hdev, u64 device_addr, u64 size) diff --git a/drivers/accel/ivpu/ivpu_fw_log.c b/drivers/accel/ivpu/ivpu_fw_log.c index ccad9aa99e5d..b51a2f9e8065 100644 --- a/drivers/accel/ivpu/ivpu_fw_log.c +++ b/drivers/accel/ivpu/ivpu_fw_log.c @@ -26,10 +26,17 @@ MODULE_PARM_DESC(ivpu_log_level, " error=" __stringify(IVPU_FW_LOG_ERROR) " fatal=" __stringify(IVPU_FW_LOG_FATAL)); +struct ivpu_fw_log_desc { + struct vpu_tracing_buffer_header *log; + u32 header_size; + u32 size; +}; + static int fw_log_ptr(struct ivpu_device *vdev, struct ivpu_bo *bo, u32 *offset, - struct vpu_tracing_buffer_header **log_header) + struct ivpu_fw_log_desc *desc) { struct vpu_tracing_buffer_header *log; + u32 header_size, size; if ((*offset + sizeof(*log)) > bo->base.size) return -EINVAL; @@ -39,26 +46,33 @@ static int fw_log_ptr(struct ivpu_device *vdev, struct ivpu_bo *bo, u32 *offset, if (log->vpu_canary_start != VPU_TRACING_BUFFER_CANARY) return -EINVAL; - if (log->header_size < sizeof(*log) || log->header_size > 1024) { - ivpu_dbg(vdev, FW_BOOT, "Invalid header size 0x%x\n", log->header_size); + header_size = READ_ONCE(log->header_size); + size = READ_ONCE(log->size); + + if (header_size < sizeof(*log) || header_size > 1024) { + ivpu_dbg(vdev, FW_BOOT, "Invalid header size 0x%x\n", header_size); return -EINVAL; } - if (log->size < log->header_size) { - ivpu_dbg(vdev, FW_BOOT, "Invalid log size 0x%x\n", log->size); + if ((char *)log + size > (char *)bo->kvaddr + bo->base.size) { + ivpu_dbg(vdev, FW_BOOT, "Invalid log size 0x%x\n", size); return -EINVAL; } - if ((char *)log + log->size > (char *)bo->kvaddr + bo->base.size) { - ivpu_dbg(vdev, FW_BOOT, "Invalid log size 0x%x\n", log->size); + if (size < header_size) { + ivpu_dbg(vdev, FW_BOOT, "Invalid log size 0x%x < header size 0x%x\n", + size, header_size); return -EINVAL; } - *log_header = log; - *offset += log->size; + desc->log = log; + desc->header_size = header_size; + desc->size = size; + *offset += size; ivpu_dbg(vdev, FW_BOOT, - "FW log name \"%s\", write offset 0x%x size 0x%x, wrap count %d, hdr version %d size %d format %d, alignment %d", - log->name, log->write_index, log->size, log->wrap_count, log->header_version, - log->header_size, log->format, log->alignment); + "FW log name \"%.*s\", write offset 0x%x size 0x%x, wrap count %d, hdr version %d size %d format %d, alignment %d", + (int)ARRAY_SIZE(log->name), log->name, log->write_index, size, + log->wrap_count, log->header_version, header_size, log->format, + log->alignment); return 0; } @@ -94,21 +108,24 @@ static void buffer_print(char *buffer, u32 size, struct drm_printer *p) drm_printf(p, "%s\n", line); } -static void fw_log_print_buffer(struct ivpu_device *vdev, struct vpu_tracing_buffer_header *log, +static void fw_log_print_buffer(struct ivpu_device *vdev, struct ivpu_fw_log_desc *desc, const char *prefix, bool only_new_msgs, struct drm_printer *p) { - char *log_buffer = (void *)log + log->header_size; - u32 log_size = log->size - log->header_size; + struct vpu_tracing_buffer_header *log = desc->log; + char *log_buffer = (void *)log + desc->header_size; + u32 log_size = desc->size - desc->header_size; u32 log_start = log->read_index; u32 log_end = log->write_index; if (!(log->write_index || log->wrap_count) || (log->write_index == log->read_index && only_new_msgs)) { - drm_printf(p, "==== %s \"%s\" log empty ====\n", prefix, log->name); + drm_printf(p, "==== %s \"%.*s\" log empty ====\n", prefix, + (int)ARRAY_SIZE(log->name), log->name); return; } - drm_printf(p, "==== %s \"%s\" log start ====\n", prefix, log->name); + drm_printf(p, "==== %s \"%.*s\" log start ====\n", prefix, (int)ARRAY_SIZE(log->name), + log->name); if (log->write_index > log->read_index) { buffer_print(log_buffer + log_start, log_end - log_start, p); } else { @@ -116,31 +133,32 @@ static void fw_log_print_buffer(struct ivpu_device *vdev, struct vpu_tracing_buf buffer_print(log_buffer, log_end, p); } drm_printf(p, "\x1b[0m"); - drm_printf(p, "==== %s \"%s\" log end ====\n", prefix, log->name); + drm_printf(p, "==== %s \"%.*s\" log end ====\n", prefix, (int)ARRAY_SIZE(log->name), + log->name); } void ivpu_fw_log_print(struct ivpu_device *vdev, bool only_new_msgs, struct drm_printer *p) { - struct vpu_tracing_buffer_header *log_header; + struct ivpu_fw_log_desc desc; u32 next = 0; - while (fw_log_ptr(vdev, vdev->fw->mem_log_crit, &next, &log_header) == 0) - fw_log_print_buffer(vdev, log_header, "VPU critical", only_new_msgs, p); + while (fw_log_ptr(vdev, vdev->fw->mem_log_crit, &next, &desc) == 0) + fw_log_print_buffer(vdev, &desc, "VPU critical", only_new_msgs, p); next = 0; - while (fw_log_ptr(vdev, vdev->fw->mem_log_verb, &next, &log_header) == 0) - fw_log_print_buffer(vdev, log_header, "VPU verbose", only_new_msgs, p); + while (fw_log_ptr(vdev, vdev->fw->mem_log_verb, &next, &desc) == 0) + fw_log_print_buffer(vdev, &desc, "VPU verbose", only_new_msgs, p); } void ivpu_fw_log_clear(struct ivpu_device *vdev) { - struct vpu_tracing_buffer_header *log_header; + struct ivpu_fw_log_desc desc; u32 next = 0; - while (fw_log_ptr(vdev, vdev->fw->mem_log_crit, &next, &log_header) == 0) - log_header->read_index = log_header->write_index; + while (fw_log_ptr(vdev, vdev->fw->mem_log_crit, &next, &desc) == 0) + desc.log->read_index = desc.log->write_index; next = 0; - while (fw_log_ptr(vdev, vdev->fw->mem_log_verb, &next, &log_header) == 0) - log_header->read_index = log_header->write_index; + while (fw_log_ptr(vdev, vdev->fw->mem_log_verb, &next, &desc) == 0) + desc.log->read_index = desc.log->write_index; } diff --git a/drivers/accel/ivpu/ivpu_gem.h b/drivers/accel/ivpu/ivpu_gem.h index 6b0ceda5f253..61458d118889 100644 --- a/drivers/accel/ivpu/ivpu_gem.h +++ b/drivers/accel/ivpu/ivpu_gem.h @@ -102,15 +102,23 @@ static inline struct ivpu_device *ivpu_bo_to_vdev(struct ivpu_bo *bo) return to_ivpu_device(bo->base.dev); } -static inline void *ivpu_to_cpu_addr(struct ivpu_bo *bo, u32 vpu_addr) +static inline void *ivpu_to_cpu_addr(struct ivpu_bo *bo, u64 vpu_addr, u64 size) { + u64 bo_size = bo->base.size; + u64 offset; + if (vpu_addr < bo->vpu_addr) return NULL; - if (vpu_addr >= (bo->vpu_addr + bo->base.size)) + if (size > bo_size) + return NULL; + + offset = vpu_addr - bo->vpu_addr; + + if (offset > bo_size - size) return NULL; - return bo->kvaddr + (vpu_addr - bo->vpu_addr); + return bo->kvaddr + offset; } static inline u32 cpu_to_vpu_addr(struct ivpu_bo *bo, void *cpu_addr) diff --git a/drivers/accel/ivpu/ivpu_ipc.c b/drivers/accel/ivpu/ivpu_ipc.c index 295c0d7b5039..fdd815961095 100644 --- a/drivers/accel/ivpu/ivpu_ipc.c +++ b/drivers/accel/ivpu/ivpu_ipc.c @@ -84,7 +84,7 @@ ivpu_ipc_tx_prepare(struct ivpu_device *vdev, struct ivpu_ipc_consumer *cons, return -ENOMEM; } - tx_buf = ivpu_to_cpu_addr(ipc->mem_tx, tx_buf_vpu_addr); + tx_buf = ivpu_to_cpu_addr(ipc->mem_tx, tx_buf_vpu_addr, sizeof(*tx_buf)); if (drm_WARN_ON(&vdev->drm, !tx_buf)) { gen_pool_free(ipc->mm_tx, tx_buf_vpu_addr, sizeof(*tx_buf)); return -EIO; @@ -376,7 +376,7 @@ int ivpu_ipc_irq_handler(struct ivpu_device *vdev) return -EIO; } - ipc_hdr = ivpu_to_cpu_addr(ipc->mem_rx, vpu_addr); + ipc_hdr = ivpu_to_cpu_addr(ipc->mem_rx, vpu_addr, sizeof(*ipc_hdr)); if (!ipc_hdr) { ivpu_warn(vdev, "IPC msg 0x%x out of range\n", vpu_addr); continue; @@ -385,7 +385,8 @@ int ivpu_ipc_irq_handler(struct ivpu_device *vdev) jsm_msg = NULL; if (ipc_hdr->channel != IVPU_IPC_CHAN_BOOT_MSG) { - jsm_msg = ivpu_to_cpu_addr(ipc->mem_rx, ipc_hdr->data_addr); + jsm_msg = ivpu_to_cpu_addr(ipc->mem_rx, ipc_hdr->data_addr, + sizeof(*jsm_msg)); if (!jsm_msg) { ivpu_warn(vdev, "JSM msg 0x%x out of range\n", ipc_hdr->data_addr); ivpu_ipc_rx_mark_free(vdev, ipc_hdr, NULL); diff --git a/drivers/accel/qaic/mhi_controller.c b/drivers/accel/qaic/mhi_controller.c index 1405623b03e4..39012b95a930 100644 --- a/drivers/accel/qaic/mhi_controller.c +++ b/drivers/accel/qaic/mhi_controller.c @@ -34,7 +34,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -50,7 +49,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -66,7 +64,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -82,7 +79,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -98,7 +94,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -114,7 +109,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -130,7 +124,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -146,7 +139,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -162,7 +154,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -178,7 +169,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -194,7 +184,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -210,7 +199,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -226,7 +214,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -242,7 +229,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -258,7 +244,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -274,7 +259,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -290,7 +274,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -306,7 +289,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -322,7 +304,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -338,7 +319,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -354,7 +334,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, { @@ -370,7 +349,6 @@ static struct mhi_channel_config aic100_channels[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, .wake_capable = false, }, }; diff --git a/drivers/acpi/acpi_platform.c b/drivers/acpi/acpi_platform.c index 48d15dd785f6..affa076a211d 100644 --- a/drivers/acpi/acpi_platform.c +++ b/drivers/acpi/acpi_platform.c @@ -71,18 +71,15 @@ static struct notifier_block acpi_platform_notifier = { .notifier_call = acpi_platform_device_remove_notify, }; -static void acpi_platform_fill_resource(struct acpi_device *adev, - const struct resource *src, struct resource *dest) +static void acpi_platform_fill_resource(struct device *parent, + const struct resource *src, + struct resource *dest) { - struct device *parent; - *dest = *src; - /* * If the device has parent we need to take its resources into * account as well because this device might consume part of those. */ - parent = acpi_get_first_physical_node(acpi_dev_parent(adev)); if (parent && dev_is_pci(parent)) dest->parent = pci_find_resource(to_pci_dev(parent), dest); } @@ -110,7 +107,8 @@ static unsigned int acpi_platform_resource_count(struct acpi_resource *ares, voi struct platform_device *acpi_create_platform_device(struct acpi_device *adev, const struct property_entry *properties) { - struct acpi_device *parent = acpi_dev_parent(adev); + struct acpi_device *p = acpi_dev_parent(adev); + struct device *parent __free(put_device) = acpi_bus_get_primary_device(p); struct platform_device *pdev = NULL; struct platform_device_info pdevinfo; const struct acpi_device_id *match; @@ -149,7 +147,7 @@ struct platform_device *acpi_create_platform_device(struct acpi_device *adev, } count = 0; list_for_each_entry(rentry, &resource_list, node) - acpi_platform_fill_resource(adev, rentry->res, + acpi_platform_fill_resource(parent, rentry->res, &resources[count++]); acpi_dev_free_resource_list(&resource_list); @@ -161,7 +159,7 @@ struct platform_device *acpi_create_platform_device(struct acpi_device *adev, * attached to it, that physical device should be the parent of the * platform device we are about to create. */ - pdevinfo.parent = parent ? acpi_get_first_physical_node(parent) : NULL; + pdevinfo.parent = parent; pdevinfo.name = dev_name(&adev->dev); pdevinfo.id = PLATFORM_DEVID_NONE; pdevinfo.res = resources; diff --git a/drivers/acpi/acpi_tad.c b/drivers/acpi/acpi_tad.c index 542e16c3ab9f..ca0d5640de05 100644 --- a/drivers/acpi/acpi_tad.c +++ b/drivers/acpi/acpi_tad.c @@ -23,6 +23,7 @@ #include #include #include +#include #include #include #include @@ -48,6 +49,9 @@ MODULE_AUTHOR("Rafael J. Wysocki"); /* Special value for disabled timer or expired timer wake policy. */ #define ACPI_TAD_WAKE_DISABLED (~(u32)0) +/* ACPI TAD RTC */ +#define ACPI_TAD_TZ_UNSPEC 2047 + struct acpi_tad_driver_data { u32 capabilities; }; @@ -66,6 +70,18 @@ struct acpi_tad_rt { u8 padding[3]; /* must be 0 */ } __packed; +static bool acpi_tad_rt_is_invalid(struct acpi_tad_rt *rt) +{ + return rt->year < 1900 || rt->year > 9999 || + rt->month < 1 || rt->month > 12 || + rt->hour > 23 || rt->minute > 59 || rt->second > 59 || + rt->tz < -1440 || + (rt->tz > 1440 && rt->tz != ACPI_TAD_TZ_UNSPEC) || + rt->daylight > 3; +} + +static DEFINE_MUTEX(acpi_tad_aml_lock); + static int acpi_tad_set_real_time(struct device *dev, struct acpi_tad_rt *rt) { acpi_handle handle = ACPI_HANDLE(dev); @@ -79,18 +95,16 @@ static int acpi_tad_set_real_time(struct device *dev, struct acpi_tad_rt *rt) unsigned long long retval; acpi_status status; - if (rt->year < 1900 || rt->year > 9999 || - rt->month < 1 || rt->month > 12 || - rt->hour > 23 || rt->minute > 59 || rt->second > 59 || - rt->tz < -1440 || (rt->tz > 1440 && rt->tz != 2047) || - rt->daylight > 3) - return -ERANGE; + if (acpi_tad_rt_is_invalid(rt)) + return -EINVAL; args[0].buffer.pointer = (u8 *)rt; args[0].buffer.length = sizeof(*rt); pm_runtime_get_sync(dev); + guard(mutex)(&acpi_tad_aml_lock); + status = acpi_evaluate_integer(handle, "_SRT", &arg_list, &retval); pm_runtime_put_sync(dev); @@ -101,43 +115,115 @@ static int acpi_tad_set_real_time(struct device *dev, struct acpi_tad_rt *rt) return 0; } -static int acpi_tad_get_real_time(struct device *dev, struct acpi_tad_rt *rt) +static int acpi_tad_evaluate_grt(struct device *dev, struct acpi_tad_rt *rt) { acpi_handle handle = ACPI_HANDLE(dev); struct acpi_buffer output = { ACPI_ALLOCATE_BUFFER }; - union acpi_object *out_obj; - struct acpi_tad_rt *data; acpi_status status; int ret = -EIO; - pm_runtime_get_sync(dev); + guard(mutex)(&acpi_tad_aml_lock); status = acpi_evaluate_object(handle, "_GRT", NULL, &output); + if (ACPI_SUCCESS(status)) { + union acpi_object *out_obj; + + out_obj = output.pointer; + if (out_obj->type == ACPI_TYPE_BUFFER && + out_obj->buffer.length == sizeof(*rt)) { + struct acpi_tad_rt *data; + + data = (struct acpi_tad_rt *)(out_obj->buffer.pointer); + if (data->valid) { + memcpy(rt, data, sizeof(*rt)); + ret = 0; + } + } + } + ACPI_FREE(output.pointer); + return ret; +} - pm_runtime_put_sync(dev); +static int __acpi_tad_get_real_time(struct device *dev, struct acpi_tad_rt *rt) +{ + int ret; - if (ACPI_FAILURE(status)) - goto out_free; + ret = acpi_tad_evaluate_grt(dev, rt); + if (ret) + return ret; - out_obj = output.pointer; - if (out_obj->type != ACPI_TYPE_BUFFER) - goto out_free; + if (acpi_tad_rt_is_invalid(rt)) + return -ENODATA; - if (out_obj->buffer.length != sizeof(*rt)) - goto out_free; + return 0; +} - data = (struct acpi_tad_rt *)(out_obj->buffer.pointer); - if (!data->valid) - goto out_free; +static int acpi_tad_get_real_time(struct device *dev, struct acpi_tad_rt *rt) +{ + int ret; - memcpy(rt, data, sizeof(*rt)); - ret = 0; + pm_runtime_get_sync(dev); + + ret = __acpi_tad_get_real_time(dev, rt); + + pm_runtime_put_sync(dev); -out_free: - ACPI_FREE(output.pointer); return ret; } +static int __acpi_tad_wake_set(struct device *dev, char *method, u32 timer_id, + u32 value) +{ + acpi_handle handle = ACPI_HANDLE(dev); + union acpi_object args[] = { + { .type = ACPI_TYPE_INTEGER, }, + { .type = ACPI_TYPE_INTEGER, }, + }; + struct acpi_object_list arg_list = { + .pointer = args, + .count = ARRAY_SIZE(args), + }; + unsigned long long retval; + acpi_status status; + + args[0].integer.value = timer_id; + args[1].integer.value = value; + + guard(mutex)(&acpi_tad_aml_lock); + + status = acpi_evaluate_integer(handle, method, &arg_list, &retval); + if (ACPI_FAILURE(status) || retval) + return -EIO; + + return 0; +} + +static int __acpi_tad_wake_read(struct device *dev, char *method, u32 timer_id, + unsigned long long *retval) +{ + acpi_handle handle = ACPI_HANDLE(dev); + union acpi_object args[] = { + { .type = ACPI_TYPE_INTEGER, }, + }; + struct acpi_object_list arg_list = { + .pointer = args, + .count = ARRAY_SIZE(args), + }; + acpi_status status; + + args[0].integer.value = timer_id; + + guard(mutex)(&acpi_tad_aml_lock); + + status = acpi_evaluate_integer(handle, method, &arg_list, retval); + if (ACPI_FAILURE(status)) + return -EIO; + + return 0; +} + +/* sysfs interface */ + static char *acpi_tad_rt_next_field(char *s, int *val) { char *p; @@ -250,31 +336,15 @@ static const struct attribute_group acpi_tad_time_attr_group = { static int acpi_tad_wake_set(struct device *dev, char *method, u32 timer_id, u32 value) { - acpi_handle handle = ACPI_HANDLE(dev); - union acpi_object args[] = { - { .type = ACPI_TYPE_INTEGER, }, - { .type = ACPI_TYPE_INTEGER, }, - }; - struct acpi_object_list arg_list = { - .pointer = args, - .count = ARRAY_SIZE(args), - }; - unsigned long long retval; - acpi_status status; - - args[0].integer.value = timer_id; - args[1].integer.value = value; + int ret; pm_runtime_get_sync(dev); - status = acpi_evaluate_integer(handle, method, &arg_list, &retval); + ret = __acpi_tad_wake_set(dev, method, timer_id, value); pm_runtime_put_sync(dev); - if (ACPI_FAILURE(status) || retval) - return -EIO; - - return 0; + return ret; } static int acpi_tad_wake_write(struct device *dev, const char *buf, char *method, @@ -300,27 +370,17 @@ static int acpi_tad_wake_write(struct device *dev, const char *buf, char *method static ssize_t acpi_tad_wake_read(struct device *dev, char *buf, char *method, u32 timer_id, const char *specval) { - acpi_handle handle = ACPI_HANDLE(dev); - union acpi_object args[] = { - { .type = ACPI_TYPE_INTEGER, }, - }; - struct acpi_object_list arg_list = { - .pointer = args, - .count = ARRAY_SIZE(args), - }; unsigned long long retval; - acpi_status status; - - args[0].integer.value = timer_id; + int ret; pm_runtime_get_sync(dev); - status = acpi_evaluate_integer(handle, method, &arg_list, &retval); + ret = __acpi_tad_wake_read(dev, method, timer_id, &retval); pm_runtime_put_sync(dev); - if (ACPI_FAILURE(status)) - return -EIO; + if (ret) + return ret; if ((u32)retval == ACPI_TAD_WAKE_DISABLED) return sprintf(buf, "%s\n", specval); @@ -371,6 +431,8 @@ static int acpi_tad_clear_status(struct device *dev, u32 timer_id) pm_runtime_get_sync(dev); + guard(mutex)(&acpi_tad_aml_lock); + status = acpi_evaluate_integer(handle, "_CWS", &arg_list, &retval); pm_runtime_put_sync(dev); @@ -412,6 +474,8 @@ static ssize_t acpi_tad_status_read(struct device *dev, char *buf, u32 timer_id) pm_runtime_get_sync(dev); + guard(mutex)(&acpi_tad_aml_lock); + status = acpi_evaluate_integer(handle, "_GWS", &arg_list, &retval); pm_runtime_put_sync(dev); diff --git a/drivers/acpi/bus.c b/drivers/acpi/bus.c index 9d2301b75898..a21bc540d836 100644 --- a/drivers/acpi/bus.c +++ b/drivers/acpi/bus.c @@ -703,30 +703,52 @@ static int __init acpi_setup_sb_notify_handler(void) Device Matching -------------------------------------------------------------------------- */ + +static struct device *primary_physical_device(struct acpi_device *adev) +{ + struct acpi_device_physical_node *pn; + + pn = list_first_entry_or_null(&adev->physical_node_list, + struct acpi_device_physical_node, node); + if (pn) + return pn->dev; + + return NULL; +} + /** - * acpi_get_first_physical_node - Get first physical node of an ACPI device + * acpi_bus_get_primary_device - Get first physical device for a given ACPI one + * @adev: ACPI device to get the first physical device for. + * + * Find the first physical device for which @adev is the ACPI companion and + * reference count it if present. + * + * Return: Pointer to the first physical counterpart of @adev or NULL if there + * are none. Callers are responsible for invoking put_device() on the returned + * device. + */ +struct device *acpi_bus_get_primary_device(struct acpi_device *adev) +{ + if (!adev) + return NULL; + + guard(mutex)(&adev->physical_node_lock); + + return get_device(primary_physical_device(adev)); +} +EXPORT_SYMBOL_GPL(acpi_bus_get_primary_device); + +/** + * acpi_get_first_physical_node - Find first physical node of an ACPI device * @adev: ACPI device in question * * Return: First physical node of ACPI device @adev */ struct device *acpi_get_first_physical_node(struct acpi_device *adev) { - struct mutex *physical_node_lock = &adev->physical_node_lock; - struct device *phys_dev; - - mutex_lock(physical_node_lock); - if (list_empty(&adev->physical_node_list)) { - phys_dev = NULL; - } else { - const struct acpi_device_physical_node *node; + guard(mutex)(&adev->physical_node_lock); - node = list_first_entry(&adev->physical_node_list, - struct acpi_device_physical_node, node); - - phys_dev = node->dev; - } - mutex_unlock(physical_node_lock); - return phys_dev; + return primary_physical_device(adev); } EXPORT_SYMBOL_GPL(acpi_get_first_physical_node); diff --git a/drivers/acpi/processor_idle.c b/drivers/acpi/processor_idle.c index d755c266e358..5db34356328c 100644 --- a/drivers/acpi/processor_idle.c +++ b/drivers/acpi/processor_idle.c @@ -1175,7 +1175,7 @@ static int acpi_processor_get_lpi_info(struct acpi_processor *pr) return 0; } -int __weak acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi) +int __weak __cpuidle acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi) { return -ENODEV; } @@ -1188,7 +1188,7 @@ int __weak acpi_processor_ffh_lpi_enter(struct acpi_lpi_state *lpi) * * Return: 0 for success or negative value for error */ -static int acpi_idle_lpi_enter(struct cpuidle_device *dev, +static int __cpuidle acpi_idle_lpi_enter(struct cpuidle_device *dev, struct cpuidle_driver *drv, int index) { struct acpi_processor *pr; diff --git a/drivers/android/binder.c b/drivers/android/binder.c index 20014ed53604..ddf8ff5eaa66 100644 --- a/drivers/android/binder.c +++ b/drivers/android/binder.c @@ -2839,8 +2839,9 @@ static int binder_proc_transaction(struct binder_transaction *t, t_outdated->buffer = NULL; buffer->transaction = NULL; trace_binder_transaction_update_buffer_release(buffer); - binder_release_entire_buffer(proc, NULL, buffer, false); + binder_release_entire_buffer(proc, NULL, buffer, true); binder_alloc_free_buf(&proc->alloc, buffer); + binder_free_txn_fixups(t_outdated); kfree(t_outdated); binder_stats_deleted(BINDER_STAT_TRANSACTION); } diff --git a/drivers/android/binder_alloc.c b/drivers/android/binder_alloc.c index 6a4be6087ae6..6c24197c48f7 100644 --- a/drivers/android/binder_alloc.c +++ b/drivers/android/binder_alloc.c @@ -234,7 +234,7 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, if (page->page_ptr) { trace_binder_alloc_lru_start(alloc, index); - on_lru = list_lru_del(&binder_alloc_lru, &page->lru); + on_lru = list_lru_del_obj(&binder_alloc_lru, &page->lru); WARN_ON(!on_lru); trace_binder_alloc_lru_end(alloc, index); @@ -285,7 +285,7 @@ static int binder_update_page_range(struct binder_alloc *alloc, int allocate, trace_binder_free_lru_start(alloc, index); - ret = list_lru_add(&binder_alloc_lru, &page->lru); + ret = list_lru_add_obj(&binder_alloc_lru, &page->lru); WARN_ON(!ret); trace_binder_free_lru_end(alloc, index); @@ -846,7 +846,7 @@ void binder_alloc_deferred_release(struct binder_alloc *alloc) if (!alloc->pages[i].page_ptr) continue; - on_lru = list_lru_del(&binder_alloc_lru, + on_lru = list_lru_del_obj(&binder_alloc_lru, &alloc->pages[i].lru); page_addr = alloc->buffer + i * PAGE_SIZE; binder_alloc_debug(BINDER_DEBUG_BUFFER_ALLOC, @@ -1283,4 +1283,3 @@ int binder_alloc_copy_from_buffer(struct binder_alloc *alloc, return binder_alloc_do_buffer_copy(alloc, false, buffer, buffer_offset, dest, bytes); } - diff --git a/drivers/base/core.c b/drivers/base/core.c index a972b585bae9..9dc293cee52b 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -1508,7 +1508,8 @@ static void __device_links_no_driver(struct device *dev) if (link->supplier->links.status == DL_DEV_DRIVER_BOUND) { WRITE_ONCE(link->status, DL_STATE_AVAILABLE); } else { - WARN_ON(!(link->flags & DL_FLAG_SYNC_STATE_ONLY)); + WARN_ON(link->supplier->links.status != DL_DEV_UNBINDING && + !(link->flags & DL_FLAG_SYNC_STATE_ONLY)); WRITE_ONCE(link->status, DL_STATE_DORMANT); } } diff --git a/drivers/base/memory.c b/drivers/base/memory.c index 468ef9b3983c..b92690a8e2b4 100644 --- a/drivers/base/memory.c +++ b/drivers/base/memory.c @@ -160,7 +160,7 @@ static ssize_t state_show(struct device *dev, struct device_attribute *attr, break; default: WARN_ON(1); - return sysfs_emit(buf, "ERROR-UNKNOWN-%ld\n", mem->state); + return sysfs_emit(buf, "ERROR-UNKNOWN-%d\n", mem->state); } return sysfs_emit(buf, "%s\n", output); @@ -414,7 +414,7 @@ static ssize_t phys_device_show(struct device *dev, static int print_allowed_zone(char *buf, int len, int nid, struct memory_group *group, unsigned long start_pfn, unsigned long nr_pages, - int online_type, struct zone *default_zone) + enum mmop online_type, struct zone *default_zone) { struct zone *zone; diff --git a/drivers/base/property.c b/drivers/base/property.c index b6a70830555e..a87cbb858d35 100644 --- a/drivers/base/property.c +++ b/drivers/base/property.c @@ -498,6 +498,41 @@ int fwnode_property_match_string(const struct fwnode_handle *fwnode, } EXPORT_SYMBOL_GPL(fwnode_property_match_string); +/** + * fwnode_property_match_property_string - find a property string value in an array and return index + * @fwnode: Firmware node to get the property of + * @propname: Name of the property holding the string value + * @array: String array to search in + * @n: Size of the @array + * + * Find a property string value in a given @array and if it is found return + * the index back. + * + * Return: index, starting from %0, if the string value was found in the @array (success), + * %-ENOENT when the string value was not found in the @array, + * %-EINVAL if given arguments are not valid, + * %-ENODATA if the property does not have a value, + * %-EPROTO or %-EILSEQ if the property is not a string, + * %-ENXIO if no suitable firmware interface is present. + */ +int fwnode_property_match_property_string(const struct fwnode_handle *fwnode, + const char *propname, const char * const *array, size_t n) +{ + const char *string; + int ret; + + ret = fwnode_property_read_string(fwnode, propname, &string); + if (ret) + return ret; + + ret = match_string(array, n, string); + if (ret < 0) + ret = -ENOENT; + + return ret; +} +EXPORT_SYMBOL_GPL(fwnode_property_match_property_string); + /** * fwnode_property_get_reference_args() - Find a reference with arguments * @fwnode: Firmware node where to look for the reference diff --git a/drivers/block/null_blk/main.c b/drivers/block/null_blk/main.c index a011ae2055d0..243565d47eb5 100644 --- a/drivers/block/null_blk/main.c +++ b/drivers/block/null_blk/main.c @@ -552,8 +552,13 @@ static ssize_t nullb_device_zone_readonly_store(struct config_item *item, const char *page, size_t count) { struct nullb_device *dev = to_nullb_device(item); + ssize_t ret; + + mutex_lock(&lock); + ret = zone_cond_store(dev, page, count, BLK_ZONE_COND_READONLY); + mutex_unlock(&lock); - return zone_cond_store(dev, page, count, BLK_ZONE_COND_READONLY); + return ret; } CONFIGFS_ATTR_WO(nullb_device_, zone_readonly); @@ -561,8 +566,13 @@ static ssize_t nullb_device_zone_offline_store(struct config_item *item, const char *page, size_t count) { struct nullb_device *dev = to_nullb_device(item); + ssize_t ret; - return zone_cond_store(dev, page, count, BLK_ZONE_COND_OFFLINE); + mutex_lock(&lock); + ret = zone_cond_store(dev, page, count, BLK_ZONE_COND_OFFLINE); + mutex_unlock(&lock); + + return ret; } CONFIGFS_ATTR_WO(nullb_device_, zone_offline); @@ -785,7 +795,6 @@ static void null_free_dev(struct nullb_device *dev) if (!dev) return; - null_free_zoned_dev(dev); badblocks_exit(&dev->badblocks); kfree(dev); } @@ -1870,6 +1879,7 @@ static void null_del_dev(struct nullb *nullb) } put_disk(nullb->disk); + null_free_zoned_dev(dev); if (dev->queue_mode == NULL_Q_MQ && nullb->tag_set == &nullb->__tag_set) blk_mq_free_tag_set(nullb->tag_set); diff --git a/drivers/block/ublk_drv.c b/drivers/block/ublk_drv.c index 356f32b6f180..ad1156e36d8a 100644 --- a/drivers/block/ublk_drv.c +++ b/drivers/block/ublk_drv.c @@ -938,8 +938,9 @@ static int ublk_map_io(const struct ublk_queue *ubq, const struct request *req, struct iovec iov; const int dir = ITER_DEST; - import_single_range(dir, u64_to_user_ptr(io->addr), rq_bytes, - &iov, &iter); + if (import_single_range(dir, u64_to_user_ptr(io->addr), rq_bytes, + &iov, &iter) < 0) + return 0; return ublk_copy_user_pages(req, 0, &iter, dir); } @@ -962,8 +963,10 @@ static int ublk_unmap_io(const struct ublk_queue *ubq, WARN_ON_ONCE(io->res > rq_bytes); - import_single_range(dir, u64_to_user_ptr(io->addr), io->res, - &iov, &iter); + if (import_single_range(dir, u64_to_user_ptr(io->addr), io->res, + &iov, &iter) < 0) + return 0; + return ublk_copy_user_pages(req, 0, &iter, dir); } return rq_bytes; @@ -1134,13 +1137,37 @@ static inline void __ublk_complete_rq(struct request *req) ublk_end_request(req, res); } +static void ublk_do_fail_rq(struct request *req) +{ + struct ublk_queue *ubq = req->mq_hctx->driver_data; + + if (ublk_queue_can_use_recovery_reissue(ubq)) + blk_mq_requeue_request(req, false); + else + __ublk_complete_rq(req); +} + +static void ublk_fail_rq_fn(struct kref *ref) +{ + struct ublk_rq_data *data = container_of(ref, struct ublk_rq_data, + ref); + struct request *req = blk_mq_rq_from_pdu(data); + + ublk_do_fail_rq(req); +} + static void ublk_complete_rq(struct kref *ref) { struct ublk_rq_data *data = container_of(ref, struct ublk_rq_data, ref); struct request *req = blk_mq_rq_from_pdu(data); + struct ublk_queue *ubq = req->mq_hctx->driver_data; - __ublk_complete_rq(req); + /* the abort path's reference may be dropped by another holder last */ + if (ubq->ios[req->tag].flags & UBLK_IO_FLAG_ABORTED) + ublk_do_fail_rq(req); + else + __ublk_complete_rq(req); } /* @@ -1158,10 +1185,13 @@ static void __ublk_fail_req(struct ublk_queue *ubq, struct ublk_io *io, if (!(io->flags & UBLK_IO_FLAG_ABORTED)) { io->flags |= UBLK_IO_FLAG_ABORTED; - if (ublk_queue_can_use_recovery_reissue(ubq)) - blk_mq_requeue_request(req, false); - else - ublk_put_req_ref(ubq, req); + if (ublk_need_req_ref(ubq)) { + struct ublk_rq_data *data = blk_mq_rq_to_pdu(req); + + kref_put(&data->ref, ublk_fail_rq_fn); + } else { + ublk_do_fail_rq(req); + } } } diff --git a/drivers/block/zram/zram_drv.c b/drivers/block/zram/zram_drv.c index 0c422e4179ad..451908dd2d67 100644 --- a/drivers/block/zram/zram_drv.c +++ b/drivers/block/zram/zram_drv.c @@ -627,7 +627,7 @@ static ssize_t writeback_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { struct zram *zram = dev_to_zram(dev); - unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; + unsigned long nr_pages; unsigned long index = 0; struct bio bio; struct bio_vec bio_vec; @@ -648,11 +648,9 @@ static ssize_t writeback_store(struct device *dev, if (strncmp(buf, PAGE_WB_SIG, sizeof(PAGE_WB_SIG) - 1)) return -EINVAL; - if (kstrtol(buf + sizeof(PAGE_WB_SIG) - 1, 10, &index) || - index >= nr_pages) + if (kstrtol(buf + sizeof(PAGE_WB_SIG) - 1, 10, &index)) return -EINVAL; - nr_pages = 1; mode = PAGE_WRITEBACK; } @@ -667,6 +665,15 @@ static ssize_t writeback_store(struct device *dev, goto release_init_lock; } + nr_pages = zram->disksize >> PAGE_SHIFT; + if (mode == PAGE_WRITEBACK) { + if (index >= nr_pages) { + ret = -EINVAL; + goto release_init_lock; + } + nr_pages = 1; + } + page = alloc_page(GFP_KERNEL); if (!page) { ret = -ENOMEM; @@ -878,8 +885,7 @@ static ssize_t read_block_state(struct file *file, char __user *buf, char *kbuf; ssize_t index, written = 0; struct zram *zram = file->private_data; - unsigned long nr_pages = zram->disksize >> PAGE_SHIFT; - struct timespec64 ts; + unsigned long nr_pages; kbuf = kvmalloc(count, GFP_KERNEL); if (!kbuf) @@ -892,8 +898,11 @@ static ssize_t read_block_state(struct file *file, char __user *buf, return -EINVAL; } + nr_pages = zram->disksize >> PAGE_SHIFT; + for (index = *ppos; index < nr_pages; index++) { int copied; + struct timespec64 ts; zram_slot_lock(zram, index); if (!zram_allocated(zram, index)) @@ -2021,6 +2030,8 @@ static void zram_destroy_comps(struct zram *zram) kfree(zram->comp_algs[prio]); zram->comp_algs[prio] = NULL; } + + comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor); } static void zram_reset_device(struct zram *zram) @@ -2038,8 +2049,6 @@ static void zram_reset_device(struct zram *zram) zram_destroy_comps(zram); memset(&zram->stats, 0, sizeof(zram->stats)); reset_bdev(zram); - - comp_algorithm_set(zram, ZRAM_PRIMARY_COMP, default_compressor); up_write(&zram->init_lock); } diff --git a/drivers/bluetooth/btintel.c b/drivers/bluetooth/btintel.c index d713bbebbc5a..8541bfaddd06 100644 --- a/drivers/bluetooth/btintel.c +++ b/drivers/bluetooth/btintel.c @@ -265,7 +265,8 @@ void btintel_hw_error(struct hci_dev *hdev, u8 code) goto unlock; } - bt_dev_err(hdev, "Exception info %s", (char *)(skb->data + 1)); + bt_dev_err(hdev, "Exception info %.*s", (int)(skb->len - 1), + (char *)(skb->data + 1)); kfree_skb(skb); @@ -384,8 +385,16 @@ int btintel_load_ddc_config(struct hci_dev *hdev, const char *ddc_name) /* DDC file contains one or more DDC structure which has * Length (1 byte), DDC ID (2 bytes), and DDC value (Length - 2). */ - while (fw->size > fw_ptr - fw->data) { - u8 cmd_plen = fw_ptr[0] + sizeof(u8); + while (fw->size > (size_t)(fw_ptr - fw->data)) { + size_t remaining = fw->size - (fw_ptr - fw->data); + unsigned int cmd_plen = fw_ptr[0] + 1U; + + if (cmd_plen < 3 || cmd_plen > U8_MAX || cmd_plen > remaining) { + bt_dev_err(hdev, "Malformed DDC record (plen=%u, remaining=%zu)", + cmd_plen, remaining); + release_firmware(fw); + return -EINVAL; + } skb = __hci_cmd_sync(hdev, 0xfc8b, cmd_plen, fw_ptr, HCI_INIT_TIMEOUT); diff --git a/drivers/bluetooth/btqcomsmd.c b/drivers/bluetooth/btqcomsmd.c index 11c7e04bf394..4ce8354ebb86 100644 --- a/drivers/bluetooth/btqcomsmd.c +++ b/drivers/bluetooth/btqcomsmd.c @@ -188,7 +188,10 @@ static int btqcomsmd_probe(struct platform_device *pdev) return 0; hci_free_dev: + rpmsg_destroy_ept(btq->cmd_channel); + rpmsg_destroy_ept(btq->acl_channel); hci_free_dev(hdev); + return ret; destroy_cmd_channel: rpmsg_destroy_ept(btq->cmd_channel); destroy_acl_channel: @@ -197,17 +200,16 @@ static int btqcomsmd_probe(struct platform_device *pdev) return ret; } -static int btqcomsmd_remove(struct platform_device *pdev) +static void btqcomsmd_remove(struct platform_device *pdev) { struct btqcomsmd *btq = platform_get_drvdata(pdev); hci_unregister_dev(btq->hdev); - hci_free_dev(btq->hdev); rpmsg_destroy_ept(btq->cmd_channel); rpmsg_destroy_ept(btq->acl_channel); - return 0; + hci_free_dev(btq->hdev); } static const struct of_device_id btqcomsmd_of_match[] = { @@ -218,7 +220,7 @@ MODULE_DEVICE_TABLE(of, btqcomsmd_of_match); static struct platform_driver btqcomsmd_driver = { .probe = btqcomsmd_probe, - .remove = btqcomsmd_remove, + .remove_new = btqcomsmd_remove, .driver = { .name = "btqcomsmd", .of_match_table = btqcomsmd_of_match, diff --git a/drivers/bluetooth/btusb.c b/drivers/bluetooth/btusb.c index ced60cd806fb..95becfdb0d2e 100644 --- a/drivers/bluetooth/btusb.c +++ b/drivers/bluetooth/btusb.c @@ -305,6 +305,8 @@ static const struct usb_device_id quirks_table[] = { BTUSB_WIDEBAND_SPEECH }, { USB_DEVICE(0x13d3, 0x3501), .driver_info = BTUSB_QCA_ROME | BTUSB_WIDEBAND_SPEECH }, + { USB_DEVICE(0x13d3, 0x3503), .driver_info = BTUSB_QCA_ROME | + BTUSB_WIDEBAND_SPEECH }, /* QCA WCN6855 chipset */ { USB_DEVICE(0x0cf3, 0xe600), .driver_info = BTUSB_QCA_WCN6855 | diff --git a/drivers/bus/fsl-mc/fsl-mc-bus.c b/drivers/bus/fsl-mc/fsl-mc-bus.c index b5a1daed1ea4..afe610817f82 100644 --- a/drivers/bus/fsl-mc/fsl-mc-bus.c +++ b/drivers/bus/fsl-mc/fsl-mc-bus.c @@ -93,12 +93,16 @@ static int fsl_mc_bus_match(struct device *dev, struct device_driver *drv) struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev); struct fsl_mc_driver *mc_drv = to_fsl_mc_driver(drv); bool found = false; + int ret; /* When driver_override is set, only bind to the matching driver */ - if (mc_dev->driver_override) { - found = !strcmp(mc_dev->driver_override, mc_drv->driver.name); + ret = device_match_driver_override(dev, drv); + if (ret > 0) { + found = true; goto out; } + if (ret == 0) + goto out; if (!mc_drv->match_id_table) goto out; @@ -187,39 +191,8 @@ static ssize_t modalias_show(struct device *dev, struct device_attribute *attr, } static DEVICE_ATTR_RO(modalias); -static ssize_t driver_override_store(struct device *dev, - struct device_attribute *attr, - const char *buf, size_t count) -{ - struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev); - int ret; - - if (WARN_ON(dev->bus != &fsl_mc_bus_type)) - return -EINVAL; - - ret = driver_set_override(dev, &mc_dev->driver_override, buf, count); - if (ret) - return ret; - - return count; -} - -static ssize_t driver_override_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - struct fsl_mc_device *mc_dev = to_fsl_mc_device(dev); - ssize_t len; - - device_lock(dev); - len = sysfs_emit(buf, "%s\n", mc_dev->driver_override); - device_unlock(dev); - return len; -} -static DEVICE_ATTR_RW(driver_override); - static struct attribute *fsl_mc_dev_attrs[] = { &dev_attr_modalias.attr, - &dev_attr_driver_override.attr, NULL, }; @@ -322,6 +295,7 @@ ATTRIBUTE_GROUPS(fsl_mc_bus); struct bus_type fsl_mc_bus_type = { .name = "fsl-mc", + .driver_override = true, .match = fsl_mc_bus_match, .uevent = fsl_mc_bus_uevent, .dma_configure = fsl_mc_dma_configure, @@ -931,9 +905,6 @@ static struct notifier_block fsl_mc_nb; */ void fsl_mc_device_remove(struct fsl_mc_device *mc_dev) { - kfree(mc_dev->driver_override); - mc_dev->driver_override = NULL; - /* * The device-specific remove callback will get invoked by device_del() */ diff --git a/drivers/bus/mhi/host/pci_generic.c b/drivers/bus/mhi/host/pci_generic.c index 6121ca846541..11377ae6a7ba 100644 --- a/drivers/bus/mhi/host/pci_generic.c +++ b/drivers/bus/mhi/host/pci_generic.c @@ -80,22 +80,6 @@ struct mhi_pci_dev_info { .doorbell_mode_switch = false, \ } -#define MHI_CHANNEL_CONFIG_DL_AUTOQUEUE(ch_num, ch_name, el_count, ev_ring) \ - { \ - .num = ch_num, \ - .name = ch_name, \ - .num_elements = el_count, \ - .event_ring = ev_ring, \ - .dir = DMA_FROM_DEVICE, \ - .ee_mask = BIT(MHI_EE_AMSS), \ - .pollcfg = 0, \ - .doorbell = MHI_DB_BRST_DISABLE, \ - .lpm_notify = false, \ - .offload_channel = false, \ - .doorbell_mode_switch = false, \ - .auto_queue = true, \ - } - #define MHI_EVENT_CONFIG_CTRL(ev_ring, el_count) \ { \ .num_elements = el_count, \ @@ -247,7 +231,7 @@ static const struct mhi_channel_config modem_qcom_v1_mhi_channels[] = { MHI_CHANNEL_CONFIG_UL(14, "QMI", 4, 0), MHI_CHANNEL_CONFIG_DL(15, "QMI", 4, 0), MHI_CHANNEL_CONFIG_UL(20, "IPCR", 8, 0), - MHI_CHANNEL_CONFIG_DL_AUTOQUEUE(21, "IPCR", 8, 0), + MHI_CHANNEL_CONFIG_DL(21, "IPCR", 8, 0), MHI_CHANNEL_CONFIG_UL_FP(34, "FIREHOSE", 32, 0), MHI_CHANNEL_CONFIG_DL_FP(35, "FIREHOSE", 32, 0), MHI_CHANNEL_CONFIG_UL(46, "IP_SW0", 64, 2), @@ -507,7 +491,7 @@ static const struct mhi_channel_config mhi_telit_fn980_hw_v1_channels[] = { MHI_CHANNEL_CONFIG_UL(14, "QMI", 32, 0), MHI_CHANNEL_CONFIG_DL(15, "QMI", 32, 0), MHI_CHANNEL_CONFIG_UL(20, "IPCR", 16, 0), - MHI_CHANNEL_CONFIG_DL_AUTOQUEUE(21, "IPCR", 16, 0), + MHI_CHANNEL_CONFIG_DL(21, "IPCR", 16, 0), MHI_CHANNEL_CONFIG_HW_UL(100, "IP_HW0", 128, 1), MHI_CHANNEL_CONFIG_HW_DL(101, "IP_HW0", 128, 2), }; diff --git a/drivers/char/hw_random/core.c b/drivers/char/hw_random/core.c index a182fe794f98..e3e730dc64c4 100644 --- a/drivers/char/hw_random/core.c +++ b/drivers/char/hw_random/core.c @@ -20,23 +20,25 @@ #include #include #include +#include #include #include #include #include #include +#include #define RNG_MODULE_NAME "hw_random" #define RNG_BUFFER_SIZE (SMP_CACHE_BYTES < 32 ? 32 : SMP_CACHE_BYTES) -static struct hwrng *current_rng; +static struct hwrng __rcu *current_rng; /* the current rng has been explicitly chosen by user via sysfs */ static int cur_rng_set_by_user; static struct task_struct *hwrng_fill; /* list of registered rngs */ static LIST_HEAD(rng_list); -/* Protects rng_list and current_rng */ +/* Protects rng_list, hwrng_fill and updating on current_rng */ static DEFINE_MUTEX(rng_mutex); /* Protects rng read functions, data_avail, rng_buffer and rng_fillbuf */ static DEFINE_MUTEX(reading_mutex); @@ -77,18 +79,39 @@ static void add_early_randomness(struct hwrng *rng) } } -static inline void cleanup_rng(struct kref *kref) +static void cleanup_rng_work(struct work_struct *work) { - struct hwrng *rng = container_of(kref, struct hwrng, ref); + struct hwrng *rng = container_of(work, struct hwrng, cleanup_work); + + /* + * Hold rng_mutex here so we serialize in case they set_current_rng + * on rng again immediately. + */ + mutex_lock(&rng_mutex); + + /* Skip if rng has been reinitialized. */ + if (kref_read(&rng->ref)) { + mutex_unlock(&rng_mutex); + return; + } if (rng->cleanup) rng->cleanup(rng); complete(&rng->cleanup_done); + mutex_unlock(&rng_mutex); +} + +static inline void cleanup_rng(struct kref *kref) +{ + struct hwrng *rng = container_of(kref, struct hwrng, ref); + + schedule_work(&rng->cleanup_work); } static int set_current_rng(struct hwrng *rng) { + struct hwrng *old_rng; int err; BUG_ON(!mutex_is_locked(&rng_mutex)); @@ -97,8 +120,14 @@ static int set_current_rng(struct hwrng *rng) if (err) return err; - drop_current_rng(); - current_rng = rng; + old_rng = rcu_dereference_protected(current_rng, + lockdep_is_held(&rng_mutex)); + rcu_assign_pointer(current_rng, rng); + + if (old_rng) { + synchronize_rcu(); + kref_put(&old_rng->ref, cleanup_rng); + } /* if necessary, start hwrng thread */ if (!hwrng_fill) { @@ -114,47 +143,56 @@ static int set_current_rng(struct hwrng *rng) static void drop_current_rng(void) { - BUG_ON(!mutex_is_locked(&rng_mutex)); - if (!current_rng) + struct hwrng *rng; + + rng = rcu_dereference_protected(current_rng, + lockdep_is_held(&rng_mutex)); + if (!rng) return; + RCU_INIT_POINTER(current_rng, NULL); + synchronize_rcu(); + + if (hwrng_fill) { + kthread_stop(hwrng_fill); + hwrng_fill = NULL; + } + /* decrease last reference for triggering the cleanup */ - kref_put(¤t_rng->ref, cleanup_rng); - current_rng = NULL; + kref_put(&rng->ref, cleanup_rng); } -/* Returns ERR_PTR(), NULL or refcounted hwrng */ +/* Returns NULL or refcounted hwrng */ static struct hwrng *get_current_rng_nolock(void) { - if (current_rng) - kref_get(¤t_rng->ref); + struct hwrng *rng; + + rng = rcu_dereference_protected(current_rng, + lockdep_is_held(&rng_mutex)); + if (rng) + kref_get(&rng->ref); - return current_rng; + return rng; } static struct hwrng *get_current_rng(void) { struct hwrng *rng; - if (mutex_lock_interruptible(&rng_mutex)) - return ERR_PTR(-ERESTARTSYS); + rcu_read_lock(); + rng = rcu_dereference(current_rng); + if (rng) + kref_get(&rng->ref); - rng = get_current_rng_nolock(); + rcu_read_unlock(); - mutex_unlock(&rng_mutex); return rng; } static void put_rng(struct hwrng *rng) { - /* - * Hold rng_mutex here so we serialize in case they set_current_rng - * on rng again immediately. - */ - mutex_lock(&rng_mutex); if (rng) kref_put(&rng->ref, cleanup_rng); - mutex_unlock(&rng_mutex); } static int hwrng_init(struct hwrng *rng) @@ -219,10 +257,6 @@ static ssize_t rng_dev_read(struct file *filp, char __user *buf, while (size) { rng = get_current_rng(); - if (IS_ERR(rng)) { - err = PTR_ERR(rng); - goto out; - } if (!rng) { err = -ENODEV; goto out; @@ -309,7 +343,7 @@ static struct miscdevice rng_miscdev = { static int enable_best_rng(void) { - struct hwrng *rng, *new_rng = NULL; + struct hwrng *rng, *cur_rng, *new_rng = NULL; int ret = -ENODEV; BUG_ON(!mutex_is_locked(&rng_mutex)); @@ -327,7 +361,9 @@ static int enable_best_rng(void) new_rng = rng; } - ret = ((new_rng == current_rng) ? 0 : set_current_rng(new_rng)); + cur_rng = rcu_dereference_protected(current_rng, + lockdep_is_held(&rng_mutex)); + ret = ((new_rng == cur_rng) ? 0 : set_current_rng(new_rng)); if (!ret) cur_rng_set_by_user = 0; @@ -345,7 +381,8 @@ static ssize_t rng_current_store(struct device *dev, if (err) return -ERESTARTSYS; - old_rng = current_rng; + old_rng = rcu_dereference_protected(current_rng, + lockdep_is_held(&rng_mutex)); if (sysfs_streq(buf, "")) { err = enable_best_rng(); } else { @@ -378,8 +415,6 @@ static ssize_t rng_current_show(struct device *dev, struct hwrng *rng; rng = get_current_rng(); - if (IS_ERR(rng)) - return PTR_ERR(rng); ret = snprintf(buf, PAGE_SIZE, "%s\n", rng ? rng->name : "none"); put_rng(rng); @@ -423,8 +458,6 @@ static ssize_t rng_quality_show(struct device *dev, struct hwrng *rng; rng = get_current_rng(); - if (IS_ERR(rng)) - return PTR_ERR(rng); if (!rng) /* no need to put_rng */ return -ENODEV; @@ -439,6 +472,7 @@ static ssize_t rng_quality_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { + struct hwrng *rng; u16 quality; int ret = -EINVAL; @@ -455,12 +489,13 @@ static ssize_t rng_quality_store(struct device *dev, goto out; } - if (!current_rng) { + rng = rcu_dereference_protected(current_rng, lockdep_is_held(&rng_mutex)); + if (!rng) { ret = -ENODEV; goto out; } - current_rng->quality = quality; + rng->quality = quality; current_quality = quality; /* obsolete */ /* the best available RNG may have changed */ @@ -506,8 +541,20 @@ static int hwrng_fillfn(void *unused) struct hwrng *rng; rng = get_current_rng(); - if (IS_ERR(rng) || !rng) + if (!rng) { + /* + * Keep the task_struct alive until kthread_stop() + * is called to avoid UAF in drop_current_rng(). + */ + while (!kthread_should_stop()) { + set_current_state(TASK_INTERRUPTIBLE); + if (!kthread_should_stop()) + schedule(); + } + set_current_state(TASK_RUNNING); break; + } + mutex_lock(&reading_mutex); rc = rng_get_data(rng, rng_fillbuf, rng_buffer_size(), 1); @@ -535,14 +582,13 @@ static int hwrng_fillfn(void *unused) add_hwgenerator_randomness((void *)rng_fillbuf, rc, entropy >> 10, true); } - hwrng_fill = NULL; return 0; } int hwrng_register(struct hwrng *rng) { int err = -EINVAL; - struct hwrng *tmp; + struct hwrng *cur_rng, *tmp; bool is_new_current = false; if (!rng->name || (!rng->data_read && !rng->read)) @@ -558,6 +604,7 @@ int hwrng_register(struct hwrng *rng) } list_add_tail(&rng->list, &rng_list); + INIT_WORK(&rng->cleanup_work, cleanup_rng_work); init_completion(&rng->cleanup_done); complete(&rng->cleanup_done); init_completion(&rng->dying); @@ -565,8 +612,10 @@ int hwrng_register(struct hwrng *rng) /* Adjust quality field to always have a proper value */ rng->quality = min_t(u16, min_t(u16, default_quality, 1024), rng->quality ?: 1024); - if (!current_rng || - (!cur_rng_set_by_user && rng->quality > current_rng->quality)) { + cur_rng = rcu_dereference_protected(current_rng, + lockdep_is_held(&rng_mutex)); + if (!cur_rng || + (!cur_rng_set_by_user && rng->quality > cur_rng->quality)) { /* * Set new rng as current as the new rng source * provides better entropy quality and was not @@ -574,7 +623,7 @@ int hwrng_register(struct hwrng *rng) */ err = set_current_rng(rng); if (err) - goto out_unlock; + goto out_list_del; /* to use current_rng in add_early_randomness() we need * to take a ref */ @@ -595,6 +644,8 @@ int hwrng_register(struct hwrng *rng) if (is_new_current) put_rng(rng); return 0; +out_list_del: + list_del_init(&rng->list); out_unlock: mutex_unlock(&rng_mutex); out: @@ -609,10 +660,12 @@ void hwrng_unregister(struct hwrng *rng) mutex_lock(&rng_mutex); - old_rng = current_rng; + old_rng = rcu_dereference_protected(current_rng, + lockdep_is_held(&rng_mutex)); list_del(&rng->list); complete_all(&rng->dying); - if (current_rng == rng) { + + if (old_rng == rng) { err = enable_best_rng(); if (err) { drop_current_rng(); @@ -621,12 +674,7 @@ void hwrng_unregister(struct hwrng *rng) } new_rng = get_current_rng_nolock(); - if (list_empty(&rng_list)) { - mutex_unlock(&rng_mutex); - if (hwrng_fill) - kthread_stop(hwrng_fill); - } else - mutex_unlock(&rng_mutex); + mutex_unlock(&rng_mutex); if (new_rng) { if (old_rng != new_rng) @@ -721,7 +769,7 @@ static int __init hwrng_modinit(void) static void __exit hwrng_modexit(void) { mutex_lock(&rng_mutex); - BUG_ON(current_rng); + WARN_ON(rcu_access_pointer(current_rng)); kfree(rng_buffer); kfree(rng_fillbuf); mutex_unlock(&rng_mutex); diff --git a/drivers/char/hw_random/stm32-rng.c b/drivers/char/hw_random/stm32-rng.c index efb6a9f9a11b..fcfb0c4cea23 100644 --- a/drivers/char/hw_random/stm32-rng.c +++ b/drivers/char/hw_random/stm32-rng.c @@ -159,18 +159,13 @@ static int stm32_rng_probe(struct platform_device *ofdev) pm_runtime_set_autosuspend_delay(dev, 100); pm_runtime_use_autosuspend(dev); - pm_runtime_enable(dev); + err = devm_pm_runtime_enable(dev); + if (err) + return err; return devm_hwrng_register(dev, &priv->rng); } -static int stm32_rng_remove(struct platform_device *ofdev) -{ - pm_runtime_disable(&ofdev->dev); - - return 0; -} - #ifdef CONFIG_PM static int stm32_rng_runtime_suspend(struct device *dev) { @@ -212,7 +207,6 @@ static struct platform_driver stm32_rng_driver = { .of_match_table = stm32_rng_match, }, .probe = stm32_rng_probe, - .remove = stm32_rng_remove, }; module_platform_driver(stm32_rng_driver); diff --git a/drivers/clk/mediatek/clk-pllfh.c b/drivers/clk/mediatek/clk-pllfh.c index 094ec8a26d66..6fd70f8d7a58 100644 --- a/drivers/clk/mediatek/clk-pllfh.c +++ b/drivers/clk/mediatek/clk-pllfh.c @@ -194,12 +194,55 @@ static void mtk_clk_unregister_pllfh(struct clk_hw *hw) kfree(fh); } +static void mtk_clk_cleanup_pllfhs(void __iomem *iomem_base, + const struct mtk_pll_data *plls, int num_plls, + void __iomem *iomem_fhctl_base, + struct mtk_pllfh_data *pllfhs, int num_fhs, + struct clk_hw_onecell_data *clk_data) +{ + void __iomem *base = iomem_base; + void __iomem *fhctl_base = iomem_fhctl_base; + int i; + + for (i = num_plls - 1; i >= 0; i--) { + const struct mtk_pll_data *pll = &plls[i]; + struct mtk_pllfh_data *pllfh; + bool use_fhctl; + + if (IS_ERR_OR_NULL(clk_data->hws[pll->id])) + continue; + + pllfh = get_pllfh_by_id(pllfhs, num_fhs, pll->id); + use_fhctl = fhctl_is_supported_and_enabled(pllfh); + + if (!base) + base = mtk_clk_pll_get_base(clk_data->hws[pll->id], + pll); + + if (use_fhctl) { + if (!fhctl_base) + fhctl_base = pllfh->state.base; + mtk_clk_unregister_pllfh(clk_data->hws[pll->id]); + } else { + mtk_clk_unregister_pll(clk_data->hws[pll->id]); + } + + clk_data->hws[pll->id] = ERR_PTR(-ENOENT); + } + + if (fhctl_base) + iounmap(fhctl_base); + + if (base) + iounmap(base); +} + int mtk_clk_register_pllfhs(struct device_node *node, const struct mtk_pll_data *plls, int num_plls, struct mtk_pllfh_data *pllfhs, int num_fhs, struct clk_hw_onecell_data *clk_data) { - void __iomem *base; + void __iomem *base, *fhctl_base = NULL; int i; struct clk_hw *hw; @@ -235,24 +278,8 @@ int mtk_clk_register_pllfhs(struct device_node *node, return 0; err: - while (--i >= 0) { - const struct mtk_pll_data *pll = &plls[i]; - struct mtk_pllfh_data *pllfh; - bool use_fhctl; - - pllfh = get_pllfh_by_id(pllfhs, num_fhs, pll->id); - use_fhctl = fhctl_is_supported_and_enabled(pllfh); - - if (use_fhctl) - mtk_clk_unregister_pllfh(clk_data->hws[pll->id]); - else - mtk_clk_unregister_pll(clk_data->hws[pll->id]); - - clk_data->hws[pll->id] = ERR_PTR(-ENOENT); - } - - iounmap(base); - + mtk_clk_cleanup_pllfhs(base, plls, i, fhctl_base, pllfhs, num_fhs, + clk_data); return PTR_ERR(hw); } EXPORT_SYMBOL_GPL(mtk_clk_register_pllfhs); @@ -261,38 +288,10 @@ void mtk_clk_unregister_pllfhs(const struct mtk_pll_data *plls, int num_plls, struct mtk_pllfh_data *pllfhs, int num_fhs, struct clk_hw_onecell_data *clk_data) { - void __iomem *base = NULL, *fhctl_base = NULL; - int i; - if (!clk_data) return; - for (i = num_plls; i > 0; i--) { - const struct mtk_pll_data *pll = &plls[i - 1]; - struct mtk_pllfh_data *pllfh; - bool use_fhctl; - - if (IS_ERR_OR_NULL(clk_data->hws[pll->id])) - continue; - - pllfh = get_pllfh_by_id(pllfhs, num_fhs, pll->id); - use_fhctl = fhctl_is_supported_and_enabled(pllfh); - - if (use_fhctl) { - fhctl_base = pllfh->state.base; - mtk_clk_unregister_pllfh(clk_data->hws[pll->id]); - } else { - base = mtk_clk_pll_get_base(clk_data->hws[pll->id], - pll); - mtk_clk_unregister_pll(clk_data->hws[pll->id]); - } - - clk_data->hws[pll->id] = ERR_PTR(-ENOENT); - } - - if (fhctl_base) - iounmap(fhctl_base); - - iounmap(base); + mtk_clk_cleanup_pllfhs(NULL, plls, num_plls, NULL, pllfhs, + num_fhs, clk_data); } EXPORT_SYMBOL_GPL(mtk_clk_unregister_pllfhs); diff --git a/drivers/clk/qcom/gcc-qcm2290.c b/drivers/clk/qcom/gcc-qcm2290.c index 7b9c978f9233..fc56996e4c41 100644 --- a/drivers/clk/qcom/gcc-qcm2290.c +++ b/drivers/clk/qcom/gcc-qcm2290.c @@ -116,7 +116,7 @@ static const struct alpha_pll_config gpll10_config = { .vco_mask = GENMASK(21, 20), .main_output_mask = BIT(0), .config_ctl_val = 0x4001055B, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, }; static struct clk_alpha_pll gpll10 = { @@ -148,7 +148,7 @@ static const struct alpha_pll_config gpll11_config = { .vco_mask = GENMASK(21, 20), .main_output_mask = BIT(0), .config_ctl_val = 0x4001055B, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, }; static struct clk_alpha_pll gpll11 = { @@ -309,7 +309,7 @@ static const struct alpha_pll_config gpll8_config = { .post_div_val = 0x1 << 8, .post_div_mask = GENMASK(11, 8), .config_ctl_val = 0x4001055B, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, }; static struct clk_alpha_pll gpll8 = { diff --git a/drivers/clk/qcom/gcc-sm6115.c b/drivers/clk/qcom/gcc-sm6115.c index 13e521cd4259..ad3abb3aa638 100644 --- a/drivers/clk/qcom/gcc-sm6115.c +++ b/drivers/clk/qcom/gcc-sm6115.c @@ -120,7 +120,7 @@ static const struct alpha_pll_config gpll10_config = { .vco_mask = GENMASK(21, 20), .main_output_mask = BIT(0), .config_ctl_val = 0x4001055b, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, .test_ctl_hi_mask = 0x1, }; @@ -173,7 +173,7 @@ static const struct alpha_pll_config gpll11_config = { .vco_val = 0x2 << 20, .vco_mask = GENMASK(21, 20), .config_ctl_val = 0x4001055b, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, .test_ctl_hi_mask = 0x1, }; @@ -367,7 +367,7 @@ static const struct alpha_pll_config gpll8_config = { .post_div_val = 0x1 << 8, .post_div_mask = GENMASK(11, 8), .config_ctl_val = 0x4001055b, - .test_ctl_hi1_val = 0x1, + .test_ctl_hi_val = 0x1, .test_ctl_hi_mask = 0x1, }; diff --git a/drivers/clk/sunxi-ng/ccu-sun50i-h616.c b/drivers/clk/sunxi-ng/ccu-sun50i-h616.c index 21e918582aa5..5831bcfbe500 100644 --- a/drivers/clk/sunxi-ng/ccu-sun50i-h616.c +++ b/drivers/clk/sunxi-ng/ccu-sun50i-h616.c @@ -318,10 +318,16 @@ static SUNXI_CCU_M_WITH_MUX_GATE(gpu0_clk, "gpu0", gpu0_parents, 0x670, 24, 1, /* mux */ BIT(31), /* gate */ CLK_SET_RATE_PARENT); + +/* + * This clk is needed as a temporary fall back during GPU PLL freq changes. + * Set CLK_IS_CRITICAL flag to prevent from being disabled. + */ +#define SUN50I_H616_GPU_CLK1_REG 0x674 static SUNXI_CCU_M_WITH_GATE(gpu1_clk, "gpu1", "pll-periph0-2x", 0x674, 0, 2, /* M */ BIT(31),/* gate */ - 0); + CLK_IS_CRITICAL); static SUNXI_CCU_GATE(bus_gpu_clk, "bus-gpu", "psi-ahb1-ahb2", 0x67c, BIT(0), 0); @@ -1090,11 +1096,37 @@ static const u32 usb2_clk_regs[] = { SUN50I_H616_USB3_CLK_REG, }; +static struct ccu_mux_nb sun50i_h616_cpu_nb = { + .common = &cpux_clk.common, + .cm = &cpux_clk.mux, + .delay_us = 1, /* manual doesn't really say */ + .bypass_index = 4, /* PLL_PERI0@600MHz, as recommended by manual */ +}; + +static struct ccu_pll_nb sun50i_h616_pll_cpu_nb = { + .common = &pll_cpux_clk.common, + .enable = BIT(29), /* LOCK_ENABLE */ + .lock = BIT(28), +}; + +static struct ccu_mux_nb sun50i_h616_gpu_nb = { + .common = &gpu0_clk.common, + .cm = &gpu0_clk.mux, + .delay_us = 1, /* manual doesn't really say */ + .bypass_index = 1, /* GPU_CLK1@400MHz */ +}; + +static struct ccu_pll_nb sun50i_h616_pll_gpu_nb = { + .common = &pll_gpu_clk.common, + .enable = BIT(29), /* LOCK_ENABLE */ + .lock = BIT(28), +}; + static int sun50i_h616_ccu_probe(struct platform_device *pdev) { void __iomem *reg; u32 val; - int i; + int ret, i; reg = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(reg)) @@ -1138,6 +1170,14 @@ static int sun50i_h616_ccu_probe(struct platform_device *pdev) val &= ~(GENMASK(21, 16) | BIT(0)); writel(val | (11 << 16) | BIT(0), reg + SUN50I_H616_PLL_AUDIO_REG); + /* + * Set the input-divider for the gpu1 clock to 3, to reach a safe 400 MHz. + */ + val = readl(reg + SUN50I_H616_GPU_CLK1_REG); + val &= ~GENMASK(1, 0); + val |= 2; + writel(val, reg + SUN50I_H616_GPU_CLK1_REG); + /* * First clock parent (osc32K) is unusable for CEC. But since there * is no good way to force parent switch (both run with same frequency), @@ -1147,7 +1187,25 @@ static int sun50i_h616_ccu_probe(struct platform_device *pdev) val |= BIT(24); writel(val, reg + SUN50I_H616_HDMI_CEC_CLK_REG); - return devm_sunxi_ccu_probe(&pdev->dev, reg, &sun50i_h616_ccu_desc); + ret = devm_sunxi_ccu_probe(&pdev->dev, reg, &sun50i_h616_ccu_desc); + if (ret) + return ret; + + /* Reparent CPU during CPU PLL rate changes */ + ccu_mux_notifier_register(pll_cpux_clk.common.hw.clk, + &sun50i_h616_cpu_nb); + + /* Re-lock the CPU PLL after any rate changes */ + ccu_pll_notifier_register(&sun50i_h616_pll_cpu_nb); + + /* Reparent GPU during GPU PLL rate changes */ + ccu_mux_notifier_register(pll_gpu_clk.common.hw.clk, + &sun50i_h616_gpu_nb); + + /* Re-lock the GPU PLL after any rate changes */ + ccu_pll_notifier_register(&sun50i_h616_pll_gpu_nb); + + return 0; } static const struct of_device_id sun50i_h616_ccu_ids[] = { diff --git a/drivers/clocksource/samsung_pwm_timer.c b/drivers/clocksource/samsung_pwm_timer.c index b9561e3f196c..0544124cf5ce 100644 --- a/drivers/clocksource/samsung_pwm_timer.c +++ b/drivers/clocksource/samsung_pwm_timer.c @@ -56,7 +56,7 @@ #define TCON_AUTORELOAD(chan) \ ((chan < 5) ? _TCON_AUTORELOAD(chan) : _TCON_AUTORELOAD4(chan)) -DEFINE_SPINLOCK(samsung_pwm_lock); +DEFINE_RAW_SPINLOCK(samsung_pwm_lock); EXPORT_SYMBOL(samsung_pwm_lock); struct samsung_pwm_clocksource { @@ -87,14 +87,14 @@ static void samsung_timer_set_prescale(unsigned int channel, u16 prescale) if (channel >= 2) shift = TCFG0_PRESCALER1_SHIFT; - spin_lock_irqsave(&samsung_pwm_lock, flags); + raw_spin_lock_irqsave(&samsung_pwm_lock, flags); reg = readl(pwm.base + REG_TCFG0); reg &= ~(TCFG0_PRESCALER_MASK << shift); reg |= (prescale - 1) << shift; writel(reg, pwm.base + REG_TCFG0); - spin_unlock_irqrestore(&samsung_pwm_lock, flags); + raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags); } static void samsung_timer_set_divisor(unsigned int channel, u8 divisor) @@ -106,14 +106,14 @@ static void samsung_timer_set_divisor(unsigned int channel, u8 divisor) bits = (fls(divisor) - 1) - pwm.variant.div_base; - spin_lock_irqsave(&samsung_pwm_lock, flags); + raw_spin_lock_irqsave(&samsung_pwm_lock, flags); reg = readl(pwm.base + REG_TCFG1); reg &= ~(TCFG1_MUX_MASK << shift); reg |= bits << shift; writel(reg, pwm.base + REG_TCFG1); - spin_unlock_irqrestore(&samsung_pwm_lock, flags); + raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags); } static void samsung_time_stop(unsigned int channel) @@ -124,13 +124,13 @@ static void samsung_time_stop(unsigned int channel) if (channel > 0) ++channel; - spin_lock_irqsave(&samsung_pwm_lock, flags); + raw_spin_lock_irqsave(&samsung_pwm_lock, flags); tcon = readl_relaxed(pwm.base + REG_TCON); tcon &= ~TCON_START(channel); writel_relaxed(tcon, pwm.base + REG_TCON); - spin_unlock_irqrestore(&samsung_pwm_lock, flags); + raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags); } static void samsung_time_setup(unsigned int channel, unsigned long tcnt) @@ -142,7 +142,7 @@ static void samsung_time_setup(unsigned int channel, unsigned long tcnt) if (tcon_chan > 0) ++tcon_chan; - spin_lock_irqsave(&samsung_pwm_lock, flags); + raw_spin_lock_irqsave(&samsung_pwm_lock, flags); tcon = readl_relaxed(pwm.base + REG_TCON); @@ -153,7 +153,7 @@ static void samsung_time_setup(unsigned int channel, unsigned long tcnt) writel_relaxed(tcnt, pwm.base + REG_TCMPB(channel)); writel_relaxed(tcon, pwm.base + REG_TCON); - spin_unlock_irqrestore(&samsung_pwm_lock, flags); + raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags); } static void samsung_time_start(unsigned int channel, bool periodic) @@ -164,7 +164,7 @@ static void samsung_time_start(unsigned int channel, bool periodic) if (channel > 0) ++channel; - spin_lock_irqsave(&samsung_pwm_lock, flags); + raw_spin_lock_irqsave(&samsung_pwm_lock, flags); tcon = readl_relaxed(pwm.base + REG_TCON); @@ -178,7 +178,7 @@ static void samsung_time_start(unsigned int channel, bool periodic) writel_relaxed(tcon, pwm.base + REG_TCON); - spin_unlock_irqrestore(&samsung_pwm_lock, flags); + raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags); } static int samsung_set_next_event(unsigned long cycles, diff --git a/drivers/cpufreq/acpi-cpufreq.c b/drivers/cpufreq/acpi-cpufreq.c index 0bfced81cc17..f9d50d81ac61 100644 --- a/drivers/cpufreq/acpi-cpufreq.c +++ b/drivers/cpufreq/acpi-cpufreq.c @@ -926,7 +926,7 @@ static int acpi_cpufreq_cpu_init(struct cpufreq_policy *policy) return result; } -static int acpi_cpufreq_cpu_exit(struct cpufreq_policy *policy) +static void acpi_cpufreq_cpu_exit(struct cpufreq_policy *policy) { struct acpi_cpufreq_data *data = policy->driver_data; @@ -939,8 +939,6 @@ static int acpi_cpufreq_cpu_exit(struct cpufreq_policy *policy) free_cpumask_var(data->freqdomain_cpus); kfree(policy->freq_table); kfree(data); - - return 0; } static int acpi_cpufreq_resume(struct cpufreq_policy *policy) diff --git a/drivers/cpufreq/amd-pstate.c b/drivers/cpufreq/amd-pstate.c index 7e7fa04188cf..664bd617a2d6 100644 --- a/drivers/cpufreq/amd-pstate.c +++ b/drivers/cpufreq/amd-pstate.c @@ -900,7 +900,7 @@ static int amd_pstate_cpu_init(struct cpufreq_policy *policy) return ret; } -static int amd_pstate_cpu_exit(struct cpufreq_policy *policy) +static void amd_pstate_cpu_exit(struct cpufreq_policy *policy) { struct amd_cpudata *cpudata = policy->driver_data; @@ -908,8 +908,6 @@ static int amd_pstate_cpu_exit(struct cpufreq_policy *policy) freq_qos_remove_request(&cpudata->req[0]); policy->fast_switch_possible = false; kfree(cpudata); - - return 0; } static int amd_pstate_cpu_resume(struct cpufreq_policy *policy) @@ -1353,7 +1351,7 @@ static int amd_pstate_epp_cpu_init(struct cpufreq_policy *policy) return ret; } -static int amd_pstate_epp_cpu_exit(struct cpufreq_policy *policy) +static void amd_pstate_epp_cpu_exit(struct cpufreq_policy *policy) { struct amd_cpudata *cpudata = policy->driver_data; @@ -1363,7 +1361,6 @@ static int amd_pstate_epp_cpu_exit(struct cpufreq_policy *policy) } pr_debug("CPU %d exiting\n", policy->cpu); - return 0; } static void amd_pstate_epp_update_limit(struct cpufreq_policy *policy) diff --git a/drivers/cpufreq/apple-soc-cpufreq.c b/drivers/cpufreq/apple-soc-cpufreq.c index 9ba6b09775f6..873c8b786475 100644 --- a/drivers/cpufreq/apple-soc-cpufreq.c +++ b/drivers/cpufreq/apple-soc-cpufreq.c @@ -223,21 +223,19 @@ static int apple_soc_cpufreq_init(struct cpufreq_policy *policy) return -ENODEV; } - ret = dev_pm_opp_of_add_table(cpu_dev); - if (ret < 0) { - dev_err(cpu_dev, "%s: failed to add OPP table: %d\n", __func__, ret); - return ret; - } + priv = kzalloc(sizeof(*priv), GFP_KERNEL); + if (!priv) + return -ENOMEM; ret = apple_soc_cpufreq_find_cluster(policy, ®_base, &info); if (ret) { dev_err(cpu_dev, "%s: failed to get cluster info: %d\n", __func__, ret); - return ret; + goto out_free_priv; } - ret = dev_pm_opp_set_sharing_cpus(cpu_dev, policy->cpus); - if (ret) { - dev_err(cpu_dev, "%s: failed to mark OPPs as shared: %d\n", __func__, ret); + ret = dev_pm_opp_of_cpumask_add_table(policy->cpus); + if (ret < 0) { + dev_err(cpu_dev, "%s: failed to add OPP table: %d\n", __func__, ret); goto out_iounmap; } @@ -245,19 +243,13 @@ static int apple_soc_cpufreq_init(struct cpufreq_policy *policy) if (ret <= 0) { dev_dbg(cpu_dev, "OPP table is not ready, deferring probe\n"); ret = -EPROBE_DEFER; - goto out_free_opp; - } - - priv = kzalloc(sizeof(*priv), GFP_KERNEL); - if (!priv) { - ret = -ENOMEM; - goto out_free_opp; + goto out_free_table; } ret = dev_pm_opp_init_cpufreq_table(cpu_dev, &freq_table); if (ret) { dev_err(cpu_dev, "failed to init cpufreq table: %d\n", ret); - goto out_free_priv; + goto out_free_table; } /* Get OPP levels (p-state indexes) and stash them in driver_data */ @@ -302,25 +294,23 @@ static int apple_soc_cpufreq_init(struct cpufreq_policy *policy) out_free_cpufreq_table: dev_pm_opp_free_cpufreq_table(cpu_dev, &freq_table); -out_free_priv: - kfree(priv); -out_free_opp: - dev_pm_opp_remove_all_dynamic(cpu_dev); +out_free_table: + dev_pm_opp_of_cpumask_remove_table(policy->cpus); out_iounmap: iounmap(reg_base); +out_free_priv: + kfree(priv); return ret; } -static int apple_soc_cpufreq_exit(struct cpufreq_policy *policy) +static void apple_soc_cpufreq_exit(struct cpufreq_policy *policy) { struct apple_cpu_priv *priv = policy->driver_data; dev_pm_opp_free_cpufreq_table(priv->cpu_dev, &policy->freq_table); - dev_pm_opp_remove_all_dynamic(priv->cpu_dev); + dev_pm_opp_of_cpumask_remove_table(policy->cpus); iounmap(priv->reg_base); kfree(priv); - - return 0; } static struct cpufreq_driver apple_soc_cpufreq_driver = { diff --git a/drivers/cpufreq/bmips-cpufreq.c b/drivers/cpufreq/bmips-cpufreq.c index 39221a9a187a..17a4c174553d 100644 --- a/drivers/cpufreq/bmips-cpufreq.c +++ b/drivers/cpufreq/bmips-cpufreq.c @@ -121,11 +121,9 @@ static int bmips_cpufreq_target_index(struct cpufreq_policy *policy, return 0; } -static int bmips_cpufreq_exit(struct cpufreq_policy *policy) +static void bmips_cpufreq_exit(struct cpufreq_policy *policy) { kfree(policy->freq_table); - - return 0; } static int bmips_cpufreq_init(struct cpufreq_policy *policy) diff --git a/drivers/cpufreq/cppc_cpufreq.c b/drivers/cpufreq/cppc_cpufreq.c index 4d96eed64fe0..a10ea3be7dc6 100644 --- a/drivers/cpufreq/cppc_cpufreq.c +++ b/drivers/cpufreq/cppc_cpufreq.c @@ -707,7 +707,7 @@ static int cppc_cpufreq_cpu_init(struct cpufreq_policy *policy) return ret; } -static int cppc_cpufreq_cpu_exit(struct cpufreq_policy *policy) +static void cppc_cpufreq_cpu_exit(struct cpufreq_policy *policy) { struct cppc_cpudata *cpu_data = policy->driver_data; struct cppc_perf_caps *caps = &cpu_data->perf_caps; @@ -724,7 +724,6 @@ static int cppc_cpufreq_cpu_exit(struct cpufreq_policy *policy) caps->lowest_perf, cpu, ret); cppc_cpufreq_put_cpu_data(policy); - return 0; } static inline u64 get_delta(u64 t1, u64 t0) diff --git a/drivers/cpufreq/cpufreq-dt.c b/drivers/cpufreq/cpufreq-dt.c index 2d83bbc65dd0..eaf02579ea74 100644 --- a/drivers/cpufreq/cpufreq-dt.c +++ b/drivers/cpufreq/cpufreq-dt.c @@ -166,10 +166,9 @@ static int cpufreq_offline(struct cpufreq_policy *policy) return 0; } -static int cpufreq_exit(struct cpufreq_policy *policy) +static void cpufreq_exit(struct cpufreq_policy *policy) { clk_put(policy->clk); - return 0; } static struct cpufreq_driver dt_cpufreq_driver = { diff --git a/drivers/cpufreq/e_powersaver.c b/drivers/cpufreq/e_powersaver.c index ab93bce8ae77..6e958b09e1b5 100644 --- a/drivers/cpufreq/e_powersaver.c +++ b/drivers/cpufreq/e_powersaver.c @@ -360,14 +360,13 @@ static int eps_cpu_init(struct cpufreq_policy *policy) return 0; } -static int eps_cpu_exit(struct cpufreq_policy *policy) +static void eps_cpu_exit(struct cpufreq_policy *policy) { unsigned int cpu = policy->cpu; /* Bye */ kfree(eps_cpu[cpu]); eps_cpu[cpu] = NULL; - return 0; } static struct cpufreq_driver eps_driver = { diff --git a/drivers/cpufreq/intel_pstate.c b/drivers/cpufreq/intel_pstate.c index df865612158a..b7441e693733 100644 --- a/drivers/cpufreq/intel_pstate.c +++ b/drivers/cpufreq/intel_pstate.c @@ -2703,13 +2703,11 @@ static int intel_pstate_cpu_offline(struct cpufreq_policy *policy) return intel_cpufreq_cpu_offline(policy); } -static int intel_pstate_cpu_exit(struct cpufreq_policy *policy) +static void intel_pstate_cpu_exit(struct cpufreq_policy *policy) { pr_debug("CPU %d exiting\n", policy->cpu); policy->fast_switch_possible = false; - - return 0; } static int __intel_pstate_cpu_init(struct cpufreq_policy *policy) @@ -3044,7 +3042,7 @@ static int intel_cpufreq_cpu_init(struct cpufreq_policy *policy) return ret; } -static int intel_cpufreq_cpu_exit(struct cpufreq_policy *policy) +static void intel_cpufreq_cpu_exit(struct cpufreq_policy *policy) { struct freq_qos_request *req; @@ -3054,7 +3052,7 @@ static int intel_cpufreq_cpu_exit(struct cpufreq_policy *policy) freq_qos_remove_request(req); kfree(req); - return intel_pstate_cpu_exit(policy); + intel_pstate_cpu_exit(policy); } static int intel_cpufreq_suspend(struct cpufreq_policy *policy) diff --git a/drivers/cpufreq/mediatek-cpufreq-hw.c b/drivers/cpufreq/mediatek-cpufreq-hw.c index 55353fe7b9e7..aeb5e6304542 100644 --- a/drivers/cpufreq/mediatek-cpufreq-hw.c +++ b/drivers/cpufreq/mediatek-cpufreq-hw.c @@ -260,7 +260,7 @@ static int mtk_cpufreq_hw_cpu_init(struct cpufreq_policy *policy) return 0; } -static int mtk_cpufreq_hw_cpu_exit(struct cpufreq_policy *policy) +static void mtk_cpufreq_hw_cpu_exit(struct cpufreq_policy *policy) { struct mtk_cpufreq_data *data = policy->driver_data; struct resource *res = data->res; @@ -270,8 +270,6 @@ static int mtk_cpufreq_hw_cpu_exit(struct cpufreq_policy *policy) writel_relaxed(0x0, data->reg_bases[REG_FREQ_ENABLE]); iounmap(base); release_mem_region(res->start, resource_size(res)); - - return 0; } static void mtk_cpufreq_register_em(struct cpufreq_policy *policy) diff --git a/drivers/cpufreq/mediatek-cpufreq.c b/drivers/cpufreq/mediatek-cpufreq.c index a0a61919bc4c..aba34fb7948e 100644 --- a/drivers/cpufreq/mediatek-cpufreq.c +++ b/drivers/cpufreq/mediatek-cpufreq.c @@ -599,13 +599,11 @@ static int mtk_cpufreq_init(struct cpufreq_policy *policy) return 0; } -static int mtk_cpufreq_exit(struct cpufreq_policy *policy) +static void mtk_cpufreq_exit(struct cpufreq_policy *policy) { struct mtk_cpu_dvfs_info *info = policy->driver_data; dev_pm_opp_free_cpufreq_table(info->cpu_dev, &policy->freq_table); - - return 0; } static struct cpufreq_driver mtk_cpufreq_driver = { diff --git a/drivers/cpufreq/omap-cpufreq.c b/drivers/cpufreq/omap-cpufreq.c index 895690856665..3458d5cc9b7f 100644 --- a/drivers/cpufreq/omap-cpufreq.c +++ b/drivers/cpufreq/omap-cpufreq.c @@ -135,11 +135,10 @@ static int omap_cpu_init(struct cpufreq_policy *policy) return 0; } -static int omap_cpu_exit(struct cpufreq_policy *policy) +static void omap_cpu_exit(struct cpufreq_policy *policy) { freq_table_free(); clk_put(policy->clk); - return 0; } static struct cpufreq_driver omap_driver = { diff --git a/drivers/cpufreq/pasemi-cpufreq.c b/drivers/cpufreq/pasemi-cpufreq.c index 039a66bbe1be..ee925b53b6b9 100644 --- a/drivers/cpufreq/pasemi-cpufreq.c +++ b/drivers/cpufreq/pasemi-cpufreq.c @@ -204,21 +204,19 @@ static int pas_cpufreq_cpu_init(struct cpufreq_policy *policy) return err; } -static int pas_cpufreq_cpu_exit(struct cpufreq_policy *policy) +static void pas_cpufreq_cpu_exit(struct cpufreq_policy *policy) { /* * We don't support CPU hotplug. Don't unmap after the system * has already made it to a running state. */ if (system_state >= SYSTEM_RUNNING) - return 0; + return; if (sdcasr_mapbase) iounmap(sdcasr_mapbase); if (sdcpwr_mapbase) iounmap(sdcpwr_mapbase); - - return 0; } static int pas_cpufreq_target(struct cpufreq_policy *policy, diff --git a/drivers/cpufreq/pcc-cpufreq.c b/drivers/cpufreq/pcc-cpufreq.c index 335429d0683b..6eef6a87c561 100644 --- a/drivers/cpufreq/pcc-cpufreq.c +++ b/drivers/cpufreq/pcc-cpufreq.c @@ -564,18 +564,12 @@ static int pcc_cpufreq_cpu_init(struct cpufreq_policy *policy) return result; } -static int pcc_cpufreq_cpu_exit(struct cpufreq_policy *policy) -{ - return 0; -} - static struct cpufreq_driver pcc_cpufreq_driver = { .flags = CPUFREQ_CONST_LOOPS, .get = pcc_get_freq, .verify = pcc_cpufreq_verify, .target = pcc_cpufreq_target, .init = pcc_cpufreq_cpu_init, - .exit = pcc_cpufreq_cpu_exit, .name = "pcc-cpufreq", }; diff --git a/drivers/cpufreq/powernow-k6.c b/drivers/cpufreq/powernow-k6.c index 41eefef95d87..f0a4a6c31204 100644 --- a/drivers/cpufreq/powernow-k6.c +++ b/drivers/cpufreq/powernow-k6.c @@ -219,7 +219,7 @@ static int powernow_k6_cpu_init(struct cpufreq_policy *policy) } -static int powernow_k6_cpu_exit(struct cpufreq_policy *policy) +static void powernow_k6_cpu_exit(struct cpufreq_policy *policy) { unsigned int i; @@ -234,10 +234,9 @@ static int powernow_k6_cpu_exit(struct cpufreq_policy *policy) cpufreq_freq_transition_begin(policy, &freqs); powernow_k6_target(policy, i); cpufreq_freq_transition_end(policy, &freqs, 0); - break; + return; } } - return 0; } static unsigned int powernow_k6_get(unsigned int cpu) diff --git a/drivers/cpufreq/powernow-k7.c b/drivers/cpufreq/powernow-k7.c index 5d515fc34836..4271446c8725 100644 --- a/drivers/cpufreq/powernow-k7.c +++ b/drivers/cpufreq/powernow-k7.c @@ -644,7 +644,7 @@ static int powernow_cpu_init(struct cpufreq_policy *policy) return 0; } -static int powernow_cpu_exit(struct cpufreq_policy *policy) +static void powernow_cpu_exit(struct cpufreq_policy *policy) { #ifdef CONFIG_X86_POWERNOW_K7_ACPI if (acpi_processor_perf) { @@ -655,7 +655,6 @@ static int powernow_cpu_exit(struct cpufreq_policy *policy) #endif kfree(powernow_table); - return 0; } static struct cpufreq_driver powernow_driver = { diff --git a/drivers/cpufreq/powernow-k8.c b/drivers/cpufreq/powernow-k8.c index 508bd6f68f1e..1260d4023a59 100644 --- a/drivers/cpufreq/powernow-k8.c +++ b/drivers/cpufreq/powernow-k8.c @@ -1090,13 +1090,13 @@ static int powernowk8_cpu_init(struct cpufreq_policy *pol) return -ENODEV; } -static int powernowk8_cpu_exit(struct cpufreq_policy *pol) +static void powernowk8_cpu_exit(struct cpufreq_policy *pol) { struct powernow_k8_data *data = per_cpu(powernow_data, pol->cpu); int cpu; if (!data) - return -EINVAL; + return; powernow_k8_cpu_exit_acpi(data); @@ -1105,8 +1105,6 @@ static int powernowk8_cpu_exit(struct cpufreq_policy *pol) /* pol->cpus will be empty here, use related_cpus instead. */ for_each_cpu(cpu, pol->related_cpus) per_cpu(powernow_data, cpu) = NULL; - - return 0; } static void query_values_on_cpu(void *_err) diff --git a/drivers/cpufreq/powernv-cpufreq.c b/drivers/cpufreq/powernv-cpufreq.c index fddbd1ea1635..50c62929f7ca 100644 --- a/drivers/cpufreq/powernv-cpufreq.c +++ b/drivers/cpufreq/powernv-cpufreq.c @@ -874,7 +874,7 @@ static int powernv_cpufreq_cpu_init(struct cpufreq_policy *policy) return 0; } -static int powernv_cpufreq_cpu_exit(struct cpufreq_policy *policy) +static void powernv_cpufreq_cpu_exit(struct cpufreq_policy *policy) { struct powernv_smp_call_data freq_data; struct global_pstate_info *gpstates = policy->driver_data; @@ -886,8 +886,6 @@ static int powernv_cpufreq_cpu_exit(struct cpufreq_policy *policy) del_timer_sync(&gpstates->timer); kfree(policy->driver_data); - - return 0; } static int powernv_cpufreq_reboot_notifier(struct notifier_block *nb, diff --git a/drivers/cpufreq/ppc_cbe_cpufreq.c b/drivers/cpufreq/ppc_cbe_cpufreq.c index 88afc49941b7..5ee4c7bfdcc5 100644 --- a/drivers/cpufreq/ppc_cbe_cpufreq.c +++ b/drivers/cpufreq/ppc_cbe_cpufreq.c @@ -113,10 +113,9 @@ static int cbe_cpufreq_cpu_init(struct cpufreq_policy *policy) return 0; } -static int cbe_cpufreq_cpu_exit(struct cpufreq_policy *policy) +static void cbe_cpufreq_cpu_exit(struct cpufreq_policy *policy) { cbe_cpufreq_pmi_policy_exit(policy); - return 0; } static int cbe_cpufreq_target(struct cpufreq_policy *policy, diff --git a/drivers/cpufreq/qcom-cpufreq-hw.c b/drivers/cpufreq/qcom-cpufreq-hw.c index 92c0fc4a3cf6..692673447a2e 100644 --- a/drivers/cpufreq/qcom-cpufreq-hw.c +++ b/drivers/cpufreq/qcom-cpufreq-hw.c @@ -574,7 +574,7 @@ static int qcom_cpufreq_hw_cpu_init(struct cpufreq_policy *policy) return qcom_cpufreq_hw_lmh_init(policy, index); } -static int qcom_cpufreq_hw_cpu_exit(struct cpufreq_policy *policy) +static void qcom_cpufreq_hw_cpu_exit(struct cpufreq_policy *policy) { struct device *cpu_dev = get_cpu_device(policy->cpu); struct qcom_cpufreq_data *data = policy->driver_data; @@ -583,9 +583,6 @@ static int qcom_cpufreq_hw_cpu_exit(struct cpufreq_policy *policy) dev_pm_opp_of_cpumask_remove_table(policy->related_cpus); qcom_cpufreq_hw_lmh_exit(data); kfree(policy->freq_table); - kfree(data); - - return 0; } static void qcom_cpufreq_ready(struct cpufreq_policy *policy) diff --git a/drivers/cpufreq/qoriq-cpufreq.c b/drivers/cpufreq/qoriq-cpufreq.c index 0aecaecbb0e6..3519bf34d397 100644 --- a/drivers/cpufreq/qoriq-cpufreq.c +++ b/drivers/cpufreq/qoriq-cpufreq.c @@ -225,7 +225,7 @@ static int qoriq_cpufreq_cpu_init(struct cpufreq_policy *policy) return -ENODEV; } -static int qoriq_cpufreq_cpu_exit(struct cpufreq_policy *policy) +static void qoriq_cpufreq_cpu_exit(struct cpufreq_policy *policy) { struct cpu_data *data = policy->driver_data; @@ -233,8 +233,6 @@ static int qoriq_cpufreq_cpu_exit(struct cpufreq_policy *policy) kfree(data->table); kfree(data); policy->driver_data = NULL; - - return 0; } static int qoriq_cpufreq_target(struct cpufreq_policy *policy, diff --git a/drivers/cpufreq/scmi-cpufreq.c b/drivers/cpufreq/scmi-cpufreq.c index 68325ebd56fe..27e5d887d131 100644 --- a/drivers/cpufreq/scmi-cpufreq.c +++ b/drivers/cpufreq/scmi-cpufreq.c @@ -247,7 +247,7 @@ static int scmi_cpufreq_init(struct cpufreq_policy *policy) return ret; } -static int scmi_cpufreq_exit(struct cpufreq_policy *policy) +static void scmi_cpufreq_exit(struct cpufreq_policy *policy) { struct scmi_data *priv = policy->driver_data; @@ -255,8 +255,6 @@ static int scmi_cpufreq_exit(struct cpufreq_policy *policy) dev_pm_opp_remove_all_dynamic(priv->cpu_dev); free_cpumask_var(priv->opp_shared_cpus); kfree(priv); - - return 0; } static void scmi_cpufreq_register_em(struct cpufreq_policy *policy) diff --git a/drivers/cpufreq/scpi-cpufreq.c b/drivers/cpufreq/scpi-cpufreq.c index 2aef39bff7d6..a191d9bdf667 100644 --- a/drivers/cpufreq/scpi-cpufreq.c +++ b/drivers/cpufreq/scpi-cpufreq.c @@ -175,7 +175,7 @@ static int scpi_cpufreq_init(struct cpufreq_policy *policy) return ret; } -static int scpi_cpufreq_exit(struct cpufreq_policy *policy) +static void scpi_cpufreq_exit(struct cpufreq_policy *policy) { struct scpi_data *priv = policy->driver_data; @@ -183,8 +183,6 @@ static int scpi_cpufreq_exit(struct cpufreq_policy *policy) dev_pm_opp_free_cpufreq_table(priv->cpu_dev, &policy->freq_table); dev_pm_opp_remove_all_dynamic(priv->cpu_dev); kfree(priv); - - return 0; } static struct cpufreq_driver scpi_cpufreq_driver = { diff --git a/drivers/cpufreq/sh-cpufreq.c b/drivers/cpufreq/sh-cpufreq.c index b8704232c27b..aa74036d0420 100644 --- a/drivers/cpufreq/sh-cpufreq.c +++ b/drivers/cpufreq/sh-cpufreq.c @@ -135,14 +135,12 @@ static int sh_cpufreq_cpu_init(struct cpufreq_policy *policy) return 0; } -static int sh_cpufreq_cpu_exit(struct cpufreq_policy *policy) +static void sh_cpufreq_cpu_exit(struct cpufreq_policy *policy) { unsigned int cpu = policy->cpu; struct clk *cpuclk = &per_cpu(sh_cpuclk, cpu); clk_put(cpuclk); - - return 0; } static struct cpufreq_driver sh_cpufreq_driver = { diff --git a/drivers/cpufreq/sparc-us2e-cpufreq.c b/drivers/cpufreq/sparc-us2e-cpufreq.c index 2783d3d55fce..8a0cd5312a59 100644 --- a/drivers/cpufreq/sparc-us2e-cpufreq.c +++ b/drivers/cpufreq/sparc-us2e-cpufreq.c @@ -296,10 +296,9 @@ static int us2e_freq_cpu_init(struct cpufreq_policy *policy) return 0; } -static int us2e_freq_cpu_exit(struct cpufreq_policy *policy) +static void us2e_freq_cpu_exit(struct cpufreq_policy *policy) { us2e_freq_target(policy, 0); - return 0; } static struct cpufreq_driver cpufreq_us2e_driver = { diff --git a/drivers/cpufreq/sparc-us3-cpufreq.c b/drivers/cpufreq/sparc-us3-cpufreq.c index 6c3657679a88..b50f9d13e6d2 100644 --- a/drivers/cpufreq/sparc-us3-cpufreq.c +++ b/drivers/cpufreq/sparc-us3-cpufreq.c @@ -140,10 +140,9 @@ static int us3_freq_cpu_init(struct cpufreq_policy *policy) return 0; } -static int us3_freq_cpu_exit(struct cpufreq_policy *policy) +static void us3_freq_cpu_exit(struct cpufreq_policy *policy) { us3_freq_target(policy, 0); - return 0; } static struct cpufreq_driver cpufreq_us3_driver = { diff --git a/drivers/cpufreq/speedstep-centrino.c b/drivers/cpufreq/speedstep-centrino.c index 75b10ecdb60f..0eb51fb73a80 100644 --- a/drivers/cpufreq/speedstep-centrino.c +++ b/drivers/cpufreq/speedstep-centrino.c @@ -400,16 +400,12 @@ static int centrino_cpu_init(struct cpufreq_policy *policy) return 0; } -static int centrino_cpu_exit(struct cpufreq_policy *policy) +static void centrino_cpu_exit(struct cpufreq_policy *policy) { unsigned int cpu = policy->cpu; - if (!per_cpu(centrino_model, cpu)) - return -ENODEV; - - per_cpu(centrino_model, cpu) = NULL; - - return 0; + if (per_cpu(centrino_model, cpu)) + per_cpu(centrino_model, cpu) = NULL; } /** diff --git a/drivers/cpufreq/tegra194-cpufreq.c b/drivers/cpufreq/tegra194-cpufreq.c index 386aed3637b4..454767c2ad6b 100644 --- a/drivers/cpufreq/tegra194-cpufreq.c +++ b/drivers/cpufreq/tegra194-cpufreq.c @@ -526,14 +526,12 @@ static int tegra194_cpufreq_offline(struct cpufreq_policy *policy) return 0; } -static int tegra194_cpufreq_exit(struct cpufreq_policy *policy) +static void tegra194_cpufreq_exit(struct cpufreq_policy *policy) { struct device *cpu_dev = get_cpu_device(policy->cpu); dev_pm_opp_remove_all_dynamic(cpu_dev); dev_pm_opp_of_cpumask_remove_table(policy->related_cpus); - - return 0; } static int tegra194_cpufreq_set_target(struct cpufreq_policy *policy, diff --git a/drivers/cpufreq/vexpress-spc-cpufreq.c b/drivers/cpufreq/vexpress-spc-cpufreq.c index 9ac4ea50b874..3fadf536c429 100644 --- a/drivers/cpufreq/vexpress-spc-cpufreq.c +++ b/drivers/cpufreq/vexpress-spc-cpufreq.c @@ -447,7 +447,7 @@ static int ve_spc_cpufreq_init(struct cpufreq_policy *policy) return 0; } -static int ve_spc_cpufreq_exit(struct cpufreq_policy *policy) +static void ve_spc_cpufreq_exit(struct cpufreq_policy *policy) { struct device *cpu_dev; @@ -455,11 +455,10 @@ static int ve_spc_cpufreq_exit(struct cpufreq_policy *policy) if (!cpu_dev) { pr_err("%s: failed to get cpu%d device\n", __func__, policy->cpu); - return -ENODEV; + return; } put_cluster_clk_and_freq_table(cpu_dev, policy->related_cpus); - return 0; } static struct cpufreq_driver ve_spc_cpufreq_driver = { diff --git a/drivers/crypto/hisilicon/qm.c b/drivers/crypto/hisilicon/qm.c index c7f874570383..b7d918b3f7cf 100644 --- a/drivers/crypto/hisilicon/qm.c +++ b/drivers/crypto/hisilicon/qm.c @@ -4666,6 +4666,15 @@ void hisi_qm_reset_done(struct pci_dev *pdev) } EXPORT_SYMBOL_GPL(hisi_qm_reset_done); +static irqreturn_t qm_rsvd_irq(int irq, void *data) +{ + struct hisi_qm *qm = data; + + dev_info(&qm->pdev->dev, "Reserved interrupt, ignore!\n"); + + return IRQ_HANDLED; +} + static irqreturn_t qm_abnormal_irq(int irq, void *data) { struct hisi_qm *qm = data; @@ -4967,7 +4976,7 @@ static void qm_unregister_abnormal_irq(struct hisi_qm *qm) struct pci_dev *pdev = qm->pdev; u32 irq_vector, val; - if (qm->fun_type == QM_HW_VF) + if (qm->fun_type == QM_HW_VF && qm->ver < QM_HW_V3) return; val = qm->cap_tables.qm_cap_table[QM_ABN_IRQ_TYPE_CAP_IDX].cap_val; @@ -4984,17 +4993,28 @@ static int qm_register_abnormal_irq(struct hisi_qm *qm) u32 irq_vector, val; int ret; - if (qm->fun_type == QM_HW_VF) - return 0; - val = qm->cap_tables.qm_cap_table[QM_ABN_IRQ_TYPE_CAP_IDX].cap_val; if (!((val >> QM_IRQ_TYPE_SHIFT) & QM_ABN_IRQ_TYPE_MASK)) return 0; - irq_vector = val & QM_IRQ_VECTOR_MASK; + + /* For VF, this is a reserved interrupt in V3 version. */ + if (qm->fun_type == QM_HW_VF) { + if (qm->ver < QM_HW_V3) + return 0; + + ret = request_irq(pci_irq_vector(pdev, irq_vector), qm_rsvd_irq, + IRQF_NO_AUTOEN, qm->dev_name, qm); + if (ret) { + dev_err(&pdev->dev, "failed to request reserved irq, ret = %d!\n", ret); + return ret; + } + return 0; + } + ret = request_irq(pci_irq_vector(pdev, irq_vector), qm_abnormal_irq, 0, qm->dev_name, qm); if (ret) - dev_err(&qm->pdev->dev, "failed to request abnormal irq, ret = %d", ret); + dev_err(&qm->pdev->dev, "failed to request abnormal irq, ret = %d!\n", ret); return ret; } @@ -5333,6 +5353,12 @@ static int hisi_qm_pci_init(struct hisi_qm *qm) pci_set_master(pdev); num_vec = qm_get_irq_num(qm); + if (!num_vec) { + dev_err(dev, "Device irq num is zero!\n"); + ret = -EINVAL; + goto err_get_pci_res; + } + num_vec = roundup_pow_of_two(num_vec); ret = pci_alloc_irq_vectors(pdev, num_vec, num_vec, PCI_IRQ_MSI); if (ret < 0) { dev_err(dev, "Failed to enable MSI vectors!\n"); diff --git a/drivers/crypto/intel/qat/qat_common/adf_init.c b/drivers/crypto/intel/qat/qat_common/adf_init.c index 9c1b587a87e5..718bedff9c16 100644 --- a/drivers/crypto/intel/qat/qat_common/adf_init.c +++ b/drivers/crypto/intel/qat/qat_common/adf_init.c @@ -140,16 +140,19 @@ static int adf_dev_init(struct adf_accel_dev *accel_dev) * This is to facilitate any ordering dependencies between services * prior to starting any of the accelerators. */ + mutex_lock(&service_lock); list_for_each(list_itr, &service_table) { service = list_entry(list_itr, struct service_hndl, list); if (service->event_hld(accel_dev, ADF_EVENT_INIT)) { dev_err(&GET_DEV(accel_dev), "Failed to initialise service %s\n", service->name); + mutex_unlock(&service_lock); return -EFAULT; } set_bit(accel_dev->accel_id, service->init_status); } + mutex_unlock(&service_lock); return 0; } @@ -212,16 +215,19 @@ static int adf_dev_start(struct adf_accel_dev *accel_dev) adf_heartbeat_start(accel_dev); + mutex_lock(&service_lock); list_for_each(list_itr, &service_table) { service = list_entry(list_itr, struct service_hndl, list); if (service->event_hld(accel_dev, ADF_EVENT_START)) { dev_err(&GET_DEV(accel_dev), "Failed to start service %s\n", service->name); + mutex_unlock(&service_lock); return -EFAULT; } set_bit(accel_dev->accel_id, service->start_status); } + mutex_unlock(&service_lock); clear_bit(ADF_STATUS_STARTING, &accel_dev->status); set_bit(ADF_STATUS_STARTED, &accel_dev->status); @@ -289,6 +295,7 @@ static void adf_dev_stop(struct adf_accel_dev *accel_dev) qat_comp_algs_unregister(); clear_bit(ADF_STATUS_COMP_ALGS_REGISTERED, &accel_dev->status); + mutex_lock(&service_lock); list_for_each(list_itr, &service_table) { service = list_entry(list_itr, struct service_hndl, list); if (!test_bit(accel_dev->accel_id, service->start_status)) @@ -301,6 +308,7 @@ static void adf_dev_stop(struct adf_accel_dev *accel_dev) clear_bit(accel_dev->accel_id, service->start_status); } } + mutex_unlock(&service_lock); if (hw_data->stop_timer) hw_data->stop_timer(accel_dev); @@ -349,6 +357,7 @@ static void adf_dev_shutdown(struct adf_accel_dev *accel_dev) &accel_dev->status); } + mutex_lock(&service_lock); list_for_each(list_itr, &service_table) { service = list_entry(list_itr, struct service_hndl, list); if (!test_bit(accel_dev->accel_id, service->init_status)) @@ -360,6 +369,7 @@ static void adf_dev_shutdown(struct adf_accel_dev *accel_dev) else clear_bit(accel_dev->accel_id, service->init_status); } + mutex_unlock(&service_lock); adf_heartbeat_shutdown(accel_dev); @@ -390,6 +400,7 @@ int adf_dev_restarting_notify(struct adf_accel_dev *accel_dev) struct service_hndl *service; struct list_head *list_itr; + mutex_lock(&service_lock); list_for_each(list_itr, &service_table) { service = list_entry(list_itr, struct service_hndl, list); if (service->event_hld(accel_dev, ADF_EVENT_RESTARTING)) @@ -397,6 +408,7 @@ int adf_dev_restarting_notify(struct adf_accel_dev *accel_dev) "Failed to restart service %s.\n", service->name); } + mutex_unlock(&service_lock); return 0; } @@ -405,6 +417,7 @@ int adf_dev_restarted_notify(struct adf_accel_dev *accel_dev) struct service_hndl *service; struct list_head *list_itr; + mutex_lock(&service_lock); list_for_each(list_itr, &service_table) { service = list_entry(list_itr, struct service_hndl, list); if (service->event_hld(accel_dev, ADF_EVENT_RESTARTED)) @@ -412,6 +425,7 @@ int adf_dev_restarted_notify(struct adf_accel_dev *accel_dev) "Failed to restart service %s.\n", service->name); } + mutex_unlock(&service_lock); return 0; } diff --git a/drivers/cxl/core/core.h b/drivers/cxl/core/core.h index 6444cc827c9c..3f05fa7c65a7 100644 --- a/drivers/cxl/core/core.h +++ b/drivers/cxl/core/core.h @@ -28,6 +28,7 @@ int cxl_region_init(void); void cxl_region_exit(void); int cxl_get_poison_by_endpoint(struct cxl_port *port); struct cxl_region *cxl_dpa_to_region(const struct cxl_memdev *cxlmd, u64 dpa); +void kill_regions(struct cxl_root_decoder *cxlrd); #else static inline @@ -42,6 +43,7 @@ static inline int cxl_get_poison_by_endpoint(struct cxl_port *port) static inline void cxl_decoder_kill_region(struct cxl_endpoint_decoder *cxled) { } +static inline void kill_regions(struct cxl_root_decoder *cxlrd) { }; static inline int cxl_region_init(void) { return 0; diff --git a/drivers/cxl/core/mbox.c b/drivers/cxl/core/mbox.c index 059e95e08620..14e0354db65e 100644 --- a/drivers/cxl/core/mbox.c +++ b/drivers/cxl/core/mbox.c @@ -306,6 +306,7 @@ static bool cxl_mem_raw_command_allowed(u16 opcode) * cxl_payload_from_user_allowed() - Check contents of in_payload. * @opcode: The mailbox command opcode. * @payload_in: Pointer to the input payload passed in from user space. + * @in_size: Size of @payload_in in bytes. * * Return: * * true - payload_in passes check for @opcode. @@ -320,12 +321,15 @@ static bool cxl_mem_raw_command_allowed(u16 opcode) * * The specific checks are determined by the opcode. */ -static bool cxl_payload_from_user_allowed(u16 opcode, void *payload_in) +static bool cxl_payload_from_user_allowed(u16 opcode, void *payload_in, + size_t in_size) { switch (opcode) { case CXL_MBOX_OP_SET_PARTITION_INFO: { struct cxl_mbox_set_partition_info *pi = payload_in; + if (in_size < sizeof(*pi)) + return false; if (pi->flags & CXL_SET_PARTITION_IMMEDIATE_FLAG) return false; break; @@ -351,7 +355,8 @@ static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox, if (IS_ERR(mbox->payload_in)) return PTR_ERR(mbox->payload_in); - if (!cxl_payload_from_user_allowed(opcode, mbox->payload_in)) { + if (!cxl_payload_from_user_allowed(opcode, mbox->payload_in, + in_size)) { dev_dbg(mds->cxlds.dev, "%s: input payload not allowed\n", cxl_mem_opcode_to_name(opcode)); kvfree(mbox->payload_in); @@ -359,11 +364,7 @@ static int cxl_mbox_cmd_ctor(struct cxl_mbox_cmd *mbox, } } - /* Prepare to handle a full payload for variable sized output */ - if (out_size == CXL_VARIABLE_PAYLOAD) - mbox->size_out = mds->payload_size; - else - mbox->size_out = out_size; + mbox->size_out = min_t(size_t, out_size, mds->payload_size); if (mbox->size_out) { mbox->payload_out = kvzalloc(mbox->size_out, GFP_KERNEL); diff --git a/drivers/cxl/core/pci.c b/drivers/cxl/core/pci.c index 6edfd0546673..e89d4a982715 100644 --- a/drivers/cxl/core/pci.c +++ b/drivers/cxl/core/pci.c @@ -11,6 +11,10 @@ #include "core.h" #include "trace.h" +/* Check that UCE header definition is maintained to keep ABI intact */ +static_assert(CXL_HEADERLOG_TRACE_SIZE_U32 == 128, + "rasdaemon ABI requires exactly 128 u32s"); + /** * DOC: cxl core pci * @@ -663,12 +667,12 @@ static void header_log_copy(struct cxl_dev_state *cxlds, u32 *log) { void __iomem *addr; u32 *log_addr; - int i, log_u32_size = CXL_HEADERLOG_SIZE / sizeof(u32); + int i; addr = cxlds->regs.ras + CXL_RAS_HEADER_LOG_OFFSET; log_addr = log; - for (i = 0; i < log_u32_size; i++) { + for (i = 0; i < CXL_HEADERLOG_SIZE_U32; i++) { *log_addr = readl(addr); log_addr++; addr += sizeof(u32); @@ -681,7 +685,7 @@ static void header_log_copy(struct cxl_dev_state *cxlds, u32 *log) */ static bool cxl_report_and_clear(struct cxl_dev_state *cxlds) { - u32 hl[CXL_HEADERLOG_SIZE_U32]; + u32 hl[CXL_HEADERLOG_TRACE_SIZE_U32] = {}; void __iomem *addr; u32 status; u32 fe; diff --git a/drivers/cxl/core/port.c b/drivers/cxl/core/port.c index 7f28d1021fa9..2589354b0ca4 100644 --- a/drivers/cxl/core/port.c +++ b/drivers/cxl/core/port.c @@ -433,6 +433,8 @@ static void cxl_root_decoder_release(struct device *dev) if (atomic_read(&cxlrd->region_id) >= 0) memregion_free(atomic_read(&cxlrd->region_id)); + mutex_destroy(&cxlrd->regions_lock); + xa_destroy(&cxlrd->regions); __cxl_decoder_release(&cxlrd->cxlsd.cxld); kfree(cxlrd); } @@ -527,6 +529,9 @@ static void cxl_port_release(struct device *dev) xa_destroy(&port->dports); xa_destroy(&port->regions); ida_free(&cxl_port_ida, port->id); + + if (!is_cxl_root(port)) + put_device(dev->parent); kfree(port); } @@ -657,6 +662,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport_dev, struct cxl_port *iter; dev->parent = &parent_port->dev; + get_device(dev->parent); port->depth = parent_port->depth + 1; port->parent_dport = parent_dport; @@ -1380,8 +1386,8 @@ static void cxl_detach_ep(void *data) dev_name(&port->dev)); delete_switch_port(port); } - put_device(&port->dev); device_unlock(&parent_port->dev); + put_device(&port->dev); } } @@ -1703,7 +1709,8 @@ struct cxl_root_decoder *cxl_root_decoder_alloc(struct cxl_port *port, } cxlrd->calc_hb = calc_hb; - mutex_init(&cxlrd->range_lock); + mutex_init(&cxlrd->regions_lock); + xa_init(&cxlrd->regions); cxld = &cxlsd->cxld; cxld->dev.type = &cxl_decoder_root_type; @@ -1890,6 +1897,8 @@ static void cxld_unregister(void *dev) cxled = to_cxl_endpoint_decoder(dev); cxl_decoder_kill_region(cxled); } + if (is_root_decoder(dev)) + kill_regions(to_cxl_root_decoder(dev)); device_unregister(dev); } diff --git a/drivers/cxl/core/region.c b/drivers/cxl/core/region.c index 97e4a4755ab2..873ee24a3431 100644 --- a/drivers/cxl/core/region.c +++ b/drivers/cxl/core/region.c @@ -2193,13 +2193,14 @@ static struct cxl_region *to_cxl_region(struct device *dev) return container_of(dev, struct cxl_region, dev); } -static void unregister_region(void *dev) +static void unregister_region(struct cxl_region *cxlr) { - struct cxl_region *cxlr = to_cxl_region(dev); + struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(cxlr->dev.parent); struct cxl_region_params *p = &cxlr->params; int i; - device_del(dev); + xa_erase(&cxlrd->regions, cxlr->id); + device_del(&cxlr->dev); /* * Now that region sysfs is shutdown, the parameter block is now @@ -2210,7 +2211,7 @@ static void unregister_region(void *dev) detach_target(cxlr, i); cxl_region_iomem_release(cxlr); - put_device(dev); + put_device(&cxlr->dev); } static struct lock_class_key cxl_region_key; @@ -2243,6 +2244,19 @@ static struct cxl_region *cxl_region_alloc(struct cxl_root_decoder *cxlrd, int i return cxlr; } +/* unwind all remaining regions */ +void kill_regions(struct cxl_root_decoder *cxlrd) +{ + unsigned long index; + struct cxl_region *cxlr; + + guard(mutex)(&cxlrd->regions_lock); + /* no more region creation */ + cxlrd->dead = true; + xa_for_each(&cxlrd->regions, index, cxlr) + unregister_region(cxlr); +} + /** * devm_cxl_add_region - Adds a region to a decoder * @cxlrd: root decoder @@ -2281,9 +2295,11 @@ static struct cxl_region *devm_cxl_add_region(struct cxl_root_decoder *cxlrd, if (rc) goto err; - rc = devm_add_action_or_reset(port->uport_dev, unregister_region, cxlr); - if (rc) + rc = xa_insert(&cxlrd->regions, cxlr->id, cxlr, GFP_KERNEL); + if (rc) { + unregister_region(cxlr); return ERR_PTR(rc); + } dev_dbg(port->uport_dev, "%s: created %s\n", dev_name(&cxlrd->cxlsd.cxld.dev), dev_name(dev)); @@ -2316,6 +2332,9 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd, { int rc; + if (cxlrd->dead) + return ERR_PTR(-ENXIO); + switch (mode) { case CXL_DECODER_RAM: case CXL_DECODER_PMEM: @@ -2349,7 +2368,12 @@ static ssize_t create_pmem_region_store(struct device *dev, if (rc != 1) return -EINVAL; + rc = mutex_lock_interruptible(&cxlrd->regions_lock); + if (rc) + return rc; + cxlr = __create_region(cxlrd, CXL_DECODER_PMEM, id); + mutex_unlock(&cxlrd->regions_lock); if (IS_ERR(cxlr)) return PTR_ERR(cxlr); @@ -2369,7 +2393,12 @@ static ssize_t create_ram_region_store(struct device *dev, if (rc != 1) return -EINVAL; + rc = mutex_lock_interruptible(&cxlrd->regions_lock); + if (rc) + return rc; + cxlr = __create_region(cxlrd, CXL_DECODER_RAM, id); + mutex_unlock(&cxlrd->regions_lock); if (IS_ERR(cxlr)) return PTR_ERR(cxlr); @@ -2397,33 +2426,32 @@ static ssize_t region_show(struct device *dev, struct device_attribute *attr, } DEVICE_ATTR_RO(region); -static struct cxl_region * -cxl_find_region_by_name(struct cxl_root_decoder *cxlrd, const char *name) -{ - struct cxl_decoder *cxld = &cxlrd->cxlsd.cxld; - struct device *region_dev; - - region_dev = device_find_child_by_name(&cxld->dev, name); - if (!region_dev) - return ERR_PTR(-ENODEV); - - return to_cxl_region(region_dev); -} - static ssize_t delete_region_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t len) { struct cxl_root_decoder *cxlrd = to_cxl_root_decoder(dev); - struct cxl_port *port = to_cxl_port(dev->parent); struct cxl_region *cxlr; + int rc, id; - cxlr = cxl_find_region_by_name(cxlrd, buf); - if (IS_ERR(cxlr)) - return PTR_ERR(cxlr); + rc = mutex_lock_interruptible(&cxlrd->regions_lock); + if (rc) + return rc; - devm_release_action(port->uport_dev, unregister_region, cxlr); - put_device(&cxlr->dev); + rc = sscanf(buf, "region%d\n", &id); + if (rc != 1) { + mutex_unlock(&cxlrd->regions_lock); + return -EINVAL; + } + + cxlr = xa_load(&cxlrd->regions, id); + if (!cxlr || !sysfs_streq(buf, dev_name(&cxlr->dev))) { + mutex_unlock(&cxlrd->regions_lock); + return -ENODEV; + } + + unregister_region(cxlr); + mutex_unlock(&cxlrd->regions_lock); return len; } @@ -2940,7 +2968,6 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, struct cxl_endpoint_decoder *cxled) { struct cxl_memdev *cxlmd = cxled_to_memdev(cxled); - struct cxl_port *port = cxlrd_to_port(cxlrd); struct range *hpa = &cxled->cxld.hpa_range; struct cxl_region_params *p; struct cxl_region *cxlr; @@ -3015,7 +3042,7 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd, err: up_write(&cxl_region_rwsem); - devm_release_action(port->uport_dev, unregister_region, cxlr); + unregister_region(cxlr); return ERR_PTR(rc); } @@ -3047,7 +3074,7 @@ int cxl_add_to_region(struct cxl_port *root, struct cxl_endpoint_decoder *cxled) * Ensure that if multiple threads race to construct_region() for @hpa * one does the construction and the others add to that. */ - mutex_lock(&cxlrd->range_lock); + guard(mutex)(&cxlrd->regions_lock); region_dev = device_find_child(&cxlrd->cxlsd.cxld.dev, hpa, match_region_by_range); if (!region_dev) { @@ -3055,7 +3082,6 @@ int cxl_add_to_region(struct cxl_port *root, struct cxl_endpoint_decoder *cxled) region_dev = &cxlr->dev; } else cxlr = to_cxl_region(region_dev); - mutex_unlock(&cxlrd->range_lock); rc = PTR_ERR_OR_ZERO(cxlr); if (rc) diff --git a/drivers/cxl/core/trace.h b/drivers/cxl/core/trace.h index 1e1e50d39bf6..54b72f943744 100644 --- a/drivers/cxl/core/trace.h +++ b/drivers/cxl/core/trace.h @@ -57,7 +57,7 @@ TRACE_EVENT(cxl_aer_uncorrectable_error, __field(u64, serial) __field(u32, status) __field(u32, first_error) - __array(u32, header_log, CXL_HEADERLOG_SIZE_U32) + __array(u32, header_log, CXL_HEADERLOG_TRACE_SIZE_U32) ), TP_fast_assign( __assign_str(memdev, dev_name(&cxlmd->dev)); @@ -66,10 +66,14 @@ TRACE_EVENT(cxl_aer_uncorrectable_error, __entry->status = status; __entry->first_error = fe; /* - * Embed the 512B headerlog data for user app retrieval and - * parsing, but no need to print this in the trace buffer. + * Embed headerlog data for user app retrieval and parsing, + * but no need to print in the trace buffer. Only + * CXL_HEADERLOG_SIZE_U32 (16) dwords are hardware data; + * the remaining entries preserve the 512-byte ABI layout + * rasdaemon depends on and are zero-filled by the caller. */ - memcpy(__entry->header_log, hl, CXL_HEADERLOG_SIZE); + memcpy(__entry->header_log, hl, + CXL_HEADERLOG_TRACE_SIZE_U32 * sizeof(u32)); ), TP_printk("memdev=%s host=%s serial=%lld: status: '%s' first_error: '%s'", __get_str(memdev), __get_str(host), __entry->serial, diff --git a/drivers/cxl/cxl.h b/drivers/cxl/cxl.h index 59e963e39d31..7a1b1721860d 100644 --- a/drivers/cxl/cxl.h +++ b/drivers/cxl/cxl.h @@ -143,8 +143,18 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw) #define CXL_RAS_CAP_CONTROL_FE_MASK GENMASK(5, 0) #define CXL_RAS_HEADER_LOG_OFFSET 0x18 #define CXL_RAS_CAPABILITY_LENGTH 0x58 -#define CXL_HEADERLOG_SIZE SZ_512 -#define CXL_HEADERLOG_SIZE_U32 SZ_512 / sizeof(u32) +#define CXL_HEADERLOG_SIZE SZ_64 +#define CXL_HEADERLOG_SIZE_U32 (CXL_HEADERLOG_SIZE / sizeof(u32)) + +/* + * The RAS UCE trace event header array was originally sized at SZ_512/sizeof(u32) + * = 128 u32s due to a bug. Userspace tools (rasdaemon) have grown a dependency + * on that 512-byte layout. Keep the trace array at 128 u32s to preserve the + * ABI; only CXL_HEADERLOG_SIZE_U32 (16) dwords are valid hardware data, the + * remainder are zero-filled. + */ +#define CXL_HEADERLOG_TRACE_SIZE SZ_512 +#define CXL_HEADERLOG_TRACE_SIZE_U32 (CXL_HEADERLOG_TRACE_SIZE / sizeof(u32)) /* CXL 2.0 8.2.8.1 Device Capabilities Array Register */ #define CXLDEV_CAP_ARRAY_OFFSET 0x0 @@ -431,7 +441,9 @@ typedef struct cxl_dport *(*cxl_calc_hb_fn)(struct cxl_root_decoder *cxlrd, * @region_id: region id for next region provisioning event * @calc_hb: which host bridge covers the n'th position by granularity * @platform_data: platform specific configuration data - * @range_lock: sync region autodiscovery by address range + * @regions_lock: sync region discovery, construction, and deletion + * @regions: regions to remove at root decoder destruct time + * @dead: root decoder dead to region creation * @cxlsd: base cxl switch decoder */ struct cxl_root_decoder { @@ -439,7 +451,9 @@ struct cxl_root_decoder { atomic_t region_id; cxl_calc_hb_fn calc_hb; void *platform_data; - struct mutex range_lock; + struct mutex regions_lock; + struct xarray regions; + bool dead; struct cxl_switch_decoder cxlsd; }; diff --git a/drivers/dma-buf/dma-buf.c b/drivers/dma-buf/dma-buf.c index 21916bba77d5..47783225ed02 100644 --- a/drivers/dma-buf/dma-buf.c +++ b/drivers/dma-buf/dma-buf.c @@ -714,6 +714,21 @@ int dma_buf_fd(struct dma_buf *dmabuf, int flags) } EXPORT_SYMBOL_NS_GPL(dma_buf_fd, DMA_BUF); +/** + * dma_buf_fd_install - install a reserved fd for a dma-buf + * @dmabuf: [in] pointer to dma_buf + * @fd: [in] fd reserved with get_unused_fd_flags() + * + * Publishes a previously reserved fd into the caller's fd table. + * Must only be called after all fallible work (e.g. copy_to_user) + * has succeeded, as it cannot be undone safely once called. + */ +void dma_buf_fd_install(struct dma_buf *dmabuf, int fd) +{ + fd_install(fd, dmabuf->file); +} +EXPORT_SYMBOL_NS_GPL(dma_buf_fd_install, DMA_BUF); + /** * dma_buf_get - returns the struct dma_buf related to an fd * @fd: [in] fd associated with the struct dma_buf to be returned diff --git a/drivers/dma-buf/dma-heap.c b/drivers/dma-buf/dma-heap.c index 84ae708fafe7..baeb3b629ead 100644 --- a/drivers/dma-buf/dma-heap.c +++ b/drivers/dma-buf/dma-heap.c @@ -49,33 +49,6 @@ static dev_t dma_heap_devt; static struct class *dma_heap_class; static DEFINE_XARRAY_ALLOC(dma_heap_minors); -static int dma_heap_buffer_alloc(struct dma_heap *heap, size_t len, - unsigned int fd_flags, - unsigned int heap_flags) -{ - struct dma_buf *dmabuf; - int fd; - - /* - * Allocations from all heaps have to begin - * and end on page boundaries. - */ - len = PAGE_ALIGN(len); - if (!len) - return -EINVAL; - - dmabuf = heap->ops->allocate(heap, len, fd_flags, heap_flags); - if (IS_ERR(dmabuf)) - return PTR_ERR(dmabuf); - - fd = dma_buf_fd(dmabuf, fd_flags); - if (fd < 0) { - dma_buf_put(dmabuf); - /* just return, as put will call release and that will free */ - } - return fd; -} - static int dma_heap_open(struct inode *inode, struct file *file) { struct dma_heap *heap; @@ -93,30 +66,42 @@ static int dma_heap_open(struct inode *inode, struct file *file) return 0; } -static long dma_heap_ioctl_allocate(struct file *file, void *data) +static struct dma_buf *dma_heap_ioctl_allocate(struct file *file, void *data) { struct dma_heap_allocation_data *heap_allocation = data; struct dma_heap *heap = file->private_data; + struct dma_buf *dmabuf; int fd; + size_t len; if (heap_allocation->fd) - return -EINVAL; + return ERR_PTR(-EINVAL); if (heap_allocation->fd_flags & ~DMA_HEAP_VALID_FD_FLAGS) - return -EINVAL; + return ERR_PTR(-EINVAL); if (heap_allocation->heap_flags & ~DMA_HEAP_VALID_HEAP_FLAGS) - return -EINVAL; + return ERR_PTR(-EINVAL); + + len = PAGE_ALIGN(heap_allocation->len); + if (!len) + return ERR_PTR(-EINVAL); + + dmabuf = heap->ops->allocate(heap, len, heap_allocation->fd_flags, + heap_allocation->heap_flags); - fd = dma_heap_buffer_alloc(heap, heap_allocation->len, - heap_allocation->fd_flags, - heap_allocation->heap_flags); - if (fd < 0) - return fd; + if (IS_ERR(dmabuf)) + return dmabuf; + + fd = get_unused_fd_flags(heap_allocation->fd_flags); + if (fd < 0) { + dma_buf_put(dmabuf); + return ERR_PTR(fd); + } heap_allocation->fd = fd; - return 0; + return dmabuf; } static unsigned int dma_heap_ioctl_cmds[] = { @@ -132,6 +117,8 @@ static long dma_heap_ioctl(struct file *file, unsigned int ucmd, unsigned int in_size, out_size, drv_size, ksize; int nr = _IOC_NR(ucmd); int ret = 0; + int fd; + struct dma_buf *dmabuf; if (nr >= ARRAY_SIZE(dma_heap_ioctl_cmds)) return -EINVAL; @@ -168,15 +155,28 @@ static long dma_heap_ioctl(struct file *file, unsigned int ucmd, switch (kcmd) { case DMA_HEAP_IOCTL_ALLOC: - ret = dma_heap_ioctl_allocate(file, kdata); + dmabuf = dma_heap_ioctl_allocate(file, kdata); + + if (IS_ERR(dmabuf)) { + ret = PTR_ERR(dmabuf); + break; + } + + fd = ((struct dma_heap_allocation_data *)kdata)->fd; + if (copy_to_user((void __user *)arg, kdata, out_size) != 0) { + put_unused_fd(fd); + dma_buf_put(dmabuf); + ret = -EFAULT; + } else { + dma_buf_fd_install(dmabuf, fd); + } + break; default: ret = -ENOTTY; goto err; } - if (copy_to_user((void __user *)arg, kdata, out_size) != 0) - ret = -EFAULT; err: if (kdata != stack_kdata) kfree(kdata); diff --git a/drivers/dma-buf/udmabuf.c b/drivers/dma-buf/udmabuf.c index cdb6c30352ae..ddf8a9a66c0c 100644 --- a/drivers/dma-buf/udmabuf.c +++ b/drivers/dma-buf/udmabuf.c @@ -146,7 +146,7 @@ static void release_udmabuf(struct dma_buf *buf) for (pg = 0; pg < ubuf->pagecount; pg++) put_page(ubuf->pages[pg]); - kfree(ubuf->pages); + kvfree(ubuf->pages); kfree(ubuf); } @@ -233,8 +233,8 @@ static long udmabuf_create(struct miscdevice *device, if (!ubuf->pagecount) goto err; - ubuf->pages = kmalloc_array(ubuf->pagecount, sizeof(*ubuf->pages), - GFP_KERNEL); + ubuf->pages = kvmalloc_array(ubuf->pagecount, sizeof(*ubuf->pages), + GFP_KERNEL); if (!ubuf->pages) { ret = -ENOMEM; goto err; @@ -285,14 +285,24 @@ static long udmabuf_create(struct miscdevice *device, flags = 0; if (head->flags & UDMABUF_FLAGS_CLOEXEC) flags |= O_CLOEXEC; - return dma_buf_fd(buf, flags); + /* + * Ownership of ubuf is held by the dmabuf from here. + * If the following dma_buf_fd() fails, dma_buf_put() cleans up both the + * dmabuf and the ubuf (through udmabuf_ops.release). + */ + + ret = dma_buf_fd(buf, flags); + if (ret < 0) + dma_buf_put(buf); + + return ret; err: while (pgbuf > 0) put_page(ubuf->pages[--pgbuf]); if (memfd) fput(memfd); - kfree(ubuf->pages); + kvfree(ubuf->pages); kfree(ubuf); return ret; } diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c index d4eb8cb5a94b..577613c3dc33 100644 --- a/drivers/dma/dmaengine.c +++ b/drivers/dma/dmaengine.c @@ -933,6 +933,36 @@ void dma_release_channel(struct dma_chan *chan) } EXPORT_SYMBOL_GPL(dma_release_channel); +static void dmaenginem_release_channel(void *chan) +{ + dma_release_channel(chan); +} + +/** + * devm_dma_request_chan - try to allocate an exclusive slave channel + * @dev: pointer to client device structure + * @name: slave channel name + * + * Returns pointer to appropriate DMA channel on success or an error pointer. + * + * The operation is managed and will be undone on driver detach. + */ + +struct dma_chan *devm_dma_request_chan(struct device *dev, const char *name) +{ + struct dma_chan *chan = dma_request_chan(dev, name); + int ret = 0; + + if (!IS_ERR(chan)) + ret = devm_add_action_or_reset(dev, dmaenginem_release_channel, chan); + + if (ret) + return ERR_PTR(ret); + + return chan; +} +EXPORT_SYMBOL_GPL(devm_dma_request_chan); + /** * dmaengine_get - register interest in dma_channels */ diff --git a/drivers/dma/idxd/init.c b/drivers/dma/idxd/init.c index 4b999c5802f4..810f3d9c4f7a 100644 --- a/drivers/dma/idxd/init.c +++ b/drivers/dma/idxd/init.c @@ -147,18 +147,12 @@ static void idxd_cleanup_interrupts(struct idxd_device *idxd) static void idxd_clean_wqs(struct idxd_device *idxd) { - struct idxd_wq *wq; struct device *conf_dev; int i; for (i = 0; i < idxd->max_wqs; i++) { - wq = idxd->wqs[i]; - if (idxd->hw.wq_cap.op_config) - bitmap_free(wq->opcap_bmap); - kfree(wq->wqcfg); - conf_dev = wq_confdev(wq); + conf_dev = wq_confdev(idxd->wqs[i]); put_device(conf_dev); - kfree(wq); } bitmap_free(idxd->wq_enable_map); kfree(idxd->wqs); @@ -200,7 +194,6 @@ static int idxd_setup_wqs(struct idxd_device *idxd) rc = dev_set_name(conf_dev, "wq%d.%d", idxd->id, wq->id); if (rc < 0) { put_device(conf_dev); - kfree(wq); goto err_unwind; } @@ -214,7 +207,6 @@ static int idxd_setup_wqs(struct idxd_device *idxd) wq->wqcfg = kzalloc_node(idxd->wqcfg_size, GFP_KERNEL, dev_to_node(dev)); if (!wq->wqcfg) { put_device(conf_dev); - kfree(wq); rc = -ENOMEM; goto err_unwind; } @@ -222,9 +214,7 @@ static int idxd_setup_wqs(struct idxd_device *idxd) if (idxd->hw.wq_cap.op_config) { wq->opcap_bmap = bitmap_zalloc(IDXD_MAX_OPCAP_BITS, GFP_KERNEL); if (!wq->opcap_bmap) { - kfree(wq->wqcfg); put_device(conf_dev); - kfree(wq); rc = -ENOMEM; goto err_unwind; } @@ -239,13 +229,8 @@ static int idxd_setup_wqs(struct idxd_device *idxd) err_unwind: while (--i >= 0) { - wq = idxd->wqs[i]; - if (idxd->hw.wq_cap.op_config) - bitmap_free(wq->opcap_bmap); - kfree(wq->wqcfg); - conf_dev = wq_confdev(wq); + conf_dev = wq_confdev(idxd->wqs[i]); put_device(conf_dev); - kfree(wq); } bitmap_free(idxd->wq_enable_map); @@ -257,15 +242,12 @@ static int idxd_setup_wqs(struct idxd_device *idxd) static void idxd_clean_engines(struct idxd_device *idxd) { - struct idxd_engine *engine; struct device *conf_dev; int i; for (i = 0; i < idxd->max_engines; i++) { - engine = idxd->engines[i]; - conf_dev = engine_confdev(engine); + conf_dev = engine_confdev(idxd->engines[i]); put_device(conf_dev); - kfree(engine); } kfree(idxd->engines); } @@ -300,7 +282,6 @@ static int idxd_setup_engines(struct idxd_device *idxd) rc = dev_set_name(conf_dev, "engine%d.%d", idxd->id, engine->id); if (rc < 0) { put_device(conf_dev); - kfree(engine); goto err; } @@ -311,10 +292,8 @@ static int idxd_setup_engines(struct idxd_device *idxd) err: while (--i >= 0) { - engine = idxd->engines[i]; - conf_dev = engine_confdev(engine); + conf_dev = engine_confdev(idxd->engines[i]); put_device(conf_dev); - kfree(engine); } kfree(idxd->engines); @@ -323,13 +302,10 @@ static int idxd_setup_engines(struct idxd_device *idxd) static void idxd_clean_groups(struct idxd_device *idxd) { - struct idxd_group *group; int i; for (i = 0; i < idxd->max_groups; i++) { - group = idxd->groups[i]; - put_device(group_confdev(group)); - kfree(group); + put_device(group_confdev(idxd->groups[i])); } kfree(idxd->groups); } @@ -364,7 +340,6 @@ static int idxd_setup_groups(struct idxd_device *idxd) rc = dev_set_name(conf_dev, "group%d.%d", idxd->id, group->id); if (rc < 0) { put_device(conf_dev); - kfree(group); goto err; } @@ -389,7 +364,6 @@ static int idxd_setup_groups(struct idxd_device *idxd) while (--i >= 0) { group = idxd->groups[i]; put_device(group_confdev(group)); - kfree(group); } kfree(idxd->groups); diff --git a/drivers/dma/ti/k3-udma-glue.c b/drivers/dma/ti/k3-udma-glue.c index 72cac5852aa8..ff84ba6d9e9e 100644 --- a/drivers/dma/ti/k3-udma-glue.c +++ b/drivers/dma/ti/k3-udma-glue.c @@ -84,6 +84,7 @@ struct k3_udma_glue_rx_channel { struct k3_udma_glue_rx_flow *flows; u32 flow_num; u32 flows_ready; + bool single_fdq; /* one FDQ for all flows */ }; static void k3_udma_chan_dev_release(struct device *dev) @@ -900,10 +901,13 @@ k3_udma_glue_request_rx_chn_priv(struct device *dev, const char *name, ep_cfg = rx_chn->common.ep_config; - if (xudma_is_pktdma(rx_chn->common.udmax)) + if (xudma_is_pktdma(rx_chn->common.udmax)) { rx_chn->udma_rchan_id = ep_cfg->mapped_channel_id; - else + rx_chn->single_fdq = false; + } else { rx_chn->udma_rchan_id = -1; + rx_chn->single_fdq = true; + } /* request and cfg UDMAP RX channel */ rx_chn->udma_rchanx = xudma_rchan_get(rx_chn->common.udmax, @@ -1064,6 +1068,9 @@ k3_udma_glue_request_remote_rx_chn(struct device *dev, const char *name, rx_chn->common.chan_dev.dma_coherent = true; dma_coerce_mask_and_coherent(&rx_chn->common.chan_dev, DMA_BIT_MASK(48)); + rx_chn->single_fdq = false; + } else { + rx_chn->single_fdq = true; } ret = k3_udma_glue_allocate_rx_flows(rx_chn, cfg); @@ -1325,7 +1332,7 @@ EXPORT_SYMBOL_GPL(k3_udma_glue_tdown_rx_chn); void k3_udma_glue_reset_rx_chn(struct k3_udma_glue_rx_channel *rx_chn, u32 flow_num, void *data, - void (*cleanup)(void *data, dma_addr_t desc_dma), bool skip_fdq) + void (*cleanup)(void *data, dma_addr_t desc_dma)) { struct k3_udma_glue_rx_flow *flow = &rx_chn->flows[flow_num]; struct device *dev = rx_chn->common.dev; @@ -1337,7 +1344,7 @@ void k3_udma_glue_reset_rx_chn(struct k3_udma_glue_rx_channel *rx_chn, dev_dbg(dev, "RX reset flow %u occ_rx %u\n", flow_num, occ_rx); /* Skip RX FDQ in case one FDQ is used for the set of flows */ - if (skip_fdq) + if (rx_chn->single_fdq && flow_num) goto do_reset; /* diff --git a/drivers/dma/xilinx/xilinx_dma.c b/drivers/dma/xilinx/xilinx_dma.c index 1e1d8829cd81..ee15da19515a 100644 --- a/drivers/dma/xilinx/xilinx_dma.c +++ b/drivers/dma/xilinx/xilinx_dma.c @@ -164,6 +164,8 @@ /* Delay loop counter to prevent hardware failure */ #define XILINX_DMA_LOOP_COUNT 1000000 +/* Delay between polls (avoid a delay of 0 to prevent CPU stalls) */ +#define XILINX_DMA_POLL_DELAY_US 10 /* AXI DMA Specific Registers/Offsets */ #define XILINX_DMA_REG_SRCDSTADDR 0x18 @@ -1313,7 +1315,8 @@ static int xilinx_dma_stop_transfer(struct xilinx_dma_chan *chan) /* Wait for the hardware to halt */ return xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val, - val & XILINX_DMA_DMASR_HALTED, 0, + val & XILINX_DMA_DMASR_HALTED, + XILINX_DMA_POLL_DELAY_US, XILINX_DMA_LOOP_COUNT); } @@ -1328,7 +1331,8 @@ static int xilinx_cdma_stop_transfer(struct xilinx_dma_chan *chan) u32 val; return xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val, - val & XILINX_DMA_DMASR_IDLE, 0, + val & XILINX_DMA_DMASR_IDLE, + XILINX_DMA_POLL_DELAY_US, XILINX_DMA_LOOP_COUNT); } @@ -1345,7 +1349,8 @@ static void xilinx_dma_start(struct xilinx_dma_chan *chan) /* Wait for the hardware to start */ err = xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMASR, val, - !(val & XILINX_DMA_DMASR_HALTED), 0, + !(val & XILINX_DMA_DMASR_HALTED), + XILINX_DMA_POLL_DELAY_US, XILINX_DMA_LOOP_COUNT); if (err) { @@ -1778,7 +1783,8 @@ static int xilinx_dma_reset(struct xilinx_dma_chan *chan) /* Wait for the hardware to finish reset */ err = xilinx_dma_poll_timeout(chan, XILINX_DMA_REG_DMACR, tmp, - !(tmp & XILINX_DMA_DMACR_RESET), 0, + !(tmp & XILINX_DMA_DMACR_RESET), + XILINX_DMA_POLL_DELAY_US, XILINX_DMA_LOOP_COUNT); if (err) { diff --git a/drivers/edac/amd64_edac.c b/drivers/edac/amd64_edac.c index a8db2bde6384..d4f6a5426ec6 100644 --- a/drivers/edac/amd64_edac.c +++ b/drivers/edac/amd64_edac.c @@ -3068,13 +3068,13 @@ static inline void decode_bus_error(int node_id, struct mce *m) * the instance_id. For example, instance_id=0xYXXXXX where Y is the channel * number. * - * For DRAM ECC errors, the Chip Select number is given in bits [2:0] of + * For DRAM ECC errors, the Chip Select number is given in bits [1:0] of * the MCA_SYND[ErrorInformation] field. */ static void umc_get_err_info(struct mce *m, struct err_info *err) { err->channel = (m->ipid & GENMASK(31, 0)) >> 20; - err->csrow = m->synd & 0x7; + err->csrow = m->synd & 0x3; } static void decode_umc_error(int node_id, struct mce *m) diff --git a/drivers/edac/igen6_edac.c b/drivers/edac/igen6_edac.c index 34a25103402e..766cd1a0cfa3 100644 --- a/drivers/edac/igen6_edac.c +++ b/drivers/edac/igen6_edac.c @@ -1007,6 +1007,11 @@ static int igen6_pci_setup(struct pci_dev *pdev, u64 *mchbar) return -ENODEV; } +static void imc_release(struct device *dev) +{ + /* Nothing to do, the 'imc' owns the 'dev' and will also release it. */ +} + static int igen6_register_mci(int mc, u64 mchbar, struct pci_dev *pdev) { struct edac_mc_layer layers[2]; @@ -1051,6 +1056,7 @@ static int igen6_register_mci(int mc, u64 mchbar, struct pci_dev *pdev) mci->pvt_info = &igen6_pvt->imc[mc]; imc = mci->pvt_info; + imc->dev.release = imc_release; device_initialize(&imc->dev); /* * EDAC core uses mci->pdev(pointer of structure device) as diff --git a/drivers/firewire/core-topology.c b/drivers/firewire/core-topology.c index f40c81534381..99a2fc88ccd2 100644 --- a/drivers/firewire/core-topology.c +++ b/drivers/firewire/core-topology.c @@ -161,6 +161,17 @@ static inline struct fw_node *fw_node(struct list_head *l) return list_entry(l, struct fw_node, link); } +typedef void (*fw_node_callback_t)(struct fw_card *card, struct fw_node *node, + struct fw_node *parent); + +static void for_each_fw_node(struct fw_card *card, struct fw_node *root, + fw_node_callback_t callback); + +static void free_fw_node(struct fw_card *card, struct fw_node *node, struct fw_node *parent) +{ + fw_node_put(node); +} + /* * This function builds the tree representation of the topology given * by the self IDs from the latest bus reset. During the construction @@ -193,19 +204,19 @@ static struct fw_node *build_tree(struct fw_card *card, if (next_sid == NULL) { fw_err(card, "inconsistent extended self IDs\n"); - return NULL; + goto error; } q = *sid; if (phy_id != SELF_ID_PHY_ID(q)) { fw_err(card, "PHY ID mismatch in self ID: %d != %d\n", phy_id, SELF_ID_PHY_ID(q)); - return NULL; + goto error; } if (child_port_count > stack_depth) { fw_err(card, "topology stack underflow\n"); - return NULL; + goto error; } /* @@ -223,7 +234,7 @@ static struct fw_node *build_tree(struct fw_card *card, node = fw_node_create(q, port_count, card->color); if (node == NULL) { fw_err(card, "out of memory while building topology\n"); - return NULL; + goto error; } if (phy_id == (card->node_id & 0x3f)) @@ -273,7 +284,7 @@ static struct fw_node *build_tree(struct fw_card *card, (next_sid < end && parent_count != 1)) { fw_err(card, "parent port inconsistency for node %d: " "parent_count=%d\n", phy_id, parent_count); - return NULL; + goto error; } /* Pop the child nodes off the stack and push the new node. */ @@ -304,12 +315,13 @@ static struct fw_node *build_tree(struct fw_card *card, card->beta_repeaters_present = beta_repeaters_present; return local_node; +error: + ++card->color; + list_for_each_entry_safe(node, child, &stack, link) + for_each_fw_node(card, node, free_fw_node); + return NULL; } -typedef void (*fw_node_callback_t)(struct fw_card * card, - struct fw_node * node, - struct fw_node * parent); - static void for_each_fw_node(struct fw_card *card, struct fw_node *root, fw_node_callback_t callback) { diff --git a/drivers/firewire/ohci.c b/drivers/firewire/ohci.c index b9ae0340b8a7..b184669cdaef 100644 --- a/drivers/firewire/ohci.c +++ b/drivers/firewire/ohci.c @@ -719,12 +719,14 @@ static void ar_context_link_page(struct ar_context *ctx, unsigned int index) static void ar_context_release(struct ar_context *ctx) { - struct device *dev = ctx->ohci->card.device; + struct device *dev; unsigned int i; if (!ctx->buffer) return; + dev = ctx->ohci->card.device; + vunmap(ctx->buffer); for (i = 0; i < AR_BUFFERS; i++) { diff --git a/drivers/firmware/arm_ffa/driver.c b/drivers/firmware/arm_ffa/driver.c index 2309a31549e2..8d6217a6b623 100644 --- a/drivers/firmware/arm_ffa/driver.c +++ b/drivers/firmware/arm_ffa/driver.c @@ -481,7 +481,7 @@ ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize, constituents = buffer; } - if ((void *)constituents - buffer > max_fragsize) { + if ((void *)constituents + sizeof(*constituents) - buffer > max_fragsize) { pr_err("Memory Region Fragment > Tx Buffer size\n"); return -EFAULT; } @@ -490,7 +490,7 @@ ffa_setup_and_transmit(u32 func_id, void *buffer, u32 max_fragsize, constituents->pg_cnt = args->sg->length / FFA_PAGE_SIZE; constituents->reserved = 0; constituents++; - frag_len += sizeof(struct ffa_mem_region_addr_range); + frag_len += sizeof(*constituents); } while ((args->sg = sg_next(args->sg))); return ffa_transmit_fragment(func_id, addr, buf_sz, frag_len, diff --git a/drivers/firmware/arm_scmi/bus.c b/drivers/firmware/arm_scmi/bus.c index 22e5ed1f7320..157717ac567a 100644 --- a/drivers/firmware/arm_scmi/bus.c +++ b/drivers/firmware/arm_scmi/bus.c @@ -249,8 +249,9 @@ static int scmi_match_by_id_table(struct device *dev, void *data) (id_table->name && !strcmp(sdev->name, id_table->name)); } -static struct scmi_device *scmi_child_dev_find(struct device *parent, - int prot_id, const char *name) +/* Returns a device_find_child() reference which must be dropped by caller. */ +static struct scmi_device * +scmi_child_dev_find_get(struct device *parent, int prot_id, const char *name) { struct scmi_device_id id_table; struct device *dev; @@ -262,9 +263,6 @@ static struct scmi_device *scmi_child_dev_find(struct device *parent, if (!dev) return NULL; - /* Drop the refcnt bumped implicitly by device_find_child */ - put_device(dev); - return to_scmi_dev(dev); } @@ -332,10 +330,19 @@ void scmi_driver_unregister(struct scmi_driver *driver) } EXPORT_SYMBOL_GPL(scmi_driver_unregister); +static void scmi_device_release_resources(struct scmi_device *scmi_dev) +{ + if (scmi_dev->id) { + ida_free(&scmi_bus_id, scmi_dev->id); + scmi_dev->id = 0; + } +} + static void scmi_device_release(struct device *dev) { struct scmi_device *scmi_dev = to_scmi_dev(dev); + scmi_device_release_resources(scmi_dev); kfree_const(scmi_dev->name); kfree(scmi_dev); } @@ -350,8 +357,9 @@ static void __scmi_device_destroy(struct scmi_device *scmi_dev) if (scmi_dev->protocol_id == SCMI_PROTOCOL_SYSTEM) atomic_set(&scmi_syspower_registered, 0); - ida_free(&scmi_bus_id, scmi_dev->id); - device_unregister(&scmi_dev->dev); + device_del(&scmi_dev->dev); + scmi_device_release_resources(scmi_dev); + put_device(&scmi_dev->dev); } static struct scmi_device * @@ -368,9 +376,11 @@ __scmi_device_create(struct device_node *np, struct device *parent, * each DT defined protocol at probe time, and the concurrent * registration of SCMI drivers. */ - scmi_dev = scmi_child_dev_find(parent, protocol, name); - if (scmi_dev) + scmi_dev = scmi_child_dev_find_get(parent, protocol, name); + if (scmi_dev) { + put_device(&scmi_dev->dev); return scmi_dev; + } /* * Ignore any possible subsequent failures while creating the device @@ -421,8 +431,8 @@ __scmi_device_create(struct device_node *np, struct device *parent, return scmi_dev; put_dev: + scmi_device_release_resources(scmi_dev); put_device(&scmi_dev->dev); - ida_free(&scmi_bus_id, id); return NULL; } @@ -494,9 +504,11 @@ void scmi_device_destroy(struct device *parent, int protocol, const char *name) { struct scmi_device *scmi_dev; - scmi_dev = scmi_child_dev_find(parent, protocol, name); - if (scmi_dev) + scmi_dev = scmi_child_dev_find_get(parent, protocol, name); + if (scmi_dev) { __scmi_device_destroy(scmi_dev); + put_device(&scmi_dev->dev); + } } EXPORT_SYMBOL_GPL(scmi_device_destroy); diff --git a/drivers/firmware/arm_scmi/driver.c b/drivers/firmware/arm_scmi/driver.c index d253e9d41d84..0c2a0dc350eb 100644 --- a/drivers/firmware/arm_scmi/driver.c +++ b/drivers/firmware/arm_scmi/driver.c @@ -2449,6 +2449,7 @@ static int scmi_chan_setup(struct scmi_info *info, struct device_node *of_node, cinfo->id = prot_id; cinfo->dev = &tdev->dev; + cinfo->handle = &info->handle; ret = info->desc->ops->chan_setup(cinfo, info->dev, tx); if (ret) { of_node_put(of_node); @@ -2482,7 +2483,6 @@ static int scmi_chan_setup(struct scmi_info *info, struct device_node *of_node, return ret; } - cinfo->handle = &info->handle; return 0; } @@ -2830,7 +2830,7 @@ static int scmi_probe(struct platform_device *pdev) /* Setup all channels described in the DT at first */ ret = scmi_channels_setup(info); if (ret) - goto clear_ida; + goto clear_txrx_setup; ret = bus_register_notifier(&scmi_bus_type, &info->bus_nb); if (ret) diff --git a/drivers/firmware/arm_scmi/mailbox.c b/drivers/firmware/arm_scmi/mailbox.c index f1d5e3fba35e..a34a3c693e15 100644 --- a/drivers/firmware/arm_scmi/mailbox.c +++ b/drivers/firmware/arm_scmi/mailbox.c @@ -211,13 +211,18 @@ static int mailbox_chan_setup(struct scmi_chan_info *cinfo, struct device *dev, cl->tx_block = false; cl->knows_txdone = tx; + cinfo->transport_info = smbox; + smbox->cinfo = cinfo; + mutex_init(&smbox->chan_lock); + smbox->chan = mbox_request_channel(cl, tx ? 0 : p2a_chan); if (IS_ERR(smbox->chan)) { ret = PTR_ERR(smbox->chan); + smbox->chan = NULL; if (ret != -EPROBE_DEFER) dev_err(cdev, "failed to request SCMI %s mailbox\n", desc); - return ret; + goto err_clear_cinfo; } /* Additional unidirectional channel for TX if needed */ @@ -225,17 +230,23 @@ static int mailbox_chan_setup(struct scmi_chan_info *cinfo, struct device *dev, smbox->chan_receiver = mbox_request_channel(cl, a2p_rx_chan); if (IS_ERR(smbox->chan_receiver)) { ret = PTR_ERR(smbox->chan_receiver); + smbox->chan_receiver = NULL; if (ret != -EPROBE_DEFER) dev_err(cdev, "failed to request SCMI Tx Receiver mailbox\n"); - return ret; + goto err_free_chan; } } - cinfo->transport_info = smbox; - smbox->cinfo = cinfo; - mutex_init(&smbox->chan_lock); - return 0; + +err_free_chan: + mbox_free_channel(smbox->chan); +err_clear_cinfo: + cinfo->transport_info = NULL; + smbox->cinfo = NULL; + devm_iounmap(dev, smbox->shmem); + devm_kfree(dev, smbox); + return ret; } static int mailbox_chan_free(int id, void *p, void *data) diff --git a/drivers/firmware/arm_scmi/smc.c b/drivers/firmware/arm_scmi/smc.c index 771797b6e268..fada68b8e15f 100644 --- a/drivers/firmware/arm_scmi/smc.c +++ b/drivers/firmware/arm_scmi/smc.c @@ -166,6 +166,12 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev, scmi_info->param_page = SHMEM_PAGE(res.start); scmi_info->param_offset = SHMEM_OFFSET(res.start); } + + scmi_info->func_id = func_id; + scmi_info->cinfo = cinfo; + smc_channel_lock_init(scmi_info); + cinfo->transport_info = scmi_info; + /* * If there is an interrupt named "a2p", then the service and * completion of a message is signaled by an interrupt rather than by @@ -177,17 +183,14 @@ static int smc_chan_setup(struct scmi_chan_info *cinfo, struct device *dev, IRQF_NO_SUSPEND, dev_name(dev), scmi_info); if (ret) { dev_err(dev, "failed to setup SCMI smc irq\n"); + cinfo->transport_info = NULL; + scmi_info->cinfo = NULL; return ret; } } else { cinfo->no_completion_irq = true; } - scmi_info->func_id = func_id; - scmi_info->cinfo = cinfo; - smc_channel_lock_init(scmi_info); - cinfo->transport_info = scmi_info; - return 0; } diff --git a/drivers/firmware/qcom_scm-smc.c b/drivers/firmware/qcom_scm-smc.c index 0a2a2c794d0e..ed57c22715b6 100644 --- a/drivers/firmware/qcom_scm-smc.c +++ b/drivers/firmware/qcom_scm-smc.c @@ -103,7 +103,7 @@ static int __scm_smc_do_quirk_handle_waitq(struct device *dev, struct arm_smccc_ wq_ctx = res->a1; smc_call_ctx = res->a2; - ret = qcom_scm_wait_for_wq_completion(wq_ctx); + ret = qcom_scm_wait_for_wq_completion(dev, wq_ctx); if (ret) return ret; diff --git a/drivers/firmware/qcom_scm.c b/drivers/firmware/qcom_scm.c index 4c5c2b73d42c..b5ada925f8da 100644 --- a/drivers/firmware/qcom_scm.c +++ b/drivers/firmware/qcom_scm.c @@ -1358,15 +1358,16 @@ static int qcom_scm_assert_valid_wq_ctx(u32 wq_ctx) return 0; } -int qcom_scm_wait_for_wq_completion(u32 wq_ctx) +int qcom_scm_wait_for_wq_completion(struct device *dev, u32 wq_ctx) { + struct qcom_scm *scm = dev_get_drvdata(dev); int ret; ret = qcom_scm_assert_valid_wq_ctx(wq_ctx); if (ret) return ret; - wait_for_completion(&__scm->waitq_comp); + wait_for_completion(&scm->waitq_comp); return 0; } @@ -1379,7 +1380,7 @@ static int qcom_scm_waitq_wakeup(struct qcom_scm *scm, unsigned int wq_ctx) if (ret) return ret; - complete(&__scm->waitq_comp); + complete(&scm->waitq_comp); return 0; } @@ -1422,6 +1423,7 @@ static int qcom_scm_probe(struct platform_device *pdev) return -ENOMEM; scm->dev = &pdev->dev; + platform_set_drvdata(pdev, scm); ret = qcom_scm_find_dload_address(&pdev->dev, &scm->dload_mode_addr); if (ret < 0) return ret; diff --git a/drivers/firmware/qcom_scm.h b/drivers/firmware/qcom_scm.h index e6e512bd57d1..2bbf65442239 100644 --- a/drivers/firmware/qcom_scm.h +++ b/drivers/firmware/qcom_scm.h @@ -60,7 +60,7 @@ struct qcom_scm_res { u64 result[MAX_QCOM_SCM_RETS]; }; -int qcom_scm_wait_for_wq_completion(u32 wq_ctx); +int qcom_scm_wait_for_wq_completion(struct device *dev, u32 wq_ctx); int scm_get_wq_ctx(u32 *wq_ctx, u32 *flags, u32 *more_pending); #define SCM_SMC_FNID(s, c) ((((s) & 0xFF) << 8) | ((c) & 0xFF)) diff --git a/drivers/fpga/dfl.c b/drivers/fpga/dfl.c index af618609d369..9aeaf7a4b210 100644 --- a/drivers/fpga/dfl.c +++ b/drivers/fpga/dfl.c @@ -1874,7 +1874,7 @@ static irqreturn_t dfl_irq_handler(int irq, void *arg) { struct eventfd_ctx *trigger = arg; - eventfd_signal(trigger, 1); + eventfd_signal(trigger); return IRQ_HANDLED; } diff --git a/drivers/gpio/gpio-exar.c b/drivers/gpio/gpio-exar.c index d5909a4f0433..e526fdee4c43 100644 --- a/drivers/gpio/gpio-exar.c +++ b/drivers/gpio/gpio-exar.c @@ -197,6 +197,7 @@ static int gpio_exar_probe(struct platform_device *pdev) index = ida_alloc(&ida_index, GFP_KERNEL); if (index < 0) return index; + exar_gpio->index = index; ret = devm_add_action_or_reset(dev, exar_devm_ida_free, exar_gpio); if (ret) @@ -212,7 +213,6 @@ static int gpio_exar_probe(struct platform_device *pdev) exar_gpio->gpio_chip.set = exar_set_value; exar_gpio->gpio_chip.base = -1; exar_gpio->gpio_chip.ngpio = ngpios; - exar_gpio->index = index; exar_gpio->first_pin = first_pin; ret = devm_gpiochip_add_data(dev, &exar_gpio->gpio_chip, exar_gpio); diff --git a/drivers/gpio/gpio-mvebu.c b/drivers/gpio/gpio-mvebu.c index f58f835d0acb..4eb0c7806714 100644 --- a/drivers/gpio/gpio-mvebu.c +++ b/drivers/gpio/gpio-mvebu.c @@ -1116,6 +1116,7 @@ static void mvebu_gpio_remove_irq_domain(void *data) { struct irq_domain *domain = data; + irq_domain_remove_generic_chips(domain); irq_domain_remove(domain); } diff --git a/drivers/gpio/gpio-pca953x.c b/drivers/gpio/gpio-pca953x.c index bd07cb24f233..f2a3961664cd 100644 --- a/drivers/gpio/gpio-pca953x.c +++ b/drivers/gpio/gpio-pca953x.c @@ -517,15 +517,23 @@ static int pca953x_read_regs(struct pca953x_chip *chip, int reg, unsigned long * return 0; } -static int pca953x_gpio_direction_input(struct gpio_chip *gc, unsigned off) +static int pca953x_gpio_direction_input_unlocked(struct gpio_chip *gc, + unsigned int off) { struct pca953x_chip *chip = gpiochip_get_data(gc); u8 dirreg = chip->recalc_addr(chip, chip->regs->direction, off); u8 bit = BIT(off % BANK_SZ); + return regmap_write_bits(chip->regmap, dirreg, bit, bit); +} + +static int pca953x_gpio_direction_input(struct gpio_chip *gc, unsigned int off) +{ + struct pca953x_chip *chip = gpiochip_get_data(gc); + guard(mutex)(&chip->i2c_lock); - return regmap_write_bits(chip->regmap, dirreg, bit, bit); + return pca953x_gpio_direction_input_unlocked(gc, off); } static int pca953x_gpio_direction_output(struct gpio_chip *gc, @@ -753,9 +761,9 @@ static void pca953x_irq_bus_sync_unlock(struct irq_data *d) DECLARE_BITMAP(reg_direction, MAX_LINE); int level; - if (chip->driver_data & PCA_PCAL) { - guard(mutex)(&chip->i2c_lock); + guard(mutex)(&chip->i2c_lock); + if (chip->driver_data & PCA_PCAL) { /* Enable latch on interrupt-enabled inputs */ pca953x_write_regs(chip, PCAL953X_IN_LATCH, chip->irq_mask); @@ -776,7 +784,7 @@ static void pca953x_irq_bus_sync_unlock(struct irq_data *d) /* Look for any newly setup interrupt */ for_each_set_bit(level, irq_mask, gc->ngpio) - pca953x_gpio_direction_input(&chip->gpio_chip, level); + pca953x_gpio_direction_input_unlocked(&chip->gpio_chip, level); mutex_unlock(&chip->irq_lock); } diff --git a/drivers/gpio/gpio-rockchip.c b/drivers/gpio/gpio-rockchip.c index ff9a4b8611d7..86cf3cbcdb40 100644 --- a/drivers/gpio/gpio-rockchip.c +++ b/drivers/gpio/gpio-rockchip.c @@ -629,10 +629,17 @@ static int rockchip_gpiolib_register(struct rockchip_pin_bank *bank) return ret; } +static void rockchip_clk_put(void *data) +{ + struct clk *clk = data; + + clk_put(clk); +} + static int rockchip_get_bank_data(struct rockchip_pin_bank *bank) { struct resource res; - int id = 0; + int id = 0, ret; if (of_address_to_resource(bank->of_node, 0, &res)) { dev_err(bank->dev, "cannot find IO resource for bank\n"); @@ -662,6 +669,11 @@ static int rockchip_get_bank_data(struct rockchip_pin_bank *bank) dev_err(bank->dev, "cannot find debounce clk\n"); return -EINVAL; } + + ret = devm_add_action_or_reset(bank->dev, rockchip_clk_put, + bank->db_clk); + if (ret) + return ret; } else { bank->gpio_regs = &gpio_regs_v1; bank->gpio_type = GPIO_TYPE_V1; @@ -773,6 +785,11 @@ static int rockchip_gpio_remove(struct platform_device *pdev) { struct rockchip_pin_bank *bank = platform_get_drvdata(pdev); + irq_set_chained_handler_and_data(bank->irq, NULL, NULL); + if (bank->domain) { + irq_domain_remove_generic_chips(bank->domain); + irq_domain_remove(bank->domain); + } gpiochip_remove(&bank->gpio_chip); return 0; diff --git a/drivers/gpio/gpio-xilinx.c b/drivers/gpio/gpio-xilinx.c index 956ea2957833..a37b84048027 100644 --- a/drivers/gpio/gpio-xilinx.c +++ b/drivers/gpio/gpio-xilinx.c @@ -293,14 +293,7 @@ static void xgpio_save_regs(struct xgpio_instance *chip) static int xgpio_request(struct gpio_chip *chip, unsigned int offset) { - int ret; - - ret = pm_runtime_get_sync(chip->parent); - /* - * If the device is already active pm_runtime_get() will return 1 on - * success, but gpio_request still needs to return 0. - */ - return ret < 0 ? ret : 0; + return pm_runtime_resume_and_get(chip->parent); } static void xgpio_free(struct gpio_chip *chip, unsigned int offset) diff --git a/drivers/gpu/drm/Makefile b/drivers/gpu/drm/Makefile index 215e78e79125..7a84b3cddeab 100644 --- a/drivers/gpu/drm/Makefile +++ b/drivers/gpu/drm/Makefile @@ -45,7 +45,7 @@ drm-y := \ drm_vblank.o \ drm_vblank_work.o \ drm_vma_manager.o \ - drm_gpuva_mgr.o \ + drm_gpuvm.o \ drm_writeback.o drm-$(CONFIG_DRM_LEGACY) += \ drm_agpsupport.o \ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu.h b/drivers/gpu/drm/amd/amdgpu/amdgpu.h index 0d2f7be98dd4..975c362f2b22 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu.h @@ -385,6 +385,7 @@ bool amdgpu_get_bios(struct amdgpu_device *adev); bool amdgpu_read_bios(struct amdgpu_device *adev); bool amdgpu_soc15_read_bios_from_rom(struct amdgpu_device *adev, u8 *bios, u32 length_bytes); +void amdgpu_bios_release(struct amdgpu_device *adev); /* * Clocks */ @@ -757,8 +758,8 @@ struct amdgpu_mqd { #define AMDGPU_RESET_MAGIC_NUM 64 #define AMDGPU_MAX_DF_PERFMONS 4 -#define AMDGPU_PRODUCT_NAME_LEN 64 struct amdgpu_reset_domain; +struct amdgpu_fru_info; /* * Non-zero (true) if the GPU has VRAM. Zero (false) otherwise. @@ -1036,11 +1037,7 @@ struct amdgpu_device { bool ucode_sysfs_en; - /* Chip product information */ - char product_number[20]; - char product_name[AMDGPU_PRODUCT_NAME_LEN]; - char serial[20]; - + struct amdgpu_fru_info *fru_info; atomic_t throttling_logging_enabled; struct ratelimit_state throttling_logging_rs; uint32_t ras_hw_enabled; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c index 31c3aaeb3bb4..4453d32323e6 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_acp.c @@ -324,7 +324,7 @@ static int acp_hw_init(void *handle) r = device_for_each_child(adev->acp.parent, &adev->acp.acp_genpd->gpd, acp_genpd_add_device); if (r) - goto failure; + goto failure_remove_mfd; break; } default: @@ -446,7 +446,7 @@ static int acp_hw_init(void *handle) r = device_for_each_child(adev->acp.parent, &adev->acp.acp_genpd->gpd, acp_genpd_add_device); if (r) - goto failure; + goto failure_remove_mfd; } /* Assert Soft reset of ACP */ @@ -464,7 +464,7 @@ static int acp_hw_init(void *handle) if (--count == 0) { dev_err(&adev->pdev->dev, "Failed to reset ACP\n"); r = -ETIMEDOUT; - goto failure; + goto failure_remove_mfd; } udelay(100); } @@ -482,7 +482,7 @@ static int acp_hw_init(void *handle) if (--count == 0) { dev_err(&adev->pdev->dev, "Failed to reset ACP\n"); r = -ETIMEDOUT; - goto failure; + goto failure_remove_mfd; } udelay(100); } @@ -492,6 +492,11 @@ static int acp_hw_init(void *handle) cgs_write_register(adev->acp.cgs_device, mmACP_SOFT_RESET, val); return 0; +failure_remove_mfd: + device_for_each_child(adev->acp.parent, NULL, + acp_genpd_remove_device); + mfd_remove_devices(adev->acp.parent); + failure: kfree(i2s_pdata); kfree(adev->acp.acp_res); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c index 7730e5644493..45cea079d6c8 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_acpi.c @@ -1110,6 +1110,7 @@ static int amdgpu_acpi_enumerate_xcc(void) GFP_KERNEL); if (!xcc_info) { DRM_ERROR("Failed to allocate memory for xcc info\n"); + acpi_dev_put(acpi_dev); return -ENOMEM; } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c index 0c5689eace67..d3d34a489a4f 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_amdkfd.c @@ -508,14 +508,18 @@ int amdgpu_amdkfd_get_dmabuf_info(struct amdgpu_device *adev, int dma_buf_fd, /* first get metadata_size by buffer = NULL */ r = amdgpu_bo_get_metadata(bo, NULL, 0, metadata_size, NULL); + if (r) + goto out_put; /* user buf_size is bigger than bo metadata_size * allocate a buf at kernel space and copy */ if (*metadata_size <= buffer_size) { *metadata_buffer = kzalloc(*metadata_size, GFP_KERNEL); - if (!*metadata_buffer) - return -ENOMEM; + if (!*metadata_buffer) { + r = -ENOMEM; + goto out_put; + } r = amdgpu_bo_get_metadata(bo, *metadata_buffer, *metadata_size, NULL, &metadata_flags); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c index a14a54a734c1..fd9e4f1054b5 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_atombios.c @@ -36,11 +36,19 @@ #include "atombios_encoders.h" #include "bif/bif_4_1_d.h" -static void amdgpu_atombios_lookup_i2c_gpio_quirks(struct amdgpu_device *adev, - ATOM_GPIO_I2C_ASSIGMENT *gpio, - u8 index) +/* VBIOS-reported table size is unchecked against the image; cap iterations and + * adev->i2c_bus[] indexing to AMDGPU_MAX_I2C_BUS. + */ +static int amdgpu_atombios_gpio_i2c_num_entries(uint16_t size) { + u32 bytes; + + if (size < sizeof(ATOM_COMMON_TABLE_HEADER)) + return 0; + bytes = size - sizeof(ATOM_COMMON_TABLE_HEADER); + return (int)min_t(u32, bytes / sizeof(ATOM_GPIO_I2C_ASSIGMENT), + AMDGPU_MAX_I2C_BUS); } static struct amdgpu_i2c_bus_rec amdgpu_atombios_get_bus_rec_for_i2c_gpio(ATOM_GPIO_I2C_ASSIGMENT *gpio) @@ -103,14 +111,10 @@ struct amdgpu_i2c_bus_rec amdgpu_atombios_lookup_i2c_gpio(struct amdgpu_device * if (amdgpu_atom_parse_data_header(ctx, index, &size, NULL, NULL, &data_offset)) { i2c_info = (struct _ATOM_GPIO_I2C_INFO *)(ctx->bios + data_offset); - num_indices = (size - sizeof(ATOM_COMMON_TABLE_HEADER)) / - sizeof(ATOM_GPIO_I2C_ASSIGMENT); + num_indices = amdgpu_atombios_gpio_i2c_num_entries(size); gpio = &i2c_info->asGPIO_Info[0]; for (i = 0; i < num_indices; i++) { - - amdgpu_atombios_lookup_i2c_gpio_quirks(adev, gpio, i); - if (gpio->sucI2cId.ucAccess == id) { i2c = amdgpu_atombios_get_bus_rec_for_i2c_gpio(gpio); break; @@ -137,13 +141,10 @@ void amdgpu_atombios_i2c_init(struct amdgpu_device *adev) if (amdgpu_atom_parse_data_header(ctx, index, &size, NULL, NULL, &data_offset)) { i2c_info = (struct _ATOM_GPIO_I2C_INFO *)(ctx->bios + data_offset); - num_indices = (size - sizeof(ATOM_COMMON_TABLE_HEADER)) / - sizeof(ATOM_GPIO_I2C_ASSIGMENT); + num_indices = amdgpu_atombios_gpio_i2c_num_entries(size); gpio = &i2c_info->asGPIO_Info[0]; for (i = 0; i < num_indices; i++) { - amdgpu_atombios_lookup_i2c_gpio_quirks(adev, gpio, i); - i2c = amdgpu_atombios_get_bus_rec_for_i2c_gpio(gpio); if (i2c.valid) { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c index 367787653bc0..1564cc68fab8 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_bios.c @@ -47,41 +47,50 @@ /* Check if current bios is an ATOM BIOS. * Return true if it is ATOM BIOS. Otherwise, return false. */ -static bool check_atom_bios(uint8_t *bios, size_t size) +static bool check_atom_bios(struct amdgpu_device *adev, size_t size) { uint16_t tmp, bios_header_start; + uint8_t *bios = adev->bios; if (!bios || size < 0x49) { - DRM_INFO("vbios mem is null or mem size is wrong\n"); + dev_dbg(adev->dev, "VBIOS mem is null or mem size is wrong\n"); return false; } if (!AMD_IS_VALID_VBIOS(bios)) { - DRM_INFO("BIOS signature incorrect %x %x\n", bios[0], bios[1]); + dev_dbg(adev->dev, "VBIOS signature incorrect %x %x\n", bios[0], + bios[1]); return false; } bios_header_start = bios[0x48] | (bios[0x49] << 8); if (!bios_header_start) { - DRM_INFO("Can't locate bios header\n"); + dev_dbg(adev->dev, "Can't locate VBIOS header\n"); return false; } tmp = bios_header_start + 4; if (size < tmp) { - DRM_INFO("BIOS header is broken\n"); + dev_dbg(adev->dev, "VBIOS header is broken\n"); return false; } if (!memcmp(bios + tmp, "ATOM", 4) || !memcmp(bios + tmp, "MOTA", 4)) { - DRM_DEBUG("ATOMBIOS detected\n"); + dev_dbg(adev->dev, "ATOMBIOS detected\n"); return true; } return false; } +void amdgpu_bios_release(struct amdgpu_device *adev) +{ + kfree(adev->bios); + adev->bios = NULL; + adev->bios_size = 0; +} + /* If you boot an IGP board with a discrete card as the primary, * the IGP rom is not accessible via the rom bar as the IGP rom is * part of the system bios. On boot, the system bios puts a @@ -117,8 +126,8 @@ static bool igp_read_bios_from_vram(struct amdgpu_device *adev) memcpy_fromio(adev->bios, bios, size); iounmap(bios); - if (!check_atom_bios(adev->bios, size)) { - kfree(adev->bios); + if (!check_atom_bios(adev, size)) { + amdgpu_bios_release(adev); return false; } @@ -145,8 +154,8 @@ bool amdgpu_read_bios(struct amdgpu_device *adev) memcpy_fromio(adev->bios, bios, size); pci_unmap_rom(adev->pdev, bios); - if (!check_atom_bios(adev->bios, size)) { - kfree(adev->bios); + if (!check_atom_bios(adev, size)) { + amdgpu_bios_release(adev); return false; } @@ -185,8 +194,8 @@ static bool amdgpu_read_bios_from_rom(struct amdgpu_device *adev) /* read complete BIOS */ amdgpu_asic_read_bios_from_rom(adev, adev->bios, len); - if (!check_atom_bios(adev->bios, len)) { - kfree(adev->bios); + if (!check_atom_bios(adev, len)) { + amdgpu_bios_release(adev); return false; } @@ -215,14 +224,15 @@ static bool amdgpu_read_platform_bios(struct amdgpu_device *adev) memcpy_fromio(adev->bios, bios, romlen); iounmap(bios); - if (!check_atom_bios(adev->bios, romlen)) + if (!check_atom_bios(adev, romlen)) goto free_bios; adev->bios_size = romlen; return true; free_bios: - kfree(adev->bios); + amdgpu_bios_release(adev); + return false; } @@ -337,8 +347,8 @@ static bool amdgpu_atrm_get_bios(struct amdgpu_device *adev) break; } - if (!check_atom_bios(adev->bios, size)) { - kfree(adev->bios); + if (!check_atom_bios(adev, size)) { + amdgpu_bios_release(adev); return false; } adev->bios_size = size; @@ -406,13 +416,14 @@ static bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev) acpi_size tbl_size; UEFI_ACPI_VFCT *vfct; unsigned int offset; + bool r = false; if (!ACPI_SUCCESS(acpi_get_table("VFCT", 1, &hdr))) return false; tbl_size = hdr->length; if (tbl_size < sizeof(UEFI_ACPI_VFCT)) { dev_info(adev->dev, "ACPI VFCT table present but broken (too short #1),skipping\n"); - return false; + goto out; } vfct = (UEFI_ACPI_VFCT *)hdr; @@ -425,13 +436,13 @@ static bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev) offset += sizeof(VFCT_IMAGE_HEADER); if (offset > tbl_size) { dev_info(adev->dev, "ACPI VFCT image header truncated,skipping\n"); - return false; + goto out; } offset += vhdr->ImageLength; if (offset > tbl_size) { dev_info(adev->dev, "ACPI VFCT image truncated,skipping\n"); - return false; + goto out; } if (vhdr->ImageLength && @@ -440,17 +451,21 @@ static bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev) vhdr->ImageLength, GFP_KERNEL); - if (!check_atom_bios(adev->bios, vhdr->ImageLength)) { - kfree(adev->bios); - return false; + if (!check_atom_bios(adev, vhdr->ImageLength)) { + amdgpu_bios_release(adev); + goto out; } adev->bios_size = vhdr->ImageLength; - return true; + r = true; + goto out; } } dev_info(adev->dev, "ACPI VFCT table present but broken (too short #2),skipping\n"); - return false; + +out: + acpi_put_table(hdr); + return r; } #else static inline bool amdgpu_acpi_vfct_bios(struct amdgpu_device *adev) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c index a5ede9ab877d..df7a5775d361 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c @@ -41,6 +41,25 @@ #include "amdgpu_gem.h" #include "amdgpu_ras.h" +/* + * Maximum IB length (dwords) for rings whose emit_ib packet format + * documents a 20-bit size field. + */ +#define AMDGPU_GFX_SDMA_IB_PACKET_SIZE_MAX_DW 0xFFFFF +#define AMDGPU_MM_IB_PACKET_SIZE_MAX_DW 0x7FFFF0 + +static u32 amdgpu_cs_ib_packet_size_max_dw(enum amdgpu_ring_type type) +{ + switch (type) { + case AMDGPU_RING_TYPE_GFX: + case AMDGPU_RING_TYPE_COMPUTE: + case AMDGPU_RING_TYPE_SDMA: + return AMDGPU_GFX_SDMA_IB_PACKET_SIZE_MAX_DW; + default: + return AMDGPU_MM_IB_PACKET_SIZE_MAX_DW; + } +} + static int amdgpu_cs_parser_init(struct amdgpu_cs_parser *p, struct amdgpu_device *adev, struct drm_file *filp, @@ -354,7 +373,6 @@ static int amdgpu_cs_p2_ib(struct amdgpu_cs_parser *p, job = p->jobs[r]; ring = amdgpu_job_ring(job); - ib = &job->ibs[job->num_ibs++]; /* MM engine doesn't support user fences */ if (p->uf_bo && ring->funcs->no_user_fence) @@ -373,6 +391,12 @@ static int amdgpu_cs_p2_ib(struct amdgpu_cs_parser *p, return -EINVAL; } + if (chunk_ib->ib_bytes / 4 > + amdgpu_cs_ib_packet_size_max_dw(ring->funcs->type)) + return -EINVAL; + + ib = &job->ibs[job->num_ibs++]; + if (chunk_ib->flags & AMDGPU_IB_FLAG_PREAMBLE) job->preamble_status |= AMDGPU_PREAMBLE_IB_PRESENT; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c index cf8804fa7e97..afcc157adc12 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ctx.c @@ -320,7 +320,6 @@ static int amdgpu_ctx_init(struct amdgpu_ctx_mgr *mgr, int32_t priority, struct drm_file *filp, struct amdgpu_ctx *ctx) { struct amdgpu_fpriv *fpriv = filp->driver_priv; - u32 current_stable_pstate; int r; r = amdgpu_ctx_priority_permit(filp, priority); @@ -338,37 +337,22 @@ static int amdgpu_ctx_init(struct amdgpu_ctx_mgr *mgr, int32_t priority, ctx->generation = amdgpu_vm_generation(mgr->adev, &fpriv->vm); ctx->init_priority = priority; ctx->override_priority = AMDGPU_CTX_PRIORITY_UNSET; - - r = amdgpu_ctx_get_stable_pstate(ctx, ¤t_stable_pstate); - if (r) - return r; - - if (mgr->adev->pm.stable_pstate_ctx) - ctx->stable_pstate = mgr->adev->pm.stable_pstate_ctx->stable_pstate; - else - ctx->stable_pstate = current_stable_pstate; + ctx->stable_pstate = AMDGPU_CTX_STABLE_PSTATE_NONE; ctx->ctx_mgr = &(fpriv->ctx_mgr); return 0; } -static int amdgpu_ctx_set_stable_pstate(struct amdgpu_ctx *ctx, - u32 stable_pstate) +static int __amdgpu_ctx_set_stable_pstate(struct amdgpu_ctx *ctx, + u32 stable_pstate) { struct amdgpu_device *adev = ctx->mgr->adev; enum amd_dpm_forced_level level; + struct amdgpu_ctx *current_ctx; u32 current_stable_pstate; - int r; + int r = 0; - mutex_lock(&adev->pm.stable_pstate_ctx_lock); - if (adev->pm.stable_pstate_ctx && adev->pm.stable_pstate_ctx != ctx) { - r = -EBUSY; - goto done; - } - - r = amdgpu_ctx_get_stable_pstate(ctx, ¤t_stable_pstate); - if (r || (stable_pstate == current_stable_pstate)) - goto done; + lockdep_assert_held(&adev->pm.stable_pstate_ctx_lock); switch (stable_pstate) { case AMDGPU_CTX_STABLE_PSTATE_NONE: @@ -387,17 +371,41 @@ static int amdgpu_ctx_set_stable_pstate(struct amdgpu_ctx *ctx, level = AMD_DPM_FORCED_LEVEL_PROFILE_PEAK; break; default: - r = -EINVAL; - goto done; + return -EINVAL; } + current_ctx = adev->pm.stable_pstate_ctx; + if (current_ctx && current_ctx != ctx) + return -EBUSY; + + r = amdgpu_ctx_get_stable_pstate(ctx, ¤t_stable_pstate); + if (r || current_stable_pstate == stable_pstate) + return r; + r = amdgpu_dpm_force_performance_level(adev, level); + if (r) + return r; - if (level == AMD_DPM_FORCED_LEVEL_AUTO) - adev->pm.stable_pstate_ctx = NULL; - else + if (!current_ctx) { adev->pm.stable_pstate_ctx = ctx; -done: + /* + * Serialized by context taking ownership for the first time + * while holding adev->pm.stable_pstate_ctx_lock). + */ + WRITE_ONCE(ctx->stable_pstate, current_stable_pstate); + } + + return 0; +} + +static int amdgpu_ctx_set_stable_pstate(struct amdgpu_ctx *ctx, + u32 stable_pstate) +{ + struct amdgpu_device *adev = ctx->mgr->adev; + int r; + + mutex_lock(&adev->pm.stable_pstate_ctx_lock); + r = __amdgpu_ctx_set_stable_pstate(ctx, stable_pstate); mutex_unlock(&adev->pm.stable_pstate_ctx_lock); return r; @@ -423,7 +431,12 @@ static void amdgpu_ctx_fini(struct kref *ref) } if (drm_dev_enter(adev_to_drm(adev), &idx)) { - amdgpu_ctx_set_stable_pstate(ctx, ctx->stable_pstate); + mutex_lock(&adev->pm.stable_pstate_ctx_lock); + if (adev->pm.stable_pstate_ctx == ctx) { + __amdgpu_ctx_set_stable_pstate(ctx, ctx->stable_pstate); + adev->pm.stable_pstate_ctx = NULL; + } + mutex_unlock(&adev->pm.stable_pstate_ctx_lock); drm_dev_exit(idx); } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c index cf31193b4b35..7c304c1c9c6f 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c @@ -114,6 +114,20 @@ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f, *pos &= (1UL << 22) - 1; + if (size > PAGE_SIZE) + return -EINVAL; + + /* + * Access the user buffer only while none of the locks below are held: + * a fault takes mmap_lock, and grbm_idx_mutex and srbm_mutex nest + * inside it. + */ + u32 *data __free(kfree) = kmalloc(size, GFP_KERNEL); + if (!data) + return -ENOMEM; + if (!read && copy_from_user(data, buf, size)) + return -EFAULT; + r = pm_runtime_get_sync(adev_to_drm(adev)->dev); if (r < 0) { pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); @@ -146,28 +160,16 @@ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f, mutex_lock(&adev->pm.mutex); while (size) { - uint32_t value; - - if (read) { - value = RREG32(*pos >> 2); - r = put_user(value, (uint32_t *)buf); - } else { - r = get_user(value, (uint32_t *)buf); - if (!r) - amdgpu_mm_wreg_mmio_rlc(adev, *pos >> 2, value, 0); - } - if (r) { - result = r; - goto end; - } + if (read) + data[result >> 2] = RREG32(*pos >> 2); + else + amdgpu_mm_wreg_mmio_rlc(adev, *pos >> 2, data[result >> 2], 0); result += 4; - buf += 4; *pos += 4; size -= 4; } -end: if (use_bank) { amdgpu_gfx_select_se_sh(adev, 0xffffffff, 0xffffffff, 0xffffffff, 0); mutex_unlock(&adev->grbm_idx_mutex); @@ -183,6 +185,10 @@ static int amdgpu_debugfs_process_reg_op(bool read, struct file *f, pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); amdgpu_virt_disable_access_debugfs(adev); + + if (read && copy_to_user(buf, data, result)) + return -EFAULT; + return result; } @@ -233,11 +239,24 @@ static ssize_t amdgpu_debugfs_regs2_op(struct file *f, char __user *buf, u32 off struct amdgpu_device *adev = rd->adev; ssize_t result = 0; int r; - uint32_t value; if (size & 0x3 || offset & 0x3) return -EINVAL; + if (size > PAGE_SIZE) + return -EINVAL; + + /* + * Access the user buffer only while none of the locks below are held: + * a fault takes mmap_lock, and grbm_idx_mutex and srbm_mutex nest + * inside it. + */ + u32 *data __free(kfree) = kmalloc(size, GFP_KERNEL); + if (!data) + return -ENOMEM; + if (write_en && copy_from_user(data, buf, size)) + return -EFAULT; + r = pm_runtime_get_sync(adev_to_drm(adev)->dev); if (r < 0) { pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); @@ -277,24 +296,16 @@ static ssize_t amdgpu_debugfs_regs2_op(struct file *f, char __user *buf, u32 off mutex_lock(&adev->pm.mutex); while (size) { - if (!write_en) { - value = RREG32(offset >> 2); - r = put_user(value, (uint32_t *)buf); - } else { - r = get_user(value, (uint32_t *)buf); - if (!r) - amdgpu_mm_wreg_mmio_rlc(adev, offset >> 2, value, rd->id.xcc_id); - } - if (r) { - result = r; - goto end; - } + if (!write_en) + data[result >> 2] = RREG32(offset >> 2); + else + amdgpu_mm_wreg_mmio_rlc(adev, offset >> 2, data[result >> 2], + rd->id.xcc_id); offset += 4; size -= 4; result += 4; - buf += 4; } -end: + if (rd->id.use_grbm) { amdgpu_gfx_select_se_sh(adev, 0xffffffff, 0xffffffff, 0xffffffff, rd->id.xcc_id); mutex_unlock(&adev->grbm_idx_mutex); @@ -314,6 +325,10 @@ static ssize_t amdgpu_debugfs_regs2_op(struct file *f, char __user *buf, u32 off pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); amdgpu_virt_disable_access_debugfs(adev); + + if (!write_en && copy_to_user(buf, data, result)) + return -EFAULT; + return result; } @@ -1777,9 +1792,14 @@ static int amdgpu_debugfs_vm_info_show(struct seq_file *m, void *unused) list_for_each_entry(file, &dev->filelist, lhead) { struct amdgpu_fpriv *fpriv = file->driver_priv; struct amdgpu_vm *vm = &fpriv->vm; + struct amdgpu_task_info *ti; + + ti = amdgpu_vm_get_task_info_vm(vm); + if (ti) { + seq_printf(m, "pid:%d\tProcess:%s ----------\n", ti->pid, ti->process_name); + amdgpu_vm_put_task_info(ti); + } - seq_printf(m, "pid:%d\tProcess:%s ----------\n", - vm->task_info.pid, vm->task_info.process_name); r = amdgpu_bo_reserve(vm->root.bo, true); if (r) break; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c index 47d6352c2def..23e14c01bc99 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_device.c @@ -2356,6 +2356,11 @@ static int amdgpu_device_fw_loading(struct amdgpu_device *adev) static int amdgpu_device_init_schedulers(struct amdgpu_device *adev) { + struct drm_sched_init_args args = { + .ops = &amdgpu_sched_ops, + .timeout_wq = adev->reset_domain->wq, + .dev = adev->dev, + }; long timeout; int r, i; @@ -2381,11 +2386,12 @@ static int amdgpu_device_init_schedulers(struct amdgpu_device *adev) break; } - r = drm_sched_init(&ring->sched, &amdgpu_sched_ops, - ring->num_hw_submission, 0, - timeout, adev->reset_domain->wq, - ring->sched_score, ring->name, - adev->dev); + args.hw_submission = ring->num_hw_submission; + args.timeout = timeout; + args.score = ring->sched_score; + args.name = ring->name; + + r = drm_sched_init(&ring->sched, &args); if (r) { DRM_ERROR("Failed to create scheduler on ring %s.\n", ring->name); @@ -3897,8 +3903,7 @@ int amdgpu_device_init(struct amdgpu_device *adev, goto failed; } /* init i2c buses */ - if (!amdgpu_device_has_dc_support(adev)) - amdgpu_atombios_i2c_init(adev); + amdgpu_i2c_init(adev); } } @@ -4158,14 +4163,15 @@ void amdgpu_device_fini_sw(struct amdgpu_device *adev) amdgpu_reset_fini(adev); /* free i2c buses */ - if (!amdgpu_device_has_dc_support(adev)) - amdgpu_i2c_fini(adev); + amdgpu_i2c_fini(adev); if (amdgpu_emu_mode != 1) amdgpu_atombios_fini(adev); - kfree(adev->bios); - adev->bios = NULL; + amdgpu_bios_release(adev); + + kfree(adev->fru_info); + adev->fru_info = NULL; px = amdgpu_device_supports_px(adev_to_drm(adev)); @@ -4867,6 +4873,9 @@ int amdgpu_device_mode1_reset(struct amdgpu_device *adev) if (ret) goto mode1_reset_failed; + /* enable mmio access after mode 1 reset completed */ + adev->no_hw_access = false; + amdgpu_device_load_pci_state(adev->pdev); ret = amdgpu_psp_wait_for_bootloader(adev); if (ret) @@ -4929,6 +4938,18 @@ int amdgpu_device_pre_asic_reset(struct amdgpu_device *adev, amdgpu_fence_driver_force_completion(ring); } + /* + * KIQ rings are polling-fence/no_scheduler like MES, so realign their + * fence too (one ring per XCC), otherwise the first post-reset KIQ + * submission polls forever on a stale seq. + */ + for (i = 0; i < AMDGPU_MAX_GC_INSTANCES; i++) { + struct amdgpu_ring *kiq_ring = &adev->gfx.kiq[i].ring; + + if (kiq_ring->fence_drv.initialized && kiq_ring->sched.ready) + amdgpu_fence_driver_force_completion(kiq_ring); + } + amdgpu_fence_driver_isr_toggle(adev, false); if (job && job->vm) @@ -5142,9 +5163,16 @@ int amdgpu_do_asic_reset(struct list_head *device_list_handle, tmp_adev->reset_vram_lost = vram_lost; memset(&tmp_adev->reset_task_info, 0, sizeof(tmp_adev->reset_task_info)); - if (reset_context->job && reset_context->job->vm) - tmp_adev->reset_task_info = - reset_context->job->vm->task_info; + if (reset_context->job && reset_context->job->vm) { + struct amdgpu_task_info *ti; + struct amdgpu_vm *vm = reset_context->job->vm; + + ti = amdgpu_vm_get_task_info_vm(vm); + if (ti) { + tmp_adev->reset_task_info = *ti; + amdgpu_vm_put_task_info(ti); + } + } amdgpu_reset_capture_coredumpm(tmp_adev); #endif if (vram_lost) { @@ -5156,6 +5184,11 @@ int amdgpu_do_asic_reset(struct list_head *device_list_handle, if (r) return r; + r = amdgpu_xcp_restore_partition_mode( + tmp_adev->xcp_mgr); + if (r) + goto out; + r = amdgpu_device_ip_resume_phase2(tmp_adev); if (r) goto out; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c index 5c4aa5ff873b..cbde9c533146 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_discovery.c @@ -1033,6 +1033,8 @@ static int amdgpu_discovery_sysfs_ips(struct amdgpu_device *adev, ip_hw_instance->kobj.kset = &ip_hw_id->hw_id_kset; res = kobject_add(&ip_hw_instance->kobj, NULL, "%d", ip_hw_instance->num_instance); + if (res) + kobject_put(&ip_hw_instance->kobj); next_ip: if (reg_base_64) ip_offset += struct_size(ip, base_address_64, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.c index 7cd0dfaeee20..b3b87b6edeb8 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.c @@ -97,8 +97,46 @@ static bool is_fru_eeprom_supported(struct amdgpu_device *adev, u32 *fru_addr) } } +/* + * IPMI FRU Product Info Area fields are TLV: one type/length byte + * (low 6 bits = data length) followed by that many data bytes. These + * helpers walk the cursor and copy a single field while bounding all + * accesses to the actual buffer length read from the EEPROM. + */ +#define FRU_FIELD_LEN(p, a) ((p)[a] & 0x3F) + +/* Advance cursor past the current TLV. Returns false if no more data. */ +static bool fru_pia_advance(u32 *addr, const unsigned char *pia, int len) +{ + if (*addr >= (u32)len) + return false; + *addr += 1 + FRU_FIELD_LEN(pia, *addr); + return true; +} + +/* + * Copy the current TLV's data into dst (NUL-terminated). Returns false if + * the TLV header or data would read past the end of pia. + */ +static bool fru_pia_copy_field(char *dst, size_t dst_size, + const unsigned char *pia, u32 addr, int len) +{ + size_t fl; + + if (addr + 1 >= (u32)len) + return false; + + fl = min3((size_t)FRU_FIELD_LEN(pia, addr), + dst_size - 1, + (size_t)(len - addr - 1)); + memcpy(dst, pia + addr + 1, fl); + dst[fl] = '\0'; + return true; +} + int amdgpu_fru_get_product_info(struct amdgpu_device *adev) { + struct amdgpu_fru_info *fru_info; unsigned char buf[8], *pia; u32 addr, fru_addr; int size, len; @@ -107,6 +145,19 @@ int amdgpu_fru_get_product_info(struct amdgpu_device *adev) if (!is_fru_eeprom_supported(adev, &fru_addr)) return 0; + if (!adev->fru_info) { + adev->fru_info = kzalloc(sizeof(*adev->fru_info), GFP_KERNEL); + if (!adev->fru_info) + return -ENOMEM; + } + + fru_info = adev->fru_info; + /* For Arcturus-and-later, default value of serial_number is unique_id + * so convert it to a 16-digit HEX string for convenience and + * backwards-compatibility. + */ + sprintf(fru_info->serial, "%llx", adev->unique_id); + /* If algo exists, it means that the i2c_adapter's initialized */ if (!adev->pm.fru_eeprom_i2c_bus || !adev->pm.fru_eeprom_i2c_bus->algo) { DRM_WARN("Cannot access FRU, EEPROM accessor not initialized"); @@ -179,36 +230,31 @@ int amdgpu_fru_get_product_info(struct amdgpu_device *adev) * Skip the Manufacturer Name at [3] and go directly to * the Product Name field. */ - addr = 3 + 1 + (pia[3] & 0x3F); - if (addr + 1 >= len) + addr = 3; + if (!fru_pia_advance(&addr, pia, len) || + !fru_pia_copy_field(fru_info->product_name, + sizeof(fru_info->product_name), + pia, addr, len)) goto Out; - memcpy(adev->product_name, pia + addr + 1, - min_t(size_t, - sizeof(adev->product_name), - pia[addr] & 0x3F)); - adev->product_name[sizeof(adev->product_name) - 1] = '\0'; /* Go to the Product Part/Model Number field. */ - addr += 1 + (pia[addr] & 0x3F); - if (addr + 1 >= len) + if (!fru_pia_advance(&addr, pia, len) || + !fru_pia_copy_field(fru_info->product_number, + sizeof(fru_info->product_number), + pia, addr, len)) goto Out; - memcpy(adev->product_number, pia + addr + 1, - min_t(size_t, - sizeof(adev->product_number), - pia[addr] & 0x3F)); - adev->product_number[sizeof(adev->product_number) - 1] = '\0'; - - /* Go to the Product Version field. */ - addr += 1 + (pia[addr] & 0x3F); - - /* Go to the Product Serial Number field. */ - addr += 1 + (pia[addr] & 0x3F); - if (addr + 1 >= len) + + /* Skip the Product Version field. */ + if (!fru_pia_advance(&addr, pia, len)) goto Out; - memcpy(adev->serial, pia + addr + 1, min_t(size_t, - sizeof(adev->serial), - pia[addr] & 0x3F)); - adev->serial[sizeof(adev->serial) - 1] = '\0'; + + /* Read the Product Serial Number field. */ + if (!fru_pia_advance(&addr, pia, len) || + !fru_pia_copy_field(fru_info->serial, + sizeof(fru_info->serial), + pia, addr, len)) + goto Out; + Out: kfree(pia); return 0; @@ -231,7 +277,7 @@ static ssize_t amdgpu_fru_product_name_show(struct device *dev, struct drm_device *ddev = dev_get_drvdata(dev); struct amdgpu_device *adev = drm_to_adev(ddev); - return sysfs_emit(buf, "%s\n", adev->product_name); + return sysfs_emit(buf, "%s\n", adev->fru_info->product_name); } static DEVICE_ATTR(product_name, 0444, amdgpu_fru_product_name_show, NULL); @@ -253,7 +299,7 @@ static ssize_t amdgpu_fru_product_number_show(struct device *dev, struct drm_device *ddev = dev_get_drvdata(dev); struct amdgpu_device *adev = drm_to_adev(ddev); - return sysfs_emit(buf, "%s\n", adev->product_number); + return sysfs_emit(buf, "%s\n", adev->fru_info->product_number); } static DEVICE_ATTR(product_number, 0444, amdgpu_fru_product_number_show, NULL); @@ -275,7 +321,7 @@ static ssize_t amdgpu_fru_serial_number_show(struct device *dev, struct drm_device *ddev = dev_get_drvdata(dev); struct amdgpu_device *adev = drm_to_adev(ddev); - return sysfs_emit(buf, "%s\n", adev->serial); + return sysfs_emit(buf, "%s\n", adev->fru_info->serial); } static DEVICE_ATTR(serial_number, 0444, amdgpu_fru_serial_number_show, NULL); @@ -289,7 +335,7 @@ static const struct attribute *amdgpu_fru_attributes[] = { int amdgpu_fru_sysfs_init(struct amdgpu_device *adev) { - if (!is_fru_eeprom_supported(adev, NULL)) + if (!is_fru_eeprom_supported(adev, NULL) || !adev->fru_info) return 0; return sysfs_create_files(&adev->dev->kobj, amdgpu_fru_attributes); @@ -297,7 +343,7 @@ int amdgpu_fru_sysfs_init(struct amdgpu_device *adev) void amdgpu_fru_sysfs_fini(struct amdgpu_device *adev) { - if (!is_fru_eeprom_supported(adev, NULL)) + if (!is_fru_eeprom_supported(adev, NULL) || !adev->fru_info) return; sysfs_remove_files(&adev->dev->kobj, amdgpu_fru_attributes); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.h index c817db17cfa7..c99c74811c78 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_fru_eeprom.h @@ -24,6 +24,15 @@ #ifndef __AMDGPU_FRU_EEPROM_H__ #define __AMDGPU_FRU_EEPROM_H__ +#define AMDGPU_PRODUCT_NAME_LEN 64 + +/* FRU product information */ +struct amdgpu_fru_info { + char product_number[20]; + char product_name[AMDGPU_PRODUCT_NAME_LEN]; + char serial[20]; +}; + int amdgpu_fru_get_product_info(struct amdgpu_device *adev); int amdgpu_fru_sysfs_init(struct amdgpu_device *adev); void amdgpu_fru_sysfs_fini(struct amdgpu_device *adev); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c index e7b053898f9e..4ade7b68ad15 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.c @@ -803,7 +803,7 @@ int amdgpu_gfx_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *r if (r) return r; - if (adev->gfx.cp_ecc_error_irq.funcs) { + if (!amdgpu_sriov_vf(adev) && adev->gfx.cp_ecc_error_irq.funcs) { r = amdgpu_irq_get(adev, &adev->gfx.cp_ecc_error_irq, 0); if (r) goto late_fini; @@ -818,6 +818,23 @@ int amdgpu_gfx_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *r return r; } +void amdgpu_gfx_ras_suspend(struct amdgpu_device *adev, + struct ras_common_if *ras_block) +{ + if (amdgpu_ras_is_supported(adev, ras_block->block) && + !amdgpu_sriov_vf(adev) && adev->gfx.cp_ecc_error_irq.funcs) + amdgpu_irq_put(adev, &adev->gfx.cp_ecc_error_irq, 0); +} + +void amdgpu_gfx_ras_fini(struct amdgpu_device *adev, + struct ras_common_if *ras_block) +{ + if (amdgpu_ras_is_supported(adev, ras_block->block) && + !amdgpu_sriov_vf(adev) && adev->gfx.cp_ecc_error_irq.funcs) + amdgpu_irq_put(adev, &adev->gfx.cp_ecc_error_irq, 0); + amdgpu_ras_block_late_fini(adev, ras_block); +} + int amdgpu_gfx_ras_sw_init(struct amdgpu_device *adev) { int err = 0; @@ -846,6 +863,12 @@ int amdgpu_gfx_ras_sw_init(struct amdgpu_device *adev) if (!ras->ras_block.ras_late_init) ras->ras_block.ras_late_init = amdgpu_gfx_ras_late_init; + if (!ras->ras_block.ras_suspend) + ras->ras_block.ras_suspend = amdgpu_gfx_ras_suspend; + + if (!ras->ras_block.ras_fini) + ras->ras_block.ras_fini = amdgpu_gfx_ras_fini; + /* If not defined special ras_cb function, use default ras_cb */ if (!ras->ras_block.ras_cb) ras->ras_block.ras_cb = amdgpu_gfx_process_ras_data_cb; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h index 0ca95c4d4bfb..405d6b73afeb 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gfx.h @@ -516,7 +516,8 @@ bool amdgpu_gfx_is_me_queue_enabled(struct amdgpu_device *adev, int me, void amdgpu_gfx_off_ctrl(struct amdgpu_device *adev, bool enable); int amdgpu_get_gfx_off_status(struct amdgpu_device *adev, uint32_t *value); int amdgpu_gfx_ras_late_init(struct amdgpu_device *adev, struct ras_common_if *ras_block); -void amdgpu_gfx_ras_fini(struct amdgpu_device *adev); +void amdgpu_gfx_ras_suspend(struct amdgpu_device *adev, struct ras_common_if *ras_block); +void amdgpu_gfx_ras_fini(struct amdgpu_device *adev, struct ras_common_if *ras_block); int amdgpu_get_gfx_off_entrycount(struct amdgpu_device *adev, u64 *value); int amdgpu_get_gfx_off_residency(struct amdgpu_device *adev, u32 *residency); int amdgpu_set_gfx_off_residency(struct amdgpu_device *adev, bool value); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c index 1584936f2951..64004e06ad79 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_gmc.c @@ -728,20 +728,17 @@ void amdgpu_gmc_get_vbios_allocations(struct amdgpu_device *adev) case CHIP_RENOIR: adev->mman.keep_stolen_vga_memory = true; break; - case CHIP_POLARIS10: - case CHIP_POLARIS11: - case CHIP_POLARIS12: - /* MacBookPros with switchable graphics put VRAM at 0 when - * the iGPU is enabled which results in cursor issues if - * the cursor ends up at 0. Reserve vram at 0 in that case. - */ - if (adev->gmc.vram_start == 0) - adev->mman.keep_stolen_vga_memory = true; - break; default: adev->mman.keep_stolen_vga_memory = false; break; } + /* MacBookPros with switchable graphics put VRAM at 0 when + * the iGPU is enabled which results in cursor issues if + * the cursor ends up at 0. Reserve vram at 0 in that case. + */ + if ((adev->pdev->subsystem_vendor == PCI_VENDOR_ID_APPLE) && + (adev->gmc.vram_start == 0)) + adev->mman.keep_stolen_vga_memory = true; if (amdgpu_sriov_vf(adev) || !amdgpu_device_has_display_hardware(adev)) { diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c index 57516a8c5db3..03b04ee725db 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_hmm.c @@ -51,8 +51,6 @@ #include "amdgpu_amdkfd.h" #include "amdgpu_hmm.h" -#define MAX_WALK_BYTE (2UL << 30) - /** * amdgpu_hmm_invalidate_gfx - callback to notify about mm change * @@ -170,12 +168,13 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, void *owner, struct page **pages, struct hmm_range **phmm_range) { + const u64 max_bytes = SZ_2G; struct hmm_range *hmm_range; - unsigned long end; unsigned long timeout; - unsigned long i; unsigned long *pfns; - int r = 0; + unsigned long end; + unsigned long i; + int r; hmm_range = kzalloc(sizeof(*hmm_range), GFP_KERNEL); if (unlikely(!hmm_range)) @@ -196,8 +195,9 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, end = start + npages * PAGE_SIZE; hmm_range->dev_private_owner = owner; + hmm_range->notifier_seq = mmu_interval_read_begin(notifier); do { - hmm_range->end = min(hmm_range->start + MAX_WALK_BYTE, end); + hmm_range->end = min(hmm_range->start + max_bytes, end); pr_debug("hmm range: start = 0x%lx, end = 0x%lx", hmm_range->start, hmm_range->end); @@ -208,7 +208,6 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, timeout = jiffies + msecs_to_jiffies(timeout); retry: - hmm_range->notifier_seq = mmu_interval_read_begin(notifier); r = hmm_range_fault(hmm_range); if (unlikely(r)) { /* @@ -222,7 +221,7 @@ int amdgpu_hmm_range_get_pages(struct mmu_interval_notifier *notifier, if (hmm_range->end == end) break; - hmm_range->hmm_pfns += MAX_WALK_BYTE >> PAGE_SHIFT; + hmm_range->hmm_pfns += max_bytes >> PAGE_SHIFT; hmm_range->start = hmm_range->end; schedule(); } while (hmm_range->end < end); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.c index 82608df43396..d5634f9b41c6 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.c @@ -226,13 +226,12 @@ void amdgpu_i2c_destroy(struct amdgpu_i2c_chan *i2c) kfree(i2c); } -/* Add the default buses */ void amdgpu_i2c_init(struct amdgpu_device *adev) { - if (amdgpu_hw_i2c) - DRM_INFO("hw_i2c forced on, you may experience display detection problems!\n"); - - amdgpu_atombios_i2c_init(adev); + if (!adev->is_atom_fw) { + if (!amdgpu_device_has_dc_support(adev)) + amdgpu_atombios_i2c_init(adev); + } } /* remove all the buses */ @@ -248,22 +247,6 @@ void amdgpu_i2c_fini(struct amdgpu_device *adev) } } -/* Add additional buses */ -void amdgpu_i2c_add(struct amdgpu_device *adev, - const struct amdgpu_i2c_bus_rec *rec, - const char *name) -{ - struct drm_device *dev = adev_to_drm(adev); - int i; - - for (i = 0; i < AMDGPU_MAX_I2C_BUS; i++) { - if (!adev->i2c_bus[i]) { - adev->i2c_bus[i] = amdgpu_i2c_create(dev, rec, name); - return; - } - } -} - /* looks up bus based on id */ struct amdgpu_i2c_chan * amdgpu_i2c_lookup(struct amdgpu_device *adev, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.h index 63c2ff7499e1..1d3d3806e0dd 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_i2c.h @@ -30,9 +30,6 @@ struct amdgpu_i2c_chan *amdgpu_i2c_create(struct drm_device *dev, void amdgpu_i2c_destroy(struct amdgpu_i2c_chan *i2c); void amdgpu_i2c_init(struct amdgpu_device *adev); void amdgpu_i2c_fini(struct amdgpu_device *adev); -void amdgpu_i2c_add(struct amdgpu_device *adev, - const struct amdgpu_i2c_bus_rec *rec, - const char *name); struct amdgpu_i2c_chan * amdgpu_i2c_lookup(struct amdgpu_device *adev, const struct amdgpu_i2c_bus_rec *i2c_bus); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c index 69dfc699d78b..eda45c34646e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.c @@ -35,10 +35,13 @@ * PASIDs are global address space identifiers that can be shared * between the GPU, an IOMMU and the driver. VMs on different devices * may use the same PASID if they share the same address - * space. Therefore PASIDs are allocated using a global IDA. VMs are - * looked up from the PASID per amdgpu_device. + * space. Therefore PASIDs are allocated using IDR cyclic allocator + * (similar to kernel PID allocation) which naturally delays reuse. + * VMs are looked up from the PASID per amdgpu_device. */ -static DEFINE_IDA(amdgpu_pasid_ida); + +static DEFINE_IDR(amdgpu_pasid_idr); +static DEFINE_SPINLOCK(amdgpu_pasid_idr_lock); /* Helper to free pasid from a fence callback */ struct amdgpu_pasid_cb { @@ -50,8 +53,8 @@ struct amdgpu_pasid_cb { * amdgpu_pasid_alloc - Allocate a PASID * @bits: Maximum width of the PASID in bits, must be at least 1 * - * Allocates a PASID of the given width while keeping smaller PASIDs - * available if possible. + * Uses kernel's IDR cyclic allocator (same as PID allocation). + * Allocates sequentially with automatic wrap-around. * * Returns a positive integer on success. Returns %-EINVAL if bits==0. * Returns %-ENOSPC if no PASID was available. Returns %-ENOMEM on @@ -59,15 +62,15 @@ struct amdgpu_pasid_cb { */ int amdgpu_pasid_alloc(unsigned int bits) { - int pasid = -EINVAL; + int pasid; - for (bits = min(bits, 31U); bits > 0; bits--) { - pasid = ida_simple_get(&amdgpu_pasid_ida, - 1U << (bits - 1), 1U << bits, - GFP_KERNEL); - if (pasid != -ENOSPC) - break; - } + if (bits == 0) + return -EINVAL; + + spin_lock(&amdgpu_pasid_idr_lock); + pasid = idr_alloc_cyclic(&amdgpu_pasid_idr, NULL, 1, + 1U << bits, GFP_KERNEL); + spin_unlock(&amdgpu_pasid_idr_lock); if (pasid >= 0) trace_amdgpu_pasid_allocated(pasid); @@ -82,7 +85,10 @@ int amdgpu_pasid_alloc(unsigned int bits) void amdgpu_pasid_free(u32 pasid) { trace_amdgpu_pasid_freed(pasid); - ida_simple_remove(&amdgpu_pasid_ida, pasid); + + spin_lock(&amdgpu_pasid_idr_lock); + idr_remove(&amdgpu_pasid_idr, pasid); + spin_unlock(&amdgpu_pasid_idr_lock); } static void amdgpu_pasid_free_cb(struct dma_fence *fence, @@ -615,3 +621,15 @@ void amdgpu_vmid_mgr_fini(struct amdgpu_device *adev) } } } + +/** + * amdgpu_pasid_mgr_cleanup - cleanup PASID manager + * + * Cleanup the IDR allocator. + */ +void amdgpu_pasid_mgr_cleanup(void) +{ + spin_lock(&amdgpu_pasid_idr_lock); + idr_destroy(&amdgpu_pasid_idr); + spin_unlock(&amdgpu_pasid_idr_lock); +} diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h index 240fa6751260..429a072ff8fb 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ids.h @@ -75,6 +75,7 @@ int amdgpu_pasid_alloc(unsigned int bits); void amdgpu_pasid_free(u32 pasid); void amdgpu_pasid_free_delayed(struct dma_resv *resv, u32 pasid); +void amdgpu_pasid_mgr_cleanup(void); bool amdgpu_vmid_had_gpu_reset(struct amdgpu_device *adev, struct amdgpu_vmid *id); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c index 8175d831abd4..56eb3aa1d9d3 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_job.c @@ -35,7 +35,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) { struct amdgpu_ring *ring = to_amdgpu_ring(s_job->sched); struct amdgpu_job *job = to_amdgpu_job(s_job); - struct amdgpu_task_info ti; + struct amdgpu_task_info *ti; struct amdgpu_device *adev = ring->adev; int idx; int r; @@ -48,7 +48,7 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) return DRM_GPU_SCHED_STAT_ENODEV; } - memset(&ti, 0, sizeof(struct amdgpu_task_info)); + adev->job_hang = true; if (amdgpu_gpu_recovery && @@ -58,12 +58,16 @@ static enum drm_gpu_sched_stat amdgpu_job_timedout(struct drm_sched_job *s_job) goto exit; } - amdgpu_vm_get_task_info(ring->adev, job->pasid, &ti); DRM_ERROR("ring %s timeout, signaled seq=%u, emitted seq=%u\n", - job->base.sched->name, atomic_read(&ring->fence_drv.last_seq), - ring->fence_drv.sync_seq); - DRM_ERROR("Process information: process %s pid %d thread %s pid %d\n", - ti.process_name, ti.tgid, ti.task_name, ti.pid); + job->base.sched->name, atomic_read(&ring->fence_drv.last_seq), + ring->fence_drv.sync_seq); + + ti = amdgpu_vm_get_task_info_pasid(ring->adev, job->pasid); + if (ti) { + DRM_ERROR("Process information: process %s pid %d thread %s pid %d\n", + ti->process_name, ti->tgid, ti->task_name, ti->pid); + amdgpu_vm_put_task_info(ti); + } if (dma_fence_get_status(&s_job->s_fence->finished) == 0) dma_fence_set_error(&s_job->s_fence->finished, -ETIME); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c index 4c678048be63..11e837017625 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c @@ -30,6 +30,7 @@ #include #include #include +#include #include "amdgpu_uvd.h" #include "amdgpu_vce.h" #include "atom.h" @@ -1404,6 +1405,9 @@ u32 amdgpu_get_vblank_counter_kms(struct drm_crtc *crtc) DRM_ERROR("Invalid crtc %u\n", pipe); return -EINVAL; } + if (!down_read_trylock(&adev->reset_domain->sem)) + return drm_crtc_vblank_count(crtc); + /* The hw increments its frame counter at start of vsync, not at start * of vblank, as is required by DRM core vblank counter handling. @@ -1449,6 +1453,7 @@ u32 amdgpu_get_vblank_counter_kms(struct drm_crtc *crtc) DRM_DEBUG_VBL("NULL mode info! Returned count may be wrong.\n"); } + up_read(&adev->reset_domain->sem); return count; } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c index 91eebf85596b..e678fd5a0a60 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.c @@ -759,7 +759,11 @@ static int psp_load_toc(struct psp_context *psp, struct psp_gfx_cmd_resp *cmd = acquire_psp_cmd_buf(psp); /* Copy toc to psp firmware private buffer */ - psp_copy_fw(psp, psp->toc.start_addr, psp->toc.size_bytes); + ret = psp_copy_fw(psp, psp->toc.start_addr, psp->toc.size_bytes); + if (ret) { + release_psp_cmd_buf(psp); + return ret; + } psp_prep_load_toc_cmd_buf(cmd, psp->fw_pri_mc_addr, psp->toc.size_bytes); @@ -994,8 +998,11 @@ static int psp_rl_load(struct amdgpu_device *adev) cmd = acquire_psp_cmd_buf(psp); - memset(psp->fw_pri_buf, 0, PSP_1_MEG); - memcpy(psp->fw_pri_buf, psp->rl.start_addr, psp->rl.size_bytes); + ret = psp_copy_fw(psp, psp->rl.start_addr, psp->rl.size_bytes); + if (ret) { + release_psp_cmd_buf(psp); + return ret; + } cmd->cmd_id = GFX_CMD_ID_LOAD_IP_FW; cmd->cmd.cmd_load_ip_fw.fw_phy_addr_lo = lower_32_bits(psp->fw_pri_mc_addr); @@ -1188,8 +1195,12 @@ int psp_ta_load(struct psp_context *psp, struct ta_context *context) cmd = acquire_psp_cmd_buf(psp); - psp_copy_fw(psp, context->bin_desc.start_addr, - context->bin_desc.size_bytes); + ret = psp_copy_fw(psp, context->bin_desc.start_addr, + context->bin_desc.size_bytes); + if (ret) { + release_psp_cmd_buf(psp); + return ret; + } psp_prep_ta_load_cmd_buf(cmd, psp->fw_pri_mc_addr, context); @@ -3576,17 +3587,24 @@ static ssize_t psp_usbc_pd_fw_sysfs_write(struct device *dev, return count; } -void psp_copy_fw(struct psp_context *psp, uint8_t *start_addr, uint32_t bin_size) +int psp_copy_fw(struct psp_context *psp, uint8_t *start_addr, uint32_t bin_size) { int idx; if (!drm_dev_enter(adev_to_drm(psp->adev), &idx)) - return; + return -ENODEV; + + if (!bin_size || bin_size > PSP_1_MEG) { + dev_err(psp->adev->dev, "PSP firmware is invalid\n"); + drm_dev_exit(idx); + return -EINVAL; + } memset(psp->fw_pri_buf, 0, PSP_1_MEG); memcpy(psp->fw_pri_buf, start_addr, bin_size); drm_dev_exit(idx); + return 0; } /** diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h index 3e67ed63e638..2431bfb59867 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_psp.h @@ -528,7 +528,7 @@ int psp_get_fw_attestation_records_addr(struct psp_context *psp, int psp_load_fw_list(struct psp_context *psp, struct amdgpu_firmware_info **ucode_list, int ucode_count); -void psp_copy_fw(struct psp_context *psp, uint8_t *start_addr, uint32_t bin_size); +int psp_copy_fw(struct psp_context *psp, uint8_t *start_addr, uint32_t bin_size); int psp_spatial_partition(struct psp_context *psp, int mode); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c index 4214bbd7a1a2..65a5edcb12ba 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.c @@ -86,6 +86,9 @@ struct amdgpu_ras_block_list { struct list_head node; struct amdgpu_ras_block_object *ras_obj; + + /* set by ras_late_init, cleared by ras_suspend/ras_fini */ + bool active; }; const char *get_ras_block_str(struct ras_common_if *ras_block) @@ -2886,10 +2889,23 @@ void amdgpu_ras_resume(struct amdgpu_device *adev) void amdgpu_ras_suspend(struct amdgpu_device *adev) { struct amdgpu_ras *con = amdgpu_ras_get_context(adev); + struct amdgpu_ras_block_list *node; + struct amdgpu_ras_block_object *obj; if (!adev->ras_enabled || !con) return; + /* run per-block ras_suspend before tearing down the RAS context */ + list_for_each_entry(node, &adev->ras_list, node) { + if (!node->active) + continue; + + obj = node->ras_obj; + if (obj && obj->ras_suspend) + obj->ras_suspend(adev, &obj->ras_comm); + node->active = false; + } + amdgpu_ras_disable_all_features(adev, 0); /* Make sure all ras objects are disabled. */ if (con->features) @@ -2920,8 +2936,15 @@ int amdgpu_ras_late_init(struct amdgpu_device *adev) obj->ras_comm.name, r); return r; } - } else - amdgpu_ras_block_late_init_default(adev, &obj->ras_comm); + } else { + r = amdgpu_ras_block_late_init_default(adev, &obj->ras_comm); + if (r) { + dev_err(adev->dev, "%s failed to execute ras_block_late_init_default! ret:%d\n", + obj->ras_comm.name, r); + return r; + } + } + node->active = true; } return 0; @@ -2955,11 +2978,12 @@ int amdgpu_ras_fini(struct amdgpu_device *adev) list_for_each_entry_safe(ras_node, tmp, &adev->ras_list, node) { if (ras_node->ras_obj) { obj = ras_node->ras_obj; - if (amdgpu_ras_is_supported(adev, obj->ras_comm.block) && - obj->ras_fini) + /* fall back to default cleanup if ras_suspend already ran */ + if (ras_node->active && obj->ras_fini) obj->ras_fini(adev, &obj->ras_comm); else amdgpu_ras_block_late_fini_default(adev, &obj->ras_comm); + ras_node->active = false; } /* Clear ras blocks from ras_list and free ras block list node */ diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h index ffb49b2d533a..742449833c8b 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_ras.h @@ -551,6 +551,7 @@ struct amdgpu_ras_block_object { int (*ras_block_match)(struct amdgpu_ras_block_object *block_obj, enum amdgpu_ras_block block, uint32_t sub_block_index); int (*ras_late_init)(struct amdgpu_device *adev, struct ras_common_if *ras_block); + void (*ras_suspend)(struct amdgpu_device *adev, struct ras_common_if *ras_block); void (*ras_fini)(struct amdgpu_device *adev, struct ras_common_if *ras_block); ras_ih_cb ras_cb; const struct amdgpu_ras_block_hw_ops *hw_ops; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_umc.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_umc.c index db0d94ca4ffc..cf6ebb10acc9 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_umc.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_umc.c @@ -162,6 +162,7 @@ static int amdgpu_umc_do_page_retirement(struct amdgpu_device *adev, } kfree(err_data->err_addr); + err_data->err_addr = NULL; return AMDGPU_RAS_SUCCESS; } diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c index aab293ec3417..52b5ddc923c8 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_uvd.c @@ -511,7 +511,8 @@ int amdgpu_uvd_resume(struct amdgpu_device *adev) } memset_io(ptr, 0, size); /* to restore uvd fence seq */ - amdgpu_fence_driver_force_completion(&adev->uvd.inst[i].ring); + if (adev->uvd.inst[i].ring.fence_drv.initialized) + amdgpu_fence_driver_force_completion(&adev->uvd.inst[i].ring); } } return 0; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c index 8329b5c60072..c653580cbdd4 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.c @@ -2134,6 +2134,108 @@ long amdgpu_vm_wait_idle(struct amdgpu_vm *vm, long timeout) return dma_fence_wait_timeout(vm->last_unlocked, true, timeout); } +static void amdgpu_vm_destroy_task_info(struct kref *kref) +{ + struct amdgpu_task_info *ti = container_of(kref, struct amdgpu_task_info, refcount); + + kfree(ti); +} + +static inline struct amdgpu_vm * +amdgpu_vm_get_vm_from_pasid(struct amdgpu_device *adev, u32 pasid) +{ + struct amdgpu_vm *vm; + unsigned long flags; + + xa_lock_irqsave(&adev->vm_manager.pasids, flags); + vm = xa_load(&adev->vm_manager.pasids, pasid); + xa_unlock_irqrestore(&adev->vm_manager.pasids, flags); + + return vm; +} + +/** + * amdgpu_vm_put_task_info - reference down the vm task_info ptr + * + * @task_info: task_info struct under discussion. + * + * frees the vm task_info ptr at the last put + */ +void amdgpu_vm_put_task_info(struct amdgpu_task_info *task_info) +{ + kref_put(&task_info->refcount, amdgpu_vm_destroy_task_info); +} + +/** + * amdgpu_vm_get_task_info_vm - Extracts task info for a vm. + * + * @vm: VM to get info from + * + * Returns the reference counted task_info structure, which must be + * referenced down with amdgpu_vm_put_task_info. + */ +struct amdgpu_task_info * +amdgpu_vm_get_task_info_vm(struct amdgpu_vm *vm) +{ + struct amdgpu_task_info *ti = NULL; + + if (vm) { + ti = vm->task_info; + kref_get(&vm->task_info->refcount); + } + + return ti; +} + +/** + * amdgpu_vm_get_task_info_pasid - Extracts task info for a PASID. + * + * @adev: drm device pointer + * @pasid: PASID identifier for VM + * + * Returns the reference counted task_info structure, which must be + * referenced down with amdgpu_vm_put_task_info. + */ +struct amdgpu_task_info * +amdgpu_vm_get_task_info_pasid(struct amdgpu_device *adev, u32 pasid) +{ + return amdgpu_vm_get_task_info_vm( + amdgpu_vm_get_vm_from_pasid(adev, pasid)); +} + +static int amdgpu_vm_create_task_info(struct amdgpu_vm *vm) +{ + vm->task_info = kzalloc(sizeof(struct amdgpu_task_info), GFP_KERNEL); + if (!vm->task_info) + return -ENOMEM; + + kref_init(&vm->task_info->refcount); + return 0; +} + +/** + * amdgpu_vm_set_task_info - Sets VMs task info. + * + * @vm: vm for which to set the info + */ +void amdgpu_vm_set_task_info(struct amdgpu_vm *vm) +{ + if (!vm->task_info) + return; + + if (vm->task_info->pid == current->pid) + return; + + vm->task_info->pid = current->pid; + get_task_comm(vm->task_info->task_name, current); + + if (current->group_leader->mm != current->mm) + return; + + vm->task_info->tgid = current->group_leader->pid; + get_task_comm(vm->task_info->process_name, current->group_leader); +} + /** * amdgpu_vm_init - initialize a vm instance * @@ -2219,6 +2321,10 @@ int amdgpu_vm_init(struct amdgpu_device *adev, struct amdgpu_vm *vm, if (r) goto error_free_root; + r = amdgpu_vm_create_task_info(vm); + if (r) + DRM_DEBUG("Failed to create task info for VM\n"); + amdgpu_bo_unreserve(vm->root.bo); amdgpu_bo_unref(&root_bo); @@ -2359,6 +2465,7 @@ void amdgpu_vm_fini(struct amdgpu_device *adev, struct amdgpu_vm *vm) root = amdgpu_bo_ref(vm->root.bo); amdgpu_bo_reserve(root, true); + amdgpu_vm_put_task_info(vm->task_info); amdgpu_vm_set_pasid(adev, vm, 0); dma_fence_wait(vm->last_unlocked, false); dma_fence_put(vm->last_unlocked); @@ -2471,6 +2578,7 @@ void amdgpu_vm_manager_fini(struct amdgpu_device *adev) xa_destroy(&adev->vm_manager.pasids); amdgpu_vmid_mgr_fini(adev); + amdgpu_pasid_mgr_cleanup(); } /** @@ -2517,48 +2625,6 @@ int amdgpu_vm_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) return 0; } -/** - * amdgpu_vm_get_task_info - Extracts task info for a PASID. - * - * @adev: drm device pointer - * @pasid: PASID identifier for VM - * @task_info: task_info to fill. - */ -void amdgpu_vm_get_task_info(struct amdgpu_device *adev, u32 pasid, - struct amdgpu_task_info *task_info) -{ - struct amdgpu_vm *vm; - unsigned long flags; - - xa_lock_irqsave(&adev->vm_manager.pasids, flags); - - vm = xa_load(&adev->vm_manager.pasids, pasid); - if (vm) - *task_info = vm->task_info; - - xa_unlock_irqrestore(&adev->vm_manager.pasids, flags); -} - -/** - * amdgpu_vm_set_task_info - Sets VMs task info. - * - * @vm: vm for which to set the info - */ -void amdgpu_vm_set_task_info(struct amdgpu_vm *vm) -{ - if (vm->task_info.pid) - return; - - vm->task_info.pid = current->pid; - get_task_comm(vm->task_info.task_name, current); - - if (current->group_leader->mm != current->mm) - return; - - vm->task_info.tgid = current->group_leader->pid; - get_task_comm(vm->task_info.process_name, current->group_leader); -} - /** * amdgpu_vm_handle_fault - graceful handling of VM faults. * @adev: amdgpu device pointer diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h index 204ab13184ed..c5500549ae66 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm.h @@ -183,10 +183,11 @@ struct amdgpu_vm_pte_funcs { }; struct amdgpu_task_info { - char process_name[TASK_COMM_LEN]; - char task_name[TASK_COMM_LEN]; - pid_t pid; - pid_t tgid; + char process_name[TASK_COMM_LEN]; + char task_name[TASK_COMM_LEN]; + pid_t pid; + pid_t tgid; + struct kref refcount; }; /** @@ -334,7 +335,7 @@ struct amdgpu_vm { uint64_t pd_phys_addr; /* Some basic info about the task */ - struct amdgpu_task_info task_info; + struct amdgpu_task_info *task_info; /* Store positions of group of BOs */ struct ttm_lru_bulk_move lru_bulk_move; @@ -466,8 +467,14 @@ bool amdgpu_vm_need_pipeline_sync(struct amdgpu_ring *ring, struct amdgpu_job *job); void amdgpu_vm_check_compute_bug(struct amdgpu_device *adev); -void amdgpu_vm_get_task_info(struct amdgpu_device *adev, u32 pasid, - struct amdgpu_task_info *task_info); +struct amdgpu_task_info * +amdgpu_vm_get_task_info_pasid(struct amdgpu_device *adev, u32 pasid); + +struct amdgpu_task_info * +amdgpu_vm_get_task_info_vm(struct amdgpu_vm *vm); + +void amdgpu_vm_put_task_info(struct amdgpu_task_info *task_info); + bool amdgpu_vm_handle_fault(struct amdgpu_device *adev, u32 pasid, u32 vmid, u32 node_id, uint64_t addr, bool write_fault); diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c index 026a3db94729..56b619b0933d 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vm_pt.c @@ -1037,7 +1037,7 @@ int amdgpu_vm_ptes_update(struct amdgpu_vm_update_params *params, trace_amdgpu_vm_update_ptes(params, frag_start, upd_end, min(nptes, 32u), dst, incr, upd_flags, - vm->task_info.tgid, + vm->task_info ? vm->task_info->tgid : 0, vm->immediate.fence_context); amdgpu_vm_pte_update_flags(params, to_amdgpu_bo_vm(pt), cursor.level, pe_start, dst, diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c index 451c37d04e45..a592f80e940e 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_vram_mgr.c @@ -212,8 +212,26 @@ static struct attribute *amdgpu_vram_mgr_attributes[] = { NULL }; +static umode_t amdgpu_vram_attrs_is_visible(struct kobject *kobj, + struct attribute *attr, int i) +{ + struct device *dev = kobj_to_dev(kobj); + struct drm_device *ddev = dev_get_drvdata(dev); + struct amdgpu_device *adev = drm_to_adev(ddev); + + if (attr == &dev_attr_mem_info_vram_vendor.attr && + !adev->gmc.vram_vendor) + return 0; + + if (!ttm_resource_manager_used(&adev->mman.vram_mgr.manager)) + return 0; + + return attr->mode; +} + const struct attribute_group amdgpu_vram_mgr_attr_group = { - .attrs = amdgpu_vram_mgr_attributes + .attrs = amdgpu_vram_mgr_attributes, + .is_visible = amdgpu_vram_attrs_is_visible }; /** diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c index 565a1fa436d4..2b99eed5ba19 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.c @@ -163,16 +163,11 @@ int amdgpu_xcp_init(struct amdgpu_xcp_mgr *xcp_mgr, int num_xcps, int mode) return 0; } -int amdgpu_xcp_switch_partition_mode(struct amdgpu_xcp_mgr *xcp_mgr, int mode) +static int __amdgpu_xcp_switch_partition_mode(struct amdgpu_xcp_mgr *xcp_mgr, + int mode) { int ret, curr_mode, num_xcps = 0; - if (!xcp_mgr || mode == AMDGPU_XCP_MODE_NONE) - return -EINVAL; - - if (xcp_mgr->mode == mode) - return 0; - if (!xcp_mgr->funcs || !xcp_mgr->funcs->switch_partition_mode) return 0; @@ -201,6 +196,25 @@ int amdgpu_xcp_switch_partition_mode(struct amdgpu_xcp_mgr *xcp_mgr, int mode) return ret; } +int amdgpu_xcp_switch_partition_mode(struct amdgpu_xcp_mgr *xcp_mgr, int mode) +{ + if (!xcp_mgr || mode == AMDGPU_XCP_MODE_NONE) + return -EINVAL; + + if (xcp_mgr->mode == mode) + return 0; + + return __amdgpu_xcp_switch_partition_mode(xcp_mgr, mode); +} + +int amdgpu_xcp_restore_partition_mode(struct amdgpu_xcp_mgr *xcp_mgr) +{ + if (!xcp_mgr || xcp_mgr->mode == AMDGPU_XCP_MODE_NONE) + return 0; + + return __amdgpu_xcp_switch_partition_mode(xcp_mgr, xcp_mgr->mode); +} + int amdgpu_xcp_query_partition_mode(struct amdgpu_xcp_mgr *xcp_mgr, u32 flags) { int mode; diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.h b/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.h index 9142238e7791..32775260556f 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.h +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_xcp.h @@ -129,6 +129,7 @@ int amdgpu_xcp_mgr_init(struct amdgpu_device *adev, int init_mode, int amdgpu_xcp_init(struct amdgpu_xcp_mgr *xcp_mgr, int num_xcps, int mode); int amdgpu_xcp_query_partition_mode(struct amdgpu_xcp_mgr *xcp_mgr, u32 flags); int amdgpu_xcp_switch_partition_mode(struct amdgpu_xcp_mgr *xcp_mgr, int mode); +int amdgpu_xcp_restore_partition_mode(struct amdgpu_xcp_mgr *xcp_mgr); int amdgpu_xcp_get_partition(struct amdgpu_xcp_mgr *xcp_mgr, enum AMDGPU_XCP_IP_BLOCK ip, int instance); diff --git a/drivers/gpu/drm/amd/amdgpu/aqua_vanjaram.c b/drivers/gpu/drm/amd/amdgpu/aqua_vanjaram.c index fb20faa2c4ff..5989264913e5 100644 --- a/drivers/gpu/drm/amd/amdgpu/aqua_vanjaram.c +++ b/drivers/gpu/drm/amd/amdgpu/aqua_vanjaram.c @@ -513,7 +513,8 @@ static int aqua_vanjaram_switch_partition_mode(struct amdgpu_xcp_mgr *xcp_mgr, return -EINVAL; } - if (adev->kfd.init_complete) + if (adev->kfd.init_complete && !amdgpu_in_reset(adev) && + !adev->in_suspend) flags |= AMDGPU_XCP_OPS_KFD; if (flags & AMDGPU_XCP_OPS_KFD) { diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c index 131be358f920..c3b92ba3f821 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v11_0.c @@ -5284,8 +5284,6 @@ static void gfx_v11_0_ring_emit_ib_gfx(struct amdgpu_ring *ring, unsigned vmid = AMDGPU_JOB_GET_VMID(job); u32 header, control = 0; - BUG_ON(ib->flags & AMDGPU_IB_FLAG_CE); - header = PACKET3(PACKET3_INDIRECT_BUFFER, 2); control |= ib->length_dw | (vmid << 24); diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c index c2a6fcc4eda7..58bea7b24b47 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v6_0.c @@ -308,6 +308,14 @@ static const u32 verde_rlc_save_restore_register_list[] = 0x00000000 }; +static void gfx_v6_0_free_microcode(struct amdgpu_device *adev) +{ + amdgpu_ucode_release(&adev->gfx.pfp_fw); + amdgpu_ucode_release(&adev->gfx.me_fw); + amdgpu_ucode_release(&adev->gfx.ce_fw); + amdgpu_ucode_release(&adev->gfx.rlc_fw); +} + static int gfx_v6_0_init_microcode(struct amdgpu_device *adev) { const char *chip_name; @@ -3185,6 +3193,7 @@ static int gfx_v6_0_sw_fini(void *handle) amdgpu_ring_fini(&adev->gfx.compute_ring[i]); amdgpu_gfx_rlc_fini(adev); + gfx_v6_0_free_microcode(adev); return 0; } diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c index b1b4f75ebad3..4bcd5a93415b 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_0.c @@ -3776,8 +3776,6 @@ static int gfx_v9_0_hw_fini(void *handle) { struct amdgpu_device *adev = (struct amdgpu_device *)handle; - if (amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__GFX)) - amdgpu_irq_put(adev, &adev->gfx.cp_ecc_error_irq, 0); amdgpu_irq_put(adev, &adev->gfx.priv_reg_irq, 0); amdgpu_irq_put(adev, &adev->gfx.priv_inst_irq, 0); diff --git a/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c b/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c index 1164937e8d20..c7b868c8fa17 100644 --- a/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c +++ b/drivers/gpu/drm/amd/amdgpu/gfx_v9_4_3.c @@ -1948,9 +1948,11 @@ static int gfx_v9_4_3_cp_resume(struct amdgpu_device *adev) { int r = 0, i, num_xcc; - if (amdgpu_xcp_query_partition_mode(adev->xcp_mgr, - AMDGPU_XCP_FL_NONE) == - AMDGPU_UNKNOWN_COMPUTE_PARTITION_MODE) + if (adev->in_suspend) + amdgpu_xcp_restore_partition_mode(adev->xcp_mgr); + else if (amdgpu_xcp_query_partition_mode(adev->xcp_mgr, + AMDGPU_XCP_FL_NONE) == + AMDGPU_UNKNOWN_COMPUTE_PARTITION_MODE) r = amdgpu_xcp_switch_partition_mode(adev->xcp_mgr, amdgpu_user_partt_mode); diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c index 62ecf4d89cb9..541aa0d1c800 100644 --- a/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v10_0.c @@ -107,7 +107,7 @@ static int gmc_v10_0_process_interrupt(struct amdgpu_device *adev, struct amdgpu_vmhub *hub = &adev->vmhub[vmhub_index]; bool retry_fault = !!(entry->src_data[1] & 0x80); bool write_fault = !!(entry->src_data[1] & 0x20); - struct amdgpu_task_info task_info; + struct amdgpu_task_info *task_info; uint32_t status = 0; u64 addr; @@ -155,18 +155,22 @@ static int gmc_v10_0_process_interrupt(struct amdgpu_device *adev, if (!printk_ratelimit()) return 0; - memset(&task_info, 0, sizeof(struct amdgpu_task_info)); - amdgpu_vm_get_task_info(adev, entry->pasid, &task_info); - dev_err(adev->dev, - "[%s] page fault (src_id:%u ring:%u vmid:%u pasid:%u, for process %s pid %d thread %s pid %d)\n", + "[%s] page fault (src_id:%u ring:%u vmid:%u pasid:%u)\n", entry->vmid_src ? "mmhub" : "gfxhub", - entry->src_id, entry->ring_id, entry->vmid, - entry->pasid, task_info.process_name, task_info.tgid, - task_info.task_name, task_info.pid); + entry->src_id, entry->ring_id, entry->vmid, entry->pasid); + task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid); + if (task_info) { + dev_err(adev->dev, + " in process %s pid %d thread %s pid %d\n", + task_info->process_name, task_info->tgid, + task_info->task_name, task_info->pid); + amdgpu_vm_put_task_info(task_info); + } + dev_err(adev->dev, " in page starting at address 0x%016llx from client 0x%x (%s)\n", - addr, entry->client_id, - soc15_ih_clientid_name[entry->client_id]); + addr, entry->client_id, + soc15_ih_clientid_name[entry->client_id]); if (!amdgpu_sriov_vf(adev)) hub->vmhub_funcs->print_l2_protection_fault_status(adev, diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c index 3d797a1adef3..88ae81aadce5 100644 --- a/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v11_0.c @@ -120,19 +120,24 @@ static int gmc_v11_0_process_interrupt(struct amdgpu_device *adev, } if (printk_ratelimit()) { - struct amdgpu_task_info task_info; - - memset(&task_info, 0, sizeof(struct amdgpu_task_info)); - amdgpu_vm_get_task_info(adev, entry->pasid, &task_info); + struct amdgpu_task_info *task_info; dev_err(adev->dev, - "[%s] page fault (src_id:%u ring:%u vmid:%u pasid:%u, for process %s pid %d thread %s pid %d)\n", + "[%s] page fault (src_id:%u ring:%u vmid:%u pasid:%u)\n", entry->vmid_src ? "mmhub" : "gfxhub", - entry->src_id, entry->ring_id, entry->vmid, - entry->pasid, task_info.process_name, task_info.tgid, - task_info.task_name, task_info.pid); + entry->src_id, entry->ring_id, entry->vmid, entry->pasid); + task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid); + if (task_info) { + dev_err(adev->dev, + " in process %s pid %d thread %s pid %d)\n", + task_info->process_name, task_info->tgid, + task_info->task_name, task_info->pid); + amdgpu_vm_put_task_info(task_info); + } + dev_err(adev->dev, " in page starting at address 0x%016llx from client %d\n", - addr, entry->client_id); + addr, entry->client_id); + if (!amdgpu_sriov_vf(adev)) hub->vmhub_funcs->print_l2_protection_fault_status(adev, status); } diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c index ed268c5739eb..13606e594fa8 100644 --- a/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v8_0.c @@ -1446,18 +1446,24 @@ static int gmc_v8_0_process_interrupt(struct amdgpu_device *adev, gmc_v8_0_set_fault_enable_default(adev, false); if (printk_ratelimit()) { - struct amdgpu_task_info task_info; + struct amdgpu_task_info *task_info; - memset(&task_info, 0, sizeof(struct amdgpu_task_info)); - amdgpu_vm_get_task_info(adev, entry->pasid, &task_info); + dev_err(adev->dev, "GPU fault detected: %d 0x%08x\n", + entry->src_id, entry->src_data[0]); + + task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid); + if (task_info) { + dev_err(adev->dev, " for process %s pid %d thread %s pid %d\n", + task_info->process_name, task_info->tgid, + task_info->task_name, task_info->pid); + amdgpu_vm_put_task_info(task_info); + } - dev_err(adev->dev, "GPU fault detected: %d 0x%08x for process %s pid %d thread %s pid %d\n", - entry->src_id, entry->src_data[0], task_info.process_name, - task_info.tgid, task_info.task_name, task_info.pid); dev_err(adev->dev, " VM_CONTEXT1_PROTECTION_FAULT_ADDR 0x%08X\n", - addr); + addr); dev_err(adev->dev, " VM_CONTEXT1_PROTECTION_FAULT_STATUS 0x%08X\n", status); + gmc_v8_0_vm_decode_fault(adev, status, addr, mc_client, entry->pasid); } diff --git a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c index fdd36e3b2f14..79d0f5c2d7f2 100644 --- a/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c +++ b/drivers/gpu/drm/amd/amdgpu/gmc_v9_0.c @@ -550,7 +550,7 @@ static int gmc_v9_0_process_interrupt(struct amdgpu_device *adev, bool retry_fault = !!(entry->src_data[1] & 0x80); bool write_fault = !!(entry->src_data[1] & 0x20); uint32_t status = 0, cid = 0, rw = 0; - struct amdgpu_task_info task_info; + struct amdgpu_task_info *task_info; struct amdgpu_vmhub *hub; const char *mmhub_cid; const char *hub_name; @@ -625,16 +625,20 @@ static int gmc_v9_0_process_interrupt(struct amdgpu_device *adev, if (!printk_ratelimit()) return 0; - - memset(&task_info, 0, sizeof(struct amdgpu_task_info)); - amdgpu_vm_get_task_info(adev, entry->pasid, &task_info); - dev_err(adev->dev, - "[%s] %s page fault (src_id:%u ring:%u vmid:%u pasid:%u, for process %s pid %d thread %s pid %d)\n", - hub_name, retry_fault ? "retry" : "no-retry", - entry->src_id, entry->ring_id, entry->vmid, - entry->pasid, task_info.process_name, task_info.tgid, - task_info.task_name, task_info.pid); + "[%s] %s page fault (src_id:%u ring:%u vmid:%u pasid:%u)\n", hub_name, + retry_fault ? "retry" : "no-retry", + entry->src_id, entry->ring_id, entry->vmid, entry->pasid); + + task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid); + if (task_info) { + dev_err(adev->dev, + " for process %s pid %d thread %s pid %d)\n", + task_info->process_name, task_info->tgid, + task_info->task_name, task_info->pid); + amdgpu_vm_put_task_info(task_info); + } + dev_err(adev->dev, " in page starting at address 0x%016llx from IH client 0x%x (%s)\n", addr, entry->client_id, soc15_ih_clientid_name[entry->client_id]); diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v11_0.c b/drivers/gpu/drm/amd/amdgpu/psp_v11_0.c index 8f84fe40abbb..7f6c45539eb8 100644 --- a/drivers/gpu/drm/amd/amdgpu/psp_v11_0.c +++ b/drivers/gpu/drm/amd/amdgpu/psp_v11_0.c @@ -193,7 +193,9 @@ static int psp_v11_0_bootloader_load_component(struct psp_context *psp, return ret; /* Copy PSP System Driver binary to memory */ - psp_copy_fw(psp, bin_desc->start_addr, bin_desc->size_bytes); + ret = psp_copy_fw(psp, bin_desc->start_addr, bin_desc->size_bytes); + if (ret) + return ret; /* Provide the sys driver to bootloader */ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36, @@ -239,7 +241,9 @@ static int psp_v11_0_bootloader_load_sos(struct psp_context *psp) return ret; /* Copy Secure OS binary to PSP memory */ - psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes); + ret = psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes); + if (ret) + return ret; /* Provide the PSP secure OS to bootloader */ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36, diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v12_0.c b/drivers/gpu/drm/amd/amdgpu/psp_v12_0.c index fcd708eae75c..14e6cd44cc01 100644 --- a/drivers/gpu/drm/amd/amdgpu/psp_v12_0.c +++ b/drivers/gpu/drm/amd/amdgpu/psp_v12_0.c @@ -90,7 +90,9 @@ static int psp_v12_0_bootloader_load_sysdrv(struct psp_context *psp) return ret; /* Copy PSP System Driver binary to memory */ - psp_copy_fw(psp, psp->sys.start_addr, psp->sys.size_bytes); + ret = psp_copy_fw(psp, psp->sys.start_addr, psp->sys.size_bytes); + if (ret) + return ret; /* Provide the sys driver to bootloader */ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36, @@ -129,7 +131,9 @@ static int psp_v12_0_bootloader_load_sos(struct psp_context *psp) return ret; /* Copy Secure OS binary to PSP memory */ - psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes); + ret = psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes); + if (ret) + return ret; /* Provide the PSP secure OS to bootloader */ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36, diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c b/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c index fe1995ed13be..a1ad01056efb 100644 --- a/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c +++ b/drivers/gpu/drm/amd/amdgpu/psp_v13_0.c @@ -215,10 +215,9 @@ static int psp_v13_0_bootloader_load_component(struct psp_context *psp, if (ret) return ret; - memset(psp->fw_pri_buf, 0, PSP_1_MEG); - - /* Copy PSP KDB binary to memory */ - memcpy(psp->fw_pri_buf, bin_desc->start_addr, bin_desc->size_bytes); + ret = psp_copy_fw(psp, bin_desc->start_addr, bin_desc->size_bytes); + if (ret) + return ret; /* Provide the PSP KDB to bootloader */ WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_36, @@ -284,10 +283,9 @@ static int psp_v13_0_bootloader_load_sos(struct psp_context *psp) if (ret) return ret; - memset(psp->fw_pri_buf, 0, PSP_1_MEG); - - /* Copy Secure OS binary to PSP memory */ - memcpy(psp->fw_pri_buf, psp->sos.start_addr, psp->sos.size_bytes); + ret = psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes); + if (ret) + return ret; /* Provide the PSP secure OS to bootloader */ WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_36, diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v13_0_4.c b/drivers/gpu/drm/amd/amdgpu/psp_v13_0_4.c index d5ba58eba3e2..28a2f2ca6246 100644 --- a/drivers/gpu/drm/amd/amdgpu/psp_v13_0_4.c +++ b/drivers/gpu/drm/amd/amdgpu/psp_v13_0_4.c @@ -107,10 +107,9 @@ static int psp_v13_0_4_bootloader_load_component(struct psp_context *psp, if (ret) return ret; - memset(psp->fw_pri_buf, 0, PSP_1_MEG); - - /* Copy PSP KDB binary to memory */ - memcpy(psp->fw_pri_buf, bin_desc->start_addr, bin_desc->size_bytes); + ret = psp_copy_fw(psp, bin_desc->start_addr, bin_desc->size_bytes); + if (ret) + return ret; /* Provide the PSP KDB to bootloader */ WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_36, @@ -170,10 +169,9 @@ static int psp_v13_0_4_bootloader_load_sos(struct psp_context *psp) if (ret) return ret; - memset(psp->fw_pri_buf, 0, PSP_1_MEG); - - /* Copy Secure OS binary to PSP memory */ - memcpy(psp->fw_pri_buf, psp->sos.start_addr, psp->sos.size_bytes); + ret = psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes); + if (ret) + return ret; /* Provide the PSP secure OS to bootloader */ WREG32_SOC15(MP0, 0, regMP0_SMN_C2PMSG_36, diff --git a/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c b/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c index f6b75e3e47ff..b0c1046c355d 100644 --- a/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c +++ b/drivers/gpu/drm/amd/amdgpu/psp_v3_1.c @@ -96,7 +96,9 @@ static int psp_v3_1_bootloader_load_sysdrv(struct psp_context *psp) return ret; /* Copy PSP System Driver binary to memory */ - psp_copy_fw(psp, psp->sys.start_addr, psp->sys.size_bytes); + ret = psp_copy_fw(psp, psp->sys.start_addr, psp->sys.size_bytes); + if (ret) + return ret; /* Provide the sys driver to bootloader */ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36, @@ -135,7 +137,9 @@ static int psp_v3_1_bootloader_load_sos(struct psp_context *psp) return ret; /* Copy Secure OS binary to PSP memory */ - psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes); + ret = psp_copy_fw(psp, psp->sos.start_addr, psp->sos.size_bytes); + if (ret) + return ret; /* Provide the PSP secure OS to bootloader */ WREG32_SOC15(MP0, 0, mmMP0_SMN_C2PMSG_36, diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c index 6a26428572ec..a7f81f570fcc 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_0.c @@ -2115,7 +2115,7 @@ static int sdma_v4_0_print_iv_entry(struct amdgpu_device *adev, struct amdgpu_iv_entry *entry) { int instance; - struct amdgpu_task_info task_info; + struct amdgpu_task_info *task_info; u64 addr; instance = sdma_v4_0_irq_id_to_seq(entry->client_id); @@ -2127,15 +2127,20 @@ static int sdma_v4_0_print_iv_entry(struct amdgpu_device *adev, addr = (u64)entry->src_data[0] << 12; addr |= ((u64)entry->src_data[1] & 0xf) << 44; - memset(&task_info, 0, sizeof(struct amdgpu_task_info)); - amdgpu_vm_get_task_info(adev, entry->pasid, &task_info); - dev_dbg_ratelimited(adev->dev, - "[sdma%d] address:0x%016llx src_id:%u ring:%u vmid:%u " - "pasid:%u, for process %s pid %d thread %s pid %d\n", - instance, addr, entry->src_id, entry->ring_id, entry->vmid, - entry->pasid, task_info.process_name, task_info.tgid, - task_info.task_name, task_info.pid); + "[sdma%d] address:0x%016llx src_id:%u ring:%u vmid:%u pasid:%u\n", + instance, addr, entry->src_id, entry->ring_id, entry->vmid, + entry->pasid); + + task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid); + if (task_info) { + dev_dbg_ratelimited(adev->dev, + " for process %s pid %d thread %s pid %d\n", + task_info->process_name, task_info->tgid, + task_info->task_name, task_info->pid); + amdgpu_vm_put_task_info(task_info); + } + return 0; } diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c b/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c index 2b9b19b6df5f..a8927bf9a29e 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v4_4_2.c @@ -1624,7 +1624,7 @@ static int sdma_v4_4_2_print_iv_entry(struct amdgpu_device *adev, struct amdgpu_iv_entry *entry) { int instance; - struct amdgpu_task_info task_info; + struct amdgpu_task_info *task_info; u64 addr; instance = sdma_v4_4_2_irq_id_to_seq(entry->client_id); @@ -1636,15 +1636,19 @@ static int sdma_v4_4_2_print_iv_entry(struct amdgpu_device *adev, addr = (u64)entry->src_data[0] << 12; addr |= ((u64)entry->src_data[1] & 0xf) << 44; - memset(&task_info, 0, sizeof(struct amdgpu_task_info)); - amdgpu_vm_get_task_info(adev, entry->pasid, &task_info); - dev_dbg_ratelimited(adev->dev, - "[sdma%d] address:0x%016llx src_id:%u ring:%u vmid:%u " - "pasid:%u, for process %s pid %d thread %s pid %d\n", - instance, addr, entry->src_id, entry->ring_id, entry->vmid, - entry->pasid, task_info.process_name, task_info.tgid, - task_info.task_name, task_info.pid); + "[sdma%d] address:0x%016llx src_id:%u ring:%u vmid:%u pasid:%u\n", + instance, addr, entry->src_id, entry->ring_id, entry->vmid, + entry->pasid); + + task_info = amdgpu_vm_get_task_info_pasid(adev, entry->pasid); + if (task_info) { + dev_dbg_ratelimited(adev->dev, " for process %s pid %d thread %s pid %d\n", + task_info->process_name, task_info->tgid, + task_info->task_name, task_info->pid); + amdgpu_vm_put_task_info(task_info); + } + return 0; } diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c b/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c index fa0459ed079c..b69096930e63 100644 --- a/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c +++ b/drivers/gpu/drm/amd/amdgpu/vcn_v2_5.c @@ -355,7 +355,9 @@ static int vcn_v2_5_hw_fini(void *handle) RREG32_SOC15(VCN, i, mmUVD_STATUS))) vcn_v2_5_set_powergating_state(adev, AMD_PG_STATE_GATE); - if (amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__VCN)) + /* VF doesn't enable interrupt operations for RAS */ + if (!amdgpu_sriov_vf(adev) && + amdgpu_ras_is_supported(adev, AMDGPU_RAS_BLOCK__VCN)) amdgpu_irq_put(adev, &adev->vcn.inst[i].ras_poison_irq, 0); } diff --git a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c index 9d3368fa9cb0..5f8d6fd50b92 100644 --- a/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c +++ b/drivers/gpu/drm/amd/amdgpu/vcn_v4_0.c @@ -1827,6 +1827,7 @@ static struct amdgpu_ring_funcs vcn_v4_0_unified_ring_vm_funcs = { .type = AMDGPU_RING_TYPE_VCN_ENC, .align_mask = 0x3f, .nop = VCN_ENC_CMD_NO_OP, + .no_user_fence = true, .get_rptr = vcn_v4_0_unified_ring_get_rptr, .get_wptr = vcn_v4_0_unified_ring_get_wptr, .set_wptr = vcn_v4_0_unified_ring_set_wptr, diff --git a/drivers/gpu/drm/amd/amdgpu/vi.c b/drivers/gpu/drm/amd/amdgpu/vi.c index 3683a5d9348b..86be88171bba 100644 --- a/drivers/gpu/drm/amd/amdgpu/vi.c +++ b/drivers/gpu/drm/amd/amdgpu/vi.c @@ -1409,10 +1409,31 @@ static uint64_t vi_get_pcie_replay_count(struct amdgpu_device *adev) return (nak_r + nak_g); } +struct vi_reset_quirk { + u16 device; + u16 subsystem_vendor; + u16 subsystem_device; + u8 revision; +}; + +static const struct vi_reset_quirk vi_reset_quirks[] = { + { 0x67ef, PCI_VENDOR_ID_APPLE, 0x0190, 0xe3 }, /* Radeon Pro 555X */ + { 0x67ef, PCI_VENDOR_ID_APPLE, 0x018f, 0xc2 }, /* Radeon Pro 560X */ + { 0x67ef, PCI_VENDOR_ID_APPLE, 0x017a, 0xc7 }, /* Radeon Pro 555 */ +}; + static bool vi_need_reset_on_init(struct amdgpu_device *adev) { + unsigned int i; u32 clock_cntl, pc; + for (i = 0; i < ARRAY_SIZE(vi_reset_quirks); i++) + if (adev->pdev->device == vi_reset_quirks[i].device && + adev->pdev->subsystem_vendor == vi_reset_quirks[i].subsystem_vendor && + adev->pdev->subsystem_device == vi_reset_quirks[i].subsystem_device && + adev->pdev->revision == vi_reset_quirks[i].revision) + return true; + if (adev->flags & AMD_IS_APU) return false; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c index 29a02c175228..2753e98dac2f 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_crat.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_crat.c @@ -1314,6 +1314,23 @@ int kfd_parse_crat_table(void *crat_image, struct list_head *device_list, sub_type_hdr = (struct crat_subtype_generic *)(crat_table+1); while ((char *)sub_type_hdr + sizeof(struct crat_subtype_generic) < ((char *)crat_image) + image_len) { + if (!sub_type_hdr->length || + sub_type_hdr->length < sizeof(struct crat_subtype_generic)) { + pr_warn("Invalid CRAT subtype length %u\n", + sub_type_hdr->length); + ret = -EINVAL; + break; + } + + /* Validate subtype fits within remaining image */ + if ((char *)sub_type_hdr + sub_type_hdr->length > + (char *)crat_image + image_len) { + pr_warn("CRAT subtype length %u exceeds image bounds\n", + sub_type_hdr->length); + ret = -EINVAL; + break; + } + if (sub_type_hdr->flags & CRAT_SUBTYPE_FLAGS_ENABLED) { ret = kfd_parse_subtype(sub_type_hdr, device_list); if (ret) diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c index 85d461340e64..7e4bba9d07f8 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.c @@ -372,12 +372,32 @@ static int allocate_doorbell(struct qcm_process_device *qpd, * we need the physical sdma engine id in order to get the * correct doorbell offset. */ - uint32_t valid_id = idx_offset[qpd->dqm->dev->node_id * - get_num_all_sdma_engines(qpd->dqm) + - q->properties.sdma_engine_id] - + (q->properties.sdma_queue_id & 1) - * KFD_QUEUE_DOORBELL_MIRROR_OFFSET - + (q->properties.sdma_queue_id >> 1); + u32 engine_base = idx_offset[dev->node_id * + get_num_all_sdma_engines(qpd->dqm) + + q->properties.sdma_engine_id]; + u32 valid_id; + + /* + * With the aqua_vanjaram doorbell layout, a shader doorbell + * write whose 32-byte block starts in the previous engine's + * range rings that engine instead. Engine ranges are 10 + * indices apart (sdma_doorbell_range = 20 dwords, halved for + * 8-byte doorbells, in aqua_vanjaram_doorbell_index_init()), + * which puts odd engines 2 indices into a shared block. + * Aligning costs at most 2 indices, and 8 queues per engine use + * only 4 of the 10, so the aligned base stays in range. The + * 512-index mirror offset keeps the alignment. + */ + if (KFD_GC_VERSION(dev) == IP_VERSION(9, 4, 3) || + KFD_GC_VERSION(dev) == IP_VERSION(9, 4, 4) || + KFD_GC_VERSION(dev) == IP_VERSION(9, 5, 0)) + engine_base = ALIGN(engine_base, + KFD_SDMA_SHADER_DOORBELL_GRANULARITY); + + valid_id = engine_base + + (q->properties.sdma_queue_id & 1) * + KFD_QUEUE_DOORBELL_MIRROR_OFFSET + + (q->properties.sdma_queue_id >> 1); if (restore_id && *restore_id != valid_id) return -EINVAL; @@ -1131,6 +1151,14 @@ static int evict_process_queues_cpsch(struct device_queue_manager *dqm, } } } + /* + * Heavy-weight TLB flush after MES removes queues to ensure + * in-flight memory accesses complete before memory is freed/migrated. + * HWS does this automatically, MES does not. + */ + if (dqm->dev->kfd->shared_resources.enable_mes) + kfd_flush_tlb(pdd, TLB_FLUSH_HEAVYWEIGHT); + pdd->last_evict_timestamp = get_jiffies_64(); if (!dqm->dev->kfd->shared_resources.enable_mes) retval = execute_queues_cpsch(dqm, @@ -2972,8 +3000,12 @@ int suspend_queues(struct kfd_process *p, if (!per_device_suspended) { dqm_unlock(dqm); mutex_unlock(&p->event_mutex); - if (total_suspended) + if (total_suspended) { amdgpu_amdkfd_debug_mem_fence(dqm->dev->adev); + /* Heavy-weight TLB flush after MES suspends queues */ + if (pdd->drm_priv) + kfd_flush_tlb(pdd, TLB_FLUSH_HEAVYWEIGHT); + } continue; } @@ -3117,6 +3149,11 @@ int debug_refresh_runlist(struct device_queue_manager *dqm) return debug_map_and_unlock(dqm); } +size_t mqd_size_from_queue_type(struct device_queue_manager *dqm, enum kfd_queue_type type) +{ + return dqm->mqd_mgrs[get_mqd_type_from_queue_type(type)]->mqd_size; +} + #if defined(CONFIG_DEBUG_FS) static void seq_reg_dump(struct seq_file *m, diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h index cf7e182588f8..d6220d77a06c 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_device_queue_manager.h @@ -302,6 +302,8 @@ void set_queue_snapshot_entry(struct queue *q, int debug_lock_and_unmap(struct device_queue_manager *dqm); int debug_map_and_unlock(struct device_queue_manager *dqm); int debug_refresh_runlist(struct device_queue_manager *dqm); +size_t mqd_size_from_queue_type(struct device_queue_manager *dqm, + enum kfd_queue_type type); static inline unsigned int get_sh_mem_bases_32(struct kfd_process_device *pdd) { diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c index 59575e2424d0..912209fd7dd7 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_migrate.c @@ -131,7 +131,7 @@ svm_migrate_copy_memory_gart(struct amdgpu_device *adev, dma_addr_t *sys, uint64_t gart_s, gart_d; struct dma_fence *next; uint64_t size; - int r; + int r = 0; mutex_lock(&adev->mman.gtt_window_lock); @@ -251,15 +251,6 @@ svm_migrate_get_sys_page(struct vm_area_struct *vma, unsigned long addr) return page; } -static void svm_migrate_put_sys_page(unsigned long addr) -{ - struct page *page; - - page = pfn_to_page(addr >> PAGE_SHIFT); - unlock_page(page); - put_page(page); -} - static unsigned long svm_migrate_successful_pages(struct migrate_vma *migrate) { unsigned long cpages = 0; @@ -298,7 +289,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, dma_addr_t *src; uint64_t *dst; uint64_t i, j; - int r; + int r = 0; pr_debug("svms 0x%p [0x%lx 0x%lx 0x%llx]\n", prange->svms, prange->start, prange->last, ttm_res_offset); @@ -322,6 +313,7 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, DMA_BIDIRECTIONAL); r = dma_mapping_error(dev, src[i]); if (r) { + src[i] = 0; dev_err(dev, "%s: fail %d dma_map_page\n", __func__, r); goto out_free_vram_pages; @@ -346,7 +338,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, pr_debug_ratelimited("dma mapping src to 0x%llx, pfn 0x%lx\n", src[i] >> PAGE_SHIFT, page_to_pfn(spage)); - if (j >= (cursor.size >> PAGE_SHIFT) - 1 && i < npages - 1) { + /* accumulated j + 1 pages reach end of current drm_buddy_block */ + if (j + 1 >= (cursor.size >> PAGE_SHIFT)) { r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j + 1, FROM_RAM_TO_VRAM, @@ -360,7 +353,8 @@ svm_migrate_copy_to_vram(struct kfd_node *node, struct svm_range *prange, } } - r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j, + if (j > 0) + r = svm_migrate_copy_memory_gart(adev, src + i - j, dst + i - j, j, FROM_RAM_TO_VRAM, mfence); out_free_vram_pages: @@ -582,9 +576,10 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, dma_addr_t *scratch, uint64_t npages) { struct device *dev = adev->dev; - uint64_t *src; + struct page *dpage = NULL; dma_addr_t *dst; - struct page *dpage; + uint64_t *src; + uint64_t i = 0, j; uint64_t addr; int r = 0; @@ -638,6 +633,7 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, r = dma_mapping_error(dev, dst[i]); if (r) { dev_err(adev->dev, "%s: fail %d dma_map_page\n", __func__, r); + dst[i] = 0; goto out_oom; } @@ -645,17 +641,39 @@ svm_migrate_copy_to_ram(struct amdgpu_device *adev, struct svm_range *prange, dst[i] >> PAGE_SHIFT, page_to_pfn(dpage)); migrate->dst[i] = migrate_pfn(page_to_pfn(dpage)); + + dpage = NULL; j++; } - r = svm_migrate_copy_memory_gart(adev, dst + i - j, src + i - j, j, - FROM_VRAM_TO_RAM, mfence); - + if (j > 0) + r = svm_migrate_copy_memory_gart(adev, dst + i - j, src + i - j, j, + FROM_VRAM_TO_RAM, mfence); out_oom: if (r) { pr_debug("failed %d copy to ram\n", r); + + /* first release current dpage when dma_map_page fail */ + if (dpage) { + unlock_page(dpage); + put_page(dpage); + } + + /* release previous allocated sys pages and unmap dma address */ while (i--) { - svm_migrate_put_sys_page(dst[i]); + + if (dst[i]) { + dma_unmap_page(dev, dst[i], PAGE_SIZE, + DMA_BIDIRECTIONAL); + dst[i] = 0; + } + + dpage = migrate_pfn_to_page(migrate->dst[i]); + if (!dpage) + continue; + + unlock_page(dpage); + put_page(dpage); migrate->dst[i] = 0; } } diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h index 12de27f58edf..8f6fe5312b26 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_priv.h +++ b/drivers/gpu/drm/amd/amdkfd/kfd_priv.h @@ -132,6 +132,13 @@ */ #define KFD_QUEUE_DOORBELL_MIRROR_OFFSET 512 +/* + * On GC 9.4.3, 9.4.4 and 9.5.0, doorbell writes issued from shaders are + * routed to an SDMA engine at 32-byte granularity, i.e. 4 indices with the + * 8-byte SDMA doorbells used on SOC15. + */ +#define KFD_SDMA_SHADER_DOORBELL_GRANULARITY 4 + /** * enum kfd_ioctl_flags - KFD ioctl flags * Various flags that can be set in &amdkfd_ioctl_desc.flags to control how @@ -426,7 +433,8 @@ enum kfd_queue_type { KFD_QUEUE_TYPE_SDMA, KFD_QUEUE_TYPE_HIQ, KFD_QUEUE_TYPE_DIQ, - KFD_QUEUE_TYPE_SDMA_XGMI + KFD_QUEUE_TYPE_SDMA_XGMI, + KFD_QUEUE_TYPE_MAX, }; enum kfd_queue_format { diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c index 3f6fc8d0984d..430a79efa0bc 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_process_queue_manager.c @@ -946,6 +946,23 @@ int kfd_criu_restore_queue(struct kfd_process *p, goto exit; } + pdd = kfd_process_device_data_by_id(p, q_data->gpu_id); + if (!pdd) { + pr_err("Failed to get pdd\n"); + ret = -EINVAL; + goto exit; + } + + if (q_data->type >= KFD_QUEUE_TYPE_MAX) { + ret = -EINVAL; + goto exit; + } + + if (q_data->mqd_size != mqd_size_from_queue_type(pdd->dev->dqm, q_data->type)) { + ret = -EINVAL; + goto exit; + } + *priv_data_offset += sizeof(*q_data); q_extra_data_size = (uint64_t)q_data->ctl_stack_size + q_data->mqd_size; @@ -968,13 +985,6 @@ int kfd_criu_restore_queue(struct kfd_process *p, *priv_data_offset += q_extra_data_size; - pdd = kfd_process_device_data_by_id(p, q_data->gpu_id); - if (!pdd) { - pr_err("Failed to get pdd\n"); - ret = -EINVAL; - goto exit; - } - /* data stored in this order: mqd, ctl_stack */ mqd = q_extra_data; ctl_stack = mqd + q_data->mqd_size; diff --git a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c index d9953c2b2661..ef4c7e39a844 100644 --- a/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c +++ b/drivers/gpu/drm/amd/amdkfd/kfd_smi_events.c @@ -238,16 +238,16 @@ void kfd_smi_event_update_thermal_throttling(struct kfd_node *dev, void kfd_smi_event_update_vmfault(struct kfd_node *dev, uint16_t pasid) { - struct amdgpu_task_info task_info; - - memset(&task_info, 0, sizeof(struct amdgpu_task_info)); - amdgpu_vm_get_task_info(dev->adev, pasid, &task_info); - /* Report VM faults from user applications, not retry from kernel */ - if (!task_info.pid) - return; - - kfd_smi_event_add(0, dev, KFD_SMI_EVENT_VMFAULT, "%x:%s\n", - task_info.pid, task_info.task_name); + struct amdgpu_task_info *task_info; + + task_info = amdgpu_vm_get_task_info_pasid(dev->adev, pasid); + if (task_info) { + /* Report VM faults from user applications, not retry from kernel */ + if (task_info->pid) + kfd_smi_event_add(task_info->tgid, dev, KFD_SMI_EVENT_VMFAULT, "%x:%s\n", + task_info->pid, task_info->task_name); + amdgpu_vm_put_task_info(task_info); + } } void kfd_smi_event_page_fault_start(struct kfd_node *node, pid_t pid, diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c index 4b03bb4b913f..bc86394d13d9 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.c @@ -413,6 +413,7 @@ static void dm_pflip_high_irq(void *interrupt_params) struct amdgpu_crtc *amdgpu_crtc; struct common_irq_params *irq_params = interrupt_params; struct amdgpu_device *adev = irq_params->adev; + struct drm_device *dev = adev_to_drm(adev); unsigned long flags; struct drm_pending_vblank_event *e; u32 vpos, hpos, v_blank_start, v_blank_end; @@ -423,18 +424,17 @@ static void dm_pflip_high_irq(void *interrupt_params) /* IRQ could occur when in initial stage */ /* TODO work and BO cleanup */ if (amdgpu_crtc == NULL) { - DC_LOG_PFLIP("CRTC is null, returning.\n"); + drm_dbg_state(dev, "CRTC is null, returning.\n"); return; } spin_lock_irqsave(&adev_to_drm(adev)->event_lock, flags); if (amdgpu_crtc->pflip_status != AMDGPU_FLIP_SUBMITTED) { - DC_LOG_PFLIP("amdgpu_crtc->pflip_status = %d !=AMDGPU_FLIP_SUBMITTED(%d) on crtc:%d[%p]\n", - amdgpu_crtc->pflip_status, - AMDGPU_FLIP_SUBMITTED, - amdgpu_crtc->crtc_id, - amdgpu_crtc); + drm_dbg_state(dev, + "amdgpu_crtc->pflip_status = %d != AMDGPU_FLIP_SUBMITTED(%d) on crtc:%d[%p]\n", + amdgpu_crtc->pflip_status, AMDGPU_FLIP_SUBMITTED, + amdgpu_crtc->crtc_id, amdgpu_crtc); spin_unlock_irqrestore(&adev_to_drm(adev)->event_lock, flags); return; } @@ -500,9 +500,9 @@ static void dm_pflip_high_irq(void *interrupt_params) amdgpu_crtc->pflip_status = AMDGPU_FLIP_NONE; spin_unlock_irqrestore(&adev_to_drm(adev)->event_lock, flags); - DC_LOG_PFLIP("crtc:%d[%p], pflip_stat:AMDGPU_FLIP_NONE, vrr[%d]-fp %d\n", - amdgpu_crtc->crtc_id, amdgpu_crtc, - vrr_active, (int) !e); + drm_dbg_state(dev, + "crtc:%d[%p], pflip_stat:AMDGPU_FLIP_NONE, vrr[%d]-fp %d\n", + amdgpu_crtc->crtc_id, amdgpu_crtc, vrr_active, (int)!e); } static void dm_vupdate_high_irq(void *interrupt_params) @@ -532,9 +532,9 @@ static void dm_vupdate_high_irq(void *interrupt_params) atomic64_set(&irq_params->previous_timestamp, vblank->time); } - DC_LOG_VBLANK("crtc:%d, vupdate-vrr:%d\n", - acrtc->crtc_id, - vrr_active); + drm_dbg_vbl(drm_dev, + "crtc:%d, vupdate-vrr:%d\n", acrtc->crtc_id, + vrr_active); /* Core vblank handling is done here after end of front-porch in * vrr mode, as vblank timestamping will give valid results @@ -585,8 +585,9 @@ static void dm_crtc_high_irq(void *interrupt_params) vrr_active = amdgpu_dm_crtc_vrr_active_irq(acrtc); - DC_LOG_VBLANK("crtc:%d, vupdate-vrr:%d, planes:%d\n", acrtc->crtc_id, - vrr_active, acrtc->dm_irq_params.active_planes); + drm_dbg_vbl(adev_to_drm(adev), + "crtc:%d, vupdate-vrr:%d, planes:%d\n", acrtc->crtc_id, + vrr_active, acrtc->dm_irq_params.active_planes); /** * Core vblank handling at start of front-porch is only possible @@ -2639,9 +2640,14 @@ static int dm_suspend(void *handle) dc_allow_idle_optimizations(adev->dm.dc, false); dm->cached_dc_state = dc_copy_state(dm->dc->current_state); + if (!dm->cached_dc_state) { + drm_err(adev_to_drm(adev), + "Failed to allocate cached DC state during suspend\n"); + mutex_unlock(&dm->dc_lock); + return -ENOMEM; + } - if (dm->cached_dc_state) - dm_gpureset_toggle_interrupts(adev, dm->cached_dc_state, false); + dm_gpureset_toggle_interrupts(adev, dm->cached_dc_state, false); amdgpu_dm_commit_zero_streams(dm->dc); @@ -2691,6 +2697,7 @@ static void emulated_link_detect(struct dc_link *link) struct display_sink_capability sink_caps = { 0 }; enum dc_edid_status edid_status; struct dc_context *dc_ctx = link->ctx; + struct drm_device *dev = adev_to_drm(dc_ctx->driver_context); struct dc_sink *sink = NULL; struct dc_sink *prev_sink = NULL; @@ -2740,7 +2747,7 @@ static void emulated_link_detect(struct dc_link *link) } default: - DC_ERROR("Invalid connector type! signal:%d\n", + drm_err(dev, "Invalid connector type! signal:%d\n", link->connector_signal); return; } @@ -2750,7 +2757,7 @@ static void emulated_link_detect(struct dc_link *link) sink = dc_sink_create(&sink_init_data); if (!sink) { - DC_ERROR("Failed to create sink!\n"); + drm_err(dev, "Failed to create sink!\n"); return; } @@ -2763,7 +2770,7 @@ static void emulated_link_detect(struct dc_link *link) sink); if (edid_status != EDID_OK) - DC_ERROR("Failed to read EDID"); + drm_err(dev, "Failed to read EDID\n"); } @@ -2782,7 +2789,7 @@ static void dm_gpureset_commit_state(struct dc_state *dc_state, bundle = kzalloc(sizeof(*bundle), GFP_KERNEL); if (!bundle) { - dm_error("Failed to allocate update bundle\n"); + drm_err(dm->ddev, "Failed to allocate update bundle\n"); goto cleanup; } @@ -3241,7 +3248,8 @@ void amdgpu_dm_update_connector_after_detect( aconnector->timing_requested = kzalloc(sizeof(struct dc_crtc_timing), GFP_KERNEL); if (!aconnector->timing_requested) - dm_error("failed to create aconnector->requested_timing\n"); + drm_err(dev, + "failed to create aconnector->requested_timing\n"); } drm_connector_update_edid_property(connector, aconnector->edid); @@ -5666,11 +5674,16 @@ decide_crtc_timing_for_drm_display_mode(struct drm_display_mode *drm_mode, const struct drm_display_mode *native_mode, bool scale_enabled) { + /* Inserted modes copy the native sync; a sink mode with equal totals may not */ if (scale_enabled) { copy_crtc_timing_for_drm_display_mode(native_mode, drm_mode); } else if (native_mode->clock == drm_mode->clock && native_mode->htotal == drm_mode->htotal && - native_mode->vtotal == drm_mode->vtotal) { + native_mode->vtotal == drm_mode->vtotal && + native_mode->hsync_start == drm_mode->hsync_start && + native_mode->hsync_end == drm_mode->hsync_end && + native_mode->vsync_start == drm_mode->vsync_start && + native_mode->vsync_end == drm_mode->vsync_end) { copy_crtc_timing_for_drm_display_mode(native_mode, drm_mode); } else { /* no scaling nor amdgpu inserted, no need to patch */ @@ -6147,10 +6160,10 @@ create_stream_for_sink(struct amdgpu_dm_connector *aconnector, requested_bpc); if (aconnector->timing_changed) { - DC_LOG_DEBUG("%s: overriding timing for automated test, bpc %d, changing to %d\n", - __func__, - stream->timing.display_color_depth, - aconnector->timing_requested->display_color_depth); + drm_dbg(aconnector->base.dev, + "overriding timing for automated test, bpc %d, changing to %d\n", + stream->timing.display_color_depth, + aconnector->timing_requested->display_color_depth); stream->timing = *aconnector->timing_requested; } @@ -7875,8 +7888,9 @@ static void prepare_flip_isr(struct amdgpu_crtc *acrtc) /* Mark this event as consumed */ acrtc->base.state->event = NULL; - DC_LOG_PFLIP("crtc:%d, pflip_stat:AMDGPU_FLIP_SUBMITTED\n", - acrtc->crtc_id); + drm_dbg_state(acrtc->base.dev, + "crtc:%d, pflip_stat:AMDGPU_FLIP_SUBMITTED\n", + acrtc->crtc_id); } static void update_freesync_state_on_stream( @@ -8119,7 +8133,7 @@ static void amdgpu_dm_commit_planes(struct drm_atomic_state *state, bundle = kzalloc(sizeof(*bundle), GFP_KERNEL); if (!bundle) { - dm_error("Failed to allocate update bundle\n"); + drm_err(dev, "Failed to allocate update bundle\n"); goto cleanup; } @@ -8708,7 +8722,9 @@ static void amdgpu_dm_commit_streams(struct drm_atomic_state *state, status = dc_stream_get_status_from_state(dc_state, dm_new_crtc_state->stream); if (!status) - DC_ERR("got no status for stream %p on acrtc%p\n", dm_new_crtc_state->stream, acrtc); + drm_err(dev, + "got no status for stream %p on acrtc%p\n", + dm_new_crtc_state->stream, acrtc); else acrtc->otg_inst = status->primary_otg_inst; } @@ -11015,7 +11031,8 @@ void dm_write_reg_func(const struct dc_context *ctx, uint32_t address, { #ifdef DM_CHECK_ADDR_0 if (address == 0) { - DC_ERR("invalid register write. address = 0"); + drm_err(adev_to_drm(ctx->driver_context), + "invalid register write. address = 0"); return; } #endif @@ -11029,7 +11046,8 @@ uint32_t dm_read_reg_func(const struct dc_context *ctx, uint32_t address, u32 value; #ifdef DM_CHECK_ADDR_0 if (address == 0) { - DC_ERR("invalid register read; address = 0\n"); + drm_err(adev_to_drm(ctx->driver_context), + "invalid register read; address = 0\n"); return 0; } #endif diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h index 8d4f2cadb915..6fc5b1e4be46 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm.h @@ -43,7 +43,7 @@ * in amdgpu_dm_kms.h file */ -#define AMDGPU_DM_MAX_DISPLAY_INDEX 31 +#define AMDGPU_DM_MAX_DISPLAY_COUNT 32 #define AMDGPU_DM_MAX_CRTC 6 diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c index dab732c6c6c7..0531ebcf2cd3 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_crtc.c @@ -270,13 +270,13 @@ static void dm_crtc_reset_state(struct drm_crtc *crtc) { struct dm_crtc_state *state; - if (crtc->state) - dm_crtc_destroy_state(crtc, crtc->state); - state = kzalloc(sizeof(*state), GFP_KERNEL); - if (WARN_ON(!state)) + if (!state) return; + if (crtc->state) + dm_crtc_destroy_state(crtc, crtc->state); + __drm_atomic_helper_crtc_reset(crtc, &state->base); } diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c index a048022d9865..98843f38f414 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.c @@ -256,7 +256,7 @@ void hdcp_reset_display(struct hdcp_workqueue *hdcp_work, unsigned int link_inde cancel_delayed_work(&hdcp_w->property_validate_dwork); - for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX; conn_index++) { + for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT; conn_index++) { hdcp_w->encryption_status[conn_index] = MOD_HDCP_ENCRYPTION_STATUS_HDCP_OFF; if (hdcp_w->aconnector[conn_index]) { @@ -303,7 +303,7 @@ static void event_property_update(struct work_struct *work) struct drm_connector *connector; struct drm_connector_state *conn_state; - for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX; conn_index++) { + for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT; conn_index++) { aconnector = hdcp_work->aconnector[conn_index]; if (!aconnector) @@ -372,7 +372,7 @@ static void event_property_validate(struct work_struct *work) guard(mutex)(&hdcp_work->mutex); - for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_INDEX; + for (conn_index = 0; conn_index < AMDGPU_DM_MAX_DISPLAY_COUNT; conn_index++) { aconnector = hdcp_work->aconnector[conn_index]; @@ -747,10 +747,10 @@ struct hdcp_workqueue *hdcp_create_workqueue(struct amdgpu_device *adev, memset(hdcp_work[i].aconnector, 0, sizeof(struct amdgpu_dm_connector *) * - AMDGPU_DM_MAX_DISPLAY_INDEX); + AMDGPU_DM_MAX_DISPLAY_COUNT); memset(hdcp_work[i].encryption_status, 0, sizeof(enum mod_hdcp_encryption_status) * - AMDGPU_DM_MAX_DISPLAY_INDEX); + AMDGPU_DM_MAX_DISPLAY_COUNT); } cp_psp->funcs.update_stream_config = update_config; diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h index 69b445b011c8..d18c4cc7d69f 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_hdcp.h @@ -43,7 +43,7 @@ struct hdcp_workqueue { struct delayed_work callback_dwork; struct delayed_work watchdog_timer_dwork; struct delayed_work property_validate_dwork; - struct amdgpu_dm_connector *aconnector[AMDGPU_DM_MAX_DISPLAY_INDEX]; + struct amdgpu_dm_connector *aconnector[AMDGPU_DM_MAX_DISPLAY_COUNT]; struct mutex mutex; struct mod_hdcp hdcp; @@ -51,7 +51,7 @@ struct hdcp_workqueue { struct mod_hdcp_display display; struct mod_hdcp_link link; - enum mod_hdcp_encryption_status encryption_status[AMDGPU_DM_MAX_DISPLAY_INDEX]; + enum mod_hdcp_encryption_status encryption_status[AMDGPU_DM_MAX_DISPLAY_COUNT]; /* when display is unplugged from mst hub, connctor will be * destroyed within dm_dp_mst_connector_destroy. connector * hdcp perperties, like type, undesired, desired, enabled, @@ -61,9 +61,9 @@ struct hdcp_workqueue { * will be retrieved from hdcp_work within dm_dp_mst_get_modes */ /* un-desired, desired, enabled */ - unsigned int content_protection[AMDGPU_DM_MAX_DISPLAY_INDEX]; + unsigned int content_protection[AMDGPU_DM_MAX_DISPLAY_COUNT]; /* hdcp1.x, hdcp2.x */ - unsigned int hdcp_content_type[AMDGPU_DM_MAX_DISPLAY_INDEX]; + unsigned int hdcp_content_type[AMDGPU_DM_MAX_DISPLAY_COUNT]; uint8_t max_link; diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c index 227a148b0f82..3a16de55f43c 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_helpers.c @@ -547,7 +547,8 @@ bool dm_helpers_dp_read_dpcd( struct amdgpu_dm_connector *aconnector = link->priv; if (!aconnector) { - DC_LOG_DC("Failed to find connector for link!\n"); + drm_dbg_dp(aconnector->base.dev, + "Failed to find connector for link!\n"); return false; } @@ -669,7 +670,7 @@ static bool execute_synaptics_rc_command(struct drm_dp_aux *aux, drm_dp_dpcd_read(aux, SYNAPTICS_RC_DATA, data, length); } - DC_LOG_DC("%s: success = %d\n", __func__, success); + drm_dbg_dp(aux->drm_dev, "success = %d\n", success); return success; } @@ -678,7 +679,7 @@ static void apply_synaptics_fifo_reset_wa(struct drm_dp_aux *aux) { unsigned char data[16] = {0}; - DC_LOG_DC("Start %s\n", __func__); + drm_dbg_dp(aux->drm_dev, "Start\n"); // Step 2 data[0] = 'P'; @@ -736,7 +737,7 @@ static void apply_synaptics_fifo_reset_wa(struct drm_dp_aux *aux) if (!execute_synaptics_rc_command(aux, true, 0x02, 0, 0, NULL)) return; - DC_LOG_DC("Done %s\n", __func__); + drm_dbg_dp(aux->drm_dev, "Done\n"); } /* MST Dock */ @@ -749,7 +750,8 @@ static uint8_t write_dsc_enable_synaptics_non_virtual_dpcd_mst( { uint8_t ret = 0; - DC_LOG_DC("Configure DSC to non-virtual dpcd synaptics\n"); + drm_dbg_dp(aux->drm_dev, + "Configure DSC to non-virtual dpcd synaptics\n"); if (enable) { /* When DSC is enabled on previous boot and reboot with the hub, @@ -787,7 +789,9 @@ bool dm_helpers_dp_write_dsc_enable( static const uint8_t DSC_DECODING = 0x01; static const uint8_t DSC_PASSTHROUGH = 0x02; - struct amdgpu_dm_connector *aconnector; + struct amdgpu_dm_connector *aconnector = + (struct amdgpu_dm_connector *)stream->dm_stream_context; + struct drm_device *dev = aconnector->base.dev; struct drm_dp_mst_port *port; uint8_t enable_dsc = enable ? DSC_DECODING : DSC_DISABLE; uint8_t enable_passthrough = enable ? DSC_PASSTHROUGH : DSC_DISABLE; @@ -797,8 +801,6 @@ bool dm_helpers_dp_write_dsc_enable( return false; if (stream->signal == SIGNAL_TYPE_DISPLAY_PORT_MST) { - aconnector = (struct amdgpu_dm_connector *)stream->dm_stream_context; - if (!aconnector->dsc_aux) return false; @@ -815,30 +817,34 @@ bool dm_helpers_dp_write_dsc_enable( ret = drm_dp_dpcd_write(port->passthrough_aux, DP_DSC_ENABLE, &enable_passthrough, 1); - DC_LOG_DC("Sent DSC pass-through enable to virtual dpcd port, ret = %u\n", - ret); + drm_dbg_dp(dev, + "Sent DSC pass-through enable to virtual dpcd port, ret = %u\n", + ret); } ret = drm_dp_dpcd_write(aconnector->dsc_aux, DP_DSC_ENABLE, &enable_dsc, 1); - DC_LOG_DC("Sent DSC decoding enable to %s port, ret = %u\n", - (port->passthrough_aux) ? "remote RX" : - "virtual dpcd", - ret); + drm_dbg_dp(dev, + "Sent DSC decoding enable to %s port, ret = %u\n", + (port->passthrough_aux) ? "remote RX" : + "virtual dpcd", + ret); } else { ret = drm_dp_dpcd_write(aconnector->dsc_aux, DP_DSC_ENABLE, &enable_dsc, 1); - DC_LOG_DC("Sent DSC decoding disable to %s port, ret = %u\n", - (port->passthrough_aux) ? "remote RX" : - "virtual dpcd", - ret); + drm_dbg_dp(dev, + "Sent DSC decoding disable to %s port, ret = %u\n", + (port->passthrough_aux) ? "remote RX" : + "virtual dpcd", + ret); if (port->passthrough_aux) { ret = drm_dp_dpcd_write(port->passthrough_aux, DP_DSC_ENABLE, &enable_passthrough, 1); - DC_LOG_DC("Sent DSC pass-through disable to virtual dpcd port, ret = %u\n", - ret); + drm_dbg_dp(dev, + "Sent DSC pass-through disable to virtual dpcd port, ret = %u\n", + ret); } } } @@ -846,10 +852,14 @@ bool dm_helpers_dp_write_dsc_enable( if (stream->signal == SIGNAL_TYPE_DISPLAY_PORT || stream->signal == SIGNAL_TYPE_EDP) { if (stream->sink->link->dpcd_caps.dongle_type == DISPLAY_DONGLE_NONE) { ret = dm_helpers_dp_write_dpcd(ctx, stream->link, DP_DSC_ENABLE, &enable_dsc, 1); - DC_LOG_DC("Send DSC %s to SST RX\n", enable_dsc ? "enable" : "disable"); + drm_dbg_dp(dev, + "Send DSC %s to SST RX\n", + enable_dsc ? "enable" : "disable"); } else if (stream->sink->link->dpcd_caps.dongle_type == DISPLAY_DONGLE_DP_HDMI_CONVERTER) { ret = dm_helpers_dp_write_dpcd(ctx, stream->link, DP_DSC_ENABLE, &enable_dsc, 1); - DC_LOG_DC("Send DSC %s to DP-HDMI PCON\n", enable_dsc ? "enable" : "disable"); + drm_dbg_dp(dev, + "Send DSC %s to DP-HDMI PCON\n", + enable_dsc ? "enable" : "disable"); } } @@ -1123,6 +1133,7 @@ bool dm_helpers_dp_handle_test_pattern_request( struct pipe_ctx *pipes = link->dc->current_state->res_ctx.pipe_ctx; struct pipe_ctx *pipe_ctx = NULL; struct amdgpu_dm_connector *aconnector = link->priv; + struct drm_device *dev = aconnector->base.dev; int i; for (i = 0; i < MAX_PIPES; i++) { @@ -1200,12 +1211,12 @@ bool dm_helpers_dp_handle_test_pattern_request( && pipe_ctx->stream->timing.display_color_depth != requestColorDepth) || (requestPixelEncoding != PIXEL_ENCODING_UNDEFINED && pipe_ctx->stream->timing.pixel_encoding != requestPixelEncoding)) { - DC_LOG_DEBUG("%s: original bpc %d pix encoding %d, changing to %d %d\n", - __func__, - pipe_ctx->stream->timing.display_color_depth, - pipe_ctx->stream->timing.pixel_encoding, - requestColorDepth, - requestPixelEncoding); + drm_dbg(dev, + "original bpc %d pix encoding %d, changing to %d %d\n", + pipe_ctx->stream->timing.display_color_depth, + pipe_ctx->stream->timing.pixel_encoding, + requestColorDepth, + requestPixelEncoding); pipe_ctx->stream->timing.display_color_depth = requestColorDepth; pipe_ctx->stream->timing.pixel_encoding = requestPixelEncoding; @@ -1216,7 +1227,7 @@ bool dm_helpers_dp_handle_test_pattern_request( if (aconnector->timing_requested) *aconnector->timing_requested = pipe_ctx->stream->timing; else - DC_LOG_ERROR("%s: timing storage failed\n", __func__); + drm_err(dev, "timing storage failed\n"); } diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c index 29e1b252ec73..eb9003d3cfae 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_irq.c @@ -908,6 +908,9 @@ void amdgpu_dm_hpd_init(struct amdgpu_device *adev) const struct dc_link *dc_link = amdgpu_dm_connector->dc_link; + if (!dc_link) + continue; + /* * Get a base driver irq reference for hpd ints for the lifetime * of dm. Note that only hpd interrupt types are registered with diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.c index ab6924d3046b..acb5362d3450 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_mst_types.c @@ -199,8 +199,9 @@ amdgpu_dm_mst_connector_early_unregister(struct drm_connector *connector) if (dc_link->sink_count) dc_link_remove_remote_sink(dc_link, dc_sink); - DC_LOG_MST("DM_MST: remove remote sink 0x%p, %d remaining\n", - dc_sink, dc_link->sink_count); + drm_dbg_dp(connector->dev, + "DM_MST: remove remote sink 0x%p, %d remaining\n", + dc_sink, dc_link->sink_count); dc_sink_release(dc_sink); aconnector->dc_sink = NULL; @@ -353,8 +354,10 @@ static int dm_dp_mst_get_modes(struct drm_connector *connector) return 0; } - DC_LOG_MST("DM_MST: add remote sink 0x%p, %d remaining\n", - dc_sink, aconnector->dc_link->sink_count); + drm_dbg_dp(connector->dev, + "DM_MST: add remote sink 0x%p, %d remaining\n", + dc_sink, + aconnector->dc_link->sink_count); dc_sink->priv = aconnector; aconnector->dc_sink = dc_sink; @@ -389,8 +392,9 @@ static int dm_dp_mst_get_modes(struct drm_connector *connector) return 0; } - DC_LOG_MST("DM_MST: add remote sink 0x%p, %d remaining\n", - dc_sink, aconnector->dc_link->sink_count); + drm_dbg_dp(connector->dev, + "DM_MST: add remote sink 0x%p, %d remaining\n", + dc_sink, aconnector->dc_link->sink_count); dc_sink->priv = aconnector; /* dc_link_add_remote_sink returns a new reference */ @@ -509,8 +513,10 @@ dm_dp_mst_detect(struct drm_connector *connector, if (aconnector->dc_link->sink_count) dc_link_remove_remote_sink(aconnector->dc_link, aconnector->dc_sink); - DC_LOG_MST("DM_MST: remove remote sink 0x%p, %d remaining\n", - aconnector->dc_link, aconnector->dc_link->sink_count); + drm_dbg_dp(connector->dev, + "DM_MST: remove remote sink 0x%p, %d remaining\n", + aconnector->dc_link, + aconnector->dc_link->sink_count); dc_sink_release(aconnector->dc_sink); aconnector->dc_sink = NULL; diff --git a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c index 746df72405eb..c1d18fc36516 100644 --- a/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c +++ b/drivers/gpu/drm/amd/display/amdgpu_dm/amdgpu_dm_plane.c @@ -110,25 +110,7 @@ void amdgpu_dm_plane_fill_blending_from_plane_state(const struct drm_plane_state if (plane_state->pixel_blend_mode == DRM_MODE_BLEND_PREMULTI || plane_state->pixel_blend_mode == DRM_MODE_BLEND_COVERAGE) { - static const uint32_t alpha_formats[] = { - DRM_FORMAT_ARGB8888, - DRM_FORMAT_RGBA8888, - DRM_FORMAT_ABGR8888, - DRM_FORMAT_ARGB2101010, - DRM_FORMAT_ABGR2101010, - DRM_FORMAT_ARGB16161616, - DRM_FORMAT_ABGR16161616, - DRM_FORMAT_ARGB16161616F, - }; - uint32_t format = plane_state->fb->format->format; - unsigned int i; - - for (i = 0; i < ARRAY_SIZE(alpha_formats); ++i) { - if (format == alpha_formats[i]) { - *per_pixel_alpha = true; - break; - } - } + *per_pixel_alpha = plane_state->fb->format->has_alpha; if (*per_pixel_alpha && plane_state->pixel_blend_mode == DRM_MODE_BLEND_COVERAGE) *pre_multiplied_alpha = false; @@ -1246,11 +1228,9 @@ void amdgpu_dm_plane_handle_cursor_update(struct drm_plane *plane, if (!plane->state->fb && !old_plane_state->fb) return; - DC_LOG_CURSOR("%s: crtc_id=%d with size %d to %d\n", - __func__, - amdgpu_crtc->crtc_id, - plane->state->crtc_w, - plane->state->crtc_h); + drm_dbg_atomic(plane->dev, "crtc_id=%d with size %d to %d\n", + amdgpu_crtc->crtc_id, plane->state->crtc_w, + plane->state->crtc_h); ret = get_cursor_position(plane, crtc, &position); if (ret) @@ -1336,16 +1316,16 @@ static const struct drm_plane_helper_funcs dm_plane_helper_funcs = { static void dm_drm_plane_reset(struct drm_plane *plane) { - struct dm_plane_state *amdgpu_state = NULL; + struct dm_plane_state *amdgpu_state; + + amdgpu_state = kzalloc(sizeof(*amdgpu_state), GFP_KERNEL); + if (!amdgpu_state) + return; if (plane->state) plane->funcs->atomic_destroy_state(plane, plane->state); - amdgpu_state = kzalloc(sizeof(*amdgpu_state), GFP_KERNEL); - WARN_ON(amdgpu_state == NULL); - - if (amdgpu_state) - __drm_atomic_helper_plane_reset(plane, &amdgpu_state->base); + __drm_atomic_helper_plane_reset(plane, &amdgpu_state->base); } static struct drm_plane_state * diff --git a/drivers/gpu/drm/amd/display/dc/bios/bios_parser.c b/drivers/gpu/drm/amd/display/dc/bios/bios_parser.c index fa8fcbcc421a..5400ab41adaf 100644 --- a/drivers/gpu/drm/amd/display/dc/bios/bios_parser.c +++ b/drivers/gpu/drm/amd/display/dc/bios/bios_parser.c @@ -2827,6 +2827,7 @@ static enum bp_result bios_get_board_layout_info( struct board_layout_info *board_layout_info) { unsigned int i; + struct bios_parser *bp; enum bp_result record_result; const unsigned int slot_index_to_vbios_id[MAX_BOARD_SLOTS] = { @@ -2835,6 +2836,8 @@ static enum bp_result bios_get_board_layout_info( 0, 0 }; + bp = BP_FROM_DCB(dcb); + if (board_layout_info == NULL) { DC_LOG_DETECTION_EDID_PARSER("Invalid board_layout_info\n"); return BP_RESULT_BADINPUT; diff --git a/drivers/gpu/drm/amd/display/dc/clk_mgr/dce110/dce110_clk_mgr.c b/drivers/gpu/drm/amd/display/dc/clk_mgr/dce110/dce110_clk_mgr.c index bce53ab36f3e..d50b9440210e 100644 --- a/drivers/gpu/drm/amd/display/dc/clk_mgr/dce110/dce110_clk_mgr.c +++ b/drivers/gpu/drm/amd/display/dc/clk_mgr/dce110/dce110_clk_mgr.c @@ -218,7 +218,7 @@ void dce11_pplib_apply_display_requirements( * , then change minimum memory clock based on real-time bandwidth * limitation. */ - if ((dc->ctx->asic_id.chip_family == FAMILY_AI) && + if (dc->bw_vbios && (dc->ctx->asic_id.chip_family == FAMILY_AI) && ASICREV_IS_VEGA20_P(dc->ctx->asic_id.hw_internal_rev) && (context->stream_count >= 2)) { pp_display_cfg->min_memory_clock_khz = max(pp_display_cfg->min_memory_clock_khz, (uint32_t) div64_s64( diff --git a/drivers/gpu/drm/amd/display/dc/clk_mgr/dcn201/dcn201_clk_mgr.c b/drivers/gpu/drm/amd/display/dc/clk_mgr/dcn201/dcn201_clk_mgr.c index 694fe4271b4d..9c90090e7351 100644 --- a/drivers/gpu/drm/amd/display/dc/clk_mgr/dcn201/dcn201_clk_mgr.c +++ b/drivers/gpu/drm/amd/display/dc/clk_mgr/dcn201/dcn201_clk_mgr.c @@ -59,8 +59,6 @@ #define CTX \ clk_mgr->base.ctx -#define DC_LOGGER \ - clk_mgr->base.ctx->logger static const struct clk_mgr_registers clk_mgr_regs = { CLK_COMMON_REG_LIST_DCN_201() diff --git a/drivers/gpu/drm/amd/display/dc/core/dc.c b/drivers/gpu/drm/amd/display/dc/core/dc.c index 50df677d509d..25edc0f0f60c 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc.c @@ -24,6 +24,8 @@ #include "dm_services.h" +#include "amdgpu.h" + #include "dc.h" #include "core_status.h" @@ -823,16 +825,18 @@ static void dc_destruct(struct dc *dc) if (dc->link_srv) link_destroy_link_service(&dc->link_srv); - if (dc->ctx->gpio_service) - dal_gpio_service_destroy(&dc->ctx->gpio_service); - - if (dc->ctx->created_bios) - dal_bios_parser_destroy(&dc->ctx->dc_bios); + if (dc->ctx) { + if (dc->ctx->gpio_service) + dal_gpio_service_destroy(&dc->ctx->gpio_service); - dc_perf_trace_destroy(&dc->ctx->perf_trace); + if (dc->ctx->created_bios) + dal_bios_parser_destroy(&dc->ctx->dc_bios); + kfree(dc->ctx->logger); + dc_perf_trace_destroy(&dc->ctx->perf_trace); - kfree(dc->ctx); - dc->ctx = NULL; + kfree(dc->ctx); + dc->ctx = NULL; + } kfree(dc->bw_vbios); dc->bw_vbios = NULL; @@ -871,6 +875,15 @@ static bool dc_construct_ctx(struct dc *dc, dc_ctx->nbio_reg_offsets = init_params->nbio_reg_offsets; /* Create logger */ + dc_ctx->logger = kmalloc(sizeof(*dc_ctx->logger), GFP_KERNEL); + + if (!dc_ctx->logger) { + kfree(dc_ctx); + return false; + } + + dc_ctx->logger->dev = adev_to_drm(init_params->driver); + dc->dml.logger = dc_ctx->logger; dc_ctx->dce_version = resource_parse_asic_id(init_params->asic_id); diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_debug.c b/drivers/gpu/drm/amd/display/dc/core/dc_debug.c index 69f1c2b89a57..801cdbc8117d 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_debug.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_debug.c @@ -36,6 +36,8 @@ #include "resource.h" +#define DC_LOGGER \ + dc->ctx->logger #define DC_LOGGER_INIT(logger) diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c b/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c index 37b156d32dd9..58f28adf7b2c 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_hw_sequencer.c @@ -522,48 +522,50 @@ void hwss_build_fast_sequence(struct dc *dc, while (current_pipe) { current_mpc_pipe = current_pipe; while (current_mpc_pipe) { - if (dc->hwss.set_flip_control_gsl && current_mpc_pipe->plane_state && current_mpc_pipe->plane_state->update_flags.raw) { - block_sequence[*num_steps].params.set_flip_control_gsl_params.pipe_ctx = current_mpc_pipe; - block_sequence[*num_steps].params.set_flip_control_gsl_params.flip_immediate = current_mpc_pipe->plane_state->flip_immediate; - block_sequence[*num_steps].func = HUBP_SET_FLIP_CONTROL_GSL; - (*num_steps)++; - } - if (dc->hwss.program_triplebuffer && dc->debug.enable_tri_buf && current_mpc_pipe->plane_state->update_flags.raw) { - block_sequence[*num_steps].params.program_triplebuffer_params.dc = dc; - block_sequence[*num_steps].params.program_triplebuffer_params.pipe_ctx = current_mpc_pipe; - block_sequence[*num_steps].params.program_triplebuffer_params.enableTripleBuffer = current_mpc_pipe->plane_state->triplebuffer_flips; - block_sequence[*num_steps].func = HUBP_PROGRAM_TRIPLEBUFFER; - (*num_steps)++; - } - if (dc->hwss.update_plane_addr && current_mpc_pipe->plane_state->update_flags.bits.addr_update) { - block_sequence[*num_steps].params.update_plane_addr_params.dc = dc; - block_sequence[*num_steps].params.update_plane_addr_params.pipe_ctx = current_mpc_pipe; - block_sequence[*num_steps].func = HUBP_UPDATE_PLANE_ADDR; - (*num_steps)++; - } + if (current_mpc_pipe->plane_state) { + if (dc->hwss.set_flip_control_gsl && current_mpc_pipe->plane_state->update_flags.raw) { + block_sequence[*num_steps].params.set_flip_control_gsl_params.pipe_ctx = current_mpc_pipe; + block_sequence[*num_steps].params.set_flip_control_gsl_params.flip_immediate = current_mpc_pipe->plane_state->flip_immediate; + block_sequence[*num_steps].func = HUBP_SET_FLIP_CONTROL_GSL; + (*num_steps)++; + } + if (dc->hwss.program_triplebuffer && dc->debug.enable_tri_buf && current_mpc_pipe->plane_state->update_flags.raw) { + block_sequence[*num_steps].params.program_triplebuffer_params.dc = dc; + block_sequence[*num_steps].params.program_triplebuffer_params.pipe_ctx = current_mpc_pipe; + block_sequence[*num_steps].params.program_triplebuffer_params.enableTripleBuffer = current_mpc_pipe->plane_state->triplebuffer_flips; + block_sequence[*num_steps].func = HUBP_PROGRAM_TRIPLEBUFFER; + (*num_steps)++; + } + if (dc->hwss.update_plane_addr && current_mpc_pipe->plane_state->update_flags.bits.addr_update) { + block_sequence[*num_steps].params.update_plane_addr_params.dc = dc; + block_sequence[*num_steps].params.update_plane_addr_params.pipe_ctx = current_mpc_pipe; + block_sequence[*num_steps].func = HUBP_UPDATE_PLANE_ADDR; + (*num_steps)++; + } - if (hws->funcs.set_input_transfer_func && current_mpc_pipe->plane_state->update_flags.bits.gamma_change) { - block_sequence[*num_steps].params.set_input_transfer_func_params.dc = dc; - block_sequence[*num_steps].params.set_input_transfer_func_params.pipe_ctx = current_mpc_pipe; - block_sequence[*num_steps].params.set_input_transfer_func_params.plane_state = current_mpc_pipe->plane_state; - block_sequence[*num_steps].func = DPP_SET_INPUT_TRANSFER_FUNC; - (*num_steps)++; - } + if (hws->funcs.set_input_transfer_func && current_mpc_pipe->plane_state->update_flags.bits.gamma_change) { + block_sequence[*num_steps].params.set_input_transfer_func_params.dc = dc; + block_sequence[*num_steps].params.set_input_transfer_func_params.pipe_ctx = current_mpc_pipe; + block_sequence[*num_steps].params.set_input_transfer_func_params.plane_state = current_mpc_pipe->plane_state; + block_sequence[*num_steps].func = DPP_SET_INPUT_TRANSFER_FUNC; + (*num_steps)++; + } - if (dc->hwss.program_gamut_remap && current_mpc_pipe->plane_state->update_flags.bits.gamut_remap_change) { - block_sequence[*num_steps].params.program_gamut_remap_params.pipe_ctx = current_mpc_pipe; - block_sequence[*num_steps].func = DPP_PROGRAM_GAMUT_REMAP; - (*num_steps)++; - } - if (current_mpc_pipe->plane_state->update_flags.bits.input_csc_change) { - block_sequence[*num_steps].params.setup_dpp_params.pipe_ctx = current_mpc_pipe; - block_sequence[*num_steps].func = DPP_SETUP_DPP; - (*num_steps)++; - } - if (current_mpc_pipe->plane_state->update_flags.bits.coeff_reduction_change) { - block_sequence[*num_steps].params.program_bias_and_scale_params.pipe_ctx = current_mpc_pipe; - block_sequence[*num_steps].func = DPP_PROGRAM_BIAS_AND_SCALE; - (*num_steps)++; + if (dc->hwss.program_gamut_remap && current_mpc_pipe->plane_state->update_flags.bits.gamut_remap_change) { + block_sequence[*num_steps].params.program_gamut_remap_params.pipe_ctx = current_mpc_pipe; + block_sequence[*num_steps].func = DPP_PROGRAM_GAMUT_REMAP; + (*num_steps)++; + } + if (current_mpc_pipe->plane_state->update_flags.bits.input_csc_change) { + block_sequence[*num_steps].params.setup_dpp_params.pipe_ctx = current_mpc_pipe; + block_sequence[*num_steps].func = DPP_SETUP_DPP; + (*num_steps)++; + } + if (current_mpc_pipe->plane_state->update_flags.bits.coeff_reduction_change) { + block_sequence[*num_steps].params.program_bias_and_scale_params.pipe_ctx = current_mpc_pipe; + block_sequence[*num_steps].func = DPP_PROGRAM_BIAS_AND_SCALE; + (*num_steps)++; + } } if (hws->funcs.set_output_transfer_func && current_mpc_pipe->stream->update_flags.bits.out_tf) { block_sequence[*num_steps].params.set_output_transfer_func_params.dc = dc; diff --git a/drivers/gpu/drm/amd/display/dc/core/dc_resource.c b/drivers/gpu/drm/amd/display/dc/core/dc_resource.c index 1f576b2a1960..7812a86c85aa 100644 --- a/drivers/gpu/drm/amd/display/dc/core/dc_resource.c +++ b/drivers/gpu/drm/amd/display/dc/core/dc_resource.c @@ -81,6 +81,8 @@ */ #define VISUAL_CONFIRM_DPP_OFFSET_DENO 240 +#define DC_LOGGER \ + dc->ctx->logger #define DC_LOGGER_INIT(logger) #define UNABLE_TO_SPLIT -1 diff --git a/drivers/gpu/drm/amd/display/dc/dc.h b/drivers/gpu/drm/amd/display/dc/dc.h index 88e64b280c90..1f4292336692 100644 --- a/drivers/gpu/drm/amd/display/dc/dc.h +++ b/drivers/gpu/drm/amd/display/dc/dc.h @@ -1512,6 +1512,7 @@ struct dc_link { struct ddc_service *ddc; enum dp_panel_mode panel_mode; + bool panel_mode_initialized; bool aux_mode; /* Private to DC core */ diff --git a/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c b/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c index b3f02f451535..624e93e22b06 100644 --- a/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c +++ b/drivers/gpu/drm/amd/display/dc/dc_dmub_srv.c @@ -79,43 +79,43 @@ void dc_dmub_srv_wait_idle(struct dc_dmub_srv *dc_dmub_srv) } } -void dc_dmub_srv_clear_inbox0_ack(struct dc_dmub_srv *dmub_srv) +void dc_dmub_srv_clear_inbox0_ack(struct dc_dmub_srv *dc_dmub_srv) { - struct dmub_srv *dmub = dmub_srv->dmub; - struct dc_context *dc_ctx = dmub_srv->ctx; + struct dmub_srv *dmub = dc_dmub_srv->dmub; + struct dc_context *dc_ctx = dc_dmub_srv->ctx; enum dmub_status status = DMUB_STATUS_OK; status = dmub_srv_clear_inbox0_ack(dmub); if (status != DMUB_STATUS_OK) { DC_ERROR("Error clearing INBOX0 ack: status=%d\n", status); - dc_dmub_srv_log_diagnostic_data(dmub_srv); + dc_dmub_srv_log_diagnostic_data(dc_dmub_srv); } } -void dc_dmub_srv_wait_for_inbox0_ack(struct dc_dmub_srv *dmub_srv) +void dc_dmub_srv_wait_for_inbox0_ack(struct dc_dmub_srv *dc_dmub_srv) { - struct dmub_srv *dmub = dmub_srv->dmub; - struct dc_context *dc_ctx = dmub_srv->ctx; + struct dmub_srv *dmub = dc_dmub_srv->dmub; + struct dc_context *dc_ctx = dc_dmub_srv->ctx; enum dmub_status status = DMUB_STATUS_OK; status = dmub_srv_wait_for_inbox0_ack(dmub, 100000); if (status != DMUB_STATUS_OK) { DC_ERROR("Error waiting for INBOX0 HW Lock Ack\n"); - dc_dmub_srv_log_diagnostic_data(dmub_srv); + dc_dmub_srv_log_diagnostic_data(dc_dmub_srv); } } -void dc_dmub_srv_send_inbox0_cmd(struct dc_dmub_srv *dmub_srv, - union dmub_inbox0_data_register data) +void dc_dmub_srv_send_inbox0_cmd(struct dc_dmub_srv *dc_dmub_srv, + union dmub_inbox0_data_register data) { - struct dmub_srv *dmub = dmub_srv->dmub; - struct dc_context *dc_ctx = dmub_srv->ctx; + struct dmub_srv *dmub = dc_dmub_srv->dmub; + struct dc_context *dc_ctx = dc_dmub_srv->ctx; enum dmub_status status = DMUB_STATUS_OK; status = dmub_srv_send_inbox0_cmd(dmub, data); if (status != DMUB_STATUS_OK) { DC_ERROR("Error sending INBOX0 cmd\n"); - dc_dmub_srv_log_diagnostic_data(dmub_srv); + dc_dmub_srv_log_diagnostic_data(dc_dmub_srv); } } @@ -793,7 +793,10 @@ void dc_dmub_srv_log_diagnostic_data(struct dc_dmub_srv *dc_dmub_srv) { struct dmub_diagnostic_data diag_data = {0}; - if (!dc_dmub_srv || !dc_dmub_srv->dmub) { + if (!dc_dmub_srv) + return; + + if (!dc_dmub_srv->dmub) { DC_LOG_ERROR("%s: invalid parameters.", __func__); return; } @@ -1034,7 +1037,10 @@ void dc_dmub_srv_enable_dpia_trace(const struct dc *dc) enum dmub_status status; static const uint32_t timeout_us = 30; - if (!dc_dmub_srv || !dc_dmub_srv->dmub) { + if (!dc_dmub_srv) + return; + + if (!dc_dmub_srv->dmub) { DC_LOG_ERROR("%s: invalid parameters.", __func__); return; } diff --git a/drivers/gpu/drm/amd/display/dc/dc_dp_types.h b/drivers/gpu/drm/amd/display/dc/dc_dp_types.h index 83719f5bea49..f834c0fa3b17 100644 --- a/drivers/gpu/drm/amd/display/dc/dc_dp_types.h +++ b/drivers/gpu/drm/amd/display/dc/dc_dp_types.h @@ -1156,7 +1156,7 @@ struct dc_lttpr_caps { uint8_t max_ext_timeout; union dp_main_link_channel_coding_lttpr_cap main_link_channel_coding; union dp_128b_132b_supported_lttpr_link_rates supported_128b_132b_rates; - uint8_t aux_rd_interval[MAX_REPEATER_CNT - 1]; + uint8_t aux_rd_interval[MAX_REPEATER_CNT]; }; struct dc_dongle_dfp_cap_ext { diff --git a/drivers/gpu/drm/amd/display/dc/dc_helper.c b/drivers/gpu/drm/amd/display/dc/dc_helper.c index 0713a503f7f6..d22f70efe5f4 100644 --- a/drivers/gpu/drm/amd/display/dc/dc_helper.c +++ b/drivers/gpu/drm/amd/display/dc/dc_helper.c @@ -36,6 +36,9 @@ #include "dc_dmub_srv.h" #include "reg_helper.h" +#define DC_LOGGER \ + ctx->logger + static inline void submit_dmub_read_modify_write( struct dc_reg_helper_state *offload, const struct dc_context *ctx) diff --git a/drivers/gpu/drm/amd/display/dc/dc_types.h b/drivers/gpu/drm/amd/display/dc/dc_types.h index 6eaa02a80344..5a678ae2f2bd 100644 --- a/drivers/gpu/drm/amd/display/dc/dc_types.h +++ b/drivers/gpu/drm/amd/display/dc/dc_types.h @@ -789,6 +789,7 @@ struct dc_context { struct dc *dc; void *driver_context; /* e.g. amdgpu_device */ + struct dal_logger *logger; struct dc_perf_trace *perf_trace; void *cgs_device; diff --git a/drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c b/drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c index ed8936405dfa..597b858369f5 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dce_clock_source.c @@ -43,7 +43,8 @@ #define CTX \ clk_src->base.ctx -#define DC_LOGGER_INIT() +#define DC_LOGGER \ + CTX->logger #undef FN #define FN(reg_name, field_name) \ @@ -287,6 +288,7 @@ static bool calc_pll_dividers_in_range( } static uint32_t calculate_pixel_clock_pll_dividers( + struct dce110_clk_src *clk_src, struct calc_pll_clock_source *calc_pll_cs, struct pll_settings *pll_settings) { @@ -475,7 +477,7 @@ static uint32_t dce110_get_pix_clk_dividers_helper ( { uint32_t field = 0; uint32_t pll_calc_error = MAX_PLL_CALC_ERROR; - DC_LOGGER_INIT(); + /* Check if reference clock is external (not pcie/xtalin) * HW Dce80 spec: * 00 - PCIE_REFCLK, 01 - XTALIN, 02 - GENERICA, 03 - GENERICB @@ -518,12 +520,14 @@ static uint32_t dce110_get_pix_clk_dividers_helper ( /*Calculate Dividers by HDMI object, no SS case or SS case */ pll_calc_error = calculate_pixel_clock_pll_dividers( + clk_src, &clk_src->calc_pll_hdmi, pll_settings); else /*Calculate Dividers by default object, no SS case or SS case */ pll_calc_error = calculate_pixel_clock_pll_dividers( + clk_src, &clk_src->calc_pll, pll_settings); @@ -568,7 +572,6 @@ static uint32_t dce110_get_pix_clk_dividers( { struct dce110_clk_src *clk_src = TO_DCE110_CLK_SRC(cs); uint32_t pll_calc_error = MAX_PLL_CALC_ERROR; - DC_LOGGER_INIT(); if (pix_clk_params == NULL || pll_settings == NULL || pix_clk_params->requested_pix_clk_100hz == 0) { @@ -600,7 +603,6 @@ static uint32_t dce112_get_pix_clk_dividers( struct pll_settings *pll_settings) { struct dce110_clk_src *clk_src = TO_DCE110_CLK_SRC(cs); - DC_LOGGER_INIT(); if (pix_clk_params == NULL || pll_settings == NULL || pix_clk_params->requested_pix_clk_100hz == 0) { @@ -1254,8 +1256,7 @@ static uint32_t dcn3_get_pix_clk_dividers( struct pll_settings *pll_settings) { unsigned long long actual_pix_clk_100Hz = pix_clk_params ? pix_clk_params->requested_pix_clk_100hz : 0; - - DC_LOGGER_INIT(); + struct dce110_clk_src *clk_src = TO_DCE110_CLK_SRC(cs); if (pix_clk_params == NULL || pll_settings == NULL || pix_clk_params->requested_pix_clk_100hz == 0) { @@ -1319,7 +1320,6 @@ static const struct clock_source_funcs dce110_clk_src_funcs = { .get_pixel_clk_frequency_100hz = get_pixel_clk_frequency_100hz }; - static void get_ss_info_from_atombios( struct dce110_clk_src *clk_src, enum as_signal_type as_signal, @@ -1332,7 +1332,7 @@ static void get_ss_info_from_atombios( struct spread_spectrum_info *ss_info_cur; struct spread_spectrum_data *ss_data_cur; uint32_t i; - DC_LOGGER_INIT(); + if (ss_entries_num == NULL) { DC_LOG_SYNC( "Invalid entry !!!\n"); @@ -1467,6 +1467,7 @@ static void ss_info_from_atombios_create( } static bool calc_pll_max_vco_construct( + struct dce110_clk_src *clk_src, struct calc_pll_clock_source *calc_pll_cs, struct calc_pll_clock_source_init_data *init_data) { @@ -1618,6 +1619,7 @@ bool dce110_clk_src_construct( ss_info_from_atombios_create(clk_src); if (!calc_pll_max_vco_construct( + clk_src, &clk_src->calc_pll, &calc_pll_cs_init_data)) { ASSERT_CRITICAL(false); @@ -1632,7 +1634,7 @@ bool dce110_clk_src_construct( if (!calc_pll_max_vco_construct( - &clk_src->calc_pll_hdmi, &calc_pll_cs_init_data_hdmi)) { + clk_src, &clk_src->calc_pll_hdmi, &calc_pll_cs_init_data_hdmi)) { ASSERT_CRITICAL(false); goto unexpected_failure; } diff --git a/drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c b/drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c index 42c802afc468..f5d1070e2b59 100644 --- a/drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c +++ b/drivers/gpu/drm/amd/display/dc/dce/dmub_abm_lcd.c @@ -262,7 +262,7 @@ bool dmub_abm_set_pipe(struct abm *abm, { union dmub_rb_cmd cmd; struct dc_context *dc = abm->ctx; - uint32_t ramping_boundary = 0xFFFF; + uint8_t ramping_boundary = 0xFF; memset(&cmd, 0, sizeof(cmd)); cmd.abm_set_pipe.header.type = DMUB_CMD__ABM; diff --git a/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c b/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c index 14f5b4f00f72..c8aa8cc09234 100644 --- a/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c +++ b/drivers/gpu/drm/amd/display/dc/dce110/dce110_hw_sequencer.c @@ -82,7 +82,10 @@ #define CTX \ hws->ctx -#define DC_LOGGER_INIT() +#define DC_LOGGER \ + ctx->logger +#define DC_LOGGER_INIT() \ + struct dc_context *ctx = dc->ctx #define REG(reg)\ hws->regs->reg @@ -954,7 +957,7 @@ void dce110_edp_backlight_control( { struct dc_context *ctx = link->ctx; struct bp_transmitter_control cntl = { 0 }; - uint8_t pwrseq_instance; + uint8_t pwrseq_instance = 0; unsigned int pre_T11_delay = OLED_PRE_T11_DELAY; unsigned int post_T7_delay = OLED_POST_T7_DELAY; @@ -1007,7 +1010,8 @@ void dce110_edp_backlight_control( */ /* dc_service_sleep_in_milliseconds(50); */ /*edp 1.2*/ - pwrseq_instance = link->panel_cntl->pwrseq_inst; + if (link->panel_cntl) + pwrseq_instance = link->panel_cntl->pwrseq_inst; if (cntl.action == TRANSMITTER_CONTROL_BACKLIGHT_ON) { if (!link->dc->config.edp_no_power_sequencing) @@ -2463,6 +2467,7 @@ static bool wait_for_reset_trigger_to_occur( struct dc_context *dc_ctx, struct timing_generator *tg) { + struct dc_context *ctx = dc_ctx; bool rc = false; /* To avoid endless loop we wait at most @@ -2506,6 +2511,7 @@ static void dce110_enable_timing_synchronization( struct dc_context *dc_ctx = dc->ctx; struct dcp_gsl_params gsl_params = { 0 }; int i; + DC_LOGGER_INIT(); DC_SYNC_INFO("GSL: Setting-up...\n"); @@ -2551,6 +2557,7 @@ static void dce110_enable_per_frame_crtc_position_reset( struct dc_context *dc_ctx = dc->ctx; struct dcp_gsl_params gsl_params = { 0 }; int i; + DC_LOGGER_INIT(); gsl_params.gsl_group = 0; gsl_params.gsl_master = 0; @@ -2696,7 +2703,6 @@ static void dce110_program_front_end_for_pipe( unsigned int i; struct dce_hwseq *hws = dc->hwseq; - DC_LOGGER_INIT(); memset(&tbl_entry, 0, sizeof(tbl_entry)); memset(&adjust, 0, sizeof(adjust)); diff --git a/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c b/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c index 2b20180f1a32..18c942cfe530 100644 --- a/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dce112/dce112_resource.c @@ -1069,7 +1069,10 @@ static void bw_calcs_data_update_from_pplib(struct dc *dc) struct dm_pp_clock_levels clks = {0}; int memory_type_multiplier = MEMORY_TYPE_MULTIPLIER_CZ; - if (dc->bw_vbios && dc->bw_vbios->memory_type == bw_def_hbm) + if (!dc->bw_vbios) + return; + + if (dc->bw_vbios->memory_type == bw_def_hbm) memory_type_multiplier = MEMORY_TYPE_HBM; /*do system clock TODO PPLIB: after PPLIB implement, diff --git a/drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.c b/drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.c index 8db9f7514466..556c453666a7 100644 --- a/drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dce60/dce60_resource.c @@ -1355,7 +1355,6 @@ static bool dce64_construct( dc->caps.max_downscale_ratio = 200; dc->caps.i2c_speed_in_khz = 40; dc->caps.max_cursor_size = 64; - dc->caps.is_apu = true; /************************************************* * Create resources * diff --git a/drivers/gpu/drm/amd/display/dc/dce80/dce80_resource.c b/drivers/gpu/drm/amd/display/dc/dce80/dce80_resource.c index 061221394ce0..616cf2cfcc38 100644 --- a/drivers/gpu/drm/amd/display/dc/dce80/dce80_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dce80/dce80_resource.c @@ -1174,6 +1174,7 @@ static bool dce81_construct( dc->caps.max_cursor_size = 128; dc->caps.min_horizontal_blanking_period = 80; dc->caps.is_apu = true; + dc->debug = debug_defaults; /************************************************* * Create resources * diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.c index bf399819ca80..22ac2b7e49ae 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hubp.c @@ -749,7 +749,8 @@ bool hubp1_is_flip_pending(struct hubp *hubp) if (flip_pending) return true; - if (earliest_inuse_address.grph.addr.quad_part != hubp->request_address.grph.addr.quad_part) + if (hubp && + earliest_inuse_address.grph.addr.quad_part != hubp->request_address.grph.addr.quad_part) return true; return false; diff --git a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c index e71b79f5a66c..dd92ba8e1ce6 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c +++ b/drivers/gpu/drm/amd/display/dc/dcn10/dcn10_hw_sequencer.c @@ -57,7 +57,10 @@ #include "dce/dmub_outbox.h" #include "link.h" -#define DC_LOGGER_INIT(logger) +#define DC_LOGGER \ + dc_logger +#define DC_LOGGER_INIT(logger) \ + struct dal_logger *dc_logger = logger #define CTX \ hws->ctx @@ -1490,7 +1493,7 @@ void dcn10_init_hw(struct dc *dc) dc->clk_mgr->funcs->init_clocks(dc->clk_mgr); /* Align bw context with hw config when system resume. */ - if (dc->clk_mgr->clks.dispclk_khz != 0 && dc->clk_mgr->clks.dppclk_khz != 0) { + if (dc->clk_mgr && dc->clk_mgr->clks.dispclk_khz != 0 && dc->clk_mgr->clks.dppclk_khz != 0) { dc->current_state->bw_ctx.bw.dcn.clk.dispclk_khz = dc->clk_mgr->clks.dispclk_khz; dc->current_state->bw_ctx.bw.dcn.clk.dppclk_khz = dc->clk_mgr->clks.dppclk_khz; } @@ -1608,7 +1611,7 @@ void dcn10_init_hw(struct dc *dc) REG_UPDATE(DCFCLK_CNTL, DCFCLK_GATE_DIS, 0); } - if (dc->clk_mgr->funcs->notify_wm_ranges) + if (dc->clk_mgr && dc->clk_mgr->funcs->notify_wm_ranges) dc->clk_mgr->funcs->notify_wm_ranges(dc->clk_mgr); } @@ -1809,7 +1812,6 @@ static void log_tf(struct dc_context *ctx, // DC_LOG_ALL_TF_POINTS logs all channels of the tf int i = 0; - DC_LOGGER_INIT(ctx->logger); DC_LOG_GAMMA("Gamma Correction TF"); DC_LOG_ALL_GAMMA("Logging all tf points..."); DC_LOG_ALL_TF_CHANNELS("Logging all channels..."); @@ -1990,6 +1992,8 @@ static bool wait_for_reset_trigger_to_occur( { bool rc = false; + DC_LOGGER_INIT(dc_ctx->logger); + /* To avoid endless loop we wait at most * frames_to_wait_on_triggered_reset frames for the reset to occur. */ const uint32_t frames_to_wait_on_triggered_reset = 10; @@ -2117,6 +2121,8 @@ static int dcn10_align_pixel_clocks(struct dc *dc, int group_size, uint32_t dp_ref_clk_100hz = dc->res_pool->dp_clock_source->ctx->dc->clk_mgr->dprefclk_khz*10; + DC_LOGGER_INIT(dc_ctx->logger); + hw_crtc_timing = kcalloc(MAX_PIPES, sizeof(*hw_crtc_timing), GFP_KERNEL); if (!hw_crtc_timing) return master; @@ -2200,6 +2206,8 @@ void dcn10_enable_vblanks_synchronization( struct timing_generator *tg; int i, width, height, master; + DC_LOGGER_INIT(dc_ctx->logger); + for (i = 1; i < group_size; i++) { opp = grouped_pipes[i]->stream_res.opp; tg = grouped_pipes[i]->stream_res.tg; @@ -2263,6 +2271,8 @@ void dcn10_enable_timing_synchronization( struct timing_generator *tg; int i, width, height; + DC_LOGGER_INIT(dc_ctx->logger); + DC_SYNC_INFO("Setting up OTG reset trigger\n"); for (i = 1; i < group_size; i++) { @@ -2340,6 +2350,8 @@ void dcn10_enable_per_frame_crtc_position_reset( struct dc_context *dc_ctx = dc->ctx; int i; + DC_LOGGER_INIT(dc_ctx->logger); + DC_SYNC_INFO("Setting up\n"); for (i = 0; i < group_size; i++) if (grouped_pipes[i]->stream_res.tg->funcs->enable_crtc_reset) @@ -2994,8 +3006,6 @@ void dcn10_post_unlock_program_front_end( { int i; - DC_LOGGER_INIT(dc->ctx->logger); - for (i = 0; i < dc->res_pool->pipe_count; i++) { struct pipe_ctx *pipe_ctx = &context->res_ctx.pipe_ctx[i]; diff --git a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubp.c b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubp.c index aa252dc26326..0c999314e01d 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubp.c +++ b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hubp.c @@ -30,6 +30,8 @@ #include "reg_helper.h" #include "basics/conversion.h" +#define DC_LOGGER \ + ctx->logger #define DC_LOGGER_INIT(logger) #define REG(reg)\ @@ -925,7 +927,8 @@ bool hubp2_is_flip_pending(struct hubp *hubp) if (flip_pending) return true; - if (earliest_inuse_address.grph.addr.quad_part != hubp->request_address.grph.addr.quad_part) + if (hubp && + earliest_inuse_address.grph.addr.quad_part != hubp->request_address.grph.addr.quad_part) return true; return false; diff --git a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c index e52c2bb1b626..327bdde41c99 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c +++ b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_hwseq.c @@ -56,7 +56,10 @@ #include "link_hwss.h" #include "link.h" -#define DC_LOGGER_INIT(logger) +#define DC_LOGGER \ + dc_logger +#define DC_LOGGER_INIT(logger) \ + struct dal_logger *dc_logger = logger #define CTX \ hws->ctx @@ -1732,9 +1735,9 @@ static void dcn20_program_pipe( dc->res_pool->hubbub->funcs->program_det_size( dc->res_pool->hubbub, pipe_ctx->plane_res.hubp->inst, pipe_ctx->det_buffer_size_kb); - if (pipe_ctx->update_flags.raw || - (pipe_ctx->plane_state && pipe_ctx->plane_state->update_flags.raw) || - pipe_ctx->stream->update_flags.raw) + if (pipe_ctx->plane_state && (pipe_ctx->update_flags.raw || + pipe_ctx->plane_state->update_flags.raw || + pipe_ctx->stream->update_flags.raw)) dcn20_update_dchubp_dpp(dc, pipe_ctx, context); if (pipe_ctx->update_flags.bits.enable || @@ -1963,8 +1966,6 @@ void dcn20_post_unlock_program_front_end( unsigned int polling_interval_us = 1; struct dce_hwseq *hwseq = dc->hwseq; - DC_LOGGER_INIT(dc->ctx->logger); - for (i = 0; i < dc->res_pool->pipe_count; i++) if (context->res_ctx.pipe_ctx[i].update_flags.bits.disable) dc->hwss.disable_plane(dc, &dc->current_state->res_ctx.pipe_ctx[i]); @@ -2017,6 +2018,9 @@ void dcn20_post_unlock_program_front_end( } } + if (!hwseq) + return; + /* P-State support transitions: * Natural -> FPO: P-State disabled in prepare, force disallow anytime is safe * FPO -> Natural: Unforce anytime after FW disable is safe (P-State will assert naturally) @@ -2024,7 +2028,7 @@ void dcn20_post_unlock_program_front_end( * FPO -> Unsupported: P-State disabled in prepare, unforce disallow anytime is safe * FPO <-> SubVP: Force disallow is maintained on the FPO / SubVP pipes */ - if (hwseq && hwseq->funcs.update_force_pstate) + if (hwseq->funcs.update_force_pstate) dc->hwseq->funcs.update_force_pstate(dc, context); /* Only program the MALL registers after all the main and phantom pipes @@ -2262,6 +2266,9 @@ bool dcn20_wait_for_blank_complete( { int counter; + if (!opp) + return false; + for (counter = 0; counter < 1000; counter++) { if (opp->funcs->dpg_is_blanked(opp)) break; @@ -2748,7 +2755,11 @@ void dcn20_enable_stream(struct pipe_ctx *pipe_ctx) dccg->funcs->set_dpstreamclk(dccg, DTBCLK0, tg->inst, dp_hpo_inst); phyd32clk = get_phyd32clk_src(link); - dccg->funcs->enable_symclk32_se(dccg, dp_hpo_inst, phyd32clk); + if (link->cur_link_settings.link_rate == LINK_RATE_UNKNOWN) { + dccg->funcs->disable_symclk32_se(dccg, dp_hpo_inst); + } else { + dccg->funcs->enable_symclk32_se(dccg, dp_hpo_inst, phyd32clk); + } } else { } if (hws->funcs.calculate_dccg_k1_k2_values && dc->res_pool->dccg->funcs->set_pixel_rate_div) { diff --git a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_vmid.c b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_vmid.c index 96c263223315..5bc3bc60a2ac 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_vmid.c +++ b/drivers/gpu/drm/amd/display/dc/dcn20/dcn20_vmid.c @@ -38,6 +38,9 @@ #define FN(reg_name, field_name) \ vmid->shifts->field_name, vmid->masks->field_name +#define DC_LOGGER \ + CTX->logger + static void dcn20_wait_for_vmid_ready(struct dcn20_vmid *vmid) { /* According the hardware spec, we need to poll for the lowest diff --git a/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hubp.c b/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hubp.c index f976fac8dc3f..68cad55c72ab 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hubp.c +++ b/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hubp.c @@ -31,6 +31,8 @@ #include "dc_dmub_srv.h" +#define DC_LOGGER \ + ctx->logger #define DC_LOGGER_INIT(logger) #define REG(reg)\ diff --git a/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hwseq.c b/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hwseq.c index 1b08749b084b..840dfd237247 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hwseq.c +++ b/drivers/gpu/drm/amd/display/dc/dcn21/dcn21_hwseq.c @@ -142,7 +142,7 @@ static bool dmub_abm_set_pipe(struct abm *abm, uint32_t otg_inst, { union dmub_rb_cmd cmd; struct dc_context *dc = abm->ctx; - uint32_t ramping_boundary = 0xFFFF; + uint8_t ramping_boundary = 0xFF; memset(&cmd, 0, sizeof(cmd)); cmd.abm_set_pipe.header.type = DMUB_CMD__ABM; diff --git a/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_resource.c b/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_resource.c index de83acd12250..a4979737d878 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dcn30/dcn30_resource.c @@ -91,6 +91,8 @@ #include "amdgpu_socbb.h" #include "dc_dmub_srv.h" +#define DC_LOGGER \ + dc->ctx->logger #define DC_LOGGER_INIT(logger) enum dcn30_clk_src_array_id { diff --git a/drivers/gpu/drm/amd/display/dc/dcn301/dcn301_resource.c b/drivers/gpu/drm/amd/display/dc/dcn301/dcn301_resource.c index 9485fda890cd..333ee2b53b51 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn301/dcn301_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dcn301/dcn301_resource.c @@ -92,6 +92,8 @@ #define TO_DCN301_RES_POOL(pool)\ container_of(pool, struct dcn301_resource_pool, base) +#define DC_LOGGER \ + dc->ctx->logger #define DC_LOGGER_INIT(logger) enum dcn301_clk_src_array_id { diff --git a/drivers/gpu/drm/amd/display/dc/dcn302/dcn302_resource.c b/drivers/gpu/drm/amd/display/dc/dcn302/dcn302_resource.c index 447abcd593be..26a194244782 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn302/dcn302_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dcn302/dcn302_resource.c @@ -74,6 +74,8 @@ #include "nbio/nbio_7_4_offset.h" #include "amdgpu_socbb.h" +#define DC_LOGGER \ + dc->ctx->logger #define DC_LOGGER_INIT(logger) static const struct dc_debug_options debug_defaults_drv = { diff --git a/drivers/gpu/drm/amd/display/dc/dcn303/dcn303_resource.c b/drivers/gpu/drm/amd/display/dc/dcn303/dcn303_resource.c index adf4989177f7..b896273d3097 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn303/dcn303_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dcn303/dcn303_resource.c @@ -56,6 +56,8 @@ #include "dml/dcn303/dcn303_fpu.h" +#define DC_LOGGER \ + dc->ctx->logger #define DC_LOGGER_INIT(logger) diff --git a/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_hwseq.c b/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_hwseq.c index 22da2007601e..31065b4fa3d9 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_hwseq.c +++ b/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_hwseq.c @@ -254,10 +254,10 @@ void dcn31_init_hw(struct dc *dc) if (!dcb->funcs->is_accelerated_mode(dcb) && dc->res_pool->hubbub->funcs->init_watermarks) dc->res_pool->hubbub->funcs->init_watermarks(dc->res_pool->hubbub); - if (dc->clk_mgr->funcs->notify_wm_ranges) + if (dc->clk_mgr && dc->clk_mgr->funcs->notify_wm_ranges) dc->clk_mgr->funcs->notify_wm_ranges(dc->clk_mgr); - if (dc->clk_mgr->funcs->set_hard_max_memclk && !dc->clk_mgr->dc_mode_softmax_enabled) + if (dc->clk_mgr && dc->clk_mgr->funcs->set_hard_max_memclk && !dc->clk_mgr->dc_mode_softmax_enabled) dc->clk_mgr->funcs->set_hard_max_memclk(dc->clk_mgr); if (dc->res_pool->hubbub->funcs->force_pstate_change_control) diff --git a/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_resource.c b/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_resource.c index 84e3df49be2f..1176c5366543 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dcn31/dcn31_resource.c @@ -104,6 +104,8 @@ #include "link_enc_cfg.h" +#define DC_LOGGER \ + dc->ctx->logger #define DC_LOGGER_INIT(logger) enum dcn31_clk_src_array_id { diff --git a/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_hwseq.c b/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_hwseq.c index 33a8626bda73..5b197807080e 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_hwseq.c +++ b/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_hwseq.c @@ -62,7 +62,7 @@ #define REG(reg)\ hws->regs->reg #define DC_LOGGER \ - dc->ctx->logger + stream->ctx->logger #undef FN diff --git a/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_resource.c b/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_resource.c index 6e52851bc031..9d0e9c38c1a6 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dcn314/dcn314_resource.c @@ -118,6 +118,8 @@ #define regBIF_BX2_BIOS_SCRATCH_6 0x003e #define regBIF_BX2_BIOS_SCRATCH_6_BASE_IDX 1 +#define DC_LOGGER \ + dc->ctx->logger #define DC_LOGGER_INIT(logger) enum dcn31_clk_src_array_id { diff --git a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_hwseq.c b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_hwseq.c index 1a24fc8b5367..14b88a78cd44 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_hwseq.c +++ b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_hwseq.c @@ -58,7 +58,7 @@ #define REG(reg)\ hws->regs->reg #define DC_LOGGER \ - dc->ctx->logger + stream->ctx->logger #undef FN diff --git a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c index ddc0a444a054..ab6b4f44f537 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource.c @@ -1645,6 +1645,9 @@ static void dcn32_enable_phantom_plane(struct dc *dc, else phantom_plane = dc_create_plane_state(dc); + if (!phantom_plane) + continue; + memcpy(&phantom_plane->address, &curr_pipe->plane_state->address, sizeof(phantom_plane->address)); memcpy(&phantom_plane->scaling_quality, &curr_pipe->plane_state->scaling_quality, sizeof(phantom_plane->scaling_quality)); @@ -2647,8 +2650,10 @@ static struct pipe_ctx *dcn32_acquire_idle_pipe_for_head_pipe_in_layer( struct resource_context *old_ctx = &stream->ctx->dc->current_state->res_ctx; int head_index; - if (!head_pipe) + if (!head_pipe) { ASSERT(0); + return NULL; + } /* * Modified from dcn20_acquire_idle_pipe_for_layer diff --git a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource_helpers.c b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource_helpers.c index 3ad2b48954e0..57a662fc99ef 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource_helpers.c +++ b/drivers/gpu/drm/amd/display/dc/dcn32/dcn32_resource_helpers.c @@ -233,12 +233,12 @@ bool dcn32_is_center_timing(struct pipe_ctx *pipe) pipe->stream->timing.v_addressable != pipe->stream->src.height) { is_center_timing = true; } - } - if (pipe->plane_state) { - if (pipe->stream->timing.v_addressable != pipe->plane_state->dst_rect.height && - pipe->stream->timing.v_addressable != pipe->plane_state->src_rect.height) { - is_center_timing = true; + if (pipe->plane_state) { + if (pipe->stream->timing.v_addressable != pipe->plane_state->dst_rect.height && + pipe->stream->timing.v_addressable != pipe->plane_state->src_rect.height) { + is_center_timing = true; + } } } diff --git a/drivers/gpu/drm/amd/display/dc/dcn321/dcn321_resource.c b/drivers/gpu/drm/amd/display/dc/dcn321/dcn321_resource.c index 4289cd1643ec..e02920969c32 100644 --- a/drivers/gpu/drm/amd/display/dc/dcn321/dcn321_resource.c +++ b/drivers/gpu/drm/amd/display/dc/dcn321/dcn321_resource.c @@ -1197,7 +1197,7 @@ static struct stream_encoder *dcn321_stream_encoder_create( vpg = dcn321_vpg_create(ctx, vpg_inst); afmt = dcn321_afmt_create(ctx, afmt_inst); - if (!enc1 || !vpg || !afmt) { + if (!enc1 || !vpg || !afmt || eng_id >= ARRAY_SIZE(stream_enc_regs)) { kfree(enc1); kfree(vpg); kfree(afmt); diff --git a/drivers/gpu/drm/amd/display/dc/dml/dcn20/dcn20_fpu.c b/drivers/gpu/drm/amd/display/dc/dml/dcn20/dcn20_fpu.c index 8a5a038fd855..9e22c539f878 100644 --- a/drivers/gpu/drm/amd/display/dc/dml/dcn20/dcn20_fpu.c +++ b/drivers/gpu/drm/amd/display/dc/dml/dcn20/dcn20_fpu.c @@ -34,6 +34,8 @@ #include "link.h" #include "dcn20_fpu.h" +#define DC_LOGGER \ + dc->ctx->logger #define DC_LOGGER_INIT(logger) #ifndef MAX diff --git a/drivers/gpu/drm/amd/display/dc/inc/hw/clk_mgr_internal.h b/drivers/gpu/drm/amd/display/dc/inc/hw/clk_mgr_internal.h index cff5fd55a0ad..5de4991d8ec7 100644 --- a/drivers/gpu/drm/amd/display/dc/inc/hw/clk_mgr_internal.h +++ b/drivers/gpu/drm/amd/display/dc/inc/hw/clk_mgr_internal.h @@ -73,7 +73,7 @@ enum dentist_divider_range { clk_mgr->base.ctx #define DC_LOGGER \ - clk_mgr->base.ctx->logger + dc->ctx->logger diff --git a/drivers/gpu/drm/amd/display/dc/link/accessories/link_dp_cts.c b/drivers/gpu/drm/amd/display/dc/link/accessories/link_dp_cts.c index fe4282771cd0..b5775a12f6e5 100644 --- a/drivers/gpu/drm/amd/display/dc/link/accessories/link_dp_cts.c +++ b/drivers/gpu/drm/amd/display/dc/link/accessories/link_dp_cts.c @@ -878,8 +878,11 @@ bool dp_set_test_pattern( break; } + if (!pipe_ctx->stream) + return false; + if (pipe_ctx->stream_res.tg->funcs->lock_doublebuffer_enable) { - if (pipe_ctx->stream && should_use_dmub_lock(pipe_ctx->stream->link)) { + if (should_use_dmub_lock(pipe_ctx->stream->link)) { union dmub_hw_lock_flags hw_locks = { 0 }; struct dmub_hw_lock_inst_flags inst_flags = { 0 }; diff --git a/drivers/gpu/drm/amd/display/dc/link/hwss/link_hwss_dio.c b/drivers/gpu/drm/amd/display/dc/link/hwss/link_hwss_dio.c index 1328a0ade342..a52a0eef06b3 100644 --- a/drivers/gpu/drm/amd/display/dc/link/hwss/link_hwss_dio.c +++ b/drivers/gpu/drm/amd/display/dc/link/hwss/link_hwss_dio.c @@ -58,7 +58,10 @@ void reset_dio_stream_encoder(struct pipe_ctx *pipe_ctx) struct link_encoder *link_enc = link_enc_cfg_get_link_enc(pipe_ctx->stream->link); struct stream_encoder *stream_enc = pipe_ctx->stream_res.stream_enc; - if (stream_enc && stream_enc->funcs->disable_fifo) + if (!stream_enc) + return; + + if (stream_enc->funcs->disable_fifo) stream_enc->funcs->disable_fifo(stream_enc); link_enc->funcs->connect_dig_be_to_fe( diff --git a/drivers/gpu/drm/amd/display/dc/link/hwss/link_hwss_dpia.c b/drivers/gpu/drm/amd/display/dc/link/hwss/link_hwss_dpia.c index 861f3cd5b356..46fb3649bc86 100644 --- a/drivers/gpu/drm/amd/display/dc/link/hwss/link_hwss_dpia.c +++ b/drivers/gpu/drm/amd/display/dc/link/hwss/link_hwss_dpia.c @@ -27,6 +27,8 @@ #include "link_hwss_dio.h" #include "link_enc_cfg.h" +#define DC_LOGGER \ + link->ctx->logger #define DC_LOGGER_INIT(logger) static void update_dpia_stream_allocation_table(struct dc_link *link, diff --git a/drivers/gpu/drm/amd/display/dc/link/link_detection.c b/drivers/gpu/drm/amd/display/dc/link/link_detection.c index f3ba8e2a6fae..0cd58cf1961b 100644 --- a/drivers/gpu/drm/amd/display/dc/link/link_detection.c +++ b/drivers/gpu/drm/amd/display/dc/link/link_detection.c @@ -47,6 +47,8 @@ #include "dm_helpers.h" #include "clk_mgr.h" +#define DC_LOGGER \ + link->ctx->logger #define DC_LOGGER_INIT(logger) #define LINK_INFO(...) \ @@ -322,6 +324,7 @@ static void query_dp_dual_mode_adaptor( bool is_type2_dongle = false; int retry_count = 2; struct dp_hdmi_dongle_signature_data *dongle_signature; + struct dc_link *link = ddc->link; /* Assume we have no valid DP passive dongle connected */ *dongle = DISPLAY_DONGLE_NONE; diff --git a/drivers/gpu/drm/amd/display/dc/link/link_dpms.c b/drivers/gpu/drm/amd/display/dc/link/link_dpms.c index 2ce2d9ff7568..0396f0a22a0f 100644 --- a/drivers/gpu/drm/amd/display/dc/link/link_dpms.c +++ b/drivers/gpu/drm/amd/display/dc/link/link_dpms.c @@ -56,7 +56,10 @@ #include "dccg.h" #include "clk_mgr.h" #include "atomfirmware.h" -#define DC_LOGGER_INIT(logger) +#define DC_LOGGER \ + dc_logger +#define DC_LOGGER_INIT(logger) \ + struct dal_logger *dc_logger = logger #define LINK_INFO(...) \ DC_LOG_HW_HOTPLUG( \ @@ -2300,13 +2303,13 @@ void link_set_dpms_off(struct pipe_ctx *pipe_ctx) struct dc_link *link = stream->sink->link; struct vpg *vpg = pipe_ctx->stream_res.stream_enc->vpg; + DC_LOGGER_INIT(pipe_ctx->stream->ctx->logger); + ASSERT(is_master_pipe_for_link(link, pipe_ctx)); if (dp_is_128b_132b_signal(pipe_ctx)) vpg = pipe_ctx->stream_res.hpo_dp_stream_enc->vpg; - DC_LOGGER_INIT(pipe_ctx->stream->ctx->logger); - if (pipe_ctx->stream->sink) { if (pipe_ctx->stream->sink->sink_signal != SIGNAL_TYPE_VIRTUAL && pipe_ctx->stream->sink->sink_signal != SIGNAL_TYPE_NONE) { @@ -2408,13 +2411,13 @@ void link_set_dpms_on( bool apply_edp_fast_boot_optimization = pipe_ctx->stream->apply_edp_fast_boot_optimization; + DC_LOGGER_INIT(pipe_ctx->stream->ctx->logger); + ASSERT(is_master_pipe_for_link(link, pipe_ctx)); if (dp_is_128b_132b_signal(pipe_ctx)) vpg = pipe_ctx->stream_res.hpo_dp_stream_enc->vpg; - DC_LOGGER_INIT(pipe_ctx->stream->ctx->logger); - if (pipe_ctx->stream->sink) { if (pipe_ctx->stream->sink->sink_signal != SIGNAL_TYPE_VIRTUAL && pipe_ctx->stream->sink->sink_signal != SIGNAL_TYPE_NONE) { diff --git a/drivers/gpu/drm/amd/display/dc/link/link_factory.c b/drivers/gpu/drm/amd/display/dc/link/link_factory.c index eb7c9f226af5..ae3e7baf846d 100644 --- a/drivers/gpu/drm/amd/display/dc/link/link_factory.c +++ b/drivers/gpu/drm/amd/display/dc/link/link_factory.c @@ -46,6 +46,8 @@ #include "gpio_service_interface.h" #include "atomfirmware.h" +#define DC_LOGGER \ + dc_ctx->logger #define DC_LOGGER_INIT(logger) #define LINK_INFO(...) \ @@ -369,6 +371,7 @@ static enum transmitter translate_encoder_to_transmitter( static uint8_t translate_dig_inst_to_pwrseq_inst(struct dc_link *link) { + struct dc_context *dc_ctx = link->ctx; uint8_t pwrseq_inst = 0xF; switch (link->eng_id) { diff --git a/drivers/gpu/drm/amd/display/dc/link/protocols/link_ddc.c b/drivers/gpu/drm/amd/display/dc/link/protocols/link_ddc.c index ecfd83299e75..c2d40979203e 100644 --- a/drivers/gpu/drm/amd/display/dc/link/protocols/link_ddc.c +++ b/drivers/gpu/drm/amd/display/dc/link/protocols/link_ddc.c @@ -38,6 +38,8 @@ #include "dm_helpers.h" #include "atomfirmware.h" +#define DC_LOGGER \ + ddc_service->ctx->logger #define DC_LOGGER_INIT(logger) static const uint8_t DP_VGA_DONGLE_BRANCH_DEV_NAME[] = "DpVga"; diff --git a/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_capability.c b/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_capability.c index 56999657f4d8..8ffa6739640e 100644 --- a/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_capability.c +++ b/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_capability.c @@ -2217,7 +2217,7 @@ bool dp_verify_link_cap_with_retries( memset(&link->verified_link_cap, 0, sizeof(struct dc_link_settings)); - if (!link_detect_connection_type(link, &type) || type == dc_connection_none) { + if (link->link_enc && (!link_detect_connection_type(link, &type) || type == dc_connection_none)) { link->verified_link_cap = fail_safe_link_settings; break; } else if (dp_verify_link_cap(link, known_limit_link_setting, &fail_count)) { diff --git a/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_irq_handler.c b/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_irq_handler.c index e047bbeaa49a..34bf8a9ef738 100644 --- a/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_irq_handler.c +++ b/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_irq_handler.c @@ -38,6 +38,8 @@ #include "link/link_dpms.h" #include "dm_helpers.h" +#define DC_LOGGER \ + link->ctx->logger #define DC_LOGGER_INIT(logger) bool dp_parse_link_loss_status( diff --git a/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_training.c b/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_training.c index 539f57858dfe..aa46e0e84ce4 100644 --- a/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_training.c +++ b/drivers/gpu/drm/amd/display/dc/link/protocols/link_dp_training.c @@ -1608,7 +1608,8 @@ bool perform_link_training_with_retries( */ bool result; result = cp_psp->funcs.enable_assr(cp_psp->handle, link); - if (!result && link->panel_mode != DP_PANEL_MODE_EDP) + if (!result && link->panel_mode_initialized && + link->panel_mode != DP_PANEL_MODE_EDP) panel_mode = DP_PANEL_MODE_DEFAULT; } } diff --git a/drivers/gpu/drm/amd/display/dc/link/protocols/link_dpcd.c b/drivers/gpu/drm/amd/display/dc/link/protocols/link_dpcd.c index fc50931c2aec..1fdbc6aa0368 100644 --- a/drivers/gpu/drm/amd/display/dc/link/protocols/link_dpcd.c +++ b/drivers/gpu/drm/amd/display/dc/link/protocols/link_dpcd.c @@ -164,7 +164,8 @@ static void dpcd_extend_address_range( if (new_addr_range.start != in_address || new_addr_range.end != end_address) { *out_address = new_addr_range.start; *out_size = ADDRESS_RANGE_SIZE(new_addr_range.start, new_addr_range.end); - *out_data = kzalloc(*out_size * sizeof(**out_data), GFP_KERNEL); + *out_data = kcalloc(*out_size, sizeof(**out_data), GFP_KERNEL); + ASSERT(*out_data); } } diff --git a/drivers/gpu/drm/amd/display/dc/link/protocols/link_edp_panel_control.c b/drivers/gpu/drm/amd/display/dc/link/protocols/link_edp_panel_control.c index d4d92da153ec..8f62c350cadb 100644 --- a/drivers/gpu/drm/amd/display/dc/link/protocols/link_edp_panel_control.c +++ b/drivers/gpu/drm/amd/display/dc/link/protocols/link_edp_panel_control.c @@ -38,6 +38,8 @@ #include "dc/dc_dmub_srv.h" #include "dce/dmub_replay.h" #include "abm.h" +#define DC_LOGGER \ + link->ctx->logger #define DC_LOGGER_INIT(logger) #define DP_SINK_PR_ENABLE_AND_CONFIGURATION 0x37B @@ -88,6 +90,7 @@ void dp_set_panel_mode(struct dc_link *link, enum dp_panel_mode panel_mode) } link->panel_mode = panel_mode; + link->panel_mode_initialized = true; DC_LOG_DETECTION_DP_CAPS("Link: %d eDP panel mode supported: %d " "eDP panel mode enabled: %d \n", link->link_index, diff --git a/drivers/gpu/drm/amd/display/dc/os_types.h b/drivers/gpu/drm/amd/display/dc/os_types.h index aad8095660c9..6c4578d347af 100644 --- a/drivers/gpu/drm/amd/display/dc/os_types.h +++ b/drivers/gpu/drm/amd/display/dc/os_types.h @@ -37,6 +37,7 @@ #include #include +#include #include #include "cgs_common.h" diff --git a/drivers/gpu/drm/amd/display/include/logger_types.h b/drivers/gpu/drm/amd/display/include/logger_types.h index 3bf08a60c45c..f49632608b08 100644 --- a/drivers/gpu/drm/amd/display/include/logger_types.h +++ b/drivers/gpu/drm/amd/display/include/logger_types.h @@ -28,53 +28,41 @@ #include "os_types.h" -#define MAX_NAME_LEN 32 - -#define DC_LOG_ERROR(...) DRM_ERROR(__VA_ARGS__) -#define DC_LOG_WARNING(...) DRM_WARN(__VA_ARGS__) -#define DC_LOG_DEBUG(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_DC(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_DTN(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_ERROR(...) drm_err((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_WARNING(...) drm_warn((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_DEBUG(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_DC(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) #define DC_LOG_SURFACE(...) pr_debug("[SURFACE]:"__VA_ARGS__) -#define DC_LOG_CURSOR(...) pr_debug("[CURSOR]:"__VA_ARGS__) -#define DC_LOG_PFLIP(...) pr_debug("[PFLIP]:"__VA_ARGS__) -#define DC_LOG_VBLANK(...) pr_debug("[VBLANK]:"__VA_ARGS__) -#define DC_LOG_HW_HOTPLUG(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_HW_HOTPLUG(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) #define DC_LOG_HW_LINK_TRAINING(...) pr_debug("[HW_LINK_TRAINING]:"__VA_ARGS__) -#define DC_LOG_HW_SET_MODE(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_HW_RESUME_S3(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_HW_RESUME_S3(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) #define DC_LOG_HW_AUDIO(...) pr_debug("[HW_AUDIO]:"__VA_ARGS__) -#define DC_LOG_HW_HPD_IRQ(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_MST(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_HW_HPD_IRQ(...) drm_dbg_dp((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_MST(...) drm_dbg_dp((DC_LOGGER)->dev, __VA_ARGS__) #define DC_LOG_SCALER(...) pr_debug("[SCALER]:"__VA_ARGS__) #define DC_LOG_BIOS(...) pr_debug("[BIOS]:"__VA_ARGS__) #define DC_LOG_BANDWIDTH_CALCS(...) pr_debug("[BANDWIDTH_CALCS]:"__VA_ARGS__) -#define DC_LOG_BANDWIDTH_VALIDATION(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_I2C_AUX(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_SYNC(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_BACKLIGHT(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_FEATURE_OVERRIDE(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_DETECTION_EDID_PARSER(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_DETECTION_DP_CAPS(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_RESOURCE(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_BANDWIDTH_VALIDATION(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_SYNC(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_BACKLIGHT(...) drm_dbg_dp((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_DETECTION_EDID_PARSER(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_DETECTION_DP_CAPS(...) drm_dbg_dp((DC_LOGGER)->dev, __VA_ARGS__) #define DC_LOG_DML(...) pr_debug("[DML]:"__VA_ARGS__) -#define DC_LOG_EVENT_MODE_SET(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_EVENT_DETECTION(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_EVENT_LINK_TRAINING(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_EVENT_LINK_LOSS(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_EVENT_UNDERFLOW(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_EVENT_MODE_SET(...) drm_dbg_kms((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_EVENT_DETECTION(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_EVENT_LINK_TRAINING(...) \ + drm_dbg_dp((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_EVENT_LINK_LOSS(...) drm_dbg_dp((DC_LOGGER)->dev, __VA_ARGS__) #define DC_LOG_IF_TRACE(...) pr_debug("[IF_TRACE]:"__VA_ARGS__) -#define DC_LOG_PERF_TRACE(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_RETIMER_REDRIVER(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_PERF_TRACE(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_RETIMER_REDRIVER(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) #define DC_LOG_GAMMA(...) pr_debug("[GAMMA]:"__VA_ARGS__) #define DC_LOG_ALL_GAMMA(...) pr_debug("[GAMMA]:"__VA_ARGS__) #define DC_LOG_ALL_TF_CHANNELS(...) pr_debug("[GAMMA]:"__VA_ARGS__) -#define DC_LOG_DSC(...) DRM_DEBUG_KMS(__VA_ARGS__) +#define DC_LOG_DSC(...) drm_dbg_dp((DC_LOGGER)->dev, __VA_ARGS__) #define DC_LOG_SMU(...) pr_debug("[SMU_MSG]:"__VA_ARGS__) -#define DC_LOG_DWB(...) DRM_DEBUG_KMS(__VA_ARGS__) -#define DC_LOG_DP2(...) DRM_DEBUG_KMS(__VA_ARGS__) - -struct dal_logger; +#define DC_LOG_DWB(...) drm_dbg((DC_LOGGER)->dev, __VA_ARGS__) +#define DC_LOG_DP2(...) drm_dbg_dp((DC_LOGGER)->dev, __VA_ARGS__) struct dc_log_buffer_ctx { char *buf; @@ -82,88 +70,8 @@ struct dc_log_buffer_ctx { size_t size; }; -enum dc_log_type { - LOG_ERROR = 0, - LOG_WARNING, - LOG_DEBUG, - LOG_DC, - LOG_DTN, - LOG_SURFACE, - LOG_HW_HOTPLUG, - LOG_HW_LINK_TRAINING, - LOG_HW_SET_MODE, - LOG_HW_RESUME_S3, - LOG_HW_AUDIO, - LOG_HW_HPD_IRQ, - LOG_MST, - LOG_SCALER, - LOG_BIOS, - LOG_BANDWIDTH_CALCS, - LOG_BANDWIDTH_VALIDATION, - LOG_I2C_AUX, - LOG_SYNC, - LOG_BACKLIGHT, - LOG_FEATURE_OVERRIDE, - LOG_DETECTION_EDID_PARSER, - LOG_DETECTION_DP_CAPS, - LOG_RESOURCE, - LOG_DML, - LOG_EVENT_MODE_SET, - LOG_EVENT_DETECTION, - LOG_EVENT_LINK_TRAINING, - LOG_EVENT_LINK_LOSS, - LOG_EVENT_UNDERFLOW, - LOG_IF_TRACE, - LOG_PERF_TRACE, - LOG_DISPLAYSTATS, - LOG_HDMI_RETIMER_REDRIVER, - LOG_DSC, - LOG_SMU_MSG, - LOG_DC2RESERVED4, - LOG_DC2RESERVED5, - LOG_DWB, - LOG_GAMMA_DEBUG, - LOG_MAX_HW_POINTS, - LOG_ALL_TF_CHANNELS, - LOG_SAMPLE_1DLUT, - LOG_DP2, - LOG_DC2RESERVED12, +struct dal_logger { + struct drm_device *dev; }; -#define DC_MIN_LOG_MASK ((1 << LOG_ERROR) | \ - (1 << LOG_DETECTION_EDID_PARSER)) - -#define DC_DEFAULT_LOG_MASK ((1ULL << LOG_ERROR) | \ - (1ULL << LOG_WARNING) | \ - (1ULL << LOG_EVENT_MODE_SET) | \ - (1ULL << LOG_EVENT_DETECTION) | \ - (1ULL << LOG_EVENT_LINK_TRAINING) | \ - (1ULL << LOG_EVENT_LINK_LOSS) | \ - (1ULL << LOG_EVENT_UNDERFLOW) | \ - (1ULL << LOG_RESOURCE) | \ - (1ULL << LOG_FEATURE_OVERRIDE) | \ - (1ULL << LOG_DETECTION_EDID_PARSER) | \ - (1ULL << LOG_DC) | \ - (1ULL << LOG_HW_HOTPLUG) | \ - (1ULL << LOG_HW_SET_MODE) | \ - (1ULL << LOG_HW_RESUME_S3) | \ - (1ULL << LOG_HW_HPD_IRQ) | \ - (1ULL << LOG_SYNC) | \ - (1ULL << LOG_BANDWIDTH_VALIDATION) | \ - (1ULL << LOG_MST) | \ - (1ULL << LOG_DETECTION_DP_CAPS) | \ - (1ULL << LOG_BACKLIGHT)) | \ - (1ULL << LOG_I2C_AUX) | \ - (1ULL << LOG_IF_TRACE) | \ - (1ULL << LOG_HDMI_FRL) | \ - (1ULL << LOG_SCALER) | \ - (1ULL << LOG_DTN) /* | \ - (1ULL << LOG_DEBUG) | \ - (1ULL << LOG_BIOS) | \ - (1ULL << LOG_SURFACE) | \ - (1ULL << LOG_DML) | \ - (1ULL << LOG_HW_LINK_TRAINING) | \ - (1ULL << LOG_HW_AUDIO)| \ - (1ULL << LOG_BANDWIDTH_CALCS)*/ - #endif /* __DAL_LOGGER_TYPES_H__ */ diff --git a/drivers/gpu/drm/amd/pm/amdgpu_pm.c b/drivers/gpu/drm/amd/pm/amdgpu_pm.c index ada58a3f1533..470d5d5e21f0 100644 --- a/drivers/gpu/drm/amd/pm/amdgpu_pm.c +++ b/drivers/gpu/drm/amd/pm/amdgpu_pm.c @@ -1442,6 +1442,8 @@ static ssize_t amdgpu_set_pp_power_profile_mode(struct device *dev, if (ret) return -EINVAL; parameter_size++; + if (!tmp_str) + break; while (isspace(*tmp_str)) tmp_str++; } diff --git a/drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c b/drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c index 442b347dcfbc..fa7bdaefb46d 100644 --- a/drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c +++ b/drivers/gpu/drm/amd/pm/legacy-dpm/si_dpm.c @@ -5239,7 +5239,8 @@ static int si_init_smc_table(struct amdgpu_device *adev) break; } - if (adev->pm.dpm.platform_caps & ATOM_PP_PLATFORM_CAP_HARDWAREDC) + if ((adev->flags & AMD_IS_MOBILITY) && + (adev->pm.dpm.platform_caps & ATOM_PP_PLATFORM_CAP_HARDWAREDC)) table->systemFlags |= PPSMC_SYSTEMFLAG_GPIO_DC; if (adev->pm.dpm.platform_caps & ATOM_PP_PLATFORM_CAP_REGULATOR_HOT) { @@ -7210,6 +7211,19 @@ static void si_parse_pplib_non_clock_info(struct amdgpu_device *adev, adev->pm.dpm.uvd_ps = rps; } +static void si_update_limits(struct amdgpu_clock_and_voltage_limits *limits, + struct rv7xx_pl *pl) +{ + if (pl->sclk > limits->sclk) + limits->sclk = pl->sclk; + if (pl->mclk > limits->mclk) + limits->mclk = pl->mclk; + if (pl->vddc > limits->vddc) + limits->vddc = pl->vddc; + if (pl->vddci > limits->vddci) + limits->vddci = pl->vddci; +} + static void si_parse_pplib_clock_info(struct amdgpu_device *adev, struct amdgpu_ps *rps, int index, union pplib_clock_info *clock_info) @@ -7218,7 +7232,6 @@ static void si_parse_pplib_clock_info(struct amdgpu_device *adev, struct evergreen_power_info *eg_pi = evergreen_get_pi(adev); struct si_power_info *si_pi = si_get_pi(adev); struct si_ps *ps = si_get_ps(rps); - struct amdgpu_clock_and_voltage_limits *limits; u16 leakage_voltage; struct rv7xx_pl *pl = &ps->performance_levels[index]; int ret; @@ -7283,26 +7296,14 @@ static void si_parse_pplib_clock_info(struct amdgpu_device *adev, * VBIOS can contain conflicting values between: * - the maximum allowed clocks and voltages on AC or DC * - the clocks and voltages in power states on AC or DC + * + * Assume that the AC limits are the maximum of all power states, + * and the DC limits are the maximum of battery power states. */ + si_update_limits(&adev->pm.dpm.dyn_state.max_clock_voltage_on_ac, pl); if ((rps->class & ATOM_PPLIB_CLASSIFICATION_UI_MASK) == - ATOM_PPLIB_CLASSIFICATION_UI_PERFORMANCE) - limits = &adev->pm.dpm.dyn_state.max_clock_voltage_on_ac; - else if ((rps->class & ATOM_PPLIB_CLASSIFICATION_UI_MASK) == ATOM_PPLIB_CLASSIFICATION_UI_BATTERY) - limits = &adev->pm.dpm.dyn_state.max_clock_voltage_on_dc; - else - limits = NULL; - - if (limits) { - if (pl->sclk > limits->sclk) - limits->sclk = pl->sclk; - if (pl->mclk > limits->mclk) - limits->mclk = pl->mclk; - if (pl->vddc > limits->vddc) - limits->vddc = pl->vddc; - if (pl->vddci > limits->vddci) - limits->vddci = pl->vddci; - } + si_update_limits(&adev->pm.dpm.dyn_state.max_clock_voltage_on_dc, pl); } union pplib_power_state { diff --git a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c index 7e37354a0341..640350bbd908 100644 --- a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c +++ b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomctrl.c @@ -269,15 +269,21 @@ static const ATOM_VOLTAGE_OBJECT_V3 *atomctrl_lookup_voltage_type_v3( unsigned int offset = offsetof(ATOM_VOLTAGE_OBJECT_INFO_V3_1, asVoltageObj[0]); uint8_t *start = (uint8_t *)voltage_object_info_table; - while (offset < size) { + while (offset + sizeof(ATOM_VOLTAGE_OBJECT_HEADER_V3) <= size) { const ATOM_VOLTAGE_OBJECT_V3 *voltage_object = (const ATOM_VOLTAGE_OBJECT_V3 *)(start + offset); + u16 obj_size; + + obj_size = le16_to_cpu(voltage_object->asGpioVoltageObj.sHeader.usSize); + if (obj_size < sizeof(voltage_object->asGpioVoltageObj.sHeader) || + offset + obj_size > size) + break; if (voltage_type == voltage_object->asGpioVoltageObj.sHeader.ucVoltageType && voltage_mode == voltage_object->asGpioVoltageObj.sHeader.ucVoltageMode) return voltage_object; - offset += le16_to_cpu(voltage_object->asGpioVoltageObj.sHeader.usSize); + offset += obj_size; } return NULL; diff --git a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomfwctrl.c b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomfwctrl.c index a47a47238e2b..e5aa78718202 100644 --- a/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomfwctrl.c +++ b/drivers/gpu/drm/amd/pm/powerplay/hwmgr/ppatomfwctrl.c @@ -36,16 +36,21 @@ static const union atom_voltage_object_v4 *pp_atomfwctrl_lookup_voltage_type_v4( offsetof(struct atom_voltage_objects_info_v4_1, voltage_object[0]); unsigned long start = (unsigned long)voltage_object_info_table; - while (offset < size) { + while (offset + sizeof(struct atom_voltage_object_header_v4) <= size) { const union atom_voltage_object_v4 *voltage_object = (const union atom_voltage_object_v4 *)(start + offset); + u16 obj_size; + + obj_size = le16_to_cpu(voltage_object->gpio_voltage_obj.header.object_size); + if (obj_size < sizeof(voltage_object->gpio_voltage_obj.header) || + offset + obj_size > size) + break; if (voltage_type == voltage_object->gpio_voltage_obj.header.voltage_type && voltage_mode == voltage_object->gpio_voltage_obj.header.voltage_mode) return voltage_object; - offset += le16_to_cpu(voltage_object->gpio_voltage_obj.header.object_size); - + offset += obj_size; } return NULL; diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu11/arcturus_ppt.c b/drivers/gpu/drm/amd/pm/swsmu/smu11/arcturus_ppt.c index a47898487de0..cb4f7f164911 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/smu11/arcturus_ppt.c +++ b/drivers/gpu/drm/amd/pm/swsmu/smu11/arcturus_ppt.c @@ -2237,10 +2237,6 @@ static void arcturus_get_unique_id(struct smu_context *smu) id = ((uint64_t)bottom32 << 32) | top32; adev->unique_id = id; - /* For Arcturus-and-later, unique_id == serial_number, so convert it to a - * 16-digit HEX string for convenience and backwards-compatibility - */ - sprintf(adev->serial, "%llx", id); } static int arcturus_set_df_cstate(struct smu_context *smu, diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu11/sienna_cichlid_ppt.c b/drivers/gpu/drm/amd/pm/swsmu/smu11/sienna_cichlid_ppt.c index 2fdb982e70ef..0da0b6792cdf 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/smu11/sienna_cichlid_ppt.c +++ b/drivers/gpu/drm/amd/pm/swsmu/smu11/sienna_cichlid_ppt.c @@ -1989,8 +1989,6 @@ static void sienna_cichlid_get_unique_id(struct smu_context *smu) out: adev->unique_id = ((uint64_t)upper32 << 32) | lower32; - if (adev->serial[0] == '\0') - sprintf(adev->serial, "%016llx", adev->unique_id); } static int sienna_cichlid_get_uclk_dpm_states(struct smu_context *smu, uint32_t *clocks_in_khz, uint32_t *num_states) diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu13/aldebaran_ppt.c b/drivers/gpu/drm/amd/pm/swsmu/smu13/aldebaran_ppt.c index d1fd643e7a48..337e7722d25e 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/smu13/aldebaran_ppt.c +++ b/drivers/gpu/drm/amd/pm/swsmu/smu13/aldebaran_ppt.c @@ -1622,8 +1622,6 @@ static void aldebaran_get_unique_id(struct smu_context *smu) out: adev->unique_id = ((uint64_t)upper32 << 32) | lower32; - if (adev->serial[0] == '\0') - sprintf(adev->serial, "%016llx", adev->unique_id); } static bool aldebaran_is_baco_supported(struct smu_context *smu) diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c index fe6c42a736d0..6ef3140edacd 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c +++ b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_0_ppt.c @@ -1931,8 +1931,6 @@ static void smu_v13_0_0_get_unique_id(struct smu_context *smu) out: adev->unique_id = ((uint64_t)upper32 << 32) | lower32; - if (adev->serial[0] == '\0') - sprintf(adev->serial, "%016llx", adev->unique_id); } static int smu_v13_0_0_get_fan_speed_pwm(struct smu_context *smu, @@ -2449,8 +2447,13 @@ static int smu_v13_0_0_mode1_reset(struct smu_context *smu) break; } - if (!ret) + if (!ret) { + /* disable mmio access while doing mode 1 reset*/ + smu->adev->no_hw_access = true; + /* ensure no_hw_access is globally visible before any MMIO */ + smp_mb(); msleep(SMU13_MODE1_RESET_WAIT_TIME_IN_MS); + } return ret; } diff --git a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c index 4edb63146122..ac2b60512e2d 100644 --- a/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c +++ b/drivers/gpu/drm/amd/pm/swsmu/smu13/smu_v13_0_6_ppt.c @@ -1858,8 +1858,6 @@ static void smu_v13_0_6_get_unique_id(struct smu_context *smu) (struct PPTable_t *)smu_table->driver_pptable; adev->unique_id = pptable->PublicSerialNumber_AID; - if (adev->serial[0] == '\0') - sprintf(adev->serial, "%016llx", adev->unique_id); } static bool smu_v13_0_6_is_baco_supported(struct smu_context *smu) diff --git a/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c b/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c index 78122b35a0cb..68d425f8e2a3 100644 --- a/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c +++ b/drivers/gpu/drm/aspeed/aspeed_gfx_drv.c @@ -193,13 +193,12 @@ static int aspeed_gfx_load(struct drm_device *drm) } reset_control_deassert(priv->rst); - priv->clk = devm_clk_get(drm->dev, NULL); + priv->clk = devm_clk_get_enabled(drm->dev, NULL); if (IS_ERR(priv->clk)) { dev_err(&pdev->dev, - "missing or invalid clk device tree entry"); + "failed to get and enable clock\n"); return PTR_ERR(priv->clk); } - clk_prepare_enable(priv->clk); /* Sanitize control registers */ writel(0, priv->base + CRT_CTRL1); diff --git a/drivers/gpu/drm/display/drm_dp_helper.c b/drivers/gpu/drm/display/drm_dp_helper.c index 24ff1cab4e25..1a54a52407b1 100644 --- a/drivers/gpu/drm/display/drm_dp_helper.c +++ b/drivers/gpu/drm/display/drm_dp_helper.c @@ -3843,7 +3843,9 @@ static int dp_aux_backlight_update_status(struct backlight_device *bd) if (!backlight_is_blank(bd)) { if (!bl->enabled) { - drm_edp_backlight_enable(bl->aux, &bl->info, brightness); + ret = drm_edp_backlight_enable(bl->aux, &bl->info, brightness); + if (ret) + return ret; bl->enabled = true; return 0; } diff --git a/drivers/gpu/drm/drm_debugfs.c b/drivers/gpu/drm/drm_debugfs.c index 41b0682c638e..4dce905d7b3e 100644 --- a/drivers/gpu/drm/drm_debugfs.c +++ b/drivers/gpu/drm/drm_debugfs.c @@ -40,7 +40,7 @@ #include #include #include -#include +#include #include "drm_crtc_internal.h" #include "drm_internal.h" @@ -182,31 +182,31 @@ static const struct file_operations drm_debugfs_fops = { /** * drm_debugfs_gpuva_info - dump the given DRM GPU VA space * @m: pointer to the &seq_file to write - * @mgr: the &drm_gpuva_manager representing the GPU VA space + * @gpuvm: the &drm_gpuvm representing the GPU VA space * * Dumps the GPU VA mappings of a given DRM GPU VA manager. * * For each DRM GPU VA space drivers should call this function from their * &drm_info_list's show callback. * - * Returns: 0 on success, -ENODEV if the &mgr is not initialized + * Returns: 0 on success, -ENODEV if the &gpuvm is not initialized */ int drm_debugfs_gpuva_info(struct seq_file *m, - struct drm_gpuva_manager *mgr) + struct drm_gpuvm *gpuvm) { - struct drm_gpuva *va, *kva = &mgr->kernel_alloc_node; + struct drm_gpuva *va, *kva = &gpuvm->kernel_alloc_node; - if (!mgr->name) + if (!gpuvm->name) return -ENODEV; seq_printf(m, "DRM GPU VA space (%s) [0x%016llx;0x%016llx]\n", - mgr->name, mgr->mm_start, mgr->mm_start + mgr->mm_range); + gpuvm->name, gpuvm->mm_start, gpuvm->mm_start + gpuvm->mm_range); seq_printf(m, "Kernel reserved node [0x%016llx;0x%016llx]\n", kva->va.addr, kva->va.addr + kva->va.range); seq_puts(m, "\n"); seq_puts(m, " VAs | start | range | end | object | object offset\n"); seq_puts(m, "-------------------------------------------------------------------------------------------------------------\n"); - drm_gpuva_for_each_va(va, mgr) { + drm_gpuvm_for_each_va(va, gpuvm) { if (unlikely(va == kva)) continue; diff --git a/drivers/gpu/drm/drm_fb_helper.c b/drivers/gpu/drm/drm_fb_helper.c index 298a8974adad..ac1a3d9809b8 100644 --- a/drivers/gpu/drm/drm_fb_helper.c +++ b/drivers/gpu/drm/drm_fb_helper.c @@ -1837,10 +1837,10 @@ __drm_fb_helper_initial_config_and_unlock(struct drm_fb_helper *fb_helper) ret = drm_fb_helper_single_fb_probe(fb_helper); if (ret < 0) { - if (ret == -EAGAIN) { - fb_helper->deferred_setup = true; + /* modesets are probed but fb_helper->fb isn't; defer restore too */ + fb_helper->deferred_setup = true; + if (ret == -EAGAIN) ret = 0; - } mutex_unlock(&fb_helper->lock); goto err_drm_fb_helper_release_info; diff --git a/drivers/gpu/drm/drm_gpuva_mgr.c b/drivers/gpu/drm/drm_gpuva_mgr.c deleted file mode 100644 index ad99c9cfedac..000000000000 --- a/drivers/gpu/drm/drm_gpuva_mgr.c +++ /dev/null @@ -1,1723 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * Copyright (c) 2022 Red Hat. - * - * Permission is hereby granted, free of charge, to any person obtaining a - * copy of this software and associated documentation files (the "Software"), - * to deal in the Software without restriction, including without limitation - * the rights to use, copy, modify, merge, publish, distribute, sublicense, - * and/or sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following conditions: - * - * The above copyright notice and this permission notice shall be included in - * all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, - * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL - * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR - * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, - * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - * - * Authors: - * Danilo Krummrich - * - */ - -#include - -#include -#include - -/** - * DOC: Overview - * - * The DRM GPU VA Manager, represented by struct drm_gpuva_manager keeps track - * of a GPU's virtual address (VA) space and manages the corresponding virtual - * mappings represented by &drm_gpuva objects. It also keeps track of the - * mapping's backing &drm_gem_object buffers. - * - * &drm_gem_object buffers maintain a list of &drm_gpuva objects representing - * all existent GPU VA mappings using this &drm_gem_object as backing buffer. - * - * GPU VAs can be flagged as sparse, such that drivers may use GPU VAs to also - * keep track of sparse PTEs in order to support Vulkan 'Sparse Resources'. - * - * The GPU VA manager internally uses a rb-tree to manage the - * &drm_gpuva mappings within a GPU's virtual address space. - * - * The &drm_gpuva_manager contains a special &drm_gpuva representing the - * portion of VA space reserved by the kernel. This node is initialized together - * with the GPU VA manager instance and removed when the GPU VA manager is - * destroyed. - * - * In a typical application drivers would embed struct drm_gpuva_manager and - * struct drm_gpuva within their own driver specific structures, there won't be - * any memory allocations of its own nor memory allocations of &drm_gpuva - * entries. - * - * The data structures needed to store &drm_gpuvas within the &drm_gpuva_manager - * are contained within struct drm_gpuva already. Hence, for inserting - * &drm_gpuva entries from within dma-fence signalling critical sections it is - * enough to pre-allocate the &drm_gpuva structures. - */ - -/** - * DOC: Split and Merge - * - * Besides its capability to manage and represent a GPU VA space, the - * &drm_gpuva_manager also provides functions to let the &drm_gpuva_manager - * calculate a sequence of operations to satisfy a given map or unmap request. - * - * Therefore the DRM GPU VA manager provides an algorithm implementing splitting - * and merging of existent GPU VA mappings with the ones that are requested to - * be mapped or unmapped. This feature is required by the Vulkan API to - * implement Vulkan 'Sparse Memory Bindings' - drivers UAPIs often refer to this - * as VM BIND. - * - * Drivers can call drm_gpuva_sm_map() to receive a sequence of callbacks - * containing map, unmap and remap operations for a given newly requested - * mapping. The sequence of callbacks represents the set of operations to - * execute in order to integrate the new mapping cleanly into the current state - * of the GPU VA space. - * - * Depending on how the new GPU VA mapping intersects with the existent mappings - * of the GPU VA space the &drm_gpuva_fn_ops callbacks contain an arbitrary - * amount of unmap operations, a maximum of two remap operations and a single - * map operation. The caller might receive no callback at all if no operation is - * required, e.g. if the requested mapping already exists in the exact same way. - * - * The single map operation represents the original map operation requested by - * the caller. - * - * &drm_gpuva_op_unmap contains a 'keep' field, which indicates whether the - * &drm_gpuva to unmap is physically contiguous with the original mapping - * request. Optionally, if 'keep' is set, drivers may keep the actual page table - * entries for this &drm_gpuva, adding the missing page table entries only and - * update the &drm_gpuva_manager's view of things accordingly. - * - * Drivers may do the same optimization, namely delta page table updates, also - * for remap operations. This is possible since &drm_gpuva_op_remap consists of - * one unmap operation and one or two map operations, such that drivers can - * derive the page table update delta accordingly. - * - * Note that there can't be more than two existent mappings to split up, one at - * the beginning and one at the end of the new mapping, hence there is a - * maximum of two remap operations. - * - * Analogous to drm_gpuva_sm_map() drm_gpuva_sm_unmap() uses &drm_gpuva_fn_ops - * to call back into the driver in order to unmap a range of GPU VA space. The - * logic behind this function is way simpler though: For all existent mappings - * enclosed by the given range unmap operations are created. For mappings which - * are only partically located within the given range, remap operations are - * created such that those mappings are split up and re-mapped partically. - * - * As an alternative to drm_gpuva_sm_map() and drm_gpuva_sm_unmap(), - * drm_gpuva_sm_map_ops_create() and drm_gpuva_sm_unmap_ops_create() can be used - * to directly obtain an instance of struct drm_gpuva_ops containing a list of - * &drm_gpuva_op, which can be iterated with drm_gpuva_for_each_op(). This list - * contains the &drm_gpuva_ops analogous to the callbacks one would receive when - * calling drm_gpuva_sm_map() or drm_gpuva_sm_unmap(). While this way requires - * more memory (to allocate the &drm_gpuva_ops), it provides drivers a way to - * iterate the &drm_gpuva_op multiple times, e.g. once in a context where memory - * allocations are possible (e.g. to allocate GPU page tables) and once in the - * dma-fence signalling critical path. - * - * To update the &drm_gpuva_manager's view of the GPU VA space - * drm_gpuva_insert() and drm_gpuva_remove() may be used. These functions can - * safely be used from &drm_gpuva_fn_ops callbacks originating from - * drm_gpuva_sm_map() or drm_gpuva_sm_unmap(). However, it might be more - * convenient to use the provided helper functions drm_gpuva_map(), - * drm_gpuva_remap() and drm_gpuva_unmap() instead. - * - * The following diagram depicts the basic relationships of existent GPU VA - * mappings, a newly requested mapping and the resulting mappings as implemented - * by drm_gpuva_sm_map() - it doesn't cover any arbitrary combinations of these. - * - * 1) Requested mapping is identical. Replace it, but indicate the backing PTEs - * could be kept. - * - * :: - * - * 0 a 1 - * old: |-----------| (bo_offset=n) - * - * 0 a 1 - * req: |-----------| (bo_offset=n) - * - * 0 a 1 - * new: |-----------| (bo_offset=n) - * - * - * 2) Requested mapping is identical, except for the BO offset, hence replace - * the mapping. - * - * :: - * - * 0 a 1 - * old: |-----------| (bo_offset=n) - * - * 0 a 1 - * req: |-----------| (bo_offset=m) - * - * 0 a 1 - * new: |-----------| (bo_offset=m) - * - * - * 3) Requested mapping is identical, except for the backing BO, hence replace - * the mapping. - * - * :: - * - * 0 a 1 - * old: |-----------| (bo_offset=n) - * - * 0 b 1 - * req: |-----------| (bo_offset=n) - * - * 0 b 1 - * new: |-----------| (bo_offset=n) - * - * - * 4) Existent mapping is a left aligned subset of the requested one, hence - * replace the existent one. - * - * :: - * - * 0 a 1 - * old: |-----| (bo_offset=n) - * - * 0 a 2 - * req: |-----------| (bo_offset=n) - * - * 0 a 2 - * new: |-----------| (bo_offset=n) - * - * .. note:: - * We expect to see the same result for a request with a different BO - * and/or non-contiguous BO offset. - * - * - * 5) Requested mapping's range is a left aligned subset of the existent one, - * but backed by a different BO. Hence, map the requested mapping and split - * the existent one adjusting its BO offset. - * - * :: - * - * 0 a 2 - * old: |-----------| (bo_offset=n) - * - * 0 b 1 - * req: |-----| (bo_offset=n) - * - * 0 b 1 a' 2 - * new: |-----|-----| (b.bo_offset=n, a.bo_offset=n+1) - * - * .. note:: - * We expect to see the same result for a request with a different BO - * and/or non-contiguous BO offset. - * - * - * 6) Existent mapping is a superset of the requested mapping. Split it up, but - * indicate that the backing PTEs could be kept. - * - * :: - * - * 0 a 2 - * old: |-----------| (bo_offset=n) - * - * 0 a 1 - * req: |-----| (bo_offset=n) - * - * 0 a 1 a' 2 - * new: |-----|-----| (a.bo_offset=n, a'.bo_offset=n+1) - * - * - * 7) Requested mapping's range is a right aligned subset of the existent one, - * but backed by a different BO. Hence, map the requested mapping and split - * the existent one, without adjusting the BO offset. - * - * :: - * - * 0 a 2 - * old: |-----------| (bo_offset=n) - * - * 1 b 2 - * req: |-----| (bo_offset=m) - * - * 0 a 1 b 2 - * new: |-----|-----| (a.bo_offset=n,b.bo_offset=m) - * - * - * 8) Existent mapping is a superset of the requested mapping. Split it up, but - * indicate that the backing PTEs could be kept. - * - * :: - * - * 0 a 2 - * old: |-----------| (bo_offset=n) - * - * 1 a 2 - * req: |-----| (bo_offset=n+1) - * - * 0 a' 1 a 2 - * new: |-----|-----| (a'.bo_offset=n, a.bo_offset=n+1) - * - * - * 9) Existent mapping is overlapped at the end by the requested mapping backed - * by a different BO. Hence, map the requested mapping and split up the - * existent one, without adjusting the BO offset. - * - * :: - * - * 0 a 2 - * old: |-----------| (bo_offset=n) - * - * 1 b 3 - * req: |-----------| (bo_offset=m) - * - * 0 a 1 b 3 - * new: |-----|-----------| (a.bo_offset=n,b.bo_offset=m) - * - * - * 10) Existent mapping is overlapped by the requested mapping, both having the - * same backing BO with a contiguous offset. Indicate the backing PTEs of - * the old mapping could be kept. - * - * :: - * - * 0 a 2 - * old: |-----------| (bo_offset=n) - * - * 1 a 3 - * req: |-----------| (bo_offset=n+1) - * - * 0 a' 1 a 3 - * new: |-----|-----------| (a'.bo_offset=n, a.bo_offset=n+1) - * - * - * 11) Requested mapping's range is a centered subset of the existent one - * having a different backing BO. Hence, map the requested mapping and split - * up the existent one in two mappings, adjusting the BO offset of the right - * one accordingly. - * - * :: - * - * 0 a 3 - * old: |-----------------| (bo_offset=n) - * - * 1 b 2 - * req: |-----| (bo_offset=m) - * - * 0 a 1 b 2 a' 3 - * new: |-----|-----|-----| (a.bo_offset=n,b.bo_offset=m,a'.bo_offset=n+2) - * - * - * 12) Requested mapping is a contiguous subset of the existent one. Split it - * up, but indicate that the backing PTEs could be kept. - * - * :: - * - * 0 a 3 - * old: |-----------------| (bo_offset=n) - * - * 1 a 2 - * req: |-----| (bo_offset=n+1) - * - * 0 a' 1 a 2 a'' 3 - * old: |-----|-----|-----| (a'.bo_offset=n, a.bo_offset=n+1, a''.bo_offset=n+2) - * - * - * 13) Existent mapping is a right aligned subset of the requested one, hence - * replace the existent one. - * - * :: - * - * 1 a 2 - * old: |-----| (bo_offset=n+1) - * - * 0 a 2 - * req: |-----------| (bo_offset=n) - * - * 0 a 2 - * new: |-----------| (bo_offset=n) - * - * .. note:: - * We expect to see the same result for a request with a different bo - * and/or non-contiguous bo_offset. - * - * - * 14) Existent mapping is a centered subset of the requested one, hence - * replace the existent one. - * - * :: - * - * 1 a 2 - * old: |-----| (bo_offset=n+1) - * - * 0 a 3 - * req: |----------------| (bo_offset=n) - * - * 0 a 3 - * new: |----------------| (bo_offset=n) - * - * .. note:: - * We expect to see the same result for a request with a different bo - * and/or non-contiguous bo_offset. - * - * - * 15) Existent mappings is overlapped at the beginning by the requested mapping - * backed by a different BO. Hence, map the requested mapping and split up - * the existent one, adjusting its BO offset accordingly. - * - * :: - * - * 1 a 3 - * old: |-----------| (bo_offset=n) - * - * 0 b 2 - * req: |-----------| (bo_offset=m) - * - * 0 b 2 a' 3 - * new: |-----------|-----| (b.bo_offset=m,a.bo_offset=n+2) - */ - -/** - * DOC: Locking - * - * Generally, the GPU VA manager does not take care of locking itself, it is - * the drivers responsibility to take care about locking. Drivers might want to - * protect the following operations: inserting, removing and iterating - * &drm_gpuva objects as well as generating all kinds of operations, such as - * split / merge or prefetch. - * - * The GPU VA manager also does not take care of the locking of the backing - * &drm_gem_object buffers GPU VA lists by itself; drivers are responsible to - * enforce mutual exclusion using either the GEMs dma_resv lock or alternatively - * a driver specific external lock. For the latter see also - * drm_gem_gpuva_set_lock(). - * - * However, the GPU VA manager contains lockdep checks to ensure callers of its - * API hold the corresponding lock whenever the &drm_gem_objects GPU VA list is - * accessed by functions such as drm_gpuva_link() or drm_gpuva_unlink(). - */ - -/** - * DOC: Examples - * - * This section gives two examples on how to let the DRM GPUVA Manager generate - * &drm_gpuva_op in order to satisfy a given map or unmap request and how to - * make use of them. - * - * The below code is strictly limited to illustrate the generic usage pattern. - * To maintain simplicitly, it doesn't make use of any abstractions for common - * code, different (asyncronous) stages with fence signalling critical paths, - * any other helpers or error handling in terms of freeing memory and dropping - * previously taken locks. - * - * 1) Obtain a list of &drm_gpuva_op to create a new mapping:: - * - * // Allocates a new &drm_gpuva. - * struct drm_gpuva * driver_gpuva_alloc(void); - * - * // Typically drivers would embedd the &drm_gpuva_manager and &drm_gpuva - * // structure in individual driver structures and lock the dma-resv with - * // drm_exec or similar helpers. - * int driver_mapping_create(struct drm_gpuva_manager *mgr, - * u64 addr, u64 range, - * struct drm_gem_object *obj, u64 offset) - * { - * struct drm_gpuva_ops *ops; - * struct drm_gpuva_op *op - * - * driver_lock_va_space(); - * ops = drm_gpuva_sm_map_ops_create(mgr, addr, range, - * obj, offset); - * if (IS_ERR(ops)) - * return PTR_ERR(ops); - * - * drm_gpuva_for_each_op(op, ops) { - * struct drm_gpuva *va; - * - * switch (op->op) { - * case DRM_GPUVA_OP_MAP: - * va = driver_gpuva_alloc(); - * if (!va) - * ; // unwind previous VA space updates, - * // free memory and unlock - * - * driver_vm_map(); - * drm_gpuva_map(mgr, va, &op->map); - * drm_gpuva_link(va); - * - * break; - * case DRM_GPUVA_OP_REMAP: { - * struct drm_gpuva *prev = NULL, *next = NULL; - * - * va = op->remap.unmap->va; - * - * if (op->remap.prev) { - * prev = driver_gpuva_alloc(); - * if (!prev) - * ; // unwind previous VA space - * // updates, free memory and - * // unlock - * } - * - * if (op->remap.next) { - * next = driver_gpuva_alloc(); - * if (!next) - * ; // unwind previous VA space - * // updates, free memory and - * // unlock - * } - * - * driver_vm_remap(); - * drm_gpuva_remap(prev, next, &op->remap); - * - * drm_gpuva_unlink(va); - * if (prev) - * drm_gpuva_link(prev); - * if (next) - * drm_gpuva_link(next); - * - * break; - * } - * case DRM_GPUVA_OP_UNMAP: - * va = op->unmap->va; - * - * driver_vm_unmap(); - * drm_gpuva_unlink(va); - * drm_gpuva_unmap(&op->unmap); - * - * break; - * default: - * break; - * } - * } - * driver_unlock_va_space(); - * - * return 0; - * } - * - * 2) Receive a callback for each &drm_gpuva_op to create a new mapping:: - * - * struct driver_context { - * struct drm_gpuva_manager *mgr; - * struct drm_gpuva *new_va; - * struct drm_gpuva *prev_va; - * struct drm_gpuva *next_va; - * }; - * - * // ops to pass to drm_gpuva_manager_init() - * static const struct drm_gpuva_fn_ops driver_gpuva_ops = { - * .sm_step_map = driver_gpuva_map, - * .sm_step_remap = driver_gpuva_remap, - * .sm_step_unmap = driver_gpuva_unmap, - * }; - * - * // Typically drivers would embedd the &drm_gpuva_manager and &drm_gpuva - * // structure in individual driver structures and lock the dma-resv with - * // drm_exec or similar helpers. - * int driver_mapping_create(struct drm_gpuva_manager *mgr, - * u64 addr, u64 range, - * struct drm_gem_object *obj, u64 offset) - * { - * struct driver_context ctx; - * struct drm_gpuva_ops *ops; - * struct drm_gpuva_op *op; - * int ret = 0; - * - * ctx.mgr = mgr; - * - * ctx.new_va = kzalloc(sizeof(*ctx.new_va), GFP_KERNEL); - * ctx.prev_va = kzalloc(sizeof(*ctx.prev_va), GFP_KERNEL); - * ctx.next_va = kzalloc(sizeof(*ctx.next_va), GFP_KERNEL); - * if (!ctx.new_va || !ctx.prev_va || !ctx.next_va) { - * ret = -ENOMEM; - * goto out; - * } - * - * driver_lock_va_space(); - * ret = drm_gpuva_sm_map(mgr, &ctx, addr, range, obj, offset); - * driver_unlock_va_space(); - * - * out: - * kfree(ctx.new_va); - * kfree(ctx.prev_va); - * kfree(ctx.next_va); - * return ret; - * } - * - * int driver_gpuva_map(struct drm_gpuva_op *op, void *__ctx) - * { - * struct driver_context *ctx = __ctx; - * - * drm_gpuva_map(ctx->mgr, ctx->new_va, &op->map); - * - * drm_gpuva_link(ctx->new_va); - * - * // prevent the new GPUVA from being freed in - * // driver_mapping_create() - * ctx->new_va = NULL; - * - * return 0; - * } - * - * int driver_gpuva_remap(struct drm_gpuva_op *op, void *__ctx) - * { - * struct driver_context *ctx = __ctx; - * - * drm_gpuva_remap(ctx->prev_va, ctx->next_va, &op->remap); - * - * drm_gpuva_unlink(op->remap.unmap->va); - * kfree(op->remap.unmap->va); - * - * if (op->remap.prev) { - * drm_gpuva_link(ctx->prev_va); - * ctx->prev_va = NULL; - * } - * - * if (op->remap.next) { - * drm_gpuva_link(ctx->next_va); - * ctx->next_va = NULL; - * } - * - * return 0; - * } - * - * int driver_gpuva_unmap(struct drm_gpuva_op *op, void *__ctx) - * { - * drm_gpuva_unlink(op->unmap.va); - * drm_gpuva_unmap(&op->unmap); - * kfree(op->unmap.va); - * - * return 0; - * } - */ - -#define to_drm_gpuva(__node) container_of((__node), struct drm_gpuva, rb.node) - -#define GPUVA_START(node) ((node)->va.addr) -#define GPUVA_LAST(node) ((node)->va.addr + (node)->va.range - 1) - -/* We do not actually use drm_gpuva_it_next(), tell the compiler to not complain - * about this. - */ -INTERVAL_TREE_DEFINE(struct drm_gpuva, rb.node, u64, rb.__subtree_last, - GPUVA_START, GPUVA_LAST, static __maybe_unused, - drm_gpuva_it) - -static int __drm_gpuva_insert(struct drm_gpuva_manager *mgr, - struct drm_gpuva *va); -static void __drm_gpuva_remove(struct drm_gpuva *va); - -static bool -drm_gpuva_check_overflow(u64 addr, u64 range) -{ - u64 end; - - return WARN(check_add_overflow(addr, range, &end), - "GPUVA address limited to %zu bytes.\n", sizeof(end)); -} - -static bool -drm_gpuva_in_mm_range(struct drm_gpuva_manager *mgr, u64 addr, u64 range) -{ - u64 end = addr + range; - u64 mm_start = mgr->mm_start; - u64 mm_end = mm_start + mgr->mm_range; - - return addr >= mm_start && end <= mm_end; -} - -static bool -drm_gpuva_in_kernel_node(struct drm_gpuva_manager *mgr, u64 addr, u64 range) -{ - u64 end = addr + range; - u64 kstart = mgr->kernel_alloc_node.va.addr; - u64 krange = mgr->kernel_alloc_node.va.range; - u64 kend = kstart + krange; - - return krange && addr < kend && kstart < end; -} - -static bool -drm_gpuva_range_valid(struct drm_gpuva_manager *mgr, - u64 addr, u64 range) -{ - return !drm_gpuva_check_overflow(addr, range) && - drm_gpuva_in_mm_range(mgr, addr, range) && - !drm_gpuva_in_kernel_node(mgr, addr, range); -} - -/** - * drm_gpuva_manager_init() - initialize a &drm_gpuva_manager - * @mgr: pointer to the &drm_gpuva_manager to initialize - * @name: the name of the GPU VA space - * @start_offset: the start offset of the GPU VA space - * @range: the size of the GPU VA space - * @reserve_offset: the start of the kernel reserved GPU VA area - * @reserve_range: the size of the kernel reserved GPU VA area - * @ops: &drm_gpuva_fn_ops called on &drm_gpuva_sm_map / &drm_gpuva_sm_unmap - * - * The &drm_gpuva_manager must be initialized with this function before use. - * - * Note that @mgr must be cleared to 0 before calling this function. The given - * &name is expected to be managed by the surrounding driver structures. - */ -void -drm_gpuva_manager_init(struct drm_gpuva_manager *mgr, - const char *name, - u64 start_offset, u64 range, - u64 reserve_offset, u64 reserve_range, - const struct drm_gpuva_fn_ops *ops) -{ - mgr->rb.tree = RB_ROOT_CACHED; - INIT_LIST_HEAD(&mgr->rb.list); - - drm_gpuva_check_overflow(start_offset, range); - mgr->mm_start = start_offset; - mgr->mm_range = range; - - mgr->name = name ? name : "unknown"; - mgr->ops = ops; - - memset(&mgr->kernel_alloc_node, 0, sizeof(struct drm_gpuva)); - - if (reserve_range) { - mgr->kernel_alloc_node.va.addr = reserve_offset; - mgr->kernel_alloc_node.va.range = reserve_range; - - if (likely(!drm_gpuva_check_overflow(reserve_offset, - reserve_range))) - __drm_gpuva_insert(mgr, &mgr->kernel_alloc_node); - } -} -EXPORT_SYMBOL_GPL(drm_gpuva_manager_init); - -/** - * drm_gpuva_manager_destroy() - cleanup a &drm_gpuva_manager - * @mgr: pointer to the &drm_gpuva_manager to clean up - * - * Note that it is a bug to call this function on a manager that still - * holds GPU VA mappings. - */ -void -drm_gpuva_manager_destroy(struct drm_gpuva_manager *mgr) -{ - mgr->name = NULL; - - if (mgr->kernel_alloc_node.va.range) - __drm_gpuva_remove(&mgr->kernel_alloc_node); - - WARN(!RB_EMPTY_ROOT(&mgr->rb.tree.rb_root), - "GPUVA tree is not empty, potentially leaking memory."); -} -EXPORT_SYMBOL_GPL(drm_gpuva_manager_destroy); - -static int -__drm_gpuva_insert(struct drm_gpuva_manager *mgr, - struct drm_gpuva *va) -{ - struct rb_node *node; - struct list_head *head; - - if (drm_gpuva_it_iter_first(&mgr->rb.tree, - GPUVA_START(va), - GPUVA_LAST(va))) - return -EEXIST; - - va->mgr = mgr; - - drm_gpuva_it_insert(va, &mgr->rb.tree); - - node = rb_prev(&va->rb.node); - if (node) - head = &(to_drm_gpuva(node))->rb.entry; - else - head = &mgr->rb.list; - - list_add(&va->rb.entry, head); - - return 0; -} - -/** - * drm_gpuva_insert() - insert a &drm_gpuva - * @mgr: the &drm_gpuva_manager to insert the &drm_gpuva in - * @va: the &drm_gpuva to insert - * - * Insert a &drm_gpuva with a given address and range into a - * &drm_gpuva_manager. - * - * It is safe to use this function using the safe versions of iterating the GPU - * VA space, such as drm_gpuva_for_each_va_safe() and - * drm_gpuva_for_each_va_range_safe(). - * - * Returns: 0 on success, negative error code on failure. - */ -int -drm_gpuva_insert(struct drm_gpuva_manager *mgr, - struct drm_gpuva *va) -{ - u64 addr = va->va.addr; - u64 range = va->va.range; - - if (unlikely(!drm_gpuva_range_valid(mgr, addr, range))) - return -EINVAL; - - return __drm_gpuva_insert(mgr, va); -} -EXPORT_SYMBOL_GPL(drm_gpuva_insert); - -static void -__drm_gpuva_remove(struct drm_gpuva *va) -{ - drm_gpuva_it_remove(va, &va->mgr->rb.tree); - list_del_init(&va->rb.entry); -} - -/** - * drm_gpuva_remove() - remove a &drm_gpuva - * @va: the &drm_gpuva to remove - * - * This removes the given &va from the underlaying tree. - * - * It is safe to use this function using the safe versions of iterating the GPU - * VA space, such as drm_gpuva_for_each_va_safe() and - * drm_gpuva_for_each_va_range_safe(). - */ -void -drm_gpuva_remove(struct drm_gpuva *va) -{ - struct drm_gpuva_manager *mgr = va->mgr; - - if (unlikely(va == &mgr->kernel_alloc_node)) { - WARN(1, "Can't destroy kernel reserved node.\n"); - return; - } - - __drm_gpuva_remove(va); -} -EXPORT_SYMBOL_GPL(drm_gpuva_remove); - -/** - * drm_gpuva_link() - link a &drm_gpuva - * @va: the &drm_gpuva to link - * - * This adds the given &va to the GPU VA list of the &drm_gem_object it is - * associated with. - * - * This function expects the caller to protect the GEM's GPUVA list against - * concurrent access using the GEMs dma_resv lock. - */ -void -drm_gpuva_link(struct drm_gpuva *va) -{ - struct drm_gem_object *obj = va->gem.obj; - - if (unlikely(!obj)) - return; - - drm_gem_gpuva_assert_lock_held(obj); - - list_add_tail(&va->gem.entry, &obj->gpuva.list); -} -EXPORT_SYMBOL_GPL(drm_gpuva_link); - -/** - * drm_gpuva_unlink() - unlink a &drm_gpuva - * @va: the &drm_gpuva to unlink - * - * This removes the given &va from the GPU VA list of the &drm_gem_object it is - * associated with. - * - * This function expects the caller to protect the GEM's GPUVA list against - * concurrent access using the GEMs dma_resv lock. - */ -void -drm_gpuva_unlink(struct drm_gpuva *va) -{ - struct drm_gem_object *obj = va->gem.obj; - - if (unlikely(!obj)) - return; - - drm_gem_gpuva_assert_lock_held(obj); - - list_del_init(&va->gem.entry); -} -EXPORT_SYMBOL_GPL(drm_gpuva_unlink); - -/** - * drm_gpuva_find_first() - find the first &drm_gpuva in the given range - * @mgr: the &drm_gpuva_manager to search in - * @addr: the &drm_gpuvas address - * @range: the &drm_gpuvas range - * - * Returns: the first &drm_gpuva within the given range - */ -struct drm_gpuva * -drm_gpuva_find_first(struct drm_gpuva_manager *mgr, - u64 addr, u64 range) -{ - u64 last = addr + range - 1; - - return drm_gpuva_it_iter_first(&mgr->rb.tree, addr, last); -} -EXPORT_SYMBOL_GPL(drm_gpuva_find_first); - -/** - * drm_gpuva_find() - find a &drm_gpuva - * @mgr: the &drm_gpuva_manager to search in - * @addr: the &drm_gpuvas address - * @range: the &drm_gpuvas range - * - * Returns: the &drm_gpuva at a given &addr and with a given &range - */ -struct drm_gpuva * -drm_gpuva_find(struct drm_gpuva_manager *mgr, - u64 addr, u64 range) -{ - struct drm_gpuva *va; - - va = drm_gpuva_find_first(mgr, addr, range); - if (!va) - goto out; - - if (va->va.addr != addr || - va->va.range != range) - goto out; - - return va; - -out: - return NULL; -} -EXPORT_SYMBOL_GPL(drm_gpuva_find); - -/** - * drm_gpuva_find_prev() - find the &drm_gpuva before the given address - * @mgr: the &drm_gpuva_manager to search in - * @start: the given GPU VA's start address - * - * Find the adjacent &drm_gpuva before the GPU VA with given &start address. - * - * Note that if there is any free space between the GPU VA mappings no mapping - * is returned. - * - * Returns: a pointer to the found &drm_gpuva or NULL if none was found - */ -struct drm_gpuva * -drm_gpuva_find_prev(struct drm_gpuva_manager *mgr, u64 start) -{ - if (!drm_gpuva_range_valid(mgr, start - 1, 1)) - return NULL; - - return drm_gpuva_it_iter_first(&mgr->rb.tree, start - 1, start); -} -EXPORT_SYMBOL_GPL(drm_gpuva_find_prev); - -/** - * drm_gpuva_find_next() - find the &drm_gpuva after the given address - * @mgr: the &drm_gpuva_manager to search in - * @end: the given GPU VA's end address - * - * Find the adjacent &drm_gpuva after the GPU VA with given &end address. - * - * Note that if there is any free space between the GPU VA mappings no mapping - * is returned. - * - * Returns: a pointer to the found &drm_gpuva or NULL if none was found - */ -struct drm_gpuva * -drm_gpuva_find_next(struct drm_gpuva_manager *mgr, u64 end) -{ - if (!drm_gpuva_range_valid(mgr, end, 1)) - return NULL; - - return drm_gpuva_it_iter_first(&mgr->rb.tree, end, end + 1); -} -EXPORT_SYMBOL_GPL(drm_gpuva_find_next); - -/** - * drm_gpuva_interval_empty() - indicate whether a given interval of the VA space - * is empty - * @mgr: the &drm_gpuva_manager to check the range for - * @addr: the start address of the range - * @range: the range of the interval - * - * Returns: true if the interval is empty, false otherwise - */ -bool -drm_gpuva_interval_empty(struct drm_gpuva_manager *mgr, u64 addr, u64 range) -{ - return !drm_gpuva_find_first(mgr, addr, range); -} -EXPORT_SYMBOL_GPL(drm_gpuva_interval_empty); - -/** - * drm_gpuva_map() - helper to insert a &drm_gpuva according to a - * &drm_gpuva_op_map - * @mgr: the &drm_gpuva_manager - * @va: the &drm_gpuva to insert - * @op: the &drm_gpuva_op_map to initialize @va with - * - * Initializes the @va from the @op and inserts it into the given @mgr. - */ -void -drm_gpuva_map(struct drm_gpuva_manager *mgr, - struct drm_gpuva *va, - struct drm_gpuva_op_map *op) -{ - drm_gpuva_init_from_op(va, op); - drm_gpuva_insert(mgr, va); -} -EXPORT_SYMBOL_GPL(drm_gpuva_map); - -/** - * drm_gpuva_remap() - helper to remap a &drm_gpuva according to a - * &drm_gpuva_op_remap - * @prev: the &drm_gpuva to remap when keeping the start of a mapping - * @next: the &drm_gpuva to remap when keeping the end of a mapping - * @op: the &drm_gpuva_op_remap to initialize @prev and @next with - * - * Removes the currently mapped &drm_gpuva and remaps it using @prev and/or - * @next. - */ -void -drm_gpuva_remap(struct drm_gpuva *prev, - struct drm_gpuva *next, - struct drm_gpuva_op_remap *op) -{ - struct drm_gpuva *curr = op->unmap->va; - struct drm_gpuva_manager *mgr = curr->mgr; - - drm_gpuva_remove(curr); - - if (op->prev) { - drm_gpuva_init_from_op(prev, op->prev); - drm_gpuva_insert(mgr, prev); - } - - if (op->next) { - drm_gpuva_init_from_op(next, op->next); - drm_gpuva_insert(mgr, next); - } -} -EXPORT_SYMBOL_GPL(drm_gpuva_remap); - -/** - * drm_gpuva_unmap() - helper to remove a &drm_gpuva according to a - * &drm_gpuva_op_unmap - * @op: the &drm_gpuva_op_unmap specifying the &drm_gpuva to remove - * - * Removes the &drm_gpuva associated with the &drm_gpuva_op_unmap. - */ -void -drm_gpuva_unmap(struct drm_gpuva_op_unmap *op) -{ - drm_gpuva_remove(op->va); -} -EXPORT_SYMBOL_GPL(drm_gpuva_unmap); - -static int -op_map_cb(const struct drm_gpuva_fn_ops *fn, void *priv, - u64 addr, u64 range, - struct drm_gem_object *obj, u64 offset) -{ - struct drm_gpuva_op op = {}; - - op.op = DRM_GPUVA_OP_MAP; - op.map.va.addr = addr; - op.map.va.range = range; - op.map.gem.obj = obj; - op.map.gem.offset = offset; - - return fn->sm_step_map(&op, priv); -} - -static int -op_remap_cb(const struct drm_gpuva_fn_ops *fn, void *priv, - struct drm_gpuva_op_map *prev, - struct drm_gpuva_op_map *next, - struct drm_gpuva_op_unmap *unmap) -{ - struct drm_gpuva_op op = {}; - struct drm_gpuva_op_remap *r; - - op.op = DRM_GPUVA_OP_REMAP; - r = &op.remap; - r->prev = prev; - r->next = next; - r->unmap = unmap; - - return fn->sm_step_remap(&op, priv); -} - -static int -op_unmap_cb(const struct drm_gpuva_fn_ops *fn, void *priv, - struct drm_gpuva *va, bool merge) -{ - struct drm_gpuva_op op = {}; - - op.op = DRM_GPUVA_OP_UNMAP; - op.unmap.va = va; - op.unmap.keep = merge; - - return fn->sm_step_unmap(&op, priv); -} - -static int -__drm_gpuva_sm_map(struct drm_gpuva_manager *mgr, - const struct drm_gpuva_fn_ops *ops, void *priv, - u64 req_addr, u64 req_range, - struct drm_gem_object *req_obj, u64 req_offset) -{ - struct drm_gpuva *va, *next; - u64 req_end = req_addr + req_range; - int ret; - - if (unlikely(!drm_gpuva_range_valid(mgr, req_addr, req_range))) - return -EINVAL; - - drm_gpuva_for_each_va_range_safe(va, next, mgr, req_addr, req_end) { - struct drm_gem_object *obj = va->gem.obj; - u64 offset = va->gem.offset; - u64 addr = va->va.addr; - u64 range = va->va.range; - u64 end = addr + range; - bool merge = !!va->gem.obj; - - if (addr == req_addr) { - merge &= obj == req_obj && - offset == req_offset; - - if (end == req_end) { - ret = op_unmap_cb(ops, priv, va, merge); - if (ret) - return ret; - break; - } - - if (end < req_end) { - ret = op_unmap_cb(ops, priv, va, merge); - if (ret) - return ret; - continue; - } - - if (end > req_end) { - struct drm_gpuva_op_map n = { - .va.addr = req_end, - .va.range = range - req_range, - .gem.obj = obj, - .gem.offset = offset + req_range, - }; - struct drm_gpuva_op_unmap u = { - .va = va, - .keep = merge, - }; - - ret = op_remap_cb(ops, priv, NULL, &n, &u); - if (ret) - return ret; - break; - } - } else if (addr < req_addr) { - u64 ls_range = req_addr - addr; - struct drm_gpuva_op_map p = { - .va.addr = addr, - .va.range = ls_range, - .gem.obj = obj, - .gem.offset = offset, - }; - struct drm_gpuva_op_unmap u = { .va = va }; - - merge &= obj == req_obj && - offset + ls_range == req_offset; - u.keep = merge; - - if (end == req_end) { - ret = op_remap_cb(ops, priv, &p, NULL, &u); - if (ret) - return ret; - break; - } - - if (end < req_end) { - ret = op_remap_cb(ops, priv, &p, NULL, &u); - if (ret) - return ret; - continue; - } - - if (end > req_end) { - struct drm_gpuva_op_map n = { - .va.addr = req_end, - .va.range = end - req_end, - .gem.obj = obj, - .gem.offset = offset + ls_range + - req_range, - }; - - ret = op_remap_cb(ops, priv, &p, &n, &u); - if (ret) - return ret; - break; - } - } else if (addr > req_addr) { - merge &= obj == req_obj && - offset == req_offset + - (addr - req_addr); - - if (end == req_end) { - ret = op_unmap_cb(ops, priv, va, merge); - if (ret) - return ret; - break; - } - - if (end < req_end) { - ret = op_unmap_cb(ops, priv, va, merge); - if (ret) - return ret; - continue; - } - - if (end > req_end) { - struct drm_gpuva_op_map n = { - .va.addr = req_end, - .va.range = end - req_end, - .gem.obj = obj, - .gem.offset = offset + req_end - addr, - }; - struct drm_gpuva_op_unmap u = { - .va = va, - .keep = merge, - }; - - ret = op_remap_cb(ops, priv, NULL, &n, &u); - if (ret) - return ret; - break; - } - } - } - - return op_map_cb(ops, priv, - req_addr, req_range, - req_obj, req_offset); -} - -static int -__drm_gpuva_sm_unmap(struct drm_gpuva_manager *mgr, - const struct drm_gpuva_fn_ops *ops, void *priv, - u64 req_addr, u64 req_range) -{ - struct drm_gpuva *va, *next; - u64 req_end = req_addr + req_range; - int ret; - - if (unlikely(!drm_gpuva_range_valid(mgr, req_addr, req_range))) - return -EINVAL; - - drm_gpuva_for_each_va_range_safe(va, next, mgr, req_addr, req_end) { - struct drm_gpuva_op_map prev = {}, next = {}; - bool prev_split = false, next_split = false; - struct drm_gem_object *obj = va->gem.obj; - u64 offset = va->gem.offset; - u64 addr = va->va.addr; - u64 range = va->va.range; - u64 end = addr + range; - - if (addr < req_addr) { - prev.va.addr = addr; - prev.va.range = req_addr - addr; - prev.gem.obj = obj; - prev.gem.offset = offset; - - prev_split = true; - } - - if (end > req_end) { - next.va.addr = req_end; - next.va.range = end - req_end; - next.gem.obj = obj; - next.gem.offset = offset + (req_end - addr); - - next_split = true; - } - - if (prev_split || next_split) { - struct drm_gpuva_op_unmap unmap = { .va = va }; - - ret = op_remap_cb(ops, priv, - prev_split ? &prev : NULL, - next_split ? &next : NULL, - &unmap); - if (ret) - return ret; - } else { - ret = op_unmap_cb(ops, priv, va, false); - if (ret) - return ret; - } - } - - return 0; -} - -/** - * drm_gpuva_sm_map() - creates the &drm_gpuva_op split/merge steps - * @mgr: the &drm_gpuva_manager representing the GPU VA space - * @req_addr: the start address of the new mapping - * @req_range: the range of the new mapping - * @req_obj: the &drm_gem_object to map - * @req_offset: the offset within the &drm_gem_object - * @priv: pointer to a driver private data structure - * - * This function iterates the given range of the GPU VA space. It utilizes the - * &drm_gpuva_fn_ops to call back into the driver providing the split and merge - * steps. - * - * Drivers may use these callbacks to update the GPU VA space right away within - * the callback. In case the driver decides to copy and store the operations for - * later processing neither this function nor &drm_gpuva_sm_unmap is allowed to - * be called before the &drm_gpuva_manager's view of the GPU VA space was - * updated with the previous set of operations. To update the - * &drm_gpuva_manager's view of the GPU VA space drm_gpuva_insert(), - * drm_gpuva_destroy_locked() and/or drm_gpuva_destroy_unlocked() should be - * used. - * - * A sequence of callbacks can contain map, unmap and remap operations, but - * the sequence of callbacks might also be empty if no operation is required, - * e.g. if the requested mapping already exists in the exact same way. - * - * There can be an arbitrary amount of unmap operations, a maximum of two remap - * operations and a single map operation. The latter one represents the original - * map operation requested by the caller. - * - * Returns: 0 on success or a negative error code - */ -int -drm_gpuva_sm_map(struct drm_gpuva_manager *mgr, void *priv, - u64 req_addr, u64 req_range, - struct drm_gem_object *req_obj, u64 req_offset) -{ - const struct drm_gpuva_fn_ops *ops = mgr->ops; - - if (unlikely(!(ops && ops->sm_step_map && - ops->sm_step_remap && - ops->sm_step_unmap))) - return -EINVAL; - - return __drm_gpuva_sm_map(mgr, ops, priv, - req_addr, req_range, - req_obj, req_offset); -} -EXPORT_SYMBOL_GPL(drm_gpuva_sm_map); - -/** - * drm_gpuva_sm_unmap() - creates the &drm_gpuva_ops to split on unmap - * @mgr: the &drm_gpuva_manager representing the GPU VA space - * @priv: pointer to a driver private data structure - * @req_addr: the start address of the range to unmap - * @req_range: the range of the mappings to unmap - * - * This function iterates the given range of the GPU VA space. It utilizes the - * &drm_gpuva_fn_ops to call back into the driver providing the operations to - * unmap and, if required, split existent mappings. - * - * Drivers may use these callbacks to update the GPU VA space right away within - * the callback. In case the driver decides to copy and store the operations for - * later processing neither this function nor &drm_gpuva_sm_map is allowed to be - * called before the &drm_gpuva_manager's view of the GPU VA space was updated - * with the previous set of operations. To update the &drm_gpuva_manager's view - * of the GPU VA space drm_gpuva_insert(), drm_gpuva_destroy_locked() and/or - * drm_gpuva_destroy_unlocked() should be used. - * - * A sequence of callbacks can contain unmap and remap operations, depending on - * whether there are actual overlapping mappings to split. - * - * There can be an arbitrary amount of unmap operations and a maximum of two - * remap operations. - * - * Returns: 0 on success or a negative error code - */ -int -drm_gpuva_sm_unmap(struct drm_gpuva_manager *mgr, void *priv, - u64 req_addr, u64 req_range) -{ - const struct drm_gpuva_fn_ops *ops = mgr->ops; - - if (unlikely(!(ops && ops->sm_step_remap && - ops->sm_step_unmap))) - return -EINVAL; - - return __drm_gpuva_sm_unmap(mgr, ops, priv, - req_addr, req_range); -} -EXPORT_SYMBOL_GPL(drm_gpuva_sm_unmap); - -static struct drm_gpuva_op * -gpuva_op_alloc(struct drm_gpuva_manager *mgr) -{ - const struct drm_gpuva_fn_ops *fn = mgr->ops; - struct drm_gpuva_op *op; - - if (fn && fn->op_alloc) - op = fn->op_alloc(); - else - op = kzalloc(sizeof(*op), GFP_KERNEL); - - if (unlikely(!op)) - return NULL; - - return op; -} - -static void -gpuva_op_free(struct drm_gpuva_manager *mgr, - struct drm_gpuva_op *op) -{ - const struct drm_gpuva_fn_ops *fn = mgr->ops; - - if (fn && fn->op_free) - fn->op_free(op); - else - kfree(op); -} - -static int -drm_gpuva_sm_step(struct drm_gpuva_op *__op, - void *priv) -{ - struct { - struct drm_gpuva_manager *mgr; - struct drm_gpuva_ops *ops; - } *args = priv; - struct drm_gpuva_manager *mgr = args->mgr; - struct drm_gpuva_ops *ops = args->ops; - struct drm_gpuva_op *op; - - op = gpuva_op_alloc(mgr); - if (unlikely(!op)) - goto err; - - memcpy(op, __op, sizeof(*op)); - - if (op->op == DRM_GPUVA_OP_REMAP) { - struct drm_gpuva_op_remap *__r = &__op->remap; - struct drm_gpuva_op_remap *r = &op->remap; - - r->unmap = kmemdup(__r->unmap, sizeof(*r->unmap), - GFP_KERNEL); - if (unlikely(!r->unmap)) - goto err_free_op; - - if (__r->prev) { - r->prev = kmemdup(__r->prev, sizeof(*r->prev), - GFP_KERNEL); - if (unlikely(!r->prev)) - goto err_free_unmap; - } - - if (__r->next) { - r->next = kmemdup(__r->next, sizeof(*r->next), - GFP_KERNEL); - if (unlikely(!r->next)) - goto err_free_prev; - } - } - - list_add_tail(&op->entry, &ops->list); - - return 0; - -err_free_unmap: - kfree(op->remap.unmap); -err_free_prev: - kfree(op->remap.prev); -err_free_op: - gpuva_op_free(mgr, op); -err: - return -ENOMEM; -} - -static const struct drm_gpuva_fn_ops gpuva_list_ops = { - .sm_step_map = drm_gpuva_sm_step, - .sm_step_remap = drm_gpuva_sm_step, - .sm_step_unmap = drm_gpuva_sm_step, -}; - -/** - * drm_gpuva_sm_map_ops_create() - creates the &drm_gpuva_ops to split and merge - * @mgr: the &drm_gpuva_manager representing the GPU VA space - * @req_addr: the start address of the new mapping - * @req_range: the range of the new mapping - * @req_obj: the &drm_gem_object to map - * @req_offset: the offset within the &drm_gem_object - * - * This function creates a list of operations to perform splitting and merging - * of existent mapping(s) with the newly requested one. - * - * The list can be iterated with &drm_gpuva_for_each_op and must be processed - * in the given order. It can contain map, unmap and remap operations, but it - * also can be empty if no operation is required, e.g. if the requested mapping - * already exists is the exact same way. - * - * There can be an arbitrary amount of unmap operations, a maximum of two remap - * operations and a single map operation. The latter one represents the original - * map operation requested by the caller. - * - * Note that before calling this function again with another mapping request it - * is necessary to update the &drm_gpuva_manager's view of the GPU VA space. The - * previously obtained operations must be either processed or abandoned. To - * update the &drm_gpuva_manager's view of the GPU VA space drm_gpuva_insert(), - * drm_gpuva_destroy_locked() and/or drm_gpuva_destroy_unlocked() should be - * used. - * - * After the caller finished processing the returned &drm_gpuva_ops, they must - * be freed with &drm_gpuva_ops_free. - * - * Returns: a pointer to the &drm_gpuva_ops on success, an ERR_PTR on failure - */ -struct drm_gpuva_ops * -drm_gpuva_sm_map_ops_create(struct drm_gpuva_manager *mgr, - u64 req_addr, u64 req_range, - struct drm_gem_object *req_obj, u64 req_offset) -{ - struct drm_gpuva_ops *ops; - struct { - struct drm_gpuva_manager *mgr; - struct drm_gpuva_ops *ops; - } args; - int ret; - - ops = kzalloc(sizeof(*ops), GFP_KERNEL); - if (unlikely(!ops)) - return ERR_PTR(-ENOMEM); - - INIT_LIST_HEAD(&ops->list); - - args.mgr = mgr; - args.ops = ops; - - ret = __drm_gpuva_sm_map(mgr, &gpuva_list_ops, &args, - req_addr, req_range, - req_obj, req_offset); - if (ret) - goto err_free_ops; - - return ops; - -err_free_ops: - drm_gpuva_ops_free(mgr, ops); - return ERR_PTR(ret); -} -EXPORT_SYMBOL_GPL(drm_gpuva_sm_map_ops_create); - -/** - * drm_gpuva_sm_unmap_ops_create() - creates the &drm_gpuva_ops to split on - * unmap - * @mgr: the &drm_gpuva_manager representing the GPU VA space - * @req_addr: the start address of the range to unmap - * @req_range: the range of the mappings to unmap - * - * This function creates a list of operations to perform unmapping and, if - * required, splitting of the mappings overlapping the unmap range. - * - * The list can be iterated with &drm_gpuva_for_each_op and must be processed - * in the given order. It can contain unmap and remap operations, depending on - * whether there are actual overlapping mappings to split. - * - * There can be an arbitrary amount of unmap operations and a maximum of two - * remap operations. - * - * Note that before calling this function again with another range to unmap it - * is necessary to update the &drm_gpuva_manager's view of the GPU VA space. The - * previously obtained operations must be processed or abandoned. To update the - * &drm_gpuva_manager's view of the GPU VA space drm_gpuva_insert(), - * drm_gpuva_destroy_locked() and/or drm_gpuva_destroy_unlocked() should be - * used. - * - * After the caller finished processing the returned &drm_gpuva_ops, they must - * be freed with &drm_gpuva_ops_free. - * - * Returns: a pointer to the &drm_gpuva_ops on success, an ERR_PTR on failure - */ -struct drm_gpuva_ops * -drm_gpuva_sm_unmap_ops_create(struct drm_gpuva_manager *mgr, - u64 req_addr, u64 req_range) -{ - struct drm_gpuva_ops *ops; - struct { - struct drm_gpuva_manager *mgr; - struct drm_gpuva_ops *ops; - } args; - int ret; - - ops = kzalloc(sizeof(*ops), GFP_KERNEL); - if (unlikely(!ops)) - return ERR_PTR(-ENOMEM); - - INIT_LIST_HEAD(&ops->list); - - args.mgr = mgr; - args.ops = ops; - - ret = __drm_gpuva_sm_unmap(mgr, &gpuva_list_ops, &args, - req_addr, req_range); - if (ret) - goto err_free_ops; - - return ops; - -err_free_ops: - drm_gpuva_ops_free(mgr, ops); - return ERR_PTR(ret); -} -EXPORT_SYMBOL_GPL(drm_gpuva_sm_unmap_ops_create); - -/** - * drm_gpuva_prefetch_ops_create() - creates the &drm_gpuva_ops to prefetch - * @mgr: the &drm_gpuva_manager representing the GPU VA space - * @addr: the start address of the range to prefetch - * @range: the range of the mappings to prefetch - * - * This function creates a list of operations to perform prefetching. - * - * The list can be iterated with &drm_gpuva_for_each_op and must be processed - * in the given order. It can contain prefetch operations. - * - * There can be an arbitrary amount of prefetch operations. - * - * After the caller finished processing the returned &drm_gpuva_ops, they must - * be freed with &drm_gpuva_ops_free. - * - * Returns: a pointer to the &drm_gpuva_ops on success, an ERR_PTR on failure - */ -struct drm_gpuva_ops * -drm_gpuva_prefetch_ops_create(struct drm_gpuva_manager *mgr, - u64 addr, u64 range) -{ - struct drm_gpuva_ops *ops; - struct drm_gpuva_op *op; - struct drm_gpuva *va; - u64 end = addr + range; - int ret; - - ops = kzalloc(sizeof(*ops), GFP_KERNEL); - if (!ops) - return ERR_PTR(-ENOMEM); - - INIT_LIST_HEAD(&ops->list); - - drm_gpuva_for_each_va_range(va, mgr, addr, end) { - op = gpuva_op_alloc(mgr); - if (!op) { - ret = -ENOMEM; - goto err_free_ops; - } - - op->op = DRM_GPUVA_OP_PREFETCH; - op->prefetch.va = va; - list_add_tail(&op->entry, &ops->list); - } - - return ops; - -err_free_ops: - drm_gpuva_ops_free(mgr, ops); - return ERR_PTR(ret); -} -EXPORT_SYMBOL_GPL(drm_gpuva_prefetch_ops_create); - -/** - * drm_gpuva_gem_unmap_ops_create() - creates the &drm_gpuva_ops to unmap a GEM - * @mgr: the &drm_gpuva_manager representing the GPU VA space - * @obj: the &drm_gem_object to unmap - * - * This function creates a list of operations to perform unmapping for every - * GPUVA attached to a GEM. - * - * The list can be iterated with &drm_gpuva_for_each_op and consists out of an - * arbitrary amount of unmap operations. - * - * After the caller finished processing the returned &drm_gpuva_ops, they must - * be freed with &drm_gpuva_ops_free. - * - * It is the callers responsibility to protect the GEMs GPUVA list against - * concurrent access using the GEMs dma_resv lock. - * - * Returns: a pointer to the &drm_gpuva_ops on success, an ERR_PTR on failure - */ -struct drm_gpuva_ops * -drm_gpuva_gem_unmap_ops_create(struct drm_gpuva_manager *mgr, - struct drm_gem_object *obj) -{ - struct drm_gpuva_ops *ops; - struct drm_gpuva_op *op; - struct drm_gpuva *va; - int ret; - - drm_gem_gpuva_assert_lock_held(obj); - - ops = kzalloc(sizeof(*ops), GFP_KERNEL); - if (!ops) - return ERR_PTR(-ENOMEM); - - INIT_LIST_HEAD(&ops->list); - - drm_gem_for_each_gpuva(va, obj) { - op = gpuva_op_alloc(mgr); - if (!op) { - ret = -ENOMEM; - goto err_free_ops; - } - - op->op = DRM_GPUVA_OP_UNMAP; - op->unmap.va = va; - list_add_tail(&op->entry, &ops->list); - } - - return ops; - -err_free_ops: - drm_gpuva_ops_free(mgr, ops); - return ERR_PTR(ret); -} -EXPORT_SYMBOL_GPL(drm_gpuva_gem_unmap_ops_create); - -/** - * drm_gpuva_ops_free() - free the given &drm_gpuva_ops - * @mgr: the &drm_gpuva_manager the ops were created for - * @ops: the &drm_gpuva_ops to free - * - * Frees the given &drm_gpuva_ops structure including all the ops associated - * with it. - */ -void -drm_gpuva_ops_free(struct drm_gpuva_manager *mgr, - struct drm_gpuva_ops *ops) -{ - struct drm_gpuva_op *op, *next; - - drm_gpuva_for_each_op_safe(op, next, ops) { - list_del(&op->entry); - - if (op->op == DRM_GPUVA_OP_REMAP) { - kfree(op->remap.prev); - kfree(op->remap.next); - kfree(op->remap.unmap); - } - - gpuva_op_free(mgr, op); - } - - kfree(ops); -} -EXPORT_SYMBOL_GPL(drm_gpuva_ops_free); diff --git a/drivers/gpu/drm/drm_gpuvm.c b/drivers/gpu/drm/drm_gpuvm.c new file mode 100644 index 000000000000..89dcc6c34de4 --- /dev/null +++ b/drivers/gpu/drm/drm_gpuvm.c @@ -0,0 +1,1723 @@ +// SPDX-License-Identifier: GPL-2.0-only +/* + * Copyright (c) 2022 Red Hat. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + * + * Authors: + * Danilo Krummrich + * + */ + +#include + +#include +#include + +/** + * DOC: Overview + * + * The DRM GPU VA Manager, represented by struct drm_gpuvm keeps track of a + * GPU's virtual address (VA) space and manages the corresponding virtual + * mappings represented by &drm_gpuva objects. It also keeps track of the + * mapping's backing &drm_gem_object buffers. + * + * &drm_gem_object buffers maintain a list of &drm_gpuva objects representing + * all existent GPU VA mappings using this &drm_gem_object as backing buffer. + * + * GPU VAs can be flagged as sparse, such that drivers may use GPU VAs to also + * keep track of sparse PTEs in order to support Vulkan 'Sparse Resources'. + * + * The GPU VA manager internally uses a rb-tree to manage the + * &drm_gpuva mappings within a GPU's virtual address space. + * + * The &drm_gpuvm structure contains a special &drm_gpuva representing the + * portion of VA space reserved by the kernel. This node is initialized together + * with the GPU VA manager instance and removed when the GPU VA manager is + * destroyed. + * + * In a typical application drivers would embed struct drm_gpuvm and + * struct drm_gpuva within their own driver specific structures, there won't be + * any memory allocations of its own nor memory allocations of &drm_gpuva + * entries. + * + * The data structures needed to store &drm_gpuvas within the &drm_gpuvm are + * contained within struct drm_gpuva already. Hence, for inserting &drm_gpuva + * entries from within dma-fence signalling critical sections it is enough to + * pre-allocate the &drm_gpuva structures. + */ + +/** + * DOC: Split and Merge + * + * Besides its capability to manage and represent a GPU VA space, the + * GPU VA manager also provides functions to let the &drm_gpuvm calculate a + * sequence of operations to satisfy a given map or unmap request. + * + * Therefore the DRM GPU VA manager provides an algorithm implementing splitting + * and merging of existent GPU VA mappings with the ones that are requested to + * be mapped or unmapped. This feature is required by the Vulkan API to + * implement Vulkan 'Sparse Memory Bindings' - drivers UAPIs often refer to this + * as VM BIND. + * + * Drivers can call drm_gpuvm_sm_map() to receive a sequence of callbacks + * containing map, unmap and remap operations for a given newly requested + * mapping. The sequence of callbacks represents the set of operations to + * execute in order to integrate the new mapping cleanly into the current state + * of the GPU VA space. + * + * Depending on how the new GPU VA mapping intersects with the existent mappings + * of the GPU VA space the &drm_gpuvm_ops callbacks contain an arbitrary amount + * of unmap operations, a maximum of two remap operations and a single map + * operation. The caller might receive no callback at all if no operation is + * required, e.g. if the requested mapping already exists in the exact same way. + * + * The single map operation represents the original map operation requested by + * the caller. + * + * &drm_gpuva_op_unmap contains a 'keep' field, which indicates whether the + * &drm_gpuva to unmap is physically contiguous with the original mapping + * request. Optionally, if 'keep' is set, drivers may keep the actual page table + * entries for this &drm_gpuva, adding the missing page table entries only and + * update the &drm_gpuvm's view of things accordingly. + * + * Drivers may do the same optimization, namely delta page table updates, also + * for remap operations. This is possible since &drm_gpuva_op_remap consists of + * one unmap operation and one or two map operations, such that drivers can + * derive the page table update delta accordingly. + * + * Note that there can't be more than two existent mappings to split up, one at + * the beginning and one at the end of the new mapping, hence there is a + * maximum of two remap operations. + * + * Analogous to drm_gpuvm_sm_map() drm_gpuvm_sm_unmap() uses &drm_gpuvm_ops to + * call back into the driver in order to unmap a range of GPU VA space. The + * logic behind this function is way simpler though: For all existent mappings + * enclosed by the given range unmap operations are created. For mappings which + * are only partically located within the given range, remap operations are + * created such that those mappings are split up and re-mapped partically. + * + * As an alternative to drm_gpuvm_sm_map() and drm_gpuvm_sm_unmap(), + * drm_gpuvm_sm_map_ops_create() and drm_gpuvm_sm_unmap_ops_create() can be used + * to directly obtain an instance of struct drm_gpuva_ops containing a list of + * &drm_gpuva_op, which can be iterated with drm_gpuva_for_each_op(). This list + * contains the &drm_gpuva_ops analogous to the callbacks one would receive when + * calling drm_gpuvm_sm_map() or drm_gpuvm_sm_unmap(). While this way requires + * more memory (to allocate the &drm_gpuva_ops), it provides drivers a way to + * iterate the &drm_gpuva_op multiple times, e.g. once in a context where memory + * allocations are possible (e.g. to allocate GPU page tables) and once in the + * dma-fence signalling critical path. + * + * To update the &drm_gpuvm's view of the GPU VA space drm_gpuva_insert() and + * drm_gpuva_remove() may be used. These functions can safely be used from + * &drm_gpuvm_ops callbacks originating from drm_gpuvm_sm_map() or + * drm_gpuvm_sm_unmap(). However, it might be more convenient to use the + * provided helper functions drm_gpuva_map(), drm_gpuva_remap() and + * drm_gpuva_unmap() instead. + * + * The following diagram depicts the basic relationships of existent GPU VA + * mappings, a newly requested mapping and the resulting mappings as implemented + * by drm_gpuvm_sm_map() - it doesn't cover any arbitrary combinations of these. + * + * 1) Requested mapping is identical. Replace it, but indicate the backing PTEs + * could be kept. + * + * :: + * + * 0 a 1 + * old: |-----------| (bo_offset=n) + * + * 0 a 1 + * req: |-----------| (bo_offset=n) + * + * 0 a 1 + * new: |-----------| (bo_offset=n) + * + * + * 2) Requested mapping is identical, except for the BO offset, hence replace + * the mapping. + * + * :: + * + * 0 a 1 + * old: |-----------| (bo_offset=n) + * + * 0 a 1 + * req: |-----------| (bo_offset=m) + * + * 0 a 1 + * new: |-----------| (bo_offset=m) + * + * + * 3) Requested mapping is identical, except for the backing BO, hence replace + * the mapping. + * + * :: + * + * 0 a 1 + * old: |-----------| (bo_offset=n) + * + * 0 b 1 + * req: |-----------| (bo_offset=n) + * + * 0 b 1 + * new: |-----------| (bo_offset=n) + * + * + * 4) Existent mapping is a left aligned subset of the requested one, hence + * replace the existent one. + * + * :: + * + * 0 a 1 + * old: |-----| (bo_offset=n) + * + * 0 a 2 + * req: |-----------| (bo_offset=n) + * + * 0 a 2 + * new: |-----------| (bo_offset=n) + * + * .. note:: + * We expect to see the same result for a request with a different BO + * and/or non-contiguous BO offset. + * + * + * 5) Requested mapping's range is a left aligned subset of the existent one, + * but backed by a different BO. Hence, map the requested mapping and split + * the existent one adjusting its BO offset. + * + * :: + * + * 0 a 2 + * old: |-----------| (bo_offset=n) + * + * 0 b 1 + * req: |-----| (bo_offset=n) + * + * 0 b 1 a' 2 + * new: |-----|-----| (b.bo_offset=n, a.bo_offset=n+1) + * + * .. note:: + * We expect to see the same result for a request with a different BO + * and/or non-contiguous BO offset. + * + * + * 6) Existent mapping is a superset of the requested mapping. Split it up, but + * indicate that the backing PTEs could be kept. + * + * :: + * + * 0 a 2 + * old: |-----------| (bo_offset=n) + * + * 0 a 1 + * req: |-----| (bo_offset=n) + * + * 0 a 1 a' 2 + * new: |-----|-----| (a.bo_offset=n, a'.bo_offset=n+1) + * + * + * 7) Requested mapping's range is a right aligned subset of the existent one, + * but backed by a different BO. Hence, map the requested mapping and split + * the existent one, without adjusting the BO offset. + * + * :: + * + * 0 a 2 + * old: |-----------| (bo_offset=n) + * + * 1 b 2 + * req: |-----| (bo_offset=m) + * + * 0 a 1 b 2 + * new: |-----|-----| (a.bo_offset=n,b.bo_offset=m) + * + * + * 8) Existent mapping is a superset of the requested mapping. Split it up, but + * indicate that the backing PTEs could be kept. + * + * :: + * + * 0 a 2 + * old: |-----------| (bo_offset=n) + * + * 1 a 2 + * req: |-----| (bo_offset=n+1) + * + * 0 a' 1 a 2 + * new: |-----|-----| (a'.bo_offset=n, a.bo_offset=n+1) + * + * + * 9) Existent mapping is overlapped at the end by the requested mapping backed + * by a different BO. Hence, map the requested mapping and split up the + * existent one, without adjusting the BO offset. + * + * :: + * + * 0 a 2 + * old: |-----------| (bo_offset=n) + * + * 1 b 3 + * req: |-----------| (bo_offset=m) + * + * 0 a 1 b 3 + * new: |-----|-----------| (a.bo_offset=n,b.bo_offset=m) + * + * + * 10) Existent mapping is overlapped by the requested mapping, both having the + * same backing BO with a contiguous offset. Indicate the backing PTEs of + * the old mapping could be kept. + * + * :: + * + * 0 a 2 + * old: |-----------| (bo_offset=n) + * + * 1 a 3 + * req: |-----------| (bo_offset=n+1) + * + * 0 a' 1 a 3 + * new: |-----|-----------| (a'.bo_offset=n, a.bo_offset=n+1) + * + * + * 11) Requested mapping's range is a centered subset of the existent one + * having a different backing BO. Hence, map the requested mapping and split + * up the existent one in two mappings, adjusting the BO offset of the right + * one accordingly. + * + * :: + * + * 0 a 3 + * old: |-----------------| (bo_offset=n) + * + * 1 b 2 + * req: |-----| (bo_offset=m) + * + * 0 a 1 b 2 a' 3 + * new: |-----|-----|-----| (a.bo_offset=n,b.bo_offset=m,a'.bo_offset=n+2) + * + * + * 12) Requested mapping is a contiguous subset of the existent one. Split it + * up, but indicate that the backing PTEs could be kept. + * + * :: + * + * 0 a 3 + * old: |-----------------| (bo_offset=n) + * + * 1 a 2 + * req: |-----| (bo_offset=n+1) + * + * 0 a' 1 a 2 a'' 3 + * old: |-----|-----|-----| (a'.bo_offset=n, a.bo_offset=n+1, a''.bo_offset=n+2) + * + * + * 13) Existent mapping is a right aligned subset of the requested one, hence + * replace the existent one. + * + * :: + * + * 1 a 2 + * old: |-----| (bo_offset=n+1) + * + * 0 a 2 + * req: |-----------| (bo_offset=n) + * + * 0 a 2 + * new: |-----------| (bo_offset=n) + * + * .. note:: + * We expect to see the same result for a request with a different bo + * and/or non-contiguous bo_offset. + * + * + * 14) Existent mapping is a centered subset of the requested one, hence + * replace the existent one. + * + * :: + * + * 1 a 2 + * old: |-----| (bo_offset=n+1) + * + * 0 a 3 + * req: |----------------| (bo_offset=n) + * + * 0 a 3 + * new: |----------------| (bo_offset=n) + * + * .. note:: + * We expect to see the same result for a request with a different bo + * and/or non-contiguous bo_offset. + * + * + * 15) Existent mappings is overlapped at the beginning by the requested mapping + * backed by a different BO. Hence, map the requested mapping and split up + * the existent one, adjusting its BO offset accordingly. + * + * :: + * + * 1 a 3 + * old: |-----------| (bo_offset=n) + * + * 0 b 2 + * req: |-----------| (bo_offset=m) + * + * 0 b 2 a' 3 + * new: |-----------|-----| (b.bo_offset=m,a.bo_offset=n+2) + */ + +/** + * DOC: Locking + * + * Generally, the GPU VA manager does not take care of locking itself, it is + * the drivers responsibility to take care about locking. Drivers might want to + * protect the following operations: inserting, removing and iterating + * &drm_gpuva objects as well as generating all kinds of operations, such as + * split / merge or prefetch. + * + * The GPU VA manager also does not take care of the locking of the backing + * &drm_gem_object buffers GPU VA lists by itself; drivers are responsible to + * enforce mutual exclusion using either the GEMs dma_resv lock or alternatively + * a driver specific external lock. For the latter see also + * drm_gem_gpuva_set_lock(). + * + * However, the GPU VA manager contains lockdep checks to ensure callers of its + * API hold the corresponding lock whenever the &drm_gem_objects GPU VA list is + * accessed by functions such as drm_gpuva_link() or drm_gpuva_unlink(). + */ + +/** + * DOC: Examples + * + * This section gives two examples on how to let the DRM GPUVA Manager generate + * &drm_gpuva_op in order to satisfy a given map or unmap request and how to + * make use of them. + * + * The below code is strictly limited to illustrate the generic usage pattern. + * To maintain simplicitly, it doesn't make use of any abstractions for common + * code, different (asyncronous) stages with fence signalling critical paths, + * any other helpers or error handling in terms of freeing memory and dropping + * previously taken locks. + * + * 1) Obtain a list of &drm_gpuva_op to create a new mapping:: + * + * // Allocates a new &drm_gpuva. + * struct drm_gpuva * driver_gpuva_alloc(void); + * + * // Typically drivers would embedd the &drm_gpuvm and &drm_gpuva + * // structure in individual driver structures and lock the dma-resv with + * // drm_exec or similar helpers. + * int driver_mapping_create(struct drm_gpuvm *gpuvm, + * u64 addr, u64 range, + * struct drm_gem_object *obj, u64 offset) + * { + * struct drm_gpuva_ops *ops; + * struct drm_gpuva_op *op + * + * driver_lock_va_space(); + * ops = drm_gpuvm_sm_map_ops_create(gpuvm, addr, range, + * obj, offset); + * if (IS_ERR(ops)) + * return PTR_ERR(ops); + * + * drm_gpuva_for_each_op(op, ops) { + * struct drm_gpuva *va; + * + * switch (op->op) { + * case DRM_GPUVA_OP_MAP: + * va = driver_gpuva_alloc(); + * if (!va) + * ; // unwind previous VA space updates, + * // free memory and unlock + * + * driver_vm_map(); + * drm_gpuva_map(gpuvm, va, &op->map); + * drm_gpuva_link(va); + * + * break; + * case DRM_GPUVA_OP_REMAP: { + * struct drm_gpuva *prev = NULL, *next = NULL; + * + * va = op->remap.unmap->va; + * + * if (op->remap.prev) { + * prev = driver_gpuva_alloc(); + * if (!prev) + * ; // unwind previous VA space + * // updates, free memory and + * // unlock + * } + * + * if (op->remap.next) { + * next = driver_gpuva_alloc(); + * if (!next) + * ; // unwind previous VA space + * // updates, free memory and + * // unlock + * } + * + * driver_vm_remap(); + * drm_gpuva_remap(prev, next, &op->remap); + * + * drm_gpuva_unlink(va); + * if (prev) + * drm_gpuva_link(prev); + * if (next) + * drm_gpuva_link(next); + * + * break; + * } + * case DRM_GPUVA_OP_UNMAP: + * va = op->unmap->va; + * + * driver_vm_unmap(); + * drm_gpuva_unlink(va); + * drm_gpuva_unmap(&op->unmap); + * + * break; + * default: + * break; + * } + * } + * driver_unlock_va_space(); + * + * return 0; + * } + * + * 2) Receive a callback for each &drm_gpuva_op to create a new mapping:: + * + * struct driver_context { + * struct drm_gpuvm *gpuvm; + * struct drm_gpuva *new_va; + * struct drm_gpuva *prev_va; + * struct drm_gpuva *next_va; + * }; + * + * // ops to pass to drm_gpuvm_init() + * static const struct drm_gpuvm_ops driver_gpuvm_ops = { + * .sm_step_map = driver_gpuva_map, + * .sm_step_remap = driver_gpuva_remap, + * .sm_step_unmap = driver_gpuva_unmap, + * }; + * + * // Typically drivers would embedd the &drm_gpuvm and &drm_gpuva + * // structure in individual driver structures and lock the dma-resv with + * // drm_exec or similar helpers. + * int driver_mapping_create(struct drm_gpuvm *gpuvm, + * u64 addr, u64 range, + * struct drm_gem_object *obj, u64 offset) + * { + * struct driver_context ctx; + * struct drm_gpuva_ops *ops; + * struct drm_gpuva_op *op; + * int ret = 0; + * + * ctx.gpuvm = gpuvm; + * + * ctx.new_va = kzalloc(sizeof(*ctx.new_va), GFP_KERNEL); + * ctx.prev_va = kzalloc(sizeof(*ctx.prev_va), GFP_KERNEL); + * ctx.next_va = kzalloc(sizeof(*ctx.next_va), GFP_KERNEL); + * if (!ctx.new_va || !ctx.prev_va || !ctx.next_va) { + * ret = -ENOMEM; + * goto out; + * } + * + * driver_lock_va_space(); + * ret = drm_gpuvm_sm_map(gpuvm, &ctx, addr, range, obj, offset); + * driver_unlock_va_space(); + * + * out: + * kfree(ctx.new_va); + * kfree(ctx.prev_va); + * kfree(ctx.next_va); + * return ret; + * } + * + * int driver_gpuva_map(struct drm_gpuva_op *op, void *__ctx) + * { + * struct driver_context *ctx = __ctx; + * + * drm_gpuva_map(ctx->vm, ctx->new_va, &op->map); + * + * drm_gpuva_link(ctx->new_va); + * + * // prevent the new GPUVA from being freed in + * // driver_mapping_create() + * ctx->new_va = NULL; + * + * return 0; + * } + * + * int driver_gpuva_remap(struct drm_gpuva_op *op, void *__ctx) + * { + * struct driver_context *ctx = __ctx; + * + * drm_gpuva_remap(ctx->prev_va, ctx->next_va, &op->remap); + * + * drm_gpuva_unlink(op->remap.unmap->va); + * kfree(op->remap.unmap->va); + * + * if (op->remap.prev) { + * drm_gpuva_link(ctx->prev_va); + * ctx->prev_va = NULL; + * } + * + * if (op->remap.next) { + * drm_gpuva_link(ctx->next_va); + * ctx->next_va = NULL; + * } + * + * return 0; + * } + * + * int driver_gpuva_unmap(struct drm_gpuva_op *op, void *__ctx) + * { + * drm_gpuva_unlink(op->unmap.va); + * drm_gpuva_unmap(&op->unmap); + * kfree(op->unmap.va); + * + * return 0; + * } + */ + +#define to_drm_gpuva(__node) container_of((__node), struct drm_gpuva, rb.node) + +#define GPUVA_START(node) ((node)->va.addr) +#define GPUVA_LAST(node) ((node)->va.addr + (node)->va.range - 1) + +/* We do not actually use drm_gpuva_it_next(), tell the compiler to not complain + * about this. + */ +INTERVAL_TREE_DEFINE(struct drm_gpuva, rb.node, u64, rb.__subtree_last, + GPUVA_START, GPUVA_LAST, static __maybe_unused, + drm_gpuva_it) + +static int __drm_gpuva_insert(struct drm_gpuvm *gpuvm, + struct drm_gpuva *va); +static void __drm_gpuva_remove(struct drm_gpuva *va); + +static bool +drm_gpuvm_check_overflow(u64 addr, u64 range) +{ + u64 end; + + return WARN(check_add_overflow(addr, range, &end), + "GPUVA address limited to %zu bytes.\n", sizeof(end)); +} + +static bool +drm_gpuvm_in_mm_range(struct drm_gpuvm *gpuvm, u64 addr, u64 range) +{ + u64 end = addr + range; + u64 mm_start = gpuvm->mm_start; + u64 mm_end = mm_start + gpuvm->mm_range; + + return addr >= mm_start && end <= mm_end; +} + +static bool +drm_gpuvm_in_kernel_node(struct drm_gpuvm *gpuvm, u64 addr, u64 range) +{ + u64 end = addr + range; + u64 kstart = gpuvm->kernel_alloc_node.va.addr; + u64 krange = gpuvm->kernel_alloc_node.va.range; + u64 kend = kstart + krange; + + return krange && addr < kend && kstart < end; +} + +static bool +drm_gpuvm_range_valid(struct drm_gpuvm *gpuvm, + u64 addr, u64 range) +{ + return !drm_gpuvm_check_overflow(addr, range) && + drm_gpuvm_in_mm_range(gpuvm, addr, range) && + !drm_gpuvm_in_kernel_node(gpuvm, addr, range); +} + +/** + * drm_gpuvm_init() - initialize a &drm_gpuvm + * @gpuvm: pointer to the &drm_gpuvm to initialize + * @name: the name of the GPU VA space + * @start_offset: the start offset of the GPU VA space + * @range: the size of the GPU VA space + * @reserve_offset: the start of the kernel reserved GPU VA area + * @reserve_range: the size of the kernel reserved GPU VA area + * @ops: &drm_gpuvm_ops called on &drm_gpuvm_sm_map / &drm_gpuvm_sm_unmap + * + * The &drm_gpuvm must be initialized with this function before use. + * + * Note that @gpuvm must be cleared to 0 before calling this function. The given + * &name is expected to be managed by the surrounding driver structures. + */ +void +drm_gpuvm_init(struct drm_gpuvm *gpuvm, + const char *name, + u64 start_offset, u64 range, + u64 reserve_offset, u64 reserve_range, + const struct drm_gpuvm_ops *ops) +{ + gpuvm->rb.tree = RB_ROOT_CACHED; + INIT_LIST_HEAD(&gpuvm->rb.list); + + drm_gpuvm_check_overflow(start_offset, range); + gpuvm->mm_start = start_offset; + gpuvm->mm_range = range; + + gpuvm->name = name ? name : "unknown"; + gpuvm->ops = ops; + + memset(&gpuvm->kernel_alloc_node, 0, sizeof(struct drm_gpuva)); + + if (reserve_range) { + gpuvm->kernel_alloc_node.va.addr = reserve_offset; + gpuvm->kernel_alloc_node.va.range = reserve_range; + + if (likely(!drm_gpuvm_check_overflow(reserve_offset, + reserve_range))) + __drm_gpuva_insert(gpuvm, &gpuvm->kernel_alloc_node); + } +} +EXPORT_SYMBOL_GPL(drm_gpuvm_init); + +/** + * drm_gpuvm_destroy() - cleanup a &drm_gpuvm + * @gpuvm: pointer to the &drm_gpuvm to clean up + * + * Note that it is a bug to call this function on a manager that still + * holds GPU VA mappings. + */ +void +drm_gpuvm_destroy(struct drm_gpuvm *gpuvm) +{ + gpuvm->name = NULL; + + if (gpuvm->kernel_alloc_node.va.range) + __drm_gpuva_remove(&gpuvm->kernel_alloc_node); + + WARN(!RB_EMPTY_ROOT(&gpuvm->rb.tree.rb_root), + "GPUVA tree is not empty, potentially leaking memory."); +} +EXPORT_SYMBOL_GPL(drm_gpuvm_destroy); + +static int +__drm_gpuva_insert(struct drm_gpuvm *gpuvm, + struct drm_gpuva *va) +{ + struct rb_node *node; + struct list_head *head; + + if (drm_gpuva_it_iter_first(&gpuvm->rb.tree, + GPUVA_START(va), + GPUVA_LAST(va))) + return -EEXIST; + + va->vm = gpuvm; + + drm_gpuva_it_insert(va, &gpuvm->rb.tree); + + node = rb_prev(&va->rb.node); + if (node) + head = &(to_drm_gpuva(node))->rb.entry; + else + head = &gpuvm->rb.list; + + list_add(&va->rb.entry, head); + + return 0; +} + +/** + * drm_gpuva_insert() - insert a &drm_gpuva + * @gpuvm: the &drm_gpuvm to insert the &drm_gpuva in + * @va: the &drm_gpuva to insert + * + * Insert a &drm_gpuva with a given address and range into a + * &drm_gpuvm. + * + * It is safe to use this function using the safe versions of iterating the GPU + * VA space, such as drm_gpuvm_for_each_va_safe() and + * drm_gpuvm_for_each_va_range_safe(). + * + * Returns: 0 on success, negative error code on failure. + */ +int +drm_gpuva_insert(struct drm_gpuvm *gpuvm, + struct drm_gpuva *va) +{ + u64 addr = va->va.addr; + u64 range = va->va.range; + + if (unlikely(!drm_gpuvm_range_valid(gpuvm, addr, range))) + return -EINVAL; + + return __drm_gpuva_insert(gpuvm, va); +} +EXPORT_SYMBOL_GPL(drm_gpuva_insert); + +static void +__drm_gpuva_remove(struct drm_gpuva *va) +{ + drm_gpuva_it_remove(va, &va->vm->rb.tree); + list_del_init(&va->rb.entry); +} + +/** + * drm_gpuva_remove() - remove a &drm_gpuva + * @va: the &drm_gpuva to remove + * + * This removes the given &va from the underlaying tree. + * + * It is safe to use this function using the safe versions of iterating the GPU + * VA space, such as drm_gpuvm_for_each_va_safe() and + * drm_gpuvm_for_each_va_range_safe(). + */ +void +drm_gpuva_remove(struct drm_gpuva *va) +{ + struct drm_gpuvm *gpuvm = va->vm; + + if (unlikely(va == &gpuvm->kernel_alloc_node)) { + WARN(1, "Can't destroy kernel reserved node.\n"); + return; + } + + __drm_gpuva_remove(va); +} +EXPORT_SYMBOL_GPL(drm_gpuva_remove); + +/** + * drm_gpuva_link() - link a &drm_gpuva + * @va: the &drm_gpuva to link + * + * This adds the given &va to the GPU VA list of the &drm_gem_object it is + * associated with. + * + * This function expects the caller to protect the GEM's GPUVA list against + * concurrent access using the GEMs dma_resv lock. + */ +void +drm_gpuva_link(struct drm_gpuva *va) +{ + struct drm_gem_object *obj = va->gem.obj; + + if (unlikely(!obj)) + return; + + drm_gem_gpuva_assert_lock_held(obj); + + list_add_tail(&va->gem.entry, &obj->gpuva.list); +} +EXPORT_SYMBOL_GPL(drm_gpuva_link); + +/** + * drm_gpuva_unlink() - unlink a &drm_gpuva + * @va: the &drm_gpuva to unlink + * + * This removes the given &va from the GPU VA list of the &drm_gem_object it is + * associated with. + * + * This function expects the caller to protect the GEM's GPUVA list against + * concurrent access using the GEMs dma_resv lock. + */ +void +drm_gpuva_unlink(struct drm_gpuva *va) +{ + struct drm_gem_object *obj = va->gem.obj; + + if (unlikely(!obj)) + return; + + drm_gem_gpuva_assert_lock_held(obj); + + list_del_init(&va->gem.entry); +} +EXPORT_SYMBOL_GPL(drm_gpuva_unlink); + +/** + * drm_gpuva_find_first() - find the first &drm_gpuva in the given range + * @gpuvm: the &drm_gpuvm to search in + * @addr: the &drm_gpuvas address + * @range: the &drm_gpuvas range + * + * Returns: the first &drm_gpuva within the given range + */ +struct drm_gpuva * +drm_gpuva_find_first(struct drm_gpuvm *gpuvm, + u64 addr, u64 range) +{ + u64 last = addr + range - 1; + + return drm_gpuva_it_iter_first(&gpuvm->rb.tree, addr, last); +} +EXPORT_SYMBOL_GPL(drm_gpuva_find_first); + +/** + * drm_gpuva_find() - find a &drm_gpuva + * @gpuvm: the &drm_gpuvm to search in + * @addr: the &drm_gpuvas address + * @range: the &drm_gpuvas range + * + * Returns: the &drm_gpuva at a given &addr and with a given &range + */ +struct drm_gpuva * +drm_gpuva_find(struct drm_gpuvm *gpuvm, + u64 addr, u64 range) +{ + struct drm_gpuva *va; + + va = drm_gpuva_find_first(gpuvm, addr, range); + if (!va) + goto out; + + if (va->va.addr != addr || + va->va.range != range) + goto out; + + return va; + +out: + return NULL; +} +EXPORT_SYMBOL_GPL(drm_gpuva_find); + +/** + * drm_gpuva_find_prev() - find the &drm_gpuva before the given address + * @gpuvm: the &drm_gpuvm to search in + * @start: the given GPU VA's start address + * + * Find the adjacent &drm_gpuva before the GPU VA with given &start address. + * + * Note that if there is any free space between the GPU VA mappings no mapping + * is returned. + * + * Returns: a pointer to the found &drm_gpuva or NULL if none was found + */ +struct drm_gpuva * +drm_gpuva_find_prev(struct drm_gpuvm *gpuvm, u64 start) +{ + if (!drm_gpuvm_range_valid(gpuvm, start - 1, 1)) + return NULL; + + return drm_gpuva_it_iter_first(&gpuvm->rb.tree, start - 1, start); +} +EXPORT_SYMBOL_GPL(drm_gpuva_find_prev); + +/** + * drm_gpuva_find_next() - find the &drm_gpuva after the given address + * @gpuvm: the &drm_gpuvm to search in + * @end: the given GPU VA's end address + * + * Find the adjacent &drm_gpuva after the GPU VA with given &end address. + * + * Note that if there is any free space between the GPU VA mappings no mapping + * is returned. + * + * Returns: a pointer to the found &drm_gpuva or NULL if none was found + */ +struct drm_gpuva * +drm_gpuva_find_next(struct drm_gpuvm *gpuvm, u64 end) +{ + if (!drm_gpuvm_range_valid(gpuvm, end, 1)) + return NULL; + + return drm_gpuva_it_iter_first(&gpuvm->rb.tree, end, end + 1); +} +EXPORT_SYMBOL_GPL(drm_gpuva_find_next); + +/** + * drm_gpuvm_interval_empty() - indicate whether a given interval of the VA space + * is empty + * @gpuvm: the &drm_gpuvm to check the range for + * @addr: the start address of the range + * @range: the range of the interval + * + * Returns: true if the interval is empty, false otherwise + */ +bool +drm_gpuvm_interval_empty(struct drm_gpuvm *gpuvm, u64 addr, u64 range) +{ + return !drm_gpuva_find_first(gpuvm, addr, range); +} +EXPORT_SYMBOL_GPL(drm_gpuvm_interval_empty); + +/** + * drm_gpuva_map() - helper to insert a &drm_gpuva according to a + * &drm_gpuva_op_map + * @gpuvm: the &drm_gpuvm + * @va: the &drm_gpuva to insert + * @op: the &drm_gpuva_op_map to initialize @va with + * + * Initializes the @va from the @op and inserts it into the given @gpuvm. + */ +void +drm_gpuva_map(struct drm_gpuvm *gpuvm, + struct drm_gpuva *va, + struct drm_gpuva_op_map *op) +{ + drm_gpuva_init_from_op(va, op); + drm_gpuva_insert(gpuvm, va); +} +EXPORT_SYMBOL_GPL(drm_gpuva_map); + +/** + * drm_gpuva_remap() - helper to remap a &drm_gpuva according to a + * &drm_gpuva_op_remap + * @prev: the &drm_gpuva to remap when keeping the start of a mapping + * @next: the &drm_gpuva to remap when keeping the end of a mapping + * @op: the &drm_gpuva_op_remap to initialize @prev and @next with + * + * Removes the currently mapped &drm_gpuva and remaps it using @prev and/or + * @next. + */ +void +drm_gpuva_remap(struct drm_gpuva *prev, + struct drm_gpuva *next, + struct drm_gpuva_op_remap *op) +{ + struct drm_gpuva *curr = op->unmap->va; + struct drm_gpuvm *gpuvm = curr->vm; + + drm_gpuva_remove(curr); + + if (op->prev) { + drm_gpuva_init_from_op(prev, op->prev); + drm_gpuva_insert(gpuvm, prev); + } + + if (op->next) { + drm_gpuva_init_from_op(next, op->next); + drm_gpuva_insert(gpuvm, next); + } +} +EXPORT_SYMBOL_GPL(drm_gpuva_remap); + +/** + * drm_gpuva_unmap() - helper to remove a &drm_gpuva according to a + * &drm_gpuva_op_unmap + * @op: the &drm_gpuva_op_unmap specifying the &drm_gpuva to remove + * + * Removes the &drm_gpuva associated with the &drm_gpuva_op_unmap. + */ +void +drm_gpuva_unmap(struct drm_gpuva_op_unmap *op) +{ + drm_gpuva_remove(op->va); +} +EXPORT_SYMBOL_GPL(drm_gpuva_unmap); + +static int +op_map_cb(const struct drm_gpuvm_ops *fn, void *priv, + u64 addr, u64 range, + struct drm_gem_object *obj, u64 offset) +{ + struct drm_gpuva_op op = {}; + + op.op = DRM_GPUVA_OP_MAP; + op.map.va.addr = addr; + op.map.va.range = range; + op.map.gem.obj = obj; + op.map.gem.offset = offset; + + return fn->sm_step_map(&op, priv); +} + +static int +op_remap_cb(const struct drm_gpuvm_ops *fn, void *priv, + struct drm_gpuva_op_map *prev, + struct drm_gpuva_op_map *next, + struct drm_gpuva_op_unmap *unmap) +{ + struct drm_gpuva_op op = {}; + struct drm_gpuva_op_remap *r; + + op.op = DRM_GPUVA_OP_REMAP; + r = &op.remap; + r->prev = prev; + r->next = next; + r->unmap = unmap; + + return fn->sm_step_remap(&op, priv); +} + +static int +op_unmap_cb(const struct drm_gpuvm_ops *fn, void *priv, + struct drm_gpuva *va, bool merge) +{ + struct drm_gpuva_op op = {}; + + op.op = DRM_GPUVA_OP_UNMAP; + op.unmap.va = va; + op.unmap.keep = merge; + + return fn->sm_step_unmap(&op, priv); +} + +static int +__drm_gpuvm_sm_map(struct drm_gpuvm *gpuvm, + const struct drm_gpuvm_ops *ops, void *priv, + u64 req_addr, u64 req_range, + struct drm_gem_object *req_obj, u64 req_offset) +{ + struct drm_gpuva *va, *next; + u64 req_end = req_addr + req_range; + int ret; + + if (unlikely(!drm_gpuvm_range_valid(gpuvm, req_addr, req_range))) + return -EINVAL; + + drm_gpuvm_for_each_va_range_safe(va, next, gpuvm, req_addr, req_end) { + struct drm_gem_object *obj = va->gem.obj; + u64 offset = va->gem.offset; + u64 addr = va->va.addr; + u64 range = va->va.range; + u64 end = addr + range; + bool merge = !!va->gem.obj; + + if (addr == req_addr) { + merge &= obj == req_obj && + offset == req_offset; + + if (end == req_end) { + ret = op_unmap_cb(ops, priv, va, merge); + if (ret) + return ret; + break; + } + + if (end < req_end) { + ret = op_unmap_cb(ops, priv, va, merge); + if (ret) + return ret; + continue; + } + + if (end > req_end) { + struct drm_gpuva_op_map n = { + .va.addr = req_end, + .va.range = range - req_range, + .gem.obj = obj, + .gem.offset = offset + req_range, + }; + struct drm_gpuva_op_unmap u = { + .va = va, + .keep = merge, + }; + + ret = op_remap_cb(ops, priv, NULL, &n, &u); + if (ret) + return ret; + break; + } + } else if (addr < req_addr) { + u64 ls_range = req_addr - addr; + struct drm_gpuva_op_map p = { + .va.addr = addr, + .va.range = ls_range, + .gem.obj = obj, + .gem.offset = offset, + }; + struct drm_gpuva_op_unmap u = { .va = va }; + + merge &= obj == req_obj && + offset + ls_range == req_offset; + u.keep = merge; + + if (end == req_end) { + ret = op_remap_cb(ops, priv, &p, NULL, &u); + if (ret) + return ret; + break; + } + + if (end < req_end) { + ret = op_remap_cb(ops, priv, &p, NULL, &u); + if (ret) + return ret; + continue; + } + + if (end > req_end) { + struct drm_gpuva_op_map n = { + .va.addr = req_end, + .va.range = end - req_end, + .gem.obj = obj, + .gem.offset = offset + ls_range + + req_range, + }; + + ret = op_remap_cb(ops, priv, &p, &n, &u); + if (ret) + return ret; + break; + } + } else if (addr > req_addr) { + merge &= obj == req_obj && + offset == req_offset + + (addr - req_addr); + + if (end == req_end) { + ret = op_unmap_cb(ops, priv, va, merge); + if (ret) + return ret; + break; + } + + if (end < req_end) { + ret = op_unmap_cb(ops, priv, va, merge); + if (ret) + return ret; + continue; + } + + if (end > req_end) { + struct drm_gpuva_op_map n = { + .va.addr = req_end, + .va.range = end - req_end, + .gem.obj = obj, + .gem.offset = offset + req_end - addr, + }; + struct drm_gpuva_op_unmap u = { + .va = va, + .keep = merge, + }; + + ret = op_remap_cb(ops, priv, NULL, &n, &u); + if (ret) + return ret; + break; + } + } + } + + return op_map_cb(ops, priv, + req_addr, req_range, + req_obj, req_offset); +} + +static int +__drm_gpuvm_sm_unmap(struct drm_gpuvm *gpuvm, + const struct drm_gpuvm_ops *ops, void *priv, + u64 req_addr, u64 req_range) +{ + struct drm_gpuva *va, *next; + u64 req_end = req_addr + req_range; + int ret; + + if (unlikely(!drm_gpuvm_range_valid(gpuvm, req_addr, req_range))) + return -EINVAL; + + drm_gpuvm_for_each_va_range_safe(va, next, gpuvm, req_addr, req_end) { + struct drm_gpuva_op_map prev = {}, next = {}; + bool prev_split = false, next_split = false; + struct drm_gem_object *obj = va->gem.obj; + u64 offset = va->gem.offset; + u64 addr = va->va.addr; + u64 range = va->va.range; + u64 end = addr + range; + + if (addr < req_addr) { + prev.va.addr = addr; + prev.va.range = req_addr - addr; + prev.gem.obj = obj; + prev.gem.offset = offset; + + prev_split = true; + } + + if (end > req_end) { + next.va.addr = req_end; + next.va.range = end - req_end; + next.gem.obj = obj; + next.gem.offset = offset + (req_end - addr); + + next_split = true; + } + + if (prev_split || next_split) { + struct drm_gpuva_op_unmap unmap = { .va = va }; + + ret = op_remap_cb(ops, priv, + prev_split ? &prev : NULL, + next_split ? &next : NULL, + &unmap); + if (ret) + return ret; + } else { + ret = op_unmap_cb(ops, priv, va, false); + if (ret) + return ret; + } + } + + return 0; +} + +/** + * drm_gpuvm_sm_map() - creates the &drm_gpuva_op split/merge steps + * @gpuvm: the &drm_gpuvm representing the GPU VA space + * @req_addr: the start address of the new mapping + * @req_range: the range of the new mapping + * @req_obj: the &drm_gem_object to map + * @req_offset: the offset within the &drm_gem_object + * @priv: pointer to a driver private data structure + * + * This function iterates the given range of the GPU VA space. It utilizes the + * &drm_gpuvm_ops to call back into the driver providing the split and merge + * steps. + * + * Drivers may use these callbacks to update the GPU VA space right away within + * the callback. In case the driver decides to copy and store the operations for + * later processing neither this function nor &drm_gpuvm_sm_unmap is allowed to + * be called before the &drm_gpuvm's view of the GPU VA space was + * updated with the previous set of operations. To update the + * &drm_gpuvm's view of the GPU VA space drm_gpuva_insert(), + * drm_gpuva_destroy_locked() and/or drm_gpuva_destroy_unlocked() should be + * used. + * + * A sequence of callbacks can contain map, unmap and remap operations, but + * the sequence of callbacks might also be empty if no operation is required, + * e.g. if the requested mapping already exists in the exact same way. + * + * There can be an arbitrary amount of unmap operations, a maximum of two remap + * operations and a single map operation. The latter one represents the original + * map operation requested by the caller. + * + * Returns: 0 on success or a negative error code + */ +int +drm_gpuvm_sm_map(struct drm_gpuvm *gpuvm, void *priv, + u64 req_addr, u64 req_range, + struct drm_gem_object *req_obj, u64 req_offset) +{ + const struct drm_gpuvm_ops *ops = gpuvm->ops; + + if (unlikely(!(ops && ops->sm_step_map && + ops->sm_step_remap && + ops->sm_step_unmap))) + return -EINVAL; + + return __drm_gpuvm_sm_map(gpuvm, ops, priv, + req_addr, req_range, + req_obj, req_offset); +} +EXPORT_SYMBOL_GPL(drm_gpuvm_sm_map); + +/** + * drm_gpuvm_sm_unmap() - creates the &drm_gpuva_ops to split on unmap + * @gpuvm: the &drm_gpuvm representing the GPU VA space + * @priv: pointer to a driver private data structure + * @req_addr: the start address of the range to unmap + * @req_range: the range of the mappings to unmap + * + * This function iterates the given range of the GPU VA space. It utilizes the + * &drm_gpuvm_ops to call back into the driver providing the operations to + * unmap and, if required, split existent mappings. + * + * Drivers may use these callbacks to update the GPU VA space right away within + * the callback. In case the driver decides to copy and store the operations for + * later processing neither this function nor &drm_gpuvm_sm_map is allowed to be + * called before the &drm_gpuvm's view of the GPU VA space was updated + * with the previous set of operations. To update the &drm_gpuvm's view + * of the GPU VA space drm_gpuva_insert(), drm_gpuva_destroy_locked() and/or + * drm_gpuva_destroy_unlocked() should be used. + * + * A sequence of callbacks can contain unmap and remap operations, depending on + * whether there are actual overlapping mappings to split. + * + * There can be an arbitrary amount of unmap operations and a maximum of two + * remap operations. + * + * Returns: 0 on success or a negative error code + */ +int +drm_gpuvm_sm_unmap(struct drm_gpuvm *gpuvm, void *priv, + u64 req_addr, u64 req_range) +{ + const struct drm_gpuvm_ops *ops = gpuvm->ops; + + if (unlikely(!(ops && ops->sm_step_remap && + ops->sm_step_unmap))) + return -EINVAL; + + return __drm_gpuvm_sm_unmap(gpuvm, ops, priv, + req_addr, req_range); +} +EXPORT_SYMBOL_GPL(drm_gpuvm_sm_unmap); + +static struct drm_gpuva_op * +gpuva_op_alloc(struct drm_gpuvm *gpuvm) +{ + const struct drm_gpuvm_ops *fn = gpuvm->ops; + struct drm_gpuva_op *op; + + if (fn && fn->op_alloc) + op = fn->op_alloc(); + else + op = kzalloc(sizeof(*op), GFP_KERNEL); + + if (unlikely(!op)) + return NULL; + + return op; +} + +static void +gpuva_op_free(struct drm_gpuvm *gpuvm, + struct drm_gpuva_op *op) +{ + const struct drm_gpuvm_ops *fn = gpuvm->ops; + + if (fn && fn->op_free) + fn->op_free(op); + else + kfree(op); +} + +static int +drm_gpuva_sm_step(struct drm_gpuva_op *__op, + void *priv) +{ + struct { + struct drm_gpuvm *vm; + struct drm_gpuva_ops *ops; + } *args = priv; + struct drm_gpuvm *gpuvm = args->vm; + struct drm_gpuva_ops *ops = args->ops; + struct drm_gpuva_op *op; + + op = gpuva_op_alloc(gpuvm); + if (unlikely(!op)) + goto err; + + memcpy(op, __op, sizeof(*op)); + + if (op->op == DRM_GPUVA_OP_REMAP) { + struct drm_gpuva_op_remap *__r = &__op->remap; + struct drm_gpuva_op_remap *r = &op->remap; + + r->unmap = kmemdup(__r->unmap, sizeof(*r->unmap), + GFP_KERNEL); + if (unlikely(!r->unmap)) + goto err_free_op; + + if (__r->prev) { + r->prev = kmemdup(__r->prev, sizeof(*r->prev), + GFP_KERNEL); + if (unlikely(!r->prev)) + goto err_free_unmap; + } + + if (__r->next) { + r->next = kmemdup(__r->next, sizeof(*r->next), + GFP_KERNEL); + if (unlikely(!r->next)) + goto err_free_prev; + } + } + + list_add_tail(&op->entry, &ops->list); + + return 0; + +err_free_unmap: + kfree(op->remap.unmap); +err_free_prev: + kfree(op->remap.prev); +err_free_op: + gpuva_op_free(gpuvm, op); +err: + return -ENOMEM; +} + +static const struct drm_gpuvm_ops gpuvm_list_ops = { + .sm_step_map = drm_gpuva_sm_step, + .sm_step_remap = drm_gpuva_sm_step, + .sm_step_unmap = drm_gpuva_sm_step, +}; + +/** + * drm_gpuvm_sm_map_ops_create() - creates the &drm_gpuva_ops to split and merge + * @gpuvm: the &drm_gpuvm representing the GPU VA space + * @req_addr: the start address of the new mapping + * @req_range: the range of the new mapping + * @req_obj: the &drm_gem_object to map + * @req_offset: the offset within the &drm_gem_object + * + * This function creates a list of operations to perform splitting and merging + * of existent mapping(s) with the newly requested one. + * + * The list can be iterated with &drm_gpuva_for_each_op and must be processed + * in the given order. It can contain map, unmap and remap operations, but it + * also can be empty if no operation is required, e.g. if the requested mapping + * already exists is the exact same way. + * + * There can be an arbitrary amount of unmap operations, a maximum of two remap + * operations and a single map operation. The latter one represents the original + * map operation requested by the caller. + * + * Note that before calling this function again with another mapping request it + * is necessary to update the &drm_gpuvm's view of the GPU VA space. The + * previously obtained operations must be either processed or abandoned. To + * update the &drm_gpuvm's view of the GPU VA space drm_gpuva_insert(), + * drm_gpuva_destroy_locked() and/or drm_gpuva_destroy_unlocked() should be + * used. + * + * After the caller finished processing the returned &drm_gpuva_ops, they must + * be freed with &drm_gpuva_ops_free. + * + * Returns: a pointer to the &drm_gpuva_ops on success, an ERR_PTR on failure + */ +struct drm_gpuva_ops * +drm_gpuvm_sm_map_ops_create(struct drm_gpuvm *gpuvm, + u64 req_addr, u64 req_range, + struct drm_gem_object *req_obj, u64 req_offset) +{ + struct drm_gpuva_ops *ops; + struct { + struct drm_gpuvm *vm; + struct drm_gpuva_ops *ops; + } args; + int ret; + + ops = kzalloc(sizeof(*ops), GFP_KERNEL); + if (unlikely(!ops)) + return ERR_PTR(-ENOMEM); + + INIT_LIST_HEAD(&ops->list); + + args.vm = gpuvm; + args.ops = ops; + + ret = __drm_gpuvm_sm_map(gpuvm, &gpuvm_list_ops, &args, + req_addr, req_range, + req_obj, req_offset); + if (ret) + goto err_free_ops; + + return ops; + +err_free_ops: + drm_gpuva_ops_free(gpuvm, ops); + return ERR_PTR(ret); +} +EXPORT_SYMBOL_GPL(drm_gpuvm_sm_map_ops_create); + +/** + * drm_gpuvm_sm_unmap_ops_create() - creates the &drm_gpuva_ops to split on + * unmap + * @gpuvm: the &drm_gpuvm representing the GPU VA space + * @req_addr: the start address of the range to unmap + * @req_range: the range of the mappings to unmap + * + * This function creates a list of operations to perform unmapping and, if + * required, splitting of the mappings overlapping the unmap range. + * + * The list can be iterated with &drm_gpuva_for_each_op and must be processed + * in the given order. It can contain unmap and remap operations, depending on + * whether there are actual overlapping mappings to split. + * + * There can be an arbitrary amount of unmap operations and a maximum of two + * remap operations. + * + * Note that before calling this function again with another range to unmap it + * is necessary to update the &drm_gpuvm's view of the GPU VA space. The + * previously obtained operations must be processed or abandoned. To update the + * &drm_gpuvm's view of the GPU VA space drm_gpuva_insert(), + * drm_gpuva_destroy_locked() and/or drm_gpuva_destroy_unlocked() should be + * used. + * + * After the caller finished processing the returned &drm_gpuva_ops, they must + * be freed with &drm_gpuva_ops_free. + * + * Returns: a pointer to the &drm_gpuva_ops on success, an ERR_PTR on failure + */ +struct drm_gpuva_ops * +drm_gpuvm_sm_unmap_ops_create(struct drm_gpuvm *gpuvm, + u64 req_addr, u64 req_range) +{ + struct drm_gpuva_ops *ops; + struct { + struct drm_gpuvm *vm; + struct drm_gpuva_ops *ops; + } args; + int ret; + + ops = kzalloc(sizeof(*ops), GFP_KERNEL); + if (unlikely(!ops)) + return ERR_PTR(-ENOMEM); + + INIT_LIST_HEAD(&ops->list); + + args.vm = gpuvm; + args.ops = ops; + + ret = __drm_gpuvm_sm_unmap(gpuvm, &gpuvm_list_ops, &args, + req_addr, req_range); + if (ret) + goto err_free_ops; + + return ops; + +err_free_ops: + drm_gpuva_ops_free(gpuvm, ops); + return ERR_PTR(ret); +} +EXPORT_SYMBOL_GPL(drm_gpuvm_sm_unmap_ops_create); + +/** + * drm_gpuvm_prefetch_ops_create() - creates the &drm_gpuva_ops to prefetch + * @gpuvm: the &drm_gpuvm representing the GPU VA space + * @addr: the start address of the range to prefetch + * @range: the range of the mappings to prefetch + * + * This function creates a list of operations to perform prefetching. + * + * The list can be iterated with &drm_gpuva_for_each_op and must be processed + * in the given order. It can contain prefetch operations. + * + * There can be an arbitrary amount of prefetch operations. + * + * After the caller finished processing the returned &drm_gpuva_ops, they must + * be freed with &drm_gpuva_ops_free. + * + * Returns: a pointer to the &drm_gpuva_ops on success, an ERR_PTR on failure + */ +struct drm_gpuva_ops * +drm_gpuvm_prefetch_ops_create(struct drm_gpuvm *gpuvm, + u64 addr, u64 range) +{ + struct drm_gpuva_ops *ops; + struct drm_gpuva_op *op; + struct drm_gpuva *va; + u64 end = addr + range; + int ret; + + ops = kzalloc(sizeof(*ops), GFP_KERNEL); + if (!ops) + return ERR_PTR(-ENOMEM); + + INIT_LIST_HEAD(&ops->list); + + drm_gpuvm_for_each_va_range(va, gpuvm, addr, end) { + op = gpuva_op_alloc(gpuvm); + if (!op) { + ret = -ENOMEM; + goto err_free_ops; + } + + op->op = DRM_GPUVA_OP_PREFETCH; + op->prefetch.va = va; + list_add_tail(&op->entry, &ops->list); + } + + return ops; + +err_free_ops: + drm_gpuva_ops_free(gpuvm, ops); + return ERR_PTR(ret); +} +EXPORT_SYMBOL_GPL(drm_gpuvm_prefetch_ops_create); + +/** + * drm_gpuvm_gem_unmap_ops_create() - creates the &drm_gpuva_ops to unmap a GEM + * @gpuvm: the &drm_gpuvm representing the GPU VA space + * @obj: the &drm_gem_object to unmap + * + * This function creates a list of operations to perform unmapping for every + * GPUVA attached to a GEM. + * + * The list can be iterated with &drm_gpuva_for_each_op and consists out of an + * arbitrary amount of unmap operations. + * + * After the caller finished processing the returned &drm_gpuva_ops, they must + * be freed with &drm_gpuva_ops_free. + * + * It is the callers responsibility to protect the GEMs GPUVA list against + * concurrent access using the GEMs dma_resv lock. + * + * Returns: a pointer to the &drm_gpuva_ops on success, an ERR_PTR on failure + */ +struct drm_gpuva_ops * +drm_gpuvm_gem_unmap_ops_create(struct drm_gpuvm *gpuvm, + struct drm_gem_object *obj) +{ + struct drm_gpuva_ops *ops; + struct drm_gpuva_op *op; + struct drm_gpuva *va; + int ret; + + drm_gem_gpuva_assert_lock_held(obj); + + ops = kzalloc(sizeof(*ops), GFP_KERNEL); + if (!ops) + return ERR_PTR(-ENOMEM); + + INIT_LIST_HEAD(&ops->list); + + drm_gem_for_each_gpuva(va, obj) { + op = gpuva_op_alloc(gpuvm); + if (!op) { + ret = -ENOMEM; + goto err_free_ops; + } + + op->op = DRM_GPUVA_OP_UNMAP; + op->unmap.va = va; + list_add_tail(&op->entry, &ops->list); + } + + return ops; + +err_free_ops: + drm_gpuva_ops_free(gpuvm, ops); + return ERR_PTR(ret); +} +EXPORT_SYMBOL_GPL(drm_gpuvm_gem_unmap_ops_create); + +/** + * drm_gpuva_ops_free() - free the given &drm_gpuva_ops + * @gpuvm: the &drm_gpuvm the ops were created for + * @ops: the &drm_gpuva_ops to free + * + * Frees the given &drm_gpuva_ops structure including all the ops associated + * with it. + */ +void +drm_gpuva_ops_free(struct drm_gpuvm *gpuvm, + struct drm_gpuva_ops *ops) +{ + struct drm_gpuva_op *op, *next; + + drm_gpuva_for_each_op_safe(op, next, ops) { + list_del(&op->entry); + + if (op->op == DRM_GPUVA_OP_REMAP) { + kfree(op->remap.prev); + kfree(op->remap.next); + kfree(op->remap.unmap); + } + + gpuva_op_free(gpuvm, op); + } + + kfree(ops); +} +EXPORT_SYMBOL_GPL(drm_gpuva_ops_free); diff --git a/drivers/gpu/drm/drm_syncobj.c b/drivers/gpu/drm/drm_syncobj.c index 7b4ed5ca0a9b..484ece4823b2 100644 --- a/drivers/gpu/drm/drm_syncobj.c +++ b/drivers/gpu/drm/drm_syncobj.c @@ -1367,7 +1367,7 @@ static void syncobj_eventfd_entry_fence_func(struct dma_fence *fence, struct syncobj_eventfd_entry *entry = container_of(cb, struct syncobj_eventfd_entry, fence_cb); - eventfd_signal(entry->ev_fd_ctx, 1); + eventfd_signal(entry->ev_fd_ctx); syncobj_eventfd_entry_free(entry); } @@ -1401,13 +1401,13 @@ syncobj_eventfd_entry_func(struct drm_syncobj *syncobj, entry->fence = fence; if (entry->flags & DRM_SYNCOBJ_WAIT_FLAGS_WAIT_AVAILABLE) { - eventfd_signal(entry->ev_fd_ctx, 1); + eventfd_signal(entry->ev_fd_ctx); syncobj_eventfd_entry_free(entry); } else { ret = dma_fence_add_callback(fence, &entry->fence_cb, syncobj_eventfd_entry_fence_func); if (ret == -ENOENT) { - eventfd_signal(entry->ev_fd_ctx, 1); + eventfd_signal(entry->ev_fd_ctx); syncobj_eventfd_entry_free(entry); } } diff --git a/drivers/gpu/drm/etnaviv/etnaviv_sched.c b/drivers/gpu/drm/etnaviv/etnaviv_sched.c index a3bd396c9d82..b98b83fdc3dd 100644 --- a/drivers/gpu/drm/etnaviv/etnaviv_sched.c +++ b/drivers/gpu/drm/etnaviv/etnaviv_sched.c @@ -134,12 +134,17 @@ int etnaviv_sched_push_job(struct etnaviv_gem_submit *submit) int etnaviv_sched_init(struct etnaviv_gpu *gpu) { + const struct drm_sched_init_args args = { + .ops = &etnaviv_sched_ops, + .hw_submission = etnaviv_hw_jobs_limit, + .hang_limit = etnaviv_job_hang_limit, + .timeout = msecs_to_jiffies(500), + .name = dev_name(gpu->dev), + .dev = gpu->dev, + }; int ret; - ret = drm_sched_init(&gpu->sched, &etnaviv_sched_ops, - etnaviv_hw_jobs_limit, etnaviv_job_hang_limit, - msecs_to_jiffies(500), NULL, NULL, - dev_name(gpu->dev), gpu->dev); + ret = drm_sched_init(&gpu->sched, &args); if (ret) return ret; diff --git a/drivers/gpu/drm/gud/gud_drv.c b/drivers/gpu/drm/gud/gud_drv.c index ab7662a31c16..434c6c4b6c93 100644 --- a/drivers/gpu/drm/gud/gud_drv.c +++ b/drivers/gpu/drm/gud/gud_drv.c @@ -288,6 +288,8 @@ static int gud_get_properties(struct gud_device *gdrm) * but mask out any additions on future devices. */ val &= GUD_ROTATION_MASK; + if (!(val & GUD_ROTATION_0)) + continue; ret = drm_plane_create_rotation_property(&gdrm->pipe.plane, DRM_MODE_ROTATE_0, val); break; diff --git a/drivers/gpu/drm/gud/gud_pipe.c b/drivers/gpu/drm/gud/gud_pipe.c index cd6ef313e1f5..df8cf5369708 100644 --- a/drivers/gpu/drm/gud/gud_pipe.c +++ b/drivers/gpu/drm/gud/gud_pipe.c @@ -480,6 +480,7 @@ int gud_pipe_check(struct drm_simple_display_pipe *pipe, for_each_new_connector_in_state(state, connector, connector_state, i) { if (connector_state->crtc) break; + connector_state = NULL; } /* diff --git a/drivers/gpu/drm/i915/display/icl_dsi.c b/drivers/gpu/drm/i915/display/icl_dsi.c index 0a0efeeb790e..1b11df1814f4 100644 --- a/drivers/gpu/drm/i915/display/icl_dsi.c +++ b/drivers/gpu/drm/i915/display/icl_dsi.c @@ -859,7 +859,7 @@ gen11_dsi_set_transcoder_timings(struct intel_encoder *encoder, * non-compressed link speeds, and simplifies down to the ratio between * compressed and non-compressed bpp. */ - if (crtc_state->dsc.compression_enable) { + if (is_vid_mode(intel_dsi) && crtc_state->dsc.compression_enable) { mul = crtc_state->dsc.compressed_bpp; div = mipi_dsi_pixel_format_to_bpp(intel_dsi->pixel_format); } @@ -1458,7 +1458,7 @@ static void gen11_dsi_get_timings(struct intel_encoder *encoder, struct drm_display_mode *adjusted_mode = &pipe_config->hw.adjusted_mode; - if (pipe_config->dsc.compressed_bpp) { + if (is_vid_mode(intel_dsi) && pipe_config->dsc.compressed_bpp) { int div = pipe_config->dsc.compressed_bpp; int mul = mipi_dsi_pixel_format_to_bpp(intel_dsi->pixel_format); diff --git a/drivers/gpu/drm/i915/display/intel_bios.c b/drivers/gpu/drm/i915/display/intel_bios.c index 27d1c49b46ec..0b7522ad8f3e 100644 --- a/drivers/gpu/drm/i915/display/intel_bios.c +++ b/drivers/gpu/drm/i915/display/intel_bios.c @@ -617,6 +617,21 @@ static void dump_pnp_id(struct drm_i915_private *i915, pnp_id->mfg_week, pnp_id->mfg_year + 1990); } +static bool is_panel_type_valid(int panel_type) +{ + return panel_type >= 0 && panel_type < 16; +} + +static bool is_panel_type_pnp(int panel_type) +{ + return panel_type == 0xff; +} + +static bool is_panel_type_valid_or_pnp(int panel_type) +{ + return is_panel_type_valid(panel_type) || is_panel_type_pnp(panel_type); +} + static int opregion_get_panel_type(struct drm_i915_private *i915, const struct intel_bios_encoder_data *devdata, const struct drm_edid *drm_edid, bool use_fallback) @@ -634,15 +649,21 @@ static int vbt_get_panel_type(struct drm_i915_private *i915, if (!lvds_options) return -1; - if (lvds_options->panel_type > 0xf && - lvds_options->panel_type != 0xff) { + if (!is_panel_type_valid_or_pnp(lvds_options->panel_type)) { drm_dbg_kms(&i915->drm, "Invalid VBT panel type 0x%x\n", lvds_options->panel_type); return -1; } - if (devdata && devdata->child.handle == DEVICE_HANDLE_LFP2) + if (devdata && devdata->child.handle == DEVICE_HANDLE_LFP2) { + if (!is_panel_type_valid_or_pnp(lvds_options->panel_type2)) { + drm_dbg_kms(&i915->drm, "Invalid VBT panel type 2 0x%x\n", + lvds_options->panel_type2); + return -1; + } + return lvds_options->panel_type2; + } drm_WARN_ON(&i915->drm, devdata && devdata->child.handle != DEVICE_HANDLE_LFP1); @@ -755,13 +776,12 @@ static int get_panel_type(struct drm_i915_private *i915, panel_types[i].name, panel_types[i].panel_type); } - if (panel_types[PANEL_TYPE_OPREGION].panel_type >= 0) + if (is_panel_type_valid(panel_types[PANEL_TYPE_OPREGION].panel_type)) i = PANEL_TYPE_OPREGION; - else if (panel_types[PANEL_TYPE_VBT].panel_type == 0xff && - panel_types[PANEL_TYPE_PNPID].panel_type >= 0) + else if (is_panel_type_pnp(panel_types[PANEL_TYPE_VBT].panel_type) && + is_panel_type_valid(panel_types[PANEL_TYPE_PNPID].panel_type)) i = PANEL_TYPE_PNPID; - else if (panel_types[PANEL_TYPE_VBT].panel_type != 0xff && - panel_types[PANEL_TYPE_VBT].panel_type >= 0) + else if (is_panel_type_valid(panel_types[PANEL_TYPE_VBT].panel_type)) i = PANEL_TYPE_VBT; else i = PANEL_TYPE_FALLBACK; diff --git a/drivers/gpu/drm/i915/display/intel_dp.c b/drivers/gpu/drm/i915/display/intel_dp.c index c3010c9e31c1..d1b587f91caf 100644 --- a/drivers/gpu/drm/i915/display/intel_dp.c +++ b/drivers/gpu/drm/i915/display/intel_dp.c @@ -4149,10 +4149,11 @@ intel_dp_check_mst_status(struct intel_dp *intel_dp) { struct drm_i915_private *i915 = dp_to_i915(intel_dp); bool link_ok = true; + int tries = 33; drm_WARN_ON_ONCE(&i915->drm, intel_dp->active_mst_links < 0); - for (;;) { + while (--tries) { u8 esi[4] = {}; u8 ack[4] = {}; @@ -4185,6 +4186,11 @@ intel_dp_check_mst_status(struct intel_dp *intel_dp) drm_dp_mst_hpd_irq_send_new_request(&intel_dp->mst_mgr); } + if (!tries) { + drm_dbg_kms(&i915->drm, "DPRX ESI not clearing, device may be stuck\n"); + link_ok = false; + } + return link_ok; } diff --git a/drivers/gpu/drm/i915/display/intel_hdcp.c b/drivers/gpu/drm/i915/display/intel_hdcp.c index f377c4484e18..89e378f71d67 100644 --- a/drivers/gpu/drm/i915/display/intel_hdcp.c +++ b/drivers/gpu/drm/i915/display/intel_hdcp.c @@ -142,11 +142,16 @@ int intel_hdcp_read_valid_bksv(struct intel_digital_port *dig_port, /* Is HDCP1.4 capable on Platform and Sink */ bool intel_hdcp_capable(struct intel_connector *connector) { - struct intel_digital_port *dig_port = intel_attached_dig_port(connector); + struct intel_digital_port *dig_port; const struct intel_hdcp_shim *shim = connector->hdcp.shim; bool capable = false; u8 bksv[5]; + if (!intel_attached_encoder(connector)) + return capable; + + dig_port = intel_attached_dig_port(connector); + if (!shim) return capable; @@ -163,11 +168,16 @@ bool intel_hdcp_capable(struct intel_connector *connector) /* Is HDCP2.2 capable on Platform and Sink */ bool intel_hdcp2_capable(struct intel_connector *connector) { - struct intel_digital_port *dig_port = intel_attached_dig_port(connector); + struct intel_digital_port *dig_port; struct drm_i915_private *i915 = to_i915(connector->base.dev); struct intel_hdcp *hdcp = &connector->hdcp; bool capable = false; + if (!intel_attached_encoder(connector)) + return capable; + + dig_port = intel_attached_dig_port(connector); + /* I915 support for HDCP2.2 */ if (!hdcp->hdcp2_supported) return false; diff --git a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c index e208cc8c2ac1..a3519fb5e29d 100644 --- a/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c +++ b/drivers/gpu/drm/i915/gem/i915_gem_execbuffer.c @@ -1764,6 +1764,7 @@ static int eb_reinit_userptr(struct i915_execbuffer *eb) static noinline int eb_relocate_parse_slow(struct i915_execbuffer *eb) { + const unsigned int count = eb->buffer_count; bool have_copy = false; struct eb_vma *ev; int err = 0; @@ -1866,7 +1867,6 @@ static noinline int eb_relocate_parse_slow(struct i915_execbuffer *eb) out: if (have_copy) { - const unsigned int count = eb->buffer_count; unsigned int i; for (i = 0; i < count; i++) { diff --git a/drivers/gpu/drm/i915/gt/intel_execlists_submission.c b/drivers/gpu/drm/i915/gt/intel_execlists_submission.c index 2065be5a196b..c789f91f29df 100644 --- a/drivers/gpu/drm/i915/gt/intel_execlists_submission.c +++ b/drivers/gpu/drm/i915/gt/intel_execlists_submission.c @@ -3935,11 +3935,11 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count, struct drm_i915_private *i915 = siblings[0]->i915; struct virtual_engine *ve; unsigned int n; - int err; + int err = -ENOMEM; ve = kzalloc(struct_size(ve, siblings, count), GFP_KERNEL); if (!ve) - return ERR_PTR(-ENOMEM); + goto err; ve->base.i915 = i915; ve->base.gt = siblings[0]->gt; @@ -3971,10 +3971,8 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count, intel_engine_init_execlists(&ve->base); ve->base.sched_engine = i915_sched_engine_create(ENGINE_VIRTUAL); - if (!ve->base.sched_engine) { - err = -ENOMEM; - goto err_put; - } + if (!ve->base.sched_engine) + goto err_noput; ve->base.sched_engine->private_data = &ve->base; ve->base.cops = &virtual_context_ops; @@ -3990,10 +3988,8 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count, intel_context_init(&ve->context, &ve->base); ve->base.breadcrumbs = intel_breadcrumbs_create(NULL); - if (!ve->base.breadcrumbs) { - err = -ENOMEM; + if (!ve->base.breadcrumbs) goto err_put; - } for (n = 0; n < count; n++) { struct intel_engine_cs *sibling = siblings[n]; @@ -4068,8 +4064,13 @@ execlists_create_virtual(struct intel_engine_cs **siblings, unsigned int count, virtual_engine_initial_hint(ve); return &ve->context; +err_noput: + kfree(ve); + goto err; + err_put: intel_context_put(&ve->context); +err: return ERR_PTR(err); } diff --git a/drivers/gpu/drm/i915/gt/intel_rps.c b/drivers/gpu/drm/i915/gt/intel_rps.c index 4feef874e6d6..ba5f721af67c 100644 --- a/drivers/gpu/drm/i915/gt/intel_rps.c +++ b/drivers/gpu/drm/i915/gt/intel_rps.c @@ -1886,7 +1886,8 @@ static void rps_work(struct work_struct *work) out: spin_lock_irq(gt->irq_lock); - gen6_gt_pm_unmask_irq(gt, rps->pm_events); + if (intel_rps_is_active(rps)) + gen6_gt_pm_unmask_irq(gt, rps->pm_events); spin_unlock_irq(gt->irq_lock); } diff --git a/drivers/gpu/drm/i915/gvt/interrupt.c b/drivers/gpu/drm/i915/gvt/interrupt.c index 80301472ac98..3f89e991dffa 100644 --- a/drivers/gpu/drm/i915/gvt/interrupt.c +++ b/drivers/gpu/drm/i915/gvt/interrupt.c @@ -436,7 +436,7 @@ static void inject_virtual_interrupt(struct intel_vgpu *vgpu) if (!test_bit(INTEL_VGPU_STATUS_ATTACHED, vgpu->status)) return; if (vgpu->msi_trigger) - eventfd_signal(vgpu->msi_trigger, 1); + eventfd_signal(vgpu->msi_trigger); } static void propagate_event(struct intel_gvt_irq *irq, diff --git a/drivers/gpu/drm/lima/lima_sched.c b/drivers/gpu/drm/lima/lima_sched.c index 1114bffe38c8..4f3954e50549 100644 --- a/drivers/gpu/drm/lima/lima_sched.c +++ b/drivers/gpu/drm/lima/lima_sched.c @@ -489,16 +489,21 @@ int lima_sched_pipe_init(struct lima_sched_pipe *pipe, const char *name) { unsigned int timeout = lima_sched_timeout_ms > 0 ? lima_sched_timeout_ms : 500; + struct drm_sched_init_args args = { + .ops = &lima_sched_ops, + .hw_submission = 1, + .hang_limit = lima_job_hang_limit, + .timeout = msecs_to_jiffies(timeout), + .name = name, + .dev = pipe->ldev->dev, + }; pipe->fence_context = dma_fence_context_alloc(1); spin_lock_init(&pipe->fence_lock); INIT_WORK(&pipe->recover_work, lima_sched_recover_work); - return drm_sched_init(&pipe->base, &lima_sched_ops, 1, - lima_job_hang_limit, - msecs_to_jiffies(timeout), NULL, - NULL, name, pipe->ldev->dev); + return drm_sched_init(&pipe->base, &args); } void lima_sched_pipe_fini(struct lima_sched_pipe *pipe) diff --git a/drivers/gpu/drm/mediatek/mtk_drm_drv.c b/drivers/gpu/drm/mediatek/mtk_drm_drv.c index ed6c650f081b..5855a578c102 100644 --- a/drivers/gpu/drm/mediatek/mtk_drm_drv.c +++ b/drivers/gpu/drm/mediatek/mtk_drm_drv.c @@ -621,8 +621,10 @@ static int mtk_drm_bind(struct device *dev) private->mtk_drm_bound = true; private->dev = dev; - if (!mtk_drm_get_all_drm_priv(dev)) + if (!mtk_drm_get_all_drm_priv(dev)) { + put_device(private->mutex_dev); return 0; + } drm = drm_dev_alloc(&mtk_drm_driver, dev); if (IS_ERR(drm)) { @@ -808,7 +810,7 @@ static int mtk_drm_probe(struct platform_device *pdev) struct mtk_drm_private *private; struct device_node *node; struct component_match *match = NULL; - struct platform_device *ovl_adaptor; + struct platform_device *ovl_adaptor = NULL; int ret; int i; @@ -840,6 +842,9 @@ static int mtk_drm_probe(struct platform_device *pdev) PLATFORM_DEVID_AUTO, (void *)private->mmsys_dev, sizeof(*private->mmsys_dev)); + if (IS_ERR(ovl_adaptor)) + return PTR_ERR(ovl_adaptor); + private->ddp_comp[DDP_COMPONENT_DRM_OVL_ADAPTOR].dev = &ovl_adaptor->dev; mtk_ddp_comp_init(dev, NULL, &private->ddp_comp[DDP_COMPONENT_DRM_OVL_ADAPTOR], DDP_COMPONENT_DRM_OVL_ADAPTOR); @@ -936,6 +941,8 @@ static int mtk_drm_probe(struct platform_device *pdev) err_pm: pm_runtime_disable(dev); err_node: + if (ovl_adaptor) + platform_device_unregister(ovl_adaptor); of_node_put(private->mutex_node); for (i = 0; i < DDP_COMPONENT_DRM_ID_MAX; i++) of_node_put(private->comp_node[i]); @@ -945,9 +952,15 @@ static int mtk_drm_probe(struct platform_device *pdev) static void mtk_drm_remove(struct platform_device *pdev) { struct mtk_drm_private *private = platform_get_drvdata(pdev); + struct device *ovl_adaptor_dev; int i; + ovl_adaptor_dev = + private->ddp_comp[DDP_COMPONENT_DRM_OVL_ADAPTOR].dev; + component_master_del(&pdev->dev, &mtk_drm_ops); + if (ovl_adaptor_dev) + platform_device_unregister(to_platform_device(ovl_adaptor_dev)); pm_runtime_disable(&pdev->dev); of_node_put(private->mutex_node); for (i = 0; i < DDP_COMPONENT_DRM_ID_MAX; i++) diff --git a/drivers/gpu/drm/msm/adreno/a6xx_gpu.c b/drivers/gpu/drm/msm/adreno/a6xx_gpu.c index 6666d8f6c17a..58233ca8d1d3 100644 --- a/drivers/gpu/drm/msm/adreno/a6xx_gpu.c +++ b/drivers/gpu/drm/msm/adreno/a6xx_gpu.c @@ -2315,7 +2315,6 @@ struct msm_gpu *a6xx_gpu_init(struct drm_device *dev) struct msm_drm_private *priv = dev->dev_private; struct platform_device *pdev = priv->gpu_pdev; struct adreno_platform_config *config = pdev->dev.platform_data; - struct device_node *node; struct a6xx_gpu *a6xx_gpu; struct adreno_gpu *adreno_gpu; struct msm_gpu *gpu; @@ -2333,7 +2332,8 @@ struct msm_gpu *a6xx_gpu_init(struct drm_device *dev) adreno_gpu->registers = NULL; /* Check if there is a GMU phandle and set it up */ - node = of_parse_phandle(pdev->dev.of_node, "qcom,gmu", 0); + struct device_node *node __free(device_node) = + of_parse_phandle(pdev->dev.of_node, "qcom,gmu", 0); /* FIXME: How do we gracefully handle this? */ BUG_ON(!node); @@ -2371,7 +2371,6 @@ struct msm_gpu *a6xx_gpu_init(struct drm_device *dev) ret = a6xx_gmu_wrapper_init(a6xx_gpu, node); else ret = a6xx_gmu_init(a6xx_gpu, node); - of_node_put(node); if (ret) { a6xx_destroy(&(a6xx_gpu->base.base)); return ERR_PTR(ret); diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_3_0_msm8998.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_3_0_msm8998.h index a857ce8e385f..1b874b101952 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_3_0_msm8998.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_3_0_msm8998.h @@ -232,7 +232,6 @@ static const struct dpu_intf_cfg msm8998_intf[] = { .prog_fetch_lines_worst_case = 21, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x6a800, .len = 0x280, @@ -241,7 +240,6 @@ static const struct dpu_intf_cfg msm8998_intf[] = { .prog_fetch_lines_worst_case = 21, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 26), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 27), - .intr_tear_rd_ptr = -1, }, { .name = "intf_2", .id = INTF_2, .base = 0x6b000, .len = 0x280, @@ -250,7 +248,6 @@ static const struct dpu_intf_cfg msm8998_intf[] = { .prog_fetch_lines_worst_case = 21, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 28), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 29), - .intr_tear_rd_ptr = -1, }, { .name = "intf_3", .id = INTF_3, .base = 0x6b800, .len = 0x280, @@ -258,7 +255,6 @@ static const struct dpu_intf_cfg msm8998_intf[] = { .prog_fetch_lines_worst_case = 21, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 30), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 31), - .intr_tear_rd_ptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_4_0_sdm845.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_4_0_sdm845.h index 3749c014870d..eb37557a6b1d 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_4_0_sdm845.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_4_0_sdm845.h @@ -248,7 +248,6 @@ static const struct dpu_intf_cfg sdm845_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x6a800, .len = 0x280, @@ -257,7 +256,6 @@ static const struct dpu_intf_cfg sdm845_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 26), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 27), - .intr_tear_rd_ptr = -1, }, { .name = "intf_2", .id = INTF_2, .base = 0x6b000, .len = 0x280, @@ -266,7 +264,6 @@ static const struct dpu_intf_cfg sdm845_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 28), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 29), - .intr_tear_rd_ptr = -1, }, { .name = "intf_3", .id = INTF_3, .base = 0x6b800, .len = 0x280, @@ -275,7 +272,6 @@ static const struct dpu_intf_cfg sdm845_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 30), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 31), - .intr_tear_rd_ptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_0_sm8150.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_0_sm8150.h index 76b668f36477..3bd9483392bd 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_0_sm8150.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_0_sm8150.h @@ -222,7 +222,6 @@ static const struct dpu_pingpong_cfg sm8150_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x70800, .len = 0xd4, @@ -230,7 +229,6 @@ static const struct dpu_pingpong_cfg sm8150_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, { .name = "pingpong_2", .id = PINGPONG_2, .base = 0x71000, .len = 0xd4, @@ -238,7 +236,6 @@ static const struct dpu_pingpong_cfg sm8150_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 10), - .intr_rdptr = -1, }, { .name = "pingpong_3", .id = PINGPONG_3, .base = 0x71800, .len = 0xd4, @@ -246,7 +243,6 @@ static const struct dpu_pingpong_cfg sm8150_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 11), - .intr_rdptr = -1, }, { .name = "pingpong_4", .id = PINGPONG_4, .base = 0x72000, .len = 0xd4, @@ -254,7 +250,6 @@ static const struct dpu_pingpong_cfg sm8150_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 30), - .intr_rdptr = -1, }, { .name = "pingpong_5", .id = PINGPONG_5, .base = 0x72800, .len = 0xd4, @@ -262,7 +257,6 @@ static const struct dpu_pingpong_cfg sm8150_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 31), - .intr_rdptr = -1, }, }; @@ -309,7 +303,6 @@ static const struct dpu_intf_cfg sm8150_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x6a800, .len = 0x2bc, @@ -339,7 +332,6 @@ static const struct dpu_intf_cfg sm8150_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 30), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 31), - .intr_tear_rd_ptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_1_sc8180x.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_1_sc8180x.h index b77ecec50733..3d524584eb41 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_1_sc8180x.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_1_sc8180x.h @@ -221,7 +221,6 @@ static const struct dpu_pingpong_cfg sc8180x_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x70800, .len = 0xd4, @@ -229,7 +228,6 @@ static const struct dpu_pingpong_cfg sc8180x_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, { .name = "pingpong_2", .id = PINGPONG_2, .base = 0x71000, .len = 0xd4, @@ -237,7 +235,6 @@ static const struct dpu_pingpong_cfg sc8180x_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 10), - .intr_rdptr = -1, }, { .name = "pingpong_3", .id = PINGPONG_3, .base = 0x71800, .len = 0xd4, @@ -245,7 +242,6 @@ static const struct dpu_pingpong_cfg sc8180x_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 11), - .intr_rdptr = -1, }, { .name = "pingpong_4", .id = PINGPONG_4, .base = 0x72000, .len = 0xd4, @@ -253,7 +249,6 @@ static const struct dpu_pingpong_cfg sc8180x_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 30), - .intr_rdptr = -1, }, { .name = "pingpong_5", .id = PINGPONG_5, .base = 0x72800, .len = 0xd4, @@ -261,7 +256,6 @@ static const struct dpu_pingpong_cfg sc8180x_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 31), - .intr_rdptr = -1, }, }; @@ -316,7 +310,6 @@ static const struct dpu_intf_cfg sc8180x_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x6a800, .len = 0x2bc, @@ -348,7 +341,6 @@ static const struct dpu_intf_cfg sc8180x_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 30), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 31), - .intr_tear_rd_ptr = -1, }, { .name = "intf_4", .id = INTF_4, .base = 0x6c000, .len = 0x280, @@ -358,7 +350,6 @@ static const struct dpu_intf_cfg sc8180x_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 20), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 21), - .intr_tear_rd_ptr = -1, }, { .name = "intf_5", .id = INTF_5, .base = 0x6c800, .len = 0x280, @@ -368,7 +359,6 @@ static const struct dpu_intf_cfg sc8180x_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 22), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 23), - .intr_tear_rd_ptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_4_sm6125.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_4_sm6125.h index 2491eed10039..cec7af6667dc 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_4_sm6125.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_5_4_sm6125.h @@ -129,7 +129,6 @@ static const struct dpu_pingpong_cfg sm6125_pp[] = { .merge_3d = 0, .sblk = &sdm845_pp_sblk, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x70800, .len = 0xd4, @@ -137,7 +136,6 @@ static const struct dpu_pingpong_cfg sm6125_pp[] = { .merge_3d = 0, .sblk = &sdm845_pp_sblk, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, }; @@ -151,7 +149,6 @@ static const struct dpu_intf_cfg sm6125_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x6a800, .len = 0x2c0, diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_0_sm8250.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_0_sm8250.h index a38fb057ceda..ef66c0649bc5 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_0_sm8250.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_0_sm8250.h @@ -221,7 +221,6 @@ static const struct dpu_pingpong_cfg sm8250_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x70800, .len = 0xd4, @@ -229,7 +228,6 @@ static const struct dpu_pingpong_cfg sm8250_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, { .name = "pingpong_2", .id = PINGPONG_2, .base = 0x71000, .len = 0xd4, @@ -237,7 +235,6 @@ static const struct dpu_pingpong_cfg sm8250_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 10), - .intr_rdptr = -1, }, { .name = "pingpong_3", .id = PINGPONG_3, .base = 0x71800, .len = 0xd4, @@ -245,7 +242,6 @@ static const struct dpu_pingpong_cfg sm8250_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 11), - .intr_rdptr = -1, }, { .name = "pingpong_4", .id = PINGPONG_4, .base = 0x72000, .len = 0xd4, @@ -253,7 +249,6 @@ static const struct dpu_pingpong_cfg sm8250_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 30), - .intr_rdptr = -1, }, { .name = "pingpong_5", .id = PINGPONG_5, .base = 0x72800, .len = 0xd4, @@ -261,7 +256,6 @@ static const struct dpu_pingpong_cfg sm8250_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 31), - .intr_rdptr = -1, }, }; @@ -308,7 +302,6 @@ static const struct dpu_intf_cfg sm8250_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x6a800, .len = 0x2c0, @@ -338,7 +331,6 @@ static const struct dpu_intf_cfg sm8250_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 30), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 31), - .intr_tear_rd_ptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_2_sc7180.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_2_sc7180.h index 69d3f7e5e095..9bfa15e4e645 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_2_sc7180.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_2_sc7180.h @@ -120,7 +120,6 @@ static const struct dpu_pingpong_cfg sc7180_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = 0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x70800, .len = 0xd4, @@ -128,7 +127,6 @@ static const struct dpu_pingpong_cfg sc7180_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = 0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, }; @@ -142,7 +140,6 @@ static const struct dpu_intf_cfg sc7180_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x6a800, .len = 0x2c0, diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_3_sm6115.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_3_sm6115.h index 7e6000167ef0..57ce14c18def 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_3_sm6115.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_3_sm6115.h @@ -83,7 +83,6 @@ static const struct dpu_pingpong_cfg sm6115_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = 0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_4_sm6350.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_4_sm6350.h index cf5db6f296bd..62db84bd15f2 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_4_sm6350.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_4_sm6350.h @@ -128,7 +128,6 @@ static struct dpu_pingpong_cfg sm6350_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = 0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x70800, .len = 0xd4, @@ -136,7 +135,6 @@ static struct dpu_pingpong_cfg sm6350_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = 0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, }; @@ -158,7 +156,6 @@ static const struct dpu_intf_cfg sm6350_intf[] = { .prog_fetch_lines_worst_case = 35, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x6a800, .len = 0x2c0, diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_5_qcm2290.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_5_qcm2290.h index 87a03aa16554..fb36fba5171c 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_5_qcm2290.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_5_qcm2290.h @@ -82,7 +82,6 @@ static const struct dpu_pingpong_cfg qcm2290_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = 0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_9_sm6375.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_9_sm6375.h index a327e21c90fb..5a3aad364c78 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_9_sm6375.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_6_9_sm6375.h @@ -85,7 +85,6 @@ static const struct dpu_pingpong_cfg sm6375_pp[] = { .sblk = &sdm845_pp_sblk, .merge_3d = 0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_7_0_sm8350.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_7_0_sm8350.h index e2f181077d11..d619185cfd7c 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_7_0_sm8350.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_7_0_sm8350.h @@ -221,7 +221,6 @@ static const struct dpu_pingpong_cfg sm8350_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x6a000, .len = 0, @@ -229,7 +228,6 @@ static const struct dpu_pingpong_cfg sm8350_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, { .name = "pingpong_2", .id = PINGPONG_2, .base = 0x6b000, .len = 0, @@ -237,7 +235,6 @@ static const struct dpu_pingpong_cfg sm8350_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 10), - .intr_rdptr = -1, }, { .name = "pingpong_3", .id = PINGPONG_3, .base = 0x6c000, .len = 0, @@ -245,7 +242,6 @@ static const struct dpu_pingpong_cfg sm8350_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 11), - .intr_rdptr = -1, }, { .name = "pingpong_4", .id = PINGPONG_4, .base = 0x6d000, .len = 0, @@ -253,7 +249,6 @@ static const struct dpu_pingpong_cfg sm8350_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 30), - .intr_rdptr = -1, }, { .name = "pingpong_5", .id = PINGPONG_5, .base = 0x6e000, .len = 0, @@ -261,7 +256,6 @@ static const struct dpu_pingpong_cfg sm8350_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 31), - .intr_rdptr = -1, }, }; @@ -332,7 +326,6 @@ static const struct dpu_intf_cfg sm8350_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x35000, .len = 0x2c4, @@ -362,7 +355,6 @@ static const struct dpu_intf_cfg sm8350_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 30), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 31), - .intr_tear_rd_ptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_7_2_sc7280.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_7_2_sc7280.h index cbaca4bf2864..72c559fffbc2 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_7_2_sc7280.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_7_2_sc7280.h @@ -132,7 +132,6 @@ static const struct dpu_pingpong_cfg sc7280_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = 0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x6a000, .len = 0, @@ -140,7 +139,6 @@ static const struct dpu_pingpong_cfg sc7280_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = 0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, { .name = "pingpong_2", .id = PINGPONG_2, .base = 0x6b000, .len = 0, @@ -148,7 +146,6 @@ static const struct dpu_pingpong_cfg sc7280_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 10), - .intr_rdptr = -1, }, { .name = "pingpong_3", .id = PINGPONG_3, .base = 0x6c000, .len = 0, @@ -156,7 +153,6 @@ static const struct dpu_pingpong_cfg sc7280_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 11), - .intr_rdptr = -1, }, }; @@ -202,7 +198,6 @@ static const struct dpu_intf_cfg sc7280_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x35000, .len = 0x2c4, @@ -222,7 +217,6 @@ static const struct dpu_intf_cfg sc7280_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 22), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 23), - .intr_tear_rd_ptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_8_0_sc8280xp.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_8_0_sc8280xp.h index ff9adb8000ac..4c0528794e7a 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_8_0_sc8280xp.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_8_0_sc8280xp.h @@ -221,7 +221,6 @@ static const struct dpu_pingpong_cfg sc8280xp_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x6a000, .len = 0, @@ -229,7 +228,6 @@ static const struct dpu_pingpong_cfg sc8280xp_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, { .name = "pingpong_2", .id = PINGPONG_2, .base = 0x6b000, .len = 0, @@ -237,7 +235,6 @@ static const struct dpu_pingpong_cfg sc8280xp_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 10), - .intr_rdptr = -1, }, { .name = "pingpong_3", .id = PINGPONG_3, .base = 0x6c000, .len = 0, @@ -245,7 +242,6 @@ static const struct dpu_pingpong_cfg sc8280xp_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 11), - .intr_rdptr = -1, }, { .name = "pingpong_4", .id = PINGPONG_4, .base = 0x6d000, .len = 0, @@ -253,7 +249,6 @@ static const struct dpu_pingpong_cfg sc8280xp_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 30), - .intr_rdptr = -1, }, { .name = "pingpong_5", .id = PINGPONG_5, .base = 0x6e000, .len = 0, @@ -261,7 +256,6 @@ static const struct dpu_pingpong_cfg sc8280xp_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 31), - .intr_rdptr = -1, }, }; @@ -328,7 +322,6 @@ static const struct dpu_intf_cfg sc8280xp_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x35000, .len = 0x300, @@ -358,7 +351,6 @@ static const struct dpu_intf_cfg sc8280xp_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 30), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 31), - .intr_tear_rd_ptr = -1, }, { .name = "intf_4", .id = INTF_4, .base = 0x38000, .len = 0x280, @@ -368,7 +360,6 @@ static const struct dpu_intf_cfg sc8280xp_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 20), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 21), - .intr_tear_rd_ptr = -1, }, { .name = "intf_5", .id = INTF_5, .base = 0x39000, .len = 0x280, @@ -378,7 +369,6 @@ static const struct dpu_intf_cfg sc8280xp_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 22), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 23), - .intr_tear_rd_ptr = -1, }, { .name = "intf_6", .id = INTF_6, .base = 0x3a000, .len = 0x280, @@ -388,7 +378,6 @@ static const struct dpu_intf_cfg sc8280xp_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 16), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 17), - .intr_tear_rd_ptr = -1, }, { .name = "intf_7", .id = INTF_7, .base = 0x3b000, .len = 0x280, @@ -398,7 +387,6 @@ static const struct dpu_intf_cfg sc8280xp_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 18), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 19), - .intr_tear_rd_ptr = -1, }, { .name = "intf_8", .id = INTF_8, .base = 0x3c000, .len = 0x280, @@ -408,7 +396,6 @@ static const struct dpu_intf_cfg sc8280xp_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 12), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 13), - .intr_tear_rd_ptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_8_1_sm8450.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_8_1_sm8450.h index 72a1726371ca..a9227f80ec31 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_8_1_sm8450.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_8_1_sm8450.h @@ -222,7 +222,6 @@ static const struct dpu_pingpong_cfg sm8450_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x6a000, .len = 0, @@ -230,7 +229,6 @@ static const struct dpu_pingpong_cfg sm8450_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, { .name = "pingpong_2", .id = PINGPONG_2, .base = 0x6b000, .len = 0, @@ -238,7 +236,6 @@ static const struct dpu_pingpong_cfg sm8450_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 10), - .intr_rdptr = -1, }, { .name = "pingpong_3", .id = PINGPONG_3, .base = 0x6c000, .len = 0, @@ -246,7 +243,6 @@ static const struct dpu_pingpong_cfg sm8450_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 11), - .intr_rdptr = -1, }, { .name = "pingpong_4", .id = PINGPONG_4, .base = 0x6d000, .len = 0, @@ -254,7 +250,6 @@ static const struct dpu_pingpong_cfg sm8450_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 30), - .intr_rdptr = -1, }, { .name = "pingpong_5", .id = PINGPONG_5, .base = 0x6e000, .len = 0, @@ -262,23 +257,18 @@ static const struct dpu_pingpong_cfg sm8450_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 31), - .intr_rdptr = -1, }, { .name = "pingpong_6", .id = PINGPONG_6, .base = 0x65800, .len = 0, .features = BIT(DPU_PINGPONG_DITHER), .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_3, - .intr_done = -1, - .intr_rdptr = -1, }, { .name = "pingpong_7", .id = PINGPONG_7, .base = 0x65c00, .len = 0, .features = BIT(DPU_PINGPONG_DITHER), .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_3, - .intr_done = -1, - .intr_rdptr = -1, }, }; @@ -352,7 +342,6 @@ static const struct dpu_intf_cfg sm8450_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x35000, .len = 0x300, @@ -382,7 +371,6 @@ static const struct dpu_intf_cfg sm8450_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 30), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 31), - .intr_tear_rd_ptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_9_0_sm8550.h b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_9_0_sm8550.h index 69b238ed01b9..ab8cdc1bdc09 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_9_0_sm8550.h +++ b/drivers/gpu/drm/msm/disp/dpu1/catalog/dpu_9_0_sm8550.h @@ -238,7 +238,6 @@ static const struct dpu_pingpong_cfg sm8550_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 8), - .intr_rdptr = -1, }, { .name = "pingpong_1", .id = PINGPONG_1, .base = 0x6a000, .len = 0, @@ -246,7 +245,6 @@ static const struct dpu_pingpong_cfg sm8550_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_0, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 9), - .intr_rdptr = -1, }, { .name = "pingpong_2", .id = PINGPONG_2, .base = 0x6b000, .len = 0, @@ -254,7 +252,6 @@ static const struct dpu_pingpong_cfg sm8550_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 10), - .intr_rdptr = -1, }, { .name = "pingpong_3", .id = PINGPONG_3, .base = 0x6c000, .len = 0, @@ -262,7 +259,6 @@ static const struct dpu_pingpong_cfg sm8550_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_1, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 11), - .intr_rdptr = -1, }, { .name = "pingpong_4", .id = PINGPONG_4, .base = 0x6d000, .len = 0, @@ -270,7 +266,6 @@ static const struct dpu_pingpong_cfg sm8550_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 30), - .intr_rdptr = -1, }, { .name = "pingpong_5", .id = PINGPONG_5, .base = 0x6e000, .len = 0, @@ -278,23 +273,18 @@ static const struct dpu_pingpong_cfg sm8550_pp[] = { .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_2, .intr_done = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR2, 31), - .intr_rdptr = -1, }, { .name = "pingpong_6", .id = PINGPONG_6, .base = 0x66000, .len = 0, .features = BIT(DPU_PINGPONG_DITHER), .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_3, - .intr_done = -1, - .intr_rdptr = -1, }, { .name = "pingpong_7", .id = PINGPONG_7, .base = 0x66400, .len = 0, .features = BIT(DPU_PINGPONG_DITHER), .sblk = &sc7280_pp_sblk, .merge_3d = MERGE_3D_3, - .intr_done = -1, - .intr_rdptr = -1, }, }; @@ -353,7 +343,6 @@ static const struct dpu_intf_cfg sm8550_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 24), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 25), - .intr_tear_rd_ptr = -1, }, { .name = "intf_1", .id = INTF_1, .base = 0x35000, .len = 0x300, @@ -383,7 +372,6 @@ static const struct dpu_intf_cfg sm8550_intf[] = { .prog_fetch_lines_worst_case = 24, .intr_underrun = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 30), .intr_vsync = DPU_IRQ_IDX(MDP_SSPP_TOP0_INTR, 31), - .intr_tear_rd_ptr = -1, }, }; diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_core_irq.h b/drivers/gpu/drm/msm/disp/dpu1/dpu_core_irq.h index ba06312cbb16..7c286bafb948 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/dpu_core_irq.h +++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_core_irq.h @@ -37,7 +37,7 @@ irqreturn_t dpu_core_irq(struct msm_kms *kms); */ u32 dpu_core_irq_read( struct dpu_kms *dpu_kms, - int irq_idx); + unsigned int irq_idx); /** * dpu_core_irq_register_callback - For registering callback function on IRQ @@ -52,7 +52,7 @@ u32 dpu_core_irq_read( */ int dpu_core_irq_register_callback( struct dpu_kms *dpu_kms, - int irq_idx, + unsigned int irq_idx, void (*irq_cb)(void *arg), void *irq_arg); @@ -67,7 +67,7 @@ int dpu_core_irq_register_callback( */ int dpu_core_irq_unregister_callback( struct dpu_kms *dpu_kms, - int irq_idx); + unsigned int irq_idx); /** * dpu_debugfs_core_irq_init - register core irq debugfs diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder.c index 99cccde5d221..dcf8127b9b6e 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder.c +++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder.c @@ -360,7 +360,7 @@ static int dpu_encoder_helper_wait_event_timeout(int32_t drm_id, u32 irq_idx, struct dpu_encoder_wait_info *info); int dpu_encoder_helper_wait_for_irq(struct dpu_encoder_phys *phys_enc, - int irq_idx, + unsigned int irq_idx, void (*func)(void *arg), struct dpu_encoder_wait_info *wait_info) { @@ -1551,7 +1551,7 @@ void dpu_encoder_helper_trigger_start(struct dpu_encoder_phys *phys_enc) static int dpu_encoder_helper_wait_event_timeout( int32_t drm_id, - u32 irq_idx, + unsigned int irq_idx, struct dpu_encoder_wait_info *info) { int rc = 0; @@ -2532,8 +2532,6 @@ unsigned int dpu_encoder_helper_get_dsc(struct dpu_encoder_phys *phys_enc) void dpu_encoder_phys_init(struct dpu_encoder_phys *phys_enc, struct dpu_enc_phys_init_params *p) { - int i; - phys_enc->hw_mdptop = p->dpu_kms->hw_mdp; phys_enc->hw_intf = p->hw_intf; phys_enc->hw_wb = p->hw_wb; @@ -2543,10 +2541,6 @@ void dpu_encoder_phys_init(struct dpu_encoder_phys *phys_enc, phys_enc->enc_spinlock = p->enc_spinlock; phys_enc->enable_state = DPU_ENC_DISABLED; - for (i = 0; i < ARRAY_SIZE(phys_enc->irq); i++) - phys_enc->irq[i] = -EINVAL; - - atomic_set(&phys_enc->vblank_refcount, 0); atomic_set(&phys_enc->pending_kickoff_cnt, 0); atomic_set(&phys_enc->pending_ctlstart_cnt, 0); diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys.h b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys.h index 57a3598f2a30..461e8c20cce8 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys.h +++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys.h @@ -154,6 +154,7 @@ enum dpu_intr_idx { * @hw_wb: Hardware interface to the wb registers * @dpu_kms: Pointer to the dpu_kms top level * @cached_mode: DRM mode cached at mode_set time, acted on in enable + * @vblank_ctl_lock: Vblank ctl mutex lock to protect vblank_refcount * @enabled: Whether the encoder has enabled and running a mode * @split_role: Role to play in a split-panel configuration * @intf_mode: Interface mode @@ -182,17 +183,18 @@ struct dpu_encoder_phys { struct dpu_hw_wb *hw_wb; struct dpu_kms *dpu_kms; struct drm_display_mode cached_mode; + struct mutex vblank_ctl_lock; enum dpu_enc_split_role split_role; enum dpu_intf_mode intf_mode; spinlock_t *enc_spinlock; enum dpu_enc_enable_state enable_state; - atomic_t vblank_refcount; + int vblank_refcount; atomic_t vsync_cnt; atomic_t underrun_cnt; atomic_t pending_ctlstart_cnt; atomic_t pending_kickoff_cnt; wait_queue_head_t pending_kickoff_wq; - int irq[INTR_IDX_MAX]; + unsigned int irq[INTR_IDX_MAX]; bool has_intf_te; }; @@ -365,7 +367,7 @@ void dpu_encoder_helper_report_irq_timeout(struct dpu_encoder_phys *phys_enc, * @Return: 0 or -ERROR */ int dpu_encoder_helper_wait_for_irq(struct dpu_encoder_phys *phys_enc, - int irq, + unsigned int irq, void (*func)(void *arg), struct dpu_encoder_wait_info *wait_info); diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_cmd.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_cmd.c index fd2400c4665d..6ec75f215d2e 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_cmd.c +++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_cmd.c @@ -252,7 +252,8 @@ static int dpu_encoder_phys_cmd_control_vblank_irq( return -EINVAL; } - refcount = atomic_read(&phys_enc->vblank_refcount); + mutex_lock(&phys_enc->vblank_ctl_lock); + refcount = phys_enc->vblank_refcount; /* Slave encoders don't report vblank */ if (!dpu_encoder_phys_cmd_is_master(phys_enc)) @@ -268,16 +269,24 @@ static int dpu_encoder_phys_cmd_control_vblank_irq( phys_enc->hw_pp->idx - PINGPONG_0, enable ? "true" : "false", refcount); - if (enable && atomic_inc_return(&phys_enc->vblank_refcount) == 1) - ret = dpu_core_irq_register_callback(phys_enc->dpu_kms, - phys_enc->irq[INTR_IDX_RDPTR], - dpu_encoder_phys_cmd_te_rd_ptr_irq, - phys_enc); - else if (!enable && atomic_dec_return(&phys_enc->vblank_refcount) == 0) - ret = dpu_core_irq_unregister_callback(phys_enc->dpu_kms, - phys_enc->irq[INTR_IDX_RDPTR]); + if (enable) { + if (phys_enc->vblank_refcount == 0) + ret = dpu_core_irq_register_callback(phys_enc->dpu_kms, + phys_enc->irq[INTR_IDX_RDPTR], + dpu_encoder_phys_cmd_te_rd_ptr_irq, + phys_enc); + if (!ret) + phys_enc->vblank_refcount++; + } else if (!enable) { + if (phys_enc->vblank_refcount == 1) + ret = dpu_core_irq_unregister_callback(phys_enc->dpu_kms, + phys_enc->irq[INTR_IDX_RDPTR]); + if (!ret) + phys_enc->vblank_refcount--; + } end: + mutex_unlock(&phys_enc->vblank_ctl_lock); if (ret) { DRM_ERROR("vblank irq err id:%u pp:%d ret:%d, enable %s/%d\n", DRMID(phys_enc->parent), @@ -293,7 +302,7 @@ static void dpu_encoder_phys_cmd_irq_control(struct dpu_encoder_phys *phys_enc, { trace_dpu_enc_phys_cmd_irq_ctrl(DRMID(phys_enc->parent), phys_enc->hw_pp->idx - PINGPONG_0, - enable, atomic_read(&phys_enc->vblank_refcount)); + enable, phys_enc->vblank_refcount); if (enable) { dpu_core_irq_register_callback(phys_enc->dpu_kms, @@ -735,6 +744,9 @@ struct dpu_encoder_phys *dpu_encoder_phys_cmd_init(struct drm_device *dev, dpu_encoder_phys_init(phys_enc, p); + mutex_init(&phys_enc->vblank_ctl_lock); + phys_enc->vblank_refcount = 0; + dpu_encoder_phys_cmd_init_ops(&phys_enc->ops); phys_enc->intf_mode = INTF_MODE_CMD; cmd_enc->stream_sel = 0; diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_vid.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_vid.c index 2cda9bbf68f9..da753f2dbd26 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_vid.c +++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_encoder_phys_vid.c @@ -373,7 +373,8 @@ static int dpu_encoder_phys_vid_control_vblank_irq( int ret = 0; int refcount; - refcount = atomic_read(&phys_enc->vblank_refcount); + mutex_lock(&phys_enc->vblank_ctl_lock); + refcount = phys_enc->vblank_refcount; /* Slave encoders don't report vblank */ if (!dpu_encoder_phys_vid_is_master(phys_enc)) @@ -386,18 +387,26 @@ static int dpu_encoder_phys_vid_control_vblank_irq( } DRM_DEBUG_VBL("id:%u enable=%d/%d\n", DRMID(phys_enc->parent), enable, - atomic_read(&phys_enc->vblank_refcount)); + refcount); - if (enable && atomic_inc_return(&phys_enc->vblank_refcount) == 1) - ret = dpu_core_irq_register_callback(phys_enc->dpu_kms, - phys_enc->irq[INTR_IDX_VSYNC], - dpu_encoder_phys_vid_vblank_irq, - phys_enc); - else if (!enable && atomic_dec_return(&phys_enc->vblank_refcount) == 0) - ret = dpu_core_irq_unregister_callback(phys_enc->dpu_kms, - phys_enc->irq[INTR_IDX_VSYNC]); + if (enable) { + if (phys_enc->vblank_refcount == 0) + ret = dpu_core_irq_register_callback(phys_enc->dpu_kms, + phys_enc->irq[INTR_IDX_VSYNC], + dpu_encoder_phys_vid_vblank_irq, + phys_enc); + if (!ret) + phys_enc->vblank_refcount++; + } else if (!enable) { + if (phys_enc->vblank_refcount == 1) + ret = dpu_core_irq_unregister_callback(phys_enc->dpu_kms, + phys_enc->irq[INTR_IDX_VSYNC]); + if (!ret) + phys_enc->vblank_refcount--; + } end: + mutex_unlock(&phys_enc->vblank_ctl_lock); if (ret) { DRM_ERROR("failed: id:%u intf:%d ret:%d enable:%d refcnt:%d\n", DRMID(phys_enc->parent), @@ -621,7 +630,7 @@ static void dpu_encoder_phys_vid_irq_control(struct dpu_encoder_phys *phys_enc, trace_dpu_enc_phys_vid_irq_ctrl(DRMID(phys_enc->parent), phys_enc->hw_intf->idx - INTF_0, enable, - atomic_read(&phys_enc->vblank_refcount)); + phys_enc->vblank_refcount); if (enable) { ret = dpu_encoder_phys_vid_control_vblank_irq(phys_enc, true); @@ -714,6 +723,8 @@ struct dpu_encoder_phys *dpu_encoder_phys_vid_init(struct drm_device *dev, DPU_DEBUG_VIDENC(phys_enc, "\n"); dpu_encoder_phys_init(phys_enc, p); + mutex_init(&phys_enc->vblank_ctl_lock); + phys_enc->vblank_refcount = 0; dpu_encoder_phys_vid_init_ops(&phys_enc->ops); phys_enc->intf_mode = INTF_MODE_VIDEO; diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_catalog.h b/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_catalog.h index 3f82d84bd1c9..5a33b5a42727 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_catalog.h +++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_catalog.h @@ -495,7 +495,7 @@ struct dpu_mdp_cfg { */ struct dpu_ctl_cfg { DPU_HW_BLK_INFO; - s32 intr_start; + unsigned int intr_start; }; /** @@ -558,8 +558,8 @@ struct dpu_dspp_cfg { struct dpu_pingpong_cfg { DPU_HW_BLK_INFO; u32 merge_3d; - s32 intr_done; - s32 intr_rdptr; + unsigned int intr_done; + unsigned int intr_rdptr; const struct dpu_pingpong_sub_blks *sblk; }; @@ -606,9 +606,9 @@ struct dpu_intf_cfg { u32 type; /* interface type*/ u32 controller_id; u32 prog_fetch_lines_worst_case; - s32 intr_underrun; - s32 intr_vsync; - s32 intr_tear_rd_ptr; + unsigned int intr_underrun; + unsigned int intr_vsync; + unsigned int intr_tear_rd_ptr; }; /** @@ -627,7 +627,7 @@ struct dpu_wb_cfg { u8 vbif_idx; u32 maxlinewidth; u32 xin_id; - s32 intr_wb_done; + unsigned int intr_wb_done; const u32 *format_list; u32 num_formats; enum dpu_clk_ctrl_type clk_ctrl; diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.c b/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.c index c8d7929ce523..6e1c4d684d2e 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.c +++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.c @@ -199,15 +199,15 @@ static const struct dpu_intr_reg dpu_intr_set_7xxx[] = { #define DPU_IRQ_MASK(irq_idx) (BIT(DPU_IRQ_BIT(irq_idx))) -static inline bool dpu_core_irq_is_valid(int irq_idx) +static inline bool dpu_core_irq_is_valid(unsigned int irq_idx) { - return irq_idx >= 0 && irq_idx < DPU_NUM_IRQS; + return irq_idx && irq_idx <= DPU_NUM_IRQS; } static inline struct dpu_hw_intr_entry *dpu_core_irq_get_entry(struct dpu_hw_intr *intr, - int irq_idx) + unsigned int irq_idx) { - return &intr->irq_tbl[irq_idx]; + return &intr->irq_tbl[irq_idx - 1]; } /** @@ -215,7 +215,7 @@ static inline struct dpu_hw_intr_entry *dpu_core_irq_get_entry(struct dpu_hw_int * @dpu_kms: Pointer to DPU's KMS structure * @irq_idx: interrupt index */ -static void dpu_core_irq_callback_handler(struct dpu_kms *dpu_kms, int irq_idx) +static void dpu_core_irq_callback_handler(struct dpu_kms *dpu_kms, unsigned int irq_idx) { struct dpu_hw_intr_entry *irq_entry = dpu_core_irq_get_entry(dpu_kms->hw_intr, irq_idx); @@ -240,7 +240,7 @@ irqreturn_t dpu_core_irq(struct msm_kms *kms) struct dpu_kms *dpu_kms = to_dpu_kms(kms); struct dpu_hw_intr *intr = dpu_kms->hw_intr; int reg_idx; - int irq_idx; + unsigned int irq_idx; u32 irq_status; u32 enable_mask; int bit; @@ -296,7 +296,8 @@ irqreturn_t dpu_core_irq(struct msm_kms *kms) return IRQ_HANDLED; } -static int dpu_hw_intr_enable_irq_locked(struct dpu_hw_intr *intr, int irq_idx) +static int dpu_hw_intr_enable_irq_locked(struct dpu_hw_intr *intr, + unsigned int irq_idx) { int reg_idx; const struct dpu_intr_reg *reg; @@ -351,7 +352,8 @@ static int dpu_hw_intr_enable_irq_locked(struct dpu_hw_intr *intr, int irq_idx) return 0; } -static int dpu_hw_intr_disable_irq_locked(struct dpu_hw_intr *intr, int irq_idx) +static int dpu_hw_intr_disable_irq_locked(struct dpu_hw_intr *intr, + unsigned int irq_idx) { int reg_idx; const struct dpu_intr_reg *reg; @@ -438,7 +440,8 @@ static void dpu_disable_all_irqs(struct dpu_kms *dpu_kms) wmb(); } -u32 dpu_core_irq_read(struct dpu_kms *dpu_kms, int irq_idx) +u32 dpu_core_irq_read(struct dpu_kms *dpu_kms, + unsigned int irq_idx) { struct dpu_hw_intr *intr = dpu_kms->hw_intr; int reg_idx; @@ -502,7 +505,7 @@ struct dpu_hw_intr *dpu_hw_intr_init(void __iomem *addr, intr->irq_mask |= BIT(MDP_INTFn_INTR(intf->id)); - if (intf->intr_tear_rd_ptr != -1) + if (intf->intr_tear_rd_ptr) intr->irq_mask |= BIT(DPU_IRQ_REG(intf->intr_tear_rd_ptr)); } @@ -516,9 +519,10 @@ void dpu_hw_intr_destroy(struct dpu_hw_intr *intr) kfree(intr); } -int dpu_core_irq_register_callback(struct dpu_kms *dpu_kms, int irq_idx, - void (*irq_cb)(void *arg), - void *irq_arg) +int dpu_core_irq_register_callback(struct dpu_kms *dpu_kms, + unsigned int irq_idx, + void (*irq_cb)(void *arg), + void *irq_arg) { struct dpu_hw_intr_entry *irq_entry; unsigned long irq_flags; @@ -565,7 +569,8 @@ int dpu_core_irq_register_callback(struct dpu_kms *dpu_kms, int irq_idx, return 0; } -int dpu_core_irq_unregister_callback(struct dpu_kms *dpu_kms, int irq_idx) +int dpu_core_irq_unregister_callback(struct dpu_kms *dpu_kms, + unsigned int irq_idx) { struct dpu_hw_intr_entry *irq_entry; unsigned long irq_flags; @@ -608,7 +613,7 @@ static int dpu_debugfs_core_irq_show(struct seq_file *s, void *v) int i, irq_count; void *cb; - for (i = 0; i < DPU_NUM_IRQS; i++) { + for (i = 1; i <= DPU_NUM_IRQS; i++) { spin_lock_irqsave(&dpu_kms->hw_intr->irq_lock, irq_flags); irq_entry = dpu_core_irq_get_entry(dpu_kms->hw_intr, i); irq_count = atomic_read(&irq_entry->count); @@ -644,7 +649,7 @@ void dpu_core_irq_preinstall(struct msm_kms *kms) dpu_disable_all_irqs(dpu_kms); pm_runtime_put_sync(&dpu_kms->pdev->dev); - for (i = 0; i < DPU_NUM_IRQS; i++) { + for (i = 1; i <= DPU_NUM_IRQS; i++) { irq_entry = dpu_core_irq_get_entry(dpu_kms->hw_intr, i); atomic_set(&irq_entry->count, 0); } @@ -660,7 +665,7 @@ void dpu_core_irq_uninstall(struct msm_kms *kms) return; pm_runtime_get_sync(&dpu_kms->pdev->dev); - for (i = 0; i < DPU_NUM_IRQS; i++) { + for (i = 1; i <= DPU_NUM_IRQS; i++) { irq_entry = dpu_core_irq_get_entry(dpu_kms->hw_intr, i); if (irq_entry->cb) DPU_ERROR("IRQ=[%d, %d] still enabled/registered\n", diff --git a/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.h b/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.h index 9df5d6e737a1..53a21ebc57e8 100644 --- a/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.h +++ b/drivers/gpu/drm/msm/disp/dpu1/dpu_hw_interrupts.h @@ -36,9 +36,9 @@ enum dpu_hw_intr_reg { #define MDP_INTFn_INTR(intf) (MDP_INTF0_INTR + (intf - INTF_0)) -#define DPU_IRQ_IDX(reg_idx, offset) (reg_idx * 32 + offset) -#define DPU_IRQ_REG(irq_idx) (irq_idx / 32) -#define DPU_IRQ_BIT(irq_idx) (irq_idx % 32) +#define DPU_IRQ_IDX(reg_idx, offset) (1 + reg_idx * 32 + offset) +#define DPU_IRQ_REG(irq_idx) ((irq_idx - 1) / 32) +#define DPU_IRQ_BIT(irq_idx) ((irq_idx - 1) % 32) #define DPU_NUM_IRQS (MDP_INTR_MAX * 32) diff --git a/drivers/gpu/drm/msm/msm_gpu.c b/drivers/gpu/drm/msm/msm_gpu.c index 5a7541597d0c..007b8b4e8139 100644 --- a/drivers/gpu/drm/msm/msm_gpu.c +++ b/drivers/gpu/drm/msm/msm_gpu.c @@ -418,25 +418,23 @@ static void recover_worker(struct kthread_work *work) msm_update_fence(ring->fctx, fence); } - if (msm_gpu_active(gpu)) { - /* retire completed submits, plus the one that hung: */ - retire_submits(gpu); + gpu->funcs->recover(gpu); - gpu->funcs->recover(gpu); + /* retire completed submits, plus the one that hung: */ + retire_submits(gpu); - /* - * Replay all remaining submits starting with highest priority - * ring - */ - for (i = 0; i < gpu->nr_rings; i++) { - struct msm_ringbuffer *ring = gpu->rb[i]; - unsigned long flags; + /* + * Replay all remaining submits starting with highest priority + * ring + */ + for (i = 0; i < gpu->nr_rings; i++) { + struct msm_ringbuffer *ring = gpu->rb[i]; + unsigned long flags; - spin_lock_irqsave(&ring->submit_lock, flags); - list_for_each_entry(submit, &ring->submits, node) - gpu->funcs->submit(gpu, submit); - spin_unlock_irqrestore(&ring->submit_lock, flags); - } + spin_lock_irqsave(&ring->submit_lock, flags); + list_for_each_entry(submit, &ring->submits, node) + gpu->funcs->submit(gpu, submit); + spin_unlock_irqrestore(&ring->submit_lock, flags); } pm_runtime_put(&gpu->pdev->dev); diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.c b/drivers/gpu/drm/msm/msm_ringbuffer.c index 7f5e0a961bba..ba9d51f47ffb 100644 --- a/drivers/gpu/drm/msm/msm_ringbuffer.c +++ b/drivers/gpu/drm/msm/msm_ringbuffer.c @@ -58,8 +58,13 @@ static const struct drm_sched_backend_ops msm_sched_ops = { struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, void *memptrs, uint64_t memptrs_iova) { + struct drm_sched_init_args args = { + .ops = &msm_sched_ops, + .hw_submission = num_hw_submissions, + .timeout = MAX_SCHEDULE_TIMEOUT, + .dev = gpu->dev->dev, + }; struct msm_ringbuffer *ring; - long sched_timeout; char name[32]; int ret; @@ -86,6 +91,7 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, } msm_gem_object_set_name(ring->bo, "ring%d", id); + args.name = to_msm_bo(ring->bo)->name; ring->end = ring->start + (MSM_GPU_RINGBUFFER_SZ >> 2); ring->next = ring->start; @@ -94,15 +100,10 @@ struct msm_ringbuffer *msm_ringbuffer_new(struct msm_gpu *gpu, int id, ring->memptrs = memptrs; ring->memptrs_iova = memptrs_iova; - /* currently managing hangcheck ourselves: */ - sched_timeout = MAX_SCHEDULE_TIMEOUT; - - ret = drm_sched_init(&ring->sched, &msm_sched_ops, - num_hw_submissions, 0, sched_timeout, - NULL, NULL, to_msm_bo(ring->bo)->name, gpu->dev->dev); - if (ret) { + ret = drm_sched_init(&ring->sched, &args); + if (ret) goto fail; - } + ring->sched_initialized = true; INIT_LIST_HEAD(&ring->submits); spin_lock_init(&ring->submit_lock); @@ -124,7 +125,8 @@ void msm_ringbuffer_destroy(struct msm_ringbuffer *ring) if (IS_ERR_OR_NULL(ring)) return; - drm_sched_fini(&ring->sched); + if (ring->sched_initialized) + drm_sched_fini(&ring->sched); msm_fence_context_free(ring->fctx); diff --git a/drivers/gpu/drm/msm/msm_ringbuffer.h b/drivers/gpu/drm/msm/msm_ringbuffer.h index 698b333abccd..eafaa7cef324 100644 --- a/drivers/gpu/drm/msm/msm_ringbuffer.h +++ b/drivers/gpu/drm/msm/msm_ringbuffer.h @@ -50,6 +50,7 @@ struct msm_ringbuffer { * The job scheduler for this ring. */ struct drm_gpu_scheduler sched; + bool sched_initialized; /* * List of in-flight submits on this ring. Protected by submit_lock. diff --git a/drivers/gpu/drm/nouveau/dispnv04/crtc.c b/drivers/gpu/drm/nouveau/dispnv04/crtc.c index 8c7fff19c97b..4c846276814c 100644 --- a/drivers/gpu/drm/nouveau/dispnv04/crtc.c +++ b/drivers/gpu/drm/nouveau/dispnv04/crtc.c @@ -1042,7 +1042,7 @@ nv04_finish_page_flip(struct nouveau_channel *chan, struct nv04_page_flip_state *ps) { struct nouveau_fence_chan *fctx = chan->fence; - struct nouveau_drm *drm = chan->drm; + struct nouveau_drm *drm = chan->cli->drm; struct drm_device *dev = drm->dev; struct nv04_page_flip_state *s; unsigned long flags; @@ -1098,7 +1098,7 @@ nv04_page_flip_emit(struct nouveau_channel *chan, struct nouveau_fence **pfence) { struct nouveau_fence_chan *fctx = chan->fence; - struct nouveau_drm *drm = chan->drm; + struct nouveau_drm *drm = chan->cli->drm; struct drm_device *dev = drm->dev; struct nvif_push *push = chan->chan.push; unsigned long flags; diff --git a/drivers/gpu/drm/nouveau/nouveau_bo0039.c b/drivers/gpu/drm/nouveau/nouveau_bo0039.c index e2ce44adaa5c..2babc6c47241 100644 --- a/drivers/gpu/drm/nouveau/nouveau_bo0039.c +++ b/drivers/gpu/drm/nouveau/nouveau_bo0039.c @@ -104,6 +104,6 @@ nv04_bo_move_init(struct nouveau_channel *chan, u32 handle) return ret; PUSH_MTHD(push, NV039, SET_OBJECT, handle); - PUSH_MTHD(push, NV039, SET_CONTEXT_DMA_NOTIFIES, chan->drm->ntfy.handle); + PUSH_MTHD(push, NV039, SET_CONTEXT_DMA_NOTIFIES, chan->cli->drm->ntfy.handle); return 0; } diff --git a/drivers/gpu/drm/nouveau/nouveau_bo5039.c b/drivers/gpu/drm/nouveau/nouveau_bo5039.c index c6cf3629a9f9..0a6b1fce1108 100644 --- a/drivers/gpu/drm/nouveau/nouveau_bo5039.c +++ b/drivers/gpu/drm/nouveau/nouveau_bo5039.c @@ -144,7 +144,7 @@ nv50_bo_move_init(struct nouveau_channel *chan, u32 handle) return ret; PUSH_MTHD(push, NV5039, SET_OBJECT, handle); - PUSH_MTHD(push, NV5039, SET_CONTEXT_DMA_NOTIFY, chan->drm->ntfy.handle, + PUSH_MTHD(push, NV5039, SET_CONTEXT_DMA_NOTIFY, chan->cli->drm->ntfy.handle, SET_CONTEXT_DMA_BUFFER_IN, chan->vram.handle, SET_CONTEXT_DMA_BUFFER_OUT, chan->vram.handle); return 0; diff --git a/drivers/gpu/drm/nouveau/nouveau_chan.c b/drivers/gpu/drm/nouveau/nouveau_chan.c index cee36b1efd39..c6ee2bfe2787 100644 --- a/drivers/gpu/drm/nouveau/nouveau_chan.c +++ b/drivers/gpu/drm/nouveau/nouveau_chan.c @@ -90,19 +90,24 @@ nouveau_channel_del(struct nouveau_channel **pchan) { struct nouveau_channel *chan = *pchan; if (chan) { - struct nouveau_cli *cli = (void *)chan->user.client; + /* + * Drop the kill-event subscription first. Its handler + * dereferences chan->fence, which the fence context teardown + * below frees, so leaving it armed across the teardown leaves + * a window for a use-after-free. + */ + nvif_event_dtor(&chan->kill); if (chan->fence) - nouveau_fence(chan->drm)->context_del(chan); + nouveau_fence(chan->cli->drm)->context_del(chan); - if (cli) + if (nvif_object_constructed(&chan->user)) nouveau_svmm_part(chan->vmm->svmm, chan->inst); nvif_object_dtor(&chan->blit); nvif_object_dtor(&chan->nvsw); nvif_object_dtor(&chan->gart); nvif_object_dtor(&chan->vram); - nvif_event_dtor(&chan->kill); nvif_object_dtor(&chan->user); nvif_mem_dtor(&chan->mem_userd); nvif_object_dtor(&chan->push.ctxdma); @@ -157,8 +162,6 @@ nouveau_channel_prep(struct nouveau_cli *cli, return -ENOMEM; chan->cli = cli; - chan->device = device; - chan->drm = drm; chan->vmm = nouveau_cli_vmm(cli); atomic_set(&chan->killed, 0); @@ -230,12 +233,11 @@ nouveau_channel_prep(struct nouveau_cli *cli, args.limit = device->info.ram_user - 1; } } else { - if (chan->drm->agp.bridge) { + if (drm->agp.bridge) { args.target = NV_DMA_V0_TARGET_AGP; args.access = NV_DMA_V0_ACCESS_RDWR; - args.start = chan->drm->agp.base; - args.limit = chan->drm->agp.base + - chan->drm->agp.size - 1; + args.start = drm->agp.base; + args.limit = drm->agp.base + drm->agp.size - 1; } else { args.target = NV_DMA_V0_TARGET_VM; args.access = NV_DMA_V0_ACCESS_RDWR; @@ -365,8 +367,9 @@ nouveau_channel_ctor(struct nouveau_cli *cli, bool priv, u64 runm, static int nouveau_channel_init(struct nouveau_channel *chan, u32 vram, u32 gart) { - struct nvif_device *device = chan->device; - struct nouveau_drm *drm = chan->drm; + struct nouveau_cli *cli = chan->cli; + struct nouveau_drm *drm = cli->drm; + struct nvif_device *device = &cli->device; struct nv_dma_v0 args = {}; int ret, i; @@ -421,12 +424,11 @@ nouveau_channel_init(struct nouveau_channel *chan, u32 vram, u32 gart) args.start = 0; args.limit = chan->vmm->vmm.limit - 1; } else - if (chan->drm->agp.bridge) { + if (drm->agp.bridge) { args.target = NV_DMA_V0_TARGET_AGP; args.access = NV_DMA_V0_ACCESS_RDWR; - args.start = chan->drm->agp.base; - args.limit = chan->drm->agp.base + - chan->drm->agp.size - 1; + args.start = drm->agp.base; + args.limit = drm->agp.base + drm->agp.size - 1; } else { args.target = NV_DMA_V0_TARGET_VM; args.access = NV_DMA_V0_ACCESS_RDWR; @@ -491,7 +493,7 @@ nouveau_channel_init(struct nouveau_channel *chan, u32 vram, u32 gart) } /* initialise synchronisation */ - return nouveau_fence(chan->drm)->context_new(chan); + return nouveau_fence(drm)->context_new(chan); } int diff --git a/drivers/gpu/drm/nouveau/nouveau_chan.h b/drivers/gpu/drm/nouveau/nouveau_chan.h index 260febd634ee..3ce9832c9528 100644 --- a/drivers/gpu/drm/nouveau/nouveau_chan.h +++ b/drivers/gpu/drm/nouveau/nouveau_chan.h @@ -13,8 +13,6 @@ struct nouveau_channel { } chan; struct nouveau_cli *cli; - struct nvif_device *device; - struct nouveau_drm *drm; struct nouveau_vmm *vmm; struct nvif_mem mem_userd; diff --git a/drivers/gpu/drm/nouveau/nouveau_dma.c b/drivers/gpu/drm/nouveau/nouveau_dma.c index b01c029f3a90..a1f329ef0641 100644 --- a/drivers/gpu/drm/nouveau/nouveau_dma.c +++ b/drivers/gpu/drm/nouveau/nouveau_dma.c @@ -72,7 +72,7 @@ void nv50_dma_push(struct nouveau_channel *chan, u64 offset, u32 length, bool no_prefetch) { - struct nvif_user *user = &chan->drm->client.device.user; + struct nvif_user *user = &chan->cli->drm->client.device.user; struct nouveau_bo *pb = chan->push.buffer; int ip = (chan->dma.ib_put * 2) + chan->dma.ib_base; diff --git a/drivers/gpu/drm/nouveau/nouveau_exec.c b/drivers/gpu/drm/nouveau/nouveau_exec.c index 5eb41aefc2af..9cc99dc5bd91 100644 --- a/drivers/gpu/drm/nouveau/nouveau_exec.c +++ b/drivers/gpu/drm/nouveau/nouveau_exec.c @@ -107,7 +107,7 @@ nouveau_exec_job_submit(struct nouveau_job *job) drm_exec_until_all_locked(exec) { struct drm_gpuva *va; - drm_gpuva_for_each_va(va, &uvmm->umgr) { + drm_gpuvm_for_each_va(va, &uvmm->umgr) { if (unlikely(va == &uvmm->umgr.kernel_alloc_node)) continue; diff --git a/drivers/gpu/drm/nouveau/nouveau_fence.c b/drivers/gpu/drm/nouveau/nouveau_fence.c index 1bfa312d6fb8..ee43a36f17bd 100644 --- a/drivers/gpu/drm/nouveau/nouveau_fence.c +++ b/drivers/gpu/drm/nouveau/nouveau_fence.c @@ -181,8 +181,9 @@ nouveau_fence_wait_uevent_handler(struct nvif_event *event, void *repv, u32 repc void nouveau_fence_context_new(struct nouveau_channel *chan, struct nouveau_fence_chan *fctx) { - struct nouveau_fence_priv *priv = (void*)chan->drm->fence; - struct nouveau_cli *cli = (void *)chan->user.client; + struct nouveau_cli *cli = chan->cli; + struct nouveau_drm *drm = cli->drm; + struct nouveau_fence_priv *priv = (void*)drm->fence; struct { struct nvif_event_v0 base; struct nvif_chan_event_v0 host; @@ -193,11 +194,11 @@ nouveau_fence_context_new(struct nouveau_channel *chan, struct nouveau_fence_cha INIT_LIST_HEAD(&fctx->flip); INIT_LIST_HEAD(&fctx->pending); spin_lock_init(&fctx->lock); - fctx->context = chan->drm->runl[chan->runlist].context_base + chan->chid; + fctx->context = drm->runl[chan->runlist].context_base + chan->chid; - if (chan == chan->drm->cechan) + if (chan == drm->cechan) strcpy(fctx->name, "copy engine channel"); - else if (chan == chan->drm->channel) + else if (chan == drm->channel) strcpy(fctx->name, "generic kernel channel"); else strcpy(fctx->name, nvxx_client(&cli->base)->name); @@ -221,7 +222,7 @@ nouveau_fence_emit(struct nouveau_fence *fence) { struct nouveau_channel *chan = unrcu_pointer(fence->channel); struct nouveau_fence_chan *fctx = chan->fence; - struct nouveau_fence_priv *priv = (void*)chan->drm->fence; + struct nouveau_fence_priv *priv = (void*)chan->cli->drm->fence; int ret; fence->timeout = jiffies + (15 * HZ); @@ -382,7 +383,7 @@ nouveau_fence_sync(struct nouveau_bo *nvbo, struct nouveau_channel *chan, if (i == 0 && usage == DMA_RESV_USAGE_WRITE) continue; - f = nouveau_local_fence(fence, chan->drm); + f = nouveau_local_fence(fence, chan->cli->drm); if (f) { struct nouveau_channel *prev; bool must_wait = true; diff --git a/drivers/gpu/drm/nouveau/nouveau_gem.c b/drivers/gpu/drm/nouveau/nouveau_gem.c index 29b0884ff419..c0efbb6085fe 100644 --- a/drivers/gpu/drm/nouveau/nouveau_gem.c +++ b/drivers/gpu/drm/nouveau/nouveau_gem.c @@ -569,10 +569,11 @@ validate_init(struct nouveau_channel *chan, struct drm_file *file_priv, } static int -validate_list(struct nouveau_channel *chan, struct nouveau_cli *cli, +validate_list(struct nouveau_channel *chan, struct list_head *list, struct drm_nouveau_gem_pushbuf_bo *pbbo) { - struct nouveau_drm *drm = chan->drm; + struct nouveau_cli *cli = chan->cli; + struct nouveau_drm *drm = cli->drm; struct nouveau_bo *nvbo; int ret, relocs = 0; @@ -644,7 +645,7 @@ nouveau_gem_pushbuf_validate(struct nouveau_channel *chan, return ret; } - ret = validate_list(chan, cli, &op->list, pbbo); + ret = validate_list(chan, &op->list, pbbo); if (unlikely(ret < 0)) { if (ret != -ERESTARTSYS) NV_PRINTK(err, cli, "validating bo list\n"); diff --git a/drivers/gpu/drm/nouveau/nouveau_sched.c b/drivers/gpu/drm/nouveau/nouveau_sched.c index 3b7ea5221226..9f80d9e81fb3 100644 --- a/drivers/gpu/drm/nouveau/nouveau_sched.c +++ b/drivers/gpu/drm/nouveau/nouveau_sched.c @@ -429,15 +429,19 @@ static const struct drm_sched_backend_ops nouveau_sched_ops = { int nouveau_sched_init(struct nouveau_drm *drm) { struct drm_gpu_scheduler *sched = &drm->sched; - long job_hang_limit = msecs_to_jiffies(NOUVEAU_SCHED_JOB_TIMEOUT_MS); + struct drm_sched_init_args args = { + .ops = &nouveau_sched_ops, + .hw_submission = NOUVEAU_SCHED_HW_SUBMISSIONS, + .timeout = msecs_to_jiffies(NOUVEAU_SCHED_JOB_TIMEOUT_MS), + .name = "nouveau_sched", + .dev = drm->dev->dev, + }; drm->sched_wq = create_singlethread_workqueue("nouveau_sched_wq"); if (!drm->sched_wq) return -ENOMEM; - return drm_sched_init(sched, &nouveau_sched_ops, - NOUVEAU_SCHED_HW_SUBMISSIONS, 0, job_hang_limit, - NULL, NULL, "nouveau_sched", drm->dev->dev); + return drm_sched_init(sched, &args); } void nouveau_sched_fini(struct nouveau_drm *drm) diff --git a/drivers/gpu/drm/nouveau/nouveau_uvmm.c b/drivers/gpu/drm/nouveau/nouveau_uvmm.c index 1310ee5f4973..ab2c2244eb54 100644 --- a/drivers/gpu/drm/nouveau/nouveau_uvmm.c +++ b/drivers/gpu/drm/nouveau/nouveau_uvmm.c @@ -329,7 +329,7 @@ nouveau_uvma_region_create(struct nouveau_uvmm *uvmm, struct nouveau_uvma_region *reg; int ret; - if (!drm_gpuva_interval_empty(&uvmm->umgr, addr, range)) + if (!drm_gpuvm_interval_empty(&uvmm->umgr, addr, range)) return -ENOSPC; ret = nouveau_uvma_region_alloc(®); @@ -384,7 +384,7 @@ nouveau_uvma_region_empty(struct nouveau_uvma_region *reg) { struct nouveau_uvmm *uvmm = reg->uvmm; - return drm_gpuva_interval_empty(&uvmm->umgr, + return drm_gpuvm_interval_empty(&uvmm->umgr, reg->va.addr, reg->va.range); } @@ -444,7 +444,7 @@ op_map_prepare_unwind(struct nouveau_uvma *uvma) static void op_unmap_prepare_unwind(struct drm_gpuva *va) { - drm_gpuva_insert(va->mgr, va); + drm_gpuva_insert(va->vm, va); } static void @@ -772,6 +772,9 @@ op_map(struct nouveau_uvma *uvma) { struct nouveau_bo *nvbo = nouveau_gem_object(uvma->va.gem.obj); + if (drm_gpuva_invalidated(&uvma->va)) + return; + nouveau_uvma_map(uvma, nouveau_mem(nvbo->bo.resource)); } @@ -1195,7 +1198,7 @@ nouveau_uvmm_bind_job_submit(struct nouveau_job *job) goto unwind_continue; } - op->ops = drm_gpuva_sm_unmap_ops_create(&uvmm->umgr, + op->ops = drm_gpuvm_sm_unmap_ops_create(&uvmm->umgr, op->va.addr, op->va.range); if (IS_ERR(op->ops)) { @@ -1242,7 +1245,7 @@ nouveau_uvmm_bind_job_submit(struct nouveau_job *job) } } - op->ops = drm_gpuva_sm_map_ops_create(&uvmm->umgr, + op->ops = drm_gpuvm_sm_map_ops_create(&uvmm->umgr, op->va.addr, op->va.range, op->gem.obj, @@ -1266,7 +1269,7 @@ nouveau_uvmm_bind_job_submit(struct nouveau_job *job) break; } case OP_UNMAP: - op->ops = drm_gpuva_sm_unmap_ops_create(&uvmm->umgr, + op->ops = drm_gpuvm_sm_unmap_ops_create(&uvmm->umgr, op->va.addr, op->va.range); if (IS_ERR(op->ops)) { @@ -1334,6 +1337,13 @@ nouveau_uvmm_bind_job_submit(struct nouveau_job *job) continue; nvbo = nouveau_gem_object(obj); + if (!(nvbo->valid_domains & + (NOUVEAU_GEM_DOMAIN_VRAM | NOUVEAU_GEM_DOMAIN_GART))) { + ret = -EINVAL; + op = list_last_op(&bind_job->ops); + goto unwind; + } + nouveau_bo_placement_set(nvbo, nvbo->valid_domains, 0); ret = nouveau_bo_validate(nvbo, true, false); if (ret) { @@ -1409,7 +1419,8 @@ nouveau_uvmm_bind_job_submit(struct nouveau_job *job) break; } - drm_gpuva_ops_free(&uvmm->umgr, op->ops); + if (!IS_ERR_OR_NULL(op->ops)) + drm_gpuva_ops_free(&uvmm->umgr, op->ops); op->ops = NULL; op->reg = NULL; } @@ -1841,11 +1852,11 @@ nouveau_uvmm_init(struct nouveau_uvmm *uvmm, struct nouveau_cli *cli, uvmm->kernel_managed_addr = kernel_managed_addr; uvmm->kernel_managed_size = kernel_managed_size; - drm_gpuva_manager_init(&uvmm->umgr, cli->name, - NOUVEAU_VA_SPACE_START, - NOUVEAU_VA_SPACE_END, - kernel_managed_addr, kernel_managed_size, - NULL); + drm_gpuvm_init(&uvmm->umgr, cli->name, + NOUVEAU_VA_SPACE_START, + NOUVEAU_VA_SPACE_END, + kernel_managed_addr, kernel_managed_size, + NULL); ret = nvif_vmm_ctor(&cli->mmu, "uvmm", cli->vmm.vmm.object.oclass, RAW, @@ -1860,7 +1871,7 @@ nouveau_uvmm_init(struct nouveau_uvmm *uvmm, struct nouveau_cli *cli, return 0; out_free_gpuva_mgr: - drm_gpuva_manager_destroy(&uvmm->umgr); + drm_gpuvm_destroy(&uvmm->umgr); out_unlock: mutex_unlock(&cli->mutex); return ret; @@ -1882,7 +1893,7 @@ nouveau_uvmm_fini(struct nouveau_uvmm *uvmm) wait_event(entity->job.wq, list_empty(&entity->job.list.head)); nouveau_uvmm_lock(uvmm); - drm_gpuva_for_each_va_safe(va, next, &uvmm->umgr) { + drm_gpuvm_for_each_va_safe(va, next, &uvmm->umgr) { struct nouveau_uvma *uvma = uvma_from_va(va); struct drm_gem_object *obj = va->gem.obj; @@ -1915,7 +1926,7 @@ nouveau_uvmm_fini(struct nouveau_uvmm *uvmm) mutex_lock(&cli->mutex); nouveau_vmm_fini(&uvmm->vmm); - drm_gpuva_manager_destroy(&uvmm->umgr); + drm_gpuvm_destroy(&uvmm->umgr); mutex_unlock(&cli->mutex); dma_resv_fini(&uvmm->resv); diff --git a/drivers/gpu/drm/nouveau/nouveau_uvmm.h b/drivers/gpu/drm/nouveau/nouveau_uvmm.h index fc7f6fd2a4e1..e96c9919d1bd 100644 --- a/drivers/gpu/drm/nouveau/nouveau_uvmm.h +++ b/drivers/gpu/drm/nouveau/nouveau_uvmm.h @@ -3,13 +3,13 @@ #ifndef __NOUVEAU_UVMM_H__ #define __NOUVEAU_UVMM_H__ -#include +#include #include "nouveau_drv.h" struct nouveau_uvmm { struct nouveau_vmm vmm; - struct drm_gpuva_manager umgr; + struct drm_gpuvm umgr; struct maple_tree region_mt; struct mutex mutex; struct dma_resv resv; @@ -44,7 +44,7 @@ struct nouveau_uvma { #define uvmm_from_mgr(x) container_of((x), struct nouveau_uvmm, umgr) #define uvma_from_va(x) container_of((x), struct nouveau_uvma, va) -#define to_uvmm(x) uvmm_from_mgr((x)->va.mgr) +#define to_uvmm(x) uvmm_from_mgr((x)->va.vm) struct nouveau_uvmm_bind_job { struct nouveau_job base; diff --git a/drivers/gpu/drm/nouveau/nv17_fence.c b/drivers/gpu/drm/nouveau/nv17_fence.c index 07c2e0878c24..7fa52ec61b30 100644 --- a/drivers/gpu/drm/nouveau/nv17_fence.c +++ b/drivers/gpu/drm/nouveau/nv17_fence.c @@ -36,8 +36,8 @@ int nv17_fence_sync(struct nouveau_fence *fence, struct nouveau_channel *prev, struct nouveau_channel *chan) { - struct nouveau_cli *cli = (void *)prev->user.client; - struct nv10_fence_priv *priv = chan->drm->fence; + struct nouveau_cli *cli = prev->cli; + struct nv10_fence_priv *priv = cli->drm->fence; struct nv10_fence_chan *fctx = chan->fence; struct nvif_push *ppush = prev->chan.push; struct nvif_push *npush = chan->chan.push; @@ -76,7 +76,7 @@ nv17_fence_sync(struct nouveau_fence *fence, static int nv17_fence_context_new(struct nouveau_channel *chan) { - struct nv10_fence_priv *priv = chan->drm->fence; + struct nv10_fence_priv *priv = chan->cli->drm->fence; struct ttm_resource *reg = priv->bo->bo.resource; struct nv10_fence_chan *fctx; u32 start = reg->start * PAGE_SIZE; diff --git a/drivers/gpu/drm/nouveau/nv50_fence.c b/drivers/gpu/drm/nouveau/nv50_fence.c index ea1e1f480bfe..8b76cb227f55 100644 --- a/drivers/gpu/drm/nouveau/nv50_fence.c +++ b/drivers/gpu/drm/nouveau/nv50_fence.c @@ -35,7 +35,7 @@ static int nv50_fence_context_new(struct nouveau_channel *chan) { - struct nv10_fence_priv *priv = chan->drm->fence; + struct nv10_fence_priv *priv = chan->cli->drm->fence; struct nv10_fence_chan *fctx; struct ttm_resource *reg = priv->bo->bo.resource; u32 start = reg->start * PAGE_SIZE; diff --git a/drivers/gpu/drm/nouveau/nv84_fence.c b/drivers/gpu/drm/nouveau/nv84_fence.c index 812b8c62eeba..7f62089dc940 100644 --- a/drivers/gpu/drm/nouveau/nv84_fence.c +++ b/drivers/gpu/drm/nouveau/nv84_fence.c @@ -79,7 +79,7 @@ nv84_fence_sync32(struct nouveau_channel *chan, u64 virtual, u32 sequence) static inline u32 nv84_fence_chid(struct nouveau_channel *chan) { - return chan->drm->runl[chan->runlist].chan_id_base + chan->chid; + return chan->cli->drm->runl[chan->runlist].chan_id_base + chan->chid; } static int @@ -105,14 +105,14 @@ nv84_fence_sync(struct nouveau_fence *fence, static u32 nv84_fence_read(struct nouveau_channel *chan) { - struct nv84_fence_priv *priv = chan->drm->fence; + struct nv84_fence_priv *priv = chan->cli->drm->fence; return nouveau_bo_rd32(priv->bo, nv84_fence_chid(chan) * 16/4); } static void nv84_fence_context_del(struct nouveau_channel *chan) { - struct nv84_fence_priv *priv = chan->drm->fence; + struct nv84_fence_priv *priv = chan->cli->drm->fence; struct nv84_fence_chan *fctx = chan->fence; nouveau_bo_wr32(priv->bo, nv84_fence_chid(chan) * 16 / 4, fctx->base.sequence); @@ -127,7 +127,7 @@ nv84_fence_context_del(struct nouveau_channel *chan) int nv84_fence_context_new(struct nouveau_channel *chan) { - struct nv84_fence_priv *priv = chan->drm->fence; + struct nv84_fence_priv *priv = chan->cli->drm->fence; struct nv84_fence_chan *fctx; int ret; diff --git a/drivers/gpu/drm/panfrost/panfrost_job.c b/drivers/gpu/drm/panfrost/panfrost_job.c index a8b4827dc425..0e70a64356ee 100644 --- a/drivers/gpu/drm/panfrost/panfrost_job.c +++ b/drivers/gpu/drm/panfrost/panfrost_job.c @@ -791,6 +791,12 @@ static irqreturn_t panfrost_job_irq_handler(int irq, void *data) int panfrost_job_init(struct panfrost_device *pfdev) { + struct drm_sched_init_args args = { + .ops = &panfrost_sched_ops, + .timeout = msecs_to_jiffies(JOB_TIMEOUT_MS), + .name = "pan_js", + .dev = pfdev->dev, + }; struct panfrost_job_slot *js; unsigned int nentries = 2; int ret, j; @@ -826,16 +832,13 @@ int panfrost_job_init(struct panfrost_device *pfdev) pfdev->reset.wq = alloc_ordered_workqueue("panfrost-reset", 0); if (!pfdev->reset.wq) return -ENOMEM; + args.timeout_wq = pfdev->reset.wq; + args.hw_submission = nentries; for (j = 0; j < NUM_JOB_SLOTS; j++) { js->queue[j].fence_context = dma_fence_context_alloc(1); - ret = drm_sched_init(&js->queue[j].sched, - &panfrost_sched_ops, - nentries, 0, - msecs_to_jiffies(JOB_TIMEOUT_MS), - pfdev->reset.wq, - NULL, "pan_js", pfdev->dev); + ret = drm_sched_init(&js->queue[j].sched, &args); if (ret) { dev_err(pfdev->dev, "Failed to create scheduler: %d.", ret); goto err_sched; diff --git a/drivers/gpu/drm/radeon/radeon_bios.c b/drivers/gpu/drm/radeon/radeon_bios.c index 63bdc9f6fc24..1fecb46d421f 100644 --- a/drivers/gpu/drm/radeon/radeon_bios.c +++ b/drivers/gpu/drm/radeon/radeon_bios.c @@ -49,6 +49,7 @@ static bool igp_read_bios_from_vram(struct radeon_device *rdev) { uint8_t __iomem *bios; + uint8_t val1, val2; resource_size_t vram_base; resource_size_t size = 256 * 1024; /* ??? */ @@ -63,7 +64,10 @@ static bool igp_read_bios_from_vram(struct radeon_device *rdev) return false; } - if (size == 0 || bios[0] != 0x55 || bios[1] != 0xaa) { + val1 = readb(&bios[0]); + val2 = readb(&bios[1]); + + if (size == 0 || val1 != 0x55 || val2 != 0xaa) { iounmap(bios); return false; } diff --git a/drivers/gpu/drm/renesas/rz-du/rzg2l_mipi_dsi.c b/drivers/gpu/drm/renesas/rz-du/rzg2l_mipi_dsi.c index a9216a99b2be..5b8da2b6cbd0 100644 --- a/drivers/gpu/drm/renesas/rz-du/rzg2l_mipi_dsi.c +++ b/drivers/gpu/drm/renesas/rz-du/rzg2l_mipi_dsi.c @@ -574,6 +574,13 @@ static void rzg2l_mipi_dsi_atomic_disable(struct drm_bridge *bridge, rzg2l_mipi_dsi_stop_video(dsi); rzg2l_mipi_dsi_stop_hs_clock(dsi); +} + +static void rzg2l_mipi_dsi_atomic_post_disable(struct drm_bridge *bridge, + struct drm_bridge_state *old_bridge_state) +{ + struct rzg2l_mipi_dsi *dsi = bridge_to_rzg2l_mipi_dsi(bridge); + rzg2l_mipi_dsi_stop(dsi); } @@ -598,6 +605,7 @@ static const struct drm_bridge_funcs rzg2l_mipi_dsi_bridge_ops = { .atomic_reset = drm_atomic_helper_bridge_reset, .atomic_enable = rzg2l_mipi_dsi_atomic_enable, .atomic_disable = rzg2l_mipi_dsi_atomic_disable, + .atomic_post_disable = rzg2l_mipi_dsi_atomic_post_disable, .mode_valid = rzg2l_mipi_dsi_bridge_mode_valid, }; diff --git a/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c b/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c index bd5bce9d8c2c..89c914bcfbf1 100644 --- a/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c +++ b/drivers/gpu/drm/rockchip/rockchip_drm_vop2.c @@ -2412,6 +2412,10 @@ static int vop2_create_crtcs(struct vop2 *vop2) if (!vp->crtc.port) continue; + if (!vp->primary_plane) + return dev_err_probe(drm->dev, -ENOENT, + "no primary plane for vp %d\n", i); + plane = &vp->primary_plane->base; ret = drm_crtc_init_with_planes(drm, &vp->crtc, plane, NULL, diff --git a/drivers/gpu/drm/scheduler/sched_main.c b/drivers/gpu/drm/scheduler/sched_main.c index 50716bc5eef6..a18693303c22 100644 --- a/drivers/gpu/drm/scheduler/sched_main.c +++ b/drivers/gpu/drm/scheduler/sched_main.c @@ -1080,33 +1080,21 @@ static int drm_sched_main(void *param) * drm_sched_init - Init a gpu scheduler instance * * @sched: scheduler instance - * @ops: backend operations for this scheduler - * @hw_submission: number of hw submissions that can be in flight - * @hang_limit: number of times to allow a job to hang before dropping it - * @timeout: timeout value in jiffies for the scheduler - * @timeout_wq: workqueue to use for timeout work. If NULL, the system_wq is - * used - * @score: optional score atomic shared with other schedulers - * @name: name used for debugging - * @dev: target &struct device + * @args: scheduler initialization arguments * * Return 0 on success, otherwise error code. */ -int drm_sched_init(struct drm_gpu_scheduler *sched, - const struct drm_sched_backend_ops *ops, - unsigned hw_submission, unsigned hang_limit, - long timeout, struct workqueue_struct *timeout_wq, - atomic_t *score, const char *name, struct device *dev) +int drm_sched_init(struct drm_gpu_scheduler *sched, const struct drm_sched_init_args *args) { int i, ret; - sched->ops = ops; - sched->hw_submission_limit = hw_submission; - sched->name = name; - sched->timeout = timeout; - sched->timeout_wq = timeout_wq ? : system_wq; - sched->hang_limit = hang_limit; - sched->score = score ? score : &sched->_score; - sched->dev = dev; + sched->ops = args->ops; + sched->hw_submission_limit = args->hw_submission; + sched->name = args->name; + sched->timeout = args->timeout; + sched->timeout_wq = args->timeout_wq ? args->timeout_wq : system_wq; + sched->hang_limit = args->hang_limit; + sched->score = args->score ? args->score : &sched->_score; + sched->dev = args->dev; for (i = DRM_SCHED_PRIORITY_MIN; i < DRM_SCHED_PRIORITY_COUNT; i++) drm_sched_rq_init(sched, &sched->sched_rq[i]); @@ -1124,7 +1112,7 @@ int drm_sched_init(struct drm_gpu_scheduler *sched, if (IS_ERR(sched->thread)) { ret = PTR_ERR(sched->thread); sched->thread = NULL; - DRM_DEV_ERROR(sched->dev, "Failed to create scheduler for %s.\n", name); + DRM_DEV_ERROR(sched->dev, "Failed to create scheduler for %s.\n", args->name); return ret; } diff --git a/drivers/gpu/drm/tiny/ofdrm.c b/drivers/gpu/drm/tiny/ofdrm.c index fe2d3b7c6125..1caef717e4cf 100644 --- a/drivers/gpu/drm/tiny/ofdrm.c +++ b/drivers/gpu/drm/tiny/ofdrm.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only #include +#include #include #include @@ -232,7 +233,7 @@ static bool is_avivo(u32 vendor, u32 device) /* This will match most R5xx */ return (vendor == PCI_VENDOR_ID_ATI) && ((device >= PCI_VENDOR_ID_ATI_R520 && device < 0x7800) || - (PCI_VENDOR_ID_ATI_R600 >= 0x9400)); + (device >= PCI_VENDOR_ID_ATI_R600)); } static enum ofdrm_model display_get_model_of(struct drm_device *dev, struct device_node *of_node) @@ -1169,7 +1170,10 @@ static struct ofdrm_device *ofdrm_device_create(struct drm_driver *drv, return ERR_PTR(-EINVAL); } - fb_size = linebytes * height; + if (check_mul_overflow(linebytes, height, &fb_size)) { + drm_err(dev, "framebuffer size exceeds maximum\n"); + return ERR_PTR(-EINVAL); + } /* * Try to figure out the address of the framebuffer. Unfortunately, Open diff --git a/drivers/gpu/drm/tiny/simpledrm.c b/drivers/gpu/drm/tiny/simpledrm.c index 30676b107303..d10574e630a8 100644 --- a/drivers/gpu/drm/tiny/simpledrm.c +++ b/drivers/gpu/drm/tiny/simpledrm.c @@ -210,6 +210,39 @@ simplefb_get_memory_of(struct drm_device *dev, struct device_node *of_node) return res; } +static int __simplefb_get_panel_size_mm_of(struct drm_device *dev, + struct device_node *of_panel_node, + const char *name) +{ + int ret; + u32 value; + + ret = of_property_read_u32(of_panel_node, name, &value); + if (ret) { + drm_dbg(dev, "simplefb: cannot parse panel %s: error %d\n", + name, ret); + return ret; + } else if (value > U16_MAX) { + drm_dbg(dev, "simplefb: panel %s of %u exceeds maximum value\n", + name, value); + return -EINVAL; + } + + return value; +} + +static int simplefb_get_panel_width_mm_of(struct drm_device *dev, + struct device_node *of_panel_node) +{ + return __simplefb_get_panel_size_mm_of(dev, of_panel_node, "width-mm"); +} + +static int simplefb_get_panel_height_mm_of(struct drm_device *dev, + struct device_node *of_panel_node) +{ + return __simplefb_get_panel_size_mm_of(dev, of_panel_node, "height-mm"); +} + /* * Simple Framebuffer device */ @@ -625,7 +658,7 @@ static struct simpledrm_device *simpledrm_device_create(struct drm_driver *drv, struct simpledrm_device *sdev; struct drm_device *dev; int width, height, stride; - int width_mm = 0, height_mm = 0; + u16 width_mm = 0, height_mm = 0; struct device_node *panel_node; const struct drm_format_info *format; struct resource *res, *mem = NULL; @@ -685,8 +718,18 @@ static struct simpledrm_device *simpledrm_device_create(struct drm_driver *drv, return ERR_CAST(mem); panel_node = of_parse_phandle(of_node, "panel", 0); if (panel_node) { - simplefb_read_u32_of(dev, panel_node, "width-mm", &width_mm); - simplefb_read_u32_of(dev, panel_node, "height-mm", &height_mm); + /* + * Ignore errors from parsing the physical panel + * size. Using the pre-initialized sizes of 0 will + * make drm_sysfb_mode() calculate a default physical + * size based on a resolution of 96 dpi. + */ + ret = simplefb_get_panel_width_mm_of(dev, panel_node); + if (ret > 0) + width_mm = ret; + ret = simplefb_get_panel_height_mm_of(dev, panel_node); + if (ret > 0) + height_mm = ret; of_node_put(panel_node); } } else { @@ -694,9 +737,15 @@ static struct simpledrm_device *simpledrm_device_create(struct drm_driver *drv, return ERR_PTR(-ENODEV); } if (!stride) { - stride = drm_format_info_min_pitch(format, 0, width); - if (drm_WARN_ON(dev, !stride)) + u64 pitch = drm_format_info_min_pitch(format, 0, width); + + if (drm_WARN_ON(dev, !pitch)) { + return ERR_PTR(-EINVAL); /* driver bug */ + } else if (pitch > INT_MAX) { + drm_warn(dev, "stride of %llu exceeds maximum\n", pitch); return ERR_PTR(-EINVAL); + } + stride = pitch; } /* diff --git a/drivers/gpu/drm/v3d/v3d_sched.c b/drivers/gpu/drm/v3d/v3d_sched.c index ab872bc81800..804891011330 100644 --- a/drivers/gpu/drm/v3d/v3d_sched.c +++ b/drivers/gpu/drm/v3d/v3d_sched.c @@ -77,15 +77,11 @@ static struct dma_fence *v3d_bin_job_run(struct drm_sched_job *sched_job) struct v3d_bin_job *job = to_bin_job(sched_job); struct v3d_dev *v3d = job->base.v3d; struct drm_device *dev = &v3d->drm; - struct dma_fence *fence; + struct dma_fence *fence = NULL; unsigned long irqflags; - if (unlikely(job->base.base.s_fence->finished.error)) { - spin_lock_irqsave(&v3d->job_lock, irqflags); - v3d->queue[V3D_BIN].active_job = NULL; - spin_unlock_irqrestore(&v3d->job_lock, irqflags); - return NULL; - } + if (unlikely(job->base.base.s_fence->finished.error)) + goto out_clean_job; /* Lock required around bin_job update vs * v3d_overflow_mem_work(). @@ -102,7 +98,7 @@ static struct dma_fence *v3d_bin_job_run(struct drm_sched_job *sched_job) fence = v3d_fence_create(v3d, V3D_BIN); if (IS_ERR(fence)) - return NULL; + goto out_clean_job; if (job->base.irq_fence) dma_fence_put(job->base.irq_fence); @@ -129,6 +125,12 @@ static struct dma_fence *v3d_bin_job_run(struct drm_sched_job *sched_job) V3D_CORE_WRITE(0, V3D_CLE_CT0QEA, job->end); return fence; + +out_clean_job: + spin_lock_irqsave(&v3d->job_lock, irqflags); + v3d->queue[V3D_BIN].active_job = NULL; + spin_unlock_irqrestore(&v3d->job_lock, irqflags); + return fence; } static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job) @@ -136,12 +138,10 @@ static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job) struct v3d_render_job *job = to_render_job(sched_job); struct v3d_dev *v3d = job->base.v3d; struct drm_device *dev = &v3d->drm; - struct dma_fence *fence; + struct dma_fence *fence = NULL; - if (unlikely(job->base.base.s_fence->finished.error)) { - v3d->queue[V3D_RENDER].active_job = NULL; - return NULL; - } + if (unlikely(job->base.base.s_fence->finished.error)) + goto out_clean_job; v3d->queue[V3D_RENDER].active_job = &job->base; @@ -155,7 +155,7 @@ static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job) fence = v3d_fence_create(v3d, V3D_RENDER); if (IS_ERR(fence)) - return NULL; + goto out_clean_job; if (job->base.irq_fence) dma_fence_put(job->base.irq_fence); @@ -175,6 +175,10 @@ static struct dma_fence *v3d_render_job_run(struct drm_sched_job *sched_job) V3D_CORE_WRITE(0, V3D_CLE_CT1QEA, job->end); return fence; + +out_clean_job: + v3d->queue[V3D_RENDER].active_job = NULL; + return fence; } static struct dma_fence * @@ -183,18 +187,16 @@ v3d_tfu_job_run(struct drm_sched_job *sched_job) struct v3d_tfu_job *job = to_tfu_job(sched_job); struct v3d_dev *v3d = job->base.v3d; struct drm_device *dev = &v3d->drm; - struct dma_fence *fence; + struct dma_fence *fence = NULL; - if (unlikely(job->base.base.s_fence->finished.error)) { - v3d->queue[V3D_TFU].active_job = NULL; - return NULL; - } + if (unlikely(job->base.base.s_fence->finished.error)) + goto out_clean_job; v3d->queue[V3D_TFU].active_job = &job->base; fence = v3d_fence_create(v3d, V3D_TFU); if (IS_ERR(fence)) - return NULL; + goto out_clean_job; if (job->base.irq_fence) dma_fence_put(job->base.irq_fence); @@ -218,6 +220,10 @@ v3d_tfu_job_run(struct drm_sched_job *sched_job) V3D_WRITE(V3D_TFU_ICFG, job->args.icfg | V3D_TFU_ICFG_IOC); return fence; + +out_clean_job: + v3d->queue[V3D_TFU].active_job = NULL; + return fence; } static struct dma_fence * @@ -226,13 +232,11 @@ v3d_csd_job_run(struct drm_sched_job *sched_job) struct v3d_csd_job *job = to_csd_job(sched_job); struct v3d_dev *v3d = job->base.v3d; struct drm_device *dev = &v3d->drm; - struct dma_fence *fence; + struct dma_fence *fence = NULL; int i; - if (unlikely(job->base.base.s_fence->finished.error)) { - v3d->queue[V3D_CSD].active_job = NULL; - return NULL; - } + if (unlikely(job->base.base.s_fence->finished.error)) + goto out_clean_job; /* The HW interprets a workgroup size of 0 as 65536; however, the * user-space driver exposes a maximum of 65535. Therefore, a 0 in @@ -250,7 +254,7 @@ v3d_csd_job_run(struct drm_sched_job *sched_job) fence = v3d_fence_create(v3d, V3D_CSD); if (IS_ERR(fence)) - return NULL; + goto out_clean_job; if (job->base.irq_fence) dma_fence_put(job->base.irq_fence); @@ -266,6 +270,10 @@ v3d_csd_job_run(struct drm_sched_job *sched_job) V3D_CORE_WRITE(0, V3D_CSD_QUEUED_CFG0, job->args.cfg[0]); return fence; + +out_clean_job: + v3d->queue[V3D_CSD].active_job = NULL; + return fence; } static struct dma_fence * @@ -423,49 +431,41 @@ static const struct drm_sched_backend_ops v3d_cache_clean_sched_ops = { int v3d_sched_init(struct v3d_dev *v3d) { - int hw_jobs_limit = 1; - int job_hang_limit = 0; - int hang_limit_ms = 500; + struct drm_sched_init_args args = { + .hw_submission = 1, + .timeout = msecs_to_jiffies(500), + .dev = v3d->drm.dev, + }; int ret; - ret = drm_sched_init(&v3d->queue[V3D_BIN].sched, - &v3d_bin_sched_ops, - hw_jobs_limit, job_hang_limit, - msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_bin", v3d->drm.dev); + args.ops = &v3d_bin_sched_ops; + args.name = "v3d_bin"; + ret = drm_sched_init(&v3d->queue[V3D_BIN].sched, &args); if (ret) return ret; - ret = drm_sched_init(&v3d->queue[V3D_RENDER].sched, - &v3d_render_sched_ops, - hw_jobs_limit, job_hang_limit, - msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_render", v3d->drm.dev); + args.ops = &v3d_render_sched_ops; + args.name = "v3d_render"; + ret = drm_sched_init(&v3d->queue[V3D_RENDER].sched, &args); if (ret) goto fail; - ret = drm_sched_init(&v3d->queue[V3D_TFU].sched, - &v3d_tfu_sched_ops, - hw_jobs_limit, job_hang_limit, - msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_tfu", v3d->drm.dev); + args.ops = &v3d_tfu_sched_ops; + args.name = "v3d_tfu"; + ret = drm_sched_init(&v3d->queue[V3D_TFU].sched, &args); if (ret) goto fail; if (v3d_has_csd(v3d)) { - ret = drm_sched_init(&v3d->queue[V3D_CSD].sched, - &v3d_csd_sched_ops, - hw_jobs_limit, job_hang_limit, - msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_csd", v3d->drm.dev); + args.ops = &v3d_csd_sched_ops; + args.name = "v3d_csd"; + ret = drm_sched_init(&v3d->queue[V3D_CSD].sched, &args); if (ret) goto fail; - ret = drm_sched_init(&v3d->queue[V3D_CACHE_CLEAN].sched, - &v3d_cache_clean_sched_ops, - hw_jobs_limit, job_hang_limit, - msecs_to_jiffies(hang_limit_ms), NULL, - NULL, "v3d_cache_clean", v3d->drm.dev); + args.ops = &v3d_cache_clean_sched_ops; + args.name = "v3d_cache_clean"; + ret = drm_sched_init(&v3d->queue[V3D_CACHE_CLEAN].sched, &args); if (ret) goto fail; } diff --git a/drivers/gpu/drm/vc4/vc4_bo.c b/drivers/gpu/drm/vc4/vc4_bo.c index cd4f528a93ce..c2d69390330e 100644 --- a/drivers/gpu/drm/vc4/vc4_bo.c +++ b/drivers/gpu/drm/vc4/vc4_bo.c @@ -1050,10 +1050,11 @@ static void vc4_bo_cache_destroy(struct drm_device *dev, void *unused) struct vc4_dev *vc4 = to_vc4_dev(dev); int i; - del_timer(&vc4->bo_cache.time_timer); + timer_shutdown_sync(&vc4->bo_cache.time_timer); cancel_work_sync(&vc4->bo_cache.time_work); vc4_bo_cache_purge(dev); + kfree(vc4->bo_cache.size_list); for (i = 0; i < vc4->num_labels; i++) { if (vc4->bo_labels[i].num_allocated) { diff --git a/drivers/gpu/drm/vc4/vc4_drv.c b/drivers/gpu/drm/vc4/vc4_drv.c index 1b3531374967..ddeb9ffc3063 100644 --- a/drivers/gpu/drm/vc4/vc4_drv.c +++ b/drivers/gpu/drm/vc4/vc4_drv.c @@ -351,8 +351,11 @@ static int vc4_drm_bind(struct device *dev) } ret = drm_aperture_remove_framebuffers(driver); - if (ret) + if (ret) { + if (firmware) + rpi_firmware_put(firmware); return ret; + } if (firmware) { ret = rpi_firmware_property(firmware, diff --git a/drivers/gpu/drm/vc4/vc4_hvs.c b/drivers/gpu/drm/vc4/vc4_hvs.c index bea1b3d35cdd..4e74617e493e 100644 --- a/drivers/gpu/drm/vc4/vc4_hvs.c +++ b/drivers/gpu/drm/vc4/vc4_hvs.c @@ -1045,7 +1045,7 @@ static int vc4_hvs_bind(struct device *dev, struct device *master, void *data) static void vc4_hvs_unbind(struct device *dev, struct device *master, void *data) { - struct drm_device *drm = dev_get_drvdata(master); + struct drm_device *drm = data; struct vc4_dev *vc4 = to_vc4_dev(drm); struct vc4_hvs *hvs = vc4->hvs; struct drm_mm_node *node, *next; diff --git a/drivers/gpu/drm/vc4/vc4_v3d.c b/drivers/gpu/drm/vc4/vc4_v3d.c index b1de828e2f90..39f37b921ae5 100644 --- a/drivers/gpu/drm/vc4/vc4_v3d.c +++ b/drivers/gpu/drm/vc4/vc4_v3d.c @@ -9,6 +9,7 @@ #include #include #include +#include #include "vc4_drv.h" #include "vc4_regs.h" @@ -504,11 +505,18 @@ static int vc4_v3d_bind(struct device *dev, struct device *master, void *data) static void vc4_v3d_unbind(struct device *dev, struct device *master, void *data) { - struct drm_device *drm = dev_get_drvdata(master); + struct drm_device *drm = data; struct vc4_dev *vc4 = to_vc4_dev(drm); + /* A straggler vc4_reset() re-enables the interrupt. */ + timer_shutdown_sync(&vc4->hangcheck.timer); + cancel_work_sync(&vc4->hangcheck.reset_work); + vc4_irq_uninstall(drm); + /* Flush rather than cancel, so queued completions release their jobs. */ + flush_work(&vc4->job_done_work); + /* Disable the binner's overflow memory address, so the next * driver probe (if any) doesn't try to reuse our old * allocation. diff --git a/drivers/gpu/drm/virtio/virtgpu_drv.h b/drivers/gpu/drm/virtio/virtgpu_drv.h index 9f9ded45cd6a..de36af56df86 100644 --- a/drivers/gpu/drm/virtio/virtgpu_drv.h +++ b/drivers/gpu/drm/virtio/virtgpu_drv.h @@ -43,6 +43,8 @@ #include #include +#include + #define DRIVER_NAME "virtio_gpu" #define DRIVER_DESC "virtio GPU" #define DRIVER_DATE "0" @@ -58,6 +60,24 @@ #define MAX_CAPSET_ID 63 #define MAX_RINGS 64 +/* + * Whether the host must be told about resource backing pages by DMA address + * rather than guest-physical address. + * + * This mirrors vring_use_map_api() in drivers/virtio/virtio_ring.c, including + * its xen_domain() case. + */ +static inline bool virtio_gpu_use_dma_api(const struct virtio_device *vdev) +{ + if (!virtio_has_dma_quirk(vdev)) + return true; + + if (xen_domain()) + return true; + + return false; +} + struct virtio_gpu_object_params { unsigned long size; bool dumb; diff --git a/drivers/gpu/drm/virtio/virtgpu_object.c b/drivers/gpu/drm/virtio/virtgpu_object.c index c7e74cf13022..c4f8cdf85f60 100644 --- a/drivers/gpu/drm/virtio/virtgpu_object.c +++ b/drivers/gpu/drm/virtio/virtgpu_object.c @@ -137,7 +137,7 @@ static int virtio_gpu_object_shmem_init(struct virtio_gpu_device *vgdev, struct virtio_gpu_mem_entry **ents, unsigned int *nents) { - bool use_dma_api = !virtio_has_dma_quirk(vgdev->vdev); + bool use_dma_api = virtio_gpu_use_dma_api(vgdev->vdev); struct scatterlist *sg; struct sg_table *pages; int si; diff --git a/drivers/gpu/drm/virtio/virtgpu_vq.c b/drivers/gpu/drm/virtio/virtgpu_vq.c index a4d1a7710325..d2c83f3e74b9 100644 --- a/drivers/gpu/drm/virtio/virtgpu_vq.c +++ b/drivers/gpu/drm/virtio/virtgpu_vq.c @@ -601,7 +601,7 @@ void virtio_gpu_cmd_transfer_to_host_2d(struct virtio_gpu_device *vgdev, struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(objs->objs[0]); struct virtio_gpu_transfer_to_host_2d *cmd_p; struct virtio_gpu_vbuffer *vbuf; - bool use_dma_api = !virtio_has_dma_quirk(vgdev->vdev); + bool use_dma_api = virtio_gpu_use_dma_api(vgdev->vdev); if (virtio_gpu_is_shmem(bo) && use_dma_api) dma_sync_sgtable_for_device(vgdev->vdev->dev.parent, @@ -1020,7 +1020,7 @@ void virtio_gpu_cmd_transfer_to_host_3d(struct virtio_gpu_device *vgdev, struct virtio_gpu_object *bo = gem_to_virtio_gpu_obj(objs->objs[0]); struct virtio_gpu_transfer_host_3d *cmd_p; struct virtio_gpu_vbuffer *vbuf; - bool use_dma_api = !virtio_has_dma_quirk(vgdev->vdev); + bool use_dma_api = virtio_gpu_use_dma_api(vgdev->vdev); if (virtio_gpu_is_shmem(bo) && use_dma_api) dma_sync_sgtable_for_device(vgdev->vdev->dev.parent, diff --git a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c index 33f73d559be7..7524403427c7 100644 --- a/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c +++ b/drivers/gpu/drm/vmwgfx/vmwgfx_kms.c @@ -370,6 +370,7 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf, unsigned long kmap_num; SVGA3dCopyBox *box; unsigned box_count; + u64 src_extent; void *virtual; bool is_iomem; struct vmw_dma_cmd { @@ -381,6 +382,8 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf, vmw_surface_get_desc(VMW_CURSOR_SNOOP_FORMAT); const u32 image_pitch = VMW_CURSOR_SNOOP_WIDTH * desc->pitchBytesPerBlock; + BUILD_BUG_ON(sizeof(*srf->snooper.image) != desc->pitchBytesPerBlock); + cmd = container_of(header, struct vmw_dma_cmd, header); /* No snooper installed, nothing to copy */ @@ -418,8 +421,19 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf, return; } + if (box->w == 0 || box->h == 0) + return; + + src_extent = (u64)(box->h - 1) * cmd->dma.guest.pitch + + (u64)box->w * desc->pitchBytesPerBlock; + if (src_extent > bo->base.size) { + DRM_ERROR("Cursor snoop source of %llu bytes exceeds the %zu byte buffer\n", + src_extent, bo->base.size); + return; + } + kmap_offset = cmd->dma.guest.ptr.offset >> PAGE_SHIFT; - kmap_num = (VMW_CURSOR_SNOOP_HEIGHT*image_pitch) >> PAGE_SHIFT; + kmap_num = PFN_UP(src_extent); ret = ttm_bo_reserve(bo, true, false, NULL); if (unlikely(ret != 0)) { @@ -433,14 +447,16 @@ void vmw_kms_cursor_snoop(struct vmw_surface *srf, virtual = ttm_kmap_obj_virtual(&map, &is_iomem); - if (box->w == VMW_CURSOR_SNOOP_WIDTH && cmd->dma.guest.pitch == image_pitch) { + if (box->w == VMW_CURSOR_SNOOP_WIDTH && + box->h == VMW_CURSOR_SNOOP_HEIGHT && + cmd->dma.guest.pitch == image_pitch) { memcpy(srf->snooper.image, virtual, VMW_CURSOR_SNOOP_HEIGHT*image_pitch); } else { - /* Image is unsigned pointer. */ + /* The destination stride is in pixels. */ for (i = 0; i < box->h; i++) - memcpy(srf->snooper.image + i * image_pitch, - virtual + i * cmd->dma.guest.pitch, + memcpy(srf->snooper.image + i * VMW_CURSOR_SNOOP_WIDTH, + virtual + (size_t)i * cmd->dma.guest.pitch, box->w * desc->pitchBytesPerBlock); } diff --git a/drivers/hid/bpf/hid_bpf_dispatch.c b/drivers/hid/bpf/hid_bpf_dispatch.c index e0c03942ff7c..9948ada290f0 100644 --- a/drivers/hid/bpf/hid_bpf_dispatch.c +++ b/drivers/hid/bpf/hid_bpf_dispatch.c @@ -18,6 +18,7 @@ #include #include #include +#include #include "hid_bpf_dispatch.h" #include "entrypoints/entrypoints.lskel.h" @@ -157,13 +158,15 @@ noinline __u8 * hid_bpf_get_data(struct hid_bpf_ctx *ctx, unsigned int offset, const size_t rdwr_buf_size) { struct hid_bpf_ctx_kern *ctx_kern; + size_t end; if (!ctx) return NULL; ctx_kern = container_of(ctx, struct hid_bpf_ctx_kern, ctx); - if (rdwr_buf_size + offset > ctx->allocated_size) + if (check_add_overflow(rdwr_buf_size, offset, &end) || + end > ctx->allocated_size) return NULL; return ctx_kern->data + offset; diff --git a/drivers/hid/hid-alps.c b/drivers/hid/hid-alps.c index 8aed5e17e581..25da18017500 100644 --- a/drivers/hid/hid-alps.c +++ b/drivers/hid/hid-alps.c @@ -825,6 +825,24 @@ static int alps_probe(struct hid_device *hdev, const struct hid_device_id *id) return 0; } +static void alps_remove(struct hid_device *hdev) +{ + struct alps_dev *data = hid_get_drvdata(hdev); + + /* + * input2 ("DualPoint Stick") is allocated separately and is not + * tracked in hdev->inputs, so the default remove path + * (hid_hw_stop -> hidinput_disconnect) does not unregister it. + * + * Stop the device first so that no URB callback can touch input2 + * while it is being unregistered, then drop it explicitly. + */ + hid_hw_stop(hdev); + + if (data->input2) + input_unregister_device(data->input2); +} + static const struct hid_device_id alps_id[] = { { HID_DEVICE(HID_BUS_ANY, HID_GROUP_ANY, USB_VENDOR_ID_ALPS_JP, HID_DEVICE_ID_ALPS_U1_DUAL) }, @@ -845,10 +863,9 @@ static struct hid_driver alps_driver = { .raw_event = alps_raw_event, .input_mapping = alps_input_mapping, .input_configured = alps_input_configured, -#ifdef CONFIG_PM - .resume = alps_post_resume, - .reset_resume = alps_post_reset, -#endif + .resume = pm_ptr(alps_post_resume), + .reset_resume = pm_ptr(alps_post_reset), + .remove = alps_remove, }; module_hid_driver(alps_driver); diff --git a/drivers/hid/hid-core.c b/drivers/hid/hid-core.c index 446ed394dc48..9607ec8b149e 100644 --- a/drivers/hid/hid-core.c +++ b/drivers/hid/hid-core.c @@ -635,7 +635,11 @@ static int hid_parser_main(struct hid_parser *parser, struct hid_item *item) ret = hid_add_field(parser, HID_FEATURE_REPORT, data); break; default: - hid_warn(parser->device, "unknown main item tag 0x%x\n", item->tag); + if (item->tag >= HID_MAIN_ITEM_TAG_RESERVED_MIN && + item->tag <= HID_MAIN_ITEM_TAG_RESERVED_MAX) + hid_warn_ratelimited(parser->device, "reserved main item tag 0x%x\n", item->tag); + else + hid_warn_ratelimited(parser->device, "unknown main item tag 0x%x\n", item->tag); ret = 0; } @@ -2045,9 +2049,9 @@ int hid_report_raw_event(struct hid_device *hid, enum hid_report_type type, u8 * } if (csize < rsize) { - dbg_hid("report %d is too short, (%d < %d)\n", report->id, - csize, rsize); - memset(cdata + csize, 0, rsize - csize); + hid_warn_ratelimited(hid, "Event data for report %d was too short (%d vs %d)\n", + report->id, rsize, csize); + return -EINVAL; } if ((hid->claimed & HID_CLAIMED_HIDDEV) && hid->hiddev_report_event) @@ -2082,14 +2086,36 @@ EXPORT_SYMBOL_GPL(hid_report_raw_event); * @interrupt: distinguish between interrupt and control transfers * * This is data entry for lower layers. + * Legacy, please use hid_safe_input_report() instead. */ int hid_input_report(struct hid_device *hid, enum hid_report_type type, u8 *data, u32 size, int interrupt) +{ + return hid_safe_input_report(hid, type, data, size, size, interrupt); +} +EXPORT_SYMBOL_GPL(hid_input_report); + +/** + * hid_safe_input_report - report data from lower layer (usb, bt...) + * + * @hid: hid device + * @type: HID report type (HID_*_REPORT) + * @data: report contents + * @bufsize: allocated size of the data buffer + * @size: useful size of data parameter + * @interrupt: distinguish between interrupt and control transfers + * + * This is data entry for lower layers. + * Please use this function instead of the non safe version because we provide + * here the size of the buffer, allowing hid-core to make smarter decisions + * regarding the incoming buffer. + */ +int hid_safe_input_report(struct hid_device *hid, enum hid_report_type type, u8 *data, + size_t bufsize, u32 size, int interrupt) { struct hid_report_enum *report_enum; struct hid_driver *hdrv; struct hid_report *report; - size_t bufsize = size; int ret = 0; if (!hid) @@ -2140,7 +2166,7 @@ int hid_input_report(struct hid_device *hid, enum hid_report_type type, u8 *data up(&hid->driver_input_lock); return ret; } -EXPORT_SYMBOL_GPL(hid_input_report); +EXPORT_SYMBOL_GPL(hid_safe_input_report); bool hid_match_one_id(const struct hid_device *hdev, const struct hid_device_id *id) diff --git a/drivers/hid/hid-logitech-hidpp.c b/drivers/hid/hid-logitech-hidpp.c index d223d1fa33f3..88b4027236cd 100644 --- a/drivers/hid/hid-logitech-hidpp.c +++ b/drivers/hid/hid-logitech-hidpp.c @@ -2879,18 +2879,19 @@ static int hidpp_ff_init(struct hidpp_device *hidpp, * ownership to FF core */ data = kmemdup(data, sizeof(*data), GFP_KERNEL); - if (!data) - return -ENOMEM; + if (!data) { + error = -ENOMEM; + goto err_destroy_ff; + } data->effect_ids = kcalloc(num_slots, sizeof(int), GFP_KERNEL); if (!data->effect_ids) { - kfree(data); - return -ENOMEM; + error = -ENOMEM; + goto err_free_data; } data->wq = create_singlethread_workqueue("hidpp-ff-sendqueue"); if (!data->wq) { - kfree(data->effect_ids); - kfree(data); - return -ENOMEM; + error = -ENOMEM; + goto err_free_effect_ids; } data->hidpp = hidpp; @@ -2920,6 +2921,14 @@ static int hidpp_ff_init(struct hidpp_device *hidpp, version); return 0; + +err_free_effect_ids: + kfree(data->effect_ids); +err_free_data: + kfree(data); +err_destroy_ff: + input_ff_destroy(dev); + return error; } /* ************************************************************************** */ @@ -4539,10 +4548,12 @@ static int hidpp_probe(struct hid_device *hdev, const struct hid_device_id *id) if (hidpp->quirks & HIDPP_QUIRK_CLASS_G920) { ret = hidpp_ff_init(hidpp, &data); - if (ret) + if (ret) { hid_warn(hidpp->hid_dev, "Unable to initialize force feedback support, errno %d\n", ret); + ret = 0; + } } /* diff --git a/drivers/hid/hid-mcp2221.c b/drivers/hid/hid-mcp2221.c index 6678b14e9855..233f27d5ce16 100644 --- a/drivers/hid/hid-mcp2221.c +++ b/drivers/hid/hid-mcp2221.c @@ -99,6 +99,7 @@ struct mcp2221 { u8 *rxbuf; u8 txbuf[64]; int rxbuf_idx; + int rxbuf_size; int status; u8 cur_i2c_clk_div; struct gpio_chip *gc; @@ -307,17 +308,19 @@ static int mcp_i2c_smbus_read(struct mcp2221 *mcp, mcp->txbuf[3] = (u8)(msg->addr << 1); total_len = msg->len; mcp->rxbuf = msg->buf; + mcp->rxbuf_size = msg->len; } else { mcp->txbuf[1] = smbus_len; mcp->txbuf[2] = 0; mcp->txbuf[3] = (u8)(smbus_addr << 1); total_len = smbus_len; mcp->rxbuf = smbus_buf; + mcp->rxbuf_size = smbus_len; } ret = mcp_send_data_req_status(mcp, mcp->txbuf, 4); if (ret) - return ret; + goto out; mcp->rxbuf_idx = 0; @@ -339,7 +342,7 @@ static int mcp_i2c_smbus_read(struct mcp2221 *mcp, } else { usleep_range(980, 1000); mcp_cancel_last_cmd(mcp); - return ret; + goto out; } } else { retries = 0; @@ -349,6 +352,10 @@ static int mcp_i2c_smbus_read(struct mcp2221 *mcp, usleep_range(980, 1000); ret = mcp_chk_last_cmd_status_free_bus(mcp); +out: + mcp->rxbuf = NULL; + mcp->rxbuf_size = 0; + return ret; } @@ -522,6 +529,7 @@ static int mcp_smbus_xfer(struct i2c_adapter *adapter, u16 addr, mcp->rxbuf_idx = 0; mcp->rxbuf = data->block; + mcp->rxbuf_size = sizeof(data->block); mcp->txbuf[0] = MCP2221_I2C_GET_DATA; ret = mcp_send_data_req_status(mcp, mcp->txbuf, 1); if (ret) @@ -545,6 +553,7 @@ static int mcp_smbus_xfer(struct i2c_adapter *adapter, u16 addr, mcp->rxbuf_idx = 0; mcp->rxbuf = data->block; + mcp->rxbuf_size = sizeof(data->block); mcp->txbuf[0] = MCP2221_I2C_GET_DATA; ret = mcp_send_data_req_status(mcp, mcp->txbuf, 1); if (ret) @@ -827,6 +836,10 @@ static int mcp2221_raw_event(struct hid_device *hdev, mcp->status = -EINVAL; break; } + if (mcp->rxbuf_idx + data[3] > mcp->rxbuf_size) { + mcp->status = -EINVAL; + break; + } buf = mcp->rxbuf; memcpy(&buf[mcp->rxbuf_idx], &data[4], data[3]); mcp->rxbuf_idx = mcp->rxbuf_idx + data[3]; diff --git a/drivers/hid/hid-multitouch.c b/drivers/hid/hid-multitouch.c index 4481bf5c9133..422172ecf2bb 100644 --- a/drivers/hid/hid-multitouch.c +++ b/drivers/hid/hid-multitouch.c @@ -1895,7 +1895,7 @@ static void mt_remove(struct hid_device *hdev) { struct mt_device *td = hid_get_drvdata(hdev); - del_timer_sync(&td->release_timer); + timer_shutdown_sync(&td->release_timer); sysfs_remove_group(&hdev->dev.kobj, &mt_attribute_group); hid_hw_stop(hdev); diff --git a/drivers/hid/hid-nintendo.c b/drivers/hid/hid-nintendo.c index 115f496368e5..7e272e79e00f 100644 --- a/drivers/hid/hid-nintendo.c +++ b/drivers/hid/hid-nintendo.c @@ -301,6 +301,7 @@ enum joycon_ctlr_state { JOYCON_CTLR_STATE_INIT, JOYCON_CTLR_STATE_READ, JOYCON_CTLR_STATE_REMOVED, + JOYCON_CTLR_STATE_SUSPENDED, }; /* Controller type received as part of device info */ @@ -2418,14 +2419,46 @@ static void nintendo_hid_remove(struct hid_device *hdev) static int nintendo_hid_resume(struct hid_device *hdev) { - int ret = joycon_init(hdev); + struct joycon_ctlr *ctlr = hid_get_drvdata(hdev); + int ret; + + hid_dbg(hdev, "resume\n"); + if (ctlr->hdev->bus != BUS_USB) { + hid_dbg(hdev, "no-op resume for bt ctlr\n"); + ctlr->ctlr_state = JOYCON_CTLR_STATE_READ; + return 0; + } + ret = joycon_init(hdev); if (ret) - hid_err(hdev, "Failed to restore controller after resume"); + hid_err(hdev, + "Failed to restore controller after resume: %d\n", + ret); + else + ctlr->ctlr_state = JOYCON_CTLR_STATE_READ; return ret; } +static int nintendo_hid_suspend(struct hid_device *hdev, pm_message_t message) +{ + struct joycon_ctlr *ctlr = hid_get_drvdata(hdev); + + hid_dbg(hdev, "suspend: %d\n", message.event); + /* + * Avoid any blocking loops in suspend/resume transitions. + * + * joycon_enforce_subcmd_rate() can result in repeated retries if for + * whatever reason the controller stops providing input reports. + * + * This has been observed with bluetooth controllers which lose + * connectivity prior to suspend (but not long enough to result in + * complete disconnection). + */ + ctlr->ctlr_state = JOYCON_CTLR_STATE_SUSPENDED; + return 0; +} + #endif static const struct hid_device_id nintendo_hid_devices[] = { @@ -2452,6 +2485,7 @@ static struct hid_driver nintendo_hid_driver = { #ifdef CONFIG_PM .resume = nintendo_hid_resume, + .suspend = nintendo_hid_suspend, #endif }; module_hid_driver(nintendo_hid_driver); diff --git a/drivers/hid/hid-sony.c b/drivers/hid/hid-sony.c index ebc0aa4e4345..a83ebda69243 100644 --- a/drivers/hid/hid-sony.c +++ b/drivers/hid/hid-sony.c @@ -1745,13 +1745,10 @@ static int sony_check_add_dev_list(struct sony_sc *sc) static void sony_remove_dev_list(struct sony_sc *sc) { - unsigned long flags; + guard(spinlock_irqsave)(&sony_dev_list_lock); - if (sc->list_node.next) { - spin_lock_irqsave(&sony_dev_list_lock, flags); - list_del(&(sc->list_node)); - spin_unlock_irqrestore(&sony_dev_list_lock, flags); - } + if (!list_empty(&sc->list_node)) + list_del_init(&sc->list_node); } static int sony_get_bt_devaddr(struct sony_sc *sc) @@ -1889,6 +1886,13 @@ static inline void sony_cancel_work_sync(struct sony_sc *sc) } } +static void sony_cleanup(struct sony_sc *sc) +{ + sony_cancel_work_sync(sc); + sony_remove_dev_list(sc); + sony_release_device_id(sc); +} + static int sony_input_configured(struct hid_device *hdev, struct hid_input *hidinput) { @@ -2049,9 +2053,7 @@ static int sony_input_configured(struct hid_device *hdev, err_close: hid_hw_close(hdev); err_stop: - sony_cancel_work_sync(sc); - sony_remove_dev_list(sc); - sony_release_device_id(sc); + sony_cleanup(sc); return ret; } @@ -2077,6 +2079,8 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id) } spin_lock_init(&sc->lock); + INIT_LIST_HEAD(&sc->list_node); + sc->device_id = -1; sc->quirks = quirks; hid_set_drvdata(hdev, sc); @@ -2105,6 +2109,7 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id) ret = hid_hw_start(hdev, connect_mask); if (ret) { hid_err(hdev, "hw start failed\n"); + sony_cleanup(sc); return ret; } @@ -2156,7 +2161,7 @@ static int sony_probe(struct hid_device *hdev, const struct hid_device_id *id) err: usb_free_urb(sc->ghl_urb); - + sony_cleanup(sc); hid_hw_stop(hdev); return ret; } @@ -2166,18 +2171,14 @@ static void sony_remove(struct hid_device *hdev) struct sony_sc *sc = hid_get_drvdata(hdev); if (sc->quirks & (GHL_GUITAR_PS3WIIU | GHL_GUITAR_PS4)) { - del_timer_sync(&sc->ghl_poke_timer); + /* poison, not kill: a pending timer must not re-submit during teardown */ + usb_poison_urb(sc->ghl_urb); + timer_shutdown_sync(&sc->ghl_poke_timer); usb_free_urb(sc->ghl_urb); } hid_hw_close(hdev); - - sony_cancel_work_sync(sc); - - sony_remove_dev_list(sc); - - sony_release_device_id(sc); - + sony_cleanup(sc); hid_hw_stop(hdev); } diff --git a/drivers/hid/hid-steam.c b/drivers/hid/hid-steam.c index 29ff4eb5194b..9086bef42be5 100644 --- a/drivers/hid/hid-steam.c +++ b/drivers/hid/hid-steam.c @@ -294,12 +294,14 @@ struct steam_device { spinlock_t lock; struct hid_device *hdev, *client_hdev; struct mutex report_mutex; - bool client_opened; + struct mutex registration_mutex; + unsigned long client_opened; struct input_dev __rcu *input; struct input_dev __rcu *sensors; unsigned long quirks; struct work_struct work_connect; bool connected; + bool registered; char serial_no[STEAM_SERIAL_LEN + 1]; struct power_supply_desc battery_desc; struct power_supply __rcu *battery; @@ -312,7 +314,6 @@ struct steam_device { u16 rumble_left; u16 rumble_right; unsigned int sensor_timestamp_us; - struct work_struct unregister_work; }; static int steam_recv_report(struct steam_device *steam, @@ -322,6 +323,13 @@ static int steam_recv_report(struct steam_device *steam, u8 *buf; int ret; + /* + * All reports start with a two byte header. + * We must read at least two bytes to get a sensible output. + */ + if (size < 2) + return -EINVAL; + r = steam->hdev->report_enum[HID_FEATURE_REPORT].report_id_hash[0]; if (!r) { hid_err(steam->hdev, "No HID_FEATURE_REPORT submitted - nothing to read\n"); @@ -344,10 +352,31 @@ static int steam_recv_report(struct steam_device *steam, ret = hid_hw_raw_request(steam->hdev, 0x00, buf, hid_report_len(r) + 1, HID_FEATURE_REPORT, HID_REQ_GET_REPORT); - if (ret > 0) - memcpy(data, buf + 1, min(size, ret - 1)); + if (ret > 0) { + /* Remove the report ID from the return buffer */ + ret--; + size = min(size, ret); + memcpy(data, buf + 1, size); + } kfree(buf); - return ret; + + if (ret < 0) + hid_err(steam->hdev, "%s: error %d\n", __func__, ret); + else + hid_dbg(steam->hdev, "Received report %*ph\n", size, data); + if (ret < 0) + return ret; + + if (ret < 2) { + hid_err(steam->hdev, "%s: reply too short\n", __func__); + return -EPROTO; + } + if (ret < data[1] + 2) { + hid_err(steam->hdev, "%s: expected %u bytes, read %i\n", + __func__, data[1] + 2, ret); + return -EPROTO; + } + return size; } static int steam_send_report(struct steam_device *steam, @@ -446,21 +475,16 @@ static int steam_get_serial(struct steam_device *steam) u8 cmd[] = {ID_GET_STRING_ATTRIBUTE, 0x15, ATTRIB_STR_UNIT_SERIAL}; u8 reply[3 + STEAM_SERIAL_LEN + 1]; - mutex_lock(&steam->report_mutex); ret = steam_send_report(steam, cmd, sizeof(cmd)); if (ret < 0) - goto out; + return ret; ret = steam_recv_report(steam, reply, sizeof(reply)); if (ret < 0) - goto out; - if (reply[0] != ID_GET_STRING_ATTRIBUTE || reply[1] != 0x15 || reply[2] != ATTRIB_STR_UNIT_SERIAL) { - ret = -EIO; - goto out; - } + return ret; + if (reply[0] != ID_GET_STRING_ATTRIBUTE || reply[1] != 0x15 || reply[2] != ATTRIB_STR_UNIT_SERIAL) + return -EIO; reply[3 + STEAM_SERIAL_LEN] = 0; strscpy(steam->serial_no, reply + 3, sizeof(steam->serial_no)); -out: - mutex_unlock(&steam->report_mutex); return ret; } @@ -503,9 +527,7 @@ static inline int steam_haptic_pulse(struct steam_device *steam, u8 pad, report[8] = count >> 8; report[9] = gain; - mutex_lock(&steam->report_mutex); ret = steam_send_report(steam, report, sizeof(report)); - mutex_unlock(&steam->report_mutex); return ret; } @@ -525,9 +547,7 @@ static inline int steam_haptic_rumble(struct steam_device *steam, report[9] = left_gain; report[10] = right_gain; - mutex_lock(&steam->report_mutex); ret = steam_send_report(steam, report, sizeof(report)); - mutex_unlock(&steam->report_mutex); return ret; } @@ -535,8 +555,11 @@ static void steam_haptic_rumble_cb(struct work_struct *work) { struct steam_device *steam = container_of(work, struct steam_device, rumble_work); + + mutex_lock(&steam->report_mutex); steam_haptic_rumble(steam, 0, steam->rumble_left, steam->rumble_right, 2, 0); + mutex_unlock(&steam->report_mutex); } #ifdef CONFIG_STEAM_FF @@ -558,14 +581,11 @@ static void steam_set_lizard_mode(struct steam_device *steam, bool enable) enable = false; if (enable) { - mutex_lock(&steam->report_mutex); /* enable esc, enter, cursors */ steam_send_report_byte(steam, ID_SET_DEFAULT_DIGITAL_MAPPINGS); /* reset settings */ steam_send_report_byte(steam, ID_LOAD_DEFAULT_SETTINGS); - mutex_unlock(&steam->report_mutex); } else { - mutex_lock(&steam->report_mutex); /* disable esc, enter, cursor */ steam_send_report_byte(steam, ID_CLEAR_DIGITAL_MAPPINGS); @@ -577,13 +597,11 @@ static void steam_set_lizard_mode(struct steam_device *steam, bool enable) SETTING_RIGHT_TRACKPAD_CLICK_PRESSURE, 0xFFFF, /* disable haptic click */ SETTING_STEAM_WATCHDOG_ENABLE, 0, /* disable watchdog that tests if Steam is active */ 0); - mutex_unlock(&steam->report_mutex); } else { steam_write_settings(steam, SETTING_LEFT_TRACKPAD_MODE, TRACKPAD_NONE, /* disable mouse */ SETTING_RIGHT_TRACKPAD_MODE, TRACKPAD_NONE, /* disable mouse */ 0); - mutex_unlock(&steam->report_mutex); } } } @@ -592,7 +610,7 @@ static int steam_input_open(struct input_dev *dev) { struct steam_device *steam = input_get_drvdata(dev); unsigned long flags; - bool set_lizard_mode; + bool client_opened; /* * Disabling lizard mode automatically is only done on the Steam @@ -601,9 +619,10 @@ static int steam_input_open(struct input_dev *dev) */ if (!(steam->quirks & STEAM_QUIRK_DECK)) { spin_lock_irqsave(&steam->lock, flags); - set_lizard_mode = !steam->client_opened && lizard_mode; + client_opened = steam->client_opened; spin_unlock_irqrestore(&steam->lock, flags); - if (set_lizard_mode) + guard(mutex)(&steam->report_mutex); + if (!client_opened && lizard_mode) steam_set_lizard_mode(steam, false); } @@ -614,13 +633,14 @@ static void steam_input_close(struct input_dev *dev) { struct steam_device *steam = input_get_drvdata(dev); unsigned long flags; - bool set_lizard_mode; + bool client_opened; if (!(steam->quirks & STEAM_QUIRK_DECK)) { spin_lock_irqsave(&steam->lock, flags); - set_lizard_mode = !steam->client_opened && lizard_mode; + client_opened = steam->client_opened; spin_unlock_irqrestore(&steam->lock, flags); - if (set_lizard_mode) + guard(mutex)(&steam->report_mutex); + if (!client_opened && lizard_mode) steam_set_lizard_mode(steam, true); } } @@ -694,6 +714,7 @@ static int steam_battery_register(struct steam_device *steam) &steam->battery_desc, &battery_cfg); if (IS_ERR(battery)) { ret = PTR_ERR(battery); + devm_kfree(&steam->hdev->dev, steam->battery_desc.name); hid_err(steam->hdev, "%s:power_supply_register failed with error %d\n", __func__, ret); @@ -947,97 +968,111 @@ static void steam_battery_unregister(struct steam_device *steam) RCU_INIT_POINTER(steam->battery, NULL); synchronize_rcu(); power_supply_unregister(battery); + devm_kfree(&steam->hdev->dev, steam->battery_desc.name); } static int steam_register(struct steam_device *steam) { int ret; - bool client_opened; - unsigned long flags; + mutex_lock(&steam->registration_mutex); /* * This function can be called several times in a row with the * wireless adaptor, without steam_unregister() between them, because * another client send a get_connection_status command, for example. * The battery and serial number are set just once per device. */ - if (!steam->serial_no[0]) { - /* - * Unlikely, but getting the serial could fail, and it is not so - * important, so make up a serial number and go on. - */ - if (steam_get_serial(steam) < 0) - strscpy(steam->serial_no, "XXXXXXXXXX", - sizeof(steam->serial_no)); + if (steam->registered) { + mutex_unlock(&steam->registration_mutex); + return 0; + } - hid_info(steam->hdev, "Steam Controller '%s' connected", - steam->serial_no); + mutex_lock(&steam->report_mutex); + /* + * Unlikely, but getting the serial could fail, and it is not so + * important, so make up a serial number and go on. + */ + if (steam_get_serial(steam) < 0 || !steam->serial_no[0]) + strscpy(steam->serial_no, "XXXXXXXXXX", + sizeof(steam->serial_no)); - /* ignore battery errors, we can live without it */ - if (steam->quirks & STEAM_QUIRK_WIRELESS) - steam_battery_register(steam); + hid_info(steam->hdev, "Steam Controller '%s' connected", + steam->serial_no); - mutex_lock(&steam_devices_lock); - if (list_empty(&steam->list)) - list_add(&steam->list, &steam_devices); - mutex_unlock(&steam_devices_lock); - } + steam_set_lizard_mode(steam, lizard_mode); + mutex_unlock(&steam->report_mutex); - spin_lock_irqsave(&steam->lock, flags); - client_opened = steam->client_opened; - spin_unlock_irqrestore(&steam->lock, flags); + /* ignore battery errors, we can live without it */ + if (steam->quirks & STEAM_QUIRK_WIRELESS) + steam_battery_register(steam); - if (!client_opened) { - steam_set_lizard_mode(steam, lizard_mode); - ret = steam_input_register(steam); - if (ret != 0) - goto steam_register_input_fail; - ret = steam_sensors_register(steam); - if (ret != 0) - goto steam_register_sensors_fail; - } + ret = steam_input_register(steam); + if (ret != 0) + goto steam_register_input_fail; + ret = steam_sensors_register(steam); + if (ret != 0) + goto steam_register_sensors_fail; + + steam->registered = true; + mutex_unlock(&steam->registration_mutex); + mutex_lock(&steam_devices_lock); + if (list_empty(&steam->list)) + list_add(&steam->list, &steam_devices); + mutex_unlock(&steam_devices_lock); return 0; steam_register_sensors_fail: steam_input_unregister(steam); steam_register_input_fail: + steam_battery_unregister(steam); + mutex_unlock(&steam->registration_mutex); + cancel_work_sync(&steam->rumble_work); + cancel_delayed_work_sync(&steam->mode_switch); return ret; } static void steam_unregister(struct steam_device *steam) { + if (!steam->registered) + return; + + hid_info(steam->hdev, "Steam Controller '%s' disconnected", + steam->serial_no); + mutex_lock(&steam->registration_mutex); + steam->registered = false; steam_battery_unregister(steam); steam_sensors_unregister(steam); steam_input_unregister(steam); - if (steam->serial_no[0]) { - hid_info(steam->hdev, "Steam Controller '%s' disconnected", - steam->serial_no); - mutex_lock(&steam_devices_lock); - list_del_init(&steam->list); - mutex_unlock(&steam_devices_lock); - steam->serial_no[0] = 0; - } + mutex_unlock(&steam->registration_mutex); + cancel_work_sync(&steam->rumble_work); + cancel_delayed_work_sync(&steam->mode_switch); + mutex_lock(&steam_devices_lock); + list_del_init(&steam->list); + mutex_unlock(&steam_devices_lock); + steam->serial_no[0] = 0; } static void steam_work_connect_cb(struct work_struct *work) { struct steam_device *steam = container_of(work, struct steam_device, work_connect); + unsigned long flags; bool connected; + bool opened; int ret; spin_lock_irqsave(&steam->lock, flags); + opened = steam->client_opened; connected = steam->connected; spin_unlock_irqrestore(&steam->lock, flags); - if (connected) { + if (connected && !opened) { ret = steam_register(steam); - if (ret) { + if (ret) hid_err(steam->hdev, "%s:steam_register failed with error %d\n", __func__, ret); - } } else { steam_unregister(steam); } @@ -1049,53 +1084,30 @@ static void steam_mode_switch_cb(struct work_struct *work) struct steam_device, mode_switch); unsigned long flags; bool client_opened; + bool gamepad_mode; + + spin_lock_irqsave(&steam->lock, flags); steam->gamepad_mode = !steam->gamepad_mode; + gamepad_mode = steam->gamepad_mode; + client_opened = steam->client_opened; + spin_unlock_irqrestore(&steam->lock, flags); if (!lizard_mode) return; - if (steam->gamepad_mode) + guard(mutex)(&steam->report_mutex); + if (gamepad_mode) steam_set_lizard_mode(steam, false); - else { - spin_lock_irqsave(&steam->lock, flags); - client_opened = steam->client_opened; - spin_unlock_irqrestore(&steam->lock, flags); - if (!client_opened) - steam_set_lizard_mode(steam, lizard_mode); - } + else if (!client_opened) + steam_set_lizard_mode(steam, lizard_mode); steam_haptic_pulse(steam, STEAM_PAD_RIGHT, 0x190, 0, 1, 0); - if (steam->gamepad_mode) { + if (gamepad_mode) { steam_haptic_pulse(steam, STEAM_PAD_LEFT, 0x14D, 0x14D, 0x2D, 0); } else { steam_haptic_pulse(steam, STEAM_PAD_LEFT, 0x1F4, 0x1F4, 0x1E, 0); } } -static void steam_work_unregister_cb(struct work_struct *work) -{ - struct steam_device *steam = container_of(work, struct steam_device, - unregister_work); - unsigned long flags; - bool connected; - bool opened; - - spin_lock_irqsave(&steam->lock, flags); - opened = steam->client_opened; - connected = steam->connected; - spin_unlock_irqrestore(&steam->lock, flags); - - if (connected) { - if (opened) { - steam_sensors_unregister(steam); - steam_input_unregister(steam); - } else { - steam_set_lizard_mode(steam, lizard_mode); - steam_input_register(steam); - steam_sensors_register(steam); - } - } -} - static bool steam_is_valve_interface(struct hid_device *hdev) { struct hid_report_enum *rep_enum; @@ -1138,10 +1150,10 @@ static int steam_client_ll_open(struct hid_device *hdev) unsigned long flags; spin_lock_irqsave(&steam->lock, flags); - steam->client_opened = true; + steam->client_opened++; spin_unlock_irqrestore(&steam->lock, flags); - schedule_work(&steam->unregister_work); + schedule_work(&steam->work_connect); return 0; } @@ -1154,11 +1166,11 @@ static void steam_client_ll_close(struct hid_device *hdev) bool connected; spin_lock_irqsave(&steam->lock, flags); - steam->client_opened = false; + steam->client_opened--; connected = steam->connected && !steam->client_opened; spin_unlock_irqrestore(&steam->lock, flags); - schedule_work(&steam->unregister_work); + schedule_work(&steam->work_connect); } static int steam_client_ll_raw_request(struct hid_device *hdev, @@ -1244,13 +1256,13 @@ static int steam_probe(struct hid_device *hdev, hid_set_drvdata(hdev, steam); spin_lock_init(&steam->lock); mutex_init(&steam->report_mutex); + mutex_init(&steam->registration_mutex); steam->quirks = id->driver_data; INIT_WORK(&steam->work_connect, steam_work_connect_cb); INIT_DELAYED_WORK(&steam->mode_switch, steam_mode_switch_cb); INIT_LIST_HEAD(&steam->list); INIT_WORK(&steam->rumble_work, steam_haptic_rumble_cb); steam->sensor_timestamp_us = 0; - INIT_WORK(&steam->unregister_work, steam_work_unregister_cb); /* * With the real steam controller interface, do not connect hidraw. @@ -1311,7 +1323,6 @@ static int steam_probe(struct hid_device *hdev, cancel_work_sync(&steam->work_connect); cancel_delayed_work_sync(&steam->mode_switch); cancel_work_sync(&steam->rumble_work); - cancel_work_sync(&steam->unregister_work); return ret; } @@ -1319,36 +1330,32 @@ static int steam_probe(struct hid_device *hdev, static void steam_remove(struct hid_device *hdev) { struct steam_device *steam = hid_get_drvdata(hdev); + unsigned long flags; if (!steam || hdev->group == HID_GROUP_STEAM) { hid_hw_stop(hdev); return; } + hid_hw_close(hdev); hid_destroy_device(steam->client_hdev); - cancel_delayed_work_sync(&steam->mode_switch); + spin_lock_irqsave(&steam->lock, flags); + steam->client_opened = 0; + spin_unlock_irqrestore(&steam->lock, flags); cancel_work_sync(&steam->work_connect); - cancel_work_sync(&steam->rumble_work); - cancel_work_sync(&steam->unregister_work); - steam->client_hdev = NULL; - steam->client_opened = false; if (steam->quirks & STEAM_QUIRK_WIRELESS) { hid_info(hdev, "Steam wireless receiver disconnected"); } - hid_hw_close(hdev); - hid_hw_stop(hdev); steam_unregister(steam); + hid_hw_stop(hdev); } static void steam_do_connect_event(struct steam_device *steam, bool connected) { - unsigned long flags; bool changed; - spin_lock_irqsave(&steam->lock, flags); changed = steam->connected != connected; steam->connected = connected; - spin_unlock_irqrestore(&steam->lock, flags); if (changed && schedule_work(&steam->work_connect) == 0) dbg_hid("%s: connected=%d event already queued\n", @@ -1718,8 +1725,6 @@ static void steam_do_deck_sensors_event(struct steam_device *steam, static void steam_do_battery_event(struct steam_device *steam, struct power_supply *battery, u8 *data) { - unsigned long flags; - s16 volts = steam_le16(data + 12); u8 batt = data[14]; @@ -1727,10 +1732,8 @@ static void steam_do_battery_event(struct steam_device *steam, rcu_read_lock(); battery = rcu_dereference(steam->battery); if (likely(battery)) { - spin_lock_irqsave(&steam->lock, flags); steam->voltage = volts; steam->battery_charge = batt; - spin_unlock_irqrestore(&steam->lock, flags); power_supply_changed(battery); } rcu_read_unlock(); @@ -1748,6 +1751,7 @@ static int steam_raw_event(struct hid_device *hdev, if (!steam) return 0; + guard(spinlock_irqsave)(&steam->lock); if (steam->client_opened) hid_input_report(steam->client_hdev, HID_FEATURE_REPORT, data, size, 0); @@ -1832,6 +1836,8 @@ static int steam_param_set_lizard_mode(const char *val, { struct steam_device *steam; int ret; + bool client_opened; + unsigned long flags; ret = param_set_bool(val, kp); if (ret) @@ -1839,8 +1845,13 @@ static int steam_param_set_lizard_mode(const char *val, mutex_lock(&steam_devices_lock); list_for_each_entry(steam, &steam_devices, list) { - if (!steam->client_opened) + spin_lock_irqsave(&steam->lock, flags); + client_opened = steam->client_opened; + spin_unlock_irqrestore(&steam->lock, flags); + if (!client_opened) { + guard(mutex)(&steam->report_mutex); steam_set_lizard_mode(steam, lizard_mode); + } } mutex_unlock(&steam_devices_lock); return 0; diff --git a/drivers/hid/hid-universal-pidff.c b/drivers/hid/hid-universal-pidff.c index 7ef5ab9146b1..6806e3fe842a 100644 --- a/drivers/hid/hid-universal-pidff.c +++ b/drivers/hid/hid-universal-pidff.c @@ -100,12 +100,14 @@ static int universal_pidff_probe(struct hid_device *hdev, error = init_function(hdev, id->driver_data); if (error) { hid_warn(hdev, "Error initialising force feedback\n"); - goto err; + goto err_stop; } hid_info(hdev, "Universal pidff driver loaded sucesfully!"); return 0; +err_stop: + hid_hw_stop(hdev); err: return error; } diff --git a/drivers/hid/i2c-hid/i2c-hid-core.c b/drivers/hid/i2c-hid/i2c-hid-core.c index b36284e2d5cb..1760f2693d31 100644 --- a/drivers/hid/i2c-hid/i2c-hid-core.c +++ b/drivers/hid/i2c-hid/i2c-hid-core.c @@ -544,9 +544,10 @@ static void i2c_hid_get_input(struct i2c_hid *ihid) if (ihid->hid->group != HID_GROUP_RMI) pm_wakeup_event(&ihid->client->dev, 0); - hid_input_report(ihid->hid, HID_INPUT_REPORT, - ihid->inbuf + sizeof(__le16), - ret_size - sizeof(__le16), 1); + hid_safe_input_report(ihid->hid, HID_INPUT_REPORT, + ihid->inbuf + sizeof(__le16), + ihid->bufsize - sizeof(__le16), + ret_size - sizeof(__le16), 1); } return; diff --git a/drivers/hid/intel-ish-hid/ishtp/bus.c b/drivers/hid/intel-ish-hid/ishtp/bus.c index 4d97d043aae4..8d00b2816c81 100644 --- a/drivers/hid/intel-ish-hid/ishtp/bus.c +++ b/drivers/hid/intel-ish-hid/ishtp/bus.c @@ -730,6 +730,8 @@ void ishtp_bus_remove_all_clients(struct ishtp_device *ishtp_dev, spin_lock_irqsave(&ishtp_dev->cl_list_lock, flags); list_for_each_entry(cl, &ishtp_dev->cl_list, link) { cl->state = ISHTP_CL_DISCONNECTED; + if (warm_reset && cl->device && cl->device->reference_count) + continue; /* * Wake any pending process. The waiter would check dev->state diff --git a/drivers/hid/intel-ish-hid/ishtp/client.c b/drivers/hid/intel-ish-hid/ishtp/client.c index 2d92fc129ce4..af2562b8efce 100644 --- a/drivers/hid/intel-ish-hid/ishtp/client.c +++ b/drivers/hid/intel-ish-hid/ishtp/client.c @@ -49,7 +49,9 @@ static void ishtp_read_list_flush(struct ishtp_cl *cl) list_for_each_entry_safe(rb, next, &cl->dev->read_list.list, list) if (rb->cl && ishtp_cl_cmp_id(cl, rb->cl)) { list_del(&rb->list); - ishtp_io_rb_free(rb); + spin_lock(&cl->free_list_spinlock); + list_add_tail(&rb->list, &cl->free_rb_list.list); + spin_unlock(&cl->free_list_spinlock); } spin_unlock_irqrestore(&cl->dev->read_list_spinlock, flags); } diff --git a/drivers/hid/uhid.c b/drivers/hid/uhid.c index 4588d2cd4ea4..9027bb23f247 100644 --- a/drivers/hid/uhid.c +++ b/drivers/hid/uhid.c @@ -596,8 +596,8 @@ static int uhid_dev_input(struct uhid_device *uhid, struct uhid_event *ev) if (!READ_ONCE(uhid->running)) return -EINVAL; - hid_input_report(uhid->hid, HID_INPUT_REPORT, ev->u.input.data, - min_t(size_t, ev->u.input.size, UHID_DATA_MAX), 0); + hid_safe_input_report(uhid->hid, HID_INPUT_REPORT, ev->u.input.data, UHID_DATA_MAX, + min_t(size_t, ev->u.input.size, UHID_DATA_MAX), 0); return 0; } @@ -607,8 +607,8 @@ static int uhid_dev_input2(struct uhid_device *uhid, struct uhid_event *ev) if (!READ_ONCE(uhid->running)) return -EINVAL; - hid_input_report(uhid->hid, HID_INPUT_REPORT, ev->u.input2.data, - min_t(size_t, ev->u.input2.size, UHID_DATA_MAX), 0); + hid_safe_input_report(uhid->hid, HID_INPUT_REPORT, ev->u.input2.data, UHID_DATA_MAX, + min_t(size_t, ev->u.input2.size, UHID_DATA_MAX), 0); return 0; } diff --git a/drivers/hid/usbhid/hid-core.c b/drivers/hid/usbhid/hid-core.c index 415d44ac1480..f8287774c697 100644 --- a/drivers/hid/usbhid/hid-core.c +++ b/drivers/hid/usbhid/hid-core.c @@ -281,9 +281,9 @@ static void hid_irq_in(struct urb *urb) break; usbhid_mark_busy(usbhid); if (!test_bit(HID_RESUME_RUNNING, &usbhid->iofl)) { - hid_input_report(urb->context, HID_INPUT_REPORT, - urb->transfer_buffer, - urb->actual_length, 1); + hid_safe_input_report(urb->context, HID_INPUT_REPORT, + urb->transfer_buffer, urb->transfer_buffer_length, + urb->actual_length, 1); /* * autosuspend refused while keys are pressed * because most keyboards don't wake up when @@ -480,9 +480,10 @@ static void hid_ctrl(struct urb *urb) switch (status) { case 0: /* success */ if (usbhid->ctrl[usbhid->ctrltail].dir == USB_DIR_IN) - hid_input_report(urb->context, + hid_safe_input_report(urb->context, usbhid->ctrl[usbhid->ctrltail].report->type, - urb->transfer_buffer, urb->actual_length, 0); + urb->transfer_buffer, urb->transfer_buffer_length, + urb->actual_length, 0); break; case -ESHUTDOWN: /* unplug */ unplug = 1; diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c index da5c42201680..9df19a69feb6 100644 --- a/drivers/hv/vmbus_drv.c +++ b/drivers/hv/vmbus_drv.c @@ -530,34 +530,6 @@ static ssize_t device_show(struct device *dev, } static DEVICE_ATTR_RO(device); -static ssize_t driver_override_store(struct device *dev, - struct device_attribute *attr, - const char *buf, size_t count) -{ - struct hv_device *hv_dev = device_to_hv_device(dev); - int ret; - - ret = driver_set_override(dev, &hv_dev->driver_override, buf, count); - if (ret) - return ret; - - return count; -} - -static ssize_t driver_override_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - struct hv_device *hv_dev = device_to_hv_device(dev); - ssize_t len; - - device_lock(dev); - len = snprintf(buf, PAGE_SIZE, "%s\n", hv_dev->driver_override); - device_unlock(dev); - - return len; -} -static DEVICE_ATTR_RW(driver_override); - /* Set up per device attributes in /sys/bus/vmbus/devices/ */ static struct attribute *vmbus_dev_attrs[] = { &dev_attr_id.attr, @@ -588,7 +560,6 @@ static struct attribute *vmbus_dev_attrs[] = { &dev_attr_channel_vp_mapping.attr, &dev_attr_vendor.attr, &dev_attr_device.attr, - &dev_attr_driver_override.attr, NULL, }; @@ -700,9 +671,11 @@ static const struct hv_vmbus_device_id *hv_vmbus_get_id(struct hv_driver *drv, { const guid_t *guid = &dev->dev_type; const struct hv_vmbus_device_id *id; + int ret; - /* When driver_override is set, only bind to the matching driver */ - if (dev->driver_override && strcmp(dev->driver_override, drv->name)) + /* If a driver override is set, only bind to the matching driver */ + ret = device_match_driver_override(&dev->device, &drv->driver); + if (ret == 0) return NULL; /* Look at the dynamic ids first, before the static ones */ @@ -710,8 +683,11 @@ static const struct hv_vmbus_device_id *hv_vmbus_get_id(struct hv_driver *drv, if (!id) id = hv_vmbus_dev_match(drv->id_table, guid); - /* driver_override will always match, send a dummy id */ - if (!id && dev->driver_override) + /* + * If there's a matching driver override, this function should succeed, + * thus return a dummy device ID if no matching ID is found. + */ + if (!id && ret > 0) id = &vmbus_device_null; return id; @@ -990,6 +966,7 @@ static const struct dev_pm_ops vmbus_pm = { /* The one and only one */ static struct bus_type hv_bus = { .name = "vmbus", + .driver_override = true, .match = vmbus_match, .shutdown = vmbus_shutdown, .remove = vmbus_remove, @@ -2663,6 +2640,13 @@ static int __init hv_acpi_init(void) return -ENODEV; if (hv_root_partition && !hv_nested) + /* + * A non-nested root partition does not need VMBus client + * functionality. However, the mshv_root module may have + * a dependency on the VMBus module as described in + * commit 840b740a35bf. Return success so the module + * loads even though no VMBus initialization is done. + */ return 0; /* @@ -2711,6 +2695,14 @@ static void __exit vmbus_exit(void) { int cpu; + if (hv_root_partition && !hv_nested) + /* + * If a non-nested root partition loaded the VMBus module, + * hv_acpi_init() did not do any VMBus initialization. + * There's nothing to clean up, so just return. + */ + return; + unregister_syscore_ops(&hv_synic_syscore_ops); hv_remove_kexec_handler(); diff --git a/drivers/hwmon/lm90.c b/drivers/hwmon/lm90.c index 7be9874b3dfb..d3996950bd5d 100644 --- a/drivers/hwmon/lm90.c +++ b/drivers/hwmon/lm90.c @@ -695,6 +695,7 @@ struct lm90_data { struct mutex update_lock; struct delayed_work alert_work; struct work_struct report_work; + bool shutdown; /* true if shutting down */ bool valid; /* true if register values are valid */ bool alarms_valid; /* true if status register values are valid */ unsigned long last_updated; /* in jiffies */ @@ -1113,6 +1114,9 @@ static void lm90_report_alarms(struct work_struct *work) static int lm90_update_alarms_locked(struct lm90_data *data, bool force) { + if (data->shutdown) + return 0; + if (force || !data->alarms_valid || time_after(jiffies, data->alarms_updated + msecs_to_jiffies(data->update_interval))) { struct i2c_client *client = data->client; @@ -2558,15 +2562,23 @@ static void lm90_restore_conf(void *_data) struct lm90_data *data = _data; struct i2c_client *client = data->client; - cancel_delayed_work_sync(&data->alert_work); - cancel_work_sync(&data->report_work); - /* Restore initial configuration */ if (data->flags & LM90_HAVE_CONVRATE) lm90_write_convrate(data, data->convrate_orig); lm90_write_reg(client, LM90_REG_CONFIG1, data->config_orig); } +static void lm90_stop_work(void *_data) +{ + struct lm90_data *data = _data; + + mutex_lock(&data->update_lock); + data->shutdown = true; + mutex_unlock(&data->update_lock); + cancel_delayed_work_sync(&data->alert_work); + cancel_work_sync(&data->report_work); +} + static int lm90_init_client(struct i2c_client *client, struct lm90_data *data) { struct device_node *np = client->dev.of_node; @@ -2899,6 +2911,10 @@ static int lm90_probe(struct i2c_client *client) data->hwmon_dev = hwmon_dev; + err = devm_add_action_or_reset(&client->dev, lm90_stop_work, data); + if (err) + return err; + if (client->irq) { dev_dbg(dev, "IRQ: %d\n", client->irq); err = devm_request_threaded_irq(dev, client->irq, @@ -2927,7 +2943,8 @@ static void lm90_alert(struct i2c_client *client, enum i2c_alert_protocol type, */ struct lm90_data *data = i2c_get_clientdata(client); - if ((data->flags & LM90_HAVE_BROKEN_ALERT) && + mutex_lock(&data->update_lock); + if (!data->shutdown && (data->flags & LM90_HAVE_BROKEN_ALERT) && (data->current_alarms & data->alert_alarms)) { if (!(data->config & 0x80)) { dev_dbg(&client->dev, "Disabling ALERT#\n"); @@ -2936,6 +2953,7 @@ static void lm90_alert(struct i2c_client *client, enum i2c_alert_protocol type, schedule_delayed_work(&data->alert_work, max_t(int, HZ, msecs_to_jiffies(data->update_interval))); } + mutex_unlock(&data->update_lock); } else { dev_dbg(&client->dev, "Everything OK\n"); } diff --git a/drivers/hwmon/pmbus/pmbus_core.c b/drivers/hwmon/pmbus/pmbus_core.c index 4be4c396b900..45720413e409 100644 --- a/drivers/hwmon/pmbus/pmbus_core.c +++ b/drivers/hwmon/pmbus/pmbus_core.c @@ -34,7 +34,7 @@ struct pmbus_sensor { struct pmbus_sensor *next; char name[PMBUS_NAME_SIZE]; /* sysfs sensor name */ - struct device_attribute attribute; + struct sensor_device_attribute attribute; u8 page; /* page number */ u8 phase; /* phase number, 0xff for all phases */ u16 reg; /* register */ @@ -58,7 +58,7 @@ struct pmbus_boolean { struct pmbus_label { char name[PMBUS_NAME_SIZE]; /* sysfs label name */ - struct device_attribute attribute; + struct sensor_device_attribute attribute; char label[PMBUS_NAME_SIZE]; /* label */ }; #define to_pmbus_label(_attr) \ @@ -1023,9 +1023,24 @@ static u16 pmbus_data2reg_direct(struct pmbus_data *data, static u16 pmbus_data2reg_vid(struct pmbus_data *data, struct pmbus_sensor *sensor, s64 val) { - val = clamp_val(val, 500, 1600); - - return 2 + DIV_ROUND_CLOSEST_ULL((1600LL - val) * 100LL, 625); + switch (data->info->vrm_version[sensor->page]) { + case vr12: + val = clamp_val(val, 250, 1520); + return 1 + DIV_ROUND_CLOSEST_ULL(val - 250, 5); + case vr13: + val = clamp_val(val, 500, 3040); + return 1 + DIV_ROUND_CLOSEST_ULL(val - 500, 10); + case imvp9: + val = clamp_val(val, 200, 2740); + return 1 + DIV_ROUND_CLOSEST_ULL(val - 200, 10); + case amd625mv: + val = clamp_val(val, 200, 1550); + return DIV_ROUND_CLOSEST_ULL((1550LL - val) * 100LL, 625); + case vr11: + default: + val = clamp_val(val, 500, 1600); + return 2 + DIV_ROUND_CLOSEST_ULL((1600LL - val) * 100LL, 625); + } } static u16 pmbus_data2reg(struct pmbus_data *data, @@ -1155,7 +1170,8 @@ static ssize_t pmbus_show_sensor(struct device *dev, struct device_attribute *devattr, char *buf) { struct i2c_client *client = to_i2c_client(dev->parent); - struct pmbus_sensor *sensor = to_pmbus_sensor(devattr); + struct sensor_device_attribute *attr = to_sensor_dev_attr(devattr); + struct pmbus_sensor *sensor = to_pmbus_sensor(attr); struct pmbus_data *data = i2c_get_clientdata(client); ssize_t ret; @@ -1175,7 +1191,8 @@ static ssize_t pmbus_set_sensor(struct device *dev, { struct i2c_client *client = to_i2c_client(dev->parent); struct pmbus_data *data = i2c_get_clientdata(client); - struct pmbus_sensor *sensor = to_pmbus_sensor(devattr); + struct sensor_device_attribute *attr = to_sensor_dev_attr(devattr); + struct pmbus_sensor *sensor = to_pmbus_sensor(attr); ssize_t rv = count; s64 val; int ret; @@ -1198,7 +1215,8 @@ static ssize_t pmbus_set_sensor(struct device *dev, static ssize_t pmbus_show_label(struct device *dev, struct device_attribute *da, char *buf) { - struct pmbus_label *label = to_pmbus_label(da); + struct sensor_device_attribute *attr = to_sensor_dev_attr(da); + struct pmbus_label *label = to_pmbus_label(attr); return sysfs_emit(buf, "%s\n", label->label); } @@ -1351,8 +1369,8 @@ static struct pmbus_sensor *pmbus_add_sensor(struct pmbus_data *data, bool update, bool readonly, bool convert) { + struct sensor_device_attribute *a; struct pmbus_sensor *sensor; - struct device_attribute *a; sensor = devm_kzalloc(data->dev, sizeof(*sensor), GFP_KERNEL); if (!sensor) @@ -1376,11 +1394,10 @@ static struct pmbus_sensor *pmbus_add_sensor(struct pmbus_data *data, sensor->update = update; sensor->convert = convert; sensor->data = -ENODATA; - pmbus_dev_attr_init(a, sensor->name, - readonly ? 0444 : 0644, - pmbus_show_sensor, pmbus_set_sensor); + pmbus_attr_init(a, sensor->name, readonly ? 0444 : 0644, + pmbus_show_sensor, pmbus_set_sensor, -1); - if (pmbus_add_attribute(data, &a->attr)) + if (pmbus_add_attribute(data, &a->dev_attr.attr)) return NULL; sensor->next = data->sensors; @@ -1397,8 +1414,8 @@ static int pmbus_add_label(struct pmbus_data *data, const char *name, int seq, const char *lstring, int index, int phase) { + struct sensor_device_attribute *a; struct pmbus_label *label; - struct device_attribute *a; label = devm_kzalloc(data->dev, sizeof(*label), GFP_KERNEL); if (!label) @@ -1423,8 +1440,8 @@ static int pmbus_add_label(struct pmbus_data *data, lstring, index, phase); } - pmbus_dev_attr_init(a, label->name, 0444, pmbus_show_label, NULL); - return pmbus_add_attribute(data, &a->attr); + pmbus_attr_init(a, label->name, 0444, pmbus_show_label, NULL, -1); + return pmbus_add_attribute(data, &a->dev_attr.attr); } /* @@ -2268,7 +2285,7 @@ struct pmbus_samples_attr { struct pmbus_samples_reg { int page; struct pmbus_samples_attr *attr; - struct device_attribute dev_attr; + struct sensor_device_attribute attribute; }; static struct pmbus_samples_attr pmbus_samples_registers[] = { @@ -2290,14 +2307,15 @@ static struct pmbus_samples_attr pmbus_samples_registers[] = { } }; -#define to_samples_reg(x) container_of(x, struct pmbus_samples_reg, dev_attr) +#define to_samples_reg(x) container_of(x, struct pmbus_samples_reg, attribute) static ssize_t pmbus_show_samples(struct device *dev, struct device_attribute *devattr, char *buf) { int val; struct i2c_client *client = to_i2c_client(dev->parent); - struct pmbus_samples_reg *reg = to_samples_reg(devattr); + struct sensor_device_attribute *attr = to_sensor_dev_attr(devattr); + struct pmbus_samples_reg *reg = to_samples_reg(attr); struct pmbus_data *data = i2c_get_clientdata(client); mutex_lock(&data->update_lock); @@ -2316,7 +2334,8 @@ static ssize_t pmbus_set_samples(struct device *dev, int ret; long val; struct i2c_client *client = to_i2c_client(dev->parent); - struct pmbus_samples_reg *reg = to_samples_reg(devattr); + struct sensor_device_attribute *attr = to_sensor_dev_attr(devattr); + struct pmbus_samples_reg *reg = to_samples_reg(attr); struct pmbus_data *data = i2c_get_clientdata(client); if (kstrtol(buf, 0, &val) < 0) @@ -2332,6 +2351,7 @@ static ssize_t pmbus_set_samples(struct device *dev, static int pmbus_add_samples_attr(struct pmbus_data *data, int page, struct pmbus_samples_attr *attr) { + struct sensor_device_attribute *a; struct pmbus_samples_reg *reg; reg = devm_kzalloc(data->dev, sizeof(*reg), GFP_KERNEL); @@ -2341,10 +2361,12 @@ static int pmbus_add_samples_attr(struct pmbus_data *data, int page, reg->attr = attr; reg->page = page; - pmbus_dev_attr_init(®->dev_attr, attr->name, 0644, - pmbus_show_samples, pmbus_set_samples); + a = ®->attribute; - return pmbus_add_attribute(data, ®->dev_attr.attr); + pmbus_attr_init(a, attr->name, 0644, + pmbus_show_samples, pmbus_set_samples, -1); + + return pmbus_add_attribute(data, &a->dev_attr.attr); } static int pmbus_add_samples_attributes(struct i2c_client *client, @@ -2799,9 +2821,15 @@ static void pmbus_notify(struct pmbus_data *data, int page, int reg, int flags) struct device_attribute *da = to_dev_attr(data->group.attrs[i]); struct sensor_device_attribute *attr = to_sensor_dev_attr(da); int index = attr->index; - u16 smask = pb_index_to_mask(index); - u8 spage = pb_index_to_page(index); - u16 sreg = pb_index_to_reg(index); + u16 smask, sreg; + u8 spage; + + if (index == -1) + continue; + + smask = pb_index_to_mask(index); + spage = pb_index_to_page(index); + sreg = pb_index_to_reg(index); if (reg == sreg && page == spage && (smask & flags)) { dev_dbg(data->dev, "sysfs notify: %s", da->attr.name); diff --git a/drivers/hwmon/pwm-fan.c b/drivers/hwmon/pwm-fan.c index b67bc9e833c0..90254f8eb56f 100644 --- a/drivers/hwmon/pwm-fan.c +++ b/drivers/hwmon/pwm-fan.c @@ -466,12 +466,18 @@ static void pwm_fan_cleanup(void *__ctx) { struct pwm_fan_ctx *ctx = __ctx; - del_timer_sync(&ctx->rpm_timer); /* Switch off everything */ ctx->enable_mode = pwm_disable_reg_disable; pwm_fan_power_off(ctx); } +static void pwm_fan_timer_cleanup(void *__ctx) +{ + struct pwm_fan_ctx *ctx = __ctx; + + timer_shutdown_sync(&ctx->rpm_timer); +} + static int pwm_fan_probe(struct platform_device *pdev) { struct thermal_cooling_device *cdev; @@ -606,6 +612,10 @@ static int pwm_fan_probe(struct platform_device *pdev) } if (ctx->tach_count > 0) { + ret = devm_add_action_or_reset(dev, pwm_fan_timer_cleanup, ctx); + if (ret) + return ret; + ctx->sample_start = ktime_get(); mod_timer(&ctx->rpm_timer, jiffies + HZ); @@ -647,6 +657,7 @@ static void pwm_fan_shutdown(struct platform_device *pdev) { struct pwm_fan_ctx *ctx = platform_get_drvdata(pdev); + pwm_fan_timer_cleanup(ctx); pwm_fan_cleanup(ctx); } diff --git a/drivers/hwmon/sht3x.c b/drivers/hwmon/sht3x.c index d8a86e60cf8c..0f757ef4bbe9 100644 --- a/drivers/hwmon/sht3x.c +++ b/drivers/hwmon/sht3x.c @@ -20,6 +20,7 @@ #include #include #include +#include /* commands (high repeatability mode) */ static const unsigned char sht3x_cmd_measure_single_hpm[] = { 0x24, 0x00 }; @@ -273,9 +274,9 @@ static struct sht3x_data *sht3x_update_client(struct device *dev) if (ret) goto out; - val = be16_to_cpup((__be16 *)buf); + val = get_unaligned_be16(buf); data->temperature = sht3x_extract_temperature(val); - val = be16_to_cpup((__be16 *)(buf + 3)); + val = get_unaligned_be16(buf + 3); data->humidity = sht3x_extract_humidity(val); data->last_update = jiffies; } @@ -333,7 +334,7 @@ static int limits_update(struct sht3x_data *data) if (ret) return ret; - raw = be16_to_cpup((__be16 *)buffer); + raw = get_unaligned_be16(buffer); temperature = sht3x_extract_temperature((raw & 0x01ff) << 7); humidity = sht3x_extract_humidity(raw & 0xfe00); data->temperature_limits[index] = temperature; @@ -386,7 +387,7 @@ static size_t limit_write(struct device *dev, raw = ((u32)(temperature + 45000) * 24543) >> (16 + 7); raw |= ((humidity * 42950) >> 16) & 0xfe00; - *((__be16 *)position) = cpu_to_be16(raw); + put_unaligned_be16(raw, position); position += SHT3X_WORD_LEN; *position = crc8(sht3x_crc8_table, position - SHT3X_WORD_LEN, diff --git a/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c b/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c index 2f271b7fb048..c88c64900698 100644 --- a/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c +++ b/drivers/hwtracing/coresight/coresight-etm3x-sysfs.c @@ -717,26 +717,19 @@ static DEVICE_ATTR_RW(cntr_rld_event); static ssize_t cntr_val_show(struct device *dev, struct device_attribute *attr, char *buf) { - int i, ret = 0; u32 val; struct etm_drvdata *drvdata = dev_get_drvdata(dev->parent); struct etm_config *config = &drvdata->config; if (!local_read(&drvdata->mode)) { spin_lock(&drvdata->spinlock); - for (i = 0; i < drvdata->nr_cntr; i++) - ret += sprintf(buf, "counter %d: %x\n", - i, config->cntr_val[i]); + val = config->cntr_val[config->cntr_idx]; spin_unlock(&drvdata->spinlock); - return ret; - } - - for (i = 0; i < drvdata->nr_cntr; i++) { - val = etm_readl(drvdata, ETMCNTVRn(i)); - ret += sprintf(buf, "counter %d: %x\n", i, val); + } else { + val = etm_readl(drvdata, ETMCNTVRn(config->cntr_idx)); } - return ret; + return sysfs_emit(buf, "%#x\n", val); } static ssize_t cntr_val_store(struct device *dev, diff --git a/drivers/hwtracing/coresight/coresight-etm4x-core.c b/drivers/hwtracing/coresight/coresight-etm4x-core.c index ace8027d7cb7..593dcfe4c945 100644 --- a/drivers/hwtracing/coresight/coresight-etm4x-core.c +++ b/drivers/hwtracing/coresight/coresight-etm4x-core.c @@ -54,10 +54,14 @@ MODULE_PARM_DESC(boot_enable, "Enable tracing on boot"); #define PARAM_PM_SAVE_NEVER 1 /* never save any state */ #define PARAM_PM_SAVE_SELF_HOSTED 2 /* save self-hosted state only */ +/* + * Save option for ETM4. ETE, sysreg ETM4s and ACPI boots ignore this option and + * will always save. + */ static int pm_save_enable = PARAM_PM_SAVE_FIRMWARE; module_param(pm_save_enable, int, 0444); MODULE_PARM_DESC(pm_save_enable, - "Save/restore state on power down: 1 = never, 2 = self-hosted"); + "Save/restore state on power down: 1 = never, 2 = self-hosted. MMIO and DT only."); static struct etmv4_drvdata *etmdrvdata[NR_CPUS]; static void etm4_set_default_config(struct etmv4_config *config); @@ -694,8 +698,6 @@ static int etm4_parse_event_config(struct coresight_device *csdev, struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); struct etmv4_config *config = &drvdata->config; struct perf_event_attr *attr = &event->attr; - unsigned long cfg_hash; - int preset; /* Clear configuration from previous run */ memset(config, 0, sizeof(struct etmv4_config)); @@ -765,20 +767,6 @@ static int etm4_parse_event_config(struct coresight_device *csdev, /* bit[12], Return stack enable bit */ config->cfg |= TRCCONFIGR_RS; - /* - * Set any selected configuration and preset. - * - * This extracts the values of PMU_FORMAT_ATTR(configid) and PMU_FORMAT_ATTR(preset) - * in the perf attributes defined in coresight-etm-perf.c. - * configid uses bits 63:32 of attr->config2, preset uses bits 3:0 of attr->config. - * A zero configid means no configuration active, preset = 0 means no preset selected. - */ - if (attr->config2 & GENMASK_ULL(63, 32)) { - cfg_hash = (u32)(attr->config2 >> 32); - preset = attr->config & 0xF; - ret = cscfg_csdev_enable_active_config(csdev, cfg_hash, preset); - } - /* branch broadcast - enable if selected and supported */ if (attr->config & BIT(ETM_OPT_BRANCH_BROADCAST)) { if (!drvdata->trcbb) { @@ -800,8 +788,10 @@ static int etm4_parse_event_config(struct coresight_device *csdev, static int etm4_enable_perf(struct coresight_device *csdev, struct perf_event *event) { - int ret = 0, trace_id; + int ret = 0, trace_id, preset; struct etmv4_drvdata *drvdata = dev_get_drvdata(csdev->dev.parent); + struct perf_event_attr *attr = &event->attr; + unsigned long cfg_hash = 0; if (WARN_ON_ONCE(drvdata->cpu != smp_processor_id())) { ret = -EINVAL; @@ -813,6 +803,22 @@ static int etm4_enable_perf(struct coresight_device *csdev, if (ret) goto out; + /* + * Set any selected configuration and preset. + * + * This extracts the values of PMU_FORMAT_ATTR(configid) and PMU_FORMAT_ATTR(preset) + * in the perf attributes defined in coresight-etm-perf.c. + * configid uses bits 63:32 of attr->config2, preset uses bits 3:0 of attr->config. + * A zero configid means no configuration active, preset = 0 means no preset selected. + */ + if (attr->config2 & GENMASK_ULL(63, 32)) { + cfg_hash = (u32)(attr->config2 >> 32); + preset = attr->config & 0xF; + ret = cscfg_csdev_enable_active_config(csdev, cfg_hash, preset); + if (ret) + goto out; + } + /* * perf allocates cpu ids as part of _setup_aux() - device needs to use * the allocated ID. This reads the current version without allocation. @@ -826,13 +832,20 @@ static int etm4_enable_perf(struct coresight_device *csdev, dev_err(&drvdata->csdev->dev, "Failed to set trace ID for %s on CPU%d\n", dev_name(&drvdata->csdev->dev), drvdata->cpu); ret = -EINVAL; - goto out; + goto err_cfg; } drvdata->trcid = (u8)trace_id; /* And enable it */ ret = etm4_enable_hw(drvdata); + if (ret) + goto err_cfg; + return 0; + +err_cfg: + if (cfg_hash) + cscfg_csdev_disable_active_config(csdev); out: return ret; } @@ -2075,6 +2088,17 @@ static void etm4_pm_clear(void) } } +static bool etm4x_always_pm_save(struct device *dev, struct csdev_access *csa) +{ + /* + * Only IO mem ETM devices will benefit from skipping PM save and only + * DT has the option to control it, not ACPI. Otherwise system register + * based ETMs and ETEs will always lose context on CPU power down, so + * always save. + */ + return !csa->io_mem || is_acpi_device_node(dev_fwnode(dev)); +} + static int etm4_add_coresight_dev(struct etm4_init_arg *init_arg) { int ret; @@ -2084,6 +2108,7 @@ static int etm4_add_coresight_dev(struct etm4_init_arg *init_arg) struct coresight_desc desc = { 0 }; u8 major, minor; char *type_name; + bool pm_save; if (!drvdata) return -EINVAL; @@ -2111,6 +2136,21 @@ static int etm4_add_coresight_dev(struct etm4_init_arg *init_arg) etm4_set_default(&drvdata->config); + if (etm4x_always_pm_save(dev, init_arg->csa)) + pm_save = true; + else if (pm_save_enable == PARAM_PM_SAVE_FIRMWARE) + pm_save = coresight_loses_context_with_cpu(dev); + else + pm_save = pm_save_enable != PARAM_PM_SAVE_NEVER; + + if (pm_save) { + drvdata->save_state = devm_kmalloc(dev, + sizeof(struct etmv4_save_state), + GFP_KERNEL); + if (!drvdata->save_state) + return -ENOMEM; + } + pdata = coresight_get_platform_data(dev); if (IS_ERR(pdata)) return PTR_ERR(pdata); @@ -2167,17 +2207,6 @@ static int etm4_probe(struct device *dev) if (IS_ERR(drvdata->atclk)) return PTR_ERR(drvdata->atclk); - if (pm_save_enable == PARAM_PM_SAVE_FIRMWARE) - pm_save_enable = coresight_loses_context_with_cpu(dev) ? - PARAM_PM_SAVE_SELF_HOSTED : PARAM_PM_SAVE_NEVER; - - if (pm_save_enable != PARAM_PM_SAVE_NEVER) { - drvdata->save_state = devm_kmalloc(dev, - sizeof(struct etmv4_save_state), GFP_KERNEL); - if (!drvdata->save_state) - return -ENOMEM; - } - spin_lock_init(&drvdata->spinlock); drvdata->cpu = coresight_get_cpu(dev); diff --git a/drivers/hwtracing/coresight/coresight-syscfg.c b/drivers/hwtracing/coresight/coresight-syscfg.c index 30a561d87481..55f36ed1af13 100644 --- a/drivers/hwtracing/coresight/coresight-syscfg.c +++ b/drivers/hwtracing/coresight/coresight-syscfg.c @@ -395,6 +395,8 @@ static void cscfg_remove_owned_csdev_configs(struct coresight_device *csdev, voi if (list_empty(&csdev->config_csdev_list)) return; + guard(spinlock_irqsave)(&csdev->cscfg_csdev_lock); + list_for_each_entry_safe(config_csdev, tmp, &csdev->config_csdev_list, node) { if (config_csdev->config_desc->load_owner == load_owner) list_del(&config_csdev->node); diff --git a/drivers/hwtracing/coresight/coresight-tmc-etr.c b/drivers/hwtracing/coresight/coresight-tmc-etr.c index f3312fbcdc0f..f8e59cf69b93 100644 --- a/drivers/hwtracing/coresight/coresight-tmc-etr.c +++ b/drivers/hwtracing/coresight/coresight-tmc-etr.c @@ -1156,6 +1156,13 @@ static struct etr_buf *tmc_etr_get_sysfs_buffer(struct coresight_device *csdev) * with the lock released. */ spin_lock_irqsave(&drvdata->spinlock, flags); + + /* + * If the ETR is already enabled, continue with the existing buffer. + */ + if (drvdata->mode == CS_MODE_SYSFS) + goto out; + sysfs_buf = READ_ONCE(drvdata->sysfs_buf); if (!sysfs_buf || (sysfs_buf->size != drvdata->size)) { spin_unlock_irqrestore(&drvdata->spinlock, flags); @@ -1205,6 +1212,15 @@ static int tmc_enable_etr_sink_sysfs(struct coresight_device *csdev) spin_lock_irqsave(&drvdata->spinlock, flags); + /* + * Since the sysfs buffer allocation and the hardware enablement is not + * in the same critical region, it's possible to race with the perf. + */ + if (drvdata->mode == CS_MODE_PERF) { + spin_unlock_irqrestore(&drvdata->spinlock, flags); + return -EBUSY; + } + /* * In sysFS mode we can have multiple writers per sink. Since this * sink is already enabled no memory is needed and the HW need not be diff --git a/drivers/i2c/busses/i2c-at91-master.c b/drivers/i2c/busses/i2c-at91-master.c index bb9874c08401..4737feedb940 100644 --- a/drivers/i2c/busses/i2c-at91-master.c +++ b/drivers/i2c/busses/i2c-at91-master.c @@ -922,8 +922,14 @@ int at91_twi_probe_master(struct platform_device *pdev, at91_calc_twi_clock(dev); rc = at91_init_twi_recovery_info(pdev, dev); - if (rc == -EPROBE_DEFER) + if (rc == -EPROBE_DEFER) { + /* + * The DMA channels were already set up above, release them + * before deferring the probe so they are not leaked. + */ + at91_twi_dma_release(dev); return rc; + } dev->adapter.algo = &at91_twi_algorithm; dev->adapter.quirks = &at91_twi_quirks; diff --git a/drivers/i2c/busses/i2c-mxs.c b/drivers/i2c/busses/i2c-mxs.c index 36def0a9c95c..e3b1ee4c4195 100644 --- a/drivers/i2c/busses/i2c-mxs.c +++ b/drivers/i2c/busses/i2c-mxs.c @@ -831,7 +831,7 @@ static int mxs_i2c_probe(struct platform_device *pdev) } /* Setup the DMA */ - i2c->dmach = dma_request_chan(dev, "rx-tx"); + i2c->dmach = devm_dma_request_chan(dev, "rx-tx"); if (IS_ERR(i2c->dmach)) { return dev_err_probe(dev, PTR_ERR(i2c->dmach), "Failed to request dma\n"); @@ -869,9 +869,6 @@ static void mxs_i2c_remove(struct platform_device *pdev) i2c_del_adapter(&i2c->adapter); - if (i2c->dmach) - dma_release_channel(i2c->dmach); - writel(MXS_I2C_CTRL0_SFTRST, i2c->regs + MXS_I2C_CTRL0_SET); } diff --git a/drivers/i2c/busses/i2c-xiic.c b/drivers/i2c/busses/i2c-xiic.c index 1d68177241a6..5e8521f86dd9 100644 --- a/drivers/i2c/busses/i2c-xiic.c +++ b/drivers/i2c/busses/i2c-xiic.c @@ -72,6 +72,9 @@ enum i2c_scl_freq { * @prev_msg_tx: Previous message is Tx * @quirks: To hold platform specific bug info * @smbus_block_read: Flag to handle block read + * @smbus_actual_len: Valid byte count (length + payload + optional PEC) of a + * padded SMBus block read. msg->len is trimmed to this on completion. + * Zero when no trimming is needed. * @input_clk: Input clock to I2C controller * @i2c_clk: I2C SCL frequency */ @@ -94,6 +97,7 @@ struct xiic_i2c { bool prev_msg_tx; u32 quirks; bool smbus_block_read; + unsigned int smbus_actual_len; unsigned long input_clk; unsigned int i2c_clk; }; @@ -238,6 +242,29 @@ static const struct timing_regs timing_reg_values[] = { static int xiic_start_xfer(struct xiic_i2c *i2c, struct i2c_msg *msgs, int num); static void __xiic_start_xfer(struct xiic_i2c *i2c); +static int __maybe_unused xiic_i2c_runtime_suspend(struct device *dev) +{ + struct xiic_i2c *i2c = dev_get_drvdata(dev); + + clk_disable(i2c->clk); + + return 0; +} + +static int __maybe_unused xiic_i2c_runtime_resume(struct device *dev) +{ + struct xiic_i2c *i2c = dev_get_drvdata(dev); + int ret; + + ret = clk_enable(i2c->clk); + if (ret) { + dev_err(dev, "Cannot enable clock.\n"); + return ret; + } + + return 0; +} + /* * For the register read and write functions, a little-endian and big-endian * version are necessary. Endianness is detected during the probe function. @@ -505,30 +532,38 @@ static void xiic_smbus_block_read_setup(struct xiic_i2c *i2c) /* Check if received length is valid */ if (rxmsg_len <= I2C_SMBUS_BLOCK_MAX) { + unsigned int pec_len = i2c->rx_msg->len - 1; + /* Set Receive fifo depth */ - if (rxmsg_len > IIC_RX_FIFO_DEPTH) { + if (rxmsg_len + pec_len > IIC_RX_FIFO_DEPTH) { /* - * When Rx msg len greater than or equal to Rx fifo capacity - * Receive fifo depth should set to Rx fifo capacity minus 1 + * Trailing bytes (payload plus any PEC) exceed Rx FIFO + * capacity, so drain in chunks. This also keeps the + * else branch below from pushing rfd_set past the + * 4-bit XIIC_RFD_REG_OFFSET field. */ rfd_set = IIC_RX_FIFO_DEPTH - 1; - i2c->rx_msg->len = rxmsg_len + 1; - } else if ((rxmsg_len == 1) || - (rxmsg_len == 0)) { + i2c->rx_msg->len = rxmsg_len + 1 + pec_len; + } else if (1 + rxmsg_len + pec_len < SMBUS_BLOCK_READ_MIN_LEN) { /* - * Minimum of 3 bytes required to exit cleanly. 1 byte - * already received, Second byte is being received. Have - * to set NACK in read_rx before receiving the last byte + * The HW needs SMBUS_BLOCK_READ_MIN_LEN bytes on the + * bus to exit cleanly: by the time the ISR reads the + * length byte the second byte is already being clocked + * in, too late to NACK. Pad the drain target and record + * the real length, trimmed back on completion so the + * PEC check sees the right byte. */ rfd_set = 0; i2c->rx_msg->len = SMBUS_BLOCK_READ_MIN_LEN; + i2c->smbus_actual_len = 1 + rxmsg_len + pec_len; } else { /* - * When Rx msg len less than Rx fifo capacity - * Receive fifo depth should set to Rx msg len minus 2 + * All trailing bytes fit in the Rx FIFO. The widened + * condition above guarantees rxmsg_len + pec_len >= 2, + * so this cannot underflow. */ - rfd_set = rxmsg_len - 2; - i2c->rx_msg->len = rxmsg_len + 1; + rfd_set = rxmsg_len + pec_len - 2; + i2c->rx_msg->len = rxmsg_len + 1 + pec_len; } xiic_setreg8(i2c, XIIC_RFD_REG_OFFSET, rfd_set); @@ -737,6 +772,10 @@ static irqreturn_t xiic_process(int irq, void *dev_id) xiic_read_rx(i2c); if (xiic_rx_space(i2c) == 0) { + /* Undo padding so the PEC check sees the right byte. */ + if (i2c->smbus_actual_len) + i2c->rx_msg->len = i2c->smbus_actual_len; + /* this is the last part of the message */ i2c->rx_msg = NULL; @@ -805,8 +844,11 @@ static irqreturn_t xiic_process(int irq, void *dev_id) if (i2c->tx_msg && i2c->smbus_block_read) { i2c->smbus_block_read = false; - /* Set requested message len=1 to indicate STATE_DONE */ - i2c->tx_msg->len = 1; + /* + * Drive xiic_tx_space() to 0 to signal STATE_DONE + * without truncating the rx_msg length. + */ + i2c->tx_pos = i2c->tx_msg->len; } if (!i2c->tx_msg) @@ -867,6 +909,9 @@ static void xiic_start_recv(struct xiic_i2c *i2c) u8 cr = 0, rfd_set = 0; struct i2c_msg *msg = i2c->rx_msg = i2c->tx_msg; + /* A stale value from an aborted block read would truncate this msg. */ + i2c->smbus_actual_len = 0; + dev_dbg(i2c->adap.dev.parent, "%s entry, ISR: 0x%x, CR: 0x%x\n", __func__, xiic_getreg32(i2c, XIIC_IISR_OFFSET), xiic_getreg8(i2c, XIIC_CR_REG_OFFSET)); @@ -1365,29 +1410,6 @@ static void xiic_i2c_remove(struct platform_device *pdev) pm_runtime_dont_use_autosuspend(&pdev->dev); } -static int __maybe_unused xiic_i2c_runtime_suspend(struct device *dev) -{ - struct xiic_i2c *i2c = dev_get_drvdata(dev); - - clk_disable(i2c->clk); - - return 0; -} - -static int __maybe_unused xiic_i2c_runtime_resume(struct device *dev) -{ - struct xiic_i2c *i2c = dev_get_drvdata(dev); - int ret; - - ret = clk_enable(i2c->clk); - if (ret) { - dev_err(dev, "Cannot enable clock.\n"); - return ret; - } - - return 0; -} - static const struct dev_pm_ops xiic_dev_pm_ops = { SET_RUNTIME_PM_OPS(xiic_i2c_runtime_suspend, xiic_i2c_runtime_resume, NULL) diff --git a/drivers/i3c/master.c b/drivers/i3c/master.c index 330f87c65bff..35d02234471b 100644 --- a/drivers/i3c/master.c +++ b/drivers/i3c/master.c @@ -667,6 +667,11 @@ static struct attribute *i3c_masterdev_attrs[] = { }; ATTRIBUTE_GROUPS(i3c_masterdev); +static void i3c_master_free_i3c_dev(struct i3c_dev_desc *dev) +{ + kfree(dev); +} + static void i3c_masterdev_release(struct device *dev) { struct i3c_master_controller *master = dev_to_i3cmaster(dev); @@ -679,6 +684,8 @@ static void i3c_masterdev_release(struct device *dev) i3c_bus_cleanup(bus); of_node_put(dev->of_node); + + i3c_master_free_i3c_dev(master->this); } static const struct device_type i3c_masterdev_type = { @@ -866,11 +873,6 @@ static void i3c_device_release(struct device *dev) kfree(i3cdev); } -static void i3c_master_free_i3c_dev(struct i3c_dev_desc *dev) -{ - kfree(dev); -} - static struct i3c_dev_desc * i3c_master_alloc_i3c_dev(struct i3c_master_controller *master, const struct i3c_device_info *info) @@ -1618,12 +1620,16 @@ static int i3c_master_early_i3c_dev_add(struct i3c_master_controller *master, static void i3c_master_register_new_i3c_devs(struct i3c_master_controller *master) { + struct i3c_device *i3cdev, *tmp; struct i3c_dev_desc *desc; + LIST_HEAD(i3c_unreg_devs); int ret; if (!master->init_done) return; + i3c_bus_maintenance_lock(&master->bus); + i3c_bus_for_each_i3cdev(&master->bus, desc) { if (desc->dev || !desc->info.dyn_addr || desc == master->this) continue; @@ -1644,15 +1650,30 @@ i3c_master_register_new_i3c_devs(struct i3c_master_controller *master) if (desc->boardinfo) desc->dev->dev.of_node = desc->boardinfo->of_node; - ret = device_register(&desc->dev->dev); - if (ret) { - dev_err(&master->dev, - "Failed to add I3C device (err = %d)\n", ret); - desc->dev->desc = NULL; - put_device(&desc->dev->dev); - desc->dev = NULL; - } + list_add_tail(&desc->dev->node, &i3c_unreg_devs); } + + i3c_bus_maintenance_unlock(&master->bus); + + list_for_each_entry_safe(i3cdev, tmp, &i3c_unreg_devs, node) { + ret = device_register(&i3cdev->dev); + if (ret) + dev_err(&master->dev, "Failed to add I3C device (err = %d)\n", ret); + else + list_del_init(&i3cdev->node); + } + + i3c_bus_maintenance_lock(&master->bus); + + list_for_each_entry_safe(i3cdev, tmp, &i3c_unreg_devs, node) { + list_del(&i3cdev->node); + desc = i3cdev->desc; + i3cdev->desc = NULL; + put_device(&i3cdev->dev); + desc->dev = NULL; + } + + i3c_bus_maintenance_unlock(&master->bus); } /** @@ -1681,9 +1702,7 @@ int i3c_master_do_daa(struct i3c_master_controller *master) if (ret) return ret; - i3c_bus_normaluse_lock(&master->bus); i3c_master_register_new_i3c_devs(master); - i3c_bus_normaluse_unlock(&master->bus); return 0; } @@ -1743,6 +1762,8 @@ int i3c_master_set_info(struct i3c_master_controller *master, return 0; err_free_dev: + master->bus.cur_master = NULL; + master->this = NULL; i3c_master_free_i3c_dev(i3cdev); return ret; @@ -1763,7 +1784,8 @@ static void i3c_master_detach_free_devs(struct i3c_master_controller *master) i3cdev->boardinfo->init_dyn_addr, I3C_ADDR_SLOT_FREE); - i3c_master_free_i3c_dev(i3cdev); + if (i3cdev != master->this) + i3c_master_free_i3c_dev(i3cdev); } list_for_each_entry_safe(i2cdev, i2ctmp, &master->bus.devs.i2c, @@ -1983,7 +2005,8 @@ i3c_master_search_i3c_dev_duplicate(struct i3c_dev_desc *refdev) struct i3c_dev_desc *i3cdev; i3c_bus_for_each_i3cdev(&master->bus, i3cdev) { - if (i3cdev != refdev && i3cdev->info.pid == refdev->info.pid) + if (i3cdev != refdev && i3cdev->info.pid == refdev->info.pid && + i3cdev != master->this) return i3cdev; } @@ -2851,9 +2874,7 @@ int i3c_master_register(struct i3c_master_controller *master, * register I3C devices discovered during the initial DAA. */ master->init_done = true; - i3c_bus_normaluse_lock(&master->bus); i3c_master_register_new_i3c_devs(master); - i3c_bus_normaluse_unlock(&master->bus); return 0; diff --git a/drivers/i3c/master/mipi-i3c-hci/core.c b/drivers/i3c/master/mipi-i3c-hci/core.c index 10c400da6640..6cc89cf01c66 100644 --- a/drivers/i3c/master/mipi-i3c-hci/core.c +++ b/drivers/i3c/master/mipi-i3c-hci/core.c @@ -747,6 +747,9 @@ static int i3c_hci_probe(struct platform_device *pdev) hci = devm_kzalloc(&pdev->dev, sizeof(*hci), GFP_KERNEL); if (!hci) return -ENOMEM; + + mutex_init(&hci->control_mutex); + hci->base_regs = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(hci->base_regs)) return PTR_ERR(hci->base_regs); diff --git a/drivers/i3c/master/mipi-i3c-hci/dma.c b/drivers/i3c/master/mipi-i3c-hci/dma.c index 2cf4974fffa7..60da4be20c5f 100644 --- a/drivers/i3c/master/mipi-i3c-hci/dma.c +++ b/drivers/i3c/master/mipi-i3c-hci/dma.c @@ -448,16 +448,25 @@ static bool hci_dma_dequeue_xfer(struct i3c_hci *hci, struct hci_rh_data *rh = &rings->headers[xfer_list[0].ring_number]; unsigned int i; bool did_unqueue = false; - - /* stop the ring */ - rh_reg_write(RING_CONTROL, RING_CTRL_ABORT); - if (wait_for_completion_timeout(&rh->op_done, HZ) == 0) { - /* - * We're deep in it if ever this condition is ever met. - * Hardware might still be writing to memory, etc. - */ - dev_crit(&hci->master.dev, "unable to abort the ring\n"); - WARN_ON(1); + u32 ring_status; + + guard(mutex)(&hci->control_mutex); + + ring_status = rh_reg_read(RING_STATUS); + if (ring_status & RING_STATUS_RUNNING) { + /* stop the ring */ + reinit_completion(&rh->op_done); + rh_reg_write(RING_CONTROL, RING_CTRL_ENABLE | RING_CTRL_ABORT); + wait_for_completion_timeout(&rh->op_done, HZ); + ring_status = rh_reg_read(RING_STATUS); + if (ring_status & RING_STATUS_RUNNING) { + /* + * We're deep in it if ever this condition is ever met. + * Hardware might still be writing to memory, etc. + */ + dev_crit(&hci->master.dev, "unable to abort the ring\n"); + WARN_ON(1); + } } for (i = 0; i < n; i++) { diff --git a/drivers/i3c/master/mipi-i3c-hci/hci.h b/drivers/i3c/master/mipi-i3c-hci/hci.h index f109923f6c3f..39490bc9f84f 100644 --- a/drivers/i3c/master/mipi-i3c-hci/hci.h +++ b/drivers/i3c/master/mipi-i3c-hci/hci.h @@ -43,6 +43,7 @@ struct i3c_hci { const struct hci_io_ops *io; void *io_data; const struct hci_cmd_ops *cmd; + struct mutex control_mutex; atomic_t next_cmd_tid; u32 caps; unsigned int quirks; diff --git a/drivers/iio/accel/kionix-kx022a.c b/drivers/iio/accel/kionix-kx022a.c index 220465d9c713..1f23130a84e5 100644 --- a/drivers/iio/accel/kionix-kx022a.c +++ b/drivers/iio/accel/kionix-kx022a.c @@ -507,7 +507,7 @@ static int kx022a_read_raw(struct iio_dev *idev, if (ret) return ret; - if ((regval & KX022A_MASK_ODR) > + if ((regval & KX022A_MASK_ODR) >= ARRAY_SIZE(kx022a_accel_samp_freq_table)) { dev_err(data->dev, "Invalid ODR\n"); return -EINVAL; diff --git a/drivers/iio/accel/kxcjk-1013.c b/drivers/iio/accel/kxcjk-1013.c index 894709286b0c..f3b912c2d39c 100644 --- a/drivers/iio/accel/kxcjk-1013.c +++ b/drivers/iio/accel/kxcjk-1013.c @@ -1050,7 +1050,7 @@ static int kxcjk1013_write_event_config(struct iio_dev *indio_dev, struct kxcjk1013_data *data = iio_priv(indio_dev); int ret; - if (state && data->ev_enable_state) + if (state == data->ev_enable_state) return 0; mutex_lock(&data->mutex); diff --git a/drivers/iio/accel/sca3000.c b/drivers/iio/accel/sca3000.c index 2b87f7f5508b..4b0497f63f83 100644 --- a/drivers/iio/accel/sca3000.c +++ b/drivers/iio/accel/sca3000.c @@ -652,7 +652,7 @@ static int sca3000_write_raw_samp_freq(struct sca3000_state *st, int val) if (val == base_freq / 2) ctrlval |= SCA3000_REG_OUT_CTRL_BUF_DIV_2; - if (val == base_freq / 4) + else if (val == base_freq / 4) ctrlval |= SCA3000_REG_OUT_CTRL_BUF_DIV_4; else if (val != base_freq) return -EINVAL; diff --git a/drivers/iio/adc/ad_sigma_delta.c b/drivers/iio/adc/ad_sigma_delta.c index 914274ed899e..45e0f638c0ed 100644 --- a/drivers/iio/adc/ad_sigma_delta.c +++ b/drivers/iio/adc/ad_sigma_delta.c @@ -343,9 +343,8 @@ EXPORT_SYMBOL_NS_GPL(ad_sigma_delta_single_conversion, IIO_AD_SIGMA_DELTA); static int ad_sd_buffer_postenable(struct iio_dev *indio_dev) { struct ad_sigma_delta *sigma_delta = iio_device_get_drvdata(indio_dev); - unsigned int i, slot, samples_buf_size; + unsigned int i, slot; unsigned int channel; - uint8_t *samples_buf; int ret; if (sigma_delta->num_slots == 1) { @@ -378,15 +377,6 @@ static int ad_sd_buffer_postenable(struct iio_dev *indio_dev) return ret; } - samples_buf_size = ALIGN(slot * indio_dev->channels[0].scan_type.storagebits / 8, 8); - samples_buf_size += sizeof(int64_t); - samples_buf = devm_krealloc(&sigma_delta->spi->dev, sigma_delta->samples_buf, - samples_buf_size, GFP_KERNEL); - if (!samples_buf) - return -ENOMEM; - - sigma_delta->samples_buf = samples_buf; - spi_bus_lock(sigma_delta->spi->master); sigma_delta->bus_locked = true; sigma_delta->keep_cs_asserted = true; @@ -620,6 +610,7 @@ static int devm_ad_sd_probe_trigger(struct device *dev, struct iio_dev *indio_de int devm_ad_sd_setup_buffer_and_trigger(struct device *dev, struct iio_dev *indio_dev) { struct ad_sigma_delta *sigma_delta = iio_device_get_drvdata(indio_dev); + unsigned int samples_buf_size; int ret; sigma_delta->slots = devm_kcalloc(dev, sigma_delta->num_slots, @@ -627,6 +618,19 @@ int devm_ad_sd_setup_buffer_and_trigger(struct device *dev, struct iio_dev *indi if (!sigma_delta->slots) return -ENOMEM; + /* + * Worst-case size: all sequencer slots can be active, capped + * at num_slots by ad_sd_validate_scan_mask(). + */ + samples_buf_size = + ALIGN(sigma_delta->num_slots * + BITS_TO_BYTES(indio_dev->channels[0].scan_type.storagebits), + sizeof(s64)); + samples_buf_size += sizeof(s64); + sigma_delta->samples_buf = devm_kzalloc(dev, samples_buf_size, GFP_KERNEL); + if (!sigma_delta->samples_buf) + return -ENOMEM; + ret = devm_iio_triggered_buffer_setup(dev, indio_dev, &iio_pollfunc_store_time, &ad_sd_trigger_handler, diff --git a/drivers/iio/adc/aspeed_adc.c b/drivers/iio/adc/aspeed_adc.c index 998e8bcc06e1..4cb2210cb38d 100644 --- a/drivers/iio/adc/aspeed_adc.c +++ b/drivers/iio/adc/aspeed_adc.c @@ -549,12 +549,13 @@ static int aspeed_adc_probe(struct platform_device *pdev) return PTR_ERR(data->clk_scaler); data->rst = devm_reset_control_get_shared(&pdev->dev, NULL); - if (IS_ERR(data->rst)) { - dev_err(&pdev->dev, - "invalid or missing reset controller device tree entry"); - return PTR_ERR(data->rst); - } - reset_control_deassert(data->rst); + if (IS_ERR(data->rst)) + return dev_err_probe(&pdev->dev, PTR_ERR(data->rst), + "invalid or missing reset controller device tree entry"); + + ret = reset_control_deassert(data->rst); + if (ret) + return ret; ret = devm_add_action_or_reset(data->dev, aspeed_adc_reset_assert, data->rst); diff --git a/drivers/iio/adc/axp288_adc.c b/drivers/iio/adc/axp288_adc.c index 49fff1cabd0d..3b7612dcaba6 100644 --- a/drivers/iio/adc/axp288_adc.c +++ b/drivers/iio/adc/axp288_adc.c @@ -207,6 +207,14 @@ static const struct dmi_system_id axp288_adc_ts_bias_override[] = { }, .driver_data = (void *)(uintptr_t)AXP288_ADC_TS_BIAS_80UA, }, + { + /* Haier HV103H */ + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "Haier"), + DMI_MATCH(DMI_PRODUCT_NAME, "HV103H"), + }, + .driver_data = (void *)(uintptr_t)AXP288_ADC_TS_BIAS_80UA, + }, {} }; diff --git a/drivers/iio/adc/max1363.c b/drivers/iio/adc/max1363.c index 5304235d733f..d5afb9440d1b 100644 --- a/drivers/iio/adc/max1363.c +++ b/drivers/iio/adc/max1363.c @@ -406,6 +406,14 @@ static int max1363_read_single_chan(struct iio_dev *indio_dev, } data = rxbuf[0]; } + + /* + * Differential channels are bipolar and the device returns the sample + * in two's complement, so sign-extend it from the resolution bit. + */ + if (chan->differential) + data = sign_extend32(data, st->chip_info->bits - 1); + *val = data; error_ret: diff --git a/drivers/iio/adc/stm32-adc.c b/drivers/iio/adc/stm32-adc.c index 0b3e487440a6..3b78581c5e78 100644 --- a/drivers/iio/adc/stm32-adc.c +++ b/drivers/iio/adc/stm32-adc.c @@ -1478,11 +1478,16 @@ static int stm32_adc_read_raw(struct iio_dev *indio_dev, ret = stm32_adc_single_conv(indio_dev, chan, val); else ret = -EINVAL; + iio_device_release_direct_mode(indio_dev); + if (ret < 0) + return ret; - if (mask == IIO_CHAN_INFO_PROCESSED) + if (mask == IIO_CHAN_INFO_PROCESSED) { + if (*val == 0) + return -EINVAL; *val = STM32_ADC_VREFINT_VOLTAGE * adc->vrefint.vrefint_cal / *val; + } - iio_device_release_direct_mode(indio_dev); return ret; case IIO_CHAN_INFO_SCALE: @@ -2127,33 +2132,37 @@ static int stm32_adc_populate_int_ch(struct iio_dev *indio_dev, const char *ch_n for (i = 0; i < STM32_ADC_INT_CH_NB; i++) { if (!strncmp(stm32_adc_ic[i].name, ch_name, STM32_ADC_CH_SZ)) { + bool na; + /* Check internal channel availability */ switch (i) { case STM32_ADC_INT_CH_VDDCORE: - if (!adc->cfg->regs->or_vddcore.reg) - dev_warn(&indio_dev->dev, - "%s channel not available\n", ch_name); + na = !adc->cfg->regs->or_vddcore.reg; break; case STM32_ADC_INT_CH_VDDCPU: - if (!adc->cfg->regs->or_vddcpu.reg) - dev_warn(&indio_dev->dev, - "%s channel not available\n", ch_name); + na = !adc->cfg->regs->or_vddcpu.reg; break; case STM32_ADC_INT_CH_VDDQ_DDR: - if (!adc->cfg->regs->or_vddq_ddr.reg) - dev_warn(&indio_dev->dev, - "%s channel not available\n", ch_name); + na = !adc->cfg->regs->or_vddq_ddr.reg; break; case STM32_ADC_INT_CH_VREFINT: - if (!adc->cfg->regs->ccr_vref.reg) - dev_warn(&indio_dev->dev, - "%s channel not available\n", ch_name); + na = !adc->cfg->regs->ccr_vref.reg; break; case STM32_ADC_INT_CH_VBAT: - if (!adc->cfg->regs->ccr_vbat.reg) - dev_warn(&indio_dev->dev, - "%s channel not available\n", ch_name); + na = !adc->cfg->regs->ccr_vbat.reg; break; + default: + return -EINVAL; + } + + if (na) { + /* + * Channel label matches an internal STM32 ADC channel. + * Warn about it, as there's normally no restriction on the + * name but that's not among available internal channels. + */ + dev_warn(&indio_dev->dev, "no %s internal channel\n", ch_name); + return 0; } if (stm32_adc_ic[i].idx != STM32_ADC_INT_CH_VREFINT) { diff --git a/drivers/iio/adc/sun4i-gpadc-iio.c b/drivers/iio/adc/sun4i-gpadc-iio.c index 25bba96367a8..3a96296ef7d5 100644 --- a/drivers/iio/adc/sun4i-gpadc-iio.c +++ b/drivers/iio/adc/sun4i-gpadc-iio.c @@ -644,10 +644,10 @@ static int sun4i_gpadc_probe(struct platform_device *pdev) * thermal because no thermal DT node is found. */ if (IS_ERR(info->tzd) && PTR_ERR(info->tzd) != -ENODEV) { - dev_err(&pdev->dev, - "could not register thermal sensor: %ld\n", - PTR_ERR(info->tzd)); - return PTR_ERR(info->tzd); + ret = dev_err_probe(&pdev->dev, + PTR_ERR(info->tzd), + "could not register thermal sensor\n"); + goto err_map; } } @@ -663,7 +663,6 @@ static int sun4i_gpadc_probe(struct platform_device *pdev) if (!info->no_irq && IS_ENABLED(CONFIG_THERMAL_OF)) iio_map_array_unregister(indio_dev); - pm_runtime_put(&pdev->dev); pm_runtime_disable(&pdev->dev); return ret; @@ -674,7 +673,6 @@ static int sun4i_gpadc_remove(struct platform_device *pdev) struct iio_dev *indio_dev = platform_get_drvdata(pdev); struct sun4i_gpadc_iio *info = iio_priv(indio_dev); - pm_runtime_put(&pdev->dev); pm_runtime_disable(&pdev->dev); if (!IS_ENABLED(CONFIG_THERMAL_OF)) diff --git a/drivers/iio/adc/xilinx-xadc-core.c b/drivers/iio/adc/xilinx-xadc-core.c index c6bedffbb47e..fbd2ddb8f7b1 100644 --- a/drivers/iio/adc/xilinx-xadc-core.c +++ b/drivers/iio/adc/xilinx-xadc-core.c @@ -1403,13 +1403,14 @@ static int xadc_probe(struct platform_device *pdev) } if (irq > 0) { - ret = devm_request_irq(dev, irq, xadc->ops->interrupt_handler, - 0, dev_name(dev), indio_dev); + /* Clear out any work queued by the interrupt */ + ret = devm_add_action_or_reset(dev, xadc_cancel_delayed_work, + &xadc->zynq_unmask_work); if (ret) return ret; - ret = devm_add_action_or_reset(dev, xadc_cancel_delayed_work, - &xadc->zynq_unmask_work); + ret = devm_request_irq(dev, irq, xadc->ops->interrupt_handler, + 0, dev_name(dev), indio_dev); if (ret) return ret; } diff --git a/drivers/iio/cdc/ad7150.c b/drivers/iio/cdc/ad7150.c index 4c03b9e834b8..0417dffd350a 100644 --- a/drivers/iio/cdc/ad7150.c +++ b/drivers/iio/cdc/ad7150.c @@ -637,11 +637,13 @@ static const struct i2c_device_id ad7150_id[] = { MODULE_DEVICE_TABLE(i2c, ad7150_id); static const struct of_device_id ad7150_of_match[] = { - { "adi,ad7150" }, - { "adi,ad7151" }, - { "adi,ad7156" }, + { .compatible = "adi,ad7150" }, + { .compatible = "adi,ad7151" }, + { .compatible = "adi,ad7156" }, {} }; +MODULE_DEVICE_TABLE(of, ad7150_of_match); + static struct i2c_driver ad7150_driver = { .driver = { .name = "ad7150", diff --git a/drivers/iio/frequency/admv1013.c b/drivers/iio/frequency/admv1013.c index 92923074f930..8b521f371e87 100644 --- a/drivers/iio/frequency/admv1013.c +++ b/drivers/iio/frequency/admv1013.c @@ -85,9 +85,9 @@ enum { }; enum { - ADMV1013_SE_MODE_POS = 6, - ADMV1013_SE_MODE_NEG = 9, - ADMV1013_SE_MODE_DIFF = 12 + ADMV1013_SE_MODE_POS, + ADMV1013_SE_MODE_NEG, + ADMV1013_SE_MODE_DIFF, }; struct admv1013_state { @@ -95,7 +95,6 @@ struct admv1013_state { struct clk *clkin; /* Protect against concurrent accesses to the device and to data */ struct mutex lock; - struct regulator *reg; struct notifier_block nb; unsigned int input_mode; unsigned int quad_se_mode; @@ -197,7 +196,7 @@ static int admv1013_read_raw(struct iio_dev *indio_dev, switch (info) { case IIO_CHAN_INFO_CALIBBIAS: - switch (chan->channel) { + switch (chan->channel2) { case IIO_MOD_I: addr = ADMV1013_REG_OFFSET_ADJUST_I; break; @@ -342,14 +341,9 @@ static int admv1013_update_quad_filters(struct admv1013_state *st) FIELD_PREP(ADMV1013_QUAD_FILTERS_MSK, filt_raw)); } -static int admv1013_update_mixer_vgate(struct admv1013_state *st) +static int admv1013_update_mixer_vgate(struct admv1013_state *st, int vcm) { unsigned int mixer_vgate; - int vcm; - - vcm = regulator_get_voltage(st->reg); - if (vcm < 0) - return vcm; if (vcm <= 1800000) mixer_vgate = (2389 * vcm / 1000000 + 8100) / 100; @@ -443,7 +437,7 @@ static const struct iio_chan_spec admv1013_channels[] = { ADMV1013_CHAN_CALIB(1, Q), }; -static int admv1013_init(struct admv1013_state *st) +static int admv1013_init(struct admv1013_state *st, int vcm_uv) { int ret; unsigned int data; @@ -476,14 +470,27 @@ static int admv1013_init(struct admv1013_state *st) if (ret) return ret; - data = FIELD_PREP(ADMV1013_QUAD_SE_MODE_MSK, st->quad_se_mode); + switch (st->quad_se_mode) { + case ADMV1013_SE_MODE_POS: + data = 6; + break; + case ADMV1013_SE_MODE_NEG: + data = 9; + break; + case ADMV1013_SE_MODE_DIFF: + data = 12; + break; + default: + return -EINVAL; + } ret = __admv1013_spi_update_bits(st, ADMV1013_REG_QUAD, - ADMV1013_QUAD_SE_MODE_MSK, data); + ADMV1013_QUAD_SE_MODE_MSK, + FIELD_PREP(ADMV1013_QUAD_SE_MODE_MSK, data)); if (ret) return ret; - ret = admv1013_update_mixer_vgate(st); + ret = admv1013_update_mixer_vgate(st, vcm_uv); if (ret) return ret; @@ -498,11 +505,6 @@ static int admv1013_init(struct admv1013_state *st) st->input_mode); } -static void admv1013_reg_disable(void *data) -{ - regulator_disable(data); -} - static void admv1013_powerdown(void *data) { unsigned int enable_reg, enable_reg_msk; @@ -525,48 +527,39 @@ static void admv1013_powerdown(void *data) admv1013_spi_update_bits(data, ADMV1013_REG_ENABLE, enable_reg_msk, enable_reg); } +static const char * const admv1013_input_modes[] = { + [ADMV1013_IQ_MODE] = "iq", + [ADMV1013_IF_MODE] = "if", +}; + +static const char * const admv1013_quad_se_modes[] = { + [ADMV1013_SE_MODE_POS] = "se-pos", + [ADMV1013_SE_MODE_NEG] = "se-neg", + [ADMV1013_SE_MODE_DIFF] = "diff", +}; + static int admv1013_properties_parse(struct admv1013_state *st) { int ret; - const char *str; - struct spi_device *spi = st->spi; + struct device *dev = &st->spi->dev; - st->det_en = device_property_read_bool(&spi->dev, "adi,detector-enable"); + st->det_en = device_property_read_bool(dev, "adi,detector-enable"); - ret = device_property_read_string(&spi->dev, "adi,input-mode", &str); - if (ret) - st->input_mode = ADMV1013_IQ_MODE; - - if (!strcmp(str, "iq")) - st->input_mode = ADMV1013_IQ_MODE; - else if (!strcmp(str, "if")) - st->input_mode = ADMV1013_IF_MODE; - else - return -EINVAL; - - ret = device_property_read_string(&spi->dev, "adi,quad-se-mode", &str); - if (ret) - st->quad_se_mode = ADMV1013_SE_MODE_DIFF; - - if (!strcmp(str, "diff")) - st->quad_se_mode = ADMV1013_SE_MODE_DIFF; - else if (!strcmp(str, "se-pos")) - st->quad_se_mode = ADMV1013_SE_MODE_POS; - else if (!strcmp(str, "se-neg")) - st->quad_se_mode = ADMV1013_SE_MODE_NEG; - else - return -EINVAL; + ret = device_property_match_property_string(dev, "adi,input-mode", + admv1013_input_modes, + ARRAY_SIZE(admv1013_input_modes)); + st->input_mode = ret >= 0 ? ret : ADMV1013_IQ_MODE; - st->reg = devm_regulator_get(&spi->dev, "vcm"); - if (IS_ERR(st->reg)) - return dev_err_probe(&spi->dev, PTR_ERR(st->reg), - "failed to get the common-mode voltage\n"); + ret = device_property_match_property_string(dev, "adi,quad-se-mode", + admv1013_quad_se_modes, + ARRAY_SIZE(admv1013_quad_se_modes)); + st->quad_se_mode = ret >= 0 ? ret : ADMV1013_SE_MODE_DIFF; - ret = devm_regulator_bulk_get_enable(&st->spi->dev, + ret = devm_regulator_bulk_get_enable(dev, ARRAY_SIZE(admv1013_vcc_regs), admv1013_vcc_regs); if (ret) { - dev_err_probe(&spi->dev, ret, + dev_err_probe(dev, ret, "Failed to request VCC regulators\n"); return ret; } @@ -578,9 +571,10 @@ static int admv1013_probe(struct spi_device *spi) { struct iio_dev *indio_dev; struct admv1013_state *st; - int ret; + struct device *dev = &spi->dev; + int ret, vcm_uv; - indio_dev = devm_iio_device_alloc(&spi->dev, sizeof(*st)); + indio_dev = devm_iio_device_alloc(dev, sizeof(*st)); if (!indio_dev) return -ENOMEM; @@ -597,40 +591,36 @@ static int admv1013_probe(struct spi_device *spi) if (ret) return ret; - ret = regulator_enable(st->reg); - if (ret) { - dev_err(&spi->dev, "Failed to enable specified Common-Mode Voltage!\n"); - return ret; - } + ret = devm_regulator_get_enable_read_voltage(dev, "vcm"); + if (ret < 0) + return dev_err_probe(dev, ret, + "failed to get the common-mode voltage\n"); - ret = devm_add_action_or_reset(&spi->dev, admv1013_reg_disable, - st->reg); - if (ret) - return ret; + vcm_uv = ret; - st->clkin = devm_clk_get_enabled(&spi->dev, "lo_in"); + st->clkin = devm_clk_get_enabled(dev, "lo_in"); if (IS_ERR(st->clkin)) - return dev_err_probe(&spi->dev, PTR_ERR(st->clkin), + return dev_err_probe(dev, PTR_ERR(st->clkin), "failed to get the LO input clock\n"); st->nb.notifier_call = admv1013_freq_change; - ret = devm_clk_notifier_register(&spi->dev, st->clkin, &st->nb); + ret = devm_clk_notifier_register(dev, st->clkin, &st->nb); if (ret) return ret; mutex_init(&st->lock); - ret = admv1013_init(st); + ret = admv1013_init(st, vcm_uv); if (ret) { dev_err(&spi->dev, "admv1013 init failed\n"); return ret; } - ret = devm_add_action_or_reset(&spi->dev, admv1013_powerdown, st); + ret = devm_add_action_or_reset(dev, admv1013_powerdown, st); if (ret) return ret; - return devm_iio_device_register(&spi->dev, indio_dev); + return devm_iio_device_register(dev, indio_dev); } static const struct spi_device_id admv1013_id[] = { diff --git a/drivers/iio/gyro/adis16136.c b/drivers/iio/gyro/adis16136.c index c95cf41be34b..1faaaf74671a 100644 --- a/drivers/iio/gyro/adis16136.c +++ b/drivers/iio/gyro/adis16136.c @@ -145,7 +145,7 @@ static int adis16136_debugfs_init(struct iio_dev *indio_dev) struct adis16136 *adis16136 = iio_priv(indio_dev); struct dentry *d = iio_get_debugfs_dentry(indio_dev); - debugfs_create_file_unsafe("serial_number", 0400, + debugfs_create_file("serial_number", 0400, d, adis16136, &adis16136_serial_fops); debugfs_create_file_unsafe("product_id", 0400, d, adis16136, &adis16136_product_id_fops); diff --git a/drivers/iio/health/max30102.c b/drivers/iio/health/max30102.c index 37e619827e8a..9419cc62a2a4 100644 --- a/drivers/iio/health/max30102.c +++ b/drivers/iio/health/max30102.c @@ -292,9 +292,15 @@ static irqreturn_t max30102_interrupt_handler(int irq, void *private) { struct iio_dev *indio_dev = private; struct max30102_data *data = iio_priv(indio_dev); - unsigned int measurements = bitmap_weight(indio_dev->active_scan_mask, - indio_dev->masklength); - int ret, cnt = 0; + unsigned int measurements; + int ret, cnt; + + cnt = max30102_fifo_count(data); + if (cnt <= 0) + return IRQ_HANDLED; + + measurements = bitmap_weight(indio_dev->active_scan_mask, + iio_get_masklength(indio_dev)); mutex_lock(&data->lock); diff --git a/drivers/iio/imu/adis16400.c b/drivers/iio/imu/adis16400.c index 3eda32e12a53..eac7cfead5a7 100644 --- a/drivers/iio/imu/adis16400.c +++ b/drivers/iio/imu/adis16400.c @@ -279,7 +279,7 @@ static int adis16400_debugfs_init(struct iio_dev *indio_dev) struct dentry *d = iio_get_debugfs_dentry(indio_dev); if (st->variant->flags & ADIS16400_HAS_SERIAL_NUMBER) - debugfs_create_file_unsafe("serial_number", 0400, + debugfs_create_file("serial_number", 0400, d, st, &adis16400_serial_number_fops); if (st->variant->flags & ADIS16400_HAS_PROD_ID) debugfs_create_file_unsafe("product_id", 0400, diff --git a/drivers/iio/imu/adis16480.c b/drivers/iio/imu/adis16480.c index fe520194a837..ef78f7074576 100644 --- a/drivers/iio/imu/adis16480.c +++ b/drivers/iio/imu/adis16480.c @@ -302,9 +302,9 @@ static int adis16480_debugfs_init(struct iio_dev *indio_dev) struct adis16480 *adis16480 = iio_priv(indio_dev); struct dentry *d = iio_get_debugfs_dentry(indio_dev); - debugfs_create_file_unsafe("firmware_revision", 0400, + debugfs_create_file("firmware_revision", 0400, d, adis16480, &adis16480_firmware_revision_fops); - debugfs_create_file_unsafe("firmware_date", 0400, + debugfs_create_file("firmware_date", 0400, d, adis16480, &adis16480_firmware_date_fops); debugfs_create_file_unsafe("serial_number", 0400, d, adis16480, &adis16480_serial_number_fops); diff --git a/drivers/iio/industrialio-buffer.c b/drivers/iio/industrialio-buffer.c index fc04c55da6c7..29b073698636 100644 --- a/drivers/iio/industrialio-buffer.c +++ b/drivers/iio/industrialio-buffer.c @@ -1294,6 +1294,10 @@ EXPORT_SYMBOL_GPL(iio_update_buffers); void iio_disable_all_buffers(struct iio_dev *indio_dev) { + struct iio_dev_opaque *iio_dev_opaque = to_iio_dev_opaque(indio_dev); + + guard(mutex)(&iio_dev_opaque->mlock); + iio_disable_buffers(indio_dev); iio_buffer_deactivate_all(indio_dev); } @@ -1913,7 +1917,8 @@ int iio_push_to_buffers_with_ts_unaligned(struct iio_dev *indio_dev, bb = devm_krealloc(&indio_dev->dev, iio_dev_opaque->bounce_buffer, - indio_dev->scan_bytes, GFP_KERNEL); + indio_dev->scan_bytes, + GFP_KERNEL | __GFP_ZERO); if (!bb) return -ENOMEM; iio_dev_opaque->bounce_buffer = bb; diff --git a/drivers/iio/industrialio-trigger.c b/drivers/iio/industrialio-trigger.c index b5fed8a000ea..56378f98d5c0 100644 --- a/drivers/iio/industrialio-trigger.c +++ b/drivers/iio/industrialio-trigger.c @@ -519,6 +519,8 @@ static void iio_trig_release(struct device *device) struct iio_trigger *trig = to_iio_trigger(device); int i; + cancel_work_sync(&trig->reenable_work); + if (trig->subirq_base) { for (i = 0; i < CONFIG_IIO_CONSUMERS_PER_TRIGGER; i++) { irq_modify_status(trig->subirq_base + i, diff --git a/drivers/iio/light/Kconfig b/drivers/iio/light/Kconfig index ddbd1838650e..c9710ba31705 100644 --- a/drivers/iio/light/Kconfig +++ b/drivers/iio/light/Kconfig @@ -305,20 +305,6 @@ config ROHM_BU27008 Typical application is adjusting LCD backlight of TVs, mobile phones and tablet PCs. -config ROHM_BU27034 - tristate "ROHM BU27034 ambient light sensor" - depends on I2C - select REGMAP_I2C - select IIO_GTS_HELPER - select IIO_BUFFER - select IIO_KFIFO_BUF - help - Enable support for the ROHM BU27034 ambient light sensor. ROHM BU27034 - is an ambient light sesnor with 3 channels and 3 photo diodes capable - of detecting a very wide range of illuminance. - Typical application is adjusting LCD and backlight power of TVs and - mobile phones. - config RPR0521 tristate "ROHM RPR0521 ALS and proximity sensor driver" depends on I2C diff --git a/drivers/iio/light/Makefile b/drivers/iio/light/Makefile index c0db4c4c36ec..789e5ca0d3b8 100644 --- a/drivers/iio/light/Makefile +++ b/drivers/iio/light/Makefile @@ -40,7 +40,6 @@ obj-$(CONFIG_OPT3001) += opt3001.o obj-$(CONFIG_OPT4001) += opt4001.o obj-$(CONFIG_PA12203001) += pa12203001.o obj-$(CONFIG_ROHM_BU27008) += rohm-bu27008.o -obj-$(CONFIG_ROHM_BU27034) += rohm-bu27034.o obj-$(CONFIG_RPR0521) += rpr0521.o obj-$(CONFIG_SI1133) += si1133.o obj-$(CONFIG_SI1145) += si1145.o diff --git a/drivers/iio/light/gp2ap020a00f.c b/drivers/iio/light/gp2ap020a00f.c index 9f41724819b6..8a3c6c3aab86 100644 --- a/drivers/iio/light/gp2ap020a00f.c +++ b/drivers/iio/light/gp2ap020a00f.c @@ -1565,6 +1565,7 @@ static int gp2ap020a00f_probe(struct i2c_client *client) iio_trigger_unregister(data->trig); error_free_irq: free_irq(client->irq, indio_dev); + irq_work_sync(&data->work); error_uninit_buffer: iio_triggered_buffer_cleanup(indio_dev); error_regulator_disable: @@ -1587,6 +1588,7 @@ static void gp2ap020a00f_remove(struct i2c_client *client) iio_device_unregister(indio_dev); iio_trigger_unregister(data->trig); free_irq(client->irq, indio_dev); + irq_work_sync(&data->work); iio_triggered_buffer_cleanup(indio_dev); regulator_disable(data->vled_reg); } diff --git a/drivers/iio/light/ltrf216a.c b/drivers/iio/light/ltrf216a.c index 8de4dd849936..45644c0699fa 100644 --- a/drivers/iio/light/ltrf216a.c +++ b/drivers/iio/light/ltrf216a.c @@ -234,22 +234,20 @@ static int ltrf216a_read_data(struct ltrf216a_data *data, u8 addr) static int ltrf216a_get_lux(struct ltrf216a_data *data) { int ret, greendata; - u64 lux, div; + u64 lux; ret = ltrf216a_set_power_state(data, true); if (ret) return ret; greendata = ltrf216a_read_data(data, LTRF216A_ALS_DATA_0); + ltrf216a_set_power_state(data, false); if (greendata < 0) return greendata; - ltrf216a_set_power_state(data, false); - - lux = greendata * 45 * LTRF216A_WIN_FAC * 100; - div = data->als_gain_fac * data->int_time_fac * 100; + lux = greendata * 45 * LTRF216A_WIN_FAC; - return div_u64(lux, div); + return lux; } static int ltrf216a_read_raw(struct iio_dev *indio_dev, @@ -279,7 +277,8 @@ static int ltrf216a_read_raw(struct iio_dev *indio_dev, if (ret < 0) return ret; *val = ret; - return IIO_VAL_INT; + *val2 = data->als_gain_fac * data->int_time_fac; + return IIO_VAL_FRACTIONAL; case IIO_CHAN_INFO_INT_TIME: mutex_lock(&data->lock); ret = ltrf216a_get_int_time(data, val, val2); diff --git a/drivers/iio/light/rohm-bu27034.c b/drivers/iio/light/rohm-bu27034.c deleted file mode 100644 index bf3de853a811..000000000000 --- a/drivers/iio/light/rohm-bu27034.c +++ /dev/null @@ -1,1528 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-only -/* - * BU27034 ROHM Ambient Light Sensor - * - * Copyright (c) 2023, ROHM Semiconductor. - * https://fscdn.rohm.com/en/products/databook/datasheet/ic/sensor/light/bu27034nuc-e.pdf - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include -#include -#include - -#define BU27034_REG_SYSTEM_CONTROL 0x40 -#define BU27034_MASK_SW_RESET BIT(7) -#define BU27034_MASK_PART_ID GENMASK(5, 0) -#define BU27034_ID 0x19 -#define BU27034_REG_MODE_CONTROL1 0x41 -#define BU27034_MASK_MEAS_MODE GENMASK(2, 0) - -#define BU27034_REG_MODE_CONTROL2 0x42 -#define BU27034_MASK_D01_GAIN GENMASK(7, 3) -#define BU27034_MASK_D2_GAIN_HI GENMASK(7, 6) -#define BU27034_MASK_D2_GAIN_LO GENMASK(2, 0) - -#define BU27034_REG_MODE_CONTROL3 0x43 -#define BU27034_REG_MODE_CONTROL4 0x44 -#define BU27034_MASK_MEAS_EN BIT(0) -#define BU27034_MASK_VALID BIT(7) -#define BU27034_REG_DATA0_LO 0x50 -#define BU27034_REG_DATA1_LO 0x52 -#define BU27034_REG_DATA2_LO 0x54 -#define BU27034_REG_DATA2_HI 0x55 -#define BU27034_REG_MANUFACTURER_ID 0x92 -#define BU27034_REG_MAX BU27034_REG_MANUFACTURER_ID - -/* - * The BU27034 does not have interrupt to trigger the data read when a - * measurement has finished. Hence we poll the VALID bit in a thread. We will - * try to wake the thread BU27034_MEAS_WAIT_PREMATURE_MS milliseconds before - * the expected sampling time to prevent the drifting. - * - * If we constantly wake up a bit too late we would eventually skip a sample. - * And because the sleep can't wake up _exactly_ at given time this would be - * inevitable even if the sensor clock would be perfectly phase-locked to CPU - * clock - which we can't say is the case. - * - * This is still fragile. No matter how big advance do we have, we will still - * risk of losing a sample because things can in a rainy-day scenario be - * delayed a lot. Yet, more we reserve the time for polling, more we also lose - * the performance by spending cycles polling the register. So, selecting this - * value is a balancing dance between severity of wasting CPU time and severity - * of losing samples. - * - * In most cases losing the samples is not _that_ crucial because light levels - * tend to change slowly. - * - * Other option that was pointed to me would be always sleeping 1/2 of the - * measurement time, checking the VALID bit and just sleeping again if the bit - * was not set. That should be pretty tolerant against missing samples due to - * the scheduling delays while also not wasting much of cycles for polling. - * Downside is that the time-stamps would be very inaccurate as the wake-up - * would not really be tied to the sensor toggling the valid bit. This would also - * result 'jumps' in the time-stamps when the delay drifted so that wake-up was - * performed during the consecutive wake-ups (Or, when sensor and CPU clocks - * were very different and scheduling the wake-ups was very close to given - * timeout - and when the time-outs were very close to the actual sensor - * sampling, Eg. once in a blue moon, two consecutive time-outs would occur - * without having a sample ready). - */ -#define BU27034_MEAS_WAIT_PREMATURE_MS 5 -#define BU27034_DATA_WAIT_TIME_US 1000 -#define BU27034_TOTAL_DATA_WAIT_TIME_US (BU27034_MEAS_WAIT_PREMATURE_MS * 1000) - -#define BU27034_RETRY_LIMIT 18 - -enum { - BU27034_CHAN_ALS, - BU27034_CHAN_DATA0, - BU27034_CHAN_DATA1, - BU27034_CHAN_DATA2, - BU27034_NUM_CHANS -}; - -static const unsigned long bu27034_scan_masks[] = { - GENMASK(BU27034_CHAN_DATA2, BU27034_CHAN_ALS), 0 -}; - -/* - * Available scales with gain 1x - 4096x, timings 55, 100, 200, 400 mS - * Time impacts to gain: 1x, 2x, 4x, 8x. - * - * => Max total gain is HWGAIN * gain by integration time (8 * 4096) = 32768 - * - * Using NANO precision for scale we must use scale 64x corresponding gain 1x - * to avoid precision loss. (32x would result scale 976 562.5(nanos). - */ -#define BU27034_SCALE_1X 64 - -/* See the data sheet for the "Gain Setting" table */ -#define BU27034_GSEL_1X 0x00 /* 00000 */ -#define BU27034_GSEL_4X 0x08 /* 01000 */ -#define BU27034_GSEL_16X 0x0a /* 01010 */ -#define BU27034_GSEL_32X 0x0b /* 01011 */ -#define BU27034_GSEL_64X 0x0c /* 01100 */ -#define BU27034_GSEL_256X 0x18 /* 11000 */ -#define BU27034_GSEL_512X 0x19 /* 11001 */ -#define BU27034_GSEL_1024X 0x1a /* 11010 */ -#define BU27034_GSEL_2048X 0x1b /* 11011 */ -#define BU27034_GSEL_4096X 0x1c /* 11100 */ - -/* Available gain settings */ -static const struct iio_gain_sel_pair bu27034_gains[] = { - GAIN_SCALE_GAIN(1, BU27034_GSEL_1X), - GAIN_SCALE_GAIN(4, BU27034_GSEL_4X), - GAIN_SCALE_GAIN(16, BU27034_GSEL_16X), - GAIN_SCALE_GAIN(32, BU27034_GSEL_32X), - GAIN_SCALE_GAIN(64, BU27034_GSEL_64X), - GAIN_SCALE_GAIN(256, BU27034_GSEL_256X), - GAIN_SCALE_GAIN(512, BU27034_GSEL_512X), - GAIN_SCALE_GAIN(1024, BU27034_GSEL_1024X), - GAIN_SCALE_GAIN(2048, BU27034_GSEL_2048X), - GAIN_SCALE_GAIN(4096, BU27034_GSEL_4096X), -}; - -/* - * The IC has 5 modes for sampling time. 5 mS mode is exceptional as it limits - * the data collection to data0-channel only and cuts the supported range to - * 10 bit. It is not supported by the driver. - * - * "normal" modes are 55, 100, 200 and 400 mS modes - which do have direct - * multiplying impact to the register values (similar to gain). - * - * This means that if meas-mode is changed for example from 400 => 200, - * the scale is doubled. Eg, time impact to total gain is x1, x2, x4, x8. - */ -#define BU27034_MEAS_MODE_100MS 0 -#define BU27034_MEAS_MODE_55MS 1 -#define BU27034_MEAS_MODE_200MS 2 -#define BU27034_MEAS_MODE_400MS 4 - -static const struct iio_itime_sel_mul bu27034_itimes[] = { - GAIN_SCALE_ITIME_US(400000, BU27034_MEAS_MODE_400MS, 8), - GAIN_SCALE_ITIME_US(200000, BU27034_MEAS_MODE_200MS, 4), - GAIN_SCALE_ITIME_US(100000, BU27034_MEAS_MODE_100MS, 2), - GAIN_SCALE_ITIME_US(55000, BU27034_MEAS_MODE_55MS, 1), -}; - -#define BU27034_CHAN_DATA(_name, _ch2) \ -{ \ - .type = IIO_INTENSITY, \ - .channel = BU27034_CHAN_##_name, \ - .channel2 = (_ch2), \ - .info_mask_separate = BIT(IIO_CHAN_INFO_RAW) | \ - BIT(IIO_CHAN_INFO_SCALE), \ - .info_mask_separate_available = BIT(IIO_CHAN_INFO_SCALE), \ - .info_mask_shared_by_all = BIT(IIO_CHAN_INFO_INT_TIME), \ - .info_mask_shared_by_all_available = \ - BIT(IIO_CHAN_INFO_INT_TIME), \ - .address = BU27034_REG_##_name##_LO, \ - .scan_index = BU27034_CHAN_##_name, \ - .scan_type = { \ - .sign = 'u', \ - .realbits = 16, \ - .storagebits = 16, \ - .endianness = IIO_LE, \ - }, \ - .indexed = 1, \ -} - -static const struct iio_chan_spec bu27034_channels[] = { - { - .type = IIO_LIGHT, - .info_mask_separate = BIT(IIO_CHAN_INFO_RAW) | - BIT(IIO_CHAN_INFO_SCALE), - .channel = BU27034_CHAN_ALS, - .scan_index = BU27034_CHAN_ALS, - .scan_type = { - .sign = 'u', - .realbits = 32, - .storagebits = 32, - .endianness = IIO_CPU, - }, - }, - /* - * The BU27034 DATA0 and DATA1 channels are both on the visible light - * area (mostly). The data0 sensitivity peaks at 500nm, DATA1 at 600nm. - * These wave lengths are pretty much on the border of colours making - * these a poor candidates for R/G/B standardization. Hence they're both - * marked as clear channels - */ - BU27034_CHAN_DATA(DATA0, IIO_MOD_LIGHT_CLEAR), - BU27034_CHAN_DATA(DATA1, IIO_MOD_LIGHT_CLEAR), - BU27034_CHAN_DATA(DATA2, IIO_MOD_LIGHT_IR), - IIO_CHAN_SOFT_TIMESTAMP(4), -}; - -struct bu27034_data { - struct regmap *regmap; - struct device *dev; - /* - * Protect gain and time during scale adjustment and data reading. - * Protect measurement enabling/disabling. - */ - struct mutex mutex; - struct iio_gts gts; - struct task_struct *task; - __le16 raw[3]; - struct { - u32 mlux; - __le16 channels[3]; - s64 ts __aligned(8); - } scan; -}; - -struct bu27034_result { - u16 ch0; - u16 ch1; - u16 ch2; -}; - -static const struct regmap_range bu27034_volatile_ranges[] = { - { - .range_min = BU27034_REG_SYSTEM_CONTROL, - .range_max = BU27034_REG_SYSTEM_CONTROL, - }, { - .range_min = BU27034_REG_MODE_CONTROL4, - .range_max = BU27034_REG_MODE_CONTROL4, - }, { - .range_min = BU27034_REG_DATA0_LO, - .range_max = BU27034_REG_DATA2_HI, - }, -}; - -static const struct regmap_access_table bu27034_volatile_regs = { - .yes_ranges = &bu27034_volatile_ranges[0], - .n_yes_ranges = ARRAY_SIZE(bu27034_volatile_ranges), -}; - -static const struct regmap_range bu27034_read_only_ranges[] = { - { - .range_min = BU27034_REG_DATA0_LO, - .range_max = BU27034_REG_DATA2_HI, - }, { - .range_min = BU27034_REG_MANUFACTURER_ID, - .range_max = BU27034_REG_MANUFACTURER_ID, - } -}; - -static const struct regmap_access_table bu27034_ro_regs = { - .no_ranges = &bu27034_read_only_ranges[0], - .n_no_ranges = ARRAY_SIZE(bu27034_read_only_ranges), -}; - -static const struct regmap_config bu27034_regmap = { - .reg_bits = 8, - .val_bits = 8, - .max_register = BU27034_REG_MAX, - .cache_type = REGCACHE_RBTREE, - .volatile_table = &bu27034_volatile_regs, - .wr_table = &bu27034_ro_regs, -}; - -struct bu27034_gain_check { - int old_gain; - int new_gain; - int chan; -}; - -static int bu27034_get_gain_sel(struct bu27034_data *data, int chan) -{ - int ret, val; - - switch (chan) { - case BU27034_CHAN_DATA0: - case BU27034_CHAN_DATA1: - { - int reg[] = { - [BU27034_CHAN_DATA0] = BU27034_REG_MODE_CONTROL2, - [BU27034_CHAN_DATA1] = BU27034_REG_MODE_CONTROL3, - }; - ret = regmap_read(data->regmap, reg[chan], &val); - if (ret) - return ret; - - return FIELD_GET(BU27034_MASK_D01_GAIN, val); - } - case BU27034_CHAN_DATA2: - { - int d2_lo_bits = fls(BU27034_MASK_D2_GAIN_LO); - - ret = regmap_read(data->regmap, BU27034_REG_MODE_CONTROL2, &val); - if (ret) - return ret; - - /* - * The data2 channel gain is composed by 5 non continuous bits - * [7:6], [2:0]. Thus when we combine the 5-bit 'selector' - * from register value we must right shift the high bits by 3. - */ - return FIELD_GET(BU27034_MASK_D2_GAIN_HI, val) << d2_lo_bits | - FIELD_GET(BU27034_MASK_D2_GAIN_LO, val); - } - default: - return -EINVAL; - } -} - -static int bu27034_get_gain(struct bu27034_data *data, int chan, int *gain) -{ - int ret, sel; - - ret = bu27034_get_gain_sel(data, chan); - if (ret < 0) - return ret; - - sel = ret; - - ret = iio_gts_find_gain_by_sel(&data->gts, sel); - if (ret < 0) { - dev_err(data->dev, "chan %u: unknown gain value 0x%x\n", chan, - sel); - - return ret; - } - - *gain = ret; - - return 0; -} - -static int bu27034_get_int_time(struct bu27034_data *data) -{ - int ret, sel; - - ret = regmap_read(data->regmap, BU27034_REG_MODE_CONTROL1, &sel); - if (ret) - return ret; - - return iio_gts_find_int_time_by_sel(&data->gts, - sel & BU27034_MASK_MEAS_MODE); -} - -static int _bu27034_get_scale(struct bu27034_data *data, int channel, int *val, - int *val2) -{ - int gain, ret; - - ret = bu27034_get_gain(data, channel, &gain); - if (ret) - return ret; - - ret = bu27034_get_int_time(data); - if (ret < 0) - return ret; - - return iio_gts_get_scale(&data->gts, gain, ret, val, val2); -} - -static int bu27034_get_scale(struct bu27034_data *data, int channel, int *val, - int *val2) -{ - int ret; - - if (channel == BU27034_CHAN_ALS) { - *val = 0; - *val2 = 1000; - return IIO_VAL_INT_PLUS_MICRO; - } - - mutex_lock(&data->mutex); - ret = _bu27034_get_scale(data, channel, val, val2); - mutex_unlock(&data->mutex); - if (ret) - return ret; - - return IIO_VAL_INT_PLUS_NANO; -} - -/* Caller should hold the lock to protect lux reading */ -static int bu27034_write_gain_sel(struct bu27034_data *data, int chan, int sel) -{ - static const int reg[] = { - [BU27034_CHAN_DATA0] = BU27034_REG_MODE_CONTROL2, - [BU27034_CHAN_DATA1] = BU27034_REG_MODE_CONTROL3, - }; - int mask, val; - - if (chan != BU27034_CHAN_DATA0 && chan != BU27034_CHAN_DATA1) - return -EINVAL; - - val = FIELD_PREP(BU27034_MASK_D01_GAIN, sel); - - mask = BU27034_MASK_D01_GAIN; - - if (chan == BU27034_CHAN_DATA0) { - /* - * We keep the same gain for channel 2 as we set for channel 0 - * We can't allow them to be individually controlled because - * setting one will impact also the other. Also, if we don't - * always update both gains we may result unsupported bit - * combinations. - * - * This is not nice but this is yet another place where the - * user space must be prepared to surprizes. Namely, see chan 2 - * gain changed when chan 0 gain is changed. - * - * This is not fatal for most users though. I don't expect the - * channel 2 to be used in any generic cases - the intensity - * values provided by the sensor for IR area are not openly - * documented. Also, channel 2 is not used for visible light. - * - * So, if there is application which is written to utilize the - * channel 2 - then it is probably specifically targeted to this - * sensor and knows how to utilize those values. It is safe to - * hope such user can also cope with the gain changes. - */ - mask |= BU27034_MASK_D2_GAIN_LO; - - /* - * The D2 gain bits are directly the lowest bits of selector. - * Just do add those bits to the value - */ - val |= sel & BU27034_MASK_D2_GAIN_LO; - } - - return regmap_update_bits(data->regmap, reg[chan], mask, val); -} - -static int bu27034_set_gain(struct bu27034_data *data, int chan, int gain) -{ - int ret; - - /* - * We don't allow setting channel 2 gain as it messes up the - * gain for channel 0 - which shares the high bits - */ - if (chan != BU27034_CHAN_DATA0 && chan != BU27034_CHAN_DATA1) - return -EINVAL; - - ret = iio_gts_find_sel_by_gain(&data->gts, gain); - if (ret < 0) - return ret; - - return bu27034_write_gain_sel(data, chan, ret); -} - -/* Caller should hold the lock to protect data->int_time */ -static int bu27034_set_int_time(struct bu27034_data *data, int time) -{ - int ret; - - ret = iio_gts_find_sel_by_int_time(&data->gts, time); - if (ret < 0) - return ret; - - return regmap_update_bits(data->regmap, BU27034_REG_MODE_CONTROL1, - BU27034_MASK_MEAS_MODE, ret); -} - -/* - * We try to change the time in such way that the scale is maintained for - * given channels by adjusting gain so that it compensates the time change. - */ -static int bu27034_try_set_int_time(struct bu27034_data *data, int time_us) -{ - struct bu27034_gain_check gains[] = { - { .chan = BU27034_CHAN_DATA0 }, - { .chan = BU27034_CHAN_DATA1 }, - }; - int numg = ARRAY_SIZE(gains); - int ret, int_time_old, i; - - mutex_lock(&data->mutex); - ret = bu27034_get_int_time(data); - if (ret < 0) - goto unlock_out; - - int_time_old = ret; - - if (!iio_gts_valid_time(&data->gts, time_us)) { - dev_err(data->dev, "Unsupported integration time %u\n", - time_us); - ret = -EINVAL; - - goto unlock_out; - } - - if (time_us == int_time_old) { - ret = 0; - goto unlock_out; - } - - for (i = 0; i < numg; i++) { - ret = bu27034_get_gain(data, gains[i].chan, &gains[i].old_gain); - if (ret) - goto unlock_out; - - ret = iio_gts_find_new_gain_by_old_gain_time(&data->gts, - gains[i].old_gain, - int_time_old, time_us, - &gains[i].new_gain); - if (ret) { - int scale1, scale2; - bool ok; - - _bu27034_get_scale(data, gains[i].chan, &scale1, &scale2); - dev_dbg(data->dev, - "chan %u, can't support time %u with scale %u %u\n", - gains[i].chan, time_us, scale1, scale2); - - if (gains[i].new_gain < 0) - goto unlock_out; - - /* - * If caller requests for integration time change and we - * can't support the scale - then the caller should be - * prepared to 'pick up the pieces and deal with the - * fact that the scale changed'. - */ - ret = iio_find_closest_gain_low(&data->gts, - gains[i].new_gain, &ok); - - if (!ok) - dev_dbg(data->dev, - "optimal gain out of range for chan %u\n", - gains[i].chan); - - if (ret < 0) { - dev_dbg(data->dev, - "Total gain increase. Risk of saturation"); - ret = iio_gts_get_min_gain(&data->gts); - if (ret < 0) - goto unlock_out; - } - dev_dbg(data->dev, "chan %u scale changed\n", - gains[i].chan); - gains[i].new_gain = ret; - dev_dbg(data->dev, "chan %u new gain %u\n", - gains[i].chan, gains[i].new_gain); - } - } - - for (i = 0; i < numg; i++) { - ret = bu27034_set_gain(data, gains[i].chan, gains[i].new_gain); - if (ret) - goto unlock_out; - } - - ret = bu27034_set_int_time(data, time_us); - -unlock_out: - mutex_unlock(&data->mutex); - - return ret; -} - -static int bu27034_set_scale(struct bu27034_data *data, int chan, - int val, int val2) -{ - int ret, time_sel, gain_sel, i; - bool found = false; - - if (chan == BU27034_CHAN_DATA2) - return -EINVAL; - - if (chan == BU27034_CHAN_ALS) { - if (val == 0 && val2 == 1000000) - return 0; - - return -EINVAL; - } - - mutex_lock(&data->mutex); - ret = regmap_read(data->regmap, BU27034_REG_MODE_CONTROL1, &time_sel); - if (ret) - goto unlock_out; - - ret = iio_gts_find_gain_sel_for_scale_using_time(&data->gts, time_sel, - val, val2, &gain_sel); - if (ret) { - /* - * Could not support scale with given time. Need to change time. - * We still want to maintain the scale for all channels - */ - struct bu27034_gain_check gain; - int new_time_sel; - - /* - * Populate information for the other channel which should also - * maintain the scale. (Due to the HW limitations the chan2 - * gets the same gain as chan0, so we only need to explicitly - * set the chan 0 and 1). - */ - if (chan == BU27034_CHAN_DATA0) - gain.chan = BU27034_CHAN_DATA1; - else if (chan == BU27034_CHAN_DATA1) - gain.chan = BU27034_CHAN_DATA0; - - ret = bu27034_get_gain(data, gain.chan, &gain.old_gain); - if (ret) - goto unlock_out; - - /* - * Iterate through all the times to see if we find one which - * can support requested scale for requested channel, while - * maintaining the scale for other channels - */ - for (i = 0; i < data->gts.num_itime; i++) { - new_time_sel = data->gts.itime_table[i].sel; - - if (new_time_sel == time_sel) - continue; - - /* Can we provide requested scale with this time? */ - ret = iio_gts_find_gain_sel_for_scale_using_time( - &data->gts, new_time_sel, val, val2, - &gain_sel); - if (ret) - continue; - - /* Can the other channel(s) maintain scale? */ - ret = iio_gts_find_new_gain_sel_by_old_gain_time( - &data->gts, gain.old_gain, time_sel, - new_time_sel, &gain.new_gain); - if (!ret) { - /* Yes - we found suitable time */ - found = true; - break; - } - } - if (!found) { - dev_dbg(data->dev, - "Can't set scale maintaining other channels\n"); - ret = -EINVAL; - - goto unlock_out; - } - - ret = bu27034_set_gain(data, gain.chan, gain.new_gain); - if (ret) - goto unlock_out; - - ret = regmap_update_bits(data->regmap, BU27034_REG_MODE_CONTROL1, - BU27034_MASK_MEAS_MODE, new_time_sel); - if (ret) - goto unlock_out; - } - - ret = bu27034_write_gain_sel(data, chan, gain_sel); -unlock_out: - mutex_unlock(&data->mutex); - - return ret; -} - -/* - * for (D1/D0 < 0.87): - * lx = 0.004521097 * D1 - 0.002663996 * D0 + - * 0.00012213 * D1 * D1 / D0 - * - * => 115.7400832 * ch1 / gain1 / mt - - * 68.1982976 * ch0 / gain0 / mt + - * 0.00012213 * 25600 * (ch1 / gain1 / mt) * 25600 * - * (ch1 /gain1 / mt) / (25600 * ch0 / gain0 / mt) - * - * A = 0.00012213 * 25600 * (ch1 /gain1 / mt) * 25600 * - * (ch1 /gain1 / mt) / (25600 * ch0 / gain0 / mt) - * => 0.00012213 * 25600 * (ch1 /gain1 / mt) * - * (ch1 /gain1 / mt) / (ch0 / gain0 / mt) - * => 0.00012213 * 25600 * (ch1 / gain1) * (ch1 /gain1 / mt) / - * (ch0 / gain0) - * => 0.00012213 * 25600 * (ch1 / gain1) * (ch1 /gain1 / mt) * - * gain0 / ch0 - * => 3.126528 * ch1 * ch1 * gain0 / gain1 / gain1 / mt /ch0 - * - * lx = (115.7400832 * ch1 / gain1 - 68.1982976 * ch0 / gain0) / - * mt + A - * => (115.7400832 * ch1 / gain1 - 68.1982976 * ch0 / gain0) / - * mt + 3.126528 * ch1 * ch1 * gain0 / gain1 / gain1 / mt / - * ch0 - * - * => (115.7400832 * ch1 / gain1 - 68.1982976 * ch0 / gain0 + - * 3.126528 * ch1 * ch1 * gain0 / gain1 / gain1 / ch0) / - * mt - * - * For (0.87 <= D1/D0 < 1.00) - * lx = (0.001331* D0 + 0.0000354 * D1) * ((D1/D0 – 0.87) * (0.385) + 1) - * => (0.001331 * 256 * 100 * ch0 / gain0 / mt + 0.0000354 * 256 * - * 100 * ch1 / gain1 / mt) * ((D1/D0 - 0.87) * (0.385) + 1) - * => (34.0736 * ch0 / gain0 / mt + 0.90624 * ch1 / gain1 / mt) * - * ((D1/D0 - 0.87) * (0.385) + 1) - * => (34.0736 * ch0 / gain0 / mt + 0.90624 * ch1 / gain1 / mt) * - * (0.385 * D1/D0 - 0.66505) - * => (34.0736 * ch0 / gain0 / mt + 0.90624 * ch1 / gain1 / mt) * - * (0.385 * 256 * 100 * ch1 / gain1 / mt / (256 * 100 * ch0 / gain0 / mt) - 0.66505) - * => (34.0736 * ch0 / gain0 / mt + 0.90624 * ch1 / gain1 / mt) * - * (9856 * ch1 / gain1 / mt / (25600 * ch0 / gain0 / mt) + 0.66505) - * => 13.118336 * ch1 / (gain1 * mt) - * + 22.66064768 * ch0 / (gain0 * mt) - * + 8931.90144 * ch1 * ch1 * gain0 / - * (25600 * ch0 * gain1 * gain1 * mt) - * + 0.602694912 * ch1 / (gain1 * mt) - * - * => [0.3489024 * ch1 * ch1 * gain0 / (ch0 * gain1 * gain1) - * + 22.66064768 * ch0 / gain0 - * + 13.721030912 * ch1 / gain1 - * ] / mt - * - * For (D1/D0 >= 1.00) - * - * lx = (0.001331* D0 + 0.0000354 * D1) * ((D1/D0 – 2.0) * (-0.05) + 1) - * => (0.001331* D0 + 0.0000354 * D1) * (-0.05D1/D0 + 1.1) - * => (0.001331 * 256 * 100 * ch0 / gain0 / mt + 0.0000354 * 256 * - * 100 * ch1 / gain1 / mt) * (-0.05D1/D0 + 1.1) - * => (34.0736 * ch0 / gain0 / mt + 0.90624 * ch1 / gain1 / mt) * - * (-0.05 * 256 * 100 * ch1 / gain1 / mt / (256 * 100 * ch0 / gain0 / mt) + 1.1) - * => (34.0736 * ch0 / gain0 / mt + 0.90624 * ch1 / gain1 / mt) * - * (-1280 * ch1 / (gain1 * mt * 25600 * ch0 / gain0 / mt) + 1.1) - * => (34.0736 * ch0 * -1280 * ch1 * gain0 * mt /( gain0 * mt * gain1 * mt * 25600 * ch0) - * + 34.0736 * 1.1 * ch0 / (gain0 * mt) - * + 0.90624 * ch1 * -1280 * ch1 *gain0 * mt / (gain1 * mt *gain1 * mt * 25600 * ch0) - * + 1.1 * 0.90624 * ch1 / (gain1 * mt) - * => -43614.208 * ch1 / (gain1 * mt * 25600) - * + 37.48096 ch0 / (gain0 * mt) - * - 1159.9872 * ch1 * ch1 * gain0 / (gain1 * gain1 * mt * 25600 * ch0) - * + 0.996864 ch1 / (gain1 * mt) - * => [ - * - 0.045312 * ch1 * ch1 * gain0 / (gain1 * gain1 * ch0) - * - 0.706816 * ch1 / gain1 - * + 37.48096 ch0 /gain0 - * ] * mt - * - * - * So, the first case (D1/D0 < 0.87) can be computed to a form: - * - * lx = (3.126528 * ch1 * ch1 * gain0 / (ch0 * gain1 * gain1) + - * 115.7400832 * ch1 / gain1 + - * -68.1982976 * ch0 / gain0 - * / mt - * - * Second case (0.87 <= D1/D0 < 1.00) goes to form: - * - * => [0.3489024 * ch1 * ch1 * gain0 / (ch0 * gain1 * gain1) + - * 13.721030912 * ch1 / gain1 + - * 22.66064768 * ch0 / gain0 - * ] / mt - * - * Third case (D1/D0 >= 1.00) goes to form: - * => [-0.045312 * ch1 * ch1 * gain0 / (ch0 * gain1 * gain1) + - * -0.706816 * ch1 / gain1 + - * 37.48096 ch0 /(gain0 - * ] / mt - * - * This can be unified to format: - * lx = [ - * A * ch1 * ch1 * gain0 / (ch0 * gain1 * gain1) + - * B * ch1 / gain1 + - * C * ch0 / gain0 - * ] / mt - * - * For case 1: - * A = 3.126528, - * B = 115.7400832 - * C = -68.1982976 - * - * For case 2: - * A = 0.3489024 - * B = 13.721030912 - * C = 22.66064768 - * - * For case 3: - * A = -0.045312 - * B = -0.706816 - * C = 37.48096 - */ - -struct bu27034_lx_coeff { - unsigned int A; - unsigned int B; - unsigned int C; - /* Indicate which of the coefficients above are negative */ - bool is_neg[3]; -}; - -static inline u64 gain_mul_div_helper(u64 val, unsigned int gain, - unsigned int div) -{ - /* - * Max gain for a channel is 4096. The max u64 (0xffffffffffffffffULL) - * divided by 4096 is 0xFFFFFFFFFFFFF (GENMASK_ULL(51, 0)) (floored). - * Thus, the 0xFFFFFFFFFFFFF is the largest value we can safely multiply - * with the gain, no matter what gain is set. - * - * So, multiplication with max gain may overflow if val is greater than - * 0xFFFFFFFFFFFFF (52 bits set).. - * - * If this is the case we divide first. - */ - if (val < GENMASK_ULL(51, 0)) { - val *= gain; - do_div(val, div); - } else { - do_div(val, div); - val *= gain; - } - - return val; -} - -static u64 bu27034_fixp_calc_t1_64bit(unsigned int coeff, unsigned int ch0, - unsigned int ch1, unsigned int gain0, - unsigned int gain1) -{ - unsigned int helper; - u64 helper64; - - helper64 = (u64)coeff * (u64)ch1 * (u64)ch1; - - helper = gain1 * gain1; - if (helper > ch0) { - do_div(helper64, helper); - - return gain_mul_div_helper(helper64, gain0, ch0); - } - - do_div(helper64, ch0); - - return gain_mul_div_helper(helper64, gain0, helper); - -} - -static u64 bu27034_fixp_calc_t1(unsigned int coeff, unsigned int ch0, - unsigned int ch1, unsigned int gain0, - unsigned int gain1) -{ - unsigned int helper, tmp; - - /* - * Here we could overflow even the 64bit value. Hence we - * multiply with gain0 only after the divisions - even though - * it may result loss of accuracy - */ - helper = coeff * ch1 * ch1; - tmp = helper * gain0; - - helper = ch1 * ch1; - - if (check_mul_overflow(helper, coeff, &helper)) - return bu27034_fixp_calc_t1_64bit(coeff, ch0, ch1, gain0, gain1); - - if (check_mul_overflow(helper, gain0, &tmp)) - return bu27034_fixp_calc_t1_64bit(coeff, ch0, ch1, gain0, gain1); - - return tmp / (gain1 * gain1) / ch0; - -} - -static u64 bu27034_fixp_calc_t23(unsigned int coeff, unsigned int ch, - unsigned int gain) -{ - unsigned int helper; - u64 helper64; - - if (!check_mul_overflow(coeff, ch, &helper)) - return helper / gain; - - helper64 = (u64)coeff * (u64)ch; - do_div(helper64, gain); - - return helper64; -} - -static int bu27034_fixp_calc_lx(unsigned int ch0, unsigned int ch1, - unsigned int gain0, unsigned int gain1, - unsigned int meastime, int coeff_idx) -{ - static const struct bu27034_lx_coeff coeff[] = { - { - .A = 31265280, /* 3.126528 */ - .B = 1157400832, /*115.7400832 */ - .C = 681982976, /* -68.1982976 */ - .is_neg = {false, false, true}, - }, { - .A = 3489024, /* 0.3489024 */ - .B = 137210309, /* 13.721030912 */ - .C = 226606476, /* 22.66064768 */ - /* All terms positive */ - }, { - .A = 453120, /* -0.045312 */ - .B = 7068160, /* -0.706816 */ - .C = 374809600, /* 37.48096 */ - .is_neg = {true, true, false}, - } - }; - const struct bu27034_lx_coeff *c = &coeff[coeff_idx]; - u64 res = 0, terms[3]; - int i; - - if (coeff_idx >= ARRAY_SIZE(coeff)) - return -EINVAL; - - terms[0] = bu27034_fixp_calc_t1(c->A, ch0, ch1, gain0, gain1); - terms[1] = bu27034_fixp_calc_t23(c->B, ch1, gain1); - terms[2] = bu27034_fixp_calc_t23(c->C, ch0, gain0); - - /* First, add positive terms */ - for (i = 0; i < 3; i++) - if (!c->is_neg[i]) - res += terms[i]; - - /* No positive term => zero lux */ - if (!res) - return 0; - - /* Then, subtract negative terms (if any) */ - for (i = 0; i < 3; i++) - if (c->is_neg[i]) { - /* - * If the negative term is greater than positive - then - * the darkness has taken over and we are all doomed! Eh, - * I mean, then we can just return 0 lx and go out - */ - if (terms[i] >= res) - return 0; - - res -= terms[i]; - } - - meastime *= 10; - do_div(res, meastime); - - return (int) res; -} - -static bool bu27034_has_valid_sample(struct bu27034_data *data) -{ - int ret, val; - - ret = regmap_read(data->regmap, BU27034_REG_MODE_CONTROL4, &val); - if (ret) { - dev_err(data->dev, "Read failed %d\n", ret); - - return false; - } - - return val & BU27034_MASK_VALID; -} - -/* - * Reading the register where VALID bit is clears this bit. (So does changing - * any gain / integration time configuration registers) The bit gets - * set when we have acquired new data. We use this bit to indicate data - * validity. - */ -static void bu27034_invalidate_read_data(struct bu27034_data *data) -{ - bu27034_has_valid_sample(data); -} - -static int bu27034_read_result(struct bu27034_data *data, int chan, int *res) -{ - int reg[] = { - [BU27034_CHAN_DATA0] = BU27034_REG_DATA0_LO, - [BU27034_CHAN_DATA1] = BU27034_REG_DATA1_LO, - [BU27034_CHAN_DATA2] = BU27034_REG_DATA2_LO, - }; - int valid, ret; - __le16 val; - - ret = regmap_read_poll_timeout(data->regmap, BU27034_REG_MODE_CONTROL4, - valid, (valid & BU27034_MASK_VALID), - BU27034_DATA_WAIT_TIME_US, 0); - if (ret) - return ret; - - ret = regmap_bulk_read(data->regmap, reg[chan], &val, sizeof(val)); - if (ret) - return ret; - - *res = le16_to_cpu(val); - - return 0; -} - -static int bu27034_get_result_unlocked(struct bu27034_data *data, __le16 *res, - int size) -{ - int ret = 0, retry_cnt = 0; - -retry: - /* Get new value from sensor if data is ready */ - if (bu27034_has_valid_sample(data)) { - ret = regmap_bulk_read(data->regmap, BU27034_REG_DATA0_LO, - res, size); - if (ret) - return ret; - - bu27034_invalidate_read_data(data); - } else { - /* No new data in sensor. Wait and retry */ - retry_cnt++; - - if (retry_cnt > BU27034_RETRY_LIMIT) { - dev_err(data->dev, "No data from sensor\n"); - - return -ETIMEDOUT; - } - - msleep(25); - - goto retry; - } - - return ret; -} - -static int bu27034_meas_set(struct bu27034_data *data, bool en) -{ - if (en) - return regmap_set_bits(data->regmap, BU27034_REG_MODE_CONTROL4, - BU27034_MASK_MEAS_EN); - - return regmap_clear_bits(data->regmap, BU27034_REG_MODE_CONTROL4, - BU27034_MASK_MEAS_EN); -} - -static int bu27034_get_single_result(struct bu27034_data *data, int chan, - int *val) -{ - int ret; - - if (chan < BU27034_CHAN_DATA0 || chan > BU27034_CHAN_DATA2) - return -EINVAL; - - ret = bu27034_meas_set(data, true); - if (ret) - return ret; - - ret = bu27034_get_int_time(data); - if (ret < 0) - return ret; - - msleep(ret / 1000); - - return bu27034_read_result(data, chan, val); -} - -/* - * The formula given by vendor for computing luxes out of data0 and data1 - * (in open air) is as follows: - * - * Let's mark: - * D0 = data0/ch0_gain/meas_time_ms * 25600 - * D1 = data1/ch1_gain/meas_time_ms * 25600 - * - * Then: - * if (D1/D0 < 0.87) - * lx = (0.001331 * D0 + 0.0000354 * D1) * ((D1 / D0 - 0.87) * 3.45 + 1) - * else if (D1/D0 < 1) - * lx = (0.001331 * D0 + 0.0000354 * D1) * ((D1 / D0 - 0.87) * 0.385 + 1) - * else - * lx = (0.001331 * D0 + 0.0000354 * D1) * ((D1 / D0 - 2) * -0.05 + 1) - * - * We use it here. Users who have for example some colored lens - * need to modify the calculation but I hope this gives a starting point for - * those working with such devices. - */ - -static int bu27034_calc_mlux(struct bu27034_data *data, __le16 *res, int *val) -{ - unsigned int gain0, gain1, meastime; - unsigned int d1_d0_ratio_scaled; - u16 ch0, ch1; - u64 helper64; - int ret; - - /* - * We return 0 lux if calculation fails. This should be reasonably - * easy to spot from the buffers especially if raw-data channels show - * valid values - */ - *val = 0; - - ch0 = max_t(u16, 1, le16_to_cpu(res[0])); - ch1 = max_t(u16, 1, le16_to_cpu(res[1])); - - ret = bu27034_get_gain(data, BU27034_CHAN_DATA0, &gain0); - if (ret) - return ret; - - ret = bu27034_get_gain(data, BU27034_CHAN_DATA1, &gain1); - if (ret) - return ret; - - ret = bu27034_get_int_time(data); - if (ret < 0) - return ret; - - meastime = ret; - - d1_d0_ratio_scaled = (unsigned int)ch1 * (unsigned int)gain0 * 100; - helper64 = (u64)ch1 * (u64)gain0 * 100LLU; - - if (helper64 != d1_d0_ratio_scaled) { - unsigned int div = (unsigned int)ch0 * gain1; - - do_div(helper64, div); - d1_d0_ratio_scaled = helper64; - } else { - d1_d0_ratio_scaled /= ch0 * gain1; - } - - if (d1_d0_ratio_scaled < 87) - ret = bu27034_fixp_calc_lx(ch0, ch1, gain0, gain1, meastime, 0); - else if (d1_d0_ratio_scaled < 100) - ret = bu27034_fixp_calc_lx(ch0, ch1, gain0, gain1, meastime, 1); - else - ret = bu27034_fixp_calc_lx(ch0, ch1, gain0, gain1, meastime, 2); - - if (ret < 0) - return ret; - - *val = ret; - - return 0; - -} - -static int bu27034_get_mlux(struct bu27034_data *data, int chan, int *val) -{ - __le16 res[3]; - int ret; - - ret = bu27034_meas_set(data, true); - if (ret) - return ret; - - ret = bu27034_get_result_unlocked(data, &res[0], sizeof(res)); - if (ret) - return ret; - - ret = bu27034_calc_mlux(data, res, val); - if (ret) - return ret; - - ret = bu27034_meas_set(data, false); - if (ret) - dev_err(data->dev, "failed to disable measurement\n"); - - return 0; -} - -static int bu27034_read_raw(struct iio_dev *idev, - struct iio_chan_spec const *chan, - int *val, int *val2, long mask) -{ - struct bu27034_data *data = iio_priv(idev); - int ret; - - switch (mask) { - case IIO_CHAN_INFO_INT_TIME: - *val = 0; - *val2 = bu27034_get_int_time(data); - if (*val2 < 0) - return *val2; - - return IIO_VAL_INT_PLUS_MICRO; - - case IIO_CHAN_INFO_SCALE: - return bu27034_get_scale(data, chan->channel, val, val2); - - case IIO_CHAN_INFO_RAW: - { - int (*result_get)(struct bu27034_data *data, int chan, int *val); - - if (chan->type == IIO_INTENSITY) - result_get = bu27034_get_single_result; - else if (chan->type == IIO_LIGHT) - result_get = bu27034_get_mlux; - else - return -EINVAL; - - /* Don't mess with measurement enabling while buffering */ - ret = iio_device_claim_direct_mode(idev); - if (ret) - return ret; - - mutex_lock(&data->mutex); - /* - * Reading one channel at a time is inefficient but we - * don't care here. Buffered version should be used if - * performance is an issue. - */ - ret = result_get(data, chan->channel, val); - - mutex_unlock(&data->mutex); - iio_device_release_direct_mode(idev); - - if (ret) - return ret; - - return IIO_VAL_INT; - } - default: - return -EINVAL; - } -} - -static int bu27034_write_raw_get_fmt(struct iio_dev *indio_dev, - struct iio_chan_spec const *chan, - long mask) -{ - - switch (mask) { - case IIO_CHAN_INFO_SCALE: - return IIO_VAL_INT_PLUS_NANO; - case IIO_CHAN_INFO_INT_TIME: - return IIO_VAL_INT_PLUS_MICRO; - default: - return -EINVAL; - } -} - -static int bu27034_write_raw(struct iio_dev *idev, - struct iio_chan_spec const *chan, - int val, int val2, long mask) -{ - struct bu27034_data *data = iio_priv(idev); - int ret; - - ret = iio_device_claim_direct_mode(idev); - if (ret) - return ret; - - switch (mask) { - case IIO_CHAN_INFO_SCALE: - ret = bu27034_set_scale(data, chan->channel, val, val2); - break; - case IIO_CHAN_INFO_INT_TIME: - if (!val) - ret = bu27034_try_set_int_time(data, val2); - else - ret = -EINVAL; - break; - default: - ret = -EINVAL; - break; - } - - iio_device_release_direct_mode(idev); - - return ret; -} - -static int bu27034_read_avail(struct iio_dev *idev, - struct iio_chan_spec const *chan, const int **vals, - int *type, int *length, long mask) -{ - struct bu27034_data *data = iio_priv(idev); - - switch (mask) { - case IIO_CHAN_INFO_INT_TIME: - return iio_gts_avail_times(&data->gts, vals, type, length); - case IIO_CHAN_INFO_SCALE: - return iio_gts_all_avail_scales(&data->gts, vals, type, length); - default: - return -EINVAL; - } -} - -static const struct iio_info bu27034_info = { - .read_raw = &bu27034_read_raw, - .write_raw = &bu27034_write_raw, - .write_raw_get_fmt = &bu27034_write_raw_get_fmt, - .read_avail = &bu27034_read_avail, -}; - -static int bu27034_chip_init(struct bu27034_data *data) -{ - int ret, sel; - - /* Reset */ - ret = regmap_write_bits(data->regmap, BU27034_REG_SYSTEM_CONTROL, - BU27034_MASK_SW_RESET, BU27034_MASK_SW_RESET); - if (ret) - return dev_err_probe(data->dev, ret, "Sensor reset failed\n"); - - msleep(1); - - ret = regmap_reinit_cache(data->regmap, &bu27034_regmap); - if (ret) { - dev_err(data->dev, "Failed to reinit reg cache\n"); - return ret; - } - - /* - * Read integration time here to ensure it is in regmap cache. We do - * this to speed-up the int-time acquisition in the start of the buffer - * handling thread where longer delays could make it more likely we end - * up skipping a sample, and where the longer delays make timestamps - * less accurate. - */ - ret = regmap_read(data->regmap, BU27034_REG_MODE_CONTROL1, &sel); - if (ret) - dev_err(data->dev, "reading integration time failed\n"); - - return 0; -} - -static int bu27034_wait_for_data(struct bu27034_data *data) -{ - int ret, val; - - ret = regmap_read_poll_timeout(data->regmap, BU27034_REG_MODE_CONTROL4, - val, val & BU27034_MASK_VALID, - BU27034_DATA_WAIT_TIME_US, - BU27034_TOTAL_DATA_WAIT_TIME_US); - if (ret) { - dev_err(data->dev, "data polling %s\n", - !(val & BU27034_MASK_VALID) ? "timeout" : "fail"); - - return ret; - } - - ret = regmap_bulk_read(data->regmap, BU27034_REG_DATA0_LO, - &data->scan.channels[0], - sizeof(data->scan.channels)); - if (ret) - return ret; - - bu27034_invalidate_read_data(data); - - return 0; -} - -static int bu27034_buffer_thread(void *arg) -{ - struct iio_dev *idev = arg; - struct bu27034_data *data; - int wait_ms; - - data = iio_priv(idev); - - wait_ms = bu27034_get_int_time(data); - wait_ms /= 1000; - - wait_ms -= BU27034_MEAS_WAIT_PREMATURE_MS; - - while (!kthread_should_stop()) { - int ret; - int64_t tstamp; - - msleep(wait_ms); - ret = bu27034_wait_for_data(data); - if (ret) - continue; - - tstamp = iio_get_time_ns(idev); - - if (test_bit(BU27034_CHAN_ALS, idev->active_scan_mask)) { - int mlux; - - ret = bu27034_calc_mlux(data, &data->scan.channels[0], - &mlux); - if (ret) - dev_err(data->dev, "failed to calculate lux\n"); - - /* - * The maximum Milli lux value we get with gain 1x time - * 55mS data ch0 = 0xffff ch1 = 0xffff fits in 26 bits - * so there should be no problem returning int from - * computations and casting it to u32 - */ - data->scan.mlux = (u32)mlux; - } - iio_push_to_buffers_with_timestamp(idev, &data->scan, tstamp); - } - - return 0; -} - -static int bu27034_buffer_enable(struct iio_dev *idev) -{ - struct bu27034_data *data = iio_priv(idev); - struct task_struct *task; - int ret; - - mutex_lock(&data->mutex); - ret = bu27034_meas_set(data, true); - if (ret) - goto unlock_out; - - task = kthread_run(bu27034_buffer_thread, idev, - "bu27034-buffering-%u", - iio_device_id(idev)); - if (IS_ERR(task)) { - ret = PTR_ERR(task); - goto unlock_out; - } - - data->task = task; - -unlock_out: - mutex_unlock(&data->mutex); - - return ret; -} - -static int bu27034_buffer_disable(struct iio_dev *idev) -{ - struct bu27034_data *data = iio_priv(idev); - int ret; - - mutex_lock(&data->mutex); - if (data->task) { - kthread_stop(data->task); - data->task = NULL; - } - - ret = bu27034_meas_set(data, false); - mutex_unlock(&data->mutex); - - return ret; -} - -static const struct iio_buffer_setup_ops bu27034_buffer_ops = { - .postenable = &bu27034_buffer_enable, - .predisable = &bu27034_buffer_disable, -}; - -static int bu27034_probe(struct i2c_client *i2c) -{ - struct device *dev = &i2c->dev; - struct bu27034_data *data; - struct regmap *regmap; - struct iio_dev *idev; - unsigned int part_id, reg; - int ret; - - regmap = devm_regmap_init_i2c(i2c, &bu27034_regmap); - if (IS_ERR(regmap)) - return dev_err_probe(dev, PTR_ERR(regmap), - "Failed to initialize Regmap\n"); - - idev = devm_iio_device_alloc(dev, sizeof(*data)); - if (!idev) - return -ENOMEM; - - ret = devm_regulator_get_enable(dev, "vdd"); - if (ret) - return dev_err_probe(dev, ret, "Failed to get regulator\n"); - - data = iio_priv(idev); - - ret = regmap_read(regmap, BU27034_REG_SYSTEM_CONTROL, ®); - if (ret) - return dev_err_probe(dev, ret, "Failed to access sensor\n"); - - part_id = FIELD_GET(BU27034_MASK_PART_ID, reg); - - if (part_id != BU27034_ID) - dev_warn(dev, "unknown device 0x%x\n", part_id); - - ret = devm_iio_init_iio_gts(dev, BU27034_SCALE_1X, 0, bu27034_gains, - ARRAY_SIZE(bu27034_gains), bu27034_itimes, - ARRAY_SIZE(bu27034_itimes), &data->gts); - if (ret) - return ret; - - mutex_init(&data->mutex); - data->regmap = regmap; - data->dev = dev; - - idev->channels = bu27034_channels; - idev->num_channels = ARRAY_SIZE(bu27034_channels); - idev->name = "bu27034"; - idev->info = &bu27034_info; - - idev->modes = INDIO_DIRECT_MODE | INDIO_BUFFER_SOFTWARE; - idev->available_scan_masks = bu27034_scan_masks; - - ret = bu27034_chip_init(data); - if (ret) - return ret; - - ret = devm_iio_kfifo_buffer_setup(dev, idev, &bu27034_buffer_ops); - if (ret) - return dev_err_probe(dev, ret, "buffer setup failed\n"); - - ret = devm_iio_device_register(dev, idev); - if (ret < 0) - return dev_err_probe(dev, ret, - "Unable to register iio device\n"); - - return ret; -} - -static const struct of_device_id bu27034_of_match[] = { - { .compatible = "rohm,bu27034" }, - { } -}; -MODULE_DEVICE_TABLE(of, bu27034_of_match); - -static struct i2c_driver bu27034_i2c_driver = { - .driver = { - .name = "bu27034-als", - .of_match_table = bu27034_of_match, - .probe_type = PROBE_PREFER_ASYNCHRONOUS, - }, - .probe = bu27034_probe, -}; -module_i2c_driver(bu27034_i2c_driver); - -MODULE_LICENSE("GPL"); -MODULE_AUTHOR("Matti Vaittinen "); -MODULE_DESCRIPTION("ROHM BU27034 ambient light sensor driver"); -MODULE_IMPORT_NS(IIO_GTS_HELPER); diff --git a/drivers/iio/pressure/bmp280-core.c b/drivers/iio/pressure/bmp280-core.c index 8d9e8a1c94c4..b5f57711c77d 100644 --- a/drivers/iio/pressure/bmp280-core.c +++ b/drivers/iio/pressure/bmp280-core.c @@ -616,7 +616,7 @@ static int bmp280_write_sampling_frequency(struct bmp280_data *data, int val, int val2) { const int (*avail)[2] = data->chip_info->sampling_freq_avail; - const int n = data->chip_info->num_sampling_freq_avail; + const int n = data->chip_info->num_sampling_freq_avail / 2; int ret, prev; int i; diff --git a/drivers/iio/proximity/isl29501.c b/drivers/iio/proximity/isl29501.c index 9fd3d2e8cea6..86c074c399e5 100644 --- a/drivers/iio/proximity/isl29501.c +++ b/drivers/iio/proximity/isl29501.c @@ -227,7 +227,7 @@ static int isl29501_register_read(struct isl29501_private *isl29501, return ret; } -static u32 isl29501_register_write(struct isl29501_private *isl29501, +static int isl29501_register_write(struct isl29501_private *isl29501, enum isl29501_register_name name, u32 value) { diff --git a/drivers/iio/proximity/pulsedlight-lidar-lite-v2.c b/drivers/iio/proximity/pulsedlight-lidar-lite-v2.c index 2913d5e0fe4f..c7ff5496f39e 100644 --- a/drivers/iio/proximity/pulsedlight-lidar-lite-v2.c +++ b/drivers/iio/proximity/pulsedlight-lidar-lite-v2.c @@ -298,12 +298,14 @@ static int lidar_probe(struct i2c_client *client) ret = pm_runtime_set_active(&client->dev); if (ret) - goto error_unreg_buffer; + goto error_unreg_dev; pm_runtime_enable(&client->dev); pm_runtime_idle(&client->dev); return 0; +error_unreg_dev: + iio_device_unregister(indio_dev); error_unreg_buffer: iio_triggered_buffer_cleanup(indio_dev); diff --git a/drivers/iio/proximity/sx9324.c b/drivers/iio/proximity/sx9324.c index 438f9c9aba6e..9b6115131cb7 100644 --- a/drivers/iio/proximity/sx9324.c +++ b/drivers/iio/proximity/sx9324.c @@ -223,7 +223,7 @@ static const struct iio_chan_spec_ext_info sx9324_channel_ext_info[] = { .scan_index = idx, \ .scan_type = { \ .sign = 's', \ - .realbits = 12, \ + .realbits = 16, \ .storagebits = 16, \ .endianness = IIO_BE, \ }, \ diff --git a/drivers/iio/proximity/vcnl3020.c b/drivers/iio/proximity/vcnl3020.c index d1ddf85f5383..1887451c0c9a 100644 --- a/drivers/iio/proximity/vcnl3020.c +++ b/drivers/iio/proximity/vcnl3020.c @@ -584,16 +584,26 @@ static irqreturn_t vcnl3020_handle_irq_thread(int irq, void *p) return IRQ_HANDLED; } - if (!(isr & VCNL_ICR_THRES_EN)) + if (!(isr & (VCNL_INT_TH_HI | VCNL_INT_TH_LOW))) return IRQ_NONE; - iio_push_event(indio_dev, - IIO_UNMOD_EVENT_CODE(IIO_PROXIMITY, 1, - IIO_EV_TYPE_THRESH, - IIO_EV_DIR_RISING), - iio_get_time_ns(indio_dev)); + if (isr & VCNL_INT_TH_HI) { + iio_push_event(indio_dev, + IIO_UNMOD_EVENT_CODE(IIO_PROXIMITY, 0, + IIO_EV_TYPE_THRESH, + IIO_EV_DIR_RISING), + iio_get_time_ns(indio_dev)); + } + + if (isr & VCNL_INT_TH_LOW) { + iio_push_event(indio_dev, + IIO_UNMOD_EVENT_CODE(IIO_PROXIMITY, 0, + IIO_EV_TYPE_THRESH, + IIO_EV_DIR_FALLING), + iio_get_time_ns(indio_dev)); + } - rc = regmap_write(data->regmap, VCNL_ISR, isr & VCNL_ICR_THRES_EN); + rc = regmap_write(data->regmap, VCNL_ISR, isr & (VCNL_INT_TH_HI | VCNL_INT_TH_LOW)); if (rc) dev_err(data->dev, "Error (%d) writing in reg (0x%x)\n", rc, VCNL_ISR); diff --git a/drivers/infiniband/core/counters.c b/drivers/infiniband/core/counters.c index 8cbca897fc53..78a76b4fa6f6 100644 --- a/drivers/infiniband/core/counters.c +++ b/drivers/infiniband/core/counters.c @@ -190,12 +190,19 @@ static struct rdma_counter *alloc_and_bind(struct ib_device *dev, u32 port, ret = __rdma_counter_bind_qp(counter, qp); if (ret) - goto err_mode; + goto err_bind; rdma_restrack_parent_name(&counter->res, &qp->res); rdma_restrack_add(&counter->res); return counter; +err_bind: + mutex_lock(&port_counter->lock); + port_counter->num_counters--; + if (!port_counter->num_counters && + port_counter->mode.mode == RDMA_COUNTER_MODE_MANUAL) + __counter_set_mode(port_counter, RDMA_COUNTER_MODE_NONE, 0); + mutex_unlock(&port_counter->lock); err_mode: rdma_free_hw_stats_struct(counter->stats); err_stats: diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.c b/drivers/infiniband/hw/bnxt_re/ib_verbs.c index 7d6c296da3b6..a6a074ec2908 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.c +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.c @@ -1775,6 +1775,8 @@ int bnxt_re_create_srq(struct ib_srq *ib_srq, srq->qplib_srq.threshold = srq_init_attr->attr.srq_limit; srq->srq_limit = srq_init_attr->attr.srq_limit; srq->qplib_srq.eventq_hw_ring_id = rdev->nq[0].ring_id; + srq->qplib_srq.sg_info.pgsize = PAGE_SIZE; + srq->qplib_srq.sg_info.pgshft = PAGE_SHIFT; nq = &rdev->nq[0]; if (udata) { @@ -4152,6 +4154,7 @@ int bnxt_re_alloc_ucontext(struct ib_ucontext *ctx, struct ib_udata *udata) goto fail; } spin_lock_init(&uctx->sh_lock); + mutex_init(&uctx->wcdpi_lock); resp.comp_mask = BNXT_RE_UCNTX_CMASK_HAVE_CCTX; chip_met_rev_num = rdev->chip_ctx->chip_num; @@ -4314,8 +4317,8 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_ALLOC_PAGE)(struct uverbs_attr_bundle * struct bnxt_re_ucontext *uctx; struct bnxt_re_dev *rdev; u64 mmap_offset; + u32 dpi = 0; u32 length; - u32 dpi; u64 addr; int err; @@ -4332,14 +4335,23 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_ALLOC_PAGE)(struct uverbs_attr_bundle * switch (alloc_type) { case BNXT_RE_ALLOC_WC_PAGE: - if (cctx->modes.db_push) { + if (cctx->modes.db_push) { + mutex_lock(&uctx->wcdpi_lock); + /* already allocated — one WC page per context */ + if (uctx->wcdpi.dbr) { + mutex_unlock(&uctx->wcdpi_lock); + return -EEXIST; + } if (bnxt_qplib_alloc_dpi(&rdev->qplib_res, &uctx->wcdpi, - uctx, BNXT_QPLIB_DPI_TYPE_WC)) + uctx, BNXT_QPLIB_DPI_TYPE_WC)) { + mutex_unlock(&uctx->wcdpi_lock); return -ENOMEM; + } length = PAGE_SIZE; dpi = uctx->wcdpi.dpi; addr = (u64)uctx->wcdpi.umdbr; mmap_flag = BNXT_RE_MMAP_WC_DB; + mutex_unlock(&uctx->wcdpi_lock); } else { return -EINVAL; } @@ -4362,8 +4374,15 @@ static int UVERBS_HANDLER(BNXT_RE_METHOD_ALLOC_PAGE)(struct uverbs_attr_bundle * } entry = bnxt_re_mmap_entry_insert(uctx, addr, mmap_flag, &mmap_offset); - if (!entry) + if (!entry) { + if (mmap_flag == BNXT_RE_MMAP_WC_DB) { + mutex_lock(&uctx->wcdpi_lock); + bnxt_qplib_dealloc_dpi(&rdev->qplib_res, &uctx->wcdpi); + uctx->wcdpi.dbr = NULL; + mutex_unlock(&uctx->wcdpi_lock); + } return -ENOMEM; + } uobj->object = entry; uverbs_finalize_uobj_create(attrs, BNXT_RE_ALLOC_PAGE_HANDLE); @@ -4394,11 +4413,16 @@ static int alloc_page_obj_cleanup(struct ib_uobject *uobject, switch (entry->mmap_flag) { case BNXT_RE_MMAP_WC_DB: - if (uctx && uctx->wcdpi.dbr) { + if (uctx) { struct bnxt_re_dev *rdev = uctx->rdev; - bnxt_qplib_dealloc_dpi(&rdev->qplib_res, &uctx->wcdpi); - uctx->wcdpi.dbr = NULL; + mutex_lock(&uctx->wcdpi_lock); + if (uctx->wcdpi.dbr) { + bnxt_qplib_dealloc_dpi(&rdev->qplib_res, + &uctx->wcdpi); + uctx->wcdpi.dbr = NULL; + } + mutex_unlock(&uctx->wcdpi_lock); } break; case BNXT_RE_MMAP_DBR_BAR: diff --git a/drivers/infiniband/hw/bnxt_re/ib_verbs.h b/drivers/infiniband/hw/bnxt_re/ib_verbs.h index ef910e6e2ccb..99ab392f9e51 100644 --- a/drivers/infiniband/hw/bnxt_re/ib_verbs.h +++ b/drivers/infiniband/hw/bnxt_re/ib_verbs.h @@ -137,6 +137,7 @@ struct bnxt_re_ucontext { struct bnxt_re_dev *rdev; struct bnxt_qplib_dpi dpi; struct bnxt_qplib_dpi wcdpi; + struct mutex wcdpi_lock; /* serialises WC DPI alloc/free */ void *shpg; spinlock_t sh_lock; /* protect shpg */ struct rdma_user_mmap_entry *shpage_mmap; diff --git a/drivers/infiniband/hw/cxgb4/mem.c b/drivers/infiniband/hw/cxgb4/mem.c index 9c24789bbb54..9ed43d23c183 100644 --- a/drivers/infiniband/hw/cxgb4/mem.c +++ b/drivers/infiniband/hw/cxgb4/mem.c @@ -199,7 +199,8 @@ static int _c4iw_write_mem_dma(struct c4iw_rdev *rdev, u32 addr, u32 len, daddr = dma_map_single(&rdev->lldi.pdev->dev, data, len, DMA_TO_DEVICE); if (dma_mapping_error(&rdev->lldi.pdev->dev, daddr)) - return -1; + return _c4iw_write_mem_inline(rdev, addr, len, data, skb, + wr_waitp); save = daddr; while (remain > inline_threshold) { @@ -235,30 +236,12 @@ static int write_adapter_mem(struct c4iw_rdev *rdev, u32 addr, u32 len, void *data, struct sk_buff *skb, struct c4iw_wr_wait *wr_waitp) { - int ret; - - if (!rdev->lldi.ulptx_memwrite_dsgl || !use_dsgl) { - ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb, - wr_waitp); - goto out; - } - - if (len <= inline_threshold) { - ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb, + if (!rdev->lldi.ulptx_memwrite_dsgl || !use_dsgl || + len <= inline_threshold) + return _c4iw_write_mem_inline(rdev, addr, len, data, skb, wr_waitp); - goto out; - } - - ret = _c4iw_write_mem_dma(rdev, addr, len, data, skb, wr_waitp); - if (ret) { - pr_warn_ratelimited("%s: dma map failure (non fatal)\n", - pci_name(rdev->lldi.pdev)); - ret = _c4iw_write_mem_inline(rdev, addr, len, data, skb, - wr_waitp); - } -out: - return ret; + return _c4iw_write_mem_dma(rdev, addr, len, data, skb, wr_waitp); } /* @@ -280,12 +263,16 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry, bool stag_idx_allocated = false; static atomic_t key; - if (c4iw_fatal_error(rdev)) + if (c4iw_fatal_error(rdev)) { + kfree_skb(skb); return -EIO; + } tpt = kmalloc(sizeof(*tpt), GFP_KERNEL); - if (!tpt) + if (!tpt) { + kfree_skb(skb); return -ENOMEM; + } stag_state = stag_state > 0; stag_idx = (*stag) >> 8; @@ -297,6 +284,7 @@ static int write_tpt_entry(struct c4iw_rdev *rdev, u32 reset_tpt_entry, rdev->stats.stag.fail++; mutex_unlock(&rdev->stats.lock); kfree(tpt); + kfree_skb(skb); return -ENOMEM; } mutex_lock(&rdev->stats.lock); @@ -471,8 +459,10 @@ struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc) FW_RI_STAG_NSMR, mhp->attr.perms, mhp->attr.mw_bind_enable, 0, 0, ~0ULL, 0, 0, 0, NULL, mhp->wr_waitp); - if (ret) - goto err_free_skb; + if (ret) { + kfree_skb(mhp->dereg_skb); + goto err_free_wr_wait; + } ret = finish_mem_reg(mhp, stag); if (ret) @@ -481,8 +471,6 @@ struct ib_mr *c4iw_get_dma_mr(struct ib_pd *pd, int acc) err_dereg_mem: dereg_mem(&rhp->rdev, mhp->attr.stag, mhp->attr.pbl_size, mhp->attr.pbl_addr, mhp->dereg_skb, mhp->wr_waitp); -err_free_skb: - kfree_skb(mhp->dereg_skb); err_free_wr_wait: c4iw_put_wr_wait(mhp->wr_waitp); err_free_mhp: diff --git a/drivers/infiniband/hw/efa/efa_com.c b/drivers/infiniband/hw/efa/efa_com.c index 00e339abc2c1..090279dd0ab9 100644 --- a/drivers/infiniband/hw/efa/efa_com.c +++ b/drivers/infiniband/hw/efa/efa_com.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 OR BSD-2-Clause /* - * Copyright 2018-2024 Amazon.com, Inc. or its affiliates. All rights reserved. + * Copyright 2018-2026 Amazon.com, Inc. or its affiliates. All rights reserved. */ #include "efa_com.h" @@ -499,7 +499,6 @@ static int efa_com_wait_and_process_admin_cq_polling(struct efa_comp_ctx *comp_c { unsigned long timeout; unsigned long flags; - int err; timeout = jiffies + usecs_to_jiffies(aq->completion_timeout); @@ -519,24 +518,20 @@ static int efa_com_wait_and_process_admin_cq_polling(struct efa_comp_ctx *comp_c atomic64_inc(&aq->stats.no_completion); clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state); - err = -ETIME; - goto out; + return -ETIME; } msleep(aq->poll_interval); } - err = efa_com_comp_status_to_errno(comp_ctx->user_cqe->acq_common_descriptor.status); -out: - efa_com_put_comp_ctx(aq, comp_ctx); - return err; + return efa_com_comp_status_to_errno( + comp_ctx->user_cqe->acq_common_descriptor.status); } static int efa_com_wait_and_process_admin_cq_interrupts(struct efa_comp_ctx *comp_ctx, struct efa_com_admin_queue *aq) { unsigned long flags; - int err; wait_for_completion_timeout(&comp_ctx->wait_event, usecs_to_jiffies(aq->completion_timeout)); @@ -570,14 +565,11 @@ static int efa_com_wait_and_process_admin_cq_interrupts(struct efa_comp_ctx *com comp_ctx, aq->sq.pc, aq->sq.cc, aq->cq.cc); clear_bit(EFA_AQ_STATE_RUNNING_BIT, &aq->state); - err = -ETIME; - goto out; + return -ETIME; } - err = efa_com_comp_status_to_errno(comp_ctx->user_cqe->acq_common_descriptor.status); -out: - efa_com_put_comp_ctx(aq, comp_ctx); - return err; + return efa_com_comp_status_to_errno( + comp_ctx->user_cqe->acq_common_descriptor.status); } /* @@ -644,13 +636,13 @@ int efa_com_cmd_exec(struct efa_com_admin_queue *aq, if (err) { ibdev_err_ratelimited( aq->efa_dev, - "Failed to process command %s (opcode %u) comp_status %d err %d\n", + "Failed to process command %s (opcode %u) err %d\n", efa_com_cmd_str(cmd->aq_common_descriptor.opcode), - cmd->aq_common_descriptor.opcode, - comp_ctx->user_cqe->acq_common_descriptor.status, err); + cmd->aq_common_descriptor.opcode, err); atomic64_inc(&aq->stats.cmd_err); } + efa_com_put_comp_ctx(aq, comp_ctx); up(&aq->avail_cmds); return err; diff --git a/drivers/infiniband/hw/erdma/erdma_cm.c b/drivers/infiniband/hw/erdma/erdma_cm.c index e349e8d2fb50..c747f77c1bdb 100644 --- a/drivers/infiniband/hw/erdma/erdma_cm.c +++ b/drivers/infiniband/hw/erdma/erdma_cm.c @@ -1017,10 +1017,9 @@ int erdma_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) if (laddr->sa_family != AF_INET || raddr->sa_family != AF_INET) return -EAFNOSUPPORT; - qp = find_qp_by_qpn(dev, params->qpn); + qp = erdma_qp_get_by_qpn(dev, params->qpn); if (!qp) return -ENOENT; - erdma_qp_get(qp); ret = sock_create(AF_INET, SOCK_STREAM, IPPROTO_TCP, &s); if (ret < 0) @@ -1149,10 +1148,9 @@ int erdma_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params) return -ECONNRESET; } - qp = find_qp_by_qpn(dev, params->qpn); + qp = erdma_qp_get_by_qpn(dev, params->qpn); if (!qp) return -ENOENT; - erdma_qp_get(qp); down_write(&qp->state_lock); if (qp->attrs.state > ERDMA_QP_STATE_RTR) { diff --git a/drivers/infiniband/hw/erdma/erdma_eq.c b/drivers/infiniband/hw/erdma/erdma_eq.c index 3fec9cd18afa..9d5b75abf06a 100644 --- a/drivers/infiniband/hw/erdma/erdma_eq.c +++ b/drivers/infiniband/hw/erdma/erdma_eq.c @@ -65,7 +65,7 @@ void erdma_aeq_event_handler(struct erdma_dev *dev) erdma_cq_put(cq); } else { qpn = le32_to_cpu(aeqe->event_data0); - qp = find_qp_by_qpn(dev, qpn); + qp = erdma_qp_get_by_qpn(dev, qpn); if (!qp) continue; @@ -75,6 +75,7 @@ void erdma_aeq_event_handler(struct erdma_dev *dev) if (qp->ibqp.event_handler) qp->ibqp.event_handler(&event, qp->ibqp.qp_context); + erdma_qp_put(qp); } } diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.c b/drivers/infiniband/hw/erdma/erdma_verbs.c index 066144ae3b24..a9b2d0890570 100644 --- a/drivers/infiniband/hw/erdma/erdma_verbs.c +++ b/drivers/infiniband/hw/erdma/erdma_verbs.c @@ -1211,8 +1211,15 @@ int erdma_dereg_mr(struct ib_mr *ibmr, struct ib_udata *udata) FIELD_PREP(ERDMA_CMD_MR_KEY_MASK, ibmr->lkey & 0xFF); ret = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL); + /* + * A timeout disables the command queue, so retry cannot succeed. Treat + * terminal command failures as diagnostic; propagating them can make + * forced uverbs cleanup discard the last software resource pointers. + */ if (ret) - return ret; + ibdev_warn_ratelimited(&dev->ibdev, + "failed to deregister MR 0x%x: %d\n", + ibmr->lkey, ret); erdma_free_idx(&dev->res_cb[ERDMA_RES_TYPE_STAG_IDX], ibmr->lkey >> 8); @@ -1238,7 +1245,9 @@ int erdma_destroy_cq(struct ib_cq *ibcq, struct ib_udata *udata) err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL); if (err) - return err; + ibdev_warn_ratelimited(&dev->ibdev, + "failed to destroy CQ %u: %d\n", + cq->cqn, err); xa_lock_irqsave(&dev->cq_xa, flags); __xa_erase(&dev->cq_xa, cq->cqn); @@ -1266,6 +1275,7 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) struct erdma_ucontext *ctx = rdma_udata_to_drv_context( udata, struct erdma_ucontext, ibucontext); struct erdma_qp_attrs qp_attrs; + unsigned long flags; int err; struct erdma_cmdq_destroy_qp_req req; @@ -1282,7 +1292,13 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) err = erdma_post_cmd_wait(&dev->cmdq, &req, sizeof(req), NULL, NULL); if (err) - return err; + ibdev_warn_ratelimited(&dev->ibdev, + "failed to destroy QP %u: %d\n", + QP_ID(qp), err); + + xa_lock_irqsave(&dev->qp_xa, flags); + __xa_erase(&dev->qp_xa, QP_ID(qp)); + xa_unlock_irqrestore(&dev->qp_xa, flags); erdma_qp_put(qp); wait_for_completion(&qp->safe_free); @@ -1306,7 +1322,6 @@ int erdma_destroy_qp(struct ib_qp *ibqp, struct ib_udata *udata) if (qp->cep) erdma_cep_put(qp->cep); - xa_erase(&dev->qp_xa, QP_ID(qp)); return 0; } diff --git a/drivers/infiniband/hw/erdma/erdma_verbs.h b/drivers/infiniband/hw/erdma/erdma_verbs.h index 10b9b8a5f6a8..687017531b95 100644 --- a/drivers/infiniband/hw/erdma/erdma_verbs.h +++ b/drivers/infiniband/hw/erdma/erdma_verbs.h @@ -280,6 +280,21 @@ static inline struct erdma_qp *find_qp_by_qpn(struct erdma_dev *dev, int id) return (struct erdma_qp *)xa_load(&dev->qp_xa, id); } +static inline struct erdma_qp *erdma_qp_get_by_qpn(struct erdma_dev *dev, + int id) +{ + struct erdma_qp *qp; + unsigned long flags; + + xa_lock_irqsave(&dev->qp_xa, flags); + qp = xa_load(&dev->qp_xa, id); + if (qp && !kref_get_unless_zero(&qp->ref)) + qp = NULL; + xa_unlock_irqrestore(&dev->qp_xa, flags); + + return qp; +} + static inline struct erdma_cq *erdma_cq_get_by_cqn(struct erdma_dev *dev, int id) { diff --git a/drivers/infiniband/hw/hns/hns_roce_cq.c b/drivers/infiniband/hw/hns/hns_roce_cq.c index 5e0d78f4e545..c72bbf21f1ae 100644 --- a/drivers/infiniband/hw/hns/hns_roce_cq.c +++ b/drivers/infiniband/hw/hns/hns_roce_cq.c @@ -119,9 +119,9 @@ static int hns_roce_create_cqc(struct hns_roce_dev *hr_dev, ret = hns_roce_create_hw_ctx(hr_dev, mailbox, HNS_ROCE_CMD_CREATE_CQC, hr_cq->cqn); if (ret) - ibdev_err(ibdev, - "failed to send create cmd for CQ(0x%lx), ret = %d.\n", - hr_cq->cqn, ret); + ibdev_err_ratelimited(ibdev, + "failed to send create cmd for CQ(0x%lx), ret = %d.\n", + hr_cq->cqn, ret); hns_roce_free_cmd_mailbox(hr_dev, mailbox); diff --git a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c index b50529a65274..ef702fd9854b 100644 --- a/drivers/infiniband/hw/hns/hns_roce_hw_v2.c +++ b/drivers/infiniband/hw/hns/hns_roce_hw_v2.c @@ -2905,14 +2905,22 @@ static int hns_roce_v2_init(struct hns_roce_dev *hr_dev) { int ret; + if (hr_dev->pci_dev->revision == PCI_REVISION_ID_HIP08) { + ret = free_mr_init(hr_dev); + if (ret) { + dev_err(hr_dev->dev, "failed to init free mr!\n"); + return ret; + } + } + /* The hns ROCEE requires the extdb info to be cleared before using */ ret = hns_roce_clear_extdb_list_info(hr_dev); if (ret) - return ret; + goto err_clear_extdb_failed; ret = get_hem_table(hr_dev); if (ret) - return ret; + goto err_clear_extdb_failed; if (hr_dev->is_vf) return 0; @@ -2927,6 +2935,9 @@ static int hns_roce_v2_init(struct hns_roce_dev *hr_dev) err_llm_init_failed: put_hem_table(hr_dev); +err_clear_extdb_failed: + if (hr_dev->pci_dev->revision == PCI_REVISION_ID_HIP08) + free_mr_exit(hr_dev); return ret; } @@ -5702,9 +5713,9 @@ static int hns_roce_v2_modify_srq(struct ib_srq *ibsrq, HNS_ROCE_CMD_MODIFY_SRQC, srq->srqn); hns_roce_free_cmd_mailbox(hr_dev, mailbox); if (ret) { - ibdev_err(&hr_dev->ib_dev, - "failed to handle cmd of modifying SRQ, ret = %d.\n", - ret); + ibdev_err_ratelimited(&hr_dev->ib_dev, + "failed to handle cmd of modifying SRQ, ret = %d.\n", + ret); return ret; } } @@ -5728,9 +5739,9 @@ static int hns_roce_v2_query_srq(struct ib_srq *ibsrq, struct ib_srq_attr *attr) ret = hns_roce_cmd_mbox(hr_dev, 0, mailbox->dma, HNS_ROCE_CMD_QUERY_SRQC, srq->srqn); if (ret) { - ibdev_err(&hr_dev->ib_dev, - "failed to process cmd of querying SRQ, ret = %d.\n", - ret); + ibdev_err_ratelimited(&hr_dev->ib_dev, + "failed to process cmd of querying SRQ, ret = %d.\n", + ret); goto out; } @@ -5831,9 +5842,9 @@ static int hns_roce_v2_query_mpt(struct hns_roce_dev *hr_dev, u32 key, ret = hns_roce_cmd_mbox(hr_dev, 0, mailbox->dma, HNS_ROCE_CMD_QUERY_MPT, key_to_hw_index(key)); if (ret) { - ibdev_err(&hr_dev->ib_dev, - "failed to process cmd when querying MPT, ret = %d.\n", - ret); + ibdev_err_ratelimited(&hr_dev->ib_dev, + "failed to process cmd when querying MPT, ret = %d.\n", + ret); goto err_mailbox; } @@ -6801,21 +6812,11 @@ static int __hns_roce_hw_v2_init_instance(struct hnae3_handle *handle) goto error_failed_roce_init; } - if (hr_dev->pci_dev->revision == PCI_REVISION_ID_HIP08) { - ret = free_mr_init(hr_dev); - if (ret) { - dev_err(hr_dev->dev, "failed to init free mr!\n"); - goto error_failed_free_mr_init; - } - } handle->priv = hr_dev; return 0; -error_failed_free_mr_init: - hns_roce_exit(hr_dev); - error_failed_roce_init: kfree(hr_dev->priv); diff --git a/drivers/infiniband/hw/hns/hns_roce_main.c b/drivers/infiniband/hw/hns/hns_roce_main.c index a67b3291e68f..26f1d1c8d7f1 100644 --- a/drivers/infiniband/hw/hns/hns_roce_main.c +++ b/drivers/infiniband/hw/hns/hns_roce_main.c @@ -924,6 +924,9 @@ static int hns_roce_setup_hca(struct hns_roce_dev *hr_dev) spin_lock_init(&hr_dev->sm_lock); + INIT_LIST_HEAD(&hr_dev->qp_list); + spin_lock_init(&hr_dev->qp_list_lock); + if (hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_CQ_RECORD_DB || hr_dev->caps.flags & HNS_ROCE_CAP_FLAG_QP_RECORD_DB) { INIT_LIST_HEAD(&hr_dev->pgdir_list); @@ -1068,8 +1071,6 @@ int hns_roce_init(struct hns_roce_dev *hr_dev) } } - INIT_LIST_HEAD(&hr_dev->qp_list); - spin_lock_init(&hr_dev->qp_list_lock); INIT_LIST_HEAD(&hr_dev->dip_list); spin_lock_init(&hr_dev->dip_list_lock); diff --git a/drivers/infiniband/hw/hns/hns_roce_mr.c b/drivers/infiniband/hw/hns/hns_roce_mr.c index aca079c120fb..6ecc292f25e9 100644 --- a/drivers/infiniband/hw/hns/hns_roce_mr.c +++ b/drivers/infiniband/hw/hns/hns_roce_mr.c @@ -163,7 +163,7 @@ static int hns_roce_mr_enable(struct hns_roce_dev *hr_dev, ret = hns_roce_create_hw_ctx(hr_dev, mailbox, HNS_ROCE_CMD_CREATE_MPT, mtpt_idx & (hr_dev->caps.num_mtpts - 1)); if (ret) { - dev_err(dev, "failed to create mpt, ret = %d.\n", ret); + dev_err_ratelimited(dev, "failed to create mpt, ret = %d.\n", ret); goto err_page; } @@ -295,7 +295,7 @@ struct ib_mr *hns_roce_rereg_user_mr(struct ib_mr *ibmr, int flags, u64 start, ret = hns_roce_destroy_hw_ctx(hr_dev, HNS_ROCE_CMD_DESTROY_MPT, mtpt_idx); if (ret) - ibdev_warn(ib_dev, "failed to destroy MPT, ret = %d.\n", ret); + ibdev_warn_ratelimited(ib_dev, "failed to destroy MPT, ret = %d.\n", ret); mr->enabled = 0; mr->iova = virt_addr; @@ -326,7 +326,7 @@ struct ib_mr *hns_roce_rereg_user_mr(struct ib_mr *ibmr, int flags, u64 start, ret = hns_roce_create_hw_ctx(hr_dev, mailbox, HNS_ROCE_CMD_CREATE_MPT, mtpt_idx); if (ret) { - ibdev_err(ib_dev, "failed to create MPT, ret = %d.\n", ret); + ibdev_err_ratelimited(ib_dev, "failed to create MPT, ret = %d.\n", ret); goto free_cmd_mbox; } diff --git a/drivers/infiniband/hw/hns/hns_roce_srq.c b/drivers/infiniband/hw/hns/hns_roce_srq.c index 80fcb1b0e8fd..c699bf48adb4 100644 --- a/drivers/infiniband/hw/hns/hns_roce_srq.c +++ b/drivers/infiniband/hw/hns/hns_roce_srq.c @@ -103,7 +103,7 @@ static int hns_roce_create_srqc(struct hns_roce_dev *hr_dev, ret = hns_roce_create_hw_ctx(hr_dev, mailbox, HNS_ROCE_CMD_CREATE_SRQ, srq->srqn); if (ret) - ibdev_err(ibdev, "failed to config SRQC, ret = %d.\n", ret); + ibdev_err_ratelimited(ibdev, "failed to config SRQC, ret = %d.\n", ret); err_mbox: hns_roce_free_cmd_mailbox(hr_dev, mailbox); diff --git a/drivers/infiniband/hw/irdma/hw.c b/drivers/infiniband/hw/irdma/hw.c index 1745f40b075f..fd18474a3d2f 100644 --- a/drivers/infiniband/hw/irdma/hw.c +++ b/drivers/infiniband/hw/irdma/hw.c @@ -1001,9 +1001,9 @@ static int irdma_create_cqp(struct irdma_pci_f *rf) INIT_LIST_HEAD(&cqp->cqp_avail_reqs); INIT_LIST_HEAD(&cqp->cqp_pending_reqs); - /* init the waitqueue of the cqp_requests and add them to the list */ + /* init the completion of the cqp_requests and add them to the list */ for (i = 0; i < sqsize; i++) { - init_waitqueue_head(&cqp->cqp_requests[i].waitq); + init_completion(&cqp->cqp_requests[i].comp); list_add_tail(&cqp->cqp_requests[i].list, &cqp->cqp_avail_reqs); } init_waitqueue_head(&cqp->remove_wq); @@ -2103,8 +2103,7 @@ void irdma_cqp_ce_handler(struct irdma_pci_f *rf, struct irdma_sc_cq *cq) cqp_request->compl_info.error = info.error; if (cqp_request->waiting) { - WRITE_ONCE(cqp_request->request_done, true); - wake_up(&cqp_request->waitq); + complete_all(&cqp_request->comp); irdma_put_cqp_request(&rf->cqp, cqp_request); } else { if (cqp_request->callback_fcn) diff --git a/drivers/infiniband/hw/irdma/main.h b/drivers/infiniband/hw/irdma/main.h index 19bc3842642f..456208395ad8 100644 --- a/drivers/infiniband/hw/irdma/main.h +++ b/drivers/infiniband/hw/irdma/main.h @@ -155,13 +155,12 @@ struct irdma_cqp_compl_info { struct irdma_cqp_request { struct cqp_cmds_info info; - wait_queue_head_t waitq; + struct completion comp; struct list_head list; refcount_t refcnt; void (*callback_fcn)(struct irdma_cqp_request *cqp_request); void *param; struct irdma_cqp_compl_info compl_info; - bool request_done; /* READ/WRITE_ONCE macros operate on it */ bool waiting:1; bool dynamic:1; }; diff --git a/drivers/infiniband/hw/irdma/utils.c b/drivers/infiniband/hw/irdma/utils.c index 21c83f000a78..834cf4254605 100644 --- a/drivers/infiniband/hw/irdma/utils.c +++ b/drivers/infiniband/hw/irdma/utils.c @@ -445,7 +445,7 @@ struct irdma_cqp_request *irdma_alloc_and_get_cqp_request(struct irdma_cqp *cqp, if (cqp_request) { cqp_request->dynamic = true; if (wait) - init_waitqueue_head(&cqp_request->waitq); + init_completion(&cqp_request->comp); } } if (!cqp_request) { @@ -482,7 +482,7 @@ void irdma_free_cqp_request(struct irdma_cqp *cqp, if (cqp_request->dynamic) { kfree(cqp_request); } else { - WRITE_ONCE(cqp_request->request_done, false); + reinit_completion(&cqp_request->comp); cqp_request->callback_fcn = NULL; cqp_request->waiting = false; @@ -516,8 +516,7 @@ irdma_free_pending_cqp_request(struct irdma_cqp *cqp, { if (cqp_request->waiting) { cqp_request->compl_info.error = true; - WRITE_ONCE(cqp_request->request_done, true); - wake_up(&cqp_request->waitq); + complete_all(&cqp_request->comp); } wait_event_timeout(cqp->remove_wq, refcount_read(&cqp_request->refcnt) == 1, 1000); @@ -571,9 +570,8 @@ static int irdma_wait_event(struct irdma_pci_f *rf, cqp_timeout.compl_cqp_cmds = atomic64_read(&rf->sc_dev.cqp->completed_ops); do { irdma_cqp_ce_handler(rf, &rf->ccq.sc_cq); - if (wait_event_timeout(cqp_request->waitq, - READ_ONCE(cqp_request->request_done), - msecs_to_jiffies(CQP_COMPL_WAIT_TIME_MS))) + if (wait_for_completion_timeout(&cqp_request->comp, + msecs_to_jiffies(CQP_COMPL_WAIT_TIME_MS))) break; irdma_check_cqp_progress(&cqp_timeout, &rf->sc_dev); diff --git a/drivers/infiniband/hw/irdma/verbs.c b/drivers/infiniband/hw/irdma/verbs.c index c732dfefe7e3..a02c551052dc 100644 --- a/drivers/infiniband/hw/irdma/verbs.c +++ b/drivers/infiniband/hw/irdma/verbs.c @@ -602,18 +602,17 @@ static int irdma_setup_umode_qp(struct ib_udata *udata, iwqp->ctx_info.qp_compl_ctx = req.user_compl_ctx; iwqp->user_mode = 1; - if (req.user_wqe_bufs) { - info->qp_uk_init_info.legacy_mode = ucontext->legacy_mode; - spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags); - iwqp->iwpbl = irdma_get_pbl((unsigned long)req.user_wqe_bufs, - &ucontext->qp_reg_mem_list); - spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags); - if (!iwqp->iwpbl) { - ret = -ENODATA; - ibdev_dbg(&iwdev->ibdev, "VERBS: no pbl info\n"); - return ret; - } + info->qp_uk_init_info.legacy_mode = ucontext->legacy_mode; + spin_lock_irqsave(&ucontext->qp_reg_mem_list_lock, flags); + iwqp->iwpbl = irdma_get_pbl((unsigned long)req.user_wqe_bufs, + &ucontext->qp_reg_mem_list); + spin_unlock_irqrestore(&ucontext->qp_reg_mem_list_lock, flags); + + if (!iwqp->iwpbl) { + ret = -ENODATA; + ibdev_dbg(&iwdev->ibdev, "VERBS: no pbl info\n"); + return ret; } if (!ucontext->use_raw_attrs) { diff --git a/drivers/infiniband/hw/mlx5/cq.c b/drivers/infiniband/hw/mlx5/cq.c index 51194137ef6d..219c010846cc 100644 --- a/drivers/infiniband/hw/mlx5/cq.c +++ b/drivers/infiniband/hw/mlx5/cq.c @@ -1012,15 +1012,18 @@ int mlx5_ib_create_cq(struct ib_cq *ibcq, const struct ib_cq_init_attr *attr, if (cq->create_flags & IB_UVERBS_CQ_FLAGS_IGNORE_OVERRUN) MLX5_SET(cqc, cqc, oi, 1); + if (udata) { + cq->mcq.comp = mlx5_add_cq_to_tasklet; + cq->mcq.tasklet_ctx.comp = mlx5_ib_cq_comp; + } else { + cq->mcq.comp = mlx5_ib_cq_comp; + } + err = mlx5_core_create_cq(dev->mdev, &cq->mcq, cqb, inlen, out, sizeof(out)); if (err) goto err_cqb; mlx5_ib_dbg(dev, "cqn 0x%x\n", cq->mcq.cqn); - if (udata) - cq->mcq.tasklet_ctx.comp = mlx5_ib_cq_comp; - else - cq->mcq.comp = mlx5_ib_cq_comp; cq->mcq.event = mlx5_ib_cq_event; INIT_LIST_HEAD(&cq->wc_list); diff --git a/drivers/infiniband/hw/mlx5/devx.c b/drivers/infiniband/hw/mlx5/devx.c index 0aee5738e298..51e7689c1731 100644 --- a/drivers/infiniband/hw/mlx5/devx.c +++ b/drivers/infiniband/hw/mlx5/devx.c @@ -2515,7 +2515,7 @@ static void dispatch_event_fd(struct list_head *fd_list, list_for_each_entry_rcu(item, fd_list, xa_list) { if (item->eventfd) - eventfd_signal(item->eventfd, 1); + eventfd_signal(item->eventfd); else deliver_event(item, data); } diff --git a/drivers/infiniband/hw/mlx5/main.c b/drivers/infiniband/hw/mlx5/main.c index df9d4b8373d1..b4875b639da5 100644 --- a/drivers/infiniband/hw/mlx5/main.c +++ b/drivers/infiniband/hw/mlx5/main.c @@ -2719,7 +2719,6 @@ static void mlx5_ib_handle_event(struct work_struct *_work) container_of(_work, struct mlx5_ib_event_work, work); struct mlx5_ib_dev *ibdev; struct ib_event ibev; - bool fatal = false; if (work->is_slave) { ibdev = mlx5_ib_get_ibdev_from_mpi(work->mpi); @@ -2730,12 +2729,6 @@ static void mlx5_ib_handle_event(struct work_struct *_work) } switch (work->event) { - case MLX5_DEV_EVENT_SYS_ERROR: - ibev.event = IB_EVENT_DEVICE_FATAL; - mlx5_ib_handle_internal_error(ibdev); - ibev.element.port_num = (u8)(unsigned long)work->param; - fatal = true; - break; case MLX5_EVENT_TYPE_PORT_CHANGE: if (handle_port_change(ibdev, work->param, &ibev)) goto out; @@ -2757,8 +2750,6 @@ static void mlx5_ib_handle_event(struct work_struct *_work) if (ibdev->ib_active) ib_dispatch_event(&ibev); - if (fatal) - ibdev->ib_active = false; out: kfree(work); } @@ -2802,6 +2793,66 @@ static int mlx5_ib_event_slave_port(struct notifier_block *nb, return NOTIFY_OK; } +static void mlx5_ib_handle_sys_error_event(struct work_struct *_work) +{ + struct mlx5_ib_event_work *work = + container_of(_work, struct mlx5_ib_event_work, work); + struct mlx5_ib_dev *ibdev = work->dev; + struct ib_event ibev; + + ibev.event = IB_EVENT_DEVICE_FATAL; + mlx5_ib_handle_internal_error(ibdev); + ibev.element.port_num = (u8)(unsigned long)work->param; + ibev.device = &ibdev->ib_dev; + + if (!rdma_is_port_valid(&ibdev->ib_dev, ibev.element.port_num)) { + mlx5_ib_warn(ibdev, "warning: event on port %d\n", ibev.element.port_num); + goto out; + } + + if (ibdev->ib_active) + ib_dispatch_event(&ibev); + + ibdev->ib_active = false; +out: + kfree(work); +} + +static int mlx5_ib_sys_error_event(struct notifier_block *nb, + unsigned long event, void *param) +{ + struct mlx5_ib_event_work *work; + + if (event != MLX5_DEV_EVENT_SYS_ERROR) + return NOTIFY_DONE; + + work = kmalloc(sizeof(*work), GFP_ATOMIC); + if (!work) + return NOTIFY_DONE; + + INIT_WORK(&work->work, mlx5_ib_handle_sys_error_event); + work->dev = container_of(nb, struct mlx5_ib_dev, sys_error_events); + work->is_slave = false; + work->param = param; + work->event = event; + + queue_work(mlx5_ib_event_wq, &work->work); + + return NOTIFY_OK; +} + +static int mlx5_ib_stage_sys_error_notifier_init(struct mlx5_ib_dev *dev) +{ + dev->sys_error_events.notifier_call = mlx5_ib_sys_error_event; + mlx5_notifier_register(dev->mdev, &dev->sys_error_events); + return 0; +} + +static void mlx5_ib_stage_sys_error_notifier_cleanup(struct mlx5_ib_dev *dev) +{ + mlx5_notifier_unregister(dev->mdev, &dev->sys_error_events); +} + static int set_has_smi_cap(struct mlx5_ib_dev *dev) { struct mlx5_hca_vport_context vport_ctx; @@ -4363,6 +4414,9 @@ static const struct mlx5_ib_profile pf_profile = { STAGE_CREATE(MLX5_IB_STAGE_WHITELIST_UID, mlx5_ib_devx_init, mlx5_ib_devx_cleanup), + STAGE_CREATE(MLX5_IB_STAGE_SYS_ERROR_NOTIFIER, + mlx5_ib_stage_sys_error_notifier_init, + mlx5_ib_stage_sys_error_notifier_cleanup), STAGE_CREATE(MLX5_IB_STAGE_IB_REG, mlx5_ib_stage_ib_reg_init, mlx5_ib_stage_ib_reg_cleanup), @@ -4423,6 +4477,9 @@ const struct mlx5_ib_profile raw_eth_profile = { STAGE_CREATE(MLX5_IB_STAGE_WHITELIST_UID, mlx5_ib_devx_init, mlx5_ib_devx_cleanup), + STAGE_CREATE(MLX5_IB_STAGE_SYS_ERROR_NOTIFIER, + mlx5_ib_stage_sys_error_notifier_init, + mlx5_ib_stage_sys_error_notifier_cleanup), STAGE_CREATE(MLX5_IB_STAGE_IB_REG, mlx5_ib_stage_ib_reg_init, mlx5_ib_stage_ib_reg_cleanup), diff --git a/drivers/infiniband/hw/mlx5/mlx5_ib.h b/drivers/infiniband/hw/mlx5/mlx5_ib.h index 9b780f48a382..e9257178454a 100644 --- a/drivers/infiniband/hw/mlx5/mlx5_ib.h +++ b/drivers/infiniband/hw/mlx5/mlx5_ib.h @@ -961,6 +961,7 @@ enum mlx5_ib_stages { MLX5_IB_STAGE_BFREG, MLX5_IB_STAGE_PRE_IB_REG_UMR, MLX5_IB_STAGE_WHITELIST_UID, + MLX5_IB_STAGE_SYS_ERROR_NOTIFIER, MLX5_IB_STAGE_IB_REG, MLX5_IB_STAGE_DEVICE_NOTIFIER, MLX5_IB_STAGE_POST_IB_REG_UMR, @@ -1116,6 +1117,7 @@ struct mlx5_ib_dev { struct ib_device ib_dev; struct mlx5_core_dev *mdev; struct notifier_block mdev_events; + struct notifier_block sys_error_events; int num_ports; /* serialize update of capability mask */ diff --git a/drivers/infiniband/hw/mlx5/mr.c b/drivers/infiniband/hw/mlx5/mr.c index 25348f732ac5..9f3644a6e503 100644 --- a/drivers/infiniband/hw/mlx5/mr.c +++ b/drivers/infiniband/hw/mlx5/mr.c @@ -1834,7 +1834,7 @@ mlx5_alloc_priv_descs(struct ib_device *device, static void mlx5_free_priv_descs(struct mlx5_ib_mr *mr) { - if (!mr->umem && mr->descs) { + if (!mr->umem && mr->ibmr.type != IB_MR_TYPE_DM && mr->descs) { struct ib_device *device = mr->ibmr.device; int size = mr->max_descs * mr->desc_size; struct mlx5_ib_dev *dev = to_mdev(device); diff --git a/drivers/infiniband/hw/mlx5/odp.c b/drivers/infiniband/hw/mlx5/odp.c index 7ad5db46ffce..98e7a55ec8a4 100644 --- a/drivers/infiniband/hw/mlx5/odp.c +++ b/drivers/infiniband/hw/mlx5/odp.c @@ -213,10 +213,23 @@ static void destroy_unused_implicit_child_mr(struct mlx5_ib_mr *mr) unsigned long idx = ib_umem_start(odp) >> MLX5_IMR_MTT_SHIFT; struct mlx5_ib_mr *imr = mr->parent; + /* + * If userspace is racing freeing the parent implicit ODP MR then we can + * loose the race with parent destruction. In this case + * mlx5_ib_free_odp_mr() will free everything in the implicit_children + * xarray so NOP is fine. This child MR cannot be destroyed here because + * we are under its umem_mutex. + */ if (!refcount_inc_not_zero(&imr->mmkey.usecount)) return; - xa_erase(&imr->implicit_children, idx); + xa_lock(&imr->implicit_children); + if (__xa_cmpxchg(&imr->implicit_children, idx, mr, NULL, GFP_KERNEL) != + mr) { + xa_unlock(&imr->implicit_children); + return; + } + xa_unlock(&imr->implicit_children); /* Freeing a MR is a sleeping operation, so bounce to a work queue */ INIT_WORK(&mr->odp_destroy.work, free_implicit_child_mr_work); diff --git a/drivers/infiniband/hw/mlx5/umr.c b/drivers/infiniband/hw/mlx5/umr.c index 50f97fff69e8..80c85b6c2492 100644 --- a/drivers/infiniband/hw/mlx5/umr.c +++ b/drivers/infiniband/hw/mlx5/umr.c @@ -200,30 +200,6 @@ void mlx5r_umr_resource_cleanup(struct mlx5_ib_dev *dev) ib_dealloc_pd(dev->umrc.pd); } -static int mlx5r_umr_recover(struct mlx5_ib_dev *dev) -{ - struct umr_common *umrc = &dev->umrc; - struct ib_qp_attr attr; - int err; - - attr.qp_state = IB_QPS_RESET; - err = ib_modify_qp(umrc->qp, &attr, IB_QP_STATE); - if (err) { - mlx5_ib_dbg(dev, "Couldn't modify UMR QP\n"); - goto err; - } - - err = mlx5r_umr_qp_rst2rts(dev, umrc->qp); - if (err) - goto err; - - umrc->state = MLX5_UMR_STATE_ACTIVE; - return 0; - -err: - umrc->state = MLX5_UMR_STATE_ERR; - return err; -} static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe, struct mlx5r_umr_wqe *wqe, bool with_data) @@ -271,6 +247,61 @@ static int mlx5r_umr_post_send(struct ib_qp *ibqp, u32 mkey, struct ib_cqe *cqe, return err; } +static int mlx5r_umr_recover(struct mlx5_ib_dev *dev, u32 mkey, + struct mlx5r_umr_context *umr_context, + struct mlx5r_umr_wqe *wqe, bool with_data) +{ + struct umr_common *umrc = &dev->umrc; + struct ib_qp_attr attr; + int err; + + mutex_lock(&umrc->lock); + /* Preventing any further WRs to be sent now */ + if (umrc->state != MLX5_UMR_STATE_RECOVER) { + mlx5_ib_warn(dev, "UMR recovery encountered an unexpected state=%d\n", + umrc->state); + umrc->state = MLX5_UMR_STATE_RECOVER; + } + mutex_unlock(&umrc->lock); + + /* Sending a final/barrier WR (the failed one) and wait for its completion. + * This will ensure that all the previous WRs got a completion before + * we set the QP state to RESET. + */ + err = mlx5r_umr_post_send(umrc->qp, mkey, &umr_context->cqe, wqe, + with_data); + if (err) { + mlx5_ib_warn(dev, "UMR recovery post send failed, err %d\n", err); + goto err; + } + + /* Since the QP is in an error state, it will only receive + * IB_WC_WR_FLUSH_ERR. However, as it serves only as a barrier + * we don't care about its status. + */ + wait_for_completion(&umr_context->done); + + attr.qp_state = IB_QPS_RESET; + err = ib_modify_qp(umrc->qp, &attr, IB_QP_STATE); + if (err) { + mlx5_ib_warn(dev, "Couldn't modify UMR QP to RESET, err=%d\n", err); + goto err; + } + + err = mlx5r_umr_qp_rst2rts(dev, umrc->qp); + if (err) { + mlx5_ib_warn(dev, "Couldn't modify UMR QP to RTS, err=%d\n", err); + goto err; + } + + umrc->state = MLX5_UMR_STATE_ACTIVE; + return 0; + +err: + umrc->state = MLX5_UMR_STATE_ERR; + return err; +} + static void mlx5r_umr_done(struct ib_cq *cq, struct ib_wc *wc) { struct mlx5_ib_umr_context *context = @@ -333,11 +364,9 @@ static int mlx5r_umr_post_send_wait(struct mlx5_ib_dev *dev, u32 mkey, WARN_ON_ONCE(1); mlx5_ib_warn(dev, - "reg umr failed (%u). Trying to recover and resubmit the flushed WQEs\n", - umr_context.status); - mutex_lock(&umrc->lock); - err = mlx5r_umr_recover(dev); - mutex_unlock(&umrc->lock); + "reg umr failed (%u). Trying to recover and resubmit the flushed WQEs, mkey = %u\n", + umr_context.status, mkey); + err = mlx5r_umr_recover(dev, mkey, &umr_context, wqe, with_data); if (err) mlx5_ib_warn(dev, "couldn't recover UMR, err %d\n", err); diff --git a/drivers/infiniband/sw/rdmavt/qp.c b/drivers/infiniband/sw/rdmavt/qp.c index dc83d0ac6a38..d77ada17e9db 100644 --- a/drivers/infiniband/sw/rdmavt/qp.c +++ b/drivers/infiniband/sw/rdmavt/qp.c @@ -92,12 +92,10 @@ static int rvt_wss_llc_size(void) static void cacheless_memcpy(void *dst, void *src, size_t n) { /* - * Use the only available X64 cacheless copy. Add a __user cast - * to quiet sparse. The src agument is already in the kernel so - * there are no security issues. The extra fault recovery machinery - * is not invoked. + * Use the only available X64 cacheless copy. + * The extra fault recovery machinery is not invoked. */ - __copy_user_nocache(dst, (void __user *)src, n); + copy_to_nontemporal(dst, src, n); } void rvt_wss_exit(struct rvt_dev_info *rdi) diff --git a/drivers/infiniband/sw/rdmavt/vt.c b/drivers/infiniband/sw/rdmavt/vt.c index d61f8de7f21c..ddc47691791e 100644 --- a/drivers/infiniband/sw/rdmavt/vt.c +++ b/drivers/infiniband/sw/rdmavt/vt.c @@ -54,8 +54,10 @@ struct rvt_dev_info *rvt_alloc_device(size_t size, int nports) return rdi; rdi->ports = kcalloc(nports, sizeof(*rdi->ports), GFP_KERNEL); - if (!rdi->ports) + if (!rdi->ports) { ib_dealloc_device(&rdi->ibdev); + return NULL; + } return rdi; } diff --git a/drivers/infiniband/sw/rxe/rxe_comp.c b/drivers/infiniband/sw/rxe/rxe_comp.c index 81b645c727a1..be546cb5988a 100644 --- a/drivers/infiniband/sw/rxe/rxe_comp.c +++ b/drivers/infiniband/sw/rxe/rxe_comp.c @@ -124,7 +124,7 @@ void retransmit_timer(struct timer_list *t) spin_lock_irqsave(&qp->state_lock, flags); if (qp->valid) { qp->comp.timeout = 1; - rxe_sched_task(&qp->comp.task); + rxe_sched_task(&qp->send_task); } spin_unlock_irqrestore(&qp->state_lock, flags); rxe_put(qp); @@ -136,14 +136,14 @@ void rxe_comp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb) must_sched = skb_queue_len(&qp->resp_pkts) > 0; if (must_sched != 0) - rxe_counter_inc(SKB_TO_PKT(skb)->rxe, RXE_CNT_COMPLETER_SCHED); + rxe_counter_inc(SKB_TO_PKT(skb)->rxe, RXE_CNT_SENDER_SCHED); skb_queue_tail(&qp->resp_pkts, skb); if (must_sched) - rxe_sched_task(&qp->comp.task); + rxe_sched_task(&qp->send_task); else - rxe_run_task(&qp->comp.task); + rxe_run_task(&qp->send_task); } static inline enum comp_state get_wqe(struct rxe_qp *qp, @@ -328,7 +328,7 @@ static inline enum comp_state check_ack(struct rxe_qp *qp, qp->comp.psn = pkt->psn; if (qp->req.wait_psn) { qp->req.wait_psn = 0; - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->send_task); } } return COMPST_ERROR_RETRY; @@ -479,7 +479,7 @@ static void do_complete(struct rxe_qp *qp, struct rxe_send_wqe *wqe) */ if (qp->req.wait_fence) { qp->req.wait_fence = 0; - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->send_task); } } @@ -518,7 +518,7 @@ static inline enum comp_state complete_ack(struct rxe_qp *qp, if (qp->req.need_rd_atomic) { qp->comp.timeout_retry = 0; qp->req.need_rd_atomic = 0; - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->send_task); } } @@ -544,7 +544,7 @@ static inline enum comp_state complete_wqe(struct rxe_qp *qp, if (qp->req.wait_psn) { qp->req.wait_psn = 0; - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->send_task); } } @@ -740,7 +740,7 @@ int rxe_completer(struct rxe_qp *qp) if (qp->req.wait_psn) { qp->req.wait_psn = 0; - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->send_task); } state = COMPST_DONE; @@ -795,7 +795,7 @@ int rxe_completer(struct rxe_qp *qp) RXE_CNT_COMP_RETRY); qp->req.need_retry = 1; qp->comp.started_retry = 1; - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->send_task); } goto done; diff --git a/drivers/infiniband/sw/rxe/rxe_hw_counters.c b/drivers/infiniband/sw/rxe/rxe_hw_counters.c index a012522b577a..437917a7d8f2 100644 --- a/drivers/infiniband/sw/rxe/rxe_hw_counters.c +++ b/drivers/infiniband/sw/rxe/rxe_hw_counters.c @@ -14,7 +14,7 @@ static const struct rdma_stat_desc rxe_counter_descs[] = { [RXE_CNT_RCV_RNR].name = "rcvd_rnr_err", [RXE_CNT_SND_RNR].name = "send_rnr_err", [RXE_CNT_RCV_SEQ_ERR].name = "rcvd_seq_err", - [RXE_CNT_COMPLETER_SCHED].name = "ack_deferred", + [RXE_CNT_SENDER_SCHED].name = "ack_deferred", [RXE_CNT_RETRY_EXCEEDED].name = "retry_exceeded_err", [RXE_CNT_RNR_RETRY_EXCEEDED].name = "retry_rnr_exceeded_err", [RXE_CNT_COMP_RETRY].name = "completer_retry_err", diff --git a/drivers/infiniband/sw/rxe/rxe_hw_counters.h b/drivers/infiniband/sw/rxe/rxe_hw_counters.h index 71f4d4fa9dc8..051f9e1c3852 100644 --- a/drivers/infiniband/sw/rxe/rxe_hw_counters.h +++ b/drivers/infiniband/sw/rxe/rxe_hw_counters.h @@ -18,7 +18,7 @@ enum rxe_counters { RXE_CNT_RCV_RNR, RXE_CNT_SND_RNR, RXE_CNT_RCV_SEQ_ERR, - RXE_CNT_COMPLETER_SCHED, + RXE_CNT_SENDER_SCHED, RXE_CNT_RETRY_EXCEEDED, RXE_CNT_RNR_RETRY_EXCEEDED, RXE_CNT_COMP_RETRY, diff --git a/drivers/infiniband/sw/rxe/rxe_loc.h b/drivers/infiniband/sw/rxe/rxe_loc.h index 4d2a8ef52c85..86675a527cd6 100644 --- a/drivers/infiniband/sw/rxe/rxe_loc.h +++ b/drivers/infiniband/sw/rxe/rxe_loc.h @@ -164,7 +164,8 @@ void rxe_dealloc(struct ib_device *ib_dev); int rxe_completer(struct rxe_qp *qp); int rxe_requester(struct rxe_qp *qp); -int rxe_responder(struct rxe_qp *qp); +int rxe_sender(struct rxe_qp *qp); +int rxe_receiver(struct rxe_qp *qp); /* rxe_icrc.c */ int rxe_icrc_init(struct rxe_dev *rxe); diff --git a/drivers/infiniband/sw/rxe/rxe_mr.c b/drivers/infiniband/sw/rxe/rxe_mr.c index c955121c455e..e5ab09a6ccf0 100644 --- a/drivers/infiniband/sw/rxe/rxe_mr.c +++ b/drivers/infiniband/sw/rxe/rxe_mr.c @@ -73,14 +73,46 @@ void rxe_mr_init_dma(int access, struct rxe_mr *mr) mr->ibmr.type = IB_MR_TYPE_DMA; } +/* + * Convert iova to page_info index. The page_info stores pages of size + * PAGE_SIZE, but MRs can have different page sizes. This function + * handles the conversion for all cases: + * + * 1. mr->page_size > PAGE_SIZE: + * The MR's iova may not be aligned to mr->page_size. We use the + * aligned base (iova & page_mask) as reference, then calculate + * which PAGE_SIZE sub-page the iova falls into. + * + * 2. mr->page_size <= PAGE_SIZE: + * Use simple shift arithmetic since each page_info entry corresponds + * to one or more MR pages. + */ static unsigned long rxe_mr_iova_to_index(struct rxe_mr *mr, u64 iova) { - return (iova >> mr->page_shift) - (mr->ibmr.iova >> mr->page_shift); + int idx; + + if (mr_page_size(mr) > PAGE_SIZE) + idx = (iova - (mr->ibmr.iova & mr->page_mask)) >> PAGE_SHIFT; + else + idx = (iova >> mr->page_shift) - + (mr->ibmr.iova >> mr->page_shift); + + WARN_ON(idx >= mr->nbuf); + return idx; } +/* + * Convert iova to offset within the page_info entry. + * + * For mr_page_size > PAGE_SIZE, the offset is within the system page. + * For mr_page_size <= PAGE_SIZE, the offset is within the MR page size. + */ static unsigned long rxe_mr_iova_to_page_offset(struct rxe_mr *mr, u64 iova) { - return iova & (mr_page_size(mr) - 1); + if (mr_page_size(mr) > PAGE_SIZE) + return iova & (PAGE_SIZE - 1); + else + return iova & (mr_page_size(mr) - 1); } static bool is_pmem_page(struct page *pg) @@ -94,37 +126,69 @@ static bool is_pmem_page(struct page *pg) static int rxe_mr_fill_pages_from_sgt(struct rxe_mr *mr, struct sg_table *sgt) { - XA_STATE(xas, &mr->page_list, 0); struct sg_page_iter sg_iter; struct page *page; bool persistent = !!(mr->access & IB_ACCESS_FLUSH_PERSISTENT); + WARN_ON(mr_page_size(mr) != PAGE_SIZE); + __sg_page_iter_start(&sg_iter, sgt->sgl, sgt->orig_nents, 0); if (!__sg_page_iter_next(&sg_iter)) return 0; - do { - xas_lock(&xas); - while (true) { - page = sg_page_iter_page(&sg_iter); - - if (persistent && !is_pmem_page(page)) { - rxe_dbg_mr(mr, "Page can't be persistent\n"); - xas_set_err(&xas, -EINVAL); - break; - } + while (true) { + page = sg_page_iter_page(&sg_iter); - xas_store(&xas, page); - if (xas_error(&xas)) - break; - xas_next(&xas); - if (!__sg_page_iter_next(&sg_iter)) - break; + if (persistent && !is_pmem_page(page)) { + rxe_dbg_mr(mr, "Page can't be persistent\n"); + return -EINVAL; } - xas_unlock(&xas); - } while (xas_nomem(&xas, GFP_KERNEL)); - return xas_error(&xas); + mr->page_info[mr->nbuf].page = page; + mr->page_info[mr->nbuf].offset = 0; + mr->nbuf++; + + if (!__sg_page_iter_next(&sg_iter)) + break; + } + + return 0; +} + +static int __alloc_mr_page_info(struct rxe_mr *mr, int num_pages) +{ + mr->page_info = kcalloc(num_pages, sizeof(struct rxe_mr_page), + GFP_KERNEL); + if (!mr->page_info) + return -ENOMEM; + + mr->max_allowed_buf = num_pages; + mr->nbuf = 0; + + return 0; +} + +static int alloc_mr_page_info(struct rxe_mr *mr, int num_pages) +{ + int ret; + + WARN_ON(mr->num_buf); + ret = __alloc_mr_page_info(mr, num_pages); + if (ret) + return ret; + + mr->num_buf = num_pages; + + return 0; +} + +static void free_mr_page_info(struct rxe_mr *mr) +{ + if (!mr->page_info) + return; + + kfree(mr->page_info); + mr->page_info = NULL; } int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, @@ -135,8 +199,6 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, rxe_mr_init(access, mr); - xa_init(&mr->page_list); - umem = ib_umem_get(&rxe->ib_dev, start, length, access); if (IS_ERR(umem)) { rxe_dbg_mr(mr, "Unable to pin memory region err = %d\n", @@ -144,46 +206,24 @@ int rxe_mr_init_user(struct rxe_dev *rxe, u64 start, u64 length, u64 iova, return PTR_ERR(umem); } + err = alloc_mr_page_info(mr, ib_umem_num_pages(umem)); + if (err) + goto err2; + err = rxe_mr_fill_pages_from_sgt(mr, &umem->sgt_append.sgt); - if (err) { - ib_umem_release(umem); - return err; - } + if (err) + goto err1; mr->umem = umem; mr->ibmr.type = IB_MR_TYPE_USER; mr->state = RXE_MR_STATE_VALID; return 0; -} - -static int rxe_mr_alloc(struct rxe_mr *mr, int num_buf) -{ - XA_STATE(xas, &mr->page_list, 0); - int i = 0; - int err; - - xa_init(&mr->page_list); - - do { - xas_lock(&xas); - while (i != num_buf) { - xas_store(&xas, XA_ZERO_ENTRY); - if (xas_error(&xas)) - break; - xas_next(&xas); - i++; - } - xas_unlock(&xas); - } while (xas_nomem(&xas, GFP_KERNEL)); - - err = xas_error(&xas); - if (err) - return err; - - mr->num_buf = num_buf; - - return 0; +err1: + free_mr_page_info(mr); +err2: + ib_umem_release(umem); + return err; } int rxe_mr_init_fast(int max_pages, struct rxe_mr *mr) @@ -193,7 +233,7 @@ int rxe_mr_init_fast(int max_pages, struct rxe_mr *mr) /* always allow remote access for FMRs */ rxe_mr_init(RXE_ACCESS_REMOTE, mr); - err = rxe_mr_alloc(mr, max_pages); + err = alloc_mr_page_info(mr, max_pages); if (err) goto err1; @@ -206,26 +246,43 @@ int rxe_mr_init_fast(int max_pages, struct rxe_mr *mr) return err; } +/* + * I) MRs with page_size >= PAGE_SIZE, + * Split a large MR page (mr->page_size) into multiple PAGE_SIZE + * sub-pages and store them in page_info, offset is always 0. + * + * Called when mr->page_size > PAGE_SIZE. Each call to rxe_set_page() + * represents one mr->page_size region, which we must split into + * (mr->page_size >> PAGE_SHIFT) individual pages. + * + * II) MRs with page_size < PAGE_SIZE, + * Save each PAGE_SIZE page and its offset within the system page in page_info. + */ static int rxe_set_page(struct ib_mr *ibmr, u64 dma_addr) { struct rxe_mr *mr = to_rmr(ibmr); - struct page *page = ib_virt_dma_to_page(dma_addr); bool persistent = !!(mr->access & IB_ACCESS_FLUSH_PERSISTENT); - int err; + u32 i, pages_per_mr = mr_page_size(mr) >> PAGE_SHIFT; - if (persistent && !is_pmem_page(page)) { - rxe_dbg_mr(mr, "Page cannot be persistent\n"); - return -EINVAL; - } + pages_per_mr = MAX(1, pages_per_mr); - if (unlikely(mr->nbuf == mr->num_buf)) - return -ENOMEM; + for (i = 0; i < pages_per_mr; i++) { + u64 addr = dma_addr + i * PAGE_SIZE; + struct page *sub_page = ib_virt_dma_to_page(addr); - err = xa_err(xa_store(&mr->page_list, mr->nbuf, page, GFP_KERNEL)); - if (err) - return err; + if (unlikely(mr->nbuf >= mr->max_allowed_buf)) + return -ENOMEM; + + if (persistent && !is_pmem_page(sub_page)) { + rxe_dbg_mr(mr, "Page cannot be persistent\n"); + return -EINVAL; + } + + mr->page_info[mr->nbuf].page = sub_page; + mr->page_info[mr->nbuf].offset = addr & (PAGE_SIZE - 1); + mr->nbuf++; + } - mr->nbuf++; return 0; } @@ -235,6 +292,31 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, struct rxe_mr *mr = to_rmr(ibmr); unsigned int page_size = mr_page_size(mr); + /* + * Ensure page_size and PAGE_SIZE are compatible for mapping. + * We require one to be a multiple of the other for correct + * iova-to-page conversion. + */ + if (!IS_ALIGNED(page_size, PAGE_SIZE) && + !IS_ALIGNED(PAGE_SIZE, page_size)) { + rxe_dbg_mr(mr, "MR page size %u must be compatible with PAGE_SIZE %lu\n", + page_size, PAGE_SIZE); + return -EINVAL; + } + + if (mr_page_size(mr) > PAGE_SIZE) { + /* resize page_info if needed */ + u32 map_mr_pages = (page_size >> PAGE_SHIFT) * mr->num_buf; + + if (map_mr_pages > mr->max_allowed_buf) { + rxe_dbg_mr(mr, "requested pages %u exceed max %u\n", + map_mr_pages, mr->max_allowed_buf); + free_mr_page_info(mr); + if (__alloc_mr_page_info(mr, map_mr_pages)) + return -ENOMEM; + } + } + mr->nbuf = 0; mr->page_shift = ilog2(page_size); mr->page_mask = ~((u64)page_size - 1); @@ -246,30 +328,30 @@ int rxe_map_mr_sg(struct ib_mr *ibmr, struct scatterlist *sgl, static int rxe_mr_copy_xarray(struct rxe_mr *mr, u64 iova, void *addr, unsigned int length, enum rxe_mr_copy_dir dir) { - unsigned int page_offset = rxe_mr_iova_to_page_offset(mr, iova); - unsigned long index = rxe_mr_iova_to_index(mr, iova); unsigned int bytes; - struct page *page; - void *va; + u8 *va; while (length) { - page = xa_load(&mr->page_list, index); - if (!page) + unsigned long index = rxe_mr_iova_to_index(mr, iova); + struct rxe_mr_page *info = &mr->page_info[index]; + unsigned int page_offset = rxe_mr_iova_to_page_offset(mr, iova); + + if (!info->page) return -EFAULT; - bytes = min_t(unsigned int, length, - mr_page_size(mr) - page_offset); - va = kmap_local_page(page); + page_offset += info->offset; + bytes = min_t(unsigned int, length, PAGE_SIZE - page_offset); + va = kmap_local_page(info->page); + if (dir == RXE_FROM_MR_OBJ) memcpy(addr, va + page_offset, bytes); else memcpy(va + page_offset, addr, bytes); kunmap_local(va); - page_offset = 0; addr += bytes; + iova += bytes; length -= bytes; - index++; } return 0; @@ -424,9 +506,6 @@ int copy_data( int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) { - unsigned int page_offset; - unsigned long index; - struct page *page; unsigned int bytes; int err; u8 *va; @@ -446,15 +525,17 @@ int rxe_flush_pmem_iova(struct rxe_mr *mr, u64 iova, unsigned int length) return err; while (length > 0) { - index = rxe_mr_iova_to_index(mr, iova); - page = xa_load(&mr->page_list, index); - page_offset = rxe_mr_iova_to_page_offset(mr, iova); - if (!page) + unsigned long index = rxe_mr_iova_to_index(mr, iova); + struct rxe_mr_page *info = &mr->page_info[index]; + unsigned int page_offset = rxe_mr_iova_to_page_offset(mr, iova); + + if (!info->page) return -EFAULT; - bytes = min_t(unsigned int, length, - mr_page_size(mr) - page_offset); - va = kmap_local_page(page); + page_offset += info->offset; + bytes = min_t(unsigned int, length, PAGE_SIZE - page_offset); + + va = kmap_local_page(info->page); arch_wb_cache_pmem(va + page_offset, bytes); kunmap_local(va); @@ -488,6 +569,7 @@ int rxe_mr_do_atomic_op(struct rxe_mr *mr, u64 iova, int opcode, } else { unsigned long index; int err; + struct rxe_mr_page *info; err = mr_check_range(mr, iova, sizeof(value)); if (err) { @@ -496,9 +578,12 @@ int rxe_mr_do_atomic_op(struct rxe_mr *mr, u64 iova, int opcode, } page_offset = rxe_mr_iova_to_page_offset(mr, iova); index = rxe_mr_iova_to_index(mr, iova); - page = xa_load(&mr->page_list, index); - if (!page) + info = &mr->page_info[index]; + if (!info->page) return RESPST_ERR_RKEY_VIOLATION; + + page_offset += info->offset; + page = info->page; } if (unlikely(page_offset & 0x7)) { @@ -545,6 +630,7 @@ int rxe_mr_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) } else { unsigned long index; int err; + struct rxe_mr_page *info; /* See IBA oA19-28 */ err = mr_check_range(mr, iova, sizeof(value)); @@ -554,9 +640,12 @@ int rxe_mr_do_atomic_write(struct rxe_mr *mr, u64 iova, u64 value) } page_offset = rxe_mr_iova_to_page_offset(mr, iova); index = rxe_mr_iova_to_index(mr, iova); - page = xa_load(&mr->page_list, index); - if (!page) + info = &mr->page_info[index]; + if (!info->page) return RESPST_ERR_RKEY_VIOLATION; + + page_offset += info->offset; + page = info->page; } /* See IBA A19.4.2 */ @@ -728,5 +817,5 @@ void rxe_mr_cleanup(struct rxe_pool_elem *elem) ib_umem_release(mr->umem); if (mr->ibmr.type != IB_MR_TYPE_DMA) - xa_destroy(&mr->page_list); + free_mr_page_info(mr); } diff --git a/drivers/infiniband/sw/rxe/rxe_net.c b/drivers/infiniband/sw/rxe/rxe_net.c index e2859bbc9a71..857988eed4ed 100644 --- a/drivers/infiniband/sw/rxe/rxe_net.c +++ b/drivers/infiniband/sw/rxe/rxe_net.c @@ -351,7 +351,7 @@ static void rxe_skb_tx_dtor(struct sk_buff *skb) if (unlikely(qp->need_req_skb && skb_out < RXE_INFLIGHT_SKBS_PER_QP_LOW)) - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->send_task); rxe_put(qp); } @@ -435,7 +435,7 @@ int rxe_xmit_packet(struct rxe_qp *qp, struct rxe_pkt_info *pkt, if ((qp_type(qp) != IB_QPT_RC) && (pkt->mask & RXE_END_MASK)) { pkt->wqe->state = wqe_state_done; - rxe_sched_task(&qp->comp.task); + rxe_sched_task(&qp->send_task); } rxe_counter_inc(rxe, RXE_CNT_SENT_PKTS); diff --git a/drivers/infiniband/sw/rxe/rxe_qp.c b/drivers/infiniband/sw/rxe/rxe_qp.c index 1d6c8eb1f97d..c3e1ada9b6a1 100644 --- a/drivers/infiniband/sw/rxe/rxe_qp.c +++ b/drivers/infiniband/sw/rxe/rxe_qp.c @@ -124,6 +124,7 @@ static void free_rd_atomic_resources(struct rxe_qp *qp) } kfree(qp->resp.resources); qp->resp.resources = NULL; + qp->resp.res = NULL; } } @@ -265,8 +266,7 @@ static int rxe_qp_init_req(struct rxe_dev *rxe, struct rxe_qp *qp, qp->req.opcode = -1; qp->comp.opcode = -1; - rxe_init_task(&qp->req.task, qp, rxe_requester); - rxe_init_task(&qp->comp.task, qp, rxe_completer); + rxe_init_task(&qp->send_task, qp, rxe_sender); qp->qp_timeout_jiffies = 0; /* Can't be set for UD/UC in modify_qp */ if (init->qp_type == IB_QPT_RC) { @@ -337,7 +337,7 @@ static int rxe_qp_init_resp(struct rxe_dev *rxe, struct rxe_qp *qp, return err; } - rxe_init_task(&qp->resp.task, qp, rxe_responder); + rxe_init_task(&qp->recv_task, qp, rxe_receiver); qp->resp.opcode = OPCODE_NONE; qp->resp.msn = 0; @@ -514,14 +514,12 @@ int rxe_qp_chk_attr(struct rxe_dev *rxe, struct rxe_qp *qp, static void rxe_qp_reset(struct rxe_qp *qp) { /* stop tasks from running */ - rxe_disable_task(&qp->resp.task); - rxe_disable_task(&qp->comp.task); - rxe_disable_task(&qp->req.task); + rxe_disable_task(&qp->recv_task); + rxe_disable_task(&qp->send_task); /* drain work and packet queuesc */ - rxe_requester(qp); - rxe_completer(qp); - rxe_responder(qp); + rxe_sender(qp); + rxe_receiver(qp); if (qp->rq.queue) rxe_queue_reset(qp->rq.queue); @@ -548,9 +546,8 @@ static void rxe_qp_reset(struct rxe_qp *qp) cleanup_rd_atomic_resources(qp); /* reenable tasks */ - rxe_enable_task(&qp->resp.task); - rxe_enable_task(&qp->comp.task); - rxe_enable_task(&qp->req.task); + rxe_enable_task(&qp->recv_task); + rxe_enable_task(&qp->send_task); } /* move the qp to the error state */ @@ -562,9 +559,8 @@ void rxe_qp_error(struct rxe_qp *qp) qp->attr.qp_state = IB_QPS_ERR; /* drain work and packet queues */ - rxe_sched_task(&qp->resp.task); - rxe_sched_task(&qp->comp.task); - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->recv_task); + rxe_sched_task(&qp->send_task); spin_unlock_irqrestore(&qp->state_lock, flags); } @@ -575,8 +571,7 @@ static void rxe_qp_sqd(struct rxe_qp *qp, struct ib_qp_attr *attr, spin_lock_irqsave(&qp->state_lock, flags); qp->attr.sq_draining = 1; - rxe_sched_task(&qp->comp.task); - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->send_task); spin_unlock_irqrestore(&qp->state_lock, flags); } @@ -664,12 +659,25 @@ int rxe_qp_from_attr(struct rxe_qp *qp, struct ib_qp_attr *attr, int mask, int max_dest_rd_atomic = attr->max_dest_rd_atomic ? roundup_pow_of_two(attr->max_dest_rd_atomic) : 0; + /* + * Not gated by IB_QP_STATE, so the responder task is live. + * Quiesce recv_task like rxe_qp_reset() before swapping the + * rd_atomic array, so rxe_receiver() cannot race the free/ + * realloc. + */ + rxe_disable_task(&qp->recv_task); free_rd_atomic_resources(qp); qp->attr.max_dest_rd_atomic = max_dest_rd_atomic; err = alloc_rd_atomic_resources(qp, max_dest_rd_atomic); + /* + * On ENOMEM leave recv_task quiesced: qp->resp.resources is + * NULL and rxe_prepare_res()/find_resource() would deref it. + * Re-enable only after a fresh array is installed. + */ if (err) return err; + rxe_enable_task(&qp->recv_task); } if (mask & IB_QP_EN_SQD_ASYNC_NOTIFY) @@ -826,19 +834,15 @@ static void rxe_qp_do_cleanup(struct work_struct *work) del_timer_sync(&qp->rnr_nak_timer); } - if (qp->resp.task.func) - rxe_cleanup_task(&qp->resp.task); + if (qp->recv_task.func) + rxe_cleanup_task(&qp->recv_task); - if (qp->req.task.func) - rxe_cleanup_task(&qp->req.task); - - if (qp->comp.task.func) - rxe_cleanup_task(&qp->comp.task); + if (qp->send_task.func) + rxe_cleanup_task(&qp->send_task); /* flush out any receive wr's or pending requests */ - rxe_requester(qp); - rxe_completer(qp); - rxe_responder(qp); + rxe_sender(qp); + rxe_receiver(qp); if (qp->sq.queue) rxe_queue_cleanup(qp->sq.queue); diff --git a/drivers/infiniband/sw/rxe/rxe_req.c b/drivers/infiniband/sw/rxe/rxe_req.c index bccf1f28467a..883ad2222bab 100644 --- a/drivers/infiniband/sw/rxe/rxe_req.c +++ b/drivers/infiniband/sw/rxe/rxe_req.c @@ -110,7 +110,7 @@ void rnr_nak_timer(struct timer_list *t) /* request a send queue retry */ qp->req.need_retry = 1; qp->req.wait_for_rnr_timer = 0; - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->send_task); } spin_unlock_irqrestore(&qp->state_lock, flags); rxe_put(qp); @@ -663,7 +663,7 @@ static int rxe_do_local_ops(struct rxe_qp *qp, struct rxe_send_wqe *wqe) * which can lead to a deadlock. So go ahead and complete * it now. */ - rxe_sched_task(&qp->comp.task); + rxe_sched_task(&qp->send_task); return 0; } @@ -807,7 +807,7 @@ int rxe_requester(struct rxe_qp *qp) qp->req.wqe_index); wqe->state = wqe_state_done; wqe->status = IB_WC_SUCCESS; - rxe_sched_task(&qp->comp.task); + rxe_sched_task(&qp->send_task); goto done; } payload = mtu; @@ -876,7 +876,7 @@ int rxe_requester(struct rxe_qp *qp) */ qp->need_req_skb = 1; - rxe_sched_task(&qp->req.task); + rxe_sched_task(&qp->send_task); goto exit; } @@ -899,3 +899,20 @@ int rxe_requester(struct rxe_qp *qp) out: return ret; } + +int rxe_sender(struct rxe_qp *qp) +{ + int req_ret; + int comp_ret; + + /* process the send queue */ + req_ret = rxe_requester(qp); + + /* process the response queue */ + comp_ret = rxe_completer(qp); + + /* exit the task loop if both requester and completer + * are ready + */ + return (req_ret && comp_ret) ? -EAGAIN : 0; +} diff --git a/drivers/infiniband/sw/rxe/rxe_resp.c b/drivers/infiniband/sw/rxe/rxe_resp.c index e03082db4243..c5ca98456262 100644 --- a/drivers/infiniband/sw/rxe/rxe_resp.c +++ b/drivers/infiniband/sw/rxe/rxe_resp.c @@ -58,9 +58,9 @@ void rxe_resp_queue_pkt(struct rxe_qp *qp, struct sk_buff *skb) (skb_queue_len(&qp->req_pkts) > 1); if (must_sched) - rxe_sched_task(&qp->resp.task); + rxe_sched_task(&qp->recv_task); else - rxe_run_task(&qp->resp.task); + rxe_run_task(&qp->recv_task); } static inline enum resp_states get_req(struct rxe_qp *qp, @@ -1540,7 +1540,7 @@ static void flush_recv_queue(struct rxe_qp *qp, bool notify) qp->resp.wqe = NULL; } -int rxe_responder(struct rxe_qp *qp) +int rxe_receiver(struct rxe_qp *qp) { struct rxe_dev *rxe = to_rdev(qp->ibqp.device); enum resp_states state; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.c b/drivers/infiniband/sw/rxe/rxe_verbs.c index a7bfa9532d69..3539817b777a 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.c +++ b/drivers/infiniband/sw/rxe/rxe_verbs.c @@ -913,12 +913,7 @@ static int rxe_post_send_kernel(struct rxe_qp *qp, /* kickoff processing of any posted wqes */ if (good) - rxe_sched_task(&qp->req.task); - - spin_lock_irqsave(&qp->state_lock, flags); - if (qp_state(qp) == IB_QPS_ERR) - rxe_sched_task(&qp->comp.task); - spin_unlock_irqrestore(&qp->state_lock, flags); + rxe_sched_task(&qp->send_task); return err; } @@ -948,7 +943,7 @@ static int rxe_post_send(struct ib_qp *ibqp, const struct ib_send_wr *wr, if (qp->is_user) { /* Utilize process context to do protocol processing */ - rxe_run_task(&qp->req.task); + rxe_run_task(&qp->send_task); } else { err = rxe_post_send_kernel(qp, wr, bad_wr); if (err) @@ -1057,7 +1052,7 @@ static int rxe_post_recv(struct ib_qp *ibqp, const struct ib_recv_wr *wr, spin_lock_irqsave(&qp->state_lock, flags); if (qp_state(qp) == IB_QPS_ERR) - rxe_sched_task(&qp->resp.task); + rxe_sched_task(&qp->recv_task); spin_unlock_irqrestore(&qp->state_lock, flags); return err; diff --git a/drivers/infiniband/sw/rxe/rxe_verbs.h b/drivers/infiniband/sw/rxe/rxe_verbs.h index 4242b24a47da..3f636ba992c1 100644 --- a/drivers/infiniband/sw/rxe/rxe_verbs.h +++ b/drivers/infiniband/sw/rxe/rxe_verbs.h @@ -113,7 +113,6 @@ struct rxe_req_info { int need_retry; int wait_for_rnr_timer; int noack_pkts; - struct rxe_task task; }; struct rxe_comp_info { @@ -124,7 +123,6 @@ struct rxe_comp_info { int started_retry; u32 retry_cnt; u32 rnr_retry; - struct rxe_task task; }; enum rdatm_res_state { @@ -196,7 +194,6 @@ struct rxe_resp_info { unsigned int res_head; unsigned int res_tail; struct resp_res *res; - struct rxe_task task; }; struct rxe_qp { @@ -229,6 +226,9 @@ struct rxe_qp { struct sk_buff_head req_pkts; struct sk_buff_head resp_pkts; + struct rxe_task send_task; + struct rxe_task recv_task; + struct rxe_req_info req; struct rxe_comp_info comp; struct rxe_resp_info resp; @@ -297,6 +297,11 @@ static inline int rkey_is_mw(u32 rkey) return (index >= RXE_MIN_MW_INDEX) && (index <= RXE_MAX_MW_INDEX); } +struct rxe_mr_page { + struct page *page; + unsigned int offset; /* offset in system page */ +}; + struct rxe_mr { struct rxe_pool_elem elem; struct ib_mr ibmr; @@ -313,10 +318,13 @@ struct rxe_mr { unsigned int page_shift; u64 page_mask; + /* size of page_info when mr allocated */ u32 num_buf; + /* real size of page_info */ + u32 max_allowed_buf; u32 nbuf; - struct xarray page_list; + struct rxe_mr_page *page_info; }; static inline unsigned int mr_page_size(struct rxe_mr *mr) diff --git a/drivers/infiniband/sw/siw/siw.h b/drivers/infiniband/sw/siw/siw.h index 58dddb143b9f..b79dad870423 100644 --- a/drivers/infiniband/sw/siw/siw.h +++ b/drivers/infiniband/sw/siw/siw.h @@ -46,6 +46,9 @@ */ #define SIW_IRQ_MAXBURST_SQ_ACTIVE 4 +/* There is always only a port 1 per siw device */ +#define SIW_PORT 1 + struct siw_dev_cap { int max_qp; int max_qp_wr; @@ -69,16 +72,12 @@ struct siw_pd { struct siw_device { struct ib_device base_dev; - struct net_device *netdev; struct siw_dev_cap attrs; u32 vendor_part_id; int numa_node; char raw_gid[ETH_ALEN]; - /* physical port state (only one port per device) */ - enum ib_port_state state; - spinlock_t lock; struct xarray qp_xa; diff --git a/drivers/infiniband/sw/siw/siw_cm.c b/drivers/infiniband/sw/siw/siw_cm.c index 3551824e371a..744c03f352f6 100644 --- a/drivers/infiniband/sw/siw/siw_cm.c +++ b/drivers/infiniband/sw/siw/siw_cm.c @@ -94,6 +94,7 @@ static void siw_socket_disassoc(struct socket *s) cep = sk_to_cep(sk); if (cep) { siw_sk_restore_upcalls(sk, cep); + cep->sock = NULL; siw_cep_put(cep); } else { pr_warn("siw: cannot restore sk callbacks: no ep\n"); @@ -373,6 +374,16 @@ static void siw_free_cm_id(struct siw_cep *cep) cep->cm_id = NULL; } +static void siw_destroy_cep_sock(struct siw_cep *cep) +{ + struct socket *s = cep->sock; + + if (s) { + siw_socket_disassoc(s); + sock_release(s); + } +} + /* * siw_qp_cm_drop() * @@ -430,14 +441,7 @@ void siw_qp_cm_drop(struct siw_qp *qp, int schedule) } cep->state = SIW_EPSTATE_CLOSED; - if (cep->sock) { - siw_socket_disassoc(cep->sock); - /* - * Immediately close socket - */ - sock_release(cep->sock); - cep->sock = NULL; - } + siw_destroy_cep_sock(cep); if (cep->qp) { cep->qp = NULL; siw_qp_put(qp); @@ -1006,7 +1010,6 @@ static void siw_accept_newconn(struct siw_cep *cep) if (new_s) { siw_socket_disassoc(new_s); sock_release(new_s); - new_cep->sock = NULL; } siw_dbg_cep(cep, "error %d\n", rv); } @@ -1158,6 +1161,8 @@ static void siw_cm_work_handler(struct work_struct *w) WARN(1, "Undefined CM work type: %d\n", work->type); } if (release_cep) { + struct socket *s = cep->sock; + siw_dbg_cep(cep, "release: timer=%s, QP[%u]\n", cep->mpa_timer ? "y" : "n", @@ -1183,10 +1188,9 @@ static void siw_cm_work_handler(struct work_struct *w) cep->qp = NULL; siw_qp_put(qp); } - if (cep->sock) { - siw_socket_disassoc(cep->sock); - sock_release(cep->sock); - cep->sock = NULL; + if (s) { + siw_socket_disassoc(s); + sock_release(s); } if (cep->cm_id) { siw_free_cm_id(cep); @@ -1516,7 +1520,6 @@ int siw_connect(struct iw_cm_id *id, struct iw_cm_conn_param *params) if (cep) { siw_socket_disassoc(s); sock_release(s); - cep->sock = NULL; cep->qp = NULL; @@ -1703,9 +1706,7 @@ int siw_accept(struct iw_cm_id *id, struct iw_cm_conn_param *params) error_unlock: up_write(&qp->state_lock); error: - siw_socket_disassoc(cep->sock); - sock_release(cep->sock); - cep->sock = NULL; + siw_destroy_cep_sock(cep); cep->state = SIW_EPSTATE_CLOSED; @@ -1750,9 +1751,7 @@ int siw_reject(struct iw_cm_id *id, const void *pdata, u8 pd_len) cep->mpa.hdr.params.bits |= MPA_RR_FLAG_REJECT; /* reject */ siw_send_mpareqrep(cep, pdata, pd_len); } - siw_socket_disassoc(cep->sock); - sock_release(cep->sock); - cep->sock = NULL; + siw_destroy_cep_sock(cep); cep->state = SIW_EPSTATE_CLOSED; @@ -1771,6 +1770,7 @@ int siw_create_listen(struct iw_cm_id *id, int backlog) { struct socket *s; struct siw_cep *cep = NULL; + struct net_device *ndev = NULL; struct siw_device *sdev = to_siw_dev(id->device); int addr_family = id->local_addr.ss_family; int rv = 0; @@ -1791,9 +1791,15 @@ int siw_create_listen(struct iw_cm_id *id, int backlog) struct sockaddr_in *laddr = &to_sockaddr_in(id->local_addr); /* For wildcard addr, limit binding to current device only */ - if (ipv4_is_zeronet(laddr->sin_addr.s_addr)) - s->sk->sk_bound_dev_if = sdev->netdev->ifindex; - + if (ipv4_is_zeronet(laddr->sin_addr.s_addr)) { + ndev = ib_device_get_netdev(id->device, SIW_PORT); + if (ndev) { + s->sk->sk_bound_dev_if = ndev->ifindex; + } else { + rv = -ENODEV; + goto error; + } + } rv = s->ops->bind(s, (struct sockaddr *)laddr, sizeof(struct sockaddr_in)); } else { @@ -1809,9 +1815,15 @@ int siw_create_listen(struct iw_cm_id *id, int backlog) } /* For wildcard addr, limit binding to current device only */ - if (ipv6_addr_any(&laddr->sin6_addr)) - s->sk->sk_bound_dev_if = sdev->netdev->ifindex; - + if (ipv6_addr_any(&laddr->sin6_addr)) { + ndev = ib_device_get_netdev(id->device, SIW_PORT); + if (ndev) { + s->sk->sk_bound_dev_if = ndev->ifindex; + } else { + rv = -ENODEV; + goto error; + } + } rv = s->ops->bind(s, (struct sockaddr *)laddr, sizeof(struct sockaddr_in6)); } @@ -1872,6 +1884,7 @@ int siw_create_listen(struct iw_cm_id *id, int backlog) } list_add_tail(&cep->listenq, (struct list_head *)id->provider_data); cep->state = SIW_EPSTATE_LISTENING; + dev_put(ndev); siw_dbg(id->device, "Listen at laddr %pISp\n", &id->local_addr); @@ -1884,13 +1897,13 @@ int siw_create_listen(struct iw_cm_id *id, int backlog) siw_cep_set_inuse(cep); siw_free_cm_id(cep); - cep->sock = NULL; siw_socket_disassoc(s); cep->state = SIW_EPSTATE_CLOSED; siw_cep_set_free_and_put(cep); } sock_release(s); + dev_put(ndev); return rv; } @@ -1905,6 +1918,7 @@ static void siw_drop_listeners(struct iw_cm_id *id) */ list_for_each_safe(p, tmp, (struct list_head *)id->provider_data) { struct siw_cep *cep = list_entry(p, struct siw_cep, listenq); + struct socket *s = cep->sock; list_del(p); @@ -1913,10 +1927,9 @@ static void siw_drop_listeners(struct iw_cm_id *id) siw_cep_set_inuse(cep); siw_free_cm_id(cep); - if (cep->sock) { - siw_socket_disassoc(cep->sock); - sock_release(cep->sock); - cep->sock = NULL; + if (s) { + siw_socket_disassoc(s); + sock_release(s); } cep->state = SIW_EPSTATE_CLOSED; siw_cep_set_free_and_put(cep); diff --git a/drivers/infiniband/sw/siw/siw_main.c b/drivers/infiniband/sw/siw/siw_main.c index d4b6e0106851..c82f0af046af 100644 --- a/drivers/infiniband/sw/siw/siw_main.c +++ b/drivers/infiniband/sw/siw/siw_main.c @@ -289,7 +289,6 @@ static struct siw_device *siw_device_create(struct net_device *netdev) return NULL; base_dev = &sdev->base_dev; - sdev->netdev = netdev; if (netdev->addr_len) { memcpy(sdev->raw_gid, netdev->dev_addr, @@ -416,7 +415,6 @@ static int siw_netdev_event(struct notifier_block *nb, unsigned long event, switch (event) { case NETDEV_UP: - sdev->state = IB_PORT_ACTIVE; siw_port_event(sdev, 1, IB_EVENT_PORT_ACTIVE); break; @@ -425,7 +423,6 @@ static int siw_netdev_event(struct notifier_block *nb, unsigned long event, break; case NETDEV_DOWN: - sdev->state = IB_PORT_DOWN; siw_port_event(sdev, 1, IB_EVENT_PORT_ERR); break; @@ -446,12 +443,8 @@ static int siw_netdev_event(struct notifier_block *nb, unsigned long event, siw_port_event(sdev, 1, IB_EVENT_LID_CHANGE); break; /* - * Todo: Below netdev events are currently not handled. + * All other events are not handled */ - case NETDEV_CHANGEMTU: - case NETDEV_CHANGE: - break; - default: break; } @@ -481,12 +474,6 @@ static int siw_newlink(const char *basedev_name, struct net_device *netdev) sdev = siw_device_create(netdev); if (sdev) { dev_dbg(&netdev->dev, "siw: new device\n"); - - if (netif_running(netdev) && netif_carrier_ok(netdev)) - sdev->state = IB_PORT_ACTIVE; - else - sdev->state = IB_PORT_DOWN; - rv = siw_device_register(sdev, basedev_name); if (rv) ib_dealloc_device(&sdev->base_dev); diff --git a/drivers/infiniband/sw/siw/siw_verbs.c b/drivers/infiniband/sw/siw/siw_verbs.c index 40e0e0a2486d..236b42fc9761 100644 --- a/drivers/infiniband/sw/siw/siw_verbs.c +++ b/drivers/infiniband/sw/siw/siw_verbs.c @@ -165,21 +165,29 @@ int siw_query_device(struct ib_device *base_dev, struct ib_device_attr *attr, int siw_query_port(struct ib_device *base_dev, u32 port, struct ib_port_attr *attr) { - struct siw_device *sdev = to_siw_dev(base_dev); + struct net_device *ndev; int rv; memset(attr, 0, sizeof(*attr)); rv = ib_get_eth_speed(base_dev, port, &attr->active_speed, &attr->active_width); + if (rv) + return rv; + + ndev = ib_device_get_netdev(base_dev, SIW_PORT); + if (!ndev) + return -ENODEV; + attr->gid_tbl_len = 1; attr->max_msg_sz = -1; - attr->max_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu); - attr->active_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu); - attr->phys_state = sdev->state == IB_PORT_ACTIVE ? + attr->max_mtu = ib_mtu_int_to_enum(ndev->max_mtu); + attr->active_mtu = ib_mtu_int_to_enum(READ_ONCE(ndev->mtu)); + attr->phys_state = (netif_running(ndev) && netif_carrier_ok(ndev)) ? IB_PORT_PHYS_STATE_LINK_UP : IB_PORT_PHYS_STATE_DISABLED; + attr->state = attr->phys_state == IB_PORT_PHYS_STATE_LINK_UP ? + IB_PORT_ACTIVE : IB_PORT_DOWN; attr->port_cap_flags = IB_PORT_CM_SUP | IB_PORT_DEVICE_MGMT_SUP; - attr->state = sdev->state; /* * All zero * @@ -193,6 +201,7 @@ int siw_query_port(struct ib_device *base_dev, u32 port, * attr->subnet_timeout = 0; * attr->init_type_repy = 0; */ + dev_put(ndev); return rv; } @@ -503,20 +512,23 @@ int siw_query_qp(struct ib_qp *base_qp, struct ib_qp_attr *qp_attr, int qp_attr_mask, struct ib_qp_init_attr *qp_init_attr) { struct siw_qp *qp; - struct siw_device *sdev; + struct net_device *ndev; - if (base_qp && qp_attr && qp_init_attr) { + if (base_qp && qp_attr && qp_init_attr) qp = to_siw_qp(base_qp); - sdev = to_siw_dev(base_qp->device); - } else { + else return -EINVAL; - } + + ndev = ib_device_get_netdev(base_qp->device, SIW_PORT); + if (!ndev) + return -ENODEV; + qp_attr->cap.max_inline_data = SIW_MAX_INLINE; qp_attr->cap.max_send_wr = qp->attrs.sq_size; qp_attr->cap.max_send_sge = qp->attrs.sq_max_sges; qp_attr->cap.max_recv_wr = qp->attrs.rq_size; qp_attr->cap.max_recv_sge = qp->attrs.rq_max_sges; - qp_attr->path_mtu = ib_mtu_int_to_enum(sdev->netdev->mtu); + qp_attr->path_mtu = ib_mtu_int_to_enum(READ_ONCE(ndev->mtu)); qp_attr->max_rd_atomic = qp->attrs.irq_size; qp_attr->max_dest_rd_atomic = qp->attrs.orq_size; @@ -531,6 +543,7 @@ int siw_query_qp(struct ib_qp *base_qp, struct ib_qp_attr *qp_attr, qp_init_attr->cap = qp_attr->cap; + dev_put(ndev); return 0; } diff --git a/drivers/input/joystick/xpad.c b/drivers/input/joystick/xpad.c index 74913aeea62f..04cc323e3014 100644 --- a/drivers/input/joystick/xpad.c +++ b/drivers/input/joystick/xpad.c @@ -330,6 +330,8 @@ static const struct xpad_device { { 0x17ef, 0x6182, "Lenovo Legion Controller for Windows", 0, XTYPE_XBOX360 }, { 0x1949, 0x041a, "Amazon Game Controller", 0, XTYPE_XBOX360 }, { 0x1a86, 0xe310, "Legion Go S", 0, XTYPE_XBOX360 }, + { 0x1b1c, 0x2b1f, "Corsair Novablade Pro (Wired)", 0, XTYPE_XBOX360 }, + { 0x1b1c, 0x2b2b, "Corsair Novablade Pro (2.4 GHz Receiver)", 0, XTYPE_XBOX360 }, { 0x1bad, 0x0002, "Harmonix Rock Band Guitar", 0, XTYPE_XBOX360 }, { 0x1bad, 0x0003, "Harmonix Rock Band Drumkit", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360 }, { 0x1bad, 0x0130, "Ion Drum Rocker", MAP_DPAD_TO_BUTTONS, XTYPE_XBOX360 }, @@ -564,6 +566,7 @@ static const struct usb_device_id xpad_table[] = { XPAD_XBOX360_VENDOR(0x17ef), /* Lenovo */ XPAD_XBOX360_VENDOR(0x1949), /* Amazon controllers */ XPAD_XBOX360_VENDOR(0x1a86), /* Nanjing Qinheng Microelectronics (WCH) */ + XPAD_XBOX360_VENDOR(0x1b1c), /* Corsair controllers */ XPAD_XBOX360_VENDOR(0x1bad), /* Harmonix Rock Band guitar and drums */ XPAD_XBOX360_VENDOR(0x1ee9), /* ZOTAC Technology Limited */ XPAD_XBOX360_VENDOR(0x20bc), /* BETOP wireless dongles */ diff --git a/drivers/input/keyboard/atkbd.c b/drivers/input/keyboard/atkbd.c index 9d40a85ff131..811d69338abe 100644 --- a/drivers/input/keyboard/atkbd.c +++ b/drivers/input/keyboard/atkbd.c @@ -1974,6 +1974,14 @@ static const struct dmi_system_id atkbd_dmi_quirk_table[] __initconst = { }, .callback = atkbd_deactivate_fixup, }, + { + /* Lenovo IdeaPad Slim 3 15IWC11 (83RR) */ + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "LENOVO"), + DMI_MATCH(DMI_PRODUCT_NAME, "83RR"), + }, + .callback = atkbd_deactivate_fixup, + }, { /* Xiaomi Book Pro 14 (TM2424) */ .matches = { diff --git a/drivers/input/keyboard/lkkbd.c b/drivers/input/keyboard/lkkbd.c index 047b654b3752..7b57577a1715 100644 --- a/drivers/input/keyboard/lkkbd.c +++ b/drivers/input/keyboard/lkkbd.c @@ -671,6 +671,7 @@ static int lkkbd_connect(struct serio *serio, struct serio_driver *drv) return 0; fail3: serio_close(serio); + cancel_work_sync(&lk->tq); fail2: serio_set_drvdata(serio, NULL); fail1: input_free_device(input_dev); kfree(lk); @@ -687,6 +688,7 @@ static void lkkbd_disconnect(struct serio *serio) input_get_device(lk->dev); input_unregister_device(lk->dev); serio_close(serio); + cancel_work_sync(&lk->tq); serio_set_drvdata(serio, NULL); input_put_device(lk->dev); kfree(lk); diff --git a/drivers/input/mouse/alps.c b/drivers/input/mouse/alps.c index e2c11d9f3868..e2a3d08791ee 100644 --- a/drivers/input/mouse/alps.c +++ b/drivers/input/mouse/alps.c @@ -2971,6 +2971,8 @@ static void alps_disconnect(struct psmouse *psmouse) psmouse_reset(psmouse); timer_shutdown_sync(&priv->timer); + psmouse_set_state(psmouse, PSMOUSE_IGNORE); + cancel_delayed_work_sync(&priv->dev3_register_work); if (priv->dev2) input_unregister_device(priv->dev2); if (!IS_ERR_OR_NULL(priv->dev3)) diff --git a/drivers/input/mouse/synaptics.c b/drivers/input/mouse/synaptics.c index 4bd797006027..df4431260aad 100644 --- a/drivers/input/mouse/synaptics.c +++ b/drivers/input/mouse/synaptics.c @@ -190,6 +190,7 @@ static const char * const smbus_pnp_ids[] = { "LEN2054", /* E480 */ "LEN2055", /* E580 */ "LEN2058", /* E490 */ + "LEN205b", /* T490 */ "LEN2068", /* T14 Gen 1 */ "SYN1221", /* TUXEDO InfinityBook Pro 14 v5 */ "SYN3003", /* HP EliteBook 850 G1 */ @@ -1838,8 +1839,10 @@ static int synaptics_setup_intertouch(struct psmouse *psmouse, return -ENXIO; } - /* Disable intertouch on known-broken board revisions */ - if (info->board_id == 2722) { + /* Disable intertouch on known-broken T440p board revisions */ + if (info->board_id == 2722 && + psmouse_matches_pnp_id(psmouse, + (const char * const []){"LEN0036", NULL})) { psmouse_info(psmouse, "Disabling intertouch for board id %u\n", info->board_id); diff --git a/drivers/input/serio/i8042-acpipnpio.h b/drivers/input/serio/i8042-acpipnpio.h index 37f7fcd8d88c..d44d17d3190a 100644 --- a/drivers/input/serio/i8042-acpipnpio.h +++ b/drivers/input/serio/i8042-acpipnpio.h @@ -670,6 +670,15 @@ static const struct dmi_system_id i8042_dmi_quirk_table[] __initconst = { }, .driver_data = (void *)(SERIO_QUIRK_NOAUX) }, + { + /* Fujitsu LIFEBOOK U7410 */ + /* Mux mode also disturbs the EC keyboard, causing spurious ACKs */ + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "FUJITSU CLIENT COMPUTING LIMITED"), + DMI_MATCH(DMI_PRODUCT_NAME, "LIFEBOOK U7410"), + }, + .driver_data = (void *)(SERIO_QUIRK_NOMUX) + }, { /* Gigabyte M912 */ .matches = { diff --git a/drivers/input/touchscreen/s6sy761.c b/drivers/input/touchscreen/s6sy761.c index 998d99d18911..6ca2e422c534 100644 --- a/drivers/input/touchscreen/s6sy761.c +++ b/drivers/input/touchscreen/s6sy761.c @@ -507,10 +507,22 @@ static int s6sy761_suspend(struct device *dev) static int s6sy761_resume(struct device *dev) { struct s6sy761_data *sdata = dev_get_drvdata(dev); + int err; + + err = s6sy761_power_on(sdata); + if (err) + return err; enable_irq(sdata->client->irq); - return s6sy761_power_on(sdata); + guard(mutex)(&sdata->input->mutex); + + if (!input_device_enabled(sdata->input)) + return 0; + + err = i2c_smbus_write_byte(sdata->client, S6SY761_SENSE_ON); + + return err; } static const struct dev_pm_ops s6sy761_pm_ops = { diff --git a/drivers/iommu/amd/init.c b/drivers/iommu/amd/init.c index 7a9119f99e98..10ab34bed410 100644 --- a/drivers/iommu/amd/init.c +++ b/drivers/iommu/amd/init.c @@ -958,7 +958,9 @@ static void free_ga_log(struct amd_iommu *iommu) { #ifdef CONFIG_IRQ_REMAP free_pages((unsigned long)iommu->ga_log, get_order(GA_LOG_SIZE)); + iommu->ga_log = NULL; free_pages((unsigned long)iommu->ga_log_tail, get_order(8)); + iommu->ga_log_tail = NULL; #endif } @@ -1003,6 +1005,9 @@ static int iommu_init_ga_log(struct amd_iommu *iommu) if (!AMD_IOMMU_GUEST_IR_VAPIC(amd_iommu_guest_ir)) return 0; + if (iommu->ga_log && iommu->ga_log_tail) + return 0; + iommu->ga_log = (u8 *)__get_free_pages(GFP_KERNEL | __GFP_ZERO, get_order(GA_LOG_SIZE)); if (!iommu->ga_log) diff --git a/drivers/iommu/amd/iommu.c b/drivers/iommu/amd/iommu.c index 695cfb2e56aa..c267dc7f92be 100644 --- a/drivers/iommu/amd/iommu.c +++ b/drivers/iommu/amd/iommu.c @@ -217,10 +217,11 @@ static struct iommu_dev_data *search_dev_data(struct amd_iommu *iommu, u16 devid return NULL; } -static int clone_alias(struct pci_dev *pdev, u16 alias, void *data) +static int clone_alias(struct pci_dev *pdev_origin, u16 alias, void *data) { struct amd_iommu *iommu; struct dev_table_entry *dev_table; + struct pci_dev *pdev = data; u16 devid = pci_dev_id(pdev); if (devid == alias) @@ -252,9 +253,9 @@ static void clone_aliases(struct amd_iommu *iommu, struct device *dev) * part of the PCI DMA aliases if it's bus differs * from the original device. */ - clone_alias(pdev, iommu->pci_seg->alias_table[pci_dev_id(pdev)], NULL); + clone_alias(pdev, iommu->pci_seg->alias_table[pci_dev_id(pdev)], pdev); - pci_for_each_dma_alias(pdev, clone_alias, NULL); + pci_for_each_dma_alias(pdev, clone_alias, pdev); } static void setup_aliases(struct amd_iommu *iommu, struct device *dev) diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 70e5d7cda0c9..2c5e61786b36 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -2864,14 +2864,9 @@ static int arm_smmu_def_domain_type(struct device *dev) return 0; } -static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid) +static void arm_smmu_remove_dev_pasid(struct device *dev, ioasid_t pasid, + struct iommu_domain *domain) { - struct iommu_domain *domain; - - domain = iommu_get_domain_for_dev_pasid(dev, pasid, IOMMU_DOMAIN_SVA); - if (WARN_ON(IS_ERR(domain)) || !domain) - return; - arm_smmu_sva_remove_dev_pasid(domain, dev, pasid); } @@ -2962,6 +2957,20 @@ static int arm_smmu_cmdq_init(struct arm_smmu_device *smmu) return 0; } +static void arm_smmu_free_iopf_action(void *data) +{ + struct iopf_queue *queue = data; + + iopf_queue_free(queue); +} + +static void arm_smmu_destroy_vmid_map(void *data) +{ + struct ida *ida = data; + + ida_destroy(ida); +} + static int arm_smmu_init_queues(struct arm_smmu_device *smmu) { int ret; @@ -2989,6 +2998,11 @@ static int arm_smmu_init_queues(struct arm_smmu_device *smmu) smmu->evtq.iopf = iopf_queue_alloc(dev_name(smmu->dev)); if (!smmu->evtq.iopf) return -ENOMEM; + ret = devm_add_action_or_reset(smmu->dev, + arm_smmu_free_iopf_action, + smmu->evtq.iopf); + if (ret) + return ret; } /* priq */ @@ -3105,7 +3119,8 @@ static int arm_smmu_init_strtab(struct arm_smmu_device *smmu) ida_init(&smmu->vmid_map); - return 0; + return devm_add_action_or_reset(smmu->dev, arm_smmu_destroy_vmid_map, + &smmu->vmid_map); } static int arm_smmu_init_structures(struct arm_smmu_device *smmu) @@ -3311,6 +3326,13 @@ static int arm_smmu_device_disable(struct arm_smmu_device *smmu) return ret; } +static void arm_smmu_disable_action(void *data) +{ + struct arm_smmu_device *smmu = data; + + arm_smmu_device_disable(smmu); +} + static int arm_smmu_device_reset(struct arm_smmu_device *smmu, bool bypass) { int ret; @@ -3892,7 +3914,7 @@ static int arm_smmu_device_probe(struct platform_device *pdev) /* Initialise in-memory data structures */ ret = arm_smmu_init_structures(smmu); if (ret) - goto err_free_iopf; + return ret; /* Record our private device structure */ platform_set_drvdata(pdev, smmu); @@ -3902,30 +3924,30 @@ static int arm_smmu_device_probe(struct platform_device *pdev) /* Reset the device */ ret = arm_smmu_device_reset(smmu, bypass); + if (ret) { + arm_smmu_device_disable(smmu); + return ret; + } + + /* Register last so it unwinds first, while the CMDQ is still up. */ + ret = devm_add_action_or_reset(smmu->dev, arm_smmu_disable_action, smmu); if (ret) - goto err_disable; + return ret; /* And we're up. Go go go! */ ret = iommu_device_sysfs_add(&smmu->iommu, dev, NULL, "smmu3.%pa", &ioaddr); if (ret) - goto err_disable; + return ret; ret = iommu_device_register(&smmu->iommu, &arm_smmu_ops, dev); if (ret) { dev_err(dev, "Failed to register iommu\n"); - goto err_free_sysfs; + iommu_device_sysfs_remove(&smmu->iommu); + return ret; } return 0; - -err_free_sysfs: - iommu_device_sysfs_remove(&smmu->iommu); -err_disable: - arm_smmu_device_disable(smmu); -err_free_iopf: - iopf_queue_free(smmu->evtq.iopf); - return ret; } static void arm_smmu_device_remove(struct platform_device *pdev) @@ -3934,9 +3956,6 @@ static void arm_smmu_device_remove(struct platform_device *pdev) iommu_device_unregister(&smmu->iommu); iommu_device_sysfs_remove(&smmu->iommu); - arm_smmu_device_disable(smmu); - iopf_queue_free(smmu->evtq.iopf); - ida_destroy(&smmu->vmid_map); } static void arm_smmu_device_shutdown(struct platform_device *pdev) diff --git a/drivers/iommu/intel/iommu.c b/drivers/iommu/intel/iommu.c index d68bf9231bd5..10257548dd7d 100644 --- a/drivers/iommu/intel/iommu.c +++ b/drivers/iommu/intel/iommu.c @@ -170,7 +170,26 @@ static phys_addr_t root_entry_uctp(struct root_entry *re) static inline void context_set_present(struct context_entry *context) { - context->lo |= 1; + u64 val; + + dma_wmb(); + val = READ_ONCE(context->lo) | 1; + WRITE_ONCE(context->lo, val); +} + +/* + * Clear the Present (P) bit (bit 0) of a context table entry. This initiates + * the transition of the entry's ownership from hardware to software. The + * caller is responsible for fulfilling the invalidation handshake recommended + * by the VT-d spec, Section 6.5.3.3 (Guidance to Software for Invalidations). + */ +static inline void context_clear_present(struct context_entry *context) +{ + u64 val; + + val = READ_ONCE(context->lo) & GENMASK_ULL(63, 1); + WRITE_ONCE(context->lo, val); + dma_wmb(); } static inline void context_set_fault_enable(struct context_entry *context) @@ -2324,7 +2343,7 @@ static void domain_context_clear_one(struct device_domain_info *info, u8 bus, u8 did_old = context_domain_id(context); } - context_clear_entry(context); + context_clear_present(context); __iommu_flush_cache(iommu, context, sizeof(*context)); spin_unlock(&iommu->lock); iommu->flush.flush_context(iommu, @@ -2343,6 +2362,9 @@ static void domain_context_clear_one(struct device_domain_info *info, u8 bus, u8 DMA_TLB_DSI_FLUSH); __iommu_flush_dev_iotlb(info, 0, MAX_AGAW_PFN_WIDTH); + + context_clear_entry(context); + __iommu_flush_cache(iommu, context, sizeof(*context)); } static int domain_setup_first_level(struct intel_iommu *iommu, @@ -4700,18 +4722,14 @@ static void intel_iommu_iotlb_sync_map(struct iommu_domain *domain, __mapping_notify_one(info->iommu, dmar_domain, pfn, pages); } -static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) +static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid, + struct iommu_domain *domain) { struct intel_iommu *iommu = device_to_iommu(dev, NULL, NULL); + struct dmar_domain *dmar_domain = to_dmar_domain(domain); struct dev_pasid_info *curr, *dev_pasid = NULL; - struct dmar_domain *dmar_domain; - struct iommu_domain *domain; unsigned long flags; - domain = iommu_get_domain_for_dev_pasid(dev, pasid, 0); - if (WARN_ON_ONCE(!domain)) - goto out_tear_down; - /* * The SVA implementation needs to handle its own stuffs like the mm * notification. Before consolidating that code into iommu core, let @@ -4722,7 +4740,6 @@ static void intel_iommu_remove_dev_pasid(struct device *dev, ioasid_t pasid) goto out_tear_down; } - dmar_domain = to_dmar_domain(domain); spin_lock_irqsave(&dmar_domain->lock, flags); list_for_each_entry(curr, &dmar_domain->dev_pasids, link_domain) { if (curr->dev == dev && curr->pasid == pasid) { diff --git a/drivers/iommu/intel/pasid.c b/drivers/iommu/intel/pasid.c index 99589ac3b96b..5ce46fddcb0b 100644 --- a/drivers/iommu/intel/pasid.c +++ b/drivers/iommu/intel/pasid.c @@ -351,9 +351,23 @@ static inline void pasid_set_wpe(struct pasid_entry *pe) */ static inline void pasid_set_present(struct pasid_entry *pe) { + dma_wmb(); pasid_set_bits(&pe->val[0], 1 << 0, 1); } +/* + * Clear the Present (P) bit (bit 0) of a scalable-mode PASID table entry. + * This initiates the transition of the entry's ownership from hardware + * to software. The caller is responsible for fulfilling the invalidation + * handshake recommended by the VT-d spec, Section 6.5.3.3 (Guidance to + * Software for Invalidations). + */ +static inline void pasid_clear_present(struct pasid_entry *pe) +{ + pasid_set_bits(&pe->val[0], 1 << 0, 0); + dma_wmb(); +} + /* * Setup Page Walk Snoop bit (Bit 87) of a scalable mode PASID * entry. @@ -463,7 +477,7 @@ void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev, did = pasid_get_domain_id(pte); pgtt = pasid_pte_get_pgtt(pte); - intel_pasid_clear_entry(dev, pasid, fault_ignore); + pasid_clear_present(pte); spin_unlock(&iommu->lock); if (!ecap_coherent(iommu->ecap)) @@ -479,6 +493,10 @@ void intel_pasid_tear_down_entry(struct intel_iommu *iommu, struct device *dev, /* Device IOTLB doesn't need to be flushed in caching mode. */ if (!cap_caching_mode(iommu->cap)) devtlb_invalidation_with_pasid(iommu, dev, pasid); + + intel_pasid_clear_entry(dev, pasid, fault_ignore); + if (!ecap_coherent(iommu->ecap)) + clflush_cache_range(pte, sizeof(*pte)); } /* diff --git a/drivers/iommu/iommu-sva.c b/drivers/iommu/iommu-sva.c index a340b805b82b..efe491deaf61 100644 --- a/drivers/iommu/iommu-sva.c +++ b/drivers/iommu/iommu-sva.c @@ -100,10 +100,9 @@ struct iommu_sva *iommu_sva_bind_device(struct device *dev, struct mm_struct *mm goto out_free_domain; domain->users = 1; out: - mutex_unlock(&iommu_sva_lock); handle->dev = dev; handle->domain = domain; - + mutex_unlock(&iommu_sva_lock); return handle; out_free_domain: diff --git a/drivers/iommu/iommu.c b/drivers/iommu/iommu.c index 3fa5699b9ff1..1927869ddf49 100644 --- a/drivers/iommu/iommu.c +++ b/drivers/iommu/iommu.c @@ -3407,20 +3407,21 @@ static int __iommu_set_group_pasid(struct iommu_domain *domain, if (device == last_gdev) break; - ops->remove_dev_pasid(device->dev, pasid); + ops->remove_dev_pasid(device->dev, pasid, domain); } return ret; } static void __iommu_remove_group_pasid(struct iommu_group *group, - ioasid_t pasid) + ioasid_t pasid, + struct iommu_domain *domain) { struct group_device *device; const struct iommu_ops *ops; for_each_group_device(group, device) { ops = dev_iommu_ops(device->dev); - ops->remove_dev_pasid(device->dev, pasid); + ops->remove_dev_pasid(device->dev, pasid, domain); } } @@ -3479,7 +3480,7 @@ void iommu_detach_device_pasid(struct iommu_domain *domain, struct device *dev, struct iommu_group *group = iommu_group_get(dev); mutex_lock(&group->mutex); - __iommu_remove_group_pasid(group, pasid); + __iommu_remove_group_pasid(group, pasid, domain); WARN_ON(xa_erase(&group->pasid_array, pasid) != domain); mutex_unlock(&group->mutex); diff --git a/drivers/iommu/msm_iommu.c b/drivers/iommu/msm_iommu.c index be94bb16f627..1aff96d655b3 100644 --- a/drivers/iommu/msm_iommu.c +++ b/drivers/iommu/msm_iommu.c @@ -787,19 +787,24 @@ static int msm_iommu_probe(struct platform_device *pdev) "msm-smmu.%pa", &ioaddr); if (ret) { pr_err("Could not add msm-smmu at %pa to sysfs\n", &ioaddr); - goto fail; + goto err_remove_list; } ret = iommu_device_register(&iommu->iommu, &msm_iommu_ops, &pdev->dev); if (ret) { pr_err("Could not register msm-smmu at %pa\n", &ioaddr); - goto fail; + goto err_remove_sysfs; } pr_info("device mapped at %p, irq %d with %d ctx banks\n", iommu->base, iommu->irq, iommu->ncb); return ret; + +err_remove_sysfs: + iommu_device_sysfs_remove(&iommu->iommu); +err_remove_list: + list_del(&iommu->dev_node); fail: clk_unprepare(iommu->clk); clk_unprepare(iommu->pclk); diff --git a/drivers/iommu/s390-iommu.c b/drivers/iommu/s390-iommu.c index fbf59a8db29b..9328ada882b7 100644 --- a/drivers/iommu/s390-iommu.c +++ b/drivers/iommu/s390-iommu.c @@ -101,6 +101,23 @@ static void __s390_iommu_detach_device(struct zpci_dev *zdev) zdev->dma_table = NULL; } +static bool reg_ioat_propagate_error(int cc, u8 status) +{ + /* + * If the device is in the error state the reset routine + * will register the IOAT of the newly set domain on re-enable + */ + if (cc == ZPCI_CC_ERR && status == ZPCI_PCI_ST_FUNC_NOT_AVAIL) + return false; + /* + * If the device was removed treat registration as success + * and let the subsequent error event trigger tear down. + */ + if (cc == ZPCI_CC_INVAL_HANDLE) + return false; + return cc != ZPCI_CC_OK; +} + static int s390_iommu_attach_device(struct iommu_domain *domain, struct device *dev) { @@ -124,11 +141,7 @@ static int s390_iommu_attach_device(struct iommu_domain *domain, cc = zpci_register_ioat(zdev, 0, zdev->start_dma, zdev->end_dma, virt_to_phys(s390_domain->dma_table), &status); - /* - * If the device is undergoing error recovery the reset code - * will re-establish the new domain. - */ - if (cc && status != ZPCI_PCI_ST_FUNC_NOT_AVAIL) + if (reg_ioat_propagate_error(cc, status)) return -EIO; zdev->dma_table = s390_domain->dma_table; diff --git a/drivers/mailbox/zynqmp-ipi-mailbox.c b/drivers/mailbox/zynqmp-ipi-mailbox.c index 90f248ef2a1f..c89989c23411 100644 --- a/drivers/mailbox/zynqmp-ipi-mailbox.c +++ b/drivers/mailbox/zynqmp-ipi-mailbox.c @@ -72,6 +72,10 @@ struct zynqmp_ipi_mchan { unsigned int chan_type; }; +struct zynqmp_ipi_mbox; + +typedef int (*setup_ipi_fn)(struct zynqmp_ipi_mbox *ipi_mbox, struct device_node *node); + /** * struct zynqmp_ipi_mbox - Description of a ZynqMP IPI mailbox * platform data. @@ -81,7 +85,7 @@ struct zynqmp_ipi_mchan { * @remote_id: remote IPI agent ID * @mbox: mailbox Controller * @mchans: array for channels, tx channel and rx channel. - * @irq: IPI agent interrupt ID + * @setup_ipi_fn: Function Pointer to set up IPI Channels */ struct zynqmp_ipi_mbox { struct zynqmp_ipi_pdata *pdata; @@ -89,6 +93,7 @@ struct zynqmp_ipi_mbox { u32 remote_id; struct mbox_controller mbox; struct zynqmp_ipi_mchan mchans[2]; + setup_ipi_fn setup_ipi_fn; }; /** @@ -416,12 +421,6 @@ static struct mbox_chan *zynqmp_ipi_of_xlate(struct mbox_controller *mbox, return chan; } -static const struct of_device_id zynqmp_ipi_of_match[] = { - { .compatible = "xlnx,zynqmp-ipi-mailbox" }, - {}, -}; -MODULE_DEVICE_TABLE(of, zynqmp_ipi_of_match); - /** * zynqmp_ipi_mbox_get_buf_res - Get buffer resource from the IPI dev node * @@ -471,12 +470,9 @@ static void zynqmp_ipi_mbox_dev_release(struct device *dev) static int zynqmp_ipi_mbox_probe(struct zynqmp_ipi_mbox *ipi_mbox, struct device_node *node) { - struct zynqmp_ipi_mchan *mchan; struct mbox_chan *chans; struct mbox_controller *mbox; - struct resource res; struct device *dev, *mdev; - const char *name; int ret; dev = ipi_mbox->pdata->dev; @@ -496,6 +492,73 @@ static int zynqmp_ipi_mbox_probe(struct zynqmp_ipi_mbox *ipi_mbox, } mdev = &ipi_mbox->dev; + /* Get the IPI remote agent ID */ + ret = of_property_read_u32(node, "xlnx,ipi-id", &ipi_mbox->remote_id); + if (ret < 0) { + dev_err(dev, "No IPI remote ID is specified.\n"); + return ret; + } + + ret = ipi_mbox->setup_ipi_fn(ipi_mbox, node); + if (ret) { + dev_err(dev, "Failed to set up IPI Buffers.\n"); + return ret; + } + + mbox = &ipi_mbox->mbox; + mbox->dev = mdev; + mbox->ops = &zynqmp_ipi_chan_ops; + mbox->num_chans = 2; + mbox->txdone_irq = false; + mbox->txdone_poll = true; + mbox->txpoll_period = 5; + mbox->of_xlate = zynqmp_ipi_of_xlate; + chans = devm_kzalloc(mdev, 2 * sizeof(*chans), GFP_KERNEL); + if (!chans) + return -ENOMEM; + mbox->chans = chans; + chans[IPI_MB_CHNL_TX].con_priv = &ipi_mbox->mchans[IPI_MB_CHNL_TX]; + chans[IPI_MB_CHNL_RX].con_priv = &ipi_mbox->mchans[IPI_MB_CHNL_RX]; + ipi_mbox->mchans[IPI_MB_CHNL_TX].chan_type = IPI_MB_CHNL_TX; + ipi_mbox->mchans[IPI_MB_CHNL_RX].chan_type = IPI_MB_CHNL_RX; + ret = devm_mbox_controller_register(mdev, mbox); + if (ret) + dev_err(mdev, + "Failed to register mbox_controller(%d)\n", ret); + else + dev_info(mdev, + "Registered ZynqMP IPI mbox with TX/RX channels.\n"); + return ret; +} + +/** + * zynqmp_ipi_setup - set up IPI Buffers for classic flow + * + * @ipi_mbox: pointer to IPI mailbox private data structure + * @node: IPI mailbox device node + * + * This will be used to set up IPI Buffers for ZynqMP SOC if user + * wishes to use classic driver usage model on new SOC's with only + * buffered IPIs. + * + * Note that bufferless IPIs and mixed usage of buffered and bufferless + * IPIs are not supported with this flow. + * + * This will be invoked with compatible string "xlnx,zynqmp-ipi-mailbox". + * + * Return: 0 for success, negative value for failure + */ +static int zynqmp_ipi_setup(struct zynqmp_ipi_mbox *ipi_mbox, + struct device_node *node) +{ + struct zynqmp_ipi_mchan *mchan; + struct device *mdev; + struct resource res; + const char *name; + int ret; + + mdev = &ipi_mbox->dev; + mchan = &ipi_mbox->mchans[IPI_MB_CHNL_TX]; name = "local_request_region"; ret = zynqmp_ipi_mbox_get_buf_res(node, name, &res); @@ -570,37 +633,7 @@ static int zynqmp_ipi_mbox_probe(struct zynqmp_ipi_mbox *ipi_mbox, if (!mchan->rx_buf) return -ENOMEM; - /* Get the IPI remote agent ID */ - ret = of_property_read_u32(node, "xlnx,ipi-id", &ipi_mbox->remote_id); - if (ret < 0) { - dev_err(dev, "No IPI remote ID is specified.\n"); - return ret; - } - - mbox = &ipi_mbox->mbox; - mbox->dev = mdev; - mbox->ops = &zynqmp_ipi_chan_ops; - mbox->num_chans = 2; - mbox->txdone_irq = false; - mbox->txdone_poll = true; - mbox->txpoll_period = 5; - mbox->of_xlate = zynqmp_ipi_of_xlate; - chans = devm_kzalloc(mdev, 2 * sizeof(*chans), GFP_KERNEL); - if (!chans) - return -ENOMEM; - mbox->chans = chans; - chans[IPI_MB_CHNL_TX].con_priv = &ipi_mbox->mchans[IPI_MB_CHNL_TX]; - chans[IPI_MB_CHNL_RX].con_priv = &ipi_mbox->mchans[IPI_MB_CHNL_RX]; - ipi_mbox->mchans[IPI_MB_CHNL_TX].chan_type = IPI_MB_CHNL_TX; - ipi_mbox->mchans[IPI_MB_CHNL_RX].chan_type = IPI_MB_CHNL_RX; - ret = devm_mbox_controller_register(mdev, mbox); - if (ret) - dev_err(mdev, - "Failed to register mbox_controller(%d)\n", ret); - else - dev_info(mdev, - "Registered ZynqMP IPI mbox with TX/RX channels.\n"); - return ret; + return 0; } /** @@ -613,7 +646,7 @@ static void zynqmp_ipi_free_mboxes(struct zynqmp_ipi_pdata *pdata) struct zynqmp_ipi_mbox *ipi_mbox; int i; - i = pdata->num_mboxes; + i = pdata->num_mboxes - 1; for (; i >= 0; i--) { ipi_mbox = &pdata->ipi_mboxes[i]; if (device_is_registered(&ipi_mbox->dev)) @@ -628,6 +661,7 @@ static int zynqmp_ipi_probe(struct platform_device *pdev) struct zynqmp_ipi_pdata *pdata; struct zynqmp_ipi_mbox *mbox; int num_mboxes, ret = -EINVAL; + setup_ipi_fn ipi_fn; num_mboxes = of_get_available_child_count(np); if (num_mboxes == 0) { @@ -648,9 +682,18 @@ static int zynqmp_ipi_probe(struct platform_device *pdev) return ret; } + ipi_fn = (setup_ipi_fn)device_get_match_data(&pdev->dev); + if (!ipi_fn) { + dev_err(dev, + "Mbox Compatible String is missing IPI Setup fn.\n"); + return -ENODEV; + } + pdata->num_mboxes = num_mboxes; mbox = pdata->ipi_mboxes; + mbox->setup_ipi_fn = ipi_fn; + for_each_available_child_of_node(np, nc) { mbox->pdata = pdata; ret = zynqmp_ipi_mbox_probe(mbox, nc); @@ -695,6 +738,14 @@ static int zynqmp_ipi_remove(struct platform_device *pdev) return 0; } +static const struct of_device_id zynqmp_ipi_of_match[] = { + { .compatible = "xlnx,zynqmp-ipi-mailbox", + .data = &zynqmp_ipi_setup, + }, + {}, +}; +MODULE_DEVICE_TABLE(of, zynqmp_ipi_of_match); + static struct platform_driver zynqmp_ipi_driver = { .probe = zynqmp_ipi_probe, .remove = zynqmp_ipi_remove, diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c index 5522f8e9443f..bbc5c8bb2b85 100644 --- a/drivers/md/dm-cache-target.c +++ b/drivers/md/dm-cache-target.c @@ -3539,8 +3539,8 @@ static void cache_io_hints(struct dm_target *ti, struct queue_limits *limits) */ if (io_opt_sectors < cache->sectors_per_block || do_div(io_opt_sectors, cache->sectors_per_block)) { - blk_limits_io_min(limits, cache->sectors_per_block << SECTOR_SHIFT); - blk_limits_io_opt(limits, cache->sectors_per_block << SECTOR_SHIFT); + limits->io_min = cache->sectors_per_block << SECTOR_SHIFT; + limits->io_opt = cache->sectors_per_block << SECTOR_SHIFT; } disable_passdown_if_not_supported(cache); diff --git a/drivers/md/dm-clone-target.c b/drivers/md/dm-clone-target.c index 94b2fc33f64b..56e9a4911d96 100644 --- a/drivers/md/dm-clone-target.c +++ b/drivers/md/dm-clone-target.c @@ -2078,8 +2078,8 @@ static void clone_io_hints(struct dm_target *ti, struct queue_limits *limits) */ if (io_opt_sectors < clone->region_size || do_div(io_opt_sectors, clone->region_size)) { - blk_limits_io_min(limits, clone->region_size << SECTOR_SHIFT); - blk_limits_io_opt(limits, clone->region_size << SECTOR_SHIFT); + limits->io_min = clone->region_size << SECTOR_SHIFT; + limits->io_opt = clone->region_size << SECTOR_SHIFT; } disable_passdown_if_not_supported(clone); diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index 1ff16b8264cb..834478059220 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -2950,8 +2950,13 @@ static int crypt_ctr_cipher_new(struct dm_target *ti, char *cipher_in, char *key } } - if (*ivmode && !strcmp(*ivmode, "lmk")) + if (*ivmode && !strcmp(*ivmode, "lmk")) { + if (crypt_integrity_aead(cc)) { + ti->error = "AEAD transforms not supported for LMK"; + return -EINVAL; + } cc->tfms_count = 64; + } if (*ivmode && !strcmp(*ivmode, "essiv")) { if (!*ivopts) { diff --git a/drivers/md/dm-ebs-target.c b/drivers/md/dm-ebs-target.c index 45b8d314ef0d..3a609ef22168 100644 --- a/drivers/md/dm-ebs-target.c +++ b/drivers/md/dm-ebs-target.c @@ -434,7 +434,7 @@ static void ebs_io_hints(struct dm_target *ti, struct queue_limits *limits) limits->logical_block_size = to_bytes(ec->e_bs); limits->physical_block_size = to_bytes(ec->u_bs); limits->alignment_offset = limits->physical_block_size; - blk_limits_io_min(limits, limits->logical_block_size); + limits->io_min = limits->logical_block_size; } static int ebs_iterate_devices(struct dm_target *ti, diff --git a/drivers/md/dm-era-target.c b/drivers/md/dm-era-target.c index 2fe173a9880b..c8508a089123 100644 --- a/drivers/md/dm-era-target.c +++ b/drivers/md/dm-era-target.c @@ -1743,8 +1743,8 @@ static void era_io_hints(struct dm_target *ti, struct queue_limits *limits) */ if (io_opt_sectors < era->sectors_per_block || do_div(io_opt_sectors, era->sectors_per_block)) { - blk_limits_io_min(limits, 0); - blk_limits_io_opt(limits, era->sectors_per_block << SECTOR_SHIFT); + limits->io_min = 0; + limits->io_opt = era->sectors_per_block << SECTOR_SHIFT; } } diff --git a/drivers/md/dm-integrity.c b/drivers/md/dm-integrity.c index 1b562d2189c5..aa6f3e71c4fa 100644 --- a/drivers/md/dm-integrity.c +++ b/drivers/md/dm-integrity.c @@ -3512,7 +3512,7 @@ static void dm_integrity_io_hints(struct dm_target *ti, struct queue_limits *lim if (ic->sectors_per_block > 1) { limits->logical_block_size = ic->sectors_per_block << SECTOR_SHIFT; limits->physical_block_size = ic->sectors_per_block << SECTOR_SHIFT; - blk_limits_io_min(limits, ic->sectors_per_block << SECTOR_SHIFT); + limits->io_min = ic->sectors_per_block << SECTOR_SHIFT; limits->dma_alignment = limits->logical_block_size - 1; } } diff --git a/drivers/md/dm-ioctl.c b/drivers/md/dm-ioctl.c index 874778c74196..862ac077696b 100644 --- a/drivers/md/dm-ioctl.c +++ b/drivers/md/dm-ioctl.c @@ -556,10 +556,16 @@ static void *get_result_buffer(struct dm_ioctl *param, size_t param_size, { param->data_start = align_ptr(param + 1) - (void *) param; - if (param->data_start < param_size) + if (param->data_start < param_size) { *len = param_size - param->data_start; - else + } else { + /* + * The buffer can be as small as offsetof(*param, data), which + * is less than sizeof(*param), so don't run off the end of it. + */ *len = 0; + param->data_start = param_size; + } return ((void *) param) + param->data_start; } @@ -1175,6 +1181,7 @@ static int do_resume(struct dm_ioctl *param) /* Do we need to load a new map ? */ if (new_map) { sector_t old_size, new_size; + blk_mode_t new_map_mode; /* Suspend if it isn't already suspended */ if (param->flags & DM_SKIP_LOCKFS_FLAG) @@ -1202,6 +1209,7 @@ static int do_resume(struct dm_ioctl *param) } } + new_map_mode = dm_table_get_mode(new_map); old_size = dm_get_size(md); old_map = dm_swap_table(md, new_map); if (IS_ERR(old_map)) { @@ -1214,7 +1222,7 @@ static int do_resume(struct dm_ioctl *param) if (old_size && new_size && old_size != new_size) need_resize_uevent = true; - if (dm_table_get_mode(new_map) & BLK_OPEN_WRITE) + if (new_map_mode & BLK_OPEN_WRITE) set_disk_ro(dm_disk(md), 0); else set_disk_ro(dm_disk(md), 1); diff --git a/drivers/md/dm-raid.c b/drivers/md/dm-raid.c index 0c4ab6865182..2d6a35803c1e 100644 --- a/drivers/md/dm-raid.c +++ b/drivers/md/dm-raid.c @@ -3782,8 +3782,8 @@ static void raid_io_hints(struct dm_target *ti, struct queue_limits *limits) struct raid_set *rs = ti->private; unsigned int chunk_size_bytes = to_bytes(rs->md.chunk_sectors); - blk_limits_io_min(limits, chunk_size_bytes); - blk_limits_io_opt(limits, chunk_size_bytes * mddev_data_stripes(rs)); + limits->io_min = chunk_size_bytes; + limits->io_opt = chunk_size_bytes * mddev_data_stripes(rs); } static void raid_postsuspend(struct dm_target *ti) diff --git a/drivers/md/dm-snap.c b/drivers/md/dm-snap.c index dcffa3441a66..ebe5fa1566c8 100644 --- a/drivers/md/dm-snap.c +++ b/drivers/md/dm-snap.c @@ -1298,6 +1298,20 @@ static int snapshot_ctr(struct dm_target *ti, unsigned int argc, char **argv) goto bad_store; } + /* + * snapshot-merge requires the optional prepare_merge/commit_merge + * callbacks; the transient exception store does not implement them. + * The handover path checks this in __validate_exception_handover(), + * but a directly constructed snapshot-merge never goes through it. + */ + if (dm_target_is_snapshot_merge(ti) && + (!s->store->type->prepare_merge || + !s->store->type->commit_merge)) { + ti->error = "Snapshot exception store does not support snapshot-merge."; + r = -EINVAL; + goto bad_hash_tables; + } + argv += args_used; argc -= args_used; diff --git a/drivers/md/dm-stripe.c b/drivers/md/dm-stripe.c index e2854a3cbd28..11d8ff3c4563 100644 --- a/drivers/md/dm-stripe.c +++ b/drivers/md/dm-stripe.c @@ -456,10 +456,15 @@ static void stripe_io_hints(struct dm_target *ti, struct queue_limits *limits) { struct stripe_c *sc = ti->private; - unsigned int chunk_size = sc->chunk_size << SECTOR_SHIFT; + unsigned int io_min, io_opt; - blk_limits_io_min(limits, chunk_size); - blk_limits_io_opt(limits, chunk_size * sc->stripes); + limits->chunk_sectors = sc->chunk_size; + + if (!check_shl_overflow(sc->chunk_size, SECTOR_SHIFT, &io_min) && + !check_mul_overflow(io_min, sc->stripes, &io_opt)) { + limits->io_min = io_min; + limits->io_opt = io_opt; + } } static struct target_type stripe_target = { diff --git a/drivers/md/dm-thin.c b/drivers/md/dm-thin.c index 7935363d13b1..1e860940293a 100644 --- a/drivers/md/dm-thin.c +++ b/drivers/md/dm-thin.c @@ -4081,10 +4081,10 @@ static void pool_io_hints(struct dm_target *ti, struct queue_limits *limits) if (io_opt_sectors < pool->sectors_per_block || !is_factor(io_opt_sectors, pool->sectors_per_block)) { if (is_factor(pool->sectors_per_block, limits->max_sectors)) - blk_limits_io_min(limits, limits->max_sectors << SECTOR_SHIFT); + limits->io_min = limits->max_sectors << SECTOR_SHIFT; else - blk_limits_io_min(limits, pool->sectors_per_block << SECTOR_SHIFT); - blk_limits_io_opt(limits, pool->sectors_per_block << SECTOR_SHIFT); + limits->io_min = pool->sectors_per_block << SECTOR_SHIFT; + limits->io_opt = pool->sectors_per_block << SECTOR_SHIFT; } /* diff --git a/drivers/md/dm-verity-fec.c b/drivers/md/dm-verity-fec.c index 5f646c4bb98a..a9e3d4498086 100644 --- a/drivers/md/dm-verity-fec.c +++ b/drivers/md/dm-verity-fec.c @@ -55,36 +55,6 @@ static int fec_decode_rs8(struct dm_verity *v, struct dm_verity_fec_io *fio, fio->erasures, 0, NULL); } -/* - * Read error-correcting codes for the requested RS block. Returns a pointer - * to the data block. Caller is responsible for releasing buf. - */ -static u8 *fec_read_parity(struct dm_verity *v, u64 rsb, int index, - unsigned int *offset, unsigned int par_buf_offset, - struct dm_buffer **buf) -{ - u64 position, block, rem; - u8 *res; - - /* We have already part of parity bytes read, skip to the next block */ - if (par_buf_offset) - index++; - - position = (index + rsb) * v->fec->roots; - block = div64_u64_rem(position, v->fec->io_size, &rem); - *offset = par_buf_offset ? 0 : (unsigned int)rem; - - res = dm_bufio_read(v->fec->bufio, block, buf); - if (IS_ERR(res)) { - DMERR("%s: FEC %llu: parity read failed (block %llu): %ld", - v->data_dev->name, (unsigned long long)rsb, - (unsigned long long)block, PTR_ERR(res)); - *buf = NULL; - } - - return res; -} - /* Loop over each preallocated buffer slot. */ #define fec_for_each_prealloc_buffer(__i) \ for (__i = 0; __i < DM_VERITY_FEC_BUF_PREALLOC; __i++) @@ -132,13 +102,27 @@ static int fec_decode_bufs(struct dm_verity *v, struct dm_verity_fec_io *fio, { int r, corrected = 0, res; struct dm_buffer *buf; - unsigned int n, i, offset, par_buf_offset = 0; + unsigned int n, i, parity_pos, to_copy; u8 *par, *block, par_buf[DM_VERITY_FEC_RSM - DM_VERITY_FEC_MIN_RSN]; + u64 parity_block; - par = fec_read_parity(v, rsb, block_offset, &offset, - par_buf_offset, &buf); - if (IS_ERR(par)) + /* + * Compute the index of the first parity block that will be needed and + * the starting position in that block. Then read that block. + * + * io_size is always a power of 2, but roots might not be. Note that + * when it's not, a codeword's parity bytes can span a block boundary. + */ + parity_block = (rsb + block_offset) * v->fec->roots; + parity_pos = parity_block & (v->fec->io_size - 1); + parity_block >>= v->data_dev_block_bits; + par = dm_bufio_read(v->fec->bufio, parity_block, &buf); + if (IS_ERR(par)) { + DMERR("%s: FEC %llu: parity read failed (block %llu): %ld", + v->data_dev->name, (unsigned long long)rsb, + (unsigned long long)parity_block, PTR_ERR(par)); return PTR_ERR(par); + } /* * Decode the RS blocks we have in bufs. Each RS block results in @@ -146,7 +130,31 @@ static int fec_decode_bufs(struct dm_verity *v, struct dm_verity_fec_io *fio, */ fec_for_each_buffer_rs_block(fio, n, i) { block = fec_buffer_rs_block(v, fio, n, i); - memcpy(&par_buf[par_buf_offset], &par[offset], v->fec->roots - par_buf_offset); + + /* + * Copy the next 'roots' parity bytes to 'par_buf', reading + * another parity block if needed. + */ + to_copy = min(v->fec->io_size - parity_pos, v->fec->roots); + memcpy(par_buf, &par[parity_pos], to_copy); + parity_pos += to_copy; + if (to_copy < v->fec->roots) { + parity_block++; + parity_pos = 0; + + dm_bufio_release(buf); + par = dm_bufio_read(v->fec->bufio, parity_block, &buf); + if (IS_ERR(par)) { + DMERR("%s: FEC %llu: parity read failed (block %llu): %ld", + v->data_dev->name, (unsigned long long)rsb, + (unsigned long long)parity_block, + PTR_ERR(par)); + return PTR_ERR(par); + } + memcpy(&par_buf[to_copy], par, v->fec->roots - to_copy); + parity_pos = v->fec->roots - to_copy; + } + res = fec_decode_rs8(v, fio, block, par_buf, neras); if (res < 0) { r = res; @@ -159,25 +167,6 @@ static int fec_decode_bufs(struct dm_verity *v, struct dm_verity_fec_io *fio, block_offset++; if (block_offset >= 1 << v->data_dev_block_bits) goto done; - - /* Read the next block when we run out of parity bytes */ - offset += (v->fec->roots - par_buf_offset); - /* Check if parity bytes are split between blocks */ - if (offset < v->fec->io_size && (offset + v->fec->roots) > v->fec->io_size) { - par_buf_offset = v->fec->io_size - offset; - memcpy(par_buf, &par[offset], par_buf_offset); - offset += par_buf_offset; - } else - par_buf_offset = 0; - - if (offset >= v->fec->io_size) { - dm_bufio_release(buf); - - par = fec_read_parity(v, rsb, block_offset, &offset, - par_buf_offset, &buf); - if (IS_ERR(par)) - return PTR_ERR(par); - } } done: r = corrected; @@ -272,7 +261,7 @@ static int fec_read_bufs(struct dm_verity *v, struct dm_verity_io *io, (unsigned long long)block, PTR_ERR(bbuf)); /* assume the block is corrupted */ - if (neras && *neras <= v->fec->roots) + if (neras && *neras < v->fec->roots) fio->erasures[(*neras)++] = i; continue; @@ -290,7 +279,7 @@ static int fec_read_bufs(struct dm_verity *v, struct dm_verity_io *io, * skip if we have already found the theoretical * maximum number (i.e. fec->roots) of erasures */ - if (neras && *neras <= v->fec->roots && + if (neras && *neras < v->fec->roots && fec_is_erasure(v, io, want_digest, bbuf)) fio->erasures[(*neras)++] = i; } diff --git a/drivers/md/dm-verity-target.c b/drivers/md/dm-verity-target.c index 7e877c6adb2e..27ef5c0167cb 100644 --- a/drivers/md/dm-verity-target.c +++ b/drivers/md/dm-verity-target.c @@ -996,7 +996,15 @@ static void verity_io_hints(struct dm_target *ti, struct queue_limits *limits) if (limits->physical_block_size < 1 << v->data_dev_block_bits) limits->physical_block_size = 1 << v->data_dev_block_bits; - blk_limits_io_min(limits, limits->logical_block_size); + limits->io_min = limits->logical_block_size; + + /* + * Similar to what dm-crypt does, opt dm-verity out of support for + * direct I/O that is aligned to less than the traditional direct I/O + * alignment requirement of logical_block_size. This prevents dm-verity + * data blocks from crossing pages, eliminating various edge cases. + */ + limits->dma_alignment = limits->logical_block_size - 1; } static void verity_dtr(struct dm_target *ti) diff --git a/drivers/md/dm-zoned-target.c b/drivers/md/dm-zoned-target.c index 36e55a5bcb0d..b57f7259f999 100644 --- a/drivers/md/dm-zoned-target.c +++ b/drivers/md/dm-zoned-target.c @@ -996,8 +996,8 @@ static void dmz_io_hints(struct dm_target *ti, struct queue_limits *limits) limits->logical_block_size = DMZ_BLOCK_SIZE; limits->physical_block_size = DMZ_BLOCK_SIZE; - blk_limits_io_min(limits, DMZ_BLOCK_SIZE); - blk_limits_io_opt(limits, DMZ_BLOCK_SIZE); + limits->io_min = DMZ_BLOCK_SIZE; + limits->io_opt = DMZ_BLOCK_SIZE; limits->discard_alignment = 0; limits->discard_granularity = DMZ_BLOCK_SIZE; diff --git a/drivers/md/md.c b/drivers/md/md.c index 3c605ba3801a..49770a698aab 100644 --- a/drivers/md/md.c +++ b/drivers/md/md.c @@ -89,7 +89,7 @@ static int remove_and_add_spares(struct mddev *mddev, struct md_rdev *this); static void mddev_detach(struct mddev *mddev); static void export_rdev(struct md_rdev *rdev, struct mddev *mddev); -static void md_wakeup_thread_directly(struct md_thread __rcu *thread); +static void md_wakeup_thread_directly(struct md_thread __rcu **thread); /* * Default number of read corrections we'll attempt on an rdev @@ -211,7 +211,8 @@ void mddev_create_serial_pool(struct mddev *mddev, struct md_rdev *rdev, { int ret = 0; - if (rdev && !rdev_need_serial(rdev) && + if (!mddev->serialize_policy && + rdev && !rdev_need_serial(rdev) && !test_bit(CollisionCheck, &rdev->flags)) return; @@ -371,25 +372,32 @@ static bool is_suspended(struct mddev *mddev, struct bio *bio) void md_handle_request(struct mddev *mddev, struct bio *bio) { check_suspended: - if (is_suspended(mddev, bio)) { - DEFINE_WAIT(__wait); - /* Bail out if REQ_NOWAIT is set for the bio */ - if (bio->bi_opf & REQ_NOWAIT) { - bio_wouldblock_error(bio); - return; - } - for (;;) { - prepare_to_wait(&mddev->sb_wait, &__wait, - TASK_UNINTERRUPTIBLE); - if (!is_suspended(mddev, bio)) - break; - schedule(); + if (unlikely(md_cloned_bio(mddev, bio))) { + /* + * This bio is an MD cloned bio and already holds an + * active_io reference, so percpu_ref_get() is safe here. + */ + percpu_ref_get(&mddev->active_io); + } else { + if (is_suspended(mddev, bio)) { + DEFINE_WAIT(__wait); + /* Bail out if REQ_NOWAIT is set for the bio */ + if (bio->bi_opf & REQ_NOWAIT) { + bio_wouldblock_error(bio); + return; + } + for (;;) { + prepare_to_wait(&mddev->sb_wait, &__wait, + TASK_UNINTERRUPTIBLE); + if (!is_suspended(mddev, bio)) + break; + schedule(); + } + finish_wait(&mddev->sb_wait, &__wait); } - finish_wait(&mddev->sb_wait, &__wait); + if (!percpu_ref_tryget_live(&mddev->active_io)) + goto check_suspended; } - if (!percpu_ref_tryget_live(&mddev->active_io)) - goto check_suspended; - if (!mddev->pers->make_request(mddev, bio)) { percpu_ref_put(&mddev->active_io); goto check_suspended; @@ -443,12 +451,22 @@ void mddev_suspend(struct mddev *mddev) lockdep_is_held(&mddev->reconfig_mutex)); WARN_ON_ONCE(thread && current == thread->tsk); - if (mddev->suspended++) + + /* can't concurrent with __mddev_suspend() and __mddev_resume() */ + mutex_lock(&mddev->suspend_mutex); + if (mddev->suspended++) { + mutex_unlock(&mddev->suspend_mutex); return; + } + wake_up(&mddev->sb_wait); set_bit(MD_ALLOW_SB_UPDATE, &mddev->flags); percpu_ref_kill(&mddev->active_io); + /* + * TODO: cleanup 'pers->prepare_suspend after all callers are replaced + * by __mddev_suspend(). + */ if (mddev->pers && mddev->pers->prepare_suspend) mddev->pers->prepare_suspend(mddev); @@ -458,14 +476,21 @@ void mddev_suspend(struct mddev *mddev) /* restrict memory reclaim I/O during raid array is suspend */ mddev->noio_flag = memalloc_noio_save(); + + mutex_unlock(&mddev->suspend_mutex); } EXPORT_SYMBOL_GPL(mddev_suspend); void mddev_resume(struct mddev *mddev) { lockdep_assert_held(&mddev->reconfig_mutex); - if (--mddev->suspended) + + /* can't concurrent with __mddev_suspend() and __mddev_resume() */ + mutex_lock(&mddev->suspend_mutex); + if (--mddev->suspended) { + mutex_unlock(&mddev->suspend_mutex); return; + } /* entred the memalloc scope from mddev_suspend() */ memalloc_noio_restore(mddev->noio_flag); @@ -476,9 +501,89 @@ void mddev_resume(struct mddev *mddev) set_bit(MD_RECOVERY_NEEDED, &mddev->recovery); md_wakeup_thread(mddev->thread); md_wakeup_thread(mddev->sync_thread); /* possibly kick off a reshape */ + + mutex_unlock(&mddev->suspend_mutex); } EXPORT_SYMBOL_GPL(mddev_resume); +int __mddev_suspend(struct mddev *mddev, bool interruptible) +{ + int err = 0; + + /* + * hold reconfig_mutex to wait for normal io will deadlock, because + * other context can't update super_block, and normal io can rely on + * updating super_block. + */ + lockdep_assert_not_held(&mddev->reconfig_mutex); + + if (interruptible) + err = mutex_lock_interruptible(&mddev->suspend_mutex); + else + mutex_lock(&mddev->suspend_mutex); + if (err) + return err; + + if (mddev->suspended) { + WRITE_ONCE(mddev->suspended, mddev->suspended + 1); + mutex_unlock(&mddev->suspend_mutex); + return 0; + } + + percpu_ref_kill(&mddev->active_io); + if (interruptible) + err = wait_event_interruptible(mddev->sb_wait, + percpu_ref_is_zero(&mddev->active_io)); + else + wait_event(mddev->sb_wait, + percpu_ref_is_zero(&mddev->active_io)); + if (err) { + percpu_ref_resurrect(&mddev->active_io); + mutex_unlock(&mddev->suspend_mutex); + return err; + } + + /* + * For raid456, io might be waiting for reshape to make progress, + * allow new reshape to start while waiting for io to be done to + * prevent deadlock. + */ + WRITE_ONCE(mddev->suspended, mddev->suspended + 1); + + del_timer_sync(&mddev->safemode_timer); + /* restrict memory reclaim I/O during raid array is suspend */ + mddev->noio_flag = memalloc_noio_save(); + + mutex_unlock(&mddev->suspend_mutex); + return 0; +} +EXPORT_SYMBOL_GPL(__mddev_suspend); + +void __mddev_resume(struct mddev *mddev) +{ + lockdep_assert_not_held(&mddev->reconfig_mutex); + + mutex_lock(&mddev->suspend_mutex); + WRITE_ONCE(mddev->suspended, mddev->suspended - 1); + if (mddev->suspended) { + mutex_unlock(&mddev->suspend_mutex); + return; + } + + /* entred the memalloc scope from __mddev_suspend() */ + memalloc_noio_restore(mddev->noio_flag); + + percpu_ref_resurrect(&mddev->active_io); + wake_up(&mddev->sb_wait); + + set_bit(MD_RECOVERY_NEEDED, &mddev->recovery); + md_wakeup_thread(mddev->thread); + md_wakeup_thread(mddev->sync_thread); /* possibly kick off a reshape */ + + mutex_unlock(&mddev->suspend_mutex); +} +EXPORT_SYMBOL_GPL(__mddev_resume); + /* * Generic flush handling for md */ @@ -672,6 +777,7 @@ void mddev_init(struct mddev *mddev) mutex_init(&mddev->open_mutex); mutex_init(&mddev->reconfig_mutex); mutex_init(&mddev->sync_mutex); + mutex_init(&mddev->suspend_mutex); mutex_init(&mddev->bitmap_info.mutex); INIT_LIST_HEAD(&mddev->disks); INIT_LIST_HEAD(&mddev->all_mddevs); @@ -4211,7 +4317,7 @@ raid_disks_store(struct mddev *mddev, const char *buf, size_t len) if (err < 0) return err; - err = mddev_lock(mddev); + err = mddev_suspend_and_lock(mddev); if (err) return err; if (mddev->pers) @@ -4236,7 +4342,7 @@ raid_disks_store(struct mddev *mddev, const char *buf, size_t len) } else mddev->raid_disks = n; out_unlock: - mddev_unlock(mddev); + mddev_unlock_and_resume(mddev); return err ? err : len; } static struct md_sysfs_entry md_raid_disks = @@ -4832,47 +4938,58 @@ action_show(struct mddev *mddev, char *page) return sprintf(page, "%s\n", type); } -static void stop_sync_thread(struct mddev *mddev) +/** + * stop_sync_thread() - wait for sync_thread to stop if it's running. + * @mddev: the array. + * @locked: if set, reconfig_mutex will still be held after this function + * return; if not set, reconfig_mutex will be released after this + * function return. + * @check_seq: if set, only wait for curent running sync_thread to stop, noted + * that new sync_thread can still start. + */ +static void stop_sync_thread(struct mddev *mddev, bool locked, bool check_seq) { - if (!test_bit(MD_RECOVERY_RUNNING, &mddev->recovery)) - return; + int sync_seq; - if (mddev_lock(mddev)) - return; + if (check_seq) + sync_seq = atomic_read(&mddev->sync_seq); - /* - * Check again in case MD_RECOVERY_RUNNING is cleared before lock is - * held. - */ if (!test_bit(MD_RECOVERY_RUNNING, &mddev->recovery)) { - mddev_unlock(mddev); + if (!locked) + mddev_unlock(mddev); return; } - if (work_pending(&mddev->sync_work)) - flush_workqueue(md_misc_wq); + mddev_unlock(mddev); set_bit(MD_RECOVERY_INTR, &mddev->recovery); /* * Thread might be blocked waiting for metadata update which will now * never happen */ - md_wakeup_thread_directly(mddev->sync_thread); + md_wakeup_thread_directly(&mddev->sync_thread); + if (work_pending(&mddev->sync_work)) + flush_work(&mddev->sync_work); - mddev_unlock(mddev); + wait_event(resync_wait, + !test_bit(MD_RECOVERY_RUNNING, &mddev->recovery) || + (check_seq && sync_seq != atomic_read(&mddev->sync_seq))); + + if (locked) + mddev_lock_nointr(mddev); } static void idle_sync_thread(struct mddev *mddev) { - int sync_seq = atomic_read(&mddev->sync_seq); - mutex_lock(&mddev->sync_mutex); clear_bit(MD_RECOVERY_FROZEN, &mddev->recovery); - stop_sync_thread(mddev); - wait_event(resync_wait, sync_seq != atomic_read(&mddev->sync_seq) || - !test_bit(MD_RECOVERY_RUNNING, &mddev->recovery)); + if (mddev_lock(mddev)) { + mutex_unlock(&mddev->sync_mutex); + return; + } + stop_sync_thread(mddev, false, true); mutex_unlock(&mddev->sync_mutex); } @@ -4880,11 +4997,13 @@ static void frozen_sync_thread(struct mddev *mddev) { mutex_lock(&mddev->sync_mutex); set_bit(MD_RECOVERY_FROZEN, &mddev->recovery); - stop_sync_thread(mddev); - wait_event(resync_wait, mddev->sync_thread == NULL && - !test_bit(MD_RECOVERY_RUNNING, &mddev->recovery)); + if (mddev_lock(mddev)) { + mutex_unlock(&mddev->sync_mutex); + return; + } + stop_sync_thread(mddev, false, false); mutex_unlock(&mddev->sync_mutex); } @@ -6305,14 +6424,7 @@ static void md_clean(struct mddev *mddev) static void __md_stop_writes(struct mddev *mddev) { - set_bit(MD_RECOVERY_FROZEN, &mddev->recovery); - if (work_pending(&mddev->sync_work)) - flush_workqueue(md_misc_wq); - if (mddev->sync_thread) { - set_bit(MD_RECOVERY_INTR, &mddev->recovery); - md_reap_sync_thread(mddev); - } - + stop_sync_thread(mddev, true, false); del_timer_sync(&mddev->safemode_timer); if (mddev->pers && mddev->pers->quiesce) { @@ -6406,18 +6518,8 @@ static int md_set_readonly(struct mddev *mddev, struct block_device *bdev) set_bit(MD_RECOVERY_FROZEN, &mddev->recovery); md_wakeup_thread(mddev->thread); } - if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery)) - set_bit(MD_RECOVERY_INTR, &mddev->recovery); - - /* - * Thread might be blocked waiting for metadata update which will now - * never happen - */ - md_wakeup_thread_directly(mddev->sync_thread); - mddev_unlock(mddev); - wait_event(resync_wait, !test_bit(MD_RECOVERY_RUNNING, - &mddev->recovery)); + stop_sync_thread(mddev, false, false); wait_event(mddev->sb_wait, !test_bit(MD_SB_CHANGE_PENDING, &mddev->sb_flags)); mddev_lock_nointr(mddev); @@ -6471,20 +6573,8 @@ static int do_md_stop(struct mddev *mddev, int mode, set_bit(MD_RECOVERY_FROZEN, &mddev->recovery); md_wakeup_thread(mddev->thread); } - if (test_bit(MD_RECOVERY_RUNNING, &mddev->recovery)) - set_bit(MD_RECOVERY_INTR, &mddev->recovery); - - /* - * Thread might be blocked waiting for metadata update which will now - * never happen - */ - md_wakeup_thread_directly(mddev->sync_thread); - mddev_unlock(mddev); - wait_event(resync_wait, (mddev->sync_thread == NULL && - !test_bit(MD_RECOVERY_RUNNING, - &mddev->recovery))); - mddev_lock_nointr(mddev); + stop_sync_thread(mddev, true, false); mutex_lock(&mddev->open_mutex); if ((mddev->pers && atomic_read(&mddev->openers) > !!bdev) || @@ -8020,31 +8110,29 @@ static int md_thread(void *arg) return 0; } -static void md_wakeup_thread_directly(struct md_thread __rcu *thread) +static void md_wakeup_thread_directly(struct md_thread __rcu **thread) { struct md_thread *t; rcu_read_lock(); - t = rcu_dereference(thread); + t = rcu_dereference(*thread); if (t) wake_up_process(t->tsk); rcu_read_unlock(); } -void md_wakeup_thread(struct md_thread __rcu *thread) +void __md_wakeup_thread(struct md_thread __rcu *thread) { struct md_thread *t; - rcu_read_lock(); t = rcu_dereference(thread); if (t) { pr_debug("md: waking up MD thread %s.\n", t->tsk->comm); set_bit(THREAD_WAKEUP, &t->flags); wake_up(&t->wqueue); } - rcu_read_unlock(); } -EXPORT_SYMBOL(md_wakeup_thread); +EXPORT_SYMBOL(__md_wakeup_thread); struct md_thread *md_register_thread(void (*run) (struct md_thread *), struct mddev *mddev, const char *name) diff --git a/drivers/md/md.h b/drivers/md/md.h index 46995558d3bd..337efde4b3e3 100644 --- a/drivers/md/md.h +++ b/drivers/md/md.h @@ -316,6 +316,7 @@ struct mddev { unsigned long sb_flags; int suspended; + struct mutex suspend_mutex; struct percpu_ref active_io; int ro; int sysfs_active; /* set when sysfs deletes @@ -756,6 +757,12 @@ struct md_io_clone { #define THREAD_WAKEUP 0 +#define md_wakeup_thread(thread) do { \ + rcu_read_lock(); \ + __md_wakeup_thread(thread); \ + rcu_read_unlock(); \ +} while (0) + static inline void safe_put_page(struct page *p) { if (p) put_page(p); @@ -773,7 +780,7 @@ extern struct md_thread *md_register_thread( struct mddev *mddev, const char *name); extern void md_unregister_thread(struct mddev *mddev, struct md_thread __rcu **threadp); -extern void md_wakeup_thread(struct md_thread __rcu *thread); +extern void __md_wakeup_thread(struct md_thread __rcu *thread); extern void md_check_recovery(struct mddev *mddev); extern void md_reap_sync_thread(struct mddev *mddev); extern int mddev_init_writes_pending(struct mddev *mddev); @@ -816,6 +823,8 @@ extern void md_rdev_clear(struct md_rdev *rdev); extern void md_handle_request(struct mddev *mddev, struct bio *bio); extern void mddev_suspend(struct mddev *mddev); extern void mddev_resume(struct mddev *mddev); +extern int __mddev_suspend(struct mddev *mddev, bool interruptible); +extern void __mddev_resume(struct mddev *mddev); extern void md_reload_sb(struct mddev *mddev, int raid_disk); extern void md_update_sb(struct mddev *mddev, int force); @@ -860,6 +869,33 @@ static inline void mddev_check_write_zeroes(struct mddev *mddev, struct bio *bio mddev->queue->limits.max_write_zeroes_sectors = 0; } +static inline int mddev_suspend_and_lock(struct mddev *mddev) +{ + int ret; + + ret = __mddev_suspend(mddev, true); + if (ret) + return ret; + + ret = mddev_lock(mddev); + if (ret) + __mddev_resume(mddev); + + return ret; +} + +static inline void mddev_suspend_and_lock_nointr(struct mddev *mddev) +{ + __mddev_suspend(mddev, false); + mutex_lock(&mddev->reconfig_mutex); +} + +static inline void mddev_unlock_and_resume(struct mddev *mddev) +{ + mddev_unlock(mddev); + __mddev_resume(mddev); +} + struct mdu_array_info_s; struct mdu_disk_info_s; @@ -872,4 +908,9 @@ int do_md_run(struct mddev *mddev); extern const struct block_device_operations md_fops; +static inline bool md_cloned_bio(struct mddev *mddev, struct bio *bio) +{ + return bio->bi_pool == &mddev->io_clone_set; +} + #endif /* _MD_MD_H */ diff --git a/drivers/md/persistent-data/dm-array.c b/drivers/md/persistent-data/dm-array.c index 5e9c17307671..1969d6cfbfa2 100644 --- a/drivers/md/persistent-data/dm-array.c +++ b/drivers/md/persistent-data/dm-array.c @@ -38,6 +38,14 @@ struct array_block { */ #define CSUM_XOR 595846735 +/* + * Each array block can hold this many values. + */ +static uint32_t calc_max_entries(size_t value_size, size_t size_of_block) +{ + return (size_of_block - sizeof(struct array_block)) / value_size; +} + static void array_block_prepare_for_write(struct dm_block_validator *v, struct dm_block *b, size_t size_of_block) @@ -55,6 +63,7 @@ static int array_block_check(struct dm_block_validator *v, size_t size_of_block) { struct array_block *bh_le = dm_block_data(b); + uint32_t nr_entries, max_entries, value_size; __le32 csum_disk; if (dm_block_location(b) != le64_to_cpu(bh_le->blocknr)) { @@ -74,6 +83,26 @@ static int array_block_check(struct dm_block_validator *v, return -EILSEQ; } + nr_entries = le32_to_cpu(bh_le->nr_entries); + max_entries = le32_to_cpu(bh_le->max_entries); + value_size = le32_to_cpu(bh_le->value_size); + + if (!value_size) { + DMERR_LIMIT("%s failed: value_size is zero", __func__); + return -EILSEQ; + } + + if (max_entries != calc_max_entries(value_size, size_of_block)) { + DMERR_LIMIT("%s failed: max_entries %u invalid for value_size %u", + __func__, max_entries, value_size); + return -EILSEQ; + } + + if (nr_entries > max_entries) { + DMERR_LIMIT("%s failed: too many entries", __func__); + return -EILSEQ; + } + return 0; } @@ -138,14 +167,6 @@ static void dec_ablock_entries(struct dm_array_info *info, struct array_block *a on_entries(info, ab, vt->dec); } -/* - * Each array block can hold this many values. - */ -static uint32_t calc_max_entries(size_t value_size, size_t size_of_block) -{ - return (size_of_block - sizeof(struct array_block)) / value_size; -} - /* * Allocate a new array block. The caller will need to unlock block. */ diff --git a/drivers/md/raid1.c b/drivers/md/raid1.c index bd57fc5ad5dd..710727814ea8 100644 --- a/drivers/md/raid1.c +++ b/drivers/md/raid1.c @@ -1220,11 +1220,18 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, char b[BDEVNAME_SIZE]; /* - * If r1_bio is set, we are blocking the raid1d thread - * so there is a tiny risk of deadlock. So ask for + * An md cloned bio indicates we are in the error path. + * This is more reliable than checking r1_bio, which might + * be NULL even in the error path if a failed bio was split. + */ + bool err_path = md_cloned_bio(mddev, bio); + + /* + * If we are in the error path, we are blocking the raid1d + * thread so there is a tiny risk of deadlock. So ask for * emergency memory if needed. */ - gfp_t gfp = r1_bio ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO; + gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO; if (r1bio_existed) { /* Need to get the block device name carefully */ @@ -1307,7 +1314,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, } r1_bio->read_disk = rdisk; - if (!r1bio_existed) { + if (likely(!md_cloned_bio(mddev, bio))) { md_account_bio(mddev, &bio); r1_bio->master_bio = bio; } @@ -1331,7 +1338,7 @@ static void raid1_read_request(struct mddev *mddev, struct bio *bio, submit_bio_noacct(read_bio); } -static void raid1_write_request(struct mddev *mddev, struct bio *bio, +static bool raid1_write_request(struct mddev *mddev, struct bio *bio, int max_write_sectors) { struct r1conf *conf = mddev->private; @@ -1351,7 +1358,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio, DEFINE_WAIT(w); if (bio->bi_opf & REQ_NOWAIT) { bio_wouldblock_error(bio); - return; + return false; } for (;;) { prepare_to_wait(&conf->wait_barrier, @@ -1373,7 +1380,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio, if (!wait_barrier(conf, bio->bi_iter.bi_sector, bio->bi_opf & REQ_NOWAIT)) { bio_wouldblock_error(bio); - return; + return false; } retry_write: @@ -1463,7 +1470,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio, if (bio->bi_opf & REQ_NOWAIT) { bio_wouldblock_error(bio); - return; + return false; } raid1_log(mddev, "wait rdev %d blocked", blocked_rdev->raid_disk); md_wait_for_blocked_rdev(blocked_rdev, mddev); @@ -1565,6 +1572,7 @@ static void raid1_write_request(struct mddev *mddev, struct bio *bio, /* In case raid1d snuck in to freeze_array */ wake_up_barrier(conf); + return true; } static bool raid1_make_request(struct mddev *mddev, struct bio *bio) @@ -1590,7 +1598,8 @@ static bool raid1_make_request(struct mddev *mddev, struct bio *bio) else { if (!md_write_start(mddev,bio)) return false; - raid1_write_request(mddev, bio, sectors); + if (!raid1_write_request(mddev, bio, sectors)) + md_write_end(mddev); } return true; } diff --git a/drivers/md/raid10.c b/drivers/md/raid10.c index ca9ae89c6d1d..2f06b406d914 100644 --- a/drivers/md/raid10.c +++ b/drivers/md/raid10.c @@ -1164,7 +1164,7 @@ static bool regular_request_wait(struct mddev *mddev, struct r10conf *conf, } static void raid10_read_request(struct mddev *mddev, struct bio *bio, - struct r10bio *r10_bio, bool io_accounting) + struct r10bio *r10_bio) { struct r10conf *conf = mddev->private; struct bio *read_bio; @@ -1173,7 +1173,21 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, char b[BDEVNAME_SIZE]; int slot = r10_bio->read_slot; struct md_rdev *err_rdev = NULL; - gfp_t gfp = GFP_NOIO; + + /* + * An md cloned bio indicates we are in the error path. + * This is more reliable than checking slot, which might + * be -1 even in the error path if a failed bio was split. + */ + bool err_path = md_cloned_bio(mddev, bio); + + /* + * If we are in the error path, we are blocking the raid10d + * thread so there is a tiny risk of deadlock. So ask for + * emergency memory if needed. + */ + gfp_t gfp = err_path ? (GFP_NOIO | __GFP_HIGH) : GFP_NOIO; + int error; if (slot >= 0 && r10_bio->devs[slot].rdev) { /* @@ -1184,11 +1198,6 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, * we lose the device name in error messages. */ int disk; - /* - * As we are blocking raid10, it is a little safer to - * use __GFP_HIGH. - */ - gfp = GFP_NOIO | __GFP_HIGH; rcu_read_lock(); disk = r10_bio->devs[slot].devnum; @@ -1226,6 +1235,10 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, if (max_sectors < bio_sectors(bio)) { struct bio *split = bio_split(bio, max_sectors, gfp, &conf->bio_split); + if (IS_ERR(split)) { + error = PTR_ERR(split); + goto err_handle; + } bio_chain(split, bio); allow_barrier(conf); submit_bio_noacct(bio); @@ -1236,7 +1249,7 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, } slot = r10_bio->read_slot; - if (io_accounting) { + if (likely(!md_cloned_bio(mddev, bio))) { md_account_bio(mddev, &bio); r10_bio->master_bio = bio; } @@ -1258,6 +1271,11 @@ static void raid10_read_request(struct mddev *mddev, struct bio *bio, r10_bio->sector); submit_bio_noacct(read_bio); return; +err_handle: + atomic_dec(&rdev->nr_pending); + bio->bi_status = errno_to_blk_status(error); + set_bit(R10BIO_Uptodate, &r10_bio->state); + raid_end_bio_io(r10_bio); } static void raid10_write_one_disk(struct mddev *mddev, struct r10bio *r10_bio, @@ -1394,13 +1412,14 @@ static void wait_blocked_dev(struct mddev *mddev, struct r10bio *r10_bio) } } -static void raid10_write_request(struct mddev *mddev, struct bio *bio, +static bool raid10_write_request(struct mddev *mddev, struct bio *bio, struct r10bio *r10_bio) { struct r10conf *conf = mddev->private; - int i; + int i, k; sector_t sectors; int max_sectors; + int error; if ((mddev_is_clustered(mddev) && md_cluster_ops->area_resyncing(mddev, WRITE, @@ -1410,7 +1429,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio, /* Bail out if REQ_NOWAIT is set for the bio */ if (bio->bi_opf & REQ_NOWAIT) { bio_wouldblock_error(bio); - return; + return false; } for (;;) { prepare_to_wait(&conf->wait_barrier, @@ -1426,7 +1445,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio, sectors = r10_bio->sectors; if (!regular_request_wait(mddev, conf, bio, sectors)) { free_r10bio(r10_bio); - return; + return false; } if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery) && @@ -1443,7 +1462,7 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio, if (bio->bi_opf & REQ_NOWAIT) { allow_barrier(conf); bio_wouldblock_error(bio); - return; + return false; } raid10_log(conf->mddev, "wait reshape metadata"); wait_event(mddev->sb_wait, @@ -1526,6 +1545,10 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio, if (r10_bio->sectors < bio_sectors(bio)) { struct bio *split = bio_split(bio, r10_bio->sectors, GFP_NOIO, &conf->bio_split); + if (IS_ERR(split)) { + error = PTR_ERR(split); + goto err_handle; + } bio_chain(split, bio); allow_barrier(conf); submit_bio_noacct(bio); @@ -1545,10 +1568,33 @@ static void raid10_write_request(struct mddev *mddev, struct bio *bio, raid10_write_one_disk(mddev, r10_bio, bio, true, i); } one_write_done(r10_bio); + return true; + +err_handle: + for (k = 0; k < i; k++) { + int d = r10_bio->devs[k].devnum; + struct md_rdev *rdev = conf->mirrors[d].rdev; + struct md_rdev *rrdev = conf->mirrors[d].replacement; + + if (r10_bio->devs[k].bio) { + rdev_dec_pending(rdev, mddev); + r10_bio->devs[k].bio = NULL; + } + if (r10_bio->devs[k].repl_bio) { + rdev_dec_pending(rrdev, mddev); + r10_bio->devs[k].repl_bio = NULL; + } + } + + bio->bi_status = errno_to_blk_status(error); + set_bit(R10BIO_Uptodate, &r10_bio->state); + raid_end_bio_io(r10_bio); + return false; } -static void __make_request(struct mddev *mddev, struct bio *bio, int sectors) +static bool __make_request(struct mddev *mddev, struct bio *bio, int sectors) { + bool ret; struct r10conf *conf = mddev->private; struct r10bio *r10_bio; @@ -1564,10 +1610,13 @@ static void __make_request(struct mddev *mddev, struct bio *bio, int sectors) memset(r10_bio->devs, 0, sizeof(r10_bio->devs[0]) * conf->geo.raid_disks); + ret = true; if (bio_data_dir(bio) == READ) - raid10_read_request(mddev, bio, r10_bio, true); + raid10_read_request(mddev, bio, r10_bio); else - raid10_write_request(mddev, bio, r10_bio); + ret = raid10_write_request(mddev, bio, r10_bio); + + return ret; } static void raid_end_discard_bio(struct r10bio *r10bio) @@ -1653,11 +1702,11 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio) if (test_bit(MD_RECOVERY_RESHAPE, &mddev->recovery)) return -EAGAIN; - if (WARN_ON_ONCE(bio->bi_opf & REQ_NOWAIT)) { + if (!wait_barrier(conf, bio->bi_opf & REQ_NOWAIT)) { bio_wouldblock_error(bio); + md_write_end(mddev); return 0; } - wait_barrier(conf, false); /* * Check reshape again to avoid reshape happens after checking @@ -1695,6 +1744,11 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio) if (remainder) { split_size = stripe_size - remainder; split = bio_split(bio, split_size, GFP_NOIO, &conf->bio_split); + if (IS_ERR(split)) { + bio->bi_status = errno_to_blk_status(PTR_ERR(split)); + bio_endio(bio); + return 0; + } bio_chain(split, bio); allow_barrier(conf); /* Resend the fist split part */ @@ -1705,6 +1759,11 @@ static int raid10_handle_discard(struct mddev *mddev, struct bio *bio) if (remainder) { split_size = bio_sectors(bio) - remainder; split = bio_split(bio, split_size, GFP_NOIO, &conf->bio_split); + if (IS_ERR(split)) { + bio->bi_status = errno_to_blk_status(PTR_ERR(split)); + bio_endio(bio); + return 0; + } bio_chain(split, bio); allow_barrier(conf); /* Resend the second split part */ @@ -1911,7 +1970,8 @@ static bool raid10_make_request(struct mddev *mddev, struct bio *bio) sectors = chunk_sects - (bio->bi_iter.bi_sector & (chunk_sects - 1)); - __make_request(mddev, bio, sectors); + if (!__make_request(mddev, bio, sectors)) + md_write_end(mddev); /* In case raid10d snuck in to freeze_array */ wake_up_barrier(conf); @@ -2975,7 +3035,7 @@ static void handle_read_error(struct mddev *mddev, struct r10bio *r10_bio) rdev_dec_pending(rdev, mddev); r10_bio->state = 0; - raid10_read_request(mddev, r10_bio->master_bio, r10_bio, false); + raid10_read_request(mddev, r10_bio->master_bio, r10_bio); /* * allow_barrier after re-submit to ensure no sync io * can be issued while regular io pending. diff --git a/drivers/md/raid5.c b/drivers/md/raid5.c index d83b23b12217..ac14655c85d6 100644 --- a/drivers/md/raid5.c +++ b/drivers/md/raid5.c @@ -4916,25 +4916,25 @@ static void break_stripe_batch_list(struct stripe_head *head_sh, sh->check_state = head_sh->check_state; sh->reconstruct_state = head_sh->reconstruct_state; spin_lock_irq(&sh->stripe_lock); - sh->batch_head = NULL; - spin_unlock_irq(&sh->stripe_lock); for (i = 0; i < sh->disks; i++) { if (test_and_clear_bit(R5_Overlap, &sh->dev[i].flags)) do_wakeup = 1; - sh->dev[i].flags = head_sh->dev[i].flags & + sh->dev[i].flags = READ_ONCE(head_sh->dev[i].flags) & (~((1 << R5_WriteError) | (1 << R5_Overlap))); } + sh->batch_head = NULL; + spin_unlock_irq(&sh->stripe_lock); if (handle_flags == 0 || sh->state & handle_flags) set_bit(STRIPE_HANDLE, &sh->state); raid5_release_stripe(sh); } spin_lock_irq(&head_sh->stripe_lock); - head_sh->batch_head = NULL; - spin_unlock_irq(&head_sh->stripe_lock); for (i = 0; i < head_sh->disks; i++) if (test_and_clear_bit(R5_Overlap, &head_sh->dev[i].flags)) do_wakeup = 1; + head_sh->batch_head = NULL; + spin_unlock_irq(&head_sh->stripe_lock); if (head_sh->state & handle_flags) set_bit(STRIPE_HANDLE, &head_sh->state); diff --git a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-core.c b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-core.c index cc44be10fdb7..50d145833934 100644 --- a/drivers/media/platform/mediatek/mdp3/mtk-mdp3-core.c +++ b/drivers/media/platform/mediatek/mdp3/mtk-mdp3-core.c @@ -215,6 +215,11 @@ static int mdp_probe(struct platform_device *pdev) goto err_destroy_clock_wq; } mdp->scp = platform_get_drvdata(mm_pdev); + if (!mdp->scp) { + ret = -EPROBE_DEFER; + goto err_destroy_clock_wq; + } + mdp->rproc_handle = scp_get_rproc(mdp->scp); dev_dbg(&pdev->dev, "MDP rproc_handle: %pK", mdp->rproc_handle); diff --git a/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c b/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c index cb032738ef95..c2a507c3da6e 100644 --- a/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c +++ b/drivers/media/platform/nxp/imx8-isi/imx8-isi-video.c @@ -937,6 +937,49 @@ static void mxc_isi_video_init_channel(struct mxc_isi_video *video) mxc_isi_channel_set_output_format(pipe, video->fmtinfo, &video->pix); } +static int mxc_isi_vb2_prepare_streaming(struct vb2_queue *q) +{ + struct mxc_isi_video *video = vb2_get_drv_priv(q); + struct media_device *mdev = &video->pipe->isi->media_dev; + struct media_pipeline *pipe; + int ret; + + /* Get a pipeline for the video node and start it. */ + scoped_guard(mutex, &mdev->graph_mutex) { + ret = mxc_isi_pipe_acquire(video->pipe, + &mxc_isi_video_frame_write_done); + if (ret) + return ret; + + pipe = media_entity_pipeline(&video->vdev.entity) + ? : &video->pipe->pipe; + + ret = __video_device_pipeline_start(&video->vdev, pipe); + if (ret) + goto err_release; + } + + /* Verify that the video format matches the output of the subdev. */ + ret = mxc_isi_video_validate_format(video); + if (ret) + goto err_stop; + + /* Allocate buffers for discard operation. */ + ret = mxc_isi_video_alloc_discard_buffers(video); + if (ret) + goto err_stop; + + video->is_streaming = true; + + return 0; + +err_stop: + video_device_pipeline_stop(&video->vdev); +err_release: + mxc_isi_pipe_release(video->pipe); + return ret; +} + static int mxc_isi_vb2_start_streaming(struct vb2_queue *q, unsigned int count) { struct mxc_isi_video *video = vb2_get_drv_priv(q); @@ -985,6 +1028,17 @@ static void mxc_isi_vb2_stop_streaming(struct vb2_queue *q) mxc_isi_video_return_buffers(video, VB2_BUF_STATE_ERROR); } +static void mxc_isi_vb2_unprepare_streaming(struct vb2_queue *q) +{ + struct mxc_isi_video *video = vb2_get_drv_priv(q); + + mxc_isi_video_free_discard_buffers(video); + video_device_pipeline_stop(&video->vdev); + mxc_isi_pipe_release(video->pipe); + + video->is_streaming = false; +} + static const struct vb2_ops mxc_isi_vb2_qops = { .queue_setup = mxc_isi_vb2_queue_setup, .buf_init = mxc_isi_vb2_buffer_init, @@ -992,8 +1046,10 @@ static const struct vb2_ops mxc_isi_vb2_qops = { .buf_queue = mxc_isi_vb2_buffer_queue, .wait_prepare = vb2_ops_wait_prepare, .wait_finish = vb2_ops_wait_finish, + .prepare_streaming = mxc_isi_vb2_prepare_streaming, .start_streaming = mxc_isi_vb2_start_streaming, .stop_streaming = mxc_isi_vb2_stop_streaming, + .unprepare_streaming = mxc_isi_vb2_unprepare_streaming, }; /* ----------------------------------------------------------------------------- @@ -1147,97 +1203,6 @@ static int mxc_isi_video_s_fmt(struct file *file, void *priv, return 0; } -static int mxc_isi_video_streamon(struct file *file, void *priv, - enum v4l2_buf_type type) -{ - struct mxc_isi_video *video = video_drvdata(file); - struct media_device *mdev = &video->pipe->isi->media_dev; - struct media_pipeline *pipe; - int ret; - - if (vb2_queue_is_busy(&video->vb2_q, file)) - return -EBUSY; - - /* - * Get a pipeline for the video node and start it. This must be done - * here and not in the queue .start_streaming() handler, so that - * pipeline start errors can be reported from VIDIOC_STREAMON and not - * delayed until subsequent VIDIOC_QBUF calls. - */ - mutex_lock(&mdev->graph_mutex); - - ret = mxc_isi_pipe_acquire(video->pipe, &mxc_isi_video_frame_write_done); - if (ret) { - mutex_unlock(&mdev->graph_mutex); - return ret; - } - - pipe = media_entity_pipeline(&video->vdev.entity) ? : &video->pipe->pipe; - - ret = __video_device_pipeline_start(&video->vdev, pipe); - if (ret) { - mutex_unlock(&mdev->graph_mutex); - goto err_release; - } - - mutex_unlock(&mdev->graph_mutex); - - /* Verify that the video format matches the output of the subdev. */ - ret = mxc_isi_video_validate_format(video); - if (ret) - goto err_stop; - - /* Allocate buffers for discard operation. */ - ret = mxc_isi_video_alloc_discard_buffers(video); - if (ret) - goto err_stop; - - ret = vb2_streamon(&video->vb2_q, type); - if (ret) - goto err_free; - - video->is_streaming = true; - - return 0; - -err_free: - mxc_isi_video_free_discard_buffers(video); -err_stop: - video_device_pipeline_stop(&video->vdev); -err_release: - mxc_isi_pipe_release(video->pipe); - return ret; -} - -static void mxc_isi_video_cleanup_streaming(struct mxc_isi_video *video) -{ - lockdep_assert_held(&video->lock); - - if (!video->is_streaming) - return; - - mxc_isi_video_free_discard_buffers(video); - video_device_pipeline_stop(&video->vdev); - mxc_isi_pipe_release(video->pipe); - - video->is_streaming = false; -} - -static int mxc_isi_video_streamoff(struct file *file, void *priv, - enum v4l2_buf_type type) -{ - struct mxc_isi_video *video = video_drvdata(file); - int ret; - - ret = vb2_ioctl_streamoff(file, priv, type); - if (ret) - return ret; - - mxc_isi_video_cleanup_streaming(video); - - return 0; -} - static int mxc_isi_video_enum_framesizes(struct file *file, void *priv, struct v4l2_frmsizeenum *fsize) { @@ -1293,9 +1258,8 @@ static const struct v4l2_ioctl_ops mxc_isi_video_ioctl_ops = { .vidioc_expbuf = vb2_ioctl_expbuf, .vidioc_prepare_buf = vb2_ioctl_prepare_buf, .vidioc_create_bufs = vb2_ioctl_create_bufs, - - .vidioc_streamon = mxc_isi_video_streamon, - .vidioc_streamoff = mxc_isi_video_streamoff, + .vidioc_streamon = vb2_ioctl_streamon, + .vidioc_streamoff = vb2_ioctl_streamoff, .vidioc_enum_framesizes = mxc_isi_video_enum_framesizes, @@ -1334,10 +1298,6 @@ static int mxc_isi_video_release(struct file *file) if (ret) dev_err(video->pipe->isi->dev, "%s fail\n", __func__); - mutex_lock(&video->lock); - mxc_isi_video_cleanup_streaming(video); - mutex_unlock(&video->lock); - pm_runtime_put(video->pipe->isi->dev); return ret; } diff --git a/drivers/media/platform/verisilicon/hantro.h b/drivers/media/platform/verisilicon/hantro.h index e9e15746f0aa..ce13812a9400 100644 --- a/drivers/media/platform/verisilicon/hantro.h +++ b/drivers/media/platform/verisilicon/hantro.h @@ -77,6 +77,7 @@ struct hantro_irq { * @double_buffer: core needs double buffering * @legacy_regs: core uses legacy register set * @late_postproc: postproc must be set up at the end of the job + * @shared_devices: an array of device ids that cannot run concurrently */ struct hantro_variant { unsigned int enc_offset; @@ -101,6 +102,7 @@ struct hantro_variant { unsigned int double_buffer : 1; unsigned int legacy_regs : 1; unsigned int late_postproc : 1; + const struct of_device_id *shared_devices; }; /** diff --git a/drivers/media/platform/verisilicon/hantro_drv.c b/drivers/media/platform/verisilicon/hantro_drv.c index 35833ee8beb5..6ebab13712af 100644 --- a/drivers/media/platform/verisilicon/hantro_drv.c +++ b/drivers/media/platform/verisilicon/hantro_drv.c @@ -13,6 +13,7 @@ #include #include #include +#include #include #include #include @@ -987,6 +988,84 @@ static const struct media_device_ops hantro_m2m_media_ops = { .req_queue = v4l2_m2m_request_queue, }; +/* + * Some SoCs, like RK3588 have multiple identical Hantro cores, but the + * kernel is currently missing support for multi-core handling. Exposing + * separate devices for each core to userspace is bad, since that does + * not allow scheduling tasks properly (and creates ABI). With this workaround + * the driver will only probe for the first core and early exit for the other + * cores. Once the driver gains multi-core support, the same technique + * for detecting the main core can be used to cluster all cores together. + */ +static int hantro_disable_multicore(struct hantro_dev *vpu) +{ + struct device_node *node = NULL; + const char *compatible; + bool is_main_core; + int ret; + + /* Intentionally ignores the fallback strings */ + ret = of_property_read_string(vpu->dev->of_node, "compatible", &compatible); + if (ret) + return ret; + + /* The first compatible and available node found is considered the main core */ + do { + node = of_find_compatible_node(node, NULL, compatible); + if (of_device_is_available(node)) + break; + } while (node); + + if (!node) + return -EINVAL; + + is_main_core = (vpu->dev->of_node == node); + + of_node_put(node); + + if (!is_main_core) { + dev_info(vpu->dev, "missing multi-core support, ignoring this instance\n"); + return -ENODEV; + } + + return 0; +} + +static struct v4l2_m2m_dev *hantro_get_v4l2_m2m_dev(struct hantro_dev *vpu) +{ + struct device_node *node; + struct hantro_dev *shared_vpu; + + if (!vpu->variant || !vpu->variant->shared_devices) + goto init_new_m2m_dev; + + for_each_matching_node(node, vpu->variant->shared_devices) { + struct platform_device *pdev; + struct v4l2_m2m_dev *m2m_dev; + + pdev = of_find_device_by_node(node); + if (!pdev) + continue; + + shared_vpu = platform_get_drvdata(pdev); + if (IS_ERR_OR_NULL(shared_vpu) || shared_vpu == vpu) { + platform_device_put(pdev); + continue; + } + + v4l2_m2m_get(shared_vpu->m2m_dev); + m2m_dev = shared_vpu->m2m_dev; + platform_device_put(pdev); + + of_node_put(node); + + return m2m_dev; + } + +init_new_m2m_dev: + return v4l2_m2m_init(&vpu_m2m_ops); +} + static int hantro_probe(struct platform_device *pdev) { const struct of_device_id *match; @@ -1006,6 +1085,10 @@ static int hantro_probe(struct platform_device *pdev) match = of_match_node(of_hantro_match, pdev->dev.of_node); vpu->variant = match->data; + ret = hantro_disable_multicore(vpu); + if (ret) + return ret; + /* * Support for nxp,imx8mq-vpu is kept for backwards compatibility * but it's deprecated. Please update your DTS file to use @@ -1134,7 +1217,7 @@ static int hantro_probe(struct platform_device *pdev) } platform_set_drvdata(pdev, vpu); - vpu->m2m_dev = v4l2_m2m_init(&vpu_m2m_ops); + vpu->m2m_dev = hantro_get_v4l2_m2m_dev(vpu); if (IS_ERR(vpu->m2m_dev)) { v4l2_err(&vpu->v4l2_dev, "Failed to init mem2mem device\n"); ret = PTR_ERR(vpu->m2m_dev); @@ -1173,7 +1256,7 @@ static int hantro_probe(struct platform_device *pdev) hantro_remove_enc_func(vpu); err_m2m_rel: media_device_cleanup(&vpu->mdev); - v4l2_m2m_release(vpu->m2m_dev); + v4l2_m2m_put(vpu->m2m_dev); err_v4l2_unreg: v4l2_device_unregister(&vpu->v4l2_dev); err_clk_unprepare: @@ -1196,7 +1279,7 @@ static void hantro_remove(struct platform_device *pdev) hantro_remove_dec_func(vpu); hantro_remove_enc_func(vpu); media_device_cleanup(&vpu->mdev); - v4l2_m2m_release(vpu->m2m_dev); + v4l2_m2m_put(vpu->m2m_dev); v4l2_device_unregister(&vpu->v4l2_dev); clk_bulk_unprepare(vpu->variant->num_clocks, vpu->clocks); reset_control_assert(vpu->resets); diff --git a/drivers/media/platform/verisilicon/imx8m_vpu_hw.c b/drivers/media/platform/verisilicon/imx8m_vpu_hw.c index 74fd985a8aad..cdaac2f18fb5 100644 --- a/drivers/media/platform/verisilicon/imx8m_vpu_hw.c +++ b/drivers/media/platform/verisilicon/imx8m_vpu_hw.c @@ -361,6 +361,12 @@ const struct hantro_variant imx8mq_vpu_variant = { .num_regs = ARRAY_SIZE(imx8mq_reg_names) }; +static const struct of_device_id imx8mq_vpu_shared_resources[] __initconst = { + { .compatible = "nxp,imx8mq-vpu-g1", }, + { .compatible = "nxp,imx8mq-vpu-g2", }, + { /* sentinel */ } +}; + const struct hantro_variant imx8mq_vpu_g1_variant = { .dec_fmts = imx8m_vpu_dec_fmts, .num_dec_fmts = ARRAY_SIZE(imx8m_vpu_dec_fmts), @@ -374,6 +380,7 @@ const struct hantro_variant imx8mq_vpu_g1_variant = { .num_irqs = ARRAY_SIZE(imx8mq_irqs), .clk_names = imx8mq_g1_clk_names, .num_clocks = ARRAY_SIZE(imx8mq_g1_clk_names), + .shared_devices = imx8mq_vpu_shared_resources, }; const struct hantro_variant imx8mq_vpu_g2_variant = { @@ -389,6 +396,7 @@ const struct hantro_variant imx8mq_vpu_g2_variant = { .num_irqs = ARRAY_SIZE(imx8mq_g2_irqs), .clk_names = imx8mq_g2_clk_names, .num_clocks = ARRAY_SIZE(imx8mq_g2_clk_names), + .shared_devices = imx8mq_vpu_shared_resources, }; const struct hantro_variant imx8mm_vpu_g1_variant = { diff --git a/drivers/media/test-drivers/vivid/vivid-core.c b/drivers/media/test-drivers/vivid/vivid-core.c index e95bdccfc18e..761f54b9f7a3 100644 --- a/drivers/media/test-drivers/vivid/vivid-core.c +++ b/drivers/media/test-drivers/vivid/vivid-core.c @@ -2125,8 +2125,10 @@ static int __init vivid_init(void) int ret; ret = platform_device_register(&vivid_pdev); - if (ret) + if (ret) { + platform_device_put(&vivid_pdev); return ret; + } ret = platform_driver_register(&vivid_pdrv); if (ret) diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c b/drivers/media/v4l2-core/v4l2-mem2mem.c index 8db9ac9c1433..494ddd7e142c 100644 --- a/drivers/media/v4l2-core/v4l2-mem2mem.c +++ b/drivers/media/v4l2-core/v4l2-mem2mem.c @@ -90,6 +90,7 @@ static const char * const m2m_entity_name[] = { * @job_work: worker to run queued jobs. * @job_queue_flags: flags of the queue status, %QUEUE_PAUSED. * @m2m_ops: driver callbacks + * @kref: device reference count */ struct v4l2_m2m_dev { struct v4l2_m2m_ctx *curr_ctx; @@ -109,6 +110,8 @@ struct v4l2_m2m_dev { unsigned long job_queue_flags; const struct v4l2_m2m_ops *m2m_ops; + + struct kref kref; }; static struct v4l2_m2m_queue_ctx *get_queue_ctx(struct v4l2_m2m_ctx *m2m_ctx, @@ -1207,6 +1210,7 @@ struct v4l2_m2m_dev *v4l2_m2m_init(const struct v4l2_m2m_ops *m2m_ops) INIT_LIST_HEAD(&m2m_dev->job_queue); spin_lock_init(&m2m_dev->job_spinlock); INIT_WORK(&m2m_dev->job_work, v4l2_m2m_device_run_work); + kref_init(&m2m_dev->kref); return m2m_dev; } @@ -1218,6 +1222,25 @@ void v4l2_m2m_release(struct v4l2_m2m_dev *m2m_dev) } EXPORT_SYMBOL_GPL(v4l2_m2m_release); +void v4l2_m2m_get(struct v4l2_m2m_dev *m2m_dev) +{ + kref_get(&m2m_dev->kref); +} +EXPORT_SYMBOL_GPL(v4l2_m2m_get); + +static void v4l2_m2m_release_from_kref(struct kref *kref) +{ + struct v4l2_m2m_dev *m2m_dev = container_of(kref, struct v4l2_m2m_dev, kref); + + v4l2_m2m_release(m2m_dev); +} + +void v4l2_m2m_put(struct v4l2_m2m_dev *m2m_dev) +{ + kref_put(&m2m_dev->kref, v4l2_m2m_release_from_kref); +} +EXPORT_SYMBOL_GPL(v4l2_m2m_put); + struct v4l2_m2m_ctx *v4l2_m2m_ctx_init(struct v4l2_m2m_dev *m2m_dev, void *drv_priv, int (*queue_init)(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq)) diff --git a/drivers/memstick/core/memstick.c b/drivers/memstick/core/memstick.c index 65af43201b69..e0895e979e35 100644 --- a/drivers/memstick/core/memstick.c +++ b/drivers/memstick/core/memstick.c @@ -367,9 +367,7 @@ int memstick_set_rw_addr(struct memstick_dev *card) { card->next_request = h_memstick_set_rw_addr; memstick_new_req(card->host); - if (!wait_for_completion_timeout(&card->mrq_complete, - msecs_to_jiffies(500))) - card->current_mrq.error = -ETIMEDOUT; + wait_for_completion(&card->mrq_complete); return card->current_mrq.error; } @@ -403,9 +401,7 @@ static struct memstick_dev *memstick_alloc_card(struct memstick_host *host) card->next_request = h_memstick_read_dev_id; memstick_new_req(host); - if (!wait_for_completion_timeout(&card->mrq_complete, - msecs_to_jiffies(500))) - card->current_mrq.error = -ETIMEDOUT; + wait_for_completion(&card->mrq_complete); if (card->current_mrq.error) goto err_out; diff --git a/drivers/memstick/host/rtsx_usb_ms.c b/drivers/memstick/host/rtsx_usb_ms.c index 43ec4948daa2..4c90bf886684 100644 --- a/drivers/memstick/host/rtsx_usb_ms.c +++ b/drivers/memstick/host/rtsx_usb_ms.c @@ -27,7 +27,6 @@ struct rtsx_usb_ms { struct memstick_host *msh; struct memstick_request *req; - struct mutex host_mutex; struct work_struct handle_req; struct delayed_work poll_card; @@ -511,6 +510,13 @@ static void rtsx_usb_ms_handle_req(struct work_struct *work) struct memstick_host *msh = host->msh; int rc; + /* Fail requests after eject so their waiters are released. */ + if (host->eject) { + while (!memstick_next_req(msh, &host->req)) + host->req->error = -ENOMEDIUM; + return; + } + if (!host->req) { pm_runtime_get_sync(ms_dev(host)); do { @@ -544,8 +550,7 @@ static void rtsx_usb_ms_request(struct memstick_host *msh) dev_dbg(ms_dev(host), "--> %s\n", __func__); - if (!host->eject) - schedule_work(&host->handle_req); + schedule_work(&host->handle_req); } static int rtsx_usb_ms_set_param(struct memstick_host *msh, @@ -778,7 +783,6 @@ static int rtsx_usb_ms_drv_probe(struct platform_device *pdev) host->power_mode = MEMSTICK_POWER_OFF; platform_set_drvdata(pdev, host); - mutex_init(&host->host_mutex); INIT_WORK(&host->handle_req, rtsx_usb_ms_handle_req); INIT_DELAYED_WORK(&host->poll_card, rtsx_usb_ms_poll_card); @@ -809,27 +813,12 @@ static int rtsx_usb_ms_drv_remove(struct platform_device *pdev) { struct rtsx_usb_ms *host = platform_get_drvdata(pdev); struct memstick_host *msh = host->msh; - int err; host->eject = true; msh->removing = true; - cancel_work_sync(&host->handle_req); + flush_work(&host->handle_req); cancel_delayed_work_sync(&host->poll_card); - mutex_lock(&host->host_mutex); - if (host->req) { - dev_dbg(ms_dev(host), - "%s: Controller removed during transfer\n", - dev_name(&msh->dev)); - host->req->error = -ENOMEDIUM; - do { - err = memstick_next_req(msh, &host->req); - if (!err) - host->req->error = -ENOMEDIUM; - } while (!err); - } - mutex_unlock(&host->host_mutex); - /* Balance possible unbalanced usage count * e.g. unconditional module removal */ @@ -838,6 +827,8 @@ static int rtsx_usb_ms_drv_remove(struct platform_device *pdev) pm_runtime_disable(ms_dev(host)); memstick_remove_host(msh); + /* No card, no new requests; wait for the last failed one to finish. */ + cancel_work_sync(&host->handle_req); dev_dbg(ms_dev(host), ": Realtek USB Memstick controller has been removed\n"); memstick_free_host(msh); diff --git a/drivers/mfd/sm501.c b/drivers/mfd/sm501.c index 1865621d62f2..5752099683eb 100644 --- a/drivers/mfd/sm501.c +++ b/drivers/mfd/sm501.c @@ -1667,9 +1667,10 @@ static void sm501_pci_remove(struct pci_dev *dev) release_mem_region(sm->io_res->start, 0x100); pci_disable_device(dev); + kfree(sm); } -static int sm501_plat_remove(struct platform_device *dev) +static void sm501_plat_remove(struct platform_device *dev) { struct sm501_devdata *sm = platform_get_drvdata(dev); @@ -1677,8 +1678,7 @@ static int sm501_plat_remove(struct platform_device *dev) iounmap(sm->regs); release_mem_region(sm->io_res->start, 0x100); - - return 0; + kfree(sm); } static const struct pci_device_id sm501_pci_tbl[] = { @@ -1709,7 +1709,7 @@ static struct platform_driver sm501_plat_driver = { .of_match_table = of_sm501_match_tbl, }, .probe = sm501_plat_probe, - .remove = sm501_plat_remove, + .remove_new = sm501_plat_remove, .suspend = pm_sleep_ptr(sm501_plat_suspend), .resume = pm_sleep_ptr(sm501_plat_resume), }; diff --git a/drivers/misc/ocxl/file.c b/drivers/misc/ocxl/file.c index 6e63f060e4cc..51766db296ab 100644 --- a/drivers/misc/ocxl/file.c +++ b/drivers/misc/ocxl/file.c @@ -185,7 +185,7 @@ static irqreturn_t irq_handler(void *private) { struct eventfd_ctx *ev_ctx = private; - eventfd_signal(ev_ctx, 1); + eventfd_signal(ev_ctx); return IRQ_HANDLED; } diff --git a/drivers/misc/sgi-gru/grufault.c b/drivers/misc/sgi-gru/grufault.c index 629edb6486de..5a87c12f444a 100644 --- a/drivers/misc/sgi-gru/grufault.c +++ b/drivers/misc/sgi-gru/grufault.c @@ -166,13 +166,8 @@ static void get_clear_fault_map(struct gru_state *gru, } /* - * Atomic (interrupt context) & non-atomic (user context) functions to - * convert a vaddr into a physical address. The size of the page - * is returned in pageshift. - * returns: - * 0 - successful - * < 0 - error code - * 1 - (atomic only) try again in non-atomic context + * Convert a user virtual address to a physical address in process context. + * The size of the page is returned in pageshift. */ static int non_atomic_pte_lookup(struct vm_area_struct *vma, unsigned long vaddr, int write, @@ -192,87 +187,25 @@ static int non_atomic_pte_lookup(struct vm_area_struct *vma, return 0; } -/* - * atomic_pte_lookup - * - * Convert a user virtual address to a physical address - * Only supports Intel large pages (2MB only) on x86_64. - * ZZZ - hugepage support is incomplete - * - * NOTE: mmap_lock is already held on entry to this function. This - * guarantees existence of the page tables. - */ -static int atomic_pte_lookup(struct vm_area_struct *vma, unsigned long vaddr, - int write, unsigned long *paddr, int *pageshift) -{ - pgd_t *pgdp; - p4d_t *p4dp; - pud_t *pudp; - pmd_t *pmdp; - pte_t pte; - - pgdp = pgd_offset(vma->vm_mm, vaddr); - if (unlikely(pgd_none(*pgdp))) - goto err; - - p4dp = p4d_offset(pgdp, vaddr); - if (unlikely(p4d_none(*p4dp))) - goto err; - - pudp = pud_offset(p4dp, vaddr); - if (unlikely(pud_none(*pudp))) - goto err; - - pmdp = pmd_offset(pudp, vaddr); - if (unlikely(pmd_none(*pmdp))) - goto err; -#ifdef CONFIG_X86_64 - if (unlikely(pmd_large(*pmdp))) - pte = ptep_get((pte_t *)pmdp); - else -#endif - pte = *pte_offset_kernel(pmdp, vaddr); - - if (unlikely(!pte_present(pte) || - (write && (!pte_write(pte) || !pte_dirty(pte))))) - return 1; - - *paddr = pte_pfn(pte) << PAGE_SHIFT; -#ifdef CONFIG_HUGETLB_PAGE - *pageshift = is_vm_hugetlb_page(vma) ? HPAGE_SHIFT : PAGE_SHIFT; -#else - *pageshift = PAGE_SHIFT; -#endif - return 0; - -err: - return 1; -} - static int gru_vtop(struct gru_thread_state *gts, unsigned long vaddr, int write, int atomic, unsigned long *gpa, int *pageshift) { struct mm_struct *mm = gts->ts_mm; struct vm_area_struct *vma; unsigned long paddr; - int ret, ps; + int ps; vma = find_vma(mm, vaddr); if (!vma) goto inval; - /* - * Atomic lookup is faster & usually works even if called in non-atomic - * context. - */ - rmb(); /* Must/check ms_range_active before loading PTEs */ - ret = atomic_pte_lookup(vma, vaddr, write, &paddr, &ps); - if (ret) { - if (atomic) - goto upm; - if (non_atomic_pte_lookup(vma, vaddr, write, &paddr, &ps)) - goto inval; - } + if (atomic) + goto upm; + + /* Order the caller's ms_range_active check before loading PTEs. */ + rmb(); + if (non_atomic_pte_lookup(vma, vaddr, write, &paddr, &ps)) + goto inval; if (is_gru_paddr(paddr)) goto inval; paddr = paddr & ~((1UL << ps) - 1); @@ -569,19 +502,9 @@ static irqreturn_t gru_intr(int chiplet, int blade) continue; } - /* - * This is running in interrupt context. Trylock the mmap_lock. - * If it fails, retry the fault in user context. - */ + /* Address translation may sleep, so retry the fault in user context. */ gts->ustats.fmm_tlbmiss++; - if (!gts->ts_force_cch_reload && - mmap_read_trylock(gts->ts_mm)) { - gru_try_dropin(gru, gts, tfh, NULL); - mmap_read_unlock(gts->ts_mm); - } else { - tfh_user_polling_mode(tfh); - STAT(intr_mm_lock_failed); - } + tfh_user_polling_mode(tfh); } return IRQ_HANDLED; } diff --git a/drivers/misc/sgi-gru/gruprocfs.c b/drivers/misc/sgi-gru/gruprocfs.c index 97b8b38ab47d..b8139c27bc7f 100644 --- a/drivers/misc/sgi-gru/gruprocfs.c +++ b/drivers/misc/sgi-gru/gruprocfs.c @@ -54,7 +54,6 @@ static int statistics_show(struct seq_file *s, void *p) printstat(s, intr_cbr); printstat(s, intr_tfh); printstat(s, intr_spurious); - printstat(s, intr_mm_lock_failed); printstat(s, call_os); printstat(s, call_os_wait_queue); printstat(s, user_flush_tlb); diff --git a/drivers/misc/sgi-gru/grutables.h b/drivers/misc/sgi-gru/grutables.h index 640daf1994df..3348552925c6 100644 --- a/drivers/misc/sgi-gru/grutables.h +++ b/drivers/misc/sgi-gru/grutables.h @@ -182,7 +182,6 @@ struct gru_stats_s { atomic_long_t intr_cbr; atomic_long_t intr_tfh; atomic_long_t intr_spurious; - atomic_long_t intr_mm_lock_failed; atomic_long_t call_os; atomic_long_t call_os_wait_queue; atomic_long_t user_flush_tlb; diff --git a/drivers/mmc/host/cavium-octeon.c b/drivers/mmc/host/cavium-octeon.c index 060ec4f4800f..e36d40f2b718 100644 --- a/drivers/mmc/host/cavium-octeon.c +++ b/drivers/mmc/host/cavium-octeon.c @@ -302,9 +302,12 @@ static void octeon_mmc_remove(struct platform_device *pdev) u64 dma_cfg; int i; - for (i = 0; i < CAVIUM_MAX_MMC; i++) + for (i = 0; i < CAVIUM_MAX_MMC; i++) { if (host->slot[i]) cvm_mmc_of_slot_remove(host->slot[i]); + if (host->slot_pdev[i]) + of_platform_device_destroy(&host->slot_pdev[i]->dev, NULL); + } dma_cfg = readq(host->dma_base + MIO_EMM_DMA_CFG(host)); dma_cfg &= ~MIO_EMM_DMA_CFG_EN; diff --git a/drivers/mmc/host/cavium-thunderx.c b/drivers/mmc/host/cavium-thunderx.c index 2e2ff984f0b3..60f5507ef8cb 100644 --- a/drivers/mmc/host/cavium-thunderx.c +++ b/drivers/mmc/host/cavium-thunderx.c @@ -174,9 +174,12 @@ static void thunder_mmc_remove(struct pci_dev *pdev) u64 dma_cfg; int i; - for (i = 0; i < CAVIUM_MAX_MMC; i++) + for (i = 0; i < CAVIUM_MAX_MMC; i++) { if (host->slot[i]) cvm_mmc_of_slot_remove(host->slot[i]); + if (host->slot_pdev[i]) + of_platform_device_destroy(&host->slot_pdev[i]->dev, NULL); + } dma_cfg = readq(host->dma_base + MIO_EMM_DMA_CFG(host)); dma_cfg &= ~MIO_EMM_DMA_CFG_EN; diff --git a/drivers/mmc/host/mtk-sd.c b/drivers/mmc/host/mtk-sd.c index cf685c0a17ed..3a84f8b968e0 100644 --- a/drivers/mmc/host/mtk-sd.c +++ b/drivers/mmc/host/mtk-sd.c @@ -2912,6 +2912,7 @@ static void msdc_drv_remove(struct platform_device *pdev) platform_set_drvdata(pdev, NULL); mmc_remove_host(mmc); + cancel_delayed_work_sync(&host->req_timeout); msdc_deinit_hw(host); msdc_gate_clock(host); diff --git a/drivers/mmc/host/sdhci-sprd.c b/drivers/mmc/host/sdhci-sprd.c index bed57a1c64b5..5e7aef47be6f 100644 --- a/drivers/mmc/host/sdhci-sprd.c +++ b/drivers/mmc/host/sdhci-sprd.c @@ -903,6 +903,10 @@ static void sdhci_sprd_remove(struct platform_device *pdev) struct sdhci_host *host = platform_get_drvdata(pdev); struct sdhci_sprd_host *sprd_host = TO_SPRD_HOST(host); + pm_runtime_get_sync(&pdev->dev); + pm_runtime_disable(&pdev->dev); + pm_runtime_put_noidle(&pdev->dev); + sdhci_remove_host(host, 0); clk_disable_unprepare(sprd_host->clk_sdio); diff --git a/drivers/mmc/host/via-sdmmc.c b/drivers/mmc/host/via-sdmmc.c index cac4b894f5f8..6634b04a7545 100644 --- a/drivers/mmc/host/via-sdmmc.c +++ b/drivers/mmc/host/via-sdmmc.c @@ -1207,6 +1207,10 @@ static void via_sd_remove(struct pci_dev *pcidev) free_irq(pcidev->irq, sdhost); + cancel_work_sync(&sdhost->carddet_work); + /* carddet_work may re-enable the interrupt via via_reset_pcictrl(). */ + writeb(0x0, sdhost->pcictrl_mmiobase + VIA_CRDR_PCIINTCTRL); + del_timer_sync(&sdhost->timer); tasklet_kill(&sdhost->finish_tasklet); diff --git a/drivers/mmc/host/vub300.c b/drivers/mmc/host/vub300.c index b420e4d82cf4..678115a08f07 100644 --- a/drivers/mmc/host/vub300.c +++ b/drivers/mmc/host/vub300.c @@ -2343,11 +2343,16 @@ static int vub300_probe(struct usb_interface *interface, interface_to_InterfaceNumber(interface)); retval = mmc_add_host(mmc); if (retval) - goto error6; + goto err_stop_io; return 0; -error6: - del_timer_sync(&vub300->inactivity_timer); + +err_stop_io: + vub300->interface = NULL; + kref_put(&vub300->kref, vub300_delete); + + return retval; + error5: mmc_free_host(mmc); /* diff --git a/drivers/most/core.c b/drivers/most/core.c index 5d073b3d2796..3d1102faec8a 100644 --- a/drivers/most/core.c +++ b/drivers/most/core.c @@ -1282,19 +1282,28 @@ int most_register_interface(struct most_interface *iface) int id; struct most_channel *c; - if (!iface || !iface->enqueue || !iface->configure || - !iface->poison_channel || (iface->num_channels > MAX_CHANNELS)) + if (!iface) return -EINVAL; + device_initialize(iface->dev); + + if (!iface->enqueue || !iface->configure || !iface->poison_channel || + (iface->num_channels > MAX_CHANNELS)) { + put_device(iface->dev); + return -EINVAL; + } + id = ida_simple_get(&mdev_id, 0, 0, GFP_KERNEL); if (id < 0) { dev_err(iface->dev, "Failed to allocate device ID\n"); + put_device(iface->dev); return id; } iface->p = kzalloc(sizeof(*iface->p), GFP_KERNEL); if (!iface->p) { ida_simple_remove(&mdev_id, id); + put_device(iface->dev); return -ENOMEM; } @@ -1304,7 +1313,7 @@ int most_register_interface(struct most_interface *iface) iface->dev->bus = &mostbus; iface->dev->groups = interface_attr_groups; dev_set_drvdata(iface->dev, iface); - if (device_register(iface->dev)) { + if (device_add(iface->dev)) { dev_err(iface->dev, "Failed to register interface device\n"); kfree(iface->p); put_device(iface->dev); diff --git a/drivers/mtd/devices/block2mtd.c b/drivers/mtd/devices/block2mtd.c index be106dc20ff3..1eb6bcab7adc 100644 --- a/drivers/mtd/devices/block2mtd.c +++ b/drivers/mtd/devices/block2mtd.c @@ -37,7 +37,7 @@ /* Info for the block device */ struct block2mtd_dev { struct list_head list; - struct block_device *blkdev; + struct bdev_handle *bdev_handle; struct mtd_info mtd; struct mutex write_mutex; }; @@ -55,7 +55,8 @@ static struct page *page_read(struct address_space *mapping, pgoff_t index) /* erase a specified part of the device */ static int _block2mtd_erase(struct block2mtd_dev *dev, loff_t to, size_t len) { - struct address_space *mapping = dev->blkdev->bd_inode->i_mapping; + struct address_space *mapping = + dev->bdev_handle->bdev->bd_inode->i_mapping; struct page *page; pgoff_t index = to >> PAGE_SHIFT; // page index int pages = len >> PAGE_SHIFT; @@ -105,6 +106,8 @@ static int block2mtd_read(struct mtd_info *mtd, loff_t from, size_t len, size_t *retlen, u_char *buf) { struct block2mtd_dev *dev = mtd->priv; + struct address_space *mapping = + dev->bdev_handle->bdev->bd_inode->i_mapping; struct page *page; pgoff_t index = from >> PAGE_SHIFT; int offset = from & (PAGE_SIZE-1); @@ -117,7 +120,7 @@ static int block2mtd_read(struct mtd_info *mtd, loff_t from, size_t len, cpylen = len; // this page len = len - cpylen; - page = page_read(dev->blkdev->bd_inode->i_mapping, index); + page = page_read(mapping, index); if (IS_ERR(page)) return PTR_ERR(page); @@ -139,7 +142,8 @@ static int _block2mtd_write(struct block2mtd_dev *dev, const u_char *buf, loff_t to, size_t len, size_t *retlen) { struct page *page; - struct address_space *mapping = dev->blkdev->bd_inode->i_mapping; + struct address_space *mapping = + dev->bdev_handle->bdev->bd_inode->i_mapping; pgoff_t index = to >> PAGE_SHIFT; // page index int offset = to & ~PAGE_MASK; // page offset int cpylen; @@ -194,7 +198,7 @@ static int block2mtd_write(struct mtd_info *mtd, loff_t to, size_t len, static void block2mtd_sync(struct mtd_info *mtd) { struct block2mtd_dev *dev = mtd->priv; - sync_blockdev(dev->blkdev); + sync_blockdev(dev->bdev_handle->bdev); return; } @@ -206,10 +210,10 @@ static void block2mtd_free_device(struct block2mtd_dev *dev) kfree(dev->mtd.name); - if (dev->blkdev) { - invalidate_mapping_pages(dev->blkdev->bd_inode->i_mapping, - 0, -1); - blkdev_put(dev->blkdev, NULL); + if (dev->bdev_handle) { + invalidate_mapping_pages( + dev->bdev_handle->bdev->bd_inode->i_mapping, 0, -1); + bdev_release(dev->bdev_handle); } kfree(dev); @@ -219,10 +223,10 @@ static void block2mtd_free_device(struct block2mtd_dev *dev) * This function is marked __ref because it calls the __init marked * early_lookup_bdev when called from the early boot code. */ -static struct block_device __ref *mdtblock_early_get_bdev(const char *devname, +static struct bdev_handle __ref *mdtblock_early_get_bdev(const char *devname, blk_mode_t mode, int timeout, struct block2mtd_dev *dev) { - struct block_device *bdev = ERR_PTR(-ENODEV); + struct bdev_handle *bdev_handle = ERR_PTR(-ENODEV); #ifndef MODULE int i; @@ -230,7 +234,7 @@ static struct block_device __ref *mdtblock_early_get_bdev(const char *devname, * We can't use early_lookup_bdev from a running system. */ if (system_state >= SYSTEM_RUNNING) - return bdev; + return bdev_handle; /* * We might not have the root device mounted at this point. @@ -249,21 +253,23 @@ static struct block_device __ref *mdtblock_early_get_bdev(const char *devname, wait_for_device_probe(); if (!early_lookup_bdev(devname, &devt)) { - bdev = blkdev_get_by_dev(devt, mode, dev, NULL); - if (!IS_ERR(bdev)) + bdev_handle = bdev_open_by_dev(devt, mode, dev, NULL); + if (!IS_ERR(bdev_handle)) break; } } #endif - return bdev; + return bdev_handle; } static struct block2mtd_dev *add_device(char *devname, int erase_size, char *label, int timeout) { const blk_mode_t mode = BLK_OPEN_READ | BLK_OPEN_WRITE; + struct bdev_handle *bdev_handle; struct block_device *bdev; struct block2mtd_dev *dev; + loff_t size; char *name; if (!devname) @@ -274,21 +280,24 @@ static struct block2mtd_dev *add_device(char *devname, int erase_size, return NULL; /* Get a handle on the device */ - bdev = blkdev_get_by_path(devname, mode, dev, NULL); - if (IS_ERR(bdev)) - bdev = mdtblock_early_get_bdev(devname, mode, timeout, dev); - if (IS_ERR(bdev)) { + bdev_handle = bdev_open_by_path(devname, mode, dev, NULL); + if (IS_ERR(bdev_handle)) + bdev_handle = mdtblock_early_get_bdev(devname, mode, timeout, + dev); + if (IS_ERR(bdev_handle)) { pr_err("error: cannot open device %s\n", devname); goto err_free_block2mtd; } - dev->blkdev = bdev; + dev->bdev_handle = bdev_handle; + bdev = bdev_handle->bdev; if (MAJOR(bdev->bd_dev) == MTD_BLOCK_MAJOR) { pr_err("attempting to use an MTD device as a block device\n"); goto err_free_block2mtd; } - if ((long)dev->blkdev->bd_inode->i_size % erase_size) { + size = bdev_nr_bytes(bdev); + if (!erase_size || (long)size % erase_size) { pr_err("erasesize must be a divisor of device size\n"); goto err_free_block2mtd; } @@ -306,7 +315,7 @@ static struct block2mtd_dev *add_device(char *devname, int erase_size, dev->mtd.name = name; - dev->mtd.size = dev->blkdev->bd_inode->i_size & PAGE_MASK; + dev->mtd.size = size & PAGE_MASK; dev->mtd.erasesize = erase_size; dev->mtd.writesize = 1; dev->mtd.writebufsize = PAGE_SIZE; diff --git a/drivers/mtd/maps/vmu-flash.c b/drivers/mtd/maps/vmu-flash.c index 3667faefe6c5..7b439b8167d8 100644 --- a/drivers/mtd/maps/vmu-flash.c +++ b/drivers/mtd/maps/vmu-flash.c @@ -610,7 +610,7 @@ static int vmu_connect(struct maple_device *mdev) basic_flash_data = be32_to_cpu(mdev->devinfo.function_data[c - 1]); - card = kmalloc(sizeof(struct memcard), GFP_KERNEL); + card = kzalloc(sizeof(struct memcard), GFP_KERNEL); if (!card) { error = -ENOMEM; goto fail_nomem; @@ -628,15 +628,15 @@ static int vmu_connect(struct maple_device *mdev) * Not sure there are actually any multi-partition devices in the * real world, but the hardware supports them, so, so will we */ - card->parts = kmalloc_array(card->partitions, sizeof(struct vmupart), - GFP_KERNEL); + card->parts = kcalloc(card->partitions, sizeof(struct vmupart), + GFP_KERNEL); if (!card->parts) { error = -ENOMEM; goto fail_partitions; } - card->mtd = kmalloc_array(card->partitions, sizeof(struct mtd_info), - GFP_KERNEL); + card->mtd = kcalloc(card->partitions, sizeof(struct mtd_info), + GFP_KERNEL); if (!card->mtd) { error = -ENOMEM; goto fail_mtd_info; diff --git a/drivers/mtd/mtdcore.c b/drivers/mtd/mtdcore.c index 506ebd4ba1b4..114127e08739 100644 --- a/drivers/mtd/mtdcore.c +++ b/drivers/mtd/mtdcore.c @@ -1255,7 +1255,7 @@ int __get_mtd_device(struct mtd_info *mtd) int err; if (master->_get_device) { - err = master->_get_device(mtd); + err = master->_get_device(master); if (err) return err; } diff --git a/drivers/mtd/nand/raw/cadence-nand-controller.c b/drivers/mtd/nand/raw/cadence-nand-controller.c index 06edf52c26c7..36f582014a1d 100644 --- a/drivers/mtd/nand/raw/cadence-nand-controller.c +++ b/drivers/mtd/nand/raw/cadence-nand-controller.c @@ -2893,6 +2893,9 @@ static int cadence_nand_init(struct cdns_nand_ctrl *cdns_ctrl) goto free_buf_desc; } + spin_lock_init(&cdns_ctrl->irq_lock); + init_completion(&cdns_ctrl->complete); + if (devm_request_irq(cdns_ctrl->dev, cdns_ctrl->irq, cadence_nand_isr, IRQF_SHARED, "cadence-nand-controller", cdns_ctrl)) { @@ -2901,9 +2904,6 @@ static int cadence_nand_init(struct cdns_nand_ctrl *cdns_ctrl) goto free_buf; } - spin_lock_init(&cdns_ctrl->irq_lock); - init_completion(&cdns_ctrl->complete); - ret = cadence_nand_hw_init(cdns_ctrl); if (ret) goto disable_irq; diff --git a/drivers/mtd/nand/raw/vf610_nfc.c b/drivers/mtd/nand/raw/vf610_nfc.c index 3f783b8f76c9..243c43c3dc02 100644 --- a/drivers/mtd/nand/raw/vf610_nfc.c +++ b/drivers/mtd/nand/raw/vf610_nfc.c @@ -504,6 +504,11 @@ static int vf610_nfc_exec_op(struct nand_chip *chip, check_only); } +static unsigned int vf610_nfc_spare_size(struct mtd_info *mtd) +{ + return min_t(unsigned int, mtd->oobsize, 64); +} + static inline int vf610_nfc_correct_data(struct nand_chip *chip, uint8_t *dat, uint8_t *oob, int page) { @@ -513,6 +518,7 @@ static inline int vf610_nfc_correct_data(struct nand_chip *chip, uint8_t *dat, u8 ecc_status; u8 ecc_count; int flips_threshold = nfc->chip.ecc.strength / 2; + int ret; ecc_status = vf610_nfc_read(nfc, ecc_status_off) & 0xff; ecc_count = ecc_status & ECC_STATUS_ERR_COUNT; @@ -520,16 +526,22 @@ static inline int vf610_nfc_correct_data(struct nand_chip *chip, uint8_t *dat, if (!(ecc_status & ECC_STATUS_MASK)) return ecc_count; + /* The failed decode leaves a bogus correction in SRAM; re-read without ECC */ nfc->data_access = true; - nand_read_oob_op(&nfc->chip, page, 0, oob, mtd->oobsize); + ret = nand_read_page_op(&nfc->chip, page, 0, dat, mtd->writesize); + if (!ret) + ret = nand_read_oob_op(&nfc->chip, page, 0, oob, + vf610_nfc_spare_size(mtd)); nfc->data_access = false; + if (ret) + return ret; /* - * On an erased page, bit count (including OOB) should be zero or - * at least less then half of the ECC strength. + * Run the erased-page check with the driver's historic threshold + * of half the ECC strength. */ - return nand_check_erased_ecc_chunk(dat, nfc->chip.ecc.size, oob, - mtd->oobsize, NULL, 0, + return nand_check_erased_ecc_chunk(dat, mtd->writesize, oob, + vf610_nfc_spare_size(mtd), NULL, 0, flips_threshold); } @@ -550,7 +562,7 @@ static int vf610_nfc_read_page(struct nand_chip *chip, uint8_t *buf, { struct vf610_nfc *nfc = chip_to_nfc(chip); struct mtd_info *mtd = nand_to_mtd(chip); - int trfr_sz = mtd->writesize + mtd->oobsize; + int trfr_sz = mtd->writesize + vf610_nfc_spare_size(mtd); u32 row = 0, cmd1 = 0, cmd2 = 0, code = 0; int stat; @@ -576,11 +588,16 @@ static int vf610_nfc_read_page(struct nand_chip *chip, uint8_t *buf, */ vf610_nfc_rd_from_sram(buf, nfc->regs + NFC_MAIN_AREA(0), mtd->writesize, false); - if (oob_required) + if (oob_required) { + unsigned int spare = vf610_nfc_spare_size(mtd); + vf610_nfc_rd_from_sram(chip->oob_poi, nfc->regs + NFC_MAIN_AREA(0) + mtd->writesize, - mtd->oobsize, false); + spare, false); + /* Not transferred, and never written: reads back erased */ + memset(chip->oob_poi + spare, 0xff, mtd->oobsize - spare); + } stat = vf610_nfc_correct_data(chip, buf, chip->oob_poi, page); @@ -598,7 +615,7 @@ static int vf610_nfc_write_page(struct nand_chip *chip, const uint8_t *buf, { struct vf610_nfc *nfc = chip_to_nfc(chip); struct mtd_info *mtd = nand_to_mtd(chip); - int trfr_sz = mtd->writesize + mtd->oobsize; + int trfr_sz = mtd->writesize + vf610_nfc_spare_size(mtd); u32 row = 0, cmd1 = 0, cmd2 = 0, code = 0; u8 status; int ret; @@ -739,6 +756,49 @@ static void vf610_nfc_init_controller(struct vf610_nfc *nfc) } } +/* + * With 64 byte OOB chips the core's large page layout matches what + * U-Boot uses, and on chips with more, U-Boot and older kernels clamped + * mtd->oobsize to 64. Modifying the OOB size is no longer possible, the + * actual chip geometry must be respected, so to avoid breaking those + * existing setups use our own layout: the core's large page one, + * computed over the first 64 spare bytes only. + */ +static int vf610_nfc_ooblayout_ecc(struct mtd_info *mtd, int section, + struct mtd_oob_region *oobregion) +{ + struct nand_device *nand = mtd_to_nanddev(mtd); + unsigned int total_ecc_bytes = nand->ecc.ctx.total; + + if (section || !total_ecc_bytes) + return -ERANGE; + + oobregion->length = total_ecc_bytes; + oobregion->offset = vf610_nfc_spare_size(mtd) - oobregion->length; + + return 0; +} + +static int vf610_nfc_ooblayout_free(struct mtd_info *mtd, int section, + struct mtd_oob_region *oobregion) +{ + struct nand_device *nand = mtd_to_nanddev(mtd); + unsigned int total_ecc_bytes = nand->ecc.ctx.total; + + if (section) + return -ERANGE; + + oobregion->length = vf610_nfc_spare_size(mtd) - total_ecc_bytes - 2; + oobregion->offset = 2; + + return 0; +} + +static const struct mtd_ooblayout_ops vf610_nfc_ooblayout_ops = { + .ecc = vf610_nfc_ooblayout_ecc, + .free = vf610_nfc_ooblayout_free, +}; + static int vf610_nfc_attach_chip(struct nand_chip *chip) { struct mtd_info *mtd = nand_to_mtd(chip); @@ -769,12 +829,7 @@ static int vf610_nfc_attach_chip(struct nand_chip *chip) return -ENXIO; } - /* Only 64 byte ECC layouts known */ - if (mtd->oobsize > 64) - mtd->oobsize = 64; - - /* Use default large page ECC layout defined in NAND core */ - mtd_set_ooblayout(mtd, nand_get_large_page_ooblayout()); + mtd_set_ooblayout(mtd, &vf610_nfc_ooblayout_ops); if (chip->ecc.strength == 32) { nfc->ecc_mode = ECC_60_BYTE; chip->ecc.bytes = 60; diff --git a/drivers/mtd/nand/spi/core.c b/drivers/mtd/nand/spi/core.c index 4c2776f52fee..79d6d6d12107 100644 --- a/drivers/mtd/nand/spi/core.c +++ b/drivers/mtd/nand/spi/core.c @@ -179,7 +179,10 @@ static int spinand_init_cfg_cache(struct spinand_device *spinand) static int spinand_init_quad_enable(struct spinand_device *spinand) { + struct nand_device *nand = spinand_to_nand(spinand); + unsigned int target; bool enable = false; + int ret; if (!(spinand->flags & SPINAND_HAS_QE_BIT)) return 0; @@ -189,8 +192,22 @@ static int spinand_init_quad_enable(struct spinand_device *spinand) spinand->op_templates.update_cache->data.buswidth == 4) enable = true; - return spinand_upd_cfg(spinand, CFG_QUAD_ENABLE, - enable ? CFG_QUAD_ENABLE : 0); + /* + * QE is a per-die setting on some devices. Program each target + * individually when enabling or disabling quad I/O mode. + */ + for (target = 0; target < nand->memorg.ntargets; target++) { + ret = spinand_select_target(spinand, target); + if (ret) + return ret; + + ret = spinand_upd_cfg(spinand, CFG_QUAD_ENABLE, + enable ? CFG_QUAD_ENABLE : 0); + if (ret) + return ret; + } + + return 0; } static int spinand_ecc_enable(struct spinand_device *spinand, @@ -884,7 +901,8 @@ static int spinand_create_dirmap(struct spinand_device *spinand, spinand->dirmaps[plane].rdesc = desc; - if (nand->ecc.engine->integration != NAND_ECC_ENGINE_INTEGRATION_PIPELINED) { + if (!nand->ecc.engine || + nand->ecc.engine->integration != NAND_ECC_ENGINE_INTEGRATION_PIPELINED) { spinand->dirmaps[plane].wdesc_ecc = spinand->dirmaps[plane].wdesc; spinand->dirmaps[plane].rdesc_ecc = spinand->dirmaps[plane].rdesc; @@ -1280,12 +1298,17 @@ static int spinand_init(struct spinand_device *spinand) mtd->_max_bad_blocks = nanddev_mtd_max_bad_blocks; mtd->_resume = spinand_mtd_resume; - if (nand->ecc.engine) { - ret = mtd_ooblayout_count_freebytes(mtd); - if (ret < 0) - goto err_cleanup_ecc_engine; + if (!nand->ecc.engine) { + if (spinand->eccinfo.ooblayout) + mtd_set_ooblayout(mtd, spinand->eccinfo.ooblayout); + else + mtd_set_ooblayout(mtd, &spinand_noecc_ooblayout); } + ret = mtd_ooblayout_count_freebytes(mtd); + if (ret < 0) + goto err_cleanup_ecc_engine; + mtd->oobavail = ret; /* Propagate ECC information to mtd_info */ diff --git a/drivers/net/amt.c b/drivers/net/amt.c index 4799f8fd03c4..9bdc6c0e7bf9 100644 --- a/drivers/net/amt.c +++ b/drivers/net/amt.c @@ -78,15 +78,6 @@ static struct in6_addr mld2_all_node = MLD2_ALL_NODE_INIT; static struct mld2_grec mldv2_zero_grec; #endif -static struct amt_skb_cb *amt_skb_cb(struct sk_buff *skb) -{ - BUILD_BUG_ON(sizeof(struct amt_skb_cb) + sizeof(struct tc_skb_cb) > - sizeof_field(struct sk_buff, cb)); - - return (struct amt_skb_cb *)((void *)skb->cb + - sizeof(struct tc_skb_cb)); -} - static void __amt_source_gc_work(void) { struct amt_source_node *snode; @@ -787,6 +778,11 @@ static void amt_send_request(struct amt_dev *amt, bool v6) rcu_read_unlock(); } +static bool amt_send_membership_query(struct amt_dev *amt, + struct sk_buff *skb, + struct amt_tunnel_list *tunnel, + bool v6); + static void amt_send_igmp_gq(struct amt_dev *amt, struct amt_tunnel_list *tunnel) { @@ -796,8 +792,11 @@ static void amt_send_igmp_gq(struct amt_dev *amt, if (!skb) return; - amt_skb_cb(skb)->tunnel = tunnel; - dev_queue_xmit(skb); + skb_pull(skb, sizeof(struct ethhdr)); + if (amt_send_membership_query(amt, skb, tunnel, false)) { + amt->dev->stats.tx_dropped++; + kfree_skb(skb); + } } #if IS_ENABLED(CONFIG_IPV6) @@ -881,8 +880,11 @@ static void amt_send_mld_gq(struct amt_dev *amt, struct amt_tunnel_list *tunnel) if (!skb) return; - amt_skb_cb(skb)->tunnel = tunnel; - dev_queue_xmit(skb); + skb_pull(skb, sizeof(struct ethhdr)); + if (amt_send_membership_query(amt, skb, tunnel, true)) { + amt->dev->stats.tx_dropped++; + kfree_skb(skb); + } } #else static void amt_send_mld_gq(struct amt_dev *amt, struct amt_tunnel_list *tunnel) @@ -1178,7 +1180,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev) #endif bool report = false; struct igmphdr *ih; - bool query = false; struct iphdr *iph; bool data = false; bool v6 = false; @@ -1196,9 +1197,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev) case IGMP_HOST_MEMBERSHIP_REPORT: report = true; break; - case IGMP_HOST_MEMBERSHIP_QUERY: - query = true; - break; default: goto free; } @@ -1220,9 +1218,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev) case ICMPV6_MLD2_REPORT: report = true; break; - case ICMPV6_MGM_QUERY: - query = true; - break; default: goto free; } @@ -1253,19 +1248,6 @@ static netdev_tx_t amt_dev_xmit(struct sk_buff *skb, struct net_device *dev) goto free; goto unlock; } else if (amt->mode == AMT_MODE_RELAY) { - if (query) { - tunnel = amt_skb_cb(skb)->tunnel; - if (!tunnel) { - WARN_ON(1); - goto free; - } - - /* Do not forward unexpected query */ - if (amt_send_membership_query(amt, skb, tunnel, v6)) - goto free; - goto unlock; - } - if (!data) goto free; list_for_each_entry_rcu(tunnel, &amt->tunnel_list, list) { @@ -3213,6 +3195,9 @@ static int amt_newlink(struct net *net, struct net_device *dev, struct amt_dev *amt = netdev_priv(dev); int err = -EINVAL; + if (!net_eq(net, dev_net(dev))) + return err; + amt->net = net; amt->mode = nla_get_u32(data[IFLA_AMT_MODE]); diff --git a/drivers/net/can/dev/skb.c b/drivers/net/can/dev/skb.c index 0da615afa04d..3ebd4f779b9b 100644 --- a/drivers/net/can/dev/skb.c +++ b/drivers/net/can/dev/skb.c @@ -202,6 +202,7 @@ static void init_can_skb_reserve(struct sk_buff *skb) skb_reset_transport_header(skb); can_skb_reserve(skb); + can_skb_prv(skb)->skbcnt = 0; } struct sk_buff *alloc_can_skb(struct net_device *dev, struct can_frame **cf) @@ -311,6 +312,7 @@ static bool can_skb_headroom_valid(struct net_device *dev, struct sk_buff *skb) if (skb->ip_summed == CHECKSUM_NONE) { /* init headroom */ can_skb_prv(skb)->ifindex = dev->ifindex; + can_skb_prv(skb)->skbcnt = 0; skb->ip_summed = CHECKSUM_UNNECESSARY; diff --git a/drivers/net/can/usb/f81604.c b/drivers/net/can/usb/f81604.c index 36b4bb1c84b2..ea2d8a10fe36 100644 --- a/drivers/net/can/usb/f81604.c +++ b/drivers/net/can/usb/f81604.c @@ -169,7 +169,7 @@ struct f81604_int_data { u8 rxerr; u8 txerr; u8 val; -} __packed __aligned(4); +} __packed; struct f81604_sff { __be16 id; diff --git a/drivers/net/dsa/b53/b53_common.c b/drivers/net/dsa/b53/b53_common.c index f8b34fa8b99e..41961e3809b2 100644 --- a/drivers/net/dsa/b53/b53_common.c +++ b/drivers/net/dsa/b53/b53_common.c @@ -2268,8 +2268,12 @@ EXPORT_SYMBOL(b53_eee_enable_set); */ int b53_eee_init(struct dsa_switch *ds, int port, struct phy_device *phy) { + struct b53_device *dev = ds->priv; int ret; + if (is5325(dev) || is5365(dev) || is63xx(dev)) + return 0; + ret = phy_init_eee(phy, false); if (ret) return 0; @@ -2286,7 +2290,7 @@ int b53_get_mac_eee(struct dsa_switch *ds, int port, struct ethtool_eee *e) struct ethtool_eee *p = &dev->ports[port].eee; u16 reg; - if (is5325(dev) || is5365(dev)) + if (is5325(dev) || is5365(dev) || is63xx(dev)) return -EOPNOTSUPP; b53_read16(dev, B53_EEE_PAGE, B53_EEE_LPI_INDICATE, ®); @@ -2302,7 +2306,7 @@ int b53_set_mac_eee(struct dsa_switch *ds, int port, struct ethtool_eee *e) struct b53_device *dev = ds->priv; struct ethtool_eee *p = &dev->ports[port].eee; - if (is5325(dev) || is5365(dev)) + if (is5325(dev) || is5365(dev) || is63xx(dev)) return -EOPNOTSUPP; p->eee_enabled = e->eee_enabled; diff --git a/drivers/net/dsa/microchip/ksz_common.h b/drivers/net/dsa/microchip/ksz_common.h index 0e51f2277381..deb3743f0ef5 100644 --- a/drivers/net/dsa/microchip/ksz_common.h +++ b/drivers/net/dsa/microchip/ksz_common.h @@ -636,7 +636,7 @@ static inline int is_lan937x(struct ksz_device *dev) #define KSZ87_FAMILY_ID 0x87 #define KSZ88_FAMILY_ID 0x88 -#define KSZ8_PORT_STATUS_0 0x08 +#define KSZ8_PORT_STATUS_0 0x18 #define KSZ8_PORT_FIBER_MODE BIT(7) #define SW_CHIP_ID_M GENMASK(7, 4) diff --git a/drivers/net/ethernet/amazon/ena/ena_com.c b/drivers/net/ethernet/amazon/ena/ena_com.c index 276f6a8631fb..a7b98c0b5f28 100644 --- a/drivers/net/ethernet/amazon/ena/ena_com.c +++ b/drivers/net/ethernet/amazon/ena/ena_com.c @@ -763,25 +763,16 @@ static int ena_com_wait_and_process_admin_cq_interrupts(struct ena_comp_ctx *com if (comp_ctx->status == ENA_CMD_COMPLETED) { netdev_err(admin_queue->ena_dev->net_device, - "The ena device sent a completion but the driver didn't receive a MSI-X interrupt (cmd %d), autopolling mode is %s\n", - comp_ctx->cmd_opcode, admin_queue->auto_polling ? "ON" : "OFF"); - /* Check if fallback to polling is enabled */ - if (admin_queue->auto_polling) - admin_queue->polling = true; + "The ena device sent a completion but the driver didn't receive a MSI-X interrupt (cmd %d)\n", + comp_ctx->cmd_opcode); } else { netdev_err(admin_queue->ena_dev->net_device, "The ena device didn't send a completion for the admin cmd %d status %d\n", comp_ctx->cmd_opcode, comp_ctx->status); } - /* Check if shifted to polling mode. - * This will happen if there is a completion without an interrupt - * and autopolling mode is enabled. Continuing normal execution in such case - */ - if (!admin_queue->polling) { - admin_queue->running_state = false; - ret = -ETIME; - goto err; - } + admin_queue->running_state = false; + ret = -ETIME; + goto err; } ret = ena_com_comp_status_to_errno(admin_queue, comp_ctx->comp_status); @@ -1658,12 +1649,6 @@ void ena_com_set_admin_polling_mode(struct ena_com_dev *ena_dev, bool polling) ena_dev->admin_queue.polling = polling; } -void ena_com_set_admin_auto_polling_mode(struct ena_com_dev *ena_dev, - bool polling) -{ - ena_dev->admin_queue.auto_polling = polling; -} - int ena_com_mmio_reg_read_request_init(struct ena_com_dev *ena_dev) { struct ena_com_mmio_read *mmio_read = &ena_dev->mmio_read; diff --git a/drivers/net/ethernet/amazon/ena/ena_com.h b/drivers/net/ethernet/amazon/ena/ena_com.h index 3c5081d9d25d..bb33fe209eec 100644 --- a/drivers/net/ethernet/amazon/ena/ena_com.h +++ b/drivers/net/ethernet/amazon/ena/ena_com.h @@ -227,9 +227,6 @@ struct ena_com_admin_queue { /* Indicate if the admin queue should poll for completion */ bool polling; - /* Define if fallback to polling mode should occur */ - bool auto_polling; - u16 curr_cmd_id; /* Indicate that the ena was initialized and can @@ -484,17 +481,6 @@ bool ena_com_get_admin_running_state(struct ena_com_dev *ena_dev); */ void ena_com_set_admin_polling_mode(struct ena_com_dev *ena_dev, bool polling); -/* ena_com_set_admin_auto_polling_mode - Enable autoswitch to polling mode - * @ena_dev: ENA communication layer struct - * @polling: Enable/Disable polling mode - * - * Set the autopolling mode. - * If autopolling is on: - * In case of missing interrupt when data is available switch to polling. - */ -void ena_com_set_admin_auto_polling_mode(struct ena_com_dev *ena_dev, - bool polling); - /* ena_com_admin_q_comp_intr_handler - admin queue interrupt handler * @ena_dev: ENA communication layer struct * diff --git a/drivers/net/ethernet/amazon/ena/ena_eth_com.c b/drivers/net/ethernet/amazon/ena/ena_eth_com.c index 933e619b3a31..4c6e07aa4bbb 100644 --- a/drivers/net/ethernet/amazon/ena/ena_eth_com.c +++ b/drivers/net/ethernet/amazon/ena/ena_eth_com.c @@ -229,30 +229,43 @@ static struct ena_eth_io_rx_cdesc_base * idx * io_cq->cdesc_entry_size_in_bytes); } -static u16 ena_com_cdesc_rx_pkt_get(struct ena_com_io_cq *io_cq, - u16 *first_cdesc_idx) +static int ena_com_cdesc_rx_pkt_get(struct ena_com_io_cq *io_cq, + u16 *first_cdesc_idx, + u16 *num_descs) { + u16 count = io_cq->cur_rx_pkt_cdesc_count, head_masked; struct ena_eth_io_rx_cdesc_base *cdesc; - u16 count = 0, head_masked; u32 last = 0; do { + u32 status; + cdesc = ena_com_get_next_rx_cdesc(io_cq); if (!cdesc) break; + status = READ_ONCE(cdesc->status); ena_com_cq_inc_head(io_cq); + if (unlikely((status & ENA_ETH_IO_RX_CDESC_BASE_FIRST_MASK) >> + ENA_ETH_IO_RX_CDESC_BASE_FIRST_SHIFT && count != 0)) { + struct ena_com_dev *dev = ena_com_io_cq_to_ena_dev(io_cq); + + netdev_err(dev->net_device, + "First bit is on in descriptor #%d on q_id: %d, req_id: %u\n", + count, io_cq->qid, cdesc->req_id); + return -EFAULT; + } count++; - last = (READ_ONCE(cdesc->status) & ENA_ETH_IO_RX_CDESC_BASE_LAST_MASK) >> - ENA_ETH_IO_RX_CDESC_BASE_LAST_SHIFT; + last = (status & ENA_ETH_IO_RX_CDESC_BASE_LAST_MASK) >> + ENA_ETH_IO_RX_CDESC_BASE_LAST_SHIFT; } while (!last); if (last) { *first_cdesc_idx = io_cq->cur_rx_pkt_cdesc_start_idx; - count += io_cq->cur_rx_pkt_cdesc_count; head_masked = io_cq->head & (io_cq->q_depth - 1); + *num_descs = count; io_cq->cur_rx_pkt_cdesc_count = 0; io_cq->cur_rx_pkt_cdesc_start_idx = head_masked; @@ -260,11 +273,11 @@ static u16 ena_com_cdesc_rx_pkt_get(struct ena_com_io_cq *io_cq, "ENA q_id: %d packets were completed. first desc idx %u descs# %d\n", io_cq->qid, *first_cdesc_idx, count); } else { - io_cq->cur_rx_pkt_cdesc_count += count; - count = 0; + io_cq->cur_rx_pkt_cdesc_count = count; + *num_descs = 0; } - return count; + return 0; } static int ena_com_create_meta(struct ena_com_io_sq *io_sq, @@ -539,10 +552,14 @@ int ena_com_rx_pkt(struct ena_com_io_cq *io_cq, u16 cdesc_idx = 0; u16 nb_hw_desc; u16 i = 0; + int rc; WARN(io_cq->direction != ENA_COM_IO_QUEUE_DIRECTION_RX, "wrong Q type"); - nb_hw_desc = ena_com_cdesc_rx_pkt_get(io_cq, &cdesc_idx); + rc = ena_com_cdesc_rx_pkt_get(io_cq, &cdesc_idx, &nb_hw_desc); + if (unlikely(rc != 0)) + return -EFAULT; + if (nb_hw_desc == 0) { ena_rx_ctx->descs = nb_hw_desc; return 0; diff --git a/drivers/net/ethernet/amazon/ena/ena_netdev.c b/drivers/net/ethernet/amazon/ena/ena_netdev.c index 4d7446bb66b7..2242fb1f3e0d 100644 --- a/drivers/net/ethernet/amazon/ena/ena_netdev.c +++ b/drivers/net/ethernet/amazon/ena/ena_netdev.c @@ -1335,6 +1335,8 @@ static int ena_clean_rx_irq(struct ena_ring *rx_ring, struct napi_struct *napi, if (rc == -ENOSPC) { ena_increase_stat(&rx_ring->rx_stats.bad_desc_num, 1, &rx_ring->syncp); ena_reset_device(adapter, ENA_REGS_RESET_TOO_MANY_RX_DESCS); + } else if (rc == -EFAULT) { + ena_reset_device(adapter, ENA_REGS_RESET_RX_DESCRIPTOR_MALFORMED); } else { ena_increase_stat(&rx_ring->rx_stats.bad_req_id, 1, &rx_ring->syncp); @@ -3988,6 +3990,7 @@ static int ena_probe(struct pci_dev *pdev, const struct pci_device_id *ent) err_device_destroy: ena_com_delete_host_info(ena_dev); ena_com_admin_destroy(ena_dev); + ena_com_mmio_reg_read_request_destroy(ena_dev); err_netdev_destroy: free_netdev(netdev); err_free_region: diff --git a/drivers/net/ethernet/amazon/ena/ena_regs_defs.h b/drivers/net/ethernet/amazon/ena/ena_regs_defs.h index 1e007a41a525..23466d62fbc5 100644 --- a/drivers/net/ethernet/amazon/ena/ena_regs_defs.h +++ b/drivers/net/ethernet/amazon/ena/ena_regs_defs.h @@ -21,6 +21,7 @@ enum ena_regs_reset_reason_types { ENA_REGS_RESET_USER_TRIGGER = 12, ENA_REGS_RESET_GENERIC = 13, ENA_REGS_RESET_MISS_INTERRUPT = 14, + ENA_REGS_RESET_RX_DESCRIPTOR_MALFORMED = 16, }; /* ena_registers offsets */ diff --git a/drivers/net/ethernet/amd/pds_core/core.c b/drivers/net/ethernet/amd/pds_core/core.c index cd0c698983de..07bd5bd92a2b 100644 --- a/drivers/net/ethernet/amd/pds_core/core.c +++ b/drivers/net/ethernet/amd/pds_core/core.c @@ -110,7 +110,6 @@ static void pdsc_qcq_intr_free(struct pdsc *pdsc, struct pdsc_qcq *qcq) return; pdsc_intr_free(pdsc, qcq->intx); - qcq->intx = PDS_CORE_INTR_INDEX_NOT_ASSIGNED; } static int pdsc_qcq_intr_alloc(struct pdsc *pdsc, struct pdsc_qcq *qcq) @@ -129,6 +128,7 @@ static int pdsc_qcq_intr_alloc(struct pdsc *pdsc, struct pdsc_qcq *qcq) if (index < 0) return index; qcq->intx = index; + qcq->cq.bound_intr = &pdsc->intr_info[index]; return 0; } @@ -144,6 +144,12 @@ void pdsc_qcq_free(struct pdsc *pdsc, struct pdsc_qcq *qcq) pdsc_qcq_intr_free(pdsc, qcq); + /* Drain any work queued by ISR before it was freed above */ + if (qcq->work.func) + cancel_work_sync(&qcq->work); + + qcq->intx = PDS_CORE_INTR_INDEX_NOT_ASSIGNED; + if (qcq->q_base) dma_free_coherent(dev, qcq->q_size, qcq->q_base, qcq->q_base_pa); @@ -227,7 +233,6 @@ int pdsc_qcq_alloc(struct pdsc *pdsc, unsigned int type, unsigned int index, goto err_out_free_irq; } - qcq->cq.bound_intr = &pdsc->intr_info[qcq->intx]; qcq->cq.num_descs = num_descs; qcq->cq.desc_size = cq_desc_size; qcq->cq.tail_idx = 0; @@ -305,6 +310,20 @@ int pdsc_qcq_alloc(struct pdsc *pdsc, unsigned int type, unsigned int index, return err; } +static void pdsc_core_uninit(struct pdsc *pdsc) +{ + /* Free adminqcq first: its work accesses notifyqcq, so we must + * disable its IRQ and drain its work before freeing notifyqcq. + */ + pdsc_qcq_free(pdsc, &pdsc->adminqcq); + pdsc_qcq_free(pdsc, &pdsc->notifyqcq); + + if (pdsc->kern_dbpage) { + iounmap(pdsc->kern_dbpage); + pdsc->kern_dbpage = NULL; + } +} + static int pdsc_core_init(struct pdsc *pdsc) { union pds_core_dev_comp comp = {}; @@ -315,9 +334,29 @@ static int pdsc_core_init(struct pdsc *pdsc) struct pds_core_dev_init_data_in cidi; u32 dbid_count; u32 dbpage_num; + int numdescs; size_t sz; int err; + numdescs = PDSC_ADMINQ_MAX_LENGTH; + err = pdsc_qcq_alloc(pdsc, PDS_CORE_QTYPE_ADMINQ, 0, "adminq", + PDS_CORE_QCQ_F_CORE | PDS_CORE_QCQ_F_INTR, + numdescs, + sizeof(union pds_core_adminq_cmd), + sizeof(union pds_core_adminq_comp), + 0, &pdsc->adminqcq); + if (err) + return err; + + err = pdsc_qcq_alloc(pdsc, PDS_CORE_QTYPE_NOTIFYQ, 0, "notifyq", + PDS_CORE_QCQ_F_NOTIFYQ, + PDSC_NOTIFYQ_LENGTH, + sizeof(struct pds_core_notifyq_cmd), + sizeof(union pds_core_notifyq_comp), + 0, &pdsc->notifyqcq); + if (err) + goto err_out_uninit; + cidi.adminq_q_base = cpu_to_le64(pdsc->adminqcq.q_base_pa); cidi.adminq_cq_base = cpu_to_le64(pdsc->adminqcq.cq_base_pa); cidi.notifyq_cq_base = cpu_to_le64(pdsc->notifyqcq.cq.base_pa); @@ -341,7 +380,7 @@ static int pdsc_core_init(struct pdsc *pdsc) if (err) { dev_err(pdsc->dev, "Device init command failed: %pe\n", ERR_PTR(err)); - return err; + goto err_out_uninit; } pdsc->hw_index = le32_to_cpu(cido.core_hw_index); @@ -351,7 +390,8 @@ static int pdsc_core_init(struct pdsc *pdsc) pdsc->kern_dbpage = pdsc_map_dbpage(pdsc, dbpage_num); if (!pdsc->kern_dbpage) { dev_err(pdsc->dev, "Cannot map dbpage, aborting\n"); - return -ENOMEM; + err = -ENOMEM; + goto err_out_uninit; } pdsc->adminqcq.q.hw_type = cido.adminq_hw_type; @@ -364,6 +404,10 @@ static int pdsc_core_init(struct pdsc *pdsc) pdsc->last_eid = 0; + return 0; + +err_out_uninit: + pdsc_core_uninit(pdsc); return err; } @@ -406,35 +450,14 @@ static int pdsc_viftypes_init(struct pdsc *pdsc) int pdsc_setup(struct pdsc *pdsc, bool init) { - int numdescs; int err; + pci_set_master(pdsc->pdev); + err = pdsc_dev_init(pdsc); if (err) return err; - numdescs = PDSC_ADMINQ_MAX_LENGTH; - err = pdsc_qcq_alloc(pdsc, PDS_CORE_QTYPE_ADMINQ, 0, "adminq", - PDS_CORE_QCQ_F_CORE | PDS_CORE_QCQ_F_INTR, - numdescs, - sizeof(union pds_core_adminq_cmd), - sizeof(union pds_core_adminq_comp), - 0, &pdsc->adminqcq); - if (err) - goto err_out_teardown; - - err = pdsc_qcq_alloc(pdsc, PDS_CORE_QTYPE_NOTIFYQ, 0, "notifyq", - PDS_CORE_QCQ_F_NOTIFYQ, - PDSC_NOTIFYQ_LENGTH, - sizeof(struct pds_core_notifyq_cmd), - sizeof(union pds_core_notifyq_comp), - 0, &pdsc->notifyqcq); - if (err) - goto err_out_teardown; - - /* NotifyQ rides on the AdminQ interrupt */ - pdsc->notifyqcq.intx = pdsc->adminqcq.intx; - /* Set up the Core with the AdminQ and NotifyQ info */ err = pdsc_core_init(pdsc); if (err) @@ -459,33 +482,18 @@ int pdsc_setup(struct pdsc *pdsc, bool init) void pdsc_teardown(struct pdsc *pdsc, bool removing) { - int i; - if (!pdsc->pdev->is_virtfn) pdsc_devcmd_reset(pdsc); - if (pdsc->adminqcq.work.func) - cancel_work_sync(&pdsc->adminqcq.work); - pdsc_qcq_free(pdsc, &pdsc->notifyqcq); - pdsc_qcq_free(pdsc, &pdsc->adminqcq); - kfree(pdsc->viftype_status); - pdsc->viftype_status = NULL; + pci_clear_master(pdsc->pdev); - if (pdsc->intr_info) { - for (i = 0; i < pdsc->nintrs; i++) - pdsc_intr_free(pdsc, i); + pdsc_core_uninit(pdsc); - kfree(pdsc->intr_info); - pdsc->intr_info = NULL; - pdsc->nintrs = 0; - } + kfree(pdsc->viftype_status); + pdsc->viftype_status = NULL; - if (pdsc->kern_dbpage) { - iounmap(pdsc->kern_dbpage); - pdsc->kern_dbpage = NULL; - } + pdsc_dev_uninit(pdsc); - pci_free_irq_vectors(pdsc->pdev); set_bit(PDSC_S_FW_DEAD, &pdsc->state); } diff --git a/drivers/net/ethernet/amd/pds_core/core.h b/drivers/net/ethernet/amd/pds_core/core.h index 421371408503..b61d7c5afda1 100644 --- a/drivers/net/ethernet/amd/pds_core/core.h +++ b/drivers/net/ethernet/amd/pds_core/core.h @@ -282,6 +282,7 @@ int pdsc_devcmd_locked(struct pdsc *pdsc, union pds_core_dev_cmd *cmd, int pdsc_devcmd_init(struct pdsc *pdsc); int pdsc_devcmd_reset(struct pdsc *pdsc); int pdsc_dev_init(struct pdsc *pdsc); +void pdsc_dev_uninit(struct pdsc *pdsc); int pdsc_intr_alloc(struct pdsc *pdsc, char *name, irq_handler_t handler, void *data); diff --git a/drivers/net/ethernet/amd/pds_core/debugfs.c b/drivers/net/ethernet/amd/pds_core/debugfs.c index 33b8e8d1fd4c..734b86804f40 100644 --- a/drivers/net/ethernet/amd/pds_core/debugfs.c +++ b/drivers/net/ethernet/amd/pds_core/debugfs.c @@ -114,7 +114,6 @@ void pdsc_debugfs_add_qcq(struct pdsc *pdsc, struct pdsc_qcq *qcq) struct dentry *qcq_dentry, *q_dentry, *cq_dentry; struct dentry *intr_dentry; struct debugfs_regset32 *intr_ctrl_regset; - struct pdsc_intr_info *intr = &pdsc->intr_info[qcq->intx]; struct pdsc_queue *q = &qcq->q; struct pdsc_cq *cq = &qcq->cq; @@ -152,6 +151,8 @@ void pdsc_debugfs_add_qcq(struct pdsc *pdsc, struct pdsc_qcq *qcq) debugfs_create_u16("tail", 0400, cq_dentry, &cq->tail_idx); if (qcq->flags & PDS_CORE_QCQ_F_INTR) { + struct pdsc_intr_info *intr = &pdsc->intr_info[qcq->intx]; + intr_dentry = debugfs_create_dir("intr", qcq->dentry); if (IS_ERR_OR_NULL(intr_dentry)) return; diff --git a/drivers/net/ethernet/amd/pds_core/dev.c b/drivers/net/ethernet/amd/pds_core/dev.c index 0e9398503a04..68f44c4a4c42 100644 --- a/drivers/net/ethernet/amd/pds_core/dev.c +++ b/drivers/net/ethernet/amd/pds_core/dev.c @@ -323,6 +323,22 @@ static int pdsc_identify(struct pdsc *pdsc) return 0; } +void pdsc_dev_uninit(struct pdsc *pdsc) +{ + if (pdsc->intr_info) { + int i; + + for (i = 0; i < pdsc->nintrs; i++) + pdsc_intr_free(pdsc, i); + + kfree(pdsc->intr_info); + pdsc->intr_info = NULL; + pdsc->nintrs = 0; + } + + pci_free_irq_vectors(pdsc->pdev); +} + int pdsc_dev_init(struct pdsc *pdsc) { unsigned int nintrs; diff --git a/drivers/net/ethernet/amd/pds_core/main.c b/drivers/net/ethernet/amd/pds_core/main.c index e88230ae6349..a5075e11f03a 100644 --- a/drivers/net/ethernet/amd/pds_core/main.c +++ b/drivers/net/ethernet/amd/pds_core/main.c @@ -432,8 +432,6 @@ static void pdsc_remove(struct pci_dev *pdev) pdsc_sriov_configure(pdev, 0); timer_shutdown_sync(&pdsc->wdtimer); - if (pdsc->wq) - destroy_workqueue(pdsc->wq); mutex_lock(&pdsc->config_lock); set_bit(PDSC_S_STOPPING_DRIVER, &pdsc->state); @@ -441,6 +439,9 @@ static void pdsc_remove(struct pci_dev *pdev) pdsc_stop(pdsc); pdsc_teardown(pdsc, PDSC_TEARDOWN_REMOVING); mutex_unlock(&pdsc->config_lock); + + if (pdsc->wq) + destroy_workqueue(pdsc->wq); mutex_destroy(&pdsc->config_lock); mutex_destroy(&pdsc->devcmd_lock); diff --git a/drivers/net/ethernet/broadcom/asp2/bcmasp.c b/drivers/net/ethernet/broadcom/asp2/bcmasp.c index d9e9ec2e8945..a419e7160887 100644 --- a/drivers/net/ethernet/broadcom/asp2/bcmasp.c +++ b/drivers/net/ethernet/broadcom/asp2/bcmasp.c @@ -1118,12 +1118,6 @@ static void bcmasp_enable_wol_shared(struct bcmasp_intf *intf, bool en) } } -static void bcmasp_wol_irq_destroy_shared(struct bcmasp_priv *priv) -{ - if (priv->wol_irq > 0) - free_irq(priv->wol_irq, priv); -} - static void bcmasp_init_wol_per_intf(struct bcmasp_priv *priv) { struct platform_device *pdev = priv->pdev; @@ -1156,16 +1150,6 @@ static void bcmasp_enable_wol_per_intf(struct bcmasp_intf *intf, bool en) device_set_wakeup_enable(dev, en); } -static void bcmasp_wol_irq_destroy_per_intf(struct bcmasp_priv *priv) -{ - struct bcmasp_intf *intf; - - list_for_each_entry(intf, &priv->intfs, list) { - if (intf->wol_irq > 0) - free_irq(intf->wol_irq, priv); - } -} - static struct bcmasp_hw_info v20_hw_info = { .rx_ctrl_flush = ASP_RX_CTRL_FLUSH, .umac2fb = UMAC2FB_OFFSET, @@ -1177,7 +1161,6 @@ static struct bcmasp_hw_info v20_hw_info = { static const struct bcmasp_plat_data v20_plat_data = { .init_wol = bcmasp_init_wol_per_intf, .enable_wol = bcmasp_enable_wol_per_intf, - .destroy_wol = bcmasp_wol_irq_destroy_per_intf, .hw_info = &v20_hw_info, }; @@ -1193,7 +1176,6 @@ static struct bcmasp_hw_info v21_hw_info = { static const struct bcmasp_plat_data v21_plat_data = { .init_wol = bcmasp_init_wol_shared, .enable_wol = bcmasp_enable_wol_shared, - .destroy_wol = bcmasp_wol_irq_destroy_shared, .hw_info = &v21_hw_info, }; @@ -1267,7 +1249,6 @@ static int bcmasp_probe(struct platform_device *pdev) priv->init_wol = pdata->init_wol; priv->enable_wol = pdata->enable_wol; - priv->destroy_wol = pdata->destroy_wol; priv->hw_info = pdata->hw_info; /* Enable all clocks to ensure successful probing */ @@ -1331,7 +1312,6 @@ static int bcmasp_probe(struct platform_device *pdev) if (ret) { netdev_err(intf->ndev, "failed to register net_device: %d\n", ret); - priv->destroy_wol(priv); bcmasp_remove_intfs(priv); goto of_put_exit; } @@ -1352,7 +1332,6 @@ static int bcmasp_remove(struct platform_device *pdev) if (!priv) return 0; - priv->destroy_wol(priv); bcmasp_remove_intfs(priv); return 0; diff --git a/drivers/net/ethernet/broadcom/asp2/bcmasp.h b/drivers/net/ethernet/broadcom/asp2/bcmasp.h index ec90add6b03e..985a47333955 100644 --- a/drivers/net/ethernet/broadcom/asp2/bcmasp.h +++ b/drivers/net/ethernet/broadcom/asp2/bcmasp.h @@ -371,7 +371,6 @@ struct bcmasp_hw_info { struct bcmasp_plat_data { void (*init_wol)(struct bcmasp_priv *priv); void (*enable_wol)(struct bcmasp_intf *intf, bool en); - void (*destroy_wol)(struct bcmasp_priv *priv); struct bcmasp_hw_info *hw_info; }; @@ -389,7 +388,6 @@ struct bcmasp_priv { void (*init_wol)(struct bcmasp_priv *priv); void (*enable_wol)(struct bcmasp_intf *intf, bool en); - void (*destroy_wol)(struct bcmasp_priv *priv); void __iomem *base; struct bcmasp_hw_info *hw_info; diff --git a/drivers/net/ethernet/broadcom/bcmsysport.c b/drivers/net/ethernet/broadcom/bcmsysport.c index 23cc2d85994e..332e0745eb75 100644 --- a/drivers/net/ethernet/broadcom/bcmsysport.c +++ b/drivers/net/ethernet/broadcom/bcmsysport.c @@ -425,8 +425,10 @@ static void bcm_sysport_update_mib_counters(struct bcm_sysport_priv *priv) if (priv->is_lite) continue; - if (s->type != BCM_SYSPORT_STAT_MIB_RX) + if (s->type == BCM_SYSPORT_STAT_MIB_TX) offset = UMAC_MIB_STAT_OFFSET; + else if (s->type == BCM_SYSPORT_STAT_RUNT) + offset = 2 * UMAC_MIB_STAT_OFFSET; val = umac_readl(priv, UMAC_MIB_START + j + offset); break; case BCM_SYSPORT_STAT_RXCHK: @@ -516,10 +518,10 @@ static void bcm_sysport_get_stats(struct net_device *dev, s->type == BCM_SYSPORT_STAT_NETDEV64) { do { start = u64_stats_fetch_begin(syncp); - data[i] = *(u64 *)p; + data[j] = *(u64 *)p; } while (u64_stats_fetch_retry(syncp, start)); } else - data[i] = *(u32 *)p; + data[j] = *(u32 *)p; j++; } diff --git a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c index 7c3f62783de3..7bab74cfb37e 100644 --- a/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c +++ b/drivers/net/ethernet/broadcom/bnx2x/bnx2x_cmn.c @@ -3538,7 +3538,7 @@ static u8 bnx2x_set_pbd_csum_enc(struct bnx2x *bp, struct sk_buff *skb, u32 *parsing_data, u32 xmit_type) { *parsing_data |= - ((((u8 *)skb_inner_transport_header(skb) - skb->data) >> 1) << + ((skb_inner_transport_offset(skb) >> 1) << ETH_TX_PARSE_BD_E2_L4_HDR_START_OFFSET_W_SHIFT) & ETH_TX_PARSE_BD_E2_L4_HDR_START_OFFSET_W; @@ -3570,7 +3570,7 @@ static u8 bnx2x_set_pbd_csum_e2(struct bnx2x *bp, struct sk_buff *skb, u32 *parsing_data, u32 xmit_type) { *parsing_data |= - ((((u8 *)skb_transport_header(skb) - skb->data) >> 1) << + ((skb_transport_offset(skb) >> 1) << ETH_TX_PARSE_BD_E2_L4_HDR_START_OFFSET_W_SHIFT) & ETH_TX_PARSE_BD_E2_L4_HDR_START_OFFSET_W; @@ -3613,7 +3613,7 @@ static u8 bnx2x_set_pbd_csum(struct bnx2x *bp, struct sk_buff *skb, struct eth_tx_parse_bd_e1x *pbd, u32 xmit_type) { - u8 hlen = (skb_network_header(skb) - skb->data) >> 1; + u8 hlen = skb_network_offset(skb) >> 1; /* for now NS flag is not used in Linux */ pbd->global_data = @@ -3667,8 +3667,7 @@ static void bnx2x_update_pbds_gso_enc(struct sk_buff *skb, u8 outerip_off, outerip_len = 0; /* from outer IP to transport */ - hlen_w = (skb_inner_transport_header(skb) - - skb_network_header(skb)) >> 1; + hlen_w = skb_inner_transport_offset(skb) >> 1; /* transport len */ hlen_w += inner_tcp_hdrlen(skb) >> 1; @@ -3714,7 +3713,7 @@ static void bnx2x_update_pbds_gso_enc(struct sk_buff *skb, 0, IPPROTO_TCP, 0)); } - outerip_off = (skb_network_header(skb) - skb->data) >> 1; + outerip_off = (skb_network_offset(skb)) >> 1; *global_data |= outerip_off | diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.c b/drivers/net/ethernet/broadcom/bnxt/bnxt.c index 0e1a88a0df3d..9cfccedebc09 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.c @@ -394,7 +394,7 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev) struct netdev_queue *txq; int i; dma_addr_t mapping; - unsigned int length, pad = 0; + unsigned int length; u32 len, free_size, vlan_tag_flags, cfa_action, flags; u16 prod, last_frag; struct pci_dev *pdev = bp->pdev; @@ -413,6 +413,14 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev) txr = &bp->tx_ring[bp->tx_ring_map[i]]; prod = txr->tx_prod; + /* Must be done before skb_shinfo(skb)->nr_frags is sampled, + * because skb_put_padto() might linearize the skb. + */ + if (skb_put_padto(skb, BNXT_MIN_PKT_SIZE)) { + /* SKB already freed. */ + goto tx_kick_pending; + } + free_size = bnxt_tx_avail(bp, txr); if (unlikely(free_size < skb_shinfo(skb)->nr_frags + 2)) { /* We must have raced with NAPI cleanup */ @@ -545,14 +553,6 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev) } normal_tx: - if (length < BNXT_MIN_PKT_SIZE) { - pad = BNXT_MIN_PKT_SIZE - length; - if (skb_pad(skb, pad)) - /* SKB already freed. */ - goto tx_kick_pending; - length = BNXT_MIN_PKT_SIZE; - } - mapping = dma_map_single(&pdev->dev, skb->data, len, DMA_TO_DEVICE); if (unlikely(dma_mapping_error(&pdev->dev, mapping))) @@ -624,10 +624,7 @@ static netdev_tx_t bnxt_start_xmit(struct sk_buff *skb, struct net_device *dev) txbd->tx_bd_len_flags_type = cpu_to_le32(flags); } - flags &= ~TX_BD_LEN; - txbd->tx_bd_len_flags_type = - cpu_to_le32(((len + pad) << TX_BD_LEN_SHIFT) | flags | - TX_BD_FLAGS_PACKET_END); + txbd->tx_bd_len_flags_type |= cpu_to_le32(TX_BD_FLAGS_PACKET_END); netdev_tx_sent_queue(txq, skb->len); @@ -818,6 +815,7 @@ int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, mapping += bp->rx_dma_offset; rx_buf->data = page; rx_buf->data_ptr = page_address(page) + offset + bp->rx_offset; + rx_buf->offset = offset; } else { u8 *data = __bnxt_alloc_rx_frag(bp, &mapping, gfp); @@ -826,6 +824,7 @@ int bnxt_alloc_rx_data(struct bnxt *bp, struct bnxt_rx_ring_info *rxr, rx_buf->data = data; rx_buf->data_ptr = data + bp->rx_offset; + rx_buf->offset = 0; } rx_buf->mapping = mapping; @@ -846,6 +845,7 @@ void bnxt_reuse_rx_data(struct bnxt_rx_ring_info *rxr, u16 cons, void *data) prod_rx_buf->data_ptr = cons_rx_buf->data_ptr; prod_rx_buf->mapping = cons_rx_buf->mapping; + prod_rx_buf->offset = cons_rx_buf->offset; prod_bd = &rxr->rx_desc_ring[RX_RING(prod)][RX_IDX(prod)]; cons_bd = &rxr->rx_desc_ring[RX_RING(cons)][RX_IDX(cons)]; @@ -983,8 +983,11 @@ static struct sk_buff *bnxt_rx_multi_page_skb(struct bnxt *bp, struct page *page = data; u16 prod = rxr->rx_prod; struct sk_buff *skb; + void *frag_start; int err; + frag_start = page_address(page) + rxr->rx_buf_ring[cons].offset; + err = bnxt_alloc_rx_data(bp, rxr, prod, GFP_ATOMIC); if (unlikely(err)) { bnxt_reuse_rx_data(rxr, cons, data); @@ -993,13 +996,13 @@ static struct sk_buff *bnxt_rx_multi_page_skb(struct bnxt *bp, dma_addr -= bp->rx_dma_offset; dma_sync_single_for_cpu(&bp->pdev->dev, dma_addr, BNXT_RX_PAGE_SIZE, bp->rx_dir); - skb = napi_build_skb(data_ptr - bp->rx_offset, BNXT_RX_PAGE_SIZE); + skb = napi_build_skb(frag_start, BNXT_RX_PAGE_SIZE); if (!skb) { page_pool_recycle_direct(rxr->page_pool, page); return NULL; } skb_mark_for_recycle(skb); - skb_reserve(skb, bp->rx_offset); + skb_reserve(skb, data_ptr - (u8 *)frag_start); __skb_put(skb, len); return skb; @@ -1800,6 +1803,7 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr, struct rx_cmp_ext *rxcmp1; u32 tmp_raw_cons = *raw_cons; u16 cfa_code, cons, prod, cp_cons = RING_CMP(tmp_raw_cons); + struct skb_shared_info *sinfo; struct bnxt_sw_rx_bd *rx_buf; unsigned int len; u8 *data_ptr, agg_bufs, cmp_type; @@ -1924,6 +1928,7 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr, false); if (!frag_len) goto oom_next_rx; + } xdp_active = true; } @@ -1933,6 +1938,12 @@ static int bnxt_rx_pkt(struct bnxt *bp, struct bnxt_cp_ring_info *cpr, rc = 1; goto next_rx; } + if (xdp_buff_has_frags(&xdp)) { + sinfo = xdp_get_shared_info_from_buff(&xdp); + agg_bufs = sinfo->nr_frags; + } else { + agg_bufs = 0; + } } if (len <= bp->rx_copy_thresh) { @@ -10502,6 +10513,21 @@ static int bnxt_reinit_after_abort(struct bnxt *bp) return rc; } +static int bnxt_tx_nr_rings(struct bnxt *bp) +{ + int num_tc = netdev_get_num_tc(bp->dev); + + return num_tc ? bp->tx_nr_rings_per_tc * num_tc : + bp->tx_nr_rings_per_tc; +} + +static int bnxt_tx_nr_rings_per_tc(struct bnxt *bp) +{ + int num_tc = netdev_get_num_tc(bp->dev); + + return num_tc ? bp->tx_nr_rings / num_tc : bp->tx_nr_rings; +} + static int __bnxt_open_nic(struct bnxt *bp, bool irq_re_init, bool link_re_init) { int rc = 0; @@ -13449,7 +13475,7 @@ static void bnxt_trim_dflt_sh_rings(struct bnxt *bp) bp->cp_nr_rings = min_t(int, bp->tx_nr_rings_per_tc, bp->rx_nr_rings); bp->rx_nr_rings = bp->cp_nr_rings; bp->tx_nr_rings_per_tc = bp->cp_nr_rings; - bp->tx_nr_rings = bp->tx_nr_rings_per_tc; + bp->tx_nr_rings = bnxt_tx_nr_rings(bp); } static int bnxt_set_dflt_rings(struct bnxt *bp, bool sh) @@ -13480,12 +13506,12 @@ static int bnxt_set_dflt_rings(struct bnxt *bp, bool sh) bnxt_trim_dflt_sh_rings(bp); else bp->cp_nr_rings = bp->tx_nr_rings_per_tc + bp->rx_nr_rings; - bp->tx_nr_rings = bp->tx_nr_rings_per_tc; + bp->tx_nr_rings = bnxt_tx_nr_rings(bp); rc = __bnxt_reserve_rings(bp); if (rc && rc != -ENODEV) netdev_warn(bp->dev, "Unable to reserve tx rings\n"); - bp->tx_nr_rings_per_tc = bp->tx_nr_rings; + bp->tx_nr_rings_per_tc = bnxt_tx_nr_rings_per_tc(bp); if (sh) bnxt_trim_dflt_sh_rings(bp); @@ -13494,7 +13520,7 @@ static int bnxt_set_dflt_rings(struct bnxt *bp, bool sh) rc = __bnxt_reserve_rings(bp); if (rc && rc != -ENODEV) netdev_warn(bp->dev, "2nd rings reservation failed.\n"); - bp->tx_nr_rings_per_tc = bp->tx_nr_rings; + bp->tx_nr_rings_per_tc = bnxt_tx_nr_rings_per_tc(bp); } if (BNXT_CHIP_TYPE_NITRO_A0(bp)) { bp->rx_nr_rings++; @@ -13528,7 +13554,7 @@ static int bnxt_init_dflt_ring_mode(struct bnxt *bp) if (rc) goto init_dflt_ring_err; - bp->tx_nr_rings_per_tc = bp->tx_nr_rings; + bp->tx_nr_rings_per_tc = bnxt_tx_nr_rings_per_tc(bp); bnxt_set_dflt_rfs(bp); diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt.h b/drivers/net/ethernet/broadcom/bnxt/bnxt.h index 739ed9d0b5ab..5a860758cf03 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt.h +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt.h @@ -729,6 +729,7 @@ struct bnxt_sw_rx_bd { void *data; u8 *data_ptr; dma_addr_t mapping; + unsigned int offset; }; struct bnxt_sw_rx_agg_bd { diff --git a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c index b6320b64b8e7..fc47d45805b7 100644 --- a/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c +++ b/drivers/net/ethernet/broadcom/bnxt/bnxt_xdp.c @@ -464,7 +464,7 @@ bnxt_xdp_build_skb(struct bnxt *bp, struct sk_buff *skb, u8 num_frags, xdp_update_skb_shared_info(skb, num_frags, sinfo->xdp_frags_size, - BNXT_RX_PAGE_SIZE * sinfo->nr_frags, + BNXT_RX_PAGE_SIZE * num_frags, xdp_buff_is_frag_pfmemalloc(xdp)); return skb; } diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c index 14b28d0c93de..affac9887f16 100644 --- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c +++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c @@ -4084,8 +4084,11 @@ static int bcmgenet_probe(struct platform_device *pdev) netif_set_real_num_rx_queues(priv->dev, priv->hw_params->rx_queues + 1); /* Set default coalescing parameters */ - for (i = 0; i <= priv->hw_params->rx_queues; i++) + for (i = 0; i <= priv->hw_params->rx_queues; i++) { + priv->rx_rings[i].priv = priv; + priv->rx_rings[i].index = i; priv->rx_rings[i].rx_max_coalesced_frames = 1; + } /* Initialize u64 stats seq counter for 32bit machines */ for (i = 0; i <= GENET_MAX_MQ_CNT; i++) { diff --git a/drivers/net/ethernet/brocade/bna/bnad.c b/drivers/net/ethernet/brocade/bna/bnad.c index 6cf06a93bedf..c01c849a8d1a 100644 --- a/drivers/net/ethernet/brocade/bna/bnad.c +++ b/drivers/net/ethernet/brocade/bna/bnad.c @@ -3727,9 +3727,11 @@ bnad_pci_probe(struct pci_dev *pdev, bnad_res_free(bnad, &bnad->mod_res_info[0], BNA_MOD_RES_T_MAX); disable_ioceth: bnad_ioceth_disable(bnad); - del_timer_sync(&bnad->bna.ioceth.ioc.ioc_timer); - del_timer_sync(&bnad->bna.ioceth.ioc.sem_timer); - del_timer_sync(&bnad->bna.ioceth.ioc.hb_timer); + /* The IOC timers rearm one another. Shut down all of them. */ + timer_shutdown_sync(&bnad->bna.ioceth.ioc.ioc_timer); + timer_shutdown_sync(&bnad->bna.ioceth.ioc.sem_timer); + timer_shutdown_sync(&bnad->bna.ioceth.ioc.hb_timer); + timer_shutdown_sync(&bnad->bna.ioceth.ioc.iocpf_timer); spin_lock_irqsave(&bnad->bna_lock, flags); bna_uninit(bna); spin_unlock_irqrestore(&bnad->bna_lock, flags); @@ -3770,9 +3772,11 @@ bnad_pci_remove(struct pci_dev *pdev) mutex_lock(&bnad->conf_mutex); bnad_ioceth_disable(bnad); - del_timer_sync(&bnad->bna.ioceth.ioc.ioc_timer); - del_timer_sync(&bnad->bna.ioceth.ioc.sem_timer); - del_timer_sync(&bnad->bna.ioceth.ioc.hb_timer); + /* The IOC timers rearm one another. Shut down all of them. */ + timer_shutdown_sync(&bnad->bna.ioceth.ioc.ioc_timer); + timer_shutdown_sync(&bnad->bna.ioceth.ioc.sem_timer); + timer_shutdown_sync(&bnad->bna.ioceth.ioc.hb_timer); + timer_shutdown_sync(&bnad->bna.ioceth.ioc.iocpf_timer); spin_lock_irqsave(&bnad->bna_lock, flags); bna_uninit(bna); spin_unlock_irqrestore(&bnad->bna_lock, flags); diff --git a/drivers/net/ethernet/cavium/liquidio/lio_main.c b/drivers/net/ethernet/cavium/liquidio/lio_main.c index 8e4e49d24dad..19e8f8bbd06e 100644 --- a/drivers/net/ethernet/cavium/liquidio/lio_main.c +++ b/drivers/net/ethernet/cavium/liquidio/lio_main.c @@ -3795,9 +3795,7 @@ static int setup_nic_devices(struct octeon_device *octeon_dev) static int octeon_enable_sriov(struct octeon_device *oct) { unsigned int num_vfs_alloced = oct->sriov_info.num_vfs_alloced; - struct pci_dev *vfdev; int err; - u32 u; if (OCTEON_CN23XX_PF(oct) && num_vfs_alloced) { err = pci_enable_sriov(oct->pci_dev, @@ -3810,23 +3808,6 @@ static int octeon_enable_sriov(struct octeon_device *oct) return err; } oct->sriov_info.sriov_enabled = 1; - - /* init lookup table that maps DPI ring number to VF pci_dev - * struct pointer - */ - u = 0; - vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM, - OCTEON_CN23XX_VF_VID, NULL); - while (vfdev) { - if (vfdev->is_virtfn && - (vfdev->physfn == oct->pci_dev)) { - oct->sriov_info.dpiring_to_vfpcidev_lut[u] = - vfdev; - u += oct->sriov_info.rings_per_vf; - } - vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM, - OCTEON_CN23XX_VF_VID, vfdev); - } } return num_vfs_alloced; @@ -3834,8 +3815,6 @@ static int octeon_enable_sriov(struct octeon_device *oct) static int lio_pci_sriov_disable(struct octeon_device *oct) { - int u; - if (pci_vfs_assigned(oct->pci_dev)) { dev_err(&oct->pci_dev->dev, "VFs are still assigned to VMs.\n"); return -EPERM; @@ -3843,12 +3822,6 @@ static int lio_pci_sriov_disable(struct octeon_device *oct) pci_disable_sriov(oct->pci_dev); - u = 0; - while (u < MAX_POSSIBLE_VFS) { - oct->sriov_info.dpiring_to_vfpcidev_lut[u] = NULL; - u += oct->sriov_info.rings_per_vf; - } - oct->sriov_info.num_vfs_alloced = 0; dev_info(&oct->pci_dev->dev, "oct->pf_num:%d disabled VFs\n", oct->pf_num); diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_device.h b/drivers/net/ethernet/cavium/liquidio/octeon_device.h index fb380b4f3e02..cfa3b769227f 100644 --- a/drivers/net/ethernet/cavium/liquidio/octeon_device.h +++ b/drivers/net/ethernet/cavium/liquidio/octeon_device.h @@ -390,9 +390,6 @@ struct octeon_sriov_info { struct lio_trusted_vf trusted_vf; - /*lookup table that maps DPI ring number to VF pci_dev struct pointer*/ - struct pci_dev *dpiring_to_vfpcidev_lut[MAX_POSSIBLE_VFS]; - u64 vf_macaddr[MAX_POSSIBLE_VFS]; u16 vf_vlantci[MAX_POSSIBLE_VFS]; diff --git a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c index ad685f5d0a13..697fcdc41e3c 100644 --- a/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c +++ b/drivers/net/ethernet/cavium/liquidio/octeon_mailbox.c @@ -26,6 +26,31 @@ #include "octeon_mailbox.h" #include "cn23xx_pf_device.h" +static struct pci_dev *lio_vf_pci_dev_by_qno(struct octeon_device *oct, u32 q_no) +{ + struct pci_dev *vfdev = NULL; + int vfidx; + + if (!oct->sriov_info.rings_per_vf) + return NULL; + + if (q_no % oct->sriov_info.rings_per_vf) + return NULL; + + vfidx = q_no / oct->sriov_info.rings_per_vf; + if (vfidx >= oct->sriov_info.num_vfs_alloced) + return NULL; + + while ((vfdev = pci_get_device(PCI_VENDOR_ID_CAVIUM, + OCTEON_CN23XX_VF_VID, vfdev))) { + if (pci_physfn(vfdev) && pci_physfn(vfdev) == oct->pci_dev && + pci_iov_vf_id(vfdev) == vfidx) + return vfdev; + } + + return NULL; +} + /** * octeon_mbox_read: * @mbox: Pointer mailbox @@ -237,6 +262,7 @@ static int octeon_mbox_process_cmd(struct octeon_mbox *mbox, struct octeon_mbox_cmd *mbox_cmd) { struct octeon_device *oct = mbox->oct_dev; + struct pci_dev *vfdev; switch (mbox_cmd->msg.s.cmd) { case OCTEON_VF_ACTIVE: @@ -260,7 +286,12 @@ static int octeon_mbox_process_cmd(struct octeon_mbox *mbox, dev_info(&oct->pci_dev->dev, "got a request for FLR from VF that owns DPI ring %u\n", mbox->q_no); - pcie_flr(oct->sriov_info.dpiring_to_vfpcidev_lut[mbox->q_no]); + vfdev = lio_vf_pci_dev_by_qno(oct, mbox->q_no); + if (!vfdev) + break; + + pcie_flr(vfdev); + pci_dev_put(vfdev); break; case OCTEON_PF_CHANGED_VF_MACADDR: diff --git a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c index bcdc7fc2f427..649d05b3fea7 100644 --- a/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c +++ b/drivers/net/ethernet/chelsio/inline_crypto/ch_ktls/chcr_ktls.c @@ -390,7 +390,7 @@ static void chcr_ktls_dev_del(struct net_device *netdev, cxgb4_remove_tid(&tx_info->adap->tids, tx_info->tx_chan, tx_info->tid, tx_info->ip_family); - xa_erase(&u_ctx->tid_list, tx_info->tid); + xa_erase_bh(&u_ctx->tid_list, tx_info->tid); } port_stats = &tx_info->adap->ch_ktls_stats.ktls_port[tx_info->port_id]; @@ -581,7 +581,7 @@ static int chcr_ktls_dev_add(struct net_device *netdev, struct sock *sk, cxgb4_remove_tid(&tx_info->adap->tids, tx_info->tx_chan, tx_info->tid, tx_info->ip_family); - xa_erase(&u_ctx->tid_list, tx_info->tid); + xa_erase_bh(&u_ctx->tid_list, tx_info->tid); put_module: /* release module refcount */ diff --git a/drivers/net/ethernet/google/gve/gve_ethtool.c b/drivers/net/ethernet/google/gve/gve_ethtool.c index b1c4df3bab29..bc20c13ba287 100644 --- a/drivers/net/ethernet/google/gve/gve_ethtool.c +++ b/drivers/net/ethernet/google/gve/gve_ethtool.c @@ -166,6 +166,8 @@ gve_get_ethtool_stats(struct net_device *netdev, struct stats *report_stats; int *rx_qid_to_stats_idx; int *tx_qid_to_stats_idx; + int num_stopped_rxqs = 0; + int num_stopped_txqs = 0; struct gve_priv *priv; bool skip_nic_stats; unsigned int start; @@ -182,12 +184,23 @@ gve_get_ethtool_stats(struct net_device *netdev, sizeof(int), GFP_KERNEL); if (!rx_qid_to_stats_idx) return; + for (ring = 0; ring < priv->rx_cfg.num_queues; ring++) { + rx_qid_to_stats_idx[ring] = -1; + if (!priv->ntfy_blocks[gve_rx_idx_to_ntfy(priv, ring)].rx) + num_stopped_rxqs++; + } tx_qid_to_stats_idx = kmalloc_array(num_tx_queues, sizeof(int), GFP_KERNEL); if (!tx_qid_to_stats_idx) { kfree(rx_qid_to_stats_idx); return; } + for (ring = 0; ring < num_tx_queues; ring++) { + tx_qid_to_stats_idx[ring] = -1; + if (!priv->ntfy_blocks[gve_tx_idx_to_ntfy(priv, ring)].tx) + num_stopped_txqs++; + } + for (rx_pkts = 0, rx_bytes = 0, rx_skb_alloc_fail = 0, rx_buf_alloc_fail = 0, rx_desc_err_dropped_pkt = 0, xdp_tx_errors = 0, xdp_redirect_errors = 0, @@ -272,11 +285,16 @@ gve_get_ethtool_stats(struct net_device *netdev, */ rx_base_stats_idx = (stats_region_len - nic_stats_len) / sizeof(struct stats); + /* The boundary between driver stats and NIC stats shifts if there are + * stopped queues. + */ + rx_base_stats_idx += NIC_RX_STATS_REPORT_NUM * num_stopped_rxqs + + NIC_TX_STATS_REPORT_NUM * num_stopped_txqs; max_rx_stats_idx = NIC_RX_STATS_REPORT_NUM * - priv->rx_cfg.num_queues + + (priv->rx_cfg.num_queues - num_stopped_rxqs) + rx_base_stats_idx; max_tx_stats_idx = NIC_TX_STATS_REPORT_NUM * - num_tx_queues + + (num_tx_queues - num_stopped_txqs) + max_rx_stats_idx; } /* Preprocess the stats report for rx, map queue id to start index */ @@ -291,6 +309,10 @@ gve_get_ethtool_stats(struct net_device *netdev, skip_nic_stats = true; break; } + if (queue_id < 0 || queue_id >= priv->rx_cfg.num_queues) { + net_err_ratelimited("Invalid rxq id in NIC stats\n"); + continue; + } rx_qid_to_stats_idx[queue_id] = stats_idx; } /* walk RX rings */ @@ -327,11 +349,11 @@ gve_get_ethtool_stats(struct net_device *netdev, data[i++] = rx->rx_copybreak_pkt; data[i++] = rx->rx_copied_pkt; /* stats from NIC */ - if (skip_nic_stats) { + stats_idx = rx_qid_to_stats_idx[ring]; + if (skip_nic_stats || stats_idx < 0) { /* skip NIC rx stats */ i += NIC_RX_STATS_REPORT_NUM; } else { - stats_idx = rx_qid_to_stats_idx[ring]; for (j = 0; j < NIC_RX_STATS_REPORT_NUM; j++) { u64 value = be64_to_cpu(report_stats[stats_idx + j].value); @@ -367,6 +389,10 @@ gve_get_ethtool_stats(struct net_device *netdev, skip_nic_stats = true; break; } + if (queue_id < 0 || queue_id >= num_tx_queues) { + net_err_ratelimited("Invalid txq id in NIC stats\n"); + continue; + } tx_qid_to_stats_idx[queue_id] = stats_idx; } /* walk TX rings */ @@ -400,11 +426,11 @@ gve_get_ethtool_stats(struct net_device *netdev, data[i++] = gve_tx_load_event_counter(priv, tx); data[i++] = tx->dma_mapping_error; /* stats from NIC */ - if (skip_nic_stats) { + stats_idx = tx_qid_to_stats_idx[ring]; + if (skip_nic_stats || stats_idx < 0) { /* skip NIC tx stats */ i += NIC_TX_STATS_REPORT_NUM; } else { - stats_idx = tx_qid_to_stats_idx[ring]; for (j = 0; j < NIC_TX_STATS_REPORT_NUM; j++) { u64 value = be64_to_cpu(report_stats[stats_idx + j].value); diff --git a/drivers/net/ethernet/google/gve/gve_main.c b/drivers/net/ethernet/google/gve/gve_main.c index 9a287a376479..5f966f0a049d 100644 --- a/drivers/net/ethernet/google/gve/gve_main.c +++ b/drivers/net/ethernet/google/gve/gve_main.c @@ -1272,17 +1272,10 @@ static void gve_unreg_xdp_info(struct gve_priv *priv) static void gve_drain_page_cache(struct gve_priv *priv) { - struct page_frag_cache *nc; int i; - for (i = 0; i < priv->rx_cfg.num_queues; i++) { - nc = &priv->rx[i].page_cache; - if (nc->va) { - __page_frag_cache_drain(virt_to_page(nc->va), - nc->pagecnt_bias); - nc->va = NULL; - } - } + for (i = 0; i < priv->rx_cfg.num_queues; i++) + page_frag_cache_drain(&priv->rx[i].page_cache); } static int gve_open(struct net_device *dev) diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c index 107e692e8c87..9ad557650770 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3_enet.c @@ -2491,9 +2491,9 @@ static netdev_features_t hns3_features_check(struct sk_buff *skb, return features; if (skb->encapsulation) - len = skb_inner_transport_header(skb) - skb->data; + len = skb_inner_transport_offset(skb); else - len = skb_transport_header(skb) - skb->data; + len = skb_transport_offset(skb); /* Assume L4 is 60 byte as TCP is the only protocol with a * a flexible value, and it's max len is 60 bytes. diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_regs.c b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_regs.c index 43c1c18fa81f..8c057192aae6 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_regs.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3pf/hclge_regs.c @@ -510,9 +510,9 @@ static int hclge_get_dfx_reg(struct hclge_dev *hdev, void *data) static int hclge_fetch_pf_reg(struct hclge_dev *hdev, void *data, struct hnae3_knic_private_info *kinfo) { -#define HCLGE_RING_REG_OFFSET 0x200 #define HCLGE_RING_INT_REG_OFFSET 0x4 + struct hnae3_queue *tqp; int i, j, reg_num; int data_num_sum; u32 *reg = data; @@ -533,10 +533,11 @@ static int hclge_fetch_pf_reg(struct hclge_dev *hdev, void *data, reg_num = ARRAY_SIZE(ring_reg_addr_list); for (j = 0; j < kinfo->num_tqps; j++) { reg += hclge_reg_get_tlv(HCLGE_REG_TAG_RING, reg_num, reg); + tqp = kinfo->tqp[j]; for (i = 0; i < reg_num; i++) - *reg++ = hclge_read_dev(&hdev->hw, - ring_reg_addr_list[i] + - HCLGE_RING_REG_OFFSET * j); + *reg++ = readl_relaxed(tqp->io_base - + HCLGE_TQP_REG_OFFSET + + ring_reg_addr_list[i]); } data_num_sum += (reg_num + HCLGE_REG_TLV_SPACE) * kinfo->num_tqps; diff --git a/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_regs.c b/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_regs.c index 65b9dcd38137..6ecf936c79b2 100644 --- a/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_regs.c +++ b/drivers/net/ethernet/hisilicon/hns3/hns3vf/hclgevf_regs.c @@ -123,10 +123,10 @@ int hclgevf_get_regs_len(struct hnae3_handle *handle) void hclgevf_get_regs(struct hnae3_handle *handle, u32 *version, void *data) { -#define HCLGEVF_RING_REG_OFFSET 0x200 #define HCLGEVF_RING_INT_REG_OFFSET 0x4 struct hclgevf_dev *hdev = hclgevf_ae_get_hdev(handle); + struct hnae3_queue *tqp; int i, j, reg_um; u32 *reg = data; @@ -147,10 +147,11 @@ void hclgevf_get_regs(struct hnae3_handle *handle, u32 *version, reg_um = sizeof(ring_reg_addr_list) / sizeof(u32); for (j = 0; j < hdev->num_tqps; j++) { reg += hclgevf_reg_get_tlv(HCLGEVF_REG_TAG_RING, reg_um, reg); + tqp = &hdev->htqp[j].q; for (i = 0; i < reg_um; i++) - *reg++ = hclgevf_read_dev(&hdev->hw, - ring_reg_addr_list[i] + - HCLGEVF_RING_REG_OFFSET * j); + *reg++ = readl_relaxed(tqp->io_base - + HCLGEVF_TQP_REG_OFFSET + + ring_reg_addr_list[i]); } reg_um = sizeof(tqp_intr_reg_addr_list) / sizeof(u32); diff --git a/drivers/net/ethernet/ibm/ibmveth.c b/drivers/net/ethernet/ibm/ibmveth.c index be12eaf080e9..69e8233920a1 100644 --- a/drivers/net/ethernet/ibm/ibmveth.c +++ b/drivers/net/ethernet/ibm/ibmveth.c @@ -465,10 +465,15 @@ static void ibmveth_rxq_harvest_buffer(struct ibmveth_adapter *adapter) static void ibmveth_free_tx_ltb(struct ibmveth_adapter *adapter, int idx) { + void *ptr = adapter->tx_ltb_ptr[idx]; + + if (!ptr) + return; + + adapter->tx_ltb_ptr[idx] = NULL; dma_unmap_single(&adapter->vdev->dev, adapter->tx_ltb_dma[idx], adapter->tx_ltb_size, DMA_TO_DEVICE); - kfree(adapter->tx_ltb_ptr[idx]); - adapter->tx_ltb_ptr[idx] = NULL; + kfree(ptr); } static int ibmveth_allocate_tx_ltb(struct ibmveth_adapter *adapter, int idx) @@ -579,7 +584,7 @@ static int ibmveth_open(struct net_device *netdev) for (i = 0; i < netdev->real_num_tx_queues; i++) { if (ibmveth_allocate_tx_ltb(adapter, i)) - goto out_free_tx_ltb; + goto out_unmap_filter_list; } adapter->rx_queue.index = 0; @@ -630,10 +635,6 @@ static int ibmveth_open(struct net_device *netdev) if (rc != 0) { netdev_err(netdev, "unable to request irq 0x%x, rc %d\n", netdev->irq, rc); - do { - lpar_rc = h_free_logical_lan(adapter->vdev->unit_address); - } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY)); - goto out_free_buffer_pools; } @@ -649,6 +650,9 @@ static int ibmveth_open(struct net_device *netdev) return 0; out_free_buffer_pools: + do { + lpar_rc = h_free_logical_lan(adapter->vdev->unit_address); + } while (H_IS_LONG_BUSY(lpar_rc) || (lpar_rc == H_BUSY)); while (--i >= 0) { if (adapter->rx_buff_pool[i].active) ibmveth_free_buffer_pool(adapter, @@ -658,10 +662,8 @@ static int ibmveth_open(struct net_device *netdev) dma_unmap_single(dev, adapter->filter_list_dma, 4096, DMA_BIDIRECTIONAL); -out_free_tx_ltb: - while (--i >= 0) { + for (i = netdev->real_num_tx_queues - 1; i >= 0; i--) ibmveth_free_tx_ltb(adapter, i); - } out_unmap_buffer_list: dma_unmap_single(dev, adapter->buffer_list_dma, 4096, diff --git a/drivers/net/ethernet/intel/i40e/i40e_main.c b/drivers/net/ethernet/intel/i40e/i40e_main.c index 063f9ced3ffd..7d575e857d38 100644 --- a/drivers/net/ethernet/intel/i40e/i40e_main.c +++ b/drivers/net/ethernet/intel/i40e/i40e_main.c @@ -13362,7 +13362,7 @@ static netdev_features_t i40e_features_check(struct sk_buff *skb, features &= ~NETIF_F_GSO_MASK; /* MACLEN can support at most 63 words */ - len = skb_network_header(skb) - skb->data; + len = skb_network_offset(skb); if (len & ~(63 * 2)) goto out_err; diff --git a/drivers/net/ethernet/intel/iavf/iavf_ethtool.c b/drivers/net/ethernet/intel/iavf/iavf_ethtool.c index 1ac97bd606e3..f2288e2aedb0 100644 --- a/drivers/net/ethernet/intel/iavf/iavf_ethtool.c +++ b/drivers/net/ethernet/intel/iavf/iavf_ethtool.c @@ -334,14 +334,13 @@ static int iavf_get_sset_count(struct net_device *netdev, int sset) { /* Report the maximum number queues, even if not every queue is * currently configured. Since allocation of queues is in pairs, - * use netdev->real_num_tx_queues * 2. The real_num_tx_queues is set - * at device creation and never changes. + * use netdev->num_tx_queues * 2. The num_tx_queues is set at + * device creation and never changes. */ if (sset == ETH_SS_STATS) return IAVF_STATS_LEN + - (IAVF_QUEUE_STATS_LEN * 2 * - netdev->real_num_tx_queues); + (IAVF_QUEUE_STATS_LEN * 2 * netdev->num_tx_queues); else if (sset == ETH_SS_PRIV_FLAGS) return IAVF_PRIV_FLAGS_STR_LEN; else @@ -368,19 +367,19 @@ static void iavf_get_ethtool_stats(struct net_device *netdev, iavf_add_ethtool_stats(&data, adapter, iavf_gstrings_stats); rcu_read_lock(); - /* As num_active_queues describe both tx and rx queues, we can use - * it to iterate over rings' stats. + /* Use num_tx_queues to report stats for the maximum number of queues. + * Queues beyond num_active_queues will report zero. */ - for (i = 0; i < adapter->num_active_queues; i++) { - struct iavf_ring *ring; + for (i = 0; i < netdev->num_tx_queues; i++) { + struct iavf_ring *tx_ring = NULL, *rx_ring = NULL; - /* Tx rings stats */ - ring = &adapter->tx_rings[i]; - iavf_add_queue_stats(&data, ring); + if (i < adapter->num_active_queues) { + tx_ring = &adapter->tx_rings[i]; + rx_ring = &adapter->rx_rings[i]; + } - /* Rx rings stats */ - ring = &adapter->rx_rings[i]; - iavf_add_queue_stats(&data, ring); + iavf_add_queue_stats(&data, tx_ring); + iavf_add_queue_stats(&data, rx_ring); } rcu_read_unlock(); } @@ -417,9 +416,9 @@ static void iavf_get_stat_strings(struct net_device *netdev, u8 *data) iavf_add_stat_strings(&data, iavf_gstrings_stats); /* Queues are always allocated in pairs, so we just use - * real_num_tx_queues for both Tx and Rx queues. + * num_tx_queues for both Tx and Rx queues. */ - for (i = 0; i < netdev->real_num_tx_queues; i++) { + for (i = 0; i < netdev->num_tx_queues; i++) { iavf_add_stat_strings(&data, iavf_gstrings_queue_stats, "tx", i); iavf_add_stat_strings(&data, iavf_gstrings_queue_stats, diff --git a/drivers/net/ethernet/intel/iavf/iavf_main.c b/drivers/net/ethernet/intel/iavf/iavf_main.c index b9b855a35fc8..3ddae9296461 100644 --- a/drivers/net/ethernet/intel/iavf/iavf_main.c +++ b/drivers/net/ethernet/intel/iavf/iavf_main.c @@ -4507,7 +4507,7 @@ static netdev_features_t iavf_features_check(struct sk_buff *skb, features &= ~NETIF_F_GSO_MASK; /* MACLEN can support at most 63 words */ - len = skb_network_header(skb) - skb->data; + len = skb_network_offset(skb); if (len & ~(63 * 2)) goto out_err; diff --git a/drivers/net/ethernet/intel/ice/ice_base.c b/drivers/net/ethernet/intel/ice/ice_base.c index 9a0682b05c4f..f3a2d564da18 100644 --- a/drivers/net/ethernet/intel/ice/ice_base.c +++ b/drivers/net/ethernet/intel/ice/ice_base.c @@ -557,13 +557,16 @@ int ice_vsi_cfg_rxq(struct ice_rx_ring *ring) err = __xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev, ring->q_index, ring->q_vector->napi.napi_id, - ring->rx_buf_len); + ice_get_frame_sz(ring)); if (err) return err; } ring->xsk_pool = ice_xsk_pool(ring); if (ring->xsk_pool) { + u32 frag_size = + xsk_pool_get_rx_frag_step(ring->xsk_pool); + xdp_rxq_info_unreg(&ring->xdp_rxq); ring->rx_buf_len = @@ -571,7 +574,7 @@ int ice_vsi_cfg_rxq(struct ice_rx_ring *ring) err = __xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev, ring->q_index, ring->q_vector->napi.napi_id, - ring->rx_buf_len); + frag_size); if (err) return err; err = xdp_rxq_info_reg_mem_model(&ring->xdp_rxq, @@ -588,7 +591,7 @@ int ice_vsi_cfg_rxq(struct ice_rx_ring *ring) err = __xdp_rxq_info_reg(&ring->xdp_rxq, ring->netdev, ring->q_index, ring->q_vector->napi.napi_id, - ring->rx_buf_len); + ice_get_frame_sz(ring)); if (err) return err; } diff --git a/drivers/net/ethernet/intel/ice/ice_eswitch.c b/drivers/net/ethernet/intel/ice/ice_eswitch.c index e7f1e53314d7..528ca4be2b64 100644 --- a/drivers/net/ethernet/intel/ice/ice_eswitch.c +++ b/drivers/net/ethernet/intel/ice/ice_eswitch.c @@ -207,7 +207,7 @@ ice_eswitch_release_reprs(struct ice_pf *pf, struct ice_vsi *ctrl_vsi) continue; ice_vsi_update_security(vsi, ice_vsi_ctx_set_antispoof); - metadata_dst_free(vf->repr->dst); + dst_release(&vf->repr->dst->dst); vf->repr->dst = NULL; ice_eswitch_del_vf_sp_rule(vf); ice_fltr_add_mac_and_broadcast(vsi, vf->hw_lan_addr, diff --git a/drivers/net/ethernet/intel/ice/ice_ptp.c b/drivers/net/ethernet/intel/ice/ice_ptp.c index 0da6b47f5b52..df88304c9a1b 100644 --- a/drivers/net/ethernet/intel/ice/ice_ptp.c +++ b/drivers/net/ethernet/intel/ice/ice_ptp.c @@ -1477,6 +1477,10 @@ void ice_ptp_extts_event(struct ice_pf *pf) u8 chan, tmr_idx; u32 hi, lo; + /* Don't process timestamp events if PTP is not ready */ + if (!test_bit(ICE_FLAG_PTP, pf->flags)) + return; + tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned; /* Event time is captured by one of the two matched registers * GLTSYN_EVNT_L: 32 LSB of sampled time event @@ -1502,27 +1506,24 @@ void ice_ptp_extts_event(struct ice_pf *pf) /** * ice_ptp_cfg_extts - Configure EXTTS pin and channel * @pf: Board private structure - * @ena: true to enable; false to disable * @chan: GPIO channel (0-3) - * @gpio_pin: GPIO pin - * @extts_flags: request flags from the ptp_extts_request.flags + * @config: desired EXTTS configuration. + * @store: If set to true, the values will be stored + * + * Configure an external timestamp event on the requested channel. */ -static int -ice_ptp_cfg_extts(struct ice_pf *pf, bool ena, unsigned int chan, u32 gpio_pin, - unsigned int extts_flags) +static void ice_ptp_cfg_extts(struct ice_pf *pf, unsigned int chan, + struct ice_extts_channel *config, bool store) { u32 func, aux_reg, gpio_reg, irq_reg; struct ice_hw *hw = &pf->hw; u8 tmr_idx; - if (chan > (unsigned int)pf->ptp.info.n_ext_ts) - return -EINVAL; - tmr_idx = hw->func_caps.ts_func_info.tmr_index_owned; irq_reg = rd32(hw, PFINT_OICR_ENA); - if (ena) { + if (config->ena) { /* Enable the interrupt */ irq_reg |= PFINT_OICR_TSYN_EVNT_M; aux_reg = GLTSYN_AUX_IN_0_INT_ENA_M; @@ -1531,9 +1532,9 @@ ice_ptp_cfg_extts(struct ice_pf *pf, bool ena, unsigned int chan, u32 gpio_pin, #define GLTSYN_AUX_IN_0_EVNTLVL_FALLING_EDGE BIT(1) /* set event level to requested edge */ - if (extts_flags & PTP_FALLING_EDGE) + if (config->flags & PTP_FALLING_EDGE) aux_reg |= GLTSYN_AUX_IN_0_EVNTLVL_FALLING_EDGE; - if (extts_flags & PTP_RISING_EDGE) + if (config->flags & PTP_RISING_EDGE) aux_reg |= GLTSYN_AUX_IN_0_EVNTLVL_RISING_EDGE; /* Write GPIO CTL reg. @@ -1555,9 +1556,47 @@ ice_ptp_cfg_extts(struct ice_pf *pf, bool ena, unsigned int chan, u32 gpio_pin, wr32(hw, PFINT_OICR_ENA, irq_reg); wr32(hw, GLTSYN_AUX_IN(chan, tmr_idx), aux_reg); - wr32(hw, GLGEN_GPIO_CTL(gpio_pin), gpio_reg); + wr32(hw, GLGEN_GPIO_CTL(config->gpio_pin), gpio_reg); - return 0; + if (store) + memcpy(&pf->ptp.extts_channels[chan], config, sizeof(*config)); +} + +/** + * ice_ptp_disable_all_extts - Disable all EXTTS channels + * @pf: Board private structure + */ +static void ice_ptp_disable_all_extts(struct ice_pf *pf) +{ + struct ice_extts_channel extts_cfg = {}; + int i; + + for (i = 0; i < pf->ptp.info.n_ext_ts; i++) { + if (pf->ptp.extts_channels[i].ena) { + extts_cfg.gpio_pin = pf->ptp.extts_channels[i].gpio_pin; + extts_cfg.ena = false; + ice_ptp_cfg_extts(pf, i, &extts_cfg, false); + } + } + + synchronize_irq(pf->oicr_irq.virq); +} + +/** + * ice_ptp_enable_all_extts - Enable all EXTTS channels + * @pf: Board private structure + * + * Called during reset to restore user configuration. + */ +static void ice_ptp_enable_all_extts(struct ice_pf *pf) +{ + int i; + + for (i = 0; i < pf->ptp.info.n_ext_ts; i++) { + if (pf->ptp.extts_channels[i].ena) + ice_ptp_cfg_extts(pf, i, &pf->ptp.extts_channels[i], + false); + } } /** @@ -1714,7 +1753,6 @@ ice_ptp_gpio_enable_e810(struct ptp_clock_info *info, struct ptp_clock_request *rq, int on) { struct ice_pf *pf = ptp_info_to_pf(info); - struct ice_perout_channel clk_cfg = {0}; bool sma_pres = false; unsigned int chan; u32 gpio_pin; @@ -1725,6 +1763,9 @@ ice_ptp_gpio_enable_e810(struct ptp_clock_info *info, switch (rq->type) { case PTP_CLK_REQ_PEROUT: + { + struct ice_perout_channel clk_cfg = {}; + chan = rq->perout.index; if (sma_pres) { if (chan == ice_pin_desc_e810t[SMA1].chan) @@ -1752,7 +1793,11 @@ ice_ptp_gpio_enable_e810(struct ptp_clock_info *info, err = ice_ptp_cfg_clkout(pf, chan, &clk_cfg, true); break; + } case PTP_CLK_REQ_EXTTS: + { + struct ice_extts_channel extts_cfg = {}; + chan = rq->extts.index; if (sma_pres) { if (chan < ice_pin_desc_e810t[SMA2].chan) @@ -1768,9 +1813,13 @@ ice_ptp_gpio_enable_e810(struct ptp_clock_info *info, gpio_pin = chan; } - err = ice_ptp_cfg_extts(pf, !!on, chan, gpio_pin, - rq->extts.flags); - break; + extts_cfg.flags = rq->extts.flags; + extts_cfg.gpio_pin = gpio_pin; + extts_cfg.ena = !!on; + + ice_ptp_cfg_extts(pf, chan, &extts_cfg, true); + return 0; + } default: return -EOPNOTSUPP; } @@ -1788,21 +1837,31 @@ static int ice_ptp_gpio_enable_e823(struct ptp_clock_info *info, struct ptp_clock_request *rq, int on) { struct ice_pf *pf = ptp_info_to_pf(info); - struct ice_perout_channel clk_cfg = {0}; int err; switch (rq->type) { case PTP_CLK_REQ_PPS: + { + struct ice_perout_channel clk_cfg = {}; + clk_cfg.gpio_pin = PPS_PIN_INDEX; clk_cfg.period = NSEC_PER_SEC; clk_cfg.ena = !!on; err = ice_ptp_cfg_clkout(pf, PPS_CLK_GEN_CHAN, &clk_cfg, true); break; + } case PTP_CLK_REQ_EXTTS: - err = ice_ptp_cfg_extts(pf, !!on, rq->extts.index, - TIME_SYNC_PIN_INDEX, rq->extts.flags); - break; + { + struct ice_extts_channel extts_cfg = {}; + + extts_cfg.flags = rq->extts.flags; + extts_cfg.gpio_pin = TIME_SYNC_PIN_INDEX; + extts_cfg.ena = !!on; + + ice_ptp_cfg_extts(pf, rq->extts.index, &extts_cfg, true); + return 0; + } default: return -EOPNOTSUPP; } @@ -2522,6 +2581,10 @@ void ice_ptp_reset(struct ice_pf *pf) goto err; } + /* Re-enable all periodic outputs and external timestamp events */ + ice_ptp_enable_all_clkout(pf); + ice_ptp_enable_all_extts(pf); + reset_ts: /* Restart the PHY timestamping block */ ice_ptp_reset_phy_timestamping(pf); @@ -2769,6 +2832,8 @@ void ice_ptp_release(struct ice_pf *pf) clear_bit(ICE_FLAG_PTP, pf->flags); + ice_ptp_disable_all_extts(pf); + kthread_cancel_delayed_work_sync(&pf->ptp.work); ice_ptp_port_phy_stop(&pf->ptp.port); diff --git a/drivers/net/ethernet/intel/ice/ice_ptp.h b/drivers/net/ethernet/intel/ice/ice_ptp.h index 995a57019ba7..3dd7584be0fa 100644 --- a/drivers/net/ethernet/intel/ice/ice_ptp.h +++ b/drivers/net/ethernet/intel/ice/ice_ptp.h @@ -33,6 +33,12 @@ struct ice_perout_channel { u64 start_time; }; +struct ice_extts_channel { + bool ena; + u32 gpio_pin; + u32 flags; +}; + /* The ice hardware captures Tx hardware timestamps in the PHY. The timestamp * is stored in a buffer of registers. Depending on the specific hardware, * this buffer might be shared across multiple PHY ports. @@ -185,6 +191,7 @@ struct ice_ptp_port { * @ext_ts_irq: the external timestamp IRQ in use * @kworker: kwork thread for handling periodic work * @perout_channels: periodic output data + * @extts_channels: channels for external timestamps * @info: structure defining PTP hardware capabilities * @clock: pointer to registered PTP clock device * @tstamp_config: hardware timestamping configuration @@ -205,6 +212,7 @@ struct ice_ptp { u8 ext_ts_irq; struct kthread_worker *kworker; struct ice_perout_channel perout_channels[GLTSYN_TGT_H_IDX_MAX]; + struct ice_extts_channel extts_channels[GLTSYN_TGT_H_IDX_MAX]; struct ptp_clock_info info; struct ptp_clock *clock; struct hwtstamp_config tstamp_config; diff --git a/drivers/net/ethernet/intel/ice/ice_trace.h b/drivers/net/ethernet/intel/ice/ice_trace.h index a41e210a310d..aac117b8d443 100644 --- a/drivers/net/ethernet/intel/ice/ice_trace.h +++ b/drivers/net/ethernet/intel/ice/ice_trace.h @@ -63,23 +63,33 @@ DECLARE_EVENT_CLASS(ice_rx_dim_template, TP_PROTO(struct ice_q_vector *q_vector, struct dim *dim), TP_ARGS(q_vector, dim), - TP_STRUCT__entry(__field(struct ice_q_vector *, q_vector) - __field(struct dim *, dim) + TP_STRUCT__entry(__field(u16, q_index) + __field(u8, state) + __field(u8, profile_ix) + __field(u8, tune_state) + __field(u8, steps_right) + __field(u8, steps_left) + __field(u8, tired) __string(devname, q_vector->rx.rx_ring->netdev->name)), - TP_fast_assign(__entry->q_vector = q_vector; - __entry->dim = dim; + TP_fast_assign(__entry->q_index = q_vector->rx.rx_ring->q_index; + __entry->state = dim->state; + __entry->profile_ix = dim->profile_ix; + __entry->tune_state = dim->tune_state; + __entry->steps_right = dim->steps_right; + __entry->steps_left = dim->steps_left; + __entry->tired = dim->tired; __assign_str(devname, q_vector->rx.rx_ring->netdev->name);), TP_printk("netdev: %s Rx-Q: %d dim-state: %d dim-profile: %d dim-tune: %d dim-st-right: %d dim-st-left: %d dim-tired: %d", __get_str(devname), - __entry->q_vector->rx.rx_ring->q_index, - __entry->dim->state, - __entry->dim->profile_ix, - __entry->dim->tune_state, - __entry->dim->steps_right, - __entry->dim->steps_left, - __entry->dim->tired) + __entry->q_index, + __entry->state, + __entry->profile_ix, + __entry->tune_state, + __entry->steps_right, + __entry->steps_left, + __entry->tired) ); DEFINE_EVENT(ice_rx_dim_template, ice_rx_dim_work, @@ -90,23 +100,33 @@ DEFINE_EVENT(ice_rx_dim_template, ice_rx_dim_work, DECLARE_EVENT_CLASS(ice_tx_dim_template, TP_PROTO(struct ice_q_vector *q_vector, struct dim *dim), TP_ARGS(q_vector, dim), - TP_STRUCT__entry(__field(struct ice_q_vector *, q_vector) - __field(struct dim *, dim) + TP_STRUCT__entry(__field(u16, q_index) + __field(u8, state) + __field(u8, profile_ix) + __field(u8, tune_state) + __field(u8, steps_right) + __field(u8, steps_left) + __field(u8, tired) __string(devname, q_vector->tx.tx_ring->netdev->name)), - TP_fast_assign(__entry->q_vector = q_vector; - __entry->dim = dim; + TP_fast_assign(__entry->q_index = q_vector->tx.tx_ring->q_index; + __entry->state = dim->state; + __entry->profile_ix = dim->profile_ix; + __entry->tune_state = dim->tune_state; + __entry->steps_right = dim->steps_right; + __entry->steps_left = dim->steps_left; + __entry->tired = dim->tired; __assign_str(devname, q_vector->tx.tx_ring->netdev->name);), TP_printk("netdev: %s Tx-Q: %d dim-state: %d dim-profile: %d dim-tune: %d dim-st-right: %d dim-st-left: %d dim-tired: %d", __get_str(devname), - __entry->q_vector->tx.tx_ring->q_index, - __entry->dim->state, - __entry->dim->profile_ix, - __entry->dim->tune_state, - __entry->dim->steps_right, - __entry->dim->steps_left, - __entry->dim->tired) + __entry->q_index, + __entry->state, + __entry->profile_ix, + __entry->tune_state, + __entry->steps_right, + __entry->steps_left, + __entry->tired) ); DEFINE_EVENT(ice_tx_dim_template, ice_tx_dim_work, diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.c b/drivers/net/ethernet/intel/ice/ice_txrx.c index 3d4caf047554..0139c393980a 100644 --- a/drivers/net/ethernet/intel/ice/ice_txrx.c +++ b/drivers/net/ethernet/intel/ice/ice_txrx.c @@ -865,10 +865,6 @@ ice_add_xdp_frag(struct ice_rx_ring *rx_ring, struct xdp_buff *xdp, __skb_fill_page_desc_noacc(sinfo, sinfo->nr_frags++, rx_buf->page, rx_buf->page_offset, size); sinfo->xdp_frags_size += size; - /* remember frag count before XDP prog execution; bpf_xdp_adjust_tail() - * can pop off frags but driver has to handle it on its own - */ - rx_ring->nr_frags = sinfo->nr_frags; if (page_is_pfmemalloc(rx_buf->page)) xdp_buff_set_frag_pfmemalloc(xdp); @@ -939,20 +935,20 @@ ice_get_rx_buf(struct ice_rx_ring *rx_ring, const unsigned int size, /** * ice_get_pgcnts - grab page_count() for gathered fragments * @rx_ring: Rx descriptor ring to store the page counts on + * @ntc: the next to clean element (not included in this frame!) * * This function is intended to be called right before running XDP * program so that the page recycling mechanism will be able to take * a correct decision regarding underlying pages; this is done in such * way as XDP program can change the refcount of page */ -static void ice_get_pgcnts(struct ice_rx_ring *rx_ring) +static void ice_get_pgcnts(struct ice_rx_ring *rx_ring, unsigned int ntc) { - u32 nr_frags = rx_ring->nr_frags + 1; u32 idx = rx_ring->first_desc; struct ice_rx_buf *rx_buf; u32 cnt = rx_ring->count; - for (int i = 0; i < nr_frags; i++) { + while (idx != ntc) { rx_buf = &rx_ring->rx_buf[idx]; rx_buf->pgcnt = page_count(rx_buf->page); @@ -1126,62 +1122,51 @@ ice_put_rx_buf(struct ice_rx_ring *rx_ring, struct ice_rx_buf *rx_buf) } /** - * ice_put_rx_mbuf - ice_put_rx_buf() caller, for all frame frags + * ice_put_rx_mbuf - ice_put_rx_buf() caller, for all buffers in frame * @rx_ring: Rx ring with all the auxiliary data * @xdp: XDP buffer carrying linear + frags part - * @xdp_xmit: XDP_TX/XDP_REDIRECT verdict storage - * @ntc: a current next_to_clean value to be stored at rx_ring + * @ntc: the next to clean element (not included in this frame!) * @verdict: return code from XDP program execution * - * Walk through gathered fragments and satisfy internal page - * recycle mechanism; we take here an action related to verdict - * returned by XDP program; + * Called after XDP program is completed, or on error with verdict set to + * ICE_XDP_CONSUMED. + * + * Walk through buffers from first_desc to the end of the frame, releasing + * buffers and satisfying internal page recycle mechanism. The action depends + * on verdict from XDP program. */ static void ice_put_rx_mbuf(struct ice_rx_ring *rx_ring, struct xdp_buff *xdp, - u32 *xdp_xmit, u32 ntc, u32 verdict) + u32 ntc, u32 verdict) { - u32 nr_frags = rx_ring->nr_frags + 1; u32 idx = rx_ring->first_desc; u32 cnt = rx_ring->count; - u32 post_xdp_frags = 1; struct ice_rx_buf *buf; - int i; + u32 xdp_frags = 0; + int i = 0; if (unlikely(xdp_buff_has_frags(xdp))) - post_xdp_frags += xdp_get_shared_info_from_buff(xdp)->nr_frags; + xdp_frags = xdp_get_shared_info_from_buff(xdp)->nr_frags; - for (i = 0; i < post_xdp_frags; i++) { + while (idx != ntc) { buf = &rx_ring->rx_buf[idx]; + if (++idx == cnt) + idx = 0; - if (verdict & (ICE_XDP_TX | ICE_XDP_REDIR)) { + /* An XDP program could release fragments from the end of the + * buffer. For these, we need to keep the pagecnt_bias as-is. + * To do this, only adjust pagecnt_bias for fragments up to + * the total remaining after the XDP program has run. + */ + if (verdict != ICE_XDP_CONSUMED) ice_rx_buf_adjust_pg_offset(buf, xdp->frame_sz); - *xdp_xmit |= verdict; - } else if (verdict & ICE_XDP_CONSUMED) { + else if (i++ <= xdp_frags) buf->pagecnt_bias++; - } else if (verdict == ICE_XDP_PASS) { - ice_rx_buf_adjust_pg_offset(buf, xdp->frame_sz); - } ice_put_rx_buf(rx_ring, buf); - - if (++idx == cnt) - idx = 0; - } - /* handle buffers that represented frags released by XDP prog; - * for these we keep pagecnt_bias as-is; refcount from struct page - * has been decremented within XDP prog and we do not have to increase - * the biased refcnt - */ - for (; i < nr_frags; i++) { - buf = &rx_ring->rx_buf[idx]; - ice_put_rx_buf(rx_ring, buf); - if (++idx == cnt) - idx = 0; } xdp->data = NULL; rx_ring->first_desc = ntc; - rx_ring->nr_frags = 0; } /** @@ -1262,6 +1247,10 @@ int ice_clean_rx_irq(struct ice_rx_ring *rx_ring, int budget) /* retrieve a buffer from the ring */ rx_buf = ice_get_rx_buf(rx_ring, size, ntc); + /* Increment ntc before calls to ice_put_rx_mbuf() */ + if (++ntc == cnt) + ntc = 0; + if (!xdp->data) { void *hard_start; @@ -1270,24 +1259,23 @@ int ice_clean_rx_irq(struct ice_rx_ring *rx_ring, int budget) xdp_prepare_buff(xdp, hard_start, offset, size, !!offset); xdp_buff_clear_frags_flag(xdp); } else if (ice_add_xdp_frag(rx_ring, xdp, rx_buf, size)) { - ice_put_rx_mbuf(rx_ring, xdp, NULL, ntc, ICE_XDP_CONSUMED); + ice_put_rx_mbuf(rx_ring, xdp, ntc, ICE_XDP_CONSUMED); break; } - if (++ntc == cnt) - ntc = 0; /* skip if it is NOP desc */ if (ice_is_non_eop(rx_ring, rx_desc)) continue; - ice_get_pgcnts(rx_ring); + ice_get_pgcnts(rx_ring, ntc); xdp_verdict = ice_run_xdp(rx_ring, xdp, xdp_prog, xdp_ring, rx_desc); if (xdp_verdict == ICE_XDP_PASS) goto construct_skb; total_rx_bytes += xdp_get_buff_len(xdp); total_rx_pkts++; - ice_put_rx_mbuf(rx_ring, xdp, &xdp_xmit, ntc, xdp_verdict); + ice_put_rx_mbuf(rx_ring, xdp, ntc, xdp_verdict); + xdp_xmit |= xdp_verdict & (ICE_XDP_TX | ICE_XDP_REDIR); continue; construct_skb: @@ -1300,7 +1288,7 @@ int ice_clean_rx_irq(struct ice_rx_ring *rx_ring, int budget) rx_ring->ring_stats->rx_stats.alloc_buf_failed++; xdp_verdict = ICE_XDP_CONSUMED; } - ice_put_rx_mbuf(rx_ring, xdp, &xdp_xmit, ntc, xdp_verdict); + ice_put_rx_mbuf(rx_ring, xdp, ntc, xdp_verdict); if (!skb) break; @@ -2362,6 +2350,9 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring) ice_trace(xmit_frame_ring, tx_ring, skb); + /* record the location of the first descriptor for this packet */ + first = &tx_ring->tx_buf[tx_ring->next_to_use]; + if (unlikely(ipv6_hopopt_jumbo_remove(skb))) goto out_drop; @@ -2390,8 +2381,6 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring) offload.tx_ring = tx_ring; - /* record the location of the first descriptor for this packet */ - first = &tx_ring->tx_buf[tx_ring->next_to_use]; first->skb = skb; first->type = ICE_TX_BUF_SKB; first->bytecount = max_t(unsigned int, skb->len, ETH_ZLEN); @@ -2453,6 +2442,7 @@ ice_xmit_frame_ring(struct sk_buff *skb, struct ice_tx_ring *tx_ring) out_drop: ice_trace(xmit_frame_ring_drop, tx_ring, skb); dev_kfree_skb_any(skb); + first->type = ICE_TX_BUF_EMPTY; return NETDEV_TX_OK; } diff --git a/drivers/net/ethernet/intel/ice/ice_txrx.h b/drivers/net/ethernet/intel/ice/ice_txrx.h index 53a155dde3e3..f2f86b90a003 100644 --- a/drivers/net/ethernet/intel/ice/ice_txrx.h +++ b/drivers/net/ethernet/intel/ice/ice_txrx.h @@ -344,7 +344,6 @@ struct ice_rx_ring { struct ice_tx_ring *xdp_ring; struct ice_rx_ring *next; /* pointer to next ring in q_vector */ struct xsk_buff_pool *xsk_pool; - u32 nr_frags; dma_addr_t dma; /* physical address of ring */ u64 cached_phctime; u16 rx_buf_len; diff --git a/drivers/net/ethernet/intel/igb/igb_main.c b/drivers/net/ethernet/intel/igb/igb_main.c index c38be2880efc..43201369226b 100644 --- a/drivers/net/ethernet/intel/igb/igb_main.c +++ b/drivers/net/ethernet/intel/igb/igb_main.c @@ -2543,7 +2543,7 @@ igb_features_check(struct sk_buff *skb, struct net_device *dev, unsigned int network_hdr_len, mac_hdr_len; /* Make certain the headers can be described by a context descriptor */ - mac_hdr_len = skb_network_header(skb) - skb->data; + mac_hdr_len = skb_network_offset(skb); if (unlikely(mac_hdr_len > IGB_MAX_MAC_HDR_LEN)) return features & ~(NETIF_F_HW_CSUM | NETIF_F_SCTP_CRC | diff --git a/drivers/net/ethernet/intel/igbvf/netdev.c b/drivers/net/ethernet/intel/igbvf/netdev.c index 404e8dbe679f..c812e4b0e29f 100644 --- a/drivers/net/ethernet/intel/igbvf/netdev.c +++ b/drivers/net/ethernet/intel/igbvf/netdev.c @@ -2653,7 +2653,7 @@ igbvf_features_check(struct sk_buff *skb, struct net_device *dev, unsigned int network_hdr_len, mac_hdr_len; /* Make certain the headers can be described by a context descriptor */ - mac_hdr_len = skb_network_header(skb) - skb->data; + mac_hdr_len = skb_network_offset(skb); if (unlikely(mac_hdr_len > IGBVF_MAX_MAC_HDR_LEN)) return features & ~(NETIF_F_HW_CSUM | NETIF_F_SCTP_CRC | diff --git a/drivers/net/ethernet/intel/igc/igc_main.c b/drivers/net/ethernet/intel/igc/igc_main.c index 6fdf009df836..b8261a630ca7 100644 --- a/drivers/net/ethernet/intel/igc/igc_main.c +++ b/drivers/net/ethernet/intel/igc/igc_main.c @@ -5269,7 +5269,7 @@ igc_features_check(struct sk_buff *skb, struct net_device *dev, unsigned int network_hdr_len, mac_hdr_len; /* Make certain the headers can be described by a context descriptor */ - mac_hdr_len = skb_network_header(skb) - skb->data; + mac_hdr_len = skb_network_offset(skb); if (unlikely(mac_hdr_len > IGC_MAX_MAC_HDR_LEN)) return features & ~(NETIF_F_HW_CSUM | NETIF_F_SCTP_CRC | diff --git a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c index 99876b765b08..3f990136d05c 100644 --- a/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c +++ b/drivers/net/ethernet/intel/ixgbe/ixgbe_main.c @@ -3514,7 +3514,8 @@ void ixgbe_configure_tx_ring(struct ixgbe_adapter *adapter, } /* initialize XPS */ - if (!test_and_set_bit(__IXGBE_TX_XPS_INIT_DONE, &ring->state)) { + if (!ring_is_xdp(ring) && + !test_and_set_bit(__IXGBE_TX_XPS_INIT_DONE, &ring->state)) { struct ixgbe_q_vector *q_vector = ring->q_vector; if (q_vector) @@ -10205,7 +10206,7 @@ ixgbe_features_check(struct sk_buff *skb, struct net_device *dev, unsigned int network_hdr_len, mac_hdr_len; /* Make certain the headers can be described by a context descriptor */ - mac_hdr_len = skb_network_header(skb) - skb->data; + mac_hdr_len = skb_network_offset(skb); if (unlikely(mac_hdr_len > IXGBE_MAX_MAC_HDR_LEN)) return features & ~(NETIF_F_HW_CSUM | NETIF_F_SCTP_CRC | diff --git a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c index 99d6aa03d4b6..3b5a5f7450a5 100644 --- a/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c +++ b/drivers/net/ethernet/intel/ixgbevf/ixgbevf_main.c @@ -4449,7 +4449,7 @@ ixgbevf_features_check(struct sk_buff *skb, struct net_device *dev, unsigned int network_hdr_len, mac_hdr_len; /* Make certain the headers can be described by a context descriptor */ - mac_hdr_len = skb_network_header(skb) - skb->data; + mac_hdr_len = skb_network_offset(skb); if (unlikely(mac_hdr_len > IXGBEVF_MAX_MAC_HDR_LEN)) return features & ~(NETIF_F_HW_CSUM | NETIF_F_SCTP_CRC | diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c index d415d5dbc110..b62ed5303278 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/cgx.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/cgx.c @@ -518,6 +518,18 @@ int cgx_set_pkind(void *cgxd, u8 lmac_id, int pkind) return 0; } +int cgx_get_pkind(void *cgxd, u8 lmac_id, int *pkind) +{ + struct cgx *cgx = cgxd; + + if (!is_lmac_valid(cgx, lmac_id)) + return -ENODEV; + + *pkind = cgx_read(cgx, lmac_id, cgx->mac_ops->rxid_map_offset); + *pkind = *pkind & 0x3F; + return 0; +} + static u8 cgx_get_lmac_type(void *cgxd, int lmac_id) { struct cgx *cgx = cgxd; diff --git a/drivers/net/ethernet/marvell/octeontx2/af/cgx.h b/drivers/net/ethernet/marvell/octeontx2/af/cgx.h index 1cf12e5c7da8..e8f010ecb284 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/cgx.h +++ b/drivers/net/ethernet/marvell/octeontx2/af/cgx.h @@ -141,6 +141,7 @@ int cgx_get_cgxid(void *cgxd); int cgx_get_lmac_cnt(void *cgxd); void *cgx_get_pdata(int cgx_id); int cgx_set_pkind(void *cgxd, u8 lmac_id, int pkind); +int cgx_get_pkind(void *cgxd, u8 lmac_id, int *pkind); int cgx_lmac_evh_register(struct cgx_event_cb *cb, void *cgxd, int lmac_id); int cgx_lmac_evh_unregister(void *cgxd, int lmac_id); int cgx_get_tx_stats(void *cgxd, int lmac_id, int idx, u64 *tx_stat); diff --git a/drivers/net/ethernet/marvell/octeontx2/af/mcs.c b/drivers/net/ethernet/marvell/octeontx2/af/mcs.c index 211c10aa5880..30c672d526a0 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/mcs.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/mcs.c @@ -314,7 +314,7 @@ int mcs_clear_all_stats(struct mcs *mcs, u16 pcifunc, int dir) } /* Clear SC stats */ - for (id = 0; id < map->secy.max; id++) { + for (id = 0; id < map->sc.max; id++) { if (map->sc2pf_map[id] != pcifunc) continue; mcs_clear_stats(mcs, MCS_SC_STATS, id, dir); @@ -738,7 +738,7 @@ int mcs_free_all_rsrc(struct mcs *mcs, int dir, u16 pcifunc) } /* free sc entries */ - for (id = 0; id < map->secy.max; id++) { + for (id = 0; id < map->sc.max; id++) { if (map->sc2pf_map[id] != pcifunc) continue; mcs_free_rsrc(&map->sc, map->sc2pf_map, id, pcifunc); diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu.h b/drivers/net/ethernet/marvell/octeontx2/af/rvu.h index f26814c49902..042195b87808 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu.h +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu.h @@ -904,6 +904,8 @@ void npc_read_mcam_entry(struct rvu *rvu, struct npc_mcam *mcam, int blkaddr, u16 src, struct mcam_entry *entry, u8 *intf, u8 *ena); bool is_cgx_config_permitted(struct rvu *rvu, u16 pcifunc); +bool rvu_cgx_check_permission_and_set_pkind(struct rvu *rvu, u16 pcifunc, int pkind); +bool rvu_cgx_is_pkind_config_permitted(struct rvu *rvu, u16 pcifunc); bool is_mac_feature_supported(struct rvu *rvu, int pf, int feature); u32 rvu_cgx_get_fifolen(struct rvu *rvu); void *rvu_first_cgx_pdata(struct rvu *rvu); diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c index d14cf2a9d207..5b84f541e6f9 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_cgx.c @@ -1349,3 +1349,82 @@ void rvu_mac_reset(struct rvu *rvu, u16 pcifunc) if (mac_ops->mac_reset(cgxd, lmac, !is_vf(pcifunc))) dev_err(rvu->dev, "Failed to reset MAC\n"); } + +/* Do not allow CGX-mapped VFs to overwrite PKIND when special parse kinds + * (HiGig, EDSA, etc.) are in use on the shared LMAC. VFs must not program + * NPC_TX_DEF_PKIND on NIX_AF_LFX_TX_PARSE_CFG in that case: the PF owns + * parse mode and no separate NPC_TX_HIGIG_PKIND is installed on the VF LF. + * TX-parse callers skip the write when denied; rvu_lf_reset() clears each LF + * before alloc so the next permitted owner programs NPC_TX_DEF_PKIND. + */ +bool rvu_cgx_is_pkind_config_permitted(struct rvu *rvu, u16 pcifunc) +{ + int pf, err, rxpkind; + u8 cgx_id, lmac_id; + void *cgxd; + + pf = rvu_get_pf(pcifunc); + + if (!(pcifunc & RVU_PFVF_FUNC_MASK)) + return true; + + if (!is_pf_cgxmapped(rvu, pf)) + return true; + + rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id); + cgxd = rvu_cgx_pdata(cgx_id, rvu); + err = cgx_get_pkind(cgxd, lmac_id, &rxpkind); + if (err) + return false; + + switch (rxpkind) { + case NPC_RX_HIGIG_PKIND: + case NPC_RX_EDSA_PKIND: + return false; + default: + return true; + } +} + +/* Do not allow CGX-mapped VFs to overwrite PKIND when special parse kinds + * (HiGig, EDSA, etc.) are in use on the shared LMAC. + */ +bool rvu_cgx_check_permission_and_set_pkind(struct rvu *rvu, u16 pcifunc, int pkind) +{ + int pf, err, rxpkind; + u8 cgx_id, lmac_id; + struct cgx *cgxd; + + pf = rvu_get_pf(pcifunc); + + if (!is_pf_cgxmapped(rvu, pf)) + return false; + + rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id); + cgxd = rvu_cgx_pdata(cgx_id, rvu); + + mutex_lock(&cgxd->lock); + if (!is_vf(pcifunc)) + goto set; + + err = cgx_get_pkind(cgxd, lmac_id, &rxpkind); + if (err) + goto err; + + switch (rxpkind) { + case NPC_RX_HIGIG_PKIND: + case NPC_RX_EDSA_PKIND: + goto err; + default: + break; + } + +set: + cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id, pkind); + mutex_unlock(&cgxd->lock); + return true; + +err: + mutex_unlock(&cgxd->lock); + return false; +} diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c index d07deeb53a53..35c8c7229f35 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_nix.c @@ -336,8 +336,8 @@ static int nix_interface_init(struct rvu *rvu, u16 pcifunc, int type, int nixlf, pfvf->tx_chan_cnt = 1; rsp->tx_link = cgx_id * hw->lmac_per_cgx + lmac_id; - cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id, pkind); - rvu_npc_set_pkind(rvu, pkind, pfvf); + if (rvu_cgx_check_permission_and_set_pkind(rvu, pcifunc, pkind)) + rvu_npc_set_pkind(rvu, pkind, pfvf); break; case NIX_INTF_TYPE_LBK: @@ -1326,11 +1326,13 @@ int rvu_mbox_handler_nix_lf_alloc(struct rvu *rvu, struct nix_lf_alloc_req *req, struct nix_lf_alloc_rsp *rsp) { - int nixlf, qints, hwctx_size, intf, err, rc = 0; + int nixlf, qints, hwctx_size, intf, err, rc = 0, pf; struct rvu_hwinfo *hw = rvu->hw; u16 pcifunc = req->hdr.pcifunc; + u8 cgx_id = 0, lmac_id = 0; struct rvu_block *block; struct rvu_pfvf *pfvf; + struct cgx *cgxd; u64 cfg, ctx_cfg; int blkaddr; @@ -1498,8 +1500,20 @@ int rvu_mbox_handler_nix_lf_alloc(struct rvu *rvu, rvu_write64(rvu, blkaddr, NIX_AF_LFX_RX_CFG(nixlf), req->rx_cfg); /* Configure pkind for TX parse config */ + + pf = rvu_get_pf(pcifunc); cfg = NPC_TX_DEF_PKIND; - rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg); + + if (is_pf_cgxmapped(rvu, pf) && is_vf(pcifunc)) { + rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id); + cgxd = rvu_cgx_pdata(cgx_id, rvu); + mutex_lock(&cgxd->lock); + if (rvu_cgx_is_pkind_config_permitted(rvu, pcifunc)) + rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg); + mutex_unlock(&cgxd->lock); + } else { + rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), cfg); + } intf = is_afvf(pcifunc) ? NIX_INTF_TYPE_LBK : NIX_INTF_TYPE_CGX; if (is_sdp_pfvf(pcifunc)) diff --git a/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c b/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c index 65ad7476e609..fb3814717f90 100644 --- a/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c +++ b/drivers/net/ethernet/marvell/octeontx2/af/rvu_npc.c @@ -16,6 +16,7 @@ #include "cgx.h" #include "npc_profile.h" #include "rvu_npc_hash.h" +#include "lmac_common.h" #define RSVD_MCAM_ENTRIES_PER_PF 3 /* Broadcast, Promisc and AllMulticast */ #define RSVD_MCAM_ENTRIES_PER_NIXLF 1 /* Ucast for LFs */ @@ -1933,7 +1934,7 @@ static int npc_mcam_rsrcs_init(struct rvu *rvu, int blkaddr) /* Alloc memory for MCAM entry to counter mapping and for tracking * counter's reference count. */ - mcam->entry2cntr_map = devm_kcalloc(rvu->dev, mcam->bmap_entries, + mcam->entry2cntr_map = devm_kcalloc(rvu->dev, mcam->total_entries, sizeof(u16), GFP_KERNEL); if (!mcam->entry2cntr_map) goto free_mem; @@ -1949,10 +1950,11 @@ static int npc_mcam_rsrcs_init(struct rvu *rvu, int blkaddr) if (!mcam->entry2target_pffunc) goto free_mem; - for (index = 0; index < mcam->bmap_entries; index++) { + for (index = 0; index < mcam->bmap_entries; index++) mcam->entry2pfvf_map[index] = NPC_MCAM_INVALID_MAP; + + for (index = 0; index < mcam->total_entries; index++) mcam->entry2cntr_map[index] = NPC_MCAM_INVALID_MAP; - } for (cntr = 0; cntr < mcam->counters.max; cntr++) mcam->cntr2pfvf_map[cntr] = NPC_MCAM_INVALID_MAP; @@ -3039,7 +3041,7 @@ int rvu_mbox_handler_npc_mcam_free_counter(struct rvu *rvu, struct npc_mcam_oper_counter_req *req, struct msg_rsp *rsp) { struct npc_mcam *mcam = &rvu->hw->mcam; - u16 index, entry = 0; + u16 index; int blkaddr, err; blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0); @@ -3057,20 +3059,16 @@ int rvu_mbox_handler_npc_mcam_free_counter(struct rvu *rvu, mcam->cntr2pfvf_map[req->cntr] = NPC_MCAM_INVALID_MAP; rvu_free_rsrc(&mcam->counters, req->cntr); - /* Disable all MCAM entry's stats which are using this counter */ - while (entry < mcam->bmap_entries) { + /* Disable all MCAM entry's stats which are using this counter. + * Scan the full MCAM index range: AF-reserved rules (e.g. CPT pass-2) + */ + for (index = 0; index < mcam->total_entries; index++) { if (!mcam->cntr_refcnt[req->cntr]) break; - - index = find_next_bit(mcam->bmap, mcam->bmap_entries, entry); - if (index >= mcam->bmap_entries) - break; - entry = index + 1; if (mcam->entry2cntr_map[index] != req->cntr) continue; - - npc_unmap_mcam_entry_and_cntr(rvu, mcam, blkaddr, - index, req->cntr); + npc_unmap_mcam_entry_and_cntr(rvu, mcam, blkaddr, index, + req->cntr); } mutex_unlock(&mcam->lock); @@ -3081,7 +3079,7 @@ int rvu_mbox_handler_npc_mcam_unmap_counter(struct rvu *rvu, struct npc_mcam_unmap_counter_req *req, struct msg_rsp *rsp) { struct npc_mcam *mcam = &rvu->hw->mcam; - u16 index, entry = 0; + u16 index; int blkaddr, rc; blkaddr = rvu_get_blkaddr(rvu, BLKTYPE_NPC, 0); @@ -3104,20 +3102,13 @@ int rvu_mbox_handler_npc_mcam_unmap_counter(struct rvu *rvu, } /* Disable all MCAM entry's stats which are using this counter */ - while (entry < mcam->bmap_entries) { + for (index = 0; index < mcam->total_entries; index++) { if (!mcam->cntr_refcnt[req->cntr]) break; - - index = find_next_bit(mcam->bmap, mcam->bmap_entries, entry); - if (index >= mcam->bmap_entries) - break; - entry = index + 1; - if (mcam->entry2cntr_map[index] != req->cntr) continue; - - npc_unmap_mcam_entry_and_cntr(rvu, mcam, blkaddr, - index, req->cntr); + npc_unmap_mcam_entry_and_cntr(rvu, mcam, blkaddr, index, + req->cntr); } exit: mutex_unlock(&mcam->lock); @@ -3339,8 +3330,9 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir, struct rvu_pfvf *pfvf = rvu_get_pfvf(rvu, pcifunc); int blkaddr, nixlf, rc, intf_mode; int pf = rvu_get_pf(pcifunc); + u8 cgx_id = 0, lmac_id = 0; u64 rxpkind, txpkind; - u8 cgx_id, lmac_id; + struct cgx *cgxd; /* use default pkind to disable edsa/higig */ rxpkind = rvu_npc_get_pkind(rvu, pf); @@ -3364,12 +3356,8 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir, /* rx pkind set req valid only for cgx mapped PFs */ if (!is_cgx_config_permitted(rvu, pcifunc)) return 0; - rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, &lmac_id); - - rc = cgx_set_pkind(rvu_cgx_pdata(cgx_id, rvu), lmac_id, - rxpkind); - if (rc) - return rc; + if (!rvu_cgx_check_permission_and_set_pkind(rvu, pcifunc, rxpkind)) + return -EINVAL; } if (dir & PKIND_TX) { @@ -3378,8 +3366,19 @@ int rvu_npc_set_parse_mode(struct rvu *rvu, u16 pcifunc, u64 mode, u8 dir, if (rc) return rc; - rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), - txpkind); + if (is_pf_cgxmapped(rvu, pf) && is_vf(pcifunc)) { + rvu_get_cgx_lmac_id(rvu->pf2cgxlmac_map[pf], &cgx_id, + &lmac_id); + cgxd = rvu_cgx_pdata(cgx_id, rvu); + mutex_lock(&cgxd->lock); + if (rvu_cgx_is_pkind_config_permitted(rvu, pcifunc)) + rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), + txpkind); + mutex_unlock(&cgxd->lock); + } else { + rvu_write64(rvu, blkaddr, NIX_AF_LFX_TX_PARSE_CFG(nixlf), + txpkind); + } } pfvf->intf_mode = intf_mode; diff --git a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c index d93b4109a144..3fe0b30d39b7 100644 --- a/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c +++ b/drivers/net/ethernet/marvell/octeontx2/nic/otx2_vf.c @@ -685,16 +685,16 @@ static int otx2vf_probe(struct pci_dev *pdev, const struct pci_device_id *id) snprintf(netdev->name, sizeof(netdev->name), "lbk%d", n); } + err = otx2_wq_init(vf); + if (err) + goto err_ptp_destroy; + err = register_netdev(netdev); if (err) { dev_err(dev, "Failed to register netdevice\n"); - goto err_ptp_destroy; + goto err_wq_destroy; } - err = otx2_wq_init(vf); - if (err) - goto err_unreg_netdev; - otx2vf_set_ethtool_ops(netdev); err = otx2vf_mcam_flow_init(vf); @@ -722,6 +722,10 @@ static int otx2vf_probe(struct pci_dev *pdev, const struct pci_device_id *id) otx2_shutdown_tc(vf); err_unreg_netdev: unregister_netdev(netdev); +err_wq_destroy: + cancel_work_sync(&vf->reset_task); + cancel_work_sync(&vf->rx_mode_work); + destroy_workqueue(vf->otx2_wq); err_ptp_destroy: otx2_ptp_destroy(vf); err_detach_rsrc: @@ -769,11 +773,13 @@ static void otx2vf_remove(struct pci_dev *pdev) } #endif - cancel_work_sync(&vf->reset_task); otx2_unregister_dl(vf); unregister_netdev(netdev); - if (vf->otx2_wq) + if (vf->otx2_wq) { + cancel_work_sync(&vf->reset_task); + cancel_work_sync(&vf->rx_mode_work); destroy_workqueue(vf->otx2_wq); + } otx2_ptp_destroy(vf); otx2_mcam_flow_del(vf); otx2_shutdown_tc(vf); diff --git a/drivers/net/ethernet/mediatek/mtk_eth_soc.c b/drivers/net/ethernet/mediatek/mtk_eth_soc.c index 261bc1a1b55a..379fc508549d 100644 --- a/drivers/net/ethernet/mediatek/mtk_eth_soc.c +++ b/drivers/net/ethernet/mediatek/mtk_eth_soc.c @@ -3237,7 +3237,7 @@ static void mtk_poll_controller(struct net_device *dev) mtk_tx_irq_disable(eth, MTK_TX_DONE_INT); mtk_rx_irq_disable(eth, eth->soc->txrx.rx_irq_done_mask); - mtk_handle_irq_rx(eth->irq[2], dev); + mtk_handle_irq_rx(eth->irq[2], eth); mtk_tx_irq_enable(eth, MTK_TX_DONE_INT); mtk_rx_irq_enable(eth, eth->soc->txrx.rx_irq_done_mask); } @@ -5031,6 +5031,22 @@ static int mtk_probe(struct platform_device *pdev) goto err_deinit_ppe; } + /* we run 2 devices on the same DMA ring so we need a dummy device + * for NAPI to work. Allocate it before registering the netdevs so a + * concurrent ndo_open (e.g. netifd bringing the first netdev up the + * instant it is registered) never observes eth->dummy_dev == NULL in + * mtk_dma_init() -> mtk_rx_alloc() -> __xdp_rxq_info_reg() (net/core/xdp.c + * "Missing net_device from driver") and fails the first open. + */ + eth->dummy_dev = alloc_netdev_dummy(0); + if (!eth->dummy_dev) { + err = -ENOMEM; + dev_err(eth->dev, "failed to allocated dummy device\n"); + goto err_unreg_netdev; + } + netif_napi_add(eth->dummy_dev, ð->tx_napi, mtk_napi_tx); + netif_napi_add(eth->dummy_dev, ð->rx_napi, mtk_napi_rx); + for (i = 0; i < MTK_MAX_DEVS; i++) { if (!eth->netdev[i]) continue; @@ -5045,18 +5061,6 @@ static int mtk_probe(struct platform_device *pdev) eth->netdev[i]->base_addr, eth->irq[0]); } - /* we run 2 devices on the same DMA ring so we need a dummy device - * for NAPI to work - */ - eth->dummy_dev = alloc_netdev_dummy(0); - if (!eth->dummy_dev) { - err = -ENOMEM; - dev_err(eth->dev, "failed to allocated dummy device\n"); - goto err_unreg_netdev; - } - netif_napi_add(eth->dummy_dev, ð->tx_napi, mtk_napi_tx); - netif_napi_add(eth->dummy_dev, ð->rx_napi, mtk_napi_rx); - platform_set_drvdata(pdev, eth); schedule_delayed_work(ð->reset.monitor_work, MTK_DMA_MONITOR_TIMEOUT); @@ -5066,6 +5070,16 @@ static int mtk_probe(struct platform_device *pdev) err_unreg_netdev: mtk_unreg_dev(eth); err_deinit_ppe: + /* A netdev opened while probe was still registering can have queued + * the frame-engine reset worker, which takes rtnl and drives the + * hardware; cancel it before anything below is torn down. + */ + cancel_work_sync(ð->pending_work); + if (eth->dummy_dev) { + netif_napi_del(ð->tx_napi); + netif_napi_del(ð->rx_napi); + free_netdev(eth->dummy_dev); + } mtk_ppe_deinit(eth); mtk_mdio_cleanup(eth); err_free_dev: diff --git a/drivers/net/ethernet/mediatek/mtk_wed_wo.c b/drivers/net/ethernet/mediatek/mtk_wed_wo.c index ae44ad5f8ce8..d3844ec7f999 100644 --- a/drivers/net/ethernet/mediatek/mtk_wed_wo.c +++ b/drivers/net/ethernet/mediatek/mtk_wed_wo.c @@ -285,7 +285,6 @@ mtk_wed_wo_queue_free(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) static void mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; int i; for (i = 0; i < q->n_desc; i++) { @@ -300,19 +299,12 @@ mtk_wed_wo_queue_tx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) entry->buf = NULL; } - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); + page_frag_cache_drain(&q->cache); } static void mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) { - struct page *page; - for (;;) { void *buf = mtk_wed_wo_dequeue(wo, q, NULL, true); @@ -322,12 +314,7 @@ mtk_wed_wo_queue_rx_clean(struct mtk_wed_wo *wo, struct mtk_wed_wo_queue *q) skb_free_frag(buf); } - if (!q->cache.va) - return; - - page = virt_to_page(q->cache.va); - __page_frag_cache_drain(page, q->cache.pagecnt_bias); - memset(&q->cache, 0, sizeof(q->cache)); + page_frag_cache_drain(&q->cache); } static void diff --git a/drivers/net/ethernet/mellanox/mlx4/srq.c b/drivers/net/ethernet/mellanox/mlx4/srq.c index dd890f5d7b72..39a01aee3bef 100644 --- a/drivers/net/ethernet/mellanox/mlx4/srq.c +++ b/drivers/net/ethernet/mellanox/mlx4/srq.c @@ -44,13 +44,14 @@ void mlx4_srq_event(struct mlx4_dev *dev, u32 srqn, int event_type) { struct mlx4_srq_table *srq_table = &mlx4_priv(dev)->srq_table; struct mlx4_srq *srq; + unsigned long flags; - rcu_read_lock(); + spin_lock_irqsave(&srq_table->lock, flags); srq = radix_tree_lookup(&srq_table->tree, srqn & (dev->caps.num_srqs - 1)); - rcu_read_unlock(); - if (srq) - refcount_inc(&srq->refcount); - else { + if (!srq || !refcount_inc_not_zero(&srq->refcount)) + srq = NULL; + spin_unlock_irqrestore(&srq_table->lock, flags); + if (!srq) { mlx4_warn(dev, "Async event for bogus SRQ %08x\n", srqn); return; } @@ -203,8 +204,9 @@ int mlx4_srq_alloc(struct mlx4_dev *dev, u32 pdn, u32 cqn, u16 xrcd, if (err) goto err_radix; - refcount_set(&srq->refcount, 1); init_completion(&srq->free); + smp_wmb(); + refcount_set(&srq->refcount, 1); return 0; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/cq.c b/drivers/net/ethernet/mellanox/mlx5/core/cq.c index 4caa1b6f40ba..d21c14bb3ecf 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/cq.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/cq.c @@ -66,8 +66,8 @@ void mlx5_cq_tasklet_cb(struct tasklet_struct *t) tasklet_schedule(&ctx->task); } -static void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq, - struct mlx5_eqe *eqe) +void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq, + struct mlx5_eqe *eqe) { unsigned long flags; struct mlx5_eq_tasklet *tasklet_ctx = cq->tasklet_ctx.priv; @@ -84,7 +84,15 @@ static void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq, } spin_unlock_irqrestore(&tasklet_ctx->lock, flags); } +EXPORT_SYMBOL(mlx5_add_cq_to_tasklet); +static void mlx5_core_cq_dummy_cb(struct mlx5_core_cq *cq, struct mlx5_eqe *eqe) +{ + mlx5_core_err(cq->eq->core.dev, + "CQ default completion callback, CQ #%u\n", cq->cqn); +} + +#define MLX5_CQ_INIT_CMD_SN cpu_to_be32(2 << 28) /* Callers must verify outbox status in case of err */ int mlx5_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq, u32 *in, int inlen, u32 *out, int outlen) @@ -110,10 +118,19 @@ int mlx5_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq, cq->arm_sn = 0; cq->eq = eq; cq->uid = MLX5_GET(create_cq_in, in, uid); + + /* Kernel CQs must set the arm_db address prior to calling + * this function, allowing for the proper value to be + * initialized. User CQs are responsible for their own + * initialization since they do not use the arm_db field. + */ + if (cq->arm_db) + *cq->arm_db = MLX5_CQ_INIT_CMD_SN; + refcount_set(&cq->refcount, 1); init_completion(&cq->free); if (!cq->comp) - cq->comp = mlx5_add_cq_to_tasklet; + cq->comp = mlx5_core_cq_dummy_cb; /* assuming CQ will be deleted before the EQ */ cq->tasklet_ctx.priv = &eq->tasklet_ctx; INIT_LIST_HEAD(&cq->tasklet_ctx.list); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en.h b/drivers/net/ethernet/mellanox/mlx5/core/en.h index 455d02b6500d..8b6c086e3293 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en.h @@ -944,8 +944,20 @@ struct mlx5e_priv { struct dentry *dfs_root; }; +static inline u16 mlx5e_stats_nch_read(const struct mlx5e_priv *priv) +{ + /* Pairs with smp_store_release in mlx5e_stats_nch_write(). */ + return smp_load_acquire(&priv->stats_nch); +} + +static inline void mlx5e_stats_nch_write(struct mlx5e_priv *priv, u16 n) +{ + /* Pairs with smp_load_acquire in mlx5e_stats_nch_read(). */ + smp_store_release(&priv->stats_nch, n); +} + struct mlx5e_dev { - struct mlx5e_priv *priv; + struct net_device *netdev; struct devlink_port dl_port; }; @@ -1202,7 +1214,7 @@ struct net_device * mlx5e_create_netdev(struct mlx5_core_dev *mdev, const struct mlx5e_profile *profile); int mlx5e_attach_netdev(struct mlx5e_priv *priv); void mlx5e_detach_netdev(struct mlx5e_priv *priv); -void mlx5e_destroy_netdev(struct mlx5e_priv *priv); +void mlx5e_destroy_netdev(struct net_device *netdev); int mlx5e_netdev_change_profile(struct mlx5e_priv *priv, const struct mlx5e_profile *new_profile, void *new_ppriv); void mlx5e_netdev_attach_nic_profile(struct mlx5e_priv *priv); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c index 1856d211339f..adfb49ff42aa 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/hv_vhca_stats.c @@ -33,9 +33,10 @@ mlx5e_hv_vhca_fill_ring_stats(struct mlx5e_priv *priv, int ch, static void mlx5e_hv_vhca_fill_stats(struct mlx5e_priv *priv, void *data, int buf_len) { + u16 nch = mlx5e_stats_nch_read(priv); int ch, i = 0; - for (ch = 0; ch < priv->stats_nch; ch++) { + for (ch = 0; ch < nch; ch++) { void *buf = data + i; if (WARN_ON_ONCE(buf + @@ -50,8 +51,9 @@ static void mlx5e_hv_vhca_fill_stats(struct mlx5e_priv *priv, void *data, static int mlx5e_hv_vhca_stats_buf_size(struct mlx5e_priv *priv) { - return (sizeof(struct mlx5e_hv_vhca_per_ring_stats) * - priv->stats_nch); + u16 nch = mlx5e_stats_nch_read(priv); + + return sizeof(struct mlx5e_hv_vhca_per_ring_stats) * nch; } static int mlx5e_hv_vhca_stats_buf_max_size(struct mlx5e_priv *priv) @@ -106,7 +108,7 @@ static void mlx5e_hv_vhca_stats_control(struct mlx5_hv_vhca_agent *agent, sagent = &priv->stats_agent; block->version = MLX5_HV_VHCA_STATS_VERSION; - block->rings = priv->stats_nch; + block->rings = mlx5e_stats_nch_read(priv); if (!block->command) { cancel_delayed_work_sync(&priv->stats_agent.work); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c index 13c7ed1bb37e..17ef61700a01 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xdp.c @@ -102,9 +102,15 @@ mlx5e_xmit_xdp_buff(struct mlx5e_xdpsq *sq, struct mlx5e_rq *rq, xdptxd->dma_addr = dma_addr; - if (unlikely(!INDIRECT_CALL_2(sq->xmit_xdp_frame, mlx5e_xmit_xdp_frame_mpwqe, - mlx5e_xmit_xdp_frame, sq, xdptxd, 0))) + if (unlikely(!INDIRECT_CALL_2(sq->xmit_xdp_frame, + mlx5e_xmit_xdp_frame_mpwqe, + mlx5e_xmit_xdp_frame, + sq, xdptxd, 0))) { + dma_unmap_single(sq->pdev, dma_addr, xdptxd->len, + DMA_TO_DEVICE); + xdp_return_frame(xdpf); return false; + } /* xmit_mode == MLX5E_XDP_XMIT_MODE_FRAME */ mlx5e_xdpi_fifo_push(&sq->db.xdpi_fifo, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c index b8dd74453655..fadc3d2309a8 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en/xsk/rx.c @@ -3,6 +3,7 @@ #include "rx.h" #include "en/xdp.h" +#include #include #include @@ -153,6 +154,7 @@ int mlx5e_xsk_alloc_rx_mpwqe(struct mlx5e_rq *rq, u16 ix) xsk_buff_free(xsk_buffs[batch]); err: + bitmap_fill(wi->skip_release_bitmap, rq->mpwqe.pages_per_wqe); rq->stats->buff_alloc_err++; return -ENOMEM; } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c index fdf664e9c46e..13c143610d6c 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_accel/ipsec.c @@ -265,7 +265,6 @@ static void mlx5e_ipsec_init_limits(struct mlx5e_ipsec_sa_entry *sa_entry, static void mlx5e_ipsec_init_macs(struct mlx5e_ipsec_sa_entry *sa_entry, struct mlx5_accel_esp_xfrm_attrs *attrs) { - struct mlx5_core_dev *mdev = mlx5e_ipsec_sa2dev(sa_entry); struct xfrm_state *x = sa_entry->x; struct net_device *netdev; struct neighbour *n; @@ -279,7 +278,7 @@ static void mlx5e_ipsec_init_macs(struct mlx5e_ipsec_sa_entry *sa_entry, netdev = x->xso.real_dev; - mlx5_query_mac_address(mdev, addr); + ether_addr_copy(addr, netdev->dev_addr); switch (attrs->dir) { case XFRM_DEV_OFFLOAD_IN: src = attrs->dmac; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c index 895bd5b1216e..a35594fe3bd1 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_main.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_main.c @@ -1936,7 +1936,6 @@ static int mlx5e_alloc_cq_common(struct mlx5e_priv *priv, mcq->set_ci_db = cq->wq_ctrl.db.db; mcq->arm_db = cq->wq_ctrl.db.db + 1; *mcq->set_ci_db = 0; - *mcq->arm_db = 0; mcq->vector = param->eq_ix; mcq->comp = mlx5e_completion_event; mcq->event = mlx5e_cq_error_event; @@ -2422,7 +2421,7 @@ static int mlx5e_channel_stats_alloc(struct mlx5e_priv *priv, int ix, int cpu) GFP_KERNEL, cpu_to_node(cpu)); if (!priv->channel_stats[ix]) return -ENOMEM; - priv->stats_nch++; + mlx5e_stats_nch_write(priv, priv->stats_nch + 1); return 0; } @@ -3686,9 +3685,10 @@ static int mlx5e_setup_tc(struct net_device *dev, enum tc_setup_type type, void mlx5e_fold_sw_stats64(struct mlx5e_priv *priv, struct rtnl_link_stats64 *s) { + u16 nch = mlx5e_stats_nch_read(priv); int i; - for (i = 0; i < priv->stats_nch; i++) { + for (i = 0; i < nch; i++) { struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i]; struct mlx5e_rq_stats *xskrq_stats = &channel_stats->xskrq; struct mlx5e_rq_stats *rq_stats = &channel_stats->rq; @@ -5984,11 +5984,12 @@ void mlx5e_netdev_attach_nic_profile(struct mlx5e_priv *priv) mlx5e_netdev_change_profile(priv, &mlx5e_nic_profile, NULL); } -void mlx5e_destroy_netdev(struct mlx5e_priv *priv) +void mlx5e_destroy_netdev(struct net_device *netdev) { - struct net_device *netdev = priv->netdev; + struct mlx5e_priv *priv = netdev_priv(netdev); - mlx5e_priv_cleanup(priv); + if (priv->profile) + mlx5e_priv_cleanup(priv); free_netdev(netdev); } @@ -5996,8 +5997,8 @@ static int mlx5e_resume(struct auxiliary_device *adev) { struct mlx5_adev *edev = container_of(adev, struct mlx5_adev, adev); struct mlx5e_dev *mlx5e_dev = auxiliary_get_drvdata(adev); - struct mlx5e_priv *priv = mlx5e_dev->priv; - struct net_device *netdev = priv->netdev; + struct mlx5e_priv *priv = netdev_priv(mlx5e_dev->netdev); + struct net_device *netdev = mlx5e_dev->netdev; struct mlx5_core_dev *mdev = edev->mdev; int err; @@ -6017,14 +6018,16 @@ static int mlx5e_resume(struct auxiliary_device *adev) return 0; } -static int mlx5e_suspend(struct auxiliary_device *adev, pm_message_t state) +static int mlx5e_suspend_common(struct auxiliary_device *adev, + bool pre_netdev_reg) { + struct mlx5_adev *edev = container_of(adev, struct mlx5_adev, adev); struct mlx5e_dev *mlx5e_dev = auxiliary_get_drvdata(adev); - struct mlx5e_priv *priv = mlx5e_dev->priv; - struct net_device *netdev = priv->netdev; - struct mlx5_core_dev *mdev = priv->mdev; + struct mlx5e_priv *priv = netdev_priv(mlx5e_dev->netdev); + struct net_device *netdev = mlx5e_dev->netdev; + struct mlx5_core_dev *mdev = edev->mdev; - if (!netif_device_present(netdev)) { + if (!pre_netdev_reg && !netif_device_present(netdev)) { if (test_bit(MLX5E_STATE_DESTROYING, &priv->state)) mlx5e_destroy_mdev_resources(mdev); return -ENODEV; @@ -6035,6 +6038,11 @@ static int mlx5e_suspend(struct auxiliary_device *adev, pm_message_t state) return 0; } +static int mlx5e_suspend(struct auxiliary_device *adev, pm_message_t state) +{ + return mlx5e_suspend_common(adev, false); +} + static int mlx5e_probe(struct auxiliary_device *adev, const struct auxiliary_device_id *id) { @@ -6043,7 +6051,6 @@ static int mlx5e_probe(struct auxiliary_device *adev, struct mlx5_core_dev *mdev = edev->mdev; struct mlx5e_dev *mlx5e_dev; struct net_device *netdev; - pm_message_t state = {}; struct mlx5e_priv *priv; int err; @@ -6065,11 +6072,11 @@ static int mlx5e_probe(struct auxiliary_device *adev, goto err_devlink_port_unregister; } SET_NETDEV_DEVLINK_PORT(netdev, &mlx5e_dev->dl_port); + mlx5e_dev->netdev = netdev; mlx5e_build_nic_netdev(netdev); priv = netdev_priv(netdev); - mlx5e_dev->priv = priv; priv->profile = profile; priv->ppriv = NULL; @@ -6106,11 +6113,11 @@ static int mlx5e_probe(struct auxiliary_device *adev, return 0; err_resume: - mlx5e_suspend(adev, state); + mlx5e_suspend_common(adev, true); err_profile_cleanup: profile->cleanup(priv); err_destroy_netdev: - mlx5e_destroy_netdev(priv); + mlx5e_destroy_netdev(netdev); err_devlink_port_unregister: mlx5e_devlink_port_unregister(mlx5e_dev); err_devlink_unregister: @@ -6120,18 +6127,23 @@ static int mlx5e_probe(struct auxiliary_device *adev, static void mlx5e_remove(struct auxiliary_device *adev) { + struct mlx5_adev *edev = container_of(adev, struct mlx5_adev, adev); struct mlx5e_dev *mlx5e_dev = auxiliary_get_drvdata(adev); - struct mlx5e_priv *priv = mlx5e_dev->priv; + struct net_device *netdev = mlx5e_dev->netdev; + struct mlx5e_priv *priv = netdev_priv(netdev); + struct mlx5_core_dev *mdev = edev->mdev; pm_message_t state = {}; - mlx5_core_uplink_netdev_set(priv->mdev, NULL); - mlx5e_dcbnl_delete_app(priv); - unregister_netdev(priv->netdev); + mlx5_core_uplink_netdev_set(mdev, NULL); + + if (priv->profile) + mlx5e_dcbnl_delete_app(priv); + unregister_netdev(netdev); mlx5e_suspend(adev, state); /* Avoid cleanup if profile rollback failed. */ if (priv->profile) priv->profile->cleanup(priv); - mlx5e_destroy_netdev(priv); + mlx5e_destroy_netdev(netdev); mlx5e_devlink_port_unregister(mlx5e_dev); mlx5e_destroy_devlink(mlx5e_dev); } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c index 656a7b65f4c7..4e299fa31342 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.c @@ -54,7 +54,6 @@ #include "lib/vxlan.h" #define CREATE_TRACE_POINTS #include "diag/en_rep_tracepoint.h" -#include "diag/reporter_vnic.h" #include "en_accel/ipsec.h" #include "en/tc/int_port.h" #include "en/ptp.h" @@ -1364,50 +1363,6 @@ static unsigned int mlx5e_ul_rep_stats_grps_num(struct mlx5e_priv *priv) return ARRAY_SIZE(mlx5e_ul_rep_stats_grps); } -static int -mlx5e_rep_vnic_reporter_diagnose(struct devlink_health_reporter *reporter, - struct devlink_fmsg *fmsg, - struct netlink_ext_ack *extack) -{ - struct mlx5e_rep_priv *rpriv = devlink_health_reporter_priv(reporter); - struct mlx5_eswitch_rep *rep = rpriv->rep; - - return mlx5_reporter_vnic_diagnose_counters(rep->esw->dev, fmsg, - rep->vport, true); -} - -static const struct devlink_health_reporter_ops mlx5_rep_vnic_reporter_ops = { - .name = "vnic", - .diagnose = mlx5e_rep_vnic_reporter_diagnose, -}; - -static void mlx5e_rep_vnic_reporter_create(struct mlx5e_priv *priv, - struct devlink_port *dl_port) -{ - struct mlx5e_rep_priv *rpriv = priv->ppriv; - struct devlink_health_reporter *reporter; - - reporter = devl_port_health_reporter_create(dl_port, - &mlx5_rep_vnic_reporter_ops, - 0, rpriv); - if (IS_ERR(reporter)) { - mlx5_core_err(priv->mdev, - "Failed to create representor vnic reporter, err = %ld\n", - PTR_ERR(reporter)); - return; - } - - rpriv->rep_vnic_reporter = reporter; -} - -static void mlx5e_rep_vnic_reporter_destroy(struct mlx5e_priv *priv) -{ - struct mlx5e_rep_priv *rpriv = priv->ppriv; - - if (!IS_ERR_OR_NULL(rpriv->rep_vnic_reporter)) - devl_health_reporter_destroy(rpriv->rep_vnic_reporter); -} - static const struct mlx5e_profile mlx5e_rep_profile = { .init = mlx5e_init_rep, .cleanup = mlx5e_cleanup_rep, @@ -1517,10 +1472,8 @@ mlx5e_vport_vf_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep) dl_port = mlx5_esw_offloads_devlink_port(dev->priv.eswitch, rpriv->rep->vport); - if (!IS_ERR(dl_port)) { + if (!IS_ERR(dl_port)) SET_NETDEV_DEVLINK_PORT(netdev, dl_port); - mlx5e_rep_vnic_reporter_create(priv, dl_port); - } err = register_netdev(netdev); if (err) { @@ -1533,13 +1486,12 @@ mlx5e_vport_vf_rep_load(struct mlx5_core_dev *dev, struct mlx5_eswitch_rep *rep) return 0; err_detach_netdev: - mlx5e_rep_vnic_reporter_destroy(priv); mlx5e_detach_netdev(netdev_priv(netdev)); err_cleanup_profile: priv->profile->cleanup(priv); err_destroy_netdev: - mlx5e_destroy_netdev(netdev_priv(netdev)); + mlx5e_destroy_netdev(netdev); return err; } @@ -1591,10 +1543,9 @@ mlx5e_vport_rep_unload(struct mlx5_eswitch_rep *rep) } unregister_netdev(netdev); - mlx5e_rep_vnic_reporter_destroy(priv); mlx5e_detach_netdev(priv); priv->profile->cleanup(priv); - mlx5e_destroy_netdev(priv); + mlx5e_destroy_netdev(netdev); free_ppriv: kvfree(ppriv); /* mlx5e_rep_priv */ } diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h index 70640fa1ad7b..bcd7b4e814d0 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rep.h @@ -118,7 +118,6 @@ struct mlx5e_rep_priv { struct rtnl_link_stats64 prev_vf_vport_stats; struct mlx5_flow_handle *send_to_vport_meta_rule; struct rhashtable tc_ht; - struct devlink_health_reporter *rep_vnic_reporter; }; static inline diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c index 1d586451900b..11a75a9917f9 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_rx.c @@ -382,8 +382,11 @@ static inline void mlx5e_free_rx_wqe(struct mlx5e_rq *rq, static void mlx5e_xsk_free_rx_wqe(struct mlx5e_wqe_frag_info *wi) { - if (!(wi->flags & BIT(MLX5E_WQE_FRAG_SKIP_RELEASE))) - xsk_buff_free(*wi->xskp); + if (wi->flags & BIT(MLX5E_WQE_FRAG_SKIP_RELEASE)) + return; + + xsk_buff_free(*wi->xskp); + wi->flags |= BIT(MLX5E_WQE_FRAG_SKIP_RELEASE); } static void mlx5e_dealloc_rx_wqe(struct mlx5e_rq *rq, u16 ix) @@ -1070,10 +1073,12 @@ int mlx5e_poll_ico_cq(struct mlx5e_cq *cq) } while (!last_wqe); } while ((++i < MLX5E_TX_CQ_POLL_BUDGET) && (cqe = mlx5_cqwq_get_cqe(&cq->wq))); - sq->cc = sqcc; - mlx5_cqwq_update_db_record(&cq->wq); + /* ensure cq space is freed before enabling more cqes */ + dma_wmb(); + + sq->cc = sqcc; return i; } @@ -1733,6 +1738,7 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi struct skb_shared_info *sinfo; u32 frag_consumed_bytes; struct bpf_prog *prog; + u8 nr_frags_free = 0; struct sk_buff *skb; dma_addr_t addr; u32 truesize; @@ -1775,15 +1781,13 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi prog = rcu_dereference(rq->xdp_prog); if (prog) { - u8 nr_frags_free, old_nr_frags = sinfo->nr_frags; + u8 old_nr_frags = sinfo->nr_frags; if (mlx5e_xdp_handle(rq, prog, mxbuf)) { if (__test_and_clear_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags)) { struct mlx5e_wqe_frag_info *pwi; - wi -= old_nr_frags - sinfo->nr_frags; - for (pwi = head_wi; pwi < wi; pwi++) pwi->frag_page->frags++; } @@ -1791,10 +1795,8 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi } nr_frags_free = old_nr_frags - sinfo->nr_frags; - if (unlikely(nr_frags_free)) { - wi -= nr_frags_free; + if (unlikely(nr_frags_free)) truesize -= nr_frags_free * frag_info->frag_stride; - } } skb = mlx5e_build_linear_skb( @@ -1810,7 +1812,8 @@ mlx5e_skb_from_cqe_nonlinear(struct mlx5e_rq *rq, struct mlx5e_wqe_frag_info *wi if (xdp_buff_has_frags(&mxbuf->xdp)) { /* sinfo->nr_frags is reset by build_skb, calculate again. */ - xdp_update_skb_shared_info(skb, wi - head_wi - 1, + xdp_update_skb_shared_info(skb, + wi - head_wi - nr_frags_free - 1, sinfo->xdp_frags_size, truesize, xdp_buff_is_frag_pfmemalloc( &mxbuf->xdp)); @@ -2094,14 +2097,13 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w if (prog) { u8 nr_frags_free, old_nr_frags = sinfo->nr_frags; + u8 new_nr_frags; u32 len; if (mlx5e_xdp_handle(rq, prog, mxbuf)) { if (__test_and_clear_bit(MLX5E_RQ_FLAG_XDP_XMIT, rq->flags)) { struct mlx5e_frag_page *pfp; - frag_page -= old_nr_frags - sinfo->nr_frags; - for (pfp = head_page; pfp < frag_page; pfp++) pfp->frags++; @@ -2111,13 +2113,12 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w return NULL; /* page/packet was consumed by XDP */ } - nr_frags_free = old_nr_frags - sinfo->nr_frags; - if (unlikely(nr_frags_free)) { - frag_page -= nr_frags_free; + new_nr_frags = sinfo->nr_frags; + nr_frags_free = old_nr_frags - new_nr_frags; + if (unlikely(nr_frags_free)) truesize -= (nr_frags_free - 1) * PAGE_SIZE + ALIGN(pg_consumed_bytes, BIT(rq->mpwqe.log_stride_sz)); - } len = mxbuf->xdp.data_end - mxbuf->xdp.data; @@ -2138,7 +2139,7 @@ mlx5e_skb_from_cqe_mpwrq_nonlinear(struct mlx5e_rq *rq, struct mlx5e_mpw_info *w struct mlx5e_frag_page *pagep; /* sinfo->nr_frags is reset by build_skb, calculate again. */ - xdp_update_skb_shared_info(skb, frag_page - head_page, + xdp_update_skb_shared_info(skb, new_nr_frags, sinfo->xdp_frags_size, truesize, xdp_buff_is_frag_pfmemalloc( &mxbuf->xdp)); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c index 1c69244e00d7..9f3ccf68b267 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/en_stats.c @@ -517,6 +517,7 @@ static void mlx5e_stats_update_stats_rq_page_pool(struct mlx5e_channel *c) static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw) { struct mlx5e_sw_stats *s = &priv->stats.sw; + u16 nch = mlx5e_stats_nch_read(priv); int i; memset(s, 0, sizeof(*s)); @@ -524,7 +525,7 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(sw) for (i = 0; i < priv->channels.num; i++) /* for active channels only */ mlx5e_stats_update_stats_rq_page_pool(priv->channels.c[i]); - for (i = 0; i < priv->stats_nch; i++) { + for (i = 0; i < nch; i++) { struct mlx5e_channel_stats *channel_stats = priv->channel_stats[i]; @@ -2358,7 +2359,7 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(ptp) { return; } static MLX5E_DECLARE_STATS_GRP_OP_NUM_STATS(channels) { - int max_nch = priv->stats_nch; + int max_nch = mlx5e_stats_nch_read(priv); return (NUM_RQ_STATS * max_nch) + (NUM_CH_STATS * max_nch) + @@ -2371,8 +2372,8 @@ static MLX5E_DECLARE_STATS_GRP_OP_NUM_STATS(channels) static MLX5E_DECLARE_STATS_GRP_OP_FILL_STRS(channels) { + int max_nch = mlx5e_stats_nch_read(priv); bool is_xsk = priv->xsk.ever_used; - int max_nch = priv->stats_nch; int i, j, tc; for (i = 0; i < max_nch; i++) @@ -2413,8 +2414,8 @@ static MLX5E_DECLARE_STATS_GRP_OP_FILL_STRS(channels) static MLX5E_DECLARE_STATS_GRP_OP_FILL_STATS(channels) { + int max_nch = mlx5e_stats_nch_read(priv); bool is_xsk = priv->xsk.ever_used; - int max_nch = priv->stats_nch; int i, j, tc; for (i = 0; i < max_nch; i++) diff --git a/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c b/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c index 91319b5acd3d..893da3d8c9e6 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/esw/devlink_port.c @@ -3,6 +3,24 @@ #include #include "eswitch.h" +#include "diag/reporter_vnic.h" + +static int +mlx5_esw_rep_vnic_reporter_diagnose(struct devlink_health_reporter *reporter, + struct devlink_fmsg *fmsg, + struct netlink_ext_ack *extack) +{ + struct mlx5_vport *vport = devlink_health_reporter_priv(reporter); + + return mlx5_reporter_vnic_diagnose_counters(vport->dev, fmsg, + vport->vport, true); +} + +static const +struct devlink_health_reporter_ops mlx5_esw_rep_vnic_reporter_ops = { + .name = "vnic", + .diagnose = mlx5_esw_rep_vnic_reporter_diagnose, +}; static void mlx5_esw_get_port_parent_id(struct mlx5_core_dev *dev, struct netdev_phys_item_id *ppid) @@ -149,6 +167,7 @@ static const struct devlink_port_ops mlx5_esw_dl_sf_port_ops = { int mlx5_esw_offloads_devlink_port_register(struct mlx5_eswitch *esw, struct mlx5_vport *vport) { + struct devlink_health_reporter *reporter; struct mlx5_core_dev *dev = esw->dev; const struct devlink_port_ops *ops; struct mlx5_devlink_port *dl_port; @@ -178,6 +197,16 @@ int mlx5_esw_offloads_devlink_port_register(struct mlx5_eswitch *esw, struct mlx if (err) goto rate_err; + reporter = devl_port_health_reporter_create( + &dl_port->dl_port, &mlx5_esw_rep_vnic_reporter_ops, 0, + vport); + if (IS_ERR(reporter)) + mlx5_core_err(dev, + "Failed to create vnic health reporter for vport %d: %pe\n", + vport_num, reporter); + else + dl_port->vnic_reporter = reporter; + return 0; rate_err: @@ -193,6 +222,11 @@ void mlx5_esw_offloads_devlink_port_unregister(struct mlx5_eswitch *esw, struct return; dl_port = vport->dl_port; + if (dl_port->vnic_reporter) { + devl_health_reporter_destroy(dl_port->vnic_reporter); + dl_port->vnic_reporter = NULL; + } + mlx5_esw_qos_vport_update_group(esw, vport, NULL, NULL); devl_rate_leaf_destroy(&dl_port->dl_port); diff --git a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h index 48bebc3b8b12..f5873ac10c02 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h +++ b/drivers/net/ethernet/mellanox/mlx5/core/eswitch.h @@ -179,6 +179,7 @@ struct mlx5_vport; struct mlx5_devlink_port { struct devlink_port dl_port; struct mlx5_vport *vport; + struct devlink_health_reporter *vnic_reporter; }; static inline void mlx5_devlink_port_init(struct mlx5_devlink_port *dl_port, diff --git a/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c b/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c index c4de6bf8d1b6..ff679c7a29ea 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/fpga/conn.c @@ -437,6 +437,14 @@ static int mlx5_fpga_conn_create_cq(struct mlx5_fpga_conn *conn, int cq_size) cqe->op_own = MLX5_CQE_INVALID << 4 | MLX5_CQE_OWNER_MASK; } + conn->cq.mcq.cqe_sz = 64; + conn->cq.mcq.set_ci_db = conn->cq.wq_ctrl.db.db; + conn->cq.mcq.arm_db = conn->cq.wq_ctrl.db.db + 1; + *conn->cq.mcq.set_ci_db = 0; + conn->cq.mcq.vector = 0; + conn->cq.mcq.comp = mlx5_fpga_conn_cq_complete; + conn->cq.mcq.uar = fdev->conn_res.uar; + inlen = MLX5_ST_SZ_BYTES(create_cq_in) + sizeof(u64) * conn->cq.wq_ctrl.buf.npages; in = kvzalloc(inlen, GFP_KERNEL); @@ -468,16 +476,7 @@ static int mlx5_fpga_conn_create_cq(struct mlx5_fpga_conn *conn, int cq_size) if (err) goto err_cqwq; - conn->cq.mcq.cqe_sz = 64; - conn->cq.mcq.set_ci_db = conn->cq.wq_ctrl.db.db; - conn->cq.mcq.arm_db = conn->cq.wq_ctrl.db.db + 1; - *conn->cq.mcq.set_ci_db = 0; - *conn->cq.mcq.arm_db = 0; - conn->cq.mcq.vector = 0; - conn->cq.mcq.comp = mlx5_fpga_conn_cq_complete; - conn->cq.mcq.uar = fdev->conn_res.uar; tasklet_setup(&conn->cq.tasklet, mlx5_fpga_conn_cq_tasklet); - mlx5_fpga_dbg(fdev, "Created CQ #0x%x\n", conn->cq.mcq.cqn); goto out; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c index baa7ef812313..e46755f0ef39 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/ipoib/ipoib.c @@ -114,10 +114,11 @@ void mlx5i_cleanup(struct mlx5e_priv *priv) static void mlx5i_grp_sw_update_stats(struct mlx5e_priv *priv) { + u16 nch = mlx5e_stats_nch_read(priv); struct rtnl_link_stats64 s = {}; int i, j; - for (i = 0; i < priv->stats_nch; i++) { + for (i = 0; i < nch; i++) { struct mlx5e_channel_stats *channel_stats; struct mlx5e_rq_stats *rq_stats; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/port.c b/drivers/net/ethernet/mellanox/mlx5/core/port.c index a5622b44385e..ec739b2f0ada 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/port.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/port.c @@ -316,7 +316,7 @@ static int mlx5_query_module_id(struct mlx5_core_dev *dev, int module_num, status); return -EIO; } - ptr = MLX5_ADDR_OF(mcia_reg, out, dword_0); + ptr = MLX5_ADDR_OF(mcia_reg, out, dwords); *module_id = ptr[0]; @@ -401,7 +401,7 @@ static int mlx5_query_mcia(struct mlx5_core_dev *dev, return -EIO; } - ptr = MLX5_ADDR_OF(mcia_reg, out, dword_0); + ptr = MLX5_ADDR_OF(mcia_reg, out, dwords); memcpy(data, ptr, size); return size; diff --git a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c index f57c84e5128b..30401ed45b17 100644 --- a/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c +++ b/drivers/net/ethernet/mellanox/mlx5/core/steering/dr_send.c @@ -1049,12 +1049,6 @@ static int dr_prepare_qp_to_rts(struct mlx5dr_domain *dmn) return 0; } -static void dr_cq_complete(struct mlx5_core_cq *mcq, - struct mlx5_eqe *eqe) -{ - pr_err("CQ completion CQ: #%u\n", mcq->cqn); -} - static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev, struct mlx5_uars_page *uar, size_t ncqe) @@ -1089,6 +1083,14 @@ static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev, cqe->op_own = MLX5_CQE_INVALID << 4 | MLX5_CQE_OWNER_MASK; } + cq->mcq.cqe_sz = 64; + cq->mcq.set_ci_db = cq->wq_ctrl.db.db; + cq->mcq.arm_db = cq->wq_ctrl.db.db + 1; + *cq->mcq.set_ci_db = 0; + cq->mcq.vector = 0; + cq->mcq.uar = uar; + cq->mdev = mdev; + inlen = MLX5_ST_SZ_BYTES(create_cq_in) + sizeof(u64) * cq->wq_ctrl.buf.npages; in = kvzalloc(inlen, GFP_KERNEL); @@ -1112,28 +1114,12 @@ static struct mlx5dr_cq *dr_create_cq(struct mlx5_core_dev *mdev, pas = (__be64 *)MLX5_ADDR_OF(create_cq_in, in, pas); mlx5_fill_page_frag_array(&cq->wq_ctrl.buf, pas); - cq->mcq.comp = dr_cq_complete; - err = mlx5_core_create_cq(mdev, &cq->mcq, in, inlen, out, sizeof(out)); kvfree(in); if (err) goto err_cqwq; - cq->mcq.cqe_sz = 64; - cq->mcq.set_ci_db = cq->wq_ctrl.db.db; - cq->mcq.arm_db = cq->wq_ctrl.db.db + 1; - *cq->mcq.set_ci_db = 0; - - /* set no-zero value, in order to avoid the HW to run db-recovery on - * CQ that used in polling mode. - */ - *cq->mcq.arm_db = cpu_to_be32(2 << 28); - - cq->mcq.vector = 0; - cq->mcq.uar = uar; - cq->mdev = mdev; - return cq; err_cqwq: diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h index 02ca2871b6f9..1422ffaff856 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum.h +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum.h @@ -998,6 +998,8 @@ void mlxsw_sp_acl_ruleset_prio_get(struct mlxsw_sp_acl_ruleset *ruleset, struct mlxsw_sp_acl_rule_info * mlxsw_sp_acl_rulei_create(struct mlxsw_sp_acl *acl, struct mlxsw_afa_block *afa_block); +void mlxsw_sp_acl_rulei_free(struct mlxsw_sp *mlxsw_sp, + const struct mlxsw_sp_acl_rule_info *rulei); void mlxsw_sp_acl_rulei_destroy(struct mlxsw_sp *mlxsw_sp, struct mlxsw_sp_acl_rule_info *rulei); int mlxsw_sp_acl_rulei_commit(struct mlxsw_sp_acl_rule_info *rulei); diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c index b01b000bc71c..eac09cd6ecd4 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_acl.c @@ -340,8 +340,8 @@ mlxsw_sp_acl_rulei_create(struct mlxsw_sp_acl *acl, return ERR_PTR(err); } -void mlxsw_sp_acl_rulei_destroy(struct mlxsw_sp *mlxsw_sp, - struct mlxsw_sp_acl_rule_info *rulei) +void mlxsw_sp_acl_rulei_free(struct mlxsw_sp *mlxsw_sp, + const struct mlxsw_sp_acl_rule_info *rulei) { if (rulei->action_created) mlxsw_afa_block_destroy(rulei->act_block); @@ -351,6 +351,12 @@ void mlxsw_sp_acl_rulei_destroy(struct mlxsw_sp *mlxsw_sp, if (rulei->dst_port_range_reg_valid) mlxsw_sp_port_range_reg_put(mlxsw_sp, rulei->dst_port_range_reg_index); +} + +void mlxsw_sp_acl_rulei_destroy(struct mlxsw_sp *mlxsw_sp, + struct mlxsw_sp_acl_rule_info *rulei) +{ + mlxsw_sp_acl_rulei_free(mlxsw_sp, rulei); kfree(rulei); } diff --git a/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c b/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c index fba545938fd7..7892c4d222ce 100644 --- a/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c +++ b/drivers/net/ethernet/mellanox/mlxsw/spectrum_flower.c @@ -848,14 +848,19 @@ int mlxsw_sp_flower_tmplt_create(struct mlxsw_sp *mlxsw_sp, memset(&rulei, 0, sizeof(rulei)); err = mlxsw_sp_flower_parse(mlxsw_sp, block, &rulei, f); if (err) - return err; + goto out; + ruleset = mlxsw_sp_acl_ruleset_get(mlxsw_sp, block, f->common.chain_index, MLXSW_SP_ACL_PROFILE_FLOWER, &rulei.values.elusage); + err = PTR_ERR_OR_ZERO(ruleset); + +out: + mlxsw_sp_acl_rulei_free(mlxsw_sp, &rulei); /* keep the reference to the ruleset */ - return PTR_ERR_OR_ZERO(ruleset); + return err; } void mlxsw_sp_flower_tmplt_destroy(struct mlxsw_sp *mlxsw_sp, diff --git a/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c b/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c index e717d68bf45a..a99feb266c36 100644 --- a/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c +++ b/drivers/net/ethernet/microchip/lan966x/lan966x_fdma.c @@ -120,8 +120,10 @@ static int lan966x_fdma_rx_alloc(struct lan966x_rx *rx) size = ALIGN(size, PAGE_SIZE); rx->dcbs = dma_alloc_coherent(lan966x->dev, size, &rx->dma, GFP_KERNEL); - if (!rx->dcbs) + if (!rx->dcbs) { + page_pool_destroy(rx->page_pool); return -ENOMEM; + } rx->last_entry = rx->dcbs; rx->db_index = 0; @@ -1048,7 +1050,9 @@ int lan966x_fdma_init(struct lan966x *lan966x) err = lan966x_fdma_tx_alloc(&lan966x->tx); if (err) { + lan966x_fdma_rx_free_pages(&lan966x->rx); lan966x_fdma_rx_free(&lan966x->rx); + page_pool_destroy(lan966x->rx.page_pool); return err; } diff --git a/drivers/net/ethernet/microchip/sparx5/sparx5_main.c b/drivers/net/ethernet/microchip/sparx5/sparx5_main.c index 98bee953234b..524601201b6f 100644 --- a/drivers/net/ethernet/microchip/sparx5/sparx5_main.c +++ b/drivers/net/ethernet/microchip/sparx5/sparx5_main.c @@ -843,6 +843,7 @@ static int mchp_sparx5_probe(struct platform_device *pdev) dev_info(sparx5->dev, "MAC addr was not set, use random MAC\n"); eth_random_addr(sparx5->base_mac); sparx5->base_mac[5] = 0; + sparx5->base_mac_assign_type = NET_ADDR_RANDOM; } sparx5->fdma_irq = platform_get_irq_byname(sparx5->pdev, "fdma"); diff --git a/drivers/net/ethernet/microchip/sparx5/sparx5_main.h b/drivers/net/ethernet/microchip/sparx5/sparx5_main.h index 316fed5f2735..40a3e9c62cfd 100644 --- a/drivers/net/ethernet/microchip/sparx5/sparx5_main.h +++ b/drivers/net/ethernet/microchip/sparx5/sparx5_main.h @@ -257,6 +257,7 @@ struct sparx5 { struct notifier_block switchdev_blocking_nb; /* Switch state */ u8 base_mac[ETH_ALEN]; + u8 base_mac_assign_type; /* Associated bridge device (when bridged) */ struct net_device *hw_bridge_dev; /* Bridged interfaces */ diff --git a/drivers/net/ethernet/microchip/sparx5/sparx5_netdev.c b/drivers/net/ethernet/microchip/sparx5/sparx5_netdev.c index 705a004b324f..62025aba558c 100644 --- a/drivers/net/ethernet/microchip/sparx5/sparx5_netdev.c +++ b/drivers/net/ethernet/microchip/sparx5/sparx5_netdev.c @@ -281,6 +281,7 @@ struct net_device *sparx5_create_netdev(struct sparx5 *sparx5, u32 portno) ndev->ethtool_ops = &sparx5_ethtool_ops; eth_hw_addr_gen(ndev, sparx5->base_mac, portno + 1); + ndev->addr_assign_type = sparx5->base_mac_assign_type; return ndev; } diff --git a/drivers/net/ethernet/microchip/vcap/vcap_api.c b/drivers/net/ethernet/microchip/vcap/vcap_api.c index 07437c94d474..8891e814840d 100644 --- a/drivers/net/ethernet/microchip/vcap/vcap_api.c +++ b/drivers/net/ethernet/microchip/vcap/vcap_api.c @@ -3512,8 +3512,8 @@ int vcap_filter_rule_keys(struct vcap_rule *rule, if (key == keylist[idx]) { list_del(&ckf->ctrl.list); kfree(ckf); - idx++; err = 0; + break; } } } diff --git a/drivers/net/ethernet/microsoft/mana/mana_en.c b/drivers/net/ethernet/microsoft/mana/mana_en.c index 5c96e9a5f14e..d4a37341534d 100644 --- a/drivers/net/ethernet/microsoft/mana/mana_en.c +++ b/drivers/net/ethernet/microsoft/mana/mana_en.c @@ -2050,9 +2050,12 @@ static void mana_destroy_rxq(struct mana_port_context *apc, netif_napi_del(napi); } - xdp_rxq_info_unreg(&rxq->xdp_rxq); - mana_destroy_wq_obj(apc, GDMA_RQ, rxq->rxobj); + if (xdp_rxq_info_is_reg(&rxq->xdp_rxq)) + xdp_rxq_info_unreg(&rxq->xdp_rxq); + + if (rxq->rxobj != INVALID_MANA_HANDLE) + mana_destroy_wq_obj(apc, GDMA_RQ, rxq->rxobj); mana_deinit_cq(apc, &rxq->rx_cq); @@ -2318,9 +2321,6 @@ static struct mana_rxq *mana_create_rxq(struct mana_port_context *apc, mana_destroy_rxq(apc, rxq, false); - if (cq) - mana_deinit_cq(apc, cq); - return NULL; } @@ -2921,11 +2921,16 @@ void mana_remove(struct gdma_dev *gd, bool suspending) { struct gdma_context *gc = gd->gdma_context; struct mana_context *ac = gd->driver_data; - struct device *dev = gc->dev; + struct device *dev; struct net_device *ndev; int err; int i; + if (!gc || !ac) + return; + + dev = gc->dev; + /* adev currently doesn't support suspending, always remove it */ if (gd->adev) remove_adev(gd); diff --git a/drivers/net/ethernet/mscc/ocelot_stats.c b/drivers/net/ethernet/mscc/ocelot_stats.c index c018783757fb..8dff857b0536 100644 --- a/drivers/net/ethernet/mscc/ocelot_stats.c +++ b/drivers/net/ethernet/mscc/ocelot_stats.c @@ -984,6 +984,6 @@ int ocelot_stats_init(struct ocelot *ocelot) void ocelot_stats_deinit(struct ocelot *ocelot) { - cancel_delayed_work(&ocelot->stats_work); + cancel_delayed_work_sync(&ocelot->stats_work); destroy_workqueue(ocelot->stats_queue); } diff --git a/drivers/net/ethernet/qlogic/qede/qede_fp.c b/drivers/net/ethernet/qlogic/qede/qede_fp.c index 5d83d10de348..a02624f6d6d5 100644 --- a/drivers/net/ethernet/qlogic/qede/qede_fp.c +++ b/drivers/net/ethernet/qlogic/qede/qede_fp.c @@ -216,7 +216,7 @@ static void qede_set_params_for_ipv6_ext(struct sk_buff *skb, bd2_bits1 |= (1 << ETH_TX_DATA_2ND_BD_IPV6_EXT_SHIFT); - bd2_bits2 |= ((((u8 *)skb_transport_header(skb) - skb->data) >> 1) & + bd2_bits2 |= ((skb_transport_offset(skb) >> 1) & ETH_TX_DATA_2ND_BD_L4_HDR_START_OFFSET_W_MASK) << ETH_TX_DATA_2ND_BD_L4_HDR_START_OFFSET_W_SHIFT; diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c index 09f20c794754..d6cc6288436d 100644 --- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c +++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_init.c @@ -6,6 +6,7 @@ #include "qlcnic.h" #include "qlcnic_hw.h" +#include struct crb_addr_pair { u32 addr; @@ -740,159 +741,212 @@ qlcnic_has_mn(struct qlcnic_adapter *adapter) return 0; } -static -struct uni_table_desc *qlcnic_get_table_desc(const u8 *unirom, int section) -{ - u32 i, entries; - struct uni_table_desc *directory = (struct uni_table_desc *) &unirom[0]; - entries = le32_to_cpu(directory->num_entries); - - for (i = 0; i < entries; i++) { +#define FILEHEADER_SIZE (14 * 4) +#define QLCNIC_UNI_DIR_TYPE_OFF (8 * sizeof(__le32)) +#define QLCNIC_UNI_DIR_ENTRY_MIN_SIZE (9 * sizeof(__le32)) +#define QLCNIC_UNI_PRODUCT_ENTRY_MIN_SIZE \ + ((QLCNIC_UNI_FIRMWARE_IDX_OFF + 1) * sizeof(__le32)) +#define QLCNIC_UNI_VERSION_TAIL_SIZE 17 +#define QLCNIC_UNI_BOOTLD_SIZE \ + (QLCNIC_IMAGE_START - QLCNIC_BOOTLD_START) + +struct qlcnic_uni_data { + u32 offset; + u32 size; +}; - u32 offs = le32_to_cpu(directory->findex) + - i * le32_to_cpu(directory->entry_size); - u32 tab_type = le32_to_cpu(*((__le32 *)&unirom[offs] + 8)); +static bool qlcnic_rom_range_valid(size_t size, size_t offset, size_t len) +{ + return offset <= size && len <= size - offset; +} - if (tab_type == section) - return (struct uni_table_desc *) &unirom[offs]; - } +static bool qlcnic_rom_table_valid(size_t size, u32 offset, u32 entries, + u32 entry_size, u32 min_entry_size) +{ + if (entry_size < min_entry_size || offset > size) + return false; - return NULL; + return entries <= (size - offset) / entry_size; } -#define FILEHEADER_SIZE (14 * 4) - -static int -qlcnic_validate_header(struct qlcnic_adapter *adapter) +static int qlcnic_get_directory(struct qlcnic_adapter *adapter, + size_t *offset, u32 *entries, u32 *entry_size) { - const u8 *unirom = adapter->fw->data; - struct uni_table_desc *directory = (struct uni_table_desc *) &unirom[0]; - u32 entries, entry_size, tab_size, fw_file_size; - - fw_file_size = adapter->fw->size; + const struct firmware *fw = adapter->fw; + const u8 *directory = fw->data; - if (fw_file_size < FILEHEADER_SIZE) + if (fw->size < FILEHEADER_SIZE) return -EINVAL; - entries = le32_to_cpu(directory->num_entries); - entry_size = le32_to_cpu(directory->entry_size); - tab_size = le32_to_cpu(directory->findex) + (entries * entry_size); + *offset = get_unaligned_le32(directory + + offsetof(struct uni_table_desc, findex)); + *entries = get_unaligned_le32(directory + + offsetof(struct uni_table_desc, num_entries)); + *entry_size = get_unaligned_le32(directory + + offsetof(struct uni_table_desc, entry_size)); - if (fw_file_size < tab_size) + if (!qlcnic_rom_table_valid(fw->size, *offset, *entries, *entry_size, + QLCNIC_UNI_DIR_ENTRY_MIN_SIZE)) return -EINVAL; return 0; } -static int -qlcnic_validate_bootld(struct qlcnic_adapter *adapter) +static int qlcnic_get_table_desc(struct qlcnic_adapter *adapter, int section, + size_t *desc_offset) { - struct uni_table_desc *tab_desc; - struct uni_data_desc *descr; - u32 offs, tab_size, data_size, idx; const u8 *unirom = adapter->fw->data; - __le32 temp; - - temp = *((__le32 *)&unirom[adapter->file_prd_off] + - QLCNIC_UNI_BOOTLD_IDX_OFF); - idx = le32_to_cpu(temp); - tab_desc = qlcnic_get_table_desc(unirom, QLCNIC_UNI_DIR_SECT_BOOTLD); - - if (!tab_desc) - return -EINVAL; - - tab_size = le32_to_cpu(tab_desc->findex) + - le32_to_cpu(tab_desc->entry_size) * (idx + 1); - - if (adapter->fw->size < tab_size) - return -EINVAL; + size_t directory_offset; + u32 entries, entry_size; + size_t i; + int ret; - offs = le32_to_cpu(tab_desc->findex) + - le32_to_cpu(tab_desc->entry_size) * idx; - descr = (struct uni_data_desc *)&unirom[offs]; + ret = qlcnic_get_directory(adapter, &directory_offset, &entries, + &entry_size); + if (ret) + return ret; - data_size = le32_to_cpu(descr->findex) + le32_to_cpu(descr->size); + for (i = 0; i < entries; i++) { + size_t offset = directory_offset + i * entry_size; + u32 table_type; - if (adapter->fw->size < data_size) - return -EINVAL; + table_type = get_unaligned_le32(unirom + offset + + QLCNIC_UNI_DIR_TYPE_OFF); + if (table_type == section) { + *desc_offset = offset; + return 0; + } + } - return 0; + return -ENOENT; } static int -qlcnic_validate_fw(struct qlcnic_adapter *adapter) +qlcnic_validate_header(struct qlcnic_adapter *adapter) { - struct uni_table_desc *tab_desc; - struct uni_data_desc *descr; - const u8 *unirom = adapter->fw->data; - u32 offs, tab_size, data_size, idx; - __le32 temp; + u32 entries, entry_size; + size_t offset; + + return qlcnic_get_directory(adapter, &offset, &entries, &entry_size); +} - temp = *((__le32 *)&unirom[adapter->file_prd_off] + - QLCNIC_UNI_FIRMWARE_IDX_OFF); - idx = le32_to_cpu(temp); - tab_desc = qlcnic_get_table_desc(unirom, QLCNIC_UNI_DIR_SECT_FW); +static int qlcnic_get_data_desc(struct qlcnic_adapter *adapter, u32 section, + u32 index_offset, struct qlcnic_uni_data *data) +{ + size_t table_desc_offset, table_offset, desc_offset; + const struct firmware *fw = adapter->fw; + const u8 *unirom = fw->data; + size_t product_index_offset; + u32 entries, entry_size, idx; + int ret; - if (!tab_desc) + product_index_offset = adapter->file_prd_off + + (size_t)index_offset * sizeof(__le32); + if (!qlcnic_rom_range_valid(fw->size, product_index_offset, + sizeof(__le32))) return -EINVAL; - tab_size = le32_to_cpu(tab_desc->findex) + - le32_to_cpu(tab_desc->entry_size) * (idx + 1); + idx = get_unaligned_le32(unirom + product_index_offset); + ret = qlcnic_get_table_desc(adapter, section, &table_desc_offset); + if (ret) + return ret; - if (adapter->fw->size < tab_size) + table_offset = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, findex)); + entries = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, num_entries)); + entry_size = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, entry_size)); + if (!qlcnic_rom_table_valid(fw->size, table_offset, entries, + entry_size, sizeof(struct uni_data_desc)) || + idx >= entries) return -EINVAL; - offs = le32_to_cpu(tab_desc->findex) + - le32_to_cpu(tab_desc->entry_size) * idx; - descr = (struct uni_data_desc *)&unirom[offs]; - data_size = le32_to_cpu(descr->findex) + le32_to_cpu(descr->size); + desc_offset = table_offset + (size_t)idx * entry_size; + data->offset = get_unaligned_le32(unirom + desc_offset + + offsetof(struct uni_data_desc, findex)); + data->size = get_unaligned_le32(unirom + desc_offset + + offsetof(struct uni_data_desc, size)); - if (adapter->fw->size < data_size) + if (!qlcnic_rom_range_valid(fw->size, data->offset, data->size)) return -EINVAL; return 0; } +static int qlcnic_validate_bootld(struct qlcnic_adapter *adapter) +{ + struct qlcnic_uni_data data; + int ret; + + ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_BOOTLD, + QLCNIC_UNI_BOOTLD_IDX_OFF, &data); + if (ret) + return ret; + + return data.size < QLCNIC_UNI_BOOTLD_SIZE ? -EINVAL : 0; +} + +static int qlcnic_validate_fw(struct qlcnic_adapter *adapter) +{ + struct qlcnic_uni_data data; + int ret; + + ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, + QLCNIC_UNI_FIRMWARE_IDX_OFF, &data); + if (ret) + return ret; + + return data.size < QLCNIC_UNI_VERSION_TAIL_SIZE ? -EINVAL : 0; +} + static int qlcnic_validate_product_offs(struct qlcnic_adapter *adapter) { - struct uni_table_desc *ptab_descr; + size_t table_desc_offset, table_offset; const u8 *unirom = adapter->fw->data; int mn_present = qlcnic_has_mn(adapter); - u32 entries, entry_size, tab_size, i; - __le32 temp; - - ptab_descr = qlcnic_get_table_desc(unirom, - QLCNIC_UNI_DIR_SECT_PRODUCT_TBL); - if (!ptab_descr) - return -EINVAL; + u32 entries, entry_size; + size_t i; + int ret; - entries = le32_to_cpu(ptab_descr->num_entries); - entry_size = le32_to_cpu(ptab_descr->entry_size); - tab_size = le32_to_cpu(ptab_descr->findex) + (entries * entry_size); + ret = qlcnic_get_table_desc(adapter, QLCNIC_UNI_DIR_SECT_PRODUCT_TBL, + &table_desc_offset); + if (ret) + return ret; - if (adapter->fw->size < tab_size) + table_offset = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, findex)); + entries = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, num_entries)); + entry_size = get_unaligned_le32(unirom + table_desc_offset + + offsetof(struct uni_table_desc, entry_size)); + if (!qlcnic_rom_table_valid(adapter->fw->size, table_offset, entries, + entry_size, + QLCNIC_UNI_PRODUCT_ENTRY_MIN_SIZE)) return -EINVAL; nomn: for (i = 0; i < entries; i++) { - - u32 flags, file_chiprev, offs; + size_t offset = table_offset + i * entry_size; u8 chiprev = adapter->ahw->revision_id; + u32 flags, file_chiprev; u32 flagbit; - offs = le32_to_cpu(ptab_descr->findex) + - i * le32_to_cpu(ptab_descr->entry_size); - temp = *((__le32 *)&unirom[offs] + QLCNIC_UNI_FLAGS_OFF); - flags = le32_to_cpu(temp); - temp = *((__le32 *)&unirom[offs] + QLCNIC_UNI_CHIP_REV_OFF); - file_chiprev = le32_to_cpu(temp); + flags = get_unaligned_le32(unirom + offset + + QLCNIC_UNI_FLAGS_OFF * sizeof(__le32)); + file_chiprev = get_unaligned_le32(unirom + offset + + QLCNIC_UNI_CHIP_REV_OFF * + sizeof(__le32)); flagbit = mn_present ? 1 : 2; if ((chiprev == file_chiprev) && ((1ULL << flagbit) & flags)) { - adapter->file_prd_off = offs; + if (offset > U32_MAX) + return -EINVAL; + + adapter->file_prd_off = offset; return 0; } } @@ -933,93 +987,81 @@ qlcnic_validate_unified_romimage(struct qlcnic_adapter *adapter) return 0; } -static -struct uni_data_desc *qlcnic_get_data_desc(struct qlcnic_adapter *adapter, - u32 section, u32 idx_offset) -{ - const u8 *unirom = adapter->fw->data; - struct uni_table_desc *tab_desc; - u32 offs, idx; - __le32 temp; - - temp = *((__le32 *)&unirom[adapter->file_prd_off] + idx_offset); - idx = le32_to_cpu(temp); - - tab_desc = qlcnic_get_table_desc(unirom, section); - - if (tab_desc == NULL) - return NULL; - - offs = le32_to_cpu(tab_desc->findex) + - le32_to_cpu(tab_desc->entry_size) * idx; - - return (struct uni_data_desc *)&unirom[offs]; -} - -static u8 * -qlcnic_get_bootld_offs(struct qlcnic_adapter *adapter) +static int qlcnic_get_bootld_data(struct qlcnic_adapter *adapter, + const u8 **bootld) { u32 offs = QLCNIC_BOOTLD_START; - struct uni_data_desc *data_desc; - - data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_BOOTLD, - QLCNIC_UNI_BOOTLD_IDX_OFF); + struct qlcnic_uni_data data; + int ret; - if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) - offs = le32_to_cpu(data_desc->findex); + if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) { + ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_BOOTLD, + QLCNIC_UNI_BOOTLD_IDX_OFF, &data); + if (ret || data.size < QLCNIC_UNI_BOOTLD_SIZE) + return -EINVAL; + offs = data.offset; + } else if (!qlcnic_rom_range_valid(adapter->fw->size, offs, + QLCNIC_UNI_BOOTLD_SIZE)) { + return -EINVAL; + } - return (u8 *)&adapter->fw->data[offs]; + *bootld = adapter->fw->data + offs; + return 0; } -static u8 * -qlcnic_get_fw_offs(struct qlcnic_adapter *adapter) +static int qlcnic_get_fw_data(struct qlcnic_adapter *adapter, + const u8 **image, u32 *image_size) { u32 offs = QLCNIC_IMAGE_START; - struct uni_data_desc *data_desc; - - data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, - QLCNIC_UNI_FIRMWARE_IDX_OFF); - if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) - offs = le32_to_cpu(data_desc->findex); - - return (u8 *)&adapter->fw->data[offs]; -} + struct qlcnic_uni_data data; + int ret; -static u32 qlcnic_get_fw_size(struct qlcnic_adapter *adapter) -{ - struct uni_data_desc *data_desc; - const u8 *unirom = adapter->fw->data; + if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) { + ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, + QLCNIC_UNI_FIRMWARE_IDX_OFF, &data); + if (ret) + return ret; + offs = data.offset; + *image_size = data.size; + } else { + if (!qlcnic_rom_range_valid(adapter->fw->size, + QLCNIC_FW_SIZE_OFFSET, + sizeof(__le32))) + return -EINVAL; + *image_size = get_unaligned_le32(adapter->fw->data + + QLCNIC_FW_SIZE_OFFSET); + } - data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, - QLCNIC_UNI_FIRMWARE_IDX_OFF); + if (!qlcnic_rom_range_valid(adapter->fw->size, offs, *image_size)) + return -EINVAL; - if (adapter->ahw->fw_type == QLCNIC_UNIFIED_ROMIMAGE) - return le32_to_cpu(data_desc->size); - else - return le32_to_cpu(*(__le32 *)&unirom[QLCNIC_FW_SIZE_OFFSET]); + *image = adapter->fw->data + offs; + return 0; } static u32 qlcnic_get_fw_version(struct qlcnic_adapter *adapter) { - struct uni_data_desc *fw_data_desc; + char ver_str[QLCNIC_UNI_VERSION_TAIL_SIZE + 1]; const struct firmware *fw = adapter->fw; + struct qlcnic_uni_data data; u32 major, minor, sub; - __le32 version_offset; - const u8 *ver_str; int i, ret; - if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE) { - version_offset = *(__le32 *)&fw->data[QLCNIC_FW_VERSION_OFFSET]; - return le32_to_cpu(version_offset); - } + if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE) + return get_unaligned_le32(fw->data + QLCNIC_FW_VERSION_OFFSET); + + ret = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, + QLCNIC_UNI_FIRMWARE_IDX_OFF, &data); + if (ret || data.size < QLCNIC_UNI_VERSION_TAIL_SIZE) + return 0; - fw_data_desc = qlcnic_get_data_desc(adapter, QLCNIC_UNI_DIR_SECT_FW, - QLCNIC_UNI_FIRMWARE_IDX_OFF); - ver_str = fw->data + le32_to_cpu(fw_data_desc->findex) + - le32_to_cpu(fw_data_desc->size) - 17; + memcpy(ver_str, fw->data + data.offset + data.size - + QLCNIC_UNI_VERSION_TAIL_SIZE, + QLCNIC_UNI_VERSION_TAIL_SIZE); + ver_str[QLCNIC_UNI_VERSION_TAIL_SIZE] = '\0'; for (i = 0; i < 12; i++) { - if (!strncmp(&ver_str[i], "REV=", 4)) { + if (!strncmp(ver_str + i, "REV=", 4)) { ret = sscanf(&ver_str[i+4], "%u.%u.%u ", &major, &minor, &sub); if (ret != 3) @@ -1034,18 +1076,15 @@ static u32 qlcnic_get_fw_version(struct qlcnic_adapter *adapter) static u32 qlcnic_get_bios_version(struct qlcnic_adapter *adapter) { - const struct firmware *fw = adapter->fw; u32 bios_ver, prd_off = adapter->file_prd_off; - u8 *version_offset; - __le32 temp; + const struct firmware *fw = adapter->fw; - if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE) { - version_offset = (u8 *)&fw->data[QLCNIC_BIOS_VERSION_OFFSET]; - return le32_to_cpu(*(__le32 *)version_offset); - } + if (adapter->ahw->fw_type != QLCNIC_UNIFIED_ROMIMAGE) + return get_unaligned_le32(fw->data + QLCNIC_BIOS_VERSION_OFFSET); - temp = *((__le32 *)(&fw->data[prd_off]) + QLCNIC_UNI_BIOS_VERSION_OFF); - bios_ver = le32_to_cpu(temp); + bios_ver = get_unaligned_le32(fw->data + prd_off + + QLCNIC_UNI_BIOS_VERSION_OFF * + sizeof(__le32)); return (bios_ver << 16) + ((bios_ver >> 8) & 0xff00) + (bios_ver >> 24); } @@ -1106,24 +1145,26 @@ static const char *fw_name[] = { int qlcnic_load_firmware(struct qlcnic_adapter *adapter) { - __le64 *ptr64; - u32 i, flashaddr, size; const struct firmware *fw = adapter->fw; struct pci_dev *pdev = adapter->pdev; + const u8 *bootld, *image; + u32 i, flashaddr, image_size; + int ret; dev_info(&pdev->dev, "loading firmware from %s\n", fw_name[adapter->ahw->fw_type]); if (fw) { + u32 words, remainder; u64 data; - size = (QLCNIC_IMAGE_START - QLCNIC_BOOTLD_START) / 8; - - ptr64 = (__le64 *)qlcnic_get_bootld_offs(adapter); + ret = qlcnic_get_bootld_data(adapter, &bootld); + if (ret) + return ret; flashaddr = QLCNIC_BOOTLD_START; - for (i = 0; i < size; i++) { - data = le64_to_cpu(ptr64[i]); + for (i = 0; i < QLCNIC_UNI_BOOTLD_SIZE / sizeof(u64); i++) { + data = get_unaligned_le64(bootld + i * sizeof(u64)); if (qlcnic_pci_mem_write_2M(adapter, flashaddr, data)) return -EIO; @@ -1131,13 +1172,15 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter) flashaddr += 8; } - size = qlcnic_get_fw_size(adapter) / 8; - - ptr64 = (__le64 *)qlcnic_get_fw_offs(adapter); + ret = qlcnic_get_fw_data(adapter, &image, &image_size); + if (ret) + return ret; + words = image_size / sizeof(u64); + remainder = image_size % sizeof(u64); flashaddr = QLCNIC_IMAGE_START; - for (i = 0; i < size; i++) { - data = le64_to_cpu(ptr64[i]); + for (i = 0; i < words; i++) { + data = get_unaligned_le64(image + i * sizeof(u64)); if (qlcnic_pci_mem_write_2M(adapter, flashaddr, data)) @@ -1146,9 +1189,11 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter) flashaddr += 8; } - size = qlcnic_get_fw_size(adapter) % 8; - if (size) { - data = le64_to_cpu(ptr64[i]); + if (remainder) { + __le64 tail = 0; + + memcpy(&tail, image + words * sizeof(u64), remainder); + data = le64_to_cpu(tail); if (qlcnic_pci_mem_write_2M(adapter, flashaddr, data)) @@ -1156,10 +1201,9 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter) } } else { - u64 data; - u32 hi, lo; - int ret; struct qlcnic_flt_entry bootld_entry; + u32 hi, lo, size; + u64 data; ret = qlcnic_get_flt_entry(adapter, QLCNIC_BOOTLD_REGION, &bootld_entry); @@ -1200,26 +1244,33 @@ qlcnic_load_firmware(struct qlcnic_adapter *adapter) static int qlcnic_validate_firmware(struct qlcnic_adapter *adapter) { - u32 val; - u32 ver, bios, min_size; - struct pci_dev *pdev = adapter->pdev; const struct firmware *fw = adapter->fw; + struct pci_dev *pdev = adapter->pdev; u8 fw_type = adapter->ahw->fw_type; + u32 ver, bios, min_size; + const u8 *data; + u32 data_size; + u32 val; + + if (fw_type == QLCNIC_UNIFIED_ROMIMAGE) + min_size = QLCNIC_UNI_FW_MIN_SIZE; + else + min_size = QLCNIC_FW_MIN_SIZE; + + if (fw->size < min_size) + return -EINVAL; if (fw_type == QLCNIC_UNIFIED_ROMIMAGE) { if (qlcnic_validate_unified_romimage(adapter)) return -EINVAL; - - min_size = QLCNIC_UNI_FW_MIN_SIZE; } else { - val = le32_to_cpu(*(__le32 *)&fw->data[QLCNIC_FW_MAGIC_OFFSET]); + val = get_unaligned_le32(fw->data + QLCNIC_FW_MAGIC_OFFSET); if (val != QLCNIC_BDINFO_MAGIC) return -EINVAL; - - min_size = QLCNIC_FW_MIN_SIZE; } - if (fw->size < min_size) + if (qlcnic_get_bootld_data(adapter, &data) || + qlcnic_get_fw_data(adapter, &data, &data_size)) return -EINVAL; val = qlcnic_get_fw_version(adapter); diff --git a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c index 41894d154013..b9dc0071c5de 100644 --- a/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c +++ b/drivers/net/ethernet/qlogic/qlcnic/qlcnic_io.c @@ -446,8 +446,7 @@ static int qlcnic_tx_encap_pkt(struct qlcnic_adapter *adapter, encap_descr |= skb_network_offset(skb) << 10; first_desc->encap_descr = cpu_to_le16(encap_descr); - first_desc->tcp_hdr_offset = skb_inner_transport_header(skb) - - skb->data; + first_desc->tcp_hdr_offset = skb_inner_transport_offset(skb); first_desc->ip_hdr_offset = skb_inner_network_offset(skb); qlcnic_set_tx_flags_opcode(first_desc, flags, opcode); diff --git a/drivers/net/ethernet/realtek/r8169_main.c b/drivers/net/ethernet/realtek/r8169_main.c index 2e731b5587a2..39d97c56166d 100644 --- a/drivers/net/ethernet/realtek/r8169_main.c +++ b/drivers/net/ethernet/realtek/r8169_main.c @@ -4669,11 +4669,7 @@ static void r8169_phylink_handler(struct net_device *ndev) if (netif_carrier_ok(ndev)) { rtl_link_chg_patch(tp); pm_request_resume(d); - netif_wake_queue(tp->dev); } else { - /* In few cases rx is broken after link-down otherwise */ - if (rtl_is_8125(tp)) - rtl_schedule_task(tp, RTL_FLAG_TASK_RESET_NO_QUEUE_WAKE); pm_runtime_idle(d); } diff --git a/drivers/net/ethernet/renesas/ravb_main.c b/drivers/net/ethernet/renesas/ravb_main.c index 1f479bcda8be..f5c3f88cee97 100644 --- a/drivers/net/ethernet/renesas/ravb_main.c +++ b/drivers/net/ethernet/renesas/ravb_main.c @@ -3012,10 +3012,12 @@ static int __maybe_unused ravb_suspend(struct device *dev) netif_device_detach(ndev); + rtnl_lock(); if (priv->wol_enabled) ret = ravb_wol_setup(ndev); else ret = ravb_close(ndev); + rtnl_unlock(); if (priv->info->ccc_gac) ravb_ptp_stop(ndev); @@ -3067,12 +3069,16 @@ static int __maybe_unused ravb_resume(struct device *dev) ravb_ptp_init(ndev, priv->pdev); if (netif_running(ndev)) { + rtnl_lock(); if (priv->wol_enabled) { ret = ravb_wol_restore(ndev); - if (ret) + if (ret) { + rtnl_unlock(); return ret; + } } ret = ravb_open(ndev); + rtnl_unlock(); if (ret < 0) return ret; ravb_set_rx_mode(ndev); diff --git a/drivers/net/ethernet/renesas/ravb_ptp.c b/drivers/net/ethernet/renesas/ravb_ptp.c index fd674f121086..5806db5d57d5 100644 --- a/drivers/net/ethernet/renesas/ravb_ptp.c +++ b/drivers/net/ethernet/renesas/ravb_ptp.c @@ -299,16 +299,17 @@ static const struct ptp_clock_info ravb_ptp_info = { void ravb_ptp_interrupt(struct net_device *ndev) { struct ravb_private *priv = netdev_priv(ndev); + struct ptp_clock *clock = READ_ONCE(priv->ptp.clock); u32 gis = ravb_read(ndev, GIS); gis &= ravb_read(ndev, GIC); - if (gis & GIS_PTCF) { + if ((gis & GIS_PTCF) && clock) { struct ptp_clock_event event; event.type = PTP_CLOCK_EXTTS; event.index = 0; event.timestamp = ravb_read(ndev, GCPT); - ptp_clock_event(priv->ptp.clock, &event); + ptp_clock_event(clock, &event); } if (gis & GIS_PTMF) { struct ravb_ptp_perout *perout = priv->ptp.perout; @@ -344,19 +345,39 @@ void ravb_ptp_init(struct net_device *ndev, struct platform_device *pdev) clock = NULL; } - priv->ptp.clock = clock; + WRITE_ONCE(priv->ptp.clock, clock); if (clock) WRITE_ONCE(priv->ptp.phc_index, ptp_clock_index(clock)); } -void ravb_ptp_stop(struct net_device *ndev) +static void ravb_ptp_disable(struct net_device *ndev) { - struct ravb_private *priv = netdev_priv(ndev); - ravb_write(ndev, 0, GIC); ravb_write(ndev, 0, GIS); +} + +static void ravb_ptp_sync_irqs(struct net_device *ndev) +{ + struct ravb_private *priv = netdev_priv(ndev); + + synchronize_irq(ndev->irq); + if (priv->info->err_mgmt_irqs) { + synchronize_irq(priv->erra_irq); + synchronize_irq(priv->mgmta_irq); + } +} + +void ravb_ptp_stop(struct net_device *ndev) +{ + struct ravb_private *priv = netdev_priv(ndev); + struct ptp_clock *clock; WRITE_ONCE(priv->ptp.phc_index, -1); - if (priv->ptp.clock) - ptp_clock_unregister(priv->ptp.clock); + clock = xchg(&priv->ptp.clock, NULL); + + ravb_ptp_disable(ndev); + ravb_ptp_sync_irqs(ndev); + + if (clock) + ptp_clock_unregister(clock); } diff --git a/drivers/net/ethernet/sfc/ef10.c b/drivers/net/ethernet/sfc/ef10.c index 6dfa062feebc..8fa6c0e9195b 100644 --- a/drivers/net/ethernet/sfc/ef10.c +++ b/drivers/net/ethernet/sfc/ef10.c @@ -3706,13 +3706,13 @@ static int efx_ef10_ptp_set_ts_sync_events(struct efx_nic *efx, bool en, } static int efx_ef10_ptp_set_ts_config_vf(struct efx_nic *efx, - struct hwtstamp_config *init) + struct kernel_hwtstamp_config *init) { return -EOPNOTSUPP; } static int efx_ef10_ptp_set_ts_config(struct efx_nic *efx, - struct hwtstamp_config *init) + struct kernel_hwtstamp_config *init) { int rc; diff --git a/drivers/net/ethernet/sfc/ef100_netdev.c b/drivers/net/ethernet/sfc/ef100_netdev.c index 7f7d560cb2b4..3a06e3b1bd6b 100644 --- a/drivers/net/ethernet/sfc/ef100_netdev.c +++ b/drivers/net/ethernet/sfc/ef100_netdev.c @@ -450,9 +450,9 @@ int ef100_probe_netdev(struct efx_probe_data *probe_data) net_dev->hw_enc_features |= efx->type->offload_features; net_dev->vlan_features |= NETIF_F_HW_CSUM | NETIF_F_SG | NETIF_F_HIGHDMA | NETIF_F_ALL_TSO; - netif_set_tso_max_segs(net_dev, - ESE_EF100_DP_GZ_TSO_MAX_HDR_NUM_SEGS_DEFAULT); - efx->mdio.dev = net_dev; + nic_data = efx->nic_data; + netif_set_tso_max_size(efx->net_dev, nic_data->tso_max_payload_len); + netif_set_tso_max_segs(efx->net_dev, nic_data->tso_max_payload_num_segs); rc = efx_ef100_init_datapath_caps(efx); if (rc < 0) @@ -478,7 +478,6 @@ int ef100_probe_netdev(struct efx_probe_data *probe_data) /* Don't fail init if RSS setup doesn't work. */ efx_mcdi_push_default_indir_table(efx, efx->n_rx_channels); - nic_data = efx->nic_data; rc = ef100_get_mac_address(efx, net_dev->perm_addr, CLIENT_HANDLE_SELF, efx->type->is_vf); if (rc) diff --git a/drivers/net/ethernet/sfc/ef100_nic.c b/drivers/net/ethernet/sfc/ef100_nic.c index 6da06931187d..5b1bdcac81d9 100644 --- a/drivers/net/ethernet/sfc/ef100_nic.c +++ b/drivers/net/ethernet/sfc/ef100_nic.c @@ -887,8 +887,7 @@ static int ef100_process_design_param(struct efx_nic *efx, case ESE_EF100_DP_GZ_TSO_MAX_HDR_NUM_SEGS: /* We always put HDR_NUM_SEGS=1 in our TSO descriptors */ if (!reader->value) { - netif_err(efx, probe, efx->net_dev, - "TSO_MAX_HDR_NUM_SEGS < 1\n"); + pci_err(efx->pci_dev, "TSO_MAX_HDR_NUM_SEGS < 1\n"); return -EOPNOTSUPP; } return 0; @@ -901,32 +900,28 @@ static int ef100_process_design_param(struct efx_nic *efx, */ if (!reader->value || reader->value > EFX_MIN_DMAQ_SIZE || EFX_MIN_DMAQ_SIZE % (u32)reader->value) { - netif_err(efx, probe, efx->net_dev, - "%s size granularity is %llu, can't guarantee safety\n", - reader->type == ESE_EF100_DP_GZ_RXQ_SIZE_GRANULARITY ? "RXQ" : "TXQ", - reader->value); + pci_err(efx->pci_dev, + "%s size granularity is %llu, can't guarantee safety\n", + reader->type == ESE_EF100_DP_GZ_RXQ_SIZE_GRANULARITY ? "RXQ" : "TXQ", + reader->value); return -EOPNOTSUPP; } return 0; case ESE_EF100_DP_GZ_TSO_MAX_PAYLOAD_LEN: nic_data->tso_max_payload_len = min_t(u64, reader->value, GSO_LEGACY_MAX_SIZE); - netif_set_tso_max_size(efx->net_dev, - nic_data->tso_max_payload_len); return 0; case ESE_EF100_DP_GZ_TSO_MAX_PAYLOAD_NUM_SEGS: nic_data->tso_max_payload_num_segs = min_t(u64, reader->value, 0xffff); - netif_set_tso_max_segs(efx->net_dev, - nic_data->tso_max_payload_num_segs); return 0; case ESE_EF100_DP_GZ_TSO_MAX_NUM_FRAMES: nic_data->tso_max_frames = min_t(u64, reader->value, 0xffff); return 0; case ESE_EF100_DP_GZ_COMPAT: if (reader->value) { - netif_err(efx, probe, efx->net_dev, - "DP_COMPAT has unknown bits %#llx, driver not compatible with this hw\n", - reader->value); + pci_err(efx->pci_dev, + "DP_COMPAT has unknown bits %#llx, driver not compatible with this hw\n", + reader->value); return -EOPNOTSUPP; } return 0; @@ -946,10 +941,10 @@ static int ef100_process_design_param(struct efx_nic *efx, * So the value of this shouldn't matter. */ if (reader->value != ESE_EF100_DP_GZ_VI_STRIDES_DEFAULT) - netif_dbg(efx, probe, efx->net_dev, - "NIC has other than default VI_STRIDES (mask " - "%#llx), early probing might use wrong one\n", - reader->value); + pci_dbg(efx->pci_dev, + "NIC has other than default VI_STRIDES (mask " + "%#llx), early probing might use wrong one\n", + reader->value); return 0; case ESE_EF100_DP_GZ_RX_MAX_RUNT: /* Driver doesn't look at L2_STATUS:LEN_ERR bit, so we don't @@ -961,9 +956,9 @@ static int ef100_process_design_param(struct efx_nic *efx, /* Host interface says "Drivers should ignore design parameters * that they do not recognise." */ - netif_dbg(efx, probe, efx->net_dev, - "Ignoring unrecognised design parameter %u\n", - reader->type); + pci_dbg(efx->pci_dev, + "Ignoring unrecognised design parameter %u\n", + reader->type); return 0; } } @@ -999,13 +994,13 @@ static int ef100_check_design_params(struct efx_nic *efx) */ if (reader.state != EF100_TLV_TYPE) { if (reader.state == EF100_TLV_TYPE_CONT) - netif_err(efx, probe, efx->net_dev, - "truncated design parameter (incomplete type %u)\n", - reader.type); + pci_err(efx->pci_dev, + "truncated design parameter (incomplete type %u)\n", + reader.type); else - netif_err(efx, probe, efx->net_dev, - "truncated design parameter %u\n", - reader.type); + pci_err(efx->pci_dev, + "truncated design parameter %u\n", + reader.type); rc = -EIO; } out: diff --git a/drivers/net/ethernet/sfc/efx.c b/drivers/net/ethernet/sfc/efx.c index 19f4b4d0b851..6395fda5906d 100644 --- a/drivers/net/ethernet/sfc/efx.c +++ b/drivers/net/ethernet/sfc/efx.c @@ -481,33 +481,6 @@ void efx_get_irq_moderation(struct efx_nic *efx, unsigned int *tx_usecs, } } -/************************************************************************** - * - * ioctls - * - *************************************************************************/ - -/* Net device ioctl - * Context: process, rtnl_lock() held. - */ -static int efx_ioctl(struct net_device *net_dev, struct ifreq *ifr, int cmd) -{ - struct efx_nic *efx = efx_netdev_priv(net_dev); - struct mii_ioctl_data *data = if_mii(ifr); - - if (cmd == SIOCSHWTSTAMP) - return efx_ptp_set_ts_config(efx, ifr); - if (cmd == SIOCGHWTSTAMP) - return efx_ptp_get_ts_config(efx, ifr); - - /* Convert phy_id from older PRTAD/DEVAD format */ - if ((cmd == SIOCGMIIREG || cmd == SIOCSMIIREG) && - (data->phy_id & 0xfc00) == 0x0400) - data->phy_id ^= MDIO_PHY_ID_C45 | 0x0400; - - return mdio_mii_ioctl(&efx->mdio, data, cmd); -} - /************************************************************************** * * Kernel net device interface @@ -581,6 +554,23 @@ static int efx_vlan_rx_kill_vid(struct net_device *net_dev, __be16 proto, u16 vi return -EOPNOTSUPP; } +static int efx_hwtstamp_set(struct net_device *net_dev, + struct kernel_hwtstamp_config *config, + struct netlink_ext_ack *extack) +{ + struct efx_nic *efx = efx_netdev_priv(net_dev); + + return efx_ptp_set_ts_config(efx, config, extack); +} + +static int efx_hwtstamp_get(struct net_device *net_dev, + struct kernel_hwtstamp_config *config) +{ + struct efx_nic *efx = efx_netdev_priv(net_dev); + + return efx_ptp_get_ts_config(efx, config); +} + static const struct net_device_ops efx_netdev_ops = { .ndo_open = efx_net_open, .ndo_stop = efx_net_stop, @@ -588,7 +578,6 @@ static const struct net_device_ops efx_netdev_ops = { .ndo_tx_timeout = efx_watchdog, .ndo_start_xmit = efx_hard_start_xmit, .ndo_validate_addr = eth_validate_addr, - .ndo_eth_ioctl = efx_ioctl, .ndo_change_mtu = efx_change_mtu, .ndo_set_mac_address = efx_set_mac_address, .ndo_set_rx_mode = efx_set_rx_mode, @@ -596,6 +585,8 @@ static const struct net_device_ops efx_netdev_ops = { .ndo_features_check = efx_features_check, .ndo_vlan_rx_add_vid = efx_vlan_rx_add_vid, .ndo_vlan_rx_kill_vid = efx_vlan_rx_kill_vid, + .ndo_hwtstamp_set = efx_hwtstamp_set, + .ndo_hwtstamp_get = efx_hwtstamp_get, #ifdef CONFIG_SFC_SRIOV .ndo_set_vf_mac = efx_sriov_set_vf_mac, .ndo_set_vf_vlan = efx_sriov_set_vf_vlan, @@ -1082,7 +1073,6 @@ static int efx_pci_probe(struct pci_dev *pci_dev, rc = efx_init_struct(efx, pci_dev); if (rc) goto fail1; - efx->mdio.dev = net_dev; pci_info(pci_dev, "Solarflare NIC detected\n"); diff --git a/drivers/net/ethernet/sfc/mcdi_port.c b/drivers/net/ethernet/sfc/mcdi_port.c index ad4694fa3dda..7b236d291d8c 100644 --- a/drivers/net/ethernet/sfc/mcdi_port.c +++ b/drivers/net/ethernet/sfc/mcdi_port.c @@ -17,58 +17,6 @@ #include "selftest.h" #include "mcdi_port_common.h" -static int efx_mcdi_mdio_read(struct net_device *net_dev, - int prtad, int devad, u16 addr) -{ - struct efx_nic *efx = efx_netdev_priv(net_dev); - MCDI_DECLARE_BUF(inbuf, MC_CMD_MDIO_READ_IN_LEN); - MCDI_DECLARE_BUF(outbuf, MC_CMD_MDIO_READ_OUT_LEN); - size_t outlen; - int rc; - - MCDI_SET_DWORD(inbuf, MDIO_READ_IN_BUS, efx->mdio_bus); - MCDI_SET_DWORD(inbuf, MDIO_READ_IN_PRTAD, prtad); - MCDI_SET_DWORD(inbuf, MDIO_READ_IN_DEVAD, devad); - MCDI_SET_DWORD(inbuf, MDIO_READ_IN_ADDR, addr); - - rc = efx_mcdi_rpc(efx, MC_CMD_MDIO_READ, inbuf, sizeof(inbuf), - outbuf, sizeof(outbuf), &outlen); - if (rc) - return rc; - - if (MCDI_DWORD(outbuf, MDIO_READ_OUT_STATUS) != - MC_CMD_MDIO_STATUS_GOOD) - return -EIO; - - return (u16)MCDI_DWORD(outbuf, MDIO_READ_OUT_VALUE); -} - -static int efx_mcdi_mdio_write(struct net_device *net_dev, - int prtad, int devad, u16 addr, u16 value) -{ - struct efx_nic *efx = efx_netdev_priv(net_dev); - MCDI_DECLARE_BUF(inbuf, MC_CMD_MDIO_WRITE_IN_LEN); - MCDI_DECLARE_BUF(outbuf, MC_CMD_MDIO_WRITE_OUT_LEN); - size_t outlen; - int rc; - - MCDI_SET_DWORD(inbuf, MDIO_WRITE_IN_BUS, efx->mdio_bus); - MCDI_SET_DWORD(inbuf, MDIO_WRITE_IN_PRTAD, prtad); - MCDI_SET_DWORD(inbuf, MDIO_WRITE_IN_DEVAD, devad); - MCDI_SET_DWORD(inbuf, MDIO_WRITE_IN_ADDR, addr); - MCDI_SET_DWORD(inbuf, MDIO_WRITE_IN_VALUE, value); - - rc = efx_mcdi_rpc(efx, MC_CMD_MDIO_WRITE, inbuf, sizeof(inbuf), - outbuf, sizeof(outbuf), &outlen); - if (rc) - return rc; - - if (MCDI_DWORD(outbuf, MDIO_WRITE_OUT_STATUS) != - MC_CMD_MDIO_STATUS_GOOD) - return -EIO; - - return 0; -} u32 efx_mcdi_phy_get_caps(struct efx_nic *efx) { @@ -97,12 +45,7 @@ int efx_mcdi_port_probe(struct efx_nic *efx) { int rc; - /* Set up MDIO structure for PHY */ - efx->mdio.mode_support = MDIO_SUPPORTS_C45 | MDIO_EMULATE_C22; - efx->mdio.mdio_read = efx_mcdi_mdio_read; - efx->mdio.mdio_write = efx_mcdi_mdio_write; - - /* Fill out MDIO structure, loopback modes, and initial link state */ + /* Fill out loopback modes and initial link state */ rc = efx_mcdi_phy_probe(efx); if (rc != 0) return rc; diff --git a/drivers/net/ethernet/sfc/mcdi_port_common.c b/drivers/net/ethernet/sfc/mcdi_port_common.c index 76ea26722ca4..dae684194ac8 100644 --- a/drivers/net/ethernet/sfc/mcdi_port_common.c +++ b/drivers/net/ethernet/sfc/mcdi_port_common.c @@ -448,15 +448,6 @@ int efx_mcdi_phy_probe(struct efx_nic *efx) efx->phy_data = phy_data; efx->phy_type = phy_data->type; - efx->mdio_bus = phy_data->channel; - efx->mdio.prtad = phy_data->port; - efx->mdio.mmds = phy_data->mmd_mask & ~(1 << MC_CMD_MMD_CLAUSE22); - efx->mdio.mode_support = 0; - if (phy_data->mmd_mask & (1 << MC_CMD_MMD_CLAUSE22)) - efx->mdio.mode_support |= MDIO_SUPPORTS_C22; - if (phy_data->mmd_mask & ~(1 << MC_CMD_MMD_CLAUSE22)) - efx->mdio.mode_support |= MDIO_SUPPORTS_C45 | MDIO_EMULATE_C22; - caps = MCDI_DWORD(outbuf, GET_LINK_OUT_CAP); if (caps & (1 << MC_CMD_PHY_CAP_AN_LBN)) mcdi_to_ethtool_linkset(phy_data->media, caps, @@ -546,8 +537,6 @@ void efx_mcdi_phy_get_link_ksettings(struct efx_nic *efx, struct ethtool_link_ks cmd->base.port = mcdi_to_ethtool_media(phy_cfg->media); cmd->base.phy_address = phy_cfg->port; cmd->base.autoneg = !!(efx->link_advertising[0] & ADVERTISED_Autoneg); - cmd->base.mdio_support = (efx->mdio.mode_support & - (MDIO_SUPPORTS_C45 | MDIO_SUPPORTS_C22)); mcdi_to_ethtool_linkset(phy_cfg->media, phy_cfg->supported_cap, cmd->link_modes.supported); diff --git a/drivers/net/ethernet/sfc/net_driver.h b/drivers/net/ethernet/sfc/net_driver.h index 27d86e90a3bb..37e7e3059315 100644 --- a/drivers/net/ethernet/sfc/net_driver.h +++ b/drivers/net/ethernet/sfc/net_driver.h @@ -15,7 +15,7 @@ #include #include #include -#include +#include #include #include #include @@ -914,8 +914,6 @@ struct efx_mae; * @stats_buffer: DMA buffer for statistics * @phy_type: PHY type * @phy_data: PHY private data (including PHY-specific stats) - * @mdio: PHY MDIO interface - * @mdio_bus: PHY MDIO bus ID (only used by Siena) * @phy_mode: PHY operating mode. Serialised by @mac_lock. * @link_advertising: Autonegotiation advertising flags * @fec_config: Forward Error Correction configuration flags. For bit positions @@ -1090,8 +1088,6 @@ struct efx_nic { unsigned int phy_type; void *phy_data; - struct mdio_if_info mdio; - unsigned int mdio_bus; enum efx_phy_mode phy_mode; __ETHTOOL_DECLARE_LINK_MODE_MASK(link_advertising); @@ -1473,7 +1469,7 @@ struct efx_nic_type { void (*ptp_write_host_time)(struct efx_nic *efx, u32 host_time); int (*ptp_set_ts_sync_events)(struct efx_nic *efx, bool en, bool temp); int (*ptp_set_ts_config)(struct efx_nic *efx, - struct hwtstamp_config *init); + struct kernel_hwtstamp_config *init); int (*sriov_configure)(struct efx_nic *efx, int num_vfs); int (*vlan_rx_add_vid)(struct efx_nic *efx, __be16 proto, u16 vid); int (*vlan_rx_kill_vid)(struct efx_nic *efx, __be16 proto, u16 vid); diff --git a/drivers/net/ethernet/sfc/ptp.c b/drivers/net/ethernet/sfc/ptp.c index f54200f03e15..f5227ae8cc43 100644 --- a/drivers/net/ethernet/sfc/ptp.c +++ b/drivers/net/ethernet/sfc/ptp.c @@ -295,7 +295,7 @@ struct efx_ptp_data { bool reset_required; struct list_head rxfilters_mcast; struct list_head rxfilters_ucast; - struct hwtstamp_config config; + struct kernel_hwtstamp_config config; bool enabled; unsigned int mode; void (*ns_to_nic_time)(s64 ns, u32 *nic_major, u32 *nic_minor); @@ -1847,7 +1847,7 @@ int efx_ptp_change_mode(struct efx_nic *efx, bool enable_wanted, return 0; } -static int efx_ptp_ts_init(struct efx_nic *efx, struct hwtstamp_config *init) +static int efx_ptp_ts_init(struct efx_nic *efx, struct kernel_hwtstamp_config *init) { int rc; @@ -1894,33 +1894,25 @@ void efx_ptp_get_ts_info(struct efx_nic *efx, struct ethtool_ts_info *ts_info) ts_info->rx_filters = ptp->efx->type->hwtstamp_filters; } -int efx_ptp_set_ts_config(struct efx_nic *efx, struct ifreq *ifr) +int efx_ptp_set_ts_config(struct efx_nic *efx, + struct kernel_hwtstamp_config *config, + struct netlink_ext_ack __always_unused *extack) { - struct hwtstamp_config config; - int rc; - /* Not a PTP enabled port */ if (!efx->ptp_data) return -EOPNOTSUPP; - if (copy_from_user(&config, ifr->ifr_data, sizeof(config))) - return -EFAULT; - - rc = efx_ptp_ts_init(efx, &config); - if (rc != 0) - return rc; - - return copy_to_user(ifr->ifr_data, &config, sizeof(config)) - ? -EFAULT : 0; + return efx_ptp_ts_init(efx, config); } -int efx_ptp_get_ts_config(struct efx_nic *efx, struct ifreq *ifr) +int efx_ptp_get_ts_config(struct efx_nic *efx, + struct kernel_hwtstamp_config *config) { + /* Not a PTP enabled port */ if (!efx->ptp_data) return -EOPNOTSUPP; - - return copy_to_user(ifr->ifr_data, &efx->ptp_data->config, - sizeof(efx->ptp_data->config)) ? -EFAULT : 0; + *config = efx->ptp_data->config; + return 0; } static void ptp_event_failure(struct efx_nic *efx, int expected_frag_len) diff --git a/drivers/net/ethernet/sfc/ptp.h b/drivers/net/ethernet/sfc/ptp.h index 7b1ef7002b3f..2f30dbb490d2 100644 --- a/drivers/net/ethernet/sfc/ptp.h +++ b/drivers/net/ethernet/sfc/ptp.h @@ -18,8 +18,11 @@ void efx_ptp_defer_probe_with_channel(struct efx_nic *efx); struct efx_channel *efx_ptp_channel(struct efx_nic *efx); void efx_ptp_update_channel(struct efx_nic *efx, struct efx_channel *channel); void efx_ptp_remove(struct efx_nic *efx); -int efx_ptp_set_ts_config(struct efx_nic *efx, struct ifreq *ifr); -int efx_ptp_get_ts_config(struct efx_nic *efx, struct ifreq *ifr); +int efx_ptp_set_ts_config(struct efx_nic *efx, + struct kernel_hwtstamp_config *config, + struct netlink_ext_ack *extack); +int efx_ptp_get_ts_config(struct efx_nic *efx, + struct kernel_hwtstamp_config *config); void efx_ptp_get_ts_info(struct efx_nic *efx, struct ethtool_ts_info *ts_info); bool efx_ptp_is_ptp_tx(struct efx_nic *efx, struct sk_buff *skb); int efx_ptp_get_mode(struct efx_nic *efx); diff --git a/drivers/net/ethernet/sfc/siena/tx_common.c b/drivers/net/ethernet/sfc/siena/tx_common.c index a7a9ab304e13..71f9b5ec5ae4 100644 --- a/drivers/net/ethernet/sfc/siena/tx_common.c +++ b/drivers/net/ethernet/sfc/siena/tx_common.c @@ -317,11 +317,10 @@ static int efx_tx_tso_header_length(struct sk_buff *skb) size_t header_len; if (skb->encapsulation) - header_len = skb_inner_transport_header(skb) - - skb->data + + header_len = skb_inner_transport_offset(skb) + (inner_tcp_hdr(skb)->doff << 2u); else - header_len = skb_transport_header(skb) - skb->data + + header_len = skb_transport_offset(skb) + (tcp_hdr(skb)->doff << 2u); return header_len; } diff --git a/drivers/net/ethernet/sfc/tx_common.c b/drivers/net/ethernet/sfc/tx_common.c index 9f2393d34371..2adb132b2f7e 100644 --- a/drivers/net/ethernet/sfc/tx_common.c +++ b/drivers/net/ethernet/sfc/tx_common.c @@ -336,11 +336,10 @@ int efx_tx_tso_header_length(struct sk_buff *skb) size_t header_len; if (skb->encapsulation) - header_len = skb_inner_transport_header(skb) - - skb->data + + header_len = skb_inner_transport_offset(skb) + (inner_tcp_hdr(skb)->doff << 2u); else - header_len = skb_transport_header(skb) - skb->data + + header_len = skb_transport_offset(skb) + (tcp_hdr(skb)->doff << 2u); return header_len; } diff --git a/drivers/net/ethernet/sfc/tx_tso.c b/drivers/net/ethernet/sfc/tx_tso.c index 64a6768f75ea..ddf149db8180 100644 --- a/drivers/net/ethernet/sfc/tx_tso.c +++ b/drivers/net/ethernet/sfc/tx_tso.c @@ -174,8 +174,8 @@ static int tso_start(struct tso_state *st, struct efx_nic *efx, unsigned int header_len, in_len; dma_addr_t dma_addr; - st->ip_off = skb_network_header(skb) - skb->data; - st->tcp_off = skb_transport_header(skb) - skb->data; + st->ip_off = skb_network_offset(skb); + st->tcp_off = skb_transport_offset(skb); header_len = st->tcp_off + (tcp_hdr(skb)->doff << 2u); in_len = skb_headlen(skb) - header_len; st->header_len = header_len; diff --git a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c index 435189e35b8a..6f7a2fec7f13 100644 --- a/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c +++ b/drivers/net/ethernet/stmicro/stmmac/stmmac_main.c @@ -4040,6 +4040,26 @@ static void stmmac_flush_tx_descriptors(struct stmmac_priv *priv, int queue) stmmac_set_tx_tail_ptr(priv, priv->ioaddr, tx_q->tx_tail_addr, queue); } +static int stmmac_tso_get_num_desc(struct stmmac_tx_queue *tx_q, + struct sk_buff *skb, u32 pay_len) +{ + int i, ndesc = 1; + + /* head payload */ + ndesc += DIV_ROUND_UP(pay_len, TSO_MAX_BUFF_SIZE); + /* frag payload */ + for (i = 0; i < skb_shinfo(skb)->nr_frags; i++) { + const skb_frag_t *frag = &skb_shinfo(skb)->frags[i]; + + ndesc += DIV_ROUND_UP(skb_frag_size(frag), + TSO_MAX_BUFF_SIZE); + } + /* MSS update requires a new descriptor */ + ndesc += !!(skb_shinfo(skb)->gso_size != tx_q->mss); + + return ndesc; +} + /** * stmmac_tso_xmit - Tx entry point of the driver for oversized frames (TSO) * @skb : the socket buffer @@ -4075,7 +4095,7 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev) u32 queue = skb_get_queue_mapping(skb); unsigned int first_entry, tx_packets; struct stmmac_txq_stats *txq_stats; - int tmp_pay_len = 0, first_tx; + int tmp_pay_len = 0, first_tx, ndesc; struct stmmac_tx_queue *tx_q; bool has_vlan, set_ic; dma_addr_t tso_des, des; @@ -4096,9 +4116,10 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev) hdr = tcp_hdrlen(skb); } - /* Desc availability based on threshold should be enough safe */ - if (unlikely(stmmac_tx_avail(priv, queue) < - (((skb->len - proto_hdr_len) / TSO_MAX_BUFF_SIZE + 1)))) { + pay_len = skb_headlen(skb) - proto_hdr_len; /* no frags */ + + ndesc = stmmac_tso_get_num_desc(tx_q, skb, pay_len); + if (unlikely(stmmac_tx_avail(priv, queue) < ndesc)) { if (!netif_tx_queue_stopped(netdev_get_tx_queue(dev, queue))) { netif_tx_stop_queue(netdev_get_tx_queue(priv->dev, queue)); @@ -4110,8 +4131,6 @@ static netdev_tx_t stmmac_tso_xmit(struct sk_buff *skb, struct net_device *dev) return NETDEV_TX_BUSY; } - pay_len = skb_headlen(skb) - proto_hdr_len; /* no frags */ - mss = skb_shinfo(skb)->gso_size; /* set new MSS value if needed */ @@ -4728,13 +4747,27 @@ static unsigned int stmmac_rx_buf2_len(struct stmmac_priv *priv, if (!priv->sph) return 0; - /* Not last descriptor */ - if (status & rx_not_ls) + /* For GMAC4, when split header is enabled, in some rare cases, the + * hardware does not fill buf2 of the first descriptor with payload. + * Thus we cannot assume buf2 is always fully filled if it is not + * the last descriptor. Otherwise, the length of buf2 of the second + * descriptor will be calculated wrong and cause an oops. + * + * If this is the last descriptor, 'plen' is the length of the + * received packet that was transferred to system memory. + * Otherwise, it is the accumulated number of bytes that have been + * transferred for the current packet. + * + * Thus 'plen - len' always gives the correct length of buf2. + */ + + /* Not GMAC4 and not last descriptor */ + if (!priv->plat->has_gmac4 && (status & rx_not_ls)) return priv->dma_conf.dma_buf_sz; + /* GMAC4 or last descriptor */ plen = stmmac_get_rx_frame_len(priv, p, coe); - /* Last descriptor */ return plen - len; } diff --git a/drivers/net/ethernet/sun/sunvnet_common.c b/drivers/net/ethernet/sun/sunvnet_common.c index 3525d5c0d694..351609f4f011 100644 --- a/drivers/net/ethernet/sun/sunvnet_common.c +++ b/drivers/net/ethernet/sun/sunvnet_common.c @@ -1144,9 +1144,9 @@ static inline struct sk_buff *vnet_skb_shape(struct sk_buff *skb, int ncookies) nskb->protocol = skb->protocol; offset = skb_mac_header(skb) - skb->data; skb_set_mac_header(nskb, offset); - offset = skb_network_header(skb) - skb->data; + offset = skb_network_offset(skb); skb_set_network_header(nskb, offset); - offset = skb_transport_header(skb) - skb->data; + offset = skb_transport_offset(skb); skb_set_transport_header(nskb, offset); offset = 0; diff --git a/drivers/net/ethernet/ti/am65-cpsw-nuss.c b/drivers/net/ethernet/ti/am65-cpsw-nuss.c index 8e7bcb3e4225..4c069bad313c 100644 --- a/drivers/net/ethernet/ti/am65-cpsw-nuss.c +++ b/drivers/net/ethernet/ti/am65-cpsw-nuss.c @@ -520,7 +520,7 @@ static int am65_cpsw_nuss_common_stop(struct am65_cpsw_common *common) for (i = 0; i < AM65_CPSW_MAX_RX_FLOWS; i++) k3_udma_glue_reset_rx_chn(common->rx_chns.rx_chn, i, &common->rx_chns, - am65_cpsw_nuss_rx_cleanup, !!i); + am65_cpsw_nuss_rx_cleanup); k3_udma_glue_disable_rx_chn(common->rx_chns.rx_chn); @@ -2757,7 +2757,7 @@ static int am65_cpsw_nuss_register_ndevs(struct am65_cpsw_common *common) for (i = 0; i < AM65_CPSW_MAX_RX_FLOWS; i++) k3_udma_glue_reset_rx_chn(rx_chan->rx_chn, i, rx_chan, - am65_cpsw_nuss_rx_cleanup, !!i); + am65_cpsw_nuss_rx_cleanup); k3_udma_glue_disable_rx_chn(rx_chan->rx_chn); diff --git a/drivers/net/ethernet/ti/icssg/icssg_prueth.c b/drivers/net/ethernet/ti/icssg/icssg_prueth.c index 7efb3e347c04..1fe5d4be7d8b 100644 --- a/drivers/net/ethernet/ti/icssg/icssg_prueth.c +++ b/drivers/net/ethernet/ti/icssg/icssg_prueth.c @@ -1131,7 +1131,7 @@ static void prueth_reset_rx_chan(struct prueth_rx_chn *chn, for (i = 0; i < num_flows; i++) k3_udma_glue_reset_rx_chn(chn->rx_chn, i, chn, - prueth_rx_cleanup, !!i); + prueth_rx_cleanup); if (disable) k3_udma_glue_disable_rx_chn(chn->rx_chn); } diff --git a/drivers/net/ethernet/wangxun/libwx/wx_lib.c b/drivers/net/ethernet/wangxun/libwx/wx_lib.c index c27e68e320e6..71c64cdb5ddb 100644 --- a/drivers/net/ethernet/wangxun/libwx/wx_lib.c +++ b/drivers/net/ethernet/wangxun/libwx/wx_lib.c @@ -1247,7 +1247,7 @@ static int wx_tso(struct wx_ring *tx_ring, struct wx_tx_buffer *first, /* compute header lengths */ l4len = enc ? inner_tcp_hdrlen(skb) : tcp_hdrlen(skb); - *hdr_len = enc ? (skb_inner_transport_header(skb) - skb->data) : + *hdr_len = enc ? skb_inner_transport_offset(skb) : skb_transport_offset(skb); *hdr_len += l4len; diff --git a/drivers/net/ieee802154/cc2520.c b/drivers/net/ieee802154/cc2520.c index a94d8dd71aad..10bdc317ae28 100644 --- a/drivers/net/ieee802154/cc2520.c +++ b/drivers/net/ieee802154/cc2520.c @@ -206,6 +206,7 @@ struct cc2520_private { bool is_tx; /* Flag for sync b/w Tx and Rx */ bool amplified; /* Flag for CC2591 */ struct gpio_desc *fifo_pin; /* FIFO GPIO pin number */ + int fifop_irq; /* FIFOP IRQ */ struct work_struct fifop_irqwork;/* Workqueue for FIFOP */ spinlock_t lock; /* Lock for is_tx*/ struct completion tx_complete; /* Work completion for Tx */ @@ -1117,8 +1118,9 @@ static int cc2520_probe(struct spi_device *spi) goto err_hw_init; /* Set up fifop interrupt */ + priv->fifop_irq = gpiod_to_irq(fifop); ret = devm_request_irq(&spi->dev, - gpiod_to_irq(fifop), + priv->fifop_irq, cc2520_fifop_isr, IRQF_TRIGGER_RISING, dev_name(&spi->dev), @@ -1156,11 +1158,11 @@ static void cc2520_remove(struct spi_device *spi) { struct cc2520_private *priv = spi_get_drvdata(spi); - mutex_destroy(&priv->buffer_mutex); - flush_work(&priv->fifop_irqwork); - + disable_irq(priv->fifop_irq); + cancel_work_sync(&priv->fifop_irqwork); ieee802154_unregister_hw(priv->hw); ieee802154_free_hw(priv->hw); + mutex_destroy(&priv->buffer_mutex); } static const struct spi_device_id cc2520_ids[] = { diff --git a/drivers/net/netconsole.c b/drivers/net/netconsole.c index fffffa3658d2..69d54f58aa97 100644 --- a/drivers/net/netconsole.c +++ b/drivers/net/netconsole.c @@ -854,7 +854,8 @@ static void send_ext_msg_udp(struct netconsole_target *nt, const char *msg, if (msg_len + release_len <= MAX_PRINT_CHUNK) { /* No fragmentation needed */ if (nt->release) { - scnprintf(buf, MAX_PRINT_CHUNK, "%s,%s", release, msg); + scnprintf(buf, MAX_PRINT_CHUNK, "%s,%.*s", release, + msg_len, msg); msg_len += release_len; msg_ready = buf; } diff --git a/drivers/net/phy/sfp.c b/drivers/net/phy/sfp.c index e352ceb2f7cb..38ab35e57127 100644 --- a/drivers/net/phy/sfp.c +++ b/drivers/net/phy/sfp.c @@ -514,6 +514,22 @@ static const struct sfp_quirk sfp_quirks[] = { SFP_QUIRK("HUAWEI", "MA5671A", sfp_quirk_2500basex, sfp_fixup_ignore_tx_fault_and_los), + // Hisense LXT-010S-H is a GPON ONT SFP (sold as LEOX LXT-010S-H) that + // can operate at 2500base-X, but reports 1000BASE-LX / 1300MBd in its + // EEPROM + SFP_QUIRK("Hisense-Leox", "LXT-010S-H", sfp_quirk_2500basex, + sfp_fixup_ignore_tx_fault), + + // Hisense ZNID-GPON-2311NA can operate at 2500base-X, but reports + // 1000BASE-LX / 1300MBd in its EEPROM + SFP_QUIRK("Hisense", "ZNID-GPON-2311NA", sfp_quirk_2500basex, + sfp_fixup_ignore_tx_fault), + + // HSGQ HSGQ-XPON-Stick can operate at 2500base-X, but reports + // 1000BASE-LX / 1300MBd in its EEPROM + SFP_QUIRK("HSGQ", "HSGQ-XPON-Stick", sfp_quirk_2500basex, + sfp_fixup_ignore_tx_fault), + // FS 2.5G Base-T SFP_QUIRK_M("FS", "SFP-2.5G-T", sfp_quirk_oem_2_5g), diff --git a/drivers/net/tap.c b/drivers/net/tap.c index 5f01c875e49e..9ce26a60dac3 100644 --- a/drivers/net/tap.c +++ b/drivers/net/tap.c @@ -645,6 +645,7 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control, int err; struct virtio_net_hdr vnet_hdr = { 0 }; int vnet_hdr_len = 0; + int hdr_len = 0; int copylen = 0; int depth; bool zerocopy = false; @@ -663,13 +664,13 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control, if (!copy_from_iter_full(&vnet_hdr, sizeof(vnet_hdr), from)) goto err; iov_iter_advance(from, vnet_hdr_len - sizeof(vnet_hdr)); - if ((vnet_hdr.flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) && - tap16_to_cpu(q, vnet_hdr.csum_start) + - tap16_to_cpu(q, vnet_hdr.csum_offset) + 2 > - tap16_to_cpu(q, vnet_hdr.hdr_len)) - vnet_hdr.hdr_len = cpu_to_tap16(q, - tap16_to_cpu(q, vnet_hdr.csum_start) + - tap16_to_cpu(q, vnet_hdr.csum_offset) + 2); + hdr_len = tap16_to_cpu(q, vnet_hdr.hdr_len); + if (vnet_hdr.flags & VIRTIO_NET_HDR_F_NEEDS_CSUM) { + hdr_len = max(tap16_to_cpu(q, vnet_hdr.csum_start) + + tap16_to_cpu(q, vnet_hdr.csum_offset) + 2, + hdr_len); + vnet_hdr.hdr_len = cpu_to_tap16(q, hdr_len); + } err = -EINVAL; if (tap16_to_cpu(q, vnet_hdr.hdr_len) > len) goto err; @@ -682,12 +683,7 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control, if (msg_control && sock_flag(&q->sk, SOCK_ZEROCOPY)) { struct iov_iter i; - copylen = vnet_hdr.hdr_len ? - tap16_to_cpu(q, vnet_hdr.hdr_len) : GOODCOPY_LEN; - if (copylen > good_linear) - copylen = good_linear; - else if (copylen < ETH_HLEN) - copylen = ETH_HLEN; + copylen = clamp(hdr_len ?: GOODCOPY_LEN, ETH_HLEN, good_linear); linear = copylen; i = *from; iov_iter_advance(&i, copylen); @@ -697,11 +693,7 @@ static ssize_t tap_get_user(struct tap_queue *q, void *msg_control, if (!zerocopy) { copylen = len; - linear = tap16_to_cpu(q, vnet_hdr.hdr_len); - if (linear > good_linear) - linear = good_linear; - else if (linear < ETH_HLEN) - linear = ETH_HLEN; + linear = clamp(hdr_len, ETH_HLEN, good_linear); } skb = tap_alloc_skb(&q->sk, TAP_RESERVE, copylen, @@ -1015,7 +1007,7 @@ static long tap_ioctl(struct file *file, unsigned int cmd, unsigned int __user *up = argp; unsigned short u; int __user *sp = argp; - struct sockaddr sa; + struct sockaddr sa = {}; int s; int ret; @@ -1200,26 +1192,31 @@ static int tap_get_user_xdp(struct tap_queue *q, struct xdp_buff *xdp) skb_reset_mac_header(skb); skb->protocol = eth_hdr(skb)->h_proto; + rcu_read_lock(); + tap = rcu_dereference(q->tap); + if (!tap) { + kfree_skb(skb); + rcu_read_unlock(); + return 0; + } + skb->dev = tap->dev; + if (vnet_hdr_len) { err = virtio_net_hdr_to_skb(skb, gso, tap_is_little_endian(q)); - if (err) + if (err) { + rcu_read_unlock(); goto err_kfree; + } } + skb_probe_transport_header(skb); + /* Move network header to the right position for VLAN tagged packets */ if (eth_type_vlan(skb->protocol) && vlan_get_protocol_and_depth(skb, skb->protocol, &depth) != 0) skb_set_network_header(skb, depth); - rcu_read_lock(); - tap = rcu_dereference(q->tap); - if (tap) { - skb->dev = tap->dev; - skb_probe_transport_header(skb); - dev_queue_xmit(skb); - } else { - kfree_skb(skb); - } + dev_queue_xmit(skb); rcu_read_unlock(); return 0; diff --git a/drivers/net/team/team_core.c b/drivers/net/team/team_core.c index a4bf020cf8e7..91abda19a3be 100644 --- a/drivers/net/team/team_core.c +++ b/drivers/net/team/team_core.c @@ -533,21 +533,23 @@ static void team_adjust_ops(struct team *team) if (!team->en_port_count || !team_is_mode_set(team) || !team->mode->ops->transmit) - team->ops.transmit = team_dummy_transmit; + WRITE_ONCE(team->ops.transmit, team_dummy_transmit); else - team->ops.transmit = team->mode->ops->transmit; + WRITE_ONCE(team->ops.transmit, team->mode->ops->transmit); if (!team->en_port_count || !team_is_mode_set(team) || !team->mode->ops->receive) - team->ops.receive = team_dummy_receive; + WRITE_ONCE(team->ops.receive, team_dummy_receive); else - team->ops.receive = team->mode->ops->receive; + WRITE_ONCE(team->ops.receive, team->mode->ops->receive); } /* - * We can benefit from the fact that it's ensured no port is present - * at the time of mode change. Therefore no packets are in fly so there's no - * need to set mode operations in any special way. + * team_change_mode() ensures no ports are present during mode change, + * but lockless readers can still reach team_xmit(). Avoid touching + * transmit/receive -- they are already set to dummies by + * team_adjust_ops() since no ports are enabled. synchronize_net() + * drains in-flight readers before destroying old mode state. */ static int __team_change_mode(struct team *team, const struct team_mode *new_mode) @@ -556,9 +558,21 @@ static int __team_change_mode(struct team *team, if (team_is_mode_set(team)) { void (*exit_op)(struct team *team) = team->ops.exit; - /* Clear ops area so no callback is called any longer */ - memset(&team->ops, 0, sizeof(struct team_mode_ops)); - team_adjust_ops(team); + /* Clear cold-path ops used only under RTNL. transmit and + * receive are already dummies (no ports) so leave them + * alone -- overwriting them is the source of the race. + */ + team->ops.init = NULL; + team->ops.exit = NULL; + team->ops.port_enter = NULL; + team->ops.port_leave = NULL; + team->ops.port_change_dev_addr = NULL; + team->ops.port_tx_disabled = NULL; + + /* Wait for in-flight readers before tearing down mode + * state they may reference. + */ + synchronize_net(); if (exit_op) exit_op(team); @@ -581,7 +595,12 @@ static int __team_change_mode(struct team *team, } team->mode = new_mode; - memcpy(&team->ops, new_mode->ops, sizeof(struct team_mode_ops)); + team->ops.init = new_mode->ops->init; + team->ops.exit = new_mode->ops->exit; + team->ops.port_enter = new_mode->ops->port_enter; + team->ops.port_leave = new_mode->ops->port_leave; + team->ops.port_change_dev_addr = new_mode->ops->port_change_dev_addr; + team->ops.port_tx_disabled = new_mode->ops->port_tx_disabled; team_adjust_ops(team); return 0; @@ -742,7 +761,7 @@ static rx_handler_result_t team_handle_frame(struct sk_buff **pskb) /* allow exact match delivery for disabled ports */ res = RX_HANDLER_EXACT; } else { - res = team->ops.receive(team, port, skb); + res = READ_ONCE(team->ops.receive)(team, port, skb); } if (res == RX_HANDLER_ANOTHER) { struct team_pcpu_stats *pcpu_stats; @@ -801,7 +820,8 @@ static struct list_head *__team_get_qom_list(struct team *team, u16 queue_id) /* * note: already called with rcu_read_lock */ -static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb) +static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb, + bool *tx_success) { struct list_head *qom_list; struct team_port *port; @@ -809,11 +829,11 @@ static bool team_queue_override_transmit(struct team *team, struct sk_buff *skb) if (!team->queue_override_enabled || !skb->queue_mapping) return false; qom_list = __team_get_qom_list(team, skb->queue_mapping); - list_for_each_entry_rcu(port, qom_list, qom_list) { - if (!team_dev_queue_xmit(team, port, skb)) - return true; - } - return false; + port = list_first_or_null_rcu(qom_list, struct team_port, qom_list); + if (!port) + return false; + *tx_success = !team_dev_queue_xmit(team, port, skb); + return true; } static void __team_queue_override_port_del(struct team *team, @@ -943,8 +963,6 @@ static void team_port_enable(struct team *team, team_port_index_hash(team, port->index)); team_adjust_ops(team); team_queue_override_port_add(team, port); - if (team->ops.port_enabled) - team->ops.port_enabled(team, port); team_notify_peers(team); team_mcast_rejoin(team); team_lower_state_changed(port); @@ -969,8 +987,8 @@ static void team_port_disable(struct team *team, { if (!team_port_enabled(port)) return; - if (team->ops.port_disabled) - team->ops.port_disabled(team, port); + if (team->ops.port_tx_disabled) + team->ops.port_tx_disabled(team, port); hlist_del_rcu(&port->hlist); __reconstruct_port_hlist(team, port->index); port->index = -1; @@ -1341,7 +1359,7 @@ static int team_port_add(struct team *team, struct net_device *port_dev, static void __team_port_change_port_removed(struct team_port *port); -static int team_port_del(struct team *team, struct net_device *port_dev) +static int team_port_del(struct team *team, struct net_device *port_dev, bool unregister) { struct net_device *dev = team->dev; struct team_port *port; @@ -1379,7 +1397,10 @@ static int team_port_del(struct team *team, struct net_device *port_dev) __team_port_change_port_removed(port); team_port_set_orig_dev_addr(port); - dev_set_mtu(port_dev, port->orig.mtu); + if (unregister) + __dev_set_mtu(port_dev, port->orig.mtu); + else + dev_set_mtu(port_dev, port->orig.mtu); kfree_rcu(port, rcu); netdev_info(dev, "Port device %s removed\n", portname); __team_compute_features(team); @@ -1684,7 +1705,7 @@ static void team_uninit(struct net_device *dev) mutex_lock(&team->lock); list_for_each_entry_safe(port, tmp, &team->port_list, list) - team_port_del(team, port->dev); + team_port_del(team, port->dev, false); __team_change_mode(team, NULL); /* cleanup */ __team_options_unregister(team, team_options, ARRAY_SIZE(team_options)); @@ -1730,9 +1751,8 @@ static netdev_tx_t team_xmit(struct sk_buff *skb, struct net_device *dev) bool tx_success; unsigned int len = skb->len; - tx_success = team_queue_override_transmit(team, skb); - if (!tx_success) - tx_success = team->ops.transmit(team, skb); + if (!team_queue_override_transmit(team, skb, &tx_success)) + tx_success = READ_ONCE(team->ops.transmit)(team, skb); if (tx_success) { struct team_pcpu_stats *pcpu_stats; @@ -2002,7 +2022,29 @@ static int team_del_slave(struct net_device *dev, struct net_device *port_dev) int err; mutex_lock(&team->lock); - err = team_port_del(team, port_dev); + err = team_port_del(team, port_dev, false); + mutex_unlock(&team->lock); + + if (err) + return err; + + if (netif_is_team_master(port_dev)) { + lockdep_unregister_key(&team->team_lock_key); + lockdep_register_key(&team->team_lock_key); + lockdep_set_class(&team->lock, &team->team_lock_key); + } + netdev_change_features(dev); + + return err; +} + +static int team_del_slave_on_unregister(struct net_device *dev, struct net_device *port_dev) +{ + struct team *team = netdev_priv(dev); + int err; + + mutex_lock(&team->lock); + err = team_port_del(team, port_dev, true); mutex_unlock(&team->lock); if (err) @@ -3107,7 +3149,7 @@ static int team_device_event(struct notifier_block *unused, !!netif_oper_up(port->dev)); break; case NETDEV_UNREGISTER: - team_del_slave(port->team->dev, dev); + team_del_slave_on_unregister(port->team->dev, dev); break; case NETDEV_FEAT_CHANGE: if (!port->team->notifier_ctx) { diff --git a/drivers/net/team/team_mode_loadbalance.c b/drivers/net/team/team_mode_loadbalance.c index 00f8989c29c0..3ca419884185 100644 --- a/drivers/net/team/team_mode_loadbalance.c +++ b/drivers/net/team/team_mode_loadbalance.c @@ -658,7 +658,7 @@ static void lb_port_leave(struct team *team, struct team_port *port) free_percpu(lb_port_priv->pcpu_stats); } -static void lb_port_disabled(struct team *team, struct team_port *port) +static void lb_port_tx_disabled(struct team *team, struct team_port *port) { lb_tx_hash_to_port_mapping_null_port(team, port); } @@ -668,7 +668,7 @@ static const struct team_mode_ops lb_mode_ops = { .exit = lb_exit, .port_enter = lb_port_enter, .port_leave = lb_port_leave, - .port_disabled = lb_port_disabled, + .port_tx_disabled = lb_port_tx_disabled, .receive = lb_receive, .transmit = lb_transmit, }; diff --git a/drivers/net/usb/ax88179_178a.c b/drivers/net/usb/ax88179_178a.c index 619093d33373..63cf49bd3d03 100644 --- a/drivers/net/usb/ax88179_178a.c +++ b/drivers/net/usb/ax88179_178a.c @@ -1461,11 +1461,11 @@ static int ax88179_rx_fixup(struct usbnet *dev, struct sk_buff *skb) return 1; } - ax_skb = netdev_alloc_skb_ip_align(dev->net, pkt_len); + ax_skb = netdev_alloc_skb_ip_align(dev->net, pkt_len - 2); if (!ax_skb) return 0; - skb_put(ax_skb, pkt_len); - memcpy(ax_skb->data, skb->data + 2, pkt_len); + skb_put(ax_skb, pkt_len - 2); + memcpy(ax_skb->data, skb->data + 2, pkt_len - 2); ax88179_rx_checksum(ax_skb, pkt_hdr); usbnet_skb_return(dev, ax_skb); diff --git a/drivers/net/usb/qmi_wwan.c b/drivers/net/usb/qmi_wwan.c index ad8ed2816a98..95f3b3fa3aec 100644 --- a/drivers/net/usb/qmi_wwan.c +++ b/drivers/net/usb/qmi_wwan.c @@ -1093,6 +1093,7 @@ static const struct usb_device_id products[] = { {QMI_MATCH_FF_FF_FF(0x2c7c, 0x0122)}, /* Quectel RG650V */ {QMI_MATCH_FF_FF_FF(0x2c7c, 0x0125)}, /* Quectel EC25, EC20 R2.0 Mini PCIe */ {QMI_MATCH_FF_FF_FF(0x2c7c, 0x0306)}, /* Quectel EP06/EG06/EM06 */ + {QMI_MATCH_FF_FF_FF(0x2c7c, 0x030b)}, /* Quectel EM060K/EG120K-EA */ {QMI_MATCH_FF_FF_FF(0x2c7c, 0x0512)}, /* Quectel EG12/EM12 */ {QMI_MATCH_FF_FF_FF(0x2c7c, 0x0620)}, /* Quectel EM160R-GL */ {QMI_MATCH_FF_FF_FF(0x2c7c, 0x0800)}, /* Quectel RM500Q-GL */ diff --git a/drivers/net/veth.c b/drivers/net/veth.c index e5daca818c70..334783a02219 100644 --- a/drivers/net/veth.c +++ b/drivers/net/veth.c @@ -1436,6 +1436,7 @@ static int veth_open(struct net_device *dev) } veth_set_xdp_features(dev); + veth_set_xdp_features(peer); return 0; } diff --git a/drivers/net/vxlan/vxlan_core.c b/drivers/net/vxlan/vxlan_core.c index 04847f0c6c14..355337d428e5 100644 --- a/drivers/net/vxlan/vxlan_core.c +++ b/drivers/net/vxlan/vxlan_core.c @@ -1475,6 +1475,10 @@ static bool vxlan_snoop(struct net_device *dev, if (likely(f)) { struct vxlan_rdst *rdst = first_remote_rcu(f); + /* Don't override an fdb with nexthop with a learnt entry */ + if (rcu_access_pointer(f->nh)) + return true; + if (likely(vxlan_addr_equal(&rdst->remote_ip, src_ip) && rdst->remote_ifindex == ifindex)) return false; @@ -1483,10 +1487,6 @@ static bool vxlan_snoop(struct net_device *dev, if (f->state & (NUD_PERMANENT | NUD_NOARP)) return true; - /* Don't override an fdb with nexthop with a learnt entry */ - if (rcu_access_pointer(f->nh)) - return true; - if (net_ratelimit()) netdev_info(dev, "%pM migrated from %pIS to %pIS\n", @@ -1898,6 +1898,8 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) n = neigh_lookup(&arp_tbl, &tip, dev); if (n) { + struct vxlan_rdst *rdst = NULL; + u8 ha[ETH_ALEN] __aligned(2); struct vxlan_fdb *f; struct sk_buff *reply; @@ -1906,15 +1908,19 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) goto out; } - f = vxlan_find_mac(vxlan, n->ha, vni); - if (f && vxlan_addr_any(&(first_remote_rcu(f)->remote_ip))) { + neigh_ha_snapshot(ha, n, n->dev); + + f = vxlan_find_mac(vxlan, ha, vni); + if (f) + rdst = first_remote_rcu(f); + if (rdst && vxlan_addr_any(&rdst->remote_ip)) { /* bridge-local neighbor */ neigh_release(n); goto out; } reply = arp_create(ARPOP_REPLY, ETH_P_ARP, sip, dev, tip, sha, - n->ha, sha); + ha, sha); neigh_release(n); @@ -1947,7 +1953,8 @@ static int arp_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) #if IS_ENABLED(CONFIG_IPV6) static struct sk_buff *vxlan_na_create(struct sk_buff *request, - struct neighbour *n, bool isrouter) + struct neighbour *n, u8 *ha, + bool isrouter) { struct net_device *dev = request->dev; struct sk_buff *reply; @@ -1995,7 +2002,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request, /* Ethernet header */ ether_addr_copy(eth_hdr(reply)->h_dest, daddr); - ether_addr_copy(eth_hdr(reply)->h_source, n->ha); + ether_addr_copy(eth_hdr(reply)->h_source, ha); eth_hdr(reply)->h_proto = htons(ETH_P_IPV6); reply->protocol = htons(ETH_P_IPV6); @@ -2024,7 +2031,7 @@ static struct sk_buff *vxlan_na_create(struct sk_buff *request, na->icmph.icmp6_override = 1; na->icmph.icmp6_solicited = 1; na->target = ns->target; - ether_addr_copy(&na->opt[2], n->ha); + ether_addr_copy(&na->opt[2], ha); na->opt[0] = ND_OPT_TARGET_LL_ADDR; na->opt[1] = na_olen >> 3; @@ -2066,6 +2073,8 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) n = neigh_lookup(ipv6_stub->nd_tbl, &msg->target, dev); if (n) { + struct vxlan_rdst *rdst = NULL; + u8 ha[ETH_ALEN] __aligned(2); struct vxlan_fdb *f; struct sk_buff *reply; @@ -2074,14 +2083,17 @@ static int neigh_reduce(struct net_device *dev, struct sk_buff *skb, __be32 vni) goto out; } - f = vxlan_find_mac(vxlan, n->ha, vni); - if (f && vxlan_addr_any(&(first_remote_rcu(f)->remote_ip))) { + neigh_ha_snapshot(ha, n, n->dev); + f = vxlan_find_mac(vxlan, ha, vni); + if (f) + rdst = first_remote_rcu(f); + if (rdst && vxlan_addr_any(&rdst->remote_ip)) { /* bridge-local neighbor */ neigh_release(n); goto out; } - reply = vxlan_na_create(skb, n, + reply = vxlan_na_create(skb, n, ha, !!(f ? f->flags & NTF_ROUTER : 0)); neigh_release(n); @@ -2475,7 +2487,7 @@ void vxlan_xmit_one(struct sk_buff *skb, struct net_device *dev, const struct iphdr *old_iph = ip_hdr(skb); union vxlan_addr *dst; union vxlan_addr remote_ip, local_ip; - struct vxlan_metadata _md; + struct vxlan_metadata _md = {}; struct vxlan_metadata *md = &_md; unsigned int pkt_len = skb->len; __be16 src_port = 0, dst_port; diff --git a/drivers/net/vxlan/vxlan_private.h b/drivers/net/vxlan/vxlan_private.h index 76a351a997d5..c6279ef98a5c 100644 --- a/drivers/net/vxlan/vxlan_private.h +++ b/drivers/net/vxlan/vxlan_private.h @@ -56,9 +56,7 @@ static inline struct hlist_head *vs_head(struct net *net, __be16 port) return &vn->sock_list[hash_32(ntohs(port), PORT_HASH_BITS)]; } -/* First remote destination for a forwarding entry. - * Guaranteed to be non-NULL because remotes are never deleted. - */ +/* First remote destination for a forwarding entry. */ static inline struct vxlan_rdst *first_remote_rcu(struct vxlan_fdb *fdb) { if (rcu_access_pointer(fdb->nh)) diff --git a/drivers/net/wireguard/noise.c b/drivers/net/wireguard/noise.c index 720952b92e78..bccaaca66e67 100644 --- a/drivers/net/wireguard/noise.c +++ b/drivers/net/wireguard/noise.c @@ -784,10 +784,11 @@ wg_noise_handshake_consume_response(struct message_handshake_response *src, /* Success! Copy everything to peer */ down_write(&handshake->lock); - /* It's important to check that the state is still the same, while we - * have an exclusive lock. + /* Check that the state is the same and that this is still the + * initiation we started with, while we have an exclusive lock. */ - if (handshake->state != state) { + if (handshake->state != state || + crypto_memneq(handshake->ephemeral_private, ephemeral_private, NOISE_PUBLIC_KEY_LEN)) { up_write(&handshake->lock); goto fail; } diff --git a/drivers/net/wireguard/queueing.h b/drivers/net/wireguard/queueing.h index 7eb76724b3ed..9fd19e98fcf8 100644 --- a/drivers/net/wireguard/queueing.h +++ b/drivers/net/wireguard/queueing.h @@ -78,12 +78,14 @@ static inline void wg_reset_packet(struct sk_buff *skb, bool encapsulating) u8 l4_hash = skb->l4_hash; u8 sw_hash = skb->sw_hash; u32 hash = skb->hash; + u8 tstamp_type = skb->tstamp_type; skb_scrub_packet(skb, true); memset(&skb->headers, 0, sizeof(skb->headers)); if (encapsulating) { skb->l4_hash = l4_hash; skb->sw_hash = sw_hash; skb->hash = hash; + skb->tstamp_type = tstamp_type; } skb->queue_mapping = 0; skb->nohdr = 0; diff --git a/drivers/net/wireguard/receive.c b/drivers/net/wireguard/receive.c index db01ec03bda0..eb8851113654 100644 --- a/drivers/net/wireguard/receive.c +++ b/drivers/net/wireguard/receive.c @@ -263,7 +263,7 @@ static bool decrypt_packet(struct sk_buff *skb, struct noise_keypair *keypair) * call skb_cow_data, so that there's no chance that data is removed * from the skb, so that later we can extract the original endpoint. */ - offset = skb->data - skb_network_header(skb); + offset = -skb_network_offset(skb); skb_push(skb, offset); num_frags = skb_cow_data(skb, 0, &trailer); offset += sizeof(struct message_data); diff --git a/drivers/net/wireless/ath/ath11k/core.c b/drivers/net/wireless/ath/ath11k/core.c index e3f730e0dd58..1a87e71625f4 100644 --- a/drivers/net/wireless/ath/ath11k/core.c +++ b/drivers/net/wireless/ath/ath11k/core.c @@ -62,7 +62,7 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .ce_ie_addr = &ath11k_ce_ie_addr_ipq8074, .single_pdev_only = false, .rxdma1_enable = true, - .num_rxmda_per_pdev = 1, + .num_rxdma_per_pdev = 1, .rx_mac_buf_ring = false, .vdev_start_delay = false, .htt_peer_map_v2 = true, @@ -147,7 +147,7 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .ce_ie_addr = &ath11k_ce_ie_addr_ipq8074, .single_pdev_only = false, .rxdma1_enable = true, - .num_rxmda_per_pdev = 1, + .num_rxdma_per_pdev = 1, .rx_mac_buf_ring = false, .vdev_start_delay = false, .htt_peer_map_v2 = true, @@ -230,7 +230,7 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .ce_ie_addr = &ath11k_ce_ie_addr_ipq8074, .single_pdev_only = true, .rxdma1_enable = false, - .num_rxmda_per_pdev = 2, + .num_rxdma_per_pdev = 2, .rx_mac_buf_ring = true, .vdev_start_delay = true, .htt_peer_map_v2 = false, @@ -314,7 +314,7 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .svc_to_ce_map_len = 18, .ce_ie_addr = &ath11k_ce_ie_addr_ipq8074, .rxdma1_enable = true, - .num_rxmda_per_pdev = 1, + .num_rxdma_per_pdev = 1, .rx_mac_buf_ring = false, .vdev_start_delay = false, .htt_peer_map_v2 = true, @@ -397,7 +397,7 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .ce_ie_addr = &ath11k_ce_ie_addr_ipq8074, .single_pdev_only = true, .rxdma1_enable = false, - .num_rxmda_per_pdev = 2, + .num_rxdma_per_pdev = 2, .rx_mac_buf_ring = true, .vdev_start_delay = true, .htt_peer_map_v2 = false, @@ -481,7 +481,7 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .svc_to_ce_map_len = 14, .single_pdev_only = true, .rxdma1_enable = false, - .num_rxmda_per_pdev = 2, + .num_rxdma_per_pdev = 2, .rx_mac_buf_ring = true, .vdev_start_delay = true, .htt_peer_map_v2 = false, @@ -565,7 +565,7 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .ce_ie_addr = &ath11k_ce_ie_addr_ipq8074, .single_pdev_only = true, .rxdma1_enable = false, - .num_rxmda_per_pdev = 1, + .num_rxdma_per_pdev = 1, .rx_mac_buf_ring = true, .vdev_start_delay = true, .htt_peer_map_v2 = false, @@ -657,7 +657,7 @@ static const struct ath11k_hw_params ath11k_hw_params[] = { .ce_ie_addr = &ath11k_ce_ie_addr_ipq5018, .ce_remap = &ath11k_ce_remap_ipq5018, .rxdma1_enable = true, - .num_rxmda_per_pdev = RXDMA_PER_PDEV_5018, + .num_rxdma_per_pdev = RXDMA_PER_PDEV_5018, .rx_mac_buf_ring = false, .vdev_start_delay = false, .htt_peer_map_v2 = true, @@ -1705,7 +1705,7 @@ static int ath11k_core_start(struct ath11k_base *ab) } /* put hardware to DBS mode */ - if (ab->hw_params.single_pdev_only && ab->hw_params.num_rxmda_per_pdev > 1) { + if (ab->hw_params.single_pdev_only && ab->hw_params.num_rxdma_per_pdev > 1) { ret = ath11k_wmi_set_hw_mode(ab, WMI_HOST_HW_MODE_DBS); if (ret) { ath11k_err(ab, "failed to send dbs mode: %d\n", ret); diff --git a/drivers/net/wireless/ath/ath11k/debugfs.c b/drivers/net/wireless/ath/ath11k/debugfs.c index 4304fed44d58..f7cb94e8658d 100644 --- a/drivers/net/wireless/ath/ath11k/debugfs.c +++ b/drivers/net/wireless/ath/ath11k/debugfs.c @@ -660,7 +660,7 @@ static ssize_t ath11k_write_extd_rx_stats(struct file *file, ar->debug.rx_filter = tlv_filter.rx_filter; - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ring_id = ar->dp.rx_mon_status_refill_ring[i].refill_buf_ring.ring_id; ret = ath11k_dp_tx_htt_rx_filter_setup(ar->ab, ring_id, ar->dp.mac_id, HAL_RXDMA_MONITOR_STATUS, @@ -1104,7 +1104,7 @@ static ssize_t ath11k_write_pktlog_filter(struct file *file, } /* Clear rx filter set for monitor mode and rx status */ - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ring_id = ar->dp.rx_mon_status_refill_ring[i].refill_buf_ring.ring_id; ret = ath11k_dp_tx_htt_rx_filter_setup(ar->ab, ring_id, ar->dp.mac_id, HAL_RXDMA_MONITOR_STATUS, @@ -1163,7 +1163,7 @@ static ssize_t ath11k_write_pktlog_filter(struct file *file, HTT_RX_FP_DATA_FILTER_FLASG3; } - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ring_id = ar->dp.rx_mon_status_refill_ring[i].refill_buf_ring.ring_id; ret = ath11k_dp_tx_htt_rx_filter_setup(ab, ring_id, ar->dp.mac_id + i, diff --git a/drivers/net/wireless/ath/ath11k/dp.c b/drivers/net/wireless/ath/ath11k/dp.c index 15fb15f58042..85af58c17e45 100644 --- a/drivers/net/wireless/ath/ath11k/dp.c +++ b/drivers/net/wireless/ath/ath11k/dp.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: BSD-3-Clause-Clear /* * Copyright (c) 2018-2019 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2023 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) 2021-2024 Qualcomm Innovation Center, Inc. All rights reserved. */ #include @@ -816,8 +816,8 @@ int ath11k_dp_service_srng(struct ath11k_base *ab, if (ab->hw_params.ring_mask->rx_mon_status[grp_id]) { for (i = 0; i < ab->num_radios; i++) { - for (j = 0; j < ab->hw_params.num_rxmda_per_pdev; j++) { - int id = i * ab->hw_params.num_rxmda_per_pdev + j; + for (j = 0; j < ab->hw_params.num_rxdma_per_pdev; j++) { + int id = i * ab->hw_params.num_rxdma_per_pdev + j; if (ab->hw_params.ring_mask->rx_mon_status[grp_id] & BIT(id)) { @@ -839,8 +839,8 @@ int ath11k_dp_service_srng(struct ath11k_base *ab, ath11k_dp_process_reo_status(ab); for (i = 0; i < ab->num_radios; i++) { - for (j = 0; j < ab->hw_params.num_rxmda_per_pdev; j++) { - int id = i * ab->hw_params.num_rxmda_per_pdev + j; + for (j = 0; j < ab->hw_params.num_rxdma_per_pdev; j++) { + int id = i * ab->hw_params.num_rxdma_per_pdev + j; if (ab->hw_params.ring_mask->rxdma2host[grp_id] & BIT(id)) { work_done = ath11k_dp_process_rxdma_err(ab, id, budget); @@ -899,7 +899,7 @@ void ath11k_dp_pdev_pre_alloc(struct ath11k_base *ab) spin_lock_init(&dp->rx_refill_buf_ring.idr_lock); atomic_set(&dp->num_tx_pending, 0); init_waitqueue_head(&dp->tx_empty_waitq); - for (j = 0; j < ab->hw_params.num_rxmda_per_pdev; j++) { + for (j = 0; j < ab->hw_params.num_rxdma_per_pdev; j++) { idr_init(&dp->rx_mon_status_refill_ring[j].bufs_idr); spin_lock_init(&dp->rx_mon_status_refill_ring[j].idr_lock); } diff --git a/drivers/net/wireless/ath/ath11k/dp_rx.c b/drivers/net/wireless/ath/ath11k/dp_rx.c index b5def3cfcad3..b3059e208446 100644 --- a/drivers/net/wireless/ath/ath11k/dp_rx.c +++ b/drivers/net/wireless/ath/ath11k/dp_rx.c @@ -311,7 +311,7 @@ static void ath11k_dp_service_mon_ring(struct timer_list *t) struct ath11k_base *ab = from_timer(ab, t, mon_reap_timer); int i; - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) ath11k_dp_rx_process_mon_rings(ab, i, NULL, DP_MON_SERVICE_BUDGET); mod_timer(&ab->mon_reap_timer, jiffies + @@ -324,7 +324,7 @@ static int ath11k_dp_purge_mon_ring(struct ath11k_base *ab) unsigned long timeout = jiffies + msecs_to_jiffies(DP_MON_PURGE_TIMEOUT_MS); do { - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) reaped += ath11k_dp_rx_process_mon_rings(ab, i, NULL, DP_MON_SERVICE_BUDGET); @@ -468,7 +468,7 @@ static int ath11k_dp_rxdma_pdev_buf_free(struct ath11k *ar) rx_ring = &dp->rxdma_mon_buf_ring; ath11k_dp_rxdma_buf_ring_free(ar, rx_ring); - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { rx_ring = &dp->rx_mon_status_refill_ring[i]; ath11k_dp_rxdma_buf_ring_free(ar, rx_ring); } @@ -506,7 +506,7 @@ static int ath11k_dp_rxdma_pdev_buf_setup(struct ath11k *ar) ath11k_dp_rxdma_ring_buf_setup(ar, rx_ring, HAL_RXDMA_MONITOR_BUF); } - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { rx_ring = &dp->rx_mon_status_refill_ring[i]; ath11k_dp_rxdma_ring_buf_setup(ar, rx_ring, HAL_RXDMA_MONITOR_STATUS); } @@ -522,7 +522,7 @@ static void ath11k_dp_rx_pdev_srng_free(struct ath11k *ar) ath11k_dp_srng_cleanup(ab, &dp->rx_refill_buf_ring.refill_buf_ring); - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { if (ab->hw_params.rx_mac_buf_ring) ath11k_dp_srng_cleanup(ab, &dp->rx_mac_buf_ring[i]); @@ -585,7 +585,7 @@ static int ath11k_dp_rx_pdev_srng_alloc(struct ath11k *ar) } if (ar->ab->hw_params.rx_mac_buf_ring) { - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ret = ath11k_dp_srng_setup(ar->ab, &dp->rx_mac_buf_ring[i], HAL_RXDMA_BUF, 1, @@ -598,7 +598,7 @@ static int ath11k_dp_rx_pdev_srng_alloc(struct ath11k *ar) } } - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ret = ath11k_dp_srng_setup(ar->ab, &dp->rxdma_err_dst_ring[i], HAL_RXDMA_DST, 0, dp->mac_id + i, DP_RXDMA_ERR_DST_RING_SIZE); @@ -608,7 +608,7 @@ static int ath11k_dp_rx_pdev_srng_alloc(struct ath11k *ar) } } - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { srng = &dp->rx_mon_status_refill_ring[i].refill_buf_ring; ret = ath11k_dp_srng_setup(ar->ab, srng, @@ -4495,7 +4495,7 @@ int ath11k_dp_rx_pdev_alloc(struct ath11k_base *ab, int mac_id) } if (ab->hw_params.rx_mac_buf_ring) { - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ring_id = dp->rx_mac_buf_ring[i].ring_id; ret = ath11k_dp_tx_htt_srng_setup(ab, ring_id, mac_id + i, HAL_RXDMA_BUF); @@ -4507,7 +4507,7 @@ int ath11k_dp_rx_pdev_alloc(struct ath11k_base *ab, int mac_id) } } - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ring_id = dp->rxdma_err_dst_ring[i].ring_id; ret = ath11k_dp_tx_htt_srng_setup(ab, ring_id, mac_id + i, HAL_RXDMA_DST); @@ -4547,7 +4547,7 @@ int ath11k_dp_rx_pdev_alloc(struct ath11k_base *ab, int mac_id) } config_refill_ring: - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ring_id = dp->rx_mon_status_refill_ring[i].refill_buf_ring.ring_id; ret = ath11k_dp_tx_htt_srng_setup(ab, ring_id, mac_id + i, HAL_RXDMA_MONITOR_STATUS); diff --git a/drivers/net/wireless/ath/ath11k/dp_tx.c b/drivers/net/wireless/ath/ath11k/dp_tx.c index c1072e66e3e8..22a0d50701ed 100644 --- a/drivers/net/wireless/ath/ath11k/dp_tx.c +++ b/drivers/net/wireless/ath/ath11k/dp_tx.c @@ -1035,7 +1035,7 @@ int ath11k_dp_tx_htt_h2t_ppdu_stats_req(struct ath11k *ar, u32 mask) int ret; int i; - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { skb = ath11k_htc_alloc_skb(ab, len); if (!skb) return -ENOMEM; @@ -1218,7 +1218,7 @@ int ath11k_dp_tx_htt_monitor_mode_ring_config(struct ath11k *ar, bool reset) &tlv_filter); } else if (!reset) { /* set in monitor mode only */ - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ring_id = dp->rx_mac_buf_ring[i].ring_id; ret = ath11k_dp_tx_htt_rx_filter_setup(ar->ab, ring_id, dp->mac_id + i, @@ -1231,7 +1231,7 @@ int ath11k_dp_tx_htt_monitor_mode_ring_config(struct ath11k *ar, bool reset) if (ret) return ret; - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ring_id = dp->rx_mon_status_refill_ring[i].refill_buf_ring.ring_id; if (!reset) { tlv_filter.rx_filter = diff --git a/drivers/net/wireless/ath/ath11k/hw.h b/drivers/net/wireless/ath/ath11k/hw.h index 1b070747a5db..34ddd4764206 100644 --- a/drivers/net/wireless/ath/ath11k/hw.h +++ b/drivers/net/wireless/ath/ath11k/hw.h @@ -1,7 +1,7 @@ /* SPDX-License-Identifier: BSD-3-Clause-Clear */ /* * Copyright (c) 2018-2019 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2023 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) 2021-2024 Qualcomm Innovation Center, Inc. All rights reserved. */ #ifndef ATH11K_HW_H @@ -167,7 +167,7 @@ struct ath11k_hw_params { bool single_pdev_only; bool rxdma1_enable; - int num_rxmda_per_pdev; + int num_rxdma_per_pdev; bool rx_mac_buf_ring; bool vdev_start_delay; bool htt_peer_map_v2; diff --git a/drivers/net/wireless/ath/ath11k/mac.c b/drivers/net/wireless/ath/ath11k/mac.c index 0937d725d4d0..fade0b8d8778 100644 --- a/drivers/net/wireless/ath/ath11k/mac.c +++ b/drivers/net/wireless/ath/ath11k/mac.c @@ -1461,7 +1461,8 @@ static int ath11k_mac_setup_bcn_tmpl_ema(struct ath11k_vif *arvif) if (!beacons || !beacons->cnt) { ath11k_warn(arvif->ar->ab, "failed to get ema beacon templates from mac80211\n"); - return -EPERM; + ret = -EPERM; + goto free; } if (tx_arvif == arvif) @@ -1491,11 +1492,11 @@ static int ath11k_mac_setup_bcn_tmpl_ema(struct ath11k_vif *arvif) } } - ieee80211_beacon_free_ema_list(beacons); - if (tx_arvif != arvif && !nontx_vif_params_set) - return -EINVAL; /* Profile not found in the beacons */ + ret = -EINVAL; /* Profile not found in the beacons */ +free: + ieee80211_beacon_free_ema_list(beacons); return ret; } @@ -1528,16 +1529,18 @@ static int ath11k_mac_setup_bcn_tmpl_mbssid(struct ath11k_vif *arvif) if (tx_arvif == arvif) ath11k_mac_set_vif_params(tx_arvif, bcn); - else if (!ath11k_mac_set_nontx_vif_params(tx_arvif, arvif, bcn)) - return -EINVAL; + else if (!ath11k_mac_set_nontx_vif_params(tx_arvif, arvif, bcn)) { + ret = -EINVAL; + goto free; + } ret = ath11k_wmi_bcn_tmpl(ar, arvif->vdev_id, &offs, bcn, 0); - kfree_skb(bcn); - if (ret) ath11k_warn(ab, "failed to submit beacon template command: %d\n", ret); +free: + kfree_skb(bcn); return ret; } @@ -6082,7 +6085,7 @@ static int ath11k_mac_config_mon_status_default(struct ath11k *ar, bool enable) tlv_filter.rx_filter = ath11k_debugfs_rx_filter(ar); } - for (i = 0; i < ab->hw_params.num_rxmda_per_pdev; i++) { + for (i = 0; i < ab->hw_params.num_rxdma_per_pdev; i++) { ring_id = ar->dp.rx_mon_status_refill_ring[i].refill_buf_ring.ring_id; ret = ath11k_dp_tx_htt_rx_filter_setup(ar->ab, ring_id, ar->dp.mac_id + i, @@ -6213,6 +6216,7 @@ static int ath11k_mac_op_start(struct ieee80211_hw *hw) ar->num_started_vdevs = 0; ar->num_created_vdevs = 0; ar->num_peers = 0; + ar->num_stations = 0; ar->allocated_vdev_map = 0; /* Configure monitor status ring with default rx_filter to get rx status @@ -7762,12 +7766,14 @@ ath11k_mac_get_tx_mcs_map(const struct ieee80211_sta_he_cap *he_cap) static bool ath11k_mac_bitrate_mask_get_single_nss(struct ath11k *ar, + struct ath11k_vif *arvif, enum nl80211_band band, const struct cfg80211_bitrate_mask *mask, int *nss) { struct ieee80211_supported_band *sband = &ar->mac.sbands[band]; u16 vht_mcs_map = le16_to_cpu(sband->vht_cap.vht_mcs.tx_mcs_map); + const struct ieee80211_sta_he_cap *he_cap; u16 he_mcs_map = 0; u8 ht_nss_mask = 0; u8 vht_nss_mask = 0; @@ -7798,7 +7804,11 @@ ath11k_mac_bitrate_mask_get_single_nss(struct ath11k *ar, return false; } - he_mcs_map = le16_to_cpu(ath11k_mac_get_tx_mcs_map(&sband->iftype_data->he_cap)); + he_cap = ieee80211_get_he_iftype_cap_vif(sband, arvif->vif); + if (!he_cap) + return false; + + he_mcs_map = le16_to_cpu(ath11k_mac_get_tx_mcs_map(he_cap)); for (i = 0; i < ARRAY_SIZE(mask->control[band].he_mcs); i++) { if (mask->control[band].he_mcs[i] == 0) @@ -8211,10 +8221,10 @@ ath11k_mac_op_set_bitrate_mask(struct ieee80211_hw *hw, arvif->vdev_id, ret); return ret; } - ieee80211_iterate_stations_atomic(ar->hw, - ath11k_mac_disable_peer_fixed_rate, - arvif); - } else if (ath11k_mac_bitrate_mask_get_single_nss(ar, band, mask, + ieee80211_iterate_stations_mtx(ar->hw, + ath11k_mac_disable_peer_fixed_rate, + arvif); + } else if (ath11k_mac_bitrate_mask_get_single_nss(ar, arvif, band, mask, &single_nss)) { rate = WMI_FIXED_RATE_NONE; nss = single_nss; @@ -8282,9 +8292,9 @@ ath11k_mac_op_set_bitrate_mask(struct ieee80211_hw *hw, } mutex_lock(&ar->conf_mutex); - ieee80211_iterate_stations_atomic(ar->hw, - ath11k_mac_disable_peer_fixed_rate, - arvif); + ieee80211_iterate_stations_mtx(ar->hw, + ath11k_mac_disable_peer_fixed_rate, + arvif); arvif->bitrate_mask = *mask; ieee80211_iterate_stations_atomic(ar->hw, diff --git a/drivers/net/wireless/ath/ath11k/mhi.c b/drivers/net/wireless/ath/ath11k/mhi.c index d3a4b561832c..8e7bb9c4f148 100644 --- a/drivers/net/wireless/ath/ath11k/mhi.c +++ b/drivers/net/wireless/ath/ath11k/mhi.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: BSD-3-Clause-Clear /* * Copyright (c) 2020 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2023 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. */ #include @@ -61,7 +61,6 @@ static struct mhi_channel_config ath11k_mhi_channels_qca6390[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, }, { .num = 21, @@ -75,7 +74,6 @@ static struct mhi_channel_config ath11k_mhi_channels_qca6390[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = true, }, }; @@ -154,7 +152,6 @@ static struct mhi_channel_config ath11k_mhi_channels_qcn9074[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, }, { .num = 21, @@ -168,7 +165,6 @@ static struct mhi_channel_config ath11k_mhi_channels_qcn9074[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = true, }, }; @@ -334,8 +330,10 @@ static void ath11k_mhi_op_status_cb(struct mhi_controller *mhi_cntrl, ath11k_warn(ab, "firmware crashed: MHI_CB_SYS_ERROR\n"); break; case MHI_CB_EE_RDDM: + spin_lock_bh(&ab->base_lock); if (!(test_bit(ATH11K_FLAG_UNREGISTERING, &ab->dev_flags))) queue_work(ab->workqueue_aux, &ab->reset_work); + spin_unlock_bh(&ab->base_lock); break; default: break; diff --git a/drivers/net/wireless/ath/ath11k/pci.c b/drivers/net/wireless/ath/ath11k/pci.c index 02c261a30a4d..2a9ac6421d71 100644 --- a/drivers/net/wireless/ath/ath11k/pci.c +++ b/drivers/net/wireless/ath/ath11k/pci.c @@ -972,6 +972,13 @@ static void ath11k_pci_shutdown(struct pci_dev *pdev) struct ath11k_pci *ab_pci = ath11k_pci_priv(ab); ath11k_pci_set_irq_affinity_hint(ab_pci, NULL); + + spin_lock_bh(&ab->base_lock); + set_bit(ATH11K_FLAG_UNREGISTERING, &ab->dev_flags); + spin_unlock_bh(&ab->base_lock); + + cancel_work_sync(&ab->reset_work); + ath11k_pci_power_down(ab); } diff --git a/drivers/net/wireless/ath/ath11k/reg.c b/drivers/net/wireless/ath/ath11k/reg.c index c9e8bbc4896f..9d30928588ba 100644 --- a/drivers/net/wireless/ath/ath11k/reg.c +++ b/drivers/net/wireless/ath/ath11k/reg.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: BSD-3-Clause-Clear /* * Copyright (c) 2018-2019 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2023 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) 2021-2025 Qualcomm Innovation Center, Inc. All rights reserved. */ #include @@ -56,6 +56,19 @@ ath11k_reg_notifier(struct wiphy *wiphy, struct regulatory_request *request) ath11k_dbg(ar->ab, ATH11K_DBG_REG, "Regulatory Notification received for %s\n", wiphy_name(wiphy)); + if (request->initiator == NL80211_REGDOM_SET_BY_DRIVER) { + ath11k_dbg(ar->ab, ATH11K_DBG_REG, + "driver initiated regd update\n"); + if (ar->state != ATH11K_STATE_ON) + return; + + ret = ath11k_reg_update_chan_list(ar, true); + if (ret) + ath11k_warn(ar->ab, "failed to update channel list: %d\n", ret); + + return; + } + /* Currently supporting only General User Hints. Cell base user * hints to be handled later. * Hints from other sources like Core, Beacons are not expected for @@ -295,12 +308,6 @@ int ath11k_regd_update(struct ath11k *ar) if (ret) goto err; - if (ar->state == ATH11K_STATE_ON) { - ret = ath11k_reg_update_chan_list(ar, true); - if (ret) - goto err; - } - return 0; err: ath11k_warn(ab, "failed to perform regd update : %d\n", ret); @@ -608,25 +615,68 @@ ath11k_reg_update_weather_radar_band(struct ath11k_base *ab, *rule_idx = i; } +enum wmi_reg_6ghz_ap_type +ath11k_reg_ap_pwr_convert(enum ieee80211_ap_reg_power power_type) +{ + switch (power_type) { + case IEEE80211_REG_LPI_AP: + return WMI_REG_INDOOR_AP; + case IEEE80211_REG_SP_AP: + return WMI_REG_STANDARD_POWER_AP; + case IEEE80211_REG_VLP_AP: + return WMI_REG_VERY_LOW_POWER_AP; + default: + return WMI_REG_MAX_AP_TYPE; + } +} + struct ieee80211_regdomain * ath11k_reg_build_regd(struct ath11k_base *ab, - struct cur_regulatory_info *reg_info, bool intersect) + struct cur_regulatory_info *reg_info, bool intersect, + enum wmi_vdev_type vdev_type, + enum ieee80211_ap_reg_power power_type) { struct ieee80211_regdomain *tmp_regd, *default_regd, *new_regd = NULL; - struct cur_reg_rule *reg_rule; + struct cur_reg_rule *reg_rule, *reg_rule_6ghz; u8 i = 0, j = 0, k = 0; u8 num_rules; u16 max_bw; - u32 flags; + u32 flags, reg_6ghz_number, max_bw_6ghz; char alpha2[3]; num_rules = reg_info->num_5ghz_reg_rules + reg_info->num_2ghz_reg_rules; - /* FIXME: Currently taking reg rules for 6 GHz only from Indoor AP mode list. - * This can be updated after complete 6 GHz regulatory support is added. - */ - if (reg_info->is_ext_reg_event) - num_rules += reg_info->num_6ghz_rules_ap[WMI_REG_INDOOR_AP]; + if (reg_info->is_ext_reg_event) { + if (vdev_type == WMI_VDEV_TYPE_STA) { + enum wmi_reg_6ghz_ap_type ap_type; + + ap_type = ath11k_reg_ap_pwr_convert(power_type); + + if (ap_type == WMI_REG_MAX_AP_TYPE) + ap_type = WMI_REG_INDOOR_AP; + + reg_6ghz_number = reg_info->num_6ghz_rules_client + [ap_type][WMI_REG_DEFAULT_CLIENT]; + + if (reg_6ghz_number == 0) { + ap_type = WMI_REG_INDOOR_AP; + reg_6ghz_number = reg_info->num_6ghz_rules_client + [ap_type][WMI_REG_DEFAULT_CLIENT]; + } + + reg_rule_6ghz = reg_info->reg_rules_6ghz_client_ptr + [ap_type][WMI_REG_DEFAULT_CLIENT]; + max_bw_6ghz = reg_info->max_bw_6ghz_client + [ap_type][WMI_REG_DEFAULT_CLIENT]; + } else { + reg_6ghz_number = reg_info->num_6ghz_rules_ap[WMI_REG_INDOOR_AP]; + reg_rule_6ghz = + reg_info->reg_rules_6ghz_ap_ptr[WMI_REG_INDOOR_AP]; + max_bw_6ghz = reg_info->max_bw_6ghz_ap[WMI_REG_INDOOR_AP]; + } + + num_rules += reg_6ghz_number; + } if (!num_rules) goto ret; @@ -673,13 +723,10 @@ ath11k_reg_build_regd(struct ath11k_base *ab, * per other BW rule flags we pass from here */ flags = NL80211_RRF_AUTO_BW; - } else if (reg_info->is_ext_reg_event && - reg_info->num_6ghz_rules_ap[WMI_REG_INDOOR_AP] && - (k < reg_info->num_6ghz_rules_ap[WMI_REG_INDOOR_AP])) { - reg_rule = reg_info->reg_rules_6ghz_ap_ptr[WMI_REG_INDOOR_AP] + - k++; - max_bw = min_t(u16, reg_rule->max_bw, - reg_info->max_bw_6ghz_ap[WMI_REG_INDOOR_AP]); + } else if (reg_info->is_ext_reg_event && reg_6ghz_number && + k < reg_6ghz_number) { + reg_rule = reg_rule_6ghz + k++; + max_bw = min_t(u16, reg_rule->max_bw, max_bw_6ghz); flags = NL80211_RRF_AUTO_BW; } else { break; @@ -767,6 +814,7 @@ void ath11k_regd_update_work(struct work_struct *work) void ath11k_reg_init(struct ath11k *ar) { ar->hw->wiphy->regulatory_flags = REGULATORY_WIPHY_SELF_MANAGED; + ar->hw->wiphy->flags |= WIPHY_FLAG_NOTIFY_REGDOM_BY_DRIVER; ar->hw->wiphy->reg_notifier = ath11k_reg_notifier; } diff --git a/drivers/net/wireless/ath/ath11k/reg.h b/drivers/net/wireless/ath/ath11k/reg.h index d873b9cf7fc4..30fce1411d1f 100644 --- a/drivers/net/wireless/ath/ath11k/reg.h +++ b/drivers/net/wireless/ath/ath11k/reg.h @@ -31,7 +31,11 @@ void ath11k_reg_free(struct ath11k_base *ab); void ath11k_regd_update_work(struct work_struct *work); struct ieee80211_regdomain * ath11k_reg_build_regd(struct ath11k_base *ab, - struct cur_regulatory_info *reg_info, bool intersect); + struct cur_regulatory_info *reg_info, bool intersect, + enum wmi_vdev_type vdev_type, + enum ieee80211_ap_reg_power power_type); int ath11k_regd_update(struct ath11k *ar); int ath11k_reg_update_chan_list(struct ath11k *ar, bool wait); +enum wmi_reg_6ghz_ap_type +ath11k_reg_ap_pwr_convert(enum ieee80211_ap_reg_power power_type); #endif diff --git a/drivers/net/wireless/ath/ath11k/wmi.c b/drivers/net/wireless/ath/ath11k/wmi.c index 4db48d02a8f3..2fc9b7a86b8c 100644 --- a/drivers/net/wireless/ath/ath11k/wmi.c +++ b/drivers/net/wireless/ath/ath11k/wmi.c @@ -4649,14 +4649,16 @@ static int ath11k_wmi_tlv_mac_phy_caps_parse(struct ath11k_base *soc, if (svc_rdy_ext->n_mac_phy_caps >= svc_rdy_ext->tot_phy_id) return -ENOBUFS; - len = min_t(u16, len, sizeof(struct wmi_mac_phy_capabilities)); if (!svc_rdy_ext->n_mac_phy_caps) { - svc_rdy_ext->mac_phy_caps = kcalloc(svc_rdy_ext->tot_phy_id, - len, GFP_ATOMIC); + svc_rdy_ext->mac_phy_caps = + kcalloc(svc_rdy_ext->tot_phy_id, + sizeof(*svc_rdy_ext->mac_phy_caps), + GFP_ATOMIC); if (!svc_rdy_ext->mac_phy_caps) return -ENOMEM; } + len = min_t(u16, len, sizeof(struct wmi_mac_phy_capabilities)); memcpy(svc_rdy_ext->mac_phy_caps + svc_rdy_ext->n_mac_phy_caps, ptr, len); svc_rdy_ext->n_mac_phy_caps++; return 0; @@ -7147,7 +7149,7 @@ static int ath11k_reg_chan_list_event(struct ath11k_base *ab, * event. Otherwise, it goes to fallback. */ if (ab->hw_params.single_pdev_only && - pdev_idx < ab->hw_params.num_rxmda_per_pdev) + pdev_idx < ab->hw_params.num_rxdma_per_pdev) goto mem_free; else goto fallback; @@ -7171,7 +7173,8 @@ static int ath11k_reg_chan_list_event(struct ath11k_base *ab, !ath11k_reg_is_world_alpha((char *)reg_info->alpha2)) intersect = true; - regd = ath11k_reg_build_regd(ab, reg_info, intersect); + regd = ath11k_reg_build_regd(ab, reg_info, intersect, + WMI_VDEV_TYPE_AP, IEEE80211_REG_LPI_AP); if (!regd) { ath11k_warn(ab, "failed to build regd from reg_info\n"); goto fallback; @@ -9109,7 +9112,7 @@ int ath11k_wmi_attach(struct ath11k_base *ab) ab->wmi_ab.preferred_hw_mode = WMI_HOST_HW_MODE_MAX; /* It's overwritten when service_ext_ready is handled */ - if (ab->hw_params.single_pdev_only && ab->hw_params.num_rxmda_per_pdev > 1) + if (ab->hw_params.single_pdev_only && ab->hw_params.num_rxdma_per_pdev > 1) ab->wmi_ab.preferred_hw_mode = WMI_HOST_HW_MODE_SINGLE; /* TODO: Init remaining wmi soc resources required */ diff --git a/drivers/net/wireless/ath/ath12k/dp.c b/drivers/net/wireless/ath/ath12k/dp.c index c918f5d12975..637d5ee6e753 100644 --- a/drivers/net/wireless/ath/ath12k/dp.c +++ b/drivers/net/wireless/ath/ath12k/dp.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: BSD-3-Clause-Clear /* * Copyright (c) 2018-2021 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2022 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) 2021-2023 Qualcomm Innovation Center, Inc. All rights reserved. */ #include @@ -970,9 +970,7 @@ int ath12k_dp_service_srng(struct ath12k_base *ab, struct ath12k_dp *dp = &ab->dp; struct dp_rxdma_ring *rx_ring = &dp->rx_refill_buf_ring; - ath12k_dp_rx_bufs_replenish(ab, 0, rx_ring, 0, - ab->hw_params->hal_params->rx_buf_rbm, - true); + ath12k_dp_rx_bufs_replenish(ab, rx_ring, 0); } /* TODO: Implement handler for other interrupts */ diff --git a/drivers/net/wireless/ath/ath12k/dp.h b/drivers/net/wireless/ath/ath12k/dp.h index 284032db0b98..7a4d342380ee 100644 --- a/drivers/net/wireless/ath/ath12k/dp.h +++ b/drivers/net/wireless/ath/ath12k/dp.h @@ -1,7 +1,7 @@ /* SPDX-License-Identifier: BSD-3-Clause-Clear */ /* * Copyright (c) 2018-2021 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2022 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) 2021-2023 Qualcomm Innovation Center, Inc. All rights reserved. */ #ifndef ATH12K_DP_H @@ -31,7 +31,7 @@ struct dp_srng { u32 ring_id; }; -struct dp_rxdma_ring { +struct dp_rxdma_mon_ring { struct dp_srng refill_buf_ring; struct idr bufs_idr; /* Protects bufs_idr */ @@ -39,6 +39,11 @@ struct dp_rxdma_ring { int bufs_max; }; +struct dp_rxdma_ring { + struct dp_srng refill_buf_ring; + int bufs_max; +}; + #define ATH12K_TX_COMPL_NEXT(x) (((x) + 1) % DP_TX_COMP_RING_SIZE) struct dp_tx_ring { @@ -353,8 +358,8 @@ struct ath12k_dp { struct dp_rxdma_ring rx_refill_buf_ring; struct dp_srng rx_mac_buf_ring[MAX_RXDMA_PER_PDEV]; struct dp_srng rxdma_err_dst_ring[MAX_RXDMA_PER_PDEV]; - struct dp_rxdma_ring rxdma_mon_buf_ring; - struct dp_rxdma_ring tx_mon_buf_ring; + struct dp_rxdma_mon_ring rxdma_mon_buf_ring; + struct dp_rxdma_mon_ring tx_mon_buf_ring; struct ath12k_reo_q_addr_lut reoq_lut; }; diff --git a/drivers/net/wireless/ath/ath12k/dp_mon.c b/drivers/net/wireless/ath/ath12k/dp_mon.c index 69bf75ebd751..0116fa963a36 100644 --- a/drivers/net/wireless/ath/ath12k/dp_mon.c +++ b/drivers/net/wireless/ath/ath12k/dp_mon.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: BSD-3-Clause-Clear /* * Copyright (c) 2019-2021 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2022 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) 2021-2023 Qualcomm Innovation Center, Inc. All rights reserved. */ #include "dp_mon.h" @@ -10,12 +10,10 @@ #include "dp_tx.h" #include "peer.h" -static void ath12k_dp_mon_rx_handle_ofdma_info(void *rx_tlv, - struct hal_rx_user_status *rx_user_status) +static void +ath12k_dp_mon_rx_handle_ofdma_info(const struct hal_rx_ppdu_end_user_stats *ppdu_end_user, + struct hal_rx_user_status *rx_user_status) { - struct hal_rx_ppdu_end_user_stats *ppdu_end_user = - (struct hal_rx_ppdu_end_user_stats *)rx_tlv; - rx_user_status->ul_ofdma_user_v0_word0 = __le32_to_cpu(ppdu_end_user->usr_resp_ref); rx_user_status->ul_ofdma_user_v0_word1 = @@ -23,13 +21,12 @@ static void ath12k_dp_mon_rx_handle_ofdma_info(void *rx_tlv, } static void -ath12k_dp_mon_rx_populate_byte_count(void *rx_tlv, void *ppduinfo, +ath12k_dp_mon_rx_populate_byte_count(const struct hal_rx_ppdu_end_user_stats *stats, + void *ppduinfo, struct hal_rx_user_status *rx_user_status) { - struct hal_rx_ppdu_end_user_stats *ppdu_end_user = - (struct hal_rx_ppdu_end_user_stats *)rx_tlv; - u32 mpdu_ok_byte_count = __le32_to_cpu(ppdu_end_user->mpdu_ok_cnt); - u32 mpdu_err_byte_count = __le32_to_cpu(ppdu_end_user->mpdu_err_cnt); + u32 mpdu_ok_byte_count = __le32_to_cpu(stats->mpdu_ok_cnt); + u32 mpdu_err_byte_count = __le32_to_cpu(stats->mpdu_err_cnt); rx_user_status->mpdu_ok_byte_count = u32_get_bits(mpdu_ok_byte_count, @@ -40,7 +37,7 @@ ath12k_dp_mon_rx_populate_byte_count(void *rx_tlv, void *ppduinfo, } static void -ath12k_dp_mon_rx_populate_mu_user_info(void *rx_tlv, +ath12k_dp_mon_rx_populate_mu_user_info(const struct hal_rx_ppdu_end_user_stats *rx_tlv, struct hal_rx_mon_ppdu_info *ppdu_info, struct hal_rx_user_status *rx_user_status) { @@ -78,11 +75,9 @@ ath12k_dp_mon_rx_populate_mu_user_info(void *rx_tlv, ath12k_dp_mon_rx_populate_byte_count(rx_tlv, ppdu_info, rx_user_status); } -static void ath12k_dp_mon_parse_vht_sig_a(u8 *tlv_data, +static void ath12k_dp_mon_parse_vht_sig_a(const struct hal_rx_vht_sig_a_info *vht_sig, struct hal_rx_mon_ppdu_info *ppdu_info) { - struct hal_rx_vht_sig_a_info *vht_sig = - (struct hal_rx_vht_sig_a_info *)tlv_data; u32 nsts, group_id, info0, info1; u8 gi_setting; @@ -124,11 +119,9 @@ static void ath12k_dp_mon_parse_vht_sig_a(u8 *tlv_data, u32_get_bits(info1, HAL_RX_VHT_SIG_A_INFO_INFO1_SU_MU_CODING); } -static void ath12k_dp_mon_parse_ht_sig(u8 *tlv_data, +static void ath12k_dp_mon_parse_ht_sig(const struct hal_rx_ht_sig_info *ht_sig, struct hal_rx_mon_ppdu_info *ppdu_info) { - struct hal_rx_ht_sig_info *ht_sig = - (struct hal_rx_ht_sig_info *)tlv_data; u32 info0 = __le32_to_cpu(ht_sig->info0); u32 info1 = __le32_to_cpu(ht_sig->info1); @@ -141,11 +134,9 @@ static void ath12k_dp_mon_parse_ht_sig(u8 *tlv_data, ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_SU; } -static void ath12k_dp_mon_parse_l_sig_b(u8 *tlv_data, +static void ath12k_dp_mon_parse_l_sig_b(const struct hal_rx_lsig_b_info *lsigb, struct hal_rx_mon_ppdu_info *ppdu_info) { - struct hal_rx_lsig_b_info *lsigb = - (struct hal_rx_lsig_b_info *)tlv_data; u32 info0 = __le32_to_cpu(lsigb->info0); u8 rate; @@ -175,11 +166,9 @@ static void ath12k_dp_mon_parse_l_sig_b(u8 *tlv_data, ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_SU; } -static void ath12k_dp_mon_parse_l_sig_a(u8 *tlv_data, +static void ath12k_dp_mon_parse_l_sig_a(const struct hal_rx_lsig_a_info *lsiga, struct hal_rx_mon_ppdu_info *ppdu_info) { - struct hal_rx_lsig_a_info *lsiga = - (struct hal_rx_lsig_a_info *)tlv_data; u32 info0 = __le32_to_cpu(lsiga->info0); u8 rate; @@ -217,14 +206,13 @@ static void ath12k_dp_mon_parse_l_sig_a(u8 *tlv_data, ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_SU; } -static void ath12k_dp_mon_parse_he_sig_b2_ofdma(u8 *tlv_data, - struct hal_rx_mon_ppdu_info *ppdu_info) +static void +ath12k_dp_mon_parse_he_sig_b2_ofdma(const struct hal_rx_he_sig_b2_ofdma_info *ofdma, + struct hal_rx_mon_ppdu_info *ppdu_info) { - struct hal_rx_he_sig_b2_ofdma_info *he_sig_b2_ofdma = - (struct hal_rx_he_sig_b2_ofdma_info *)tlv_data; u32 info0, value; - info0 = __le32_to_cpu(he_sig_b2_ofdma->info0); + info0 = __le32_to_cpu(ofdma->info0); ppdu_info->he_data1 |= HE_MCS_KNOWN | HE_DCM_KNOWN | HE_CODING_KNOWN; @@ -255,11 +243,10 @@ static void ath12k_dp_mon_parse_he_sig_b2_ofdma(u8 *tlv_data, ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_MU_OFDMA; } -static void ath12k_dp_mon_parse_he_sig_b2_mu(u8 *tlv_data, - struct hal_rx_mon_ppdu_info *ppdu_info) +static void +ath12k_dp_mon_parse_he_sig_b2_mu(const struct hal_rx_he_sig_b2_mu_info *he_sig_b2_mu, + struct hal_rx_mon_ppdu_info *ppdu_info) { - struct hal_rx_he_sig_b2_mu_info *he_sig_b2_mu = - (struct hal_rx_he_sig_b2_mu_info *)tlv_data; u32 info0, value; info0 = __le32_to_cpu(he_sig_b2_mu->info0); @@ -282,11 +269,10 @@ static void ath12k_dp_mon_parse_he_sig_b2_mu(u8 *tlv_data, ppdu_info->nss = u32_get_bits(info0, HAL_RX_HE_SIG_B2_MU_INFO_INFO0_STA_NSTS); } -static void ath12k_dp_mon_parse_he_sig_b1_mu(u8 *tlv_data, - struct hal_rx_mon_ppdu_info *ppdu_info) +static void +ath12k_dp_mon_parse_he_sig_b1_mu(const struct hal_rx_he_sig_b1_mu_info *he_sig_b1_mu, + struct hal_rx_mon_ppdu_info *ppdu_info) { - struct hal_rx_he_sig_b1_mu_info *he_sig_b1_mu = - (struct hal_rx_he_sig_b1_mu_info *)tlv_data; u32 info0 = __le32_to_cpu(he_sig_b1_mu->info0); u16 ru_tones; @@ -297,11 +283,10 @@ static void ath12k_dp_mon_parse_he_sig_b1_mu(u8 *tlv_data, ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_MU_MIMO; } -static void ath12k_dp_mon_parse_he_sig_mu(u8 *tlv_data, - struct hal_rx_mon_ppdu_info *ppdu_info) +static void +ath12k_dp_mon_parse_he_sig_mu(const struct hal_rx_he_sig_a_mu_dl_info *he_sig_a_mu_dl, + struct hal_rx_mon_ppdu_info *ppdu_info) { - struct hal_rx_he_sig_a_mu_dl_info *he_sig_a_mu_dl = - (struct hal_rx_he_sig_a_mu_dl_info *)tlv_data; u32 info0, info1, value; u16 he_gi = 0, he_ltf = 0; @@ -432,11 +417,9 @@ static void ath12k_dp_mon_parse_he_sig_mu(u8 *tlv_data, ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_MU_MIMO; } -static void ath12k_dp_mon_parse_he_sig_su(u8 *tlv_data, +static void ath12k_dp_mon_parse_he_sig_su(const struct hal_rx_he_sig_a_su_info *he_sig_a, struct hal_rx_mon_ppdu_info *ppdu_info) { - struct hal_rx_he_sig_a_su_info *he_sig_a = - (struct hal_rx_he_sig_a_su_info *)tlv_data; u32 info0, info1, value; u32 dcm; u8 he_dcm = 0, he_stbc = 0; @@ -585,15 +568,15 @@ static void ath12k_dp_mon_parse_he_sig_su(u8 *tlv_data, static enum hal_rx_mon_status ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab, struct ath12k_mon_data *pmon, - u32 tlv_tag, u8 *tlv_data, u32 userid) + u32 tlv_tag, const void *tlv_data, + u32 userid) { struct hal_rx_mon_ppdu_info *ppdu_info = &pmon->mon_ppdu_info; u32 info[7]; switch (tlv_tag) { case HAL_RX_PPDU_START: { - struct hal_rx_ppdu_start *ppdu_start = - (struct hal_rx_ppdu_start *)tlv_data; + const struct hal_rx_ppdu_start *ppdu_start = tlv_data; info[0] = __le32_to_cpu(ppdu_start->info0); @@ -612,8 +595,8 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab, break; } case HAL_RX_PPDU_END_USER_STATS: { - struct hal_rx_ppdu_end_user_stats *eu_stats = - (struct hal_rx_ppdu_end_user_stats *)tlv_data; + const struct hal_rx_ppdu_end_user_stats *eu_stats = tlv_data; + u32 tid_bitmap; info[0] = __le32_to_cpu(eu_stats->info0); info[1] = __le32_to_cpu(eu_stats->info1); @@ -626,10 +609,9 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab, u32_get_bits(info[2], HAL_RX_PPDU_END_USER_STATS_INFO2_AST_INDEX); ppdu_info->fc_valid = u32_get_bits(info[1], HAL_RX_PPDU_END_USER_STATS_INFO1_FC_VALID); - ppdu_info->tid = - ffs(u32_get_bits(info[6], - HAL_RX_PPDU_END_USER_STATS_INFO6_TID_BITMAP) - - 1); + tid_bitmap = u32_get_bits(info[6], + HAL_RX_PPDU_END_USER_STATS_INFO6_TID_BITMAP); + ppdu_info->tid = ffs(tid_bitmap) - 1; ppdu_info->tcp_msdu_count = u32_get_bits(info[4], HAL_RX_PPDU_END_USER_STATS_INFO4_TCP_MSDU_CNT); @@ -670,8 +652,8 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab, &ppdu_info->userstats[userid]; ppdu_info->num_users += 1; - ath12k_dp_mon_rx_handle_ofdma_info(tlv_data, rxuser_stats); - ath12k_dp_mon_rx_populate_mu_user_info(tlv_data, ppdu_info, + ath12k_dp_mon_rx_handle_ofdma_info(eu_stats, rxuser_stats); + ath12k_dp_mon_rx_populate_mu_user_info(eu_stats, ppdu_info, rxuser_stats); } ppdu_info->mpdu_fcs_ok_bitmap[0] = __le32_to_cpu(eu_stats->rsvd1[0]); @@ -679,8 +661,8 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab, break; } case HAL_RX_PPDU_END_USER_STATS_EXT: { - struct hal_rx_ppdu_end_user_stats_ext *eu_stats = - (struct hal_rx_ppdu_end_user_stats_ext *)tlv_data; + const struct hal_rx_ppdu_end_user_stats_ext *eu_stats = tlv_data; + ppdu_info->mpdu_fcs_ok_bitmap[2] = __le32_to_cpu(eu_stats->info1); ppdu_info->mpdu_fcs_ok_bitmap[3] = __le32_to_cpu(eu_stats->info2); ppdu_info->mpdu_fcs_ok_bitmap[4] = __le32_to_cpu(eu_stats->info3); @@ -726,40 +708,25 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab, break; case HAL_PHYRX_RSSI_LEGACY: { - struct hal_rx_phyrx_rssi_legacy_info *rssi = - (struct hal_rx_phyrx_rssi_legacy_info *)tlv_data; - u32 reception_type = 0; - u32 rssi_legacy_info = __le32_to_cpu(rssi->rsvd[0]); + const struct hal_rx_phyrx_rssi_legacy_info *rssi = tlv_data; info[0] = __le32_to_cpu(rssi->info0); + info[1] = __le32_to_cpu(rssi->info1); /* TODO: Please note that the combined rssi will not be accurate * in MU case. Rssi in MU needs to be retrieved from * PHYRX_OTHER_RECEIVE_INFO TLV. */ ppdu_info->rssi_comb = - u32_get_bits(info[0], - HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RSSI_COMB); - reception_type = - u32_get_bits(rssi_legacy_info, - HAL_RX_PHYRX_RSSI_LEGACY_INFO_RSVD1_RECEPTION); - - switch (reception_type) { - case HAL_RECEPTION_TYPE_ULOFMDA: - ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_MU_OFDMA; - break; - case HAL_RECEPTION_TYPE_ULMIMO: - ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_MU_MIMO; - break; - default: - ppdu_info->reception_type = HAL_RX_RECEPTION_TYPE_SU; - break; - } + u32_get_bits(info[1], + HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO1_RSSI_COMB); + + ppdu_info->bw = u32_get_bits(info[0], + HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RX_BW); break; } case HAL_RXPCU_PPDU_END_INFO: { - struct hal_rx_ppdu_end_duration *ppdu_rx_duration = - (struct hal_rx_ppdu_end_duration *)tlv_data; + const struct hal_rx_ppdu_end_duration *ppdu_rx_duration = tlv_data; info[0] = __le32_to_cpu(ppdu_rx_duration->info0); ppdu_info->rx_duration = @@ -770,9 +737,7 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab, break; } case HAL_RX_MPDU_START: { - struct hal_rx_mpdu_start *mpdu_start = - (struct hal_rx_mpdu_start *)tlv_data; - struct dp_mon_mpdu *mon_mpdu = pmon->mon_mpdu; + const struct hal_rx_mpdu_start *mpdu_start = tlv_data; u16 peer_id; info[1] = __le32_to_cpu(mpdu_start->info1); @@ -789,67 +754,17 @@ ath12k_dp_mon_rx_parse_status_tlv(struct ath12k_base *ab, u32_get_bits(info[0], HAL_RX_MPDU_START_INFO1_PEERID); } - mon_mpdu = kzalloc(sizeof(*mon_mpdu), GFP_ATOMIC); - if (!mon_mpdu) - return HAL_RX_MON_STATUS_PPDU_NOT_DONE; - break; } case HAL_RX_MSDU_START: /* TODO: add msdu start parsing logic */ break; - case HAL_MON_BUF_ADDR: { - struct dp_rxdma_ring *buf_ring = &ab->dp.rxdma_mon_buf_ring; - struct dp_mon_packet_info *packet_info = - (struct dp_mon_packet_info *)tlv_data; - int buf_id = u32_get_bits(packet_info->cookie, - DP_RXDMA_BUF_COOKIE_BUF_ID); - struct sk_buff *msdu; - struct dp_mon_mpdu *mon_mpdu = pmon->mon_mpdu; - struct ath12k_skb_rxcb *rxcb; - - spin_lock_bh(&buf_ring->idr_lock); - msdu = idr_remove(&buf_ring->bufs_idr, buf_id); - spin_unlock_bh(&buf_ring->idr_lock); - - if (unlikely(!msdu)) { - ath12k_warn(ab, "monitor destination with invalid buf_id %d\n", - buf_id); - return HAL_RX_MON_STATUS_PPDU_NOT_DONE; - } - - rxcb = ATH12K_SKB_RXCB(msdu); - dma_unmap_single(ab->dev, rxcb->paddr, - msdu->len + skb_tailroom(msdu), - DMA_FROM_DEVICE); - - if (mon_mpdu->tail) - mon_mpdu->tail->next = msdu; - else - mon_mpdu->tail = msdu; - - ath12k_dp_mon_buf_replenish(ab, buf_ring, 1); - - break; - } - case HAL_RX_MSDU_END: { - struct rx_msdu_end_qcn9274 *msdu_end = - (struct rx_msdu_end_qcn9274 *)tlv_data; - bool is_first_msdu_in_mpdu; - u16 msdu_end_info; - - msdu_end_info = __le16_to_cpu(msdu_end->info5); - is_first_msdu_in_mpdu = u32_get_bits(msdu_end_info, - RX_MSDU_END_INFO5_FIRST_MSDU); - if (is_first_msdu_in_mpdu) { - pmon->mon_mpdu->head = pmon->mon_mpdu->tail; - pmon->mon_mpdu->tail = NULL; - } - break; - } + case HAL_MON_BUF_ADDR: + return HAL_RX_MON_STATUS_BUF_ADDR; + case HAL_RX_MSDU_END: + return HAL_RX_MON_STATUS_MSDU_END; case HAL_RX_MPDU_END: - list_add_tail(&pmon->mon_mpdu->list, &pmon->dp_rx_mon_mpdu_list); - break; + return HAL_RX_MON_STATUS_MPDU_END; case HAL_DUMMY: return HAL_RX_MON_STATUS_BUF_DONE; case HAL_RX_PPDU_END_STATUS_DONE: @@ -1226,7 +1141,10 @@ ath12k_dp_mon_parse_rx_dest(struct ath12k_base *ab, struct ath12k_mon_data *pmon if ((ptr - skb->data) >= DP_RX_BUFFER_SIZE) break; - } while (hal_status == HAL_RX_MON_STATUS_PPDU_NOT_DONE); + } while ((hal_status == HAL_RX_MON_STATUS_PPDU_NOT_DONE) || + (hal_status == HAL_RX_MON_STATUS_BUF_ADDR) || + (hal_status == HAL_RX_MON_STATUS_MPDU_END) || + (hal_status == HAL_RX_MON_STATUS_MSDU_END)); return hal_status; } @@ -1263,7 +1181,7 @@ ath12k_dp_mon_rx_parse_mon_status(struct ath12k *ar, } int ath12k_dp_mon_buf_replenish(struct ath12k_base *ab, - struct dp_rxdma_ring *buf_ring, + struct dp_rxdma_mon_ring *buf_ring, int req_entries) { struct hal_mon_buf_ring *mon_buf; @@ -1598,7 +1516,7 @@ ath12k_dp_mon_tx_gen_prot_frame(struct dp_mon_tx_ppdu_info *tx_ppdu_info) static enum dp_mon_tx_tlv_status ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, struct ath12k_mon_data *pmon, - u16 tlv_tag, u8 *tlv_data, u32 userid) + u16 tlv_tag, const void *tlv_data, u32 userid) { struct dp_mon_tx_ppdu_info *tx_ppdu_info; enum dp_mon_tx_tlv_status status = DP_MON_TX_STATUS_PPDU_NOT_DONE; @@ -1608,8 +1526,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, switch (tlv_tag) { case HAL_TX_FES_SETUP: { - struct hal_tx_fes_setup *tx_fes_setup = - (struct hal_tx_fes_setup *)tlv_data; + const struct hal_tx_fes_setup *tx_fes_setup = tlv_data; info[0] = __le32_to_cpu(tx_fes_setup->info0); tx_ppdu_info->ppdu_id = __le32_to_cpu(tx_fes_setup->schedule_id); @@ -1620,8 +1537,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, } case HAL_TX_FES_STATUS_END: { - struct hal_tx_fes_status_end *tx_fes_status_end = - (struct hal_tx_fes_status_end *)tlv_data; + const struct hal_tx_fes_status_end *tx_fes_status_end = tlv_data; u32 tst_15_0, tst_31_16; info[0] = __le32_to_cpu(tx_fes_status_end->info0); @@ -1638,8 +1554,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, } case HAL_RX_RESPONSE_REQUIRED_INFO: { - struct hal_rx_resp_req_info *rx_resp_req_info = - (struct hal_rx_resp_req_info *)tlv_data; + const struct hal_rx_resp_req_info *rx_resp_req_info = tlv_data; u32 addr_32; u16 addr_16; @@ -1684,8 +1599,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, } case HAL_PCU_PPDU_SETUP_INIT: { - struct hal_tx_pcu_ppdu_setup_init *ppdu_setup = - (struct hal_tx_pcu_ppdu_setup_init *)tlv_data; + const struct hal_tx_pcu_ppdu_setup_init *ppdu_setup = tlv_data; u32 addr_32; u16 addr_16; @@ -1731,8 +1645,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, } case HAL_TX_QUEUE_EXTENSION: { - struct hal_tx_queue_exten *tx_q_exten = - (struct hal_tx_queue_exten *)tlv_data; + const struct hal_tx_queue_exten *tx_q_exten = tlv_data; info[0] = __le32_to_cpu(tx_q_exten->info0); @@ -1744,8 +1657,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, } case HAL_TX_FES_STATUS_START: { - struct hal_tx_fes_status_start *tx_fes_start = - (struct hal_tx_fes_status_start *)tlv_data; + const struct hal_tx_fes_status_start *tx_fes_start = tlv_data; info[0] = __le32_to_cpu(tx_fes_start->info0); @@ -1756,8 +1668,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, } case HAL_TX_FES_STATUS_PROT: { - struct hal_tx_fes_status_prot *tx_fes_status = - (struct hal_tx_fes_status_prot *)tlv_data; + const struct hal_tx_fes_status_prot *tx_fes_status = tlv_data; u32 start_timestamp; u32 end_timestamp; @@ -1784,8 +1695,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, case HAL_TX_FES_STATUS_START_PPDU: case HAL_TX_FES_STATUS_START_PROT: { - struct hal_tx_fes_status_start_prot *tx_fes_stat_start = - (struct hal_tx_fes_status_start_prot *)tlv_data; + const struct hal_tx_fes_status_start_prot *tx_fes_stat_start = tlv_data; u64 ppdu_ts; info[0] = __le32_to_cpu(tx_fes_stat_start->info0); @@ -1800,8 +1710,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, } case HAL_TX_FES_STATUS_USER_PPDU: { - struct hal_tx_fes_status_user_ppdu *tx_fes_usr_ppdu = - (struct hal_tx_fes_status_user_ppdu *)tlv_data; + const struct hal_tx_fes_status_user_ppdu *tx_fes_usr_ppdu = tlv_data; info[0] = __le32_to_cpu(tx_fes_usr_ppdu->info0); @@ -1844,8 +1753,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, break; case HAL_RX_FRAME_BITMAP_ACK: { - struct hal_rx_frame_bitmap_ack *fbm_ack = - (struct hal_rx_frame_bitmap_ack *)tlv_data; + const struct hal_rx_frame_bitmap_ack *fbm_ack = tlv_data; u32 addr_32; u16 addr_16; @@ -1863,8 +1771,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, } case HAL_MACTX_PHY_DESC: { - struct hal_tx_phy_desc *tx_phy_desc = - (struct hal_tx_phy_desc *)tlv_data; + const struct hal_tx_phy_desc *tx_phy_desc = tlv_data; info[0] = __le32_to_cpu(tx_phy_desc->info0); info[1] = __le32_to_cpu(tx_phy_desc->info1); @@ -1906,7 +1813,7 @@ ath12k_dp_mon_tx_parse_status_tlv(struct ath12k_base *ab, } case HAL_MON_BUF_ADDR: { - struct dp_rxdma_ring *buf_ring = &ab->dp.tx_mon_buf_ring; + struct dp_rxdma_mon_ring *buf_ring = &ab->dp.tx_mon_buf_ring; struct dp_mon_packet_info *packet_info = (struct dp_mon_packet_info *)tlv_data; int buf_id = u32_get_bits(packet_info->cookie, @@ -2071,7 +1978,7 @@ int ath12k_dp_mon_srng_process(struct ath12k *ar, int mac_id, int *budget, struct ath12k_skb_rxcb *rxcb; struct dp_srng *mon_dst_ring; struct hal_srng *srng; - struct dp_rxdma_ring *buf_ring; + struct dp_rxdma_mon_ring *buf_ring; u64 cookie; u32 ppdu_id; int num_buffs_reaped = 0, srng_id, buf_id; @@ -2484,7 +2391,7 @@ int ath12k_dp_mon_rx_process_stats(struct ath12k *ar, int mac_id, struct ath12k_skb_rxcb *rxcb; struct dp_srng *mon_dst_ring; struct hal_srng *srng; - struct dp_rxdma_ring *buf_ring; + struct dp_rxdma_mon_ring *buf_ring; struct ath12k_sta *arsta = NULL; struct ath12k_peer *peer; u64 cookie; diff --git a/drivers/net/wireless/ath/ath12k/dp_mon.h b/drivers/net/wireless/ath/ath12k/dp_mon.h index c18c385798a1..fb9e9c176ce5 100644 --- a/drivers/net/wireless/ath/ath12k/dp_mon.h +++ b/drivers/net/wireless/ath/ath12k/dp_mon.h @@ -1,7 +1,7 @@ /* SPDX-License-Identifier: BSD-3-Clause-Clear */ /* * Copyright (c) 2019-2021 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2022 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) 2021-2023 Qualcomm Innovation Center, Inc. All rights reserved. */ #ifndef ATH12K_DP_MON_H @@ -80,7 +80,7 @@ ath12k_dp_mon_rx_parse_mon_status(struct ath12k *ar, int mac_id, struct sk_buff *skb, struct napi_struct *napi); int ath12k_dp_mon_buf_replenish(struct ath12k_base *ab, - struct dp_rxdma_ring *buf_ring, + struct dp_rxdma_mon_ring *buf_ring, int req_entries); int ath12k_dp_mon_srng_process(struct ath12k *ar, int mac_id, int *budget, enum dp_monitor_mode monitor_mode, diff --git a/drivers/net/wireless/ath/ath12k/dp_rx.c b/drivers/net/wireless/ath/ath12k/dp_rx.c index 56dda76d066c..1f8357e9ae03 100644 --- a/drivers/net/wireless/ath/ath12k/dp_rx.c +++ b/drivers/net/wireless/ath/ath12k/dp_rx.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: BSD-3-Clause-Clear /* * Copyright (c) 2018-2021 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2022 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. */ #include @@ -256,22 +256,20 @@ static int ath12k_dp_purge_mon_ring(struct ath12k_base *ab) } /* Returns number of Rx buffers replenished */ -int ath12k_dp_rx_bufs_replenish(struct ath12k_base *ab, int mac_id, +int ath12k_dp_rx_bufs_replenish(struct ath12k_base *ab, struct dp_rxdma_ring *rx_ring, - int req_entries, - enum hal_rx_buf_return_buf_manager mgr, - bool hw_cc) + int req_entries) { struct ath12k_buffer_addr *desc; struct hal_srng *srng; struct sk_buff *skb; int num_free; int num_remain; - int buf_id; u32 cookie; dma_addr_t paddr; struct ath12k_dp *dp = &ab->dp; struct ath12k_rx_desc_info *rx_desc; + enum hal_rx_buf_return_buf_manager mgr = ab->hw_params->hal_params->rx_buf_rbm; req_entries = min(req_entries, rx_ring->bufs_max); @@ -307,42 +305,29 @@ int ath12k_dp_rx_bufs_replenish(struct ath12k_base *ab, int mac_id, if (dma_mapping_error(ab->dev, paddr)) goto fail_free_skb; - if (hw_cc) { - spin_lock_bh(&dp->rx_desc_lock); - - /* Get desc from free list and store in used list - * for cleanup purposes - * - * TODO: pass the removed descs rather than - * add/read to optimize - */ - rx_desc = list_first_entry_or_null(&dp->rx_desc_free_list, - struct ath12k_rx_desc_info, - list); - if (!rx_desc) { - spin_unlock_bh(&dp->rx_desc_lock); - goto fail_dma_unmap; - } - - rx_desc->skb = skb; - cookie = rx_desc->cookie; - list_del(&rx_desc->list); - list_add_tail(&rx_desc->list, &dp->rx_desc_used_list); + spin_lock_bh(&dp->rx_desc_lock); + /* Get desc from free list and store in used list + * for cleanup purposes + * + * TODO: pass the removed descs rather than + * add/read to optimize + */ + rx_desc = list_first_entry_or_null(&dp->rx_desc_free_list, + struct ath12k_rx_desc_info, + list); + if (!rx_desc) { spin_unlock_bh(&dp->rx_desc_lock); - } else { - spin_lock_bh(&rx_ring->idr_lock); - buf_id = idr_alloc(&rx_ring->bufs_idr, skb, 0, - rx_ring->bufs_max * 3, GFP_ATOMIC); - spin_unlock_bh(&rx_ring->idr_lock); - if (buf_id < 0) - goto fail_dma_unmap; - cookie = u32_encode_bits(mac_id, - DP_RXDMA_BUF_COOKIE_PDEV_ID) | - u32_encode_bits(buf_id, - DP_RXDMA_BUF_COOKIE_BUF_ID); + goto fail_dma_unmap; } + rx_desc->skb = skb; + cookie = rx_desc->cookie; + list_del(&rx_desc->list); + list_add_tail(&rx_desc->list, &dp->rx_desc_used_list); + + spin_unlock_bh(&dp->rx_desc_lock); + desc = ath12k_hal_srng_src_get_next_entry(ab, srng); if (!desc) goto fail_buf_unassign; @@ -361,17 +346,11 @@ int ath12k_dp_rx_bufs_replenish(struct ath12k_base *ab, int mac_id, return req_entries - num_remain; fail_buf_unassign: - if (hw_cc) { - spin_lock_bh(&dp->rx_desc_lock); - list_del(&rx_desc->list); - list_add_tail(&rx_desc->list, &dp->rx_desc_free_list); - rx_desc->skb = NULL; - spin_unlock_bh(&dp->rx_desc_lock); - } else { - spin_lock_bh(&rx_ring->idr_lock); - idr_remove(&rx_ring->bufs_idr, buf_id); - spin_unlock_bh(&rx_ring->idr_lock); - } + spin_lock_bh(&dp->rx_desc_lock); + list_del(&rx_desc->list); + list_add_tail(&rx_desc->list, &dp->rx_desc_free_list); + rx_desc->skb = NULL; + spin_unlock_bh(&dp->rx_desc_lock); fail_dma_unmap: dma_unmap_single(ab->dev, paddr, skb->len + skb_tailroom(skb), DMA_FROM_DEVICE); @@ -385,8 +364,8 @@ int ath12k_dp_rx_bufs_replenish(struct ath12k_base *ab, int mac_id, return req_entries - num_remain; } -static int ath12k_dp_rxdma_buf_ring_free(struct ath12k_base *ab, - struct dp_rxdma_ring *rx_ring) +static int ath12k_dp_rxdma_mon_buf_ring_free(struct ath12k_base *ab, + struct dp_rxdma_mon_ring *rx_ring) { struct sk_buff *skb; int buf_id; @@ -411,46 +390,49 @@ static int ath12k_dp_rxdma_buf_ring_free(struct ath12k_base *ab, static int ath12k_dp_rxdma_buf_free(struct ath12k_base *ab) { struct ath12k_dp *dp = &ab->dp; - struct dp_rxdma_ring *rx_ring = &dp->rx_refill_buf_ring; - ath12k_dp_rxdma_buf_ring_free(ab, rx_ring); + ath12k_dp_rxdma_mon_buf_ring_free(ab, &dp->rxdma_mon_buf_ring); + + ath12k_dp_rxdma_mon_buf_ring_free(ab, &dp->tx_mon_buf_ring); + + return 0; +} - rx_ring = &dp->rxdma_mon_buf_ring; - ath12k_dp_rxdma_buf_ring_free(ab, rx_ring); +static int ath12k_dp_rxdma_mon_ring_buf_setup(struct ath12k_base *ab, + struct dp_rxdma_mon_ring *rx_ring, + u32 ringtype) +{ + int num_entries; - rx_ring = &dp->tx_mon_buf_ring; - ath12k_dp_rxdma_buf_ring_free(ab, rx_ring); + num_entries = rx_ring->refill_buf_ring.size / + ath12k_hal_srng_get_entrysize(ab, ringtype); + + rx_ring->bufs_max = num_entries; + ath12k_dp_mon_buf_replenish(ab, rx_ring, num_entries); return 0; } static int ath12k_dp_rxdma_ring_buf_setup(struct ath12k_base *ab, - struct dp_rxdma_ring *rx_ring, - u32 ringtype) + struct dp_rxdma_ring *rx_ring) { int num_entries; num_entries = rx_ring->refill_buf_ring.size / - ath12k_hal_srng_get_entrysize(ab, ringtype); + ath12k_hal_srng_get_entrysize(ab, HAL_RXDMA_BUF); rx_ring->bufs_max = num_entries; - if ((ringtype == HAL_RXDMA_MONITOR_BUF) || (ringtype == HAL_TX_MONITOR_BUF)) - ath12k_dp_mon_buf_replenish(ab, rx_ring, num_entries); - else - ath12k_dp_rx_bufs_replenish(ab, 0, rx_ring, num_entries, - ab->hw_params->hal_params->rx_buf_rbm, - ringtype == HAL_RXDMA_BUF); + ath12k_dp_rx_bufs_replenish(ab, rx_ring, num_entries); + return 0; } static int ath12k_dp_rxdma_buf_setup(struct ath12k_base *ab) { struct ath12k_dp *dp = &ab->dp; - struct dp_rxdma_ring *rx_ring = &dp->rx_refill_buf_ring; int ret; - ret = ath12k_dp_rxdma_ring_buf_setup(ab, rx_ring, - HAL_RXDMA_BUF); + ret = ath12k_dp_rxdma_ring_buf_setup(ab, &dp->rx_refill_buf_ring); if (ret) { ath12k_warn(ab, "failed to setup HAL_RXDMA_BUF\n"); @@ -458,18 +440,18 @@ static int ath12k_dp_rxdma_buf_setup(struct ath12k_base *ab) } if (ab->hw_params->rxdma1_enable) { - rx_ring = &dp->rxdma_mon_buf_ring; - ret = ath12k_dp_rxdma_ring_buf_setup(ab, rx_ring, - HAL_RXDMA_MONITOR_BUF); + ret = ath12k_dp_rxdma_mon_ring_buf_setup(ab, + &dp->rxdma_mon_buf_ring, + HAL_RXDMA_MONITOR_BUF); if (ret) { ath12k_warn(ab, "failed to setup HAL_RXDMA_MONITOR_BUF\n"); return ret; } - rx_ring = &dp->tx_mon_buf_ring; - ret = ath12k_dp_rxdma_ring_buf_setup(ab, rx_ring, - HAL_TX_MONITOR_BUF); + ret = ath12k_dp_rxdma_mon_ring_buf_setup(ab, + &dp->tx_mon_buf_ring, + HAL_TX_MONITOR_BUF); if (ret) { ath12k_warn(ab, "failed to setup HAL_TX_MONITOR_BUF\n"); @@ -1783,6 +1765,7 @@ static int ath12k_dp_rx_msdu_coalesce(struct ath12k *ar, struct hal_rx_desc *ldesc; int space_extra, rem_len, buf_len; u32 hal_rx_desc_sz = ar->ab->hw_params->hal_desc_sz; + bool is_continuation; /* As the msdu is spread across multiple rx buffers, * find the offset to the start of msdu for computing @@ -1831,7 +1814,8 @@ static int ath12k_dp_rx_msdu_coalesce(struct ath12k *ar, rem_len = msdu_len - buf_first_len; while ((skb = __skb_dequeue(msdu_list)) != NULL && rem_len > 0) { rxcb = ATH12K_SKB_RXCB(skb); - if (rxcb->is_continuation) + is_continuation = rxcb->is_continuation; + if (is_continuation) buf_len = DP_RX_BUFFER_SIZE - hal_rx_desc_sz; else buf_len = rem_len; @@ -1849,7 +1833,7 @@ static int ath12k_dp_rx_msdu_coalesce(struct ath12k *ar, dev_kfree_skb_any(skb); rem_len -= buf_len; - if (!rxcb->is_continuation) + if (!is_continuation) break; } @@ -2745,9 +2729,7 @@ int ath12k_dp_rx_process(struct ath12k_base *ab, int ring_id, if (!total_msdu_reaped) goto exit; - /* TODO: Move to implicit BM? */ - ath12k_dp_rx_bufs_replenish(ab, 0, rx_ring, num_buffs_reaped, - ab->hw_params->hal_params->rx_buf_rbm, true); + ath12k_dp_rx_bufs_replenish(ab, rx_ring, num_buffs_reaped); ath12k_dp_rx_process_received_packets(ab, napi, &msdu_list, ring_id); @@ -2895,8 +2877,10 @@ static int ath12k_dp_rx_h_verify_tkip_mic(struct ath12k *ar, struct ath12k_peer RX_FLAG_IV_STRIPPED | RX_FLAG_DECRYPTED; skb_pull(msdu, hal_rx_desc_sz); - if (unlikely(!ath12k_dp_rx_check_nwifi_hdr_len_valid(ab, rx_desc, msdu))) + if (unlikely(!ath12k_dp_rx_check_nwifi_hdr_len_valid(ab, rx_desc, msdu))) { + dev_kfree_skb_any(msdu); return -EINVAL; + } ath12k_dp_rx_h_ppdu(ar, rx_desc, rxs); ath12k_dp_rx_h_undecap(ar, msdu, rx_desc, @@ -3429,6 +3413,50 @@ ath12k_dp_process_rx_err_buf(struct ath12k *ar, struct hal_reo_dest_ring *desc, return 0; } +static int ath12k_dp_h_msdu_buffer_type(struct ath12k_base *ab, + struct hal_reo_dest_ring *desc) +{ + struct ath12k_rx_desc_info *desc_info; + struct ath12k_skb_rxcb *rxcb; + struct sk_buff *msdu; + u64 desc_va; + + desc_va = (u64)le32_to_cpu(desc->buf_va_hi) << 32 | + le32_to_cpu(desc->buf_va_lo); + desc_info = (struct ath12k_rx_desc_info *)(uintptr_t)desc_va; + if (!desc_info) { + u32 cookie; + + cookie = le32_get_bits(desc->buf_addr_info.info1, + BUFFER_ADDR_INFO1_SW_COOKIE); + desc_info = ath12k_dp_get_rx_desc(ab, cookie); + if (!desc_info) { + ath12k_warn(ab, "Invalid cookie in manual descriptor retrieval: 0x%x\n", + cookie); + return -EINVAL; + } + } + + if (desc_info->magic != ATH12K_DP_RX_DESC_MAGIC) { + ath12k_warn(ab, "rx exception, magic check failed with value: %u\n", + desc_info->magic); + return -EINVAL; + } + + msdu = desc_info->skb; + desc_info->skb = NULL; + spin_lock_bh(&ab->dp.rx_desc_lock); + list_move_tail(&desc_info->list, &ab->dp.rx_desc_free_list); + spin_unlock_bh(&ab->dp.rx_desc_lock); + + rxcb = ATH12K_SKB_RXCB(msdu); + dma_unmap_single(ab->dev, rxcb->paddr, msdu->len + skb_tailroom(msdu), + DMA_FROM_DEVICE); + dev_kfree_skb_any(msdu); + + return 0; +} + int ath12k_dp_rx_process_err(struct ath12k_base *ab, struct napi_struct *napi, int budget) { @@ -3466,6 +3494,18 @@ int ath12k_dp_rx_process_err(struct ath12k_base *ab, struct napi_struct *napi, while (budget && (reo_desc = ath12k_hal_srng_dst_get_next_entry(ab, srng))) { ab->soc_stats.err_ring_pkts++; + + /* Below case is added to handle data packet from un-associated clients. + * As it is expected that AST lookup will fail for + * un-associated station's data packets. + */ + if (le32_get_bits(reo_desc->info0, HAL_REO_DEST_RING_INFO0_BUFFER_TYPE) == + HAL_REO_DEST_RING_BUFFER_TYPE_MSDU) { + if (!ath12k_dp_h_msdu_buffer_type(ab, reo_desc)) + tot_n_bufs_reaped++; + goto next_desc; + } + ret = ath12k_hal_desc_reo_parse_err(ab, reo_desc, &paddr, &desc_bank); if (ret) { @@ -3512,6 +3552,7 @@ int ath12k_dp_rx_process_err(struct ath12k_base *ab, struct napi_struct *napi, tot_n_bufs_reaped++; } +next_desc: if (tot_n_bufs_reaped >= quota) { tot_n_bufs_reaped = quota; goto exit; @@ -3527,8 +3568,7 @@ int ath12k_dp_rx_process_err(struct ath12k_base *ab, struct napi_struct *napi, rx_ring = &dp->rx_refill_buf_ring; - ath12k_dp_rx_bufs_replenish(ab, 0, rx_ring, tot_n_bufs_reaped, - ab->hw_params->hal_params->rx_buf_rbm, true); + ath12k_dp_rx_bufs_replenish(ab, rx_ring, tot_n_bufs_reaped); return tot_n_bufs_reaped; } @@ -3808,7 +3848,7 @@ int ath12k_dp_rx_process_wbm_err(struct ath12k_base *ab, continue; } - desc_info = (struct ath12k_rx_desc_info *)err_info.rx_desc; + desc_info = err_info.rx_desc; /* retry manual desc retrieval if hw cc is not done */ if (!desc_info) { @@ -3867,8 +3907,7 @@ int ath12k_dp_rx_process_wbm_err(struct ath12k_base *ab, if (!num_buffs_reaped) goto done; - ath12k_dp_rx_bufs_replenish(ab, 0, rx_ring, num_buffs_reaped, - ab->hw_params->hal_params->rx_buf_rbm, true); + ath12k_dp_rx_bufs_replenish(ab, rx_ring, num_buffs_reaped); rcu_read_lock(); for (i = 0; i < ab->num_radios; i++) { @@ -4149,9 +4188,6 @@ int ath12k_dp_rx_alloc(struct ath12k_base *ab) struct ath12k_dp *dp = &ab->dp; int i, ret; - idr_init(&dp->rx_refill_buf_ring.bufs_idr); - spin_lock_init(&dp->rx_refill_buf_ring.idr_lock); - idr_init(&dp->rxdma_mon_buf_ring.bufs_idr); spin_lock_init(&dp->rxdma_mon_buf_ring.idr_lock); diff --git a/drivers/net/wireless/ath/ath12k/dp_rx.h b/drivers/net/wireless/ath/ath12k/dp_rx.h index c955b5c859d1..05b3d5581dbe 100644 --- a/drivers/net/wireless/ath/ath12k/dp_rx.h +++ b/drivers/net/wireless/ath/ath12k/dp_rx.h @@ -1,7 +1,7 @@ /* SPDX-License-Identifier: BSD-3-Clause-Clear */ /* * Copyright (c) 2018-2021 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2022 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) 2021-2023 Qualcomm Innovation Center, Inc. All rights reserved. */ #ifndef ATH12K_DP_RX_H #define ATH12K_DP_RX_H @@ -116,11 +116,9 @@ int ath12k_dp_rx_process_err(struct ath12k_base *ab, struct napi_struct *napi, int ath12k_dp_rx_process(struct ath12k_base *ab, int mac_id, struct napi_struct *napi, int budget); -int ath12k_dp_rx_bufs_replenish(struct ath12k_base *ab, int mac_id, +int ath12k_dp_rx_bufs_replenish(struct ath12k_base *ab, struct dp_rxdma_ring *rx_ring, - int req_entries, - enum hal_rx_buf_return_buf_manager mgr, - bool hw_cc); + int req_entries); int ath12k_dp_rx_pdev_mon_attach(struct ath12k *ar); int ath12k_dp_rx_peer_frag_setup(struct ath12k *ar, const u8 *peer_mac, int vdev_id); diff --git a/drivers/net/wireless/ath/ath12k/dp_tx.c b/drivers/net/wireless/ath/ath12k/dp_tx.c index 474e0d4d406e..cd5cc64ed541 100644 --- a/drivers/net/wireless/ath/ath12k/dp_tx.c +++ b/drivers/net/wireless/ath/ath12k/dp_tx.c @@ -10,10 +10,9 @@ #include "hw.h" static enum hal_tcl_encap_type -ath12k_dp_tx_get_encap_type(struct ath12k_vif *arvif, struct sk_buff *skb) +ath12k_dp_tx_get_encap_type(struct ath12k_base *ab, struct sk_buff *skb) { struct ieee80211_tx_info *tx_info = IEEE80211_SKB_CB(skb); - struct ath12k_base *ab = arvif->ar->ab; if (test_bit(ATH12K_FLAG_RAW_MODE, &ab->dev_flags)) return HAL_TCL_ENCAP_TYPE_RAW; @@ -106,11 +105,10 @@ static struct ath12k_tx_desc_info *ath12k_dp_tx_assign_buffer(struct ath12k_dp * return desc; } -static void ath12k_hal_tx_cmd_ext_desc_setup(struct ath12k_base *ab, void *cmd, +static void ath12k_hal_tx_cmd_ext_desc_setup(struct ath12k_base *ab, + struct hal_tx_msdu_ext_desc *tcl_ext_cmd, struct hal_tx_info *ti) { - struct hal_tx_msdu_ext_desc *tcl_ext_cmd = (struct hal_tx_msdu_ext_desc *)cmd; - tcl_ext_cmd->info0 = le32_encode_bits(ti->paddr, HAL_TX_MSDU_EXT_INFO0_BUF_PTR_LO); tcl_ext_cmd->info1 = le32_encode_bits(0x0, @@ -196,7 +194,7 @@ int ath12k_dp_tx(struct ath12k *ar, struct ath12k_vif *arvif, msdu_ext_desc = true; } - ti.encap_type = ath12k_dp_tx_get_encap_type(arvif, skb); + ti.encap_type = ath12k_dp_tx_get_encap_type(ab, skb); ti.addr_search_flags = arvif->hal_addr_search_flags; ti.search_type = arvif->search_type; ti.type = HAL_TCL_DESC_TYPE_BUFFER; diff --git a/drivers/net/wireless/ath/ath12k/hal_rx.c b/drivers/net/wireless/ath/ath12k/hal_rx.c index 4d1b89cdffe1..bc72b9c966ce 100644 --- a/drivers/net/wireless/ath/ath12k/hal_rx.c +++ b/drivers/net/wireless/ath/ath12k/hal_rx.c @@ -1,7 +1,7 @@ // SPDX-License-Identifier: BSD-3-Clause-Clear /* * Copyright (c) 2018-2021 The Linux Foundation. All rights reserved. - * Copyright (c) 2021-2023 Qualcomm Innovation Center, Inc. All rights reserved. + * Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. */ #include "debug.h" @@ -320,7 +320,7 @@ int ath12k_hal_desc_reo_parse_err(struct ath12k_base *ab, { enum hal_reo_dest_ring_push_reason push_reason; enum hal_reo_dest_ring_error_code err_code; - u32 cookie, val; + u32 cookie; push_reason = le32_get_bits(desc->info0, HAL_REO_DEST_RING_INFO0_PUSH_REASON); @@ -335,12 +335,6 @@ int ath12k_hal_desc_reo_parse_err(struct ath12k_base *ab, return -EINVAL; } - val = le32_get_bits(desc->info0, HAL_REO_DEST_RING_INFO0_BUFFER_TYPE); - if (val != HAL_REO_DEST_RING_BUFFER_TYPE_LINK_DESC) { - ath12k_warn(ab, "expected buffer type link_desc"); - return -EINVAL; - } - ath12k_hal_rx_reo_ent_paddr_get(ab, &desc->buf_addr_info, paddr, &cookie); *desc_bank = u32_get_bits(cookie, DP_LINK_DESC_BANK_MASK); diff --git a/drivers/net/wireless/ath/ath12k/hal_rx.h b/drivers/net/wireless/ath/ath12k/hal_rx.h index fcfb6c819047..e99aa5b4b652 100644 --- a/drivers/net/wireless/ath/ath12k/hal_rx.h +++ b/drivers/net/wireless/ath/ath12k/hal_rx.h @@ -107,6 +107,9 @@ enum hal_rx_mon_status { HAL_RX_MON_STATUS_PPDU_NOT_DONE, HAL_RX_MON_STATUS_PPDU_DONE, HAL_RX_MON_STATUS_BUF_DONE, + HAL_RX_MON_STATUS_BUF_ADDR, + HAL_RX_MON_STATUS_MPDU_END, + HAL_RX_MON_STATUS_MSDU_END, }; #define HAL_RX_MAX_MPDU 256 @@ -435,12 +438,15 @@ enum hal_rx_ul_reception_type { HAL_RECEPTION_TYPE_FRAMELESS }; -#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RSSI_COMB GENMASK(15, 8) -#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_RSVD1_RECEPTION GENMASK(3, 0) +#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RECEPTION GENMASK(3, 0) +#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO0_RX_BW GENMASK(7, 5) +#define HAL_RX_PHYRX_RSSI_LEGACY_INFO_INFO1_RSSI_COMB GENMASK(15, 8) struct hal_rx_phyrx_rssi_legacy_info { - __le32 rsvd[35]; __le32 info0; + __le32 rsvd0[39]; + __le32 info1; + __le32 rsvd1; } __packed; #define HAL_RX_MPDU_START_INFO0_PPDU_ID GENMASK(31, 16) diff --git a/drivers/net/wireless/ath/ath12k/hw.c b/drivers/net/wireless/ath/ath12k/hw.c index 97ed179be228..bcd799b514ce 100644 --- a/drivers/net/wireless/ath/ath12k/hw.c +++ b/drivers/net/wireless/ath/ath12k/hw.c @@ -673,6 +673,8 @@ static const struct ath12k_hw_regs qcn9274_v1_regs = { /* REO status ring address */ .hal_reo_status_ring_base = 0x00000a84, + + .gcc_gcc_pcie_hot_rst = 0x1e38338, }; static const struct ath12k_hw_regs qcn9274_v2_regs = { @@ -753,6 +755,8 @@ static const struct ath12k_hw_regs qcn9274_v2_regs = { /* REO status ring address */ .hal_reo_status_ring_base = 0x00000aa0, + + .gcc_gcc_pcie_hot_rst = 0x1e38338, }; static const struct ath12k_hw_regs wcn7850_regs = { @@ -829,6 +833,8 @@ static const struct ath12k_hw_regs wcn7850_regs = { /* REO status ring address */ .hal_reo_status_ring_base = 0x00000a84, + + .gcc_gcc_pcie_hot_rst = 0x1e40304, }; static const struct ath12k_hw_hal_params ath12k_hw_hal_params_qcn9274 = { diff --git a/drivers/net/wireless/ath/ath12k/hw.h b/drivers/net/wireless/ath/ath12k/hw.h index 0b0b2a4f70f2..a56e5e504b9d 100644 --- a/drivers/net/wireless/ath/ath12k/hw.h +++ b/drivers/net/wireless/ath/ath12k/hw.h @@ -307,6 +307,8 @@ struct ath12k_hw_regs { u32 hal_reo_cmd_ring_base; u32 hal_reo_status_ring_base; + + u32 gcc_gcc_pcie_hot_rst; }; int ath12k_hw_init(struct ath12k_base *ab); diff --git a/drivers/net/wireless/ath/ath12k/mhi.c b/drivers/net/wireless/ath/ath12k/mhi.c index 380328fa2822..5020ae43de34 100644 --- a/drivers/net/wireless/ath/ath12k/mhi.c +++ b/drivers/net/wireless/ath/ath12k/mhi.c @@ -55,7 +55,6 @@ static const struct mhi_channel_config ath12k_mhi_channels_qcn9274[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, }, { .num = 21, @@ -69,7 +68,6 @@ static const struct mhi_channel_config ath12k_mhi_channels_qcn9274[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = true, }, }; @@ -148,7 +146,6 @@ static const struct mhi_channel_config ath12k_mhi_channels_wcn7850[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = false, }, { .num = 21, @@ -162,7 +159,6 @@ static const struct mhi_channel_config ath12k_mhi_channels_wcn7850[] = { .lpm_notify = false, .offload_channel = false, .doorbell_mode_switch = false, - .auto_queue = true, }, }; diff --git a/drivers/net/wireless/ath/ath12k/pci.c b/drivers/net/wireless/ath/ath12k/pci.c index 62be78de3300..8db438988ae2 100644 --- a/drivers/net/wireless/ath/ath12k/pci.c +++ b/drivers/net/wireless/ath/ath12k/pci.c @@ -280,10 +280,10 @@ static void ath12k_pci_enable_ltssm(struct ath12k_base *ab) ath12k_dbg(ab, ATH12K_DBG_PCI, "pci ltssm 0x%x\n", val); - val = ath12k_pci_read32(ab, GCC_GCC_PCIE_HOT_RST); + val = ath12k_pci_read32(ab, GCC_GCC_PCIE_HOT_RST(ab)); val |= GCC_GCC_PCIE_HOT_RST_VAL; - ath12k_pci_write32(ab, GCC_GCC_PCIE_HOT_RST, val); - val = ath12k_pci_read32(ab, GCC_GCC_PCIE_HOT_RST); + ath12k_pci_write32(ab, GCC_GCC_PCIE_HOT_RST(ab), val); + val = ath12k_pci_read32(ab, GCC_GCC_PCIE_HOT_RST(ab)); ath12k_dbg(ab, ATH12K_DBG_PCI, "pci pcie_hot_rst 0x%x\n", val); diff --git a/drivers/net/wireless/ath/ath12k/pci.h b/drivers/net/wireless/ath/ath12k/pci.h index 9a17a7dcdd6a..74dffd7b7549 100644 --- a/drivers/net/wireless/ath/ath12k/pci.h +++ b/drivers/net/wireless/ath/ath12k/pci.h @@ -28,7 +28,9 @@ #define PCIE_PCIE_PARF_LTSSM 0x1e081b0 #define PARM_LTSSM_VALUE 0x111 -#define GCC_GCC_PCIE_HOT_RST 0x1e38338 +#define GCC_GCC_PCIE_HOT_RST(ab) \ + ((ab)->hw_params->regs->gcc_gcc_pcie_hot_rst) + #define GCC_GCC_PCIE_HOT_RST_VAL 0x10 #define PCIE_PCIE_INT_ALL_CLEAR 0x1e08228 diff --git a/drivers/net/wireless/ath/ath12k/qmi.c b/drivers/net/wireless/ath/ath12k/qmi.c index 48d0043fcee9..8dc1c8e0c7b1 100644 --- a/drivers/net/wireless/ath/ath12k/qmi.c +++ b/drivers/net/wireless/ath/ath12k/qmi.c @@ -2518,9 +2518,10 @@ static void ath12k_qmi_m3_free(struct ath12k_base *ab) if (!m3_mem->vaddr) return; - dma_free_coherent(ab->dev, m3_mem->size, + dma_free_coherent(ab->dev, m3_mem->total_size, m3_mem->vaddr, m3_mem->paddr); m3_mem->vaddr = NULL; + m3_mem->total_size = 0; m3_mem->size = 0; } @@ -2542,10 +2543,8 @@ static int ath12k_qmi_m3_load(struct ath12k_base *ab) /* In recovery/resume cases, M3 buffer is not freed, try to reuse that */ if (m3_mem->vaddr) { - if (m3_mem->size >= fw->size) { - release_firmware(fw); - return 0; - } + if (m3_mem->total_size >= fw->size) + goto skip_m3_alloc; /* Old buffer is too small, free and reallocate */ ath12k_qmi_m3_free(ab); @@ -2561,6 +2560,9 @@ static int ath12k_qmi_m3_load(struct ath12k_base *ab) return -ENOMEM; } + m3_mem->total_size = fw->size; + +skip_m3_alloc: memcpy(m3_mem->vaddr, fw->data, fw->size); m3_mem->size = fw->size; release_firmware(fw); diff --git a/drivers/net/wireless/ath/ath12k/qmi.h b/drivers/net/wireless/ath/ath12k/qmi.h index a996c6d57aa7..62c7a526f111 100644 --- a/drivers/net/wireless/ath/ath12k/qmi.h +++ b/drivers/net/wireless/ath/ath12k/qmi.h @@ -119,6 +119,9 @@ struct target_info { }; struct m3_mem_region { + /* total memory allocated */ + u32 total_size; + /* actual memory being used */ u32 size; dma_addr_t paddr; void *vaddr; diff --git a/drivers/net/wireless/ath/ath12k/wmi.c b/drivers/net/wireless/ath/ath12k/wmi.c index 5b5001314d5e..183ac212bb03 100644 --- a/drivers/net/wireless/ath/ath12k/wmi.c +++ b/drivers/net/wireless/ath/ath12k/wmi.c @@ -160,6 +160,8 @@ static const struct ath12k_wmi_tlv_policy ath12k_wmi_tlv_policies[] = { .min_len = sizeof(struct wmi_probe_resp_tx_status_event) }, [WMI_TAG_VDEV_DELETE_RESP_EVENT] = { .min_len = sizeof(struct wmi_vdev_delete_resp_event) }, + [WMI_TAG_PDEV_CSA_SWITCH_COUNT_STATUS_EVENT] = { + .min_len = sizeof(struct ath12k_wmi_pdev_csa_event) }, }; static __le32 ath12k_wmi_tlv_hdr(u32 cmd, u32 len) @@ -376,6 +378,13 @@ ath12k_wmi_tlv_parse_alloc(struct ath12k_base *ab, const void *ptr, return tb; } +static u32 ath12k_wmi_tlv_data_len(const void *data) +{ + const struct wmi_tlv *tlv = (const struct wmi_tlv *)data - 1; + + return le32_get_bits(tlv->header, WMI_TLV_LEN); +} + static int ath12k_wmi_cmd_send_nowait(struct ath12k_wmi_pdev *wmi, struct sk_buff *skb, u32 cmd_id) { @@ -3830,14 +3839,16 @@ static int ath12k_wmi_mac_phy_caps_parse(struct ath12k_base *soc, if (svc_rdy_ext->n_mac_phy_caps >= svc_rdy_ext->tot_phy_id) return -ENOBUFS; - len = min_t(u16, len, sizeof(struct ath12k_wmi_mac_phy_caps_params)); if (!svc_rdy_ext->n_mac_phy_caps) { - svc_rdy_ext->mac_phy_caps = kzalloc((svc_rdy_ext->tot_phy_id) * len, - GFP_ATOMIC); + svc_rdy_ext->mac_phy_caps = + kcalloc(svc_rdy_ext->tot_phy_id, + sizeof(*svc_rdy_ext->mac_phy_caps), + GFP_ATOMIC); if (!svc_rdy_ext->mac_phy_caps) return -ENOMEM; } + len = min_t(u16, len, sizeof(struct ath12k_wmi_mac_phy_caps_params)); memcpy(svc_rdy_ext->mac_phy_caps + svc_rdy_ext->n_mac_phy_caps, ptr, len); svc_rdy_ext->n_mac_phy_caps++; return 0; @@ -6447,6 +6458,8 @@ ath12k_wmi_process_csa_switch_count_event(struct ath12k_base *ab, const struct ath12k_wmi_pdev_csa_event *ev, const u32 *vdev_ids) { + u32 vdev_ids_len = ath12k_wmi_tlv_data_len(vdev_ids); + u32 num_vdevs = le32_to_cpu(ev->num_vdevs); int i; struct ath12k_vif *arvif; @@ -6454,6 +6467,12 @@ ath12k_wmi_process_csa_switch_count_event(struct ath12k_base *ab, if (ev->current_switch_count) return; + if (num_vdevs > vdev_ids_len / sizeof(*vdev_ids)) { + ath12k_warn(ab, "csa switch count num_vdevs %u exceeds tlv array length %u\n", + num_vdevs, vdev_ids_len); + return; + } + rcu_read_lock(); for (i = 0; i < le32_to_cpu(ev->num_vdevs); i++) { arvif = ath12k_mac_get_arvif_by_vdev_id(ab, vdev_ids[i]); diff --git a/drivers/net/wireless/ath/wcn36xx/dxe.c b/drivers/net/wireless/ath/wcn36xx/dxe.c index 9013f056eecb..02e66839ab3a 100644 --- a/drivers/net/wireless/ath/wcn36xx/dxe.c +++ b/drivers/net/wireless/ath/wcn36xx/dxe.c @@ -1055,7 +1055,7 @@ void wcn36xx_dxe_deinit(struct wcn36xx *wcn) free_irq(wcn->tx_irq, wcn); free_irq(wcn->rx_irq, wcn); - del_timer(&wcn->tx_ack_timer); + timer_shutdown_sync(&wcn->tx_ack_timer); if (wcn->tx_ack_skb) { ieee80211_tx_status_irqsafe(wcn->hw, wcn->tx_ack_skb); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c index 17be8fcb5ab1..4013dd9b05f2 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/core.c @@ -1479,8 +1479,31 @@ void brcmf_detach(struct device *dev) brcmf_bus_change_state(bus_if, BRCMF_BUS_DOWN); /* make sure primary interface removed last */ for (i = BRCMF_MAX_IFS - 1; i > -1; i--) { - if (drvr->iflist[i]) - brcmf_remove_interface(drvr->iflist[i], false); + struct brcmf_if *ifp = drvr->iflist[i]; + + if (!ifp) + continue; + + if (ifp->ndev) { + brcmf_remove_interface(ifp, false); + continue; + } + + /* The P2P device interface has no netdev. Its removal can + * race with NL80211_CMD_DEL_INTERFACE issued by a user space + * process that is exiting (e.g. wpa_supplicant), which ends + * in brcmf_p2p_del_vif() under RTNL and the wiphy mutex. If + * both paths remove the interface, the wdev is unregistered + * twice and the vif is used after being freed. Take the same + * locks and re-read iflist, so only one path removes it. + */ + rtnl_lock(); + wiphy_lock(drvr->wiphy); + ifp = drvr->iflist[i]; + if (ifp) + brcmf_remove_interface(ifp, true); + wiphy_unlock(drvr->wiphy); + rtnl_unlock(); } brcmf_bus_stop(drvr->bus_if); diff --git a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c index 33f27c83c33e..c2c713f2dae6 100644 --- a/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c +++ b/drivers/net/wireless/broadcom/brcm80211/brcmfmac/p2p.c @@ -1281,6 +1281,9 @@ static s32 brcmf_p2p_abort_action_frame(struct brcmf_cfg80211_info *cfg) brcmf_dbg(TRACE, "Enter\n"); vif = p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif; + if (!vif) + vif = p2p->bss_idx[P2PAPI_BSSCFG_PRIMARY].vif; + err = brcmf_fil_bsscfg_data_set(vif->ifp, "actframe_abort", &int_val, sizeof(s32)); if (err) @@ -1817,6 +1820,7 @@ bool brcmf_p2p_send_action_frame(struct brcmf_if *ifp, /* validate channel and p2p ies */ if (config_af_params.search_channel && IS_P2P_SOCIAL_CHANNEL(le32_to_cpu(af_params->channel)) && + p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif && p2p->bss_idx[P2PAPI_BSSCFG_DEVICE].vif->saved_ie.probe_req_ie_len) { afx_hdl = &p2p->afx_hdl; afx_hdl->peer_listen_chan = le32_to_cpu(af_params->channel); diff --git a/drivers/net/wireless/intel/iwlegacy/3945-mac.c b/drivers/net/wireless/intel/iwlegacy/3945-mac.c index 5d61cebbdc40..0fd19150782f 100644 --- a/drivers/net/wireless/intel/iwlegacy/3945-mac.c +++ b/drivers/net/wireless/intel/iwlegacy/3945-mac.c @@ -3840,6 +3840,7 @@ il3945_pci_remove(struct pci_dev *pdev) il_free_channel_map(il); il_free_geos(il); + il_eeprom_free(il); kfree(il->scan_cmd); dev_kfree_skb(il->beacon_skb); ieee80211_free_hw(il->hw); diff --git a/drivers/net/wireless/intel/iwlwifi/dvm/main.c b/drivers/net/wireless/intel/iwlwifi/dvm/main.c index b490a88b97ca..73b1f5a9bde6 100644 --- a/drivers/net/wireless/intel/iwlwifi/dvm/main.c +++ b/drivers/net/wireless/intel/iwlwifi/dvm/main.c @@ -1245,7 +1245,7 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans, STATISTICS_NOTIFICATION, REPLY_TX, }; - int i; + int i, err; /************************ * 1. Allocating HW data @@ -1253,6 +1253,7 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans, hw = iwl_alloc_all(); if (!hw) { pr_err("%s: Cannot allocate network device\n", trans->name); + err = -ENOMEM; goto out; } @@ -1303,8 +1304,10 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans, break; } - if (WARN_ON(!priv->lib)) + if (WARN_ON(!priv->lib)) { + err = -ENODEV; goto out_free_hw; + } /* * Populate the state variables that the transport layer needs @@ -1383,14 +1386,16 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans, IWL_INFO(priv, "Detected %s, REV=0x%X\n", priv->trans->name, priv->trans->hw_rev); - if (iwl_trans_start_hw(priv->trans)) - goto out_free_hw; + err = iwl_trans_start_hw(priv->trans); + if (err) + goto out_leave_trans; /* Read the EEPROM */ - if (iwl_read_eeprom(priv->trans, &priv->eeprom_blob, - &priv->eeprom_blob_size)) { + err = iwl_read_eeprom(priv->trans, &priv->eeprom_blob, + &priv->eeprom_blob_size); + if (err) { IWL_ERR(priv, "Unable to init EEPROM\n"); - goto out_free_hw; + goto out_leave_trans; } /* Reset chip to save power until we load uCode during "up". */ @@ -1399,13 +1404,17 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans, priv->nvm_data = iwl_parse_eeprom_data(priv->trans, priv->cfg, priv->eeprom_blob, priv->eeprom_blob_size); - if (!priv->nvm_data) + if (!priv->nvm_data) { + err = -ENOMEM; goto out_free_eeprom_blob; + } - if (iwl_nvm_check_version(priv->nvm_data, priv->trans)) + err = iwl_nvm_check_version(priv->nvm_data, priv->trans); + if (err) goto out_free_eeprom; - if (iwl_eeprom_init_hw_params(priv)) + err = iwl_eeprom_init_hw_params(priv); + if (err) goto out_free_eeprom; /* extract MAC Address */ @@ -1452,7 +1461,8 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans, atomic_set(&priv->queue_stop_count[i], 0); } - if (iwl_init_drv(priv)) + err = iwl_init_drv(priv); + if (err) goto out_free_eeprom; /* At this point both hw and priv are initialized. */ @@ -1460,7 +1470,8 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans, /******************** * 6. Setup services ********************/ - if (iwl_setup_deferred_work(priv)) + err = iwl_setup_deferred_work(priv); + if (err) goto out_uninit_drv; iwl_setup_rx_handlers(priv); @@ -1488,7 +1499,8 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans, * * 7. Setup and register with mac80211 and debugfs **************************************************/ - if (iwlagn_mac_setup_register(priv, &fw->ucode_capa)) + err = iwlagn_mac_setup_register(priv, &fw->ucode_capa); + if (err) goto out_destroy_workqueue; iwl_dbgfs_register(priv, dbgfs_dir); @@ -1502,15 +1514,16 @@ static struct iwl_op_mode *iwl_op_mode_dvm_start(struct iwl_trans *trans, priv->workqueue = NULL; out_uninit_drv: iwl_uninit_drv(priv); -out_free_eeprom_blob: - kfree(priv->eeprom_blob); out_free_eeprom: kfree(priv->nvm_data); +out_free_eeprom_blob: + kfree(priv->eeprom_blob); +out_leave_trans: + iwl_trans_op_mode_leave(priv->trans); out_free_hw: ieee80211_free_hw(priv->hw); out: - op_mode = NULL; - return op_mode; + return ERR_PTR(err); } static void iwl_op_mode_dvm_stop(struct iwl_op_mode *op_mode) diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/commands.h b/drivers/net/wireless/intel/iwlwifi/fw/api/commands.h index 13cb0d53a1a3..196f88564f54 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/commands.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/commands.h @@ -141,8 +141,8 @@ enum iwl_legacy_cmds { /** * @TX_CMD: uses &struct iwl_tx_cmd or &struct iwl_tx_cmd_gen2 or * &struct iwl_tx_cmd_gen3, - * response in &struct iwl_mvm_tx_resp or - * &struct iwl_mvm_tx_resp_v3 + * response in &struct iwl_tx_resp or + * &struct iwl_tx_resp_v3 */ TX_CMD = 0x1c, diff --git a/drivers/net/wireless/intel/iwlwifi/fw/api/tx.h b/drivers/net/wireless/intel/iwlwifi/fw/api/tx.h index 842360b1e995..b98a9d19ba00 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/api/tx.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/api/tx.h @@ -484,7 +484,7 @@ struct agg_tx_status { #define IWL_MVM_TX_RES_GET_RA(_ra_tid) ((_ra_tid) >> 4) /** - * struct iwl_mvm_tx_resp_v3 - notifies that fw is TXing a packet + * struct iwl_tx_resp_v3 - notifies that fw is TXing a packet * ( REPLY_TX = 0x1c ) * @frame_count: 1 no aggregation, >1 aggregation * @bt_kill_count: num of times blocked by bluetooth (unused for agg) @@ -515,7 +515,7 @@ struct agg_tx_status { * After the array of statuses comes the SSN of the SCD. Look at * %iwl_mvm_get_scd_ssn for more details. */ -struct iwl_mvm_tx_resp_v3 { +struct iwl_tx_resp_v3 { u8 frame_count; u8 bt_kill_count; u8 failure_rts; @@ -541,7 +541,7 @@ struct iwl_mvm_tx_resp_v3 { } __packed; /* TX_RSP_API_S_VER_3 */ /** - * struct iwl_mvm_tx_resp - notifies that fw is TXing a packet + * struct iwl_tx_resp - notifies that fw is TXing a packet * ( REPLY_TX = 0x1c ) * @frame_count: 1 no aggregation, >1 aggregation * @bt_kill_count: num of times blocked by bluetooth (unused for agg) @@ -573,7 +573,7 @@ struct iwl_mvm_tx_resp_v3 { * After the array of statuses comes the SSN of the SCD. Look at * %iwl_mvm_get_scd_ssn for more details. */ -struct iwl_mvm_tx_resp { +struct iwl_tx_resp { u8 frame_count; u8 bt_kill_count; u8 failure_rts; @@ -814,7 +814,7 @@ struct iwl_mac_beacon_cmd { */ struct iwl_beacon_notif { - struct iwl_mvm_tx_resp beacon_notify_hdr; + struct iwl_tx_resp beacon_notify_hdr; __le64 tsf; __le32 ibss_mgr_status; } __packed; @@ -827,7 +827,7 @@ struct iwl_beacon_notif { * @gp2: last beacon time in gp2 */ struct iwl_extended_beacon_notif_v5 { - struct iwl_mvm_tx_resp beacon_notify_hdr; + struct iwl_tx_resp beacon_notify_hdr; __le64 tsf; __le32 ibss_mgr_status; __le32 gp2; diff --git a/drivers/net/wireless/intel/iwlwifi/fw/dbg.h b/drivers/net/wireless/intel/iwlwifi/fw/dbg.h index 4227fbd2b977..27568a07ff06 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/dbg.h +++ b/drivers/net/wireless/intel/iwlwifi/fw/dbg.h @@ -331,14 +331,14 @@ void iwl_send_dbg_dump_complete_cmd(struct iwl_fw_runtime *fwrt, u32 timepoint_data); #define IWL_FW_CHECK_FAILED(_obj, _fmt, ...) \ - IWL_ERR_LIMIT(_obj, _fmt, __VA_ARGS__) + IWL_ERR_LIMIT(_obj, _fmt, ##__VA_ARGS__) #define IWL_FW_CHECK(_obj, _cond, _fmt, ...) \ ({ \ bool __cond = (_cond); \ \ if (unlikely(__cond)) \ - IWL_FW_CHECK_FAILED(_obj, _fmt, __VA_ARGS__); \ + IWL_FW_CHECK_FAILED(_obj, _fmt, ##__VA_ARGS__); \ \ unlikely(__cond); \ }) diff --git a/drivers/net/wireless/intel/iwlwifi/fw/smem.c b/drivers/net/wireless/intel/iwlwifi/fw/smem.c index 3f1272014daf..e2fc2f5de442 100644 --- a/drivers/net/wireless/intel/iwlwifi/fw/smem.c +++ b/drivers/net/wireless/intel/iwlwifi/fw/smem.c @@ -6,6 +6,7 @@ */ #include "iwl-drv.h" #include "runtime.h" +#include "dbg.h" #include "fw/api/commands.h" static void iwl_parse_shared_mem_22000(struct iwl_fw_runtime *fwrt, @@ -17,7 +18,9 @@ static void iwl_parse_shared_mem_22000(struct iwl_fw_runtime *fwrt, u8 api_ver = iwl_fw_lookup_notif_ver(fwrt->fw, SYSTEM_GROUP, SHARED_MEM_CFG_CMD, 0); - if (WARN_ON(lmac_num > ARRAY_SIZE(mem_cfg->lmac_smem))) + /* Note: notification has 3 entries, but we only expect 2 */ + if (IWL_FW_CHECK(fwrt, lmac_num > ARRAY_SIZE(fwrt->smem_cfg.lmac), + "FW advertises %d LMACs\n", lmac_num)) return; fwrt->smem_cfg.num_lmacs = lmac_num; @@ -26,7 +29,8 @@ static void iwl_parse_shared_mem_22000(struct iwl_fw_runtime *fwrt, fwrt->smem_cfg.rxfifo2_size = le32_to_cpu(mem_cfg->rxfifo2_size); if (api_ver >= 4 && - !WARN_ON_ONCE(iwl_rx_packet_payload_len(pkt) < sizeof(*mem_cfg))) { + !IWL_FW_CHECK(fwrt, iwl_rx_packet_payload_len(pkt) < sizeof(*mem_cfg), + "bad shared mem notification size\n")) { fwrt->smem_cfg.rxfifo2_control_size = le32_to_cpu(mem_cfg->rxfifo2_control_size); } diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-drv.c b/drivers/net/wireless/intel/iwlwifi/iwl-drv.c index 36e8d0e8c91b..5c5a664dddc6 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-drv.c +++ b/drivers/net/wireless/intel/iwlwifi/iwl-drv.c @@ -1438,7 +1438,7 @@ _iwl_op_mode_start(struct iwl_drv *drv, struct iwlwifi_opmode_table *op) op_mode = ops->start(drv->trans, drv->trans->cfg, &drv->fw, dbgfs_dir); - if (op_mode) + if (!IS_ERR(op_mode)) return op_mode; IWL_ERR(drv, "retry init count %d\n", retry); diff --git a/drivers/net/wireless/intel/iwlwifi/iwl-trans.h b/drivers/net/wireless/intel/iwlwifi/iwl-trans.h index e8f48cb8d2da..8c0762d2dc00 100644 --- a/drivers/net/wireless/intel/iwlwifi/iwl-trans.h +++ b/drivers/net/wireless/intel/iwlwifi/iwl-trans.h @@ -1276,6 +1276,9 @@ static inline void iwl_trans_reclaim(struct iwl_trans *trans, int queue, int ssn, struct sk_buff_head *skbs, bool is_flush) { + if (unlikely(test_bit(STATUS_FW_ERROR, &trans->status))) + return; + if (WARN_ON_ONCE(trans->state != IWL_TRANS_FW_ALIVE)) { IWL_ERR(trans, "%s bad state = %d\n", __func__, trans->state); return; diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/binding.c b/drivers/net/wireless/intel/iwlwifi/mvm/binding.c index 458b97930059..9d1f8e7657e7 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/binding.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/binding.c @@ -2,7 +2,7 @@ /* * Copyright (C) 2012-2014, 2020 Intel Corporation * Copyright (C) 2016 Intel Deutschland GmbH - * Copyright (C) 2022 Intel Corporation + * Copyright (C) 2022, 2026 Intel Corporation */ #include #include "fw-api.h" @@ -76,6 +76,9 @@ static void iwl_mvm_iface_iterator(void *_data, u8 *mac, if (vif == data->ignore_vif) return; + if (vif->type == NL80211_IFTYPE_P2P_DEVICE && !mvmvif->p2p_in_binding) + return; + if (mvmvif->deflink.phy_ctxt != data->phyctxt) return; diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c b/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c index 60472c89fca3..b086300bbafe 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mac-ctxt.c @@ -1496,7 +1496,7 @@ void iwl_mvm_rx_beacon_notif(struct iwl_mvm *mvm, mvm->ap_last_beacon_gp2 = le32_to_cpu(beacon->gp2); if (!iwl_mvm_is_short_beacon_notif_supported(mvm)) { - struct iwl_mvm_tx_resp *beacon_notify_hdr = + struct iwl_tx_resp *beacon_notify_hdr = &beacon_v5->beacon_notify_hdr; if (unlikely(pkt_len < sizeof(*beacon_v5))) diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c index 4ec2ea44f488..6d5cfcfb7a18 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mac80211.c @@ -1022,6 +1022,8 @@ static void iwl_mvm_cleanup_iterator(void *data, u8 *mac, iwl_mvm_te_clear_data(mvm, &mvmvif->time_event_data); spin_unlock_bh(&mvm->time_event_lock); + mvmvif->p2p_in_binding = false; + memset(&mvmvif->bf_data, 0, sizeof(mvmvif->bf_data)); mvmvif->ap_sta = NULL; @@ -4582,6 +4584,7 @@ static int iwl_mvm_add_aux_sta_for_hs20(struct iwl_mvm *mvm, u32 lmac_id) static int iwl_mvm_roc_link(struct iwl_mvm *mvm, struct ieee80211_vif *vif) { + struct iwl_mvm_vif *mvmvif = iwl_mvm_vif_from_mac80211(vif); int ret; lockdep_assert_held(&mvm->mutex); @@ -4590,10 +4593,18 @@ static int iwl_mvm_roc_link(struct iwl_mvm *mvm, struct ieee80211_vif *vif) if (WARN(ret, "Failed binding P2P_DEVICE\n")) return ret; + mvmvif->p2p_in_binding = true; + /* The station and queue allocation must be done only after the binding * is done, as otherwise the FW might incorrectly configure its state. */ - return iwl_mvm_add_p2p_bcast_sta(mvm, vif); + ret = iwl_mvm_add_p2p_bcast_sta(mvm, vif); + if (ret) { + iwl_mvm_binding_remove_vif(mvm, vif); + mvmvif->p2p_in_binding = false; + } + + return ret; } static int iwl_mvm_roc(struct ieee80211_hw *hw, diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h b/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h index bace9d01fd58..3509eabe0fa2 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h +++ b/drivers/net/wireless/intel/iwlwifi/mvm/mvm.h @@ -363,6 +363,8 @@ struct iwl_mvm_vif_link_info { * @csa_countdown: indicates that CSA countdown may be started * @csa_failed: CSA failed to schedule time event, report an error later * @csa_bcn_pending: indicates that we are waiting for a beacon on a new channel + * @p2p_in_binding: indicates that this P2P-Device interface should be + * added to the binding, i.e. is running ROC right now * @features: hw features active for this vif */ struct iwl_mvm_vif { @@ -437,6 +439,7 @@ struct iwl_mvm_vif { struct iwl_mvm_time_event_data time_event_data; struct iwl_mvm_time_event_data hs_time_event_data; + bool p2p_in_binding; /* TCP Checksum Offload */ netdev_features_t features; @@ -1526,9 +1529,9 @@ static inline struct agg_tx_status * iwl_mvm_get_agg_status(struct iwl_mvm *mvm, void *tx_resp) { if (iwl_mvm_has_new_tx_api(mvm)) - return &((struct iwl_mvm_tx_resp *)tx_resp)->status; + return &((struct iwl_tx_resp *)tx_resp)->status; else - return ((struct iwl_mvm_tx_resp_v3 *)tx_resp)->status; + return ((struct iwl_tx_resp_v3 *)tx_resp)->status; } static inline bool iwl_mvm_is_tt_in_fw(struct iwl_mvm *mvm) diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c index 7ed684d638bd..fdebe570ab8c 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/ops.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/ops.c @@ -309,7 +309,7 @@ struct iwl_rx_handlers { */ static const struct iwl_rx_handlers iwl_mvm_rx_handlers[] = { RX_HANDLER(TX_CMD, iwl_mvm_rx_tx_cmd, RX_HANDLER_SYNC, - struct iwl_mvm_tx_resp), + struct iwl_tx_resp), RX_HANDLER(BA_NOTIF, iwl_mvm_rx_ba_notif, RX_HANDLER_SYNC, struct iwl_mvm_ba_notif), @@ -1112,6 +1112,7 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, size_t scan_size; u32 min_backoff; struct iwl_mvm_csme_conn_info *csme_conn_info __maybe_unused; + int err; /* * We use IWL_MVM_STATION_COUNT_MAX to check the validity of the station @@ -1129,7 +1130,7 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, iwl_mvm_has_mld_api(fw) ? &iwl_mvm_mld_hw_ops : &iwl_mvm_hw_ops); if (!hw) - return NULL; + return ERR_PTR(-ENOMEM); if (trans->trans_cfg->device_family >= IWL_DEVICE_FAMILY_BZ) max_agg = IEEE80211_MAX_AMPDU_BUF_EHT; @@ -1173,8 +1174,10 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, trans->rx_mpdu_cmd_hdr_size = sizeof(struct iwl_rx_mpdu_res_start); - if (WARN_ON(trans->num_rx_queues > 1)) + if (WARN_ON(trans->num_rx_queues > 1)) { + err = -EINVAL; goto out_free; + } } mvm->fw_restart = iwlwifi_mod_params.fw_restart ? -1 : 0; @@ -1244,8 +1247,10 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, iwl_fw_lookup_notif_ver(mvm->fw, LOCATION_GROUP, TOF_RANGE_RESPONSE_NOTIF, 5); /* we only support up to version 9 */ - if (WARN_ON_ONCE(mvm->cmd_ver.range_resp > 9)) + if (WARN_ON_ONCE(mvm->cmd_ver.range_resp > 9)) { + err = -EINVAL; goto out_free; + } /* * Populate the state variables that the transport layer needs @@ -1329,6 +1334,7 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, mvm->phy_db = iwl_phy_db_init(trans); if (!mvm->phy_db) { IWL_ERR(mvm, "Cannot init phy_db\n"); + err = -ENOMEM; goto out_free; } @@ -1344,8 +1350,10 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, scan_size = iwl_mvm_scan_size(mvm); mvm->scan_cmd = kmalloc(scan_size, GFP_KERNEL); - if (!mvm->scan_cmd) + if (!mvm->scan_cmd) { + err = -ENOMEM; goto out_free; + } mvm->scan_cmd_size = scan_size; /* invalidate ids to prevent accidental removal of sta_id 0 */ @@ -1374,7 +1382,8 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, iwl_mvm_mei_scan_filter_init(&mvm->mei_scan_filter); - if (iwl_mvm_start_get_nvm(mvm)) { + err = iwl_mvm_start_get_nvm(mvm); + if (err) { /* * Getting NVM failed while CSME is the owner, but we are * registered to MEI, we'll get the NVM later when it'll be @@ -1387,7 +1396,8 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, } - if (iwl_mvm_start_post_nvm(mvm)) + err = iwl_mvm_start_post_nvm(mvm); + if (err) goto out_thermal_exit; return op_mode; @@ -1409,7 +1419,7 @@ iwl_op_mode_mvm_start(struct iwl_trans *trans, const struct iwl_cfg *cfg, iwl_trans_op_mode_leave(trans); ieee80211_free_hw(mvm->hw); - return NULL; + return ERR_PTR(err); } void iwl_mvm_stop_device(struct iwl_mvm *mvm) diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c b/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c index 158266719ffd..752414f16d63 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/time-event.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause /* - * Copyright (C) 2012-2014, 2018-2023 Intel Corporation + * Copyright (C) 2012-2014, 2018-2023, 2026 Intel Corporation * Copyright (C) 2013-2015 Intel Mobile Communications GmbH * Copyright (C) 2017 Intel Deutschland GmbH */ @@ -94,6 +94,7 @@ void iwl_mvm_roc_done_wk(struct work_struct *wk) } else { iwl_mvm_rm_p2p_bcast_sta(mvm, vif); iwl_mvm_binding_remove_vif(mvm, vif); + mvmvif->p2p_in_binding = false; } /* Do not remove the PHY context as removing and adding diff --git a/drivers/net/wireless/intel/iwlwifi/mvm/tx.c b/drivers/net/wireless/intel/iwlwifi/mvm/tx.c index 2594e8a977e9..824f54c354a7 100644 --- a/drivers/net/wireless/intel/iwlwifi/mvm/tx.c +++ b/drivers/net/wireless/intel/iwlwifi/mvm/tx.c @@ -1622,7 +1622,7 @@ static void iwl_mvm_tx_status_check_trigger(struct iwl_mvm *mvm, * For 22000-series and lower, this is just 12 bits. For later, 16 bits. */ static inline u32 iwl_mvm_get_scd_ssn(struct iwl_mvm *mvm, - struct iwl_mvm_tx_resp *tx_resp) + struct iwl_tx_resp *tx_resp) { u32 val = le32_to_cpup((__le32 *)iwl_mvm_get_agg_status(mvm, tx_resp) + tx_resp->frame_count); @@ -1632,14 +1632,25 @@ static inline u32 iwl_mvm_get_scd_ssn(struct iwl_mvm *mvm, return val & 0xFFF; } +static inline size_t iwl_mvm_tx_resp_min_len(struct iwl_mvm *mvm, + struct iwl_tx_resp *tx_resp) +{ + struct agg_tx_status *agg_status = + iwl_mvm_get_agg_status(mvm, tx_resp); + + /* The aggregate response ends with a trailing SCD SSN __le32 word. */ + return (u8 *)(agg_status + tx_resp->frame_count) - (u8 *)tx_resp + + sizeof(__le32); +} + static void iwl_mvm_rx_tx_cmd_single(struct iwl_mvm *mvm, struct iwl_rx_packet *pkt) { struct ieee80211_sta *sta; u16 sequence = le16_to_cpu(pkt->hdr.sequence); int txq_id = SEQ_TO_QUEUE(sequence); - /* struct iwl_mvm_tx_resp_v3 is almost the same */ - struct iwl_mvm_tx_resp *tx_resp = (void *)pkt->data; + /* struct iwl_tx_resp_v3 is almost the same */ + struct iwl_tx_resp *tx_resp = (void *)pkt->data; int sta_id = IWL_MVM_TX_RES_GET_RA(tx_resp->ra_tid); int tid = IWL_MVM_TX_RES_GET_TID(tx_resp->ra_tid); struct agg_tx_status *agg_status = @@ -1895,7 +1906,7 @@ static const char *iwl_get_agg_tx_status(u16 status) static void iwl_mvm_rx_tx_cmd_agg_dbg(struct iwl_mvm *mvm, struct iwl_rx_packet *pkt) { - struct iwl_mvm_tx_resp *tx_resp = (void *)pkt->data; + struct iwl_tx_resp *tx_resp = (void *)pkt->data; struct agg_tx_status *frame_status = iwl_mvm_get_agg_status(mvm, tx_resp); int i; @@ -1929,7 +1940,7 @@ static void iwl_mvm_rx_tx_cmd_agg_dbg(struct iwl_mvm *mvm, static void iwl_mvm_rx_tx_cmd_agg(struct iwl_mvm *mvm, struct iwl_rx_packet *pkt) { - struct iwl_mvm_tx_resp *tx_resp = (void *)pkt->data; + struct iwl_tx_resp *tx_resp = (void *)pkt->data; int sta_id = IWL_MVM_TX_RES_GET_RA(tx_resp->ra_tid); int tid = IWL_MVM_TX_RES_GET_TID(tx_resp->ra_tid); u16 sequence = le16_to_cpu(pkt->hdr.sequence); @@ -1937,6 +1948,9 @@ static void iwl_mvm_rx_tx_cmd_agg(struct iwl_mvm *mvm, int queue = SEQ_TO_QUEUE(sequence); struct ieee80211_sta *sta; + if (WARN_ON_ONCE(iwl_mvm_has_new_tx_api(mvm))) + return; + if (WARN_ON_ONCE(queue < IWL_MVM_DQA_MIN_DATA_QUEUE && (queue != IWL_MVM_DQA_BSS_CLIENT_QUEUE))) return; @@ -1970,7 +1984,27 @@ static void iwl_mvm_rx_tx_cmd_agg(struct iwl_mvm *mvm, void iwl_mvm_rx_tx_cmd(struct iwl_mvm *mvm, struct iwl_rx_cmd_buffer *rxb) { struct iwl_rx_packet *pkt = rxb_addr(rxb); - struct iwl_mvm_tx_resp *tx_resp = (void *)pkt->data; + struct iwl_tx_resp *tx_resp = (void *)pkt->data; + size_t min_len; + + if (IWL_FW_CHECK(mvm, !tx_resp->frame_count, + "invalid TX_CMD frame_count %u\n", + tx_resp->frame_count)) + return; + + if (IWL_FW_CHECK(mvm, + iwl_mvm_has_new_tx_api(mvm) && + tx_resp->frame_count != 1, + "invalid TX_CMD frame_count %u for new TX API\n", + tx_resp->frame_count)) + return; + + min_len = iwl_mvm_tx_resp_min_len(mvm, tx_resp); + if (IWL_FW_CHECK(mvm, iwl_rx_packet_payload_len(pkt) < min_len, + "invalid TX_CMD len %u (frame_count %u, min %zu)\n", + iwl_rx_packet_payload_len(pkt), tx_resp->frame_count, + min_len)) + return; if (tx_resp->frame_count == 1) iwl_mvm_rx_tx_cmd_single(mvm, pkt); diff --git a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c index be9b5a19e2a7..618d8eac37d6 100644 --- a/drivers/net/wireless/intel/iwlwifi/pcie/rx.c +++ b/drivers/net/wireless/intel/iwlwifi/pcie/rx.c @@ -1,6 +1,6 @@ // SPDX-License-Identifier: GPL-2.0 OR BSD-3-Clause /* - * Copyright (C) 2003-2014, 2018-2023 Intel Corporation + * Copyright (C) 2003-2014, 2018-2023, 2026 Intel Corporation * Copyright (C) 2013-2015 Intel Mobile Communications GmbH * Copyright (C) 2016-2017 Intel Deutschland GmbH */ @@ -1236,11 +1236,16 @@ void iwl_pcie_rx_free(struct iwl_trans *trans) } } kfree(trans_pcie->rx_pool); + trans_pcie->rx_pool = NULL; kfree(trans_pcie->global_table); + trans_pcie->global_table = NULL; kfree(trans_pcie->rxq); + trans_pcie->rxq = NULL; - if (trans_pcie->alloc_page) + if (trans_pcie->alloc_page) { __free_pages(trans_pcie->alloc_page, trans_pcie->rx_page_order); + trans_pcie->alloc_page = NULL; + } } static void iwl_pcie_rx_move_to_allocator(struct iwl_rxq *rxq, diff --git a/drivers/net/wireless/marvell/libertas/tx.c b/drivers/net/wireless/marvell/libertas/tx.c index 13d08022e414..27304a98787d 100644 --- a/drivers/net/wireless/marvell/libertas/tx.c +++ b/drivers/net/wireless/marvell/libertas/tx.c @@ -117,13 +117,6 @@ netdev_tx_t lbs_hard_start_xmit(struct sk_buff *skb, struct net_device *dev) if (priv->wdev->iftype == NL80211_IFTYPE_MONITOR) { struct tx_radiotap_hdr *rtap_hdr = (void *)skb->data; - if (skb->len < sizeof(*rtap_hdr) + 4 + ETH_ALEN) { - lbs_deb_tx("tx err: short monitor frame %u\n", skb->len); - dev->stats.tx_dropped++; - dev->stats.tx_errors++; - goto free; - } - /* set txpd fields from the radiotap header */ txpd->tx_control = cpu_to_le32(convert_radiotap_rate_to_mv(rtap_hdr->rate)); diff --git a/drivers/net/wireless/mediatek/mt76/mac80211.c b/drivers/net/wireless/mediatek/mt76/mac80211.c index 61543ab7c256..3f09bf58130d 100644 --- a/drivers/net/wireless/mediatek/mt76/mac80211.c +++ b/drivers/net/wireless/mediatek/mt76/mac80211.c @@ -763,6 +763,43 @@ void mt76_free_device(struct mt76_dev *dev) } EXPORT_SYMBOL_GPL(mt76_free_device); +static void mt76_reset_phy(struct mt76_phy *phy) +{ + if (!phy) + return; + + INIT_LIST_HEAD(&phy->tx_list); +} + +void mt76_reset_device(struct mt76_dev *dev) +{ + int i; + + rcu_read_lock(); + for (i = 0; i < ARRAY_SIZE(dev->wcid); i++) { + struct mt76_wcid *wcid; + + wcid = rcu_dereference(dev->wcid[i]); + if (!wcid) + continue; + + wcid->sta = 0; + mt76_wcid_cleanup(dev, wcid); + rcu_assign_pointer(dev->wcid[i], NULL); + } + rcu_read_unlock(); + + INIT_LIST_HEAD(&dev->wcid_list); + INIT_LIST_HEAD(&dev->sta_poll_list); + dev->vif_mask = 0; + memset(dev->wcid_mask, 0, sizeof(dev->wcid_mask)); + + mt76_reset_phy(&dev->phy); + for (i = 0; i < ARRAY_SIZE(dev->phys); i++) + mt76_reset_phy(dev->phys[i]); +} +EXPORT_SYMBOL_GPL(mt76_reset_device); + static void mt76_rx_release_amsdu(struct mt76_phy *phy, enum mt76_rxq_id q) { struct sk_buff *skb = phy->rx_amsdu[q].head; diff --git a/drivers/net/wireless/mediatek/mt76/mt76.h b/drivers/net/wireless/mediatek/mt76/mt76.h index df0ea638370b..5050f640530f 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76.h +++ b/drivers/net/wireless/mediatek/mt76/mt76.h @@ -1080,6 +1080,7 @@ int mt76_register_device(struct mt76_dev *dev, bool vht, struct ieee80211_rate *rates, int n_rates); void mt76_unregister_device(struct mt76_dev *dev); void mt76_free_device(struct mt76_dev *dev); +void mt76_reset_device(struct mt76_dev *dev); void mt76_unregister_phy(struct mt76_phy *phy); struct mt76_phy *mt76_alloc_phy(struct mt76_dev *dev, unsigned int size, diff --git a/drivers/net/wireless/mediatek/mt76/mt7603/init.c b/drivers/net/wireless/mediatek/mt76/mt7603/init.c index 0762de3ce5ac..02d7b36ea786 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7603/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7603/init.c @@ -222,6 +222,7 @@ mt7603_mac_init(struct mt7603_dev *dev) eth_broadcast_addr(bc_addr); mt7603_wtbl_init(dev, MT7603_WTBL_RESERVED, -1, bc_addr); + mt76_wcid_init(&dev->global_sta.wcid); dev->global_sta.wcid.idx = MT7603_WTBL_RESERVED; rcu_assign_pointer(dev->mt76.wcid[MT7603_WTBL_RESERVED], &dev->global_sta.wcid); diff --git a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c index ef1e68154bb6..c459b18ddc14 100644 --- a/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt76_connac_mcu.c @@ -1261,8 +1261,10 @@ int mt76_connac_mcu_sta_ba(struct mt76_dev *dev, struct mt76_vif *mvif, wtbl_hdr); ret = mt76_connac_mcu_sta_wed_update(dev, skb); - if (ret) + if (ret) { + dev_kfree_skb(skb); return ret; + } ret = mt76_mcu_skb_send_msg(dev, skb, cmd, true); if (ret) @@ -1275,8 +1277,10 @@ int mt76_connac_mcu_sta_ba(struct mt76_dev *dev, struct mt76_vif *mvif, mt76_connac_mcu_sta_ba_tlv(skb, params, enable, tx); ret = mt76_connac_mcu_sta_wed_update(dev, skb); - if (ret) + if (ret) { + dev_kfree_skb(skb); return ret; + } return mt76_mcu_skb_send_msg(dev, skb, cmd, true); } @@ -2775,12 +2779,16 @@ int mt76_connac_mcu_add_key(struct mt76_dev *dev, struct ieee80211_vif *vif, return PTR_ERR(skb); ret = mt76_connac_mcu_sta_key_tlv(sta_key_conf, skb, key, cmd); - if (ret) + if (ret) { + dev_kfree_skb(skb); return ret; + } ret = mt76_connac_mcu_sta_wed_update(dev, skb); - if (ret) + if (ret) { + dev_kfree_skb(skb); return ret; + } return mt76_mcu_skb_send_msg(dev, skb, mcu_cmd, true); } diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c b/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c index 450f4d221184..1646328bde9c 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/debugfs.c @@ -444,6 +444,11 @@ mt7915_rdd_monitor(struct seq_file *s, void *data) mutex_lock(&dev->mt76.mutex); + if (!mt7915_eeprom_has_background_radar(dev)) { + seq_puts(s, "no background radar capability\n"); + goto out; + } + if (!cfg80211_chandef_valid(chandef)) { ret = -EINVAL; goto out; diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c index 0425cf8b5650..cf9fe6b533fc 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.c @@ -136,7 +136,7 @@ static int mt7915_eeprom_load(struct mt7915_dev *dev) /* read eeprom data from efuse */ block_num = DIV_ROUND_UP(eeprom_size, eeprom_blk_size); for (i = 0; i < block_num; i++) { - ret = mt7915_mcu_get_eeprom(dev, i * eeprom_blk_size); + ret = mt7915_mcu_get_eeprom(dev, i * eeprom_blk_size, NULL); if (ret < 0) return ret; } @@ -353,6 +353,37 @@ s8 mt7915_eeprom_get_power_delta(struct mt7915_dev *dev, int band) return val & MT_EE_RATE_DELTA_SIGN ? delta : -delta; } +bool +mt7915_eeprom_has_background_radar(struct mt7915_dev *dev) +{ + u8 val, buf[MT7915_EEPROM_BLOCK_SIZE]; + u8 band_sel, tx_path, rx_path; + int offs = MT_EE_WIFI_CONF + 1; + + switch (mt76_chip(&dev->mt76)) { + case 0x7915: + return true; + case 0x7906: + /* read efuse to check background radar capability */ + if (mt7915_mcu_get_eeprom(dev, offs, buf)) + break; + + val = buf[offs % MT7915_EEPROM_BLOCK_SIZE]; + band_sel = u8_get_bits(val, MT_EE_WIFI_CONF0_BAND_SEL); + tx_path = u8_get_bits(val, MT_EE_WIFI_CONF0_TX_PATH); + rx_path = u8_get_bits(val, MT_EE_WIFI_CONF0_RX_PATH); + + return (band_sel == MT_EE_V2_BAND_SEL_5GHZ && + tx_path == rx_path && rx_path == 2); + case 0x7981: + case 0x7986: + default: + break; + } + + return false; +} + const u8 mt7915_sku_group_len[] = { [SKU_CCK] = 4, [SKU_OFDM] = 8, diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h index adc26a222823..ca0506df1ef3 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h +++ b/drivers/net/wireless/mediatek/mt76/mt7915/eeprom.h @@ -45,6 +45,7 @@ enum mt7915_eeprom_field { #define MT_EE_CAL_DPD_SIZE (54 * MT_EE_CAL_UNIT) #define MT_EE_WIFI_CONF0_TX_PATH GENMASK(2, 0) +#define MT_EE_WIFI_CONF0_RX_PATH GENMASK(5, 3) #define MT_EE_WIFI_CONF0_BAND_SEL GENMASK(7, 6) #define MT_EE_WIFI_CONF1_BAND_SEL GENMASK(7, 6) #define MT_EE_WIFI_CONF_STREAM_NUM GENMASK(7, 5) diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/init.c b/drivers/net/wireless/mediatek/mt76/mt7915/init.c index ba8da7e2e7e6..711de6b1ce5b 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/init.c @@ -377,9 +377,10 @@ mt7915_init_wiphy(struct mt7915_phy *phy) if (!is_mt7915(&dev->mt76)) wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_STA_TX_PWR); - if (!mdev->dev->of_node || - !of_property_read_bool(mdev->dev->of_node, - "mediatek,disable-radar-background")) + if (mt7915_eeprom_has_background_radar(phy->dev) && + (!mdev->dev->of_node || + !of_property_read_bool(mdev->dev->of_node, + "mediatek,disable-radar-background"))) wiphy_ext_feature_set(wiphy, NL80211_EXT_FEATURE_RADAR_BACKGROUND); @@ -1260,6 +1261,8 @@ int mt7915_register_device(struct mt7915_dev *dev) void mt7915_unregister_device(struct mt7915_dev *dev) { cancel_work_sync(&dev->dump_work); + cancel_work_sync(&dev->reset_work); + cancel_work_sync(&dev->rc_work); mt7915_unregister_ext_phy(dev); mt7915_coredump_unregister(dev); mt7915_unregister_thermal(&dev->phy); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mac.c b/drivers/net/wireless/mediatek/mt76/mt7915/mac.c index a594a88f2054..fd743ef86e7a 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mac.c @@ -1452,17 +1452,15 @@ mt7915_mac_full_reset(struct mt7915_dev *dev) if (i == 10) dev_err(dev->mt76.dev, "chip full reset failed\n"); - spin_lock_bh(&dev->mt76.sta_poll_lock); - while (!list_empty(&dev->mt76.sta_poll_list)) - list_del_init(dev->mt76.sta_poll_list.next); - spin_unlock_bh(&dev->mt76.sta_poll_lock); - - memset(dev->mt76.wcid_mask, 0, sizeof(dev->mt76.wcid_mask)); - dev->mt76.vif_mask = 0; dev->phy.omac_mask = 0; if (phy2) phy2->omac_mask = 0; + mt76_reset_device(&dev->mt76); + + INIT_LIST_HEAD(&dev->sta_rc_list); + INIT_LIST_HEAD(&dev->twt_list); + i = mt76_wcid_alloc(dev->mt76.wcid_mask, MT7915_WTBL_STA); dev->mt76.global_wcid.idx = i; dev->recovery.hw_full_reset = false; diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/main.c b/drivers/net/wireless/mediatek/mt76/mt7915/main.c index 19355a11e0c7..864f636666e5 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/main.c @@ -245,7 +245,11 @@ static int mt7915_add_interface(struct ieee80211_hw *hw, dev->mt76.vif_mask |= BIT_ULL(mvif->mt76.idx); phy->omac_mask |= BIT_ULL(mvif->mt76.omac_idx); - idx = MT7915_WTBL_RESERVED - mvif->mt76.idx; + idx = mt76_wcid_alloc(dev->mt76.wcid_mask, mt7915_wtbl_size(dev)); + if (idx < 0) { + ret = -ENOSPC; + goto err; + } INIT_LIST_HEAD(&mvif->sta.rc_list); INIT_LIST_HEAD(&mvif->sta.wcid.poll_list); @@ -275,7 +279,17 @@ static int mt7915_add_interface(struct ieee80211_hw *hw, mt7915_mcu_add_sta(dev, vif, NULL, true); rcu_assign_pointer(dev->mt76.wcid[idx], &mvif->sta.wcid); + mutex_unlock(&dev->mt76.mutex); + + return 0; + +err: + dev->mt76.vif_mask &= ~BIT_ULL(mvif->mt76.idx); + phy->omac_mask &= ~BIT_ULL(mvif->mt76.omac_idx); + mt7915_mcu_add_dev_info(phy, vif, false); out: + if (phy->monitor_vif == vif) + phy->monitor_vif = NULL; mutex_unlock(&dev->mt76.mutex); return ret; @@ -292,6 +306,7 @@ static void mt7915_remove_interface(struct ieee80211_hw *hw, mt7915_mcu_add_bss_info(phy, vif, false); mt7915_mcu_add_sta(dev, vif, NULL, false); + mt76_wcid_mask_clear(dev->mt76.wcid_mask, mvif->sta.wcid.idx); mutex_lock(&dev->mt76.mutex); mt76_testmode_reset(phy->mt76, true); diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c index 6c0ae5715ca6..8e2ee4ef0c81 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mcu.c @@ -1717,8 +1717,10 @@ int mt7915_mcu_add_sta(struct mt7915_dev *dev, struct ieee80211_vif *vif, } out: ret = mt76_connac_mcu_sta_wed_update(&dev->mt76, skb); - if (ret) + if (ret) { + dev_kfree_skb(skb); return ret; + } return mt76_mcu_skb_send_msg(&dev->mt76, skb, MCU_EXT_CMD(STA_REC_UPDATE), true); @@ -2838,7 +2840,7 @@ int mt7915_mcu_set_eeprom(struct mt7915_dev *dev) &req, sizeof(req), true); } -int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset) +int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset, u8 *read_buf) { struct mt7915_mcu_eeprom_info req = { .addr = cpu_to_le32(round_down(offset, @@ -2846,8 +2848,8 @@ int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset) }; struct mt7915_mcu_eeprom_info *res; struct sk_buff *skb; + u8 *buf = read_buf; int ret; - u8 *buf; ret = mt76_mcu_send_and_get_msg(&dev->mt76, MCU_EXT_QUERY(EFUSE_ACCESS), @@ -2856,8 +2858,17 @@ int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset) return ret; res = (struct mt7915_mcu_eeprom_info *)skb->data; - buf = dev->mt76.eeprom.data + le32_to_cpu(res->addr); + if (!buf) { + u32 addr = le32_to_cpu(res->addr); + + if (addr > dev->mt76.eeprom.size - MT7915_EEPROM_BLOCK_SIZE) { + dev_kfree_skb(skb); + return -EINVAL; + } + buf = dev->mt76.eeprom.data + addr; + } memcpy(buf, res->data, MT7915_EEPROM_BLOCK_SIZE); + dev_kfree_skb(skb); return 0; diff --git a/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h b/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h index 19e3a773d9c2..94dbe3045260 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h +++ b/drivers/net/wireless/mediatek/mt76/mt7915/mt7915.h @@ -439,6 +439,7 @@ int mt7915_eeprom_get_target_power(struct mt7915_dev *dev, struct ieee80211_channel *chan, u8 chain_idx); s8 mt7915_eeprom_get_power_delta(struct mt7915_dev *dev, int band); +bool mt7915_eeprom_has_background_radar(struct mt7915_dev *dev); int mt7915_dma_init(struct mt7915_dev *dev, struct mt7915_phy *phy2); void mt7915_dma_prefetch(struct mt7915_dev *dev); void mt7915_dma_cleanup(struct mt7915_dev *dev); @@ -490,7 +491,7 @@ int mt7915_mcu_set_fixed_rate_ctrl(struct mt7915_dev *dev, struct ieee80211_sta *sta, void *data, u32 field); int mt7915_mcu_set_eeprom(struct mt7915_dev *dev); -int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset); +int mt7915_mcu_get_eeprom(struct mt7915_dev *dev, u32 offset, u8 *read_buf); int mt7915_mcu_get_eeprom_free_block(struct mt7915_dev *dev, u8 *block_num); int mt7915_mcu_set_mac(struct mt7915_dev *dev, int band, bool enable, bool hdr_trans); diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/main.c b/drivers/net/wireless/mediatek/mt76/mt7921/main.c index 5ffe08b8b85c..8f0976e9c295 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/main.c +++ b/drivers/net/wireless/mediatek/mt76/mt7921/main.c @@ -726,6 +726,9 @@ int mt7921_mac_sta_add(struct mt76_dev *mdev, struct ieee80211_vif *vif, struct mt792x_vif *mvif = (struct mt792x_vif *)vif->drv_priv; int ret, idx; + if (sta->aid > MT7921_MAX_AID) + return -ENOENT; + idx = mt76_wcid_alloc(dev->mt76.wcid_mask, MT792x_WTBL_STA - 1); if (idx < 0) return -ENOSPC; diff --git a/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h b/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h index 87dd06855f68..bb54832358f5 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h +++ b/drivers/net/wireless/mediatek/mt76/mt7921/mt7921.h @@ -7,6 +7,8 @@ #include "../mt792x.h" #include "regs.h" +#define MT7921_MAX_AID 20 + #define MT7921_TX_RING_SIZE 2048 #define MT7921_TX_MCU_RING_SIZE 256 #define MT7921_TX_FWDL_RING_SIZE 128 diff --git a/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.c b/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.c index c4e3bfcc519e..3643db2ce8c8 100644 --- a/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.c +++ b/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.c @@ -77,68 +77,98 @@ mt792x_asar_acpi_read_mtcl(struct mt792x_dev *dev, u8 **table, u8 *version) return ret; } +/* A table's layout is decided by the version byte in MTCL, but some firmware + * labels a version 2 table as version 1. Accept whichever layout the table + * actually has: for every accepted entry count the two do not overlap, since + * one is 6 + 5n bytes and the other 7 + 7n. + */ +static bool +mt792x_asar_fits(u8 *table, int len, int prelen, int sarlen, int nr_off, + int min, int max) +{ + int tblcnt; + + if (len < prelen || (len - prelen) % sarlen) + return false; + + tblcnt = (len - prelen) / sarlen; + if (tblcnt < min || tblcnt > max) + return false; + + return tblcnt == table[nr_off]; +} + /* MTDS : Dynamic SAR Power Table */ static int -mt792x_asar_acpi_read_mtds(struct mt792x_dev *dev, u8 **table, u8 version) +mt792x_asar_acpi_read_mtds(struct mt792x_dev *dev, u8 **table, u8 version, + u8 *used) { - int len, ret, sarlen, prelen, tblcnt; - bool enable; + int len, ret; + u8 *t; ret = mt792x_acpi_read(dev, MT792x_ACPI_MTDS, table, &len); if (ret) return ret; + t = *table; + /* Table content validation */ - switch (version) { - case 1: - enable = ((struct mt792x_asar_dyn *)*table)->enable; - sarlen = sizeof(struct mt792x_asar_dyn_limit); - prelen = sizeof(struct mt792x_asar_dyn); - break; - case 2: - enable = ((struct mt792x_asar_dyn_v2 *)*table)->enable; - sarlen = sizeof(struct mt792x_asar_dyn_limit_v2); - prelen = sizeof(struct mt792x_asar_dyn_v2); - break; - default: + if (mt792x_asar_fits(t, len, sizeof(struct mt792x_asar_dyn), + sizeof(struct mt792x_asar_dyn_limit), + offsetof(struct mt792x_asar_dyn, nr_tbl), + MT792x_ASAR_MIN_DYN, MT792x_ASAR_MAX_DYN)) + *used = 1; + else if (mt792x_asar_fits(t, len, sizeof(struct mt792x_asar_dyn_v2), + sizeof(struct mt792x_asar_dyn_limit_v2), + offsetof(struct mt792x_asar_dyn_v2, nr_tbl), + MT792x_ASAR_MIN_DYN, MT792x_ASAR_MAX_DYN)) + *used = 2; + else return -EINVAL; - } - tblcnt = (len - prelen) / sarlen; - if (!enable || - tblcnt > MT792x_ASAR_MAX_DYN || tblcnt < MT792x_ASAR_MIN_DYN) + if (!t[offsetof(struct mt792x_asar_dyn, enable)]) return -EINVAL; + if (version && *used != version) + dev_info(dev->mt76.dev, + "MTDS is v%u, MTCL says v%u; using v%u\n", + *used, version, *used); + return 0; } /* MTGS : Geo SAR Power Table */ static int -mt792x_asar_acpi_read_mtgs(struct mt792x_dev *dev, u8 **table, u8 version) +mt792x_asar_acpi_read_mtgs(struct mt792x_dev *dev, u8 **table, u8 version, + u8 *used) { - int len, ret, sarlen, prelen, tblcnt; + int len, ret; + u8 *t; ret = mt792x_acpi_read(dev, MT792x_ACPI_MTGS, table, &len); if (ret) return ret; + t = *table; + /* Table content validation */ - switch (version) { - case 1: - sarlen = sizeof(struct mt792x_asar_geo_limit); - prelen = sizeof(struct mt792x_asar_geo); - break; - case 2: - sarlen = sizeof(struct mt792x_asar_geo_limit_v2); - prelen = sizeof(struct mt792x_asar_geo_v2); - break; - default: + if (mt792x_asar_fits(t, len, sizeof(struct mt792x_asar_geo), + sizeof(struct mt792x_asar_geo_limit), + offsetof(struct mt792x_asar_geo, nr_tbl), + MT792x_ASAR_MIN_GEO, MT792x_ASAR_MAX_GEO)) + *used = 1; + else if (mt792x_asar_fits(t, len, sizeof(struct mt792x_asar_geo_v2), + sizeof(struct mt792x_asar_geo_limit_v2), + offsetof(struct mt792x_asar_geo_v2, nr_tbl), + MT792x_ASAR_MIN_GEO, MT792x_ASAR_MAX_GEO)) + *used = 2; + else return -EINVAL; - } - tblcnt = (len - prelen) / sarlen; - if (tblcnt > MT792x_ASAR_MAX_GEO || tblcnt < MT792x_ASAR_MIN_GEO) - return -EINVAL; + if (version && *used != version) + dev_info(dev->mt76.dev, + "MTGS is v%u, MTCL says v%u; using v%u\n", + *used, version, *used); return 0; } @@ -174,14 +204,16 @@ int mt792x_init_acpi_sar(struct mt792x_dev *dev) asar->countrylist = NULL; } - ret = mt792x_asar_acpi_read_mtds(dev, (u8 **)&asar->dyn, asar->ver); + ret = mt792x_asar_acpi_read_mtds(dev, (u8 **)&asar->dyn, asar->ver, + &asar->dyn_ver); if (ret) { devm_kfree(dev->mt76.dev, asar->dyn); asar->dyn = NULL; } /* MTGS is optional */ - ret = mt792x_asar_acpi_read_mtgs(dev, (u8 **)&asar->geo, asar->ver); + ret = mt792x_asar_acpi_read_mtgs(dev, (u8 **)&asar->geo, asar->ver, + &asar->geo_ver); if (ret) { devm_kfree(dev->mt76.dev, asar->geo); asar->geo = NULL; @@ -223,7 +255,7 @@ mt792x_asar_get_geo_pwr(struct mt792x_phy *phy, break; } - if (asar->ver == 1) { + if (asar->geo_ver == 1) { band_pwr = &asar->geo->tbl[idx].band[0]; max = ARRAY_SIZE(asar->geo->tbl[idx].band); } else { @@ -248,10 +280,18 @@ mt792x_asar_get_geo_pwr(struct mt792x_phy *phy, if (idx >= max) return dyn_power; - geo_power = (band_pwr + idx)->pwr; - dyn_power += (band_pwr + idx)->offset; + band_pwr += idx; + + /* Some OEM tables carry 0xff here; narrowed to s8 that is -1 and + * wins the min() below. Take it as no limit. + */ + if (band_pwr->pwr == 0xff) + return dyn_power; + + geo_power = band_pwr->pwr; - return min(geo_power, dyn_power); + return min_t(int, geo_power, + min_t(int, dyn_power + band_pwr->offset, 127)); } static s8 @@ -266,7 +306,7 @@ mt792x_asar_range_pwr(struct mt792x_phy *phy, if (!capa) return 127; - if (asar->ver == 1) { + if (asar->dyn_ver == 1) { limit = &asar->dyn->tbl[0].frp[0]; max = ARRAY_SIZE(asar->dyn->tbl[0].frp); } else { @@ -284,7 +324,9 @@ mt792x_asar_range_pwr(struct mt792x_phy *phy, else band = NL80211_BAND_2GHZ; - return mt792x_asar_get_geo_pwr(phy, band, limit[idx]); + /* 0xff: no dynamic limit, see mt792x_asar_get_geo_pwr() */ + return mt792x_asar_get_geo_pwr(phy, band, + limit[idx] == 0xff ? 127 : limit[idx]); } int mt792x_init_acpi_sar_power(struct mt792x_phy *phy, bool set_default) diff --git a/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.h b/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.h index d6d332e863ba..ed2e266d0637 100644 --- a/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.h +++ b/drivers/net/wireless/mediatek/mt76/mt792x_acpi_sar.h @@ -90,6 +90,8 @@ struct mt792x_asar_fg { struct mt792x_acpi_sar { u8 ver; + u8 dyn_ver; + u8 geo_ver; union { struct mt792x_asar_dyn *dyn; struct mt792x_asar_dyn_v2 *dyn_v2; diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/init.c b/drivers/net/wireless/mediatek/mt76/mt7996/init.c index fa81a1c30704..6acd7c10d28a 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/init.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/init.c @@ -930,6 +930,9 @@ int mt7996_register_device(struct mt7996_dev *dev) void mt7996_unregister_device(struct mt7996_dev *dev) { + cancel_work_sync(&dev->dump_work); + cancel_work_sync(&dev->reset_work); + cancel_work_sync(&dev->rc_work); mt7996_unregister_phy(mt7996_phy3(dev), MT_BAND2); mt7996_unregister_phy(mt7996_phy2(dev), MT_BAND1); mt7996_coredump_unregister(dev); diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c index 74ba07577bfd..14c84219ef75 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c @@ -479,7 +479,13 @@ mt7996_mac_fill_rx(struct mt7996_dev *dev, struct sk_buff *skb) memset(status, 0, sizeof(*status)); band_idx = FIELD_GET(MT_RXD1_NORMAL_BAND_IDX, rxd1); + if (band_idx >= __MT_MAX_BAND) + return -EINVAL; + mphy = dev->mt76.phys[band_idx]; + if (!mphy) + return -EINVAL; + phy = mphy->priv; status->phy_idx = mphy->band_idx; @@ -1845,13 +1851,14 @@ void mt7996_mac_reset_work(struct work_struct *work) set_bit(MT76_RESET, &phy3->mt76->state); cancel_delayed_work_sync(&phy3->mt76->mac_work); } + + mutex_lock(&dev->mt76.mutex); + mt76_worker_disable(&dev->mt76.tx_worker); mt76_for_each_q_rx(&dev->mt76, i) napi_disable(&dev->mt76.napi[i]); napi_disable(&dev->mt76.tx_napi); - mutex_lock(&dev->mt76.mutex); - mt76_wr(dev, MT_MCU_INT_EVENT, MT_MCU_INT_EVENT_DMA_STOPPED); if (mt7996_wait_reset_state(dev, MT_MCU_CMD_RESET_DONE)) { diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c index 3699a43df5a9..6c9517e4cea4 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mcu.c @@ -3107,7 +3107,13 @@ int mt7996_mcu_get_eeprom(struct mt7996_dev *dev, u32 offset) valid = le32_to_cpu(*(__le32 *)(skb->data + 16)); if (valid) { u32 addr = le32_to_cpu(*(__le32 *)(skb->data + 12)); - u8 *buf = (u8 *)dev->mt76.eeprom.data + addr; + u8 *buf; + + if (addr > dev->mt76.eeprom.size - MT7996_EEPROM_BLOCK_SIZE) { + dev_kfree_skb(skb); + return -EINVAL; + } + buf = (u8 *)dev->mt76.eeprom.data + addr; skb_pull(skb, 48); memcpy(buf, skb->data, MT7996_EEPROM_BLOCK_SIZE); diff --git a/drivers/net/wireless/mediatek/mt76/tx.c b/drivers/net/wireless/mediatek/mt76/tx.c index 47cdccdbed6a..c1353ee158a3 100644 --- a/drivers/net/wireless/mediatek/mt76/tx.c +++ b/drivers/net/wireless/mediatek/mt76/tx.c @@ -313,7 +313,6 @@ __mt76_tx_queue_skb(struct mt76_phy *phy, int qid, struct sk_buff *skb, if (idx < 0 || !sta) return idx; - wcid = (struct mt76_wcid *)sta->drv_priv; q->entry[idx].wcid = wcid->idx; if (!non_aql) diff --git a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c index 5a57ffb53ace..bbaebdd821c8 100644 --- a/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c +++ b/drivers/net/wireless/realtek/rtl8xxxu/rtl8xxxu_core.c @@ -5829,14 +5829,19 @@ static void rtl8xxxu_queue_rx_urb(struct rtl8xxxu_priv *priv, { struct sk_buff *skb; unsigned long flags; - int pending = 0; spin_lock_irqsave(&priv->rx_urb_lock, flags); if (!priv->shutdown) { list_add_tail(&rx_urb->list, &priv->rx_urb_pending_list); priv->rx_urb_pending_count++; - pending = priv->rx_urb_pending_count; + /* + * Arm the worker under rx_urb_lock so this is atomic with the + * shutdown check: moving it out of the lock would let a + * completion arm the work after rtl8xxxu_stop() canceled it. + */ + if (priv->rx_urb_pending_count > RTL8XXXU_RX_URB_PENDING_WATER) + schedule_work(&priv->rx_urb_wq); } else { skb = (struct sk_buff *)rx_urb->urb.context; dev_kfree_skb_irq(skb); @@ -5844,9 +5849,6 @@ static void rtl8xxxu_queue_rx_urb(struct rtl8xxxu_priv *priv, } spin_unlock_irqrestore(&priv->rx_urb_lock, flags); - - if (pending > RTL8XXXU_RX_URB_PENDING_WATER) - schedule_work(&priv->rx_urb_wq); } static void rtl8xxxu_rx_urb_work(struct work_struct *work) @@ -7281,6 +7283,13 @@ static void rtl8xxxu_stop(struct ieee80211_hw *hw) priv->shutdown = true; spin_unlock_irqrestore(&priv->rx_urb_lock, flags); + /* + * Cancel before killing rx_anchor: the worker re-anchors every URB + * it drained via rtl8xxxu_submit_rx_urb(), so a worker still running + * after the kill could submit a URB that escapes it. + */ + cancel_work_sync(&priv->rx_urb_wq); + usb_kill_anchored_urbs(&priv->rx_anchor); usb_kill_anchored_urbs(&priv->tx_anchor); if (priv->usb_interrupts) diff --git a/drivers/net/wireless/realtek/rtw88/debug.c b/drivers/net/wireless/realtek/rtw88/debug.c index 35bc37a3c469..83376a6c9abb 100644 --- a/drivers/net/wireless/realtek/rtw88/debug.c +++ b/drivers/net/wireless/realtek/rtw88/debug.c @@ -43,6 +43,62 @@ struct rtw_debugfs_priv { }; }; +struct rtw_debugfs { + struct rtw_debugfs_priv mac_0; + struct rtw_debugfs_priv mac_1; + struct rtw_debugfs_priv mac_2; + struct rtw_debugfs_priv mac_3; + struct rtw_debugfs_priv mac_4; + struct rtw_debugfs_priv mac_5; + struct rtw_debugfs_priv mac_6; + struct rtw_debugfs_priv mac_7; + struct rtw_debugfs_priv mac_10; + struct rtw_debugfs_priv mac_11; + struct rtw_debugfs_priv mac_12; + struct rtw_debugfs_priv mac_13; + struct rtw_debugfs_priv mac_14; + struct rtw_debugfs_priv mac_15; + struct rtw_debugfs_priv mac_16; + struct rtw_debugfs_priv mac_17; + struct rtw_debugfs_priv bb_8; + struct rtw_debugfs_priv bb_9; + struct rtw_debugfs_priv bb_a; + struct rtw_debugfs_priv bb_b; + struct rtw_debugfs_priv bb_c; + struct rtw_debugfs_priv bb_d; + struct rtw_debugfs_priv bb_e; + struct rtw_debugfs_priv bb_f; + struct rtw_debugfs_priv bb_18; + struct rtw_debugfs_priv bb_19; + struct rtw_debugfs_priv bb_1a; + struct rtw_debugfs_priv bb_1b; + struct rtw_debugfs_priv bb_1c; + struct rtw_debugfs_priv bb_1d; + struct rtw_debugfs_priv bb_1e; + struct rtw_debugfs_priv bb_1f; + struct rtw_debugfs_priv bb_2c; + struct rtw_debugfs_priv bb_2d; + struct rtw_debugfs_priv bb_40; + struct rtw_debugfs_priv bb_41; + struct rtw_debugfs_priv rf_dump; + struct rtw_debugfs_priv tx_pwr_tbl; + struct rtw_debugfs_priv write_reg; + struct rtw_debugfs_priv h2c; + struct rtw_debugfs_priv rf_write; + struct rtw_debugfs_priv rf_read; + struct rtw_debugfs_priv read_reg; + struct rtw_debugfs_priv fix_rate; + struct rtw_debugfs_priv dump_cam; + struct rtw_debugfs_priv rsvd_page; + struct rtw_debugfs_priv phy_info; + struct rtw_debugfs_priv coex_enable; + struct rtw_debugfs_priv coex_info; + struct rtw_debugfs_priv edcca_enable; + struct rtw_debugfs_priv fw_crash; + struct rtw_debugfs_priv force_lowest_basic_rate; + struct rtw_debugfs_priv dm_cap; +}; + static const char * const rtw_dm_cap_strs[] = { [RTW_DM_CAP_NA] = "NA", [RTW_DM_CAP_TXGAPK] = "TXGAPK", @@ -541,7 +597,7 @@ static int rtw_debug_get_bb_page(struct seq_file *m, void *v) return 0; } -static int rtw_debug_get_rf_dump(struct seq_file *m, void *v) +static int rtw_debugfs_get_rf_dump(struct seq_file *m, void *v) { struct rtw_debugfs_priv *debugfs_priv = m->private; struct rtw_dev *rtwdev = debugfs_priv->rtwdev; @@ -1102,139 +1158,102 @@ static int rtw_debugfs_get_dm_cap(struct seq_file *m, void *v) return 0; } -#define rtw_debug_impl_mac(page, addr) \ -static struct rtw_debugfs_priv rtw_debug_priv_mac_ ##page = { \ +#define rtw_debug_priv_mac(addr) \ +{ \ .cb_read = rtw_debug_get_mac_page, \ .cb_data = addr, \ } -rtw_debug_impl_mac(0, 0x0000); -rtw_debug_impl_mac(1, 0x0100); -rtw_debug_impl_mac(2, 0x0200); -rtw_debug_impl_mac(3, 0x0300); -rtw_debug_impl_mac(4, 0x0400); -rtw_debug_impl_mac(5, 0x0500); -rtw_debug_impl_mac(6, 0x0600); -rtw_debug_impl_mac(7, 0x0700); -rtw_debug_impl_mac(10, 0x1000); -rtw_debug_impl_mac(11, 0x1100); -rtw_debug_impl_mac(12, 0x1200); -rtw_debug_impl_mac(13, 0x1300); -rtw_debug_impl_mac(14, 0x1400); -rtw_debug_impl_mac(15, 0x1500); -rtw_debug_impl_mac(16, 0x1600); -rtw_debug_impl_mac(17, 0x1700); - -#define rtw_debug_impl_bb(page, addr) \ -static struct rtw_debugfs_priv rtw_debug_priv_bb_ ##page = { \ +#define rtw_debug_priv_bb(addr) \ +{ \ .cb_read = rtw_debug_get_bb_page, \ .cb_data = addr, \ } -rtw_debug_impl_bb(8, 0x0800); -rtw_debug_impl_bb(9, 0x0900); -rtw_debug_impl_bb(a, 0x0a00); -rtw_debug_impl_bb(b, 0x0b00); -rtw_debug_impl_bb(c, 0x0c00); -rtw_debug_impl_bb(d, 0x0d00); -rtw_debug_impl_bb(e, 0x0e00); -rtw_debug_impl_bb(f, 0x0f00); -rtw_debug_impl_bb(18, 0x1800); -rtw_debug_impl_bb(19, 0x1900); -rtw_debug_impl_bb(1a, 0x1a00); -rtw_debug_impl_bb(1b, 0x1b00); -rtw_debug_impl_bb(1c, 0x1c00); -rtw_debug_impl_bb(1d, 0x1d00); -rtw_debug_impl_bb(1e, 0x1e00); -rtw_debug_impl_bb(1f, 0x1f00); -rtw_debug_impl_bb(2c, 0x2c00); -rtw_debug_impl_bb(2d, 0x2d00); -rtw_debug_impl_bb(40, 0x4000); -rtw_debug_impl_bb(41, 0x4100); - -static struct rtw_debugfs_priv rtw_debug_priv_rf_dump = { - .cb_read = rtw_debug_get_rf_dump, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_tx_pwr_tbl = { - .cb_read = rtw_debugfs_get_tx_pwr_tbl, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_write_reg = { - .cb_write = rtw_debugfs_set_write_reg, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_h2c = { - .cb_write = rtw_debugfs_set_h2c, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_rf_write = { - .cb_write = rtw_debugfs_set_rf_write, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_rf_read = { - .cb_write = rtw_debugfs_set_rf_read, - .cb_read = rtw_debugfs_get_rf_read, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_read_reg = { - .cb_write = rtw_debugfs_set_read_reg, - .cb_read = rtw_debugfs_get_read_reg, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_fix_rate = { - .cb_write = rtw_debugfs_set_fix_rate, - .cb_read = rtw_debugfs_get_fix_rate, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_dump_cam = { - .cb_write = rtw_debugfs_set_single_input, - .cb_read = rtw_debugfs_get_dump_cam, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_rsvd_page = { - .cb_write = rtw_debugfs_set_rsvd_page, - .cb_read = rtw_debugfs_get_rsvd_page, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_phy_info = { - .cb_read = rtw_debugfs_get_phy_info, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_coex_enable = { - .cb_write = rtw_debugfs_set_coex_enable, - .cb_read = rtw_debugfs_get_coex_enable, -}; - -static struct rtw_debugfs_priv rtw_debug_priv_coex_info = { - .cb_read = rtw_debugfs_get_coex_info, -}; +#define rtw_debug_priv_get(name) \ +{ \ + .cb_read = rtw_debugfs_get_ ##name, \ +} -static struct rtw_debugfs_priv rtw_debug_priv_edcca_enable = { - .cb_write = rtw_debugfs_set_edcca_enable, - .cb_read = rtw_debugfs_get_edcca_enable, -}; +#define rtw_debug_priv_set(name) \ +{ \ + .cb_write = rtw_debugfs_set_ ##name, \ +} -static struct rtw_debugfs_priv rtw_debug_priv_fw_crash = { - .cb_write = rtw_debugfs_set_fw_crash, - .cb_read = rtw_debugfs_get_fw_crash, -}; +#define rtw_debug_priv_set_and_get(name) \ +{ \ + .cb_write = rtw_debugfs_set_ ##name, \ + .cb_read = rtw_debugfs_get_ ##name, \ +} -static struct rtw_debugfs_priv rtw_debug_priv_force_lowest_basic_rate = { - .cb_write = rtw_debugfs_set_force_lowest_basic_rate, - .cb_read = rtw_debugfs_get_force_lowest_basic_rate, -}; +#define rtw_debug_priv_set_single_and_get(name) \ +{ \ + .cb_write = rtw_debugfs_set_single_input, \ + .cb_read = rtw_debugfs_get_ ##name, \ +} -static struct rtw_debugfs_priv rtw_debug_priv_dm_cap = { - .cb_write = rtw_debugfs_set_dm_cap, - .cb_read = rtw_debugfs_get_dm_cap, +static const struct rtw_debugfs rtw_debugfs_templ = { + .mac_0 = rtw_debug_priv_mac(0x0000), + .mac_1 = rtw_debug_priv_mac(0x0100), + .mac_2 = rtw_debug_priv_mac(0x0200), + .mac_3 = rtw_debug_priv_mac(0x0300), + .mac_4 = rtw_debug_priv_mac(0x0400), + .mac_5 = rtw_debug_priv_mac(0x0500), + .mac_6 = rtw_debug_priv_mac(0x0600), + .mac_7 = rtw_debug_priv_mac(0x0700), + .mac_10 = rtw_debug_priv_mac(0x1000), + .mac_11 = rtw_debug_priv_mac(0x1100), + .mac_12 = rtw_debug_priv_mac(0x1200), + .mac_13 = rtw_debug_priv_mac(0x1300), + .mac_14 = rtw_debug_priv_mac(0x1400), + .mac_15 = rtw_debug_priv_mac(0x1500), + .mac_16 = rtw_debug_priv_mac(0x1600), + .mac_17 = rtw_debug_priv_mac(0x1700), + .bb_8 = rtw_debug_priv_bb(0x0800), + .bb_9 = rtw_debug_priv_bb(0x0900), + .bb_a = rtw_debug_priv_bb(0x0a00), + .bb_b = rtw_debug_priv_bb(0x0b00), + .bb_c = rtw_debug_priv_bb(0x0c00), + .bb_d = rtw_debug_priv_bb(0x0d00), + .bb_e = rtw_debug_priv_bb(0x0e00), + .bb_f = rtw_debug_priv_bb(0x0f00), + .bb_18 = rtw_debug_priv_bb(0x1800), + .bb_19 = rtw_debug_priv_bb(0x1900), + .bb_1a = rtw_debug_priv_bb(0x1a00), + .bb_1b = rtw_debug_priv_bb(0x1b00), + .bb_1c = rtw_debug_priv_bb(0x1c00), + .bb_1d = rtw_debug_priv_bb(0x1d00), + .bb_1e = rtw_debug_priv_bb(0x1e00), + .bb_1f = rtw_debug_priv_bb(0x1f00), + .bb_2c = rtw_debug_priv_bb(0x2c00), + .bb_2d = rtw_debug_priv_bb(0x2d00), + .bb_40 = rtw_debug_priv_bb(0x4000), + .bb_41 = rtw_debug_priv_bb(0x4100), + .rf_dump = rtw_debug_priv_get(rf_dump), + .tx_pwr_tbl = rtw_debug_priv_get(tx_pwr_tbl), + .write_reg = rtw_debug_priv_set(write_reg), + .h2c = rtw_debug_priv_set(h2c), + .rf_write = rtw_debug_priv_set(rf_write), + .rf_read = rtw_debug_priv_set_and_get(rf_read), + .read_reg = rtw_debug_priv_set_and_get(read_reg), + .fix_rate = rtw_debug_priv_set_and_get(fix_rate), + .dump_cam = rtw_debug_priv_set_single_and_get(dump_cam), + .rsvd_page = rtw_debug_priv_set_and_get(rsvd_page), + .phy_info = rtw_debug_priv_get(phy_info), + .coex_enable = rtw_debug_priv_set_and_get(coex_enable), + .coex_info = rtw_debug_priv_get(coex_info), + .edcca_enable = rtw_debug_priv_set_and_get(edcca_enable), + .fw_crash = rtw_debug_priv_set_and_get(fw_crash), + .force_lowest_basic_rate = rtw_debug_priv_set_and_get(force_lowest_basic_rate), + .dm_cap = rtw_debug_priv_set_and_get(dm_cap), }; #define rtw_debugfs_add_core(name, mode, fopname, parent) \ do { \ - rtw_debug_priv_ ##name.rtwdev = rtwdev; \ + struct rtw_debugfs_priv *priv = &rtwdev->debugfs->name; \ + priv->rtwdev = rtwdev; \ if (IS_ERR(debugfs_create_file(#name, mode, \ - parent, &rtw_debug_priv_ ##name,\ + parent, priv, \ &file_ops_ ##fopname))) \ pr_debug("Unable to initialize debugfs:%s\n", \ #name); \ @@ -1247,12 +1266,9 @@ static struct rtw_debugfs_priv rtw_debug_priv_dm_cap = { #define rtw_debugfs_add_r(name) \ rtw_debugfs_add_core(name, S_IFREG | 0444, single_r, debugfs_topdir) -void rtw_debugfs_init(struct rtw_dev *rtwdev) +static +void rtw_debugfs_add_basic(struct rtw_dev *rtwdev, struct dentry *debugfs_topdir) { - struct dentry *debugfs_topdir; - - debugfs_topdir = debugfs_create_dir("rtw88", - rtwdev->hw->wiphy->debugfsdir); rtw_debugfs_add_w(write_reg); rtw_debugfs_add_rw(read_reg); rtw_debugfs_add_w(rf_write); @@ -1264,6 +1280,17 @@ void rtw_debugfs_init(struct rtw_dev *rtwdev) rtw_debugfs_add_r(coex_info); rtw_debugfs_add_rw(coex_enable); rtw_debugfs_add_w(h2c); + rtw_debugfs_add_r(rf_dump); + rtw_debugfs_add_r(tx_pwr_tbl); + rtw_debugfs_add_rw(edcca_enable); + rtw_debugfs_add_rw(fw_crash); + rtw_debugfs_add_rw(force_lowest_basic_rate); + rtw_debugfs_add_rw(dm_cap); +} + +static +void rtw_debugfs_add_sec0(struct rtw_dev *rtwdev, struct dentry *debugfs_topdir) +{ rtw_debugfs_add_r(mac_0); rtw_debugfs_add_r(mac_1); rtw_debugfs_add_r(mac_2); @@ -1280,6 +1307,11 @@ void rtw_debugfs_init(struct rtw_dev *rtwdev) rtw_debugfs_add_r(bb_d); rtw_debugfs_add_r(bb_e); rtw_debugfs_add_r(bb_f); +} + +static +void rtw_debugfs_add_sec1(struct rtw_dev *rtwdev, struct dentry *debugfs_topdir) +{ rtw_debugfs_add_r(mac_10); rtw_debugfs_add_r(mac_11); rtw_debugfs_add_r(mac_12); @@ -1302,14 +1334,29 @@ void rtw_debugfs_init(struct rtw_dev *rtwdev) rtw_debugfs_add_r(bb_40); rtw_debugfs_add_r(bb_41); } - rtw_debugfs_add_r(rf_dump); - rtw_debugfs_add_r(tx_pwr_tbl); - rtw_debugfs_add_rw(edcca_enable); - rtw_debugfs_add_rw(fw_crash); - rtw_debugfs_add_rw(force_lowest_basic_rate); - rtw_debugfs_add_rw(dm_cap); } +void rtw_debugfs_init(struct rtw_dev *rtwdev) +{ + struct dentry *debugfs_topdir; + + rtwdev->debugfs = kmemdup(&rtw_debugfs_templ, sizeof(rtw_debugfs_templ), + GFP_KERNEL); + if (!rtwdev->debugfs) + return; + + debugfs_topdir = debugfs_create_dir("rtw88", + rtwdev->hw->wiphy->debugfsdir); + + rtw_debugfs_add_basic(rtwdev, debugfs_topdir); + rtw_debugfs_add_sec0(rtwdev, debugfs_topdir); + rtw_debugfs_add_sec1(rtwdev, debugfs_topdir); +} + +void rtw_debugfs_deinit(struct rtw_dev *rtwdev) +{ + kfree(rtwdev->debugfs); +} #endif /* CONFIG_RTW88_DEBUGFS */ #ifdef CONFIG_RTW88_DEBUG diff --git a/drivers/net/wireless/realtek/rtw88/debug.h b/drivers/net/wireless/realtek/rtw88/debug.h index a9149c6c2b48..92d589ba002c 100644 --- a/drivers/net/wireless/realtek/rtw88/debug.h +++ b/drivers/net/wireless/realtek/rtw88/debug.h @@ -32,11 +32,13 @@ enum rtw_debug_mask { #ifdef CONFIG_RTW88_DEBUGFS void rtw_debugfs_init(struct rtw_dev *rtwdev); +void rtw_debugfs_deinit(struct rtw_dev *rtwdev); void rtw_debugfs_get_simple_phy_info(struct seq_file *m); #else static inline void rtw_debugfs_init(struct rtw_dev *rtwdev) {} +static inline void rtw_debugfs_deinit(struct rtw_dev *rtwdev) {} #endif /* CONFIG_RTW88_DEBUGFS */ diff --git a/drivers/net/wireless/realtek/rtw88/main.c b/drivers/net/wireless/realtek/rtw88/main.c index 9a4c23163fba..2afb273ff7a6 100644 --- a/drivers/net/wireless/realtek/rtw88/main.c +++ b/drivers/net/wireless/realtek/rtw88/main.c @@ -1633,11 +1633,13 @@ static u16 rtw_get_max_scan_ie_len(struct rtw_dev *rtwdev) static void rtw_set_supported_band(struct ieee80211_hw *hw, const struct rtw_chip_info *chip) { - struct rtw_dev *rtwdev = hw->priv; struct ieee80211_supported_band *sband; + struct rtw_dev *rtwdev = hw->priv; + struct device *dev = rtwdev->dev; if (chip->band & RTW_BAND_2G) { - sband = kmemdup(&rtw_band_2ghz, sizeof(*sband), GFP_KERNEL); + sband = devm_kmemdup(dev, &rtw_band_2ghz, sizeof(*sband), + GFP_KERNEL); if (!sband) goto err_out; if (chip->ht_supported) @@ -1646,7 +1648,8 @@ static void rtw_set_supported_band(struct ieee80211_hw *hw, } if (chip->band & RTW_BAND_5G) { - sband = kmemdup(&rtw_band_5ghz, sizeof(*sband), GFP_KERNEL); + sband = devm_kmemdup(dev, &rtw_band_5ghz, sizeof(*sband), + GFP_KERNEL); if (!sband) goto err_out; if (chip->ht_supported) @@ -1662,13 +1665,6 @@ static void rtw_set_supported_band(struct ieee80211_hw *hw, rtw_err(rtwdev, "failed to set supported band\n"); } -static void rtw_unset_supported_band(struct ieee80211_hw *hw, - const struct rtw_chip_info *chip) -{ - kfree(hw->wiphy->bands[NL80211_BAND_2GHZ]); - kfree(hw->wiphy->bands[NL80211_BAND_5GHZ]); -} - static void rtw_vif_smps_iter(void *data, u8 *mac, struct ieee80211_vif *vif) { @@ -2279,10 +2275,8 @@ EXPORT_SYMBOL(rtw_register_hw); void rtw_unregister_hw(struct rtw_dev *rtwdev, struct ieee80211_hw *hw) { - const struct rtw_chip_info *chip = rtwdev->chip; - ieee80211_unregister_hw(hw); - rtw_unset_supported_band(hw, chip); + rtw_debugfs_deinit(rtwdev); } EXPORT_SYMBOL(rtw_unregister_hw); diff --git a/drivers/net/wireless/realtek/rtw88/main.h b/drivers/net/wireless/realtek/rtw88/main.h index 6e0e1c9c28f7..17d417ce0aa4 100644 --- a/drivers/net/wireless/realtek/rtw88/main.h +++ b/drivers/net/wireless/realtek/rtw88/main.h @@ -50,6 +50,7 @@ extern const struct ieee80211_ops rtw_ops; #define RTW_MAX_CHANNEL_NUM_5G 49 struct rtw_dev; +struct rtw_debugfs; enum rtw_hci_type { RTW_HCI_TYPE_PCIE, @@ -2038,7 +2039,7 @@ struct rtw_dev { bool beacon_loss; struct completion lps_leave_check; - struct dentry *debugfs; + struct rtw_debugfs *debugfs; u8 sta_cnt; u32 rts_threshold; diff --git a/drivers/net/wireless/realtek/rtw89/mac80211.c b/drivers/net/wireless/realtek/rtw89/mac80211.c index 5b9de1f41dc7..aa04e20c2bc3 100644 --- a/drivers/net/wireless/realtek/rtw89/mac80211.c +++ b/drivers/net/wireless/realtek/rtw89/mac80211.c @@ -872,11 +872,14 @@ static void rtw89_ops_cancel_hw_scan(struct ieee80211_hw *hw, if (!RTW89_CHK_FW_FEATURE(SCAN_OFFLOAD, &rtwdev->fw)) return; + mutex_lock(&rtwdev->mutex); + if (!rtwdev->scanning) - return; + goto out; - mutex_lock(&rtwdev->mutex); rtw89_hw_scan_abort(rtwdev, vif); + +out: mutex_unlock(&rtwdev->mutex); } diff --git a/drivers/net/wireless/realtek/rtw89/pci.c b/drivers/net/wireless/realtek/rtw89/pci.c index 1a1027dd01d3..7b0630d5d528 100644 --- a/drivers/net/wireless/realtek/rtw89/pci.c +++ b/drivers/net/wireless/realtek/rtw89/pci.c @@ -484,11 +484,16 @@ static void rtw89_pci_release_rpp(struct rtw89_dev *rtwdev, tx_status = le32_get_bits(rpp->dword, RTW89_PCI_RPP_TX_STATUS); txch = rtw89_core_get_ch_dma(rtwdev, qsel); - if (txch == RTW89_TXCH_CH12) { + if (unlikely(txch == RTW89_TXCH_CH12)) { rtw89_warn(rtwdev, "should no fwcmd release report\n"); return; } + if (unlikely(seq >= RTW89_PCI_TXWD_NUM_MAX)) { + rtw89_warn(rtwdev, "invalid seq %d\n", seq); + return; + } + tx_ring = &rtwpci->tx_rings[txch]; wd_ring = &tx_ring->wd_ring; txwd = &wd_ring->pages[seq]; diff --git a/drivers/net/wireless/silabs/wfx/hif_tx.c b/drivers/net/wireless/silabs/wfx/hif_tx.c index 9402503fbde3..852406fc7f3c 100644 --- a/drivers/net/wireless/silabs/wfx/hif_tx.c +++ b/drivers/net/wireless/silabs/wfx/hif_tx.c @@ -189,9 +189,11 @@ int wfx_hif_read_mib(struct wfx_dev *wdev, int vif_id, u16 mib_id, void *val, si dev_warn(wdev->dev, "%s: confirmation mismatch request\n", __func__); ret = -EIO; } - if (ret == -ENOMEM) + if (!ret && le16_to_cpu(reply->length) > val_len) { dev_err(wdev->dev, "buffer is too small to receive %s (%zu < %d)\n", wfx_get_mib_name(mib_id), val_len, le16_to_cpu(reply->length)); + ret = -EIO; + } if (!ret) memcpy(val, &reply->mib_data, le16_to_cpu(reply->length)); else diff --git a/drivers/net/wireless/st/cw1200/bh.c b/drivers/net/wireless/st/cw1200/bh.c index 37232ee22037..91d4287f5e4a 100644 --- a/drivers/net/wireless/st/cw1200/bh.c +++ b/drivers/net/wireless/st/cw1200/bh.c @@ -343,6 +343,7 @@ static int cw1200_bh_tx_helper(struct cw1200_common *priv, int *tx_burst) { size_t tx_len; + size_t aligned_len; u8 *data; int ret; struct wsm_hdr *wsm; @@ -375,9 +376,14 @@ static int cw1200_bh_tx_helper(struct cw1200_common *priv, atomic_inc(&priv->bh_tx); - tx_len = priv->hwbus_ops->align_size( + aligned_len = priv->hwbus_ops->align_size( priv->hwbus_priv, tx_len); + /* Zero the bus alignment padding so no stale data is sent. */ + if (aligned_len > tx_len) + memset(data + tx_len, 0, aligned_len - tx_len); + tx_len = aligned_len; + /* Check if not exceeding CW1200 capabilities */ if (WARN_ON_ONCE(tx_len > EFFECTIVE_BUF_SIZE)) pr_debug("Write aligned len: %zu\n", tx_len); diff --git a/drivers/net/wireless/st/cw1200/txrx.c b/drivers/net/wireless/st/cw1200/txrx.c index e16e9ae90d20..413193aac2e0 100644 --- a/drivers/net/wireless/st/cw1200/txrx.c +++ b/drivers/net/wireless/st/cw1200/txrx.c @@ -14,6 +14,7 @@ #include "wsm.h" #include "bh.h" #include "sta.h" +#include "hwbus.h" #include "debug.h" #define CW1200_INVALID_RATE_ID (0xFF) @@ -716,6 +717,7 @@ void cw1200_tx(struct ieee80211_hw *dev, struct ieee80211_sta *sta; struct wsm_tx *wsm; bool tid_update = false; + size_t pad_len; u8 flags = 0; int ret; @@ -764,6 +766,18 @@ void cw1200_tx(struct ieee80211_hw *dev, sta = t.sta; + /* + * The bus transfer is padded up to the bus alignment and the + * padding is transferred from the frame buffer. Make sure the + * buffer has room for the padding so the device never receives + * data past the end of the frame. + */ + pad_len = priv->hwbus_ops->align_size(priv->hwbus_priv, skb->len) - + skb->len; + if (skb_tailroom(skb) < pad_len && + pskb_expand_head(skb, 0, pad_len, GFP_ATOMIC)) + goto drop; + spin_lock_bh(&priv->ps_state_lock); { tid_update = cw1200_tx_h_pm_state(priv, &t); diff --git a/drivers/net/wireless/ti/wl12xx/main.c b/drivers/net/wireless/ti/wl12xx/main.c index d06a2c419447..de045fe4ca1e 100644 --- a/drivers/net/wireless/ti/wl12xx/main.c +++ b/drivers/net/wireless/ti/wl12xx/main.c @@ -1919,7 +1919,7 @@ static int wl12xx_probe(struct platform_device *pdev) return ret; } -static int wl12xx_remove(struct platform_device *pdev) +static void wl12xx_remove(struct platform_device *pdev) { struct wl1271 *wl = platform_get_drvdata(pdev); struct wl12xx_priv *priv; @@ -1928,7 +1928,7 @@ static int wl12xx_remove(struct platform_device *pdev) kfree(priv->rx_mem_addr); - return wlcore_remove(pdev); + wlcore_remove(pdev); } static const struct platform_device_id wl12xx_id_table[] = { @@ -1939,7 +1939,7 @@ MODULE_DEVICE_TABLE(platform, wl12xx_id_table); static struct platform_driver wl12xx_driver = { .probe = wl12xx_probe, - .remove = wl12xx_remove, + .remove_new = wl12xx_remove, .id_table = wl12xx_id_table, .driver = { .name = "wl12xx_driver", diff --git a/drivers/net/wireless/ti/wl18xx/main.c b/drivers/net/wireless/ti/wl18xx/main.c index 0b3cf8477c6c..d4a89401f2c4 100644 --- a/drivers/net/wireless/ti/wl18xx/main.c +++ b/drivers/net/wireless/ti/wl18xx/main.c @@ -2033,7 +2033,7 @@ MODULE_DEVICE_TABLE(platform, wl18xx_id_table); static struct platform_driver wl18xx_driver = { .probe = wl18xx_probe, - .remove = wlcore_remove, + .remove_new = wlcore_remove, .id_table = wl18xx_id_table, .driver = { .name = "wl18xx_driver", diff --git a/drivers/net/wireless/ti/wlcore/main.c b/drivers/net/wireless/ti/wlcore/main.c index f472d4bd77fb..45563dd3d212 100644 --- a/drivers/net/wireless/ti/wlcore/main.c +++ b/drivers/net/wireless/ti/wlcore/main.c @@ -6735,7 +6735,7 @@ int wlcore_probe(struct wl1271 *wl, struct platform_device *pdev) } EXPORT_SYMBOL_GPL(wlcore_probe); -int wlcore_remove(struct platform_device *pdev) +void wlcore_remove(struct platform_device *pdev) { struct wlcore_platdev_data *pdev_data = dev_get_platdata(&pdev->dev); struct wl1271 *wl = platform_get_drvdata(pdev); @@ -6749,8 +6749,10 @@ int wlcore_remove(struct platform_device *pdev) if (pdev_data->family && pdev_data->family->nvs_name) wait_for_completion(&wl->nvs_loading_complete); - if (!wl->initialized) - return 0; + if (!wl->initialized) { + pm_runtime_put_noidle(wl->dev); + return; + } if (wl->wakeirq >= 0) { dev_pm_clear_wake_irq(wl->dev); @@ -6770,8 +6772,6 @@ int wlcore_remove(struct platform_device *pdev) free_irq(wl->irq, wl); wlcore_free_hw(wl); - - return 0; } EXPORT_SYMBOL_GPL(wlcore_remove); diff --git a/drivers/net/wireless/ti/wlcore/wlcore.h b/drivers/net/wireless/ti/wlcore/wlcore.h index 81c94d390623..1f8511bf9bb3 100644 --- a/drivers/net/wireless/ti/wlcore/wlcore.h +++ b/drivers/net/wireless/ti/wlcore/wlcore.h @@ -497,7 +497,7 @@ struct wl1271 { }; int wlcore_probe(struct wl1271 *wl, struct platform_device *pdev); -int wlcore_remove(struct platform_device *pdev); +void wlcore_remove(struct platform_device *pdev); struct ieee80211_hw *wlcore_alloc_hw(size_t priv_size, u32 aggr_buf_size, u32 mbox_size); int wlcore_free_hw(struct wl1271 *wl); diff --git a/drivers/net/xen-netfront.c b/drivers/net/xen-netfront.c index 5b59c0ee6c66..ff619619d71a 100644 --- a/drivers/net/xen-netfront.c +++ b/drivers/net/xen-netfront.c @@ -1233,8 +1233,16 @@ static int handle_incoming_queue(struct netfront_queue *queue, while ((skb = __skb_dequeue(rxq)) != NULL) { int pull_to = NETFRONT_SKB_CB(skb)->pull_to; - if (pull_to > skb_headlen(skb)) - __pskb_pull_tail(skb, pull_to - skb_headlen(skb)); + /* pull_to comes from the first slot's length, which the + * backend controls. Make sure the head holds at least an + * Ethernet header for eth_type_trans(). + */ + if (!pskb_may_pull(skb, max(pull_to, ETH_HLEN))) { + kfree_skb(skb); + packets_dropped++; + queue->info->netdev->stats.rx_errors++; + continue; + } /* Ethernet work: Delayed to here as it peeks the header. */ skb->protocol = eth_type_trans(skb, queue->info->netdev); @@ -1337,8 +1345,10 @@ static int xennet_poll(struct napi_struct *napi, int budget) skb->data_len = rx->status; skb->len += rx->status; - if (unlikely(xennet_fill_frags(queue, skb, &tmpq))) + if (unlikely(xennet_fill_frags(queue, skb, &tmpq))) { + __skb_queue_head(&tmpq, skb); goto err; + } if (rx->flags & XEN_NETRXF_csum_blank) skb->ip_summed = CHECKSUM_PARTIAL; diff --git a/drivers/ntb/ntb_transport.c b/drivers/ntb/ntb_transport.c index f62110713d77..df86c99c554f 100644 --- a/drivers/ntb/ntb_transport.c +++ b/drivers/ntb/ntb_transport.c @@ -211,6 +211,7 @@ struct ntb_transport_mw { void *alloc_addr; void *virt_addr; dma_addr_t dma_addr; + dma_addr_t original_dma_addr; }; struct ntb_transport_client_dev { @@ -791,14 +792,14 @@ static void ntb_transport_msi_desc_changed(void *data) static void ntb_free_mw(struct ntb_transport_ctx *nt, int num_mw) { struct ntb_transport_mw *mw = &nt->mw_vec[num_mw]; - struct pci_dev *pdev = nt->ndev->pdev; + struct device *dma_dev = ntb_get_dma_dev(nt->ndev); if (!mw->virt_addr) return; ntb_mw_clear_trans(nt->ndev, PIDX, num_mw); - dma_free_coherent(&pdev->dev, mw->alloc_size, - mw->alloc_addr, mw->dma_addr); + dma_free_coherent(dma_dev, mw->alloc_size, + mw->alloc_addr, mw->original_dma_addr); mw->xlat_size = 0; mw->buff_size = 0; mw->alloc_size = 0; @@ -834,6 +835,7 @@ static int ntb_alloc_mw_buffer(struct ntb_transport_mw *mw, return -ENOMEM; } virt_addr = alloc_addr; + mw->original_dma_addr = dma_addr; /* * we must ensure that the memory address allocated is BAR size @@ -867,7 +869,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw, resource_size_t size) { struct ntb_transport_mw *mw = &nt->mw_vec[num_mw]; - struct pci_dev *pdev = nt->ndev->pdev; + struct device *dma_dev = ntb_get_dma_dev(nt->ndev); size_t xlat_size, buff_size; resource_size_t xlat_align; resource_size_t xlat_align_size; @@ -896,12 +898,12 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw, mw->buff_size = buff_size; mw->alloc_size = buff_size; - rc = ntb_alloc_mw_buffer(mw, &pdev->dev, xlat_align); + rc = ntb_alloc_mw_buffer(mw, dma_dev, xlat_align); if (rc) { mw->alloc_size *= 2; - rc = ntb_alloc_mw_buffer(mw, &pdev->dev, xlat_align); + rc = ntb_alloc_mw_buffer(mw, dma_dev, xlat_align); if (rc) { - dev_err(&pdev->dev, + dev_err(dma_dev, "Unable to alloc aligned MW buff\n"); mw->xlat_size = 0; mw->buff_size = 0; @@ -914,7 +916,7 @@ static int ntb_set_mw(struct ntb_transport_ctx *nt, int num_mw, rc = ntb_mw_set_trans(nt->ndev, PIDX, num_mw, mw->dma_addr, mw->xlat_size); if (rc) { - dev_err(&pdev->dev, "Unable to set mw%d translation", num_mw); + dev_err(dma_dev, "Unable to set mw%d translation", num_mw); ntb_free_mw(nt, num_mw); return -EIO; } @@ -1805,12 +1807,13 @@ static void ntb_tx_copy_callback(void *data, static void ntb_memcpy_tx(struct ntb_queue_entry *entry, void __iomem *offset) { -#ifdef ARCH_HAS_NOCACHE_UACCESS +#ifdef copy_to_nontemporal /* * Using non-temporal mov to improve performance on non-cached - * writes, even though we aren't actually copying from user space. + * writes. This only works if __iomem is strictly memory-like, + * but that is the case on x86-64 */ - __copy_from_user_inatomic_nocache(offset, entry->buf, entry->len); + copy_to_nontemporal(offset, entry->buf, entry->len); #else memcpy_toio(offset, entry->buf, entry->len); #endif diff --git a/drivers/nvdimm/nd_virtio.c b/drivers/nvdimm/nd_virtio.c index e5a7b031da2d..585d46963388 100644 --- a/drivers/nvdimm/nd_virtio.c +++ b/drivers/nvdimm/nd_virtio.c @@ -9,6 +9,14 @@ #include "virtio_pmem.h" #include "nd.h" +static void virtio_pmem_req_release(struct kref *kref) +{ + struct virtio_pmem_request *req; + + req = container_of(kref, struct virtio_pmem_request, kref); + kfree(req); +} + /* The interrupt handler */ void virtio_pmem_host_ack(struct virtqueue *vq) { @@ -29,6 +37,7 @@ void virtio_pmem_host_ack(struct virtqueue *vq) wake_up(&req_buf->wq_buf); list_del(&req_buf->list); } + kref_put(&req_data->kref, virtio_pmem_req_release); } spin_unlock_irqrestore(&vpmem->pmem_lock, flags); } @@ -59,6 +68,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region) if (!req_data) return -ENOMEM; + kref_init(&req_data->kref); req_data->done = false; init_waitqueue_head(&req_data->host_acked); init_waitqueue_head(&req_data->wq_buf); @@ -76,10 +86,23 @@ static int virtio_pmem_flush(struct nd_region *nd_region) * to req_list and wait for host_ack to wake us up when free * slots are available. */ - while ((err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data, - GFP_ATOMIC)) == -ENOSPC) { + for (;;) { + err = virtqueue_add_sgs(vpmem->req_vq, sgs, 1, 1, req_data, + GFP_ATOMIC); + if (!err) { + /* + * Take the virtqueue reference while @pmem_lock is + * held so completion cannot run concurrently. + */ + kref_get(&req_data->kref); + break; + } + + if (err != -ENOSPC) + break; - dev_info(&vdev->dev, "failed to send command to virtio pmem device, no free slots in the virtqueue\n"); + dev_info_ratelimited(&vdev->dev, + "failed to send command to virtio pmem device, no free slots in the virtqueue\n"); req_data->wq_buf_avail = false; list_add_tail(&req_data->list, &vpmem->req_list); spin_unlock_irqrestore(&vpmem->pmem_lock, flags); @@ -88,6 +111,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region) wait_event(req_data->wq_buf, req_data->wq_buf_avail); spin_lock_irqsave(&vpmem->pmem_lock, flags); } + err1 = virtqueue_kick(vpmem->req_vq); spin_unlock_irqrestore(&vpmem->pmem_lock, flags); /* @@ -103,7 +127,7 @@ static int virtio_pmem_flush(struct nd_region *nd_region) err = le32_to_cpu(req_data->resp.ret); } - kfree(req_data); + kref_put(&req_data->kref, virtio_pmem_req_release); return err; }; diff --git a/drivers/nvdimm/pmem.c b/drivers/nvdimm/pmem.c index 4e8fdcb3f1c8..e82c8e7ec372 100644 --- a/drivers/nvdimm/pmem.c +++ b/drivers/nvdimm/pmem.c @@ -209,8 +209,14 @@ static void pmem_submit_bio(struct bio *bio) struct pmem_device *pmem = bio->bi_bdev->bd_disk->private_data; struct nd_region *nd_region = to_region(pmem); - if (bio->bi_opf & REQ_PREFLUSH) - ret = nvdimm_flush(nd_region, bio); + if (bio->bi_opf & REQ_PREFLUSH) { + ret = nvdimm_flush(nd_region, NULL); + if (ret) { + bio->bi_status = errno_to_blk_status(ret); + bio_endio(bio); + return; + } + } do_acct = blk_queue_io_stat(bio->bi_bdev->bd_disk->queue); if (do_acct) @@ -230,7 +236,7 @@ static void pmem_submit_bio(struct bio *bio) if (do_acct) bio_end_io_acct(bio, start); - if (bio->bi_opf & REQ_FUA) + if ((bio->bi_opf & REQ_FUA) && !bio->bi_status) ret = nvdimm_flush(nd_region, bio); if (ret) diff --git a/drivers/nvdimm/virtio_pmem.h b/drivers/nvdimm/virtio_pmem.h index f72cf17f9518..1017e498c9b4 100644 --- a/drivers/nvdimm/virtio_pmem.h +++ b/drivers/nvdimm/virtio_pmem.h @@ -12,11 +12,13 @@ #include #include +#include #include #include #include struct virtio_pmem_request { + struct kref kref; struct virtio_pmem_req req; struct virtio_pmem_resp resp; diff --git a/drivers/nvme/host/core.c b/drivers/nvme/host/core.c index d02ca033db43..af71dd259c3c 100644 --- a/drivers/nvme/host/core.c +++ b/drivers/nvme/host/core.c @@ -4,6 +4,7 @@ * Copyright (c) 2011-2014, Intel Corporation. */ +#include #include #include #include @@ -121,8 +122,6 @@ static dev_t nvme_ns_chr_devt; static struct class *nvme_ns_chr_class; static void nvme_put_subsystem(struct nvme_subsystem *subsys); -static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl, - unsigned nsid); static void nvme_update_keep_alive(struct nvme_ctrl *ctrl, struct nvme_command *cmd); @@ -3699,6 +3698,9 @@ static void nvme_alloc_ns(struct nvme_ctrl *ctrl, struct nvme_ns_info *info) if (list_empty(&ns->head->list)) list_del_init(&ns->head->entry); mutex_unlock(&ctrl->subsys->lock); + + /* guarantee not available in head->list */ + synchronize_srcu(&ns->head->srcu); nvme_put_ns_head(ns->head); out_cleanup_disk: put_disk(disk); @@ -3829,15 +3831,45 @@ static void nvme_scan_ns(struct nvme_ctrl *ctrl, unsigned nsid) } } -static void nvme_remove_invalid_namespaces(struct nvme_ctrl *ctrl, - unsigned nsid) +/** + * struct async_scan_info - keeps track of controller & NSIDs to scan + * @ctrl: Controller on which namespaces are being scanned + * @next_nsid: Index of next NSID to scan in ns_list + * @ns_list: Pointer to list of NSIDs to scan + * + * Note: There is a single async_scan_info structure shared by all instances + * of nvme_scan_ns_async() scanning a given controller, so the atomic + * operations on next_nsid are critical to ensure each instance scans a unique + * NSID. + */ +struct async_scan_info { + struct nvme_ctrl *ctrl; + atomic_t next_nsid; + __le32 *ns_list; +}; + +static void nvme_scan_ns_async(void *data, async_cookie_t cookie) +{ + struct async_scan_info *scan_info = data; + int idx; + u32 nsid; + + idx = (u32)atomic_fetch_inc(&scan_info->next_nsid); + nsid = le32_to_cpu(scan_info->ns_list[idx]); + + nvme_scan_ns(scan_info->ctrl, nsid); +} + +static void nvme_remove_nsid_range(struct nvme_ctrl *ctrl, u32 start, u32 end) { struct nvme_ns *ns, *next; LIST_HEAD(rm_list); mutex_lock(&ctrl->namespaces_lock); list_for_each_entry_safe(ns, next, &ctrl->namespaces, list) { - if (ns->head->ns_id > nsid) { + if (ns->head->ns_id >= end) + break; + if (ns->head->ns_id > start) { list_del_rcu(&ns->list); synchronize_srcu(&ctrl->srcu); list_add_tail_rcu(&ns->list, &rm_list); @@ -3855,11 +3887,15 @@ static int nvme_scan_ns_list(struct nvme_ctrl *ctrl) __le32 *ns_list; u32 prev = 0; int ret = 0, i; + ASYNC_DOMAIN(domain); + struct async_scan_info scan_info; ns_list = kzalloc(NVME_IDENTIFY_DATA_SIZE, GFP_KERNEL); if (!ns_list) return -ENOMEM; + scan_info.ctrl = ctrl; + scan_info.ns_list = ns_list; for (;;) { struct nvme_command cmd = { .identify.opcode = nvme_admin_identify, @@ -3875,19 +3911,24 @@ static int nvme_scan_ns_list(struct nvme_ctrl *ctrl) goto free; } + atomic_set(&scan_info.next_nsid, 0); for (i = 0; i < nr_entries; i++) { u32 nsid = le32_to_cpu(ns_list[i]); if (!nsid) /* end of the list? */ goto out; - nvme_scan_ns(ctrl, nsid); - while (++prev < nsid) - nvme_ns_remove_by_nsid(ctrl, prev); + async_schedule_domain(nvme_scan_ns_async, &scan_info, + &domain); + if (prev + 1 < nsid) + nvme_remove_nsid_range(ctrl, prev, nsid); + prev = max(prev + 1, nsid); } + async_synchronize_full_domain(&domain); } out: - nvme_remove_invalid_namespaces(ctrl, prev); + nvme_remove_nsid_range(ctrl, prev, UINT_MAX); free: + async_synchronize_full_domain(&domain); kfree(ns_list); return ret; } @@ -3905,7 +3946,7 @@ static void nvme_scan_ns_sequential(struct nvme_ctrl *ctrl) for (i = 1; i <= nn; i++) nvme_scan_ns(ctrl, i); - nvme_remove_invalid_namespaces(ctrl, nn); + nvme_remove_nsid_range(ctrl, nn, UINT_MAX); } static void nvme_clear_changed_ns_log(struct nvme_ctrl *ctrl) diff --git a/drivers/nvme/host/fabrics.c b/drivers/nvme/host/fabrics.c index 8adcc8c97c42..f910db503294 100644 --- a/drivers/nvme/host/fabrics.c +++ b/drivers/nvme/host/fabrics.c @@ -952,6 +952,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, } if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) { pr_err("Invalid DH-CHAP secret %s\n", p); + kfree_sensitive(p); ret = -EINVAL; goto out; } @@ -966,6 +967,7 @@ static int nvmf_parse_options(struct nvmf_ctrl_options *opts, } if (strlen(p) < 11 || strncmp(p, "DHHC-1:", 7)) { pr_err("Invalid DH-CHAP secret %s\n", p); + kfree_sensitive(p); ret = -EINVAL; goto out; } diff --git a/drivers/nvme/host/multipath.c b/drivers/nvme/host/multipath.c index fdd64ce6dedb..923c00c04869 100644 --- a/drivers/nvme/host/multipath.c +++ b/drivers/nvme/host/multipath.c @@ -725,7 +725,8 @@ static int nvme_parse_ana_log(struct nvme_ctrl *ctrl, void *data, u32 nr_nsids; size_t nsid_buf_size; - if (WARN_ON_ONCE(offset > ctrl->ana_log_size - sizeof(*desc))) + if (WARN_ON_ONCE(offset > ctrl->ana_log_size || + sizeof(*desc) > ctrl->ana_log_size - offset)) return -EINVAL; nr_nsids = le32_to_cpu(desc->nnsids); @@ -741,7 +742,7 @@ static int nvme_parse_ana_log(struct nvme_ctrl *ctrl, void *data, return -EINVAL; offset += sizeof(*desc); - if (WARN_ON_ONCE(offset > ctrl->ana_log_size - nsid_buf_size)) + if (WARN_ON_ONCE(nsid_buf_size > ctrl->ana_log_size - offset)) return -EINVAL; error = cb(ctrl, desc, data); diff --git a/drivers/nvme/host/pci.c b/drivers/nvme/host/pci.c index e4d35e985a0b..4e8701775b99 100644 --- a/drivers/nvme/host/pci.c +++ b/drivers/nvme/host/pci.c @@ -3556,6 +3556,8 @@ static const struct pci_device_id nvme_id_table[] = { .driver_data = NVME_QUIRK_IGNORE_DEV_SUBNQN }, { PCI_DEVICE(0x1344, 0x6001), /* Micron Nitro NVMe */ .driver_data = NVME_QUIRK_BOGUS_NID, }, + { PCI_DEVICE(0x1344, 0x6004), /* Micron 4100AT */ + .driver_data = NVME_QUIRK_DMAPOOL_ALIGN_512, }, { PCI_DEVICE(0x1c5c, 0x1504), /* SK Hynix PC400 */ .driver_data = NVME_QUIRK_DISABLE_WRITE_ZEROES, }, { PCI_DEVICE(0x1c5c, 0x174a), /* SK Hynix P31 SSD */ diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 88489a787e67..abfa680bdd21 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -190,6 +190,19 @@ static inline int nvme_tcp_queue_id(struct nvme_tcp_queue *queue) return queue - queue->ctrl->queues; } +static inline bool nvme_tcp_recv_pdu_supported(enum nvme_tcp_pdu_type type) +{ + switch (type) { + case nvme_tcp_c2h_term: + case nvme_tcp_c2h_data: + case nvme_tcp_r2t: + case nvme_tcp_rsp: + return true; + default: + return false; + } +} + static inline struct blk_mq_tags *nvme_tcp_tagset(struct nvme_tcp_queue *queue) { u32 queue_idx = nvme_tcp_queue_id(queue); @@ -385,7 +398,8 @@ nvme_tcp_fetch_request(struct nvme_tcp_queue *queue) return NULL; } - list_del(&req->entry); + list_del_init(&req->entry); + init_llist_node(&req->lentry); return req; } @@ -493,6 +507,8 @@ static int nvme_tcp_init_request(struct blk_mq_tag_set *set, req->queue = queue; nvme_req(rq)->ctrl = &ctrl->ctrl; nvme_req(rq)->cmd = &pdu->cmd; + init_llist_node(&req->lentry); + INIT_LIST_HEAD(&req->entry); return 0; } @@ -739,6 +755,14 @@ static int nvme_tcp_handle_r2t(struct nvme_tcp_queue *queue, return -EPROTO; } + if (llist_on_list(&req->lentry) || + !list_empty(&req->entry)) { + dev_err(queue->ctrl->ctrl.device, + "req %d unexpected r2t while processing request\n", + rq->tag); + return -EPROTO; + } + req->pdu_len = 0; req->h2cdata_left = r2t_length; req->h2cdata_offset = r2t_offset; @@ -805,6 +829,16 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, return 0; hdr = queue->pdu; + if (unlikely(hdr->hlen != sizeof(struct nvme_tcp_rsp_pdu))) { + if (!nvme_tcp_recv_pdu_supported(hdr->type)) + goto unsupported_pdu; + + dev_err(queue->ctrl->ctrl.device, + "pdu type %d has unexpected header length (%d)\n", + hdr->type, hdr->hlen); + return -EPROTO; + } + if (unlikely(hdr->type == nvme_tcp_c2h_term)) { /* * C2HTermReq never includes Header or Data digests. @@ -837,10 +871,13 @@ static int nvme_tcp_recv_pdu(struct nvme_tcp_queue *queue, struct sk_buff *skb, nvme_tcp_init_recv_ctx(queue); return nvme_tcp_handle_r2t(queue, (void *)queue->pdu); default: - dev_err(queue->ctrl->ctrl.device, - "unsupported pdu type (%d)\n", hdr->type); - return -EINVAL; + goto unsupported_pdu; } + +unsupported_pdu: + dev_err(queue->ctrl->ctrl.device, + "unsupported pdu type (%d)\n", hdr->type); + return -EINVAL; } static inline void nvme_tcp_end_request(struct request *rq, u16 status) @@ -1412,7 +1449,6 @@ static int nvme_tcp_alloc_async_req(struct nvme_tcp_ctrl *ctrl) static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) { - struct page *page; struct nvme_tcp_ctrl *ctrl = to_tcp_ctrl(nctrl); struct nvme_tcp_queue *queue = &ctrl->queues[qid]; unsigned int noreclaim_flag; @@ -1423,11 +1459,7 @@ static void nvme_tcp_free_queue(struct nvme_ctrl *nctrl, int qid) if (queue->hdr_digest || queue->data_digest) nvme_tcp_free_crypto(queue); - if (queue->pf_cache.va) { - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); - queue->pf_cache.va = NULL; - } + page_frag_cache_drain(&queue->pf_cache); noreclaim_flag = memalloc_noreclaim_save(); sock_release(queue->sock); @@ -2374,6 +2406,8 @@ static void nvme_tcp_submit_async_event(struct nvme_ctrl *arg) ctrl->async_req.offset = 0; ctrl->async_req.curr_bio = NULL; ctrl->async_req.data_len = 0; + init_llist_node(&ctrl->async_req.lentry); + INIT_LIST_HEAD(&ctrl->async_req.entry); nvme_tcp_queue_request(&ctrl->async_req, true, true); } diff --git a/drivers/nvme/target/admin-cmd.c b/drivers/nvme/target/admin-cmd.c index e79b5fb91d59..c974beefa766 100644 --- a/drivers/nvme/target/admin-cmd.c +++ b/drivers/nvme/target/admin-cmd.c @@ -818,7 +818,7 @@ u16 nvmet_set_feat_async_event(struct nvmet_req *req, u32 mask) void nvmet_execute_set_features(struct nvmet_req *req) { - struct nvmet_subsys *subsys = nvmet_req_subsys(req); + struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req); u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10); u32 cdw11 = le32_to_cpu(req->cmd->common.cdw11); u16 status = 0; @@ -837,7 +837,7 @@ void nvmet_execute_set_features(struct nvmet_req *req) break; } nvmet_set_result(req, - (subsys->max_qid - 1) | ((subsys->max_qid - 1) << 16)); + (ctrl->max_qid - 1) | ((ctrl->max_qid - 1) << 16)); break; case NVME_FEAT_KATO: status = nvmet_set_feat_kato(req); @@ -892,7 +892,7 @@ void nvmet_get_feat_async_event(struct nvmet_req *req) void nvmet_execute_get_features(struct nvmet_req *req) { - struct nvmet_subsys *subsys = nvmet_req_subsys(req); + struct nvmet_ctrl *ctrl = nvmet_req_ctrl(req); u32 cdw10 = le32_to_cpu(req->cmd->common.cdw10); u16 status = 0; @@ -929,7 +929,7 @@ void nvmet_execute_get_features(struct nvmet_req *req) break; case NVME_FEAT_NUM_QUEUES: nvmet_set_result(req, - (subsys->max_qid-1) | ((subsys->max_qid-1) << 16)); + (ctrl->max_qid-1) | ((ctrl->max_qid-1) << 16)); break; case NVME_FEAT_KATO: nvmet_get_feat_kato(req); @@ -975,7 +975,7 @@ void nvmet_execute_async_event(struct nvmet_req *req) ctrl->async_event_cmds[ctrl->nr_async_event_cmds++] = req; mutex_unlock(&ctrl->lock); - queue_work(nvmet_wq, &ctrl->async_event_work); + queue_work(nvmet_aen_wq, &ctrl->async_event_work); } void nvmet_execute_keep_alive(struct nvmet_req *req) @@ -998,6 +998,27 @@ void nvmet_execute_keep_alive(struct nvmet_req *req) nvmet_req_complete(req, status); } +u32 nvmet_admin_cmd_data_len(struct nvmet_req *req) +{ + struct nvme_command *cmd = req->cmd; + + if (nvme_is_fabrics(cmd)) + return nvmet_fabrics_admin_cmd_data_len(req); + if (nvmet_is_disc_subsys(nvmet_req_subsys(req))) + return nvmet_discovery_cmd_data_len(req); + + switch (cmd->common.opcode) { + case nvme_admin_get_log_page: + return nvmet_get_log_page_len(cmd); + case nvme_admin_identify: + return NVME_IDENTIFY_DATA_SIZE; + case nvme_admin_get_features: + return nvmet_feat_data_len(req, le32_to_cpu(cmd->common.cdw10)); + default: + return 0; + } +} + u16 nvmet_parse_admin_cmd(struct nvmet_req *req) { struct nvme_command *cmd = req->cmd; diff --git a/drivers/nvme/target/configfs.c b/drivers/nvme/target/configfs.c index 384cd2b540d0..b266ed13ab32 100644 --- a/drivers/nvme/target/configfs.c +++ b/drivers/nvme/target/configfs.c @@ -346,6 +346,7 @@ static ssize_t nvmet_ns_device_path_store(struct config_item *item, struct nvmet_subsys *subsys = ns->subsys; size_t len; int ret; + char *new_path = NULL; mutex_lock(&subsys->lock); ret = -EBUSY; @@ -357,12 +358,14 @@ static ssize_t nvmet_ns_device_path_store(struct config_item *item, if (!len) goto out_unlock; - kfree(ns->device_path); ret = -ENOMEM; - ns->device_path = kmemdup_nul(page, len, GFP_KERNEL); - if (!ns->device_path) + new_path = kmemdup_nul(page, len, GFP_KERNEL); + if (!new_path) goto out_unlock; + kfree(ns->device_path); + ns->device_path = new_path; + mutex_unlock(&subsys->lock); return count; diff --git a/drivers/nvme/target/core.c b/drivers/nvme/target/core.c index 3b1b72022a97..8e0da771fc1c 100644 --- a/drivers/nvme/target/core.c +++ b/drivers/nvme/target/core.c @@ -25,6 +25,8 @@ static DEFINE_IDA(cntlid_ida); struct workqueue_struct *nvmet_wq; EXPORT_SYMBOL_GPL(nvmet_wq); +struct workqueue_struct *nvmet_aen_wq; +EXPORT_SYMBOL_GPL(nvmet_aen_wq); /* * This read/write semaphore is used to synchronize access to configuration @@ -211,7 +213,7 @@ void nvmet_add_async_event(struct nvmet_ctrl *ctrl, u8 event_type, list_add_tail(&aen->entry, &ctrl->async_events); mutex_unlock(&ctrl->lock); - queue_work(nvmet_wq, &ctrl->async_event_work); + queue_work(nvmet_aen_wq, &ctrl->async_event_work); } static void nvmet_add_to_changed_ns_log(struct nvmet_ctrl *ctrl, __le32 nsid) @@ -883,6 +885,33 @@ static inline u16 nvmet_io_cmd_check_access(struct nvmet_req *req) return 0; } +static u32 nvmet_io_cmd_transfer_len(struct nvmet_req *req) +{ + struct nvme_command *cmd = req->cmd; + u32 metadata_len = 0; + + if (nvme_is_fabrics(cmd)) + return nvmet_fabrics_io_cmd_data_len(req); + + if (!req->ns) + return 0; + + switch (req->cmd->common.opcode) { + case nvme_cmd_read: + case nvme_cmd_write: + case nvme_cmd_zone_append: + if (req->sq->ctrl->pi_support && nvmet_ns_has_pi(req->ns)) + metadata_len = nvmet_rw_metadata_len(req); + return nvmet_rw_data_len(req) + metadata_len; + case nvme_cmd_dsm: + return nvmet_dsm_len(req); + case nvme_cmd_zone_mgmt_recv: + return (le32_to_cpu(req->cmd->zmr.numd) + 1) << 2; + default: + return 0; + } +} + static u16 nvmet_parse_io_cmd(struct nvmet_req *req) { struct nvme_command *cmd = req->cmd; @@ -1007,6 +1036,16 @@ void nvmet_req_uninit(struct nvmet_req *req) } EXPORT_SYMBOL_GPL(nvmet_req_uninit); +size_t nvmet_req_transfer_len(struct nvmet_req *req) +{ + if (likely(req->sq->qid != 0)) + return nvmet_io_cmd_transfer_len(req); + if (unlikely(!req->sq->ctrl)) + return nvmet_connect_cmd_data_len(req); + return nvmet_admin_cmd_data_len(req); +} +EXPORT_SYMBOL_GPL(nvmet_req_transfer_len); + bool nvmet_check_transfer_len(struct nvmet_req *req, size_t len) { if (unlikely(len != req->transfer_len)) { @@ -1054,7 +1093,7 @@ static int nvmet_req_alloc_p2pmem_sgls(struct pci_dev *p2p_dev, return 0; out_free_sg: - pci_p2pmem_free_sgl(req->p2p_dev, req->sg); + pci_p2pmem_free_sgl(p2p_dev, req->sg); out_err: return -ENOMEM; } @@ -1432,7 +1471,25 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, if (!ctrl->changed_ns_list) goto out_free_ctrl; - ctrl->sqs = kcalloc(subsys->max_qid + 1, + /* + * Discovery controllers may use some arbitrary high value + * in order to cleanup stale discovery sessions + */ + if (nvmet_is_disc_subsys(ctrl->subsys) && !kato) + kato = NVMET_DISC_KATO_MS; + + /* keep-alive timeout in seconds */ + ctrl->kato = DIV_ROUND_UP(kato, 1000); + + ctrl->err_counter = 0; + spin_lock_init(&ctrl->error_lock); + + down_read(&nvmet_config_sem); + mutex_lock(&subsys->lock); + + ctrl->max_qid = subsys->max_qid; + + ctrl->sqs = kcalloc(ctrl->max_qid + 1, sizeof(struct nvmet_sq *), GFP_KERNEL); if (!ctrl->sqs) @@ -1450,25 +1507,12 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, } ctrl->cntlid = ret; - /* - * Discovery controllers may use some arbitrary high value - * in order to cleanup stale discovery sessions - */ - if (nvmet_is_disc_subsys(ctrl->subsys) && !kato) - kato = NVMET_DISC_KATO_MS; - - /* keep-alive timeout in seconds */ - ctrl->kato = DIV_ROUND_UP(kato, 1000); - - ctrl->err_counter = 0; - spin_lock_init(&ctrl->error_lock); - - nvmet_start_keep_alive_timer(ctrl); - - mutex_lock(&subsys->lock); list_add_tail(&ctrl->subsys_entry, &subsys->ctrls); nvmet_setup_p2p_ns_map(ctrl, req); mutex_unlock(&subsys->lock); + up_read(&nvmet_config_sem); + + nvmet_start_keep_alive_timer(ctrl); *ctrlp = ctrl; return 0; @@ -1476,6 +1520,8 @@ u16 nvmet_alloc_ctrl(const char *subsysnqn, const char *hostnqn, out_free_sqs: kfree(ctrl->sqs); out_free_changed_ns_list: + mutex_unlock(&subsys->lock); + up_read(&nvmet_config_sem); kfree(ctrl->changed_ns_list); out_free_ctrl: kfree(ctrl); @@ -1682,9 +1728,14 @@ static int __init nvmet_init(void) if (!nvmet_wq) goto out_free_buffered_work_queue; + nvmet_aen_wq = alloc_workqueue("nvmet-aen-wq", + WQ_MEM_RECLAIM | WQ_UNBOUND, 0); + if (!nvmet_aen_wq) + goto out_free_nvmet_work_queue; + error = nvmet_init_discovery(); if (error) - goto out_free_nvmet_work_queue; + goto out_free_nvmet_aen_work_queue; error = nvmet_init_configfs(); if (error) @@ -1693,6 +1744,8 @@ static int __init nvmet_init(void) out_exit_discovery: nvmet_exit_discovery(); +out_free_nvmet_aen_work_queue: + destroy_workqueue(nvmet_aen_wq); out_free_nvmet_work_queue: destroy_workqueue(nvmet_wq); out_free_buffered_work_queue: @@ -1709,6 +1762,7 @@ static void __exit nvmet_exit(void) nvmet_exit_configfs(); nvmet_exit_discovery(); ida_destroy(&cntlid_ida); + destroy_workqueue(nvmet_aen_wq); destroy_workqueue(nvmet_wq); destroy_workqueue(buffered_io_wq); destroy_workqueue(zbd_wq); diff --git a/drivers/nvme/target/discovery.c b/drivers/nvme/target/discovery.c index af3c6793aff0..4ea79ee87f79 100644 --- a/drivers/nvme/target/discovery.c +++ b/drivers/nvme/target/discovery.c @@ -166,6 +166,7 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req) u64 offset = nvmet_get_log_page_offset(req->cmd); size_t data_len = nvmet_get_log_page_len(req->cmd); size_t alloc_len; + size_t copy_len; struct nvmet_subsys_link *p; struct nvmet_port *r; u32 numrec = 0; @@ -239,7 +240,27 @@ static void nvmet_execute_disc_get_log_page(struct nvmet_req *req) up_read(&nvmet_config_sem); - status = nvmet_copy_to_sgl(req, 0, buffer + offset, data_len); + /* + * Validate the host-supplied log page offset before copying out. + * Without this check, the host controls a 64-bit byte offset into + * a small kzalloc'd buffer: a value past the log page lets the + * subsequent memcpy read adjacent kernel heap, and a value aimed + * at unmapped kernel memory faults the in-kernel copy and crashes + * the target host. The Discovery controller is unauthenticated, + * so the bug is reachable from any reachable fabric peer. + */ + if (offset > alloc_len) { + req->error_loc = + offsetof(struct nvme_get_log_page_command, lpo); + status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR; + goto out_free_buffer; + } + + copy_len = min_t(size_t, data_len, alloc_len - offset); + status = nvmet_copy_to_sgl(req, 0, buffer + offset, copy_len); + if (!status && copy_len < data_len) + status = nvmet_zero_sgl(req, copy_len, data_len - copy_len); +out_free_buffer: kfree(buffer); out: nvmet_req_complete(req, status); @@ -352,6 +373,20 @@ static void nvmet_execute_disc_get_features(struct nvmet_req *req) nvmet_req_complete(req, stat); } +u32 nvmet_discovery_cmd_data_len(struct nvmet_req *req) +{ + struct nvme_command *cmd = req->cmd; + + switch (cmd->common.opcode) { + case nvme_admin_get_log_page: + return nvmet_get_log_page_len(req->cmd); + case nvme_admin_identify: + return NVME_IDENTIFY_DATA_SIZE; + default: + return 0; + } +} + u16 nvmet_parse_discovery_cmd(struct nvmet_req *req) { struct nvme_command *cmd = req->cmd; diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c index 193bc75de833..935510c377a6 100644 --- a/drivers/nvme/target/fabrics-cmd-auth.c +++ b/drivers/nvme/target/fabrics-cmd-auth.c @@ -196,6 +196,11 @@ static u8 nvmet_auth_failure2(void *d) return data->rescode_exp; } +u32 nvmet_auth_send_data_len(struct nvmet_req *req) +{ + return le32_to_cpu(req->cmd->auth_send.tl); +} + void nvmet_execute_auth_send(struct nvmet_req *req) { struct nvmet_ctrl *ctrl = req->sq->ctrl; @@ -223,7 +228,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req) offsetof(struct nvmf_auth_send_command, spsp1); goto done; } - tl = le32_to_cpu(req->cmd->auth_send.tl); + tl = nvmet_auth_send_data_len(req); if (!tl) { status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR; req->error_loc = @@ -373,7 +378,7 @@ static int nvmet_auth_challenge(struct nvmet_req *req, void *d, int al) struct nvmet_ctrl *ctrl = req->sq->ctrl; int ret = 0; int hash_len = nvme_auth_hmac_hash_len(ctrl->shash_id); - int data_size = sizeof(*d) + hash_len; + int data_size = sizeof(*data) + hash_len; if (ctrl->dh_tfm) data_size += ctrl->dh_keysize; @@ -446,6 +451,35 @@ static void nvmet_auth_failure1(struct nvmet_req *req, void *d, int al) data->rescode_exp = req->sq->dhchap_status; } +u32 nvmet_auth_receive_data_len(struct nvmet_req *req) +{ + struct nvmet_ctrl *ctrl = req->sq->ctrl; + u32 al = le32_to_cpu(req->cmd->auth_receive.al); + u32 min_len; + + /* + * Reject too-short al before kmalloc(al), since the SUCCESS1 and + * FAILURE1/default builders write fixed response headers into it. + */ + switch (req->sq->dhchap_step) { + case NVME_AUTH_DHCHAP_MESSAGE_CHALLENGE: + return al; + case NVME_AUTH_DHCHAP_MESSAGE_SUCCESS1: + min_len = sizeof(struct nvmf_auth_dhchap_success1_data); + if (req->sq->dhchap_c2) + min_len += nvme_auth_hmac_hash_len(ctrl->shash_id); + break; + default: + min_len = sizeof(struct nvmf_auth_dhchap_failure_data); + break; + } + + if (al < min_len) + return 0; + + return al; +} + void nvmet_execute_auth_receive(struct nvmet_req *req) { struct nvmet_ctrl *ctrl = req->sq->ctrl; @@ -471,7 +505,7 @@ void nvmet_execute_auth_receive(struct nvmet_req *req) offsetof(struct nvmf_auth_receive_command, spsp1); goto done; } - al = le32_to_cpu(req->cmd->auth_receive.al); + al = nvmet_auth_receive_data_len(req); if (!al) { status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR; req->error_loc = diff --git a/drivers/nvme/target/fabrics-cmd.c b/drivers/nvme/target/fabrics-cmd.c index da56af9bd8d2..7807cee08790 100644 --- a/drivers/nvme/target/fabrics-cmd.c +++ b/drivers/nvme/target/fabrics-cmd.c @@ -82,6 +82,22 @@ static void nvmet_execute_prop_get(struct nvmet_req *req) nvmet_req_complete(req, status); } +u32 nvmet_fabrics_admin_cmd_data_len(struct nvmet_req *req) +{ + struct nvme_command *cmd = req->cmd; + + switch (cmd->fabrics.fctype) { +#ifdef CONFIG_NVME_TARGET_AUTH + case nvme_fabrics_type_auth_send: + return nvmet_auth_send_data_len(req); + case nvme_fabrics_type_auth_receive: + return nvmet_auth_receive_data_len(req); +#endif + default: + return 0; + } +} + u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req) { struct nvme_command *cmd = req->cmd; @@ -111,6 +127,22 @@ u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req) return 0; } +u32 nvmet_fabrics_io_cmd_data_len(struct nvmet_req *req) +{ + struct nvme_command *cmd = req->cmd; + + switch (cmd->fabrics.fctype) { +#ifdef CONFIG_NVME_TARGET_AUTH + case nvme_fabrics_type_auth_send: + return nvmet_auth_send_data_len(req); + case nvme_fabrics_type_auth_receive: + return nvmet_auth_receive_data_len(req); +#endif + default: + return 0; + } +} + u16 nvmet_parse_fabrics_io_cmd(struct nvmet_req *req) { struct nvme_command *cmd = req->cmd; @@ -319,7 +351,7 @@ static void nvmet_execute_io_connect(struct nvmet_req *req) goto out; } - if (unlikely(qid > ctrl->subsys->max_qid)) { + if (unlikely(qid > ctrl->max_qid)) { pr_warn("invalid queue id (%d)\n", qid); status = NVME_SC_CONNECT_INVALID_PARAM | NVME_STATUS_DNR; req->cqe->result.u32 = IPO_IATTR_CONNECT_SQE(qid); @@ -343,6 +375,17 @@ static void nvmet_execute_io_connect(struct nvmet_req *req) goto out; } +u32 nvmet_connect_cmd_data_len(struct nvmet_req *req) +{ + struct nvme_command *cmd = req->cmd; + + if (!nvme_is_fabrics(cmd) || + cmd->fabrics.fctype != nvme_fabrics_type_connect) + return 0; + + return sizeof(struct nvmf_connect_data); +} + u16 nvmet_parse_connect_cmd(struct nvmet_req *req) { struct nvme_command *cmd = req->cmd; diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index c037d1daf94d..2189e5a5985b 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -191,6 +191,7 @@ struct nvmet_ctrl { uuid_t hostid; u16 cntlid; + u16 max_qid; u32 kato; struct nvmet_port *port; @@ -400,6 +401,7 @@ extern struct kmem_cache *nvmet_bvec_cache; extern struct workqueue_struct *buffered_io_wq; extern struct workqueue_struct *zbd_wq; extern struct workqueue_struct *nvmet_wq; +extern struct workqueue_struct *nvmet_aen_wq; static inline void nvmet_set_result(struct nvmet_req *req, u32 result) { @@ -446,18 +448,24 @@ void nvmet_start_keep_alive_timer(struct nvmet_ctrl *ctrl); void nvmet_stop_keep_alive_timer(struct nvmet_ctrl *ctrl); u16 nvmet_parse_connect_cmd(struct nvmet_req *req); +u32 nvmet_connect_cmd_data_len(struct nvmet_req *req); void nvmet_bdev_set_limits(struct block_device *bdev, struct nvme_id_ns *id); u16 nvmet_bdev_parse_io_cmd(struct nvmet_req *req); u16 nvmet_file_parse_io_cmd(struct nvmet_req *req); u16 nvmet_bdev_zns_parse_io_cmd(struct nvmet_req *req); +u32 nvmet_admin_cmd_data_len(struct nvmet_req *req); u16 nvmet_parse_admin_cmd(struct nvmet_req *req); +u32 nvmet_discovery_cmd_data_len(struct nvmet_req *req); u16 nvmet_parse_discovery_cmd(struct nvmet_req *req); u16 nvmet_parse_fabrics_admin_cmd(struct nvmet_req *req); +u32 nvmet_fabrics_admin_cmd_data_len(struct nvmet_req *req); u16 nvmet_parse_fabrics_io_cmd(struct nvmet_req *req); +u32 nvmet_fabrics_io_cmd_data_len(struct nvmet_req *req); bool nvmet_req_init(struct nvmet_req *req, struct nvmet_cq *cq, struct nvmet_sq *sq, const struct nvmet_fabrics_ops *ops); void nvmet_req_uninit(struct nvmet_req *req); +size_t nvmet_req_transfer_len(struct nvmet_req *req); bool nvmet_check_transfer_len(struct nvmet_req *req, size_t len); bool nvmet_check_data_len_lte(struct nvmet_req *req, size_t data_len); void nvmet_req_complete(struct nvmet_req *req, u16 status); @@ -613,6 +621,11 @@ static inline struct nvmet_subsys *nvmet_req_subsys(struct nvmet_req *req) return req->sq->ctrl->subsys; } +static inline struct nvmet_ctrl *nvmet_req_ctrl(struct nvmet_req *req) +{ + return req->sq->ctrl; +} + static inline bool nvmet_is_disc_subsys(struct nvmet_subsys *subsys) { return subsys->type != NVME_NQN_NVME; @@ -695,7 +708,9 @@ static inline void nvmet_req_bio_put(struct nvmet_req *req, struct bio *bio) } #ifdef CONFIG_NVME_TARGET_AUTH +u32 nvmet_auth_send_data_len(struct nvmet_req *req); void nvmet_execute_auth_send(struct nvmet_req *req); +u32 nvmet_auth_receive_data_len(struct nvmet_req *req); void nvmet_execute_auth_receive(struct nvmet_req *req); int nvmet_auth_set_key(struct nvmet_host *host, const char *secret, bool set_ctrl); diff --git a/drivers/nvme/target/rdma.c b/drivers/nvme/target/rdma.c index 0dfe264968ff..380c0aea840d 100644 --- a/drivers/nvme/target/rdma.c +++ b/drivers/nvme/target/rdma.c @@ -2098,6 +2098,7 @@ static void nvmet_rdma_remove_one(struct ib_device *ib_device, void *client_data mutex_unlock(&nvmet_rdma_queue_mutex); flush_workqueue(nvmet_wq); + flush_workqueue(nvmet_aen_wq); } static struct ib_client nvmet_rdma_ib_client = { diff --git a/drivers/nvme/target/tcp.c b/drivers/nvme/target/tcp.c index 15b2e05d1319..52a435499f9c 100644 --- a/drivers/nvme/target/tcp.c +++ b/drivers/nvme/target/tcp.c @@ -1585,7 +1585,6 @@ static void nvmet_tcp_free_cmd_data_in_buffers(struct nvmet_tcp_queue *queue) static void nvmet_tcp_release_queue_work(struct work_struct *w) { - struct page *page; struct nvmet_tcp_queue *queue = container_of(w, struct nvmet_tcp_queue, release_work); @@ -1602,14 +1601,13 @@ static void nvmet_tcp_release_queue_work(struct work_struct *w) nvmet_sq_destroy(&queue->nvme_sq); cancel_work_sync(&queue->io_work); nvmet_tcp_free_cmd_data_in_buffers(queue); - sock_release(queue->sock); + /* ->sock will be released by fput() */ + fput(queue->sock->file); nvmet_tcp_free_cmds(queue); if (queue->hdr_digest || queue->data_digest) nvmet_tcp_free_crypto(queue); ida_free(&nvmet_tcp_queue_ida, queue->idx); - - page = virt_to_head_page(queue->pf_cache.va); - __page_frag_cache_drain(page, queue->pf_cache.pagecnt_bias); + page_frag_cache_drain(&queue->pf_cache); kfree(queue); } @@ -1730,15 +1728,18 @@ static int nvmet_tcp_set_queue_sock(struct nvmet_tcp_queue *queue) return ret; } -static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, +static void nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, struct socket *newsock) { struct nvmet_tcp_queue *queue; + struct file *sock_file = NULL; int ret; queue = kzalloc(sizeof(*queue), GFP_KERNEL); - if (!queue) - return -ENOMEM; + if (!queue) { + ret = -ENOMEM; + goto out_release; + } INIT_WORK(&queue->release_work, nvmet_tcp_release_queue_work); INIT_WORK(&queue->io_work, nvmet_tcp_io_work); @@ -1751,10 +1752,16 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, init_llist_head(&queue->resp_list); INIT_LIST_HEAD(&queue->resp_send_list); + sock_file = sock_alloc_file(queue->sock, O_CLOEXEC, NULL); + if (IS_ERR(sock_file)) { + ret = PTR_ERR(sock_file); + goto out_free_queue; + } + queue->idx = ida_alloc(&nvmet_tcp_queue_ida, GFP_KERNEL); if (queue->idx < 0) { ret = queue->idx; - goto out_free_queue; + goto out_sock; } ret = nvmet_tcp_alloc_cmd(queue, &queue->connect); @@ -1775,7 +1782,7 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, if (ret) goto out_destroy_sq; - return 0; + return; out_destroy_sq: mutex_lock(&nvmet_tcp_queue_mutex); list_del_init(&queue->queue_list); @@ -1785,9 +1792,20 @@ static int nvmet_tcp_alloc_queue(struct nvmet_tcp_port *port, nvmet_tcp_free_cmd(&queue->connect); out_ida_remove: ida_free(&nvmet_tcp_queue_ida, queue->idx); + /* + * Drain the page fragment cache if any allocations were done. + * The first allocation using pf_cache is nvmet_tcp_alloc_cmd() + * for queue->connect after ida_alloc(). + */ + page_frag_cache_drain(&queue->pf_cache); +out_sock: + fput(queue->sock->file); out_free_queue: kfree(queue); - return ret; +out_release: + pr_err("failed to allocate queue, error %d\n", ret); + if (!sock_file) + sock_release(newsock); } static void nvmet_tcp_accept_work(struct work_struct *w) @@ -1804,11 +1822,7 @@ static void nvmet_tcp_accept_work(struct work_struct *w) pr_warn("failed to accept err=%d\n", ret); return; } - ret = nvmet_tcp_alloc_queue(port, newsock); - if (ret) { - pr_err("failed to allocate queue\n"); - sock_release(newsock); - } + nvmet_tcp_alloc_queue(port, newsock); } } diff --git a/drivers/nvmem/Kconfig b/drivers/nvmem/Kconfig index 5bc9c4874fe3..283134498fbc 100644 --- a/drivers/nvmem/Kconfig +++ b/drivers/nvmem/Kconfig @@ -1,6 +1,7 @@ # SPDX-License-Identifier: GPL-2.0-only menuconfig NVMEM bool "NVMEM Support" + imply NVMEM_LAYOUTS help Support for NVMEM(Non Volatile Memory) devices like EEPROM, EFUSES... diff --git a/drivers/nvmem/Makefile b/drivers/nvmem/Makefile index 423baf089515..cdd01fbf1313 100644 --- a/drivers/nvmem/Makefile +++ b/drivers/nvmem/Makefile @@ -5,6 +5,8 @@ obj-$(CONFIG_NVMEM) += nvmem_core.o nvmem_core-y := core.o +obj-$(CONFIG_NVMEM_LAYOUTS) += nvmem_layouts.o +nvmem_layouts-y := layouts.o obj-y += layouts/ # Devices diff --git a/drivers/nvmem/core.c b/drivers/nvmem/core.c index 5b819bbd451a..788111fb3495 100644 --- a/drivers/nvmem/core.c +++ b/drivers/nvmem/core.c @@ -55,9 +55,6 @@ static LIST_HEAD(nvmem_lookup_list); static BLOCKING_NOTIFIER_HEAD(nvmem_notifier); -static DEFINE_SPINLOCK(nvmem_layout_lock); -static LIST_HEAD(nvmem_layouts); - static int __nvmem_reg_read(struct nvmem_device *nvmem, unsigned int offset, void *val, size_t bytes) { @@ -754,105 +751,22 @@ static int nvmem_add_cells_from_fixed_layout(struct nvmem_device *nvmem) return err; } -int __nvmem_layout_register(struct nvmem_layout *layout, struct module *owner) +int nvmem_layout_register(struct nvmem_layout *layout) { - layout->owner = owner; - - spin_lock(&nvmem_layout_lock); - list_add(&layout->node, &nvmem_layouts); - spin_unlock(&nvmem_layout_lock); - - blocking_notifier_call_chain(&nvmem_notifier, NVMEM_LAYOUT_ADD, layout); + if (!layout->add_cells) + return -EINVAL; - return 0; + /* Populate the cells */ + return layout->add_cells(&layout->nvmem->dev, layout->nvmem); } -EXPORT_SYMBOL_GPL(__nvmem_layout_register); +EXPORT_SYMBOL_GPL(nvmem_layout_register); void nvmem_layout_unregister(struct nvmem_layout *layout) { - blocking_notifier_call_chain(&nvmem_notifier, NVMEM_LAYOUT_REMOVE, layout); - - spin_lock(&nvmem_layout_lock); - list_del(&layout->node); - spin_unlock(&nvmem_layout_lock); + /* Keep the API even with an empty stub in case we need it later */ } EXPORT_SYMBOL_GPL(nvmem_layout_unregister); -static struct nvmem_layout *nvmem_layout_get(struct nvmem_device *nvmem) -{ - struct device_node *layout_np; - struct nvmem_layout *l, *layout = ERR_PTR(-EPROBE_DEFER); - - layout_np = of_nvmem_layout_get_container(nvmem); - if (!layout_np) - return NULL; - - /* Fixed layouts don't have a matching driver */ - if (of_device_is_compatible(layout_np, "fixed-layout")) { - of_node_put(layout_np); - return NULL; - } - - /* - * In case the nvmem device was built-in while the layout was built as a - * module, we shall manually request the layout driver loading otherwise - * we'll never have any match. - */ - of_request_module(layout_np); - - spin_lock(&nvmem_layout_lock); - - list_for_each_entry(l, &nvmem_layouts, node) { - if (of_match_node(l->of_match_table, layout_np)) { - if (try_module_get(l->owner)) - layout = l; - - break; - } - } - - spin_unlock(&nvmem_layout_lock); - of_node_put(layout_np); - - return layout; -} - -static void nvmem_layout_put(struct nvmem_layout *layout) -{ - if (layout) - module_put(layout->owner); -} - -static int nvmem_add_cells_from_layout(struct nvmem_device *nvmem) -{ - struct nvmem_layout *layout = nvmem->layout; - int ret; - - if (layout && layout->add_cells) { - ret = layout->add_cells(&nvmem->dev, nvmem); - if (ret) - return ret; - } - - return 0; -} - -#if IS_ENABLED(CONFIG_OF) -/** - * of_nvmem_layout_get_container() - Get OF node to layout container. - * - * @nvmem: nvmem device. - * - * Return: a node pointer with refcount incremented or NULL if no - * container exists. Use of_node_put() on it when done. - */ -struct device_node *of_nvmem_layout_get_container(struct nvmem_device *nvmem) -{ - return of_get_child_by_name(nvmem->dev.of_node, "nvmem-layout"); -} -EXPORT_SYMBOL_GPL(of_nvmem_layout_get_container); -#endif - const void *nvmem_layout_get_match_data(struct nvmem_device *nvmem, struct nvmem_layout *layout) { @@ -860,7 +774,7 @@ const void *nvmem_layout_get_match_data(struct nvmem_device *nvmem, const struct of_device_id *match; layout_np = of_nvmem_layout_get_container(nvmem); - match = of_match_node(layout->of_match_table, layout_np); + match = of_match_node(layout->dev.driver->of_match_table, layout_np); return match ? match->data : NULL; } @@ -972,19 +886,6 @@ struct nvmem_device *nvmem_register(const struct nvmem_config *config) goto err_put_device; } - /* - * If the driver supplied a layout by config->layout, the module - * pointer will be NULL and nvmem_layout_put() will be a noop. - */ - nvmem->layout = config->layout ?: nvmem_layout_get(nvmem); - if (IS_ERR(nvmem->layout)) { - rval = PTR_ERR(nvmem->layout); - nvmem->layout = NULL; - - if (rval == -EPROBE_DEFER) - goto err_teardown_compat; - } - if (config->cells) { rval = nvmem_add_cells(nvmem, config->cells, config->ncells); if (rval) @@ -1005,24 +906,24 @@ struct nvmem_device *nvmem_register(const struct nvmem_config *config) if (rval) goto err_remove_cells; - rval = nvmem_add_cells_from_layout(nvmem); - if (rval) - goto err_remove_cells; - dev_dbg(&nvmem->dev, "Registering nvmem device %s\n", config->name); rval = device_add(&nvmem->dev); if (rval) goto err_remove_cells; + rval = nvmem_populate_layout(nvmem); + if (rval) + goto err_remove_dev; + blocking_notifier_call_chain(&nvmem_notifier, NVMEM_ADD, nvmem); return nvmem; +err_remove_dev: + device_del(&nvmem->dev); err_remove_cells: nvmem_device_remove_all_cells(nvmem); - nvmem_layout_put(nvmem->layout); -err_teardown_compat: if (config->compat) nvmem_sysfs_remove_compat(nvmem, config); err_put_device: @@ -1044,7 +945,7 @@ static void nvmem_device_release(struct kref *kref) device_remove_bin_file(nvmem->base_dev, &nvmem->eeprom); nvmem_device_remove_all_cells(nvmem); - nvmem_layout_put(nvmem->layout); + nvmem_destroy_layout(nvmem); device_unregister(&nvmem->dev); } @@ -1346,6 +1247,12 @@ nvmem_cell_get_from_lookup(struct device *dev, const char *con_id) return cell; } +static void nvmem_layout_module_put(struct nvmem_device *nvmem) +{ + if (nvmem->layout && nvmem->layout->dev.driver) + module_put(nvmem->layout->dev.driver->owner); +} + #if IS_ENABLED(CONFIG_OF) static struct nvmem_cell_entry * nvmem_find_cell_entry_by_node(struct nvmem_device *nvmem, struct device_node *np) @@ -1364,6 +1271,18 @@ nvmem_find_cell_entry_by_node(struct nvmem_device *nvmem, struct device_node *np return cell; } +static int nvmem_layout_module_get_optional(struct nvmem_device *nvmem) +{ + if (!nvmem->layout) + return 0; + + if (!nvmem->layout->dev.driver || + !try_module_get(nvmem->layout->dev.driver->owner)) + return -EPROBE_DEFER; + + return 0; +} + /** * of_nvmem_cell_get() - Get a nvmem cell from given device node and cell id * @@ -1426,16 +1345,27 @@ struct nvmem_cell *of_nvmem_cell_get(struct device_node *np, const char *id) return ERR_CAST(nvmem); } + ret = nvmem_layout_module_get_optional(nvmem); + if (ret) { + of_node_put(cell_np); + __nvmem_device_put(nvmem); + return ERR_PTR(ret); + } + cell_entry = nvmem_find_cell_entry_by_node(nvmem, cell_np); of_node_put(cell_np); if (!cell_entry) { + nvmem_layout_module_put(nvmem); + ret = nvmem->layout ? -EPROBE_DEFER : -ENOENT; __nvmem_device_put(nvmem); - return ERR_PTR(-ENOENT); + return ERR_PTR(ret); } cell = nvmem_create_cell(cell_entry, id, cell_index); - if (IS_ERR(cell)) + if (IS_ERR(cell)) { + nvmem_layout_module_put(nvmem); __nvmem_device_put(nvmem); + } return cell; } @@ -1548,6 +1478,7 @@ void nvmem_cell_put(struct nvmem_cell *cell) kfree_const(cell->id); kfree(cell); + nvmem_layout_module_put(nvmem); __nvmem_device_put(nvmem); } EXPORT_SYMBOL_GPL(nvmem_cell_put); @@ -2141,11 +2072,22 @@ EXPORT_SYMBOL_GPL(nvmem_dev_size); static int __init nvmem_init(void) { - return bus_register(&nvmem_bus_type); + int ret; + + ret = bus_register(&nvmem_bus_type); + if (ret) + return ret; + + ret = nvmem_layout_bus_register(); + if (ret) + bus_unregister(&nvmem_bus_type); + + return ret; } static void __exit nvmem_exit(void) { + nvmem_layout_bus_unregister(); bus_unregister(&nvmem_bus_type); } diff --git a/drivers/nvmem/internals.h b/drivers/nvmem/internals.h index 893553fbdf51..4946456c76c7 100644 --- a/drivers/nvmem/internals.h +++ b/drivers/nvmem/internals.h @@ -34,4 +34,25 @@ struct nvmem_device { void *priv; }; +#if IS_ENABLED(CONFIG_OF) +int nvmem_layout_bus_register(void); +void nvmem_layout_bus_unregister(void); +int nvmem_populate_layout(struct nvmem_device *nvmem); +void nvmem_destroy_layout(struct nvmem_device *nvmem); +#else /* CONFIG_OF */ +static inline int nvmem_layout_bus_register(void) +{ + return 0; +} + +static inline void nvmem_layout_bus_unregister(void) {} + +static inline int nvmem_populate_layout(struct nvmem_device *nvmem) +{ + return 0; +} + +static inline void nvmem_destroy_layout(struct nvmem_device *nvmem) { } +#endif /* CONFIG_OF */ + #endif /* ifndef _LINUX_NVMEM_INTERNALS_H */ diff --git a/drivers/nvmem/layouts.c b/drivers/nvmem/layouts.c new file mode 100644 index 000000000000..6a6aa58369ff --- /dev/null +++ b/drivers/nvmem/layouts.c @@ -0,0 +1,201 @@ +// SPDX-License-Identifier: GPL-2.0 +/* + * NVMEM layout bus handling + * + * Copyright (C) 2023 Bootlin + * Author: Miquel Raynal +#include +#include +#include +#include +#include +#include + +#include "internals.h" + +#define to_nvmem_layout_driver(drv) \ + (container_of((drv), struct nvmem_layout_driver, driver)) +#define to_nvmem_layout_device(_dev) \ + container_of((_dev), struct nvmem_layout, dev) + +static int nvmem_layout_bus_match(struct device *dev, struct device_driver *drv) +{ + return of_driver_match_device(dev, drv); +} + +static int nvmem_layout_bus_probe(struct device *dev) +{ + struct nvmem_layout_driver *drv = to_nvmem_layout_driver(dev->driver); + struct nvmem_layout *layout = to_nvmem_layout_device(dev); + + if (!drv->probe || !drv->remove) + return -EINVAL; + + return drv->probe(layout); +} + +static void nvmem_layout_bus_remove(struct device *dev) +{ + struct nvmem_layout_driver *drv = to_nvmem_layout_driver(dev->driver); + struct nvmem_layout *layout = to_nvmem_layout_device(dev); + + return drv->remove(layout); +} + +static struct bus_type nvmem_layout_bus_type = { + .name = "nvmem-layout", + .match = nvmem_layout_bus_match, + .probe = nvmem_layout_bus_probe, + .remove = nvmem_layout_bus_remove, +}; + +int nvmem_layout_driver_register(struct nvmem_layout_driver *drv) +{ + drv->driver.bus = &nvmem_layout_bus_type; + + return driver_register(&drv->driver); +} +EXPORT_SYMBOL_GPL(nvmem_layout_driver_register); + +void nvmem_layout_driver_unregister(struct nvmem_layout_driver *drv) +{ + driver_unregister(&drv->driver); +} +EXPORT_SYMBOL_GPL(nvmem_layout_driver_unregister); + +static void nvmem_layout_release_device(struct device *dev) +{ + struct nvmem_layout *layout = to_nvmem_layout_device(dev); + + of_node_put(layout->dev.of_node); + kfree(layout); +} + +static int nvmem_layout_create_device(struct nvmem_device *nvmem, + struct device_node *np) +{ + struct nvmem_layout *layout; + struct device *dev; + int ret; + + layout = kzalloc(sizeof(*layout), GFP_KERNEL); + if (!layout) + return -ENOMEM; + + /* Create a bidirectional link */ + layout->nvmem = nvmem; + nvmem->layout = layout; + + /* Device model registration */ + dev = &layout->dev; + device_initialize(dev); + dev->parent = &nvmem->dev; + dev->bus = &nvmem_layout_bus_type; + dev->release = nvmem_layout_release_device; + dev->coherent_dma_mask = DMA_BIT_MASK(32); + dev->dma_mask = &dev->coherent_dma_mask; + device_set_node(dev, of_fwnode_handle(of_node_get(np))); + of_device_make_bus_id(dev); + of_msi_configure(dev, dev->of_node); + + ret = device_add(dev); + if (ret) { + put_device(dev); + return ret; + } + + return 0; +} + +static const struct of_device_id of_nvmem_layout_skip_table[] = { + { .compatible = "fixed-layout", }, + {} +}; + +static int nvmem_layout_bus_populate(struct nvmem_device *nvmem, + struct device_node *layout_dn) +{ + int ret; + + /* Make sure it has a compatible property */ + if (!of_get_property(layout_dn, "compatible", NULL)) { + pr_debug("%s() - skipping %pOF, no compatible prop\n", + __func__, layout_dn); + return 0; + } + + /* Fixed layouts are parsed manually somewhere else for now */ + if (of_match_node(of_nvmem_layout_skip_table, layout_dn)) { + pr_debug("%s() - skipping %pOF node\n", __func__, layout_dn); + return 0; + } + + if (of_node_check_flag(layout_dn, OF_POPULATED_BUS)) { + pr_debug("%s() - skipping %pOF, already populated\n", + __func__, layout_dn); + + return 0; + } + + /* NVMEM layout buses expect only a single device representing the layout */ + ret = nvmem_layout_create_device(nvmem, layout_dn); + if (ret) + return ret; + + of_node_set_flag(layout_dn, OF_POPULATED_BUS); + + return 0; +} + +struct device_node *of_nvmem_layout_get_container(struct nvmem_device *nvmem) +{ + return of_get_child_by_name(nvmem->dev.of_node, "nvmem-layout"); +} +EXPORT_SYMBOL_GPL(of_nvmem_layout_get_container); + +/* + * Returns the number of devices populated, 0 if the operation was not relevant + * for this nvmem device, an error code otherwise. + */ +int nvmem_populate_layout(struct nvmem_device *nvmem) +{ + struct device_node *layout_dn; + int ret; + + layout_dn = of_nvmem_layout_get_container(nvmem); + if (!layout_dn) + return 0; + + /* Populate the layout device */ + device_links_supplier_sync_state_pause(); + ret = nvmem_layout_bus_populate(nvmem, layout_dn); + device_links_supplier_sync_state_resume(); + + of_node_put(layout_dn); + return ret; +} + +void nvmem_destroy_layout(struct nvmem_device *nvmem) +{ + struct device *dev; + + if (!nvmem->layout) + return; + + dev = &nvmem->layout->dev; + of_node_clear_flag(dev->of_node, OF_POPULATED_BUS); + device_unregister(dev); +} + +int nvmem_layout_bus_register(void) +{ + return bus_register(&nvmem_layout_bus_type); +} + +void nvmem_layout_bus_unregister(void) +{ + bus_unregister(&nvmem_layout_bus_type); +} diff --git a/drivers/nvmem/layouts/Kconfig b/drivers/nvmem/layouts/Kconfig index 7ff1ee1c1f05..9c6e672fc350 100644 --- a/drivers/nvmem/layouts/Kconfig +++ b/drivers/nvmem/layouts/Kconfig @@ -1,5 +1,11 @@ # SPDX-License-Identifier: GPL-2.0 +config NVMEM_LAYOUTS + bool + depends on OF + +if NVMEM_LAYOUTS + menu "Layout Types" config NVMEM_LAYOUT_SL28_VPD @@ -21,3 +27,5 @@ config NVMEM_LAYOUT_ONIE_TLV If unsure, say N. endmenu + +endif diff --git a/drivers/nvmem/layouts/onie-tlv.c b/drivers/nvmem/layouts/onie-tlv.c index 46ae52674b35..607821721c24 100644 --- a/drivers/nvmem/layouts/onie-tlv.c +++ b/drivers/nvmem/layouts/onie-tlv.c @@ -226,16 +226,32 @@ static int onie_tlv_parse_table(struct device *dev, struct nvmem_device *nvmem) return 0; } +static int onie_tlv_probe(struct nvmem_layout *layout) +{ + layout->add_cells = onie_tlv_parse_table; + + return nvmem_layout_register(layout); +} + +static void onie_tlv_remove(struct nvmem_layout *layout) +{ + nvmem_layout_unregister(layout); +} + static const struct of_device_id onie_tlv_of_match_table[] = { { .compatible = "onie,tlv-layout", }, {}, }; MODULE_DEVICE_TABLE(of, onie_tlv_of_match_table); -static struct nvmem_layout onie_tlv_layout = { - .name = "ONIE tlv layout", - .of_match_table = onie_tlv_of_match_table, - .add_cells = onie_tlv_parse_table, +static struct nvmem_layout_driver onie_tlv_layout = { + .driver = { + .owner = THIS_MODULE, + .name = "onie-tlv-layout", + .of_match_table = onie_tlv_of_match_table, + }, + .probe = onie_tlv_probe, + .remove = onie_tlv_remove, }; module_nvmem_layout_driver(onie_tlv_layout); diff --git a/drivers/nvmem/layouts/sl28vpd.c b/drivers/nvmem/layouts/sl28vpd.c index 26c7cf21b523..b8ffae646cc2 100644 --- a/drivers/nvmem/layouts/sl28vpd.c +++ b/drivers/nvmem/layouts/sl28vpd.c @@ -134,16 +134,32 @@ static int sl28vpd_add_cells(struct device *dev, struct nvmem_device *nvmem) return 0; } +static int sl28vpd_probe(struct nvmem_layout *layout) +{ + layout->add_cells = sl28vpd_add_cells; + + return nvmem_layout_register(layout); +} + +static void sl28vpd_remove(struct nvmem_layout *layout) +{ + nvmem_layout_unregister(layout); +} + static const struct of_device_id sl28vpd_of_match_table[] = { { .compatible = "kontron,sl28-vpd" }, {}, }; MODULE_DEVICE_TABLE(of, sl28vpd_of_match_table); -static struct nvmem_layout sl28vpd_layout = { - .name = "sl28-vpd", - .of_match_table = sl28vpd_of_match_table, - .add_cells = sl28vpd_add_cells, +static struct nvmem_layout_driver sl28vpd_layout = { + .driver = { + .owner = THIS_MODULE, + .name = "kontron-sl28vpd-layout", + .of_match_table = sl28vpd_of_match_table, + }, + .probe = sl28vpd_probe, + .remove = sl28vpd_remove, }; module_nvmem_layout_driver(sl28vpd_layout); diff --git a/drivers/of/device.c b/drivers/of/device.c index 1ca42ad9dd15..6e9572c4af83 100644 --- a/drivers/of/device.c +++ b/drivers/of/device.c @@ -304,3 +304,44 @@ int of_device_uevent_modalias(const struct device *dev, struct kobj_uevent_env * return 0; } EXPORT_SYMBOL_GPL(of_device_uevent_modalias); + +/** + * of_device_make_bus_id - Use the device node data to assign a unique name + * @dev: pointer to device structure that is linked to a device tree node + * + * This routine will first try using the translated bus address to + * derive a unique name. If it cannot, then it will prepend names from + * parent nodes until a unique name can be derived. + */ +void of_device_make_bus_id(struct device *dev) +{ + struct device_node *node = dev->of_node; + const __be32 *reg; + u64 addr; + u32 mask; + + /* Construct the name, using parent nodes if necessary to ensure uniqueness */ + while (node->parent) { + /* + * If the address can be translated, then that is as much + * uniqueness as we need. Make it the first component and return + */ + reg = of_get_property(node, "reg", NULL); + if (reg && (addr = of_translate_address(node, reg)) != OF_BAD_ADDR) { + if (!of_property_read_u32(node, "mask", &mask)) + dev_set_name(dev, dev_name(dev) ? "%llx.%x.%pOFn:%s" : "%llx.%x.%pOFn", + addr, ffs(mask) - 1, node, dev_name(dev)); + + else + dev_set_name(dev, dev_name(dev) ? "%llx.%pOFn:%s" : "%llx.%pOFn", + addr, node, dev_name(dev)); + return; + } + + /* format arguments only used if dev_name() resolves to NULL */ + dev_set_name(dev, dev_name(dev) ? "%s:%s" : "%s", + kbasename(node->full_name), dev_name(dev)); + node = node->parent; + } +} +EXPORT_SYMBOL_GPL(of_device_make_bus_id); diff --git a/drivers/of/irq.c b/drivers/of/irq.c index 1ec2a198c669..e027f39b88bf 100644 --- a/drivers/of/irq.c +++ b/drivers/of/irq.c @@ -660,6 +660,8 @@ void __init of_irq_init(const struct of_device_id *matches) list_for_each_entry_safe(desc, temp_desc, &intc_parent_list, list) { list_del(&desc->list); + of_node_put(desc->interrupt_parent); + of_node_put(desc->dev); kfree(desc); } err: diff --git a/drivers/of/overlay.c b/drivers/of/overlay.c index 1a52acff72ad..bee2cc714761 100644 --- a/drivers/of/overlay.c +++ b/drivers/of/overlay.c @@ -250,6 +250,9 @@ static struct property *dup_and_fixup_symbol_prop( if (!target_path) return NULL; target_path_len = strlen(target_path); + /* a root target renders as "/"; drop it to avoid "//" results */ + if (target_path_len == 1 && target_path[0] == '/' && path_tail_len) + target_path_len = 0; new_prop = kzalloc(sizeof(*new_prop), GFP_KERNEL); if (!new_prop) @@ -353,12 +356,13 @@ static int add_changeset_property(struct overlay_changeset *ovcs, return -ENOMEM; if (!prop) { - if (!target->in_livetree) { + ret = of_changeset_add_property(&ovcs->cset, target->np, + new_prop); + /* the detached node owns the property until the apply */ + if (!ret && !target->in_livetree) { new_prop->next = target->np->deadprops; target->np->deadprops = new_prop; } - ret = of_changeset_add_property(&ovcs->cset, target->np, - new_prop); } else { ret = of_changeset_update_property(&ovcs->cset, target->np, new_prop); @@ -850,6 +854,10 @@ static int init_overlay_changeset(struct overlay_changeset *ovcs, err_out: pr_err("%s() failed, ret = %d\n", __func__, ret); + /* let free_overlay_changeset() put the fragments set up so far */ + if (ovcs->fragments) + ovcs->count = cnt; + return ret; } @@ -860,7 +868,8 @@ static void free_overlay_changeset(struct overlay_changeset *ovcs) if (ovcs->cset.entries.next) of_changeset_destroy(&ovcs->cset); - if (ovcs->id) { + /* a failed idr_alloc() leaves its negative error in ovcs->id */ + if (ovcs->id > 0) { idr_remove(&ovcs_idr, ovcs->id); list_del(&ovcs->ovcs_list); ovcs->id = 0; diff --git a/drivers/of/platform.c b/drivers/of/platform.c index d1dfd857f2cd..3a36678cf05e 100644 --- a/drivers/of/platform.c +++ b/drivers/of/platform.c @@ -97,46 +97,6 @@ static const struct of_device_id of_skipped_node_table[] = { * mechanism for creating devices from device tree nodes. */ -/** - * of_device_make_bus_id - Use the device node data to assign a unique name - * @dev: pointer to device structure that is linked to a device tree node - * - * This routine will first try using the translated bus address to - * derive a unique name. If it cannot, then it will prepend names from - * parent nodes until a unique name can be derived. - */ -static void of_device_make_bus_id(struct device *dev) -{ - struct device_node *node = dev->of_node; - const __be32 *reg; - u64 addr; - u32 mask; - - /* Construct the name, using parent nodes if necessary to ensure uniqueness */ - while (node->parent) { - /* - * If the address can be translated, then that is as much - * uniqueness as we need. Make it the first component and return - */ - reg = of_get_property(node, "reg", NULL); - if (reg && (addr = of_translate_address(node, reg)) != OF_BAD_ADDR) { - if (!of_property_read_u32(node, "mask", &mask)) - dev_set_name(dev, dev_name(dev) ? "%llx.%x.%pOFn:%s" : "%llx.%x.%pOFn", - addr, ffs(mask) - 1, node, dev_name(dev)); - - else - dev_set_name(dev, dev_name(dev) ? "%llx.%pOFn:%s" : "%llx.%pOFn", - addr, node, dev_name(dev)); - return; - } - - /* format arguments only used if dev_name() resolves to NULL */ - dev_set_name(dev, dev_name(dev) ? "%s:%s" : "%s", - kbasename(node->full_name), dev_name(dev)); - node = node->parent; - } -} - /** * of_device_alloc - Allocate and initialize an of_device * @np: device node to assign to device diff --git a/drivers/of/unittest.c b/drivers/of/unittest.c index 9af9d3f7418f..9109667a3510 100644 --- a/drivers/of/unittest.c +++ b/drivers/of/unittest.c @@ -1636,7 +1636,6 @@ static void attach_node_and_children(struct device_node *np) */ static int __init unittest_data_add(void) { - void *unittest_data; void *unittest_data_align; struct device_node *unittest_data_node = NULL, *np; /* @@ -1655,7 +1654,7 @@ static int __init unittest_data_add(void) } /* creating copy */ - unittest_data = kmalloc(size + FDT_ALIGN_SIZE, GFP_KERNEL); + void *unittest_data __free(kfree) = kmalloc(size + FDT_ALIGN_SIZE, GFP_KERNEL); if (!unittest_data) return -ENOMEM; @@ -1665,12 +1664,10 @@ static int __init unittest_data_add(void) ret = of_fdt_unflatten_tree(unittest_data_align, NULL, &unittest_data_node); if (!ret) { pr_warn("%s: unflatten testcases tree failed\n", __func__); - kfree(unittest_data); return -ENODATA; } if (!unittest_data_node) { pr_warn("%s: testcases tree is empty\n", __func__); - kfree(unittest_data); return -ENODATA; } @@ -1693,6 +1690,7 @@ static int __init unittest_data_add(void) of_aliases = of_find_node_by_path("/aliases"); of_chosen = of_find_node_by_path("/chosen"); of_overlay_mutex_unlock(); + (void)no_free_ptr(unittest_data); return 0; } @@ -1715,6 +1713,7 @@ static int __init unittest_data_add(void) of_overlay_mutex_unlock(); + (void)no_free_ptr(unittest_data); return 0; } diff --git a/drivers/opp/core.c b/drivers/opp/core.c index 218cb2e7f360..02db807adad4 100644 --- a/drivers/opp/core.c +++ b/drivers/opp/core.c @@ -1998,11 +1998,10 @@ int _opp_add(struct device *dev, struct dev_pm_opp *new_opp, return ret; } - list_add(&new_opp->node, head); - mutex_unlock(&opp_table->lock); - new_opp->opp_table = opp_table; kref_init(&new_opp->kref); + list_add(&new_opp->node, head); + mutex_unlock(&opp_table->lock); opp_debug_create_one(new_opp, opp_table); diff --git a/drivers/pci/controller/dwc/pci-imx6.c b/drivers/pci/controller/dwc/pci-imx6.c index c438cf888ad5..20ffa27ec982 100644 --- a/drivers/pci/controller/dwc/pci-imx6.c +++ b/drivers/pci/controller/dwc/pci-imx6.c @@ -60,6 +60,11 @@ enum imx6_pcie_variants { #define IMX6_PCIE_FLAG_IMX6_PHY BIT(0) #define IMX6_PCIE_FLAG_IMX6_SPEED_CHANGE BIT(1) #define IMX6_PCIE_FLAG_SUPPORTS_SUSPEND BIT(2) +/* + * Because of ERR005723 (PCIe does not support L2 power down) we need to + * workaround suspend resume on some devices which are affected by this errata. + */ +#define IMX6_PCIE_FLAG_BROKEN_SUSPEND BIT(9) #define IMX6_PCIE_MAX_CLKS 6 @@ -1171,9 +1176,19 @@ static int imx6_pcie_suspend_noirq(struct device *dev) return 0; imx6_pcie_msi_save_restore(imx6_pcie, true); - imx6_pcie_pm_turnoff(imx6_pcie); - imx6_pcie_stop_link(imx6_pcie->pci); - imx6_pcie_host_exit(pp); + if (imx6_pcie->drvdata->flags & IMX6_PCIE_FLAG_BROKEN_SUSPEND) { + /* + * The minimum for a workaround would be to set PERST# and to + * set the PCIE_TEST_PD flag. However, we can also disable the + * clock which saves some power. + */ + imx6_pcie_assert_core_reset(imx6_pcie); + imx6_pcie_disable_ref_clk(imx6_pcie); + } else { + imx6_pcie_pm_turnoff(imx6_pcie); + imx6_pcie_stop_link(imx6_pcie->pci); + imx6_pcie_host_exit(pp); + } return 0; } @@ -1187,14 +1202,32 @@ static int imx6_pcie_resume_noirq(struct device *dev) if (!(imx6_pcie->drvdata->flags & IMX6_PCIE_FLAG_SUPPORTS_SUSPEND)) return 0; - ret = imx6_pcie_host_init(pp); - if (ret) - return ret; - imx6_pcie_msi_save_restore(imx6_pcie, false); - dw_pcie_setup_rc(pp); + if (imx6_pcie->drvdata->flags & IMX6_PCIE_FLAG_BROKEN_SUSPEND) { + ret = imx6_pcie_enable_ref_clk(imx6_pcie); + if (ret) + return ret; + ret = imx6_pcie_deassert_core_reset(imx6_pcie); + if (ret) + return ret; + /* + * Using PCIE_TEST_PD seems to disable MSI and powers down the + * root complex. This is why we have to setup the rc again and + * why we have to restore the MSI register. + */ + ret = dw_pcie_setup_rc(&imx6_pcie->pci->pp); + if (ret) + return ret; + imx6_pcie_msi_save_restore(imx6_pcie, false); + } else { + ret = imx6_pcie_host_init(pp); + if (ret) + return ret; + imx6_pcie_msi_save_restore(imx6_pcie, false); + dw_pcie_setup_rc(pp); - if (imx6_pcie->link_is_up) - imx6_pcie_start_link(imx6_pcie->pci); + if (imx6_pcie->link_is_up) + imx6_pcie_start_link(imx6_pcie->pci); + } return 0; } @@ -1418,7 +1451,9 @@ static const struct imx6_pcie_drvdata drvdata[] = { [IMX6Q] = { .variant = IMX6Q, .flags = IMX6_PCIE_FLAG_IMX6_PHY | - IMX6_PCIE_FLAG_IMX6_SPEED_CHANGE, + IMX6_PCIE_FLAG_IMX6_SPEED_CHANGE | + IMX6_PCIE_FLAG_BROKEN_SUSPEND | + IMX6_PCIE_FLAG_SUPPORTS_SUSPEND, .dbi_length = 0x200, .gpr = "fsl,imx6q-iomuxc-gpr", .clk_names = imx6q_clks, diff --git a/drivers/pci/controller/dwc/pcie-designware-ep.c b/drivers/pci/controller/dwc/pcie-designware-ep.c index 26ad643fb424..2860645d65bf 100644 --- a/drivers/pci/controller/dwc/pcie-designware-ep.c +++ b/drivers/pci/controller/dwc/pcie-designware-ep.c @@ -247,6 +247,23 @@ static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no, reg = PCI_BASE_ADDRESS_0 + (4 * bar) + func_offset; + /* + * Certain EPF drivers dynamically change the physical address of a BAR + * (i.e. they call set_bar() twice, without ever calling clear_bar(), as + * calling clear_bar() would clear the BAR's PCI address assigned by the + * host). + */ + if (ep->epf_bar[bar]) { + /* + * We can only dynamically change a BAR if the new BAR size and + * BAR flags do not differ from the existing configuration. + */ + if (ep->epf_bar[bar]->barno != bar || + ep->epf_bar[bar]->size != size || + ep->epf_bar[bar]->flags != flags) + return -EINVAL; + } + if (!(flags & PCI_BASE_ADDRESS_SPACE)) type = PCIE_ATU_TYPE_MEM; else @@ -256,6 +273,10 @@ static int dw_pcie_ep_set_bar(struct pci_epc *epc, u8 func_no, u8 vfunc_no, if (ret) return ret; + /* + * When dynamically changing a BAR, skip writing the BAR reg, as + * that would clear the BAR's PCI address assigned by the host. + */ if (ep->epf_bar[bar]) return 0; @@ -614,6 +635,9 @@ int dw_pcie_ep_raise_msix_irq(struct dw_pcie_ep *ep, u8 func_no, writel(msg_data, ep->msi_mem + aligned_offset); + /* flush posted write before unmap */ + readl(ep->msi_mem + aligned_offset); + dw_pcie_ep_unmap_addr(epc, func_no, 0, ep->msi_mem_phys); return 0; diff --git a/drivers/pci/endpoint/functions/pci-epf-ntb.c b/drivers/pci/endpoint/functions/pci-epf-ntb.c index cba6bab067b6..ee3ce8a02529 100644 --- a/drivers/pci/endpoint/functions/pci-epf-ntb.c +++ b/drivers/pci/endpoint/functions/pci-epf-ntb.c @@ -2075,6 +2075,11 @@ static int __init epf_ntb_init(void) kpcintb_workqueue = alloc_workqueue("kpcintb", WQ_MEM_RECLAIM | WQ_HIGHPRI, 0); + if (!kpcintb_workqueue) { + pr_err("Failed to allocate kpcintb workqueue\n"); + return -ENOMEM; + } + ret = pci_epf_register_driver(&epf_ntb_driver); if (ret) { destroy_workqueue(kpcintb_workqueue); diff --git a/drivers/pci/endpoint/functions/pci-epf-test.c b/drivers/pci/endpoint/functions/pci-epf-test.c index 4cf20ca25ea7..cf6e00802b53 100644 --- a/drivers/pci/endpoint/functions/pci-epf-test.c +++ b/drivers/pci/endpoint/functions/pci-epf-test.c @@ -706,6 +706,7 @@ static void pci_epf_test_unbind(struct pci_epf *epf) epf_bar); pci_epf_free_space(epf, epf_test->reg[bar], bar, PRIMARY_INTERFACE); + epf_test->reg[bar] = NULL; } } } @@ -740,6 +741,7 @@ static int pci_epf_test_set_bar(struct pci_epf *epf) if (ret) { pci_epf_free_space(epf, epf_test->reg[bar], bar, PRIMARY_INTERFACE); + epf_test->reg[bar] = NULL; dev_err(dev, "Failed to set BAR%d\n", bar); if (bar == test_reg_bar) return ret; @@ -753,19 +755,11 @@ static int pci_epf_test_core_init(struct pci_epf *epf) { struct pci_epf_test *epf_test = epf_get_drvdata(epf); struct pci_epf_header *header = epf->header; - const struct pci_epc_features *epc_features; + const struct pci_epc_features *epc_features = epf_test->epc_features; struct pci_epc *epc = epf->epc; struct device *dev = &epf->dev; - bool msix_capable = false; - bool msi_capable = true; int ret; - epc_features = pci_epc_get_features(epc, epf->func_no, epf->vfunc_no); - if (epc_features) { - msix_capable = epc_features->msix_capable; - msi_capable = epc_features->msi_capable; - } - if (epf->vfunc_no <= 1) { ret = pci_epc_write_header(epc, epf->func_no, epf->vfunc_no, header); if (ret) { @@ -778,7 +772,7 @@ static int pci_epf_test_core_init(struct pci_epf *epf) if (ret) return ret; - if (msi_capable) { + if (epc_features->msi_capable) { ret = pci_epc_set_msi(epc, epf->func_no, epf->vfunc_no, epf->msi_interrupts); if (ret) { @@ -787,7 +781,7 @@ static int pci_epf_test_core_init(struct pci_epf *epf) } } - if (msix_capable) { + if (epc_features->msix_capable) { ret = pci_epc_set_msix(epc, epf->func_no, epf->vfunc_no, epf->msix_interrupts, epf_test->test_reg_bar, diff --git a/drivers/pci/endpoint/functions/pci-epf-vntb.c b/drivers/pci/endpoint/functions/pci-epf-vntb.c index 0ebd37e7ecd0..22364ebd73ed 100644 --- a/drivers/pci/endpoint/functions/pci-epf-vntb.c +++ b/drivers/pci/endpoint/functions/pci-epf-vntb.c @@ -1432,6 +1432,11 @@ static int __init epf_ntb_init(void) kpcintb_workqueue = alloc_workqueue("kpcintb", WQ_MEM_RECLAIM | WQ_HIGHPRI, 0); + if (!kpcintb_workqueue) { + pr_err("Failed to allocate kpcintb workqueue\n"); + return -ENOMEM; + } + ret = pci_epf_register_driver(&epf_ntb_driver); if (ret) { destroy_workqueue(kpcintb_workqueue); diff --git a/drivers/pci/hotplug/pnv_php.c b/drivers/pci/hotplug/pnv_php.c index ec7828ad6661..2bd2b7599d1c 100644 --- a/drivers/pci/hotplug/pnv_php.c +++ b/drivers/pci/hotplug/pnv_php.c @@ -799,7 +799,7 @@ static struct pnv_php_slot *pnv_php_alloc_slot(struct device_node *dn) if (dn->child && PCI_DN(dn->child)) php_slot->slot_no = PCI_SLOT(PCI_DN(dn->child)->devfn); else - php_slot->slot_no = -1; /* Placeholder slot */ + php_slot->slot_no = PCI_SLOT_PLACEHOLDER; /* Placeholder slot */ kref_init(&php_slot->kref); php_slot->state = PNV_PHP_STATE_INITIALIZED; diff --git a/drivers/pci/hotplug/rpaphp_slot.c b/drivers/pci/hotplug/rpaphp_slot.c index 779eab12e981..ac8c512190df 100644 --- a/drivers/pci/hotplug/rpaphp_slot.c +++ b/drivers/pci/hotplug/rpaphp_slot.c @@ -85,7 +85,7 @@ int rpaphp_register_slot(struct slot *slot) struct device_node *child; u32 my_index; int retval; - int slotno = -1; + int slotno = PCI_SLOT_PLACEHOLDER; dbg("%s registering slot:path[%pOF] index[%x], name[%s] pdomain[%x] type[%d]\n", __func__, slot->dn, slot->index, slot->name, diff --git a/drivers/pci/pci-driver.c b/drivers/pci/pci-driver.c index cfb8b81250db..02c63637dcef 100644 --- a/drivers/pci/pci-driver.c +++ b/drivers/pci/pci-driver.c @@ -29,6 +29,47 @@ struct pci_dynid { struct pci_device_id id; }; +/** + * do_pci_add_dynid - Add a new PCI device ID to this driver and re-probe + * @drv: target PCI driver + * @id: ID to be added + * @check_dup: whether to check if matching ID is already present + * + * Add a new dynamic PCI device ID to this driver and causes the driver to + * probe for all devices again. @drv must have been registered prior to calling + * this function. + * + * Context: Does GFP_KERNEL allocation. + * + * Return: 0 on success, -errno on failure. + */ +static int do_pci_add_dynid(struct pci_driver *drv, + const struct pci_device_id *id, + bool check_dup) +{ + struct pci_dynid *dynid, *existing_dynid; + + dynid = kzalloc(sizeof(*dynid), GFP_KERNEL); + if (!dynid) + return -ENOMEM; + + dynid->id = *id; + + scoped_guard(spinlock, &drv->dynids.lock) { + if (check_dup) { + list_for_each_entry(existing_dynid, &drv->dynids.list, node) { + if (pci_match_one_id(&existing_dynid->id, id)) { + kfree(dynid); + return -EEXIST; + } + } + } + list_add_tail(&dynid->node, &drv->dynids.list); + } + + return driver_attach(&drv->driver); +} + /** * pci_add_dynid - add a new PCI device ID to this driver and re-probe devices * @drv: target pci driver @@ -56,25 +97,17 @@ int pci_add_dynid(struct pci_driver *drv, unsigned int class, unsigned int class_mask, unsigned long driver_data) { - struct pci_dynid *dynid; - - dynid = kzalloc(sizeof(*dynid), GFP_KERNEL); - if (!dynid) - return -ENOMEM; - - dynid->id.vendor = vendor; - dynid->id.device = device; - dynid->id.subvendor = subvendor; - dynid->id.subdevice = subdevice; - dynid->id.class = class; - dynid->id.class_mask = class_mask; - dynid->id.driver_data = driver_data; - - spin_lock(&drv->dynids.lock); - list_add_tail(&dynid->node, &drv->dynids.list); - spin_unlock(&drv->dynids.lock); - - return driver_attach(&drv->driver); + struct pci_device_id id = { + .vendor = vendor, + .device = device, + .subvendor = subvendor, + .subdevice = subdevice, + .class = class, + .class_mask = class_mask, + .driver_data = driver_data, + }; + + return do_pci_add_dynid(drv, &id, false); } EXPORT_SYMBOL_GPL(pci_add_dynid); @@ -90,6 +123,31 @@ static void pci_free_dynids(struct pci_driver *drv) spin_unlock(&drv->dynids.lock); } +/** + * do_pci_match_id - See if a PCI ID matches a given pci_id table + * @ids: array of PCI device ID structures to search in + * @dev_id: the actual PCI device ID structure to match against. + * @include_override_only: also match against device ID entries marked as + * override only. + * + * Return: the matching pci_device_id structure or %NULL if there is no match. + */ +static const struct pci_device_id * +do_pci_match_id(const struct pci_device_id *ids, + const struct pci_device_id *dev_id, + bool include_override_only) +{ + if (ids) { + while (ids->vendor || ids->subvendor || ids->class_mask) { + if ((!ids->override_only || include_override_only) && + pci_match_one_id(ids, dev_id)) + return ids; + ids++; + } + } + return NULL; +} + /** * pci_match_id - See if a PCI device matches a given pci_id table * @ids: array of PCI device ID structures to search in @@ -105,14 +163,9 @@ static void pci_free_dynids(struct pci_driver *drv) const struct pci_device_id *pci_match_id(const struct pci_device_id *ids, struct pci_dev *dev) { - if (ids) { - while (ids->vendor || ids->subvendor || ids->class_mask) { - if (pci_match_one_device(ids, dev)) - return ids; - ids++; - } - } - return NULL; + struct pci_device_id dev_id = pci_id_from_device(dev); + + return do_pci_match_id(ids, &dev_id, true); } EXPORT_SYMBOL(pci_match_id); @@ -127,6 +180,7 @@ static const struct pci_device_id pci_device_id_any = { * pci_match_device - See if a device matches a driver's list of IDs * @drv: the PCI driver to match against * @dev: the PCI device structure to match against + * @id_copy: place to store copy of pci_device_id for dynamic ID * * Used by a driver to check whether a PCI device is in its list of * supported devices or in the dynids list, which may have been augmented @@ -134,45 +188,37 @@ static const struct pci_device_id pci_device_id_any = { * structure or %NULL if there is no match. */ static const struct pci_device_id *pci_match_device(struct pci_driver *drv, - struct pci_dev *dev) + struct pci_dev *dev, + struct pci_device_id *id_copy) { - struct pci_dynid *dynid; - const struct pci_device_id *found_id = NULL, *ids; + const struct pci_device_id *found_id = NULL; + struct pci_device_id dev_id; + int ret; /* When driver_override is set, only bind to the matching driver */ - if (dev->driver_override && strcmp(dev->driver_override, drv->name)) + ret = device_match_driver_override(&dev->dev, &drv->driver); + if (ret == 0) return NULL; + dev_id = pci_id_from_device(dev); /* Look at the dynamic ids first, before the static ones */ - spin_lock(&drv->dynids.lock); - list_for_each_entry(dynid, &drv->dynids.list, node) { - if (pci_match_one_device(&dynid->id, dev)) { - found_id = &dynid->id; - break; + scoped_guard(spinlock, &drv->dynids.lock) { + struct pci_dynid *dynid; + + list_for_each_entry(dynid, &drv->dynids.list, node) { + if (pci_match_one_id(&dynid->id, &dev_id)) { + *id_copy = dynid->id; + return id_copy; + } } } - spin_unlock(&drv->dynids.lock); + found_id = do_pci_match_id(drv->id_table, &dev_id, ret > 0); if (found_id) return found_id; - for (ids = drv->id_table; (found_id = pci_match_id(ids, dev)); - ids = found_id + 1) { - /* - * The match table is split based on driver_override. - * In case override_only was set, enforce driver_override - * matching. - */ - if (found_id->override_only) { - if (dev->driver_override) - return found_id; - } else { - return found_id; - } - } - /* driver_override will always match, send a dummy id */ - if (dev->driver_override) + if (ret > 0) return &pci_device_id_any; return NULL; } @@ -190,36 +236,22 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf, { struct pci_driver *pdrv = to_pci_driver(driver); const struct pci_device_id *ids = pdrv->id_table; - u32 vendor, device, subvendor = PCI_ANY_ID, - subdevice = PCI_ANY_ID, class = 0, class_mask = 0; - unsigned long driver_data = 0; + struct pci_device_id id = { + .subvendor = PCI_ANY_ID, + .subdevice = PCI_ANY_ID + }; int fields; int retval = 0; fields = sscanf(buf, "%x %x %x %x %x %x %lx", - &vendor, &device, &subvendor, &subdevice, - &class, &class_mask, &driver_data); + &id.vendor, &id.device, &id.subvendor, &id.subdevice, + &id.class, &id.class_mask, &id.driver_data); if (fields < 2) return -EINVAL; if (fields != 7) { - struct pci_dev *pdev = kzalloc(sizeof(*pdev), GFP_KERNEL); - if (!pdev) - return -ENOMEM; - - pdev->vendor = vendor; - pdev->device = device; - pdev->subsystem_vendor = subvendor; - pdev->subsystem_device = subdevice; - pdev->class = class; - - if (pci_match_device(pdrv, pdev)) - retval = -EEXIST; - - kfree(pdev); - - if (retval) - return retval; + if (do_pci_match_id(pdrv->id_table, &id, false)) + return -EEXIST; } /* Only accept driver_data values that match an existing id_table @@ -227,7 +259,7 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf, if (ids) { retval = -EINVAL; while (ids->vendor || ids->subvendor || ids->class_mask) { - if (driver_data == ids->driver_data) { + if (id.driver_data == ids->driver_data) { retval = 0; break; } @@ -237,8 +269,7 @@ static ssize_t new_id_store(struct device_driver *driver, const char *buf, return retval; } - retval = pci_add_dynid(pdrv, vendor, device, subvendor, subdevice, - class, class_mask, driver_data); + retval = do_pci_add_dynid(pdrv, &id, fields != 7); if (retval) return retval; return count; @@ -407,12 +438,13 @@ static int pci_call_probe(struct pci_driver *drv, struct pci_dev *dev, static int __pci_device_probe(struct pci_driver *drv, struct pci_dev *pci_dev) { const struct pci_device_id *id; + struct pci_device_id id_copy; int error = 0; if (drv->probe) { error = -ENODEV; - id = pci_match_device(drv, pci_dev); + id = pci_match_device(drv, pci_dev, &id_copy); if (id) error = pci_call_probe(drv, pci_dev, id); } @@ -432,7 +464,7 @@ void __weak pcibios_free_irq(struct pci_dev *dev) static inline bool pci_device_can_probe(struct pci_dev *pdev) { return (!pdev->is_virtfn || pdev->physfn->sriov->drivers_autoprobe || - pdev->driver_override); + device_has_driver_override(&pdev->dev)); } #else static inline bool pci_device_can_probe(struct pci_dev *pdev) @@ -1518,12 +1550,13 @@ static int pci_bus_match(struct device *dev, struct device_driver *drv) struct pci_dev *pci_dev = to_pci_dev(dev); struct pci_driver *pci_drv; const struct pci_device_id *found_id; + struct pci_device_id id_copy; if (!pci_dev->match_driver) return 0; pci_drv = to_pci_driver(drv); - found_id = pci_match_device(pci_drv, pci_dev); + found_id = pci_match_device(pci_drv, pci_dev, &id_copy); if (found_id) return 1; @@ -1683,6 +1716,7 @@ static void pci_dma_cleanup(struct device *dev) struct bus_type pci_bus_type = { .name = "pci", + .driver_override = true, .match = pci_bus_match, .uevent = pci_uevent, .probe = pci_device_probe, diff --git a/drivers/pci/pci-sysfs.c b/drivers/pci/pci-sysfs.c index 346630f3a6f2..2bfbc917fa8c 100644 --- a/drivers/pci/pci-sysfs.c +++ b/drivers/pci/pci-sysfs.c @@ -608,33 +608,6 @@ static ssize_t devspec_show(struct device *dev, static DEVICE_ATTR_RO(devspec); #endif -static ssize_t driver_override_store(struct device *dev, - struct device_attribute *attr, - const char *buf, size_t count) -{ - struct pci_dev *pdev = to_pci_dev(dev); - int ret; - - ret = driver_set_override(dev, &pdev->driver_override, buf, count); - if (ret) - return ret; - - return count; -} - -static ssize_t driver_override_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - struct pci_dev *pdev = to_pci_dev(dev); - ssize_t len; - - device_lock(dev); - len = sysfs_emit(buf, "%s\n", pdev->driver_override); - device_unlock(dev); - return len; -} -static DEVICE_ATTR_RW(driver_override); - static struct attribute *pci_dev_attrs[] = { &dev_attr_power_state.attr, &dev_attr_resource.attr, @@ -662,7 +635,6 @@ static struct attribute *pci_dev_attrs[] = { #ifdef CONFIG_OF &dev_attr_devspec.attr, #endif - &dev_attr_driver_override.attr, &dev_attr_ari_enabled.attr, NULL, }; @@ -1437,71 +1409,84 @@ static const struct attribute_group pci_dev_reset_attr_group = { .is_visible = pci_dev_reset_attr_is_visible, }; +static ssize_t __resource_resize_show(struct device *dev, int n, char *buf) +{ + struct pci_dev *pdev = to_pci_dev(dev); + ssize_t ret; + + pci_config_pm_runtime_get(pdev); + + ret = sysfs_emit(buf, "%016llx\n", + (u64)pci_rebar_get_possible_sizes(pdev, n)); + + pci_config_pm_runtime_put(pdev); + + return ret; +} + +static ssize_t __resource_resize_store(struct device *dev, int n, + const char *buf, size_t count) +{ + struct pci_dev *pdev = to_pci_dev(dev); + unsigned long size; + int ret; + u16 cmd; + + if (!capable(CAP_SYS_ADMIN)) + return -EPERM; + + if (kstrtoul(buf, 0, &size) < 0) + return -EINVAL; + + device_lock(dev); + if (dev->driver) { + ret = -EBUSY; + goto unlock; + } + + pci_config_pm_runtime_get(pdev); + + if ((pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA) { + ret = aperture_remove_conflicting_pci_devices(pdev, + "resourceN_resize"); + if (ret) + goto pm_put; + } + + pci_read_config_word(pdev, PCI_COMMAND, &cmd); + pci_write_config_word(pdev, PCI_COMMAND, + cmd & ~PCI_COMMAND_MEMORY); + + pci_remove_resource_files(pdev); + + ret = pci_resize_resource(pdev, n, size, 0); + + pci_assign_unassigned_bus_resources(pdev->bus); + + if (pci_create_resource_files(pdev)) + pci_warn(pdev, "Failed to recreate resource files after BAR resizing\n"); + + pci_write_config_word(pdev, PCI_COMMAND, cmd); +pm_put: + pci_config_pm_runtime_put(pdev); +unlock: + device_unlock(dev); + + return ret ? ret : count; +} + #define pci_dev_resource_resize_attr(n) \ static ssize_t resource##n##_resize_show(struct device *dev, \ struct device_attribute *attr, \ - char * buf) \ + char *buf) \ { \ - struct pci_dev *pdev = to_pci_dev(dev); \ - ssize_t ret; \ - \ - pci_config_pm_runtime_get(pdev); \ - \ - ret = sysfs_emit(buf, "%016llx\n", \ - (u64)pci_rebar_get_possible_sizes(pdev, n)); \ - \ - pci_config_pm_runtime_put(pdev); \ - \ - return ret; \ + return __resource_resize_show(dev, n, buf); \ } \ - \ static ssize_t resource##n##_resize_store(struct device *dev, \ struct device_attribute *attr,\ const char *buf, size_t count)\ { \ - struct pci_dev *pdev = to_pci_dev(dev); \ - unsigned long size; \ - int ret; \ - u16 cmd; \ - \ - if (kstrtoul(buf, 0, &size) < 0) \ - return -EINVAL; \ - \ - device_lock(dev); \ - if (dev->driver) { \ - ret = -EBUSY; \ - goto unlock; \ - } \ - \ - pci_config_pm_runtime_get(pdev); \ - \ - if ((pdev->class >> 8) == PCI_CLASS_DISPLAY_VGA) { \ - ret = aperture_remove_conflicting_pci_devices(pdev, \ - "resourceN_resize"); \ - if (ret) \ - goto pm_put; \ - } \ - \ - pci_read_config_word(pdev, PCI_COMMAND, &cmd); \ - pci_write_config_word(pdev, PCI_COMMAND, \ - cmd & ~PCI_COMMAND_MEMORY); \ - \ - pci_remove_resource_files(pdev); \ - \ - ret = pci_resize_resource(pdev, n, size, 0); \ - \ - pci_assign_unassigned_bus_resources(pdev->bus); \ - \ - if (pci_create_resource_files(pdev)) \ - pci_warn(pdev, "Failed to recreate resource files after BAR resizing\n");\ - \ - pci_write_config_word(pdev, PCI_COMMAND, cmd); \ -pm_put: \ - pci_config_pm_runtime_put(pdev); \ -unlock: \ - device_unlock(dev); \ - \ - return ret ? ret : count; \ + return __resource_resize_store(dev, n, buf, count); \ } \ static DEVICE_ATTR_RW(resource##n##_resize) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index a3fa91d0b509..e8186da64463 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -3868,8 +3868,21 @@ int pci_enable_atomic_ops_to_root(struct pci_dev *dev, u32 cap_mask) } root = pcie_find_root_port(dev); - if (!root) + if (!root) { + + /* + * A hypervisor may expose a topology with the root port + * not visible to the guest. If the hypervisor has already + * set AtomicOp Requester Enable in the endpoint, we assume + * it has verified support in the root port, so it is safe + * for the driver to use AtomicOps. + */ + pcie_capability_read_dword(dev, PCI_EXP_DEVCTL2, &ctl2); + if (ctl2 & PCI_EXP_DEVCTL2_ATOMIC_REQ) + return 0; + return -EINVAL; + } pcie_capability_read_dword(root, PCI_EXP_DEVCAP2, &cap); if ((cap & cap_mask) != cap_mask) @@ -5255,8 +5268,9 @@ static int pci_reset_hotplug_slot(struct hotplug_slot *hotplug, bool probe) static int pci_dev_reset_slot_function(struct pci_dev *dev, bool probe) { - if (dev->multifunction || dev->subordinate || !dev->slot || - dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET) + if (dev->subordinate || !dev->slot || + dev->dev_flags & PCI_DEV_FLAGS_NO_BUS_RESET || + (dev->multifunction && !dev->slot->per_func_slot)) return -ENOTTY; return pci_reset_hotplug_slot(dev->slot->hotplug, probe); diff --git a/drivers/pci/pci.h b/drivers/pci/pci.h index 278af5b263bf..c2ab22a35301 100644 --- a/drivers/pci/pci.h +++ b/drivers/pci/pci.h @@ -222,21 +222,40 @@ extern unsigned long pci_hotplug_mmio_pref_size; extern unsigned long pci_hotplug_bus_size; /** - * pci_match_one_device - Tell if a PCI device structure has a matching - * PCI device id structure - * @id: single PCI device id structure to match - * @dev: the PCI device structure to match against + * pci_id_from_device - Obtain a pci_device_id from a PCI device + * @dev: the PCI device * - * Returns the matching pci_device_id structure or %NULL if there is no match. + * Return: a pci_device_id filled. */ -static inline const struct pci_device_id * -pci_match_one_device(const struct pci_device_id *id, const struct pci_dev *dev) +static inline struct pci_device_id pci_id_from_device(const struct pci_dev *dev) { - if ((id->vendor == PCI_ANY_ID || id->vendor == dev->vendor) && - (id->device == PCI_ANY_ID || id->device == dev->device) && - (id->subvendor == PCI_ANY_ID || id->subvendor == dev->subsystem_vendor) && - (id->subdevice == PCI_ANY_ID || id->subdevice == dev->subsystem_device) && - !((id->class ^ dev->class) & id->class_mask)) + return (struct pci_device_id) { + .vendor = dev->vendor, + .device = dev->device, + .subvendor = dev->subsystem_vendor, + .subdevice = dev->subsystem_device, + .class = dev->class, + }; +} + +/** + * pci_match_one_id - Tell if a PCI device ID matches a needle PCI device ID + * @id: single PCI device id structure to match against (needle) + * @dev_id: the actual ID from the PCI device + * + * ID can be retrieved from device using pci_id_from_device(). + * + * Return: the matching pci_device_id structure or %NULL if there is no match. + */ +static inline const struct pci_device_id * +pci_match_one_id(const struct pci_device_id *id, + const struct pci_device_id *dev_id) +{ + if ((id->vendor == PCI_ANY_ID || id->vendor == dev_id->vendor) && + (id->device == PCI_ANY_ID || id->device == dev_id->device) && + (id->subvendor == PCI_ANY_ID || id->subvendor == dev_id->subvendor) && + (id->subdevice == PCI_ANY_ID || id->subdevice == dev_id->subdevice) && + !((id->class ^ dev_id->class) & id->class_mask)) return id; return NULL; } diff --git a/drivers/pci/probe.c b/drivers/pci/probe.c index d90ffbb47f0e..9701917ce838 100644 --- a/drivers/pci/probe.c +++ b/drivers/pci/probe.c @@ -2371,7 +2371,6 @@ static void pci_release_dev(struct device *dev) pci_release_of_node(pci_dev); pcibios_release_device(pci_dev); pci_bus_put(pci_dev->bus); - kfree(pci_dev->driver_override); bitmap_free(pci_dev->dma_alias_mask); dev_dbg(dev, "device released\n"); kfree(pci_dev); diff --git a/drivers/pci/rebar.c b/drivers/pci/rebar.c index 920ef3ab4bda..d5e432d084e5 100644 --- a/drivers/pci/rebar.c +++ b/drivers/pci/rebar.c @@ -177,7 +177,6 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size, int exclude_bars) { struct pci_host_bridge *host; - int old, ret; u32 sizes; u16 cmd; @@ -197,22 +196,6 @@ int pci_resize_resource(struct pci_dev *dev, int resno, int size, if (!(sizes & BIT(size))) return -EINVAL; - old = pci_rebar_get_current_size(dev, resno); - if (old < 0) - return old; - - ret = pci_rebar_set_size(dev, resno, size); - if (ret) - return ret; - - ret = pci_do_resource_release_and_resize(dev, resno, size, exclude_bars); - if (ret) - goto error_resize; - - return 0; - -error_resize: - pci_rebar_set_size(dev, resno, old); - return ret; + return pci_do_resource_release_and_resize(dev, resno, size, exclude_bars); } EXPORT_SYMBOL(pci_resize_resource); diff --git a/drivers/pci/search.c b/drivers/pci/search.c index 53840634fbfc..705ab725321a 100644 --- a/drivers/pci/search.c +++ b/drivers/pci/search.c @@ -243,8 +243,10 @@ static int match_pci_dev_by_id(struct device *dev, const void *data) { struct pci_dev *pdev = to_pci_dev(dev); const struct pci_device_id *id = data; + struct pci_device_id dev_id; - if (pci_match_one_device(id, pdev)) + dev_id = pci_id_from_device(pdev); + if (pci_match_one_id(id, &dev_id)) return 1; return 0; } @@ -368,9 +370,9 @@ EXPORT_SYMBOL(pci_get_class); * @class: search for a PCI device with this base class code * @from: Previous PCI device found in search, or %NULL for new search. * - * Iterates through the list of known PCI devices. If a PCI device is found + * Iterate through the list of known PCI devices. If a PCI device is found * with a matching base class code, the reference count to the device is - * incremented. See pci_match_one_device() to figure out how does this works. + * incremented. See pci_match_one_id() to figure out how this works. * A new search is initiated by passing %NULL as the @from argument. * Otherwise if @from is not %NULL, searches continue from next device on the * global list. The reference count for @from is always decremented if it is diff --git a/drivers/pci/setup-bus.c b/drivers/pci/setup-bus.c index 98d0038e32ef..df39fa852997 100644 --- a/drivers/pci/setup-bus.c +++ b/drivers/pci/setup-bus.c @@ -2292,6 +2292,11 @@ static int pci_reassign_bridge_resources(struct pci_dev *bridge, unsigned long t return 0; } +/* Keep the resize follow-up compatible with the 6.12 resource helpers. */ +#define pbus_reassign_bridge_resources(bus, res, saved) \ + pci_reassign_bridge_resources((bus)->self, (res)->flags, (saved)) +#define resource_assigned(res) ((res)->parent != NULL) + int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size, int exclude_bars) { @@ -2299,10 +2304,21 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size unsigned long flags = res->flags; struct pci_dev_resource *dev_res; struct pci_bus *bus = pdev->bus; - struct resource *r; + struct pci_dev *bridge = pci_upstream_bridge(pdev); + struct resource *b_win, *r; LIST_HEAD(saved); unsigned int i; - int ret = 0; + int old, ret; + + b_win = res->parent; + + old = pci_rebar_get_current_size(pdev, resno); + if (old < 0) + return old; + + ret = pci_rebar_set_size(pdev, resno, size); + if (ret) + return ret; down_read(&pci_bus_sem); @@ -2313,7 +2329,10 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size if (exclude_bars & BIT(i)) continue; - if (!pci_resource_len(pdev, i) || r->flags != flags) + if (!pci_resource_len(pdev, i)) + continue; + + if (b_win ? r->parent != b_win : r->flags != flags) continue; ret = add_to_list(&saved, pdev, r, 0, 0); @@ -2324,12 +2343,21 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size res->end = res->start + pci_rebar_size_to_bytes(size) - 1; - if (!bus->self) - goto out; + if (bridge) { + ret = pbus_reassign_bridge_resources(bus, res, &saved); + if (ret) + goto restore; + } else { + /* No bridge window to adjust; let the core reassign the bus. */ + pci_bus_assign_resources(bus); - ret = pci_reassign_bridge_resources(bus->self, res->flags, &saved); - if (ret) - goto restore; + list_for_each_entry(dev_res, &saved, list) { + if (!resource_assigned(dev_res->res)) { + ret = -ENOSPC; + goto restore; + } + } + } out: up_read(&pci_bus_sem); @@ -2337,14 +2365,22 @@ int pci_do_resource_release_and_resize(struct pci_dev *pdev, int resno, int size return ret; restore: - /* Revert to the old configuration */ + /* + * Revert to the old configuration. + * + * BAR Size must be restored first because it affects the read-only + * bits in BAR (the old address might not be restorable otherwise + * due to low address bits). + */ + pci_rebar_set_size(pdev, resno, old); + list_for_each_entry(dev_res, &saved, list) { struct resource *res = dev_res->res; struct pci_dev *dev = dev_res->dev; i = res - dev->resource; - if (res->parent) { + if (resource_assigned(res)) { release_child_resources(res); pci_release_resource(dev, i); } diff --git a/drivers/pci/slot.c b/drivers/pci/slot.c index ed645c7a4e4b..9f2255419c6d 100644 --- a/drivers/pci/slot.c +++ b/drivers/pci/slot.c @@ -38,7 +38,7 @@ static const struct sysfs_ops pci_slot_sysfs_ops = { static ssize_t address_read_file(struct pci_slot *slot, char *buf) { - if (slot->number == 0xff) + if (slot->number == PCI_SLOT_PLACEHOLDER) return sysfs_emit(buf, "%04x:%02x\n", pci_domain_nr(slot->bus), slot->bus->number); @@ -64,6 +64,23 @@ static ssize_t cur_speed_read_file(struct pci_slot *slot, char *buf) return bus_speed_read(slot->bus->cur_bus_speed, buf); } +static bool pci_dev_matches_slot(struct pci_dev *dev, struct pci_slot *slot) +{ + if (slot->per_func_slot) + return dev->devfn == slot->number; + + return slot->number == PCI_SLOT_ALL_DEVICES || + PCI_SLOT(dev->devfn) == slot->number; +} + +static bool pci_slot_enabled_per_func(void) +{ + if (IS_ENABLED(CONFIG_S390)) + return true; + + return false; +} + static void pci_slot_release(struct kobject *kobj) { struct pci_dev *dev; @@ -74,7 +91,7 @@ static void pci_slot_release(struct kobject *kobj) down_read(&pci_bus_sem); list_for_each_entry(dev, &slot->bus->devices, bus_list) - if (PCI_SLOT(dev->devfn) == slot->number) + if (pci_dev_matches_slot(dev, slot)) dev->slot = NULL; up_read(&pci_bus_sem); @@ -167,7 +184,7 @@ void pci_dev_assign_slot(struct pci_dev *dev) mutex_lock(&pci_slot_mutex); list_for_each_entry(slot, &dev->bus->slots, list) - if (PCI_SLOT(dev->devfn) == slot->number) + if (pci_dev_matches_slot(dev, slot)) dev->slot = slot; mutex_unlock(&pci_slot_mutex); } @@ -189,7 +206,7 @@ static struct pci_slot *get_slot(struct pci_bus *parent, int slot_nr) /** * pci_create_slot - create or increment refcount for physical PCI slot * @parent: struct pci_bus of parent bridge - * @slot_nr: PCI_SLOT(pci_dev->devfn) or -1 for placeholder + * @slot_nr: PCI_SLOT(pci_dev->devfn) or PCI_SLOT_PLACEHOLDER for placeholder * @name: user visible string presented in /sys/bus/pci/slots/ * @hotplug: set if caller is hotplug driver, NULL otherwise * @@ -214,15 +231,16 @@ static struct pci_slot *get_slot(struct pci_bus *parent, int slot_nr) * In most cases, @pci_bus, @slot_nr will be sufficient to uniquely identify * a slot. There is one notable exception - pSeries (rpaphp), where the * @slot_nr cannot be determined until a device is actually inserted into - * the slot. In this scenario, the caller may pass -1 for @slot_nr. + * the slot. In this scenario, the caller may pass PCI_SLOT_PLACEHOLDER for @slot_nr. * * The following semantics are imposed when the caller passes @slot_nr == - * -1. First, we no longer check for an existing %struct pci_slot, as there - * may be many slots with @slot_nr of -1. The other change in semantics is - * user-visible, which is the 'address' parameter presented in sysfs will - * consist solely of a dddd:bb tuple, where dddd is the PCI domain of the - * %struct pci_bus and bb is the bus number. In other words, the devfn of - * the 'placeholder' slot will not be displayed. + * PCI_SLOT_PLACEHOLDER. First, we no longer check for an existing %struct + * pci_slot, as there may be many slots with @slot_nr of + * PCI_SLOT_PLACEHOLDER. The other change in semantics is user-visible, + * which is the 'address' parameter presented in sysfs will consist solely + * of a dddd:bb tuple, where dddd is the PCI domain of the %struct pci_bus + * and bb is the bus number. In other words, the devfn of the 'placeholder' + * slot will not be displayed. */ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr, const char *name, @@ -235,7 +253,7 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr, mutex_lock(&pci_slot_mutex); - if (slot_nr == -1) + if (slot_nr == PCI_SLOT_PLACEHOLDER) goto placeholder; /* @@ -265,6 +283,9 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr, slot->bus = pci_bus_get(parent); slot->number = slot_nr; + if (pci_slot_enabled_per_func()) + slot->per_func_slot = 1; + slot->kobj.kset = pci_slots_kset; slot_name = make_slot_name(name); @@ -287,7 +308,7 @@ struct pci_slot *pci_create_slot(struct pci_bus *parent, int slot_nr, down_read(&pci_bus_sem); list_for_each_entry(dev, &parent->devices, bus_list) - if (PCI_SLOT(dev->devfn) == slot_nr) + if (pci_dev_matches_slot(dev, slot)) dev->slot = slot; up_read(&pci_bus_sem); diff --git a/drivers/perf/riscv_pmu_legacy.c b/drivers/perf/riscv_pmu_legacy.c index fa0bccf4edf2..4b25e0a76eae 100644 --- a/drivers/perf/riscv_pmu_legacy.c +++ b/drivers/perf/riscv_pmu_legacy.c @@ -111,8 +111,9 @@ static void pmu_legacy_init(struct riscv_pmu *pmu) { pr_info("Legacy PMU implementation is available\n"); - pmu->cmask = BIT(RISCV_PMU_LEGACY_CYCLE) | - BIT(RISCV_PMU_LEGACY_INSTRET); + bitmap_zero(pmu->cmask, RISCV_MAX_COUNTERS); + set_bit(RISCV_PMU_LEGACY_CYCLE, pmu->cmask); + set_bit(RISCV_PMU_LEGACY_INSTRET, pmu->cmask); pmu->ctr_start = pmu_legacy_ctr_start; pmu->ctr_stop = NULL; pmu->event_map = pmu_legacy_event_map; diff --git a/drivers/perf/riscv_pmu_sbi.c b/drivers/perf/riscv_pmu_sbi.c index 901da688ea3f..6b826e7d9dd4 100644 --- a/drivers/perf/riscv_pmu_sbi.c +++ b/drivers/perf/riscv_pmu_sbi.c @@ -64,7 +64,7 @@ static unsigned int riscv_pmu_irq_num; static unsigned int riscv_pmu_irq; /* Cache the available counters in a bitmask */ -static unsigned long cmask; +static DECLARE_BITMAP(cmask, RISCV_MAX_COUNTERS); struct sbi_pmu_event_data { union { @@ -265,6 +265,21 @@ static const struct sbi_pmu_event_data pmu_cache_event_map[PERF_COUNT_HW_CACHE_M }, }; +static struct sbiret pmu_sbi_ctr_cfg_match(unsigned long cbase, + unsigned long ctr_mask, + unsigned long cflags, + unsigned long event_idx, + u64 config) +{ +#if defined(CONFIG_32BIT) + return sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase, + ctr_mask, cflags, event_idx, config, config >> 32); +#else + return sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase, + ctr_mask, cflags, event_idx, config, 0); +#endif +} + static int pmu_sbi_ctr_get_width(int idx) { return pmu_ctr_list[idx].width; @@ -292,10 +307,10 @@ int riscv_pmu_get_hpm_info(u32 *hw_ctr_width, u32 *num_hw_ctr) union sbi_pmu_ctr_info *info; u32 hpm_width = 0, hpm_count = 0; - if (!cmask) + if (bitmap_empty(cmask, RISCV_MAX_COUNTERS)) return -EINVAL; - for_each_set_bit(i, &cmask, RISCV_MAX_COUNTERS) { + for_each_set_bit(i, cmask, RISCV_MAX_COUNTERS) { info = &pmu_ctr_list[i]; if (!info) continue; @@ -344,8 +359,8 @@ static int pmu_sbi_ctr_get_idx(struct perf_event *event) struct riscv_pmu *rvpmu = to_riscv_pmu(event->pmu); struct cpu_hw_events *cpuc = this_cpu_ptr(rvpmu->hw_events); struct sbiret ret; - int idx; - uint64_t cbase = 0, cmask = rvpmu->cmask; + int idx, i; + u64 cbase = 0, cmask = 0; unsigned long cflags = 0; cflags = pmu_sbi_get_filter_flags(event); @@ -366,14 +381,21 @@ static int pmu_sbi_ctr_get_idx(struct perf_event *event) } /* retrieve the available counter index */ -#if defined(CONFIG_32BIT) - ret = sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase, - cmask, cflags, hwc->event_base, hwc->config, - hwc->config >> 32); -#else - ret = sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_CFG_MATCH, cbase, - cmask, cflags, hwc->event_base, hwc->config, 0); -#endif + if (cmask) { + ret = pmu_sbi_ctr_cfg_match(cbase, cmask, cflags, hwc->event_base, + hwc->config); + } else { + ret.error = SBI_ERR_NOT_SUPPORTED; + for (i = 0; i < BITS_TO_LONGS(RISCV_MAX_COUNTERS); i++) { + if (!rvpmu->cmask[i]) + continue; + cbase = i * BITS_PER_LONG; + ret = pmu_sbi_ctr_cfg_match(cbase, rvpmu->cmask[i], cflags, + hwc->event_base, hwc->config); + if (!ret.error) + break; + } + } if (ret.error) { pr_debug("Not able to find a counter for event %lx config %llx\n", hwc->event_base, hwc->config); @@ -381,7 +403,7 @@ static int pmu_sbi_ctr_get_idx(struct perf_event *event) } idx = ret.value; - if (!test_bit(idx, &rvpmu->cmask) || !pmu_ctr_list[idx].value) + if (!test_bit(idx, rvpmu->cmask) || !pmu_ctr_list[idx].value) return -ENOENT; /* Additional sanity check for the counter id */ @@ -589,7 +611,7 @@ static int pmu_sbi_get_ctrinfo(int nctr, unsigned long *mask) /* The logical counter ids are not expected to be contiguous */ continue; - *mask |= BIT(i); + set_bit(i, mask); cinfo.value = ret.value; if (cinfo.type == SBI_PMU_CTR_TYPE_FW) @@ -606,12 +628,19 @@ static int pmu_sbi_get_ctrinfo(int nctr, unsigned long *mask) static inline void pmu_sbi_stop_all(struct riscv_pmu *pmu) { + int i; + /* * No need to check the error because we are disabling all the counters * which may include counters that are not enabled yet. */ - sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_STOP, - 0, pmu->cmask, SBI_PMU_STOP_FLAG_RESET, 0, 0, 0); + for (i = 0; i < BITS_TO_LONGS(RISCV_MAX_COUNTERS); i++) { + if (!pmu->cmask[i]) + continue; + sbi_ecall(SBI_EXT_PMU, SBI_EXT_PMU_COUNTER_STOP, + i * BITS_PER_LONG, pmu->cmask[i], + SBI_PMU_STOP_FLAG_RESET, 0, 0, 0); + } } static inline void pmu_sbi_stop_hw_ctrs(struct riscv_pmu *pmu) @@ -1043,7 +1072,7 @@ static int pmu_sbi_device_probe(struct platform_device *pdev) } /* cache all the information about counters now */ - if (pmu_sbi_get_ctrinfo(num_counters, &cmask)) + if (pmu_sbi_get_ctrinfo(num_counters, cmask)) goto out_free; ret = pmu_sbi_setup_irqs(pmu, pdev); @@ -1054,7 +1083,7 @@ static int pmu_sbi_device_probe(struct platform_device *pdev) } pmu->pmu.attr_groups = riscv_pmu_attr_groups; - pmu->cmask = cmask; + bitmap_copy(pmu->cmask, cmask, RISCV_MAX_COUNTERS); pmu->ctr_start = pmu_sbi_ctr_start; pmu->ctr_stop = pmu_sbi_ctr_stop; pmu->event_map = pmu_sbi_event_map; diff --git a/drivers/pinctrl/mediatek/pinctrl-moore.c b/drivers/pinctrl/mediatek/pinctrl-moore.c index c3b559cc1644..19a19861b709 100644 --- a/drivers/pinctrl/mediatek/pinctrl-moore.c +++ b/drivers/pinctrl/mediatek/pinctrl-moore.c @@ -575,7 +575,7 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw) chip->base = -1; chip->ngpio = hw->soc->npins; - ret = gpiochip_add_data(chip, hw); + ret = devm_gpiochip_add_data(hw->dev, chip, hw); if (ret < 0) return ret; @@ -589,10 +589,8 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw) if (!of_property_present(hw->dev->of_node, "gpio-ranges")) { ret = gpiochip_add_pin_range(chip, dev_name(hw->dev), 0, 0, chip->ngpio); - if (ret < 0) { - gpiochip_remove(chip); + if (ret < 0) return ret; - } } return 0; diff --git a/drivers/pinctrl/mediatek/pinctrl-mtk-common.c b/drivers/pinctrl/mediatek/pinctrl-mtk-common.c index 0c3732875e67..d4f49b662362 100644 --- a/drivers/pinctrl/mediatek/pinctrl-mtk-common.c +++ b/drivers/pinctrl/mediatek/pinctrl-mtk-common.c @@ -1127,30 +1127,24 @@ int mtk_pctrl_init(struct platform_device *pdev, pctl->chip->parent = &pdev->dev; pctl->chip->base = -1; - ret = gpiochip_add_data(pctl->chip, pctl); + ret = devm_gpiochip_add_data(&pdev->dev, pctl->chip, pctl); if (ret) return -EINVAL; /* Register the GPIO to pin mappings. */ ret = gpiochip_add_pin_range(pctl->chip, dev_name(&pdev->dev), 0, 0, pctl->devdata->npins); - if (ret) { - ret = -EINVAL; - goto chip_error; - } + if (ret) + return -EINVAL; /* Only initialize EINT if we have EINT pins */ if (data->eint_hw.ap_num > 0) { ret = mtk_eint_init(pctl, pdev); if (ret) - goto chip_error; + return ret; } return 0; - -chip_error: - gpiochip_remove(pctl->chip); - return ret; } int mtk_pctrl_common_probe(struct platform_device *pdev) diff --git a/drivers/pinctrl/mediatek/pinctrl-paris.c b/drivers/pinctrl/mediatek/pinctrl-paris.c index 2d0b970168a3..b2c99f0facf9 100644 --- a/drivers/pinctrl/mediatek/pinctrl-paris.c +++ b/drivers/pinctrl/mediatek/pinctrl-paris.c @@ -775,7 +775,6 @@ static const struct pinmux_ops mtk_pmxops = { .get_function_name = mtk_pmx_get_func_name, .get_function_groups = mtk_pmx_get_func_groups, .set_mux = mtk_pmx_set_mux, - .gpio_set_direction = mtk_pinmux_gpio_set_direction, .gpio_request_enable = mtk_pinmux_gpio_request_enable, }; @@ -904,7 +903,7 @@ static int mtk_gpio_direction_input(struct gpio_chip *chip, unsigned int gpio) if (gpio >= hw->soc->npins) return -EINVAL; - return pinctrl_gpio_direction_input(chip, gpio); + return mtk_pinmux_gpio_set_direction(hw->pctrl, NULL, gpio, true); } static int mtk_gpio_direction_output(struct gpio_chip *chip, unsigned int gpio, @@ -917,7 +916,7 @@ static int mtk_gpio_direction_output(struct gpio_chip *chip, unsigned int gpio, mtk_gpio_set(chip, gpio, value); - return pinctrl_gpio_direction_output(chip, gpio); + return mtk_pinmux_gpio_set_direction(hw->pctrl, NULL, gpio, false); } static int mtk_gpio_to_irq(struct gpio_chip *chip, unsigned int offset) @@ -974,7 +973,7 @@ static int mtk_build_gpiochip(struct mtk_pinctrl *hw) chip->base = -1; chip->ngpio = hw->soc->npins; - ret = gpiochip_add_data(chip, hw); + ret = devm_gpiochip_add_data(hw->dev, chip, hw); if (ret < 0) return ret; diff --git a/drivers/pinctrl/sunxi/pinctrl-sunxi.c b/drivers/pinctrl/sunxi/pinctrl-sunxi.c index c5b954c9537d..e182812c66fa 100644 --- a/drivers/pinctrl/sunxi/pinctrl-sunxi.c +++ b/drivers/pinctrl/sunxi/pinctrl-sunxi.c @@ -792,6 +792,21 @@ static void sunxi_pmx_set(struct pinctrl_dev *pctldev, writel((readl(pctl->membase + reg) & ~mask) | config << shift, pctl->membase + reg); + /* + * A pin muxed to gpio_out directly through a pinmux node bypasses + * sunxi_pinctrl_gpio_set() and drives whatever its output latch + * holds. Now that the pin is in output mode the data register + * reads back the latch, so refresh the shadow to keep such pins + * driving their pre-existing level. + */ + if (config == SUN4I_FUNC_OUTPUT) { + u32 *shadow = &pctl->dat_shadow[pin / PINS_PER_BANK]; + + sunxi_data_reg(pctl, pin, ®, &shift, &mask); + *shadow = (*shadow & ~mask) | + (readl(pctl->membase + reg) & mask); + } + raw_spin_unlock_irqrestore(&pctl->lock, flags); } @@ -948,21 +963,29 @@ static void sunxi_pinctrl_gpio_set(struct gpio_chip *chip, unsigned offset, int value) { struct sunxi_pinctrl *pctl = gpiochip_get_data(chip); - u32 reg, shift, mask, val; + u32 *shadow = &pctl->dat_shadow[offset / PINS_PER_BANK]; + u32 reg, shift, mask; unsigned long flags; sunxi_data_reg(pctl, offset, ®, &shift, &mask); raw_spin_lock_irqsave(&pctl->lock, flags); - val = readl(pctl->membase + reg); - + /* + * Reading the data register returns the pin level, not the output + * latch, for pins muxed as inputs. A read-modify-write based on + * the register would therefore corrupt the latches of input-muxed + * pins in the same bank (e.g. an emulated open-drain I2C line + * released high), making them drive the wrong level once switched + * to output. Base the read-modify-write on a shadow copy of the + * latches instead. + */ if (value) - val |= mask; + *shadow |= mask; else - val &= ~mask; + *shadow &= ~mask; - writel(val, pctl->membase + reg); + writel(*shadow, pctl->membase + reg); raw_spin_unlock_irqrestore(&pctl->lock, flags); } @@ -1486,7 +1509,7 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev, struct pinctrl_pin_desc *pins; struct sunxi_pinctrl *pctl; struct pinmux_ops *pmxops; - int i, ret, last_pin, pin_idx; + int i, ret, last_pin, pin_idx, nbanks; struct clk *clk; pctl = devm_kzalloc(&pdev->dev, sizeof(*pctl), GFP_KERNEL); @@ -1520,6 +1543,37 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev, if (!pctl->irq_array) return -ENOMEM; + /* + * The bus clock has to be enabled before the pinctrl device + * registers, as the pin hogs claimed from there access registers. + */ + ret = of_clk_get_parent_count(node); + clk = devm_clk_get_enabled(&pdev->dev, ret == 1 ? NULL : "apb"); + if (IS_ERR(clk)) + return PTR_ERR(clk); + + /* + * Seed the output latch shadow from the hardware so pins the + * bootloader left in output mode keep their state; see + * sunxi_pinctrl_gpio_set() for why a shadow is needed. This must + * happen before the pinctrl device registers, as pin hogs can mux + * pins to gpio_out and thereby update the shadow. + */ + last_pin = pctl->desc->pins[pctl->desc->npins - 1].pin.number; + nbanks = DIV_ROUND_UP(last_pin + 1 - pctl->desc->pin_base, + PINS_PER_BANK); + pctl->dat_shadow = devm_kcalloc(&pdev->dev, nbanks, + sizeof(*pctl->dat_shadow), GFP_KERNEL); + if (!pctl->dat_shadow) + return -ENOMEM; + + for (i = 0; i < nbanks; i++) { + u32 reg, shift, mask; + + sunxi_data_reg(pctl, i * PINS_PER_BANK, ®, &shift, &mask); + pctl->dat_shadow[i] = readl(pctl->membase + reg); + } + ret = sunxi_pinctrl_build_state(pdev); if (ret) { dev_err(&pdev->dev, "dt probe failed: %d\n", ret); @@ -1574,7 +1628,6 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev, if (!pctl->chip) return -ENOMEM; - last_pin = pctl->desc->pins[pctl->desc->npins - 1].pin.number; pctl->chip->owner = THIS_MODULE; pctl->chip->request = gpiochip_generic_request; pctl->chip->free = gpiochip_generic_free; @@ -1607,31 +1660,20 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev, goto gpiochip_error; } - ret = of_clk_get_parent_count(node); - clk = devm_clk_get(&pdev->dev, ret == 1 ? NULL : "apb"); - if (IS_ERR(clk)) { - ret = PTR_ERR(clk); - goto gpiochip_error; - } - - ret = clk_prepare_enable(clk); - if (ret) - goto gpiochip_error; - pctl->irq = devm_kcalloc(&pdev->dev, pctl->desc->irq_banks, sizeof(*pctl->irq), GFP_KERNEL); if (!pctl->irq) { ret = -ENOMEM; - goto clk_error; + goto gpiochip_error; } for (i = 0; i < pctl->desc->irq_banks; i++) { pctl->irq[i] = platform_get_irq(pdev, i); if (pctl->irq[i] < 0) { ret = pctl->irq[i]; - goto clk_error; + goto gpiochip_error; } } @@ -1642,7 +1684,7 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev, if (!pctl->domain) { dev_err(&pdev->dev, "Couldn't register IRQ domain\n"); ret = -ENOMEM; - goto clk_error; + goto gpiochip_error; } for (i = 0; i < (pctl->desc->irq_banks * IRQ_PER_BANK); i++) { @@ -1674,8 +1716,6 @@ int sunxi_pinctrl_init_with_variant(struct platform_device *pdev, return 0; -clk_error: - clk_disable_unprepare(clk); gpiochip_error: gpiochip_remove(pctl->chip); return ret; diff --git a/drivers/pinctrl/sunxi/pinctrl-sunxi.h b/drivers/pinctrl/sunxi/pinctrl-sunxi.h index a87a2f944d60..244b0039480a 100644 --- a/drivers/pinctrl/sunxi/pinctrl-sunxi.h +++ b/drivers/pinctrl/sunxi/pinctrl-sunxi.h @@ -80,6 +80,7 @@ #define IO_BIAS_MASK GENMASK(3, 0) #define SUN4I_FUNC_INPUT 0 +#define SUN4I_FUNC_OUTPUT 1 #define SUN4I_FUNC_IRQ 6 #define PINCTRL_SUN5I_A10S BIT(1) @@ -173,6 +174,12 @@ struct sunxi_pinctrl { int *irq; unsigned *irq_array; raw_spinlock_t lock; + /* + * Output latch shadow, one word per bank. Seeded lockless at + * probe before the pinctrl device registers, protected by @lock + * afterwards. + */ + u32 *dat_shadow; struct pinctrl_dev *pctl_dev; unsigned long variant; u32 bank_mem_size; diff --git a/drivers/platform/x86/amd/pmc/pmc.c b/drivers/platform/x86/amd/pmc/pmc.c index bf6c7ca247e2..6b67f8517d90 100644 --- a/drivers/platform/x86/amd/pmc/pmc.c +++ b/drivers/platform/x86/amd/pmc/pmc.c @@ -1005,34 +1005,46 @@ static int amd_pmc_s2d_init(struct amd_pmc_dev *dev) /* Get num of IP blocks within the SoC */ amd_pmc_get_ip_info(dev); - amd_pmc_send_cmd(dev, S2D_TELEMETRY_SIZE, &size, dev->s2d_msg_id, true); - if (size != S2D_TELEMETRY_BYTES_MAX) - return -EIO; + ret = amd_pmc_send_cmd(dev, S2D_TELEMETRY_SIZE, &size, dev->s2d_msg_id, true); + if (ret) + goto out; + if (size != S2D_TELEMETRY_BYTES_MAX) { + ret = -EIO; + goto out; + } - /* Get DRAM size */ - ret = amd_pmc_send_cmd(dev, S2D_DRAM_SIZE, &dev->dram_size, dev->s2d_msg_id, true); - if (ret || !dev->dram_size) + /* Get DRAM size; fall back to the default if the query fails */ + if (amd_pmc_send_cmd(dev, S2D_DRAM_SIZE, &dev->dram_size, dev->s2d_msg_id, true) || + !dev->dram_size) dev->dram_size = S2D_TELEMETRY_DRAMBYTES_MAX; /* Get STB DRAM address */ - amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_LOW, &phys_addr_low, dev->s2d_msg_id, true); - amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_HIGH, &phys_addr_hi, dev->s2d_msg_id, true); + ret = amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_LOW, &phys_addr_low, + dev->s2d_msg_id, true); + if (ret) + goto out; + ret = amd_pmc_send_cmd(dev, S2D_PHYS_ADDR_HIGH, &phys_addr_hi, + dev->s2d_msg_id, true); + if (ret) + goto out; if (!phys_addr_hi && !phys_addr_low) { dev_err(dev->dev, "STB is not enabled on the system; disable enable_stb or contact system vendor\n"); - return -EINVAL; + ret = -EINVAL; + goto out; } stb_phys_addr = ((u64)phys_addr_hi << 32 | phys_addr_low); - /* Clear msg_port for other SMU operation */ - dev->msg_port = 0; - dev->stb_virt_addr = devm_ioremap(dev->dev, stb_phys_addr, dev->dram_size); if (!dev->stb_virt_addr) - return -ENOMEM; + ret = -ENOMEM; - return 0; +out: + /* Clear msg_port for other SMU operation */ + dev->msg_port = 0; + + return ret; } static int amd_pmc_write_stb(struct amd_pmc_dev *dev, u32 data) diff --git a/drivers/platform/x86/asus-wmi.c b/drivers/platform/x86/asus-wmi.c index b56be64fcc63..a76d43ca647d 100644 --- a/drivers/platform/x86/asus-wmi.c +++ b/drivers/platform/x86/asus-wmi.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -133,6 +134,11 @@ module_param(fnlock_default, bool, 0444); #define NVIDIA_TEMP_MIN 75 #define NVIDIA_TEMP_MAX 87 +/* Controls the power state of the USB0 hub on ROG Ally which input is on */ +#define ASUS_USB0_PWR_EC0_CSEE "\\_SB.PCI0.SBRG.EC0.CSEE" +/* 300ms so far seems to produce a reliable result on AC and battery */ +#define ASUS_USB0_PWR_EC0_CSEE_WAIT 300 + static const char * const ashs_ids[] = { "ATK4001", "ATK4002", NULL }; static int throttle_thermal_policy_write(struct asus_wmi *); @@ -300,6 +306,9 @@ struct asus_wmi { bool fnlock_locked; + /* The ROG Ally device requires the MCU USB device be disconnected before suspend */ + bool ally_mcu_usb_switch; + struct asus_wmi_debug debug; struct asus_wmi_driver *driver; @@ -4367,6 +4376,8 @@ static int asus_wmi_add(struct platform_device *pdev) asus->nv_temp_tgt_available = asus_wmi_dev_is_present(asus, ASUS_WMI_DEVID_NV_THERM_TARGET); asus->panel_overdrive_available = asus_wmi_dev_is_present(asus, ASUS_WMI_DEVID_PANEL_OD); asus->mini_led_mode_available = asus_wmi_dev_is_present(asus, ASUS_WMI_DEVID_MINI_LED_MODE); + asus->ally_mcu_usb_switch = acpi_has_method(NULL, ASUS_USB0_PWR_EC0_CSEE) + && dmi_match(DMI_BOARD_NAME, "RC71L"); if (asus_wmi_dev_is_present(asus, ASUS_WMI_DEVID_THROTTLE_THERMAL_POLICY)) asus->throttle_thermal_policy_dev = ASUS_WMI_DEVID_THROTTLE_THERMAL_POLICY; @@ -4459,12 +4470,12 @@ static int asus_wmi_add(struct platform_device *pdev) fail_rfkill: asus_wmi_led_exit(asus); fail_leds: +fail_custom_fan_curve: fail_hwmon: asus_wmi_input_exit(asus); fail_input: asus_wmi_sysfs_exit(asus->platform_device); fail_sysfs: -fail_custom_fan_curve: fail_platform_profile_setup: if (asus->platform_profile_support) platform_profile_remove(); @@ -4531,6 +4542,43 @@ static int asus_hotk_resume(struct device *device) asus_wmi_fnlock_update(asus); asus_wmi_tablet_mode_get_state(asus); + + return 0; +} + +static int asus_hotk_resume_early(struct device *device) +{ + struct asus_wmi *asus = dev_get_drvdata(device); + + if (asus->ally_mcu_usb_switch) { + if (ACPI_FAILURE(acpi_execute_simple_method(NULL, ASUS_USB0_PWR_EC0_CSEE, 0xB8))) + dev_err(device, "ROG Ally MCU failed to connect USB dev\n"); + else + msleep(ASUS_USB0_PWR_EC0_CSEE_WAIT); + } + return 0; +} + +static int asus_hotk_prepare(struct device *device) +{ + struct asus_wmi *asus = dev_get_drvdata(device); + int result, err; + + if (asus->ally_mcu_usb_switch) { + /* When powersave is enabled it causes many issues with resume of USB hub */ + result = asus_wmi_get_devstate_simple(asus, ASUS_WMI_DEVID_MCU_POWERSAVE); + if (result == 1) { + dev_warn(device, "MCU powersave enabled, disabling to prevent resume issues"); + err = asus_wmi_set_devstate(ASUS_WMI_DEVID_MCU_POWERSAVE, 0, &result); + if (err || result != 1) + dev_err(device, "Failed to set MCU powersave mode: %d\n", err); + } + /* sleep required to ensure USB0 is disabled before sleep continues */ + if (ACPI_FAILURE(acpi_execute_simple_method(NULL, ASUS_USB0_PWR_EC0_CSEE, 0xB7))) + dev_err(device, "ROG Ally MCU failed to disconnect USB dev\n"); + else + msleep(ASUS_USB0_PWR_EC0_CSEE_WAIT); + } return 0; } @@ -4578,6 +4626,8 @@ static const struct dev_pm_ops asus_pm_ops = { .thaw = asus_hotk_thaw, .restore = asus_hotk_restore, .resume = asus_hotk_resume, + .resume_early = asus_hotk_resume_early, + .prepare = asus_hotk_prepare, }; /* Registration ***************************************************************/ diff --git a/drivers/platform/x86/dell/dell-wmi-base.c b/drivers/platform/x86/dell/dell-wmi-base.c index 713ec2c12a04..4838992ee5f9 100644 --- a/drivers/platform/x86/dell/dell-wmi-base.c +++ b/drivers/platform/x86/dell/dell-wmi-base.c @@ -442,7 +442,7 @@ static int dell_wmi_process_key(struct wmi_device *wdev, int type, int code, u16 SW_TABLET_MODE, !buffer[0]); return 1; } else if (type == 0x0012 && code == 0x000d && remaining > 0) { - value = (buffer[2] == 2); + value = (buffer[0] == 2); used = 1; } diff --git a/drivers/platform/x86/intel/int1092/intel_sar.c b/drivers/platform/x86/intel/int1092/intel_sar.c index 6246c066ade2..1cfb268f6b71 100644 --- a/drivers/platform/x86/intel/int1092/intel_sar.c +++ b/drivers/platform/x86/intel/int1092/intel_sar.c @@ -91,8 +91,10 @@ static acpi_status parse_package(struct wwan_sar_context *context, union acpi_ob item->package.count <= data->total_dev_mode) return AE_ERROR; - data->device_mode_info = kmalloc_array(data->total_dev_mode, - sizeof(struct wwan_device_mode_info), GFP_KERNEL); + data->device_mode_info = devm_kmalloc_array(&context->sar_device->dev, + data->total_dev_mode, + sizeof(*data->device_mode_info), + GFP_KERNEL); if (!data->device_mode_info) return AE_ERROR; @@ -245,21 +247,26 @@ static void sar_get_data(int reg, struct wwan_sar_context *context) static int sar_probe(struct platform_device *device) { struct wwan_sar_context *context; + acpi_handle handle; int reg; int result; - context = kzalloc(sizeof(*context), GFP_KERNEL); + handle = ACPI_HANDLE(&device->dev); + if (!handle) + return -ENODEV; + + context = devm_kzalloc(&device->dev, sizeof(*context), GFP_KERNEL); if (!context) return -ENOMEM; context->sar_device = device; - context->handle = ACPI_HANDLE(&device->dev); + context->handle = handle; dev_set_drvdata(&device->dev, context); result = guid_parse(SAR_DSM_UUID, &context->guid); if (result) { dev_err(&device->dev, "SAR UUID parse error: %d\n", result); - goto r_free; + return result; } for (reg = 0; reg < MAX_REGULATORY; reg++) @@ -267,43 +274,29 @@ static int sar_probe(struct platform_device *device) if (sar_get_device_mode(device) != AE_OK) { dev_err(&device->dev, "Failed to get device mode\n"); - result = -EIO; - goto r_free; + return -EIO; } result = sysfs_create_group(&device->dev.kobj, &intcsar_group); if (result) { dev_err(&device->dev, "sysfs creation failed\n"); - goto r_free; + return result; } if (acpi_install_notify_handler(ACPI_HANDLE(&device->dev), ACPI_DEVICE_NOTIFY, sar_notify, (void *)device) != AE_OK) { dev_err(&device->dev, "Failed acpi_install_notify_handler\n"); - result = -EIO; - goto r_sys; + sysfs_remove_group(&device->dev.kobj, &intcsar_group); + return -EIO; } return 0; - -r_sys: - sysfs_remove_group(&device->dev.kobj, &intcsar_group); -r_free: - kfree(context); - return result; } static void sar_remove(struct platform_device *device) { - struct wwan_sar_context *context = dev_get_drvdata(&device->dev); - int reg; - acpi_remove_notify_handler(ACPI_HANDLE(&device->dev), ACPI_DEVICE_NOTIFY, sar_notify); sysfs_remove_group(&device->dev.kobj, &intcsar_group); - for (reg = 0; reg < MAX_REGULATORY; reg++) - kfree(context->config_data[reg].device_mode_info); - - kfree(context); } static struct platform_driver sar_driver = { diff --git a/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c b/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c index 7a4f3086e97b..30f4f047ef5a 100644 --- a/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c +++ b/drivers/platform/x86/intel/speed_select_if/isst_tpmi_core.c @@ -450,6 +450,9 @@ static bool disable_dynamic_sst_features(void) #define SST_CP_PRIORITY_TYPE_START 1 #define SST_CP_PRIORITY_TYPE_WIDTH 1 +#define SST_CP_MAX_ENABLE 1 +#define SST_CP_MAX_PRIORITY_TYPE 1 + static long isst_if_core_power_state(void __user *argp) { struct tpmi_per_power_domain_info *power_domain_info; @@ -466,6 +469,10 @@ static long isst_if_core_power_state(void __user *argp) return -EINVAL; if (core_power.get_set) { + if (core_power.enable > SST_CP_MAX_ENABLE || + core_power.priority_type > SST_CP_MAX_PRIORITY_TYPE) + return -EINVAL; + _write_cp_info("cp_enable", core_power.enable, SST_CP_CONTROL_OFFSET, SST_CP_ENABLE_START, SST_CP_ENABLE_WIDTH, SST_MUL_FACTOR_NONE) _write_cp_info("cp_prio_type", core_power.priority_type, SST_CP_CONTROL_OFFSET, @@ -763,6 +770,9 @@ static int isst_if_set_perf_level(void __user *argp) if (!power_domain_info) return -EINVAL; + if (perf_level.level > power_domain_info->max_level) + return -EINVAL; + if (!(power_domain_info->pp_header.allowed_level_mask & BIT(perf_level.level))) return -EINVAL; @@ -1143,9 +1153,12 @@ static int isst_if_get_tpmi_instance_count(void __user *argp) if (tpmi_inst.socket_id >= topology_max_packages()) return -EINVAL; - tpmi_inst.count = isst_common.sst_inst[tpmi_inst.socket_id]->number_of_power_domains; - sst_inst = isst_common.sst_inst[tpmi_inst.socket_id]; + if (!sst_inst) + return -EINVAL; + + tpmi_inst.count = sst_inst->number_of_power_domains; + tpmi_inst.valid_mask = 0; for (i = 0; i < sst_inst->number_of_power_domains; ++i) { struct tpmi_per_power_domain_info *pd_info; diff --git a/drivers/platform/x86/lg-laptop.c b/drivers/platform/x86/lg-laptop.c index 78c48a1f9c68..16c37634925d 100644 --- a/drivers/platform/x86/lg-laptop.c +++ b/drivers/platform/x86/lg-laptop.c @@ -71,6 +71,8 @@ static u32 inited; #define INIT_SPARSE_KEYMAP 0x80 static int battery_limit_use_wmbb; +static bool kbd_backlight_available; +static bool tpad_led_available; static struct led_classdev kbd_backlight; static enum led_brightness get_kbd_backlight_level(struct device *dev); @@ -188,6 +190,7 @@ static void wmi_notify(u32 value, void *context) union acpi_object *obj; acpi_status status; long data = (long)context; + unsigned int brightness; pr_debug("event guid %li\n", data); status = wmi_get_event_data(value, &response); @@ -205,8 +208,11 @@ static void wmi_notify(u32 value, void *context) struct key_entry *key; if (eventcode == 0x10000000) { - led_classdev_notify_brightness_hw_changed( - &kbd_backlight, get_kbd_backlight_level(kbd_backlight.dev->parent)); + if (kbd_backlight_available) { + brightness = get_kbd_backlight_level(kbd_backlight.dev->parent); + led_classdev_notify_brightness_hw_changed(&kbd_backlight, + brightness); + } } else { key = sparse_keymap_entry_from_scancode( wmi_input_dev, eventcode); @@ -725,8 +731,11 @@ static int acpi_add(struct acpi_device *device) goto out_platform_device; /* LEDs are optional */ - led_classdev_register(&pf_device->dev, &kbd_backlight); - led_classdev_register(&pf_device->dev, &tpad_led); + ret = led_classdev_register(&pf_device->dev, &kbd_backlight); + kbd_backlight_available = !ret; + + ret = led_classdev_register(&pf_device->dev, &tpad_led); + tpad_led_available = !ret; wmi_input_setup(); battery_hook_register(&battery_hook); @@ -744,11 +753,16 @@ static void acpi_remove(struct acpi_device *device) { sysfs_remove_group(&pf_device->dev.kobj, &dev_attribute_group); - led_classdev_unregister(&tpad_led); - led_classdev_unregister(&kbd_backlight); - battery_hook_unregister(&battery_hook); wmi_input_destroy(); + + if (tpad_led_available) + led_classdev_unregister(&tpad_led); + if (kbd_backlight_available) { + kbd_backlight_available = false; + led_classdev_unregister(&kbd_backlight); + } + platform_device_unregister(pf_device); pf_device = NULL; platform_driver_unregister(&pf_driver); diff --git a/drivers/pmdomain/imx/imx8m-blk-ctrl.c b/drivers/pmdomain/imx/imx8m-blk-ctrl.c index a2e9a6bdc2b8..8d1a625250f6 100644 --- a/drivers/pmdomain/imx/imx8m-blk-ctrl.c +++ b/drivers/pmdomain/imx/imx8m-blk-ctrl.c @@ -15,6 +15,7 @@ #include #include #include +#include #include #include @@ -34,6 +35,12 @@ struct imx8m_blk_ctrl { struct regmap *regmap; struct imx8m_blk_ctrl_domain *domains; struct genpd_onecell_data onecell_data; + /* + * Serializes the blk-ctrl reset/clock sequence across sibling domains; + * their transitions interact through the shared VPUMIX bus domain, + * VPU_NOC and the not-ack-verified ADB400 handshake (ERR050531). + */ + struct mutex power_lock; }; struct imx8m_blk_ctrl_domain_data { @@ -98,6 +105,8 @@ static int imx8m_blk_ctrl_power_on(struct generic_pm_domain *genpd) struct imx8m_blk_ctrl *bc = domain->bc; int ret; + guard(mutex)(&bc->power_lock); + /* make sure bus domain is awake */ ret = pm_runtime_get_sync(bc->bus_power_dev); if (ret < 0) { @@ -164,6 +173,8 @@ static int imx8m_blk_ctrl_power_off(struct generic_pm_domain *genpd) const struct imx8m_blk_ctrl_domain_data *data = domain->data; struct imx8m_blk_ctrl *bc = domain->bc; + guard(mutex)(&bc->power_lock); + /* put devices into reset and disable clocks */ if (data->mipi_phy_rst_mask) regmap_clear_bits(bc->regmap, BLK_MIPI_RESET_DIV, data->mipi_phy_rst_mask); @@ -202,6 +213,10 @@ static int imx8m_blk_ctrl_probe(struct platform_device *pdev) bc->dev = dev; + ret = devm_mutex_init(dev, &bc->power_lock); + if (ret) + return ret; + bc_data = of_device_get_match_data(dev); base = devm_platform_ioremap_resource(pdev, 0); diff --git a/drivers/pmdomain/rockchip/pm-domains.c b/drivers/pmdomain/rockchip/pm-domains.c index 9b76b62869d0..1f82e2e5a3b6 100644 --- a/drivers/pmdomain/rockchip/pm-domains.c +++ b/drivers/pmdomain/rockchip/pm-domains.c @@ -592,7 +592,19 @@ static int rockchip_pd_attach_dev(struct generic_pm_domain *genpd, } i = 0; - while ((clk = of_clk_get(dev->of_node, i++)) && !IS_ERR(clk)) { + while (1) { + clk = of_clk_get(dev->of_node, i++); + if (IS_ERR(clk)) { + error = PTR_ERR(clk); + if (error == -ENOENT) + break; + + dev_err(dev, "failed to get clock %d: %d\n", i - 1, + error); + pm_clk_destroy(dev); + return error; + } + dev_dbg(dev, "adding clock '%pC' to list of PM clocks\n", clk); error = pm_clk_add_clk(dev, clk); if (error) { @@ -674,7 +686,7 @@ static int rockchip_pm_add_one_domain(struct rockchip_pmu *pmu, dev_err(pmu->dev, "%pOFn: failed to get clk at index %d: %d\n", node, i, error); - return error; + goto err_put_clocks; } } @@ -842,7 +854,9 @@ static int rockchip_pm_add_subdomain(struct rockchip_pmu *pmu, parent_domain->name, child_domain->name); } - rockchip_pm_add_subdomain(pmu, np); + error = rockchip_pm_add_subdomain(pmu, np); + if (error) + goto err_out; } return 0; diff --git a/drivers/power/supply/charger-manager.c b/drivers/power/supply/charger-manager.c index b874947b84f2..48b706c87646 100644 --- a/drivers/power/supply/charger-manager.c +++ b/drivers/power/supply/charger-manager.c @@ -1017,6 +1017,29 @@ static int charger_extcon_init(struct charger_manager *cm, return 0; } +static int charger_manager_get_regulators(struct charger_manager *cm) +{ + struct charger_desc *desc = cm->desc; + struct charger_regulator *charger; + int i, ret; + + for (i = 0; i < desc->num_charger_regulators; i++) { + charger = &desc->charger_regulators[i]; + charger->consumer = regulator_get(cm->dev, + charger->regulator_name); + if (IS_ERR(charger->consumer)) { + dev_err(cm->dev, "Cannot find charger(%s)\n", + charger->regulator_name); + ret = PTR_ERR(charger->consumer); + while (i-- > 0) + regulator_put(desc->charger_regulators[i].consumer); + return ret; + } + charger->cm = cm; + } + return 0; +} + /** * charger_manager_register_extcon - Register extcon device to receive state * of charger cable. @@ -1039,15 +1062,6 @@ static int charger_manager_register_extcon(struct charger_manager *cm) for (i = 0; i < desc->num_charger_regulators; i++) { charger = &desc->charger_regulators[i]; - charger->consumer = regulator_get(cm->dev, - charger->regulator_name); - if (IS_ERR(charger->consumer)) { - dev_err(cm->dev, "Cannot find charger(%s)\n", - charger->regulator_name); - return PTR_ERR(charger->consumer); - } - charger->cm = cm; - for (j = 0; j < charger->num_cables; j++) { struct charger_cable *cable = &charger->cables[j]; @@ -1582,13 +1596,23 @@ static int charger_manager_probe(struct platform_device *pdev) } psy_cfg.attr_grp = desc->sysfs_groups; + /* + * Acquire charger regulators before exposing the sysfs entries, so + * userspace cannot reach externally_control before the regulators + * (and charger->cm) are available. Mirrors the order in remove(). + */ + ret = charger_manager_get_regulators(cm); + if (ret < 0) + return ret; + cm->charger_psy = power_supply_register(&pdev->dev, &cm->charger_psy_desc, &psy_cfg); if (IS_ERR(cm->charger_psy)) { dev_err(&pdev->dev, "Cannot register charger-manager with name \"%s\"\n", cm->charger_psy_desc.name); - return PTR_ERR(cm->charger_psy); + ret = PTR_ERR(cm->charger_psy); + goto err_regulator; } /* Register extcon device for charger cable */ @@ -1622,15 +1646,15 @@ static int charger_manager_probe(struct platform_device *pdev) return 0; err_reg_extcon: + power_supply_unregister(cm->charger_psy); +err_regulator: for (i = 0; i < desc->num_charger_regulators; i++) regulator_put(desc->charger_regulators[i].consumer); - power_supply_unregister(cm->charger_psy); - return ret; } -static int charger_manager_remove(struct platform_device *pdev) +static void charger_manager_remove(struct platform_device *pdev) { struct charger_manager *cm = platform_get_drvdata(pdev); struct charger_desc *desc = cm->desc; @@ -1644,14 +1668,12 @@ static int charger_manager_remove(struct platform_device *pdev) cancel_work_sync(&setup_polling); cancel_delayed_work_sync(&cm_monitor_work); - for (i = 0 ; i < desc->num_charger_regulators ; i++) - regulator_put(desc->charger_regulators[i].consumer); + try_charger_enable(cm, false); power_supply_unregister(cm->charger_psy); - try_charger_enable(cm, false); - - return 0; + for (i = 0 ; i < desc->num_charger_regulators ; i++) + regulator_put(desc->charger_regulators[i].consumer); } static const struct platform_device_id charger_manager_id[] = { @@ -1742,7 +1764,7 @@ static struct platform_driver charger_manager_driver = { .of_match_table = charger_manager_match, }, .probe = charger_manager_probe, - .remove = charger_manager_remove, + .remove_new = charger_manager_remove, .id_table = charger_manager_id, }; diff --git a/drivers/power/supply/lp8788-charger.c b/drivers/power/supply/lp8788-charger.c index 755b6a4379b8..8d9a60d2f7da 100644 --- a/drivers/power/supply/lp8788-charger.c +++ b/drivers/power/supply/lp8788-charger.c @@ -406,12 +406,6 @@ static const struct power_supply_desc lp8788_psy_battery_desc = { .get_property = lp8788_battery_get_property, }; -static void lp8788_psy_unregister(struct lp8788_charger *pchg) -{ - power_supply_unregister(pchg->battery); - power_supply_unregister(pchg->charger); -} - static void lp8788_charger_event(struct work_struct *work) { struct lp8788_charger *pchg = @@ -666,18 +660,16 @@ static int lp8788_psy_register(struct platform_device *pdev, charger_cfg.supplied_to = battery_supplied_to; charger_cfg.num_supplicants = ARRAY_SIZE(battery_supplied_to); - pchg->charger = power_supply_register(&pdev->dev, - &lp8788_psy_charger_desc, - &charger_cfg); + pchg->charger = devm_power_supply_register(&pdev->dev, + &lp8788_psy_charger_desc, + &charger_cfg); if (IS_ERR(pchg->charger)) return -EPERM; - pchg->battery = power_supply_register(&pdev->dev, - &lp8788_psy_battery_desc, NULL); - if (IS_ERR(pchg->battery)) { - power_supply_unregister(pchg->charger); + pchg->battery = devm_power_supply_register(&pdev->dev, + &lp8788_psy_battery_desc, NULL); + if (IS_ERR(pchg->battery)) return -EPERM; - } return 0; } @@ -714,20 +706,17 @@ static int lp8788_charger_probe(struct platform_device *pdev) return 0; } -static int lp8788_charger_remove(struct platform_device *pdev) +static void lp8788_charger_remove(struct platform_device *pdev) { struct lp8788_charger *pchg = platform_get_drvdata(pdev); - flush_work(&pchg->charger_work); lp8788_irq_unregister(pdev, pchg); - lp8788_psy_unregister(pchg); - - return 0; + cancel_work_sync(&pchg->charger_work); } static struct platform_driver lp8788_charger_driver = { .probe = lp8788_charger_probe, - .remove = lp8788_charger_remove, + .remove_new = lp8788_charger_remove, .driver = { .name = LP8788_DEV_CHARGER, }, diff --git a/drivers/power/supply/max17040_battery.c b/drivers/power/supply/max17040_battery.c index 2814327c8e7d..3fe8f499f29a 100644 --- a/drivers/power/supply/max17040_battery.c +++ b/drivers/power/supply/max17040_battery.c @@ -190,8 +190,11 @@ static int max17040_raw_vcell_to_uvolts(struct max17040_chip *chip, u16 vcell) static int max17040_get_vcell(struct max17040_chip *chip) { u32 vcell; + int ret; - regmap_read(chip->regmap, MAX17040_VCELL, &vcell); + ret = regmap_read(chip->regmap, MAX17040_VCELL, &vcell); + if (ret) + return ret; return max17040_raw_vcell_to_uvolts(chip, vcell); } @@ -199,8 +202,11 @@ static int max17040_get_vcell(struct max17040_chip *chip) static int max17040_get_soc(struct max17040_chip *chip) { u32 soc; + int ret; - regmap_read(chip->regmap, MAX17040_SOC, &soc); + ret = regmap_read(chip->regmap, MAX17040_SOC, &soc); + if (ret) + return ret; return soc >> (chip->quirk_double_soc ? 9 : 8); } @@ -259,7 +265,11 @@ static int max17040_get_of_data(struct max17040_chip *chip) static void max17040_check_changes(struct max17040_chip *chip) { - chip->soc = max17040_get_soc(chip); + int soc; + + soc = max17040_get_soc(chip); + if (soc >= 0) + chip->soc = soc; } static void max17040_queue_work(struct max17040_chip *chip) @@ -386,16 +396,23 @@ static int max17040_get_property(struct power_supply *psy, union power_supply_propval *val) { struct max17040_chip *chip = power_supply_get_drvdata(psy); + int ret; switch (psp) { case POWER_SUPPLY_PROP_ONLINE: val->intval = max17040_get_online(chip); break; case POWER_SUPPLY_PROP_VOLTAGE_NOW: - val->intval = max17040_get_vcell(chip); + ret = max17040_get_vcell(chip); + if (ret < 0) + return ret; + val->intval = ret; break; case POWER_SUPPLY_PROP_CAPACITY: - val->intval = max17040_get_soc(chip); + ret = max17040_get_soc(chip); + if (ret < 0) + return ret; + val->intval = ret; break; case POWER_SUPPLY_PROP_CAPACITY_ALERT_MIN: val->intval = chip->low_soc_alert; diff --git a/drivers/power/supply/qcom_battmgr.c b/drivers/power/supply/qcom_battmgr.c index 653f70403f44..7b51f827c34c 100644 --- a/drivers/power/supply/qcom_battmgr.c +++ b/drivers/power/supply/qcom_battmgr.c @@ -4,6 +4,7 @@ * Copyright (c) 2022, Linaro Ltd */ #include +#include #include #include #include @@ -1329,6 +1330,7 @@ static int qcom_battmgr_probe(struct auxiliary_device *adev, const struct of_device_id *match; struct qcom_battmgr *battmgr; struct device *dev = &adev->dev; + int ret; battmgr = devm_kzalloc(dev, sizeof(*battmgr), GFP_KERNEL); if (!battmgr) @@ -1344,7 +1346,6 @@ static int qcom_battmgr_probe(struct auxiliary_device *adev, psy_cfg_supply.supplied_to = qcom_battmgr_battery; psy_cfg_supply.num_supplicants = 1; - INIT_WORK(&battmgr->enable_work, qcom_battmgr_enable_worker); mutex_init(&battmgr->lock); init_completion(&battmgr->ack); @@ -1391,6 +1392,11 @@ static int qcom_battmgr_probe(struct auxiliary_device *adev, "failed to register wireless charing power supply\n"); } + ret = devm_work_autocancel(dev, &battmgr->enable_work, + qcom_battmgr_enable_worker); + if (ret) + return ret; + battmgr->client = devm_pmic_glink_client_alloc(dev, PMIC_GLINK_OWNER_BATTMGR, qcom_battmgr_callback, qcom_battmgr_pdr_notify, diff --git a/drivers/ptp/ptp_ocp.c b/drivers/ptp/ptp_ocp.c index aa233a301012..af4f6fc94ff4 100644 --- a/drivers/ptp/ptp_ocp.c +++ b/drivers/ptp/ptp_ocp.c @@ -4143,8 +4143,7 @@ ptp_ocp_detach(struct ptp_ocp *bp) ptp_ocp_debugfs_remove_device(bp); ptp_ocp_detach_sysfs(bp); ptp_ocp_attr_group_del(bp); - if (timer_pending(&bp->watchdog)) - del_timer_sync(&bp->watchdog); + del_timer_sync(&bp->watchdog); if (bp->ts0) ptp_ocp_unregister_ext(bp->ts0); if (bp->ts1) diff --git a/drivers/ptp/ptp_private.h b/drivers/ptp/ptp_private.h index e25be08fa6ae..d5bf50ee33d4 100644 --- a/drivers/ptp/ptp_private.h +++ b/drivers/ptp/ptp_private.h @@ -19,6 +19,7 @@ #define PTP_MAX_TIMESTAMPS 128 #define PTP_BUF_TIMESTAMPS 30 #define PTP_DEFAULT_MAX_VCLOCKS 20 +#define PTP_MAX_VCLOCKS_LIMIT (KMALLOC_MAX_SIZE/(sizeof(int))) enum { PTP_LOCK_PHYSICAL = 0, diff --git a/drivers/ptp/ptp_sysfs.c b/drivers/ptp/ptp_sysfs.c index aefc06ae5d09..02111e6d87a1 100644 --- a/drivers/ptp/ptp_sysfs.c +++ b/drivers/ptp/ptp_sysfs.c @@ -282,7 +282,7 @@ static ssize_t max_vclocks_store(struct device *dev, size_t size; u32 max; - if (kstrtou32(buf, 0, &max) || max == 0) + if (kstrtou32(buf, 0, &max) || max == 0 || max > PTP_MAX_VCLOCKS_LIMIT) return -EINVAL; if (max == ptp->max_vclocks) diff --git a/drivers/pwm/pwm-samsung.c b/drivers/pwm/pwm-samsung.c index 7d0966b07dff..0b83af3e8655 100644 --- a/drivers/pwm/pwm-samsung.c +++ b/drivers/pwm/pwm-samsung.c @@ -102,7 +102,7 @@ struct samsung_pwm_chip { * IP. Should this change, both drivers will need to be modified to * properly synchronize accesses to particular instances. */ -static DEFINE_SPINLOCK(samsung_pwm_lock); +static DEFINE_RAW_SPINLOCK(samsung_pwm_lock); #endif static inline @@ -141,14 +141,14 @@ static void pwm_samsung_set_divisor(struct samsung_pwm_chip *pwm, bits = (fls(divisor) - 1) - pwm->variant.div_base; - spin_lock_irqsave(&samsung_pwm_lock, flags); + raw_spin_lock_irqsave(&samsung_pwm_lock, flags); reg = readl(pwm->base + REG_TCFG1); reg &= ~(TCFG1_MUX_MASK << shift); reg |= bits << shift; writel(reg, pwm->base + REG_TCFG1); - spin_unlock_irqrestore(&samsung_pwm_lock, flags); + raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags); } static int pwm_samsung_is_tdiv(struct samsung_pwm_chip *chip, unsigned int chan) @@ -258,7 +258,7 @@ static int pwm_samsung_enable(struct pwm_chip *chip, struct pwm_device *pwm) unsigned long flags; u32 tcon; - spin_lock_irqsave(&samsung_pwm_lock, flags); + raw_spin_lock_irqsave(&samsung_pwm_lock, flags); tcon = readl(our_chip->base + REG_TCON); @@ -272,7 +272,7 @@ static int pwm_samsung_enable(struct pwm_chip *chip, struct pwm_device *pwm) our_chip->disabled_mask &= ~BIT(pwm->hwpwm); - spin_unlock_irqrestore(&samsung_pwm_lock, flags); + raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags); return 0; } @@ -284,7 +284,7 @@ static void pwm_samsung_disable(struct pwm_chip *chip, struct pwm_device *pwm) unsigned long flags; u32 tcon; - spin_lock_irqsave(&samsung_pwm_lock, flags); + raw_spin_lock_irqsave(&samsung_pwm_lock, flags); tcon = readl(our_chip->base + REG_TCON); tcon &= ~TCON_AUTORELOAD(tcon_chan); @@ -299,7 +299,7 @@ static void pwm_samsung_disable(struct pwm_chip *chip, struct pwm_device *pwm) our_chip->disabled_mask |= BIT(pwm->hwpwm); - spin_unlock_irqrestore(&samsung_pwm_lock, flags); + raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags); } static void pwm_samsung_manual_update(struct samsung_pwm_chip *chip, @@ -307,11 +307,11 @@ static void pwm_samsung_manual_update(struct samsung_pwm_chip *chip, { unsigned long flags; - spin_lock_irqsave(&samsung_pwm_lock, flags); + raw_spin_lock_irqsave(&samsung_pwm_lock, flags); __pwm_samsung_manual_update(chip, pwm); - spin_unlock_irqrestore(&samsung_pwm_lock, flags); + raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags); } static int __pwm_samsung_config(struct pwm_chip *chip, struct pwm_device *pwm, @@ -400,7 +400,7 @@ static void pwm_samsung_set_invert(struct samsung_pwm_chip *chip, unsigned long flags; u32 tcon; - spin_lock_irqsave(&samsung_pwm_lock, flags); + raw_spin_lock_irqsave(&samsung_pwm_lock, flags); tcon = readl(chip->base + REG_TCON); @@ -414,7 +414,7 @@ static void pwm_samsung_set_invert(struct samsung_pwm_chip *chip, writel(tcon, chip->base + REG_TCON); - spin_unlock_irqrestore(&samsung_pwm_lock, flags); + raw_spin_unlock_irqrestore(&samsung_pwm_lock, flags); } static int pwm_samsung_set_polarity(struct pwm_chip *chip, diff --git a/drivers/regulator/ltc3676.c b/drivers/regulator/ltc3676.c index 768cdce0d4ec..1ec218951829 100644 --- a/drivers/regulator/ltc3676.c +++ b/drivers/regulator/ltc3676.c @@ -232,7 +232,7 @@ static const struct regulator_desc ltc3676_regulators[LTC3676_NUM_REGULATORS] = LTC3676_FIXED_REG(LDO4, ldo4, LDOB, 2), }; -static bool ltc3676_readable_writeable_reg(struct device *dev, unsigned int reg) +static bool ltc3676_writeable_reg(struct device *dev, unsigned int reg) { switch (reg) { case LTC3676_BUCK1 ... LTC3676_IRQSTAT: @@ -243,6 +243,15 @@ static bool ltc3676_readable_writeable_reg(struct device *dev, unsigned int reg) return false; } +static bool ltc3676_readable_reg(struct device *dev, unsigned int reg) +{ + switch (reg) { + case LTC3676_BUCK1 ... LTC3676_IRQSTAT: + return true; + } + return false; +} + static bool ltc3676_volatile_reg(struct device *dev, unsigned int reg) { switch (reg) { @@ -255,8 +264,8 @@ static bool ltc3676_volatile_reg(struct device *dev, unsigned int reg) static const struct regmap_config ltc3676_regmap_config = { .reg_bits = 8, .val_bits = 8, - .writeable_reg = ltc3676_readable_writeable_reg, - .readable_reg = ltc3676_readable_writeable_reg, + .writeable_reg = ltc3676_writeable_reg, + .readable_reg = ltc3676_readable_reg, .volatile_reg = ltc3676_volatile_reg, .max_register = LTC3676_CLIRQ, .use_single_read = true, diff --git a/drivers/regulator/tps6594-regulator.c b/drivers/regulator/tps6594-regulator.c index 9ede06398648..4dfa93c74ab4 100644 --- a/drivers/regulator/tps6594-regulator.c +++ b/drivers/regulator/tps6594-regulator.c @@ -241,21 +241,21 @@ static const struct regulator_desc tps65224_buck_regs[] = { TPS65224_MASK_BUCKS_VSET, TPS6594_REG_BUCKX_CTRL(1), TPS6594_BIT_BUCK_EN, 0, 0, tps65224_bucks_2_3_4_ranges, - 4, 0, 0, NULL, 0, 0), + ARRAY_SIZE(tps65224_bucks_2_3_4_ranges), 0, 0, NULL, 0, 0), TPS6594_REGULATOR("BUCK3", "buck3", TPS6594_BUCK_3, REGULATOR_VOLTAGE, tps6594_bucks_ops, TPS65224_MASK_BUCKS_VSET, TPS6594_REG_BUCKX_VOUT_1(2), TPS65224_MASK_BUCKS_VSET, TPS6594_REG_BUCKX_CTRL(2), TPS6594_BIT_BUCK_EN, 0, 0, tps65224_bucks_2_3_4_ranges, - 4, 0, 0, NULL, 0, 0), + ARRAY_SIZE(tps65224_bucks_2_3_4_ranges), 0, 0, NULL, 0, 0), TPS6594_REGULATOR("BUCK4", "buck4", TPS6594_BUCK_4, REGULATOR_VOLTAGE, tps6594_bucks_ops, TPS65224_MASK_BUCKS_VSET, TPS6594_REG_BUCKX_VOUT_1(3), TPS65224_MASK_BUCKS_VSET, TPS6594_REG_BUCKX_CTRL(3), TPS6594_BIT_BUCK_EN, 0, 0, tps65224_bucks_2_3_4_ranges, - 4, 0, 0, NULL, 0, 0), + ARRAY_SIZE(tps65224_bucks_2_3_4_ranges), 0, 0, NULL, 0, 0), }; static const struct tps6594_regulator_irq_type tps6594_buck1_irq_types[] = { diff --git a/drivers/remoteproc/qcom_wcnss.c b/drivers/remoteproc/qcom_wcnss.c index 37174b544113..133cf6ac4b4b 100644 --- a/drivers/remoteproc/qcom_wcnss.c +++ b/drivers/remoteproc/qcom_wcnss.c @@ -441,25 +441,31 @@ static void wcnss_release_pds(struct qcom_wcnss *wcnss) } static int wcnss_init_regulators(struct qcom_wcnss *wcnss, - const struct wcnss_vreg_info *info, - int num_vregs, int num_pd_vregs) + const struct wcnss_data *data) { + const struct wcnss_vreg_info *info = data->vregs; struct regulator_bulk_data *bulk; + size_t i, possible_pds = 0, num_vregs = data->num_vregs; int ret; - int i; + + for (i = 0; i < WCNSS_MAX_PDS; i++) + if (data->pd_names[i]) + possible_pds++; /* * If attaching the power domains suceeded we can skip requesting * the regulators for the power domains. For old device trees we need to * reserve extra space to manage them through the regulator interface. */ - if (wcnss->num_pds) { + if (possible_pds >= num_vregs) { + /* Do nothing if vregs do not include PD regulators (pronto-v3) */ + } else if (wcnss->num_pds) { info += wcnss->num_pds; /* Handle single power domain case */ - if (wcnss->num_pds < num_pd_vregs) - num_vregs += num_pd_vregs - wcnss->num_pds; + if (wcnss->num_pds < data->num_pd_vregs) + num_vregs += data->num_pd_vregs - wcnss->num_pds; } else { - num_vregs += num_pd_vregs; + num_vregs += data->num_pd_vregs; } bulk = devm_kcalloc(wcnss->dev, @@ -614,8 +620,7 @@ static int wcnss_probe(struct platform_device *pdev) if (ret && (ret != -ENODATA || !data->num_pd_vregs)) goto free_rproc; - ret = wcnss_init_regulators(wcnss, data->vregs, data->num_vregs, - data->num_pd_vregs); + ret = wcnss_init_regulators(wcnss, data); if (ret) goto detach_pds; diff --git a/drivers/rtc/dev.c b/drivers/rtc/dev.c index 4aad9bb99868..8ff910017b10 100644 --- a/drivers/rtc/dev.c +++ b/drivers/rtc/dev.c @@ -207,7 +207,7 @@ static long rtc_dev_ioctl(struct file *file, struct rtc_device *rtc = file->private_data; const struct rtc_class_ops *ops = rtc->ops; struct rtc_time tm; - struct rtc_wkalrm alarm; + struct rtc_wkalrm alarm = { 0 }; struct rtc_param param; void __user *uarg = (void __user *)arg; diff --git a/drivers/rtc/rtc-ac100.c b/drivers/rtc/rtc-ac100.c index eaf2c9ab9661..86dabeabb62f 100644 --- a/drivers/rtc/rtc-ac100.c +++ b/drivers/rtc/rtc-ac100.c @@ -316,6 +316,8 @@ static int ac100_rtc_register_clks(struct ac100_rtc_dev *chip) if (!chip->clk_data) return -ENOMEM; + chip->clk_data->num = AC100_CLKOUT_NUM; + chip->rtc_32k_clk = clk_hw_register_fixed_rate(chip->dev, AC100_RTC_32K_NAME, NULL, 0, @@ -359,7 +361,6 @@ static int ac100_rtc_register_clks(struct ac100_rtc_dev *chip) chip->clk_data->hws[i] = &clk->hw; } - chip->clk_data->num = i; ret = of_clk_add_hw_provider(np, of_clk_hw_onecell_get, chip->clk_data); if (ret) goto err_unregister_rtc_32k; diff --git a/drivers/rtc/rtc-mpfs.c b/drivers/rtc/rtc-mpfs.c index aa724a945c08..8335e8c7357e 100644 --- a/drivers/rtc/rtc-mpfs.c +++ b/drivers/rtc/rtc-mpfs.c @@ -256,8 +256,12 @@ static int mpfs_rtc_probe(struct platform_device *pdev) return ret; } + clk = devm_clk_get(&pdev->dev, "rtcref"); + if (IS_ERR(clk)) + return PTR_ERR(clk); + /* prescaler hardware adds 1 to reg value */ - prescaler = clk_get_rate(devm_clk_get(&pdev->dev, "rtcref")) - 1; + prescaler = clk_get_rate(clk) - 1; if (prescaler > MAX_PRESCALER_COUNT) { dev_dbg(&pdev->dev, "invalid prescaler %lu\n", prescaler); return -EINVAL; diff --git a/drivers/rtc/rtc-rzn1.c b/drivers/rtc/rtc-rzn1.c index c64eaf50bb39..668935dbcefe 100644 --- a/drivers/rtc/rtc-rzn1.c +++ b/drivers/rtc/rtc-rzn1.c @@ -193,13 +193,24 @@ static int rzn1_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm) if (ret) return ret; + ctl1 = readl(rtc->base + RZN1_RTC_CTL1); + alrm->enabled = !!(ctl1 & RZN1_RTC_CTL1_ALME); + min = readl(rtc->base + RZN1_RTC_ALM); hour = readl(rtc->base + RZN1_RTC_ALH); - wday = readl(rtc->base + RZN1_RTC_ALW); tm->tm_sec = 0; tm->tm_min = bcd2bin(min); tm->tm_hour = bcd2bin(hour); + + /* + * If wday is zero, no bit is set in RZN1_RTC_ALW. This is the + * register's power-on reset value. + */ + wday = readl(rtc->base + RZN1_RTC_ALW); + if (!wday) + return 0; + delta_days = ((fls(wday) - 1) - tm->tm_wday + 7) % 7; tm->tm_wday = fls(wday) - 1; @@ -208,9 +219,6 @@ static int rzn1_rtc_read_alarm(struct device *dev, struct rtc_wkalrm *alrm) rtc_time64_to_tm(alarm, tm); } - ctl1 = readl(rtc->base + RZN1_RTC_CTL1); - alrm->enabled = !!(ctl1 & RZN1_RTC_CTL1_ALME); - return 0; } diff --git a/drivers/rtc/rtc-spear.c b/drivers/rtc/rtc-spear.c index 1df5c7e94198..14b84d297f4b 100644 --- a/drivers/rtc/rtc-spear.c +++ b/drivers/rtc/rtc-spear.c @@ -363,6 +363,12 @@ static int spear_rtc_probe(struct platform_device *pdev) if (irq < 0) return irq; + config->ioaddr = devm_platform_ioremap_resource(pdev, 0); + if (IS_ERR(config->ioaddr)) + return PTR_ERR(config->ioaddr); + + spin_lock_init(&config->lock); + status = devm_request_irq(&pdev->dev, irq, spear_rtc_irq, 0, pdev->name, config); if (status) { @@ -371,10 +377,6 @@ static int spear_rtc_probe(struct platform_device *pdev) return status; } - config->ioaddr = devm_platform_ioremap_resource(pdev, 0); - if (IS_ERR(config->ioaddr)) - return PTR_ERR(config->ioaddr); - config->clk = devm_clk_get(&pdev->dev, NULL); if (IS_ERR(config->clk)) return PTR_ERR(config->clk); @@ -383,7 +385,6 @@ static int spear_rtc_probe(struct platform_device *pdev) if (status < 0) return status; - spin_lock_init(&config->lock); platform_set_drvdata(pdev, config); config->rtc->ops = &spear_rtc_ops; diff --git a/drivers/s390/cio/vfio_ccw_chp.c b/drivers/s390/cio/vfio_ccw_chp.c index a518234d0753..7708eb4d6de0 100644 --- a/drivers/s390/cio/vfio_ccw_chp.c +++ b/drivers/s390/cio/vfio_ccw_chp.c @@ -132,7 +132,7 @@ static ssize_t vfio_ccw_crw_region_read(struct vfio_ccw_private *private, /* Notify the guest if more CRWs are on our queue */ spin_lock_irqsave(&private->crw_lock, flags); if (!list_empty(&private->crw) && private->crw_trigger) - eventfd_signal(private->crw_trigger, 1); + eventfd_signal(private->crw_trigger); spin_unlock_irqrestore(&private->crw_lock, flags); out: diff --git a/drivers/s390/cio/vfio_ccw_drv.c b/drivers/s390/cio/vfio_ccw_drv.c index e3e91aa10682..6ebd42636f16 100644 --- a/drivers/s390/cio/vfio_ccw_drv.c +++ b/drivers/s390/cio/vfio_ccw_drv.c @@ -112,7 +112,7 @@ void vfio_ccw_sch_io_todo(struct work_struct *work) mutex_unlock(&private->io_mutex); if (private->io_trigger) - eventfd_signal(private->io_trigger, 1); + eventfd_signal(private->io_trigger); } void vfio_ccw_crw_todo(struct work_struct *work) @@ -124,7 +124,7 @@ void vfio_ccw_crw_todo(struct work_struct *work) spin_lock_irqsave(&private->crw_lock, flags); if (!list_empty(&private->crw) && private->crw_trigger) - eventfd_signal(private->crw_trigger, 1); + eventfd_signal(private->crw_trigger); spin_unlock_irqrestore(&private->crw_lock, flags); } diff --git a/drivers/s390/cio/vfio_ccw_ops.c b/drivers/s390/cio/vfio_ccw_ops.c index d1c17969cb98..b7acc2532096 100644 --- a/drivers/s390/cio/vfio_ccw_ops.c +++ b/drivers/s390/cio/vfio_ccw_ops.c @@ -452,7 +452,7 @@ static int vfio_ccw_mdev_set_irqs(struct vfio_ccw_private *private, case VFIO_IRQ_SET_DATA_NONE: { if (*ctx) - eventfd_signal(*ctx, 1); + eventfd_signal(*ctx); return 0; } case VFIO_IRQ_SET_DATA_BOOL: @@ -463,7 +463,7 @@ static int vfio_ccw_mdev_set_irqs(struct vfio_ccw_private *private, return -EFAULT; if (trigger && *ctx) - eventfd_signal(*ctx, 1); + eventfd_signal(*ctx); return 0; } case VFIO_IRQ_SET_DATA_EVENTFD: @@ -643,7 +643,7 @@ static void vfio_ccw_mdev_request(struct vfio_device *vdev, unsigned int count) "Relaying device request to user (#%u)\n", count); - eventfd_signal(private->req_trigger, 1); + eventfd_signal(private->req_trigger); } else if (count == 0) { dev_notice(dev, "No device request channel registered, blocked until released by user\n"); diff --git a/drivers/s390/crypto/pkey_api.c b/drivers/s390/crypto/pkey_api.c index 21fc6151af79..d4d0bec00c92 100644 --- a/drivers/s390/crypto/pkey_api.c +++ b/drivers/s390/crypto/pkey_api.c @@ -336,7 +336,8 @@ static int pkey_verifykey(const struct pkey_seckey *seckey, int rc; /* check the secure key for valid AES secure key */ - rc = cca_check_secaeskeytoken(debug_info, 3, (u8 *)seckey, 0); + rc = cca_check_secaeskeytoken(debug_info, 3, (u8 *)seckey, + sizeof(*seckey), 0); if (rc) goto out; if (pattributes) @@ -839,7 +840,7 @@ static int pkey_verifykey2(const u8 *key, size_t keylen, hdr->version == TOKVER_CCA_AES) { struct secaeskeytoken *t = (struct secaeskeytoken *)key; - rc = cca_check_secaeskeytoken(debug_info, 3, key, 0); + rc = cca_check_secaeskeytoken(debug_info, 3, key, keylen, 0); if (rc) goto out; if (ktype) @@ -869,7 +870,7 @@ static int pkey_verifykey2(const u8 *key, size_t keylen, hdr->version == TOKVER_CCA_VLSC) { struct cipherkeytoken *t = (struct cipherkeytoken *)key; - rc = cca_check_secaescipherkey(debug_info, 3, key, 0, 1); + rc = cca_check_secaescipherkey(debug_info, 3, key, keylen, 0, 1); if (rc) goto out; if (ktype) @@ -981,12 +982,12 @@ static int pkey_keyblob2pkey2(const struct pkey_apqn *apqns, size_t nr_apqns, if (hdr->version == TOKVER_CCA_AES) { if (keylen != sizeof(struct secaeskeytoken)) return -EINVAL; - if (cca_check_secaeskeytoken(debug_info, 3, key, 0)) + if (cca_check_secaeskeytoken(debug_info, 3, key, keylen, 0)) return -EINVAL; } else if (hdr->version == TOKVER_CCA_VLSC) { if (keylen < hdr->len || keylen > MAXCCAVLSCTOKENSIZE) return -EINVAL; - if (cca_check_secaescipherkey(debug_info, 3, key, 0, 1)) + if (cca_check_secaescipherkey(debug_info, 3, key, keylen, 0, 1)) return -EINVAL; } else { DEBUG_ERR("%s unknown CCA internal token version %d\n", @@ -1253,13 +1254,13 @@ static int pkey_keyblob2pkey3(const struct pkey_apqn *apqns, size_t nr_apqns, /* CCA AES data key */ if (keylen != sizeof(struct secaeskeytoken)) return -EINVAL; - if (cca_check_secaeskeytoken(debug_info, 3, key, 0)) + if (cca_check_secaeskeytoken(debug_info, 3, key, keylen, 0)) return -EINVAL; } else if (hdr->version == TOKVER_CCA_VLSC) { /* CCA AES cipher key */ if (keylen < hdr->len || keylen > MAXCCAVLSCTOKENSIZE) return -EINVAL; - if (cca_check_secaescipherkey(debug_info, 3, key, 0, 1)) + if (cca_check_secaescipherkey(debug_info, 3, key, keylen, 0, 1)) return -EINVAL; } else { DEBUG_ERR("%s unknown CCA internal token version %d\n", @@ -1350,11 +1351,10 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, return -EFAULT; rc = cca_genseckey(kgs.cardnr, kgs.domain, kgs.keytype, kgs.seckey.seckey); - DEBUG_DBG("%s cca_genseckey()=%d\n", __func__, rc); - if (rc) - break; - if (copy_to_user(ugs, &kgs, sizeof(kgs))) - return -EFAULT; + pr_debug("%s cca_genseckey()=%d\n", __func__, rc); + if (!rc && copy_to_user(ugs, &kgs, sizeof(kgs))) + rc = -EFAULT; + memzero_explicit(&kgs, sizeof(kgs)); break; } case PKEY_CLR2SECK: { @@ -1381,11 +1381,10 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, rc = cca_sec2protkey(ksp.cardnr, ksp.domain, ksp.seckey.seckey, ksp.protkey.protkey, &ksp.protkey.len, &ksp.protkey.type); - DEBUG_DBG("%s cca_sec2protkey()=%d\n", __func__, rc); - if (rc) - break; - if (copy_to_user(usp, &ksp, sizeof(ksp))) - return -EFAULT; + pr_debug("%s cca_sec2protkey()=%d\n", __func__, rc); + if (!rc && copy_to_user(usp, &ksp, sizeof(ksp))) + rc = -EFAULT; + memzero_explicit(&ksp, sizeof(ksp)); break; } case PKEY_CLR2PROTK: { @@ -1428,11 +1427,10 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, ksp.protkey.len = sizeof(ksp.protkey.protkey); rc = pkey_skey2pkey(ksp.seckey.seckey, ksp.protkey.protkey, &ksp.protkey.len, &ksp.protkey.type); - DEBUG_DBG("%s pkey_skey2pkey()=%d\n", __func__, rc); - if (rc) - break; - if (copy_to_user(usp, &ksp, sizeof(ksp))) - return -EFAULT; + pr_debug("%s pkey_skey2pkey()=%d\n", __func__, rc); + if (!rc && copy_to_user(usp, &ksp, sizeof(ksp))) + rc = -EFAULT; + memzero_explicit(&ksp, sizeof(ksp)); break; } case PKEY_VERIFYKEY: { @@ -1443,11 +1441,10 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, return -EFAULT; rc = pkey_verifykey(&kvk.seckey, &kvk.cardnr, &kvk.domain, &kvk.keysize, &kvk.attributes); - DEBUG_DBG("%s pkey_verifykey()=%d\n", __func__, rc); - if (rc) - break; - if (copy_to_user(uvk, &kvk, sizeof(kvk))) - return -EFAULT; + pr_debug("%s pkey_verifykey()=%d\n", __func__, rc); + if (!rc && copy_to_user(uvk, &kvk, sizeof(kvk))) + rc = -EFAULT; + memzero_explicit(&kvk, sizeof(kvk)); break; } case PKEY_GENPROTK: { @@ -1459,11 +1456,10 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, kgp.protkey.len = sizeof(kgp.protkey.protkey); rc = pkey_genprotkey(kgp.keytype, kgp.protkey.protkey, &kgp.protkey.len, &kgp.protkey.type); - DEBUG_DBG("%s pkey_genprotkey()=%d\n", __func__, rc); - if (rc) - break; - if (copy_to_user(ugp, &kgp, sizeof(kgp))) - return -EFAULT; + pr_debug("%s pkey_genprotkey()=%d\n", __func__, rc); + if (!rc && copy_to_user(ugp, &kgp, sizeof(kgp))) + rc = -EFAULT; + memzero_explicit(&kgp, sizeof(kgp)); break; } case PKEY_VERIFYPROTK: { @@ -1474,7 +1470,8 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, return -EFAULT; rc = pkey_verifyprotkey(kvp.protkey.protkey, kvp.protkey.len, kvp.protkey.type); - DEBUG_DBG("%s pkey_verifyprotkey()=%d\n", __func__, rc); + pr_debug("%s pkey_verifyprotkey()=%d\n", __func__, rc); + memzero_explicit(&kvp, sizeof(kvp)); break; } case PKEY_KBLOB2PROTK: { @@ -1490,13 +1487,11 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, ktp.protkey.len = sizeof(ktp.protkey.protkey); rc = pkey_keyblob2pkey(kkey, ktp.keylen, ktp.protkey.protkey, &ktp.protkey.len, &ktp.protkey.type); - DEBUG_DBG("%s pkey_keyblob2pkey()=%d\n", __func__, rc); - memzero_explicit(kkey, ktp.keylen); - kfree(kkey); - if (rc) - break; - if (copy_to_user(utp, &ktp, sizeof(ktp))) - return -EFAULT; + pr_debug("%s pkey_keyblob2pkey()=%d\n", __func__, rc); + kfree_sensitive(kkey); + if (!rc && copy_to_user(utp, &ktp, sizeof(ktp))) + rc = -EFAULT; + memzero_explicit(&ktp, sizeof(ktp)); break; } case PKEY_GENSECK2: { @@ -1519,26 +1514,26 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, rc = pkey_genseckey2(apqns, kgs.apqn_entries, kgs.type, kgs.size, kgs.keygenflags, kkey, &klen); - DEBUG_DBG("%s pkey_genseckey2()=%d\n", __func__, rc); + pr_debug("%s pkey_genseckey2()=%d\n", __func__, rc); kfree(apqns); if (rc) { - kfree(kkey); + kfree_sensitive(kkey); break; } if (kgs.key) { if (kgs.keylen < klen) { - kfree(kkey); + kfree_sensitive(kkey); return -EINVAL; } if (copy_to_user(kgs.key, kkey, klen)) { - kfree(kkey); + kfree_sensitive(kkey); return -EFAULT; } } kgs.keylen = klen; if (copy_to_user(ugs, &kgs, sizeof(kgs))) rc = -EFAULT; - kfree(kkey); + kfree_sensitive(kkey); break; } case PKEY_CLR2SECK2: { @@ -1564,21 +1559,21 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, rc = pkey_clr2seckey2(apqns, kcs.apqn_entries, kcs.type, kcs.size, kcs.keygenflags, kcs.clrkey.clrkey, kkey, &klen); - DEBUG_DBG("%s pkey_clr2seckey2()=%d\n", __func__, rc); + pr_debug("%s pkey_clr2seckey2()=%d\n", __func__, rc); kfree(apqns); if (rc) { - kfree(kkey); + kfree_sensitive(kkey); memzero_explicit(&kcs, sizeof(kcs)); break; } if (kcs.key) { if (kcs.keylen < klen) { - kfree(kkey); + kfree_sensitive(kkey); memzero_explicit(&kcs, sizeof(kcs)); return -EINVAL; } if (copy_to_user(kcs.key, kkey, klen)) { - kfree(kkey); + kfree_sensitive(kkey); memzero_explicit(&kcs, sizeof(kcs)); return -EFAULT; } @@ -1587,7 +1582,7 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, if (copy_to_user(ucs, &kcs, sizeof(kcs))) rc = -EFAULT; memzero_explicit(&kcs, sizeof(kcs)); - kfree(kkey); + kfree_sensitive(kkey); break; } case PKEY_VERIFYKEY2: { @@ -1603,8 +1598,8 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, rc = pkey_verifykey2(kkey, kvk.keylen, &kvk.cardnr, &kvk.domain, &kvk.type, &kvk.size, &kvk.flags); - DEBUG_DBG("%s pkey_verifykey2()=%d\n", __func__, rc); - kfree(kkey); + pr_debug("%s pkey_verifykey2()=%d\n", __func__, rc); + kfree_sensitive(kkey); if (rc) break; if (copy_to_user(uvk, &kvk, sizeof(kvk))) @@ -1632,14 +1627,12 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, kkey, ktp.keylen, ktp.protkey.protkey, &ktp.protkey.len, &ktp.protkey.type); - DEBUG_DBG("%s pkey_keyblob2pkey2()=%d\n", __func__, rc); + pr_debug("%s pkey_keyblob2pkey2()=%d\n", __func__, rc); kfree(apqns); - memzero_explicit(kkey, ktp.keylen); - kfree(kkey); - if (rc) - break; - if (copy_to_user(utp, &ktp, sizeof(ktp))) - return -EFAULT; + kfree_sensitive(kkey); + if (!rc && copy_to_user(utp, &ktp, sizeof(ktp))) + rc = -EFAULT; + memzero_explicit(&ktp, sizeof(ktp)); break; } case PKEY_APQNS4K: { @@ -1666,8 +1659,8 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, } rc = pkey_apqns4key(kkey, kak.keylen, kak.flags, apqns, &nr_apqns); - DEBUG_DBG("%s pkey_apqns4key()=%d\n", __func__, rc); - kfree(kkey); + pr_debug("%s pkey_apqns4key()=%d\n", __func__, rc); + kfree_sensitive(kkey); if (rc && rc != -ENOSPC) { kfree(apqns); break; @@ -1753,7 +1746,7 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, protkey = kmalloc(protkeylen, GFP_KERNEL); if (!protkey) { kfree(apqns); - kfree(kkey); + kfree_sensitive(kkey); return -ENOMEM; } rc = pkey_keyblob2pkey3(apqns, ktp.apqn_entries, @@ -1761,23 +1754,22 @@ static long pkey_unlocked_ioctl(struct file *filp, unsigned int cmd, protkey, &protkeylen, &ktp.pkeytype); DEBUG_DBG("%s pkey_keyblob2pkey3()=%d\n", __func__, rc); kfree(apqns); - memzero_explicit(kkey, ktp.keylen); - kfree(kkey); + kfree_sensitive(kkey); if (rc) { - kfree(protkey); + kfree_sensitive(protkey); break; } if (ktp.pkey && ktp.pkeylen) { if (protkeylen > ktp.pkeylen) { - kfree(protkey); + kfree_sensitive(protkey); return -EINVAL; } if (copy_to_user(ktp.pkey, protkey, protkeylen)) { - kfree(protkey); + kfree_sensitive(protkey); return -EFAULT; } } - kfree(protkey); + kfree_sensitive(protkey); ktp.pkeylen = protkeylen; if (copy_to_user(utp, &ktp, sizeof(ktp))) return -EFAULT; diff --git a/drivers/s390/crypto/vfio_ap_ops.c b/drivers/s390/crypto/vfio_ap_ops.c index f82c9d56f07b..d6c60802c978 100644 --- a/drivers/s390/crypto/vfio_ap_ops.c +++ b/drivers/s390/crypto/vfio_ap_ops.c @@ -770,11 +770,17 @@ static int vfio_ap_mdev_probe(struct mdev_device *mdev) ret = vfio_register_emulated_iommu_dev(&matrix_mdev->vdev); if (ret) goto err_put_vdev; - matrix_mdev->req_trigger = NULL; + + /* + * Take the matrix_dev->guests_lock mutex before adding the matrix_mdev + * to the mdev_list. All functions that traverse the list must also hold + * this lock to guard against additions to or removals from the list + * while it is being traversed. + */ + mutex_lock(&matrix_dev->guests_lock); dev_set_drvdata(&mdev->dev, matrix_mdev); - mutex_lock(&matrix_dev->mdevs_lock); list_add(&matrix_mdev->node, &matrix_dev->mdev_list); - mutex_unlock(&matrix_dev->mdevs_lock); + mutex_unlock(&matrix_dev->guests_lock); return 0; err_put_vdev: @@ -1877,7 +1883,7 @@ static void vfio_ap_mdev_request(struct vfio_device *vdev, unsigned int count) "Relaying device request to user (#%u)\n", count); - eventfd_signal(matrix_mdev->req_trigger, 1); + eventfd_signal(matrix_mdev->req_trigger); } else if (count == 0) { dev_notice(dev, "No device request registered, blocked until released by user\n"); @@ -2044,6 +2050,8 @@ static struct ap_matrix_mdev *vfio_ap_mdev_for_queue(struct vfio_ap_queue *q) unsigned long apid = AP_QID_CARD(q->apqn); unsigned long apqi = AP_QID_QUEUE(q->apqn); + lockdep_assert_held(&matrix_dev->guests_lock); + list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { if (test_bit_inv(apid, matrix_mdev->matrix.apm) && test_bit_inv(apqi, matrix_mdev->matrix.aqm)) @@ -2063,6 +2071,7 @@ static ssize_t status_show(struct device *dev, struct ap_matrix_mdev *matrix_mdev; struct ap_device *apdev = to_ap_dev(dev); + mutex_lock(&matrix_dev->guests_lock); mutex_lock(&matrix_dev->mdevs_lock); q = dev_get_drvdata(&apdev->device); @@ -2111,6 +2120,7 @@ static ssize_t status_show(struct device *dev, done: mutex_unlock(&matrix_dev->mdevs_lock); + mutex_unlock(&matrix_dev->guests_lock); return nchars; } @@ -2540,6 +2550,12 @@ static void vfio_ap_mdev_cfg_add(unsigned long *apm_add, unsigned long *aqm_add, vfio_ap_filter_apid_by_qtype(apm_add, aqm_add); + /* + * It is safe to traverse this list here because the + * required guard - matrix_dev->guests_lock - is taken in the + * vfio_ap_on_cfg_changed function prior to this function getting + * called. + */ list_for_each_entry(matrix_mdev, &matrix_dev->mdev_list, node) { /* * The mdevs_lock must be held in order to access fields @@ -2607,6 +2623,10 @@ void vfio_ap_on_cfg_changed(struct ap_config_info *cur_cfg_info, if (!cur_cfg_info || !prev_cfg_info) return; + /* + * Take the guests_lock mutex here to guard access to the + * matrix_dev->mdev_list in the two functions called below. + */ mutex_lock(&matrix_dev->guests_lock); vfio_ap_mdev_on_cfg_remove(cur_cfg_info, prev_cfg_info); diff --git a/drivers/s390/crypto/zcrypt_api.c b/drivers/s390/crypto/zcrypt_api.c index 2d3a766991a0..d9d0d3887a88 100644 --- a/drivers/s390/crypto/zcrypt_api.c +++ b/drivers/s390/crypto/zcrypt_api.c @@ -26,6 +26,7 @@ #include #include #include +#include #include #define CREATE_TRACE_POINTS @@ -871,6 +872,7 @@ static long _zcrypt_send_cprb(bool userspace, struct ap_perms *perms, tdom = *domain; if (perms != &ap_perms && tdom < AP_DOMAINS) { if (ap_msg.flags & AP_MSG_FLAG_ADMIN) { + tdom = array_index_nospec(tdom, AP_DOMAINS); if (!test_bit_inv(tdom, perms->adm)) { rc = -ENODEV; goto out; @@ -1050,6 +1052,7 @@ static long _zcrypt_send_ep11_cprb(bool userspace, struct ap_perms *perms, if (perms != &ap_perms && domain < AP_DOMAINS) { if (ap_msg.flags & AP_MSG_FLAG_ADMIN) { + domain = array_index_nospec(domain, AP_DOMAINS); if (!test_bit_inv(domain, perms->adm)) { rc = -ENODEV; goto out_free; diff --git a/drivers/s390/crypto/zcrypt_ccamisc.c b/drivers/s390/crypto/zcrypt_ccamisc.c index b8526219eceb..a9ab757aef68 100644 --- a/drivers/s390/crypto/zcrypt_ccamisc.c +++ b/drivers/s390/crypto/zcrypt_ccamisc.c @@ -51,12 +51,18 @@ static DEFINE_SPINLOCK(cca_info_list_lock); * also checked. Returns 0 on success or errno value on failure. */ int cca_check_secaeskeytoken(debug_info_t *dbg, int dbflvl, - const u8 *token, int keybitsize) + const u8 *token, u32 keysize, int keybitsize) { struct secaeskeytoken *t = (struct secaeskeytoken *)token; #define DBF(...) debug_sprintf_event(dbg, dbflvl, ##__VA_ARGS__) + if (keysize < sizeof(*t)) { + if (dbg) + DBF("%s keysize %u < min token size %zu\n", + __func__, keysize, sizeof(*t)); + return -EINVAL; + } if (t->type != TOKTYPE_CCA_INTERNAL) { if (dbg) DBF("%s token check failed, type 0x%02x != 0x%02x\n", @@ -90,14 +96,20 @@ EXPORT_SYMBOL(cca_check_secaeskeytoken); * Returns 0 on success or errno value on failure. */ int cca_check_secaescipherkey(debug_info_t *dbg, int dbflvl, - const u8 *token, int keybitsize, - int checkcpacfexport) + const u8 *token, u32 keysize, + int keybitsize, int checkcpacfexport) { struct cipherkeytoken *t = (struct cipherkeytoken *)token; bool keybitsizeok = true; #define DBF(...) debug_sprintf_event(dbg, dbflvl, ##__VA_ARGS__) + if (keysize < sizeof(*t)) { + if (dbg) + DBF("%s keysize %u < min token size %zu\n", + __func__, keysize, sizeof(*t)); + return -EINVAL; + } if (t->type != TOKTYPE_CCA_INTERNAL) { if (dbg) DBF("%s token check failed, type 0x%02x != 0x%02x\n", @@ -110,6 +122,18 @@ int cca_check_secaescipherkey(debug_info_t *dbg, int dbflvl, __func__, (int)t->version, TOKVER_CCA_VLSC); return -EINVAL; } + if (t->len > keysize) { + if (dbg) + DBF("%s token check failed, len %d > keysize %u\n", + __func__, (int)t->len, keysize); + return -EINVAL; + } + if (t->len < sizeof(*t)) { + if (dbg) + DBF("%s token check failed, len %d < min token size %zu\n", + __func__, (int)t->len, sizeof(*t)); + return -EINVAL; + } if (t->algtype != 0x02) { if (dbg) DBF("%s token check failed, algtype 0x%02x != 0x02\n", @@ -184,6 +208,12 @@ int cca_check_sececckeytoken(debug_info_t *dbg, int dbflvl, #define DBF(...) debug_sprintf_event(dbg, dbflvl, ##__VA_ARGS__) + if (keysize < sizeof(*t)) { + if (dbg) + DBF("%s keysize %zu < min token size %zu\n", + __func__, keysize, sizeof(*t)); + return -EINVAL; + } if (t->type != TOKTYPE_CCA_INTERNAL_PKA) { if (dbg) DBF("%s token check failed, type 0x%02x != 0x%02x\n", @@ -196,6 +226,12 @@ int cca_check_sececckeytoken(debug_info_t *dbg, int dbflvl, __func__, (int)t->len, keysize); return -EINVAL; } + if (t->len < sizeof(*t)) { + if (dbg) + DBF("%s token check failed, len %d < min token size %zu\n", + __func__, (int)t->len, sizeof(*t)); + return -EINVAL; + } if (t->secid != 0x20) { if (dbg) DBF("%s token check failed, secid 0x%02x != 0x20\n", @@ -419,7 +455,8 @@ int cca_genseckey(u16 cardnr, u16 domain, /* check secure key token */ rc = cca_check_secaeskeytoken(zcrypt_dbf_info, DBF_ERR, - prepparm->lv3.keyblock.tok, 8 * keysize); + prepparm->lv3.keyblock.tok, + seckeysize, 8 * keysize); if (rc) { rc = -EIO; goto out; @@ -557,7 +594,8 @@ int cca_clr2seckey(u16 cardnr, u16 domain, u32 keybitsize, /* check secure key token */ rc = cca_check_secaeskeytoken(zcrypt_dbf_info, DBF_ERR, - prepparm->lv3.keyblock.tok, 8 * keysize); + prepparm->lv3.keyblock.tok, + seckeysize, 8 * keysize); if (rc) { rc = -EIO; goto out; @@ -816,6 +854,7 @@ int cca_gencipherkey(u16 cardnr, u16 domain, u32 keybitsize, u32 keygenflags, } kb; } __packed * prepparm; struct cipherkeytoken *t; + u32 keybuflen; /* get already prepared memory for 2 cprbs with param block each */ rc = alloc_and_prep_cprbmem(PARMBSIZE, &mem, &preqcblk, &prepcblk); @@ -912,23 +951,28 @@ int cca_gencipherkey(u16 cardnr, u16 domain, u32 keybitsize, u32 keygenflags, } /* and some checks on the generated key */ + t = (struct cipherkeytoken *)prepparm->kb.tlv1.gen_key; + if (prepparm->kb.tlv1.len < 2 * sizeof(uint16_t) + sizeof(*t)) { + rc = -EIO; + goto out; + } + keybuflen = prepparm->kb.tlv1.len - 2 * sizeof(uint16_t); rc = cca_check_secaescipherkey(zcrypt_dbf_info, DBF_ERR, prepparm->kb.tlv1.gen_key, - keybitsize, 1); + keybuflen, keybitsize, 1); if (rc) { rc = -EIO; goto out; } /* copy the generated vlsc key token */ - t = (struct cipherkeytoken *)prepparm->kb.tlv1.gen_key; if (keybuf) { - if (*keybufsize >= t->len) - memcpy(keybuf, t, t->len); + if (*keybufsize >= keybuflen) + memcpy(keybuf, t, keybuflen); else rc = -EINVAL; } - *keybufsize = t->len; + *keybufsize = keybuflen; out: free_cprbmem(mem, PARMBSIZE, 0); diff --git a/drivers/s390/crypto/zcrypt_ccamisc.h b/drivers/s390/crypto/zcrypt_ccamisc.h index 5ddf02f965f9..44bf3fb27ce5 100644 --- a/drivers/s390/crypto/zcrypt_ccamisc.h +++ b/drivers/s390/crypto/zcrypt_ccamisc.h @@ -135,7 +135,7 @@ struct eccprivkeytoken { * also checked. Returns 0 on success or errno value on failure. */ int cca_check_secaeskeytoken(debug_info_t *dbg, int dbflvl, - const u8 *token, int keybitsize); + const u8 *token, u32 keysize, int keybitsize); /* * Simple check if the token is a valid CCA secure AES cipher key @@ -145,8 +145,8 @@ int cca_check_secaeskeytoken(debug_info_t *dbg, int dbflvl, * Returns 0 on success or errno value on failure. */ int cca_check_secaescipherkey(debug_info_t *dbg, int dbflvl, - const u8 *token, int keybitsize, - int checkcpacfexport); + const u8 *token, u32 keysize, + int keybitsize, int checkcpacfexport); /* * Simple check if the token is a valid CCA secure ECC private diff --git a/drivers/s390/net/ism_drv.c b/drivers/s390/net/ism_drv.c index 950f45ff8217..183b2f274ebd 100644 --- a/drivers/s390/net/ism_drv.c +++ b/drivers/s390/net/ism_drv.c @@ -243,13 +243,16 @@ static int unregister_sba(struct ism_dev *ism) if (ret && ret != ISM_ERROR) return -EIO; + return 0; +} + +static void ism_free_sba(struct ism_dev *ism) +{ dma_free_coherent(&ism->pdev->dev, PAGE_SIZE, ism->sba, ism->sba_dma_addr); ism->sba = NULL; ism->sba_dma_addr = 0; - - return 0; } static int unregister_ieq(struct ism_dev *ism) @@ -263,13 +266,16 @@ static int unregister_ieq(struct ism_dev *ism) if (ret && ret != ISM_ERROR) return -EIO; + return 0; +} + +static void ism_free_ieq(struct ism_dev *ism) +{ dma_free_coherent(&ism->pdev->dev, PAGE_SIZE, ism->ieq, ism->ieq_dma_addr); ism->ieq = NULL; ism->ieq_dma_addr = 0; - - return 0; } static int ism_read_local_gid(struct ism_dev *ism) @@ -603,8 +609,10 @@ static int ism_dev_init(struct ism_dev *ism) unreg_ieq: unregister_ieq(ism); + ism_free_ieq(ism); unreg_sba: unregister_sba(ism); + ism_free_sba(ism); free_irq: free_irq(pci_irq_vector(pdev, 0), ism); free_client_arr: @@ -702,9 +710,13 @@ static void ism_dev_exit(struct ism_dev *ism) if (SYSTEM_EID.serial_number[0] != '0' || SYSTEM_EID.type[0] != '0') ism_del_vlan_id(ism, ISM_RESERVED_VLANID); + /* ism will only generate new IRQs while ieq & sba are registered */ unregister_ieq(ism); unregister_sba(ism); + /* drain ongoing irpt handlers */ free_irq(pci_irq_vector(pdev, 0), ism); + ism_free_ieq(ism); + ism_free_sba(ism); kfree(ism->sba_client_arr); pci_free_irq_vectors(pdev); list_del_init(&ism->list); diff --git a/drivers/scsi/ch.c b/drivers/scsi/ch.c index cb0a399be1cc..b4c877bb4163 100644 --- a/drivers/scsi/ch.c +++ b/drivers/scsi/ch.c @@ -110,7 +110,6 @@ typedef struct { int minor; char name[8]; struct scsi_device *device; - struct scsi_device **dt; /* ptrs to data transfer elements */ u_int firsts[CH_TYPES]; u_int counts[CH_TYPES]; u_int unit_attention; @@ -358,16 +357,10 @@ ch_readconfig(scsi_changer *ch) vendor_labels[i]); } - /* look up the devices of the data transfer elements */ - ch->dt = kcalloc(ch->counts[CHET_DT], sizeof(*ch->dt), - GFP_KERNEL); - - if (!ch->dt) { - kfree(buffer); - return -ENOMEM; - } - + /* report the devices of the data transfer elements */ for (elem = 0; elem < ch->counts[CHET_DT]; elem++) { + struct scsi_device *sdev; + id = -1; lun = 0; if (elem < CH_DT_MAX && -1 != dt_id[elem]) { @@ -383,10 +376,8 @@ ch_readconfig(scsi_changer *ch) VPRINTK(KERN_INFO, "dt 0x%x: ",elem+ch->firsts[CHET_DT]); if (data[6] & 0x80) { VPRINTK(KERN_CONT, "not this SCSI bus\n"); - ch->dt[elem] = NULL; } else if (0 == (data[6] & 0x30)) { VPRINTK(KERN_CONT, "ID/LUN unknown\n"); - ch->dt[elem] = NULL; } else { id = ch->device->id; lun = 0; @@ -396,18 +387,16 @@ ch_readconfig(scsi_changer *ch) } if (-1 != id) { VPRINTK(KERN_CONT, "ID %i, LUN %i, ",id,lun); - ch->dt[elem] = - scsi_device_lookup(ch->device->host, - ch->device->channel, - id,lun); - if (!ch->dt[elem]) { + sdev = scsi_device_lookup(ch->device->host, + ch->device->channel, + id, lun); + if (!sdev) { /* should not happen */ VPRINTK(KERN_CONT, "Huh? device not found!\n"); } else { VPRINTK(KERN_CONT, "name: %8.8s %16.16s %4.4s\n", - ch->dt[elem]->vendor, - ch->dt[elem]->model, - ch->dt[elem]->rev); + sdev->vendor, sdev->model, sdev->rev); + scsi_device_put(sdev); } } } @@ -571,7 +560,6 @@ static void ch_destroy(struct kref *ref) scsi_changer *ch = container_of(ref, scsi_changer, ref); ch->device = NULL; - kfree(ch->dt); kfree(ch); } diff --git a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c index d386672d59e4..30cad657bb57 100644 --- a/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c +++ b/drivers/scsi/hisi_sas/hisi_sas_v3_hw.c @@ -2977,7 +2977,7 @@ static int slave_configure_v3_hw(struct scsi_device *sdev) return 0; if (!device_link_add(&sdev->sdev_gendev, dev, - DL_FLAG_PM_RUNTIME | DL_FLAG_RPM_ACTIVE)) { + DL_FLAG_STATELESS | DL_FLAG_PM_RUNTIME | DL_FLAG_RPM_ACTIVE)) { if (pm_runtime_enabled(dev)) { dev_info(dev, "add device link failed, disable runtime PM for the host\n"); pm_runtime_disable(dev); @@ -2987,6 +2987,15 @@ static int slave_configure_v3_hw(struct scsi_device *sdev) return 0; } +static void hisi_sas_sdev_destroy(struct scsi_device *sdev) +{ + struct Scsi_Host *shost = dev_to_shost(&sdev->sdev_gendev); + struct hisi_hba *hisi_hba = shost_priv(shost); + struct device *dev = hisi_hba->dev; + + device_link_remove(&sdev->sdev_gendev, dev); +} + static struct attribute *host_v3_hw_attrs[] = { &dev_attr_phy_event_threshold.attr, &dev_attr_intr_conv_v3_hw.attr, @@ -3407,6 +3416,7 @@ static const struct scsi_host_template sht_v3_hw = { .eh_device_reset_handler = sas_eh_device_reset_handler, .eh_target_reset_handler = sas_eh_target_reset_handler, .slave_alloc = hisi_sas_slave_alloc, + .slave_destroy = hisi_sas_sdev_destroy, .target_destroy = sas_target_destroy, .ioctl = sas_ioctl, #ifdef CONFIG_COMPAT diff --git a/drivers/scsi/hosts.c b/drivers/scsi/hosts.c index f274812aee1e..03c2cb126988 100644 --- a/drivers/scsi/hosts.c +++ b/drivers/scsi/hosts.c @@ -344,6 +344,7 @@ static void scsi_host_dev_release(struct device *dev) /* Wait for functions invoked through call_rcu(&scmd->rcu, ...) */ rcu_barrier(); + cancel_work_sync(&shost->eh_work); if (shost->tmf_work_q) destroy_workqueue(shost->tmf_work_q); if (shost->ehandler) @@ -409,6 +410,7 @@ struct Scsi_Host *scsi_host_alloc(const struct scsi_host_template *sht, int priv INIT_LIST_HEAD(&shost->starved_list); init_waitqueue_head(&shost->host_wait); mutex_init(&shost->scan_mutex); + INIT_WORK(&shost->eh_work, scsi_rcu_eh_wakeup); index = ida_alloc(&host_index_ida, GFP_KERNEL); if (index < 0) { diff --git a/drivers/scsi/imm.c b/drivers/scsi/imm.c index 07db98161a03..dbe0598ce4bf 100644 --- a/drivers/scsi/imm.c +++ b/drivers/scsi/imm.c @@ -1269,6 +1269,7 @@ static void imm_detach(struct parport *pb) if (dev->dev->port == pb) { list_del_init(&dev->list); scsi_remove_host(dev->host); + cancel_delayed_work_sync(&dev->imm_tq); scsi_host_put(dev->host); parport_unregister_device(dev->dev); kfree(dev); diff --git a/drivers/scsi/lpfc/lpfc.h b/drivers/scsi/lpfc/lpfc.h index 9670cb2bf198..ce816ecc5302 100644 --- a/drivers/scsi/lpfc/lpfc.h +++ b/drivers/scsi/lpfc/lpfc.h @@ -535,6 +535,36 @@ struct lpfc_cgn_acqe_stat { atomic64_t warn; }; +enum lpfc_fc_flag { + /* Several of these flags are HBA centric and should be moved to + * phba->link_flag (e.g. FC_PTP, FC_PUBLIC_LOOP) + */ + FC_PT2PT, /* pt2pt with no fabric */ + FC_PT2PT_PLOGI, /* pt2pt initiate PLOGI */ + FC_DISC_TMO, /* Discovery timer running */ + FC_PUBLIC_LOOP, /* Public loop */ + FC_LBIT, /* LOGIN bit in loopinit set */ + FC_RSCN_MODE, /* RSCN cmd rcv'ed */ + FC_NLP_MORE, /* More node to process in node tbl */ + FC_OFFLINE_MODE, /* Interface is offline for diag */ + FC_FABRIC, /* We are fabric attached */ + FC_VPORT_LOGO_RCVD, /* LOGO received on vport */ + FC_RSCN_DISCOVERY, /* Auth all devices after RSCN */ + FC_LOGO_RCVD_DID_CHNG, /* FDISC on phys port detect DID chng */ + FC_PT2PT_NO_NVME, /* Don't send NVME PRLI */ + FC_SCSI_SCAN_TMO, /* scsi scan timer running */ + FC_ABORT_DISCOVERY, /* we want to abort discovery */ + FC_NDISC_ACTIVE, /* NPort discovery active */ + FC_BYPASSED_MODE, /* NPort is in bypassed mode */ + FC_VPORT_NEEDS_REG_VPI, /* Needs to have its vpi registered */ + FC_RSCN_DEFERRED, /* A deferred RSCN being processed */ + FC_VPORT_NEEDS_INIT_VPI, /* Need to INIT_VPI before FDISC */ + FC_VPORT_CVL_RCVD, /* VLink failed due to CVL */ + FC_VFI_REGISTERED, /* VFI is registered */ + FC_FDISC_COMPLETED, /* FDISC completed */ + FC_DISC_DELAYED, /* Delay NPort discovery */ +}; + struct lpfc_vport { struct lpfc_hba *phba; struct list_head listentry; @@ -549,34 +579,7 @@ struct lpfc_vport { uint8_t vpi_state; #define LPFC_VPI_REGISTERED 0x1 - uint32_t fc_flag; /* FC flags */ -/* Several of these flags are HBA centric and should be moved to - * phba->link_flag (e.g. FC_PTP, FC_PUBLIC_LOOP) - */ -#define FC_PT2PT 0x1 /* pt2pt with no fabric */ -#define FC_PT2PT_PLOGI 0x2 /* pt2pt initiate PLOGI */ -#define FC_DISC_TMO 0x4 /* Discovery timer running */ -#define FC_PUBLIC_LOOP 0x8 /* Public loop */ -#define FC_LBIT 0x10 /* LOGIN bit in loopinit set */ -#define FC_RSCN_MODE 0x20 /* RSCN cmd rcv'ed */ -#define FC_NLP_MORE 0x40 /* More node to process in node tbl */ -#define FC_OFFLINE_MODE 0x80 /* Interface is offline for diag */ -#define FC_FABRIC 0x100 /* We are fabric attached */ -#define FC_VPORT_LOGO_RCVD 0x200 /* LOGO received on vport */ -#define FC_RSCN_DISCOVERY 0x400 /* Auth all devices after RSCN */ -#define FC_LOGO_RCVD_DID_CHNG 0x800 /* FDISC on phys port detect DID chng*/ -#define FC_PT2PT_NO_NVME 0x1000 /* Don't send NVME PRLI */ -#define FC_SCSI_SCAN_TMO 0x4000 /* scsi scan timer running */ -#define FC_ABORT_DISCOVERY 0x8000 /* we want to abort discovery */ -#define FC_NDISC_ACTIVE 0x10000 /* NPort discovery active */ -#define FC_BYPASSED_MODE 0x20000 /* NPort is in bypassed mode */ -#define FC_VPORT_NEEDS_REG_VPI 0x80000 /* Needs to have its vpi registered */ -#define FC_RSCN_DEFERRED 0x100000 /* A deferred RSCN being processed */ -#define FC_VPORT_NEEDS_INIT_VPI 0x200000 /* Need to INIT_VPI before FDISC */ -#define FC_VPORT_CVL_RCVD 0x400000 /* VLink failed due to CVL */ -#define FC_VFI_REGISTERED 0x800000 /* VFI is registered */ -#define FC_FDISC_COMPLETED 0x1000000/* FDISC completed */ -#define FC_DISC_DELAYED 0x2000000/* Delay NPort discovery */ + unsigned long fc_flag; /* FC flags */ uint32_t ct_flags; #define FC_CT_RFF_ID 0x1 /* RFF_ID accepted by switch */ @@ -587,16 +590,18 @@ struct lpfc_vport { #define FC_CT_RPRT_DEFER 0x20 /* Defer issuing FDMI RPRT */ struct list_head fc_nodes; + spinlock_t fc_nodes_list_lock; /* spinlock for fc_nodes list */ /* Keep counters for the number of entries in each list. */ - uint16_t fc_plogi_cnt; - uint16_t fc_adisc_cnt; - uint16_t fc_reglogin_cnt; - uint16_t fc_prli_cnt; - uint16_t fc_unmap_cnt; - uint16_t fc_map_cnt; - uint16_t fc_npr_cnt; - uint16_t fc_unused_cnt; + atomic_t fc_plogi_cnt; + atomic_t fc_adisc_cnt; + atomic_t fc_reglogin_cnt; + atomic_t fc_prli_cnt; + atomic_t fc_unmap_cnt; + atomic_t fc_map_cnt; + atomic_t fc_npr_cnt; + atomic_t fc_unused_cnt; + struct serv_parm fc_sparam; /* buffer for our service parameters */ uint32_t fc_myDID; /* fibre channel S_ID */ diff --git a/drivers/scsi/lpfc/lpfc_attr.c b/drivers/scsi/lpfc/lpfc_attr.c index 8123062ec2fa..82bffbc2b72e 100644 --- a/drivers/scsi/lpfc/lpfc_attr.c +++ b/drivers/scsi/lpfc/lpfc_attr.c @@ -344,6 +344,7 @@ lpfc_nvme_info_show(struct device *dev, struct device_attribute *attr, struct lpfc_fc4_ctrl_stat *cstat; uint64_t data1, data2, data3; uint64_t totin, totout, tot; + unsigned long iflags; char *statep; int i; int len = 0; @@ -543,7 +544,7 @@ lpfc_nvme_info_show(struct device *dev, struct device_attribute *attr, if (strlcat(buf, tmp, PAGE_SIZE) >= PAGE_SIZE) goto buffer_done; - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_for_each_entry(ndlp, &vport->fc_nodes, nlp_listp) { nrport = NULL; @@ -617,7 +618,7 @@ lpfc_nvme_info_show(struct device *dev, struct device_attribute *attr, if (strlcat(buf, tmp, PAGE_SIZE) >= PAGE_SIZE) goto unlock_buf_done; } - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); if (!lport) goto buffer_done; @@ -681,7 +682,7 @@ lpfc_nvme_info_show(struct device *dev, struct device_attribute *attr, goto buffer_done; unlock_buf_done: - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); buffer_done: len = strnlen(buf, PAGE_SIZE); @@ -1091,14 +1092,14 @@ lpfc_link_state_show(struct device *dev, struct device_attribute *attr, break; } if (phba->fc_topology == LPFC_TOPOLOGY_LOOP) { - if (vport->fc_flag & FC_PUBLIC_LOOP) + if (test_bit(FC_PUBLIC_LOOP, &vport->fc_flag)) len += scnprintf(buf + len, PAGE_SIZE-len, " Public Loop\n"); else len += scnprintf(buf + len, PAGE_SIZE-len, " Private Loop\n"); } else { - if (vport->fc_flag & FC_FABRIC) { + if (test_bit(FC_FABRIC, &vport->fc_flag)) { if (phba->sli_rev == LPFC_SLI_REV4 && vport->port_type == LPFC_PHYSICAL_PORT && phba->sli4_hba.fawwpn_flag & @@ -1260,7 +1261,8 @@ lpfc_num_discovered_ports_show(struct device *dev, struct lpfc_vport *vport = (struct lpfc_vport *) shost->hostdata; return scnprintf(buf, PAGE_SIZE, "%d\n", - vport->fc_map_cnt + vport->fc_unmap_cnt); + atomic_read(&vport->fc_map_cnt) + + atomic_read(&vport->fc_unmap_cnt)); } /** @@ -1289,7 +1291,7 @@ lpfc_issue_lip(struct Scsi_Host *shost) * If the link is offline, disabled or BLOCK_MGMT_IO * it doesn't make any sense to allow issue_lip */ - if ((vport->fc_flag & FC_OFFLINE_MODE) || + if (test_bit(FC_OFFLINE_MODE, &vport->fc_flag) || (phba->hba_flag & LINK_DISABLED) || (phba->sli.sli_flag & LPFC_BLOCK_MGMT_IO)) return -EPERM; @@ -1303,8 +1305,8 @@ lpfc_issue_lip(struct Scsi_Host *shost) pmboxq->u.mb.mbxCommand = MBX_DOWN_LINK; pmboxq->u.mb.mbxOwner = OWN_HOST; - if ((vport->fc_flag & FC_PT2PT) && (vport->fc_flag & FC_PT2PT_NO_NVME)) - vport->fc_flag &= ~FC_PT2PT_NO_NVME; + if (test_bit(FC_PT2PT, &vport->fc_flag)) + clear_bit(FC_PT2PT_NO_NVME, &vport->fc_flag); mbxstatus = lpfc_sli_issue_mbox_wait(phba, pmboxq, LPFC_MBOX_TMO * 2); @@ -1494,7 +1496,8 @@ lpfc_reset_pci_bus(struct lpfc_hba *phba) if (shost) { phba_other = ((struct lpfc_vport *)shost->hostdata)->phba; - if (!(phba_other->pport->fc_flag & FC_OFFLINE_MODE)) { + if (!test_bit(FC_OFFLINE_MODE, + &phba_other->pport->fc_flag)) { lpfc_printf_log(phba_other, KERN_INFO, LOG_INIT, "8349 WWPN = 0x%02x%02x%02x%02x" "%02x%02x%02x%02x is not " @@ -1549,7 +1552,7 @@ lpfc_selective_reset(struct lpfc_hba *phba) if (!phba->cfg_enable_hba_reset) return -EACCES; - if (!(phba->pport->fc_flag & FC_OFFLINE_MODE)) { + if (!test_bit(FC_OFFLINE_MODE, &phba->pport->fc_flag)) { status = lpfc_do_offline(phba, LPFC_EVT_OFFLINE); if (status != 0) @@ -1688,7 +1691,7 @@ lpfc_sli4_pdev_reg_request(struct lpfc_hba *phba, uint32_t opcode) { struct completion online_compl; struct pci_dev *pdev = phba->pcidev; - uint32_t before_fc_flag; + unsigned long before_fc_flag; uint32_t sriov_nr_virtfn; uint32_t reg_val; int status = 0, rc = 0; @@ -1759,7 +1762,7 @@ lpfc_sli4_pdev_reg_request(struct lpfc_hba *phba, uint32_t opcode) } /* keep the original port state */ - if (before_fc_flag & FC_OFFLINE_MODE) { + if (test_bit(FC_OFFLINE_MODE, &before_fc_flag)) { if (phba->fw_dump_cmpl) phba->fw_dump_cmpl = NULL; goto out; @@ -2102,7 +2105,7 @@ lpfc_board_mode_store(struct device *dev, struct device_attribute *attr, *board_mode_str = '\0'; lpfc_printf_vlog(vport, KERN_ERR, LOG_INIT, "3097 Failed \"%s\", status(%d), " - "fc_flag(x%x)\n", + "fc_flag(x%lx)\n", buf, status, phba->pport->fc_flag); return status; } @@ -2161,7 +2164,7 @@ lpfc_get_hba_info(struct lpfc_hba *phba, pmb->mbxOwner = OWN_HOST; pmboxq->ctx_buf = NULL; - if (phba->pport->fc_flag & FC_OFFLINE_MODE) + if (test_bit(FC_OFFLINE_MODE, &phba->pport->fc_flag)) rc = MBX_NOT_FINISHED; else rc = lpfc_sli_issue_mbox_wait(phba, pmboxq, phba->fc_ratov * 2); @@ -3769,15 +3772,14 @@ lpfc_nodev_tmo_init(struct lpfc_vport *vport, int val) static void lpfc_update_rport_devloss_tmo(struct lpfc_vport *vport) { - struct Scsi_Host *shost; struct lpfc_nodelist *ndlp; + unsigned long iflags; #if (IS_ENABLED(CONFIG_NVME_FC)) struct lpfc_nvme_rport *rport; struct nvme_fc_remote_port *remoteport = NULL; #endif - shost = lpfc_shost_from_vport(vport); - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_for_each_entry(ndlp, &vport->fc_nodes, nlp_listp) { if (ndlp->rport) ndlp->rport->dev_loss_tmo = vport->cfg_devloss_tmo; @@ -3792,7 +3794,7 @@ lpfc_update_rport_devloss_tmo(struct lpfc_vport *vport) vport->cfg_devloss_tmo); #endif } - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); } /** @@ -3978,8 +3980,8 @@ lpfc_vport_param_init(tgt_queue_depth, LPFC_MAX_TGT_QDEPTH, static int lpfc_tgt_queue_depth_set(struct lpfc_vport *vport, uint val) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_nodelist *ndlp; + unsigned long iflags; if (!lpfc_rangecheck(val, LPFC_MIN_TGT_QDEPTH, LPFC_MAX_TGT_QDEPTH)) return -EINVAL; @@ -3987,14 +3989,13 @@ lpfc_tgt_queue_depth_set(struct lpfc_vport *vport, uint val) if (val == vport->cfg_tgt_queue_depth) return 0; - spin_lock_irq(shost->host_lock); vport->cfg_tgt_queue_depth = val; /* Next loop thru nodelist and change cmd_qdepth */ + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_for_each_entry(ndlp, &vport->fc_nodes, nlp_listp) ndlp->cmd_qdepth = vport->cfg_tgt_queue_depth; - - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); return 0; } @@ -5240,8 +5241,8 @@ lpfc_vport_param_show(max_scsicmpl_time); static int lpfc_max_scsicmpl_time_set(struct lpfc_vport *vport, int val) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_nodelist *ndlp, *next_ndlp; + unsigned long iflags; if (val == vport->cfg_max_scsicmpl_time) return 0; @@ -5249,13 +5250,13 @@ lpfc_max_scsicmpl_time_set(struct lpfc_vport *vport, int val) return -EINVAL; vport->cfg_max_scsicmpl_time = val; - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_for_each_entry_safe(ndlp, next_ndlp, &vport->fc_nodes, nlp_listp) { if (ndlp->nlp_state == NLP_STE_UNUSED_NODE) continue; ndlp->cmd_qdepth = vport->cfg_tgt_queue_depth; } - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); return 0; } lpfc_vport_param_store(max_scsicmpl_time); @@ -6205,7 +6206,7 @@ sysfs_ctlreg_write(struct file *filp, struct kobject *kobj, if (memcmp(buf, LPFC_REG_WRITE_KEY, LPFC_REG_WRITE_KEY_SIZE)) return -EINVAL; - if (!(vport->fc_flag & FC_OFFLINE_MODE)) + if (!test_bit(FC_OFFLINE_MODE, &vport->fc_flag)) return -EPERM; spin_lock_irq(&phba->hbalock); @@ -6434,26 +6435,22 @@ lpfc_get_host_port_type(struct Scsi_Host *shost) struct lpfc_vport *vport = (struct lpfc_vport *) shost->hostdata; struct lpfc_hba *phba = vport->phba; - spin_lock_irq(shost->host_lock); - if (vport->port_type == LPFC_NPIV_PORT) { fc_host_port_type(shost) = FC_PORTTYPE_NPIV; } else if (lpfc_is_link_up(phba)) { if (phba->fc_topology == LPFC_TOPOLOGY_LOOP) { - if (vport->fc_flag & FC_PUBLIC_LOOP) + if (test_bit(FC_PUBLIC_LOOP, &vport->fc_flag)) fc_host_port_type(shost) = FC_PORTTYPE_NLPORT; else fc_host_port_type(shost) = FC_PORTTYPE_LPORT; } else { - if (vport->fc_flag & FC_FABRIC) + if (test_bit(FC_FABRIC, &vport->fc_flag)) fc_host_port_type(shost) = FC_PORTTYPE_NPORT; else fc_host_port_type(shost) = FC_PORTTYPE_PTP; } } else fc_host_port_type(shost) = FC_PORTTYPE_UNKNOWN; - - spin_unlock_irq(shost->host_lock); } /** @@ -6466,9 +6463,7 @@ lpfc_get_host_port_state(struct Scsi_Host *shost) struct lpfc_vport *vport = (struct lpfc_vport *) shost->hostdata; struct lpfc_hba *phba = vport->phba; - spin_lock_irq(shost->host_lock); - - if (vport->fc_flag & FC_OFFLINE_MODE) + if (test_bit(FC_OFFLINE_MODE, &vport->fc_flag)) fc_host_port_state(shost) = FC_PORTSTATE_OFFLINE; else { switch (phba->link_state) { @@ -6495,8 +6490,6 @@ lpfc_get_host_port_state(struct Scsi_Host *shost) break; } } - - spin_unlock_irq(shost->host_lock); } /** @@ -6509,8 +6502,6 @@ lpfc_get_host_speed(struct Scsi_Host *shost) struct lpfc_vport *vport = (struct lpfc_vport *) shost->hostdata; struct lpfc_hba *phba = vport->phba; - spin_lock_irq(shost->host_lock); - if ((lpfc_is_link_up(phba)) && (!(phba->hba_flag & HBA_FCOE_MODE))) { switch(phba->fc_linkspeed) { case LPFC_LINK_SPEED_1GHZ: @@ -6573,8 +6564,6 @@ lpfc_get_host_speed(struct Scsi_Host *shost) } } else fc_host_speed(shost) = FC_PORTSPEED_UNKNOWN; - - spin_unlock_irq(shost->host_lock); } /** @@ -6588,19 +6577,15 @@ lpfc_get_host_fabric_name (struct Scsi_Host *shost) struct lpfc_hba *phba = vport->phba; u64 node_name; - spin_lock_irq(shost->host_lock); - - if ((vport->port_state > LPFC_FLOGI) && - ((vport->fc_flag & FC_FABRIC) || - ((phba->fc_topology == LPFC_TOPOLOGY_LOOP) && - (vport->fc_flag & FC_PUBLIC_LOOP)))) + if (vport->port_state > LPFC_FLOGI && + (test_bit(FC_FABRIC, &vport->fc_flag) || + (phba->fc_topology == LPFC_TOPOLOGY_LOOP && + test_bit(FC_PUBLIC_LOOP, &vport->fc_flag)))) node_name = wwn_to_u64(phba->fc_fabparam.nodeName.u.wwn); else /* fabric is local port if there is no F/FL_Port */ node_name = 0; - spin_unlock_irq(shost->host_lock); - fc_host_fabric_name(shost) = node_name; } @@ -6651,7 +6636,7 @@ lpfc_get_stats(struct Scsi_Host *shost) pmboxq->ctx_buf = NULL; pmboxq->vport = vport; - if (vport->fc_flag & FC_OFFLINE_MODE) { + if (test_bit(FC_OFFLINE_MODE, &vport->fc_flag)) { rc = lpfc_sli_issue_mbox(phba, pmboxq, MBX_POLL); if (rc != MBX_SUCCESS) { mempool_free(pmboxq, phba->mbox_mem_pool); @@ -6704,7 +6689,7 @@ lpfc_get_stats(struct Scsi_Host *shost) pmboxq->ctx_buf = NULL; pmboxq->vport = vport; - if (vport->fc_flag & FC_OFFLINE_MODE) { + if (test_bit(FC_OFFLINE_MODE, &vport->fc_flag)) { rc = lpfc_sli_issue_mbox(phba, pmboxq, MBX_POLL); if (rc != MBX_SUCCESS) { mempool_free(pmboxq, phba->mbox_mem_pool); @@ -6791,8 +6776,8 @@ lpfc_reset_stats(struct Scsi_Host *shost) pmboxq->ctx_buf = NULL; pmboxq->vport = vport; - if ((vport->fc_flag & FC_OFFLINE_MODE) || - (!(psli->sli_flag & LPFC_SLI_ACTIVE))) { + if (test_bit(FC_OFFLINE_MODE, &vport->fc_flag) || + !(psli->sli_flag & LPFC_SLI_ACTIVE)) { rc = lpfc_sli_issue_mbox(phba, pmboxq, MBX_POLL); if (rc != MBX_SUCCESS) { mempool_free(pmboxq, phba->mbox_mem_pool); @@ -6813,8 +6798,8 @@ lpfc_reset_stats(struct Scsi_Host *shost) pmboxq->ctx_buf = NULL; pmboxq->vport = vport; - if ((vport->fc_flag & FC_OFFLINE_MODE) || - (!(psli->sli_flag & LPFC_SLI_ACTIVE))) { + if (test_bit(FC_OFFLINE_MODE, &vport->fc_flag) || + !(psli->sli_flag & LPFC_SLI_ACTIVE)) { rc = lpfc_sli_issue_mbox(phba, pmboxq, MBX_POLL); if (rc != MBX_SUCCESS) { mempool_free(pmboxq, phba->mbox_mem_pool); @@ -6873,17 +6858,19 @@ lpfc_get_node_by_target(struct scsi_target *starget) struct Scsi_Host *shost = dev_to_shost(starget->dev.parent); struct lpfc_vport *vport = (struct lpfc_vport *) shost->hostdata; struct lpfc_nodelist *ndlp; + unsigned long iflags; - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); /* Search for this, mapped, target ID */ list_for_each_entry(ndlp, &vport->fc_nodes, nlp_listp) { if (ndlp->nlp_state == NLP_STE_MAPPED_NODE && starget->id == ndlp->nlp_sid) { - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, + iflags); return ndlp; } } - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); return NULL; } diff --git a/drivers/scsi/lpfc/lpfc_bsg.c b/drivers/scsi/lpfc/lpfc_bsg.c index 0166f86c7b71..62588aaed13e 100644 --- a/drivers/scsi/lpfc/lpfc_bsg.c +++ b/drivers/scsi/lpfc/lpfc_bsg.c @@ -1977,7 +1977,7 @@ lpfc_sli4_bsg_set_loopback_mode(struct lpfc_hba *phba, int mode, static int lpfc_sli4_diag_fcport_reg_setup(struct lpfc_hba *phba) { - if (phba->pport->fc_flag & FC_VFI_REGISTERED) { + if (test_bit(FC_VFI_REGISTERED, &phba->pport->fc_flag)) { lpfc_printf_log(phba, KERN_WARNING, LOG_LIBDFC, "3136 Port still had vfi registered: " "mydid:x%x, fcfi:%d, vfi:%d, vpi:%d\n", @@ -3448,7 +3448,7 @@ static int lpfc_bsg_check_cmd_access(struct lpfc_hba *phba, case MBX_RUN_DIAGS: case MBX_RESTART: case MBX_SET_MASK: - if (!(vport->fc_flag & FC_OFFLINE_MODE)) { + if (!test_bit(FC_OFFLINE_MODE, &vport->fc_flag)) { lpfc_printf_log(phba, KERN_WARNING, LOG_LIBDFC, "2743 Command 0x%x is illegal in on-line " "state\n", @@ -4747,7 +4747,7 @@ lpfc_bsg_issue_mbox(struct lpfc_hba *phba, struct bsg_job *job, if (mbox_req->inExtWLen || mbox_req->outExtWLen) { from = pmbx; ext = from + sizeof(MAILBOX_t); - pmboxq->ctx_buf = ext; + pmboxq->ext_buf = ext; pmboxq->in_ext_byte_len = mbox_req->inExtWLen * sizeof(uint32_t); pmboxq->out_ext_byte_len = @@ -4886,7 +4886,7 @@ lpfc_bsg_issue_mbox(struct lpfc_hba *phba, struct bsg_job *job, dd_data->context_un.mbox.outExtWLen = mbox_req->outExtWLen; job->dd_data = dd_data; - if ((vport->fc_flag & FC_OFFLINE_MODE) || + if (test_bit(FC_OFFLINE_MODE, &vport->fc_flag) || (!(phba->sli.sli_flag & LPFC_SLI_ACTIVE))) { rc = lpfc_sli_issue_mbox(phba, pmboxq, MBX_POLL); if (rc != MBX_SUCCESS) { diff --git a/drivers/scsi/lpfc/lpfc_ct.c b/drivers/scsi/lpfc/lpfc_ct.c index 177511523986..9443e0fd5cb5 100644 --- a/drivers/scsi/lpfc/lpfc_ct.c +++ b/drivers/scsi/lpfc/lpfc_ct.c @@ -265,7 +265,7 @@ lpfc_ct_reject_event(struct lpfc_nodelist *ndlp, kfree(mp); ct_exit: lpfc_printf_vlog(vport, KERN_ERR, LOG_ELS, - "6440 Unsol CT: Rsp err %d Data: x%x\n", + "6440 Unsol CT: Rsp err %d Data: x%lx\n", rc, vport->fc_flag); } @@ -298,7 +298,7 @@ lpfc_ct_handle_mibreq(struct lpfc_hba *phba, struct lpfc_iocbq *ctiocbq) } /* Ignore traffic received during vport shutdown */ - if (vport->fc_flag & FC_UNLOADING) + if (test_bit(FC_UNLOADING, &vport->fc_flag)) return; ndlp = lpfc_findnode_did(vport, did); @@ -723,7 +723,7 @@ lpfc_prep_node_fc4type(struct lpfc_vport *vport, uint32_t Did, uint8_t fc4_type) if (ndlp) { lpfc_debugfs_disc_trc(vport, LPFC_DISC_TRC_CT, - "Parse GID_FTrsp: did:x%x flg:x%x x%x", + "Parse GID_FTrsp: did:x%x flg:x%lx x%x", Did, ndlp->nlp_flag, vport->fc_flag); /* By default, the driver expects to support FCP FC4 */ @@ -735,7 +735,7 @@ lpfc_prep_node_fc4type(struct lpfc_vport *vport, uint32_t Did, uint8_t fc4_type) lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0238 Process x%06x NameServer Rsp " - "Data: x%x x%x x%x x%x x%x\n", Did, + "Data: x%x x%x x%x x%lx x%x\n", Did, ndlp->nlp_flag, ndlp->nlp_fc4_type, ndlp->nlp_state, vport->fc_flag, vport->fc_rscn_id_cnt); @@ -751,20 +751,20 @@ lpfc_prep_node_fc4type(struct lpfc_vport *vport, uint32_t Did, uint8_t fc4_type) } } else { lpfc_debugfs_disc_trc(vport, LPFC_DISC_TRC_CT, - "Skip1 GID_FTrsp: did:x%x flg:x%x cnt:%d", + "Skip1 GID_FTrsp: did:x%x flg:x%lx cnt:%d", Did, vport->fc_flag, vport->fc_rscn_id_cnt); lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0239 Skip x%06x NameServer Rsp " - "Data: x%x x%x x%px\n", + "Data: x%lx x%x x%px\n", Did, vport->fc_flag, vport->fc_rscn_id_cnt, ndlp); } } else { - if (!(vport->fc_flag & FC_RSCN_MODE) || + if (!test_bit(FC_RSCN_MODE, &vport->fc_flag) || lpfc_rscn_payload_check(vport, Did)) { lpfc_debugfs_disc_trc(vport, LPFC_DISC_TRC_CT, - "Query GID_FTrsp: did:x%x flg:x%x cnt:%d", + "Query GID_FTrsp: did:x%x flg:x%lx cnt:%d", Did, vport->fc_flag, vport->fc_rscn_id_cnt); /* @@ -787,12 +787,12 @@ lpfc_prep_node_fc4type(struct lpfc_vport *vport, uint32_t Did, uint8_t fc4_type) lpfc_setup_disc_node(vport, Did); } else { lpfc_debugfs_disc_trc(vport, LPFC_DISC_TRC_CT, - "Skip2 GID_FTrsp: did:x%x flg:x%x cnt:%d", + "Skip2 GID_FTrsp: did:x%x flg:x%lx cnt:%d", Did, vport->fc_flag, vport->fc_rscn_id_cnt); lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0245 Skip x%06x NameServer Rsp " - "Data: x%x x%x\n", Did, + "Data: x%lx x%x\n", Did, vport->fc_flag, vport->fc_rscn_id_cnt); } @@ -914,7 +914,6 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, struct lpfc_iocbq *rspiocb) { struct lpfc_vport *vport = cmdiocb->vport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_dmabuf *outp; struct lpfc_dmabuf *inp; struct lpfc_sli_ct_request *CTrsp; @@ -943,9 +942,9 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, goto out; } - /* Don't bother processing response if vport is being torn down. */ - if (vport->load_flag & FC_UNLOADING) { - if (vport->fc_flag & FC_RSCN_MODE) + /* Skip processing response on pport if unloading */ + if (vport == phba->pport && vport->load_flag & FC_UNLOADING) { + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) lpfc_els_flush_rscn(vport); goto out; } @@ -953,7 +952,7 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, if (lpfc_els_chk_latt(vport)) { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0216 Link event during NS query\n"); - if (vport->fc_flag & FC_RSCN_MODE) + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) lpfc_els_flush_rscn(vport); lpfc_vport_set_state(vport, FC_VPORT_FAILED); goto out; @@ -961,22 +960,18 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, if (lpfc_error_lost_link(vport, ulp_status, ulp_word4)) { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0226 NS query failed due to link event: " - "ulp_status x%x ulp_word4 x%x fc_flag x%x " + "ulp_status x%x ulp_word4 x%x fc_flag x%lx " "port_state x%x gidft_inp x%x\n", ulp_status, ulp_word4, vport->fc_flag, vport->port_state, vport->gidft_inp); - if (vport->fc_flag & FC_RSCN_MODE) + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) lpfc_els_flush_rscn(vport); if (vport->gidft_inp) vport->gidft_inp--; goto out; } - spin_lock_irq(shost->host_lock); - if (vport->fc_flag & FC_RSCN_DEFERRED) { - vport->fc_flag &= ~FC_RSCN_DEFERRED; - spin_unlock_irq(shost->host_lock); - + if (test_and_clear_bit(FC_RSCN_DEFERRED, &vport->fc_flag)) { /* This is a GID_FT completing so the gidft_inp counter was * incremented before the GID_FT was issued to the wire. */ @@ -988,13 +983,12 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, * Re-issue the NS cmd */ lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, - "0151 Process Deferred RSCN Data: x%x x%x\n", + "0151 Process Deferred RSCN Data: x%lx x%x\n", vport->fc_flag, vport->fc_rscn_id_cnt); lpfc_els_handle_rscn(vport); goto out; } - spin_unlock_irq(shost->host_lock); if (ulp_status) { /* Check for retry */ @@ -1018,7 +1012,7 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, vport->gidft_inp--; } } - if (vport->fc_flag & FC_RSCN_MODE) + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) lpfc_els_flush_rscn(vport); lpfc_vport_set_state(vport, FC_VPORT_FAILED); lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, @@ -1031,7 +1025,7 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, if (CTrsp->CommandResponse.bits.CmdRsp == cpu_to_be16(SLI_CT_RESPONSE_FS_ACC)) { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, - "0208 NameServer Rsp Data: x%x x%x " + "0208 NameServer Rsp Data: x%lx x%x " "x%x x%x sz x%x\n", vport->fc_flag, CTreq->un.gid.Fc4Type, @@ -1051,7 +1045,7 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0269 No NameServer Entries " - "Data: x%x x%x x%x x%x\n", + "Data: x%x x%x x%x x%lx\n", be16_to_cpu(CTrsp->CommandResponse.bits.CmdRsp), (uint32_t) CTrsp->ReasonCode, (uint32_t) CTrsp->Explanation, @@ -1066,7 +1060,7 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0240 NameServer Rsp Error " - "Data: x%x x%x x%x x%x\n", + "Data: x%x x%x x%x x%lx\n", be16_to_cpu(CTrsp->CommandResponse.bits.CmdRsp), (uint32_t) CTrsp->ReasonCode, (uint32_t) CTrsp->Explanation, @@ -1084,7 +1078,7 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, /* NameServer Rsp Error */ lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, "0241 NameServer Rsp Error " - "Data: x%x x%x x%x x%x\n", + "Data: x%x x%x x%x x%lx\n", be16_to_cpu(CTrsp->CommandResponse.bits.CmdRsp), (uint32_t) CTrsp->ReasonCode, (uint32_t) CTrsp->Explanation, @@ -1113,14 +1107,13 @@ lpfc_cmpl_ct_cmd_gid_ft(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, * current driver state. */ if (vport->port_state >= LPFC_DISC_AUTH) { - if (vport->fc_flag & FC_RSCN_MODE) { + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) { lpfc_els_flush_rscn(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_RSCN_MODE; /* RSCN still */ - spin_unlock_irq(shost->host_lock); - } - else + /* RSCN still */ + set_bit(FC_RSCN_MODE, &vport->fc_flag); + } else { lpfc_els_flush_rscn(vport); + } } lpfc_disc_start(vport); @@ -1136,7 +1129,6 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, struct lpfc_iocbq *rspiocb) { struct lpfc_vport *vport = cmdiocb->vport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_dmabuf *outp; struct lpfc_dmabuf *inp; struct lpfc_sli_ct_request *CTrsp; @@ -1166,9 +1158,9 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, goto out; } - /* Don't bother processing response if vport is being torn down. */ - if (vport->load_flag & FC_UNLOADING) { - if (vport->fc_flag & FC_RSCN_MODE) + /* Skip processing response on pport if unloading */ + if (vport == phba->pport && vport->load_flag & FC_UNLOADING) { + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) lpfc_els_flush_rscn(vport); goto out; } @@ -1176,7 +1168,7 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, if (lpfc_els_chk_latt(vport)) { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "4108 Link event during NS query\n"); - if (vport->fc_flag & FC_RSCN_MODE) + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) lpfc_els_flush_rscn(vport); lpfc_vport_set_state(vport, FC_VPORT_FAILED); goto out; @@ -1184,22 +1176,18 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, if (lpfc_error_lost_link(vport, ulp_status, ulp_word4)) { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "4166 NS query failed due to link event: " - "ulp_status x%x ulp_word4 x%x fc_flag x%x " + "ulp_status x%x ulp_word4 x%x fc_flag x%lx " "port_state x%x gidft_inp x%x\n", ulp_status, ulp_word4, vport->fc_flag, vport->port_state, vport->gidft_inp); - if (vport->fc_flag & FC_RSCN_MODE) + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) lpfc_els_flush_rscn(vport); if (vport->gidft_inp) vport->gidft_inp--; goto out; } - spin_lock_irq(shost->host_lock); - if (vport->fc_flag & FC_RSCN_DEFERRED) { - vport->fc_flag &= ~FC_RSCN_DEFERRED; - spin_unlock_irq(shost->host_lock); - + if (test_and_clear_bit(FC_RSCN_DEFERRED, &vport->fc_flag)) { /* This is a GID_PT completing so the gidft_inp counter was * incremented before the GID_PT was issued to the wire. */ @@ -1211,13 +1199,12 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, * Re-issue the NS cmd */ lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, - "4167 Process Deferred RSCN Data: x%x x%x\n", + "4167 Process Deferred RSCN Data: x%lx x%x\n", vport->fc_flag, vport->fc_rscn_id_cnt); lpfc_els_handle_rscn(vport); goto out; } - spin_unlock_irq(shost->host_lock); if (ulp_status) { /* Check for retry */ @@ -1237,7 +1224,7 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, vport->gidft_inp--; } } - if (vport->fc_flag & FC_RSCN_MODE) + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) lpfc_els_flush_rscn(vport); lpfc_vport_set_state(vport, FC_VPORT_FAILED); lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, @@ -1250,7 +1237,7 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, if (be16_to_cpu(CTrsp->CommandResponse.bits.CmdRsp) == SLI_CT_RESPONSE_FS_ACC) { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, - "4105 NameServer Rsp Data: x%x x%x " + "4105 NameServer Rsp Data: x%lx x%x " "x%x x%x sz x%x\n", vport->fc_flag, CTreq->un.gid.Fc4Type, @@ -1270,7 +1257,7 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, lpfc_printf_vlog( vport, KERN_INFO, LOG_DISCOVERY, "4106 No NameServer Entries " - "Data: x%x x%x x%x x%x\n", + "Data: x%x x%x x%x x%lx\n", be16_to_cpu(CTrsp->CommandResponse.bits.CmdRsp), (uint32_t)CTrsp->ReasonCode, (uint32_t)CTrsp->Explanation, @@ -1286,7 +1273,7 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, lpfc_printf_vlog( vport, KERN_INFO, LOG_DISCOVERY, "4107 NameServer Rsp Error " - "Data: x%x x%x x%x x%x\n", + "Data: x%x x%x x%x x%lx\n", be16_to_cpu(CTrsp->CommandResponse.bits.CmdRsp), (uint32_t)CTrsp->ReasonCode, (uint32_t)CTrsp->Explanation, @@ -1303,7 +1290,7 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, /* NameServer Rsp Error */ lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, "4109 NameServer Rsp Error " - "Data: x%x x%x x%x x%x\n", + "Data: x%x x%x x%x x%lx\n", be16_to_cpu(CTrsp->CommandResponse.bits.CmdRsp), (uint32_t)CTrsp->ReasonCode, (uint32_t)CTrsp->Explanation, @@ -1333,11 +1320,10 @@ lpfc_cmpl_ct_cmd_gid_pt(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, * current driver state. */ if (vport->port_state >= LPFC_DISC_AUTH) { - if (vport->fc_flag & FC_RSCN_MODE) { + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) { lpfc_els_flush_rscn(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_RSCN_MODE; /* RSCN still */ - spin_unlock_irq(shost->host_lock); + /* RSCN still */ + set_bit(FC_RSCN_MODE, &vport->fc_flag); } else { lpfc_els_flush_rscn(vport); } @@ -1355,7 +1341,6 @@ lpfc_cmpl_ct_cmd_gff_id(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, struct lpfc_iocbq *rspiocb) { struct lpfc_vport *vport = cmdiocb->vport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_dmabuf *inp = cmdiocb->cmd_dmabuf; struct lpfc_dmabuf *outp = cmdiocb->rsp_dmabuf; struct lpfc_sli_ct_request *CTrsp; @@ -1445,7 +1430,7 @@ lpfc_cmpl_ct_cmd_gff_id(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, } lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, "0267 NameServer GFF Rsp " - "x%x Error (%d %d) Data: x%x x%x\n", + "x%x Error (%d %d) Data: x%lx x%x\n", did, ulp_status, ulp_word4, vport->fc_flag, vport->fc_rscn_id_cnt); } @@ -1455,13 +1440,13 @@ lpfc_cmpl_ct_cmd_gff_id(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, if (ndlp) { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0242 Process x%x GFF " - "NameServer Rsp Data: x%x x%x x%x\n", + "NameServer Rsp Data: x%x x%lx x%x\n", did, ndlp->nlp_flag, vport->fc_flag, vport->fc_rscn_id_cnt); } else { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0243 Skip x%x GFF " - "NameServer Rsp Data: x%x x%x\n", did, + "NameServer Rsp Data: x%lx x%x\n", did, vport->fc_flag, vport->fc_rscn_id_cnt); } out: @@ -1480,14 +1465,13 @@ lpfc_cmpl_ct_cmd_gff_id(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, * current driver state. */ if (vport->port_state >= LPFC_DISC_AUTH) { - if (vport->fc_flag & FC_RSCN_MODE) { + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) { lpfc_els_flush_rscn(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_RSCN_MODE; /* RSCN still */ - spin_unlock_irq(shost->host_lock); - } - else + /* RSCN still */ + set_bit(FC_RSCN_MODE, &vport->fc_flag); + } else { lpfc_els_flush_rscn(vport); + } } lpfc_disc_start(vport); } @@ -1865,11 +1849,10 @@ static uint32_t lpfc_find_map_node(struct lpfc_vport *vport) { struct lpfc_nodelist *ndlp, *next_ndlp; - struct Scsi_Host *shost; + unsigned long iflags; uint32_t cnt = 0; - shost = lpfc_shost_from_vport(vport); - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_for_each_entry_safe(ndlp, next_ndlp, &vport->fc_nodes, nlp_listp) { if (ndlp->nlp_type & NLP_FABRIC) continue; @@ -1877,7 +1860,7 @@ lpfc_find_map_node(struct lpfc_vport *vport) (ndlp->nlp_state == NLP_STE_UNMAPPED_NODE)) cnt++; } - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); return cnt; } @@ -1962,7 +1945,7 @@ lpfc_ns_cmd(struct lpfc_vport *vport, int cmdcode, /* NameServer Req */ lpfc_printf_vlog(vport, KERN_INFO ,LOG_DISCOVERY, - "0236 NameServer Req Data: x%x x%x x%x x%x\n", + "0236 NameServer Req Data: x%x x%lx x%x x%x\n", cmdcode, vport->fc_flag, vport->fc_rscn_id_cnt, context); @@ -2179,7 +2162,8 @@ lpfc_ns_cmd(struct lpfc_vport *vport, int cmdcode, kfree(mp); ns_cmd_exit: lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, - "0266 Issue NameServer Req x%x err %d Data: x%x x%x\n", + "0266 Issue NameServer Req x%x err %d Data: x%lx " + "x%x\n", cmdcode, rc, vport->fc_flag, vport->fc_rscn_id_cnt); return 1; } @@ -2465,7 +2449,7 @@ lpfc_fdmi_change_check(struct lpfc_vport *vport) return; /* Must be connected to a Fabric */ - if (!(vport->fc_flag & FC_FABRIC)) + if (!test_bit(FC_FABRIC, &vport->fc_flag)) return; ndlp = lpfc_findnode_did(vport, FDMI_DID); @@ -3245,7 +3229,7 @@ lpfc_fdmi_cmd(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, /* FDMI request */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, - "0218 FDMI Request x%x mask x%x Data: x%x x%x x%x\n", + "0218 FDMI Request x%x mask x%x Data: x%x x%lx x%x\n", cmdcode, new_mask, vport->fdmi_port_mask, vport->fc_flag, vport->port_state); @@ -3482,15 +3466,8 @@ lpfc_delayed_disc_tmo(struct timer_list *t) void lpfc_delayed_disc_timeout_handler(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); - - spin_lock_irq(shost->host_lock); - if (!(vport->fc_flag & FC_DISC_DELAYED)) { - spin_unlock_irq(shost->host_lock); + if (!test_and_clear_bit(FC_DISC_DELAYED, &vport->fc_flag)) return; - } - vport->fc_flag &= ~FC_DISC_DELAYED; - spin_unlock_irq(shost->host_lock); lpfc_do_scr_ns_plogi(vport->phba, vport); } @@ -3741,7 +3718,7 @@ lpfc_vmid_cmd(struct lpfc_vport *vport, INIT_LIST_HEAD(&bmp->list); lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, - "3275 VMID Request Data: x%x x%x x%x\n", + "3275 VMID Request Data: x%lx x%x x%x\n", vport->fc_flag, vport->port_state, cmdcode); ctreq = (struct lpfc_sli_ct_request *)mp->virt; data = mp->virt; diff --git a/drivers/scsi/lpfc/lpfc_debugfs.c b/drivers/scsi/lpfc/lpfc_debugfs.c index 6b6c964e8076..e7da5d562f48 100644 --- a/drivers/scsi/lpfc/lpfc_debugfs.c +++ b/drivers/scsi/lpfc/lpfc_debugfs.c @@ -806,10 +806,10 @@ lpfc_debugfs_nodelist_data(struct lpfc_vport *vport, char *buf, int size) { int len = 0; int i, iocnt, outio, cnt; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; struct lpfc_nodelist *ndlp; unsigned char *statep; + unsigned long iflags; struct nvme_fc_local_port *localport; struct nvme_fc_remote_port *nrport = NULL; struct lpfc_nvme_rport *rport; @@ -818,7 +818,7 @@ lpfc_debugfs_nodelist_data(struct lpfc_vport *vport, char *buf, int size) outio = 0; len += scnprintf(buf+len, size-len, "\nFCP Nodelist Entries ...\n"); - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_for_each_entry(ndlp, &vport->fc_nodes, nlp_listp) { iocnt = 0; if (!cnt) { @@ -908,7 +908,7 @@ lpfc_debugfs_nodelist_data(struct lpfc_vport *vport, char *buf, int size) ndlp->nlp_defer_did); len += scnprintf(buf+len, size-len, "\n"); } - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); len += scnprintf(buf + len, size - len, "\nOutstanding IO x%x\n", outio); @@ -940,8 +940,6 @@ lpfc_debugfs_nodelist_data(struct lpfc_vport *vport, char *buf, int size) if (!localport) goto out_exit; - spin_lock_irq(shost->host_lock); - /* Port state is only one of two values for now. */ if (localport->port_id) statep = "ONLINE"; @@ -953,6 +951,7 @@ lpfc_debugfs_nodelist_data(struct lpfc_vport *vport, char *buf, int size) localport->port_id, statep); len += scnprintf(buf + len, size - len, "\tRport List:\n"); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_for_each_entry(ndlp, &vport->fc_nodes, nlp_listp) { /* local short-hand pointer. */ spin_lock(&ndlp->lock); @@ -1006,8 +1005,7 @@ lpfc_debugfs_nodelist_data(struct lpfc_vport *vport, char *buf, int size) /* Terminate the string. */ len += scnprintf(buf + len, size - len, "\n"); } - - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); out_exit: return len; } diff --git a/drivers/scsi/lpfc/lpfc_els.c b/drivers/scsi/lpfc/lpfc_els.c index 2e9972a58781..3611d4d7ddc5 100644 --- a/drivers/scsi/lpfc/lpfc_els.c +++ b/drivers/scsi/lpfc/lpfc_els.c @@ -93,7 +93,6 @@ static void lpfc_vmid_put_cs_ctl(struct lpfc_vport *vport, u32 ctcl_vmid); int lpfc_els_chk_latt(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; uint32_t ha_copy; @@ -121,9 +120,7 @@ lpfc_els_chk_latt(struct lpfc_vport *vport) * will cleanup any left over in-progress discovery * events. */ - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_ABORT_DISCOVERY; - spin_unlock_irq(shost->host_lock); + set_bit(FC_ABORT_DISCOVERY, &vport->fc_flag); if (phba->link_state != LPFC_CLEAR_LA) lpfc_issue_clear_la(phba, vport); @@ -292,7 +289,7 @@ lpfc_prep_els_iocb(struct lpfc_vport *vport, u8 expect_rsp, lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, "0116 Xmit ELS command x%x to remote " "NPORT x%x I/O tag: x%x, port state:x%x " - "rpi x%x fc_flag:x%x\n", + "rpi x%x fc_flag:x%lx\n", elscmd, did, elsiocb->iotag, vport->port_state, ndlp->nlp_rpi, vport->fc_flag); @@ -301,7 +298,7 @@ lpfc_prep_els_iocb(struct lpfc_vport *vport, u8 expect_rsp, lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, "0117 Xmit ELS response x%x to remote " "NPORT x%x I/O tag: x%x, size: x%x " - "port_state x%x rpi x%x fc_flag x%x\n", + "port_state x%x rpi x%x fc_flag x%lx\n", elscmd, ndlp->nlp_DID, elsiocb->iotag, cmd_size, vport->port_state, ndlp->nlp_rpi, vport->fc_flag); @@ -443,7 +440,7 @@ lpfc_issue_reg_vfi(struct lpfc_vport *vport) /* move forward in case of SLI4 FC port loopback test and pt2pt mode */ if ((phba->sli_rev == LPFC_SLI_REV4) && !(phba->link_flag & LS_LOOPBACK_MODE) && - !(vport->fc_flag & FC_PT2PT)) { + !test_bit(FC_PT2PT, &vport->fc_flag)) { ndlp = lpfc_findnode_did(vport, Fabric_DID); if (!ndlp) { rc = -ENODEV; @@ -458,7 +455,8 @@ lpfc_issue_reg_vfi(struct lpfc_vport *vport) } /* Supply CSP's only if we are fabric connect or pt-to-pt connect */ - if ((vport->fc_flag & FC_FABRIC) || (vport->fc_flag & FC_PT2PT)) { + if (test_bit(FC_FABRIC, &vport->fc_flag) || + test_bit(FC_PT2PT, &vport->fc_flag)) { rc = lpfc_mbox_rsrc_prep(phba, mboxq); if (rc) { rc = -ENOMEM; @@ -511,7 +509,6 @@ int lpfc_issue_unreg_vfi(struct lpfc_vport *vport) { struct lpfc_hba *phba = vport->phba; - struct Scsi_Host *shost; LPFC_MBOXQ_t *mboxq; int rc; @@ -537,10 +534,7 @@ lpfc_issue_unreg_vfi(struct lpfc_vport *vport) return -EIO; } - shost = lpfc_shost_from_vport(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_VFI_REGISTERED; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_VFI_REGISTERED, &vport->fc_flag); return 0; } @@ -568,7 +562,6 @@ lpfc_check_clean_addr_bit(struct lpfc_vport *vport, { struct lpfc_hba *phba = vport->phba; uint8_t fabric_param_changed = 0; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); if ((vport->fc_prevDID != vport->fc_myDID) || memcmp(&vport->fabric_portname, &sp->portName, @@ -590,11 +583,8 @@ lpfc_check_clean_addr_bit(struct lpfc_vport *vport, * - lpfc_delay_discovery module parameter is set. */ if (fabric_param_changed && !sp->cmn.clean_address_bit && - (vport->fc_prevDID || phba->cfg_delay_discovery)) { - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_DISC_DELAYED; - spin_unlock_irq(shost->host_lock); - } + (vport->fc_prevDID || phba->cfg_delay_discovery)) + set_bit(FC_DISC_DELAYED, &vport->fc_flag); return fabric_param_changed; } @@ -624,15 +614,12 @@ static int lpfc_cmpl_els_flogi_fabric(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, struct serv_parm *sp, uint32_t ulp_word4) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; struct lpfc_nodelist *np; struct lpfc_nodelist *next_np; uint8_t fabric_param_changed; - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_FABRIC; - spin_unlock_irq(shost->host_lock); + set_bit(FC_FABRIC, &vport->fc_flag); phba->fc_edtov = be32_to_cpu(sp->cmn.e_d_tov); if (sp->cmn.edtovResolution) /* E_D_TOV ticks are in nanoseconds */ @@ -641,11 +628,8 @@ lpfc_cmpl_els_flogi_fabric(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, phba->fc_edtovResol = sp->cmn.edtovResolution; phba->fc_ratov = (be32_to_cpu(sp->cmn.w2.r_a_tov) + 999) / 1000; - if (phba->fc_topology == LPFC_TOPOLOGY_LOOP) { - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_PUBLIC_LOOP; - spin_unlock_irq(shost->host_lock); - } + if (phba->fc_topology == LPFC_TOPOLOGY_LOOP) + set_bit(FC_PUBLIC_LOOP, &vport->fc_flag); vport->fc_myDID = ulp_word4 & Mask_DID; memcpy(&ndlp->nlp_portname, &sp->portName, sizeof(struct lpfc_name)); @@ -719,12 +703,12 @@ lpfc_cmpl_els_flogi_fabric(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, lpfc_unregister_fcf_prep(phba); /* This should just update the VFI CSPs*/ - if (vport->fc_flag & FC_VFI_REGISTERED) + if (test_bit(FC_VFI_REGISTERED, &vport->fc_flag)) lpfc_issue_reg_vfi(vport); } if (fabric_param_changed && - !(vport->fc_flag & FC_VPORT_NEEDS_REG_VPI)) { + !test_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag)) { /* If our NportID changed, we need to ensure all * remaining NPORTs get unreg_login'ed. @@ -744,20 +728,16 @@ lpfc_cmpl_els_flogi_fabric(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, if (phba->sli_rev == LPFC_SLI_REV4) { lpfc_sli4_unreg_all_rpis(vport); lpfc_mbx_unreg_vpi(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_VPORT_NEEDS_INIT_VPI; - spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_NEEDS_INIT_VPI, &vport->fc_flag); } /* * For SLI3 and SLI4, the VPI needs to be reregistered in * response to this fabric parameter change event. */ - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_VPORT_NEEDS_REG_VPI; - spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); } else if ((phba->sli_rev == LPFC_SLI_REV4) && - !(vport->fc_flag & FC_VPORT_NEEDS_REG_VPI)) { + !test_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag)) { /* * Driver needs to re-reg VPI in order for f/w * to update the MAC address. @@ -770,18 +750,18 @@ lpfc_cmpl_els_flogi_fabric(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, if (phba->sli_rev < LPFC_SLI_REV4) { lpfc_nlp_set_state(vport, ndlp, NLP_STE_REG_LOGIN_ISSUE); if (phba->sli3_options & LPFC_SLI3_NPIV_ENABLED && - vport->fc_flag & FC_VPORT_NEEDS_REG_VPI) + test_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag)) lpfc_register_new_vport(phba, vport, ndlp); else lpfc_issue_fabric_reglogin(vport); } else { ndlp->nlp_type |= NLP_FABRIC; lpfc_nlp_set_state(vport, ndlp, NLP_STE_UNMAPPED_NODE); - if ((!(vport->fc_flag & FC_VPORT_NEEDS_REG_VPI)) && - (vport->vpi_state & LPFC_VPI_REGISTERED)) { + if ((!test_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag)) && + (vport->vpi_state & LPFC_VPI_REGISTERED)) { lpfc_start_fdiscs(phba); lpfc_do_scr_ns_plogi(phba, vport); - } else if (vport->fc_flag & FC_VFI_REGISTERED) + } else if (test_bit(FC_VFI_REGISTERED, &vport->fc_flag)) lpfc_issue_init_vpi(vport); else { lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, @@ -817,15 +797,13 @@ static int lpfc_cmpl_els_flogi_nport(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, struct serv_parm *sp) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; LPFC_MBOXQ_t *mbox; int rc; - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~(FC_FABRIC | FC_PUBLIC_LOOP); - vport->fc_flag |= FC_PT2PT; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_FABRIC, &vport->fc_flag); + clear_bit(FC_PUBLIC_LOOP, &vport->fc_flag); + set_bit(FC_PT2PT, &vport->fc_flag); /* If we are pt2pt with another NPort, force NPIV off! */ phba->sli3_options &= ~LPFC_SLI3_NPIV_ENABLED; @@ -833,10 +811,7 @@ lpfc_cmpl_els_flogi_nport(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, /* If physical FC port changed, unreg VFI and ALL VPIs / RPIs */ if ((phba->sli_rev == LPFC_SLI_REV4) && phba->fc_topology_changed) { lpfc_unregister_fcf_prep(phba); - - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_VFI_REGISTERED; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_VFI_REGISTERED, &vport->fc_flag); phba->fc_topology_changed = 0; } @@ -845,9 +820,7 @@ lpfc_cmpl_els_flogi_nport(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, if (rc >= 0) { /* This side will initiate the PLOGI */ - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_PT2PT_PLOGI; - spin_unlock_irq(shost->host_lock); + set_bit(FC_PT2PT_PLOGI, &vport->fc_flag); /* * N_Port ID cannot be 0, set our Id to LocalID @@ -944,7 +917,6 @@ lpfc_cmpl_els_flogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, struct lpfc_iocbq *rspiocb) { struct lpfc_vport *vport = cmdiocb->vport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_nodelist *ndlp = cmdiocb->ndlp; IOCB_t *irsp; struct lpfc_dmabuf *pcmd = cmdiocb->cmd_dmabuf, *prsp; @@ -1060,10 +1032,9 @@ lpfc_cmpl_els_flogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, } /* FLOGI failed, so there is no fabric */ - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~(FC_FABRIC | FC_PUBLIC_LOOP | - FC_PT2PT_NO_NVME); - spin_unlock_irq(shost->host_lock); + clear_bit(FC_FABRIC, &vport->fc_flag); + clear_bit(FC_PUBLIC_LOOP, &vport->fc_flag); + clear_bit(FC_PT2PT_NO_NVME, &vport->fc_flag); /* If private loop, then allow max outstanding els to be * LPFC_MAX_DISC_THREADS (32). Scanning in the case of no @@ -1072,15 +1043,14 @@ lpfc_cmpl_els_flogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, if (phba->alpa_map[0] == 0) vport->cfg_discovery_threads = LPFC_MAX_DISC_THREADS; if ((phba->sli_rev == LPFC_SLI_REV4) && - (!(vport->fc_flag & FC_VFI_REGISTERED) || + (!test_bit(FC_VFI_REGISTERED, &vport->fc_flag) || (vport->fc_prevDID != vport->fc_myDID) || phba->fc_topology_changed)) { - if (vport->fc_flag & FC_VFI_REGISTERED) { + if (test_bit(FC_VFI_REGISTERED, &vport->fc_flag)) { if (phba->fc_topology_changed) { lpfc_unregister_fcf_prep(phba); - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_VFI_REGISTERED; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_VFI_REGISTERED, + &vport->fc_flag); phba->fc_topology_changed = 0; } else { lpfc_sli4_unreg_all_rpis(vport); @@ -1095,10 +1065,8 @@ lpfc_cmpl_els_flogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, } goto flogifail; } - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_VPORT_CVL_RCVD; - vport->fc_flag &= ~FC_VPORT_LOGO_RCVD; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_VPORT_CVL_RCVD, &vport->fc_flag); + clear_bit(FC_VPORT_LOGO_RCVD, &vport->fc_flag); /* * The FLOGI succeeded. Sync the data for the CPU before @@ -1112,7 +1080,7 @@ lpfc_cmpl_els_flogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, /* FLOGI completes successfully */ lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, "0101 FLOGI completes successfully, I/O tag:x%x " - "xri x%x Data: x%x x%x x%x x%x x%x x%x x%x %d\n", + "xri x%x Data: x%x x%x x%x x%x x%x x%lx x%x %d\n", cmdiocb->iotag, cmdiocb->sli4_xritag, ulp_word4, sp->cmn.e_d_tov, sp->cmn.w2.r_a_tov, sp->cmn.edtovResolution, @@ -1191,7 +1159,7 @@ lpfc_cmpl_els_flogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, goto out; } } else if (vport->port_state > LPFC_FLOGI && - vport->fc_flag & FC_PT2PT) { + test_bit(FC_PT2PT, &vport->fc_flag)) { /* * In a p2p topology, it is possible that discovery has * already progressed, and this completion can be ignored. @@ -1495,8 +1463,9 @@ lpfc_els_abort_flogi(struct lpfc_hba *phba) if (ulp_command == CMD_ELS_REQUEST64_CR) { ndlp = iocb->ndlp; if (ndlp && ndlp->nlp_DID == Fabric_DID) { - if ((phba->pport->fc_flag & FC_PT2PT) && - !(phba->pport->fc_flag & FC_PT2PT_PLOGI)) + if (test_bit(FC_PT2PT, &phba->pport->fc_flag) && + !test_bit(FC_PT2PT_PLOGI, + &phba->pport->fc_flag)) iocb->fabric_cmd_cmpl = lpfc_ignore_els_cmpl; lpfc_sli_issue_abort_iotag(phba, pring, iocb, @@ -1551,7 +1520,7 @@ lpfc_initial_flogi(struct lpfc_vport *vport) } /* Reset the Fabric flag, topology change may have happened */ - vport->fc_flag &= ~FC_FABRIC; + clear_bit(FC_FABRIC, &vport->fc_flag); if (lpfc_issue_els_flogi(vport, ndlp, 0)) { /* A node reference should be retained while registered with a * transport or dev-loss-evt work is pending. @@ -1634,11 +1603,12 @@ lpfc_more_plogi(struct lpfc_vport *vport) /* Continue discovery with PLOGIs to go */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0232 Continue discovery with %d PLOGIs to go " - "Data: x%x x%x x%x\n", - vport->num_disc_nodes, vport->fc_plogi_cnt, + "Data: x%x x%lx x%x\n", + vport->num_disc_nodes, + atomic_read(&vport->fc_plogi_cnt), vport->fc_flag, vport->port_state); /* Check to see if there are more PLOGIs to be sent */ - if (vport->fc_flag & FC_NLP_MORE) + if (test_bit(FC_NLP_MORE, &vport->fc_flag)) /* go thru NPR nodes and issue any remaining ELS PLOGIs */ lpfc_els_disc_plogi(vport); @@ -1742,7 +1712,7 @@ lpfc_plogi_confirm_nport(struct lpfc_hba *phba, uint32_t *prsp, * would have updated nlp_fc4_type in ndlp, so we must ensure * new_ndlp has the right value. */ - if (vport->fc_flag & FC_FABRIC) { + if (test_bit(FC_FABRIC, &vport->fc_flag)) { keep_nlp_fc4_type = new_ndlp->nlp_fc4_type; new_ndlp->nlp_fc4_type = ndlp->nlp_fc4_type; } @@ -1903,21 +1873,17 @@ lpfc_plogi_confirm_nport(struct lpfc_hba *phba, uint32_t *prsp, void lpfc_end_rscn(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); - if (vport->fc_flag & FC_RSCN_MODE) { + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) { /* * Check to see if more RSCNs came in while we were * processing this one. */ if (vport->fc_rscn_id_cnt || - (vport->fc_flag & FC_RSCN_DISCOVERY) != 0) + test_bit(FC_RSCN_DISCOVERY, &vport->fc_flag)) lpfc_els_handle_rscn(vport); - else { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_RSCN_MODE; - spin_unlock_irq(shost->host_lock); - } + else + clear_bit(FC_RSCN_MODE, &vport->fc_flag); } } @@ -2004,7 +1970,6 @@ lpfc_cmpl_els_plogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, struct lpfc_iocbq *rspiocb) { struct lpfc_vport *vport = cmdiocb->vport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); IOCB_t *irsp; struct lpfc_nodelist *ndlp, *free_ndlp; struct lpfc_dmabuf *prsp; @@ -2146,9 +2111,7 @@ lpfc_cmpl_els_plogi(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, lpfc_more_plogi(vport); if (vport->num_disc_nodes == 0) { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_NDISC_ACTIVE; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_NDISC_ACTIVE, &vport->fc_flag); lpfc_can_disctmo(vport); lpfc_end_rscn(vport); @@ -2210,7 +2173,7 @@ lpfc_issue_els_plogi(struct lpfc_vport *vport, uint32_t did, uint8_t retry) */ if ((ndlp->nlp_flag & (NLP_IGNR_REG_CMPL | NLP_UNREG_INP)) && ((ndlp->nlp_DID & Fabric_DID_MASK) != Fabric_DID_MASK) && - !(vport->fc_flag & FC_OFFLINE_MODE)) { + !test_bit(FC_OFFLINE_MODE, &vport->fc_flag)) { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "4110 Issue PLOGI x%x deferred " "on NPort x%x rpi x%x flg x%x Data:" @@ -2242,7 +2205,8 @@ lpfc_issue_els_plogi(struct lpfc_vport *vport, uint32_t did, uint8_t retry) * If we are a N-port connected to a Fabric, fix-up paramm's so logins * to device on remote loops work. */ - if ((vport->fc_flag & FC_FABRIC) && !(vport->fc_flag & FC_PUBLIC_LOOP)) + if (test_bit(FC_FABRIC, &vport->fc_flag) && + !test_bit(FC_PUBLIC_LOOP, &vport->fc_flag)) sp->cmn.altBbCredit = 1; if (sp->cmn.fcphLow < FC_PH_4_3) @@ -2365,8 +2329,8 @@ lpfc_cmpl_els_prli(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, /* If we don't send GFT_ID to Fabric, a PRLI error * could be expected. */ - if ((vport->fc_flag & FC_FABRIC) || - (vport->cfg_enable_fc4_type != LPFC_ENABLE_BOTH)) { + if (test_bit(FC_FABRIC, &vport->fc_flag) || + vport->cfg_enable_fc4_type != LPFC_ENABLE_BOTH) { mode = KERN_ERR; loglevel = LOG_TRACE_EVENT; } else { @@ -2407,7 +2371,7 @@ lpfc_cmpl_els_prli(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, * For P2P topology, retain the node so that PLOGI can be * attempted on it again. */ - if (vport->fc_flag & FC_PT2PT) + if (test_bit(FC_PT2PT, &vport->fc_flag)) goto out; /* As long as this node is not registered with the SCSI @@ -2483,7 +2447,7 @@ lpfc_issue_els_prli(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, * the remote NPort beng a NVME Target. */ if (phba->sli_rev == LPFC_SLI_REV4 && - vport->fc_flag & FC_RSCN_MODE && + test_bit(FC_RSCN_MODE, &vport->fc_flag) && vport->nvmei_support) ndlp->nlp_fc4_type |= NLP_FC4_NVME; local_nlp_type = ndlp->nlp_fc4_type; @@ -2660,7 +2624,7 @@ lpfc_rscn_disc(struct lpfc_vport *vport) /* RSCN discovery */ /* go thru NPR nodes and issue ELS PLOGIs */ - if (vport->fc_npr_cnt) + if (atomic_read(&vport->fc_npr_cnt)) if (lpfc_els_disc_plogi(vport)) return; @@ -2680,7 +2644,6 @@ lpfc_rscn_disc(struct lpfc_vport *vport) static void lpfc_adisc_done(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; /* @@ -2688,7 +2651,7 @@ lpfc_adisc_done(struct lpfc_vport *vport) * and continue discovery. */ if ((phba->sli3_options & LPFC_SLI3_NPIV_ENABLED) && - !(vport->fc_flag & FC_RSCN_MODE) && + !test_bit(FC_RSCN_MODE, &vport->fc_flag) && (phba->sli_rev < LPFC_SLI_REV4)) { /* @@ -2717,15 +2680,13 @@ lpfc_adisc_done(struct lpfc_vport *vport) if (vport->port_state < LPFC_VPORT_READY) { /* If we get here, there is nothing to ADISC */ lpfc_issue_clear_la(phba, vport); - if (!(vport->fc_flag & FC_ABORT_DISCOVERY)) { + if (!test_bit(FC_ABORT_DISCOVERY, &vport->fc_flag)) { vport->num_disc_nodes = 0; /* go thru NPR list, issue ELS PLOGIs */ - if (vport->fc_npr_cnt) + if (atomic_read(&vport->fc_npr_cnt)) lpfc_els_disc_plogi(vport); if (!vport->num_disc_nodes) { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_NDISC_ACTIVE; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_NDISC_ACTIVE, &vport->fc_flag); lpfc_can_disctmo(vport); lpfc_end_rscn(vport); } @@ -2752,11 +2713,12 @@ lpfc_more_adisc(struct lpfc_vport *vport) /* Continue discovery with ADISCs to go */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0210 Continue discovery with %d ADISCs to go " - "Data: x%x x%x x%x\n", - vport->num_disc_nodes, vport->fc_adisc_cnt, + "Data: x%x x%lx x%x\n", + vport->num_disc_nodes, + atomic_read(&vport->fc_adisc_cnt), vport->fc_flag, vport->port_state); /* Check to see if there are more ADISCs to be sent */ - if (vport->fc_flag & FC_NLP_MORE) { + if (test_bit(FC_NLP_MORE, &vport->fc_flag)) { lpfc_set_disctmo(vport); /* go thru NPR nodes and issue any remaining ELS ADISCs */ lpfc_els_disc_adisc(vport); @@ -3591,10 +3553,10 @@ lpfc_issue_els_rscn(struct lpfc_vport *vport, uint8_t retry) /* Not supported for private loop */ if (phba->fc_topology == LPFC_TOPOLOGY_LOOP && - !(vport->fc_flag & FC_PUBLIC_LOOP)) + !test_bit(FC_PUBLIC_LOOP, &vport->fc_flag)) return 1; - if (vport->fc_flag & FC_PT2PT) { + if (test_bit(FC_PT2PT, &vport->fc_flag)) { /* find any mapped nport - that would be the other nport */ ndlp = lpfc_findnode_mapped(vport); if (!ndlp) @@ -4369,7 +4331,6 @@ lpfc_issue_els_edc(struct lpfc_vport *vport, uint8_t retry) void lpfc_cancel_retry_delay_tmo(struct lpfc_vport *vport, struct lpfc_nodelist *nlp) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_work_evt *evtp; if (!(nlp->nlp_flag & NLP_DELAY_TMO)) @@ -4397,9 +4358,8 @@ lpfc_cancel_retry_delay_tmo(struct lpfc_vport *vport, struct lpfc_nodelist *nlp) /* Check if there are more PLOGIs to be sent */ lpfc_more_plogi(vport); if (vport->num_disc_nodes == 0) { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_NDISC_ACTIVE; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_NDISC_ACTIVE, + &vport->fc_flag); lpfc_can_disctmo(vport); lpfc_end_rscn(vport); } @@ -4516,7 +4476,7 @@ lpfc_els_retry_delay_handler(struct lpfc_nodelist *ndlp) } break; case ELS_CMD_FDISC: - if (!(vport->fc_flag & FC_VPORT_NEEDS_INIT_VPI)) + if (!test_bit(FC_VPORT_NEEDS_INIT_VPI, &vport->fc_flag)) lpfc_issue_els_fdisc(vport, ndlp, retry); break; } @@ -4754,7 +4714,7 @@ lpfc_els_retry(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, /* Added for Vendor specifc support * Just keep retrying for these Rsn / Exp codes */ - if ((vport->fc_flag & FC_PT2PT) && + if (test_bit(FC_PT2PT, &vport->fc_flag) && cmd == ELS_CMD_NVMEPRLI) { switch (stat.un.b.lsRjtRsnCode) { case LSRJT_UNABLE_TPC: @@ -4767,7 +4727,7 @@ lpfc_els_retry(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, "support NVME, disabling NVME\n", stat.un.b.lsRjtRsnCode); retry = 0; - vport->fc_flag |= FC_PT2PT_NO_NVME; + set_bit(FC_PT2PT_NO_NVME, &vport->fc_flag); goto out_retry; } } @@ -4990,7 +4950,7 @@ lpfc_els_retry(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, /* If discovery / RSCN timer is running, reset it */ if (timer_pending(&vport->fc_disctmo) || - (vport->fc_flag & FC_RSCN_MODE)) + test_bit(FC_RSCN_MODE, &vport->fc_flag)) lpfc_set_disctmo(vport); } @@ -5378,7 +5338,7 @@ lpfc_cmpl_els_rsp(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, if (ulp_status == 0 && (ndlp->nlp_flag & NLP_ACC_REGLOGIN)) { if (!lpfc_unreg_rpi(vport, ndlp) && - (!(vport->fc_flag & FC_PT2PT))) { + !test_bit(FC_PT2PT, &vport->fc_flag)) { if (ndlp->nlp_state == NLP_STE_PLOGI_ISSUE || ndlp->nlp_state == NLP_STE_REG_LOGIN_ISSUE) { @@ -5750,7 +5710,7 @@ lpfc_els_rsp_acc(struct lpfc_vport *vport, uint32_t flag, lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, "0128 Xmit ELS ACC response Status: x%x, IoTag: x%x, " "XRI: x%x, DID: x%x, nlp_flag: x%x nlp_state: x%x " - "RPI: x%x, fc_flag x%x refcnt %d\n", + "RPI: x%x, fc_flag x%lx refcnt %d\n", rc, elsiocb->iotag, elsiocb->sli4_xritag, ndlp->nlp_DID, ndlp->nlp_flag, ndlp->nlp_state, ndlp->nlp_rpi, vport->fc_flag, kref_read(&ndlp->kref)); @@ -5956,7 +5916,7 @@ lpfc_issue_els_edc_rsp(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, "0152 Xmit EDC ACC response Status: x%x, IoTag: x%x, " "XRI: x%x, DID: x%x, nlp_flag: x%x nlp_state: x%x " - "RPI: x%x, fc_flag x%x\n", + "RPI: x%x, fc_flag x%lx\n", rc, elsiocb->iotag, elsiocb->sli4_xritag, ndlp->nlp_DID, ndlp->nlp_flag, ndlp->nlp_state, ndlp->nlp_rpi, vport->fc_flag); @@ -6523,7 +6483,6 @@ lpfc_els_rsp_echo_acc(struct lpfc_vport *vport, uint8_t *data, int lpfc_els_disc_adisc(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_nodelist *ndlp, *next_ndlp; int sentadisc = 0; @@ -6558,18 +6517,13 @@ lpfc_els_disc_adisc(struct lpfc_vport *vport) vport->num_disc_nodes++; if (vport->num_disc_nodes >= vport->cfg_discovery_threads) { - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_NLP_MORE; - spin_unlock_irq(shost->host_lock); + set_bit(FC_NLP_MORE, &vport->fc_flag); break; } } - if (sentadisc == 0) { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_NLP_MORE; - spin_unlock_irq(shost->host_lock); - } + if (sentadisc == 0) + clear_bit(FC_NLP_MORE, &vport->fc_flag); return sentadisc; } @@ -6595,7 +6549,6 @@ lpfc_els_disc_adisc(struct lpfc_vport *vport) int lpfc_els_disc_plogi(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_nodelist *ndlp, *next_ndlp; int sentplogi = 0; @@ -6612,26 +6565,20 @@ lpfc_els_disc_plogi(struct lpfc_vport *vport) vport->num_disc_nodes++; if (vport->num_disc_nodes >= vport->cfg_discovery_threads) { - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_NLP_MORE; - spin_unlock_irq(shost->host_lock); + set_bit(FC_NLP_MORE, &vport->fc_flag); break; } } } lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, - "6452 Discover PLOGI %d flag x%x\n", + "6452 Discover PLOGI %d flag x%lx\n", sentplogi, vport->fc_flag); - if (sentplogi) { + if (sentplogi) lpfc_set_disctmo(vport); - } - else { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_NLP_MORE; - spin_unlock_irq(shost->host_lock); - } + else + clear_bit(FC_NLP_MORE, &vport->fc_flag); return sentplogi; } @@ -7042,7 +6989,7 @@ lpfc_rdp_res_attach_port_names(struct fc_rdp_port_name_desc *desc, { desc->tag = cpu_to_be32(RDP_PORT_NAMES_DESC_TAG); - if (vport->fc_flag & FC_FABRIC) { + if (test_bit(FC_FABRIC, &vport->fc_flag)) { memcpy(desc->port_names.wwnn, &vport->fabric_nodename, sizeof(desc->port_names.wwnn)); @@ -7300,7 +7247,7 @@ int lpfc_get_sfp_info_wait(struct lpfc_hba *phba, mbox->in_ext_byte_len = DMP_SFF_PAGE_A0_SIZE; mbox->out_ext_byte_len = DMP_SFF_PAGE_A0_SIZE; mbox->mbox_offset_word = 5; - mbox->ctx_buf = virt; + mbox->ext_buf = virt; } else { bf_set(lpfc_mbx_memory_dump_type3_length, &mbox->u.mqe.un.mem_dump_type3, DMP_SFF_PAGE_A0_SIZE); @@ -7309,13 +7256,13 @@ int lpfc_get_sfp_info_wait(struct lpfc_hba *phba, } mbox->vport = phba->pport; mbox->ctx_ndlp = (struct lpfc_rdp_context *)rdp_context; - - rc = lpfc_sli_issue_mbox_wait(phba, mbox, 30); + rc = lpfc_sli_issue_mbox_wait(phba, mbox, LPFC_MBOX_SLI4_CONFIG_TMO); if (rc == MBX_NOT_FINISHED) { rc = 1; goto error; } - + if (rc == MBX_TIMEOUT) + goto error; if (phba->sli_rev == LPFC_SLI_REV4) mp = (struct lpfc_dmabuf *)(mbox->ctx_buf); else @@ -7360,7 +7307,7 @@ int lpfc_get_sfp_info_wait(struct lpfc_hba *phba, mbox->in_ext_byte_len = DMP_SFF_PAGE_A2_SIZE; mbox->out_ext_byte_len = DMP_SFF_PAGE_A2_SIZE; mbox->mbox_offset_word = 5; - mbox->ctx_buf = virt; + mbox->ext_buf = virt; } else { bf_set(lpfc_mbx_memory_dump_type3_length, &mbox->u.mqe.un.mem_dump_type3, DMP_SFF_PAGE_A2_SIZE); @@ -7369,7 +7316,10 @@ int lpfc_get_sfp_info_wait(struct lpfc_hba *phba, } mbox->ctx_ndlp = (struct lpfc_rdp_context *)rdp_context; - rc = lpfc_sli_issue_mbox_wait(phba, mbox, 30); + rc = lpfc_sli_issue_mbox_wait(phba, mbox, LPFC_MBOX_SLI4_CONFIG_TMO); + + if (rc == MBX_TIMEOUT) + goto error; if (bf_get(lpfc_mqe_status, &mbox->u.mqe)) { rc = 1; goto error; @@ -7380,8 +7330,10 @@ int lpfc_get_sfp_info_wait(struct lpfc_hba *phba, DMP_SFF_PAGE_A2_SIZE); error: - mbox->ctx_buf = mpsave; - lpfc_mbox_rsrc_cleanup(phba, mbox, MBOX_THD_UNLOCKED); + if (mbox->mbox_flag & LPFC_MBX_WAKE) { + mbox->ctx_buf = mpsave; + lpfc_mbox_rsrc_cleanup(phba, mbox, MBOX_THD_UNLOCKED); + } return rc; @@ -7826,9 +7778,10 @@ lpfc_els_flush_rscn(struct lpfc_vport *vport) lpfc_in_buf_free(phba, vport->fc_rscn_id_list[i]); vport->fc_rscn_id_list[i] = NULL; } + clear_bit(FC_RSCN_MODE, &vport->fc_flag); + clear_bit(FC_RSCN_DISCOVERY, &vport->fc_flag); spin_lock_irq(shost->host_lock); vport->fc_rscn_id_cnt = 0; - vport->fc_flag &= ~(FC_RSCN_MODE | FC_RSCN_DISCOVERY); spin_unlock_irq(shost->host_lock); lpfc_can_disctmo(vport); /* Indicate we are done walking this fc_rscn_id_list */ @@ -7863,7 +7816,7 @@ lpfc_rscn_payload_check(struct lpfc_vport *vport, uint32_t did) return 0; /* If we are doing a FULL RSCN rediscovery, match everything */ - if (vport->fc_flag & FC_RSCN_DISCOVERY) + if (test_bit(FC_RSCN_DISCOVERY, &vport->fc_flag)) return did; spin_lock_irq(shost->host_lock); @@ -8042,7 +7995,7 @@ lpfc_els_rcv_rscn(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, payload_len -= sizeof(uint32_t); /* take off word 0 */ /* RSCN received */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, - "0214 RSCN received Data: x%x x%x x%x x%x\n", + "0214 RSCN received Data: x%lx x%x x%x x%x\n", vport->fc_flag, payload_len, *lp, vport->fc_rscn_id_cnt); @@ -8054,10 +8007,10 @@ lpfc_els_rcv_rscn(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, FCH_EVT_RSCN, lp[i]); /* Check if RSCN is coming from a direct-connected remote NPort */ - if (vport->fc_flag & FC_PT2PT) { + if (test_bit(FC_PT2PT, &vport->fc_flag)) { /* If so, just ACC it, no other action needed for now */ lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, - "2024 pt2pt RSCN %08x Data: x%x x%x\n", + "2024 pt2pt RSCN %08x Data: x%lx x%x\n", *lp, vport->fc_flag, payload_len); lpfc_els_rsp_acc(vport, ELS_CMD_ACC, cmdiocb, ndlp, NULL); @@ -8101,7 +8054,7 @@ lpfc_els_rcv_rscn(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, /* ALL NPortIDs in RSCN are on HBA */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0219 Ignore RSCN " - "Data: x%x x%x x%x x%x\n", + "Data: x%lx x%x x%x x%x\n", vport->fc_flag, payload_len, *lp, vport->fc_rscn_id_cnt); lpfc_debugfs_disc_trc(vport, LPFC_DISC_TRC_ELS_UNSOL, @@ -8112,7 +8065,7 @@ lpfc_els_rcv_rscn(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, lpfc_els_rsp_acc(vport, ELS_CMD_ACC, cmdiocb, ndlp, NULL); /* Restart disctmo if its already running */ - if (vport->fc_flag & FC_DISC_TMO) { + if (test_bit(FC_DISC_TMO, &vport->fc_flag)) { tmo = ((phba->fc_ratov * 3) + 3); mod_timer(&vport->fc_disctmo, jiffies + @@ -8125,8 +8078,8 @@ lpfc_els_rcv_rscn(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, spin_lock_irq(shost->host_lock); if (vport->fc_rscn_flush) { /* Another thread is walking fc_rscn_id_list on this vport */ - vport->fc_flag |= FC_RSCN_DISCOVERY; spin_unlock_irq(shost->host_lock); + set_bit(FC_RSCN_DISCOVERY, &vport->fc_flag); /* Send back ACC */ lpfc_els_rsp_acc(vport, ELS_CMD_ACC, cmdiocb, ndlp, NULL); return 0; @@ -8139,24 +8092,23 @@ lpfc_els_rcv_rscn(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, /* If we are already processing an RSCN, save the received * RSCN payload buffer, cmdiocb->cmd_dmabuf to process later. */ - if (vport->fc_flag & (FC_RSCN_MODE | FC_NDISC_ACTIVE)) { + if (test_bit(FC_RSCN_MODE, &vport->fc_flag) || + test_bit(FC_NDISC_ACTIVE, &vport->fc_flag)) { lpfc_debugfs_disc_trc(vport, LPFC_DISC_TRC_ELS_UNSOL, "RCV RSCN defer: did:x%x/ste:x%x flg:x%x", ndlp->nlp_DID, vport->port_state, ndlp->nlp_flag); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_RSCN_DEFERRED; + set_bit(FC_RSCN_DEFERRED, &vport->fc_flag); /* Restart disctmo if its already running */ - if (vport->fc_flag & FC_DISC_TMO) { + if (test_bit(FC_DISC_TMO, &vport->fc_flag)) { tmo = ((phba->fc_ratov * 3) + 3); mod_timer(&vport->fc_disctmo, jiffies + msecs_to_jiffies(1000 * tmo)); } if ((rscn_cnt < FC_MAX_HOLD_RSCN) && - !(vport->fc_flag & FC_RSCN_DISCOVERY)) { - vport->fc_flag |= FC_RSCN_MODE; - spin_unlock_irq(shost->host_lock); + !test_bit(FC_RSCN_DISCOVERY, &vport->fc_flag)) { + set_bit(FC_RSCN_MODE, &vport->fc_flag); if (rscn_cnt) { cmd = vport->fc_rscn_id_list[rscn_cnt-1]->virt; length = be32_to_cpu(*cmd & ~ELS_CMD_MASK); @@ -8178,16 +8130,15 @@ lpfc_els_rcv_rscn(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, /* Deferred RSCN */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0235 Deferred RSCN " - "Data: x%x x%x x%x\n", + "Data: x%x x%lx x%x\n", vport->fc_rscn_id_cnt, vport->fc_flag, vport->port_state); } else { - vport->fc_flag |= FC_RSCN_DISCOVERY; - spin_unlock_irq(shost->host_lock); + set_bit(FC_RSCN_DISCOVERY, &vport->fc_flag); /* ReDiscovery RSCN */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0234 ReDiscovery RSCN " - "Data: x%x x%x x%x\n", + "Data: x%x x%lx x%x\n", vport->fc_rscn_id_cnt, vport->fc_flag, vport->port_state); } @@ -8203,9 +8154,7 @@ lpfc_els_rcv_rscn(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, "RCV RSCN: did:x%x/ste:x%x flg:x%x", ndlp->nlp_DID, vport->port_state, ndlp->nlp_flag); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_RSCN_MODE; - spin_unlock_irq(shost->host_lock); + set_bit(FC_RSCN_MODE, &vport->fc_flag); vport->fc_rscn_id_list[vport->fc_rscn_id_cnt++] = pcmd; /* Indicate we are done walking fc_rscn_id_list on this vport */ vport->fc_rscn_flush = 0; @@ -8255,7 +8204,7 @@ lpfc_els_handle_rscn(struct lpfc_vport *vport) /* RSCN processed */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, - "0215 RSCN processed Data: x%x x%x x%x x%x x%x x%x\n", + "0215 RSCN processed Data: x%lx x%x x%x x%x x%x x%x\n", vport->fc_flag, 0, vport->fc_rscn_id_cnt, vport->port_state, vport->num_disc_nodes, vport->gidft_inp); @@ -8344,7 +8293,7 @@ lpfc_els_rcv_flogi(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, LPFC_MBOXQ_t *mbox; uint32_t cmd, did; int rc; - uint32_t fc_flag = 0; + unsigned long fc_flag = 0; uint32_t port_state = 0; /* Clear external loopback plug detected flag */ @@ -8414,9 +8363,7 @@ lpfc_els_rcv_flogi(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, return 0; } else if (rc > 0) { /* greater than */ - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_PT2PT_PLOGI; - spin_unlock_irq(shost->host_lock); + set_bit(FC_PT2PT_PLOGI, &vport->fc_flag); /* If we have the high WWPN we can assign our own * myDID; otherwise, we have to WAIT for a PLOGI @@ -8435,17 +8382,17 @@ lpfc_els_rcv_flogi(struct lpfc_vport *vport, struct lpfc_iocbq *cmdiocb, spin_lock_irq(shost->host_lock); fc_flag = vport->fc_flag; port_state = vport->port_state; - vport->fc_flag |= FC_PT2PT; - vport->fc_flag &= ~(FC_FABRIC | FC_PUBLIC_LOOP); - /* Acking an unsol FLOGI. Count 1 for link bounce * work-around. */ vport->rcv_flogi_cnt++; spin_unlock_irq(shost->host_lock); + set_bit(FC_PT2PT, &vport->fc_flag); + clear_bit(FC_FABRIC, &vport->fc_flag); + clear_bit(FC_PUBLIC_LOOP, &vport->fc_flag); lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, "3311 Rcv Flogi PS x%x new PS x%x " - "fc_flag x%x new fc_flag x%x\n", + "fc_flag x%lx new fc_flag x%lx\n", port_state, vport->port_state, fc_flag, vport->fc_flag); @@ -10381,8 +10328,8 @@ lpfc_els_unsol_buffer(struct lpfc_hba *phba, struct lpfc_sli_ring *pring, goto dropit; /* If NPort discovery is delayed drop incoming ELS */ - if ((vport->fc_flag & FC_DISC_DELAYED) && - (cmd != ELS_CMD_PLOGI)) + if (test_bit(FC_DISC_DELAYED, &vport->fc_flag) && + cmd != ELS_CMD_PLOGI) goto dropit; ndlp = lpfc_findnode_did(vport, did); @@ -10426,14 +10373,14 @@ lpfc_els_unsol_buffer(struct lpfc_hba *phba, struct lpfc_sli_ring *pring, /* ELS command received from NPORT */ lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, "0112 ELS command x%x received from NPORT x%x " - "refcnt %d Data: x%x x%x x%x x%x\n", + "refcnt %d Data: x%x x%lx x%x x%x\n", cmd, did, kref_read(&ndlp->kref), vport->port_state, vport->fc_flag, vport->fc_myDID, vport->fc_prevDID); /* reject till our FLOGI completes or PLOGI assigned DID via PT2PT */ if ((vport->port_state < LPFC_FABRIC_CFG_LINK) && (cmd != ELS_CMD_FLOGI) && - !((cmd == ELS_CMD_PLOGI) && (vport->fc_flag & FC_PT2PT))) { + !((cmd == ELS_CMD_PLOGI) && test_bit(FC_PT2PT, &vport->fc_flag))) { rjt_err = LSRJT_LOGICAL_BSY; rjt_exp = LSEXP_NOTHING_MORE; goto lsrjt; @@ -10448,7 +10395,7 @@ lpfc_els_unsol_buffer(struct lpfc_hba *phba, struct lpfc_sli_ring *pring, phba->fc_stat.elsRcvPLOGI++; ndlp = lpfc_plogi_confirm_nport(phba, payload, ndlp); if (phba->sli_rev == LPFC_SLI_REV4 && - (phba->pport->fc_flag & FC_PT2PT)) { + test_bit(FC_PT2PT, &phba->pport->fc_flag)) { vport->fc_prevDID = vport->fc_myDID; /* Our DID needs to be updated before registering * the vfi. This is done in lpfc_rcv_plogi but @@ -10466,15 +10413,15 @@ lpfc_els_unsol_buffer(struct lpfc_hba *phba, struct lpfc_sli_ring *pring, lpfc_send_els_event(vport, ndlp, payload); /* If Nport discovery is delayed, reject PLOGIs */ - if (vport->fc_flag & FC_DISC_DELAYED) { + if (test_bit(FC_DISC_DELAYED, &vport->fc_flag)) { rjt_err = LSRJT_UNABLE_TPC; rjt_exp = LSEXP_NOTHING_MORE; break; } if (vport->port_state < LPFC_DISC_AUTH) { - if (!(phba->pport->fc_flag & FC_PT2PT) || - (phba->pport->fc_flag & FC_PT2PT_PLOGI)) { + if (!test_bit(FC_PT2PT, &phba->pport->fc_flag) || + test_bit(FC_PT2PT_PLOGI, &phba->pport->fc_flag)) { rjt_err = LSRJT_UNABLE_TPC; rjt_exp = LSEXP_NOTHING_MORE; break; @@ -10500,7 +10447,7 @@ lpfc_els_unsol_buffer(struct lpfc_hba *phba, struct lpfc_sli_ring *pring, * bounce the link. There is some descrepancy. */ if (vport->port_state >= LPFC_LOCAL_CFG_LINK && - vport->fc_flag & FC_PT2PT && + test_bit(FC_PT2PT, &vport->fc_flag) && vport->rcv_flogi_cnt >= 1) { rjt_err = LSRJT_LOGICAL_BSY; rjt_exp = LSEXP_NOTHING_MORE; @@ -10623,7 +10570,7 @@ lpfc_els_unsol_buffer(struct lpfc_hba *phba, struct lpfc_sli_ring *pring, phba->fc_stat.elsRcvPRLI++; if ((vport->port_state < LPFC_DISC_AUTH) && - (vport->fc_flag & FC_FABRIC)) { + test_bit(FC_FABRIC, &vport->fc_flag)) { rjt_err = LSRJT_UNABLE_TPC; rjt_exp = LSEXP_NOTHING_MORE; break; @@ -10952,16 +10899,13 @@ void lpfc_do_scr_ns_plogi(struct lpfc_hba *phba, struct lpfc_vport *vport) { struct lpfc_nodelist *ndlp; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); /* * If lpfc_delay_discovery parameter is set and the clean address * bit is cleared and fc fabric parameters chenged, delay FC NPort * discovery. */ - spin_lock_irq(shost->host_lock); - if (vport->fc_flag & FC_DISC_DELAYED) { - spin_unlock_irq(shost->host_lock); + if (test_bit(FC_DISC_DELAYED, &vport->fc_flag)) { lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, "3334 Delay fc port discovery for %d secs\n", phba->fc_ratov); @@ -10969,7 +10913,6 @@ lpfc_do_scr_ns_plogi(struct lpfc_hba *phba, struct lpfc_vport *vport) jiffies + msecs_to_jiffies(1000 * phba->fc_ratov)); return; } - spin_unlock_irq(shost->host_lock); ndlp = lpfc_findnode_did(vport, NameServer_DID); if (!ndlp) { @@ -11019,14 +10962,12 @@ static void lpfc_cmpl_reg_new_vport(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) { struct lpfc_vport *vport = pmb->vport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); + struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_nodelist *ndlp = pmb->ctx_ndlp; MAILBOX_t *mb = &pmb->u.mb; int rc; - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_VPORT_NEEDS_REG_VPI; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); if (mb->mbxStatus) { lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, @@ -11043,16 +10984,13 @@ lpfc_cmpl_reg_new_vport(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) case 0x9602: /* Link event since CLEAR_LA */ /* giving up on vport registration */ lpfc_vport_set_state(vport, FC_VPORT_FAILED); - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~(FC_FABRIC | FC_PUBLIC_LOOP); - spin_unlock_irq(shost->host_lock); + clear_bit(FC_FABRIC, &vport->fc_flag); + clear_bit(FC_PUBLIC_LOOP, &vport->fc_flag); lpfc_can_disctmo(vport); break; /* If reg_vpi fail with invalid VPI status, re-init VPI */ case 0x20: - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_VPORT_NEEDS_REG_VPI; - spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); lpfc_init_vpi(phba, pmb, vport->vpi); pmb->vport = vport; pmb->mbox_cmpl = lpfc_init_vpi_cmpl; @@ -11073,13 +11011,11 @@ lpfc_cmpl_reg_new_vport(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) if (phba->sli_rev == LPFC_SLI_REV4) lpfc_sli4_unreg_all_rpis(vport); lpfc_mbx_unreg_vpi(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_VPORT_NEEDS_REG_VPI; - spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); if (mb->mbxStatus == MBX_NOT_FINISHED) break; if ((vport->port_type == LPFC_PHYSICAL_PORT) && - !(vport->fc_flag & FC_LOGO_RCVD_DID_CHNG)) { + !test_bit(FC_LOGO_RCVD_DID_CHNG, &vport->fc_flag)) { if (phba->sli_rev == LPFC_SLI_REV4) lpfc_issue_init_vfi(vport); else @@ -11140,7 +11076,6 @@ void lpfc_register_new_vport(struct lpfc_hba *phba, struct lpfc_vport *vport, struct lpfc_nodelist *ndlp) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); LPFC_MBOXQ_t *mbox; mbox = mempool_alloc(phba->mbox_mem_pool, GFP_KERNEL); @@ -11175,9 +11110,7 @@ lpfc_register_new_vport(struct lpfc_hba *phba, struct lpfc_vport *vport, mbox_err_exit: lpfc_vport_set_state(vport, FC_VPORT_FAILED); - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_VPORT_NEEDS_REG_VPI; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); return; } @@ -11292,7 +11225,6 @@ lpfc_cmpl_els_fdisc(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, struct lpfc_iocbq *rspiocb) { struct lpfc_vport *vport = cmdiocb->vport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_nodelist *ndlp = cmdiocb->ndlp; struct lpfc_nodelist *np; struct lpfc_nodelist *next_np; @@ -11340,13 +11272,11 @@ lpfc_cmpl_els_fdisc(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, lpfc_check_nlp_post_devloss(vport, ndlp); - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_VPORT_CVL_RCVD; - vport->fc_flag &= ~FC_VPORT_LOGO_RCVD; - vport->fc_flag |= FC_FABRIC; + clear_bit(FC_VPORT_CVL_RCVD, &vport->fc_flag); + clear_bit(FC_VPORT_LOGO_RCVD, &vport->fc_flag); + set_bit(FC_FABRIC, &vport->fc_flag); if (vport->phba->fc_topology == LPFC_TOPOLOGY_LOOP) - vport->fc_flag |= FC_PUBLIC_LOOP; - spin_unlock_irq(shost->host_lock); + set_bit(FC_PUBLIC_LOOP, &vport->fc_flag); vport->fc_myDID = ulp_word4 & Mask_DID; lpfc_vport_set_state(vport, FC_VPORT_ACTIVE); @@ -11360,7 +11290,7 @@ lpfc_cmpl_els_fdisc(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, memcpy(&vport->fabric_nodename, &sp->nodeName, sizeof(struct lpfc_name)); if (fabric_param_changed && - !(vport->fc_flag & FC_VPORT_NEEDS_REG_VPI)) { + !test_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag)) { /* If our NportID changed, we need to ensure all * remaining NPORTs get unreg_login'ed so we can * issue unreg_vpi. @@ -11381,15 +11311,13 @@ lpfc_cmpl_els_fdisc(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, lpfc_sli4_unreg_all_rpis(vport); lpfc_mbx_unreg_vpi(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_VPORT_NEEDS_REG_VPI; + set_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); if (phba->sli_rev == LPFC_SLI_REV4) - vport->fc_flag |= FC_VPORT_NEEDS_INIT_VPI; + set_bit(FC_VPORT_NEEDS_INIT_VPI, &vport->fc_flag); else - vport->fc_flag |= FC_LOGO_RCVD_DID_CHNG; - spin_unlock_irq(shost->host_lock); + set_bit(FC_LOGO_RCVD_DID_CHNG, &vport->fc_flag); } else if ((phba->sli_rev == LPFC_SLI_REV4) && - !(vport->fc_flag & FC_VPORT_NEEDS_REG_VPI)) { + !test_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag)) { /* * Driver needs to re-reg VPI in order for f/w * to update the MAC address. @@ -11399,9 +11327,9 @@ lpfc_cmpl_els_fdisc(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, goto out; } - if (vport->fc_flag & FC_VPORT_NEEDS_INIT_VPI) + if (test_bit(FC_VPORT_NEEDS_INIT_VPI, &vport->fc_flag)) lpfc_issue_init_vpi(vport); - else if (vport->fc_flag & FC_VPORT_NEEDS_REG_VPI) + else if (test_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag)) lpfc_register_new_vport(phba, vport, ndlp); else lpfc_do_scr_ns_plogi(phba, vport); @@ -11554,7 +11482,6 @@ lpfc_cmpl_els_npiv_logo(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, struct lpfc_vport *vport = cmdiocb->vport; IOCB_t *irsp; struct lpfc_nodelist *ndlp; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); u32 ulp_status, ulp_word4, did, tmo; ndlp = cmdiocb->ndlp; @@ -11585,10 +11512,8 @@ lpfc_cmpl_els_npiv_logo(struct lpfc_hba *phba, struct lpfc_iocbq *cmdiocb, ndlp->fc4_xpt_flags); if (ulp_status == IOSTAT_SUCCESS) { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_NDISC_ACTIVE; - vport->fc_flag &= ~FC_FABRIC; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_NDISC_ACTIVE, &vport->fc_flag); + clear_bit(FC_FABRIC, &vport->fc_flag); lpfc_can_disctmo(vport); } diff --git a/drivers/scsi/lpfc/lpfc_hbadisc.c b/drivers/scsi/lpfc/lpfc_hbadisc.c index ff4428333877..4f531164ad32 100644 --- a/drivers/scsi/lpfc/lpfc_hbadisc.c +++ b/drivers/scsi/lpfc/lpfc_hbadisc.c @@ -1149,7 +1149,6 @@ lpfc_workq_post_event(struct lpfc_hba *phba, void *arg1, void *arg2, void lpfc_cleanup_rpis(struct lpfc_vport *vport, int remove) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; struct lpfc_nodelist *ndlp, *next_ndlp; @@ -1180,9 +1179,7 @@ lpfc_cleanup_rpis(struct lpfc_vport *vport, int remove) if (phba->sli_rev == LPFC_SLI_REV4) lpfc_sli4_unreg_all_rpis(vport); lpfc_mbx_unreg_vpi(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_VPORT_NEEDS_REG_VPI; - spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); } } @@ -1210,7 +1207,7 @@ void lpfc_linkdown_port(struct lpfc_vport *vport) { struct lpfc_hba *phba = vport->phba; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); + struct Scsi_Host *shost = lpfc_shost_from_vport(vport); if (vport->cfg_enable_fc4_type != LPFC_ENABLE_NVME) fc_host_post_event(shost, fc_get_event_number(), @@ -1223,9 +1220,7 @@ lpfc_linkdown_port(struct lpfc_vport *vport) lpfc_port_link_failure(vport); /* Stop delayed Nport discovery */ - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_DISC_DELAYED; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_DISC_DELAYED, &vport->fc_flag); del_timer_sync(&vport->delayed_disc_tmo); if (phba->sli_rev == LPFC_SLI_REV4 && @@ -1240,7 +1235,7 @@ int lpfc_linkdown(struct lpfc_hba *phba) { struct lpfc_vport *vport = phba->pport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); + struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_vport **vports; LPFC_MBOXQ_t *mb; int i; @@ -1273,9 +1268,7 @@ lpfc_linkdown(struct lpfc_hba *phba) phba->sli4_hba.link_state.logical_speed = LPFC_LINK_SPEED_UNKNOWN; } - spin_lock_irq(shost->host_lock); - phba->pport->fc_flag &= ~FC_LBIT; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_LBIT, &phba->pport->fc_flag); } vports = lpfc_create_vport_work_array(phba); if (vports != NULL) { @@ -1313,7 +1306,7 @@ lpfc_linkdown(struct lpfc_hba *phba) skip_unreg_did: /* Setup myDID for link up if we are in pt2pt mode */ - if (phba->pport->fc_flag & FC_PT2PT) { + if (test_bit(FC_PT2PT, &phba->pport->fc_flag)) { mb = mempool_alloc(phba->mbox_mem_pool, GFP_KERNEL); if (mb) { lpfc_config_link(phba, mb); @@ -1324,8 +1317,9 @@ lpfc_linkdown(struct lpfc_hba *phba) mempool_free(mb, phba->mbox_mem_pool); } } + clear_bit(FC_PT2PT, &phba->pport->fc_flag); + clear_bit(FC_PT2PT_PLOGI, &phba->pport->fc_flag); spin_lock_irq(shost->host_lock); - phba->pport->fc_flag &= ~(FC_PT2PT | FC_PT2PT_PLOGI); phba->pport->rcv_flogi_cnt = 0; spin_unlock_irq(shost->host_lock); } @@ -1376,19 +1370,22 @@ lpfc_linkup_port(struct lpfc_vport *vport) (vport != phba->pport)) return; - if (vport->cfg_enable_fc4_type != LPFC_ENABLE_NVME) - fc_host_post_event(shost, fc_get_event_number(), - FCH_EVT_LINKUP, 0); + if (phba->defer_flogi_acc_flag) { + clear_bit(FC_ABORT_DISCOVERY, &vport->fc_flag); + clear_bit(FC_RSCN_MODE, &vport->fc_flag); + clear_bit(FC_NLP_MORE, &vport->fc_flag); + clear_bit(FC_RSCN_DISCOVERY, &vport->fc_flag); + } else { + clear_bit(FC_PT2PT, &vport->fc_flag); + clear_bit(FC_PT2PT_PLOGI, &vport->fc_flag); + clear_bit(FC_ABORT_DISCOVERY, &vport->fc_flag); + clear_bit(FC_RSCN_MODE, &vport->fc_flag); + clear_bit(FC_NLP_MORE, &vport->fc_flag); + clear_bit(FC_RSCN_DISCOVERY, &vport->fc_flag); + } + set_bit(FC_NDISC_ACTIVE, &vport->fc_flag); spin_lock_irq(shost->host_lock); - if (phba->defer_flogi_acc_flag) - vport->fc_flag &= ~(FC_ABORT_DISCOVERY | FC_RSCN_MODE | - FC_NLP_MORE | FC_RSCN_DISCOVERY); - else - vport->fc_flag &= ~(FC_PT2PT | FC_PT2PT_PLOGI | - FC_ABORT_DISCOVERY | FC_RSCN_MODE | - FC_NLP_MORE | FC_RSCN_DISCOVERY); - vport->fc_flag |= FC_NDISC_ACTIVE; vport->fc_ns_retry = 0; spin_unlock_irq(shost->host_lock); lpfc_setup_fdmi_mask(vport); @@ -1439,7 +1436,6 @@ static void lpfc_mbx_cmpl_clear_la(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) { struct lpfc_vport *vport = pmb->vport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_sli *psli = &phba->sli; MAILBOX_t *mb = &pmb->u.mb; uint32_t control; @@ -1478,9 +1474,7 @@ lpfc_mbx_cmpl_clear_la(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) "0225 Device Discovery completes\n"); mempool_free(pmb, phba->mbox_mem_pool); - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_ABORT_DISCOVERY; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_ABORT_DISCOVERY, &vport->fc_flag); lpfc_can_disctmo(vport); @@ -1517,8 +1511,8 @@ lpfc_mbx_cmpl_local_config_link(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) return; if (phba->fc_topology == LPFC_TOPOLOGY_LOOP && - vport->fc_flag & FC_PUBLIC_LOOP && - !(vport->fc_flag & FC_LBIT)) { + test_bit(FC_PUBLIC_LOOP, &vport->fc_flag) && + !test_bit(FC_LBIT, &vport->fc_flag)) { /* Need to wait for FAN - use discovery timer * for timeout. port_state is identically * LPFC_LOCAL_CFG_LINK while waiting for FAN @@ -1560,7 +1554,7 @@ lpfc_mbx_cmpl_local_config_link(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) lpfc_initial_flogi(vport); } } else { - if (vport->fc_flag & FC_PT2PT) + if (test_bit(FC_PT2PT, &vport->fc_flag)) lpfc_disc_start(vport); } return; @@ -1884,7 +1878,7 @@ lpfc_register_fcf(struct lpfc_hba *phba) phba->fcf.fcf_flag |= (FCF_SCAN_DONE | FCF_IN_USE); phba->hba_flag &= ~FCF_TS_INPROG; if (phba->pport->port_state != LPFC_FLOGI && - phba->pport->fc_flag & FC_FABRIC) { + test_bit(FC_FABRIC, &phba->pport->fc_flag)) { phba->hba_flag |= FCF_RR_INPROG; spin_unlock_irq(&phba->hbalock); lpfc_initial_flogi(phba->pport); @@ -2742,7 +2736,7 @@ lpfc_mbx_cmpl_fcf_scan_read_fcf_rec(struct lpfc_hba *phba, LPFC_MBOXQ_t *mboxq) lpfc_printf_log(phba, KERN_INFO, LOG_FIP, "2836 New FCF matches in-use " "FCF (x%x), port_state:x%x, " - "fc_flag:x%x\n", + "fc_flag:x%lx\n", phba->fcf.current_rec.fcf_indx, phba->pport->port_state, phba->pport->fc_flag); @@ -3218,7 +3212,6 @@ lpfc_init_vpi_cmpl(struct lpfc_hba *phba, LPFC_MBOXQ_t *mboxq) { struct lpfc_vport *vport = mboxq->vport; struct lpfc_nodelist *ndlp; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); if (mboxq->u.mb.mbxStatus) { lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, @@ -3228,9 +3221,7 @@ lpfc_init_vpi_cmpl(struct lpfc_hba *phba, LPFC_MBOXQ_t *mboxq) lpfc_vport_set_state(vport, FC_VPORT_FAILED); return; } - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_VPORT_NEEDS_INIT_VPI; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_VPORT_NEEDS_INIT_VPI, &vport->fc_flag); /* If this port is physical port or FDISC is done, do reg_vpi */ if ((phba->pport == vport) || (vport->port_state == LPFC_FDISC)) { @@ -3328,7 +3319,8 @@ lpfc_start_fdiscs(struct lpfc_hba *phba) FC_VPORT_LINKDOWN); continue; } - if (vports[i]->fc_flag & FC_VPORT_NEEDS_INIT_VPI) { + if (test_bit(FC_VPORT_NEEDS_INIT_VPI, + &vports[i]->fc_flag)) { lpfc_issue_init_vpi(vports[i]); continue; } @@ -3380,17 +3372,17 @@ lpfc_mbx_cmpl_reg_vfi(struct lpfc_hba *phba, LPFC_MBOXQ_t *mboxq) * Unless this was a VFI update and we are in PT2PT mode, then * we should drop through to set the port state to ready. */ - if (vport->fc_flag & FC_VFI_REGISTERED) + if (test_bit(FC_VFI_REGISTERED, &vport->fc_flag)) if (!(phba->sli_rev == LPFC_SLI_REV4 && - vport->fc_flag & FC_PT2PT)) + test_bit(FC_PT2PT, &vport->fc_flag))) goto out_free_mem; /* The VPI is implicitly registered when the VFI is registered */ + set_bit(FC_VFI_REGISTERED, &vport->fc_flag); + clear_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); + clear_bit(FC_VPORT_NEEDS_INIT_VPI, &vport->fc_flag); spin_lock_irq(shost->host_lock); vport->vpi_state |= LPFC_VPI_REGISTERED; - vport->fc_flag |= FC_VFI_REGISTERED; - vport->fc_flag &= ~FC_VPORT_NEEDS_REG_VPI; - vport->fc_flag &= ~FC_VPORT_NEEDS_INIT_VPI; spin_unlock_irq(shost->host_lock); /* In case SLI4 FC loopback test, we are ready */ @@ -3401,8 +3393,8 @@ lpfc_mbx_cmpl_reg_vfi(struct lpfc_hba *phba, LPFC_MBOXQ_t *mboxq) } lpfc_printf_vlog(vport, KERN_INFO, LOG_SLI, - "3313 cmpl reg vfi port_state:%x fc_flag:%x myDid:%x " - "alpacnt:%d LinkState:%x topology:%x\n", + "3313 cmpl reg vfi port_state:%x fc_flag:%lx " + "myDid:%x alpacnt:%d LinkState:%x topology:%x\n", vport->port_state, vport->fc_flag, vport->fc_myDID, vport->phba->alpa_map[0], phba->link_state, phba->fc_topology); @@ -3412,14 +3404,14 @@ lpfc_mbx_cmpl_reg_vfi(struct lpfc_hba *phba, LPFC_MBOXQ_t *mboxq) * For private loop or for NPort pt2pt, * just start discovery and we are done. */ - if ((vport->fc_flag & FC_PT2PT) || - ((phba->fc_topology == LPFC_TOPOLOGY_LOOP) && - !(vport->fc_flag & FC_PUBLIC_LOOP))) { + if (test_bit(FC_PT2PT, &vport->fc_flag) || + (phba->fc_topology == LPFC_TOPOLOGY_LOOP && + !test_bit(FC_PUBLIC_LOOP, &vport->fc_flag))) { /* Use loop map to make discovery list */ lpfc_disc_list_loopmap(vport); /* Start discovery */ - if (vport->fc_flag & FC_PT2PT) + if (test_bit(FC_PT2PT, &vport->fc_flag)) vport->port_state = LPFC_VPORT_READY; else lpfc_disc_start(vport); @@ -3496,11 +3488,9 @@ lpfc_mbx_process_link_up(struct lpfc_hba *phba, struct lpfc_mbx_read_top *la) { struct lpfc_vport *vport = phba->pport; LPFC_MBOXQ_t *sparam_mbox, *cfglink_mbox = NULL; - struct Scsi_Host *shost; int i; int rc; struct fcf_record *fcf_record; - uint32_t fc_flags = 0; unsigned long iflags; spin_lock_irqsave(&phba->hbalock, iflags); @@ -3537,7 +3527,6 @@ lpfc_mbx_process_link_up(struct lpfc_hba *phba, struct lpfc_mbx_read_top *la) phba->fc_topology = bf_get(lpfc_mbx_read_top_topology, la); phba->link_flag &= ~(LS_NPIV_FAB_SUPPORTED | LS_CT_VEN_RPA); - shost = lpfc_shost_from_vport(vport); if (phba->fc_topology == LPFC_TOPOLOGY_LOOP) { phba->sli3_options &= ~LPFC_SLI3_NPIV_ENABLED; @@ -3550,7 +3539,7 @@ lpfc_mbx_process_link_up(struct lpfc_hba *phba, struct lpfc_mbx_read_top *la) "topology\n"); /* Get Loop Map information */ if (bf_get(lpfc_mbx_read_top_il, la)) - fc_flags |= FC_LBIT; + set_bit(FC_LBIT, &vport->fc_flag); vport->fc_myDID = bf_get(lpfc_mbx_read_top_alpa_granted, la); i = la->lilpBde64.tus.f.bdeSize; @@ -3599,16 +3588,10 @@ lpfc_mbx_process_link_up(struct lpfc_hba *phba, struct lpfc_mbx_read_top *la) phba->sli3_options |= LPFC_SLI3_NPIV_ENABLED; } vport->fc_myDID = phba->fc_pref_DID; - fc_flags |= FC_LBIT; + set_bit(FC_LBIT, &vport->fc_flag); } spin_unlock_irqrestore(&phba->hbalock, iflags); - if (fc_flags) { - spin_lock_irqsave(shost->host_lock, iflags); - vport->fc_flag |= fc_flags; - spin_unlock_irqrestore(shost->host_lock, iflags); - } - lpfc_linkup(phba); sparam_mbox = NULL; @@ -3751,13 +3734,11 @@ void lpfc_mbx_cmpl_read_topology(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) { struct lpfc_vport *vport = pmb->vport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_mbx_read_top *la; struct lpfc_sli_ring *pring; MAILBOX_t *mb = &pmb->u.mb; struct lpfc_dmabuf *mp = (struct lpfc_dmabuf *)(pmb->ctx_buf); uint8_t attn_type; - unsigned long iflags; /* Unblock ELS traffic */ pring = lpfc_phba_elsring(phba); @@ -3779,12 +3760,10 @@ lpfc_mbx_cmpl_read_topology(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) memcpy(&phba->alpa_map[0], mp->virt, 128); - spin_lock_irqsave(shost->host_lock, iflags); if (bf_get(lpfc_mbx_read_top_pb, la)) - vport->fc_flag |= FC_BYPASSED_MODE; + set_bit(FC_BYPASSED_MODE, &vport->fc_flag); else - vport->fc_flag &= ~FC_BYPASSED_MODE; - spin_unlock_irqrestore(shost->host_lock, iflags); + clear_bit(FC_BYPASSED_MODE, &vport->fc_flag); if (phba->fc_eventTag <= la->eventTag) { phba->fc_stat.LinkMultiEvent++; @@ -3832,20 +3811,20 @@ lpfc_mbx_cmpl_read_topology(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) lpfc_printf_log(phba, KERN_ERR, LOG_LINK_EVENT, "1308 Link Down Event in loop back mode " "x%x received " - "Data: x%x x%x x%x\n", + "Data: x%x x%x x%lx\n", la->eventTag, phba->fc_eventTag, phba->pport->port_state, vport->fc_flag); else if (attn_type == LPFC_ATT_UNEXP_WWPN) lpfc_printf_log(phba, KERN_ERR, LOG_LINK_EVENT, "1313 Link Down Unexpected FA WWPN Event x%x " - "received Data: x%x x%x x%x x%x\n", + "received Data: x%x x%x x%lx x%x\n", la->eventTag, phba->fc_eventTag, phba->pport->port_state, vport->fc_flag, bf_get(lpfc_mbx_read_top_fa, la)); else lpfc_printf_log(phba, KERN_ERR, LOG_LINK_EVENT, "1305 Link Down Event x%x received " - "Data: x%x x%x x%x x%x\n", + "Data: x%x x%x x%lx x%x\n", la->eventTag, phba->fc_eventTag, phba->pport->port_state, vport->fc_flag, bf_get(lpfc_mbx_read_top_fa, la)); @@ -3949,9 +3928,10 @@ lpfc_mbx_cmpl_unreg_vpi(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) lpfc_workq_post_event(phba, NULL, NULL, LPFC_EVT_RESET_HBA); } + + set_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); spin_lock_irq(shost->host_lock); vport->vpi_state &= ~LPFC_VPI_REGISTERED; - vport->fc_flag |= FC_VPORT_NEEDS_REG_VPI; spin_unlock_irq(shost->host_lock); mempool_free(pmb, phba->mbox_mem_pool); lpfc_cleanup_vports_rrqs(vport, NULL); @@ -4002,9 +3982,8 @@ lpfc_mbx_cmpl_reg_vpi(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) "0912 cmpl_reg_vpi, mb status = 0x%x\n", mb->mbxStatus); lpfc_vport_set_state(vport, FC_VPORT_FAILED); - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~(FC_FABRIC | FC_PUBLIC_LOOP); - spin_unlock_irq(shost->host_lock); + clear_bit(FC_FABRIC, &vport->fc_flag); + clear_bit(FC_PUBLIC_LOOP, &vport->fc_flag); vport->fc_myDID = 0; if ((vport->cfg_enable_fc4_type == LPFC_ENABLE_BOTH) || @@ -4017,19 +3996,17 @@ lpfc_mbx_cmpl_reg_vpi(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) goto out; } + clear_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); spin_lock_irq(shost->host_lock); vport->vpi_state |= LPFC_VPI_REGISTERED; - vport->fc_flag &= ~FC_VPORT_NEEDS_REG_VPI; spin_unlock_irq(shost->host_lock); vport->num_disc_nodes = 0; /* go thru NPR list and issue ELS PLOGIs */ - if (vport->fc_npr_cnt) + if (atomic_read(&vport->fc_npr_cnt)) lpfc_els_disc_plogi(vport); if (!vport->num_disc_nodes) { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_NDISC_ACTIVE; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_NDISC_ACTIVE, &vport->fc_flag); lpfc_can_disctmo(vport); } vport->port_state = LPFC_VPORT_READY; @@ -4193,7 +4170,6 @@ lpfc_mbx_cmpl_fabric_reg_login(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) struct lpfc_vport *vport = pmb->vport; MAILBOX_t *mb = &pmb->u.mb; struct lpfc_nodelist *ndlp = (struct lpfc_nodelist *)pmb->ctx_ndlp; - struct Scsi_Host *shost; pmb->ctx_ndlp = NULL; @@ -4232,14 +4208,8 @@ lpfc_mbx_cmpl_fabric_reg_login(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) if (vport->port_state == LPFC_FABRIC_CFG_LINK) { /* when physical port receive logo donot start * vport discovery */ - if (!(vport->fc_flag & FC_LOGO_RCVD_DID_CHNG)) + if (!test_and_clear_bit(FC_LOGO_RCVD_DID_CHNG, &vport->fc_flag)) lpfc_start_fdiscs(phba); - else { - shost = lpfc_shost_from_vport(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_LOGO_RCVD_DID_CHNG ; - spin_unlock_irq(shost->host_lock); - } lpfc_do_scr_ns_plogi(phba, vport); } @@ -4600,40 +4570,35 @@ lpfc_unregister_remote_port(struct lpfc_nodelist *ndlp) static void lpfc_nlp_counters(struct lpfc_vport *vport, int state, int count) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); - unsigned long iflags; - - spin_lock_irqsave(shost->host_lock, iflags); switch (state) { case NLP_STE_UNUSED_NODE: - vport->fc_unused_cnt += count; + atomic_add(count, &vport->fc_unused_cnt); break; case NLP_STE_PLOGI_ISSUE: - vport->fc_plogi_cnt += count; + atomic_add(count, &vport->fc_plogi_cnt); break; case NLP_STE_ADISC_ISSUE: - vport->fc_adisc_cnt += count; + atomic_add(count, &vport->fc_adisc_cnt); break; case NLP_STE_REG_LOGIN_ISSUE: - vport->fc_reglogin_cnt += count; + atomic_add(count, &vport->fc_reglogin_cnt); break; case NLP_STE_PRLI_ISSUE: - vport->fc_prli_cnt += count; + atomic_add(count, &vport->fc_prli_cnt); break; case NLP_STE_UNMAPPED_NODE: - vport->fc_unmap_cnt += count; + atomic_add(count, &vport->fc_unmap_cnt); break; case NLP_STE_MAPPED_NODE: - vport->fc_map_cnt += count; + atomic_add(count, &vport->fc_map_cnt); break; case NLP_STE_NPR_NODE: - if (vport->fc_npr_cnt == 0 && count == -1) - vport->fc_npr_cnt = 0; + if (!atomic_read(&vport->fc_npr_cnt) && count == -1) + atomic_set(&vport->fc_npr_cnt, 0); else - vport->fc_npr_cnt += count; + atomic_add(count, &vport->fc_npr_cnt); break; } - spin_unlock_irqrestore(shost->host_lock, iflags); } /* Register a node with backend if not already done */ @@ -4865,10 +4830,10 @@ void lpfc_nlp_set_state(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, int state) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); int old_state = ndlp->nlp_state; int node_dropped = ndlp->nlp_flag & NLP_DROPPED; char name1[16], name2[16]; + unsigned long iflags; lpfc_printf_vlog(vport, KERN_INFO, LOG_NODE, "0904 NPort state transition x%06x, %s -> %s\n", @@ -4895,9 +4860,9 @@ lpfc_nlp_set_state(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, } if (list_empty(&ndlp->nlp_listp)) { - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_add_tail(&ndlp->nlp_listp, &vport->fc_nodes); - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); } else if (old_state) lpfc_nlp_counters(vport, old_state, -1); @@ -4909,26 +4874,26 @@ lpfc_nlp_set_state(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, void lpfc_enqueue_node(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); + unsigned long iflags; if (list_empty(&ndlp->nlp_listp)) { - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_add_tail(&ndlp->nlp_listp, &vport->fc_nodes); - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); } } void lpfc_dequeue_node(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); + unsigned long iflags; lpfc_cancel_retry_delay_tmo(vport, ndlp); if (ndlp->nlp_state && !list_empty(&ndlp->nlp_listp)) lpfc_nlp_counters(vport, ndlp->nlp_state, -1); - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_del_init(&ndlp->nlp_listp); - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); lpfc_nlp_state_cleanup(vport, ndlp, ndlp->nlp_state, NLP_STE_UNUSED_NODE); } @@ -5003,7 +4968,6 @@ lpfc_drop_node(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp) void lpfc_set_disctmo(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; uint32_t tmo; @@ -5025,17 +4989,16 @@ lpfc_set_disctmo(struct lpfc_vport *vport) } mod_timer(&vport->fc_disctmo, jiffies + msecs_to_jiffies(1000 * tmo)); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_DISC_TMO; - spin_unlock_irq(shost->host_lock); + set_bit(FC_DISC_TMO, &vport->fc_flag); /* Start Discovery Timer state */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0247 Start Discovery Timer state x%x " "Data: x%x x%lx x%x x%x\n", vport->port_state, tmo, - (unsigned long)&vport->fc_disctmo, vport->fc_plogi_cnt, - vport->fc_adisc_cnt); + (unsigned long)&vport->fc_disctmo, + atomic_read(&vport->fc_plogi_cnt), + atomic_read(&vport->fc_adisc_cnt)); return; } @@ -5046,7 +5009,6 @@ lpfc_set_disctmo(struct lpfc_vport *vport) int lpfc_can_disctmo(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); unsigned long iflags; lpfc_debugfs_disc_trc(vport, LPFC_DISC_TRC_ELS_CMD, @@ -5054,11 +5016,9 @@ lpfc_can_disctmo(struct lpfc_vport *vport) vport->port_state, vport->fc_ns_retry, vport->fc_flag); /* Turn off discovery timer if its running */ - if (vport->fc_flag & FC_DISC_TMO || + if (test_bit(FC_DISC_TMO, &vport->fc_flag) || timer_pending(&vport->fc_disctmo)) { - spin_lock_irqsave(shost->host_lock, iflags); - vport->fc_flag &= ~FC_DISC_TMO; - spin_unlock_irqrestore(shost->host_lock, iflags); + clear_bit(FC_DISC_TMO, &vport->fc_flag); del_timer_sync(&vport->fc_disctmo); spin_lock_irqsave(&vport->work_port_lock, iflags); vport->work_port_events &= ~WORKER_DISC_TMO; @@ -5068,9 +5028,10 @@ lpfc_can_disctmo(struct lpfc_vport *vport) /* Cancel Discovery Timer state */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0248 Cancel Discovery Timer state x%x " - "Data: x%x x%x x%x\n", + "Data: x%lx x%x x%x\n", vport->port_state, vport->fc_flag, - vport->fc_plogi_cnt, vport->fc_adisc_cnt); + atomic_read(&vport->fc_plogi_cnt), + atomic_read(&vport->fc_adisc_cnt)); return 0; } @@ -5356,7 +5317,7 @@ lpfc_unreg_rpi(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp) acc_plogi = 0; if (((ndlp->nlp_DID & Fabric_DID_MASK) != Fabric_DID_MASK) && - (!(vport->fc_flag & FC_OFFLINE_MODE))) + (!test_bit(FC_OFFLINE_MODE, &vport->fc_flag))) ndlp->nlp_flag |= NLP_UNREG_INP; lpfc_printf_vlog(vport, KERN_INFO, @@ -5424,8 +5385,8 @@ lpfc_unreg_hba_rpis(struct lpfc_hba *phba) { struct lpfc_vport **vports; struct lpfc_nodelist *ndlp; - struct Scsi_Host *shost; int i; + unsigned long iflags; vports = lpfc_create_vport_work_array(phba); if (!vports) { @@ -5434,17 +5395,18 @@ lpfc_unreg_hba_rpis(struct lpfc_hba *phba) return; } for (i = 0; i <= phba->max_vports && vports[i] != NULL; i++) { - shost = lpfc_shost_from_vport(vports[i]); - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vports[i]->fc_nodes_list_lock, iflags); list_for_each_entry(ndlp, &vports[i]->fc_nodes, nlp_listp) { if (ndlp->nlp_flag & NLP_RPI_REGISTERED) { /* The mempool_alloc might sleep */ - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vports[i]->fc_nodes_list_lock, + iflags); lpfc_unreg_rpi(vports[i], ndlp); - spin_lock_irq(shost->host_lock); + spin_lock_irqsave(&vports[i]->fc_nodes_list_lock, + iflags); } } - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vports[i]->fc_nodes_list_lock, iflags); } lpfc_destroy_vport_work_array(phba, vports); } @@ -5693,12 +5655,11 @@ lpfc_findnode_did(struct lpfc_vport *vport, uint32_t did) struct lpfc_nodelist * lpfc_findnode_mapped(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_nodelist *ndlp; uint32_t data1; unsigned long iflags; - spin_lock_irqsave(shost->host_lock, iflags); + spin_lock_irqsave(&vport->fc_nodes_list_lock, iflags); list_for_each_entry(ndlp, &vport->fc_nodes, nlp_listp) { if (ndlp->nlp_state == NLP_STE_UNMAPPED_NODE || @@ -5707,7 +5668,8 @@ lpfc_findnode_mapped(struct lpfc_vport *vport) ((uint32_t)ndlp->nlp_xri << 16) | ((uint32_t)ndlp->nlp_type << 8) | ((uint32_t)ndlp->nlp_rpi & 0xff)); - spin_unlock_irqrestore(shost->host_lock, iflags); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, + iflags); lpfc_printf_vlog(vport, KERN_INFO, LOG_NODE, "2025 FIND node DID " "Data: x%px x%x x%x x%x x%px\n", @@ -5717,7 +5679,7 @@ lpfc_findnode_mapped(struct lpfc_vport *vport) return ndlp; } } - spin_unlock_irqrestore(shost->host_lock, iflags); + spin_unlock_irqrestore(&vport->fc_nodes_list_lock, iflags); /* FIND node did NOT FOUND */ lpfc_printf_vlog(vport, KERN_INFO, LOG_NODE, @@ -5734,7 +5696,7 @@ lpfc_setup_disc_node(struct lpfc_vport *vport, uint32_t did) if (!ndlp) { if (vport->phba->nvmet_support) return NULL; - if ((vport->fc_flag & FC_RSCN_MODE) != 0 && + if (test_bit(FC_RSCN_MODE, &vport->fc_flag) && lpfc_rscn_payload_check(vport, did) == 0) return NULL; ndlp = lpfc_nlp_init(vport, did); @@ -5744,7 +5706,7 @@ lpfc_setup_disc_node(struct lpfc_vport *vport, uint32_t did) lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "6453 Setup New Node 2B_DISC x%x " - "Data:x%x x%x x%x\n", + "Data:x%x x%x x%lx\n", ndlp->nlp_DID, ndlp->nlp_flag, ndlp->nlp_state, vport->fc_flag); @@ -5758,8 +5720,8 @@ lpfc_setup_disc_node(struct lpfc_vport *vport, uint32_t did) * The goal is to allow the target to reset its state and clear * pending IO in preparation for the initiator to recover. */ - if ((vport->fc_flag & FC_RSCN_MODE) && - !(vport->fc_flag & FC_NDISC_ACTIVE)) { + if (test_bit(FC_RSCN_MODE, &vport->fc_flag) && + !test_bit(FC_NDISC_ACTIVE, &vport->fc_flag)) { if (lpfc_rscn_payload_check(vport, did)) { /* Since this node is marked for discovery, @@ -5769,7 +5731,7 @@ lpfc_setup_disc_node(struct lpfc_vport *vport, uint32_t did) lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "6455 Setup RSCN Node 2B_DISC x%x " - "Data:x%x x%x x%x\n", + "Data:x%x x%x x%lx\n", ndlp->nlp_DID, ndlp->nlp_flag, ndlp->nlp_state, vport->fc_flag); @@ -5801,7 +5763,7 @@ lpfc_setup_disc_node(struct lpfc_vport *vport, uint32_t did) } else { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "6456 Skip Setup RSCN Node x%x " - "Data:x%x x%x x%x\n", + "Data:x%x x%x x%lx\n", ndlp->nlp_DID, ndlp->nlp_flag, ndlp->nlp_state, vport->fc_flag); ndlp = NULL; @@ -5809,7 +5771,7 @@ lpfc_setup_disc_node(struct lpfc_vport *vport, uint32_t did) } else { lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "6457 Setup Active Node 2B_DISC x%x " - "Data:x%x x%x x%x\n", + "Data:x%x x%x x%lx\n", ndlp->nlp_DID, ndlp->nlp_flag, ndlp->nlp_state, vport->fc_flag); @@ -5937,7 +5899,6 @@ lpfc_issue_reg_vpi(struct lpfc_hba *phba, struct lpfc_vport *vport) void lpfc_disc_start(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; uint32_t num_sent; uint32_t clear_la_pending; @@ -5965,9 +5926,11 @@ lpfc_disc_start(struct lpfc_vport *vport) /* Start Discovery state */ lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, "0202 Start Discovery port state x%x " - "flg x%x Data: x%x x%x x%x\n", - vport->port_state, vport->fc_flag, vport->fc_plogi_cnt, - vport->fc_adisc_cnt, vport->fc_npr_cnt); + "flg x%lx Data: x%x x%x x%x\n", + vport->port_state, vport->fc_flag, + atomic_read(&vport->fc_plogi_cnt), + atomic_read(&vport->fc_adisc_cnt), + atomic_read(&vport->fc_npr_cnt)); /* First do ADISCs - if any */ num_sent = lpfc_els_disc_adisc(vport); @@ -5977,8 +5940,8 @@ lpfc_disc_start(struct lpfc_vport *vport) /* Register the VPI for SLI3, NPIV only. */ if ((phba->sli3_options & LPFC_SLI3_NPIV_ENABLED) && - !(vport->fc_flag & FC_PT2PT) && - !(vport->fc_flag & FC_RSCN_MODE) && + !test_bit(FC_PT2PT, &vport->fc_flag) && + !test_bit(FC_RSCN_MODE, &vport->fc_flag) && (phba->sli_rev < LPFC_SLI_REV4)) { lpfc_issue_clear_la(phba, vport); lpfc_issue_reg_vpi(phba, vport); @@ -5993,16 +5956,14 @@ lpfc_disc_start(struct lpfc_vport *vport) /* If we get here, there is nothing to ADISC */ lpfc_issue_clear_la(phba, vport); - if (!(vport->fc_flag & FC_ABORT_DISCOVERY)) { + if (!test_bit(FC_ABORT_DISCOVERY, &vport->fc_flag)) { vport->num_disc_nodes = 0; /* go thru NPR nodes and issue ELS PLOGIs */ - if (vport->fc_npr_cnt) + if (atomic_read(&vport->fc_npr_cnt)) lpfc_els_disc_plogi(vport); if (!vport->num_disc_nodes) { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_NDISC_ACTIVE; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_NDISC_ACTIVE, &vport->fc_flag); lpfc_can_disctmo(vport); } } @@ -6014,18 +5975,17 @@ lpfc_disc_start(struct lpfc_vport *vport) if (num_sent) return; - if (vport->fc_flag & FC_RSCN_MODE) { + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) { /* Check to see if more RSCNs came in while we * were processing this one. */ - if ((vport->fc_rscn_id_cnt == 0) && - (!(vport->fc_flag & FC_RSCN_DISCOVERY))) { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_RSCN_MODE; - spin_unlock_irq(shost->host_lock); + if (vport->fc_rscn_id_cnt == 0 && + !test_bit(FC_RSCN_DISCOVERY, &vport->fc_flag)) { + clear_bit(FC_RSCN_MODE, &vport->fc_flag); lpfc_can_disctmo(vport); - } else + } else { lpfc_els_handle_rscn(vport); + } } } return; @@ -6092,7 +6052,8 @@ lpfc_disc_flush_list(struct lpfc_vport *vport) struct lpfc_nodelist *ndlp, *next_ndlp; struct lpfc_hba *phba = vport->phba; - if (vport->fc_plogi_cnt || vport->fc_adisc_cnt) { + if (atomic_read(&vport->fc_plogi_cnt) || + atomic_read(&vport->fc_adisc_cnt)) { list_for_each_entry_safe(ndlp, next_ndlp, &vport->fc_nodes, nlp_listp) { if (ndlp->nlp_state == NLP_STE_PLOGI_ISSUE || @@ -6173,20 +6134,15 @@ lpfc_disc_timeout(struct timer_list *t) static void lpfc_disc_timeout_handler(struct lpfc_vport *vport) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; struct lpfc_sli *psli = &phba->sli; struct lpfc_nodelist *ndlp, *next_ndlp; LPFC_MBOXQ_t *initlinkmbox; int rc, clrlaerr = 0; - if (!(vport->fc_flag & FC_DISC_TMO)) + if (!test_and_clear_bit(FC_DISC_TMO, &vport->fc_flag)) return; - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_DISC_TMO; - spin_unlock_irq(shost->host_lock); - lpfc_debugfs_disc_trc(vport, LPFC_DISC_TRC_ELS_CMD, "disc timeout: state:x%x rtry:x%x flg:x%x", vport->port_state, vport->fc_ns_retry, vport->fc_flag); @@ -6340,7 +6296,7 @@ lpfc_disc_timeout_handler(struct lpfc_vport *vport) break; case LPFC_VPORT_READY: - if (vport->fc_flag & FC_RSCN_MODE) { + if (test_bit(FC_RSCN_MODE, &vport->fc_flag)) { lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, "0231 RSCN timeout Data: x%x " @@ -6757,7 +6713,7 @@ lpfc_fcf_inuse(struct lpfc_hba *phba) struct lpfc_vport **vports; int i, ret = 0; struct lpfc_nodelist *ndlp; - struct Scsi_Host *shost; + unsigned long iflags; vports = lpfc_create_vport_work_array(phba); @@ -6766,24 +6722,23 @@ lpfc_fcf_inuse(struct lpfc_hba *phba) return 1; for (i = 0; i <= phba->max_vports && vports[i] != NULL; i++) { - shost = lpfc_shost_from_vport(vports[i]); - spin_lock_irq(shost->host_lock); /* * IF the CVL_RCVD bit is not set then we have sent the * flogi. * If dev_loss fires while we are waiting we do not want to * unreg the fcf. */ - if (!(vports[i]->fc_flag & FC_VPORT_CVL_RCVD)) { - spin_unlock_irq(shost->host_lock); + if (!test_bit(FC_VPORT_CVL_RCVD, &vports[i]->fc_flag)) { ret = 1; goto out; } + spin_lock_irqsave(&vports[i]->fc_nodes_list_lock, iflags); list_for_each_entry(ndlp, &vports[i]->fc_nodes, nlp_listp) { if (ndlp->rport && (ndlp->rport->roles & FC_RPORT_ROLE_FCP_TARGET)) { ret = 1; - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vports[i]->fc_nodes_list_lock, + iflags); goto out; } else if (ndlp->nlp_flag & NLP_RPI_REGISTERED) { ret = 1; @@ -6795,7 +6750,7 @@ lpfc_fcf_inuse(struct lpfc_hba *phba) ndlp->nlp_flag); } } - spin_unlock_irq(shost->host_lock); + spin_unlock_irqrestore(&vports[i]->fc_nodes_list_lock, iflags); } out: lpfc_destroy_vport_work_array(phba, vports); @@ -6813,7 +6768,6 @@ void lpfc_unregister_vfi_cmpl(struct lpfc_hba *phba, LPFC_MBOXQ_t *mboxq) { struct lpfc_vport *vport = mboxq->vport; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); if (mboxq->u.mb.mbxStatus) { lpfc_printf_log(phba, KERN_ERR, LOG_TRACE_EVENT, @@ -6821,9 +6775,7 @@ lpfc_unregister_vfi_cmpl(struct lpfc_hba *phba, LPFC_MBOXQ_t *mboxq) "HBA state x%x\n", mboxq->u.mb.mbxStatus, vport->port_state); } - spin_lock_irq(shost->host_lock); - phba->pport->fc_flag &= ~FC_VFI_REGISTERED; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_VFI_REGISTERED, &phba->pport->fc_flag); mempool_free(mboxq, phba->mbox_mem_pool); return; } @@ -6887,9 +6839,9 @@ lpfc_unregister_fcf_prep(struct lpfc_hba *phba) lpfc_mbx_unreg_vpi(vports[i]); shost = lpfc_shost_from_vport(vports[i]); spin_lock_irq(shost->host_lock); - vports[i]->fc_flag |= FC_VPORT_NEEDS_INIT_VPI; vports[i]->vpi_state &= ~LPFC_VPI_REGISTERED; spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_NEEDS_INIT_VPI, &vports[i]->fc_flag); } lpfc_destroy_vport_work_array(phba, vports); if (i == 0 && (!(phba->sli3_options & LPFC_SLI3_NPIV_ENABLED))) { @@ -6902,9 +6854,9 @@ lpfc_unregister_fcf_prep(struct lpfc_hba *phba) lpfc_mbx_unreg_vpi(phba->pport); shost = lpfc_shost_from_vport(phba->pport); spin_lock_irq(shost->host_lock); - phba->pport->fc_flag |= FC_VPORT_NEEDS_INIT_VPI; phba->pport->vpi_state &= ~LPFC_VPI_REGISTERED; spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_NEEDS_INIT_VPI, &phba->pport->fc_flag); } /* Cleanup any outstanding ELS commands */ diff --git a/drivers/scsi/lpfc/lpfc_init.c b/drivers/scsi/lpfc/lpfc_init.c index f78e087f2aad..899839a44c5a 100644 --- a/drivers/scsi/lpfc/lpfc_init.c +++ b/drivers/scsi/lpfc/lpfc_init.c @@ -1268,9 +1268,9 @@ lpfc_hb_mbox_cmpl(struct lpfc_hba * phba, LPFC_MBOXQ_t * pmboxq) /* Check and reset heart-beat timer if necessary */ mempool_free(pmboxq, phba->mbox_mem_pool); - if (!(phba->pport->fc_flag & FC_OFFLINE_MODE) && - !(phba->link_state == LPFC_HBA_ERROR) && - !(phba->pport->load_flag & FC_UNLOADING)) + if (!test_bit(FC_OFFLINE_MODE, &phba->pport->fc_flag) && + !(phba->link_state == LPFC_HBA_ERROR) && + !(phba->pport->load_flag & FC_UNLOADING)) mod_timer(&phba->hb_tmofunc, jiffies + msecs_to_jiffies(1000 * LPFC_HB_MBOX_INTERVAL)); @@ -1301,7 +1301,7 @@ lpfc_idle_stat_delay_work(struct work_struct *work) return; if (phba->link_state == LPFC_HBA_ERROR || - phba->pport->fc_flag & FC_OFFLINE_MODE || + test_bit(FC_OFFLINE_MODE, &phba->pport->fc_flag) || phba->cmf_active_mode != LPFC_CFG_OFF) goto requeue; @@ -1362,7 +1362,7 @@ lpfc_hb_eq_delay_work(struct work_struct *work) return; if (phba->link_state == LPFC_HBA_ERROR || - phba->pport->fc_flag & FC_OFFLINE_MODE) + test_bit(FC_OFFLINE_MODE, &phba->pport->fc_flag)) goto requeue; ena_delay = kcalloc(phba->sli4_hba.num_possible_cpu, sizeof(*ena_delay), @@ -1535,7 +1535,7 @@ lpfc_hb_timeout_handler(struct lpfc_hba *phba) if ((phba->link_state == LPFC_HBA_ERROR) || (phba->pport->load_flag & FC_UNLOADING) || - (phba->pport->fc_flag & FC_OFFLINE_MODE)) + test_bit(FC_OFFLINE_MODE, &phba->pport->fc_flag)) return; if (phba->elsbuf_cnt && @@ -3690,7 +3690,7 @@ lpfc_online(struct lpfc_hba *phba) return 0; vport = phba->pport; - if (!(vport->fc_flag & FC_OFFLINE_MODE)) + if (!test_bit(FC_OFFLINE_MODE, &vport->fc_flag)) return 0; lpfc_printf_log(phba, KERN_WARNING, LOG_INIT, @@ -3730,20 +3730,18 @@ lpfc_online(struct lpfc_hba *phba) vports = lpfc_create_vport_work_array(phba); if (vports != NULL) { for (i = 0; i <= phba->max_vports && vports[i] != NULL; i++) { - struct Scsi_Host *shost; - shost = lpfc_shost_from_vport(vports[i]); - spin_lock_irq(shost->host_lock); - vports[i]->fc_flag &= ~FC_OFFLINE_MODE; + clear_bit(FC_OFFLINE_MODE, &vports[i]->fc_flag); if (phba->sli3_options & LPFC_SLI3_NPIV_ENABLED) - vports[i]->fc_flag |= FC_VPORT_NEEDS_REG_VPI; + set_bit(FC_VPORT_NEEDS_REG_VPI, + &vports[i]->fc_flag); if (phba->sli_rev == LPFC_SLI_REV4) { - vports[i]->fc_flag |= FC_VPORT_NEEDS_INIT_VPI; + set_bit(FC_VPORT_NEEDS_INIT_VPI, + &vports[i]->fc_flag); if ((vpis_cleared) && (vports[i]->port_type != LPFC_PHYSICAL_PORT)) vports[i]->vpi = 0; } - spin_unlock_irq(shost->host_lock); } } lpfc_destroy_vport_work_array(phba, vports); @@ -3798,7 +3796,7 @@ lpfc_offline_prep(struct lpfc_hba *phba, int mbx_action) int offline; bool hba_pci_err; - if (vport->fc_flag & FC_OFFLINE_MODE) + if (test_bit(FC_OFFLINE_MODE, &vport->fc_flag)) return; lpfc_block_mgmt_io(phba, mbx_action); @@ -3817,11 +3815,10 @@ lpfc_offline_prep(struct lpfc_hba *phba, int mbx_action) shost = lpfc_shost_from_vport(vports[i]); spin_lock_irq(shost->host_lock); vports[i]->vpi_state &= ~LPFC_VPI_REGISTERED; - vports[i]->fc_flag |= FC_VPORT_NEEDS_REG_VPI; - vports[i]->fc_flag &= ~FC_VFI_REGISTERED; spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_NEEDS_REG_VPI, &vports[i]->fc_flag); + clear_bit(FC_VFI_REGISTERED, &vports[i]->fc_flag); - shost = lpfc_shost_from_vport(vports[i]); list_for_each_entry_safe(ndlp, next_ndlp, &vports[i]->fc_nodes, nlp_listp) { @@ -3903,7 +3900,7 @@ lpfc_offline(struct lpfc_hba *phba) struct lpfc_vport **vports; int i; - if (phba->pport->fc_flag & FC_OFFLINE_MODE) + if (test_bit(FC_OFFLINE_MODE, &phba->pport->fc_flag)) return; /* stop port and all timers associated with this hba */ @@ -3934,14 +3931,14 @@ lpfc_offline(struct lpfc_hba *phba) shost = lpfc_shost_from_vport(vports[i]); spin_lock_irq(shost->host_lock); vports[i]->work_port_events = 0; - vports[i]->fc_flag |= FC_OFFLINE_MODE; spin_unlock_irq(shost->host_lock); + set_bit(FC_OFFLINE_MODE, &vports[i]->fc_flag); } lpfc_destroy_vport_work_array(phba, vports); /* If OFFLINE flag is clear (i.e. unloading), cpuhp removal is handled * in hba_unset */ - if (phba->pport->fc_flag & FC_OFFLINE_MODE) + if (test_bit(FC_OFFLINE_MODE, &phba->pport->fc_flag)) __lpfc_cpuhp_remove(phba); if (phba->cfg_xri_rebalancing) @@ -4760,8 +4757,16 @@ lpfc_create_port(struct lpfc_hba *phba, int instance, struct device *dev) vport = (struct lpfc_vport *) shost->hostdata; vport->phba = phba; vport->load_flag |= FC_LOADING; - vport->fc_flag |= FC_VPORT_NEEDS_REG_VPI; + set_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); vport->fc_rscn_flush = 0; + atomic_set(&vport->fc_plogi_cnt, 0); + atomic_set(&vport->fc_adisc_cnt, 0); + atomic_set(&vport->fc_reglogin_cnt, 0); + atomic_set(&vport->fc_prli_cnt, 0); + atomic_set(&vport->fc_unmap_cnt, 0); + atomic_set(&vport->fc_map_cnt, 0); + atomic_set(&vport->fc_npr_cnt, 0); + atomic_set(&vport->fc_unused_cnt, 0); lpfc_get_vport_cfgparam(vport); /* Adjust value in vport */ @@ -4817,6 +4822,7 @@ lpfc_create_port(struct lpfc_hba *phba, int instance, struct device *dev) /* Initialize all internally managed lists. */ INIT_LIST_HEAD(&vport->fc_nodes); + spin_lock_init(&vport->fc_nodes_list_lock); INIT_LIST_HEAD(&vport->rcv_buffer_list); spin_lock_init(&vport->work_port_lock); @@ -4938,7 +4944,8 @@ int lpfc_scan_finished(struct Scsi_Host *shost, unsigned long time) goto finished; if (vport->num_disc_nodes || vport->fc_prli_sent) goto finished; - if (vport->fc_map_cnt == 0 && time < msecs_to_jiffies(2 * 1000)) + if (!atomic_read(&vport->fc_map_cnt) && + time < msecs_to_jiffies(2 * 1000)) goto finished; if ((phba->sli.sli_flag & LPFC_SLI_MBOX_ACTIVE) != 0) goto finished; @@ -6682,9 +6689,7 @@ lpfc_sli4_perform_vport_cvl(struct lpfc_vport *vport) return NULL; lpfc_linkdown_port(vport); lpfc_cleanup_pending_mbox(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_VPORT_CVL_RCVD; - spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_CVL_RCVD, &vport->fc_flag); return ndlp; } @@ -6881,9 +6886,9 @@ lpfc_sli4_async_fip_evt(struct lpfc_hba *phba, if (vports) { for (i = 0; i <= phba->max_vports && vports[i] != NULL; i++) { - if ((!(vports[i]->fc_flag & - FC_VPORT_CVL_RCVD)) && - (vports[i]->port_state > LPFC_FDISC)) { + if (!test_bit(FC_VPORT_CVL_RCVD, + &vports[i]->fc_flag) && + vports[i]->port_state > LPFC_FDISC) { active_vlink_present = 1; break; } @@ -12772,7 +12777,8 @@ static void __lpfc_cpuhp_remove(struct lpfc_hba *phba) static void lpfc_cpuhp_remove(struct lpfc_hba *phba) { - if (phba->pport && (phba->pport->fc_flag & FC_OFFLINE_MODE)) + if (phba->pport && + test_bit(FC_OFFLINE_MODE, &phba->pport->fc_flag)) return; __lpfc_cpuhp_remove(phba); diff --git a/drivers/scsi/lpfc/lpfc_mbox.c b/drivers/scsi/lpfc/lpfc_mbox.c index 0dfdc0c4c08c..43af05c8af07 100644 --- a/drivers/scsi/lpfc/lpfc_mbox.c +++ b/drivers/scsi/lpfc/lpfc_mbox.c @@ -949,7 +949,7 @@ lpfc_reg_vpi(struct lpfc_vport *vport, LPFC_MBOXQ_t *pmb) * Set the re-reg VPI bit for f/w to update the MAC address. */ if ((phba->sli_rev == LPFC_SLI_REV4) && - !(vport->fc_flag & FC_VPORT_NEEDS_REG_VPI)) + !test_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag)) mb->un.varRegVpi.upd = 1; mb->un.varRegVpi.vpi = phba->vpi_ids[vport->vpi]; @@ -2242,7 +2242,7 @@ lpfc_reg_vfi(struct lpfcMboxq *mbox, struct lpfc_vport *vport, dma_addr_t phys) /* Only FC supports upd bit */ if ((phba->sli4_hba.lnk_info.lnk_tp == LPFC_LNK_TYPE_FC) && - (vport->fc_flag & FC_VFI_REGISTERED) && + test_bit(FC_VFI_REGISTERED, &vport->fc_flag) && (!phba->fc_topology_changed)) bf_set(lpfc_reg_vfi_upd, reg_vfi, 1); @@ -2269,8 +2269,8 @@ lpfc_reg_vfi(struct lpfcMboxq *mbox, struct lpfc_vport *vport, dma_addr_t phys) } lpfc_printf_vlog(vport, KERN_INFO, LOG_MBOX, "3134 Register VFI, mydid:x%x, fcfi:%d, " - " vfi:%d, vpi:%d, fc_pname:%x%x fc_flag:x%x" - " port_state:x%x topology chg:%d bbscn_fabric :%d\n", + "vfi:%d, vpi:%d, fc_pname:%x%x fc_flag:x%lx " + "port_state:x%x topology chg:%d bbscn_fabric :%d\n", vport->fc_myDID, phba->fcf.fcfi, phba->sli4_hba.vfi_ids[vport->vfi], diff --git a/drivers/scsi/lpfc/lpfc_nportdisc.c b/drivers/scsi/lpfc/lpfc_nportdisc.c index fe174062e494..3bc5d293ad77 100644 --- a/drivers/scsi/lpfc/lpfc_nportdisc.c +++ b/drivers/scsi/lpfc/lpfc_nportdisc.c @@ -382,7 +382,7 @@ lpfc_rcv_plogi(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, /* PLOGI chkparm OK */ lpfc_printf_vlog(vport, KERN_INFO, LOG_ELS, "0114 PLOGI chkparm OK Data: x%x x%x x%x " - "x%x x%x x%x\n", + "x%x x%x x%lx\n", ndlp->nlp_DID, ndlp->nlp_state, ndlp->nlp_flag, ndlp->nlp_rpi, vport->port_state, vport->fc_flag); @@ -464,8 +464,8 @@ lpfc_rcv_plogi(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, save_iocb = NULL; /* Check for Nport to NPort pt2pt protocol */ - if ((vport->fc_flag & FC_PT2PT) && - !(vport->fc_flag & FC_PT2PT_PLOGI)) { + if (test_bit(FC_PT2PT, &vport->fc_flag) && + !test_bit(FC_PT2PT_PLOGI, &vport->fc_flag)) { /* rcv'ed PLOGI decides what our NPortId will be */ if (phba->sli_rev == LPFC_SLI_REV4) { vport->fc_myDID = bf_get(els_rsp64_sid, @@ -580,7 +580,7 @@ lpfc_rcv_plogi(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, * This only applies to a fabric environment. */ if ((ndlp->nlp_state == NLP_STE_PLOGI_ISSUE) && - (vport->fc_flag & FC_FABRIC)) { + test_bit(FC_FABRIC, &vport->fc_flag)) { /* software abort outstanding PLOGI */ lpfc_els_abort(phba, ndlp); } @@ -804,7 +804,6 @@ static int lpfc_rcv_logo(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, struct lpfc_iocbq *cmdiocb, uint32_t els_cmd) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; struct lpfc_vport **vports; int i, active_vlink_present = 0 ; @@ -837,19 +836,17 @@ lpfc_rcv_logo(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, if (ndlp->nlp_DID == Fabric_DID) { if (vport->port_state <= LPFC_FDISC || - vport->fc_flag & FC_PT2PT) + test_bit(FC_PT2PT, &vport->fc_flag)) goto out; lpfc_linkdown_port(vport); - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_VPORT_LOGO_RCVD; - spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_LOGO_RCVD, &vport->fc_flag); vports = lpfc_create_vport_work_array(phba); if (vports) { for (i = 0; i <= phba->max_vports && vports[i] != NULL; i++) { - if ((!(vports[i]->fc_flag & - FC_VPORT_LOGO_RCVD)) && - (vports[i]->port_state > LPFC_FDISC)) { + if (!test_bit(FC_VPORT_LOGO_RCVD, + &vports[i]->fc_flag) && + vports[i]->port_state > LPFC_FDISC) { active_vlink_present = 1; break; } @@ -876,23 +873,21 @@ lpfc_rcv_logo(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, ndlp->nlp_last_elscmd = ELS_CMD_FDISC; vport->port_state = LPFC_FDISC; } else { - spin_lock_irq(shost->host_lock); - phba->pport->fc_flag &= ~FC_LOGO_RCVD_DID_CHNG; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_LOGO_RCVD_DID_CHNG, &phba->pport->fc_flag); lpfc_retry_pport_discovery(phba); } } else { lpfc_printf_vlog(vport, KERN_INFO, LOG_NODE | LOG_ELS | LOG_DISCOVERY, "3203 LOGO recover nport x%06x state x%x " - "ntype x%x fc_flag x%x\n", + "ntype x%x fc_flag x%lx\n", ndlp->nlp_DID, ndlp->nlp_state, ndlp->nlp_type, vport->fc_flag); /* Special cases for rports that recover post LOGO. */ if ((!(ndlp->nlp_type == NLP_FABRIC) && (ndlp->nlp_type & (NLP_FCP_TARGET | NLP_NVME_TARGET) || - vport->fc_flag & FC_PT2PT)) || + test_bit(FC_PT2PT, &vport->fc_flag))) || (ndlp->nlp_state >= NLP_STE_ADISC_ISSUE || ndlp->nlp_state <= NLP_STE_PRLI_ISSUE)) { mod_timer(&ndlp->nlp_delayfunc, @@ -1047,9 +1042,10 @@ lpfc_disc_set_adisc(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp) return 0; } - if (!(vport->fc_flag & FC_PT2PT)) { + if (!test_bit(FC_PT2PT, &vport->fc_flag)) { /* Check config parameter use-adisc or FCP-2 */ - if (vport->cfg_use_adisc && ((vport->fc_flag & FC_RSCN_MODE) || + if (vport->cfg_use_adisc && + (test_bit(FC_RSCN_MODE, &vport->fc_flag) || ((ndlp->nlp_fcp_info & NLP_FCP_2_DEVICE) && (ndlp->nlp_type & NLP_FCP_TARGET)))) { spin_lock_irq(&ndlp->lock); @@ -1113,7 +1109,7 @@ lpfc_release_rpi(struct lpfc_hba *phba, struct lpfc_vport *vport, } if (((ndlp->nlp_DID & Fabric_DID_MASK) != Fabric_DID_MASK) && - (!(vport->fc_flag & FC_OFFLINE_MODE))) + (!test_bit(FC_OFFLINE_MODE, &vport->fc_flag))) ndlp->nlp_flag |= NLP_UNREG_INP; lpfc_printf_vlog(vport, KERN_INFO, LOG_DISCOVERY, @@ -1236,7 +1232,6 @@ static uint32_t lpfc_rcv_plogi_plogi_issue(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, void *arg, uint32_t evt) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); struct lpfc_hba *phba = vport->phba; struct lpfc_iocbq *cmdiocb = arg; struct lpfc_dmabuf *pcmd = cmdiocb->cmd_dmabuf; @@ -1271,9 +1266,7 @@ lpfc_rcv_plogi_plogi_issue(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, /* Check if there are more PLOGIs to be sent */ lpfc_more_plogi(vport); if (vport->num_disc_nodes == 0) { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~FC_NDISC_ACTIVE; - spin_unlock_irq(shost->host_lock); + clear_bit(FC_NDISC_ACTIVE, &vport->fc_flag); lpfc_can_disctmo(vport); lpfc_end_rscn(vport); } @@ -1413,8 +1406,8 @@ lpfc_cmpl_plogi_plogi_issue(struct lpfc_vport *vport, ndlp->nlp_maxframe = ((sp->cmn.bbRcvSizeMsb & 0x0F) << 8) | sp->cmn.bbRcvSizeLsb; - if ((vport->fc_flag & FC_PT2PT) && - (vport->fc_flag & FC_PT2PT_PLOGI)) { + if (test_bit(FC_PT2PT, &vport->fc_flag) && + test_bit(FC_PT2PT_PLOGI, &vport->fc_flag)) { ed_tov = be32_to_cpu(sp->cmn.e_d_tov); if (sp->cmn.edtovResolution) { /* E_D_TOV ticks are in nanoseconds */ @@ -1605,7 +1598,7 @@ lpfc_device_recov_plogi_issue(struct lpfc_vport *vport, /* Don't do anything that will mess up processing of the * previous RSCN. */ - if (vport->fc_flag & FC_RSCN_DEFERRED) + if (test_bit(FC_RSCN_DEFERRED, &vport->fc_flag)) return ndlp->nlp_state; /* software abort outstanding PLOGI */ @@ -1791,7 +1784,7 @@ lpfc_device_recov_adisc_issue(struct lpfc_vport *vport, /* Don't do anything that will mess up processing of the * previous RSCN. */ - if (vport->fc_flag & FC_RSCN_DEFERRED) + if (test_bit(FC_RSCN_DEFERRED, &vport->fc_flag)) return ndlp->nlp_state; /* software abort outstanding ADISC */ @@ -1981,13 +1974,13 @@ lpfc_cmpl_reglogin_reglogin_issue(struct lpfc_vport *vport, * know what PRLI to send yet. Figure that out now and * call PRLI depending on the outcome. */ - if (vport->fc_flag & FC_PT2PT) { + if (test_bit(FC_PT2PT, &vport->fc_flag)) { /* If we are pt2pt, there is no Fabric to determine * the FC4 type of the remote nport. So if NVME * is configured try it. */ ndlp->nlp_fc4_type |= NLP_FC4_FCP; - if ((!(vport->fc_flag & FC_PT2PT_NO_NVME)) && + if ((!test_bit(FC_PT2PT_NO_NVME, &vport->fc_flag)) && (vport->cfg_enable_fc4_type == LPFC_ENABLE_BOTH || vport->cfg_enable_fc4_type == LPFC_ENABLE_NVME)) { ndlp->nlp_fc4_type |= NLP_FC4_NVME; @@ -2019,7 +2012,7 @@ lpfc_cmpl_reglogin_reglogin_issue(struct lpfc_vport *vport, lpfc_nlp_set_state(vport, ndlp, NLP_STE_NPR_NODE); } } else { - if ((vport->fc_flag & FC_PT2PT) && phba->nvmet_support) + if (test_bit(FC_PT2PT, &vport->fc_flag) && phba->nvmet_support) phba->targetport->port_id = vport->fc_myDID; /* Only Fabric ports should transition. NVME target @@ -2060,7 +2053,7 @@ lpfc_device_recov_reglogin_issue(struct lpfc_vport *vport, /* Don't do anything that will mess up processing of the * previous RSCN. */ - if (vport->fc_flag & FC_RSCN_DEFERRED) + if (test_bit(FC_RSCN_DEFERRED, &vport->fc_flag)) return ndlp->nlp_state; ndlp->nlp_prev_state = NLP_STE_REG_LOGIN_ISSUE; @@ -2376,7 +2369,7 @@ lpfc_device_recov_prli_issue(struct lpfc_vport *vport, /* Don't do anything that will mess up processing of the * previous RSCN. */ - if (vport->fc_flag & FC_RSCN_DEFERRED) + if (test_bit(FC_RSCN_DEFERRED, &vport->fc_flag)) return ndlp->nlp_state; /* software abort outstanding PRLI */ @@ -2838,13 +2831,10 @@ static uint32_t lpfc_cmpl_logo_npr_node(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, void *arg, uint32_t evt) { - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); - /* For the fabric port just clear the fc flags. */ if (ndlp->nlp_DID == Fabric_DID) { - spin_lock_irq(shost->host_lock); - vport->fc_flag &= ~(FC_FABRIC | FC_PUBLIC_LOOP); - spin_unlock_irq(shost->host_lock); + clear_bit(FC_FABRIC, &vport->fc_flag); + clear_bit(FC_PUBLIC_LOOP, &vport->fc_flag); } lpfc_unreg_rpi(vport, ndlp); return ndlp->nlp_state; @@ -2916,7 +2906,7 @@ lpfc_device_recov_npr_node(struct lpfc_vport *vport, struct lpfc_nodelist *ndlp, /* Don't do anything that will mess up processing of the * previous RSCN. */ - if (vport->fc_flag & FC_RSCN_DEFERRED) + if (test_bit(FC_RSCN_DEFERRED, &vport->fc_flag)) return ndlp->nlp_state; lpfc_cancel_retry_delay_tmo(vport, ndlp); diff --git a/drivers/scsi/lpfc/lpfc_sli.c b/drivers/scsi/lpfc/lpfc_sli.c index c88e224feed8..6651cfd2bc0c 100644 --- a/drivers/scsi/lpfc/lpfc_sli.c +++ b/drivers/scsi/lpfc/lpfc_sli.c @@ -2909,8 +2909,8 @@ lpfc_sli_def_mbox_cmpl(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmb) shost = lpfc_shost_from_vport(vport); spin_lock_irq(shost->host_lock); vport->vpi_state |= LPFC_VPI_REGISTERED; - vport->fc_flag &= ~FC_VPORT_NEEDS_REG_VPI; spin_unlock_irq(shost->host_lock); + clear_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); } if (pmb->u.mb.mbxCommand == MBX_REG_LOGIN64) { @@ -9558,8 +9558,8 @@ lpfc_sli_issue_mbox_s3(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmbox, } /* Copy the mailbox extension data */ - if (pmbox->in_ext_byte_len && pmbox->ctx_buf) { - lpfc_sli_pcimem_bcopy(pmbox->ctx_buf, + if (pmbox->in_ext_byte_len && pmbox->ext_buf) { + lpfc_sli_pcimem_bcopy(pmbox->ext_buf, (uint8_t *)phba->mbox_ext, pmbox->in_ext_byte_len); } @@ -9572,10 +9572,10 @@ lpfc_sli_issue_mbox_s3(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmbox, = MAILBOX_HBA_EXT_OFFSET; /* Copy the mailbox extension data */ - if (pmbox->in_ext_byte_len && pmbox->ctx_buf) + if (pmbox->in_ext_byte_len && pmbox->ext_buf) lpfc_memcpy_to_slim(phba->MBslimaddr + MAILBOX_HBA_EXT_OFFSET, - pmbox->ctx_buf, pmbox->in_ext_byte_len); + pmbox->ext_buf, pmbox->in_ext_byte_len); if (mbx->mbxCommand == MBX_CONFIG_PORT) /* copy command data into host mbox for cmpl */ @@ -9698,9 +9698,9 @@ lpfc_sli_issue_mbox_s3(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmbox, lpfc_sli_pcimem_bcopy(phba->mbox, mbx, MAILBOX_CMD_SIZE); /* Copy the mailbox extension data */ - if (pmbox->out_ext_byte_len && pmbox->ctx_buf) { + if (pmbox->out_ext_byte_len && pmbox->ext_buf) { lpfc_sli_pcimem_bcopy(phba->mbox_ext, - pmbox->ctx_buf, + pmbox->ext_buf, pmbox->out_ext_byte_len); } } else { @@ -9708,9 +9708,9 @@ lpfc_sli_issue_mbox_s3(struct lpfc_hba *phba, LPFC_MBOXQ_t *pmbox, lpfc_memcpy_from_slim(mbx, phba->MBslimaddr, MAILBOX_CMD_SIZE); /* Copy the mailbox extension data */ - if (pmbox->out_ext_byte_len && pmbox->ctx_buf) { + if (pmbox->out_ext_byte_len && pmbox->ext_buf) { lpfc_memcpy_from_slim( - pmbox->ctx_buf, + pmbox->ext_buf, phba->MBslimaddr + MAILBOX_HBA_EXT_OFFSET, pmbox->out_ext_byte_len); @@ -10897,7 +10897,7 @@ __lpfc_sli_prep_els_req_rsp_s4(struct lpfc_iocbq *cmdiocbq, * all ELS pt2pt protocol traffic as well. */ if ((phba->sli3_options & LPFC_SLI3_NPIV_ENABLED) || - (vport->fc_flag & FC_PT2PT)) { + test_bit(FC_PT2PT, &vport->fc_flag)) { if (expect_rsp) { bf_set(els_req64_sid, &wqe->els_req, vport->fc_myDID); @@ -13822,10 +13822,10 @@ lpfc_sli_sp_intr_handler(int irq, void *dev_id) lpfc_sli_pcimem_bcopy(mbox, pmbox, MAILBOX_CMD_SIZE); if (pmb->out_ext_byte_len && - pmb->ctx_buf) + pmb->ext_buf) lpfc_sli_pcimem_bcopy( phba->mbox_ext, - pmb->ctx_buf, + pmb->ext_buf, pmb->out_ext_byte_len); } if (pmb->mbox_flag & LPFC_MBX_IMED_UNREG) { @@ -18614,8 +18614,8 @@ lpfc_fc_frame_to_vport(struct lpfc_hba *phba, struct fc_frame_header *fc_hdr, if (did == Fabric_DID) return phba->pport; - if ((phba->pport->fc_flag & FC_PT2PT) && - !(phba->link_state == LPFC_HBA_READY)) + if (test_bit(FC_PT2PT, &phba->pport->fc_flag) && + phba->link_state != LPFC_HBA_READY) return phba->pport; vports = lpfc_create_vport_work_array(phba); @@ -19559,8 +19559,8 @@ lpfc_sli4_handle_received_buffer(struct lpfc_hba *phba, * The pt2pt protocol allows for discovery frames * to be received without a registered VPI. */ - if (!(vport->fc_flag & FC_PT2PT) || - (phba->link_state == LPFC_HBA_READY)) { + if (!test_bit(FC_PT2PT, &vport->fc_flag) || + phba->link_state == LPFC_HBA_READY) { lpfc_in_buf_free(phba, &dmabuf->dbuf); return; } @@ -22712,7 +22712,7 @@ lpfc_sli_prep_wqe(struct lpfc_hba *phba, struct lpfc_iocbq *job) if_type = bf_get(lpfc_sli_intf_if_type, &phba->sli4_hba.sli_intf); if (if_type >= LPFC_SLI_INTF_IF_TYPE_2) { - if (job->vport->fc_flag & FC_PT2PT) { + if (test_bit(FC_PT2PT, &job->vport->fc_flag)) { bf_set(els_rsp64_sp, &wqe->xmit_els_rsp, 1); bf_set(els_rsp64_sid, &wqe->xmit_els_rsp, job->vport->fc_myDID); diff --git a/drivers/scsi/lpfc/lpfc_sli.h b/drivers/scsi/lpfc/lpfc_sli.h index cd33dfec758c..350bd4f147e8 100644 --- a/drivers/scsi/lpfc/lpfc_sli.h +++ b/drivers/scsi/lpfc/lpfc_sli.h @@ -184,6 +184,7 @@ typedef struct lpfcMboxq { struct lpfc_vport *vport; /* virtual port pointer */ void *ctx_ndlp; /* caller ndlp information */ void *ctx_buf; /* caller buffer information */ + void *ext_buf; /* extended mailbox payload */ void *context3; void (*mbox_cmpl) (struct lpfc_hba *, struct lpfcMboxq *); diff --git a/drivers/scsi/lpfc/lpfc_vport.c b/drivers/scsi/lpfc/lpfc_vport.c index 256ee797adb3..82ea0b0a5078 100644 --- a/drivers/scsi/lpfc/lpfc_vport.c +++ b/drivers/scsi/lpfc/lpfc_vport.c @@ -238,13 +238,9 @@ lpfc_unique_wwpn(struct lpfc_hba *phba, struct lpfc_vport *new_vport) static void lpfc_discovery_wait(struct lpfc_vport *vport) { struct lpfc_hba *phba = vport->phba; - uint32_t wait_flags = 0; unsigned long wait_time_max; unsigned long start_time; - wait_flags = FC_RSCN_MODE | FC_RSCN_DISCOVERY | FC_NLP_MORE | - FC_RSCN_DEFERRED | FC_NDISC_ACTIVE | FC_DISC_TMO; - /* * The time constraint on this loop is a balance between the * fabric RA_TOV value and dev_loss tmo. The driver's @@ -255,14 +251,19 @@ static void lpfc_discovery_wait(struct lpfc_vport *vport) start_time = jiffies; while (time_before(jiffies, wait_time_max)) { if ((vport->num_disc_nodes > 0) || - (vport->fc_flag & wait_flags) || + test_bit(FC_RSCN_MODE, &vport->fc_flag) || + test_bit(FC_RSCN_DISCOVERY, &vport->fc_flag) || + test_bit(FC_NLP_MORE, &vport->fc_flag) || + test_bit(FC_RSCN_DEFERRED, &vport->fc_flag) || + test_bit(FC_NDISC_ACTIVE, &vport->fc_flag) || + test_bit(FC_DISC_TMO, &vport->fc_flag) || ((vport->port_state > LPFC_VPORT_FAILED) && (vport->port_state < LPFC_VPORT_READY))) { lpfc_printf_vlog(vport, KERN_INFO, LOG_VPORT, - "1833 Vport discovery quiesce Wait:" - " state x%x fc_flags x%x" - " num_nodes x%x, waiting 1000 msecs" - " total wait msecs x%x\n", + "1833 Vport discovery quiesce Wait: " + "state x%x fc_flags x%lx " + "num_nodes x%x, waiting 1000 msecs " + "total wait msecs x%x\n", vport->port_state, vport->fc_flag, vport->num_disc_nodes, jiffies_to_msecs(jiffies - start_time)); @@ -270,9 +271,9 @@ static void lpfc_discovery_wait(struct lpfc_vport *vport) } else { /* Base case. Wait variants satisfied. Break out */ lpfc_printf_vlog(vport, KERN_INFO, LOG_VPORT, - "1834 Vport discovery quiesced:" - " state x%x fc_flags x%x" - " wait msecs x%x\n", + "1834 Vport discovery quiesced: " + "state x%x fc_flags x%lx " + "wait msecs x%x\n", vport->port_state, vport->fc_flag, jiffies_to_msecs(jiffies - start_time)); @@ -283,7 +284,7 @@ static void lpfc_discovery_wait(struct lpfc_vport *vport) if (time_after(jiffies, wait_time_max)) lpfc_printf_vlog(vport, KERN_ERR, LOG_TRACE_EVENT, "1835 Vport discovery quiesce failed:" - " state x%x fc_flags x%x wait msecs x%x\n", + " state x%x fc_flags x%lx wait msecs x%x\n", vport->port_state, vport->fc_flag, jiffies_to_msecs(jiffies - start_time)); } @@ -420,7 +421,7 @@ lpfc_vport_create(struct fc_vport *fc_vport, bool disable) * by the port. */ if ((phba->sli_rev == LPFC_SLI_REV4) && - (pport->fc_flag & FC_VFI_REGISTERED)) { + test_bit(FC_VFI_REGISTERED, &pport->fc_flag)) { rc = lpfc_sli4_init_vpi(vport); if (rc) { lpfc_printf_log(phba, KERN_ERR, LOG_TRACE_EVENT, @@ -435,7 +436,7 @@ lpfc_vport_create(struct fc_vport *fc_vport, bool disable) * Driver cannot INIT_VPI now. Set the flags to * init_vpi when reg_vfi complete. */ - vport->fc_flag |= FC_VPORT_NEEDS_INIT_VPI; + set_bit(FC_VPORT_NEEDS_INIT_VPI, &vport->fc_flag); lpfc_vport_set_state(vport, FC_VPORT_LINKDOWN); rc = VPORT_OK; goto out; @@ -535,7 +536,6 @@ disable_vport(struct fc_vport *fc_vport) struct lpfc_vport *vport = *(struct lpfc_vport **)fc_vport->dd_data; struct lpfc_hba *phba = vport->phba; struct lpfc_nodelist *ndlp = NULL; - struct Scsi_Host *shost = lpfc_shost_from_vport(vport); /* Can't disable during an outstanding delete. */ if (vport->load_flag & FC_UNLOADING) @@ -556,11 +556,8 @@ disable_vport(struct fc_vport *fc_vport) * scsi_host_put() to release the vport. */ lpfc_mbx_unreg_vpi(vport); - if (phba->sli_rev == LPFC_SLI_REV4) { - spin_lock_irq(shost->host_lock); - vport->fc_flag |= FC_VPORT_NEEDS_INIT_VPI; - spin_unlock_irq(shost->host_lock); - } + if (phba->sli_rev == LPFC_SLI_REV4) + set_bit(FC_VPORT_NEEDS_INIT_VPI, &vport->fc_flag); lpfc_vport_set_state(vport, FC_VPORT_DISABLED); lpfc_printf_vlog(vport, KERN_ERR, LOG_VPORT, @@ -584,14 +581,13 @@ enable_vport(struct fc_vport *fc_vport) spin_lock_irq(shost->host_lock); vport->load_flag |= FC_LOADING; - if (vport->fc_flag & FC_VPORT_NEEDS_INIT_VPI) { - spin_unlock_irq(shost->host_lock); + spin_unlock_irq(shost->host_lock); + if (test_bit(FC_VPORT_NEEDS_INIT_VPI, &vport->fc_flag)) { lpfc_issue_init_vpi(vport); goto out; } - vport->fc_flag |= FC_VPORT_NEEDS_REG_VPI; - spin_unlock_irq(shost->host_lock); + set_bit(FC_VPORT_NEEDS_REG_VPI, &vport->fc_flag); /* Use the Physical nodes Fabric NDLP to determine if the link is * up and ready to FDISC. diff --git a/drivers/scsi/mpi3mr/mpi3mr.h b/drivers/scsi/mpi3mr/mpi3mr.h index 90da783457fb..51ad4164a5d5 100644 --- a/drivers/scsi/mpi3mr/mpi3mr.h +++ b/drivers/scsi/mpi3mr/mpi3mr.h @@ -63,7 +63,7 @@ extern atomic64_t event_counter; #define MPI3MR_DRIVER_AUTHOR "Broadcom Inc. " #define MPI3MR_DRIVER_DESC "MPI3 Storage Controller Device Driver" -#define MPI3MR_NAME_LENGTH 32 +#define MPI3MR_NAME_LENGTH 64 #define IOCNAME "%s: " #define MPI3MR_DEFAULT_MAX_IO_SIZE (1 * 1024 * 1024) @@ -131,8 +131,6 @@ extern atomic64_t event_counter; #define MPI3MR_WATCHDOG_INTERVAL 1000 /* in milli seconds */ -#define MPI3MR_DEFAULT_CFG_PAGE_SZ 1024 /* in bytes */ - #define MPI3MR_RESET_TOPOLOGY_SETTLE_TIME 10 #define MPI3MR_SCMD_TIMEOUT (60 * HZ) @@ -290,6 +288,10 @@ enum mpi3mr_reset_reason { MPI3MR_RESET_FROM_SAS_TRANSPORT_TIMEOUT = 30, }; +#define MPI3MR_RESET_REASON_OSTYPE_LINUX 1 +#define MPI3MR_RESET_REASON_OSTYPE_SHIFT 28 +#define MPI3MR_RESET_REASON_IOCNUM_SHIFT 20 + /* Queue type definitions */ enum queue_type { MPI3MR_DEFAULT_QUEUE = 0, @@ -991,6 +993,7 @@ struct scmd_priv { * @ts_update_counter: Timestamp update counter * @reset_in_progress: Reset in progress flag * @unrecoverable: Controller unrecoverable flag + * @io_admin_reset_sync: Manage state of I/O ops during an admin reset process * @prev_reset_result: Result of previous reset * @reset_mutex: Controller reset mutex * @reset_waitq: Controller reset wait queue @@ -1033,9 +1036,6 @@ struct scmd_priv { * @io_throttle_low: I/O size to stop throttle in 512b blocks * @num_io_throttle_group: Maximum number of throttle groups * @throttle_groups: Pointer to throttle group info structures - * @cfg_page: Default memory for configuration pages - * @cfg_page_dma: Configuration page DMA address - * @cfg_page_sz: Default configuration page memory size * @sas_transport_enabled: SAS transport enabled or not * @scsi_device_channel: Channel ID for SCSI devices * @transport_cmds: Command tracker for SAS transport commands @@ -1131,7 +1131,7 @@ struct mpi3mr_ioc { spinlock_t fwevt_lock; struct list_head fwevt_list; - char watchdog_work_q_name[20]; + char watchdog_work_q_name[50]; struct workqueue_struct *watchdog_work_q; struct delayed_work watchdog_work; spinlock_t watchdog_lock; @@ -1167,6 +1167,7 @@ struct mpi3mr_ioc { u32 ts_update_counter; u8 reset_in_progress; u8 unrecoverable; + u8 io_admin_reset_sync; int prev_reset_result; struct mutex reset_mutex; wait_queue_head_t reset_waitq; @@ -1219,10 +1220,6 @@ struct mpi3mr_ioc { u16 num_io_throttle_group; struct mpi3mr_throttle_group_info *throttle_groups; - void *cfg_page; - dma_addr_t cfg_page_dma; - u16 cfg_page_sz; - u8 sas_transport_enabled; u8 scsi_device_channel; struct mpi3mr_drv_cmd transport_cmds; @@ -1321,7 +1318,7 @@ void mpi3mr_start_watchdog(struct mpi3mr_ioc *mrioc); void mpi3mr_stop_watchdog(struct mpi3mr_ioc *mrioc); int mpi3mr_soft_reset_handler(struct mpi3mr_ioc *mrioc, - u32 reset_reason, u8 snapdump); + u16 reset_reason, u8 snapdump); void mpi3mr_ioc_disable_intr(struct mpi3mr_ioc *mrioc); void mpi3mr_ioc_enable_intr(struct mpi3mr_ioc *mrioc); diff --git a/drivers/scsi/mpi3mr/mpi3mr_app.c b/drivers/scsi/mpi3mr/mpi3mr_app.c index 8e9e278d0449..27fcffa94713 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_app.c +++ b/drivers/scsi/mpi3mr/mpi3mr_app.c @@ -1534,6 +1534,7 @@ void mpi3mr_bsg_init(struct mpi3mr_ioc *mrioc) { struct device *bsg_dev = &mrioc->bsg_dev; struct device *parent = &mrioc->shost->shost_gendev; + struct request_queue *q; device_initialize(bsg_dev); @@ -1549,9 +1550,9 @@ void mpi3mr_bsg_init(struct mpi3mr_ioc *mrioc) return; } - mrioc->bsg_queue = bsg_setup_queue(bsg_dev, dev_name(bsg_dev), + q = bsg_setup_queue(bsg_dev, dev_name(bsg_dev), mpi3mr_bsg_request, NULL, 0); - if (IS_ERR(mrioc->bsg_queue)) { + if (IS_ERR(q)) { ioc_err(mrioc, "%s: bsg registration failed\n", dev_name(bsg_dev)); device_del(bsg_dev); @@ -1559,6 +1560,8 @@ void mpi3mr_bsg_init(struct mpi3mr_ioc *mrioc) return; } + mrioc->bsg_queue = q; + blk_queue_max_segments(mrioc->bsg_queue, MPI3MR_MAX_APP_XFER_SEGMENTS); blk_queue_max_hw_sectors(mrioc->bsg_queue, MPI3MR_MAX_APP_XFER_SECTORS); diff --git a/drivers/scsi/mpi3mr/mpi3mr_fw.c b/drivers/scsi/mpi3mr/mpi3mr_fw.c index b742ece3f050..119e6db20101 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_fw.c +++ b/drivers/scsi/mpi3mr/mpi3mr_fw.c @@ -11,13 +11,13 @@ #include static int -mpi3mr_issue_reset(struct mpi3mr_ioc *mrioc, u16 reset_type, u32 reset_reason); +mpi3mr_issue_reset(struct mpi3mr_ioc *mrioc, u16 reset_type, u16 reset_reason); static int mpi3mr_setup_admin_qpair(struct mpi3mr_ioc *mrioc); static void mpi3mr_process_factsdata(struct mpi3mr_ioc *mrioc, struct mpi3_ioc_facts_data *facts_data); static void mpi3mr_pel_wait_complete(struct mpi3mr_ioc *mrioc, struct mpi3mr_drv_cmd *drv_cmd); - +static int mpi3mr_check_op_admin_proc(struct mpi3mr_ioc *mrioc); static int poll_queues; module_param(poll_queues, int, 0444); MODULE_PARM_DESC(poll_queues, "Number of queues for io_uring poll mode. (Range 1 - 126)"); @@ -449,7 +449,7 @@ int mpi3mr_process_admin_reply_q(struct mpi3mr_ioc *mrioc) } do { - if (mrioc->unrecoverable) + if (mrioc->unrecoverable || mrioc->io_admin_reset_sync) break; mrioc->admin_req_ci = le16_to_cpu(reply_desc->request_queue_ci); @@ -538,7 +538,7 @@ int mpi3mr_process_op_reply_q(struct mpi3mr_ioc *mrioc, } do { - if (mrioc->unrecoverable) + if (mrioc->unrecoverable || mrioc->io_admin_reset_sync) break; req_q_idx = le16_to_cpu(reply_desc->request_queue_id) - 1; @@ -1097,7 +1097,7 @@ static inline void mpi3mr_clear_reset_history(struct mpi3mr_ioc *mrioc) static int mpi3mr_issue_and_process_mur(struct mpi3mr_ioc *mrioc, u32 reset_reason) { - u32 ioc_config, timeout, ioc_status; + u32 ioc_config, timeout, ioc_status, scratch_pad0; int retval = -1; ioc_info(mrioc, "Issuing Message unit Reset(MUR)\n"); @@ -1106,7 +1106,11 @@ static int mpi3mr_issue_and_process_mur(struct mpi3mr_ioc *mrioc, return retval; } mpi3mr_clear_reset_history(mrioc); - writel(reset_reason, &mrioc->sysif_regs->scratchpad[0]); + scratch_pad0 = ((MPI3MR_RESET_REASON_OSTYPE_LINUX << + MPI3MR_RESET_REASON_OSTYPE_SHIFT) | + (mrioc->facts.ioc_num << + MPI3MR_RESET_REASON_IOCNUM_SHIFT) | reset_reason); + writel(scratch_pad0, &mrioc->sysif_regs->scratchpad[0]); ioc_config = readl(&mrioc->sysif_regs->ioc_configuration); ioc_config &= ~MPI3_SYSIF_IOC_CONFIG_ENABLE_IOC; writel(ioc_config, &mrioc->sysif_regs->ioc_configuration); @@ -1422,11 +1426,11 @@ static inline void mpi3mr_set_diagsave(struct mpi3mr_ioc *mrioc) * Return: 0 on success, non-zero on failure. */ static int mpi3mr_issue_reset(struct mpi3mr_ioc *mrioc, u16 reset_type, - u32 reset_reason) + u16 reset_reason) { int retval = -1; u8 unlock_retry_count = 0; - u32 host_diagnostic, ioc_status, ioc_config; + u32 host_diagnostic, ioc_status, ioc_config, scratch_pad0; u32 timeout = MPI3MR_RESET_ACK_TIMEOUT * 10; if ((reset_type != MPI3_SYSIF_HOST_DIAG_RESET_ACTION_SOFT_RESET) && @@ -1478,6 +1482,9 @@ static int mpi3mr_issue_reset(struct mpi3mr_ioc *mrioc, u16 reset_type, unlock_retry_count, host_diagnostic); } while (!(host_diagnostic & MPI3_SYSIF_HOST_DIAG_DIAG_WRITE_ENABLE)); + scratch_pad0 = ((MPI3MR_RESET_REASON_OSTYPE_LINUX << + MPI3MR_RESET_REASON_OSTYPE_SHIFT) | (mrioc->facts.ioc_num << + MPI3MR_RESET_REASON_IOCNUM_SHIFT) | reset_reason); writel(reset_reason, &mrioc->sysif_regs->scratchpad[0]); writel(host_diagnostic | reset_type, &mrioc->sysif_regs->host_diagnostic); @@ -2493,7 +2500,7 @@ static void mpi3mr_watchdog_work(struct work_struct *work) unsigned long flags; enum mpi3mr_iocstate ioc_state; u32 fault, host_diagnostic, ioc_status; - u32 reset_reason = MPI3MR_RESET_FROM_FAULT_WATCH; + u16 reset_reason = MPI3MR_RESET_FROM_FAULT_WATCH; if (mrioc->reset_in_progress) return; @@ -3880,17 +3887,6 @@ int mpi3mr_init_ioc(struct mpi3mr_ioc *mrioc) mpi3mr_print_ioc_info(mrioc); - if (!mrioc->cfg_page) { - dprint_init(mrioc, "allocating config page buffers\n"); - mrioc->cfg_page_sz = MPI3MR_DEFAULT_CFG_PAGE_SZ; - mrioc->cfg_page = dma_alloc_coherent(&mrioc->pdev->dev, - mrioc->cfg_page_sz, &mrioc->cfg_page_dma, GFP_KERNEL); - if (!mrioc->cfg_page) { - retval = -1; - goto out_failed_noretry; - } - } - if (!mrioc->init_cmds.reply) { retval = mpi3mr_alloc_reply_sense_bufs(mrioc); if (retval) { @@ -4017,6 +4013,7 @@ int mpi3mr_reinit_ioc(struct mpi3mr_ioc *mrioc, u8 is_resume) goto out_failed_noretry; } + mrioc->io_admin_reset_sync = 0; if (is_resume) { dprint_reset(mrioc, "setting up single ISR\n"); retval = mpi3mr_setup_isr(mrioc, 1); @@ -4434,11 +4431,7 @@ void mpi3mr_free_mem(struct mpi3mr_ioc *mrioc) mrioc->admin_req_base, mrioc->admin_req_dma); mrioc->admin_req_base = NULL; } - if (mrioc->cfg_page) { - dma_free_coherent(&mrioc->pdev->dev, mrioc->cfg_page_sz, - mrioc->cfg_page, mrioc->cfg_page_dma); - mrioc->cfg_page = NULL; - } + if (mrioc->pel_seqnum_virt) { dma_free_coherent(&mrioc->pdev->dev, mrioc->pel_seqnum_sz, mrioc->pel_seqnum_virt, mrioc->pel_seqnum_dma); @@ -4855,6 +4848,55 @@ void mpi3mr_pel_get_seqnum_complete(struct mpi3mr_ioc *mrioc, drv_cmd->retry_count = 0; } +/** + * mpi3mr_check_op_admin_proc - + * @mrioc: Adapter instance reference + * + * Check if any of the operation reply queues + * or the admin reply queue are currently in use. + * If any queue is in use, this function waits for + * a maximum of 10 seconds for them to become available. + * + * Return: 0 on success, non-zero on failure. + */ +static int mpi3mr_check_op_admin_proc(struct mpi3mr_ioc *mrioc) +{ + + u16 timeout = 10 * 10; + u16 elapsed_time = 0; + bool op_admin_in_use = false; + + do { + op_admin_in_use = false; + + /* Check admin_reply queue first to exit early */ + if (atomic_read(&mrioc->admin_reply_q_in_use) == 1) + op_admin_in_use = true; + else { + /* Check op_reply queues */ + int i; + + for (i = 0; i < mrioc->num_queues; i++) { + if (atomic_read(&mrioc->op_reply_qinfo[i].in_use) == 1) { + op_admin_in_use = true; + break; + } + } + } + + if (!op_admin_in_use) + break; + + msleep(100); + + } while (++elapsed_time < timeout); + + if (op_admin_in_use) + return 1; + + return 0; +} + /** * mpi3mr_soft_reset_handler - Reset the controller * @mrioc: Adapter instance reference @@ -4877,7 +4919,7 @@ void mpi3mr_pel_get_seqnum_complete(struct mpi3mr_ioc *mrioc, * Return: 0 on success, non-zero on failure. */ int mpi3mr_soft_reset_handler(struct mpi3mr_ioc *mrioc, - u32 reset_reason, u8 snapdump) + u16 reset_reason, u8 snapdump) { int retval = 0, i; unsigned long flags; @@ -4928,6 +4970,7 @@ int mpi3mr_soft_reset_handler(struct mpi3mr_ioc *mrioc, mpi3mr_wait_for_host_io(mrioc, MPI3MR_RESET_HOST_IOWAIT_TIMEOUT); mpi3mr_ioc_disable_intr(mrioc); + mrioc->io_admin_reset_sync = 1; if (snapdump) { mpi3mr_set_diagsave(mrioc); @@ -4951,6 +4994,16 @@ int mpi3mr_soft_reset_handler(struct mpi3mr_ioc *mrioc, ioc_err(mrioc, "Failed to issue soft reset to the ioc\n"); goto out; } + + retval = mpi3mr_check_op_admin_proc(mrioc); + if (retval) { + ioc_err(mrioc, "Soft reset failed due to an Admin or I/O queue polling\n" + "thread still processing replies even after a 10 second\n" + "timeout. Marking the controller as unrecoverable!\n"); + + goto out; + } + if (mrioc->num_io_throttle_group != mrioc->facts.max_io_throttle_group) { ioc_err(mrioc, @@ -5023,55 +5076,6 @@ int mpi3mr_soft_reset_handler(struct mpi3mr_ioc *mrioc, return retval; } - -/** - * mpi3mr_free_config_dma_memory - free memory for config page - * @mrioc: Adapter instance reference - * @mem_desc: memory descriptor structure - * - * Check whether the size of the buffer specified by the memory - * descriptor is greater than the default page size if so then - * free the memory pointed by the descriptor. - * - * Return: Nothing. - */ -static void mpi3mr_free_config_dma_memory(struct mpi3mr_ioc *mrioc, - struct dma_memory_desc *mem_desc) -{ - if ((mem_desc->size > mrioc->cfg_page_sz) && mem_desc->addr) { - dma_free_coherent(&mrioc->pdev->dev, mem_desc->size, - mem_desc->addr, mem_desc->dma_addr); - mem_desc->addr = NULL; - } -} - -/** - * mpi3mr_alloc_config_dma_memory - Alloc memory for config page - * @mrioc: Adapter instance reference - * @mem_desc: Memory descriptor to hold dma memory info - * - * This function allocates new dmaable memory or provides the - * default config page dmaable memory based on the memory size - * described by the descriptor. - * - * Return: 0 on success, non-zero on failure. - */ -static int mpi3mr_alloc_config_dma_memory(struct mpi3mr_ioc *mrioc, - struct dma_memory_desc *mem_desc) -{ - if (mem_desc->size > mrioc->cfg_page_sz) { - mem_desc->addr = dma_alloc_coherent(&mrioc->pdev->dev, - mem_desc->size, &mem_desc->dma_addr, GFP_KERNEL); - if (!mem_desc->addr) - return -ENOMEM; - } else { - mem_desc->addr = mrioc->cfg_page; - mem_desc->dma_addr = mrioc->cfg_page_dma; - memset(mem_desc->addr, 0, mrioc->cfg_page_sz); - } - return 0; -} - /** * mpi3mr_post_cfg_req - Issue config requests and wait * @mrioc: Adapter instance reference @@ -5227,8 +5231,12 @@ static int mpi3mr_process_cfg_req(struct mpi3mr_ioc *mrioc, cfg_req->page_length = cfg_hdr->page_length; cfg_req->page_version = cfg_hdr->page_version; } - if (mpi3mr_alloc_config_dma_memory(mrioc, &mem_desc)) - goto out; + + mem_desc.addr = dma_alloc_coherent(&mrioc->pdev->dev, + mem_desc.size, &mem_desc.dma_addr, GFP_KERNEL); + + if (!mem_desc.addr) + return retval; mpi3mr_add_sg_single(&cfg_req->sgl, sgl_flags, mem_desc.size, mem_desc.dma_addr); @@ -5257,7 +5265,12 @@ static int mpi3mr_process_cfg_req(struct mpi3mr_ioc *mrioc, } out: - mpi3mr_free_config_dma_memory(mrioc, &mem_desc); + if (mem_desc.addr) { + dma_free_coherent(&mrioc->pdev->dev, mem_desc.size, + mem_desc.addr, mem_desc.dma_addr); + mem_desc.addr = NULL; + } + return retval; } diff --git a/drivers/scsi/mpi3mr/mpi3mr_os.c b/drivers/scsi/mpi3mr/mpi3mr_os.c index 1bf3572c7aac..f25772103384 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_os.c +++ b/drivers/scsi/mpi3mr/mpi3mr_os.c @@ -4053,20 +4053,45 @@ static inline void mpi3mr_setup_divert_ws(struct mpi3mr_ioc *mrioc, * mpi3mr_eh_host_reset - Host reset error handling callback * @scmd: SCSI command reference * - * Issue controller reset if the scmd is for a Physical Device, - * if the scmd is for RAID volume, then wait for - * MPI3MR_RAID_ERRREC_RESET_TIMEOUT and checke whether any - * pending I/Os prior to issuing reset to the controller. + * Issue controller reset * * Return: SUCCESS of successful reset else FAILED */ static int mpi3mr_eh_host_reset(struct scsi_cmnd *scmd) +{ + struct mpi3mr_ioc *mrioc = shost_priv(scmd->device->host); + int retval = FAILED, ret; + + ret = mpi3mr_soft_reset_handler(mrioc, + MPI3MR_RESET_FROM_EH_HOS, 1); + if (ret) + goto out; + + retval = SUCCESS; +out: + sdev_printk(KERN_INFO, scmd->device, + "Host reset is %s for scmd(%p)\n", + ((retval == SUCCESS) ? "SUCCESS" : "FAILED"), scmd); + + return retval; +} + +/** + * mpi3mr_eh_bus_reset - Bus reset error handling callback + * @scmd: SCSI command reference + * + * Checks whether pending I/Os are present for the RAID volume; + * if not there's no need to reset the adapter. + * + * Return: SUCCESS of successful reset else FAILED + */ +static int mpi3mr_eh_bus_reset(struct scsi_cmnd *scmd) { struct mpi3mr_ioc *mrioc = shost_priv(scmd->device->host); struct mpi3mr_stgt_priv_data *stgt_priv_data; struct mpi3mr_sdev_priv_data *sdev_priv_data; u8 dev_type = MPI3_DEVICE_DEVFORM_VD; - int retval = FAILED, ret; + int retval = FAILED; sdev_priv_data = scmd->device->hostdata; if (sdev_priv_data && sdev_priv_data->tgt_priv_data) { @@ -4076,25 +4101,16 @@ static int mpi3mr_eh_host_reset(struct scsi_cmnd *scmd) if (dev_type == MPI3_DEVICE_DEVFORM_VD) { mpi3mr_wait_for_host_io(mrioc, - MPI3MR_RAID_ERRREC_RESET_TIMEOUT); - if (!mpi3mr_get_fw_pending_ios(mrioc)) { + MPI3MR_RAID_ERRREC_RESET_TIMEOUT); + if (!mpi3mr_get_fw_pending_ios(mrioc)) retval = SUCCESS; - goto out; - } } + if (retval == FAILED) + mpi3mr_print_pending_host_io(mrioc); - mpi3mr_print_pending_host_io(mrioc); - ret = mpi3mr_soft_reset_handler(mrioc, - MPI3MR_RESET_FROM_EH_HOS, 1); - if (ret) - goto out; - - retval = SUCCESS; -out: sdev_printk(KERN_INFO, scmd->device, - "Host reset is %s for scmd(%p)\n", - ((retval == SUCCESS) ? "SUCCESS" : "FAILED"), scmd); - + "Bus reset is %s for scmd(%p)\n", + ((retval == SUCCESS) ? "SUCCESS" : "FAILED"), scmd); return retval; } @@ -4941,6 +4957,7 @@ static const struct scsi_host_template mpi3mr_driver_template = { .change_queue_depth = mpi3mr_change_queue_depth, .eh_device_reset_handler = mpi3mr_eh_dev_reset, .eh_target_reset_handler = mpi3mr_eh_target_reset, + .eh_bus_reset_handler = mpi3mr_eh_bus_reset, .eh_host_reset_handler = mpi3mr_eh_host_reset, .bios_param = mpi3mr_bios_param, .map_queues = mpi3mr_map_queues, diff --git a/drivers/scsi/qedi/qedi_main.c b/drivers/scsi/qedi/qedi_main.c index ede8d1f6ae23..49760f39abc8 100644 --- a/drivers/scsi/qedi/qedi_main.c +++ b/drivers/scsi/qedi/qedi_main.c @@ -2664,8 +2664,6 @@ static int __qedi_probe(struct pci_dev *pdev, int mode) snprintf(host_buf, sizeof(host_buf), "host_%d", qedi->shost->host_no); qedi_ops->common->set_name(qedi->cdev, host_buf); - qedi_ops->register_ops(qedi->cdev, &qedi_cb_ops, qedi); - memset(¶ms, 0, sizeof(params)); params.mtu = DEF_PATH_MTU + IPV6_HDR_LEN + TCP_HDR_LEN; qedi->ll2_mtu = DEF_PATH_MTU; @@ -2682,6 +2680,7 @@ static int __qedi_probe(struct pci_dev *pdev, int mode) spin_lock_init(&qedi->task_idx_lock); mutex_init(&qedi->stats_lock); } + qedi_ops->register_ops(qedi->cdev, &qedi_cb_ops, qedi); qedi_ops->ll2->register_cb_ops(qedi->cdev, &qedi_ll2_cb_ops, qedi); qedi_ops->ll2->start(qedi->cdev, ¶ms); diff --git a/drivers/scsi/qla2xxx/qla_bsg.c b/drivers/scsi/qla2xxx/qla_bsg.c index 3982f7d576a8..0fbf6dba5a36 100644 --- a/drivers/scsi/qla2xxx/qla_bsg.c +++ b/drivers/scsi/qla2xxx/qla_bsg.c @@ -158,6 +158,12 @@ qla24xx_proc_fcp_prio_cfg_cmd(struct bsg_job *bsg_job) goto exit_fcp_prio_cfg; } + if (bsg_job->request_len < + sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t)) { + ret = -EINVAL; + goto exit_fcp_prio_cfg; + } + /* Get the sub command */ oper = bsg_request->rqst_data.h_vendor.vendor_cmd[1]; @@ -756,6 +762,10 @@ qla2x00_process_loopback(struct bsg_job *bsg_job) return -EIO; } + if (bsg_job->request_len < + sizeof(struct fc_bsg_request) + 3 * sizeof(uint32_t)) + return -EINVAL; + memset(&elreq, 0, sizeof(elreq)); elreq.req_sg_cnt = dma_map_sg(&ha->pdev->dev, @@ -988,6 +998,10 @@ qla84xx_reset(struct bsg_job *bsg_job) return -EINVAL; } + if (bsg_job->request_len < + sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t)) + return -EINVAL; + flag = bsg_request->rqst_data.h_vendor.vendor_cmd[1]; rval = qla84xx_reset_chip(vha, flag == A84_ISSUE_RESET_DIAG_FW); @@ -1032,6 +1046,10 @@ qla84xx_updatefw(struct bsg_job *bsg_job) return -EINVAL; } + if (bsg_job->request_len < + sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t)) + return -EINVAL; + sg_cnt = dma_map_sg(&ha->pdev->dev, bsg_job->request_payload.sg_list, bsg_job->request_payload.sg_cnt, DMA_TO_DEVICE); if (!sg_cnt) { @@ -1482,6 +1500,10 @@ qla2x00_read_optrom(struct bsg_job *bsg_job) struct qla_hw_data *ha = vha->hw; int rval = 0; + if (bsg_job->request_len < + sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t)) + return -EINVAL; + if (ha->flags.nic_core_reset_hdlr_active) return -EBUSY; @@ -1519,6 +1541,10 @@ qla2x00_update_optrom(struct bsg_job *bsg_job) struct qla_hw_data *ha = vha->hw; int rval = 0; + if (bsg_job->request_len < + sizeof(struct fc_bsg_request) + 2 * sizeof(uint32_t)) + return -EINVAL; + mutex_lock(&ha->optrom_mutex); rval = qla2x00_optrom_setup(bsg_job, vha, 1); if (rval) { @@ -2010,6 +2036,11 @@ qlafx00_mgmt_cmd(struct bsg_job *bsg_job) struct fc_port *fcport; char *type = "FC_BSG_HST_FX_MGMT"; + if (bsg_job->request_len < + sizeof(struct fc_bsg_request) + sizeof(uint32_t) + + sizeof(struct qla_mt_iocb_rqst_fx00)) + return -EINVAL; + /* Copy the IOCB specific information */ piocb_rqst = (struct qla_mt_iocb_rqst_fx00 *) &bsg_request->rqst_data.h_vendor.vendor_cmd[1]; @@ -2412,7 +2443,7 @@ qla2x00_do_dport_diagnostics(struct bsg_job *bsg_job) !IS_QLA28XX(vha->hw)) return -EPERM; - dd = kmalloc(sizeof(*dd), GFP_KERNEL); + dd = kzalloc(sizeof(*dd), GFP_KERNEL); if (!dd) { ql_log(ql_log_warn, vha, 0x70db, "Failed to allocate memory for dport.\n"); @@ -2899,6 +2930,13 @@ qla2x00_process_vendor_specific(struct scsi_qla_host *vha, struct bsg_job *bsg_j { struct fc_bsg_request *bsg_request = bsg_job->request; + if (bsg_job->request_len < + sizeof(struct fc_bsg_request) + sizeof(uint32_t)) { + ql_log(ql_log_warn, vha, 0x7000, + "BSG request too small for vendor cmd.\n"); + return -EINVAL; + } + ql_dbg(ql_dbg_edif, vha, 0x911b, "%s FC_BSG_HST_VENDOR cmd[0]=0x%x\n", __func__, bsg_request->rqst_data.h_vendor.vendor_cmd[0]); @@ -3024,8 +3062,11 @@ qla24xx_bsg_request(struct bsg_job *bsg_job) } /* Disable port will bring down the chip, allow enable command */ - if (bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_MANAGE_HOST_PORT || - bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_GET_HOST_STATS) + if (bsg_request->msgcode == FC_BSG_HST_VENDOR && + bsg_job->request_len >= + sizeof(struct fc_bsg_request) + sizeof(uint32_t) && + (bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_MANAGE_HOST_PORT || + bsg_request->rqst_data.h_vendor.vendor_cmd[0] == QL_VND_GET_HOST_STATS)) goto skip_chip_chk; if (vha->hw->flags.port_isolated) { diff --git a/drivers/scsi/qla2xxx/qla_def.h b/drivers/scsi/qla2xxx/qla_def.h index 4f7248643335..8d90ddb07c0d 100644 --- a/drivers/scsi/qla2xxx/qla_def.h +++ b/drivers/scsi/qla2xxx/qla_def.h @@ -2570,6 +2570,8 @@ typedef struct fc_port { struct list_head list; struct scsi_qla_host *vha; struct list_head unsol_ctx_head; + /* Serializes unsol_ctx_head against ISR, DPC and NVMe transport. */ + spinlock_t unsol_ctx_lock; unsigned int conf_compl_supported:1; unsigned int deleted:2; diff --git a/drivers/scsi/qla2xxx/qla_init.c b/drivers/scsi/qla2xxx/qla_init.c index c89ad5d7b4d5..914b5993d110 100644 --- a/drivers/scsi/qla2xxx/qla_init.c +++ b/drivers/scsi/qla2xxx/qla_init.c @@ -5636,6 +5636,7 @@ qla2x00_alloc_fcport(scsi_qla_host_t *vha, gfp_t flags) INIT_LIST_HEAD(&fcport->gnl_entry); INIT_LIST_HEAD(&fcport->list); INIT_LIST_HEAD(&fcport->unsol_ctx_head); + spin_lock_init(&fcport->unsol_ctx_lock); INIT_LIST_HEAD(&fcport->sess_cmd_list); spin_lock_init(&fcport->sess_cmd_lock); diff --git a/drivers/scsi/qla2xxx/qla_iocb.c b/drivers/scsi/qla2xxx/qla_iocb.c index 0b41e8a06602..c8a44969a6c3 100644 --- a/drivers/scsi/qla2xxx/qla_iocb.c +++ b/drivers/scsi/qla2xxx/qla_iocb.c @@ -11,6 +11,10 @@ #include +/* All adapters supported by this tree use the 64-byte request ring. */ +#define qla_req_ring_slot(ha, req) ((req)->ring_ptr) +#define qla_req_entry_size(ha) REQUEST_ENTRY_SIZE + static int qla_start_scsi_type6(srb_t *sp); /** * qla2x00_get_cmd_direction() - Determine control_flag data direction. @@ -2286,10 +2290,9 @@ __qla2x00_alloc_iocbs(struct qla_qpair *qpair, srb_t *sp) struct req_que *req = qpair->req; device_reg_t *reg = ISP_QUE_REG(ha, req->id); uint32_t handle; - request_t *pkt; uint16_t cnt, req_cnt; + request_t *pkt = NULL; - pkt = NULL; req_cnt = 1; handle = 0; @@ -2343,14 +2346,18 @@ __qla2x00_alloc_iocbs(struct qla_qpair *qpair, srb_t *sp) sp->handle = handle; } - /* Prep packet */ + /* + * Prep the current request-ring slot. + */ req->cnt -= req_cnt; - pkt = req->ring_ptr; - memset(pkt, 0, REQUEST_ENTRY_SIZE); + pkt = qla_req_ring_slot(ha, req); if (IS_QLAFX00(ha)) { + memset_io((void __iomem __force *)pkt, 0, + qla_req_entry_size(ha)); wrt_reg_byte((u8 __force __iomem *)&pkt->entry_count, req_cnt); wrt_reg_dword((__le32 __force __iomem *)&pkt->handle, handle); } else { + memset(pkt, 0, qla_req_entry_size(ha)); pkt->entry_count = req_cnt; pkt->handle = handle; } @@ -2751,7 +2758,6 @@ qla24xx_els_dcmd_iocb(scsi_qla_host_t *vha, int els_opcode, if (!elsio->u.els_logo.els_logo_pyld) { /* ref: INIT */ kref_put(&sp->cmd_kref, qla2x00_sp_release); - qla2x00_free_fcport(fcport); return QLA_FUNCTION_FAILED; } @@ -2776,7 +2782,6 @@ qla24xx_els_dcmd_iocb(scsi_qla_host_t *vha, int els_opcode, if (rval != QLA_SUCCESS) { /* ref: INIT */ kref_put(&sp->cmd_kref, qla2x00_sp_release); - qla2x00_free_fcport(fcport); return QLA_FUNCTION_FAILED; } @@ -3828,6 +3833,12 @@ qla25xx_ctrlvp_iocb(srb_t *sp, struct vp_ctrl_entry_24xx *vce) */ map = (sp->u.iocb_cmd.u.ctrlvp.vp_index - 1) / 8; pos = (sp->u.iocb_cmd.u.ctrlvp.vp_index - 1) & 7; + if (map >= ARRAY_SIZE(vce->vp_idx_map)) { + ql_log(ql_log_warn, sp->vha, 0x307c, + "ctrlvp: vp_index %u exceeds vp_idx_map capacity\n", + sp->u.iocb_cmd.u.ctrlvp.vp_index); + return; + } vce->vp_idx_map[map] |= 1 << pos; } diff --git a/drivers/scsi/qla2xxx/qla_isr.c b/drivers/scsi/qla2xxx/qla_isr.c index 2df5cd3d8262..028f1bc224ca 100644 --- a/drivers/scsi/qla2xxx/qla_isr.c +++ b/drivers/scsi/qla2xxx/qla_isr.c @@ -921,14 +921,6 @@ qla27xx_copy_multiple_pkt(struct scsi_qla_host *vha, void **pkt, do { while ((total_bytes > 0) && (entry_count_remaining > 0)) { - if (rsp_q->ring_ptr->signature == RESPONSE_PROCESSED) { - ql_dbg(ql_dbg_async, vha, 0x5084, - "Ran out of IOCBs, partial data 0x%x\n", - buffer_copy_offset); - cpu_relax(); - continue; - } - new_pkt = (sts_cont_entry_t *)rsp_q->ring_ptr; *pkt = new_pkt; @@ -1205,14 +1197,6 @@ qla27xx_copy_fpin_pkt(struct scsi_qla_host *vha, void **pkt, do { while ((total_bytes > 0) && (entry_count_remaining > 0)) { - if (rsp_q->ring_ptr->signature == RESPONSE_PROCESSED) { - ql_dbg(ql_dbg_async, vha, 0x5084, - "Ran out of IOCBs, partial data 0x%x\n", - buffer_copy_offset); - cpu_relax(); - continue; - } - new_pkt = (sts_cont_entry_t *)rsp_q->ring_ptr; *pkt = new_pkt; @@ -4148,9 +4132,24 @@ void qla24xx_process_response_queue(struct scsi_qla_host *vha, "SCM not active for this port\n"); break; } + if (qla_chk_cont_iocb_avail(vha, rsp, + (response_t *)pkt, rsp_in)) { + /* + * ring_ptr and ring_index were + * pre-incremented above. Reset them + * back to current. Wait for next + * interrupt with all IOCBs to arrive + * and re-process. + */ + rsp->ring_ptr = (response_t *)pkt; + rsp->ring_index = cur_ring_index; + + ql_dbg(ql_dbg_init, vha, 0x5095, + "Defer processing FPIN...\n"); + return; + } pure_item = qla27xx_copy_fpin_pkt(vha, (void **)&pkt, &rsp); - __update_rsp_in(is_shadow_hba, rsp, rsp_in); if (!pure_item) break; qla24xx_queue_purex_item(vha, pure_item, diff --git a/drivers/scsi/qla2xxx/qla_mbx.c b/drivers/scsi/qla2xxx/qla_mbx.c index 5bb9241ebebd..391a69878705 100644 --- a/drivers/scsi/qla2xxx/qla_mbx.c +++ b/drivers/scsi/qla2xxx/qla_mbx.c @@ -4147,6 +4147,8 @@ qla24xx_report_id_acquisition(scsi_qla_host_t *vha, spin_lock_irqsave(&ha->vport_slock, flags); list_for_each_entry(vp, &ha->vp_list, list) { if (rptid_entry->vp_idx == vp->vp_idx) { + if (test_bit(VPORT_DELETE, &vp->dpc_flags)) + break; found = 1; atomic_inc(&vp->vref_count); break; @@ -5658,7 +5660,7 @@ qla2x00_get_data_rate(scsi_qla_host_t *vha) ql_dbg(ql_dbg_mbx, vha, 0x1107, "Failed=%x mb[0]=%x.\n", rval, mcp->mb[0]); } else { - if (mcp->mb[1] != 0x7) + if (mcp->mb[1] != 0x7 || IS_QLA28XX(ha)) ha->link_data_rate = mcp->mb[1]; if (IS_QLA83XX(ha) || IS_QLA27XX(ha) || IS_QLA28XX(ha)) { @@ -5669,8 +5671,6 @@ qla2x00_get_data_rate(scsi_qla_host_t *vha) ql_dbg(ql_dbg_mbx + ql_dbg_verbose, vha, 0x1108, "Done %s.\n", __func__); - if (mcp->mb[1] != 0x7) - ha->link_data_rate = mcp->mb[1]; } return rval; diff --git a/drivers/scsi/qla2xxx/qla_mid.c b/drivers/scsi/qla2xxx/qla_mid.c index 9946899dd83b..86c0217d1763 100644 --- a/drivers/scsi/qla2xxx/qla_mid.c +++ b/drivers/scsi/qla2xxx/qla_mid.c @@ -574,16 +574,19 @@ qla25xx_free_req_que(struct scsi_qla_host *vha, struct req_que *req) { struct qla_hw_data *ha = vha->hw; uint16_t que_id = req->id; + size_t req_entry_size = sizeof(request_t); - dma_free_coherent(&ha->pdev->dev, (req->length + 1) * - sizeof(request_t), req->ring, req->dma); + if (req->ring) + dma_free_coherent(&ha->pdev->dev, + (req->length + 1) * req_entry_size, + req->ring, req->dma); req->ring = NULL; req->dma = 0; if (que_id) { + mutex_lock(&ha->mq_lock); ha->req_q_map[que_id] = NULL; - mutex_lock(&ha->vport_lock); clear_bit(que_id, ha->req_qid_map); - mutex_unlock(&ha->vport_lock); + mutex_unlock(&ha->mq_lock); } kfree(req->outstanding_cmds); kfree(req); @@ -594,6 +597,7 @@ qla25xx_free_rsp_que(struct scsi_qla_host *vha, struct rsp_que *rsp) { struct qla_hw_data *ha = vha->hw; uint16_t que_id = rsp->id; + size_t rsp_entry_size = sizeof(response_t); if (rsp->msix && rsp->msix->have_irq) { free_irq(rsp->msix->vector, rsp->msix->handle); @@ -601,15 +605,22 @@ qla25xx_free_rsp_que(struct scsi_qla_host *vha, struct rsp_que *rsp) rsp->msix->in_use = 0; rsp->msix->handle = NULL; } - dma_free_coherent(&ha->pdev->dev, (rsp->length + 1) * - sizeof(response_t), rsp->ring, rsp->dma); + + /* Flush any queued response work before freeing the queue/qpair. */ + if (rsp->qpair && ha->wq) + cancel_work_sync(&rsp->qpair->q_work); + + if (rsp->ring) + dma_free_coherent(&ha->pdev->dev, + (rsp->length + 1) * rsp_entry_size, + rsp->ring, rsp->dma); rsp->ring = NULL; rsp->dma = 0; if (que_id) { + mutex_lock(&ha->mq_lock); ha->rsp_q_map[que_id] = NULL; - mutex_lock(&ha->vport_lock); clear_bit(que_id, ha->rsp_qid_map); - mutex_unlock(&ha->vport_lock); + mutex_unlock(&ha->mq_lock); } kfree(rsp); } @@ -794,9 +805,6 @@ qla25xx_create_req_que(struct qla_hw_data *ha, uint16_t options, if (ret != QLA_SUCCESS) { ql_log(ql_log_fatal, base_vha, 0x00df, "%s failed.\n", __func__); - mutex_lock(&ha->mq_lock); - clear_bit(que_id, ha->req_qid_map); - mutex_unlock(&ha->mq_lock); goto que_failed; } vha->flags.qpairs_req_created = 1; @@ -908,9 +916,6 @@ qla25xx_create_rsp_que(struct qla_hw_data *ha, uint16_t options, if (ret != QLA_SUCCESS) { ql_log(ql_log_fatal, base_vha, 0x00e7, "%s failed.\n", __func__); - mutex_lock(&ha->mq_lock); - clear_bit(que_id, ha->rsp_qid_map); - mutex_unlock(&ha->mq_lock); goto que_failed; } vha->flags.qpairs_rsp_created = 1; @@ -957,6 +962,14 @@ int qla24xx_control_vp(scsi_qla_host_t *vha, int cmd) if (vp_index == 0 || vp_index >= ha->max_npiv_vports) return QLA_PARAMETER_ERROR; + /* + * The VP_CTRL IOCB selects the target VP through a fixed 128-bit + * (16-byte) vp_idx_map bitmap, so vp_index must fit within it even + * if firmware advertises more NPIV vports. + */ + if (vp_index > sizeof_field(struct vp_ctrl_entry_24xx, vp_idx_map) * 8) + return QLA_PARAMETER_ERROR; + /* ref: INIT */ sp = qla2x00_get_sp(base_vha, NULL, GFP_KERNEL); if (!sp) diff --git a/drivers/scsi/qla2xxx/qla_nvme.c b/drivers/scsi/qla2xxx/qla_nvme.c index 23481b5d42b3..65978b7e5784 100644 --- a/drivers/scsi/qla2xxx/qla_nvme.c +++ b/drivers/scsi/qla2xxx/qla_nvme.c @@ -258,7 +258,9 @@ static void qla_nvme_release_lsrsp_cmd_kref(struct kref *kref) fd_rsp = uctx->fd_rsp; + spin_lock_irqsave(&uctx->fcport->unsol_ctx_lock, flags); list_del(&uctx->elem); + spin_unlock_irqrestore(&uctx->fcport->unsol_ctx_lock, flags); fd_rsp->done(fd_rsp); kfree(uctx); @@ -447,6 +449,9 @@ static int qla_nvme_xmt_ls_rsp(struct nvme_fc_local_port *lport, qla_nvme_ls_reject_iocb(vha, ha->base_qpair, &a, true); spin_unlock_irqrestore(ha->base_qpair->qp_lock_ptr, flags); } + spin_lock_irqsave(&uctx->fcport->unsol_ctx_lock, flags); + list_del(&uctx->elem); + spin_unlock_irqrestore(&uctx->fcport->unsol_ctx_lock, flags); kfree(uctx); return rval; } @@ -1216,7 +1221,9 @@ qla2xxx_process_purls_pkt(struct scsi_qla_host *vha, struct purex_item *item) spin_unlock_irqrestore(vha->hw->base_qpair->qp_lock_ptr, flags); } + spin_lock_irqsave(&uctx->fcport->unsol_ctx_lock, flags); list_del(&uctx->elem); + spin_unlock_irqrestore(&uctx->fcport->unsol_ctx_lock, flags); kfree(uctx); } } @@ -1258,6 +1265,7 @@ void qla2xxx_process_purls_iocb(void **pkt, struct rsp_que **rsp) struct purex_item *item; port_id_t d_id = {0}; port_id_t id = {0}; + unsigned long flags; u8 *opcode; bool xmt_reject = false; @@ -1323,7 +1331,9 @@ void qla2xxx_process_purls_iocb(void **pkt, struct rsp_que **rsp) uctx->ox_id = p->ox_id; qla_rport->uctx = uctx; INIT_LIST_HEAD(&uctx->elem); + spin_lock_irqsave(&fcport->unsol_ctx_lock, flags); list_add_tail(&uctx->elem, &fcport->unsol_ctx_head); + spin_unlock_irqrestore(&fcport->unsol_ctx_lock, flags); item->purls_context = (void *)uctx; ql_dbg(ql_dbg_unsol, vha, 0x2121, diff --git a/drivers/scsi/qla2xxx/qla_os.c b/drivers/scsi/qla2xxx/qla_os.c index ec165cd64033..cac01f733492 100644 --- a/drivers/scsi/qla2xxx/qla_os.c +++ b/drivers/scsi/qla2xxx/qla_os.c @@ -1888,10 +1888,26 @@ __qla2x00_abort_all_cmds(struct qla_qpair *qp, int res) continue; } cmd = (struct qla_tgt_cmd *)sp; - cmd->aborted = 1; + + if (cmd->sg_mapped) + qlt_unmap_sg(vha, cmd); + + if (cmd->state == QLA_TGT_STATE_NEED_DATA) { + cmd->aborted = 1; + cmd->write_data_transferred = 0; + cmd->state = QLA_TGT_STATE_DATA_IN; + ha->tgt.tgt_ops->handle_data(cmd); + } else { + ha->tgt.tgt_ops->free_cmd(cmd); + } break; case TYPE_TGT_TMCMD: - /* Skip task management functions. */ + /* + * Currently, only ABTS response gets on the + * outstanding_cmds[] + */ + ha->tgt.tgt_ops->free_mcmd( + (struct qla_tgt_mgmt_cmd *) sp); break; default: break; @@ -4480,25 +4496,40 @@ qla2x00_mem_alloc(struct qla_hw_data *ha, uint16_t req_len, uint16_t rsp_len, fail_lsrjt: dma_free_coherent(&ha->pdev->dev, ha->elsrej.size, ha->elsrej.c, ha->elsrej.cdma); + ha->elsrej.c = NULL; + ha->elsrej.cdma = 0; fail_elsrej: dma_pool_destroy(ha->purex_dma_pool); + ha->purex_dma_pool = NULL; fail_flt: dma_free_coherent(&ha->pdev->dev, sizeof(struct qla_flt_header) + FLT_REGIONS_SIZE, ha->flt, ha->flt_dma); + ha->flt = NULL; + ha->flt_dma = 0; fail_flt_buffer: dma_free_coherent(&ha->pdev->dev, SFP_DEV_SIZE, ha->sfp_data, ha->sfp_data_dma); + ha->sfp_data = NULL; + ha->sfp_data_dma = 0; fail_sfp_data: kfree(ha->loop_id_map); + ha->loop_id_map = NULL; fail_loop_id_map: dma_pool_free(ha->s_dma_pool, ha->async_pd, ha->async_pd_dma); + ha->async_pd = NULL; + ha->async_pd_dma = 0; fail_async_pd: dma_pool_free(ha->s_dma_pool, ha->sf_init_cb, ha->sf_init_cb_dma); + ha->sf_init_cb = NULL; + ha->sf_init_cb_dma = 0; fail_sf_init_cb: dma_pool_free(ha->s_dma_pool, ha->ex_init_cb, ha->ex_init_cb_dma); + ha->ex_init_cb = NULL; + ha->ex_init_cb_dma = 0; fail_ex_init_cb: kfree(ha->npiv_info); + ha->npiv_info = NULL; fail_npiv_info: dma_free_coherent(&ha->pdev->dev, ((*rsp)->length + 1) * sizeof(response_t), (*rsp)->ring, (*rsp)->dma); diff --git a/drivers/scsi/qla2xxx/qla_target.c b/drivers/scsi/qla2xxx/qla_target.c index d7551b1443e4..ed9a60a292c3 100644 --- a/drivers/scsi/qla2xxx/qla_target.c +++ b/drivers/scsi/qla2xxx/qla_target.c @@ -2487,7 +2487,7 @@ static int qlt_pci_map_calc_cnt(struct qla_tgt_prm *prm) return -1; } -static void qlt_unmap_sg(struct scsi_qla_host *vha, struct qla_tgt_cmd *cmd) +void qlt_unmap_sg(struct scsi_qla_host *vha, struct qla_tgt_cmd *cmd) { struct qla_hw_data *ha; struct qla_qpair *qpair; @@ -3817,9 +3817,6 @@ int qlt_abort_cmd(struct qla_tgt_cmd *cmd) spin_lock_irqsave(&cmd->cmd_lock, flags); if (cmd->aborted) { - if (cmd->sg_mapped) - qlt_unmap_sg(vha, cmd); - spin_unlock_irqrestore(&cmd->cmd_lock, flags); /* * It's normal to see 2 calls in this path: diff --git a/drivers/scsi/qla2xxx/qla_target.h b/drivers/scsi/qla2xxx/qla_target.h index 354fca2e7feb..3644ab140ff9 100644 --- a/drivers/scsi/qla2xxx/qla_target.h +++ b/drivers/scsi/qla2xxx/qla_target.h @@ -1059,6 +1059,7 @@ extern int qlt_abort_cmd(struct qla_tgt_cmd *); extern void qlt_xmit_tm_rsp(struct qla_tgt_mgmt_cmd *); extern void qlt_free_mcmd(struct qla_tgt_mgmt_cmd *); extern void qlt_free_cmd(struct qla_tgt_cmd *cmd); +extern void qlt_unmap_sg(struct scsi_qla_host *vha, struct qla_tgt_cmd *cmd); extern void qlt_async_event(uint16_t, struct scsi_qla_host *, uint16_t *); extern void qlt_enable_vha(struct scsi_qla_host *); extern void qlt_vport_create(struct scsi_qla_host *, struct qla_hw_data *); diff --git a/drivers/scsi/scsi_error.c b/drivers/scsi/scsi_error.c index bfa2e477a5b6..e7c9764152e9 100644 --- a/drivers/scsi/scsi_error.c +++ b/drivers/scsi/scsi_error.c @@ -73,6 +73,26 @@ void scsi_eh_wakeup(struct Scsi_Host *shost, unsigned int busy) } } +void scsi_rcu_eh_wakeup(struct work_struct *work) +{ + struct Scsi_Host *shost = container_of(work, struct Scsi_Host, eh_work); + unsigned long flags; + unsigned int busy; + + /* + * Ensure any running scsi_dec_host_busy has completed its rcu section + * so changes to host state and host_eh_scheduled are visible to all + * future calls of scsi_dec_host_busy + */ + synchronize_rcu(); + + busy = scsi_host_busy(shost); + + spin_lock_irqsave(shost->host_lock, flags); + scsi_eh_wakeup(shost, busy); + spin_unlock_irqrestore(shost->host_lock, flags); +} + /** * scsi_schedule_eh - schedule EH for SCSI host * @shost: SCSI host to invoke error handling on. @@ -88,7 +108,7 @@ void scsi_schedule_eh(struct Scsi_Host *shost) if (scsi_host_set_state(shost, SHOST_RECOVERY) == 0 || scsi_host_set_state(shost, SHOST_CANCEL_RECOVERY) == 0) { shost->host_eh_scheduled++; - scsi_eh_wakeup(shost, scsi_host_busy(shost)); + queue_work(shost->tmf_work_q, &shost->eh_work); } spin_unlock_irqrestore(shost->host_lock, flags); @@ -2317,6 +2337,7 @@ static void scsi_unjam_host(struct Scsi_Host *shost) int scsi_error_handler(void *data) { struct Scsi_Host *shost = data; + bool eh_noresume; /* * We use TASK_INTERRUPTIBLE so that the thread is not @@ -2358,7 +2379,8 @@ int scsi_error_handler(void *data) * what we need to do to get it up and online again (if we can). * If we fail, we end up taking the thing offline. */ - if (!shost->eh_noresume && scsi_autopm_get_host(shost) != 0) { + eh_noresume = READ_ONCE(shost->eh_noresume); + if (!eh_noresume && scsi_autopm_get_host(shost) != 0) { SCSI_LOG_ERROR_RECOVERY(1, shost_printk(KERN_ERR, shost, "scsi_eh_%d: unable to autoresume\n", @@ -2382,7 +2404,7 @@ int scsi_error_handler(void *data) * which are still online. */ scsi_restart_operations(shost); - if (!shost->eh_noresume) + if (!eh_noresume) scsi_autopm_put_host(shost); } __set_current_state(TASK_RUNNING); diff --git a/drivers/scsi/scsi_priv.h b/drivers/scsi/scsi_priv.h index 1fbfe1b52c9f..c4ffe5f53da3 100644 --- a/drivers/scsi/scsi_priv.h +++ b/drivers/scsi/scsi_priv.h @@ -93,6 +93,7 @@ extern enum blk_eh_timer_return scsi_timeout(struct request *req); extern int scsi_error_handler(void *host); extern enum scsi_disposition scsi_decide_disposition(struct scsi_cmnd *cmd); extern void scsi_eh_wakeup(struct Scsi_Host *shost, unsigned int busy); +extern void scsi_rcu_eh_wakeup(struct work_struct *work); extern void scsi_eh_scmd_add(struct scsi_cmnd *); void scsi_eh_ready_devs(struct Scsi_Host *shost, struct list_head *work_q, diff --git a/drivers/scsi/sd.c b/drivers/scsi/sd.c index a6c8d786125e..ce9165f67919 100644 --- a/drivers/scsi/sd.c +++ b/drivers/scsi/sd.c @@ -872,6 +872,17 @@ static void sd_config_discard(struct scsi_disk *sdkp, unsigned int mode) blk_queue_max_discard_sectors(q, max_blocks * (logical_block_size >> 9)); } +static void sd_uninit_command(struct scsi_cmnd *cmd) +{ + struct request *rq = scsi_cmd_to_rq(cmd); + + if (!(rq->rq_flags & RQF_SPECIAL_PAYLOAD)) + return; + + mempool_free(rq->special_vec.bv_page, sd_page_pool); + rq->rq_flags &= ~RQF_SPECIAL_PAYLOAD; +} + static void *sd_set_special_bvec(struct request *rq, unsigned int data_len) { struct page *page; @@ -894,6 +905,7 @@ static blk_status_t sd_setup_unmap_cmnd(struct scsi_cmnd *cmd) u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq)); unsigned int data_len = 24; char *buf; + blk_status_t ret; buf = sd_set_special_bvec(rq, data_len); if (!buf) @@ -912,7 +924,10 @@ static blk_status_t sd_setup_unmap_cmnd(struct scsi_cmnd *cmd) cmd->transfersize = data_len; rq->timeout = SD_TIMEOUT; - return scsi_alloc_sgtables(cmd); + ret = scsi_alloc_sgtables(cmd); + if (ret != BLK_STS_OK) + sd_uninit_command(cmd); + return ret; } static blk_status_t sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd, @@ -924,6 +939,7 @@ static blk_status_t sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd, u64 lba = sectors_to_logical(sdp, blk_rq_pos(rq)); u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq)); u32 data_len = sdp->sector_size; + blk_status_t ret; if (!sd_set_special_bvec(rq, data_len)) return BLK_STS_RESOURCE; @@ -939,7 +955,10 @@ static blk_status_t sd_setup_write_same16_cmnd(struct scsi_cmnd *cmd, cmd->transfersize = data_len; rq->timeout = unmap ? SD_TIMEOUT : SD_WRITE_SAME_TIMEOUT; - return scsi_alloc_sgtables(cmd); + ret = scsi_alloc_sgtables(cmd); + if (ret != BLK_STS_OK) + sd_uninit_command(cmd); + return ret; } static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd, @@ -951,6 +970,7 @@ static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd, u64 lba = sectors_to_logical(sdp, blk_rq_pos(rq)); u32 nr_blocks = sectors_to_logical(sdp, blk_rq_sectors(rq)); u32 data_len = sdp->sector_size; + blk_status_t ret; if (!sd_set_special_bvec(rq, data_len)) return BLK_STS_RESOURCE; @@ -966,7 +986,10 @@ static blk_status_t sd_setup_write_same10_cmnd(struct scsi_cmnd *cmd, cmd->transfersize = data_len; rq->timeout = unmap ? SD_TIMEOUT : SD_WRITE_SAME_TIMEOUT; - return scsi_alloc_sgtables(cmd); + ret = scsi_alloc_sgtables(cmd); + if (ret != BLK_STS_OK) + sd_uninit_command(cmd); + return ret; } static blk_status_t sd_setup_write_zeroes_cmnd(struct scsi_cmnd *cmd) @@ -1340,14 +1363,6 @@ static blk_status_t sd_init_command(struct scsi_cmnd *cmd) } } -static void sd_uninit_command(struct scsi_cmnd *SCpnt) -{ - struct request *rq = scsi_cmd_to_rq(SCpnt); - - if (rq->rq_flags & RQF_SPECIAL_PAYLOAD) - mempool_free(rq->special_vec.bv_page, sd_page_pool); -} - static bool sd_need_revalidate(struct gendisk *disk, struct scsi_disk *sdkp) { if (sdkp->device->removable || sdkp->write_prot) { diff --git a/drivers/soc/aspeed/aspeed-lpc-snoop.c b/drivers/soc/aspeed/aspeed-lpc-snoop.c index 3dd2ab8336aa..3527f99cf499 100644 --- a/drivers/soc/aspeed/aspeed-lpc-snoop.c +++ b/drivers/soc/aspeed/aspeed-lpc-snoop.c @@ -11,6 +11,7 @@ */ #include +#include #include #include #include @@ -59,6 +60,7 @@ struct aspeed_lpc_snoop_model_data { struct aspeed_lpc_snoop_channel { bool enabled; + spinlock_t lock; struct kfifo fifo; wait_queue_head_t wq; struct miscdevice miscdev; @@ -82,6 +84,7 @@ static ssize_t snoop_file_read(struct file *file, char __user *buffer, size_t count, loff_t *ppos) { struct aspeed_lpc_snoop_channel *chan = snoop_file_to_chan(file); + u8 *buf __free(kfree) = NULL; unsigned int copied; int ret = 0; @@ -93,9 +96,16 @@ static ssize_t snoop_file_read(struct file *file, char __user *buffer, if (ret == -ERESTARTSYS) return -EINTR; } - ret = kfifo_to_user(&chan->fifo, buffer, count, &copied); - if (ret) - return ret; + + count = min_t(size_t, count, SNOOP_FIFO_SIZE); + + buf = kmalloc(count, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + copied = kfifo_out_spinlocked(&chan->fifo, buf, count, &chan->lock); + if (copied && copy_to_user(buffer, buf, copied)) + return -EFAULT; return copied; } @@ -119,11 +129,13 @@ static const struct file_operations snoop_fops = { /* Save a byte to a FIFO and discard the oldest byte if FIFO is full */ static void put_fifo_with_discard(struct aspeed_lpc_snoop_channel *chan, u8 val) { - if (!kfifo_initialized(&chan->fifo)) - return; - if (kfifo_is_full(&chan->fifo)) - kfifo_skip(&chan->fifo); - kfifo_put(&chan->fifo, val); + scoped_guard(spinlock, &chan->lock) { + if (!kfifo_initialized(&chan->fifo)) + return; + if (kfifo_is_full(&chan->fifo)) + kfifo_skip(&chan->fifo); + kfifo_put(&chan->fifo, val); + } wake_up_interruptible(&chan->wq); } @@ -292,6 +304,9 @@ static int aspeed_lpc_snoop_probe(struct platform_device *pdev) if (!lpc_snoop) return -ENOMEM; + spin_lock_init(&lpc_snoop->chan[0].lock); + spin_lock_init(&lpc_snoop->chan[1].lock); + np = pdev->dev.parent->of_node; if (!of_device_is_compatible(np, "aspeed,ast2400-lpc-v2") && !of_device_is_compatible(np, "aspeed,ast2500-lpc-v2") && diff --git a/drivers/soc/xilinx/zynqmp_power.c b/drivers/soc/xilinx/zynqmp_power.c index d7c784d77208..92c6efb209ef 100644 --- a/drivers/soc/xilinx/zynqmp_power.c +++ b/drivers/soc/xilinx/zynqmp_power.c @@ -277,16 +277,16 @@ static int zynqmp_pm_probe(struct platform_device *pdev) return 0; } -static int zynqmp_pm_remove(struct platform_device *pdev) +static void zynqmp_pm_remove(struct platform_device *pdev) { sysfs_remove_file(&pdev->dev.kobj, &dev_attr_suspend_mode.attr); if (event_registered) xlnx_unregister_event(PM_INIT_SUSPEND_CB, 0, 0, suspend_event_callback, NULL); - if (!rx_chan) + if (rx_chan) { mbox_free_channel(rx_chan); - - return 0; + rx_chan = NULL; + } } static const struct of_device_id pm_of_match[] = { @@ -297,7 +297,7 @@ MODULE_DEVICE_TABLE(of, pm_of_match); static struct platform_driver zynqmp_pm_platform_driver = { .probe = zynqmp_pm_probe, - .remove = zynqmp_pm_remove, + .remove_new = zynqmp_pm_remove, .driver = { .name = "zynqmp_power", .of_match_table = pm_of_match, diff --git a/drivers/soundwire/generic_bandwidth_allocation.c b/drivers/soundwire/generic_bandwidth_allocation.c index 31162f2b5638..b9b7308977a2 100644 --- a/drivers/soundwire/generic_bandwidth_allocation.c +++ b/drivers/soundwire/generic_bandwidth_allocation.c @@ -201,28 +201,26 @@ static int sdw_add_element_group_count(struct sdw_group *group, int num = group->count; int i; - for (i = 0; i <= num; i++) { + for (i = 0; i < num; i++) { if (rate == group->rates[i]) - break; - - if (i != num) - continue; + return 0; + } - if (group->count >= group->max_size) { - unsigned int *rates; + if (group->count >= group->max_size) { + unsigned int *rates; - group->max_size += 1; - rates = krealloc(group->rates, - (sizeof(int) * group->max_size), - GFP_KERNEL); - if (!rates) - return -ENOMEM; - group->rates = rates; - } + group->max_size += 1; + rates = krealloc(group->rates, + (sizeof(int) * group->max_size), + GFP_KERNEL); + if (!rates) + return -ENOMEM; - group->rates[group->count++] = rate; + group->rates = rates; } + group->rates[group->count++] = rate; + return 0; } diff --git a/drivers/spi/spi-dw-dma.c b/drivers/spi/spi-dw-dma.c index 96fe5fd0228e..62fb031ada30 100644 --- a/drivers/spi/spi-dw-dma.c +++ b/drivers/spi/spi-dw-dma.c @@ -271,7 +271,7 @@ static int dw_spi_dma_wait(struct dw_spi *dws, unsigned int len, u32 speed) msecs_to_jiffies(ms)); if (ms == 0) { - dev_err(&dws->host->cur_msg->spi->dev, + dev_err(&dws->host->dev, "DMA transaction timed out\n"); return -ETIMEDOUT; } diff --git a/drivers/spi/spi-microchip-core-qspi.c b/drivers/spi/spi-microchip-core-qspi.c index 160861891b46..f8cee68a9347 100644 --- a/drivers/spi/spi-microchip-core-qspi.c +++ b/drivers/spi/spi-microchip-core-qspi.c @@ -74,6 +74,13 @@ #define STATUS_FLAGSX4 BIT(8) #define STATUS_MASK GENMASK(8, 0) +/* + * QSPI Direct Access register defines + */ +#define DIRECT_ACCESS_EN_SSEL BIT(0) +#define DIRECT_ACCESS_OP_SSEL BIT(1) +#define DIRECT_ACCESS_OP_SSEL_SHIFT 1 + #define BYTESUPPER_MASK GENMASK(31, 16) #define BYTESLOWER_MASK GENMASK(15, 0) @@ -158,6 +165,38 @@ static int mchp_coreqspi_set_mode(struct mchp_coreqspi *qspi, const struct spi_m return 0; } +static void mchp_coreqspi_set_cs(struct spi_device *spi, bool enable) +{ + struct mchp_coreqspi *qspi = spi_controller_get_devdata(spi->controller); + u32 val; + + val = readl(qspi->regs + REG_DIRECT_ACCESS); + + val &= ~DIRECT_ACCESS_OP_SSEL; + val |= !enable << DIRECT_ACCESS_OP_SSEL_SHIFT; + + writel(val, qspi->regs + REG_DIRECT_ACCESS); +} + +static int mchp_coreqspi_setup(struct spi_device *spi) +{ + struct mchp_coreqspi *qspi = spi_controller_get_devdata(spi->controller); + u32 val; + + /* + * Active low devices need to be specifically set to their inactive + * states during probe. + */ + if (spi->mode & SPI_CS_HIGH) + return 0; + + val = readl(qspi->regs + REG_DIRECT_ACCESS); + val |= DIRECT_ACCESS_OP_SSEL; + writel(val, qspi->regs + REG_DIRECT_ACCESS); + + return 0; +} + static inline void mchp_coreqspi_read_op(struct mchp_coreqspi *qspi) { u32 control, data; @@ -298,19 +337,6 @@ static int mchp_coreqspi_setup_clock(struct mchp_coreqspi *qspi, struct spi_devi return 0; } -static int mchp_coreqspi_setup_op(struct spi_device *spi_dev) -{ - struct spi_controller *ctlr = spi_dev->master; - struct mchp_coreqspi *qspi = spi_controller_get_devdata(ctlr); - u32 control = readl_relaxed(qspi->regs + REG_CONTROL); - - control |= (CONTROL_MASTER | CONTROL_ENABLE); - control &= ~CONTROL_CLKIDLE; - writel_relaxed(control, qspi->regs + REG_CONTROL); - - return 0; -} - static inline void mchp_coreqspi_config_op(struct mchp_coreqspi *qspi, const struct spi_mem_op *op) { u32 idle_cycles = 0; @@ -409,6 +435,7 @@ static int mchp_coreqspi_exec_op(struct spi_mem *mem, const struct spi_mem_op *o reinit_completion(&qspi->data_completion); mchp_coreqspi_config_op(qspi, op); + mchp_coreqspi_set_cs(mem->spi, true); if (op->cmd.opcode) { qspi->txbuf = &opcode; qspi->rxbuf = NULL; @@ -449,6 +476,7 @@ static int mchp_coreqspi_exec_op(struct spi_mem *mem, const struct spi_mem_op *o err = -ETIMEDOUT; error: + mchp_coreqspi_set_cs(mem->spi, false); mutex_unlock(&qspi->op_lock); mchp_coreqspi_disable_ints(qspi); @@ -505,6 +533,7 @@ static int mchp_coreqspi_probe(struct platform_device *pdev) struct device *dev = &pdev->dev; struct device_node *np = dev->of_node; int ret; + u32 num_cs, val; ctlr = devm_spi_alloc_master(&pdev->dev, sizeof(*qspi)); if (!ctlr) @@ -538,12 +567,34 @@ static int mchp_coreqspi_probe(struct platform_device *pdev) return ret; } + /* + * The IP core only has a single CS, any more have to be provided via + * gpios + */ + if (of_property_read_u32(pdev->dev.of_node, "num-cs", &num_cs)) + num_cs = 1; + + ctlr->num_chipselect = num_cs; + ctlr->bits_per_word_mask = SPI_BPW_MASK(8); ctlr->mem_ops = &mchp_coreqspi_mem_ops; - ctlr->setup = mchp_coreqspi_setup_op; ctlr->mode_bits = SPI_CPOL | SPI_CPHA | SPI_RX_DUAL | SPI_RX_QUAD | SPI_TX_DUAL | SPI_TX_QUAD; ctlr->dev.of_node = np; + ctlr->setup = mchp_coreqspi_setup; + ctlr->set_cs = mchp_coreqspi_set_cs; + ctlr->use_gpio_descriptors = true; + + val = readl_relaxed(qspi->regs + REG_CONTROL); + val |= (CONTROL_MASTER | CONTROL_ENABLE); + writel_relaxed(val, qspi->regs + REG_CONTROL); + + /* + * Put cs into software controlled mode + */ + val = readl_relaxed(qspi->regs + REG_DIRECT_ACCESS); + val |= DIRECT_ACCESS_EN_SSEL; + writel(val, qspi->regs + REG_DIRECT_ACCESS); ret = spi_register_controller(ctlr); if (ret) diff --git a/drivers/spi/spi-mpc52xx.c b/drivers/spi/spi-mpc52xx.c index 16ab333af009..8f2f0821b1e1 100644 --- a/drivers/spi/spi-mpc52xx.c +++ b/drivers/spi/spi-mpc52xx.c @@ -62,7 +62,7 @@ MODULE_LICENSE("GPL"); /* Driver internal data */ struct mpc52xx_spi { - struct spi_master *master; + struct spi_controller *host; void __iomem *regs; int irq0; /* MODF irq */ int irq1; /* SPIF irq */ @@ -152,7 +152,7 @@ mpc52xx_spi_fsmstate_idle(int irq, struct mpc52xx_spi *ms, u8 status, u8 data) u8 ctrl1; if (status && irq) - dev_err(&ms->master->dev, "spurious irq, status=0x%.2x\n", + dev_err(&ms->host->dev, "spurious irq, status=0x%.2x\n", status); /* Check if there is another transfer waiting. */ @@ -235,7 +235,7 @@ static int mpc52xx_spi_fsmstate_transfer(int irq, struct mpc52xx_spi *ms, return FSM_CONTINUE; } else if (status & SPI_STATUS_MODF) { ms->modf_count++; - dev_err(&ms->master->dev, "mode fault\n"); + dev_err(&ms->host->dev, "mode fault\n"); mpc52xx_spi_chipsel(ms, 0); ms->message->status = -EIO; if (ms->message->complete) @@ -280,7 +280,7 @@ static int mpc52xx_spi_fsmstate_wait(int irq, struct mpc52xx_spi *ms, u8 status, u8 data) { if (status && irq) - dev_err(&ms->master->dev, "spurious irq, status=0x%.2x\n", + dev_err(&ms->host->dev, "spurious irq, status=0x%.2x\n", status); if (((int)mftb()) - ms->timestamp < 0) @@ -361,12 +361,12 @@ static void mpc52xx_spi_wq(struct work_struct *work) } /* - * spi_master ops + * spi_controller ops */ static int mpc52xx_spi_transfer(struct spi_device *spi, struct spi_message *m) { - struct mpc52xx_spi *ms = spi_master_get_devdata(spi->master); + struct mpc52xx_spi *ms = spi_controller_get_devdata(spi->controller); unsigned long flags; m->actual_length = 0; @@ -385,7 +385,7 @@ static int mpc52xx_spi_transfer(struct spi_device *spi, struct spi_message *m) */ static int mpc52xx_spi_probe(struct platform_device *op) { - struct spi_master *master; + struct spi_controller *host; struct mpc52xx_spi *ms; struct gpio_desc *gpio_cs; void __iomem *regs; @@ -406,7 +406,7 @@ static int mpc52xx_spi_probe(struct platform_device *op) out_8(regs + SPI_PORTDATA, 0x8); /* Deassert /SS signal */ /* Clear the status register and re-read it to check for a MODF - * failure. This driver cannot currently handle multiple masters + * failure. This driver cannot currently handle multiple hosts * on the SPI bus. This fault will also occur if the SPI signals * are not connected to any pins (port_config setting) */ in_8(regs + SPI_STATUS); @@ -419,22 +419,22 @@ static int mpc52xx_spi_probe(struct platform_device *op) goto err_init; } - dev_dbg(&op->dev, "allocating spi_master struct\n"); - master = spi_alloc_master(&op->dev, sizeof(*ms)); - if (!master) { + dev_dbg(&op->dev, "allocating spi_controller struct\n"); + host = spi_alloc_host(&op->dev, sizeof(*ms)); + if (!host) { rc = -ENOMEM; goto err_alloc; } - master->transfer = mpc52xx_spi_transfer; - master->mode_bits = SPI_CPOL | SPI_CPHA | SPI_LSB_FIRST; - master->bits_per_word_mask = SPI_BPW_MASK(8); - master->dev.of_node = op->dev.of_node; + host->transfer = mpc52xx_spi_transfer; + host->mode_bits = SPI_CPOL | SPI_CPHA | SPI_LSB_FIRST; + host->bits_per_word_mask = SPI_BPW_MASK(8); + host->dev.of_node = op->dev.of_node; - platform_set_drvdata(op, master); + platform_set_drvdata(op, host); - ms = spi_master_get_devdata(master); - ms->master = master; + ms = spi_controller_get_devdata(host); + ms->host = host; ms->regs = regs; ms->irq0 = irq_of_parse_and_map(op->dev.of_node, 0); ms->irq1 = irq_of_parse_and_map(op->dev.of_node, 1); @@ -442,7 +442,7 @@ static int mpc52xx_spi_probe(struct platform_device *op) ms->ipb_freq = mpc5xxx_get_bus_frequency(&op->dev); ms->gpio_cs_count = gpiod_count(&op->dev, NULL); if (ms->gpio_cs_count > 0) { - master->num_chipselect = ms->gpio_cs_count; + host->num_chipselect = ms->gpio_cs_count; ms->gpio_cs = kmalloc_array(ms->gpio_cs_count, sizeof(*ms->gpio_cs), GFP_KERNEL); @@ -489,24 +489,27 @@ static int mpc52xx_spi_probe(struct platform_device *op) if (!ms->irq0) dev_info(&op->dev, "using polled mode\n"); - dev_dbg(&op->dev, "registering spi_master struct\n"); - rc = spi_register_master(master); + dev_dbg(&op->dev, "registering spi_controller struct\n"); + rc = spi_register_controller(host); if (rc) goto err_register; - dev_info(&ms->master->dev, "registered MPC5200 SPI bus\n"); + dev_info(&ms->host->dev, "registered MPC5200 SPI bus\n"); return rc; err_register: - dev_err(&ms->master->dev, "initialization failed\n"); + dev_err(&ms->host->dev, "initialization failed\n"); + free_irq(ms->irq0, ms); + free_irq(ms->irq1, ms); + cancel_work_sync(&ms->work); err_gpio: while (i-- > 0) gpiod_put(ms->gpio_cs[i]); kfree(ms->gpio_cs); err_alloc_gpio: - spi_master_put(master); + spi_controller_put(host); err_alloc: err_init: iounmap(regs); @@ -515,10 +518,12 @@ static int mpc52xx_spi_probe(struct platform_device *op) static void mpc52xx_spi_remove(struct platform_device *op) { - struct spi_master *master = spi_master_get(platform_get_drvdata(op)); - struct mpc52xx_spi *ms = spi_master_get_devdata(master); + struct spi_controller *host = spi_controller_get(platform_get_drvdata(op)); + struct mpc52xx_spi *ms = spi_controller_get_devdata(host); int i; + spi_unregister_controller(host); + free_irq(ms->irq0, ms); free_irq(ms->irq1, ms); @@ -528,9 +533,8 @@ static void mpc52xx_spi_remove(struct platform_device *op) gpiod_put(ms->gpio_cs[i]); kfree(ms->gpio_cs); - spi_unregister_master(master); iounmap(ms->regs); - spi_master_put(master); + spi_controller_put(host); } static const struct of_device_id mpc52xx_spi_match[] = { diff --git a/drivers/spi/spi-oc-tiny.c b/drivers/spi/spi-oc-tiny.c index 3af499838e84..7ed76a9e7273 100644 --- a/drivers/spi/spi-oc-tiny.c +++ b/drivers/spi/spi-oc-tiny.c @@ -213,11 +213,11 @@ static int tiny_spi_probe(struct platform_device *pdev) struct tiny_spi_platform_data *platp = dev_get_platdata(&pdev->dev); struct tiny_spi *hw; struct spi_master *master; - int err = -ENODEV; + int err; - master = spi_alloc_master(&pdev->dev, sizeof(struct tiny_spi)); + master = devm_spi_alloc_master(&pdev->dev, sizeof(struct tiny_spi)); if (!master) - return err; + return -ENOMEM; /* setup the master state. */ master->bus_num = pdev->id; @@ -235,10 +235,8 @@ static int tiny_spi_probe(struct platform_device *pdev) /* find and map our resources */ hw->base = devm_platform_ioremap_resource(pdev, 0); - if (IS_ERR(hw->base)) { - err = PTR_ERR(hw->base); - goto exit; - } + if (IS_ERR(hw->base)) + return PTR_ERR(hw->base); /* irq is optional */ hw->irq = platform_get_irq(pdev, 0); if (hw->irq >= 0) { @@ -246,7 +244,7 @@ static int tiny_spi_probe(struct platform_device *pdev) err = devm_request_irq(&pdev->dev, hw->irq, tiny_spi_irq, 0, pdev->name, hw); if (err) - goto exit; + return err; } /* find platform data */ if (platp) { @@ -255,29 +253,23 @@ static int tiny_spi_probe(struct platform_device *pdev) } else { err = tiny_spi_of_probe(pdev); if (err) - goto exit; + return err; } /* register our spi controller */ err = spi_bitbang_start(&hw->bitbang); if (err) - goto exit; + return err; dev_info(&pdev->dev, "base %p, irq %d\n", hw->base, hw->irq); return 0; - -exit: - spi_master_put(master); - return err; } static void tiny_spi_remove(struct platform_device *pdev) { struct tiny_spi *hw = platform_get_drvdata(pdev); - struct spi_master *master = hw->bitbang.master; spi_bitbang_stop(&hw->bitbang); - spi_master_put(master); } #ifdef CONFIG_OF diff --git a/drivers/spi/spi-tegra210-quad.c b/drivers/spi/spi-tegra210-quad.c index f7677b960c34..8af5892ac738 100644 --- a/drivers/spi/spi-tegra210-quad.c +++ b/drivers/spi/spi-tegra210-quad.c @@ -1351,6 +1351,11 @@ static irqreturn_t handle_cpu_based_xfer(struct tegra_qspi *tqspi) spin_lock_irqsave(&tqspi->lock, flags); t = tqspi->curr_xfer; + if (!t) { + spin_unlock_irqrestore(&tqspi->lock, flags); + return IRQ_HANDLED; + } + if (tqspi->tx_status || tqspi->rx_status) { tegra_qspi_handle_error(tqspi); complete(&tqspi->xfer_completion); @@ -1419,6 +1424,11 @@ static irqreturn_t handle_dma_based_xfer(struct tegra_qspi *tqspi) spin_lock_irqsave(&tqspi->lock, flags); t = tqspi->curr_xfer; + if (!t) { + spin_unlock_irqrestore(&tqspi->lock, flags); + return IRQ_HANDLED; + } + if (err) { tegra_qspi_dma_unmap_xfer(tqspi, t); tegra_qspi_handle_error(tqspi); @@ -1457,6 +1467,7 @@ static irqreturn_t handle_dma_based_xfer(struct tegra_qspi *tqspi) static irqreturn_t tegra_qspi_isr_thread(int irq, void *context_data) { struct tegra_qspi *tqspi = context_data; + unsigned long flags; u32 status; /* @@ -1474,7 +1485,9 @@ static irqreturn_t tegra_qspi_isr_thread(int irq, void *context_data) * If no transfer is in progress, check if this was a real interrupt * that the timeout handler already processed, or a spurious one. */ + spin_lock_irqsave(&tqspi->lock, flags); if (!tqspi->curr_xfer) { + spin_unlock_irqrestore(&tqspi->lock, flags); /* Spurious interrupt - transfer not ready */ if (!(status & QSPI_RDY)) return IRQ_NONE; @@ -1491,7 +1504,14 @@ static irqreturn_t tegra_qspi_isr_thread(int irq, void *context_data) tqspi->rx_status = tqspi->status_reg & (QSPI_RX_FIFO_OVF | QSPI_RX_FIFO_UNF); tegra_qspi_mask_clear_irq(tqspi); + spin_unlock_irqrestore(&tqspi->lock, flags); + /* + * Lock is released here but handlers safely re-check curr_xfer under + * lock before dereferencing. + * DMA handler also needs to sleep in wait_for_completion_*(), which + * cannot be done while holding spinlock. + */ if (!tqspi->is_curr_dma_xfer) return handle_cpu_based_xfer(tqspi); diff --git a/drivers/spi/spi.c b/drivers/spi/spi.c index 79e68b97dfec..9a011544d154 100644 --- a/drivers/spi/spi.c +++ b/drivers/spi/spi.c @@ -51,7 +51,6 @@ static void spidev_release(struct device *dev) struct spi_device *spi = to_spi_device(dev); spi_controller_put(spi->controller); - kfree(spi->driver_override); free_percpu(spi->pcpu_statistics); kfree(spi); } @@ -74,10 +73,9 @@ static ssize_t driver_override_store(struct device *dev, struct device_attribute *a, const char *buf, size_t count) { - struct spi_device *spi = to_spi_device(dev); int ret; - ret = driver_set_override(dev, &spi->driver_override, buf, count); + ret = __device_set_driver_override(dev, buf, count); if (ret) return ret; @@ -87,13 +85,8 @@ static ssize_t driver_override_store(struct device *dev, static ssize_t driver_override_show(struct device *dev, struct device_attribute *a, char *buf) { - const struct spi_device *spi = to_spi_device(dev); - ssize_t len; - - device_lock(dev); - len = sysfs_emit(buf, "%s\n", spi->driver_override ? : ""); - device_unlock(dev); - return len; + guard(spinlock)(&dev->driver_override.lock); + return sysfs_emit(buf, "%s\n", dev->driver_override.name ?: ""); } static DEVICE_ATTR_RW(driver_override); @@ -383,10 +376,12 @@ static int spi_match_device(struct device *dev, struct device_driver *drv) { const struct spi_device *spi = to_spi_device(dev); const struct spi_driver *sdrv = to_spi_driver(drv); + int ret; /* Check override first, and if set, only use the named driver */ - if (spi->driver_override) - return strcmp(spi->driver_override, drv->name) == 0; + ret = device_match_driver_override(dev, drv); + if (ret >= 0) + return ret; /* Attempt an OF style match */ if (of_driver_match_device(dev, drv)) diff --git a/drivers/staging/greybus/raw.c b/drivers/staging/greybus/raw.c index 8bca8cb12cc6..685ff55e5894 100644 --- a/drivers/staging/greybus/raw.c +++ b/drivers/staging/greybus/raw.c @@ -21,9 +21,10 @@ struct gb_raw { struct list_head list; int list_data; struct mutex list_lock; - dev_t dev; + struct rw_semaphore disconnect_lock; + bool disconnected; struct cdev cdev; - struct device *device; + struct device dev; }; struct raw_data { @@ -145,6 +146,15 @@ static int gb_raw_send(struct gb_raw *raw, u32 len, const char __user *data) return retval; } +static void raw_dev_release(struct device *dev) +{ + struct gb_raw *raw = container_of(dev, struct gb_raw, dev); + + ida_simple_remove(&minors, MINOR(raw->dev.devt)); + + kfree(raw); +} + static int gb_raw_probe(struct gb_bundle *bundle, const struct greybus_bundle_id *id) { @@ -161,63 +171,59 @@ static int gb_raw_probe(struct gb_bundle *bundle, if (cport_desc->protocol_id != GREYBUS_PROTOCOL_RAW) return -ENODEV; + minor = ida_simple_get(&minors, 0, 0, GFP_KERNEL); + if (minor < 0) + return minor; + raw = kzalloc(sizeof(*raw), GFP_KERNEL); - if (!raw) + if (!raw) { + ida_simple_remove(&minors, minor); return -ENOMEM; + } + + device_initialize(&raw->dev); + raw->dev.devt = MKDEV(raw_major, minor); + raw->dev.class = raw_class; + raw->dev.parent = &bundle->dev; + raw->dev.release = raw_dev_release; + retval = dev_set_name(&raw->dev, "gb!raw%d", minor); + if (retval) + goto error_put_device; connection = gb_connection_create(bundle, le16_to_cpu(cport_desc->id), gb_raw_request_handler); if (IS_ERR(connection)) { retval = PTR_ERR(connection); - goto error_free; + goto error_put_device; } INIT_LIST_HEAD(&raw->list); mutex_init(&raw->list_lock); + init_rwsem(&raw->disconnect_lock); raw->connection = connection; greybus_set_drvdata(bundle, raw); - minor = ida_simple_get(&minors, 0, 0, GFP_KERNEL); - if (minor < 0) { - retval = minor; - goto error_connection_destroy; - } - - raw->dev = MKDEV(raw_major, minor); cdev_init(&raw->cdev, &raw_fops); retval = gb_connection_enable(connection); if (retval) - goto error_remove_ida; + goto error_connection_destroy; - retval = cdev_add(&raw->cdev, raw->dev, 1); + retval = cdev_device_add(&raw->cdev, &raw->dev); if (retval) goto error_connection_disable; - raw->device = device_create(raw_class, &connection->bundle->dev, - raw->dev, raw, "gb!raw%d", minor); - if (IS_ERR(raw->device)) { - retval = PTR_ERR(raw->device); - goto error_del_cdev; - } - return 0; -error_del_cdev: - cdev_del(&raw->cdev); - error_connection_disable: gb_connection_disable(connection); -error_remove_ida: - ida_simple_remove(&minors, minor); - error_connection_destroy: gb_connection_destroy(connection); -error_free: - kfree(raw); +error_put_device: + put_device(&raw->dev); return retval; } @@ -228,11 +234,13 @@ static void gb_raw_disconnect(struct gb_bundle *bundle) struct raw_data *raw_data; struct raw_data *temp; - // FIXME - handle removing a connection when the char device node is open. - device_destroy(raw_class, raw->dev); - cdev_del(&raw->cdev); + cdev_device_del(&raw->cdev, &raw->dev); + + down_write(&raw->disconnect_lock); + raw->disconnected = true; + up_write(&raw->disconnect_lock); + gb_connection_disable(connection); - ida_simple_remove(&minors, MINOR(raw->dev)); gb_connection_destroy(connection); mutex_lock(&raw->list_lock); @@ -241,8 +249,7 @@ static void gb_raw_disconnect(struct gb_bundle *bundle) kfree(raw_data); } mutex_unlock(&raw->list_lock); - - kfree(raw); + put_device(&raw->dev); } /* @@ -275,11 +282,22 @@ static ssize_t raw_write(struct file *file, const char __user *buf, if (count > MAX_PACKET_SIZE) return -E2BIG; + down_read(&raw->disconnect_lock); + + if (raw->disconnected) { + retval = -ENODEV; + goto exit; + } + retval = gb_raw_send(raw, count, buf); if (retval) - return retval; + goto exit; + + retval = count; +exit: + up_read(&raw->disconnect_lock); - return count; + return retval; } static ssize_t raw_read(struct file *file, char __user *buf, size_t count, diff --git a/drivers/staging/greybus/uart.c b/drivers/staging/greybus/uart.c index 526573556cda..f067f26fefcb 100644 --- a/drivers/staging/greybus/uart.c +++ b/drivers/staging/greybus/uart.c @@ -954,7 +954,8 @@ static int gb_tty_init(void) gb_tty_driver->minor_start = 0; gb_tty_driver->type = TTY_DRIVER_TYPE_SERIAL; gb_tty_driver->subtype = SERIAL_TYPE_NORMAL; - gb_tty_driver->flags = TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV; + gb_tty_driver->flags = TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV | + TTY_DRIVER_RESET_SAVED_TERMIOS; gb_tty_driver->init_termios = tty_std_termios; gb_tty_driver->init_termios.c_cflag = B9600 | CS8 | CREAD | HUPCL | CLOCAL; diff --git a/drivers/staging/media/atomisp/i2c/atomisp-gc2235.c b/drivers/staging/media/atomisp/i2c/atomisp-gc2235.c index 9fa390fbc5f3..1130d514891f 100644 --- a/drivers/staging/media/atomisp/i2c/atomisp-gc2235.c +++ b/drivers/staging/media/atomisp/i2c/atomisp-gc2235.c @@ -813,7 +813,7 @@ static int gc2235_probe(struct i2c_client *client) ret = gc2235_s_config(&dev->sd, client->irq, gcpdev); if (ret) - goto out_free; + goto err_unregister_subdev; dev->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE; dev->pad.flags = MEDIA_PAD_FL_SOURCE; @@ -822,18 +822,16 @@ static int gc2235_probe(struct i2c_client *client) ret = v4l2_ctrl_handler_init(&dev->ctrl_handler, ARRAY_SIZE(gc2235_controls)); - if (ret) { - gc2235_remove(client); - return ret; - } + if (ret) + goto err_csi_cfg; for (i = 0; i < ARRAY_SIZE(gc2235_controls); i++) v4l2_ctrl_new_custom(&dev->ctrl_handler, &gc2235_controls[i], NULL); if (dev->ctrl_handler.error) { - gc2235_remove(client); - return dev->ctrl_handler.error; + ret = dev->ctrl_handler.error; + goto err_ctrl_handler; } /* Use same lock for controls as for everything else. */ @@ -842,14 +840,23 @@ static int gc2235_probe(struct i2c_client *client) ret = media_entity_pads_init(&dev->sd.entity, 1, &dev->pad); if (ret) - gc2235_remove(client); + goto err_ctrl_handler; + + ret = atomisp_register_i2c_module(&dev->sd, gcpdev, RAW_CAMERA); + if (ret) + goto err_media_cleanup; - return atomisp_register_i2c_module(&dev->sd, gcpdev, RAW_CAMERA); + return 0; -out_free: +err_media_cleanup: + media_entity_cleanup(&dev->sd.entity); +err_ctrl_handler: + v4l2_ctrl_handler_free(&dev->ctrl_handler); +err_csi_cfg: + dev->platform_data->csi_cfg(&dev->sd, 0); +err_unregister_subdev: v4l2_device_unregister_subdev(&dev->sd); kfree(dev); - return ret; } diff --git a/drivers/staging/media/av7110/av7110.h b/drivers/staging/media/av7110/av7110.h index 809d938ae166..f77766008b97 100644 --- a/drivers/staging/media/av7110/av7110.h +++ b/drivers/staging/media/av7110/av7110.h @@ -86,6 +86,8 @@ struct infrared { u32 ir_config; }; +#define MAX_CI_SLOTS 2 + /* place to store all the necessary device information */ struct av7110 { @@ -166,7 +168,7 @@ struct av7110 { /* CA */ - struct ca_slot_info ci_slot[2]; + struct ca_slot_info ci_slot[MAX_CI_SLOTS]; enum av7110_video_mode vidmode; struct dmxdev dmxdev; diff --git a/drivers/staging/media/av7110/av7110_ca.c b/drivers/staging/media/av7110/av7110_ca.c index c1338e074a3d..642af03fbac8 100644 --- a/drivers/staging/media/av7110/av7110_ca.c +++ b/drivers/staging/media/av7110/av7110_ca.c @@ -12,6 +12,7 @@ */ #include +#include #include #include #include @@ -26,23 +27,28 @@ void CI_handle(struct av7110 *av7110, u8 *data, u16 len) { + unsigned slot_num; + dprintk(8, "av7110:%p\n",av7110); if (len < 3) return; switch (data[0]) { case CI_MSG_CI_INFO: - if (data[2] != 1 && data[2] != 2) + if (data[2] != 1 && data[2] != MAX_CI_SLOTS) break; + + slot_num = array_index_nospec(data[2] - 1, MAX_CI_SLOTS); + switch (data[1]) { case 0: - av7110->ci_slot[data[2] - 1].flags = 0; + av7110->ci_slot[slot_num].flags = 0; break; case 1: - av7110->ci_slot[data[2] - 1].flags |= CA_CI_MODULE_PRESENT; + av7110->ci_slot[slot_num].flags |= CA_CI_MODULE_PRESENT; break; case 2: - av7110->ci_slot[data[2] - 1].flags |= CA_CI_MODULE_READY; + av7110->ci_slot[slot_num].flags |= CA_CI_MODULE_READY; break; } break; @@ -264,15 +270,19 @@ static int dvb_ca_ioctl(struct file *file, unsigned int cmd, void *parg) case CA_GET_SLOT_INFO: { struct ca_slot_info *info=(struct ca_slot_info *)parg; + unsigned int slot_num; if (info->num < 0 || info->num > 1) { mutex_unlock(&av7110->ioctl_mutex); return -EINVAL; } - av7110->ci_slot[info->num].num = info->num; - av7110->ci_slot[info->num].type = FW_CI_LL_SUPPORT(av7110->arm_app) ? - CA_CI_LINK : CA_CI; - memcpy(info, &av7110->ci_slot[info->num], sizeof(struct ca_slot_info)); + slot_num = array_index_nospec(info->num, MAX_CI_SLOTS); + + av7110->ci_slot[slot_num].num = info->num; + av7110->ci_slot[slot_num].type = FW_CI_LL_SUPPORT(av7110->arm_app) ? + CA_CI_LINK : CA_CI; + memcpy(info, &av7110->ci_slot[slot_num], + sizeof(struct ca_slot_info)); break; } diff --git a/drivers/staging/media/rkvdec/rkvdec.c b/drivers/staging/media/rkvdec/rkvdec.c index a1d941b0be00..85dcabe11bf9 100644 --- a/drivers/staging/media/rkvdec/rkvdec.c +++ b/drivers/staging/media/rkvdec/rkvdec.c @@ -1044,8 +1044,8 @@ static int rkvdec_probe(struct platform_device *pdev) vb2_dma_contig_set_max_seg_size(&pdev->dev, DMA_BIT_MASK(32)); irq = platform_get_irq(pdev, 0); - if (irq <= 0) - return -ENXIO; + if (irq < 0) + return irq; ret = devm_request_threaded_irq(&pdev->dev, irq, NULL, rkvdec_irq_handler, IRQF_ONESHOT, diff --git a/drivers/staging/rtl8723bs/core/rtw_mlme.c b/drivers/staging/rtl8723bs/core/rtw_mlme.c index 1d44f6a3db18..061634bafae5 100644 --- a/drivers/staging/rtl8723bs/core/rtw_mlme.c +++ b/drivers/staging/rtl8723bs/core/rtw_mlme.c @@ -22,7 +22,6 @@ int rtw_init_mlme_priv(struct adapter *padapter) pmlmepriv->pscanned = NULL; pmlmepriv->fw_state = WIFI_STATION_STATE; /* Must sync with rtw_wdev_alloc() */ - /* wdev->iftype = NL80211_IFTYPE_STATION */ pmlmepriv->cur_network.network.infrastructure_mode = Ndis802_11AutoUnknown; pmlmepriv->scan_mode = SCAN_ACTIVE;/* 1: active, 0: passive. Maybe someday we should rename this varable to "active_mode" (Jeff) */ @@ -109,32 +108,6 @@ void _rtw_free_mlme_priv(struct mlme_priv *pmlmepriv) } } -/* -struct wlan_network *_rtw_dequeue_network(struct __queue *queue) -{ - _irqL irqL; - - struct wlan_network *pnetwork; - - spin_lock_bh(&queue->lock); - - if (list_empty(&queue->queue)) - - pnetwork = NULL; - - else - { - pnetwork = container_of(get_next(&queue->queue), struct wlan_network, list); - - list_del_init(&(pnetwork->list)); - } - - spin_unlock_bh(&queue->lock); - - return pnetwork; -} -*/ - struct wlan_network *rtw_alloc_network(struct mlme_priv *pmlmepriv) { struct wlan_network *pnetwork; @@ -147,7 +120,7 @@ struct wlan_network *rtw_alloc_network(struct mlme_priv *pmlmepriv) pnetwork = NULL; goto exit; } - plist = get_next(&(free_queue->queue)); + plist = get_next(&free_queue->queue); pnetwork = container_of(plist, struct wlan_network, list); @@ -169,8 +142,7 @@ void _rtw_free_network(struct mlme_priv *pmlmepriv, struct wlan_network *pnetwor { unsigned int delta_time; u32 lifetime = SCANQUEUE_LIFETIME; -/* _irqL irqL; */ - struct __queue *free_queue = &(pmlmepriv->free_bss_pool); + struct __queue *free_queue = &pmlmepriv->free_bss_pool; if (!pnetwork) return; @@ -190,9 +162,9 @@ void _rtw_free_network(struct mlme_priv *pmlmepriv, struct wlan_network *pnetwor spin_lock_bh(&free_queue->lock); - list_del_init(&(pnetwork->list)); + list_del_init(&pnetwork->list); - list_add_tail(&(pnetwork->list), &(free_queue->queue)); + list_add_tail(&pnetwork->list, &free_queue->queue); spin_unlock_bh(&free_queue->lock); } @@ -200,7 +172,7 @@ void _rtw_free_network(struct mlme_priv *pmlmepriv, struct wlan_network *pnetwor void _rtw_free_network_nolock(struct mlme_priv *pmlmepriv, struct wlan_network *pnetwork) { - struct __queue *free_queue = &(pmlmepriv->free_bss_pool); + struct __queue *free_queue = &pmlmepriv->free_bss_pool; if (!pnetwork) return; @@ -208,13 +180,9 @@ void _rtw_free_network_nolock(struct mlme_priv *pmlmepriv, struct wlan_network * if (pnetwork->fixed) return; - /* spin_lock_irqsave(&free_queue->lock, irqL); */ - - list_del_init(&(pnetwork->list)); - - list_add_tail(&(pnetwork->list), get_list_head(free_queue)); + list_del_init(&pnetwork->list); - /* spin_unlock_irqrestore(&free_queue->lock, irqL); */ + list_add_tail(&pnetwork->list, get_list_head(free_queue)); } /* @@ -232,8 +200,6 @@ struct wlan_network *_rtw_find_network(struct __queue *scanned_queue, u8 *addr) goto exit; } - /* spin_lock_bh(&scanned_queue->lock); */ - phead = get_list_head(scanned_queue); list_for_each(plist, phead) { pnetwork = list_entry(plist, struct wlan_network, list); @@ -245,8 +211,6 @@ struct wlan_network *_rtw_find_network(struct __queue *scanned_queue, u8 *addr) if (plist == phead) pnetwork = NULL; - /* spin_unlock_bh(&scanned_queue->lock); */ - exit: return pnetwork; } @@ -321,20 +285,10 @@ void rtw_free_mlme_priv(struct mlme_priv *pmlmepriv) _rtw_free_mlme_priv(pmlmepriv); } -/* -static struct wlan_network *rtw_dequeue_network(struct __queue *queue) -{ - struct wlan_network *pnetwork; - - pnetwork = _rtw_dequeue_network(queue); - return pnetwork; -} -*/ - void rtw_free_network_nolock(struct adapter *padapter, struct wlan_network *pnetwork); void rtw_free_network_nolock(struct adapter *padapter, struct wlan_network *pnetwork) { - _rtw_free_network_nolock(&(padapter->mlmepriv), pnetwork); + _rtw_free_network_nolock(&padapter->mlmepriv, pnetwork); rtw_cfg80211_unlink_bss(padapter, pnetwork); } @@ -389,7 +343,6 @@ int is_same_network(struct wlan_bssid_ex *src, struct wlan_bssid_ex *dst, u8 fea d_cap = le16_to_cpu(tmpd); return (src->ssid.ssid_length == dst->ssid.ssid_length) && - /* (src->configuration.ds_config == dst->configuration.ds_config) && */ ((!memcmp(src->mac_address, dst->mac_address, ETH_ALEN))) && ((!memcmp(src->ssid.ssid, dst->ssid.ssid, src->ssid.ssid_length))) && ((s_cap & WLAN_CAPABILITY_IBSS) == @@ -452,7 +405,7 @@ void update_network(struct wlan_bssid_ex *dst, struct wlan_bssid_ex *src, long rssi_final; /* The rule below is 1/5 for sample value, 4/5 for history value */ - if (check_fwstate(&padapter->mlmepriv, _FW_LINKED) && is_same_network(&(padapter->mlmepriv.cur_network.network), src, 0)) { + if (check_fwstate(&padapter->mlmepriv, _FW_LINKED) && is_same_network(&padapter->mlmepriv.cur_network.network, src, 0)) { /* Take the recvpriv's value for the connected AP*/ ss_final = padapter->recvpriv.signal_strength; sq_final = padapter->recvpriv.signal_qual; @@ -488,20 +441,20 @@ void update_network(struct wlan_bssid_ex *dst, struct wlan_bssid_ex *src, static void update_current_network(struct adapter *adapter, struct wlan_bssid_ex *pnetwork) { - struct mlme_priv *pmlmepriv = &(adapter->mlmepriv); + struct mlme_priv *pmlmepriv = &adapter->mlmepriv; - rtw_bug_check(&(pmlmepriv->cur_network.network), - &(pmlmepriv->cur_network.network), - &(pmlmepriv->cur_network.network), - &(pmlmepriv->cur_network.network)); + rtw_bug_check(&pmlmepriv->cur_network.network, + &pmlmepriv->cur_network.network, + &pmlmepriv->cur_network.network, + &pmlmepriv->cur_network.network); - if ((check_fwstate(pmlmepriv, _FW_LINKED) == true) && (is_same_network(&(pmlmepriv->cur_network.network), pnetwork, 0))) { - /* if (pmlmepriv->cur_network.network.ie_length<= pnetwork->ie_length) */ - { - update_network(&(pmlmepriv->cur_network.network), pnetwork, adapter, true); - rtw_update_protection(adapter, (pmlmepriv->cur_network.network.ies) + sizeof(struct ndis_802_11_fix_ie), - pmlmepriv->cur_network.network.ie_length); - } + if ((check_fwstate(pmlmepriv, _FW_LINKED) == true) && (is_same_network(&pmlmepriv->cur_network.network, pnetwork, 0))) { + update_network(&pmlmepriv->cur_network.network, pnetwork, adapter, true); + if (pmlmepriv->cur_network.network.ie_length < sizeof(struct ndis_802_11_fix_ie)) + return; + + rtw_update_protection(adapter, (pmlmepriv->cur_network.network.ies) + sizeof(struct ndis_802_11_fix_ie), + pmlmepriv->cur_network.network.ie_length - sizeof(struct ndis_802_11_fix_ie)); } } @@ -512,8 +465,8 @@ void rtw_update_scanned_network(struct adapter *adapter, struct wlan_bssid_ex *t { struct list_head *plist, *phead; u32 bssid_ex_sz; - struct mlme_priv *pmlmepriv = &(adapter->mlmepriv); - struct __queue *queue = &(pmlmepriv->scanned_queue); + struct mlme_priv *pmlmepriv = &adapter->mlmepriv; + struct __queue *queue = &pmlmepriv->scanned_queue; struct wlan_network *pnetwork = NULL; struct wlan_network *oldest = NULL; int target_find = 0; @@ -526,7 +479,7 @@ void rtw_update_scanned_network(struct adapter *adapter, struct wlan_bssid_ex *t rtw_bug_check(pnetwork, pnetwork, pnetwork, pnetwork); - if (is_same_network(&(pnetwork->network), target, feature)) { + if (is_same_network(&pnetwork->network, target, feature)) { target_find = 1; break; } @@ -542,7 +495,6 @@ void rtw_update_scanned_network(struct adapter *adapter, struct wlan_bssid_ex *t /* If we didn't find a match, then get a new network slot to initialize * with this beacon's information */ - /* if (phead == plist) { */ if (!target_find) { if (list_empty(&pmlmepriv->free_bss_pool.queue)) { /* If there are no more slots, expire the oldest */ @@ -551,7 +503,7 @@ void rtw_update_scanned_network(struct adapter *adapter, struct wlan_bssid_ex *t if (!pnetwork) goto exit; - memcpy(&(pnetwork->network), target, get_wlan_bssid_ex_sz(target)); + memcpy(&pnetwork->network, target, get_wlan_bssid_ex_sz(target)); /* variable initialize */ pnetwork->fixed = false; pnetwork->last_scanned = jiffies; @@ -573,7 +525,7 @@ void rtw_update_scanned_network(struct adapter *adapter, struct wlan_bssid_ex *t bssid_ex_sz = get_wlan_bssid_ex_sz(target); target->length = bssid_ex_sz; - memcpy(&(pnetwork->network), target, bssid_ex_sz); + memcpy(&pnetwork->network, target, bssid_ex_sz); pnetwork->last_scanned = jiffies; @@ -581,7 +533,7 @@ void rtw_update_scanned_network(struct adapter *adapter, struct wlan_bssid_ex *t if (pnetwork->network.phy_info.signal_quality == 101) pnetwork->network.phy_info.signal_quality = 0; - list_add_tail(&(pnetwork->list), &(queue->queue)); + list_add_tail(&pnetwork->list, &queue->queue); } } else { @@ -605,7 +557,7 @@ void rtw_update_scanned_network(struct adapter *adapter, struct wlan_bssid_ex *t update_ie = false; } - update_network(&(pnetwork->network), target, adapter, update_ie); + update_network(&pnetwork->network, target, adapter, update_ie); } exit: @@ -615,15 +567,8 @@ void rtw_update_scanned_network(struct adapter *adapter, struct wlan_bssid_ex *t void rtw_add_network(struct adapter *adapter, struct wlan_bssid_ex *pnetwork); void rtw_add_network(struct adapter *adapter, struct wlan_bssid_ex *pnetwork) { - /* struct __queue *queue = &(pmlmepriv->scanned_queue); */ - - /* spin_lock_bh(&queue->lock); */ - update_current_network(adapter, pnetwork); - rtw_update_scanned_network(adapter, pnetwork); - - /* spin_unlock_bh(&queue->lock); */ } /* select the desired network based on the capability of the (i)bss. */ @@ -639,16 +584,15 @@ int rtw_is_desired_network(struct adapter *adapter, struct wlan_network *pnetwor struct mlme_priv *pmlmepriv = &adapter->mlmepriv; u32 desired_encmode; u32 privacy; - - /* u8 wps_ie[512]; */ uint wps_ielen; - int bselected = true; desired_encmode = psecuritypriv->ndisencryptstatus; privacy = pnetwork->network.privacy; if (check_fwstate(pmlmepriv, WIFI_UNDER_WPS)) { + if (pnetwork->network.ie_length < _FIXED_IE_LENGTH_) + return false; if (rtw_get_wps_ie(pnetwork->network.ies+_FIXED_IE_LENGTH_, pnetwork->network.ie_length-_FIXED_IE_LENGTH_, NULL, &wps_ielen)) return true; else @@ -663,11 +607,15 @@ int rtw_is_desired_network(struct adapter *adapter, struct wlan_network *pnetwor bselected = false; if (psecuritypriv->ndisauthtype == Ndis802_11AuthModeWPA2PSK) { - p = rtw_get_ie(pnetwork->network.ies + _BEACON_IE_OFFSET_, WLAN_EID_RSN, &ie_len, (pnetwork->network.ie_length - _BEACON_IE_OFFSET_)); - if (p && ie_len > 0) - bselected = true; - else + if (pnetwork->network.ie_length < _BEACON_IE_OFFSET_) { bselected = false; + } else { + p = rtw_get_ie(pnetwork->network.ies + _BEACON_IE_OFFSET_, WLAN_EID_RSN, &ie_len, (pnetwork->network.ie_length - _BEACON_IE_OFFSET_)); + if (p && ie_len > 0) + bselected = true; + else + bselected = false; + } } } @@ -691,7 +639,7 @@ void rtw_survey_event_callback(struct adapter *adapter, u8 *pbuf) { u32 len; struct wlan_bssid_ex *pnetwork; - struct mlme_priv *pmlmepriv = &(adapter->mlmepriv); + struct mlme_priv *pmlmepriv = &adapter->mlmepriv; pnetwork = (struct wlan_bssid_ex *)pbuf; @@ -703,18 +651,18 @@ void rtw_survey_event_callback(struct adapter *adapter, u8 *pbuf) /* update IBSS_network 's timestamp */ if ((check_fwstate(pmlmepriv, WIFI_ADHOC_MASTER_STATE)) == true) { - if (!memcmp(&(pmlmepriv->cur_network.network.mac_address), pnetwork->mac_address, ETH_ALEN)) { + if (!memcmp(&pmlmepriv->cur_network.network.mac_address, pnetwork->mac_address, ETH_ALEN)) { struct wlan_network *ibss_wlan = NULL; memcpy(pmlmepriv->cur_network.network.ies, pnetwork->ies, 8); - spin_lock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_lock_bh(&pmlmepriv->scanned_queue.lock); ibss_wlan = rtw_find_network(&pmlmepriv->scanned_queue, pnetwork->mac_address); if (ibss_wlan) { memcpy(ibss_wlan->network.ies, pnetwork->ies, 8); - spin_unlock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_unlock_bh(&pmlmepriv->scanned_queue.lock); goto exit; } - spin_unlock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_unlock_bh(&pmlmepriv->scanned_queue.lock); } } @@ -732,7 +680,7 @@ void rtw_survey_event_callback(struct adapter *adapter, u8 *pbuf) void rtw_surveydone_event_callback(struct adapter *adapter, u8 *pbuf) { - struct mlme_priv *pmlmepriv = &(adapter->mlmepriv); + struct mlme_priv *pmlmepriv = &adapter->mlmepriv; spin_lock_bh(&pmlmepriv->lock); if (pmlmepriv->wps_probe_req_ie) { @@ -759,7 +707,7 @@ void rtw_surveydone_event_callback(struct adapter *adapter, u8 *pbuf) _set_timer(&pmlmepriv->assoc_timer, MAX_JOIN_TIMEOUT); } else { u8 ret = _SUCCESS; - struct wlan_bssid_ex *pdev_network = &(adapter->registrypriv.dev_network); + struct wlan_bssid_ex *pdev_network = &adapter->registrypriv.dev_network; u8 *pibss = adapter->registrypriv.dev_network.mac_address; /* pmlmepriv->fw_state ^= _FW_UNDER_SURVEY;because don't set assoc_timer */ @@ -987,8 +935,8 @@ inline void rtw_indicate_scan_done(struct adapter *padapter, bool aborted) void rtw_scan_abort(struct adapter *adapter) { unsigned long start; - struct mlme_priv *pmlmepriv = &(adapter->mlmepriv); - struct mlme_ext_priv *pmlmeext = &(adapter->mlmeextpriv); + struct mlme_priv *pmlmepriv = &adapter->mlmepriv; + struct mlme_ext_priv *pmlmeext = &adapter->mlmeextpriv; start = jiffies; pmlmeext->scan_abort = true; @@ -1056,9 +1004,8 @@ static struct sta_info *rtw_joinbss_update_stainfo(struct adapter *padapter, str memset((u8 *)&psta->dot11rxpn, 0, sizeof(union pn48)); } - /* Commented by Albert 2012/07/21 */ - /* When doing the WPS, the wps_ie_len won't equal to 0 */ - /* And the Wi-Fi driver shouldn't allow the data packet to be transmitted. */ + /* When doing the WPS, the wps_ie_len won't equal to 0 */ + /* And the Wi-Fi driver shouldn't allow the data packet to be transmitted. */ if (padapter->securitypriv.wps_ie_len != 0) { psta->ieee8021x_blocked = true; padapter->securitypriv.wps_ie_len = 0; @@ -1068,7 +1015,6 @@ static struct sta_info *rtw_joinbss_update_stainfo(struct adapter *padapter, str /* if A-MPDU Rx is enabled, resetting rx_ordering_ctrl wstart_b(indicate_seq) to default value = 0xffff */ /* todo: check if AP can send A-MPDU packets */ for (i = 0; i < 16 ; i++) { - /* preorder_ctrl = &precvpriv->recvreorder_ctrl[i]; */ preorder_ctrl = &psta->recvreorder_ctrl[i]; preorder_ctrl->enable = false; preorder_ctrl->indicate_seq = 0xffff; @@ -1079,7 +1025,6 @@ static struct sta_info *rtw_joinbss_update_stainfo(struct adapter *padapter, str bmc_sta = rtw_get_bcmc_stainfo(padapter); if (bmc_sta) { for (i = 0; i < 16 ; i++) { - /* preorder_ctrl = &precvpriv->recvreorder_ctrl[i]; */ preorder_ctrl = &bmc_sta->recvreorder_ctrl[i]; preorder_ctrl->enable = false; preorder_ctrl->indicate_seq = 0xffff; @@ -1097,8 +1042,8 @@ static struct sta_info *rtw_joinbss_update_stainfo(struct adapter *padapter, str /* ptarget_wlan: found from scanned_queue */ static void rtw_joinbss_update_network(struct adapter *padapter, struct wlan_network *ptarget_wlan, struct wlan_network *pnetwork) { - struct mlme_priv *pmlmepriv = &(padapter->mlmepriv); - struct wlan_network *cur_network = &(pmlmepriv->cur_network); + struct mlme_priv *pmlmepriv = &padapter->mlmepriv; + struct wlan_network *cur_network = &pmlmepriv->cur_network; /* why not use ptarget_wlan?? */ memcpy(&cur_network->network, &pnetwork->network, pnetwork->network.length); @@ -1135,8 +1080,11 @@ static void rtw_joinbss_update_network(struct adapter *padapter, struct wlan_net break; } + if (cur_network->network.ie_length < sizeof(struct ndis_802_11_fix_ie)) + return; + rtw_update_protection(padapter, (cur_network->network.ies) + sizeof(struct ndis_802_11_fix_ie), - (cur_network->network.ie_length)); + (cur_network->network.ie_length - sizeof(struct ndis_802_11_fix_ie))); rtw_update_ht_cap(padapter, cur_network->network.ies, cur_network->network.ie_length, (u8) cur_network->network.configuration.ds_config); } @@ -1154,9 +1102,9 @@ void rtw_joinbss_event_prehandle(struct adapter *adapter, u8 *pbuf) static u8 __maybe_unused retry; struct sta_info *ptarget_sta = NULL, *pcur_sta = NULL; struct sta_priv *pstapriv = &adapter->stapriv; - struct mlme_priv *pmlmepriv = &(adapter->mlmepriv); + struct mlme_priv *pmlmepriv = &adapter->mlmepriv; struct wlan_network *pnetwork = (struct wlan_network *)pbuf; - struct wlan_network *cur_network = &(pmlmepriv->cur_network); + struct wlan_network *cur_network = &pmlmepriv->cur_network; struct wlan_network *pcur_wlan = NULL, *ptarget_wlan = NULL; unsigned int the_same_macaddr = false; @@ -1174,7 +1122,7 @@ void rtw_joinbss_event_prehandle(struct adapter *adapter, u8 *pbuf) pmlmepriv->LinkDetectInfo.LowPowerTransitionCount = 0; if (pnetwork->join_res > 0) { - spin_lock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_lock_bh(&pmlmepriv->scanned_queue.lock); retry = 0; if (check_fwstate(pmlmepriv, _FW_UNDER_LINKING)) { /* s1. find ptarget_wlan */ @@ -1211,7 +1159,7 @@ void rtw_joinbss_event_prehandle(struct adapter *adapter, u8 *pbuf) } else { netdev_dbg(adapter->pnetdev, "Can't find ptarget_wlan when joinbss_event callback\n"); - spin_unlock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_unlock_bh(&pmlmepriv->scanned_queue.lock); goto ignore_joinbss_callback; } @@ -1219,7 +1167,7 @@ void rtw_joinbss_event_prehandle(struct adapter *adapter, u8 *pbuf) if (check_fwstate(pmlmepriv, WIFI_STATION_STATE) == true) { ptarget_sta = rtw_joinbss_update_stainfo(adapter, pnetwork); if (!ptarget_sta) { - spin_unlock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_unlock_bh(&pmlmepriv->scanned_queue.lock); goto ignore_joinbss_callback; } } @@ -1237,14 +1185,12 @@ void rtw_joinbss_event_prehandle(struct adapter *adapter, u8 *pbuf) del_timer_sync(&pmlmepriv->assoc_timer); spin_lock_bh(&pmlmepriv->lock); } else { - spin_unlock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_unlock_bh(&pmlmepriv->scanned_queue.lock); } } else if (pnetwork->join_res == -4) { rtw_reset_securitypriv(adapter); _set_timer(&pmlmepriv->assoc_timer, 1); - /* rtw_free_assoc_resources(adapter, 1); */ - if ((check_fwstate(pmlmepriv, _FW_UNDER_LINKING)) == true) _clr_fwstate_(pmlmepriv, _FW_UNDER_LINKING); @@ -1266,7 +1212,6 @@ void rtw_joinbss_event_prehandle(struct adapter *adapter, u8 *pbuf) #endif _set_timer(&pmlmepriv->assoc_timer, 1); - /* rtw_free_assoc_resources(adapter, 1); */ _clr_fwstate_(pmlmepriv, _FW_UNDER_LINKING); #ifdef REJOIN @@ -1304,9 +1249,9 @@ void rtw_sta_media_status_rpt(struct adapter *adapter, struct sta_info *psta, u3 void rtw_stassoc_event_callback(struct adapter *adapter, u8 *pbuf) { struct sta_info *psta; - struct mlme_priv *pmlmepriv = &(adapter->mlmepriv); + struct mlme_priv *pmlmepriv = &adapter->mlmepriv; struct stassoc_event *pstassoc = (struct stassoc_event *)pbuf; - struct wlan_network *cur_network = &(pmlmepriv->cur_network); + struct wlan_network *cur_network = &pmlmepriv->cur_network; struct wlan_network *ptarget_wlan = NULL; if (rtw_access_ctrl(adapter, pstassoc->macaddr) == false) @@ -1361,7 +1306,6 @@ void rtw_stassoc_event_callback(struct adapter *adapter, u8 *pbuf) /* to do : init sta_info variable */ psta->qos_option = 0; psta->mac_id = (uint)pstassoc->cam_id; - /* psta->aid = (uint)pstassoc->cam_id; */ /* for ad-hoc mode */ rtw_hal_set_odm_var(adapter, HAL_ODM_STA_INFO, psta, true); @@ -1378,12 +1322,12 @@ void rtw_stassoc_event_callback(struct adapter *adapter, u8 *pbuf) if ((check_fwstate(pmlmepriv, WIFI_ADHOC_MASTER_STATE) == true) || (check_fwstate(pmlmepriv, WIFI_ADHOC_STATE) == true)) { if (adapter->stapriv.asoc_sta_count == 2) { - spin_lock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_lock_bh(&pmlmepriv->scanned_queue.lock); ptarget_wlan = rtw_find_network(&pmlmepriv->scanned_queue, cur_network->network.mac_address); pmlmepriv->cur_network_scanned = ptarget_wlan; if (ptarget_wlan) ptarget_wlan->fixed = true; - spin_unlock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_unlock_bh(&pmlmepriv->scanned_queue.lock); /* a sta + bc/mc_stainfo (not Ibss_stainfo) */ rtw_indicate_connect(adapter); } @@ -1401,11 +1345,11 @@ void rtw_stadel_event_callback(struct adapter *adapter, u8 *pbuf) struct wlan_network *pwlan = NULL; struct wlan_bssid_ex *pdev_network = NULL; u8 *pibss = NULL; - struct mlme_priv *pmlmepriv = &(adapter->mlmepriv); + struct mlme_priv *pmlmepriv = &adapter->mlmepriv; struct stadel_event *pstadel = (struct stadel_event *)pbuf; - struct wlan_network *tgt_network = &(pmlmepriv->cur_network); + struct wlan_network *tgt_network = &pmlmepriv->cur_network; struct mlme_ext_priv *pmlmeext = &adapter->mlmeextpriv; - struct mlme_ext_info *pmlmeinfo = &(pmlmeext->mlmext_info); + struct mlme_ext_info *pmlmeinfo = &pmlmeext->mlmext_info; psta = rtw_get_stainfo(&adapter->stapriv, pstadel->macaddr); if (psta) @@ -1457,14 +1401,14 @@ void rtw_stadel_event_callback(struct adapter *adapter, u8 *pbuf) rtw_free_assoc_resources(adapter, 1); rtw_indicate_disconnect(adapter); - spin_lock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_lock_bh(&pmlmepriv->scanned_queue.lock); /* remove the network entry in scanned_queue */ pwlan = rtw_find_network(&pmlmepriv->scanned_queue, tgt_network->network.mac_address); if (pwlan) { pwlan->fixed = false; rtw_free_network_nolock(adapter, pwlan); } - spin_unlock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_unlock_bh(&pmlmepriv->scanned_queue.lock); _rtw_roaming(adapter, roam_target); } @@ -1476,18 +1420,16 @@ void rtw_stadel_event_callback(struct adapter *adapter, u8 *pbuf) if (adapter->stapriv.asoc_sta_count == 1) {/* a sta + bc/mc_stainfo (not Ibss_stainfo) */ u8 ret = _SUCCESS; - /* rtw_indicate_disconnect(adapter);removed@20091105 */ - spin_lock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_lock_bh(&pmlmepriv->scanned_queue.lock); /* free old ibss network */ - /* pwlan = rtw_find_network(&pmlmepriv->scanned_queue, pstadel->macaddr); */ pwlan = rtw_find_network(&pmlmepriv->scanned_queue, tgt_network->network.mac_address); if (pwlan) { pwlan->fixed = false; rtw_free_network_nolock(adapter, pwlan); } - spin_unlock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_unlock_bh(&pmlmepriv->scanned_queue.lock); /* re-create ibss */ - pdev_network = &(adapter->registrypriv.dev_network); + pdev_network = &adapter->registrypriv.dev_network; pibss = adapter->registrypriv.dev_network.mac_address; memcpy(pdev_network, &tgt_network->network, get_wlan_bssid_ex_sz(&tgt_network->network)); @@ -1595,7 +1537,7 @@ void rtw_mlme_reset_auto_scan_int(struct adapter *adapter) { struct mlme_priv *mlme = &adapter->mlmepriv; struct mlme_ext_priv *pmlmeext = &adapter->mlmeextpriv; - struct mlme_ext_info *pmlmeinfo = &(pmlmeext->mlmext_info); + struct mlme_ext_info *pmlmeinfo = &pmlmeext->mlmext_info; if (pmlmeinfo->VHT_enable) /* disable auto scan when connect to 11AC AP */ mlme->auto_scan_int_ms = 0; @@ -1737,7 +1679,7 @@ int rtw_select_roaming_candidate(struct mlme_priv *mlme) { int ret = _FAIL; struct list_head *phead; - struct __queue *queue = &(mlme->scanned_queue); + struct __queue *queue = &mlme->scanned_queue; struct wlan_network *pnetwork = NULL; struct wlan_network *candidate = NULL; @@ -1746,7 +1688,7 @@ int rtw_select_roaming_candidate(struct mlme_priv *mlme) return ret; } - spin_lock_bh(&(mlme->scanned_queue.lock)); + spin_lock_bh(&mlme->scanned_queue.lock); phead = get_list_head(queue); list_for_each(mlme->pscanned, phead) { @@ -1770,7 +1712,7 @@ int rtw_select_roaming_candidate(struct mlme_priv *mlme) ret = _SUCCESS; exit: - spin_unlock_bh(&(mlme->scanned_queue.lock)); + spin_unlock_bh(&mlme->scanned_queue.lock); return ret; } @@ -1831,13 +1773,13 @@ int rtw_select_and_join_from_scanned_queue(struct mlme_priv *pmlmepriv) int ret; struct list_head *phead; struct adapter *adapter; - struct __queue *queue = &(pmlmepriv->scanned_queue); + struct __queue *queue = &pmlmepriv->scanned_queue; struct wlan_network *pnetwork = NULL; struct wlan_network *candidate = NULL; adapter = (struct adapter *)pmlmepriv->nic_hdl; - spin_lock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_lock_bh(&pmlmepriv->scanned_queue.lock); if (pmlmepriv->roam_network) { candidate = pmlmepriv->roam_network; @@ -1875,7 +1817,7 @@ int rtw_select_and_join_from_scanned_queue(struct mlme_priv *pmlmepriv) ret = rtw_joinbss_cmd(adapter, candidate); exit: - spin_unlock_bh(&(pmlmepriv->scanned_queue.lock)); + spin_unlock_bh(&pmlmepriv->scanned_queue.lock); return ret; } @@ -1883,7 +1825,7 @@ signed int rtw_set_auth(struct adapter *adapter, struct security_priv *psecurity { struct cmd_obj *pcmd; struct setauth_parm *psetauthparm; - struct cmd_priv *pcmdpriv = &(adapter->cmdpriv); + struct cmd_priv *pcmdpriv = &adapter->cmdpriv; signed int res = _SUCCESS; pcmd = rtw_zmalloc(sizeof(struct cmd_obj)); @@ -1920,7 +1862,7 @@ signed int rtw_set_key(struct adapter *adapter, struct security_priv *psecurityp u8 keylen; struct cmd_obj *pcmd; struct setkey_parm *psetkeyparm; - struct cmd_priv *pcmdpriv = &(adapter->cmdpriv); + struct cmd_priv *pcmdpriv = &adapter->cmdpriv; signed int res = _SUCCESS; psetkeyparm = rtw_zmalloc(sizeof(struct setkey_parm)); @@ -1943,11 +1885,11 @@ signed int rtw_set_key(struct adapter *adapter, struct security_priv *psecurityp case _WEP40_: keylen = 5; - memcpy(&(psetkeyparm->key[0]), &(psecuritypriv->dot11DefKey[keyid].skey[0]), keylen); + memcpy(&psetkeyparm->key[0], &psecuritypriv->dot11DefKey[keyid].skey[0], keylen); break; case _WEP104_: keylen = 13; - memcpy(&(psetkeyparm->key[0]), &(psecuritypriv->dot11DefKey[keyid].skey[0]), keylen); + memcpy(&psetkeyparm->key[0], &psecuritypriv->dot11DefKey[keyid].skey[0], keylen); break; case _TKIP_: keylen = 16; @@ -2015,7 +1957,10 @@ int rtw_restruct_wmm_ie(struct adapter *adapter, u8 *in_ie, u8 *out_ie, uint in_ break; } - i += (in_ie[i+1]+2); /* to the next IE element */ + if (i + 1 >= in_len) + break; + + i += (in_ie[i + 1] + 2); /* to the next IE element */ } return ielength; @@ -2095,14 +2040,6 @@ signed int rtw_restruct_sec_ie(struct adapter *adapter, u8 *in_ie, u8 *out_ie, u } else if ((authmode == WLAN_EID_VENDOR_SPECIFIC) || (authmode == WLAN_EID_RSN)) { /* copy RSN or SSN */ memcpy(&out_ie[ielength], &psecuritypriv->supplicant_ie[0], psecuritypriv->supplicant_ie[1]+2); - /* debug for CONFIG_IEEE80211W - { - int jj; - printk("supplicant_ie_length =%d &&&&&&&&&&&&&&&&&&&\n", psecuritypriv->supplicant_ie[1]+2); - for (jj = 0; jj < psecuritypriv->supplicant_ie[1]+2; jj++) - printk(" %02x ", psecuritypriv->supplicant_ie[jj]); - printk("\n"); - }*/ ielength += psecuritypriv->supplicant_ie[1]+2; rtw_report_sec_ie(adapter, authmode, psecuritypriv->supplicant_ie); } @@ -2139,7 +2076,6 @@ void rtw_update_registrypriv_dev_network(struct adapter *adapter) struct wlan_bssid_ex *pdev_network = &pregistrypriv->dev_network; struct security_priv *psecuritypriv = &adapter->securitypriv; struct wlan_network *cur_network = &adapter->mlmepriv.cur_network; - /* struct xmit_priv *pxmitpriv = &adapter->xmitpriv; */ pdev_network->privacy = (psecuritypriv->dot11PrivacyAlgrthm > 0 ? 1 : 0) ; /* adhoc no 802.1x */ @@ -2388,7 +2324,6 @@ unsigned int rtw_restructure_ht_ie(struct adapter *padapter, u8 *in_ie, u8 *out_ rtw_hal_get_def_var(padapter, HW_VAR_MAX_RX_AMPDU_FACTOR, &max_rx_ampdu_factor); - /* rtw_hal_get_def_var(padapter, HW_VAR_MAX_RX_AMPDU_FACTOR, &max_rx_ampdu_factor); */ ht_capie.ampdu_params_info = (max_rx_ampdu_factor&0x03); if (padapter->securitypriv.dot11PrivacyAlgrthm == _AES_) @@ -2418,16 +2353,12 @@ void rtw_update_ht_cap(struct adapter *padapter, u8 *pie, uint ie_len, u8 channe { u8 *p, max_ampdu_sz; int len; - /* struct sta_info *bmc_sta, *psta; */ struct ieee80211_ht_cap *pht_capie; - /* struct recv_reorder_ctrl *preorder_ctrl; */ struct mlme_priv *pmlmepriv = &padapter->mlmepriv; struct ht_priv *phtpriv = &pmlmepriv->htpriv; - /* struct recv_priv *precvpriv = &padapter->recvpriv; */ struct registry_priv *pregistrypriv = &padapter->registrypriv; - /* struct wlan_network *pcur_network = &(pmlmepriv->cur_network);; */ struct mlme_ext_priv *pmlmeext = &padapter->mlmeextpriv; - struct mlme_ext_info *pmlmeinfo = &(pmlmeext->mlmext_info); + struct mlme_ext_info *pmlmeinfo = &pmlmeext->mlmext_info; u8 cbw40_enable = 0; if (!phtpriv->ht_option) diff --git a/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c b/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c index 98b46182afd6..9125b6fed82b 100644 --- a/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c +++ b/drivers/staging/rtl8723bs/core/rtw_mlme_ext.c @@ -2259,8 +2259,11 @@ void issue_beacon(struct adapter *padapter, int timeout_ms) spin_unlock_bh(&pmlmepriv->bcn_update_lock); - if ((pattrib->pktlen + TXDESC_SIZE) > 512) + if ((pattrib->pktlen + TXDESC_SIZE) > 512) { + rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf); + rtw_free_xmitframe(pxmitpriv, pmgntframe); return; + } pattrib->last_txcmdsz = pattrib->pktlen; @@ -2323,8 +2326,11 @@ void issue_probersp(struct adapter *padapter, unsigned char *da, u8 is_valid_p2p pframe += pattrib->hdrlen; - if (cur_network->ie_length > MAX_IE_SZ) + if (cur_network->ie_length > MAX_IE_SZ) { + rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf); + rtw_free_xmitframe(pxmitpriv, pmgntframe); return; + } if ((pmlmeinfo->state&0x03) == WIFI_FW_AP_STATE) { pwps_ie = rtw_get_wps_ie(cur_network->ies+_FIXED_IE_LENGTH_, cur_network->ie_length-_FIXED_IE_LENGTH_, NULL, &wps_ielen); @@ -2371,8 +2377,11 @@ void issue_probersp(struct adapter *padapter, unsigned char *da, u8 is_valid_p2p u8 *ies = pmgntframe->buf_addr+TXDESC_OFFSET+sizeof(struct ieee80211_hdr_3addr); buf = rtw_zmalloc(MAX_IE_SZ); - if (!buf) + if (!buf) { + rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf); + rtw_free_xmitframe(pxmitpriv, pmgntframe); return; + } ssid_ie = rtw_get_ie(ies+_FIXED_IE_LENGTH_, WLAN_EID_SSID, &ssid_ielen, (pframe-ies)-_FIXED_IE_LENGTH_); @@ -2765,10 +2774,13 @@ void issue_asocrsp(struct adapter *padapter, unsigned short status, struct sta_i SetSeqNum(pwlanhdr, pmlmeext->mgnt_seq); pmlmeext->mgnt_seq++; - if ((pkt_type == WIFI_ASSOCRSP) || (pkt_type == WIFI_REASSOCRSP)) + if ((pkt_type == WIFI_ASSOCRSP) || (pkt_type == WIFI_REASSOCRSP)) { SetFrameSubType(pwlanhdr, pkt_type); - else + } else { + rtw_free_xmitbuf(pxmitpriv, pmgntframe->pxmitbuf); + rtw_free_xmitframe(pxmitpriv, pmgntframe); return; + } pattrib->hdrlen = sizeof(struct ieee80211_hdr_3addr); pattrib->pktlen += pattrib->hdrlen; diff --git a/drivers/staging/sm750fb/sm750.c b/drivers/staging/sm750fb/sm750.c index 86781bbd5ce0..c2202bab0ce5 100644 --- a/drivers/staging/sm750fb/sm750.c +++ b/drivers/staging/sm750fb/sm750.c @@ -275,7 +275,7 @@ static void lynxfb_ops_imageblit(struct fb_info *info, spin_lock(&sm750_dev->slock); sm750_dev->accel.de_imageblit(&sm750_dev->accel, - image->data, image->width >> 3, 0, + image->data, 0, base, pitch, Bpp, image->dx, image->dy, image->width, image->height, diff --git a/drivers/staging/sm750fb/sm750.h b/drivers/staging/sm750fb/sm750.h index aff69661c8e6..a693d9f1d0d5 100644 --- a/drivers/staging/sm750fb/sm750.h +++ b/drivers/staging/sm750fb/sm750.h @@ -73,7 +73,7 @@ struct lynx_accel { u32 rop2); int (*de_imageblit)(struct lynx_accel *accel, const char *p_srcbuf, - u32 src_delta, u32 start_bit, u32 d_base, u32 d_pitch, + u32 start_bit, u32 d_base, u32 d_pitch, u32 byte_per_pixel, u32 dx, u32 dy, u32 width, u32 height, u32 f_color, u32 b_color, u32 rop2); diff --git a/drivers/staging/sm750fb/sm750_accel.c b/drivers/staging/sm750fb/sm750_accel.c index 44b9e3fe3a41..37217a496ba7 100644 --- a/drivers/staging/sm750fb/sm750_accel.c +++ b/drivers/staging/sm750fb/sm750_accel.c @@ -300,8 +300,6 @@ static unsigned int deGetTransparency(struct lynx_accel *accel) * sm750_hw_imageblit * @accel: Acceleration device data * @pSrcbuf: pointer to start of source buffer in system memory - * @srcDelta: Pitch value (in bytes) of the source buffer, +ive means top down - * and -ive mean button up * @startBit: Mono data can start at any bit in a byte, this value should be * 0 to 7 * @dBase: Address of destination: offset in frame buffer @@ -316,7 +314,7 @@ static unsigned int deGetTransparency(struct lynx_accel *accel) * @rop2: ROP value */ int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf, - u32 srcDelta, u32 startBit, u32 dBase, u32 dPitch, + u32 startBit, u32 dBase, u32 dPitch, u32 bytePerPixel, u32 dx, u32 dy, u32 width, u32 height, u32 fColor, u32 bColor, u32 rop2) { @@ -405,7 +403,7 @@ int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf, write_dpPort(accel, *(unsigned int *)ajRemain); } - pSrcbuf += srcDelta; + pSrcbuf += ulBytesPerScan; } return 0; diff --git a/drivers/staging/sm750fb/sm750_accel.h b/drivers/staging/sm750fb/sm750_accel.h index 2c79cb730a0a..df48ed57a054 100644 --- a/drivers/staging/sm750fb/sm750_accel.h +++ b/drivers/staging/sm750fb/sm750_accel.h @@ -220,8 +220,6 @@ int sm750_hw_copyarea(struct lynx_accel *accel, /** * sm750_hw_imageblit * @pSrcbuf: pointer to start of source buffer in system memory - * @srcDelta: Pitch value (in bytes) of the source buffer, +ive means top down - *>----- and -ive mean button up * @startBit: Mono data can start at any bit in a byte, this value should be *>----- 0 to 7 * @dBase: Address of destination: offset in frame buffer @@ -236,7 +234,7 @@ int sm750_hw_copyarea(struct lynx_accel *accel, * @rop2: ROP value */ int sm750_hw_imageblit(struct lynx_accel *accel, const char *pSrcbuf, - u32 srcDelta, u32 startBit, u32 dBase, u32 dPitch, + u32 startBit, u32 dBase, u32 dPitch, u32 bytePerPixel, u32 dx, u32 dy, u32 width, u32 height, u32 fColor, u32 bColor, u32 rop2); diff --git a/drivers/tee/optee/ffa_abi.c b/drivers/tee/optee/ffa_abi.c index 927c3d7947f9..f0ecd931035e 100644 --- a/drivers/tee/optee/ffa_abi.c +++ b/drivers/tee/optee/ffa_abi.c @@ -844,14 +844,6 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev) } optee->supp_teedev = teedev; - rc = tee_device_register(optee->teedev); - if (rc) - goto err_unreg_supp_teedev; - - rc = tee_device_register(optee->supp_teedev); - if (rc) - goto err_unreg_supp_teedev; - rc = rhashtable_init(&optee->ffa.global_ids, &shm_rhash_params); if (rc) goto err_unreg_supp_teedev; @@ -871,6 +863,14 @@ static int optee_ffa_probe(struct ffa_device *ffa_dev) if (rc) goto err_close_ctx; + rc = tee_device_register(optee->teedev); + if (rc) + goto err_unregister_devices; + + rc = tee_device_register(optee->supp_teedev); + if (rc) + goto err_unregister_devices; + rc = optee_enumerate_devices(PTA_CMD_GET_DEVICES); if (rc) goto err_unregister_devices; diff --git a/drivers/tee/optee/smc_abi.c b/drivers/tee/optee/smc_abi.c index d5b28fd35d66..39941bcf1bb9 100644 --- a/drivers/tee/optee/smc_abi.c +++ b/drivers/tee/optee/smc_abi.c @@ -1717,14 +1717,6 @@ static int optee_probe(struct platform_device *pdev) } optee->supp_teedev = teedev; - rc = tee_device_register(optee->teedev); - if (rc) - goto err_unreg_supp_teedev; - - rc = tee_device_register(optee->supp_teedev); - if (rc) - goto err_unreg_supp_teedev; - mutex_init(&optee->call_queue.mutex); INIT_LIST_HEAD(&optee->call_queue.waiters); optee_supp_init(&optee->supp); @@ -1781,6 +1773,14 @@ static int optee_probe(struct platform_device *pdev) if (optee->smc.sec_caps & OPTEE_SMC_SEC_CAP_DYNAMIC_SHM) pr_info("dynamic shared memory is enabled\n"); + rc = tee_device_register(optee->teedev); + if (rc) + goto err_disable_shm_cache; + + rc = tee_device_register(optee->supp_teedev); + if (rc) + goto err_disable_shm_cache; + rc = optee_enumerate_devices(PTA_CMD_GET_DEVICES); if (rc) goto err_disable_shm_cache; @@ -1801,7 +1801,6 @@ static int optee_probe(struct platform_device *pdev) optee_shm_arg_cache_uninit(optee); optee_supp_uninit(&optee->supp); mutex_destroy(&optee->call_queue.mutex); -err_unreg_supp_teedev: tee_device_unregister(optee->supp_teedev); err_unreg_teedev: tee_device_unregister(optee->teedev); diff --git a/drivers/thermal/thermal_hwmon.c b/drivers/thermal/thermal_hwmon.c index 79a4de5f9f0e..55a9bd111d25 100644 --- a/drivers/thermal/thermal_hwmon.c +++ b/drivers/thermal/thermal_hwmon.c @@ -101,34 +101,12 @@ thermal_hwmon_lookup_by_type(const struct thermal_zone_device *tz) struct thermal_hwmon_device *hwmon; char type[THERMAL_NAME_LENGTH]; - mutex_lock(&thermal_hwmon_list_lock); list_for_each_entry(hwmon, &thermal_hwmon_list, node) { strcpy(type, tz->type); strreplace(type, '-', '_'); - if (!strcmp(hwmon->type, type)) { - mutex_unlock(&thermal_hwmon_list_lock); + if (!strcmp(hwmon->type, type)) return hwmon; - } } - mutex_unlock(&thermal_hwmon_list_lock); - - return NULL; -} - -/* Find the temperature input matching a given thermal zone */ -static struct thermal_hwmon_temp * -thermal_hwmon_lookup_temp(const struct thermal_hwmon_device *hwmon, - const struct thermal_zone_device *tz) -{ - struct thermal_hwmon_temp *temp; - - mutex_lock(&thermal_hwmon_list_lock); - list_for_each_entry(temp, &hwmon->tz_list, hwmon_node) - if (temp->tz == tz) { - mutex_unlock(&thermal_hwmon_list_lock); - return temp; - } - mutex_unlock(&thermal_hwmon_list_lock); return NULL; } @@ -144,7 +122,9 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz) struct thermal_hwmon_device *hwmon; struct thermal_hwmon_temp *temp; int new_hwmon_device = 1; - int result; + int result = 0; + + mutex_lock(&thermal_hwmon_list_lock); hwmon = thermal_hwmon_lookup_by_type(tz); if (hwmon) { @@ -153,8 +133,10 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz) } hwmon = kzalloc(sizeof(*hwmon), GFP_KERNEL); - if (!hwmon) - return -ENOMEM; + if (!hwmon) { + result = -ENOMEM; + goto unlock; + } INIT_LIST_HEAD(&hwmon->tz_list); strscpy(hwmon->type, tz->type, THERMAL_NAME_LENGTH); @@ -202,24 +184,24 @@ int thermal_add_hwmon_sysfs(struct thermal_zone_device *tz) temp->temp_crit_present = true; } - mutex_lock(&thermal_hwmon_list_lock); if (new_hwmon_device) list_add_tail(&hwmon->node, &thermal_hwmon_list); list_add_tail(&temp->hwmon_node, &hwmon->tz_list); - mutex_unlock(&thermal_hwmon_list_lock); - return 0; + goto unlock; - unregister_input: +unregister_input: device_remove_file(hwmon->device, &temp->temp_input.attr); - free_temp_mem: +free_temp_mem: kfree(temp); - unregister_name: +unregister_name: if (new_hwmon_device) hwmon_device_unregister(hwmon->device); - free_mem: +free_mem: if (new_hwmon_device) kfree(hwmon); +unlock: + mutex_unlock(&thermal_hwmon_list_lock); return result; } @@ -227,8 +209,11 @@ EXPORT_SYMBOL_GPL(thermal_add_hwmon_sysfs); void thermal_remove_hwmon_sysfs(struct thermal_zone_device *tz) { + struct thermal_hwmon_temp *temp, *entry; struct thermal_hwmon_device *hwmon; - struct thermal_hwmon_temp *temp; + bool unregister; + + guard(mutex)(&thermal_hwmon_list_lock); hwmon = thermal_hwmon_lookup_by_type(tz); if (unlikely(!hwmon)) { @@ -237,29 +222,25 @@ void thermal_remove_hwmon_sysfs(struct thermal_zone_device *tz) return; } - temp = thermal_hwmon_lookup_temp(hwmon, tz); - if (unlikely(!temp)) { - /* Should never happen... */ - dev_dbg(&tz->device, "temperature input lookup failed!\n"); - return; - } + unregister = hwmon->device->parent == &tz->device; - device_remove_file(hwmon->device, &temp->temp_input.attr); - if (temp->temp_crit_present) - device_remove_file(hwmon->device, &temp->temp_crit.attr); + list_for_each_entry_safe_reverse(temp, entry, &hwmon->tz_list, hwmon_node) { + if (!unregister && temp->tz != tz) + continue; - mutex_lock(&thermal_hwmon_list_lock); - list_del(&temp->hwmon_node); - kfree(temp); - if (!list_empty(&hwmon->tz_list)) { - mutex_unlock(&thermal_hwmon_list_lock); - return; + device_remove_file(hwmon->device, &temp->temp_input.attr); + if (temp->temp_crit_present) + device_remove_file(hwmon->device, &temp->temp_crit.attr); + + list_del(&temp->hwmon_node); + kfree(temp); } - list_del(&hwmon->node); - mutex_unlock(&thermal_hwmon_list_lock); - hwmon_device_unregister(hwmon->device); - kfree(hwmon); + if (unregister) { + list_del(&hwmon->node); + hwmon_device_unregister(hwmon->device); + kfree(hwmon); + } } EXPORT_SYMBOL_GPL(thermal_remove_hwmon_sysfs); diff --git a/drivers/thunderbolt/ctl.c b/drivers/thunderbolt/ctl.c index a60d7afc041f..3a7cc231079a 100644 --- a/drivers/thunderbolt/ctl.c +++ b/drivers/thunderbolt/ctl.c @@ -65,7 +65,6 @@ struct tb_ctl { #define tb_ctl_dbg(ctl, format, arg...) \ dev_dbg(&(ctl)->nhi->pdev->dev, format, ## arg) -static DECLARE_WAIT_QUEUE_HEAD(tb_cfg_request_cancel_queue); /* Serializes access to request kref_get/put */ static DEFINE_MUTEX(tb_cfg_request_lock); @@ -123,41 +122,42 @@ void tb_cfg_request_put(struct tb_cfg_request *req) static int tb_cfg_request_enqueue(struct tb_ctl *ctl, struct tb_cfg_request *req) { + tb_cfg_request_get(req); + WARN_ON(test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags)); WARN_ON(req->ctl); - mutex_lock(&ctl->request_queue_lock); + guard(mutex)(&ctl->request_queue_lock); if (!ctl->running) { - mutex_unlock(&ctl->request_queue_lock); + tb_cfg_request_put(req); return -ENOTCONN; } req->ctl = ctl; list_add_tail(&req->list, &ctl->request_queue); set_bit(TB_CFG_REQUEST_ACTIVE, &req->flags); - mutex_unlock(&ctl->request_queue_lock); return 0; } -static void tb_cfg_request_dequeue(struct tb_cfg_request *req) +static bool tb_cfg_request_is_active(struct tb_cfg_request *req) { - struct tb_ctl *ctl = req->ctl; - - mutex_lock(&ctl->request_queue_lock); - if (!test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags)) { - mutex_unlock(&ctl->request_queue_lock); - return; - } + return test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags); +} - list_del(&req->list); - clear_bit(TB_CFG_REQUEST_ACTIVE, &req->flags); - if (test_bit(TB_CFG_REQUEST_CANCELED, &req->flags)) - wake_up(&tb_cfg_request_cancel_queue); - mutex_unlock(&ctl->request_queue_lock); +static bool tb_cfg_request_is_canceled(struct tb_cfg_request *req) +{ + return test_bit(TB_CFG_REQUEST_CANCELED, &req->flags); } -static bool tb_cfg_request_is_active(struct tb_cfg_request *req) +static void tb_cfg_request_dequeue(struct tb_cfg_request *req) { - return test_bit(TB_CFG_REQUEST_ACTIVE, &req->flags); + struct tb_ctl *ctl = req->ctl; + + guard(mutex)(&ctl->request_queue_lock); + if (tb_cfg_request_is_active(req)) { + list_del(&req->list); + clear_bit(TB_CFG_REQUEST_ACTIVE, &req->flags); + tb_cfg_request_put(req); + } } static struct tb_cfg_request * @@ -168,7 +168,7 @@ tb_cfg_request_find(struct tb_ctl *ctl, struct ctl_pkg *pkg) mutex_lock(&pkg->ctl->request_queue_lock); list_for_each_entry(iter, &pkg->ctl->request_queue, list) { tb_cfg_request_get(iter); - if (iter->match(iter, pkg)) { + if (!tb_cfg_request_is_canceled(iter) && iter->match(iter, pkg)) { req = iter; break; } @@ -495,8 +495,11 @@ static void tb_ctl_rx_callback(struct tb_ring *ring, struct ring_frame *frame, */ req = tb_cfg_request_find(pkg->ctl, pkg); if (req) { - if (req->copy(req, pkg)) - schedule_work(&req->work); + scoped_guard(mutex, &pkg->ctl->request_queue_lock) { + if (!tb_cfg_request_is_canceled(req) && + req->copy(req, pkg)) + schedule_work(&req->work); + } tb_cfg_request_put(req); } @@ -508,11 +511,10 @@ static void tb_cfg_request_work(struct work_struct *work) { struct tb_cfg_request *req = container_of(work, typeof(*req), work); - if (!test_bit(TB_CFG_REQUEST_CANCELED, &req->flags)) + if (!tb_cfg_request_is_canceled(req)) req->callback(req->callback_data); tb_cfg_request_dequeue(req); - tb_cfg_request_put(req); } /** @@ -536,10 +538,9 @@ int tb_cfg_request(struct tb_ctl *ctl, struct tb_cfg_request *req, INIT_WORK(&req->work, tb_cfg_request_work); INIT_LIST_HEAD(&req->list); - tb_cfg_request_get(req); ret = tb_cfg_request_enqueue(ctl, req); if (ret) - goto err_put; + return ret; ret = tb_ctl_tx(ctl, req->request, req->request_size, req->request_type); @@ -553,9 +554,6 @@ int tb_cfg_request(struct tb_ctl *ctl, struct tb_cfg_request *req, err_dequeue: tb_cfg_request_dequeue(req); -err_put: - tb_cfg_request_put(req); - return ret; } @@ -569,9 +567,10 @@ int tb_cfg_request(struct tb_ctl *ctl, struct tb_cfg_request *req, */ void tb_cfg_request_cancel(struct tb_cfg_request *req, int err) { - set_bit(TB_CFG_REQUEST_CANCELED, &req->flags); - schedule_work(&req->work); - wait_event(tb_cfg_request_cancel_queue, !tb_cfg_request_is_active(req)); + scoped_guard(mutex, &req->ctl->request_queue_lock) + set_bit(TB_CFG_REQUEST_CANCELED, &req->flags); + cancel_work_sync(&req->work); + tb_cfg_request_dequeue(req); req->result.err = err; } diff --git a/drivers/thunderbolt/quirks.c b/drivers/thunderbolt/quirks.c index e81de9c30eac..520afebebd42 100644 --- a/drivers/thunderbolt/quirks.c +++ b/drivers/thunderbolt/quirks.c @@ -92,6 +92,9 @@ static const struct tb_quirk tb_quirks[] = { quirk_usb3_maximum_bandwidth }, { 0x8087, PCI_DEVICE_ID_INTEL_BARLOW_RIDGE_HUB_40G_BRIDGE, 0x0000, 0x0000, quirk_usb3_maximum_bandwidth }, + /* CLx is unstable on the Anker Prime TB5 dock (DROM 01ea:83b5) */ + { 0x8087, PCI_DEVICE_ID_INTEL_BARLOW_RIDGE_HUB_80G_BRIDGE, 0x01ea, 0x83b5, + quirk_clx_disable }, /* * Block Runtime PM in DP redrive mode for Intel Barlow Ridge host * controllers. diff --git a/drivers/thunderbolt/xdomain.c b/drivers/thunderbolt/xdomain.c index 316aa2738cfe..c831a8323514 100644 --- a/drivers/thunderbolt/xdomain.c +++ b/drivers/thunderbolt/xdomain.c @@ -377,6 +377,10 @@ static int tb_xdp_properties_request(struct tb_ctl *ctl, u64 route, len += sizeof(res->hdr.xd_hdr) / 4; len -= sizeof(*res) / 4; + if (len > TB_XDP_PROPERTIES_MAX_DATA_LENGTH) { + ret = -EINVAL; + goto err; + } if (res->offset != req.offset) { ret = -EINVAL; diff --git a/drivers/tty/amiserial.c b/drivers/tty/amiserial.c index 785558c65ae8..822ef406fce5 100644 --- a/drivers/tty/amiserial.c +++ b/drivers/tty/amiserial.c @@ -1140,11 +1140,6 @@ static int rs_ioctl(struct tty_struct *tty, local_irq_save(flags); cnow = info->icount; /* atomic copy */ local_irq_restore(flags); - if (cnow.rng == cprev.rng && cnow.dsr == cprev.dsr && - cnow.dcd == cprev.dcd && cnow.cts == cprev.cts) { - ret = -EIO; /* no change => error */ - break; - } if ( ((arg & TIOCM_RNG) && (cnow.rng != cprev.rng)) || ((arg & TIOCM_DSR) && (cnow.dsr != cprev.dsr)) || ((arg & TIOCM_CD) && (cnow.dcd != cprev.dcd)) || @@ -1152,6 +1147,10 @@ static int rs_ioctl(struct tty_struct *tty, ret = 0; break; } + if (tty_io_error(tty)) { + ret = -EIO; + break; + } schedule(); /* see if a signal did it */ if (signal_pending(current)) { diff --git a/drivers/tty/n_gsm.c b/drivers/tty/n_gsm.c index 901f5c8311fb..2c1523b58313 100644 --- a/drivers/tty/n_gsm.c +++ b/drivers/tty/n_gsm.c @@ -453,14 +453,13 @@ static const u8 gsm_fcs8[256] = { static void gsm_dlci_close(struct gsm_dlci *dlci); static int gsmld_output(struct gsm_mux *gsm, u8 *data, int len); -static int gsm_modem_update(struct gsm_dlci *dlci, u8 brk); +static int gsm_modem_update(struct gsm_dlci *dlci, u8 brk, bool wait); static struct gsm_msg *gsm_data_alloc(struct gsm_mux *gsm, u8 addr, int len, u8 ctrl); static int gsm_send_packet(struct gsm_mux *gsm, struct gsm_msg *msg); static struct gsm_dlci *gsm_dlci_alloc(struct gsm_mux *gsm, int addr); static void gsmld_write_trigger(struct gsm_mux *gsm); static void gsmld_write_task(struct work_struct *work); -static int gsm_modem_send_initial_msc(struct gsm_dlci *dlci); /** * gsm_fcs_add - update FCS @@ -2174,7 +2173,7 @@ static void gsm_dlci_open(struct gsm_dlci *dlci) pr_debug("DLCI %d goes open.\n", dlci->addr); /* Send current modem state */ if (dlci->addr) { - gsm_modem_send_initial_msc(dlci); + gsm_modem_update(dlci, 0, false); } else { /* Start keep-alive control */ gsm->ka_num = 0; @@ -4132,9 +4131,10 @@ static void gsm_modem_upd_via_data(struct gsm_dlci *dlci, u8 brk) * gsm_modem_upd_via_msc - send modem bits via control frame * @dlci: channel * @brk: break signal + * @wait: wait for the MSC command */ -static int gsm_modem_upd_via_msc(struct gsm_dlci *dlci, u8 brk) +static int gsm_modem_upd_via_msc(struct gsm_dlci *dlci, u8 brk, bool wait) { u8 modembits[3]; struct gsm_control *ctrl; @@ -4151,41 +4151,22 @@ static int gsm_modem_upd_via_msc(struct gsm_dlci *dlci, u8 brk) modembits[2] = (brk << 4) | 2 | EA; /* Length, Break, EA */ len++; } + if (!wait) + return gsm_control_command(dlci->gsm, CMD_MSC, modembits, len); ctrl = gsm_control_send(dlci->gsm, CMD_MSC, modembits, len); if (ctrl == NULL) return -ENOMEM; return gsm_control_wait(dlci->gsm, ctrl); } -/** - * gsm_modem_send_initial_msc - Send initial modem status message - * - * @dlci channel - * - * Send an initial MSC message after DLCI open to set the initial - * modem status lines. This is only done for basic mode. - * Does not wait for a response as we cannot block the input queue - * processing. - */ -static int gsm_modem_send_initial_msc(struct gsm_dlci *dlci) -{ - u8 modembits[2]; - - if (dlci->adaption != 1 || dlci->gsm->encoding != GSM_BASIC_OPT) - return 0; - - modembits[0] = (dlci->addr << 2) | 2 | EA; /* DLCI, Valid, EA */ - modembits[1] = (gsm_encode_modem(dlci) << 1) | EA; - return gsm_control_command(dlci->gsm, CMD_MSC, (const u8 *)&modembits, 2); -} - /** * gsm_modem_update - send modem status line state * @dlci: channel * @brk: break signal + * @wait: wait for the MSC command */ -static int gsm_modem_update(struct gsm_dlci *dlci, u8 brk) +static int gsm_modem_update(struct gsm_dlci *dlci, u8 brk, bool wait) { if (dlci->gsm->dead) return -EL2HLT; @@ -4195,7 +4176,7 @@ static int gsm_modem_update(struct gsm_dlci *dlci, u8 brk) return 0; } else if (dlci->gsm->encoding == GSM_BASIC_OPT) { /* Send as MSC control message. */ - return gsm_modem_upd_via_msc(dlci, brk); + return gsm_modem_upd_via_msc(dlci, brk, wait); } /* Modem status lines are not supported. */ @@ -4261,7 +4242,7 @@ static void gsm_dtr_rts(struct tty_port *port, bool active) modem_tx &= ~(TIOCM_DTR | TIOCM_RTS); if (modem_tx != dlci->modem_tx) { dlci->modem_tx = modem_tx; - gsm_modem_update(dlci, 0); + gsm_modem_update(dlci, 0, true); } } @@ -4457,7 +4438,7 @@ static int gsmtty_tiocmset(struct tty_struct *tty, if (modem_tx != dlci->modem_tx) { dlci->modem_tx = modem_tx; - return gsm_modem_update(dlci, 0); + return gsm_modem_update(dlci, 0, true); } return 0; } @@ -4539,7 +4520,7 @@ static void gsmtty_throttle(struct tty_struct *tty) dlci->modem_tx &= ~TIOCM_RTS; dlci->throttled = true; /* Send an MSC with RTS cleared */ - gsm_modem_update(dlci, 0); + gsm_modem_update(dlci, 0, true); } static void gsmtty_unthrottle(struct tty_struct *tty) @@ -4551,7 +4532,7 @@ static void gsmtty_unthrottle(struct tty_struct *tty) dlci->modem_tx |= TIOCM_RTS; dlci->throttled = false; /* Send an MSC with RTS set */ - gsm_modem_update(dlci, 0); + gsm_modem_update(dlci, 0, true); } static int gsmtty_break_ctl(struct tty_struct *tty, int state) @@ -4569,7 +4550,7 @@ static int gsmtty_break_ctl(struct tty_struct *tty, int state) if (encode > 0x0F) encode = 0x0F; /* Best effort */ } - return gsm_modem_update(dlci, encode); + return gsm_modem_update(dlci, encode, true); } static void gsmtty_cleanup(struct tty_struct *tty) diff --git a/drivers/tty/serial/8250/8250_bcm7271.c b/drivers/tty/serial/8250/8250_bcm7271.c index 9afd5979c9e0..5b56f96dc950 100644 --- a/drivers/tty/serial/8250/8250_bcm7271.c +++ b/drivers/tty/serial/8250/8250_bcm7271.c @@ -1153,8 +1153,8 @@ static int brcmuart_remove(struct platform_device *pdev) struct brcmuart_priv *priv = platform_get_drvdata(pdev); debugfs_remove_recursive(priv->debugfs_dir); - hrtimer_cancel(&priv->hrt); serial8250_unregister_port(priv->line); + hrtimer_cancel(&priv->hrt); brcmuart_free_bufs(&pdev->dev, priv); clk_disable_unprepare(priv->baud_mux_clk); if (priv->dma_enabled) diff --git a/drivers/tty/serial/8250/8250_core.c b/drivers/tty/serial/8250/8250_core.c index c536028e92dc..cc28943254b3 100644 --- a/drivers/tty/serial/8250/8250_core.c +++ b/drivers/tty/serial/8250/8250_core.c @@ -1034,139 +1034,140 @@ int serial8250_register_8250_port(const struct uart_8250_port *up) nr_uarts++; } - if (uart->port.type != PORT_8250_CIR) { - struct mctrl_gpios *gpios; - - if (uart->port.dev) - uart_remove_one_port(&serial8250_reg, &uart->port); - - uart->port.ctrl_id = up->port.ctrl_id; - uart->port.port_id = up->port.port_id; - uart->port.iobase = up->port.iobase; - uart->port.membase = up->port.membase; - uart->port.irq = up->port.irq; - uart->port.irqflags = up->port.irqflags; - uart->port.uartclk = up->port.uartclk; - uart->port.fifosize = up->port.fifosize; - uart->port.regshift = up->port.regshift; - uart->port.iotype = up->port.iotype; - uart->port.flags = up->port.flags | UPF_BOOT_AUTOCONF; - uart->bugs = up->bugs; - uart->port.mapbase = up->port.mapbase; - uart->port.mapsize = up->port.mapsize; - uart->port.private_data = up->port.private_data; - uart->tx_loadsz = up->tx_loadsz; - uart->capabilities = up->capabilities; - uart->port.throttle = up->port.throttle; - uart->port.unthrottle = up->port.unthrottle; - uart->port.rs485_config = up->port.rs485_config; - uart->port.rs485_supported = up->port.rs485_supported; - uart->port.rs485 = up->port.rs485; - uart->rs485_start_tx = up->rs485_start_tx; - uart->rs485_stop_tx = up->rs485_stop_tx; - uart->lsr_save_mask = up->lsr_save_mask; - uart->dma = up->dma; - - /* Take tx_loadsz from fifosize if it wasn't set separately */ - if (uart->port.fifosize && !uart->tx_loadsz) - uart->tx_loadsz = uart->port.fifosize; - - if (up->port.dev) { - uart->port.dev = up->port.dev; - ret = uart_get_rs485_mode(&uart->port); - if (ret) - goto err; - } + if (uart->port.type == PORT_8250_CIR) { + ret = -ENODEV; + goto unlock; + } - if (up->port.flags & UPF_FIXED_TYPE) - uart->port.type = up->port.type; + if (uart->port.dev) + uart_remove_one_port(&serial8250_reg, &uart->port); + + uart->port.ctrl_id = up->port.ctrl_id; + uart->port.port_id = up->port.port_id; + uart->port.iobase = up->port.iobase; + uart->port.membase = up->port.membase; + uart->port.irq = up->port.irq; + uart->port.irqflags = up->port.irqflags; + uart->port.uartclk = up->port.uartclk; + uart->port.fifosize = up->port.fifosize; + uart->port.regshift = up->port.regshift; + uart->port.iotype = up->port.iotype; + uart->port.flags = up->port.flags | UPF_BOOT_AUTOCONF; + uart->bugs = up->bugs; + uart->port.mapbase = up->port.mapbase; + uart->port.mapsize = up->port.mapsize; + uart->port.private_data = up->port.private_data; + uart->tx_loadsz = up->tx_loadsz; + uart->capabilities = up->capabilities; + uart->port.throttle = up->port.throttle; + uart->port.unthrottle = up->port.unthrottle; + uart->port.rs485_config = up->port.rs485_config; + uart->port.rs485_supported = up->port.rs485_supported; + uart->port.rs485 = up->port.rs485; + uart->rs485_start_tx = up->rs485_start_tx; + uart->rs485_stop_tx = up->rs485_stop_tx; + uart->lsr_save_mask = up->lsr_save_mask; + uart->dma = up->dma; + + /* Take tx_loadsz from fifosize if it wasn't set separately */ + if (uart->port.fifosize && !uart->tx_loadsz) + uart->tx_loadsz = uart->port.fifosize; + + if (up->port.dev) { + uart->port.dev = up->port.dev; + ret = uart_get_rs485_mode(&uart->port); + if (ret) + goto err; + } - /* - * Only call mctrl_gpio_init(), if the device has no ACPI - * companion device - */ - if (!has_acpi_companion(uart->port.dev)) { - gpios = mctrl_gpio_init(&uart->port, 0); - if (IS_ERR(gpios)) { - ret = PTR_ERR(gpios); - goto err; - } else { - uart->gpios = gpios; - } - } + if (up->port.flags & UPF_FIXED_TYPE) + uart->port.type = up->port.type; - serial8250_set_defaults(uart); - - /* Possibly override default I/O functions. */ - if (up->port.serial_in) - uart->port.serial_in = up->port.serial_in; - if (up->port.serial_out) - uart->port.serial_out = up->port.serial_out; - if (up->port.handle_irq) - uart->port.handle_irq = up->port.handle_irq; - /* Possibly override set_termios call */ - if (up->port.set_termios) - uart->port.set_termios = up->port.set_termios; - if (up->port.set_ldisc) - uart->port.set_ldisc = up->port.set_ldisc; - if (up->port.get_mctrl) - uart->port.get_mctrl = up->port.get_mctrl; - if (up->port.set_mctrl) - uart->port.set_mctrl = up->port.set_mctrl; - if (up->port.get_divisor) - uart->port.get_divisor = up->port.get_divisor; - if (up->port.set_divisor) - uart->port.set_divisor = up->port.set_divisor; - if (up->port.startup) - uart->port.startup = up->port.startup; - if (up->port.shutdown) - uart->port.shutdown = up->port.shutdown; - if (up->port.pm) - uart->port.pm = up->port.pm; - if (up->port.handle_break) - uart->port.handle_break = up->port.handle_break; - if (up->dl_read) - uart->dl_read = up->dl_read; - if (up->dl_write) - uart->dl_write = up->dl_write; - - if (uart->port.type != PORT_8250_CIR) { - if (uart_console_registered(&uart->port)) - pm_runtime_get_sync(uart->port.dev); - - if (serial8250_isa_config != NULL) - serial8250_isa_config(0, &uart->port, - &uart->capabilities); - - serial8250_apply_quirks(uart); - ret = uart_add_one_port(&serial8250_reg, - &uart->port); - if (ret) - goto err; - - ret = uart->port.line; + /* + * Only call mctrl_gpio_init(), if the device has no ACPI + * companion device + */ + if (uart->port.dev && !has_acpi_companion(uart->port.dev)) { + struct mctrl_gpios *gpios = mctrl_gpio_init(&uart->port, 0); + if (IS_ERR(gpios)) { + ret = PTR_ERR(gpios); + goto err; } else { - dev_info(uart->port.dev, - "skipping CIR port at 0x%lx / 0x%llx, IRQ %d\n", - uart->port.iobase, - (unsigned long long)uart->port.mapbase, - uart->port.irq); - - ret = 0; + uart->gpios = gpios; } + } - if (!uart->lsr_save_mask) - uart->lsr_save_mask = LSR_SAVE_FLAGS; /* Use default LSR mask */ + serial8250_set_defaults(uart); + + /* Possibly override default I/O functions. */ + if (up->port.serial_in) + uart->port.serial_in = up->port.serial_in; + if (up->port.serial_out) + uart->port.serial_out = up->port.serial_out; + if (up->port.handle_irq) + uart->port.handle_irq = up->port.handle_irq; + /* Possibly override set_termios call */ + if (up->port.set_termios) + uart->port.set_termios = up->port.set_termios; + if (up->port.set_ldisc) + uart->port.set_ldisc = up->port.set_ldisc; + if (up->port.get_mctrl) + uart->port.get_mctrl = up->port.get_mctrl; + if (up->port.set_mctrl) + uart->port.set_mctrl = up->port.set_mctrl; + if (up->port.get_divisor) + uart->port.get_divisor = up->port.get_divisor; + if (up->port.set_divisor) + uart->port.set_divisor = up->port.set_divisor; + if (up->port.startup) + uart->port.startup = up->port.startup; + if (up->port.shutdown) + uart->port.shutdown = up->port.shutdown; + if (up->port.pm) + uart->port.pm = up->port.pm; + if (up->port.handle_break) + uart->port.handle_break = up->port.handle_break; + if (up->dl_read) + uart->dl_read = up->dl_read; + if (up->dl_write) + uart->dl_write = up->dl_write; - /* Initialise interrupt backoff work if required */ - if (up->overrun_backoff_time_ms > 0) { - uart->overrun_backoff_time_ms = - up->overrun_backoff_time_ms; - INIT_DELAYED_WORK(&uart->overrun_backoff, - serial_8250_overrun_backoff_work); - } else { - uart->overrun_backoff_time_ms = 0; - } + if (uart->port.type != PORT_8250_CIR) { + if (uart_console_registered(&uart->port)) + pm_runtime_get_sync(uart->port.dev); + + if (serial8250_isa_config != NULL) + serial8250_isa_config(0, &uart->port, + &uart->capabilities); + + serial8250_apply_quirks(uart); + ret = uart_add_one_port(&serial8250_reg, + &uart->port); + if (ret) + goto err; + + ret = uart->port.line; + } else { + dev_info(uart->port.dev, + "skipping CIR port at 0x%lx / 0x%llx, IRQ %d\n", + uart->port.iobase, + (unsigned long long)uart->port.mapbase, + uart->port.irq); + + ret = 0; + } + + if (!uart->lsr_save_mask) + uart->lsr_save_mask = LSR_SAVE_FLAGS; /* Use default LSR mask */ + + /* Initialise interrupt backoff work if required */ + if (up->overrun_backoff_time_ms > 0) { + uart->overrun_backoff_time_ms = + up->overrun_backoff_time_ms; + INIT_DELAYED_WORK(&uart->overrun_backoff, + serial_8250_overrun_backoff_work); + } else { + uart->overrun_backoff_time_ms = 0; } unlock: diff --git a/drivers/tty/serial/8250/8250_of.c b/drivers/tty/serial/8250/8250_of.c index 51329625c48a..b704248f6bb1 100644 --- a/drivers/tty/serial/8250/8250_of.c +++ b/drivers/tty/serial/8250/8250_of.c @@ -25,6 +25,46 @@ struct of_serial_info { int line; }; +static int lpc32xx_handle_irq(struct uart_port *port) +{ + struct uart_8250_port *up = up_to_u8250p(port); + unsigned long flags; + unsigned int iir; + u16 status; + int ret; + + serial8250_rpm_get(up); + + iir = serial_port_in(port, UART_IIR); + if (iir & UART_IIR_NO_INT) { + serial8250_rpm_put(up); + return 0; + } + + /* + * The LPC32xx UART can assert an RX character-timeout interrupt while + * the RX FIFO is empty: IIR reports UART_IIR_RX_TIMEOUT but LSR.DR is + * clear. The timeout is only cleared by reading RHR, but the core RX + * path skips that read when the FIFO is empty, so the level-triggered + * IRQ re-fires forever and livelocks this single-core SoC. Do one + * throwaway RHR read to clear it; a healthy UART never reports a + * timeout with DR/BI clear, so no received data is ever discarded. + */ + if ((iir & 0x3f) == UART_IIR_RX_TIMEOUT) { + spin_lock_irqsave(&port->lock, flags); + status = serial_lsr_in(up); + if (!(status & (UART_LSR_DR | UART_LSR_BI))) + serial_port_in(port, UART_RX); + spin_unlock_irqrestore(&port->lock, flags); + } + + ret = serial8250_handle_irq(port, iir); + + serial8250_rpm_put(up); + + return ret; +} + /* * Fill a struct uart_port for a given device node */ @@ -175,6 +215,9 @@ static int of_platform_serial_setup(struct platform_device *ofdev, if (ret) goto err_unprepare; break; + case PORT_LPC3220: + port->handle_irq = lpc32xx_handle_irq; + break; } if (IS_REACHABLE(CONFIG_SERIAL_8250_FSL) && diff --git a/drivers/tty/serial/imx.c b/drivers/tty/serial/imx.c index 60d48d857b1c..8e8391d879a3 100644 --- a/drivers/tty/serial/imx.c +++ b/drivers/tty/serial/imx.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include #include @@ -2018,6 +2019,9 @@ static const struct uart_ops imx_uart_pops = { static struct imx_port *imx_uart_ports[UART_NR]; +/* Held across uart_add/remove_one_port(); console callbacks must not take it. */ +static DEFINE_MUTEX(imx_uart_ports_lock); + #if IS_ENABLED(CONFIG_SERIAL_IMX_CONSOLE) static void imx_uart_console_putchar(struct uart_port *port, unsigned char ch) { @@ -2490,11 +2494,19 @@ static int imx_uart_probe(struct platform_device *pdev) } } - imx_uart_ports[sport->port.line] = sport; - platform_set_drvdata(pdev, sport); - ret = uart_add_one_port(&imx_uart_uart_driver, &sport->port); + scoped_guard(mutex, &imx_uart_ports_lock) { + if (imx_uart_ports[sport->port.line]) { + ret = -EBUSY; + } else { + imx_uart_ports[sport->port.line] = sport; + ret = uart_add_one_port(&imx_uart_uart_driver, + &sport->port); + if (ret) + imx_uart_ports[sport->port.line] = NULL; + } + } err_clk: clk_disable_unprepare(sport->clk_ipg); @@ -2502,13 +2514,13 @@ static int imx_uart_probe(struct platform_device *pdev) return ret; } -static int imx_uart_remove(struct platform_device *pdev) +static void imx_uart_remove(struct platform_device *pdev) { struct imx_port *sport = platform_get_drvdata(pdev); + guard(mutex)(&imx_uart_ports_lock); uart_remove_one_port(&imx_uart_uart_driver, &sport->port); - - return 0; + imx_uart_ports[sport->port.line] = NULL; } static void imx_uart_restore_context(struct imx_port *sport) @@ -2677,7 +2689,7 @@ static const struct dev_pm_ops imx_uart_pm_ops = { static struct platform_driver imx_uart_platform_driver = { .probe = imx_uart_probe, - .remove = imx_uart_remove, + .remove_new = imx_uart_remove, .driver = { .name = "imx-uart", diff --git a/drivers/tty/serial/ip22zilog.c b/drivers/tty/serial/ip22zilog.c index 845ff706bc59..570cbdf1bb5a 100644 --- a/drivers/tty/serial/ip22zilog.c +++ b/drivers/tty/serial/ip22zilog.c @@ -30,6 +30,7 @@ #include #include #include +#include #include #include @@ -50,8 +51,9 @@ #define ZSDELAY_LONG() udelay(20) #define ZS_WSYNC(channel) do { } while (0) -#define NUM_IP22ZILOG 1 -#define NUM_CHANNELS (NUM_IP22ZILOG * 2) +#define NUM_CHANNELS 2 +#define CHANNEL_B 0 +#define CHANNEL_A 1 #define ZS_CLOCK 3672000 /* Zilog input clock rate. */ #define ZS_CLOCK_DIVISOR 16 /* Divisor this driver uses. */ @@ -62,9 +64,6 @@ struct uart_ip22zilog_port { struct uart_port port; - /* IRQ servicing chain. */ - struct uart_ip22zilog_port *next; - /* Current values of Zilog write registers. */ unsigned char curregs[NUM_ZSREGS]; @@ -72,7 +71,6 @@ struct uart_ip22zilog_port { #define IP22ZILOG_FLAG_IS_CONS 0x00000004 #define IP22ZILOG_FLAG_IS_KGDB 0x00000008 #define IP22ZILOG_FLAG_MODEM_STATUS 0x00000010 -#define IP22ZILOG_FLAG_IS_CHANNEL_A 0x00000020 #define IP22ZILOG_FLAG_REGS_HELD 0x00000040 #define IP22ZILOG_FLAG_TX_STOPPED 0x00000080 #define IP22ZILOG_FLAG_TX_ACTIVE 0x00000100 @@ -84,6 +82,8 @@ struct uart_ip22zilog_port { unsigned char prev_status; }; +static struct uart_ip22zilog_port ip22zilog_port_table[NUM_CHANNELS]; + #define ZILOG_CHANNEL_FROM_PORT(PORT) ((struct zilog_channel *)((PORT)->membase)) #define UART_ZILOG(PORT) ((struct uart_ip22zilog_port *)(PORT)) #define IP22ZILOG_GET_CURR_REG(PORT, REGNUM) \ @@ -93,7 +93,6 @@ struct uart_ip22zilog_port { #define ZS_IS_CONS(UP) ((UP)->flags & IP22ZILOG_FLAG_IS_CONS) #define ZS_IS_KGDB(UP) ((UP)->flags & IP22ZILOG_FLAG_IS_KGDB) #define ZS_WANTS_MODEM_STATUS(UP) ((UP)->flags & IP22ZILOG_FLAG_MODEM_STATUS) -#define ZS_IS_CHANNEL_A(UP) ((UP)->flags & IP22ZILOG_FLAG_IS_CHANNEL_A) #define ZS_REGS_HELD(UP) ((UP)->flags & IP22ZILOG_FLAG_REGS_HELD) #define ZS_TX_STOPPED(UP) ((UP)->flags & IP22ZILOG_FLAG_TX_STOPPED) #define ZS_TX_ACTIVE(UP) ((UP)->flags & IP22ZILOG_FLAG_TX_ACTIVE) @@ -424,60 +423,57 @@ static void ip22zilog_transmit_chars(struct uart_ip22zilog_port *up, static irqreturn_t ip22zilog_interrupt(int irq, void *dev_id) { - struct uart_ip22zilog_port *up = dev_id; - - while (up) { - struct zilog_channel *channel - = ZILOG_CHANNEL_FROM_PORT(&up->port); - unsigned char r3; - bool push = false; - - spin_lock(&up->port.lock); - r3 = read_zsreg(channel, R3); + struct uart_ip22zilog_port *up; + struct zilog_channel *channel; + unsigned char r3; + bool push = false; - /* Channel A */ - if (r3 & (CHAEXT | CHATxIP | CHARxIP)) { - writeb(RES_H_IUS, &channel->control); - ZSDELAY(); - ZS_WSYNC(channel); + up = &ip22zilog_port_table[CHANNEL_A]; + channel = ZILOG_CHANNEL_FROM_PORT(&up->port); - if (r3 & CHARxIP) - push = ip22zilog_receive_chars(up, channel); - if (r3 & CHAEXT) - ip22zilog_status_handle(up, channel); - if (r3 & CHATxIP) - ip22zilog_transmit_chars(up, channel); - } - spin_unlock(&up->port.lock); + uart_port_lock(&up->port); + r3 = read_zsreg(channel, R3); - if (push) - tty_flip_buffer_push(&up->port.state->port); + /* Channel A */ + if (r3 & (CHAEXT | CHATxIP | CHARxIP)) { + writeb(RES_H_IUS, &channel->control); + ZSDELAY(); + ZS_WSYNC(channel); - /* Channel B */ - up = up->next; - channel = ZILOG_CHANNEL_FROM_PORT(&up->port); - push = false; + if (r3 & CHARxIP) + push = ip22zilog_receive_chars(up, channel); + if (r3 & CHAEXT) + ip22zilog_status_handle(up, channel); + if (r3 & CHATxIP) + ip22zilog_transmit_chars(up, channel); + } + uart_port_unlock(&up->port); - spin_lock(&up->port.lock); - if (r3 & (CHBEXT | CHBTxIP | CHBRxIP)) { - writeb(RES_H_IUS, &channel->control); - ZSDELAY(); - ZS_WSYNC(channel); + if (push) + tty_flip_buffer_push(&up->port.state->port); - if (r3 & CHBRxIP) - push = ip22zilog_receive_chars(up, channel); - if (r3 & CHBEXT) - ip22zilog_status_handle(up, channel); - if (r3 & CHBTxIP) - ip22zilog_transmit_chars(up, channel); - } - spin_unlock(&up->port.lock); + /* Channel B */ + up = &ip22zilog_port_table[CHANNEL_B]; + channel = ZILOG_CHANNEL_FROM_PORT(&up->port); + push = false; - if (push) - tty_flip_buffer_push(&up->port.state->port); + uart_port_lock(&up->port); + if (r3 & (CHBEXT | CHBTxIP | CHBRxIP)) { + writeb(RES_H_IUS, &channel->control); + ZSDELAY(); + ZS_WSYNC(channel); - up = up->next; + if (r3 & CHBRxIP) + push = ip22zilog_receive_chars(up, channel); + if (r3 & CHBEXT) + ip22zilog_status_handle(up, channel); + if (r3 & CHBTxIP) + ip22zilog_transmit_chars(up, channel); } + uart_port_unlock(&up->port); + + if (push) + tty_flip_buffer_push(&up->port.state->port); return IRQ_HANDLED; } @@ -504,11 +500,11 @@ static unsigned int ip22zilog_tx_empty(struct uart_port *port) unsigned char status; unsigned int ret; - spin_lock_irqsave(&port->lock, flags); + uart_port_lock_irqsave(port, &flags); status = ip22zilog_read_channel_status(port); - spin_unlock_irqrestore(&port->lock, flags); + uart_port_unlock_irqrestore(port, flags); if (status & Tx_BUF_EMP) ret = TIOCSER_TEMT; @@ -664,7 +660,7 @@ static void ip22zilog_break_ctl(struct uart_port *port, int break_state) else clear_bits |= SND_BRK; - spin_lock_irqsave(&port->lock, flags); + uart_port_lock_irqsave(port, &flags); new_reg = (up->curregs[R5] | set_bits) & ~clear_bits; if (new_reg != up->curregs[R5]) { @@ -674,7 +670,7 @@ static void ip22zilog_break_ctl(struct uart_port *port, int break_state) write_zsreg(channel, R5, up->curregs[R5]); } - spin_unlock_irqrestore(&port->lock, flags); + uart_port_unlock_irqrestore(port, flags); } static void __ip22zilog_reset(struct uart_ip22zilog_port *up) @@ -694,16 +690,16 @@ static void __ip22zilog_reset(struct uart_ip22zilog_port *up) udelay(100); } - if (!ZS_IS_CHANNEL_A(up)) { - up++; - channel = ZILOG_CHANNEL_FROM_PORT(&up->port); - } + up = &ip22zilog_port_table[CHANNEL_A]; + channel = ZILOG_CHANNEL_FROM_PORT(&up->port); + write_zsreg(channel, R9, FHWRES); ZSDELAY_LONG(); (void) read_zsreg(channel, R0); up->flags |= IP22ZILOG_FLAG_RESET_DONE; - up->next->flags |= IP22ZILOG_FLAG_RESET_DONE; + up = &ip22zilog_port_table[CHANNEL_B]; + up->flags |= IP22ZILOG_FLAG_RESET_DONE; } static void __ip22zilog_startup(struct uart_ip22zilog_port *up) @@ -735,9 +731,9 @@ static int ip22zilog_startup(struct uart_port *port) if (ZS_IS_CONS(up)) return 0; - spin_lock_irqsave(&port->lock, flags); + uart_port_lock_irqsave(port, &flags); __ip22zilog_startup(up); - spin_unlock_irqrestore(&port->lock, flags); + uart_port_unlock_irqrestore(port, flags); return 0; } @@ -775,7 +771,7 @@ static void ip22zilog_shutdown(struct uart_port *port) if (ZS_IS_CONS(up)) return; - spin_lock_irqsave(&port->lock, flags); + uart_port_lock_irqsave(port, &flags); channel = ZILOG_CHANNEL_FROM_PORT(port); @@ -788,7 +784,7 @@ static void ip22zilog_shutdown(struct uart_port *port) up->curregs[R5] &= ~SND_BRK; ip22zilog_maybe_update_regs(up, channel); - spin_unlock_irqrestore(&port->lock, flags); + uart_port_unlock_irqrestore(port, flags); } /* Shared by TTY driver and serial console setup. The port lock is held @@ -880,7 +876,7 @@ ip22zilog_set_termios(struct uart_port *port, struct ktermios *termios, baud = uart_get_baud_rate(port, termios, old, 1200, 76800); - spin_lock_irqsave(&up->port.lock, flags); + uart_port_lock_irqsave(&up->port, &flags); brg = BPS_TO_BRG(baud, ZS_CLOCK / ZS_CLOCK_DIVISOR); @@ -894,7 +890,7 @@ ip22zilog_set_termios(struct uart_port *port, struct ktermios *termios, ip22zilog_maybe_update_regs(up, ZILOG_CHANNEL_FROM_PORT(port)); uart_update_timeout(port, termios->c_cflag, baud); - spin_unlock_irqrestore(&up->port.lock, flags); + uart_port_unlock_irqrestore(&up->port, flags); } static const char *ip22zilog_type(struct uart_port *port) @@ -944,47 +940,6 @@ static const struct uart_ops ip22zilog_pops = { .verify_port = ip22zilog_verify_port, }; -static struct uart_ip22zilog_port *ip22zilog_port_table; -static struct zilog_layout **ip22zilog_chip_regs; - -static struct uart_ip22zilog_port *ip22zilog_irq_chain; -static int zilog_irq = -1; - -static void * __init alloc_one_table(unsigned long size) -{ - return kzalloc(size, GFP_KERNEL); -} - -static void __init ip22zilog_alloc_tables(void) -{ - ip22zilog_port_table = (struct uart_ip22zilog_port *) - alloc_one_table(NUM_CHANNELS * sizeof(struct uart_ip22zilog_port)); - ip22zilog_chip_regs = (struct zilog_layout **) - alloc_one_table(NUM_IP22ZILOG * sizeof(struct zilog_layout *)); - - if (ip22zilog_port_table == NULL || ip22zilog_chip_regs == NULL) { - panic("IP22-Zilog: Cannot allocate IP22-Zilog tables."); - } -} - -/* Get the address of the registers for IP22-Zilog instance CHIP. */ -static struct zilog_layout * __init get_zs(int chip) -{ - unsigned long base; - - if (chip < 0 || chip >= NUM_IP22ZILOG) { - panic("IP22-Zilog: Illegal chip number %d in get_zs.", chip); - } - - /* Not probe-able, hard code it. */ - base = (unsigned long) &sgioc->uart; - - zilog_irq = SGI_SERIAL_IRQ; - request_mem_region(base, 8, "IP22-Zilog"); - - return (struct zilog_layout *) base; -} - #define ZS_PUT_CHAR_MAX_DELAY 2000 /* 10 ms */ #ifdef CONFIG_SERIAL_IP22_ZILOG_CONSOLE @@ -1016,10 +971,10 @@ ip22zilog_console_write(struct console *con, const char *s, unsigned int count) struct uart_ip22zilog_port *up = &ip22zilog_port_table[con->index]; unsigned long flags; - spin_lock_irqsave(&up->port.lock, flags); + uart_port_lock_irqsave(&up->port, &flags); uart_console_write(&up->port, s, count, ip22zilog_put_char); udelay(2); - spin_unlock_irqrestore(&up->port.lock, flags); + uart_port_unlock_irqrestore(&up->port, flags); } static int __init ip22zilog_console_setup(struct console *con, char *options) @@ -1034,13 +989,13 @@ static int __init ip22zilog_console_setup(struct console *con, char *options) printk(KERN_INFO "Console: ttyS%d (IP22-Zilog)\n", con->index); - spin_lock_irqsave(&up->port.lock, flags); + uart_port_lock_irqsave(&up->port, &flags); up->curregs[R15] |= BRKIE; __ip22zilog_startup(up); - spin_unlock_irqrestore(&up->port.lock, flags); + uart_port_unlock_irqrestore(&up->port, flags); if (options) uart_parse_options(options, &baud, &parity, &bits, &flow); @@ -1072,144 +1027,123 @@ static struct uart_driver ip22zilog_reg = { #endif }; -static void __init ip22zilog_prepare(void) +static void __init ip22zilog_prepare(struct uart_ip22zilog_port *up) { unsigned char sysrq_on = IS_ENABLED(CONFIG_SERIAL_IP22_ZILOG_CONSOLE); + int brg; + + spin_lock_init(&up->port.lock); + + up->port.iotype = UPIO_MEM; + up->port.uartclk = ZS_CLOCK; + up->port.fifosize = 1; + up->port.has_sysrq = sysrq_on; + up->port.ops = &ip22zilog_pops; + up->port.type = PORT_IP22ZILOG; + + /* Normal serial TTY. */ + up->parity_mask = 0xff; + up->curregs[R1] = EXT_INT_ENAB | INT_ALL_Rx | TxINT_ENAB; + up->curregs[R4] = PAR_EVEN | X16CLK | SB1; + up->curregs[R3] = RxENAB | Rx8; + up->curregs[R5] = TxENAB | Tx8; + up->curregs[R9] = NV | MIE; + up->curregs[R10] = NRZ; + up->curregs[R11] = TCBR | RCBR; + brg = BPS_TO_BRG(9600, ZS_CLOCK / ZS_CLOCK_DIVISOR); + up->curregs[R12] = (brg & 0xff); + up->curregs[R13] = (brg >> 8) & 0xff; + up->curregs[R14] = BRENAB; +} + +static int ip22zilog_probe(struct platform_device *pdev) +{ struct uart_ip22zilog_port *up; - struct zilog_layout *rp; - int channel, chip; + char __iomem *membase; + struct resource *res; + int irq; + int i; - /* - * Temporary fix. - */ - for (channel = 0; channel < NUM_CHANNELS; channel++) - spin_lock_init(&ip22zilog_port_table[channel].port.lock); - - ip22zilog_irq_chain = &ip22zilog_port_table[NUM_CHANNELS - 1]; - up = &ip22zilog_port_table[0]; - for (channel = NUM_CHANNELS - 1 ; channel > 0; channel--) - up[channel].next = &up[channel - 1]; - up[channel].next = NULL; - - for (chip = 0; chip < NUM_IP22ZILOG; chip++) { - if (!ip22zilog_chip_regs[chip]) { - ip22zilog_chip_regs[chip] = rp = get_zs(chip); - - up[(chip * 2) + 0].port.membase = (char *) &rp->channelB; - up[(chip * 2) + 1].port.membase = (char *) &rp->channelA; - - /* In theory mapbase is the physical address ... */ - up[(chip * 2) + 0].port.mapbase = - (unsigned long) ioremap((unsigned long) &rp->channelB, 8); - up[(chip * 2) + 1].port.mapbase = - (unsigned long) ioremap((unsigned long) &rp->channelA, 8); - } + up = &ip22zilog_port_table[CHANNEL_B]; + if (up->port.dev) + return -ENOSPC; - /* Channel A */ - up[(chip * 2) + 0].port.iotype = UPIO_MEM; - up[(chip * 2) + 0].port.irq = zilog_irq; - up[(chip * 2) + 0].port.uartclk = ZS_CLOCK; - up[(chip * 2) + 0].port.fifosize = 1; - up[(chip * 2) + 0].port.has_sysrq = sysrq_on; - up[(chip * 2) + 0].port.ops = &ip22zilog_pops; - up[(chip * 2) + 0].port.type = PORT_IP22ZILOG; - up[(chip * 2) + 0].port.flags = 0; - up[(chip * 2) + 0].port.line = (chip * 2) + 0; - up[(chip * 2) + 0].flags = 0; - - /* Channel B */ - up[(chip * 2) + 1].port.iotype = UPIO_MEM; - up[(chip * 2) + 1].port.irq = zilog_irq; - up[(chip * 2) + 1].port.uartclk = ZS_CLOCK; - up[(chip * 2) + 1].port.fifosize = 1; - up[(chip * 2) + 1].port.has_sysrq = sysrq_on; - up[(chip * 2) + 1].port.ops = &ip22zilog_pops; - up[(chip * 2) + 1].port.type = PORT_IP22ZILOG; - up[(chip * 2) + 1].port.line = (chip * 2) + 1; - up[(chip * 2) + 1].flags |= IP22ZILOG_FLAG_IS_CHANNEL_A; - } + irq = platform_get_irq(pdev, 0); + if (irq < 0) + return irq; - for (channel = 0; channel < NUM_CHANNELS; channel++) { - struct uart_ip22zilog_port *up = &ip22zilog_port_table[channel]; - int brg; + membase = devm_platform_get_and_ioremap_resource(pdev, 0, &res); + if (IS_ERR(membase)) + return PTR_ERR(membase); - /* Normal serial TTY. */ - up->parity_mask = 0xff; - up->curregs[R1] = EXT_INT_ENAB | INT_ALL_Rx | TxINT_ENAB; - up->curregs[R4] = PAR_EVEN | X16CLK | SB1; - up->curregs[R3] = RxENAB | Rx8; - up->curregs[R5] = TxENAB | Tx8; - up->curregs[R9] = NV | MIE; - up->curregs[R10] = NRZ; - up->curregs[R11] = TCBR | RCBR; - brg = BPS_TO_BRG(9600, ZS_CLOCK / ZS_CLOCK_DIVISOR); - up->curregs[R12] = (brg & 0xff); - up->curregs[R13] = (brg >> 8) & 0xff; - up->curregs[R14] = BRENAB; - } -} + ip22zilog_prepare(up); -static int __init ip22zilog_ports_init(void) -{ - int ret; + up->port.mapbase = res->start + offsetof(struct zilog_layout, channelB); + up->port.membase = membase + offsetof(struct zilog_layout, channelB); + up->port.line = 0; + up->port.dev = &pdev->dev; + up->port.irq = irq; - printk(KERN_INFO "Serial: IP22 Zilog driver (%d chips).\n", NUM_IP22ZILOG); + up = &ip22zilog_port_table[CHANNEL_A]; + ip22zilog_prepare(up); - ip22zilog_prepare(); + up->port.mapbase = res->start + offsetof(struct zilog_layout, channelA); + up->port.membase = membase + offsetof(struct zilog_layout, channelA); + up->port.line = 1; + up->port.dev = &pdev->dev; + up->port.irq = irq; - if (request_irq(zilog_irq, ip22zilog_interrupt, 0, - "IP22-Zilog", ip22zilog_irq_chain)) { + if (request_irq(irq, ip22zilog_interrupt, 0, + "IP22-Zilog", NULL)) { panic("IP22-Zilog: Unable to register zs interrupt handler.\n"); } - ret = uart_register_driver(&ip22zilog_reg); - if (ret == 0) { - int i; - - for (i = 0; i < NUM_CHANNELS; i++) { - struct uart_ip22zilog_port *up = &ip22zilog_port_table[i]; - - uart_add_one_port(&ip22zilog_reg, &up->port); - } - } - - return ret; -} - -static int __init ip22zilog_init(void) -{ - /* IP22 Zilog setup is hard coded, no probing to do. */ - ip22zilog_alloc_tables(); - ip22zilog_ports_init(); + for (i = 0; i < NUM_CHANNELS; i++) + uart_add_one_port(&ip22zilog_reg, + &ip22zilog_port_table[i].port); return 0; } -static void __exit ip22zilog_exit(void) +static void ip22zilog_remove(struct platform_device *pdev) { int i; - struct uart_ip22zilog_port *up; for (i = 0; i < NUM_CHANNELS; i++) { - up = &ip22zilog_port_table[i]; - - uart_remove_one_port(&ip22zilog_reg, &up->port); + uart_remove_one_port(&ip22zilog_reg, + &ip22zilog_port_table[i].port); + ip22zilog_port_table[i].port.dev = NULL; } +} - /* Free IO mem */ - up = &ip22zilog_port_table[0]; - for (i = 0; i < NUM_IP22ZILOG; i++) { - if (up[(i * 2) + 0].port.mapbase) { - iounmap((void*)up[(i * 2) + 0].port.mapbase); - up[(i * 2) + 0].port.mapbase = 0; - } - if (up[(i * 2) + 1].port.mapbase) { - iounmap((void*)up[(i * 2) + 1].port.mapbase); - up[(i * 2) + 1].port.mapbase = 0; - } +static struct platform_driver ip22zilog_driver = { + .probe = ip22zilog_probe, + .remove = ip22zilog_remove, + .driver = { + .name = "ip22zilog" } +}; + +static int __init ip22zilog_init(void) +{ + int ret; + + ret = uart_register_driver(&ip22zilog_reg); + if (ret) + return ret; + + ret = platform_driver_register(&ip22zilog_driver); + if (ret) + uart_unregister_driver(&ip22zilog_reg); + return ret; + +} + +static void __exit ip22zilog_exit(void) +{ uart_unregister_driver(&ip22zilog_reg); + platform_driver_unregister(&ip22zilog_driver); } module_init(ip22zilog_init); diff --git a/drivers/tty/serial/kgdboc.c b/drivers/tty/serial/kgdboc.c index 58ea1e1391ce..10f55aa4e18b 100644 --- a/drivers/tty/serial/kgdboc.c +++ b/drivers/tty/serial/kgdboc.c @@ -190,6 +190,11 @@ static void cleanup_kgdboc(void) return; kgdboc_unregister_kbd(); kgdb_unregister_io_module(&kgdboc_io_ops); + + if (kgdb_tty_driver) { + tty_driver_kref_put(kgdb_tty_driver); + kgdb_tty_driver = NULL; + } } static int configure_kgdboc(void) @@ -262,6 +267,10 @@ static int configure_kgdboc(void) kgdb_unregister_io_module(&kgdboc_io_ops); noconfig: kgdboc_unregister_kbd(); + if (kgdb_tty_driver) { + tty_driver_kref_put(kgdb_tty_driver); + kgdb_tty_driver = NULL; + } configured = 0; return err; diff --git a/drivers/tty/serial/ma35d1_serial.c b/drivers/tty/serial/ma35d1_serial.c index 6e5458b08554..225feb5868e8 100644 --- a/drivers/tty/serial/ma35d1_serial.c +++ b/drivers/tty/serial/ma35d1_serial.c @@ -725,7 +725,7 @@ static int ma35d1serial_probe(struct platform_device *pdev) ret = clk_prepare_enable(up->clk); if (ret) - goto err_iounmap; + goto err_put_clk; if (up->port.line != 0) up->port.uartclk = clk_get_rate(up->clk); @@ -752,6 +752,9 @@ static int ma35d1serial_probe(struct platform_device *pdev) err_clk_disable: clk_disable_unprepare(up->clk); +err_put_clk: + clk_put(up->clk); + err_iounmap: iounmap(up->port.membase); return ret; @@ -760,14 +763,14 @@ static int ma35d1serial_probe(struct platform_device *pdev) /* * Remove serial ports registered against a platform device. */ -static int ma35d1serial_remove(struct platform_device *dev) +static void ma35d1serial_remove(struct platform_device *dev) { struct uart_port *port = platform_get_drvdata(dev); struct uart_ma35d1_port *up = to_ma35d1_uart_port(port); uart_remove_one_port(&ma35d1serial_reg, port); clk_disable_unprepare(up->clk); - return 0; + clk_put(up->clk); } static int ma35d1serial_suspend(struct platform_device *dev, pm_message_t state) @@ -800,7 +803,7 @@ static int ma35d1serial_resume(struct platform_device *dev) static struct platform_driver ma35d1serial_driver = { .probe = ma35d1serial_probe, - .remove = ma35d1serial_remove, + .remove_new = ma35d1serial_remove, .suspend = ma35d1serial_suspend, .resume = ma35d1serial_resume, .driver = { diff --git a/drivers/tty/serial/max3100.c b/drivers/tty/serial/max3100.c index 3d2b83d6ab51..ec230c2f081d 100644 --- a/drivers/tty/serial/max3100.c +++ b/drivers/tty/serial/max3100.c @@ -567,11 +567,11 @@ static void max3100_shutdown(struct uart_port *port) del_timer_sync(&s->timer); if (s->workqueue) { + free_irq(s->irq, s); + cancel_work_sync(&s->work); destroy_workqueue(s->workqueue); s->workqueue = NULL; } - if (s->irq) - free_irq(s->irq, s); /* set shutdown mode to save power */ if (s->max3100_hw_suspend) @@ -830,6 +830,15 @@ static void max3100_remove(struct spi_device *spi) if (max3100s[i] == s) { dev_dbg(&spi->dev, "%s: removing port %d\n", __func__, i); uart_remove_one_port(&max3100_uart_driver, &max3100s[i]->port); + + s->force_end_work = 1; + timer_shutdown_sync(&s->timer); + if (s->workqueue) { + free_irq(s->irq, s); + cancel_work_sync(&s->work); + destroy_workqueue(s->workqueue); + s->workqueue = NULL; + } kfree(max3100s[i]); max3100s[i] = NULL; break; diff --git a/drivers/tty/serial/qcom_geni_serial.c b/drivers/tty/serial/qcom_geni_serial.c index 91c502e833e1..46145dda5e93 100644 --- a/drivers/tty/serial/qcom_geni_serial.c +++ b/drivers/tty/serial/qcom_geni_serial.c @@ -1304,9 +1304,8 @@ static void qcom_geni_serial_set_termios(struct uart_port *uport, * Bump up BW vote on CPU and CORE path as driver supports FIFO mode * only. */ - avg_bw_core = (baud > 115200) ? Bps_to_icc(CORE_2X_50_MHZ) - : GENI_DEFAULT_BW; - port->se.icc_paths[GENI_TO_CORE].avg_bw = avg_bw_core; + avg_bw_core = baud > 115200 ? CORE_2X_50_MHZ : CORE_2X_19_2_MHZ; + port->se.icc_paths[GENI_TO_CORE].avg_bw = Bps_to_icc(avg_bw_core); port->se.icc_paths[CPU_TO_GENI].avg_bw = Bps_to_icc(baud); geni_icc_set_bw(&port->se); diff --git a/drivers/tty/serial/sc16is7xx.c b/drivers/tty/serial/sc16is7xx.c index 02b74a695c8e..ac2380978ae7 100644 --- a/drivers/tty/serial/sc16is7xx.c +++ b/drivers/tty/serial/sc16is7xx.c @@ -336,6 +336,7 @@ struct sc16is7xx_one { struct kthread_work reg_work; struct kthread_delayed_work ms_work; struct sc16is7xx_one_config config; + unsigned char buf[SC16IS7XX_FIFO_SIZE]; /* Rx buffer. */ unsigned int old_mctrl; u8 old_lcr; /* Value before EFR access. */ bool irda_mode; @@ -349,7 +350,6 @@ struct sc16is7xx_port { unsigned long gpio_valid_mask; #endif u8 mctrl_mask; - unsigned char buf[SC16IS7XX_FIFO_SIZE]; struct kthread_worker kworker; struct task_struct *kworker_task; struct sc16is7xx_one p[]; @@ -382,17 +382,15 @@ static void sc16is7xx_port_write(struct uart_port *port, u8 reg, u8 val) regmap_write(one->regmap, reg, val); } -static void sc16is7xx_fifo_read(struct uart_port *port, unsigned int rxlen) +static void sc16is7xx_fifo_read(struct uart_port *port, u8 *rxbuf, unsigned int rxlen) { - struct sc16is7xx_port *s = dev_get_drvdata(port->dev); struct sc16is7xx_one *one = to_sc16is7xx_one(port, port); - regmap_noinc_read(one->regmap, SC16IS7XX_RHR_REG, s->buf, rxlen); + regmap_noinc_read(one->regmap, SC16IS7XX_RHR_REG, rxbuf, rxlen); } -static void sc16is7xx_fifo_write(struct uart_port *port, u8 to_send) +static void sc16is7xx_fifo_write(struct uart_port *port, u8 *txbuf, u8 to_send) { - struct sc16is7xx_port *s = dev_get_drvdata(port->dev); struct sc16is7xx_one *one = to_sc16is7xx_one(port, port); /* @@ -402,7 +400,7 @@ static void sc16is7xx_fifo_write(struct uart_port *port, u8 to_send) if (unlikely(!to_send)) return; - regmap_noinc_write(one->regmap, SC16IS7XX_THR_REG, s->buf, to_send); + regmap_noinc_write(one->regmap, SC16IS7XX_THR_REG, txbuf, to_send); } static void sc16is7xx_port_update(struct uart_port *port, u8 reg, @@ -618,18 +616,18 @@ static int sc16is7xx_set_baud(struct uart_port *port, int baud) static void sc16is7xx_handle_rx(struct uart_port *port, unsigned int rxlen, unsigned int iir) { - struct sc16is7xx_port *s = dev_get_drvdata(port->dev); + struct sc16is7xx_one *one = to_sc16is7xx_one(port, port); unsigned int lsr = 0, bytes_read, i; bool read_lsr = (iir == SC16IS7XX_IIR_RLSE_SRC) ? true : false; u8 ch, flag; - if (unlikely(rxlen >= sizeof(s->buf))) { + if (unlikely(rxlen >= sizeof(one->buf))) { dev_warn_ratelimited(port->dev, "ttySC%i: Possible RX FIFO overrun: %d\n", port->line, rxlen); port->icount.buf_overrun++; /* Ensure sanity of RX level */ - rxlen = sizeof(s->buf); + rxlen = sizeof(one->buf); } while (rxlen) { @@ -642,10 +640,10 @@ static void sc16is7xx_handle_rx(struct uart_port *port, unsigned int rxlen, lsr = 0; if (read_lsr) { - s->buf[0] = sc16is7xx_port_read(port, SC16IS7XX_RHR_REG); + one->buf[0] = sc16is7xx_port_read(port, SC16IS7XX_RHR_REG); bytes_read = 1; } else { - sc16is7xx_fifo_read(port, rxlen); + sc16is7xx_fifo_read(port, one->buf, rxlen); bytes_read = rxlen; } @@ -678,7 +676,7 @@ static void sc16is7xx_handle_rx(struct uart_port *port, unsigned int rxlen, } for (i = 0; i < bytes_read; ++i) { - ch = s->buf[i]; + ch = one->buf[i]; if (uart_handle_sysrq_char(port, ch)) continue; @@ -696,9 +694,8 @@ static void sc16is7xx_handle_rx(struct uart_port *port, unsigned int rxlen, static void sc16is7xx_handle_tx(struct uart_port *port) { - struct sc16is7xx_port *s = dev_get_drvdata(port->dev); struct circ_buf *xmit = &port->state->xmit; - unsigned int txlen, to_send, i; + unsigned int txlen, to_send; unsigned long flags; if (unlikely(port->x_char)) { @@ -728,13 +725,11 @@ static void sc16is7xx_handle_tx(struct uart_port *port) } to_send = (to_send > txlen) ? txlen : to_send; - /* Convert to linear buffer */ - for (i = 0; i < to_send; ++i) { - s->buf[i] = xmit->buf[xmit->tail]; - uart_xmit_advance(port, 1); - } + to_send = min_t(unsigned int, to_send, + CIRC_CNT_TO_END(xmit->head, xmit->tail, UART_XMIT_SIZE)); - sc16is7xx_fifo_write(port, to_send); + sc16is7xx_fifo_write(port, &xmit->buf[xmit->tail], to_send); + uart_xmit_advance(port, to_send); } uart_port_lock_irqsave(port, &flags); diff --git a/drivers/tty/serial/serial-tegra.c b/drivers/tty/serial/serial-tegra.c index d4ec943cb8e9..53d7da8adc8a 100644 --- a/drivers/tty/serial/serial-tegra.c +++ b/drivers/tty/serial/serial-tegra.c @@ -306,11 +306,19 @@ static void tegra_uart_fifo_reset(struct tegra_uart_port *tup, u8 fcr_bits) { unsigned long fcr = tup->fcr_shadow; unsigned int lsr, tmout = 10000; + bool clear_tx = !!(fcr_bits & UART_FCR_CLEAR_XMIT); if (tup->rts_active) set_rts(tup, false); - if (tup->cdata->allow_txfifo_reset_fifo_mode) { + /* + * Leaving FIFO mode below is a workaround for a Tegra30 + * restriction on clearing the Tx FIFO while FIFO mode is + * enabled. It empties both FIFOs, so applying it to an + * Rx-only reset would destroy an in-flight transmission. + * Only take that path when the caller asked for CLEAR_XMIT. + */ + if (tup->cdata->allow_txfifo_reset_fifo_mode || !clear_tx) { fcr |= fcr_bits & (UART_FCR_CLEAR_RCVR | UART_FCR_CLEAR_XMIT); tegra_uart_write(tup, fcr, UART_FCR); } else { @@ -335,9 +343,15 @@ static void tegra_uart_fifo_reset(struct tegra_uart_port *tup, u8 fcr_bits) */ tegra_uart_wait_cycle_time(tup, 32); + /* + * Only wait for the transmitter to drain when the Tx FIFO was + * part of the reset. For an Rx-only reset it is left intact, + * and waiting for TEMT here would spin for a full frame time. + */ do { lsr = tegra_uart_read(tup, UART_LSR); - if ((lsr & UART_LSR_TEMT) && !(lsr & UART_LSR_DR)) + if ((!clear_tx || (lsr & UART_LSR_TEMT)) && + !(lsr & UART_LSR_DR)) break; udelay(1); } while (--tmout); diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c index c3a51bd59ff7..d3abab6177b1 100644 --- a/drivers/tty/serial/serial_core.c +++ b/drivers/tty/serial/serial_core.c @@ -643,7 +643,10 @@ static unsigned int uart_write_room(struct tty_struct *tty) unsigned int ret; port = uart_port_lock(state, flags); - ret = uart_circ_chars_free(&state->xmit); + if (!state->xmit.buf) + ret = 0; + else + ret = uart_circ_chars_free(&state->xmit); uart_port_unlock(port, flags); return ret; } @@ -1253,12 +1256,12 @@ static int uart_wait_modem_status(struct uart_state *state, unsigned long arg) add_wait_queue(&port->delta_msr_wait, &wait); for (;;) { + set_current_state(TASK_INTERRUPTIBLE); + spin_lock_irq(&uport->lock); memcpy(&cnow, &uport->icount, sizeof(struct uart_icount)); spin_unlock_irq(&uport->lock); - set_current_state(TASK_INTERRUPTIBLE); - if (((arg & TIOCM_RNG) && (cnow.rng != cprev.rng)) || ((arg & TIOCM_DSR) && (cnow.dsr != cprev.dsr)) || ((arg & TIOCM_CD) && (cnow.dcd != cprev.dcd)) || @@ -2874,8 +2877,10 @@ int uart_register_driver(struct uart_driver *drv) for (i = 0; i < drv->nr; i++) tty_port_destroy(&drv->state[i].port); tty_driver_kref_put(normal); + drv->tty_driver = NULL; out_kfree: kfree(drv->state); + drv->state = NULL; out: return retval; } @@ -3155,7 +3160,6 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u struct tty_port *port; int ret = 0; struct device *tty_dev; - int num_groups; if (uport->line >= drv->nr) return -EINVAL; @@ -3169,6 +3173,25 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u goto out; } + uport->name = kasprintf(GFP_KERNEL, "%s%d", drv->dev_name, + drv->tty_driver->name_base + uport->line); + if (!uport->name) { + ret = -ENOMEM; + goto out; + } + + /* + * uart_configure_port() may set uport->attr_group and register the + * console. Allocate room for both groups and a NULL terminator first. + */ + uport->tty_groups = kcalloc(3, sizeof(*uport->tty_groups), GFP_KERNEL); + if (!uport->tty_groups) { + kfree(uport->name); + ret = -ENOMEM; + goto out; + } + uport->tty_groups[0] = &tty_dev_attr_group; + /* Link the port to the driver state table and vice versa */ atomic_set(&state->refcount, 1); init_waitqueue_head(&state->remove_wait); @@ -3178,12 +3201,6 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u state->pm_state = UART_PM_STATE_UNDEFINED; uport->cons = drv->cons; uport->minor = drv->tty_driver->minor_start + uport->line; - uport->name = kasprintf(GFP_KERNEL, "%s%d", drv->dev_name, - drv->tty_driver->name_base + uport->line); - if (!uport->name) { - ret = -ENOMEM; - goto out; - } /* * If this port is in use as a console then the spinlock is already @@ -3200,17 +3217,6 @@ static int serial_core_add_one_port(struct uart_driver *drv, struct uart_port *u port->console = uart_console(uport); - num_groups = 2; - if (uport->attr_group) - num_groups++; - - uport->tty_groups = kcalloc(num_groups, sizeof(*uport->tty_groups), - GFP_KERNEL); - if (!uport->tty_groups) { - ret = -ENOMEM; - goto out; - } - uport->tty_groups[0] = &tty_dev_attr_group; if (uport->attr_group) uport->tty_groups[1] = uport->attr_group; diff --git a/drivers/tty/serial/vt8500_serial.c b/drivers/tty/serial/vt8500_serial.c index c5d5c2765119..4aa4efdedb7f 100644 --- a/drivers/tty/serial/vt8500_serial.c +++ b/drivers/tty/serial/vt8500_serial.c @@ -656,6 +656,7 @@ static int vt8500_serial_probe(struct platform_device *pdev) ret = clk_prepare_enable(vt8500_port->clk); if (ret) { dev_err(&pdev->dev, "failed to enable clock\n"); + clk_put(vt8500_port->clk); return ret; } diff --git a/drivers/tty/tty_io.c b/drivers/tty/tty_io.c index 6781bac852cf..817370cc4165 100644 --- a/drivers/tty/tty_io.c +++ b/drivers/tty/tty_io.c @@ -629,6 +629,8 @@ static void __tty_hangup(struct tty_struct *tty, int exit_session) tty_ldisc_hangup(tty, cons_filp != NULL); + wake_up_interruptible(&tty->break_wait); + spin_lock_irq(&tty->ctrl.lock); clear_bit(TTY_THROTTLED, &tty->flags); clear_bit(TTY_DO_WRITE_WAKEUP, &tty->flags); @@ -2460,6 +2462,7 @@ static int tiocgetd(struct tty_struct *tty, int __user *p) * send_break - performed time break * @tty: device to break on * @duration: timeout in mS + * @file: file object * * Perform a timed break on hardware that lacks its own driver level timed * break functionality. @@ -2467,8 +2470,9 @@ static int tiocgetd(struct tty_struct *tty, int __user *p) * Locking: * @tty->atomic_write_lock serializes */ -static int send_break(struct tty_struct *tty, unsigned int duration) +static int send_break(struct file *file, struct tty_struct *tty, unsigned int duration) { + long timeout; int retval; if (tty->ops->break_ctl == NULL) @@ -2482,13 +2486,26 @@ static int send_break(struct tty_struct *tty, unsigned int duration) return -EINTR; retval = tty->ops->break_ctl(tty, -1); - if (!retval) { - msleep_interruptible(duration); - retval = tty->ops->break_ctl(tty, 0); - } else if (retval == -EOPNOTSUPP) { - /* some drivers can tell only dynamically */ - retval = 0; + if (retval) { + if (retval == -EOPNOTSUPP) { + /* some drivers can tell only dynamically */ + retval = 0; + } + goto out_unlock; + } + + timeout = msecs_to_jiffies(duration); + timeout = wait_event_interruptible_timeout(tty->break_wait, + tty_hung_up_p(file), + timeout); + /* return early on hangup only */ + if (timeout > 0) { + retval = -EIO; + goto out_unlock; } + + retval = tty->ops->break_ctl(tty, 0); +out_unlock: tty_write_unlock(tty); if (signal_pending(current)) @@ -2741,10 +2758,10 @@ long tty_ioctl(struct file *file, unsigned int cmd, unsigned long arg) * This is used by the tcdrain() termios function. */ if (!arg) - return send_break(tty, 250); + return send_break(file, tty, 250); return 0; case TCSBRKP: /* support for POSIX tcsendbreak() */ - return send_break(tty, arg ? arg*100 : 250); + return send_break(file, tty, arg ? arg * 100 : 250); case TIOCMGET: return tty_tiocmget(tty, p); @@ -3107,6 +3124,7 @@ struct tty_struct *alloc_tty_struct(struct tty_driver *driver, int idx) init_ldsem(&tty->ldisc_sem); init_waitqueue_head(&tty->write_wait); init_waitqueue_head(&tty->read_wait); + init_waitqueue_head(&tty->break_wait); INIT_WORK(&tty->hangup_work, do_tty_hangup); mutex_init(&tty->atomic_write_lock); spin_lock_init(&tty->ctrl.lock); @@ -3255,14 +3273,16 @@ struct device *tty_register_device_attr(struct tty_driver *driver, goto err_put; if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC)) { - /* - * Free any saved termios data so that the termios state is - * reset when reusing a minor number. - */ - tp = driver->termios[index]; - if (tp) { - driver->termios[index] = NULL; - kfree(tp); + if (driver->flags & TTY_DRIVER_RESET_SAVED_TERMIOS) { + /* + * Free any saved termios data so that the termios + * state is reset when reusing a minor number. + */ + tp = driver->termios[index]; + if (tp) { + driver->termios[index] = NULL; + kfree(tp); + } } retval = tty_cdev_add(driver, devt, index, 1); @@ -3297,7 +3317,7 @@ EXPORT_SYMBOL_GPL(tty_register_device_attr); void tty_unregister_device(struct tty_driver *driver, unsigned index) { device_destroy(&tty_class, MKDEV(driver->major, driver->minor_start) + index); - if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC)) { + if (!(driver->flags & TTY_DRIVER_DYNAMIC_ALLOC) && driver->cdevs[index]) { cdev_del(driver->cdevs[index]); driver->cdevs[index] = NULL; } diff --git a/drivers/tty/tty_port.c b/drivers/tty/tty_port.c index 21430e64b412..75a2f1a61e63 100644 --- a/drivers/tty/tty_port.c +++ b/drivers/tty/tty_port.c @@ -348,21 +348,12 @@ void tty_port_tty_set(struct tty_port *port, struct tty_struct *tty) } EXPORT_SYMBOL(tty_port_tty_set); -/** - * tty_port_shutdown - internal helper to shutdown the device - * @port: tty port to be shut down - * @tty: the associated tty - * - * It is used by tty_port_hangup() and tty_port_close(). Its task is to - * shutdown the device if it was initialized (note consoles remain - * functioning). It lowers DTR/RTS (if @tty has HUPCL set) and invokes - * @port->ops->shutdown(). - */ -static void tty_port_shutdown(struct tty_port *port, struct tty_struct *tty) +static void tty_port_shutdown_locked(struct tty_port *port, struct tty_struct *tty) { - mutex_lock(&port->mutex); + lockdep_assert_held(&port->mutex); + if (port->console) - goto out; + return; if (tty_port_initialized(port)) { tty_port_set_initialized(port, false); @@ -376,7 +367,22 @@ static void tty_port_shutdown(struct tty_port *port, struct tty_struct *tty) if (port->ops->shutdown) port->ops->shutdown(port); } -out: +} + +/** + * tty_port_shutdown - internal helper to shutdown the device + * @port: tty port to be shut down + * @tty: the associated tty + * + * It is used by tty_port_hangup() and tty_port_close(). Its task is to + * shutdown the device if it was initialized (note consoles remain + * functioning). It lowers DTR/RTS (if @tty has HUPCL set) and invokes + * @port->ops->shutdown(). + */ +static void tty_port_shutdown(struct tty_port *port, struct tty_struct *tty) +{ + mutex_lock(&port->mutex); + tty_port_shutdown_locked(port, tty); mutex_unlock(&port->mutex); } @@ -394,6 +400,7 @@ void tty_port_hangup(struct tty_port *port) struct tty_struct *tty; unsigned long flags; + mutex_lock(&port->mutex); spin_lock_irqsave(&port->lock, flags); port->count = 0; tty = port->tty; @@ -402,26 +409,47 @@ void tty_port_hangup(struct tty_port *port) port->tty = NULL; spin_unlock_irqrestore(&port->lock, flags); tty_port_set_active(port, false); - tty_port_shutdown(port, tty); + tty_port_shutdown_locked(port, tty); + mutex_unlock(&port->mutex); tty_kref_put(tty); wake_up_interruptible(&port->open_wait); wake_up_interruptible(&port->delta_msr_wait); } EXPORT_SYMBOL(tty_port_hangup); -void __tty_port_tty_hangup(struct tty_port *port, bool check_clocal, bool async) +/** + * tty_port_tty_hangup - helper to hang up a tty asynchronously + * @port: tty port + * @check_clocal: hang only ttys with %CLOCAL unset? + */ +void tty_port_tty_hangup(struct tty_port *port, bool check_clocal) { struct tty_struct *tty = tty_port_tty_get(port); - if (tty && (!check_clocal || !C_CLOCAL(tty))) { - if (async) - tty_hangup(tty); - else - tty_vhangup(tty); - } + if (tty && (!check_clocal || !C_CLOCAL(tty))) + tty_hangup(tty); tty_kref_put(tty); } -EXPORT_SYMBOL_GPL(__tty_port_tty_hangup); +EXPORT_SYMBOL_GPL(tty_port_tty_hangup); + +/** + * tty_port_tty_vhangup - helper to hang up a tty synchronously + * @port: tty port + */ +void tty_port_tty_vhangup(struct tty_port *port) +{ + struct tty_struct *tty; + + mutex_lock(&port->mutex); + tty = tty_port_tty_get(port); + mutex_unlock(&port->mutex); + + if (tty) { + tty_vhangup(tty); + tty_kref_put(tty); + } +} +EXPORT_SYMBOL_GPL(tty_port_tty_vhangup); /** * tty_port_tty_wakeup - helper to wake up a tty @@ -781,6 +809,7 @@ int tty_port_open(struct tty_port *port, struct tty_struct *tty, int retval = port->ops->activate(port, tty); if (retval) { + set_bit(TTY_IO_ERROR, &tty->flags); mutex_unlock(&port->mutex); return retval; } diff --git a/drivers/tty/vcc.c b/drivers/tty/vcc.c index 5b625f20233b..8a27376bc924 100644 --- a/drivers/tty/vcc.c +++ b/drivers/tty/vcc.c @@ -492,7 +492,7 @@ static ssize_t domain_show(struct device *dev, static int vcc_send_ctl(struct vcc_port *port, int ctl) { - struct vio_vcc pkt; + struct vio_vcc pkt = {}; int rv; pkt.tag.type = VIO_TYPE_CTRL; @@ -986,7 +986,8 @@ static void vcc_cleanup(struct tty_struct *tty) port = vcc_get(tty->index, true); if (port) { - port->tty = NULL; + scoped_guard(spinlock_irqsave, &port->lock) + port->tty = NULL; if (port->removed) { vcc_table_remove(tty->index); diff --git a/drivers/tty/vt/vc_screen.c b/drivers/tty/vt/vc_screen.c index 5c417b6426bb..a7b3fac5304d 100644 --- a/drivers/tty/vt/vc_screen.c +++ b/drivers/tty/vt/vc_screen.c @@ -643,6 +643,19 @@ vcs_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos) ret = copy_from_user(con_buf, buf, this_round); console_lock(); + /* The vc might have been freed or vcs_size might have changed + * while we slept to grab the user buffer; recheck here, before + * if (ret), so every break path below passes a fresh vc to the + * post-loop vcs_scr_updated(). Return data written so far. + */ + vc = vcs_vc(inode, &viewed); + if (!vc) { + if (written) + break; + ret = -ENXIO; + goto unlock_out; + } + if (ret) { this_round -= ret; if (!this_round) { @@ -655,18 +668,6 @@ vcs_write(struct file *file, const char __user *buf, size_t count, loff_t *ppos) goto unlock_out; } } - - /* The vc might have been freed or vcs_size might have changed - * while we slept to grab the user buffer, so recheck. - * Return data written up to now on failure. - */ - vc = vcs_vc(inode, &viewed); - if (!vc) { - if (written) - break; - ret = -ENXIO; - goto unlock_out; - } size = vcs_size(vc, attr, false); if (size < 0) { if (written) diff --git a/drivers/tty/vt/vt.c b/drivers/tty/vt/vt.c index 60c878ea95f9..fd56dd2c4cc1 100644 --- a/drivers/tty/vt/vt.c +++ b/drivers/tty/vt/vt.c @@ -1109,6 +1109,8 @@ int vc_allocate(unsigned int currcons) /* return 0 on success */ return 0; err_free: visual_deinit(vc); + if (*vc->uni_pagedict_loc) + con_free_unimap(vc); kfree(vc); vc_cons[currcons].d = NULL; return err; @@ -2497,7 +2499,8 @@ static void do_con_trol(struct tty_struct *tty, struct vc_data *vc, int c) csi_J(vc, 2); vc->vc_video_erase_char = (vc->vc_video_erase_char & 0xff00) | ' '; - do_update_region(vc, vc->vc_origin, vc->vc_screenbuf_size / 2); + if (con_should_update(vc)) + do_update_region(vc, vc->vc_origin, vc->vc_screenbuf_size / 2); } return; case ESsetG0: diff --git a/drivers/ufs/core/ufs-mcq.c b/drivers/ufs/core/ufs-mcq.c index cb8f0652a4be..0b930160a430 100644 --- a/drivers/ufs/core/ufs-mcq.c +++ b/drivers/ufs/core/ufs-mcq.c @@ -255,9 +255,7 @@ EXPORT_SYMBOL_GPL(ufshcd_mcq_write_cqis); * Current MCQ specification doesn't provide a Task Tag or its equivalent in * the Completion Queue Entry. Find the Task Tag using an indirect method. */ -static int ufshcd_mcq_get_tag(struct ufs_hba *hba, - struct ufs_hw_queue *hwq, - struct cq_entry *cqe) +static int ufshcd_mcq_get_tag(struct ufs_hba *hba, struct cq_entry *cqe) { u64 addr; @@ -271,19 +269,34 @@ static int ufshcd_mcq_get_tag(struct ufs_hba *hba, return div_u64(addr, ufshcd_get_ucd_size(hba)); } +static void ufshcd_mcq_compl_cqe(struct ufs_hba *hba, struct cq_entry *cqe) +{ + int tag = ufshcd_mcq_get_tag(hba, cqe); + + ufshcd_compl_one_cqe(hba, tag, cqe); + /* After processing the CQE, mark it as an empty (invalid) entry. */ + cqe->command_desc_base_addr = 0; +} + static void ufshcd_mcq_process_cqe(struct ufs_hba *hba, struct ufs_hw_queue *hwq) { struct cq_entry *cqe = ufshcd_mcq_cur_cqe(hwq); - int tag = ufshcd_mcq_get_tag(hba, hwq, cqe); - if (cqe->command_desc_base_addr) { - ufshcd_compl_one_cqe(hba, tag, cqe); - /* After processed the cqe, mark it empty (invalid) entry */ - cqe->command_desc_base_addr = 0; - } + if (cqe->command_desc_base_addr) + ufshcd_mcq_compl_cqe(hba, cqe); + else + dev_err(hba->dev, "Abnormal CQ entry!\n"); } +/* + * This function is called from the UFS error handler with the UFS host + * controller disabled (HCE = 0). Reading host controller registers, e.g. the + * CQ tail pointer (CQTPy), may not be safe with the host controller disabled. + * Hence, iterate over all completion queue entries. This won't result in + * double completions because ufshcd_mcq_compl_cqe() clears a CQE after it + * has been processed. + */ void ufshcd_mcq_compl_all_cqes_lock(struct ufs_hba *hba, struct ufs_hw_queue *hwq) { @@ -292,13 +305,13 @@ void ufshcd_mcq_compl_all_cqes_lock(struct ufs_hba *hba, spin_lock_irqsave(&hwq->cq_lock, flags); while (entries > 0) { - ufshcd_mcq_process_cqe(hba, hwq); + struct cq_entry *cqe = ufshcd_mcq_cur_cqe(hwq); + + if (cqe->command_desc_base_addr) + ufshcd_mcq_compl_cqe(hba, cqe); ufshcd_mcq_inc_cq_head_slot(hwq); entries--; } - - ufshcd_mcq_update_cq_tail_slot(hwq); - hwq->cq_head_slot = hwq->cq_tail_slot; spin_unlock_irqrestore(&hwq->cq_lock, flags); } diff --git a/drivers/ufs/core/ufshcd-priv.h b/drivers/ufs/core/ufshcd-priv.h index e33466609052..29153629f001 100644 --- a/drivers/ufs/core/ufshcd-priv.h +++ b/drivers/ufs/core/ufshcd-priv.h @@ -83,11 +83,17 @@ int ufshcd_try_to_abort_task(struct ufs_hba *hba, int tag); void ufshcd_release_scsi_cmd(struct ufs_hba *hba, struct ufshcd_lrb *lrbp); -#define SD_ASCII_STD true -#define SD_RAW false -int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, - u8 **buf, bool ascii); +/** + * enum ufs_descr_fmt - UFS string descriptor format + * @SD_RAW: Raw UTF-16 format + * @SD_ASCII_STD: Convert to null-terminated ASCII string + */ +enum ufs_descr_fmt { + SD_RAW = 0, + SD_ASCII_STD = 1, +}; +int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, u8 **buf, enum ufs_descr_fmt fmt); int ufshcd_send_uic_cmd(struct ufs_hba *hba, struct uic_command *uic_cmd); int ufshcd_send_bsg_uic_cmd(struct ufs_hba *hba, struct uic_command *uic_cmd); diff --git a/drivers/ufs/core/ufshcd.c b/drivers/ufs/core/ufshcd.c index 368459dd8b3f..55de8d4a5eef 100644 --- a/drivers/ufs/core/ufshcd.c +++ b/drivers/ufs/core/ufshcd.c @@ -3672,20 +3672,18 @@ static inline char ufshcd_remove_non_printable(u8 ch) * @desc_index: descriptor index * @buf: pointer to buffer where descriptor would be read, * the caller should free the memory. - * @ascii: if true convert from unicode to ascii characters - * null terminated string. + * @fmt: if %SD_ASCII_STD, convert from UTF-16 to ASCII * * Return: * * string size on success. * * -ENOMEM: on allocation failure * * -EINVAL: on a wrong parameter */ -int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, - u8 **buf, bool ascii) +int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, u8 **buf, enum ufs_descr_fmt fmt) { struct uc_string_id *uc_str; u8 *str; - int ret; + int ret, uc_len; if (!buf) return -EINVAL; @@ -3710,11 +3708,19 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, goto out; } - if (ascii) { + uc_len = uc_str->len - QUERY_DESC_HDR_SIZE; + if (uc_len % sizeof(*uc_str->uc)) { + dev_err(hba->dev, "String Desc has an odd UTF-16 payload length\n"); + str = NULL; + ret = -EINVAL; + goto out; + } + + if (fmt == SD_ASCII_STD) { ssize_t ascii_len; int i; - /* remove header and divide by 2 to move from UTF16 to UTF8 */ - ascii_len = (uc_str->len - QUERY_DESC_HDR_SIZE) / 2 + 1; + /* Allow up to three UTF-8 bytes per UTF-16 code unit plus a NUL. */ + ascii_len = uc_len / sizeof(*uc_str->uc) * 3 + 1; str = kzalloc(ascii_len, GFP_KERNEL); if (!str) { ret = -ENOMEM; @@ -3726,7 +3732,7 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, * we need to convert to utf-8 so it can be displayed */ ret = utf16s_to_utf8s(uc_str->uc, - uc_str->len - QUERY_DESC_HDR_SIZE, + uc_len / sizeof(*uc_str->uc), UTF16_BIG_ENDIAN, str, ascii_len - 1); /* replace non-printable or non-ASCII characters with spaces */ @@ -3736,11 +3742,17 @@ int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, str[ret++] = '\0'; } else { - str = kmemdup(uc_str->uc, uc_str->len, GFP_KERNEL); + /* + * Keep the bLength-sized raw output for the RPMB device ID ABI. + * The two bytes beyond the UTF-16 payload are explicitly zeroed + * instead of being read past the descriptor buffer. + */ + str = kzalloc(uc_str->len, GFP_KERNEL); if (!str) { ret = -ENOMEM; goto out; } + memcpy(str, uc_str->uc, uc_len); ret = uc_str->len; } out: @@ -5639,7 +5651,17 @@ static void ufshcd_mcq_compl_pending_transfer(struct ufs_hba *hba, struct scsi_cmnd *cmd; unsigned long flags; u32 hwq_num, utag; - int tag; + int tag, i; + + for (i = 0; i < hba->nr_hw_queues; i++) { + if (force_compl) + ufshcd_mcq_compl_all_cqes_lock(hba, &hba->uhq[i]); + else + ufshcd_mcq_poll_cqe_lock(hba, &hba->uhq[i]); + } + + if (!force_compl) + return; for (tag = 0; tag < hba->nutrs; tag++) { lrbp = &hba->lrb[tag]; @@ -5652,21 +5674,16 @@ static void ufshcd_mcq_compl_pending_transfer(struct ufs_hba *hba, hwq_num = blk_mq_unique_tag_to_hwq(utag); hwq = &hba->uhq[hwq_num]; - if (force_compl) { - ufshcd_mcq_compl_all_cqes_lock(hba, hwq); - /* - * For those cmds of which the cqes are not present - * in the cq, complete them explicitly. - */ - if (cmd && !test_bit(SCMD_STATE_COMPLETE, &cmd->state)) { - spin_lock_irqsave(&hwq->cq_lock, flags); - set_host_byte(cmd, DID_REQUEUE); - ufshcd_release_scsi_cmd(hba, lrbp); - scsi_done(cmd); - spin_unlock_irqrestore(&hwq->cq_lock, flags); - } - } else { - ufshcd_mcq_poll_cqe_lock(hba, hwq); + /* + * For those cmds of which the cqes are not present + * in the cq, complete them explicitly. + */ + if (cmd && !test_bit(SCMD_STATE_COMPLETE, &cmd->state)) { + spin_lock_irqsave(&hwq->cq_lock, flags); + set_host_byte(cmd, DID_REQUEUE); + ufshcd_release_scsi_cmd(hba, lrbp); + scsi_done(cmd); + spin_unlock_irqrestore(&hwq->cq_lock, flags); } } } @@ -9609,7 +9626,7 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba, * we are functional while we are here, skip host resume in error * handling context. */ - hba->host->eh_noresume = 1; + WRITE_ONCE(hba->host->eh_noresume, 1); /* * Current function would be generally called from the power management @@ -9639,7 +9656,7 @@ static int ufshcd_set_dev_pwr_mode(struct ufs_hba *hba, } scsi_device_put(sdp); - hba->host->eh_noresume = 0; + WRITE_ONCE(hba->host->eh_noresume, 0); return ret; } diff --git a/drivers/ufs/host/ufs-exynos.c b/drivers/ufs/host/ufs-exynos.c index 14c1b855e10a..3e3ec906cb60 100644 --- a/drivers/ufs/host/ufs-exynos.c +++ b/drivers/ufs/host/ufs-exynos.c @@ -200,8 +200,8 @@ static int exynosauto_ufs_drv_init(struct device *dev, struct exynos_ufs *ufs) /* IO Coherency setting */ if (ufs->sysreg) { return regmap_update_bits(ufs->sysreg, - ufs->shareability_reg_offset, - UFS_SHARABLE, UFS_SHARABLE); + ufs->iocc_offset, + UFS_SHARABLE, ufs->iocc_val); } attr->tx_dif_p_nsec = 3200000; @@ -1102,12 +1102,21 @@ static int exynos_ufs_parse_dt(struct device *dev, struct exynos_ufs *ufs) ufs->sysreg = NULL; else { if (of_property_read_u32_index(np, "samsung,sysreg", 1, - &ufs->shareability_reg_offset)) { + &ufs->iocc_offset)) { dev_warn(dev, "can't get an offset from sysreg. Set to default value\n"); - ufs->shareability_reg_offset = UFS_SHAREABILITY_OFFSET; + ufs->iocc_offset = UFS_SHAREABILITY_OFFSET; } } + /* + * no 'dma-coherent' property means the descriptors are + * non-cacheable so iocc shareability should be disabled. + */ + if (of_dma_is_coherent(dev->of_node)) + ufs->iocc_val = UFS_SHARABLE; + else + ufs->iocc_val = 0; + ufs->pclk_avail_min = PCLK_AVAIL_MIN; ufs->pclk_avail_max = PCLK_AVAIL_MAX; diff --git a/drivers/ufs/host/ufs-exynos.h b/drivers/ufs/host/ufs-exynos.h index a4bd6646d7f1..65aa04d41c3e 100644 --- a/drivers/ufs/host/ufs-exynos.h +++ b/drivers/ufs/host/ufs-exynos.h @@ -212,7 +212,8 @@ struct exynos_ufs { ktime_t entry_hibern8_t; const struct exynos_ufs_drv_data *drv_data; struct regmap *sysreg; - u32 shareability_reg_offset; + u32 iocc_offset; + u32 iocc_val; u32 opts; #define EXYNOS_UFS_OPT_HAS_APB_CLK_CTRL BIT(0) diff --git a/drivers/ufs/host/ufs-qcom.c b/drivers/ufs/host/ufs-qcom.c index c94824c999cc..d639df475bca 100644 --- a/drivers/ufs/host/ufs-qcom.c +++ b/drivers/ufs/host/ufs-qcom.c @@ -731,25 +731,29 @@ static int ufs_qcom_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op, enum ufs_notify_change_status status) { struct ufs_qcom_host *host = ufshcd_get_variant(hba); - struct phy *phy = host->generic_phy; if (status == PRE_CHANGE) return 0; - if (ufs_qcom_is_link_off(hba)) { - /* - * Disable the tx/rx lane symbol clocks before PHY is - * powered down as the PLL source should be disabled - * after downstream clocks are disabled. - */ + if (!ufs_qcom_is_link_active(hba)) ufs_qcom_disable_lane_clks(host); - phy_power_off(phy); - /* reset the connected UFS device during power down */ - ufs_qcom_device_reset_ctrl(hba, true); - } else if (!ufs_qcom_is_link_active(hba)) { - ufs_qcom_disable_lane_clks(host); + /* reset the connected UFS device during power down */ + if (ufs_qcom_is_link_off(hba) && host->device_reset) { + ufs_qcom_device_reset_ctrl(hba, true); + /* + * After sending the SSU command, asserting the rst_n + * line causes the device firmware to wake up and + * execute its reset routine. + * + * During this process, the device may draw current + * beyond the permissible limit for low-power mode (LPM). + * A 10ms delay, based on experimental observations, + * allows the UFS device to complete its hardware reset + * before transitioning the power rail to LPM. + */ + usleep_range(10000, 11000); } return ufs_qcom_ice_suspend(host); @@ -758,26 +762,11 @@ static int ufs_qcom_suspend(struct ufs_hba *hba, enum ufs_pm_op pm_op, static int ufs_qcom_resume(struct ufs_hba *hba, enum ufs_pm_op pm_op) { struct ufs_qcom_host *host = ufshcd_get_variant(hba); - struct phy *phy = host->generic_phy; int err; - if (ufs_qcom_is_link_off(hba)) { - err = phy_power_on(phy); - if (err) { - dev_err(hba->dev, "%s: failed PHY power on: %d\n", - __func__, err); - return err; - } - - err = ufs_qcom_enable_lane_clks(host); - if (err) - return err; - - } else if (!ufs_qcom_is_link_active(hba)) { - err = ufs_qcom_enable_lane_clks(host); - if (err) - return err; - } + err = ufs_qcom_enable_lane_clks(host); + if (err) + return err; return ufs_qcom_ice_resume(host); } @@ -1065,12 +1054,20 @@ static void ufs_qcom_set_caps(struct ufs_hba *hba) * @on: If true, enable clocks else disable them. * @status: PRE_CHANGE or POST_CHANGE notify * + * There are certain clocks which comes from the PHY so it needs + * to be managed together along with controller clocks which also + * provides a better power saving. Hence keep phy_power_off/on calls + * in ufs_qcom_setup_clocks, so that PHY's regulators & clks can be + * turned on/off along with UFS's clocks. + * * Return: 0 on success, non-zero on failure. */ static int ufs_qcom_setup_clocks(struct ufs_hba *hba, bool on, enum ufs_notify_change_status status) { struct ufs_qcom_host *host = ufshcd_get_variant(hba); + struct phy *phy = host->generic_phy; + int err; /* * In case ufs_qcom_init() is not yet done, simply ignore. @@ -1089,10 +1086,22 @@ static int ufs_qcom_setup_clocks(struct ufs_hba *hba, bool on, /* disable device ref_clk */ ufs_qcom_dev_ref_clk_ctrl(host, false); } + + err = phy_power_off(phy); + if (err) { + dev_err(hba->dev, "phy power off failed, ret=%d\n", err); + return err; + } } break; case POST_CHANGE: if (on) { + err = phy_power_on(phy); + if (err) { + dev_err(hba->dev, "phy power on failed, ret = %d\n", err); + return err; + } + /* enable the device ref clock for HS mode*/ if (ufshcd_is_hs_mode(&hba->pwr_info)) ufs_qcom_dev_ref_clk_ctrl(host, true); diff --git a/drivers/usb/cdns3/cdns3-gadget.c b/drivers/usb/cdns3/cdns3-gadget.c index b52570c6deff..ba5713dc551b 100644 --- a/drivers/usb/cdns3/cdns3-gadget.c +++ b/drivers/usb/cdns3/cdns3-gadget.c @@ -3269,6 +3269,10 @@ static void cdns3_gadget_exit(struct cdns *cdns) usb_del_gadget(&priv_dev->gadget); devm_free_irq(cdns->dev, cdns->dev_irq, priv_dev); + /* The works can still be queued until the IRQ is freed. */ + cancel_work_sync(&priv_dev->pending_status_wq); + cancel_work_sync(&priv_dev->aligned_buf_wq); + cdns3_free_all_eps(priv_dev); while (!list_empty(&priv_dev->aligned_buf_list)) { diff --git a/drivers/usb/cdns3/cdns3-pci-wrap.c b/drivers/usb/cdns3/cdns3-pci-wrap.c index 92ae5d443350..2f321ce033be 100644 --- a/drivers/usb/cdns3/cdns3-pci-wrap.c +++ b/drivers/usb/cdns3/cdns3-pci-wrap.c @@ -99,6 +99,11 @@ static int cdns3_pci_probe(struct pci_dev *pdev, if (pci_is_enabled(func)) { wrap = pci_get_drvdata(func); + if (!wrap) { + dev_err(&pdev->dev, + "second function not initialized, retrying\n"); + return -EPROBE_DEFER; + } } else { wrap = kzalloc(sizeof(*wrap), GFP_KERNEL); if (!wrap) diff --git a/drivers/usb/chipidea/ci_hdrc_tegra.c b/drivers/usb/chipidea/ci_hdrc_tegra.c index 8e78bf643e25..0e77dada1582 100644 --- a/drivers/usb/chipidea/ci_hdrc_tegra.c +++ b/drivers/usb/chipidea/ci_hdrc_tegra.c @@ -308,8 +308,10 @@ static int tegra_usb_probe(struct platform_device *pdev) pm_runtime_enable(&pdev->dev); err = pm_runtime_resume_and_get(&pdev->dev); - if (err) + if (err) { + pm_runtime_disable(&pdev->dev); return err; + } if (device_property_present(&pdev->dev, "nvidia,needs-double-reset")) usb->needs_double_reset = true; diff --git a/drivers/usb/chipidea/udc.c b/drivers/usb/chipidea/udc.c index 0bee561420af..301eba2758aa 100644 --- a/drivers/usb/chipidea/udc.c +++ b/drivers/usb/chipidea/udc.c @@ -773,6 +773,10 @@ __acquires(hwep->lock) list_del_init(&hwreq->queue); hwreq->req.status = -ESHUTDOWN; + /* Unmap DMA and clean up bounce buffers before giving back */ + usb_gadget_unmap_request_by_dev(hwep->ci->dev->parent, + &hwreq->req, hwep->dir); + if (hwreq->req.complete != NULL) { spin_unlock(hwep->lock); usb_gadget_giveback_request(&hwep->ep, &hwreq->req); diff --git a/drivers/usb/class/cdc-acm.c b/drivers/usb/class/cdc-acm.c index 659884a46437..7c6ea7d1d6d7 100644 --- a/drivers/usb/class/cdc-acm.c +++ b/drivers/usb/class/cdc-acm.c @@ -331,7 +331,6 @@ static void acm_process_notification(struct acm *acm, unsigned char *buf) spin_lock_irqsave(&acm->read_lock, flags); acm->ctrlin = newctrl; - acm->oldcount = acm->iocount; if (difference & USB_CDC_SERIAL_STATE_DSR) acm->iocount.dsr++; @@ -788,6 +787,9 @@ static void acm_port_shutdown(struct tty_port *port) usb_autopm_put_interface_async(acm->control); } + if (acm->disconnected) + return; + acm_unpoison_urbs(acm); if (acm->quirks & ALWAYS_POLL_CTRL) { @@ -1027,11 +1029,16 @@ static int wait_serial_change(struct acm *acm, unsigned long arg) DECLARE_WAITQUEUE(wait, current); struct async_icount old, new; - do { + spin_lock_irq(&acm->read_lock); + old = acm->iocount; + spin_unlock_irq(&acm->read_lock); + + add_wait_queue(&acm->wioctl, &wait); + for (;;) { + set_current_state(TASK_INTERRUPTIBLE); + spin_lock_irq(&acm->read_lock); - old = acm->oldcount; new = acm->iocount; - acm->oldcount = new; spin_unlock_irq(&acm->read_lock); if ((arg & TIOCM_DSR) && @@ -1044,22 +1051,20 @@ static int wait_serial_change(struct acm *acm, unsigned long arg) old.rng != new.rng) break; - add_wait_queue(&acm->wioctl, &wait); - set_current_state(TASK_INTERRUPTIBLE); - schedule(); - remove_wait_queue(&acm->wioctl, &wait); if (acm->disconnected) { - if (arg & TIOCM_CD) - break; - else - rv = -ENODEV; - } else { - if (signal_pending(current)) - rv = -ERESTARTSYS; + rv = -ENODEV; + break; } - } while (!rv); - + schedule(); + + if (signal_pending(current)) { + rv = -ERESTARTSYS; + break; + } + } + __set_current_state(TASK_RUNNING); + remove_wait_queue(&acm->wioctl, &wait); return rv; } @@ -2140,7 +2145,7 @@ static int __init acm_init(void) { int retval; acm_tty_driver = tty_alloc_driver(ACM_TTY_MINORS, TTY_DRIVER_REAL_RAW | - TTY_DRIVER_DYNAMIC_DEV); + TTY_DRIVER_DYNAMIC_DEV | TTY_DRIVER_RESET_SAVED_TERMIOS); if (IS_ERR(acm_tty_driver)) return PTR_ERR(acm_tty_driver); acm_tty_driver->driver_name = "acm", diff --git a/drivers/usb/class/cdc-acm.h b/drivers/usb/class/cdc-acm.h index 01f448a783c0..d245e60eb9f5 100644 --- a/drivers/usb/class/cdc-acm.h +++ b/drivers/usb/class/cdc-acm.h @@ -90,7 +90,6 @@ struct acm { unsigned int ctrlin; /* input control lines (DCD, DSR, RI, break, overruns) */ unsigned int ctrlout; /* output control lines (DTR, RTS) */ struct async_icount iocount; /* counters for control line changes */ - struct async_icount oldcount; /* for comparison of counter */ wait_queue_head_t wioctl; /* for ioctl */ unsigned int writesize; /* max packet size for the output bulk endpoint */ unsigned int readsize,ctrlsize; /* buffer sizes for freeing */ diff --git a/drivers/usb/core/hub.c b/drivers/usb/core/hub.c index c0d8259975e3..0a7fa1e50be9 100644 --- a/drivers/usb/core/hub.c +++ b/drivers/usb/core/hub.c @@ -69,8 +69,11 @@ /* * Give SS hubs 200ms time after wake to train downstream links before * assuming no port activity and allowing hub to runtime suspend back. + * Root hubs have no upstream hub whose wake propagation needs to be + * accounted for, so they need less time, use 120ms for them. */ #define USB_SS_PORT_U0_WAKE_TIME 200 /* ms */ +#define USB_SS_RH_PORT_U0_WAKE_TIME 120 /* ms */ /* Protect struct usb_device->state and ->children members * Note: Both are also protected by ->dev.sem, except that ->state can @@ -1332,7 +1335,9 @@ static void hub_activate(struct usb_hub *hub, enum hub_activation_type type) queue_delayed_work(system_power_efficient_wq, &hub->post_resume_work, - msecs_to_jiffies(USB_SS_PORT_U0_WAKE_TIME)); + msecs_to_jiffies(hdev->parent ? + USB_SS_PORT_U0_WAKE_TIME : + USB_SS_RH_PORT_U0_WAKE_TIME)); return; } diff --git a/drivers/usb/core/message.c b/drivers/usb/core/message.c index 119d07c4c62b..6e5f25a63692 100644 --- a/drivers/usb/core/message.c +++ b/drivers/usb/core/message.c @@ -1333,7 +1333,8 @@ static void remove_intf_ep_devs(struct usb_interface *intf) * * Disables the endpoint for URB submission and nukes all pending URBs. * If @reset_hardware is set then also deallocates hcd/hardware state - * for the endpoint. + * for the endpoint (clearing both ep_in and ep_out pointers for + * bidirectional non-ep0 control endpoints). */ void usb_disable_endpoint(struct usb_device *dev, unsigned int epaddr, bool reset_hardware) @@ -1354,6 +1355,11 @@ void usb_disable_endpoint(struct usb_device *dev, unsigned int epaddr, dev->ep_in[epnum] = NULL; } if (ep) { + if (reset_hardware && epnum != 0 && + usb_endpoint_xfer_control(&ep->desc)) { + dev->ep_out[epnum] = NULL; + dev->ep_in[epnum] = NULL; + } ep->enabled = 0; usb_hcd_flush_endpoint(dev, ep); if (reset_hardware) diff --git a/drivers/usb/dwc2/hcd.c b/drivers/usb/dwc2/hcd.c index e0f84726f4b1..5e71094a22b2 100644 --- a/drivers/usb/dwc2/hcd.c +++ b/drivers/usb/dwc2/hcd.c @@ -5082,14 +5082,13 @@ static void dwc2_hcd_free(struct dwc2_hsotg *hsotg) } cancel_work_sync(&hsotg->phy_reset_work); - - del_timer(&hsotg->wkp_timer); } static void dwc2_hcd_release(struct dwc2_hsotg *hsotg) { /* Turn off all host-specific interrupts */ dwc2_disable_host_interrupts(hsotg); + timer_shutdown_sync(&hsotg->wkp_timer); dwc2_hcd_free(hsotg); } diff --git a/drivers/usb/dwc3/core.h b/drivers/usb/dwc3/core.h index 45084ca66806..52b75592294c 100644 --- a/drivers/usb/dwc3/core.h +++ b/drivers/usb/dwc3/core.h @@ -42,6 +42,7 @@ #define DWC3_ENDPOINTS_NUM 32 #define DWC3_XHCI_RESOURCES_NUM 2 #define DWC3_ISOC_MAX_RETRIES 5 +#define DWC3_ERR_RECOVERY_MAX 3 #define DWC3_SCRATCHBUF_SIZE 4096 /* each buffer is assumed to be 4KiB */ #define DWC3_EVENT_BUFFERS_SIZE 4096 @@ -726,6 +727,7 @@ struct dwc3_event_buffer { */ struct dwc3_ep { struct usb_ep endpoint; + struct delayed_work nostream_work; struct list_head cancelled_list; struct list_head pending_list; struct list_head started_list; @@ -748,7 +750,7 @@ struct dwc3_ep { #define DWC3_EP_WAIT_TRANSFER_COMPLETE BIT(7) #define DWC3_EP_IGNORE_NEXT_NOSTREAM BIT(8) #define DWC3_EP_FORCE_RESTART_STREAM BIT(9) -#define DWC3_EP_FIRST_STREAM_PRIMED BIT(10) +#define DWC3_EP_STREAM_PRIMED BIT(10) #define DWC3_EP_PENDING_CLEAR_STALL BIT(11) #define DWC3_EP_TXFIFO_RESIZED BIT(12) #define DWC3_EP_DELAY_STOP BIT(13) @@ -824,6 +826,12 @@ enum dwc3_link_state { DWC3_LINK_STATE_MASK = 0x0f, }; +enum dwc3_err_state { + DWC3_ERR_NONE = 0, + DWC3_ERR_RECOVERY, + DWC3_ERR_UNRECOVERABLE, +}; + /* TRB Length, PCM and Status */ #define DWC3_TRB_SIZE_MASK (0x00ffffff) #define DWC3_TRB_SIZE_LENGTH(n) ((n) & DWC3_TRB_SIZE_MASK) @@ -983,6 +991,7 @@ struct dwc3_scratchpad_array { /** * struct dwc3 - representation of our controller * @drd_work: workqueue used for role swapping + * @err_recovery_work: workqueue used for controller error recovery * @ep0_trb: trb which is used for the ctrl_req * @bounce: address of bounce buffer * @setup_buf: used while precessing STD USB requests @@ -992,6 +1001,7 @@ struct dwc3_scratchpad_array { * @ep0_in_setup: one control transfer is completed and enter setup phase * @lock: for synchronizing * @mutex: for mode switching + * @connect_mutex: for the pull-up and err_recovery_work * @dev: pointer to our struct device * @sysdev: pointer to the DMA-capable device * @xhci: pointer to our xHCI child @@ -1051,6 +1061,9 @@ struct dwc3_scratchpad_array { * @ep0_next_event: hold the next expected event * @ep0state: state of endpoint zero * @link_state: link state + * @err_state: current error recovery state. + * @err_recovery_count: number of consecutive error recovery attempts until + * confirmed healthy and reset to 0 on reset event. * @speed: device speed (super, high, full, low) * @hwparams: copy of hwparams registers * @regset: debugfs pointer to regdump file @@ -1148,6 +1161,7 @@ struct dwc3_scratchpad_array { */ struct dwc3 { struct work_struct drd_work; + struct work_struct err_recovery_work; struct dwc3_trb *ep0_trb; void *bounce; u8 *setup_buf; @@ -1162,6 +1176,9 @@ struct dwc3 { /* mode switching lock */ struct mutex mutex; + /* serializes pull-up run/stop vs error recovery */ + struct mutex connect_mutex; + struct device *dev; struct device *sysdev; @@ -1280,6 +1297,9 @@ struct dwc3 { enum dwc3_ep0_next ep0_next_event; enum dwc3_ep0_state ep0state; enum dwc3_link_state link_state; + enum dwc3_err_state err_state; + + u32 err_recovery_count; u16 u2sel; u16 u2pel; diff --git a/drivers/usb/dwc3/dwc3-am62.c b/drivers/usb/dwc3/dwc3-am62.c index 47d9f56333fe..793e5fb958ba 100644 --- a/drivers/usb/dwc3/dwc3-am62.c +++ b/drivers/usb/dwc3/dwc3-am62.c @@ -263,6 +263,7 @@ static int dwc3_ti_probe(struct platform_device *pdev) err_pm_disable: clk_disable_unprepare(am62->usb2_refclk); + pm_runtime_put_noidle(dev); pm_runtime_disable(dev); pm_runtime_set_suspended(dev); return ret; diff --git a/drivers/usb/dwc3/ep0.c b/drivers/usb/dwc3/ep0.c index bcc9cc398068..58d80db07295 100644 --- a/drivers/usb/dwc3/ep0.c +++ b/drivers/usb/dwc3/ep0.c @@ -200,6 +200,11 @@ int dwc3_gadget_ep0_queue(struct usb_ep *ep, struct usb_request *request, int ret; spin_lock_irqsave(&dwc->lock, flags); + if (dwc->err_state != DWC3_ERR_NONE) { + ret = -ESHUTDOWN; + goto out; + } + if (!dep->endpoint.desc || !dwc->pullups_connected || !dwc->connected) { dev_err(dwc->dev, "%s: can't queue to disabled endpoint\n", dep->name); @@ -271,6 +276,10 @@ int dwc3_gadget_ep0_set_halt(struct usb_ep *ep, int value) int ret; spin_lock_irqsave(&dwc->lock, flags); + if (dwc->err_state != DWC3_ERR_NONE) { + spin_unlock_irqrestore(&dwc->lock, flags); + return -ESHUTDOWN; + } ret = __dwc3_gadget_ep0_set_halt(ep, value); spin_unlock_irqrestore(&dwc->lock, flags); @@ -304,7 +313,7 @@ void dwc3_ep0_out_start(struct dwc3 *dwc) dwc3_ep->flags &= ~DWC3_EP_DELAY_STOP; if (dwc->connected) - dwc3_stop_active_transfer(dwc3_ep, true, true); + dwc3_stop_active_transfer(dwc3_ep, false, true); else dwc3_remove_requests(dwc, dwc3_ep, -ESHUTDOWN); } diff --git a/drivers/usb/dwc3/gadget.c b/drivers/usb/dwc3/gadget.c index 1d40e5136ef0..5bd30788301b 100644 --- a/drivers/usb/dwc3/gadget.c +++ b/drivers/usb/dwc3/gadget.c @@ -993,12 +993,11 @@ static int __dwc3_gadget_ep_enable(struct dwc3_ep *dep, unsigned int action) * controller to generate an ERDY to initiate the * stream. */ - dwc3_stop_active_transfer(dep, true, true); + dwc3_stop_active_transfer(dep, false, true); /* * All stream eps will reinitiate stream on NoStream - * rejection until we can determine that the host can - * prime after the first transfer. + * rejection. * * However, if the controller is capable of * TXF_FLUSH_BYPASS, then IN direction endpoints will @@ -1022,7 +1021,7 @@ void dwc3_remove_requests(struct dwc3 *dwc, struct dwc3_ep *dep, int status) { struct dwc3_request *req; - dwc3_stop_active_transfer(dep, true, false); + dwc3_stop_active_transfer(dep, false, false); /* If endxfer is delayed, avoid unmapping requests */ if (dep->flags & DWC3_EP_DELAY_STOP) @@ -1139,6 +1138,10 @@ static int dwc3_gadget_ep_enable(struct usb_ep *ep, return 0; spin_lock_irqsave(&dwc->lock, flags); + if (dwc->err_state != DWC3_ERR_NONE) { + spin_unlock_irqrestore(&dwc->lock, flags); + return -ESHUTDOWN; + } ret = __dwc3_gadget_ep_enable(dep, DWC3_DEPCFG_ACTION_INIT); spin_unlock_irqrestore(&dwc->lock, flags); @@ -1712,7 +1715,7 @@ static int __dwc3_gadget_kick_transfer(struct dwc3_ep *dep) if (ret == -EAGAIN) return ret; - dwc3_stop_active_transfer(dep, true, true); + dwc3_stop_active_transfer(dep, false, true); list_for_each_entry_safe(req, tmp, &dep->started_list, list) dwc3_gadget_move_cancelled_request(req, DWC3_REQUEST_STATUS_DEQUEUED); @@ -1749,6 +1752,11 @@ static int __dwc3_gadget_get_frame(struct dwc3 *dwc) * the controller won't update the TRB progress on command * completion. It also won't clear the HWO bit in the TRB. * The command will also not complete immediately in that case. + * + * Older programming guide revisions recommended setting ForceRM to 1 + * when ending a transfer. Newer programming guide revisions now + * recommend keeping ForceRM cleared, and TRBs are properly updated + * on command completion. */ static int __dwc3_stop_active_transfer(struct dwc3_ep *dep, bool force, bool interrupt) { @@ -1874,7 +1882,7 @@ static int dwc3_gadget_start_isoc_quirk(struct dwc3_ep *dep) * to wait for the next XferNotReady to test the command again */ if (cmd_status == 0) { - dwc3_stop_active_transfer(dep, true, true); + dwc3_stop_active_transfer(dep, false, true); return 0; } } @@ -2047,6 +2055,10 @@ static int dwc3_gadget_ep_queue(struct usb_ep *ep, struct usb_request *request, int ret; spin_lock_irqsave(&dwc->lock, flags); + if (dwc->err_state != DWC3_ERR_NONE) { + spin_unlock_irqrestore(&dwc->lock, flags); + return -ESHUTDOWN; + } ret = __dwc3_gadget_ep_queue(dep, req); spin_unlock_irqrestore(&dwc->lock, flags); @@ -2157,7 +2169,7 @@ static int dwc3_gadget_ep_dequeue(struct usb_ep *ep, struct dwc3_request *t; /* wait until it is processed */ - dwc3_stop_active_transfer(dep, true, true); + dwc3_stop_active_transfer(dep, false, true); /* * Remove any started request if the transfer is @@ -2234,7 +2246,7 @@ int __dwc3_gadget_ep_set_halt(struct dwc3_ep *dep, int value, int protocol) return 0; } - dwc3_stop_active_transfer(dep, true, true); + dwc3_stop_active_transfer(dep, false, true); list_for_each_entry_safe(req, tmp, &dep->started_list, list) dwc3_gadget_move_cancelled_request(req, DWC3_REQUEST_STATUS_STALLED); @@ -2279,6 +2291,10 @@ static int dwc3_gadget_ep_set_halt(struct usb_ep *ep, int value) int ret; spin_lock_irqsave(&dwc->lock, flags); + if (dwc->err_state != DWC3_ERR_NONE) { + spin_unlock_irqrestore(&dwc->lock, flags); + return -ESHUTDOWN; + } ret = __dwc3_gadget_ep_set_halt(dep, value, false); spin_unlock_irqrestore(&dwc->lock, flags); @@ -2293,6 +2309,10 @@ static int dwc3_gadget_ep_set_wedge(struct usb_ep *ep) int ret; spin_lock_irqsave(&dwc->lock, flags); + if (dwc->err_state != DWC3_ERR_NONE) { + spin_unlock_irqrestore(&dwc->lock, flags); + return -ESHUTDOWN; + } dep->flags |= DWC3_EP_WEDGE; if (dep->number == 0 || dep->number == 1) @@ -2446,6 +2466,10 @@ static int dwc3_gadget_wakeup(struct usb_gadget *g) } spin_lock_irqsave(&dwc->lock, flags); + if (dwc->err_state != DWC3_ERR_NONE) { + spin_unlock_irqrestore(&dwc->lock, flags); + return -ESHUTDOWN; + } if (!dwc->gadget->wakeup_armed) { dev_err(dwc->dev, "not armed for remote wakeup\n"); spin_unlock_irqrestore(&dwc->lock, flags); @@ -2473,6 +2497,10 @@ static int dwc3_gadget_func_wakeup(struct usb_gadget *g, int intf_id) } spin_lock_irqsave(&dwc->lock, flags); + if (dwc->err_state != DWC3_ERR_NONE) { + spin_unlock_irqrestore(&dwc->lock, flags); + return -ESHUTDOWN; + } /* * If the link is in U3, signal for remote wakeup and wait for the * link to transition to U0 before sending device notification. @@ -2843,10 +2871,13 @@ static int dwc3_gadget_pullup(struct usb_gadget *g, int is_on) synchronize_irq(dwc->irq_gadget); + /* Serialize against dwc3_err_recovery_work() */ + mutex_lock(&dwc->connect_mutex); if (!is_on) ret = dwc3_gadget_soft_disconnect(dwc); else ret = dwc3_gadget_soft_connect(dwc); + mutex_unlock(&dwc->connect_mutex); pm_runtime_put(dwc->dev); @@ -3337,6 +3368,50 @@ static int dwc3_gadget_init_out_endpoint(struct dwc3_ep *dep) return dwc3_alloc_trb_pool(dep); } +#define nostream_work_to_dep(w) (container_of(to_delayed_work(w), struct dwc3_ep, nostream_work)) +static void dwc3_nostream_work(struct work_struct *work) +{ + struct dwc3_ep *dep = nostream_work_to_dep(work); + struct dwc3 *dwc = dep->dwc; + unsigned long flags; + + spin_lock_irqsave(&dwc->lock, flags); + if (dep->flags & DWC3_EP_STREAM_PRIMED) + goto out; + + if ((dep->flags & DWC3_EP_IGNORE_NEXT_NOSTREAM) || + (!DWC3_MST_CAPABLE(&dwc->hwparams) && + !(dep->flags & DWC3_EP_WAIT_TRANSFER_COMPLETE))) + goto out; + /* + * If the host rejects a stream due to no active stream, by the + * USB and xHCI spec, the endpoint will be put back to idle + * state. When the host is ready (buffer added/updated), it will + * prime the endpoint to inform the usb device controller. This + * triggers the device controller to issue ERDY to restart the + * stream. However, some hosts don't follow this and keep the + * endpoint in the idle state. No prime will come despite host + * streams are updated, and the device controller will not be + * triggered to generate ERDY to move the next stream data. To + * workaround this and maintain compatibility with various + * hosts, force to reinitiate the stream until the host is ready + * instead of waiting for the host to prime the endpoint. + */ + if (DWC3_VER_IS_WITHIN(DWC32, 100A, ANY)) { + unsigned int cmd = DWC3_DGCMD_SET_ENDPOINT_PRIME; + + dwc3_send_gadget_generic_command(dwc, cmd, dep->number); + } else { + dep->flags |= DWC3_EP_DELAY_START; + dwc3_stop_active_transfer(dep, false, true); + spin_unlock_irqrestore(&dwc->lock, flags); + return; + } +out: + dep->flags &= ~DWC3_EP_IGNORE_NEXT_NOSTREAM; + spin_unlock_irqrestore(&dwc->lock, flags); +} + static int dwc3_gadget_init_endpoint(struct dwc3 *dwc, u8 epnum) { struct dwc3_ep *dep; @@ -3382,6 +3457,7 @@ static int dwc3_gadget_init_endpoint(struct dwc3 *dwc, u8 epnum) INIT_LIST_HEAD(&dep->pending_list); INIT_LIST_HEAD(&dep->started_list); INIT_LIST_HEAD(&dep->cancelled_list); + INIT_DELAYED_WORK(&dep->nostream_work, dwc3_nostream_work); dwc3_debugfs_create_endpoint_dir(dep); @@ -3671,7 +3747,7 @@ static bool dwc3_gadget_endpoint_trbs_complete(struct dwc3_ep *dep, if (usb_endpoint_xfer_isoc(dep->endpoint.desc) && list_empty(&dep->started_list) && (list_empty(&dep->pending_list) || status == -EXDEV)) - dwc3_stop_active_transfer(dep, true, true); + dwc3_stop_active_transfer(dep, false, true); else if (dwc3_gadget_ep_should_continue(dep)) if (__dwc3_gadget_kick_transfer(dep) == 0) no_started_trb = false; @@ -3817,66 +3893,27 @@ static void dwc3_gadget_endpoint_command_complete(struct dwc3_ep *dep, static void dwc3_gadget_endpoint_stream_event(struct dwc3_ep *dep, const struct dwc3_event_depevt *event) { - struct dwc3 *dwc = dep->dwc; - if (event->status == DEPEVT_STREAMEVT_FOUND) { - dep->flags |= DWC3_EP_FIRST_STREAM_PRIMED; - goto out; + cancel_delayed_work(&dep->nostream_work); + dep->flags |= DWC3_EP_STREAM_PRIMED; + dep->flags &= ~DWC3_EP_IGNORE_NEXT_NOSTREAM; + return; } /* Note: NoStream rejection event param value is 0 and not 0xFFFF */ switch (event->parameters) { case DEPEVT_STREAM_PRIME: - /* - * If the host can properly transition the endpoint state from - * idle to prime after a NoStream rejection, there's no need to - * force restarting the endpoint to reinitiate the stream. To - * simplify the check, assume the host follows the USB spec if - * it primed the endpoint more than once. - */ - if (dep->flags & DWC3_EP_FORCE_RESTART_STREAM) { - if (dep->flags & DWC3_EP_FIRST_STREAM_PRIMED) - dep->flags &= ~DWC3_EP_FORCE_RESTART_STREAM; - else - dep->flags |= DWC3_EP_FIRST_STREAM_PRIMED; - } - + cancel_delayed_work(&dep->nostream_work); + dep->flags |= DWC3_EP_STREAM_PRIMED; + dep->flags &= ~DWC3_EP_IGNORE_NEXT_NOSTREAM; break; case DEPEVT_STREAM_NOSTREAM: - if ((dep->flags & DWC3_EP_IGNORE_NEXT_NOSTREAM) || - !(dep->flags & DWC3_EP_FORCE_RESTART_STREAM) || - (!DWC3_MST_CAPABLE(&dwc->hwparams) && - !(dep->flags & DWC3_EP_WAIT_TRANSFER_COMPLETE))) - break; - - /* - * If the host rejects a stream due to no active stream, by the - * USB and xHCI spec, the endpoint will be put back to idle - * state. When the host is ready (buffer added/updated), it will - * prime the endpoint to inform the usb device controller. This - * triggers the device controller to issue ERDY to restart the - * stream. However, some hosts don't follow this and keep the - * endpoint in the idle state. No prime will come despite host - * streams are updated, and the device controller will not be - * triggered to generate ERDY to move the next stream data. To - * workaround this and maintain compatibility with various - * hosts, force to reinitiate the stream until the host is ready - * instead of waiting for the host to prime the endpoint. - */ - if (DWC3_VER_IS_WITHIN(DWC32, 100A, ANY)) { - unsigned int cmd = DWC3_DGCMD_SET_ENDPOINT_PRIME; - - dwc3_send_gadget_generic_command(dwc, cmd, dep->number); - } else { - dep->flags |= DWC3_EP_DELAY_START; - dwc3_stop_active_transfer(dep, true, true); - return; - } + dep->flags &= ~DWC3_EP_STREAM_PRIMED; + if (dep->flags & DWC3_EP_FORCE_RESTART_STREAM) + queue_delayed_work(system_wq, &dep->nostream_work, + msecs_to_jiffies(100)); break; } - -out: - dep->flags &= ~DWC3_EP_IGNORE_NEXT_NOSTREAM; } static void dwc3_endpoint_interrupt(struct dwc3 *dwc, @@ -4132,6 +4169,10 @@ static void dwc3_gadget_reset_interrupt(struct dwc3 *dwc) dwc->suspended = false; + /* The controller is recovered. */ + if (dwc->err_state == DWC3_ERR_NONE) + dwc->err_recovery_count = 0; + /* * Ideally, dwc3_reset_gadget() would trigger the function * drivers to stop any active transfers through ep disable. @@ -4605,6 +4646,12 @@ static irqreturn_t dwc3_thread_interrupt(int irq, void *_evt) return ret; } +static void dwc3_schedule_err_recovery(struct dwc3 *dwc) +{ + dwc->err_state = DWC3_ERR_RECOVERY; + schedule_work(&dwc->err_recovery_work); +} + static irqreturn_t dwc3_check_event_buf(struct dwc3_event_buffer *evt) { struct dwc3 *dwc = evt->dwc; @@ -4638,9 +4685,21 @@ static irqreturn_t dwc3_check_event_buf(struct dwc3_event_buffer *evt) return IRQ_NONE; if (count > evt->length) { - dev_err_ratelimited(dwc->dev, "invalid count(%u) > evt->length(%u)\n", + dev_err(dwc->dev, "invalid count(%u) > evt->length(%u)\n", count, evt->length); - return IRQ_NONE; + /* + * This is a fatal error - the driver and controller are out of + * sync on which event has been consumed. Reinitializing the + * controller is required to recover. Write the bogus count back + * to GEVNTCOUNT to clear the IRQ source, consistent with the + * stale event clearing in dwc3_event_buffers_setup(), then + * schedule error recovery. + */ + spin_lock(&dwc->lock); + dwc3_schedule_err_recovery(dwc); + spin_unlock(&dwc->lock); + dwc3_writel(dwc->regs, DWC3_GEVNTCOUNT(0), count); + return IRQ_HANDLED; } evt->count = count; @@ -4700,6 +4759,57 @@ static void dwc_gadget_release(struct device *dev) kfree(gadget); } +static void dwc3_err_recovery_work(struct work_struct *work) +{ + struct dwc3 *dwc = container_of(work, struct dwc3, err_recovery_work); + unsigned long flags; + int ret; + + /* serializes against dwc3_gadget_pullup() */ + mutex_lock(&dwc->connect_mutex); + + ret = dwc3_gadget_soft_disconnect(dwc); + + dwc3_disconnect_gadget_sleepable(dwc); + + if (ret) + goto err_unrecoverable; + + if (dwc->softconnect) { + u32 count; + /* + * Wait irq to finish before soft_connect resets evt->lpos and + * the event buffer registers to avoid racing with + * dwc3_process_event_buf(). + */ + synchronize_irq(dwc->irq_gadget); + + spin_lock_irqsave(&dwc->lock, flags); + count = ++dwc->err_recovery_count; + spin_unlock_irqrestore(&dwc->lock, flags); + + if (count > DWC3_ERR_RECOVERY_MAX) + goto err_unrecoverable; + + ret = dwc3_gadget_soft_connect(dwc); + if (ret) + goto err_unrecoverable; + } + mutex_unlock(&dwc->connect_mutex); + + spin_lock_irqsave(&dwc->lock, flags); + dwc->err_state = DWC3_ERR_NONE; + spin_unlock_irqrestore(&dwc->lock, flags); + return; + +err_unrecoverable: + dev_err(dwc->dev, "Unable to recover the controller\n"); + mutex_unlock(&dwc->connect_mutex); + spin_lock_irqsave(&dwc->lock, flags); + dwc->err_state = DWC3_ERR_UNRECOVERABLE; + spin_unlock_irqrestore(&dwc->lock, flags); +} + /** * dwc3_gadget_init - initializes gadget related registers * @dwc: pointer to our controller context structure @@ -4743,6 +4853,8 @@ int dwc3_gadget_init(struct dwc3 *dwc) } init_completion(&dwc->ep0_in_setup); + INIT_WORK(&dwc->err_recovery_work, dwc3_err_recovery_work); + mutex_init(&dwc->connect_mutex); dwc->gadget = kzalloc(sizeof(struct usb_gadget), GFP_KERNEL); if (!dwc->gadget) { ret = -ENOMEM; @@ -4838,6 +4950,8 @@ void dwc3_gadget_exit(struct dwc3 *dwc) if (!dwc->gadget) return; + cancel_work_sync(&dwc->err_recovery_work); + mutex_destroy(&dwc->connect_mutex); dwc3_enable_susphy(dwc, true); usb_del_gadget(dwc->gadget); dwc3_gadget_free_endpoints(dwc); diff --git a/drivers/usb/gadget/function/f_ecm.c b/drivers/usb/gadget/function/f_ecm.c index 7bb63b9e3f78..752971cbc638 100644 --- a/drivers/usb/gadget/function/f_ecm.c +++ b/drivers/usb/gadget/function/f_ecm.c @@ -681,6 +681,7 @@ ecm_bind(struct usb_configuration *c, struct usb_function *f) struct usb_ep *ep; struct f_ecm_opts *ecm_opts; + struct net_device *net __free(detach_gadget) = NULL; struct usb_request *request __free(free_usb_request) = NULL; if (!can_support_ecm(cdev->gadget)) @@ -688,18 +689,18 @@ ecm_bind(struct usb_configuration *c, struct usb_function *f) ecm_opts = container_of(f->fi, struct f_ecm_opts, func_inst); - mutex_lock(&ecm_opts->lock); - - gether_set_gadget(ecm_opts->net, cdev->gadget); - - if (!ecm_opts->bound) { - status = gether_register_netdev(ecm_opts->net); - ecm_opts->bound = true; - } - - mutex_unlock(&ecm_opts->lock); - if (status) - return status; + scoped_guard(mutex, &ecm_opts->lock) + if (ecm_opts->bind_count == 0 && !ecm_opts->bound) { + if (!device_is_registered(&ecm_opts->net->dev)) { + gether_set_gadget(ecm_opts->net, cdev->gadget); + status = gether_register_netdev(ecm_opts->net); + } else + status = gether_attach_gadget(ecm_opts->net, cdev->gadget); + + if (status) + return status; + net = ecm_opts->net; + } ecm_string_defs[1].s = ecm->ethaddr; @@ -790,6 +791,9 @@ ecm_bind(struct usb_configuration *c, struct usb_function *f) ecm->notify_req = no_free_ptr(request); + ecm_opts->bind_count++; + no_free_ptr(net); + DBG(cdev, "CDC Ethernet: IN/%s OUT/%s NOTIFY/%s\n", ecm->port.in_ep->name, ecm->port.out_ep->name, ecm->notify->name); @@ -836,7 +840,7 @@ static void ecm_free_inst(struct usb_function_instance *f) struct f_ecm_opts *opts; opts = container_of(f, struct f_ecm_opts, func_inst); - if (opts->bound) + if (device_is_registered(&opts->net->dev)) gether_cleanup(netdev_priv(opts->net)); else free_netdev(opts->net); @@ -906,9 +910,12 @@ static void ecm_free(struct usb_function *f) static void ecm_unbind(struct usb_configuration *c, struct usb_function *f) { struct f_ecm *ecm = func_to_ecm(f); + struct f_ecm_opts *ecm_opts; DBG(c->cdev, "ecm unbind\n"); + ecm_opts = container_of(f->fi, struct f_ecm_opts, func_inst); + usb_free_all_descriptors(f); if (atomic_read(&ecm->notify_count)) { @@ -918,6 +925,10 @@ static void ecm_unbind(struct usb_configuration *c, struct usb_function *f) kfree(ecm->notify_req->buf); usb_ep_free_request(ecm->notify, ecm->notify_req); + + ecm_opts->bind_count--; + if (ecm_opts->bind_count == 0 && !ecm_opts->bound) + gether_detach_gadget(ecm_opts->net); } static struct usb_function *ecm_alloc(struct usb_function_instance *fi) diff --git a/drivers/usb/gadget/function/f_eem.c b/drivers/usb/gadget/function/f_eem.c index f2478b48a30b..8480f130db11 100644 --- a/drivers/usb/gadget/function/f_eem.c +++ b/drivers/usb/gadget/function/f_eem.c @@ -7,6 +7,7 @@ * Copyright (C) 2009 EF Johnson Technologies */ +#include #include #include #include @@ -251,24 +252,22 @@ static int eem_bind(struct usb_configuration *c, struct usb_function *f) struct usb_ep *ep; struct f_eem_opts *eem_opts; + struct net_device *net __free(detach_gadget) = NULL; eem_opts = container_of(f->fi, struct f_eem_opts, func_inst); - /* - * in drivers/usb/gadget/configfs.c:configfs_composite_bind() - * configurations are bound in sequence with list_for_each_entry, - * in each configuration its functions are bound in sequence - * with list_for_each_entry, so we assume no race condition - * with regard to eem_opts->bound access - */ - if (!eem_opts->bound) { - mutex_lock(&eem_opts->lock); - gether_set_gadget(eem_opts->net, cdev->gadget); - status = gether_register_netdev(eem_opts->net); - mutex_unlock(&eem_opts->lock); - if (status) - return status; - eem_opts->bound = true; - } + + scoped_guard(mutex, &eem_opts->lock) + if (eem_opts->bind_count == 0 && !eem_opts->bound) { + if (!device_is_registered(&eem_opts->net->dev)) { + gether_set_gadget(eem_opts->net, cdev->gadget); + status = gether_register_netdev(eem_opts->net); + } else + status = gether_attach_gadget(eem_opts->net, cdev->gadget); + + if (status) + return status; + net = eem_opts->net; + } us = usb_gstrings_attach(cdev, eem_strings, ARRAY_SIZE(eem_string_defs)); @@ -279,21 +278,19 @@ static int eem_bind(struct usb_configuration *c, struct usb_function *f) /* allocate instance-specific interface IDs */ status = usb_interface_id(c, f); if (status < 0) - goto fail; + return status; eem->ctrl_id = status; eem_intf.bInterfaceNumber = status; - status = -ENODEV; - /* allocate instance-specific endpoints */ ep = usb_ep_autoconfig(cdev->gadget, &eem_fs_in_desc); if (!ep) - goto fail; + return -ENODEV; eem->port.in_ep = ep; ep = usb_ep_autoconfig(cdev->gadget, &eem_fs_out_desc); if (!ep) - goto fail; + return -ENODEV; eem->port.out_ep = ep; /* support all relevant hardware speeds... we expect that when @@ -309,16 +306,14 @@ static int eem_bind(struct usb_configuration *c, struct usb_function *f) status = usb_assign_descriptors(f, eem_fs_function, eem_hs_function, eem_ss_function, eem_ss_function); if (status) - goto fail; + return status; + + eem_opts->bind_count++; + no_free_ptr(net); DBG(cdev, "CDC Ethernet (EEM): IN/%s OUT/%s\n", eem->port.in_ep->name, eem->port.out_ep->name); return 0; - -fail: - ERROR(cdev, "%s: can't bind, err %d\n", f->name, status); - - return status; } static void eem_cmd_complete(struct usb_ep *ep, struct usb_request *req) @@ -597,7 +592,7 @@ static void eem_free_inst(struct usb_function_instance *f) struct f_eem_opts *opts; opts = container_of(f, struct f_eem_opts, func_inst); - if (opts->bound) + if (device_is_registered(&opts->net->dev)) gether_cleanup(netdev_priv(opts->net)); else free_netdev(opts->net); @@ -640,9 +635,17 @@ static void eem_free(struct usb_function *f) static void eem_unbind(struct usb_configuration *c, struct usb_function *f) { + struct f_eem_opts *opts; + DBG(c->cdev, "eem unbind\n"); + opts = container_of(f->fi, struct f_eem_opts, func_inst); + usb_free_all_descriptors(f); + + opts->bind_count--; + if (opts->bind_count == 0 && !opts->bound) + gether_detach_gadget(opts->net); } static struct usb_function *eem_alloc(struct usb_function_instance *fi) diff --git a/drivers/usb/gadget/function/f_fs.c b/drivers/usb/gadget/function/f_fs.c index 9a8c94bd9072..81465fcd1315 100644 --- a/drivers/usb/gadget/function/f_fs.c +++ b/drivers/usb/gadget/function/f_fs.c @@ -51,8 +51,8 @@ static struct ffs_data *__must_check ffs_data_new(const char *dev_name) __attribute__((malloc)); /* Opened counter handling. */ -static void ffs_data_opened(struct ffs_data *ffs); static void ffs_data_closed(struct ffs_data *ffs); +static void ffs_data_reset(struct ffs_data *ffs); /* Called with ffs->mutex held; take over ownership of data. */ static int __must_check @@ -613,13 +613,35 @@ static ssize_t ffs_ep0_read(struct file *file, char __user *buf, static int ffs_ep0_open(struct inode *inode, struct file *file) { - struct ffs_data *ffs = inode->i_private; + struct ffs_data *ffs = inode->i_sb->s_fs_info; + bool reset; + int ret; + + /* Acquire mutex */ + ret = ffs_mutex_lock(&ffs->mutex, file->f_flags & O_NONBLOCK); + if (ret < 0) + return ret; - if (ffs->state == FFS_CLOSING) + if (ffs->state == FFS_CLOSING) { + mutex_unlock(&ffs->mutex); return -EBUSY; + } + /* + * ffs_data_reset() ends up taking ffs->mutex via gadget unbind, so + * it can't run under it: count the open without resetting, and + * reset once the mutex is dropped. + */ + refcount_inc(&ffs->ref); + reset = atomic_add_return(1, &ffs->opened) == 1 && + ffs->state == FFS_DEACTIVATED; + if (reset) + ffs->state = FFS_CLOSING; + mutex_unlock(&ffs->mutex); file->private_data = ffs; - ffs_data_opened(ffs); + + if (reset) + ffs_data_reset(ffs); return stream_open(inode, file); } @@ -831,15 +853,21 @@ static void ffs_user_copy_worker(struct work_struct *work) unsigned long flags; if (io_data->read && ret > 0) { - kthread_use_mm(io_data->mm); - ret = ffs_copy_to_iter(io_data->buf, ret, &io_data->data); - kthread_unuse_mm(io_data->mm); + if (mmget_not_zero(io_data->mm)) { + kthread_use_mm(io_data->mm); + ret = ffs_copy_to_iter(io_data->buf, ret, &io_data->data); + kthread_unuse_mm(io_data->mm); + mmput(io_data->mm); + } else { + ret = -EFAULT; + } + mmdrop(io_data->mm); } io_data->kiocb->ki_complete(io_data->kiocb, ret); if (io_data->ffs->ffs_eventfd && !kiocb_has_eventfd) - eventfd_signal(io_data->ffs->ffs_eventfd, 1); + eventfd_signal(io_data->ffs->ffs_eventfd); spin_lock_irqsave(&io_data->ffs->eps_lock, flags); usb_ep_free_request(io_data->ep, io_data->req); @@ -946,31 +974,44 @@ static ssize_t __ffs_epfile_read_data(struct ffs_epfile *epfile, return ret; } -static ssize_t ffs_epfile_io(struct file *file, struct ffs_io_data *io_data) +static struct ffs_ep *ffs_epfile_wait_ep(struct file *file) { struct ffs_epfile *epfile = file->private_data; - struct usb_request *req; struct ffs_ep *ep; - char *data = NULL; - ssize_t ret, data_len = -EINVAL; - int halt; - - /* Are we still active? */ - if (WARN_ON(epfile->ffs->state != FFS_ACTIVE)) - return -ENODEV; + int ret; /* Wait for endpoint to be enabled */ ep = epfile->ep; if (!ep) { if (file->f_flags & O_NONBLOCK) - return -EAGAIN; + return ERR_PTR(-EAGAIN); ret = wait_event_interruptible( epfile->ffs->wait, (ep = epfile->ep)); if (ret) - return -EINTR; + return ERR_PTR(-EINTR); } + return ep; +} + +static ssize_t ffs_epfile_io(struct file *file, struct ffs_io_data *io_data) +{ + struct ffs_epfile *epfile = file->private_data; + struct usb_request *req; + struct ffs_ep *ep; + char *data = NULL; + ssize_t ret, data_len = -EINVAL; + int halt; + + /* Are we still active? */ + if (WARN_ON(epfile->ffs->state != FFS_ACTIVE)) + return -ENODEV; + + ep = ffs_epfile_wait_ep(file); + if (IS_ERR(ep)) + return PTR_ERR(ep); + /* Do we halt? */ halt = (!io_data->read == !epfile->in); if (halt && epfile->isoc) @@ -1158,14 +1199,34 @@ static ssize_t ffs_epfile_io(struct file *file, struct ffs_io_data *io_data) static int ffs_epfile_open(struct inode *inode, struct file *file) { - struct ffs_epfile *epfile = inode->i_private; + struct ffs_data *ffs = inode->i_sb->s_fs_info; + struct ffs_epfile *epfile; + int ret; - if (WARN_ON(epfile->ffs->state != FFS_ACTIVE)) + /* Acquire mutex */ + ret = ffs_mutex_lock(&ffs->mutex, file->f_flags & O_NONBLOCK); + if (ret < 0) + return ret; + + if (!atomic_inc_not_zero(&ffs->opened)) { + mutex_unlock(&ffs->mutex); + return -ENODEV; + } + refcount_inc(&ffs->ref); + /* + * we want the state to be FFS_ACTIVE; FFS_ACTIVE alone is + * not enough, though - we might have been through FFS_CLOSING + * and back to FFS_ACTIVE, with our file already removed. + */ + epfile = smp_load_acquire(&inode->i_private); + if (unlikely(ffs->state != FFS_ACTIVE || !epfile)) { + mutex_unlock(&ffs->mutex); + ffs_data_closed(ffs); return -ENODEV; + } + mutex_unlock(&ffs->mutex); file->private_data = epfile; - ffs_data_opened(epfile->ffs); - return stream_open(inode, file); } @@ -1210,16 +1271,22 @@ static ssize_t ffs_epfile_write_iter(struct kiocb *kiocb, struct iov_iter *from) kiocb->private = p; - if (p->aio) + if (p->aio) { + mmgrab(p->mm); kiocb_set_cancel_fn(kiocb, ffs_aio_cancel); + } res = ffs_epfile_io(kiocb->ki_filp, p); if (res == -EIOCBQUEUED) return res; - if (p->aio) + if (p->aio) { + kiocb->ki_complete(kiocb, res); + mmdrop(p->mm); kfree(p); - else + return -EIOCBQUEUED; + } else { *from = p->data; + } return res; } @@ -1254,16 +1321,21 @@ static ssize_t ffs_epfile_read_iter(struct kiocb *kiocb, struct iov_iter *to) kiocb->private = p; - if (p->aio) + if (p->aio) { + mmgrab(p->mm); kiocb_set_cancel_fn(kiocb, ffs_aio_cancel); + } res = ffs_epfile_io(kiocb->ki_filp, p); if (res == -EIOCBQUEUED) return res; if (p->aio) { + kiocb->ki_complete(kiocb, res); + mmdrop(p->mm); kfree(p->to_free); kfree(p); + return -EIOCBQUEUED; } else { *to = p->data; } @@ -1273,7 +1345,7 @@ static ssize_t ffs_epfile_read_iter(struct kiocb *kiocb, struct iov_iter *to) static int ffs_epfile_release(struct inode *inode, struct file *file) { - struct ffs_epfile *epfile = inode->i_private; + struct ffs_epfile *epfile = file->private_data; ffs_data_closed(epfile->ffs); @@ -1291,16 +1363,9 @@ static long ffs_epfile_ioctl(struct file *file, unsigned code, return -ENODEV; /* Wait for endpoint to be enabled */ - ep = epfile->ep; - if (!ep) { - if (file->f_flags & O_NONBLOCK) - return -EAGAIN; - - ret = wait_event_interruptible( - epfile->ffs->wait, (ep = epfile->ep)); - if (ret) - return -EINTR; - } + ep = ffs_epfile_wait_ep(file); + if (IS_ERR(ep)) + return PTR_ERR(ep); spin_lock_irq(&epfile->ffs->eps_lock); @@ -1326,8 +1391,9 @@ static long ffs_epfile_ioctl(struct file *file, unsigned code, break; case FUNCTIONFS_ENDPOINT_DESC: { + const struct usb_endpoint_descriptor *desc; + struct usb_endpoint_descriptor desc1; int desc_idx; - struct usb_endpoint_descriptor desc1, *desc; switch (epfile->ffs->gadget->speed) { case USB_SPEED_SUPER: @@ -1341,7 +1407,17 @@ static long ffs_epfile_ioctl(struct file *file, unsigned code, desc_idx = 0; } - desc = epfile->ep->descs[desc_idx]; + do { + desc = epfile->ep->descs[desc_idx]; + } while (!desc && --desc_idx >= 0); + + if (!desc) + desc = epfile->ep->ep->desc; + if (!desc) { + ret = -EINVAL; + break; + } + memcpy(&desc1, desc, desc->bLength); spin_unlock_irq(&epfile->ffs->eps_lock); @@ -1662,16 +1738,6 @@ static void ffs_data_get(struct ffs_data *ffs) refcount_inc(&ffs->ref); } -static void ffs_data_opened(struct ffs_data *ffs) -{ - refcount_inc(&ffs->ref); - if (atomic_add_return(1, &ffs->opened) == 1 && - ffs->state == FFS_DEACTIVATED) { - ffs->state = FFS_CLOSING; - ffs_data_reset(ffs); - } -} - static void ffs_data_put(struct ffs_data *ffs) { if (refcount_dec_and_test(&ffs->ref)) { @@ -1892,6 +1958,11 @@ static int ffs_epfiles_create(struct ffs_data *ffs) return 0; } +static void clear_one(struct dentry *dentry) +{ + smp_store_release(&dentry->d_inode->i_private, NULL); +} + static void ffs_epfiles_destroy(struct ffs_epfile *epfiles, unsigned count) { struct ffs_epfile *epfile = epfiles; @@ -1900,8 +1971,7 @@ static void ffs_epfiles_destroy(struct ffs_epfile *epfiles, unsigned count) BUG_ON(mutex_is_locked(&epfile->mutex)); __ffs_epfile_read_buffer_free(epfile); if (epfile->dentry) { - d_delete(epfile->dentry); - dput(epfile->dentry); + simple_recursive_removal(epfile->dentry, clear_one); epfile->dentry = NULL; } } @@ -2757,7 +2827,7 @@ static void __ffs_event_add(struct ffs_data *ffs, ffs->ev.types[ffs->ev.count++] = type; wake_up_locked(&ffs->ev.waitq); if (ffs->ffs_eventfd) - eventfd_signal(ffs->ffs_eventfd, 1); + eventfd_signal(ffs->ffs_eventfd); } static void ffs_event_add(struct ffs_data *ffs, diff --git a/drivers/usb/gadget/function/f_hid.c b/drivers/usb/gadget/function/f_hid.c index 124696479098..d7bc83b5f1a9 100644 --- a/drivers/usb/gadget/function/f_hid.c +++ b/drivers/usb/gadget/function/f_hid.c @@ -76,7 +76,7 @@ struct f_hidg { struct usb_request *req; struct device dev; - struct cdev cdev; + struct cdev *cdev; struct usb_function func; struct usb_ep *in_ep; @@ -558,8 +558,9 @@ static int f_hidg_release(struct inode *inode, struct file *fd) static int f_hidg_open(struct inode *inode, struct file *fd) { + struct kobject *parent = inode->i_cdev->kobj.parent; struct f_hidg *hidg = - container_of(inode->i_cdev, struct f_hidg, cdev); + container_of(parent, struct f_hidg, dev.kobj); fd->private_data = hidg; @@ -1004,8 +1005,14 @@ static int hidg_bind(struct usb_configuration *c, struct usb_function *f) hidg->req = NULL; /* create char device */ - cdev_init(&hidg->cdev, &f_hidg_fops); - status = cdev_device_add(&hidg->cdev, &hidg->dev); + hidg->cdev = cdev_alloc(); + if (!hidg->cdev) { + status = -ENOMEM; + goto fail_free_descs; + } + hidg->cdev->ops = &f_hidg_fops; + + status = cdev_device_add(hidg->cdev, &hidg->dev); if (status) goto fail_free_descs; @@ -1250,7 +1257,7 @@ static void hidg_unbind(struct usb_configuration *c, struct usb_function *f) { struct f_hidg *hidg = func_to_hidg(f); - cdev_device_del(&hidg->cdev, &hidg->dev); + cdev_device_del(hidg->cdev, &hidg->dev); usb_free_all_descriptors(f); } diff --git a/drivers/usb/gadget/function/f_mass_storage.c b/drivers/usb/gadget/function/f_mass_storage.c index 01c35deaefe1..4a7de2ba91cb 100644 --- a/drivers/usb/gadget/function/f_mass_storage.c +++ b/drivers/usb/gadget/function/f_mass_storage.c @@ -2748,6 +2748,9 @@ int fsg_common_set_num_buffers(struct fsg_common *common, unsigned int n) struct fsg_buffhd *bh, *buffhds; int i; + if (n < 2) + return -EINVAL; + buffhds = kcalloc(n, sizeof(*buffhds), GFP_KERNEL); if (!buffhds) return -ENOMEM; diff --git a/drivers/usb/gadget/function/f_midi.c b/drivers/usb/gadget/function/f_midi.c index 680651e91a52..33ee0e3f18eb 100644 --- a/drivers/usb/gadget/function/f_midi.c +++ b/drivers/usb/gadget/function/f_midi.c @@ -99,7 +99,7 @@ struct f_midi { unsigned int in_last_port; unsigned char free_ref; - struct gmidi_in_port in_ports_array[/* in_ports */]; + struct gmidi_in_port in_ports_array[] __counted_by(in_ports); }; static inline struct f_midi *func_to_midi(struct usb_function *f) @@ -877,7 +877,6 @@ static int f_midi_bind(struct usb_configuration *c, struct usb_function *f) int status, n, jack = 1, i = 0, endpoint_descriptor_index = 0; midi->gadget = cdev->gadget; - INIT_WORK(&midi->work, f_midi_in_work); status = f_midi_register_card(midi); if (status < 0) goto fail_register; @@ -1359,6 +1358,7 @@ static struct usb_function *f_midi_alloc(struct usb_function_instance *fi) status = -ENOMEM; goto setup_fail; } + midi->in_ports = opts->in_ports; for (i = 0; i < opts->in_ports; i++) midi->in_ports_array[i].cable = i; @@ -1369,7 +1369,7 @@ static struct usb_function *f_midi_alloc(struct usb_function_instance *fi) status = -ENOMEM; goto midi_free; } - midi->in_ports = opts->in_ports; + INIT_WORK(&midi->work, f_midi_in_work); midi->out_ports = opts->out_ports; midi->index = opts->index; midi->buflen = opts->buflen; diff --git a/drivers/usb/gadget/function/f_midi2.c b/drivers/usb/gadget/function/f_midi2.c index 0f47cca32c3c..13d5be51b4bf 100644 --- a/drivers/usb/gadget/function/f_midi2.c +++ b/drivers/usb/gadget/function/f_midi2.c @@ -1636,8 +1636,8 @@ struct f_midi2_usb_config { /* MIDI 1.0 jacks */ unsigned char jack_in, jack_out, jack_id; - struct usb_midi_in_jack_descriptor jack_ins[MAX_CABLES]; - struct usb_midi_out_jack_descriptor_1 jack_outs[MAX_CABLES]; + struct usb_midi_in_jack_descriptor jack_ins[MAX_CABLES * 2]; + struct usb_midi_out_jack_descriptor_1 jack_outs[MAX_CABLES * 2]; }; static int append_config(struct f_midi2_usb_config *config, void *d) diff --git a/drivers/usb/gadget/function/f_rndis.c b/drivers/usb/gadget/function/f_rndis.c index c11728644273..bac60612ffa6 100644 --- a/drivers/usb/gadget/function/f_rndis.c +++ b/drivers/usb/gadget/function/f_rndis.c @@ -666,6 +666,7 @@ rndis_bind(struct usb_configuration *c, struct usb_function *f) struct f_rndis_opts *rndis_opts; struct usb_os_desc_table *os_desc_table __free(kfree) = NULL; + struct net_device *net __free(detach_gadget) = NULL; struct usb_request *request __free(free_usb_request) = NULL; if (!can_support_rndis(c)) @@ -683,21 +684,18 @@ rndis_bind(struct usb_configuration *c, struct usb_function *f) rndis_iad_descriptor.bFunctionClass = rndis_opts->class; rndis_iad_descriptor.bFunctionSubClass = rndis_opts->subclass; rndis_iad_descriptor.bFunctionProtocol = rndis_opts->protocol; - } - /* - * in drivers/usb/gadget/configfs.c:configfs_composite_bind() - * configurations are bound in sequence with list_for_each_entry, - * in each configuration its functions are bound in sequence - * with list_for_each_entry, so we assume no race condition - * with regard to rndis_opts->bound access - */ - if (!rndis_opts->bound) { - gether_set_gadget(rndis_opts->net, cdev->gadget); - status = gether_register_netdev(rndis_opts->net); - if (status) - return status; - rndis_opts->bound = true; + if (rndis_opts->bind_count == 0 && !rndis_opts->borrowed_net) { + if (!device_is_registered(&rndis_opts->net->dev)) { + gether_set_gadget(rndis_opts->net, cdev->gadget); + status = gether_register_netdev(rndis_opts->net); + } else + status = gether_attach_gadget(rndis_opts->net, cdev->gadget); + + if (status) + return status; + net = rndis_opts->net; + } } us = usb_gstrings_attach(cdev, rndis_strings, @@ -796,6 +794,9 @@ rndis_bind(struct usb_configuration *c, struct usb_function *f) } rndis->notify_req = no_free_ptr(request); + rndis_opts->bind_count++; + no_free_ptr(net); + /* NOTE: all that is done without knowing or caring about * the network link ... which is unavailable to this code * until we're activated via set_alt(). @@ -812,11 +813,11 @@ void rndis_borrow_net(struct usb_function_instance *f, struct net_device *net) struct f_rndis_opts *opts; opts = container_of(f, struct f_rndis_opts, func_inst); - if (opts->bound) + if (device_is_registered(&opts->net->dev)) gether_cleanup(netdev_priv(opts->net)); else free_netdev(opts->net); - opts->borrowed_net = opts->bound = true; + opts->borrowed_net = true; opts->net = net; } EXPORT_SYMBOL_GPL(rndis_borrow_net); @@ -874,7 +875,7 @@ static void rndis_free_inst(struct usb_function_instance *f) opts = container_of(f, struct f_rndis_opts, func_inst); if (!opts->borrowed_net) { - if (opts->bound) + if (device_is_registered(&opts->net->dev)) gether_cleanup(netdev_priv(opts->net)); else free_netdev(opts->net); @@ -943,6 +944,9 @@ static void rndis_free(struct usb_function *f) static void rndis_unbind(struct usb_configuration *c, struct usb_function *f) { struct f_rndis *rndis = func_to_rndis(f); + struct f_rndis_opts *rndis_opts; + + rndis_opts = container_of(f->fi, struct f_rndis_opts, func_inst); kfree(f->os_desc_table); f->os_desc_n = 0; @@ -950,6 +954,10 @@ static void rndis_unbind(struct usb_configuration *c, struct usb_function *f) kfree(rndis->notify_req->buf); usb_ep_free_request(rndis->notify, rndis->notify_req); + + rndis_opts->bind_count--; + if (rndis_opts->bind_count == 0 && !rndis_opts->borrowed_net) + gether_detach_gadget(rndis_opts->net); } static struct usb_function *rndis_alloc(struct usb_function_instance *fi) diff --git a/drivers/usb/gadget/function/f_subset.c b/drivers/usb/gadget/function/f_subset.c index 7d209382c068..489b1f57f5c1 100644 --- a/drivers/usb/gadget/function/f_subset.c +++ b/drivers/usb/gadget/function/f_subset.c @@ -299,25 +299,22 @@ geth_bind(struct usb_configuration *c, struct usb_function *f) struct usb_ep *ep; struct f_gether_opts *gether_opts; + struct net_device *net __free(detach_gadget) = NULL; gether_opts = container_of(f->fi, struct f_gether_opts, func_inst); - /* - * in drivers/usb/gadget/configfs.c:configfs_composite_bind() - * configurations are bound in sequence with list_for_each_entry, - * in each configuration its functions are bound in sequence - * with list_for_each_entry, so we assume no race condition - * with regard to gether_opts->bound access - */ - if (!gether_opts->bound) { - mutex_lock(&gether_opts->lock); - gether_set_gadget(gether_opts->net, cdev->gadget); - status = gether_register_netdev(gether_opts->net); - mutex_unlock(&gether_opts->lock); - if (status) - return status; - gether_opts->bound = true; - } + scoped_guard(mutex, &gether_opts->lock) + if (gether_opts->bind_count == 0 && !gether_opts->bound) { + if (!device_is_registered(&gether_opts->net->dev)) { + gether_set_gadget(gether_opts->net, cdev->gadget); + status = gether_register_netdev(gether_opts->net); + } else + status = gether_attach_gadget(gether_opts->net, cdev->gadget); + + if (status) + return status; + net = gether_opts->net; + } us = usb_gstrings_attach(cdev, geth_strings, ARRAY_SIZE(geth_string_defs)); @@ -330,20 +327,18 @@ geth_bind(struct usb_configuration *c, struct usb_function *f) /* allocate instance-specific interface IDs */ status = usb_interface_id(c, f); if (status < 0) - goto fail; + return status; subset_data_intf.bInterfaceNumber = status; - status = -ENODEV; - /* allocate instance-specific endpoints */ ep = usb_ep_autoconfig(cdev->gadget, &fs_subset_in_desc); if (!ep) - goto fail; + return -ENODEV; geth->port.in_ep = ep; ep = usb_ep_autoconfig(cdev->gadget, &fs_subset_out_desc); if (!ep) - goto fail; + return -ENODEV; geth->port.out_ep = ep; /* support all relevant hardware speeds... we expect that when @@ -361,21 +356,19 @@ geth_bind(struct usb_configuration *c, struct usb_function *f) status = usb_assign_descriptors(f, fs_eth_function, hs_eth_function, ss_eth_function, ss_eth_function); if (status) - goto fail; + return status; /* NOTE: all that is done without knowing or caring about * the network link ... which is unavailable to this code * until we're activated via set_alt(). */ + gether_opts->bind_count++; + no_free_ptr(net); + DBG(cdev, "CDC Subset: IN/%s OUT/%s\n", geth->port.in_ep->name, geth->port.out_ep->name); return 0; - -fail: - ERROR(cdev, "%s: can't bind, err %d\n", f->name, status); - - return status; } static inline struct f_gether_opts *to_f_gether_opts(struct config_item *item) @@ -418,7 +411,7 @@ static void geth_free_inst(struct usb_function_instance *f) struct f_gether_opts *opts; opts = container_of(f, struct f_gether_opts, func_inst); - if (opts->bound) + if (device_is_registered(&opts->net->dev)) gether_cleanup(netdev_priv(opts->net)); else free_netdev(opts->net); @@ -462,8 +455,16 @@ static void geth_free(struct usb_function *f) static void geth_unbind(struct usb_configuration *c, struct usb_function *f) { + struct f_gether_opts *opts; + + opts = container_of(f->fi, struct f_gether_opts, func_inst); + geth_string_defs[0].id = 0; usb_free_all_descriptors(f); + + opts->bind_count--; + if (opts->bind_count == 0 && !opts->bound) + gether_detach_gadget(opts->net); } static struct usb_function *geth_alloc(struct usb_function_instance *fi) diff --git a/drivers/usb/gadget/function/f_uac1.c b/drivers/usb/gadget/function/f_uac1.c index 6f0e1d803dc2..7939dadd52a5 100644 --- a/drivers/usb/gadget/function/f_uac1.c +++ b/drivers/usb/gadget/function/f_uac1.c @@ -1521,7 +1521,8 @@ static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \ const char *page, size_t len) \ { \ struct f_uac1_opts *opts = to_f_uac1_opts(item); \ - char *split_page = NULL; \ + char *buf = NULL; \ + char *split_page; \ int ret = -EINVAL; \ char *token; \ u32 num; \ @@ -1535,18 +1536,22 @@ static ssize_t f_uac1_opts_##name##_store(struct config_item *item, \ \ i = 0; \ memset(opts->name##s, 0x00, sizeof(opts->name##s)); \ - split_page = kstrdup(page, GFP_KERNEL); \ + buf = kstrdup(page, GFP_KERNEL); \ + split_page = buf; \ while ((token = strsep(&split_page, ",")) != NULL) { \ ret = kstrtou32(token, 0, &num); \ if (ret) \ goto end; \ - \ + if (i >= UAC_MAX_RATES) { \ + ret = -EINVAL; \ + goto end; \ + } \ opts->name##s[i++] = num; \ ret = len; \ }; \ \ end: \ - kfree(split_page); \ + kfree(buf); \ mutex_unlock(&opts->lock); \ return ret; \ } \ diff --git a/drivers/usb/gadget/function/f_uac1_legacy.c b/drivers/usb/gadget/function/f_uac1_legacy.c index 6b1efbb57072..6724228da7ab 100644 --- a/drivers/usb/gadget/function/f_uac1_legacy.c +++ b/drivers/usb/gadget/function/f_uac1_legacy.c @@ -797,6 +797,9 @@ f_audio_bind(struct usb_configuration *c, struct usb_function *f) static int generic_set_cmd(struct usb_audio_control *con, u8 cmd, int value) { + if (cmd >= ARRAY_SIZE(con->data)) + return -EINVAL; + con->data[cmd] = value; return 0; @@ -804,6 +807,9 @@ static int generic_set_cmd(struct usb_audio_control *con, u8 cmd, int value) static int generic_get_cmd(struct usb_audio_control *con, u8 cmd) { + if (cmd >= ARRAY_SIZE(con->data)) + return -EINVAL; + return con->data[cmd]; } diff --git a/drivers/usb/gadget/function/f_uac2.c b/drivers/usb/gadget/function/f_uac2.c index eabd10817e1e..b7d3988caa9c 100644 --- a/drivers/usb/gadget/function/f_uac2.c +++ b/drivers/usb/gadget/function/f_uac2.c @@ -2003,7 +2003,8 @@ static ssize_t f_uac2_opts_##name##_store(struct config_item *item, \ const char *page, size_t len) \ { \ struct f_uac2_opts *opts = to_f_uac2_opts(item); \ - char *split_page = NULL; \ + char *buf = NULL; \ + char *split_page; \ int ret = -EINVAL; \ char *token; \ u32 num; \ @@ -2017,18 +2018,22 @@ static ssize_t f_uac2_opts_##name##_store(struct config_item *item, \ \ i = 0; \ memset(opts->name##s, 0x00, sizeof(opts->name##s)); \ - split_page = kstrdup(page, GFP_KERNEL); \ + buf = kstrdup(page, GFP_KERNEL); \ + split_page = buf; \ while ((token = strsep(&split_page, ",")) != NULL) { \ ret = kstrtou32(token, 0, &num); \ if (ret) \ goto end; \ - \ + if (i >= UAC_MAX_RATES) { \ + ret = -EINVAL; \ + goto end; \ + } \ opts->name##s[i++] = num; \ ret = len; \ }; \ \ end: \ - kfree(split_page); \ + kfree(buf); \ mutex_unlock(&opts->lock); \ return ret; \ } \ diff --git a/drivers/usb/gadget/function/u_ecm.h b/drivers/usb/gadget/function/u_ecm.h index 77cfb89932be..7f666b9dea02 100644 --- a/drivers/usb/gadget/function/u_ecm.h +++ b/drivers/usb/gadget/function/u_ecm.h @@ -15,17 +15,26 @@ #include +/** + * struct f_ecm_opts - ECM function options + * @func_inst: USB function instance. + * @net: The net_device associated with the ECM function. + * @bound: True if the net_device is shared and pre-registered during the + * legacy composite driver's bind phase (e.g., multi.c). If false, + * the ECM function will register the net_device during its own + * bind phase. + * @bind_count: Tracks the number of configurations the ECM function is + * bound to, preventing double-registration of the @net device. + * @lock: Protects the data from concurrent access by configfs read/write + * and create symlink/remove symlink operations. + * @refcnt: Reference counter for the function instance. + */ struct f_ecm_opts { struct usb_function_instance func_inst; struct net_device *net; bool bound; + int bind_count; - /* - * Read/write access to configfs attributes is handled by configfs. - * - * This is to protect the data from concurrent access by read/write - * and create symlink/remove symlink. - */ struct mutex lock; int refcnt; }; diff --git a/drivers/usb/gadget/function/u_eem.h b/drivers/usb/gadget/function/u_eem.h index 3bd85dfcd71c..78ef55815219 100644 --- a/drivers/usb/gadget/function/u_eem.h +++ b/drivers/usb/gadget/function/u_eem.h @@ -15,17 +15,26 @@ #include +/** + * struct f_eem_opts - EEM function options + * @func_inst: USB function instance. + * @net: The net_device associated with the EEM function. + * @bound: True if the net_device is shared and pre-registered during the + * legacy composite driver's bind phase (e.g., multi.c). If false, + * the EEM function will register the net_device during its own + * bind phase. + * @bind_count: Tracks the number of configurations the EEM function is + * bound to, preventing double-registration of the @net device. + * @lock: Protects the data from concurrent access by configfs read/write + * and create symlink/remove symlink operations. + * @refcnt: Reference counter for the function instance. + */ struct f_eem_opts { struct usb_function_instance func_inst; struct net_device *net; bool bound; + int bind_count; - /* - * Read/write access to configfs attributes is handled by configfs. - * - * This is to protect the data from concurrent access by read/write - * and create symlink/remove symlink. - */ struct mutex lock; int refcnt; }; diff --git a/drivers/usb/gadget/function/u_gether.h b/drivers/usb/gadget/function/u_gether.h index 2f7a373ed449..e7b6b51f69c1 100644 --- a/drivers/usb/gadget/function/u_gether.h +++ b/drivers/usb/gadget/function/u_gether.h @@ -15,17 +15,25 @@ #include +/** + * struct f_gether_opts - subset function options + * @func_inst: USB function instance. + * @net: The net_device associated with the subset function. + * @bound: True if the net_device is shared and pre-registered during the + * legacy composite driver's bind phase (e.g., multi.c). If false, + * the subset function will register the net_device during its own + * bind phase. + * @bind_count: Tracks the number of configurations the subset function is + * bound to, preventing double-registration of the @net device. + * @lock: Protects the data from concurrent access by configfs read/write + * and create symlink/remove symlink operations. + * @refcnt: Reference counter for the function instance. + */ struct f_gether_opts { struct usb_function_instance func_inst; struct net_device *net; bool bound; - - /* - * Read/write access to configfs attributes is handled by configfs. - * - * This is to protect the data from concurrent access by read/write - * and create symlink/remove symlink. - */ + int bind_count; struct mutex lock; int refcnt; }; diff --git a/drivers/usb/gadget/function/u_rndis.h b/drivers/usb/gadget/function/u_rndis.h index a8c409b2f52f..4e64619714dc 100644 --- a/drivers/usb/gadget/function/u_rndis.h +++ b/drivers/usb/gadget/function/u_rndis.h @@ -15,12 +15,34 @@ #include +/** + * struct f_rndis_opts - RNDIS function options + * @func_inst: USB function instance. + * @vendor_id: Vendor ID. + * @manufacturer: Manufacturer string. + * @net: The net_device associated with the RNDIS function. + * @bind_count: Tracks the number of configurations the RNDIS function is + * bound to, preventing double-registration of the @net device. + * @borrowed_net: True if the net_device is shared and pre-registered during + * the legacy composite driver's bind phase (e.g., multi.c). + * If false, the RNDIS function will register the net_device + * during its own bind phase. + * @rndis_interf_group: ConfigFS group for RNDIS interface. + * @rndis_os_desc: USB OS descriptor for RNDIS. + * @rndis_ext_compat_id: Extended compatibility ID. + * @class: USB class. + * @subclass: USB subclass. + * @protocol: USB protocol. + * @lock: Protects the data from concurrent access by configfs read/write + * and create symlink/remove symlink operations. + * @refcnt: Reference counter for the function instance. + */ struct f_rndis_opts { struct usb_function_instance func_inst; u32 vendor_id; const char *manufacturer; struct net_device *net; - bool bound; + int bind_count; bool borrowed_net; struct config_group *rndis_interf_group; @@ -30,13 +52,6 @@ struct f_rndis_opts { u8 class; u8 subclass; u8 protocol; - - /* - * Read/write access to configfs attributes is handled by configfs. - * - * This is to protect the data from concurrent access by read/write - * and create symlink/remove symlink. - */ struct mutex lock; int refcnt; }; diff --git a/drivers/usb/gadget/legacy/inode.c b/drivers/usb/gadget/legacy/inode.c index 07b1efc94428..3889b0ebdb90 100644 --- a/drivers/usb/gadget/legacy/inode.c +++ b/drivers/usb/gadget/legacy/inode.c @@ -1547,7 +1547,6 @@ static void destroy_ep_files (struct dev_data *dev) spin_lock_irq (&dev->lock); while (!list_empty(&dev->epfiles)) { struct ep_data *ep; - struct inode *parent; struct dentry *dentry; /* break link to FS */ @@ -1557,7 +1556,6 @@ static void destroy_ep_files (struct dev_data *dev) dentry = ep->dentry; ep->dentry = NULL; - parent = d_inode(dentry->d_parent); /* break link to controller */ mutex_lock(&ep->lock); @@ -1572,10 +1570,7 @@ static void destroy_ep_files (struct dev_data *dev) put_ep (ep); /* break link to dcache */ - inode_lock(parent); - d_delete (dentry); - dput (dentry); - inode_unlock(parent); + simple_recursive_removal(dentry, NULL); spin_lock_irq (&dev->lock); } diff --git a/drivers/usb/gadget/udc/aspeed-vhub/core.c b/drivers/usb/gadget/udc/aspeed-vhub/core.c index f60a019bb173..d34575059f9e 100644 --- a/drivers/usb/gadget/udc/aspeed-vhub/core.c +++ b/drivers/usb/gadget/udc/aspeed-vhub/core.c @@ -266,6 +266,9 @@ static void ast_vhub_remove(struct platform_device *pdev) for (i = 0; i < vhub->max_ports; i++) ast_vhub_del_dev(&vhub->ports[i].dev); + /* Final drain; the worker takes vhub->lock, so stay outside of it */ + cancel_work_sync(&vhub->wake_work); + spin_lock_irqsave(&vhub->lock, flags); /* Mask & ack all interrupts */ @@ -324,6 +327,7 @@ static int ast_vhub_probe(struct platform_device *pdev) return -ENOMEM; spin_lock_init(&vhub->lock); + INIT_WORK(&vhub->wake_work, ast_vhub_wake_work); vhub->pdev = pdev; vhub->port_irq_mask = GENMASK(VHUB_IRQ_DEV1_BIT + vhub->max_ports - 1, VHUB_IRQ_DEV1_BIT); diff --git a/drivers/usb/gadget/udc/aspeed-vhub/dev.c b/drivers/usb/gadget/udc/aspeed-vhub/dev.c index a09f72772e6e..477d0ae789c2 100644 --- a/drivers/usb/gadget/udc/aspeed-vhub/dev.c +++ b/drivers/usb/gadget/udc/aspeed-vhub/dev.c @@ -280,7 +280,7 @@ static int ast_vhub_udc_wakeup(struct usb_gadget* gadget) int rc = -EINVAL; spin_lock_irqsave(&d->vhub->lock, flags); - if (!d->wakeup_en) + if (!d->wakeup_en || !d->registered) goto err; DDBG(d, "Device initiated wakeup\n"); diff --git a/drivers/usb/gadget/udc/aspeed-vhub/hub.c b/drivers/usb/gadget/udc/aspeed-vhub/hub.c index a63e4af60a56..c60255b3fe9e 100644 --- a/drivers/usb/gadget/udc/aspeed-vhub/hub.c +++ b/drivers/usb/gadget/udc/aspeed-vhub/hub.c @@ -557,7 +557,7 @@ void ast_vhub_device_connect(struct ast_vhub *vhub, ast_vhub_send_host_wakeup(vhub); } -static void ast_vhub_wake_work(struct work_struct *work) +void ast_vhub_wake_work(struct work_struct *work) { struct ast_vhub *vhub = container_of(work, struct ast_vhub, @@ -587,6 +587,8 @@ static void ast_vhub_wake_work(struct work_struct *work) void ast_vhub_hub_wake_all(struct ast_vhub *vhub) { + lockdep_assert_held(&vhub->lock); + /* * A device is trying to wake the world, because this * can recurse into the device, we break the call chain @@ -1075,7 +1077,6 @@ static int ast_vhub_init_desc(struct ast_vhub *vhub) int ast_vhub_init_hub(struct ast_vhub *vhub) { vhub->speed = USB_SPEED_UNKNOWN; - INIT_WORK(&vhub->wake_work, ast_vhub_wake_work); return ast_vhub_init_desc(vhub); } diff --git a/drivers/usb/gadget/udc/aspeed-vhub/vhub.h b/drivers/usb/gadget/udc/aspeed-vhub/vhub.h index 6b9dfa6e10eb..0ca168b39d96 100644 --- a/drivers/usb/gadget/udc/aspeed-vhub/vhub.h +++ b/drivers/usb/gadget/udc/aspeed-vhub/vhub.h @@ -546,6 +546,7 @@ void ast_vhub_hub_suspend(struct ast_vhub *vhub); void ast_vhub_hub_resume(struct ast_vhub *vhub); void ast_vhub_hub_reset(struct ast_vhub *vhub); void ast_vhub_hub_wake_all(struct ast_vhub *vhub); +void ast_vhub_wake_work(struct work_struct *work); /* dev.c */ int ast_vhub_init_dev(struct ast_vhub *vhub, unsigned int idx); diff --git a/drivers/usb/gadget/udc/at91_udc.c b/drivers/usb/gadget/udc/at91_udc.c index 922b4187004b..408013eb23f6 100644 --- a/drivers/usb/gadget/udc/at91_udc.c +++ b/drivers/usb/gadget/udc/at91_udc.c @@ -1793,6 +1793,19 @@ static void at91udc_of_init(struct at91_udc *udc, struct device_node *np) udc->caps = match->data; } +/* + * The work handler re-arms this timer, so shut the timer down before + * draining the work; otherwise it restarts the polling cycle. + */ +static void at91_udc_shutdown_vbus_timer(struct at91_udc *udc) +{ + if (!(udc->board.vbus_pin && udc->board.vbus_polled)) + return; + + timer_shutdown_sync(&udc->vbus_timer); + cancel_work_sync(&udc->vbus_timer_work); +} + static int at91udc_probe(struct platform_device *pdev) { struct device *dev = &pdev->dev; @@ -1906,7 +1919,7 @@ static int at91udc_probe(struct platform_device *pdev) } retval = usb_add_gadget_udc(dev, &udc->gadget); if (retval) - goto err_unprepare_iclk; + goto err_shutdown_vbus; dev_set_drvdata(dev, udc); device_init_wakeup(dev, 1); create_debug_file(udc); @@ -1914,6 +1927,8 @@ static int at91udc_probe(struct platform_device *pdev) INFO("%s version %s\n", driver_name, DRIVER_VERSION); return 0; +err_shutdown_vbus: + at91_udc_shutdown_vbus_timer(udc); err_unprepare_iclk: clk_unprepare(udc->iclk); err_unprepare_fclk: @@ -1924,7 +1939,7 @@ static int at91udc_probe(struct platform_device *pdev) return retval; } -static int at91udc_remove(struct platform_device *pdev) +static void at91udc_remove(struct platform_device *pdev) { struct at91_udc *udc = platform_get_drvdata(pdev); unsigned long flags; @@ -1932,8 +1947,14 @@ static int at91udc_remove(struct platform_device *pdev) DBG("remove\n"); usb_del_gadget_udc(&udc->gadget); - if (udc->driver) - return -EBUSY; + + at91_udc_shutdown_vbus_timer(udc); + + if (udc->driver) { + dev_err(&pdev->dev, + "Driver still in use but removing anyhow\n"); + return; + } spin_lock_irqsave(&udc->lock, flags); pullup(udc, 0); @@ -1943,8 +1964,6 @@ static int at91udc_remove(struct platform_device *pdev) remove_debug_file(udc); clk_unprepare(udc->fclk); clk_unprepare(udc->iclk); - - return 0; } #ifdef CONFIG_PM @@ -2000,7 +2019,7 @@ static int at91udc_resume(struct platform_device *pdev) #endif static struct platform_driver at91_udc_driver = { - .remove = at91udc_remove, + .remove_new = at91udc_remove, .shutdown = at91udc_shutdown, .suspend = at91udc_suspend, .resume = at91udc_resume, diff --git a/drivers/usb/gadget/udc/core.c b/drivers/usb/gadget/udc/core.c index 2ef28760e9ad..365d1f77856e 100644 --- a/drivers/usb/gadget/udc/core.c +++ b/drivers/usb/gadget/udc/core.c @@ -913,6 +913,11 @@ int usb_gadget_map_request_by_dev(struct device *dev, if (req->length == 0) return 0; + if (req->sg_was_mapped) { + req->num_mapped_sgs = req->num_sgs; + return 0; + } + if (req->num_sgs) { int mapped; @@ -958,7 +963,7 @@ EXPORT_SYMBOL_GPL(usb_gadget_map_request); void usb_gadget_unmap_request_by_dev(struct device *dev, struct usb_request *req, int is_in) { - if (req->length == 0) + if (req->length == 0 || req->sg_was_mapped) return; if (req->num_mapped_sgs) { diff --git a/drivers/usb/gadget/udc/dummy_hcd.c b/drivers/usb/gadget/udc/dummy_hcd.c index 0947746790be..26c2a87e9cd7 100644 --- a/drivers/usb/gadget/udc/dummy_hcd.c +++ b/drivers/usb/gadget/udc/dummy_hcd.c @@ -342,9 +342,11 @@ static void dummy_giveback(struct dummy *dum, struct usb_ep *_ep, { bool fifo = req == &dum->fifo_req; + ++dum->callback_usage; spin_unlock(&dum->lock); usb_gadget_giveback_request(_ep, &req->req); spin_lock(&dum->lock); + --dum->callback_usage; if (fifo) dum->fifo_req_busy = 0; } @@ -758,11 +760,13 @@ static int dummy_queue(struct usb_ep *_ep, struct usb_request *_req, req->req.complete = fifo_complete; list_add_tail(&req->queue, &ep->queue); + ++dum->callback_usage; spin_unlock(&dum->lock); _req->actual = _req->length; _req->status = 0; usb_gadget_giveback_request(_ep, _req); spin_lock(&dum->lock); + --dum->callback_usage; } else list_add_tail(&req->queue, &ep->queue); spin_unlock_irqrestore(&dum->lock, flags); diff --git a/drivers/usb/host/octeon-hcd.c b/drivers/usb/host/octeon-hcd.c index 19d5777f5db2..371e13f84061 100644 --- a/drivers/usb/host/octeon-hcd.c +++ b/drivers/usb/host/octeon-hcd.c @@ -578,7 +578,7 @@ static int cvmx_wait_tx_rx(struct octeon_hcd *usb, int fflsh_type) int result; u64 address = CVMX_USBCX_GRSTCTL(usb->index); u64 done = cvmx_get_cycle() + 100 * - (u64)octeon_get_clock_rate / 1000000; + (u64)octeon_get_clock_rate() / 1000000; union cvmx_usbcx_grstctl c; while (1) { diff --git a/drivers/usb/host/ohci-da8xx.c b/drivers/usb/host/ohci-da8xx.c index d9adae53466b..9fd76c57f91c 100644 --- a/drivers/usb/host/ohci-da8xx.c +++ b/drivers/usb/host/ohci-da8xx.c @@ -464,6 +464,7 @@ static int ohci_da8xx_probe(struct platform_device *pdev) err_remove_hcd: usb_remove_hcd(hcd); + device_wakeup_disable(hcd->self.controller); err: usb_put_hcd(hcd); return error; @@ -474,6 +475,7 @@ static void ohci_da8xx_remove(struct platform_device *pdev) struct usb_hcd *hcd = platform_get_drvdata(pdev); usb_remove_hcd(hcd); + device_wakeup_disable(hcd->self.controller); usb_put_hcd(hcd); } diff --git a/drivers/usb/host/ohci-s3c2410.c b/drivers/usb/host/ohci-s3c2410.c index c5c9b4cbcb9a..3273e6b1ce21 100644 --- a/drivers/usb/host/ohci-s3c2410.c +++ b/drivers/usb/host/ohci-s3c2410.c @@ -335,6 +335,7 @@ ohci_hcd_s3c2410_remove(struct platform_device *dev) struct usb_hcd *hcd = platform_get_drvdata(dev); usb_remove_hcd(hcd); + device_wakeup_disable(hcd->self.controller); s3c2410_stop_hc(dev); usb_put_hcd(hcd); } diff --git a/drivers/usb/host/ohci-spear.c b/drivers/usb/host/ohci-spear.c index 993f347c5c28..f9414b2de65b 100644 --- a/drivers/usb/host/ohci-spear.c +++ b/drivers/usb/host/ohci-spear.c @@ -103,8 +103,8 @@ static void spear_ohci_hcd_drv_remove(struct platform_device *pdev) struct spear_ohci *sohci_p = to_spear_ohci(hcd); usb_remove_hcd(hcd); - if (sohci_p->clk) - clk_disable_unprepare(sohci_p->clk); + device_wakeup_disable(hcd->self.controller); + clk_disable_unprepare(sohci_p->clk); usb_put_hcd(hcd); } diff --git a/drivers/usb/host/ohci-st.c b/drivers/usb/host/ohci-st.c index 214342013f7e..2267b1ba784f 100644 --- a/drivers/usb/host/ohci-st.c +++ b/drivers/usb/host/ohci-st.c @@ -235,6 +235,7 @@ static void st_ohci_platform_remove(struct platform_device *dev) int clk; usb_remove_hcd(hcd); + device_wakeup_disable(hcd->self.controller); if (pdata->power_off) pdata->power_off(dev); diff --git a/drivers/usb/misc/iowarrior.c b/drivers/usb/misc/iowarrior.c index 5ff51af16a60..3a215e2ae5b4 100644 --- a/drivers/usb/misc/iowarrior.c +++ b/drivers/usb/misc/iowarrior.c @@ -71,6 +71,7 @@ static struct usb_driver iowarrior_driver; /* Structure to hold all of our device specific stuff */ struct iowarrior { + struct kref kref; struct mutex mutex; /* locks this structure */ struct usb_device *udev; /* save off the usb device pointer */ struct usb_interface *interface; /* the interface for this device */ @@ -243,8 +244,10 @@ static void iowarrior_write_callback(struct urb *urb) /* * iowarrior_delete */ -static inline void iowarrior_delete(struct iowarrior *dev) +static inline void iowarrior_delete(struct kref *kref) { + struct iowarrior *dev = container_of(kref, struct iowarrior, kref); + kfree(dev->int_in_buffer); usb_free_urb(dev->int_in_urb); kfree(dev->read_queue); @@ -641,6 +644,9 @@ static int iowarrior_open(struct inode *inode, struct file *file) } /* increment our usage count for the driver */ ++dev->opened; + + kref_get(&dev->kref); + /* save our object in the file's private structure */ file->private_data = dev; retval = 0; @@ -656,7 +662,6 @@ static int iowarrior_open(struct inode *inode, struct file *file) static int iowarrior_release(struct inode *inode, struct file *file) { struct iowarrior *dev; - int retval = 0; dev = file->private_data; if (!dev) @@ -664,29 +669,18 @@ static int iowarrior_release(struct inode *inode, struct file *file) /* lock our device */ mutex_lock(&dev->mutex); + dev->opened = 0; /* we're closing now */ - if (dev->opened <= 0) { - retval = -ENODEV; /* close called more than once */ - mutex_unlock(&dev->mutex); - } else { - dev->opened = 0; /* we're closing now */ - retval = 0; - if (dev->present) { - /* - The device is still connected so we only shutdown - pending read-/write-ops. - */ - usb_kill_urb(dev->int_in_urb); - wake_up_interruptible(&dev->read_wait); - wake_up_interruptible(&dev->write_wait); - mutex_unlock(&dev->mutex); - } else { - /* The device was unplugged, cleanup resources */ - mutex_unlock(&dev->mutex); - iowarrior_delete(dev); - } + if (dev->present) { + usb_kill_urb(dev->int_in_urb); + wake_up_interruptible(&dev->read_wait); + wake_up_interruptible(&dev->write_wait); } - return retval; + mutex_unlock(&dev->mutex); + + kref_put(&dev->kref, iowarrior_delete); + + return 0; } static __poll_t iowarrior_poll(struct file *file, poll_table * wait) @@ -771,6 +765,7 @@ static int iowarrior_probe(struct usb_interface *interface, if (!dev) return retval; + kref_init(&dev->kref); mutex_init(&dev->mutex); atomic_set(&dev->intr_idx, 0); @@ -889,7 +884,8 @@ static int iowarrior_probe(struct usb_interface *interface, return retval; error: - iowarrior_delete(dev); + kref_put(&dev->kref, iowarrior_delete); + return retval; } @@ -913,19 +909,14 @@ static void iowarrior_disconnect(struct usb_interface *interface) usb_kill_anchored_urbs(&dev->submitted); if (dev->opened) { - /* There is a process that holds a filedescriptor to the device , - so we only shutdown read-/write-ops going on. - Deleting the device is postponed until close() was called. - */ usb_kill_urb(dev->int_in_urb); wake_up_interruptible(&dev->read_wait); wake_up_interruptible(&dev->write_wait); - mutex_unlock(&dev->mutex); - } else { - /* no process is using the device, cleanup now */ - mutex_unlock(&dev->mutex); - iowarrior_delete(dev); } + + mutex_unlock(&dev->mutex); + + kref_put(&dev->kref, iowarrior_delete); } /* usb specific object needed to register this driver with the usb subsystem */ diff --git a/drivers/usb/serial/bus.c b/drivers/usb/serial/bus.c index a1dde939f311..e75f70675ded 100644 --- a/drivers/usb/serial/bus.c +++ b/drivers/usb/serial/bus.c @@ -165,7 +165,11 @@ int usb_serial_bus_register(struct usb_serial_driver *driver) void usb_serial_bus_deregister(struct usb_serial_driver *driver) { - free_dynids(driver); driver_unregister(&driver->driver); + free_dynids(driver); } +void usb_serial_bus_remove_new_id(struct usb_serial_driver *driver) +{ + driver_remove_file(&driver->driver, &driver_attr_new_id); +} diff --git a/drivers/usb/serial/cp210x.c b/drivers/usb/serial/cp210x.c index e3c391397e0f..d124ead13b84 100644 --- a/drivers/usb/serial/cp210x.c +++ b/drivers/usb/serial/cp210x.c @@ -223,6 +223,7 @@ static const struct usb_device_id id_table[] = { { USB_DEVICE(0x19CF, 0x3000) }, /* Parrot NMEA GPS Flight Recorder */ { USB_DEVICE(0x1ADB, 0x0001) }, /* Schweitzer Engineering C662 Cable */ { USB_DEVICE(0x1B1C, 0x1C00) }, /* Corsair USB Dongle */ + { USB_DEVICE(0x1B1C, 0x1C02) }, /* Corsair AX1500i Power Supply */ { USB_DEVICE(0x1B93, 0x1013) }, /* Phoenix Contact UPS Device */ { USB_DEVICE(0x1BA4, 0x0002) }, /* Silicon Labs 358x factory default */ { USB_DEVICE(0x1BE3, 0x07A6) }, /* WAGO 750-923 USB Service Cable */ diff --git a/drivers/usb/serial/generic.c b/drivers/usb/serial/generic.c index 15b6dee3a8e5..64890f6d78e0 100644 --- a/drivers/usb/serial/generic.c +++ b/drivers/usb/serial/generic.c @@ -268,6 +268,7 @@ EXPORT_SYMBOL_GPL(usb_serial_generic_chars_in_buffer); void usb_serial_generic_wait_until_sent(struct tty_struct *tty, long timeout) { struct usb_serial_port *port = tty->driver_data; + struct tty_port *tport = &port->port; unsigned int bps; unsigned long period; unsigned long expire; @@ -287,7 +288,14 @@ void usb_serial_generic_wait_until_sent(struct tty_struct *tty, long timeout) __func__, jiffies_to_msecs(timeout), jiffies_to_msecs(period)); expire = jiffies + timeout; - while (!port->serial->type->tx_empty(port)) { + for (;;) { + mutex_lock(&tport->mutex); + if (tty_io_error(tty) || port->serial->type->tx_empty(port)) { + mutex_unlock(&tport->mutex); + break; + } + mutex_unlock(&tport->mutex); + schedule_timeout_interruptible(period); if (signal_pending(current)) break; diff --git a/drivers/usb/serial/option.c b/drivers/usb/serial/option.c index 6982911a814b..b1f903fd6950 100644 --- a/drivers/usb/serial/option.c +++ b/drivers/usb/serial/option.c @@ -261,6 +261,7 @@ static void option_instat_callback(struct urb *urb); #define QUECTEL_PRODUCT_EM060K_12a 0x012a #define QUECTEL_PRODUCT_EM060K_12b 0x012b #define QUECTEL_PRODUCT_EM060K_12c 0x012c +#define QUECTEL_PRODUCT_RG660QB 0x013d #define QUECTEL_PRODUCT_EG91 0x0191 #define QUECTEL_PRODUCT_EG95 0x0195 #define QUECTEL_PRODUCT_BG96 0x0296 @@ -281,6 +282,7 @@ static void option_instat_callback(struct urb *urb); #define QUECTEL_PRODUCT_EC200U 0x0901 #define QUECTEL_PRODUCT_EG912Y 0x6001 #define QUECTEL_PRODUCT_EC200S_CN 0x6002 +#define QUECTEL_PRODUCT_EG060W 0x6004 #define QUECTEL_PRODUCT_EC200A 0x6005 #define QUECTEL_PRODUCT_EG916Q 0x6007 #define QUECTEL_PRODUCT_EM061K_LWW 0x6008 @@ -1269,12 +1271,15 @@ static const struct usb_device_id option_ids[] = { { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EC200A, 0xff, 0, 0) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EC200U, 0xff, 0, 0) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EC200S_CN, 0xff, 0, 0) }, + { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EG060W, 0xff, 0, 0) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EC200T, 0xff, 0, 0) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EG912Y, 0xff, 0, 0) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_EG916Q, 0xff, 0x00, 0x00) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RM500K, 0xff, 0x00, 0x00) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG650V, 0xff, 0xff, 0x30) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG650V, 0xff, 0, 0) }, + { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG660QB, 0xff, 0xff, 0x30) }, + { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG660QB, 0xff, 0, 0) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG255C, 0xff, 0xff, 0x30) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG255C, 0xff, 0, 0) }, { USB_DEVICE_AND_INTERFACE_INFO(QUECTEL_VENDOR_ID, QUECTEL_PRODUCT_RG255C, 0xff, 0xff, 0x40) }, @@ -2169,6 +2174,8 @@ static const struct usb_device_id option_ids[] = { { USB_DEVICE_INTERFACE_CLASS(0x1e0e, 0x9003, 0xff) }, /* Simcom SIM7500/SIM7600 MBIM mode */ { USB_DEVICE_INTERFACE_CLASS(0x1e0e, 0x9011, 0xff), /* Simcom SIM7500/SIM7600 RNDIS mode */ .driver_info = RSVD(7) }, + { USB_DEVICE_INTERFACE_CLASS(0x1e0e, 0x902b, 0xff), + .driver_info = RSVD(4) }, { USB_DEVICE(0x1e0e, 0x9071), /* Simcom SIM8230 RMNET mode */ .driver_info = RSVD(3) | RSVD(4) }, { USB_DEVICE_INTERFACE_CLASS(0x1e0e, 0x9078, 0xff), /* Simcom SIM8230 ECM mode */ @@ -2430,6 +2437,13 @@ static const struct usb_device_id option_ids[] = { .driver_info = RSVD(5) }, { USB_DEVICE_INTERFACE_CLASS(0x0489, 0xe167, 0xff), /* Foxconn T99W640 MBIM */ .driver_info = RSVD(3) }, + { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x4d22, 0xff, 0x10, 0x01) }, /* Compal EXC-T1 */ + { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x4d22, 0xff, 0x10, 0x02) }, + { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x4d22, 0xff, 0x10, 0x03) }, + { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x4d22, 0xff, 0x10, 0x04) }, + { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x8217, 0xff, 0xff, 0x30) }, /* Compal EXM-G1x */ + { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x8217, 0xff, 0xff, 0x40) }, + { USB_DEVICE_AND_INTERFACE_INFO(0x04b7, 0x8217, 0xff, 0xff, 0x60) }, { USB_DEVICE(0x1508, 0x1001), /* Fibocom NL668 (IOT version) */ .driver_info = RSVD(4) | RSVD(5) | RSVD(6) }, { USB_DEVICE(0x1782, 0x4d10) }, /* Fibocom L610 (AT mode) */ diff --git a/drivers/usb/serial/quatech2.c b/drivers/usb/serial/quatech2.c index 4cde2bd3885f..4ebe739c7472 100644 --- a/drivers/usb/serial/quatech2.c +++ b/drivers/usb/serial/quatech2.c @@ -54,6 +54,7 @@ #define QT2_XMIT_FLUSH 0x05 /* no following info */ #define QT2_CONTROL_ESCAPE 0xff /* pass through previous 2 control bytes */ +#define MIN_BAUD_RATE 50 #define MAX_BAUD_RATE 921600 #define DEFAULT_BAUD_RATE 9600 @@ -158,7 +159,7 @@ static inline int calc_baud_divisor(int baudrate) static inline int qt2_set_port_config(struct usb_device *dev, unsigned char port_number, - u16 baudrate, u16 lcr) + speed_t baudrate, u16 lcr) { int divisor = calc_baud_divisor(baudrate); u16 index = ((u16) (lcr << 8) | (u16) (port_number)); @@ -259,9 +260,9 @@ static void qt2_set_termios(struct tty_struct *tty, struct usb_device *dev = port->serial->dev; struct qt2_port_private *port_priv; struct ktermios *termios = &tty->termios; - u16 baud; unsigned int cflag = termios->c_cflag; u16 new_lcr = 0; + speed_t baud; int status; port_priv = usb_get_serial_port_data(port); @@ -279,6 +280,18 @@ static void qt2_set_termios(struct tty_struct *tty, if (!baud) baud = 9600; + if (baud < MIN_BAUD_RATE || baud > MAX_BAUD_RATE) { + if (old_termios) + baud = tty_termios_baud_rate(old_termios); + else + baud = clamp(baud, MIN_BAUD_RATE, MAX_BAUD_RATE); + + tty_encode_baud_rate(tty, baud, baud); + + if (!baud) + baud = 9600; + } + status = qt2_set_port_config(dev, port_priv->device_port, baud, new_lcr); if (status < 0) @@ -375,7 +388,7 @@ static int qt2_open(struct tty_struct *tty, struct usb_serial_port *port) port_priv->device_port = (u8) device_port; if (tty) - qt2_set_termios(tty, port, &tty->termios); + qt2_set_termios(tty, port, NULL); return 0; diff --git a/drivers/usb/serial/ssu100.c b/drivers/usb/serial/ssu100.c index 1e1888b66305..52b239d16a0e 100644 --- a/drivers/usb/serial/ssu100.c +++ b/drivers/usb/serial/ssu100.c @@ -34,6 +34,7 @@ #define SERIAL_EVEN_PARITY (UART_LCR_PARITY | UART_LCR_EPAR) +#define MIN_BAUD_RATE 50 #define MAX_BAUD_RATE 460800 #define ATC_DISABLED 0x00 @@ -219,9 +220,10 @@ static void ssu100_set_termios(struct tty_struct *tty, { struct usb_device *dev = port->serial->dev; struct ktermios *termios = &tty->termios; - u16 baud, divisor, remainder; + speed_t baud, remainder; unsigned int cflag = termios->c_cflag; u16 urb_value = 0; /* will hold the new flags */ + u16 divisor; int result; if (cflag & PARENB) { @@ -237,6 +239,18 @@ static void ssu100_set_termios(struct tty_struct *tty, if (!baud) baud = 9600; + if (baud < MIN_BAUD_RATE || baud > MAX_BAUD_RATE) { + if (old_termios) + baud = tty_termios_baud_rate(old_termios); + else + baud = clamp(baud, MIN_BAUD_RATE, MAX_BAUD_RATE); + + tty_encode_baud_rate(tty, baud, baud); + + if (!baud) + baud = 9600; + } + dev_dbg(&port->dev, "%s - got baud = %d\n", __func__, baud); @@ -312,7 +326,7 @@ static int ssu100_open(struct tty_struct *tty, struct usb_serial_port *port) dev_dbg(&port->dev, "%s - set uart failed\n", __func__); if (tty) - ssu100_set_termios(tty, port, &tty->termios); + ssu100_set_termios(tty, port, NULL); return usb_serial_generic_open(tty, port); } diff --git a/drivers/usb/serial/usb-serial.c b/drivers/usb/serial/usb-serial.c index bd90af9b3dbc..e2c78e422570 100644 --- a/drivers/usb/serial/usb-serial.c +++ b/drivers/usb/serial/usb-serial.c @@ -403,17 +403,13 @@ static unsigned int serial_chars_in_buffer(struct tty_struct *tty) static void serial_wait_until_sent(struct tty_struct *tty, int timeout) { struct usb_serial_port *port = tty->driver_data; - struct usb_serial *serial = port->serial; dev_dbg(&port->dev, "%s\n", __func__); if (!port->serial->type->wait_until_sent) return; - mutex_lock(&serial->disc_mutex); - if (!serial->disconnected) - port->serial->type->wait_until_sent(tty, timeout); - mutex_unlock(&serial->disc_mutex); + port->serial->type->wait_until_sent(tty, timeout); } static void serial_throttle(struct tty_struct *tty) @@ -441,8 +437,13 @@ static int serial_get_serial(struct tty_struct *tty, struct serial_struct *ss) struct usb_serial_port *port = tty->driver_data; struct tty_port *tport = &port->port; unsigned int close_delay, closing_wait; + int ret = 0; mutex_lock(&tport->mutex); + if (tty_io_error(tty)) { + ret = -EIO; + goto out_unlock; + } close_delay = jiffies_to_msecs(tport->close_delay) / 10; closing_wait = tport->closing_wait; @@ -455,10 +456,10 @@ static int serial_get_serial(struct tty_struct *tty, struct serial_struct *ss) if (port->serial->type->get_serial) port->serial->type->get_serial(tty, ss); - +out_unlock: mutex_unlock(&tport->mutex); - return 0; + return ret; } static int serial_set_serial(struct tty_struct *tty, struct serial_struct *ss) @@ -474,6 +475,10 @@ static int serial_set_serial(struct tty_struct *tty, struct serial_struct *ss) closing_wait = msecs_to_jiffies(closing_wait * 10); mutex_lock(&tport->mutex); + if (tty_io_error(tty)) { + ret = -EIO; + goto out_unlock; + } if (!capable(CAP_SYS_ADMIN)) { if (close_delay != tport->close_delay || @@ -501,6 +506,7 @@ static int serial_ioctl(struct tty_struct *tty, unsigned int cmd, unsigned long arg) { struct usb_serial_port *port = tty->driver_data; + struct tty_port *tport = &port->port; int retval = -ENOIOCTLCMD; dev_dbg(&port->dev, "%s - cmd 0x%04x\n", __func__, cmd); @@ -511,8 +517,21 @@ static int serial_ioctl(struct tty_struct *tty, retval = port->serial->type->tiocmiwait(tty, arg); break; default: - if (port->serial->type->ioctl) + if (!port->serial->type->ioctl) + break; + + if (cmd == TIOCSRS485) + down_write(&tty->termios_rwsem); + + mutex_lock(&tport->mutex); + if (tty_io_error(tty)) + retval = -EIO; + else retval = port->serial->type->ioctl(tty, cmd, arg); + mutex_unlock(&tport->mutex); + + if (cmd == TIOCSRS485) + up_write(&tty->termios_rwsem); } return retval; @@ -522,25 +541,40 @@ static void serial_set_termios(struct tty_struct *tty, const struct ktermios *old) { struct usb_serial_port *port = tty->driver_data; + struct tty_port *tport = &port->port; dev_dbg(&port->dev, "%s\n", __func__); - if (port->serial->type->set_termios) - port->serial->type->set_termios(tty, port, old); - else + if (!port->serial->type->set_termios) { tty_termios_copy_hw(&tty->termios, old); + return; + } + + mutex_lock(&tport->mutex); + if (!tty_io_error(tty)) + port->serial->type->set_termios(tty, port, old); + mutex_unlock(&tport->mutex); } static int serial_break(struct tty_struct *tty, int break_state) { struct usb_serial_port *port = tty->driver_data; + struct tty_port *tport = &port->port; + int ret; dev_dbg(&port->dev, "%s\n", __func__); - if (port->serial->type->break_ctl) - return port->serial->type->break_ctl(tty, break_state); + if (!port->serial->type->break_ctl) + return -ENOTTY; - return -ENOTTY; + mutex_lock(&tport->mutex); + if (tty_io_error(tty)) + ret = -EIO; + else + ret = port->serial->type->break_ctl(tty, break_state); + mutex_unlock(&tport->mutex); + + return ret; } static int serial_proc_show(struct seq_file *m, void *v) @@ -581,24 +615,44 @@ static int serial_proc_show(struct seq_file *m, void *v) static int serial_tiocmget(struct tty_struct *tty) { struct usb_serial_port *port = tty->driver_data; + struct tty_port *tport = &port->port; + int ret; dev_dbg(&port->dev, "%s\n", __func__); - if (port->serial->type->tiocmget) - return port->serial->type->tiocmget(tty); - return -ENOTTY; + if (!port->serial->type->tiocmget) + return -ENOTTY; + + mutex_lock(&tport->mutex); + if (tty_io_error(tty)) + ret = -EIO; + else + ret = port->serial->type->tiocmget(tty); + mutex_unlock(&tport->mutex); + + return ret; } static int serial_tiocmset(struct tty_struct *tty, unsigned int set, unsigned int clear) { struct usb_serial_port *port = tty->driver_data; + struct tty_port *tport = &port->port; + int ret; dev_dbg(&port->dev, "%s\n", __func__); - if (port->serial->type->tiocmset) - return port->serial->type->tiocmset(tty, set, clear); - return -ENOTTY; + if (!port->serial->type->tiocmset) + return -ENOTTY; + + mutex_lock(&tport->mutex); + if (tty_io_error(tty)) + ret = -EIO; + else + ret = port->serial->type->tiocmset(tty, set, clear); + mutex_unlock(&tport->mutex); + + return ret; } static int serial_get_icount(struct tty_struct *tty, @@ -1194,12 +1248,17 @@ static void usb_serial_disconnect(struct usb_interface *interface) usb_serial_port_poison_urbs(port); wake_up_interruptible(&port->port.delta_msr_wait); cancel_work_sync(&port->work); - if (device_is_registered(&port->dev)) - device_del(&port->dev); } + if (serial->type->disconnect) serial->type->disconnect(serial); + for (i = 0; i < serial->num_ports; ++i) { + port = serial->port[i]; + if (device_is_registered(&port->dev)) + device_del(&port->dev); + } + release_sibling(serial, interface); /* let the last holder of this object cause it to be cleaned up */ @@ -1316,7 +1375,8 @@ static int __init usb_serial_init(void) int result; usb_serial_tty_driver = tty_alloc_driver(USB_SERIAL_TTY_MINORS, - TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV); + TTY_DRIVER_REAL_RAW | TTY_DRIVER_DYNAMIC_DEV | + TTY_DRIVER_RESET_SAVED_TERMIOS); if (IS_ERR(usb_serial_tty_driver)) return PTR_ERR(usb_serial_tty_driver); @@ -1466,7 +1526,7 @@ int usb_serial_register_drivers(struct usb_serial_driver *const serial_drivers[] { int rc; struct usb_driver *udriver; - struct usb_serial_driver * const *sd; + struct usb_serial_driver * const *sd, * const *s; /* * udriver must be registered before any of the serial drivers, @@ -1518,9 +1578,11 @@ int usb_serial_register_drivers(struct usb_serial_driver *const serial_drivers[] return 0; err_deregister_drivers: + for (s = serial_drivers; s < sd; ++s) + usb_serial_bus_remove_new_id(*s); + usb_deregister(udriver); while (sd-- > serial_drivers) usb_serial_deregister(*sd); - usb_deregister(udriver); err_free_driver: kfree(udriver); return rc; @@ -1538,10 +1600,23 @@ EXPORT_SYMBOL_GPL(usb_serial_register_drivers); void usb_serial_deregister_drivers(struct usb_serial_driver *const serial_drivers[]) { struct usb_driver *udriver = (*serial_drivers)->usb_driver; + struct usb_serial_driver * const *sd; + + /* + * udriver must be deregistered before the serial drivers so that + * I/O is stopped before unbinding the ports. + * + * Remove the new_id attributes to prevent ids from being added and + * triggering a probe of udriver after it has been deregistered. + */ + for (sd = serial_drivers; *sd; ++sd) + usb_serial_bus_remove_new_id(*sd); + + usb_deregister(udriver); for (; *serial_drivers; ++serial_drivers) usb_serial_deregister(*serial_drivers); - usb_deregister(udriver); + kfree(udriver); } EXPORT_SYMBOL_GPL(usb_serial_deregister_drivers); diff --git a/drivers/usb/serial/xr_serial.c b/drivers/usb/serial/xr_serial.c index 1d9a12628f81..b9338187a588 100644 --- a/drivers/usb/serial/xr_serial.c +++ b/drivers/usb/serial/xr_serial.c @@ -849,12 +849,9 @@ static int xr_get_rs485_config(struct tty_struct *tty, struct usb_serial_port *port = tty->driver_data; struct xr_data *data = usb_get_serial_port_data(port); - down_read(&tty->termios_rwsem); - if (copy_to_user(argp, &data->rs485, sizeof(data->rs485))) { - up_read(&tty->termios_rwsem); + /* core holds port mutex */ + if (copy_to_user(argp, &data->rs485, sizeof(data->rs485))) return -EFAULT; - } - up_read(&tty->termios_rwsem); return 0; } @@ -870,10 +867,9 @@ static int xr_set_rs485_config(struct tty_struct *tty, return -EFAULT; xr_sanitize_serial_rs485(&rs485); - down_write(&tty->termios_rwsem); + /* core holds termios rwsem and port mutex */ data->rs485 = rs485; xr_set_flow_mode(tty, port, NULL); - up_write(&tty->termios_rwsem); if (copy_to_user(argp, &rs485, sizeof(rs485))) return -EFAULT; diff --git a/drivers/usb/storage/realtek_cr.c b/drivers/usb/storage/realtek_cr.c index a026c6cb6e68..de82d8260d72 100644 --- a/drivers/usb/storage/realtek_cr.c +++ b/drivers/usb/storage/realtek_cr.c @@ -915,7 +915,6 @@ static int realtek_cr_autosuspend_setup(struct us_data *us) us->proto_handler = rts51x_invoke_transport; chip->timer_expires = 0; - timer_setup(&chip->rts51x_suspend_timer, rts51x_suspend_timer_fn, 0); fw5895_init(us); /* enable autosuspend function of the usb device */ @@ -933,10 +932,7 @@ static void realtek_cr_destructor(void *extra) return; #ifdef CONFIG_REALTEK_AUTOPM - if (ss_en) { - del_timer(&chip->rts51x_suspend_timer); - chip->timer_expires = 0; - } + timer_shutdown_sync(&chip->rts51x_suspend_timer); #endif kfree(chip->status); } @@ -981,6 +977,9 @@ static int init_realtek_cr(struct us_data *us) us->extra = chip; us->extra_destructor = realtek_cr_destructor; +#ifdef CONFIG_REALTEK_AUTOPM + timer_setup(&chip->rts51x_suspend_timer, rts51x_suspend_timer_fn, 0); +#endif us->max_lun = chip->max_lun = rts51x_get_max_lun(us); chip->us = us; diff --git a/drivers/usb/storage/sierra_ms.c b/drivers/usb/storage/sierra_ms.c index 0774ba22fb66..26b98413b85c 100644 --- a/drivers/usb/storage/sierra_ms.c +++ b/drivers/usb/storage/sierra_ms.c @@ -77,6 +77,13 @@ static int sierra_get_swoc_info(struct usb_device *udev, sizeof(struct swoc_info), /* __u16 size */ USB_CTRL_SET_TIMEOUT); /* int timeout */ + /* + * A short IN transfer leaves the tail of swocInfo uninitialized; + * only a full transfer is valid. + */ + if (result != sizeof(struct swoc_info)) + return -EIO; + swocInfo->LinuxSKU = le16_to_cpu(swocInfo->LinuxSKU); swocInfo->LinuxVer = le16_to_cpu(swocInfo->LinuxVer); return result; diff --git a/drivers/usb/typec/anx7411.c b/drivers/usb/typec/anx7411.c index 978d0cae1c7d..4bfdd33eb2e6 100644 --- a/drivers/usb/typec/anx7411.c +++ b/drivers/usb/typec/anx7411.c @@ -1571,8 +1571,11 @@ static void anx7411_i2c_remove(struct i2c_client *client) anx7411_partner_unregister_altmode(plat); anx7411_unregister_partner(plat); - if (plat->workqueue) + if (plat->workqueue) { + disable_irq(plat->intp_irq); + cancel_work_sync(&plat->work); destroy_workqueue(plat->workqueue); + } if (plat->spi_client) i2c_unregister_device(plat->spi_client); diff --git a/drivers/usb/typec/tcpm/tcpm.c b/drivers/usb/typec/tcpm/tcpm.c index 2eabd89c36cb..cac8f8167aa8 100644 --- a/drivers/usb/typec/tcpm/tcpm.c +++ b/drivers/usb/typec/tcpm/tcpm.c @@ -531,9 +531,12 @@ static const char * const pd_rev[] = { (tcpm_cc_is_source((port)->cc2) && \ !tcpm_cc_is_source((port)->cc1))) -#define tcpm_port_is_debug(port) \ +#define tcpm_port_is_debug_source(port) \ (tcpm_cc_is_source((port)->cc1) && tcpm_cc_is_source((port)->cc2)) +#define tcpm_port_is_debug(port) \ + tcpm_port_is_debug_source(port) + #define tcpm_port_is_audio(port) \ (tcpm_cc_is_audio((port)->cc1) && tcpm_cc_is_audio((port)->cc2)) @@ -1482,6 +1485,9 @@ static void svdm_consume_identity(struct tcpm_port *port, const u32 *p, int cnt) u32 vdo = p[VDO_INDEX_IDH]; u32 product = p[VDO_INDEX_PRODUCT]; + if (cnt <= VDO_INDEX_PRODUCT) + return; + memset(&port->mode_data, 0, sizeof(port->mode_data)); port->partner_ident.id_header = vdo; @@ -1550,17 +1556,16 @@ static void svdm_consume_modes(struct tcpm_port *port, const u32 *p, int cnt) struct typec_altmode_desc *paltmode; int i; - if (pmdata->altmodes >= ARRAY_SIZE(port->partner_altmode)) { - /* Already logged in svdm_consume_svids() */ - return; - } - if (pmdata->svid_index < 0 || pmdata->svid_index >= pmdata->nsvids) { tcpm_log(port, "Invalid SVID index %d", pmdata->svid_index); return; } for (i = 1; i < cnt; i++) { + if (pmdata->altmodes >= ALTMODE_DISCOVERY_MAX) { + /* Already logged in svdm_consume_svids() */ + return; + } paltmode = &pmdata->altmode_desc[pmdata->altmodes]; memset(paltmode, 0, sizeof(*paltmode)); @@ -3974,7 +3979,7 @@ static void run_state_machine(struct tcpm_port *port) tcpm_set_state(port, SNK_UNATTACHED, PD_T_DRP_SNK); break; case SRC_ATTACH_WAIT: - if (tcpm_port_is_debug(port)) + if (tcpm_port_is_debug_source(port)) tcpm_set_state(port, DEBUG_ACC_ATTACHED, PD_T_CC_DEBOUNCE); else if (tcpm_port_is_audio(port)) @@ -4998,7 +5003,7 @@ static void _tcpm_cc_change(struct tcpm_port *port, enum typec_cc_status cc1, switch (port->state) { case TOGGLING: - if (tcpm_port_is_debug(port) || tcpm_port_is_audio(port) || + if (tcpm_port_is_debug_source(port) || tcpm_port_is_audio(port) || tcpm_port_is_source(port)) tcpm_set_state(port, SRC_ATTACH_WAIT, 0); else if (tcpm_port_is_sink(port)) @@ -5009,7 +5014,7 @@ static void _tcpm_cc_change(struct tcpm_port *port, enum typec_cc_status cc1, break; case SRC_UNATTACHED: case ACC_UNATTACHED: - if (tcpm_port_is_debug(port) || tcpm_port_is_audio(port) || + if (tcpm_port_is_debug_source(port) || tcpm_port_is_audio(port) || tcpm_port_is_source(port)) tcpm_set_state(port, SRC_ATTACH_WAIT, 0); break; @@ -5506,22 +5511,38 @@ static void tcpm_pd_event_handler(struct kthread_work *work) port->upcoming_state = FR_SWAP_SEND; ret = tcpm_ams_start(port, FAST_ROLE_SWAP); if (ret == -EAGAIN) - port->upcoming_state = INVALID_STATE; + tcpm_set_state(port, ERROR_RECOVERY, 0); } else { tcpm_log(port, "Discarding FRS_SIGNAL! Not in sink ready"); } } if (events & TCPM_SOURCING_VBUS) { - tcpm_log(port, "sourcing vbus"); /* * In fast role swap case TCPC autonomously sources vbus. Set vbus_source - * true as TCPM wouldn't have called tcpm_set_vbus. + * true conditionally as TCPM wouldn't have called tcpm_set_vbus. + * If TCPM calls tcpm_set_vbus to source vbus, vbus_source would already + * be true. * - * When vbus is sourced on the command on TCPM i.e. TCPM called - * tcpm_set_vbus to source vbus, vbus_source would already be true. + * When TCPM_FRS_EVENT and TCPM_SOURCING_VBUS arrive simultaneously, + * handling TCPM_FRS_EVENT above transitions the state to AMS_START + * with upcoming_state FR_SWAP_SEND. */ - port->vbus_source = true; - _tcpm_pd_vbus_on(port); + + if (tcpm_port_is_source(port) || + tcpm_port_is_debug_source(port) || + (port->state == AMS_START && port->upcoming_state == FR_SWAP_SEND) || + port->state == FR_SWAP_SEND || + port->state == FR_SWAP_SEND_TIMEOUT || + port->state == FR_SWAP_SNK_SRC_TRANSITION_TO_OFF || + port->state == FR_SWAP_SNK_SRC_NEW_SINK_READY || + port->state == FR_SWAP_SNK_SRC_SOURCE_VBUS_APPLIED) { + tcpm_log(port, "sourcing vbus"); + port->vbus_source = true; + _tcpm_pd_vbus_on(port); + } else { + tcpm_log(port, "Discarding sourcing vbus! Invalid state %s", + tcpm_states[port->state]); + } } if (events & TCPM_PORT_CLEAN) { tcpm_log(port, "port clean"); diff --git a/drivers/usb/typec/tipd/core.c b/drivers/usb/typec/tipd/core.c index 3559781d01a3..60fbefbde027 100644 --- a/drivers/usb/typec/tipd/core.c +++ b/drivers/usb/typec/tipd/core.c @@ -226,7 +226,7 @@ static void tps6598x_set_data_role(struct tps6598x *tps, static int tps6598x_connect(struct tps6598x *tps, u32 status) { - struct typec_partner_desc desc; + struct typec_partner_desc desc = { }; enum typec_pwr_opmode mode; int ret; @@ -237,7 +237,6 @@ static int tps6598x_connect(struct tps6598x *tps, u32 status) desc.usb_pd = mode == TYPEC_PWR_MODE_PD; desc.accessory = TYPEC_ACCESSORY_NONE; /* XXX: handle accessories */ - desc.identity = NULL; if (desc.usb_pd) { ret = tps6598x_read_partner_identity(tps); diff --git a/drivers/usb/typec/ucsi/displayport.c b/drivers/usb/typec/ucsi/displayport.c index 5d78f9ba3b02..1d92020b0664 100644 --- a/drivers/usb/typec/ucsi/displayport.c +++ b/drivers/usb/typec/ucsi/displayport.c @@ -71,11 +71,14 @@ static int ucsi_displayport_enter(struct typec_altmode *alt, u32 *vdo) if (ret < 0) { if (ucsi->version > 0x0100) goto err_unlock; - cur = 0xff; } - if (cur < UCSI_MAX_ALTMODES) { - ret = dp->con->port_altmode[cur] == alt ? 0 : -EBUSY; + if (cur != 0xff) { + if (cur < UCSI_MAX_ALTMODES) + ret = dp->con->port_altmode[cur] == alt ? 0 : -EBUSY; + else + ret = -EINVAL; + goto err_unlock; } diff --git a/drivers/usb/typec/ucsi/ucsi.c b/drivers/usb/typec/ucsi/ucsi.c index 46fe16b5a2a0..37dcd5dafdde 100644 --- a/drivers/usb/typec/ucsi/ucsi.c +++ b/drivers/usb/typec/ucsi/ucsi.c @@ -1190,10 +1190,21 @@ static struct fwnode_handle *ucsi_find_fwnode(struct ucsi_connector *con) { struct fwnode_handle *fwnode; int i = 1; + int ret; + u32 port; - device_for_each_child_node(con->ucsi->dev, fwnode) - if (i++ == con->num) - return fwnode; + device_for_each_child_node(con->ucsi->dev, fwnode) { + ret = fwnode_property_read_u32(fwnode, "reg", &port); + if (ret < 0) { + if (i == con->num) + return fwnode; + } else { + if (port == con->num - 1) + return fwnode; + } + + i++; + } return NULL; } diff --git a/drivers/usb/typec/ucsi/ucsi_acpi.c b/drivers/usb/typec/ucsi/ucsi_acpi.c index 1054947497bb..7a4969a086df 100644 --- a/drivers/usb/typec/ucsi/ucsi_acpi.c +++ b/drivers/usb/typec/ucsi/ucsi_acpi.c @@ -107,6 +107,31 @@ static const struct ucsi_operations ucsi_acpi_ops = { .async_write = ucsi_acpi_async_write }; +static int ucsi_acer_read(struct ucsi *ucsi, unsigned int offset, + void *val, size_t val_len) +{ + struct ucsi_acpi *ua = ucsi_get_drvdata(ucsi); + u16 *version = val; + int ret; + + ret = ucsi_acpi_read(ucsi, offset, val, val_len); + if (ret) + return ret; + + if (offset == UCSI_VERSION && val_len == sizeof(*version) && !*version) { + dev_warn(ua->dev, "UCSI version is zero, assuming 1.2\n"); + *version = UCSI_VERSION_1_2; + } + + return 0; +} + +static const struct ucsi_operations ucsi_acer_ops = { + .read = ucsi_acer_read, + .sync_write = ucsi_acpi_sync_write, + .async_write = ucsi_acpi_async_write +}; + static int ucsi_zenbook_read(struct ucsi *ucsi, unsigned int offset, void *val, size_t val_len) { @@ -174,6 +199,13 @@ static const struct ucsi_operations ucsi_dell_ops = { }; static const struct dmi_system_id ucsi_acpi_quirks[] = { + { + .matches = { + DMI_MATCH(DMI_SYS_VENDOR, "Acer"), + DMI_MATCH(DMI_PRODUCT_NAME, "Nitro ANV15-41"), + }, + .driver_data = (void *)&ucsi_acer_ops, + }, { .matches = { DMI_MATCH(DMI_SYS_VENDOR, "ASUSTeK COMPUTER INC."), diff --git a/drivers/vdpa/mlx5/core/mlx5_vdpa.h b/drivers/vdpa/mlx5/core/mlx5_vdpa.h index ca56242972b3..869219529ff1 100644 --- a/drivers/vdpa/mlx5/core/mlx5_vdpa.h +++ b/drivers/vdpa/mlx5/core/mlx5_vdpa.h @@ -11,6 +11,8 @@ #define MLX5V_ETH_HARD_MTU (ETH_HLEN + VLAN_HLEN + ETH_FCS_LEN) +extern int mlx5_vdpa_max_iotlb_entries; + struct mlx5_vdpa_direct_mr { u64 start; u64 end; diff --git a/drivers/vdpa/mlx5/core/resources.c b/drivers/vdpa/mlx5/core/resources.c index d5a59c9035fb..00b66438ab53 100644 --- a/drivers/vdpa/mlx5/core/resources.c +++ b/drivers/vdpa/mlx5/core/resources.c @@ -3,8 +3,14 @@ #include #include +#include #include "mlx5_vdpa.h" +int mlx5_vdpa_max_iotlb_entries = 2048; +module_param_named(max_iotlb_entries, mlx5_vdpa_max_iotlb_entries, int, 0444); +MODULE_PARM_DESC(max_iotlb_entries, + "Maximum number of iotlb entries. (default: 2048)"); + static int alloc_pd(struct mlx5_vdpa_dev *dev, u32 *pdn, u16 uid) { struct mlx5_core_dev *mdev = dev->mdev; @@ -229,7 +235,10 @@ int mlx5_vdpa_destroy_mkey(struct mlx5_vdpa_dev *mvdev, u32 mkey) static int init_ctrl_vq(struct mlx5_vdpa_dev *mvdev) { - mvdev->cvq.iotlb = vhost_iotlb_alloc(0, 0); + if (mlx5_vdpa_max_iotlb_entries < 2) + return -EINVAL; + + mvdev->cvq.iotlb = vhost_iotlb_alloc(mlx5_vdpa_max_iotlb_entries, 0); if (!mvdev->cvq.iotlb) return -ENOMEM; diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c index 9b8b70ffde5a..822c23e73053 100644 --- a/drivers/vdpa/mlx5/net/mlx5_vnet.c +++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c @@ -553,6 +553,8 @@ static int cq_create(struct mlx5_vdpa_net *ndev, u16 idx, u32 num_ent) vcq->mcq.set_ci_db = vcq->db.db; vcq->mcq.arm_db = vcq->db.db + 1; vcq->mcq.cqe_sz = 64; + vcq->mcq.comp = mlx5_vdpa_cq_comp; + vcq->cqe = num_ent; err = cq_frag_buf_alloc(ndev, &vcq->buf, num_ent); if (err) @@ -592,10 +594,6 @@ static int cq_create(struct mlx5_vdpa_net *ndev, u16 idx, u32 num_ent) if (err) goto err_vec; - vcq->mcq.comp = mlx5_vdpa_cq_comp; - vcq->cqe = num_ent; - vcq->mcq.set_ci_db = vcq->db.db; - vcq->mcq.arm_db = vcq->db.db + 1; mlx5_cq_arm(&mvq->cq.mcq, MLX5_CQ_DB_REQ_NOT, uar_page, mvq->cq.mcq.cons_index); kfree(in); return 0; @@ -3180,9 +3178,6 @@ static int mlx5_set_group_asid(struct vdpa_device *vdev, u32 group, { struct mlx5_vdpa_dev *mvdev = to_mvdev(vdev); - if (group >= MLX5_VDPA_NUMVQ_GROUPS) - return -EINVAL; - mvdev->group2asid[group] = asid; return 0; } diff --git a/drivers/vdpa/vdpa.c b/drivers/vdpa/vdpa.c index a7612e0783b3..28a2fb1d0fa4 100644 --- a/drivers/vdpa/vdpa.c +++ b/drivers/vdpa/vdpa.c @@ -67,57 +67,20 @@ static void vdpa_dev_remove(struct device *d) static int vdpa_dev_match(struct device *dev, struct device_driver *drv) { - struct vdpa_device *vdev = dev_to_vdpa(dev); + int ret; /* Check override first, and if set, only use the named driver */ - if (vdev->driver_override) - return strcmp(vdev->driver_override, drv->name) == 0; + ret = device_match_driver_override(dev, drv); + if (ret >= 0) + return ret; /* Currently devices must be supported by all vDPA bus drivers */ return 1; } -static ssize_t driver_override_store(struct device *dev, - struct device_attribute *attr, - const char *buf, size_t count) -{ - struct vdpa_device *vdev = dev_to_vdpa(dev); - int ret; - - ret = driver_set_override(dev, &vdev->driver_override, buf, count); - if (ret) - return ret; - - return count; -} - -static ssize_t driver_override_show(struct device *dev, - struct device_attribute *attr, char *buf) -{ - struct vdpa_device *vdev = dev_to_vdpa(dev); - ssize_t len; - - device_lock(dev); - len = snprintf(buf, PAGE_SIZE, "%s\n", vdev->driver_override); - device_unlock(dev); - - return len; -} -static DEVICE_ATTR_RW(driver_override); - -static struct attribute *vdpa_dev_attrs[] = { - &dev_attr_driver_override.attr, - NULL, -}; - -static const struct attribute_group vdpa_dev_group = { - .attrs = vdpa_dev_attrs, -}; -__ATTRIBUTE_GROUPS(vdpa_dev); - static struct bus_type vdpa_bus = { .name = "vdpa", - .dev_groups = vdpa_dev_groups, + .driver_override = true, .match = vdpa_dev_match, .probe = vdpa_dev_probe, .remove = vdpa_dev_remove, @@ -132,7 +95,6 @@ static void vdpa_release_dev(struct device *d) ops->free(vdev); ida_simple_remove(&vdpa_index_ida, vdev->index); - kfree(vdev->driver_override); kfree(vdev); } diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c index 58cb711e981d..20edde71280e 100644 --- a/drivers/vdpa/vdpa_sim/vdpa_sim.c +++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c @@ -34,7 +34,7 @@ MODULE_PARM_DESC(batch_mapping, "Batched mapping 1 -Enable; 0 - Disable"); static int max_iotlb_entries = 2048; module_param(max_iotlb_entries, int, 0444); MODULE_PARM_DESC(max_iotlb_entries, - "Maximum number of iotlb entries for each address space. 0 means unlimited. (default: 2048)"); + "Maximum number of iotlb entries for each address space. (default: 2048)"); static bool use_va = true; module_param(use_va, bool, 0444); @@ -199,6 +199,8 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr, if (!dev_attr->alloc_size) return ERR_PTR(-EINVAL); + if (max_iotlb_entries < 2) + return ERR_PTR(-EINVAL); if (config->mask & BIT_ULL(VDPA_ATTR_DEV_FEATURES)) { if (config->device_features & @@ -587,12 +589,6 @@ static int vdpasim_set_group_asid(struct vdpa_device *vdpa, unsigned int group, struct vhost_iotlb *iommu; int i; - if (group > vdpasim->dev_attr.ngroups) - return -EINVAL; - - if (asid >= vdpasim->dev_attr.nas) - return -EINVAL; - iommu = &vdpasim->iommu[asid]; mutex_lock(&vdpasim->mutex); diff --git a/drivers/vdpa/vdpa_user/iova_domain.c b/drivers/vdpa/vdpa_user/iova_domain.c index cc8d26b97187..cc230af339e5 100644 --- a/drivers/vdpa/vdpa_user/iova_domain.c +++ b/drivers/vdpa/vdpa_user/iova_domain.c @@ -12,11 +12,17 @@ #include #include #include +#include #include #include #include "iova_domain.h" +static int max_iotlb_entries = 2048; +module_param(max_iotlb_entries, int, 0444); +MODULE_PARM_DESC(max_iotlb_entries, + "Maximum number of iotlb entries. (default: 2048)"); + static int vduse_iotlb_add_range(struct vduse_iova_domain *domain, u64 start, u64 last, u64 addr, unsigned int perm, @@ -608,11 +614,14 @@ vduse_domain_create(unsigned long iova_limit, size_t bounce_size) if (iova_limit <= bounce_size) return NULL; + if (max_iotlb_entries <= 0) + return NULL; + domain = kzalloc(sizeof(*domain), GFP_KERNEL); if (!domain) return NULL; - domain->iotlb = vhost_iotlb_alloc(0, 0); + domain->iotlb = vhost_iotlb_alloc(max_iotlb_entries, 0); if (!domain->iotlb) goto err_iotlb; diff --git a/drivers/vdpa/vdpa_user/vduse_dev.c b/drivers/vdpa/vdpa_user/vduse_dev.c index 85f9bbb563c5..2784d6025e5c 100644 --- a/drivers/vdpa/vdpa_user/vduse_dev.c +++ b/drivers/vdpa/vdpa_user/vduse_dev.c @@ -519,7 +519,7 @@ static void vduse_vq_kick(struct vduse_virtqueue *vq) goto unlock; if (vq->kickfd) - eventfd_signal(vq->kickfd, 1); + eventfd_signal(vq->kickfd); else vq->kicked = true; unlock: @@ -943,7 +943,7 @@ static int vduse_kickfd_setup(struct vduse_dev *dev, eventfd_ctx_put(vq->kickfd); vq->kickfd = ctx; if (vq->ready && vq->kicked && vq->kickfd) { - eventfd_signal(vq->kickfd, 1); + eventfd_signal(vq->kickfd); vq->kicked = false; } spin_unlock(&vq->kick_lock); @@ -992,7 +992,7 @@ static bool vduse_vq_signal_irqfd(struct vduse_virtqueue *vq) spin_lock_irq(&vq->irq_lock); if (vq->ready && vq->cb.trigger) { - eventfd_signal(vq->cb.trigger, 1); + eventfd_signal(vq->cb.trigger); signal = true; } spin_unlock_irq(&vq->irq_lock); @@ -1708,7 +1708,9 @@ static bool vduse_validate_config(struct vduse_dev_config *config) sizeof(config->reserved))) return false; - if (config->vq_align > PAGE_SIZE) + if (config->vq_align < VRING_USED_ALIGN_SIZE || + !is_power_of_2(config->vq_align) || + config->vq_align > PAGE_SIZE) return false; if (config->config_size > PAGE_SIZE) diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc.c b/drivers/vfio/fsl-mc/vfio_fsl_mc.c index f65d91c01f2e..03600872c480 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc.c @@ -430,9 +430,7 @@ static int vfio_fsl_mc_bus_notifier(struct notifier_block *nb, if (action == BUS_NOTIFY_ADD_DEVICE && vdev->mc_dev == mc_cont) { - mc_dev->driver_override = kasprintf(GFP_KERNEL, "%s", - vfio_fsl_mc_ops.name); - if (!mc_dev->driver_override) + if (device_set_driver_override(dev, vfio_fsl_mc_ops.name)) dev_warn(dev, "VFIO_FSL_MC: Setting driver override for device in dprc %s failed\n", dev_name(&mc_cont->dev)); else diff --git a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c index 1a1d0d5ec35c..82b2afa9b7e3 100644 --- a/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c +++ b/drivers/vfio/fsl-mc/vfio_fsl_mc_intr.c @@ -54,7 +54,7 @@ static irqreturn_t vfio_fsl_mc_irq_handler(int irq_num, void *arg) { struct vfio_fsl_mc_irq *mc_irq = (struct vfio_fsl_mc_irq *)arg; - eventfd_signal(mc_irq->trigger, 1); + eventfd_signal(mc_irq->trigger); return IRQ_HANDLED; } @@ -142,13 +142,13 @@ static int vfio_fsl_mc_set_irq_trigger(struct vfio_fsl_mc_device *vdev, if (flags & VFIO_IRQ_SET_DATA_NONE) { if (irq->trigger) - eventfd_signal(irq->trigger, 1); + eventfd_signal(irq->trigger); } else if (flags & VFIO_IRQ_SET_DATA_BOOL) { u8 trigger = *(u8 *)data; if (trigger && irq->trigger) - eventfd_signal(irq->trigger, 1); + eventfd_signal(irq->trigger); } return 0; diff --git a/drivers/vfio/pci/mlx5/main.c b/drivers/vfio/pci/mlx5/main.c index 42ec574a8622..b6c6643ab0f4 100644 --- a/drivers/vfio/pci/mlx5/main.c +++ b/drivers/vfio/pci/mlx5/main.c @@ -71,6 +71,7 @@ int mlx5vf_add_migration_pages(struct mlx5_vhca_data_buffer *buf, unsigned long filled; unsigned int to_fill; int ret; + int i; to_fill = min_t(unsigned int, npages, PAGE_SIZE / sizeof(*page_list)); page_list = kvzalloc(to_fill * sizeof(*page_list), GFP_KERNEL_ACCOUNT); @@ -91,7 +92,7 @@ int mlx5vf_add_migration_pages(struct mlx5_vhca_data_buffer *buf, GFP_KERNEL_ACCOUNT); if (ret) - goto err; + goto err_append; buf->allocated_length += filled * PAGE_SIZE; /* clean input for another bulk allocation */ memset(page_list, 0, filled * sizeof(*page_list)); @@ -102,6 +103,9 @@ int mlx5vf_add_migration_pages(struct mlx5_vhca_data_buffer *buf, kvfree(page_list); return 0; +err_append: + for (i = filled - 1; i >= 0; i--) + __free_page(page_list[i]); err: kvfree(page_list); return ret; diff --git a/drivers/vfio/pci/vfio_pci_core.c b/drivers/vfio/pci/vfio_pci_core.c index db31aeb9df70..0bf56e21c155 100644 --- a/drivers/vfio/pci/vfio_pci_core.c +++ b/drivers/vfio/pci/vfio_pci_core.c @@ -446,7 +446,7 @@ static int vfio_pci_core_runtime_resume(struct device *dev) */ down_write(&vdev->memory_lock); if (vdev->pm_wake_eventfd_ctx) { - eventfd_signal(vdev->pm_wake_eventfd_ctx, 1); + eventfd_signal(vdev->pm_wake_eventfd_ctx); __vfio_pci_runtime_pm_exit(vdev); } up_write(&vdev->memory_lock); @@ -1776,7 +1776,7 @@ void vfio_pci_core_request(struct vfio_device *core_vdev, unsigned int count) pci_notice_ratelimited(pdev, "Relaying device request to user (#%u)\n", count); - eventfd_signal(vdev->req_trigger, 1); + eventfd_signal(vdev->req_trigger); } else if (count == 0) { pci_warn(pdev, "No device request channel registered, blocked until released by user\n"); @@ -1938,8 +1938,8 @@ static int vfio_pci_bus_notifier(struct notifier_block *nb, pdev->is_virtfn && physfn == vdev->pdev) { pci_info(vdev->pdev, "Captured SR-IOV VF %s driver_override\n", pci_name(pdev)); - pdev->driver_override = kasprintf(GFP_KERNEL, "%s", - vdev->vdev.ops->name); + WARN_ON(device_set_driver_override(&pdev->dev, + vdev->vdev.ops->name)); } else if (action == BUS_NOTIFY_BOUND_DRIVER && pdev->is_virtfn && physfn == vdev->pdev) { struct pci_driver *drv = pci_dev_driver(pdev); @@ -2195,7 +2195,7 @@ pci_ers_result_t vfio_pci_core_aer_err_detected(struct pci_dev *pdev, mutex_lock(&vdev->igate); if (vdev->err_trigger) - eventfd_signal(vdev->err_trigger, 1); + eventfd_signal(vdev->err_trigger); mutex_unlock(&vdev->igate); diff --git a/drivers/vfio/pci/vfio_pci_intrs.c b/drivers/vfio/pci/vfio_pci_intrs.c index c4322faca2bd..b05620236f70 100644 --- a/drivers/vfio/pci/vfio_pci_intrs.c +++ b/drivers/vfio/pci/vfio_pci_intrs.c @@ -98,7 +98,7 @@ static void vfio_send_intx_eventfd(void *opaque, void *unused) trigger = READ_ONCE(ctx->trigger); if (likely(trigger)) - eventfd_signal(trigger, 1); + eventfd_signal(trigger); } } @@ -376,7 +376,7 @@ static irqreturn_t vfio_msihandler(int irq, void *arg) { struct eventfd_ctx *trigger = arg; - eventfd_signal(trigger, 1); + eventfd_signal(trigger); return IRQ_HANDLED; } @@ -727,11 +727,11 @@ static int vfio_pci_set_msi_trigger(struct vfio_pci_core_device *vdev, if (!ctx) continue; if (flags & VFIO_IRQ_SET_DATA_NONE) { - eventfd_signal(ctx->trigger, 1); + eventfd_signal(ctx->trigger); } else if (flags & VFIO_IRQ_SET_DATA_BOOL) { uint8_t *bools = data; if (bools[i - start]) - eventfd_signal(ctx->trigger, 1); + eventfd_signal(ctx->trigger); } } return 0; @@ -745,7 +745,7 @@ static int vfio_pci_set_ctx_trigger_single(struct eventfd_ctx **ctx, if (flags & VFIO_IRQ_SET_DATA_NONE) { if (*ctx) { if (count) { - eventfd_signal(*ctx, 1); + eventfd_signal(*ctx); } else { eventfd_ctx_put(*ctx); *ctx = NULL; @@ -760,7 +760,7 @@ static int vfio_pci_set_ctx_trigger_single(struct eventfd_ctx **ctx, trigger = *(uint8_t *)data; if (trigger && *ctx) - eventfd_signal(*ctx, 1); + eventfd_signal(*ctx); return 0; } else if (flags & VFIO_IRQ_SET_DATA_EVENTFD) { diff --git a/drivers/vfio/platform/vfio_platform_irq.c b/drivers/vfio/platform/vfio_platform_irq.c index 31636d1414a0..ef41ecef83af 100644 --- a/drivers/vfio/platform/vfio_platform_irq.c +++ b/drivers/vfio/platform/vfio_platform_irq.c @@ -143,7 +143,7 @@ static int vfio_platform_set_irq_unmask(struct vfio_platform_device *vdev, static void vfio_send_eventfd(struct vfio_platform_irq *irq_ctx) { if (likely(irq_ctx->trigger)) - eventfd_signal(irq_ctx->trigger, 1); + eventfd_signal(irq_ctx->trigger); } static irqreturn_t vfio_automasked_irq_handler(int irq, void *dev_id) diff --git a/drivers/vhost/iotlb.c b/drivers/vhost/iotlb.c index ea61330a3431..a6228dd24931 100644 --- a/drivers/vhost/iotlb.c +++ b/drivers/vhost/iotlb.c @@ -20,6 +20,14 @@ INTERVAL_TREE_DEFINE(struct vhost_iotlb_map, rb, __u64, __subtree_last, START, LAST, static inline, vhost_iotlb_itree); +static void vhost_iotlb_map_unlink(struct vhost_iotlb *iotlb, + struct vhost_iotlb_map *map) +{ + vhost_iotlb_itree_remove(map, &iotlb->root); + list_del(&map->link); + iotlb->nmaps--; +} + /** * vhost_iotlb_map_free - remove a map node and free it * @iotlb: the IOTLB @@ -28,10 +36,8 @@ INTERVAL_TREE_DEFINE(struct vhost_iotlb_map, void vhost_iotlb_map_free(struct vhost_iotlb *iotlb, struct vhost_iotlb_map *map) { - vhost_iotlb_itree_remove(map, &iotlb->root); - list_del(&map->link); + vhost_iotlb_map_unlink(iotlb, map); kfree(map); - iotlb->nmaps--; } EXPORT_SYMBOL_GPL(vhost_iotlb_map_free); @@ -57,14 +63,25 @@ int vhost_iotlb_add_range_ctx(struct vhost_iotlb *iotlb, if (last < start) return -EFAULT; + if (!iotlb->limit) + return -EINVAL; + /* If the range being mapped is [0, ULONG_MAX], split it into two entries * otherwise its size would overflow u64. */ if (start == 0 && last == ULONG_MAX) { u64 mid = last / 2; - int err = vhost_iotlb_add_range_ctx(iotlb, start, mid, addr, - perm, opaque); + int err; + + if (iotlb->limit < 2) + return -ENOSPC; + if (!(iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) && + iotlb->nmaps > iotlb->limit - 2) + return -ENOSPC; + + err = vhost_iotlb_add_range_ctx(iotlb, start, mid, addr, + perm, opaque); if (err) return err; @@ -72,17 +89,19 @@ int vhost_iotlb_add_range_ctx(struct vhost_iotlb *iotlb, start = mid + 1; } - if (iotlb->limit && - iotlb->nmaps == iotlb->limit && - iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) { - map = list_first_entry(&iotlb->list, typeof(*map), link); - vhost_iotlb_map_free(iotlb, map); + if (iotlb->nmaps >= iotlb->limit) { + if (iotlb->flags & VHOST_IOTLB_FLAG_RETIRE) { + map = list_first_entry(&iotlb->list, typeof(*map), link); + vhost_iotlb_map_unlink(iotlb, map); + } else { + return -ENOSPC; + } + } else { + map = kmalloc(sizeof(*map), GFP_ATOMIC); + if (!map) + return -ENOMEM; } - map = kmalloc(sizeof(*map), GFP_ATOMIC); - if (!map) - return -ENOMEM; - map->start = start; map->size = last - start + 1; map->last = last; diff --git a/drivers/vhost/net.c b/drivers/vhost/net.c index 2953ab6e0c1d..bf636bd35177 100644 --- a/drivers/vhost/net.c +++ b/drivers/vhost/net.c @@ -1667,6 +1667,8 @@ static int vhost_net_set_features(struct vhost_net *n, u64 features) if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) { if (vhost_init_device_iotlb(&n->dev)) goto out_unlock; + } else { + vhost_clear_device_iotlb(&n->dev); } for (i = 0; i < VHOST_NET_VQ_MAX; ++i) { diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c index d2442063423c..d408f830364a 100644 --- a/drivers/vhost/vdpa.c +++ b/drivers/vhost/vdpa.c @@ -34,6 +34,11 @@ enum { #define VHOST_VDPA_DEV_MAX (1U << MINORBITS) +static int max_iotlb_entries = 2048; +module_param(max_iotlb_entries, int, 0444); +MODULE_PARM_DESC(max_iotlb_entries, + "Maximum number of iotlb entries. (default: 2048)"); + #define VHOST_VDPA_IOTLB_BUCKETS 16 struct vhost_vdpa_as { @@ -57,6 +62,8 @@ struct vhost_vdpa { int virtio_id; int minor; struct eventfd_ctx *config_ctx; + /* Serialises vhost_vdpa_config_cb() against config_ctx being replaced. */ + spinlock_t config_lock; int in_batch; struct vdpa_iova_range range; u32 batch_asid; @@ -109,12 +116,14 @@ static struct vhost_vdpa_as *vhost_vdpa_alloc_as(struct vhost_vdpa *v, u32 asid) if (asid >= v->vdpa->nas) return NULL; + if (max_iotlb_entries <= 0) + return NULL; as = kmalloc(sizeof(*as), GFP_KERNEL); if (!as) return NULL; - vhost_iotlb_init(&as->iotlb, 0, 0); + vhost_iotlb_init(&as->iotlb, max_iotlb_entries, 0); as->id = asid; hlist_add_head(&as->hash_link, head); @@ -162,7 +171,7 @@ static irqreturn_t vhost_vdpa_virtqueue_cb(void *private) struct eventfd_ctx *call_ctx = vq->call_ctx.ctx; if (call_ctx) - eventfd_signal(call_ctx, 1); + eventfd_signal(call_ctx); return IRQ_HANDLED; } @@ -170,10 +179,12 @@ static irqreturn_t vhost_vdpa_virtqueue_cb(void *private) static irqreturn_t vhost_vdpa_config_cb(void *private) { struct vhost_vdpa *v = private; - struct eventfd_ctx *config_ctx = v->config_ctx; + unsigned long flags; - if (config_ctx) - eventfd_signal(config_ctx, 1); + spin_lock_irqsave(&v->config_lock, flags); + if (v->config_ctx) + eventfd_signal(v->config_ctx); + spin_unlock_irqrestore(&v->config_lock, flags); return IRQ_HANDLED; } @@ -501,15 +512,22 @@ static long vhost_vdpa_get_vring_num(struct vhost_vdpa *v, u16 __user *argp) static void vhost_vdpa_config_put(struct vhost_vdpa *v) { - if (v->config_ctx) { - eventfd_ctx_put(v->config_ctx); - v->config_ctx = NULL; - } + struct eventfd_ctx *ctx; + unsigned long flags; + + spin_lock_irqsave(&v->config_lock, flags); + ctx = v->config_ctx; + v->config_ctx = NULL; + spin_unlock_irqrestore(&v->config_lock, flags); + + if (ctx) + eventfd_ctx_put(ctx); } static long vhost_vdpa_set_config_call(struct vhost_vdpa *v, u32 __user *argp) { struct vdpa_callback cb; + unsigned long flags; int fd; struct eventfd_ctx *ctx; @@ -522,8 +540,14 @@ static long vhost_vdpa_set_config_call(struct vhost_vdpa *v, u32 __user *argp) if (IS_ERR(ctx)) return PTR_ERR(ctx); + spin_lock_irqsave(&v->config_lock, flags); swap(ctx, v->config_ctx); + spin_unlock_irqrestore(&v->config_lock, flags); + /* + * The callback can no longer reach the old context, so this is the + * last reference to it. + */ if (ctx) eventfd_ctx_put(ctx); @@ -663,7 +687,7 @@ static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd, case VHOST_VDPA_SET_GROUP_ASID: if (copy_from_user(&s, argp, sizeof(s))) return -EFAULT; - if (s.num >= vdpa->nas) + if (idx >= vdpa->ngroups || s.num >= vdpa->nas) return -EINVAL; if (!ops->set_group_asid) return -EOPNOTSUPP; @@ -1571,6 +1595,7 @@ static int vhost_vdpa_probe(struct vdpa_device *vdpa) } atomic_set(&v->opened, 0); + spin_lock_init(&v->config_lock); v->minor = minor; v->vdpa = vdpa; v->nvqs = vdpa->nvqs; diff --git a/drivers/vhost/vhost.c b/drivers/vhost/vhost.c index fd8298490a31..7e7f80cb5dab 100644 --- a/drivers/vhost/vhost.c +++ b/drivers/vhost/vhost.c @@ -348,6 +348,17 @@ static void __vhost_vq_meta_reset(struct vhost_virtqueue *vq) vq->meta_iotlb[j] = NULL; } +/* Caller must hold the virtqueue mutex. */ +static void vhost_vq_invalidate_access(struct vhost_virtqueue *vq) +{ + vq->desc = NULL; + vq->avail = NULL; + vq->used = NULL; + vq->log_used = false; + vq->log_addr = -1ull; + __vhost_vq_meta_reset(vq); +} + static void vhost_vq_meta_reset(struct vhost_dev *d) { int i; @@ -977,6 +988,9 @@ EXPORT_SYMBOL_GPL(vhost_dev_set_owner); static struct vhost_iotlb *iotlb_alloc(void) { + if (max_iotlb_entries <= 0) + return NULL; + return vhost_iotlb_alloc(max_iotlb_entries, VHOST_IOTLB_FLAG_RETIRE); } @@ -1358,10 +1372,39 @@ static void vhost_dev_unlock_vqs(struct vhost_dev *d) mutex_unlock(&d->vqs[i]->mutex); } -static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq, - __virtio16 *idx) +static inline int vhost_get_avail_idx(struct vhost_virtqueue *vq) { - return vhost_get_avail(vq, *idx, &vq->avail->idx); + __virtio16 idx; + u16 avail_idx; + int r; + + r = vhost_get_avail(vq, idx, &vq->avail->idx); + if (unlikely(r < 0)) { + vq_err(vq, "Failed to access available index at %p (%d)\n", + &vq->avail->idx, r); + return r; + } + + /* Check it isn't doing very strange thing with available indexes */ + avail_idx = vhost16_to_cpu(vq, idx); + if (unlikely((u16)(avail_idx - vq->last_avail_idx) > vq->num)) { + vq_err(vq, "Invalid available index change from %u to %u", + vq->last_avail_idx, avail_idx); + return -EINVAL; + } + + /* We're done if there is nothing new */ + if (avail_idx == vq->avail_idx) + return 0; + + vq->avail_idx = avail_idx; + + /* + * We updated vq->avail_idx so we need a memory barrier between + * the index read above and the caller reading avail ring entries. + */ + smp_rmb(); + return 1; } static inline int vhost_get_avail_head(struct vhost_virtqueue *vq, @@ -1721,6 +1764,13 @@ int vq_meta_prefetch(struct vhost_virtqueue *vq) { unsigned int num = vq->num; + /* + * vhost_vq_invalidate_access() clears all three addresses together. + * A single zero address may be a valid GIOVA in IOTLB mode. + */ + if (!vq->desc && !vq->avail && !vq->used) + return 0; + if (!vq->iotlb) return 1; @@ -1791,6 +1841,8 @@ static long vhost_set_memory(struct vhost_dev *d, struct vhost_memory __user *m) return -EOPNOTSUPP; if (mem.nregions > max_mem_regions) return -E2BIG; + if (max_iotlb_entries <= 0) + return -EINVAL; newmem = kvzalloc(struct_size(newmem, regions, mem.nregions), GFP_KERNEL); if (!newmem) @@ -2088,11 +2140,48 @@ long vhost_vring_ioctl(struct vhost_dev *d, unsigned int ioctl, void __user *arg } EXPORT_SYMBOL_GPL(vhost_vring_ioctl); +/* Caller must hold the device mutex. */ +void vhost_clear_device_iotlb(struct vhost_dev *d) +{ + struct vhost_iotlb *iotlb; + int i; + + iotlb = d->iotlb; + if (!iotlb) + return; + + vhost_dev_lock_vqs(d); + + /* + * vhost_dev_lock_vqs() takes all VQ mutexes in index order. Drop the + * device-wide view while they are held, then clear each per-VQ view + * and its cached ring access before releasing the locks. Workers + * cannot observe a mixed address-space state during this handoff. + */ + d->iotlb = NULL; + + for (i = 0; i < d->nvqs; ++i) { + struct vhost_virtqueue *vq = d->vqs[i]; + + vq->iotlb = NULL; + vhost_vq_invalidate_access(vq); + } + + vhost_dev_unlock_vqs(d); + vhost_clear_msg(d); + vhost_iotlb_free(iotlb); + wake_up_interruptible_poll(&d->wait, EPOLLIN | EPOLLRDNORM); +} +EXPORT_SYMBOL_GPL(vhost_clear_device_iotlb); + int vhost_init_device_iotlb(struct vhost_dev *d) { struct vhost_iotlb *niotlb, *oiotlb; int i; + if (max_iotlb_entries <= 0) + return -EINVAL; + niotlb = iotlb_alloc(); if (!niotlb) return -ENOMEM; @@ -2105,7 +2194,10 @@ int vhost_init_device_iotlb(struct vhost_dev *d) mutex_lock(&vq->mutex); vq->iotlb = niotlb; - __vhost_vq_meta_reset(vq); + if (oiotlb) + __vhost_vq_meta_reset(vq); + else + vhost_vq_invalidate_access(vq); mutex_unlock(&vq->mutex); } @@ -2324,7 +2416,7 @@ int vhost_log_write(struct vhost_virtqueue *vq, struct vhost_log *log, len -= l; if (!len) { if (vq->log_ctx) - eventfd_signal(vq->log_ctx, 1); + eventfd_signal(vq->log_ctx); return 0; } } @@ -2347,7 +2439,7 @@ static int vhost_update_used_flags(struct vhost_virtqueue *vq) log_used(vq, (used - (void __user *)vq->used), sizeof vq->used->flags); if (vq->log_ctx) - eventfd_signal(vq->log_ctx, 1); + eventfd_signal(vq->log_ctx); } return 0; } @@ -2365,7 +2457,7 @@ static int vhost_update_avail_event(struct vhost_virtqueue *vq) log_used(vq, (used - (void __user *)vq->used), sizeof *vhost_avail_event(vq)); if (vq->log_ctx) - eventfd_signal(vq->log_ctx, 1); + eventfd_signal(vq->log_ctx); } return 0; } @@ -2574,38 +2666,17 @@ int vhost_get_vq_desc(struct vhost_virtqueue *vq, { struct vring_desc desc; unsigned int i, head, found = 0; - u16 last_avail_idx; - __virtio16 avail_idx; + u16 last_avail_idx = vq->last_avail_idx; __virtio16 ring_head; int ret, access; - /* Check it isn't doing very strange things with descriptor numbers. */ - last_avail_idx = vq->last_avail_idx; - if (vq->avail_idx == vq->last_avail_idx) { - if (unlikely(vhost_get_avail_idx(vq, &avail_idx))) { - vq_err(vq, "Failed to access avail idx at %p\n", - &vq->avail->idx); - return -EFAULT; - } - vq->avail_idx = vhost16_to_cpu(vq, avail_idx); - - if (unlikely((u16)(vq->avail_idx - last_avail_idx) > vq->num)) { - vq_err(vq, "Guest moved used index from %u to %u", - last_avail_idx, vq->avail_idx); - return -EFAULT; - } + ret = vhost_get_avail_idx(vq); + if (unlikely(ret < 0)) + return ret; - /* If there's nothing new since last we looked, return - * invalid. - */ - if (vq->avail_idx == last_avail_idx) + if (!ret) return vq->num; - - /* Only get avail ring entries after they have been - * exposed by guest. - */ - smp_rmb(); } /* Grab the next descriptor number they're advertising, and increment @@ -2794,7 +2865,7 @@ int vhost_add_used_n(struct vhost_virtqueue *vq, struct vring_used_elem *heads, log_used(vq, offsetof(struct vring_used, idx), sizeof vq->used->idx); if (vq->log_ctx) - eventfd_signal(vq->log_ctx, 1); + eventfd_signal(vq->log_ctx); } return r; } @@ -2842,7 +2913,7 @@ void vhost_signal(struct vhost_dev *dev, struct vhost_virtqueue *vq) { /* Signal the Guest tell them we used something up. */ if (vq->call_ctx.ctx && vhost_notify(dev, vq)) - eventfd_signal(vq->call_ctx.ctx, 1); + eventfd_signal(vq->call_ctx.ctx); } EXPORT_SYMBOL_GPL(vhost_signal); @@ -2869,35 +2940,21 @@ EXPORT_SYMBOL_GPL(vhost_add_used_and_signal_n); /* return true if we're sure that avaiable ring is empty */ bool vhost_vq_avail_empty(struct vhost_dev *dev, struct vhost_virtqueue *vq) { - __virtio16 avail_idx; int r; if (vq->avail_idx != vq->last_avail_idx) return false; - r = vhost_get_avail_idx(vq, &avail_idx); - if (unlikely(r)) - return false; + r = vhost_get_avail_idx(vq); - vq->avail_idx = vhost16_to_cpu(vq, avail_idx); - if (vq->avail_idx != vq->last_avail_idx) { - /* Since we have updated avail_idx, the following - * call to vhost_get_vq_desc() will read available - * ring entries. Make sure that read happens after - * the avail_idx read. - */ - smp_rmb(); - return false; - } - - return true; + /* Note: we treat error as non-empty here */ + return r == 0; } EXPORT_SYMBOL_GPL(vhost_vq_avail_empty); /* OK, now we need to know about added descriptors. */ bool vhost_enable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) { - __virtio16 avail_idx; int r; if (!(vq->used_flags & VRING_USED_F_NO_NOTIFY)) @@ -2921,25 +2978,13 @@ bool vhost_enable_notify(struct vhost_dev *dev, struct vhost_virtqueue *vq) /* They could have slipped one in as we were doing that: make * sure it's written, then check again. */ smp_mb(); - r = vhost_get_avail_idx(vq, &avail_idx); - if (r) { - vq_err(vq, "Failed to check avail idx at %p: %d\n", - &vq->avail->idx, r); - return false; - } - vq->avail_idx = vhost16_to_cpu(vq, avail_idx); - if (vq->avail_idx != vq->last_avail_idx) { - /* Since we have updated avail_idx, the following - * call to vhost_get_vq_desc() will read available - * ring entries. Make sure that read happens after - * the avail_idx read. - */ - smp_rmb(); - return true; - } + r = vhost_get_avail_idx(vq); + /* Note: we treat error as empty here */ + if (unlikely(r < 0)) + return false; - return false; + return r; } EXPORT_SYMBOL_GPL(vhost_enable_notify); diff --git a/drivers/vhost/vhost.h b/drivers/vhost/vhost.h index 8014d2b35950..9bb7afafbea9 100644 --- a/drivers/vhost/vhost.h +++ b/drivers/vhost/vhost.h @@ -243,6 +243,7 @@ ssize_t vhost_chr_read_iter(struct vhost_dev *dev, struct iov_iter *to, int noblock); ssize_t vhost_chr_write_iter(struct vhost_dev *dev, struct iov_iter *from); +void vhost_clear_device_iotlb(struct vhost_dev *d); int vhost_init_device_iotlb(struct vhost_dev *d); void vhost_iotlb_map_free(struct vhost_iotlb *iotlb, @@ -251,7 +252,7 @@ void vhost_iotlb_map_free(struct vhost_iotlb *iotlb, #define vq_err(vq, fmt, ...) do { \ pr_debug(pr_fmt(fmt), ##__VA_ARGS__); \ if ((vq)->error_ctx) \ - eventfd_signal((vq)->error_ctx, 1);\ + eventfd_signal((vq)->error_ctx);\ } while (0) enum { diff --git a/drivers/vhost/vsock.c b/drivers/vhost/vsock.c index 912e5153b1f8..c3fceb789298 100644 --- a/drivers/vhost/vsock.c +++ b/drivers/vhost/vsock.c @@ -807,6 +807,8 @@ static int vhost_vsock_set_features(struct vhost_vsock *vsock, u64 features) if ((features & (1ULL << VIRTIO_F_ACCESS_PLATFORM))) { if (vhost_init_device_iotlb(&vsock->dev)) goto err; + } else { + vhost_clear_device_iotlb(&vsock->dev); } vsock->seqpacket_allow = features & (1ULL << VIRTIO_VSOCK_F_SEQPACKET); diff --git a/drivers/video/fbdev/core/fb_chrdev.c b/drivers/video/fbdev/core/fb_chrdev.c index eadb81f53a82..fa28da74ae14 100644 --- a/drivers/video/fbdev/core/fb_chrdev.c +++ b/drivers/video/fbdev/core/fb_chrdev.c @@ -85,11 +85,7 @@ static long do_fb_ioctl(struct fb_info *info, unsigned int cmd, var.activate &= ~FB_ACTIVATE_KD_TEXT; console_lock(); lock_fb_info(info); - ret = fbcon_modechange_possible(info, &var); - if (!ret) - ret = fb_set_var(info, &var); - if (!ret) - fbcon_update_vcs(info, var.activate & FB_ACTIVATE_ALL); + ret = fb_set_var_from_user(info, &var); unlock_fb_info(info); console_unlock(); if (!ret && copy_to_user(argp, &var, sizeof(var))) diff --git a/drivers/video/fbdev/core/fbcon.c b/drivers/video/fbdev/core/fbcon.c index 6bb86a415c6c..97642a95faa7 100644 --- a/drivers/video/fbdev/core/fbcon.c +++ b/drivers/video/fbdev/core/fbcon.c @@ -2770,7 +2770,6 @@ void fbcon_update_vcs(struct fb_info *info, bool all) else fbcon_modechanged(info); } -EXPORT_SYMBOL(fbcon_update_vcs); /* let fbcon check if it supports a new screen resolution */ int fbcon_modechange_possible(struct fb_info *info, struct fb_var_screeninfo *var) @@ -2798,7 +2797,6 @@ int fbcon_modechange_possible(struct fb_info *info, struct fb_var_screeninfo *va return 0; } -EXPORT_SYMBOL_GPL(fbcon_modechange_possible); int fbcon_mode_deleted(struct fb_info *info, struct fb_videomode *mode) diff --git a/drivers/video/fbdev/core/fbmem.c b/drivers/video/fbdev/core/fbmem.c index d4e41c6116ff..581bc616e418 100644 --- a/drivers/video/fbdev/core/fbmem.c +++ b/drivers/video/fbdev/core/fbmem.c @@ -881,6 +881,19 @@ fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var) } EXPORT_SYMBOL(fb_set_var); +int fb_set_var_from_user(struct fb_info *info, struct fb_var_screeninfo *var) +{ + int ret = fbcon_modechange_possible(info, var); + + if (!ret) + ret = fb_set_var(info, var); + if (!ret) + fbcon_update_vcs(info, var->activate & FB_ACTIVATE_ALL); + + return ret; +} +EXPORT_SYMBOL(fb_set_var_from_user); + int fb_blank(struct fb_info *info, int blank) { diff --git a/drivers/video/fbdev/core/fbsysfs.c b/drivers/video/fbdev/core/fbsysfs.c index 10c3cfb959f4..572bf45d9d98 100644 --- a/drivers/video/fbdev/core/fbsysfs.c +++ b/drivers/video/fbdev/core/fbsysfs.c @@ -22,9 +22,7 @@ static int activate(struct fb_info *fb_info, struct fb_var_screeninfo *var) var->activate |= FB_ACTIVATE_FORCE; console_lock(); lock_fb_info(fb_info); - err = fb_set_var(fb_info, var); - if (!err) - fbcon_update_vcs(fb_info, var->activate & FB_ACTIVATE_ALL); + err = fb_set_var_from_user(fb_info, var); unlock_fb_info(fb_info); console_unlock(); if (err) diff --git a/drivers/video/fbdev/hyperv_fb.c b/drivers/video/fbdev/hyperv_fb.c index 80e8ec36b7db..e0d68953aa1a 100644 --- a/drivers/video/fbdev/hyperv_fb.c +++ b/drivers/video/fbdev/hyperv_fb.c @@ -283,6 +283,8 @@ static uint screen_depth; static uint screen_fb_size; static uint dio_fb_size; /* FB size for deferred IO */ +static void hvfb_putmem(struct fb_info *info); + /* Send message to Hyper-V host */ static inline int synthvid_send(struct hv_device *hdev, struct synthvid_msg *msg) @@ -887,6 +889,17 @@ static void hvfb_cfb_imageblit(struct fb_info *p, image->width, image->height); } +/* + * fb_ops.fb_destroy is called by the last put_fb_info() call at the end + * of unregister_framebuffer() or fb_release(). Do any cleanup related to + * framebuffer here. + */ +static void hvfb_destroy(struct fb_info *info) +{ + hvfb_putmem(info); + framebuffer_release(info); +} + static const struct fb_ops hvfb_ops = { .owner = THIS_MODULE, .fb_check_var = hvfb_check_var, @@ -897,6 +910,7 @@ static const struct fb_ops hvfb_ops = { .fb_imageblit = hvfb_cfb_imageblit, .fb_blank = hvfb_blank, .fb_mmap = fb_deferred_io_mmap, + .fb_destroy = hvfb_destroy, }; @@ -973,7 +987,7 @@ static phys_addr_t hvfb_get_phymem(struct hv_device *hdev, } /* Release contiguous physical memory */ -static void hvfb_release_phymem(struct hv_device *hdev, +static void hvfb_release_phymem(struct device *device, phys_addr_t paddr, unsigned int size) { unsigned int order = get_order(size); @@ -981,7 +995,7 @@ static void hvfb_release_phymem(struct hv_device *hdev, if (order <= MAX_ORDER) __free_pages(pfn_to_page(paddr >> PAGE_SHIFT), order); else - dma_free_coherent(&hdev->device, + dma_free_coherent(device, round_up(size, PAGE_SIZE), phys_to_virt(paddr), paddr); @@ -1100,7 +1114,7 @@ static int hvfb_getmem(struct hv_device *hdev, struct fb_info *info) } /* Release the framebuffer */ -static void hvfb_putmem(struct hv_device *hdev, struct fb_info *info) +static void hvfb_putmem(struct fb_info *info) { struct hvfb_par *par = info->par; @@ -1109,7 +1123,7 @@ static void hvfb_putmem(struct hv_device *hdev, struct fb_info *info) iounmap(par->mmio_vp); vmbus_free_mmio(par->mem->start, screen_fb_size); } else { - hvfb_release_phymem(hdev, info->fix.smem_start, + hvfb_release_phymem(info->device, info->fix.smem_start, screen_fb_size); } @@ -1223,7 +1237,7 @@ static int hvfb_probe(struct hv_device *hdev, error: fb_deferred_io_cleanup(info); - hvfb_putmem(hdev, info); + hvfb_putmem(info); error2: vmbus_close(hdev->channel); error1: @@ -1246,14 +1260,11 @@ static void hvfb_remove(struct hv_device *hdev) fb_deferred_io_cleanup(info); - unregister_framebuffer(info); cancel_delayed_work_sync(&par->dwork); vmbus_close(hdev->channel); hv_set_drvdata(hdev, NULL); - - hvfb_putmem(hdev, info); - framebuffer_release(info); + unregister_framebuffer(info); } static int hvfb_suspend(struct hv_device *hdev) diff --git a/drivers/video/fbdev/omap2/omapfb/omapfb-main.c b/drivers/video/fbdev/omap2/omapfb/omapfb-main.c index b5acad8eb279..91170476f053 100644 --- a/drivers/video/fbdev/omap2/omapfb/omapfb-main.c +++ b/drivers/video/fbdev/omap2/omapfb/omapfb-main.c @@ -1097,7 +1097,11 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma) rg = omapfb_get_mem_region(ofbi->region); - start = omapfb_get_region_paddr(ofbi); + if (ofbi->rotation_type == OMAP_DSS_ROT_VRFB) + start = rg->vrfb.paddr[0]; + else + start = rg->paddr; + len = fix->smem_len; DBG("user mmap region start %lx, len %d, off %lx\n", start, len, @@ -1107,6 +1111,8 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma) vma->vm_ops = &mmap_user_ops; vma->vm_private_data = rg; + atomic_inc(&rg->map_count); + r = vm_iomap_memory(vma, start, len); if (r) goto error; @@ -1119,7 +1125,8 @@ static int omapfb_mmap(struct fb_info *fbi, struct vm_area_struct *vma) return 0; error: - omapfb_put_mem_region(ofbi->region); + atomic_dec(&rg->map_count); + omapfb_put_mem_region(rg); return r; } diff --git a/drivers/video/fbdev/ps3fb.c b/drivers/video/fbdev/ps3fb.c index 64d291d6b153..a76b1101ef60 100644 --- a/drivers/video/fbdev/ps3fb.c +++ b/drivers/video/fbdev/ps3fb.c @@ -29,7 +29,6 @@ #include #include #include -#include #include #include @@ -828,9 +827,7 @@ static int ps3fb_ioctl(struct fb_info *info, unsigned int cmd, /* Force, in case only special bits changed */ var.activate |= FB_ACTIVATE_FORCE; par->new_mode_id = val; - retval = fb_set_var(info, &var); - if (!retval) - fbcon_update_vcs(info, var.activate & FB_ACTIVATE_ALL); + retval = fb_set_var_from_user(info, &var); console_unlock(); } break; diff --git a/drivers/video/fbdev/sh_mobile_lcdcfb.c b/drivers/video/fbdev/sh_mobile_lcdcfb.c index 2a4794ec1947..e5a070676504 100644 --- a/drivers/video/fbdev/sh_mobile_lcdcfb.c +++ b/drivers/video/fbdev/sh_mobile_lcdcfb.c @@ -15,7 +15,6 @@ #include #include #include -#include #include #include #include @@ -1769,11 +1768,9 @@ static void sh_mobile_fb_reconfig(struct fb_info *info) var.height = ch->display.height; var.activate = FB_ACTIVATE_NOW; - if (fb_set_var(info, &var) < 0) + if (fb_set_var_from_user(info, &var) < 0) /* Couldn't reconfigure, hopefully, can continue as before */ return; - - fbcon_update_vcs(info, true); } /* diff --git a/drivers/virt/acrn/ioeventfd.c b/drivers/virt/acrn/ioeventfd.c index ac4037e9f947..4e845c6ca0b5 100644 --- a/drivers/virt/acrn/ioeventfd.c +++ b/drivers/virt/acrn/ioeventfd.c @@ -223,7 +223,7 @@ static int acrn_ioeventfd_handler(struct acrn_ioreq_client *client, mutex_lock(&client->vm->ioeventfds_lock); p = hsm_ioeventfd_match(client->vm, addr, val, size, req->type); if (p) - eventfd_signal(p->eventfd, 1); + eventfd_signal(p->eventfd); mutex_unlock(&client->vm->ioeventfds_lock); return 0; diff --git a/drivers/virt/acrn/irqfd.c b/drivers/virt/acrn/irqfd.c index d4ad211dce7a..e6280a100572 100644 --- a/drivers/virt/acrn/irqfd.c +++ b/drivers/virt/acrn/irqfd.c @@ -18,6 +18,9 @@ static LIST_HEAD(acrn_irqfd_clients); +/* Cleanup work has been queued; set via test_and_set_bit(). */ +#define HSM_IRQFD_FLAG_SHUTDOWN 0 + /** * struct hsm_irqfd - Properties of HSM irqfd * @vm: Associated VM pointer @@ -27,6 +30,7 @@ static LIST_HEAD(acrn_irqfd_clients); * @list: Entry within &acrn_vm.irqfds of irqfds of a VM * @pt: Structure for select/poll on the associated eventfd * @msi: MSI data + * @flags: Internal lifecycle flags (HSM_IRQFD_FLAG_*) */ struct hsm_irqfd { struct acrn_vm *vm; @@ -36,6 +40,7 @@ struct hsm_irqfd { struct list_head list; poll_table pt; struct acrn_msi_entry msi; + unsigned long flags; }; static void acrn_irqfd_inject(struct hsm_irqfd *irqfd) @@ -46,30 +51,29 @@ static void acrn_irqfd_inject(struct hsm_irqfd *irqfd) irqfd->msi.msi_data); } -static void hsm_irqfd_shutdown(struct hsm_irqfd *irqfd) +/* Queue the cleanup work at most once. Safe from atomic context. */ +static void hsm_irqfd_queue_shutdown(struct hsm_irqfd *irqfd) { - u64 cnt; - - lockdep_assert_held(&irqfd->vm->irqfds_lock); - - /* remove from wait queue */ - list_del_init(&irqfd->list); - eventfd_ctx_remove_wait_queue(irqfd->eventfd, &irqfd->wait, &cnt); - eventfd_ctx_put(irqfd->eventfd); - kfree(irqfd); + if (!test_and_set_bit(HSM_IRQFD_FLAG_SHUTDOWN, &irqfd->flags)) + queue_work(irqfd->vm->irqfd_wq, &irqfd->shutdown); } +/* Sole owner of @irqfd: unhook waitqueue, drop eventfd ref, free. */ static void hsm_irqfd_shutdown_work(struct work_struct *work) { - struct hsm_irqfd *irqfd; - struct acrn_vm *vm; + struct hsm_irqfd *irqfd = container_of(work, struct hsm_irqfd, + shutdown); + struct acrn_vm *vm = irqfd->vm; + u64 cnt; - irqfd = container_of(work, struct hsm_irqfd, shutdown); - vm = irqfd->vm; mutex_lock(&vm->irqfds_lock); if (!list_empty(&irqfd->list)) - hsm_irqfd_shutdown(irqfd); + list_del_init(&irqfd->list); mutex_unlock(&vm->irqfds_lock); + + eventfd_ctx_remove_wait_queue(irqfd->eventfd, &irqfd->wait, &cnt); + eventfd_ctx_put(irqfd->eventfd); + kfree(irqfd); } /* Called with wqh->lock held and interrupts disabled */ @@ -78,17 +82,16 @@ static int hsm_irqfd_wakeup(wait_queue_entry_t *wait, unsigned int mode, { unsigned long poll_bits = (unsigned long)key; struct hsm_irqfd *irqfd; - struct acrn_vm *vm; irqfd = container_of(wait, struct hsm_irqfd, wait); - vm = irqfd->vm; + if (poll_bits & POLLIN) /* An event has been signaled, inject an interrupt */ acrn_irqfd_inject(irqfd); if (poll_bits & POLLHUP) - /* Do shutdown work in thread to hold wqh->lock */ - queue_work(vm->irqfd_wq, &irqfd->shutdown); + /* Defer teardown to the cleanup work; can't sleep here. */ + hsm_irqfd_queue_shutdown(irqfd); return 0; } @@ -145,6 +148,12 @@ static int acrn_irqfd_assign(struct acrn_vm *vm, struct acrn_irqfd *args) init_waitqueue_func_entry(&irqfd->wait, hsm_irqfd_wakeup); init_poll_funcptr(&irqfd->pt, hsm_irqfd_poll_func); + /* + * Hold irqfds_lock across waitqueue install and list_add so the + * irqfd is not visible to deassign/deinit before its waitqueue + * entry is in place, and any racing EPOLLHUP cleanup work blocks + * on irqfds_lock until publication completes. + */ mutex_lock(&vm->irqfds_lock); list_for_each_entry(tmp, &vm->irqfds, list) { if (irqfd->eventfd != tmp->eventfd) @@ -153,14 +162,12 @@ static int acrn_irqfd_assign(struct acrn_vm *vm, struct acrn_irqfd *args) mutex_unlock(&vm->irqfds_lock); goto fail; } - list_add_tail(&irqfd->list, &vm->irqfds); - mutex_unlock(&vm->irqfds_lock); - /* Check the pending event in this stage */ events = vfs_poll(f.file, &irqfd->pt); - + list_add_tail(&irqfd->list, &vm->irqfds); if (events & EPOLLIN) acrn_irqfd_inject(irqfd); + mutex_unlock(&vm->irqfds_lock); fdput(f); return 0; @@ -187,13 +194,17 @@ static int acrn_irqfd_deassign(struct acrn_vm *vm, mutex_lock(&vm->irqfds_lock); list_for_each_entry_safe(irqfd, tmp, &vm->irqfds, list) { if (irqfd->eventfd == eventfd) { - hsm_irqfd_shutdown(irqfd); + list_del_init(&irqfd->list); + hsm_irqfd_queue_shutdown(irqfd); break; } } mutex_unlock(&vm->irqfds_lock); eventfd_ctx_put(eventfd); + /* Wait for cleanup work to finish so the eventfd is fully detached. */ + flush_workqueue(vm->irqfd_wq); + return 0; } @@ -226,9 +237,15 @@ void acrn_irqfd_deinit(struct acrn_vm *vm) struct hsm_irqfd *irqfd, *next; dev_dbg(acrn_dev.this_device, "VM %u irqfd deinit.\n", vm->vmid); - destroy_workqueue(vm->irqfd_wq); + mutex_lock(&vm->irqfds_lock); - list_for_each_entry_safe(irqfd, next, &vm->irqfds, list) - hsm_irqfd_shutdown(irqfd); + list_for_each_entry_safe(irqfd, next, &vm->irqfds, list) { + list_del_init(&irqfd->list); + hsm_irqfd_queue_shutdown(irqfd); + } mutex_unlock(&vm->irqfds_lock); + + /* Drain all cleanup work before tearing the workqueue down. */ + flush_workqueue(vm->irqfd_wq); + destroy_workqueue(vm->irqfd_wq); } diff --git a/drivers/virt/nitro_enclaves/ne_misc_dev.c b/drivers/virt/nitro_enclaves/ne_misc_dev.c index 241b94f62e56..2cd7cd177795 100644 --- a/drivers/virt/nitro_enclaves/ne_misc_dev.c +++ b/drivers/virt/nitro_enclaves/ne_misc_dev.c @@ -1710,6 +1710,7 @@ static int ne_create_vm_ioctl(struct ne_pci_dev *ne_pci_dev, u64 __user *slot_ui return enclave_fd; put_file: + enclave_file->private_data = NULL; fput(enclave_file); put_fd: put_unused_fd(enclave_fd); diff --git a/drivers/virtio/virtio.c b/drivers/virtio/virtio.c index 71dee622b771..d09ab0ee74d6 100644 --- a/drivers/virtio/virtio.c +++ b/drivers/virtio/virtio.c @@ -340,13 +340,61 @@ static void virtio_dev_remove(struct device *_d) of_node_put(dev->dev.of_node); } -static struct bus_type virtio_bus = { +/* + * virtio_irq_get_affinity - get IRQ affinity mask for device + * @_d: ptr to dev structure + * @irq_vec: interrupt vector number + * + * Return the CPU affinity mask for @_d and @irq_vec. + */ +static const struct cpumask *virtio_irq_get_affinity(struct device *_d, + unsigned int irq_vec) +{ + struct virtio_device *dev = dev_to_virtio(_d); + + if (!dev->config->get_vq_affinity) + return NULL; + + return dev->config->get_vq_affinity(dev, irq_vec); +} + +static void virtio_dev_shutdown(struct device *_d) +{ + struct virtio_device *dev = dev_to_virtio(_d); + struct virtio_driver *drv = drv_to_virtio(dev->dev.driver); + + /* + * Stop accesses to or from the device. + * We only need to do it if there's a driver - no accesses otherwise. + */ + if (!drv) + return; + + /* + * Some devices get wedged if you kick them after they are + * reset. Mark all vqs as broken to make sure we don't. + */ + virtio_break_device(dev); + /* + * Guarantee that any callback will see vq->broken as true. + */ + virtio_synchronize_cbs(dev); + /* + * As IOMMUs are reset on shutdown, this will block device access to memory. + * Some devices get wedged if this happens, so reset to make sure it does not. + */ + dev->config->reset(dev); +} + +static const struct bus_type virtio_bus = { .name = "virtio", .match = virtio_dev_match, .dev_groups = virtio_dev_groups, .uevent = virtio_uevent, .probe = virtio_dev_probe, .remove = virtio_dev_remove, + .irq_get_affinity = virtio_irq_get_affinity, + .shutdown = virtio_dev_shutdown, }; int register_virtio_driver(struct virtio_driver *driver) diff --git a/drivers/virtio/virtio_mmio.c b/drivers/virtio/virtio_mmio.c index 049c4f4ee56f..649e9d5aa132 100644 --- a/drivers/virtio/virtio_mmio.c +++ b/drivers/virtio/virtio_mmio.c @@ -65,7 +65,6 @@ #include #include #include -#include #include #include #include @@ -89,21 +88,10 @@ struct virtio_mmio_device { void __iomem *base; unsigned long version; - /* a list of queues so we can dispatch IRQs */ - spinlock_t lock; - struct list_head virtqueues; + /* True if enable_irq_wake() succeeded for the shared IRQ. */ + bool wake_irq_enabled; }; -struct virtio_mmio_vq_info { - /* the actual virtqueue */ - struct virtqueue *vq; - - /* the list node for the virtqueues list */ - struct list_head node; -}; - - - /* Configuration interface */ static u64 vm_get_features(struct virtio_device *vdev) @@ -300,9 +288,8 @@ static bool vm_notify_with_data(struct virtqueue *vq) static irqreturn_t vm_interrupt(int irq, void *opaque) { struct virtio_mmio_device *vm_dev = opaque; - struct virtio_mmio_vq_info *info; + struct virtqueue *vq; unsigned long status; - unsigned long flags; irqreturn_t ret = IRQ_NONE; /* Read and acknowledge interrupts */ @@ -315,10 +302,8 @@ static irqreturn_t vm_interrupt(int irq, void *opaque) } if (likely(status & VIRTIO_MMIO_INT_VRING)) { - spin_lock_irqsave(&vm_dev->lock, flags); - list_for_each_entry(info, &vm_dev->virtqueues, node) - ret |= vring_interrupt(irq, info->vq); - spin_unlock_irqrestore(&vm_dev->lock, flags); + virtio_device_for_each_vq(&vm_dev->vdev, vq) + ret |= vring_interrupt(irq, vq); } return ret; @@ -329,14 +314,8 @@ static irqreturn_t vm_interrupt(int irq, void *opaque) static void vm_del_vq(struct virtqueue *vq) { struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vq->vdev); - struct virtio_mmio_vq_info *info = vq->priv; - unsigned long flags; unsigned int index = vq->index; - spin_lock_irqsave(&vm_dev->lock, flags); - list_del(&info->node); - spin_unlock_irqrestore(&vm_dev->lock, flags); - /* Select and deactivate the queue */ writel(index, vm_dev->base + VIRTIO_MMIO_QUEUE_SEL); if (vm_dev->version == 1) { @@ -347,19 +326,23 @@ static void vm_del_vq(struct virtqueue *vq) } vring_del_virtqueue(vq); - - kfree(info); } static void vm_del_vqs(struct virtio_device *vdev) { struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev); struct virtqueue *vq, *n; + int irq = platform_get_irq(vm_dev->pdev, 0); list_for_each_entry_safe(vq, n, &vdev->vqs, list) vm_del_vq(vq); - free_irq(platform_get_irq(vm_dev->pdev, 0), vm_dev); + if (vm_dev->wake_irq_enabled) { + disable_irq_wake(irq); + vm_dev->wake_irq_enabled = false; + } + + free_irq(irq, vm_dev); } static void vm_synchronize_cbs(struct virtio_device *vdev) @@ -375,9 +358,7 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in { struct virtio_mmio_device *vm_dev = to_virtio_mmio_device(vdev); bool (*notify)(struct virtqueue *vq); - struct virtio_mmio_vq_info *info; struct virtqueue *vq; - unsigned long flags; unsigned int num; int err; @@ -399,13 +380,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in goto error_available; } - /* Allocate and fill out our active queue description */ - info = kmalloc(sizeof(*info), GFP_KERNEL); - if (!info) { - err = -ENOMEM; - goto error_kmalloc; - } - num = readl(vm_dev->base + VIRTIO_MMIO_QUEUE_NUM_MAX); if (num == 0) { err = -ENOENT; @@ -463,13 +437,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in writel(1, vm_dev->base + VIRTIO_MMIO_QUEUE_READY); } - vq->priv = info; - info->vq = vq; - - spin_lock_irqsave(&vm_dev->lock, flags); - list_add(&info->node, &vm_dev->virtqueues); - spin_unlock_irqrestore(&vm_dev->lock, flags); - return vq; error_bad_pfn: @@ -481,8 +448,6 @@ static struct virtqueue *vm_setup_vq(struct virtio_device *vdev, unsigned int in writel(0, vm_dev->base + VIRTIO_MMIO_QUEUE_READY); WARN_ON(readl(vm_dev->base + VIRTIO_MMIO_QUEUE_READY)); } - kfree(info); -error_kmalloc: error_available: return ERR_PTR(err); } @@ -506,8 +471,9 @@ static int vm_find_vqs(struct virtio_device *vdev, unsigned int nvqs, if (err) return err; - if (of_property_read_bool(vm_dev->pdev->dev.of_node, "wakeup-source")) - enable_irq_wake(irq); + if (of_property_read_bool(vm_dev->pdev->dev.of_node, "wakeup-source") && + !enable_irq_wake(irq)) + vm_dev->wake_irq_enabled = true; for (i = 0; i < nvqs; ++i) { if (!names[i]) { @@ -627,8 +593,6 @@ static int virtio_mmio_probe(struct platform_device *pdev) vm_dev->vdev.dev.release = virtio_mmio_release_dev; vm_dev->vdev.config = &virtio_mmio_config_ops; vm_dev->pdev = pdev; - INIT_LIST_HEAD(&vm_dev->virtqueues); - spin_lock_init(&vm_dev->lock); vm_dev->base = devm_platform_ioremap_resource(pdev, 0); if (IS_ERR(vm_dev->base)) { diff --git a/drivers/virtio/virtio_ring.c b/drivers/virtio/virtio_ring.c index db4582687b95..a7abe94b5dd8 100644 --- a/drivers/virtio/virtio_ring.c +++ b/drivers/virtio/virtio_ring.c @@ -1412,7 +1412,7 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, struct scatterlist *sg; unsigned int i, n, c, descs_used, err_idx; __le16 head_flags, flags; - u16 head, id, prev, curr, avail_used_flags; + u16 head, id, prev, curr, avail_used_flags, unpub_flags; int err; START_USE(vq); @@ -1535,15 +1535,30 @@ static inline int virtqueue_add_packed(struct virtqueue *_vq, curr = vq->free_head; vq->packed.avail_used_flags = avail_used_flags; + unpub_flags = avail_used_flags ^ (1 << VRING_PACKED_DESC_F_AVAIL | + 1 << VRING_PACKED_DESC_F_USED); for (n = 0; n < total_sg; n++) { if (i == err_idx) break; + /* + * The mapping loop made every descriptor but the head + * available. Stamp the previous wrap counter's AVAIL and USED + * bits on those, so that a later and shorter chain at this head + * does not leave one of them available beyond its own last + * descriptor. Marking them used instead would hand + * is_used_desc_packed() a completion we never made. + */ + if (i != head) + desc[i].flags = cpu_to_le16(unpub_flags); vring_unmap_extra_packed(vq, &vq->packed.desc_extra[curr]); curr = vq->packed.desc_extra[curr].next; i++; - if (i >= vq->packed.vring.num) + if (i >= vq->packed.vring.num) { i = 0; + unpub_flags ^= 1 << VRING_PACKED_DESC_F_AVAIL | + 1 << VRING_PACKED_DESC_F_USED; + } } END_USE(vq); diff --git a/drivers/watchdog/at91sam9_wdt.c b/drivers/watchdog/at91sam9_wdt.c index fed7be246442..4e5171f4d006 100644 --- a/drivers/watchdog/at91sam9_wdt.c +++ b/drivers/watchdog/at91sam9_wdt.c @@ -242,7 +242,7 @@ static int at91_wdt_init(struct platform_device *pdev, struct at91wdt *wdt) return 0; out_stop_timer: - del_timer(&wdt->timer); + timer_shutdown_sync(&wdt->timer); return err; } @@ -387,7 +387,7 @@ static int __exit at91wdt_remove(struct platform_device *pdev) watchdog_unregister_device(&wdt->wdd); pr_warn("I quit now, hardware will probably reboot!\n"); - del_timer(&wdt->timer); + timer_shutdown_sync(&wdt->timer); clk_disable_unprepare(wdt->sclk); return 0; diff --git a/drivers/xen/balloon.c b/drivers/xen/balloon.c index a76d5530533f..af34de2a8a23 100644 --- a/drivers/xen/balloon.c +++ b/drivers/xen/balloon.c @@ -671,7 +671,7 @@ void xen_free_ballooned_pages(unsigned int nr_pages, struct page **pages) } EXPORT_SYMBOL(xen_free_ballooned_pages); -static int __init balloon_add_regions(void) +static int __init balloon_add_regions(bool append) { unsigned long start_pfn, pages; unsigned long pfn, extra_pfn_end; @@ -695,19 +695,26 @@ static int __init balloon_add_regions(void) balloon_append(pfn_to_page(pfn)); /* - * Extra regions are accounted for in the physmap, but need - * decreasing from current_pages and target_pages to balloon - * down the initial allocation, because they are already - * accounted for in total_pages. + * There are two different use-cases depending on how the + * initial memory target is fetched. For PVH dom0 and PV the + * target is usually set to reflect the domain assigned memory, + * and hence extra regions need adding. + * + * OTOH for HVM and PVH domU the target is set to the amount of + * RAM reported in the memory map, and hence extra regions need + * subtracting to reflect the real memory usage. */ pages = extra_pfn_end - start_pfn; - if (pages >= balloon_stats.current_pages || - pages >= balloon_stats.target_pages) { + if (append) { + balloon_stats.total_pages += pages; + } else if (pages >= balloon_stats.current_pages || + pages >= balloon_stats.target_pages) { WARN(1, "Extra pages underflow current target"); return -ERANGE; + } else { + balloon_stats.current_pages -= pages; + balloon_stats.target_pages -= pages; } - balloon_stats.current_pages -= pages; - balloon_stats.target_pages -= pages; } return 0; @@ -716,7 +723,9 @@ static int __init balloon_add_regions(void) static int __init balloon_init(void) { struct task_struct *task; - unsigned long current_pages; + long current_pages = 0; + domid_t domid = DOMID_SELF; + bool append = true; int rc; if (!xen_domain()) @@ -724,15 +733,22 @@ static int __init balloon_init(void) pr_info("Initialising balloon driver\n"); - if (xen_pv_domain()) { - if (xen_released_pages >= xen_start_info->nr_pages) - goto underflow; - current_pages = min(xen_start_info->nr_pages - - xen_released_pages, max_pfn); - } else { - if (xen_unpopulated_pages >= get_num_physpages()) - goto underflow; - current_pages = get_num_physpages() - xen_unpopulated_pages; + if (xen_initial_domain()) + current_pages = HYPERVISOR_memory_op(XENMEM_current_reservation, + &domid); + if (current_pages <= 0) { + if (xen_pv_domain()) { + if (xen_released_pages >= xen_start_info->nr_pages) + goto underflow; + current_pages = min(xen_start_info->nr_pages - + xen_released_pages, max_pfn); + } else { + if (xen_unpopulated_pages >= get_num_physpages()) + goto underflow; + append = false; + current_pages = get_num_physpages() - + xen_unpopulated_pages; + } } balloon_stats.current_pages = current_pages; @@ -752,7 +768,7 @@ static int __init balloon_init(void) register_sysctl_init("xen/balloon", balloon_table); #endif - rc = balloon_add_regions(); + rc = balloon_add_regions(append); if (rc) return rc; diff --git a/drivers/xen/gntdev-dmabuf.c b/drivers/xen/gntdev-dmabuf.c index 42adc2c1e06b..d7b56466bec5 100644 --- a/drivers/xen/gntdev-dmabuf.c +++ b/drivers/xen/gntdev-dmabuf.c @@ -357,8 +357,12 @@ struct gntdev_dmabuf_export_args { static int dmabuf_exp_from_pages(struct gntdev_dmabuf_export_args *args) { DEFINE_DMA_BUF_EXPORT_INFO(exp_info); - struct gntdev_dmabuf *gntdev_dmabuf; - int ret; + struct gntdev_dmabuf *gntdev_dmabuf __free(kfree) = NULL; + CLASS(get_unused_fd, ret)(O_CLOEXEC); + int fd; + + if (ret < 0) + return ret; gntdev_dmabuf = kzalloc(sizeof(*gntdev_dmabuf), GFP_KERNEL); if (!gntdev_dmabuf) @@ -383,32 +387,23 @@ static int dmabuf_exp_from_pages(struct gntdev_dmabuf_export_args *args) exp_info.priv = gntdev_dmabuf; gntdev_dmabuf->dmabuf = dma_buf_export(&exp_info); - if (IS_ERR(gntdev_dmabuf->dmabuf)) { - ret = PTR_ERR(gntdev_dmabuf->dmabuf); - gntdev_dmabuf->dmabuf = NULL; - goto fail; - } - - ret = dma_buf_fd(gntdev_dmabuf->dmabuf, O_CLOEXEC); - if (ret < 0) - goto fail; + if (IS_ERR(gntdev_dmabuf->dmabuf)) + return PTR_ERR(gntdev_dmabuf->dmabuf); gntdev_dmabuf->fd = ret; args->fd = ret; pr_debug("Exporting DMA buffer with fd %d\n", ret); + get_file(gntdev_dmabuf->priv->filp); mutex_lock(&args->dmabuf_priv->lock); list_add(&gntdev_dmabuf->next, &args->dmabuf_priv->exp_list); mutex_unlock(&args->dmabuf_priv->lock); - get_file(gntdev_dmabuf->priv->filp); - return 0; -fail: - if (gntdev_dmabuf->dmabuf) - dma_buf_put(gntdev_dmabuf->dmabuf); - kfree(gntdev_dmabuf); - return ret; + fd = ret; + ret = -EBADF; + fd_install(fd, no_free_ptr(gntdev_dmabuf)->dmabuf->file); + return 0; } static struct gntdev_grant_map * diff --git a/drivers/xen/pvcalls-front.c b/drivers/xen/pvcalls-front.c index b72ee9379d77..d2bdd43703e8 100644 --- a/drivers/xen/pvcalls-front.c +++ b/drivers/xen/pvcalls-front.c @@ -32,6 +32,7 @@ struct pvcalls_bedata { struct xen_pvcalls_front_ring ring; grant_ref_t ref; int irq; + bool disabled; struct list_head socket_mappings; spinlock_t socket_lock; @@ -131,6 +132,20 @@ static inline int get_request(struct pvcalls_bedata *bedata, int *req_id) return 0; } +/* + * Wait for the backend's response to req_id, or for the frontend to be + * disabled because the backend violated the wire protocol. Returns 0 once + * the response has arrived, or -EIO if the frontend was disabled. + */ +static int pvcalls_front_wait_rsp(struct pvcalls_bedata *bedata, u32 req_id) +{ + wait_event(bedata->inflight_req, + READ_ONCE(bedata->rsp[req_id].req_id) == req_id || + READ_ONCE(bedata->disabled)); + + return READ_ONCE(bedata->disabled) ? -EIO : 0; +} + static bool pvcalls_front_write_todo(struct sock_mapping *map) { struct pvcalls_data_intf *intf = map->active.ring; @@ -168,7 +183,8 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id) struct pvcalls_bedata *bedata; struct xen_pvcalls_response *rsp; uint8_t *src, *dst; - int req_id = 0, more = 0, done = 0; + u32 req_id = 0; + int more = 0, done = 0; if (dev == NULL) return IRQ_HANDLED; @@ -179,12 +195,31 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id) pvcalls_exit(); return IRQ_HANDLED; } + if (READ_ONCE(bedata->disabled)) { + pvcalls_exit(); + return IRQ_HANDLED; + } again: while (RING_HAS_UNCONSUMED_RESPONSES(&bedata->ring)) { rsp = RING_GET_RESPONSE(&bedata->ring, bedata->ring.rsp_cons); req_id = rsp->req_id; + if (req_id >= PVCALLS_NR_RSP_PER_RING) { + /* + * The backend supplied a req_id that would index + * bedata->rsp[] out of bounds: a protocol violation + * from a malicious or buggy backend. Log once, stop + * trusting this backend and disable the frontend rather + * than silently dropping the response and continuing. + */ + pr_err_once("pvcalls: backend sent out-of-range req_id %u, disabling frontend\n", + req_id); + WRITE_ONCE(bedata->disabled, true); + bedata->ring.rsp_cons++; + done = 1; + break; + } if (rsp->cmd == PVCALLS_POLL) { struct sock_mapping *map = (struct sock_mapping *)(uintptr_t) rsp->u.poll.id; @@ -217,7 +252,7 @@ static irqreturn_t pvcalls_front_event_handler(int irq, void *dev_id) } RING_FINAL_CHECK_FOR_RESPONSES(&bedata->ring, more); - if (more) + if (more && !READ_ONCE(bedata->disabled)) goto again; if (done) wake_up(&bedata->inflight_req); @@ -330,8 +365,11 @@ int pvcalls_front_socket(struct socket *sock) if (notify) notify_remote_via_irq(bedata->irq); - wait_event(bedata->inflight_req, - READ_ONCE(bedata->rsp[req_id].req_id) == req_id); + ret = pvcalls_front_wait_rsp(bedata, req_id); + if (ret) { + pvcalls_exit(); + return ret; + } /* read req_id, then the content */ smp_rmb(); @@ -476,8 +514,11 @@ int pvcalls_front_connect(struct socket *sock, struct sockaddr *addr, if (notify) notify_remote_via_irq(bedata->irq); - wait_event(bedata->inflight_req, - READ_ONCE(bedata->rsp[req_id].req_id) == req_id); + ret = pvcalls_front_wait_rsp(bedata, req_id); + if (ret) { + pvcalls_exit_sock(sock); + return ret; + } /* read req_id, then the content */ smp_rmb(); @@ -707,8 +748,11 @@ int pvcalls_front_bind(struct socket *sock, struct sockaddr *addr, int addr_len) if (notify) notify_remote_via_irq(bedata->irq); - wait_event(bedata->inflight_req, - READ_ONCE(bedata->rsp[req_id].req_id) == req_id); + ret = pvcalls_front_wait_rsp(bedata, req_id); + if (ret) { + pvcalls_exit_sock(sock); + return ret; + } /* read req_id, then the content */ smp_rmb(); @@ -756,8 +800,11 @@ int pvcalls_front_listen(struct socket *sock, int backlog) if (notify) notify_remote_via_irq(bedata->irq); - wait_event(bedata->inflight_req, - READ_ONCE(bedata->rsp[req_id].req_id) == req_id); + ret = pvcalls_front_wait_rsp(bedata, req_id); + if (ret) { + pvcalls_exit_sock(sock); + return ret; + } /* read req_id, then the content */ smp_rmb(); @@ -813,6 +860,14 @@ int pvcalls_front_accept(struct socket *sock, struct socket *newsock, int flags) } } + if (READ_ONCE(bedata->disabled)) { + clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT, + (void *)&map->passive.flags); + wake_up(&map->passive.inflight_accept_req); + pvcalls_exit_sock(sock); + return -EIO; + } + map2 = kzalloc(sizeof(*map2), GFP_KERNEL); if (map2 == NULL) { clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT, @@ -873,10 +928,18 @@ int pvcalls_front_accept(struct socket *sock, struct socket *newsock, int flags) } if (wait_event_interruptible(bedata->inflight_req, - READ_ONCE(bedata->rsp[req_id].req_id) == req_id)) { + READ_ONCE(bedata->rsp[req_id].req_id) == req_id || + READ_ONCE(bedata->disabled))) { pvcalls_exit_sock(sock); return -EINTR; } + if (READ_ONCE(bedata->disabled)) { + clear_bit(PVCALLS_FLAG_ACCEPT_INFLIGHT, + (void *)&map->passive.flags); + wake_up(&map->passive.inflight_accept_req); + pvcalls_exit_sock(sock); + return -EIO; + } /* read req_id, then the content */ smp_rmb(); @@ -1045,7 +1108,8 @@ int pvcalls_front_release(struct socket *sock) notify_remote_via_irq(bedata->irq); wait_event(bedata->inflight_req, - READ_ONCE(bedata->rsp[req_id].req_id) == req_id); + READ_ONCE(bedata->rsp[req_id].req_id) == req_id || + READ_ONCE(bedata->disabled)); if (map->active_socket) { /* diff --git a/drivers/xen/xen-pciback/pci_stub.c b/drivers/xen/xen-pciback/pci_stub.c index e34b623e4b41..42e630239b9b 100644 --- a/drivers/xen/xen-pciback/pci_stub.c +++ b/drivers/xen/xen-pciback/pci_stub.c @@ -555,6 +555,8 @@ static int pcistub_seize(struct pci_dev *dev, return err; } +static struct pci_driver xen_pcibk_pci_driver; + /* Called when 'bind'. This means we must _NOT_ call pci_reset_function or * other functions that take the sysfs lock. */ static int pcistub_probe(struct pci_dev *dev, const struct pci_device_id *id) @@ -566,8 +568,8 @@ static int pcistub_probe(struct pci_dev *dev, const struct pci_device_id *id) match = pcistub_match(dev); - if ((dev->driver_override && - !strcmp(dev->driver_override, PCISTUB_DRIVER_NAME)) || + if (device_match_driver_override(&dev->dev, + &xen_pcibk_pci_driver.driver) > 0 || match) { if (dev->hdr_type != PCI_HEADER_TYPE_NORMAL diff --git a/fs/9p/vfs_inode_dotl.c b/fs/9p/vfs_inode_dotl.c index 96e791c2b7b4..f309464aee4d 100644 --- a/fs/9p/vfs_inode_dotl.c +++ b/fs/9p/vfs_inode_dotl.c @@ -402,32 +402,17 @@ static int v9fs_vfs_mkdir_dotl(struct mnt_idmap *idmap, } /* instantiate inode and assign the unopened fid to the dentry */ - if (v9ses->cache & (CACHE_META|CACHE_LOOSE)) { - inode = v9fs_get_new_inode_from_fid(v9ses, fid, dir->i_sb); - if (IS_ERR(inode)) { - err = PTR_ERR(inode); - p9_debug(P9_DEBUG_VFS, "inode creation failed %d\n", - err); - goto error; - } - v9fs_fid_add(dentry, &fid); - v9fs_set_create_acl(inode, fid, dacl, pacl); - d_instantiate(dentry, inode); - err = 0; - } else { - /* - * Not in cached mode. No need to populate - * inode with stat. We need to get an inode - * so that we can set the acl with dentry - */ - inode = v9fs_get_inode(dir->i_sb, mode, 0); - if (IS_ERR(inode)) { - err = PTR_ERR(inode); - goto error; - } - v9fs_set_create_acl(inode, fid, dacl, pacl); - d_instantiate(dentry, inode); + inode = v9fs_get_new_inode_from_fid(v9ses, fid, dir->i_sb); + if (IS_ERR(inode)) { + err = PTR_ERR(inode); + p9_debug(P9_DEBUG_VFS, "inode creation failed %d\n", + err); + goto error; } + v9fs_set_create_acl(inode, fid, dacl, pacl); + v9fs_fid_add(dentry, &fid); + d_instantiate(dentry, inode); + err = 0; inc_nlink(dir); v9fs_invalidate_inode_attr(dir); error: diff --git a/fs/Kconfig b/fs/Kconfig index 85ae4953f2d7..f2bb0c07aaf2 100644 --- a/fs/Kconfig +++ b/fs/Kconfig @@ -158,7 +158,6 @@ menu "DOS/FAT/EXFAT/NT Filesystems" source "fs/fat/Kconfig" source "fs/exfat/Kconfig" -source "fs/ntfs/Kconfig" source "fs/ntfs3/Kconfig" endmenu diff --git a/fs/Makefile b/fs/Makefile index f9541f40be4e..259608369c8a 100644 --- a/fs/Makefile +++ b/fs/Makefile @@ -91,7 +91,6 @@ obj-y += unicode/ obj-$(CONFIG_SYSV_FS) += sysv/ obj-$(CONFIG_SMBFS) += smb/ obj-$(CONFIG_HPFS_FS) += hpfs/ -obj-$(CONFIG_NTFS_FS) += ntfs/ obj-$(CONFIG_NTFS3_FS) += ntfs3/ obj-$(CONFIG_UFS_FS) += ufs/ obj-$(CONFIG_EFS_FS) += efs/ diff --git a/fs/afs/inode.c b/fs/afs/inode.c index 1c794a1896aa..069fb731687a 100644 --- a/fs/afs/inode.c +++ b/fs/afs/inode.c @@ -820,6 +820,7 @@ void afs_evict_inode(struct inode *inode) ASSERTCMP(inode->i_ino, ==, vnode->fid.vnode); + flush_delayed_work(&vnode->lock_work); truncate_inode_pages_final(&inode->i_data); afs_set_cache_aux(vnode, &aux); diff --git a/fs/afs/internal.h b/fs/afs/internal.h index d1287cc3d938..85059a453b6a 100644 --- a/fs/afs/internal.h +++ b/fs/afs/internal.h @@ -102,6 +102,7 @@ struct afs_call { wait_queue_head_t waitq; /* processes awaiting completion */ struct work_struct async_work; /* async I/O processor */ struct work_struct work; /* actual work processor */ + struct work_struct free_work; /* Deferred free processor */ struct rxrpc_call *rxcall; /* RxRPC call handle */ struct key *key; /* security for this call */ struct afs_net *net; /* The network namespace */ @@ -1282,6 +1283,7 @@ extern int __net_init afs_open_socket(struct afs_net *); extern void __net_exit afs_close_socket(struct afs_net *); extern void afs_charge_preallocation(struct work_struct *); extern void afs_put_call(struct afs_call *); +void afs_deferred_put_call(struct afs_call *call); extern void afs_make_call(struct afs_addr_cursor *, struct afs_call *, gfp_t); extern long afs_wait_for_call_to_complete(struct afs_call *, struct afs_addr_cursor *); extern struct afs_call *afs_alloc_flat_call(struct afs_net *, diff --git a/fs/afs/rxrpc.c b/fs/afs/rxrpc.c index 3aa0d2686420..523272b86ad4 100644 --- a/fs/afs/rxrpc.c +++ b/fs/afs/rxrpc.c @@ -18,6 +18,7 @@ struct workqueue_struct *afs_async_calls; +static void afs_deferred_free_worker(struct work_struct *work); static void afs_wake_up_call_waiter(struct sock *, struct rxrpc_call *, unsigned long); static void afs_wake_up_async_call(struct sock *, struct rxrpc_call *, unsigned long); static void afs_process_async_call(struct work_struct *); @@ -161,6 +162,7 @@ static struct afs_call *afs_alloc_call(struct afs_net *net, call->debug_id = atomic_inc_return(&rxrpc_debug_id); refcount_set(&call->ref, 1); INIT_WORK(&call->async_work, afs_process_async_call); + INIT_WORK(&call->free_work, afs_deferred_free_worker); init_waitqueue_head(&call->waitq); spin_lock_init(&call->state_lock); call->iter = &call->def_iter; @@ -171,6 +173,35 @@ static struct afs_call *afs_alloc_call(struct afs_net *net, return call; } +static void afs_free_call(struct afs_call *call) +{ + struct afs_net *net = call->net; + int o; + + ASSERT(!work_pending(&call->async_work)); + + if (call->rxcall) { + rxrpc_kernel_shutdown_call(net->socket, call->rxcall); + rxrpc_kernel_put_call(net->socket, call->rxcall); + call->rxcall = NULL; + } + if (call->type->destructor) + call->type->destructor(call); + + afs_unuse_server_notime(call->net, call->server, afs_server_trace_put_call); + afs_put_addrlist(call->alist); + kfree(call->request); + + o = atomic_read(&net->nr_outstanding_calls); + trace_afs_call(call->debug_id, afs_call_trace_free, 0, o, + __builtin_return_address(0)); + kfree(call); + + o = atomic_dec_return(&net->nr_outstanding_calls); + if (o == 0) + wake_up_var(&net->nr_outstanding_calls); +} + /* * Dispose of a reference on a call. */ @@ -185,31 +216,34 @@ void afs_put_call(struct afs_call *call) o = atomic_read(&net->nr_outstanding_calls); trace_afs_call(debug_id, afs_call_trace_put, r - 1, o, __builtin_return_address(0)); + if (zero) + afs_free_call(call); +} - if (zero) { - ASSERT(!work_pending(&call->async_work)); - ASSERT(call->type->name != NULL); - - if (call->rxcall) { - rxrpc_kernel_shutdown_call(net->socket, call->rxcall); - rxrpc_kernel_put_call(net->socket, call->rxcall); - call->rxcall = NULL; - } - if (call->type->destructor) - call->type->destructor(call); +static void afs_deferred_free_worker(struct work_struct *work) +{ + struct afs_call *call = container_of(work, struct afs_call, free_work); - afs_unuse_server_notime(call->net, call->server, afs_server_trace_put_call); - afs_put_addrlist(call->alist); - kfree(call->request); + afs_free_call(call); +} - trace_afs_call(call->debug_id, afs_call_trace_free, 0, o, - __builtin_return_address(0)); - kfree(call); +/* + * Dispose of a reference on a call, deferring the cleanup to a workqueue + * to avoid lock recursion. + */ +void afs_deferred_put_call(struct afs_call *call) +{ + struct afs_net *net = call->net; + unsigned int debug_id = call->debug_id; + bool zero; + int r, o; - o = atomic_dec_return(&net->nr_outstanding_calls); - if (o == 0) - wake_up_var(&net->nr_outstanding_calls); - } + zero = __refcount_dec_and_test(&call->ref, &r); + o = atomic_read(&net->nr_outstanding_calls); + trace_afs_call(debug_id, afs_call_trace_put, r - 1, o, + __builtin_return_address(0)); + if (zero) + schedule_work(&call->free_work); } static struct afs_call *afs_get_call(struct afs_call *call, @@ -683,7 +717,8 @@ static void afs_wake_up_call_waiter(struct sock *sk, struct rxrpc_call *rxcall, } /* - * wake up an asynchronous call + * Wake up an asynchronous call. The caller is holding the call notify + * spinlock around this, so we can't call afs_put_call(). */ static void afs_wake_up_async_call(struct sock *sk, struct rxrpc_call *rxcall, unsigned long call_user_ID) @@ -700,7 +735,7 @@ static void afs_wake_up_async_call(struct sock *sk, struct rxrpc_call *rxcall, __builtin_return_address(0)); if (!queue_work(afs_async_calls, &call->async_work)) - afs_put_call(call); + afs_deferred_put_call(call); } } diff --git a/fs/afs/super.c b/fs/afs/super.c index 1dd6c7c9713b..3fa0c38d7a7c 100644 --- a/fs/afs/super.c +++ b/fs/afs/super.c @@ -670,7 +670,6 @@ static void afs_i_init_once(void *_vnode) INIT_LIST_HEAD(&vnode->wb_keys); INIT_LIST_HEAD(&vnode->pending_locks); INIT_LIST_HEAD(&vnode->granted_locks); - INIT_DELAYED_WORK(&vnode->lock_work, afs_lock_work); INIT_LIST_HEAD(&vnode->cb_mmap_link); seqlock_init(&vnode->cb_lock); } @@ -702,6 +701,7 @@ static struct inode *afs_alloc_inode(struct super_block *sb) init_rwsem(&vnode->rmdir_lock); INIT_WORK(&vnode->cb_work, afs_invalidate_mmap_work); + INIT_DELAYED_WORK(&vnode->lock_work, afs_lock_work); _leave(" = %p", &vnode->netfs.inode); return &vnode->netfs.inode; diff --git a/fs/aio.c b/fs/aio.c index 4a9b5e4719ee..be12d7c049ef 100644 --- a/fs/aio.c +++ b/fs/aio.c @@ -1177,7 +1177,7 @@ static void aio_complete(struct aio_kiocb *iocb) * from IRQ context. */ if (iocb->ki_eventfd) - eventfd_signal(iocb->ki_eventfd, 1); + eventfd_signal(iocb->ki_eventfd); /* * We have to order our ring_info tail store above and test diff --git a/fs/attr.c b/fs/attr.c index a8ae5f6d9b16..dd3b0b5115af 100644 --- a/fs/attr.c +++ b/fs/attr.c @@ -173,7 +173,17 @@ int setattr_prepare(struct mnt_idmap *idmap, struct dentry *dentry, * ATTR_FORCE. */ if (ia_valid & ATTR_SIZE) { - int error = inode_newsize_ok(inode, attr->ia_size); + int error; + + /* + * Verity files are immutable, so deny truncates. This isn't + * covered by the open-time check because sys_truncate() takes a + * path, not an open file. + */ + if (IS_VERITY(inode)) + return -EPERM; + + error = inode_newsize_ok(inode, attr->ia_size); if (error) return error; } diff --git a/fs/btrfs/acl.c b/fs/btrfs/acl.c index 7427449a04a3..eb78685f1c46 100644 --- a/fs/btrfs/acl.c +++ b/fs/btrfs/acl.c @@ -117,6 +117,9 @@ int btrfs_set_acl(struct mnt_idmap *idmap, struct dentry *dentry, struct inode *inode = d_inode(dentry); umode_t old_mode = inode->i_mode; + if (btrfs_root_readonly(BTRFS_I(inode)->root)) + return -EROFS; + if (type == ACL_TYPE_ACCESS && acl) { ret = posix_acl_update_mode(idmap, inode, &inode->i_mode, &acl); diff --git a/fs/btrfs/bio.c b/fs/btrfs/bio.c index 36b5ec9701d2..578391215c89 100644 --- a/fs/btrfs/bio.c +++ b/fs/btrfs/bio.c @@ -780,9 +780,15 @@ int btrfs_repair_io_failure(struct btrfs_fs_info *fs_info, u64 ino, u64 start, struct bio bio; int ret = 0; - ASSERT(!(fs_info->sb->s_flags & SB_RDONLY)); BUG_ON(!mirror_num); + /* + * The fs either mounted RO or hit critical errors, no need + * to continue repairing. + */ + if (unlikely(sb_rdonly(fs_info->sb))) + return 0; + if (btrfs_repair_one_zone(fs_info, logical)) return 0; diff --git a/fs/btrfs/block-group.c b/fs/btrfs/block-group.c index 705724089f75..4ea1fb222ba9 100644 --- a/fs/btrfs/block-group.c +++ b/fs/btrfs/block-group.c @@ -1308,7 +1308,7 @@ int btrfs_remove_block_group(struct btrfs_trans_handle *trans, /* Once for the lookup reference */ btrfs_put_block_group(block_group); if (remove_rsv) - btrfs_delayed_refs_rsv_release(fs_info, 1); + btrfs_dec_delayed_refs_rsv_bg_updates(fs_info); btrfs_free_path(path); return ret; } @@ -2799,9 +2799,12 @@ void btrfs_create_pending_block_groups(struct btrfs_trans_handle *trans) /* Already aborted the transaction if it failed. */ next: - btrfs_delayed_refs_rsv_release(fs_info, 1); + btrfs_dec_delayed_refs_rsv_bg_inserts(fs_info); + + spin_lock(&fs_info->unused_bgs_lock); list_del_init(&block_group->bg_list); clear_bit(BLOCK_GROUP_FLAG_NEW, &block_group->runtime_flags); + spin_unlock(&fs_info->unused_bgs_lock); } btrfs_trans_release_chunk_metadata(trans); } @@ -2909,8 +2912,7 @@ struct btrfs_block_group *btrfs_make_block_group(struct btrfs_trans_handle *tran #endif list_add_tail(&cache->bg_list, &trans->new_bgs); - trans->delayed_ref_updates++; - btrfs_update_delayed_refs_rsv(trans); + btrfs_inc_delayed_refs_rsv_bg_inserts(fs_info); set_avail_alloc_bits(fs_info, type); return cache; @@ -3462,7 +3464,7 @@ int btrfs_start_dirty_block_groups(struct btrfs_trans_handle *trans) if (should_put) btrfs_put_block_group(cache); if (drop_reserve) - btrfs_delayed_refs_rsv_release(fs_info, 1); + btrfs_dec_delayed_refs_rsv_bg_updates(fs_info); /* * Avoid blocking other tasks for too long. It might even save * us from writing caches for block groups that are going to be @@ -3566,8 +3568,7 @@ int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans) cache_save_setup(cache, trans, path); if (!ret) - ret = btrfs_run_delayed_refs(trans, - (unsigned long) -1); + ret = btrfs_run_delayed_refs(trans, U64_MAX); if (!ret && cache->disk_cache_state == BTRFS_DC_SETUP) { cache->io_ctl.inode = NULL; @@ -3610,7 +3611,7 @@ int btrfs_write_dirty_block_groups(struct btrfs_trans_handle *trans) /* If its not on the io list, we need to put the block group */ if (should_put) btrfs_put_block_group(cache); - btrfs_delayed_refs_rsv_release(fs_info, 1); + btrfs_dec_delayed_refs_rsv_bg_updates(fs_info); spin_lock(&cur_trans->dirty_bgs_lock); } spin_unlock(&cur_trans->dirty_bgs_lock); @@ -3635,12 +3636,12 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans, u64 bytenr, u64 num_bytes, bool alloc) { struct btrfs_fs_info *info = trans->fs_info; - struct btrfs_block_group *cache = NULL; - u64 total = num_bytes; + struct btrfs_space_info *space_info; + struct btrfs_block_group *cache; u64 old_val; - u64 byte_in_group; + bool reclaim = false; + bool bg_already_dirty = true; int factor; - int ret = 0; /* Block accounting for super block */ spin_lock(&info->delalloc_root_lock); @@ -3652,97 +3653,86 @@ int btrfs_update_block_group(struct btrfs_trans_handle *trans, btrfs_set_super_bytes_used(info->super_copy, old_val); spin_unlock(&info->delalloc_root_lock); - while (total) { - struct btrfs_space_info *space_info; - bool reclaim = false; - - cache = btrfs_lookup_block_group(info, bytenr); - if (!cache) { - ret = -ENOENT; - break; - } - space_info = cache->space_info; - factor = btrfs_bg_type_to_factor(cache->flags); + cache = btrfs_lookup_block_group(info, bytenr); + if (!cache) + return -ENOENT; - /* - * If this block group has free space cache written out, we - * need to make sure to load it if we are removing space. This - * is because we need the unpinning stage to actually add the - * space back to the block group, otherwise we will leak space. - */ - if (!alloc && !btrfs_block_group_done(cache)) - btrfs_cache_block_group(cache, true); + /* An extent can not span multiple block groups. */ + ASSERT(bytenr + num_bytes <= cache->start + cache->length); - byte_in_group = bytenr - cache->start; - WARN_ON(byte_in_group > cache->length); + space_info = cache->space_info; + factor = btrfs_bg_type_to_factor(cache->flags); - spin_lock(&space_info->lock); - spin_lock(&cache->lock); + /* + * If this block group has free space cache written out, we need to make + * sure to load it if we are removing space. This is because we need + * the unpinning stage to actually add the space back to the block group, + * otherwise we will leak space. + */ + if (!alloc && !btrfs_block_group_done(cache)) + btrfs_cache_block_group(cache, true); - if (btrfs_test_opt(info, SPACE_CACHE) && - cache->disk_cache_state < BTRFS_DC_CLEAR) - cache->disk_cache_state = BTRFS_DC_CLEAR; + spin_lock(&space_info->lock); + spin_lock(&cache->lock); - old_val = cache->used; - num_bytes = min(total, cache->length - byte_in_group); - if (alloc) { - old_val += num_bytes; - cache->used = old_val; - cache->reserved -= num_bytes; - space_info->bytes_reserved -= num_bytes; - space_info->bytes_used += num_bytes; - space_info->disk_used += num_bytes * factor; - spin_unlock(&cache->lock); - spin_unlock(&space_info->lock); - } else { - old_val -= num_bytes; - cache->used = old_val; - cache->pinned += num_bytes; - btrfs_space_info_update_bytes_pinned(info, space_info, - num_bytes); - space_info->bytes_used -= num_bytes; - space_info->disk_used -= num_bytes * factor; + if (btrfs_test_opt(info, SPACE_CACHE) && + cache->disk_cache_state < BTRFS_DC_CLEAR) + cache->disk_cache_state = BTRFS_DC_CLEAR; - reclaim = should_reclaim_block_group(cache, num_bytes); + old_val = cache->used; + if (alloc) { + old_val += num_bytes; + cache->used = old_val; + cache->reserved -= num_bytes; + space_info->bytes_reserved -= num_bytes; + space_info->bytes_used += num_bytes; + space_info->disk_used += num_bytes * factor; + spin_unlock(&cache->lock); + spin_unlock(&space_info->lock); + } else { + old_val -= num_bytes; + cache->used = old_val; + cache->pinned += num_bytes; + btrfs_space_info_update_bytes_pinned(info, space_info, num_bytes); + space_info->bytes_used -= num_bytes; + space_info->disk_used -= num_bytes * factor; - spin_unlock(&cache->lock); - spin_unlock(&space_info->lock); + reclaim = should_reclaim_block_group(cache, num_bytes); - set_extent_bit(&trans->transaction->pinned_extents, - bytenr, bytenr + num_bytes - 1, - EXTENT_DIRTY, NULL); - } + spin_unlock(&cache->lock); + spin_unlock(&space_info->lock); - spin_lock(&trans->transaction->dirty_bgs_lock); - if (list_empty(&cache->dirty_list)) { - list_add_tail(&cache->dirty_list, - &trans->transaction->dirty_bgs); - trans->delayed_ref_updates++; - btrfs_get_block_group(cache); - } - spin_unlock(&trans->transaction->dirty_bgs_lock); + set_extent_bit(&trans->transaction->pinned_extents, bytenr, + bytenr + num_bytes - 1, EXTENT_DIRTY, NULL); + } - /* - * No longer have used bytes in this block group, queue it for - * deletion. We do this after adding the block group to the - * dirty list to avoid races between cleaner kthread and space - * cache writeout. - */ - if (!alloc && old_val == 0) { - if (!btrfs_test_opt(info, DISCARD_ASYNC)) - btrfs_mark_bg_unused(cache); - } else if (!alloc && reclaim) { - btrfs_mark_bg_to_reclaim(cache); - } + spin_lock(&trans->transaction->dirty_bgs_lock); + if (list_empty(&cache->dirty_list)) { + list_add_tail(&cache->dirty_list, &trans->transaction->dirty_bgs); + bg_already_dirty = false; + btrfs_get_block_group(cache); + } + spin_unlock(&trans->transaction->dirty_bgs_lock); - btrfs_put_block_group(cache); - total -= num_bytes; - bytenr += num_bytes; + /* + * No longer have used bytes in this block group, queue it for deletion. + * We do this after adding the block group to the dirty list to avoid + * races between cleaner kthread and space cache writeout. + */ + if (!alloc && old_val == 0) { + if (!btrfs_test_opt(info, DISCARD_ASYNC)) + btrfs_mark_bg_unused(cache); + } else if (!alloc && reclaim) { + btrfs_mark_bg_to_reclaim(cache); } + btrfs_put_block_group(cache); + /* Modified block groups are accounted for in the delayed_refs_rsv. */ - btrfs_update_delayed_refs_rsv(trans); - return ret; + if (!bg_already_dirty) + btrfs_inc_delayed_refs_rsv_bg_updates(info); + + return 0; } /* diff --git a/fs/btrfs/block-rsv.c b/fs/btrfs/block-rsv.c index 07bf07431a7f..14530d436d6e 100644 --- a/fs/btrfs/block-rsv.c +++ b/fs/btrfs/block-rsv.c @@ -324,10 +324,25 @@ void btrfs_block_rsv_add_bytes(struct btrfs_block_rsv *block_rsv, void btrfs_update_global_block_rsv(struct btrfs_fs_info *fs_info) { struct btrfs_block_rsv *block_rsv = &fs_info->global_block_rsv; - struct btrfs_space_info *sinfo = block_rsv->space_info; + struct btrfs_space_info *sinfo; struct btrfs_root *root, *tmp; - u64 num_bytes = btrfs_root_used(&fs_info->tree_root->root_item); unsigned int min_items = 1; + u64 num_bytes; + + /* + * A full read-only mount (rescue options) cannot start transactions, + * so the global reserve is never consumed. Mark it as full and skip + * the accounting. + */ + if (btrfs_is_full_ro(fs_info)) { + spin_lock(&block_rsv->lock); + block_rsv->full = true; + spin_unlock(&block_rsv->lock); + return; + } + + sinfo = block_rsv->space_info; + num_bytes = btrfs_root_used(&fs_info->tree_root->root_item); /* * The global block rsv is based on the size of the extent tree, the diff --git a/fs/btrfs/defrag.c b/fs/btrfs/defrag.c index e1475dfdf7a8..c7b0582d1444 100644 --- a/fs/btrfs/defrag.c +++ b/fs/btrfs/defrag.c @@ -1017,26 +1017,21 @@ static_assert(PAGE_ALIGNED(CLUSTER_SIZE)); * * - Extent bits are locked */ -static int defrag_one_locked_target(struct btrfs_inode *inode, - struct defrag_target_range *target, - struct page **pages, int nr_pages, - struct extent_state **cached_state) +static void defrag_one_locked_target(struct btrfs_inode *inode, + struct defrag_target_range *target, + struct page **pages, int nr_pages, + struct extent_state **cached_state) { struct btrfs_fs_info *fs_info = inode->root->fs_info; - struct extent_changeset *data_reserved = NULL; const u64 start = target->start; const u64 len = target->len; unsigned long last_index = (start + len - 1) >> PAGE_SHIFT; unsigned long start_index = start >> PAGE_SHIFT; unsigned long first_index = page_index(pages[0]); - int ret = 0; int i; ASSERT(last_index - first_index + 1 <= nr_pages); - ret = btrfs_delalloc_reserve_space(inode, &data_reserved, start, len); - if (ret < 0) - return ret; clear_extent_bit(&inode->io_tree, start, start + len - 1, EXTENT_DELALLOC | EXTENT_DO_ACCOUNTING | EXTENT_DEFRAG, cached_state); @@ -1048,10 +1043,6 @@ static int defrag_one_locked_target(struct btrfs_inode *inode, ClearPageChecked(pages[i]); btrfs_page_clamp_set_dirty(fs_info, pages[i], start, len); } - btrfs_delalloc_release_extents(inode, len); - extent_changeset_free(data_reserved); - - return ret; } static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len, @@ -1067,6 +1058,8 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len, u64 last_index = (start + len - 1) >> PAGE_SHIFT; u64 start_index = start >> PAGE_SHIFT; unsigned int nr_pages = last_index - start_index + 1; + struct extent_changeset *data_reserved = NULL; + u64 last_defrag_end = start; int ret = 0; int i; @@ -1077,6 +1070,22 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len, if (!pages) return -ENOMEM; + /* + * Reserve delalloc space before locking the range and before locking + * and dirtying any folios - otherwise we could deadlock, for example + * after defrag of one range we dirty folios and keep them locked when + * we move to the next range, so reserving delalloc space right before + * each range could trigger flushing of delalloc and deadlock on the + * extent lock or trigger a transaction commit with flushoncommit, which + * can either deadlock on the lock of a folio made dirty in the previous + * range or the extent lock. + */ + ret = btrfs_delalloc_reserve_space(inode, &data_reserved, start, len); + if (ret < 0) { + kfree(pages); + return ret; + } + /* Prepare all pages */ for (i = 0; i < nr_pages; i++) { pages[i] = defrag_prepare_one_page(inode, start_index + i); @@ -1107,10 +1116,11 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len, goto unlock_extent; list_for_each_entry(entry, &target_list, list) { - ret = defrag_one_locked_target(inode, entry, pages, nr_pages, - &cached_state); - if (ret < 0) - break; + defrag_one_locked_target(inode, entry, pages, nr_pages, &cached_state); + if (entry->start > last_defrag_end) + btrfs_delalloc_release_space(inode, data_reserved, last_defrag_end, + entry->start - last_defrag_end, true); + last_defrag_end = entry->start + entry->len; } list_for_each_entry_safe(entry, tmp, &target_list, list) { @@ -1129,6 +1139,12 @@ static int defrag_one_range(struct btrfs_inode *inode, u64 start, u32 len, } } kfree(pages); + btrfs_delalloc_release_extents(inode, len); + if (last_defrag_end < start + len) + btrfs_delalloc_release_space(inode, data_reserved, last_defrag_end, + start + len - last_defrag_end, true); + extent_changeset_free(data_reserved); + return ret; } diff --git a/fs/btrfs/delayed-inode.c b/fs/btrfs/delayed-inode.c index c39e39142abf..4fde25d02298 100644 --- a/fs/btrfs/delayed-inode.c +++ b/fs/btrfs/delayed-inode.c @@ -1835,19 +1835,19 @@ static void fill_stack_inode_item(struct btrfs_trans_handle *trans, btrfs_set_stack_inode_block_group(inode_item, 0); btrfs_set_stack_timespec_sec(&inode_item->atime, - inode->i_atime.tv_sec); + inode_get_atime_sec(inode)); btrfs_set_stack_timespec_nsec(&inode_item->atime, - inode->i_atime.tv_nsec); + inode_get_atime_nsec(inode)); btrfs_set_stack_timespec_sec(&inode_item->mtime, - inode->i_mtime.tv_sec); + inode_get_mtime_sec(inode)); btrfs_set_stack_timespec_nsec(&inode_item->mtime, - inode->i_mtime.tv_nsec); + inode_get_mtime_nsec(inode)); btrfs_set_stack_timespec_sec(&inode_item->ctime, - inode_get_ctime(inode).tv_sec); + inode_get_ctime_sec(inode)); btrfs_set_stack_timespec_nsec(&inode_item->ctime, - inode_get_ctime(inode).tv_nsec); + inode_get_ctime_nsec(inode)); btrfs_set_stack_timespec_sec(&inode_item->otime, BTRFS_I(inode)->i_otime_sec); btrfs_set_stack_timespec_nsec(&inode_item->otime, BTRFS_I(inode)->i_otime_nsec); @@ -1890,11 +1890,11 @@ int btrfs_fill_inode(struct inode *inode, u32 *rdev) btrfs_inode_split_flags(btrfs_stack_inode_flags(inode_item), &BTRFS_I(inode)->flags, &BTRFS_I(inode)->ro_flags); - inode->i_atime.tv_sec = btrfs_stack_timespec_sec(&inode_item->atime); - inode->i_atime.tv_nsec = btrfs_stack_timespec_nsec(&inode_item->atime); + inode_set_atime(inode, btrfs_stack_timespec_sec(&inode_item->atime), + btrfs_stack_timespec_nsec(&inode_item->atime)); - inode->i_mtime.tv_sec = btrfs_stack_timespec_sec(&inode_item->mtime); - inode->i_mtime.tv_nsec = btrfs_stack_timespec_nsec(&inode_item->mtime); + inode_set_mtime(inode, btrfs_stack_timespec_sec(&inode_item->mtime), + btrfs_stack_timespec_nsec(&inode_item->mtime)); inode_set_ctime(inode, btrfs_stack_timespec_sec(&inode_item->ctime), btrfs_stack_timespec_nsec(&inode_item->ctime)); diff --git a/fs/btrfs/delayed-ref.c b/fs/btrfs/delayed-ref.c index 6f2e48d697dd..29e0a22e3d67 100644 --- a/fs/btrfs/delayed-ref.c +++ b/fs/btrfs/delayed-ref.c @@ -57,16 +57,20 @@ bool btrfs_check_space_for_delayed_refs(struct btrfs_fs_info *fs_info) * Release a ref head's reservation. * * @fs_info: the filesystem - * @nr: number of items to drop + * @nr_refs: number of delayed refs to drop + * @nr_csums: number of csum items to drop * * Drops the delayed ref head's count from the delayed refs rsv and free any * excess reservation we had. */ -void btrfs_delayed_refs_rsv_release(struct btrfs_fs_info *fs_info, int nr) +void btrfs_delayed_refs_rsv_release(struct btrfs_fs_info *fs_info, int nr_refs, int nr_csums) { struct btrfs_block_rsv *block_rsv = &fs_info->delayed_refs_rsv; - const u64 num_bytes = btrfs_calc_delayed_ref_bytes(fs_info, nr); - u64 released = 0; + u64 num_bytes; + u64 released; + + num_bytes = btrfs_calc_delayed_ref_bytes(fs_info, nr_refs); + num_bytes += btrfs_calc_delayed_ref_csum_bytes(fs_info, nr_csums); released = btrfs_block_rsv_release(fs_info, block_rsv, num_bytes, NULL); if (released) @@ -77,8 +81,9 @@ void btrfs_delayed_refs_rsv_release(struct btrfs_fs_info *fs_info, int nr) /* * Adjust the size of the delayed refs rsv. * - * This is to be called anytime we may have adjusted trans->delayed_ref_updates, - * it'll calculate the additional size and add it to the delayed_refs_rsv. + * This is to be called anytime we may have adjusted trans->delayed_ref_updates + * or trans->delayed_ref_csum_deletions, it'll calculate the additional size and + * add it to the delayed_refs_rsv. */ void btrfs_update_delayed_refs_rsv(struct btrfs_trans_handle *trans) { @@ -86,17 +91,89 @@ void btrfs_update_delayed_refs_rsv(struct btrfs_trans_handle *trans) struct btrfs_block_rsv *delayed_rsv = &fs_info->delayed_refs_rsv; u64 num_bytes; - if (!trans->delayed_ref_updates) - return; + num_bytes = btrfs_calc_delayed_ref_bytes(fs_info, trans->delayed_ref_updates); + num_bytes += btrfs_calc_delayed_ref_csum_bytes(fs_info, + trans->delayed_ref_csum_deletions); - num_bytes = btrfs_calc_delayed_ref_bytes(fs_info, - trans->delayed_ref_updates); + if (num_bytes == 0) + return; spin_lock(&delayed_rsv->lock); delayed_rsv->size += num_bytes; delayed_rsv->full = false; spin_unlock(&delayed_rsv->lock); trans->delayed_ref_updates = 0; + trans->delayed_ref_csum_deletions = 0; +} + +/* + * Adjust the size of the delayed refs block reserve for 1 block group item + * insertion, used after allocating a block group. + */ +void btrfs_inc_delayed_refs_rsv_bg_inserts(struct btrfs_fs_info *fs_info) +{ + struct btrfs_block_rsv *delayed_rsv = &fs_info->delayed_refs_rsv; + + spin_lock(&delayed_rsv->lock); + /* + * Inserting a block group item does not require changing the free space + * tree, only the extent tree or the block group tree, so this is all we + * need. + */ + delayed_rsv->size += btrfs_calc_insert_metadata_size(fs_info, 1); + delayed_rsv->full = false; + spin_unlock(&delayed_rsv->lock); +} + +/* + * Adjust the size of the delayed refs block reserve to release space for 1 + * block group item insertion. + */ +void btrfs_dec_delayed_refs_rsv_bg_inserts(struct btrfs_fs_info *fs_info) +{ + struct btrfs_block_rsv *delayed_rsv = &fs_info->delayed_refs_rsv; + const u64 num_bytes = btrfs_calc_insert_metadata_size(fs_info, 1); + u64 released; + + released = btrfs_block_rsv_release(fs_info, delayed_rsv, num_bytes, NULL); + if (released > 0) + trace_btrfs_space_reservation(fs_info, "delayed_refs_rsv", + 0, released, 0); +} + +/* + * Adjust the size of the delayed refs block reserve for 1 block group item + * update. + */ +void btrfs_inc_delayed_refs_rsv_bg_updates(struct btrfs_fs_info *fs_info) +{ + struct btrfs_block_rsv *delayed_rsv = &fs_info->delayed_refs_rsv; + + spin_lock(&delayed_rsv->lock); + /* + * Updating a block group item does not result in new nodes/leaves and + * does not require changing the free space tree, only the extent tree + * or the block group tree, so this is all we need. + */ + delayed_rsv->size += btrfs_calc_metadata_size(fs_info, 1); + delayed_rsv->full = false; + spin_unlock(&delayed_rsv->lock); +} + +/* + * Adjust the size of the delayed refs block reserve to release space for 1 + * block group item update. + */ +void btrfs_dec_delayed_refs_rsv_bg_updates(struct btrfs_fs_info *fs_info) +{ + struct btrfs_block_rsv *delayed_rsv = &fs_info->delayed_refs_rsv; + const u64 num_bytes = btrfs_calc_metadata_size(fs_info, 1); + u64 released; + + released = btrfs_block_rsv_release(fs_info, delayed_rsv, num_bytes, NULL); + if (released > 0) + trace_btrfs_space_reservation(fs_info, "delayed_refs_rsv", + 0, released, 0); } /* @@ -422,7 +499,8 @@ int btrfs_delayed_ref_lock(struct btrfs_delayed_ref_root *delayed_refs, return 0; } -static inline void drop_delayed_ref(struct btrfs_delayed_ref_root *delayed_refs, +static inline void drop_delayed_ref(struct btrfs_fs_info *fs_info, + struct btrfs_delayed_ref_root *delayed_refs, struct btrfs_delayed_ref_head *head, struct btrfs_delayed_ref_node *ref) { @@ -433,9 +511,11 @@ static inline void drop_delayed_ref(struct btrfs_delayed_ref_root *delayed_refs, list_del(&ref->add_list); btrfs_put_delayed_ref(ref); atomic_dec(&delayed_refs->num_entries); + btrfs_delayed_refs_rsv_release(fs_info, 1, 0); } -static bool merge_ref(struct btrfs_delayed_ref_root *delayed_refs, +static bool merge_ref(struct btrfs_fs_info *fs_info, + struct btrfs_delayed_ref_root *delayed_refs, struct btrfs_delayed_ref_head *head, struct btrfs_delayed_ref_node *ref, u64 seq) @@ -464,10 +544,10 @@ static bool merge_ref(struct btrfs_delayed_ref_root *delayed_refs, mod = -next->ref_mod; } - drop_delayed_ref(delayed_refs, head, next); + drop_delayed_ref(fs_info, delayed_refs, head, next); ref->ref_mod += mod; if (ref->ref_mod == 0) { - drop_delayed_ref(delayed_refs, head, ref); + drop_delayed_ref(fs_info, delayed_refs, head, ref); done = true; } else { /* @@ -505,7 +585,7 @@ void btrfs_merge_delayed_refs(struct btrfs_fs_info *fs_info, ref = rb_entry(node, struct btrfs_delayed_ref_node, ref_node); if (seq && ref->seq >= seq) continue; - if (merge_ref(delayed_refs, head, ref, seq)) + if (merge_ref(fs_info, delayed_refs, head, ref, seq)) goto again; } } @@ -584,10 +664,11 @@ void btrfs_delete_ref_head(struct btrfs_delayed_ref_root *delayed_refs, * Return true if the ref was merged into an existing one (and therefore can be * freed by the caller). */ -static bool insert_delayed_ref(struct btrfs_delayed_ref_root *root, +static bool insert_delayed_ref(struct btrfs_trans_handle *trans, struct btrfs_delayed_ref_head *href, struct btrfs_delayed_ref_node *ref) { + struct btrfs_delayed_ref_root *root = &trans->transaction->delayed_refs; struct btrfs_delayed_ref_node *exist; int mod; @@ -598,6 +679,7 @@ static bool insert_delayed_ref(struct btrfs_delayed_ref_root *root, list_add_tail(&ref->add_list, &href->ref_add_list); atomic_inc(&root->num_entries); spin_unlock(&href->lock); + trans->delayed_ref_updates++; return false; } @@ -626,7 +708,7 @@ static bool insert_delayed_ref(struct btrfs_delayed_ref_root *root, /* remove existing tail if its ref_mod is zero */ if (exist->ref_mod == 0) - drop_delayed_ref(root, href, exist); + drop_delayed_ref(trans->fs_info, root, href, exist); spin_unlock(&href->lock); return true; } @@ -695,6 +777,8 @@ static noinline void update_existing_head_ref(struct btrfs_trans_handle *trans, /* * If we are going to from a positive ref mod to a negative or vice * versa we need to make sure to adjust pending_csums accordingly. + * We reserve bytes for csum deletion when adding or updating a ref head + * see add_delayed_ref_head() for more details. */ if (existing->is_data) { u64 csum_leaves = @@ -703,11 +787,11 @@ static noinline void update_existing_head_ref(struct btrfs_trans_handle *trans, if (existing->total_ref_mod >= 0 && old_ref_mod < 0) { delayed_refs->pending_csums -= existing->num_bytes; - btrfs_delayed_refs_rsv_release(fs_info, csum_leaves); + btrfs_delayed_refs_rsv_release(fs_info, 0, csum_leaves); } if (existing->total_ref_mod < 0 && old_ref_mod >= 0) { delayed_refs->pending_csums += existing->num_bytes; - trans->delayed_ref_updates += csum_leaves; + trans->delayed_ref_csum_deletions += csum_leaves; } } @@ -819,16 +903,21 @@ add_delayed_ref_head(struct btrfs_trans_handle *trans, kmem_cache_free(btrfs_delayed_ref_head_cachep, head_ref); head_ref = existing; } else { + /* + * We reserve the amount of bytes needed to delete csums when + * adding the ref head and not when adding individual drop refs + * since the csum items are deleted only after running the last + * delayed drop ref (the data extent's ref count drops to 0). + */ if (head_ref->is_data && head_ref->ref_mod < 0) { delayed_refs->pending_csums += head_ref->num_bytes; - trans->delayed_ref_updates += + trans->delayed_ref_csum_deletions += btrfs_csum_bytes_to_leaves(trans->fs_info, head_ref->num_bytes); } delayed_refs->num_heads++; delayed_refs->num_heads_ready++; atomic_inc(&delayed_refs->num_entries); - trans->delayed_ref_updates++; } if (qrecord_inserted_ret) *qrecord_inserted_ret = qrecord_inserted; @@ -959,7 +1048,7 @@ int btrfs_add_delayed_tree_ref(struct btrfs_trans_handle *trans, head_ref = add_delayed_ref_head(trans, head_ref, record, action, &qrecord_inserted); - merged = insert_delayed_ref(delayed_refs, head_ref, &ref->node); + merged = insert_delayed_ref(trans, head_ref, &ref->node); spin_unlock(&delayed_refs->lock); /* @@ -1051,7 +1140,7 @@ int btrfs_add_delayed_data_ref(struct btrfs_trans_handle *trans, head_ref = add_delayed_ref_head(trans, head_ref, record, action, &qrecord_inserted); - merged = insert_delayed_ref(delayed_refs, head_ref, &ref->node); + merged = insert_delayed_ref(trans, head_ref, &ref->node); spin_unlock(&delayed_refs->lock); /* diff --git a/fs/btrfs/delayed-ref.h b/fs/btrfs/delayed-ref.h index fd9bf2b709c0..5dc4f2dc4f59 100644 --- a/fs/btrfs/delayed-ref.h +++ b/fs/btrfs/delayed-ref.h @@ -277,6 +277,17 @@ static inline u64 btrfs_calc_delayed_ref_bytes(const struct btrfs_fs_info *fs_in return num_bytes; } +static inline u64 btrfs_calc_delayed_ref_csum_bytes(const struct btrfs_fs_info *fs_info, + int num_csum_items) +{ + /* + * Deleting csum items does not result in new nodes/leaves and does not + * require changing the free space tree, only the csum tree, so this is + * all we need. + */ + return btrfs_calc_metadata_size(fs_info, num_csum_items); +} + static inline void btrfs_init_generic_ref(struct btrfs_ref *generic_ref, int action, u64 bytenr, u64 len, u64 parent) { @@ -402,8 +413,12 @@ struct btrfs_delayed_ref_head *btrfs_select_ref_head( int btrfs_check_delayed_seq(struct btrfs_fs_info *fs_info, u64 seq); -void btrfs_delayed_refs_rsv_release(struct btrfs_fs_info *fs_info, int nr); +void btrfs_delayed_refs_rsv_release(struct btrfs_fs_info *fs_info, int nr_refs, int nr_csums); void btrfs_update_delayed_refs_rsv(struct btrfs_trans_handle *trans); +void btrfs_inc_delayed_refs_rsv_bg_inserts(struct btrfs_fs_info *fs_info); +void btrfs_dec_delayed_refs_rsv_bg_inserts(struct btrfs_fs_info *fs_info); +void btrfs_inc_delayed_refs_rsv_bg_updates(struct btrfs_fs_info *fs_info); +void btrfs_dec_delayed_refs_rsv_bg_updates(struct btrfs_fs_info *fs_info); int btrfs_delayed_refs_rsv_refill(struct btrfs_fs_info *fs_info, enum btrfs_reserve_flush_enum flush); void btrfs_migrate_to_delayed_refs_rsv(struct btrfs_fs_info *fs_info, diff --git a/fs/btrfs/dev-replace.c b/fs/btrfs/dev-replace.c index f77ef719a3b1..4f97a9222b74 100644 --- a/fs/btrfs/dev-replace.c +++ b/fs/btrfs/dev-replace.c @@ -506,6 +506,7 @@ static int mark_block_group_to_copy(struct btrfs_fs_info *fs_info, path->reada = READA_FORWARD; path->search_commit_root = 1; path->skip_locking = 1; + path->need_commit_sem = 1; key.objectid = src_dev->devid; key.type = BTRFS_DEV_EXTENT_KEY; diff --git a/fs/btrfs/disk-io.c b/fs/btrfs/disk-io.c index c7746baa86f8..20253e3e5f82 100644 --- a/fs/btrfs/disk-io.c +++ b/fs/btrfs/disk-io.c @@ -3298,6 +3298,10 @@ int __cold open_ctree(struct super_block *sb, struct btrfs_fs_devices *fs_device if (ret) goto fail_alloc; + /* If the fs has any rescue options, no transaction is allowed. */ + if (btrfs_is_full_ro(fs_info)) + WRITE_ONCE(fs_info->fs_error, -EROFS); + ret = btrfs_check_features(fs_info, !sb_rdonly(sb)); if (ret < 0) goto fail_alloc; @@ -4661,6 +4665,7 @@ static void btrfs_destroy_delayed_refs(struct btrfs_transaction *trans, list_del(&ref->add_list); atomic_dec(&delayed_refs->num_entries); btrfs_put_delayed_ref(ref); + btrfs_delayed_refs_rsv_release(fs_info, 1, 0); } if (head->must_insert_reserved) pin_bytes = true; @@ -4781,6 +4786,7 @@ static void btrfs_destroy_marked_extents(struct btrfs_fs_info *fs_info, free_extent_buffer_stale(eb); } } + extent_io_tree_release(dirty_pages); } static void btrfs_destroy_pinned_extent(struct btrfs_fs_info *fs_info, @@ -4858,7 +4864,7 @@ void btrfs_cleanup_dirty_bgs(struct btrfs_transaction *cur_trans, spin_unlock(&cur_trans->dirty_bgs_lock); btrfs_put_block_group(cache); - btrfs_delayed_refs_rsv_release(fs_info, 1); + btrfs_dec_delayed_refs_rsv_bg_updates(fs_info); spin_lock(&cur_trans->dirty_bgs_lock); } spin_unlock(&cur_trans->dirty_bgs_lock); diff --git a/fs/btrfs/extent-tree.c b/fs/btrfs/extent-tree.c index ebb94ec9f3d7..b23864330b2b 100644 --- a/fs/btrfs/extent-tree.c +++ b/fs/btrfs/extent-tree.c @@ -1766,6 +1766,7 @@ static int run_one_delayed_ref(struct btrfs_trans_handle *trans, struct btrfs_delayed_extent_op *extent_op, bool insert_reserved) { + struct btrfs_fs_info *fs_info = trans->fs_info; int ret = 0; if (TRANS_ABORTED(trans)) { @@ -1775,19 +1776,22 @@ static int run_one_delayed_ref(struct btrfs_trans_handle *trans, } if (node->type == BTRFS_TREE_BLOCK_REF_KEY || - node->type == BTRFS_SHARED_BLOCK_REF_KEY) + node->type == BTRFS_SHARED_BLOCK_REF_KEY) { ret = run_delayed_tree_ref(trans, node, extent_op, insert_reserved); - else if (node->type == BTRFS_EXTENT_DATA_REF_KEY || - node->type == BTRFS_SHARED_DATA_REF_KEY) + } else if (node->type == BTRFS_EXTENT_DATA_REF_KEY || + node->type == BTRFS_SHARED_DATA_REF_KEY) { ret = run_delayed_data_ref(trans, node, extent_op, insert_reserved); - else - BUG(); + } else { + ret = -EUCLEAN; + btrfs_err(fs_info, "unexpected delayed ref node type: %u", node->type); + } + if (ret && insert_reserved) btrfs_pin_extent(trans, node->bytenr, node->num_bytes, 1); if (ret < 0) - btrfs_err(trans->fs_info, + btrfs_err(fs_info, "failed to run delayed ref for logical %llu num_bytes %llu type %u action %u ref_mod %d: %d", node->bytenr, node->num_bytes, node->type, node->action, node->ref_mod, ret); @@ -1860,28 +1864,33 @@ static int run_and_cleanup_extent_op(struct btrfs_trans_handle *trans, return ret ? ret : 1; } -void btrfs_cleanup_ref_head_accounting(struct btrfs_fs_info *fs_info, +u64 btrfs_cleanup_ref_head_accounting(struct btrfs_fs_info *fs_info, struct btrfs_delayed_ref_root *delayed_refs, struct btrfs_delayed_ref_head *head) { - int nr_items = 1; /* Dropping this ref head update. */ - /* * We had csum deletions accounted for in our delayed refs rsv, we need * to drop the csum leaves for this update from our delayed_refs_rsv. */ if (head->total_ref_mod < 0 && head->is_data) { + int nr_csums; + spin_lock(&delayed_refs->lock); delayed_refs->pending_csums -= head->num_bytes; spin_unlock(&delayed_refs->lock); - nr_items += btrfs_csum_bytes_to_leaves(fs_info, head->num_bytes); + nr_csums = btrfs_csum_bytes_to_leaves(fs_info, head->num_bytes); + + btrfs_delayed_refs_rsv_release(fs_info, 0, nr_csums); + + return btrfs_calc_delayed_ref_csum_bytes(fs_info, nr_csums); } - btrfs_delayed_refs_rsv_release(fs_info, nr_items); + return 0; } static int cleanup_ref_head(struct btrfs_trans_handle *trans, - struct btrfs_delayed_ref_head *head) + struct btrfs_delayed_ref_head *head, + u64 *bytes_released) { struct btrfs_fs_info *fs_info = trans->fs_info; @@ -1926,7 +1935,7 @@ static int cleanup_ref_head(struct btrfs_trans_handle *trans, } } - btrfs_cleanup_ref_head_accounting(fs_info, delayed_refs, head); + *bytes_released += btrfs_cleanup_ref_head_accounting(fs_info, delayed_refs, head); trace_run_delayed_ref_head(fs_info, head, 0); btrfs_delayed_ref_unlock(head); @@ -1968,7 +1977,8 @@ static struct btrfs_delayed_ref_head *btrfs_obtain_ref_head( } static int btrfs_run_delayed_refs_for_head(struct btrfs_trans_handle *trans, - struct btrfs_delayed_ref_head *locked_ref) + struct btrfs_delayed_ref_head *locked_ref, + u64 *bytes_released) { struct btrfs_fs_info *fs_info = trans->fs_info; struct btrfs_delayed_ref_root *delayed_refs; @@ -2024,7 +2034,8 @@ static int btrfs_run_delayed_refs_for_head(struct btrfs_trans_handle *trans, ret = run_one_delayed_ref(trans, ref, extent_op, must_insert_reserved); - + btrfs_delayed_refs_rsv_release(fs_info, 1, 0); + *bytes_released += btrfs_calc_delayed_ref_bytes(fs_info, 1); btrfs_free_delayed_extent_op(extent_op); if (ret) { unselect_delayed_ref_head(delayed_refs, locked_ref); @@ -2047,15 +2058,22 @@ static int btrfs_run_delayed_refs_for_head(struct btrfs_trans_handle *trans, * Returns -ENOMEM or -EIO on failure and will abort the transaction. */ static noinline int __btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, - unsigned long nr) + u64 min_bytes) { struct btrfs_fs_info *fs_info = trans->fs_info; struct btrfs_delayed_ref_root *delayed_refs; struct btrfs_delayed_ref_head *locked_ref = NULL; int ret; unsigned long count = 0; + unsigned long max_count = 0; + u64 bytes_processed = 0; delayed_refs = &trans->transaction->delayed_refs; + if (min_bytes == 0) { + max_count = delayed_refs->num_heads_ready; + min_bytes = U64_MAX; + } + do { if (!locked_ref) { locked_ref = btrfs_obtain_ref_head(trans); @@ -2084,7 +2102,7 @@ static noinline int __btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, spin_lock(&locked_ref->lock); btrfs_merge_delayed_refs(fs_info, delayed_refs, locked_ref); - ret = btrfs_run_delayed_refs_for_head(trans, locked_ref); + ret = btrfs_run_delayed_refs_for_head(trans, locked_ref, &bytes_processed); if (ret < 0 && ret != -EAGAIN) { /* * Error, btrfs_run_delayed_refs_for_head already @@ -2096,7 +2114,7 @@ static noinline int __btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, * Success, perform the usual cleanup of a processed * head */ - ret = cleanup_ref_head(trans, locked_ref); + ret = cleanup_ref_head(trans, locked_ref, &bytes_processed); if (ret > 0 ) { /* We dropped our lock, we need to loop. */ ret = 0; @@ -2113,7 +2131,9 @@ static noinline int __btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, again: locked_ref = NULL; cond_resched(); - } while ((nr != -1 && count < nr) || locked_ref); + } while ((min_bytes != U64_MAX && bytes_processed < min_bytes) || + (max_count > 0 && count < max_count) || + locked_ref); return 0; } @@ -2162,24 +2182,27 @@ static u64 find_middle(struct rb_root *root) #endif /* - * this starts processing the delayed reference count updates and - * extent insertions we have queued up so far. count can be - * 0, which means to process everything in the tree at the start - * of the run (but not newly added entries), or it can be some target - * number you'd like to process. + * Start processing the delayed reference count updates and extent insertions + * we have queued up so far. + * + * @trans: Transaction handle. + * @min_bytes: How many bytes of delayed references to process. After this + * many bytes we stop processing delayed references if there are + * any more. If 0 it means to run all existing delayed references, + * but not new ones added after running all existing ones. + * Use (u64)-1 (U64_MAX) to run all existing delayed references + * plus any new ones that are added. * * Returns 0 on success or if called with an aborted transaction * Returns <0 on error and aborts the transaction */ -int btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, - unsigned long count) +int btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, u64 min_bytes) { struct btrfs_fs_info *fs_info = trans->fs_info; struct rb_node *node; struct btrfs_delayed_ref_root *delayed_refs; struct btrfs_delayed_ref_head *head; int ret; - int run_all = count == (unsigned long)-1; /* We'll clean this up in btrfs_cleanup_transaction */ if (TRANS_ABORTED(trans)) @@ -2189,20 +2212,17 @@ int btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, return 0; delayed_refs = &trans->transaction->delayed_refs; - if (count == 0) - count = delayed_refs->num_heads_ready; - again: #ifdef SCRAMBLE_DELAYED_REFS delayed_refs->run_delayed_start = find_middle(&delayed_refs->root); #endif - ret = __btrfs_run_delayed_refs(trans, count); + ret = __btrfs_run_delayed_refs(trans, min_bytes); if (ret < 0) { btrfs_abort_transaction(trans, ret); return ret; } - if (run_all) { + if (min_bytes == U64_MAX) { btrfs_create_pending_block_groups(trans); spin_lock(&delayed_refs->lock); @@ -2868,7 +2888,15 @@ int btrfs_finish_extent_commit(struct btrfs_trans_handle *trans) block_group->length, &trimmed); + /* + * Not strictly necessary to lock, as the block_group should be + * read-only from btrfs_delete_unused_bgs(). + */ + ASSERT(block_group->ro); + spin_lock(&fs_info->unused_bgs_lock); list_del_init(&block_group->bg_list); + spin_unlock(&fs_info->unused_bgs_lock); + btrfs_unfreeze_block_group(block_group); btrfs_put_block_group(block_group); diff --git a/fs/btrfs/extent-tree.h b/fs/btrfs/extent-tree.h index ef1c1c99294e..c155f398fcb6 100644 --- a/fs/btrfs/extent-tree.h +++ b/fs/btrfs/extent-tree.h @@ -91,8 +91,8 @@ int btrfs_get_extent_inline_ref_type(const struct extent_buffer *eb, enum btrfs_inline_ref_type is_data); u64 hash_extent_data_ref(u64 root_objectid, u64 owner, u64 offset); -int btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, unsigned long count); -void btrfs_cleanup_ref_head_accounting(struct btrfs_fs_info *fs_info, +int btrfs_run_delayed_refs(struct btrfs_trans_handle *trans, u64 min_bytes); +u64 btrfs_cleanup_ref_head_accounting(struct btrfs_fs_info *fs_info, struct btrfs_delayed_ref_root *delayed_refs, struct btrfs_delayed_ref_head *head); int btrfs_lookup_data_extent(struct btrfs_fs_info *fs_info, u64 start, u64 len); diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c index 7db2344a748c..fe742200154e 100644 --- a/fs/btrfs/file.c +++ b/fs/btrfs/file.c @@ -956,70 +956,64 @@ static noinline int prepare_pages(struct inode *inode, struct page **pages, /* * This function locks the extent and properly waits for data=ordered extents - * to finish before allowing the pages to be modified if need. + * to finish before allowing the pages to be modified. * * The return value: - * 1 - the extent is locked - * 0 - the extent is not locked, and everything is OK + * 0 - the extent is locked * -EAGAIN - need re-prepare the pages * the other < 0 number - Something wrong happens */ static noinline int -lock_and_cleanup_extent_if_need(struct btrfs_inode *inode, struct page **pages, - size_t num_pages, loff_t pos, - size_t write_bytes, - u64 *lockstart, u64 *lockend, bool nowait, - struct extent_state **cached_state) +lock_and_cleanup_extent(struct btrfs_inode *inode, struct page **pages, + size_t num_pages, loff_t pos, + size_t write_bytes, + u64 *lockstart, u64 *lockend, bool nowait, + struct extent_state **cached_state) { struct btrfs_fs_info *fs_info = inode->root->fs_info; + struct btrfs_ordered_extent *ordered; u64 start_pos; u64 last_pos; int i; - int ret = 0; start_pos = round_down(pos, fs_info->sectorsize); last_pos = round_up(pos + write_bytes, fs_info->sectorsize) - 1; - if (start_pos < inode->vfs_inode.i_size) { - struct btrfs_ordered_extent *ordered; - - if (nowait) { - if (!try_lock_extent(&inode->io_tree, start_pos, last_pos, - cached_state)) { - for (i = 0; i < num_pages; i++) { - unlock_page(pages[i]); - put_page(pages[i]); - pages[i] = NULL; - } - - return -EAGAIN; - } - } else { - lock_extent(&inode->io_tree, start_pos, last_pos, cached_state); - } - - ordered = btrfs_lookup_ordered_range(inode, start_pos, - last_pos - start_pos + 1); - if (ordered && - ordered->file_offset + ordered->num_bytes > start_pos && - ordered->file_offset <= last_pos) { - unlock_extent(&inode->io_tree, start_pos, last_pos, - cached_state); + if (nowait) { + if (!try_lock_extent(&inode->io_tree, start_pos, last_pos, + cached_state)) { for (i = 0; i < num_pages; i++) { unlock_page(pages[i]); put_page(pages[i]); + pages[i] = NULL; } - btrfs_start_ordered_extent(ordered); - btrfs_put_ordered_extent(ordered); + return -EAGAIN; } - if (ordered) - btrfs_put_ordered_extent(ordered); + } else { + lock_extent(&inode->io_tree, start_pos, last_pos, cached_state); + } - *lockstart = start_pos; - *lockend = last_pos; - ret = 1; + ordered = btrfs_lookup_ordered_range(inode, start_pos, + last_pos - start_pos + 1); + if (ordered && + ordered->file_offset + ordered->num_bytes > start_pos && + ordered->file_offset <= last_pos) { + unlock_extent(&inode->io_tree, start_pos, last_pos, + cached_state); + for (i = 0; i < num_pages; i++) { + unlock_page(pages[i]); + put_page(pages[i]); + } + btrfs_start_ordered_extent(ordered); + btrfs_put_ordered_extent(ordered); + return -EAGAIN; } + if (ordered) + btrfs_put_ordered_extent(ordered); + + *lockstart = start_pos; + *lockend = last_pos; /* * We should be called after prepare_pages() which should have locked @@ -1028,7 +1022,7 @@ lock_and_cleanup_extent_if_need(struct btrfs_inode *inode, struct page **pages, for (i = 0; i < num_pages; i++) WARN_ON(!PageLocked(pages[i])); - return ret; + return 0; } /* @@ -1100,17 +1094,18 @@ void btrfs_check_nocow_unlock(struct btrfs_inode *inode) static void update_time_for_write(struct inode *inode) { - struct timespec64 now, ctime; + struct timespec64 now, ts; if (IS_NOCMTIME(inode)) return; now = current_time(inode); - if (!timespec64_equal(&inode->i_mtime, &now)) - inode->i_mtime = now; + ts = inode_get_mtime(inode); + if (!timespec64_equal(&ts, &now)) + inode_set_mtime_to_ts(inode, now); - ctime = inode_get_ctime(inode); - if (!timespec64_equal(&ctime, &now)) + ts = inode_get_ctime(inode); + if (!timespec64_equal(&ts, &now)) inode_set_ctime_to_ts(inode, now); if (IS_I_VERSION(inode)) @@ -1223,7 +1218,6 @@ static noinline ssize_t btrfs_buffered_write(struct kiocb *iocb, size_t copied; size_t dirty_sectors; size_t num_sectors; - int extents_locked; /* * Fault pages before locking them in prepare_pages @@ -1307,17 +1301,16 @@ static noinline ssize_t btrfs_buffered_write(struct kiocb *iocb, break; } - extents_locked = lock_and_cleanup_extent_if_need( - BTRFS_I(inode), pages, - num_pages, pos, write_bytes, &lockstart, - &lockend, nowait, &cached_state); - if (extents_locked < 0) { - if (!nowait && extents_locked == -EAGAIN) + ret = lock_and_cleanup_extent(BTRFS_I(inode), pages, + num_pages, pos, write_bytes, + &lockstart, &lockend, nowait, + &cached_state); + if (ret < 0) { + if (!nowait && ret == -EAGAIN) goto again; btrfs_delalloc_release_extents(BTRFS_I(inode), reserve_bytes); - ret = extents_locked; break; } @@ -1370,18 +1363,8 @@ static noinline ssize_t btrfs_buffered_write(struct kiocb *iocb, dirty_pages, pos, copied, &cached_state, only_release_metadata); - /* - * If we have not locked the extent range, because the range's - * start offset is >= i_size, we might still have a non-NULL - * cached extent state, acquired while marking the extent range - * as delalloc through btrfs_dirty_pages(). Therefore free any - * possible cached extent state to avoid a memory leak. - */ - if (extents_locked) - unlock_extent(&BTRFS_I(inode)->io_tree, lockstart, - lockend, &cached_state); - else - free_extent_state(cached_state); + unlock_extent(&BTRFS_I(inode)->io_tree, lockstart, lockend, + &cached_state); btrfs_delalloc_release_extents(BTRFS_I(inode), reserve_bytes); if (ret) { @@ -2511,7 +2494,8 @@ int btrfs_replace_file_extents(struct btrfs_inode *inode, inode_inc_iversion(&inode->vfs_inode); if (!extent_info || extent_info->update_times) - inode->vfs_inode.i_mtime = inode_set_ctime_current(&inode->vfs_inode); + inode_set_mtime_to_ts(&inode->vfs_inode, + inode_set_ctime_current(&inode->vfs_inode)); ret = btrfs_update_inode(trans, inode); if (unlikely(ret)) { @@ -2754,7 +2738,7 @@ static int btrfs_punch_hole(struct file *file, loff_t offset, loff_t len) ASSERT(trans != NULL); inode_inc_iversion(inode); - inode->i_mtime = inode_set_ctime_current(inode); + inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode)); ret = btrfs_update_inode(trans, BTRFS_I(inode)); updated_inode = true; btrfs_end_transaction(trans); @@ -2774,7 +2758,7 @@ static int btrfs_punch_hole(struct file *file, loff_t offset, loff_t len) struct timespec64 now = inode_set_ctime_current(inode); inode_inc_iversion(inode); - inode->i_mtime = now; + inode_set_mtime_to_ts(inode, now); trans = btrfs_start_transaction(root, 1); if (IS_ERR(trans)) { ret = PTR_ERR(trans); diff --git a/fs/btrfs/fs.h b/fs/btrfs/fs.h index b8b9ce8921ba..88c779635483 100644 --- a/fs/btrfs/fs.h +++ b/fs/btrfs/fs.h @@ -186,6 +186,12 @@ enum { BTRFS_MOUNT_NODISCARD = (1UL << 31), }; +/* These mount options require a full read-only fs, no new transaction is allowed. */ +#define BTRFS_MOUNT_FULL_RO_MASK \ + (BTRFS_MOUNT_NOLOGREPLAY | \ + BTRFS_MOUNT_IGNOREBADROOTS | \ + BTRFS_MOUNT_IGNOREDATACSUMS) + /* * Compat flags that we support. If any incompat flags are set other than the * ones specified below then we will fail to mount @@ -942,6 +948,15 @@ do { \ btrfs_clear_opt(fs_info->mount_opt, opt); \ } while (0) +static inline bool btrfs_is_full_ro(const struct btrfs_fs_info *fs_info) +{ + if (!sb_rdonly(fs_info->sb)) + return false; + if (unlikely(fs_info->mount_opt & BTRFS_MOUNT_FULL_RO_MASK)) + return true; + return false; +} + static inline int btrfs_fs_closing(struct btrfs_fs_info *fs_info) { /* Do it this way so we only ever do one test_bit in the normal case. */ diff --git a/fs/btrfs/inode.c b/fs/btrfs/inode.c index d1bc1901edb0..c8034d86f844 100644 --- a/fs/btrfs/inode.c +++ b/fs/btrfs/inode.c @@ -1368,6 +1368,17 @@ static noinline int cow_file_range(struct btrfs_inode *inode, alloc_hint = get_extent_allocation_hint(inode, start, num_bytes); + /* + * We're not doing compressed IO, don't unlock the first page (which + * the caller expects to stay locked), don't clear any dirty bits and + * don't set any writeback bits. + * + * Do set the Ordered (Private2) bit so we know this page was properly + * setup for writepage. + */ + page_ops = (keep_locked ? 0 : PAGE_UNLOCK); + page_ops |= PAGE_SET_ORDERED; + /* * Relocation relies on the relocated extents to have exactly the same * size as the original extents. Normally writeback for relocation data @@ -1411,8 +1422,13 @@ static noinline int cow_file_range(struct btrfs_inode *inode, continue; } if (done_offset) { - *done_offset = start - 1; - return 0; + /* + * Move @end to the end of the processed range, + * and exit the loop to unlock the processed extents. + */ + end = start - 1; + ret = 0; + break; } ret = -ENOSPC; } @@ -1468,21 +1484,6 @@ static noinline int cow_file_range(struct btrfs_inode *inode, btrfs_dec_block_group_reservations(fs_info, ins.objectid); - /* - * We're not doing compressed IO, don't unlock the first page - * (which the caller expects to stay locked), don't clear any - * dirty bits and don't set any writeback bits - * - * Do set the Ordered (Private2) bit so we know this page was - * properly setup for writepage. - */ - page_ops = (keep_locked ? 0 : PAGE_UNLOCK); - page_ops |= PAGE_SET_ORDERED; - - extent_clear_unlock_delalloc(inode, start, start + ram_size - 1, - locked_page, - EXTENT_LOCKED | EXTENT_DELALLOC, - page_ops); if (num_bytes < cur_alloc_size) num_bytes = 0; else @@ -1499,6 +1500,8 @@ static noinline int cow_file_range(struct btrfs_inode *inode, if (ret) goto out_unlock; } + extent_clear_unlock_delalloc(inode, orig_start, end, locked_page, + EXTENT_LOCKED | EXTENT_DELALLOC, page_ops); done: if (done_offset) *done_offset = end; @@ -1519,29 +1522,31 @@ static noinline int cow_file_range(struct btrfs_inode *inode, * We process each region below. */ - clear_bits = EXTENT_LOCKED | EXTENT_DELALLOC | EXTENT_DELALLOC_NEW | - EXTENT_DEFRAG | EXTENT_CLEAR_META_RESV; - page_ops = PAGE_UNLOCK | PAGE_START_WRITEBACK | PAGE_END_WRITEBACK; - /* * For the range (1). We have already instantiated the ordered extents * for this region. They are cleaned up by * btrfs_cleanup_ordered_extents() in e.g, - * btrfs_run_delalloc_range(). EXTENT_LOCKED | EXTENT_DELALLOC are - * already cleared in the above loop. And, EXTENT_DELALLOC_NEW | - * EXTENT_DEFRAG | EXTENT_CLEAR_META_RESV are handled by the cleanup - * function. + * btrfs_run_delalloc_range(). + * EXTENT_DELALLOC_NEW | EXTENT_DEFRAG | EXTENT_CLEAR_META_RESV + * are also handled by the cleanup function. * - * However, in case of @keep_locked, we still need to unlock the pages - * (except @locked_page) to ensure all the pages are unlocked. + * So here we only clear EXTENT_LOCKED and EXTENT_DELALLOC flag, and + * finish the writeback of the involved folios, which will be never submitted. */ - if (keep_locked && orig_start < start) { + if (orig_start < start) { + clear_bits = EXTENT_LOCKED | EXTENT_DELALLOC; + page_ops = PAGE_UNLOCK | PAGE_START_WRITEBACK | PAGE_END_WRITEBACK; + if (!locked_page) mapping_set_error(inode->vfs_inode.i_mapping, ret); extent_clear_unlock_delalloc(inode, orig_start, start - 1, - locked_page, 0, page_ops); + locked_page, clear_bits, page_ops); } + clear_bits = EXTENT_LOCKED | EXTENT_DELALLOC | EXTENT_DELALLOC_NEW | + EXTENT_DEFRAG | EXTENT_CLEAR_META_RESV; + page_ops = PAGE_UNLOCK | PAGE_START_WRITEBACK | PAGE_END_WRITEBACK; + /* * For the range (2). If we reserved an extent for our delalloc range * (or a subrange) and failed to create the respective ordered extent, @@ -1968,6 +1973,53 @@ static int can_nocow_file_extent(struct btrfs_path *path, return ret < 0 ? ret : can_nocow; } +/* + * Cleanup the dirty folios which will never be submitted due to error. + * + * When running a delalloc range, we may need to split the ranges (due to + * fragmentation or NOCOW). If we hit an error in the later part, we will error + * out and previously successfully executed range will never be submitted, thus + * we have to cleanup those folios by clearing their dirty flag, starting and + * finishing the writeback. + */ +static void cleanup_dirty_folios(struct btrfs_inode *inode, + struct page *locked_page, + u64 start, u64 end, int error) +{ + struct btrfs_fs_info *fs_info = inode->root->fs_info; + struct address_space *mapping = inode->vfs_inode.i_mapping; + pgoff_t start_index = start >> PAGE_SHIFT; + pgoff_t end_index = end >> PAGE_SHIFT; + u32 len; + + ASSERT(end + 1 - start < U32_MAX); + ASSERT(IS_ALIGNED(start, fs_info->sectorsize) && + IS_ALIGNED(end + 1, fs_info->sectorsize)); + len = end + 1 - start; + + /* + * Handle the locked page first. + * The btrfs_page_clamp_*() helpers can handle range out of the page case. + */ + btrfs_page_clamp_finish_io(fs_info, locked_page, start, len); + + for (pgoff_t index = start_index; index <= end_index; index++) { + struct page *page; + + /* Already handled at the beginning. */ + if (index == locked_page->index) + continue; + page = pagecache_get_page(mapping, index, FGP_LOCK, GFP_NOFS); + /* Cache already dropped, no need to do any cleanup. */ + if (!page) + continue; + btrfs_page_clamp_finish_io(fs_info, page, start, len); + unlock_page(page); + put_page(page); + } + mapping_set_error(mapping, error); +} + /* * when nowcow writeback call back. This checks for snapshots or COW copies * of the extents that exist in the file, and COWs the file as required. @@ -1983,6 +2035,11 @@ static noinline int run_delalloc_nocow(struct btrfs_inode *inode, struct btrfs_root *root = inode->root; struct btrfs_path *path; u64 cow_start = (u64)-1; + /* + * If not 0, represents the inclusive end of the last fallback_to_cow() + * range. Only for error handling. + */ + u64 cow_end = 0; u64 cur_offset = start; int ret; bool check_prev = true; @@ -2143,6 +2200,7 @@ static noinline int run_delalloc_nocow(struct btrfs_inode *inode, cow_start, found_key.offset - 1); cow_start = (u64)-1; if (ret) { + cow_end = found_key.offset - 1; btrfs_dec_nocow_writers(nocow_bg); goto error; } @@ -2219,24 +2277,59 @@ static noinline int run_delalloc_nocow(struct btrfs_inode *inode, cow_start = cur_offset; if (cow_start != (u64)-1) { - cur_offset = end; ret = fallback_to_cow(inode, locked_page, cow_start, end); cow_start = (u64)-1; - if (ret) + if (ret) { + cow_end = end; goto error; + } } btrfs_free_path(path); return 0; error: + /* + * There are several error cases: + * + * 1) Failed without falling back to COW + * start cur_offset end + * |/////////////| | + * + * For range [start, cur_offset) the folios are already unlocked (except + * @locked_page), EXTENT_DELALLOC already removed. + * Only need to clear the dirty flag as they will never be submitted. + * Ordered extent and extent maps are handled by + * btrfs_mark_ordered_io_finished() inside run_delalloc_range(). + * + * 2) Failed with error from fallback_to_cow() + * start cur_offset cow_end end + * |/////////////|-----------| | + * + * For range [start, cur_offset) it's the same as case 1). + * But for range [cur_offset, cow_end), the folios have dirty flag + * cleared and unlocked, EXTENT_DEALLLOC cleared by cow_file_range(). + * + * Thus we should not call extent_clear_unlock_delalloc() on range + * [cur_offset, cow_end), as the folios are already unlocked. + * + * So clear the folio dirty flags for [start, cur_offset) first. + */ + if (cur_offset > start) + cleanup_dirty_folios(inode, locked_page, start, cur_offset - 1, ret); + /* * If an error happened while a COW region is outstanding, cur_offset - * needs to be reset to cow_start to ensure the COW region is unlocked - * as well. + * needs to be reset to @cow_end + 1 to skip the COW range, as + * cow_file_range() will do the proper cleanup at error. + */ + if (cow_end) + cur_offset = cow_end + 1; + + /* + * We need to lock the extent here because we're clearing DELALLOC and + * we're not locked at this point. */ - if (cow_start != (u64)-1) - cur_offset = cow_start; if (cur_offset < end) { extent_clear_unlock_delalloc(inode, cur_offset, end, locked_page, EXTENT_LOCKED | @@ -3845,11 +3938,11 @@ static int btrfs_read_locked_inode(struct inode *inode, btrfs_inode_set_file_extent_range(BTRFS_I(inode), 0, round_up(i_size_read(inode), fs_info->sectorsize)); - inode->i_atime.tv_sec = btrfs_timespec_sec(leaf, &inode_item->atime); - inode->i_atime.tv_nsec = btrfs_timespec_nsec(leaf, &inode_item->atime); + inode_set_atime(inode, btrfs_timespec_sec(leaf, &inode_item->atime), + btrfs_timespec_nsec(leaf, &inode_item->atime)); - inode->i_mtime.tv_sec = btrfs_timespec_sec(leaf, &inode_item->mtime); - inode->i_mtime.tv_nsec = btrfs_timespec_nsec(leaf, &inode_item->mtime); + inode_set_mtime(inode, btrfs_timespec_sec(leaf, &inode_item->mtime), + btrfs_timespec_nsec(leaf, &inode_item->mtime)); inode_set_ctime(inode, btrfs_timespec_sec(leaf, &inode_item->ctime), btrfs_timespec_nsec(leaf, &inode_item->ctime)); @@ -4011,19 +4104,19 @@ static void fill_inode_item(struct btrfs_trans_handle *trans, btrfs_set_token_inode_nlink(&token, item, inode->i_nlink); btrfs_set_token_timespec_sec(&token, &item->atime, - inode->i_atime.tv_sec); + inode_get_atime_sec(inode)); btrfs_set_token_timespec_nsec(&token, &item->atime, - inode->i_atime.tv_nsec); + inode_get_atime_nsec(inode)); btrfs_set_token_timespec_sec(&token, &item->mtime, - inode->i_mtime.tv_sec); + inode_get_mtime_sec(inode)); btrfs_set_token_timespec_nsec(&token, &item->mtime, - inode->i_mtime.tv_nsec); + inode_get_mtime_nsec(inode)); btrfs_set_token_timespec_sec(&token, &item->ctime, - inode_get_ctime(inode).tv_sec); + inode_get_ctime_sec(inode)); btrfs_set_token_timespec_nsec(&token, &item->ctime, - inode_get_ctime(inode).tv_nsec); + inode_get_ctime_nsec(inode)); btrfs_set_token_timespec_sec(&token, &item->otime, BTRFS_I(inode)->i_otime_sec); btrfs_set_token_timespec_nsec(&token, &item->otime, BTRFS_I(inode)->i_otime_nsec); @@ -4220,8 +4313,7 @@ static int __btrfs_unlink_inode(struct btrfs_trans_handle *trans, inode_inc_iversion(&inode->vfs_inode); inode_set_ctime_current(&inode->vfs_inode); inode_inc_iversion(&dir->vfs_inode); - inode_set_ctime_current(&inode->vfs_inode); - dir->vfs_inode.i_mtime = inode_set_ctime_current(&dir->vfs_inode); + inode_set_mtime_to_ts(&dir->vfs_inode, inode_set_ctime_current(&dir->vfs_inode)); ret = btrfs_update_inode(trans, dir); out: return ret; @@ -4391,7 +4483,7 @@ static int btrfs_unlink_subvol(struct btrfs_trans_handle *trans, btrfs_i_size_write(dir, dir->vfs_inode.i_size - fname.disk_name.len * 2); inode_inc_iversion(&dir->vfs_inode); - dir->vfs_inode.i_mtime = inode_set_ctime_current(&dir->vfs_inode); + inode_set_mtime_to_ts(&dir->vfs_inode, inode_set_ctime_current(&dir->vfs_inode)); ret = btrfs_update_inode_fallback(trans, dir); if (ret) btrfs_abort_transaction(trans, ret); @@ -5060,7 +5152,8 @@ static int btrfs_setsize(struct inode *inode, struct iattr *attr) if (newsize != oldsize) { inode_inc_iversion(inode); if (!(mask & (ATTR_CTIME | ATTR_MTIME))) { - inode->i_mtime = inode_set_ctime_current(inode); + inode_set_mtime_to_ts(inode, + inode_set_ctime_current(inode)); } } @@ -5704,10 +5797,10 @@ static struct inode *new_simple_dir(struct inode *dir, inode->i_opflags &= ~IOP_XATTR; inode->i_fop = &simple_dir_operations; inode->i_mode = S_IFDIR | S_IRUGO | S_IWUSR | S_IXUGO; - inode->i_mtime = inode_set_ctime_current(inode); - inode->i_atime = dir->i_atime; - BTRFS_I(inode)->i_otime_sec = inode->i_mtime.tv_sec; - BTRFS_I(inode)->i_otime_nsec = inode->i_mtime.tv_nsec; + inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode)); + inode_set_atime_to_ts(inode, inode_get_atime(dir)); + BTRFS_I(inode)->i_otime_sec = inode_get_mtime_sec(inode); + BTRFS_I(inode)->i_otime_nsec = inode_get_mtime_nsec(inode); inode->i_uid = dir->i_uid; inode->i_gid = dir->i_gid; @@ -6401,10 +6494,9 @@ int btrfs_create_new_inode(struct btrfs_trans_handle *trans, goto discard; } - inode->i_mtime = inode_set_ctime_current(inode); - inode->i_atime = inode->i_mtime; - BTRFS_I(inode)->i_otime_sec = inode->i_mtime.tv_sec; - BTRFS_I(inode)->i_otime_nsec = inode->i_mtime.tv_nsec; + simple_inode_init_ts(inode); + BTRFS_I(inode)->i_otime_sec = inode_get_mtime_sec(inode); + BTRFS_I(inode)->i_otime_nsec = inode_get_mtime_nsec(inode); /* * We're going to fill the inode item now, so at this point the inode @@ -6569,8 +6661,8 @@ int btrfs_add_link(struct btrfs_trans_handle *trans, * values (the ones it had when the fsync was done). */ if (!test_bit(BTRFS_FS_LOG_RECOVERING, &root->fs_info->flags)) - parent_inode->vfs_inode.i_mtime = - inode_set_ctime_current(&parent_inode->vfs_inode); + inode_set_mtime_to_ts(&parent_inode->vfs_inode, + inode_set_ctime_current(&parent_inode->vfs_inode)); ret = btrfs_update_inode(trans, parent_inode); if (ret) @@ -7463,17 +7555,24 @@ static int btrfs_get_blocks_direct_write(struct extent_map **map, len, orig_block_len, ram_bytes, type); btrfs_dec_nocow_writers(bg); - if (type == BTRFS_ORDERED_PREALLOC) { - free_extent_map(em); - *map = em2; - em = em2; - } - if (IS_ERR(em2)) { ret = PTR_ERR(em2); + free_extent_map(em); + *map = NULL; goto out; } + /* + * True NOCOW writes don't need to create a new extent map, + * while PREALLOC writes must replace the existing one. + */ + if (em2) { + ASSERT(type == BTRFS_ORDERED_PREALLOC); + free_extent_map(em); + *map = em2; + em = em2; + } + dio_data->nocow_done = true; } else { /* Our caller expects us to free the input extent map. */ diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c index 8a5216affade..e4df4f5daefd 100644 --- a/fs/btrfs/ioctl.c +++ b/fs/btrfs/ioctl.c @@ -3779,7 +3779,8 @@ static long btrfs_ioctl_qgroup_assign(struct file *file, void __user *arg) goto drop_write; } - trans = btrfs_join_transaction(root); + /* 2 BTRFS_QGROUP_RELATION_KEY items. */ + trans = btrfs_start_transaction(root, 2); if (IS_ERR(trans)) { ret = PTR_ERR(trans); goto out; @@ -3841,7 +3842,11 @@ static long btrfs_ioctl_qgroup_create(struct file *file, void __user *arg) goto out; } - trans = btrfs_join_transaction(root); + /* + * 1 BTRFS_QGROUP_INFO_KEY item. + * 1 BTRFS_QGROUP_LIMIT_KEY item. + */ + trans = btrfs_start_transaction(root, 2); if (IS_ERR(trans)) { ret = PTR_ERR(trans); goto out; @@ -3887,7 +3892,8 @@ static long btrfs_ioctl_qgroup_limit(struct file *file, void __user *arg) goto drop_write; } - trans = btrfs_join_transaction(root); + /* 1 BTRFS_QGROUP_LIMIT_KEY item. */ + trans = btrfs_start_transaction(root, 1); if (IS_ERR(trans)) { ret = PTR_ERR(trans); goto out; diff --git a/fs/btrfs/lzo.c b/fs/btrfs/lzo.c index d3fcfc628a4f..222aeb40b26d 100644 --- a/fs/btrfs/lzo.c +++ b/fs/btrfs/lzo.c @@ -392,6 +392,17 @@ int lzo_decompress_bio(struct list_head *ws, struct compressed_bio *cb) return -EIO; } + /* The segment must not extend beyond the compressed input. */ + if (unlikely(cur_in + seg_len > cb->compressed_len)) { + struct btrfs_inode *inode = cb->bbio.inode; + + btrfs_err(fs_info, + "lzo segment overflows compressed input, root %llu inode %llu offset %llu cur_in %u len %u compressed len %u", + btrfs_root_id(inode->root), btrfs_ino(inode), + cb->start, cur_in, seg_len, cb->compressed_len); + return -EUCLEAN; + } + /* Copy the compressed segment payload into workspace */ copy_compressed_segment(cb, workspace->cbuf, seg_len, &cur_in); @@ -436,7 +447,7 @@ int lzo_decompress(struct list_head *ws, const u8 *data_in, char *kaddr; unsigned long bytes; - if (srclen < LZO_LEN || srclen > max_segment_len + LZO_LEN * 2) + if (srclen <= LZO_LEN * 2 || srclen > max_segment_len + LZO_LEN * 2) return -EUCLEAN; in_len = read_compress_length(data_in); diff --git a/fs/btrfs/qgroup.c b/fs/btrfs/qgroup.c index 622febdb61e2..e4c4d369eb8d 100644 --- a/fs/btrfs/qgroup.c +++ b/fs/btrfs/qgroup.c @@ -3897,12 +3897,13 @@ static int qgroup_free_reserved_data(struct btrfs_inode *inode, struct ulist_node *unode; struct ulist_iterator uiter; struct extent_changeset changeset; + const u32 sectorsize = root->fs_info->sectorsize; + const u64 aligned_start = round_down(start, sectorsize); + const u64 aligned_len = round_up(start + len, sectorsize) - aligned_start; u64 freed = 0; int ret; extent_changeset_init(&changeset); - len = round_up(start + len, root->fs_info->sectorsize); - start = round_down(start, root->fs_info->sectorsize); ULIST_ITER_INIT(&uiter); while ((unode = ulist_next(&reserved->range_changed, &uiter))) { @@ -3914,12 +3915,15 @@ static int qgroup_free_reserved_data(struct btrfs_inode *inode, extent_changeset_release(&changeset); - /* Only free range in range [start, start + len) */ - if (range_start >= start + len || - range_start + range_len <= start) + /* + * Only free the range within + * [aligned_start, aligned_start + aligned_len). + */ + if (range_start >= aligned_start + aligned_len || + range_start + range_len <= aligned_start) continue; - free_start = max(range_start, start); - free_len = min(start + len, range_start + range_len) - + free_start = max(range_start, aligned_start); + free_len = min(aligned_start + aligned_len, range_start + range_len) - free_start; /* * TODO: To also modify reserved->ranges_reserved to reflect diff --git a/fs/btrfs/reflink.c b/fs/btrfs/reflink.c index 60e78c446b38..30dedd7c6b7f 100644 --- a/fs/btrfs/reflink.c +++ b/fs/btrfs/reflink.c @@ -29,7 +29,7 @@ static int clone_finish_inode_update(struct btrfs_trans_handle *trans, inode_inc_iversion(inode); if (!no_time_update) { - inode->i_mtime = inode_set_ctime_current(inode); + inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode)); } /* * We round up to the block size at eof when determining which @@ -179,12 +179,10 @@ static int clone_copy_inline_extent(struct inode *dst, struct btrfs_drop_extents_args drop_args = { 0 }; int ret; struct btrfs_key key; + bool copied_inline_to_page = false; - if (new_key->offset > 0) { - ret = copy_inline_to_page(BTRFS_I(dst), new_key->offset, - inline_data, size, datal, comp_type); - goto out; - } + if (new_key->offset > 0) + goto copy_to_page; key.objectid = btrfs_ino(BTRFS_I(dst)); key.type = BTRFS_EXTENT_DATA_KEY; @@ -282,6 +280,60 @@ static int clone_copy_inline_extent(struct inode *dst, ret = btrfs_inode_set_file_extent_range(BTRFS_I(dst), 0, aligned_end); out: if (!ret && !trans) { + if (copied_inline_to_page && + new_key->offset + datal > i_size_read(dst)) { + /* + * If we copied the inline extent data to a page/folio + * beyond the i_size of the destination inode, then we + * need to increase the i_size before we start a + * transaction to update the inode item. This is to + * prevent a deadlock when the flushoncommit mount + * option is used, which happens like this: + * + * 1) Task A clones an inline extent from inode X to an + * offset of inode Y that is beyond Y's current + * i_size. This means we copied the inline extent's + * data to a folio of inode Y that is beyond its EOF, + * using the call above to copy_inline_to_page(); + * + * 2) Task B starts a transaction commit and calls + * btrfs_start_delalloc_flush() to flush delalloc; + * + * 3) The delalloc flushing sees the new dirty folio of + * inode Y and when it attempts to flush it, it ends + * up at extent_writepage() and sees that the offset + * of the folio is beyond the i_size of inode Y, so + * it attempts to invalidate the folio by calling + * folio_invalidate(), which ends up at btrfs' folio + * invalidate callback - btrfs_invalidate_folio(). + * There it tries to lock the folio's range in inode + * Y's extent io tree, but it blocks since it's + * currently locked by task A - during reflink we + * lock the inodes and the source and destination + * ranges after flushing all delalloc and waiting for + * ordered extent completion - after that we don't + * expect to have dirty folios in the ranges, the + * exception is if we have to copy an inline extent's + * data (because the destination offset is not zero); + * + * 4) Task A then does the 'goto out' below and attempts + * to start a transaction to update the inode item, + * and then it's blocked since the current + * transaction is in the TRANS_STATE_COMMIT_START + * state. Therefore task A has to wait for the + * current transaction to become unblocked (its + * state >= TRANS_STATE_UNBLOCKED). + * + * This leads to a deadlock - the task committing the + * transaction waiting for the delalloc flushing which + * is blocked during folio invalidation on the inode's + * extent lock and the reflink task waiting for the + * current transaction to be unblocked so that it can + * start a new one to update the inode item (while + * holding the extent lock). + */ + i_size_write(dst, new_key->offset + datal); + } /* * No transaction here means we copied the inline extent into a * page of the destination inode. @@ -316,6 +368,8 @@ static int clone_copy_inline_extent(struct inode *dst, ret = copy_inline_to_page(BTRFS_I(dst), new_key->offset, inline_data, size, datal, comp_type); + copied_inline_to_page = (ret == 0); + goto out; } diff --git a/fs/btrfs/relocation.c b/fs/btrfs/relocation.c index fd9e674c1346..d5efca405085 100644 --- a/fs/btrfs/relocation.c +++ b/fs/btrfs/relocation.c @@ -4188,6 +4188,24 @@ static noinline_for_stack int mark_garbage_root(struct btrfs_root *root) return ret; } +static void release_recovered_fs_roots(struct list_head *roots, bool drop_reloc_refs) +{ + struct btrfs_root *root; + struct btrfs_root *next; + + list_for_each_entry_safe(root, next, roots, reloc_dirty_list) { + list_del_init(&root->reloc_dirty_list); + if (drop_reloc_refs) { + struct btrfs_root *reloc_root = root->reloc_root; + + ASSERT(reloc_root); + root->reloc_root = NULL; + btrfs_put_root(reloc_root); + } + btrfs_put_root(root); + } +} + /* * recover relocation interrupted by system crash. * @@ -4197,6 +4215,7 @@ static noinline_for_stack int mark_garbage_root(struct btrfs_root *root) int btrfs_recover_relocation(struct btrfs_fs_info *fs_info) { LIST_HEAD(reloc_roots); + LIST_HEAD(recovered_roots); struct btrfs_key key; struct btrfs_root *fs_root; struct btrfs_root *reloc_root; @@ -4204,8 +4223,8 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info) struct extent_buffer *leaf; struct reloc_control *rc = NULL; struct btrfs_trans_handle *trans; - int ret; - int err = 0; + int ret2; + int ret = 0; path = btrfs_alloc_path(); if (!path) @@ -4219,15 +4238,14 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info) while (1) { ret = btrfs_search_slot(NULL, fs_info->tree_root, &key, path, 0, 0); - if (ret < 0) { - err = ret; + if (ret < 0) goto out; - } if (ret > 0) { if (path->slots[0] == 0) break; path->slots[0]--; } + ret = 0; leaf = path->nodes[0]; btrfs_item_key_to_cpu(leaf, &key, path->slots[0]); btrfs_release_path(path); @@ -4238,7 +4256,7 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info) reloc_root = btrfs_read_tree_root(fs_info->tree_root, &key); if (IS_ERR(reloc_root)) { - err = PTR_ERR(reloc_root); + ret = PTR_ERR(reloc_root); goto out; } @@ -4250,15 +4268,12 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info) reloc_root->root_key.offset, false); if (IS_ERR(fs_root)) { ret = PTR_ERR(fs_root); - if (ret != -ENOENT) { - err = ret; + if (ret != -ENOENT) goto out; - } ret = mark_garbage_root(reloc_root); - if (ret < 0) { - err = ret; + if (ret < 0) goto out; - } + ret = 0; } else { btrfs_put_root(fs_root); } @@ -4276,15 +4291,13 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info) rc = alloc_reloc_control(fs_info); if (!rc) { - err = -ENOMEM; + ret = -ENOMEM; goto out; } ret = reloc_chunk_start(fs_info); - if (ret < 0) { - err = ret; + if (ret < 0) goto out_end; - } rc->extent_root = btrfs_extent_root(fs_info, 0); @@ -4292,7 +4305,7 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info) trans = btrfs_join_transaction(rc->extent_root); if (IS_ERR(trans)) { - err = PTR_ERR(trans); + ret = PTR_ERR(trans); goto out_unset; } @@ -4312,27 +4325,29 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info) fs_root = btrfs_get_fs_root(fs_info, reloc_root->root_key.offset, false); if (IS_ERR(fs_root)) { - err = PTR_ERR(fs_root); + ret = PTR_ERR(fs_root); list_add_tail(&reloc_root->root_list, &reloc_roots); btrfs_end_transaction(trans); - goto out_unset; + goto out_drop_reloc_refs; } - err = __add_reloc_root(reloc_root); - ASSERT(err != -EEXIST); - if (err) { + ret = __add_reloc_root(reloc_root); + ASSERT(ret != -EEXIST); + if (ret) { list_add_tail(&reloc_root->root_list, &reloc_roots); btrfs_put_root(fs_root); btrfs_end_transaction(trans); - goto out_unset; + goto out_drop_reloc_refs; } + ASSERT(list_empty(&fs_root->reloc_dirty_list)); fs_root->reloc_root = btrfs_grab_root(reloc_root); - btrfs_put_root(fs_root); + list_add_tail(&fs_root->reloc_dirty_list, &recovered_roots); } - err = btrfs_commit_transaction(trans); - if (err) - goto out_unset; + ret = btrfs_commit_transaction(trans); + if (ret) + goto out_drop_reloc_refs; + release_recovered_fs_roots(&recovered_roots, false); merge_reloc_roots(rc); @@ -4340,14 +4355,16 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info) trans = btrfs_join_transaction(rc->extent_root); if (IS_ERR(trans)) { - err = PTR_ERR(trans); + ret = PTR_ERR(trans); goto out_clean; } - err = btrfs_commit_transaction(trans); + ret = btrfs_commit_transaction(trans); out_clean: - ret = clean_dirty_subvols(rc); - if (ret < 0 && !err) - err = ret; + ret2 = clean_dirty_subvols(rc); + if (ret2 < 0 && !ret) + ret = ret2; +out_drop_reloc_refs: + release_recovered_fs_roots(&recovered_roots, true); out_unset: unset_reloc_control(rc); reloc_chunk_end(fs_info); @@ -4358,14 +4375,14 @@ int btrfs_recover_relocation(struct btrfs_fs_info *fs_info) btrfs_free_path(path); - if (err == 0) { + if (ret == 0) { /* cleanup orphan inode in data relocation tree */ fs_root = btrfs_grab_root(fs_info->data_reloc_root); ASSERT(fs_root); - err = btrfs_orphan_cleanup(fs_root); + ret = btrfs_orphan_cleanup(fs_root); btrfs_put_root(fs_root); } - return err; + return ret; } /* diff --git a/fs/btrfs/scrub.c b/fs/btrfs/scrub.c index d2d2548eea05..0da80e65d798 100644 --- a/fs/btrfs/scrub.c +++ b/fs/btrfs/scrub.c @@ -446,8 +446,10 @@ static int scrub_print_warning_inode(u64 inum, u64 offset, u64 num_bytes, } ret = paths_from_inode(inum, ipath); - if (ret < 0) + if (ret < 0) { + btrfs_put_root(local_root); goto err; + } /* * we deliberately ignore the bit ipath might have been too small to diff --git a/fs/btrfs/space-info.c b/fs/btrfs/space-info.c index a50f0db75633..e786078402f8 100644 --- a/fs/btrfs/space-info.c +++ b/fs/btrfs/space-info.c @@ -601,18 +601,6 @@ static inline u64 calc_reclaim_items_nr(const struct btrfs_fs_info *fs_info, return nr; } -static inline u64 calc_delayed_refs_nr(const struct btrfs_fs_info *fs_info, - u64 to_reclaim) -{ - const u64 bytes = btrfs_calc_delayed_ref_bytes(fs_info, 1); - u64 nr; - - nr = div64_u64(to_reclaim, bytes); - if (!nr) - nr = 1; - return nr; -} - #define EXTENT_SIZE_PER_ITEM SZ_256K /* @@ -794,10 +782,9 @@ static void flush_space(struct btrfs_fs_info *fs_info, break; } if (state == FLUSH_DELAYED_REFS_NR) - nr = calc_delayed_refs_nr(fs_info, num_bytes); + btrfs_run_delayed_refs(trans, num_bytes); else - nr = 0; - btrfs_run_delayed_refs(trans, nr); + btrfs_run_delayed_refs(trans, 0); btrfs_end_transaction(trans); break; case ALLOC_CHUNK: diff --git a/fs/btrfs/subpage.h b/fs/btrfs/subpage.h index 5cbf67ccbdeb..b2cfc48c80b0 100644 --- a/fs/btrfs/subpage.h +++ b/fs/btrfs/subpage.h @@ -145,6 +145,19 @@ DECLARE_BTRFS_SUBPAGE_OPS(writeback); DECLARE_BTRFS_SUBPAGE_OPS(ordered); DECLARE_BTRFS_SUBPAGE_OPS(checked); +/* + * Helper for error cleanup, where a page will have its dirty flag cleared, + * with writeback started and finished. + */ +static inline void btrfs_page_clamp_finish_io(struct btrfs_fs_info *fs_info, + struct page *locked_page, + u64 start, u32 len) +{ + btrfs_page_clamp_clear_dirty(fs_info, locked_page, start, len); + btrfs_page_clamp_set_writeback(fs_info, locked_page, start, len); + btrfs_page_clamp_clear_writeback(fs_info, locked_page, start, len); +} + bool btrfs_subpage_clear_and_test_dirty(const struct btrfs_fs_info *fs_info, struct page *page, u64 start, u32 len); diff --git a/fs/btrfs/transaction.c b/fs/btrfs/transaction.c index 8f14dc4691e4..769dfed15f00 100644 --- a/fs/btrfs/transaction.c +++ b/fs/btrfs/transaction.c @@ -164,7 +164,13 @@ void btrfs_put_transaction(struct btrfs_transaction *transaction) cache = list_first_entry(&transaction->deleted_bgs, struct btrfs_block_group, bg_list); + /* + * Not strictly necessary to lock, as no other task will be using a + * block_group on the deleted_bgs list during a transaction abort. + */ + spin_lock(&transaction->fs_info->unused_bgs_lock); list_del_init(&cache->bg_list); + spin_unlock(&transaction->fs_info->unused_bgs_lock); btrfs_unfreeze_block_group(cache); btrfs_put_block_group(cache); } @@ -1235,14 +1241,13 @@ static int btrfs_write_and_wait_transaction(struct btrfs_trans_handle *trans) blk_finish_plug(&plug); ret2 = btrfs_wait_extents(fs_info, dirty_pages); - extent_io_tree_release(&trans->transaction->dirty_pages); - if (ret) return ret; - else if (ret2) + if (ret2) return ret2; - else - return 0; + + extent_io_tree_release(&trans->transaction->dirty_pages); + return 0; } /* @@ -1346,7 +1351,7 @@ static noinline int commit_cowonly_roots(struct btrfs_trans_handle *trans) } /* Now flush any delayed refs generated by updating all of the roots */ - ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); + ret = btrfs_run_delayed_refs(trans, U64_MAX); if (ret) return ret; @@ -1361,7 +1366,7 @@ static noinline int commit_cowonly_roots(struct btrfs_trans_handle *trans) * so we want to keep this flushing in this loop to make sure * everything gets run. */ - ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); + ret = btrfs_run_delayed_refs(trans, U64_MAX); if (ret) return ret; } @@ -1579,7 +1584,7 @@ static int qgroup_account_snapshot(struct btrfs_trans_handle *trans, * for now flush the delayed refs to narrow the race window where the * qgroup counters could end up wrong. */ - ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); + ret = btrfs_run_delayed_refs(trans, U64_MAX); if (ret) { btrfs_abort_transaction(trans, ret); return ret; @@ -1872,7 +1877,8 @@ static noinline int create_pending_snapshot(struct btrfs_trans_handle *trans, btrfs_i_size_write(BTRFS_I(parent_inode), parent_inode->i_size + fname.disk_name.len * 2); - parent_inode->i_mtime = inode_set_ctime_current(parent_inode); + inode_set_mtime_to_ts(parent_inode, + inode_set_ctime_current(parent_inode)); ret = btrfs_update_inode_fallback(trans, BTRFS_I(parent_inode)); if (ret) { btrfs_abort_transaction(trans, ret); @@ -2112,8 +2118,14 @@ static void btrfs_cleanup_pending_block_groups(struct btrfs_trans_handle *trans) struct btrfs_block_group *block_group, *tmp; list_for_each_entry_safe(block_group, tmp, &trans->new_bgs, bg_list) { - btrfs_delayed_refs_rsv_release(fs_info, 1); + btrfs_dec_delayed_refs_rsv_bg_inserts(fs_info); + /* + * Not strictly necessary to lock, as no other task will be using a + * block_group on the new_bgs list during a transaction abort. + */ + spin_lock(&fs_info->unused_bgs_lock); list_del_init(&block_group->bg_list); + spin_unlock(&fs_info->unused_bgs_lock); } } @@ -2431,7 +2443,7 @@ int btrfs_commit_transaction(struct btrfs_trans_handle *trans) if (ret) goto unlock_reloc; - ret = btrfs_run_delayed_refs(trans, (unsigned long)-1); + ret = btrfs_run_delayed_refs(trans, U64_MAX); if (ret) goto unlock_reloc; diff --git a/fs/btrfs/transaction.h b/fs/btrfs/transaction.h index 7623db359881..50169f1bfd80 100644 --- a/fs/btrfs/transaction.h +++ b/fs/btrfs/transaction.h @@ -129,6 +129,7 @@ struct btrfs_trans_handle { u64 bytes_reserved; u64 chunk_bytes_reserved; unsigned long delayed_ref_updates; + unsigned long delayed_ref_csum_deletions; struct btrfs_transaction *transaction; struct btrfs_block_rsv *block_rsv; struct btrfs_block_rsv *orig_rsv; diff --git a/fs/btrfs/tree-log.c b/fs/btrfs/tree-log.c index 642d35816a61..651a742f7191 100644 --- a/fs/btrfs/tree-log.c +++ b/fs/btrfs/tree-log.c @@ -4252,44 +4252,45 @@ static void fill_inode_item(struct btrfs_trans_handle *trans, struct inode *inode, int log_inode_only, u64 logged_isize) { - struct btrfs_map_token token; + u64 gen = BTRFS_I(inode)->generation; u64 flags; - btrfs_init_map_token(&token, leaf); - if (log_inode_only) { - /* set the generation to zero so the recover code - * can tell the difference between an logging - * just to say 'this inode exists' and a logging - * to say 'update this inode with these values' + /* + * Set the generation to zero so the recover code can tell the + * difference between a logging just to say 'this inode exists' + * and a logging to say 'update this inode with these values'. + * But only if the inode was not already logged before. + * We access ->logged_trans directly since it was already set + * up in the call chain by btrfs_log_inode(), and data_race() + * to avoid false alerts from KCSAN and since it was set already + * and one can set it to 0 since that only happens on eviction + * and we are holding a ref on the inode. */ - btrfs_set_token_inode_generation(&token, item, 0); - btrfs_set_token_inode_size(&token, item, logged_isize); + ASSERT(data_race(BTRFS_I(inode)->logged_trans) > 0); + if (data_race(BTRFS_I(inode)->logged_trans) < trans->transid) + gen = 0; + + btrfs_set_inode_size(leaf, item, logged_isize); } else { - btrfs_set_token_inode_generation(&token, item, - BTRFS_I(inode)->generation); - btrfs_set_token_inode_size(&token, item, inode->i_size); + btrfs_set_inode_size(leaf, item, inode->i_size); } - btrfs_set_token_inode_uid(&token, item, i_uid_read(inode)); - btrfs_set_token_inode_gid(&token, item, i_gid_read(inode)); - btrfs_set_token_inode_mode(&token, item, inode->i_mode); - btrfs_set_token_inode_nlink(&token, item, inode->i_nlink); + btrfs_set_inode_generation(leaf, item, gen); + + btrfs_set_inode_uid(leaf, item, i_uid_read(inode)); + btrfs_set_inode_gid(leaf, item, i_gid_read(inode)); + btrfs_set_inode_mode(leaf, item, inode->i_mode); + btrfs_set_inode_nlink(leaf, item, inode->i_nlink); - btrfs_set_token_timespec_sec(&token, &item->atime, - inode->i_atime.tv_sec); - btrfs_set_token_timespec_nsec(&token, &item->atime, - inode->i_atime.tv_nsec); + btrfs_set_timespec_sec(leaf, &item->atime, inode_get_atime_sec(inode)); + btrfs_set_timespec_nsec(leaf, &item->atime, inode_get_atime_nsec(inode)); - btrfs_set_token_timespec_sec(&token, &item->mtime, - inode->i_mtime.tv_sec); - btrfs_set_token_timespec_nsec(&token, &item->mtime, - inode->i_mtime.tv_nsec); + btrfs_set_timespec_sec(leaf, &item->mtime, inode_get_mtime_sec(inode)); + btrfs_set_timespec_nsec(leaf, &item->mtime, inode_get_mtime_nsec(inode)); - btrfs_set_token_timespec_sec(&token, &item->ctime, - inode_get_ctime(inode).tv_sec); - btrfs_set_token_timespec_nsec(&token, &item->ctime, - inode_get_ctime(inode).tv_nsec); + btrfs_set_timespec_sec(leaf, &item->ctime, inode_get_ctime_sec(inode)); + btrfs_set_timespec_nsec(leaf, &item->ctime, inode_get_ctime_nsec(inode)); btrfs_set_timespec_sec(leaf, &item->otime, BTRFS_I(inode)->i_otime_sec); btrfs_set_timespec_nsec(leaf, &item->otime, BTRFS_I(inode)->i_otime_nsec); @@ -4303,13 +4304,13 @@ static void fill_inode_item(struct btrfs_trans_handle *trans, * inode item in subvolume tree as needed (see overwrite_item()). */ - btrfs_set_token_inode_sequence(&token, item, inode_peek_iversion(inode)); - btrfs_set_token_inode_transid(&token, item, trans->transid); - btrfs_set_token_inode_rdev(&token, item, inode->i_rdev); + btrfs_set_inode_sequence(leaf, item, inode_peek_iversion(inode)); + btrfs_set_inode_transid(leaf, item, trans->transid); + btrfs_set_inode_rdev(leaf, item, inode->i_rdev); flags = btrfs_inode_combine_flags(BTRFS_I(inode)->flags, BTRFS_I(inode)->ro_flags); - btrfs_set_token_inode_flags(&token, item, flags); - btrfs_set_token_inode_block_group(&token, item, 0); + btrfs_set_inode_flags(leaf, item, flags); + btrfs_set_inode_block_group(leaf, item, 0); } static int log_inode_item(struct btrfs_trans_handle *trans, @@ -5081,42 +5082,63 @@ static int btrfs_log_changed_extents(struct btrfs_trans_handle *trans, return 0; } -static int logged_inode_size(struct btrfs_root *log, struct btrfs_inode *inode, - struct btrfs_path *path, u64 *size_ret) +static int get_inode_size_to_log(struct btrfs_trans_handle *trans, + struct btrfs_inode *inode, + struct btrfs_path *path, u64 *size_ret) { struct btrfs_key key; + struct btrfs_inode_item *item; int ret; key.objectid = btrfs_ino(inode); key.type = BTRFS_INODE_ITEM_KEY; key.offset = 0; - ret = btrfs_search_slot(NULL, log, &key, path, 0, 0); - if (ret < 0) { - return ret; - } else if (ret > 0) { - *size_ret = 0; - } else { - struct btrfs_inode_item *item; + /* + * Our caller called inode_logged(), so logged_trans is up to date. + * Use data_race() to silence any warning from KCSAN. Once logged_trans + * is set, it can only be reset to 0 after inode eviction. + */ + if (data_race(inode->logged_trans) == trans->transid) { + ret = btrfs_search_slot(NULL, inode->root->log_root, &key, path, 0, 0); + } else if (inode->generation < trans->transid) { + path->search_commit_root = true; + path->skip_locking = true; + ret = btrfs_search_slot(NULL, inode->root, &key, path, 0, 0); + path->search_commit_root = false; + path->skip_locking = false; - item = btrfs_item_ptr(path->nodes[0], path->slots[0], - struct btrfs_inode_item); - *size_ret = btrfs_inode_size(path->nodes[0], item); - /* - * If the in-memory inode's i_size is smaller then the inode - * size stored in the btree, return the inode's i_size, so - * that we get a correct inode size after replaying the log - * when before a power failure we had a shrinking truncate - * followed by addition of a new name (rename / new hard link). - * Otherwise return the inode size from the btree, to avoid - * data loss when replaying a log due to previously doing a - * write that expands the inode's size and logging a new name - * immediately after. - */ - if (*size_ret > inode->vfs_inode.i_size) - *size_ret = inode->vfs_inode.i_size; + } else { + *size_ret = 0; + return 0; } + /* + * If the inode was logged before or is from a past transaction, then + * its inode item must exist in the log root or in the commit root. + */ + ASSERT(ret <= 0); + if (WARN_ON_ONCE(ret > 0)) + ret = -ENOENT; + + if (ret < 0) + return ret; + + item = btrfs_item_ptr(path->nodes[0], path->slots[0], + struct btrfs_inode_item); + *size_ret = btrfs_inode_size(path->nodes[0], item); + /* + * If the in-memory inode's i_size is smaller then the inode size stored + * in the btree, return the inode's i_size, so that we get a correct + * inode size after replaying the log when before a power failure we had + * a shrinking truncate followed by addition of a new name (rename / new + * hard link). Otherwise return the inode size from the btree, to avoid + * data loss when replaying a log due to previously doing a write that + * expands the inode's size and logging a new name immediately after. + */ + if (*size_ret > inode->vfs_inode.i_size) + *size_ret = inode->vfs_inode.i_size; + btrfs_release_path(path); return 0; } @@ -6633,7 +6655,7 @@ static int btrfs_log_inode(struct btrfs_trans_handle *trans, ret = drop_inode_items(trans, log, path, inode, BTRFS_XATTR_ITEM_KEY); } else { - if (inode_only == LOG_INODE_EXISTS && ctx->logged_before) { + if (inode_only == LOG_INODE_EXISTS) { /* * Make sure the new inode item we write to the log has * the same isize as the current one (if it exists). @@ -6647,7 +6669,7 @@ static int btrfs_log_inode(struct btrfs_trans_handle *trans, * (zeroes), as if an expanding truncate happened, * instead of getting a file of 4Kb only. */ - ret = logged_inode_size(log, inode, path, &logged_isize); + ret = get_inode_size_to_log(trans, inode, path, &logged_isize); if (ret) goto out_unlock; } diff --git a/fs/btrfs/volumes.c b/fs/btrfs/volumes.c index da9b08afdddd..e371bcd14e8c 100644 --- a/fs/btrfs/volumes.c +++ b/fs/btrfs/volumes.c @@ -1459,30 +1459,158 @@ struct btrfs_device *btrfs_scan_one_device(const char *path, blk_mode_t flags) } /* - * Try to find a chunk that intersects [start, start + len] range and when one - * such is found, record the end of it in *start + * Find the first pending extent intersecting a range. + * + * @device: the device to search + * @start: start of the range to check + * @len: length of the range to check + * @pending_start: output pointer for the start of the found pending extent + * @pending_end: output pointer for the end of the found pending extent (inclusive) + * + * Search for a pending chunk allocation that intersects the half-open range + * [start, start + len). + * + * Return: true if a pending extent was found, false otherwise. + * If the return value is true, store the first pending extent in + * [*pending_start, *pending_end]. Otherwise, the two output variables + * may still be modified, to something outside the range and should not + * be used. */ -static bool contains_pending_extent(struct btrfs_device *device, u64 *start, - u64 len) +static bool first_pending_extent(struct btrfs_device *device, u64 start, u64 len, + u64 *pending_start, u64 *pending_end) { - u64 physical_start, physical_end; - lockdep_assert_held(&device->fs_info->chunk_mutex); - if (find_first_extent_bit(&device->alloc_state, *start, - &physical_start, &physical_end, + if (find_first_extent_bit(&device->alloc_state, start, + pending_start, pending_end, CHUNK_ALLOCATED, NULL)) { - if (in_range(physical_start, *start, len) || - in_range(*start, physical_start, - physical_end + 1 - physical_start)) { - *start = physical_end + 1; + if (in_range(*pending_start, start, len) || + in_range(start, *pending_start, *pending_end + 1 - *pending_start)) { return true; } } return false; } +/* + * Find the first real hole accounting for pending extents. + * + * @device: the device containing the candidate hole + * @start: input/output pointer for the hole start position + * @len: input/output pointer for the hole length + * @min_hole_size: the size of hole we are looking for + * + * Given a potential hole specified by [*start, *start + *len), check for pending + * chunk allocations within that range. If pending extents are found, the hole is + * adjusted to represent the first true free space that is large enough when + * accounting for pending chunks. + * + * Note that this function must handle various cases involving non consecutive + * pending extents. + * + * Returns: true if a suitable hole was found and false otherwise. + * If the return value is true, then *start and *len are set to represent the hole. + * If the return value is false, then *start is set to the largest hole we + * found and *len is set to its length. + * If there are no holes at all, then *start is set to the end of the range and + * *len is set to 0. + */ +static bool find_hole_in_pending_extents(struct btrfs_device *device, u64 *start, + u64 *len, u64 min_hole_size) +{ + u64 pending_start, pending_end; + u64 end; + u64 max_hole_start = 0; + u64 max_hole_len = 0; + + lockdep_assert_held(&device->fs_info->chunk_mutex); + + if (*len == 0) + return false; + + end = *start + *len - 1; + + /* + * Loop until we either see a large enough hole or check every pending + * extent overlapping the candidate hole. + * At every hole that we observe, record it if it is the new max. + * At the end of the iteration, set the output variables to the max hole. + */ + while (true) { + if (first_pending_extent(device, *start, *len, &pending_start, &pending_end)) { + /* + * Case 1: the pending extent overlaps the start of + * candidate hole. That means the true hole is after the + * pending extent, but we need to find the next pending + * extent to properly size the hole. In the next loop, + * we will reduce to case 2 or 3. + * e.g., + * + * |----pending A----| real hole |----pending B----| + * | candidate hole | + * *start end + */ + if (pending_start <= *start) { + *start = pending_end + 1; + goto next; + } + /* + * Case 2: The pending extent starts after *start (and overlaps + * [*start, end), so the first hole just goes up to the start + * of the pending extent. + * e.g., + * + * | real hole |----pending A----| + * | candidate hole | + * *start end + */ + *len = pending_start - *start; + if (*len > max_hole_len) { + max_hole_start = *start; + max_hole_len = *len; + } + if (*len >= min_hole_size) + break; + /* + * If the hole wasn't big enough, then we advance past + * the pending extent and keep looking. + */ + *start = pending_end + 1; + goto next; + } else { + /* + * Case 3: There is no pending extent overlapping the + * range [*start, *start + *len - 1], so the only remaining + * hole is the remaining range. + * e.g., + * + * | candidate hole | + * | real hole | + * *start end + */ + + if (*len > max_hole_len) { + max_hole_start = *start; + max_hole_len = *len; + } + break; + } +next: + if (*start > end) + break; + *len = end - *start + 1; + } + if (max_hole_len) { + *start = max_hole_start; + *len = max_hole_len; + } else { + *start = end + 1; + *len = 0; + } + return max_hole_len >= min_hole_size; +} + static u64 dev_extent_search_start(struct btrfs_device *device) { switch (device->fs_devices->chunk_alloc_policy) { @@ -1545,58 +1673,56 @@ static bool dev_extent_hole_check_zoned(struct btrfs_device *device, } /* - * Check if specified hole is suitable for allocation. + * Validate and adjust a hole for chunk allocation * - * @device: the device which we have the hole - * @hole_start: starting position of the hole - * @hole_size: the size of the hole - * @num_bytes: the size of the free space that we need + * @device: the device containing the candidate hole + * @hole_start: input/output pointer for the hole start position + * @hole_size: input/output pointer for the hole size + * @num_bytes: minimum allocation size required * - * This function may modify @hole_start and @hole_size to reflect the suitable - * position for allocation. Returns 1 if hole position is updated, 0 otherwise. + * Check if the specified hole is suitable for allocation and adjust it if + * necessary. The hole may be modified to skip over pending chunk allocations + * and to satisfy stricter zoned requirements on zoned filesystems. + * + * For regular (non-zoned) allocation, if the hole after adjustment is smaller + * than @num_bytes, the search continues past additional pending extents until + * either a sufficiently large hole is found or no more pending extents exist. + * + * Return: true if a suitable hole was found and false otherwise. + * If the return value is true, then *hole_start and *hole_size are set to + * represent the hole we found. + * If the return value is false, then *hole_start is set to the largest + * hole we found and *hole_size is set to its length. + * If there are no holes at all, then *hole_start is set to the end of the range + * and *hole_size is set to 0. */ static bool dev_extent_hole_check(struct btrfs_device *device, u64 *hole_start, u64 *hole_size, u64 num_bytes) { - bool changed = false; - u64 hole_end = *hole_start + *hole_size; + bool found = false; + const u64 hole_end = *hole_start + *hole_size - 1; - for (;;) { - /* - * Check before we set max_hole_start, otherwise we could end up - * sending back this offset anyway. - */ - if (contains_pending_extent(device, hole_start, *hole_size)) { - if (hole_end >= *hole_start) - *hole_size = hole_end - *hole_start; - else - *hole_size = 0; - changed = true; - } + ASSERT(*hole_size > 0); - switch (device->fs_devices->chunk_alloc_policy) { - case BTRFS_CHUNK_ALLOC_REGULAR: - /* No extra check */ - break; - case BTRFS_CHUNK_ALLOC_ZONED: - if (dev_extent_hole_check_zoned(device, hole_start, - hole_size, num_bytes)) { - changed = true; - /* - * The changed hole can contain pending extent. - * Loop again to check that. - */ - continue; - } - break; - default: - BUG(); - } +again: + *hole_size = hole_end - *hole_start + 1; + found = find_hole_in_pending_extents(device, hole_start, hole_size, num_bytes); + if (!found) + return found; + ASSERT(*hole_size >= num_bytes); + switch (device->fs_devices->chunk_alloc_policy) { + case BTRFS_CHUNK_ALLOC_REGULAR: + return found; + case BTRFS_CHUNK_ALLOC_ZONED: + if (dev_extent_hole_check_zoned(device, hole_start, hole_size, num_bytes)) + goto again; break; + default: + BUG(); } - return changed; + return found; } /* @@ -1655,7 +1781,7 @@ static int find_free_dev_extent(struct btrfs_device *device, u64 num_bytes, ret = -ENOMEM; goto out; } -again: + if (search_start >= search_end || test_bit(BTRFS_DEV_STATE_REPLACE_TGT, &device->dev_state)) { ret = -ENOSPC; @@ -1742,11 +1868,7 @@ static int find_free_dev_extent(struct btrfs_device *device, u64 num_bytes, */ if (search_end > search_start) { hole_size = search_end - search_start; - if (dev_extent_hole_check(device, &search_start, &hole_size, - num_bytes)) { - btrfs_release_path(path); - goto again; - } + dev_extent_hole_check(device, &search_start, &hole_size, num_bytes); if (hole_size > max_hole_size) { max_hole_start = search_start; @@ -4909,6 +5031,7 @@ int btrfs_shrink_device(struct btrfs_device *device, u64 new_size) u64 old_size = btrfs_device_get_total_bytes(device); u64 diff; u64 start; + u64 pending_start, pending_end; new_size = round_down(new_size, fs_info->sectorsize); start = new_size; @@ -4942,7 +5065,7 @@ int btrfs_shrink_device(struct btrfs_device *device, u64 new_size) * in-memory chunks are synced to disk so that the loop below sees them * and relocates them accordingly. */ - if (contains_pending_extent(device, &start, diff)) { + if (first_pending_extent(device, start, diff, &pending_start, &pending_end)) { mutex_unlock(&fs_info->chunk_mutex); ret = btrfs_commit_transaction(trans); if (ret) diff --git a/fs/btrfs/xattr.c b/fs/btrfs/xattr.c index 76ff93b3eb27..3055b9ffb9d6 100644 --- a/fs/btrfs/xattr.c +++ b/fs/btrfs/xattr.c @@ -174,6 +174,7 @@ int btrfs_setxattr(struct btrfs_trans_handle *trans, struct inode *inode, const u16 old_data_len = btrfs_dir_data_len(leaf, di); const u32 item_size = btrfs_item_size(leaf, slot); const u32 data_size = sizeof(*di) + name_len + size; + unsigned long name_ptr; unsigned long data_ptr; char *ptr; @@ -202,8 +203,16 @@ int btrfs_setxattr(struct btrfs_trans_handle *trans, struct inode *inode, ptr = btrfs_item_ptr(leaf, slot, char); ptr += btrfs_item_size(leaf, slot) - data_size; di = (struct btrfs_dir_item *)ptr; + memzero_extent_buffer(leaf, (unsigned long)ptr + + offsetof(struct btrfs_dir_item, location), + sizeof(struct btrfs_disk_key)); + btrfs_set_dir_flags(leaf, di, BTRFS_FT_XATTR); + btrfs_set_dir_transid(leaf, di, trans->transid); + btrfs_set_dir_name_len(leaf, di, name_len); btrfs_set_dir_data_len(leaf, di, size); + name_ptr = (unsigned long)(di + 1); data_ptr = ((unsigned long)(di + 1)) + name_len; + write_extent_buffer(leaf, name, name_ptr, name_len); write_extent_buffer(leaf, value, data_ptr, size); btrfs_mark_buffer_dirty(trans, leaf); } else { diff --git a/fs/btrfs/zoned.c b/fs/btrfs/zoned.c index 92ec2d91c41f..7f6d0d44e751 100644 --- a/fs/btrfs/zoned.c +++ b/fs/btrfs/zoned.c @@ -2193,14 +2193,14 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writ GFP_NOFS); if (ret) - return ret; + break; if (!(block_group->flags & BTRFS_BLOCK_GROUP_DATA)) zinfo->reserved_active_zones++; btrfs_dev_clear_active_zone(device, physical); } - if (!fully_written) + if (!ret && !fully_written) btrfs_dec_block_group_ro(block_group); spin_lock(&fs_info->zone_active_bgs_lock); @@ -2213,7 +2213,7 @@ static int do_zone_finish(struct btrfs_block_group *block_group, bool fully_writ clear_and_wake_up_bit(BTRFS_FS_NEED_ZONE_FINISH, &fs_info->flags); - return 0; + return ret; } int btrfs_zone_finish(struct btrfs_block_group *block_group) diff --git a/fs/btrfs/zstd.c b/fs/btrfs/zstd.c index e7ac4ec809a4..a02eb80ebe2a 100644 --- a/fs/btrfs/zstd.c +++ b/fs/btrfs/zstd.c @@ -286,8 +286,17 @@ struct list_head *zstd_get_workspace(unsigned int level) DEFINE_WAIT(wait); prepare_to_wait(&wsm.wait, &wait, TASK_UNINTERRUPTIBLE); - schedule(); + /* + * Re-check after being queued: zstd_put_workspace() only wakes + * a queue that already has a sleeper, so a workspace returned + * since the failed allocation woke nobody. + */ + ws = zstd_find_workspace(level); + if (!ws) + schedule(); finish_wait(&wsm.wait, &wait); + if (ws) + return ws; goto again; } diff --git a/fs/ceph/addr.c b/fs/ceph/addr.c index 7d04b301ea3c..a5b21f448194 100644 --- a/fs/ceph/addr.c +++ b/fs/ceph/addr.c @@ -1160,6 +1160,10 @@ static int ceph_writepages_start(struct address_space *mapping, /* better not fail on first page! */ BUG_ON(locked_pages == 0); pages[locked_pages] = NULL; + /* Did we just begin a new contiguous op? Nevermind! */ + if (len == 0) + num_ops--; + redirty_page_for_writepage(wbc, page); unlock_page(page); break; diff --git a/fs/ceph/crypto.c b/fs/ceph/crypto.c index 42b066729d97..29ec2f705a50 100644 --- a/fs/ceph/crypto.c +++ b/fs/ceph/crypto.c @@ -369,6 +369,11 @@ int ceph_encode_encrypted_fname(struct inode *parent, struct dentry *dentry, * Otherwise, base64 decode the string, and then ask fscrypt to format it * for userland presentation. * + * Though the fscrypt/crypto subsystems broadly expect all buffers to be in the + * linear-mapped region, this function slightly relaxes those requirements: + * fname->ctext, fname->name, and oname->name may be vmalloc(), but not + * tname->name. + * * Returns 0 on success or negative error code on error. */ int ceph_fname_to_usr(const struct ceph_fname *fname, struct fscrypt_str *tname, @@ -376,11 +381,15 @@ int ceph_fname_to_usr(const struct ceph_fname *fname, struct fscrypt_str *tname, { struct inode *dir = fname->dir; struct fscrypt_str _tname = FSTR_INIT(NULL, 0); + struct fscrypt_str _oname; struct fscrypt_str iname; char *name = fname->name; int name_len = fname->name_len; int ret; + if (WARN_ON_ONCE(tname && is_vmalloc_addr(tname->name))) + return -EIO; + /* Sanity check that the resulting name will fit in the buffer */ if (fname->name_len > NAME_MAX || fname->ctext_len > NAME_MAX) return -EIO; @@ -421,16 +430,19 @@ int ceph_fname_to_usr(const struct ceph_fname *fname, struct fscrypt_str *tname, goto out_inode; } + if ((!tname || !tname->name) && + (fname->ctext_len == 0 || + unlikely(is_vmalloc_addr(fname->ctext)) || + unlikely(is_vmalloc_addr(oname->name)))) { + ret = fscrypt_fname_alloc_buffer(NAME_MAX, &_tname); + if (ret) + goto out_inode; + tname = &_tname; + } + if (fname->ctext_len == 0) { int declen; - if (!tname) { - ret = fscrypt_fname_alloc_buffer(NAME_MAX, &_tname); - if (ret) - goto out_inode; - tname = &_tname; - } - declen = ceph_base64_decode(name, name_len, tname->name); if (declen <= 0) { ret = -EIO; @@ -438,13 +450,29 @@ int ceph_fname_to_usr(const struct ceph_fname *fname, struct fscrypt_str *tname, } iname.name = tname->name; iname.len = declen; + } else if (unlikely(is_vmalloc_addr(fname->ctext))) { + memcpy(tname->name, fname->ctext, fname->ctext_len); + + iname.name = tname->name; + iname.len = fname->ctext_len; } else { iname.name = fname->ctext; iname.len = fname->ctext_len; } - ret = fscrypt_fname_disk_to_usr(dir, 0, 0, &iname, oname); - if (!ret && (dir != fname->dir)) { + _oname.name = unlikely(is_vmalloc_addr(oname->name)) ? + tname->name : oname->name; + _oname.len = oname->len; + + ret = fscrypt_fname_disk_to_usr(dir, 0, 0, &iname, &_oname); + if (ret) + goto out; + + if (unlikely(is_vmalloc_addr(oname->name))) + memcpy(oname->name, _oname.name, _oname.len); + oname->len = _oname.len; + + if (dir != fname->dir) { char tmp_buf[CEPH_BASE64_CHARS(NAME_MAX)]; name_len = snprintf(tmp_buf, sizeof(tmp_buf), "_%.*s_%ld", diff --git a/fs/ceph/debugfs.c b/fs/ceph/debugfs.c index 24c08078f5aa..c79a11ce7a4c 100644 --- a/fs/ceph/debugfs.c +++ b/fs/ceph/debugfs.c @@ -55,8 +55,6 @@ static int mdsc_show(struct seq_file *s, void *p) struct ceph_mds_client *mdsc = fsc->mdsc; struct ceph_mds_request *req; struct rb_node *rp; - int pathlen = 0; - u64 pathbase; char *path; mutex_lock(&mdsc->mutex); @@ -81,8 +79,8 @@ static int mdsc_show(struct seq_file *s, void *p) if (req->r_inode) { seq_printf(s, " #%llx", ceph_ino(req->r_inode)); } else if (req->r_dentry) { - path = ceph_mdsc_build_path(mdsc, req->r_dentry, &pathlen, - &pathbase, 0); + struct ceph_path_info path_info = {0}; + path = ceph_mdsc_build_path(mdsc, req->r_dentry, &path_info, 0); if (IS_ERR(path)) path = NULL; spin_lock(&req->r_dentry->d_lock); @@ -91,7 +89,7 @@ static int mdsc_show(struct seq_file *s, void *p) req->r_dentry, path ? path : ""); spin_unlock(&req->r_dentry->d_lock); - ceph_mdsc_free_path(path, pathlen); + ceph_mdsc_free_path_info(&path_info); } else if (req->r_path1) { seq_printf(s, " #%llx/%s", req->r_ino1.ino, req->r_path1); @@ -100,8 +98,8 @@ static int mdsc_show(struct seq_file *s, void *p) } if (req->r_old_dentry) { - path = ceph_mdsc_build_path(mdsc, req->r_old_dentry, &pathlen, - &pathbase, 0); + struct ceph_path_info path_info = {0}; + path = ceph_mdsc_build_path(mdsc, req->r_old_dentry, &path_info, 0); if (IS_ERR(path)) path = NULL; spin_lock(&req->r_old_dentry->d_lock); @@ -111,7 +109,7 @@ static int mdsc_show(struct seq_file *s, void *p) req->r_old_dentry, path ? path : ""); spin_unlock(&req->r_old_dentry->d_lock); - ceph_mdsc_free_path(path, pathlen); + ceph_mdsc_free_path_info(&path_info); } else if (req->r_path2 && req->r_op != CEPH_MDS_OP_SYMLINK) { if (req->r_ino2.ino) seq_printf(s, " #%llx/%s", req->r_ino2.ino, diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c index 0b764dcbc7bf..fe8a050cf7b9 100644 --- a/fs/ceph/dir.c +++ b/fs/ceph/dir.c @@ -1266,10 +1266,8 @@ static void ceph_async_unlink_cb(struct ceph_mds_client *mdsc, /* If op failed, mark everyone involved for errors */ if (result) { - int pathlen = 0; - u64 base = 0; - char *path = ceph_mdsc_build_path(mdsc, dentry, &pathlen, - &base, 0); + struct ceph_path_info path_info = {0}; + char *path = ceph_mdsc_build_path(mdsc, dentry, &path_info, 0); /* mark error on parent + clear complete */ mapping_set_error(req->r_parent->i_mapping, result); @@ -1283,8 +1281,8 @@ static void ceph_async_unlink_cb(struct ceph_mds_client *mdsc, mapping_set_error(req->r_old_inode->i_mapping, result); pr_warn_client(cl, "failure path=(%llx)%s result=%d!\n", - base, IS_ERR(path) ? "<>" : path, result); - ceph_mdsc_free_path(path, pathlen); + path_info.vino.ino, IS_ERR(path) ? "<>" : path, result); + ceph_mdsc_free_path_info(&path_info); } out: iput(req->r_old_inode); @@ -1722,11 +1720,11 @@ static int __dir_lease_check(struct dentry *dentry, void *arg) if (ret > 0) { if (time_before(jiffies, di->time + lwc->dir_lease_ttl)) return STOP; + if (!lwc->expire_dir_lease) + return KEEP; /* Move dentry to tail of dir lease list if we don't want * to delete it. So dentries in the list are checked in a * round robin manner */ - if (!lwc->expire_dir_lease) - return TOUCH; if (dentry->d_lockref.count > 0 || (di->flags & CEPH_DENTRY_REFERENCED)) return TOUCH; @@ -1753,7 +1751,7 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc) lwc.dir_lease = false; lwc.nr_to_scan = CEPH_CAPS_PER_RELEASE * 2; freed = __dentry_leases_walk(mdsc, &lwc, __dentry_lease_check); - if (!lwc.nr_to_scan) /* more invalid leases */ + if (freed > 0 && !lwc.nr_to_scan) /* more invalid leases */ return -EAGAIN; if (lwc.nr_to_scan < CEPH_CAPS_PER_RELEASE) @@ -1763,6 +1761,10 @@ int ceph_trim_dentries(struct ceph_mds_client *mdsc) lwc.expire_dir_lease = freed < count; lwc.dir_lease_ttl = mdsc->fsc->mount_options->caps_wanted_delay_max * HZ; freed +=__dentry_leases_walk(mdsc, &lwc, __dir_lease_check); + if (freed == 0 && count == 0) + /* no progress possible currently, retry futile */ + return 0; + if (!lwc.nr_to_scan) /* more to check */ return -EAGAIN; diff --git a/fs/ceph/file.c b/fs/ceph/file.c index 15d88d0a49ae..1c7abc4ef882 100644 --- a/fs/ceph/file.c +++ b/fs/ceph/file.c @@ -582,15 +582,13 @@ static void ceph_async_create_cb(struct ceph_mds_client *mdsc, mapping_set_error(req->r_parent->i_mapping, result); if (result) { - int pathlen = 0; - u64 base = 0; - char *path = ceph_mdsc_build_path(mdsc, req->r_dentry, &pathlen, - &base, 0); + struct ceph_path_info path_info = {0}; + char *path = ceph_mdsc_build_path(mdsc, req->r_dentry, &path_info, 0); pr_warn_client(cl, "async create failure path=(%llx)%s result=%d!\n", - base, IS_ERR(path) ? "<>" : path, result); - ceph_mdsc_free_path(path, pathlen); + path_info.vino.ino, IS_ERR(path) ? "<>" : path, result); + ceph_mdsc_free_path_info(&path_info); ceph_dir_clear_complete(req->r_parent); if (!d_unhashed(dentry)) diff --git a/fs/ceph/mds_client.c b/fs/ceph/mds_client.c index b49f30417047..a32f6341fbeb 100644 --- a/fs/ceph/mds_client.c +++ b/fs/ceph/mds_client.c @@ -495,9 +495,13 @@ static int parse_reply_info_readdir(void **p, void *end, * to do the base64_decode in-place. It's * safe because the decoded string should * always be shorter, which is 3/4 of origin - * string. + * string. If this message was allocated with + * vmalloc() (happens, but rarely), leave it + * NULL and let ceph_fname_to_usr() allocate + * suitable temporary working space instead. */ - tname.name = _name; + if (likely(!is_vmalloc_addr(_name))) + tname.name = _name; /* * Set oname to _name too, and this will be @@ -569,10 +573,36 @@ static int parse_reply_info_filelock(void **p, void *end, #define DELEGATED_INO_AVAILABLE xa_mk_value(1) +static int ceph_insert_deleg_ino(struct ceph_mds_session *s, u64 ino) +{ + struct ceph_client *cl = s->s_mdsc->fsc->client; + int err; + + /* + * Cap how many delegated inodes a single session may hold. This is + * the only place that grows the count, so atomic_add_unless() bounds + * it at exactly CEPH_MAX_DELEG_INOS; s_num_deleg_inos can never exceed + * that. + */ + if (!atomic_add_unless(&s->s_num_deleg_inos, 1, CEPH_MAX_DELEG_INOS)) { + pr_warn_ratelimited_client(cl, + "MDS session already holds %d delegated inodes\n", + CEPH_MAX_DELEG_INOS); + return -EOVERFLOW; + } + + err = xa_insert(&s->s_delegated_inos, ino, DELEGATED_INO_AVAILABLE, + GFP_KERNEL); + if (err) + atomic_dec(&s->s_num_deleg_inos); + return err; +} + static int ceph_parse_deleg_inos(void **p, void *end, struct ceph_mds_session *s) { struct ceph_client *cl = s->s_mdsc->fsc->client; + u64 msg_deleg_inos = 0; u32 sets; ceph_decode_32_safe(p, end, sets, bad); @@ -590,16 +620,34 @@ static int ceph_parse_deleg_inos(void **p, void *end, start, len); continue; } + + /* + * Bound the number of inodes one reply may delegate. + * ceph_insert_deleg_ino() separately caps the per-session + * population, so this only has to stop one reply from spinning + * the insert loop under an attacker-controlled len. + */ + if (len > (u64)CEPH_MAX_DELEG_INOS || + msg_deleg_inos > (u64)CEPH_MAX_DELEG_INOS - len) { + pr_warn_ratelimited_client(cl, + "MDS reply delegates too many inodes (have %llu, +%llu, max %d)\n", + msg_deleg_inos, len, CEPH_MAX_DELEG_INOS); + return -EIO; + } + msg_deleg_inos += len; + while (len--) { - int err = xa_insert(&s->s_delegated_inos, start++, - DELEGATED_INO_AVAILABLE, - GFP_KERNEL); + int err = ceph_insert_deleg_ino(s, start++); + if (!err) { doutc(cl, "added delegated inode 0x%llx\n", start - 1); } else if (err == -EBUSY) { pr_warn_client(cl, "MDS delegated inode 0x%llx more than once.\n", start - 1); + } else if (err == -EOVERFLOW) { + /* ceph_insert_deleg_ino() already warned. */ + return -EIO; } else { return err; } @@ -617,16 +665,17 @@ u64 ceph_get_deleg_ino(struct ceph_mds_session *s) xa_for_each(&s->s_delegated_inos, ino, val) { val = xa_erase(&s->s_delegated_inos, ino); - if (val == DELEGATED_INO_AVAILABLE) + if (val == DELEGATED_INO_AVAILABLE) { + atomic_dec(&s->s_num_deleg_inos); return ino; + } } return 0; } int ceph_restore_deleg_ino(struct ceph_mds_session *s, u64 ino) { - return xa_insert(&s->s_delegated_inos, ino, DELEGATED_INO_AVAILABLE, - GFP_KERNEL); + return ceph_insert_deleg_ino(s, ino); } #else /* BITS_PER_LONG == 64 */ /* @@ -1010,6 +1059,7 @@ static struct ceph_mds_session *register_session(struct ceph_mds_client *mdsc, INIT_LIST_HEAD(&s->s_waiting); INIT_LIST_HEAD(&s->s_unsafe); xa_init(&s->s_delegated_inos); + atomic_set(&s->s_num_deleg_inos, 0); INIT_LIST_HEAD(&s->s_cap_releases); INIT_WORK(&s->s_cap_release_work, ceph_cap_release_work); @@ -2624,8 +2674,7 @@ static u8 *get_fscrypt_altname(const struct ceph_mds_request *req, u32 *plen) * ceph_mdsc_build_path - build a path string to a given dentry * @mdsc: mds client * @dentry: dentry to which path should be built - * @plen: returned length of string - * @pbase: returned base inode number + * @path_info: output path, length, base ino+snap, and freepath ownership flag * @for_wire: is this path going to be sent to the MDS? * * Build a string that represents the path to the dentry. This is mostly called @@ -2643,7 +2692,7 @@ static u8 *get_fscrypt_altname(const struct ceph_mds_request *req, u32 *plen) * foo/.snap/bar -> foo//bar */ char *ceph_mdsc_build_path(struct ceph_mds_client *mdsc, struct dentry *dentry, - int *plen, u64 *pbase, int for_wire) + struct ceph_path_info *path_info, int for_wire) { struct ceph_client *cl = mdsc->fsc->client; struct dentry *cur; @@ -2756,16 +2805,28 @@ char *ceph_mdsc_build_path(struct ceph_mds_client *mdsc, struct dentry *dentry, return ERR_PTR(-ENAMETOOLONG); } - *pbase = base; - *plen = PATH_MAX - 1 - pos; + /* Initialize the output structure */ + memset(path_info, 0, sizeof(*path_info)); + + path_info->vino.ino = base; + path_info->pathlen = PATH_MAX - 1 - pos; + path_info->path = path + pos; + path_info->freepath = true; + + /* Set snap from dentry if available */ + if (d_inode(dentry)) + path_info->vino.snap = ceph_snap(d_inode(dentry)); + else + path_info->vino.snap = CEPH_NOSNAP; + doutc(cl, "on %p %d built %llx '%.*s'\n", dentry, d_count(dentry), - base, *plen, path + pos); + base, PATH_MAX - 1 - pos, path + pos); return path + pos; } static int build_dentry_path(struct ceph_mds_client *mdsc, struct dentry *dentry, - struct inode *dir, const char **ppath, int *ppathlen, - u64 *pino, bool *pfreepath, bool parent_locked) + struct inode *dir, struct ceph_path_info *path_info, + bool parent_locked) { char *path; @@ -2774,41 +2835,47 @@ static int build_dentry_path(struct ceph_mds_client *mdsc, struct dentry *dentry dir = d_inode_rcu(dentry->d_parent); if (dir && parent_locked && ceph_snap(dir) == CEPH_NOSNAP && !IS_ENCRYPTED(dir)) { - *pino = ceph_ino(dir); + path_info->vino.ino = ceph_ino(dir); + path_info->vino.snap = ceph_snap(dir); rcu_read_unlock(); - *ppath = dentry->d_name.name; - *ppathlen = dentry->d_name.len; + path_info->path = dentry->d_name.name; + path_info->pathlen = dentry->d_name.len; + path_info->freepath = false; return 0; } rcu_read_unlock(); - path = ceph_mdsc_build_path(mdsc, dentry, ppathlen, pino, 1); + path = ceph_mdsc_build_path(mdsc, dentry, path_info, 1); if (IS_ERR(path)) return PTR_ERR(path); - *ppath = path; - *pfreepath = true; + /* + * ceph_mdsc_build_path already fills path_info, including snap handling. + */ return 0; } -static int build_inode_path(struct inode *inode, - const char **ppath, int *ppathlen, u64 *pino, - bool *pfreepath) +static int build_inode_path(struct inode *inode, struct ceph_path_info *path_info) { struct ceph_mds_client *mdsc = ceph_sb_to_mdsc(inode->i_sb); struct dentry *dentry; char *path; if (ceph_snap(inode) == CEPH_NOSNAP) { - *pino = ceph_ino(inode); - *ppathlen = 0; + path_info->vino.ino = ceph_ino(inode); + path_info->vino.snap = ceph_snap(inode); + path_info->pathlen = 0; + path_info->freepath = false; return 0; } dentry = d_find_alias(inode); - path = ceph_mdsc_build_path(mdsc, dentry, ppathlen, pino, 1); + path = ceph_mdsc_build_path(mdsc, dentry, path_info, 1); dput(dentry); if (IS_ERR(path)) return PTR_ERR(path); - *ppath = path; - *pfreepath = true; + /* + * ceph_mdsc_build_path already fills path_info, including snap from dentry. + * Override with inode's snap since that's what this function is for. + */ + path_info->vino.snap = ceph_snap(inode); return 0; } @@ -2818,26 +2885,32 @@ static int build_inode_path(struct inode *inode, */ static int set_request_path_attr(struct ceph_mds_client *mdsc, struct inode *rinode, struct dentry *rdentry, struct inode *rdiri, - const char *rpath, u64 rino, const char **ppath, - int *pathlen, u64 *ino, bool *freepath, + const char *rpath, u64 rino, + struct ceph_path_info *path_info, bool parent_locked) { struct ceph_client *cl = mdsc->fsc->client; int r = 0; + /* Initialize the output structure */ + memset(path_info, 0, sizeof(*path_info)); + if (rinode) { - r = build_inode_path(rinode, ppath, pathlen, ino, freepath); + r = build_inode_path(rinode, path_info); doutc(cl, " inode %p %llx.%llx\n", rinode, ceph_ino(rinode), ceph_snap(rinode)); } else if (rdentry) { - r = build_dentry_path(mdsc, rdentry, rdiri, ppath, pathlen, ino, - freepath, parent_locked); - doutc(cl, " dentry %p %llx/%.*s\n", rdentry, *ino, *pathlen, *ppath); + r = build_dentry_path(mdsc, rdentry, rdiri, path_info, parent_locked); + doutc(cl, " dentry %p %llx/%.*s\n", rdentry, path_info->vino.ino, + path_info->pathlen, path_info->path); } else if (rpath || rino) { - *ino = rino; - *ppath = rpath; - *pathlen = rpath ? strlen(rpath) : 0; - doutc(cl, " path %.*s\n", *pathlen, rpath); + path_info->vino.ino = rino; + path_info->vino.snap = CEPH_NOSNAP; + path_info->path = rpath; + path_info->pathlen = rpath ? strlen(rpath) : 0; + path_info->freepath = false; + + doutc(cl, " path %.*s\n", path_info->pathlen, rpath); } return r; @@ -2902,11 +2975,8 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, struct ceph_mds_client *mdsc = session->s_mdsc; struct ceph_msg *msg; struct ceph_mds_request_head_legacy *lhead; - const char *path1 = NULL; - const char *path2 = NULL; - u64 ino1 = 0, ino2 = 0; - int pathlen1 = 0, pathlen2 = 0; - bool freepath1 = false, freepath2 = false; + struct ceph_path_info path_info1 = {0}; + struct ceph_path_info path_info2 = {0}; struct dentry *old_dentry = NULL; int len; u16 releases; @@ -2915,25 +2985,49 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, bool legacy = !(session->s_con.peer_features & CEPH_FEATURE_FS_BTIME); bool old_version = !test_bit(CEPHFS_FEATURE_32BITS_RETRY_FWD, &session->s_features); + bool parent_locked = test_bit(CEPH_MDS_R_PARENT_LOCKED, &req->r_req_flags); ret = set_request_path_attr(mdsc, req->r_inode, req->r_dentry, - req->r_parent, req->r_path1, req->r_ino1.ino, - &path1, &pathlen1, &ino1, &freepath1, - test_bit(CEPH_MDS_R_PARENT_LOCKED, - &req->r_req_flags)); + req->r_parent, req->r_path1, req->r_ino1.ino, + &path_info1, parent_locked); if (ret < 0) { msg = ERR_PTR(ret); goto out; } + /* + * When the parent directory's i_rwsem is *not* locked, req->r_parent may + * have become stale (e.g. after a concurrent rename) between the time the + * dentry was looked up and now. If we detect that the stored r_parent + * does not match the inode number we just encoded for the request, switch + * to the correct inode so that the MDS receives a valid parent reference. + */ + if (!parent_locked && req->r_parent && path_info1.vino.ino && + ceph_ino(req->r_parent) != path_info1.vino.ino) { + struct inode *old_parent = req->r_parent; + struct inode *correct_dir = ceph_get_inode(mdsc->fsc->sb, path_info1.vino, NULL); + if (!IS_ERR(correct_dir)) { + WARN_ONCE(1, "ceph: r_parent mismatch (had %llx wanted %llx) - updating\n", + ceph_ino(old_parent), path_info1.vino.ino); + /* + * Transfer CEPH_CAP_PIN from the old parent to the new one. + * The pin was taken earlier in ceph_mdsc_submit_request(). + */ + ceph_put_cap_refs(ceph_inode(old_parent), CEPH_CAP_PIN); + iput(old_parent); + req->r_parent = correct_dir; + ceph_get_cap_refs(ceph_inode(req->r_parent), CEPH_CAP_PIN); + } + } + /* If r_old_dentry is set, then assume that its parent is locked */ if (req->r_old_dentry && !(req->r_old_dentry->d_flags & DCACHE_DISCONNECTED)) old_dentry = req->r_old_dentry; ret = set_request_path_attr(mdsc, NULL, old_dentry, - req->r_old_dentry_dir, - req->r_path2, req->r_ino2.ino, - &path2, &pathlen2, &ino2, &freepath2, true); + req->r_old_dentry_dir, + req->r_path2, req->r_ino2.ino, + &path_info2, true); if (ret < 0) { msg = ERR_PTR(ret); goto out_free1; @@ -2962,7 +3056,7 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, /* filepaths */ len += 2 * (1 + sizeof(u32) + sizeof(u64)); - len += pathlen1 + pathlen2; + len += path_info1.pathlen + path_info2.pathlen; /* cap releases */ len += sizeof(struct ceph_mds_request_release) * @@ -2970,9 +3064,9 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, !!req->r_old_inode_drop + !!req->r_old_dentry_drop); if (req->r_dentry_drop) - len += pathlen1; + len += path_info1.pathlen; if (req->r_old_dentry_drop) - len += pathlen2; + len += path_info2.pathlen; /* MClientRequest tail */ @@ -3038,8 +3132,8 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, lhead->ino = cpu_to_le64(req->r_deleg_ino); lhead->args = req->r_args; - ceph_encode_filepath(&p, end, ino1, path1); - ceph_encode_filepath(&p, end, ino2, path2); + ceph_encode_filepath(&p, end, path_info1.vino.ino, path_info1.path); + ceph_encode_filepath(&p, end, path_info2.vino.ino, path_info2.path); /* make note of release offset, in case we need to replay */ req->r_request_release_offset = p - msg->front.iov_base; @@ -3102,11 +3196,9 @@ static struct ceph_msg *create_request_message(struct ceph_mds_session *session, msg->hdr.data_off = cpu_to_le16(0); out_free2: - if (freepath2) - ceph_mdsc_free_path((char *)path2, pathlen2); + ceph_mdsc_free_path_info(&path_info2); out_free1: - if (freepath1) - ceph_mdsc_free_path((char *)path1, pathlen1); + ceph_mdsc_free_path_info(&path_info1); out: return msg; out_err: @@ -4370,24 +4462,20 @@ static int reconnect_caps_cb(struct inode *inode, int mds, void *arg) struct ceph_pagelist *pagelist = recon_state->pagelist; struct dentry *dentry; struct ceph_cap *cap; - char *path; - int pathlen = 0, err; - u64 pathbase; + struct ceph_path_info path_info = {0}; + int err; u64 snap_follows; dentry = d_find_primary(inode); if (dentry) { /* set pathbase to parent dir when msg_version >= 2 */ - path = ceph_mdsc_build_path(mdsc, dentry, &pathlen, &pathbase, + char *path = ceph_mdsc_build_path(mdsc, dentry, &path_info, recon_state->msg_version >= 2); dput(dentry); if (IS_ERR(path)) { err = PTR_ERR(path); goto out_err; } - } else { - path = NULL; - pathbase = 0; } spin_lock(&ci->i_ceph_lock); @@ -4420,7 +4508,7 @@ static int reconnect_caps_cb(struct inode *inode, int mds, void *arg) rec.v2.wanted = cpu_to_le32(__ceph_caps_wanted(ci)); rec.v2.issued = cpu_to_le32(cap->issued); rec.v2.snaprealm = cpu_to_le64(ci->i_snap_realm->ino); - rec.v2.pathbase = cpu_to_le64(pathbase); + rec.v2.pathbase = cpu_to_le64(path_info.vino.ino); rec.v2.flock_len = (__force __le32) ((ci->i_ceph_flags & CEPH_I_ERROR_FILELOCK) ? 0 : 1); } else { @@ -4431,7 +4519,7 @@ static int reconnect_caps_cb(struct inode *inode, int mds, void *arg) ceph_encode_timespec64(&rec.v1.mtime, &inode->i_mtime); ceph_encode_timespec64(&rec.v1.atime, &inode->i_atime); rec.v1.snaprealm = cpu_to_le64(ci->i_snap_realm->ino); - rec.v1.pathbase = cpu_to_le64(pathbase); + rec.v1.pathbase = cpu_to_le64(path_info.vino.ino); } if (list_empty(&ci->i_cap_snaps)) { @@ -4493,7 +4581,7 @@ static int reconnect_caps_cb(struct inode *inode, int mds, void *arg) sizeof(struct ceph_filelock); rec.v2.flock_len = cpu_to_le32(struct_len); - struct_len += sizeof(u32) + pathlen + sizeof(rec.v2); + struct_len += sizeof(u32) + path_info.pathlen + sizeof(rec.v2); if (struct_v >= 2) struct_len += sizeof(u64); /* snap_follows */ @@ -4517,7 +4605,7 @@ static int reconnect_caps_cb(struct inode *inode, int mds, void *arg) ceph_pagelist_encode_8(pagelist, 1); ceph_pagelist_encode_32(pagelist, struct_len); } - ceph_pagelist_encode_string(pagelist, path, pathlen); + ceph_pagelist_encode_string(pagelist, (char *)path_info.path, path_info.pathlen); ceph_pagelist_append(pagelist, &rec, sizeof(rec.v2)); ceph_locks_to_pagelist(flocks, pagelist, num_fcntl_locks, num_flock_locks); @@ -4528,17 +4616,17 @@ static int reconnect_caps_cb(struct inode *inode, int mds, void *arg) } else { err = ceph_pagelist_reserve(pagelist, sizeof(u64) + sizeof(u32) + - pathlen + sizeof(rec.v1)); + path_info.pathlen + sizeof(rec.v1)); if (err) goto out_err; ceph_pagelist_encode_64(pagelist, ceph_ino(inode)); - ceph_pagelist_encode_string(pagelist, path, pathlen); + ceph_pagelist_encode_string(pagelist, (char *)path_info.path, path_info.pathlen); ceph_pagelist_append(pagelist, &rec, sizeof(rec.v1)); } out_err: - ceph_mdsc_free_path(path, pathlen); + ceph_mdsc_free_path_info(&path_info); if (!err) recon_state->nr_caps++; return err; @@ -4638,6 +4726,7 @@ static void send_mds_reconnect(struct ceph_mds_client *mdsc, goto fail_nomsg; xa_destroy(&session->s_delegated_inos); + atomic_set(&session->s_num_deleg_inos, 0); mutex_lock(&session->s_mutex); session->s_state = CEPH_MDS_SESSION_RECONNECTING; @@ -4796,6 +4885,7 @@ static void check_new_map(struct ceph_mds_client *mdsc, { int i, j, err; int oldstate, newstate; + bool holding; struct ceph_mds_session *s; unsigned long targets[DIV_ROUND_UP(CEPH_MAX_MDS, sizeof(unsigned long))] = {0}; struct ceph_client *cl = mdsc->fsc->client; @@ -4842,10 +4932,13 @@ static void check_new_map(struct ceph_mds_client *mdsc, continue; } + holding = false; if (memcmp(ceph_mdsmap_get_addr(oldmap, i), ceph_mdsmap_get_addr(newmap, i), sizeof(struct ceph_entity_addr))) { /* just close it */ + holding = true; + ceph_get_mds_session(s); mutex_unlock(&mdsc->mutex); mutex_lock(&s->s_mutex); mutex_lock(&mdsc->mutex); @@ -4861,6 +4954,10 @@ static void check_new_map(struct ceph_mds_client *mdsc, */ if (s->s_state == CEPH_MDS_SESSION_RESTARTING && newstate >= CEPH_MDS_STATE_RECONNECT) { + if (!holding) { + holding = true; + ceph_get_mds_session(s); + } mutex_unlock(&mdsc->mutex); clear_bit(i, targets); send_mds_reconnect(mdsc, s); @@ -4877,13 +4974,18 @@ static void check_new_map(struct ceph_mds_client *mdsc, pr_info_client(cl, "mds%d recovery completed\n", s->s_mds); kick_requests(mdsc, i); + ceph_get_mds_session(s); mutex_unlock(&mdsc->mutex); mutex_lock(&s->s_mutex); mutex_lock(&mdsc->mutex); ceph_kick_flushing_caps(mdsc, s); mutex_unlock(&s->s_mutex); wake_up_session_caps(s, RECONNECT); + ceph_put_mds_session(s); } + + if (holding) + ceph_put_mds_session(s); } /* diff --git a/fs/ceph/mds_client.h b/fs/ceph/mds_client.h index 4237c6615a3f..9916acd901ab 100644 --- a/fs/ceph/mds_client.h +++ b/fs/ceph/mds_client.h @@ -230,6 +230,7 @@ struct ceph_mds_session { struct list_head s_waiting; /* waiting requests */ struct list_head s_unsafe; /* unsafe requests */ struct xarray s_delegated_inos; + atomic_t s_num_deleg_inos; }; /* @@ -577,14 +578,24 @@ extern int ceph_iterate_session_caps(struct ceph_mds_session *session, void *arg); extern void ceph_mdsc_pre_umount(struct ceph_mds_client *mdsc); -static inline void ceph_mdsc_free_path(char *path, int len) +/* + * Structure to group path-related output parameters for build_*_path functions + */ +struct ceph_path_info { + const char *path; + int pathlen; + struct ceph_vino vino; + bool freepath; +}; + +static inline void ceph_mdsc_free_path_info(const struct ceph_path_info *path_info) { - if (!IS_ERR_OR_NULL(path)) - __putname(path - (PATH_MAX - 1 - len)); + if (path_info && path_info->freepath && !IS_ERR_OR_NULL(path_info->path)) + __putname((char *)path_info->path - (PATH_MAX - 1 - path_info->pathlen)); } extern char *ceph_mdsc_build_path(struct ceph_mds_client *mdsc, - struct dentry *dentry, int *plen, u64 *base, + struct dentry *dentry, struct ceph_path_info *path_info, int for_wire); extern void __ceph_mdsc_drop_dentry_lease(struct dentry *dentry); diff --git a/fs/ceph/super.h b/fs/ceph/super.h index e31ecbed609d..c921ffcb93bd 100644 --- a/fs/ceph/super.h +++ b/fs/ceph/super.h @@ -596,6 +596,15 @@ static inline int ceph_ino_compare(struct inode *inode, void *data) #define CEPH_MDS_INO_LOG_OFFSET (2 * CEPH_MAX_MDS) #define CEPH_INO_SYSTEM_BASE ((6*CEPH_MAX_MDS) + (CEPH_MAX_MDS * CEPH_NUM_STRAY)) +/* + * Upper bound on the number of delegated inodes a single MDS session may + * hold. The MDS normally hands out a small preallocation window (the + * userspace mds_client_prealloc_inos option defaults to 1000) and refills + * it as the client consumes entries. This leaves generous headroom while + * bounding the CPU and memory a malformed delegation interval can consume. + */ +#define CEPH_MAX_DELEG_INOS 8192 + static inline bool ceph_vino_is_reserved(const struct ceph_vino vino) { if (vino.ino >= CEPH_INO_SYSTEM_BASE || diff --git a/fs/configfs/dir.c b/fs/configfs/dir.c index 46540846d815..e7ca50f11996 100644 --- a/fs/configfs/dir.c +++ b/fs/configfs/dir.c @@ -395,8 +395,17 @@ static void remove_dir(struct dentry * d) configfs_remove_dirent(d); - if (d_really_is_positive(d)) - simple_rmdir(d_inode(parent),d); + if (d_really_is_positive(d)) { + if (!simple_rmdir(d_inode(parent), d)) + /* + * configfs_get_config_item() takes a hashed dentry as + * proof that ->s_element is still alive. Our caller + * is about to drop the last reference to the item and + * the VFS will not unhash until after we return, so + * unhash it here. + */ + d_drop(d); + } pr_debug(" o %pd removing done (%d)\n", d, d_count(d)); diff --git a/fs/configfs/symlink.c b/fs/configfs/symlink.c index 69133ec1fac2..b42f07e2ea7f 100644 --- a/fs/configfs/symlink.c +++ b/fs/configfs/symlink.c @@ -76,9 +76,9 @@ static int configfs_get_target_path(struct config_item *item, static int create_link(struct config_item *parent_item, struct config_item *item, + struct configfs_dirent *target_sd, struct dentry *dentry) { - struct configfs_dirent *target_sd = item->ci_dentry->d_fsdata; char *body; int ret; @@ -114,26 +114,35 @@ static int create_link(struct config_item *parent_item, } -static int get_target(const char *symname, struct path *path, - struct config_item **target, struct super_block *sb) +static int get_target(const char *symname, struct config_item **target, + struct configfs_dirent **target_sd, + struct super_block *sb) { + struct path path __free(path_put) = {}; int ret; - ret = kern_path(symname, LOOKUP_FOLLOW|LOOKUP_DIRECTORY, path); - if (!ret) { - if (path->dentry->d_sb == sb) { - *target = configfs_get_config_item(path->dentry); - if (!*target) { - ret = -ENOENT; - path_put(path); - } - } else { - ret = -EPERM; - path_put(path); - } - } + ret = kern_path(symname, LOOKUP_FOLLOW|LOOKUP_DIRECTORY, &path); + if (ret) + return ret; + if (path.dentry->d_sb != sb) + return -EPERM; + /* + * A hashed dentry guarantees that neither the item nor the dirent + * have been released yet, as removals unhash before dropping. + * Grab both references here. An item reference alone would not keep + * ->ci_dentry alive. + */ + spin_lock(&path.dentry->d_lock); + if (!d_unhashed(path.dentry)) { + struct configfs_dirent *sd = path.dentry->d_fsdata; - return ret; + *target = config_item_get(sd->s_element); + *target_sd = configfs_get(sd); + } + spin_unlock(&path.dentry->d_lock); + if (!*target) + return -ENOENT; + return 0; } @@ -141,10 +150,10 @@ int configfs_symlink(struct mnt_idmap *idmap, struct inode *dir, struct dentry *dentry, const char *symname) { int ret; - struct path path; struct configfs_dirent *sd; struct config_item *parent_item; struct config_item *target_item = NULL; + struct configfs_dirent *target_sd = NULL; const struct config_item_type *type; sd = dentry->d_parent->d_fsdata; @@ -188,7 +197,7 @@ int configfs_symlink(struct mnt_idmap *idmap, struct inode *dir, * AV, a thoroughly annoyed bastard. */ inode_unlock(dir); - ret = get_target(symname, &path, &target_item, dentry->d_sb); + ret = get_target(symname, &target_item, &target_sd, dentry->d_sb); inode_lock(dir); if (ret) goto out_put; @@ -202,15 +211,15 @@ int configfs_symlink(struct mnt_idmap *idmap, struct inode *dir, ret = type->ct_item_ops->allow_link(parent_item, target_item); if (!ret) { mutex_lock(&configfs_symlink_mutex); - ret = create_link(parent_item, target_item, dentry); + ret = create_link(parent_item, target_item, target_sd, dentry); mutex_unlock(&configfs_symlink_mutex); if (ret && type->ct_item_ops->drop_link) type->ct_item_ops->drop_link(parent_item, target_item); } + configfs_put(target_sd); config_item_put(target_item); - path_put(&path); out_put: config_item_put(parent_item); diff --git a/fs/dcache.c b/fs/dcache.c index b422baff02c9..e1f4a9e61e58 100644 --- a/fs/dcache.c +++ b/fs/dcache.c @@ -432,7 +432,8 @@ static void d_lru_add(struct dentry *dentry) this_cpu_inc(nr_dentry_unused); if (d_is_negative(dentry)) this_cpu_inc(nr_dentry_negative); - WARN_ON_ONCE(!list_lru_add(&dentry->d_sb->s_dentry_lru, &dentry->d_lru)); + WARN_ON_ONCE(!list_lru_add_obj( + &dentry->d_sb->s_dentry_lru, &dentry->d_lru)); } static void d_lru_del(struct dentry *dentry) @@ -442,7 +443,8 @@ static void d_lru_del(struct dentry *dentry) this_cpu_dec(nr_dentry_unused); if (d_is_negative(dentry)) this_cpu_dec(nr_dentry_negative); - WARN_ON_ONCE(!list_lru_del(&dentry->d_sb->s_dentry_lru, &dentry->d_lru)); + WARN_ON_ONCE(!list_lru_del_obj( + &dentry->d_sb->s_dentry_lru, &dentry->d_lru)); } static void d_shrink_del(struct dentry *dentry) @@ -1245,7 +1247,7 @@ static enum lru_status dentry_lru_isolate(struct list_head *item, * * This is guaranteed by the fact that all LRU management * functions are intermediated by the LRU API calls like - * list_lru_add and list_lru_del. List movement in this file + * list_lru_add_obj and list_lru_del_obj. List movement in this file * only ever occur through this functions or through callbacks * like this one, that are called from the LRU API. * @@ -1502,6 +1504,8 @@ EXPORT_SYMBOL(path_has_submounts); * * Only one of d_invalidate() and d_set_mounted() must succeed. For * this reason take rename_lock and d_lock on dentry and ancestors. + * Likewise for dont_mount() which marks a dentry that is being removed + * under d_lock. */ int d_set_mounted(struct dentry *dentry) { @@ -1518,7 +1522,7 @@ int d_set_mounted(struct dentry *dentry) spin_unlock(&p->d_lock); } spin_lock(&dentry->d_lock); - if (!d_unlinked(dentry)) { + if (!d_unlinked(dentry) && !cant_mount(dentry)) { ret = -EBUSY; if (!d_mountpoint(dentry)) { dentry->d_flags |= DCACHE_MOUNTED; diff --git a/fs/dlm/lock.c b/fs/dlm/lock.c index 6712d733fc90..dd2fa54f2b7d 100644 --- a/fs/dlm/lock.c +++ b/fs/dlm/lock.c @@ -444,6 +444,8 @@ int dlm_search_rsb_tree(struct rb_root *tree, const void *name, int len, struct dlm_rsb *r; int rc; + if (len > DLM_RESNAME_MAXLEN) + return -EINVAL; while (node) { r = rb_entry(node, struct dlm_rsb, res_hashnode); rc = rsb_cmp(r, name, len); diff --git a/fs/erofs/decompressor.c b/fs/erofs/decompressor.c index 18e4da98309d..c4a6391eb1fa 100644 --- a/fs/erofs/decompressor.c +++ b/fs/erofs/decompressor.c @@ -7,11 +7,6 @@ #include #include -#ifndef LZ4_DISTANCE_MAX /* history window size */ -#define LZ4_DISTANCE_MAX 65535 /* set to maximum value by default */ -#endif - -#define LZ4_MAX_DISTANCE_PAGES (DIV_ROUND_UP(LZ4_DISTANCE_MAX, PAGE_SIZE) + 1) #ifndef LZ4_DECOMPRESS_INPLACE_MARGIN #define LZ4_DECOMPRESS_INPLACE_MARGIN(srcsize) (((srcsize) >> 8) + 32) #endif @@ -29,15 +24,12 @@ static int z_erofs_load_lz4_config(struct super_block *sb, { struct erofs_sb_info *sbi = EROFS_SB(sb); struct z_erofs_lz4_cfgs *lz4 = data; - u16 distance; if (lz4) { if (size < sizeof(struct z_erofs_lz4_cfgs)) { erofs_err(sb, "invalid lz4 cfgs, size=%u", size); return -EINVAL; } - distance = le16_to_cpu(lz4->max_distance); - sbi->lz4.max_pclusterblks = le16_to_cpu(lz4->max_pclusterblks); if (!sbi->lz4.max_pclusterblks) { sbi->lz4.max_pclusterblks = 1; /* reserved case */ @@ -48,49 +40,28 @@ static int z_erofs_load_lz4_config(struct super_block *sb, return -EINVAL; } } else { - distance = le16_to_cpu(dsb->u1.lz4_max_distance); sbi->lz4.max_pclusterblks = 1; } - - sbi->lz4.max_distance_pages = distance ? - DIV_ROUND_UP(distance, PAGE_SIZE) + 1 : - LZ4_MAX_DISTANCE_PAGES; return erofs_pcpubuf_growsize(sbi->lz4.max_pclusterblks); } /* - * Fill all gaps with bounce pages if it's a sparse page list. Also check if - * all physical pages are consecutive, which can be seen for moderate CR. + * Fill all gaps with bounce pages if it's a sparse page list (for example some + * folios are already uptodate and thus can be mapped into userspace). Also + * check if pages are physically consecutive, which can be seen for moderate CR. */ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, struct page **pagepool) { struct z_erofs_decompress_req *rq = ctx->rq; - struct page *availables[LZ4_MAX_DISTANCE_PAGES] = { NULL }; - unsigned long bounced[DIV_ROUND_UP(LZ4_MAX_DISTANCE_PAGES, - BITS_PER_LONG)] = { 0 }; - unsigned int lz4_max_distance_pages = - EROFS_SB(rq->sb)->lz4.max_distance_pages; void *kaddr = NULL; - unsigned int i, j, top; - - top = 0; - for (i = j = 0; i < ctx->outpages; ++i, ++j) { - struct page *const page = rq->out[i]; - struct page *victim; - - if (j >= lz4_max_distance_pages) - j = 0; + unsigned int i; - /* 'valid' bounced can only be tested after a complete round */ - if (!rq->fillgaps && test_bit(j, bounced)) { - DBG_BUGON(i < lz4_max_distance_pages); - DBG_BUGON(top >= lz4_max_distance_pages); - availables[top++] = rq->out[i - lz4_max_distance_pages]; - } + for (i = 0; i < ctx->outpages; ++i) { + struct page *page, *victim; + page = rq->out[i]; if (page) { - __clear_bit(j, bounced); if (!PageHighMem(page)) { if (!i) { kaddr = page_address(page); @@ -102,21 +73,13 @@ static int z_erofs_lz4_prepare_dstpages(struct z_erofs_lz4_decompress_ctx *ctx, continue; } } - kaddr = NULL; - continue; - } - kaddr = NULL; - __set_bit(j, bounced); - - if (top) { - victim = availables[--top]; - get_page(victim); } else { victim = erofs_allocpage(pagepool, GFP_KERNEL | __GFP_NOFAIL); set_page_private(victim, Z_EROFS_SHORTLIVED_PAGE); + rq->out[i] = victim; } - rq->out[i] = victim; + kaddr = NULL; } return kaddr ? 1 : 0; } diff --git a/fs/erofs/decompressor_lzma.c b/fs/erofs/decompressor_lzma.c index 613fe99cf629..0ea93a7b7a4e 100644 --- a/fs/erofs/decompressor_lzma.c +++ b/fs/erofs/decompressor_lzma.c @@ -6,6 +6,7 @@ struct z_erofs_lzma { struct z_erofs_lzma *next; struct xz_dec_microlzma *state; + unsigned int dict_size; struct xz_buf buf; u8 bounce[PAGE_SIZE]; }; @@ -132,11 +133,19 @@ int z_erofs_load_lzma_config(struct super_block *sb, err = 0; /* 2. walk each isolated stream and grow max dict_size if needed */ for (strm = head; strm; strm = strm->next) { + struct xz_dec_microlzma *state; + + if (strm->dict_size >= dict_size) + continue; + state = xz_dec_microlzma_alloc(XZ_PREALLOC, dict_size); + if (!state) { + err = -ENOMEM; + break; + } if (strm->state) xz_dec_microlzma_end(strm->state); - strm->state = xz_dec_microlzma_alloc(XZ_PREALLOC, dict_size); - if (!strm->state) - err = -ENOMEM; + strm->state = state; + strm->dict_size = dict_size; } /* 3. push back all to the global list and update max dict_size */ @@ -146,7 +155,8 @@ int z_erofs_load_lzma_config(struct super_block *sb, spin_unlock(&z_erofs_lzma_lock); wake_up_all(&z_erofs_lzma_wq); - z_erofs_lzma_max_dictsize = dict_size; + if (!err) + z_erofs_lzma_max_dictsize = dict_size; mutex_unlock(&lzma_resize_mutex); return err; } diff --git a/fs/erofs/internal.h b/fs/erofs/internal.h index 787cc9ff9029..7c439addf102 100644 --- a/fs/erofs/internal.h +++ b/fs/erofs/internal.h @@ -82,12 +82,8 @@ struct erofs_dev_context { bool flatdev; }; -/* all filesystem-wide lz4 configurations */ struct erofs_sb_lz4_info { - /* # of pages needed for EROFS lz4 rolling decompression */ - u16 max_distance_pages; - /* maximum possible blocks for pclusters in the filesystem */ - u16 max_pclusterblks; + u16 max_pclusterblks; /* maximum physical blocks for LZ4 pclusters */ }; struct erofs_domain { diff --git a/fs/erofs/sysfs.c b/fs/erofs/sysfs.c index 435e515c0792..92f0bc13623c 100644 --- a/fs/erofs/sysfs.c +++ b/fs/erofs/sysfs.c @@ -78,6 +78,7 @@ EROFS_ATTR_FEATURE(sb_chksum); EROFS_ATTR_FEATURE(ztailpacking); EROFS_ATTR_FEATURE(fragments); EROFS_ATTR_FEATURE(dedupe); +EROFS_ATTR_FEATURE(xattr_prefixes); static struct attribute *erofs_feat_attrs[] = { ATTR_LIST(zero_padding), @@ -90,6 +91,7 @@ static struct attribute *erofs_feat_attrs[] = { ATTR_LIST(ztailpacking), ATTR_LIST(fragments), ATTR_LIST(dedupe), + ATTR_LIST(xattr_prefixes), NULL, }; ATTRIBUTE_GROUPS(erofs_feat); diff --git a/fs/erofs/xattr.c b/fs/erofs/xattr.c index 09d341675e89..411f5f3edeb0 100644 --- a/fs/erofs/xattr.c +++ b/fs/erofs/xattr.c @@ -105,6 +105,7 @@ static int erofs_init_inode_xattrs(struct inode *inode) it.kaddr = erofs_bread(&it.buf, erofs_blknr(sb, it.pos), EROFS_KMAP); if (IS_ERR(it.kaddr)) { + erofs_put_metabuf(&it.buf); kfree(vi->xattr_shared_xattrs); vi->xattr_shared_xattrs = NULL; ret = PTR_ERR(it.kaddr); diff --git a/fs/erofs/zdata.c b/fs/erofs/zdata.c index 5b3eb4a22793..ede86dd93b5d 100644 --- a/fs/erofs/zdata.c +++ b/fs/erofs/zdata.c @@ -863,9 +863,20 @@ static int z_erofs_pcluster_begin(struct z_erofs_decompress_frontend *fe) if (!(map->m_flags & EROFS_MAP_META)) { grp = erofs_find_workgroup(sb, blknr); - } else if ((map->m_pa & ~PAGE_MASK) + map->m_plen > PAGE_SIZE) { - DBG_BUGON(1); - return -EFSCORRUPTED; + } else { + void *mptr; + + if ((map->m_pa & ~PAGE_MASK) + map->m_plen > PAGE_SIZE) { + DBG_BUGON(1); + return -EFSCORRUPTED; + } + + mptr = erofs_read_metabuf(&map->buf, sb, blknr, EROFS_NO_KMAP); + if (IS_ERR(mptr)) { + ret = PTR_ERR(mptr); + erofs_err(sb, "failed to get inline data %d", ret); + return ret; + } } if (grp) { @@ -888,14 +899,6 @@ static int z_erofs_pcluster_begin(struct z_erofs_decompress_frontend *fe) /* bind cache first when cached decompression is preferred */ z_erofs_bind_cache(fe); } else { - void *mptr; - - mptr = erofs_read_metabuf(&map->buf, sb, blknr, EROFS_NO_KMAP); - if (IS_ERR(mptr)) { - ret = PTR_ERR(mptr); - erofs_err(sb, "failed to get inline data %d", ret); - return ret; - } get_page(map->buf.page); WRITE_ONCE(fe->pcl->compressed_bvecs[0].page, map->buf.page); fe->pcl->pageofs_in = map->m_pa & ~PAGE_MASK; @@ -1427,6 +1430,9 @@ static void z_erofs_decompress_kickoff(struct z_erofs_decompressqueue *io, if (atomic_add_return(bios, &io->pending_bios)) return; if (z_erofs_in_atomic()) { + /* See `sync_decompress` in sysfs-fs-erofs for more details */ + if (sbi->opt.sync_decompress == EROFS_SYNC_DECOMPRESS_AUTO) + sbi->opt.sync_decompress = EROFS_SYNC_DECOMPRESS_FORCE_ON; #ifdef CONFIG_EROFS_FS_PCPU_KTHREAD struct kthread_worker *worker; @@ -1443,9 +1449,6 @@ static void z_erofs_decompress_kickoff(struct z_erofs_decompressqueue *io, #else queue_work(z_erofs_workqueue, &io->u.work); #endif - /* enable sync decompression for readahead */ - if (sbi->opt.sync_decompress == EROFS_SYNC_DECOMPRESS_AUTO) - sbi->opt.sync_decompress = EROFS_SYNC_DECOMPRESS_FORCE_ON; return; } gfp_flag = memalloc_noio_save(); diff --git a/fs/erofs/zmap.c b/fs/erofs/zmap.c index 76566c2cbf63..8c227ff5b847 100644 --- a/fs/erofs/zmap.c +++ b/fs/erofs/zmap.c @@ -12,7 +12,7 @@ struct z_erofs_maprecorder { struct erofs_map_blocks *map; void *kaddr; - unsigned long lcn; + u64 lcn; /* compression extent information gathered */ u8 type, headtype; u16 clusterofs; @@ -22,8 +22,7 @@ struct z_erofs_maprecorder { bool partialref; }; -static int z_erofs_load_full_lcluster(struct z_erofs_maprecorder *m, - unsigned long lcn) +static int z_erofs_load_full_lcluster(struct z_erofs_maprecorder *m, u64 lcn) { struct inode *const inode = m->inode; struct erofs_inode *const vi = EROFS_I(inode); @@ -67,10 +66,6 @@ static int z_erofs_load_full_lcluster(struct z_erofs_maprecorder *m, if (advise & Z_EROFS_LI_PARTIAL_REF) m->partialref = true; m->clusterofs = le16_to_cpu(di->di_clusterofs); - if (m->clusterofs >= 1 << vi->z_logical_clusterbits) { - DBG_BUGON(1); - return -EFSCORRUPTED; - } m->pblk = le32_to_cpu(di->di_u.blkaddr); break; default: @@ -223,7 +218,7 @@ static int unpack_compacted_index(struct z_erofs_maprecorder *m, } static int z_erofs_load_compact_lcluster(struct z_erofs_maprecorder *m, - unsigned long lcn, bool lookahead) + u64 lcn, bool lookahead) { struct inode *const inode = m->inode; struct erofs_inode *const vi = EROFS_I(inode); @@ -274,16 +269,36 @@ static int z_erofs_load_compact_lcluster(struct z_erofs_maprecorder *m, } static int z_erofs_load_lcluster_from_disk(struct z_erofs_maprecorder *m, - unsigned int lcn, bool lookahead) + u64 lcn, bool lookahead) { - switch (EROFS_I(m->inode)->datalayout) { + struct erofs_inode *vi = EROFS_I(m->inode); + int err; + + switch (vi->datalayout) { case EROFS_INODE_COMPRESSED_FULL: - return z_erofs_load_full_lcluster(m, lcn); + err = z_erofs_load_full_lcluster(m, lcn); + break; case EROFS_INODE_COMPRESSED_COMPACT: - return z_erofs_load_compact_lcluster(m, lcn, lookahead); + err = z_erofs_load_compact_lcluster(m, lcn, lookahead); + break; default: return -EINVAL; } + if (err) + return err; + + if (m->type >= Z_EROFS_LCLUSTER_TYPE_MAX) { + erofs_err(m->inode->i_sb, "unknown type %u @ lcn %llu of nid %llu", + m->type, lcn, vi->nid); + DBG_BUGON(1); + return -EOPNOTSUPP; + } + if (m->type != Z_EROFS_LCLUSTER_TYPE_NONHEAD && + m->clusterofs >= (1 << vi->z_logical_clusterbits)) { + DBG_BUGON(1); + return -EFSCORRUPTED; + } + return 0; } static int z_erofs_extent_lookback(struct z_erofs_maprecorder *m, @@ -294,7 +309,7 @@ static int z_erofs_extent_lookback(struct z_erofs_maprecorder *m, const unsigned int lclusterbits = vi->z_logical_clusterbits; while (m->lcn >= lookback_distance) { - unsigned long lcn = m->lcn - lookback_distance; + u64 lcn = m->lcn - lookback_distance; int err; err = z_erofs_load_lcluster_from_disk(m, lcn, false); @@ -314,14 +329,14 @@ static int z_erofs_extent_lookback(struct z_erofs_maprecorder *m, m->map->m_la = (lcn << lclusterbits) | m->clusterofs; return 0; default: - erofs_err(sb, "unknown type %u @ lcn %lu of nid %llu", + erofs_err(sb, "unknown type %u @ lcn %llu of nid %llu", m->type, lcn, vi->nid); DBG_BUGON(1); return -EOPNOTSUPP; } } err_bogus: - erofs_err(sb, "bogus lookback distance %u @ lcn %lu of nid %llu", + erofs_err(sb, "bogus lookback distance %u @ lcn %llu of nid %llu", lookback_distance, m->lcn, vi->nid); DBG_BUGON(1); return -EFSCORRUPTED; @@ -334,7 +349,7 @@ static int z_erofs_get_extent_compressedlen(struct z_erofs_maprecorder *m, struct erofs_inode *const vi = EROFS_I(m->inode); struct erofs_map_blocks *const map = m->map; const unsigned int lclusterbits = vi->z_logical_clusterbits; - unsigned long lcn; + u64 lcn; int err; DBG_BUGON(m->type != Z_EROFS_LCLUSTER_TYPE_PLAIN && @@ -386,7 +401,7 @@ static int z_erofs_get_extent_compressedlen(struct z_erofs_maprecorder *m, break; fallthrough; default: - erofs_err(sb, "cannot found CBLKCNT @ lcn %lu of nid %llu", lcn, + erofs_err(sb, "cannot found CBLKCNT @ lcn %llu of nid %llu", lcn, vi->nid); DBG_BUGON(1); return -EFSCORRUPTED; @@ -395,7 +410,7 @@ static int z_erofs_get_extent_compressedlen(struct z_erofs_maprecorder *m, map->m_plen = erofs_pos(sb, m->compressedblks); return 0; err_bonus_cblkcnt: - erofs_err(sb, "bogus CBLKCNT @ lcn %lu of nid %llu", lcn, vi->nid); + erofs_err(sb, "bogus CBLKCNT @ lcn %llu of nid %llu", lcn, vi->nid); DBG_BUGON(1); return -EFSCORRUPTED; } @@ -544,7 +559,8 @@ static int z_erofs_do_map_blocks(struct inode *inode, err = -EFSCORRUPTED; goto unmap_out; } - afmt = vi->z_advise & Z_EROFS_ADVISE_INTERLACED_PCLUSTER ? + afmt = (vi->z_advise & Z_EROFS_ADVISE_INTERLACED_PCLUSTER) && + !(map->m_flags & EROFS_MAP_META) ? Z_EROFS_COMPRESSION_INTERLACED : Z_EROFS_COMPRESSION_SHIFTED; } else { diff --git a/fs/eventfd.c b/fs/eventfd.c index 33a918f9566c..d2f7d2d8a351 100644 --- a/fs/eventfd.c +++ b/fs/eventfd.c @@ -72,22 +72,19 @@ __u64 eventfd_signal_mask(struct eventfd_ctx *ctx, __u64 n, __poll_t mask) } /** - * eventfd_signal - Adds @n to the eventfd counter. + * eventfd_signal - Increment the event counter * @ctx: [in] Pointer to the eventfd context. - * @n: [in] Value of the counter to be added to the eventfd internal counter. - * The value cannot be negative. * * This function is supposed to be called by the kernel in paths that do not * allow sleeping. In this function we allow the counter to reach the ULLONG_MAX * value, and we signal this as overflow condition by returning a EPOLLERR * to poll(2). * - * Returns the amount by which the counter was incremented. This will be less - * than @n if the counter has overflowed. + * Returns the amount by which the counter was incremented. */ -__u64 eventfd_signal(struct eventfd_ctx *ctx, __u64 n) +__u64 eventfd_signal(struct eventfd_ctx *ctx) { - return eventfd_signal_mask(ctx, n, 0); + return eventfd_signal_mask(ctx, 1, 0); } EXPORT_SYMBOL_GPL(eventfd_signal); diff --git a/fs/exfat/exfat_fs.h b/fs/exfat/exfat_fs.h index 96b1faa25aaa..ee843c7dba2a 100644 --- a/fs/exfat/exfat_fs.h +++ b/fs/exfat/exfat_fs.h @@ -552,6 +552,7 @@ void __exfat_fs_error(struct super_block *sb, int report, const char *fmt, ...) void exfat_get_entry_time(struct exfat_sb_info *sbi, struct timespec64 *ts, u8 tz, __le16 time, __le16 date, u8 time_cs); void exfat_truncate_atime(struct timespec64 *ts); +void exfat_truncate_inode_atime(struct inode *inode); void exfat_set_entry_time(struct exfat_sb_info *sbi, struct timespec64 *ts, u8 *tz, __le16 *time, __le16 *date, u8 *time_cs); u16 exfat_calc_chksum16(void *data, int len, u16 chksum, int type); diff --git a/fs/exfat/file.c b/fs/exfat/file.c index 32395ef686a2..30ee2c8d36a5 100644 --- a/fs/exfat/file.c +++ b/fs/exfat/file.c @@ -22,7 +22,7 @@ static int exfat_cont_expand(struct inode *inode, loff_t size) if (err) return err; - inode->i_mtime = inode_set_ctime_current(inode); + inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode)); mark_inode_dirty(inode); if (!IS_SYNC(inode)) @@ -290,10 +290,9 @@ int exfat_setattr(struct mnt_idmap *idmap, struct dentry *dentry, } if (attr->ia_valid & ATTR_SIZE) - inode->i_mtime = inode_set_ctime_current(inode); + inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode)); - setattr_copy(&nop_mnt_idmap, inode, attr); - exfat_truncate_atime(&inode->i_atime); + exfat_truncate_inode_atime(inode); if (attr->ia_valid & ATTR_SIZE) { error = exfat_block_truncate_page(inode, attr->ia_size); diff --git a/fs/exfat/inode.c b/fs/exfat/inode.c index 81019484296d..b6291969cd11 100644 --- a/fs/exfat/inode.c +++ b/fs/exfat/inode.c @@ -26,6 +26,7 @@ int __exfat_write_inode(struct inode *inode, int sync) struct exfat_sb_info *sbi = EXFAT_SB(sb); struct exfat_inode_info *ei = EXFAT_I(inode); bool is_dir = (ei->type == TYPE_DIR) ? true : false; + struct timespec64 ts; if (inode->i_ino == EXFAT_ROOT_INO) return 0; @@ -55,16 +56,18 @@ int __exfat_write_inode(struct inode *inode, int sync) &ep->dentry.file.create_time, &ep->dentry.file.create_date, &ep->dentry.file.create_time_cs); - exfat_set_entry_time(sbi, &inode->i_mtime, - &ep->dentry.file.modify_tz, - &ep->dentry.file.modify_time, - &ep->dentry.file.modify_date, - &ep->dentry.file.modify_time_cs); - exfat_set_entry_time(sbi, &inode->i_atime, - &ep->dentry.file.access_tz, - &ep->dentry.file.access_time, - &ep->dentry.file.access_date, - NULL); + exfat_set_entry_time(sbi, &ts, + &ep->dentry.file.modify_tz, + &ep->dentry.file.modify_time, + &ep->dentry.file.modify_date, + &ep->dentry.file.modify_time_cs); + inode_set_mtime_to_ts(inode, ts); + exfat_set_entry_time(sbi, &ts, + &ep->dentry.file.access_tz, + &ep->dentry.file.access_time, + &ep->dentry.file.access_date, + NULL); + inode_set_atime_to_ts(inode, ts); /* File size should be zero if there is no cluster allocated */ on_disk_size = i_size_read(inode); @@ -355,7 +358,7 @@ static void exfat_write_failed(struct address_space *mapping, loff_t to) if (to > i_size_read(inode)) { truncate_pagecache(inode, i_size_read(inode)); - inode->i_mtime = inode_set_ctime_current(inode); + inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode)); exfat_truncate(inode); } } @@ -398,7 +401,7 @@ static int exfat_write_end(struct file *file, struct address_space *mapping, exfat_write_failed(mapping, pos+len); if (!(err < 0) && !(ei->attr & ATTR_ARCHIVE)) { - inode->i_mtime = inode_set_ctime_current(inode); + inode_set_mtime_to_ts(inode, inode_set_ctime_current(inode)); ei->attr |= ATTR_ARCHIVE; mark_inode_dirty(inode); } @@ -576,10 +579,10 @@ static int exfat_fill_inode(struct inode *inode, struct exfat_dir_entry *info) exfat_save_attr(inode, info->attr); inode->i_blocks = round_up(i_size_read(inode), sbi->cluster_size) >> 9; - inode->i_mtime = info->mtime; + inode_set_mtime_to_ts(inode, info->mtime); inode_set_ctime_to_ts(inode, info->mtime); ei->i_crtime = info->crtime; - inode->i_atime = info->atime; + inode_set_atime_to_ts(inode, info->atime); return 0; } diff --git a/fs/exfat/misc.c b/fs/exfat/misc.c index 2e1a1a6b1021..fa8459828046 100644 --- a/fs/exfat/misc.c +++ b/fs/exfat/misc.c @@ -126,6 +126,14 @@ void exfat_truncate_atime(struct timespec64 *ts) ts->tv_nsec = 0; } +void exfat_truncate_inode_atime(struct inode *inode) +{ + struct timespec64 atime = inode_get_atime(inode); + + exfat_truncate_atime(&atime); + inode_set_atime_to_ts(inode, atime); +} + u16 exfat_calc_chksum16(void *data, int len, u16 chksum, int type) { int i; diff --git a/fs/exfat/namei.c b/fs/exfat/namei.c index 83a146d3c684..3549c80f927a 100644 --- a/fs/exfat/namei.c +++ b/fs/exfat/namei.c @@ -577,7 +577,7 @@ static int exfat_create(struct mnt_idmap *idmap, struct inode *dir, goto unlock; inode_inc_iversion(dir); - dir->i_mtime = inode_set_ctime_current(dir); + inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir)); if (IS_DIRSYNC(dir)) exfat_sync_inode(dir); else @@ -590,8 +590,9 @@ static int exfat_create(struct mnt_idmap *idmap, struct inode *dir, goto unlock; inode_inc_iversion(inode); - inode->i_mtime = inode->i_atime = EXFAT_I(inode)->i_crtime = inode_set_ctime_current(inode); - exfat_truncate_atime(&inode->i_atime); + EXFAT_I(inode)->i_crtime = simple_inode_init_ts(inode); + exfat_truncate_inode_atime(inode); + /* timestamp is already written, so mark_inode_dirty() is unneeded. */ d_instantiate(dentry, inode); @@ -656,14 +657,19 @@ static int exfat_find(struct inode *dir, struct qstr *qname, info->type = exfat_get_entry_type(ep); info->attr = le16_to_cpu(ep->dentry.file.attr); info->size = le64_to_cpu(ep2->dentry.stream.valid_size); + + info->start_clu = le32_to_cpu(ep2->dentry.stream.start_clu); + if (!is_valid_cluster(sbi, info->start_clu) && info->size) { + exfat_warn(sb, "start_clu is invalid cluster(0x%x)", + info->start_clu); + info->size = 0; + } + if (info->size == 0) { info->flags = ALLOC_NO_FAT_CHAIN; info->start_clu = EXFAT_EOF_CLUSTER; - } else { + } else info->flags = ep2->dentry.stream.flags; - info->start_clu = - le32_to_cpu(ep2->dentry.stream.start_clu); - } exfat_get_entry_time(sbi, &info->crtime, ep->dentry.file.create_tz, @@ -825,16 +831,16 @@ static int exfat_unlink(struct inode *dir, struct dentry *dentry) ei->dir.dir = DIR_DELETED; inode_inc_iversion(dir); - dir->i_mtime = dir->i_atime = inode_set_ctime_current(dir); - exfat_truncate_atime(&dir->i_atime); + simple_inode_init_ts(dir); + exfat_truncate_inode_atime(dir); if (IS_DIRSYNC(dir)) exfat_sync_inode(dir); else mark_inode_dirty(dir); clear_nlink(inode); - inode->i_mtime = inode->i_atime = inode_set_ctime_current(inode); - exfat_truncate_atime(&inode->i_atime); + simple_inode_init_ts(inode); + exfat_truncate_inode_atime(inode); exfat_unhash_inode(inode); exfat_d_version_set(dentry, inode_query_iversion(dir)); unlock: @@ -860,7 +866,7 @@ static int exfat_mkdir(struct mnt_idmap *idmap, struct inode *dir, goto unlock; inode_inc_iversion(dir); - dir->i_mtime = inode_set_ctime_current(dir); + inode_set_mtime_to_ts(dir, inode_set_ctime_current(dir)); if (IS_DIRSYNC(dir)) exfat_sync_inode(dir); else @@ -874,8 +880,8 @@ static int exfat_mkdir(struct mnt_idmap *idmap, struct inode *dir, goto unlock; inode_inc_iversion(inode); - inode->i_mtime = inode->i_atime = EXFAT_I(inode)->i_crtime = inode_set_ctime_current(inode); - exfat_truncate_atime(&inode->i_atime); + EXFAT_I(inode)->i_crtime = simple_inode_init_ts(inode); + exfat_truncate_inode_atime(inode); /* timestamp is already written, so mark_inode_dirty() is unneeded. */ d_instantiate(dentry, inode); @@ -992,8 +998,8 @@ static int exfat_rmdir(struct inode *dir, struct dentry *dentry) ei->dir.dir = DIR_DELETED; inode_inc_iversion(dir); - dir->i_mtime = dir->i_atime = inode_set_ctime_current(dir); - exfat_truncate_atime(&dir->i_atime); + simple_inode_init_ts(dir); + exfat_truncate_inode_atime(dir); if (IS_DIRSYNC(dir)) exfat_sync_inode(dir); else @@ -1001,8 +1007,8 @@ static int exfat_rmdir(struct inode *dir, struct dentry *dentry) drop_nlink(dir); clear_nlink(inode); - inode->i_mtime = inode->i_atime = inode_set_ctime_current(inode); - exfat_truncate_atime(&inode->i_atime); + simple_inode_init_ts(inode); + exfat_truncate_inode_atime(inode); exfat_unhash_inode(inode); exfat_d_version_set(dentry, inode_query_iversion(dir)); unlock: @@ -1368,7 +1374,7 @@ static int exfat_rename(struct mnt_idmap *idmap, inode_inc_iversion(new_dir); simple_rename_timestamp(old_dir, old_dentry, new_dir, new_dentry); EXFAT_I(new_dir)->i_crtime = current_time(new_dir); - exfat_truncate_atime(&new_dir->i_atime); + exfat_truncate_inode_atime(new_dir); if (IS_DIRSYNC(new_dir)) exfat_sync_inode(new_dir); else diff --git a/fs/exfat/nls.c b/fs/exfat/nls.c index 0e05c6b5a286..4bfe807716de 100644 --- a/fs/exfat/nls.c +++ b/fs/exfat/nls.c @@ -778,15 +778,23 @@ int exfat_create_upcase_table(struct super_block *sb) tbl_clu = le32_to_cpu(ep->dentry.upcase.start_clu); tbl_size = le64_to_cpu(ep->dentry.upcase.size); - - sector = exfat_cluster_to_sector(sbi, tbl_clu); - num_sectors = ((tbl_size - 1) >> blksize_bits) + 1; - ret = exfat_load_upcase_table(sb, sector, num_sectors, - le32_to_cpu(ep->dentry.upcase.checksum)); - + if (tbl_size) { + sector = exfat_cluster_to_sector(sbi, tbl_clu); + num_sectors = ((tbl_size - 1) >> blksize_bits) + 1; + ret = exfat_load_upcase_table(sb, sector, num_sectors, + le32_to_cpu(ep->dentry.upcase.checksum)); + } else { + exfat_fs_error(sb, + "bad upcase table size (0 bytes). Please run fsck"); + ret = -EINVAL; + } brelse(bh); - if (ret && ret != -EIO) + + if (ret && ret != -EIO) { + /* free memory from exfat_load_upcase_table call */ + exfat_free_upcase_table(sbi); goto load_default; + } /* load successfully */ return ret; @@ -796,6 +804,8 @@ int exfat_create_upcase_table(struct super_block *sb) return -EIO; } + exfat_fs_error(sb, "no upcase table entry. Please run fsck"); + load_default: /* load default upcase table */ return exfat_load_default_upcase_table(sb); diff --git a/fs/exfat/super.c b/fs/exfat/super.c index da3e03b059cf..25917a2e5f97 100644 --- a/fs/exfat/super.c +++ b/fs/exfat/super.c @@ -366,8 +366,8 @@ static int exfat_read_root(struct inode *inode, struct exfat_chain *root_clu) ei->i_size_ondisk = i_size_read(inode); exfat_save_attr(inode, ATTR_SUBDIR); - inode->i_mtime = inode->i_atime = ei->i_crtime = inode_set_ctime_current(inode); - exfat_truncate_atime(&inode->i_atime); + ei->i_crtime = simple_inode_init_ts(inode); + exfat_truncate_inode_atime(inode); return 0; } diff --git a/fs/ext2/inode.c b/fs/ext2/inode.c index 6eccee033617..e93db18b1466 100644 --- a/fs/ext2/inode.c +++ b/fs/ext2/inode.c @@ -1302,11 +1302,10 @@ static int ext2_setsize(struct inode *inode, loff_t newsize) filemap_invalidate_unlock(inode->i_mapping); inode->i_mtime = inode_set_ctime_current(inode); + mark_inode_dirty(inode); if (inode_needs_sync(inode)) { sync_mapping_buffers(inode->i_mapping); sync_inode_metadata(inode, 1); - } else { - mark_inode_dirty(inode); } return 0; diff --git a/fs/ext2/xattr.c b/fs/ext2/xattr.c index 20f741184673..c9706d61e6dd 100644 --- a/fs/ext2/xattr.c +++ b/fs/ext2/xattr.c @@ -777,6 +777,7 @@ ext2_xattr_set2(struct inode *inode, struct buffer_head *old_bh, /* Update the inode. */ EXT2_I(inode)->i_file_acl = new_bh ? new_bh->b_blocknr : 0; inode_set_ctime_current(inode); + mark_inode_dirty(inode); if (IS_SYNC(inode)) { error = sync_inode_metadata(inode, 1); /* In case sync failed due to ENOSPC the inode was actually @@ -789,8 +790,7 @@ ext2_xattr_set2(struct inode *inode, struct buffer_head *old_bh, } goto cleanup; } - } else - mark_inode_dirty(inode); + } error = 0; if (old_bh && old_bh != new_bh) { diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 387509da86a2..5654a1cd2b4a 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -1816,7 +1816,8 @@ static inline int ext4_valid_inum(struct super_block *sb, unsigned long ino) */ enum { EXT4_MF_MNTDIR_SAMPLED, - EXT4_MF_FC_INELIGIBLE /* Fast commit ineligible */ + EXT4_MF_FC_INELIGIBLE, /* Fast commit ineligible */ + EXT4_MF_JOURNAL_DESTROY /* Journal is in process of destroying */ }; static inline void ext4_set_mount_flag(struct super_block *sb, int bit) @@ -2978,7 +2979,8 @@ typedef enum { EXT4_IGET_SPECIAL = 0x0001, /* OK to iget a system inode */ EXT4_IGET_HANDLE = 0x0002, /* Inode # is from a handle */ EXT4_IGET_BAD = 0x0004, /* Allow to iget a bad inode */ - EXT4_IGET_EA_INODE = 0x0008 /* Inode should contain an EA value */ + EXT4_IGET_EA_INODE = 0x0008, /* Inode should contain an EA value */ + EXT4_IGET_NOWAIT = 0x0010 /* Non-blocking lookup (skip if freeing) */ } ext4_iget_flags; extern struct inode *__ext4_iget(struct super_block *sb, unsigned long ino, diff --git a/fs/ext4/ext4_jbd2.h b/fs/ext4/ext4_jbd2.h index 0c77697d5e90..ada46189b086 100644 --- a/fs/ext4/ext4_jbd2.h +++ b/fs/ext4/ext4_jbd2.h @@ -513,4 +513,33 @@ static inline int ext4_should_dioread_nolock(struct inode *inode) return 1; } +/* + * Pass journal explicitly as it may not be cached in the sbi->s_journal in some + * cases + */ +static inline int ext4_journal_destroy(struct ext4_sb_info *sbi, journal_t *journal) +{ + int err = 0; + + /* + * At this point only two things can be operating on the journal. + * JBD2 thread performing transaction commit and s_sb_upd_work + * issuing sb update through the journal. Once we set + * EXT4_JOURNAL_DESTROY, new ext4_handle_error() calls will not + * queue s_sb_upd_work and ext4_force_commit() makes sure any + * ext4_handle_error() calls from the running transaction commit are + * finished. Hence no new s_sb_upd_work can be queued after we + * flush it here. + */ + ext4_set_mount_flag(sbi->s_sb, EXT4_MF_JOURNAL_DESTROY); + + ext4_force_commit(sbi->s_sb); + flush_work(&sbi->s_sb_upd_work); + + err = jbd2_journal_destroy(journal); + sbi->s_journal = NULL; + + return err; +} + #endif /* _EXT4_JBD2_H */ diff --git a/fs/ext4/inode.c b/fs/ext4/inode.c index dea6d27e5028..4253c9a757e9 100644 --- a/fs/ext4/inode.c +++ b/fs/ext4/inode.c @@ -4823,6 +4823,17 @@ static int check_igot_inode(struct inode *inode, ext4_iget_flags flags, return -EFSCORRUPTED; } +static int ext4_iget_match(struct inode *inode, unsigned long ino, void *data) +{ + if (inode->i_ino != ino) + return 0; + if (READ_ONCE(inode->i_state) & I_CREATING) + return -1; + if (!igrab(inode)) + return -1; + return 1; +} + struct inode *__ext4_iget(struct super_block *sb, unsigned long ino, ext4_iget_flags flags, const char *function, unsigned int line) @@ -4856,9 +4867,24 @@ struct inode *__ext4_iget(struct super_block *sb, unsigned long ino, return ERR_PTR(-EFSCORRUPTED); } - inode = iget_locked(sb, ino); - if (!inode) - return ERR_PTR(-ENOMEM); + if (flags & EXT4_IGET_NOWAIT) { + inode = find_inode_nowait(sb, ino, ext4_iget_match, NULL); + if (!inode) + return ERR_PTR(-ENOENT); + + if (inode->i_state & I_NEW) + wait_on_inode(inode); + + if (unlikely(inode_unhashed(inode))) { + iput(inode); + return ERR_PTR(-ENOENT); + } + } else { + inode = iget_locked(sb, ino); + if (!inode) + return ERR_PTR(-ENOMEM); + } + if (!(inode->i_state & I_NEW)) { ret = check_igot_inode(inode, flags, function, line); if (ret) { @@ -5484,10 +5510,6 @@ int ext4_setattr(struct mnt_idmap *idmap, struct dentry *dentry, if (error) return error; - error = fsverity_prepare_setattr(dentry, attr); - if (error) - return error; - if (is_quota_modification(idmap, inode, attr)) { error = dquot_initialize(inode); if (error) diff --git a/fs/ext4/ioctl.c b/fs/ext4/ioctl.c index e27eae7f163d..242974bf3c0a 100644 --- a/fs/ext4/ioctl.c +++ b/fs/ext4/ioctl.c @@ -1373,6 +1373,11 @@ static long __ext4_ioctl(struct file *filp, unsigned int cmd, unsigned long arg) goto mext_out; } + if (file_inode(filp)->i_sb != file_inode(donor.file)->i_sb) { + err = -EXDEV; + goto mext_out; + } + err = mnt_want_write_file(filp); if (err) goto mext_out; diff --git a/fs/ext4/super.c b/fs/ext4/super.c index 045c1cf3c596..349e2b087769 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -728,9 +728,13 @@ static void ext4_handle_error(struct super_block *sb, bool force_ro, int error, * In case the fs should keep running, we need to writeout * superblock through the journal. Due to lock ordering * constraints, it may not be safe to do it right here so we - * defer superblock flushing to a workqueue. + * defer superblock flushing to a workqueue. We just need to be + * careful when the journal is already shutting down. If we get + * here in that case, just update the sb directly as the last + * transaction won't commit anyway. */ - if (continue_fs && journal) + if (continue_fs && journal && + !ext4_test_mount_flag(sb, EXT4_MF_JOURNAL_DESTROY)) schedule_work(&EXT4_SB(sb)->s_sb_upd_work); else ext4_commit_super(sb); @@ -1311,18 +1315,17 @@ static void ext4_put_super(struct super_block *sb) ext4_unregister_li_request(sb); ext4_quotas_off(sb, EXT4_MAXQUOTAS); - flush_work(&sbi->s_sb_upd_work); destroy_workqueue(sbi->rsv_conversion_wq); ext4_release_orphan_info(sb); if (sbi->s_journal) { aborted = is_journal_aborted(sbi->s_journal); - err = jbd2_journal_destroy(sbi->s_journal); - sbi->s_journal = NULL; + err = ext4_journal_destroy(sbi, sbi->s_journal); if ((err < 0) && !aborted) { ext4_abort(sb, -err, "Couldn't clean up the journal"); } - } + } else + flush_work(&sbi->s_sb_upd_work); ext4_es_unregister_shrinker(sbi); timer_shutdown_sync(&sbi->s_err_report); @@ -4993,10 +4996,7 @@ static int ext4_load_and_init_journal(struct super_block *sb, return 0; out: - /* flush s_sb_upd_work before destroying the journal. */ - flush_work(&sbi->s_sb_upd_work); - jbd2_journal_destroy(sbi->s_journal); - sbi->s_journal = NULL; + ext4_journal_destroy(sbi, sbi->s_journal); return -EINVAL; } @@ -5381,7 +5381,7 @@ static int __ext4_fill_super(struct fs_context *fc, struct super_block *sb) sb->s_qcop = &ext4_qctl_operations; sb->s_quota_types = QTYPE_MASK_USR | QTYPE_MASK_GRP | QTYPE_MASK_PRJ; #endif - memcpy(&sb->s_uuid, es->s_uuid, sizeof(es->s_uuid)); + super_set_uuid(sb, es->s_uuid, sizeof(es->s_uuid)); INIT_LIST_HEAD(&sbi->s_orphan); /* unlinked but open files */ mutex_init(&sbi->s_orphan_lock); @@ -5684,10 +5684,7 @@ failed_mount8: __maybe_unused sbi->s_ea_block_cache = NULL; if (sbi->s_journal) { - /* flush s_sb_upd_work before journal destroy. */ - flush_work(&sbi->s_sb_upd_work); - jbd2_journal_destroy(sbi->s_journal); - sbi->s_journal = NULL; + ext4_journal_destroy(sbi, sbi->s_journal); } failed_mount3a: ext4_es_unregister_shrinker(sbi); @@ -5995,7 +5992,7 @@ static journal_t *ext4_open_dev_journal(struct super_block *sb, return journal; out_journal: - jbd2_journal_destroy(journal); + ext4_journal_destroy(EXT4_SB(sb), journal); out_bdev: blkdev_put(journal_bdev, sb); return ERR_PTR(errno); @@ -6112,8 +6109,7 @@ static int ext4_load_journal(struct super_block *sb, EXT4_SB(sb)->s_journal = journal; err = ext4_clear_journal_err(sb, es); if (err) { - EXT4_SB(sb)->s_journal = NULL; - jbd2_journal_destroy(journal); + ext4_journal_destroy(EXT4_SB(sb), journal); return err; } @@ -6131,7 +6127,7 @@ static int ext4_load_journal(struct super_block *sb, return 0; err_out: - jbd2_journal_destroy(journal); + ext4_journal_destroy(EXT4_SB(sb), journal); return err; } diff --git a/fs/ext4/xattr.c b/fs/ext4/xattr.c index 5d068b2905e4..03755aed501b 100644 --- a/fs/ext4/xattr.c +++ b/fs/ext4/xattr.c @@ -1550,7 +1550,7 @@ ext4_xattr_inode_cache_find(struct inode *inode, const void *value, while (ce) { ea_inode = ext4_iget(inode->i_sb, ce->e_value, - EXT4_IGET_EA_INODE); + EXT4_IGET_EA_INODE | EXT4_IGET_NOWAIT); if (IS_ERR(ea_inode)) goto next_entry; ext4_xattr_inode_set_class(ea_inode); diff --git a/fs/f2fs/checkpoint.c b/fs/f2fs/checkpoint.c index c629b5580b72..74bdce66f866 100644 --- a/fs/f2fs/checkpoint.c +++ b/fs/f2fs/checkpoint.c @@ -99,7 +99,7 @@ static struct page *__get_meta_page(struct f2fs_sb_info *sbi, pgoff_t index, } if (unlikely(!PageUptodate(page))) { - f2fs_handle_page_eio(sbi, page->index, META); + f2fs_handle_page_eio(sbi, page_folio(page), META); f2fs_put_page(page, 1); return ERR_PTR(-EIO); } diff --git a/fs/f2fs/compress.c b/fs/f2fs/compress.c index c9758fe396a5..7a7e6f8e6970 100644 --- a/fs/f2fs/compress.c +++ b/fs/f2fs/compress.c @@ -1114,6 +1114,7 @@ static int prepare_compress_overwrite(struct compress_ctx *cc, f2fs_compress_ctx_add_page(cc, page); if (!PageUptodate(page)) { + f2fs_handle_page_eio(sbi, page_folio(page), DATA); release_and_retry: f2fs_put_rpages(cc); f2fs_unlock_rpages(cc, i + 1); diff --git a/fs/f2fs/data.c b/fs/f2fs/data.c index d068111436d7..b0c633c5deb7 100644 --- a/fs/f2fs/data.c +++ b/fs/f2fs/data.c @@ -2216,6 +2216,12 @@ int f2fs_read_multi_pages(struct compress_ctx *cc, struct bio **bio_ret, int i; int ret = 0; + if (unlikely(f2fs_cp_error(sbi))) { + ret = -EIO; + from_dnode = false; + goto out_put_dnode; + } + f2fs_bug_on(sbi, f2fs_cluster_is_empty(cc)); last_block_in_file = bytes_to_blks(inode, @@ -2256,10 +2262,6 @@ int f2fs_read_multi_pages(struct compress_ctx *cc, struct bio **bio_ret, if (ret) goto out; - if (unlikely(f2fs_cp_error(sbi))) { - ret = -EIO; - goto out_put_dnode; - } f2fs_bug_on(sbi, dn.data_blkaddr != COMPRESS_ADDR); skip_reading_dnode: @@ -3549,7 +3551,7 @@ static int __reserve_data_block(struct inode *inode, pgoff_t index, static int prepare_atomic_write_begin(struct f2fs_sb_info *sbi, struct page *page, loff_t pos, unsigned int len, - block_t *blk_addr, bool *node_changed, bool *use_cow) + block_t *blk_addr, bool *node_changed) { struct inode *inode = page->mapping->host; struct inode *cow_inode = F2FS_I(inode)->cow_inode; @@ -3564,14 +3566,14 @@ static int prepare_atomic_write_begin(struct f2fs_sb_info *sbi, /* Look for the block in COW inode first */ err = __find_data_block(cow_inode, index, blk_addr); - if (err) { + if (err) return err; - } else if (__is_valid_data_blkaddr(*blk_addr)) { - *use_cow = true; + + if (__is_valid_data_blkaddr(*blk_addr)) return 0; - } else if (*blk_addr == NEW_ADDR) { + + if (*blk_addr == NEW_ADDR) cow_has_reserved_block = true; - } if (is_inode_flag_set(inode, FI_ATOMIC_REPLACE)) goto reserve_block; @@ -3604,7 +3606,6 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping, struct page *page = NULL; pgoff_t index = ((unsigned long long) pos) >> PAGE_SHIFT; bool need_balance = false; - bool use_cow = false; block_t blkaddr = NULL_ADDR; int err = 0; @@ -3664,7 +3665,7 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping, if (f2fs_is_atomic_file(inode)) err = prepare_atomic_write_begin(sbi, page, pos, len, - &blkaddr, &need_balance, &use_cow); + &blkaddr, &need_balance); else err = prepare_write_begin(sbi, page, pos, len, &blkaddr, &need_balance); @@ -3704,9 +3705,15 @@ static int f2fs_write_begin(struct file *file, struct address_space *mapping, f2fs_handle_error(sbi, ERROR_INVALID_BLKADDR); goto fail; } - err = f2fs_submit_page_read(use_cow ? - F2FS_I(inode)->cow_inode : inode, page, - blkaddr, 0, true); + /* + * Although the block may be stored in the COW inode, the folio + * belongs to @inode and its data was encrypted (or not) using + * @inode's context (see f2fs_encrypt_one_page()). Read with + * @inode so the post-read decryption decision matches the + * folio's owner; otherwise an unencrypted @inode whose COW inode + * is encrypted hits a NULL ->i_crypt_info on decryption. + */ + err = f2fs_submit_page_read(inode, page, blkaddr, 0, true); if (err) goto fail; diff --git a/fs/f2fs/debug.c b/fs/f2fs/debug.c index 68773b48ed4d..eeb37785de7c 100644 --- a/fs/f2fs/debug.c +++ b/fs/f2fs/debug.c @@ -286,10 +286,6 @@ static void update_mem_info(struct f2fs_sb_info *sbi) get_cache: si->cache_mem = 0; - /* build gc */ - if (sbi->gc_thread) - si->cache_mem += sizeof(struct f2fs_gc_kthread); - /* build merge flush thread */ if (SM_I(sbi)->fcc_info) si->cache_mem += sizeof(struct flush_cmd_control); diff --git a/fs/f2fs/dir.c b/fs/f2fs/dir.c index 13268d1f8c75..570f79fa65a5 100644 --- a/fs/f2fs/dir.c +++ b/fs/f2fs/dir.c @@ -341,6 +341,7 @@ static struct f2fs_dir_entry *find_in_level(struct inode *dir, de = find_in_block(dir, dentry_page, fname, &max_slots, use_hash); if (IS_ERR(de)) { + f2fs_put_page(dentry_page, 0); *res_page = ERR_CAST(de); de = NULL; break; diff --git a/fs/f2fs/f2fs.h b/fs/f2fs/f2fs.h index 63d3450f895e..d0d5bb469d8d 100644 --- a/fs/f2fs/f2fs.h +++ b/fs/f2fs/f2fs.h @@ -1556,6 +1556,26 @@ struct decompress_io_ctx { #define MAX_COMPRESS_LOG_SIZE 8 #define MAX_COMPRESS_WINDOW_SIZE(log_size) ((PAGE_SIZE) << (log_size)) +struct f2fs_gc_kthread { + struct task_struct *f2fs_gc_task; + wait_queue_head_t gc_wait_queue_head; + + /* for gc sleep time */ + unsigned int urgent_sleep_time; + unsigned int min_sleep_time; + unsigned int max_sleep_time; + unsigned int no_gc_sleep_time; + + /* for changing gc mode */ + bool gc_wake; + + /* for GC_MERGE mount option */ + wait_queue_head_t fggc_wq; /* + * caller of f2fs_balance_fs() + * will wait on this wait queue. + */ +}; + struct f2fs_sb_info { struct super_block *sb; /* pointer to VFS super block */ struct proc_dir_entry *s_proc; /* proc entry */ @@ -1677,7 +1697,7 @@ struct f2fs_sb_info { * semaphore for GC, avoid * race between GC and GC or CP */ - struct f2fs_gc_kthread *gc_thread; /* GC thread */ + struct f2fs_gc_kthread gc_thread; /* GC thread */ struct atgc_management am; /* atgc management */ unsigned int cur_victim_sec; /* current victim section num */ unsigned int gc_mode; /* current GC state */ @@ -3921,7 +3941,7 @@ int f2fs_gc(struct f2fs_sb_info *sbi, struct f2fs_gc_control *gc_control); void f2fs_build_gc_manager(struct f2fs_sb_info *sbi); int f2fs_gc_range(struct f2fs_sb_info *sbi, unsigned int start_seg, unsigned int end_seg, - bool dry_run, unsigned int dry_run_sections); + bool dry_run, unsigned int dry_run_sections, bool lock); int f2fs_resize_fs(struct file *filp, __u64 block_count); int __init f2fs_create_garbage_collection_cache(void); void f2fs_destroy_garbage_collection_cache(void); @@ -4698,9 +4718,11 @@ static inline void f2fs_io_schedule_timeout(long timeout) io_schedule_timeout(timeout); } -static inline void f2fs_handle_page_eio(struct f2fs_sb_info *sbi, pgoff_t ofs, - enum page_type type) +static inline void f2fs_handle_page_eio(struct f2fs_sb_info *sbi, + struct folio *folio, enum page_type type) { + pgoff_t ofs = folio->index; + if (unlikely(f2fs_cp_error(sbi))) return; diff --git a/fs/f2fs/file.c b/fs/f2fs/file.c index 7125117b6322..793b4c89a9df 100644 --- a/fs/f2fs/file.c +++ b/fs/f2fs/file.c @@ -1065,10 +1065,6 @@ int f2fs_setattr(struct mnt_idmap *idmap, struct dentry *dentry, if (err) return err; - err = fsverity_prepare_setattr(dentry, attr); - if (err) - return err; - if (unlikely(IS_IMMUTABLE(inode))) return -EPERM; @@ -3065,8 +3061,9 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in, struct inode *src = file_inode(file_in); struct inode *dst = file_inode(file_out); struct f2fs_sb_info *sbi = F2FS_I_SB(src); - size_t olen = len, dst_max_i_size = 0; - size_t dst_osize; + size_t olen = len; + loff_t dst_max_i_size = 0; + loff_t dst_osize, dst_end; int ret; if (file_in->f_path.mnt != file_out->f_path.mnt || @@ -3123,8 +3120,15 @@ static int f2fs_move_file_range(struct file *file_in, loff_t pos_in, } dst_osize = dst->i_size; - if (pos_out + olen > dst->i_size) - dst_max_i_size = pos_out + olen; + if (olen > LLONG_MAX - pos_out) + goto out_unlock; + dst_end = pos_out + olen; + if (dst_end > dst->i_size) { + ret = inode_newsize_ok(dst, dst_end); + if (ret) + goto out_unlock; + dst_max_i_size = dst_end; + } /* verify the end result is block aligned */ if (!IS_ALIGNED(pos_in, F2FS_BLKSIZE) || diff --git a/fs/f2fs/gc.c b/fs/f2fs/gc.c index ac21950f53cd..663050f4bc19 100644 --- a/fs/f2fs/gc.c +++ b/fs/f2fs/gc.c @@ -31,9 +31,9 @@ static unsigned int count_bits(const unsigned long *addr, static int gc_thread_func(void *data) { struct f2fs_sb_info *sbi = data; - struct f2fs_gc_kthread *gc_th = sbi->gc_thread; - wait_queue_head_t *wq = &sbi->gc_thread->gc_wait_queue_head; - wait_queue_head_t *fggc_wq = &sbi->gc_thread->fggc_wq; + struct f2fs_gc_kthread *gc_th = &sbi->gc_thread; + wait_queue_head_t *wq = &sbi->gc_thread.gc_wait_queue_head; + wait_queue_head_t *fggc_wq = &sbi->gc_thread.fggc_wq; unsigned int wait_ms; struct f2fs_gc_control gc_control = { .victim_segno = NULL_SEGNO, @@ -171,13 +171,9 @@ static int gc_thread_func(void *data) int f2fs_start_gc_thread(struct f2fs_sb_info *sbi) { - struct f2fs_gc_kthread *gc_th; + struct f2fs_gc_kthread *gc_th = &sbi->gc_thread; dev_t dev = sbi->sb->s_bdev->bd_dev; - gc_th = f2fs_kmalloc(sbi, sizeof(struct f2fs_gc_kthread), GFP_KERNEL); - if (!gc_th) - return -ENOMEM; - gc_th->urgent_sleep_time = DEF_GC_THREAD_URGENT_SLEEP_TIME; gc_th->min_sleep_time = DEF_GC_THREAD_MIN_SLEEP_TIME; gc_th->max_sleep_time = DEF_GC_THREAD_MAX_SLEEP_TIME; @@ -185,16 +181,14 @@ int f2fs_start_gc_thread(struct f2fs_sb_info *sbi) gc_th->gc_wake = false; - sbi->gc_thread = gc_th; - init_waitqueue_head(&sbi->gc_thread->gc_wait_queue_head); - init_waitqueue_head(&sbi->gc_thread->fggc_wq); - sbi->gc_thread->f2fs_gc_task = kthread_run(gc_thread_func, sbi, + init_waitqueue_head(&gc_th->gc_wait_queue_head); + init_waitqueue_head(&gc_th->fggc_wq); + gc_th->f2fs_gc_task = kthread_run(gc_thread_func, sbi, "f2fs_gc-%u:%u", MAJOR(dev), MINOR(dev)); if (IS_ERR(gc_th->f2fs_gc_task)) { int err = PTR_ERR(gc_th->f2fs_gc_task); - kfree(gc_th); - sbi->gc_thread = NULL; + gc_th->f2fs_gc_task = NULL; return err; } @@ -203,14 +197,14 @@ int f2fs_start_gc_thread(struct f2fs_sb_info *sbi) void f2fs_stop_gc_thread(struct f2fs_sb_info *sbi) { - struct f2fs_gc_kthread *gc_th = sbi->gc_thread; + struct f2fs_gc_kthread *gc_th = &sbi->gc_thread; - if (!gc_th) + if (!gc_th->f2fs_gc_task) return; + kthread_stop(gc_th->f2fs_gc_task); + gc_th->f2fs_gc_task = NULL; wake_up_all(&gc_th->fggc_wq); - kfree(gc_th); - sbi->gc_thread = NULL; } static int select_gc_type(struct f2fs_sb_info *sbi, int gc_type) @@ -1187,7 +1181,7 @@ static int ra_data_block(struct inode *inode, pgoff_t index) .encrypted_page = NULL, .in_list = 0, }; - int err; + int err = 0; f2fs_down_read(&F2FS_I(inode)->i_sem); if (f2fs_is_cow_file(inode)) { @@ -1254,6 +1248,9 @@ static int ra_data_block(struct inode *inode, pgoff_t index) goto put_page; } + if (PageUptodate(fio.encrypted_page)) + goto put_encrypted_page; + err = f2fs_submit_page_bio(&fio); if (err) goto put_encrypted_page; @@ -2034,7 +2031,7 @@ void f2fs_build_gc_manager(struct f2fs_sb_info *sbi) int f2fs_gc_range(struct f2fs_sb_info *sbi, unsigned int start_seg, unsigned int end_seg, - bool dry_run, unsigned int dry_run_sections) + bool dry_run, unsigned int dry_run_sections, bool lock) { unsigned int segno; unsigned int gc_secs = dry_run_sections; @@ -2045,29 +2042,47 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi, .ilist = LIST_HEAD_INIT(gc_list.ilist), .iroot = RADIX_TREE_INIT(gc_list.iroot, GFP_NOFS), }; + int err = 0; + + if (lock) + f2fs_down_write(&sbi->gc_lock); /* * avoid migrating empty section, as it can be allocated by * log in parallel. */ if (!get_valid_blocks(sbi, segno, true)) - continue; + goto next; if (IS_CURSEC(sbi, GET_SEC_FROM_SEG(sbi, segno))) - continue; + goto next; do_garbage_collect(sbi, segno, &gc_list, FG_GC, dry_run_sections == 0); put_gc_inode(&gc_list); - if (!dry_run && get_valid_blocks(sbi, segno, true)) - return -EAGAIN; + /* reset all pinned status during fggc */ + f2fs_unpin_all_sections(sbi, true); + + if (!dry_run && get_valid_blocks(sbi, segno, true)) { + err = -EAGAIN; + goto next; + } if (dry_run && dry_run_sections && - !get_valid_blocks(sbi, segno, true) && --gc_secs == 0) - break; + !get_valid_blocks(sbi, segno, true)) { + --gc_secs; + goto next; + } if (fatal_signal_pending(current)) - return -ERESTARTSYS; + err = -ERESTARTSYS; +next: + if (lock) + f2fs_up_write(&sbi->gc_lock); + if (err) + return err; + if (dry_run && dry_run_sections && !gc_secs) + return 0; } return 0; @@ -2076,9 +2091,10 @@ int f2fs_gc_range(struct f2fs_sb_info *sbi, static int free_segment_range(struct f2fs_sb_info *sbi, unsigned int secs, bool dry_run) { - unsigned int next_inuse, start, end; + unsigned int secno, next_inuse, start, end, end_secno; struct cp_control cpc = { CP_RESIZE, 0, 0, 0 }; int gc_mode, gc_type; + unsigned int freed_secs = 0; int err = 0; int type; @@ -2086,6 +2102,7 @@ static int free_segment_range(struct f2fs_sb_info *sbi, MAIN_SECS(sbi) -= secs; start = MAIN_SECS(sbi) * SEGS_PER_SEC(sbi); end = MAIN_SEGS(sbi) - 1; + end_secno = GET_SEC_FROM_SEG(sbi, end); mutex_lock(&DIRTY_I(sbi)->seglist_lock); for (gc_mode = 0; gc_mode < MAX_GC_POLICY; gc_mode++) @@ -2097,12 +2114,20 @@ static int free_segment_range(struct f2fs_sb_info *sbi, sbi->next_victim_seg[gc_type] = NULL_SEGNO; mutex_unlock(&DIRTY_I(sbi)->seglist_lock); + spin_lock(&FREE_I(sbi)->segmap_lock); + for (secno = MAIN_SECS(sbi); secno <= end_secno; secno++) { + if (!test_bit(secno, FREE_I(sbi)->free_secmap)) + freed_secs++; + } + FREE_I(sbi)->free_sections -= freed_secs; + spin_unlock(&FREE_I(sbi)->segmap_lock); + /* Move out cursegs from the target range */ - for (type = CURSEG_HOT_DATA; type < NR_CURSEG_PERSIST_TYPE; type++) + for (type = CURSEG_HOT_DATA; type < NR_CURSEG_TYPE; type++) f2fs_allocate_segment_for_resize(sbi, type, start, end); /* do GC to move out valid blocks in the range */ - err = f2fs_gc_range(sbi, start, end, dry_run, 0); + err = f2fs_gc_range(sbi, start, end, dry_run, 0, false); if (err || dry_run) goto out; @@ -2118,6 +2143,9 @@ static int free_segment_range(struct f2fs_sb_info *sbi, f2fs_bug_on(sbi, 1); } out: + spin_lock(&FREE_I(sbi)->segmap_lock); + FREE_I(sbi)->free_sections += freed_secs; + spin_unlock(&FREE_I(sbi)->segmap_lock); MAIN_SECS(sbi) += secs; return err; } diff --git a/fs/f2fs/gc.h b/fs/f2fs/gc.h index 28a00942802c..0ff8138f0174 100644 --- a/fs/f2fs/gc.h +++ b/fs/f2fs/gc.h @@ -32,26 +32,6 @@ #define NR_GC_CHECKPOINT_SECS (3) /* data/node/dentry sections */ -struct f2fs_gc_kthread { - struct task_struct *f2fs_gc_task; - wait_queue_head_t gc_wait_queue_head; - - /* for gc sleep time */ - unsigned int urgent_sleep_time; - unsigned int min_sleep_time; - unsigned int max_sleep_time; - unsigned int no_gc_sleep_time; - - /* for changing gc mode */ - bool gc_wake; - - /* for GC_MERGE mount option */ - wait_queue_head_t fggc_wq; /* - * caller of f2fs_balance_fs() - * will wait on this wait queue. - */ -}; - struct gc_inode_list { struct list_head ilist; struct radix_tree_root iroot; diff --git a/fs/f2fs/node.c b/fs/f2fs/node.c index 586a90643ddd..8aa52629f788 100644 --- a/fs/f2fs/node.c +++ b/fs/f2fs/node.c @@ -419,6 +419,7 @@ bool f2fs_need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino) struct nat_entry *e; bool need_update = true; + f2fs_down_read(&sbi->node_write); f2fs_down_read(&nm_i->nat_tree_lock); e = __lookup_nat_cache(nm_i, ino); if (e && get_nat_flag(e, HAS_LAST_FSYNC) && @@ -426,6 +427,7 @@ bool f2fs_need_inode_block_update(struct f2fs_sb_info *sbi, nid_t ino) get_nat_flag(e, HAS_FSYNCED_INODE))) need_update = false; f2fs_up_read(&nm_i->nat_tree_lock); + f2fs_up_read(&sbi->node_write); return need_update; } @@ -1515,11 +1517,11 @@ static struct page *__get_node_page(struct f2fs_sb_info *sbi, pgoff_t nid, f2fs_handle_error(sbi, ERROR_INCONSISTENT_FOOTER); err = -EFSCORRUPTED; out_err: - ClearPageUptodate(page); + clear_node_page_dirty(page); out_put_err: /* ENOENT comes from read_node_page which is not an error. */ if (err != -ENOENT) - f2fs_handle_page_eio(sbi, page->index, NODE); + f2fs_handle_page_eio(sbi, page_folio(page), NODE); f2fs_put_page(page, 1); return ERR_PTR(err); } @@ -1630,9 +1632,10 @@ static struct page *last_fsync_dnode(struct f2fs_sb_info *sbi, nid_t ino) return last_page; } -static int __write_node_page(struct page *page, bool atomic, bool *submitted, - struct writeback_control *wbc, bool do_balance, - enum iostat_type io_type, unsigned int *seq_id) +static int __write_node_page(struct page *page, bool atomic, bool do_fsync, + bool *submitted, struct writeback_control *wbc, + bool do_balance, enum iostat_type io_type, + unsigned int *seq_id) { struct f2fs_sb_info *sbi = F2FS_P_SB(page); nid_t nid; @@ -1704,11 +1707,14 @@ static int __write_node_page(struct page *page, bool atomic, bool *submitted, if (atomic) { if (!test_opt(sbi, NOBARRIER) && !f2fs_sb_has_blkzoned(sbi)) fio.op_flags |= REQ_PREFLUSH | REQ_FUA; - if (IS_INODE(page)) - set_dentry_mark(page, - f2fs_need_dentry_mark(sbi, ino_of_node(page))); } + set_dentry_mark(page, false); + set_fsync_mark(page, do_fsync); + if (IS_INODE(page) && (atomic || is_fsync_dnode(page))) + set_dentry_mark(page, + f2fs_need_dentry_mark(sbi, ino_of_node(page))); + /* should add to global list before clearing PAGECACHE status */ if (f2fs_in_warm_node_list(sbi, page)) { seq = f2fs_add_fsync_node_entry(sbi, page); @@ -1767,7 +1773,7 @@ int f2fs_move_node_page(struct page *node_page, int gc_type) goto out_page; } - if (__write_node_page(node_page, false, NULL, + if (__write_node_page(node_page, false, false, NULL, &wbc, false, FS_GC_NODE_IO, NULL)) { err = -EAGAIN; unlock_page(node_page); @@ -1788,7 +1794,7 @@ int f2fs_move_node_page(struct page *node_page, int gc_type) static int f2fs_write_node_page(struct page *page, struct writeback_control *wbc) { - return __write_node_page(page, false, NULL, wbc, false, + return __write_node_page(page, false, false, NULL, wbc, false, FS_NODE_IO, NULL); } @@ -1822,6 +1828,7 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode, for (i = 0; i < nr_folios; i++) { struct page *page = &fbatch.folios[i]->page; bool submitted = false; + bool do_fsync = false; if (unlikely(f2fs_cp_error(sbi))) { f2fs_put_page(last_page, 0); @@ -1852,19 +1859,13 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode, f2fs_wait_on_page_writeback(page, NODE, true, true); - set_fsync_mark(page, 0); - set_dentry_mark(page, 0); - if (!atomic || page == last_page) { - set_fsync_mark(page, 1); + do_fsync = true; percpu_counter_inc(&sbi->rf_node_block_count); if (IS_INODE(page)) { if (is_inode_flag_set(inode, FI_DIRTY_INODE)) f2fs_update_inode(inode, page); - if (!atomic) - set_dentry_mark(page, - f2fs_need_dentry_mark(sbi, ino)); } /* may be written by other thread */ if (!PageDirty(page)) @@ -1876,8 +1877,9 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode, ret = __write_node_page(page, atomic && page == last_page, - &submitted, wbc, true, - FS_NODE_IO, seq_id); + do_fsync, &submitted, + wbc, true, FS_NODE_IO, + seq_id); if (ret) { unlock_page(page); f2fs_put_page(last_page, 0); @@ -1902,6 +1904,11 @@ int f2fs_fsync_node_pages(struct f2fs_sb_info *sbi, struct inode *inode, f2fs_debug(sbi, "Retry to write fsync mark: ino=%u, idx=%lx", ino, last_page->index); lock_page(last_page); + if (unlikely(last_page->mapping != NODE_MAPPING(sbi))) { + f2fs_put_page(last_page, 1); + ret = -EAGAIN; + goto out; + } f2fs_wait_on_page_writeback(last_page, NODE, true, true); set_page_dirty(last_page); unlock_page(last_page); @@ -2084,10 +2091,7 @@ int f2fs_sync_node_pages(struct f2fs_sb_info *sbi, if (!clear_page_dirty_for_io(page)) goto continue_unlock; - set_fsync_mark(page, 0); - set_dentry_mark(page, 0); - - ret = __write_node_page(page, false, &submitted, + ret = __write_node_page(page, false, false, &submitted, wbc, do_balance, io_type, NULL); if (ret) unlock_page(page); diff --git a/fs/f2fs/recovery.c b/fs/f2fs/recovery.c index 332996ff5cce..5748e3d98ff5 100644 --- a/fs/f2fs/recovery.c +++ b/fs/f2fs/recovery.c @@ -176,7 +176,8 @@ static int recover_dentry(struct inode *inode, struct page *ipage, struct inode *dir, *einode; struct fsync_inode_entry *entry; int err = 0; - char *name; + const char *name; + int name_len; entry = get_fsync_inode(dir_list, pino); if (!entry) { @@ -235,12 +236,16 @@ static int recover_dentry(struct inode *inode, struct page *ipage, out_put: f2fs_put_page(page, 0); out: - if (file_enc_name(inode)) + if (file_enc_name(inode)) { name = ""; - else + name_len = sizeof("") - 1; + } else { name = raw_inode->i_name; - f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %s, dir = %lx, err = %d", - __func__, ino_of_node(ipage), name, + name_len = min_t(unsigned int, le32_to_cpu(raw_inode->i_namelen), + F2FS_NAME_LEN); + } + f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, dir = %lx, err = %d", + __func__, ino_of_node(ipage), name_len, name, IS_ERR(dir) ? 0 : dir->i_ino, err); return err; } @@ -287,7 +292,8 @@ static void recover_inline_flags(struct inode *inode, struct f2fs_inode *ri) static int recover_inode(struct inode *inode, struct page *page) { struct f2fs_inode *raw = F2FS_INODE(page); - char *name; + const char *name; + int name_len; int err; inode->i_mode = le16_to_cpu(raw->i_mode); @@ -336,13 +342,18 @@ static int recover_inode(struct inode *inode, struct page *page) f2fs_mark_inode_dirty_sync(inode, true); - if (file_enc_name(inode)) + if (file_enc_name(inode)) { name = ""; - else - name = F2FS_INODE(page)->i_name; + name_len = sizeof("") - 1; + } else { + name = raw->i_name; + name_len = min_t(unsigned int, le32_to_cpu(raw->i_namelen), + F2FS_NAME_LEN); + } - f2fs_notice(F2FS_I_SB(inode), "recover_inode: ino = %x, name = %s, inline = %x", - ino_of_node(page), name, raw->i_inline); + f2fs_notice(F2FS_I_SB(inode), "%s: ino = %x, name = %.*s, inline = %x", + __func__, ino_of_node(page), name_len, name, + raw->i_inline); return 0; } diff --git a/fs/f2fs/segment.c b/fs/f2fs/segment.c index 9c27353f384b..3ee092684355 100644 --- a/fs/f2fs/segment.c +++ b/fs/f2fs/segment.c @@ -440,15 +440,14 @@ void f2fs_balance_fs(struct f2fs_sb_info *sbi, bool need) f2fs_submit_merged_write(sbi, DATA); f2fs_submit_all_merged_ipu_writes(sbi); - if (test_opt(sbi, GC_MERGE) && sbi->gc_thread && - sbi->gc_thread->f2fs_gc_task) { + if (test_opt(sbi, GC_MERGE) && sbi->gc_thread.f2fs_gc_task) { DEFINE_WAIT(wait); - prepare_to_wait(&sbi->gc_thread->fggc_wq, &wait, + prepare_to_wait(&sbi->gc_thread.fggc_wq, &wait, TASK_UNINTERRUPTIBLE); - wake_up(&sbi->gc_thread->gc_wait_queue_head); + wake_up(&sbi->gc_thread.gc_wait_queue_head); io_schedule(); - finish_wait(&sbi->gc_thread->fggc_wq, &wait); + finish_wait(&sbi->gc_thread.fggc_wq, &wait); } else { struct f2fs_gc_control gc_control = { .victim_segno = NULL_SEGNO, @@ -3147,12 +3146,14 @@ int f2fs_allocate_pinning_section(struct f2fs_sb_info *sbi) f2fs_unlock_op(sbi); if (f2fs_sb_has_blkzoned(sbi) && err && gc_required) { - f2fs_down_write(&sbi->gc_lock); - f2fs_gc_range(sbi, 0, GET_SEGNO(sbi, FDEV(0).end_blk), true, 1); - f2fs_up_write(&sbi->gc_lock); - - gc_required = false; - goto retry; + err = f2fs_gc_range(sbi, 0, GET_SEGNO(sbi, FDEV(0).end_blk), true, 1, true); + if (err) + return err; + err = f2fs_sync_fs(sbi->sb, 1); + if (!err) { + gc_required = false; + goto retry; + } } return err; diff --git a/fs/f2fs/super.c b/fs/f2fs/super.c index 95adc2c4c3c0..b94dab34e591 100644 --- a/fs/f2fs/super.c +++ b/fs/f2fs/super.c @@ -2457,11 +2457,11 @@ static int f2fs_remount(struct super_block *sb, int *flags, char *data) if ((*flags & SB_RDONLY) || (F2FS_OPTION(sbi).bggc_mode == BGGC_MODE_OFF && !test_opt(sbi, GC_MERGE))) { - if (sbi->gc_thread) { + if (sbi->gc_thread.f2fs_gc_task) { f2fs_stop_gc_thread(sbi); need_restart_gc = true; } - } else if (!sbi->gc_thread) { + } else if (!sbi->gc_thread.f2fs_gc_task) { err = f2fs_start_gc_thread(sbi); if (err) goto restore_opts; @@ -4483,7 +4483,7 @@ static int f2fs_fill_super(struct super_block *sb, void *data, int silent) sb->s_time_gran = 1; sb->s_flags = (sb->s_flags & ~SB_POSIXACL) | (test_opt(sbi, POSIX_ACL) ? SB_POSIXACL : 0); - memcpy(&sb->s_uuid, raw_super->uuid, sizeof(raw_super->uuid)); + super_set_uuid(sb, (void *) raw_super->uuid, sizeof(raw_super->uuid)); sb->s_iflags |= SB_I_CGROUPWB; /* init f2fs-specific super block info */ diff --git a/fs/f2fs/sysfs.c b/fs/f2fs/sysfs.c index 4cd2d4bde217..694bca245a52 100644 --- a/fs/f2fs/sysfs.c +++ b/fs/f2fs/sysfs.c @@ -74,7 +74,7 @@ static ssize_t f2fs_sbi_show(struct f2fs_attr *a, static unsigned char *__struct_ptr(struct f2fs_sb_info *sbi, int struct_type) { if (struct_type == GC_THREAD) - return (unsigned char *)sbi->gc_thread; + return (unsigned char *)&sbi->gc_thread; else if (struct_type == SM_INFO) return (unsigned char *)SM_I(sbi); else if (struct_type == DCC_INFO) @@ -339,13 +339,13 @@ static ssize_t f2fs_sbi_show(struct f2fs_attr *a, if (!strcmp(a->attr.name, "ckpt_thread_ioprio")) { struct ckpt_req_control *cprc = &sbi->cprc_info; int class = IOPRIO_PRIO_CLASS(cprc->ckpt_thread_ioprio); - int data = IOPRIO_PRIO_DATA(cprc->ckpt_thread_ioprio); + int level = IOPRIO_PRIO_LEVEL(cprc->ckpt_thread_ioprio); if (class != IOPRIO_CLASS_RT && class != IOPRIO_CLASS_BE) return -EINVAL; return sysfs_emit(buf, "%s,%d\n", - class == IOPRIO_CLASS_RT ? "rt" : "be", data); + class == IOPRIO_CLASS_RT ? "rt" : "be", level); } #ifdef CONFIG_F2FS_FS_COMPRESSION @@ -470,7 +470,7 @@ static ssize_t __sbi_store(struct f2fs_attr *a, const char *name = strim((char *)buf); struct ckpt_req_control *cprc = &sbi->cprc_info; int class; - long data; + long level; int ret; if (!strncmp(name, "rt,", 3)) @@ -481,14 +481,14 @@ static ssize_t __sbi_store(struct f2fs_attr *a, return -EINVAL; name += 3; - ret = kstrtol(name, 10, &data); + ret = kstrtol(name, 10, &level); if (ret) return ret; - if (data >= IOPRIO_NR_LEVELS || data < 0) + if (level >= IOPRIO_NR_LEVELS || level < 0) return -EINVAL; - cprc->ckpt_thread_ioprio = IOPRIO_PRIO_VALUE(class, data); - if (test_opt(sbi, MERGE_CHECKPOINT)) { + cprc->ckpt_thread_ioprio = IOPRIO_PRIO_VALUE(class, level); + if (cprc->f2fs_issue_ckpt) { ret = set_task_ioprio(cprc->f2fs_issue_ckpt, cprc->ckpt_thread_ioprio); if (ret) @@ -579,20 +579,20 @@ static ssize_t __sbi_store(struct f2fs_attr *a, sbi->gc_mode = GC_NORMAL; } else if (t == 1) { sbi->gc_mode = GC_URGENT_HIGH; - if (sbi->gc_thread) { - sbi->gc_thread->gc_wake = true; + if (sbi->gc_thread.f2fs_gc_task) { + sbi->gc_thread.gc_wake = true; wake_up_interruptible_all( - &sbi->gc_thread->gc_wait_queue_head); + &sbi->gc_thread.gc_wait_queue_head); wake_up_discard_thread(sbi, true); } } else if (t == 2) { sbi->gc_mode = GC_URGENT_LOW; } else if (t == 3) { sbi->gc_mode = GC_URGENT_MID; - if (sbi->gc_thread) { - sbi->gc_thread->gc_wake = true; + if (sbi->gc_thread.f2fs_gc_task) { + sbi->gc_thread.gc_wake = true; wake_up_interruptible_all( - &sbi->gc_thread->gc_wait_queue_head); + &sbi->gc_thread.gc_wait_queue_head); } } else { return -EINVAL; @@ -799,13 +799,14 @@ static ssize_t f2fs_sbi_store(struct f2fs_attr *a, ssize_t ret; bool gc_entry = (!strcmp(a->attr.name, "gc_urgent") || a->struct_type == GC_THREAD); + bool thread_entry = !strcmp(a->attr.name, "ckpt_thread_ioprio"); - if (gc_entry) { + if (gc_entry || thread_entry) { if (!down_read_trylock(&sbi->sb->s_umount)) return -EAGAIN; } ret = __sbi_store(a, sbi, buf, count); - if (gc_entry) + if (gc_entry || thread_entry) up_read(&sbi->sb->s_umount); return ret; diff --git a/fs/fcntl.c b/fs/fcntl.c index ee4699bcfc4d..89f327e51f14 100644 --- a/fs/fcntl.c +++ b/fs/fcntl.c @@ -291,6 +291,9 @@ static long fcntl_rw_hint(struct file *file, unsigned int cmd, return -EFAULT; return 0; case F_SET_RW_HINT: + /* nothing is ever written to it */ + if (IS_IMMUTABLE(inode)) + return -EPERM; if (copy_from_user(&hint, argp, sizeof(hint))) return -EFAULT; if (!rw_hint_valid(hint)) diff --git a/fs/fsopen.c b/fs/fsopen.c index 3576b1084ec2..664137c4478e 100644 --- a/fs/fsopen.c +++ b/fs/fsopen.c @@ -191,6 +191,9 @@ SYSCALL_DEFINE3(fspick, int, dfd, const char __user *, path, unsigned int, flags ret = -EINVAL; if (target.mnt->mnt_root != target.dentry) goto err_path; + /* kernel-internal superblocks are nobody's to reconfigure */ + if (target.dentry->d_sb->s_flags & SB_NOUSER) + goto err_path; fc = fs_context_for_reconfigure(target.dentry, 0, 0); if (IS_ERR(fc)) { diff --git a/fs/fuse/file.c b/fs/fuse/file.c index 553c163afbc2..13257bfd6ca4 100644 --- a/fs/fuse/file.c +++ b/fs/fuse/file.c @@ -658,7 +658,7 @@ void fuse_read_args_fill(struct fuse_io_args *ia, struct file *file, loff_t pos, args->out_args[0].size = count; } -static void fuse_release_user_pages(struct fuse_args_pages *ap, +static void fuse_release_user_pages(struct fuse_args_pages *ap, ssize_t nres, bool should_dirty) { unsigned int i; @@ -666,8 +666,12 @@ static void fuse_release_user_pages(struct fuse_args_pages *ap, for (i = 0; i < ap->num_pages; i++) { if (should_dirty) set_page_dirty_lock(ap->pages[i]); - put_page(ap->pages[i]); + if (ap->args.is_pinned) + unpin_user_page(ap->pages[i]); } + + if (nres > 0 && ap->args.invalidate_vmap) + invalidate_kernel_vmap_range(ap->args.vmap_base, nres); } static void fuse_io_release(struct kref *kref) @@ -766,25 +770,29 @@ static void fuse_aio_complete_req(struct fuse_mount *fm, struct fuse_args *args, struct fuse_io_args *ia = container_of(args, typeof(*ia), ap.args); struct fuse_io_priv *io = ia->io; ssize_t pos = -1; - - fuse_release_user_pages(&ia->ap, io->should_dirty); + size_t nres; if (err) { /* Nothing */ } else if (io->write) { if (ia->write.out.size > ia->write.in.size) { err = -EIO; - } else if (ia->write.in.size != ia->write.out.size) { - pos = ia->write.in.offset - io->offset + - ia->write.out.size; + } else { + nres = ia->write.out.size; + if (ia->write.in.size != ia->write.out.size) + pos = ia->write.in.offset - io->offset + + ia->write.out.size; } } else { u32 outsize = args->out_args[0].size; + nres = outsize; if (ia->read.in.size != outsize) pos = ia->read.in.offset - io->offset + outsize; } + fuse_release_user_pages(&ia->ap, err ?: nres, io->should_dirty); + fuse_aio_complete(io, err, pos); fuse_io_free(ia); } @@ -1401,33 +1409,49 @@ static inline size_t fuse_get_frag_size(const struct iov_iter *ii, static int fuse_get_user_pages(struct fuse_args_pages *ap, struct iov_iter *ii, size_t *nbytesp, int write, - unsigned int max_pages) + unsigned int max_pages, + bool use_pages_for_kvec_io) { + bool flush_or_invalidate = false; size_t nbytes = 0; /* # bytes already packed in req */ ssize_t ret = 0; - /* Special case for kernel I/O: can copy directly into the buffer */ + /* Special case for kernel I/O: can copy directly into the buffer. + * However if the implementation of fuse_conn requires pages instead of + * pointer (e.g., virtio-fs), use iov_iter_extract_pages() instead. + */ if (iov_iter_is_kvec(ii)) { - unsigned long user_addr = fuse_get_user_addr(ii); - size_t frag_size = fuse_get_frag_size(ii, *nbytesp); + void *user_addr = (void *)fuse_get_user_addr(ii); - if (write) - ap->args.in_args[1].value = (void *) user_addr; - else - ap->args.out_args[0].value = (void *) user_addr; + if (!use_pages_for_kvec_io) { + size_t frag_size = fuse_get_frag_size(ii, *nbytesp); - iov_iter_advance(ii, frag_size); - *nbytesp = frag_size; - return 0; + if (write) + ap->args.in_args[1].value = user_addr; + else + ap->args.out_args[0].value = user_addr; + + iov_iter_advance(ii, frag_size); + *nbytesp = frag_size; + return 0; + } + + if (is_vmalloc_addr(user_addr)) { + ap->args.vmap_base = user_addr; + flush_or_invalidate = true; + } } while (nbytes < *nbytesp && ap->num_pages < max_pages) { unsigned npages; size_t start; - ret = iov_iter_get_pages2(ii, &ap->pages[ap->num_pages], - *nbytesp - nbytes, - max_pages - ap->num_pages, - &start); + struct page **pt_pages; + + pt_pages = &ap->pages[ap->num_pages]; + ret = iov_iter_extract_pages(ii, &pt_pages, + *nbytesp - nbytes, + max_pages - ap->num_pages, + 0, &start); if (ret < 0) break; @@ -1444,6 +1468,11 @@ static int fuse_get_user_pages(struct fuse_args_pages *ap, struct iov_iter *ii, (PAGE_SIZE - ret) & (PAGE_SIZE - 1); } + if (write && flush_or_invalidate) + flush_kernel_vmap_range(ap->args.vmap_base, nbytes); + + ap->args.invalidate_vmap = !write && flush_or_invalidate; + ap->args.is_pinned = iov_iter_extract_will_pin(ii); ap->args.user_pages = true; if (write) ap->args.in_pages = true; @@ -1511,7 +1540,7 @@ ssize_t fuse_direct_io(struct fuse_io_priv *io, struct iov_iter *iter, size_t nbytes = min(count, nmax); err = fuse_get_user_pages(&ia->ap, iter, &nbytes, write, - max_pages); + max_pages, fc->use_pages_for_kvec_io); if (err && !nbytes) break; @@ -1525,7 +1554,7 @@ ssize_t fuse_direct_io(struct fuse_io_priv *io, struct iov_iter *iter, } if (!io->async || nres < 0) { - fuse_release_user_pages(&ia->ap, io->should_dirty); + fuse_release_user_pages(&ia->ap, nres, io->should_dirty); fuse_io_free(ia); } ia = NULL; diff --git a/fs/fuse/fuse_i.h b/fs/fuse/fuse_i.h index 8751217aaafa..c4cbc0befc00 100644 --- a/fs/fuse/fuse_i.h +++ b/fs/fuse/fuse_i.h @@ -289,9 +289,13 @@ struct fuse_args { bool page_replace:1; bool may_block:1; bool is_ext:1; + bool is_pinned:1; + bool invalidate_vmap:1; struct fuse_in_arg in_args[3]; struct fuse_arg out_args[2]; void (*end)(struct fuse_mount *fm, struct fuse_args *args, int error); + /* Used for kvec iter backed by vmalloc address */ + void *vmap_base; }; struct fuse_args_pages { @@ -834,6 +838,9 @@ struct fuse_conn { /* Is statx not implemented by fs? */ unsigned int no_statx:1; + /* Use pages instead of pointer for kernel I/O */ + unsigned int use_pages_for_kvec_io:1; + /** The number of requests waiting for completion */ atomic_t num_waiting; diff --git a/fs/fuse/virtio_fs.c b/fs/fuse/virtio_fs.c index a6814bea0e0a..b1028a4dbb99 100644 --- a/fs/fuse/virtio_fs.c +++ b/fs/fuse/virtio_fs.c @@ -1461,6 +1461,7 @@ static int virtio_fs_get_tree(struct fs_context *fsc) fc->delete_stale = true; fc->auto_submounts = true; fc->sync_fs = true; + fc->use_pages_for_kvec_io = true; /* Tell FUSE to split requests that exceed the virtqueue's size */ fc->max_pages_limit = min_t(unsigned int, fc->max_pages_limit, diff --git a/fs/gfs2/inode.c b/fs/gfs2/inode.c index 18f75beea089..4eb5716ad028 100644 --- a/fs/gfs2/inode.c +++ b/fs/gfs2/inode.c @@ -89,6 +89,19 @@ static int iget_set(struct inode *inode, void *opaque) return 0; } +void gfs2_setup_inode(struct inode *inode) +{ + gfp_t gfp_mask; + + /* + * Ensure all page cache allocations are done from GFP_NOFS context to + * prevent direct reclaim recursion back into the filesystem and blowing + * stacks or deadlocking. + */ + gfp_mask = mapping_gfp_mask(inode->i_mapping); + mapping_set_gfp_mask(inode->i_mapping, gfp_mask & ~__GFP_FS); +} + /** * gfs2_inode_lookup - Lookup an inode * @sb: The super block @@ -132,6 +145,7 @@ struct inode *gfs2_inode_lookup(struct super_block *sb, unsigned int type, struct gfs2_glock *io_gl; int extra_flags = 0; + gfs2_setup_inode(inode); error = gfs2_glock_get(sdp, no_addr, &gfs2_inode_glops, CREATE, &ip->i_gl); if (unlikely(error)) @@ -684,6 +698,7 @@ static int gfs2_create_inode(struct inode *dir, struct dentry *dentry, error = -ENOMEM; if (!inode) goto fail_gunlock; + gfs2_setup_inode(inode); ip = GFS2_I(inode); error = posix_acl_create(dir, &mode, &default_acl, &acl); diff --git a/fs/gfs2/inode.h b/fs/gfs2/inode.h index ce70cf26b497..6f7c9ec21bbf 100644 --- a/fs/gfs2/inode.h +++ b/fs/gfs2/inode.h @@ -88,6 +88,7 @@ static inline int gfs2_check_internal_file_size(struct inode *inode, return -EIO; } +void gfs2_setup_inode(struct inode *inode); struct inode *gfs2_inode_lookup(struct super_block *sb, unsigned type, u64 no_addr, u64 no_formal_ino, unsigned int blktype); diff --git a/fs/gfs2/ops_fstype.c b/fs/gfs2/ops_fstype.c index f4c066aa24b9..3a13cd3781cc 100644 --- a/fs/gfs2/ops_fstype.c +++ b/fs/gfs2/ops_fstype.c @@ -215,7 +215,7 @@ static void gfs2_sb_in(struct gfs2_sbd *sdp, const void *buf) memcpy(sb->sb_lockproto, str->sb_lockproto, GFS2_LOCKNAME_LEN); memcpy(sb->sb_locktable, str->sb_locktable, GFS2_LOCKNAME_LEN); - memcpy(&s->s_uuid, str->sb_uuid, 16); + super_set_uuid(s, str->sb_uuid, 16); } /** @@ -1292,7 +1292,6 @@ static int gfs2_fill_super(struct super_block *sb, struct fs_context *fc) if (error) { gfs2_freeze_unlock(&sdp->sd_freeze_gh); - gfs2_destroy_threads(sdp); fs_err(sdp, "can't make FS RW: %d\n", error); goto fail_per_node; } @@ -1302,6 +1301,7 @@ static int gfs2_fill_super(struct super_block *sb, struct fs_context *fc) fail_per_node: init_per_node(sdp, UNDO); + gfs2_destroy_threads(sdp); fail_inodes: init_inodes(sdp, UNDO); fail_sb: diff --git a/fs/gfs2/quota.c b/fs/gfs2/quota.c index c537e1d02cf3..6794223b01fc 100644 --- a/fs/gfs2/quota.c +++ b/fs/gfs2/quota.c @@ -255,7 +255,7 @@ static struct gfs2_quota_data *gfs2_qd_search_bucket(unsigned int hash, if (qd->qd_sbd != sdp) continue; if (lockref_get_not_dead(&qd->qd_lockref)) { - list_lru_del(&gfs2_qd_lru, &qd->qd_lru); + list_lru_del_obj(&gfs2_qd_lru, &qd->qd_lru); return qd; } } @@ -323,12 +323,13 @@ static void qd_put(struct gfs2_quota_data *qd) lockref_mark_dead(&qd->qd_lockref); spin_unlock(&qd->qd_lockref.lock); + list_lru_del_obj(&gfs2_qd_lru, &qd->qd_lru); gfs2_qd_dispose(qd); return; } qd->qd_lockref.count = 0; - list_lru_add(&gfs2_qd_lru, &qd->qd_lru); + list_lru_add_obj(&gfs2_qd_lru, &qd->qd_lru); spin_unlock(&qd->qd_lockref.lock); } @@ -1492,7 +1493,7 @@ void gfs2_quota_cleanup(struct gfs2_sbd *sdp) lockref_mark_dead(&qd->qd_lockref); spin_unlock(&qd->qd_lockref.lock); - list_lru_del(&gfs2_qd_lru, &qd->qd_lru); + list_lru_del_obj(&gfs2_qd_lru, &qd->qd_lru); list_add(&qd->qd_lru, &dispose); } spin_unlock(&qd_lock); diff --git a/fs/gfs2/super.c b/fs/gfs2/super.c index ee370c586014..a5024cfd618d 100644 --- a/fs/gfs2/super.c +++ b/fs/gfs2/super.c @@ -157,8 +157,10 @@ int gfs2_make_fs_rw(struct gfs2_sbd *sdp) gfs2_log_pointers_init(sdp, head.lh_blkno); error = gfs2_quota_init(sdp); - if (!error && gfs2_withdrawing_or_withdrawn(sdp)) + if (!error && gfs2_withdrawing_or_withdrawn(sdp)) { + gfs2_quota_cleanup(sdp); error = -EIO; + } if (!error) set_bit(SDF_JOURNAL_LIVE, &sdp->sd_flags); return error; diff --git a/fs/inode.c b/fs/inode.c index 030e07b169c2..5ebe79138109 100644 --- a/fs/inode.c +++ b/fs/inode.c @@ -464,7 +464,7 @@ static void __inode_add_lru(struct inode *inode, bool rotate) if (!mapping_shrinkable(&inode->i_data)) return; - if (list_lru_add(&inode->i_sb->s_inode_lru, &inode->i_lru)) + if (list_lru_add_obj(&inode->i_sb->s_inode_lru, &inode->i_lru)) this_cpu_inc(nr_unused); else if (rotate) inode->i_state |= I_REFERENCED; @@ -482,7 +482,7 @@ void inode_add_lru(struct inode *inode) static void inode_lru_list_del(struct inode *inode) { - if (list_lru_del(&inode->i_sb->s_inode_lru, &inode->i_lru)) + if (list_lru_del_obj(&inode->i_sb->s_inode_lru, &inode->i_lru)) this_cpu_dec(nr_unused); } @@ -642,21 +642,18 @@ void dump_mapping(const struct address_space *mapping) void clear_inode(struct inode *inode) { /* - * We have to cycle the i_pages lock here because reclaim can be in the - * process of removing the last page (in __filemap_remove_folio()) - * and we must not free the mapping under it. + * We have to cycle the i_pages lock here because reclaim + * can be in the process of removing the last page (in + * __filemap_remove_folio()) and we must not free the mapping + * under it. We also remove nodes which are empty; these + * can occur in two different ways. The first is that radix + * tree expansion can fail partway and the second is that THP + * collapse_file() can allocate some temporary nodes and not + * clean them up. */ - xa_lock_irq(&inode->i_data.i_pages); + xa_destroy(&inode->i_data.i_pages); + BUG_ON(inode->i_data.nrpages); - /* - * Almost always, mapping_empty(&inode->i_data) here; but there are - * two known and long-standing ways in which nodes may get left behind - * (when deep radix-tree node allocation failed partway; or when THP - * collapse_file() failed). Until those two known cases are cleaned up, - * or a cleanup function is called here, do not BUG_ON(!mapping_empty), - * nor even WARN_ON(!mapping_empty). - */ - xa_unlock_irq(&inode->i_data.i_pages); BUG_ON(!list_empty(&inode->i_data.private_list)); BUG_ON(!(inode->i_state & I_FREEING)); BUG_ON(inode->i_state & I_CLEAR); diff --git a/fs/jbd2/checkpoint.c b/fs/jbd2/checkpoint.c index 32ed28cfb3a0..95177cddf73d 100644 --- a/fs/jbd2/checkpoint.c +++ b/fs/jbd2/checkpoint.c @@ -355,20 +355,19 @@ int jbd2_cleanup_journal_tail(journal_t *journal) /* Checkpoint list management */ -enum shrink_type {SHRINK_DESTROY, SHRINK_BUSY_STOP, SHRINK_BUSY_SKIP}; - /* * journal_shrink_one_cp_list * - * Find all the written-back checkpoint buffers in the given list - * and try to release them. If the whole transaction is released, set - * the 'released' parameter. Return the number of released checkpointed - * buffers. + * Find written-back checkpoint buffers in the given list and try to release + * them. If 'nr_to_scan' is set, scan at most that many buffers. If the whole + * transaction is released, set the 'released' parameter. Return the number of + * released checkpointed buffers. * * Called with j_list_lock held. */ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, enum shrink_type type, + unsigned long *nr_to_scan, bool *released) { struct journal_head *last_jh; @@ -377,13 +376,15 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, int ret; *released = false; - if (!jh) + if (!jh || (nr_to_scan && !*nr_to_scan)) return 0; last_jh = jh->b_cpprev; do { jh = next_jh; next_jh = jh->b_cpnext; + if (nr_to_scan) + (*nr_to_scan)--; if (type == SHRINK_DESTROY) { ret = __jbd2_journal_remove_checkpoint(jh); @@ -391,7 +392,7 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, ret = jbd2_journal_try_remove_checkpoint(jh); if (ret < 0) { if (type == SHRINK_BUSY_SKIP) - continue; + goto next; break; } } @@ -402,9 +403,10 @@ static unsigned long journal_shrink_one_cp_list(struct journal_head *jh, break; } +next: if (need_resched()) break; - } while (jh != last_jh); + } while (jh != last_jh && (!nr_to_scan || *nr_to_scan)); return nr_freed; } @@ -426,7 +428,6 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal, tid_t first_tid = 0, last_tid = 0, next_tid = 0; tid_t tid = 0; unsigned long nr_freed = 0; - unsigned long freed; bool first_set = false; again: @@ -459,10 +460,9 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal, next_transaction = transaction->t_cpnext; tid = transaction->t_tid; - freed = journal_shrink_one_cp_list(transaction->t_checkpoint_list, - SHRINK_BUSY_SKIP, &released); - nr_freed += freed; - (*nr_to_scan) -= min(*nr_to_scan, freed); + nr_freed += journal_shrink_one_cp_list(transaction->t_checkpoint_list, + SHRINK_BUSY_SKIP, + nr_to_scan, &released); if (*nr_to_scan == 0) break; if (need_resched() || spin_needbreak(&journal->j_list_lock)) @@ -493,28 +493,32 @@ unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal, * journal_clean_checkpoint_list * * Find all the written-back checkpoint buffers in the journal and release them. - * If 'destroy' is set, release all buffers unconditionally. + * If 'type' is SHRINK_DESTROY, release all buffers unconditionally. If 'type' + * is SHRINK_BUSY_STOP, will stop release buffers if encounters a busy buffer. + * To avoid wasting CPU cycles scanning the buffer list in some cases, don't + * pass SHRINK_BUSY_SKIP 'type' for this function. * * Called with j_list_lock held. */ -void __jbd2_journal_clean_checkpoint_list(journal_t *journal, bool destroy) +void __jbd2_journal_clean_checkpoint_list(journal_t *journal, + enum shrink_type type) { transaction_t *transaction, *last_transaction, *next_transaction; - enum shrink_type type; bool released; + WARN_ON_ONCE(type == SHRINK_BUSY_SKIP); + transaction = journal->j_checkpoint_transactions; if (!transaction) return; - type = destroy ? SHRINK_DESTROY : SHRINK_BUSY_STOP; last_transaction = transaction->t_cpprev; next_transaction = transaction; do { transaction = next_transaction; next_transaction = transaction->t_cpnext; journal_shrink_one_cp_list(transaction->t_checkpoint_list, - type, &released); + type, NULL, &released); /* * This function only frees up some memory if possible so we * dont have an obligation to finish processing. Bail out if @@ -548,7 +552,7 @@ void jbd2_journal_destroy_checkpoint(journal_t *journal) spin_unlock(&journal->j_list_lock); break; } - __jbd2_journal_clean_checkpoint_list(journal, true); + __jbd2_journal_clean_checkpoint_list(journal, SHRINK_DESTROY); spin_unlock(&journal->j_list_lock); cond_resched(); } diff --git a/fs/jbd2/commit.c b/fs/jbd2/commit.c index 84663ff7dc50..4de8b49afd22 100644 --- a/fs/jbd2/commit.c +++ b/fs/jbd2/commit.c @@ -501,7 +501,7 @@ void jbd2_journal_commit_transaction(journal_t *journal) * frees some memory */ spin_lock(&journal->j_list_lock); - __jbd2_journal_clean_checkpoint_list(journal, false); + __jbd2_journal_clean_checkpoint_list(journal, SHRINK_BUSY_STOP); spin_unlock(&journal->j_list_lock); jbd2_debug(3, "JBD2: commit phase 1\n"); diff --git a/fs/jfs/jfs_dtree.c b/fs/jfs/jfs_dtree.c index 93db6eec4465..ab11849cf9cc 100644 --- a/fs/jfs/jfs_dtree.c +++ b/fs/jfs/jfs_dtree.c @@ -2613,7 +2613,7 @@ void dtInitRoot(tid_t tid, struct inode *ip, u32 idotdot) * fsck.jfs should really fix this, but it currently does not. * Called from jfs_readdir when bad index is detected. */ -static void add_missing_indices(struct inode *inode, s64 bn) +static int add_missing_indices(struct inode *inode, s64 bn) { struct ldtentry *d; struct dt_lock *dtlck; @@ -2622,7 +2622,7 @@ static void add_missing_indices(struct inode *inode, s64 bn) struct lv *lv; struct metapage *mp; dtpage_t *p; - int rc; + int rc = 0; s8 *stbl; tid_t tid; struct tlock *tlck; @@ -2647,6 +2647,16 @@ static void add_missing_indices(struct inode *inode, s64 bn) stbl = DT_GETSTBL(p); for (i = 0; i < p->header.nextindex; i++) { + if (stbl[i] < 0) { + jfs_err("jfs: add_missing_indices: Invalid stbl[%d] = %d for inode %ld, block = %lld", + i, stbl[i], (long)inode->i_ino, (long long)bn); + rc = -EIO; + + DT_PUTPAGE(mp); + txAbort(tid, 0); + goto end; + } + d = (struct ldtentry *) &p->slot[stbl[i]]; index = le32_to_cpu(d->index); if ((index < 2) || (index >= JFS_IP(inode)->next_index)) { @@ -2664,6 +2674,7 @@ static void add_missing_indices(struct inode *inode, s64 bn) (void) txCommit(tid, 1, &inode, 0); end: txEnd(tid); + return rc; } /* @@ -3017,7 +3028,8 @@ int jfs_readdir(struct file *file, struct dir_context *ctx) } if (fix_page) { - add_missing_indices(ip, bn); + if ((rc = add_missing_indices(ip, bn))) + goto out; page_fixed = 1; } diff --git a/fs/kernfs/mount.c b/fs/kernfs/mount.c index c4bf26142eec..e3f28ee49f91 100644 --- a/fs/kernfs/mount.c +++ b/fs/kernfs/mount.c @@ -360,7 +360,9 @@ int kernfs_get_tree(struct fs_context *fc) } sb->s_flags |= SB_ACTIVE; - uuid_gen(&sb->s_uuid); + uuid_t uuid; + uuid_gen(&uuid); + super_set_uuid(sb, uuid.b, sizeof(uuid)); down_write(&root->kernfs_supers_rwsem); list_add(&info->node, &info->root->supers); @@ -396,8 +398,8 @@ void kernfs_kill_sb(struct super_block *sb) up_write(&root->kernfs_supers_rwsem); /* - * Remove the superblock from fs_supers/s_instances - * so we can't find it, before freeing kernfs_super_info. + * Mark the superblock dead so sget_fc() can't find it, + * before freeing kernfs_super_info. */ kill_anon_super(sb); kfree(info); diff --git a/fs/libfs.c b/fs/libfs.c index b913ab238cc1..f839c70616b0 100644 --- a/fs/libfs.c +++ b/fs/libfs.c @@ -344,22 +344,22 @@ void simple_offset_remove(struct offset_ctx *octx, struct dentry *dentry) * User space expects the directory offset value of the replaced * (new) directory entry to be unchanged after a rename. * - * Returns zero on success, a negative errno value on failure. + * Caller must have grabbed a slot for new_dentry in the xarray + * associated with new_dir, even if dentry is negative. */ -int simple_offset_rename(struct inode *old_dir, struct dentry *old_dentry, - struct inode *new_dir, struct dentry *new_dentry) +void simple_offset_rename(struct inode *old_dir, struct dentry *old_dentry, + struct inode *new_dir, struct dentry *new_dentry) { struct offset_ctx *old_ctx = old_dir->i_op->get_offset_ctx(old_dir); struct offset_ctx *new_ctx = new_dir->i_op->get_offset_ctx(new_dir); long new_offset = dentry2offset(new_dentry); - simple_offset_remove(old_ctx, old_dentry); + if (WARN_ON(!new_offset)) + return; - if (new_offset) { - offset_set(new_dentry, 0); - return simple_offset_replace(new_ctx, old_dentry, new_offset); - } - return simple_offset_add(new_ctx, old_dentry); + simple_offset_remove(old_ctx, old_dentry); + offset_set(new_dentry, 0); + WARN_ON(simple_offset_replace(new_ctx, old_dentry, new_offset)); } /** @@ -386,31 +386,23 @@ int simple_offset_rename_exchange(struct inode *old_dir, u32 new_index = dentry2offset(new_dentry); int ret; - simple_offset_remove(old_ctx, old_dentry); - simple_offset_remove(new_ctx, new_dentry); + if (WARN_ON(!old_index || !new_index)) + return -EINVAL; - ret = simple_offset_replace(new_ctx, old_dentry, new_index); - if (ret) - goto out_restore; + ret = xa_err(xa_store(&new_ctx->xa, new_index, old_dentry, GFP_KERNEL)); + if (WARN_ON(ret)) + return ret; - ret = simple_offset_replace(old_ctx, new_dentry, old_index); - if (ret) { - simple_offset_remove(new_ctx, old_dentry); - goto out_restore; + ret = xa_err(xa_store(&old_ctx->xa, old_index, new_dentry, GFP_KERNEL)); + if (WARN_ON(ret)) { + xa_store(&new_ctx->xa, new_index, new_dentry, GFP_KERNEL); + return ret; } - ret = simple_rename_exchange(old_dir, old_dentry, new_dir, new_dentry); - if (ret) { - simple_offset_remove(new_ctx, old_dentry); - simple_offset_remove(old_ctx, new_dentry); - goto out_restore; - } + offset_set(old_dentry, new_index); + offset_set(new_dentry, old_index); + simple_rename_exchange(old_dir, old_dentry, new_dir, new_dentry); return 0; - -out_restore: - (void)simple_offset_replace(old_ctx, old_dentry, old_index); - (void)simple_offset_replace(new_ctx, new_dentry, new_index); - return ret; } /** diff --git a/fs/lockd/clnt4xdr.c b/fs/lockd/clnt4xdr.c index 8161667c976f..527458db4525 100644 --- a/fs/lockd/clnt4xdr.c +++ b/fs/lockd/clnt4xdr.c @@ -243,7 +243,7 @@ static void encode_nlm4_holder(struct xdr_stream *xdr, u64 l_offset, l_len; __be32 *p; - encode_bool(xdr, lock->fl.fl_type == F_RDLCK); + encode_bool(xdr, lock->fl.c.flc_type == F_RDLCK); encode_int32(xdr, lock->svid); encode_netobj(xdr, lock->oh.data, lock->oh.len); @@ -270,7 +270,7 @@ static int decode_nlm4_holder(struct xdr_stream *xdr, struct nlm_res *result) goto out_overflow; exclusive = be32_to_cpup(p++); lock->svid = be32_to_cpup(p); - fl->fl_pid = (pid_t)lock->svid; + fl->c.flc_pid = (pid_t)lock->svid; error = decode_netobj(xdr, &lock->oh); if (unlikely(error)) @@ -280,8 +280,8 @@ static int decode_nlm4_holder(struct xdr_stream *xdr, struct nlm_res *result) if (unlikely(p == NULL)) goto out_overflow; - fl->fl_flags = FL_POSIX; - fl->fl_type = exclusive != 0 ? F_WRLCK : F_RDLCK; + fl->c.flc_flags = FL_POSIX; + fl->c.flc_type = exclusive != 0 ? F_WRLCK : F_RDLCK; p = xdr_decode_hyper(p, &l_offset); xdr_decode_hyper(p, &l_len); nlm4svc_set_file_lock_range(fl, l_offset, l_len); @@ -357,7 +357,7 @@ static void nlm4_xdr_enc_testargs(struct rpc_rqst *req, const struct nlm_lock *lock = &args->lock; encode_cookie(xdr, &args->cookie); - encode_bool(xdr, lock->fl.fl_type == F_WRLCK); + encode_bool(xdr, lock->fl.c.flc_type == F_WRLCK); encode_nlm4_lock(xdr, lock); } @@ -380,7 +380,7 @@ static void nlm4_xdr_enc_lockargs(struct rpc_rqst *req, encode_cookie(xdr, &args->cookie); encode_bool(xdr, args->block); - encode_bool(xdr, lock->fl.fl_type == F_WRLCK); + encode_bool(xdr, lock->fl.c.flc_type == F_WRLCK); encode_nlm4_lock(xdr, lock); encode_bool(xdr, args->reclaim); encode_int32(xdr, args->state); @@ -403,7 +403,7 @@ static void nlm4_xdr_enc_cancargs(struct rpc_rqst *req, encode_cookie(xdr, &args->cookie); encode_bool(xdr, args->block); - encode_bool(xdr, lock->fl.fl_type == F_WRLCK); + encode_bool(xdr, lock->fl.c.flc_type == F_WRLCK); encode_nlm4_lock(xdr, lock); } diff --git a/fs/lockd/clntlock.c b/fs/lockd/clntlock.c index 5d85715be763..a7e0519ec024 100644 --- a/fs/lockd/clntlock.c +++ b/fs/lockd/clntlock.c @@ -185,7 +185,7 @@ __be32 nlmclnt_grant(const struct sockaddr *addr, const struct nlm_lock *lock) continue; if (!rpc_cmp_addr(nlm_addr(block->b_host), addr)) continue; - if (nfs_compare_fh(NFS_FH(file_inode(fl_blocked->fl_file)), fh) != 0) + if (nfs_compare_fh(NFS_FH(file_inode(fl_blocked->c.flc_file)), fh) != 0) continue; /* Alright, we found a lock. Set the return status * and wake up the caller diff --git a/fs/lockd/clntproc.c b/fs/lockd/clntproc.c index fba6c7fa7474..2926002e9020 100644 --- a/fs/lockd/clntproc.c +++ b/fs/lockd/clntproc.c @@ -133,7 +133,8 @@ static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) char *nodename = req->a_host->h_rpcclnt->cl_nodename; nlmclnt_next_cookie(&argp->cookie); - memcpy(&lock->fh, NFS_FH(file_inode(fl->fl_file)), sizeof(struct nfs_fh)); + memcpy(&lock->fh, NFS_FH(file_inode(fl->c.flc_file)), + sizeof(struct nfs_fh)); lock->caller = nodename; lock->oh.data = req->a_owner; lock->oh.len = snprintf(req->a_owner, sizeof(req->a_owner), "%u@%s", @@ -142,7 +143,7 @@ static void nlmclnt_setlockargs(struct nlm_rqst *req, struct file_lock *fl) lock->svid = fl->fl_u.nfs_fl.owner->pid; lock->fl.fl_start = fl->fl_start; lock->fl.fl_end = fl->fl_end; - lock->fl.fl_type = fl->fl_type; + lock->fl.c.flc_type = fl->c.flc_type; } static void nlmclnt_release_lockargs(struct nlm_rqst *req) @@ -182,7 +183,7 @@ int nlmclnt_proc(struct nlm_host *host, int cmd, struct file_lock *fl, void *dat call->a_callback_data = data; if (IS_SETLK(cmd) || IS_SETLKW(cmd)) { - if (fl->fl_type != F_UNLCK) { + if (fl->c.flc_type != F_UNLCK) { call->a_args.block = IS_SETLKW(cmd) ? 1 : 0; status = nlmclnt_lock(call, fl); } else @@ -432,13 +433,14 @@ nlmclnt_test(struct nlm_rqst *req, struct file_lock *fl) { int status; - status = nlmclnt_call(nfs_file_cred(fl->fl_file), req, NLMPROC_TEST); + status = nlmclnt_call(nfs_file_cred(fl->c.flc_file), req, + NLMPROC_TEST); if (status < 0) goto out; switch (req->a_res.status) { case nlm_granted: - fl->fl_type = F_UNLCK; + fl->c.flc_type = F_UNLCK; break; case nlm_lck_denied: /* @@ -446,8 +448,8 @@ nlmclnt_test(struct nlm_rqst *req, struct file_lock *fl) */ fl->fl_start = req->a_res.lock.fl.fl_start; fl->fl_end = req->a_res.lock.fl.fl_end; - fl->fl_type = req->a_res.lock.fl.fl_type; - fl->fl_pid = -req->a_res.lock.fl.fl_pid; + fl->c.flc_type = req->a_res.lock.fl.c.flc_type; + fl->c.flc_pid = -req->a_res.lock.fl.c.flc_pid; break; default: status = nlm_stat_to_errno(req->a_res.status); @@ -485,14 +487,18 @@ static const struct file_lock_operations nlmclnt_lock_ops = { static void nlmclnt_locks_init_private(struct file_lock *fl, struct nlm_host *host) { fl->fl_u.nfs_fl.state = 0; - fl->fl_u.nfs_fl.owner = nlmclnt_find_lockowner(host, fl->fl_owner); + fl->fl_ops = NULL; + fl->fl_u.nfs_fl.owner = nlmclnt_find_lockowner(host, + fl->c.flc_owner); INIT_LIST_HEAD(&fl->fl_u.nfs_fl.list); + if (!fl->fl_u.nfs_fl.owner) + return; fl->fl_ops = &nlmclnt_lock_ops; } static int do_vfs_lock(struct file_lock *fl) { - return locks_lock_file_wait(fl->fl_file, fl); + return locks_lock_file_wait(fl->c.flc_file, fl); } /* @@ -518,11 +524,11 @@ static int do_vfs_lock(struct file_lock *fl) static int nlmclnt_lock(struct nlm_rqst *req, struct file_lock *fl) { - const struct cred *cred = nfs_file_cred(fl->fl_file); + const struct cred *cred = nfs_file_cred(fl->c.flc_file); struct nlm_host *host = req->a_host; struct nlm_res *resp = &req->a_res; struct nlm_wait block; - unsigned char fl_flags = fl->fl_flags; + unsigned char fl_flags = fl->c.flc_flags; unsigned char fl_type; __be32 b_status; int status = -ENOLCK; @@ -531,9 +537,9 @@ nlmclnt_lock(struct nlm_rqst *req, struct file_lock *fl) goto out; req->a_args.state = nsm_local_state; - fl->fl_flags |= FL_ACCESS; + fl->c.flc_flags |= FL_ACCESS; status = do_vfs_lock(fl); - fl->fl_flags = fl_flags; + fl->c.flc_flags = fl_flags; if (status < 0) goto out; @@ -591,11 +597,11 @@ nlmclnt_lock(struct nlm_rqst *req, struct file_lock *fl) goto again; } /* Ensure the resulting lock will get added to granted list */ - fl->fl_flags |= FL_SLEEP; + fl->c.flc_flags |= FL_SLEEP; if (do_vfs_lock(fl) < 0) printk(KERN_WARNING "%s: VFS is out of sync with lock manager!\n", __func__); up_read(&host->h_rwsem); - fl->fl_flags = fl_flags; + fl->c.flc_flags = fl_flags; status = 0; } if (status < 0) @@ -622,13 +628,13 @@ nlmclnt_lock(struct nlm_rqst *req, struct file_lock *fl) req->a_host->h_addrlen, req->a_res.status); dprintk("lockd: lock attempt ended in fatal error.\n" " Attempting to unlock.\n"); - fl_type = fl->fl_type; - fl->fl_type = F_UNLCK; + fl_type = fl->c.flc_type; + fl->c.flc_type = F_UNLCK; down_read(&host->h_rwsem); do_vfs_lock(fl); up_read(&host->h_rwsem); - fl->fl_type = fl_type; - fl->fl_flags = fl_flags; + fl->c.flc_type = fl_type; + fl->c.flc_flags = fl_flags; nlmclnt_async_call(cred, req, NLMPROC_UNLOCK, &nlmclnt_unlock_ops); return status; } @@ -651,12 +657,14 @@ nlmclnt_reclaim(struct nlm_host *host, struct file_lock *fl, nlmclnt_setlockargs(req, fl); req->a_args.reclaim = 1; - status = nlmclnt_call(nfs_file_cred(fl->fl_file), req, NLMPROC_LOCK); + status = nlmclnt_call(nfs_file_cred(fl->c.flc_file), req, + NLMPROC_LOCK); if (status >= 0 && req->a_res.status == nlm_granted) return 0; printk(KERN_WARNING "lockd: failed to reclaim lock for pid %d " - "(errno %d, status %d)\n", fl->fl_pid, + "(errno %d, status %d)\n", + fl->c.flc_pid, status, ntohl(req->a_res.status)); /* @@ -683,26 +691,26 @@ nlmclnt_unlock(struct nlm_rqst *req, struct file_lock *fl) struct nlm_host *host = req->a_host; struct nlm_res *resp = &req->a_res; int status; - unsigned char fl_flags = fl->fl_flags; + unsigned char fl_flags = fl->c.flc_flags; /* * Note: the server is supposed to either grant us the unlock * request, or to deny it with NLM_LCK_DENIED_GRACE_PERIOD. In either * case, we want to unlock. */ - fl->fl_flags |= FL_EXISTS; + fl->c.flc_flags |= FL_EXISTS; down_read(&host->h_rwsem); status = do_vfs_lock(fl); up_read(&host->h_rwsem); - fl->fl_flags = fl_flags; + fl->c.flc_flags = fl_flags; if (status == -ENOENT) { status = 0; goto out; } refcount_inc(&req->a_count); - status = nlmclnt_async_call(nfs_file_cred(fl->fl_file), req, - NLMPROC_UNLOCK, &nlmclnt_unlock_ops); + status = nlmclnt_async_call(nfs_file_cred(fl->c.flc_file), req, + NLMPROC_UNLOCK, &nlmclnt_unlock_ops); if (status < 0) goto out; @@ -795,8 +803,8 @@ static int nlmclnt_cancel(struct nlm_host *host, int block, struct file_lock *fl req->a_args.block = block; refcount_inc(&req->a_count); - status = nlmclnt_async_call(nfs_file_cred(fl->fl_file), req, - NLMPROC_CANCEL, &nlmclnt_cancel_ops); + status = nlmclnt_async_call(nfs_file_cred(fl->c.flc_file), req, + NLMPROC_CANCEL, &nlmclnt_cancel_ops); if (status == 0 && req->a_res.status == nlm_lck_denied) status = -ENOLCK; nlmclnt_release_call(req); diff --git a/fs/lockd/clntxdr.c b/fs/lockd/clntxdr.c index 4df62f635529..a3e97278b997 100644 --- a/fs/lockd/clntxdr.c +++ b/fs/lockd/clntxdr.c @@ -238,7 +238,7 @@ static void encode_nlm_holder(struct xdr_stream *xdr, u32 l_offset, l_len; __be32 *p; - encode_bool(xdr, lock->fl.fl_type == F_RDLCK); + encode_bool(xdr, lock->fl.c.flc_type == F_RDLCK); encode_int32(xdr, lock->svid); encode_netobj(xdr, lock->oh.data, lock->oh.len); @@ -265,7 +265,7 @@ static int decode_nlm_holder(struct xdr_stream *xdr, struct nlm_res *result) goto out_overflow; exclusive = be32_to_cpup(p++); lock->svid = be32_to_cpup(p); - fl->fl_pid = (pid_t)lock->svid; + fl->c.flc_pid = (pid_t)lock->svid; error = decode_netobj(xdr, &lock->oh); if (unlikely(error)) @@ -275,8 +275,8 @@ static int decode_nlm_holder(struct xdr_stream *xdr, struct nlm_res *result) if (unlikely(p == NULL)) goto out_overflow; - fl->fl_flags = FL_POSIX; - fl->fl_type = exclusive != 0 ? F_WRLCK : F_RDLCK; + fl->c.flc_flags = FL_POSIX; + fl->c.flc_type = exclusive != 0 ? F_WRLCK : F_RDLCK; l_offset = be32_to_cpup(p++); l_len = be32_to_cpup(p); end = l_offset + l_len - 1; @@ -357,7 +357,7 @@ static void nlm_xdr_enc_testargs(struct rpc_rqst *req, const struct nlm_lock *lock = &args->lock; encode_cookie(xdr, &args->cookie); - encode_bool(xdr, lock->fl.fl_type == F_WRLCK); + encode_bool(xdr, lock->fl.c.flc_type == F_WRLCK); encode_nlm_lock(xdr, lock); } @@ -380,7 +380,7 @@ static void nlm_xdr_enc_lockargs(struct rpc_rqst *req, encode_cookie(xdr, &args->cookie); encode_bool(xdr, args->block); - encode_bool(xdr, lock->fl.fl_type == F_WRLCK); + encode_bool(xdr, lock->fl.c.flc_type == F_WRLCK); encode_nlm_lock(xdr, lock); encode_bool(xdr, args->reclaim); encode_int32(xdr, args->state); @@ -403,7 +403,7 @@ static void nlm_xdr_enc_cancargs(struct rpc_rqst *req, encode_cookie(xdr, &args->cookie); encode_bool(xdr, args->block); - encode_bool(xdr, lock->fl.fl_type == F_WRLCK); + encode_bool(xdr, lock->fl.c.flc_type == F_WRLCK); encode_nlm_lock(xdr, lock); } diff --git a/fs/lockd/svc.c b/fs/lockd/svc.c index a62331487ebf..35405e935ff0 100644 --- a/fs/lockd/svc.c +++ b/fs/lockd/svc.c @@ -48,7 +48,7 @@ static struct svc_program nlmsvc_program; -const struct nlmsvc_binding *nlmsvc_ops; +const struct nlmsvc_binding __rcu *nlmsvc_ops; EXPORT_SYMBOL_GPL(nlmsvc_ops); static DEFINE_MUTEX(nlmsvc_mutex); @@ -142,7 +142,7 @@ lockd(void *vrqstp) nlmsvc_retry_blocked(); svc_recv(rqstp); } - if (nlmsvc_ops) + if (rcu_access_pointer(nlmsvc_ops)) nlmsvc_invalidate_all(); nlm_shutdown_hosts(); cancel_delayed_work_sync(&ln->grace_period_end); diff --git a/fs/lockd/svc4proc.c b/fs/lockd/svc4proc.c index 28ba7b1460aa..ea3b689f8b93 100644 --- a/fs/lockd/svc4proc.c +++ b/fs/lockd/svc4proc.c @@ -29,7 +29,7 @@ nlm4svc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp, __be32 error = 0; /* nfsd callbacks must have been installed for this procedure */ - if (!nlmsvc_ops) + if (!rcu_access_pointer(nlmsvc_ops)) return nlm_lck_denied_nolocks; if (lock->lock_start > OFFSET_MAX || @@ -52,16 +52,16 @@ nlm4svc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp, *filp = file; /* Set up the missing parts of the file_lock structure */ - lock->fl.fl_flags = FL_POSIX; - lock->fl.fl_file = file->f_file[mode]; - lock->fl.fl_pid = current->tgid; + lock->fl.c.flc_flags = FL_POSIX; + lock->fl.c.flc_file = file->f_file[mode]; + lock->fl.c.flc_pid = current->tgid; lock->fl.fl_start = (loff_t)lock->lock_start; lock->fl.fl_end = lock->lock_len ? (loff_t)(lock->lock_start + lock->lock_len - 1) : OFFSET_MAX; lock->fl.fl_lmops = &nlmsvc_lock_operations; nlmsvc_locks_init_private(&lock->fl, host, (pid_t)lock->svid); - if (!lock->fl.fl_owner) { + if (!lock->fl.c.flc_owner) { /* lockowner allocation has failed */ nlmsvc_release_host(host); return nlm_lck_denied_nolocks; @@ -487,7 +487,7 @@ nlm4svc_proc_granted_res(struct svc_rqst *rqstp) { struct nlm_res *argp = rqstp->rq_argp; - if (!nlmsvc_ops) + if (!rcu_access_pointer(nlmsvc_ops)) return rpc_success; dprintk("lockd: GRANTED_RES called\n"); diff --git a/fs/lockd/svclock.c b/fs/lockd/svclock.c index ef6d8061460d..13af9793a3ab 100644 --- a/fs/lockd/svclock.c +++ b/fs/lockd/svclock.c @@ -151,16 +151,17 @@ nlmsvc_lookup_block(struct nlm_file *file, struct nlm_lock *lock) struct file_lock *fl; dprintk("lockd: nlmsvc_lookup_block f=%p pd=%d %Ld-%Ld ty=%d\n", - file, lock->fl.fl_pid, + file, lock->fl.c.flc_pid, (long long)lock->fl.fl_start, - (long long)lock->fl.fl_end, lock->fl.fl_type); + (long long)lock->fl.fl_end, + lock->fl.c.flc_type); spin_lock(&nlm_blocked_lock); list_for_each_entry(block, &nlm_blocked, b_list) { fl = &block->b_call->a_args.lock.fl; dprintk("lockd: check f=%p pd=%d %Ld-%Ld ty=%d cookie=%s\n", - block->b_file, fl->fl_pid, + block->b_file, fl->c.flc_pid, (long long)fl->fl_start, - (long long)fl->fl_end, fl->fl_type, + (long long)fl->fl_end, fl->c.flc_type, nlmdbg_cookie2a(&block->b_call->a_args.cookie)); if (block->b_file == file && nlm_compare_locks(fl, &lock->fl)) { kref_get(&block->b_count); @@ -245,7 +246,7 @@ nlmsvc_create_block(struct svc_rqst *rqstp, struct nlm_host *host, goto failed_free; /* Set notifier function for VFS, and init args */ - call->a_args.lock.fl.fl_flags |= FL_SLEEP; + call->a_args.lock.fl.c.flc_flags |= FL_SLEEP; call->a_args.lock.fl.fl_lmops = &nlmsvc_lock_operations; nlmclnt_next_cookie(&call->a_args.cookie); @@ -403,14 +404,14 @@ static struct nlm_lockowner *nlmsvc_find_lockowner(struct nlm_host *host, pid_t void nlmsvc_release_lockowner(struct nlm_lock *lock) { - if (lock->fl.fl_owner) - nlmsvc_put_lockowner(lock->fl.fl_owner); + if (lock->fl.c.flc_owner) + nlmsvc_put_lockowner(lock->fl.c.flc_owner); } void nlmsvc_locks_init_private(struct file_lock *fl, struct nlm_host *host, pid_t pid) { - fl->fl_owner = nlmsvc_find_lockowner(host, pid); + fl->c.flc_owner = nlmsvc_find_lockowner(host, pid); } /* @@ -426,7 +427,7 @@ static int nlmsvc_setgrantargs(struct nlm_rqst *call, struct nlm_lock *lock) /* set default data area */ call->a_args.lock.oh.data = call->a_owner; - call->a_args.lock.svid = ((struct nlm_lockowner *)lock->fl.fl_owner)->pid; + call->a_args.lock.svid = ((struct nlm_lockowner *) lock->fl.c.flc_owner)->pid; if (lock->oh.len > NLMCLNT_OHSIZE) { void *data = kmalloc(lock->oh.len, GFP_KERNEL); @@ -492,7 +493,8 @@ nlmsvc_lock(struct svc_rqst *rqstp, struct nlm_file *file, dprintk("lockd: nlmsvc_lock(%s/%ld, ty=%d, pi=%d, %Ld-%Ld, bl=%d)\n", inode->i_sb->s_id, inode->i_ino, - lock->fl.fl_type, lock->fl.fl_pid, + lock->fl.c.flc_type, + lock->fl.c.flc_pid, (long long)lock->fl.fl_start, (long long)lock->fl.fl_end, wait); @@ -515,7 +517,7 @@ nlmsvc_lock(struct svc_rqst *rqstp, struct nlm_file *file, goto out; lock = &block->b_call->a_args.lock; } else - lock->fl.fl_flags &= ~FL_SLEEP; + lock->fl.c.flc_flags &= ~FL_SLEEP; if (block->b_flags & B_QUEUED) { dprintk("lockd: nlmsvc_lock deferred block %p flags %d\n", @@ -544,10 +546,10 @@ nlmsvc_lock(struct svc_rqst *rqstp, struct nlm_file *file, } if (!wait) - lock->fl.fl_flags &= ~FL_SLEEP; + lock->fl.c.flc_flags &= ~FL_SLEEP; mode = lock_to_openmode(&lock->fl); error = vfs_lock_file(file->f_file[mode], F_SETLK, &lock->fl, NULL); - lock->fl.fl_flags &= ~FL_SLEEP; + lock->fl.c.flc_flags &= ~FL_SLEEP; dprintk("lockd: vfs_lock_file returned %d\n", error); switch (error) { @@ -605,7 +607,7 @@ nlmsvc_testlock(struct svc_rqst *rqstp, struct nlm_file *file, dprintk("lockd: nlmsvc_testlock(%s/%ld, ty=%d, %Ld-%Ld)\n", nlmsvc_file_inode(file)->i_sb->s_id, nlmsvc_file_inode(file)->i_ino, - lock->fl.fl_type, + lock->fl.c.flc_type, (long long)lock->fl.fl_start, (long long)lock->fl.fl_end); @@ -617,10 +619,10 @@ nlmsvc_testlock(struct svc_rqst *rqstp, struct nlm_file *file, mode = lock_to_openmode(&lock->fl); locks_init_lock(&conflock->fl); /* vfs_test_lock only uses start, end, and owner, but tests fl_file */ - conflock->fl.fl_file = lock->fl.fl_file; + conflock->fl.c.flc_file = lock->fl.c.flc_file; conflock->fl.fl_start = lock->fl.fl_start; conflock->fl.fl_end = lock->fl.fl_end; - conflock->fl.fl_owner = lock->fl.fl_owner; + conflock->fl.c.flc_owner = lock->fl.c.flc_owner; error = vfs_test_lock(file->f_file[mode], &conflock->fl); if (error) { /* We can't currently deal with deferred test requests */ @@ -631,18 +633,18 @@ nlmsvc_testlock(struct svc_rqst *rqstp, struct nlm_file *file, goto out; } - if (conflock->fl.fl_type == F_UNLCK) { + if (conflock->fl.c.flc_type == F_UNLCK) { ret = nlm_granted; goto out; } dprintk("lockd: conflicting lock(ty=%d, %Ld-%Ld)\n", - conflock->fl.fl_type, (long long)conflock->fl.fl_start, + conflock->fl.c.flc_type, (long long)conflock->fl.fl_start, (long long)conflock->fl.fl_end); conflock->caller = "somehost"; /* FIXME */ conflock->len = strlen(conflock->caller); conflock->oh.len = 0; /* don't return OH info */ - conflock->svid = conflock->fl.fl_pid; + conflock->svid = conflock->fl.c.flc_pid; locks_release_private(&conflock->fl); ret = nlm_lck_denied; @@ -665,21 +667,21 @@ nlmsvc_unlock(struct net *net, struct nlm_file *file, struct nlm_lock *lock) dprintk("lockd: nlmsvc_unlock(%s/%ld, pi=%d, %Ld-%Ld)\n", nlmsvc_file_inode(file)->i_sb->s_id, nlmsvc_file_inode(file)->i_ino, - lock->fl.fl_pid, + lock->fl.c.flc_pid, (long long)lock->fl.fl_start, (long long)lock->fl.fl_end); /* First, cancel any lock that might be there */ nlmsvc_cancel_blocked(net, file, lock); - lock->fl.fl_type = F_UNLCK; - lock->fl.fl_file = file->f_file[O_RDONLY]; - if (lock->fl.fl_file) - error = vfs_lock_file(lock->fl.fl_file, F_SETLK, + lock->fl.c.flc_type = F_UNLCK; + lock->fl.c.flc_file = file->f_file[O_RDONLY]; + if (lock->fl.c.flc_file) + error = vfs_lock_file(lock->fl.c.flc_file, F_SETLK, &lock->fl, NULL); - lock->fl.fl_file = file->f_file[O_WRONLY]; - if (lock->fl.fl_file) - error |= vfs_lock_file(lock->fl.fl_file, F_SETLK, + lock->fl.c.flc_file = file->f_file[O_WRONLY]; + if (lock->fl.c.flc_file) + error |= vfs_lock_file(lock->fl.c.flc_file, F_SETLK, &lock->fl, NULL); return (error < 0)? nlm_lck_denied_nolocks : nlm_granted; @@ -702,7 +704,7 @@ nlmsvc_cancel_blocked(struct net *net, struct nlm_file *file, struct nlm_lock *l dprintk("lockd: nlmsvc_cancel(%s/%ld, pi=%d, %Ld-%Ld)\n", nlmsvc_file_inode(file)->i_sb->s_id, nlmsvc_file_inode(file)->i_ino, - lock->fl.fl_pid, + lock->fl.c.flc_pid, (long long)lock->fl.fl_start, (long long)lock->fl.fl_end); @@ -855,12 +857,12 @@ nlmsvc_grant_blocked(struct nlm_block *block) /* vfs_lock_file() can mangle fl_start and fl_end, but we need * them unchanged for the GRANT_MSG */ - lock->fl.fl_flags |= FL_SLEEP; + lock->fl.c.flc_flags |= FL_SLEEP; fl_start = lock->fl.fl_start; fl_end = lock->fl.fl_end; mode = lock_to_openmode(&lock->fl); error = vfs_lock_file(file->f_file[mode], F_SETLK, &lock->fl, NULL); - lock->fl.fl_flags &= ~FL_SLEEP; + lock->fl.c.flc_flags &= ~FL_SLEEP; lock->fl.fl_start = fl_start; lock->fl.fl_end = fl_end; @@ -985,8 +987,8 @@ nlmsvc_grant_reply(struct nlm_cookie *cookie, __be32 status) /* Client doesn't want it, just unlock it */ nlmsvc_unlink_block(block); fl = &block->b_call->a_args.lock.fl; - fl->fl_type = F_UNLCK; - error = vfs_lock_file(fl->fl_file, F_SETLK, fl, NULL); + fl->c.flc_type = F_UNLCK; + error = vfs_lock_file(fl->c.flc_file, F_SETLK, fl, NULL); if (error) pr_warn("lockd: unable to unlock lock rejected by client!\n"); break; diff --git a/fs/lockd/svcproc.c b/fs/lockd/svcproc.c index 1a4459763644..44da63392a09 100644 --- a/fs/lockd/svcproc.c +++ b/fs/lockd/svcproc.c @@ -59,7 +59,7 @@ nlmsvc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp, __be32 error = 0; /* nfsd callbacks must have been installed for this procedure */ - if (!nlmsvc_ops) + if (!rcu_access_pointer(nlmsvc_ops)) return nlm_lck_denied_nolocks; /* Obtain host handle */ @@ -77,12 +77,12 @@ nlmsvc_retrieve_args(struct svc_rqst *rqstp, struct nlm_args *argp, /* Set up the missing parts of the file_lock structure */ mode = lock_to_openmode(&lock->fl); - lock->fl.fl_flags = FL_POSIX; - lock->fl.fl_file = file->f_file[mode]; - lock->fl.fl_pid = current->tgid; + lock->fl.c.flc_flags = FL_POSIX; + lock->fl.c.flc_file = file->f_file[mode]; + lock->fl.c.flc_pid = current->tgid; lock->fl.fl_lmops = &nlmsvc_lock_operations; nlmsvc_locks_init_private(&lock->fl, host, (pid_t)lock->svid); - if (!lock->fl.fl_owner) { + if (!lock->fl.c.flc_owner) { /* lockowner allocation has failed */ nlmsvc_release_host(host); return nlm_lck_denied_nolocks; @@ -521,7 +521,7 @@ nlmsvc_proc_granted_res(struct svc_rqst *rqstp) { struct nlm_res *argp = rqstp->rq_argp; - if (!nlmsvc_ops) + if (!rcu_access_pointer(nlmsvc_ops)) return rpc_success; dprintk("lockd: GRANTED_RES called\n"); diff --git a/fs/lockd/svcsubs.c b/fs/lockd/svcsubs.c index 9e2a67bfd070..592ff6938de1 100644 --- a/fs/lockd/svcsubs.c +++ b/fs/lockd/svcsubs.c @@ -73,7 +73,7 @@ static inline unsigned int file_hash(struct nfs_fh *f) int lock_to_openmode(struct file_lock *lock) { - return (lock->fl_type == F_WRLCK) ? O_WRONLY : O_RDONLY; + return (lock->c.flc_type == F_WRLCK) ? O_WRONLY : O_RDONLY; } /* @@ -87,11 +87,22 @@ static __be32 nlm_do_fopen(struct svc_rqst *rqstp, struct nlm_file *file, int mode) { struct file **fp = &file->f_file[mode]; + const struct nlmsvc_binding *ops; __be32 nfserr; if (*fp) return 0; - nfserr = nlmsvc_ops->fopen(rqstp, &file->f_handle, fp, mode); + + rcu_read_lock(); + ops = rcu_dereference(nlmsvc_ops); + if (!ops || !try_module_get(ops->owner)) { + rcu_read_unlock(); + return nlm_lck_denied_nolocks; + } + rcu_read_unlock(); + + nfserr = ops->fopen(rqstp, &file->f_handle, fp, mode); + module_put(ops->owner); if (nfserr) dprintk("lockd: open failed (error %d)\n", nfserr); return nfserr; @@ -167,11 +178,28 @@ nlm_delete_file(struct nlm_file *file) { nlm_debug_print_file("closing file", file); if (!hlist_unhashed(&file->f_list)) { + const struct nlmsvc_binding *ops; + bool have_ops; + hlist_del(&file->f_list); - if (file->f_file[O_RDONLY]) - nlmsvc_ops->fclose(file->f_file[O_RDONLY]); - if (file->f_file[O_WRONLY]) - nlmsvc_ops->fclose(file->f_file[O_WRONLY]); + + rcu_read_lock(); + ops = rcu_dereference(nlmsvc_ops); + have_ops = ops && try_module_get(ops->owner); + rcu_read_unlock(); + + if (have_ops) { + if (file->f_file[O_RDONLY]) + ops->fclose(file->f_file[O_RDONLY]); + if (file->f_file[O_WRONLY]) + ops->fclose(file->f_file[O_WRONLY]); + module_put(ops->owner); + } else { + if (file->f_file[O_RDONLY]) + fput(file->f_file[O_RDONLY]); + if (file->f_file[O_WRONLY]) + fput(file->f_file[O_WRONLY]); + } kfree(file); } else { printk(KERN_WARNING "lockd: attempt to release unknown file!\n"); @@ -183,18 +211,18 @@ static int nlm_unlock_files(struct nlm_file *file, const struct file_lock *fl) struct file_lock lock; locks_init_lock(&lock); - lock.fl_type = F_UNLCK; + lock.c.flc_type = F_UNLCK; lock.fl_start = 0; lock.fl_end = OFFSET_MAX; - lock.fl_owner = fl->fl_owner; - lock.fl_pid = fl->fl_pid; - lock.fl_flags = FL_POSIX; + lock.c.flc_owner = fl->c.flc_owner; + lock.c.flc_pid = fl->c.flc_pid; + lock.c.flc_flags = FL_POSIX; - lock.fl_file = file->f_file[O_RDONLY]; - if (lock.fl_file && vfs_lock_file(lock.fl_file, F_SETLK, &lock, NULL)) + lock.c.flc_file = file->f_file[O_RDONLY]; + if (lock.c.flc_file && vfs_lock_file(lock.c.flc_file, F_SETLK, &lock, NULL)) goto out_err; - lock.fl_file = file->f_file[O_WRONLY]; - if (lock.fl_file && vfs_lock_file(lock.fl_file, F_SETLK, &lock, NULL)) + lock.c.flc_file = file->f_file[O_WRONLY]; + if (lock.c.flc_file && vfs_lock_file(lock.c.flc_file, F_SETLK, &lock, NULL)) goto out_err; return 0; out_err: @@ -227,7 +255,7 @@ nlm_traverse_locks(struct nlm_host *host, struct nlm_file *file, /* update current lock count */ file->f_locks++; - lockhost = ((struct nlm_lockowner *)fl->fl_owner)->host; + lockhost = ((struct nlm_lockowner *) fl->c.flc_owner)->host; if (match(lockhost, host)) { spin_unlock(&flctx->flc_lock); @@ -493,7 +521,7 @@ EXPORT_SYMBOL_GPL(nlmsvc_unlock_all_by_sb); static int nlmsvc_match_ip(void *datap, struct nlm_host *host) { - return rpc_cmp_addr(nlm_srcaddr(host), datap); + return rpc_cmp_addr(nlm_srcaddr(datap), (struct sockaddr *)host); } /** diff --git a/fs/lockd/xdr.c b/fs/lockd/xdr.c index 2fb5748dae0c..adfcce2bf11b 100644 --- a/fs/lockd/xdr.c +++ b/fs/lockd/xdr.c @@ -88,8 +88,8 @@ svcxdr_decode_lock(struct xdr_stream *xdr, struct nlm_lock *lock) return false; locks_init_lock(fl); - fl->fl_flags = FL_POSIX; - fl->fl_type = F_RDLCK; + fl->c.flc_flags = FL_POSIX; + fl->c.flc_type = F_RDLCK; end = start + len - 1; fl->fl_start = s32_to_loff_t(start); if (len == 0 || end < 0) @@ -107,7 +107,7 @@ svcxdr_encode_holder(struct xdr_stream *xdr, const struct nlm_lock *lock) s32 start, len; /* exclusive */ - if (xdr_stream_encode_bool(xdr, fl->fl_type != F_RDLCK) < 0) + if (xdr_stream_encode_bool(xdr, fl->c.flc_type != F_RDLCK) < 0) return false; if (xdr_stream_encode_u32(xdr, lock->svid) < 0) return false; @@ -164,7 +164,7 @@ nlmsvc_decode_testargs(struct svc_rqst *rqstp, struct xdr_stream *xdr) if (!svcxdr_decode_lock(xdr, &argp->lock)) return false; if (exclusive) - argp->lock.fl.fl_type = F_WRLCK; + argp->lock.fl.c.flc_type = F_WRLCK; return true; } @@ -184,7 +184,7 @@ nlmsvc_decode_lockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr) if (!svcxdr_decode_lock(xdr, &argp->lock)) return false; if (exclusive) - argp->lock.fl.fl_type = F_WRLCK; + argp->lock.fl.c.flc_type = F_WRLCK; if (xdr_stream_decode_bool(xdr, &argp->reclaim) < 0) return false; if (xdr_stream_decode_u32(xdr, &argp->state) < 0) @@ -209,7 +209,7 @@ nlmsvc_decode_cancargs(struct svc_rqst *rqstp, struct xdr_stream *xdr) if (!svcxdr_decode_lock(xdr, &argp->lock)) return false; if (exclusive) - argp->lock.fl.fl_type = F_WRLCK; + argp->lock.fl.c.flc_type = F_WRLCK; return true; } @@ -223,7 +223,7 @@ nlmsvc_decode_unlockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr) return false; if (!svcxdr_decode_lock(xdr, &argp->lock)) return false; - argp->lock.fl.fl_type = F_UNLCK; + argp->lock.fl.c.flc_type = F_UNLCK; return true; } diff --git a/fs/lockd/xdr4.c b/fs/lockd/xdr4.c index 5fcbf30cd275..3d28b9c3ed15 100644 --- a/fs/lockd/xdr4.c +++ b/fs/lockd/xdr4.c @@ -89,8 +89,8 @@ svcxdr_decode_lock(struct xdr_stream *xdr, struct nlm_lock *lock) return false; locks_init_lock(fl); - fl->fl_flags = FL_POSIX; - fl->fl_type = F_RDLCK; + fl->c.flc_flags = FL_POSIX; + fl->c.flc_type = F_RDLCK; nlm4svc_set_file_lock_range(fl, lock->lock_start, lock->lock_len); return true; } @@ -102,7 +102,7 @@ svcxdr_encode_holder(struct xdr_stream *xdr, const struct nlm_lock *lock) s64 start, len; /* exclusive */ - if (xdr_stream_encode_bool(xdr, fl->fl_type != F_RDLCK) < 0) + if (xdr_stream_encode_bool(xdr, fl->c.flc_type != F_RDLCK) < 0) return false; if (xdr_stream_encode_u32(xdr, lock->svid) < 0) return false; @@ -159,7 +159,7 @@ nlm4svc_decode_testargs(struct svc_rqst *rqstp, struct xdr_stream *xdr) if (!svcxdr_decode_lock(xdr, &argp->lock)) return false; if (exclusive) - argp->lock.fl.fl_type = F_WRLCK; + argp->lock.fl.c.flc_type = F_WRLCK; return true; } @@ -179,7 +179,7 @@ nlm4svc_decode_lockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr) if (!svcxdr_decode_lock(xdr, &argp->lock)) return false; if (exclusive) - argp->lock.fl.fl_type = F_WRLCK; + argp->lock.fl.c.flc_type = F_WRLCK; if (xdr_stream_decode_bool(xdr, &argp->reclaim) < 0) return false; if (xdr_stream_decode_u32(xdr, &argp->state) < 0) @@ -204,7 +204,7 @@ nlm4svc_decode_cancargs(struct svc_rqst *rqstp, struct xdr_stream *xdr) if (!svcxdr_decode_lock(xdr, &argp->lock)) return false; if (exclusive) - argp->lock.fl.fl_type = F_WRLCK; + argp->lock.fl.c.flc_type = F_WRLCK; return true; } @@ -218,7 +218,7 @@ nlm4svc_decode_unlockargs(struct svc_rqst *rqstp, struct xdr_stream *xdr) return false; if (!svcxdr_decode_lock(xdr, &argp->lock)) return false; - argp->lock.fl.fl_type = F_UNLCK; + argp->lock.fl.c.flc_type = F_UNLCK; return true; } diff --git a/fs/locks.c b/fs/locks.c index 19e9fcd1d4ca..90ec12e0f4f9 100644 --- a/fs/locks.c +++ b/fs/locks.c @@ -675,7 +675,7 @@ static void __locks_wake_up_blocks(struct file_lock *blocker) if (waiter->fl_lmops && waiter->fl_lmops->lm_notify) waiter->fl_lmops->lm_notify(waiter); else - wake_up(&waiter->fl_wait); + locks_wake_up(waiter); /* * The setting of fl_blocker to NULL marks the "done" @@ -842,9 +842,9 @@ locks_delete_lock_ctx(struct file_lock *fl, struct list_head *dispose) static bool locks_conflict(struct file_lock *caller_fl, struct file_lock *sys_fl) { - if (sys_fl->fl_type == F_WRLCK) + if (lock_is_write(sys_fl)) return true; - if (caller_fl->fl_type == F_WRLCK) + if (lock_is_write(caller_fl)) return true; return false; } @@ -875,7 +875,7 @@ static bool posix_test_locks_conflict(struct file_lock *caller_fl, struct file_lock *sys_fl) { /* F_UNLCK checks any locks on the same fd. */ - if (caller_fl->fl_type == F_UNLCK) { + if (lock_is_unlock(caller_fl)) { if (!posix_same_owner(caller_fl, sys_fl)) return false; return locks_overlap(caller_fl, sys_fl); @@ -1056,7 +1056,7 @@ static int flock_lock_inode(struct inode *inode, struct file_lock *request) break; } - if (request->fl_type == F_UNLCK) { + if (lock_is_unlock(request)) { if ((request->fl_flags & FL_EXISTS) && !found) error = -ENOENT; goto out; @@ -1108,7 +1108,7 @@ static int posix_lock_inode(struct inode *inode, struct file_lock *request, ctx = locks_get_lock_context(inode, request->fl_type); if (!ctx) - return (request->fl_type == F_UNLCK) ? 0 : -ENOMEM; + return lock_is_unlock(request) ? 0 : -ENOMEM; /* * We may need two file_lock structures for this operation, @@ -1229,7 +1229,7 @@ static int posix_lock_inode(struct inode *inode, struct file_lock *request, continue; if (fl->fl_start > request->fl_end) break; - if (request->fl_type == F_UNLCK) + if (lock_is_unlock(request)) added = true; if (fl->fl_start < request->fl_start) left = fl; @@ -1280,7 +1280,7 @@ static int posix_lock_inode(struct inode *inode, struct file_lock *request, error = 0; if (!added) { - if (request->fl_type == F_UNLCK) { + if (lock_is_unlock(request)) { if (request->fl_flags & FL_EXISTS) error = -ENOENT; goto out; @@ -1609,7 +1609,7 @@ void lease_get_mtime(struct inode *inode, struct timespec64 *time) spin_lock(&ctx->flc_lock); fl = list_first_entry_or_null(&ctx->flc_lease, struct file_lock, fl_list); - if (fl && (fl->fl_type == F_WRLCK)) + if (fl && lock_is_write(fl)) has_lease = true; spin_unlock(&ctx->flc_lock); } diff --git a/fs/mbcache.c b/fs/mbcache.c index 2a4b8b549e93..637e30744613 100644 --- a/fs/mbcache.c +++ b/fs/mbcache.c @@ -406,6 +406,7 @@ void mb_cache_destroy(struct mb_cache *cache) { struct mb_cache_entry *entry, *next; + cancel_work_sync(&cache->c_shrink_work); unregister_shrinker(&cache->c_shrink); /* diff --git a/fs/namei.c b/fs/namei.c index 72918875d62a..bdca82064d3d 100644 --- a/fs/namei.c +++ b/fs/namei.c @@ -126,6 +126,11 @@ #define EMBEDDED_NAME_MAX (PATH_MAX - offsetof(struct filename, iname)) +/* + * Type of the last component on LOOKUP_PARENT + */ +enum {LAST_NORM, LAST_ROOT, LAST_DOT, LAST_DOTDOT}; + struct filename * getname_flags(const char __user *filename, int flags, int *empty) { @@ -2672,15 +2677,22 @@ EXPORT_SYMBOL(kern_path); * @flags: lookup flags * @parent: pointer to struct path to fill * @last: last component - * @type: type of the last component * @root: pointer to struct path of the base directory */ int vfs_path_parent_lookup(struct filename *filename, unsigned int flags, - struct path *parent, struct qstr *last, int *type, + struct path *parent, struct qstr *last, const struct path *root) { - return __filename_parentat(AT_FDCWD, filename, flags, parent, last, - type, root); + int type; + int err = __filename_parentat(AT_FDCWD, filename, flags, parent, last, + &type, root); + if (err) + return err; + if (unlikely(type != LAST_NORM)) { + path_put(parent); + return -EINVAL; + } + return 0; } EXPORT_SYMBOL(vfs_path_parent_lookup); diff --git a/fs/namespace.c b/fs/namespace.c index f185de5f5df4..0d5f71d56b12 100644 --- a/fs/namespace.c +++ b/fs/namespace.c @@ -1191,7 +1191,7 @@ static struct mount *clone_mnt(struct mount *old, struct dentry *root, } mnt->mnt.mnt_flags = old->mnt.mnt_flags; - mnt->mnt.mnt_flags &= ~(MNT_WRITE_HOLD|MNT_MARKED|MNT_INTERNAL); + mnt->mnt.mnt_flags &= ~(MNT_WRITE_HOLD|MNT_MARKED|MNT_INTERNAL|MNT_UMOUNT); atomic_inc(&sb->s_active); mnt->mnt.mnt_idmap = mnt_idmap_get(mnt_idmap(&old->mnt)); @@ -1640,6 +1640,8 @@ static void umount_tree(struct mount *mnt, enum umount_tree_flags how) /* Gather the mounts to umount */ for (p = mnt; p; p = next_mnt(p, mnt)) { + /* A mount is unmounted once. */ + WARN_ON_ONCE(p->mnt.mnt_flags & MNT_UMOUNT); p->mnt.mnt_flags |= MNT_UMOUNT; list_move(&p->mnt_list, &tmp_list); } @@ -1814,6 +1816,9 @@ static int do_umount(struct mount *mnt, int flags) * detach_mounts allows lazily unmounting those mounts instead of * leaking them. * + * The dentry is unhashed before the mounts go so that no lookup finds + * what they covered. The caller removes it for good afterwards. + * * The caller may hold dentry->d_inode->i_mutex. */ void __detach_mounts(struct dentry *dentry) @@ -1827,6 +1832,8 @@ void __detach_mounts(struct dentry *dentry) if (!mp) goto out_unlock; + /* the name goes first, what covered it goes second */ + d_drop(dentry); event++; while (!hlist_empty(&mp->m_list)) { mnt = hlist_entry(mp->m_list.first, struct mount, mnt_mp_list); @@ -2313,15 +2320,31 @@ static int attach_recursive_mnt(struct mount *source_mnt, struct mountpoint *dest_mp, enum mnt_tree_flags_t flags) { - struct user_namespace *user_ns = current->nsproxy->mnt_ns->user_ns; HLIST_HEAD(tree_list); struct mnt_namespace *ns = top_mnt->mnt_ns; + struct user_namespace *user_ns = ns->user_ns; struct mountpoint *smp; struct mount *child, *dest_mnt, *p; struct hlist_node *n; int err = 0; bool moving = flags & MNT_TREE_MOVE, beneath = flags & MNT_TREE_BENEATH; + /* + * A caller in an unprivileged mount namespaces may trigger an + * automount and propagate locked mounts into privileged mount + * namespaces. Take ownership from the target mount namespace. + * It's equivalent for everything but the automount case. + * + * Detached trees in anonymous mount namespaces by be handed + * over via SCM_RIGHTS or inherited in other ways on purpose + * the attaching task's mount namespace is authoritative, not + * the creator of the detached tree. + */ + if (is_anon_ns(ns)) + user_ns = current->nsproxy->mnt_ns->user_ns; + else + user_ns = ns->user_ns; + /* * Preallocate a mountpoint in case the new mounts need to be * mounted beneath mounts on the same mountpoint. @@ -2614,6 +2637,9 @@ static struct mount *__do_loopback(struct path *old_path, int recurse) if (!check_mnt(old) && old_path->dentry->d_op != &ns_dentry_operations) return mnt; + if (recurse && !old->mnt_ns) + return ERR_PTR(-EINVAL); + if (!recurse && has_locked_children(old, old_path->dentry)) return mnt; @@ -3274,7 +3300,7 @@ static int do_add_mount(struct mount *newmnt, struct mountpoint *mp, if (!(mnt_flags & MNT_SHRINKABLE)) return -EINVAL; /* ... and for those we'd better have mountpoint still alive */ - if (!parent->mnt_ns) + if (!is_mounted(&parent->mnt)) return -EINVAL; } @@ -3503,6 +3529,11 @@ void mark_mounts_for_expiry(struct list_head *mounts) } while (!list_empty(&graveyard)) { mnt = list_first_entry(&graveyard, struct mount, mnt_expire); + /* an earlier umount_tree() may have moved a busy mount here */ + if (propagate_mount_busy(mnt, 1)) { + list_move(&mnt->mnt_expire, mounts); + continue; + } touch_mnt_namespace(mnt->mnt_ns); umount_tree(mnt, UMOUNT_PROPAGATE|UMOUNT_SYNC); } @@ -3512,17 +3543,37 @@ void mark_mounts_for_expiry(struct list_head *mounts) EXPORT_SYMBOL_GPL(mark_mounts_for_expiry); +/* + * Unmount @mnt if it's a shrinkable mount without children that nobody uses. + * + * mount_lock must be held for write + */ +static bool shrink_submount(struct mount *mnt) +{ + if (propagate_mount_busy(mnt, 1)) + return false; + touch_mnt_namespace(mnt->mnt_ns); + umount_tree(mnt, UMOUNT_PROPAGATE|UMOUNT_SYNC); + return true; +} + /* * Ripoff of 'select_parent()' * - * search the list of submounts for a given mountpoint, and move any - * shrinkable submounts to the 'graveyard' list. + * unmount the shrinkable submounts of @parent that aren't busy, children + * before their parent, and say whether anything went + * + * The cursor into the children of @this_parent survives the umount of a + * child mount without child mounts. The mounts that get umounted together with + * it are located under receiving mounts of @this_parent and never under + * @this_parent itself. The one exception is @this_parent getting unmounted + * then the walk starts over. */ -static int select_submounts(struct mount *parent, struct list_head *graveyard) +static bool __shrink_submounts(struct mount *parent) { struct mount *this_parent = parent; struct list_head *next; - int found = 0; + bool shrunk = false; repeat: next = this_parent->mnt_mounts.next; @@ -3541,42 +3592,45 @@ static int select_submounts(struct mount *parent, struct list_head *graveyard) this_parent = mnt; goto repeat; } - - if (!propagate_mount_busy(mnt, 1)) { - list_move_tail(&mnt->mnt_expire, graveyard); - found++; - } + if (!shrink_submount(mnt)) + continue; + shrunk = true; + if (unlikely(this_parent->mnt.mnt_flags & MNT_UMOUNT)) + return true; } /* * All done at this level ... ascend and resume the search */ if (this_parent != parent) { - next = this_parent->mnt_child.next; - this_parent = this_parent->mnt_parent; + struct mount *mnt = this_parent; + + next = mnt->mnt_child.next; + this_parent = mnt->mnt_parent; + /* its children are gone, maybe it can go as well */ + if (shrink_submount(mnt)) { + shrunk = true; + if (unlikely(this_parent->mnt.mnt_flags & MNT_UMOUNT)) + return true; + } goto resume; } - return found; + return shrunk; } /* - * process a list of expirable mountpoints with the intent of discarding any - * submounts of a specific parent mountpoint + * unmount the shrinkable submounts of @mnt that aren't busy + * + * The busy check and the umount of a mount are adjacent. An umount can + * still empty or move a mount in a part of the tree that was walked + * already, so walk again until nothing goes. * * mount_lock must be held for write */ static void shrink_submounts(struct mount *mnt) { - LIST_HEAD(graveyard); - struct mount *m; - - /* extract submounts of 'mountpoint' from the expiration list */ - while (select_submounts(mnt, &graveyard)) { - while (!list_empty(&graveyard)) { - m = list_first_entry(&graveyard, struct mount, - mnt_expire); - touch_mnt_namespace(m->mnt_ns); - umount_tree(m, UMOUNT_PROPAGATE|UMOUNT_SYNC); - } + for (;;) { + if (!__shrink_submounts(mnt)) + break; } } @@ -4254,14 +4308,15 @@ SYSCALL_DEFINE2(pivot_root, const char __user *, new_root, new_mnt = real_mount(new.mnt); root_mnt = real_mount(root.mnt); old_mnt = real_mount(old.mnt); + /* only a mounted mount has a parent that namespace_sem pins */ + if (!check_mnt(root_mnt) || !check_mnt(new_mnt)) + goto out4; ex_parent = new_mnt->mnt_parent; root_parent = root_mnt->mnt_parent; if (IS_MNT_SHARED(old_mnt) || IS_MNT_SHARED(ex_parent) || IS_MNT_SHARED(root_parent)) goto out4; - if (!check_mnt(root_mnt) || !check_mnt(new_mnt)) - goto out4; if (new_mnt->mnt.mnt_flags & MNT_LOCKED) goto out4; error = -ENOENT; @@ -4496,6 +4551,13 @@ static int do_mount_setattr(struct path *path, struct mount_kattr *kattr) * propagation. */ namespace_lock(); + /* invent_group_ids() walks the tree, only walk a mounted one */ + if (!is_mounted(&mnt->mnt) || + ((mnt_has_parent(mnt) || !is_anon_ns(mnt->mnt_ns)) && + !check_mnt(mnt))) { + namespace_unlock(); + return -EINVAL; + } if (kattr->propagation == MS_SHARED) { err = invent_group_ids(mnt, kattr->recurse); if (err) { @@ -4508,6 +4570,7 @@ static int do_mount_setattr(struct path *path, struct mount_kattr *kattr) err = -EINVAL; lock_mount_hash(); + /* Checked under namespace_sem already if the propagation changes. */ /* Ensure that this isn't anything purely vfs internal. */ if (!is_mounted(&mnt->mnt)) goto out; diff --git a/fs/nfs/blocklayout/dev.c b/fs/nfs/blocklayout/dev.c index 61ee0b6c0fba..9bb745ad8976 100644 --- a/fs/nfs/blocklayout/dev.c +++ b/fs/nfs/blocklayout/dev.c @@ -16,12 +16,14 @@ static void bl_free_device(struct pnfs_block_dev *dev) { - if (dev->nr_children) { + if (dev->children) { int i; for (i = 0; i < dev->nr_children; i++) bl_free_device(&dev->children[i]); kfree(dev->children); + dev->children = NULL; + dev->nr_children = 0; } else { if (dev->pr_registered) { const struct pr_ops *ops = @@ -34,8 +36,10 @@ bl_free_device(struct pnfs_block_dev *dev) pr_err("failed to unregister PR key.\n"); } - if (dev->bdev) + if (dev->bdev) { blkdev_put(dev->bdev, NULL); + dev->bdev = NULL; + } } } @@ -377,6 +381,7 @@ bl_parse_scsi(struct nfs_server *server, struct pnfs_block_dev *d, out_blkdev_put: blkdev_put(d->bdev, NULL); + d->bdev = NULL; return error; } @@ -412,8 +417,11 @@ bl_parse_concat(struct nfs_server *server, struct pnfs_block_dev *d, for (i = 0; i < v->concat.volumes_count; i++) { ret = bl_parse_deviceid(server, &d->children[i], volumes, v->concat.volumes[i], gfp_mask); - if (ret) + if (ret) { + bl_free_device(&d->children[i]); + bl_free_device(d); return ret; + } d->nr_children++; d->children[i].start += len; @@ -441,8 +449,11 @@ bl_parse_stripe(struct nfs_server *server, struct pnfs_block_dev *d, for (i = 0; i < v->stripe.volumes_count; i++) { ret = bl_parse_deviceid(server, &d->children[i], volumes, v->stripe.volumes[i], gfp_mask); - if (ret) + if (ret) { + bl_free_device(&d->children[i]); + bl_free_device(d); return ret; + } d->nr_children++; len += d->children[i].len; diff --git a/fs/nfs/callback_xdr.c b/fs/nfs/callback_xdr.c index d5f6437da352..a03e7e1e4867 100644 --- a/fs/nfs/callback_xdr.c +++ b/fs/nfs/callback_xdr.c @@ -419,9 +419,9 @@ static __be32 decode_cb_sequence_args(struct svc_rqst *rqstp, args->csa_nrclists = ntohl(*p++); args->csa_rclists = NULL; if (args->csa_nrclists) { - args->csa_rclists = kmalloc_array(args->csa_nrclists, - sizeof(*args->csa_rclists), - GFP_KERNEL); + args->csa_rclists = kcalloc(args->csa_nrclists, + sizeof(*args->csa_rclists), + GFP_KERNEL); if (unlikely(args->csa_rclists == NULL)) return htonl(NFS4ERR_RESOURCE); diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c index bbd582d8a7dc..af7751f05bbd 100644 --- a/fs/nfs/delegation.c +++ b/fs/nfs/delegation.c @@ -436,11 +436,14 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred, struct nfs_inode *nfsi = NFS_I(inode); struct nfs_delegation *delegation, *old_delegation; struct nfs_delegation *freeme = NULL; + bool orphaned = false; int status = 0; delegation = kmalloc(sizeof(*delegation), GFP_KERNEL_ACCOUNT); - if (delegation == NULL) + if (delegation == NULL) { + nfs4_proc_delegreturn(inode, cred, stateid, 0); return -ENOMEM; + } nfs4_stateid_copy(&delegation->stateid, stateid); refcount_set(&delegation->refcount, 1); delegation->type = type; @@ -482,12 +485,16 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred, goto out; } if (test_and_set_bit(NFS_DELEGATION_RETURNING, - &old_delegation->flags)) + &old_delegation->flags)) { + orphaned = true; goto out; + } } freeme = nfs_detach_delegation_locked(nfsi, old_delegation, clp); - if (freeme == NULL) + if (freeme == NULL) { + orphaned = true; goto out; + } add_new: /* * If we didn't revalidate the change attribute before setting @@ -514,8 +521,11 @@ int nfs_inode_set_delegation(struct inode *inode, const struct cred *cred, trace_nfs4_set_delegation(inode, type); out: spin_unlock(&clp->cl_lock); - if (delegation != NULL) + if (delegation != NULL) { + if (orphaned) + nfs_do_return_delegation(inode, delegation, 0); __nfs_free_delegation(delegation); + } if (freeme != NULL) { nfs_do_return_delegation(inode, freeme, 0); nfs_free_delegation(freeme); diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c index 3c60edc22b8d..a1dc55724597 100644 --- a/fs/nfs/flexfilelayout/flexfilelayout.c +++ b/fs/nfs/flexfilelayout/flexfilelayout.c @@ -1127,7 +1127,8 @@ static int ff_layout_async_handle_error_v4(struct rpc_task *task, struct pnfs_layout_hdr *lo = lseg->pls_layout; struct inode *inode = lo->plh_inode; struct nfs4_deviceid_node *devid = FF_LAYOUT_DEVID_NODE(lseg, idx); - struct nfs4_slot_table *tbl = &clp->cl_session->fc_slot_table; + struct nfs4_slot_table *tbl = nfs4_has_session(clp) ? + &clp->cl_session->fc_slot_table : clp->cl_slot_tbl; switch (op_status) { case NFS4_OK: diff --git a/fs/nfs/nfs42xattr.c b/fs/nfs/nfs42xattr.c index 911f634ba3da..c3744b15579c 100644 --- a/fs/nfs/nfs42xattr.c +++ b/fs/nfs/nfs42xattr.c @@ -132,7 +132,7 @@ nfs4_xattr_entry_lru_add(struct nfs4_xattr_entry *entry) lru = (entry->flags & NFS4_XATTR_ENTRY_EXTVAL) ? &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru; - return list_lru_add(lru, &entry->lru); + return list_lru_add_obj(lru, &entry->lru); } static bool @@ -143,7 +143,7 @@ nfs4_xattr_entry_lru_del(struct nfs4_xattr_entry *entry) lru = (entry->flags & NFS4_XATTR_ENTRY_EXTVAL) ? &nfs4_xattr_large_entry_lru : &nfs4_xattr_entry_lru; - return list_lru_del(lru, &entry->lru); + return list_lru_del_obj(lru, &entry->lru); } /* @@ -349,7 +349,7 @@ nfs4_xattr_cache_unlink(struct inode *inode) oldcache = nfsi->xattr_cache; if (oldcache != NULL) { - list_lru_del(&nfs4_xattr_cache_lru, &oldcache->lru); + list_lru_del_obj(&nfs4_xattr_cache_lru, &oldcache->lru); oldcache->inode = NULL; } nfsi->xattr_cache = NULL; @@ -474,7 +474,7 @@ nfs4_xattr_get_cache(struct inode *inode, int add) kref_get(&cache->ref); nfsi->xattr_cache = cache; cache->inode = inode; - list_lru_add(&nfs4_xattr_cache_lru, &cache->lru); + list_lru_add_obj(&nfs4_xattr_cache_lru, &cache->lru); } spin_unlock(&inode->i_lock); diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index ce3ab0a9c0ef..b226dbe85386 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -7635,6 +7635,7 @@ static int nfs4_add_lease(struct file *file, int arg, struct file_lock **lease, { struct inode *inode = file_inode(file); fmode_t type = arg == F_RDLCK ? FMODE_READ : FMODE_WRITE; + fl_owner_t owner = (*lease)->c.flc_owner; int ret; /* No delegation, no lease */ @@ -7644,7 +7645,8 @@ static int nfs4_add_lease(struct file *file, int arg, struct file_lock **lease, if (ret || nfs4_have_delegation(inode, type)) return ret; /* We raced with a delegation return */ - nfs4_delete_lease(file, priv); + dprintk("%s: raced with a delegation return\n", __func__); + nfs4_delete_lease(file, &owner); return -EAGAIN; } diff --git a/fs/nfs/nfs4session.c b/fs/nfs/nfs4session.c index 5db460476bf2..c59e710eb7cc 100644 --- a/fs/nfs/nfs4session.c +++ b/fs/nfs/nfs4session.c @@ -634,16 +634,22 @@ int nfs4_init_ds_session(struct nfs_client *clp, unsigned long lease_time) int ret; spin_lock(&clp->cl_lock); - if (test_and_clear_bit(NFS4_SESSION_INITING, &session->session_state)) { - /* - * Do not set NFS_CS_CHECK_LEASE_TIME instead set the - * DS lease to be equal to the MDS lease. - */ + /* + * Do not set NFS_CS_CHECK_LEASE_TIME instead set the + * DS lease to be equal to the MDS lease. + * + * A v4.0 DS has no session, so seed the lease every time. + */ + if (!session || + test_and_clear_bit(NFS4_SESSION_INITING, &session->session_state)) { clp->cl_lease_time = lease_time; clp->cl_last_renewal = jiffies; } spin_unlock(&clp->cl_lock); + if (!session) + return 0; + ret = nfs41_check_session_ready(clp); if (ret) return ret; diff --git a/fs/nfsd/auth.c b/fs/nfsd/auth.c index e6beaaf4f170..93e33d1ee891 100644 --- a/fs/nfsd/auth.c +++ b/fs/nfsd/auth.c @@ -5,26 +5,26 @@ #include "nfsd.h" #include "auth.h" -int nfsexp_flags(struct svc_rqst *rqstp, struct svc_export *exp) +int nfsexp_flags(struct svc_cred *cred, struct svc_export *exp) { struct exp_flavor_info *f; struct exp_flavor_info *end = exp->ex_flavors + exp->ex_nflavors; for (f = exp->ex_flavors; f < end; f++) { - if (f->pseudoflavor == rqstp->rq_cred.cr_flavor) + if (f->pseudoflavor == cred->cr_flavor) return f->flags; } return exp->ex_flags; } -int nfsd_setuser(struct svc_rqst *rqstp, struct svc_export *exp) +int nfsd_setuser(struct svc_cred *cred, struct svc_export *exp) { struct group_info *rqgi; struct group_info *gi; struct cred *new; int i; - int flags = nfsexp_flags(rqstp, exp); + int flags = nfsexp_flags(cred, exp); /* discard any old override before preparing the new set */ revert_creds(get_cred(current_real_cred())); @@ -32,10 +32,10 @@ int nfsd_setuser(struct svc_rqst *rqstp, struct svc_export *exp) if (!new) return -ENOMEM; - new->fsuid = rqstp->rq_cred.cr_uid; - new->fsgid = rqstp->rq_cred.cr_gid; + new->fsuid = cred->cr_uid; + new->fsgid = cred->cr_gid; - rqgi = rqstp->rq_cred.cr_group_info; + rqgi = cred->cr_group_info; if (flags & NFSEXP_ALLSQUASH) { new->fsuid = exp->ex_anon_uid; diff --git a/fs/nfsd/auth.h b/fs/nfsd/auth.h index dbd66424f600..8c5031bbbcee 100644 --- a/fs/nfsd/auth.h +++ b/fs/nfsd/auth.h @@ -12,6 +12,6 @@ * Set the current process's fsuid/fsgid etc to those of the NFS * client user */ -int nfsd_setuser(struct svc_rqst *, struct svc_export *); +int nfsd_setuser(struct svc_cred *cred, struct svc_export *exp); #endif /* LINUX_NFSD_AUTH_H */ diff --git a/fs/nfsd/export.h b/fs/nfsd/export.h index 4a48b2ad5606..3174fc05bf02 100644 --- a/fs/nfsd/export.h +++ b/fs/nfsd/export.h @@ -99,7 +99,8 @@ struct svc_expkey { #define EX_NOHIDE(exp) ((exp)->ex_flags & NFSEXP_NOHIDE) #define EX_WGATHER(exp) ((exp)->ex_flags & NFSEXP_GATHERED_WRITES) -int nfsexp_flags(struct svc_rqst *rqstp, struct svc_export *exp); +struct svc_cred; +int nfsexp_flags(struct svc_cred *cred, struct svc_export *exp); __be32 check_xprtsec_policy(struct svc_export *exp, struct svc_rqst *rqstp); __be32 check_security_flavor(struct svc_export *exp, struct svc_rqst *rqstp); __be32 check_nfsd_access(struct svc_export *exp, struct svc_rqst *rqstp); diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index 0ff07d53931f..c3ce0c100446 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -60,12 +60,6 @@ static DEFINE_PER_CPU(unsigned long, nfsd_file_releases); static DEFINE_PER_CPU(unsigned long, nfsd_file_total_age); static DEFINE_PER_CPU(unsigned long, nfsd_file_evictions); -struct nfsd_fcache_disposal { - struct work_struct work; - spinlock_t lock; - struct list_head freeme; -}; - static struct workqueue_struct *nfsd_filecache_wq __read_mostly; static struct kmem_cache *nfsd_file_slab; @@ -323,7 +317,7 @@ nfsd_file_check_writeback(struct nfsd_file *nf) static bool nfsd_file_lru_add(struct nfsd_file *nf) { set_bit(NFSD_FILE_REFERENCED, &nf->nf_flags); - if (list_lru_add(&nfsd_file_lru, &nf->nf_lru)) { + if (list_lru_add_obj(&nfsd_file_lru, &nf->nf_lru)) { trace_nfsd_file_lru_add(nf); return true; } @@ -332,7 +326,7 @@ static bool nfsd_file_lru_add(struct nfsd_file *nf) static bool nfsd_file_lru_remove(struct nfsd_file *nf) { - if (list_lru_del(&nfsd_file_lru, &nf->nf_lru)) { + if (list_lru_del_obj(&nfsd_file_lru, &nf->nf_lru)) { trace_nfsd_file_lru_del(nf); return true; } @@ -417,12 +411,11 @@ nfsd_file_dispose_list_delayed(struct list_head *dispose) struct nfsd_file *nf = list_first_entry(dispose, struct nfsd_file, nf_gc); struct nfsd_net *nn = net_generic(nf->nf_net, nfsd_net_id); - struct nfsd_fcache_disposal *l = nn->fcache_disposal; - spin_lock(&l->lock); - list_move_tail(&nf->nf_gc, &l->freeme); - spin_unlock(&l->lock); - queue_work(nfsd_filecache_wq, &l->work); + spin_lock(&nn->fcache_dispose_lock); + list_move_tail(&nf->nf_gc, &nn->fcache_dispose_list); + spin_unlock(&nn->fcache_dispose_lock); + queue_work(nfsd_filecache_wq, &nn->fcache_dispose_work); } } @@ -651,12 +644,12 @@ static void nfsd_file_delayed_close(struct work_struct *work) { LIST_HEAD(head); - struct nfsd_fcache_disposal *l = container_of(work, - struct nfsd_fcache_disposal, work); + struct nfsd_net *nn = container_of(work, struct nfsd_net, + fcache_dispose_work); - spin_lock(&l->lock); - list_splice_init(&l->freeme, &head); - spin_unlock(&l->lock); + spin_lock(&nn->fcache_dispose_lock); + list_splice_init(&nn->fcache_dispose_list, &head); + spin_unlock(&nn->fcache_dispose_lock); nfsd_file_dispose_list(&head); } @@ -825,44 +818,22 @@ __nfsd_file_cache_purge(struct net *net) nfsd_file_dispose_list(&dispose); } -static struct nfsd_fcache_disposal * -nfsd_alloc_fcache_disposal(void) -{ - struct nfsd_fcache_disposal *l; - - l = kmalloc(sizeof(*l), GFP_KERNEL); - if (!l) - return NULL; - INIT_WORK(&l->work, nfsd_file_delayed_close); - spin_lock_init(&l->lock); - INIT_LIST_HEAD(&l->freeme); - return l; -} - -static void -nfsd_free_fcache_disposal(struct nfsd_fcache_disposal *l) -{ - cancel_work_sync(&l->work); - nfsd_file_dispose_list(&l->freeme); - kfree(l); -} - -static void -nfsd_free_fcache_disposal_net(struct net *net) -{ - struct nfsd_net *nn = net_generic(net, nfsd_net_id); - struct nfsd_fcache_disposal *l = nn->fcache_disposal; - - nfsd_free_fcache_disposal(l); -} - int nfsd_file_cache_start_net(struct net *net) { struct nfsd_net *nn = net_generic(net, nfsd_net_id); - nn->fcache_disposal = nfsd_alloc_fcache_disposal(); - return nn->fcache_disposal ? 0 : -ENOMEM; + /* + * The dispose state lives as long as the net namespace and may still + * be reached by the shrinker after a previous shutdown, so only + * initialize it once. + */ + if (!nn->fcache_dispose_work.func) { + INIT_WORK(&nn->fcache_dispose_work, nfsd_file_delayed_close); + spin_lock_init(&nn->fcache_dispose_lock); + INIT_LIST_HEAD(&nn->fcache_dispose_list); + } + return 0; } /** @@ -881,8 +852,11 @@ nfsd_file_cache_purge(struct net *net) void nfsd_file_cache_shutdown_net(struct net *net) { + struct nfsd_net *nn = net_generic(net, nfsd_net_id); + nfsd_file_cache_purge(net); - nfsd_free_fcache_disposal_net(net); + cancel_work_sync(&nn->fcache_dispose_work); + nfsd_file_dispose_list(&nn->fcache_dispose_list); } void diff --git a/fs/nfsd/lockd.c b/fs/nfsd/lockd.c index b02886f38925..35b2a0541dc4 100644 --- a/fs/nfsd/lockd.c +++ b/fs/nfsd/lockd.c @@ -79,6 +79,7 @@ nlm_fclose(struct file *filp) } static const struct nlmsvc_binding nfsd_nlm_ops = { + .owner = THIS_MODULE, .fopen = nlm_fopen, /* open file for locking */ .fclose = nlm_fclose, /* close file */ }; @@ -87,11 +88,12 @@ void nfsd_lockd_init(void) { dprintk("nfsd: initializing lockd\n"); - nlmsvc_ops = &nfsd_nlm_ops; + rcu_assign_pointer(nlmsvc_ops, &nfsd_nlm_ops); } void nfsd_lockd_shutdown(void) { - nlmsvc_ops = NULL; + RCU_INIT_POINTER(nlmsvc_ops, NULL); + synchronize_rcu(); } diff --git a/fs/nfsd/netns.h b/fs/nfsd/netns.h index 0ecc5527a8f1..798f66da0536 100644 --- a/fs/nfsd/netns.h +++ b/fs/nfsd/netns.h @@ -26,6 +26,16 @@ struct cld_net; struct nfsd4_client_tracking_ops; +enum nfsd_net_flag { + NFSD_NET_GRACE_ENDED, + NFSD_NET_GRACE_END_FORCED, + NFSD_NET_IN_GRACE, + NFSD_NET_SOMEBODY_RECLAIMED, + NFSD_NET_TRACK_RECLAIM_COMPLETES, + NFSD_NET_UP, + NFSD_NET_LOCKD_UP, +}; + enum { /* cache misses due only to checksum comparison failures */ NFSD_STATS_PAYLOAD_MISSES, @@ -64,10 +74,10 @@ struct nfsd_net { struct cache_detail *nametoid_cache; struct lock_manager nfsd4_manager; - bool grace_ended; - bool grace_end_forced; + unsigned long flags; bool client_tracking_active; time64_t boot_time; + time64_t boot_time_bt; /* same instant in CLOCK_BOOTTIME */ struct dentry *nfsd_client_dir; @@ -113,19 +123,13 @@ struct nfsd_net { spinlock_t blocked_locks_lock; struct file *rec_file; - bool in_grace; const struct nfsd4_client_tracking_ops *client_tracking_ops; time64_t nfsd4_lease; time64_t nfsd4_grace; - bool somebody_reclaimed; - bool track_reclaim_completes; atomic_t nr_reclaim_complete; - bool nfsd_net_up; - bool lockd_up; - seqlock_t writeverf_lock; unsigned char writeverf[8]; @@ -207,7 +211,9 @@ struct nfsd_net { /* utsname taken from the process that starts the server */ char nfsd_name[UNX_MAXNODENAME+1]; - struct nfsd_fcache_disposal *fcache_disposal; + spinlock_t fcache_dispose_lock; + struct list_head fcache_dispose_list; + struct work_struct fcache_dispose_work; siphash_key_t siphash_key; diff --git a/fs/nfsd/nfs3proc.c b/fs/nfsd/nfs3proc.c index f4ccbb1f49ba..13ec77350eb9 100644 --- a/fs/nfsd/nfs3proc.c +++ b/fs/nfsd/nfs3proc.c @@ -28,6 +28,25 @@ static int nfs3_ftypes[] = { S_IFIFO, /* NF3FIFO */ }; +/* + * Reject a client-supplied atime or mtime whose nanoseconds field is out + * of range. Such a value is well-formed on the wire but is not a valid + * timespec64, and storing it verbatim can corrupt on-disk timestamps. + * tv_nsec is a long, so it is cast to unsigned long (the same width) to + * catch both an over-large value and one that became negative when an + * out-of-range u32 wire nseconds was assigned to a 32-bit long. + */ +static bool nfsd3_time_in_range(const struct iattr *iap) +{ + if ((iap->ia_valid & ATTR_ATIME_SET) && + (unsigned long)iap->ia_atime.tv_nsec >= NSEC_PER_SEC) + return false; + if ((iap->ia_valid & ATTR_MTIME_SET) && + (unsigned long)iap->ia_mtime.tv_nsec >= NSEC_PER_SEC) + return false; + return true; +} + /* * NULL call. */ @@ -77,9 +96,14 @@ nfsd3_proc_setattr(struct svc_rqst *rqstp) SVCFH_fmt(&argp->fh)); fh_copy(&resp->fh, &argp->fh); + if (!nfsd3_time_in_range(&argp->attrs)) { + resp->status = nfserr_inval; + goto out; + } if (argp->check_guard) guardtime = &argp->guardtime; resp->status = nfsd_setattr(rqstp, &resp->fh, &attrs, guardtime); +out: return rpc_success; } @@ -235,6 +259,8 @@ nfsd3_create_file(struct svc_rqst *rqstp, struct svc_fh *fhp, __be32 status; int host_err; + if (!nfsd3_time_in_range(iap)) + return nfserr_inval; if (isdotent(argp->name, argp->len)) return nfserr_exist; if (!(iap->ia_valid & ATTR_MODE)) @@ -381,8 +407,13 @@ nfsd3_proc_mkdir(struct svc_rqst *rqstp) argp->attrs.ia_valid &= ~ATTR_SIZE; fh_copy(&resp->dirfh, &argp->fh); fh_init(&resp->fh, NFS3_FHSIZE); + if (!nfsd3_time_in_range(&argp->attrs)) { + resp->status = nfserr_inval; + goto out; + } resp->status = nfsd_create(rqstp, &resp->dirfh, argp->name, argp->len, &attrs, S_IFDIR, 0, &resp->fh); +out: return rpc_success; } @@ -395,6 +426,10 @@ nfsd3_proc_symlink(struct svc_rqst *rqstp) .na_iattr = &argp->attrs, }; + if (!nfsd3_time_in_range(&argp->attrs)) { + resp->status = nfserr_inval; + goto out; + } if (argp->tlen == 0) { resp->status = nfserr_inval; goto out; @@ -460,6 +495,11 @@ nfsd3_proc_mknod(struct svc_rqst *rqstp) goto out; } + if (!nfsd3_time_in_range(&argp->attrs)) { + resp->status = nfserr_inval; + goto out; + } + type = nfs3_ftypes[argp->ftype]; resp->status = nfsd_create(rqstp, &resp->dirfh, argp->name, argp->len, &attrs, type, rdev, &resp->fh); diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c index 84bb200e24ad..bea7513a0de9 100644 --- a/fs/nfsd/nfs4layouts.c +++ b/fs/nfsd/nfs4layouts.c @@ -235,11 +235,17 @@ nfsd4_alloc_layout_stateid(struct nfsd4_compound_state *cstate, nfsd4_init_cb(&ls->ls_recall, clp, &nfsd4_cb_layout_ops, NFSPROC4_CLNT_CB_LAYOUT); - if (parent->sc_type == NFS4_DELEG_STID) + if (parent->sc_type == SC_TYPE_DELEG) { + spin_lock(&fp->fi_lock); ls->ls_file = nfsd_file_get(fp->fi_deleg_file); - else + spin_unlock(&fp->fi_lock); + } else { ls->ls_file = find_any_file(fp); - BUG_ON(!ls->ls_file); + } + if (!ls->ls_file) { + nfs4_put_stid(stp); + return NULL; + } if (nfsd4_layout_setlease(ls)) { nfs4_put_stid(stp); @@ -247,7 +253,7 @@ nfsd4_alloc_layout_stateid(struct nfsd4_compound_state *cstate, } spin_lock(&clp->cl_lock); - stp->sc_type = NFS4_LAYOUT_STID; + stp->sc_type = SC_TYPE_LAYOUT; list_add(&ls->ls_perclnt, &clp->cl_lo_states); spin_unlock(&clp->cl_lock); @@ -266,13 +272,13 @@ nfsd4_preprocess_layout_stateid(struct svc_rqst *rqstp, { struct nfs4_layout_stateid *ls; struct nfs4_stid *stid; - unsigned char typemask = NFS4_LAYOUT_STID; + unsigned short typemask = SC_TYPE_LAYOUT; __be32 status; if (create) - typemask |= (NFS4_OPEN_STID | NFS4_LOCK_STID | NFS4_DELEG_STID); + typemask |= (SC_TYPE_OPEN | SC_TYPE_LOCK | SC_TYPE_DELEG); - status = nfsd4_lookup_stateid(cstate, stateid, typemask, &stid, + status = nfsd4_lookup_stateid(cstate, stateid, typemask, 0, &stid, net_generic(SVC_NET(rqstp), nfsd_net_id)); if (status) goto out; @@ -283,7 +289,7 @@ nfsd4_preprocess_layout_stateid(struct svc_rqst *rqstp, goto out_put_stid; } - if (stid->sc_type != NFS4_LAYOUT_STID) { + if (stid->sc_type != SC_TYPE_LAYOUT) { ls = nfsd4_alloc_layout_stateid(cstate, stid, layout_type); nfs4_put_stid(stid); @@ -512,11 +518,11 @@ nfsd4_return_file_layouts(struct svc_rqst *rqstp, if (!list_empty(&ls->ls_layouts)) { if (found) nfs4_inc_and_copy_stateid(&lrp->lr_sid, &ls->ls_stid); - lrp->lrs_present = 1; + lrp->lrs_present = true; } else { trace_nfsd_layoutstate_unhash(&ls->ls_stid.sc_stateid); - nfs4_unhash_stid(&ls->ls_stid); - lrp->lrs_present = 0; + ls->ls_stid.sc_status |= SC_STATUS_CLOSED; + lrp->lrs_present = false; } spin_unlock(&ls->ls_lock); @@ -536,7 +542,7 @@ nfsd4_return_client_layouts(struct svc_rqst *rqstp, struct nfs4_layout *lp, *t; LIST_HEAD(reaplist); - lrp->lrs_present = 0; + lrp->lrs_present = false; spin_lock(&clp->cl_lock); list_for_each_entry_safe(ls, n, &clp->cl_lo_states, ls_perclnt) { diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c index c82d96089da5..ed5ee128864b 100644 --- a/fs/nfsd/nfs4proc.c +++ b/fs/nfsd/nfs4proc.c @@ -626,7 +626,7 @@ nfsd4_open(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, pr_warn("nfsd4_process_open2 failed to open newly-created file: status=%u\n", be32_to_cpu(status)); if (reclaim && !status) - nn->somebody_reclaimed = true; + set_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags); out: if (open->op_filp) { fput(open->op_filp); @@ -795,7 +795,23 @@ nfsd4_create(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, if (status) return status; + /* Sanitize cr_type to avoid returning ATTRNOTSUPP. */ + switch (create->cr_type) { + case NF4LNK: + case NF4BLK: + case NF4CHR: + case NF4SOCK: + case NF4FIFO: + case NF4DIR: + break; + default: + status = nfserr_badtype; + goto out_aftermask; + } + status = nfsd4_acl_to_attr(create->cr_type, create->cr_acl, &attrs); + if (status != nfs_ok) + goto out_aftermask; current->fs->umask = create->cr_umask; switch (create->cr_type) { case NF4LNK: @@ -862,6 +878,7 @@ nfsd4_create(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, fh_put(&resfh); out_umask: current->fs->umask = 0; +out_aftermask: nfsd_attrs_free(&attrs); return status; } @@ -1810,12 +1827,14 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, goto out; } status = nfsd4_setup_inter_ssc(rqstp, cstate, copy); - if (status) - return nfserr_offload_denied; + if (status) { + status = nfserr_offload_denied; + goto out; + } } else { status = nfsd4_setup_intra_ssc(rqstp, cstate, copy); if (status) - return status; + goto out; } memcpy(©->fh, &cstate->current_fh.fh_handle, @@ -1834,11 +1853,14 @@ nfsd4_copy(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, async_copy->cp_src = kmalloc(sizeof(*async_copy->cp_src), GFP_KERNEL); if (!async_copy->cp_src) goto out_err; - if (!nfs4_init_copy_state(nn, copy)) + dup_copy_fields(copy, async_copy); + + if (!nfs4_init_copy_state(nn, async_copy)) goto out_err; - memcpy(&result->cb_stateid, ©->cp_stateid.cs_stid, + memcpy(&result->cb_stateid, &async_copy->cp_stateid.cs_stid, + sizeof(result->cb_stateid)); + memcpy(&async_copy->cp_res.cb_stateid, &result->cb_stateid, sizeof(result->cb_stateid)); - dup_copy_fields(copy, async_copy); async_copy->copy_task = kthread_create(nfsd4_do_async_copy, async_copy, "%s", "copy thread"); if (IS_ERR(async_copy->copy_task)) diff --git a/fs/nfsd/nfs4recover.c b/fs/nfsd/nfs4recover.c index f4060d0cec8a..12f5c60c400e 100644 --- a/fs/nfsd/nfs4recover.c +++ b/fs/nfsd/nfs4recover.c @@ -228,7 +228,7 @@ nfsd4_create_clid_dir(struct nfs4_client *clp) out_unlock: inode_unlock(d_inode(dir)); if (status == 0) { - if (nn->in_grace) + if (test_bit(NFSD_NET_IN_GRACE, &nn->flags)) __nfsd4_create_reclaim_record_grace(clp, dname, nn); vfs_fsync(nn->rec_file, 0); } else { @@ -388,7 +388,7 @@ nfsd4_remove_clid_dir(struct nfs4_client *clp) nfs4_reset_creds(original_cred); if (status == 0) { vfs_fsync(nn->rec_file, 0); - if (nn->in_grace) + if (test_bit(NFSD_NET_IN_GRACE, &nn->flags)) __nfsd4_remove_reclaim_record_grace(dname, HEXDIR_LEN, nn); } @@ -446,7 +446,7 @@ nfsd4_recdir_purge_old(struct nfsd_net *nn) { int status; - nn->in_grace = false; + clear_bit(NFSD_NET_IN_GRACE, &nn->flags); if (!nn->rec_file) return; status = mnt_want_write_file(nn->rec_file); @@ -528,7 +528,7 @@ nfsd4_init_recdir(struct net *net) nfs4_reset_creds(original_cred); if (!status) - nn->in_grace = true; + set_bit(NFSD_NET_IN_GRACE, &nn->flags); return status; } @@ -1476,7 +1476,7 @@ nfs4_cld_state_init(struct net *net) for (i = 0; i < CLIENT_HASH_SIZE; i++) INIT_LIST_HEAD(&nn->reclaim_str_hashtbl[i]); nn->reclaim_str_hashtbl_size = 0; - nn->track_reclaim_completes = true; + set_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags); atomic_set(&nn->nr_reclaim_complete, 0); return 0; @@ -1487,7 +1487,7 @@ nfs4_cld_state_shutdown(struct net *net) { struct nfsd_net *nn = net_generic(net, nfsd_net_id); - nn->track_reclaim_completes = false; + clear_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags); kfree(nn->reclaim_str_hashtbl); } diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index b3dadc6bfdf0..fb2a6b4affca 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -329,6 +329,16 @@ free_blocked_lock(struct nfsd4_blocked_lock *nbl) kref_put(&nbl->nbl_kref, free_nbl); } +/* A blocked lock's fl_owner is its nfs4_lockowner. */ +static struct nfs4_client * +nbl_client(struct nfsd4_blocked_lock *nbl) +{ + struct nfs4_lockowner *lo; + + lo = (struct nfs4_lockowner *)nbl->nbl_lock.fl_owner; + return lo->lo_owner.so_client; +} + static void remove_blocked_locks(struct nfs4_lockowner *lo) { @@ -1336,11 +1346,6 @@ static void destroy_unhashed_deleg(struct nfs4_delegation *dp) nfs4_put_stid(&dp->dl_stid); } -void nfs4_unhash_stid(struct nfs4_stid *s) -{ - s->sc_type = 0; -} - /** * nfs4_delegation_exists - Discover if this delegation already exists * @clp: a pointer to the nfs4_client we're granting a delegation to @@ -1392,7 +1397,7 @@ hash_delegation_locked(struct nfs4_delegation *dp, struct nfs4_file *fp) if (nfs4_delegation_exists(clp, fp)) return -EAGAIN; refcount_inc(&dp->dl_stid.sc_count); - dp->dl_stid.sc_type = NFS4_DELEG_STID; + dp->dl_stid.sc_type = SC_TYPE_DELEG; list_add(&dp->dl_perfile, &fp->fi_delegations); list_add(&dp->dl_perclnt, &clp->cl_delegations); return 0; @@ -1413,7 +1418,7 @@ unhash_delegation_locked(struct nfs4_delegation *dp) if (!delegation_hashed(dp)) return false; - dp->dl_stid.sc_type = NFS4_CLOSED_DELEG_STID; + dp->dl_stid.sc_status |= SC_STATUS_CLOSED; /* Ensure that deleg break won't try to requeue it */ ++dp->dl_time; spin_lock(&fp->fi_lock); @@ -1445,7 +1450,7 @@ static void revoke_delegation(struct nfs4_delegation *dp) if (clp->cl_minorversion) { spin_lock(&clp->cl_lock); - dp->dl_stid.sc_type = NFS4_REVOKED_DELEG_STID; + dp->dl_stid.sc_status = SC_STATUS_REVOKED; refcount_inc(&dp->dl_stid.sc_count); list_add(&dp->dl_recall_lru, &clp->cl_revoked); spin_unlock(&clp->cl_lock); @@ -1453,8 +1458,8 @@ static void revoke_delegation(struct nfs4_delegation *dp) destroy_unhashed_deleg(dp); } -/* - * SETCLIENTID state +/* + * SETCLIENTID state */ static unsigned int clientid_hashval(u32 id) @@ -1618,7 +1623,7 @@ static bool unhash_lock_stateid(struct nfs4_ol_stateid *stp) if (!unhash_ol_stateid(stp)) return false; list_del_init(&stp->st_locks); - nfs4_unhash_stid(&stp->st_stid); + stp->st_stid.sc_status |= SC_STATUS_CLOSED; return true; } @@ -1697,6 +1702,7 @@ static void release_open_stateid(struct nfs4_ol_stateid *stp) LIST_HEAD(reaplist); spin_lock(&stp->st_stid.sc_client->cl_lock); + stp->st_stid.sc_status |= SC_STATUS_CLOSED; if (unhash_open_stateid(stp, &reaplist)) put_ol_stateid_locked(stp, &reaplist); spin_unlock(&stp->st_stid.sc_client->cl_lock); @@ -2376,7 +2382,7 @@ static void inc_reclaim_complete(struct nfs4_client *clp) { struct nfsd_net *nn = net_generic(clp->net, nfsd_net_id); - if (!nn->track_reclaim_completes) + if (!test_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags)) return; if (!nfsd4_find_reclaim_client(clp->cl_name, nn)) return; @@ -2557,14 +2563,16 @@ find_stateid_locked(struct nfs4_client *cl, stateid_t *t) } static struct nfs4_stid * -find_stateid_by_type(struct nfs4_client *cl, stateid_t *t, char typemask) +find_stateid_by_type(struct nfs4_client *cl, stateid_t *t, + unsigned short typemask, unsigned short ok_states) { struct nfs4_stid *s; spin_lock(&cl->cl_lock); s = find_stateid_locked(cl, t); if (s != NULL) { - if (typemask & s->sc_type) + if ((s->sc_status & ~ok_states) == 0 && + (typemask & s->sc_type)) refcount_inc(&s->sc_count); else s = NULL; @@ -2717,7 +2725,7 @@ static int nfs4_show_open(struct seq_file *s, struct nfs4_stid *st) struct nfs4_stateowner *oo; unsigned int access, deny; - if (st->sc_type != NFS4_OPEN_STID && st->sc_type != NFS4_LOCK_STID) + if (st->sc_type != SC_TYPE_OPEN && st->sc_type != SC_TYPE_LOCK) return 0; /* XXX: or SEQ_SKIP? */ ols = openlockstateid(st); oo = ols->st_stateowner; @@ -2849,14 +2857,17 @@ static int states_show(struct seq_file *s, void *v) { struct nfs4_stid *st = v; + if (st->sc_status) + return 0; + switch (st->sc_type) { - case NFS4_OPEN_STID: + case SC_TYPE_OPEN: return nfs4_show_open(s, st); - case NFS4_LOCK_STID: + case SC_TYPE_LOCK: return nfs4_show_lock(s, st); - case NFS4_DELEG_STID: + case SC_TYPE_DELEG: return nfs4_show_deleg(s, st); - case NFS4_LAYOUT_STID: + case SC_TYPE_LAYOUT: return nfs4_show_layout(s, st); default: return 0; /* XXX: or SEQ_SKIP? */ @@ -4533,8 +4544,6 @@ nfsd4_init_leases_net(struct nfsd_net *nn) nn->nfsd4_lease = 90; /* default lease time */ nn->nfsd4_grace = 90; - nn->somebody_reclaimed = false; - nn->track_reclaim_completes = false; nn->clverifier_counter = get_random_u32(); nn->clientid_base = get_random_u32(); nn->clientid_counter = nn->clientid_base + 1; @@ -4638,7 +4647,8 @@ nfsd4_find_existing_open(struct nfs4_file *fp, struct nfsd4_open *open) continue; if (local->st_stateowner != &oo->oo_owner) continue; - if (local->st_stid.sc_type == NFS4_OPEN_STID) { + if (local->st_stid.sc_type == SC_TYPE_OPEN && + !local->st_stid.sc_status) { ret = local; refcount_inc(&ret->st_stid.sc_count); break; @@ -4652,17 +4662,10 @@ nfsd4_verify_open_stid(struct nfs4_stid *s) { __be32 ret = nfs_ok; - switch (s->sc_type) { - default: - break; - case 0: - case NFS4_CLOSED_STID: - case NFS4_CLOSED_DELEG_STID: - ret = nfserr_bad_stateid; - break; - case NFS4_REVOKED_DELEG_STID: + if (s->sc_status & SC_STATUS_REVOKED) ret = nfserr_deleg_revoked; - } + else if (s->sc_status & SC_STATUS_CLOSED) + ret = nfserr_bad_stateid; return ret; } @@ -4754,7 +4757,7 @@ init_open_stateid(struct nfs4_file *fp, struct nfsd4_open *open) open->op_stp = NULL; refcount_inc(&stp->st_stid.sc_count); - stp->st_stid.sc_type = NFS4_OPEN_STID; + stp->st_stid.sc_type = SC_TYPE_OPEN; INIT_LIST_HEAD(&stp->st_locks); stp->st_stateowner = nfs4_get_stateowner(&oo->oo_owner); get_nfs4_file(fp); @@ -4981,9 +4984,9 @@ static int nfsd4_cb_recall_done(struct nfsd4_callback *cb, trace_nfsd_cb_recall_done(&dp->dl_stid.sc_stateid, task); - if (dp->dl_stid.sc_type == NFS4_CLOSED_DELEG_STID || - dp->dl_stid.sc_type == NFS4_REVOKED_DELEG_STID) - return 1; + if (dp->dl_stid.sc_status) + /* CLOSED or REVOKED */ + return 1; switch (task->tk_status) { case 0: @@ -5230,12 +5233,12 @@ static int share_access_to_flags(u32 share_access) return share_access == NFS4_SHARE_ACCESS_READ ? RD_STATE : WR_STATE; } -static struct nfs4_delegation *find_deleg_stateid(struct nfs4_client *cl, stateid_t *s) +static struct nfs4_delegation *find_deleg_stateid(struct nfs4_client *cl, + stateid_t *s) { struct nfs4_stid *ret; - ret = find_stateid_by_type(cl, s, - NFS4_DELEG_STID|NFS4_REVOKED_DELEG_STID); + ret = find_stateid_by_type(cl, s, SC_TYPE_DELEG, SC_STATUS_REVOKED); if (!ret) return NULL; return delegstateid(ret); @@ -5258,7 +5261,7 @@ nfs4_check_deleg(struct nfs4_client *cl, struct nfsd4_open *open, deleg = find_deleg_stateid(cl, &open->op_delegate_stateid); if (deleg == NULL) goto out; - if (deleg->dl_stid.sc_type == NFS4_REVOKED_DELEG_STID) { + if (deleg->dl_stid.sc_status & SC_STATUS_REVOKED) { nfs4_put_stid(&deleg->dl_stid); if (cl->cl_minorversion) status = nfserr_deleg_revoked; @@ -5910,7 +5913,6 @@ nfsd4_process_open2(struct svc_rqst *rqstp, struct svc_fh *current_fh, struct nf } else { status = nfs4_get_vfs_file(rqstp, fp, current_fh, stp, open, true); if (status) { - stp->st_stid.sc_type = NFS4_CLOSED_STID; release_open_stateid(stp); mutex_unlock(&stp->st_mutex); goto out; @@ -6008,12 +6010,21 @@ nfsd4_renew(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, static void nfsd4_end_grace(struct nfsd_net *nn) { - /* do nothing if grace period already ended */ - if (nn->grace_ended) + /* + * nfsd4_end_grace() can be entered concurrently from the + * laundromat workqueue and from an nfsd compound thread + * handling RECLAIM_COMPLETE. Without serialization, both + * callers can observe NFSD_NET_GRACE_ENDED clear and proceed + * into nfsd4_record_grace_done(). For tracking ops whose + * grace_done drains reclaim_str_hashtbl, that results in + * list corruption and a double free of every + * nfs4_client_reclaim entry. Use an atomic test-and-set so + * exactly one caller proceeds. + */ + if (test_and_set_bit(NFSD_NET_GRACE_ENDED, &nn->flags)) return; trace_nfsd_grace_complete(nn); - nn->grace_ended = true; /* * If the server goes down again right now, an NFSv4 * client will still be allowed to reclaim after it comes back up, @@ -6055,11 +6066,11 @@ bool nfsd4_force_end_grace(struct nfsd_net *nn) if (!nn->client_tracking_ops) return false; spin_lock(&nn->client_lock); - if (nn->grace_ended || !nn->client_tracking_active) { + if (test_bit(NFSD_NET_GRACE_ENDED, &nn->flags) || !nn->client_tracking_active) { spin_unlock(&nn->client_lock); return false; } - WRITE_ONCE(nn->grace_end_forced, true); + set_bit(NFSD_NET_GRACE_END_FORCED, &nn->flags); mod_delayed_work(laundry_wq, &nn->laundromat_work, 0); spin_unlock(&nn->client_lock); return true; @@ -6071,18 +6082,18 @@ bool nfsd4_force_end_grace(struct nfsd_net *nn) */ static bool clients_still_reclaiming(struct nfsd_net *nn) { - time64_t double_grace_period_end = nn->boot_time + + time64_t double_grace_period_end = nn->boot_time_bt + 2 * nn->nfsd4_lease; - if (READ_ONCE(nn->grace_end_forced)) + if (test_bit(NFSD_NET_GRACE_END_FORCED, &nn->flags)) return false; - if (nn->track_reclaim_completes && + if (test_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags) && atomic_read(&nn->nr_reclaim_complete) == nn->reclaim_str_hashtbl_size) return false; - if (!nn->somebody_reclaimed) + if (!test_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags)) return false; - nn->somebody_reclaimed = false; + clear_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags); /* * If we've given them *two* lease times to reclaim, and they're * still not done, give up: @@ -6312,6 +6323,7 @@ nfs4_laundromat(struct nfsd_net *nn) .new_timeo = nn->nfsd4_lease }; struct nfs4_cpntf_state *cps; + struct nfs4_client *clp; copy_stateid_t *cps_t; int i; @@ -6337,6 +6349,18 @@ nfs4_laundromat(struct nfsd_net *nn) dp = list_entry (pos, struct nfs4_delegation, dl_recall_lru); if (!state_expired(<, dp->dl_time)) break; + clp = dp->dl_stid.sc_client; + spin_lock(&nn->client_lock); + if (is_client_expired(clp)) { + spin_unlock(&nn->client_lock); + continue; + } + /* + * Pin without reviving: get_client_locked() would + * flip a courtesy client back to NFSD4_ACTIVE. + */ + atomic_inc(&clp->cl_rpc_users); + spin_unlock(&nn->client_lock); WARN_ON(!unhash_delegation_locked(dp)); list_add(&dp->dl_recall_lru, &reaplist); } @@ -6344,8 +6368,18 @@ nfs4_laundromat(struct nfsd_net *nn) while (!list_empty(&reaplist)) { dp = list_first_entry(&reaplist, struct nfs4_delegation, dl_recall_lru); + clp = dp->dl_stid.sc_client; list_del_init(&dp->dl_recall_lru); revoke_delegation(dp); + /* + * Unpin without renewing: put_client_renew() would + * renew the reaped client's lease. + */ + if (atomic_dec_and_lock(&clp->cl_rpc_users, &nn->client_lock)) { + if (is_client_expired(clp)) + wake_up_all(&expiry_wq); + spin_unlock(&nn->client_lock); + } } spin_lock(&nn->client_lock); @@ -6355,11 +6389,18 @@ nfs4_laundromat(struct nfsd_net *nn) if (!state_expired(<, oo->oo_time)) break; list_del_init(&oo->oo_close_lru); + clp = oo->oo_owner.so_client; + if (is_client_expired(clp)) + continue; stp = oo->oo_last_closed_stid; oo->oo_last_closed_stid = NULL; + atomic_inc(&clp->cl_rpc_users); spin_unlock(&nn->client_lock); nfs4_put_stid(&stp->st_stid); spin_lock(&nn->client_lock); + if (atomic_dec_and_test(&clp->cl_rpc_users) && + is_client_expired(clp)) + wake_up_all(&expiry_wq); } spin_unlock(&nn->client_lock); @@ -6375,22 +6416,33 @@ nfs4_laundromat(struct nfsd_net *nn) * indefinitely once the lock does become free. */ BUG_ON(!list_empty(&reaplist)); + spin_lock(&nn->client_lock); spin_lock(&nn->blocked_locks_lock); - while (!list_empty(&nn->blocked_locks_lru)) { - nbl = list_first_entry(&nn->blocked_locks_lru, - struct nfsd4_blocked_lock, nbl_lru); + list_for_each_safe(pos, next, &nn->blocked_locks_lru) { + nbl = list_entry(pos, struct nfsd4_blocked_lock, nbl_lru); if (!state_expired(<, nbl->nbl_time)) break; + clp = nbl_client(nbl); + if (is_client_expired(clp)) + continue; + atomic_inc(&clp->cl_rpc_users); list_move(&nbl->nbl_lru, &reaplist); list_del_init(&nbl->nbl_list); } spin_unlock(&nn->blocked_locks_lock); + spin_unlock(&nn->client_lock); while (!list_empty(&reaplist)) { nbl = list_first_entry(&reaplist, struct nfsd4_blocked_lock, nbl_lru); + clp = nbl_client(nbl); list_del_init(&nbl->nbl_lru); free_blocked_lock(nbl); + if (atomic_dec_and_lock(&clp->cl_rpc_users, &nn->client_lock)) { + if (is_client_expired(clp)) + wake_up_all(&expiry_wq); + spin_unlock(&nn->client_lock); + } } #ifdef CONFIG_NFSD_V4_2_INTER_SSC /* service the server-to-server copy delayed unmount list */ @@ -6581,22 +6633,20 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid) status = nfsd4_stid_check_stateid_generation(stateid, s, 1); if (status) goto out_unlock; + status = nfsd4_verify_open_stid(s); + if (status) + goto out_unlock; + switch (s->sc_type) { - case NFS4_DELEG_STID: + case SC_TYPE_DELEG: status = nfs_ok; break; - case NFS4_REVOKED_DELEG_STID: - status = nfserr_deleg_revoked; - break; - case NFS4_OPEN_STID: - case NFS4_LOCK_STID: + case SC_TYPE_OPEN: + case SC_TYPE_LOCK: status = nfsd4_check_openowner_confirmed(openlockstateid(s)); break; default: printk("unknown stateid type %x\n", s->sc_type); - fallthrough; - case NFS4_CLOSED_STID: - case NFS4_CLOSED_DELEG_STID: status = nfserr_bad_stateid; } out_unlock: @@ -6606,7 +6656,8 @@ static __be32 nfsd4_validate_stateid(struct nfs4_client *cl, stateid_t *stateid) __be32 nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, - stateid_t *stateid, unsigned char typemask, + stateid_t *stateid, + unsigned short typemask, unsigned short statusmask, struct nfs4_stid **s, struct nfsd_net *nn) { __be32 status; @@ -6617,10 +6668,13 @@ nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, * only return revoked delegations if explicitly asked. * otherwise we report revoked or bad_stateid status. */ - if (typemask & NFS4_REVOKED_DELEG_STID) + if (statusmask & SC_STATUS_REVOKED) return_revoked = true; - else if (typemask & NFS4_DELEG_STID) - typemask |= NFS4_REVOKED_DELEG_STID; + if (typemask & SC_TYPE_DELEG) + /* Always allow REVOKED for DELEG so we can + * retturn the appropriate error. + */ + statusmask |= SC_STATUS_REVOKED; if (ZERO_STATEID(stateid) || ONE_STATEID(stateid) || CLOSE_STATEID(stateid)) @@ -6633,14 +6687,12 @@ nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, } if (status) return status; - stid = find_stateid_by_type(cstate->clp, stateid, typemask); + stid = find_stateid_by_type(cstate->clp, stateid, typemask, statusmask); if (!stid) return nfserr_bad_stateid; - if ((stid->sc_type == NFS4_REVOKED_DELEG_STID) && !return_revoked) { + if ((stid->sc_status & SC_STATUS_REVOKED) && !return_revoked) { nfs4_put_stid(stid); - if (cstate->minorversion) - return nfserr_deleg_revoked; - return nfserr_bad_stateid; + return nfserr_deleg_revoked; } *s = stid; return nfs_ok; @@ -6651,17 +6703,17 @@ nfs4_find_file(struct nfs4_stid *s, int flags) { struct nfsd_file *ret = NULL; - if (!s) + if (!s || s->sc_status) return NULL; switch (s->sc_type) { - case NFS4_DELEG_STID: + case SC_TYPE_DELEG: spin_lock(&s->sc_file->fi_lock); ret = nfsd_file_get(s->sc_file->fi_deleg_file); spin_unlock(&s->sc_file->fi_lock); break; - case NFS4_OPEN_STID: - case NFS4_LOCK_STID: + case SC_TYPE_OPEN: + case SC_TYPE_LOCK: if (flags & RD_STATE) ret = find_readable_file(s->sc_file); else @@ -6692,7 +6744,8 @@ nfs4_check_file(struct svc_rqst *rqstp, struct svc_fh *fhp, struct nfs4_stid *s, nf = nfs4_find_file(s, flags); if (nf) { - status = nfsd_permission(rqstp, fhp->fh_export, fhp->fh_dentry, + status = nfsd_permission(&rqstp->rq_cred, + fhp->fh_export, fhp->fh_dentry, acc | NFSD_MAY_OWNER_OVERRIDE); if (status) { nfsd_file_put(nf); @@ -6782,7 +6835,8 @@ static __be32 find_cpntf_state(struct nfsd_net *nn, stateid_t *st, goto out; *stid = find_stateid_by_type(found, &cps->cp_p_stateid, - NFS4_DELEG_STID|NFS4_OPEN_STID|NFS4_LOCK_STID); + SC_TYPE_DELEG|SC_TYPE_OPEN|SC_TYPE_LOCK, + 0); if (*stid) status = nfs_ok; else @@ -6839,8 +6893,8 @@ nfs4_preprocess_stateid_op(struct svc_rqst *rqstp, } status = nfsd4_lookup_stateid(cstate, stateid, - NFS4_DELEG_STID|NFS4_OPEN_STID|NFS4_LOCK_STID, - &s, nn); + SC_TYPE_DELEG|SC_TYPE_OPEN|SC_TYPE_LOCK, + 0, &s, nn); if (status == nfserr_bad_stateid) status = find_cpntf_state(nn, stateid, &s); if (status) @@ -6851,16 +6905,13 @@ nfs4_preprocess_stateid_op(struct svc_rqst *rqstp, goto out; switch (s->sc_type) { - case NFS4_DELEG_STID: + case SC_TYPE_DELEG: status = nfs4_check_delegmode(delegstateid(s), flags); break; - case NFS4_OPEN_STID: - case NFS4_LOCK_STID: + case SC_TYPE_OPEN: + case SC_TYPE_LOCK: status = nfs4_check_olstateid(openlockstateid(s), flags); break; - default: - status = nfserr_bad_stateid; - break; } if (status) goto out; @@ -6939,33 +6990,34 @@ nfsd4_free_stateid(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, spin_lock(&cl->cl_lock); s = find_stateid_locked(cl, stateid); - if (!s) + if (!s || s->sc_status & SC_STATUS_CLOSED) goto out_unlock; spin_lock(&s->sc_lock); switch (s->sc_type) { - case NFS4_DELEG_STID: + case SC_TYPE_DELEG: + if (s->sc_status & SC_STATUS_REVOKED) { + spin_unlock(&s->sc_lock); + dp = delegstateid(s); + list_del_init(&dp->dl_recall_lru); + spin_unlock(&cl->cl_lock); + nfs4_put_stid(s); + ret = nfs_ok; + goto out; + } ret = nfserr_locks_held; break; - case NFS4_OPEN_STID: + case SC_TYPE_OPEN: ret = check_stateid_generation(stateid, &s->sc_stateid, 1); if (ret) break; ret = nfserr_locks_held; break; - case NFS4_LOCK_STID: + case SC_TYPE_LOCK: spin_unlock(&s->sc_lock); refcount_inc(&s->sc_count); spin_unlock(&cl->cl_lock); ret = nfsd4_free_lock_stateid(stateid, s); goto out; - case NFS4_REVOKED_DELEG_STID: - spin_unlock(&s->sc_lock); - dp = delegstateid(s); - list_del_init(&dp->dl_recall_lru); - spin_unlock(&cl->cl_lock); - nfs4_put_stid(s); - ret = nfs_ok; - goto out; /* Default falls through and returns nfserr_bad_stateid */ } spin_unlock(&s->sc_lock); @@ -7008,6 +7060,7 @@ static __be32 nfs4_seqid_op_checks(struct nfsd4_compound_state *cstate, stateid_ * @seqid: seqid (provided by client) * @stateid: stateid (provided by client) * @typemask: mask of allowable types for this operation + * @statusmask: mask of allowed states: 0 or STID_CLOSED * @stpp: return pointer for the stateid found * @nn: net namespace for request * @@ -7017,7 +7070,8 @@ static __be32 nfs4_seqid_op_checks(struct nfsd4_compound_state *cstate, stateid_ */ static __be32 nfs4_preprocess_seqid_op(struct nfsd4_compound_state *cstate, u32 seqid, - stateid_t *stateid, char typemask, + stateid_t *stateid, + unsigned short typemask, unsigned short statusmask, struct nfs4_ol_stateid **stpp, struct nfsd_net *nn) { @@ -7028,7 +7082,8 @@ nfs4_preprocess_seqid_op(struct nfsd4_compound_state *cstate, u32 seqid, trace_nfsd_preprocess(seqid, stateid); *stpp = NULL; - status = nfsd4_lookup_stateid(cstate, stateid, typemask, &s, nn); + status = nfsd4_lookup_stateid(cstate, stateid, + typemask, statusmask, &s, nn); if (status) return status; stp = openlockstateid(s); @@ -7050,7 +7105,7 @@ static __be32 nfs4_preprocess_confirmed_seqid_op(struct nfsd4_compound_state *cs struct nfs4_ol_stateid *stp; status = nfs4_preprocess_seqid_op(cstate, seqid, stateid, - NFS4_OPEN_STID, &stp, nn); + SC_TYPE_OPEN, 0, &stp, nn); if (status) return status; oo = openowner(stp->st_stateowner); @@ -7081,8 +7136,8 @@ nfsd4_open_confirm(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, return status; status = nfs4_preprocess_seqid_op(cstate, - oc->oc_seqid, &oc->oc_req_stateid, - NFS4_OPEN_STID, &stp, nn); + oc->oc_seqid, &oc->oc_req_stateid, + SC_TYPE_OPEN, 0, &stp, nn); if (status) goto out; oo = openowner(stp->st_stateowner); @@ -7212,18 +7267,20 @@ nfsd4_close(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, struct net *net = SVC_NET(rqstp); struct nfsd_net *nn = net_generic(net, nfsd_net_id); - dprintk("NFSD: nfsd4_close on file %pd\n", + dprintk("NFSD: nfsd4_close on file %pd\n", cstate->current_fh.fh_dentry); status = nfs4_preprocess_seqid_op(cstate, close->cl_seqid, - &close->cl_stateid, - NFS4_OPEN_STID|NFS4_CLOSED_STID, - &stp, nn); + &close->cl_stateid, + SC_TYPE_OPEN, SC_STATUS_CLOSED, + &stp, nn); nfsd4_bump_seqid(cstate, status); if (status) - goto out; + goto out; - stp->st_stid.sc_type = NFS4_CLOSED_STID; + spin_lock(&stp->st_stid.sc_client->cl_lock); + stp->st_stid.sc_status |= SC_STATUS_CLOSED; + spin_unlock(&stp->st_stid.sc_client->cl_lock); /* * Technically we don't _really_ have to increment or copy it, since @@ -7265,7 +7322,7 @@ nfsd4_delegreturn(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, if ((status = fh_verify(rqstp, &cstate->current_fh, S_IFREG, 0))) return status; - status = nfsd4_lookup_stateid(cstate, stateid, NFS4_DELEG_STID, &s, nn); + status = nfsd4_lookup_stateid(cstate, stateid, SC_TYPE_DELEG, 0, &s, nn); if (status) goto out; dp = delegstateid(s); @@ -7536,7 +7593,7 @@ init_lock_stateid(struct nfs4_ol_stateid *stp, struct nfs4_lockowner *lo, if (retstp) goto out_found; refcount_inc(&stp->st_stid.sc_count); - stp->st_stid.sc_type = NFS4_LOCK_STID; + stp->st_stid.sc_type = SC_TYPE_LOCK; stp->st_stateowner = nfs4_get_stateowner(&lo->lo_owner); get_nfs4_file(fp); stp->st_stid.sc_file = fp; @@ -7721,9 +7778,10 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, &lock_stp, &new); } else { status = nfs4_preprocess_seqid_op(cstate, - lock->lk_old_lock_seqid, - &lock->lk_old_lock_stateid, - NFS4_LOCK_STID, &lock_stp, nn); + lock->lk_old_lock_seqid, + &lock->lk_old_lock_stateid, + SC_TYPE_LOCK, 0, &lock_stp, + nn); } if (status) goto out; @@ -7833,7 +7891,7 @@ nfsd4_lock(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, nfs4_inc_and_copy_stateid(&lock->lk_resp_stateid, &lock_stp->st_stid); status = 0; if (lock->lk_reclaim) - nn->somebody_reclaimed = true; + set_bit(NFSD_NET_SOMEBODY_RECLAIMED, &nn->flags); break; case FILE_LOCK_DEFERRED: kref_put(&nbl->nbl_kref, free_nbl); @@ -8021,8 +8079,8 @@ nfsd4_locku(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, return nfserr_inval; status = nfs4_preprocess_seqid_op(cstate, locku->lu_seqid, - &locku->lu_stateid, NFS4_LOCK_STID, - &stp, nn); + &locku->lu_stateid, SC_TYPE_LOCK, 0, + &stp, nn); if (status) goto out; nf = find_any_file(stp->st_stid.sc_file); @@ -8343,8 +8401,9 @@ static int nfs4_state_create_net(struct net *net) nn->conf_name_tree = RB_ROOT; nn->unconf_name_tree = RB_ROOT; nn->boot_time = ktime_get_real_seconds(); - nn->grace_ended = false; - nn->grace_end_forced = false; + nn->boot_time_bt = ktime_get_boottime_seconds(); + clear_bit(NFSD_NET_GRACE_ENDED, &nn->flags); + clear_bit(NFSD_NET_GRACE_END_FORCED, &nn->flags); nn->client_tracking_active = false; nn->nfsd4_manager.block_opens = true; INIT_LIST_HEAD(&nn->nfsd4_manager.list); @@ -8426,7 +8485,8 @@ nfs4_state_start_net(struct net *net) spin_lock(&nn->client_lock); nn->client_tracking_active = true; spin_unlock(&nn->client_lock); - if (nn->track_reclaim_completes && nn->reclaim_str_hashtbl_size == 0) + if (test_bit(NFSD_NET_TRACK_RECLAIM_COMPLETES, &nn->flags) && + nn->reclaim_str_hashtbl_size == 0) goto skip_grace; printk(KERN_INFO "NFSD: starting %lld-second grace period (net %x)\n", nn->nfsd4_grace, net->ns.inum); diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c index 9fe19a889ae6..f01abd874cf5 100644 --- a/fs/nfsd/nfs4xdr.c +++ b/fs/nfsd/nfs4xdr.c @@ -3730,18 +3730,18 @@ nfsd4_encode_clientid4(struct xdr_stream *xdr, const clientid_t *clientid) return nfs_ok; } +/* This is a frequently-encoded item; open-coded for speed */ static __be32 -nfsd4_encode_stateid(struct xdr_stream *xdr, stateid_t *sid) +nfsd4_encode_stateid4(struct xdr_stream *xdr, const stateid_t *sid) { __be32 *p; - p = xdr_reserve_space(xdr, sizeof(stateid_t)); + p = xdr_reserve_space(xdr, NFS4_STATEID_SIZE); if (!p) return nfserr_resource; *p++ = cpu_to_be32(sid->si_generation); - p = xdr_encode_opaque_fixed(p, &sid->si_opaque, - sizeof(stateid_opaque_t)); - return 0; + memcpy(p, &sid->si_opaque, sizeof(sid->si_opaque)); + return nfs_ok; } static __be32 @@ -3785,7 +3785,8 @@ nfsd4_encode_close(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_close *close = &u->close; struct xdr_stream *xdr = resp->xdr; - return nfsd4_encode_stateid(xdr, &close->cl_stateid); + /* open_stateid */ + return nfsd4_encode_stateid4(xdr, &close->cl_stateid); } @@ -3889,7 +3890,7 @@ nfsd4_encode_lock(struct nfsd4_compoundres *resp, __be32 nfserr, struct xdr_stream *xdr = resp->xdr; if (!nfserr) - nfserr = nfsd4_encode_stateid(xdr, &lock->lk_resp_stateid); + nfserr = nfsd4_encode_stateid4(xdr, &lock->lk_resp_stateid); else if (nfserr == nfserr_denied) nfserr = nfsd4_encode_lock_denied(xdr, &lock->lk_denied); @@ -3915,7 +3916,8 @@ nfsd4_encode_locku(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_locku *locku = &u->locku; struct xdr_stream *xdr = resp->xdr; - return nfsd4_encode_stateid(xdr, &locku->lu_stateid); + /* lock_stateid */ + return nfsd4_encode_stateid4(xdr, &locku->lu_stateid); } @@ -3938,7 +3940,7 @@ nfsd4_encode_open(struct nfsd4_compoundres *resp, __be32 nfserr, struct xdr_stream *xdr = resp->xdr; __be32 *p; - nfserr = nfsd4_encode_stateid(xdr, &open->op_stateid); + nfserr = nfsd4_encode_stateid4(xdr, &open->op_stateid); if (nfserr) return nfserr; nfserr = nfsd4_encode_change_info4(xdr, &open->op_cinfo); @@ -3961,7 +3963,7 @@ nfsd4_encode_open(struct nfsd4_compoundres *resp, __be32 nfserr, case NFS4_OPEN_DELEGATE_NONE: break; case NFS4_OPEN_DELEGATE_READ: - nfserr = nfsd4_encode_stateid(xdr, &open->op_delegate_stateid); + nfserr = nfsd4_encode_stateid4(xdr, &open->op_delegate_stateid); if (nfserr) return nfserr; p = xdr_reserve_space(xdr, 20); @@ -3978,7 +3980,7 @@ nfsd4_encode_open(struct nfsd4_compoundres *resp, __be32 nfserr, *p++ = cpu_to_be32(0); /* XXX: is NULL principal ok? */ break; case NFS4_OPEN_DELEGATE_WRITE: - nfserr = nfsd4_encode_stateid(xdr, &open->op_delegate_stateid); + nfserr = nfsd4_encode_stateid4(xdr, &open->op_delegate_stateid); if (nfserr) return nfserr; @@ -4036,7 +4038,8 @@ nfsd4_encode_open_confirm(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_open_confirm *oc = &u->open_confirm; struct xdr_stream *xdr = resp->xdr; - return nfsd4_encode_stateid(xdr, &oc->oc_resp_stateid); + /* open_stateid */ + return nfsd4_encode_stateid4(xdr, &oc->oc_resp_stateid); } static __be32 @@ -4046,7 +4049,8 @@ nfsd4_encode_open_downgrade(struct nfsd4_compoundres *resp, __be32 nfserr, struct nfsd4_open_downgrade *od = &u->open_downgrade; struct xdr_stream *xdr = resp->xdr; - return nfsd4_encode_stateid(xdr, &od->od_stateid); + /* open_stateid */ + return nfsd4_encode_stateid4(xdr, &od->od_stateid); } /* @@ -4783,15 +4787,15 @@ nfsd4_encode_layoutreturn(struct nfsd4_compoundres *resp, __be32 nfserr, { struct nfsd4_layoutreturn *lrp = &u->layoutreturn; struct xdr_stream *xdr = resp->xdr; - __be32 *p; - p = xdr_reserve_space(xdr, 4); - if (!p) - return nfserr_resource; - *p++ = cpu_to_be32(lrp->lrs_present); + /* lrs_present */ + nfserr = nfsd4_encode_bool(xdr, lrp->lrs_present); + if (nfserr != nfs_ok) + return nfserr; if (lrp->lrs_present) - return nfsd4_encode_stateid(xdr, &lrp->lr_sid); - return 0; + /* lrs_stateid */ + return nfsd4_encode_stateid4(xdr, &lrp->lr_sid); + return nfs_ok; } #endif /* CONFIG_NFSD_PNFS */ @@ -4809,7 +4813,7 @@ nfsd42_encode_write_res(struct nfsd4_compoundres *resp, else { __be32 nfserr; *p++ = cpu_to_be32(1); - nfserr = nfsd4_encode_stateid(resp->xdr, &write->cb_stateid); + nfserr = nfsd4_encode_stateid4(resp->xdr, &write->cb_stateid); if (nfserr) return nfserr; } @@ -4993,7 +4997,7 @@ nfsd4_encode_copy_notify(struct nfsd4_compoundres *resp, __be32 nfserr, *p++ = cpu_to_be32(cn->cpn_nsec); /* cnr_stateid */ - nfserr = nfsd4_encode_stateid(xdr, &cn->cpn_cnr_stateid); + nfserr = nfsd4_encode_stateid4(xdr, &cn->cpn_cnr_stateid); if (nfserr) return nfserr; diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index 9b7f31452551..c7f46f57935d 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -1130,7 +1130,7 @@ static ssize_t write_v4_end_grace(struct file *file, char *buf, size_t size) } return scnprintf(buf, SIMPLE_TRANSACTION_LIMIT, "%c\n", - nn->grace_ended ? 'Y' : 'N'); + test_bit(NFSD_NET_GRACE_ENDED, &nn->flags) ? 'Y' : 'N'); } #endif diff --git a/fs/nfsd/nfsfh.c b/fs/nfsd/nfsfh.c index c4af2ee271cb..cc4dd4ef5cf1 100644 --- a/fs/nfsd/nfsfh.c +++ b/fs/nfsd/nfsfh.c @@ -100,7 +100,7 @@ static bool nfsd_originating_port_ok(struct svc_rqst *rqstp, int flags) static __be32 nfsd_setuser_and_check_port(struct svc_rqst *rqstp, struct svc_export *exp) { - int flags = nfsexp_flags(rqstp, exp); + int flags = nfsexp_flags(&rqstp->rq_cred, exp); /* Check if the request originated from a secure port. */ if (!nfsd_originating_port_ok(rqstp, flags)) { @@ -111,7 +111,7 @@ static __be32 nfsd_setuser_and_check_port(struct svc_rqst *rqstp, } /* Set user creds for this exportpoint */ - return nfserrno(nfsd_setuser(rqstp, exp)); + return nfserrno(nfsd_setuser(&rqstp->rq_cred, exp)); } static inline __be32 check_pseudo_root(struct svc_rqst *rqstp, @@ -400,7 +400,7 @@ fh_verify(struct svc_rqst *rqstp, struct svc_fh *fhp, umode_t type, int access) skip_pseudoflavor_check: /* Finally, check access permissions. */ - error = nfsd_permission(rqstp, exp, dentry, access); + error = nfsd_permission(&rqstp->rq_cred, exp, dentry, access); out: trace_nfsd_fh_verify_err(rqstp, fhp, type, access, error); if (error == nfserr_stale) diff --git a/fs/nfsd/nfsproc.c b/fs/nfsd/nfsproc.c index aca0391b1544..c09153d4d358 100644 --- a/fs/nfsd/nfsproc.c +++ b/fs/nfsd/nfsproc.c @@ -338,10 +338,11 @@ nfsd_proc_create(struct svc_rqst *rqstp) * echo thing > device-special-file-or-pipe * by doing a CREATE with type==0 */ - resp->status = nfsd_permission(rqstp, - newfhp->fh_export, - newfhp->fh_dentry, - NFSD_MAY_WRITE|NFSD_MAY_LOCAL_ACCESS); + resp->status = nfsd_permission( + &rqstp->rq_cred, + newfhp->fh_export, + newfhp->fh_dentry, + NFSD_MAY_WRITE|NFSD_MAY_LOCAL_ACCESS); if (resp->status && resp->status != nfserr_rofs) goto out_unlock; } diff --git a/fs/nfsd/nfssvc.c b/fs/nfsd/nfssvc.c index 409f36cb21f2..650e2a41c718 100644 --- a/fs/nfsd/nfssvc.c +++ b/fs/nfsd/nfssvc.c @@ -407,7 +407,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred) struct nfsd_net *nn = net_generic(net, nfsd_net_id); int ret; - if (nn->nfsd_net_up) + if (test_bit(NFSD_NET_UP, &nn->flags)) return 0; ret = nfsd_startup_generic(); @@ -417,11 +417,11 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred) if (ret) goto out_socks; - if (nfsd_needs_lockd(nn) && !nn->lockd_up) { + if (nfsd_needs_lockd(nn) && !test_bit(NFSD_NET_LOCKD_UP, &nn->flags)) { ret = lockd_up(net, cred); if (ret) goto out_socks; - nn->lockd_up = true; + set_bit(NFSD_NET_LOCKD_UP, &nn->flags); } ret = nfsd_file_cache_start_net(net); @@ -439,7 +439,7 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred) if (ret) goto out_reply_cache; - nn->nfsd_net_up = true; + set_bit(NFSD_NET_UP, &nn->flags); return 0; out_reply_cache: @@ -447,9 +447,9 @@ static int nfsd_startup_net(struct net *net, const struct cred *cred) out_filecache: nfsd_file_cache_shutdown_net(net); out_lockd: - if (nn->lockd_up) { + if (test_bit(NFSD_NET_LOCKD_UP, &nn->flags)) { lockd_down(net); - nn->lockd_up = false; + clear_bit(NFSD_NET_LOCKD_UP, &nn->flags); } out_socks: nfsd_shutdown_generic(); @@ -463,11 +463,11 @@ static void nfsd_shutdown_net(struct net *net) nfs4_state_shutdown_net(net); nfsd_reply_cache_shutdown(nn); nfsd_file_cache_shutdown_net(net); - if (nn->lockd_up) { + if (test_bit(NFSD_NET_LOCKD_UP, &nn->flags)) { lockd_down(net); - nn->lockd_up = false; + clear_bit(NFSD_NET_LOCKD_UP, &nn->flags); } - nn->nfsd_net_up = false; + clear_bit(NFSD_NET_UP, &nn->flags); nfsd_shutdown_generic(); } @@ -563,7 +563,7 @@ void nfsd_last_thread(struct net *net) * other initialization has been done except the rpcb information. */ svc_rpcb_cleanup(serv, net); - if (!nn->nfsd_net_up) + if (!test_bit(NFSD_NET_UP, &nn->flags)) return; nfsd_shutdown_net(net); diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h index d1450fe7e2b9..55d13442ca8f 100644 --- a/fs/nfsd/state.h +++ b/fs/nfsd/state.h @@ -88,17 +88,33 @@ struct nfsd4_callback_ops { */ struct nfs4_stid { refcount_t sc_count; -#define NFS4_OPEN_STID 1 -#define NFS4_LOCK_STID 2 -#define NFS4_DELEG_STID 4 -/* For an open stateid kept around *only* to process close replays: */ -#define NFS4_CLOSED_STID 8 + + /* A new stateid is added to the cl_stateids idr early before it + * is fully initialised. Its sc_type is then zero. After + * initialisation the sc_type it set under cl_lock, and then + * never changes. + */ +#define SC_TYPE_OPEN BIT(0) +#define SC_TYPE_LOCK BIT(1) +#define SC_TYPE_DELEG BIT(2) +#define SC_TYPE_LAYOUT BIT(3) + unsigned short sc_type; + +/* state_lock protects sc_status for delegation stateids. + * ->cl_lock protects sc_status for open and lock stateids. + * ->st_mutex also protect sc_status for open stateids. + * ->ls_lock protects sc_status for layout stateids. + */ +/* + * For an open stateid kept around *only* to process close replays. + * For deleg stateid, kept in idr until last reference is dropped. + */ +#define SC_STATUS_CLOSED BIT(0) /* For a deleg stateid kept around only to process free_stateid's: */ -#define NFS4_REVOKED_DELEG_STID 16 -#define NFS4_CLOSED_DELEG_STID 32 -#define NFS4_LAYOUT_STID 64 +#define SC_STATUS_REVOKED BIT(1) + unsigned short sc_status; + struct list_head sc_cp_list; - unsigned char sc_type; stateid_t sc_stateid; spinlock_t sc_lock; struct nfs4_client *sc_client; @@ -681,15 +697,15 @@ extern __be32 nfs4_preprocess_stateid_op(struct svc_rqst *rqstp, stateid_t *stateid, int flags, struct nfsd_file **filp, struct nfs4_stid **cstid); __be32 nfsd4_lookup_stateid(struct nfsd4_compound_state *cstate, - stateid_t *stateid, unsigned char typemask, - struct nfs4_stid **s, struct nfsd_net *nn); + stateid_t *stateid, unsigned short typemask, + unsigned short statusmask, + struct nfs4_stid **s, struct nfsd_net *nn); struct nfs4_stid *nfs4_alloc_stid(struct nfs4_client *cl, struct kmem_cache *slab, void (*sc_free)(struct nfs4_stid *)); int nfs4_init_copy_state(struct nfsd_net *nn, struct nfsd4_copy *copy); void nfs4_free_copy_state(struct nfsd4_copy *copy); struct nfs4_cpntf_state *nfs4_alloc_init_cpntf_state(struct nfsd_net *nn, struct nfs4_stid *p_stid); -void nfs4_unhash_stid(struct nfs4_stid *s); void nfs4_put_stid(struct nfs4_stid *s); void nfs4_inc_and_copy_stateid(stateid_t *dst, struct nfs4_stid *stid); void nfs4_remove_reclaim_record(struct nfs4_client_reclaim *, struct nfsd_net *); diff --git a/fs/nfsd/trace.h b/fs/nfsd/trace.h index d7ed49eef591..34e6ae5b8a9d 100644 --- a/fs/nfsd/trace.h +++ b/fs/nfsd/trace.h @@ -201,7 +201,7 @@ TRACE_EVENT(nfsd_fh_verify, TP_ARGS(rqstp, fhp, type, access), TP_STRUCT__entry( __field(unsigned int, netns_ino) - __sockaddr(server, rqstp->rq_xprt->xpt_remotelen) + __sockaddr(server, rqstp->rq_xprt->xpt_locallen) __sockaddr(client, rqstp->rq_xprt->xpt_remotelen) __field(u32, xid) __field(u32, fh_hash) @@ -240,7 +240,7 @@ TRACE_EVENT_CONDITION(nfsd_fh_verify_err, TP_CONDITION(error), TP_STRUCT__entry( __field(unsigned int, netns_ino) - __sockaddr(server, rqstp->rq_xprt->xpt_remotelen) + __sockaddr(server, rqstp->rq_xprt->xpt_locallen) __sockaddr(client, rqstp->rq_xprt->xpt_remotelen) __field(u32, xid) __field(u32, fh_hash) @@ -641,23 +641,17 @@ DEFINE_EVENT(nfsd_stateseqid_class, nfsd_##name, \ DEFINE_STATESEQID_EVENT(preprocess); DEFINE_STATESEQID_EVENT(open_confirm); -TRACE_DEFINE_ENUM(NFS4_OPEN_STID); -TRACE_DEFINE_ENUM(NFS4_LOCK_STID); -TRACE_DEFINE_ENUM(NFS4_DELEG_STID); -TRACE_DEFINE_ENUM(NFS4_CLOSED_STID); -TRACE_DEFINE_ENUM(NFS4_REVOKED_DELEG_STID); -TRACE_DEFINE_ENUM(NFS4_CLOSED_DELEG_STID); -TRACE_DEFINE_ENUM(NFS4_LAYOUT_STID); - #define show_stid_type(x) \ __print_flags(x, "|", \ - { NFS4_OPEN_STID, "OPEN" }, \ - { NFS4_LOCK_STID, "LOCK" }, \ - { NFS4_DELEG_STID, "DELEG" }, \ - { NFS4_CLOSED_STID, "CLOSED" }, \ - { NFS4_REVOKED_DELEG_STID, "REVOKED" }, \ - { NFS4_CLOSED_DELEG_STID, "CLOSED_DELEG" }, \ - { NFS4_LAYOUT_STID, "LAYOUT" }) + { SC_TYPE_OPEN, "OPEN" }, \ + { SC_TYPE_LOCK, "LOCK" }, \ + { SC_TYPE_DELEG, "DELEG" }, \ + { SC_TYPE_LAYOUT, "LAYOUT" }) + +#define show_stid_status(x) \ + __print_flags(x, "|", \ + { SC_STATUS_CLOSED, "CLOSED" }, \ + { SC_STATUS_REVOKED, "REVOKED" }) \ DECLARE_EVENT_CLASS(nfsd_stid_class, TP_PROTO( @@ -666,6 +660,7 @@ DECLARE_EVENT_CLASS(nfsd_stid_class, TP_ARGS(stid), TP_STRUCT__entry( __field(unsigned long, sc_type) + __field(unsigned long, sc_status) __field(int, sc_count) __field(u32, cl_boot) __field(u32, cl_id) @@ -676,16 +671,18 @@ DECLARE_EVENT_CLASS(nfsd_stid_class, const stateid_t *stp = &stid->sc_stateid; __entry->sc_type = stid->sc_type; + __entry->sc_status = stid->sc_status; __entry->sc_count = refcount_read(&stid->sc_count); __entry->cl_boot = stp->si_opaque.so_clid.cl_boot; __entry->cl_id = stp->si_opaque.so_clid.cl_id; __entry->si_id = stp->si_opaque.so_id; __entry->si_generation = stp->si_generation; ), - TP_printk("client %08x:%08x stateid %08x:%08x ref=%d type=%s", + TP_printk("client %08x:%08x stateid %08x:%08x ref=%d type=%s state=%s", __entry->cl_boot, __entry->cl_id, __entry->si_id, __entry->si_generation, - __entry->sc_count, show_stid_type(__entry->sc_type) + __entry->sc_count, show_stid_type(__entry->sc_type), + show_stid_status(__entry->sc_status) ) ); diff --git a/fs/nfsd/vfs.c b/fs/nfsd/vfs.c index 9a5f364954b4..4da93e691e00 100644 --- a/fs/nfsd/vfs.c +++ b/fs/nfsd/vfs.c @@ -417,20 +417,22 @@ nfsd_sanitize_attrs(struct inode *inode, struct iattr *iap) } static __be32 -nfsd_get_write_access(struct svc_rqst *rqstp, struct svc_fh *fhp, - struct iattr *iap) +nfsd_may_truncate(struct svc_rqst *rqstp, struct svc_fh *fhp, + struct iattr *iap) { struct inode *inode = d_inode(fhp->fh_dentry); - if (iap->ia_size < inode->i_size) { - __be32 err; + if (iap->ia_size >= i_size_read(inode)) + return nfs_ok; - err = nfsd_permission(rqstp, fhp->fh_export, fhp->fh_dentry, - NFSD_MAY_TRUNC | NFSD_MAY_OWNER_OVERRIDE); - if (err) - return err; - } - return nfserrno(get_write_access(inode)); + return nfsd_permission(&rqstp->rq_cred, fhp->fh_export, fhp->fh_dentry, + NFSD_MAY_TRUNC | NFSD_MAY_OWNER_OVERRIDE); +} + +static __be32 +nfsd_get_write_access(struct svc_fh *fhp) +{ + return nfserrno(get_write_access(d_inode(fhp->fh_dentry))); } static int __nfsd_setattr(struct dentry *dentry, struct iattr *iap) @@ -547,12 +549,17 @@ nfsd_setattr(struct svc_rqst *rqstp, struct svc_fh *fhp, * setattr call. */ if (size_change) { - err = nfsd_get_write_access(rqstp, fhp, iap); + err = nfsd_get_write_access(fhp); if (err) return err; } inode_lock(inode); + if (size_change) { + err = nfsd_may_truncate(rqstp, fhp, iap); + if (err) + goto out_unlock; + } err = fh_fill_pre_attrs(fhp); if (err) goto out_unlock; @@ -811,7 +818,8 @@ nfsd_access(struct svc_rqst *rqstp, struct svc_fh *fhp, u32 *access, u32 *suppor sresult |= map->access; - err2 = nfsd_permission(rqstp, export, dentry, map->how); + err2 = nfsd_permission(&rqstp->rq_cred, export, + dentry, map->how); switch (err2) { case nfs_ok: result |= map->access; @@ -1455,7 +1463,8 @@ nfsd_create_locked(struct svc_rqst *rqstp, struct svc_fh *fhp, dirp = d_inode(dentry); dchild = dget(resfhp->fh_dentry); - err = nfsd_permission(rqstp, fhp->fh_export, dentry, NFSD_MAY_CREATE); + err = nfsd_permission(&rqstp->rq_cred, fhp->fh_export, dentry, + NFSD_MAY_CREATE); if (err) goto out; @@ -2182,9 +2191,9 @@ nfsd_statfs(struct svc_rqst *rqstp, struct svc_fh *fhp, struct kstatfs *stat, in return err; } -static int exp_rdonly(struct svc_rqst *rqstp, struct svc_export *exp) +static int exp_rdonly(struct svc_cred *cred, struct svc_export *exp) { - return nfsexp_flags(rqstp, exp) & NFSEXP_READONLY; + return nfsexp_flags(cred, exp) & NFSEXP_READONLY; } #ifdef CONFIG_NFSD_V4 @@ -2428,8 +2437,8 @@ nfsd_setxattr(struct svc_rqst *rqstp, struct svc_fh *fhp, char *name, * Check for a user's access permissions to this inode. */ __be32 -nfsd_permission(struct svc_rqst *rqstp, struct svc_export *exp, - struct dentry *dentry, int acc) +nfsd_permission(struct svc_cred *cred, struct svc_export *exp, + struct dentry *dentry, int acc) { struct inode *inode = d_inode(dentry); int err; @@ -2460,7 +2469,7 @@ nfsd_permission(struct svc_rqst *rqstp, struct svc_export *exp, */ if (!(acc & NFSD_MAY_LOCAL_ACCESS)) if (acc & (NFSD_MAY_WRITE | NFSD_MAY_SATTR | NFSD_MAY_TRUNC)) { - if (exp_rdonly(rqstp, exp) || + if (exp_rdonly(cred, exp) || __mnt_is_readonly(exp->ex_path.mnt)) return nfserr_rofs; if (/* (acc & NFSD_MAY_WRITE) && */ IS_IMMUTABLE(inode)) diff --git a/fs/nfsd/vfs.h b/fs/nfsd/vfs.h index bf695d5cc04e..6c41770efe33 100644 --- a/fs/nfsd/vfs.h +++ b/fs/nfsd/vfs.h @@ -153,8 +153,8 @@ __be32 nfsd_readdir(struct svc_rqst *, struct svc_fh *, __be32 nfsd_statfs(struct svc_rqst *, struct svc_fh *, struct kstatfs *, int access); -__be32 nfsd_permission(struct svc_rqst *, struct svc_export *, - struct dentry *, int); +__be32 nfsd_permission(struct svc_cred *cred, struct svc_export *exp, + struct dentry *dentry, int acc); static inline int fh_want_write(struct svc_fh *fh) { diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h index a83a52b0f518..74ba476002b5 100644 --- a/fs/nfsd/xdr4.h +++ b/fs/nfsd/xdr4.h @@ -50,6 +50,117 @@ #define HAS_CSTATE_FLAG(c, f) ((c)->sid_flags & (f)) #define CLEAR_CSTATE_FLAG(c, f) ((c)->sid_flags &= ~(f)) +/** + * nfsd4_encode_bool - Encode an XDR bool type result + * @xdr: target XDR stream + * @val: boolean value to encode + * + * Return values: + * %nfs_ok: @val encoded; @xdr advanced to next position + * %nfserr_resource: stream buffer space exhausted + */ +static __always_inline __be32 +nfsd4_encode_bool(struct xdr_stream *xdr, bool val) +{ + __be32 *p = xdr_reserve_space(xdr, XDR_UNIT); + + if (unlikely(p == NULL)) + return nfserr_resource; + *p = val ? xdr_one : xdr_zero; + return nfs_ok; +} + +/** + * nfsd4_encode_uint32_t - Encode an XDR uint32_t type result + * @xdr: target XDR stream + * @val: integer value to encode + * + * Return values: + * %nfs_ok: @val encoded; @xdr advanced to next position + * %nfserr_resource: stream buffer space exhausted + */ +static __always_inline __be32 +nfsd4_encode_uint32_t(struct xdr_stream *xdr, u32 val) +{ + __be32 *p = xdr_reserve_space(xdr, XDR_UNIT); + + if (unlikely(p == NULL)) + return nfserr_resource; + *p = cpu_to_be32(val); + return nfs_ok; +} + +/** + * nfsd4_encode_uint64_t - Encode an XDR uint64_t type result + * @xdr: target XDR stream + * @val: integer value to encode + * + * Return values: + * %nfs_ok: @val encoded; @xdr advanced to next position + * %nfserr_resource: stream buffer space exhausted + */ +static __always_inline __be32 +nfsd4_encode_uint64_t(struct xdr_stream *xdr, u64 val) +{ + __be32 *p = xdr_reserve_space(xdr, XDR_UNIT * 2); + + if (unlikely(p == NULL)) + return nfserr_resource; + put_unaligned_be64(val, p); + return nfs_ok; +} + +/** + * nfsd4_encode_opaque_fixed - Encode a fixed-length XDR opaque type result + * @xdr: target XDR stream + * @data: pointer to data + * @size: length of data in bytes + * + * Return values: + * %nfs_ok: @data encoded; @xdr advanced to next position + * %nfserr_resource: stream buffer space exhausted + */ +static __always_inline __be32 +nfsd4_encode_opaque_fixed(struct xdr_stream *xdr, const void *data, + size_t size) +{ + __be32 *p = xdr_reserve_space(xdr, xdr_align_size(size)); + size_t pad = xdr_pad_size(size); + + if (unlikely(p == NULL)) + return nfserr_resource; + memcpy(p, data, size); + if (pad) + memset((char *)p + size, 0, pad); + return nfs_ok; +} + +/** + * nfsd4_encode_opaque - Encode a variable-length XDR opaque type result + * @xdr: target XDR stream + * @data: pointer to data + * @size: length of data in bytes + * + * Return values: + * %nfs_ok: @data encoded; @xdr advanced to next position + * %nfserr_resource: stream buffer space exhausted + */ +static __always_inline __be32 +nfsd4_encode_opaque(struct xdr_stream *xdr, const void *data, size_t size) +{ + size_t pad = xdr_pad_size(size); + __be32 *p; + + p = xdr_reserve_space(xdr, XDR_UNIT + xdr_align_size(size)); + if (unlikely(p == NULL)) + return nfserr_resource; + *p++ = cpu_to_be32(size); + memcpy(p, data, size); + if (pad) + memset((char *)p + size, 0, pad); + return nfs_ok; +} + struct nfsd4_compound_state { struct svc_fh current_fh; struct svc_fh save_fh; @@ -542,7 +653,7 @@ struct nfsd4_layoutreturn { u32 lrf_body_len; /* request */ void *lrf_body; /* request */ stateid_t lr_sid; /* request/response */ - u32 lrs_present; /* response */ + bool lrs_present; /* response */ }; struct nfsd4_fallocate { diff --git a/fs/nilfs2/page.c b/fs/nilfs2/page.c index dd40357d836b..2dbf74936a7e 100644 --- a/fs/nilfs2/page.c +++ b/fs/nilfs2/page.c @@ -405,6 +405,11 @@ void nilfs_clear_dirty_pages(struct address_space *mapping, bool silent) * nilfs_clear_dirty_page - discard dirty page * @page: dirty page that will be discarded * @silent: suppress [true] or print [false] warning messages + * + * nilfs_clear_dirty_page() clears working states including dirty state for + * the page and its buffers. If the page has buffers, clear only if it is + * confirmed that none of the buffer heads are busy (none have valid + * references and none are locked). */ void nilfs_clear_dirty_page(struct page *page, bool silent) { @@ -417,19 +422,34 @@ void nilfs_clear_dirty_page(struct page *page, bool silent) nilfs_warn(sb, "discard dirty page: offset=%lld, ino=%lu", page_offset(page), inode->i_ino); - ClearPageUptodate(page); - ClearPageMappedToDisk(page); - ClearPageChecked(page); - if (page_has_buffers(page)) { - struct buffer_head *bh, *head; + struct buffer_head *bh, *head = page_buffers(page); const unsigned long clear_bits = (BIT(BH_Uptodate) | BIT(BH_Dirty) | BIT(BH_Mapped) | BIT(BH_Async_Write) | BIT(BH_NILFS_Volatile) | BIT(BH_NILFS_Checked) | BIT(BH_NILFS_Redirected) | BIT(BH_Delay)); + bool busy, invalidated = false; + +recheck_buffers: + busy = false; + bh = head; + do { + if (atomic_read(&bh->b_count) | buffer_locked(bh)) { + busy = true; + break; + } + } while (bh = bh->b_this_page, bh != head); + + if (busy) { + if (invalidated) + return; + invalidate_bh_lrus(); + invalidated = true; + goto recheck_buffers; + } - bh = head = page_buffers(page); + bh = head; do { lock_buffer(bh); if (!silent) @@ -442,6 +462,9 @@ void nilfs_clear_dirty_page(struct page *page, bool silent) } while (bh = bh->b_this_page, bh != head); } + ClearPageUptodate(page); + ClearPageMappedToDisk(page); + ClearPageChecked(page); __nilfs_clear_page_dirty(page); } diff --git a/fs/notify/mark.c b/fs/notify/mark.c index b419a5ccf192..951d7de29f82 100644 --- a/fs/notify/mark.c +++ b/fs/notify/mark.c @@ -153,7 +153,12 @@ static struct inode *fsnotify_update_iref(struct fsnotify_mark_connector *conn, return inode; } -static void *__fsnotify_recalc_mask(struct fsnotify_mark_connector *conn) +/* + * Calculate mask of events for a list of marks. + * + * Return true if any of the attached marks want to hold an inode reference. + */ +static bool __fsnotify_recalc_mask(struct fsnotify_mark_connector *conn) { u32 new_mask = 0; bool want_iref = false; @@ -173,6 +178,34 @@ static void *__fsnotify_recalc_mask(struct fsnotify_mark_connector *conn) } *fsnotify_conn_mask_p(conn) = new_mask; + return want_iref; +} + +/* + * Calculate mask of events for a list of marks after attach/modify mark + * and get an inode reference for the connector if needed. + * + * A concurrent add of evictable mark and detach of non-evictable mark can + * lead to __fsnotify_recalc_mask() returning false want_iref, but in this + * case we defer clearing iref to fsnotify_recalc_mask_clear_iref() called + * from fsnotify_put_mark(). + */ +static void fsnotify_recalc_mask_set_iref(struct fsnotify_mark_connector *conn) +{ + bool has_iref = conn->flags & FSNOTIFY_CONN_FLAG_HAS_IREF; + bool want_iref = __fsnotify_recalc_mask(conn) || has_iref; + + (void) fsnotify_update_iref(conn, want_iref); +} + +/* + * Calculate mask of events for a list of marks after detach mark + * and return the inode object if its reference is no longer needed. + */ +static void *fsnotify_recalc_mask_clear_iref(struct fsnotify_mark_connector *conn) +{ + bool want_iref = __fsnotify_recalc_mask(conn); + return fsnotify_update_iref(conn, want_iref); } @@ -209,7 +242,7 @@ void fsnotify_recalc_mask(struct fsnotify_mark_connector *conn) spin_lock(&conn->lock); update_children = !fsnotify_conn_watches_children(conn); - __fsnotify_recalc_mask(conn); + fsnotify_recalc_mask_set_iref(conn); update_children &= fsnotify_conn_watches_children(conn); spin_unlock(&conn->lock); /* @@ -342,7 +375,7 @@ void fsnotify_put_mark(struct fsnotify_mark *mark) objp = fsnotify_detach_connector_from_object(conn, &type); free_conn = true; } else { - objp = __fsnotify_recalc_mask(conn); + objp = fsnotify_recalc_mask_clear_iref(conn); type = conn->type; } WRITE_ONCE(mark->connector, NULL); diff --git a/fs/ntfs/Kconfig b/fs/ntfs/Kconfig deleted file mode 100644 index 7b2509741735..000000000000 --- a/fs/ntfs/Kconfig +++ /dev/null @@ -1,81 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0-only -config NTFS_FS - tristate "NTFS file system support" - select BUFFER_HEAD - select NLS - help - NTFS is the file system of Microsoft Windows NT, 2000, XP and 2003. - - Saying Y or M here enables read support. There is partial, but - safe, write support available. For write support you must also - say Y to "NTFS write support" below. - - There are also a number of user-space tools available, called - ntfsprogs. These include ntfsundelete and ntfsresize, that work - without NTFS support enabled in the kernel. - - This is a rewrite from scratch of Linux NTFS support and replaced - the old NTFS code starting with Linux 2.5.11. A backport to - the Linux 2.4 kernel series is separately available as a patch - from the project web site. - - For more information see - and . - - To compile this file system support as a module, choose M here: the - module will be called ntfs. - - If you are not using Windows NT, 2000, XP or 2003 in addition to - Linux on your computer it is safe to say N. - -config NTFS_DEBUG - bool "NTFS debugging support" - depends on NTFS_FS - help - If you are experiencing any problems with the NTFS file system, say - Y here. This will result in additional consistency checks to be - performed by the driver as well as additional debugging messages to - be written to the system log. Note that debugging messages are - disabled by default. To enable them, supply the option debug_msgs=1 - at the kernel command line when booting the kernel or as an option - to insmod when loading the ntfs module. Once the driver is active, - you can enable debugging messages by doing (as root): - echo 1 > /proc/sys/fs/ntfs-debug - Replacing the "1" with "0" would disable debug messages. - - If you leave debugging messages disabled, this results in little - overhead, but enabling debug messages results in very significant - slowdown of the system. - - When reporting bugs, please try to have available a full dump of - debugging messages while the misbehaviour was occurring. - -config NTFS_RW - bool "NTFS write support" - depends on NTFS_FS - depends on PAGE_SIZE_LESS_THAN_64KB - help - This enables the partial, but safe, write support in the NTFS driver. - - The only supported operation is overwriting existing files, without - changing the file length. No file or directory creation, deletion or - renaming is possible. Note only non-resident files can be written to - so you may find that some very small files (<500 bytes or so) cannot - be written to. - - While we cannot guarantee that it will not damage any data, we have - so far not received a single report where the driver would have - damaged someones data so we assume it is perfectly safe to use. - - Note: While write support is safe in this version (a rewrite from - scratch of the NTFS support), it should be noted that the old NTFS - write support, included in Linux 2.5.10 and before (since 1997), - is not safe. - - This is currently useful with TopologiLinux. TopologiLinux is run - on top of any DOS/Microsoft Windows system without partitioning your - hard disk. Unlike other Linux distributions TopologiLinux does not - need its own partition. For more information see - - - It is perfectly safe to say N here. diff --git a/fs/ntfs/Makefile b/fs/ntfs/Makefile deleted file mode 100644 index 3e736572ed00..000000000000 --- a/fs/ntfs/Makefile +++ /dev/null @@ -1,15 +0,0 @@ -# SPDX-License-Identifier: GPL-2.0 -# Rules for making the NTFS driver. - -obj-$(CONFIG_NTFS_FS) += ntfs.o - -ntfs-y := aops.o attrib.o collate.o compress.o debug.o dir.o file.o \ - index.o inode.o mft.o mst.o namei.o runlist.o super.o sysctl.o \ - unistr.o upcase.o - -ntfs-$(CONFIG_NTFS_RW) += bitmap.o lcnalloc.o logfile.o quota.o usnjrnl.o - -ccflags-y := -DNTFS_VERSION=\"2.1.32\" -ccflags-$(CONFIG_NTFS_DEBUG) += -DDEBUG -ccflags-$(CONFIG_NTFS_RW) += -DNTFS_RW - diff --git a/fs/ntfs/aops.c b/fs/ntfs/aops.c deleted file mode 100644 index 4e158bce4192..000000000000 --- a/fs/ntfs/aops.c +++ /dev/null @@ -1,1761 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * aops.c - NTFS kernel address space operations and page cache handling. - * - * Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc. - * Copyright (c) 2002 Richard Russon - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "aops.h" -#include "attrib.h" -#include "debug.h" -#include "inode.h" -#include "mft.h" -#include "runlist.h" -#include "types.h" -#include "ntfs.h" - -/** - * ntfs_end_buffer_async_read - async io completion for reading attributes - * @bh: buffer head on which io is completed - * @uptodate: whether @bh is now uptodate or not - * - * Asynchronous I/O completion handler for reading pages belonging to the - * attribute address space of an inode. The inodes can either be files or - * directories or they can be fake inodes describing some attribute. - * - * If NInoMstProtected(), perform the post read mst fixups when all IO on the - * page has been completed and mark the page uptodate or set the error bit on - * the page. To determine the size of the records that need fixing up, we - * cheat a little bit by setting the index_block_size in ntfs_inode to the ntfs - * record size, and index_block_size_bits, to the log(base 2) of the ntfs - * record size. - */ -static void ntfs_end_buffer_async_read(struct buffer_head *bh, int uptodate) -{ - unsigned long flags; - struct buffer_head *first, *tmp; - struct page *page; - struct inode *vi; - ntfs_inode *ni; - int page_uptodate = 1; - - page = bh->b_page; - vi = page->mapping->host; - ni = NTFS_I(vi); - - if (likely(uptodate)) { - loff_t i_size; - s64 file_ofs, init_size; - - set_buffer_uptodate(bh); - - file_ofs = ((s64)page->index << PAGE_SHIFT) + - bh_offset(bh); - read_lock_irqsave(&ni->size_lock, flags); - init_size = ni->initialized_size; - i_size = i_size_read(vi); - read_unlock_irqrestore(&ni->size_lock, flags); - if (unlikely(init_size > i_size)) { - /* Race with shrinking truncate. */ - init_size = i_size; - } - /* Check for the current buffer head overflowing. */ - if (unlikely(file_ofs + bh->b_size > init_size)) { - int ofs; - void *kaddr; - - ofs = 0; - if (file_ofs < init_size) - ofs = init_size - file_ofs; - kaddr = kmap_atomic(page); - memset(kaddr + bh_offset(bh) + ofs, 0, - bh->b_size - ofs); - flush_dcache_page(page); - kunmap_atomic(kaddr); - } - } else { - clear_buffer_uptodate(bh); - SetPageError(page); - ntfs_error(ni->vol->sb, "Buffer I/O error, logical block " - "0x%llx.", (unsigned long long)bh->b_blocknr); - } - first = page_buffers(page); - spin_lock_irqsave(&first->b_uptodate_lock, flags); - clear_buffer_async_read(bh); - unlock_buffer(bh); - tmp = bh; - do { - if (!buffer_uptodate(tmp)) - page_uptodate = 0; - if (buffer_async_read(tmp)) { - if (likely(buffer_locked(tmp))) - goto still_busy; - /* Async buffers must be locked. */ - BUG(); - } - tmp = tmp->b_this_page; - } while (tmp != bh); - spin_unlock_irqrestore(&first->b_uptodate_lock, flags); - /* - * If none of the buffers had errors then we can set the page uptodate, - * but we first have to perform the post read mst fixups, if the - * attribute is mst protected, i.e. if NInoMstProteced(ni) is true. - * Note we ignore fixup errors as those are detected when - * map_mft_record() is called which gives us per record granularity - * rather than per page granularity. - */ - if (!NInoMstProtected(ni)) { - if (likely(page_uptodate && !PageError(page))) - SetPageUptodate(page); - } else { - u8 *kaddr; - unsigned int i, recs; - u32 rec_size; - - rec_size = ni->itype.index.block_size; - recs = PAGE_SIZE / rec_size; - /* Should have been verified before we got here... */ - BUG_ON(!recs); - kaddr = kmap_atomic(page); - for (i = 0; i < recs; i++) - post_read_mst_fixup((NTFS_RECORD*)(kaddr + - i * rec_size), rec_size); - kunmap_atomic(kaddr); - flush_dcache_page(page); - if (likely(page_uptodate && !PageError(page))) - SetPageUptodate(page); - } - unlock_page(page); - return; -still_busy: - spin_unlock_irqrestore(&first->b_uptodate_lock, flags); - return; -} - -/** - * ntfs_read_block - fill a @page of an address space with data - * @page: page cache page to fill with data - * - * Fill the page @page of the address space belonging to the @page->host inode. - * We read each buffer asynchronously and when all buffers are read in, our io - * completion handler ntfs_end_buffer_read_async(), if required, automatically - * applies the mst fixups to the page before finally marking it uptodate and - * unlocking it. - * - * We only enforce allocated_size limit because i_size is checked for in - * generic_file_read(). - * - * Return 0 on success and -errno on error. - * - * Contains an adapted version of fs/buffer.c::block_read_full_folio(). - */ -static int ntfs_read_block(struct page *page) -{ - loff_t i_size; - VCN vcn; - LCN lcn; - s64 init_size; - struct inode *vi; - ntfs_inode *ni; - ntfs_volume *vol; - runlist_element *rl; - struct buffer_head *bh, *head, *arr[MAX_BUF_PER_PAGE]; - sector_t iblock, lblock, zblock; - unsigned long flags; - unsigned int blocksize, vcn_ofs; - int i, nr; - unsigned char blocksize_bits; - - vi = page->mapping->host; - ni = NTFS_I(vi); - vol = ni->vol; - - /* $MFT/$DATA must have its complete runlist in memory at all times. */ - BUG_ON(!ni->runlist.rl && !ni->mft_no && !NInoAttr(ni)); - - blocksize = vol->sb->s_blocksize; - blocksize_bits = vol->sb->s_blocksize_bits; - - if (!page_has_buffers(page)) { - create_empty_buffers(page, blocksize, 0); - if (unlikely(!page_has_buffers(page))) { - unlock_page(page); - return -ENOMEM; - } - } - bh = head = page_buffers(page); - BUG_ON(!bh); - - /* - * We may be racing with truncate. To avoid some of the problems we - * now take a snapshot of the various sizes and use those for the whole - * of the function. In case of an extending truncate it just means we - * may leave some buffers unmapped which are now allocated. This is - * not a problem since these buffers will just get mapped when a write - * occurs. In case of a shrinking truncate, we will detect this later - * on due to the runlist being incomplete and if the page is being - * fully truncated, truncate will throw it away as soon as we unlock - * it so no need to worry what we do with it. - */ - iblock = (s64)page->index << (PAGE_SHIFT - blocksize_bits); - read_lock_irqsave(&ni->size_lock, flags); - lblock = (ni->allocated_size + blocksize - 1) >> blocksize_bits; - init_size = ni->initialized_size; - i_size = i_size_read(vi); - read_unlock_irqrestore(&ni->size_lock, flags); - if (unlikely(init_size > i_size)) { - /* Race with shrinking truncate. */ - init_size = i_size; - } - zblock = (init_size + blocksize - 1) >> blocksize_bits; - - /* Loop through all the buffers in the page. */ - rl = NULL; - nr = i = 0; - do { - int err = 0; - - if (unlikely(buffer_uptodate(bh))) - continue; - if (unlikely(buffer_mapped(bh))) { - arr[nr++] = bh; - continue; - } - bh->b_bdev = vol->sb->s_bdev; - /* Is the block within the allowed limits? */ - if (iblock < lblock) { - bool is_retry = false; - - /* Convert iblock into corresponding vcn and offset. */ - vcn = (VCN)iblock << blocksize_bits >> - vol->cluster_size_bits; - vcn_ofs = ((VCN)iblock << blocksize_bits) & - vol->cluster_size_mask; - if (!rl) { -lock_retry_remap: - down_read(&ni->runlist.lock); - rl = ni->runlist.rl; - } - if (likely(rl != NULL)) { - /* Seek to element containing target vcn. */ - while (rl->length && rl[1].vcn <= vcn) - rl++; - lcn = ntfs_rl_vcn_to_lcn(rl, vcn); - } else - lcn = LCN_RL_NOT_MAPPED; - /* Successful remap. */ - if (lcn >= 0) { - /* Setup buffer head to correct block. */ - bh->b_blocknr = ((lcn << vol->cluster_size_bits) - + vcn_ofs) >> blocksize_bits; - set_buffer_mapped(bh); - /* Only read initialized data blocks. */ - if (iblock < zblock) { - arr[nr++] = bh; - continue; - } - /* Fully non-initialized data block, zero it. */ - goto handle_zblock; - } - /* It is a hole, need to zero it. */ - if (lcn == LCN_HOLE) - goto handle_hole; - /* If first try and runlist unmapped, map and retry. */ - if (!is_retry && lcn == LCN_RL_NOT_MAPPED) { - is_retry = true; - /* - * Attempt to map runlist, dropping lock for - * the duration. - */ - up_read(&ni->runlist.lock); - err = ntfs_map_runlist(ni, vcn); - if (likely(!err)) - goto lock_retry_remap; - rl = NULL; - } else if (!rl) - up_read(&ni->runlist.lock); - /* - * If buffer is outside the runlist, treat it as a - * hole. This can happen due to concurrent truncate - * for example. - */ - if (err == -ENOENT || lcn == LCN_ENOENT) { - err = 0; - goto handle_hole; - } - /* Hard error, zero out region. */ - if (!err) - err = -EIO; - bh->b_blocknr = -1; - SetPageError(page); - ntfs_error(vol->sb, "Failed to read from inode 0x%lx, " - "attribute type 0x%x, vcn 0x%llx, " - "offset 0x%x because its location on " - "disk could not be determined%s " - "(error code %i).", ni->mft_no, - ni->type, (unsigned long long)vcn, - vcn_ofs, is_retry ? " even after " - "retrying" : "", err); - } - /* - * Either iblock was outside lblock limits or - * ntfs_rl_vcn_to_lcn() returned error. Just zero that portion - * of the page and set the buffer uptodate. - */ -handle_hole: - bh->b_blocknr = -1UL; - clear_buffer_mapped(bh); -handle_zblock: - zero_user(page, i * blocksize, blocksize); - if (likely(!err)) - set_buffer_uptodate(bh); - } while (i++, iblock++, (bh = bh->b_this_page) != head); - - /* Release the lock if we took it. */ - if (rl) - up_read(&ni->runlist.lock); - - /* Check we have at least one buffer ready for i/o. */ - if (nr) { - struct buffer_head *tbh; - - /* Lock the buffers. */ - for (i = 0; i < nr; i++) { - tbh = arr[i]; - lock_buffer(tbh); - tbh->b_end_io = ntfs_end_buffer_async_read; - set_buffer_async_read(tbh); - } - /* Finally, start i/o on the buffers. */ - for (i = 0; i < nr; i++) { - tbh = arr[i]; - if (likely(!buffer_uptodate(tbh))) - submit_bh(REQ_OP_READ, tbh); - else - ntfs_end_buffer_async_read(tbh, 1); - } - return 0; - } - /* No i/o was scheduled on any of the buffers. */ - if (likely(!PageError(page))) - SetPageUptodate(page); - else /* Signal synchronous i/o error. */ - nr = -EIO; - unlock_page(page); - return nr; -} - -/** - * ntfs_read_folio - fill a @folio of a @file with data from the device - * @file: open file to which the folio @folio belongs or NULL - * @folio: page cache folio to fill with data - * - * For non-resident attributes, ntfs_read_folio() fills the @folio of the open - * file @file by calling the ntfs version of the generic block_read_full_folio() - * function, ntfs_read_block(), which in turn creates and reads in the buffers - * associated with the folio asynchronously. - * - * For resident attributes, OTOH, ntfs_read_folio() fills @folio by copying the - * data from the mft record (which at this stage is most likely in memory) and - * fills the remainder with zeroes. Thus, in this case, I/O is synchronous, as - * even if the mft record is not cached at this point in time, we need to wait - * for it to be read in before we can do the copy. - * - * Return 0 on success and -errno on error. - */ -static int ntfs_read_folio(struct file *file, struct folio *folio) -{ - struct page *page = &folio->page; - loff_t i_size; - struct inode *vi; - ntfs_inode *ni, *base_ni; - u8 *addr; - ntfs_attr_search_ctx *ctx; - MFT_RECORD *mrec; - unsigned long flags; - u32 attr_len; - int err = 0; - -retry_readpage: - BUG_ON(!PageLocked(page)); - vi = page->mapping->host; - i_size = i_size_read(vi); - /* Is the page fully outside i_size? (truncate in progress) */ - if (unlikely(page->index >= (i_size + PAGE_SIZE - 1) >> - PAGE_SHIFT)) { - zero_user(page, 0, PAGE_SIZE); - ntfs_debug("Read outside i_size - truncated?"); - goto done; - } - /* - * This can potentially happen because we clear PageUptodate() during - * ntfs_writepage() of MstProtected() attributes. - */ - if (PageUptodate(page)) { - unlock_page(page); - return 0; - } - ni = NTFS_I(vi); - /* - * Only $DATA attributes can be encrypted and only unnamed $DATA - * attributes can be compressed. Index root can have the flags set but - * this means to create compressed/encrypted files, not that the - * attribute is compressed/encrypted. Note we need to check for - * AT_INDEX_ALLOCATION since this is the type of both directory and - * index inodes. - */ - if (ni->type != AT_INDEX_ALLOCATION) { - /* If attribute is encrypted, deny access, just like NT4. */ - if (NInoEncrypted(ni)) { - BUG_ON(ni->type != AT_DATA); - err = -EACCES; - goto err_out; - } - /* Compressed data streams are handled in compress.c. */ - if (NInoNonResident(ni) && NInoCompressed(ni)) { - BUG_ON(ni->type != AT_DATA); - BUG_ON(ni->name_len); - return ntfs_read_compressed_block(page); - } - } - /* NInoNonResident() == NInoIndexAllocPresent() */ - if (NInoNonResident(ni)) { - /* Normal, non-resident data stream. */ - return ntfs_read_block(page); - } - /* - * Attribute is resident, implying it is not compressed or encrypted. - * This also means the attribute is smaller than an mft record and - * hence smaller than a page, so can simply zero out any pages with - * index above 0. Note the attribute can actually be marked compressed - * but if it is resident the actual data is not compressed so we are - * ok to ignore the compressed flag here. - */ - if (unlikely(page->index > 0)) { - zero_user(page, 0, PAGE_SIZE); - goto done; - } - if (!NInoAttr(ni)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - /* Map, pin, and lock the mft record. */ - mrec = map_mft_record(base_ni); - if (IS_ERR(mrec)) { - err = PTR_ERR(mrec); - goto err_out; - } - /* - * If a parallel write made the attribute non-resident, drop the mft - * record and retry the read_folio. - */ - if (unlikely(NInoNonResident(ni))) { - unmap_mft_record(base_ni); - goto retry_readpage; - } - ctx = ntfs_attr_get_search_ctx(base_ni, mrec); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto unm_err_out; - } - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) - goto put_unm_err_out; - attr_len = le32_to_cpu(ctx->attr->data.resident.value_length); - read_lock_irqsave(&ni->size_lock, flags); - if (unlikely(attr_len > ni->initialized_size)) - attr_len = ni->initialized_size; - i_size = i_size_read(vi); - read_unlock_irqrestore(&ni->size_lock, flags); - if (unlikely(attr_len > i_size)) { - /* Race with shrinking truncate. */ - attr_len = i_size; - } - addr = kmap_atomic(page); - /* Copy the data to the page. */ - memcpy(addr, (u8*)ctx->attr + - le16_to_cpu(ctx->attr->data.resident.value_offset), - attr_len); - /* Zero the remainder of the page. */ - memset(addr + attr_len, 0, PAGE_SIZE - attr_len); - flush_dcache_page(page); - kunmap_atomic(addr); -put_unm_err_out: - ntfs_attr_put_search_ctx(ctx); -unm_err_out: - unmap_mft_record(base_ni); -done: - SetPageUptodate(page); -err_out: - unlock_page(page); - return err; -} - -#ifdef NTFS_RW - -/** - * ntfs_write_block - write a @page to the backing store - * @page: page cache page to write out - * @wbc: writeback control structure - * - * This function is for writing pages belonging to non-resident, non-mst - * protected attributes to their backing store. - * - * For a page with buffers, map and write the dirty buffers asynchronously - * under page writeback. For a page without buffers, create buffers for the - * page, then proceed as above. - * - * If a page doesn't have buffers the page dirty state is definitive. If a page - * does have buffers, the page dirty state is just a hint, and the buffer dirty - * state is definitive. (A hint which has rules: dirty buffers against a clean - * page is illegal. Other combinations are legal and need to be handled. In - * particular a dirty page containing clean buffers for example.) - * - * Return 0 on success and -errno on error. - * - * Based on ntfs_read_block() and __block_write_full_folio(). - */ -static int ntfs_write_block(struct page *page, struct writeback_control *wbc) -{ - VCN vcn; - LCN lcn; - s64 initialized_size; - loff_t i_size; - sector_t block, dblock, iblock; - struct inode *vi; - ntfs_inode *ni; - ntfs_volume *vol; - runlist_element *rl; - struct buffer_head *bh, *head; - unsigned long flags; - unsigned int blocksize, vcn_ofs; - int err; - bool need_end_writeback; - unsigned char blocksize_bits; - - vi = page->mapping->host; - ni = NTFS_I(vi); - vol = ni->vol; - - ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, page index " - "0x%lx.", ni->mft_no, ni->type, page->index); - - BUG_ON(!NInoNonResident(ni)); - BUG_ON(NInoMstProtected(ni)); - blocksize = vol->sb->s_blocksize; - blocksize_bits = vol->sb->s_blocksize_bits; - if (!page_has_buffers(page)) { - BUG_ON(!PageUptodate(page)); - create_empty_buffers(page, blocksize, - (1 << BH_Uptodate) | (1 << BH_Dirty)); - if (unlikely(!page_has_buffers(page))) { - ntfs_warning(vol->sb, "Error allocating page " - "buffers. Redirtying page so we try " - "again later."); - /* - * Put the page back on mapping->dirty_pages, but leave - * its buffers' dirty state as-is. - */ - redirty_page_for_writepage(wbc, page); - unlock_page(page); - return 0; - } - } - bh = head = page_buffers(page); - BUG_ON(!bh); - - /* NOTE: Different naming scheme to ntfs_read_block()! */ - - /* The first block in the page. */ - block = (s64)page->index << (PAGE_SHIFT - blocksize_bits); - - read_lock_irqsave(&ni->size_lock, flags); - i_size = i_size_read(vi); - initialized_size = ni->initialized_size; - read_unlock_irqrestore(&ni->size_lock, flags); - - /* The first out of bounds block for the data size. */ - dblock = (i_size + blocksize - 1) >> blocksize_bits; - - /* The last (fully or partially) initialized block. */ - iblock = initialized_size >> blocksize_bits; - - /* - * Be very careful. We have no exclusion from block_dirty_folio - * here, and the (potentially unmapped) buffers may become dirty at - * any time. If a buffer becomes dirty here after we've inspected it - * then we just miss that fact, and the page stays dirty. - * - * Buffers outside i_size may be dirtied by block_dirty_folio; - * handle that here by just cleaning them. - */ - - /* - * Loop through all the buffers in the page, mapping all the dirty - * buffers to disk addresses and handling any aliases from the - * underlying block device's mapping. - */ - rl = NULL; - err = 0; - do { - bool is_retry = false; - - if (unlikely(block >= dblock)) { - /* - * Mapped buffers outside i_size will occur, because - * this page can be outside i_size when there is a - * truncate in progress. The contents of such buffers - * were zeroed by ntfs_writepage(). - * - * FIXME: What about the small race window where - * ntfs_writepage() has not done any clearing because - * the page was within i_size but before we get here, - * vmtruncate() modifies i_size? - */ - clear_buffer_dirty(bh); - set_buffer_uptodate(bh); - continue; - } - - /* Clean buffers are not written out, so no need to map them. */ - if (!buffer_dirty(bh)) - continue; - - /* Make sure we have enough initialized size. */ - if (unlikely((block >= iblock) && - (initialized_size < i_size))) { - /* - * If this page is fully outside initialized - * size, zero out all pages between the current - * initialized size and the current page. Just - * use ntfs_read_folio() to do the zeroing - * transparently. - */ - if (block > iblock) { - // TODO: - // For each page do: - // - read_cache_page() - // Again for each page do: - // - wait_on_page_locked() - // - Check (PageUptodate(page) && - // !PageError(page)) - // Update initialized size in the attribute and - // in the inode. - // Again, for each page do: - // block_dirty_folio(); - // put_page() - // We don't need to wait on the writes. - // Update iblock. - } - /* - * The current page straddles initialized size. Zero - * all non-uptodate buffers and set them uptodate (and - * dirty?). Note, there aren't any non-uptodate buffers - * if the page is uptodate. - * FIXME: For an uptodate page, the buffers may need to - * be written out because they were not initialized on - * disk before. - */ - if (!PageUptodate(page)) { - // TODO: - // Zero any non-uptodate buffers up to i_size. - // Set them uptodate and dirty. - } - // TODO: - // Update initialized size in the attribute and in the - // inode (up to i_size). - // Update iblock. - // FIXME: This is inefficient. Try to batch the two - // size changes to happen in one go. - ntfs_error(vol->sb, "Writing beyond initialized size " - "is not supported yet. Sorry."); - err = -EOPNOTSUPP; - break; - // Do NOT set_buffer_new() BUT DO clear buffer range - // outside write request range. - // set_buffer_uptodate() on complete buffers as well as - // set_buffer_dirty(). - } - - /* No need to map buffers that are already mapped. */ - if (buffer_mapped(bh)) - continue; - - /* Unmapped, dirty buffer. Need to map it. */ - bh->b_bdev = vol->sb->s_bdev; - - /* Convert block into corresponding vcn and offset. */ - vcn = (VCN)block << blocksize_bits; - vcn_ofs = vcn & vol->cluster_size_mask; - vcn >>= vol->cluster_size_bits; - if (!rl) { -lock_retry_remap: - down_read(&ni->runlist.lock); - rl = ni->runlist.rl; - } - if (likely(rl != NULL)) { - /* Seek to element containing target vcn. */ - while (rl->length && rl[1].vcn <= vcn) - rl++; - lcn = ntfs_rl_vcn_to_lcn(rl, vcn); - } else - lcn = LCN_RL_NOT_MAPPED; - /* Successful remap. */ - if (lcn >= 0) { - /* Setup buffer head to point to correct block. */ - bh->b_blocknr = ((lcn << vol->cluster_size_bits) + - vcn_ofs) >> blocksize_bits; - set_buffer_mapped(bh); - continue; - } - /* It is a hole, need to instantiate it. */ - if (lcn == LCN_HOLE) { - u8 *kaddr; - unsigned long *bpos, *bend; - - /* Check if the buffer is zero. */ - kaddr = kmap_atomic(page); - bpos = (unsigned long *)(kaddr + bh_offset(bh)); - bend = (unsigned long *)((u8*)bpos + blocksize); - do { - if (unlikely(*bpos)) - break; - } while (likely(++bpos < bend)); - kunmap_atomic(kaddr); - if (bpos == bend) { - /* - * Buffer is zero and sparse, no need to write - * it. - */ - bh->b_blocknr = -1; - clear_buffer_dirty(bh); - continue; - } - // TODO: Instantiate the hole. - // clear_buffer_new(bh); - // clean_bdev_bh_alias(bh); - ntfs_error(vol->sb, "Writing into sparse regions is " - "not supported yet. Sorry."); - err = -EOPNOTSUPP; - break; - } - /* If first try and runlist unmapped, map and retry. */ - if (!is_retry && lcn == LCN_RL_NOT_MAPPED) { - is_retry = true; - /* - * Attempt to map runlist, dropping lock for - * the duration. - */ - up_read(&ni->runlist.lock); - err = ntfs_map_runlist(ni, vcn); - if (likely(!err)) - goto lock_retry_remap; - rl = NULL; - } else if (!rl) - up_read(&ni->runlist.lock); - /* - * If buffer is outside the runlist, truncate has cut it out - * of the runlist. Just clean and clear the buffer and set it - * uptodate so it can get discarded by the VM. - */ - if (err == -ENOENT || lcn == LCN_ENOENT) { - bh->b_blocknr = -1; - clear_buffer_dirty(bh); - zero_user(page, bh_offset(bh), blocksize); - set_buffer_uptodate(bh); - err = 0; - continue; - } - /* Failed to map the buffer, even after retrying. */ - if (!err) - err = -EIO; - bh->b_blocknr = -1; - ntfs_error(vol->sb, "Failed to write to inode 0x%lx, " - "attribute type 0x%x, vcn 0x%llx, offset 0x%x " - "because its location on disk could not be " - "determined%s (error code %i).", ni->mft_no, - ni->type, (unsigned long long)vcn, - vcn_ofs, is_retry ? " even after " - "retrying" : "", err); - break; - } while (block++, (bh = bh->b_this_page) != head); - - /* Release the lock if we took it. */ - if (rl) - up_read(&ni->runlist.lock); - - /* For the error case, need to reset bh to the beginning. */ - bh = head; - - /* Just an optimization, so ->read_folio() is not called later. */ - if (unlikely(!PageUptodate(page))) { - int uptodate = 1; - do { - if (!buffer_uptodate(bh)) { - uptodate = 0; - bh = head; - break; - } - } while ((bh = bh->b_this_page) != head); - if (uptodate) - SetPageUptodate(page); - } - - /* Setup all mapped, dirty buffers for async write i/o. */ - do { - if (buffer_mapped(bh) && buffer_dirty(bh)) { - lock_buffer(bh); - if (test_clear_buffer_dirty(bh)) { - BUG_ON(!buffer_uptodate(bh)); - mark_buffer_async_write(bh); - } else - unlock_buffer(bh); - } else if (unlikely(err)) { - /* - * For the error case. The buffer may have been set - * dirty during attachment to a dirty page. - */ - if (err != -ENOMEM) - clear_buffer_dirty(bh); - } - } while ((bh = bh->b_this_page) != head); - - if (unlikely(err)) { - // TODO: Remove the -EOPNOTSUPP check later on... - if (unlikely(err == -EOPNOTSUPP)) - err = 0; - else if (err == -ENOMEM) { - ntfs_warning(vol->sb, "Error allocating memory. " - "Redirtying page so we try again " - "later."); - /* - * Put the page back on mapping->dirty_pages, but - * leave its buffer's dirty state as-is. - */ - redirty_page_for_writepage(wbc, page); - err = 0; - } else - SetPageError(page); - } - - BUG_ON(PageWriteback(page)); - set_page_writeback(page); /* Keeps try_to_free_buffers() away. */ - - /* Submit the prepared buffers for i/o. */ - need_end_writeback = true; - do { - struct buffer_head *next = bh->b_this_page; - if (buffer_async_write(bh)) { - submit_bh(REQ_OP_WRITE, bh); - need_end_writeback = false; - } - bh = next; - } while (bh != head); - unlock_page(page); - - /* If no i/o was started, need to end_page_writeback(). */ - if (unlikely(need_end_writeback)) - end_page_writeback(page); - - ntfs_debug("Done."); - return err; -} - -/** - * ntfs_write_mst_block - write a @page to the backing store - * @page: page cache page to write out - * @wbc: writeback control structure - * - * This function is for writing pages belonging to non-resident, mst protected - * attributes to their backing store. The only supported attributes are index - * allocation and $MFT/$DATA. Both directory inodes and index inodes are - * supported for the index allocation case. - * - * The page must remain locked for the duration of the write because we apply - * the mst fixups, write, and then undo the fixups, so if we were to unlock the - * page before undoing the fixups, any other user of the page will see the - * page contents as corrupt. - * - * We clear the page uptodate flag for the duration of the function to ensure - * exclusion for the $MFT/$DATA case against someone mapping an mft record we - * are about to apply the mst fixups to. - * - * Return 0 on success and -errno on error. - * - * Based on ntfs_write_block(), ntfs_mft_writepage(), and - * write_mft_record_nolock(). - */ -static int ntfs_write_mst_block(struct page *page, - struct writeback_control *wbc) -{ - sector_t block, dblock, rec_block; - struct inode *vi = page->mapping->host; - ntfs_inode *ni = NTFS_I(vi); - ntfs_volume *vol = ni->vol; - u8 *kaddr; - unsigned int rec_size = ni->itype.index.block_size; - ntfs_inode *locked_nis[PAGE_SIZE / NTFS_BLOCK_SIZE]; - struct buffer_head *bh, *head, *tbh, *rec_start_bh; - struct buffer_head *bhs[MAX_BUF_PER_PAGE]; - runlist_element *rl; - int i, nr_locked_nis, nr_recs, nr_bhs, max_bhs, bhs_per_rec, err, err2; - unsigned bh_size, rec_size_bits; - bool sync, is_mft, page_is_dirty, rec_is_dirty; - unsigned char bh_size_bits; - - if (WARN_ON(rec_size < NTFS_BLOCK_SIZE)) - return -EINVAL; - - ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, page index " - "0x%lx.", vi->i_ino, ni->type, page->index); - BUG_ON(!NInoNonResident(ni)); - BUG_ON(!NInoMstProtected(ni)); - is_mft = (S_ISREG(vi->i_mode) && !vi->i_ino); - /* - * NOTE: ntfs_write_mst_block() would be called for $MFTMirr if a page - * in its page cache were to be marked dirty. However this should - * never happen with the current driver and considering we do not - * handle this case here we do want to BUG(), at least for now. - */ - BUG_ON(!(is_mft || S_ISDIR(vi->i_mode) || - (NInoAttr(ni) && ni->type == AT_INDEX_ALLOCATION))); - bh_size = vol->sb->s_blocksize; - bh_size_bits = vol->sb->s_blocksize_bits; - max_bhs = PAGE_SIZE / bh_size; - BUG_ON(!max_bhs); - BUG_ON(max_bhs > MAX_BUF_PER_PAGE); - - /* Were we called for sync purposes? */ - sync = (wbc->sync_mode == WB_SYNC_ALL); - - /* Make sure we have mapped buffers. */ - bh = head = page_buffers(page); - BUG_ON(!bh); - - rec_size_bits = ni->itype.index.block_size_bits; - BUG_ON(!(PAGE_SIZE >> rec_size_bits)); - bhs_per_rec = rec_size >> bh_size_bits; - BUG_ON(!bhs_per_rec); - - /* The first block in the page. */ - rec_block = block = (sector_t)page->index << - (PAGE_SHIFT - bh_size_bits); - - /* The first out of bounds block for the data size. */ - dblock = (i_size_read(vi) + bh_size - 1) >> bh_size_bits; - - rl = NULL; - err = err2 = nr_bhs = nr_recs = nr_locked_nis = 0; - page_is_dirty = rec_is_dirty = false; - rec_start_bh = NULL; - do { - bool is_retry = false; - - if (likely(block < rec_block)) { - if (unlikely(block >= dblock)) { - clear_buffer_dirty(bh); - set_buffer_uptodate(bh); - continue; - } - /* - * This block is not the first one in the record. We - * ignore the buffer's dirty state because we could - * have raced with a parallel mark_ntfs_record_dirty(). - */ - if (!rec_is_dirty) - continue; - if (unlikely(err2)) { - if (err2 != -ENOMEM) - clear_buffer_dirty(bh); - continue; - } - } else /* if (block == rec_block) */ { - BUG_ON(block > rec_block); - /* This block is the first one in the record. */ - rec_block += bhs_per_rec; - err2 = 0; - if (unlikely(block >= dblock)) { - clear_buffer_dirty(bh); - continue; - } - if (!buffer_dirty(bh)) { - /* Clean records are not written out. */ - rec_is_dirty = false; - continue; - } - rec_is_dirty = true; - rec_start_bh = bh; - } - /* Need to map the buffer if it is not mapped already. */ - if (unlikely(!buffer_mapped(bh))) { - VCN vcn; - LCN lcn; - unsigned int vcn_ofs; - - bh->b_bdev = vol->sb->s_bdev; - /* Obtain the vcn and offset of the current block. */ - vcn = (VCN)block << bh_size_bits; - vcn_ofs = vcn & vol->cluster_size_mask; - vcn >>= vol->cluster_size_bits; - if (!rl) { -lock_retry_remap: - down_read(&ni->runlist.lock); - rl = ni->runlist.rl; - } - if (likely(rl != NULL)) { - /* Seek to element containing target vcn. */ - while (rl->length && rl[1].vcn <= vcn) - rl++; - lcn = ntfs_rl_vcn_to_lcn(rl, vcn); - } else - lcn = LCN_RL_NOT_MAPPED; - /* Successful remap. */ - if (likely(lcn >= 0)) { - /* Setup buffer head to correct block. */ - bh->b_blocknr = ((lcn << - vol->cluster_size_bits) + - vcn_ofs) >> bh_size_bits; - set_buffer_mapped(bh); - } else { - /* - * Remap failed. Retry to map the runlist once - * unless we are working on $MFT which always - * has the whole of its runlist in memory. - */ - if (!is_mft && !is_retry && - lcn == LCN_RL_NOT_MAPPED) { - is_retry = true; - /* - * Attempt to map runlist, dropping - * lock for the duration. - */ - up_read(&ni->runlist.lock); - err2 = ntfs_map_runlist(ni, vcn); - if (likely(!err2)) - goto lock_retry_remap; - if (err2 == -ENOMEM) - page_is_dirty = true; - lcn = err2; - } else { - err2 = -EIO; - if (!rl) - up_read(&ni->runlist.lock); - } - /* Hard error. Abort writing this record. */ - if (!err || err == -ENOMEM) - err = err2; - bh->b_blocknr = -1; - ntfs_error(vol->sb, "Cannot write ntfs record " - "0x%llx (inode 0x%lx, " - "attribute type 0x%x) because " - "its location on disk could " - "not be determined (error " - "code %lli).", - (long long)block << - bh_size_bits >> - vol->mft_record_size_bits, - ni->mft_no, ni->type, - (long long)lcn); - /* - * If this is not the first buffer, remove the - * buffers in this record from the list of - * buffers to write and clear their dirty bit - * if not error -ENOMEM. - */ - if (rec_start_bh != bh) { - while (bhs[--nr_bhs] != rec_start_bh) - ; - if (err2 != -ENOMEM) { - do { - clear_buffer_dirty( - rec_start_bh); - } while ((rec_start_bh = - rec_start_bh-> - b_this_page) != - bh); - } - } - continue; - } - } - BUG_ON(!buffer_uptodate(bh)); - BUG_ON(nr_bhs >= max_bhs); - bhs[nr_bhs++] = bh; - } while (block++, (bh = bh->b_this_page) != head); - if (unlikely(rl)) - up_read(&ni->runlist.lock); - /* If there were no dirty buffers, we are done. */ - if (!nr_bhs) - goto done; - /* Map the page so we can access its contents. */ - kaddr = kmap(page); - /* Clear the page uptodate flag whilst the mst fixups are applied. */ - BUG_ON(!PageUptodate(page)); - ClearPageUptodate(page); - for (i = 0; i < nr_bhs; i++) { - unsigned int ofs; - - /* Skip buffers which are not at the beginning of records. */ - if (i % bhs_per_rec) - continue; - tbh = bhs[i]; - ofs = bh_offset(tbh); - if (is_mft) { - ntfs_inode *tni; - unsigned long mft_no; - - /* Get the mft record number. */ - mft_no = (((s64)page->index << PAGE_SHIFT) + ofs) - >> rec_size_bits; - /* Check whether to write this mft record. */ - tni = NULL; - if (!ntfs_may_write_mft_record(vol, mft_no, - (MFT_RECORD*)(kaddr + ofs), &tni)) { - /* - * The record should not be written. This - * means we need to redirty the page before - * returning. - */ - page_is_dirty = true; - /* - * Remove the buffers in this mft record from - * the list of buffers to write. - */ - do { - bhs[i] = NULL; - } while (++i % bhs_per_rec); - continue; - } - /* - * The record should be written. If a locked ntfs - * inode was returned, add it to the array of locked - * ntfs inodes. - */ - if (tni) - locked_nis[nr_locked_nis++] = tni; - } - /* Apply the mst protection fixups. */ - err2 = pre_write_mst_fixup((NTFS_RECORD*)(kaddr + ofs), - rec_size); - if (unlikely(err2)) { - if (!err || err == -ENOMEM) - err = -EIO; - ntfs_error(vol->sb, "Failed to apply mst fixups " - "(inode 0x%lx, attribute type 0x%x, " - "page index 0x%lx, page offset 0x%x)!" - " Unmount and run chkdsk.", vi->i_ino, - ni->type, page->index, ofs); - /* - * Mark all the buffers in this record clean as we do - * not want to write corrupt data to disk. - */ - do { - clear_buffer_dirty(bhs[i]); - bhs[i] = NULL; - } while (++i % bhs_per_rec); - continue; - } - nr_recs++; - } - /* If no records are to be written out, we are done. */ - if (!nr_recs) - goto unm_done; - flush_dcache_page(page); - /* Lock buffers and start synchronous write i/o on them. */ - for (i = 0; i < nr_bhs; i++) { - tbh = bhs[i]; - if (!tbh) - continue; - if (!trylock_buffer(tbh)) - BUG(); - /* The buffer dirty state is now irrelevant, just clean it. */ - clear_buffer_dirty(tbh); - BUG_ON(!buffer_uptodate(tbh)); - BUG_ON(!buffer_mapped(tbh)); - get_bh(tbh); - tbh->b_end_io = end_buffer_write_sync; - submit_bh(REQ_OP_WRITE, tbh); - } - /* Synchronize the mft mirror now if not @sync. */ - if (is_mft && !sync) - goto do_mirror; -do_wait: - /* Wait on i/o completion of buffers. */ - for (i = 0; i < nr_bhs; i++) { - tbh = bhs[i]; - if (!tbh) - continue; - wait_on_buffer(tbh); - if (unlikely(!buffer_uptodate(tbh))) { - ntfs_error(vol->sb, "I/O error while writing ntfs " - "record buffer (inode 0x%lx, " - "attribute type 0x%x, page index " - "0x%lx, page offset 0x%lx)! Unmount " - "and run chkdsk.", vi->i_ino, ni->type, - page->index, bh_offset(tbh)); - if (!err || err == -ENOMEM) - err = -EIO; - /* - * Set the buffer uptodate so the page and buffer - * states do not become out of sync. - */ - set_buffer_uptodate(tbh); - } - } - /* If @sync, now synchronize the mft mirror. */ - if (is_mft && sync) { -do_mirror: - for (i = 0; i < nr_bhs; i++) { - unsigned long mft_no; - unsigned int ofs; - - /* - * Skip buffers which are not at the beginning of - * records. - */ - if (i % bhs_per_rec) - continue; - tbh = bhs[i]; - /* Skip removed buffers (and hence records). */ - if (!tbh) - continue; - ofs = bh_offset(tbh); - /* Get the mft record number. */ - mft_no = (((s64)page->index << PAGE_SHIFT) + ofs) - >> rec_size_bits; - if (mft_no < vol->mftmirr_size) - ntfs_sync_mft_mirror(vol, mft_no, - (MFT_RECORD*)(kaddr + ofs), - sync); - } - if (!sync) - goto do_wait; - } - /* Remove the mst protection fixups again. */ - for (i = 0; i < nr_bhs; i++) { - if (!(i % bhs_per_rec)) { - tbh = bhs[i]; - if (!tbh) - continue; - post_write_mst_fixup((NTFS_RECORD*)(kaddr + - bh_offset(tbh))); - } - } - flush_dcache_page(page); -unm_done: - /* Unlock any locked inodes. */ - while (nr_locked_nis-- > 0) { - ntfs_inode *tni, *base_tni; - - tni = locked_nis[nr_locked_nis]; - /* Get the base inode. */ - mutex_lock(&tni->extent_lock); - if (tni->nr_extents >= 0) - base_tni = tni; - else { - base_tni = tni->ext.base_ntfs_ino; - BUG_ON(!base_tni); - } - mutex_unlock(&tni->extent_lock); - ntfs_debug("Unlocking %s inode 0x%lx.", - tni == base_tni ? "base" : "extent", - tni->mft_no); - mutex_unlock(&tni->mrec_lock); - atomic_dec(&tni->count); - iput(VFS_I(base_tni)); - } - SetPageUptodate(page); - kunmap(page); -done: - if (unlikely(err && err != -ENOMEM)) { - /* - * Set page error if there is only one ntfs record in the page. - * Otherwise we would loose per-record granularity. - */ - if (ni->itype.index.block_size == PAGE_SIZE) - SetPageError(page); - NVolSetErrors(vol); - } - if (page_is_dirty) { - ntfs_debug("Page still contains one or more dirty ntfs " - "records. Redirtying the page starting at " - "record 0x%lx.", page->index << - (PAGE_SHIFT - rec_size_bits)); - redirty_page_for_writepage(wbc, page); - unlock_page(page); - } else { - /* - * Keep the VM happy. This must be done otherwise the - * radix-tree tag PAGECACHE_TAG_DIRTY remains set even though - * the page is clean. - */ - BUG_ON(PageWriteback(page)); - set_page_writeback(page); - unlock_page(page); - end_page_writeback(page); - } - if (likely(!err)) - ntfs_debug("Done."); - return err; -} - -/** - * ntfs_writepage - write a @page to the backing store - * @page: page cache page to write out - * @wbc: writeback control structure - * - * This is called from the VM when it wants to have a dirty ntfs page cache - * page cleaned. The VM has already locked the page and marked it clean. - * - * For non-resident attributes, ntfs_writepage() writes the @page by calling - * the ntfs version of the generic block_write_full_page() function, - * ntfs_write_block(), which in turn if necessary creates and writes the - * buffers associated with the page asynchronously. - * - * For resident attributes, OTOH, ntfs_writepage() writes the @page by copying - * the data to the mft record (which at this stage is most likely in memory). - * The mft record is then marked dirty and written out asynchronously via the - * vfs inode dirty code path for the inode the mft record belongs to or via the - * vm page dirty code path for the page the mft record is in. - * - * Based on ntfs_read_folio() and fs/buffer.c::block_write_full_page(). - * - * Return 0 on success and -errno on error. - */ -static int ntfs_writepage(struct page *page, struct writeback_control *wbc) -{ - loff_t i_size; - struct inode *vi = page->mapping->host; - ntfs_inode *base_ni = NULL, *ni = NTFS_I(vi); - char *addr; - ntfs_attr_search_ctx *ctx = NULL; - MFT_RECORD *m = NULL; - u32 attr_len; - int err; - -retry_writepage: - BUG_ON(!PageLocked(page)); - i_size = i_size_read(vi); - /* Is the page fully outside i_size? (truncate in progress) */ - if (unlikely(page->index >= (i_size + PAGE_SIZE - 1) >> - PAGE_SHIFT)) { - struct folio *folio = page_folio(page); - /* - * The page may have dirty, unmapped buffers. Make them - * freeable here, so the page does not leak. - */ - block_invalidate_folio(folio, 0, folio_size(folio)); - folio_unlock(folio); - ntfs_debug("Write outside i_size - truncated?"); - return 0; - } - /* - * Only $DATA attributes can be encrypted and only unnamed $DATA - * attributes can be compressed. Index root can have the flags set but - * this means to create compressed/encrypted files, not that the - * attribute is compressed/encrypted. Note we need to check for - * AT_INDEX_ALLOCATION since this is the type of both directory and - * index inodes. - */ - if (ni->type != AT_INDEX_ALLOCATION) { - /* If file is encrypted, deny access, just like NT4. */ - if (NInoEncrypted(ni)) { - unlock_page(page); - BUG_ON(ni->type != AT_DATA); - ntfs_debug("Denying write access to encrypted file."); - return -EACCES; - } - /* Compressed data streams are handled in compress.c. */ - if (NInoNonResident(ni) && NInoCompressed(ni)) { - BUG_ON(ni->type != AT_DATA); - BUG_ON(ni->name_len); - // TODO: Implement and replace this with - // return ntfs_write_compressed_block(page); - unlock_page(page); - ntfs_error(vi->i_sb, "Writing to compressed files is " - "not supported yet. Sorry."); - return -EOPNOTSUPP; - } - // TODO: Implement and remove this check. - if (NInoNonResident(ni) && NInoSparse(ni)) { - unlock_page(page); - ntfs_error(vi->i_sb, "Writing to sparse files is not " - "supported yet. Sorry."); - return -EOPNOTSUPP; - } - } - /* NInoNonResident() == NInoIndexAllocPresent() */ - if (NInoNonResident(ni)) { - /* We have to zero every time due to mmap-at-end-of-file. */ - if (page->index >= (i_size >> PAGE_SHIFT)) { - /* The page straddles i_size. */ - unsigned int ofs = i_size & ~PAGE_MASK; - zero_user_segment(page, ofs, PAGE_SIZE); - } - /* Handle mst protected attributes. */ - if (NInoMstProtected(ni)) - return ntfs_write_mst_block(page, wbc); - /* Normal, non-resident data stream. */ - return ntfs_write_block(page, wbc); - } - /* - * Attribute is resident, implying it is not compressed, encrypted, or - * mst protected. This also means the attribute is smaller than an mft - * record and hence smaller than a page, so can simply return error on - * any pages with index above 0. Note the attribute can actually be - * marked compressed but if it is resident the actual data is not - * compressed so we are ok to ignore the compressed flag here. - */ - BUG_ON(page_has_buffers(page)); - BUG_ON(!PageUptodate(page)); - if (unlikely(page->index > 0)) { - ntfs_error(vi->i_sb, "BUG()! page->index (0x%lx) > 0. " - "Aborting write.", page->index); - BUG_ON(PageWriteback(page)); - set_page_writeback(page); - unlock_page(page); - end_page_writeback(page); - return -EIO; - } - if (!NInoAttr(ni)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - /* Map, pin, and lock the mft record. */ - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - m = NULL; - ctx = NULL; - goto err_out; - } - /* - * If a parallel write made the attribute non-resident, drop the mft - * record and retry the writepage. - */ - if (unlikely(NInoNonResident(ni))) { - unmap_mft_record(base_ni); - goto retry_writepage; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) - goto err_out; - /* - * Keep the VM happy. This must be done otherwise the radix-tree tag - * PAGECACHE_TAG_DIRTY remains set even though the page is clean. - */ - BUG_ON(PageWriteback(page)); - set_page_writeback(page); - unlock_page(page); - attr_len = le32_to_cpu(ctx->attr->data.resident.value_length); - i_size = i_size_read(vi); - if (unlikely(attr_len > i_size)) { - /* Race with shrinking truncate or a failed truncate. */ - attr_len = i_size; - /* - * If the truncate failed, fix it up now. If a concurrent - * truncate, we do its job, so it does not have to do anything. - */ - err = ntfs_resident_attr_value_resize(ctx->mrec, ctx->attr, - attr_len); - /* Shrinking cannot fail. */ - BUG_ON(err); - } - addr = kmap_atomic(page); - /* Copy the data from the page to the mft record. */ - memcpy((u8*)ctx->attr + - le16_to_cpu(ctx->attr->data.resident.value_offset), - addr, attr_len); - /* Zero out of bounds area in the page cache page. */ - memset(addr + attr_len, 0, PAGE_SIZE - attr_len); - kunmap_atomic(addr); - flush_dcache_page(page); - flush_dcache_mft_record_page(ctx->ntfs_ino); - /* We are done with the page. */ - end_page_writeback(page); - /* Finally, mark the mft record dirty, so it gets written back. */ - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - return 0; -err_out: - if (err == -ENOMEM) { - ntfs_warning(vi->i_sb, "Error allocating memory. Redirtying " - "page so we try again later."); - /* - * Put the page back on mapping->dirty_pages, but leave its - * buffers' dirty state as-is. - */ - redirty_page_for_writepage(wbc, page); - err = 0; - } else { - ntfs_error(vi->i_sb, "Resident attribute write failed with " - "error %i.", err); - SetPageError(page); - NVolSetErrors(ni->vol); - } - unlock_page(page); - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(base_ni); - return err; -} - -#endif /* NTFS_RW */ - -/** - * ntfs_bmap - map logical file block to physical device block - * @mapping: address space mapping to which the block to be mapped belongs - * @block: logical block to map to its physical device block - * - * For regular, non-resident files (i.e. not compressed and not encrypted), map - * the logical @block belonging to the file described by the address space - * mapping @mapping to its physical device block. - * - * The size of the block is equal to the @s_blocksize field of the super block - * of the mounted file system which is guaranteed to be smaller than or equal - * to the cluster size thus the block is guaranteed to fit entirely inside the - * cluster which means we do not need to care how many contiguous bytes are - * available after the beginning of the block. - * - * Return the physical device block if the mapping succeeded or 0 if the block - * is sparse or there was an error. - * - * Note: This is a problem if someone tries to run bmap() on $Boot system file - * as that really is in block zero but there is nothing we can do. bmap() is - * just broken in that respect (just like it cannot distinguish sparse from - * not available or error). - */ -static sector_t ntfs_bmap(struct address_space *mapping, sector_t block) -{ - s64 ofs, size; - loff_t i_size; - LCN lcn; - unsigned long blocksize, flags; - ntfs_inode *ni = NTFS_I(mapping->host); - ntfs_volume *vol = ni->vol; - unsigned delta; - unsigned char blocksize_bits, cluster_size_shift; - - ntfs_debug("Entering for mft_no 0x%lx, logical block 0x%llx.", - ni->mft_no, (unsigned long long)block); - if (ni->type != AT_DATA || !NInoNonResident(ni) || NInoEncrypted(ni)) { - ntfs_error(vol->sb, "BMAP does not make sense for %s " - "attributes, returning 0.", - (ni->type != AT_DATA) ? "non-data" : - (!NInoNonResident(ni) ? "resident" : - "encrypted")); - return 0; - } - /* None of these can happen. */ - BUG_ON(NInoCompressed(ni)); - BUG_ON(NInoMstProtected(ni)); - blocksize = vol->sb->s_blocksize; - blocksize_bits = vol->sb->s_blocksize_bits; - ofs = (s64)block << blocksize_bits; - read_lock_irqsave(&ni->size_lock, flags); - size = ni->initialized_size; - i_size = i_size_read(VFS_I(ni)); - read_unlock_irqrestore(&ni->size_lock, flags); - /* - * If the offset is outside the initialized size or the block straddles - * the initialized size then pretend it is a hole unless the - * initialized size equals the file size. - */ - if (unlikely(ofs >= size || (ofs + blocksize > size && size < i_size))) - goto hole; - cluster_size_shift = vol->cluster_size_bits; - down_read(&ni->runlist.lock); - lcn = ntfs_attr_vcn_to_lcn_nolock(ni, ofs >> cluster_size_shift, false); - up_read(&ni->runlist.lock); - if (unlikely(lcn < LCN_HOLE)) { - /* - * Step down to an integer to avoid gcc doing a long long - * comparision in the switch when we know @lcn is between - * LCN_HOLE and LCN_EIO (i.e. -1 to -5). - * - * Otherwise older gcc (at least on some architectures) will - * try to use __cmpdi2() which is of course not available in - * the kernel. - */ - switch ((int)lcn) { - case LCN_ENOENT: - /* - * If the offset is out of bounds then pretend it is a - * hole. - */ - goto hole; - case LCN_ENOMEM: - ntfs_error(vol->sb, "Not enough memory to complete " - "mapping for inode 0x%lx. " - "Returning 0.", ni->mft_no); - break; - default: - ntfs_error(vol->sb, "Failed to complete mapping for " - "inode 0x%lx. Run chkdsk. " - "Returning 0.", ni->mft_no); - break; - } - return 0; - } - if (lcn < 0) { - /* It is a hole. */ -hole: - ntfs_debug("Done (returning hole)."); - return 0; - } - /* - * The block is really allocated and fullfils all our criteria. - * Convert the cluster to units of block size and return the result. - */ - delta = ofs & vol->cluster_size_mask; - if (unlikely(sizeof(block) < sizeof(lcn))) { - block = lcn = ((lcn << cluster_size_shift) + delta) >> - blocksize_bits; - /* If the block number was truncated return 0. */ - if (unlikely(block != lcn)) { - ntfs_error(vol->sb, "Physical block 0x%llx is too " - "large to be returned, returning 0.", - (long long)lcn); - return 0; - } - } else - block = ((lcn << cluster_size_shift) + delta) >> - blocksize_bits; - ntfs_debug("Done (returning block 0x%llx).", (unsigned long long)lcn); - return block; -} - -/* - * ntfs_normal_aops - address space operations for normal inodes and attributes - * - * Note these are not used for compressed or mst protected inodes and - * attributes. - */ -const struct address_space_operations ntfs_normal_aops = { - .read_folio = ntfs_read_folio, -#ifdef NTFS_RW - .writepage = ntfs_writepage, - .dirty_folio = block_dirty_folio, -#endif /* NTFS_RW */ - .bmap = ntfs_bmap, - .migrate_folio = buffer_migrate_folio, - .is_partially_uptodate = block_is_partially_uptodate, - .error_remove_page = generic_error_remove_page, -}; - -/* - * ntfs_compressed_aops - address space operations for compressed inodes - */ -const struct address_space_operations ntfs_compressed_aops = { - .read_folio = ntfs_read_folio, -#ifdef NTFS_RW - .writepage = ntfs_writepage, - .dirty_folio = block_dirty_folio, -#endif /* NTFS_RW */ - .migrate_folio = buffer_migrate_folio, - .is_partially_uptodate = block_is_partially_uptodate, - .error_remove_page = generic_error_remove_page, -}; - -/* - * ntfs_mst_aops - general address space operations for mst protecteed inodes - * and attributes - */ -const struct address_space_operations ntfs_mst_aops = { - .read_folio = ntfs_read_folio, /* Fill page with data. */ -#ifdef NTFS_RW - .writepage = ntfs_writepage, /* Write dirty page to disk. */ - .dirty_folio = filemap_dirty_folio, -#endif /* NTFS_RW */ - .migrate_folio = buffer_migrate_folio, - .is_partially_uptodate = block_is_partially_uptodate, - .error_remove_page = generic_error_remove_page, -}; - -#ifdef NTFS_RW - -/** - * mark_ntfs_record_dirty - mark an ntfs record dirty - * @page: page containing the ntfs record to mark dirty - * @ofs: byte offset within @page at which the ntfs record begins - * - * Set the buffers and the page in which the ntfs record is located dirty. - * - * The latter also marks the vfs inode the ntfs record belongs to dirty - * (I_DIRTY_PAGES only). - * - * If the page does not have buffers, we create them and set them uptodate. - * The page may not be locked which is why we need to handle the buffers under - * the mapping->private_lock. Once the buffers are marked dirty we no longer - * need the lock since try_to_free_buffers() does not free dirty buffers. - */ -void mark_ntfs_record_dirty(struct page *page, const unsigned int ofs) { - struct address_space *mapping = page->mapping; - ntfs_inode *ni = NTFS_I(mapping->host); - struct buffer_head *bh, *head, *buffers_to_free = NULL; - unsigned int end, bh_size, bh_ofs; - - BUG_ON(!PageUptodate(page)); - end = ofs + ni->itype.index.block_size; - bh_size = VFS_I(ni)->i_sb->s_blocksize; - spin_lock(&mapping->private_lock); - if (unlikely(!page_has_buffers(page))) { - spin_unlock(&mapping->private_lock); - bh = head = alloc_page_buffers(page, bh_size, true); - spin_lock(&mapping->private_lock); - if (likely(!page_has_buffers(page))) { - struct buffer_head *tail; - - do { - set_buffer_uptodate(bh); - tail = bh; - bh = bh->b_this_page; - } while (bh); - tail->b_this_page = head; - attach_page_private(page, head); - } else - buffers_to_free = bh; - } - bh = head = page_buffers(page); - BUG_ON(!bh); - do { - bh_ofs = bh_offset(bh); - if (bh_ofs + bh_size <= ofs) - continue; - if (unlikely(bh_ofs >= end)) - break; - set_buffer_dirty(bh); - } while ((bh = bh->b_this_page) != head); - spin_unlock(&mapping->private_lock); - filemap_dirty_folio(mapping, page_folio(page)); - if (unlikely(buffers_to_free)) { - do { - bh = buffers_to_free->b_this_page; - free_buffer_head(buffers_to_free); - buffers_to_free = bh; - } while (buffers_to_free); - } -} - -#endif /* NTFS_RW */ diff --git a/fs/ntfs/aops.h b/fs/ntfs/aops.h deleted file mode 100644 index 8d0958a149cb..000000000000 --- a/fs/ntfs/aops.h +++ /dev/null @@ -1,88 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * aops.h - Defines for NTFS kernel address space operations and page cache - * handling. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2004 Anton Altaparmakov - * Copyright (c) 2002 Richard Russon - */ - -#ifndef _LINUX_NTFS_AOPS_H -#define _LINUX_NTFS_AOPS_H - -#include -#include -#include -#include - -#include "inode.h" - -/** - * ntfs_unmap_page - release a page that was mapped using ntfs_map_page() - * @page: the page to release - * - * Unpin, unmap and release a page that was obtained from ntfs_map_page(). - */ -static inline void ntfs_unmap_page(struct page *page) -{ - kunmap(page); - put_page(page); -} - -/** - * ntfs_map_page - map a page into accessible memory, reading it if necessary - * @mapping: address space for which to obtain the page - * @index: index into the page cache for @mapping of the page to map - * - * Read a page from the page cache of the address space @mapping at position - * @index, where @index is in units of PAGE_SIZE, and not in bytes. - * - * If the page is not in memory it is loaded from disk first using the - * read_folio method defined in the address space operations of @mapping - * and the page is added to the page cache of @mapping in the process. - * - * If the page belongs to an mst protected attribute and it is marked as such - * in its ntfs inode (NInoMstProtected()) the mst fixups are applied but no - * error checking is performed. This means the caller has to verify whether - * the ntfs record(s) contained in the page are valid or not using one of the - * ntfs_is_XXXX_record{,p}() macros, where XXXX is the record type you are - * expecting to see. (For details of the macros, see fs/ntfs/layout.h.) - * - * If the page is in high memory it is mapped into memory directly addressible - * by the kernel. - * - * Finally the page count is incremented, thus pinning the page into place. - * - * The above means that page_address(page) can be used on all pages obtained - * with ntfs_map_page() to get the kernel virtual address of the page. - * - * When finished with the page, the caller has to call ntfs_unmap_page() to - * unpin, unmap and release the page. - * - * Note this does not grant exclusive access. If such is desired, the caller - * must provide it independently of the ntfs_{un}map_page() calls by using - * a {rw_}semaphore or other means of serialization. A spin lock cannot be - * used as ntfs_map_page() can block. - * - * The unlocked and uptodate page is returned on success or an encoded error - * on failure. Caller has to test for error using the IS_ERR() macro on the - * return value. If that evaluates to 'true', the negative error code can be - * obtained using PTR_ERR() on the return value of ntfs_map_page(). - */ -static inline struct page *ntfs_map_page(struct address_space *mapping, - unsigned long index) -{ - struct page *page = read_mapping_page(mapping, index, NULL); - - if (!IS_ERR(page)) - kmap(page); - return page; -} - -#ifdef NTFS_RW - -extern void mark_ntfs_record_dirty(struct page *page, const unsigned int ofs); - -#endif /* NTFS_RW */ - -#endif /* _LINUX_NTFS_AOPS_H */ diff --git a/fs/ntfs/attrib.c b/fs/ntfs/attrib.c deleted file mode 100644 index f79408f9127a..000000000000 --- a/fs/ntfs/attrib.c +++ /dev/null @@ -1,2624 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * attrib.c - NTFS attribute operations. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2012 Anton Altaparmakov and Tuxera Inc. - * Copyright (c) 2002 Richard Russon - */ - -#include -#include -#include -#include -#include - -#include "attrib.h" -#include "debug.h" -#include "layout.h" -#include "lcnalloc.h" -#include "malloc.h" -#include "mft.h" -#include "ntfs.h" -#include "types.h" - -/** - * ntfs_map_runlist_nolock - map (a part of) a runlist of an ntfs inode - * @ni: ntfs inode for which to map (part of) a runlist - * @vcn: map runlist part containing this vcn - * @ctx: active attribute search context if present or NULL if not - * - * Map the part of a runlist containing the @vcn of the ntfs inode @ni. - * - * If @ctx is specified, it is an active search context of @ni and its base mft - * record. This is needed when ntfs_map_runlist_nolock() encounters unmapped - * runlist fragments and allows their mapping. If you do not have the mft - * record mapped, you can specify @ctx as NULL and ntfs_map_runlist_nolock() - * will perform the necessary mapping and unmapping. - * - * Note, ntfs_map_runlist_nolock() saves the state of @ctx on entry and - * restores it before returning. Thus, @ctx will be left pointing to the same - * attribute on return as on entry. However, the actual pointers in @ctx may - * point to different memory locations on return, so you must remember to reset - * any cached pointers from the @ctx, i.e. after the call to - * ntfs_map_runlist_nolock(), you will probably want to do: - * m = ctx->mrec; - * a = ctx->attr; - * Assuming you cache ctx->attr in a variable @a of type ATTR_RECORD * and that - * you cache ctx->mrec in a variable @m of type MFT_RECORD *. - * - * Return 0 on success and -errno on error. There is one special error code - * which is not an error as such. This is -ENOENT. It means that @vcn is out - * of bounds of the runlist. - * - * Note the runlist can be NULL after this function returns if @vcn is zero and - * the attribute has zero allocated size, i.e. there simply is no runlist. - * - * WARNING: If @ctx is supplied, regardless of whether success or failure is - * returned, you need to check IS_ERR(@ctx->mrec) and if 'true' the @ctx - * is no longer valid, i.e. you need to either call - * ntfs_attr_reinit_search_ctx() or ntfs_attr_put_search_ctx() on it. - * In that case PTR_ERR(@ctx->mrec) will give you the error code for - * why the mapping of the old inode failed. - * - * Locking: - The runlist described by @ni must be locked for writing on entry - * and is locked on return. Note the runlist will be modified. - * - If @ctx is NULL, the base mft record of @ni must not be mapped on - * entry and it will be left unmapped on return. - * - If @ctx is not NULL, the base mft record must be mapped on entry - * and it will be left mapped on return. - */ -int ntfs_map_runlist_nolock(ntfs_inode *ni, VCN vcn, ntfs_attr_search_ctx *ctx) -{ - VCN end_vcn; - unsigned long flags; - ntfs_inode *base_ni; - MFT_RECORD *m; - ATTR_RECORD *a; - runlist_element *rl; - struct page *put_this_page = NULL; - int err = 0; - bool ctx_is_temporary, ctx_needs_reset; - ntfs_attr_search_ctx old_ctx = { NULL, }; - - ntfs_debug("Mapping runlist part containing vcn 0x%llx.", - (unsigned long long)vcn); - if (!NInoAttr(ni)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - if (!ctx) { - ctx_is_temporary = ctx_needs_reset = true; - m = map_mft_record(base_ni); - if (IS_ERR(m)) - return PTR_ERR(m); - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - } else { - VCN allocated_size_vcn; - - BUG_ON(IS_ERR(ctx->mrec)); - a = ctx->attr; - BUG_ON(!a->non_resident); - ctx_is_temporary = false; - end_vcn = sle64_to_cpu(a->data.non_resident.highest_vcn); - read_lock_irqsave(&ni->size_lock, flags); - allocated_size_vcn = ni->allocated_size >> - ni->vol->cluster_size_bits; - read_unlock_irqrestore(&ni->size_lock, flags); - if (!a->data.non_resident.lowest_vcn && end_vcn <= 0) - end_vcn = allocated_size_vcn - 1; - /* - * If we already have the attribute extent containing @vcn in - * @ctx, no need to look it up again. We slightly cheat in - * that if vcn exceeds the allocated size, we will refuse to - * map the runlist below, so there is definitely no need to get - * the right attribute extent. - */ - if (vcn >= allocated_size_vcn || (a->type == ni->type && - a->name_length == ni->name_len && - !memcmp((u8*)a + le16_to_cpu(a->name_offset), - ni->name, ni->name_len) && - sle64_to_cpu(a->data.non_resident.lowest_vcn) - <= vcn && end_vcn >= vcn)) - ctx_needs_reset = false; - else { - /* Save the old search context. */ - old_ctx = *ctx; - /* - * If the currently mapped (extent) inode is not the - * base inode we will unmap it when we reinitialize the - * search context which means we need to get a - * reference to the page containing the mapped mft - * record so we do not accidentally drop changes to the - * mft record when it has not been marked dirty yet. - */ - if (old_ctx.base_ntfs_ino && old_ctx.ntfs_ino != - old_ctx.base_ntfs_ino) { - put_this_page = old_ctx.ntfs_ino->page; - get_page(put_this_page); - } - /* - * Reinitialize the search context so we can lookup the - * needed attribute extent. - */ - ntfs_attr_reinit_search_ctx(ctx); - ctx_needs_reset = true; - } - } - if (ctx_needs_reset) { - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, vcn, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - err = -EIO; - goto err_out; - } - BUG_ON(!ctx->attr->non_resident); - } - a = ctx->attr; - /* - * Only decompress the mapping pairs if @vcn is inside it. Otherwise - * we get into problems when we try to map an out of bounds vcn because - * we then try to map the already mapped runlist fragment and - * ntfs_mapping_pairs_decompress() fails. - */ - end_vcn = sle64_to_cpu(a->data.non_resident.highest_vcn) + 1; - if (unlikely(vcn && vcn >= end_vcn)) { - err = -ENOENT; - goto err_out; - } - rl = ntfs_mapping_pairs_decompress(ni->vol, a, ni->runlist.rl); - if (IS_ERR(rl)) - err = PTR_ERR(rl); - else - ni->runlist.rl = rl; -err_out: - if (ctx_is_temporary) { - if (likely(ctx)) - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - } else if (ctx_needs_reset) { - /* - * If there is no attribute list, restoring the search context - * is accomplished simply by copying the saved context back over - * the caller supplied context. If there is an attribute list, - * things are more complicated as we need to deal with mapping - * of mft records and resulting potential changes in pointers. - */ - if (NInoAttrList(base_ni)) { - /* - * If the currently mapped (extent) inode is not the - * one we had before, we need to unmap it and map the - * old one. - */ - if (ctx->ntfs_ino != old_ctx.ntfs_ino) { - /* - * If the currently mapped inode is not the - * base inode, unmap it. - */ - if (ctx->base_ntfs_ino && ctx->ntfs_ino != - ctx->base_ntfs_ino) { - unmap_extent_mft_record(ctx->ntfs_ino); - ctx->mrec = ctx->base_mrec; - BUG_ON(!ctx->mrec); - } - /* - * If the old mapped inode is not the base - * inode, map it. - */ - if (old_ctx.base_ntfs_ino && - old_ctx.ntfs_ino != - old_ctx.base_ntfs_ino) { -retry_map: - ctx->mrec = map_mft_record( - old_ctx.ntfs_ino); - /* - * Something bad has happened. If out - * of memory retry till it succeeds. - * Any other errors are fatal and we - * return the error code in ctx->mrec. - * Let the caller deal with it... We - * just need to fudge things so the - * caller can reinit and/or put the - * search context safely. - */ - if (IS_ERR(ctx->mrec)) { - if (PTR_ERR(ctx->mrec) == - -ENOMEM) { - schedule(); - goto retry_map; - } else - old_ctx.ntfs_ino = - old_ctx. - base_ntfs_ino; - } - } - } - /* Update the changed pointers in the saved context. */ - if (ctx->mrec != old_ctx.mrec) { - if (!IS_ERR(ctx->mrec)) - old_ctx.attr = (ATTR_RECORD*)( - (u8*)ctx->mrec + - ((u8*)old_ctx.attr - - (u8*)old_ctx.mrec)); - old_ctx.mrec = ctx->mrec; - } - } - /* Restore the search context to the saved one. */ - *ctx = old_ctx; - /* - * We drop the reference on the page we took earlier. In the - * case that IS_ERR(ctx->mrec) is true this means we might lose - * some changes to the mft record that had been made between - * the last time it was marked dirty/written out and now. This - * at this stage is not a problem as the mapping error is fatal - * enough that the mft record cannot be written out anyway and - * the caller is very likely to shutdown the whole inode - * immediately and mark the volume dirty for chkdsk to pick up - * the pieces anyway. - */ - if (put_this_page) - put_page(put_this_page); - } - return err; -} - -/** - * ntfs_map_runlist - map (a part of) a runlist of an ntfs inode - * @ni: ntfs inode for which to map (part of) a runlist - * @vcn: map runlist part containing this vcn - * - * Map the part of a runlist containing the @vcn of the ntfs inode @ni. - * - * Return 0 on success and -errno on error. There is one special error code - * which is not an error as such. This is -ENOENT. It means that @vcn is out - * of bounds of the runlist. - * - * Locking: - The runlist must be unlocked on entry and is unlocked on return. - * - This function takes the runlist lock for writing and may modify - * the runlist. - */ -int ntfs_map_runlist(ntfs_inode *ni, VCN vcn) -{ - int err = 0; - - down_write(&ni->runlist.lock); - /* Make sure someone else didn't do the work while we were sleeping. */ - if (likely(ntfs_rl_vcn_to_lcn(ni->runlist.rl, vcn) <= - LCN_RL_NOT_MAPPED)) - err = ntfs_map_runlist_nolock(ni, vcn, NULL); - up_write(&ni->runlist.lock); - return err; -} - -/** - * ntfs_attr_vcn_to_lcn_nolock - convert a vcn into a lcn given an ntfs inode - * @ni: ntfs inode of the attribute whose runlist to search - * @vcn: vcn to convert - * @write_locked: true if the runlist is locked for writing - * - * Find the virtual cluster number @vcn in the runlist of the ntfs attribute - * described by the ntfs inode @ni and return the corresponding logical cluster - * number (lcn). - * - * If the @vcn is not mapped yet, the attempt is made to map the attribute - * extent containing the @vcn and the vcn to lcn conversion is retried. - * - * If @write_locked is true the caller has locked the runlist for writing and - * if false for reading. - * - * Since lcns must be >= 0, we use negative return codes with special meaning: - * - * Return code Meaning / Description - * ========================================== - * LCN_HOLE Hole / not allocated on disk. - * LCN_ENOENT There is no such vcn in the runlist, i.e. @vcn is out of bounds. - * LCN_ENOMEM Not enough memory to map runlist. - * LCN_EIO Critical error (runlist/file is corrupt, i/o error, etc). - * - * Locking: - The runlist must be locked on entry and is left locked on return. - * - If @write_locked is 'false', i.e. the runlist is locked for reading, - * the lock may be dropped inside the function so you cannot rely on - * the runlist still being the same when this function returns. - */ -LCN ntfs_attr_vcn_to_lcn_nolock(ntfs_inode *ni, const VCN vcn, - const bool write_locked) -{ - LCN lcn; - unsigned long flags; - bool is_retry = false; - - BUG_ON(!ni); - ntfs_debug("Entering for i_ino 0x%lx, vcn 0x%llx, %s_locked.", - ni->mft_no, (unsigned long long)vcn, - write_locked ? "write" : "read"); - BUG_ON(!NInoNonResident(ni)); - BUG_ON(vcn < 0); - if (!ni->runlist.rl) { - read_lock_irqsave(&ni->size_lock, flags); - if (!ni->allocated_size) { - read_unlock_irqrestore(&ni->size_lock, flags); - return LCN_ENOENT; - } - read_unlock_irqrestore(&ni->size_lock, flags); - } -retry_remap: - /* Convert vcn to lcn. If that fails map the runlist and retry once. */ - lcn = ntfs_rl_vcn_to_lcn(ni->runlist.rl, vcn); - if (likely(lcn >= LCN_HOLE)) { - ntfs_debug("Done, lcn 0x%llx.", (long long)lcn); - return lcn; - } - if (lcn != LCN_RL_NOT_MAPPED) { - if (lcn != LCN_ENOENT) - lcn = LCN_EIO; - } else if (!is_retry) { - int err; - - if (!write_locked) { - up_read(&ni->runlist.lock); - down_write(&ni->runlist.lock); - if (unlikely(ntfs_rl_vcn_to_lcn(ni->runlist.rl, vcn) != - LCN_RL_NOT_MAPPED)) { - up_write(&ni->runlist.lock); - down_read(&ni->runlist.lock); - goto retry_remap; - } - } - err = ntfs_map_runlist_nolock(ni, vcn, NULL); - if (!write_locked) { - up_write(&ni->runlist.lock); - down_read(&ni->runlist.lock); - } - if (likely(!err)) { - is_retry = true; - goto retry_remap; - } - if (err == -ENOENT) - lcn = LCN_ENOENT; - else if (err == -ENOMEM) - lcn = LCN_ENOMEM; - else - lcn = LCN_EIO; - } - if (lcn != LCN_ENOENT) - ntfs_error(ni->vol->sb, "Failed with error code %lli.", - (long long)lcn); - return lcn; -} - -/** - * ntfs_attr_find_vcn_nolock - find a vcn in the runlist of an ntfs inode - * @ni: ntfs inode describing the runlist to search - * @vcn: vcn to find - * @ctx: active attribute search context if present or NULL if not - * - * Find the virtual cluster number @vcn in the runlist described by the ntfs - * inode @ni and return the address of the runlist element containing the @vcn. - * - * If the @vcn is not mapped yet, the attempt is made to map the attribute - * extent containing the @vcn and the vcn to lcn conversion is retried. - * - * If @ctx is specified, it is an active search context of @ni and its base mft - * record. This is needed when ntfs_attr_find_vcn_nolock() encounters unmapped - * runlist fragments and allows their mapping. If you do not have the mft - * record mapped, you can specify @ctx as NULL and ntfs_attr_find_vcn_nolock() - * will perform the necessary mapping and unmapping. - * - * Note, ntfs_attr_find_vcn_nolock() saves the state of @ctx on entry and - * restores it before returning. Thus, @ctx will be left pointing to the same - * attribute on return as on entry. However, the actual pointers in @ctx may - * point to different memory locations on return, so you must remember to reset - * any cached pointers from the @ctx, i.e. after the call to - * ntfs_attr_find_vcn_nolock(), you will probably want to do: - * m = ctx->mrec; - * a = ctx->attr; - * Assuming you cache ctx->attr in a variable @a of type ATTR_RECORD * and that - * you cache ctx->mrec in a variable @m of type MFT_RECORD *. - * Note you need to distinguish between the lcn of the returned runlist element - * being >= 0 and LCN_HOLE. In the later case you have to return zeroes on - * read and allocate clusters on write. - * - * Return the runlist element containing the @vcn on success and - * ERR_PTR(-errno) on error. You need to test the return value with IS_ERR() - * to decide if the return is success or failure and PTR_ERR() to get to the - * error code if IS_ERR() is true. - * - * The possible error return codes are: - * -ENOENT - No such vcn in the runlist, i.e. @vcn is out of bounds. - * -ENOMEM - Not enough memory to map runlist. - * -EIO - Critical error (runlist/file is corrupt, i/o error, etc). - * - * WARNING: If @ctx is supplied, regardless of whether success or failure is - * returned, you need to check IS_ERR(@ctx->mrec) and if 'true' the @ctx - * is no longer valid, i.e. you need to either call - * ntfs_attr_reinit_search_ctx() or ntfs_attr_put_search_ctx() on it. - * In that case PTR_ERR(@ctx->mrec) will give you the error code for - * why the mapping of the old inode failed. - * - * Locking: - The runlist described by @ni must be locked for writing on entry - * and is locked on return. Note the runlist may be modified when - * needed runlist fragments need to be mapped. - * - If @ctx is NULL, the base mft record of @ni must not be mapped on - * entry and it will be left unmapped on return. - * - If @ctx is not NULL, the base mft record must be mapped on entry - * and it will be left mapped on return. - */ -runlist_element *ntfs_attr_find_vcn_nolock(ntfs_inode *ni, const VCN vcn, - ntfs_attr_search_ctx *ctx) -{ - unsigned long flags; - runlist_element *rl; - int err = 0; - bool is_retry = false; - - BUG_ON(!ni); - ntfs_debug("Entering for i_ino 0x%lx, vcn 0x%llx, with%s ctx.", - ni->mft_no, (unsigned long long)vcn, ctx ? "" : "out"); - BUG_ON(!NInoNonResident(ni)); - BUG_ON(vcn < 0); - if (!ni->runlist.rl) { - read_lock_irqsave(&ni->size_lock, flags); - if (!ni->allocated_size) { - read_unlock_irqrestore(&ni->size_lock, flags); - return ERR_PTR(-ENOENT); - } - read_unlock_irqrestore(&ni->size_lock, flags); - } -retry_remap: - rl = ni->runlist.rl; - if (likely(rl && vcn >= rl[0].vcn)) { - while (likely(rl->length)) { - if (unlikely(vcn < rl[1].vcn)) { - if (likely(rl->lcn >= LCN_HOLE)) { - ntfs_debug("Done."); - return rl; - } - break; - } - rl++; - } - if (likely(rl->lcn != LCN_RL_NOT_MAPPED)) { - if (likely(rl->lcn == LCN_ENOENT)) - err = -ENOENT; - else - err = -EIO; - } - } - if (!err && !is_retry) { - /* - * If the search context is invalid we cannot map the unmapped - * region. - */ - if (IS_ERR(ctx->mrec)) - err = PTR_ERR(ctx->mrec); - else { - /* - * The @vcn is in an unmapped region, map the runlist - * and retry. - */ - err = ntfs_map_runlist_nolock(ni, vcn, ctx); - if (likely(!err)) { - is_retry = true; - goto retry_remap; - } - } - if (err == -EINVAL) - err = -EIO; - } else if (!err) - err = -EIO; - if (err != -ENOENT) - ntfs_error(ni->vol->sb, "Failed with error code %i.", err); - return ERR_PTR(err); -} - -/** - * ntfs_attr_find - find (next) attribute in mft record - * @type: attribute type to find - * @name: attribute name to find (optional, i.e. NULL means don't care) - * @name_len: attribute name length (only needed if @name present) - * @ic: IGNORE_CASE or CASE_SENSITIVE (ignored if @name not present) - * @val: attribute value to find (optional, resident attributes only) - * @val_len: attribute value length - * @ctx: search context with mft record and attribute to search from - * - * You should not need to call this function directly. Use ntfs_attr_lookup() - * instead. - * - * ntfs_attr_find() takes a search context @ctx as parameter and searches the - * mft record specified by @ctx->mrec, beginning at @ctx->attr, for an - * attribute of @type, optionally @name and @val. - * - * If the attribute is found, ntfs_attr_find() returns 0 and @ctx->attr will - * point to the found attribute. - * - * If the attribute is not found, ntfs_attr_find() returns -ENOENT and - * @ctx->attr will point to the attribute before which the attribute being - * searched for would need to be inserted if such an action were to be desired. - * - * On actual error, ntfs_attr_find() returns -EIO. In this case @ctx->attr is - * undefined and in particular do not rely on it not changing. - * - * If @ctx->is_first is 'true', the search begins with @ctx->attr itself. If it - * is 'false', the search begins after @ctx->attr. - * - * If @ic is IGNORE_CASE, the @name comparisson is not case sensitive and - * @ctx->ntfs_ino must be set to the ntfs inode to which the mft record - * @ctx->mrec belongs. This is so we can get at the ntfs volume and hence at - * the upcase table. If @ic is CASE_SENSITIVE, the comparison is case - * sensitive. When @name is present, @name_len is the @name length in Unicode - * characters. - * - * If @name is not present (NULL), we assume that the unnamed attribute is - * being searched for. - * - * Finally, the resident attribute value @val is looked for, if present. If - * @val is not present (NULL), @val_len is ignored. - * - * ntfs_attr_find() only searches the specified mft record and it ignores the - * presence of an attribute list attribute (unless it is the one being searched - * for, obviously). If you need to take attribute lists into consideration, - * use ntfs_attr_lookup() instead (see below). This also means that you cannot - * use ntfs_attr_find() to search for extent records of non-resident - * attributes, as extents with lowest_vcn != 0 are usually described by the - * attribute list attribute only. - Note that it is possible that the first - * extent is only in the attribute list while the last extent is in the base - * mft record, so do not rely on being able to find the first extent in the - * base mft record. - * - * Warning: Never use @val when looking for attribute types which can be - * non-resident as this most likely will result in a crash! - */ -static int ntfs_attr_find(const ATTR_TYPE type, const ntfschar *name, - const u32 name_len, const IGNORE_CASE_BOOL ic, - const u8 *val, const u32 val_len, ntfs_attr_search_ctx *ctx) -{ - ATTR_RECORD *a; - ntfs_volume *vol = ctx->ntfs_ino->vol; - ntfschar *upcase = vol->upcase; - u32 upcase_len = vol->upcase_len; - - /* - * Iterate over attributes in mft record starting at @ctx->attr, or the - * attribute following that, if @ctx->is_first is 'true'. - */ - if (ctx->is_first) { - a = ctx->attr; - ctx->is_first = false; - } else - a = (ATTR_RECORD*)((u8*)ctx->attr + - le32_to_cpu(ctx->attr->length)); - for (;; a = (ATTR_RECORD*)((u8*)a + le32_to_cpu(a->length))) { - u8 *mrec_end = (u8 *)ctx->mrec + - le32_to_cpu(ctx->mrec->bytes_allocated); - u8 *name_end; - - /* check whether ATTR_RECORD wrap */ - if ((u8 *)a < (u8 *)ctx->mrec) - break; - - /* check whether Attribute Record Header is within bounds */ - if ((u8 *)a > mrec_end || - (u8 *)a + sizeof(ATTR_RECORD) > mrec_end) - break; - - /* check whether ATTR_RECORD's name is within bounds */ - name_end = (u8 *)a + le16_to_cpu(a->name_offset) + - a->name_length * sizeof(ntfschar); - if (name_end > mrec_end) - break; - - ctx->attr = a; - if (unlikely(le32_to_cpu(a->type) > le32_to_cpu(type) || - a->type == AT_END)) - return -ENOENT; - if (unlikely(!a->length)) - break; - - /* check whether ATTR_RECORD's length wrap */ - if ((u8 *)a + le32_to_cpu(a->length) < (u8 *)a) - break; - /* check whether ATTR_RECORD's length is within bounds */ - if ((u8 *)a + le32_to_cpu(a->length) > mrec_end) - break; - - if (a->type != type) - continue; - /* - * If @name is present, compare the two names. If @name is - * missing, assume we want an unnamed attribute. - */ - if (!name) { - /* The search failed if the found attribute is named. */ - if (a->name_length) - return -ENOENT; - } else if (!ntfs_are_names_equal(name, name_len, - (ntfschar*)((u8*)a + le16_to_cpu(a->name_offset)), - a->name_length, ic, upcase, upcase_len)) { - register int rc; - - rc = ntfs_collate_names(name, name_len, - (ntfschar*)((u8*)a + - le16_to_cpu(a->name_offset)), - a->name_length, 1, IGNORE_CASE, - upcase, upcase_len); - /* - * If @name collates before a->name, there is no - * matching attribute. - */ - if (rc == -1) - return -ENOENT; - /* If the strings are not equal, continue search. */ - if (rc) - continue; - rc = ntfs_collate_names(name, name_len, - (ntfschar*)((u8*)a + - le16_to_cpu(a->name_offset)), - a->name_length, 1, CASE_SENSITIVE, - upcase, upcase_len); - if (rc == -1) - return -ENOENT; - if (rc) - continue; - } - /* - * The names match or @name not present and attribute is - * unnamed. If no @val specified, we have found the attribute - * and are done. - */ - if (!val) - return 0; - /* @val is present; compare values. */ - else { - register int rc; - - rc = memcmp(val, (u8*)a + le16_to_cpu( - a->data.resident.value_offset), - min_t(u32, val_len, le32_to_cpu( - a->data.resident.value_length))); - /* - * If @val collates before the current attribute's - * value, there is no matching attribute. - */ - if (!rc) { - register u32 avl; - - avl = le32_to_cpu( - a->data.resident.value_length); - if (val_len == avl) - return 0; - if (val_len < avl) - return -ENOENT; - } else if (rc < 0) - return -ENOENT; - } - } - ntfs_error(vol->sb, "Inode is corrupt. Run chkdsk."); - NVolSetErrors(vol); - return -EIO; -} - -/** - * load_attribute_list - load an attribute list into memory - * @vol: ntfs volume from which to read - * @runlist: runlist of the attribute list - * @al_start: destination buffer - * @size: size of the destination buffer in bytes - * @initialized_size: initialized size of the attribute list - * - * Walk the runlist @runlist and load all clusters from it copying them into - * the linear buffer @al. The maximum number of bytes copied to @al is @size - * bytes. Note, @size does not need to be a multiple of the cluster size. If - * @initialized_size is less than @size, the region in @al between - * @initialized_size and @size will be zeroed and not read from disk. - * - * Return 0 on success or -errno on error. - */ -int load_attribute_list(ntfs_volume *vol, runlist *runlist, u8 *al_start, - const s64 size, const s64 initialized_size) -{ - LCN lcn; - u8 *al = al_start; - u8 *al_end = al + initialized_size; - runlist_element *rl; - struct buffer_head *bh; - struct super_block *sb; - unsigned long block_size; - unsigned long block, max_block; - int err = 0; - unsigned char block_size_bits; - - ntfs_debug("Entering."); - if (!vol || !runlist || !al || size <= 0 || initialized_size < 0 || - initialized_size > size) - return -EINVAL; - if (!initialized_size) { - memset(al, 0, size); - return 0; - } - sb = vol->sb; - block_size = sb->s_blocksize; - block_size_bits = sb->s_blocksize_bits; - down_read(&runlist->lock); - rl = runlist->rl; - if (!rl) { - ntfs_error(sb, "Cannot read attribute list since runlist is " - "missing."); - goto err_out; - } - /* Read all clusters specified by the runlist one run at a time. */ - while (rl->length) { - lcn = ntfs_rl_vcn_to_lcn(rl, rl->vcn); - ntfs_debug("Reading vcn = 0x%llx, lcn = 0x%llx.", - (unsigned long long)rl->vcn, - (unsigned long long)lcn); - /* The attribute list cannot be sparse. */ - if (lcn < 0) { - ntfs_error(sb, "ntfs_rl_vcn_to_lcn() failed. Cannot " - "read attribute list."); - goto err_out; - } - block = lcn << vol->cluster_size_bits >> block_size_bits; - /* Read the run from device in chunks of block_size bytes. */ - max_block = block + (rl->length << vol->cluster_size_bits >> - block_size_bits); - ntfs_debug("max_block = 0x%lx.", max_block); - do { - ntfs_debug("Reading block = 0x%lx.", block); - bh = sb_bread(sb, block); - if (!bh) { - ntfs_error(sb, "sb_bread() failed. Cannot " - "read attribute list."); - goto err_out; - } - if (al + block_size >= al_end) - goto do_final; - memcpy(al, bh->b_data, block_size); - brelse(bh); - al += block_size; - } while (++block < max_block); - rl++; - } - if (initialized_size < size) { -initialize: - memset(al_start + initialized_size, 0, size - initialized_size); - } -done: - up_read(&runlist->lock); - return err; -do_final: - if (al < al_end) { - /* - * Partial block. - * - * Note: The attribute list can be smaller than its allocation - * by multiple clusters. This has been encountered by at least - * two people running Windows XP, thus we cannot do any - * truncation sanity checking here. (AIA) - */ - memcpy(al, bh->b_data, al_end - al); - brelse(bh); - if (initialized_size < size) - goto initialize; - goto done; - } - brelse(bh); - /* Real overflow! */ - ntfs_error(sb, "Attribute list buffer overflow. Read attribute list " - "is truncated."); -err_out: - err = -EIO; - goto done; -} - -/** - * ntfs_external_attr_find - find an attribute in the attribute list of an inode - * @type: attribute type to find - * @name: attribute name to find (optional, i.e. NULL means don't care) - * @name_len: attribute name length (only needed if @name present) - * @ic: IGNORE_CASE or CASE_SENSITIVE (ignored if @name not present) - * @lowest_vcn: lowest vcn to find (optional, non-resident attributes only) - * @val: attribute value to find (optional, resident attributes only) - * @val_len: attribute value length - * @ctx: search context with mft record and attribute to search from - * - * You should not need to call this function directly. Use ntfs_attr_lookup() - * instead. - * - * Find an attribute by searching the attribute list for the corresponding - * attribute list entry. Having found the entry, map the mft record if the - * attribute is in a different mft record/inode, ntfs_attr_find() the attribute - * in there and return it. - * - * On first search @ctx->ntfs_ino must be the base mft record and @ctx must - * have been obtained from a call to ntfs_attr_get_search_ctx(). On subsequent - * calls @ctx->ntfs_ino can be any extent inode, too (@ctx->base_ntfs_ino is - * then the base inode). - * - * After finishing with the attribute/mft record you need to call - * ntfs_attr_put_search_ctx() to cleanup the search context (unmapping any - * mapped inodes, etc). - * - * If the attribute is found, ntfs_external_attr_find() returns 0 and - * @ctx->attr will point to the found attribute. @ctx->mrec will point to the - * mft record in which @ctx->attr is located and @ctx->al_entry will point to - * the attribute list entry for the attribute. - * - * If the attribute is not found, ntfs_external_attr_find() returns -ENOENT and - * @ctx->attr will point to the attribute in the base mft record before which - * the attribute being searched for would need to be inserted if such an action - * were to be desired. @ctx->mrec will point to the mft record in which - * @ctx->attr is located and @ctx->al_entry will point to the attribute list - * entry of the attribute before which the attribute being searched for would - * need to be inserted if such an action were to be desired. - * - * Thus to insert the not found attribute, one wants to add the attribute to - * @ctx->mrec (the base mft record) and if there is not enough space, the - * attribute should be placed in a newly allocated extent mft record. The - * attribute list entry for the inserted attribute should be inserted in the - * attribute list attribute at @ctx->al_entry. - * - * On actual error, ntfs_external_attr_find() returns -EIO. In this case - * @ctx->attr is undefined and in particular do not rely on it not changing. - */ -static int ntfs_external_attr_find(const ATTR_TYPE type, - const ntfschar *name, const u32 name_len, - const IGNORE_CASE_BOOL ic, const VCN lowest_vcn, - const u8 *val, const u32 val_len, ntfs_attr_search_ctx *ctx) -{ - ntfs_inode *base_ni, *ni; - ntfs_volume *vol; - ATTR_LIST_ENTRY *al_entry, *next_al_entry; - u8 *al_start, *al_end; - ATTR_RECORD *a; - ntfschar *al_name; - u32 al_name_len; - int err = 0; - static const char *es = " Unmount and run chkdsk."; - - ni = ctx->ntfs_ino; - base_ni = ctx->base_ntfs_ino; - ntfs_debug("Entering for inode 0x%lx, type 0x%x.", ni->mft_no, type); - if (!base_ni) { - /* First call happens with the base mft record. */ - base_ni = ctx->base_ntfs_ino = ctx->ntfs_ino; - ctx->base_mrec = ctx->mrec; - } - if (ni == base_ni) - ctx->base_attr = ctx->attr; - if (type == AT_END) - goto not_found; - vol = base_ni->vol; - al_start = base_ni->attr_list; - al_end = al_start + base_ni->attr_list_size; - if (!ctx->al_entry) - ctx->al_entry = (ATTR_LIST_ENTRY*)al_start; - /* - * Iterate over entries in attribute list starting at @ctx->al_entry, - * or the entry following that, if @ctx->is_first is 'true'. - */ - if (ctx->is_first) { - al_entry = ctx->al_entry; - ctx->is_first = false; - } else - al_entry = (ATTR_LIST_ENTRY*)((u8*)ctx->al_entry + - le16_to_cpu(ctx->al_entry->length)); - for (;; al_entry = next_al_entry) { - /* Out of bounds check. */ - if ((u8*)al_entry < base_ni->attr_list || - (u8*)al_entry > al_end) - break; /* Inode is corrupt. */ - ctx->al_entry = al_entry; - /* Catch the end of the attribute list. */ - if ((u8*)al_entry == al_end) - goto not_found; - if (!al_entry->length) - break; - if ((u8*)al_entry + 6 > al_end || (u8*)al_entry + - le16_to_cpu(al_entry->length) > al_end) - break; - next_al_entry = (ATTR_LIST_ENTRY*)((u8*)al_entry + - le16_to_cpu(al_entry->length)); - if (le32_to_cpu(al_entry->type) > le32_to_cpu(type)) - goto not_found; - if (type != al_entry->type) - continue; - /* - * If @name is present, compare the two names. If @name is - * missing, assume we want an unnamed attribute. - */ - al_name_len = al_entry->name_length; - al_name = (ntfschar*)((u8*)al_entry + al_entry->name_offset); - if (!name) { - if (al_name_len) - goto not_found; - } else if (!ntfs_are_names_equal(al_name, al_name_len, name, - name_len, ic, vol->upcase, vol->upcase_len)) { - register int rc; - - rc = ntfs_collate_names(name, name_len, al_name, - al_name_len, 1, IGNORE_CASE, - vol->upcase, vol->upcase_len); - /* - * If @name collates before al_name, there is no - * matching attribute. - */ - if (rc == -1) - goto not_found; - /* If the strings are not equal, continue search. */ - if (rc) - continue; - /* - * FIXME: Reverse engineering showed 0, IGNORE_CASE but - * that is inconsistent with ntfs_attr_find(). The - * subsequent rc checks were also different. Perhaps I - * made a mistake in one of the two. Need to recheck - * which is correct or at least see what is going on... - * (AIA) - */ - rc = ntfs_collate_names(name, name_len, al_name, - al_name_len, 1, CASE_SENSITIVE, - vol->upcase, vol->upcase_len); - if (rc == -1) - goto not_found; - if (rc) - continue; - } - /* - * The names match or @name not present and attribute is - * unnamed. Now check @lowest_vcn. Continue search if the - * next attribute list entry still fits @lowest_vcn. Otherwise - * we have reached the right one or the search has failed. - */ - if (lowest_vcn && (u8*)next_al_entry >= al_start && - (u8*)next_al_entry + 6 < al_end && - (u8*)next_al_entry + le16_to_cpu( - next_al_entry->length) <= al_end && - sle64_to_cpu(next_al_entry->lowest_vcn) <= - lowest_vcn && - next_al_entry->type == al_entry->type && - next_al_entry->name_length == al_name_len && - ntfs_are_names_equal((ntfschar*)((u8*) - next_al_entry + - next_al_entry->name_offset), - next_al_entry->name_length, - al_name, al_name_len, CASE_SENSITIVE, - vol->upcase, vol->upcase_len)) - continue; - if (MREF_LE(al_entry->mft_reference) == ni->mft_no) { - if (MSEQNO_LE(al_entry->mft_reference) != ni->seq_no) { - ntfs_error(vol->sb, "Found stale mft " - "reference in attribute list " - "of base inode 0x%lx.%s", - base_ni->mft_no, es); - err = -EIO; - break; - } - } else { /* Mft references do not match. */ - /* If there is a mapped record unmap it first. */ - if (ni != base_ni) - unmap_extent_mft_record(ni); - /* Do we want the base record back? */ - if (MREF_LE(al_entry->mft_reference) == - base_ni->mft_no) { - ni = ctx->ntfs_ino = base_ni; - ctx->mrec = ctx->base_mrec; - } else { - /* We want an extent record. */ - ctx->mrec = map_extent_mft_record(base_ni, - le64_to_cpu( - al_entry->mft_reference), &ni); - if (IS_ERR(ctx->mrec)) { - ntfs_error(vol->sb, "Failed to map " - "extent mft record " - "0x%lx of base inode " - "0x%lx.%s", - MREF_LE(al_entry-> - mft_reference), - base_ni->mft_no, es); - err = PTR_ERR(ctx->mrec); - if (err == -ENOENT) - err = -EIO; - /* Cause @ctx to be sanitized below. */ - ni = NULL; - break; - } - ctx->ntfs_ino = ni; - } - ctx->attr = (ATTR_RECORD*)((u8*)ctx->mrec + - le16_to_cpu(ctx->mrec->attrs_offset)); - } - /* - * ctx->vfs_ino, ctx->mrec, and ctx->attr now point to the - * mft record containing the attribute represented by the - * current al_entry. - */ - /* - * We could call into ntfs_attr_find() to find the right - * attribute in this mft record but this would be less - * efficient and not quite accurate as ntfs_attr_find() ignores - * the attribute instance numbers for example which become - * important when one plays with attribute lists. Also, - * because a proper match has been found in the attribute list - * entry above, the comparison can now be optimized. So it is - * worth re-implementing a simplified ntfs_attr_find() here. - */ - a = ctx->attr; - /* - * Use a manual loop so we can still use break and continue - * with the same meanings as above. - */ -do_next_attr_loop: - if ((u8*)a < (u8*)ctx->mrec || (u8*)a > (u8*)ctx->mrec + - le32_to_cpu(ctx->mrec->bytes_allocated)) - break; - if (a->type == AT_END) - break; - if (!a->length) - break; - if (al_entry->instance != a->instance) - goto do_next_attr; - /* - * If the type and/or the name are mismatched between the - * attribute list entry and the attribute record, there is - * corruption so we break and return error EIO. - */ - if (al_entry->type != a->type) - break; - if (!ntfs_are_names_equal((ntfschar*)((u8*)a + - le16_to_cpu(a->name_offset)), a->name_length, - al_name, al_name_len, CASE_SENSITIVE, - vol->upcase, vol->upcase_len)) - break; - ctx->attr = a; - /* - * If no @val specified or @val specified and it matches, we - * have found it! - */ - if (!val || (!a->non_resident && le32_to_cpu( - a->data.resident.value_length) == val_len && - !memcmp((u8*)a + - le16_to_cpu(a->data.resident.value_offset), - val, val_len))) { - ntfs_debug("Done, found."); - return 0; - } -do_next_attr: - /* Proceed to the next attribute in the current mft record. */ - a = (ATTR_RECORD*)((u8*)a + le32_to_cpu(a->length)); - goto do_next_attr_loop; - } - if (!err) { - ntfs_error(vol->sb, "Base inode 0x%lx contains corrupt " - "attribute list attribute.%s", base_ni->mft_no, - es); - err = -EIO; - } - if (ni != base_ni) { - if (ni) - unmap_extent_mft_record(ni); - ctx->ntfs_ino = base_ni; - ctx->mrec = ctx->base_mrec; - ctx->attr = ctx->base_attr; - } - if (err != -ENOMEM) - NVolSetErrors(vol); - return err; -not_found: - /* - * If we were looking for AT_END, we reset the search context @ctx and - * use ntfs_attr_find() to seek to the end of the base mft record. - */ - if (type == AT_END) { - ntfs_attr_reinit_search_ctx(ctx); - return ntfs_attr_find(AT_END, name, name_len, ic, val, val_len, - ctx); - } - /* - * The attribute was not found. Before we return, we want to ensure - * @ctx->mrec and @ctx->attr indicate the position at which the - * attribute should be inserted in the base mft record. Since we also - * want to preserve @ctx->al_entry we cannot reinitialize the search - * context using ntfs_attr_reinit_search_ctx() as this would set - * @ctx->al_entry to NULL. Thus we do the necessary bits manually (see - * ntfs_attr_init_search_ctx() below). Note, we _only_ preserve - * @ctx->al_entry as the remaining fields (base_*) are identical to - * their non base_ counterparts and we cannot set @ctx->base_attr - * correctly yet as we do not know what @ctx->attr will be set to by - * the call to ntfs_attr_find() below. - */ - if (ni != base_ni) - unmap_extent_mft_record(ni); - ctx->mrec = ctx->base_mrec; - ctx->attr = (ATTR_RECORD*)((u8*)ctx->mrec + - le16_to_cpu(ctx->mrec->attrs_offset)); - ctx->is_first = true; - ctx->ntfs_ino = base_ni; - ctx->base_ntfs_ino = NULL; - ctx->base_mrec = NULL; - ctx->base_attr = NULL; - /* - * In case there are multiple matches in the base mft record, need to - * keep enumerating until we get an attribute not found response (or - * another error), otherwise we would keep returning the same attribute - * over and over again and all programs using us for enumeration would - * lock up in a tight loop. - */ - do { - err = ntfs_attr_find(type, name, name_len, ic, val, val_len, - ctx); - } while (!err); - ntfs_debug("Done, not found."); - return err; -} - -/** - * ntfs_attr_lookup - find an attribute in an ntfs inode - * @type: attribute type to find - * @name: attribute name to find (optional, i.e. NULL means don't care) - * @name_len: attribute name length (only needed if @name present) - * @ic: IGNORE_CASE or CASE_SENSITIVE (ignored if @name not present) - * @lowest_vcn: lowest vcn to find (optional, non-resident attributes only) - * @val: attribute value to find (optional, resident attributes only) - * @val_len: attribute value length - * @ctx: search context with mft record and attribute to search from - * - * Find an attribute in an ntfs inode. On first search @ctx->ntfs_ino must - * be the base mft record and @ctx must have been obtained from a call to - * ntfs_attr_get_search_ctx(). - * - * This function transparently handles attribute lists and @ctx is used to - * continue searches where they were left off at. - * - * After finishing with the attribute/mft record you need to call - * ntfs_attr_put_search_ctx() to cleanup the search context (unmapping any - * mapped inodes, etc). - * - * Return 0 if the search was successful and -errno if not. - * - * When 0, @ctx->attr is the found attribute and it is in mft record - * @ctx->mrec. If an attribute list attribute is present, @ctx->al_entry is - * the attribute list entry of the found attribute. - * - * When -ENOENT, @ctx->attr is the attribute which collates just after the - * attribute being searched for, i.e. if one wants to add the attribute to the - * mft record this is the correct place to insert it into. If an attribute - * list attribute is present, @ctx->al_entry is the attribute list entry which - * collates just after the attribute list entry of the attribute being searched - * for, i.e. if one wants to add the attribute to the mft record this is the - * correct place to insert its attribute list entry into. - * - * When -errno != -ENOENT, an error occurred during the lookup. @ctx->attr is - * then undefined and in particular you should not rely on it not changing. - */ -int ntfs_attr_lookup(const ATTR_TYPE type, const ntfschar *name, - const u32 name_len, const IGNORE_CASE_BOOL ic, - const VCN lowest_vcn, const u8 *val, const u32 val_len, - ntfs_attr_search_ctx *ctx) -{ - ntfs_inode *base_ni; - - ntfs_debug("Entering."); - BUG_ON(IS_ERR(ctx->mrec)); - if (ctx->base_ntfs_ino) - base_ni = ctx->base_ntfs_ino; - else - base_ni = ctx->ntfs_ino; - /* Sanity check, just for debugging really. */ - BUG_ON(!base_ni); - if (!NInoAttrList(base_ni) || type == AT_ATTRIBUTE_LIST) - return ntfs_attr_find(type, name, name_len, ic, val, val_len, - ctx); - return ntfs_external_attr_find(type, name, name_len, ic, lowest_vcn, - val, val_len, ctx); -} - -/** - * ntfs_attr_init_search_ctx - initialize an attribute search context - * @ctx: attribute search context to initialize - * @ni: ntfs inode with which to initialize the search context - * @mrec: mft record with which to initialize the search context - * - * Initialize the attribute search context @ctx with @ni and @mrec. - */ -static inline void ntfs_attr_init_search_ctx(ntfs_attr_search_ctx *ctx, - ntfs_inode *ni, MFT_RECORD *mrec) -{ - *ctx = (ntfs_attr_search_ctx) { - .mrec = mrec, - /* Sanity checks are performed elsewhere. */ - .attr = (ATTR_RECORD*)((u8*)mrec + - le16_to_cpu(mrec->attrs_offset)), - .is_first = true, - .ntfs_ino = ni, - }; -} - -/** - * ntfs_attr_reinit_search_ctx - reinitialize an attribute search context - * @ctx: attribute search context to reinitialize - * - * Reinitialize the attribute search context @ctx, unmapping an associated - * extent mft record if present, and initialize the search context again. - * - * This is used when a search for a new attribute is being started to reset - * the search context to the beginning. - */ -void ntfs_attr_reinit_search_ctx(ntfs_attr_search_ctx *ctx) -{ - if (likely(!ctx->base_ntfs_ino)) { - /* No attribute list. */ - ctx->is_first = true; - /* Sanity checks are performed elsewhere. */ - ctx->attr = (ATTR_RECORD*)((u8*)ctx->mrec + - le16_to_cpu(ctx->mrec->attrs_offset)); - /* - * This needs resetting due to ntfs_external_attr_find() which - * can leave it set despite having zeroed ctx->base_ntfs_ino. - */ - ctx->al_entry = NULL; - return; - } /* Attribute list. */ - if (ctx->ntfs_ino != ctx->base_ntfs_ino) - unmap_extent_mft_record(ctx->ntfs_ino); - ntfs_attr_init_search_ctx(ctx, ctx->base_ntfs_ino, ctx->base_mrec); - return; -} - -/** - * ntfs_attr_get_search_ctx - allocate/initialize a new attribute search context - * @ni: ntfs inode with which to initialize the search context - * @mrec: mft record with which to initialize the search context - * - * Allocate a new attribute search context, initialize it with @ni and @mrec, - * and return it. Return NULL if allocation failed. - */ -ntfs_attr_search_ctx *ntfs_attr_get_search_ctx(ntfs_inode *ni, MFT_RECORD *mrec) -{ - ntfs_attr_search_ctx *ctx; - - ctx = kmem_cache_alloc(ntfs_attr_ctx_cache, GFP_NOFS); - if (ctx) - ntfs_attr_init_search_ctx(ctx, ni, mrec); - return ctx; -} - -/** - * ntfs_attr_put_search_ctx - release an attribute search context - * @ctx: attribute search context to free - * - * Release the attribute search context @ctx, unmapping an associated extent - * mft record if present. - */ -void ntfs_attr_put_search_ctx(ntfs_attr_search_ctx *ctx) -{ - if (ctx->base_ntfs_ino && ctx->ntfs_ino != ctx->base_ntfs_ino) - unmap_extent_mft_record(ctx->ntfs_ino); - kmem_cache_free(ntfs_attr_ctx_cache, ctx); - return; -} - -#ifdef NTFS_RW - -/** - * ntfs_attr_find_in_attrdef - find an attribute in the $AttrDef system file - * @vol: ntfs volume to which the attribute belongs - * @type: attribute type which to find - * - * Search for the attribute definition record corresponding to the attribute - * @type in the $AttrDef system file. - * - * Return the attribute type definition record if found and NULL if not found. - */ -static ATTR_DEF *ntfs_attr_find_in_attrdef(const ntfs_volume *vol, - const ATTR_TYPE type) -{ - ATTR_DEF *ad; - - BUG_ON(!vol->attrdef); - BUG_ON(!type); - for (ad = vol->attrdef; (u8*)ad - (u8*)vol->attrdef < - vol->attrdef_size && ad->type; ++ad) { - /* We have not found it yet, carry on searching. */ - if (likely(le32_to_cpu(ad->type) < le32_to_cpu(type))) - continue; - /* We found the attribute; return it. */ - if (likely(ad->type == type)) - return ad; - /* We have gone too far already. No point in continuing. */ - break; - } - /* Attribute not found. */ - ntfs_debug("Attribute type 0x%x not found in $AttrDef.", - le32_to_cpu(type)); - return NULL; -} - -/** - * ntfs_attr_size_bounds_check - check a size of an attribute type for validity - * @vol: ntfs volume to which the attribute belongs - * @type: attribute type which to check - * @size: size which to check - * - * Check whether the @size in bytes is valid for an attribute of @type on the - * ntfs volume @vol. This information is obtained from $AttrDef system file. - * - * Return 0 if valid, -ERANGE if not valid, or -ENOENT if the attribute is not - * listed in $AttrDef. - */ -int ntfs_attr_size_bounds_check(const ntfs_volume *vol, const ATTR_TYPE type, - const s64 size) -{ - ATTR_DEF *ad; - - BUG_ON(size < 0); - /* - * $ATTRIBUTE_LIST has a maximum size of 256kiB, but this is not - * listed in $AttrDef. - */ - if (unlikely(type == AT_ATTRIBUTE_LIST && size > 256 * 1024)) - return -ERANGE; - /* Get the $AttrDef entry for the attribute @type. */ - ad = ntfs_attr_find_in_attrdef(vol, type); - if (unlikely(!ad)) - return -ENOENT; - /* Do the bounds check. */ - if (((sle64_to_cpu(ad->min_size) > 0) && - size < sle64_to_cpu(ad->min_size)) || - ((sle64_to_cpu(ad->max_size) > 0) && size > - sle64_to_cpu(ad->max_size))) - return -ERANGE; - return 0; -} - -/** - * ntfs_attr_can_be_non_resident - check if an attribute can be non-resident - * @vol: ntfs volume to which the attribute belongs - * @type: attribute type which to check - * - * Check whether the attribute of @type on the ntfs volume @vol is allowed to - * be non-resident. This information is obtained from $AttrDef system file. - * - * Return 0 if the attribute is allowed to be non-resident, -EPERM if not, and - * -ENOENT if the attribute is not listed in $AttrDef. - */ -int ntfs_attr_can_be_non_resident(const ntfs_volume *vol, const ATTR_TYPE type) -{ - ATTR_DEF *ad; - - /* Find the attribute definition record in $AttrDef. */ - ad = ntfs_attr_find_in_attrdef(vol, type); - if (unlikely(!ad)) - return -ENOENT; - /* Check the flags and return the result. */ - if (ad->flags & ATTR_DEF_RESIDENT) - return -EPERM; - return 0; -} - -/** - * ntfs_attr_can_be_resident - check if an attribute can be resident - * @vol: ntfs volume to which the attribute belongs - * @type: attribute type which to check - * - * Check whether the attribute of @type on the ntfs volume @vol is allowed to - * be resident. This information is derived from our ntfs knowledge and may - * not be completely accurate, especially when user defined attributes are - * present. Basically we allow everything to be resident except for index - * allocation and $EA attributes. - * - * Return 0 if the attribute is allowed to be non-resident and -EPERM if not. - * - * Warning: In the system file $MFT the attribute $Bitmap must be non-resident - * otherwise windows will not boot (blue screen of death)! We cannot - * check for this here as we do not know which inode's $Bitmap is - * being asked about so the caller needs to special case this. - */ -int ntfs_attr_can_be_resident(const ntfs_volume *vol, const ATTR_TYPE type) -{ - if (type == AT_INDEX_ALLOCATION) - return -EPERM; - return 0; -} - -/** - * ntfs_attr_record_resize - resize an attribute record - * @m: mft record containing attribute record - * @a: attribute record to resize - * @new_size: new size in bytes to which to resize the attribute record @a - * - * Resize the attribute record @a, i.e. the resident part of the attribute, in - * the mft record @m to @new_size bytes. - * - * Return 0 on success and -errno on error. The following error codes are - * defined: - * -ENOSPC - Not enough space in the mft record @m to perform the resize. - * - * Note: On error, no modifications have been performed whatsoever. - * - * Warning: If you make a record smaller without having copied all the data you - * are interested in the data may be overwritten. - */ -int ntfs_attr_record_resize(MFT_RECORD *m, ATTR_RECORD *a, u32 new_size) -{ - ntfs_debug("Entering for new_size %u.", new_size); - /* Align to 8 bytes if it is not already done. */ - if (new_size & 7) - new_size = (new_size + 7) & ~7; - /* If the actual attribute length has changed, move things around. */ - if (new_size != le32_to_cpu(a->length)) { - u32 new_muse = le32_to_cpu(m->bytes_in_use) - - le32_to_cpu(a->length) + new_size; - /* Not enough space in this mft record. */ - if (new_muse > le32_to_cpu(m->bytes_allocated)) - return -ENOSPC; - /* Move attributes following @a to their new location. */ - memmove((u8*)a + new_size, (u8*)a + le32_to_cpu(a->length), - le32_to_cpu(m->bytes_in_use) - ((u8*)a - - (u8*)m) - le32_to_cpu(a->length)); - /* Adjust @m to reflect the change in used space. */ - m->bytes_in_use = cpu_to_le32(new_muse); - /* Adjust @a to reflect the new size. */ - if (new_size >= offsetof(ATTR_REC, length) + sizeof(a->length)) - a->length = cpu_to_le32(new_size); - } - return 0; -} - -/** - * ntfs_resident_attr_value_resize - resize the value of a resident attribute - * @m: mft record containing attribute record - * @a: attribute record whose value to resize - * @new_size: new size in bytes to which to resize the attribute value of @a - * - * Resize the value of the attribute @a in the mft record @m to @new_size bytes. - * If the value is made bigger, the newly allocated space is cleared. - * - * Return 0 on success and -errno on error. The following error codes are - * defined: - * -ENOSPC - Not enough space in the mft record @m to perform the resize. - * - * Note: On error, no modifications have been performed whatsoever. - * - * Warning: If you make a record smaller without having copied all the data you - * are interested in the data may be overwritten. - */ -int ntfs_resident_attr_value_resize(MFT_RECORD *m, ATTR_RECORD *a, - const u32 new_size) -{ - u32 old_size; - - /* Resize the resident part of the attribute record. */ - if (ntfs_attr_record_resize(m, a, - le16_to_cpu(a->data.resident.value_offset) + new_size)) - return -ENOSPC; - /* - * The resize succeeded! If we made the attribute value bigger, clear - * the area between the old size and @new_size. - */ - old_size = le32_to_cpu(a->data.resident.value_length); - if (new_size > old_size) - memset((u8*)a + le16_to_cpu(a->data.resident.value_offset) + - old_size, 0, new_size - old_size); - /* Finally update the length of the attribute value. */ - a->data.resident.value_length = cpu_to_le32(new_size); - return 0; -} - -/** - * ntfs_attr_make_non_resident - convert a resident to a non-resident attribute - * @ni: ntfs inode describing the attribute to convert - * @data_size: size of the resident data to copy to the non-resident attribute - * - * Convert the resident ntfs attribute described by the ntfs inode @ni to a - * non-resident one. - * - * @data_size must be equal to the attribute value size. This is needed since - * we need to know the size before we can map the mft record and our callers - * always know it. The reason we cannot simply read the size from the vfs - * inode i_size is that this is not necessarily uptodate. This happens when - * ntfs_attr_make_non_resident() is called in the ->truncate call path(s). - * - * Return 0 on success and -errno on error. The following error return codes - * are defined: - * -EPERM - The attribute is not allowed to be non-resident. - * -ENOMEM - Not enough memory. - * -ENOSPC - Not enough disk space. - * -EINVAL - Attribute not defined on the volume. - * -EIO - I/o error or other error. - * Note that -ENOSPC is also returned in the case that there is not enough - * space in the mft record to do the conversion. This can happen when the mft - * record is already very full. The caller is responsible for trying to make - * space in the mft record and trying again. FIXME: Do we need a separate - * error return code for this kind of -ENOSPC or is it always worth trying - * again in case the attribute may then fit in a resident state so no need to - * make it non-resident at all? Ho-hum... (AIA) - * - * NOTE to self: No changes in the attribute list are required to move from - * a resident to a non-resident attribute. - * - * Locking: - The caller must hold i_mutex on the inode. - */ -int ntfs_attr_make_non_resident(ntfs_inode *ni, const u32 data_size) -{ - s64 new_size; - struct inode *vi = VFS_I(ni); - ntfs_volume *vol = ni->vol; - ntfs_inode *base_ni; - MFT_RECORD *m; - ATTR_RECORD *a; - ntfs_attr_search_ctx *ctx; - struct page *page; - runlist_element *rl; - u8 *kaddr; - unsigned long flags; - int mp_size, mp_ofs, name_ofs, arec_size, err, err2; - u32 attr_size; - u8 old_res_attr_flags; - - /* Check that the attribute is allowed to be non-resident. */ - err = ntfs_attr_can_be_non_resident(vol, ni->type); - if (unlikely(err)) { - if (err == -EPERM) - ntfs_debug("Attribute is not allowed to be " - "non-resident."); - else - ntfs_debug("Attribute not defined on the NTFS " - "volume!"); - return err; - } - /* - * FIXME: Compressed and encrypted attributes are not supported when - * writing and we should never have gotten here for them. - */ - BUG_ON(NInoCompressed(ni)); - BUG_ON(NInoEncrypted(ni)); - /* - * The size needs to be aligned to a cluster boundary for allocation - * purposes. - */ - new_size = (data_size + vol->cluster_size - 1) & - ~(vol->cluster_size - 1); - if (new_size > 0) { - /* - * Will need the page later and since the page lock nests - * outside all ntfs locks, we need to get the page now. - */ - page = find_or_create_page(vi->i_mapping, 0, - mapping_gfp_mask(vi->i_mapping)); - if (unlikely(!page)) - return -ENOMEM; - /* Start by allocating clusters to hold the attribute value. */ - rl = ntfs_cluster_alloc(vol, 0, new_size >> - vol->cluster_size_bits, -1, DATA_ZONE, true); - if (IS_ERR(rl)) { - err = PTR_ERR(rl); - ntfs_debug("Failed to allocate cluster%s, error code " - "%i.", (new_size >> - vol->cluster_size_bits) > 1 ? "s" : "", - err); - goto page_err_out; - } - } else { - rl = NULL; - page = NULL; - } - /* Determine the size of the mapping pairs array. */ - mp_size = ntfs_get_size_for_mapping_pairs(vol, rl, 0, -1); - if (unlikely(mp_size < 0)) { - err = mp_size; - ntfs_debug("Failed to get size for mapping pairs array, error " - "code %i.", err); - goto rl_err_out; - } - down_write(&ni->runlist.lock); - if (!NInoAttr(ni)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - m = NULL; - ctx = NULL; - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - err = -EIO; - goto err_out; - } - m = ctx->mrec; - a = ctx->attr; - BUG_ON(NInoNonResident(ni)); - BUG_ON(a->non_resident); - /* - * Calculate new offsets for the name and the mapping pairs array. - */ - if (NInoSparse(ni) || NInoCompressed(ni)) - name_ofs = (offsetof(ATTR_REC, - data.non_resident.compressed_size) + - sizeof(a->data.non_resident.compressed_size) + - 7) & ~7; - else - name_ofs = (offsetof(ATTR_REC, - data.non_resident.compressed_size) + 7) & ~7; - mp_ofs = (name_ofs + a->name_length * sizeof(ntfschar) + 7) & ~7; - /* - * Determine the size of the resident part of the now non-resident - * attribute record. - */ - arec_size = (mp_ofs + mp_size + 7) & ~7; - /* - * If the page is not uptodate bring it uptodate by copying from the - * attribute value. - */ - attr_size = le32_to_cpu(a->data.resident.value_length); - BUG_ON(attr_size != data_size); - if (page && !PageUptodate(page)) { - kaddr = kmap_atomic(page); - memcpy(kaddr, (u8*)a + - le16_to_cpu(a->data.resident.value_offset), - attr_size); - memset(kaddr + attr_size, 0, PAGE_SIZE - attr_size); - kunmap_atomic(kaddr); - flush_dcache_page(page); - SetPageUptodate(page); - } - /* Backup the attribute flag. */ - old_res_attr_flags = a->data.resident.flags; - /* Resize the resident part of the attribute record. */ - err = ntfs_attr_record_resize(m, a, arec_size); - if (unlikely(err)) - goto err_out; - /* - * Convert the resident part of the attribute record to describe a - * non-resident attribute. - */ - a->non_resident = 1; - /* Move the attribute name if it exists and update the offset. */ - if (a->name_length) - memmove((u8*)a + name_ofs, (u8*)a + le16_to_cpu(a->name_offset), - a->name_length * sizeof(ntfschar)); - a->name_offset = cpu_to_le16(name_ofs); - /* Setup the fields specific to non-resident attributes. */ - a->data.non_resident.lowest_vcn = 0; - a->data.non_resident.highest_vcn = cpu_to_sle64((new_size - 1) >> - vol->cluster_size_bits); - a->data.non_resident.mapping_pairs_offset = cpu_to_le16(mp_ofs); - memset(&a->data.non_resident.reserved, 0, - sizeof(a->data.non_resident.reserved)); - a->data.non_resident.allocated_size = cpu_to_sle64(new_size); - a->data.non_resident.data_size = - a->data.non_resident.initialized_size = - cpu_to_sle64(attr_size); - if (NInoSparse(ni) || NInoCompressed(ni)) { - a->data.non_resident.compression_unit = 0; - if (NInoCompressed(ni) || vol->major_ver < 3) - a->data.non_resident.compression_unit = 4; - a->data.non_resident.compressed_size = - a->data.non_resident.allocated_size; - } else - a->data.non_resident.compression_unit = 0; - /* Generate the mapping pairs array into the attribute record. */ - err = ntfs_mapping_pairs_build(vol, (u8*)a + mp_ofs, - arec_size - mp_ofs, rl, 0, -1, NULL); - if (unlikely(err)) { - ntfs_debug("Failed to build mapping pairs, error code %i.", - err); - goto undo_err_out; - } - /* Setup the in-memory attribute structure to be non-resident. */ - ni->runlist.rl = rl; - write_lock_irqsave(&ni->size_lock, flags); - ni->allocated_size = new_size; - if (NInoSparse(ni) || NInoCompressed(ni)) { - ni->itype.compressed.size = ni->allocated_size; - if (a->data.non_resident.compression_unit) { - ni->itype.compressed.block_size = 1U << (a->data. - non_resident.compression_unit + - vol->cluster_size_bits); - ni->itype.compressed.block_size_bits = - ffs(ni->itype.compressed.block_size) - - 1; - ni->itype.compressed.block_clusters = 1U << - a->data.non_resident.compression_unit; - } else { - ni->itype.compressed.block_size = 0; - ni->itype.compressed.block_size_bits = 0; - ni->itype.compressed.block_clusters = 0; - } - vi->i_blocks = ni->itype.compressed.size >> 9; - } else - vi->i_blocks = ni->allocated_size >> 9; - write_unlock_irqrestore(&ni->size_lock, flags); - /* - * This needs to be last since the address space operations ->read_folio - * and ->writepage can run concurrently with us as they are not - * serialized on i_mutex. Note, we are not allowed to fail once we flip - * this switch, which is another reason to do this last. - */ - NInoSetNonResident(ni); - /* Mark the mft record dirty, so it gets written back. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - up_write(&ni->runlist.lock); - if (page) { - set_page_dirty(page); - unlock_page(page); - put_page(page); - } - ntfs_debug("Done."); - return 0; -undo_err_out: - /* Convert the attribute back into a resident attribute. */ - a->non_resident = 0; - /* Move the attribute name if it exists and update the offset. */ - name_ofs = (offsetof(ATTR_RECORD, data.resident.reserved) + - sizeof(a->data.resident.reserved) + 7) & ~7; - if (a->name_length) - memmove((u8*)a + name_ofs, (u8*)a + le16_to_cpu(a->name_offset), - a->name_length * sizeof(ntfschar)); - mp_ofs = (name_ofs + a->name_length * sizeof(ntfschar) + 7) & ~7; - a->name_offset = cpu_to_le16(name_ofs); - arec_size = (mp_ofs + attr_size + 7) & ~7; - /* Resize the resident part of the attribute record. */ - err2 = ntfs_attr_record_resize(m, a, arec_size); - if (unlikely(err2)) { - /* - * This cannot happen (well if memory corruption is at work it - * could happen in theory), but deal with it as well as we can. - * If the old size is too small, truncate the attribute, - * otherwise simply give it a larger allocated size. - * FIXME: Should check whether chkdsk complains when the - * allocated size is much bigger than the resident value size. - */ - arec_size = le32_to_cpu(a->length); - if ((mp_ofs + attr_size) > arec_size) { - err2 = attr_size; - attr_size = arec_size - mp_ofs; - ntfs_error(vol->sb, "Failed to undo partial resident " - "to non-resident attribute " - "conversion. Truncating inode 0x%lx, " - "attribute type 0x%x from %i bytes to " - "%i bytes to maintain metadata " - "consistency. THIS MEANS YOU ARE " - "LOSING %i BYTES DATA FROM THIS %s.", - vi->i_ino, - (unsigned)le32_to_cpu(ni->type), - err2, attr_size, err2 - attr_size, - ((ni->type == AT_DATA) && - !ni->name_len) ? "FILE": "ATTRIBUTE"); - write_lock_irqsave(&ni->size_lock, flags); - ni->initialized_size = attr_size; - i_size_write(vi, attr_size); - write_unlock_irqrestore(&ni->size_lock, flags); - } - } - /* Setup the fields specific to resident attributes. */ - a->data.resident.value_length = cpu_to_le32(attr_size); - a->data.resident.value_offset = cpu_to_le16(mp_ofs); - a->data.resident.flags = old_res_attr_flags; - memset(&a->data.resident.reserved, 0, - sizeof(a->data.resident.reserved)); - /* Copy the data from the page back to the attribute value. */ - if (page) { - kaddr = kmap_atomic(page); - memcpy((u8*)a + mp_ofs, kaddr, attr_size); - kunmap_atomic(kaddr); - } - /* Setup the allocated size in the ntfs inode in case it changed. */ - write_lock_irqsave(&ni->size_lock, flags); - ni->allocated_size = arec_size - mp_ofs; - write_unlock_irqrestore(&ni->size_lock, flags); - /* Mark the mft record dirty, so it gets written back. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); -err_out: - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(base_ni); - ni->runlist.rl = NULL; - up_write(&ni->runlist.lock); -rl_err_out: - if (rl) { - if (ntfs_cluster_free_from_rl(vol, rl) < 0) { - ntfs_error(vol->sb, "Failed to release allocated " - "cluster(s) in error code path. Run " - "chkdsk to recover the lost " - "cluster(s)."); - NVolSetErrors(vol); - } - ntfs_free(rl); -page_err_out: - unlock_page(page); - put_page(page); - } - if (err == -EINVAL) - err = -EIO; - return err; -} - -/** - * ntfs_attr_extend_allocation - extend the allocated space of an attribute - * @ni: ntfs inode of the attribute whose allocation to extend - * @new_alloc_size: new size in bytes to which to extend the allocation to - * @new_data_size: new size in bytes to which to extend the data to - * @data_start: beginning of region which is required to be non-sparse - * - * Extend the allocated space of an attribute described by the ntfs inode @ni - * to @new_alloc_size bytes. If @data_start is -1, the whole extension may be - * implemented as a hole in the file (as long as both the volume and the ntfs - * inode @ni have sparse support enabled). If @data_start is >= 0, then the - * region between the old allocated size and @data_start - 1 may be made sparse - * but the regions between @data_start and @new_alloc_size must be backed by - * actual clusters. - * - * If @new_data_size is -1, it is ignored. If it is >= 0, then the data size - * of the attribute is extended to @new_data_size. Note that the i_size of the - * vfs inode is not updated. Only the data size in the base attribute record - * is updated. The caller has to update i_size separately if this is required. - * WARNING: It is a BUG() for @new_data_size to be smaller than the old data - * size as well as for @new_data_size to be greater than @new_alloc_size. - * - * For resident attributes this involves resizing the attribute record and if - * necessary moving it and/or other attributes into extent mft records and/or - * converting the attribute to a non-resident attribute which in turn involves - * extending the allocation of a non-resident attribute as described below. - * - * For non-resident attributes this involves allocating clusters in the data - * zone on the volume (except for regions that are being made sparse) and - * extending the run list to describe the allocated clusters as well as - * updating the mapping pairs array of the attribute. This in turn involves - * resizing the attribute record and if necessary moving it and/or other - * attributes into extent mft records and/or splitting the attribute record - * into multiple extent attribute records. - * - * Also, the attribute list attribute is updated if present and in some of the - * above cases (the ones where extent mft records/attributes come into play), - * an attribute list attribute is created if not already present. - * - * Return the new allocated size on success and -errno on error. In the case - * that an error is encountered but a partial extension at least up to - * @data_start (if present) is possible, the allocation is partially extended - * and this is returned. This means the caller must check the returned size to - * determine if the extension was partial. If @data_start is -1 then partial - * allocations are not performed. - * - * WARNING: Do not call ntfs_attr_extend_allocation() for $MFT/$DATA. - * - * Locking: This function takes the runlist lock of @ni for writing as well as - * locking the mft record of the base ntfs inode. These locks are maintained - * throughout execution of the function. These locks are required so that the - * attribute can be resized safely and so that it can for example be converted - * from resident to non-resident safely. - * - * TODO: At present attribute list attribute handling is not implemented. - * - * TODO: At present it is not safe to call this function for anything other - * than the $DATA attribute(s) of an uncompressed and unencrypted file. - */ -s64 ntfs_attr_extend_allocation(ntfs_inode *ni, s64 new_alloc_size, - const s64 new_data_size, const s64 data_start) -{ - VCN vcn; - s64 ll, allocated_size, start = data_start; - struct inode *vi = VFS_I(ni); - ntfs_volume *vol = ni->vol; - ntfs_inode *base_ni; - MFT_RECORD *m; - ATTR_RECORD *a; - ntfs_attr_search_ctx *ctx; - runlist_element *rl, *rl2; - unsigned long flags; - int err, mp_size; - u32 attr_len = 0; /* Silence stupid gcc warning. */ - bool mp_rebuilt; - -#ifdef DEBUG - read_lock_irqsave(&ni->size_lock, flags); - allocated_size = ni->allocated_size; - read_unlock_irqrestore(&ni->size_lock, flags); - ntfs_debug("Entering for i_ino 0x%lx, attribute type 0x%x, " - "old_allocated_size 0x%llx, " - "new_allocated_size 0x%llx, new_data_size 0x%llx, " - "data_start 0x%llx.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), - (unsigned long long)allocated_size, - (unsigned long long)new_alloc_size, - (unsigned long long)new_data_size, - (unsigned long long)start); -#endif -retry_extend: - /* - * For non-resident attributes, @start and @new_size need to be aligned - * to cluster boundaries for allocation purposes. - */ - if (NInoNonResident(ni)) { - if (start > 0) - start &= ~(s64)vol->cluster_size_mask; - new_alloc_size = (new_alloc_size + vol->cluster_size - 1) & - ~(s64)vol->cluster_size_mask; - } - BUG_ON(new_data_size >= 0 && new_data_size > new_alloc_size); - /* Check if new size is allowed in $AttrDef. */ - err = ntfs_attr_size_bounds_check(vol, ni->type, new_alloc_size); - if (unlikely(err)) { - /* Only emit errors when the write will fail completely. */ - read_lock_irqsave(&ni->size_lock, flags); - allocated_size = ni->allocated_size; - read_unlock_irqrestore(&ni->size_lock, flags); - if (start < 0 || start >= allocated_size) { - if (err == -ERANGE) { - ntfs_error(vol->sb, "Cannot extend allocation " - "of inode 0x%lx, attribute " - "type 0x%x, because the new " - "allocation would exceed the " - "maximum allowed size for " - "this attribute type.", - vi->i_ino, (unsigned) - le32_to_cpu(ni->type)); - } else { - ntfs_error(vol->sb, "Cannot extend allocation " - "of inode 0x%lx, attribute " - "type 0x%x, because this " - "attribute type is not " - "defined on the NTFS volume. " - "Possible corruption! You " - "should run chkdsk!", - vi->i_ino, (unsigned) - le32_to_cpu(ni->type)); - } - } - /* Translate error code to be POSIX conformant for write(2). */ - if (err == -ERANGE) - err = -EFBIG; - else - err = -EIO; - return err; - } - if (!NInoAttr(ni)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - /* - * We will be modifying both the runlist (if non-resident) and the mft - * record so lock them both down. - */ - down_write(&ni->runlist.lock); - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - m = NULL; - ctx = NULL; - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - read_lock_irqsave(&ni->size_lock, flags); - allocated_size = ni->allocated_size; - read_unlock_irqrestore(&ni->size_lock, flags); - /* - * If non-resident, seek to the last extent. If resident, there is - * only one extent, so seek to that. - */ - vcn = NInoNonResident(ni) ? allocated_size >> vol->cluster_size_bits : - 0; - /* - * Abort if someone did the work whilst we waited for the locks. If we - * just converted the attribute from resident to non-resident it is - * likely that exactly this has happened already. We cannot quite - * abort if we need to update the data size. - */ - if (unlikely(new_alloc_size <= allocated_size)) { - ntfs_debug("Allocated size already exceeds requested size."); - new_alloc_size = allocated_size; - if (new_data_size < 0) - goto done; - /* - * We want the first attribute extent so that we can update the - * data size. - */ - vcn = 0; - } - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, vcn, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - err = -EIO; - goto err_out; - } - m = ctx->mrec; - a = ctx->attr; - /* Use goto to reduce indentation. */ - if (a->non_resident) - goto do_non_resident_extend; - BUG_ON(NInoNonResident(ni)); - /* The total length of the attribute value. */ - attr_len = le32_to_cpu(a->data.resident.value_length); - /* - * Extend the attribute record to be able to store the new attribute - * size. ntfs_attr_record_resize() will not do anything if the size is - * not changing. - */ - if (new_alloc_size < vol->mft_record_size && - !ntfs_attr_record_resize(m, a, - le16_to_cpu(a->data.resident.value_offset) + - new_alloc_size)) { - /* The resize succeeded! */ - write_lock_irqsave(&ni->size_lock, flags); - ni->allocated_size = le32_to_cpu(a->length) - - le16_to_cpu(a->data.resident.value_offset); - write_unlock_irqrestore(&ni->size_lock, flags); - if (new_data_size >= 0) { - BUG_ON(new_data_size < attr_len); - a->data.resident.value_length = - cpu_to_le32((u32)new_data_size); - } - goto flush_done; - } - /* - * We have to drop all the locks so we can call - * ntfs_attr_make_non_resident(). This could be optimised by try- - * locking the first page cache page and only if that fails dropping - * the locks, locking the page, and redoing all the locking and - * lookups. While this would be a huge optimisation, it is not worth - * it as this is definitely a slow code path. - */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - up_write(&ni->runlist.lock); - /* - * Not enough space in the mft record, try to make the attribute - * non-resident and if successful restart the extension process. - */ - err = ntfs_attr_make_non_resident(ni, attr_len); - if (likely(!err)) - goto retry_extend; - /* - * Could not make non-resident. If this is due to this not being - * permitted for this attribute type or there not being enough space, - * try to make other attributes non-resident. Otherwise fail. - */ - if (unlikely(err != -EPERM && err != -ENOSPC)) { - /* Only emit errors when the write will fail completely. */ - read_lock_irqsave(&ni->size_lock, flags); - allocated_size = ni->allocated_size; - read_unlock_irqrestore(&ni->size_lock, flags); - if (start < 0 || start >= allocated_size) - ntfs_error(vol->sb, "Cannot extend allocation of " - "inode 0x%lx, attribute type 0x%x, " - "because the conversion from resident " - "to non-resident attribute failed " - "with error code %i.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), err); - if (err != -ENOMEM) - err = -EIO; - goto conv_err_out; - } - /* TODO: Not implemented from here, abort. */ - read_lock_irqsave(&ni->size_lock, flags); - allocated_size = ni->allocated_size; - read_unlock_irqrestore(&ni->size_lock, flags); - if (start < 0 || start >= allocated_size) { - if (err == -ENOSPC) - ntfs_error(vol->sb, "Not enough space in the mft " - "record/on disk for the non-resident " - "attribute value. This case is not " - "implemented yet."); - else /* if (err == -EPERM) */ - ntfs_error(vol->sb, "This attribute type may not be " - "non-resident. This case is not " - "implemented yet."); - } - err = -EOPNOTSUPP; - goto conv_err_out; -#if 0 - // TODO: Attempt to make other attributes non-resident. - if (!err) - goto do_resident_extend; - /* - * Both the attribute list attribute and the standard information - * attribute must remain in the base inode. Thus, if this is one of - * these attributes, we have to try to move other attributes out into - * extent mft records instead. - */ - if (ni->type == AT_ATTRIBUTE_LIST || - ni->type == AT_STANDARD_INFORMATION) { - // TODO: Attempt to move other attributes into extent mft - // records. - err = -EOPNOTSUPP; - if (!err) - goto do_resident_extend; - goto err_out; - } - // TODO: Attempt to move this attribute to an extent mft record, but - // only if it is not already the only attribute in an mft record in - // which case there would be nothing to gain. - err = -EOPNOTSUPP; - if (!err) - goto do_resident_extend; - /* There is nothing we can do to make enough space. )-: */ - goto err_out; -#endif -do_non_resident_extend: - BUG_ON(!NInoNonResident(ni)); - if (new_alloc_size == allocated_size) { - BUG_ON(vcn); - goto alloc_done; - } - /* - * If the data starts after the end of the old allocation, this is a - * $DATA attribute and sparse attributes are enabled on the volume and - * for this inode, then create a sparse region between the old - * allocated size and the start of the data. Otherwise simply proceed - * with filling the whole space between the old allocated size and the - * new allocated size with clusters. - */ - if ((start >= 0 && start <= allocated_size) || ni->type != AT_DATA || - !NVolSparseEnabled(vol) || NInoSparseDisabled(ni)) - goto skip_sparse; - // TODO: This is not implemented yet. We just fill in with real - // clusters for now... - ntfs_debug("Inserting holes is not-implemented yet. Falling back to " - "allocating real clusters instead."); -skip_sparse: - rl = ni->runlist.rl; - if (likely(rl)) { - /* Seek to the end of the runlist. */ - while (rl->length) - rl++; - } - /* If this attribute extent is not mapped, map it now. */ - if (unlikely(!rl || rl->lcn == LCN_RL_NOT_MAPPED || - (rl->lcn == LCN_ENOENT && rl > ni->runlist.rl && - (rl-1)->lcn == LCN_RL_NOT_MAPPED))) { - if (!rl && !allocated_size) - goto first_alloc; - rl = ntfs_mapping_pairs_decompress(vol, a, ni->runlist.rl); - if (IS_ERR(rl)) { - err = PTR_ERR(rl); - if (start < 0 || start >= allocated_size) - ntfs_error(vol->sb, "Cannot extend allocation " - "of inode 0x%lx, attribute " - "type 0x%x, because the " - "mapping of a runlist " - "fragment failed with error " - "code %i.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), - err); - if (err != -ENOMEM) - err = -EIO; - goto err_out; - } - ni->runlist.rl = rl; - /* Seek to the end of the runlist. */ - while (rl->length) - rl++; - } - /* - * We now know the runlist of the last extent is mapped and @rl is at - * the end of the runlist. We want to begin allocating clusters - * starting at the last allocated cluster to reduce fragmentation. If - * there are no valid LCNs in the attribute we let the cluster - * allocator choose the starting cluster. - */ - /* If the last LCN is a hole or simillar seek back to last real LCN. */ - while (rl->lcn < 0 && rl > ni->runlist.rl) - rl--; -first_alloc: - // FIXME: Need to implement partial allocations so at least part of the - // write can be performed when start >= 0. (Needed for POSIX write(2) - // conformance.) - rl2 = ntfs_cluster_alloc(vol, allocated_size >> vol->cluster_size_bits, - (new_alloc_size - allocated_size) >> - vol->cluster_size_bits, (rl && (rl->lcn >= 0)) ? - rl->lcn + rl->length : -1, DATA_ZONE, true); - if (IS_ERR(rl2)) { - err = PTR_ERR(rl2); - if (start < 0 || start >= allocated_size) - ntfs_error(vol->sb, "Cannot extend allocation of " - "inode 0x%lx, attribute type 0x%x, " - "because the allocation of clusters " - "failed with error code %i.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), err); - if (err != -ENOMEM && err != -ENOSPC) - err = -EIO; - goto err_out; - } - rl = ntfs_runlists_merge(ni->runlist.rl, rl2); - if (IS_ERR(rl)) { - err = PTR_ERR(rl); - if (start < 0 || start >= allocated_size) - ntfs_error(vol->sb, "Cannot extend allocation of " - "inode 0x%lx, attribute type 0x%x, " - "because the runlist merge failed " - "with error code %i.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), err); - if (err != -ENOMEM) - err = -EIO; - if (ntfs_cluster_free_from_rl(vol, rl2)) { - ntfs_error(vol->sb, "Failed to release allocated " - "cluster(s) in error code path. Run " - "chkdsk to recover the lost " - "cluster(s)."); - NVolSetErrors(vol); - } - ntfs_free(rl2); - goto err_out; - } - ni->runlist.rl = rl; - ntfs_debug("Allocated 0x%llx clusters.", (long long)(new_alloc_size - - allocated_size) >> vol->cluster_size_bits); - /* Find the runlist element with which the attribute extent starts. */ - ll = sle64_to_cpu(a->data.non_resident.lowest_vcn); - rl2 = ntfs_rl_find_vcn_nolock(rl, ll); - BUG_ON(!rl2); - BUG_ON(!rl2->length); - BUG_ON(rl2->lcn < LCN_HOLE); - mp_rebuilt = false; - /* Get the size for the new mapping pairs array for this extent. */ - mp_size = ntfs_get_size_for_mapping_pairs(vol, rl2, ll, -1); - if (unlikely(mp_size <= 0)) { - err = mp_size; - if (start < 0 || start >= allocated_size) - ntfs_error(vol->sb, "Cannot extend allocation of " - "inode 0x%lx, attribute type 0x%x, " - "because determining the size for the " - "mapping pairs failed with error code " - "%i.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), err); - err = -EIO; - goto undo_alloc; - } - /* Extend the attribute record to fit the bigger mapping pairs array. */ - attr_len = le32_to_cpu(a->length); - err = ntfs_attr_record_resize(m, a, mp_size + - le16_to_cpu(a->data.non_resident.mapping_pairs_offset)); - if (unlikely(err)) { - BUG_ON(err != -ENOSPC); - // TODO: Deal with this by moving this extent to a new mft - // record or by starting a new extent in a new mft record, - // possibly by extending this extent partially and filling it - // and creating a new extent for the remainder, or by making - // other attributes non-resident and/or by moving other - // attributes out of this mft record. - if (start < 0 || start >= allocated_size) - ntfs_error(vol->sb, "Not enough space in the mft " - "record for the extended attribute " - "record. This case is not " - "implemented yet."); - err = -EOPNOTSUPP; - goto undo_alloc; - } - mp_rebuilt = true; - /* Generate the mapping pairs array directly into the attr record. */ - err = ntfs_mapping_pairs_build(vol, (u8*)a + - le16_to_cpu(a->data.non_resident.mapping_pairs_offset), - mp_size, rl2, ll, -1, NULL); - if (unlikely(err)) { - if (start < 0 || start >= allocated_size) - ntfs_error(vol->sb, "Cannot extend allocation of " - "inode 0x%lx, attribute type 0x%x, " - "because building the mapping pairs " - "failed with error code %i.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), err); - err = -EIO; - goto undo_alloc; - } - /* Update the highest_vcn. */ - a->data.non_resident.highest_vcn = cpu_to_sle64((new_alloc_size >> - vol->cluster_size_bits) - 1); - /* - * We now have extended the allocated size of the attribute. Reflect - * this in the ntfs_inode structure and the attribute record. - */ - if (a->data.non_resident.lowest_vcn) { - /* - * We are not in the first attribute extent, switch to it, but - * first ensure the changes will make it to disk later. - */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_reinit_search_ctx(ctx); - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) - goto restore_undo_alloc; - /* @m is not used any more so no need to set it. */ - a = ctx->attr; - } - write_lock_irqsave(&ni->size_lock, flags); - ni->allocated_size = new_alloc_size; - a->data.non_resident.allocated_size = cpu_to_sle64(new_alloc_size); - /* - * FIXME: This would fail if @ni is a directory, $MFT, or an index, - * since those can have sparse/compressed set. For example can be - * set compressed even though it is not compressed itself and in that - * case the bit means that files are to be created compressed in the - * directory... At present this is ok as this code is only called for - * regular files, and only for their $DATA attribute(s). - * FIXME: The calculation is wrong if we created a hole above. For now - * it does not matter as we never create holes. - */ - if (NInoSparse(ni) || NInoCompressed(ni)) { - ni->itype.compressed.size += new_alloc_size - allocated_size; - a->data.non_resident.compressed_size = - cpu_to_sle64(ni->itype.compressed.size); - vi->i_blocks = ni->itype.compressed.size >> 9; - } else - vi->i_blocks = new_alloc_size >> 9; - write_unlock_irqrestore(&ni->size_lock, flags); -alloc_done: - if (new_data_size >= 0) { - BUG_ON(new_data_size < - sle64_to_cpu(a->data.non_resident.data_size)); - a->data.non_resident.data_size = cpu_to_sle64(new_data_size); - } -flush_done: - /* Ensure the changes make it to disk. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); -done: - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - up_write(&ni->runlist.lock); - ntfs_debug("Done, new_allocated_size 0x%llx.", - (unsigned long long)new_alloc_size); - return new_alloc_size; -restore_undo_alloc: - if (start < 0 || start >= allocated_size) - ntfs_error(vol->sb, "Cannot complete extension of allocation " - "of inode 0x%lx, attribute type 0x%x, because " - "lookup of first attribute extent failed with " - "error code %i.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), err); - if (err == -ENOENT) - err = -EIO; - ntfs_attr_reinit_search_ctx(ctx); - if (ntfs_attr_lookup(ni->type, ni->name, ni->name_len, CASE_SENSITIVE, - allocated_size >> vol->cluster_size_bits, NULL, 0, - ctx)) { - ntfs_error(vol->sb, "Failed to find last attribute extent of " - "attribute in error code path. Run chkdsk to " - "recover."); - write_lock_irqsave(&ni->size_lock, flags); - ni->allocated_size = new_alloc_size; - /* - * FIXME: This would fail if @ni is a directory... See above. - * FIXME: The calculation is wrong if we created a hole above. - * For now it does not matter as we never create holes. - */ - if (NInoSparse(ni) || NInoCompressed(ni)) { - ni->itype.compressed.size += new_alloc_size - - allocated_size; - vi->i_blocks = ni->itype.compressed.size >> 9; - } else - vi->i_blocks = new_alloc_size >> 9; - write_unlock_irqrestore(&ni->size_lock, flags); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - up_write(&ni->runlist.lock); - /* - * The only thing that is now wrong is the allocated size of the - * base attribute extent which chkdsk should be able to fix. - */ - NVolSetErrors(vol); - return err; - } - ctx->attr->data.non_resident.highest_vcn = cpu_to_sle64( - (allocated_size >> vol->cluster_size_bits) - 1); -undo_alloc: - ll = allocated_size >> vol->cluster_size_bits; - if (ntfs_cluster_free(ni, ll, -1, ctx) < 0) { - ntfs_error(vol->sb, "Failed to release allocated cluster(s) " - "in error code path. Run chkdsk to recover " - "the lost cluster(s)."); - NVolSetErrors(vol); - } - m = ctx->mrec; - a = ctx->attr; - /* - * If the runlist truncation fails and/or the search context is no - * longer valid, we cannot resize the attribute record or build the - * mapping pairs array thus we mark the inode bad so that no access to - * the freed clusters can happen. - */ - if (ntfs_rl_truncate_nolock(vol, &ni->runlist, ll) || IS_ERR(m)) { - ntfs_error(vol->sb, "Failed to %s in error code path. Run " - "chkdsk to recover.", IS_ERR(m) ? - "restore attribute search context" : - "truncate attribute runlist"); - NVolSetErrors(vol); - } else if (mp_rebuilt) { - if (ntfs_attr_record_resize(m, a, attr_len)) { - ntfs_error(vol->sb, "Failed to restore attribute " - "record in error code path. Run " - "chkdsk to recover."); - NVolSetErrors(vol); - } else /* if (success) */ { - if (ntfs_mapping_pairs_build(vol, (u8*)a + le16_to_cpu( - a->data.non_resident. - mapping_pairs_offset), attr_len - - le16_to_cpu(a->data.non_resident. - mapping_pairs_offset), rl2, ll, -1, - NULL)) { - ntfs_error(vol->sb, "Failed to restore " - "mapping pairs array in error " - "code path. Run chkdsk to " - "recover."); - NVolSetErrors(vol); - } - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - } - } -err_out: - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(base_ni); - up_write(&ni->runlist.lock); -conv_err_out: - ntfs_debug("Failed. Returning error code %i.", err); - return err; -} - -/** - * ntfs_attr_set - fill (a part of) an attribute with a byte - * @ni: ntfs inode describing the attribute to fill - * @ofs: offset inside the attribute at which to start to fill - * @cnt: number of bytes to fill - * @val: the unsigned 8-bit value with which to fill the attribute - * - * Fill @cnt bytes of the attribute described by the ntfs inode @ni starting at - * byte offset @ofs inside the attribute with the constant byte @val. - * - * This function is effectively like memset() applied to an ntfs attribute. - * Note this function actually only operates on the page cache pages belonging - * to the ntfs attribute and it marks them dirty after doing the memset(). - * Thus it relies on the vm dirty page write code paths to cause the modified - * pages to be written to the mft record/disk. - * - * Return 0 on success and -errno on error. An error code of -ESPIPE means - * that @ofs + @cnt were outside the end of the attribute and no write was - * performed. - */ -int ntfs_attr_set(ntfs_inode *ni, const s64 ofs, const s64 cnt, const u8 val) -{ - ntfs_volume *vol = ni->vol; - struct address_space *mapping; - struct page *page; - u8 *kaddr; - pgoff_t idx, end; - unsigned start_ofs, end_ofs, size; - - ntfs_debug("Entering for ofs 0x%llx, cnt 0x%llx, val 0x%hx.", - (long long)ofs, (long long)cnt, val); - BUG_ON(ofs < 0); - BUG_ON(cnt < 0); - if (!cnt) - goto done; - /* - * FIXME: Compressed and encrypted attributes are not supported when - * writing and we should never have gotten here for them. - */ - BUG_ON(NInoCompressed(ni)); - BUG_ON(NInoEncrypted(ni)); - mapping = VFS_I(ni)->i_mapping; - /* Work out the starting index and page offset. */ - idx = ofs >> PAGE_SHIFT; - start_ofs = ofs & ~PAGE_MASK; - /* Work out the ending index and page offset. */ - end = ofs + cnt; - end_ofs = end & ~PAGE_MASK; - /* If the end is outside the inode size return -ESPIPE. */ - if (unlikely(end > i_size_read(VFS_I(ni)))) { - ntfs_error(vol->sb, "Request exceeds end of attribute."); - return -ESPIPE; - } - end >>= PAGE_SHIFT; - /* If there is a first partial page, need to do it the slow way. */ - if (start_ofs) { - page = read_mapping_page(mapping, idx, NULL); - if (IS_ERR(page)) { - ntfs_error(vol->sb, "Failed to read first partial " - "page (error, index 0x%lx).", idx); - return PTR_ERR(page); - } - /* - * If the last page is the same as the first page, need to - * limit the write to the end offset. - */ - size = PAGE_SIZE; - if (idx == end) - size = end_ofs; - kaddr = kmap_atomic(page); - memset(kaddr + start_ofs, val, size - start_ofs); - flush_dcache_page(page); - kunmap_atomic(kaddr); - set_page_dirty(page); - put_page(page); - balance_dirty_pages_ratelimited(mapping); - cond_resched(); - if (idx == end) - goto done; - idx++; - } - /* Do the whole pages the fast way. */ - for (; idx < end; idx++) { - /* Find or create the current page. (The page is locked.) */ - page = grab_cache_page(mapping, idx); - if (unlikely(!page)) { - ntfs_error(vol->sb, "Insufficient memory to grab " - "page (index 0x%lx).", idx); - return -ENOMEM; - } - kaddr = kmap_atomic(page); - memset(kaddr, val, PAGE_SIZE); - flush_dcache_page(page); - kunmap_atomic(kaddr); - /* - * If the page has buffers, mark them uptodate since buffer - * state and not page state is definitive in 2.6 kernels. - */ - if (page_has_buffers(page)) { - struct buffer_head *bh, *head; - - bh = head = page_buffers(page); - do { - set_buffer_uptodate(bh); - } while ((bh = bh->b_this_page) != head); - } - /* Now that buffers are uptodate, set the page uptodate, too. */ - SetPageUptodate(page); - /* - * Set the page and all its buffers dirty and mark the inode - * dirty, too. The VM will write the page later on. - */ - set_page_dirty(page); - /* Finally unlock and release the page. */ - unlock_page(page); - put_page(page); - balance_dirty_pages_ratelimited(mapping); - cond_resched(); - } - /* If there is a last partial page, need to do it the slow way. */ - if (end_ofs) { - page = read_mapping_page(mapping, idx, NULL); - if (IS_ERR(page)) { - ntfs_error(vol->sb, "Failed to read last partial page " - "(error, index 0x%lx).", idx); - return PTR_ERR(page); - } - kaddr = kmap_atomic(page); - memset(kaddr, val, end_ofs); - flush_dcache_page(page); - kunmap_atomic(kaddr); - set_page_dirty(page); - put_page(page); - balance_dirty_pages_ratelimited(mapping); - cond_resched(); - } -done: - ntfs_debug("Done."); - return 0; -} - -#endif /* NTFS_RW */ diff --git a/fs/ntfs/attrib.h b/fs/ntfs/attrib.h deleted file mode 100644 index fe0890d3d072..000000000000 --- a/fs/ntfs/attrib.h +++ /dev/null @@ -1,102 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * attrib.h - Defines for attribute handling in NTFS Linux kernel driver. - * Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2005 Anton Altaparmakov - * Copyright (c) 2002 Richard Russon - */ - -#ifndef _LINUX_NTFS_ATTRIB_H -#define _LINUX_NTFS_ATTRIB_H - -#include "endian.h" -#include "types.h" -#include "layout.h" -#include "inode.h" -#include "runlist.h" -#include "volume.h" - -/** - * ntfs_attr_search_ctx - used in attribute search functions - * @mrec: buffer containing mft record to search - * @attr: attribute record in @mrec where to begin/continue search - * @is_first: if true ntfs_attr_lookup() begins search with @attr, else after - * - * Structure must be initialized to zero before the first call to one of the - * attribute search functions. Initialize @mrec to point to the mft record to - * search, and @attr to point to the first attribute within @mrec (not necessary - * if calling the _first() functions), and set @is_first to 'true' (not necessary - * if calling the _first() functions). - * - * If @is_first is 'true', the search begins with @attr. If @is_first is 'false', - * the search begins after @attr. This is so that, after the first call to one - * of the search attribute functions, we can call the function again, without - * any modification of the search context, to automagically get the next - * matching attribute. - */ -typedef struct { - MFT_RECORD *mrec; - ATTR_RECORD *attr; - bool is_first; - ntfs_inode *ntfs_ino; - ATTR_LIST_ENTRY *al_entry; - ntfs_inode *base_ntfs_ino; - MFT_RECORD *base_mrec; - ATTR_RECORD *base_attr; -} ntfs_attr_search_ctx; - -extern int ntfs_map_runlist_nolock(ntfs_inode *ni, VCN vcn, - ntfs_attr_search_ctx *ctx); -extern int ntfs_map_runlist(ntfs_inode *ni, VCN vcn); - -extern LCN ntfs_attr_vcn_to_lcn_nolock(ntfs_inode *ni, const VCN vcn, - const bool write_locked); - -extern runlist_element *ntfs_attr_find_vcn_nolock(ntfs_inode *ni, - const VCN vcn, ntfs_attr_search_ctx *ctx); - -int ntfs_attr_lookup(const ATTR_TYPE type, const ntfschar *name, - const u32 name_len, const IGNORE_CASE_BOOL ic, - const VCN lowest_vcn, const u8 *val, const u32 val_len, - ntfs_attr_search_ctx *ctx); - -extern int load_attribute_list(ntfs_volume *vol, runlist *rl, u8 *al_start, - const s64 size, const s64 initialized_size); - -static inline s64 ntfs_attr_size(const ATTR_RECORD *a) -{ - if (!a->non_resident) - return (s64)le32_to_cpu(a->data.resident.value_length); - return sle64_to_cpu(a->data.non_resident.data_size); -} - -extern void ntfs_attr_reinit_search_ctx(ntfs_attr_search_ctx *ctx); -extern ntfs_attr_search_ctx *ntfs_attr_get_search_ctx(ntfs_inode *ni, - MFT_RECORD *mrec); -extern void ntfs_attr_put_search_ctx(ntfs_attr_search_ctx *ctx); - -#ifdef NTFS_RW - -extern int ntfs_attr_size_bounds_check(const ntfs_volume *vol, - const ATTR_TYPE type, const s64 size); -extern int ntfs_attr_can_be_non_resident(const ntfs_volume *vol, - const ATTR_TYPE type); -extern int ntfs_attr_can_be_resident(const ntfs_volume *vol, - const ATTR_TYPE type); - -extern int ntfs_attr_record_resize(MFT_RECORD *m, ATTR_RECORD *a, u32 new_size); -extern int ntfs_resident_attr_value_resize(MFT_RECORD *m, ATTR_RECORD *a, - const u32 new_size); - -extern int ntfs_attr_make_non_resident(ntfs_inode *ni, const u32 data_size); - -extern s64 ntfs_attr_extend_allocation(ntfs_inode *ni, s64 new_alloc_size, - const s64 new_data_size, const s64 data_start); - -extern int ntfs_attr_set(ntfs_inode *ni, const s64 ofs, const s64 cnt, - const u8 val); - -#endif /* NTFS_RW */ - -#endif /* _LINUX_NTFS_ATTRIB_H */ diff --git a/fs/ntfs/bitmap.c b/fs/ntfs/bitmap.c deleted file mode 100644 index 0675b2400873..000000000000 --- a/fs/ntfs/bitmap.c +++ /dev/null @@ -1,179 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * bitmap.c - NTFS kernel bitmap handling. Part of the Linux-NTFS project. - * - * Copyright (c) 2004-2005 Anton Altaparmakov - */ - -#ifdef NTFS_RW - -#include - -#include "bitmap.h" -#include "debug.h" -#include "aops.h" -#include "ntfs.h" - -/** - * __ntfs_bitmap_set_bits_in_run - set a run of bits in a bitmap to a value - * @vi: vfs inode describing the bitmap - * @start_bit: first bit to set - * @count: number of bits to set - * @value: value to set the bits to (i.e. 0 or 1) - * @is_rollback: if 'true' this is a rollback operation - * - * Set @count bits starting at bit @start_bit in the bitmap described by the - * vfs inode @vi to @value, where @value is either 0 or 1. - * - * @is_rollback should always be 'false', it is for internal use to rollback - * errors. You probably want to use ntfs_bitmap_set_bits_in_run() instead. - * - * Return 0 on success and -errno on error. - */ -int __ntfs_bitmap_set_bits_in_run(struct inode *vi, const s64 start_bit, - const s64 count, const u8 value, const bool is_rollback) -{ - s64 cnt = count; - pgoff_t index, end_index; - struct address_space *mapping; - struct page *page; - u8 *kaddr; - int pos, len; - u8 bit; - - BUG_ON(!vi); - ntfs_debug("Entering for i_ino 0x%lx, start_bit 0x%llx, count 0x%llx, " - "value %u.%s", vi->i_ino, (unsigned long long)start_bit, - (unsigned long long)cnt, (unsigned int)value, - is_rollback ? " (rollback)" : ""); - BUG_ON(start_bit < 0); - BUG_ON(cnt < 0); - BUG_ON(value > 1); - /* - * Calculate the indices for the pages containing the first and last - * bits, i.e. @start_bit and @start_bit + @cnt - 1, respectively. - */ - index = start_bit >> (3 + PAGE_SHIFT); - end_index = (start_bit + cnt - 1) >> (3 + PAGE_SHIFT); - - /* Get the page containing the first bit (@start_bit). */ - mapping = vi->i_mapping; - page = ntfs_map_page(mapping, index); - if (IS_ERR(page)) { - if (!is_rollback) - ntfs_error(vi->i_sb, "Failed to map first page (error " - "%li), aborting.", PTR_ERR(page)); - return PTR_ERR(page); - } - kaddr = page_address(page); - - /* Set @pos to the position of the byte containing @start_bit. */ - pos = (start_bit >> 3) & ~PAGE_MASK; - - /* Calculate the position of @start_bit in the first byte. */ - bit = start_bit & 7; - - /* If the first byte is partial, modify the appropriate bits in it. */ - if (bit) { - u8 *byte = kaddr + pos; - while ((bit & 7) && cnt) { - cnt--; - if (value) - *byte |= 1 << bit++; - else - *byte &= ~(1 << bit++); - } - /* If we are done, unmap the page and return success. */ - if (!cnt) - goto done; - - /* Update @pos to the new position. */ - pos++; - } - /* - * Depending on @value, modify all remaining whole bytes in the page up - * to @cnt. - */ - len = min_t(s64, cnt >> 3, PAGE_SIZE - pos); - memset(kaddr + pos, value ? 0xff : 0, len); - cnt -= len << 3; - - /* Update @len to point to the first not-done byte in the page. */ - if (cnt < 8) - len += pos; - - /* If we are not in the last page, deal with all subsequent pages. */ - while (index < end_index) { - BUG_ON(cnt <= 0); - - /* Update @index and get the next page. */ - flush_dcache_page(page); - set_page_dirty(page); - ntfs_unmap_page(page); - page = ntfs_map_page(mapping, ++index); - if (IS_ERR(page)) - goto rollback; - kaddr = page_address(page); - /* - * Depending on @value, modify all remaining whole bytes in the - * page up to @cnt. - */ - len = min_t(s64, cnt >> 3, PAGE_SIZE); - memset(kaddr, value ? 0xff : 0, len); - cnt -= len << 3; - } - /* - * The currently mapped page is the last one. If the last byte is - * partial, modify the appropriate bits in it. Note, @len is the - * position of the last byte inside the page. - */ - if (cnt) { - u8 *byte; - - BUG_ON(cnt > 7); - - bit = cnt; - byte = kaddr + len; - while (bit--) { - if (value) - *byte |= 1 << bit; - else - *byte &= ~(1 << bit); - } - } -done: - /* We are done. Unmap the page and return success. */ - flush_dcache_page(page); - set_page_dirty(page); - ntfs_unmap_page(page); - ntfs_debug("Done."); - return 0; -rollback: - /* - * Current state: - * - no pages are mapped - * - @count - @cnt is the number of bits that have been modified - */ - if (is_rollback) - return PTR_ERR(page); - if (count != cnt) - pos = __ntfs_bitmap_set_bits_in_run(vi, start_bit, count - cnt, - value ? 0 : 1, true); - else - pos = 0; - if (!pos) { - /* Rollback was successful. */ - ntfs_error(vi->i_sb, "Failed to map subsequent page (error " - "%li), aborting.", PTR_ERR(page)); - } else { - /* Rollback failed. */ - ntfs_error(vi->i_sb, "Failed to map subsequent page (error " - "%li) and rollback failed (error %i). " - "Aborting and leaving inconsistent metadata. " - "Unmount and run chkdsk.", PTR_ERR(page), pos); - NVolSetErrors(NTFS_SB(vi->i_sb)); - } - return PTR_ERR(page); -} - -#endif /* NTFS_RW */ diff --git a/fs/ntfs/bitmap.h b/fs/ntfs/bitmap.h deleted file mode 100644 index 9dd2224ca9c4..000000000000 --- a/fs/ntfs/bitmap.h +++ /dev/null @@ -1,104 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * bitmap.h - Defines for NTFS kernel bitmap handling. Part of the Linux-NTFS - * project. - * - * Copyright (c) 2004 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_BITMAP_H -#define _LINUX_NTFS_BITMAP_H - -#ifdef NTFS_RW - -#include - -#include "types.h" - -extern int __ntfs_bitmap_set_bits_in_run(struct inode *vi, const s64 start_bit, - const s64 count, const u8 value, const bool is_rollback); - -/** - * ntfs_bitmap_set_bits_in_run - set a run of bits in a bitmap to a value - * @vi: vfs inode describing the bitmap - * @start_bit: first bit to set - * @count: number of bits to set - * @value: value to set the bits to (i.e. 0 or 1) - * - * Set @count bits starting at bit @start_bit in the bitmap described by the - * vfs inode @vi to @value, where @value is either 0 or 1. - * - * Return 0 on success and -errno on error. - */ -static inline int ntfs_bitmap_set_bits_in_run(struct inode *vi, - const s64 start_bit, const s64 count, const u8 value) -{ - return __ntfs_bitmap_set_bits_in_run(vi, start_bit, count, value, - false); -} - -/** - * ntfs_bitmap_set_run - set a run of bits in a bitmap - * @vi: vfs inode describing the bitmap - * @start_bit: first bit to set - * @count: number of bits to set - * - * Set @count bits starting at bit @start_bit in the bitmap described by the - * vfs inode @vi. - * - * Return 0 on success and -errno on error. - */ -static inline int ntfs_bitmap_set_run(struct inode *vi, const s64 start_bit, - const s64 count) -{ - return ntfs_bitmap_set_bits_in_run(vi, start_bit, count, 1); -} - -/** - * ntfs_bitmap_clear_run - clear a run of bits in a bitmap - * @vi: vfs inode describing the bitmap - * @start_bit: first bit to clear - * @count: number of bits to clear - * - * Clear @count bits starting at bit @start_bit in the bitmap described by the - * vfs inode @vi. - * - * Return 0 on success and -errno on error. - */ -static inline int ntfs_bitmap_clear_run(struct inode *vi, const s64 start_bit, - const s64 count) -{ - return ntfs_bitmap_set_bits_in_run(vi, start_bit, count, 0); -} - -/** - * ntfs_bitmap_set_bit - set a bit in a bitmap - * @vi: vfs inode describing the bitmap - * @bit: bit to set - * - * Set bit @bit in the bitmap described by the vfs inode @vi. - * - * Return 0 on success and -errno on error. - */ -static inline int ntfs_bitmap_set_bit(struct inode *vi, const s64 bit) -{ - return ntfs_bitmap_set_run(vi, bit, 1); -} - -/** - * ntfs_bitmap_clear_bit - clear a bit in a bitmap - * @vi: vfs inode describing the bitmap - * @bit: bit to clear - * - * Clear bit @bit in the bitmap described by the vfs inode @vi. - * - * Return 0 on success and -errno on error. - */ -static inline int ntfs_bitmap_clear_bit(struct inode *vi, const s64 bit) -{ - return ntfs_bitmap_clear_run(vi, bit, 1); -} - -#endif /* NTFS_RW */ - -#endif /* defined _LINUX_NTFS_BITMAP_H */ diff --git a/fs/ntfs/collate.c b/fs/ntfs/collate.c deleted file mode 100644 index 3ab6ec96abfe..000000000000 --- a/fs/ntfs/collate.c +++ /dev/null @@ -1,110 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * collate.c - NTFS kernel collation handling. Part of the Linux-NTFS project. - * - * Copyright (c) 2004 Anton Altaparmakov - */ - -#include "collate.h" -#include "debug.h" -#include "ntfs.h" - -static int ntfs_collate_binary(ntfs_volume *vol, - const void *data1, const int data1_len, - const void *data2, const int data2_len) -{ - int rc; - - ntfs_debug("Entering."); - rc = memcmp(data1, data2, min(data1_len, data2_len)); - if (!rc && (data1_len != data2_len)) { - if (data1_len < data2_len) - rc = -1; - else - rc = 1; - } - ntfs_debug("Done, returning %i", rc); - return rc; -} - -static int ntfs_collate_ntofs_ulong(ntfs_volume *vol, - const void *data1, const int data1_len, - const void *data2, const int data2_len) -{ - int rc; - u32 d1, d2; - - ntfs_debug("Entering."); - // FIXME: We don't really want to bug here. - BUG_ON(data1_len != data2_len); - BUG_ON(data1_len != 4); - d1 = le32_to_cpup(data1); - d2 = le32_to_cpup(data2); - if (d1 < d2) - rc = -1; - else { - if (d1 == d2) - rc = 0; - else - rc = 1; - } - ntfs_debug("Done, returning %i", rc); - return rc; -} - -typedef int (*ntfs_collate_func_t)(ntfs_volume *, const void *, const int, - const void *, const int); - -static ntfs_collate_func_t ntfs_do_collate0x0[3] = { - ntfs_collate_binary, - NULL/*ntfs_collate_file_name*/, - NULL/*ntfs_collate_unicode_string*/, -}; - -static ntfs_collate_func_t ntfs_do_collate0x1[4] = { - ntfs_collate_ntofs_ulong, - NULL/*ntfs_collate_ntofs_sid*/, - NULL/*ntfs_collate_ntofs_security_hash*/, - NULL/*ntfs_collate_ntofs_ulongs*/, -}; - -/** - * ntfs_collate - collate two data items using a specified collation rule - * @vol: ntfs volume to which the data items belong - * @cr: collation rule to use when comparing the items - * @data1: first data item to collate - * @data1_len: length in bytes of @data1 - * @data2: second data item to collate - * @data2_len: length in bytes of @data2 - * - * Collate the two data items @data1 and @data2 using the collation rule @cr - * and return -1, 0, ir 1 if @data1 is found, respectively, to collate before, - * to match, or to collate after @data2. - * - * For speed we use the collation rule @cr as an index into two tables of - * function pointers to call the appropriate collation function. - */ -int ntfs_collate(ntfs_volume *vol, COLLATION_RULE cr, - const void *data1, const int data1_len, - const void *data2, const int data2_len) { - int i; - - ntfs_debug("Entering."); - /* - * FIXME: At the moment we only support COLLATION_BINARY and - * COLLATION_NTOFS_ULONG, so we BUG() for everything else for now. - */ - BUG_ON(cr != COLLATION_BINARY && cr != COLLATION_NTOFS_ULONG); - i = le32_to_cpu(cr); - BUG_ON(i < 0); - if (i <= 0x02) - return ntfs_do_collate0x0[i](vol, data1, data1_len, - data2, data2_len); - BUG_ON(i < 0x10); - i -= 0x10; - if (likely(i <= 3)) - return ntfs_do_collate0x1[i](vol, data1, data1_len, - data2, data2_len); - BUG(); - return 0; -} diff --git a/fs/ntfs/collate.h b/fs/ntfs/collate.h deleted file mode 100644 index f2255619b4f4..000000000000 --- a/fs/ntfs/collate.h +++ /dev/null @@ -1,36 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * collate.h - Defines for NTFS kernel collation handling. Part of the - * Linux-NTFS project. - * - * Copyright (c) 2004 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_COLLATE_H -#define _LINUX_NTFS_COLLATE_H - -#include "types.h" -#include "volume.h" - -static inline bool ntfs_is_collation_rule_supported(COLLATION_RULE cr) { - int i; - - /* - * FIXME: At the moment we only support COLLATION_BINARY and - * COLLATION_NTOFS_ULONG, so we return false for everything else for - * now. - */ - if (unlikely(cr != COLLATION_BINARY && cr != COLLATION_NTOFS_ULONG)) - return false; - i = le32_to_cpu(cr); - if (likely(((i >= 0) && (i <= 0x02)) || - ((i >= 0x10) && (i <= 0x13)))) - return true; - return false; -} - -extern int ntfs_collate(ntfs_volume *vol, COLLATION_RULE cr, - const void *data1, const int data1_len, - const void *data2, const int data2_len); - -#endif /* _LINUX_NTFS_COLLATE_H */ diff --git a/fs/ntfs/compress.c b/fs/ntfs/compress.c deleted file mode 100644 index 761aaa0195d6..000000000000 --- a/fs/ntfs/compress.c +++ /dev/null @@ -1,950 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * compress.c - NTFS kernel compressed attributes handling. - * Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2004 Anton Altaparmakov - * Copyright (c) 2002 Richard Russon - */ - -#include -#include -#include -#include -#include - -#include "attrib.h" -#include "inode.h" -#include "debug.h" -#include "ntfs.h" - -/** - * ntfs_compression_constants - enum of constants used in the compression code - */ -typedef enum { - /* Token types and access mask. */ - NTFS_SYMBOL_TOKEN = 0, - NTFS_PHRASE_TOKEN = 1, - NTFS_TOKEN_MASK = 1, - - /* Compression sub-block constants. */ - NTFS_SB_SIZE_MASK = 0x0fff, - NTFS_SB_SIZE = 0x1000, - NTFS_SB_IS_COMPRESSED = 0x8000, - - /* - * The maximum compression block size is by definition 16 * the cluster - * size, with the maximum supported cluster size being 4kiB. Thus the - * maximum compression buffer size is 64kiB, so we use this when - * initializing the compression buffer. - */ - NTFS_MAX_CB_SIZE = 64 * 1024, -} ntfs_compression_constants; - -/* - * ntfs_compression_buffer - one buffer for the decompression engine - */ -static u8 *ntfs_compression_buffer; - -/* - * ntfs_cb_lock - spinlock which protects ntfs_compression_buffer - */ -static DEFINE_SPINLOCK(ntfs_cb_lock); - -/** - * allocate_compression_buffers - allocate the decompression buffers - * - * Caller has to hold the ntfs_lock mutex. - * - * Return 0 on success or -ENOMEM if the allocations failed. - */ -int allocate_compression_buffers(void) -{ - BUG_ON(ntfs_compression_buffer); - - ntfs_compression_buffer = vmalloc(NTFS_MAX_CB_SIZE); - if (!ntfs_compression_buffer) - return -ENOMEM; - return 0; -} - -/** - * free_compression_buffers - free the decompression buffers - * - * Caller has to hold the ntfs_lock mutex. - */ -void free_compression_buffers(void) -{ - BUG_ON(!ntfs_compression_buffer); - vfree(ntfs_compression_buffer); - ntfs_compression_buffer = NULL; -} - -/** - * zero_partial_compressed_page - zero out of bounds compressed page region - */ -static void zero_partial_compressed_page(struct page *page, - const s64 initialized_size) -{ - u8 *kp = page_address(page); - unsigned int kp_ofs; - - ntfs_debug("Zeroing page region outside initialized size."); - if (((s64)page->index << PAGE_SHIFT) >= initialized_size) { - clear_page(kp); - return; - } - kp_ofs = initialized_size & ~PAGE_MASK; - memset(kp + kp_ofs, 0, PAGE_SIZE - kp_ofs); - return; -} - -/** - * handle_bounds_compressed_page - test for&handle out of bounds compressed page - */ -static inline void handle_bounds_compressed_page(struct page *page, - const loff_t i_size, const s64 initialized_size) -{ - if ((page->index >= (initialized_size >> PAGE_SHIFT)) && - (initialized_size < i_size)) - zero_partial_compressed_page(page, initialized_size); - return; -} - -/** - * ntfs_decompress - decompress a compression block into an array of pages - * @dest_pages: destination array of pages - * @completed_pages: scratch space to track completed pages - * @dest_index: current index into @dest_pages (IN/OUT) - * @dest_ofs: current offset within @dest_pages[@dest_index] (IN/OUT) - * @dest_max_index: maximum index into @dest_pages (IN) - * @dest_max_ofs: maximum offset within @dest_pages[@dest_max_index] (IN) - * @xpage: the target page (-1 if none) (IN) - * @xpage_done: set to 1 if xpage was completed successfully (IN/OUT) - * @cb_start: compression block to decompress (IN) - * @cb_size: size of compression block @cb_start in bytes (IN) - * @i_size: file size when we started the read (IN) - * @initialized_size: initialized file size when we started the read (IN) - * - * The caller must have disabled preemption. ntfs_decompress() reenables it when - * the critical section is finished. - * - * This decompresses the compression block @cb_start into the array of - * destination pages @dest_pages starting at index @dest_index into @dest_pages - * and at offset @dest_pos into the page @dest_pages[@dest_index]. - * - * When the page @dest_pages[@xpage] is completed, @xpage_done is set to 1. - * If xpage is -1 or @xpage has not been completed, @xpage_done is not modified. - * - * @cb_start is a pointer to the compression block which needs decompressing - * and @cb_size is the size of @cb_start in bytes (8-64kiB). - * - * Return 0 if success or -EOVERFLOW on error in the compressed stream. - * @xpage_done indicates whether the target page (@dest_pages[@xpage]) was - * completed during the decompression of the compression block (@cb_start). - * - * Warning: This function *REQUIRES* PAGE_SIZE >= 4096 or it will blow up - * unpredicatbly! You have been warned! - * - * Note to hackers: This function may not sleep until it has finished accessing - * the compression block @cb_start as it is a per-CPU buffer. - */ -static int ntfs_decompress(struct page *dest_pages[], int completed_pages[], - int *dest_index, int *dest_ofs, const int dest_max_index, - const int dest_max_ofs, const int xpage, char *xpage_done, - u8 *const cb_start, const u32 cb_size, const loff_t i_size, - const s64 initialized_size) -{ - /* - * Pointers into the compressed data, i.e. the compression block (cb), - * and the therein contained sub-blocks (sb). - */ - u8 *cb_end = cb_start + cb_size; /* End of cb. */ - u8 *cb = cb_start; /* Current position in cb. */ - u8 *cb_sb_start; /* Beginning of the current sb in the cb. */ - u8 *cb_sb_end; /* End of current sb / beginning of next sb. */ - - /* Variables for uncompressed data / destination. */ - struct page *dp; /* Current destination page being worked on. */ - u8 *dp_addr; /* Current pointer into dp. */ - u8 *dp_sb_start; /* Start of current sub-block in dp. */ - u8 *dp_sb_end; /* End of current sb in dp (dp_sb_start + - NTFS_SB_SIZE). */ - u16 do_sb_start; /* @dest_ofs when starting this sub-block. */ - u16 do_sb_end; /* @dest_ofs of end of this sb (do_sb_start + - NTFS_SB_SIZE). */ - - /* Variables for tag and token parsing. */ - u8 tag; /* Current tag. */ - int token; /* Loop counter for the eight tokens in tag. */ - int nr_completed_pages = 0; - - /* Default error code. */ - int err = -EOVERFLOW; - - ntfs_debug("Entering, cb_size = 0x%x.", cb_size); -do_next_sb: - ntfs_debug("Beginning sub-block at offset = 0x%zx in the cb.", - cb - cb_start); - /* - * Have we reached the end of the compression block or the end of the - * decompressed data? The latter can happen for example if the current - * position in the compression block is one byte before its end so the - * first two checks do not detect it. - */ - if (cb == cb_end || !le16_to_cpup((le16*)cb) || - (*dest_index == dest_max_index && - *dest_ofs == dest_max_ofs)) { - int i; - - ntfs_debug("Completed. Returning success (0)."); - err = 0; -return_error: - /* We can sleep from now on, so we drop lock. */ - spin_unlock(&ntfs_cb_lock); - /* Second stage: finalize completed pages. */ - if (nr_completed_pages > 0) { - for (i = 0; i < nr_completed_pages; i++) { - int di = completed_pages[i]; - - dp = dest_pages[di]; - /* - * If we are outside the initialized size, zero - * the out of bounds page range. - */ - handle_bounds_compressed_page(dp, i_size, - initialized_size); - flush_dcache_page(dp); - kunmap(dp); - SetPageUptodate(dp); - unlock_page(dp); - if (di == xpage) - *xpage_done = 1; - else - put_page(dp); - dest_pages[di] = NULL; - } - } - return err; - } - - /* Setup offsets for the current sub-block destination. */ - do_sb_start = *dest_ofs; - do_sb_end = do_sb_start + NTFS_SB_SIZE; - - /* Check that we are still within allowed boundaries. */ - if (*dest_index == dest_max_index && do_sb_end > dest_max_ofs) - goto return_overflow; - - /* Does the minimum size of a compressed sb overflow valid range? */ - if (cb + 6 > cb_end) - goto return_overflow; - - /* Setup the current sub-block source pointers and validate range. */ - cb_sb_start = cb; - cb_sb_end = cb_sb_start + (le16_to_cpup((le16*)cb) & NTFS_SB_SIZE_MASK) - + 3; - if (cb_sb_end > cb_end) - goto return_overflow; - - /* Get the current destination page. */ - dp = dest_pages[*dest_index]; - if (!dp) { - /* No page present. Skip decompression of this sub-block. */ - cb = cb_sb_end; - - /* Advance destination position to next sub-block. */ - *dest_ofs = (*dest_ofs + NTFS_SB_SIZE) & ~PAGE_MASK; - if (!*dest_ofs && (++*dest_index > dest_max_index)) - goto return_overflow; - goto do_next_sb; - } - - /* We have a valid destination page. Setup the destination pointers. */ - dp_addr = (u8*)page_address(dp) + do_sb_start; - - /* Now, we are ready to process the current sub-block (sb). */ - if (!(le16_to_cpup((le16*)cb) & NTFS_SB_IS_COMPRESSED)) { - ntfs_debug("Found uncompressed sub-block."); - /* This sb is not compressed, just copy it into destination. */ - - /* Advance source position to first data byte. */ - cb += 2; - - /* An uncompressed sb must be full size. */ - if (cb_sb_end - cb != NTFS_SB_SIZE) - goto return_overflow; - - /* Copy the block and advance the source position. */ - memcpy(dp_addr, cb, NTFS_SB_SIZE); - cb += NTFS_SB_SIZE; - - /* Advance destination position to next sub-block. */ - *dest_ofs += NTFS_SB_SIZE; - if (!(*dest_ofs &= ~PAGE_MASK)) { -finalize_page: - /* - * First stage: add current page index to array of - * completed pages. - */ - completed_pages[nr_completed_pages++] = *dest_index; - if (++*dest_index > dest_max_index) - goto return_overflow; - } - goto do_next_sb; - } - ntfs_debug("Found compressed sub-block."); - /* This sb is compressed, decompress it into destination. */ - - /* Setup destination pointers. */ - dp_sb_start = dp_addr; - dp_sb_end = dp_sb_start + NTFS_SB_SIZE; - - /* Forward to the first tag in the sub-block. */ - cb += 2; -do_next_tag: - if (cb == cb_sb_end) { - /* Check if the decompressed sub-block was not full-length. */ - if (dp_addr < dp_sb_end) { - int nr_bytes = do_sb_end - *dest_ofs; - - ntfs_debug("Filling incomplete sub-block with " - "zeroes."); - /* Zero remainder and update destination position. */ - memset(dp_addr, 0, nr_bytes); - *dest_ofs += nr_bytes; - } - /* We have finished the current sub-block. */ - if (!(*dest_ofs &= ~PAGE_MASK)) - goto finalize_page; - goto do_next_sb; - } - - /* Check we are still in range. */ - if (cb > cb_sb_end || dp_addr > dp_sb_end) - goto return_overflow; - - /* Get the next tag and advance to first token. */ - tag = *cb++; - - /* Parse the eight tokens described by the tag. */ - for (token = 0; token < 8; token++, tag >>= 1) { - u16 lg, pt, length, max_non_overlap; - register u16 i; - u8 *dp_back_addr; - - /* Check if we are done / still in range. */ - if (cb >= cb_sb_end || dp_addr > dp_sb_end) - break; - - /* Determine token type and parse appropriately.*/ - if ((tag & NTFS_TOKEN_MASK) == NTFS_SYMBOL_TOKEN) { - /* - * We have a symbol token, copy the symbol across, and - * advance the source and destination positions. - */ - *dp_addr++ = *cb++; - ++*dest_ofs; - - /* Continue with the next token. */ - continue; - } - - /* - * We have a phrase token. Make sure it is not the first tag in - * the sb as this is illegal and would confuse the code below. - */ - if (dp_addr == dp_sb_start) - goto return_overflow; - - /* - * Determine the number of bytes to go back (p) and the number - * of bytes to copy (l). We use an optimized algorithm in which - * we first calculate log2(current destination position in sb), - * which allows determination of l and p in O(1) rather than - * O(n). We just need an arch-optimized log2() function now. - */ - lg = 0; - for (i = *dest_ofs - do_sb_start - 1; i >= 0x10; i >>= 1) - lg++; - - /* Get the phrase token into i. */ - pt = le16_to_cpup((le16*)cb); - - /* - * Calculate starting position of the byte sequence in - * the destination using the fact that p = (pt >> (12 - lg)) + 1 - * and make sure we don't go too far back. - */ - dp_back_addr = dp_addr - (pt >> (12 - lg)) - 1; - if (dp_back_addr < dp_sb_start) - goto return_overflow; - - /* Now calculate the length of the byte sequence. */ - length = (pt & (0xfff >> lg)) + 3; - - /* Advance destination position and verify it is in range. */ - *dest_ofs += length; - if (*dest_ofs > do_sb_end) - goto return_overflow; - - /* The number of non-overlapping bytes. */ - max_non_overlap = dp_addr - dp_back_addr; - - if (length <= max_non_overlap) { - /* The byte sequence doesn't overlap, just copy it. */ - memcpy(dp_addr, dp_back_addr, length); - - /* Advance destination pointer. */ - dp_addr += length; - } else { - /* - * The byte sequence does overlap, copy non-overlapping - * part and then do a slow byte by byte copy for the - * overlapping part. Also, advance the destination - * pointer. - */ - memcpy(dp_addr, dp_back_addr, max_non_overlap); - dp_addr += max_non_overlap; - dp_back_addr += max_non_overlap; - length -= max_non_overlap; - while (length--) - *dp_addr++ = *dp_back_addr++; - } - - /* Advance source position and continue with the next token. */ - cb += 2; - } - - /* No tokens left in the current tag. Continue with the next tag. */ - goto do_next_tag; - -return_overflow: - ntfs_error(NULL, "Failed. Returning -EOVERFLOW."); - goto return_error; -} - -/** - * ntfs_read_compressed_block - read a compressed block into the page cache - * @page: locked page in the compression block(s) we need to read - * - * When we are called the page has already been verified to be locked and the - * attribute is known to be non-resident, not encrypted, but compressed. - * - * 1. Determine which compression block(s) @page is in. - * 2. Get hold of all pages corresponding to this/these compression block(s). - * 3. Read the (first) compression block. - * 4. Decompress it into the corresponding pages. - * 5. Throw the compressed data away and proceed to 3. for the next compression - * block or return success if no more compression blocks left. - * - * Warning: We have to be careful what we do about existing pages. They might - * have been written to so that we would lose data if we were to just overwrite - * them with the out-of-date uncompressed data. - * - * FIXME: For PAGE_SIZE > cb_size we are not doing the Right Thing(TM) at - * the end of the file I think. We need to detect this case and zero the out - * of bounds remainder of the page in question and mark it as handled. At the - * moment we would just return -EIO on such a page. This bug will only become - * apparent if pages are above 8kiB and the NTFS volume only uses 512 byte - * clusters so is probably not going to be seen by anyone. Still this should - * be fixed. (AIA) - * - * FIXME: Again for PAGE_SIZE > cb_size we are screwing up both in - * handling sparse and compressed cbs. (AIA) - * - * FIXME: At the moment we don't do any zeroing out in the case that - * initialized_size is less than data_size. This should be safe because of the - * nature of the compression algorithm used. Just in case we check and output - * an error message in read inode if the two sizes are not equal for a - * compressed file. (AIA) - */ -int ntfs_read_compressed_block(struct page *page) -{ - loff_t i_size; - s64 initialized_size; - struct address_space *mapping = page->mapping; - ntfs_inode *ni = NTFS_I(mapping->host); - ntfs_volume *vol = ni->vol; - struct super_block *sb = vol->sb; - runlist_element *rl; - unsigned long flags, block_size = sb->s_blocksize; - unsigned char block_size_bits = sb->s_blocksize_bits; - u8 *cb, *cb_pos, *cb_end; - struct buffer_head **bhs; - unsigned long offset, index = page->index; - u32 cb_size = ni->itype.compressed.block_size; - u64 cb_size_mask = cb_size - 1UL; - VCN vcn; - LCN lcn; - /* The first wanted vcn (minimum alignment is PAGE_SIZE). */ - VCN start_vcn = (((s64)index << PAGE_SHIFT) & ~cb_size_mask) >> - vol->cluster_size_bits; - /* - * The first vcn after the last wanted vcn (minimum alignment is again - * PAGE_SIZE. - */ - VCN end_vcn = ((((s64)(index + 1UL) << PAGE_SHIFT) + cb_size - 1) - & ~cb_size_mask) >> vol->cluster_size_bits; - /* Number of compression blocks (cbs) in the wanted vcn range. */ - unsigned int nr_cbs = (end_vcn - start_vcn) << vol->cluster_size_bits - >> ni->itype.compressed.block_size_bits; - /* - * Number of pages required to store the uncompressed data from all - * compression blocks (cbs) overlapping @page. Due to alignment - * guarantees of start_vcn and end_vcn, no need to round up here. - */ - unsigned int nr_pages = (end_vcn - start_vcn) << - vol->cluster_size_bits >> PAGE_SHIFT; - unsigned int xpage, max_page, cur_page, cur_ofs, i; - unsigned int cb_clusters, cb_max_ofs; - int block, max_block, cb_max_page, bhs_size, nr_bhs, err = 0; - struct page **pages; - int *completed_pages; - unsigned char xpage_done = 0; - - ntfs_debug("Entering, page->index = 0x%lx, cb_size = 0x%x, nr_pages = " - "%i.", index, cb_size, nr_pages); - /* - * Bad things happen if we get here for anything that is not an - * unnamed $DATA attribute. - */ - BUG_ON(ni->type != AT_DATA); - BUG_ON(ni->name_len); - - pages = kmalloc_array(nr_pages, sizeof(struct page *), GFP_NOFS); - completed_pages = kmalloc_array(nr_pages + 1, sizeof(int), GFP_NOFS); - - /* Allocate memory to store the buffer heads we need. */ - bhs_size = cb_size / block_size * sizeof(struct buffer_head *); - bhs = kmalloc(bhs_size, GFP_NOFS); - - if (unlikely(!pages || !bhs || !completed_pages)) { - kfree(bhs); - kfree(pages); - kfree(completed_pages); - unlock_page(page); - ntfs_error(vol->sb, "Failed to allocate internal buffers."); - return -ENOMEM; - } - - /* - * We have already been given one page, this is the one we must do. - * Once again, the alignment guarantees keep it simple. - */ - offset = start_vcn << vol->cluster_size_bits >> PAGE_SHIFT; - xpage = index - offset; - pages[xpage] = page; - /* - * The remaining pages need to be allocated and inserted into the page - * cache, alignment guarantees keep all the below much simpler. (-8 - */ - read_lock_irqsave(&ni->size_lock, flags); - i_size = i_size_read(VFS_I(ni)); - initialized_size = ni->initialized_size; - read_unlock_irqrestore(&ni->size_lock, flags); - max_page = ((i_size + PAGE_SIZE - 1) >> PAGE_SHIFT) - - offset; - /* Is the page fully outside i_size? (truncate in progress) */ - if (xpage >= max_page) { - kfree(bhs); - kfree(pages); - kfree(completed_pages); - zero_user(page, 0, PAGE_SIZE); - ntfs_debug("Compressed read outside i_size - truncated?"); - SetPageUptodate(page); - unlock_page(page); - return 0; - } - if (nr_pages < max_page) - max_page = nr_pages; - for (i = 0; i < max_page; i++, offset++) { - if (i != xpage) - pages[i] = grab_cache_page_nowait(mapping, offset); - page = pages[i]; - if (page) { - /* - * We only (re)read the page if it isn't already read - * in and/or dirty or we would be losing data or at - * least wasting our time. - */ - if (!PageDirty(page) && (!PageUptodate(page) || - PageError(page))) { - ClearPageError(page); - kmap(page); - continue; - } - unlock_page(page); - put_page(page); - pages[i] = NULL; - } - } - - /* - * We have the runlist, and all the destination pages we need to fill. - * Now read the first compression block. - */ - cur_page = 0; - cur_ofs = 0; - cb_clusters = ni->itype.compressed.block_clusters; -do_next_cb: - nr_cbs--; - nr_bhs = 0; - - /* Read all cb buffer heads one cluster at a time. */ - rl = NULL; - for (vcn = start_vcn, start_vcn += cb_clusters; vcn < start_vcn; - vcn++) { - bool is_retry = false; - - if (!rl) { -lock_retry_remap: - down_read(&ni->runlist.lock); - rl = ni->runlist.rl; - } - if (likely(rl != NULL)) { - /* Seek to element containing target vcn. */ - while (rl->length && rl[1].vcn <= vcn) - rl++; - lcn = ntfs_rl_vcn_to_lcn(rl, vcn); - } else - lcn = LCN_RL_NOT_MAPPED; - ntfs_debug("Reading vcn = 0x%llx, lcn = 0x%llx.", - (unsigned long long)vcn, - (unsigned long long)lcn); - if (lcn < 0) { - /* - * When we reach the first sparse cluster we have - * finished with the cb. - */ - if (lcn == LCN_HOLE) - break; - if (is_retry || lcn != LCN_RL_NOT_MAPPED) - goto rl_err; - is_retry = true; - /* - * Attempt to map runlist, dropping lock for the - * duration. - */ - up_read(&ni->runlist.lock); - if (!ntfs_map_runlist(ni, vcn)) - goto lock_retry_remap; - goto map_rl_err; - } - block = lcn << vol->cluster_size_bits >> block_size_bits; - /* Read the lcn from device in chunks of block_size bytes. */ - max_block = block + (vol->cluster_size >> block_size_bits); - do { - ntfs_debug("block = 0x%x.", block); - if (unlikely(!(bhs[nr_bhs] = sb_getblk(sb, block)))) - goto getblk_err; - nr_bhs++; - } while (++block < max_block); - } - - /* Release the lock if we took it. */ - if (rl) - up_read(&ni->runlist.lock); - - /* Setup and initiate io on all buffer heads. */ - for (i = 0; i < nr_bhs; i++) { - struct buffer_head *tbh = bhs[i]; - - if (!trylock_buffer(tbh)) - continue; - if (unlikely(buffer_uptodate(tbh))) { - unlock_buffer(tbh); - continue; - } - get_bh(tbh); - tbh->b_end_io = end_buffer_read_sync; - submit_bh(REQ_OP_READ, tbh); - } - - /* Wait for io completion on all buffer heads. */ - for (i = 0; i < nr_bhs; i++) { - struct buffer_head *tbh = bhs[i]; - - if (buffer_uptodate(tbh)) - continue; - wait_on_buffer(tbh); - /* - * We need an optimization barrier here, otherwise we start - * hitting the below fixup code when accessing a loopback - * mounted ntfs partition. This indicates either there is a - * race condition in the loop driver or, more likely, gcc - * overoptimises the code without the barrier and it doesn't - * do the Right Thing(TM). - */ - barrier(); - if (unlikely(!buffer_uptodate(tbh))) { - ntfs_warning(vol->sb, "Buffer is unlocked but not " - "uptodate! Unplugging the disk queue " - "and rescheduling."); - get_bh(tbh); - io_schedule(); - put_bh(tbh); - if (unlikely(!buffer_uptodate(tbh))) - goto read_err; - ntfs_warning(vol->sb, "Buffer is now uptodate. Good."); - } - } - - /* - * Get the compression buffer. We must not sleep any more - * until we are finished with it. - */ - spin_lock(&ntfs_cb_lock); - cb = ntfs_compression_buffer; - - BUG_ON(!cb); - - cb_pos = cb; - cb_end = cb + cb_size; - - /* Copy the buffer heads into the contiguous buffer. */ - for (i = 0; i < nr_bhs; i++) { - memcpy(cb_pos, bhs[i]->b_data, block_size); - cb_pos += block_size; - } - - /* Just a precaution. */ - if (cb_pos + 2 <= cb + cb_size) - *(u16*)cb_pos = 0; - - /* Reset cb_pos back to the beginning. */ - cb_pos = cb; - - /* We now have both source (if present) and destination. */ - ntfs_debug("Successfully read the compression block."); - - /* The last page and maximum offset within it for the current cb. */ - cb_max_page = (cur_page << PAGE_SHIFT) + cur_ofs + cb_size; - cb_max_ofs = cb_max_page & ~PAGE_MASK; - cb_max_page >>= PAGE_SHIFT; - - /* Catch end of file inside a compression block. */ - if (cb_max_page > max_page) - cb_max_page = max_page; - - if (vcn == start_vcn - cb_clusters) { - /* Sparse cb, zero out page range overlapping the cb. */ - ntfs_debug("Found sparse compression block."); - /* We can sleep from now on, so we drop lock. */ - spin_unlock(&ntfs_cb_lock); - if (cb_max_ofs) - cb_max_page--; - for (; cur_page < cb_max_page; cur_page++) { - page = pages[cur_page]; - if (page) { - if (likely(!cur_ofs)) - clear_page(page_address(page)); - else - memset(page_address(page) + cur_ofs, 0, - PAGE_SIZE - - cur_ofs); - flush_dcache_page(page); - kunmap(page); - SetPageUptodate(page); - unlock_page(page); - if (cur_page == xpage) - xpage_done = 1; - else - put_page(page); - pages[cur_page] = NULL; - } - cb_pos += PAGE_SIZE - cur_ofs; - cur_ofs = 0; - if (cb_pos >= cb_end) - break; - } - /* If we have a partial final page, deal with it now. */ - if (cb_max_ofs && cb_pos < cb_end) { - page = pages[cur_page]; - if (page) - memset(page_address(page) + cur_ofs, 0, - cb_max_ofs - cur_ofs); - /* - * No need to update cb_pos at this stage: - * cb_pos += cb_max_ofs - cur_ofs; - */ - cur_ofs = cb_max_ofs; - } - } else if (vcn == start_vcn) { - /* We can't sleep so we need two stages. */ - unsigned int cur2_page = cur_page; - unsigned int cur_ofs2 = cur_ofs; - u8 *cb_pos2 = cb_pos; - - ntfs_debug("Found uncompressed compression block."); - /* Uncompressed cb, copy it to the destination pages. */ - /* - * TODO: As a big optimization, we could detect this case - * before we read all the pages and use block_read_full_folio() - * on all full pages instead (we still have to treat partial - * pages especially but at least we are getting rid of the - * synchronous io for the majority of pages. - * Or if we choose not to do the read-ahead/-behind stuff, we - * could just return block_read_full_folio(pages[xpage]) as long - * as PAGE_SIZE <= cb_size. - */ - if (cb_max_ofs) - cb_max_page--; - /* First stage: copy data into destination pages. */ - for (; cur_page < cb_max_page; cur_page++) { - page = pages[cur_page]; - if (page) - memcpy(page_address(page) + cur_ofs, cb_pos, - PAGE_SIZE - cur_ofs); - cb_pos += PAGE_SIZE - cur_ofs; - cur_ofs = 0; - if (cb_pos >= cb_end) - break; - } - /* If we have a partial final page, deal with it now. */ - if (cb_max_ofs && cb_pos < cb_end) { - page = pages[cur_page]; - if (page) - memcpy(page_address(page) + cur_ofs, cb_pos, - cb_max_ofs - cur_ofs); - cb_pos += cb_max_ofs - cur_ofs; - cur_ofs = cb_max_ofs; - } - /* We can sleep from now on, so drop lock. */ - spin_unlock(&ntfs_cb_lock); - /* Second stage: finalize pages. */ - for (; cur2_page < cb_max_page; cur2_page++) { - page = pages[cur2_page]; - if (page) { - /* - * If we are outside the initialized size, zero - * the out of bounds page range. - */ - handle_bounds_compressed_page(page, i_size, - initialized_size); - flush_dcache_page(page); - kunmap(page); - SetPageUptodate(page); - unlock_page(page); - if (cur2_page == xpage) - xpage_done = 1; - else - put_page(page); - pages[cur2_page] = NULL; - } - cb_pos2 += PAGE_SIZE - cur_ofs2; - cur_ofs2 = 0; - if (cb_pos2 >= cb_end) - break; - } - } else { - /* Compressed cb, decompress it into the destination page(s). */ - unsigned int prev_cur_page = cur_page; - - ntfs_debug("Found compressed compression block."); - err = ntfs_decompress(pages, completed_pages, &cur_page, - &cur_ofs, cb_max_page, cb_max_ofs, xpage, - &xpage_done, cb_pos, cb_size - (cb_pos - cb), - i_size, initialized_size); - /* - * We can sleep from now on, lock already dropped by - * ntfs_decompress(). - */ - if (err) { - ntfs_error(vol->sb, "ntfs_decompress() failed in inode " - "0x%lx with error code %i. Skipping " - "this compression block.", - ni->mft_no, -err); - /* Release the unfinished pages. */ - for (; prev_cur_page < cur_page; prev_cur_page++) { - page = pages[prev_cur_page]; - if (page) { - flush_dcache_page(page); - kunmap(page); - unlock_page(page); - if (prev_cur_page != xpage) - put_page(page); - pages[prev_cur_page] = NULL; - } - } - } - } - - /* Release the buffer heads. */ - for (i = 0; i < nr_bhs; i++) - brelse(bhs[i]); - - /* Do we have more work to do? */ - if (nr_cbs) - goto do_next_cb; - - /* We no longer need the list of buffer heads. */ - kfree(bhs); - - /* Clean up if we have any pages left. Should never happen. */ - for (cur_page = 0; cur_page < max_page; cur_page++) { - page = pages[cur_page]; - if (page) { - ntfs_error(vol->sb, "Still have pages left! " - "Terminating them with extreme " - "prejudice. Inode 0x%lx, page index " - "0x%lx.", ni->mft_no, page->index); - flush_dcache_page(page); - kunmap(page); - unlock_page(page); - if (cur_page != xpage) - put_page(page); - pages[cur_page] = NULL; - } - } - - /* We no longer need the list of pages. */ - kfree(pages); - kfree(completed_pages); - - /* If we have completed the requested page, we return success. */ - if (likely(xpage_done)) - return 0; - - ntfs_debug("Failed. Returning error code %s.", err == -EOVERFLOW ? - "EOVERFLOW" : (!err ? "EIO" : "unknown error")); - return err < 0 ? err : -EIO; - -read_err: - ntfs_error(vol->sb, "IO error while reading compressed data."); - /* Release the buffer heads. */ - for (i = 0; i < nr_bhs; i++) - brelse(bhs[i]); - goto err_out; - -map_rl_err: - ntfs_error(vol->sb, "ntfs_map_runlist() failed. Cannot read " - "compression block."); - goto err_out; - -rl_err: - up_read(&ni->runlist.lock); - ntfs_error(vol->sb, "ntfs_rl_vcn_to_lcn() failed. Cannot read " - "compression block."); - goto err_out; - -getblk_err: - up_read(&ni->runlist.lock); - ntfs_error(vol->sb, "getblk() failed. Cannot read compression block."); - -err_out: - kfree(bhs); - for (i = cur_page; i < max_page; i++) { - page = pages[i]; - if (page) { - flush_dcache_page(page); - kunmap(page); - unlock_page(page); - if (i != xpage) - put_page(page); - } - } - kfree(pages); - kfree(completed_pages); - return -EIO; -} diff --git a/fs/ntfs/debug.c b/fs/ntfs/debug.c deleted file mode 100644 index a3c1c5656f8f..000000000000 --- a/fs/ntfs/debug.c +++ /dev/null @@ -1,159 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * debug.c - NTFS kernel debug support. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2004 Anton Altaparmakov - */ -#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt -#include "debug.h" - -/** - * __ntfs_warning - output a warning to the syslog - * @function: name of function outputting the warning - * @sb: super block of mounted ntfs filesystem - * @fmt: warning string containing format specifications - * @...: a variable number of arguments specified in @fmt - * - * Outputs a warning to the syslog for the mounted ntfs filesystem described - * by @sb. - * - * @fmt and the corresponding @... is printf style format string containing - * the warning string and the corresponding format arguments, respectively. - * - * @function is the name of the function from which __ntfs_warning is being - * called. - * - * Note, you should be using debug.h::ntfs_warning(@sb, @fmt, @...) instead - * as this provides the @function parameter automatically. - */ -void __ntfs_warning(const char *function, const struct super_block *sb, - const char *fmt, ...) -{ - struct va_format vaf; - va_list args; - int flen = 0; - -#ifndef DEBUG - if (!printk_ratelimit()) - return; -#endif - if (function) - flen = strlen(function); - va_start(args, fmt); - vaf.fmt = fmt; - vaf.va = &args; - if (sb) - pr_warn("(device %s): %s(): %pV\n", - sb->s_id, flen ? function : "", &vaf); - else - pr_warn("%s(): %pV\n", flen ? function : "", &vaf); - va_end(args); -} - -/** - * __ntfs_error - output an error to the syslog - * @function: name of function outputting the error - * @sb: super block of mounted ntfs filesystem - * @fmt: error string containing format specifications - * @...: a variable number of arguments specified in @fmt - * - * Outputs an error to the syslog for the mounted ntfs filesystem described - * by @sb. - * - * @fmt and the corresponding @... is printf style format string containing - * the error string and the corresponding format arguments, respectively. - * - * @function is the name of the function from which __ntfs_error is being - * called. - * - * Note, you should be using debug.h::ntfs_error(@sb, @fmt, @...) instead - * as this provides the @function parameter automatically. - */ -void __ntfs_error(const char *function, const struct super_block *sb, - const char *fmt, ...) -{ - struct va_format vaf; - va_list args; - int flen = 0; - -#ifndef DEBUG - if (!printk_ratelimit()) - return; -#endif - if (function) - flen = strlen(function); - va_start(args, fmt); - vaf.fmt = fmt; - vaf.va = &args; - if (sb) - pr_err("(device %s): %s(): %pV\n", - sb->s_id, flen ? function : "", &vaf); - else - pr_err("%s(): %pV\n", flen ? function : "", &vaf); - va_end(args); -} - -#ifdef DEBUG - -/* If 1, output debug messages, and if 0, don't. */ -int debug_msgs = 0; - -void __ntfs_debug(const char *file, int line, const char *function, - const char *fmt, ...) -{ - struct va_format vaf; - va_list args; - int flen = 0; - - if (!debug_msgs) - return; - if (function) - flen = strlen(function); - va_start(args, fmt); - vaf.fmt = fmt; - vaf.va = &args; - pr_debug("(%s, %d): %s(): %pV", file, line, flen ? function : "", &vaf); - va_end(args); -} - -/* Dump a runlist. Caller has to provide synchronisation for @rl. */ -void ntfs_debug_dump_runlist(const runlist_element *rl) -{ - int i; - const char *lcn_str[5] = { "LCN_HOLE ", "LCN_RL_NOT_MAPPED", - "LCN_ENOENT ", "LCN_unknown " }; - - if (!debug_msgs) - return; - pr_debug("Dumping runlist (values in hex):\n"); - if (!rl) { - pr_debug("Run list not present.\n"); - return; - } - pr_debug("VCN LCN Run length\n"); - for (i = 0; ; i++) { - LCN lcn = (rl + i)->lcn; - - if (lcn < (LCN)0) { - int index = -lcn - 1; - - if (index > -LCN_ENOENT - 1) - index = 3; - pr_debug("%-16Lx %s %-16Lx%s\n", - (long long)(rl + i)->vcn, lcn_str[index], - (long long)(rl + i)->length, - (rl + i)->length ? "" : - " (runlist end)"); - } else - pr_debug("%-16Lx %-16Lx %-16Lx%s\n", - (long long)(rl + i)->vcn, - (long long)(rl + i)->lcn, - (long long)(rl + i)->length, - (rl + i)->length ? "" : - " (runlist end)"); - if (!(rl + i)->length) - break; - } -} - -#endif diff --git a/fs/ntfs/debug.h b/fs/ntfs/debug.h deleted file mode 100644 index 6fdef388f129..000000000000 --- a/fs/ntfs/debug.h +++ /dev/null @@ -1,57 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * debug.h - NTFS kernel debug support. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2004 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_DEBUG_H -#define _LINUX_NTFS_DEBUG_H - -#include - -#include "runlist.h" - -#ifdef DEBUG - -extern int debug_msgs; - -extern __printf(4, 5) -void __ntfs_debug(const char *file, int line, const char *function, - const char *format, ...); -/** - * ntfs_debug - write a debug level message to syslog - * @f: a printf format string containing the message - * @...: the variables to substitute into @f - * - * ntfs_debug() writes a DEBUG level message to the syslog but only if the - * driver was compiled with -DDEBUG. Otherwise, the call turns into a NOP. - */ -#define ntfs_debug(f, a...) \ - __ntfs_debug(__FILE__, __LINE__, __func__, f, ##a) - -extern void ntfs_debug_dump_runlist(const runlist_element *rl); - -#else /* !DEBUG */ - -#define ntfs_debug(fmt, ...) \ -do { \ - if (0) \ - no_printk(fmt, ##__VA_ARGS__); \ -} while (0) - -#define ntfs_debug_dump_runlist(rl) do {} while (0) - -#endif /* !DEBUG */ - -extern __printf(3, 4) -void __ntfs_warning(const char *function, const struct super_block *sb, - const char *fmt, ...); -#define ntfs_warning(sb, f, a...) __ntfs_warning(__func__, sb, f, ##a) - -extern __printf(3, 4) -void __ntfs_error(const char *function, const struct super_block *sb, - const char *fmt, ...); -#define ntfs_error(sb, f, a...) __ntfs_error(__func__, sb, f, ##a) - -#endif /* _LINUX_NTFS_DEBUG_H */ diff --git a/fs/ntfs/dir.c b/fs/ntfs/dir.c deleted file mode 100644 index 4596c90e7b7c..000000000000 --- a/fs/ntfs/dir.c +++ /dev/null @@ -1,1539 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * dir.c - NTFS kernel directory operations. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2007 Anton Altaparmakov - * Copyright (c) 2002 Richard Russon - */ - -#include -#include -#include - -#include "dir.h" -#include "aops.h" -#include "attrib.h" -#include "mft.h" -#include "debug.h" -#include "ntfs.h" - -/* - * The little endian Unicode string $I30 as a global constant. - */ -ntfschar I30[5] = { cpu_to_le16('$'), cpu_to_le16('I'), - cpu_to_le16('3'), cpu_to_le16('0'), 0 }; - -/** - * ntfs_lookup_inode_by_name - find an inode in a directory given its name - * @dir_ni: ntfs inode of the directory in which to search for the name - * @uname: Unicode name for which to search in the directory - * @uname_len: length of the name @uname in Unicode characters - * @res: return the found file name if necessary (see below) - * - * Look for an inode with name @uname in the directory with inode @dir_ni. - * ntfs_lookup_inode_by_name() walks the contents of the directory looking for - * the Unicode name. If the name is found in the directory, the corresponding - * inode number (>= 0) is returned as a mft reference in cpu format, i.e. it - * is a 64-bit number containing the sequence number. - * - * On error, a negative value is returned corresponding to the error code. In - * particular if the inode is not found -ENOENT is returned. Note that you - * can't just check the return value for being negative, you have to check the - * inode number for being negative which you can extract using MREC(return - * value). - * - * Note, @uname_len does not include the (optional) terminating NULL character. - * - * Note, we look for a case sensitive match first but we also look for a case - * insensitive match at the same time. If we find a case insensitive match, we - * save that for the case that we don't find an exact match, where we return - * the case insensitive match and setup @res (which we allocate!) with the mft - * reference, the file name type, length and with a copy of the little endian - * Unicode file name itself. If we match a file name which is in the DOS name - * space, we only return the mft reference and file name type in @res. - * ntfs_lookup() then uses this to find the long file name in the inode itself. - * This is to avoid polluting the dcache with short file names. We want them to - * work but we don't care for how quickly one can access them. This also fixes - * the dcache aliasing issues. - * - * Locking: - Caller must hold i_mutex on the directory. - * - Each page cache page in the index allocation mapping must be - * locked whilst being accessed otherwise we may find a corrupt - * page due to it being under ->writepage at the moment which - * applies the mst protection fixups before writing out and then - * removes them again after the write is complete after which it - * unlocks the page. - */ -MFT_REF ntfs_lookup_inode_by_name(ntfs_inode *dir_ni, const ntfschar *uname, - const int uname_len, ntfs_name **res) -{ - ntfs_volume *vol = dir_ni->vol; - struct super_block *sb = vol->sb; - MFT_RECORD *m; - INDEX_ROOT *ir; - INDEX_ENTRY *ie; - INDEX_ALLOCATION *ia; - u8 *index_end; - u64 mref; - ntfs_attr_search_ctx *ctx; - int err, rc; - VCN vcn, old_vcn; - struct address_space *ia_mapping; - struct page *page; - u8 *kaddr; - ntfs_name *name = NULL; - - BUG_ON(!S_ISDIR(VFS_I(dir_ni)->i_mode)); - BUG_ON(NInoAttr(dir_ni)); - /* Get hold of the mft record for the directory. */ - m = map_mft_record(dir_ni); - if (IS_ERR(m)) { - ntfs_error(sb, "map_mft_record() failed with error code %ld.", - -PTR_ERR(m)); - return ERR_MREF(PTR_ERR(m)); - } - ctx = ntfs_attr_get_search_ctx(dir_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - /* Find the index root attribute in the mft record. */ - err = ntfs_attr_lookup(AT_INDEX_ROOT, I30, 4, CASE_SENSITIVE, 0, NULL, - 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) { - ntfs_error(sb, "Index root attribute missing in " - "directory inode 0x%lx.", - dir_ni->mft_no); - err = -EIO; - } - goto err_out; - } - /* Get to the index root value (it's been verified in read_inode). */ - ir = (INDEX_ROOT*)((u8*)ctx->attr + - le16_to_cpu(ctx->attr->data.resident.value_offset)); - index_end = (u8*)&ir->index + le32_to_cpu(ir->index.index_length); - /* The first index entry. */ - ie = (INDEX_ENTRY*)((u8*)&ir->index + - le32_to_cpu(ir->index.entries_offset)); - /* - * Loop until we exceed valid memory (corruption case) or until we - * reach the last entry. - */ - for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) { - /* Bounds checks. */ - if ((u8*)ie < (u8*)ctx->mrec || (u8*)ie + - sizeof(INDEX_ENTRY_HEADER) > index_end || - (u8*)ie + le16_to_cpu(ie->key_length) > - index_end) - goto dir_err_out; - /* - * The last entry cannot contain a name. It can however contain - * a pointer to a child node in the B+tree so we just break out. - */ - if (ie->flags & INDEX_ENTRY_END) - break; - /* - * We perform a case sensitive comparison and if that matches - * we are done and return the mft reference of the inode (i.e. - * the inode number together with the sequence number for - * consistency checking). We convert it to cpu format before - * returning. - */ - if (ntfs_are_names_equal(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, - CASE_SENSITIVE, vol->upcase, vol->upcase_len)) { -found_it: - /* - * We have a perfect match, so we don't need to care - * about having matched imperfectly before, so we can - * free name and set *res to NULL. - * However, if the perfect match is a short file name, - * we need to signal this through *res, so that - * ntfs_lookup() can fix dcache aliasing issues. - * As an optimization we just reuse an existing - * allocation of *res. - */ - if (ie->key.file_name.file_name_type == FILE_NAME_DOS) { - if (!name) { - name = kmalloc(sizeof(ntfs_name), - GFP_NOFS); - if (!name) { - err = -ENOMEM; - goto err_out; - } - } - name->mref = le64_to_cpu( - ie->data.dir.indexed_file); - name->type = FILE_NAME_DOS; - name->len = 0; - *res = name; - } else { - kfree(name); - *res = NULL; - } - mref = le64_to_cpu(ie->data.dir.indexed_file); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(dir_ni); - return mref; - } - /* - * For a case insensitive mount, we also perform a case - * insensitive comparison (provided the file name is not in the - * POSIX namespace). If the comparison matches, and the name is - * in the WIN32 namespace, we cache the filename in *res so - * that the caller, ntfs_lookup(), can work on it. If the - * comparison matches, and the name is in the DOS namespace, we - * only cache the mft reference and the file name type (we set - * the name length to zero for simplicity). - */ - if (!NVolCaseSensitive(vol) && - ie->key.file_name.file_name_type && - ntfs_are_names_equal(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, - IGNORE_CASE, vol->upcase, vol->upcase_len)) { - int name_size = sizeof(ntfs_name); - u8 type = ie->key.file_name.file_name_type; - u8 len = ie->key.file_name.file_name_length; - - /* Only one case insensitive matching name allowed. */ - if (name) { - ntfs_error(sb, "Found already allocated name " - "in phase 1. Please run chkdsk " - "and if that doesn't find any " - "errors please report you saw " - "this message to " - "linux-ntfs-dev@lists." - "sourceforge.net."); - goto dir_err_out; - } - - if (type != FILE_NAME_DOS) - name_size += len * sizeof(ntfschar); - name = kmalloc(name_size, GFP_NOFS); - if (!name) { - err = -ENOMEM; - goto err_out; - } - name->mref = le64_to_cpu(ie->data.dir.indexed_file); - name->type = type; - if (type != FILE_NAME_DOS) { - name->len = len; - memcpy(name->name, ie->key.file_name.file_name, - len * sizeof(ntfschar)); - } else - name->len = 0; - *res = name; - } - /* - * Not a perfect match, need to do full blown collation so we - * know which way in the B+tree we have to go. - */ - rc = ntfs_collate_names(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, 1, - IGNORE_CASE, vol->upcase, vol->upcase_len); - /* - * If uname collates before the name of the current entry, there - * is definitely no such name in this index but we might need to - * descend into the B+tree so we just break out of the loop. - */ - if (rc == -1) - break; - /* The names are not equal, continue the search. */ - if (rc) - continue; - /* - * Names match with case insensitive comparison, now try the - * case sensitive comparison, which is required for proper - * collation. - */ - rc = ntfs_collate_names(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, 1, - CASE_SENSITIVE, vol->upcase, vol->upcase_len); - if (rc == -1) - break; - if (rc) - continue; - /* - * Perfect match, this will never happen as the - * ntfs_are_names_equal() call will have gotten a match but we - * still treat it correctly. - */ - goto found_it; - } - /* - * We have finished with this index without success. Check for the - * presence of a child node and if not present return -ENOENT, unless - * we have got a matching name cached in name in which case return the - * mft reference associated with it. - */ - if (!(ie->flags & INDEX_ENTRY_NODE)) { - if (name) { - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(dir_ni); - return name->mref; - } - ntfs_debug("Entry not found."); - err = -ENOENT; - goto err_out; - } /* Child node present, descend into it. */ - /* Consistency check: Verify that an index allocation exists. */ - if (!NInoIndexAllocPresent(dir_ni)) { - ntfs_error(sb, "No index allocation attribute but index entry " - "requires one. Directory inode 0x%lx is " - "corrupt or driver bug.", dir_ni->mft_no); - goto err_out; - } - /* Get the starting vcn of the index_block holding the child node. */ - vcn = sle64_to_cpup((sle64*)((u8*)ie + le16_to_cpu(ie->length) - 8)); - ia_mapping = VFS_I(dir_ni)->i_mapping; - /* - * We are done with the index root and the mft record. Release them, - * otherwise we deadlock with ntfs_map_page(). - */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(dir_ni); - m = NULL; - ctx = NULL; -descend_into_child_node: - /* - * Convert vcn to index into the index allocation attribute in units - * of PAGE_SIZE and map the page cache page, reading it from - * disk if necessary. - */ - page = ntfs_map_page(ia_mapping, vcn << - dir_ni->itype.index.vcn_size_bits >> PAGE_SHIFT); - if (IS_ERR(page)) { - ntfs_error(sb, "Failed to map directory index page, error %ld.", - -PTR_ERR(page)); - err = PTR_ERR(page); - goto err_out; - } - lock_page(page); - kaddr = (u8*)page_address(page); -fast_descend_into_child_node: - /* Get to the index allocation block. */ - ia = (INDEX_ALLOCATION*)(kaddr + ((vcn << - dir_ni->itype.index.vcn_size_bits) & ~PAGE_MASK)); - /* Bounds checks. */ - if ((u8*)ia < kaddr || (u8*)ia > kaddr + PAGE_SIZE) { - ntfs_error(sb, "Out of bounds check failed. Corrupt directory " - "inode 0x%lx or driver bug.", dir_ni->mft_no); - goto unm_err_out; - } - /* Catch multi sector transfer fixup errors. */ - if (unlikely(!ntfs_is_indx_record(ia->magic))) { - ntfs_error(sb, "Directory index record with vcn 0x%llx is " - "corrupt. Corrupt inode 0x%lx. Run chkdsk.", - (unsigned long long)vcn, dir_ni->mft_no); - goto unm_err_out; - } - if (sle64_to_cpu(ia->index_block_vcn) != vcn) { - ntfs_error(sb, "Actual VCN (0x%llx) of index buffer is " - "different from expected VCN (0x%llx). " - "Directory inode 0x%lx is corrupt or driver " - "bug.", (unsigned long long) - sle64_to_cpu(ia->index_block_vcn), - (unsigned long long)vcn, dir_ni->mft_no); - goto unm_err_out; - } - if (le32_to_cpu(ia->index.allocated_size) + 0x18 != - dir_ni->itype.index.block_size) { - ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode " - "0x%lx has a size (%u) differing from the " - "directory specified size (%u). Directory " - "inode is corrupt or driver bug.", - (unsigned long long)vcn, dir_ni->mft_no, - le32_to_cpu(ia->index.allocated_size) + 0x18, - dir_ni->itype.index.block_size); - goto unm_err_out; - } - index_end = (u8*)ia + dir_ni->itype.index.block_size; - if (index_end > kaddr + PAGE_SIZE) { - ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode " - "0x%lx crosses page boundary. Impossible! " - "Cannot access! This is probably a bug in the " - "driver.", (unsigned long long)vcn, - dir_ni->mft_no); - goto unm_err_out; - } - index_end = (u8*)&ia->index + le32_to_cpu(ia->index.index_length); - if (index_end > (u8*)ia + dir_ni->itype.index.block_size) { - ntfs_error(sb, "Size of index buffer (VCN 0x%llx) of directory " - "inode 0x%lx exceeds maximum size.", - (unsigned long long)vcn, dir_ni->mft_no); - goto unm_err_out; - } - /* The first index entry. */ - ie = (INDEX_ENTRY*)((u8*)&ia->index + - le32_to_cpu(ia->index.entries_offset)); - /* - * Iterate similar to above big loop but applied to index buffer, thus - * loop until we exceed valid memory (corruption case) or until we - * reach the last entry. - */ - for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) { - /* Bounds check. */ - if ((u8*)ie < (u8*)ia || (u8*)ie + - sizeof(INDEX_ENTRY_HEADER) > index_end || - (u8*)ie + le16_to_cpu(ie->key_length) > - index_end) { - ntfs_error(sb, "Index entry out of bounds in " - "directory inode 0x%lx.", - dir_ni->mft_no); - goto unm_err_out; - } - /* - * The last entry cannot contain a name. It can however contain - * a pointer to a child node in the B+tree so we just break out. - */ - if (ie->flags & INDEX_ENTRY_END) - break; - /* - * We perform a case sensitive comparison and if that matches - * we are done and return the mft reference of the inode (i.e. - * the inode number together with the sequence number for - * consistency checking). We convert it to cpu format before - * returning. - */ - if (ntfs_are_names_equal(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, - CASE_SENSITIVE, vol->upcase, vol->upcase_len)) { -found_it2: - /* - * We have a perfect match, so we don't need to care - * about having matched imperfectly before, so we can - * free name and set *res to NULL. - * However, if the perfect match is a short file name, - * we need to signal this through *res, so that - * ntfs_lookup() can fix dcache aliasing issues. - * As an optimization we just reuse an existing - * allocation of *res. - */ - if (ie->key.file_name.file_name_type == FILE_NAME_DOS) { - if (!name) { - name = kmalloc(sizeof(ntfs_name), - GFP_NOFS); - if (!name) { - err = -ENOMEM; - goto unm_err_out; - } - } - name->mref = le64_to_cpu( - ie->data.dir.indexed_file); - name->type = FILE_NAME_DOS; - name->len = 0; - *res = name; - } else { - kfree(name); - *res = NULL; - } - mref = le64_to_cpu(ie->data.dir.indexed_file); - unlock_page(page); - ntfs_unmap_page(page); - return mref; - } - /* - * For a case insensitive mount, we also perform a case - * insensitive comparison (provided the file name is not in the - * POSIX namespace). If the comparison matches, and the name is - * in the WIN32 namespace, we cache the filename in *res so - * that the caller, ntfs_lookup(), can work on it. If the - * comparison matches, and the name is in the DOS namespace, we - * only cache the mft reference and the file name type (we set - * the name length to zero for simplicity). - */ - if (!NVolCaseSensitive(vol) && - ie->key.file_name.file_name_type && - ntfs_are_names_equal(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, - IGNORE_CASE, vol->upcase, vol->upcase_len)) { - int name_size = sizeof(ntfs_name); - u8 type = ie->key.file_name.file_name_type; - u8 len = ie->key.file_name.file_name_length; - - /* Only one case insensitive matching name allowed. */ - if (name) { - ntfs_error(sb, "Found already allocated name " - "in phase 2. Please run chkdsk " - "and if that doesn't find any " - "errors please report you saw " - "this message to " - "linux-ntfs-dev@lists." - "sourceforge.net."); - unlock_page(page); - ntfs_unmap_page(page); - goto dir_err_out; - } - - if (type != FILE_NAME_DOS) - name_size += len * sizeof(ntfschar); - name = kmalloc(name_size, GFP_NOFS); - if (!name) { - err = -ENOMEM; - goto unm_err_out; - } - name->mref = le64_to_cpu(ie->data.dir.indexed_file); - name->type = type; - if (type != FILE_NAME_DOS) { - name->len = len; - memcpy(name->name, ie->key.file_name.file_name, - len * sizeof(ntfschar)); - } else - name->len = 0; - *res = name; - } - /* - * Not a perfect match, need to do full blown collation so we - * know which way in the B+tree we have to go. - */ - rc = ntfs_collate_names(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, 1, - IGNORE_CASE, vol->upcase, vol->upcase_len); - /* - * If uname collates before the name of the current entry, there - * is definitely no such name in this index but we might need to - * descend into the B+tree so we just break out of the loop. - */ - if (rc == -1) - break; - /* The names are not equal, continue the search. */ - if (rc) - continue; - /* - * Names match with case insensitive comparison, now try the - * case sensitive comparison, which is required for proper - * collation. - */ - rc = ntfs_collate_names(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, 1, - CASE_SENSITIVE, vol->upcase, vol->upcase_len); - if (rc == -1) - break; - if (rc) - continue; - /* - * Perfect match, this will never happen as the - * ntfs_are_names_equal() call will have gotten a match but we - * still treat it correctly. - */ - goto found_it2; - } - /* - * We have finished with this index buffer without success. Check for - * the presence of a child node. - */ - if (ie->flags & INDEX_ENTRY_NODE) { - if ((ia->index.flags & NODE_MASK) == LEAF_NODE) { - ntfs_error(sb, "Index entry with child node found in " - "a leaf node in directory inode 0x%lx.", - dir_ni->mft_no); - goto unm_err_out; - } - /* Child node present, descend into it. */ - old_vcn = vcn; - vcn = sle64_to_cpup((sle64*)((u8*)ie + - le16_to_cpu(ie->length) - 8)); - if (vcn >= 0) { - /* If vcn is in the same page cache page as old_vcn we - * recycle the mapped page. */ - if (old_vcn << vol->cluster_size_bits >> - PAGE_SHIFT == vcn << - vol->cluster_size_bits >> - PAGE_SHIFT) - goto fast_descend_into_child_node; - unlock_page(page); - ntfs_unmap_page(page); - goto descend_into_child_node; - } - ntfs_error(sb, "Negative child node vcn in directory inode " - "0x%lx.", dir_ni->mft_no); - goto unm_err_out; - } - /* - * No child node present, return -ENOENT, unless we have got a matching - * name cached in name in which case return the mft reference - * associated with it. - */ - if (name) { - unlock_page(page); - ntfs_unmap_page(page); - return name->mref; - } - ntfs_debug("Entry not found."); - err = -ENOENT; -unm_err_out: - unlock_page(page); - ntfs_unmap_page(page); -err_out: - if (!err) - err = -EIO; - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(dir_ni); - if (name) { - kfree(name); - *res = NULL; - } - return ERR_MREF(err); -dir_err_out: - ntfs_error(sb, "Corrupt directory. Aborting lookup."); - goto err_out; -} - -#if 0 - -// TODO: (AIA) -// The algorithm embedded in this code will be required for the time when we -// want to support adding of entries to directories, where we require correct -// collation of file names in order not to cause corruption of the filesystem. - -/** - * ntfs_lookup_inode_by_name - find an inode in a directory given its name - * @dir_ni: ntfs inode of the directory in which to search for the name - * @uname: Unicode name for which to search in the directory - * @uname_len: length of the name @uname in Unicode characters - * - * Look for an inode with name @uname in the directory with inode @dir_ni. - * ntfs_lookup_inode_by_name() walks the contents of the directory looking for - * the Unicode name. If the name is found in the directory, the corresponding - * inode number (>= 0) is returned as a mft reference in cpu format, i.e. it - * is a 64-bit number containing the sequence number. - * - * On error, a negative value is returned corresponding to the error code. In - * particular if the inode is not found -ENOENT is returned. Note that you - * can't just check the return value for being negative, you have to check the - * inode number for being negative which you can extract using MREC(return - * value). - * - * Note, @uname_len does not include the (optional) terminating NULL character. - */ -u64 ntfs_lookup_inode_by_name(ntfs_inode *dir_ni, const ntfschar *uname, - const int uname_len) -{ - ntfs_volume *vol = dir_ni->vol; - struct super_block *sb = vol->sb; - MFT_RECORD *m; - INDEX_ROOT *ir; - INDEX_ENTRY *ie; - INDEX_ALLOCATION *ia; - u8 *index_end; - u64 mref; - ntfs_attr_search_ctx *ctx; - int err, rc; - IGNORE_CASE_BOOL ic; - VCN vcn, old_vcn; - struct address_space *ia_mapping; - struct page *page; - u8 *kaddr; - - /* Get hold of the mft record for the directory. */ - m = map_mft_record(dir_ni); - if (IS_ERR(m)) { - ntfs_error(sb, "map_mft_record() failed with error code %ld.", - -PTR_ERR(m)); - return ERR_MREF(PTR_ERR(m)); - } - ctx = ntfs_attr_get_search_ctx(dir_ni, m); - if (!ctx) { - err = -ENOMEM; - goto err_out; - } - /* Find the index root attribute in the mft record. */ - err = ntfs_attr_lookup(AT_INDEX_ROOT, I30, 4, CASE_SENSITIVE, 0, NULL, - 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) { - ntfs_error(sb, "Index root attribute missing in " - "directory inode 0x%lx.", - dir_ni->mft_no); - err = -EIO; - } - goto err_out; - } - /* Get to the index root value (it's been verified in read_inode). */ - ir = (INDEX_ROOT*)((u8*)ctx->attr + - le16_to_cpu(ctx->attr->data.resident.value_offset)); - index_end = (u8*)&ir->index + le32_to_cpu(ir->index.index_length); - /* The first index entry. */ - ie = (INDEX_ENTRY*)((u8*)&ir->index + - le32_to_cpu(ir->index.entries_offset)); - /* - * Loop until we exceed valid memory (corruption case) or until we - * reach the last entry. - */ - for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) { - /* Bounds checks. */ - if ((u8*)ie < (u8*)ctx->mrec || (u8*)ie + - sizeof(INDEX_ENTRY_HEADER) > index_end || - (u8*)ie + le16_to_cpu(ie->key_length) > - index_end) - goto dir_err_out; - /* - * The last entry cannot contain a name. It can however contain - * a pointer to a child node in the B+tree so we just break out. - */ - if (ie->flags & INDEX_ENTRY_END) - break; - /* - * If the current entry has a name type of POSIX, the name is - * case sensitive and not otherwise. This has the effect of us - * not being able to access any POSIX file names which collate - * after the non-POSIX one when they only differ in case, but - * anyone doing screwy stuff like that deserves to burn in - * hell... Doing that kind of stuff on NT4 actually causes - * corruption on the partition even when using SP6a and Linux - * is not involved at all. - */ - ic = ie->key.file_name.file_name_type ? IGNORE_CASE : - CASE_SENSITIVE; - /* - * If the names match perfectly, we are done and return the - * mft reference of the inode (i.e. the inode number together - * with the sequence number for consistency checking. We - * convert it to cpu format before returning. - */ - if (ntfs_are_names_equal(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, ic, - vol->upcase, vol->upcase_len)) { -found_it: - mref = le64_to_cpu(ie->data.dir.indexed_file); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(dir_ni); - return mref; - } - /* - * Not a perfect match, need to do full blown collation so we - * know which way in the B+tree we have to go. - */ - rc = ntfs_collate_names(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, 1, - IGNORE_CASE, vol->upcase, vol->upcase_len); - /* - * If uname collates before the name of the current entry, there - * is definitely no such name in this index but we might need to - * descend into the B+tree so we just break out of the loop. - */ - if (rc == -1) - break; - /* The names are not equal, continue the search. */ - if (rc) - continue; - /* - * Names match with case insensitive comparison, now try the - * case sensitive comparison, which is required for proper - * collation. - */ - rc = ntfs_collate_names(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, 1, - CASE_SENSITIVE, vol->upcase, vol->upcase_len); - if (rc == -1) - break; - if (rc) - continue; - /* - * Perfect match, this will never happen as the - * ntfs_are_names_equal() call will have gotten a match but we - * still treat it correctly. - */ - goto found_it; - } - /* - * We have finished with this index without success. Check for the - * presence of a child node. - */ - if (!(ie->flags & INDEX_ENTRY_NODE)) { - /* No child node, return -ENOENT. */ - err = -ENOENT; - goto err_out; - } /* Child node present, descend into it. */ - /* Consistency check: Verify that an index allocation exists. */ - if (!NInoIndexAllocPresent(dir_ni)) { - ntfs_error(sb, "No index allocation attribute but index entry " - "requires one. Directory inode 0x%lx is " - "corrupt or driver bug.", dir_ni->mft_no); - goto err_out; - } - /* Get the starting vcn of the index_block holding the child node. */ - vcn = sle64_to_cpup((u8*)ie + le16_to_cpu(ie->length) - 8); - ia_mapping = VFS_I(dir_ni)->i_mapping; - /* - * We are done with the index root and the mft record. Release them, - * otherwise we deadlock with ntfs_map_page(). - */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(dir_ni); - m = NULL; - ctx = NULL; -descend_into_child_node: - /* - * Convert vcn to index into the index allocation attribute in units - * of PAGE_SIZE and map the page cache page, reading it from - * disk if necessary. - */ - page = ntfs_map_page(ia_mapping, vcn << - dir_ni->itype.index.vcn_size_bits >> PAGE_SHIFT); - if (IS_ERR(page)) { - ntfs_error(sb, "Failed to map directory index page, error %ld.", - -PTR_ERR(page)); - err = PTR_ERR(page); - goto err_out; - } - lock_page(page); - kaddr = (u8*)page_address(page); -fast_descend_into_child_node: - /* Get to the index allocation block. */ - ia = (INDEX_ALLOCATION*)(kaddr + ((vcn << - dir_ni->itype.index.vcn_size_bits) & ~PAGE_MASK)); - /* Bounds checks. */ - if ((u8*)ia < kaddr || (u8*)ia > kaddr + PAGE_SIZE) { - ntfs_error(sb, "Out of bounds check failed. Corrupt directory " - "inode 0x%lx or driver bug.", dir_ni->mft_no); - goto unm_err_out; - } - /* Catch multi sector transfer fixup errors. */ - if (unlikely(!ntfs_is_indx_record(ia->magic))) { - ntfs_error(sb, "Directory index record with vcn 0x%llx is " - "corrupt. Corrupt inode 0x%lx. Run chkdsk.", - (unsigned long long)vcn, dir_ni->mft_no); - goto unm_err_out; - } - if (sle64_to_cpu(ia->index_block_vcn) != vcn) { - ntfs_error(sb, "Actual VCN (0x%llx) of index buffer is " - "different from expected VCN (0x%llx). " - "Directory inode 0x%lx is corrupt or driver " - "bug.", (unsigned long long) - sle64_to_cpu(ia->index_block_vcn), - (unsigned long long)vcn, dir_ni->mft_no); - goto unm_err_out; - } - if (le32_to_cpu(ia->index.allocated_size) + 0x18 != - dir_ni->itype.index.block_size) { - ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode " - "0x%lx has a size (%u) differing from the " - "directory specified size (%u). Directory " - "inode is corrupt or driver bug.", - (unsigned long long)vcn, dir_ni->mft_no, - le32_to_cpu(ia->index.allocated_size) + 0x18, - dir_ni->itype.index.block_size); - goto unm_err_out; - } - index_end = (u8*)ia + dir_ni->itype.index.block_size; - if (index_end > kaddr + PAGE_SIZE) { - ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode " - "0x%lx crosses page boundary. Impossible! " - "Cannot access! This is probably a bug in the " - "driver.", (unsigned long long)vcn, - dir_ni->mft_no); - goto unm_err_out; - } - index_end = (u8*)&ia->index + le32_to_cpu(ia->index.index_length); - if (index_end > (u8*)ia + dir_ni->itype.index.block_size) { - ntfs_error(sb, "Size of index buffer (VCN 0x%llx) of directory " - "inode 0x%lx exceeds maximum size.", - (unsigned long long)vcn, dir_ni->mft_no); - goto unm_err_out; - } - /* The first index entry. */ - ie = (INDEX_ENTRY*)((u8*)&ia->index + - le32_to_cpu(ia->index.entries_offset)); - /* - * Iterate similar to above big loop but applied to index buffer, thus - * loop until we exceed valid memory (corruption case) or until we - * reach the last entry. - */ - for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) { - /* Bounds check. */ - if ((u8*)ie < (u8*)ia || (u8*)ie + - sizeof(INDEX_ENTRY_HEADER) > index_end || - (u8*)ie + le16_to_cpu(ie->key_length) > - index_end) { - ntfs_error(sb, "Index entry out of bounds in " - "directory inode 0x%lx.", - dir_ni->mft_no); - goto unm_err_out; - } - /* - * The last entry cannot contain a name. It can however contain - * a pointer to a child node in the B+tree so we just break out. - */ - if (ie->flags & INDEX_ENTRY_END) - break; - /* - * If the current entry has a name type of POSIX, the name is - * case sensitive and not otherwise. This has the effect of us - * not being able to access any POSIX file names which collate - * after the non-POSIX one when they only differ in case, but - * anyone doing screwy stuff like that deserves to burn in - * hell... Doing that kind of stuff on NT4 actually causes - * corruption on the partition even when using SP6a and Linux - * is not involved at all. - */ - ic = ie->key.file_name.file_name_type ? IGNORE_CASE : - CASE_SENSITIVE; - /* - * If the names match perfectly, we are done and return the - * mft reference of the inode (i.e. the inode number together - * with the sequence number for consistency checking. We - * convert it to cpu format before returning. - */ - if (ntfs_are_names_equal(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, ic, - vol->upcase, vol->upcase_len)) { -found_it2: - mref = le64_to_cpu(ie->data.dir.indexed_file); - unlock_page(page); - ntfs_unmap_page(page); - return mref; - } - /* - * Not a perfect match, need to do full blown collation so we - * know which way in the B+tree we have to go. - */ - rc = ntfs_collate_names(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, 1, - IGNORE_CASE, vol->upcase, vol->upcase_len); - /* - * If uname collates before the name of the current entry, there - * is definitely no such name in this index but we might need to - * descend into the B+tree so we just break out of the loop. - */ - if (rc == -1) - break; - /* The names are not equal, continue the search. */ - if (rc) - continue; - /* - * Names match with case insensitive comparison, now try the - * case sensitive comparison, which is required for proper - * collation. - */ - rc = ntfs_collate_names(uname, uname_len, - (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, 1, - CASE_SENSITIVE, vol->upcase, vol->upcase_len); - if (rc == -1) - break; - if (rc) - continue; - /* - * Perfect match, this will never happen as the - * ntfs_are_names_equal() call will have gotten a match but we - * still treat it correctly. - */ - goto found_it2; - } - /* - * We have finished with this index buffer without success. Check for - * the presence of a child node. - */ - if (ie->flags & INDEX_ENTRY_NODE) { - if ((ia->index.flags & NODE_MASK) == LEAF_NODE) { - ntfs_error(sb, "Index entry with child node found in " - "a leaf node in directory inode 0x%lx.", - dir_ni->mft_no); - goto unm_err_out; - } - /* Child node present, descend into it. */ - old_vcn = vcn; - vcn = sle64_to_cpup((u8*)ie + le16_to_cpu(ie->length) - 8); - if (vcn >= 0) { - /* If vcn is in the same page cache page as old_vcn we - * recycle the mapped page. */ - if (old_vcn << vol->cluster_size_bits >> - PAGE_SHIFT == vcn << - vol->cluster_size_bits >> - PAGE_SHIFT) - goto fast_descend_into_child_node; - unlock_page(page); - ntfs_unmap_page(page); - goto descend_into_child_node; - } - ntfs_error(sb, "Negative child node vcn in directory inode " - "0x%lx.", dir_ni->mft_no); - goto unm_err_out; - } - /* No child node, return -ENOENT. */ - ntfs_debug("Entry not found."); - err = -ENOENT; -unm_err_out: - unlock_page(page); - ntfs_unmap_page(page); -err_out: - if (!err) - err = -EIO; - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(dir_ni); - return ERR_MREF(err); -dir_err_out: - ntfs_error(sb, "Corrupt directory. Aborting lookup."); - goto err_out; -} - -#endif - -/** - * ntfs_filldir - ntfs specific filldir method - * @vol: current ntfs volume - * @ndir: ntfs inode of current directory - * @ia_page: page in which the index allocation buffer @ie is in resides - * @ie: current index entry - * @name: buffer to use for the converted name - * @actor: what to feed the entries to - * - * Convert the Unicode @name to the loaded NLS and pass it to the @filldir - * callback. - * - * If @ia_page is not NULL it is the locked page containing the index - * allocation block containing the index entry @ie. - * - * Note, we drop (and then reacquire) the page lock on @ia_page across the - * @filldir() call otherwise we would deadlock with NFSd when it calls ->lookup - * since ntfs_lookup() will lock the same page. As an optimization, we do not - * retake the lock if we are returning a non-zero value as ntfs_readdir() - * would need to drop the lock immediately anyway. - */ -static inline int ntfs_filldir(ntfs_volume *vol, - ntfs_inode *ndir, struct page *ia_page, INDEX_ENTRY *ie, - u8 *name, struct dir_context *actor) -{ - unsigned long mref; - int name_len; - unsigned dt_type; - FILE_NAME_TYPE_FLAGS name_type; - - name_type = ie->key.file_name.file_name_type; - if (name_type == FILE_NAME_DOS) { - ntfs_debug("Skipping DOS name space entry."); - return 0; - } - if (MREF_LE(ie->data.dir.indexed_file) == FILE_root) { - ntfs_debug("Skipping root directory self reference entry."); - return 0; - } - if (MREF_LE(ie->data.dir.indexed_file) < FILE_first_user && - !NVolShowSystemFiles(vol)) { - ntfs_debug("Skipping system file."); - return 0; - } - name_len = ntfs_ucstonls(vol, (ntfschar*)&ie->key.file_name.file_name, - ie->key.file_name.file_name_length, &name, - NTFS_MAX_NAME_LEN * NLS_MAX_CHARSET_SIZE + 1); - if (name_len <= 0) { - ntfs_warning(vol->sb, "Skipping unrepresentable inode 0x%llx.", - (long long)MREF_LE(ie->data.dir.indexed_file)); - return 0; - } - if (ie->key.file_name.file_attributes & - FILE_ATTR_DUP_FILE_NAME_INDEX_PRESENT) - dt_type = DT_DIR; - else - dt_type = DT_REG; - mref = MREF_LE(ie->data.dir.indexed_file); - /* - * Drop the page lock otherwise we deadlock with NFS when it calls - * ->lookup since ntfs_lookup() will lock the same page. - */ - if (ia_page) - unlock_page(ia_page); - ntfs_debug("Calling filldir for %s with len %i, fpos 0x%llx, inode " - "0x%lx, DT_%s.", name, name_len, actor->pos, mref, - dt_type == DT_DIR ? "DIR" : "REG"); - if (!dir_emit(actor, name, name_len, mref, dt_type)) - return 1; - /* Relock the page but not if we are aborting ->readdir. */ - if (ia_page) - lock_page(ia_page); - return 0; -} - -/* - * We use the same basic approach as the old NTFS driver, i.e. we parse the - * index root entries and then the index allocation entries that are marked - * as in use in the index bitmap. - * - * While this will return the names in random order this doesn't matter for - * ->readdir but OTOH results in a faster ->readdir. - * - * VFS calls ->readdir without BKL but with i_mutex held. This protects the VFS - * parts (e.g. ->f_pos and ->i_size, and it also protects against directory - * modifications). - * - * Locking: - Caller must hold i_mutex on the directory. - * - Each page cache page in the index allocation mapping must be - * locked whilst being accessed otherwise we may find a corrupt - * page due to it being under ->writepage at the moment which - * applies the mst protection fixups before writing out and then - * removes them again after the write is complete after which it - * unlocks the page. - */ -static int ntfs_readdir(struct file *file, struct dir_context *actor) -{ - s64 ia_pos, ia_start, prev_ia_pos, bmp_pos; - loff_t i_size; - struct inode *bmp_vi, *vdir = file_inode(file); - struct super_block *sb = vdir->i_sb; - ntfs_inode *ndir = NTFS_I(vdir); - ntfs_volume *vol = NTFS_SB(sb); - MFT_RECORD *m; - INDEX_ROOT *ir = NULL; - INDEX_ENTRY *ie; - INDEX_ALLOCATION *ia; - u8 *name = NULL; - int rc, err, ir_pos, cur_bmp_pos; - struct address_space *ia_mapping, *bmp_mapping; - struct page *bmp_page = NULL, *ia_page = NULL; - u8 *kaddr, *bmp, *index_end; - ntfs_attr_search_ctx *ctx; - - ntfs_debug("Entering for inode 0x%lx, fpos 0x%llx.", - vdir->i_ino, actor->pos); - rc = err = 0; - /* Are we at end of dir yet? */ - i_size = i_size_read(vdir); - if (actor->pos >= i_size + vol->mft_record_size) - return 0; - /* Emulate . and .. for all directories. */ - if (!dir_emit_dots(file, actor)) - return 0; - m = NULL; - ctx = NULL; - /* - * Allocate a buffer to store the current name being processed - * converted to format determined by current NLS. - */ - name = kmalloc(NTFS_MAX_NAME_LEN * NLS_MAX_CHARSET_SIZE + 1, GFP_NOFS); - if (unlikely(!name)) { - err = -ENOMEM; - goto err_out; - } - /* Are we jumping straight into the index allocation attribute? */ - if (actor->pos >= vol->mft_record_size) - goto skip_index_root; - /* Get hold of the mft record for the directory. */ - m = map_mft_record(ndir); - if (IS_ERR(m)) { - err = PTR_ERR(m); - m = NULL; - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(ndir, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - /* Get the offset into the index root attribute. */ - ir_pos = (s64)actor->pos; - /* Find the index root attribute in the mft record. */ - err = ntfs_attr_lookup(AT_INDEX_ROOT, I30, 4, CASE_SENSITIVE, 0, NULL, - 0, ctx); - if (unlikely(err)) { - ntfs_error(sb, "Index root attribute missing in directory " - "inode 0x%lx.", vdir->i_ino); - goto err_out; - } - /* - * Copy the index root attribute value to a buffer so that we can put - * the search context and unmap the mft record before calling the - * filldir() callback. We need to do this because of NFSd which calls - * ->lookup() from its filldir callback() and this causes NTFS to - * deadlock as ntfs_lookup() maps the mft record of the directory and - * we have got it mapped here already. The only solution is for us to - * unmap the mft record here so that a call to ntfs_lookup() is able to - * map the mft record without deadlocking. - */ - rc = le32_to_cpu(ctx->attr->data.resident.value_length); - ir = kmalloc(rc, GFP_NOFS); - if (unlikely(!ir)) { - err = -ENOMEM; - goto err_out; - } - /* Copy the index root value (it has been verified in read_inode). */ - memcpy(ir, (u8*)ctx->attr + - le16_to_cpu(ctx->attr->data.resident.value_offset), rc); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(ndir); - ctx = NULL; - m = NULL; - index_end = (u8*)&ir->index + le32_to_cpu(ir->index.index_length); - /* The first index entry. */ - ie = (INDEX_ENTRY*)((u8*)&ir->index + - le32_to_cpu(ir->index.entries_offset)); - /* - * Loop until we exceed valid memory (corruption case) or until we - * reach the last entry or until filldir tells us it has had enough - * or signals an error (both covered by the rc test). - */ - for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) { - ntfs_debug("In index root, offset 0x%zx.", (u8*)ie - (u8*)ir); - /* Bounds checks. */ - if (unlikely((u8*)ie < (u8*)ir || (u8*)ie + - sizeof(INDEX_ENTRY_HEADER) > index_end || - (u8*)ie + le16_to_cpu(ie->key_length) > - index_end)) - goto err_out; - /* The last entry cannot contain a name. */ - if (ie->flags & INDEX_ENTRY_END) - break; - /* Skip index root entry if continuing previous readdir. */ - if (ir_pos > (u8*)ie - (u8*)ir) - continue; - /* Advance the position even if going to skip the entry. */ - actor->pos = (u8*)ie - (u8*)ir; - /* Submit the name to the filldir callback. */ - rc = ntfs_filldir(vol, ndir, NULL, ie, name, actor); - if (rc) { - kfree(ir); - goto abort; - } - } - /* We are done with the index root and can free the buffer. */ - kfree(ir); - ir = NULL; - /* If there is no index allocation attribute we are finished. */ - if (!NInoIndexAllocPresent(ndir)) - goto EOD; - /* Advance fpos to the beginning of the index allocation. */ - actor->pos = vol->mft_record_size; -skip_index_root: - kaddr = NULL; - prev_ia_pos = -1LL; - /* Get the offset into the index allocation attribute. */ - ia_pos = (s64)actor->pos - vol->mft_record_size; - ia_mapping = vdir->i_mapping; - ntfs_debug("Inode 0x%lx, getting index bitmap.", vdir->i_ino); - bmp_vi = ntfs_attr_iget(vdir, AT_BITMAP, I30, 4); - if (IS_ERR(bmp_vi)) { - ntfs_error(sb, "Failed to get bitmap attribute."); - err = PTR_ERR(bmp_vi); - goto err_out; - } - bmp_mapping = bmp_vi->i_mapping; - /* Get the starting bitmap bit position and sanity check it. */ - bmp_pos = ia_pos >> ndir->itype.index.block_size_bits; - if (unlikely(bmp_pos >> 3 >= i_size_read(bmp_vi))) { - ntfs_error(sb, "Current index allocation position exceeds " - "index bitmap size."); - goto iput_err_out; - } - /* Get the starting bit position in the current bitmap page. */ - cur_bmp_pos = bmp_pos & ((PAGE_SIZE * 8) - 1); - bmp_pos &= ~(u64)((PAGE_SIZE * 8) - 1); -get_next_bmp_page: - ntfs_debug("Reading bitmap with page index 0x%llx, bit ofs 0x%llx", - (unsigned long long)bmp_pos >> (3 + PAGE_SHIFT), - (unsigned long long)bmp_pos & - (unsigned long long)((PAGE_SIZE * 8) - 1)); - bmp_page = ntfs_map_page(bmp_mapping, - bmp_pos >> (3 + PAGE_SHIFT)); - if (IS_ERR(bmp_page)) { - ntfs_error(sb, "Reading index bitmap failed."); - err = PTR_ERR(bmp_page); - bmp_page = NULL; - goto iput_err_out; - } - bmp = (u8*)page_address(bmp_page); - /* Find next index block in use. */ - while (!(bmp[cur_bmp_pos >> 3] & (1 << (cur_bmp_pos & 7)))) { -find_next_index_buffer: - cur_bmp_pos++; - /* - * If we have reached the end of the bitmap page, get the next - * page, and put away the old one. - */ - if (unlikely((cur_bmp_pos >> 3) >= PAGE_SIZE)) { - ntfs_unmap_page(bmp_page); - bmp_pos += PAGE_SIZE * 8; - cur_bmp_pos = 0; - goto get_next_bmp_page; - } - /* If we have reached the end of the bitmap, we are done. */ - if (unlikely(((bmp_pos + cur_bmp_pos) >> 3) >= i_size)) - goto unm_EOD; - ia_pos = (bmp_pos + cur_bmp_pos) << - ndir->itype.index.block_size_bits; - } - ntfs_debug("Handling index buffer 0x%llx.", - (unsigned long long)bmp_pos + cur_bmp_pos); - /* If the current index buffer is in the same page we reuse the page. */ - if ((prev_ia_pos & (s64)PAGE_MASK) != - (ia_pos & (s64)PAGE_MASK)) { - prev_ia_pos = ia_pos; - if (likely(ia_page != NULL)) { - unlock_page(ia_page); - ntfs_unmap_page(ia_page); - } - /* - * Map the page cache page containing the current ia_pos, - * reading it from disk if necessary. - */ - ia_page = ntfs_map_page(ia_mapping, ia_pos >> PAGE_SHIFT); - if (IS_ERR(ia_page)) { - ntfs_error(sb, "Reading index allocation data failed."); - err = PTR_ERR(ia_page); - ia_page = NULL; - goto err_out; - } - lock_page(ia_page); - kaddr = (u8*)page_address(ia_page); - } - /* Get the current index buffer. */ - ia = (INDEX_ALLOCATION*)(kaddr + (ia_pos & ~PAGE_MASK & - ~(s64)(ndir->itype.index.block_size - 1))); - /* Bounds checks. */ - if (unlikely((u8*)ia < kaddr || (u8*)ia > kaddr + PAGE_SIZE)) { - ntfs_error(sb, "Out of bounds check failed. Corrupt directory " - "inode 0x%lx or driver bug.", vdir->i_ino); - goto err_out; - } - /* Catch multi sector transfer fixup errors. */ - if (unlikely(!ntfs_is_indx_record(ia->magic))) { - ntfs_error(sb, "Directory index record with vcn 0x%llx is " - "corrupt. Corrupt inode 0x%lx. Run chkdsk.", - (unsigned long long)ia_pos >> - ndir->itype.index.vcn_size_bits, vdir->i_ino); - goto err_out; - } - if (unlikely(sle64_to_cpu(ia->index_block_vcn) != (ia_pos & - ~(s64)(ndir->itype.index.block_size - 1)) >> - ndir->itype.index.vcn_size_bits)) { - ntfs_error(sb, "Actual VCN (0x%llx) of index buffer is " - "different from expected VCN (0x%llx). " - "Directory inode 0x%lx is corrupt or driver " - "bug. ", (unsigned long long) - sle64_to_cpu(ia->index_block_vcn), - (unsigned long long)ia_pos >> - ndir->itype.index.vcn_size_bits, vdir->i_ino); - goto err_out; - } - if (unlikely(le32_to_cpu(ia->index.allocated_size) + 0x18 != - ndir->itype.index.block_size)) { - ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode " - "0x%lx has a size (%u) differing from the " - "directory specified size (%u). Directory " - "inode is corrupt or driver bug.", - (unsigned long long)ia_pos >> - ndir->itype.index.vcn_size_bits, vdir->i_ino, - le32_to_cpu(ia->index.allocated_size) + 0x18, - ndir->itype.index.block_size); - goto err_out; - } - index_end = (u8*)ia + ndir->itype.index.block_size; - if (unlikely(index_end > kaddr + PAGE_SIZE)) { - ntfs_error(sb, "Index buffer (VCN 0x%llx) of directory inode " - "0x%lx crosses page boundary. Impossible! " - "Cannot access! This is probably a bug in the " - "driver.", (unsigned long long)ia_pos >> - ndir->itype.index.vcn_size_bits, vdir->i_ino); - goto err_out; - } - ia_start = ia_pos & ~(s64)(ndir->itype.index.block_size - 1); - index_end = (u8*)&ia->index + le32_to_cpu(ia->index.index_length); - if (unlikely(index_end > (u8*)ia + ndir->itype.index.block_size)) { - ntfs_error(sb, "Size of index buffer (VCN 0x%llx) of directory " - "inode 0x%lx exceeds maximum size.", - (unsigned long long)ia_pos >> - ndir->itype.index.vcn_size_bits, vdir->i_ino); - goto err_out; - } - /* The first index entry in this index buffer. */ - ie = (INDEX_ENTRY*)((u8*)&ia->index + - le32_to_cpu(ia->index.entries_offset)); - /* - * Loop until we exceed valid memory (corruption case) or until we - * reach the last entry or until filldir tells us it has had enough - * or signals an error (both covered by the rc test). - */ - for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) { - ntfs_debug("In index allocation, offset 0x%llx.", - (unsigned long long)ia_start + - (unsigned long long)((u8*)ie - (u8*)ia)); - /* Bounds checks. */ - if (unlikely((u8*)ie < (u8*)ia || (u8*)ie + - sizeof(INDEX_ENTRY_HEADER) > index_end || - (u8*)ie + le16_to_cpu(ie->key_length) > - index_end)) - goto err_out; - /* The last entry cannot contain a name. */ - if (ie->flags & INDEX_ENTRY_END) - break; - /* Skip index block entry if continuing previous readdir. */ - if (ia_pos - ia_start > (u8*)ie - (u8*)ia) - continue; - /* Advance the position even if going to skip the entry. */ - actor->pos = (u8*)ie - (u8*)ia + - (sle64_to_cpu(ia->index_block_vcn) << - ndir->itype.index.vcn_size_bits) + - vol->mft_record_size; - /* - * Submit the name to the @filldir callback. Note, - * ntfs_filldir() drops the lock on @ia_page but it retakes it - * before returning, unless a non-zero value is returned in - * which case the page is left unlocked. - */ - rc = ntfs_filldir(vol, ndir, ia_page, ie, name, actor); - if (rc) { - /* @ia_page is already unlocked in this case. */ - ntfs_unmap_page(ia_page); - ntfs_unmap_page(bmp_page); - iput(bmp_vi); - goto abort; - } - } - goto find_next_index_buffer; -unm_EOD: - if (ia_page) { - unlock_page(ia_page); - ntfs_unmap_page(ia_page); - } - ntfs_unmap_page(bmp_page); - iput(bmp_vi); -EOD: - /* We are finished, set fpos to EOD. */ - actor->pos = i_size + vol->mft_record_size; -abort: - kfree(name); - return 0; -err_out: - if (bmp_page) { - ntfs_unmap_page(bmp_page); -iput_err_out: - iput(bmp_vi); - } - if (ia_page) { - unlock_page(ia_page); - ntfs_unmap_page(ia_page); - } - kfree(ir); - kfree(name); - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(ndir); - if (!err) - err = -EIO; - ntfs_debug("Failed. Returning error code %i.", -err); - return err; -} - -/** - * ntfs_dir_open - called when an inode is about to be opened - * @vi: inode to be opened - * @filp: file structure describing the inode - * - * Limit directory size to the page cache limit on architectures where unsigned - * long is 32-bits. This is the most we can do for now without overflowing the - * page cache page index. Doing it this way means we don't run into problems - * because of existing too large directories. It would be better to allow the - * user to read the accessible part of the directory but I doubt very much - * anyone is going to hit this check on a 32-bit architecture, so there is no - * point in adding the extra complexity required to support this. - * - * On 64-bit architectures, the check is hopefully optimized away by the - * compiler. - */ -static int ntfs_dir_open(struct inode *vi, struct file *filp) -{ - if (sizeof(unsigned long) < 8) { - if (i_size_read(vi) > MAX_LFS_FILESIZE) - return -EFBIG; - } - return 0; -} - -#ifdef NTFS_RW - -/** - * ntfs_dir_fsync - sync a directory to disk - * @filp: directory to be synced - * @dentry: dentry describing the directory to sync - * @datasync: if non-zero only flush user data and not metadata - * - * Data integrity sync of a directory to disk. Used for fsync, fdatasync, and - * msync system calls. This function is based on file.c::ntfs_file_fsync(). - * - * Write the mft record and all associated extent mft records as well as the - * $INDEX_ALLOCATION and $BITMAP attributes and then sync the block device. - * - * If @datasync is true, we do not wait on the inode(s) to be written out - * but we always wait on the page cache pages to be written out. - * - * Note: In the past @filp could be NULL so we ignore it as we don't need it - * anyway. - * - * Locking: Caller must hold i_mutex on the inode. - * - * TODO: We should probably also write all attribute/index inodes associated - * with this inode but since we have no simple way of getting to them we ignore - * this problem for now. We do write the $BITMAP attribute if it is present - * which is the important one for a directory so things are not too bad. - */ -static int ntfs_dir_fsync(struct file *filp, loff_t start, loff_t end, - int datasync) -{ - struct inode *bmp_vi, *vi = filp->f_mapping->host; - int err, ret; - ntfs_attr na; - - ntfs_debug("Entering for inode 0x%lx.", vi->i_ino); - - err = file_write_and_wait_range(filp, start, end); - if (err) - return err; - inode_lock(vi); - - BUG_ON(!S_ISDIR(vi->i_mode)); - /* If the bitmap attribute inode is in memory sync it, too. */ - na.mft_no = vi->i_ino; - na.type = AT_BITMAP; - na.name = I30; - na.name_len = 4; - bmp_vi = ilookup5(vi->i_sb, vi->i_ino, ntfs_test_inode, &na); - if (bmp_vi) { - write_inode_now(bmp_vi, !datasync); - iput(bmp_vi); - } - ret = __ntfs_write_inode(vi, 1); - write_inode_now(vi, !datasync); - err = sync_blockdev(vi->i_sb->s_bdev); - if (unlikely(err && !ret)) - ret = err; - if (likely(!ret)) - ntfs_debug("Done."); - else - ntfs_warning(vi->i_sb, "Failed to f%ssync inode 0x%lx. Error " - "%u.", datasync ? "data" : "", vi->i_ino, -ret); - inode_unlock(vi); - return ret; -} - -#endif /* NTFS_RW */ - -WRAP_DIR_ITER(ntfs_readdir) // FIXME! -const struct file_operations ntfs_dir_ops = { - .llseek = generic_file_llseek, /* Seek inside directory. */ - .read = generic_read_dir, /* Return -EISDIR. */ - .iterate_shared = shared_ntfs_readdir, /* Read directory contents. */ -#ifdef NTFS_RW - .fsync = ntfs_dir_fsync, /* Sync a directory to disk. */ -#endif /* NTFS_RW */ - /*.ioctl = ,*/ /* Perform function on the - mounted filesystem. */ - .open = ntfs_dir_open, /* Open directory. */ -}; diff --git a/fs/ntfs/dir.h b/fs/ntfs/dir.h deleted file mode 100644 index 0e326753df40..000000000000 --- a/fs/ntfs/dir.h +++ /dev/null @@ -1,34 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * dir.h - Defines for directory handling in NTFS Linux kernel driver. Part of - * the Linux-NTFS project. - * - * Copyright (c) 2002-2004 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_DIR_H -#define _LINUX_NTFS_DIR_H - -#include "layout.h" -#include "inode.h" -#include "types.h" - -/* - * ntfs_name is used to return the file name to the caller of - * ntfs_lookup_inode_by_name() in order for the caller (namei.c::ntfs_lookup()) - * to be able to deal with dcache aliasing issues. - */ -typedef struct { - MFT_REF mref; - FILE_NAME_TYPE_FLAGS type; - u8 len; - ntfschar name[0]; -} __attribute__ ((__packed__)) ntfs_name; - -/* The little endian Unicode string $I30 as a global constant. */ -extern ntfschar I30[5]; - -extern MFT_REF ntfs_lookup_inode_by_name(ntfs_inode *dir_ni, - const ntfschar *uname, const int uname_len, ntfs_name **res); - -#endif /* _LINUX_NTFS_FS_DIR_H */ diff --git a/fs/ntfs/endian.h b/fs/ntfs/endian.h deleted file mode 100644 index f30c139bf9ae..000000000000 --- a/fs/ntfs/endian.h +++ /dev/null @@ -1,79 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * endian.h - Defines for endianness handling in NTFS Linux kernel driver. - * Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2004 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_ENDIAN_H -#define _LINUX_NTFS_ENDIAN_H - -#include -#include "types.h" - -/* - * Signed endianness conversion functions. - */ - -static inline s16 sle16_to_cpu(sle16 x) -{ - return le16_to_cpu((__force le16)x); -} - -static inline s32 sle32_to_cpu(sle32 x) -{ - return le32_to_cpu((__force le32)x); -} - -static inline s64 sle64_to_cpu(sle64 x) -{ - return le64_to_cpu((__force le64)x); -} - -static inline s16 sle16_to_cpup(sle16 *x) -{ - return le16_to_cpu(*(__force le16*)x); -} - -static inline s32 sle32_to_cpup(sle32 *x) -{ - return le32_to_cpu(*(__force le32*)x); -} - -static inline s64 sle64_to_cpup(sle64 *x) -{ - return le64_to_cpu(*(__force le64*)x); -} - -static inline sle16 cpu_to_sle16(s16 x) -{ - return (__force sle16)cpu_to_le16(x); -} - -static inline sle32 cpu_to_sle32(s32 x) -{ - return (__force sle32)cpu_to_le32(x); -} - -static inline sle64 cpu_to_sle64(s64 x) -{ - return (__force sle64)cpu_to_le64(x); -} - -static inline sle16 cpu_to_sle16p(s16 *x) -{ - return (__force sle16)cpu_to_le16(*x); -} - -static inline sle32 cpu_to_sle32p(s32 *x) -{ - return (__force sle32)cpu_to_le32(*x); -} - -static inline sle64 cpu_to_sle64p(s64 *x) -{ - return (__force sle64)cpu_to_le64(*x); -} - -#endif /* _LINUX_NTFS_ENDIAN_H */ diff --git a/fs/ntfs/file.c b/fs/ntfs/file.c deleted file mode 100644 index cbc545999cfe..000000000000 --- a/fs/ntfs/file.c +++ /dev/null @@ -1,2004 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * file.c - NTFS kernel file operations. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2015 Anton Altaparmakov and Tuxera Inc. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include -#include - -#include "attrib.h" -#include "bitmap.h" -#include "inode.h" -#include "debug.h" -#include "lcnalloc.h" -#include "malloc.h" -#include "mft.h" -#include "ntfs.h" - -/** - * ntfs_file_open - called when an inode is about to be opened - * @vi: inode to be opened - * @filp: file structure describing the inode - * - * Limit file size to the page cache limit on architectures where unsigned long - * is 32-bits. This is the most we can do for now without overflowing the page - * cache page index. Doing it this way means we don't run into problems because - * of existing too large files. It would be better to allow the user to read - * the beginning of the file but I doubt very much anyone is going to hit this - * check on a 32-bit architecture, so there is no point in adding the extra - * complexity required to support this. - * - * On 64-bit architectures, the check is hopefully optimized away by the - * compiler. - * - * After the check passes, just call generic_file_open() to do its work. - */ -static int ntfs_file_open(struct inode *vi, struct file *filp) -{ - if (sizeof(unsigned long) < 8) { - if (i_size_read(vi) > MAX_LFS_FILESIZE) - return -EOVERFLOW; - } - return generic_file_open(vi, filp); -} - -#ifdef NTFS_RW - -/** - * ntfs_attr_extend_initialized - extend the initialized size of an attribute - * @ni: ntfs inode of the attribute to extend - * @new_init_size: requested new initialized size in bytes - * - * Extend the initialized size of an attribute described by the ntfs inode @ni - * to @new_init_size bytes. This involves zeroing any non-sparse space between - * the old initialized size and @new_init_size both in the page cache and on - * disk (if relevant complete pages are already uptodate in the page cache then - * these are simply marked dirty). - * - * As a side-effect, the file size (vfs inode->i_size) may be incremented as, - * in the resident attribute case, it is tied to the initialized size and, in - * the non-resident attribute case, it may not fall below the initialized size. - * - * Note that if the attribute is resident, we do not need to touch the page - * cache at all. This is because if the page cache page is not uptodate we - * bring it uptodate later, when doing the write to the mft record since we - * then already have the page mapped. And if the page is uptodate, the - * non-initialized region will already have been zeroed when the page was - * brought uptodate and the region may in fact already have been overwritten - * with new data via mmap() based writes, so we cannot just zero it. And since - * POSIX specifies that the behaviour of resizing a file whilst it is mmap()ped - * is unspecified, we choose not to do zeroing and thus we do not need to touch - * the page at all. For a more detailed explanation see ntfs_truncate() in - * fs/ntfs/inode.c. - * - * Return 0 on success and -errno on error. In the case that an error is - * encountered it is possible that the initialized size will already have been - * incremented some way towards @new_init_size but it is guaranteed that if - * this is the case, the necessary zeroing will also have happened and that all - * metadata is self-consistent. - * - * Locking: i_mutex on the vfs inode corrseponsind to the ntfs inode @ni must be - * held by the caller. - */ -static int ntfs_attr_extend_initialized(ntfs_inode *ni, const s64 new_init_size) -{ - s64 old_init_size; - loff_t old_i_size; - pgoff_t index, end_index; - unsigned long flags; - struct inode *vi = VFS_I(ni); - ntfs_inode *base_ni; - MFT_RECORD *m = NULL; - ATTR_RECORD *a; - ntfs_attr_search_ctx *ctx = NULL; - struct address_space *mapping; - struct page *page = NULL; - u8 *kattr; - int err; - u32 attr_len; - - read_lock_irqsave(&ni->size_lock, flags); - old_init_size = ni->initialized_size; - old_i_size = i_size_read(vi); - BUG_ON(new_init_size > ni->allocated_size); - read_unlock_irqrestore(&ni->size_lock, flags); - ntfs_debug("Entering for i_ino 0x%lx, attribute type 0x%x, " - "old_initialized_size 0x%llx, " - "new_initialized_size 0x%llx, i_size 0x%llx.", - vi->i_ino, (unsigned)le32_to_cpu(ni->type), - (unsigned long long)old_init_size, - (unsigned long long)new_init_size, old_i_size); - if (!NInoAttr(ni)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - /* Use goto to reduce indentation and we need the label below anyway. */ - if (NInoNonResident(ni)) - goto do_non_resident_extend; - BUG_ON(old_init_size != old_i_size); - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - m = NULL; - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - err = -EIO; - goto err_out; - } - m = ctx->mrec; - a = ctx->attr; - BUG_ON(a->non_resident); - /* The total length of the attribute value. */ - attr_len = le32_to_cpu(a->data.resident.value_length); - BUG_ON(old_i_size != (loff_t)attr_len); - /* - * Do the zeroing in the mft record and update the attribute size in - * the mft record. - */ - kattr = (u8*)a + le16_to_cpu(a->data.resident.value_offset); - memset(kattr + attr_len, 0, new_init_size - attr_len); - a->data.resident.value_length = cpu_to_le32((u32)new_init_size); - /* Finally, update the sizes in the vfs and ntfs inodes. */ - write_lock_irqsave(&ni->size_lock, flags); - i_size_write(vi, new_init_size); - ni->initialized_size = new_init_size; - write_unlock_irqrestore(&ni->size_lock, flags); - goto done; -do_non_resident_extend: - /* - * If the new initialized size @new_init_size exceeds the current file - * size (vfs inode->i_size), we need to extend the file size to the - * new initialized size. - */ - if (new_init_size > old_i_size) { - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - m = NULL; - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - err = -EIO; - goto err_out; - } - m = ctx->mrec; - a = ctx->attr; - BUG_ON(!a->non_resident); - BUG_ON(old_i_size != (loff_t) - sle64_to_cpu(a->data.non_resident.data_size)); - a->data.non_resident.data_size = cpu_to_sle64(new_init_size); - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - /* Update the file size in the vfs inode. */ - i_size_write(vi, new_init_size); - ntfs_attr_put_search_ctx(ctx); - ctx = NULL; - unmap_mft_record(base_ni); - m = NULL; - } - mapping = vi->i_mapping; - index = old_init_size >> PAGE_SHIFT; - end_index = (new_init_size + PAGE_SIZE - 1) >> PAGE_SHIFT; - do { - /* - * Read the page. If the page is not present, this will zero - * the uninitialized regions for us. - */ - page = read_mapping_page(mapping, index, NULL); - if (IS_ERR(page)) { - err = PTR_ERR(page); - goto init_err_out; - } - /* - * Update the initialized size in the ntfs inode. This is - * enough to make ntfs_writepage() work. - */ - write_lock_irqsave(&ni->size_lock, flags); - ni->initialized_size = (s64)(index + 1) << PAGE_SHIFT; - if (ni->initialized_size > new_init_size) - ni->initialized_size = new_init_size; - write_unlock_irqrestore(&ni->size_lock, flags); - /* Set the page dirty so it gets written out. */ - set_page_dirty(page); - put_page(page); - /* - * Play nice with the vm and the rest of the system. This is - * very much needed as we can potentially be modifying the - * initialised size from a very small value to a really huge - * value, e.g. - * f = open(somefile, O_TRUNC); - * truncate(f, 10GiB); - * seek(f, 10GiB); - * write(f, 1); - * And this would mean we would be marking dirty hundreds of - * thousands of pages or as in the above example more than - * two and a half million pages! - * - * TODO: For sparse pages could optimize this workload by using - * the FsMisc / MiscFs page bit as a "PageIsSparse" bit. This - * would be set in read_folio for sparse pages and here we would - * not need to mark dirty any pages which have this bit set. - * The only caveat is that we have to clear the bit everywhere - * where we allocate any clusters that lie in the page or that - * contain the page. - * - * TODO: An even greater optimization would be for us to only - * call read_folio() on pages which are not in sparse regions as - * determined from the runlist. This would greatly reduce the - * number of pages we read and make dirty in the case of sparse - * files. - */ - balance_dirty_pages_ratelimited(mapping); - cond_resched(); - } while (++index < end_index); - read_lock_irqsave(&ni->size_lock, flags); - BUG_ON(ni->initialized_size != new_init_size); - read_unlock_irqrestore(&ni->size_lock, flags); - /* Now bring in sync the initialized_size in the mft record. */ - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - m = NULL; - goto init_err_out; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto init_err_out; - } - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - err = -EIO; - goto init_err_out; - } - m = ctx->mrec; - a = ctx->attr; - BUG_ON(!a->non_resident); - a->data.non_resident.initialized_size = cpu_to_sle64(new_init_size); -done: - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(base_ni); - ntfs_debug("Done, initialized_size 0x%llx, i_size 0x%llx.", - (unsigned long long)new_init_size, i_size_read(vi)); - return 0; -init_err_out: - write_lock_irqsave(&ni->size_lock, flags); - ni->initialized_size = old_init_size; - write_unlock_irqrestore(&ni->size_lock, flags); -err_out: - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(base_ni); - ntfs_debug("Failed. Returning error code %i.", err); - return err; -} - -static ssize_t ntfs_prepare_file_for_write(struct kiocb *iocb, - struct iov_iter *from) -{ - loff_t pos; - s64 end, ll; - ssize_t err; - unsigned long flags; - struct file *file = iocb->ki_filp; - struct inode *vi = file_inode(file); - ntfs_inode *ni = NTFS_I(vi); - ntfs_volume *vol = ni->vol; - - ntfs_debug("Entering for i_ino 0x%lx, attribute type 0x%x, pos " - "0x%llx, count 0x%zx.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), - (unsigned long long)iocb->ki_pos, - iov_iter_count(from)); - err = generic_write_checks(iocb, from); - if (unlikely(err <= 0)) - goto out; - /* - * All checks have passed. Before we start doing any writing we want - * to abort any totally illegal writes. - */ - BUG_ON(NInoMstProtected(ni)); - BUG_ON(ni->type != AT_DATA); - /* If file is encrypted, deny access, just like NT4. */ - if (NInoEncrypted(ni)) { - /* Only $DATA attributes can be encrypted. */ - /* - * Reminder for later: Encrypted files are _always_ - * non-resident so that the content can always be encrypted. - */ - ntfs_debug("Denying write access to encrypted file."); - err = -EACCES; - goto out; - } - if (NInoCompressed(ni)) { - /* Only unnamed $DATA attribute can be compressed. */ - BUG_ON(ni->name_len); - /* - * Reminder for later: If resident, the data is not actually - * compressed. Only on the switch to non-resident does - * compression kick in. This is in contrast to encrypted files - * (see above). - */ - ntfs_error(vi->i_sb, "Writing to compressed files is not " - "implemented yet. Sorry."); - err = -EOPNOTSUPP; - goto out; - } - err = file_remove_privs(file); - if (unlikely(err)) - goto out; - /* - * Our ->update_time method always succeeds thus file_update_time() - * cannot fail either so there is no need to check the return code. - */ - file_update_time(file); - pos = iocb->ki_pos; - /* The first byte after the last cluster being written to. */ - end = (pos + iov_iter_count(from) + vol->cluster_size_mask) & - ~(u64)vol->cluster_size_mask; - /* - * If the write goes beyond the allocated size, extend the allocation - * to cover the whole of the write, rounded up to the nearest cluster. - */ - read_lock_irqsave(&ni->size_lock, flags); - ll = ni->allocated_size; - read_unlock_irqrestore(&ni->size_lock, flags); - if (end > ll) { - /* - * Extend the allocation without changing the data size. - * - * Note we ensure the allocation is big enough to at least - * write some data but we do not require the allocation to be - * complete, i.e. it may be partial. - */ - ll = ntfs_attr_extend_allocation(ni, end, -1, pos); - if (likely(ll >= 0)) { - BUG_ON(pos >= ll); - /* If the extension was partial truncate the write. */ - if (end > ll) { - ntfs_debug("Truncating write to inode 0x%lx, " - "attribute type 0x%x, because " - "the allocation was only " - "partially extended.", - vi->i_ino, (unsigned) - le32_to_cpu(ni->type)); - iov_iter_truncate(from, ll - pos); - } - } else { - err = ll; - read_lock_irqsave(&ni->size_lock, flags); - ll = ni->allocated_size; - read_unlock_irqrestore(&ni->size_lock, flags); - /* Perform a partial write if possible or fail. */ - if (pos < ll) { - ntfs_debug("Truncating write to inode 0x%lx " - "attribute type 0x%x, because " - "extending the allocation " - "failed (error %d).", - vi->i_ino, (unsigned) - le32_to_cpu(ni->type), - (int)-err); - iov_iter_truncate(from, ll - pos); - } else { - if (err != -ENOSPC) - ntfs_error(vi->i_sb, "Cannot perform " - "write to inode " - "0x%lx, attribute " - "type 0x%x, because " - "extending the " - "allocation failed " - "(error %ld).", - vi->i_ino, (unsigned) - le32_to_cpu(ni->type), - (long)-err); - else - ntfs_debug("Cannot perform write to " - "inode 0x%lx, " - "attribute type 0x%x, " - "because there is not " - "space left.", - vi->i_ino, (unsigned) - le32_to_cpu(ni->type)); - goto out; - } - } - } - /* - * If the write starts beyond the initialized size, extend it up to the - * beginning of the write and initialize all non-sparse space between - * the old initialized size and the new one. This automatically also - * increments the vfs inode->i_size to keep it above or equal to the - * initialized_size. - */ - read_lock_irqsave(&ni->size_lock, flags); - ll = ni->initialized_size; - read_unlock_irqrestore(&ni->size_lock, flags); - if (pos > ll) { - /* - * Wait for ongoing direct i/o to complete before proceeding. - * New direct i/o cannot start as we hold i_mutex. - */ - inode_dio_wait(vi); - err = ntfs_attr_extend_initialized(ni, pos); - if (unlikely(err < 0)) - ntfs_error(vi->i_sb, "Cannot perform write to inode " - "0x%lx, attribute type 0x%x, because " - "extending the initialized size " - "failed (error %d).", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), - (int)-err); - } -out: - return err; -} - -/** - * __ntfs_grab_cache_pages - obtain a number of locked pages - * @mapping: address space mapping from which to obtain page cache pages - * @index: starting index in @mapping at which to begin obtaining pages - * @nr_pages: number of page cache pages to obtain - * @pages: array of pages in which to return the obtained page cache pages - * @cached_page: allocated but as yet unused page - * - * Obtain @nr_pages locked page cache pages from the mapping @mapping and - * starting at index @index. - * - * If a page is newly created, add it to lru list - * - * Note, the page locks are obtained in ascending page index order. - */ -static inline int __ntfs_grab_cache_pages(struct address_space *mapping, - pgoff_t index, const unsigned nr_pages, struct page **pages, - struct page **cached_page) -{ - int err, nr; - - BUG_ON(!nr_pages); - err = nr = 0; - do { - pages[nr] = find_get_page_flags(mapping, index, FGP_LOCK | - FGP_ACCESSED); - if (!pages[nr]) { - if (!*cached_page) { - *cached_page = page_cache_alloc(mapping); - if (unlikely(!*cached_page)) { - err = -ENOMEM; - goto err_out; - } - } - err = add_to_page_cache_lru(*cached_page, mapping, - index, - mapping_gfp_constraint(mapping, GFP_KERNEL)); - if (unlikely(err)) { - if (err == -EEXIST) - continue; - goto err_out; - } - pages[nr] = *cached_page; - *cached_page = NULL; - } - index++; - nr++; - } while (nr < nr_pages); -out: - return err; -err_out: - while (nr > 0) { - unlock_page(pages[--nr]); - put_page(pages[nr]); - } - goto out; -} - -static inline void ntfs_submit_bh_for_read(struct buffer_head *bh) -{ - lock_buffer(bh); - get_bh(bh); - bh->b_end_io = end_buffer_read_sync; - submit_bh(REQ_OP_READ, bh); -} - -/** - * ntfs_prepare_pages_for_non_resident_write - prepare pages for receiving data - * @pages: array of destination pages - * @nr_pages: number of pages in @pages - * @pos: byte position in file at which the write begins - * @bytes: number of bytes to be written - * - * This is called for non-resident attributes from ntfs_file_buffered_write() - * with i_mutex held on the inode (@pages[0]->mapping->host). There are - * @nr_pages pages in @pages which are locked but not kmap()ped. The source - * data has not yet been copied into the @pages. - * - * Need to fill any holes with actual clusters, allocate buffers if necessary, - * ensure all the buffers are mapped, and bring uptodate any buffers that are - * only partially being written to. - * - * If @nr_pages is greater than one, we are guaranteed that the cluster size is - * greater than PAGE_SIZE, that all pages in @pages are entirely inside - * the same cluster and that they are the entirety of that cluster, and that - * the cluster is sparse, i.e. we need to allocate a cluster to fill the hole. - * - * i_size is not to be modified yet. - * - * Return 0 on success or -errno on error. - */ -static int ntfs_prepare_pages_for_non_resident_write(struct page **pages, - unsigned nr_pages, s64 pos, size_t bytes) -{ - VCN vcn, highest_vcn = 0, cpos, cend, bh_cpos, bh_cend; - LCN lcn; - s64 bh_pos, vcn_len, end, initialized_size; - sector_t lcn_block; - struct page *page; - struct inode *vi; - ntfs_inode *ni, *base_ni = NULL; - ntfs_volume *vol; - runlist_element *rl, *rl2; - struct buffer_head *bh, *head, *wait[2], **wait_bh = wait; - ntfs_attr_search_ctx *ctx = NULL; - MFT_RECORD *m = NULL; - ATTR_RECORD *a = NULL; - unsigned long flags; - u32 attr_rec_len = 0; - unsigned blocksize, u; - int err, mp_size; - bool rl_write_locked, was_hole, is_retry; - unsigned char blocksize_bits; - struct { - u8 runlist_merged:1; - u8 mft_attr_mapped:1; - u8 mp_rebuilt:1; - u8 attr_switched:1; - } status = { 0, 0, 0, 0 }; - - BUG_ON(!nr_pages); - BUG_ON(!pages); - BUG_ON(!*pages); - vi = pages[0]->mapping->host; - ni = NTFS_I(vi); - vol = ni->vol; - ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, start page " - "index 0x%lx, nr_pages 0x%x, pos 0x%llx, bytes 0x%zx.", - vi->i_ino, ni->type, pages[0]->index, nr_pages, - (long long)pos, bytes); - blocksize = vol->sb->s_blocksize; - blocksize_bits = vol->sb->s_blocksize_bits; - u = 0; - do { - page = pages[u]; - BUG_ON(!page); - /* - * create_empty_buffers() will create uptodate/dirty buffers if - * the page is uptodate/dirty. - */ - if (!page_has_buffers(page)) { - create_empty_buffers(page, blocksize, 0); - if (unlikely(!page_has_buffers(page))) - return -ENOMEM; - } - } while (++u < nr_pages); - rl_write_locked = false; - rl = NULL; - err = 0; - vcn = lcn = -1; - vcn_len = 0; - lcn_block = -1; - was_hole = false; - cpos = pos >> vol->cluster_size_bits; - end = pos + bytes; - cend = (end + vol->cluster_size - 1) >> vol->cluster_size_bits; - /* - * Loop over each page and for each page over each buffer. Use goto to - * reduce indentation. - */ - u = 0; -do_next_page: - page = pages[u]; - bh_pos = (s64)page->index << PAGE_SHIFT; - bh = head = page_buffers(page); - do { - VCN cdelta; - s64 bh_end; - unsigned bh_cofs; - - /* Clear buffer_new on all buffers to reinitialise state. */ - if (buffer_new(bh)) - clear_buffer_new(bh); - bh_end = bh_pos + blocksize; - bh_cpos = bh_pos >> vol->cluster_size_bits; - bh_cofs = bh_pos & vol->cluster_size_mask; - if (buffer_mapped(bh)) { - /* - * The buffer is already mapped. If it is uptodate, - * ignore it. - */ - if (buffer_uptodate(bh)) - continue; - /* - * The buffer is not uptodate. If the page is uptodate - * set the buffer uptodate and otherwise ignore it. - */ - if (PageUptodate(page)) { - set_buffer_uptodate(bh); - continue; - } - /* - * Neither the page nor the buffer are uptodate. If - * the buffer is only partially being written to, we - * need to read it in before the write, i.e. now. - */ - if ((bh_pos < pos && bh_end > pos) || - (bh_pos < end && bh_end > end)) { - /* - * If the buffer is fully or partially within - * the initialized size, do an actual read. - * Otherwise, simply zero the buffer. - */ - read_lock_irqsave(&ni->size_lock, flags); - initialized_size = ni->initialized_size; - read_unlock_irqrestore(&ni->size_lock, flags); - if (bh_pos < initialized_size) { - ntfs_submit_bh_for_read(bh); - *wait_bh++ = bh; - } else { - zero_user(page, bh_offset(bh), - blocksize); - set_buffer_uptodate(bh); - } - } - continue; - } - /* Unmapped buffer. Need to map it. */ - bh->b_bdev = vol->sb->s_bdev; - /* - * If the current buffer is in the same clusters as the map - * cache, there is no need to check the runlist again. The - * map cache is made up of @vcn, which is the first cached file - * cluster, @vcn_len which is the number of cached file - * clusters, @lcn is the device cluster corresponding to @vcn, - * and @lcn_block is the block number corresponding to @lcn. - */ - cdelta = bh_cpos - vcn; - if (likely(!cdelta || (cdelta > 0 && cdelta < vcn_len))) { -map_buffer_cached: - BUG_ON(lcn < 0); - bh->b_blocknr = lcn_block + - (cdelta << (vol->cluster_size_bits - - blocksize_bits)) + - (bh_cofs >> blocksize_bits); - set_buffer_mapped(bh); - /* - * If the page is uptodate so is the buffer. If the - * buffer is fully outside the write, we ignore it if - * it was already allocated and we mark it dirty so it - * gets written out if we allocated it. On the other - * hand, if we allocated the buffer but we are not - * marking it dirty we set buffer_new so we can do - * error recovery. - */ - if (PageUptodate(page)) { - if (!buffer_uptodate(bh)) - set_buffer_uptodate(bh); - if (unlikely(was_hole)) { - /* We allocated the buffer. */ - clean_bdev_bh_alias(bh); - if (bh_end <= pos || bh_pos >= end) - mark_buffer_dirty(bh); - else - set_buffer_new(bh); - } - continue; - } - /* Page is _not_ uptodate. */ - if (likely(!was_hole)) { - /* - * Buffer was already allocated. If it is not - * uptodate and is only partially being written - * to, we need to read it in before the write, - * i.e. now. - */ - if (!buffer_uptodate(bh) && bh_pos < end && - bh_end > pos && - (bh_pos < pos || - bh_end > end)) { - /* - * If the buffer is fully or partially - * within the initialized size, do an - * actual read. Otherwise, simply zero - * the buffer. - */ - read_lock_irqsave(&ni->size_lock, - flags); - initialized_size = ni->initialized_size; - read_unlock_irqrestore(&ni->size_lock, - flags); - if (bh_pos < initialized_size) { - ntfs_submit_bh_for_read(bh); - *wait_bh++ = bh; - } else { - zero_user(page, bh_offset(bh), - blocksize); - set_buffer_uptodate(bh); - } - } - continue; - } - /* We allocated the buffer. */ - clean_bdev_bh_alias(bh); - /* - * If the buffer is fully outside the write, zero it, - * set it uptodate, and mark it dirty so it gets - * written out. If it is partially being written to, - * zero region surrounding the write but leave it to - * commit write to do anything else. Finally, if the - * buffer is fully being overwritten, do nothing. - */ - if (bh_end <= pos || bh_pos >= end) { - if (!buffer_uptodate(bh)) { - zero_user(page, bh_offset(bh), - blocksize); - set_buffer_uptodate(bh); - } - mark_buffer_dirty(bh); - continue; - } - set_buffer_new(bh); - if (!buffer_uptodate(bh) && - (bh_pos < pos || bh_end > end)) { - u8 *kaddr; - unsigned pofs; - - kaddr = kmap_atomic(page); - if (bh_pos < pos) { - pofs = bh_pos & ~PAGE_MASK; - memset(kaddr + pofs, 0, pos - bh_pos); - } - if (bh_end > end) { - pofs = end & ~PAGE_MASK; - memset(kaddr + pofs, 0, bh_end - end); - } - kunmap_atomic(kaddr); - flush_dcache_page(page); - } - continue; - } - /* - * Slow path: this is the first buffer in the cluster. If it - * is outside allocated size and is not uptodate, zero it and - * set it uptodate. - */ - read_lock_irqsave(&ni->size_lock, flags); - initialized_size = ni->allocated_size; - read_unlock_irqrestore(&ni->size_lock, flags); - if (bh_pos > initialized_size) { - if (PageUptodate(page)) { - if (!buffer_uptodate(bh)) - set_buffer_uptodate(bh); - } else if (!buffer_uptodate(bh)) { - zero_user(page, bh_offset(bh), blocksize); - set_buffer_uptodate(bh); - } - continue; - } - is_retry = false; - if (!rl) { - down_read(&ni->runlist.lock); -retry_remap: - rl = ni->runlist.rl; - } - if (likely(rl != NULL)) { - /* Seek to element containing target cluster. */ - while (rl->length && rl[1].vcn <= bh_cpos) - rl++; - lcn = ntfs_rl_vcn_to_lcn(rl, bh_cpos); - if (likely(lcn >= 0)) { - /* - * Successful remap, setup the map cache and - * use that to deal with the buffer. - */ - was_hole = false; - vcn = bh_cpos; - vcn_len = rl[1].vcn - vcn; - lcn_block = lcn << (vol->cluster_size_bits - - blocksize_bits); - cdelta = 0; - /* - * If the number of remaining clusters touched - * by the write is smaller or equal to the - * number of cached clusters, unlock the - * runlist as the map cache will be used from - * now on. - */ - if (likely(vcn + vcn_len >= cend)) { - if (rl_write_locked) { - up_write(&ni->runlist.lock); - rl_write_locked = false; - } else - up_read(&ni->runlist.lock); - rl = NULL; - } - goto map_buffer_cached; - } - } else - lcn = LCN_RL_NOT_MAPPED; - /* - * If it is not a hole and not out of bounds, the runlist is - * probably unmapped so try to map it now. - */ - if (unlikely(lcn != LCN_HOLE && lcn != LCN_ENOENT)) { - if (likely(!is_retry && lcn == LCN_RL_NOT_MAPPED)) { - /* Attempt to map runlist. */ - if (!rl_write_locked) { - /* - * We need the runlist locked for - * writing, so if it is locked for - * reading relock it now and retry in - * case it changed whilst we dropped - * the lock. - */ - up_read(&ni->runlist.lock); - down_write(&ni->runlist.lock); - rl_write_locked = true; - goto retry_remap; - } - err = ntfs_map_runlist_nolock(ni, bh_cpos, - NULL); - if (likely(!err)) { - is_retry = true; - goto retry_remap; - } - /* - * If @vcn is out of bounds, pretend @lcn is - * LCN_ENOENT. As long as the buffer is out - * of bounds this will work fine. - */ - if (err == -ENOENT) { - lcn = LCN_ENOENT; - err = 0; - goto rl_not_mapped_enoent; - } - } else - err = -EIO; - /* Failed to map the buffer, even after retrying. */ - bh->b_blocknr = -1; - ntfs_error(vol->sb, "Failed to write to inode 0x%lx, " - "attribute type 0x%x, vcn 0x%llx, " - "vcn offset 0x%x, because its " - "location on disk could not be " - "determined%s (error code %i).", - ni->mft_no, ni->type, - (unsigned long long)bh_cpos, - (unsigned)bh_pos & - vol->cluster_size_mask, - is_retry ? " even after retrying" : "", - err); - break; - } -rl_not_mapped_enoent: - /* - * The buffer is in a hole or out of bounds. We need to fill - * the hole, unless the buffer is in a cluster which is not - * touched by the write, in which case we just leave the buffer - * unmapped. This can only happen when the cluster size is - * less than the page cache size. - */ - if (unlikely(vol->cluster_size < PAGE_SIZE)) { - bh_cend = (bh_end + vol->cluster_size - 1) >> - vol->cluster_size_bits; - if ((bh_cend <= cpos || bh_cpos >= cend)) { - bh->b_blocknr = -1; - /* - * If the buffer is uptodate we skip it. If it - * is not but the page is uptodate, we can set - * the buffer uptodate. If the page is not - * uptodate, we can clear the buffer and set it - * uptodate. Whether this is worthwhile is - * debatable and this could be removed. - */ - if (PageUptodate(page)) { - if (!buffer_uptodate(bh)) - set_buffer_uptodate(bh); - } else if (!buffer_uptodate(bh)) { - zero_user(page, bh_offset(bh), - blocksize); - set_buffer_uptodate(bh); - } - continue; - } - } - /* - * Out of bounds buffer is invalid if it was not really out of - * bounds. - */ - BUG_ON(lcn != LCN_HOLE); - /* - * We need the runlist locked for writing, so if it is locked - * for reading relock it now and retry in case it changed - * whilst we dropped the lock. - */ - BUG_ON(!rl); - if (!rl_write_locked) { - up_read(&ni->runlist.lock); - down_write(&ni->runlist.lock); - rl_write_locked = true; - goto retry_remap; - } - /* Find the previous last allocated cluster. */ - BUG_ON(rl->lcn != LCN_HOLE); - lcn = -1; - rl2 = rl; - while (--rl2 >= ni->runlist.rl) { - if (rl2->lcn >= 0) { - lcn = rl2->lcn + rl2->length; - break; - } - } - rl2 = ntfs_cluster_alloc(vol, bh_cpos, 1, lcn, DATA_ZONE, - false); - if (IS_ERR(rl2)) { - err = PTR_ERR(rl2); - ntfs_debug("Failed to allocate cluster, error code %i.", - err); - break; - } - lcn = rl2->lcn; - rl = ntfs_runlists_merge(ni->runlist.rl, rl2); - if (IS_ERR(rl)) { - err = PTR_ERR(rl); - if (err != -ENOMEM) - err = -EIO; - if (ntfs_cluster_free_from_rl(vol, rl2)) { - ntfs_error(vol->sb, "Failed to release " - "allocated cluster in error " - "code path. Run chkdsk to " - "recover the lost cluster."); - NVolSetErrors(vol); - } - ntfs_free(rl2); - break; - } - ni->runlist.rl = rl; - status.runlist_merged = 1; - ntfs_debug("Allocated cluster, lcn 0x%llx.", - (unsigned long long)lcn); - /* Map and lock the mft record and get the attribute record. */ - if (!NInoAttr(ni)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - break; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - unmap_mft_record(base_ni); - break; - } - status.mft_attr_mapped = 1; - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, bh_cpos, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - err = -EIO; - break; - } - m = ctx->mrec; - a = ctx->attr; - /* - * Find the runlist element with which the attribute extent - * starts. Note, we cannot use the _attr_ version because we - * have mapped the mft record. That is ok because we know the - * runlist fragment must be mapped already to have ever gotten - * here, so we can just use the _rl_ version. - */ - vcn = sle64_to_cpu(a->data.non_resident.lowest_vcn); - rl2 = ntfs_rl_find_vcn_nolock(rl, vcn); - BUG_ON(!rl2); - BUG_ON(!rl2->length); - BUG_ON(rl2->lcn < LCN_HOLE); - highest_vcn = sle64_to_cpu(a->data.non_resident.highest_vcn); - /* - * If @highest_vcn is zero, calculate the real highest_vcn - * (which can really be zero). - */ - if (!highest_vcn) - highest_vcn = (sle64_to_cpu( - a->data.non_resident.allocated_size) >> - vol->cluster_size_bits) - 1; - /* - * Determine the size of the mapping pairs array for the new - * extent, i.e. the old extent with the hole filled. - */ - mp_size = ntfs_get_size_for_mapping_pairs(vol, rl2, vcn, - highest_vcn); - if (unlikely(mp_size <= 0)) { - if (!(err = mp_size)) - err = -EIO; - ntfs_debug("Failed to get size for mapping pairs " - "array, error code %i.", err); - break; - } - /* - * Resize the attribute record to fit the new mapping pairs - * array. - */ - attr_rec_len = le32_to_cpu(a->length); - err = ntfs_attr_record_resize(m, a, mp_size + le16_to_cpu( - a->data.non_resident.mapping_pairs_offset)); - if (unlikely(err)) { - BUG_ON(err != -ENOSPC); - // TODO: Deal with this by using the current attribute - // and fill it with as much of the mapping pairs - // array as possible. Then loop over each attribute - // extent rewriting the mapping pairs arrays as we go - // along and if when we reach the end we have not - // enough space, try to resize the last attribute - // extent and if even that fails, add a new attribute - // extent. - // We could also try to resize at each step in the hope - // that we will not need to rewrite every single extent. - // Note, we may need to decompress some extents to fill - // the runlist as we are walking the extents... - ntfs_error(vol->sb, "Not enough space in the mft " - "record for the extended attribute " - "record. This case is not " - "implemented yet."); - err = -EOPNOTSUPP; - break ; - } - status.mp_rebuilt = 1; - /* - * Generate the mapping pairs array directly into the attribute - * record. - */ - err = ntfs_mapping_pairs_build(vol, (u8*)a + le16_to_cpu( - a->data.non_resident.mapping_pairs_offset), - mp_size, rl2, vcn, highest_vcn, NULL); - if (unlikely(err)) { - ntfs_error(vol->sb, "Cannot fill hole in inode 0x%lx, " - "attribute type 0x%x, because building " - "the mapping pairs failed with error " - "code %i.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), err); - err = -EIO; - break; - } - /* Update the highest_vcn but only if it was not set. */ - if (unlikely(!a->data.non_resident.highest_vcn)) - a->data.non_resident.highest_vcn = - cpu_to_sle64(highest_vcn); - /* - * If the attribute is sparse/compressed, update the compressed - * size in the ntfs_inode structure and the attribute record. - */ - if (likely(NInoSparse(ni) || NInoCompressed(ni))) { - /* - * If we are not in the first attribute extent, switch - * to it, but first ensure the changes will make it to - * disk later. - */ - if (a->data.non_resident.lowest_vcn) { - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_reinit_search_ctx(ctx); - err = ntfs_attr_lookup(ni->type, ni->name, - ni->name_len, CASE_SENSITIVE, - 0, NULL, 0, ctx); - if (unlikely(err)) { - status.attr_switched = 1; - break; - } - /* @m is not used any more so do not set it. */ - a = ctx->attr; - } - write_lock_irqsave(&ni->size_lock, flags); - ni->itype.compressed.size += vol->cluster_size; - a->data.non_resident.compressed_size = - cpu_to_sle64(ni->itype.compressed.size); - write_unlock_irqrestore(&ni->size_lock, flags); - } - /* Ensure the changes make it to disk. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - /* Successfully filled the hole. */ - status.runlist_merged = 0; - status.mft_attr_mapped = 0; - status.mp_rebuilt = 0; - /* Setup the map cache and use that to deal with the buffer. */ - was_hole = true; - vcn = bh_cpos; - vcn_len = 1; - lcn_block = lcn << (vol->cluster_size_bits - blocksize_bits); - cdelta = 0; - /* - * If the number of remaining clusters in the @pages is smaller - * or equal to the number of cached clusters, unlock the - * runlist as the map cache will be used from now on. - */ - if (likely(vcn + vcn_len >= cend)) { - up_write(&ni->runlist.lock); - rl_write_locked = false; - rl = NULL; - } - goto map_buffer_cached; - } while (bh_pos += blocksize, (bh = bh->b_this_page) != head); - /* If there are no errors, do the next page. */ - if (likely(!err && ++u < nr_pages)) - goto do_next_page; - /* If there are no errors, release the runlist lock if we took it. */ - if (likely(!err)) { - if (unlikely(rl_write_locked)) { - up_write(&ni->runlist.lock); - rl_write_locked = false; - } else if (unlikely(rl)) - up_read(&ni->runlist.lock); - rl = NULL; - } - /* If we issued read requests, let them complete. */ - read_lock_irqsave(&ni->size_lock, flags); - initialized_size = ni->initialized_size; - read_unlock_irqrestore(&ni->size_lock, flags); - while (wait_bh > wait) { - bh = *--wait_bh; - wait_on_buffer(bh); - if (likely(buffer_uptodate(bh))) { - page = bh->b_page; - bh_pos = ((s64)page->index << PAGE_SHIFT) + - bh_offset(bh); - /* - * If the buffer overflows the initialized size, need - * to zero the overflowing region. - */ - if (unlikely(bh_pos + blocksize > initialized_size)) { - int ofs = 0; - - if (likely(bh_pos < initialized_size)) - ofs = initialized_size - bh_pos; - zero_user_segment(page, bh_offset(bh) + ofs, - blocksize); - } - } else /* if (unlikely(!buffer_uptodate(bh))) */ - err = -EIO; - } - if (likely(!err)) { - /* Clear buffer_new on all buffers. */ - u = 0; - do { - bh = head = page_buffers(pages[u]); - do { - if (buffer_new(bh)) - clear_buffer_new(bh); - } while ((bh = bh->b_this_page) != head); - } while (++u < nr_pages); - ntfs_debug("Done."); - return err; - } - if (status.attr_switched) { - /* Get back to the attribute extent we modified. */ - ntfs_attr_reinit_search_ctx(ctx); - if (ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, bh_cpos, NULL, 0, ctx)) { - ntfs_error(vol->sb, "Failed to find required " - "attribute extent of attribute in " - "error code path. Run chkdsk to " - "recover."); - write_lock_irqsave(&ni->size_lock, flags); - ni->itype.compressed.size += vol->cluster_size; - write_unlock_irqrestore(&ni->size_lock, flags); - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - /* - * The only thing that is now wrong is the compressed - * size of the base attribute extent which chkdsk - * should be able to fix. - */ - NVolSetErrors(vol); - } else { - m = ctx->mrec; - a = ctx->attr; - status.attr_switched = 0; - } - } - /* - * If the runlist has been modified, need to restore it by punching a - * hole into it and we then need to deallocate the on-disk cluster as - * well. Note, we only modify the runlist if we are able to generate a - * new mapping pairs array, i.e. only when the mapped attribute extent - * is not switched. - */ - if (status.runlist_merged && !status.attr_switched) { - BUG_ON(!rl_write_locked); - /* Make the file cluster we allocated sparse in the runlist. */ - if (ntfs_rl_punch_nolock(vol, &ni->runlist, bh_cpos, 1)) { - ntfs_error(vol->sb, "Failed to punch hole into " - "attribute runlist in error code " - "path. Run chkdsk to recover the " - "lost cluster."); - NVolSetErrors(vol); - } else /* if (success) */ { - status.runlist_merged = 0; - /* - * Deallocate the on-disk cluster we allocated but only - * if we succeeded in punching its vcn out of the - * runlist. - */ - down_write(&vol->lcnbmp_lock); - if (ntfs_bitmap_clear_bit(vol->lcnbmp_ino, lcn)) { - ntfs_error(vol->sb, "Failed to release " - "allocated cluster in error " - "code path. Run chkdsk to " - "recover the lost cluster."); - NVolSetErrors(vol); - } - up_write(&vol->lcnbmp_lock); - } - } - /* - * Resize the attribute record to its old size and rebuild the mapping - * pairs array. Note, we only can do this if the runlist has been - * restored to its old state which also implies that the mapped - * attribute extent is not switched. - */ - if (status.mp_rebuilt && !status.runlist_merged) { - if (ntfs_attr_record_resize(m, a, attr_rec_len)) { - ntfs_error(vol->sb, "Failed to restore attribute " - "record in error code path. Run " - "chkdsk to recover."); - NVolSetErrors(vol); - } else /* if (success) */ { - if (ntfs_mapping_pairs_build(vol, (u8*)a + - le16_to_cpu(a->data.non_resident. - mapping_pairs_offset), attr_rec_len - - le16_to_cpu(a->data.non_resident. - mapping_pairs_offset), ni->runlist.rl, - vcn, highest_vcn, NULL)) { - ntfs_error(vol->sb, "Failed to restore " - "mapping pairs array in error " - "code path. Run chkdsk to " - "recover."); - NVolSetErrors(vol); - } - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - } - } - /* Release the mft record and the attribute. */ - if (status.mft_attr_mapped) { - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - } - /* Release the runlist lock. */ - if (rl_write_locked) - up_write(&ni->runlist.lock); - else if (rl) - up_read(&ni->runlist.lock); - /* - * Zero out any newly allocated blocks to avoid exposing stale data. - * If BH_New is set, we know that the block was newly allocated above - * and that it has not been fully zeroed and marked dirty yet. - */ - nr_pages = u; - u = 0; - end = bh_cpos << vol->cluster_size_bits; - do { - page = pages[u]; - bh = head = page_buffers(page); - do { - if (u == nr_pages && - ((s64)page->index << PAGE_SHIFT) + - bh_offset(bh) >= end) - break; - if (!buffer_new(bh)) - continue; - clear_buffer_new(bh); - if (!buffer_uptodate(bh)) { - if (PageUptodate(page)) - set_buffer_uptodate(bh); - else { - zero_user(page, bh_offset(bh), - blocksize); - set_buffer_uptodate(bh); - } - } - mark_buffer_dirty(bh); - } while ((bh = bh->b_this_page) != head); - } while (++u <= nr_pages); - ntfs_error(vol->sb, "Failed. Returning error code %i.", err); - return err; -} - -static inline void ntfs_flush_dcache_pages(struct page **pages, - unsigned nr_pages) -{ - BUG_ON(!nr_pages); - /* - * Warning: Do not do the decrement at the same time as the call to - * flush_dcache_page() because it is a NULL macro on i386 and hence the - * decrement never happens so the loop never terminates. - */ - do { - --nr_pages; - flush_dcache_page(pages[nr_pages]); - } while (nr_pages > 0); -} - -/** - * ntfs_commit_pages_after_non_resident_write - commit the received data - * @pages: array of destination pages - * @nr_pages: number of pages in @pages - * @pos: byte position in file at which the write begins - * @bytes: number of bytes to be written - * - * See description of ntfs_commit_pages_after_write(), below. - */ -static inline int ntfs_commit_pages_after_non_resident_write( - struct page **pages, const unsigned nr_pages, - s64 pos, size_t bytes) -{ - s64 end, initialized_size; - struct inode *vi; - ntfs_inode *ni, *base_ni; - struct buffer_head *bh, *head; - ntfs_attr_search_ctx *ctx; - MFT_RECORD *m; - ATTR_RECORD *a; - unsigned long flags; - unsigned blocksize, u; - int err; - - vi = pages[0]->mapping->host; - ni = NTFS_I(vi); - blocksize = vi->i_sb->s_blocksize; - end = pos + bytes; - u = 0; - do { - s64 bh_pos; - struct page *page; - bool partial; - - page = pages[u]; - bh_pos = (s64)page->index << PAGE_SHIFT; - bh = head = page_buffers(page); - partial = false; - do { - s64 bh_end; - - bh_end = bh_pos + blocksize; - if (bh_end <= pos || bh_pos >= end) { - if (!buffer_uptodate(bh)) - partial = true; - } else { - set_buffer_uptodate(bh); - mark_buffer_dirty(bh); - } - } while (bh_pos += blocksize, (bh = bh->b_this_page) != head); - /* - * If all buffers are now uptodate but the page is not, set the - * page uptodate. - */ - if (!partial && !PageUptodate(page)) - SetPageUptodate(page); - } while (++u < nr_pages); - /* - * Finally, if we do not need to update initialized_size or i_size we - * are finished. - */ - read_lock_irqsave(&ni->size_lock, flags); - initialized_size = ni->initialized_size; - read_unlock_irqrestore(&ni->size_lock, flags); - if (end <= initialized_size) { - ntfs_debug("Done."); - return 0; - } - /* - * Update initialized_size/i_size as appropriate, both in the inode and - * the mft record. - */ - if (!NInoAttr(ni)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - /* Map, pin, and lock the mft record. */ - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - m = NULL; - ctx = NULL; - goto err_out; - } - BUG_ON(!NInoNonResident(ni)); - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - err = -EIO; - goto err_out; - } - a = ctx->attr; - BUG_ON(!a->non_resident); - write_lock_irqsave(&ni->size_lock, flags); - BUG_ON(end > ni->allocated_size); - ni->initialized_size = end; - a->data.non_resident.initialized_size = cpu_to_sle64(end); - if (end > i_size_read(vi)) { - i_size_write(vi, end); - a->data.non_resident.data_size = - a->data.non_resident.initialized_size; - } - write_unlock_irqrestore(&ni->size_lock, flags); - /* Mark the mft record dirty, so it gets written back. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - ntfs_debug("Done."); - return 0; -err_out: - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(base_ni); - ntfs_error(vi->i_sb, "Failed to update initialized_size/i_size (error " - "code %i).", err); - if (err != -ENOMEM) - NVolSetErrors(ni->vol); - return err; -} - -/** - * ntfs_commit_pages_after_write - commit the received data - * @pages: array of destination pages - * @nr_pages: number of pages in @pages - * @pos: byte position in file at which the write begins - * @bytes: number of bytes to be written - * - * This is called from ntfs_file_buffered_write() with i_mutex held on the inode - * (@pages[0]->mapping->host). There are @nr_pages pages in @pages which are - * locked but not kmap()ped. The source data has already been copied into the - * @page. ntfs_prepare_pages_for_non_resident_write() has been called before - * the data was copied (for non-resident attributes only) and it returned - * success. - * - * Need to set uptodate and mark dirty all buffers within the boundary of the - * write. If all buffers in a page are uptodate we set the page uptodate, too. - * - * Setting the buffers dirty ensures that they get written out later when - * ntfs_writepage() is invoked by the VM. - * - * Finally, we need to update i_size and initialized_size as appropriate both - * in the inode and the mft record. - * - * This is modelled after fs/buffer.c::generic_commit_write(), which marks - * buffers uptodate and dirty, sets the page uptodate if all buffers in the - * page are uptodate, and updates i_size if the end of io is beyond i_size. In - * that case, it also marks the inode dirty. - * - * If things have gone as outlined in - * ntfs_prepare_pages_for_non_resident_write(), we do not need to do any page - * content modifications here for non-resident attributes. For resident - * attributes we need to do the uptodate bringing here which we combine with - * the copying into the mft record which means we save one atomic kmap. - * - * Return 0 on success or -errno on error. - */ -static int ntfs_commit_pages_after_write(struct page **pages, - const unsigned nr_pages, s64 pos, size_t bytes) -{ - s64 end, initialized_size; - loff_t i_size; - struct inode *vi; - ntfs_inode *ni, *base_ni; - struct page *page; - ntfs_attr_search_ctx *ctx; - MFT_RECORD *m; - ATTR_RECORD *a; - char *kattr, *kaddr; - unsigned long flags; - u32 attr_len; - int err; - - BUG_ON(!nr_pages); - BUG_ON(!pages); - page = pages[0]; - BUG_ON(!page); - vi = page->mapping->host; - ni = NTFS_I(vi); - ntfs_debug("Entering for inode 0x%lx, attribute type 0x%x, start page " - "index 0x%lx, nr_pages 0x%x, pos 0x%llx, bytes 0x%zx.", - vi->i_ino, ni->type, page->index, nr_pages, - (long long)pos, bytes); - if (NInoNonResident(ni)) - return ntfs_commit_pages_after_non_resident_write(pages, - nr_pages, pos, bytes); - BUG_ON(nr_pages > 1); - /* - * Attribute is resident, implying it is not compressed, encrypted, or - * sparse. - */ - if (!NInoAttr(ni)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - BUG_ON(NInoNonResident(ni)); - /* Map, pin, and lock the mft record. */ - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - m = NULL; - ctx = NULL; - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - err = -EIO; - goto err_out; - } - a = ctx->attr; - BUG_ON(a->non_resident); - /* The total length of the attribute value. */ - attr_len = le32_to_cpu(a->data.resident.value_length); - i_size = i_size_read(vi); - BUG_ON(attr_len != i_size); - BUG_ON(pos > attr_len); - end = pos + bytes; - BUG_ON(end > le32_to_cpu(a->length) - - le16_to_cpu(a->data.resident.value_offset)); - kattr = (u8*)a + le16_to_cpu(a->data.resident.value_offset); - kaddr = kmap_atomic(page); - /* Copy the received data from the page to the mft record. */ - memcpy(kattr + pos, kaddr + pos, bytes); - /* Update the attribute length if necessary. */ - if (end > attr_len) { - attr_len = end; - a->data.resident.value_length = cpu_to_le32(attr_len); - } - /* - * If the page is not uptodate, bring the out of bounds area(s) - * uptodate by copying data from the mft record to the page. - */ - if (!PageUptodate(page)) { - if (pos > 0) - memcpy(kaddr, kattr, pos); - if (end < attr_len) - memcpy(kaddr + end, kattr + end, attr_len - end); - /* Zero the region outside the end of the attribute value. */ - memset(kaddr + attr_len, 0, PAGE_SIZE - attr_len); - flush_dcache_page(page); - SetPageUptodate(page); - } - kunmap_atomic(kaddr); - /* Update initialized_size/i_size if necessary. */ - read_lock_irqsave(&ni->size_lock, flags); - initialized_size = ni->initialized_size; - BUG_ON(end > ni->allocated_size); - read_unlock_irqrestore(&ni->size_lock, flags); - BUG_ON(initialized_size != i_size); - if (end > initialized_size) { - write_lock_irqsave(&ni->size_lock, flags); - ni->initialized_size = end; - i_size_write(vi, end); - write_unlock_irqrestore(&ni->size_lock, flags); - } - /* Mark the mft record dirty, so it gets written back. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - ntfs_debug("Done."); - return 0; -err_out: - if (err == -ENOMEM) { - ntfs_warning(vi->i_sb, "Error allocating memory required to " - "commit the write."); - if (PageUptodate(page)) { - ntfs_warning(vi->i_sb, "Page is uptodate, setting " - "dirty so the write will be retried " - "later on by the VM."); - /* - * Put the page on mapping->dirty_pages, but leave its - * buffers' dirty state as-is. - */ - __set_page_dirty_nobuffers(page); - err = 0; - } else - ntfs_error(vi->i_sb, "Page is not uptodate. Written " - "data has been lost."); - } else { - ntfs_error(vi->i_sb, "Resident attribute commit write failed " - "with error %i.", err); - NVolSetErrors(ni->vol); - } - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(base_ni); - return err; -} - -/* - * Copy as much as we can into the pages and return the number of bytes which - * were successfully copied. If a fault is encountered then clear the pages - * out to (ofs + bytes) and return the number of bytes which were copied. - */ -static size_t ntfs_copy_from_user_iter(struct page **pages, unsigned nr_pages, - unsigned ofs, struct iov_iter *i, size_t bytes) -{ - struct page **last_page = pages + nr_pages; - size_t total = 0; - unsigned len, copied; - - do { - len = PAGE_SIZE - ofs; - if (len > bytes) - len = bytes; - copied = copy_page_from_iter_atomic(*pages, ofs, len, i); - total += copied; - bytes -= copied; - if (!bytes) - break; - if (copied < len) - goto err; - ofs = 0; - } while (++pages < last_page); -out: - return total; -err: - /* Zero the rest of the target like __copy_from_user(). */ - len = PAGE_SIZE - copied; - do { - if (len > bytes) - len = bytes; - zero_user(*pages, copied, len); - bytes -= len; - copied = 0; - len = PAGE_SIZE; - } while (++pages < last_page); - goto out; -} - -/** - * ntfs_perform_write - perform buffered write to a file - * @file: file to write to - * @i: iov_iter with data to write - * @pos: byte offset in file at which to begin writing to - */ -static ssize_t ntfs_perform_write(struct file *file, struct iov_iter *i, - loff_t pos) -{ - struct address_space *mapping = file->f_mapping; - struct inode *vi = mapping->host; - ntfs_inode *ni = NTFS_I(vi); - ntfs_volume *vol = ni->vol; - struct page *pages[NTFS_MAX_PAGES_PER_CLUSTER]; - struct page *cached_page = NULL; - VCN last_vcn; - LCN lcn; - size_t bytes; - ssize_t status, written = 0; - unsigned nr_pages; - - ntfs_debug("Entering for i_ino 0x%lx, attribute type 0x%x, pos " - "0x%llx, count 0x%lx.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), - (unsigned long long)pos, - (unsigned long)iov_iter_count(i)); - /* - * If a previous ntfs_truncate() failed, repeat it and abort if it - * fails again. - */ - if (unlikely(NInoTruncateFailed(ni))) { - int err; - - inode_dio_wait(vi); - err = ntfs_truncate(vi); - if (err || NInoTruncateFailed(ni)) { - if (!err) - err = -EIO; - ntfs_error(vol->sb, "Cannot perform write to inode " - "0x%lx, attribute type 0x%x, because " - "ntfs_truncate() failed (error code " - "%i).", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), err); - return err; - } - } - /* - * Determine the number of pages per cluster for non-resident - * attributes. - */ - nr_pages = 1; - if (vol->cluster_size > PAGE_SIZE && NInoNonResident(ni)) - nr_pages = vol->cluster_size >> PAGE_SHIFT; - last_vcn = -1; - do { - VCN vcn; - pgoff_t start_idx; - unsigned ofs, do_pages, u; - size_t copied; - - start_idx = pos >> PAGE_SHIFT; - ofs = pos & ~PAGE_MASK; - bytes = PAGE_SIZE - ofs; - do_pages = 1; - if (nr_pages > 1) { - vcn = pos >> vol->cluster_size_bits; - if (vcn != last_vcn) { - last_vcn = vcn; - /* - * Get the lcn of the vcn the write is in. If - * it is a hole, need to lock down all pages in - * the cluster. - */ - down_read(&ni->runlist.lock); - lcn = ntfs_attr_vcn_to_lcn_nolock(ni, pos >> - vol->cluster_size_bits, false); - up_read(&ni->runlist.lock); - if (unlikely(lcn < LCN_HOLE)) { - if (lcn == LCN_ENOMEM) - status = -ENOMEM; - else { - status = -EIO; - ntfs_error(vol->sb, "Cannot " - "perform write to " - "inode 0x%lx, " - "attribute type 0x%x, " - "because the attribute " - "is corrupt.", - vi->i_ino, (unsigned) - le32_to_cpu(ni->type)); - } - break; - } - if (lcn == LCN_HOLE) { - start_idx = (pos & ~(s64) - vol->cluster_size_mask) - >> PAGE_SHIFT; - bytes = vol->cluster_size - (pos & - vol->cluster_size_mask); - do_pages = nr_pages; - } - } - } - if (bytes > iov_iter_count(i)) - bytes = iov_iter_count(i); -again: - /* - * Bring in the user page(s) that we will copy from _first_. - * Otherwise there is a nasty deadlock on copying from the same - * page(s) as we are writing to, without it/them being marked - * up-to-date. Note, at present there is nothing to stop the - * pages being swapped out between us bringing them into memory - * and doing the actual copying. - */ - if (unlikely(fault_in_iov_iter_readable(i, bytes))) { - status = -EFAULT; - break; - } - /* Get and lock @do_pages starting at index @start_idx. */ - status = __ntfs_grab_cache_pages(mapping, start_idx, do_pages, - pages, &cached_page); - if (unlikely(status)) - break; - /* - * For non-resident attributes, we need to fill any holes with - * actual clusters and ensure all bufferes are mapped. We also - * need to bring uptodate any buffers that are only partially - * being written to. - */ - if (NInoNonResident(ni)) { - status = ntfs_prepare_pages_for_non_resident_write( - pages, do_pages, pos, bytes); - if (unlikely(status)) { - do { - unlock_page(pages[--do_pages]); - put_page(pages[do_pages]); - } while (do_pages); - break; - } - } - u = (pos >> PAGE_SHIFT) - pages[0]->index; - copied = ntfs_copy_from_user_iter(pages + u, do_pages - u, ofs, - i, bytes); - ntfs_flush_dcache_pages(pages + u, do_pages - u); - status = 0; - if (likely(copied == bytes)) { - status = ntfs_commit_pages_after_write(pages, do_pages, - pos, bytes); - } - do { - unlock_page(pages[--do_pages]); - put_page(pages[do_pages]); - } while (do_pages); - if (unlikely(status < 0)) { - iov_iter_revert(i, copied); - break; - } - cond_resched(); - if (unlikely(copied < bytes)) { - iov_iter_revert(i, copied); - if (copied) - bytes = copied; - else if (bytes > PAGE_SIZE - ofs) - bytes = PAGE_SIZE - ofs; - goto again; - } - pos += copied; - written += copied; - balance_dirty_pages_ratelimited(mapping); - if (fatal_signal_pending(current)) { - status = -EINTR; - break; - } - } while (iov_iter_count(i)); - if (cached_page) - put_page(cached_page); - ntfs_debug("Done. Returning %s (written 0x%lx, status %li).", - written ? "written" : "status", (unsigned long)written, - (long)status); - return written ? written : status; -} - -/** - * ntfs_file_write_iter - simple wrapper for ntfs_file_write_iter_nolock() - * @iocb: IO state structure - * @from: iov_iter with data to write - * - * Basically the same as generic_file_write_iter() except that it ends up - * up calling ntfs_perform_write() instead of generic_perform_write() and that - * O_DIRECT is not implemented. - */ -static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from) -{ - struct file *file = iocb->ki_filp; - struct inode *vi = file_inode(file); - ssize_t written = 0; - ssize_t err; - - inode_lock(vi); - /* We can write back this queue in page reclaim. */ - err = ntfs_prepare_file_for_write(iocb, from); - if (iov_iter_count(from) && !err) - written = ntfs_perform_write(file, from, iocb->ki_pos); - inode_unlock(vi); - iocb->ki_pos += written; - if (likely(written > 0)) - written = generic_write_sync(iocb, written); - return written ? written : err; -} - -/** - * ntfs_file_fsync - sync a file to disk - * @filp: file to be synced - * @datasync: if non-zero only flush user data and not metadata - * - * Data integrity sync of a file to disk. Used for fsync, fdatasync, and msync - * system calls. This function is inspired by fs/buffer.c::file_fsync(). - * - * If @datasync is false, write the mft record and all associated extent mft - * records as well as the $DATA attribute and then sync the block device. - * - * If @datasync is true and the attribute is non-resident, we skip the writing - * of the mft record and all associated extent mft records (this might still - * happen due to the write_inode_now() call). - * - * Also, if @datasync is true, we do not wait on the inode to be written out - * but we always wait on the page cache pages to be written out. - * - * Locking: Caller must hold i_mutex on the inode. - * - * TODO: We should probably also write all attribute/index inodes associated - * with this inode but since we have no simple way of getting to them we ignore - * this problem for now. - */ -static int ntfs_file_fsync(struct file *filp, loff_t start, loff_t end, - int datasync) -{ - struct inode *vi = filp->f_mapping->host; - int err, ret = 0; - - ntfs_debug("Entering for inode 0x%lx.", vi->i_ino); - - err = file_write_and_wait_range(filp, start, end); - if (err) - return err; - inode_lock(vi); - - BUG_ON(S_ISDIR(vi->i_mode)); - if (!datasync || !NInoNonResident(NTFS_I(vi))) - ret = __ntfs_write_inode(vi, 1); - write_inode_now(vi, !datasync); - /* - * NOTE: If we were to use mapping->private_list (see ext2 and - * fs/buffer.c) for dirty blocks then we could optimize the below to be - * sync_mapping_buffers(vi->i_mapping). - */ - err = sync_blockdev(vi->i_sb->s_bdev); - if (unlikely(err && !ret)) - ret = err; - if (likely(!ret)) - ntfs_debug("Done."); - else - ntfs_warning(vi->i_sb, "Failed to f%ssync inode 0x%lx. Error " - "%u.", datasync ? "data" : "", vi->i_ino, -ret); - inode_unlock(vi); - return ret; -} - -#endif /* NTFS_RW */ - -const struct file_operations ntfs_file_ops = { - .llseek = generic_file_llseek, - .read_iter = generic_file_read_iter, -#ifdef NTFS_RW - .write_iter = ntfs_file_write_iter, - .fsync = ntfs_file_fsync, -#endif /* NTFS_RW */ - .mmap = generic_file_mmap, - .open = ntfs_file_open, - .splice_read = filemap_splice_read, -}; - -const struct inode_operations ntfs_file_inode_ops = { -#ifdef NTFS_RW - .setattr = ntfs_setattr, -#endif /* NTFS_RW */ -}; - -const struct file_operations ntfs_empty_file_ops = {}; - -const struct inode_operations ntfs_empty_inode_ops = {}; diff --git a/fs/ntfs/index.c b/fs/ntfs/index.c deleted file mode 100644 index d46c2c03a032..000000000000 --- a/fs/ntfs/index.c +++ /dev/null @@ -1,440 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * index.c - NTFS kernel index handling. Part of the Linux-NTFS project. - * - * Copyright (c) 2004-2005 Anton Altaparmakov - */ - -#include - -#include "aops.h" -#include "collate.h" -#include "debug.h" -#include "index.h" -#include "ntfs.h" - -/** - * ntfs_index_ctx_get - allocate and initialize a new index context - * @idx_ni: ntfs index inode with which to initialize the context - * - * Allocate a new index context, initialize it with @idx_ni and return it. - * Return NULL if allocation failed. - * - * Locking: Caller must hold i_mutex on the index inode. - */ -ntfs_index_context *ntfs_index_ctx_get(ntfs_inode *idx_ni) -{ - ntfs_index_context *ictx; - - ictx = kmem_cache_alloc(ntfs_index_ctx_cache, GFP_NOFS); - if (ictx) - *ictx = (ntfs_index_context){ .idx_ni = idx_ni }; - return ictx; -} - -/** - * ntfs_index_ctx_put - release an index context - * @ictx: index context to free - * - * Release the index context @ictx, releasing all associated resources. - * - * Locking: Caller must hold i_mutex on the index inode. - */ -void ntfs_index_ctx_put(ntfs_index_context *ictx) -{ - if (ictx->entry) { - if (ictx->is_in_root) { - if (ictx->actx) - ntfs_attr_put_search_ctx(ictx->actx); - if (ictx->base_ni) - unmap_mft_record(ictx->base_ni); - } else { - struct page *page = ictx->page; - if (page) { - BUG_ON(!PageLocked(page)); - unlock_page(page); - ntfs_unmap_page(page); - } - } - } - kmem_cache_free(ntfs_index_ctx_cache, ictx); - return; -} - -/** - * ntfs_index_lookup - find a key in an index and return its index entry - * @key: [IN] key for which to search in the index - * @key_len: [IN] length of @key in bytes - * @ictx: [IN/OUT] context describing the index and the returned entry - * - * Before calling ntfs_index_lookup(), @ictx must have been obtained from a - * call to ntfs_index_ctx_get(). - * - * Look for the @key in the index specified by the index lookup context @ictx. - * ntfs_index_lookup() walks the contents of the index looking for the @key. - * - * If the @key is found in the index, 0 is returned and @ictx is setup to - * describe the index entry containing the matching @key. @ictx->entry is the - * index entry and @ictx->data and @ictx->data_len are the index entry data and - * its length in bytes, respectively. - * - * If the @key is not found in the index, -ENOENT is returned and @ictx is - * setup to describe the index entry whose key collates immediately after the - * search @key, i.e. this is the position in the index at which an index entry - * with a key of @key would need to be inserted. - * - * If an error occurs return the negative error code and @ictx is left - * untouched. - * - * When finished with the entry and its data, call ntfs_index_ctx_put() to free - * the context and other associated resources. - * - * If the index entry was modified, call flush_dcache_index_entry_page() - * immediately after the modification and either ntfs_index_entry_mark_dirty() - * or ntfs_index_entry_write() before the call to ntfs_index_ctx_put() to - * ensure that the changes are written to disk. - * - * Locking: - Caller must hold i_mutex on the index inode. - * - Each page cache page in the index allocation mapping must be - * locked whilst being accessed otherwise we may find a corrupt - * page due to it being under ->writepage at the moment which - * applies the mst protection fixups before writing out and then - * removes them again after the write is complete after which it - * unlocks the page. - */ -int ntfs_index_lookup(const void *key, const int key_len, - ntfs_index_context *ictx) -{ - VCN vcn, old_vcn; - ntfs_inode *idx_ni = ictx->idx_ni; - ntfs_volume *vol = idx_ni->vol; - struct super_block *sb = vol->sb; - ntfs_inode *base_ni = idx_ni->ext.base_ntfs_ino; - MFT_RECORD *m; - INDEX_ROOT *ir; - INDEX_ENTRY *ie; - INDEX_ALLOCATION *ia; - u8 *index_end, *kaddr; - ntfs_attr_search_ctx *actx; - struct address_space *ia_mapping; - struct page *page; - int rc, err = 0; - - ntfs_debug("Entering."); - BUG_ON(!NInoAttr(idx_ni)); - BUG_ON(idx_ni->type != AT_INDEX_ALLOCATION); - BUG_ON(idx_ni->nr_extents != -1); - BUG_ON(!base_ni); - BUG_ON(!key); - BUG_ON(key_len <= 0); - if (!ntfs_is_collation_rule_supported( - idx_ni->itype.index.collation_rule)) { - ntfs_error(sb, "Index uses unsupported collation rule 0x%x. " - "Aborting lookup.", le32_to_cpu( - idx_ni->itype.index.collation_rule)); - return -EOPNOTSUPP; - } - /* Get hold of the mft record for the index inode. */ - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - ntfs_error(sb, "map_mft_record() failed with error code %ld.", - -PTR_ERR(m)); - return PTR_ERR(m); - } - actx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!actx)) { - err = -ENOMEM; - goto err_out; - } - /* Find the index root attribute in the mft record. */ - err = ntfs_attr_lookup(AT_INDEX_ROOT, idx_ni->name, idx_ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, actx); - if (unlikely(err)) { - if (err == -ENOENT) { - ntfs_error(sb, "Index root attribute missing in inode " - "0x%lx.", idx_ni->mft_no); - err = -EIO; - } - goto err_out; - } - /* Get to the index root value (it has been verified in read_inode). */ - ir = (INDEX_ROOT*)((u8*)actx->attr + - le16_to_cpu(actx->attr->data.resident.value_offset)); - index_end = (u8*)&ir->index + le32_to_cpu(ir->index.index_length); - /* The first index entry. */ - ie = (INDEX_ENTRY*)((u8*)&ir->index + - le32_to_cpu(ir->index.entries_offset)); - /* - * Loop until we exceed valid memory (corruption case) or until we - * reach the last entry. - */ - for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) { - /* Bounds checks. */ - if ((u8*)ie < (u8*)actx->mrec || (u8*)ie + - sizeof(INDEX_ENTRY_HEADER) > index_end || - (u8*)ie + le16_to_cpu(ie->length) > index_end) - goto idx_err_out; - /* - * The last entry cannot contain a key. It can however contain - * a pointer to a child node in the B+tree so we just break out. - */ - if (ie->flags & INDEX_ENTRY_END) - break; - /* Further bounds checks. */ - if ((u32)sizeof(INDEX_ENTRY_HEADER) + - le16_to_cpu(ie->key_length) > - le16_to_cpu(ie->data.vi.data_offset) || - (u32)le16_to_cpu(ie->data.vi.data_offset) + - le16_to_cpu(ie->data.vi.data_length) > - le16_to_cpu(ie->length)) - goto idx_err_out; - /* If the keys match perfectly, we setup @ictx and return 0. */ - if ((key_len == le16_to_cpu(ie->key_length)) && !memcmp(key, - &ie->key, key_len)) { -ir_done: - ictx->is_in_root = true; - ictx->ir = ir; - ictx->actx = actx; - ictx->base_ni = base_ni; - ictx->ia = NULL; - ictx->page = NULL; -done: - ictx->entry = ie; - ictx->data = (u8*)ie + - le16_to_cpu(ie->data.vi.data_offset); - ictx->data_len = le16_to_cpu(ie->data.vi.data_length); - ntfs_debug("Done."); - return err; - } - /* - * Not a perfect match, need to do full blown collation so we - * know which way in the B+tree we have to go. - */ - rc = ntfs_collate(vol, idx_ni->itype.index.collation_rule, key, - key_len, &ie->key, le16_to_cpu(ie->key_length)); - /* - * If @key collates before the key of the current entry, there - * is definitely no such key in this index but we might need to - * descend into the B+tree so we just break out of the loop. - */ - if (rc == -1) - break; - /* - * A match should never happen as the memcmp() call should have - * cought it, but we still treat it correctly. - */ - if (!rc) - goto ir_done; - /* The keys are not equal, continue the search. */ - } - /* - * We have finished with this index without success. Check for the - * presence of a child node and if not present setup @ictx and return - * -ENOENT. - */ - if (!(ie->flags & INDEX_ENTRY_NODE)) { - ntfs_debug("Entry not found."); - err = -ENOENT; - goto ir_done; - } /* Child node present, descend into it. */ - /* Consistency check: Verify that an index allocation exists. */ - if (!NInoIndexAllocPresent(idx_ni)) { - ntfs_error(sb, "No index allocation attribute but index entry " - "requires one. Inode 0x%lx is corrupt or " - "driver bug.", idx_ni->mft_no); - goto err_out; - } - /* Get the starting vcn of the index_block holding the child node. */ - vcn = sle64_to_cpup((sle64*)((u8*)ie + le16_to_cpu(ie->length) - 8)); - ia_mapping = VFS_I(idx_ni)->i_mapping; - /* - * We are done with the index root and the mft record. Release them, - * otherwise we deadlock with ntfs_map_page(). - */ - ntfs_attr_put_search_ctx(actx); - unmap_mft_record(base_ni); - m = NULL; - actx = NULL; -descend_into_child_node: - /* - * Convert vcn to index into the index allocation attribute in units - * of PAGE_SIZE and map the page cache page, reading it from - * disk if necessary. - */ - page = ntfs_map_page(ia_mapping, vcn << - idx_ni->itype.index.vcn_size_bits >> PAGE_SHIFT); - if (IS_ERR(page)) { - ntfs_error(sb, "Failed to map index page, error %ld.", - -PTR_ERR(page)); - err = PTR_ERR(page); - goto err_out; - } - lock_page(page); - kaddr = (u8*)page_address(page); -fast_descend_into_child_node: - /* Get to the index allocation block. */ - ia = (INDEX_ALLOCATION*)(kaddr + ((vcn << - idx_ni->itype.index.vcn_size_bits) & ~PAGE_MASK)); - /* Bounds checks. */ - if ((u8*)ia < kaddr || (u8*)ia > kaddr + PAGE_SIZE) { - ntfs_error(sb, "Out of bounds check failed. Corrupt inode " - "0x%lx or driver bug.", idx_ni->mft_no); - goto unm_err_out; - } - /* Catch multi sector transfer fixup errors. */ - if (unlikely(!ntfs_is_indx_record(ia->magic))) { - ntfs_error(sb, "Index record with vcn 0x%llx is corrupt. " - "Corrupt inode 0x%lx. Run chkdsk.", - (long long)vcn, idx_ni->mft_no); - goto unm_err_out; - } - if (sle64_to_cpu(ia->index_block_vcn) != vcn) { - ntfs_error(sb, "Actual VCN (0x%llx) of index buffer is " - "different from expected VCN (0x%llx). Inode " - "0x%lx is corrupt or driver bug.", - (unsigned long long) - sle64_to_cpu(ia->index_block_vcn), - (unsigned long long)vcn, idx_ni->mft_no); - goto unm_err_out; - } - if (le32_to_cpu(ia->index.allocated_size) + 0x18 != - idx_ni->itype.index.block_size) { - ntfs_error(sb, "Index buffer (VCN 0x%llx) of inode 0x%lx has " - "a size (%u) differing from the index " - "specified size (%u). Inode is corrupt or " - "driver bug.", (unsigned long long)vcn, - idx_ni->mft_no, - le32_to_cpu(ia->index.allocated_size) + 0x18, - idx_ni->itype.index.block_size); - goto unm_err_out; - } - index_end = (u8*)ia + idx_ni->itype.index.block_size; - if (index_end > kaddr + PAGE_SIZE) { - ntfs_error(sb, "Index buffer (VCN 0x%llx) of inode 0x%lx " - "crosses page boundary. Impossible! Cannot " - "access! This is probably a bug in the " - "driver.", (unsigned long long)vcn, - idx_ni->mft_no); - goto unm_err_out; - } - index_end = (u8*)&ia->index + le32_to_cpu(ia->index.index_length); - if (index_end > (u8*)ia + idx_ni->itype.index.block_size) { - ntfs_error(sb, "Size of index buffer (VCN 0x%llx) of inode " - "0x%lx exceeds maximum size.", - (unsigned long long)vcn, idx_ni->mft_no); - goto unm_err_out; - } - /* The first index entry. */ - ie = (INDEX_ENTRY*)((u8*)&ia->index + - le32_to_cpu(ia->index.entries_offset)); - /* - * Iterate similar to above big loop but applied to index buffer, thus - * loop until we exceed valid memory (corruption case) or until we - * reach the last entry. - */ - for (;; ie = (INDEX_ENTRY*)((u8*)ie + le16_to_cpu(ie->length))) { - /* Bounds checks. */ - if ((u8*)ie < (u8*)ia || (u8*)ie + - sizeof(INDEX_ENTRY_HEADER) > index_end || - (u8*)ie + le16_to_cpu(ie->length) > index_end) { - ntfs_error(sb, "Index entry out of bounds in inode " - "0x%lx.", idx_ni->mft_no); - goto unm_err_out; - } - /* - * The last entry cannot contain a key. It can however contain - * a pointer to a child node in the B+tree so we just break out. - */ - if (ie->flags & INDEX_ENTRY_END) - break; - /* Further bounds checks. */ - if ((u32)sizeof(INDEX_ENTRY_HEADER) + - le16_to_cpu(ie->key_length) > - le16_to_cpu(ie->data.vi.data_offset) || - (u32)le16_to_cpu(ie->data.vi.data_offset) + - le16_to_cpu(ie->data.vi.data_length) > - le16_to_cpu(ie->length)) { - ntfs_error(sb, "Index entry out of bounds in inode " - "0x%lx.", idx_ni->mft_no); - goto unm_err_out; - } - /* If the keys match perfectly, we setup @ictx and return 0. */ - if ((key_len == le16_to_cpu(ie->key_length)) && !memcmp(key, - &ie->key, key_len)) { -ia_done: - ictx->is_in_root = false; - ictx->actx = NULL; - ictx->base_ni = NULL; - ictx->ia = ia; - ictx->page = page; - goto done; - } - /* - * Not a perfect match, need to do full blown collation so we - * know which way in the B+tree we have to go. - */ - rc = ntfs_collate(vol, idx_ni->itype.index.collation_rule, key, - key_len, &ie->key, le16_to_cpu(ie->key_length)); - /* - * If @key collates before the key of the current entry, there - * is definitely no such key in this index but we might need to - * descend into the B+tree so we just break out of the loop. - */ - if (rc == -1) - break; - /* - * A match should never happen as the memcmp() call should have - * cought it, but we still treat it correctly. - */ - if (!rc) - goto ia_done; - /* The keys are not equal, continue the search. */ - } - /* - * We have finished with this index buffer without success. Check for - * the presence of a child node and if not present return -ENOENT. - */ - if (!(ie->flags & INDEX_ENTRY_NODE)) { - ntfs_debug("Entry not found."); - err = -ENOENT; - goto ia_done; - } - if ((ia->index.flags & NODE_MASK) == LEAF_NODE) { - ntfs_error(sb, "Index entry with child node found in a leaf " - "node in inode 0x%lx.", idx_ni->mft_no); - goto unm_err_out; - } - /* Child node present, descend into it. */ - old_vcn = vcn; - vcn = sle64_to_cpup((sle64*)((u8*)ie + le16_to_cpu(ie->length) - 8)); - if (vcn >= 0) { - /* - * If vcn is in the same page cache page as old_vcn we recycle - * the mapped page. - */ - if (old_vcn << vol->cluster_size_bits >> - PAGE_SHIFT == vcn << - vol->cluster_size_bits >> - PAGE_SHIFT) - goto fast_descend_into_child_node; - unlock_page(page); - ntfs_unmap_page(page); - goto descend_into_child_node; - } - ntfs_error(sb, "Negative child node vcn in inode 0x%lx.", - idx_ni->mft_no); -unm_err_out: - unlock_page(page); - ntfs_unmap_page(page); -err_out: - if (!err) - err = -EIO; - if (actx) - ntfs_attr_put_search_ctx(actx); - if (m) - unmap_mft_record(base_ni); - return err; -idx_err_out: - ntfs_error(sb, "Corrupt index. Aborting lookup."); - goto err_out; -} diff --git a/fs/ntfs/index.h b/fs/ntfs/index.h deleted file mode 100644 index bb3c3ae55138..000000000000 --- a/fs/ntfs/index.h +++ /dev/null @@ -1,134 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * index.h - Defines for NTFS kernel index handling. Part of the Linux-NTFS - * project. - * - * Copyright (c) 2004 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_INDEX_H -#define _LINUX_NTFS_INDEX_H - -#include - -#include "types.h" -#include "layout.h" -#include "inode.h" -#include "attrib.h" -#include "mft.h" -#include "aops.h" - -/** - * @idx_ni: index inode containing the @entry described by this context - * @entry: index entry (points into @ir or @ia) - * @data: index entry data (points into @entry) - * @data_len: length in bytes of @data - * @is_in_root: 'true' if @entry is in @ir and 'false' if it is in @ia - * @ir: index root if @is_in_root and NULL otherwise - * @actx: attribute search context if @is_in_root and NULL otherwise - * @base_ni: base inode if @is_in_root and NULL otherwise - * @ia: index block if @is_in_root is 'false' and NULL otherwise - * @page: page if @is_in_root is 'false' and NULL otherwise - * - * @idx_ni is the index inode this context belongs to. - * - * @entry is the index entry described by this context. @data and @data_len - * are the index entry data and its length in bytes, respectively. @data - * simply points into @entry. This is probably what the user is interested in. - * - * If @is_in_root is 'true', @entry is in the index root attribute @ir described - * by the attribute search context @actx and the base inode @base_ni. @ia and - * @page are NULL in this case. - * - * If @is_in_root is 'false', @entry is in the index allocation attribute and @ia - * and @page point to the index allocation block and the mapped, locked page it - * is in, respectively. @ir, @actx and @base_ni are NULL in this case. - * - * To obtain a context call ntfs_index_ctx_get(). - * - * We use this context to allow ntfs_index_lookup() to return the found index - * @entry and its @data without having to allocate a buffer and copy the @entry - * and/or its @data into it. - * - * When finished with the @entry and its @data, call ntfs_index_ctx_put() to - * free the context and other associated resources. - * - * If the index entry was modified, call flush_dcache_index_entry_page() - * immediately after the modification and either ntfs_index_entry_mark_dirty() - * or ntfs_index_entry_write() before the call to ntfs_index_ctx_put() to - * ensure that the changes are written to disk. - */ -typedef struct { - ntfs_inode *idx_ni; - INDEX_ENTRY *entry; - void *data; - u16 data_len; - bool is_in_root; - INDEX_ROOT *ir; - ntfs_attr_search_ctx *actx; - ntfs_inode *base_ni; - INDEX_ALLOCATION *ia; - struct page *page; -} ntfs_index_context; - -extern ntfs_index_context *ntfs_index_ctx_get(ntfs_inode *idx_ni); -extern void ntfs_index_ctx_put(ntfs_index_context *ictx); - -extern int ntfs_index_lookup(const void *key, const int key_len, - ntfs_index_context *ictx); - -#ifdef NTFS_RW - -/** - * ntfs_index_entry_flush_dcache_page - flush_dcache_page() for index entries - * @ictx: ntfs index context describing the index entry - * - * Call flush_dcache_page() for the page in which an index entry resides. - * - * This must be called every time an index entry is modified, just after the - * modification. - * - * If the index entry is in the index root attribute, simply flush the page - * containing the mft record containing the index root attribute. - * - * If the index entry is in an index block belonging to the index allocation - * attribute, simply flush the page cache page containing the index block. - */ -static inline void ntfs_index_entry_flush_dcache_page(ntfs_index_context *ictx) -{ - if (ictx->is_in_root) - flush_dcache_mft_record_page(ictx->actx->ntfs_ino); - else - flush_dcache_page(ictx->page); -} - -/** - * ntfs_index_entry_mark_dirty - mark an index entry dirty - * @ictx: ntfs index context describing the index entry - * - * Mark the index entry described by the index entry context @ictx dirty. - * - * If the index entry is in the index root attribute, simply mark the mft - * record containing the index root attribute dirty. This ensures the mft - * record, and hence the index root attribute, will be written out to disk - * later. - * - * If the index entry is in an index block belonging to the index allocation - * attribute, mark the buffers belonging to the index record as well as the - * page cache page the index block is in dirty. This automatically marks the - * VFS inode of the ntfs index inode to which the index entry belongs dirty, - * too (I_DIRTY_PAGES) and this in turn ensures the page buffers, and hence the - * dirty index block, will be written out to disk later. - */ -static inline void ntfs_index_entry_mark_dirty(ntfs_index_context *ictx) -{ - if (ictx->is_in_root) - mark_mft_record_dirty(ictx->actx->ntfs_ino); - else - mark_ntfs_record_dirty(ictx->page, - (u8*)ictx->ia - (u8*)page_address(ictx->page)); -} - -#endif /* NTFS_RW */ - -#endif /* _LINUX_NTFS_INDEX_H */ diff --git a/fs/ntfs/inode.c b/fs/ntfs/inode.c deleted file mode 100644 index 99ac6ea277c4..000000000000 --- a/fs/ntfs/inode.c +++ /dev/null @@ -1,3101 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * inode.c - NTFS kernel inode handling. - * - * Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc. - */ - -#include -#include -#include -#include -#include -#include -#include -#include -#include - -#include "aops.h" -#include "attrib.h" -#include "bitmap.h" -#include "dir.h" -#include "debug.h" -#include "inode.h" -#include "lcnalloc.h" -#include "malloc.h" -#include "mft.h" -#include "time.h" -#include "ntfs.h" - -/** - * ntfs_test_inode - compare two (possibly fake) inodes for equality - * @vi: vfs inode which to test - * @data: data which is being tested with - * - * Compare the ntfs attribute embedded in the ntfs specific part of the vfs - * inode @vi for equality with the ntfs attribute @data. - * - * If searching for the normal file/directory inode, set @na->type to AT_UNUSED. - * @na->name and @na->name_len are then ignored. - * - * Return 1 if the attributes match and 0 if not. - * - * NOTE: This function runs with the inode_hash_lock spin lock held so it is not - * allowed to sleep. - */ -int ntfs_test_inode(struct inode *vi, void *data) -{ - ntfs_attr *na = (ntfs_attr *)data; - ntfs_inode *ni; - - if (vi->i_ino != na->mft_no) - return 0; - ni = NTFS_I(vi); - /* If !NInoAttr(ni), @vi is a normal file or directory inode. */ - if (likely(!NInoAttr(ni))) { - /* If not looking for a normal inode this is a mismatch. */ - if (unlikely(na->type != AT_UNUSED)) - return 0; - } else { - /* A fake inode describing an attribute. */ - if (ni->type != na->type) - return 0; - if (ni->name_len != na->name_len) - return 0; - if (na->name_len && memcmp(ni->name, na->name, - na->name_len * sizeof(ntfschar))) - return 0; - } - /* Match! */ - return 1; -} - -/** - * ntfs_init_locked_inode - initialize an inode - * @vi: vfs inode to initialize - * @data: data which to initialize @vi to - * - * Initialize the vfs inode @vi with the values from the ntfs attribute @data in - * order to enable ntfs_test_inode() to do its work. - * - * If initializing the normal file/directory inode, set @na->type to AT_UNUSED. - * In that case, @na->name and @na->name_len should be set to NULL and 0, - * respectively. Although that is not strictly necessary as - * ntfs_read_locked_inode() will fill them in later. - * - * Return 0 on success and -errno on error. - * - * NOTE: This function runs with the inode->i_lock spin lock held so it is not - * allowed to sleep. (Hence the GFP_ATOMIC allocation.) - */ -static int ntfs_init_locked_inode(struct inode *vi, void *data) -{ - ntfs_attr *na = (ntfs_attr *)data; - ntfs_inode *ni = NTFS_I(vi); - - vi->i_ino = na->mft_no; - - ni->type = na->type; - if (na->type == AT_INDEX_ALLOCATION) - NInoSetMstProtected(ni); - - ni->name = na->name; - ni->name_len = na->name_len; - - /* If initializing a normal inode, we are done. */ - if (likely(na->type == AT_UNUSED)) { - BUG_ON(na->name); - BUG_ON(na->name_len); - return 0; - } - - /* It is a fake inode. */ - NInoSetAttr(ni); - - /* - * We have I30 global constant as an optimization as it is the name - * in >99.9% of named attributes! The other <0.1% incur a GFP_ATOMIC - * allocation but that is ok. And most attributes are unnamed anyway, - * thus the fraction of named attributes with name != I30 is actually - * absolutely tiny. - */ - if (na->name_len && na->name != I30) { - unsigned int i; - - BUG_ON(!na->name); - i = na->name_len * sizeof(ntfschar); - ni->name = kmalloc(i + sizeof(ntfschar), GFP_ATOMIC); - if (!ni->name) - return -ENOMEM; - memcpy(ni->name, na->name, i); - ni->name[na->name_len] = 0; - } - return 0; -} - -static int ntfs_read_locked_inode(struct inode *vi); -static int ntfs_read_locked_attr_inode(struct inode *base_vi, struct inode *vi); -static int ntfs_read_locked_index_inode(struct inode *base_vi, - struct inode *vi); - -/** - * ntfs_iget - obtain a struct inode corresponding to a specific normal inode - * @sb: super block of mounted volume - * @mft_no: mft record number / inode number to obtain - * - * Obtain the struct inode corresponding to a specific normal inode (i.e. a - * file or directory). - * - * If the inode is in the cache, it is just returned with an increased - * reference count. Otherwise, a new struct inode is allocated and initialized, - * and finally ntfs_read_locked_inode() is called to read in the inode and - * fill in the remainder of the inode structure. - * - * Return the struct inode on success. Check the return value with IS_ERR() and - * if true, the function failed and the error code is obtained from PTR_ERR(). - */ -struct inode *ntfs_iget(struct super_block *sb, unsigned long mft_no) -{ - struct inode *vi; - int err; - ntfs_attr na; - - na.mft_no = mft_no; - na.type = AT_UNUSED; - na.name = NULL; - na.name_len = 0; - - vi = iget5_locked(sb, mft_no, ntfs_test_inode, - ntfs_init_locked_inode, &na); - if (unlikely(!vi)) - return ERR_PTR(-ENOMEM); - - err = 0; - - /* If this is a freshly allocated inode, need to read it now. */ - if (vi->i_state & I_NEW) { - err = ntfs_read_locked_inode(vi); - unlock_new_inode(vi); - } - /* - * There is no point in keeping bad inodes around if the failure was - * due to ENOMEM. We want to be able to retry again later. - */ - if (unlikely(err == -ENOMEM)) { - iput(vi); - vi = ERR_PTR(err); - } - return vi; -} - -/** - * ntfs_attr_iget - obtain a struct inode corresponding to an attribute - * @base_vi: vfs base inode containing the attribute - * @type: attribute type - * @name: Unicode name of the attribute (NULL if unnamed) - * @name_len: length of @name in Unicode characters (0 if unnamed) - * - * Obtain the (fake) struct inode corresponding to the attribute specified by - * @type, @name, and @name_len, which is present in the base mft record - * specified by the vfs inode @base_vi. - * - * If the attribute inode is in the cache, it is just returned with an - * increased reference count. Otherwise, a new struct inode is allocated and - * initialized, and finally ntfs_read_locked_attr_inode() is called to read the - * attribute and fill in the inode structure. - * - * Note, for index allocation attributes, you need to use ntfs_index_iget() - * instead of ntfs_attr_iget() as working with indices is a lot more complex. - * - * Return the struct inode of the attribute inode on success. Check the return - * value with IS_ERR() and if true, the function failed and the error code is - * obtained from PTR_ERR(). - */ -struct inode *ntfs_attr_iget(struct inode *base_vi, ATTR_TYPE type, - ntfschar *name, u32 name_len) -{ - struct inode *vi; - int err; - ntfs_attr na; - - /* Make sure no one calls ntfs_attr_iget() for indices. */ - BUG_ON(type == AT_INDEX_ALLOCATION); - - na.mft_no = base_vi->i_ino; - na.type = type; - na.name = name; - na.name_len = name_len; - - vi = iget5_locked(base_vi->i_sb, na.mft_no, ntfs_test_inode, - ntfs_init_locked_inode, &na); - if (unlikely(!vi)) - return ERR_PTR(-ENOMEM); - - err = 0; - - /* If this is a freshly allocated inode, need to read it now. */ - if (vi->i_state & I_NEW) { - err = ntfs_read_locked_attr_inode(base_vi, vi); - unlock_new_inode(vi); - } - /* - * There is no point in keeping bad attribute inodes around. This also - * simplifies things in that we never need to check for bad attribute - * inodes elsewhere. - */ - if (unlikely(err)) { - iput(vi); - vi = ERR_PTR(err); - } - return vi; -} - -/** - * ntfs_index_iget - obtain a struct inode corresponding to an index - * @base_vi: vfs base inode containing the index related attributes - * @name: Unicode name of the index - * @name_len: length of @name in Unicode characters - * - * Obtain the (fake) struct inode corresponding to the index specified by @name - * and @name_len, which is present in the base mft record specified by the vfs - * inode @base_vi. - * - * If the index inode is in the cache, it is just returned with an increased - * reference count. Otherwise, a new struct inode is allocated and - * initialized, and finally ntfs_read_locked_index_inode() is called to read - * the index related attributes and fill in the inode structure. - * - * Return the struct inode of the index inode on success. Check the return - * value with IS_ERR() and if true, the function failed and the error code is - * obtained from PTR_ERR(). - */ -struct inode *ntfs_index_iget(struct inode *base_vi, ntfschar *name, - u32 name_len) -{ - struct inode *vi; - int err; - ntfs_attr na; - - na.mft_no = base_vi->i_ino; - na.type = AT_INDEX_ALLOCATION; - na.name = name; - na.name_len = name_len; - - vi = iget5_locked(base_vi->i_sb, na.mft_no, ntfs_test_inode, - ntfs_init_locked_inode, &na); - if (unlikely(!vi)) - return ERR_PTR(-ENOMEM); - - err = 0; - - /* If this is a freshly allocated inode, need to read it now. */ - if (vi->i_state & I_NEW) { - err = ntfs_read_locked_index_inode(base_vi, vi); - unlock_new_inode(vi); - } - /* - * There is no point in keeping bad index inodes around. This also - * simplifies things in that we never need to check for bad index - * inodes elsewhere. - */ - if (unlikely(err)) { - iput(vi); - vi = ERR_PTR(err); - } - return vi; -} - -struct inode *ntfs_alloc_big_inode(struct super_block *sb) -{ - ntfs_inode *ni; - - ntfs_debug("Entering."); - ni = alloc_inode_sb(sb, ntfs_big_inode_cache, GFP_NOFS); - if (likely(ni != NULL)) { - ni->state = 0; - return VFS_I(ni); - } - ntfs_error(sb, "Allocation of NTFS big inode structure failed."); - return NULL; -} - -void ntfs_free_big_inode(struct inode *inode) -{ - kmem_cache_free(ntfs_big_inode_cache, NTFS_I(inode)); -} - -static inline ntfs_inode *ntfs_alloc_extent_inode(void) -{ - ntfs_inode *ni; - - ntfs_debug("Entering."); - ni = kmem_cache_alloc(ntfs_inode_cache, GFP_NOFS); - if (likely(ni != NULL)) { - ni->state = 0; - return ni; - } - ntfs_error(NULL, "Allocation of NTFS inode structure failed."); - return NULL; -} - -static void ntfs_destroy_extent_inode(ntfs_inode *ni) -{ - ntfs_debug("Entering."); - BUG_ON(ni->page); - if (!atomic_dec_and_test(&ni->count)) - BUG(); - kmem_cache_free(ntfs_inode_cache, ni); -} - -/* - * The attribute runlist lock has separate locking rules from the - * normal runlist lock, so split the two lock-classes: - */ -static struct lock_class_key attr_list_rl_lock_class; - -/** - * __ntfs_init_inode - initialize ntfs specific part of an inode - * @sb: super block of mounted volume - * @ni: freshly allocated ntfs inode which to initialize - * - * Initialize an ntfs inode to defaults. - * - * NOTE: ni->mft_no, ni->state, ni->type, ni->name, and ni->name_len are left - * untouched. Make sure to initialize them elsewhere. - * - * Return zero on success and -ENOMEM on error. - */ -void __ntfs_init_inode(struct super_block *sb, ntfs_inode *ni) -{ - ntfs_debug("Entering."); - rwlock_init(&ni->size_lock); - ni->initialized_size = ni->allocated_size = 0; - ni->seq_no = 0; - atomic_set(&ni->count, 1); - ni->vol = NTFS_SB(sb); - ntfs_init_runlist(&ni->runlist); - mutex_init(&ni->mrec_lock); - ni->page = NULL; - ni->page_ofs = 0; - ni->attr_list_size = 0; - ni->attr_list = NULL; - ntfs_init_runlist(&ni->attr_list_rl); - lockdep_set_class(&ni->attr_list_rl.lock, - &attr_list_rl_lock_class); - ni->itype.index.block_size = 0; - ni->itype.index.vcn_size = 0; - ni->itype.index.collation_rule = 0; - ni->itype.index.block_size_bits = 0; - ni->itype.index.vcn_size_bits = 0; - mutex_init(&ni->extent_lock); - ni->nr_extents = 0; - ni->ext.base_ntfs_ino = NULL; -} - -/* - * Extent inodes get MFT-mapped in a nested way, while the base inode - * is still mapped. Teach this nesting to the lock validator by creating - * a separate class for nested inode's mrec_lock's: - */ -static struct lock_class_key extent_inode_mrec_lock_key; - -inline ntfs_inode *ntfs_new_extent_inode(struct super_block *sb, - unsigned long mft_no) -{ - ntfs_inode *ni = ntfs_alloc_extent_inode(); - - ntfs_debug("Entering."); - if (likely(ni != NULL)) { - __ntfs_init_inode(sb, ni); - lockdep_set_class(&ni->mrec_lock, &extent_inode_mrec_lock_key); - ni->mft_no = mft_no; - ni->type = AT_UNUSED; - ni->name = NULL; - ni->name_len = 0; - } - return ni; -} - -/** - * ntfs_is_extended_system_file - check if a file is in the $Extend directory - * @ctx: initialized attribute search context - * - * Search all file name attributes in the inode described by the attribute - * search context @ctx and check if any of the names are in the $Extend system - * directory. - * - * Return values: - * 1: file is in $Extend directory - * 0: file is not in $Extend directory - * -errno: failed to determine if the file is in the $Extend directory - */ -static int ntfs_is_extended_system_file(ntfs_attr_search_ctx *ctx) -{ - int nr_links, err; - - /* Restart search. */ - ntfs_attr_reinit_search_ctx(ctx); - - /* Get number of hard links. */ - nr_links = le16_to_cpu(ctx->mrec->link_count); - - /* Loop through all hard links. */ - while (!(err = ntfs_attr_lookup(AT_FILE_NAME, NULL, 0, 0, 0, NULL, 0, - ctx))) { - FILE_NAME_ATTR *file_name_attr; - ATTR_RECORD *attr = ctx->attr; - u8 *p, *p2; - - nr_links--; - /* - * Maximum sanity checking as we are called on an inode that - * we suspect might be corrupt. - */ - p = (u8*)attr + le32_to_cpu(attr->length); - if (p < (u8*)ctx->mrec || (u8*)p > (u8*)ctx->mrec + - le32_to_cpu(ctx->mrec->bytes_in_use)) { -err_corrupt_attr: - ntfs_error(ctx->ntfs_ino->vol->sb, "Corrupt file name " - "attribute. You should run chkdsk."); - return -EIO; - } - if (attr->non_resident) { - ntfs_error(ctx->ntfs_ino->vol->sb, "Non-resident file " - "name. You should run chkdsk."); - return -EIO; - } - if (attr->flags) { - ntfs_error(ctx->ntfs_ino->vol->sb, "File name with " - "invalid flags. You should run " - "chkdsk."); - return -EIO; - } - if (!(attr->data.resident.flags & RESIDENT_ATTR_IS_INDEXED)) { - ntfs_error(ctx->ntfs_ino->vol->sb, "Unindexed file " - "name. You should run chkdsk."); - return -EIO; - } - file_name_attr = (FILE_NAME_ATTR*)((u8*)attr + - le16_to_cpu(attr->data.resident.value_offset)); - p2 = (u8 *)file_name_attr + le32_to_cpu(attr->data.resident.value_length); - if (p2 < (u8*)attr || p2 > p) - goto err_corrupt_attr; - /* This attribute is ok, but is it in the $Extend directory? */ - if (MREF_LE(file_name_attr->parent_directory) == FILE_Extend) - return 1; /* YES, it's an extended system file. */ - } - if (unlikely(err != -ENOENT)) - return err; - if (unlikely(nr_links)) { - ntfs_error(ctx->ntfs_ino->vol->sb, "Inode hard link count " - "doesn't match number of name attributes. You " - "should run chkdsk."); - return -EIO; - } - return 0; /* NO, it is not an extended system file. */ -} - -/** - * ntfs_read_locked_inode - read an inode from its device - * @vi: inode to read - * - * ntfs_read_locked_inode() is called from ntfs_iget() to read the inode - * described by @vi into memory from the device. - * - * The only fields in @vi that we need to/can look at when the function is - * called are i_sb, pointing to the mounted device's super block, and i_ino, - * the number of the inode to load. - * - * ntfs_read_locked_inode() maps, pins and locks the mft record number i_ino - * for reading and sets up the necessary @vi fields as well as initializing - * the ntfs inode. - * - * Q: What locks are held when the function is called? - * A: i_state has I_NEW set, hence the inode is locked, also - * i_count is set to 1, so it is not going to go away - * i_flags is set to 0 and we have no business touching it. Only an ioctl() - * is allowed to write to them. We should of course be honouring them but - * we need to do that using the IS_* macros defined in include/linux/fs.h. - * In any case ntfs_read_locked_inode() has nothing to do with i_flags. - * - * Return 0 on success and -errno on error. In the error case, the inode will - * have had make_bad_inode() executed on it. - */ -static int ntfs_read_locked_inode(struct inode *vi) -{ - ntfs_volume *vol = NTFS_SB(vi->i_sb); - ntfs_inode *ni; - struct inode *bvi; - MFT_RECORD *m; - ATTR_RECORD *a; - STANDARD_INFORMATION *si; - ntfs_attr_search_ctx *ctx; - int err = 0; - - ntfs_debug("Entering for i_ino 0x%lx.", vi->i_ino); - - /* Setup the generic vfs inode parts now. */ - vi->i_uid = vol->uid; - vi->i_gid = vol->gid; - vi->i_mode = 0; - - /* - * Initialize the ntfs specific part of @vi special casing - * FILE_MFT which we need to do at mount time. - */ - if (vi->i_ino != FILE_MFT) - ntfs_init_big_inode(vi); - ni = NTFS_I(vi); - - m = map_mft_record(ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(ni, m); - if (!ctx) { - err = -ENOMEM; - goto unm_err_out; - } - - if (!(m->flags & MFT_RECORD_IN_USE)) { - ntfs_error(vi->i_sb, "Inode is not in use!"); - goto unm_err_out; - } - if (m->base_mft_record) { - ntfs_error(vi->i_sb, "Inode is an extent inode!"); - goto unm_err_out; - } - - /* Transfer information from mft record into vfs and ntfs inodes. */ - vi->i_generation = ni->seq_no = le16_to_cpu(m->sequence_number); - - /* - * FIXME: Keep in mind that link_count is two for files which have both - * a long file name and a short file name as separate entries, so if - * we are hiding short file names this will be too high. Either we need - * to account for the short file names by subtracting them or we need - * to make sure we delete files even though i_nlink is not zero which - * might be tricky due to vfs interactions. Need to think about this - * some more when implementing the unlink command. - */ - set_nlink(vi, le16_to_cpu(m->link_count)); - /* - * FIXME: Reparse points can have the directory bit set even though - * they would be S_IFLNK. Need to deal with this further below when we - * implement reparse points / symbolic links but it will do for now. - * Also if not a directory, it could be something else, rather than - * a regular file. But again, will do for now. - */ - /* Everyone gets all permissions. */ - vi->i_mode |= S_IRWXUGO; - /* If read-only, no one gets write permissions. */ - if (IS_RDONLY(vi)) - vi->i_mode &= ~S_IWUGO; - if (m->flags & MFT_RECORD_IS_DIRECTORY) { - vi->i_mode |= S_IFDIR; - /* - * Apply the directory permissions mask set in the mount - * options. - */ - vi->i_mode &= ~vol->dmask; - /* Things break without this kludge! */ - if (vi->i_nlink > 1) - set_nlink(vi, 1); - } else { - vi->i_mode |= S_IFREG; - /* Apply the file permissions mask set in the mount options. */ - vi->i_mode &= ~vol->fmask; - } - /* - * Find the standard information attribute in the mft record. At this - * stage we haven't setup the attribute list stuff yet, so this could - * in fact fail if the standard information is in an extent record, but - * I don't think this actually ever happens. - */ - err = ntfs_attr_lookup(AT_STANDARD_INFORMATION, NULL, 0, 0, 0, NULL, 0, - ctx); - if (unlikely(err)) { - if (err == -ENOENT) { - /* - * TODO: We should be performing a hot fix here (if the - * recover mount option is set) by creating a new - * attribute. - */ - ntfs_error(vi->i_sb, "$STANDARD_INFORMATION attribute " - "is missing."); - } - goto unm_err_out; - } - a = ctx->attr; - /* Get the standard information attribute value. */ - if ((u8 *)a + le16_to_cpu(a->data.resident.value_offset) - + le32_to_cpu(a->data.resident.value_length) > - (u8 *)ctx->mrec + vol->mft_record_size) { - ntfs_error(vi->i_sb, "Corrupt standard information attribute in inode."); - goto unm_err_out; - } - si = (STANDARD_INFORMATION*)((u8*)a + - le16_to_cpu(a->data.resident.value_offset)); - - /* Transfer information from the standard information into vi. */ - /* - * Note: The i_?times do not quite map perfectly onto the NTFS times, - * but they are close enough, and in the end it doesn't really matter - * that much... - */ - /* - * mtime is the last change of the data within the file. Not changed - * when only metadata is changed, e.g. a rename doesn't affect mtime. - */ - vi->i_mtime = ntfs2utc(si->last_data_change_time); - /* - * ctime is the last change of the metadata of the file. This obviously - * always changes, when mtime is changed. ctime can be changed on its - * own, mtime is then not changed, e.g. when a file is renamed. - */ - inode_set_ctime_to_ts(vi, ntfs2utc(si->last_mft_change_time)); - /* - * Last access to the data within the file. Not changed during a rename - * for example but changed whenever the file is written to. - */ - vi->i_atime = ntfs2utc(si->last_access_time); - - /* Find the attribute list attribute if present. */ - ntfs_attr_reinit_search_ctx(ctx); - err = ntfs_attr_lookup(AT_ATTRIBUTE_LIST, NULL, 0, 0, 0, NULL, 0, ctx); - if (err) { - if (unlikely(err != -ENOENT)) { - ntfs_error(vi->i_sb, "Failed to lookup attribute list " - "attribute."); - goto unm_err_out; - } - } else /* if (!err) */ { - if (vi->i_ino == FILE_MFT) - goto skip_attr_list_load; - ntfs_debug("Attribute list found in inode 0x%lx.", vi->i_ino); - NInoSetAttrList(ni); - a = ctx->attr; - if (a->flags & ATTR_COMPRESSION_MASK) { - ntfs_error(vi->i_sb, "Attribute list attribute is " - "compressed."); - goto unm_err_out; - } - if (a->flags & ATTR_IS_ENCRYPTED || - a->flags & ATTR_IS_SPARSE) { - if (a->non_resident) { - ntfs_error(vi->i_sb, "Non-resident attribute " - "list attribute is encrypted/" - "sparse."); - goto unm_err_out; - } - ntfs_warning(vi->i_sb, "Resident attribute list " - "attribute in inode 0x%lx is marked " - "encrypted/sparse which is not true. " - "However, Windows allows this and " - "chkdsk does not detect or correct it " - "so we will just ignore the invalid " - "flags and pretend they are not set.", - vi->i_ino); - } - /* Now allocate memory for the attribute list. */ - ni->attr_list_size = (u32)ntfs_attr_size(a); - ni->attr_list = ntfs_malloc_nofs(ni->attr_list_size); - if (!ni->attr_list) { - ntfs_error(vi->i_sb, "Not enough memory to allocate " - "buffer for attribute list."); - err = -ENOMEM; - goto unm_err_out; - } - if (a->non_resident) { - NInoSetAttrListNonResident(ni); - if (a->data.non_resident.lowest_vcn) { - ntfs_error(vi->i_sb, "Attribute list has non " - "zero lowest_vcn."); - goto unm_err_out; - } - /* - * Setup the runlist. No need for locking as we have - * exclusive access to the inode at this time. - */ - ni->attr_list_rl.rl = ntfs_mapping_pairs_decompress(vol, - a, NULL); - if (IS_ERR(ni->attr_list_rl.rl)) { - err = PTR_ERR(ni->attr_list_rl.rl); - ni->attr_list_rl.rl = NULL; - ntfs_error(vi->i_sb, "Mapping pairs " - "decompression failed."); - goto unm_err_out; - } - /* Now load the attribute list. */ - if ((err = load_attribute_list(vol, &ni->attr_list_rl, - ni->attr_list, ni->attr_list_size, - sle64_to_cpu(a->data.non_resident. - initialized_size)))) { - ntfs_error(vi->i_sb, "Failed to load " - "attribute list attribute."); - goto unm_err_out; - } - } else /* if (!a->non_resident) */ { - if ((u8*)a + le16_to_cpu(a->data.resident.value_offset) - + le32_to_cpu( - a->data.resident.value_length) > - (u8*)ctx->mrec + vol->mft_record_size) { - ntfs_error(vi->i_sb, "Corrupt attribute list " - "in inode."); - goto unm_err_out; - } - /* Now copy the attribute list. */ - memcpy(ni->attr_list, (u8*)a + le16_to_cpu( - a->data.resident.value_offset), - le32_to_cpu( - a->data.resident.value_length)); - } - } -skip_attr_list_load: - /* - * If an attribute list is present we now have the attribute list value - * in ntfs_ino->attr_list and it is ntfs_ino->attr_list_size bytes. - */ - if (S_ISDIR(vi->i_mode)) { - loff_t bvi_size; - ntfs_inode *bni; - INDEX_ROOT *ir; - u8 *ir_end, *index_end; - - /* It is a directory, find index root attribute. */ - ntfs_attr_reinit_search_ctx(ctx); - err = ntfs_attr_lookup(AT_INDEX_ROOT, I30, 4, CASE_SENSITIVE, - 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) { - // FIXME: File is corrupt! Hot-fix with empty - // index root attribute if recovery option is - // set. - ntfs_error(vi->i_sb, "$INDEX_ROOT attribute " - "is missing."); - } - goto unm_err_out; - } - a = ctx->attr; - /* Set up the state. */ - if (unlikely(a->non_resident)) { - ntfs_error(vol->sb, "$INDEX_ROOT attribute is not " - "resident."); - goto unm_err_out; - } - /* Ensure the attribute name is placed before the value. */ - if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >= - le16_to_cpu(a->data.resident.value_offset)))) { - ntfs_error(vol->sb, "$INDEX_ROOT attribute name is " - "placed after the attribute value."); - goto unm_err_out; - } - /* - * Compressed/encrypted index root just means that the newly - * created files in that directory should be created compressed/ - * encrypted. However index root cannot be both compressed and - * encrypted. - */ - if (a->flags & ATTR_COMPRESSION_MASK) - NInoSetCompressed(ni); - if (a->flags & ATTR_IS_ENCRYPTED) { - if (a->flags & ATTR_COMPRESSION_MASK) { - ntfs_error(vi->i_sb, "Found encrypted and " - "compressed attribute."); - goto unm_err_out; - } - NInoSetEncrypted(ni); - } - if (a->flags & ATTR_IS_SPARSE) - NInoSetSparse(ni); - ir = (INDEX_ROOT*)((u8*)a + - le16_to_cpu(a->data.resident.value_offset)); - ir_end = (u8*)ir + le32_to_cpu(a->data.resident.value_length); - if (ir_end > (u8*)ctx->mrec + vol->mft_record_size) { - ntfs_error(vi->i_sb, "$INDEX_ROOT attribute is " - "corrupt."); - goto unm_err_out; - } - index_end = (u8*)&ir->index + - le32_to_cpu(ir->index.index_length); - if (index_end > ir_end) { - ntfs_error(vi->i_sb, "Directory index is corrupt."); - goto unm_err_out; - } - if (ir->type != AT_FILE_NAME) { - ntfs_error(vi->i_sb, "Indexed attribute is not " - "$FILE_NAME."); - goto unm_err_out; - } - if (ir->collation_rule != COLLATION_FILE_NAME) { - ntfs_error(vi->i_sb, "Index collation rule is not " - "COLLATION_FILE_NAME."); - goto unm_err_out; - } - ni->itype.index.collation_rule = ir->collation_rule; - ni->itype.index.block_size = le32_to_cpu(ir->index_block_size); - if (ni->itype.index.block_size & - (ni->itype.index.block_size - 1)) { - ntfs_error(vi->i_sb, "Index block size (%u) is not a " - "power of two.", - ni->itype.index.block_size); - goto unm_err_out; - } - if (ni->itype.index.block_size > PAGE_SIZE) { - ntfs_error(vi->i_sb, "Index block size (%u) > " - "PAGE_SIZE (%ld) is not " - "supported. Sorry.", - ni->itype.index.block_size, - PAGE_SIZE); - err = -EOPNOTSUPP; - goto unm_err_out; - } - if (ni->itype.index.block_size < NTFS_BLOCK_SIZE) { - ntfs_error(vi->i_sb, "Index block size (%u) < " - "NTFS_BLOCK_SIZE (%i) is not " - "supported. Sorry.", - ni->itype.index.block_size, - NTFS_BLOCK_SIZE); - err = -EOPNOTSUPP; - goto unm_err_out; - } - ni->itype.index.block_size_bits = - ffs(ni->itype.index.block_size) - 1; - /* Determine the size of a vcn in the directory index. */ - if (vol->cluster_size <= ni->itype.index.block_size) { - ni->itype.index.vcn_size = vol->cluster_size; - ni->itype.index.vcn_size_bits = vol->cluster_size_bits; - } else { - ni->itype.index.vcn_size = vol->sector_size; - ni->itype.index.vcn_size_bits = vol->sector_size_bits; - } - - /* Setup the index allocation attribute, even if not present. */ - NInoSetMstProtected(ni); - ni->type = AT_INDEX_ALLOCATION; - ni->name = I30; - ni->name_len = 4; - - if (!(ir->index.flags & LARGE_INDEX)) { - /* No index allocation. */ - vi->i_size = ni->initialized_size = - ni->allocated_size = 0; - /* We are done with the mft record, so we release it. */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(ni); - m = NULL; - ctx = NULL; - goto skip_large_dir_stuff; - } /* LARGE_INDEX: Index allocation present. Setup state. */ - NInoSetIndexAllocPresent(ni); - /* Find index allocation attribute. */ - ntfs_attr_reinit_search_ctx(ctx); - err = ntfs_attr_lookup(AT_INDEX_ALLOCATION, I30, 4, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - ntfs_error(vi->i_sb, "$INDEX_ALLOCATION " - "attribute is not present but " - "$INDEX_ROOT indicated it is."); - else - ntfs_error(vi->i_sb, "Failed to lookup " - "$INDEX_ALLOCATION " - "attribute."); - goto unm_err_out; - } - a = ctx->attr; - if (!a->non_resident) { - ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute " - "is resident."); - goto unm_err_out; - } - /* - * Ensure the attribute name is placed before the mapping pairs - * array. - */ - if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >= - le16_to_cpu( - a->data.non_resident.mapping_pairs_offset)))) { - ntfs_error(vol->sb, "$INDEX_ALLOCATION attribute name " - "is placed after the mapping pairs " - "array."); - goto unm_err_out; - } - if (a->flags & ATTR_IS_ENCRYPTED) { - ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute " - "is encrypted."); - goto unm_err_out; - } - if (a->flags & ATTR_IS_SPARSE) { - ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute " - "is sparse."); - goto unm_err_out; - } - if (a->flags & ATTR_COMPRESSION_MASK) { - ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute " - "is compressed."); - goto unm_err_out; - } - if (a->data.non_resident.lowest_vcn) { - ntfs_error(vi->i_sb, "First extent of " - "$INDEX_ALLOCATION attribute has non " - "zero lowest_vcn."); - goto unm_err_out; - } - vi->i_size = sle64_to_cpu(a->data.non_resident.data_size); - ni->initialized_size = sle64_to_cpu( - a->data.non_resident.initialized_size); - ni->allocated_size = sle64_to_cpu( - a->data.non_resident.allocated_size); - /* - * We are done with the mft record, so we release it. Otherwise - * we would deadlock in ntfs_attr_iget(). - */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(ni); - m = NULL; - ctx = NULL; - /* Get the index bitmap attribute inode. */ - bvi = ntfs_attr_iget(vi, AT_BITMAP, I30, 4); - if (IS_ERR(bvi)) { - ntfs_error(vi->i_sb, "Failed to get bitmap attribute."); - err = PTR_ERR(bvi); - goto unm_err_out; - } - bni = NTFS_I(bvi); - if (NInoCompressed(bni) || NInoEncrypted(bni) || - NInoSparse(bni)) { - ntfs_error(vi->i_sb, "$BITMAP attribute is compressed " - "and/or encrypted and/or sparse."); - goto iput_unm_err_out; - } - /* Consistency check bitmap size vs. index allocation size. */ - bvi_size = i_size_read(bvi); - if ((bvi_size << 3) < (vi->i_size >> - ni->itype.index.block_size_bits)) { - ntfs_error(vi->i_sb, "Index bitmap too small (0x%llx) " - "for index allocation (0x%llx).", - bvi_size << 3, vi->i_size); - goto iput_unm_err_out; - } - /* No longer need the bitmap attribute inode. */ - iput(bvi); -skip_large_dir_stuff: - /* Setup the operations for this inode. */ - vi->i_op = &ntfs_dir_inode_ops; - vi->i_fop = &ntfs_dir_ops; - vi->i_mapping->a_ops = &ntfs_mst_aops; - } else { - /* It is a file. */ - ntfs_attr_reinit_search_ctx(ctx); - - /* Setup the data attribute, even if not present. */ - ni->type = AT_DATA; - ni->name = NULL; - ni->name_len = 0; - - /* Find first extent of the unnamed data attribute. */ - err = ntfs_attr_lookup(AT_DATA, NULL, 0, 0, 0, NULL, 0, ctx); - if (unlikely(err)) { - vi->i_size = ni->initialized_size = - ni->allocated_size = 0; - if (err != -ENOENT) { - ntfs_error(vi->i_sb, "Failed to lookup $DATA " - "attribute."); - goto unm_err_out; - } - /* - * FILE_Secure does not have an unnamed $DATA - * attribute, so we special case it here. - */ - if (vi->i_ino == FILE_Secure) - goto no_data_attr_special_case; - /* - * Most if not all the system files in the $Extend - * system directory do not have unnamed data - * attributes so we need to check if the parent - * directory of the file is FILE_Extend and if it is - * ignore this error. To do this we need to get the - * name of this inode from the mft record as the name - * contains the back reference to the parent directory. - */ - if (ntfs_is_extended_system_file(ctx) > 0) - goto no_data_attr_special_case; - // FIXME: File is corrupt! Hot-fix with empty data - // attribute if recovery option is set. - ntfs_error(vi->i_sb, "$DATA attribute is missing."); - goto unm_err_out; - } - a = ctx->attr; - /* Setup the state. */ - if (a->flags & (ATTR_COMPRESSION_MASK | ATTR_IS_SPARSE)) { - if (a->flags & ATTR_COMPRESSION_MASK) { - NInoSetCompressed(ni); - if (vol->cluster_size > 4096) { - ntfs_error(vi->i_sb, "Found " - "compressed data but " - "compression is " - "disabled due to " - "cluster size (%i) > " - "4kiB.", - vol->cluster_size); - goto unm_err_out; - } - if ((a->flags & ATTR_COMPRESSION_MASK) - != ATTR_IS_COMPRESSED) { - ntfs_error(vi->i_sb, "Found unknown " - "compression method " - "or corrupt file."); - goto unm_err_out; - } - } - if (a->flags & ATTR_IS_SPARSE) - NInoSetSparse(ni); - } - if (a->flags & ATTR_IS_ENCRYPTED) { - if (NInoCompressed(ni)) { - ntfs_error(vi->i_sb, "Found encrypted and " - "compressed data."); - goto unm_err_out; - } - NInoSetEncrypted(ni); - } - if (a->non_resident) { - NInoSetNonResident(ni); - if (NInoCompressed(ni) || NInoSparse(ni)) { - if (NInoCompressed(ni) && a->data.non_resident. - compression_unit != 4) { - ntfs_error(vi->i_sb, "Found " - "non-standard " - "compression unit (%u " - "instead of 4). " - "Cannot handle this.", - a->data.non_resident. - compression_unit); - err = -EOPNOTSUPP; - goto unm_err_out; - } - if (a->data.non_resident.compression_unit) { - ni->itype.compressed.block_size = 1U << - (a->data.non_resident. - compression_unit + - vol->cluster_size_bits); - ni->itype.compressed.block_size_bits = - ffs(ni->itype. - compressed. - block_size) - 1; - ni->itype.compressed.block_clusters = - 1U << a->data. - non_resident. - compression_unit; - } else { - ni->itype.compressed.block_size = 0; - ni->itype.compressed.block_size_bits = - 0; - ni->itype.compressed.block_clusters = - 0; - } - ni->itype.compressed.size = sle64_to_cpu( - a->data.non_resident. - compressed_size); - } - if (a->data.non_resident.lowest_vcn) { - ntfs_error(vi->i_sb, "First extent of $DATA " - "attribute has non zero " - "lowest_vcn."); - goto unm_err_out; - } - vi->i_size = sle64_to_cpu( - a->data.non_resident.data_size); - ni->initialized_size = sle64_to_cpu( - a->data.non_resident.initialized_size); - ni->allocated_size = sle64_to_cpu( - a->data.non_resident.allocated_size); - } else { /* Resident attribute. */ - vi->i_size = ni->initialized_size = le32_to_cpu( - a->data.resident.value_length); - ni->allocated_size = le32_to_cpu(a->length) - - le16_to_cpu( - a->data.resident.value_offset); - if (vi->i_size > ni->allocated_size) { - ntfs_error(vi->i_sb, "Resident data attribute " - "is corrupt (size exceeds " - "allocation)."); - goto unm_err_out; - } - } -no_data_attr_special_case: - /* We are done with the mft record, so we release it. */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(ni); - m = NULL; - ctx = NULL; - /* Setup the operations for this inode. */ - vi->i_op = &ntfs_file_inode_ops; - vi->i_fop = &ntfs_file_ops; - vi->i_mapping->a_ops = &ntfs_normal_aops; - if (NInoMstProtected(ni)) - vi->i_mapping->a_ops = &ntfs_mst_aops; - else if (NInoCompressed(ni)) - vi->i_mapping->a_ops = &ntfs_compressed_aops; - } - /* - * The number of 512-byte blocks used on disk (for stat). This is in so - * far inaccurate as it doesn't account for any named streams or other - * special non-resident attributes, but that is how Windows works, too, - * so we are at least consistent with Windows, if not entirely - * consistent with the Linux Way. Doing it the Linux Way would cause a - * significant slowdown as it would involve iterating over all - * attributes in the mft record and adding the allocated/compressed - * sizes of all non-resident attributes present to give us the Linux - * correct size that should go into i_blocks (after division by 512). - */ - if (S_ISREG(vi->i_mode) && (NInoCompressed(ni) || NInoSparse(ni))) - vi->i_blocks = ni->itype.compressed.size >> 9; - else - vi->i_blocks = ni->allocated_size >> 9; - ntfs_debug("Done."); - return 0; -iput_unm_err_out: - iput(bvi); -unm_err_out: - if (!err) - err = -EIO; - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(ni); -err_out: - ntfs_error(vol->sb, "Failed with error code %i. Marking corrupt " - "inode 0x%lx as bad. Run chkdsk.", err, vi->i_ino); - make_bad_inode(vi); - if (err != -EOPNOTSUPP && err != -ENOMEM) - NVolSetErrors(vol); - return err; -} - -/** - * ntfs_read_locked_attr_inode - read an attribute inode from its base inode - * @base_vi: base inode - * @vi: attribute inode to read - * - * ntfs_read_locked_attr_inode() is called from ntfs_attr_iget() to read the - * attribute inode described by @vi into memory from the base mft record - * described by @base_ni. - * - * ntfs_read_locked_attr_inode() maps, pins and locks the base inode for - * reading and looks up the attribute described by @vi before setting up the - * necessary fields in @vi as well as initializing the ntfs inode. - * - * Q: What locks are held when the function is called? - * A: i_state has I_NEW set, hence the inode is locked, also - * i_count is set to 1, so it is not going to go away - * - * Return 0 on success and -errno on error. In the error case, the inode will - * have had make_bad_inode() executed on it. - * - * Note this cannot be called for AT_INDEX_ALLOCATION. - */ -static int ntfs_read_locked_attr_inode(struct inode *base_vi, struct inode *vi) -{ - ntfs_volume *vol = NTFS_SB(vi->i_sb); - ntfs_inode *ni, *base_ni; - MFT_RECORD *m; - ATTR_RECORD *a; - ntfs_attr_search_ctx *ctx; - int err = 0; - - ntfs_debug("Entering for i_ino 0x%lx.", vi->i_ino); - - ntfs_init_big_inode(vi); - - ni = NTFS_I(vi); - base_ni = NTFS_I(base_vi); - - /* Just mirror the values from the base inode. */ - vi->i_uid = base_vi->i_uid; - vi->i_gid = base_vi->i_gid; - set_nlink(vi, base_vi->i_nlink); - vi->i_mtime = base_vi->i_mtime; - inode_set_ctime_to_ts(vi, inode_get_ctime(base_vi)); - vi->i_atime = base_vi->i_atime; - vi->i_generation = ni->seq_no = base_ni->seq_no; - - /* Set inode type to zero but preserve permissions. */ - vi->i_mode = base_vi->i_mode & ~S_IFMT; - - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (!ctx) { - err = -ENOMEM; - goto unm_err_out; - } - /* Find the attribute. */ - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) - goto unm_err_out; - a = ctx->attr; - if (a->flags & (ATTR_COMPRESSION_MASK | ATTR_IS_SPARSE)) { - if (a->flags & ATTR_COMPRESSION_MASK) { - NInoSetCompressed(ni); - if ((ni->type != AT_DATA) || (ni->type == AT_DATA && - ni->name_len)) { - ntfs_error(vi->i_sb, "Found compressed " - "non-data or named data " - "attribute. Please report " - "you saw this message to " - "linux-ntfs-dev@lists." - "sourceforge.net"); - goto unm_err_out; - } - if (vol->cluster_size > 4096) { - ntfs_error(vi->i_sb, "Found compressed " - "attribute but compression is " - "disabled due to cluster size " - "(%i) > 4kiB.", - vol->cluster_size); - goto unm_err_out; - } - if ((a->flags & ATTR_COMPRESSION_MASK) != - ATTR_IS_COMPRESSED) { - ntfs_error(vi->i_sb, "Found unknown " - "compression method."); - goto unm_err_out; - } - } - /* - * The compressed/sparse flag set in an index root just means - * to compress all files. - */ - if (NInoMstProtected(ni) && ni->type != AT_INDEX_ROOT) { - ntfs_error(vi->i_sb, "Found mst protected attribute " - "but the attribute is %s. Please " - "report you saw this message to " - "linux-ntfs-dev@lists.sourceforge.net", - NInoCompressed(ni) ? "compressed" : - "sparse"); - goto unm_err_out; - } - if (a->flags & ATTR_IS_SPARSE) - NInoSetSparse(ni); - } - if (a->flags & ATTR_IS_ENCRYPTED) { - if (NInoCompressed(ni)) { - ntfs_error(vi->i_sb, "Found encrypted and compressed " - "data."); - goto unm_err_out; - } - /* - * The encryption flag set in an index root just means to - * encrypt all files. - */ - if (NInoMstProtected(ni) && ni->type != AT_INDEX_ROOT) { - ntfs_error(vi->i_sb, "Found mst protected attribute " - "but the attribute is encrypted. " - "Please report you saw this message " - "to linux-ntfs-dev@lists.sourceforge." - "net"); - goto unm_err_out; - } - if (ni->type != AT_DATA) { - ntfs_error(vi->i_sb, "Found encrypted non-data " - "attribute."); - goto unm_err_out; - } - NInoSetEncrypted(ni); - } - if (!a->non_resident) { - /* Ensure the attribute name is placed before the value. */ - if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >= - le16_to_cpu(a->data.resident.value_offset)))) { - ntfs_error(vol->sb, "Attribute name is placed after " - "the attribute value."); - goto unm_err_out; - } - if (NInoMstProtected(ni)) { - ntfs_error(vi->i_sb, "Found mst protected attribute " - "but the attribute is resident. " - "Please report you saw this message to " - "linux-ntfs-dev@lists.sourceforge.net"); - goto unm_err_out; - } - vi->i_size = ni->initialized_size = le32_to_cpu( - a->data.resident.value_length); - ni->allocated_size = le32_to_cpu(a->length) - - le16_to_cpu(a->data.resident.value_offset); - if (vi->i_size > ni->allocated_size) { - ntfs_error(vi->i_sb, "Resident attribute is corrupt " - "(size exceeds allocation)."); - goto unm_err_out; - } - } else { - NInoSetNonResident(ni); - /* - * Ensure the attribute name is placed before the mapping pairs - * array. - */ - if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >= - le16_to_cpu( - a->data.non_resident.mapping_pairs_offset)))) { - ntfs_error(vol->sb, "Attribute name is placed after " - "the mapping pairs array."); - goto unm_err_out; - } - if (NInoCompressed(ni) || NInoSparse(ni)) { - if (NInoCompressed(ni) && a->data.non_resident. - compression_unit != 4) { - ntfs_error(vi->i_sb, "Found non-standard " - "compression unit (%u instead " - "of 4). Cannot handle this.", - a->data.non_resident. - compression_unit); - err = -EOPNOTSUPP; - goto unm_err_out; - } - if (a->data.non_resident.compression_unit) { - ni->itype.compressed.block_size = 1U << - (a->data.non_resident. - compression_unit + - vol->cluster_size_bits); - ni->itype.compressed.block_size_bits = - ffs(ni->itype.compressed. - block_size) - 1; - ni->itype.compressed.block_clusters = 1U << - a->data.non_resident. - compression_unit; - } else { - ni->itype.compressed.block_size = 0; - ni->itype.compressed.block_size_bits = 0; - ni->itype.compressed.block_clusters = 0; - } - ni->itype.compressed.size = sle64_to_cpu( - a->data.non_resident.compressed_size); - } - if (a->data.non_resident.lowest_vcn) { - ntfs_error(vi->i_sb, "First extent of attribute has " - "non-zero lowest_vcn."); - goto unm_err_out; - } - vi->i_size = sle64_to_cpu(a->data.non_resident.data_size); - ni->initialized_size = sle64_to_cpu( - a->data.non_resident.initialized_size); - ni->allocated_size = sle64_to_cpu( - a->data.non_resident.allocated_size); - } - vi->i_mapping->a_ops = &ntfs_normal_aops; - if (NInoMstProtected(ni)) - vi->i_mapping->a_ops = &ntfs_mst_aops; - else if (NInoCompressed(ni)) - vi->i_mapping->a_ops = &ntfs_compressed_aops; - if ((NInoCompressed(ni) || NInoSparse(ni)) && ni->type != AT_INDEX_ROOT) - vi->i_blocks = ni->itype.compressed.size >> 9; - else - vi->i_blocks = ni->allocated_size >> 9; - /* - * Make sure the base inode does not go away and attach it to the - * attribute inode. - */ - igrab(base_vi); - ni->ext.base_ntfs_ino = base_ni; - ni->nr_extents = -1; - - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - - ntfs_debug("Done."); - return 0; - -unm_err_out: - if (!err) - err = -EIO; - if (ctx) - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); -err_out: - ntfs_error(vol->sb, "Failed with error code %i while reading attribute " - "inode (mft_no 0x%lx, type 0x%x, name_len %i). " - "Marking corrupt inode and base inode 0x%lx as bad. " - "Run chkdsk.", err, vi->i_ino, ni->type, ni->name_len, - base_vi->i_ino); - make_bad_inode(vi); - if (err != -ENOMEM) - NVolSetErrors(vol); - return err; -} - -/** - * ntfs_read_locked_index_inode - read an index inode from its base inode - * @base_vi: base inode - * @vi: index inode to read - * - * ntfs_read_locked_index_inode() is called from ntfs_index_iget() to read the - * index inode described by @vi into memory from the base mft record described - * by @base_ni. - * - * ntfs_read_locked_index_inode() maps, pins and locks the base inode for - * reading and looks up the attributes relating to the index described by @vi - * before setting up the necessary fields in @vi as well as initializing the - * ntfs inode. - * - * Note, index inodes are essentially attribute inodes (NInoAttr() is true) - * with the attribute type set to AT_INDEX_ALLOCATION. Apart from that, they - * are setup like directory inodes since directories are a special case of - * indices ao they need to be treated in much the same way. Most importantly, - * for small indices the index allocation attribute might not actually exist. - * However, the index root attribute always exists but this does not need to - * have an inode associated with it and this is why we define a new inode type - * index. Also, like for directories, we need to have an attribute inode for - * the bitmap attribute corresponding to the index allocation attribute and we - * can store this in the appropriate field of the inode, just like we do for - * normal directory inodes. - * - * Q: What locks are held when the function is called? - * A: i_state has I_NEW set, hence the inode is locked, also - * i_count is set to 1, so it is not going to go away - * - * Return 0 on success and -errno on error. In the error case, the inode will - * have had make_bad_inode() executed on it. - */ -static int ntfs_read_locked_index_inode(struct inode *base_vi, struct inode *vi) -{ - loff_t bvi_size; - ntfs_volume *vol = NTFS_SB(vi->i_sb); - ntfs_inode *ni, *base_ni, *bni; - struct inode *bvi; - MFT_RECORD *m; - ATTR_RECORD *a; - ntfs_attr_search_ctx *ctx; - INDEX_ROOT *ir; - u8 *ir_end, *index_end; - int err = 0; - - ntfs_debug("Entering for i_ino 0x%lx.", vi->i_ino); - ntfs_init_big_inode(vi); - ni = NTFS_I(vi); - base_ni = NTFS_I(base_vi); - /* Just mirror the values from the base inode. */ - vi->i_uid = base_vi->i_uid; - vi->i_gid = base_vi->i_gid; - set_nlink(vi, base_vi->i_nlink); - vi->i_mtime = base_vi->i_mtime; - inode_set_ctime_to_ts(vi, inode_get_ctime(base_vi)); - vi->i_atime = base_vi->i_atime; - vi->i_generation = ni->seq_no = base_ni->seq_no; - /* Set inode type to zero but preserve permissions. */ - vi->i_mode = base_vi->i_mode & ~S_IFMT; - /* Map the mft record for the base inode. */ - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (!ctx) { - err = -ENOMEM; - goto unm_err_out; - } - /* Find the index root attribute. */ - err = ntfs_attr_lookup(AT_INDEX_ROOT, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - ntfs_error(vi->i_sb, "$INDEX_ROOT attribute is " - "missing."); - goto unm_err_out; - } - a = ctx->attr; - /* Set up the state. */ - if (unlikely(a->non_resident)) { - ntfs_error(vol->sb, "$INDEX_ROOT attribute is not resident."); - goto unm_err_out; - } - /* Ensure the attribute name is placed before the value. */ - if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >= - le16_to_cpu(a->data.resident.value_offset)))) { - ntfs_error(vol->sb, "$INDEX_ROOT attribute name is placed " - "after the attribute value."); - goto unm_err_out; - } - /* - * Compressed/encrypted/sparse index root is not allowed, except for - * directories of course but those are not dealt with here. - */ - if (a->flags & (ATTR_COMPRESSION_MASK | ATTR_IS_ENCRYPTED | - ATTR_IS_SPARSE)) { - ntfs_error(vi->i_sb, "Found compressed/encrypted/sparse index " - "root attribute."); - goto unm_err_out; - } - ir = (INDEX_ROOT*)((u8*)a + le16_to_cpu(a->data.resident.value_offset)); - ir_end = (u8*)ir + le32_to_cpu(a->data.resident.value_length); - if (ir_end > (u8*)ctx->mrec + vol->mft_record_size) { - ntfs_error(vi->i_sb, "$INDEX_ROOT attribute is corrupt."); - goto unm_err_out; - } - index_end = (u8*)&ir->index + le32_to_cpu(ir->index.index_length); - if (index_end > ir_end) { - ntfs_error(vi->i_sb, "Index is corrupt."); - goto unm_err_out; - } - if (ir->type) { - ntfs_error(vi->i_sb, "Index type is not 0 (type is 0x%x).", - le32_to_cpu(ir->type)); - goto unm_err_out; - } - ni->itype.index.collation_rule = ir->collation_rule; - ntfs_debug("Index collation rule is 0x%x.", - le32_to_cpu(ir->collation_rule)); - ni->itype.index.block_size = le32_to_cpu(ir->index_block_size); - if (!is_power_of_2(ni->itype.index.block_size)) { - ntfs_error(vi->i_sb, "Index block size (%u) is not a power of " - "two.", ni->itype.index.block_size); - goto unm_err_out; - } - if (ni->itype.index.block_size > PAGE_SIZE) { - ntfs_error(vi->i_sb, "Index block size (%u) > PAGE_SIZE " - "(%ld) is not supported. Sorry.", - ni->itype.index.block_size, PAGE_SIZE); - err = -EOPNOTSUPP; - goto unm_err_out; - } - if (ni->itype.index.block_size < NTFS_BLOCK_SIZE) { - ntfs_error(vi->i_sb, "Index block size (%u) < NTFS_BLOCK_SIZE " - "(%i) is not supported. Sorry.", - ni->itype.index.block_size, NTFS_BLOCK_SIZE); - err = -EOPNOTSUPP; - goto unm_err_out; - } - ni->itype.index.block_size_bits = ffs(ni->itype.index.block_size) - 1; - /* Determine the size of a vcn in the index. */ - if (vol->cluster_size <= ni->itype.index.block_size) { - ni->itype.index.vcn_size = vol->cluster_size; - ni->itype.index.vcn_size_bits = vol->cluster_size_bits; - } else { - ni->itype.index.vcn_size = vol->sector_size; - ni->itype.index.vcn_size_bits = vol->sector_size_bits; - } - /* Check for presence of index allocation attribute. */ - if (!(ir->index.flags & LARGE_INDEX)) { - /* No index allocation. */ - vi->i_size = ni->initialized_size = ni->allocated_size = 0; - /* We are done with the mft record, so we release it. */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - m = NULL; - ctx = NULL; - goto skip_large_index_stuff; - } /* LARGE_INDEX: Index allocation present. Setup state. */ - NInoSetIndexAllocPresent(ni); - /* Find index allocation attribute. */ - ntfs_attr_reinit_search_ctx(ctx); - err = ntfs_attr_lookup(AT_INDEX_ALLOCATION, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) - ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute is " - "not present but $INDEX_ROOT " - "indicated it is."); - else - ntfs_error(vi->i_sb, "Failed to lookup " - "$INDEX_ALLOCATION attribute."); - goto unm_err_out; - } - a = ctx->attr; - if (!a->non_resident) { - ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute is " - "resident."); - goto unm_err_out; - } - /* - * Ensure the attribute name is placed before the mapping pairs array. - */ - if (unlikely(a->name_length && (le16_to_cpu(a->name_offset) >= - le16_to_cpu( - a->data.non_resident.mapping_pairs_offset)))) { - ntfs_error(vol->sb, "$INDEX_ALLOCATION attribute name is " - "placed after the mapping pairs array."); - goto unm_err_out; - } - if (a->flags & ATTR_IS_ENCRYPTED) { - ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute is " - "encrypted."); - goto unm_err_out; - } - if (a->flags & ATTR_IS_SPARSE) { - ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute is sparse."); - goto unm_err_out; - } - if (a->flags & ATTR_COMPRESSION_MASK) { - ntfs_error(vi->i_sb, "$INDEX_ALLOCATION attribute is " - "compressed."); - goto unm_err_out; - } - if (a->data.non_resident.lowest_vcn) { - ntfs_error(vi->i_sb, "First extent of $INDEX_ALLOCATION " - "attribute has non zero lowest_vcn."); - goto unm_err_out; - } - vi->i_size = sle64_to_cpu(a->data.non_resident.data_size); - ni->initialized_size = sle64_to_cpu( - a->data.non_resident.initialized_size); - ni->allocated_size = sle64_to_cpu(a->data.non_resident.allocated_size); - /* - * We are done with the mft record, so we release it. Otherwise - * we would deadlock in ntfs_attr_iget(). - */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - m = NULL; - ctx = NULL; - /* Get the index bitmap attribute inode. */ - bvi = ntfs_attr_iget(base_vi, AT_BITMAP, ni->name, ni->name_len); - if (IS_ERR(bvi)) { - ntfs_error(vi->i_sb, "Failed to get bitmap attribute."); - err = PTR_ERR(bvi); - goto unm_err_out; - } - bni = NTFS_I(bvi); - if (NInoCompressed(bni) || NInoEncrypted(bni) || - NInoSparse(bni)) { - ntfs_error(vi->i_sb, "$BITMAP attribute is compressed and/or " - "encrypted and/or sparse."); - goto iput_unm_err_out; - } - /* Consistency check bitmap size vs. index allocation size. */ - bvi_size = i_size_read(bvi); - if ((bvi_size << 3) < (vi->i_size >> ni->itype.index.block_size_bits)) { - ntfs_error(vi->i_sb, "Index bitmap too small (0x%llx) for " - "index allocation (0x%llx).", bvi_size << 3, - vi->i_size); - goto iput_unm_err_out; - } - iput(bvi); -skip_large_index_stuff: - /* Setup the operations for this index inode. */ - vi->i_mapping->a_ops = &ntfs_mst_aops; - vi->i_blocks = ni->allocated_size >> 9; - /* - * Make sure the base inode doesn't go away and attach it to the - * index inode. - */ - igrab(base_vi); - ni->ext.base_ntfs_ino = base_ni; - ni->nr_extents = -1; - - ntfs_debug("Done."); - return 0; -iput_unm_err_out: - iput(bvi); -unm_err_out: - if (!err) - err = -EIO; - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(base_ni); -err_out: - ntfs_error(vi->i_sb, "Failed with error code %i while reading index " - "inode (mft_no 0x%lx, name_len %i.", err, vi->i_ino, - ni->name_len); - make_bad_inode(vi); - if (err != -EOPNOTSUPP && err != -ENOMEM) - NVolSetErrors(vol); - return err; -} - -/* - * The MFT inode has special locking, so teach the lock validator - * about this by splitting off the locking rules of the MFT from - * the locking rules of other inodes. The MFT inode can never be - * accessed from the VFS side (or even internally), only by the - * map_mft functions. - */ -static struct lock_class_key mft_ni_runlist_lock_key, mft_ni_mrec_lock_key; - -/** - * ntfs_read_inode_mount - special read_inode for mount time use only - * @vi: inode to read - * - * Read inode FILE_MFT at mount time, only called with super_block lock - * held from within the read_super() code path. - * - * This function exists because when it is called the page cache for $MFT/$DATA - * is not initialized and hence we cannot get at the contents of mft records - * by calling map_mft_record*(). - * - * Further it needs to cope with the circular references problem, i.e. cannot - * load any attributes other than $ATTRIBUTE_LIST until $DATA is loaded, because - * we do not know where the other extent mft records are yet and again, because - * we cannot call map_mft_record*() yet. Obviously this applies only when an - * attribute list is actually present in $MFT inode. - * - * We solve these problems by starting with the $DATA attribute before anything - * else and iterating using ntfs_attr_lookup($DATA) over all extents. As each - * extent is found, we ntfs_mapping_pairs_decompress() including the implied - * ntfs_runlists_merge(). Each step of the iteration necessarily provides - * sufficient information for the next step to complete. - * - * This should work but there are two possible pit falls (see inline comments - * below), but only time will tell if they are real pits or just smoke... - */ -int ntfs_read_inode_mount(struct inode *vi) -{ - VCN next_vcn, last_vcn, highest_vcn; - s64 block; - struct super_block *sb = vi->i_sb; - ntfs_volume *vol = NTFS_SB(sb); - struct buffer_head *bh; - ntfs_inode *ni; - MFT_RECORD *m = NULL; - ATTR_RECORD *a; - ntfs_attr_search_ctx *ctx; - unsigned int i, nr_blocks; - int err; - - ntfs_debug("Entering."); - - /* Initialize the ntfs specific part of @vi. */ - ntfs_init_big_inode(vi); - - ni = NTFS_I(vi); - - /* Setup the data attribute. It is special as it is mst protected. */ - NInoSetNonResident(ni); - NInoSetMstProtected(ni); - NInoSetSparseDisabled(ni); - ni->type = AT_DATA; - ni->name = NULL; - ni->name_len = 0; - /* - * This sets up our little cheat allowing us to reuse the async read io - * completion handler for directories. - */ - ni->itype.index.block_size = vol->mft_record_size; - ni->itype.index.block_size_bits = vol->mft_record_size_bits; - - /* Very important! Needed to be able to call map_mft_record*(). */ - vol->mft_ino = vi; - - /* Allocate enough memory to read the first mft record. */ - if (vol->mft_record_size > 64 * 1024) { - ntfs_error(sb, "Unsupported mft record size %i (max 64kiB).", - vol->mft_record_size); - goto err_out; - } - i = vol->mft_record_size; - if (i < sb->s_blocksize) - i = sb->s_blocksize; - m = (MFT_RECORD*)ntfs_malloc_nofs(i); - if (!m) { - ntfs_error(sb, "Failed to allocate buffer for $MFT record 0."); - goto err_out; - } - - /* Determine the first block of the $MFT/$DATA attribute. */ - block = vol->mft_lcn << vol->cluster_size_bits >> - sb->s_blocksize_bits; - nr_blocks = vol->mft_record_size >> sb->s_blocksize_bits; - if (!nr_blocks) - nr_blocks = 1; - - /* Load $MFT/$DATA's first mft record. */ - for (i = 0; i < nr_blocks; i++) { - bh = sb_bread(sb, block++); - if (!bh) { - ntfs_error(sb, "Device read failed."); - goto err_out; - } - memcpy((char*)m + (i << sb->s_blocksize_bits), bh->b_data, - sb->s_blocksize); - brelse(bh); - } - - if (le32_to_cpu(m->bytes_allocated) != vol->mft_record_size) { - ntfs_error(sb, "Incorrect mft record size %u in superblock, should be %u.", - le32_to_cpu(m->bytes_allocated), vol->mft_record_size); - goto err_out; - } - - /* Apply the mst fixups. */ - if (post_read_mst_fixup((NTFS_RECORD*)m, vol->mft_record_size)) { - /* FIXME: Try to use the $MFTMirr now. */ - ntfs_error(sb, "MST fixup failed. $MFT is corrupt."); - goto err_out; - } - - /* Sanity check offset to the first attribute */ - if (le16_to_cpu(m->attrs_offset) >= le32_to_cpu(m->bytes_allocated)) { - ntfs_error(sb, "Incorrect mft offset to the first attribute %u in superblock.", - le16_to_cpu(m->attrs_offset)); - goto err_out; - } - - /* Need this to sanity check attribute list references to $MFT. */ - vi->i_generation = ni->seq_no = le16_to_cpu(m->sequence_number); - - /* Provides read_folio() for map_mft_record(). */ - vi->i_mapping->a_ops = &ntfs_mst_aops; - - ctx = ntfs_attr_get_search_ctx(ni, m); - if (!ctx) { - err = -ENOMEM; - goto err_out; - } - - /* Find the attribute list attribute if present. */ - err = ntfs_attr_lookup(AT_ATTRIBUTE_LIST, NULL, 0, 0, 0, NULL, 0, ctx); - if (err) { - if (unlikely(err != -ENOENT)) { - ntfs_error(sb, "Failed to lookup attribute list " - "attribute. You should run chkdsk."); - goto put_err_out; - } - } else /* if (!err) */ { - ATTR_LIST_ENTRY *al_entry, *next_al_entry; - u8 *al_end; - static const char *es = " Not allowed. $MFT is corrupt. " - "You should run chkdsk."; - - ntfs_debug("Attribute list attribute found in $MFT."); - NInoSetAttrList(ni); - a = ctx->attr; - if (a->flags & ATTR_COMPRESSION_MASK) { - ntfs_error(sb, "Attribute list attribute is " - "compressed.%s", es); - goto put_err_out; - } - if (a->flags & ATTR_IS_ENCRYPTED || - a->flags & ATTR_IS_SPARSE) { - if (a->non_resident) { - ntfs_error(sb, "Non-resident attribute list " - "attribute is encrypted/" - "sparse.%s", es); - goto put_err_out; - } - ntfs_warning(sb, "Resident attribute list attribute " - "in $MFT system file is marked " - "encrypted/sparse which is not true. " - "However, Windows allows this and " - "chkdsk does not detect or correct it " - "so we will just ignore the invalid " - "flags and pretend they are not set."); - } - /* Now allocate memory for the attribute list. */ - ni->attr_list_size = (u32)ntfs_attr_size(a); - if (!ni->attr_list_size) { - ntfs_error(sb, "Attr_list_size is zero"); - goto put_err_out; - } - ni->attr_list = ntfs_malloc_nofs(ni->attr_list_size); - if (!ni->attr_list) { - ntfs_error(sb, "Not enough memory to allocate buffer " - "for attribute list."); - goto put_err_out; - } - if (a->non_resident) { - NInoSetAttrListNonResident(ni); - if (a->data.non_resident.lowest_vcn) { - ntfs_error(sb, "Attribute list has non zero " - "lowest_vcn. $MFT is corrupt. " - "You should run chkdsk."); - goto put_err_out; - } - /* Setup the runlist. */ - ni->attr_list_rl.rl = ntfs_mapping_pairs_decompress(vol, - a, NULL); - if (IS_ERR(ni->attr_list_rl.rl)) { - err = PTR_ERR(ni->attr_list_rl.rl); - ni->attr_list_rl.rl = NULL; - ntfs_error(sb, "Mapping pairs decompression " - "failed with error code %i.", - -err); - goto put_err_out; - } - /* Now load the attribute list. */ - if ((err = load_attribute_list(vol, &ni->attr_list_rl, - ni->attr_list, ni->attr_list_size, - sle64_to_cpu(a->data. - non_resident.initialized_size)))) { - ntfs_error(sb, "Failed to load attribute list " - "attribute with error code %i.", - -err); - goto put_err_out; - } - } else /* if (!ctx.attr->non_resident) */ { - if ((u8*)a + le16_to_cpu( - a->data.resident.value_offset) + - le32_to_cpu( - a->data.resident.value_length) > - (u8*)ctx->mrec + vol->mft_record_size) { - ntfs_error(sb, "Corrupt attribute list " - "attribute."); - goto put_err_out; - } - /* Now copy the attribute list. */ - memcpy(ni->attr_list, (u8*)a + le16_to_cpu( - a->data.resident.value_offset), - le32_to_cpu( - a->data.resident.value_length)); - } - /* The attribute list is now setup in memory. */ - /* - * FIXME: I don't know if this case is actually possible. - * According to logic it is not possible but I have seen too - * many weird things in MS software to rely on logic... Thus we - * perform a manual search and make sure the first $MFT/$DATA - * extent is in the base inode. If it is not we abort with an - * error and if we ever see a report of this error we will need - * to do some magic in order to have the necessary mft record - * loaded and in the right place in the page cache. But - * hopefully logic will prevail and this never happens... - */ - al_entry = (ATTR_LIST_ENTRY*)ni->attr_list; - al_end = (u8*)al_entry + ni->attr_list_size; - for (;; al_entry = next_al_entry) { - /* Out of bounds check. */ - if ((u8*)al_entry < ni->attr_list || - (u8*)al_entry > al_end) - goto em_put_err_out; - /* Catch the end of the attribute list. */ - if ((u8*)al_entry == al_end) - goto em_put_err_out; - if (!al_entry->length) - goto em_put_err_out; - if ((u8*)al_entry + 6 > al_end || (u8*)al_entry + - le16_to_cpu(al_entry->length) > al_end) - goto em_put_err_out; - next_al_entry = (ATTR_LIST_ENTRY*)((u8*)al_entry + - le16_to_cpu(al_entry->length)); - if (le32_to_cpu(al_entry->type) > le32_to_cpu(AT_DATA)) - goto em_put_err_out; - if (AT_DATA != al_entry->type) - continue; - /* We want an unnamed attribute. */ - if (al_entry->name_length) - goto em_put_err_out; - /* Want the first entry, i.e. lowest_vcn == 0. */ - if (al_entry->lowest_vcn) - goto em_put_err_out; - /* First entry has to be in the base mft record. */ - if (MREF_LE(al_entry->mft_reference) != vi->i_ino) { - /* MFT references do not match, logic fails. */ - ntfs_error(sb, "BUG: The first $DATA extent " - "of $MFT is not in the base " - "mft record. Please report " - "you saw this message to " - "linux-ntfs-dev@lists." - "sourceforge.net"); - goto put_err_out; - } else { - /* Sequence numbers must match. */ - if (MSEQNO_LE(al_entry->mft_reference) != - ni->seq_no) - goto em_put_err_out; - /* Got it. All is ok. We can stop now. */ - break; - } - } - } - - ntfs_attr_reinit_search_ctx(ctx); - - /* Now load all attribute extents. */ - a = NULL; - next_vcn = last_vcn = highest_vcn = 0; - while (!(err = ntfs_attr_lookup(AT_DATA, NULL, 0, 0, next_vcn, NULL, 0, - ctx))) { - runlist_element *nrl; - - /* Cache the current attribute. */ - a = ctx->attr; - /* $MFT must be non-resident. */ - if (!a->non_resident) { - ntfs_error(sb, "$MFT must be non-resident but a " - "resident extent was found. $MFT is " - "corrupt. Run chkdsk."); - goto put_err_out; - } - /* $MFT must be uncompressed and unencrypted. */ - if (a->flags & ATTR_COMPRESSION_MASK || - a->flags & ATTR_IS_ENCRYPTED || - a->flags & ATTR_IS_SPARSE) { - ntfs_error(sb, "$MFT must be uncompressed, " - "non-sparse, and unencrypted but a " - "compressed/sparse/encrypted extent " - "was found. $MFT is corrupt. Run " - "chkdsk."); - goto put_err_out; - } - /* - * Decompress the mapping pairs array of this extent and merge - * the result into the existing runlist. No need for locking - * as we have exclusive access to the inode at this time and we - * are a mount in progress task, too. - */ - nrl = ntfs_mapping_pairs_decompress(vol, a, ni->runlist.rl); - if (IS_ERR(nrl)) { - ntfs_error(sb, "ntfs_mapping_pairs_decompress() " - "failed with error code %ld. $MFT is " - "corrupt.", PTR_ERR(nrl)); - goto put_err_out; - } - ni->runlist.rl = nrl; - - /* Are we in the first extent? */ - if (!next_vcn) { - if (a->data.non_resident.lowest_vcn) { - ntfs_error(sb, "First extent of $DATA " - "attribute has non zero " - "lowest_vcn. $MFT is corrupt. " - "You should run chkdsk."); - goto put_err_out; - } - /* Get the last vcn in the $DATA attribute. */ - last_vcn = sle64_to_cpu( - a->data.non_resident.allocated_size) - >> vol->cluster_size_bits; - /* Fill in the inode size. */ - vi->i_size = sle64_to_cpu( - a->data.non_resident.data_size); - ni->initialized_size = sle64_to_cpu( - a->data.non_resident.initialized_size); - ni->allocated_size = sle64_to_cpu( - a->data.non_resident.allocated_size); - /* - * Verify the number of mft records does not exceed - * 2^32 - 1. - */ - if ((vi->i_size >> vol->mft_record_size_bits) >= - (1ULL << 32)) { - ntfs_error(sb, "$MFT is too big! Aborting."); - goto put_err_out; - } - /* - * We have got the first extent of the runlist for - * $MFT which means it is now relatively safe to call - * the normal ntfs_read_inode() function. - * Complete reading the inode, this will actually - * re-read the mft record for $MFT, this time entering - * it into the page cache with which we complete the - * kick start of the volume. It should be safe to do - * this now as the first extent of $MFT/$DATA is - * already known and we would hope that we don't need - * further extents in order to find the other - * attributes belonging to $MFT. Only time will tell if - * this is really the case. If not we will have to play - * magic at this point, possibly duplicating a lot of - * ntfs_read_inode() at this point. We will need to - * ensure we do enough of its work to be able to call - * ntfs_read_inode() on extents of $MFT/$DATA. But lets - * hope this never happens... - */ - ntfs_read_locked_inode(vi); - if (is_bad_inode(vi)) { - ntfs_error(sb, "ntfs_read_inode() of $MFT " - "failed. BUG or corrupt $MFT. " - "Run chkdsk and if no errors " - "are found, please report you " - "saw this message to " - "linux-ntfs-dev@lists." - "sourceforge.net"); - ntfs_attr_put_search_ctx(ctx); - /* Revert to the safe super operations. */ - ntfs_free(m); - return -1; - } - /* - * Re-initialize some specifics about $MFT's inode as - * ntfs_read_inode() will have set up the default ones. - */ - /* Set uid and gid to root. */ - vi->i_uid = GLOBAL_ROOT_UID; - vi->i_gid = GLOBAL_ROOT_GID; - /* Regular file. No access for anyone. */ - vi->i_mode = S_IFREG; - /* No VFS initiated operations allowed for $MFT. */ - vi->i_op = &ntfs_empty_inode_ops; - vi->i_fop = &ntfs_empty_file_ops; - } - - /* Get the lowest vcn for the next extent. */ - highest_vcn = sle64_to_cpu(a->data.non_resident.highest_vcn); - next_vcn = highest_vcn + 1; - - /* Only one extent or error, which we catch below. */ - if (next_vcn <= 0) - break; - - /* Avoid endless loops due to corruption. */ - if (next_vcn < sle64_to_cpu( - a->data.non_resident.lowest_vcn)) { - ntfs_error(sb, "$MFT has corrupt attribute list " - "attribute. Run chkdsk."); - goto put_err_out; - } - } - if (err != -ENOENT) { - ntfs_error(sb, "Failed to lookup $MFT/$DATA attribute extent. " - "$MFT is corrupt. Run chkdsk."); - goto put_err_out; - } - if (!a) { - ntfs_error(sb, "$MFT/$DATA attribute not found. $MFT is " - "corrupt. Run chkdsk."); - goto put_err_out; - } - if (highest_vcn && highest_vcn != last_vcn - 1) { - ntfs_error(sb, "Failed to load the complete runlist for " - "$MFT/$DATA. Driver bug or corrupt $MFT. " - "Run chkdsk."); - ntfs_debug("highest_vcn = 0x%llx, last_vcn - 1 = 0x%llx", - (unsigned long long)highest_vcn, - (unsigned long long)last_vcn - 1); - goto put_err_out; - } - ntfs_attr_put_search_ctx(ctx); - ntfs_debug("Done."); - ntfs_free(m); - - /* - * Split the locking rules of the MFT inode from the - * locking rules of other inodes: - */ - lockdep_set_class(&ni->runlist.lock, &mft_ni_runlist_lock_key); - lockdep_set_class(&ni->mrec_lock, &mft_ni_mrec_lock_key); - - return 0; - -em_put_err_out: - ntfs_error(sb, "Couldn't find first extent of $DATA attribute in " - "attribute list. $MFT is corrupt. Run chkdsk."); -put_err_out: - ntfs_attr_put_search_ctx(ctx); -err_out: - ntfs_error(sb, "Failed. Marking inode as bad."); - make_bad_inode(vi); - ntfs_free(m); - return -1; -} - -static void __ntfs_clear_inode(ntfs_inode *ni) -{ - /* Free all alocated memory. */ - down_write(&ni->runlist.lock); - if (ni->runlist.rl) { - ntfs_free(ni->runlist.rl); - ni->runlist.rl = NULL; - } - up_write(&ni->runlist.lock); - - if (ni->attr_list) { - ntfs_free(ni->attr_list); - ni->attr_list = NULL; - } - - down_write(&ni->attr_list_rl.lock); - if (ni->attr_list_rl.rl) { - ntfs_free(ni->attr_list_rl.rl); - ni->attr_list_rl.rl = NULL; - } - up_write(&ni->attr_list_rl.lock); - - if (ni->name_len && ni->name != I30) { - /* Catch bugs... */ - BUG_ON(!ni->name); - kfree(ni->name); - } -} - -void ntfs_clear_extent_inode(ntfs_inode *ni) -{ - ntfs_debug("Entering for inode 0x%lx.", ni->mft_no); - - BUG_ON(NInoAttr(ni)); - BUG_ON(ni->nr_extents != -1); - -#ifdef NTFS_RW - if (NInoDirty(ni)) { - if (!is_bad_inode(VFS_I(ni->ext.base_ntfs_ino))) - ntfs_error(ni->vol->sb, "Clearing dirty extent inode! " - "Losing data! This is a BUG!!!"); - // FIXME: Do something!!! - } -#endif /* NTFS_RW */ - - __ntfs_clear_inode(ni); - - /* Bye, bye... */ - ntfs_destroy_extent_inode(ni); -} - -/** - * ntfs_evict_big_inode - clean up the ntfs specific part of an inode - * @vi: vfs inode pending annihilation - * - * When the VFS is going to remove an inode from memory, ntfs_clear_big_inode() - * is called, which deallocates all memory belonging to the NTFS specific part - * of the inode and returns. - * - * If the MFT record is dirty, we commit it before doing anything else. - */ -void ntfs_evict_big_inode(struct inode *vi) -{ - ntfs_inode *ni = NTFS_I(vi); - - truncate_inode_pages_final(&vi->i_data); - clear_inode(vi); - -#ifdef NTFS_RW - if (NInoDirty(ni)) { - bool was_bad = (is_bad_inode(vi)); - - /* Committing the inode also commits all extent inodes. */ - ntfs_commit_inode(vi); - - if (!was_bad && (is_bad_inode(vi) || NInoDirty(ni))) { - ntfs_error(vi->i_sb, "Failed to commit dirty inode " - "0x%lx. Losing data!", vi->i_ino); - // FIXME: Do something!!! - } - } -#endif /* NTFS_RW */ - - /* No need to lock at this stage as no one else has a reference. */ - if (ni->nr_extents > 0) { - int i; - - for (i = 0; i < ni->nr_extents; i++) - ntfs_clear_extent_inode(ni->ext.extent_ntfs_inos[i]); - kfree(ni->ext.extent_ntfs_inos); - } - - __ntfs_clear_inode(ni); - - if (NInoAttr(ni)) { - /* Release the base inode if we are holding it. */ - if (ni->nr_extents == -1) { - iput(VFS_I(ni->ext.base_ntfs_ino)); - ni->nr_extents = 0; - ni->ext.base_ntfs_ino = NULL; - } - } - BUG_ON(ni->page); - if (!atomic_dec_and_test(&ni->count)) - BUG(); - return; -} - -/** - * ntfs_show_options - show mount options in /proc/mounts - * @sf: seq_file in which to write our mount options - * @root: root of the mounted tree whose mount options to display - * - * Called by the VFS once for each mounted ntfs volume when someone reads - * /proc/mounts in order to display the NTFS specific mount options of each - * mount. The mount options of fs specified by @root are written to the seq file - * @sf and success is returned. - */ -int ntfs_show_options(struct seq_file *sf, struct dentry *root) -{ - ntfs_volume *vol = NTFS_SB(root->d_sb); - int i; - - seq_printf(sf, ",uid=%i", from_kuid_munged(&init_user_ns, vol->uid)); - seq_printf(sf, ",gid=%i", from_kgid_munged(&init_user_ns, vol->gid)); - if (vol->fmask == vol->dmask) - seq_printf(sf, ",umask=0%o", vol->fmask); - else { - seq_printf(sf, ",fmask=0%o", vol->fmask); - seq_printf(sf, ",dmask=0%o", vol->dmask); - } - seq_printf(sf, ",nls=%s", vol->nls_map->charset); - if (NVolCaseSensitive(vol)) - seq_printf(sf, ",case_sensitive"); - if (NVolShowSystemFiles(vol)) - seq_printf(sf, ",show_sys_files"); - if (!NVolSparseEnabled(vol)) - seq_printf(sf, ",disable_sparse"); - for (i = 0; on_errors_arr[i].val; i++) { - if (on_errors_arr[i].val & vol->on_errors) - seq_printf(sf, ",errors=%s", on_errors_arr[i].str); - } - seq_printf(sf, ",mft_zone_multiplier=%i", vol->mft_zone_multiplier); - return 0; -} - -#ifdef NTFS_RW - -static const char *es = " Leaving inconsistent metadata. Unmount and run " - "chkdsk."; - -/** - * ntfs_truncate - called when the i_size of an ntfs inode is changed - * @vi: inode for which the i_size was changed - * - * We only support i_size changes for normal files at present, i.e. not - * compressed and not encrypted. This is enforced in ntfs_setattr(), see - * below. - * - * The kernel guarantees that @vi is a regular file (S_ISREG() is true) and - * that the change is allowed. - * - * This implies for us that @vi is a file inode rather than a directory, index, - * or attribute inode as well as that @vi is a base inode. - * - * Returns 0 on success or -errno on error. - * - * Called with ->i_mutex held. - */ -int ntfs_truncate(struct inode *vi) -{ - s64 new_size, old_size, nr_freed, new_alloc_size, old_alloc_size; - VCN highest_vcn; - unsigned long flags; - ntfs_inode *base_ni, *ni = NTFS_I(vi); - ntfs_volume *vol = ni->vol; - ntfs_attr_search_ctx *ctx; - MFT_RECORD *m; - ATTR_RECORD *a; - const char *te = " Leaving file length out of sync with i_size."; - int err, mp_size, size_change, alloc_change; - - ntfs_debug("Entering for inode 0x%lx.", vi->i_ino); - BUG_ON(NInoAttr(ni)); - BUG_ON(S_ISDIR(vi->i_mode)); - BUG_ON(NInoMstProtected(ni)); - BUG_ON(ni->nr_extents < 0); -retry_truncate: - /* - * Lock the runlist for writing and map the mft record to ensure it is - * safe to mess with the attribute runlist and sizes. - */ - down_write(&ni->runlist.lock); - if (!NInoAttr(ni)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - m = map_mft_record(base_ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - ntfs_error(vi->i_sb, "Failed to map mft record for inode 0x%lx " - "(error code %d).%s", vi->i_ino, err, te); - ctx = NULL; - m = NULL; - goto old_bad_out; - } - ctx = ntfs_attr_get_search_ctx(base_ni, m); - if (unlikely(!ctx)) { - ntfs_error(vi->i_sb, "Failed to allocate a search context for " - "inode 0x%lx (not enough memory).%s", - vi->i_ino, te); - err = -ENOMEM; - goto old_bad_out; - } - err = ntfs_attr_lookup(ni->type, ni->name, ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - if (err == -ENOENT) { - ntfs_error(vi->i_sb, "Open attribute is missing from " - "mft record. Inode 0x%lx is corrupt. " - "Run chkdsk.%s", vi->i_ino, te); - err = -EIO; - } else - ntfs_error(vi->i_sb, "Failed to lookup attribute in " - "inode 0x%lx (error code %d).%s", - vi->i_ino, err, te); - goto old_bad_out; - } - m = ctx->mrec; - a = ctx->attr; - /* - * The i_size of the vfs inode is the new size for the attribute value. - */ - new_size = i_size_read(vi); - /* The current size of the attribute value is the old size. */ - old_size = ntfs_attr_size(a); - /* Calculate the new allocated size. */ - if (NInoNonResident(ni)) - new_alloc_size = (new_size + vol->cluster_size - 1) & - ~(s64)vol->cluster_size_mask; - else - new_alloc_size = (new_size + 7) & ~7; - /* The current allocated size is the old allocated size. */ - read_lock_irqsave(&ni->size_lock, flags); - old_alloc_size = ni->allocated_size; - read_unlock_irqrestore(&ni->size_lock, flags); - /* - * The change in the file size. This will be 0 if no change, >0 if the - * size is growing, and <0 if the size is shrinking. - */ - size_change = -1; - if (new_size - old_size >= 0) { - size_change = 1; - if (new_size == old_size) - size_change = 0; - } - /* As above for the allocated size. */ - alloc_change = -1; - if (new_alloc_size - old_alloc_size >= 0) { - alloc_change = 1; - if (new_alloc_size == old_alloc_size) - alloc_change = 0; - } - /* - * If neither the size nor the allocation are being changed there is - * nothing to do. - */ - if (!size_change && !alloc_change) - goto unm_done; - /* If the size is changing, check if new size is allowed in $AttrDef. */ - if (size_change) { - err = ntfs_attr_size_bounds_check(vol, ni->type, new_size); - if (unlikely(err)) { - if (err == -ERANGE) { - ntfs_error(vol->sb, "Truncate would cause the " - "inode 0x%lx to %simum size " - "for its attribute type " - "(0x%x). Aborting truncate.", - vi->i_ino, - new_size > old_size ? "exceed " - "the max" : "go under the min", - le32_to_cpu(ni->type)); - err = -EFBIG; - } else { - ntfs_error(vol->sb, "Inode 0x%lx has unknown " - "attribute type 0x%x. " - "Aborting truncate.", - vi->i_ino, - le32_to_cpu(ni->type)); - err = -EIO; - } - /* Reset the vfs inode size to the old size. */ - i_size_write(vi, old_size); - goto err_out; - } - } - if (NInoCompressed(ni) || NInoEncrypted(ni)) { - ntfs_warning(vi->i_sb, "Changes in inode size are not " - "supported yet for %s files, ignoring.", - NInoCompressed(ni) ? "compressed" : - "encrypted"); - err = -EOPNOTSUPP; - goto bad_out; - } - if (a->non_resident) - goto do_non_resident_truncate; - BUG_ON(NInoNonResident(ni)); - /* Resize the attribute record to best fit the new attribute size. */ - if (new_size < vol->mft_record_size && - !ntfs_resident_attr_value_resize(m, a, new_size)) { - /* The resize succeeded! */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - write_lock_irqsave(&ni->size_lock, flags); - /* Update the sizes in the ntfs inode and all is done. */ - ni->allocated_size = le32_to_cpu(a->length) - - le16_to_cpu(a->data.resident.value_offset); - /* - * Note ntfs_resident_attr_value_resize() has already done any - * necessary data clearing in the attribute record. When the - * file is being shrunk vmtruncate() will already have cleared - * the top part of the last partial page, i.e. since this is - * the resident case this is the page with index 0. However, - * when the file is being expanded, the page cache page data - * between the old data_size, i.e. old_size, and the new_size - * has not been zeroed. Fortunately, we do not need to zero it - * either since on one hand it will either already be zero due - * to both read_folio and writepage clearing partial page data - * beyond i_size in which case there is nothing to do or in the - * case of the file being mmap()ped at the same time, POSIX - * specifies that the behaviour is unspecified thus we do not - * have to do anything. This means that in our implementation - * in the rare case that the file is mmap()ped and a write - * occurred into the mmap()ped region just beyond the file size - * and writepage has not yet been called to write out the page - * (which would clear the area beyond the file size) and we now - * extend the file size to incorporate this dirty region - * outside the file size, a write of the page would result in - * this data being written to disk instead of being cleared. - * Given both POSIX and the Linux mmap(2) man page specify that - * this corner case is undefined, we choose to leave it like - * that as this is much simpler for us as we cannot lock the - * relevant page now since we are holding too many ntfs locks - * which would result in a lock reversal deadlock. - */ - ni->initialized_size = new_size; - write_unlock_irqrestore(&ni->size_lock, flags); - goto unm_done; - } - /* If the above resize failed, this must be an attribute extension. */ - BUG_ON(size_change < 0); - /* - * We have to drop all the locks so we can call - * ntfs_attr_make_non_resident(). This could be optimised by try- - * locking the first page cache page and only if that fails dropping - * the locks, locking the page, and redoing all the locking and - * lookups. While this would be a huge optimisation, it is not worth - * it as this is definitely a slow code path as it only ever can happen - * once for any given file. - */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - up_write(&ni->runlist.lock); - /* - * Not enough space in the mft record, try to make the attribute - * non-resident and if successful restart the truncation process. - */ - err = ntfs_attr_make_non_resident(ni, old_size); - if (likely(!err)) - goto retry_truncate; - /* - * Could not make non-resident. If this is due to this not being - * permitted for this attribute type or there not being enough space, - * try to make other attributes non-resident. Otherwise fail. - */ - if (unlikely(err != -EPERM && err != -ENOSPC)) { - ntfs_error(vol->sb, "Cannot truncate inode 0x%lx, attribute " - "type 0x%x, because the conversion from " - "resident to non-resident attribute failed " - "with error code %i.", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), err); - if (err != -ENOMEM) - err = -EIO; - goto conv_err_out; - } - /* TODO: Not implemented from here, abort. */ - if (err == -ENOSPC) - ntfs_error(vol->sb, "Not enough space in the mft record/on " - "disk for the non-resident attribute value. " - "This case is not implemented yet."); - else /* if (err == -EPERM) */ - ntfs_error(vol->sb, "This attribute type may not be " - "non-resident. This case is not implemented " - "yet."); - err = -EOPNOTSUPP; - goto conv_err_out; -#if 0 - // TODO: Attempt to make other attributes non-resident. - if (!err) - goto do_resident_extend; - /* - * Both the attribute list attribute and the standard information - * attribute must remain in the base inode. Thus, if this is one of - * these attributes, we have to try to move other attributes out into - * extent mft records instead. - */ - if (ni->type == AT_ATTRIBUTE_LIST || - ni->type == AT_STANDARD_INFORMATION) { - // TODO: Attempt to move other attributes into extent mft - // records. - err = -EOPNOTSUPP; - if (!err) - goto do_resident_extend; - goto err_out; - } - // TODO: Attempt to move this attribute to an extent mft record, but - // only if it is not already the only attribute in an mft record in - // which case there would be nothing to gain. - err = -EOPNOTSUPP; - if (!err) - goto do_resident_extend; - /* There is nothing we can do to make enough space. )-: */ - goto err_out; -#endif -do_non_resident_truncate: - BUG_ON(!NInoNonResident(ni)); - if (alloc_change < 0) { - highest_vcn = sle64_to_cpu(a->data.non_resident.highest_vcn); - if (highest_vcn > 0 && - old_alloc_size >> vol->cluster_size_bits > - highest_vcn + 1) { - /* - * This attribute has multiple extents. Not yet - * supported. - */ - ntfs_error(vol->sb, "Cannot truncate inode 0x%lx, " - "attribute type 0x%x, because the " - "attribute is highly fragmented (it " - "consists of multiple extents) and " - "this case is not implemented yet.", - vi->i_ino, - (unsigned)le32_to_cpu(ni->type)); - err = -EOPNOTSUPP; - goto bad_out; - } - } - /* - * If the size is shrinking, need to reduce the initialized_size and - * the data_size before reducing the allocation. - */ - if (size_change < 0) { - /* - * Make the valid size smaller (i_size is already up-to-date). - */ - write_lock_irqsave(&ni->size_lock, flags); - if (new_size < ni->initialized_size) { - ni->initialized_size = new_size; - a->data.non_resident.initialized_size = - cpu_to_sle64(new_size); - } - a->data.non_resident.data_size = cpu_to_sle64(new_size); - write_unlock_irqrestore(&ni->size_lock, flags); - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - /* If the allocated size is not changing, we are done. */ - if (!alloc_change) - goto unm_done; - /* - * If the size is shrinking it makes no sense for the - * allocation to be growing. - */ - BUG_ON(alloc_change > 0); - } else /* if (size_change >= 0) */ { - /* - * The file size is growing or staying the same but the - * allocation can be shrinking, growing or staying the same. - */ - if (alloc_change > 0) { - /* - * We need to extend the allocation and possibly update - * the data size. If we are updating the data size, - * since we are not touching the initialized_size we do - * not need to worry about the actual data on disk. - * And as far as the page cache is concerned, there - * will be no pages beyond the old data size and any - * partial region in the last page between the old and - * new data size (or the end of the page if the new - * data size is outside the page) does not need to be - * modified as explained above for the resident - * attribute truncate case. To do this, we simply drop - * the locks we hold and leave all the work to our - * friendly helper ntfs_attr_extend_allocation(). - */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - up_write(&ni->runlist.lock); - err = ntfs_attr_extend_allocation(ni, new_size, - size_change > 0 ? new_size : -1, -1); - /* - * ntfs_attr_extend_allocation() will have done error - * output already. - */ - goto done; - } - if (!alloc_change) - goto alloc_done; - } - /* alloc_change < 0 */ - /* Free the clusters. */ - nr_freed = ntfs_cluster_free(ni, new_alloc_size >> - vol->cluster_size_bits, -1, ctx); - m = ctx->mrec; - a = ctx->attr; - if (unlikely(nr_freed < 0)) { - ntfs_error(vol->sb, "Failed to release cluster(s) (error code " - "%lli). Unmount and run chkdsk to recover " - "the lost cluster(s).", (long long)nr_freed); - NVolSetErrors(vol); - nr_freed = 0; - } - /* Truncate the runlist. */ - err = ntfs_rl_truncate_nolock(vol, &ni->runlist, - new_alloc_size >> vol->cluster_size_bits); - /* - * If the runlist truncation failed and/or the search context is no - * longer valid, we cannot resize the attribute record or build the - * mapping pairs array thus we mark the inode bad so that no access to - * the freed clusters can happen. - */ - if (unlikely(err || IS_ERR(m))) { - ntfs_error(vol->sb, "Failed to %s (error code %li).%s", - IS_ERR(m) ? - "restore attribute search context" : - "truncate attribute runlist", - IS_ERR(m) ? PTR_ERR(m) : err, es); - err = -EIO; - goto bad_out; - } - /* Get the size for the shrunk mapping pairs array for the runlist. */ - mp_size = ntfs_get_size_for_mapping_pairs(vol, ni->runlist.rl, 0, -1); - if (unlikely(mp_size <= 0)) { - ntfs_error(vol->sb, "Cannot shrink allocation of inode 0x%lx, " - "attribute type 0x%x, because determining the " - "size for the mapping pairs failed with error " - "code %i.%s", vi->i_ino, - (unsigned)le32_to_cpu(ni->type), mp_size, es); - err = -EIO; - goto bad_out; - } - /* - * Shrink the attribute record for the new mapping pairs array. Note, - * this cannot fail since we are making the attribute smaller thus by - * definition there is enough space to do so. - */ - err = ntfs_attr_record_resize(m, a, mp_size + - le16_to_cpu(a->data.non_resident.mapping_pairs_offset)); - BUG_ON(err); - /* - * Generate the mapping pairs array directly into the attribute record. - */ - err = ntfs_mapping_pairs_build(vol, (u8*)a + - le16_to_cpu(a->data.non_resident.mapping_pairs_offset), - mp_size, ni->runlist.rl, 0, -1, NULL); - if (unlikely(err)) { - ntfs_error(vol->sb, "Cannot shrink allocation of inode 0x%lx, " - "attribute type 0x%x, because building the " - "mapping pairs failed with error code %i.%s", - vi->i_ino, (unsigned)le32_to_cpu(ni->type), - err, es); - err = -EIO; - goto bad_out; - } - /* Update the allocated/compressed size as well as the highest vcn. */ - a->data.non_resident.highest_vcn = cpu_to_sle64((new_alloc_size >> - vol->cluster_size_bits) - 1); - write_lock_irqsave(&ni->size_lock, flags); - ni->allocated_size = new_alloc_size; - a->data.non_resident.allocated_size = cpu_to_sle64(new_alloc_size); - if (NInoSparse(ni) || NInoCompressed(ni)) { - if (nr_freed) { - ni->itype.compressed.size -= nr_freed << - vol->cluster_size_bits; - BUG_ON(ni->itype.compressed.size < 0); - a->data.non_resident.compressed_size = cpu_to_sle64( - ni->itype.compressed.size); - vi->i_blocks = ni->itype.compressed.size >> 9; - } - } else - vi->i_blocks = new_alloc_size >> 9; - write_unlock_irqrestore(&ni->size_lock, flags); - /* - * We have shrunk the allocation. If this is a shrinking truncate we - * have already dealt with the initialized_size and the data_size above - * and we are done. If the truncate is only changing the allocation - * and not the data_size, we are also done. If this is an extending - * truncate, need to extend the data_size now which is ensured by the - * fact that @size_change is positive. - */ -alloc_done: - /* - * If the size is growing, need to update it now. If it is shrinking, - * we have already updated it above (before the allocation change). - */ - if (size_change > 0) - a->data.non_resident.data_size = cpu_to_sle64(new_size); - /* Ensure the modified mft record is written out. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); -unm_done: - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(base_ni); - up_write(&ni->runlist.lock); -done: - /* Update the mtime and ctime on the base inode. */ - /* normally ->truncate shouldn't update ctime or mtime, - * but ntfs did before so it got a copy & paste version - * of file_update_time. one day someone should fix this - * for real. - */ - if (!IS_NOCMTIME(VFS_I(base_ni)) && !IS_RDONLY(VFS_I(base_ni))) { - struct timespec64 now = current_time(VFS_I(base_ni)); - struct timespec64 ctime = inode_get_ctime(VFS_I(base_ni)); - int sync_it = 0; - - if (!timespec64_equal(&VFS_I(base_ni)->i_mtime, &now) || - !timespec64_equal(&ctime, &now)) - sync_it = 1; - inode_set_ctime_to_ts(VFS_I(base_ni), now); - VFS_I(base_ni)->i_mtime = now; - - if (sync_it) - mark_inode_dirty_sync(VFS_I(base_ni)); - } - - if (likely(!err)) { - NInoClearTruncateFailed(ni); - ntfs_debug("Done."); - } - return err; -old_bad_out: - old_size = -1; -bad_out: - if (err != -ENOMEM && err != -EOPNOTSUPP) - NVolSetErrors(vol); - if (err != -EOPNOTSUPP) - NInoSetTruncateFailed(ni); - else if (old_size >= 0) - i_size_write(vi, old_size); -err_out: - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(base_ni); - up_write(&ni->runlist.lock); -out: - ntfs_debug("Failed. Returning error code %i.", err); - return err; -conv_err_out: - if (err != -ENOMEM && err != -EOPNOTSUPP) - NVolSetErrors(vol); - if (err != -EOPNOTSUPP) - NInoSetTruncateFailed(ni); - else - i_size_write(vi, old_size); - goto out; -} - -/** - * ntfs_truncate_vfs - wrapper for ntfs_truncate() that has no return value - * @vi: inode for which the i_size was changed - * - * Wrapper for ntfs_truncate() that has no return value. - * - * See ntfs_truncate() description above for details. - */ -#ifdef NTFS_RW -void ntfs_truncate_vfs(struct inode *vi) { - ntfs_truncate(vi); -} -#endif - -/** - * ntfs_setattr - called from notify_change() when an attribute is being changed - * @idmap: idmap of the mount the inode was found from - * @dentry: dentry whose attributes to change - * @attr: structure describing the attributes and the changes - * - * We have to trap VFS attempts to truncate the file described by @dentry as - * soon as possible, because we do not implement changes in i_size yet. So we - * abort all i_size changes here. - * - * We also abort all changes of user, group, and mode as we do not implement - * the NTFS ACLs yet. - * - * Called with ->i_mutex held. - */ -int ntfs_setattr(struct mnt_idmap *idmap, struct dentry *dentry, - struct iattr *attr) -{ - struct inode *vi = d_inode(dentry); - int err; - unsigned int ia_valid = attr->ia_valid; - - err = setattr_prepare(&nop_mnt_idmap, dentry, attr); - if (err) - goto out; - /* We do not support NTFS ACLs yet. */ - if (ia_valid & (ATTR_UID | ATTR_GID | ATTR_MODE)) { - ntfs_warning(vi->i_sb, "Changes in user/group/mode are not " - "supported yet, ignoring."); - err = -EOPNOTSUPP; - goto out; - } - if (ia_valid & ATTR_SIZE) { - if (attr->ia_size != i_size_read(vi)) { - ntfs_inode *ni = NTFS_I(vi); - /* - * FIXME: For now we do not support resizing of - * compressed or encrypted files yet. - */ - if (NInoCompressed(ni) || NInoEncrypted(ni)) { - ntfs_warning(vi->i_sb, "Changes in inode size " - "are not supported yet for " - "%s files, ignoring.", - NInoCompressed(ni) ? - "compressed" : "encrypted"); - err = -EOPNOTSUPP; - } else { - truncate_setsize(vi, attr->ia_size); - ntfs_truncate_vfs(vi); - } - if (err || ia_valid == ATTR_SIZE) - goto out; - } else { - /* - * We skipped the truncate but must still update - * timestamps. - */ - ia_valid |= ATTR_MTIME | ATTR_CTIME; - } - } - if (ia_valid & ATTR_ATIME) - vi->i_atime = attr->ia_atime; - if (ia_valid & ATTR_MTIME) - vi->i_mtime = attr->ia_mtime; - if (ia_valid & ATTR_CTIME) - inode_set_ctime_to_ts(vi, attr->ia_ctime); - mark_inode_dirty(vi); -out: - return err; -} - -/** - * __ntfs_write_inode - write out a dirty inode - * @vi: inode to write out - * @sync: if true, write out synchronously - * - * Write out a dirty inode to disk including any extent inodes if present. - * - * If @sync is true, commit the inode to disk and wait for io completion. This - * is done using write_mft_record(). - * - * If @sync is false, just schedule the write to happen but do not wait for i/o - * completion. In 2.6 kernels, scheduling usually happens just by virtue of - * marking the page (and in this case mft record) dirty but we do not implement - * this yet as write_mft_record() largely ignores the @sync parameter and - * always performs synchronous writes. - * - * Return 0 on success and -errno on error. - */ -int __ntfs_write_inode(struct inode *vi, int sync) -{ - sle64 nt; - ntfs_inode *ni = NTFS_I(vi); - ntfs_attr_search_ctx *ctx; - MFT_RECORD *m; - STANDARD_INFORMATION *si; - int err = 0; - bool modified = false; - - ntfs_debug("Entering for %sinode 0x%lx.", NInoAttr(ni) ? "attr " : "", - vi->i_ino); - /* - * Dirty attribute inodes are written via their real inodes so just - * clean them here. Access time updates are taken care off when the - * real inode is written. - */ - if (NInoAttr(ni)) { - NInoClearDirty(ni); - ntfs_debug("Done."); - return 0; - } - /* Map, pin, and lock the mft record belonging to the inode. */ - m = map_mft_record(ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - goto err_out; - } - /* Update the access times in the standard information attribute. */ - ctx = ntfs_attr_get_search_ctx(ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto unm_err_out; - } - err = ntfs_attr_lookup(AT_STANDARD_INFORMATION, NULL, 0, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - ntfs_attr_put_search_ctx(ctx); - goto unm_err_out; - } - si = (STANDARD_INFORMATION*)((u8*)ctx->attr + - le16_to_cpu(ctx->attr->data.resident.value_offset)); - /* Update the access times if they have changed. */ - nt = utc2ntfs(vi->i_mtime); - if (si->last_data_change_time != nt) { - ntfs_debug("Updating mtime for inode 0x%lx: old = 0x%llx, " - "new = 0x%llx", vi->i_ino, (long long) - sle64_to_cpu(si->last_data_change_time), - (long long)sle64_to_cpu(nt)); - si->last_data_change_time = nt; - modified = true; - } - nt = utc2ntfs(inode_get_ctime(vi)); - if (si->last_mft_change_time != nt) { - ntfs_debug("Updating ctime for inode 0x%lx: old = 0x%llx, " - "new = 0x%llx", vi->i_ino, (long long) - sle64_to_cpu(si->last_mft_change_time), - (long long)sle64_to_cpu(nt)); - si->last_mft_change_time = nt; - modified = true; - } - nt = utc2ntfs(vi->i_atime); - if (si->last_access_time != nt) { - ntfs_debug("Updating atime for inode 0x%lx: old = 0x%llx, " - "new = 0x%llx", vi->i_ino, - (long long)sle64_to_cpu(si->last_access_time), - (long long)sle64_to_cpu(nt)); - si->last_access_time = nt; - modified = true; - } - /* - * If we just modified the standard information attribute we need to - * mark the mft record it is in dirty. We do this manually so that - * mark_inode_dirty() is not called which would redirty the inode and - * hence result in an infinite loop of trying to write the inode. - * There is no need to mark the base inode nor the base mft record - * dirty, since we are going to write this mft record below in any case - * and the base mft record may actually not have been modified so it - * might not need to be written out. - * NOTE: It is not a problem when the inode for $MFT itself is being - * written out as mark_ntfs_record_dirty() will only set I_DIRTY_PAGES - * on the $MFT inode and hence __ntfs_write_inode() will not be - * re-invoked because of it which in turn is ok since the dirtied mft - * record will be cleaned and written out to disk below, i.e. before - * this function returns. - */ - if (modified) { - flush_dcache_mft_record_page(ctx->ntfs_ino); - if (!NInoTestSetDirty(ctx->ntfs_ino)) - mark_ntfs_record_dirty(ctx->ntfs_ino->page, - ctx->ntfs_ino->page_ofs); - } - ntfs_attr_put_search_ctx(ctx); - /* Now the access times are updated, write the base mft record. */ - if (NInoDirty(ni)) - err = write_mft_record(ni, m, sync); - /* Write all attached extent mft records. */ - mutex_lock(&ni->extent_lock); - if (ni->nr_extents > 0) { - ntfs_inode **extent_nis = ni->ext.extent_ntfs_inos; - int i; - - ntfs_debug("Writing %i extent inodes.", ni->nr_extents); - for (i = 0; i < ni->nr_extents; i++) { - ntfs_inode *tni = extent_nis[i]; - - if (NInoDirty(tni)) { - MFT_RECORD *tm = map_mft_record(tni); - int ret; - - if (IS_ERR(tm)) { - if (!err || err == -ENOMEM) - err = PTR_ERR(tm); - continue; - } - ret = write_mft_record(tni, tm, sync); - unmap_mft_record(tni); - if (unlikely(ret)) { - if (!err || err == -ENOMEM) - err = ret; - } - } - } - } - mutex_unlock(&ni->extent_lock); - unmap_mft_record(ni); - if (unlikely(err)) - goto err_out; - ntfs_debug("Done."); - return 0; -unm_err_out: - unmap_mft_record(ni); -err_out: - if (err == -ENOMEM) { - ntfs_warning(vi->i_sb, "Not enough memory to write inode. " - "Marking the inode dirty again, so the VFS " - "retries later."); - mark_inode_dirty(vi); - } else { - ntfs_error(vi->i_sb, "Failed (error %i): Run chkdsk.", -err); - NVolSetErrors(ni->vol); - } - return err; -} - -#endif /* NTFS_RW */ diff --git a/fs/ntfs/inode.h b/fs/ntfs/inode.h deleted file mode 100644 index 147ef4ddb691..000000000000 --- a/fs/ntfs/inode.h +++ /dev/null @@ -1,310 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * inode.h - Defines for inode structures NTFS Linux kernel driver. Part of - * the Linux-NTFS project. - * - * Copyright (c) 2001-2007 Anton Altaparmakov - * Copyright (c) 2002 Richard Russon - */ - -#ifndef _LINUX_NTFS_INODE_H -#define _LINUX_NTFS_INODE_H - -#include - -#include -#include -#include -#include -#include - -#include "layout.h" -#include "volume.h" -#include "types.h" -#include "runlist.h" -#include "debug.h" - -typedef struct _ntfs_inode ntfs_inode; - -/* - * The NTFS in-memory inode structure. It is just used as an extension to the - * fields already provided in the VFS inode. - */ -struct _ntfs_inode { - rwlock_t size_lock; /* Lock serializing access to inode sizes. */ - s64 initialized_size; /* Copy from the attribute record. */ - s64 allocated_size; /* Copy from the attribute record. */ - unsigned long state; /* NTFS specific flags describing this inode. - See ntfs_inode_state_bits below. */ - unsigned long mft_no; /* Number of the mft record / inode. */ - u16 seq_no; /* Sequence number of the mft record. */ - atomic_t count; /* Inode reference count for book keeping. */ - ntfs_volume *vol; /* Pointer to the ntfs volume of this inode. */ - /* - * If NInoAttr() is true, the below fields describe the attribute which - * this fake inode belongs to. The actual inode of this attribute is - * pointed to by base_ntfs_ino and nr_extents is always set to -1 (see - * below). For real inodes, we also set the type (AT_DATA for files and - * AT_INDEX_ALLOCATION for directories), with the name = NULL and - * name_len = 0 for files and name = I30 (global constant) and - * name_len = 4 for directories. - */ - ATTR_TYPE type; /* Attribute type of this fake inode. */ - ntfschar *name; /* Attribute name of this fake inode. */ - u32 name_len; /* Attribute name length of this fake inode. */ - runlist runlist; /* If state has the NI_NonResident bit set, - the runlist of the unnamed data attribute - (if a file) or of the index allocation - attribute (directory) or of the attribute - described by the fake inode (if NInoAttr()). - If runlist.rl is NULL, the runlist has not - been read in yet or has been unmapped. If - NI_NonResident is clear, the attribute is - resident (file and fake inode) or there is - no $I30 index allocation attribute - (small directory). In the latter case - runlist.rl is always NULL.*/ - /* - * The following fields are only valid for real inodes and extent - * inodes. - */ - struct mutex mrec_lock; /* Lock for serializing access to the - mft record belonging to this inode. */ - struct page *page; /* The page containing the mft record of the - inode. This should only be touched by the - (un)map_mft_record*() functions. */ - int page_ofs; /* Offset into the page at which the mft record - begins. This should only be touched by the - (un)map_mft_record*() functions. */ - /* - * Attribute list support (only for use by the attribute lookup - * functions). Setup during read_inode for all inodes with attribute - * lists. Only valid if NI_AttrList is set in state, and attr_list_rl is - * further only valid if NI_AttrListNonResident is set. - */ - u32 attr_list_size; /* Length of attribute list value in bytes. */ - u8 *attr_list; /* Attribute list value itself. */ - runlist attr_list_rl; /* Run list for the attribute list value. */ - union { - struct { /* It is a directory, $MFT, or an index inode. */ - u32 block_size; /* Size of an index block. */ - u32 vcn_size; /* Size of a vcn in this - index. */ - COLLATION_RULE collation_rule; /* The collation rule - for the index. */ - u8 block_size_bits; /* Log2 of the above. */ - u8 vcn_size_bits; /* Log2 of the above. */ - } index; - struct { /* It is a compressed/sparse file/attribute inode. */ - s64 size; /* Copy of compressed_size from - $DATA. */ - u32 block_size; /* Size of a compression block - (cb). */ - u8 block_size_bits; /* Log2 of the size of a cb. */ - u8 block_clusters; /* Number of clusters per cb. */ - } compressed; - } itype; - struct mutex extent_lock; /* Lock for accessing/modifying the - below . */ - s32 nr_extents; /* For a base mft record, the number of attached extent - inodes (0 if none), for extent records and for fake - inodes describing an attribute this is -1. */ - union { /* This union is only used if nr_extents != 0. */ - ntfs_inode **extent_ntfs_inos; /* For nr_extents > 0, array of - the ntfs inodes of the extent - mft records belonging to - this base inode which have - been loaded. */ - ntfs_inode *base_ntfs_ino; /* For nr_extents == -1, the - ntfs inode of the base mft - record. For fake inodes, the - real (base) inode to which - the attribute belongs. */ - } ext; -}; - -/* - * Defined bits for the state field in the ntfs_inode structure. - * (f) = files only, (d) = directories only, (a) = attributes/fake inodes only - */ -typedef enum { - NI_Dirty, /* 1: Mft record needs to be written to disk. */ - NI_AttrList, /* 1: Mft record contains an attribute list. */ - NI_AttrListNonResident, /* 1: Attribute list is non-resident. Implies - NI_AttrList is set. */ - - NI_Attr, /* 1: Fake inode for attribute i/o. - 0: Real inode or extent inode. */ - - NI_MstProtected, /* 1: Attribute is protected by MST fixups. - 0: Attribute is not protected by fixups. */ - NI_NonResident, /* 1: Unnamed data attr is non-resident (f). - 1: Attribute is non-resident (a). */ - NI_IndexAllocPresent = NI_NonResident, /* 1: $I30 index alloc attr is - present (d). */ - NI_Compressed, /* 1: Unnamed data attr is compressed (f). - 1: Create compressed files by default (d). - 1: Attribute is compressed (a). */ - NI_Encrypted, /* 1: Unnamed data attr is encrypted (f). - 1: Create encrypted files by default (d). - 1: Attribute is encrypted (a). */ - NI_Sparse, /* 1: Unnamed data attr is sparse (f). - 1: Create sparse files by default (d). - 1: Attribute is sparse (a). */ - NI_SparseDisabled, /* 1: May not create sparse regions. */ - NI_TruncateFailed, /* 1: Last ntfs_truncate() call failed. */ -} ntfs_inode_state_bits; - -/* - * NOTE: We should be adding dirty mft records to a list somewhere and they - * should be independent of the (ntfs/vfs) inode structure so that an inode can - * be removed but the record can be left dirty for syncing later. - */ - -/* - * Macro tricks to expand the NInoFoo(), NInoSetFoo(), and NInoClearFoo() - * functions. - */ -#define NINO_FNS(flag) \ -static inline int NIno##flag(ntfs_inode *ni) \ -{ \ - return test_bit(NI_##flag, &(ni)->state); \ -} \ -static inline void NInoSet##flag(ntfs_inode *ni) \ -{ \ - set_bit(NI_##flag, &(ni)->state); \ -} \ -static inline void NInoClear##flag(ntfs_inode *ni) \ -{ \ - clear_bit(NI_##flag, &(ni)->state); \ -} - -/* - * As above for NInoTestSetFoo() and NInoTestClearFoo(). - */ -#define TAS_NINO_FNS(flag) \ -static inline int NInoTestSet##flag(ntfs_inode *ni) \ -{ \ - return test_and_set_bit(NI_##flag, &(ni)->state); \ -} \ -static inline int NInoTestClear##flag(ntfs_inode *ni) \ -{ \ - return test_and_clear_bit(NI_##flag, &(ni)->state); \ -} - -/* Emit the ntfs inode bitops functions. */ -NINO_FNS(Dirty) -TAS_NINO_FNS(Dirty) -NINO_FNS(AttrList) -NINO_FNS(AttrListNonResident) -NINO_FNS(Attr) -NINO_FNS(MstProtected) -NINO_FNS(NonResident) -NINO_FNS(IndexAllocPresent) -NINO_FNS(Compressed) -NINO_FNS(Encrypted) -NINO_FNS(Sparse) -NINO_FNS(SparseDisabled) -NINO_FNS(TruncateFailed) - -/* - * The full structure containing a ntfs_inode and a vfs struct inode. Used for - * all real and fake inodes but not for extent inodes which lack the vfs struct - * inode. - */ -typedef struct { - ntfs_inode ntfs_inode; - struct inode vfs_inode; /* The vfs inode structure. */ -} big_ntfs_inode; - -/** - * NTFS_I - return the ntfs inode given a vfs inode - * @inode: VFS inode - * - * NTFS_I() returns the ntfs inode associated with the VFS @inode. - */ -static inline ntfs_inode *NTFS_I(struct inode *inode) -{ - return (ntfs_inode *)container_of(inode, big_ntfs_inode, vfs_inode); -} - -static inline struct inode *VFS_I(ntfs_inode *ni) -{ - return &((big_ntfs_inode *)ni)->vfs_inode; -} - -/** - * ntfs_attr - ntfs in memory attribute structure - * @mft_no: mft record number of the base mft record of this attribute - * @name: Unicode name of the attribute (NULL if unnamed) - * @name_len: length of @name in Unicode characters (0 if unnamed) - * @type: attribute type (see layout.h) - * - * This structure exists only to provide a small structure for the - * ntfs_{attr_}iget()/ntfs_test_inode()/ntfs_init_locked_inode() mechanism. - * - * NOTE: Elements are ordered by size to make the structure as compact as - * possible on all architectures. - */ -typedef struct { - unsigned long mft_no; - ntfschar *name; - u32 name_len; - ATTR_TYPE type; -} ntfs_attr; - -extern int ntfs_test_inode(struct inode *vi, void *data); - -extern struct inode *ntfs_iget(struct super_block *sb, unsigned long mft_no); -extern struct inode *ntfs_attr_iget(struct inode *base_vi, ATTR_TYPE type, - ntfschar *name, u32 name_len); -extern struct inode *ntfs_index_iget(struct inode *base_vi, ntfschar *name, - u32 name_len); - -extern struct inode *ntfs_alloc_big_inode(struct super_block *sb); -extern void ntfs_free_big_inode(struct inode *inode); -extern void ntfs_evict_big_inode(struct inode *vi); - -extern void __ntfs_init_inode(struct super_block *sb, ntfs_inode *ni); - -static inline void ntfs_init_big_inode(struct inode *vi) -{ - ntfs_inode *ni = NTFS_I(vi); - - ntfs_debug("Entering."); - __ntfs_init_inode(vi->i_sb, ni); - ni->mft_no = vi->i_ino; -} - -extern ntfs_inode *ntfs_new_extent_inode(struct super_block *sb, - unsigned long mft_no); -extern void ntfs_clear_extent_inode(ntfs_inode *ni); - -extern int ntfs_read_inode_mount(struct inode *vi); - -extern int ntfs_show_options(struct seq_file *sf, struct dentry *root); - -#ifdef NTFS_RW - -extern int ntfs_truncate(struct inode *vi); -extern void ntfs_truncate_vfs(struct inode *vi); - -extern int ntfs_setattr(struct mnt_idmap *idmap, - struct dentry *dentry, struct iattr *attr); - -extern int __ntfs_write_inode(struct inode *vi, int sync); - -static inline void ntfs_commit_inode(struct inode *vi) -{ - if (!is_bad_inode(vi)) - __ntfs_write_inode(vi, 1); - return; -} - -#else - -static inline void ntfs_truncate_vfs(struct inode *vi) {} - -#endif /* NTFS_RW */ - -#endif /* _LINUX_NTFS_INODE_H */ diff --git a/fs/ntfs/layout.h b/fs/ntfs/layout.h deleted file mode 100644 index 5d4bf7a3259f..000000000000 --- a/fs/ntfs/layout.h +++ /dev/null @@ -1,2421 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * layout.h - All NTFS associated on-disk structures. Part of the Linux-NTFS - * project. - * - * Copyright (c) 2001-2005 Anton Altaparmakov - * Copyright (c) 2002 Richard Russon - */ - -#ifndef _LINUX_NTFS_LAYOUT_H -#define _LINUX_NTFS_LAYOUT_H - -#include -#include -#include -#include - -#include "types.h" - -/* The NTFS oem_id "NTFS " */ -#define magicNTFS cpu_to_le64(0x202020205346544eULL) - -/* - * Location of bootsector on partition: - * The standard NTFS_BOOT_SECTOR is on sector 0 of the partition. - * On NT4 and above there is one backup copy of the boot sector to - * be found on the last sector of the partition (not normally accessible - * from within Windows as the bootsector contained number of sectors - * value is one less than the actual value!). - * On versions of NT 3.51 and earlier, the backup copy was located at - * number of sectors/2 (integer divide), i.e. in the middle of the volume. - */ - -/* - * BIOS parameter block (bpb) structure. - */ -typedef struct { - le16 bytes_per_sector; /* Size of a sector in bytes. */ - u8 sectors_per_cluster; /* Size of a cluster in sectors. */ - le16 reserved_sectors; /* zero */ - u8 fats; /* zero */ - le16 root_entries; /* zero */ - le16 sectors; /* zero */ - u8 media_type; /* 0xf8 = hard disk */ - le16 sectors_per_fat; /* zero */ - le16 sectors_per_track; /* irrelevant */ - le16 heads; /* irrelevant */ - le32 hidden_sectors; /* zero */ - le32 large_sectors; /* zero */ -} __attribute__ ((__packed__)) BIOS_PARAMETER_BLOCK; - -/* - * NTFS boot sector structure. - */ -typedef struct { - u8 jump[3]; /* Irrelevant (jump to boot up code).*/ - le64 oem_id; /* Magic "NTFS ". */ - BIOS_PARAMETER_BLOCK bpb; /* See BIOS_PARAMETER_BLOCK. */ - u8 unused[4]; /* zero, NTFS diskedit.exe states that - this is actually: - __u8 physical_drive; // 0x80 - __u8 current_head; // zero - __u8 extended_boot_signature; - // 0x80 - __u8 unused; // zero - */ -/*0x28*/sle64 number_of_sectors; /* Number of sectors in volume. Gives - maximum volume size of 2^63 sectors. - Assuming standard sector size of 512 - bytes, the maximum byte size is - approx. 4.7x10^21 bytes. (-; */ - sle64 mft_lcn; /* Cluster location of mft data. */ - sle64 mftmirr_lcn; /* Cluster location of copy of mft. */ - s8 clusters_per_mft_record; /* Mft record size in clusters. */ - u8 reserved0[3]; /* zero */ - s8 clusters_per_index_record; /* Index block size in clusters. */ - u8 reserved1[3]; /* zero */ - le64 volume_serial_number; /* Irrelevant (serial number). */ - le32 checksum; /* Boot sector checksum. */ -/*0x54*/u8 bootstrap[426]; /* Irrelevant (boot up code). */ - le16 end_of_sector_marker; /* End of bootsector magic. Always is - 0xaa55 in little endian. */ -/* sizeof() = 512 (0x200) bytes */ -} __attribute__ ((__packed__)) NTFS_BOOT_SECTOR; - -/* - * Magic identifiers present at the beginning of all ntfs record containing - * records (like mft records for example). - */ -enum { - /* Found in $MFT/$DATA. */ - magic_FILE = cpu_to_le32(0x454c4946), /* Mft entry. */ - magic_INDX = cpu_to_le32(0x58444e49), /* Index buffer. */ - magic_HOLE = cpu_to_le32(0x454c4f48), /* ? (NTFS 3.0+?) */ - - /* Found in $LogFile/$DATA. */ - magic_RSTR = cpu_to_le32(0x52545352), /* Restart page. */ - magic_RCRD = cpu_to_le32(0x44524352), /* Log record page. */ - - /* Found in $LogFile/$DATA. (May be found in $MFT/$DATA, also?) */ - magic_CHKD = cpu_to_le32(0x444b4843), /* Modified by chkdsk. */ - - /* Found in all ntfs record containing records. */ - magic_BAAD = cpu_to_le32(0x44414142), /* Failed multi sector - transfer was detected. */ - /* - * Found in $LogFile/$DATA when a page is full of 0xff bytes and is - * thus not initialized. Page must be initialized before using it. - */ - magic_empty = cpu_to_le32(0xffffffff) /* Record is empty. */ -}; - -typedef le32 NTFS_RECORD_TYPE; - -/* - * Generic magic comparison macros. Finally found a use for the ## preprocessor - * operator! (-8 - */ - -static inline bool __ntfs_is_magic(le32 x, NTFS_RECORD_TYPE r) -{ - return (x == r); -} -#define ntfs_is_magic(x, m) __ntfs_is_magic(x, magic_##m) - -static inline bool __ntfs_is_magicp(le32 *p, NTFS_RECORD_TYPE r) -{ - return (*p == r); -} -#define ntfs_is_magicp(p, m) __ntfs_is_magicp(p, magic_##m) - -/* - * Specialised magic comparison macros for the NTFS_RECORD_TYPEs defined above. - */ -#define ntfs_is_file_record(x) ( ntfs_is_magic (x, FILE) ) -#define ntfs_is_file_recordp(p) ( ntfs_is_magicp(p, FILE) ) -#define ntfs_is_mft_record(x) ( ntfs_is_file_record (x) ) -#define ntfs_is_mft_recordp(p) ( ntfs_is_file_recordp(p) ) -#define ntfs_is_indx_record(x) ( ntfs_is_magic (x, INDX) ) -#define ntfs_is_indx_recordp(p) ( ntfs_is_magicp(p, INDX) ) -#define ntfs_is_hole_record(x) ( ntfs_is_magic (x, HOLE) ) -#define ntfs_is_hole_recordp(p) ( ntfs_is_magicp(p, HOLE) ) - -#define ntfs_is_rstr_record(x) ( ntfs_is_magic (x, RSTR) ) -#define ntfs_is_rstr_recordp(p) ( ntfs_is_magicp(p, RSTR) ) -#define ntfs_is_rcrd_record(x) ( ntfs_is_magic (x, RCRD) ) -#define ntfs_is_rcrd_recordp(p) ( ntfs_is_magicp(p, RCRD) ) - -#define ntfs_is_chkd_record(x) ( ntfs_is_magic (x, CHKD) ) -#define ntfs_is_chkd_recordp(p) ( ntfs_is_magicp(p, CHKD) ) - -#define ntfs_is_baad_record(x) ( ntfs_is_magic (x, BAAD) ) -#define ntfs_is_baad_recordp(p) ( ntfs_is_magicp(p, BAAD) ) - -#define ntfs_is_empty_record(x) ( ntfs_is_magic (x, empty) ) -#define ntfs_is_empty_recordp(p) ( ntfs_is_magicp(p, empty) ) - -/* - * The Update Sequence Array (usa) is an array of the le16 values which belong - * to the end of each sector protected by the update sequence record in which - * this array is contained. Note that the first entry is the Update Sequence - * Number (usn), a cyclic counter of how many times the protected record has - * been written to disk. The values 0 and -1 (ie. 0xffff) are not used. All - * last le16's of each sector have to be equal to the usn (during reading) or - * are set to it (during writing). If they are not, an incomplete multi sector - * transfer has occurred when the data was written. - * The maximum size for the update sequence array is fixed to: - * maximum size = usa_ofs + (usa_count * 2) = 510 bytes - * The 510 bytes comes from the fact that the last le16 in the array has to - * (obviously) finish before the last le16 of the first 512-byte sector. - * This formula can be used as a consistency check in that usa_ofs + - * (usa_count * 2) has to be less than or equal to 510. - */ -typedef struct { - NTFS_RECORD_TYPE magic; /* A four-byte magic identifying the record - type and/or status. */ - le16 usa_ofs; /* Offset to the Update Sequence Array (usa) - from the start of the ntfs record. */ - le16 usa_count; /* Number of le16 sized entries in the usa - including the Update Sequence Number (usn), - thus the number of fixups is the usa_count - minus 1. */ -} __attribute__ ((__packed__)) NTFS_RECORD; - -/* - * System files mft record numbers. All these files are always marked as used - * in the bitmap attribute of the mft; presumably in order to avoid accidental - * allocation for random other mft records. Also, the sequence number for each - * of the system files is always equal to their mft record number and it is - * never modified. - */ -typedef enum { - FILE_MFT = 0, /* Master file table (mft). Data attribute - contains the entries and bitmap attribute - records which ones are in use (bit==1). */ - FILE_MFTMirr = 1, /* Mft mirror: copy of first four mft records - in data attribute. If cluster size > 4kiB, - copy of first N mft records, with - N = cluster_size / mft_record_size. */ - FILE_LogFile = 2, /* Journalling log in data attribute. */ - FILE_Volume = 3, /* Volume name attribute and volume information - attribute (flags and ntfs version). Windows - refers to this file as volume DASD (Direct - Access Storage Device). */ - FILE_AttrDef = 4, /* Array of attribute definitions in data - attribute. */ - FILE_root = 5, /* Root directory. */ - FILE_Bitmap = 6, /* Allocation bitmap of all clusters (lcns) in - data attribute. */ - FILE_Boot = 7, /* Boot sector (always at cluster 0) in data - attribute. */ - FILE_BadClus = 8, /* Contains all bad clusters in the non-resident - data attribute. */ - FILE_Secure = 9, /* Shared security descriptors in data attribute - and two indexes into the descriptors. - Appeared in Windows 2000. Before that, this - file was named $Quota but was unused. */ - FILE_UpCase = 10, /* Uppercase equivalents of all 65536 Unicode - characters in data attribute. */ - FILE_Extend = 11, /* Directory containing other system files (eg. - $ObjId, $Quota, $Reparse and $UsnJrnl). This - is new to NTFS3.0. */ - FILE_reserved12 = 12, /* Reserved for future use (records 12-15). */ - FILE_reserved13 = 13, - FILE_reserved14 = 14, - FILE_reserved15 = 15, - FILE_first_user = 16, /* First user file, used as test limit for - whether to allow opening a file or not. */ -} NTFS_SYSTEM_FILES; - -/* - * These are the so far known MFT_RECORD_* flags (16-bit) which contain - * information about the mft record in which they are present. - */ -enum { - MFT_RECORD_IN_USE = cpu_to_le16(0x0001), - MFT_RECORD_IS_DIRECTORY = cpu_to_le16(0x0002), -} __attribute__ ((__packed__)); - -typedef le16 MFT_RECORD_FLAGS; - -/* - * mft references (aka file references or file record segment references) are - * used whenever a structure needs to refer to a record in the mft. - * - * A reference consists of a 48-bit index into the mft and a 16-bit sequence - * number used to detect stale references. - * - * For error reporting purposes we treat the 48-bit index as a signed quantity. - * - * The sequence number is a circular counter (skipping 0) describing how many - * times the referenced mft record has been (re)used. This has to match the - * sequence number of the mft record being referenced, otherwise the reference - * is considered stale and removed (FIXME: only ntfsck or the driver itself?). - * - * If the sequence number is zero it is assumed that no sequence number - * consistency checking should be performed. - * - * FIXME: Since inodes are 32-bit as of now, the driver needs to always check - * for high_part being 0 and if not either BUG(), cause a panic() or handle - * the situation in some other way. This shouldn't be a problem as a volume has - * to become HUGE in order to need more than 32-bits worth of mft records. - * Assuming the standard mft record size of 1kb only the records (never mind - * the non-resident attributes, etc.) would require 4Tb of space on their own - * for the first 32 bits worth of records. This is only if some strange person - * doesn't decide to foul play and make the mft sparse which would be a really - * horrible thing to do as it would trash our current driver implementation. )-: - * Do I hear screams "we want 64-bit inodes!" ?!? (-; - * - * FIXME: The mft zone is defined as the first 12% of the volume. This space is - * reserved so that the mft can grow contiguously and hence doesn't become - * fragmented. Volume free space includes the empty part of the mft zone and - * when the volume's free 88% are used up, the mft zone is shrunk by a factor - * of 2, thus making more space available for more files/data. This process is - * repeated every time there is no more free space except for the mft zone until - * there really is no more free space. - */ - -/* - * Typedef the MFT_REF as a 64-bit value for easier handling. - * Also define two unpacking macros to get to the reference (MREF) and - * sequence number (MSEQNO) respectively. - * The _LE versions are to be applied on little endian MFT_REFs. - * Note: The _LE versions will return a CPU endian formatted value! - */ -#define MFT_REF_MASK_CPU 0x0000ffffffffffffULL -#define MFT_REF_MASK_LE cpu_to_le64(MFT_REF_MASK_CPU) - -typedef u64 MFT_REF; -typedef le64 leMFT_REF; - -#define MK_MREF(m, s) ((MFT_REF)(((MFT_REF)(s) << 48) | \ - ((MFT_REF)(m) & MFT_REF_MASK_CPU))) -#define MK_LE_MREF(m, s) cpu_to_le64(MK_MREF(m, s)) - -#define MREF(x) ((unsigned long)((x) & MFT_REF_MASK_CPU)) -#define MSEQNO(x) ((u16)(((x) >> 48) & 0xffff)) -#define MREF_LE(x) ((unsigned long)(le64_to_cpu(x) & MFT_REF_MASK_CPU)) -#define MSEQNO_LE(x) ((u16)((le64_to_cpu(x) >> 48) & 0xffff)) - -#define IS_ERR_MREF(x) (((x) & 0x0000800000000000ULL) ? true : false) -#define ERR_MREF(x) ((u64)((s64)(x))) -#define MREF_ERR(x) ((int)((s64)(x))) - -/* - * The mft record header present at the beginning of every record in the mft. - * This is followed by a sequence of variable length attribute records which - * is terminated by an attribute of type AT_END which is a truncated attribute - * in that it only consists of the attribute type code AT_END and none of the - * other members of the attribute structure are present. - */ -typedef struct { -/*Ofs*/ -/* 0 NTFS_RECORD; -- Unfolded here as gcc doesn't like unnamed structs. */ - NTFS_RECORD_TYPE magic; /* Usually the magic is "FILE". */ - le16 usa_ofs; /* See NTFS_RECORD definition above. */ - le16 usa_count; /* See NTFS_RECORD definition above. */ - -/* 8*/ le64 lsn; /* $LogFile sequence number for this record. - Changed every time the record is modified. */ -/* 16*/ le16 sequence_number; /* Number of times this mft record has been - reused. (See description for MFT_REF - above.) NOTE: The increment (skipping zero) - is done when the file is deleted. NOTE: If - this is zero it is left zero. */ -/* 18*/ le16 link_count; /* Number of hard links, i.e. the number of - directory entries referencing this record. - NOTE: Only used in mft base records. - NOTE: When deleting a directory entry we - check the link_count and if it is 1 we - delete the file. Otherwise we delete the - FILE_NAME_ATTR being referenced by the - directory entry from the mft record and - decrement the link_count. - FIXME: Careful with Win32 + DOS names! */ -/* 20*/ le16 attrs_offset; /* Byte offset to the first attribute in this - mft record from the start of the mft record. - NOTE: Must be aligned to 8-byte boundary. */ -/* 22*/ MFT_RECORD_FLAGS flags; /* Bit array of MFT_RECORD_FLAGS. When a file - is deleted, the MFT_RECORD_IN_USE flag is - set to zero. */ -/* 24*/ le32 bytes_in_use; /* Number of bytes used in this mft record. - NOTE: Must be aligned to 8-byte boundary. */ -/* 28*/ le32 bytes_allocated; /* Number of bytes allocated for this mft - record. This should be equal to the mft - record size. */ -/* 32*/ leMFT_REF base_mft_record;/* This is zero for base mft records. - When it is not zero it is a mft reference - pointing to the base mft record to which - this record belongs (this is then used to - locate the attribute list attribute present - in the base record which describes this - extension record and hence might need - modification when the extension record - itself is modified, also locating the - attribute list also means finding the other - potential extents, belonging to the non-base - mft record). */ -/* 40*/ le16 next_attr_instance;/* The instance number that will be assigned to - the next attribute added to this mft record. - NOTE: Incremented each time after it is used. - NOTE: Every time the mft record is reused - this number is set to zero. NOTE: The first - instance number is always 0. */ -/* The below fields are specific to NTFS 3.1+ (Windows XP and above): */ -/* 42*/ le16 reserved; /* Reserved/alignment. */ -/* 44*/ le32 mft_record_number; /* Number of this mft record. */ -/* sizeof() = 48 bytes */ -/* - * When (re)using the mft record, we place the update sequence array at this - * offset, i.e. before we start with the attributes. This also makes sense, - * otherwise we could run into problems with the update sequence array - * containing in itself the last two bytes of a sector which would mean that - * multi sector transfer protection wouldn't work. As you can't protect data - * by overwriting it since you then can't get it back... - * When reading we obviously use the data from the ntfs record header. - */ -} __attribute__ ((__packed__)) MFT_RECORD; - -/* This is the version without the NTFS 3.1+ specific fields. */ -typedef struct { -/*Ofs*/ -/* 0 NTFS_RECORD; -- Unfolded here as gcc doesn't like unnamed structs. */ - NTFS_RECORD_TYPE magic; /* Usually the magic is "FILE". */ - le16 usa_ofs; /* See NTFS_RECORD definition above. */ - le16 usa_count; /* See NTFS_RECORD definition above. */ - -/* 8*/ le64 lsn; /* $LogFile sequence number for this record. - Changed every time the record is modified. */ -/* 16*/ le16 sequence_number; /* Number of times this mft record has been - reused. (See description for MFT_REF - above.) NOTE: The increment (skipping zero) - is done when the file is deleted. NOTE: If - this is zero it is left zero. */ -/* 18*/ le16 link_count; /* Number of hard links, i.e. the number of - directory entries referencing this record. - NOTE: Only used in mft base records. - NOTE: When deleting a directory entry we - check the link_count and if it is 1 we - delete the file. Otherwise we delete the - FILE_NAME_ATTR being referenced by the - directory entry from the mft record and - decrement the link_count. - FIXME: Careful with Win32 + DOS names! */ -/* 20*/ le16 attrs_offset; /* Byte offset to the first attribute in this - mft record from the start of the mft record. - NOTE: Must be aligned to 8-byte boundary. */ -/* 22*/ MFT_RECORD_FLAGS flags; /* Bit array of MFT_RECORD_FLAGS. When a file - is deleted, the MFT_RECORD_IN_USE flag is - set to zero. */ -/* 24*/ le32 bytes_in_use; /* Number of bytes used in this mft record. - NOTE: Must be aligned to 8-byte boundary. */ -/* 28*/ le32 bytes_allocated; /* Number of bytes allocated for this mft - record. This should be equal to the mft - record size. */ -/* 32*/ leMFT_REF base_mft_record;/* This is zero for base mft records. - When it is not zero it is a mft reference - pointing to the base mft record to which - this record belongs (this is then used to - locate the attribute list attribute present - in the base record which describes this - extension record and hence might need - modification when the extension record - itself is modified, also locating the - attribute list also means finding the other - potential extents, belonging to the non-base - mft record). */ -/* 40*/ le16 next_attr_instance;/* The instance number that will be assigned to - the next attribute added to this mft record. - NOTE: Incremented each time after it is used. - NOTE: Every time the mft record is reused - this number is set to zero. NOTE: The first - instance number is always 0. */ -/* sizeof() = 42 bytes */ -/* - * When (re)using the mft record, we place the update sequence array at this - * offset, i.e. before we start with the attributes. This also makes sense, - * otherwise we could run into problems with the update sequence array - * containing in itself the last two bytes of a sector which would mean that - * multi sector transfer protection wouldn't work. As you can't protect data - * by overwriting it since you then can't get it back... - * When reading we obviously use the data from the ntfs record header. - */ -} __attribute__ ((__packed__)) MFT_RECORD_OLD; - -/* - * System defined attributes (32-bit). Each attribute type has a corresponding - * attribute name (Unicode string of maximum 64 character length) as described - * by the attribute definitions present in the data attribute of the $AttrDef - * system file. On NTFS 3.0 volumes the names are just as the types are named - * in the below defines exchanging AT_ for the dollar sign ($). If that is not - * a revealing choice of symbol I do not know what is... (-; - */ -enum { - AT_UNUSED = cpu_to_le32( 0), - AT_STANDARD_INFORMATION = cpu_to_le32( 0x10), - AT_ATTRIBUTE_LIST = cpu_to_le32( 0x20), - AT_FILE_NAME = cpu_to_le32( 0x30), - AT_OBJECT_ID = cpu_to_le32( 0x40), - AT_SECURITY_DESCRIPTOR = cpu_to_le32( 0x50), - AT_VOLUME_NAME = cpu_to_le32( 0x60), - AT_VOLUME_INFORMATION = cpu_to_le32( 0x70), - AT_DATA = cpu_to_le32( 0x80), - AT_INDEX_ROOT = cpu_to_le32( 0x90), - AT_INDEX_ALLOCATION = cpu_to_le32( 0xa0), - AT_BITMAP = cpu_to_le32( 0xb0), - AT_REPARSE_POINT = cpu_to_le32( 0xc0), - AT_EA_INFORMATION = cpu_to_le32( 0xd0), - AT_EA = cpu_to_le32( 0xe0), - AT_PROPERTY_SET = cpu_to_le32( 0xf0), - AT_LOGGED_UTILITY_STREAM = cpu_to_le32( 0x100), - AT_FIRST_USER_DEFINED_ATTRIBUTE = cpu_to_le32( 0x1000), - AT_END = cpu_to_le32(0xffffffff) -}; - -typedef le32 ATTR_TYPE; - -/* - * The collation rules for sorting views/indexes/etc (32-bit). - * - * COLLATION_BINARY - Collate by binary compare where the first byte is most - * significant. - * COLLATION_UNICODE_STRING - Collate Unicode strings by comparing their binary - * Unicode values, except that when a character can be uppercased, the - * upper case value collates before the lower case one. - * COLLATION_FILE_NAME - Collate file names as Unicode strings. The collation - * is done very much like COLLATION_UNICODE_STRING. In fact I have no idea - * what the difference is. Perhaps the difference is that file names - * would treat some special characters in an odd way (see - * unistr.c::ntfs_collate_names() and unistr.c::legal_ansi_char_array[] - * for what I mean but COLLATION_UNICODE_STRING would not give any special - * treatment to any characters at all, but this is speculation. - * COLLATION_NTOFS_ULONG - Sorting is done according to ascending le32 key - * values. E.g. used for $SII index in FILE_Secure, which sorts by - * security_id (le32). - * COLLATION_NTOFS_SID - Sorting is done according to ascending SID values. - * E.g. used for $O index in FILE_Extend/$Quota. - * COLLATION_NTOFS_SECURITY_HASH - Sorting is done first by ascending hash - * values and second by ascending security_id values. E.g. used for $SDH - * index in FILE_Secure. - * COLLATION_NTOFS_ULONGS - Sorting is done according to a sequence of ascending - * le32 key values. E.g. used for $O index in FILE_Extend/$ObjId, which - * sorts by object_id (16-byte), by splitting up the object_id in four - * le32 values and using them as individual keys. E.g. take the following - * two security_ids, stored as follows on disk: - * 1st: a1 61 65 b7 65 7b d4 11 9e 3d 00 e0 81 10 42 59 - * 2nd: 38 14 37 d2 d2 f3 d4 11 a5 21 c8 6b 79 b1 97 45 - * To compare them, they are split into four le32 values each, like so: - * 1st: 0xb76561a1 0x11d47b65 0xe0003d9e 0x59421081 - * 2nd: 0xd2371438 0x11d4f3d2 0x6bc821a5 0x4597b179 - * Now, it is apparent why the 2nd object_id collates after the 1st: the - * first le32 value of the 1st object_id is less than the first le32 of - * the 2nd object_id. If the first le32 values of both object_ids were - * equal then the second le32 values would be compared, etc. - */ -enum { - COLLATION_BINARY = cpu_to_le32(0x00), - COLLATION_FILE_NAME = cpu_to_le32(0x01), - COLLATION_UNICODE_STRING = cpu_to_le32(0x02), - COLLATION_NTOFS_ULONG = cpu_to_le32(0x10), - COLLATION_NTOFS_SID = cpu_to_le32(0x11), - COLLATION_NTOFS_SECURITY_HASH = cpu_to_le32(0x12), - COLLATION_NTOFS_ULONGS = cpu_to_le32(0x13), -}; - -typedef le32 COLLATION_RULE; - -/* - * The flags (32-bit) describing attribute properties in the attribute - * definition structure. FIXME: This information is based on Regis's - * information and, according to him, it is not certain and probably - * incomplete. The INDEXABLE flag is fairly certainly correct as only the file - * name attribute has this flag set and this is the only attribute indexed in - * NT4. - */ -enum { - ATTR_DEF_INDEXABLE = cpu_to_le32(0x02), /* Attribute can be - indexed. */ - ATTR_DEF_MULTIPLE = cpu_to_le32(0x04), /* Attribute type - can be present multiple times in the - mft records of an inode. */ - ATTR_DEF_NOT_ZERO = cpu_to_le32(0x08), /* Attribute value - must contain at least one non-zero - byte. */ - ATTR_DEF_INDEXED_UNIQUE = cpu_to_le32(0x10), /* Attribute must be - indexed and the attribute value must be - unique for the attribute type in all of - the mft records of an inode. */ - ATTR_DEF_NAMED_UNIQUE = cpu_to_le32(0x20), /* Attribute must be - named and the name must be unique for - the attribute type in all of the mft - records of an inode. */ - ATTR_DEF_RESIDENT = cpu_to_le32(0x40), /* Attribute must be - resident. */ - ATTR_DEF_ALWAYS_LOG = cpu_to_le32(0x80), /* Always log - modifications to this attribute, - regardless of whether it is resident or - non-resident. Without this, only log - modifications if the attribute is - resident. */ -}; - -typedef le32 ATTR_DEF_FLAGS; - -/* - * The data attribute of FILE_AttrDef contains a sequence of attribute - * definitions for the NTFS volume. With this, it is supposed to be safe for an - * older NTFS driver to mount a volume containing a newer NTFS version without - * damaging it (that's the theory. In practice it's: not damaging it too much). - * Entries are sorted by attribute type. The flags describe whether the - * attribute can be resident/non-resident and possibly other things, but the - * actual bits are unknown. - */ -typedef struct { -/*hex ofs*/ -/* 0*/ ntfschar name[0x40]; /* Unicode name of the attribute. Zero - terminated. */ -/* 80*/ ATTR_TYPE type; /* Type of the attribute. */ -/* 84*/ le32 display_rule; /* Default display rule. - FIXME: What does it mean? (AIA) */ -/* 88*/ COLLATION_RULE collation_rule; /* Default collation rule. */ -/* 8c*/ ATTR_DEF_FLAGS flags; /* Flags describing the attribute. */ -/* 90*/ sle64 min_size; /* Optional minimum attribute size. */ -/* 98*/ sle64 max_size; /* Maximum size of attribute. */ -/* sizeof() = 0xa0 or 160 bytes */ -} __attribute__ ((__packed__)) ATTR_DEF; - -/* - * Attribute flags (16-bit). - */ -enum { - ATTR_IS_COMPRESSED = cpu_to_le16(0x0001), - ATTR_COMPRESSION_MASK = cpu_to_le16(0x00ff), /* Compression method - mask. Also, first - illegal value. */ - ATTR_IS_ENCRYPTED = cpu_to_le16(0x4000), - ATTR_IS_SPARSE = cpu_to_le16(0x8000), -} __attribute__ ((__packed__)); - -typedef le16 ATTR_FLAGS; - -/* - * Attribute compression. - * - * Only the data attribute is ever compressed in the current ntfs driver in - * Windows. Further, compression is only applied when the data attribute is - * non-resident. Finally, to use compression, the maximum allowed cluster size - * on a volume is 4kib. - * - * The compression method is based on independently compressing blocks of X - * clusters, where X is determined from the compression_unit value found in the - * non-resident attribute record header (more precisely: X = 2^compression_unit - * clusters). On Windows NT/2k, X always is 16 clusters (compression_unit = 4). - * - * There are three different cases of how a compression block of X clusters - * can be stored: - * - * 1) The data in the block is all zero (a sparse block): - * This is stored as a sparse block in the runlist, i.e. the runlist - * entry has length = X and lcn = -1. The mapping pairs array actually - * uses a delta_lcn value length of 0, i.e. delta_lcn is not present at - * all, which is then interpreted by the driver as lcn = -1. - * NOTE: Even uncompressed files can be sparse on NTFS 3.0 volumes, then - * the same principles apply as above, except that the length is not - * restricted to being any particular value. - * - * 2) The data in the block is not compressed: - * This happens when compression doesn't reduce the size of the block - * in clusters. I.e. if compression has a small effect so that the - * compressed data still occupies X clusters, then the uncompressed data - * is stored in the block. - * This case is recognised by the fact that the runlist entry has - * length = X and lcn >= 0. The mapping pairs array stores this as - * normal with a run length of X and some specific delta_lcn, i.e. - * delta_lcn has to be present. - * - * 3) The data in the block is compressed: - * The common case. This case is recognised by the fact that the run - * list entry has length L < X and lcn >= 0. The mapping pairs array - * stores this as normal with a run length of X and some specific - * delta_lcn, i.e. delta_lcn has to be present. This runlist entry is - * immediately followed by a sparse entry with length = X - L and - * lcn = -1. The latter entry is to make up the vcn counting to the - * full compression block size X. - * - * In fact, life is more complicated because adjacent entries of the same type - * can be coalesced. This means that one has to keep track of the number of - * clusters handled and work on a basis of X clusters at a time being one - * block. An example: if length L > X this means that this particular runlist - * entry contains a block of length X and part of one or more blocks of length - * L - X. Another example: if length L < X, this does not necessarily mean that - * the block is compressed as it might be that the lcn changes inside the block - * and hence the following runlist entry describes the continuation of the - * potentially compressed block. The block would be compressed if the - * following runlist entry describes at least X - L sparse clusters, thus - * making up the compression block length as described in point 3 above. (Of - * course, there can be several runlist entries with small lengths so that the - * sparse entry does not follow the first data containing entry with - * length < X.) - * - * NOTE: At the end of the compressed attribute value, there most likely is not - * just the right amount of data to make up a compression block, thus this data - * is not even attempted to be compressed. It is just stored as is, unless - * the number of clusters it occupies is reduced when compressed in which case - * it is stored as a compressed compression block, complete with sparse - * clusters at the end. - */ - -/* - * Flags of resident attributes (8-bit). - */ -enum { - RESIDENT_ATTR_IS_INDEXED = 0x01, /* Attribute is referenced in an index - (has implications for deleting and - modifying the attribute). */ -} __attribute__ ((__packed__)); - -typedef u8 RESIDENT_ATTR_FLAGS; - -/* - * Attribute record header. Always aligned to 8-byte boundary. - */ -typedef struct { -/*Ofs*/ -/* 0*/ ATTR_TYPE type; /* The (32-bit) type of the attribute. */ -/* 4*/ le32 length; /* Byte size of the resident part of the - attribute (aligned to 8-byte boundary). - Used to get to the next attribute. */ -/* 8*/ u8 non_resident; /* If 0, attribute is resident. - If 1, attribute is non-resident. */ -/* 9*/ u8 name_length; /* Unicode character size of name of attribute. - 0 if unnamed. */ -/* 10*/ le16 name_offset; /* If name_length != 0, the byte offset to the - beginning of the name from the attribute - record. Note that the name is stored as a - Unicode string. When creating, place offset - just at the end of the record header. Then, - follow with attribute value or mapping pairs - array, resident and non-resident attributes - respectively, aligning to an 8-byte - boundary. */ -/* 12*/ ATTR_FLAGS flags; /* Flags describing the attribute. */ -/* 14*/ le16 instance; /* The instance of this attribute record. This - number is unique within this mft record (see - MFT_RECORD/next_attribute_instance notes in - mft.h for more details). */ -/* 16*/ union { - /* Resident attributes. */ - struct { -/* 16 */ le32 value_length;/* Byte size of attribute value. */ -/* 20 */ le16 value_offset;/* Byte offset of the attribute - value from the start of the - attribute record. When creating, - align to 8-byte boundary if we - have a name present as this might - not have a length of a multiple - of 8-bytes. */ -/* 22 */ RESIDENT_ATTR_FLAGS flags; /* See above. */ -/* 23 */ s8 reserved; /* Reserved/alignment to 8-byte - boundary. */ - } __attribute__ ((__packed__)) resident; - /* Non-resident attributes. */ - struct { -/* 16*/ leVCN lowest_vcn;/* Lowest valid virtual cluster number - for this portion of the attribute value or - 0 if this is the only extent (usually the - case). - Only when an attribute list is used - does lowest_vcn != 0 ever occur. */ -/* 24*/ leVCN highest_vcn;/* Highest valid vcn of this extent of - the attribute value. - Usually there is only one - portion, so this usually equals the attribute - value size in clusters minus 1. Can be -1 for - zero length files. Can be 0 for "single extent" - attributes. */ -/* 32*/ le16 mapping_pairs_offset; /* Byte offset from the - beginning of the structure to the mapping pairs - array which contains the mappings between the - vcns and the logical cluster numbers (lcns). - When creating, place this at the end of this - record header aligned to 8-byte boundary. */ -/* 34*/ u8 compression_unit; /* The compression unit expressed - as the log to the base 2 of the number of - clusters in a compression unit. 0 means not - compressed. (This effectively limits the - compression unit size to be a power of two - clusters.) WinNT4 only uses a value of 4. - Sparse files have this set to 0 on XPSP2. */ -/* 35*/ u8 reserved[5]; /* Align to 8-byte boundary. */ -/* The sizes below are only used when lowest_vcn is zero, as otherwise it would - be difficult to keep them up-to-date.*/ -/* 40*/ sle64 allocated_size; /* Byte size of disk space - allocated to hold the attribute value. Always - is a multiple of the cluster size. When a file - is compressed, this field is a multiple of the - compression block size (2^compression_unit) and - it represents the logically allocated space - rather than the actual on disk usage. For this - use the compressed_size (see below). */ -/* 48*/ sle64 data_size; /* Byte size of the attribute - value. Can be larger than allocated_size if - attribute value is compressed or sparse. */ -/* 56*/ sle64 initialized_size; /* Byte size of initialized - portion of the attribute value. Usually equals - data_size. */ -/* sizeof(uncompressed attr) = 64*/ -/* 64*/ sle64 compressed_size; /* Byte size of the attribute - value after compression. Only present when - compressed or sparse. Always is a multiple of - the cluster size. Represents the actual amount - of disk space being used on the disk. */ -/* sizeof(compressed attr) = 72*/ - } __attribute__ ((__packed__)) non_resident; - } __attribute__ ((__packed__)) data; -} __attribute__ ((__packed__)) ATTR_RECORD; - -typedef ATTR_RECORD ATTR_REC; - -/* - * File attribute flags (32-bit) appearing in the file_attributes fields of the - * STANDARD_INFORMATION attribute of MFT_RECORDs and the FILENAME_ATTR - * attributes of MFT_RECORDs and directory index entries. - * - * All of the below flags appear in the directory index entries but only some - * appear in the STANDARD_INFORMATION attribute whilst only some others appear - * in the FILENAME_ATTR attribute of MFT_RECORDs. Unless otherwise stated the - * flags appear in all of the above. - */ -enum { - FILE_ATTR_READONLY = cpu_to_le32(0x00000001), - FILE_ATTR_HIDDEN = cpu_to_le32(0x00000002), - FILE_ATTR_SYSTEM = cpu_to_le32(0x00000004), - /* Old DOS volid. Unused in NT. = cpu_to_le32(0x00000008), */ - - FILE_ATTR_DIRECTORY = cpu_to_le32(0x00000010), - /* Note, FILE_ATTR_DIRECTORY is not considered valid in NT. It is - reserved for the DOS SUBDIRECTORY flag. */ - FILE_ATTR_ARCHIVE = cpu_to_le32(0x00000020), - FILE_ATTR_DEVICE = cpu_to_le32(0x00000040), - FILE_ATTR_NORMAL = cpu_to_le32(0x00000080), - - FILE_ATTR_TEMPORARY = cpu_to_le32(0x00000100), - FILE_ATTR_SPARSE_FILE = cpu_to_le32(0x00000200), - FILE_ATTR_REPARSE_POINT = cpu_to_le32(0x00000400), - FILE_ATTR_COMPRESSED = cpu_to_le32(0x00000800), - - FILE_ATTR_OFFLINE = cpu_to_le32(0x00001000), - FILE_ATTR_NOT_CONTENT_INDEXED = cpu_to_le32(0x00002000), - FILE_ATTR_ENCRYPTED = cpu_to_le32(0x00004000), - - FILE_ATTR_VALID_FLAGS = cpu_to_le32(0x00007fb7), - /* Note, FILE_ATTR_VALID_FLAGS masks out the old DOS VolId and the - FILE_ATTR_DEVICE and preserves everything else. This mask is used - to obtain all flags that are valid for reading. */ - FILE_ATTR_VALID_SET_FLAGS = cpu_to_le32(0x000031a7), - /* Note, FILE_ATTR_VALID_SET_FLAGS masks out the old DOS VolId, the - F_A_DEVICE, F_A_DIRECTORY, F_A_SPARSE_FILE, F_A_REPARSE_POINT, - F_A_COMPRESSED, and F_A_ENCRYPTED and preserves the rest. This mask - is used to obtain all flags that are valid for setting. */ - /* - * The flag FILE_ATTR_DUP_FILENAME_INDEX_PRESENT is present in all - * FILENAME_ATTR attributes but not in the STANDARD_INFORMATION - * attribute of an mft record. - */ - FILE_ATTR_DUP_FILE_NAME_INDEX_PRESENT = cpu_to_le32(0x10000000), - /* Note, this is a copy of the corresponding bit from the mft record, - telling us whether this is a directory or not, i.e. whether it has - an index root attribute or not. */ - FILE_ATTR_DUP_VIEW_INDEX_PRESENT = cpu_to_le32(0x20000000), - /* Note, this is a copy of the corresponding bit from the mft record, - telling us whether this file has a view index present (eg. object id - index, quota index, one of the security indexes or the encrypting - filesystem related indexes). */ -}; - -typedef le32 FILE_ATTR_FLAGS; - -/* - * NOTE on times in NTFS: All times are in MS standard time format, i.e. they - * are the number of 100-nanosecond intervals since 1st January 1601, 00:00:00 - * universal coordinated time (UTC). (In Linux time starts 1st January 1970, - * 00:00:00 UTC and is stored as the number of 1-second intervals since then.) - */ - -/* - * Attribute: Standard information (0x10). - * - * NOTE: Always resident. - * NOTE: Present in all base file records on a volume. - * NOTE: There is conflicting information about the meaning of each of the time - * fields but the meaning as defined below has been verified to be - * correct by practical experimentation on Windows NT4 SP6a and is hence - * assumed to be the one and only correct interpretation. - */ -typedef struct { -/*Ofs*/ -/* 0*/ sle64 creation_time; /* Time file was created. Updated when - a filename is changed(?). */ -/* 8*/ sle64 last_data_change_time; /* Time the data attribute was last - modified. */ -/* 16*/ sle64 last_mft_change_time; /* Time this mft record was last - modified. */ -/* 24*/ sle64 last_access_time; /* Approximate time when the file was - last accessed (obviously this is not - updated on read-only volumes). In - Windows this is only updated when - accessed if some time delta has - passed since the last update. Also, - last access time updates can be - disabled altogether for speed. */ -/* 32*/ FILE_ATTR_FLAGS file_attributes; /* Flags describing the file. */ -/* 36*/ union { - /* NTFS 1.2 */ - struct { - /* 36*/ u8 reserved12[12]; /* Reserved/alignment to 8-byte - boundary. */ - } __attribute__ ((__packed__)) v1; - /* sizeof() = 48 bytes */ - /* NTFS 3.x */ - struct { -/* - * If a volume has been upgraded from a previous NTFS version, then these - * fields are present only if the file has been accessed since the upgrade. - * Recognize the difference by comparing the length of the resident attribute - * value. If it is 48, then the following fields are missing. If it is 72 then - * the fields are present. Maybe just check like this: - * if (resident.ValueLength < sizeof(STANDARD_INFORMATION)) { - * Assume NTFS 1.2- format. - * If (volume version is 3.x) - * Upgrade attribute to NTFS 3.x format. - * else - * Use NTFS 1.2- format for access. - * } else - * Use NTFS 3.x format for access. - * Only problem is that it might be legal to set the length of the value to - * arbitrarily large values thus spoiling this check. - But chkdsk probably - * views that as a corruption, assuming that it behaves like this for all - * attributes. - */ - /* 36*/ le32 maximum_versions; /* Maximum allowed versions for - file. Zero if version numbering is disabled. */ - /* 40*/ le32 version_number; /* This file's version (if any). - Set to zero if maximum_versions is zero. */ - /* 44*/ le32 class_id; /* Class id from bidirectional - class id index (?). */ - /* 48*/ le32 owner_id; /* Owner_id of the user owning - the file. Translate via $Q index in FILE_Extend - /$Quota to the quota control entry for the user - owning the file. Zero if quotas are disabled. */ - /* 52*/ le32 security_id; /* Security_id for the file. - Translate via $SII index and $SDS data stream - in FILE_Secure to the security descriptor. */ - /* 56*/ le64 quota_charged; /* Byte size of the charge to - the quota for all streams of the file. Note: Is - zero if quotas are disabled. */ - /* 64*/ leUSN usn; /* Last update sequence number - of the file. This is a direct index into the - transaction log file ($UsnJrnl). It is zero if - the usn journal is disabled or this file has - not been subject to logging yet. See usnjrnl.h - for details. */ - } __attribute__ ((__packed__)) v3; - /* sizeof() = 72 bytes (NTFS 3.x) */ - } __attribute__ ((__packed__)) ver; -} __attribute__ ((__packed__)) STANDARD_INFORMATION; - -/* - * Attribute: Attribute list (0x20). - * - * - Can be either resident or non-resident. - * - Value consists of a sequence of variable length, 8-byte aligned, - * ATTR_LIST_ENTRY records. - * - The list is not terminated by anything at all! The only way to know when - * the end is reached is to keep track of the current offset and compare it to - * the attribute value size. - * - The attribute list attribute contains one entry for each attribute of - * the file in which the list is located, except for the list attribute - * itself. The list is sorted: first by attribute type, second by attribute - * name (if present), third by instance number. The extents of one - * non-resident attribute (if present) immediately follow after the initial - * extent. They are ordered by lowest_vcn and have their instace set to zero. - * It is not allowed to have two attributes with all sorting keys equal. - * - Further restrictions: - * - If not resident, the vcn to lcn mapping array has to fit inside the - * base mft record. - * - The attribute list attribute value has a maximum size of 256kb. This - * is imposed by the Windows cache manager. - * - Attribute lists are only used when the attributes of mft record do not - * fit inside the mft record despite all attributes (that can be made - * non-resident) having been made non-resident. This can happen e.g. when: - * - File has a large number of hard links (lots of file name - * attributes present). - * - The mapping pairs array of some non-resident attribute becomes so - * large due to fragmentation that it overflows the mft record. - * - The security descriptor is very complex (not applicable to - * NTFS 3.0 volumes). - * - There are many named streams. - */ -typedef struct { -/*Ofs*/ -/* 0*/ ATTR_TYPE type; /* Type of referenced attribute. */ -/* 4*/ le16 length; /* Byte size of this entry (8-byte aligned). */ -/* 6*/ u8 name_length; /* Size in Unicode chars of the name of the - attribute or 0 if unnamed. */ -/* 7*/ u8 name_offset; /* Byte offset to beginning of attribute name - (always set this to where the name would - start even if unnamed). */ -/* 8*/ leVCN lowest_vcn; /* Lowest virtual cluster number of this portion - of the attribute value. This is usually 0. It - is non-zero for the case where one attribute - does not fit into one mft record and thus - several mft records are allocated to hold - this attribute. In the latter case, each mft - record holds one extent of the attribute and - there is one attribute list entry for each - extent. NOTE: This is DEFINITELY a signed - value! The windows driver uses cmp, followed - by jg when comparing this, thus it treats it - as signed. */ -/* 16*/ leMFT_REF mft_reference;/* The reference of the mft record holding - the ATTR_RECORD for this portion of the - attribute value. */ -/* 24*/ le16 instance; /* If lowest_vcn = 0, the instance of the - attribute being referenced; otherwise 0. */ -/* 26*/ ntfschar name[0]; /* Use when creating only. When reading use - name_offset to determine the location of the - name. */ -/* sizeof() = 26 + (attribute_name_length * 2) bytes */ -} __attribute__ ((__packed__)) ATTR_LIST_ENTRY; - -/* - * The maximum allowed length for a file name. - */ -#define MAXIMUM_FILE_NAME_LENGTH 255 - -/* - * Possible namespaces for filenames in ntfs (8-bit). - */ -enum { - FILE_NAME_POSIX = 0x00, - /* This is the largest namespace. It is case sensitive and allows all - Unicode characters except for: '\0' and '/'. Beware that in - WinNT/2k/2003 by default files which eg have the same name except - for their case will not be distinguished by the standard utilities - and thus a "del filename" will delete both "filename" and "fileName" - without warning. However if for example Services For Unix (SFU) are - installed and the case sensitive option was enabled at installation - time, then you can create/access/delete such files. - Note that even SFU places restrictions on the filenames beyond the - '\0' and '/' and in particular the following set of characters is - not allowed: '"', '/', '<', '>', '\'. All other characters, - including the ones no allowed in WIN32 namespace are allowed. - Tested with SFU 3.5 (this is now free) running on Windows XP. */ - FILE_NAME_WIN32 = 0x01, - /* The standard WinNT/2k NTFS long filenames. Case insensitive. All - Unicode chars except: '\0', '"', '*', '/', ':', '<', '>', '?', '\', - and '|'. Further, names cannot end with a '.' or a space. */ - FILE_NAME_DOS = 0x02, - /* The standard DOS filenames (8.3 format). Uppercase only. All 8-bit - characters greater space, except: '"', '*', '+', ',', '/', ':', ';', - '<', '=', '>', '?', and '\'. */ - FILE_NAME_WIN32_AND_DOS = 0x03, - /* 3 means that both the Win32 and the DOS filenames are identical and - hence have been saved in this single filename record. */ -} __attribute__ ((__packed__)); - -typedef u8 FILE_NAME_TYPE_FLAGS; - -/* - * Attribute: Filename (0x30). - * - * NOTE: Always resident. - * NOTE: All fields, except the parent_directory, are only updated when the - * filename is changed. Until then, they just become out of sync with - * reality and the more up to date values are present in the standard - * information attribute. - * NOTE: There is conflicting information about the meaning of each of the time - * fields but the meaning as defined below has been verified to be - * correct by practical experimentation on Windows NT4 SP6a and is hence - * assumed to be the one and only correct interpretation. - */ -typedef struct { -/*hex ofs*/ -/* 0*/ leMFT_REF parent_directory; /* Directory this filename is - referenced from. */ -/* 8*/ sle64 creation_time; /* Time file was created. */ -/* 10*/ sle64 last_data_change_time; /* Time the data attribute was last - modified. */ -/* 18*/ sle64 last_mft_change_time; /* Time this mft record was last - modified. */ -/* 20*/ sle64 last_access_time; /* Time this mft record was last - accessed. */ -/* 28*/ sle64 allocated_size; /* Byte size of on-disk allocated space - for the unnamed data attribute. So - for normal $DATA, this is the - allocated_size from the unnamed - $DATA attribute and for compressed - and/or sparse $DATA, this is the - compressed_size from the unnamed - $DATA attribute. For a directory or - other inode without an unnamed $DATA - attribute, this is always 0. NOTE: - This is a multiple of the cluster - size. */ -/* 30*/ sle64 data_size; /* Byte size of actual data in unnamed - data attribute. For a directory or - other inode without an unnamed $DATA - attribute, this is always 0. */ -/* 38*/ FILE_ATTR_FLAGS file_attributes; /* Flags describing the file. */ -/* 3c*/ union { - /* 3c*/ struct { - /* 3c*/ le16 packed_ea_size; /* Size of the buffer needed to - pack the extended attributes - (EAs), if such are present.*/ - /* 3e*/ le16 reserved; /* Reserved for alignment. */ - } __attribute__ ((__packed__)) ea; - /* 3c*/ struct { - /* 3c*/ le32 reparse_point_tag; /* Type of reparse point, - present only in reparse - points and only if there are - no EAs. */ - } __attribute__ ((__packed__)) rp; - } __attribute__ ((__packed__)) type; -/* 40*/ u8 file_name_length; /* Length of file name in - (Unicode) characters. */ -/* 41*/ FILE_NAME_TYPE_FLAGS file_name_type; /* Namespace of the file name.*/ -/* 42*/ ntfschar file_name[0]; /* File name in Unicode. */ -} __attribute__ ((__packed__)) FILE_NAME_ATTR; - -/* - * GUID structures store globally unique identifiers (GUID). A GUID is a - * 128-bit value consisting of one group of eight hexadecimal digits, followed - * by three groups of four hexadecimal digits each, followed by one group of - * twelve hexadecimal digits. GUIDs are Microsoft's implementation of the - * distributed computing environment (DCE) universally unique identifier (UUID). - * Example of a GUID: - * 1F010768-5A73-BC91-0010A52216A7 - */ -typedef struct { - le32 data1; /* The first eight hexadecimal digits of the GUID. */ - le16 data2; /* The first group of four hexadecimal digits. */ - le16 data3; /* The second group of four hexadecimal digits. */ - u8 data4[8]; /* The first two bytes are the third group of four - hexadecimal digits. The remaining six bytes are the - final 12 hexadecimal digits. */ -} __attribute__ ((__packed__)) GUID; - -/* - * FILE_Extend/$ObjId contains an index named $O. This index contains all - * object_ids present on the volume as the index keys and the corresponding - * mft_record numbers as the index entry data parts. The data part (defined - * below) also contains three other object_ids: - * birth_volume_id - object_id of FILE_Volume on which the file was first - * created. Optional (i.e. can be zero). - * birth_object_id - object_id of file when it was first created. Usually - * equals the object_id. Optional (i.e. can be zero). - * domain_id - Reserved (always zero). - */ -typedef struct { - leMFT_REF mft_reference;/* Mft record containing the object_id in - the index entry key. */ - union { - struct { - GUID birth_volume_id; - GUID birth_object_id; - GUID domain_id; - } __attribute__ ((__packed__)) origin; - u8 extended_info[48]; - } __attribute__ ((__packed__)) opt; -} __attribute__ ((__packed__)) OBJ_ID_INDEX_DATA; - -/* - * Attribute: Object id (NTFS 3.0+) (0x40). - * - * NOTE: Always resident. - */ -typedef struct { - GUID object_id; /* Unique id assigned to the - file.*/ - /* The following fields are optional. The attribute value size is 16 - bytes, i.e. sizeof(GUID), if these are not present at all. Note, - the entries can be present but one or more (or all) can be zero - meaning that that particular value(s) is(are) not defined. */ - union { - struct { - GUID birth_volume_id; /* Unique id of volume on which - the file was first created.*/ - GUID birth_object_id; /* Unique id of file when it was - first created. */ - GUID domain_id; /* Reserved, zero. */ - } __attribute__ ((__packed__)) origin; - u8 extended_info[48]; - } __attribute__ ((__packed__)) opt; -} __attribute__ ((__packed__)) OBJECT_ID_ATTR; - -/* - * The pre-defined IDENTIFIER_AUTHORITIES used as SID_IDENTIFIER_AUTHORITY in - * the SID structure (see below). - */ -//typedef enum { /* SID string prefix. */ -// SECURITY_NULL_SID_AUTHORITY = {0, 0, 0, 0, 0, 0}, /* S-1-0 */ -// SECURITY_WORLD_SID_AUTHORITY = {0, 0, 0, 0, 0, 1}, /* S-1-1 */ -// SECURITY_LOCAL_SID_AUTHORITY = {0, 0, 0, 0, 0, 2}, /* S-1-2 */ -// SECURITY_CREATOR_SID_AUTHORITY = {0, 0, 0, 0, 0, 3}, /* S-1-3 */ -// SECURITY_NON_UNIQUE_AUTHORITY = {0, 0, 0, 0, 0, 4}, /* S-1-4 */ -// SECURITY_NT_SID_AUTHORITY = {0, 0, 0, 0, 0, 5}, /* S-1-5 */ -//} IDENTIFIER_AUTHORITIES; - -/* - * These relative identifiers (RIDs) are used with the above identifier - * authorities to make up universal well-known SIDs. - * - * Note: The relative identifier (RID) refers to the portion of a SID, which - * identifies a user or group in relation to the authority that issued the SID. - * For example, the universal well-known SID Creator Owner ID (S-1-3-0) is - * made up of the identifier authority SECURITY_CREATOR_SID_AUTHORITY (3) and - * the relative identifier SECURITY_CREATOR_OWNER_RID (0). - */ -typedef enum { /* Identifier authority. */ - SECURITY_NULL_RID = 0, /* S-1-0 */ - SECURITY_WORLD_RID = 0, /* S-1-1 */ - SECURITY_LOCAL_RID = 0, /* S-1-2 */ - - SECURITY_CREATOR_OWNER_RID = 0, /* S-1-3 */ - SECURITY_CREATOR_GROUP_RID = 1, /* S-1-3 */ - - SECURITY_CREATOR_OWNER_SERVER_RID = 2, /* S-1-3 */ - SECURITY_CREATOR_GROUP_SERVER_RID = 3, /* S-1-3 */ - - SECURITY_DIALUP_RID = 1, - SECURITY_NETWORK_RID = 2, - SECURITY_BATCH_RID = 3, - SECURITY_INTERACTIVE_RID = 4, - SECURITY_SERVICE_RID = 6, - SECURITY_ANONYMOUS_LOGON_RID = 7, - SECURITY_PROXY_RID = 8, - SECURITY_ENTERPRISE_CONTROLLERS_RID=9, - SECURITY_SERVER_LOGON_RID = 9, - SECURITY_PRINCIPAL_SELF_RID = 0xa, - SECURITY_AUTHENTICATED_USER_RID = 0xb, - SECURITY_RESTRICTED_CODE_RID = 0xc, - SECURITY_TERMINAL_SERVER_RID = 0xd, - - SECURITY_LOGON_IDS_RID = 5, - SECURITY_LOGON_IDS_RID_COUNT = 3, - - SECURITY_LOCAL_SYSTEM_RID = 0x12, - - SECURITY_NT_NON_UNIQUE = 0x15, - - SECURITY_BUILTIN_DOMAIN_RID = 0x20, - - /* - * Well-known domain relative sub-authority values (RIDs). - */ - - /* Users. */ - DOMAIN_USER_RID_ADMIN = 0x1f4, - DOMAIN_USER_RID_GUEST = 0x1f5, - DOMAIN_USER_RID_KRBTGT = 0x1f6, - - /* Groups. */ - DOMAIN_GROUP_RID_ADMINS = 0x200, - DOMAIN_GROUP_RID_USERS = 0x201, - DOMAIN_GROUP_RID_GUESTS = 0x202, - DOMAIN_GROUP_RID_COMPUTERS = 0x203, - DOMAIN_GROUP_RID_CONTROLLERS = 0x204, - DOMAIN_GROUP_RID_CERT_ADMINS = 0x205, - DOMAIN_GROUP_RID_SCHEMA_ADMINS = 0x206, - DOMAIN_GROUP_RID_ENTERPRISE_ADMINS= 0x207, - DOMAIN_GROUP_RID_POLICY_ADMINS = 0x208, - - /* Aliases. */ - DOMAIN_ALIAS_RID_ADMINS = 0x220, - DOMAIN_ALIAS_RID_USERS = 0x221, - DOMAIN_ALIAS_RID_GUESTS = 0x222, - DOMAIN_ALIAS_RID_POWER_USERS = 0x223, - - DOMAIN_ALIAS_RID_ACCOUNT_OPS = 0x224, - DOMAIN_ALIAS_RID_SYSTEM_OPS = 0x225, - DOMAIN_ALIAS_RID_PRINT_OPS = 0x226, - DOMAIN_ALIAS_RID_BACKUP_OPS = 0x227, - - DOMAIN_ALIAS_RID_REPLICATOR = 0x228, - DOMAIN_ALIAS_RID_RAS_SERVERS = 0x229, - DOMAIN_ALIAS_RID_PREW2KCOMPACCESS = 0x22a, -} RELATIVE_IDENTIFIERS; - -/* - * The universal well-known SIDs: - * - * NULL_SID S-1-0-0 - * WORLD_SID S-1-1-0 - * LOCAL_SID S-1-2-0 - * CREATOR_OWNER_SID S-1-3-0 - * CREATOR_GROUP_SID S-1-3-1 - * CREATOR_OWNER_SERVER_SID S-1-3-2 - * CREATOR_GROUP_SERVER_SID S-1-3-3 - * - * (Non-unique IDs) S-1-4 - * - * NT well-known SIDs: - * - * NT_AUTHORITY_SID S-1-5 - * DIALUP_SID S-1-5-1 - * - * NETWORD_SID S-1-5-2 - * BATCH_SID S-1-5-3 - * INTERACTIVE_SID S-1-5-4 - * SERVICE_SID S-1-5-6 - * ANONYMOUS_LOGON_SID S-1-5-7 (aka null logon session) - * PROXY_SID S-1-5-8 - * SERVER_LOGON_SID S-1-5-9 (aka domain controller account) - * SELF_SID S-1-5-10 (self RID) - * AUTHENTICATED_USER_SID S-1-5-11 - * RESTRICTED_CODE_SID S-1-5-12 (running restricted code) - * TERMINAL_SERVER_SID S-1-5-13 (running on terminal server) - * - * (Logon IDs) S-1-5-5-X-Y - * - * (NT non-unique IDs) S-1-5-0x15-... - * - * (Built-in domain) S-1-5-0x20 - */ - -/* - * The SID_IDENTIFIER_AUTHORITY is a 48-bit value used in the SID structure. - * - * NOTE: This is stored as a big endian number, hence the high_part comes - * before the low_part. - */ -typedef union { - struct { - u16 high_part; /* High 16-bits. */ - u32 low_part; /* Low 32-bits. */ - } __attribute__ ((__packed__)) parts; - u8 value[6]; /* Value as individual bytes. */ -} __attribute__ ((__packed__)) SID_IDENTIFIER_AUTHORITY; - -/* - * The SID structure is a variable-length structure used to uniquely identify - * users or groups. SID stands for security identifier. - * - * The standard textual representation of the SID is of the form: - * S-R-I-S-S... - * Where: - * - The first "S" is the literal character 'S' identifying the following - * digits as a SID. - * - R is the revision level of the SID expressed as a sequence of digits - * either in decimal or hexadecimal (if the later, prefixed by "0x"). - * - I is the 48-bit identifier_authority, expressed as digits as R above. - * - S... is one or more sub_authority values, expressed as digits as above. - * - * Example SID; the domain-relative SID of the local Administrators group on - * Windows NT/2k: - * S-1-5-32-544 - * This translates to a SID with: - * revision = 1, - * sub_authority_count = 2, - * identifier_authority = {0,0,0,0,0,5}, // SECURITY_NT_AUTHORITY - * sub_authority[0] = 32, // SECURITY_BUILTIN_DOMAIN_RID - * sub_authority[1] = 544 // DOMAIN_ALIAS_RID_ADMINS - */ -typedef struct { - u8 revision; - u8 sub_authority_count; - SID_IDENTIFIER_AUTHORITY identifier_authority; - le32 sub_authority[1]; /* At least one sub_authority. */ -} __attribute__ ((__packed__)) SID; - -/* - * Current constants for SIDs. - */ -typedef enum { - SID_REVISION = 1, /* Current revision level. */ - SID_MAX_SUB_AUTHORITIES = 15, /* Maximum number of those. */ - SID_RECOMMENDED_SUB_AUTHORITIES = 1, /* Will change to around 6 in - a future revision. */ -} SID_CONSTANTS; - -/* - * The predefined ACE types (8-bit, see below). - */ -enum { - ACCESS_MIN_MS_ACE_TYPE = 0, - ACCESS_ALLOWED_ACE_TYPE = 0, - ACCESS_DENIED_ACE_TYPE = 1, - SYSTEM_AUDIT_ACE_TYPE = 2, - SYSTEM_ALARM_ACE_TYPE = 3, /* Not implemented as of Win2k. */ - ACCESS_MAX_MS_V2_ACE_TYPE = 3, - - ACCESS_ALLOWED_COMPOUND_ACE_TYPE= 4, - ACCESS_MAX_MS_V3_ACE_TYPE = 4, - - /* The following are Win2k only. */ - ACCESS_MIN_MS_OBJECT_ACE_TYPE = 5, - ACCESS_ALLOWED_OBJECT_ACE_TYPE = 5, - ACCESS_DENIED_OBJECT_ACE_TYPE = 6, - SYSTEM_AUDIT_OBJECT_ACE_TYPE = 7, - SYSTEM_ALARM_OBJECT_ACE_TYPE = 8, - ACCESS_MAX_MS_OBJECT_ACE_TYPE = 8, - - ACCESS_MAX_MS_V4_ACE_TYPE = 8, - - /* This one is for WinNT/2k. */ - ACCESS_MAX_MS_ACE_TYPE = 8, -} __attribute__ ((__packed__)); - -typedef u8 ACE_TYPES; - -/* - * The ACE flags (8-bit) for audit and inheritance (see below). - * - * SUCCESSFUL_ACCESS_ACE_FLAG is only used with system audit and alarm ACE - * types to indicate that a message is generated (in Windows!) for successful - * accesses. - * - * FAILED_ACCESS_ACE_FLAG is only used with system audit and alarm ACE types - * to indicate that a message is generated (in Windows!) for failed accesses. - */ -enum { - /* The inheritance flags. */ - OBJECT_INHERIT_ACE = 0x01, - CONTAINER_INHERIT_ACE = 0x02, - NO_PROPAGATE_INHERIT_ACE = 0x04, - INHERIT_ONLY_ACE = 0x08, - INHERITED_ACE = 0x10, /* Win2k only. */ - VALID_INHERIT_FLAGS = 0x1f, - - /* The audit flags. */ - SUCCESSFUL_ACCESS_ACE_FLAG = 0x40, - FAILED_ACCESS_ACE_FLAG = 0x80, -} __attribute__ ((__packed__)); - -typedef u8 ACE_FLAGS; - -/* - * An ACE is an access-control entry in an access-control list (ACL). - * An ACE defines access to an object for a specific user or group or defines - * the types of access that generate system-administration messages or alarms - * for a specific user or group. The user or group is identified by a security - * identifier (SID). - * - * Each ACE starts with an ACE_HEADER structure (aligned on 4-byte boundary), - * which specifies the type and size of the ACE. The format of the subsequent - * data depends on the ACE type. - */ -typedef struct { -/*Ofs*/ -/* 0*/ ACE_TYPES type; /* Type of the ACE. */ -/* 1*/ ACE_FLAGS flags; /* Flags describing the ACE. */ -/* 2*/ le16 size; /* Size in bytes of the ACE. */ -} __attribute__ ((__packed__)) ACE_HEADER; - -/* - * The access mask (32-bit). Defines the access rights. - * - * The specific rights (bits 0 to 15). These depend on the type of the object - * being secured by the ACE. - */ -enum { - /* Specific rights for files and directories are as follows: */ - - /* Right to read data from the file. (FILE) */ - FILE_READ_DATA = cpu_to_le32(0x00000001), - /* Right to list contents of a directory. (DIRECTORY) */ - FILE_LIST_DIRECTORY = cpu_to_le32(0x00000001), - - /* Right to write data to the file. (FILE) */ - FILE_WRITE_DATA = cpu_to_le32(0x00000002), - /* Right to create a file in the directory. (DIRECTORY) */ - FILE_ADD_FILE = cpu_to_le32(0x00000002), - - /* Right to append data to the file. (FILE) */ - FILE_APPEND_DATA = cpu_to_le32(0x00000004), - /* Right to create a subdirectory. (DIRECTORY) */ - FILE_ADD_SUBDIRECTORY = cpu_to_le32(0x00000004), - - /* Right to read extended attributes. (FILE/DIRECTORY) */ - FILE_READ_EA = cpu_to_le32(0x00000008), - - /* Right to write extended attributes. (FILE/DIRECTORY) */ - FILE_WRITE_EA = cpu_to_le32(0x00000010), - - /* Right to execute a file. (FILE) */ - FILE_EXECUTE = cpu_to_le32(0x00000020), - /* Right to traverse the directory. (DIRECTORY) */ - FILE_TRAVERSE = cpu_to_le32(0x00000020), - - /* - * Right to delete a directory and all the files it contains (its - * children), even if the files are read-only. (DIRECTORY) - */ - FILE_DELETE_CHILD = cpu_to_le32(0x00000040), - - /* Right to read file attributes. (FILE/DIRECTORY) */ - FILE_READ_ATTRIBUTES = cpu_to_le32(0x00000080), - - /* Right to change file attributes. (FILE/DIRECTORY) */ - FILE_WRITE_ATTRIBUTES = cpu_to_le32(0x00000100), - - /* - * The standard rights (bits 16 to 23). These are independent of the - * type of object being secured. - */ - - /* Right to delete the object. */ - DELETE = cpu_to_le32(0x00010000), - - /* - * Right to read the information in the object's security descriptor, - * not including the information in the SACL, i.e. right to read the - * security descriptor and owner. - */ - READ_CONTROL = cpu_to_le32(0x00020000), - - /* Right to modify the DACL in the object's security descriptor. */ - WRITE_DAC = cpu_to_le32(0x00040000), - - /* Right to change the owner in the object's security descriptor. */ - WRITE_OWNER = cpu_to_le32(0x00080000), - - /* - * Right to use the object for synchronization. Enables a process to - * wait until the object is in the signalled state. Some object types - * do not support this access right. - */ - SYNCHRONIZE = cpu_to_le32(0x00100000), - - /* - * The following STANDARD_RIGHTS_* are combinations of the above for - * convenience and are defined by the Win32 API. - */ - - /* These are currently defined to READ_CONTROL. */ - STANDARD_RIGHTS_READ = cpu_to_le32(0x00020000), - STANDARD_RIGHTS_WRITE = cpu_to_le32(0x00020000), - STANDARD_RIGHTS_EXECUTE = cpu_to_le32(0x00020000), - - /* Combines DELETE, READ_CONTROL, WRITE_DAC, and WRITE_OWNER access. */ - STANDARD_RIGHTS_REQUIRED = cpu_to_le32(0x000f0000), - - /* - * Combines DELETE, READ_CONTROL, WRITE_DAC, WRITE_OWNER, and - * SYNCHRONIZE access. - */ - STANDARD_RIGHTS_ALL = cpu_to_le32(0x001f0000), - - /* - * The access system ACL and maximum allowed access types (bits 24 to - * 25, bits 26 to 27 are reserved). - */ - ACCESS_SYSTEM_SECURITY = cpu_to_le32(0x01000000), - MAXIMUM_ALLOWED = cpu_to_le32(0x02000000), - - /* - * The generic rights (bits 28 to 31). These map onto the standard and - * specific rights. - */ - - /* Read, write, and execute access. */ - GENERIC_ALL = cpu_to_le32(0x10000000), - - /* Execute access. */ - GENERIC_EXECUTE = cpu_to_le32(0x20000000), - - /* - * Write access. For files, this maps onto: - * FILE_APPEND_DATA | FILE_WRITE_ATTRIBUTES | FILE_WRITE_DATA | - * FILE_WRITE_EA | STANDARD_RIGHTS_WRITE | SYNCHRONIZE - * For directories, the mapping has the same numerical value. See - * above for the descriptions of the rights granted. - */ - GENERIC_WRITE = cpu_to_le32(0x40000000), - - /* - * Read access. For files, this maps onto: - * FILE_READ_ATTRIBUTES | FILE_READ_DATA | FILE_READ_EA | - * STANDARD_RIGHTS_READ | SYNCHRONIZE - * For directories, the mapping has the same numberical value. See - * above for the descriptions of the rights granted. - */ - GENERIC_READ = cpu_to_le32(0x80000000), -}; - -typedef le32 ACCESS_MASK; - -/* - * The generic mapping array. Used to denote the mapping of each generic - * access right to a specific access mask. - * - * FIXME: What exactly is this and what is it for? (AIA) - */ -typedef struct { - ACCESS_MASK generic_read; - ACCESS_MASK generic_write; - ACCESS_MASK generic_execute; - ACCESS_MASK generic_all; -} __attribute__ ((__packed__)) GENERIC_MAPPING; - -/* - * The predefined ACE type structures are as defined below. - */ - -/* - * ACCESS_ALLOWED_ACE, ACCESS_DENIED_ACE, SYSTEM_AUDIT_ACE, SYSTEM_ALARM_ACE - */ -typedef struct { -/* 0 ACE_HEADER; -- Unfolded here as gcc doesn't like unnamed structs. */ - ACE_TYPES type; /* Type of the ACE. */ - ACE_FLAGS flags; /* Flags describing the ACE. */ - le16 size; /* Size in bytes of the ACE. */ -/* 4*/ ACCESS_MASK mask; /* Access mask associated with the ACE. */ - -/* 8*/ SID sid; /* The SID associated with the ACE. */ -} __attribute__ ((__packed__)) ACCESS_ALLOWED_ACE, ACCESS_DENIED_ACE, - SYSTEM_AUDIT_ACE, SYSTEM_ALARM_ACE; - -/* - * The object ACE flags (32-bit). - */ -enum { - ACE_OBJECT_TYPE_PRESENT = cpu_to_le32(1), - ACE_INHERITED_OBJECT_TYPE_PRESENT = cpu_to_le32(2), -}; - -typedef le32 OBJECT_ACE_FLAGS; - -typedef struct { -/* 0 ACE_HEADER; -- Unfolded here as gcc doesn't like unnamed structs. */ - ACE_TYPES type; /* Type of the ACE. */ - ACE_FLAGS flags; /* Flags describing the ACE. */ - le16 size; /* Size in bytes of the ACE. */ -/* 4*/ ACCESS_MASK mask; /* Access mask associated with the ACE. */ - -/* 8*/ OBJECT_ACE_FLAGS object_flags; /* Flags describing the object ACE. */ -/* 12*/ GUID object_type; -/* 28*/ GUID inherited_object_type; - -/* 44*/ SID sid; /* The SID associated with the ACE. */ -} __attribute__ ((__packed__)) ACCESS_ALLOWED_OBJECT_ACE, - ACCESS_DENIED_OBJECT_ACE, - SYSTEM_AUDIT_OBJECT_ACE, - SYSTEM_ALARM_OBJECT_ACE; - -/* - * An ACL is an access-control list (ACL). - * An ACL starts with an ACL header structure, which specifies the size of - * the ACL and the number of ACEs it contains. The ACL header is followed by - * zero or more access control entries (ACEs). The ACL as well as each ACE - * are aligned on 4-byte boundaries. - */ -typedef struct { - u8 revision; /* Revision of this ACL. */ - u8 alignment1; - le16 size; /* Allocated space in bytes for ACL. Includes this - header, the ACEs and the remaining free space. */ - le16 ace_count; /* Number of ACEs in the ACL. */ - le16 alignment2; -/* sizeof() = 8 bytes */ -} __attribute__ ((__packed__)) ACL; - -/* - * Current constants for ACLs. - */ -typedef enum { - /* Current revision. */ - ACL_REVISION = 2, - ACL_REVISION_DS = 4, - - /* History of revisions. */ - ACL_REVISION1 = 1, - MIN_ACL_REVISION = 2, - ACL_REVISION2 = 2, - ACL_REVISION3 = 3, - ACL_REVISION4 = 4, - MAX_ACL_REVISION = 4, -} ACL_CONSTANTS; - -/* - * The security descriptor control flags (16-bit). - * - * SE_OWNER_DEFAULTED - This boolean flag, when set, indicates that the SID - * pointed to by the Owner field was provided by a defaulting mechanism - * rather than explicitly provided by the original provider of the - * security descriptor. This may affect the treatment of the SID with - * respect to inheritance of an owner. - * - * SE_GROUP_DEFAULTED - This boolean flag, when set, indicates that the SID in - * the Group field was provided by a defaulting mechanism rather than - * explicitly provided by the original provider of the security - * descriptor. This may affect the treatment of the SID with respect to - * inheritance of a primary group. - * - * SE_DACL_PRESENT - This boolean flag, when set, indicates that the security - * descriptor contains a discretionary ACL. If this flag is set and the - * Dacl field of the SECURITY_DESCRIPTOR is null, then a null ACL is - * explicitly being specified. - * - * SE_DACL_DEFAULTED - This boolean flag, when set, indicates that the ACL - * pointed to by the Dacl field was provided by a defaulting mechanism - * rather than explicitly provided by the original provider of the - * security descriptor. This may affect the treatment of the ACL with - * respect to inheritance of an ACL. This flag is ignored if the - * DaclPresent flag is not set. - * - * SE_SACL_PRESENT - This boolean flag, when set, indicates that the security - * descriptor contains a system ACL pointed to by the Sacl field. If this - * flag is set and the Sacl field of the SECURITY_DESCRIPTOR is null, then - * an empty (but present) ACL is being specified. - * - * SE_SACL_DEFAULTED - This boolean flag, when set, indicates that the ACL - * pointed to by the Sacl field was provided by a defaulting mechanism - * rather than explicitly provided by the original provider of the - * security descriptor. This may affect the treatment of the ACL with - * respect to inheritance of an ACL. This flag is ignored if the - * SaclPresent flag is not set. - * - * SE_SELF_RELATIVE - This boolean flag, when set, indicates that the security - * descriptor is in self-relative form. In this form, all fields of the - * security descriptor are contiguous in memory and all pointer fields are - * expressed as offsets from the beginning of the security descriptor. - */ -enum { - SE_OWNER_DEFAULTED = cpu_to_le16(0x0001), - SE_GROUP_DEFAULTED = cpu_to_le16(0x0002), - SE_DACL_PRESENT = cpu_to_le16(0x0004), - SE_DACL_DEFAULTED = cpu_to_le16(0x0008), - - SE_SACL_PRESENT = cpu_to_le16(0x0010), - SE_SACL_DEFAULTED = cpu_to_le16(0x0020), - - SE_DACL_AUTO_INHERIT_REQ = cpu_to_le16(0x0100), - SE_SACL_AUTO_INHERIT_REQ = cpu_to_le16(0x0200), - SE_DACL_AUTO_INHERITED = cpu_to_le16(0x0400), - SE_SACL_AUTO_INHERITED = cpu_to_le16(0x0800), - - SE_DACL_PROTECTED = cpu_to_le16(0x1000), - SE_SACL_PROTECTED = cpu_to_le16(0x2000), - SE_RM_CONTROL_VALID = cpu_to_le16(0x4000), - SE_SELF_RELATIVE = cpu_to_le16(0x8000) -} __attribute__ ((__packed__)); - -typedef le16 SECURITY_DESCRIPTOR_CONTROL; - -/* - * Self-relative security descriptor. Contains the owner and group SIDs as well - * as the sacl and dacl ACLs inside the security descriptor itself. - */ -typedef struct { - u8 revision; /* Revision level of the security descriptor. */ - u8 alignment; - SECURITY_DESCRIPTOR_CONTROL control; /* Flags qualifying the type of - the descriptor as well as the following fields. */ - le32 owner; /* Byte offset to a SID representing an object's - owner. If this is NULL, no owner SID is present in - the descriptor. */ - le32 group; /* Byte offset to a SID representing an object's - primary group. If this is NULL, no primary group - SID is present in the descriptor. */ - le32 sacl; /* Byte offset to a system ACL. Only valid, if - SE_SACL_PRESENT is set in the control field. If - SE_SACL_PRESENT is set but sacl is NULL, a NULL ACL - is specified. */ - le32 dacl; /* Byte offset to a discretionary ACL. Only valid, if - SE_DACL_PRESENT is set in the control field. If - SE_DACL_PRESENT is set but dacl is NULL, a NULL ACL - (unconditionally granting access) is specified. */ -/* sizeof() = 0x14 bytes */ -} __attribute__ ((__packed__)) SECURITY_DESCRIPTOR_RELATIVE; - -/* - * Absolute security descriptor. Does not contain the owner and group SIDs, nor - * the sacl and dacl ACLs inside the security descriptor. Instead, it contains - * pointers to these structures in memory. Obviously, absolute security - * descriptors are only useful for in memory representations of security - * descriptors. On disk, a self-relative security descriptor is used. - */ -typedef struct { - u8 revision; /* Revision level of the security descriptor. */ - u8 alignment; - SECURITY_DESCRIPTOR_CONTROL control; /* Flags qualifying the type of - the descriptor as well as the following fields. */ - SID *owner; /* Points to a SID representing an object's owner. If - this is NULL, no owner SID is present in the - descriptor. */ - SID *group; /* Points to a SID representing an object's primary - group. If this is NULL, no primary group SID is - present in the descriptor. */ - ACL *sacl; /* Points to a system ACL. Only valid, if - SE_SACL_PRESENT is set in the control field. If - SE_SACL_PRESENT is set but sacl is NULL, a NULL ACL - is specified. */ - ACL *dacl; /* Points to a discretionary ACL. Only valid, if - SE_DACL_PRESENT is set in the control field. If - SE_DACL_PRESENT is set but dacl is NULL, a NULL ACL - (unconditionally granting access) is specified. */ -} __attribute__ ((__packed__)) SECURITY_DESCRIPTOR; - -/* - * Current constants for security descriptors. - */ -typedef enum { - /* Current revision. */ - SECURITY_DESCRIPTOR_REVISION = 1, - SECURITY_DESCRIPTOR_REVISION1 = 1, - - /* The sizes of both the absolute and relative security descriptors is - the same as pointers, at least on ia32 architecture are 32-bit. */ - SECURITY_DESCRIPTOR_MIN_LENGTH = sizeof(SECURITY_DESCRIPTOR), -} SECURITY_DESCRIPTOR_CONSTANTS; - -/* - * Attribute: Security descriptor (0x50). A standard self-relative security - * descriptor. - * - * NOTE: Can be resident or non-resident. - * NOTE: Not used in NTFS 3.0+, as security descriptors are stored centrally - * in FILE_Secure and the correct descriptor is found using the security_id - * from the standard information attribute. - */ -typedef SECURITY_DESCRIPTOR_RELATIVE SECURITY_DESCRIPTOR_ATTR; - -/* - * On NTFS 3.0+, all security descriptors are stored in FILE_Secure. Only one - * referenced instance of each unique security descriptor is stored. - * - * FILE_Secure contains no unnamed data attribute, i.e. it has zero length. It - * does, however, contain two indexes ($SDH and $SII) as well as a named data - * stream ($SDS). - * - * Every unique security descriptor is assigned a unique security identifier - * (security_id, not to be confused with a SID). The security_id is unique for - * the NTFS volume and is used as an index into the $SII index, which maps - * security_ids to the security descriptor's storage location within the $SDS - * data attribute. The $SII index is sorted by ascending security_id. - * - * A simple hash is computed from each security descriptor. This hash is used - * as an index into the $SDH index, which maps security descriptor hashes to - * the security descriptor's storage location within the $SDS data attribute. - * The $SDH index is sorted by security descriptor hash and is stored in a B+ - * tree. When searching $SDH (with the intent of determining whether or not a - * new security descriptor is already present in the $SDS data stream), if a - * matching hash is found, but the security descriptors do not match, the - * search in the $SDH index is continued, searching for a next matching hash. - * - * When a precise match is found, the security_id coresponding to the security - * descriptor in the $SDS attribute is read from the found $SDH index entry and - * is stored in the $STANDARD_INFORMATION attribute of the file/directory to - * which the security descriptor is being applied. The $STANDARD_INFORMATION - * attribute is present in all base mft records (i.e. in all files and - * directories). - * - * If a match is not found, the security descriptor is assigned a new unique - * security_id and is added to the $SDS data attribute. Then, entries - * referencing the this security descriptor in the $SDS data attribute are - * added to the $SDH and $SII indexes. - * - * Note: Entries are never deleted from FILE_Secure, even if nothing - * references an entry any more. - */ - -/* - * This header precedes each security descriptor in the $SDS data stream. - * This is also the index entry data part of both the $SII and $SDH indexes. - */ -typedef struct { - le32 hash; /* Hash of the security descriptor. */ - le32 security_id; /* The security_id assigned to the descriptor. */ - le64 offset; /* Byte offset of this entry in the $SDS stream. */ - le32 length; /* Size in bytes of this entry in $SDS stream. */ -} __attribute__ ((__packed__)) SECURITY_DESCRIPTOR_HEADER; - -/* - * The $SDS data stream contains the security descriptors, aligned on 16-byte - * boundaries, sorted by security_id in a B+ tree. Security descriptors cannot - * cross 256kib boundaries (this restriction is imposed by the Windows cache - * manager). Each security descriptor is contained in a SDS_ENTRY structure. - * Also, each security descriptor is stored twice in the $SDS stream with a - * fixed offset of 0x40000 bytes (256kib, the Windows cache manager's max size) - * between them; i.e. if a SDS_ENTRY specifies an offset of 0x51d0, then the - * first copy of the security descriptor will be at offset 0x51d0 in the - * $SDS data stream and the second copy will be at offset 0x451d0. - */ -typedef struct { -/*Ofs*/ -/* 0 SECURITY_DESCRIPTOR_HEADER; -- Unfolded here as gcc doesn't like - unnamed structs. */ - le32 hash; /* Hash of the security descriptor. */ - le32 security_id; /* The security_id assigned to the descriptor. */ - le64 offset; /* Byte offset of this entry in the $SDS stream. */ - le32 length; /* Size in bytes of this entry in $SDS stream. */ -/* 20*/ SECURITY_DESCRIPTOR_RELATIVE sid; /* The self-relative security - descriptor. */ -} __attribute__ ((__packed__)) SDS_ENTRY; - -/* - * The index entry key used in the $SII index. The collation type is - * COLLATION_NTOFS_ULONG. - */ -typedef struct { - le32 security_id; /* The security_id assigned to the descriptor. */ -} __attribute__ ((__packed__)) SII_INDEX_KEY; - -/* - * The index entry key used in the $SDH index. The keys are sorted first by - * hash and then by security_id. The collation rule is - * COLLATION_NTOFS_SECURITY_HASH. - */ -typedef struct { - le32 hash; /* Hash of the security descriptor. */ - le32 security_id; /* The security_id assigned to the descriptor. */ -} __attribute__ ((__packed__)) SDH_INDEX_KEY; - -/* - * Attribute: Volume name (0x60). - * - * NOTE: Always resident. - * NOTE: Present only in FILE_Volume. - */ -typedef struct { - ntfschar name[0]; /* The name of the volume in Unicode. */ -} __attribute__ ((__packed__)) VOLUME_NAME; - -/* - * Possible flags for the volume (16-bit). - */ -enum { - VOLUME_IS_DIRTY = cpu_to_le16(0x0001), - VOLUME_RESIZE_LOG_FILE = cpu_to_le16(0x0002), - VOLUME_UPGRADE_ON_MOUNT = cpu_to_le16(0x0004), - VOLUME_MOUNTED_ON_NT4 = cpu_to_le16(0x0008), - - VOLUME_DELETE_USN_UNDERWAY = cpu_to_le16(0x0010), - VOLUME_REPAIR_OBJECT_ID = cpu_to_le16(0x0020), - - VOLUME_CHKDSK_UNDERWAY = cpu_to_le16(0x4000), - VOLUME_MODIFIED_BY_CHKDSK = cpu_to_le16(0x8000), - - VOLUME_FLAGS_MASK = cpu_to_le16(0xc03f), - - /* To make our life easier when checking if we must mount read-only. */ - VOLUME_MUST_MOUNT_RO_MASK = cpu_to_le16(0xc027), -} __attribute__ ((__packed__)); - -typedef le16 VOLUME_FLAGS; - -/* - * Attribute: Volume information (0x70). - * - * NOTE: Always resident. - * NOTE: Present only in FILE_Volume. - * NOTE: Windows 2000 uses NTFS 3.0 while Windows NT4 service pack 6a uses - * NTFS 1.2. I haven't personally seen other values yet. - */ -typedef struct { - le64 reserved; /* Not used (yet?). */ - u8 major_ver; /* Major version of the ntfs format. */ - u8 minor_ver; /* Minor version of the ntfs format. */ - VOLUME_FLAGS flags; /* Bit array of VOLUME_* flags. */ -} __attribute__ ((__packed__)) VOLUME_INFORMATION; - -/* - * Attribute: Data attribute (0x80). - * - * NOTE: Can be resident or non-resident. - * - * Data contents of a file (i.e. the unnamed stream) or of a named stream. - */ -typedef struct { - u8 data[0]; /* The file's data contents. */ -} __attribute__ ((__packed__)) DATA_ATTR; - -/* - * Index header flags (8-bit). - */ -enum { - /* - * When index header is in an index root attribute: - */ - SMALL_INDEX = 0, /* The index is small enough to fit inside the index - root attribute and there is no index allocation - attribute present. */ - LARGE_INDEX = 1, /* The index is too large to fit in the index root - attribute and/or an index allocation attribute is - present. */ - /* - * When index header is in an index block, i.e. is part of index - * allocation attribute: - */ - LEAF_NODE = 0, /* This is a leaf node, i.e. there are no more nodes - branching off it. */ - INDEX_NODE = 1, /* This node indexes other nodes, i.e. it is not a leaf - node. */ - NODE_MASK = 1, /* Mask for accessing the *_NODE bits. */ -} __attribute__ ((__packed__)); - -typedef u8 INDEX_HEADER_FLAGS; - -/* - * This is the header for indexes, describing the INDEX_ENTRY records, which - * follow the INDEX_HEADER. Together the index header and the index entries - * make up a complete index. - * - * IMPORTANT NOTE: The offset, length and size structure members are counted - * relative to the start of the index header structure and not relative to the - * start of the index root or index allocation structures themselves. - */ -typedef struct { - le32 entries_offset; /* Byte offset to first INDEX_ENTRY - aligned to 8-byte boundary. */ - le32 index_length; /* Data size of the index in bytes, - i.e. bytes used from allocated - size, aligned to 8-byte boundary. */ - le32 allocated_size; /* Byte size of this index (block), - multiple of 8 bytes. */ - /* NOTE: For the index root attribute, the above two numbers are always - equal, as the attribute is resident and it is resized as needed. In - the case of the index allocation attribute the attribute is not - resident and hence the allocated_size is a fixed value and must - equal the index_block_size specified by the INDEX_ROOT attribute - corresponding to the INDEX_ALLOCATION attribute this INDEX_BLOCK - belongs to. */ - INDEX_HEADER_FLAGS flags; /* Bit field of INDEX_HEADER_FLAGS. */ - u8 reserved[3]; /* Reserved/align to 8-byte boundary. */ -} __attribute__ ((__packed__)) INDEX_HEADER; - -/* - * Attribute: Index root (0x90). - * - * NOTE: Always resident. - * - * This is followed by a sequence of index entries (INDEX_ENTRY structures) - * as described by the index header. - * - * When a directory is small enough to fit inside the index root then this - * is the only attribute describing the directory. When the directory is too - * large to fit in the index root, on the other hand, two additional attributes - * are present: an index allocation attribute, containing sub-nodes of the B+ - * directory tree (see below), and a bitmap attribute, describing which virtual - * cluster numbers (vcns) in the index allocation attribute are in use by an - * index block. - * - * NOTE: The root directory (FILE_root) contains an entry for itself. Other - * directories do not contain entries for themselves, though. - */ -typedef struct { - ATTR_TYPE type; /* Type of the indexed attribute. Is - $FILE_NAME for directories, zero - for view indexes. No other values - allowed. */ - COLLATION_RULE collation_rule; /* Collation rule used to sort the - index entries. If type is $FILE_NAME, - this must be COLLATION_FILE_NAME. */ - le32 index_block_size; /* Size of each index block in bytes (in - the index allocation attribute). */ - u8 clusters_per_index_block; /* Cluster size of each index block (in - the index allocation attribute), when - an index block is >= than a cluster, - otherwise this will be the log of - the size (like how the encoding of - the mft record size and the index - record size found in the boot sector - work). Has to be a power of 2. */ - u8 reserved[3]; /* Reserved/align to 8-byte boundary. */ - INDEX_HEADER index; /* Index header describing the - following index entries. */ -} __attribute__ ((__packed__)) INDEX_ROOT; - -/* - * Attribute: Index allocation (0xa0). - * - * NOTE: Always non-resident (doesn't make sense to be resident anyway!). - * - * This is an array of index blocks. Each index block starts with an - * INDEX_BLOCK structure containing an index header, followed by a sequence of - * index entries (INDEX_ENTRY structures), as described by the INDEX_HEADER. - */ -typedef struct { -/* 0 NTFS_RECORD; -- Unfolded here as gcc doesn't like unnamed structs. */ - NTFS_RECORD_TYPE magic; /* Magic is "INDX". */ - le16 usa_ofs; /* See NTFS_RECORD definition. */ - le16 usa_count; /* See NTFS_RECORD definition. */ - -/* 8*/ sle64 lsn; /* $LogFile sequence number of the last - modification of this index block. */ -/* 16*/ leVCN index_block_vcn; /* Virtual cluster number of the index block. - If the cluster_size on the volume is <= the - index_block_size of the directory, - index_block_vcn counts in units of clusters, - and in units of sectors otherwise. */ -/* 24*/ INDEX_HEADER index; /* Describes the following index entries. */ -/* sizeof()= 40 (0x28) bytes */ -/* - * When creating the index block, we place the update sequence array at this - * offset, i.e. before we start with the index entries. This also makes sense, - * otherwise we could run into problems with the update sequence array - * containing in itself the last two bytes of a sector which would mean that - * multi sector transfer protection wouldn't work. As you can't protect data - * by overwriting it since you then can't get it back... - * When reading use the data from the ntfs record header. - */ -} __attribute__ ((__packed__)) INDEX_BLOCK; - -typedef INDEX_BLOCK INDEX_ALLOCATION; - -/* - * The system file FILE_Extend/$Reparse contains an index named $R listing - * all reparse points on the volume. The index entry keys are as defined - * below. Note, that there is no index data associated with the index entries. - * - * The index entries are sorted by the index key file_id. The collation rule is - * COLLATION_NTOFS_ULONGS. FIXME: Verify whether the reparse_tag is not the - * primary key / is not a key at all. (AIA) - */ -typedef struct { - le32 reparse_tag; /* Reparse point type (inc. flags). */ - leMFT_REF file_id; /* Mft record of the file containing the - reparse point attribute. */ -} __attribute__ ((__packed__)) REPARSE_INDEX_KEY; - -/* - * Quota flags (32-bit). - * - * The user quota flags. Names explain meaning. - */ -enum { - QUOTA_FLAG_DEFAULT_LIMITS = cpu_to_le32(0x00000001), - QUOTA_FLAG_LIMIT_REACHED = cpu_to_le32(0x00000002), - QUOTA_FLAG_ID_DELETED = cpu_to_le32(0x00000004), - - QUOTA_FLAG_USER_MASK = cpu_to_le32(0x00000007), - /* This is a bit mask for the user quota flags. */ - - /* - * These flags are only present in the quota defaults index entry, i.e. - * in the entry where owner_id = QUOTA_DEFAULTS_ID. - */ - QUOTA_FLAG_TRACKING_ENABLED = cpu_to_le32(0x00000010), - QUOTA_FLAG_ENFORCEMENT_ENABLED = cpu_to_le32(0x00000020), - QUOTA_FLAG_TRACKING_REQUESTED = cpu_to_le32(0x00000040), - QUOTA_FLAG_LOG_THRESHOLD = cpu_to_le32(0x00000080), - - QUOTA_FLAG_LOG_LIMIT = cpu_to_le32(0x00000100), - QUOTA_FLAG_OUT_OF_DATE = cpu_to_le32(0x00000200), - QUOTA_FLAG_CORRUPT = cpu_to_le32(0x00000400), - QUOTA_FLAG_PENDING_DELETES = cpu_to_le32(0x00000800), -}; - -typedef le32 QUOTA_FLAGS; - -/* - * The system file FILE_Extend/$Quota contains two indexes $O and $Q. Quotas - * are on a per volume and per user basis. - * - * The $Q index contains one entry for each existing user_id on the volume. The - * index key is the user_id of the user/group owning this quota control entry, - * i.e. the key is the owner_id. The user_id of the owner of a file, i.e. the - * owner_id, is found in the standard information attribute. The collation rule - * for $Q is COLLATION_NTOFS_ULONG. - * - * The $O index contains one entry for each user/group who has been assigned - * a quota on that volume. The index key holds the SID of the user_id the - * entry belongs to, i.e. the owner_id. The collation rule for $O is - * COLLATION_NTOFS_SID. - * - * The $O index entry data is the user_id of the user corresponding to the SID. - * This user_id is used as an index into $Q to find the quota control entry - * associated with the SID. - * - * The $Q index entry data is the quota control entry and is defined below. - */ -typedef struct { - le32 version; /* Currently equals 2. */ - QUOTA_FLAGS flags; /* Flags describing this quota entry. */ - le64 bytes_used; /* How many bytes of the quota are in use. */ - sle64 change_time; /* Last time this quota entry was changed. */ - sle64 threshold; /* Soft quota (-1 if not limited). */ - sle64 limit; /* Hard quota (-1 if not limited). */ - sle64 exceeded_time; /* How long the soft quota has been exceeded. */ - SID sid; /* The SID of the user/object associated with - this quota entry. Equals zero for the quota - defaults entry (and in fact on a WinXP - volume, it is not present at all). */ -} __attribute__ ((__packed__)) QUOTA_CONTROL_ENTRY; - -/* - * Predefined owner_id values (32-bit). - */ -enum { - QUOTA_INVALID_ID = cpu_to_le32(0x00000000), - QUOTA_DEFAULTS_ID = cpu_to_le32(0x00000001), - QUOTA_FIRST_USER_ID = cpu_to_le32(0x00000100), -}; - -/* - * Current constants for quota control entries. - */ -typedef enum { - /* Current version. */ - QUOTA_VERSION = 2, -} QUOTA_CONTROL_ENTRY_CONSTANTS; - -/* - * Index entry flags (16-bit). - */ -enum { - INDEX_ENTRY_NODE = cpu_to_le16(1), /* This entry contains a - sub-node, i.e. a reference to an index block in form of - a virtual cluster number (see below). */ - INDEX_ENTRY_END = cpu_to_le16(2), /* This signifies the last - entry in an index block. The index entry does not - represent a file but it can point to a sub-node. */ - - INDEX_ENTRY_SPACE_FILLER = cpu_to_le16(0xffff), /* gcc: Force - enum bit width to 16-bit. */ -} __attribute__ ((__packed__)); - -typedef le16 INDEX_ENTRY_FLAGS; - -/* - * This the index entry header (see below). - */ -typedef struct { -/* 0*/ union { - struct { /* Only valid when INDEX_ENTRY_END is not set. */ - leMFT_REF indexed_file; /* The mft reference of the file - described by this index - entry. Used for directory - indexes. */ - } __attribute__ ((__packed__)) dir; - struct { /* Used for views/indexes to find the entry's data. */ - le16 data_offset; /* Data byte offset from this - INDEX_ENTRY. Follows the - index key. */ - le16 data_length; /* Data length in bytes. */ - le32 reservedV; /* Reserved (zero). */ - } __attribute__ ((__packed__)) vi; - } __attribute__ ((__packed__)) data; -/* 8*/ le16 length; /* Byte size of this index entry, multiple of - 8-bytes. */ -/* 10*/ le16 key_length; /* Byte size of the key value, which is in the - index entry. It follows field reserved. Not - multiple of 8-bytes. */ -/* 12*/ INDEX_ENTRY_FLAGS flags; /* Bit field of INDEX_ENTRY_* flags. */ -/* 14*/ le16 reserved; /* Reserved/align to 8-byte boundary. */ -/* sizeof() = 16 bytes */ -} __attribute__ ((__packed__)) INDEX_ENTRY_HEADER; - -/* - * This is an index entry. A sequence of such entries follows each INDEX_HEADER - * structure. Together they make up a complete index. The index follows either - * an index root attribute or an index allocation attribute. - * - * NOTE: Before NTFS 3.0 only filename attributes were indexed. - */ -typedef struct { -/*Ofs*/ -/* 0 INDEX_ENTRY_HEADER; -- Unfolded here as gcc dislikes unnamed structs. */ - union { - struct { /* Only valid when INDEX_ENTRY_END is not set. */ - leMFT_REF indexed_file; /* The mft reference of the file - described by this index - entry. Used for directory - indexes. */ - } __attribute__ ((__packed__)) dir; - struct { /* Used for views/indexes to find the entry's data. */ - le16 data_offset; /* Data byte offset from this - INDEX_ENTRY. Follows the - index key. */ - le16 data_length; /* Data length in bytes. */ - le32 reservedV; /* Reserved (zero). */ - } __attribute__ ((__packed__)) vi; - } __attribute__ ((__packed__)) data; - le16 length; /* Byte size of this index entry, multiple of - 8-bytes. */ - le16 key_length; /* Byte size of the key value, which is in the - index entry. It follows field reserved. Not - multiple of 8-bytes. */ - INDEX_ENTRY_FLAGS flags; /* Bit field of INDEX_ENTRY_* flags. */ - le16 reserved; /* Reserved/align to 8-byte boundary. */ - -/* 16*/ union { /* The key of the indexed attribute. NOTE: Only present - if INDEX_ENTRY_END bit in flags is not set. NOTE: On - NTFS versions before 3.0 the only valid key is the - FILE_NAME_ATTR. On NTFS 3.0+ the following - additional index keys are defined: */ - FILE_NAME_ATTR file_name;/* $I30 index in directories. */ - SII_INDEX_KEY sii; /* $SII index in $Secure. */ - SDH_INDEX_KEY sdh; /* $SDH index in $Secure. */ - GUID object_id; /* $O index in FILE_Extend/$ObjId: The - object_id of the mft record found in - the data part of the index. */ - REPARSE_INDEX_KEY reparse; /* $R index in - FILE_Extend/$Reparse. */ - SID sid; /* $O index in FILE_Extend/$Quota: - SID of the owner of the user_id. */ - le32 owner_id; /* $Q index in FILE_Extend/$Quota: - user_id of the owner of the quota - control entry in the data part of - the index. */ - } __attribute__ ((__packed__)) key; - /* The (optional) index data is inserted here when creating. */ - // leVCN vcn; /* If INDEX_ENTRY_NODE bit in flags is set, the last - // eight bytes of this index entry contain the virtual - // cluster number of the index block that holds the - // entries immediately preceding the current entry (the - // vcn references the corresponding cluster in the data - // of the non-resident index allocation attribute). If - // the key_length is zero, then the vcn immediately - // follows the INDEX_ENTRY_HEADER. Regardless of - // key_length, the address of the 8-byte boundary - // aligned vcn of INDEX_ENTRY{_HEADER} *ie is given by - // (char*)ie + le16_to_cpu(ie*)->length) - sizeof(VCN), - // where sizeof(VCN) can be hardcoded as 8 if wanted. */ -} __attribute__ ((__packed__)) INDEX_ENTRY; - -/* - * Attribute: Bitmap (0xb0). - * - * Contains an array of bits (aka a bitfield). - * - * When used in conjunction with the index allocation attribute, each bit - * corresponds to one index block within the index allocation attribute. Thus - * the number of bits in the bitmap * index block size / cluster size is the - * number of clusters in the index allocation attribute. - */ -typedef struct { - u8 bitmap[0]; /* Array of bits. */ -} __attribute__ ((__packed__)) BITMAP_ATTR; - -/* - * The reparse point tag defines the type of the reparse point. It also - * includes several flags, which further describe the reparse point. - * - * The reparse point tag is an unsigned 32-bit value divided in three parts: - * - * 1. The least significant 16 bits (i.e. bits 0 to 15) specifiy the type of - * the reparse point. - * 2. The 13 bits after this (i.e. bits 16 to 28) are reserved for future use. - * 3. The most significant three bits are flags describing the reparse point. - * They are defined as follows: - * bit 29: Name surrogate bit. If set, the filename is an alias for - * another object in the system. - * bit 30: High-latency bit. If set, accessing the first byte of data will - * be slow. (E.g. the data is stored on a tape drive.) - * bit 31: Microsoft bit. If set, the tag is owned by Microsoft. User - * defined tags have to use zero here. - * - * These are the predefined reparse point tags: - */ -enum { - IO_REPARSE_TAG_IS_ALIAS = cpu_to_le32(0x20000000), - IO_REPARSE_TAG_IS_HIGH_LATENCY = cpu_to_le32(0x40000000), - IO_REPARSE_TAG_IS_MICROSOFT = cpu_to_le32(0x80000000), - - IO_REPARSE_TAG_RESERVED_ZERO = cpu_to_le32(0x00000000), - IO_REPARSE_TAG_RESERVED_ONE = cpu_to_le32(0x00000001), - IO_REPARSE_TAG_RESERVED_RANGE = cpu_to_le32(0x00000001), - - IO_REPARSE_TAG_NSS = cpu_to_le32(0x68000005), - IO_REPARSE_TAG_NSS_RECOVER = cpu_to_le32(0x68000006), - IO_REPARSE_TAG_SIS = cpu_to_le32(0x68000007), - IO_REPARSE_TAG_DFS = cpu_to_le32(0x68000008), - - IO_REPARSE_TAG_MOUNT_POINT = cpu_to_le32(0x88000003), - - IO_REPARSE_TAG_HSM = cpu_to_le32(0xa8000004), - - IO_REPARSE_TAG_SYMBOLIC_LINK = cpu_to_le32(0xe8000000), - - IO_REPARSE_TAG_VALID_VALUES = cpu_to_le32(0xe000ffff), -}; - -/* - * Attribute: Reparse point (0xc0). - * - * NOTE: Can be resident or non-resident. - */ -typedef struct { - le32 reparse_tag; /* Reparse point type (inc. flags). */ - le16 reparse_data_length; /* Byte size of reparse data. */ - le16 reserved; /* Align to 8-byte boundary. */ - u8 reparse_data[0]; /* Meaning depends on reparse_tag. */ -} __attribute__ ((__packed__)) REPARSE_POINT; - -/* - * Attribute: Extended attribute (EA) information (0xd0). - * - * NOTE: Always resident. (Is this true???) - */ -typedef struct { - le16 ea_length; /* Byte size of the packed extended - attributes. */ - le16 need_ea_count; /* The number of extended attributes which have - the NEED_EA bit set. */ - le32 ea_query_length; /* Byte size of the buffer required to query - the extended attributes when calling - ZwQueryEaFile() in Windows NT/2k. I.e. the - byte size of the unpacked extended - attributes. */ -} __attribute__ ((__packed__)) EA_INFORMATION; - -/* - * Extended attribute flags (8-bit). - */ -enum { - NEED_EA = 0x80 /* If set the file to which the EA belongs - cannot be interpreted without understanding - the associates extended attributes. */ -} __attribute__ ((__packed__)); - -typedef u8 EA_FLAGS; - -/* - * Attribute: Extended attribute (EA) (0xe0). - * - * NOTE: Can be resident or non-resident. - * - * Like the attribute list and the index buffer list, the EA attribute value is - * a sequence of EA_ATTR variable length records. - */ -typedef struct { - le32 next_entry_offset; /* Offset to the next EA_ATTR. */ - EA_FLAGS flags; /* Flags describing the EA. */ - u8 ea_name_length; /* Length of the name of the EA in bytes - excluding the '\0' byte terminator. */ - le16 ea_value_length; /* Byte size of the EA's value. */ - u8 ea_name[0]; /* Name of the EA. Note this is ASCII, not - Unicode and it is zero terminated. */ - u8 ea_value[0]; /* The value of the EA. Immediately follows - the name. */ -} __attribute__ ((__packed__)) EA_ATTR; - -/* - * Attribute: Property set (0xf0). - * - * Intended to support Native Structure Storage (NSS) - a feature removed from - * NTFS 3.0 during beta testing. - */ -typedef struct { - /* Irrelevant as feature unused. */ -} __attribute__ ((__packed__)) PROPERTY_SET; - -/* - * Attribute: Logged utility stream (0x100). - * - * NOTE: Can be resident or non-resident. - * - * Operations on this attribute are logged to the journal ($LogFile) like - * normal metadata changes. - * - * Used by the Encrypting File System (EFS). All encrypted files have this - * attribute with the name $EFS. - */ -typedef struct { - /* Can be anything the creator chooses. */ - /* EFS uses it as follows: */ - // FIXME: Type this info, verifying it along the way. (AIA) -} __attribute__ ((__packed__)) LOGGED_UTILITY_STREAM, EFS_ATTR; - -#endif /* _LINUX_NTFS_LAYOUT_H */ diff --git a/fs/ntfs/lcnalloc.c b/fs/ntfs/lcnalloc.c deleted file mode 100644 index eda9972e6159..000000000000 --- a/fs/ntfs/lcnalloc.c +++ /dev/null @@ -1,1000 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * lcnalloc.c - Cluster (de)allocation code. Part of the Linux-NTFS project. - * - * Copyright (c) 2004-2005 Anton Altaparmakov - */ - -#ifdef NTFS_RW - -#include - -#include "lcnalloc.h" -#include "debug.h" -#include "bitmap.h" -#include "inode.h" -#include "volume.h" -#include "attrib.h" -#include "malloc.h" -#include "aops.h" -#include "ntfs.h" - -/** - * ntfs_cluster_free_from_rl_nolock - free clusters from runlist - * @vol: mounted ntfs volume on which to free the clusters - * @rl: runlist describing the clusters to free - * - * Free all the clusters described by the runlist @rl on the volume @vol. In - * the case of an error being returned, at least some of the clusters were not - * freed. - * - * Return 0 on success and -errno on error. - * - * Locking: - The volume lcn bitmap must be locked for writing on entry and is - * left locked on return. - */ -int ntfs_cluster_free_from_rl_nolock(ntfs_volume *vol, - const runlist_element *rl) -{ - struct inode *lcnbmp_vi = vol->lcnbmp_ino; - int ret = 0; - - ntfs_debug("Entering."); - if (!rl) - return 0; - for (; rl->length; rl++) { - int err; - - if (rl->lcn < 0) - continue; - err = ntfs_bitmap_clear_run(lcnbmp_vi, rl->lcn, rl->length); - if (unlikely(err && (!ret || ret == -ENOMEM) && ret != err)) - ret = err; - } - ntfs_debug("Done."); - return ret; -} - -/** - * ntfs_cluster_alloc - allocate clusters on an ntfs volume - * @vol: mounted ntfs volume on which to allocate the clusters - * @start_vcn: vcn to use for the first allocated cluster - * @count: number of clusters to allocate - * @start_lcn: starting lcn at which to allocate the clusters (or -1 if none) - * @zone: zone from which to allocate the clusters - * @is_extension: if 'true', this is an attribute extension - * - * Allocate @count clusters preferably starting at cluster @start_lcn or at the - * current allocator position if @start_lcn is -1, on the mounted ntfs volume - * @vol. @zone is either DATA_ZONE for allocation of normal clusters or - * MFT_ZONE for allocation of clusters for the master file table, i.e. the - * $MFT/$DATA attribute. - * - * @start_vcn specifies the vcn of the first allocated cluster. This makes - * merging the resulting runlist with the old runlist easier. - * - * If @is_extension is 'true', the caller is allocating clusters to extend an - * attribute and if it is 'false', the caller is allocating clusters to fill a - * hole in an attribute. Practically the difference is that if @is_extension - * is 'true' the returned runlist will be terminated with LCN_ENOENT and if - * @is_extension is 'false' the runlist will be terminated with - * LCN_RL_NOT_MAPPED. - * - * You need to check the return value with IS_ERR(). If this is false, the - * function was successful and the return value is a runlist describing the - * allocated cluster(s). If IS_ERR() is true, the function failed and - * PTR_ERR() gives you the error code. - * - * Notes on the allocation algorithm - * ================================= - * - * There are two data zones. First is the area between the end of the mft zone - * and the end of the volume, and second is the area between the start of the - * volume and the start of the mft zone. On unmodified/standard NTFS 1.x - * volumes, the second data zone does not exist due to the mft zone being - * expanded to cover the start of the volume in order to reserve space for the - * mft bitmap attribute. - * - * This is not the prettiest function but the complexity stems from the need of - * implementing the mft vs data zoned approach and from the fact that we have - * access to the lcn bitmap in portions of up to 8192 bytes at a time, so we - * need to cope with crossing over boundaries of two buffers. Further, the - * fact that the allocator allows for caller supplied hints as to the location - * of where allocation should begin and the fact that the allocator keeps track - * of where in the data zones the next natural allocation should occur, - * contribute to the complexity of the function. But it should all be - * worthwhile, because this allocator should: 1) be a full implementation of - * the MFT zone approach used by Windows NT, 2) cause reduction in - * fragmentation, and 3) be speedy in allocations (the code is not optimized - * for speed, but the algorithm is, so further speed improvements are probably - * possible). - * - * FIXME: We should be monitoring cluster allocation and increment the MFT zone - * size dynamically but this is something for the future. We will just cause - * heavier fragmentation by not doing it and I am not even sure Windows would - * grow the MFT zone dynamically, so it might even be correct not to do this. - * The overhead in doing dynamic MFT zone expansion would be very large and - * unlikely worth the effort. (AIA) - * - * TODO: I have added in double the required zone position pointer wrap around - * logic which can be optimized to having only one of the two logic sets. - * However, having the double logic will work fine, but if we have only one of - * the sets and we get it wrong somewhere, then we get into trouble, so - * removing the duplicate logic requires _very_ careful consideration of _all_ - * possible code paths. So at least for now, I am leaving the double logic - - * better safe than sorry... (AIA) - * - * Locking: - The volume lcn bitmap must be unlocked on entry and is unlocked - * on return. - * - This function takes the volume lcn bitmap lock for writing and - * modifies the bitmap contents. - */ -runlist_element *ntfs_cluster_alloc(ntfs_volume *vol, const VCN start_vcn, - const s64 count, const LCN start_lcn, - const NTFS_CLUSTER_ALLOCATION_ZONES zone, - const bool is_extension) -{ - LCN zone_start, zone_end, bmp_pos, bmp_initial_pos, last_read_pos, lcn; - LCN prev_lcn = 0, prev_run_len = 0, mft_zone_size; - s64 clusters; - loff_t i_size; - struct inode *lcnbmp_vi; - runlist_element *rl = NULL; - struct address_space *mapping; - struct page *page = NULL; - u8 *buf, *byte; - int err = 0, rlpos, rlsize, buf_size; - u8 pass, done_zones, search_zone, need_writeback = 0, bit; - - ntfs_debug("Entering for start_vcn 0x%llx, count 0x%llx, start_lcn " - "0x%llx, zone %s_ZONE.", (unsigned long long)start_vcn, - (unsigned long long)count, - (unsigned long long)start_lcn, - zone == MFT_ZONE ? "MFT" : "DATA"); - BUG_ON(!vol); - lcnbmp_vi = vol->lcnbmp_ino; - BUG_ON(!lcnbmp_vi); - BUG_ON(start_vcn < 0); - BUG_ON(count < 0); - BUG_ON(start_lcn < -1); - BUG_ON(zone < FIRST_ZONE); - BUG_ON(zone > LAST_ZONE); - - /* Return NULL if @count is zero. */ - if (!count) - return NULL; - /* Take the lcnbmp lock for writing. */ - down_write(&vol->lcnbmp_lock); - /* - * If no specific @start_lcn was requested, use the current data zone - * position, otherwise use the requested @start_lcn but make sure it - * lies outside the mft zone. Also set done_zones to 0 (no zones done) - * and pass depending on whether we are starting inside a zone (1) or - * at the beginning of a zone (2). If requesting from the MFT_ZONE, - * we either start at the current position within the mft zone or at - * the specified position. If the latter is out of bounds then we start - * at the beginning of the MFT_ZONE. - */ - done_zones = 0; - pass = 1; - /* - * zone_start and zone_end are the current search range. search_zone - * is 1 for mft zone, 2 for data zone 1 (end of mft zone till end of - * volume) and 4 for data zone 2 (start of volume till start of mft - * zone). - */ - zone_start = start_lcn; - if (zone_start < 0) { - if (zone == DATA_ZONE) - zone_start = vol->data1_zone_pos; - else - zone_start = vol->mft_zone_pos; - if (!zone_start) { - /* - * Zone starts at beginning of volume which means a - * single pass is sufficient. - */ - pass = 2; - } - } else if (zone == DATA_ZONE && zone_start >= vol->mft_zone_start && - zone_start < vol->mft_zone_end) { - zone_start = vol->mft_zone_end; - /* - * Starting at beginning of data1_zone which means a single - * pass in this zone is sufficient. - */ - pass = 2; - } else if (zone == MFT_ZONE && (zone_start < vol->mft_zone_start || - zone_start >= vol->mft_zone_end)) { - zone_start = vol->mft_lcn; - if (!vol->mft_zone_end) - zone_start = 0; - /* - * Starting at beginning of volume which means a single pass - * is sufficient. - */ - pass = 2; - } - if (zone == MFT_ZONE) { - zone_end = vol->mft_zone_end; - search_zone = 1; - } else /* if (zone == DATA_ZONE) */ { - /* Skip searching the mft zone. */ - done_zones |= 1; - if (zone_start >= vol->mft_zone_end) { - zone_end = vol->nr_clusters; - search_zone = 2; - } else { - zone_end = vol->mft_zone_start; - search_zone = 4; - } - } - /* - * bmp_pos is the current bit position inside the bitmap. We use - * bmp_initial_pos to determine whether or not to do a zone switch. - */ - bmp_pos = bmp_initial_pos = zone_start; - - /* Loop until all clusters are allocated, i.e. clusters == 0. */ - clusters = count; - rlpos = rlsize = 0; - mapping = lcnbmp_vi->i_mapping; - i_size = i_size_read(lcnbmp_vi); - while (1) { - ntfs_debug("Start of outer while loop: done_zones 0x%x, " - "search_zone %i, pass %i, zone_start 0x%llx, " - "zone_end 0x%llx, bmp_initial_pos 0x%llx, " - "bmp_pos 0x%llx, rlpos %i, rlsize %i.", - done_zones, search_zone, pass, - (unsigned long long)zone_start, - (unsigned long long)zone_end, - (unsigned long long)bmp_initial_pos, - (unsigned long long)bmp_pos, rlpos, rlsize); - /* Loop until we run out of free clusters. */ - last_read_pos = bmp_pos >> 3; - ntfs_debug("last_read_pos 0x%llx.", - (unsigned long long)last_read_pos); - if (last_read_pos > i_size) { - ntfs_debug("End of attribute reached. " - "Skipping to zone_pass_done."); - goto zone_pass_done; - } - if (likely(page)) { - if (need_writeback) { - ntfs_debug("Marking page dirty."); - flush_dcache_page(page); - set_page_dirty(page); - need_writeback = 0; - } - ntfs_unmap_page(page); - } - page = ntfs_map_page(mapping, last_read_pos >> - PAGE_SHIFT); - if (IS_ERR(page)) { - err = PTR_ERR(page); - ntfs_error(vol->sb, "Failed to map page."); - goto out; - } - buf_size = last_read_pos & ~PAGE_MASK; - buf = page_address(page) + buf_size; - buf_size = PAGE_SIZE - buf_size; - if (unlikely(last_read_pos + buf_size > i_size)) - buf_size = i_size - last_read_pos; - buf_size <<= 3; - lcn = bmp_pos & 7; - bmp_pos &= ~(LCN)7; - ntfs_debug("Before inner while loop: buf_size %i, lcn 0x%llx, " - "bmp_pos 0x%llx, need_writeback %i.", buf_size, - (unsigned long long)lcn, - (unsigned long long)bmp_pos, need_writeback); - while (lcn < buf_size && lcn + bmp_pos < zone_end) { - byte = buf + (lcn >> 3); - ntfs_debug("In inner while loop: buf_size %i, " - "lcn 0x%llx, bmp_pos 0x%llx, " - "need_writeback %i, byte ofs 0x%x, " - "*byte 0x%x.", buf_size, - (unsigned long long)lcn, - (unsigned long long)bmp_pos, - need_writeback, - (unsigned int)(lcn >> 3), - (unsigned int)*byte); - /* Skip full bytes. */ - if (*byte == 0xff) { - lcn = (lcn + 8) & ~(LCN)7; - ntfs_debug("Continuing while loop 1."); - continue; - } - bit = 1 << (lcn & 7); - ntfs_debug("bit 0x%x.", bit); - /* If the bit is already set, go onto the next one. */ - if (*byte & bit) { - lcn++; - ntfs_debug("Continuing while loop 2."); - continue; - } - /* - * Allocate more memory if needed, including space for - * the terminator element. - * ntfs_malloc_nofs() operates on whole pages only. - */ - if ((rlpos + 2) * sizeof(*rl) > rlsize) { - runlist_element *rl2; - - ntfs_debug("Reallocating memory."); - if (!rl) - ntfs_debug("First free bit is at LCN " - "0x%llx.", - (unsigned long long) - (lcn + bmp_pos)); - rl2 = ntfs_malloc_nofs(rlsize + (int)PAGE_SIZE); - if (unlikely(!rl2)) { - err = -ENOMEM; - ntfs_error(vol->sb, "Failed to " - "allocate memory."); - goto out; - } - memcpy(rl2, rl, rlsize); - ntfs_free(rl); - rl = rl2; - rlsize += PAGE_SIZE; - ntfs_debug("Reallocated memory, rlsize 0x%x.", - rlsize); - } - /* Allocate the bitmap bit. */ - *byte |= bit; - /* We need to write this bitmap page to disk. */ - need_writeback = 1; - ntfs_debug("*byte 0x%x, need_writeback is set.", - (unsigned int)*byte); - /* - * Coalesce with previous run if adjacent LCNs. - * Otherwise, append a new run. - */ - ntfs_debug("Adding run (lcn 0x%llx, len 0x%llx), " - "prev_lcn 0x%llx, lcn 0x%llx, " - "bmp_pos 0x%llx, prev_run_len 0x%llx, " - "rlpos %i.", - (unsigned long long)(lcn + bmp_pos), - 1ULL, (unsigned long long)prev_lcn, - (unsigned long long)lcn, - (unsigned long long)bmp_pos, - (unsigned long long)prev_run_len, - rlpos); - if (prev_lcn == lcn + bmp_pos - prev_run_len && rlpos) { - ntfs_debug("Coalescing to run (lcn 0x%llx, " - "len 0x%llx).", - (unsigned long long) - rl[rlpos - 1].lcn, - (unsigned long long) - rl[rlpos - 1].length); - rl[rlpos - 1].length = ++prev_run_len; - ntfs_debug("Run now (lcn 0x%llx, len 0x%llx), " - "prev_run_len 0x%llx.", - (unsigned long long) - rl[rlpos - 1].lcn, - (unsigned long long) - rl[rlpos - 1].length, - (unsigned long long) - prev_run_len); - } else { - if (likely(rlpos)) { - ntfs_debug("Adding new run, (previous " - "run lcn 0x%llx, " - "len 0x%llx).", - (unsigned long long) - rl[rlpos - 1].lcn, - (unsigned long long) - rl[rlpos - 1].length); - rl[rlpos].vcn = rl[rlpos - 1].vcn + - prev_run_len; - } else { - ntfs_debug("Adding new run, is first " - "run."); - rl[rlpos].vcn = start_vcn; - } - rl[rlpos].lcn = prev_lcn = lcn + bmp_pos; - rl[rlpos].length = prev_run_len = 1; - rlpos++; - } - /* Done? */ - if (!--clusters) { - LCN tc; - /* - * Update the current zone position. Positions - * of already scanned zones have been updated - * during the respective zone switches. - */ - tc = lcn + bmp_pos + 1; - ntfs_debug("Done. Updating current zone " - "position, tc 0x%llx, " - "search_zone %i.", - (unsigned long long)tc, - search_zone); - switch (search_zone) { - case 1: - ntfs_debug("Before checks, " - "vol->mft_zone_pos " - "0x%llx.", - (unsigned long long) - vol->mft_zone_pos); - if (tc >= vol->mft_zone_end) { - vol->mft_zone_pos = - vol->mft_lcn; - if (!vol->mft_zone_end) - vol->mft_zone_pos = 0; - } else if ((bmp_initial_pos >= - vol->mft_zone_pos || - tc > vol->mft_zone_pos) - && tc >= vol->mft_lcn) - vol->mft_zone_pos = tc; - ntfs_debug("After checks, " - "vol->mft_zone_pos " - "0x%llx.", - (unsigned long long) - vol->mft_zone_pos); - break; - case 2: - ntfs_debug("Before checks, " - "vol->data1_zone_pos " - "0x%llx.", - (unsigned long long) - vol->data1_zone_pos); - if (tc >= vol->nr_clusters) - vol->data1_zone_pos = - vol->mft_zone_end; - else if ((bmp_initial_pos >= - vol->data1_zone_pos || - tc > vol->data1_zone_pos) - && tc >= vol->mft_zone_end) - vol->data1_zone_pos = tc; - ntfs_debug("After checks, " - "vol->data1_zone_pos " - "0x%llx.", - (unsigned long long) - vol->data1_zone_pos); - break; - case 4: - ntfs_debug("Before checks, " - "vol->data2_zone_pos " - "0x%llx.", - (unsigned long long) - vol->data2_zone_pos); - if (tc >= vol->mft_zone_start) - vol->data2_zone_pos = 0; - else if (bmp_initial_pos >= - vol->data2_zone_pos || - tc > vol->data2_zone_pos) - vol->data2_zone_pos = tc; - ntfs_debug("After checks, " - "vol->data2_zone_pos " - "0x%llx.", - (unsigned long long) - vol->data2_zone_pos); - break; - default: - BUG(); - } - ntfs_debug("Finished. Going to out."); - goto out; - } - lcn++; - } - bmp_pos += buf_size; - ntfs_debug("After inner while loop: buf_size 0x%x, lcn " - "0x%llx, bmp_pos 0x%llx, need_writeback %i.", - buf_size, (unsigned long long)lcn, - (unsigned long long)bmp_pos, need_writeback); - if (bmp_pos < zone_end) { - ntfs_debug("Continuing outer while loop, " - "bmp_pos 0x%llx, zone_end 0x%llx.", - (unsigned long long)bmp_pos, - (unsigned long long)zone_end); - continue; - } -zone_pass_done: /* Finished with the current zone pass. */ - ntfs_debug("At zone_pass_done, pass %i.", pass); - if (pass == 1) { - /* - * Now do pass 2, scanning the first part of the zone - * we omitted in pass 1. - */ - pass = 2; - zone_end = zone_start; - switch (search_zone) { - case 1: /* mft_zone */ - zone_start = vol->mft_zone_start; - break; - case 2: /* data1_zone */ - zone_start = vol->mft_zone_end; - break; - case 4: /* data2_zone */ - zone_start = 0; - break; - default: - BUG(); - } - /* Sanity check. */ - if (zone_end < zone_start) - zone_end = zone_start; - bmp_pos = zone_start; - ntfs_debug("Continuing outer while loop, pass 2, " - "zone_start 0x%llx, zone_end 0x%llx, " - "bmp_pos 0x%llx.", - (unsigned long long)zone_start, - (unsigned long long)zone_end, - (unsigned long long)bmp_pos); - continue; - } /* pass == 2 */ -done_zones_check: - ntfs_debug("At done_zones_check, search_zone %i, done_zones " - "before 0x%x, done_zones after 0x%x.", - search_zone, done_zones, - done_zones | search_zone); - done_zones |= search_zone; - if (done_zones < 7) { - ntfs_debug("Switching zone."); - /* Now switch to the next zone we haven't done yet. */ - pass = 1; - switch (search_zone) { - case 1: - ntfs_debug("Switching from mft zone to data1 " - "zone."); - /* Update mft zone position. */ - if (rlpos) { - LCN tc; - - ntfs_debug("Before checks, " - "vol->mft_zone_pos " - "0x%llx.", - (unsigned long long) - vol->mft_zone_pos); - tc = rl[rlpos - 1].lcn + - rl[rlpos - 1].length; - if (tc >= vol->mft_zone_end) { - vol->mft_zone_pos = - vol->mft_lcn; - if (!vol->mft_zone_end) - vol->mft_zone_pos = 0; - } else if ((bmp_initial_pos >= - vol->mft_zone_pos || - tc > vol->mft_zone_pos) - && tc >= vol->mft_lcn) - vol->mft_zone_pos = tc; - ntfs_debug("After checks, " - "vol->mft_zone_pos " - "0x%llx.", - (unsigned long long) - vol->mft_zone_pos); - } - /* Switch from mft zone to data1 zone. */ -switch_to_data1_zone: search_zone = 2; - zone_start = bmp_initial_pos = - vol->data1_zone_pos; - zone_end = vol->nr_clusters; - if (zone_start == vol->mft_zone_end) - pass = 2; - if (zone_start >= zone_end) { - vol->data1_zone_pos = zone_start = - vol->mft_zone_end; - pass = 2; - } - break; - case 2: - ntfs_debug("Switching from data1 zone to " - "data2 zone."); - /* Update data1 zone position. */ - if (rlpos) { - LCN tc; - - ntfs_debug("Before checks, " - "vol->data1_zone_pos " - "0x%llx.", - (unsigned long long) - vol->data1_zone_pos); - tc = rl[rlpos - 1].lcn + - rl[rlpos - 1].length; - if (tc >= vol->nr_clusters) - vol->data1_zone_pos = - vol->mft_zone_end; - else if ((bmp_initial_pos >= - vol->data1_zone_pos || - tc > vol->data1_zone_pos) - && tc >= vol->mft_zone_end) - vol->data1_zone_pos = tc; - ntfs_debug("After checks, " - "vol->data1_zone_pos " - "0x%llx.", - (unsigned long long) - vol->data1_zone_pos); - } - /* Switch from data1 zone to data2 zone. */ - search_zone = 4; - zone_start = bmp_initial_pos = - vol->data2_zone_pos; - zone_end = vol->mft_zone_start; - if (!zone_start) - pass = 2; - if (zone_start >= zone_end) { - vol->data2_zone_pos = zone_start = - bmp_initial_pos = 0; - pass = 2; - } - break; - case 4: - ntfs_debug("Switching from data2 zone to " - "data1 zone."); - /* Update data2 zone position. */ - if (rlpos) { - LCN tc; - - ntfs_debug("Before checks, " - "vol->data2_zone_pos " - "0x%llx.", - (unsigned long long) - vol->data2_zone_pos); - tc = rl[rlpos - 1].lcn + - rl[rlpos - 1].length; - if (tc >= vol->mft_zone_start) - vol->data2_zone_pos = 0; - else if (bmp_initial_pos >= - vol->data2_zone_pos || - tc > vol->data2_zone_pos) - vol->data2_zone_pos = tc; - ntfs_debug("After checks, " - "vol->data2_zone_pos " - "0x%llx.", - (unsigned long long) - vol->data2_zone_pos); - } - /* Switch from data2 zone to data1 zone. */ - goto switch_to_data1_zone; - default: - BUG(); - } - ntfs_debug("After zone switch, search_zone %i, " - "pass %i, bmp_initial_pos 0x%llx, " - "zone_start 0x%llx, zone_end 0x%llx.", - search_zone, pass, - (unsigned long long)bmp_initial_pos, - (unsigned long long)zone_start, - (unsigned long long)zone_end); - bmp_pos = zone_start; - if (zone_start == zone_end) { - ntfs_debug("Empty zone, going to " - "done_zones_check."); - /* Empty zone. Don't bother searching it. */ - goto done_zones_check; - } - ntfs_debug("Continuing outer while loop."); - continue; - } /* done_zones == 7 */ - ntfs_debug("All zones are finished."); - /* - * All zones are finished! If DATA_ZONE, shrink mft zone. If - * MFT_ZONE, we have really run out of space. - */ - mft_zone_size = vol->mft_zone_end - vol->mft_zone_start; - ntfs_debug("vol->mft_zone_start 0x%llx, vol->mft_zone_end " - "0x%llx, mft_zone_size 0x%llx.", - (unsigned long long)vol->mft_zone_start, - (unsigned long long)vol->mft_zone_end, - (unsigned long long)mft_zone_size); - if (zone == MFT_ZONE || mft_zone_size <= 0) { - ntfs_debug("No free clusters left, going to out."); - /* Really no more space left on device. */ - err = -ENOSPC; - goto out; - } /* zone == DATA_ZONE && mft_zone_size > 0 */ - ntfs_debug("Shrinking mft zone."); - zone_end = vol->mft_zone_end; - mft_zone_size >>= 1; - if (mft_zone_size > 0) - vol->mft_zone_end = vol->mft_zone_start + mft_zone_size; - else /* mft zone and data2 zone no longer exist. */ - vol->data2_zone_pos = vol->mft_zone_start = - vol->mft_zone_end = 0; - if (vol->mft_zone_pos >= vol->mft_zone_end) { - vol->mft_zone_pos = vol->mft_lcn; - if (!vol->mft_zone_end) - vol->mft_zone_pos = 0; - } - bmp_pos = zone_start = bmp_initial_pos = - vol->data1_zone_pos = vol->mft_zone_end; - search_zone = 2; - pass = 2; - done_zones &= ~2; - ntfs_debug("After shrinking mft zone, mft_zone_size 0x%llx, " - "vol->mft_zone_start 0x%llx, " - "vol->mft_zone_end 0x%llx, " - "vol->mft_zone_pos 0x%llx, search_zone 2, " - "pass 2, dones_zones 0x%x, zone_start 0x%llx, " - "zone_end 0x%llx, vol->data1_zone_pos 0x%llx, " - "continuing outer while loop.", - (unsigned long long)mft_zone_size, - (unsigned long long)vol->mft_zone_start, - (unsigned long long)vol->mft_zone_end, - (unsigned long long)vol->mft_zone_pos, - done_zones, (unsigned long long)zone_start, - (unsigned long long)zone_end, - (unsigned long long)vol->data1_zone_pos); - } - ntfs_debug("After outer while loop."); -out: - ntfs_debug("At out."); - /* Add runlist terminator element. */ - if (likely(rl)) { - rl[rlpos].vcn = rl[rlpos - 1].vcn + rl[rlpos - 1].length; - rl[rlpos].lcn = is_extension ? LCN_ENOENT : LCN_RL_NOT_MAPPED; - rl[rlpos].length = 0; - } - if (likely(page && !IS_ERR(page))) { - if (need_writeback) { - ntfs_debug("Marking page dirty."); - flush_dcache_page(page); - set_page_dirty(page); - need_writeback = 0; - } - ntfs_unmap_page(page); - } - if (likely(!err)) { - up_write(&vol->lcnbmp_lock); - ntfs_debug("Done."); - return rl; - } - ntfs_error(vol->sb, "Failed to allocate clusters, aborting " - "(error %i).", err); - if (rl) { - int err2; - - if (err == -ENOSPC) - ntfs_debug("Not enough space to complete allocation, " - "err -ENOSPC, first free lcn 0x%llx, " - "could allocate up to 0x%llx " - "clusters.", - (unsigned long long)rl[0].lcn, - (unsigned long long)(count - clusters)); - /* Deallocate all allocated clusters. */ - ntfs_debug("Attempting rollback..."); - err2 = ntfs_cluster_free_from_rl_nolock(vol, rl); - if (err2) { - ntfs_error(vol->sb, "Failed to rollback (error %i). " - "Leaving inconsistent metadata! " - "Unmount and run chkdsk.", err2); - NVolSetErrors(vol); - } - /* Free the runlist. */ - ntfs_free(rl); - } else if (err == -ENOSPC) - ntfs_debug("No space left at all, err = -ENOSPC, first free " - "lcn = 0x%llx.", - (long long)vol->data1_zone_pos); - up_write(&vol->lcnbmp_lock); - return ERR_PTR(err); -} - -/** - * __ntfs_cluster_free - free clusters on an ntfs volume - * @ni: ntfs inode whose runlist describes the clusters to free - * @start_vcn: vcn in the runlist of @ni at which to start freeing clusters - * @count: number of clusters to free or -1 for all clusters - * @ctx: active attribute search context if present or NULL if not - * @is_rollback: true if this is a rollback operation - * - * Free @count clusters starting at the cluster @start_vcn in the runlist - * described by the vfs inode @ni. - * - * If @count is -1, all clusters from @start_vcn to the end of the runlist are - * deallocated. Thus, to completely free all clusters in a runlist, use - * @start_vcn = 0 and @count = -1. - * - * If @ctx is specified, it is an active search context of @ni and its base mft - * record. This is needed when __ntfs_cluster_free() encounters unmapped - * runlist fragments and allows their mapping. If you do not have the mft - * record mapped, you can specify @ctx as NULL and __ntfs_cluster_free() will - * perform the necessary mapping and unmapping. - * - * Note, __ntfs_cluster_free() saves the state of @ctx on entry and restores it - * before returning. Thus, @ctx will be left pointing to the same attribute on - * return as on entry. However, the actual pointers in @ctx may point to - * different memory locations on return, so you must remember to reset any - * cached pointers from the @ctx, i.e. after the call to __ntfs_cluster_free(), - * you will probably want to do: - * m = ctx->mrec; - * a = ctx->attr; - * Assuming you cache ctx->attr in a variable @a of type ATTR_RECORD * and that - * you cache ctx->mrec in a variable @m of type MFT_RECORD *. - * - * @is_rollback should always be 'false', it is for internal use to rollback - * errors. You probably want to use ntfs_cluster_free() instead. - * - * Note, __ntfs_cluster_free() does not modify the runlist, so you have to - * remove from the runlist or mark sparse the freed runs later. - * - * Return the number of deallocated clusters (not counting sparse ones) on - * success and -errno on error. - * - * WARNING: If @ctx is supplied, regardless of whether success or failure is - * returned, you need to check IS_ERR(@ctx->mrec) and if 'true' the @ctx - * is no longer valid, i.e. you need to either call - * ntfs_attr_reinit_search_ctx() or ntfs_attr_put_search_ctx() on it. - * In that case PTR_ERR(@ctx->mrec) will give you the error code for - * why the mapping of the old inode failed. - * - * Locking: - The runlist described by @ni must be locked for writing on entry - * and is locked on return. Note the runlist may be modified when - * needed runlist fragments need to be mapped. - * - The volume lcn bitmap must be unlocked on entry and is unlocked - * on return. - * - This function takes the volume lcn bitmap lock for writing and - * modifies the bitmap contents. - * - If @ctx is NULL, the base mft record of @ni must not be mapped on - * entry and it will be left unmapped on return. - * - If @ctx is not NULL, the base mft record must be mapped on entry - * and it will be left mapped on return. - */ -s64 __ntfs_cluster_free(ntfs_inode *ni, const VCN start_vcn, s64 count, - ntfs_attr_search_ctx *ctx, const bool is_rollback) -{ - s64 delta, to_free, total_freed, real_freed; - ntfs_volume *vol; - struct inode *lcnbmp_vi; - runlist_element *rl; - int err; - - BUG_ON(!ni); - ntfs_debug("Entering for i_ino 0x%lx, start_vcn 0x%llx, count " - "0x%llx.%s", ni->mft_no, (unsigned long long)start_vcn, - (unsigned long long)count, - is_rollback ? " (rollback)" : ""); - vol = ni->vol; - lcnbmp_vi = vol->lcnbmp_ino; - BUG_ON(!lcnbmp_vi); - BUG_ON(start_vcn < 0); - BUG_ON(count < -1); - /* - * Lock the lcn bitmap for writing but only if not rolling back. We - * must hold the lock all the way including through rollback otherwise - * rollback is not possible because once we have cleared a bit and - * dropped the lock, anyone could have set the bit again, thus - * allocating the cluster for another use. - */ - if (likely(!is_rollback)) - down_write(&vol->lcnbmp_lock); - - total_freed = real_freed = 0; - - rl = ntfs_attr_find_vcn_nolock(ni, start_vcn, ctx); - if (IS_ERR(rl)) { - if (!is_rollback) - ntfs_error(vol->sb, "Failed to find first runlist " - "element (error %li), aborting.", - PTR_ERR(rl)); - err = PTR_ERR(rl); - goto err_out; - } - if (unlikely(rl->lcn < LCN_HOLE)) { - if (!is_rollback) - ntfs_error(vol->sb, "First runlist element has " - "invalid lcn, aborting."); - err = -EIO; - goto err_out; - } - /* Find the starting cluster inside the run that needs freeing. */ - delta = start_vcn - rl->vcn; - - /* The number of clusters in this run that need freeing. */ - to_free = rl->length - delta; - if (count >= 0 && to_free > count) - to_free = count; - - if (likely(rl->lcn >= 0)) { - /* Do the actual freeing of the clusters in this run. */ - err = ntfs_bitmap_set_bits_in_run(lcnbmp_vi, rl->lcn + delta, - to_free, likely(!is_rollback) ? 0 : 1); - if (unlikely(err)) { - if (!is_rollback) - ntfs_error(vol->sb, "Failed to clear first run " - "(error %i), aborting.", err); - goto err_out; - } - /* We have freed @to_free real clusters. */ - real_freed = to_free; - }; - /* Go to the next run and adjust the number of clusters left to free. */ - ++rl; - if (count >= 0) - count -= to_free; - - /* Keep track of the total "freed" clusters, including sparse ones. */ - total_freed = to_free; - /* - * Loop over the remaining runs, using @count as a capping value, and - * free them. - */ - for (; rl->length && count != 0; ++rl) { - if (unlikely(rl->lcn < LCN_HOLE)) { - VCN vcn; - - /* Attempt to map runlist. */ - vcn = rl->vcn; - rl = ntfs_attr_find_vcn_nolock(ni, vcn, ctx); - if (IS_ERR(rl)) { - err = PTR_ERR(rl); - if (!is_rollback) - ntfs_error(vol->sb, "Failed to map " - "runlist fragment or " - "failed to find " - "subsequent runlist " - "element."); - goto err_out; - } - if (unlikely(rl->lcn < LCN_HOLE)) { - if (!is_rollback) - ntfs_error(vol->sb, "Runlist element " - "has invalid lcn " - "(0x%llx).", - (unsigned long long) - rl->lcn); - err = -EIO; - goto err_out; - } - } - /* The number of clusters in this run that need freeing. */ - to_free = rl->length; - if (count >= 0 && to_free > count) - to_free = count; - - if (likely(rl->lcn >= 0)) { - /* Do the actual freeing of the clusters in the run. */ - err = ntfs_bitmap_set_bits_in_run(lcnbmp_vi, rl->lcn, - to_free, likely(!is_rollback) ? 0 : 1); - if (unlikely(err)) { - if (!is_rollback) - ntfs_error(vol->sb, "Failed to clear " - "subsequent run."); - goto err_out; - } - /* We have freed @to_free real clusters. */ - real_freed += to_free; - } - /* Adjust the number of clusters left to free. */ - if (count >= 0) - count -= to_free; - - /* Update the total done clusters. */ - total_freed += to_free; - } - if (likely(!is_rollback)) - up_write(&vol->lcnbmp_lock); - - BUG_ON(count > 0); - - /* We are done. Return the number of actually freed clusters. */ - ntfs_debug("Done."); - return real_freed; -err_out: - if (is_rollback) - return err; - /* If no real clusters were freed, no need to rollback. */ - if (!real_freed) { - up_write(&vol->lcnbmp_lock); - return err; - } - /* - * Attempt to rollback and if that succeeds just return the error code. - * If rollback fails, set the volume errors flag, emit an error - * message, and return the error code. - */ - delta = __ntfs_cluster_free(ni, start_vcn, total_freed, ctx, true); - if (delta < 0) { - ntfs_error(vol->sb, "Failed to rollback (error %i). Leaving " - "inconsistent metadata! Unmount and run " - "chkdsk.", (int)delta); - NVolSetErrors(vol); - } - up_write(&vol->lcnbmp_lock); - ntfs_error(vol->sb, "Aborting (error %i).", err); - return err; -} - -#endif /* NTFS_RW */ diff --git a/fs/ntfs/lcnalloc.h b/fs/ntfs/lcnalloc.h deleted file mode 100644 index 1589a6d8434b..000000000000 --- a/fs/ntfs/lcnalloc.h +++ /dev/null @@ -1,131 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * lcnalloc.h - Exports for NTFS kernel cluster (de)allocation. Part of the - * Linux-NTFS project. - * - * Copyright (c) 2004-2005 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_LCNALLOC_H -#define _LINUX_NTFS_LCNALLOC_H - -#ifdef NTFS_RW - -#include - -#include "attrib.h" -#include "types.h" -#include "inode.h" -#include "runlist.h" -#include "volume.h" - -typedef enum { - FIRST_ZONE = 0, /* For sanity checking. */ - MFT_ZONE = 0, /* Allocate from $MFT zone. */ - DATA_ZONE = 1, /* Allocate from $DATA zone. */ - LAST_ZONE = 1, /* For sanity checking. */ -} NTFS_CLUSTER_ALLOCATION_ZONES; - -extern runlist_element *ntfs_cluster_alloc(ntfs_volume *vol, - const VCN start_vcn, const s64 count, const LCN start_lcn, - const NTFS_CLUSTER_ALLOCATION_ZONES zone, - const bool is_extension); - -extern s64 __ntfs_cluster_free(ntfs_inode *ni, const VCN start_vcn, - s64 count, ntfs_attr_search_ctx *ctx, const bool is_rollback); - -/** - * ntfs_cluster_free - free clusters on an ntfs volume - * @ni: ntfs inode whose runlist describes the clusters to free - * @start_vcn: vcn in the runlist of @ni at which to start freeing clusters - * @count: number of clusters to free or -1 for all clusters - * @ctx: active attribute search context if present or NULL if not - * - * Free @count clusters starting at the cluster @start_vcn in the runlist - * described by the ntfs inode @ni. - * - * If @count is -1, all clusters from @start_vcn to the end of the runlist are - * deallocated. Thus, to completely free all clusters in a runlist, use - * @start_vcn = 0 and @count = -1. - * - * If @ctx is specified, it is an active search context of @ni and its base mft - * record. This is needed when ntfs_cluster_free() encounters unmapped runlist - * fragments and allows their mapping. If you do not have the mft record - * mapped, you can specify @ctx as NULL and ntfs_cluster_free() will perform - * the necessary mapping and unmapping. - * - * Note, ntfs_cluster_free() saves the state of @ctx on entry and restores it - * before returning. Thus, @ctx will be left pointing to the same attribute on - * return as on entry. However, the actual pointers in @ctx may point to - * different memory locations on return, so you must remember to reset any - * cached pointers from the @ctx, i.e. after the call to ntfs_cluster_free(), - * you will probably want to do: - * m = ctx->mrec; - * a = ctx->attr; - * Assuming you cache ctx->attr in a variable @a of type ATTR_RECORD * and that - * you cache ctx->mrec in a variable @m of type MFT_RECORD *. - * - * Note, ntfs_cluster_free() does not modify the runlist, so you have to remove - * from the runlist or mark sparse the freed runs later. - * - * Return the number of deallocated clusters (not counting sparse ones) on - * success and -errno on error. - * - * WARNING: If @ctx is supplied, regardless of whether success or failure is - * returned, you need to check IS_ERR(@ctx->mrec) and if 'true' the @ctx - * is no longer valid, i.e. you need to either call - * ntfs_attr_reinit_search_ctx() or ntfs_attr_put_search_ctx() on it. - * In that case PTR_ERR(@ctx->mrec) will give you the error code for - * why the mapping of the old inode failed. - * - * Locking: - The runlist described by @ni must be locked for writing on entry - * and is locked on return. Note the runlist may be modified when - * needed runlist fragments need to be mapped. - * - The volume lcn bitmap must be unlocked on entry and is unlocked - * on return. - * - This function takes the volume lcn bitmap lock for writing and - * modifies the bitmap contents. - * - If @ctx is NULL, the base mft record of @ni must not be mapped on - * entry and it will be left unmapped on return. - * - If @ctx is not NULL, the base mft record must be mapped on entry - * and it will be left mapped on return. - */ -static inline s64 ntfs_cluster_free(ntfs_inode *ni, const VCN start_vcn, - s64 count, ntfs_attr_search_ctx *ctx) -{ - return __ntfs_cluster_free(ni, start_vcn, count, ctx, false); -} - -extern int ntfs_cluster_free_from_rl_nolock(ntfs_volume *vol, - const runlist_element *rl); - -/** - * ntfs_cluster_free_from_rl - free clusters from runlist - * @vol: mounted ntfs volume on which to free the clusters - * @rl: runlist describing the clusters to free - * - * Free all the clusters described by the runlist @rl on the volume @vol. In - * the case of an error being returned, at least some of the clusters were not - * freed. - * - * Return 0 on success and -errno on error. - * - * Locking: - This function takes the volume lcn bitmap lock for writing and - * modifies the bitmap contents. - * - The caller must have locked the runlist @rl for reading or - * writing. - */ -static inline int ntfs_cluster_free_from_rl(ntfs_volume *vol, - const runlist_element *rl) -{ - int ret; - - down_write(&vol->lcnbmp_lock); - ret = ntfs_cluster_free_from_rl_nolock(vol, rl); - up_write(&vol->lcnbmp_lock); - return ret; -} - -#endif /* NTFS_RW */ - -#endif /* defined _LINUX_NTFS_LCNALLOC_H */ diff --git a/fs/ntfs/logfile.c b/fs/ntfs/logfile.c deleted file mode 100644 index 6ce60ffc6ac0..000000000000 --- a/fs/ntfs/logfile.c +++ /dev/null @@ -1,849 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * logfile.c - NTFS kernel journal handling. Part of the Linux-NTFS project. - * - * Copyright (c) 2002-2007 Anton Altaparmakov - */ - -#ifdef NTFS_RW - -#include -#include -#include -#include -#include -#include -#include - -#include "attrib.h" -#include "aops.h" -#include "debug.h" -#include "logfile.h" -#include "malloc.h" -#include "volume.h" -#include "ntfs.h" - -/** - * ntfs_check_restart_page_header - check the page header for consistency - * @vi: $LogFile inode to which the restart page header belongs - * @rp: restart page header to check - * @pos: position in @vi at which the restart page header resides - * - * Check the restart page header @rp for consistency and return 'true' if it is - * consistent and 'false' otherwise. - * - * This function only needs NTFS_BLOCK_SIZE bytes in @rp, i.e. it does not - * require the full restart page. - */ -static bool ntfs_check_restart_page_header(struct inode *vi, - RESTART_PAGE_HEADER *rp, s64 pos) -{ - u32 logfile_system_page_size, logfile_log_page_size; - u16 ra_ofs, usa_count, usa_ofs, usa_end = 0; - bool have_usa = true; - - ntfs_debug("Entering."); - /* - * If the system or log page sizes are smaller than the ntfs block size - * or either is not a power of 2 we cannot handle this log file. - */ - logfile_system_page_size = le32_to_cpu(rp->system_page_size); - logfile_log_page_size = le32_to_cpu(rp->log_page_size); - if (logfile_system_page_size < NTFS_BLOCK_SIZE || - logfile_log_page_size < NTFS_BLOCK_SIZE || - logfile_system_page_size & - (logfile_system_page_size - 1) || - !is_power_of_2(logfile_log_page_size)) { - ntfs_error(vi->i_sb, "$LogFile uses unsupported page size."); - return false; - } - /* - * We must be either at !pos (1st restart page) or at pos = system page - * size (2nd restart page). - */ - if (pos && pos != logfile_system_page_size) { - ntfs_error(vi->i_sb, "Found restart area in incorrect " - "position in $LogFile."); - return false; - } - /* We only know how to handle version 1.1. */ - if (sle16_to_cpu(rp->major_ver) != 1 || - sle16_to_cpu(rp->minor_ver) != 1) { - ntfs_error(vi->i_sb, "$LogFile version %i.%i is not " - "supported. (This driver supports version " - "1.1 only.)", (int)sle16_to_cpu(rp->major_ver), - (int)sle16_to_cpu(rp->minor_ver)); - return false; - } - /* - * If chkdsk has been run the restart page may not be protected by an - * update sequence array. - */ - if (ntfs_is_chkd_record(rp->magic) && !le16_to_cpu(rp->usa_count)) { - have_usa = false; - goto skip_usa_checks; - } - /* Verify the size of the update sequence array. */ - usa_count = 1 + (logfile_system_page_size >> NTFS_BLOCK_SIZE_BITS); - if (usa_count != le16_to_cpu(rp->usa_count)) { - ntfs_error(vi->i_sb, "$LogFile restart page specifies " - "inconsistent update sequence array count."); - return false; - } - /* Verify the position of the update sequence array. */ - usa_ofs = le16_to_cpu(rp->usa_ofs); - usa_end = usa_ofs + usa_count * sizeof(u16); - if (usa_ofs < sizeof(RESTART_PAGE_HEADER) || - usa_end > NTFS_BLOCK_SIZE - sizeof(u16)) { - ntfs_error(vi->i_sb, "$LogFile restart page specifies " - "inconsistent update sequence array offset."); - return false; - } -skip_usa_checks: - /* - * Verify the position of the restart area. It must be: - * - aligned to 8-byte boundary, - * - after the update sequence array, and - * - within the system page size. - */ - ra_ofs = le16_to_cpu(rp->restart_area_offset); - if (ra_ofs & 7 || (have_usa ? ra_ofs < usa_end : - ra_ofs < sizeof(RESTART_PAGE_HEADER)) || - ra_ofs > logfile_system_page_size) { - ntfs_error(vi->i_sb, "$LogFile restart page specifies " - "inconsistent restart area offset."); - return false; - } - /* - * Only restart pages modified by chkdsk are allowed to have chkdsk_lsn - * set. - */ - if (!ntfs_is_chkd_record(rp->magic) && sle64_to_cpu(rp->chkdsk_lsn)) { - ntfs_error(vi->i_sb, "$LogFile restart page is not modified " - "by chkdsk but a chkdsk LSN is specified."); - return false; - } - ntfs_debug("Done."); - return true; -} - -/** - * ntfs_check_restart_area - check the restart area for consistency - * @vi: $LogFile inode to which the restart page belongs - * @rp: restart page whose restart area to check - * - * Check the restart area of the restart page @rp for consistency and return - * 'true' if it is consistent and 'false' otherwise. - * - * This function assumes that the restart page header has already been - * consistency checked. - * - * This function only needs NTFS_BLOCK_SIZE bytes in @rp, i.e. it does not - * require the full restart page. - */ -static bool ntfs_check_restart_area(struct inode *vi, RESTART_PAGE_HEADER *rp) -{ - u64 file_size; - RESTART_AREA *ra; - u16 ra_ofs, ra_len, ca_ofs; - u8 fs_bits; - - ntfs_debug("Entering."); - ra_ofs = le16_to_cpu(rp->restart_area_offset); - ra = (RESTART_AREA*)((u8*)rp + ra_ofs); - /* - * Everything before ra->file_size must be before the first word - * protected by an update sequence number. This ensures that it is - * safe to access ra->client_array_offset. - */ - if (ra_ofs + offsetof(RESTART_AREA, file_size) > - NTFS_BLOCK_SIZE - sizeof(u16)) { - ntfs_error(vi->i_sb, "$LogFile restart area specifies " - "inconsistent file offset."); - return false; - } - /* - * Now that we can access ra->client_array_offset, make sure everything - * up to the log client array is before the first word protected by an - * update sequence number. This ensures we can access all of the - * restart area elements safely. Also, the client array offset must be - * aligned to an 8-byte boundary. - */ - ca_ofs = le16_to_cpu(ra->client_array_offset); - if (((ca_ofs + 7) & ~7) != ca_ofs || - ra_ofs + ca_ofs > NTFS_BLOCK_SIZE - sizeof(u16)) { - ntfs_error(vi->i_sb, "$LogFile restart area specifies " - "inconsistent client array offset."); - return false; - } - /* - * The restart area must end within the system page size both when - * calculated manually and as specified by ra->restart_area_length. - * Also, the calculated length must not exceed the specified length. - */ - ra_len = ca_ofs + le16_to_cpu(ra->log_clients) * - sizeof(LOG_CLIENT_RECORD); - if (ra_ofs + ra_len > le32_to_cpu(rp->system_page_size) || - ra_ofs + le16_to_cpu(ra->restart_area_length) > - le32_to_cpu(rp->system_page_size) || - ra_len > le16_to_cpu(ra->restart_area_length)) { - ntfs_error(vi->i_sb, "$LogFile restart area is out of bounds " - "of the system page size specified by the " - "restart page header and/or the specified " - "restart area length is inconsistent."); - return false; - } - /* - * The ra->client_free_list and ra->client_in_use_list must be either - * LOGFILE_NO_CLIENT or less than ra->log_clients or they are - * overflowing the client array. - */ - if ((ra->client_free_list != LOGFILE_NO_CLIENT && - le16_to_cpu(ra->client_free_list) >= - le16_to_cpu(ra->log_clients)) || - (ra->client_in_use_list != LOGFILE_NO_CLIENT && - le16_to_cpu(ra->client_in_use_list) >= - le16_to_cpu(ra->log_clients))) { - ntfs_error(vi->i_sb, "$LogFile restart area specifies " - "overflowing client free and/or in use lists."); - return false; - } - /* - * Check ra->seq_number_bits against ra->file_size for consistency. - * We cannot just use ffs() because the file size is not a power of 2. - */ - file_size = (u64)sle64_to_cpu(ra->file_size); - fs_bits = 0; - while (file_size) { - file_size >>= 1; - fs_bits++; - } - if (le32_to_cpu(ra->seq_number_bits) != 67 - fs_bits) { - ntfs_error(vi->i_sb, "$LogFile restart area specifies " - "inconsistent sequence number bits."); - return false; - } - /* The log record header length must be a multiple of 8. */ - if (((le16_to_cpu(ra->log_record_header_length) + 7) & ~7) != - le16_to_cpu(ra->log_record_header_length)) { - ntfs_error(vi->i_sb, "$LogFile restart area specifies " - "inconsistent log record header length."); - return false; - } - /* Dito for the log page data offset. */ - if (((le16_to_cpu(ra->log_page_data_offset) + 7) & ~7) != - le16_to_cpu(ra->log_page_data_offset)) { - ntfs_error(vi->i_sb, "$LogFile restart area specifies " - "inconsistent log page data offset."); - return false; - } - ntfs_debug("Done."); - return true; -} - -/** - * ntfs_check_log_client_array - check the log client array for consistency - * @vi: $LogFile inode to which the restart page belongs - * @rp: restart page whose log client array to check - * - * Check the log client array of the restart page @rp for consistency and - * return 'true' if it is consistent and 'false' otherwise. - * - * This function assumes that the restart page header and the restart area have - * already been consistency checked. - * - * Unlike ntfs_check_restart_page_header() and ntfs_check_restart_area(), this - * function needs @rp->system_page_size bytes in @rp, i.e. it requires the full - * restart page and the page must be multi sector transfer deprotected. - */ -static bool ntfs_check_log_client_array(struct inode *vi, - RESTART_PAGE_HEADER *rp) -{ - RESTART_AREA *ra; - LOG_CLIENT_RECORD *ca, *cr; - u16 nr_clients, idx; - bool in_free_list, idx_is_first; - - ntfs_debug("Entering."); - ra = (RESTART_AREA*)((u8*)rp + le16_to_cpu(rp->restart_area_offset)); - ca = (LOG_CLIENT_RECORD*)((u8*)ra + - le16_to_cpu(ra->client_array_offset)); - /* - * Check the ra->client_free_list first and then check the - * ra->client_in_use_list. Check each of the log client records in - * each of the lists and check that the array does not overflow the - * ra->log_clients value. Also keep track of the number of records - * visited as there cannot be more than ra->log_clients records and - * that way we detect eventual loops in within a list. - */ - nr_clients = le16_to_cpu(ra->log_clients); - idx = le16_to_cpu(ra->client_free_list); - in_free_list = true; -check_list: - for (idx_is_first = true; idx != LOGFILE_NO_CLIENT_CPU; nr_clients--, - idx = le16_to_cpu(cr->next_client)) { - if (!nr_clients || idx >= le16_to_cpu(ra->log_clients)) - goto err_out; - /* Set @cr to the current log client record. */ - cr = ca + idx; - /* The first log client record must not have a prev_client. */ - if (idx_is_first) { - if (cr->prev_client != LOGFILE_NO_CLIENT) - goto err_out; - idx_is_first = false; - } - } - /* Switch to and check the in use list if we just did the free list. */ - if (in_free_list) { - in_free_list = false; - idx = le16_to_cpu(ra->client_in_use_list); - goto check_list; - } - ntfs_debug("Done."); - return true; -err_out: - ntfs_error(vi->i_sb, "$LogFile log client array is corrupt."); - return false; -} - -/** - * ntfs_check_and_load_restart_page - check the restart page for consistency - * @vi: $LogFile inode to which the restart page belongs - * @rp: restart page to check - * @pos: position in @vi at which the restart page resides - * @wrp: [OUT] copy of the multi sector transfer deprotected restart page - * @lsn: [OUT] set to the current logfile lsn on success - * - * Check the restart page @rp for consistency and return 0 if it is consistent - * and -errno otherwise. The restart page may have been modified by chkdsk in - * which case its magic is CHKD instead of RSTR. - * - * This function only needs NTFS_BLOCK_SIZE bytes in @rp, i.e. it does not - * require the full restart page. - * - * If @wrp is not NULL, on success, *@wrp will point to a buffer containing a - * copy of the complete multi sector transfer deprotected page. On failure, - * *@wrp is undefined. - * - * Simillarly, if @lsn is not NULL, on success *@lsn will be set to the current - * logfile lsn according to this restart page. On failure, *@lsn is undefined. - * - * The following error codes are defined: - * -EINVAL - The restart page is inconsistent. - * -ENOMEM - Not enough memory to load the restart page. - * -EIO - Failed to reading from $LogFile. - */ -static int ntfs_check_and_load_restart_page(struct inode *vi, - RESTART_PAGE_HEADER *rp, s64 pos, RESTART_PAGE_HEADER **wrp, - LSN *lsn) -{ - RESTART_AREA *ra; - RESTART_PAGE_HEADER *trp; - int size, err; - - ntfs_debug("Entering."); - /* Check the restart page header for consistency. */ - if (!ntfs_check_restart_page_header(vi, rp, pos)) { - /* Error output already done inside the function. */ - return -EINVAL; - } - /* Check the restart area for consistency. */ - if (!ntfs_check_restart_area(vi, rp)) { - /* Error output already done inside the function. */ - return -EINVAL; - } - ra = (RESTART_AREA*)((u8*)rp + le16_to_cpu(rp->restart_area_offset)); - /* - * Allocate a buffer to store the whole restart page so we can multi - * sector transfer deprotect it. - */ - trp = ntfs_malloc_nofs(le32_to_cpu(rp->system_page_size)); - if (!trp) { - ntfs_error(vi->i_sb, "Failed to allocate memory for $LogFile " - "restart page buffer."); - return -ENOMEM; - } - /* - * Read the whole of the restart page into the buffer. If it fits - * completely inside @rp, just copy it from there. Otherwise map all - * the required pages and copy the data from them. - */ - size = PAGE_SIZE - (pos & ~PAGE_MASK); - if (size >= le32_to_cpu(rp->system_page_size)) { - memcpy(trp, rp, le32_to_cpu(rp->system_page_size)); - } else { - pgoff_t idx; - struct page *page; - int have_read, to_read; - - /* First copy what we already have in @rp. */ - memcpy(trp, rp, size); - /* Copy the remaining data one page at a time. */ - have_read = size; - to_read = le32_to_cpu(rp->system_page_size) - size; - idx = (pos + size) >> PAGE_SHIFT; - BUG_ON((pos + size) & ~PAGE_MASK); - do { - page = ntfs_map_page(vi->i_mapping, idx); - if (IS_ERR(page)) { - ntfs_error(vi->i_sb, "Error mapping $LogFile " - "page (index %lu).", idx); - err = PTR_ERR(page); - if (err != -EIO && err != -ENOMEM) - err = -EIO; - goto err_out; - } - size = min_t(int, to_read, PAGE_SIZE); - memcpy((u8*)trp + have_read, page_address(page), size); - ntfs_unmap_page(page); - have_read += size; - to_read -= size; - idx++; - } while (to_read > 0); - } - /* - * Perform the multi sector transfer deprotection on the buffer if the - * restart page is protected. - */ - if ((!ntfs_is_chkd_record(trp->magic) || le16_to_cpu(trp->usa_count)) - && post_read_mst_fixup((NTFS_RECORD*)trp, - le32_to_cpu(rp->system_page_size))) { - /* - * A multi sector tranfer error was detected. We only need to - * abort if the restart page contents exceed the multi sector - * transfer fixup of the first sector. - */ - if (le16_to_cpu(rp->restart_area_offset) + - le16_to_cpu(ra->restart_area_length) > - NTFS_BLOCK_SIZE - sizeof(u16)) { - ntfs_error(vi->i_sb, "Multi sector transfer error " - "detected in $LogFile restart page."); - err = -EINVAL; - goto err_out; - } - } - /* - * If the restart page is modified by chkdsk or there are no active - * logfile clients, the logfile is consistent. Otherwise, need to - * check the log client records for consistency, too. - */ - err = 0; - if (ntfs_is_rstr_record(rp->magic) && - ra->client_in_use_list != LOGFILE_NO_CLIENT) { - if (!ntfs_check_log_client_array(vi, trp)) { - err = -EINVAL; - goto err_out; - } - } - if (lsn) { - if (ntfs_is_rstr_record(rp->magic)) - *lsn = sle64_to_cpu(ra->current_lsn); - else /* if (ntfs_is_chkd_record(rp->magic)) */ - *lsn = sle64_to_cpu(rp->chkdsk_lsn); - } - ntfs_debug("Done."); - if (wrp) - *wrp = trp; - else { -err_out: - ntfs_free(trp); - } - return err; -} - -/** - * ntfs_check_logfile - check the journal for consistency - * @log_vi: struct inode of loaded journal $LogFile to check - * @rp: [OUT] on success this is a copy of the current restart page - * - * Check the $LogFile journal for consistency and return 'true' if it is - * consistent and 'false' if not. On success, the current restart page is - * returned in *@rp. Caller must call ntfs_free(*@rp) when finished with it. - * - * At present we only check the two restart pages and ignore the log record - * pages. - * - * Note that the MstProtected flag is not set on the $LogFile inode and hence - * when reading pages they are not deprotected. This is because we do not know - * if the $LogFile was created on a system with a different page size to ours - * yet and mst deprotection would fail if our page size is smaller. - */ -bool ntfs_check_logfile(struct inode *log_vi, RESTART_PAGE_HEADER **rp) -{ - s64 size, pos; - LSN rstr1_lsn, rstr2_lsn; - ntfs_volume *vol = NTFS_SB(log_vi->i_sb); - struct address_space *mapping = log_vi->i_mapping; - struct page *page = NULL; - u8 *kaddr = NULL; - RESTART_PAGE_HEADER *rstr1_ph = NULL; - RESTART_PAGE_HEADER *rstr2_ph = NULL; - int log_page_size, err; - bool logfile_is_empty = true; - u8 log_page_bits; - - ntfs_debug("Entering."); - /* An empty $LogFile must have been clean before it got emptied. */ - if (NVolLogFileEmpty(vol)) - goto is_empty; - size = i_size_read(log_vi); - /* Make sure the file doesn't exceed the maximum allowed size. */ - if (size > MaxLogFileSize) - size = MaxLogFileSize; - /* - * Truncate size to a multiple of the page cache size or the default - * log page size if the page cache size is between the default log page - * log page size if the page cache size is between the default log page - * size and twice that. - */ - if (PAGE_SIZE >= DefaultLogPageSize && PAGE_SIZE <= - DefaultLogPageSize * 2) - log_page_size = DefaultLogPageSize; - else - log_page_size = PAGE_SIZE; - /* - * Use ntfs_ffs() instead of ffs() to enable the compiler to - * optimize log_page_size and log_page_bits into constants. - */ - log_page_bits = ntfs_ffs(log_page_size) - 1; - size &= ~(s64)(log_page_size - 1); - /* - * Ensure the log file is big enough to store at least the two restart - * pages and the minimum number of log record pages. - */ - if (size < log_page_size * 2 || (size - log_page_size * 2) >> - log_page_bits < MinLogRecordPages) { - ntfs_error(vol->sb, "$LogFile is too small."); - return false; - } - /* - * Read through the file looking for a restart page. Since the restart - * page header is at the beginning of a page we only need to search at - * what could be the beginning of a page (for each page size) rather - * than scanning the whole file byte by byte. If all potential places - * contain empty and uninitialzed records, the log file can be assumed - * to be empty. - */ - for (pos = 0; pos < size; pos <<= 1) { - pgoff_t idx = pos >> PAGE_SHIFT; - if (!page || page->index != idx) { - if (page) - ntfs_unmap_page(page); - page = ntfs_map_page(mapping, idx); - if (IS_ERR(page)) { - ntfs_error(vol->sb, "Error mapping $LogFile " - "page (index %lu).", idx); - goto err_out; - } - } - kaddr = (u8*)page_address(page) + (pos & ~PAGE_MASK); - /* - * A non-empty block means the logfile is not empty while an - * empty block after a non-empty block has been encountered - * means we are done. - */ - if (!ntfs_is_empty_recordp((le32*)kaddr)) - logfile_is_empty = false; - else if (!logfile_is_empty) - break; - /* - * A log record page means there cannot be a restart page after - * this so no need to continue searching. - */ - if (ntfs_is_rcrd_recordp((le32*)kaddr)) - break; - /* If not a (modified by chkdsk) restart page, continue. */ - if (!ntfs_is_rstr_recordp((le32*)kaddr) && - !ntfs_is_chkd_recordp((le32*)kaddr)) { - if (!pos) - pos = NTFS_BLOCK_SIZE >> 1; - continue; - } - /* - * Check the (modified by chkdsk) restart page for consistency - * and get a copy of the complete multi sector transfer - * deprotected restart page. - */ - err = ntfs_check_and_load_restart_page(log_vi, - (RESTART_PAGE_HEADER*)kaddr, pos, - !rstr1_ph ? &rstr1_ph : &rstr2_ph, - !rstr1_ph ? &rstr1_lsn : &rstr2_lsn); - if (!err) { - /* - * If we have now found the first (modified by chkdsk) - * restart page, continue looking for the second one. - */ - if (!pos) { - pos = NTFS_BLOCK_SIZE >> 1; - continue; - } - /* - * We have now found the second (modified by chkdsk) - * restart page, so we can stop looking. - */ - break; - } - /* - * Error output already done inside the function. Note, we do - * not abort if the restart page was invalid as we might still - * find a valid one further in the file. - */ - if (err != -EINVAL) { - ntfs_unmap_page(page); - goto err_out; - } - /* Continue looking. */ - if (!pos) - pos = NTFS_BLOCK_SIZE >> 1; - } - if (page) - ntfs_unmap_page(page); - if (logfile_is_empty) { - NVolSetLogFileEmpty(vol); -is_empty: - ntfs_debug("Done. ($LogFile is empty.)"); - return true; - } - if (!rstr1_ph) { - BUG_ON(rstr2_ph); - ntfs_error(vol->sb, "Did not find any restart pages in " - "$LogFile and it was not empty."); - return false; - } - /* If both restart pages were found, use the more recent one. */ - if (rstr2_ph) { - /* - * If the second restart area is more recent, switch to it. - * Otherwise just throw it away. - */ - if (rstr2_lsn > rstr1_lsn) { - ntfs_debug("Using second restart page as it is more " - "recent."); - ntfs_free(rstr1_ph); - rstr1_ph = rstr2_ph; - /* rstr1_lsn = rstr2_lsn; */ - } else { - ntfs_debug("Using first restart page as it is more " - "recent."); - ntfs_free(rstr2_ph); - } - rstr2_ph = NULL; - } - /* All consistency checks passed. */ - if (rp) - *rp = rstr1_ph; - else - ntfs_free(rstr1_ph); - ntfs_debug("Done."); - return true; -err_out: - if (rstr1_ph) - ntfs_free(rstr1_ph); - return false; -} - -/** - * ntfs_is_logfile_clean - check in the journal if the volume is clean - * @log_vi: struct inode of loaded journal $LogFile to check - * @rp: copy of the current restart page - * - * Analyze the $LogFile journal and return 'true' if it indicates the volume was - * shutdown cleanly and 'false' if not. - * - * At present we only look at the two restart pages and ignore the log record - * pages. This is a little bit crude in that there will be a very small number - * of cases where we think that a volume is dirty when in fact it is clean. - * This should only affect volumes that have not been shutdown cleanly but did - * not have any pending, non-check-pointed i/o, i.e. they were completely idle - * at least for the five seconds preceding the unclean shutdown. - * - * This function assumes that the $LogFile journal has already been consistency - * checked by a call to ntfs_check_logfile() and in particular if the $LogFile - * is empty this function requires that NVolLogFileEmpty() is true otherwise an - * empty volume will be reported as dirty. - */ -bool ntfs_is_logfile_clean(struct inode *log_vi, const RESTART_PAGE_HEADER *rp) -{ - ntfs_volume *vol = NTFS_SB(log_vi->i_sb); - RESTART_AREA *ra; - - ntfs_debug("Entering."); - /* An empty $LogFile must have been clean before it got emptied. */ - if (NVolLogFileEmpty(vol)) { - ntfs_debug("Done. ($LogFile is empty.)"); - return true; - } - BUG_ON(!rp); - if (!ntfs_is_rstr_record(rp->magic) && - !ntfs_is_chkd_record(rp->magic)) { - ntfs_error(vol->sb, "Restart page buffer is invalid. This is " - "probably a bug in that the $LogFile should " - "have been consistency checked before calling " - "this function."); - return false; - } - ra = (RESTART_AREA*)((u8*)rp + le16_to_cpu(rp->restart_area_offset)); - /* - * If the $LogFile has active clients, i.e. it is open, and we do not - * have the RESTART_VOLUME_IS_CLEAN bit set in the restart area flags, - * we assume there was an unclean shutdown. - */ - if (ra->client_in_use_list != LOGFILE_NO_CLIENT && - !(ra->flags & RESTART_VOLUME_IS_CLEAN)) { - ntfs_debug("Done. $LogFile indicates a dirty shutdown."); - return false; - } - /* $LogFile indicates a clean shutdown. */ - ntfs_debug("Done. $LogFile indicates a clean shutdown."); - return true; -} - -/** - * ntfs_empty_logfile - empty the contents of the $LogFile journal - * @log_vi: struct inode of loaded journal $LogFile to empty - * - * Empty the contents of the $LogFile journal @log_vi and return 'true' on - * success and 'false' on error. - * - * This function assumes that the $LogFile journal has already been consistency - * checked by a call to ntfs_check_logfile() and that ntfs_is_logfile_clean() - * has been used to ensure that the $LogFile is clean. - */ -bool ntfs_empty_logfile(struct inode *log_vi) -{ - VCN vcn, end_vcn; - ntfs_inode *log_ni = NTFS_I(log_vi); - ntfs_volume *vol = log_ni->vol; - struct super_block *sb = vol->sb; - runlist_element *rl; - unsigned long flags; - unsigned block_size, block_size_bits; - int err; - bool should_wait = true; - - ntfs_debug("Entering."); - if (NVolLogFileEmpty(vol)) { - ntfs_debug("Done."); - return true; - } - /* - * We cannot use ntfs_attr_set() because we may be still in the middle - * of a mount operation. Thus we do the emptying by hand by first - * zapping the page cache pages for the $LogFile/$DATA attribute and - * then emptying each of the buffers in each of the clusters specified - * by the runlist by hand. - */ - block_size = sb->s_blocksize; - block_size_bits = sb->s_blocksize_bits; - vcn = 0; - read_lock_irqsave(&log_ni->size_lock, flags); - end_vcn = (log_ni->initialized_size + vol->cluster_size_mask) >> - vol->cluster_size_bits; - read_unlock_irqrestore(&log_ni->size_lock, flags); - truncate_inode_pages(log_vi->i_mapping, 0); - down_write(&log_ni->runlist.lock); - rl = log_ni->runlist.rl; - if (unlikely(!rl || vcn < rl->vcn || !rl->length)) { -map_vcn: - err = ntfs_map_runlist_nolock(log_ni, vcn, NULL); - if (err) { - ntfs_error(sb, "Failed to map runlist fragment (error " - "%d).", -err); - goto err; - } - rl = log_ni->runlist.rl; - BUG_ON(!rl || vcn < rl->vcn || !rl->length); - } - /* Seek to the runlist element containing @vcn. */ - while (rl->length && vcn >= rl[1].vcn) - rl++; - do { - LCN lcn; - sector_t block, end_block; - s64 len; - - /* - * If this run is not mapped map it now and start again as the - * runlist will have been updated. - */ - lcn = rl->lcn; - if (unlikely(lcn == LCN_RL_NOT_MAPPED)) { - vcn = rl->vcn; - goto map_vcn; - } - /* If this run is not valid abort with an error. */ - if (unlikely(!rl->length || lcn < LCN_HOLE)) - goto rl_err; - /* Skip holes. */ - if (lcn == LCN_HOLE) - continue; - block = lcn << vol->cluster_size_bits >> block_size_bits; - len = rl->length; - if (rl[1].vcn > end_vcn) - len = end_vcn - rl->vcn; - end_block = (lcn + len) << vol->cluster_size_bits >> - block_size_bits; - /* Iterate over the blocks in the run and empty them. */ - do { - struct buffer_head *bh; - - /* Obtain the buffer, possibly not uptodate. */ - bh = sb_getblk(sb, block); - BUG_ON(!bh); - /* Setup buffer i/o submission. */ - lock_buffer(bh); - bh->b_end_io = end_buffer_write_sync; - get_bh(bh); - /* Set the entire contents of the buffer to 0xff. */ - memset(bh->b_data, -1, block_size); - if (!buffer_uptodate(bh)) - set_buffer_uptodate(bh); - if (buffer_dirty(bh)) - clear_buffer_dirty(bh); - /* - * Submit the buffer and wait for i/o to complete but - * only for the first buffer so we do not miss really - * serious i/o errors. Once the first buffer has - * completed ignore errors afterwards as we can assume - * that if one buffer worked all of them will work. - */ - submit_bh(REQ_OP_WRITE, bh); - if (should_wait) { - should_wait = false; - wait_on_buffer(bh); - if (unlikely(!buffer_uptodate(bh))) - goto io_err; - } - brelse(bh); - } while (++block < end_block); - } while ((++rl)->vcn < end_vcn); - up_write(&log_ni->runlist.lock); - /* - * Zap the pages again just in case any got instantiated whilst we were - * emptying the blocks by hand. FIXME: We may not have completed - * writing to all the buffer heads yet so this may happen too early. - * We really should use a kernel thread to do the emptying - * asynchronously and then we can also set the volume dirty and output - * an error message if emptying should fail. - */ - truncate_inode_pages(log_vi->i_mapping, 0); - /* Set the flag so we do not have to do it again on remount. */ - NVolSetLogFileEmpty(vol); - ntfs_debug("Done."); - return true; -io_err: - ntfs_error(sb, "Failed to write buffer. Unmount and run chkdsk."); - goto dirty_err; -rl_err: - ntfs_error(sb, "Runlist is corrupt. Unmount and run chkdsk."); -dirty_err: - NVolSetErrors(vol); - err = -EIO; -err: - up_write(&log_ni->runlist.lock); - ntfs_error(sb, "Failed to fill $LogFile with 0xff bytes (error %d).", - -err); - return false; -} - -#endif /* NTFS_RW */ diff --git a/fs/ntfs/logfile.h b/fs/ntfs/logfile.h deleted file mode 100644 index 429d4909cc72..000000000000 --- a/fs/ntfs/logfile.h +++ /dev/null @@ -1,295 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * logfile.h - Defines for NTFS kernel journal ($LogFile) handling. Part of - * the Linux-NTFS project. - * - * Copyright (c) 2000-2005 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_LOGFILE_H -#define _LINUX_NTFS_LOGFILE_H - -#ifdef NTFS_RW - -#include - -#include "types.h" -#include "endian.h" -#include "layout.h" - -/* - * Journal ($LogFile) organization: - * - * Two restart areas present in the first two pages (restart pages, one restart - * area in each page). When the volume is dismounted they should be identical, - * except for the update sequence array which usually has a different update - * sequence number. - * - * These are followed by log records organized in pages headed by a log record - * header going up to log file size. Not all pages contain log records when a - * volume is first formatted, but as the volume ages, all records will be used. - * When the log file fills up, the records at the beginning are purged (by - * modifying the oldest_lsn to a higher value presumably) and writing begins - * at the beginning of the file. Effectively, the log file is viewed as a - * circular entity. - * - * NOTE: Windows NT, 2000, and XP all use log file version 1.1 but they accept - * versions <= 1.x, including 0.-1. (Yes, that is a minus one in there!) We - * probably only want to support 1.1 as this seems to be the current version - * and we don't know how that differs from the older versions. The only - * exception is if the journal is clean as marked by the two restart pages - * then it doesn't matter whether we are on an earlier version. We can just - * reinitialize the logfile and start again with version 1.1. - */ - -/* Some $LogFile related constants. */ -#define MaxLogFileSize 0x100000000ULL -#define DefaultLogPageSize 4096 -#define MinLogRecordPages 48 - -/* - * Log file restart page header (begins the restart area). - */ -typedef struct { -/*Ofs*/ -/* 0 NTFS_RECORD; -- Unfolded here as gcc doesn't like unnamed structs. */ -/* 0*/ NTFS_RECORD_TYPE magic; /* The magic is "RSTR". */ -/* 4*/ le16 usa_ofs; /* See NTFS_RECORD definition in layout.h. - When creating, set this to be immediately - after this header structure (without any - alignment). */ -/* 6*/ le16 usa_count; /* See NTFS_RECORD definition in layout.h. */ - -/* 8*/ leLSN chkdsk_lsn; /* The last log file sequence number found by - chkdsk. Only used when the magic is changed - to "CHKD". Otherwise this is zero. */ -/* 16*/ le32 system_page_size; /* Byte size of system pages when the log file - was created, has to be >= 512 and a power of - 2. Use this to calculate the required size - of the usa (usa_count) and add it to usa_ofs. - Then verify that the result is less than the - value of the restart_area_offset. */ -/* 20*/ le32 log_page_size; /* Byte size of log file pages, has to be >= - 512 and a power of 2. The default is 4096 - and is used when the system page size is - between 4096 and 8192. Otherwise this is - set to the system page size instead. */ -/* 24*/ le16 restart_area_offset;/* Byte offset from the start of this header to - the RESTART_AREA. Value has to be aligned - to 8-byte boundary. When creating, set this - to be after the usa. */ -/* 26*/ sle16 minor_ver; /* Log file minor version. Only check if major - version is 1. */ -/* 28*/ sle16 major_ver; /* Log file major version. We only support - version 1.1. */ -/* sizeof() = 30 (0x1e) bytes */ -} __attribute__ ((__packed__)) RESTART_PAGE_HEADER; - -/* - * Constant for the log client indices meaning that there are no client records - * in this particular client array. Also inside the client records themselves, - * this means that there are no client records preceding or following this one. - */ -#define LOGFILE_NO_CLIENT cpu_to_le16(0xffff) -#define LOGFILE_NO_CLIENT_CPU 0xffff - -/* - * These are the so far known RESTART_AREA_* flags (16-bit) which contain - * information about the log file in which they are present. - */ -enum { - RESTART_VOLUME_IS_CLEAN = cpu_to_le16(0x0002), - RESTART_SPACE_FILLER = cpu_to_le16(0xffff), /* gcc: Force enum bit width to 16. */ -} __attribute__ ((__packed__)); - -typedef le16 RESTART_AREA_FLAGS; - -/* - * Log file restart area record. The offset of this record is found by adding - * the offset of the RESTART_PAGE_HEADER to the restart_area_offset value found - * in it. See notes at restart_area_offset above. - */ -typedef struct { -/*Ofs*/ -/* 0*/ leLSN current_lsn; /* The current, i.e. last LSN inside the log - when the restart area was last written. - This happens often but what is the interval? - Is it just fixed time or is it every time a - check point is written or somethine else? - On create set to 0. */ -/* 8*/ le16 log_clients; /* Number of log client records in the array of - log client records which follows this - restart area. Must be 1. */ -/* 10*/ le16 client_free_list; /* The index of the first free log client record - in the array of log client records. - LOGFILE_NO_CLIENT means that there are no - free log client records in the array. - If != LOGFILE_NO_CLIENT, check that - log_clients > client_free_list. On Win2k - and presumably earlier, on a clean volume - this is != LOGFILE_NO_CLIENT, and it should - be 0, i.e. the first (and only) client - record is free and thus the logfile is - closed and hence clean. A dirty volume - would have left the logfile open and hence - this would be LOGFILE_NO_CLIENT. On WinXP - and presumably later, the logfile is always - open, even on clean shutdown so this should - always be LOGFILE_NO_CLIENT. */ -/* 12*/ le16 client_in_use_list;/* The index of the first in-use log client - record in the array of log client records. - LOGFILE_NO_CLIENT means that there are no - in-use log client records in the array. If - != LOGFILE_NO_CLIENT check that log_clients - > client_in_use_list. On Win2k and - presumably earlier, on a clean volume this - is LOGFILE_NO_CLIENT, i.e. there are no - client records in use and thus the logfile - is closed and hence clean. A dirty volume - would have left the logfile open and hence - this would be != LOGFILE_NO_CLIENT, and it - should be 0, i.e. the first (and only) - client record is in use. On WinXP and - presumably later, the logfile is always - open, even on clean shutdown so this should - always be 0. */ -/* 14*/ RESTART_AREA_FLAGS flags;/* Flags modifying LFS behaviour. On Win2k - and presumably earlier this is always 0. On - WinXP and presumably later, if the logfile - was shutdown cleanly, the second bit, - RESTART_VOLUME_IS_CLEAN, is set. This bit - is cleared when the volume is mounted by - WinXP and set when the volume is dismounted, - thus if the logfile is dirty, this bit is - clear. Thus we don't need to check the - Windows version to determine if the logfile - is clean. Instead if the logfile is closed, - we know it must be clean. If it is open and - this bit is set, we also know it must be - clean. If on the other hand the logfile is - open and this bit is clear, we can be almost - certain that the logfile is dirty. */ -/* 16*/ le32 seq_number_bits; /* How many bits to use for the sequence - number. This is calculated as 67 - the - number of bits required to store the logfile - size in bytes and this can be used in with - the specified file_size as a consistency - check. */ -/* 20*/ le16 restart_area_length;/* Length of the restart area including the - client array. Following checks required if - version matches. Otherwise, skip them. - restart_area_offset + restart_area_length - has to be <= system_page_size. Also, - restart_area_length has to be >= - client_array_offset + (log_clients * - sizeof(log client record)). */ -/* 22*/ le16 client_array_offset;/* Offset from the start of this record to - the first log client record if versions are - matched. When creating, set this to be - after this restart area structure, aligned - to 8-bytes boundary. If the versions do not - match, this is ignored and the offset is - assumed to be (sizeof(RESTART_AREA) + 7) & - ~7, i.e. rounded up to first 8-byte - boundary. Either way, client_array_offset - has to be aligned to an 8-byte boundary. - Also, restart_area_offset + - client_array_offset has to be <= 510. - Finally, client_array_offset + (log_clients - * sizeof(log client record)) has to be <= - system_page_size. On Win2k and presumably - earlier, this is 0x30, i.e. immediately - following this record. On WinXP and - presumably later, this is 0x40, i.e. there - are 16 extra bytes between this record and - the client array. This probably means that - the RESTART_AREA record is actually bigger - in WinXP and later. */ -/* 24*/ sle64 file_size; /* Usable byte size of the log file. If the - restart_area_offset + the offset of the - file_size are > 510 then corruption has - occurred. This is the very first check when - starting with the restart_area as if it - fails it means that some of the above values - will be corrupted by the multi sector - transfer protection. The file_size has to - be rounded down to be a multiple of the - log_page_size in the RESTART_PAGE_HEADER and - then it has to be at least big enough to - store the two restart pages and 48 (0x30) - log record pages. */ -/* 32*/ le32 last_lsn_data_length;/* Length of data of last LSN, not including - the log record header. On create set to - 0. */ -/* 36*/ le16 log_record_header_length;/* Byte size of the log record header. - If the version matches then check that the - value of log_record_header_length is a - multiple of 8, i.e. - (log_record_header_length + 7) & ~7 == - log_record_header_length. When creating set - it to sizeof(LOG_RECORD_HEADER), aligned to - 8 bytes. */ -/* 38*/ le16 log_page_data_offset;/* Offset to the start of data in a log record - page. Must be a multiple of 8. On create - set it to immediately after the update - sequence array of the log record page. */ -/* 40*/ le32 restart_log_open_count;/* A counter that gets incremented every - time the logfile is restarted which happens - at mount time when the logfile is opened. - When creating set to a random value. Win2k - sets it to the low 32 bits of the current - system time in NTFS format (see time.h). */ -/* 44*/ le32 reserved; /* Reserved/alignment to 8-byte boundary. */ -/* sizeof() = 48 (0x30) bytes */ -} __attribute__ ((__packed__)) RESTART_AREA; - -/* - * Log client record. The offset of this record is found by adding the offset - * of the RESTART_AREA to the client_array_offset value found in it. - */ -typedef struct { -/*Ofs*/ -/* 0*/ leLSN oldest_lsn; /* Oldest LSN needed by this client. On create - set to 0. */ -/* 8*/ leLSN client_restart_lsn;/* LSN at which this client needs to restart - the volume, i.e. the current position within - the log file. At present, if clean this - should = current_lsn in restart area but it - probably also = current_lsn when dirty most - of the time. At create set to 0. */ -/* 16*/ le16 prev_client; /* The offset to the previous log client record - in the array of log client records. - LOGFILE_NO_CLIENT means there is no previous - client record, i.e. this is the first one. - This is always LOGFILE_NO_CLIENT. */ -/* 18*/ le16 next_client; /* The offset to the next log client record in - the array of log client records. - LOGFILE_NO_CLIENT means there are no next - client records, i.e. this is the last one. - This is always LOGFILE_NO_CLIENT. */ -/* 20*/ le16 seq_number; /* On Win2k and presumably earlier, this is set - to zero every time the logfile is restarted - and it is incremented when the logfile is - closed at dismount time. Thus it is 0 when - dirty and 1 when clean. On WinXP and - presumably later, this is always 0. */ -/* 22*/ u8 reserved[6]; /* Reserved/alignment. */ -/* 28*/ le32 client_name_length;/* Length of client name in bytes. Should - always be 8. */ -/* 32*/ ntfschar client_name[64];/* Name of the client in Unicode. Should - always be "NTFS" with the remaining bytes - set to 0. */ -/* sizeof() = 160 (0xa0) bytes */ -} __attribute__ ((__packed__)) LOG_CLIENT_RECORD; - -extern bool ntfs_check_logfile(struct inode *log_vi, - RESTART_PAGE_HEADER **rp); - -extern bool ntfs_is_logfile_clean(struct inode *log_vi, - const RESTART_PAGE_HEADER *rp); - -extern bool ntfs_empty_logfile(struct inode *log_vi); - -#endif /* NTFS_RW */ - -#endif /* _LINUX_NTFS_LOGFILE_H */ diff --git a/fs/ntfs/malloc.h b/fs/ntfs/malloc.h deleted file mode 100644 index 7068425735f1..000000000000 --- a/fs/ntfs/malloc.h +++ /dev/null @@ -1,77 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * malloc.h - NTFS kernel memory handling. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2005 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_MALLOC_H -#define _LINUX_NTFS_MALLOC_H - -#include -#include -#include - -/** - * __ntfs_malloc - allocate memory in multiples of pages - * @size: number of bytes to allocate - * @gfp_mask: extra flags for the allocator - * - * Internal function. You probably want ntfs_malloc_nofs()... - * - * Allocates @size bytes of memory, rounded up to multiples of PAGE_SIZE and - * returns a pointer to the allocated memory. - * - * If there was insufficient memory to complete the request, return NULL. - * Depending on @gfp_mask the allocation may be guaranteed to succeed. - */ -static inline void *__ntfs_malloc(unsigned long size, gfp_t gfp_mask) -{ - if (likely(size <= PAGE_SIZE)) { - BUG_ON(!size); - /* kmalloc() has per-CPU caches so is faster for now. */ - return kmalloc(PAGE_SIZE, gfp_mask & ~__GFP_HIGHMEM); - /* return (void *)__get_free_page(gfp_mask); */ - } - if (likely((size >> PAGE_SHIFT) < totalram_pages())) - return __vmalloc(size, gfp_mask); - return NULL; -} - -/** - * ntfs_malloc_nofs - allocate memory in multiples of pages - * @size: number of bytes to allocate - * - * Allocates @size bytes of memory, rounded up to multiples of PAGE_SIZE and - * returns a pointer to the allocated memory. - * - * If there was insufficient memory to complete the request, return NULL. - */ -static inline void *ntfs_malloc_nofs(unsigned long size) -{ - return __ntfs_malloc(size, GFP_NOFS | __GFP_HIGHMEM); -} - -/** - * ntfs_malloc_nofs_nofail - allocate memory in multiples of pages - * @size: number of bytes to allocate - * - * Allocates @size bytes of memory, rounded up to multiples of PAGE_SIZE and - * returns a pointer to the allocated memory. - * - * This function guarantees that the allocation will succeed. It will sleep - * for as long as it takes to complete the allocation. - * - * If there was insufficient memory to complete the request, return NULL. - */ -static inline void *ntfs_malloc_nofs_nofail(unsigned long size) -{ - return __ntfs_malloc(size, GFP_NOFS | __GFP_HIGHMEM | __GFP_NOFAIL); -} - -static inline void ntfs_free(void *addr) -{ - kvfree(addr); -} - -#endif /* _LINUX_NTFS_MALLOC_H */ diff --git a/fs/ntfs/mft.c b/fs/ntfs/mft.c deleted file mode 100644 index ad1a8f72da22..000000000000 --- a/fs/ntfs/mft.c +++ /dev/null @@ -1,2907 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * mft.c - NTFS kernel mft record operations. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2012 Anton Altaparmakov and Tuxera Inc. - * Copyright (c) 2002 Richard Russon - */ - -#include -#include -#include -#include - -#include "attrib.h" -#include "aops.h" -#include "bitmap.h" -#include "debug.h" -#include "dir.h" -#include "lcnalloc.h" -#include "malloc.h" -#include "mft.h" -#include "ntfs.h" - -#define MAX_BHS (PAGE_SIZE / NTFS_BLOCK_SIZE) - -/** - * map_mft_record_page - map the page in which a specific mft record resides - * @ni: ntfs inode whose mft record page to map - * - * This maps the page in which the mft record of the ntfs inode @ni is situated - * and returns a pointer to the mft record within the mapped page. - * - * Return value needs to be checked with IS_ERR() and if that is true PTR_ERR() - * contains the negative error code returned. - */ -static inline MFT_RECORD *map_mft_record_page(ntfs_inode *ni) -{ - loff_t i_size; - ntfs_volume *vol = ni->vol; - struct inode *mft_vi = vol->mft_ino; - struct page *page; - unsigned long index, end_index; - unsigned ofs; - - BUG_ON(ni->page); - /* - * The index into the page cache and the offset within the page cache - * page of the wanted mft record. FIXME: We need to check for - * overflowing the unsigned long, but I don't think we would ever get - * here if the volume was that big... - */ - index = (u64)ni->mft_no << vol->mft_record_size_bits >> - PAGE_SHIFT; - ofs = (ni->mft_no << vol->mft_record_size_bits) & ~PAGE_MASK; - - i_size = i_size_read(mft_vi); - /* The maximum valid index into the page cache for $MFT's data. */ - end_index = i_size >> PAGE_SHIFT; - - /* If the wanted index is out of bounds the mft record doesn't exist. */ - if (unlikely(index >= end_index)) { - if (index > end_index || (i_size & ~PAGE_MASK) < ofs + - vol->mft_record_size) { - page = ERR_PTR(-ENOENT); - ntfs_error(vol->sb, "Attempt to read mft record 0x%lx, " - "which is beyond the end of the mft. " - "This is probably a bug in the ntfs " - "driver.", ni->mft_no); - goto err_out; - } - } - /* Read, map, and pin the page. */ - page = ntfs_map_page(mft_vi->i_mapping, index); - if (!IS_ERR(page)) { - /* Catch multi sector transfer fixup errors. */ - if (likely(ntfs_is_mft_recordp((le32*)(page_address(page) + - ofs)))) { - ni->page = page; - ni->page_ofs = ofs; - return page_address(page) + ofs; - } - ntfs_error(vol->sb, "Mft record 0x%lx is corrupt. " - "Run chkdsk.", ni->mft_no); - ntfs_unmap_page(page); - page = ERR_PTR(-EIO); - NVolSetErrors(vol); - } -err_out: - ni->page = NULL; - ni->page_ofs = 0; - return (void*)page; -} - -/** - * map_mft_record - map, pin and lock an mft record - * @ni: ntfs inode whose MFT record to map - * - * First, take the mrec_lock mutex. We might now be sleeping, while waiting - * for the mutex if it was already locked by someone else. - * - * The page of the record is mapped using map_mft_record_page() before being - * returned to the caller. - * - * This in turn uses ntfs_map_page() to get the page containing the wanted mft - * record (it in turn calls read_cache_page() which reads it in from disk if - * necessary, increments the use count on the page so that it cannot disappear - * under us and returns a reference to the page cache page). - * - * If read_cache_page() invokes ntfs_readpage() to load the page from disk, it - * sets PG_locked and clears PG_uptodate on the page. Once I/O has completed - * and the post-read mst fixups on each mft record in the page have been - * performed, the page gets PG_uptodate set and PG_locked cleared (this is done - * in our asynchronous I/O completion handler end_buffer_read_mft_async()). - * ntfs_map_page() waits for PG_locked to become clear and checks if - * PG_uptodate is set and returns an error code if not. This provides - * sufficient protection against races when reading/using the page. - * - * However there is the write mapping to think about. Doing the above described - * checking here will be fine, because when initiating the write we will set - * PG_locked and clear PG_uptodate making sure nobody is touching the page - * contents. Doing the locking this way means that the commit to disk code in - * the page cache code paths is automatically sufficiently locked with us as - * we will not touch a page that has been locked or is not uptodate. The only - * locking problem then is them locking the page while we are accessing it. - * - * So that code will end up having to own the mrec_lock of all mft - * records/inodes present in the page before I/O can proceed. In that case we - * wouldn't need to bother with PG_locked and PG_uptodate as nobody will be - * accessing anything without owning the mrec_lock mutex. But we do need to - * use them because of the read_cache_page() invocation and the code becomes so - * much simpler this way that it is well worth it. - * - * The mft record is now ours and we return a pointer to it. You need to check - * the returned pointer with IS_ERR() and if that is true, PTR_ERR() will return - * the error code. - * - * NOTE: Caller is responsible for setting the mft record dirty before calling - * unmap_mft_record(). This is obviously only necessary if the caller really - * modified the mft record... - * Q: Do we want to recycle one of the VFS inode state bits instead? - * A: No, the inode ones mean we want to change the mft record, not we want to - * write it out. - */ -MFT_RECORD *map_mft_record(ntfs_inode *ni) -{ - MFT_RECORD *m; - - ntfs_debug("Entering for mft_no 0x%lx.", ni->mft_no); - - /* Make sure the ntfs inode doesn't go away. */ - atomic_inc(&ni->count); - - /* Serialize access to this mft record. */ - mutex_lock(&ni->mrec_lock); - - m = map_mft_record_page(ni); - if (!IS_ERR(m)) - return m; - - mutex_unlock(&ni->mrec_lock); - atomic_dec(&ni->count); - ntfs_error(ni->vol->sb, "Failed with error code %lu.", -PTR_ERR(m)); - return m; -} - -/** - * unmap_mft_record_page - unmap the page in which a specific mft record resides - * @ni: ntfs inode whose mft record page to unmap - * - * This unmaps the page in which the mft record of the ntfs inode @ni is - * situated and returns. This is a NOOP if highmem is not configured. - * - * The unmap happens via ntfs_unmap_page() which in turn decrements the use - * count on the page thus releasing it from the pinned state. - * - * We do not actually unmap the page from memory of course, as that will be - * done by the page cache code itself when memory pressure increases or - * whatever. - */ -static inline void unmap_mft_record_page(ntfs_inode *ni) -{ - BUG_ON(!ni->page); - - // TODO: If dirty, blah... - ntfs_unmap_page(ni->page); - ni->page = NULL; - ni->page_ofs = 0; - return; -} - -/** - * unmap_mft_record - release a mapped mft record - * @ni: ntfs inode whose MFT record to unmap - * - * We release the page mapping and the mrec_lock mutex which unmaps the mft - * record and releases it for others to get hold of. We also release the ntfs - * inode by decrementing the ntfs inode reference count. - * - * NOTE: If caller has modified the mft record, it is imperative to set the mft - * record dirty BEFORE calling unmap_mft_record(). - */ -void unmap_mft_record(ntfs_inode *ni) -{ - struct page *page = ni->page; - - BUG_ON(!page); - - ntfs_debug("Entering for mft_no 0x%lx.", ni->mft_no); - - unmap_mft_record_page(ni); - mutex_unlock(&ni->mrec_lock); - atomic_dec(&ni->count); - /* - * If pure ntfs_inode, i.e. no vfs inode attached, we leave it to - * ntfs_clear_extent_inode() in the extent inode case, and to the - * caller in the non-extent, yet pure ntfs inode case, to do the actual - * tear down of all structures and freeing of all allocated memory. - */ - return; -} - -/** - * map_extent_mft_record - load an extent inode and attach it to its base - * @base_ni: base ntfs inode - * @mref: mft reference of the extent inode to load - * @ntfs_ino: on successful return, pointer to the ntfs_inode structure - * - * Load the extent mft record @mref and attach it to its base inode @base_ni. - * Return the mapped extent mft record if IS_ERR(result) is false. Otherwise - * PTR_ERR(result) gives the negative error code. - * - * On successful return, @ntfs_ino contains a pointer to the ntfs_inode - * structure of the mapped extent inode. - */ -MFT_RECORD *map_extent_mft_record(ntfs_inode *base_ni, MFT_REF mref, - ntfs_inode **ntfs_ino) -{ - MFT_RECORD *m; - ntfs_inode *ni = NULL; - ntfs_inode **extent_nis = NULL; - int i; - unsigned long mft_no = MREF(mref); - u16 seq_no = MSEQNO(mref); - bool destroy_ni = false; - - ntfs_debug("Mapping extent mft record 0x%lx (base mft record 0x%lx).", - mft_no, base_ni->mft_no); - /* Make sure the base ntfs inode doesn't go away. */ - atomic_inc(&base_ni->count); - /* - * Check if this extent inode has already been added to the base inode, - * in which case just return it. If not found, add it to the base - * inode before returning it. - */ - mutex_lock(&base_ni->extent_lock); - if (base_ni->nr_extents > 0) { - extent_nis = base_ni->ext.extent_ntfs_inos; - for (i = 0; i < base_ni->nr_extents; i++) { - if (mft_no != extent_nis[i]->mft_no) - continue; - ni = extent_nis[i]; - /* Make sure the ntfs inode doesn't go away. */ - atomic_inc(&ni->count); - break; - } - } - if (likely(ni != NULL)) { - mutex_unlock(&base_ni->extent_lock); - atomic_dec(&base_ni->count); - /* We found the record; just have to map and return it. */ - m = map_mft_record(ni); - /* map_mft_record() has incremented this on success. */ - atomic_dec(&ni->count); - if (!IS_ERR(m)) { - /* Verify the sequence number. */ - if (likely(le16_to_cpu(m->sequence_number) == seq_no)) { - ntfs_debug("Done 1."); - *ntfs_ino = ni; - return m; - } - unmap_mft_record(ni); - ntfs_error(base_ni->vol->sb, "Found stale extent mft " - "reference! Corrupt filesystem. " - "Run chkdsk."); - return ERR_PTR(-EIO); - } -map_err_out: - ntfs_error(base_ni->vol->sb, "Failed to map extent " - "mft record, error code %ld.", -PTR_ERR(m)); - return m; - } - /* Record wasn't there. Get a new ntfs inode and initialize it. */ - ni = ntfs_new_extent_inode(base_ni->vol->sb, mft_no); - if (unlikely(!ni)) { - mutex_unlock(&base_ni->extent_lock); - atomic_dec(&base_ni->count); - return ERR_PTR(-ENOMEM); - } - ni->vol = base_ni->vol; - ni->seq_no = seq_no; - ni->nr_extents = -1; - ni->ext.base_ntfs_ino = base_ni; - /* Now map the record. */ - m = map_mft_record(ni); - if (IS_ERR(m)) { - mutex_unlock(&base_ni->extent_lock); - atomic_dec(&base_ni->count); - ntfs_clear_extent_inode(ni); - goto map_err_out; - } - /* Verify the sequence number if it is present. */ - if (seq_no && (le16_to_cpu(m->sequence_number) != seq_no)) { - ntfs_error(base_ni->vol->sb, "Found stale extent mft " - "reference! Corrupt filesystem. Run chkdsk."); - destroy_ni = true; - m = ERR_PTR(-EIO); - goto unm_err_out; - } - /* Attach extent inode to base inode, reallocating memory if needed. */ - if (!(base_ni->nr_extents & 3)) { - ntfs_inode **tmp; - int new_size = (base_ni->nr_extents + 4) * sizeof(ntfs_inode *); - - tmp = kmalloc(new_size, GFP_NOFS); - if (unlikely(!tmp)) { - ntfs_error(base_ni->vol->sb, "Failed to allocate " - "internal buffer."); - destroy_ni = true; - m = ERR_PTR(-ENOMEM); - goto unm_err_out; - } - if (base_ni->nr_extents) { - BUG_ON(!base_ni->ext.extent_ntfs_inos); - memcpy(tmp, base_ni->ext.extent_ntfs_inos, new_size - - 4 * sizeof(ntfs_inode *)); - kfree(base_ni->ext.extent_ntfs_inos); - } - base_ni->ext.extent_ntfs_inos = tmp; - } - base_ni->ext.extent_ntfs_inos[base_ni->nr_extents++] = ni; - mutex_unlock(&base_ni->extent_lock); - atomic_dec(&base_ni->count); - ntfs_debug("Done 2."); - *ntfs_ino = ni; - return m; -unm_err_out: - unmap_mft_record(ni); - mutex_unlock(&base_ni->extent_lock); - atomic_dec(&base_ni->count); - /* - * If the extent inode was not attached to the base inode we need to - * release it or we will leak memory. - */ - if (destroy_ni) - ntfs_clear_extent_inode(ni); - return m; -} - -#ifdef NTFS_RW - -/** - * __mark_mft_record_dirty - set the mft record and the page containing it dirty - * @ni: ntfs inode describing the mapped mft record - * - * Internal function. Users should call mark_mft_record_dirty() instead. - * - * Set the mapped (extent) mft record of the (base or extent) ntfs inode @ni, - * as well as the page containing the mft record, dirty. Also, mark the base - * vfs inode dirty. This ensures that any changes to the mft record are - * written out to disk. - * - * NOTE: We only set I_DIRTY_DATASYNC (and not I_DIRTY_PAGES) - * on the base vfs inode, because even though file data may have been modified, - * it is dirty in the inode meta data rather than the data page cache of the - * inode, and thus there are no data pages that need writing out. Therefore, a - * full mark_inode_dirty() is overkill. A mark_inode_dirty_sync(), on the - * other hand, is not sufficient, because ->write_inode needs to be called even - * in case of fdatasync. This needs to happen or the file data would not - * necessarily hit the device synchronously, even though the vfs inode has the - * O_SYNC flag set. Also, I_DIRTY_DATASYNC simply "feels" better than just - * I_DIRTY_SYNC, since the file data has not actually hit the block device yet, - * which is not what I_DIRTY_SYNC on its own would suggest. - */ -void __mark_mft_record_dirty(ntfs_inode *ni) -{ - ntfs_inode *base_ni; - - ntfs_debug("Entering for inode 0x%lx.", ni->mft_no); - BUG_ON(NInoAttr(ni)); - mark_ntfs_record_dirty(ni->page, ni->page_ofs); - /* Determine the base vfs inode and mark it dirty, too. */ - mutex_lock(&ni->extent_lock); - if (likely(ni->nr_extents >= 0)) - base_ni = ni; - else - base_ni = ni->ext.base_ntfs_ino; - mutex_unlock(&ni->extent_lock); - __mark_inode_dirty(VFS_I(base_ni), I_DIRTY_DATASYNC); -} - -static const char *ntfs_please_email = "Please email " - "linux-ntfs-dev@lists.sourceforge.net and say that you saw " - "this message. Thank you."; - -/** - * ntfs_sync_mft_mirror_umount - synchronise an mft record to the mft mirror - * @vol: ntfs volume on which the mft record to synchronize resides - * @mft_no: mft record number of mft record to synchronize - * @m: mapped, mst protected (extent) mft record to synchronize - * - * Write the mapped, mst protected (extent) mft record @m with mft record - * number @mft_no to the mft mirror ($MFTMirr) of the ntfs volume @vol, - * bypassing the page cache and the $MFTMirr inode itself. - * - * This function is only for use at umount time when the mft mirror inode has - * already been disposed off. We BUG() if we are called while the mft mirror - * inode is still attached to the volume. - * - * On success return 0. On error return -errno. - * - * NOTE: This function is not implemented yet as I am not convinced it can - * actually be triggered considering the sequence of commits we do in super.c:: - * ntfs_put_super(). But just in case we provide this place holder as the - * alternative would be either to BUG() or to get a NULL pointer dereference - * and Oops. - */ -static int ntfs_sync_mft_mirror_umount(ntfs_volume *vol, - const unsigned long mft_no, MFT_RECORD *m) -{ - BUG_ON(vol->mftmirr_ino); - ntfs_error(vol->sb, "Umount time mft mirror syncing is not " - "implemented yet. %s", ntfs_please_email); - return -EOPNOTSUPP; -} - -/** - * ntfs_sync_mft_mirror - synchronize an mft record to the mft mirror - * @vol: ntfs volume on which the mft record to synchronize resides - * @mft_no: mft record number of mft record to synchronize - * @m: mapped, mst protected (extent) mft record to synchronize - * @sync: if true, wait for i/o completion - * - * Write the mapped, mst protected (extent) mft record @m with mft record - * number @mft_no to the mft mirror ($MFTMirr) of the ntfs volume @vol. - * - * On success return 0. On error return -errno and set the volume errors flag - * in the ntfs volume @vol. - * - * NOTE: We always perform synchronous i/o and ignore the @sync parameter. - * - * TODO: If @sync is false, want to do truly asynchronous i/o, i.e. just - * schedule i/o via ->writepage or do it via kntfsd or whatever. - */ -int ntfs_sync_mft_mirror(ntfs_volume *vol, const unsigned long mft_no, - MFT_RECORD *m, int sync) -{ - struct page *page; - unsigned int blocksize = vol->sb->s_blocksize; - int max_bhs = vol->mft_record_size / blocksize; - struct buffer_head *bhs[MAX_BHS]; - struct buffer_head *bh, *head; - u8 *kmirr; - runlist_element *rl; - unsigned int block_start, block_end, m_start, m_end, page_ofs; - int i_bhs, nr_bhs, err = 0; - unsigned char blocksize_bits = vol->sb->s_blocksize_bits; - - ntfs_debug("Entering for inode 0x%lx.", mft_no); - BUG_ON(!max_bhs); - if (WARN_ON(max_bhs > MAX_BHS)) - return -EINVAL; - if (unlikely(!vol->mftmirr_ino)) { - /* This could happen during umount... */ - err = ntfs_sync_mft_mirror_umount(vol, mft_no, m); - if (likely(!err)) - return err; - goto err_out; - } - /* Get the page containing the mirror copy of the mft record @m. */ - page = ntfs_map_page(vol->mftmirr_ino->i_mapping, mft_no >> - (PAGE_SHIFT - vol->mft_record_size_bits)); - if (IS_ERR(page)) { - ntfs_error(vol->sb, "Failed to map mft mirror page."); - err = PTR_ERR(page); - goto err_out; - } - lock_page(page); - BUG_ON(!PageUptodate(page)); - ClearPageUptodate(page); - /* Offset of the mft mirror record inside the page. */ - page_ofs = (mft_no << vol->mft_record_size_bits) & ~PAGE_MASK; - /* The address in the page of the mirror copy of the mft record @m. */ - kmirr = page_address(page) + page_ofs; - /* Copy the mst protected mft record to the mirror. */ - memcpy(kmirr, m, vol->mft_record_size); - /* Create uptodate buffers if not present. */ - if (unlikely(!page_has_buffers(page))) { - struct buffer_head *tail; - - bh = head = alloc_page_buffers(page, blocksize, true); - do { - set_buffer_uptodate(bh); - tail = bh; - bh = bh->b_this_page; - } while (bh); - tail->b_this_page = head; - attach_page_private(page, head); - } - bh = head = page_buffers(page); - BUG_ON(!bh); - rl = NULL; - nr_bhs = 0; - block_start = 0; - m_start = kmirr - (u8*)page_address(page); - m_end = m_start + vol->mft_record_size; - do { - block_end = block_start + blocksize; - /* If the buffer is outside the mft record, skip it. */ - if (block_end <= m_start) - continue; - if (unlikely(block_start >= m_end)) - break; - /* Need to map the buffer if it is not mapped already. */ - if (unlikely(!buffer_mapped(bh))) { - VCN vcn; - LCN lcn; - unsigned int vcn_ofs; - - bh->b_bdev = vol->sb->s_bdev; - /* Obtain the vcn and offset of the current block. */ - vcn = ((VCN)mft_no << vol->mft_record_size_bits) + - (block_start - m_start); - vcn_ofs = vcn & vol->cluster_size_mask; - vcn >>= vol->cluster_size_bits; - if (!rl) { - down_read(&NTFS_I(vol->mftmirr_ino)-> - runlist.lock); - rl = NTFS_I(vol->mftmirr_ino)->runlist.rl; - /* - * $MFTMirr always has the whole of its runlist - * in memory. - */ - BUG_ON(!rl); - } - /* Seek to element containing target vcn. */ - while (rl->length && rl[1].vcn <= vcn) - rl++; - lcn = ntfs_rl_vcn_to_lcn(rl, vcn); - /* For $MFTMirr, only lcn >= 0 is a successful remap. */ - if (likely(lcn >= 0)) { - /* Setup buffer head to correct block. */ - bh->b_blocknr = ((lcn << - vol->cluster_size_bits) + - vcn_ofs) >> blocksize_bits; - set_buffer_mapped(bh); - } else { - bh->b_blocknr = -1; - ntfs_error(vol->sb, "Cannot write mft mirror " - "record 0x%lx because its " - "location on disk could not " - "be determined (error code " - "%lli).", mft_no, - (long long)lcn); - err = -EIO; - } - } - BUG_ON(!buffer_uptodate(bh)); - BUG_ON(!nr_bhs && (m_start != block_start)); - BUG_ON(nr_bhs >= max_bhs); - bhs[nr_bhs++] = bh; - BUG_ON((nr_bhs >= max_bhs) && (m_end != block_end)); - } while (block_start = block_end, (bh = bh->b_this_page) != head); - if (unlikely(rl)) - up_read(&NTFS_I(vol->mftmirr_ino)->runlist.lock); - if (likely(!err)) { - /* Lock buffers and start synchronous write i/o on them. */ - for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) { - struct buffer_head *tbh = bhs[i_bhs]; - - if (!trylock_buffer(tbh)) - BUG(); - BUG_ON(!buffer_uptodate(tbh)); - clear_buffer_dirty(tbh); - get_bh(tbh); - tbh->b_end_io = end_buffer_write_sync; - submit_bh(REQ_OP_WRITE, tbh); - } - /* Wait on i/o completion of buffers. */ - for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) { - struct buffer_head *tbh = bhs[i_bhs]; - - wait_on_buffer(tbh); - if (unlikely(!buffer_uptodate(tbh))) { - err = -EIO; - /* - * Set the buffer uptodate so the page and - * buffer states do not become out of sync. - */ - set_buffer_uptodate(tbh); - } - } - } else /* if (unlikely(err)) */ { - /* Clean the buffers. */ - for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) - clear_buffer_dirty(bhs[i_bhs]); - } - /* Current state: all buffers are clean, unlocked, and uptodate. */ - /* Remove the mst protection fixups again. */ - post_write_mst_fixup((NTFS_RECORD*)kmirr); - flush_dcache_page(page); - SetPageUptodate(page); - unlock_page(page); - ntfs_unmap_page(page); - if (likely(!err)) { - ntfs_debug("Done."); - } else { - ntfs_error(vol->sb, "I/O error while writing mft mirror " - "record 0x%lx!", mft_no); -err_out: - ntfs_error(vol->sb, "Failed to synchronize $MFTMirr (error " - "code %i). Volume will be left marked dirty " - "on umount. Run ntfsfix on the partition " - "after umounting to correct this.", -err); - NVolSetErrors(vol); - } - return err; -} - -/** - * write_mft_record_nolock - write out a mapped (extent) mft record - * @ni: ntfs inode describing the mapped (extent) mft record - * @m: mapped (extent) mft record to write - * @sync: if true, wait for i/o completion - * - * Write the mapped (extent) mft record @m described by the (regular or extent) - * ntfs inode @ni to backing store. If the mft record @m has a counterpart in - * the mft mirror, that is also updated. - * - * We only write the mft record if the ntfs inode @ni is dirty and the first - * buffer belonging to its mft record is dirty, too. We ignore the dirty state - * of subsequent buffers because we could have raced with - * fs/ntfs/aops.c::mark_ntfs_record_dirty(). - * - * On success, clean the mft record and return 0. On error, leave the mft - * record dirty and return -errno. - * - * NOTE: We always perform synchronous i/o and ignore the @sync parameter. - * However, if the mft record has a counterpart in the mft mirror and @sync is - * true, we write the mft record, wait for i/o completion, and only then write - * the mft mirror copy. This ensures that if the system crashes either the mft - * or the mft mirror will contain a self-consistent mft record @m. If @sync is - * false on the other hand, we start i/o on both and then wait for completion - * on them. This provides a speedup but no longer guarantees that you will end - * up with a self-consistent mft record in the case of a crash but if you asked - * for asynchronous writing you probably do not care about that anyway. - * - * TODO: If @sync is false, want to do truly asynchronous i/o, i.e. just - * schedule i/o via ->writepage or do it via kntfsd or whatever. - */ -int write_mft_record_nolock(ntfs_inode *ni, MFT_RECORD *m, int sync) -{ - ntfs_volume *vol = ni->vol; - struct page *page = ni->page; - unsigned int blocksize = vol->sb->s_blocksize; - unsigned char blocksize_bits = vol->sb->s_blocksize_bits; - int max_bhs = vol->mft_record_size / blocksize; - struct buffer_head *bhs[MAX_BHS]; - struct buffer_head *bh, *head; - runlist_element *rl; - unsigned int block_start, block_end, m_start, m_end; - int i_bhs, nr_bhs, err = 0; - - ntfs_debug("Entering for inode 0x%lx.", ni->mft_no); - BUG_ON(NInoAttr(ni)); - BUG_ON(!max_bhs); - BUG_ON(!PageLocked(page)); - if (WARN_ON(max_bhs > MAX_BHS)) { - err = -EINVAL; - goto err_out; - } - /* - * If the ntfs_inode is clean no need to do anything. If it is dirty, - * mark it as clean now so that it can be redirtied later on if needed. - * There is no danger of races since the caller is holding the locks - * for the mft record @m and the page it is in. - */ - if (!NInoTestClearDirty(ni)) - goto done; - bh = head = page_buffers(page); - BUG_ON(!bh); - rl = NULL; - nr_bhs = 0; - block_start = 0; - m_start = ni->page_ofs; - m_end = m_start + vol->mft_record_size; - do { - block_end = block_start + blocksize; - /* If the buffer is outside the mft record, skip it. */ - if (block_end <= m_start) - continue; - if (unlikely(block_start >= m_end)) - break; - /* - * If this block is not the first one in the record, we ignore - * the buffer's dirty state because we could have raced with a - * parallel mark_ntfs_record_dirty(). - */ - if (block_start == m_start) { - /* This block is the first one in the record. */ - if (!buffer_dirty(bh)) { - BUG_ON(nr_bhs); - /* Clean records are not written out. */ - break; - } - } - /* Need to map the buffer if it is not mapped already. */ - if (unlikely(!buffer_mapped(bh))) { - VCN vcn; - LCN lcn; - unsigned int vcn_ofs; - - bh->b_bdev = vol->sb->s_bdev; - /* Obtain the vcn and offset of the current block. */ - vcn = ((VCN)ni->mft_no << vol->mft_record_size_bits) + - (block_start - m_start); - vcn_ofs = vcn & vol->cluster_size_mask; - vcn >>= vol->cluster_size_bits; - if (!rl) { - down_read(&NTFS_I(vol->mft_ino)->runlist.lock); - rl = NTFS_I(vol->mft_ino)->runlist.rl; - BUG_ON(!rl); - } - /* Seek to element containing target vcn. */ - while (rl->length && rl[1].vcn <= vcn) - rl++; - lcn = ntfs_rl_vcn_to_lcn(rl, vcn); - /* For $MFT, only lcn >= 0 is a successful remap. */ - if (likely(lcn >= 0)) { - /* Setup buffer head to correct block. */ - bh->b_blocknr = ((lcn << - vol->cluster_size_bits) + - vcn_ofs) >> blocksize_bits; - set_buffer_mapped(bh); - } else { - bh->b_blocknr = -1; - ntfs_error(vol->sb, "Cannot write mft record " - "0x%lx because its location " - "on disk could not be " - "determined (error code %lli).", - ni->mft_no, (long long)lcn); - err = -EIO; - } - } - BUG_ON(!buffer_uptodate(bh)); - BUG_ON(!nr_bhs && (m_start != block_start)); - BUG_ON(nr_bhs >= max_bhs); - bhs[nr_bhs++] = bh; - BUG_ON((nr_bhs >= max_bhs) && (m_end != block_end)); - } while (block_start = block_end, (bh = bh->b_this_page) != head); - if (unlikely(rl)) - up_read(&NTFS_I(vol->mft_ino)->runlist.lock); - if (!nr_bhs) - goto done; - if (unlikely(err)) - goto cleanup_out; - /* Apply the mst protection fixups. */ - err = pre_write_mst_fixup((NTFS_RECORD*)m, vol->mft_record_size); - if (err) { - ntfs_error(vol->sb, "Failed to apply mst fixups!"); - goto cleanup_out; - } - flush_dcache_mft_record_page(ni); - /* Lock buffers and start synchronous write i/o on them. */ - for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) { - struct buffer_head *tbh = bhs[i_bhs]; - - if (!trylock_buffer(tbh)) - BUG(); - BUG_ON(!buffer_uptodate(tbh)); - clear_buffer_dirty(tbh); - get_bh(tbh); - tbh->b_end_io = end_buffer_write_sync; - submit_bh(REQ_OP_WRITE, tbh); - } - /* Synchronize the mft mirror now if not @sync. */ - if (!sync && ni->mft_no < vol->mftmirr_size) - ntfs_sync_mft_mirror(vol, ni->mft_no, m, sync); - /* Wait on i/o completion of buffers. */ - for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) { - struct buffer_head *tbh = bhs[i_bhs]; - - wait_on_buffer(tbh); - if (unlikely(!buffer_uptodate(tbh))) { - err = -EIO; - /* - * Set the buffer uptodate so the page and buffer - * states do not become out of sync. - */ - if (PageUptodate(page)) - set_buffer_uptodate(tbh); - } - } - /* If @sync, now synchronize the mft mirror. */ - if (sync && ni->mft_no < vol->mftmirr_size) - ntfs_sync_mft_mirror(vol, ni->mft_no, m, sync); - /* Remove the mst protection fixups again. */ - post_write_mst_fixup((NTFS_RECORD*)m); - flush_dcache_mft_record_page(ni); - if (unlikely(err)) { - /* I/O error during writing. This is really bad! */ - ntfs_error(vol->sb, "I/O error while writing mft record " - "0x%lx! Marking base inode as bad. You " - "should unmount the volume and run chkdsk.", - ni->mft_no); - goto err_out; - } -done: - ntfs_debug("Done."); - return 0; -cleanup_out: - /* Clean the buffers. */ - for (i_bhs = 0; i_bhs < nr_bhs; i_bhs++) - clear_buffer_dirty(bhs[i_bhs]); -err_out: - /* - * Current state: all buffers are clean, unlocked, and uptodate. - * The caller should mark the base inode as bad so that no more i/o - * happens. ->clear_inode() will still be invoked so all extent inodes - * and other allocated memory will be freed. - */ - if (err == -ENOMEM) { - ntfs_error(vol->sb, "Not enough memory to write mft record. " - "Redirtying so the write is retried later."); - mark_mft_record_dirty(ni); - err = 0; - } else - NVolSetErrors(vol); - return err; -} - -/** - * ntfs_may_write_mft_record - check if an mft record may be written out - * @vol: [IN] ntfs volume on which the mft record to check resides - * @mft_no: [IN] mft record number of the mft record to check - * @m: [IN] mapped mft record to check - * @locked_ni: [OUT] caller has to unlock this ntfs inode if one is returned - * - * Check if the mapped (base or extent) mft record @m with mft record number - * @mft_no belonging to the ntfs volume @vol may be written out. If necessary - * and possible the ntfs inode of the mft record is locked and the base vfs - * inode is pinned. The locked ntfs inode is then returned in @locked_ni. The - * caller is responsible for unlocking the ntfs inode and unpinning the base - * vfs inode. - * - * Return 'true' if the mft record may be written out and 'false' if not. - * - * The caller has locked the page and cleared the uptodate flag on it which - * means that we can safely write out any dirty mft records that do not have - * their inodes in icache as determined by ilookup5() as anyone - * opening/creating such an inode would block when attempting to map the mft - * record in read_cache_page() until we are finished with the write out. - * - * Here is a description of the tests we perform: - * - * If the inode is found in icache we know the mft record must be a base mft - * record. If it is dirty, we do not write it and return 'false' as the vfs - * inode write paths will result in the access times being updated which would - * cause the base mft record to be redirtied and written out again. (We know - * the access time update will modify the base mft record because Windows - * chkdsk complains if the standard information attribute is not in the base - * mft record.) - * - * If the inode is in icache and not dirty, we attempt to lock the mft record - * and if we find the lock was already taken, it is not safe to write the mft - * record and we return 'false'. - * - * If we manage to obtain the lock we have exclusive access to the mft record, - * which also allows us safe writeout of the mft record. We then set - * @locked_ni to the locked ntfs inode and return 'true'. - * - * Note we cannot just lock the mft record and sleep while waiting for the lock - * because this would deadlock due to lock reversal (normally the mft record is - * locked before the page is locked but we already have the page locked here - * when we try to lock the mft record). - * - * If the inode is not in icache we need to perform further checks. - * - * If the mft record is not a FILE record or it is a base mft record, we can - * safely write it and return 'true'. - * - * We now know the mft record is an extent mft record. We check if the inode - * corresponding to its base mft record is in icache and obtain a reference to - * it if it is. If it is not, we can safely write it and return 'true'. - * - * We now have the base inode for the extent mft record. We check if it has an - * ntfs inode for the extent mft record attached and if not it is safe to write - * the extent mft record and we return 'true'. - * - * The ntfs inode for the extent mft record is attached to the base inode so we - * attempt to lock the extent mft record and if we find the lock was already - * taken, it is not safe to write the extent mft record and we return 'false'. - * - * If we manage to obtain the lock we have exclusive access to the extent mft - * record, which also allows us safe writeout of the extent mft record. We - * set the ntfs inode of the extent mft record clean and then set @locked_ni to - * the now locked ntfs inode and return 'true'. - * - * Note, the reason for actually writing dirty mft records here and not just - * relying on the vfs inode dirty code paths is that we can have mft records - * modified without them ever having actual inodes in memory. Also we can have - * dirty mft records with clean ntfs inodes in memory. None of the described - * cases would result in the dirty mft records being written out if we only - * relied on the vfs inode dirty code paths. And these cases can really occur - * during allocation of new mft records and in particular when the - * initialized_size of the $MFT/$DATA attribute is extended and the new space - * is initialized using ntfs_mft_record_format(). The clean inode can then - * appear if the mft record is reused for a new inode before it got written - * out. - */ -bool ntfs_may_write_mft_record(ntfs_volume *vol, const unsigned long mft_no, - const MFT_RECORD *m, ntfs_inode **locked_ni) -{ - struct super_block *sb = vol->sb; - struct inode *mft_vi = vol->mft_ino; - struct inode *vi; - ntfs_inode *ni, *eni, **extent_nis; - int i; - ntfs_attr na; - - ntfs_debug("Entering for inode 0x%lx.", mft_no); - /* - * Normally we do not return a locked inode so set @locked_ni to NULL. - */ - BUG_ON(!locked_ni); - *locked_ni = NULL; - /* - * Check if the inode corresponding to this mft record is in the VFS - * inode cache and obtain a reference to it if it is. - */ - ntfs_debug("Looking for inode 0x%lx in icache.", mft_no); - na.mft_no = mft_no; - na.name = NULL; - na.name_len = 0; - na.type = AT_UNUSED; - /* - * Optimize inode 0, i.e. $MFT itself, since we have it in memory and - * we get here for it rather often. - */ - if (!mft_no) { - /* Balance the below iput(). */ - vi = igrab(mft_vi); - BUG_ON(vi != mft_vi); - } else { - /* - * Have to use ilookup5_nowait() since ilookup5() waits for the - * inode lock which causes ntfs to deadlock when a concurrent - * inode write via the inode dirty code paths and the page - * dirty code path of the inode dirty code path when writing - * $MFT occurs. - */ - vi = ilookup5_nowait(sb, mft_no, ntfs_test_inode, &na); - } - if (vi) { - ntfs_debug("Base inode 0x%lx is in icache.", mft_no); - /* The inode is in icache. */ - ni = NTFS_I(vi); - /* Take a reference to the ntfs inode. */ - atomic_inc(&ni->count); - /* If the inode is dirty, do not write this record. */ - if (NInoDirty(ni)) { - ntfs_debug("Inode 0x%lx is dirty, do not write it.", - mft_no); - atomic_dec(&ni->count); - iput(vi); - return false; - } - ntfs_debug("Inode 0x%lx is not dirty.", mft_no); - /* The inode is not dirty, try to take the mft record lock. */ - if (unlikely(!mutex_trylock(&ni->mrec_lock))) { - ntfs_debug("Mft record 0x%lx is already locked, do " - "not write it.", mft_no); - atomic_dec(&ni->count); - iput(vi); - return false; - } - ntfs_debug("Managed to lock mft record 0x%lx, write it.", - mft_no); - /* - * The write has to occur while we hold the mft record lock so - * return the locked ntfs inode. - */ - *locked_ni = ni; - return true; - } - ntfs_debug("Inode 0x%lx is not in icache.", mft_no); - /* The inode is not in icache. */ - /* Write the record if it is not a mft record (type "FILE"). */ - if (!ntfs_is_mft_record(m->magic)) { - ntfs_debug("Mft record 0x%lx is not a FILE record, write it.", - mft_no); - return true; - } - /* Write the mft record if it is a base inode. */ - if (!m->base_mft_record) { - ntfs_debug("Mft record 0x%lx is a base record, write it.", - mft_no); - return true; - } - /* - * This is an extent mft record. Check if the inode corresponding to - * its base mft record is in icache and obtain a reference to it if it - * is. - */ - na.mft_no = MREF_LE(m->base_mft_record); - ntfs_debug("Mft record 0x%lx is an extent record. Looking for base " - "inode 0x%lx in icache.", mft_no, na.mft_no); - if (!na.mft_no) { - /* Balance the below iput(). */ - vi = igrab(mft_vi); - BUG_ON(vi != mft_vi); - } else - vi = ilookup5_nowait(sb, na.mft_no, ntfs_test_inode, - &na); - if (!vi) { - /* - * The base inode is not in icache, write this extent mft - * record. - */ - ntfs_debug("Base inode 0x%lx is not in icache, write the " - "extent record.", na.mft_no); - return true; - } - ntfs_debug("Base inode 0x%lx is in icache.", na.mft_no); - /* - * The base inode is in icache. Check if it has the extent inode - * corresponding to this extent mft record attached. - */ - ni = NTFS_I(vi); - mutex_lock(&ni->extent_lock); - if (ni->nr_extents <= 0) { - /* - * The base inode has no attached extent inodes, write this - * extent mft record. - */ - mutex_unlock(&ni->extent_lock); - iput(vi); - ntfs_debug("Base inode 0x%lx has no attached extent inodes, " - "write the extent record.", na.mft_no); - return true; - } - /* Iterate over the attached extent inodes. */ - extent_nis = ni->ext.extent_ntfs_inos; - for (eni = NULL, i = 0; i < ni->nr_extents; ++i) { - if (mft_no == extent_nis[i]->mft_no) { - /* - * Found the extent inode corresponding to this extent - * mft record. - */ - eni = extent_nis[i]; - break; - } - } - /* - * If the extent inode was not attached to the base inode, write this - * extent mft record. - */ - if (!eni) { - mutex_unlock(&ni->extent_lock); - iput(vi); - ntfs_debug("Extent inode 0x%lx is not attached to its base " - "inode 0x%lx, write the extent record.", - mft_no, na.mft_no); - return true; - } - ntfs_debug("Extent inode 0x%lx is attached to its base inode 0x%lx.", - mft_no, na.mft_no); - /* Take a reference to the extent ntfs inode. */ - atomic_inc(&eni->count); - mutex_unlock(&ni->extent_lock); - /* - * Found the extent inode coresponding to this extent mft record. - * Try to take the mft record lock. - */ - if (unlikely(!mutex_trylock(&eni->mrec_lock))) { - atomic_dec(&eni->count); - iput(vi); - ntfs_debug("Extent mft record 0x%lx is already locked, do " - "not write it.", mft_no); - return false; - } - ntfs_debug("Managed to lock extent mft record 0x%lx, write it.", - mft_no); - if (NInoTestClearDirty(eni)) - ntfs_debug("Extent inode 0x%lx is dirty, marking it clean.", - mft_no); - /* - * The write has to occur while we hold the mft record lock so return - * the locked extent ntfs inode. - */ - *locked_ni = eni; - return true; -} - -static const char *es = " Leaving inconsistent metadata. Unmount and run " - "chkdsk."; - -/** - * ntfs_mft_bitmap_find_and_alloc_free_rec_nolock - see name - * @vol: volume on which to search for a free mft record - * @base_ni: open base inode if allocating an extent mft record or NULL - * - * Search for a free mft record in the mft bitmap attribute on the ntfs volume - * @vol. - * - * If @base_ni is NULL start the search at the default allocator position. - * - * If @base_ni is not NULL start the search at the mft record after the base - * mft record @base_ni. - * - * Return the free mft record on success and -errno on error. An error code of - * -ENOSPC means that there are no free mft records in the currently - * initialized mft bitmap. - * - * Locking: Caller must hold vol->mftbmp_lock for writing. - */ -static int ntfs_mft_bitmap_find_and_alloc_free_rec_nolock(ntfs_volume *vol, - ntfs_inode *base_ni) -{ - s64 pass_end, ll, data_pos, pass_start, ofs, bit; - unsigned long flags; - struct address_space *mftbmp_mapping; - u8 *buf, *byte; - struct page *page; - unsigned int page_ofs, size; - u8 pass, b; - - ntfs_debug("Searching for free mft record in the currently " - "initialized mft bitmap."); - mftbmp_mapping = vol->mftbmp_ino->i_mapping; - /* - * Set the end of the pass making sure we do not overflow the mft - * bitmap. - */ - read_lock_irqsave(&NTFS_I(vol->mft_ino)->size_lock, flags); - pass_end = NTFS_I(vol->mft_ino)->allocated_size >> - vol->mft_record_size_bits; - read_unlock_irqrestore(&NTFS_I(vol->mft_ino)->size_lock, flags); - read_lock_irqsave(&NTFS_I(vol->mftbmp_ino)->size_lock, flags); - ll = NTFS_I(vol->mftbmp_ino)->initialized_size << 3; - read_unlock_irqrestore(&NTFS_I(vol->mftbmp_ino)->size_lock, flags); - if (pass_end > ll) - pass_end = ll; - pass = 1; - if (!base_ni) - data_pos = vol->mft_data_pos; - else - data_pos = base_ni->mft_no + 1; - if (data_pos < 24) - data_pos = 24; - if (data_pos >= pass_end) { - data_pos = 24; - pass = 2; - /* This happens on a freshly formatted volume. */ - if (data_pos >= pass_end) - return -ENOSPC; - } - pass_start = data_pos; - ntfs_debug("Starting bitmap search: pass %u, pass_start 0x%llx, " - "pass_end 0x%llx, data_pos 0x%llx.", pass, - (long long)pass_start, (long long)pass_end, - (long long)data_pos); - /* Loop until a free mft record is found. */ - for (; pass <= 2;) { - /* Cap size to pass_end. */ - ofs = data_pos >> 3; - page_ofs = ofs & ~PAGE_MASK; - size = PAGE_SIZE - page_ofs; - ll = ((pass_end + 7) >> 3) - ofs; - if (size > ll) - size = ll; - size <<= 3; - /* - * If we are still within the active pass, search the next page - * for a zero bit. - */ - if (size) { - page = ntfs_map_page(mftbmp_mapping, - ofs >> PAGE_SHIFT); - if (IS_ERR(page)) { - ntfs_error(vol->sb, "Failed to read mft " - "bitmap, aborting."); - return PTR_ERR(page); - } - buf = (u8*)page_address(page) + page_ofs; - bit = data_pos & 7; - data_pos &= ~7ull; - ntfs_debug("Before inner for loop: size 0x%x, " - "data_pos 0x%llx, bit 0x%llx", size, - (long long)data_pos, (long long)bit); - for (; bit < size && data_pos + bit < pass_end; - bit &= ~7ull, bit += 8) { - byte = buf + (bit >> 3); - if (*byte == 0xff) - continue; - b = ffz((unsigned long)*byte); - if (b < 8 && b >= (bit & 7)) { - ll = data_pos + (bit & ~7ull) + b; - if (unlikely(ll > (1ll << 32))) { - ntfs_unmap_page(page); - return -ENOSPC; - } - *byte |= 1 << b; - flush_dcache_page(page); - set_page_dirty(page); - ntfs_unmap_page(page); - ntfs_debug("Done. (Found and " - "allocated mft record " - "0x%llx.)", - (long long)ll); - return ll; - } - } - ntfs_debug("After inner for loop: size 0x%x, " - "data_pos 0x%llx, bit 0x%llx", size, - (long long)data_pos, (long long)bit); - data_pos += size; - ntfs_unmap_page(page); - /* - * If the end of the pass has not been reached yet, - * continue searching the mft bitmap for a zero bit. - */ - if (data_pos < pass_end) - continue; - } - /* Do the next pass. */ - if (++pass == 2) { - /* - * Starting the second pass, in which we scan the first - * part of the zone which we omitted earlier. - */ - pass_end = pass_start; - data_pos = pass_start = 24; - ntfs_debug("pass %i, pass_start 0x%llx, pass_end " - "0x%llx.", pass, (long long)pass_start, - (long long)pass_end); - if (data_pos >= pass_end) - break; - } - } - /* No free mft records in currently initialized mft bitmap. */ - ntfs_debug("Done. (No free mft records left in currently initialized " - "mft bitmap.)"); - return -ENOSPC; -} - -/** - * ntfs_mft_bitmap_extend_allocation_nolock - extend mft bitmap by a cluster - * @vol: volume on which to extend the mft bitmap attribute - * - * Extend the mft bitmap attribute on the ntfs volume @vol by one cluster. - * - * Note: Only changes allocated_size, i.e. does not touch initialized_size or - * data_size. - * - * Return 0 on success and -errno on error. - * - * Locking: - Caller must hold vol->mftbmp_lock for writing. - * - This function takes NTFS_I(vol->mftbmp_ino)->runlist.lock for - * writing and releases it before returning. - * - This function takes vol->lcnbmp_lock for writing and releases it - * before returning. - */ -static int ntfs_mft_bitmap_extend_allocation_nolock(ntfs_volume *vol) -{ - LCN lcn; - s64 ll; - unsigned long flags; - struct page *page; - ntfs_inode *mft_ni, *mftbmp_ni; - runlist_element *rl, *rl2 = NULL; - ntfs_attr_search_ctx *ctx = NULL; - MFT_RECORD *mrec; - ATTR_RECORD *a = NULL; - int ret, mp_size; - u32 old_alen = 0; - u8 *b, tb; - struct { - u8 added_cluster:1; - u8 added_run:1; - u8 mp_rebuilt:1; - } status = { 0, 0, 0 }; - - ntfs_debug("Extending mft bitmap allocation."); - mft_ni = NTFS_I(vol->mft_ino); - mftbmp_ni = NTFS_I(vol->mftbmp_ino); - /* - * Determine the last lcn of the mft bitmap. The allocated size of the - * mft bitmap cannot be zero so we are ok to do this. - */ - down_write(&mftbmp_ni->runlist.lock); - read_lock_irqsave(&mftbmp_ni->size_lock, flags); - ll = mftbmp_ni->allocated_size; - read_unlock_irqrestore(&mftbmp_ni->size_lock, flags); - rl = ntfs_attr_find_vcn_nolock(mftbmp_ni, - (ll - 1) >> vol->cluster_size_bits, NULL); - if (IS_ERR(rl) || unlikely(!rl->length || rl->lcn < 0)) { - up_write(&mftbmp_ni->runlist.lock); - ntfs_error(vol->sb, "Failed to determine last allocated " - "cluster of mft bitmap attribute."); - if (!IS_ERR(rl)) - ret = -EIO; - else - ret = PTR_ERR(rl); - return ret; - } - lcn = rl->lcn + rl->length; - ntfs_debug("Last lcn of mft bitmap attribute is 0x%llx.", - (long long)lcn); - /* - * Attempt to get the cluster following the last allocated cluster by - * hand as it may be in the MFT zone so the allocator would not give it - * to us. - */ - ll = lcn >> 3; - page = ntfs_map_page(vol->lcnbmp_ino->i_mapping, - ll >> PAGE_SHIFT); - if (IS_ERR(page)) { - up_write(&mftbmp_ni->runlist.lock); - ntfs_error(vol->sb, "Failed to read from lcn bitmap."); - return PTR_ERR(page); - } - b = (u8*)page_address(page) + (ll & ~PAGE_MASK); - tb = 1 << (lcn & 7ull); - down_write(&vol->lcnbmp_lock); - if (*b != 0xff && !(*b & tb)) { - /* Next cluster is free, allocate it. */ - *b |= tb; - flush_dcache_page(page); - set_page_dirty(page); - up_write(&vol->lcnbmp_lock); - ntfs_unmap_page(page); - /* Update the mft bitmap runlist. */ - rl->length++; - rl[1].vcn++; - status.added_cluster = 1; - ntfs_debug("Appending one cluster to mft bitmap."); - } else { - up_write(&vol->lcnbmp_lock); - ntfs_unmap_page(page); - /* Allocate a cluster from the DATA_ZONE. */ - rl2 = ntfs_cluster_alloc(vol, rl[1].vcn, 1, lcn, DATA_ZONE, - true); - if (IS_ERR(rl2)) { - up_write(&mftbmp_ni->runlist.lock); - ntfs_error(vol->sb, "Failed to allocate a cluster for " - "the mft bitmap."); - return PTR_ERR(rl2); - } - rl = ntfs_runlists_merge(mftbmp_ni->runlist.rl, rl2); - if (IS_ERR(rl)) { - up_write(&mftbmp_ni->runlist.lock); - ntfs_error(vol->sb, "Failed to merge runlists for mft " - "bitmap."); - if (ntfs_cluster_free_from_rl(vol, rl2)) { - ntfs_error(vol->sb, "Failed to deallocate " - "allocated cluster.%s", es); - NVolSetErrors(vol); - } - ntfs_free(rl2); - return PTR_ERR(rl); - } - mftbmp_ni->runlist.rl = rl; - status.added_run = 1; - ntfs_debug("Adding one run to mft bitmap."); - /* Find the last run in the new runlist. */ - for (; rl[1].length; rl++) - ; - } - /* - * Update the attribute record as well. Note: @rl is the last - * (non-terminator) runlist element of mft bitmap. - */ - mrec = map_mft_record(mft_ni); - if (IS_ERR(mrec)) { - ntfs_error(vol->sb, "Failed to map mft record."); - ret = PTR_ERR(mrec); - goto undo_alloc; - } - ctx = ntfs_attr_get_search_ctx(mft_ni, mrec); - if (unlikely(!ctx)) { - ntfs_error(vol->sb, "Failed to get search context."); - ret = -ENOMEM; - goto undo_alloc; - } - ret = ntfs_attr_lookup(mftbmp_ni->type, mftbmp_ni->name, - mftbmp_ni->name_len, CASE_SENSITIVE, rl[1].vcn, NULL, - 0, ctx); - if (unlikely(ret)) { - ntfs_error(vol->sb, "Failed to find last attribute extent of " - "mft bitmap attribute."); - if (ret == -ENOENT) - ret = -EIO; - goto undo_alloc; - } - a = ctx->attr; - ll = sle64_to_cpu(a->data.non_resident.lowest_vcn); - /* Search back for the previous last allocated cluster of mft bitmap. */ - for (rl2 = rl; rl2 > mftbmp_ni->runlist.rl; rl2--) { - if (ll >= rl2->vcn) - break; - } - BUG_ON(ll < rl2->vcn); - BUG_ON(ll >= rl2->vcn + rl2->length); - /* Get the size for the new mapping pairs array for this extent. */ - mp_size = ntfs_get_size_for_mapping_pairs(vol, rl2, ll, -1); - if (unlikely(mp_size <= 0)) { - ntfs_error(vol->sb, "Get size for mapping pairs failed for " - "mft bitmap attribute extent."); - ret = mp_size; - if (!ret) - ret = -EIO; - goto undo_alloc; - } - /* Expand the attribute record if necessary. */ - old_alen = le32_to_cpu(a->length); - ret = ntfs_attr_record_resize(ctx->mrec, a, mp_size + - le16_to_cpu(a->data.non_resident.mapping_pairs_offset)); - if (unlikely(ret)) { - if (ret != -ENOSPC) { - ntfs_error(vol->sb, "Failed to resize attribute " - "record for mft bitmap attribute."); - goto undo_alloc; - } - // TODO: Deal with this by moving this extent to a new mft - // record or by starting a new extent in a new mft record or by - // moving other attributes out of this mft record. - // Note: It will need to be a special mft record and if none of - // those are available it gets rather complicated... - ntfs_error(vol->sb, "Not enough space in this mft record to " - "accommodate extended mft bitmap attribute " - "extent. Cannot handle this yet."); - ret = -EOPNOTSUPP; - goto undo_alloc; - } - status.mp_rebuilt = 1; - /* Generate the mapping pairs array directly into the attr record. */ - ret = ntfs_mapping_pairs_build(vol, (u8*)a + - le16_to_cpu(a->data.non_resident.mapping_pairs_offset), - mp_size, rl2, ll, -1, NULL); - if (unlikely(ret)) { - ntfs_error(vol->sb, "Failed to build mapping pairs array for " - "mft bitmap attribute."); - goto undo_alloc; - } - /* Update the highest_vcn. */ - a->data.non_resident.highest_vcn = cpu_to_sle64(rl[1].vcn - 1); - /* - * We now have extended the mft bitmap allocated_size by one cluster. - * Reflect this in the ntfs_inode structure and the attribute record. - */ - if (a->data.non_resident.lowest_vcn) { - /* - * We are not in the first attribute extent, switch to it, but - * first ensure the changes will make it to disk later. - */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_reinit_search_ctx(ctx); - ret = ntfs_attr_lookup(mftbmp_ni->type, mftbmp_ni->name, - mftbmp_ni->name_len, CASE_SENSITIVE, 0, NULL, - 0, ctx); - if (unlikely(ret)) { - ntfs_error(vol->sb, "Failed to find first attribute " - "extent of mft bitmap attribute."); - goto restore_undo_alloc; - } - a = ctx->attr; - } - write_lock_irqsave(&mftbmp_ni->size_lock, flags); - mftbmp_ni->allocated_size += vol->cluster_size; - a->data.non_resident.allocated_size = - cpu_to_sle64(mftbmp_ni->allocated_size); - write_unlock_irqrestore(&mftbmp_ni->size_lock, flags); - /* Ensure the changes make it to disk. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(mft_ni); - up_write(&mftbmp_ni->runlist.lock); - ntfs_debug("Done."); - return 0; -restore_undo_alloc: - ntfs_attr_reinit_search_ctx(ctx); - if (ntfs_attr_lookup(mftbmp_ni->type, mftbmp_ni->name, - mftbmp_ni->name_len, CASE_SENSITIVE, rl[1].vcn, NULL, - 0, ctx)) { - ntfs_error(vol->sb, "Failed to find last attribute extent of " - "mft bitmap attribute.%s", es); - write_lock_irqsave(&mftbmp_ni->size_lock, flags); - mftbmp_ni->allocated_size += vol->cluster_size; - write_unlock_irqrestore(&mftbmp_ni->size_lock, flags); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(mft_ni); - up_write(&mftbmp_ni->runlist.lock); - /* - * The only thing that is now wrong is ->allocated_size of the - * base attribute extent which chkdsk should be able to fix. - */ - NVolSetErrors(vol); - return ret; - } - a = ctx->attr; - a->data.non_resident.highest_vcn = cpu_to_sle64(rl[1].vcn - 2); -undo_alloc: - if (status.added_cluster) { - /* Truncate the last run in the runlist by one cluster. */ - rl->length--; - rl[1].vcn--; - } else if (status.added_run) { - lcn = rl->lcn; - /* Remove the last run from the runlist. */ - rl->lcn = rl[1].lcn; - rl->length = 0; - } - /* Deallocate the cluster. */ - down_write(&vol->lcnbmp_lock); - if (ntfs_bitmap_clear_bit(vol->lcnbmp_ino, lcn)) { - ntfs_error(vol->sb, "Failed to free allocated cluster.%s", es); - NVolSetErrors(vol); - } - up_write(&vol->lcnbmp_lock); - if (status.mp_rebuilt) { - if (ntfs_mapping_pairs_build(vol, (u8*)a + le16_to_cpu( - a->data.non_resident.mapping_pairs_offset), - old_alen - le16_to_cpu( - a->data.non_resident.mapping_pairs_offset), - rl2, ll, -1, NULL)) { - ntfs_error(vol->sb, "Failed to restore mapping pairs " - "array.%s", es); - NVolSetErrors(vol); - } - if (ntfs_attr_record_resize(ctx->mrec, a, old_alen)) { - ntfs_error(vol->sb, "Failed to restore attribute " - "record.%s", es); - NVolSetErrors(vol); - } - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - } - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (!IS_ERR(mrec)) - unmap_mft_record(mft_ni); - up_write(&mftbmp_ni->runlist.lock); - return ret; -} - -/** - * ntfs_mft_bitmap_extend_initialized_nolock - extend mftbmp initialized data - * @vol: volume on which to extend the mft bitmap attribute - * - * Extend the initialized portion of the mft bitmap attribute on the ntfs - * volume @vol by 8 bytes. - * - * Note: Only changes initialized_size and data_size, i.e. requires that - * allocated_size is big enough to fit the new initialized_size. - * - * Return 0 on success and -error on error. - * - * Locking: Caller must hold vol->mftbmp_lock for writing. - */ -static int ntfs_mft_bitmap_extend_initialized_nolock(ntfs_volume *vol) -{ - s64 old_data_size, old_initialized_size; - unsigned long flags; - struct inode *mftbmp_vi; - ntfs_inode *mft_ni, *mftbmp_ni; - ntfs_attr_search_ctx *ctx; - MFT_RECORD *mrec; - ATTR_RECORD *a; - int ret; - - ntfs_debug("Extending mft bitmap initiailized (and data) size."); - mft_ni = NTFS_I(vol->mft_ino); - mftbmp_vi = vol->mftbmp_ino; - mftbmp_ni = NTFS_I(mftbmp_vi); - /* Get the attribute record. */ - mrec = map_mft_record(mft_ni); - if (IS_ERR(mrec)) { - ntfs_error(vol->sb, "Failed to map mft record."); - return PTR_ERR(mrec); - } - ctx = ntfs_attr_get_search_ctx(mft_ni, mrec); - if (unlikely(!ctx)) { - ntfs_error(vol->sb, "Failed to get search context."); - ret = -ENOMEM; - goto unm_err_out; - } - ret = ntfs_attr_lookup(mftbmp_ni->type, mftbmp_ni->name, - mftbmp_ni->name_len, CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(ret)) { - ntfs_error(vol->sb, "Failed to find first attribute extent of " - "mft bitmap attribute."); - if (ret == -ENOENT) - ret = -EIO; - goto put_err_out; - } - a = ctx->attr; - write_lock_irqsave(&mftbmp_ni->size_lock, flags); - old_data_size = i_size_read(mftbmp_vi); - old_initialized_size = mftbmp_ni->initialized_size; - /* - * We can simply update the initialized_size before filling the space - * with zeroes because the caller is holding the mft bitmap lock for - * writing which ensures that no one else is trying to access the data. - */ - mftbmp_ni->initialized_size += 8; - a->data.non_resident.initialized_size = - cpu_to_sle64(mftbmp_ni->initialized_size); - if (mftbmp_ni->initialized_size > old_data_size) { - i_size_write(mftbmp_vi, mftbmp_ni->initialized_size); - a->data.non_resident.data_size = - cpu_to_sle64(mftbmp_ni->initialized_size); - } - write_unlock_irqrestore(&mftbmp_ni->size_lock, flags); - /* Ensure the changes make it to disk. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(mft_ni); - /* Initialize the mft bitmap attribute value with zeroes. */ - ret = ntfs_attr_set(mftbmp_ni, old_initialized_size, 8, 0); - if (likely(!ret)) { - ntfs_debug("Done. (Wrote eight initialized bytes to mft " - "bitmap."); - return 0; - } - ntfs_error(vol->sb, "Failed to write to mft bitmap."); - /* Try to recover from the error. */ - mrec = map_mft_record(mft_ni); - if (IS_ERR(mrec)) { - ntfs_error(vol->sb, "Failed to map mft record.%s", es); - NVolSetErrors(vol); - return ret; - } - ctx = ntfs_attr_get_search_ctx(mft_ni, mrec); - if (unlikely(!ctx)) { - ntfs_error(vol->sb, "Failed to get search context.%s", es); - NVolSetErrors(vol); - goto unm_err_out; - } - if (ntfs_attr_lookup(mftbmp_ni->type, mftbmp_ni->name, - mftbmp_ni->name_len, CASE_SENSITIVE, 0, NULL, 0, ctx)) { - ntfs_error(vol->sb, "Failed to find first attribute extent of " - "mft bitmap attribute.%s", es); - NVolSetErrors(vol); -put_err_out: - ntfs_attr_put_search_ctx(ctx); -unm_err_out: - unmap_mft_record(mft_ni); - goto err_out; - } - a = ctx->attr; - write_lock_irqsave(&mftbmp_ni->size_lock, flags); - mftbmp_ni->initialized_size = old_initialized_size; - a->data.non_resident.initialized_size = - cpu_to_sle64(old_initialized_size); - if (i_size_read(mftbmp_vi) != old_data_size) { - i_size_write(mftbmp_vi, old_data_size); - a->data.non_resident.data_size = cpu_to_sle64(old_data_size); - } - write_unlock_irqrestore(&mftbmp_ni->size_lock, flags); - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(mft_ni); -#ifdef DEBUG - read_lock_irqsave(&mftbmp_ni->size_lock, flags); - ntfs_debug("Restored status of mftbmp: allocated_size 0x%llx, " - "data_size 0x%llx, initialized_size 0x%llx.", - (long long)mftbmp_ni->allocated_size, - (long long)i_size_read(mftbmp_vi), - (long long)mftbmp_ni->initialized_size); - read_unlock_irqrestore(&mftbmp_ni->size_lock, flags); -#endif /* DEBUG */ -err_out: - return ret; -} - -/** - * ntfs_mft_data_extend_allocation_nolock - extend mft data attribute - * @vol: volume on which to extend the mft data attribute - * - * Extend the mft data attribute on the ntfs volume @vol by 16 mft records - * worth of clusters or if not enough space for this by one mft record worth - * of clusters. - * - * Note: Only changes allocated_size, i.e. does not touch initialized_size or - * data_size. - * - * Return 0 on success and -errno on error. - * - * Locking: - Caller must hold vol->mftbmp_lock for writing. - * - This function takes NTFS_I(vol->mft_ino)->runlist.lock for - * writing and releases it before returning. - * - This function calls functions which take vol->lcnbmp_lock for - * writing and release it before returning. - */ -static int ntfs_mft_data_extend_allocation_nolock(ntfs_volume *vol) -{ - LCN lcn; - VCN old_last_vcn; - s64 min_nr, nr, ll; - unsigned long flags; - ntfs_inode *mft_ni; - runlist_element *rl, *rl2; - ntfs_attr_search_ctx *ctx = NULL; - MFT_RECORD *mrec; - ATTR_RECORD *a = NULL; - int ret, mp_size; - u32 old_alen = 0; - bool mp_rebuilt = false; - - ntfs_debug("Extending mft data allocation."); - mft_ni = NTFS_I(vol->mft_ino); - /* - * Determine the preferred allocation location, i.e. the last lcn of - * the mft data attribute. The allocated size of the mft data - * attribute cannot be zero so we are ok to do this. - */ - down_write(&mft_ni->runlist.lock); - read_lock_irqsave(&mft_ni->size_lock, flags); - ll = mft_ni->allocated_size; - read_unlock_irqrestore(&mft_ni->size_lock, flags); - rl = ntfs_attr_find_vcn_nolock(mft_ni, - (ll - 1) >> vol->cluster_size_bits, NULL); - if (IS_ERR(rl) || unlikely(!rl->length || rl->lcn < 0)) { - up_write(&mft_ni->runlist.lock); - ntfs_error(vol->sb, "Failed to determine last allocated " - "cluster of mft data attribute."); - if (!IS_ERR(rl)) - ret = -EIO; - else - ret = PTR_ERR(rl); - return ret; - } - lcn = rl->lcn + rl->length; - ntfs_debug("Last lcn of mft data attribute is 0x%llx.", (long long)lcn); - /* Minimum allocation is one mft record worth of clusters. */ - min_nr = vol->mft_record_size >> vol->cluster_size_bits; - if (!min_nr) - min_nr = 1; - /* Want to allocate 16 mft records worth of clusters. */ - nr = vol->mft_record_size << 4 >> vol->cluster_size_bits; - if (!nr) - nr = min_nr; - /* Ensure we do not go above 2^32-1 mft records. */ - read_lock_irqsave(&mft_ni->size_lock, flags); - ll = mft_ni->allocated_size; - read_unlock_irqrestore(&mft_ni->size_lock, flags); - if (unlikely((ll + (nr << vol->cluster_size_bits)) >> - vol->mft_record_size_bits >= (1ll << 32))) { - nr = min_nr; - if (unlikely((ll + (nr << vol->cluster_size_bits)) >> - vol->mft_record_size_bits >= (1ll << 32))) { - ntfs_warning(vol->sb, "Cannot allocate mft record " - "because the maximum number of inodes " - "(2^32) has already been reached."); - up_write(&mft_ni->runlist.lock); - return -ENOSPC; - } - } - ntfs_debug("Trying mft data allocation with %s cluster count %lli.", - nr > min_nr ? "default" : "minimal", (long long)nr); - old_last_vcn = rl[1].vcn; - do { - rl2 = ntfs_cluster_alloc(vol, old_last_vcn, nr, lcn, MFT_ZONE, - true); - if (!IS_ERR(rl2)) - break; - if (PTR_ERR(rl2) != -ENOSPC || nr == min_nr) { - ntfs_error(vol->sb, "Failed to allocate the minimal " - "number of clusters (%lli) for the " - "mft data attribute.", (long long)nr); - up_write(&mft_ni->runlist.lock); - return PTR_ERR(rl2); - } - /* - * There is not enough space to do the allocation, but there - * might be enough space to do a minimal allocation so try that - * before failing. - */ - nr = min_nr; - ntfs_debug("Retrying mft data allocation with minimal cluster " - "count %lli.", (long long)nr); - } while (1); - rl = ntfs_runlists_merge(mft_ni->runlist.rl, rl2); - if (IS_ERR(rl)) { - up_write(&mft_ni->runlist.lock); - ntfs_error(vol->sb, "Failed to merge runlists for mft data " - "attribute."); - if (ntfs_cluster_free_from_rl(vol, rl2)) { - ntfs_error(vol->sb, "Failed to deallocate clusters " - "from the mft data attribute.%s", es); - NVolSetErrors(vol); - } - ntfs_free(rl2); - return PTR_ERR(rl); - } - mft_ni->runlist.rl = rl; - ntfs_debug("Allocated %lli clusters.", (long long)nr); - /* Find the last run in the new runlist. */ - for (; rl[1].length; rl++) - ; - /* Update the attribute record as well. */ - mrec = map_mft_record(mft_ni); - if (IS_ERR(mrec)) { - ntfs_error(vol->sb, "Failed to map mft record."); - ret = PTR_ERR(mrec); - goto undo_alloc; - } - ctx = ntfs_attr_get_search_ctx(mft_ni, mrec); - if (unlikely(!ctx)) { - ntfs_error(vol->sb, "Failed to get search context."); - ret = -ENOMEM; - goto undo_alloc; - } - ret = ntfs_attr_lookup(mft_ni->type, mft_ni->name, mft_ni->name_len, - CASE_SENSITIVE, rl[1].vcn, NULL, 0, ctx); - if (unlikely(ret)) { - ntfs_error(vol->sb, "Failed to find last attribute extent of " - "mft data attribute."); - if (ret == -ENOENT) - ret = -EIO; - goto undo_alloc; - } - a = ctx->attr; - ll = sle64_to_cpu(a->data.non_resident.lowest_vcn); - /* Search back for the previous last allocated cluster of mft bitmap. */ - for (rl2 = rl; rl2 > mft_ni->runlist.rl; rl2--) { - if (ll >= rl2->vcn) - break; - } - BUG_ON(ll < rl2->vcn); - BUG_ON(ll >= rl2->vcn + rl2->length); - /* Get the size for the new mapping pairs array for this extent. */ - mp_size = ntfs_get_size_for_mapping_pairs(vol, rl2, ll, -1); - if (unlikely(mp_size <= 0)) { - ntfs_error(vol->sb, "Get size for mapping pairs failed for " - "mft data attribute extent."); - ret = mp_size; - if (!ret) - ret = -EIO; - goto undo_alloc; - } - /* Expand the attribute record if necessary. */ - old_alen = le32_to_cpu(a->length); - ret = ntfs_attr_record_resize(ctx->mrec, a, mp_size + - le16_to_cpu(a->data.non_resident.mapping_pairs_offset)); - if (unlikely(ret)) { - if (ret != -ENOSPC) { - ntfs_error(vol->sb, "Failed to resize attribute " - "record for mft data attribute."); - goto undo_alloc; - } - // TODO: Deal with this by moving this extent to a new mft - // record or by starting a new extent in a new mft record or by - // moving other attributes out of this mft record. - // Note: Use the special reserved mft records and ensure that - // this extent is not required to find the mft record in - // question. If no free special records left we would need to - // move an existing record away, insert ours in its place, and - // then place the moved record into the newly allocated space - // and we would then need to update all references to this mft - // record appropriately. This is rather complicated... - ntfs_error(vol->sb, "Not enough space in this mft record to " - "accommodate extended mft data attribute " - "extent. Cannot handle this yet."); - ret = -EOPNOTSUPP; - goto undo_alloc; - } - mp_rebuilt = true; - /* Generate the mapping pairs array directly into the attr record. */ - ret = ntfs_mapping_pairs_build(vol, (u8*)a + - le16_to_cpu(a->data.non_resident.mapping_pairs_offset), - mp_size, rl2, ll, -1, NULL); - if (unlikely(ret)) { - ntfs_error(vol->sb, "Failed to build mapping pairs array of " - "mft data attribute."); - goto undo_alloc; - } - /* Update the highest_vcn. */ - a->data.non_resident.highest_vcn = cpu_to_sle64(rl[1].vcn - 1); - /* - * We now have extended the mft data allocated_size by nr clusters. - * Reflect this in the ntfs_inode structure and the attribute record. - * @rl is the last (non-terminator) runlist element of mft data - * attribute. - */ - if (a->data.non_resident.lowest_vcn) { - /* - * We are not in the first attribute extent, switch to it, but - * first ensure the changes will make it to disk later. - */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_reinit_search_ctx(ctx); - ret = ntfs_attr_lookup(mft_ni->type, mft_ni->name, - mft_ni->name_len, CASE_SENSITIVE, 0, NULL, 0, - ctx); - if (unlikely(ret)) { - ntfs_error(vol->sb, "Failed to find first attribute " - "extent of mft data attribute."); - goto restore_undo_alloc; - } - a = ctx->attr; - } - write_lock_irqsave(&mft_ni->size_lock, flags); - mft_ni->allocated_size += nr << vol->cluster_size_bits; - a->data.non_resident.allocated_size = - cpu_to_sle64(mft_ni->allocated_size); - write_unlock_irqrestore(&mft_ni->size_lock, flags); - /* Ensure the changes make it to disk. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(mft_ni); - up_write(&mft_ni->runlist.lock); - ntfs_debug("Done."); - return 0; -restore_undo_alloc: - ntfs_attr_reinit_search_ctx(ctx); - if (ntfs_attr_lookup(mft_ni->type, mft_ni->name, mft_ni->name_len, - CASE_SENSITIVE, rl[1].vcn, NULL, 0, ctx)) { - ntfs_error(vol->sb, "Failed to find last attribute extent of " - "mft data attribute.%s", es); - write_lock_irqsave(&mft_ni->size_lock, flags); - mft_ni->allocated_size += nr << vol->cluster_size_bits; - write_unlock_irqrestore(&mft_ni->size_lock, flags); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(mft_ni); - up_write(&mft_ni->runlist.lock); - /* - * The only thing that is now wrong is ->allocated_size of the - * base attribute extent which chkdsk should be able to fix. - */ - NVolSetErrors(vol); - return ret; - } - ctx->attr->data.non_resident.highest_vcn = - cpu_to_sle64(old_last_vcn - 1); -undo_alloc: - if (ntfs_cluster_free(mft_ni, old_last_vcn, -1, ctx) < 0) { - ntfs_error(vol->sb, "Failed to free clusters from mft data " - "attribute.%s", es); - NVolSetErrors(vol); - } - - if (ntfs_rl_truncate_nolock(vol, &mft_ni->runlist, old_last_vcn)) { - ntfs_error(vol->sb, "Failed to truncate mft data attribute " - "runlist.%s", es); - NVolSetErrors(vol); - } - if (ctx) { - a = ctx->attr; - if (mp_rebuilt && !IS_ERR(ctx->mrec)) { - if (ntfs_mapping_pairs_build(vol, (u8 *)a + le16_to_cpu( - a->data.non_resident.mapping_pairs_offset), - old_alen - le16_to_cpu( - a->data.non_resident.mapping_pairs_offset), - rl2, ll, -1, NULL)) { - ntfs_error(vol->sb, "Failed to restore mapping pairs " - "array.%s", es); - NVolSetErrors(vol); - } - if (ntfs_attr_record_resize(ctx->mrec, a, old_alen)) { - ntfs_error(vol->sb, "Failed to restore attribute " - "record.%s", es); - NVolSetErrors(vol); - } - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - } else if (IS_ERR(ctx->mrec)) { - ntfs_error(vol->sb, "Failed to restore attribute search " - "context.%s", es); - NVolSetErrors(vol); - } - ntfs_attr_put_search_ctx(ctx); - } - if (!IS_ERR(mrec)) - unmap_mft_record(mft_ni); - up_write(&mft_ni->runlist.lock); - return ret; -} - -/** - * ntfs_mft_record_layout - layout an mft record into a memory buffer - * @vol: volume to which the mft record will belong - * @mft_no: mft reference specifying the mft record number - * @m: destination buffer of size >= @vol->mft_record_size bytes - * - * Layout an empty, unused mft record with the mft record number @mft_no into - * the buffer @m. The volume @vol is needed because the mft record structure - * was modified in NTFS 3.1 so we need to know which volume version this mft - * record will be used on. - * - * Return 0 on success and -errno on error. - */ -static int ntfs_mft_record_layout(const ntfs_volume *vol, const s64 mft_no, - MFT_RECORD *m) -{ - ATTR_RECORD *a; - - ntfs_debug("Entering for mft record 0x%llx.", (long long)mft_no); - if (mft_no >= (1ll << 32)) { - ntfs_error(vol->sb, "Mft record number 0x%llx exceeds " - "maximum of 2^32.", (long long)mft_no); - return -ERANGE; - } - /* Start by clearing the whole mft record to gives us a clean slate. */ - memset(m, 0, vol->mft_record_size); - /* Aligned to 2-byte boundary. */ - if (vol->major_ver < 3 || (vol->major_ver == 3 && !vol->minor_ver)) - m->usa_ofs = cpu_to_le16((sizeof(MFT_RECORD_OLD) + 1) & ~1); - else { - m->usa_ofs = cpu_to_le16((sizeof(MFT_RECORD) + 1) & ~1); - /* - * Set the NTFS 3.1+ specific fields while we know that the - * volume version is 3.1+. - */ - m->reserved = 0; - m->mft_record_number = cpu_to_le32((u32)mft_no); - } - m->magic = magic_FILE; - if (vol->mft_record_size >= NTFS_BLOCK_SIZE) - m->usa_count = cpu_to_le16(vol->mft_record_size / - NTFS_BLOCK_SIZE + 1); - else { - m->usa_count = cpu_to_le16(1); - ntfs_warning(vol->sb, "Sector size is bigger than mft record " - "size. Setting usa_count to 1. If chkdsk " - "reports this as corruption, please email " - "linux-ntfs-dev@lists.sourceforge.net stating " - "that you saw this message and that the " - "modified filesystem created was corrupt. " - "Thank you."); - } - /* Set the update sequence number to 1. */ - *(le16*)((u8*)m + le16_to_cpu(m->usa_ofs)) = cpu_to_le16(1); - m->lsn = 0; - m->sequence_number = cpu_to_le16(1); - m->link_count = 0; - /* - * Place the attributes straight after the update sequence array, - * aligned to 8-byte boundary. - */ - m->attrs_offset = cpu_to_le16((le16_to_cpu(m->usa_ofs) + - (le16_to_cpu(m->usa_count) << 1) + 7) & ~7); - m->flags = 0; - /* - * Using attrs_offset plus eight bytes (for the termination attribute). - * attrs_offset is already aligned to 8-byte boundary, so no need to - * align again. - */ - m->bytes_in_use = cpu_to_le32(le16_to_cpu(m->attrs_offset) + 8); - m->bytes_allocated = cpu_to_le32(vol->mft_record_size); - m->base_mft_record = 0; - m->next_attr_instance = 0; - /* Add the termination attribute. */ - a = (ATTR_RECORD*)((u8*)m + le16_to_cpu(m->attrs_offset)); - a->type = AT_END; - a->length = 0; - ntfs_debug("Done."); - return 0; -} - -/** - * ntfs_mft_record_format - format an mft record on an ntfs volume - * @vol: volume on which to format the mft record - * @mft_no: mft record number to format - * - * Format the mft record @mft_no in $MFT/$DATA, i.e. lay out an empty, unused - * mft record into the appropriate place of the mft data attribute. This is - * used when extending the mft data attribute. - * - * Return 0 on success and -errno on error. - */ -static int ntfs_mft_record_format(const ntfs_volume *vol, const s64 mft_no) -{ - loff_t i_size; - struct inode *mft_vi = vol->mft_ino; - struct page *page; - MFT_RECORD *m; - pgoff_t index, end_index; - unsigned int ofs; - int err; - - ntfs_debug("Entering for mft record 0x%llx.", (long long)mft_no); - /* - * The index into the page cache and the offset within the page cache - * page of the wanted mft record. - */ - index = mft_no << vol->mft_record_size_bits >> PAGE_SHIFT; - ofs = (mft_no << vol->mft_record_size_bits) & ~PAGE_MASK; - /* The maximum valid index into the page cache for $MFT's data. */ - i_size = i_size_read(mft_vi); - end_index = i_size >> PAGE_SHIFT; - if (unlikely(index >= end_index)) { - if (unlikely(index > end_index || ofs + vol->mft_record_size >= - (i_size & ~PAGE_MASK))) { - ntfs_error(vol->sb, "Tried to format non-existing mft " - "record 0x%llx.", (long long)mft_no); - return -ENOENT; - } - } - /* Read, map, and pin the page containing the mft record. */ - page = ntfs_map_page(mft_vi->i_mapping, index); - if (IS_ERR(page)) { - ntfs_error(vol->sb, "Failed to map page containing mft record " - "to format 0x%llx.", (long long)mft_no); - return PTR_ERR(page); - } - lock_page(page); - BUG_ON(!PageUptodate(page)); - ClearPageUptodate(page); - m = (MFT_RECORD*)((u8*)page_address(page) + ofs); - err = ntfs_mft_record_layout(vol, mft_no, m); - if (unlikely(err)) { - ntfs_error(vol->sb, "Failed to layout mft record 0x%llx.", - (long long)mft_no); - SetPageUptodate(page); - unlock_page(page); - ntfs_unmap_page(page); - return err; - } - flush_dcache_page(page); - SetPageUptodate(page); - unlock_page(page); - /* - * Make sure the mft record is written out to disk. We could use - * ilookup5() to check if an inode is in icache and so on but this is - * unnecessary as ntfs_writepage() will write the dirty record anyway. - */ - mark_ntfs_record_dirty(page, ofs); - ntfs_unmap_page(page); - ntfs_debug("Done."); - return 0; -} - -/** - * ntfs_mft_record_alloc - allocate an mft record on an ntfs volume - * @vol: [IN] volume on which to allocate the mft record - * @mode: [IN] mode if want a file or directory, i.e. base inode or 0 - * @base_ni: [IN] open base inode if allocating an extent mft record or NULL - * @mrec: [OUT] on successful return this is the mapped mft record - * - * Allocate an mft record in $MFT/$DATA of an open ntfs volume @vol. - * - * If @base_ni is NULL make the mft record a base mft record, i.e. a file or - * direvctory inode, and allocate it at the default allocator position. In - * this case @mode is the file mode as given to us by the caller. We in - * particular use @mode to distinguish whether a file or a directory is being - * created (S_IFDIR(mode) and S_IFREG(mode), respectively). - * - * If @base_ni is not NULL make the allocated mft record an extent record, - * allocate it starting at the mft record after the base mft record and attach - * the allocated and opened ntfs inode to the base inode @base_ni. In this - * case @mode must be 0 as it is meaningless for extent inodes. - * - * You need to check the return value with IS_ERR(). If false, the function - * was successful and the return value is the now opened ntfs inode of the - * allocated mft record. *@mrec is then set to the allocated, mapped, pinned, - * and locked mft record. If IS_ERR() is true, the function failed and the - * error code is obtained from PTR_ERR(return value). *@mrec is undefined in - * this case. - * - * Allocation strategy: - * - * To find a free mft record, we scan the mft bitmap for a zero bit. To - * optimize this we start scanning at the place specified by @base_ni or if - * @base_ni is NULL we start where we last stopped and we perform wrap around - * when we reach the end. Note, we do not try to allocate mft records below - * number 24 because numbers 0 to 15 are the defined system files anyway and 16 - * to 24 are special in that they are used for storing extension mft records - * for the $DATA attribute of $MFT. This is required to avoid the possibility - * of creating a runlist with a circular dependency which once written to disk - * can never be read in again. Windows will only use records 16 to 24 for - * normal files if the volume is completely out of space. We never use them - * which means that when the volume is really out of space we cannot create any - * more files while Windows can still create up to 8 small files. We can start - * doing this at some later time, it does not matter much for now. - * - * When scanning the mft bitmap, we only search up to the last allocated mft - * record. If there are no free records left in the range 24 to number of - * allocated mft records, then we extend the $MFT/$DATA attribute in order to - * create free mft records. We extend the allocated size of $MFT/$DATA by 16 - * records at a time or one cluster, if cluster size is above 16kiB. If there - * is not sufficient space to do this, we try to extend by a single mft record - * or one cluster, if cluster size is above the mft record size. - * - * No matter how many mft records we allocate, we initialize only the first - * allocated mft record, incrementing mft data size and initialized size - * accordingly, open an ntfs_inode for it and return it to the caller, unless - * there are less than 24 mft records, in which case we allocate and initialize - * mft records until we reach record 24 which we consider as the first free mft - * record for use by normal files. - * - * If during any stage we overflow the initialized data in the mft bitmap, we - * extend the initialized size (and data size) by 8 bytes, allocating another - * cluster if required. The bitmap data size has to be at least equal to the - * number of mft records in the mft, but it can be bigger, in which case the - * superflous bits are padded with zeroes. - * - * Thus, when we return successfully (IS_ERR() is false), we will have: - * - initialized / extended the mft bitmap if necessary, - * - initialized / extended the mft data if necessary, - * - set the bit corresponding to the mft record being allocated in the - * mft bitmap, - * - opened an ntfs_inode for the allocated mft record, and we will have - * - returned the ntfs_inode as well as the allocated mapped, pinned, and - * locked mft record. - * - * On error, the volume will be left in a consistent state and no record will - * be allocated. If rolling back a partial operation fails, we may leave some - * inconsistent metadata in which case we set NVolErrors() so the volume is - * left dirty when unmounted. - * - * Note, this function cannot make use of most of the normal functions, like - * for example for attribute resizing, etc, because when the run list overflows - * the base mft record and an attribute list is used, it is very important that - * the extension mft records used to store the $DATA attribute of $MFT can be - * reached without having to read the information contained inside them, as - * this would make it impossible to find them in the first place after the - * volume is unmounted. $MFT/$BITMAP probably does not need to follow this - * rule because the bitmap is not essential for finding the mft records, but on - * the other hand, handling the bitmap in this special way would make life - * easier because otherwise there might be circular invocations of functions - * when reading the bitmap. - */ -ntfs_inode *ntfs_mft_record_alloc(ntfs_volume *vol, const int mode, - ntfs_inode *base_ni, MFT_RECORD **mrec) -{ - s64 ll, bit, old_data_initialized, old_data_size; - unsigned long flags; - struct inode *vi; - struct page *page; - ntfs_inode *mft_ni, *mftbmp_ni, *ni; - ntfs_attr_search_ctx *ctx; - MFT_RECORD *m; - ATTR_RECORD *a; - pgoff_t index; - unsigned int ofs; - int err; - le16 seq_no, usn; - bool record_formatted = false; - - if (base_ni) { - ntfs_debug("Entering (allocating an extent mft record for " - "base mft record 0x%llx).", - (long long)base_ni->mft_no); - /* @mode and @base_ni are mutually exclusive. */ - BUG_ON(mode); - } else - ntfs_debug("Entering (allocating a base mft record)."); - if (mode) { - /* @mode and @base_ni are mutually exclusive. */ - BUG_ON(base_ni); - /* We only support creation of normal files and directories. */ - if (!S_ISREG(mode) && !S_ISDIR(mode)) - return ERR_PTR(-EOPNOTSUPP); - } - BUG_ON(!mrec); - mft_ni = NTFS_I(vol->mft_ino); - mftbmp_ni = NTFS_I(vol->mftbmp_ino); - down_write(&vol->mftbmp_lock); - bit = ntfs_mft_bitmap_find_and_alloc_free_rec_nolock(vol, base_ni); - if (bit >= 0) { - ntfs_debug("Found and allocated free record (#1), bit 0x%llx.", - (long long)bit); - goto have_alloc_rec; - } - if (bit != -ENOSPC) { - up_write(&vol->mftbmp_lock); - return ERR_PTR(bit); - } - /* - * No free mft records left. If the mft bitmap already covers more - * than the currently used mft records, the next records are all free, - * so we can simply allocate the first unused mft record. - * Note: We also have to make sure that the mft bitmap at least covers - * the first 24 mft records as they are special and whilst they may not - * be in use, we do not allocate from them. - */ - read_lock_irqsave(&mft_ni->size_lock, flags); - ll = mft_ni->initialized_size >> vol->mft_record_size_bits; - read_unlock_irqrestore(&mft_ni->size_lock, flags); - read_lock_irqsave(&mftbmp_ni->size_lock, flags); - old_data_initialized = mftbmp_ni->initialized_size; - read_unlock_irqrestore(&mftbmp_ni->size_lock, flags); - if (old_data_initialized << 3 > ll && old_data_initialized > 3) { - bit = ll; - if (bit < 24) - bit = 24; - if (unlikely(bit >= (1ll << 32))) - goto max_err_out; - ntfs_debug("Found free record (#2), bit 0x%llx.", - (long long)bit); - goto found_free_rec; - } - /* - * The mft bitmap needs to be expanded until it covers the first unused - * mft record that we can allocate. - * Note: The smallest mft record we allocate is mft record 24. - */ - bit = old_data_initialized << 3; - if (unlikely(bit >= (1ll << 32))) - goto max_err_out; - read_lock_irqsave(&mftbmp_ni->size_lock, flags); - old_data_size = mftbmp_ni->allocated_size; - ntfs_debug("Status of mftbmp before extension: allocated_size 0x%llx, " - "data_size 0x%llx, initialized_size 0x%llx.", - (long long)old_data_size, - (long long)i_size_read(vol->mftbmp_ino), - (long long)old_data_initialized); - read_unlock_irqrestore(&mftbmp_ni->size_lock, flags); - if (old_data_initialized + 8 > old_data_size) { - /* Need to extend bitmap by one more cluster. */ - ntfs_debug("mftbmp: initialized_size + 8 > allocated_size."); - err = ntfs_mft_bitmap_extend_allocation_nolock(vol); - if (unlikely(err)) { - up_write(&vol->mftbmp_lock); - goto err_out; - } -#ifdef DEBUG - read_lock_irqsave(&mftbmp_ni->size_lock, flags); - ntfs_debug("Status of mftbmp after allocation extension: " - "allocated_size 0x%llx, data_size 0x%llx, " - "initialized_size 0x%llx.", - (long long)mftbmp_ni->allocated_size, - (long long)i_size_read(vol->mftbmp_ino), - (long long)mftbmp_ni->initialized_size); - read_unlock_irqrestore(&mftbmp_ni->size_lock, flags); -#endif /* DEBUG */ - } - /* - * We now have sufficient allocated space, extend the initialized_size - * as well as the data_size if necessary and fill the new space with - * zeroes. - */ - err = ntfs_mft_bitmap_extend_initialized_nolock(vol); - if (unlikely(err)) { - up_write(&vol->mftbmp_lock); - goto err_out; - } -#ifdef DEBUG - read_lock_irqsave(&mftbmp_ni->size_lock, flags); - ntfs_debug("Status of mftbmp after initialized extension: " - "allocated_size 0x%llx, data_size 0x%llx, " - "initialized_size 0x%llx.", - (long long)mftbmp_ni->allocated_size, - (long long)i_size_read(vol->mftbmp_ino), - (long long)mftbmp_ni->initialized_size); - read_unlock_irqrestore(&mftbmp_ni->size_lock, flags); -#endif /* DEBUG */ - ntfs_debug("Found free record (#3), bit 0x%llx.", (long long)bit); -found_free_rec: - /* @bit is the found free mft record, allocate it in the mft bitmap. */ - ntfs_debug("At found_free_rec."); - err = ntfs_bitmap_set_bit(vol->mftbmp_ino, bit); - if (unlikely(err)) { - ntfs_error(vol->sb, "Failed to allocate bit in mft bitmap."); - up_write(&vol->mftbmp_lock); - goto err_out; - } - ntfs_debug("Set bit 0x%llx in mft bitmap.", (long long)bit); -have_alloc_rec: - /* - * The mft bitmap is now uptodate. Deal with mft data attribute now. - * Note, we keep hold of the mft bitmap lock for writing until all - * modifications to the mft data attribute are complete, too, as they - * will impact decisions for mft bitmap and mft record allocation done - * by a parallel allocation and if the lock is not maintained a - * parallel allocation could allocate the same mft record as this one. - */ - ll = (bit + 1) << vol->mft_record_size_bits; - read_lock_irqsave(&mft_ni->size_lock, flags); - old_data_initialized = mft_ni->initialized_size; - read_unlock_irqrestore(&mft_ni->size_lock, flags); - if (ll <= old_data_initialized) { - ntfs_debug("Allocated mft record already initialized."); - goto mft_rec_already_initialized; - } - ntfs_debug("Initializing allocated mft record."); - /* - * The mft record is outside the initialized data. Extend the mft data - * attribute until it covers the allocated record. The loop is only - * actually traversed more than once when a freshly formatted volume is - * first written to so it optimizes away nicely in the common case. - */ - read_lock_irqsave(&mft_ni->size_lock, flags); - ntfs_debug("Status of mft data before extension: " - "allocated_size 0x%llx, data_size 0x%llx, " - "initialized_size 0x%llx.", - (long long)mft_ni->allocated_size, - (long long)i_size_read(vol->mft_ino), - (long long)mft_ni->initialized_size); - while (ll > mft_ni->allocated_size) { - read_unlock_irqrestore(&mft_ni->size_lock, flags); - err = ntfs_mft_data_extend_allocation_nolock(vol); - if (unlikely(err)) { - ntfs_error(vol->sb, "Failed to extend mft data " - "allocation."); - goto undo_mftbmp_alloc_nolock; - } - read_lock_irqsave(&mft_ni->size_lock, flags); - ntfs_debug("Status of mft data after allocation extension: " - "allocated_size 0x%llx, data_size 0x%llx, " - "initialized_size 0x%llx.", - (long long)mft_ni->allocated_size, - (long long)i_size_read(vol->mft_ino), - (long long)mft_ni->initialized_size); - } - read_unlock_irqrestore(&mft_ni->size_lock, flags); - /* - * Extend mft data initialized size (and data size of course) to reach - * the allocated mft record, formatting the mft records allong the way. - * Note: We only modify the ntfs_inode structure as that is all that is - * needed by ntfs_mft_record_format(). We will update the attribute - * record itself in one fell swoop later on. - */ - write_lock_irqsave(&mft_ni->size_lock, flags); - old_data_initialized = mft_ni->initialized_size; - old_data_size = vol->mft_ino->i_size; - while (ll > mft_ni->initialized_size) { - s64 new_initialized_size, mft_no; - - new_initialized_size = mft_ni->initialized_size + - vol->mft_record_size; - mft_no = mft_ni->initialized_size >> vol->mft_record_size_bits; - if (new_initialized_size > i_size_read(vol->mft_ino)) - i_size_write(vol->mft_ino, new_initialized_size); - write_unlock_irqrestore(&mft_ni->size_lock, flags); - ntfs_debug("Initializing mft record 0x%llx.", - (long long)mft_no); - err = ntfs_mft_record_format(vol, mft_no); - if (unlikely(err)) { - ntfs_error(vol->sb, "Failed to format mft record."); - goto undo_data_init; - } - write_lock_irqsave(&mft_ni->size_lock, flags); - mft_ni->initialized_size = new_initialized_size; - } - write_unlock_irqrestore(&mft_ni->size_lock, flags); - record_formatted = true; - /* Update the mft data attribute record to reflect the new sizes. */ - m = map_mft_record(mft_ni); - if (IS_ERR(m)) { - ntfs_error(vol->sb, "Failed to map mft record."); - err = PTR_ERR(m); - goto undo_data_init; - } - ctx = ntfs_attr_get_search_ctx(mft_ni, m); - if (unlikely(!ctx)) { - ntfs_error(vol->sb, "Failed to get search context."); - err = -ENOMEM; - unmap_mft_record(mft_ni); - goto undo_data_init; - } - err = ntfs_attr_lookup(mft_ni->type, mft_ni->name, mft_ni->name_len, - CASE_SENSITIVE, 0, NULL, 0, ctx); - if (unlikely(err)) { - ntfs_error(vol->sb, "Failed to find first attribute extent of " - "mft data attribute."); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(mft_ni); - goto undo_data_init; - } - a = ctx->attr; - read_lock_irqsave(&mft_ni->size_lock, flags); - a->data.non_resident.initialized_size = - cpu_to_sle64(mft_ni->initialized_size); - a->data.non_resident.data_size = - cpu_to_sle64(i_size_read(vol->mft_ino)); - read_unlock_irqrestore(&mft_ni->size_lock, flags); - /* Ensure the changes make it to disk. */ - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(mft_ni); - read_lock_irqsave(&mft_ni->size_lock, flags); - ntfs_debug("Status of mft data after mft record initialization: " - "allocated_size 0x%llx, data_size 0x%llx, " - "initialized_size 0x%llx.", - (long long)mft_ni->allocated_size, - (long long)i_size_read(vol->mft_ino), - (long long)mft_ni->initialized_size); - BUG_ON(i_size_read(vol->mft_ino) > mft_ni->allocated_size); - BUG_ON(mft_ni->initialized_size > i_size_read(vol->mft_ino)); - read_unlock_irqrestore(&mft_ni->size_lock, flags); -mft_rec_already_initialized: - /* - * We can finally drop the mft bitmap lock as the mft data attribute - * has been fully updated. The only disparity left is that the - * allocated mft record still needs to be marked as in use to match the - * set bit in the mft bitmap but this is actually not a problem since - * this mft record is not referenced from anywhere yet and the fact - * that it is allocated in the mft bitmap means that no-one will try to - * allocate it either. - */ - up_write(&vol->mftbmp_lock); - /* - * We now have allocated and initialized the mft record. Calculate the - * index of and the offset within the page cache page the record is in. - */ - index = bit << vol->mft_record_size_bits >> PAGE_SHIFT; - ofs = (bit << vol->mft_record_size_bits) & ~PAGE_MASK; - /* Read, map, and pin the page containing the mft record. */ - page = ntfs_map_page(vol->mft_ino->i_mapping, index); - if (IS_ERR(page)) { - ntfs_error(vol->sb, "Failed to map page containing allocated " - "mft record 0x%llx.", (long long)bit); - err = PTR_ERR(page); - goto undo_mftbmp_alloc; - } - lock_page(page); - BUG_ON(!PageUptodate(page)); - ClearPageUptodate(page); - m = (MFT_RECORD*)((u8*)page_address(page) + ofs); - /* If we just formatted the mft record no need to do it again. */ - if (!record_formatted) { - /* Sanity check that the mft record is really not in use. */ - if (ntfs_is_file_record(m->magic) && - (m->flags & MFT_RECORD_IN_USE)) { - ntfs_error(vol->sb, "Mft record 0x%llx was marked " - "free in mft bitmap but is marked " - "used itself. Corrupt filesystem. " - "Unmount and run chkdsk.", - (long long)bit); - err = -EIO; - SetPageUptodate(page); - unlock_page(page); - ntfs_unmap_page(page); - NVolSetErrors(vol); - goto undo_mftbmp_alloc; - } - /* - * We need to (re-)format the mft record, preserving the - * sequence number if it is not zero as well as the update - * sequence number if it is not zero or -1 (0xffff). This - * means we do not need to care whether or not something went - * wrong with the previous mft record. - */ - seq_no = m->sequence_number; - usn = *(le16*)((u8*)m + le16_to_cpu(m->usa_ofs)); - err = ntfs_mft_record_layout(vol, bit, m); - if (unlikely(err)) { - ntfs_error(vol->sb, "Failed to layout allocated mft " - "record 0x%llx.", (long long)bit); - SetPageUptodate(page); - unlock_page(page); - ntfs_unmap_page(page); - goto undo_mftbmp_alloc; - } - if (seq_no) - m->sequence_number = seq_no; - if (usn && le16_to_cpu(usn) != 0xffff) - *(le16*)((u8*)m + le16_to_cpu(m->usa_ofs)) = usn; - } - /* Set the mft record itself in use. */ - m->flags |= MFT_RECORD_IN_USE; - if (S_ISDIR(mode)) - m->flags |= MFT_RECORD_IS_DIRECTORY; - flush_dcache_page(page); - SetPageUptodate(page); - if (base_ni) { - MFT_RECORD *m_tmp; - - /* - * Setup the base mft record in the extent mft record. This - * completes initialization of the allocated extent mft record - * and we can simply use it with map_extent_mft_record(). - */ - m->base_mft_record = MK_LE_MREF(base_ni->mft_no, - base_ni->seq_no); - /* - * Allocate an extent inode structure for the new mft record, - * attach it to the base inode @base_ni and map, pin, and lock - * its, i.e. the allocated, mft record. - */ - m_tmp = map_extent_mft_record(base_ni, bit, &ni); - if (IS_ERR(m_tmp)) { - ntfs_error(vol->sb, "Failed to map allocated extent " - "mft record 0x%llx.", (long long)bit); - err = PTR_ERR(m_tmp); - /* Set the mft record itself not in use. */ - m->flags &= cpu_to_le16( - ~le16_to_cpu(MFT_RECORD_IN_USE)); - flush_dcache_page(page); - /* Make sure the mft record is written out to disk. */ - mark_ntfs_record_dirty(page, ofs); - unlock_page(page); - ntfs_unmap_page(page); - goto undo_mftbmp_alloc; - } - BUG_ON(m != m_tmp); - /* - * Make sure the allocated mft record is written out to disk. - * No need to set the inode dirty because the caller is going - * to do that anyway after finishing with the new extent mft - * record (e.g. at a minimum a new attribute will be added to - * the mft record. - */ - mark_ntfs_record_dirty(page, ofs); - unlock_page(page); - /* - * Need to unmap the page since map_extent_mft_record() mapped - * it as well so we have it mapped twice at the moment. - */ - ntfs_unmap_page(page); - } else { - /* - * Allocate a new VFS inode and set it up. NOTE: @vi->i_nlink - * is set to 1 but the mft record->link_count is 0. The caller - * needs to bear this in mind. - */ - vi = new_inode(vol->sb); - if (unlikely(!vi)) { - err = -ENOMEM; - /* Set the mft record itself not in use. */ - m->flags &= cpu_to_le16( - ~le16_to_cpu(MFT_RECORD_IN_USE)); - flush_dcache_page(page); - /* Make sure the mft record is written out to disk. */ - mark_ntfs_record_dirty(page, ofs); - unlock_page(page); - ntfs_unmap_page(page); - goto undo_mftbmp_alloc; - } - vi->i_ino = bit; - - /* The owner and group come from the ntfs volume. */ - vi->i_uid = vol->uid; - vi->i_gid = vol->gid; - - /* Initialize the ntfs specific part of @vi. */ - ntfs_init_big_inode(vi); - ni = NTFS_I(vi); - /* - * Set the appropriate mode, attribute type, and name. For - * directories, also setup the index values to the defaults. - */ - if (S_ISDIR(mode)) { - vi->i_mode = S_IFDIR | S_IRWXUGO; - vi->i_mode &= ~vol->dmask; - - NInoSetMstProtected(ni); - ni->type = AT_INDEX_ALLOCATION; - ni->name = I30; - ni->name_len = 4; - - ni->itype.index.block_size = 4096; - ni->itype.index.block_size_bits = ntfs_ffs(4096) - 1; - ni->itype.index.collation_rule = COLLATION_FILE_NAME; - if (vol->cluster_size <= ni->itype.index.block_size) { - ni->itype.index.vcn_size = vol->cluster_size; - ni->itype.index.vcn_size_bits = - vol->cluster_size_bits; - } else { - ni->itype.index.vcn_size = vol->sector_size; - ni->itype.index.vcn_size_bits = - vol->sector_size_bits; - } - } else { - vi->i_mode = S_IFREG | S_IRWXUGO; - vi->i_mode &= ~vol->fmask; - - ni->type = AT_DATA; - ni->name = NULL; - ni->name_len = 0; - } - if (IS_RDONLY(vi)) - vi->i_mode &= ~S_IWUGO; - - /* Set the inode times to the current time. */ - vi->i_atime = vi->i_mtime = inode_set_ctime_current(vi); - /* - * Set the file size to 0, the ntfs inode sizes are set to 0 by - * the call to ntfs_init_big_inode() below. - */ - vi->i_size = 0; - vi->i_blocks = 0; - - /* Set the sequence number. */ - vi->i_generation = ni->seq_no = le16_to_cpu(m->sequence_number); - /* - * Manually map, pin, and lock the mft record as we already - * have its page mapped and it is very easy to do. - */ - atomic_inc(&ni->count); - mutex_lock(&ni->mrec_lock); - ni->page = page; - ni->page_ofs = ofs; - /* - * Make sure the allocated mft record is written out to disk. - * NOTE: We do not set the ntfs inode dirty because this would - * fail in ntfs_write_inode() because the inode does not have a - * standard information attribute yet. Also, there is no need - * to set the inode dirty because the caller is going to do - * that anyway after finishing with the new mft record (e.g. at - * a minimum some new attributes will be added to the mft - * record. - */ - mark_ntfs_record_dirty(page, ofs); - unlock_page(page); - - /* Add the inode to the inode hash for the superblock. */ - insert_inode_hash(vi); - - /* Update the default mft allocation position. */ - vol->mft_data_pos = bit + 1; - } - /* - * Return the opened, allocated inode of the allocated mft record as - * well as the mapped, pinned, and locked mft record. - */ - ntfs_debug("Returning opened, allocated %sinode 0x%llx.", - base_ni ? "extent " : "", (long long)bit); - *mrec = m; - return ni; -undo_data_init: - write_lock_irqsave(&mft_ni->size_lock, flags); - mft_ni->initialized_size = old_data_initialized; - i_size_write(vol->mft_ino, old_data_size); - write_unlock_irqrestore(&mft_ni->size_lock, flags); - goto undo_mftbmp_alloc_nolock; -undo_mftbmp_alloc: - down_write(&vol->mftbmp_lock); -undo_mftbmp_alloc_nolock: - if (ntfs_bitmap_clear_bit(vol->mftbmp_ino, bit)) { - ntfs_error(vol->sb, "Failed to clear bit in mft bitmap.%s", es); - NVolSetErrors(vol); - } - up_write(&vol->mftbmp_lock); -err_out: - return ERR_PTR(err); -max_err_out: - ntfs_warning(vol->sb, "Cannot allocate mft record because the maximum " - "number of inodes (2^32) has already been reached."); - up_write(&vol->mftbmp_lock); - return ERR_PTR(-ENOSPC); -} - -/** - * ntfs_extent_mft_record_free - free an extent mft record on an ntfs volume - * @ni: ntfs inode of the mapped extent mft record to free - * @m: mapped extent mft record of the ntfs inode @ni - * - * Free the mapped extent mft record @m of the extent ntfs inode @ni. - * - * Note that this function unmaps the mft record and closes and destroys @ni - * internally and hence you cannot use either @ni nor @m any more after this - * function returns success. - * - * On success return 0 and on error return -errno. @ni and @m are still valid - * in this case and have not been freed. - * - * For some errors an error message is displayed and the success code 0 is - * returned and the volume is then left dirty on umount. This makes sense in - * case we could not rollback the changes that were already done since the - * caller no longer wants to reference this mft record so it does not matter to - * the caller if something is wrong with it as long as it is properly detached - * from the base inode. - */ -int ntfs_extent_mft_record_free(ntfs_inode *ni, MFT_RECORD *m) -{ - unsigned long mft_no = ni->mft_no; - ntfs_volume *vol = ni->vol; - ntfs_inode *base_ni; - ntfs_inode **extent_nis; - int i, err; - le16 old_seq_no; - u16 seq_no; - - BUG_ON(NInoAttr(ni)); - BUG_ON(ni->nr_extents != -1); - - mutex_lock(&ni->extent_lock); - base_ni = ni->ext.base_ntfs_ino; - mutex_unlock(&ni->extent_lock); - - BUG_ON(base_ni->nr_extents <= 0); - - ntfs_debug("Entering for extent inode 0x%lx, base inode 0x%lx.\n", - mft_no, base_ni->mft_no); - - mutex_lock(&base_ni->extent_lock); - - /* Make sure we are holding the only reference to the extent inode. */ - if (atomic_read(&ni->count) > 2) { - ntfs_error(vol->sb, "Tried to free busy extent inode 0x%lx, " - "not freeing.", base_ni->mft_no); - mutex_unlock(&base_ni->extent_lock); - return -EBUSY; - } - - /* Dissociate the ntfs inode from the base inode. */ - extent_nis = base_ni->ext.extent_ntfs_inos; - err = -ENOENT; - for (i = 0; i < base_ni->nr_extents; i++) { - if (ni != extent_nis[i]) - continue; - extent_nis += i; - base_ni->nr_extents--; - memmove(extent_nis, extent_nis + 1, (base_ni->nr_extents - i) * - sizeof(ntfs_inode*)); - err = 0; - break; - } - - mutex_unlock(&base_ni->extent_lock); - - if (unlikely(err)) { - ntfs_error(vol->sb, "Extent inode 0x%lx is not attached to " - "its base inode 0x%lx.", mft_no, - base_ni->mft_no); - BUG(); - } - - /* - * The extent inode is no longer attached to the base inode so no one - * can get a reference to it any more. - */ - - /* Mark the mft record as not in use. */ - m->flags &= ~MFT_RECORD_IN_USE; - - /* Increment the sequence number, skipping zero, if it is not zero. */ - old_seq_no = m->sequence_number; - seq_no = le16_to_cpu(old_seq_no); - if (seq_no == 0xffff) - seq_no = 1; - else if (seq_no) - seq_no++; - m->sequence_number = cpu_to_le16(seq_no); - - /* - * Set the ntfs inode dirty and write it out. We do not need to worry - * about the base inode here since whatever caused the extent mft - * record to be freed is guaranteed to do it already. - */ - NInoSetDirty(ni); - err = write_mft_record(ni, m, 0); - if (unlikely(err)) { - ntfs_error(vol->sb, "Failed to write mft record 0x%lx, not " - "freeing.", mft_no); - goto rollback; - } -rollback_error: - /* Unmap and throw away the now freed extent inode. */ - unmap_extent_mft_record(ni); - ntfs_clear_extent_inode(ni); - - /* Clear the bit in the $MFT/$BITMAP corresponding to this record. */ - down_write(&vol->mftbmp_lock); - err = ntfs_bitmap_clear_bit(vol->mftbmp_ino, mft_no); - up_write(&vol->mftbmp_lock); - if (unlikely(err)) { - /* - * The extent inode is gone but we failed to deallocate it in - * the mft bitmap. Just emit a warning and leave the volume - * dirty on umount. - */ - ntfs_error(vol->sb, "Failed to clear bit in mft bitmap.%s", es); - NVolSetErrors(vol); - } - return 0; -rollback: - /* Rollback what we did... */ - mutex_lock(&base_ni->extent_lock); - extent_nis = base_ni->ext.extent_ntfs_inos; - if (!(base_ni->nr_extents & 3)) { - int new_size = (base_ni->nr_extents + 4) * sizeof(ntfs_inode*); - - extent_nis = kmalloc(new_size, GFP_NOFS); - if (unlikely(!extent_nis)) { - ntfs_error(vol->sb, "Failed to allocate internal " - "buffer during rollback.%s", es); - mutex_unlock(&base_ni->extent_lock); - NVolSetErrors(vol); - goto rollback_error; - } - if (base_ni->nr_extents) { - BUG_ON(!base_ni->ext.extent_ntfs_inos); - memcpy(extent_nis, base_ni->ext.extent_ntfs_inos, - new_size - 4 * sizeof(ntfs_inode*)); - kfree(base_ni->ext.extent_ntfs_inos); - } - base_ni->ext.extent_ntfs_inos = extent_nis; - } - m->flags |= MFT_RECORD_IN_USE; - m->sequence_number = old_seq_no; - extent_nis[base_ni->nr_extents++] = ni; - mutex_unlock(&base_ni->extent_lock); - mark_mft_record_dirty(ni); - return err; -} -#endif /* NTFS_RW */ diff --git a/fs/ntfs/mft.h b/fs/ntfs/mft.h deleted file mode 100644 index 49c001af16ed..000000000000 --- a/fs/ntfs/mft.h +++ /dev/null @@ -1,110 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * mft.h - Defines for mft record handling in NTFS Linux kernel driver. - * Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2004 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_MFT_H -#define _LINUX_NTFS_MFT_H - -#include -#include -#include - -#include "inode.h" - -extern MFT_RECORD *map_mft_record(ntfs_inode *ni); -extern void unmap_mft_record(ntfs_inode *ni); - -extern MFT_RECORD *map_extent_mft_record(ntfs_inode *base_ni, MFT_REF mref, - ntfs_inode **ntfs_ino); - -static inline void unmap_extent_mft_record(ntfs_inode *ni) -{ - unmap_mft_record(ni); - return; -} - -#ifdef NTFS_RW - -/** - * flush_dcache_mft_record_page - flush_dcache_page() for mft records - * @ni: ntfs inode structure of mft record - * - * Call flush_dcache_page() for the page in which an mft record resides. - * - * This must be called every time an mft record is modified, just after the - * modification. - */ -static inline void flush_dcache_mft_record_page(ntfs_inode *ni) -{ - flush_dcache_page(ni->page); -} - -extern void __mark_mft_record_dirty(ntfs_inode *ni); - -/** - * mark_mft_record_dirty - set the mft record and the page containing it dirty - * @ni: ntfs inode describing the mapped mft record - * - * Set the mapped (extent) mft record of the (base or extent) ntfs inode @ni, - * as well as the page containing the mft record, dirty. Also, mark the base - * vfs inode dirty. This ensures that any changes to the mft record are - * written out to disk. - * - * NOTE: Do not do anything if the mft record is already marked dirty. - */ -static inline void mark_mft_record_dirty(ntfs_inode *ni) -{ - if (!NInoTestSetDirty(ni)) - __mark_mft_record_dirty(ni); -} - -extern int ntfs_sync_mft_mirror(ntfs_volume *vol, const unsigned long mft_no, - MFT_RECORD *m, int sync); - -extern int write_mft_record_nolock(ntfs_inode *ni, MFT_RECORD *m, int sync); - -/** - * write_mft_record - write out a mapped (extent) mft record - * @ni: ntfs inode describing the mapped (extent) mft record - * @m: mapped (extent) mft record to write - * @sync: if true, wait for i/o completion - * - * This is just a wrapper for write_mft_record_nolock() (see mft.c), which - * locks the page for the duration of the write. This ensures that there are - * no race conditions between writing the mft record via the dirty inode code - * paths and via the page cache write back code paths or between writing - * neighbouring mft records residing in the same page. - * - * Locking the page also serializes us against ->read_folio() if the page is not - * uptodate. - * - * On success, clean the mft record and return 0. On error, leave the mft - * record dirty and return -errno. - */ -static inline int write_mft_record(ntfs_inode *ni, MFT_RECORD *m, int sync) -{ - struct page *page = ni->page; - int err; - - BUG_ON(!page); - lock_page(page); - err = write_mft_record_nolock(ni, m, sync); - unlock_page(page); - return err; -} - -extern bool ntfs_may_write_mft_record(ntfs_volume *vol, - const unsigned long mft_no, const MFT_RECORD *m, - ntfs_inode **locked_ni); - -extern ntfs_inode *ntfs_mft_record_alloc(ntfs_volume *vol, const int mode, - ntfs_inode *base_ni, MFT_RECORD **mrec); -extern int ntfs_extent_mft_record_free(ntfs_inode *ni, MFT_RECORD *m); - -#endif /* NTFS_RW */ - -#endif /* _LINUX_NTFS_MFT_H */ diff --git a/fs/ntfs/mst.c b/fs/ntfs/mst.c deleted file mode 100644 index 16b3c884abfc..000000000000 --- a/fs/ntfs/mst.c +++ /dev/null @@ -1,189 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * mst.c - NTFS multi sector transfer protection handling code. Part of the - * Linux-NTFS project. - * - * Copyright (c) 2001-2004 Anton Altaparmakov - */ - -#include "ntfs.h" - -/** - * post_read_mst_fixup - deprotect multi sector transfer protected data - * @b: pointer to the data to deprotect - * @size: size in bytes of @b - * - * Perform the necessary post read multi sector transfer fixup and detect the - * presence of incomplete multi sector transfers. - In that case, overwrite the - * magic of the ntfs record header being processed with "BAAD" (in memory only!) - * and abort processing. - * - * Return 0 on success and -EINVAL on error ("BAAD" magic will be present). - * - * NOTE: We consider the absence / invalidity of an update sequence array to - * mean that the structure is not protected at all and hence doesn't need to - * be fixed up. Thus, we return success and not failure in this case. This is - * in contrast to pre_write_mst_fixup(), see below. - */ -int post_read_mst_fixup(NTFS_RECORD *b, const u32 size) -{ - u16 usa_ofs, usa_count, usn; - u16 *usa_pos, *data_pos; - - /* Setup the variables. */ - usa_ofs = le16_to_cpu(b->usa_ofs); - /* Decrement usa_count to get number of fixups. */ - usa_count = le16_to_cpu(b->usa_count) - 1; - /* Size and alignment checks. */ - if ( size & (NTFS_BLOCK_SIZE - 1) || - usa_ofs & 1 || - usa_ofs + (usa_count * 2) > size || - (size >> NTFS_BLOCK_SIZE_BITS) != usa_count) - return 0; - /* Position of usn in update sequence array. */ - usa_pos = (u16*)b + usa_ofs/sizeof(u16); - /* - * The update sequence number which has to be equal to each of the - * u16 values before they are fixed up. Note no need to care for - * endianness since we are comparing and moving data for on disk - * structures which means the data is consistent. - If it is - * consistenty the wrong endianness it doesn't make any difference. - */ - usn = *usa_pos; - /* - * Position in protected data of first u16 that needs fixing up. - */ - data_pos = (u16*)b + NTFS_BLOCK_SIZE/sizeof(u16) - 1; - /* - * Check for incomplete multi sector transfer(s). - */ - while (usa_count--) { - if (*data_pos != usn) { - /* - * Incomplete multi sector transfer detected! )-: - * Set the magic to "BAAD" and return failure. - * Note that magic_BAAD is already converted to le32. - */ - b->magic = magic_BAAD; - return -EINVAL; - } - data_pos += NTFS_BLOCK_SIZE/sizeof(u16); - } - /* Re-setup the variables. */ - usa_count = le16_to_cpu(b->usa_count) - 1; - data_pos = (u16*)b + NTFS_BLOCK_SIZE/sizeof(u16) - 1; - /* Fixup all sectors. */ - while (usa_count--) { - /* - * Increment position in usa and restore original data from - * the usa into the data buffer. - */ - *data_pos = *(++usa_pos); - /* Increment position in data as well. */ - data_pos += NTFS_BLOCK_SIZE/sizeof(u16); - } - return 0; -} - -/** - * pre_write_mst_fixup - apply multi sector transfer protection - * @b: pointer to the data to protect - * @size: size in bytes of @b - * - * Perform the necessary pre write multi sector transfer fixup on the data - * pointer to by @b of @size. - * - * Return 0 if fixup applied (success) or -EINVAL if no fixup was performed - * (assumed not needed). This is in contrast to post_read_mst_fixup() above. - * - * NOTE: We consider the absence / invalidity of an update sequence array to - * mean that the structure is not subject to protection and hence doesn't need - * to be fixed up. This means that you have to create a valid update sequence - * array header in the ntfs record before calling this function, otherwise it - * will fail (the header needs to contain the position of the update sequence - * array together with the number of elements in the array). You also need to - * initialise the update sequence number before calling this function - * otherwise a random word will be used (whatever was in the record at that - * position at that time). - */ -int pre_write_mst_fixup(NTFS_RECORD *b, const u32 size) -{ - le16 *usa_pos, *data_pos; - u16 usa_ofs, usa_count, usn; - le16 le_usn; - - /* Sanity check + only fixup if it makes sense. */ - if (!b || ntfs_is_baad_record(b->magic) || - ntfs_is_hole_record(b->magic)) - return -EINVAL; - /* Setup the variables. */ - usa_ofs = le16_to_cpu(b->usa_ofs); - /* Decrement usa_count to get number of fixups. */ - usa_count = le16_to_cpu(b->usa_count) - 1; - /* Size and alignment checks. */ - if ( size & (NTFS_BLOCK_SIZE - 1) || - usa_ofs & 1 || - usa_ofs + (usa_count * 2) > size || - (size >> NTFS_BLOCK_SIZE_BITS) != usa_count) - return -EINVAL; - /* Position of usn in update sequence array. */ - usa_pos = (le16*)((u8*)b + usa_ofs); - /* - * Cyclically increment the update sequence number - * (skipping 0 and -1, i.e. 0xffff). - */ - usn = le16_to_cpup(usa_pos) + 1; - if (usn == 0xffff || !usn) - usn = 1; - le_usn = cpu_to_le16(usn); - *usa_pos = le_usn; - /* Position in data of first u16 that needs fixing up. */ - data_pos = (le16*)b + NTFS_BLOCK_SIZE/sizeof(le16) - 1; - /* Fixup all sectors. */ - while (usa_count--) { - /* - * Increment the position in the usa and save the - * original data from the data buffer into the usa. - */ - *(++usa_pos) = *data_pos; - /* Apply fixup to data. */ - *data_pos = le_usn; - /* Increment position in data as well. */ - data_pos += NTFS_BLOCK_SIZE/sizeof(le16); - } - return 0; -} - -/** - * post_write_mst_fixup - fast deprotect multi sector transfer protected data - * @b: pointer to the data to deprotect - * - * Perform the necessary post write multi sector transfer fixup, not checking - * for any errors, because we assume we have just used pre_write_mst_fixup(), - * thus the data will be fine or we would never have gotten here. - */ -void post_write_mst_fixup(NTFS_RECORD *b) -{ - le16 *usa_pos, *data_pos; - - u16 usa_ofs = le16_to_cpu(b->usa_ofs); - u16 usa_count = le16_to_cpu(b->usa_count) - 1; - - /* Position of usn in update sequence array. */ - usa_pos = (le16*)b + usa_ofs/sizeof(le16); - - /* Position in protected data of first u16 that needs fixing up. */ - data_pos = (le16*)b + NTFS_BLOCK_SIZE/sizeof(le16) - 1; - - /* Fixup all sectors. */ - while (usa_count--) { - /* - * Increment position in usa and restore original data from - * the usa into the data buffer. - */ - *data_pos = *(++usa_pos); - - /* Increment position in data as well. */ - data_pos += NTFS_BLOCK_SIZE/sizeof(le16); - } -} diff --git a/fs/ntfs/namei.c b/fs/ntfs/namei.c deleted file mode 100644 index ab44f2db533b..000000000000 --- a/fs/ntfs/namei.c +++ /dev/null @@ -1,391 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * namei.c - NTFS kernel directory inode operations. Part of the Linux-NTFS - * project. - * - * Copyright (c) 2001-2006 Anton Altaparmakov - */ - -#include -#include -#include -#include - -#include "attrib.h" -#include "debug.h" -#include "dir.h" -#include "mft.h" -#include "ntfs.h" - -/** - * ntfs_lookup - find the inode represented by a dentry in a directory inode - * @dir_ino: directory inode in which to look for the inode - * @dent: dentry representing the inode to look for - * @flags: lookup flags - * - * In short, ntfs_lookup() looks for the inode represented by the dentry @dent - * in the directory inode @dir_ino and if found attaches the inode to the - * dentry @dent. - * - * In more detail, the dentry @dent specifies which inode to look for by - * supplying the name of the inode in @dent->d_name.name. ntfs_lookup() - * converts the name to Unicode and walks the contents of the directory inode - * @dir_ino looking for the converted Unicode name. If the name is found in the - * directory, the corresponding inode is loaded by calling ntfs_iget() on its - * inode number and the inode is associated with the dentry @dent via a call to - * d_splice_alias(). - * - * If the name is not found in the directory, a NULL inode is inserted into the - * dentry @dent via a call to d_add(). The dentry is then termed a negative - * dentry. - * - * Only if an actual error occurs, do we return an error via ERR_PTR(). - * - * In order to handle the case insensitivity issues of NTFS with regards to the - * dcache and the dcache requiring only one dentry per directory, we deal with - * dentry aliases that only differ in case in ->ntfs_lookup() while maintaining - * a case sensitive dcache. This means that we get the full benefit of dcache - * speed when the file/directory is looked up with the same case as returned by - * ->ntfs_readdir() but that a lookup for any other case (or for the short file - * name) will not find anything in dcache and will enter ->ntfs_lookup() - * instead, where we search the directory for a fully matching file name - * (including case) and if that is not found, we search for a file name that - * matches with different case and if that has non-POSIX semantics we return - * that. We actually do only one search (case sensitive) and keep tabs on - * whether we have found a case insensitive match in the process. - * - * To simplify matters for us, we do not treat the short vs long filenames as - * two hard links but instead if the lookup matches a short filename, we - * return the dentry for the corresponding long filename instead. - * - * There are three cases we need to distinguish here: - * - * 1) @dent perfectly matches (i.e. including case) a directory entry with a - * file name in the WIN32 or POSIX namespaces. In this case - * ntfs_lookup_inode_by_name() will return with name set to NULL and we - * just d_splice_alias() @dent. - * 2) @dent matches (not including case) a directory entry with a file name in - * the WIN32 namespace. In this case ntfs_lookup_inode_by_name() will return - * with name set to point to a kmalloc()ed ntfs_name structure containing - * the properly cased little endian Unicode name. We convert the name to the - * current NLS code page, search if a dentry with this name already exists - * and if so return that instead of @dent. At this point things are - * complicated by the possibility of 'disconnected' dentries due to NFS - * which we deal with appropriately (see the code comments). The VFS will - * then destroy the old @dent and use the one we returned. If a dentry is - * not found, we allocate a new one, d_splice_alias() it, and return it as - * above. - * 3) @dent matches either perfectly or not (i.e. we don't care about case) a - * directory entry with a file name in the DOS namespace. In this case - * ntfs_lookup_inode_by_name() will return with name set to point to a - * kmalloc()ed ntfs_name structure containing the mft reference (cpu endian) - * of the inode. We use the mft reference to read the inode and to find the - * file name in the WIN32 namespace corresponding to the matched short file - * name. We then convert the name to the current NLS code page, and proceed - * searching for a dentry with this name, etc, as in case 2), above. - * - * Locking: Caller must hold i_mutex on the directory. - */ -static struct dentry *ntfs_lookup(struct inode *dir_ino, struct dentry *dent, - unsigned int flags) -{ - ntfs_volume *vol = NTFS_SB(dir_ino->i_sb); - struct inode *dent_inode; - ntfschar *uname; - ntfs_name *name = NULL; - MFT_REF mref; - unsigned long dent_ino; - int uname_len; - - ntfs_debug("Looking up %pd in directory inode 0x%lx.", - dent, dir_ino->i_ino); - /* Convert the name of the dentry to Unicode. */ - uname_len = ntfs_nlstoucs(vol, dent->d_name.name, dent->d_name.len, - &uname); - if (uname_len < 0) { - if (uname_len != -ENAMETOOLONG) - ntfs_error(vol->sb, "Failed to convert name to " - "Unicode."); - return ERR_PTR(uname_len); - } - mref = ntfs_lookup_inode_by_name(NTFS_I(dir_ino), uname, uname_len, - &name); - kmem_cache_free(ntfs_name_cache, uname); - if (!IS_ERR_MREF(mref)) { - dent_ino = MREF(mref); - ntfs_debug("Found inode 0x%lx. Calling ntfs_iget.", dent_ino); - dent_inode = ntfs_iget(vol->sb, dent_ino); - if (!IS_ERR(dent_inode)) { - /* Consistency check. */ - if (is_bad_inode(dent_inode) || MSEQNO(mref) == - NTFS_I(dent_inode)->seq_no || - dent_ino == FILE_MFT) { - /* Perfect WIN32/POSIX match. -- Case 1. */ - if (!name) { - ntfs_debug("Done. (Case 1.)"); - return d_splice_alias(dent_inode, dent); - } - /* - * We are too indented. Handle imperfect - * matches and short file names further below. - */ - goto handle_name; - } - ntfs_error(vol->sb, "Found stale reference to inode " - "0x%lx (reference sequence number = " - "0x%x, inode sequence number = 0x%x), " - "returning -EIO. Run chkdsk.", - dent_ino, MSEQNO(mref), - NTFS_I(dent_inode)->seq_no); - iput(dent_inode); - dent_inode = ERR_PTR(-EIO); - } else - ntfs_error(vol->sb, "ntfs_iget(0x%lx) failed with " - "error code %li.", dent_ino, - PTR_ERR(dent_inode)); - kfree(name); - /* Return the error code. */ - return ERR_CAST(dent_inode); - } - /* It is guaranteed that @name is no longer allocated at this point. */ - if (MREF_ERR(mref) == -ENOENT) { - ntfs_debug("Entry was not found, adding negative dentry."); - /* The dcache will handle negative entries. */ - d_add(dent, NULL); - ntfs_debug("Done."); - return NULL; - } - ntfs_error(vol->sb, "ntfs_lookup_ino_by_name() failed with error " - "code %i.", -MREF_ERR(mref)); - return ERR_PTR(MREF_ERR(mref)); - // TODO: Consider moving this lot to a separate function! (AIA) -handle_name: - { - MFT_RECORD *m; - ntfs_attr_search_ctx *ctx; - ntfs_inode *ni = NTFS_I(dent_inode); - int err; - struct qstr nls_name; - - nls_name.name = NULL; - if (name->type != FILE_NAME_DOS) { /* Case 2. */ - ntfs_debug("Case 2."); - nls_name.len = (unsigned)ntfs_ucstonls(vol, - (ntfschar*)&name->name, name->len, - (unsigned char**)&nls_name.name, 0); - kfree(name); - } else /* if (name->type == FILE_NAME_DOS) */ { /* Case 3. */ - FILE_NAME_ATTR *fn; - - ntfs_debug("Case 3."); - kfree(name); - - /* Find the WIN32 name corresponding to the matched DOS name. */ - ni = NTFS_I(dent_inode); - m = map_mft_record(ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - m = NULL; - ctx = NULL; - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(ni, m); - if (unlikely(!ctx)) { - err = -ENOMEM; - goto err_out; - } - do { - ATTR_RECORD *a; - u32 val_len; - - err = ntfs_attr_lookup(AT_FILE_NAME, NULL, 0, 0, 0, - NULL, 0, ctx); - if (unlikely(err)) { - ntfs_error(vol->sb, "Inode corrupt: No WIN32 " - "namespace counterpart to DOS " - "file name. Run chkdsk."); - if (err == -ENOENT) - err = -EIO; - goto err_out; - } - /* Consistency checks. */ - a = ctx->attr; - if (a->non_resident || a->flags) - goto eio_err_out; - val_len = le32_to_cpu(a->data.resident.value_length); - if (le16_to_cpu(a->data.resident.value_offset) + - val_len > le32_to_cpu(a->length)) - goto eio_err_out; - fn = (FILE_NAME_ATTR*)((u8*)ctx->attr + le16_to_cpu( - ctx->attr->data.resident.value_offset)); - if ((u32)(fn->file_name_length * sizeof(ntfschar) + - sizeof(FILE_NAME_ATTR)) > val_len) - goto eio_err_out; - } while (fn->file_name_type != FILE_NAME_WIN32); - - /* Convert the found WIN32 name to current NLS code page. */ - nls_name.len = (unsigned)ntfs_ucstonls(vol, - (ntfschar*)&fn->file_name, fn->file_name_length, - (unsigned char**)&nls_name.name, 0); - - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(ni); - } - m = NULL; - ctx = NULL; - - /* Check if a conversion error occurred. */ - if ((signed)nls_name.len < 0) { - err = (signed)nls_name.len; - goto err_out; - } - nls_name.hash = full_name_hash(dent, nls_name.name, nls_name.len); - - dent = d_add_ci(dent, dent_inode, &nls_name); - kfree(nls_name.name); - return dent; - -eio_err_out: - ntfs_error(vol->sb, "Illegal file name attribute. Run chkdsk."); - err = -EIO; -err_out: - if (ctx) - ntfs_attr_put_search_ctx(ctx); - if (m) - unmap_mft_record(ni); - iput(dent_inode); - ntfs_error(vol->sb, "Failed, returning error code %i.", err); - return ERR_PTR(err); - } -} - -/* - * Inode operations for directories. - */ -const struct inode_operations ntfs_dir_inode_ops = { - .lookup = ntfs_lookup, /* VFS: Lookup directory. */ -}; - -/** - * ntfs_get_parent - find the dentry of the parent of a given directory dentry - * @child_dent: dentry of the directory whose parent directory to find - * - * Find the dentry for the parent directory of the directory specified by the - * dentry @child_dent. This function is called from - * fs/exportfs/expfs.c::find_exported_dentry() which in turn is called from the - * default ->decode_fh() which is export_decode_fh() in the same file. - * - * The code is based on the ext3 ->get_parent() implementation found in - * fs/ext3/namei.c::ext3_get_parent(). - * - * Note: ntfs_get_parent() is called with @d_inode(child_dent)->i_mutex down. - * - * Return the dentry of the parent directory on success or the error code on - * error (IS_ERR() is true). - */ -static struct dentry *ntfs_get_parent(struct dentry *child_dent) -{ - struct inode *vi = d_inode(child_dent); - ntfs_inode *ni = NTFS_I(vi); - MFT_RECORD *mrec; - ntfs_attr_search_ctx *ctx; - ATTR_RECORD *attr; - FILE_NAME_ATTR *fn; - unsigned long parent_ino; - int err; - - ntfs_debug("Entering for inode 0x%lx.", vi->i_ino); - /* Get the mft record of the inode belonging to the child dentry. */ - mrec = map_mft_record(ni); - if (IS_ERR(mrec)) - return ERR_CAST(mrec); - /* Find the first file name attribute in the mft record. */ - ctx = ntfs_attr_get_search_ctx(ni, mrec); - if (unlikely(!ctx)) { - unmap_mft_record(ni); - return ERR_PTR(-ENOMEM); - } -try_next: - err = ntfs_attr_lookup(AT_FILE_NAME, NULL, 0, CASE_SENSITIVE, 0, NULL, - 0, ctx); - if (unlikely(err)) { - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(ni); - if (err == -ENOENT) - ntfs_error(vi->i_sb, "Inode 0x%lx does not have a " - "file name attribute. Run chkdsk.", - vi->i_ino); - return ERR_PTR(err); - } - attr = ctx->attr; - if (unlikely(attr->non_resident)) - goto try_next; - fn = (FILE_NAME_ATTR *)((u8 *)attr + - le16_to_cpu(attr->data.resident.value_offset)); - if (unlikely((u8 *)fn + le32_to_cpu(attr->data.resident.value_length) > - (u8*)attr + le32_to_cpu(attr->length))) - goto try_next; - /* Get the inode number of the parent directory. */ - parent_ino = MREF_LE(fn->parent_directory); - /* Release the search context and the mft record of the child. */ - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(ni); - - return d_obtain_alias(ntfs_iget(vi->i_sb, parent_ino)); -} - -static struct inode *ntfs_nfs_get_inode(struct super_block *sb, - u64 ino, u32 generation) -{ - struct inode *inode; - - inode = ntfs_iget(sb, ino); - if (!IS_ERR(inode)) { - if (is_bad_inode(inode) || inode->i_generation != generation) { - iput(inode); - inode = ERR_PTR(-ESTALE); - } - } - - return inode; -} - -static struct dentry *ntfs_fh_to_dentry(struct super_block *sb, struct fid *fid, - int fh_len, int fh_type) -{ - return generic_fh_to_dentry(sb, fid, fh_len, fh_type, - ntfs_nfs_get_inode); -} - -static struct dentry *ntfs_fh_to_parent(struct super_block *sb, struct fid *fid, - int fh_len, int fh_type) -{ - return generic_fh_to_parent(sb, fid, fh_len, fh_type, - ntfs_nfs_get_inode); -} - -/* - * Export operations allowing NFS exporting of mounted NTFS partitions. - * - * We use the default ->encode_fh() for now. Note that they - * use 32 bits to store the inode number which is an unsigned long so on 64-bit - * architectures is usually 64 bits so it would all fail horribly on huge - * volumes. I guess we need to define our own encode and decode fh functions - * that store 64-bit inode numbers at some point but for now we will ignore the - * problem... - * - * We also use the default ->get_name() helper (used by ->decode_fh() via - * fs/exportfs/expfs.c::find_exported_dentry()) as that is completely fs - * independent. - * - * The default ->get_parent() just returns -EACCES so we have to provide our - * own and the default ->get_dentry() is incompatible with NTFS due to not - * allowing the inode number 0 which is used in NTFS for the system file $MFT - * and due to using iget() whereas NTFS needs ntfs_iget(). - */ -const struct export_operations ntfs_export_ops = { - .get_parent = ntfs_get_parent, /* Find the parent of a given - directory. */ - .fh_to_dentry = ntfs_fh_to_dentry, - .fh_to_parent = ntfs_fh_to_parent, -}; diff --git a/fs/ntfs/ntfs.h b/fs/ntfs/ntfs.h deleted file mode 100644 index e81376ea9152..000000000000 --- a/fs/ntfs/ntfs.h +++ /dev/null @@ -1,150 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * ntfs.h - Defines for NTFS Linux kernel driver. - * - * Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc. - * Copyright (C) 2002 Richard Russon - */ - -#ifndef _LINUX_NTFS_H -#define _LINUX_NTFS_H - -#include -#include -#include -#include -#include -#include -#include -#include - -#include "types.h" -#include "volume.h" -#include "layout.h" - -typedef enum { - NTFS_BLOCK_SIZE = 512, - NTFS_BLOCK_SIZE_BITS = 9, - NTFS_SB_MAGIC = 0x5346544e, /* 'NTFS' */ - NTFS_MAX_NAME_LEN = 255, - NTFS_MAX_ATTR_NAME_LEN = 255, - NTFS_MAX_CLUSTER_SIZE = 64 * 1024, /* 64kiB */ - NTFS_MAX_PAGES_PER_CLUSTER = NTFS_MAX_CLUSTER_SIZE / PAGE_SIZE, -} NTFS_CONSTANTS; - -/* Global variables. */ - -/* Slab caches (from super.c). */ -extern struct kmem_cache *ntfs_name_cache; -extern struct kmem_cache *ntfs_inode_cache; -extern struct kmem_cache *ntfs_big_inode_cache; -extern struct kmem_cache *ntfs_attr_ctx_cache; -extern struct kmem_cache *ntfs_index_ctx_cache; - -/* The various operations structs defined throughout the driver files. */ -extern const struct address_space_operations ntfs_normal_aops; -extern const struct address_space_operations ntfs_compressed_aops; -extern const struct address_space_operations ntfs_mst_aops; - -extern const struct file_operations ntfs_file_ops; -extern const struct inode_operations ntfs_file_inode_ops; - -extern const struct file_operations ntfs_dir_ops; -extern const struct inode_operations ntfs_dir_inode_ops; - -extern const struct file_operations ntfs_empty_file_ops; -extern const struct inode_operations ntfs_empty_inode_ops; - -extern const struct export_operations ntfs_export_ops; - -/** - * NTFS_SB - return the ntfs volume given a vfs super block - * @sb: VFS super block - * - * NTFS_SB() returns the ntfs volume associated with the VFS super block @sb. - */ -static inline ntfs_volume *NTFS_SB(struct super_block *sb) -{ - return sb->s_fs_info; -} - -/* Declarations of functions and global variables. */ - -/* From fs/ntfs/compress.c */ -extern int ntfs_read_compressed_block(struct page *page); -extern int allocate_compression_buffers(void); -extern void free_compression_buffers(void); - -/* From fs/ntfs/super.c */ -#define default_upcase_len 0x10000 -extern struct mutex ntfs_lock; - -typedef struct { - int val; - char *str; -} option_t; -extern const option_t on_errors_arr[]; - -/* From fs/ntfs/mst.c */ -extern int post_read_mst_fixup(NTFS_RECORD *b, const u32 size); -extern int pre_write_mst_fixup(NTFS_RECORD *b, const u32 size); -extern void post_write_mst_fixup(NTFS_RECORD *b); - -/* From fs/ntfs/unistr.c */ -extern bool ntfs_are_names_equal(const ntfschar *s1, size_t s1_len, - const ntfschar *s2, size_t s2_len, - const IGNORE_CASE_BOOL ic, - const ntfschar *upcase, const u32 upcase_size); -extern int ntfs_collate_names(const ntfschar *name1, const u32 name1_len, - const ntfschar *name2, const u32 name2_len, - const int err_val, const IGNORE_CASE_BOOL ic, - const ntfschar *upcase, const u32 upcase_len); -extern int ntfs_ucsncmp(const ntfschar *s1, const ntfschar *s2, size_t n); -extern int ntfs_ucsncasecmp(const ntfschar *s1, const ntfschar *s2, size_t n, - const ntfschar *upcase, const u32 upcase_size); -extern void ntfs_upcase_name(ntfschar *name, u32 name_len, - const ntfschar *upcase, const u32 upcase_len); -extern void ntfs_file_upcase_value(FILE_NAME_ATTR *file_name_attr, - const ntfschar *upcase, const u32 upcase_len); -extern int ntfs_file_compare_values(FILE_NAME_ATTR *file_name_attr1, - FILE_NAME_ATTR *file_name_attr2, - const int err_val, const IGNORE_CASE_BOOL ic, - const ntfschar *upcase, const u32 upcase_len); -extern int ntfs_nlstoucs(const ntfs_volume *vol, const char *ins, - const int ins_len, ntfschar **outs); -extern int ntfs_ucstonls(const ntfs_volume *vol, const ntfschar *ins, - const int ins_len, unsigned char **outs, int outs_len); - -/* From fs/ntfs/upcase.c */ -extern ntfschar *generate_default_upcase(void); - -static inline int ntfs_ffs(int x) -{ - int r = 1; - - if (!x) - return 0; - if (!(x & 0xffff)) { - x >>= 16; - r += 16; - } - if (!(x & 0xff)) { - x >>= 8; - r += 8; - } - if (!(x & 0xf)) { - x >>= 4; - r += 4; - } - if (!(x & 3)) { - x >>= 2; - r += 2; - } - if (!(x & 1)) { - x >>= 1; - r += 1; - } - return r; -} - -#endif /* _LINUX_NTFS_H */ diff --git a/fs/ntfs/quota.c b/fs/ntfs/quota.c deleted file mode 100644 index 9160480222fd..000000000000 --- a/fs/ntfs/quota.c +++ /dev/null @@ -1,103 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * quota.c - NTFS kernel quota ($Quota) handling. Part of the Linux-NTFS - * project. - * - * Copyright (c) 2004 Anton Altaparmakov - */ - -#ifdef NTFS_RW - -#include "index.h" -#include "quota.h" -#include "debug.h" -#include "ntfs.h" - -/** - * ntfs_mark_quotas_out_of_date - mark the quotas out of date on an ntfs volume - * @vol: ntfs volume on which to mark the quotas out of date - * - * Mark the quotas out of date on the ntfs volume @vol and return 'true' on - * success and 'false' on error. - */ -bool ntfs_mark_quotas_out_of_date(ntfs_volume *vol) -{ - ntfs_index_context *ictx; - QUOTA_CONTROL_ENTRY *qce; - const le32 qid = QUOTA_DEFAULTS_ID; - int err; - - ntfs_debug("Entering."); - if (NVolQuotaOutOfDate(vol)) - goto done; - if (!vol->quota_ino || !vol->quota_q_ino) { - ntfs_error(vol->sb, "Quota inodes are not open."); - return false; - } - inode_lock(vol->quota_q_ino); - ictx = ntfs_index_ctx_get(NTFS_I(vol->quota_q_ino)); - if (!ictx) { - ntfs_error(vol->sb, "Failed to get index context."); - goto err_out; - } - err = ntfs_index_lookup(&qid, sizeof(qid), ictx); - if (err) { - if (err == -ENOENT) - ntfs_error(vol->sb, "Quota defaults entry is not " - "present."); - else - ntfs_error(vol->sb, "Lookup of quota defaults entry " - "failed."); - goto err_out; - } - if (ictx->data_len < offsetof(QUOTA_CONTROL_ENTRY, sid)) { - ntfs_error(vol->sb, "Quota defaults entry size is invalid. " - "Run chkdsk."); - goto err_out; - } - qce = (QUOTA_CONTROL_ENTRY*)ictx->data; - if (le32_to_cpu(qce->version) != QUOTA_VERSION) { - ntfs_error(vol->sb, "Quota defaults entry version 0x%x is not " - "supported.", le32_to_cpu(qce->version)); - goto err_out; - } - ntfs_debug("Quota defaults flags = 0x%x.", le32_to_cpu(qce->flags)); - /* If quotas are already marked out of date, no need to do anything. */ - if (qce->flags & QUOTA_FLAG_OUT_OF_DATE) - goto set_done; - /* - * If quota tracking is neither requested, nor enabled and there are no - * pending deletes, no need to mark the quotas out of date. - */ - if (!(qce->flags & (QUOTA_FLAG_TRACKING_ENABLED | - QUOTA_FLAG_TRACKING_REQUESTED | - QUOTA_FLAG_PENDING_DELETES))) - goto set_done; - /* - * Set the QUOTA_FLAG_OUT_OF_DATE bit thus marking quotas out of date. - * This is verified on WinXP to be sufficient to cause windows to - * rescan the volume on boot and update all quota entries. - */ - qce->flags |= QUOTA_FLAG_OUT_OF_DATE; - /* Ensure the modified flags are written to disk. */ - ntfs_index_entry_flush_dcache_page(ictx); - ntfs_index_entry_mark_dirty(ictx); -set_done: - ntfs_index_ctx_put(ictx); - inode_unlock(vol->quota_q_ino); - /* - * We set the flag so we do not try to mark the quotas out of date - * again on remount. - */ - NVolSetQuotaOutOfDate(vol); -done: - ntfs_debug("Done."); - return true; -err_out: - if (ictx) - ntfs_index_ctx_put(ictx); - inode_unlock(vol->quota_q_ino); - return false; -} - -#endif /* NTFS_RW */ diff --git a/fs/ntfs/quota.h b/fs/ntfs/quota.h deleted file mode 100644 index fe3132a3d6d2..000000000000 --- a/fs/ntfs/quota.h +++ /dev/null @@ -1,21 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * quota.h - Defines for NTFS kernel quota ($Quota) handling. Part of the - * Linux-NTFS project. - * - * Copyright (c) 2004 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_QUOTA_H -#define _LINUX_NTFS_QUOTA_H - -#ifdef NTFS_RW - -#include "types.h" -#include "volume.h" - -extern bool ntfs_mark_quotas_out_of_date(ntfs_volume *vol); - -#endif /* NTFS_RW */ - -#endif /* _LINUX_NTFS_QUOTA_H */ diff --git a/fs/ntfs/runlist.c b/fs/ntfs/runlist.c deleted file mode 100644 index 0d448e9881f7..000000000000 --- a/fs/ntfs/runlist.c +++ /dev/null @@ -1,1893 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * runlist.c - NTFS runlist handling code. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2007 Anton Altaparmakov - * Copyright (c) 2002-2005 Richard Russon - */ - -#include "debug.h" -#include "dir.h" -#include "endian.h" -#include "malloc.h" -#include "ntfs.h" - -/** - * ntfs_rl_mm - runlist memmove - * - * It is up to the caller to serialize access to the runlist @base. - */ -static inline void ntfs_rl_mm(runlist_element *base, int dst, int src, - int size) -{ - if (likely((dst != src) && (size > 0))) - memmove(base + dst, base + src, size * sizeof(*base)); -} - -/** - * ntfs_rl_mc - runlist memory copy - * - * It is up to the caller to serialize access to the runlists @dstbase and - * @srcbase. - */ -static inline void ntfs_rl_mc(runlist_element *dstbase, int dst, - runlist_element *srcbase, int src, int size) -{ - if (likely(size > 0)) - memcpy(dstbase + dst, srcbase + src, size * sizeof(*dstbase)); -} - -/** - * ntfs_rl_realloc - Reallocate memory for runlists - * @rl: original runlist - * @old_size: number of runlist elements in the original runlist @rl - * @new_size: number of runlist elements we need space for - * - * As the runlists grow, more memory will be required. To prevent the - * kernel having to allocate and reallocate large numbers of small bits of - * memory, this function returns an entire page of memory. - * - * It is up to the caller to serialize access to the runlist @rl. - * - * N.B. If the new allocation doesn't require a different number of pages in - * memory, the function will return the original pointer. - * - * On success, return a pointer to the newly allocated, or recycled, memory. - * On error, return -errno. The following error codes are defined: - * -ENOMEM - Not enough memory to allocate runlist array. - * -EINVAL - Invalid parameters were passed in. - */ -static inline runlist_element *ntfs_rl_realloc(runlist_element *rl, - int old_size, int new_size) -{ - runlist_element *new_rl; - - old_size = PAGE_ALIGN(old_size * sizeof(*rl)); - new_size = PAGE_ALIGN(new_size * sizeof(*rl)); - if (old_size == new_size) - return rl; - - new_rl = ntfs_malloc_nofs(new_size); - if (unlikely(!new_rl)) - return ERR_PTR(-ENOMEM); - - if (likely(rl != NULL)) { - if (unlikely(old_size > new_size)) - old_size = new_size; - memcpy(new_rl, rl, old_size); - ntfs_free(rl); - } - return new_rl; -} - -/** - * ntfs_rl_realloc_nofail - Reallocate memory for runlists - * @rl: original runlist - * @old_size: number of runlist elements in the original runlist @rl - * @new_size: number of runlist elements we need space for - * - * As the runlists grow, more memory will be required. To prevent the - * kernel having to allocate and reallocate large numbers of small bits of - * memory, this function returns an entire page of memory. - * - * This function guarantees that the allocation will succeed. It will sleep - * for as long as it takes to complete the allocation. - * - * It is up to the caller to serialize access to the runlist @rl. - * - * N.B. If the new allocation doesn't require a different number of pages in - * memory, the function will return the original pointer. - * - * On success, return a pointer to the newly allocated, or recycled, memory. - * On error, return -errno. The following error codes are defined: - * -ENOMEM - Not enough memory to allocate runlist array. - * -EINVAL - Invalid parameters were passed in. - */ -static inline runlist_element *ntfs_rl_realloc_nofail(runlist_element *rl, - int old_size, int new_size) -{ - runlist_element *new_rl; - - old_size = PAGE_ALIGN(old_size * sizeof(*rl)); - new_size = PAGE_ALIGN(new_size * sizeof(*rl)); - if (old_size == new_size) - return rl; - - new_rl = ntfs_malloc_nofs_nofail(new_size); - BUG_ON(!new_rl); - - if (likely(rl != NULL)) { - if (unlikely(old_size > new_size)) - old_size = new_size; - memcpy(new_rl, rl, old_size); - ntfs_free(rl); - } - return new_rl; -} - -/** - * ntfs_are_rl_mergeable - test if two runlists can be joined together - * @dst: original runlist - * @src: new runlist to test for mergeability with @dst - * - * Test if two runlists can be joined together. For this, their VCNs and LCNs - * must be adjacent. - * - * It is up to the caller to serialize access to the runlists @dst and @src. - * - * Return: true Success, the runlists can be merged. - * false Failure, the runlists cannot be merged. - */ -static inline bool ntfs_are_rl_mergeable(runlist_element *dst, - runlist_element *src) -{ - BUG_ON(!dst); - BUG_ON(!src); - - /* We can merge unmapped regions even if they are misaligned. */ - if ((dst->lcn == LCN_RL_NOT_MAPPED) && (src->lcn == LCN_RL_NOT_MAPPED)) - return true; - /* If the runs are misaligned, we cannot merge them. */ - if ((dst->vcn + dst->length) != src->vcn) - return false; - /* If both runs are non-sparse and contiguous, we can merge them. */ - if ((dst->lcn >= 0) && (src->lcn >= 0) && - ((dst->lcn + dst->length) == src->lcn)) - return true; - /* If we are merging two holes, we can merge them. */ - if ((dst->lcn == LCN_HOLE) && (src->lcn == LCN_HOLE)) - return true; - /* Cannot merge. */ - return false; -} - -/** - * __ntfs_rl_merge - merge two runlists without testing if they can be merged - * @dst: original, destination runlist - * @src: new runlist to merge with @dst - * - * Merge the two runlists, writing into the destination runlist @dst. The - * caller must make sure the runlists can be merged or this will corrupt the - * destination runlist. - * - * It is up to the caller to serialize access to the runlists @dst and @src. - */ -static inline void __ntfs_rl_merge(runlist_element *dst, runlist_element *src) -{ - dst->length += src->length; -} - -/** - * ntfs_rl_append - append a runlist after a given element - * @dst: original runlist to be worked on - * @dsize: number of elements in @dst (including end marker) - * @src: runlist to be inserted into @dst - * @ssize: number of elements in @src (excluding end marker) - * @loc: append the new runlist @src after this element in @dst - * - * Append the runlist @src after element @loc in @dst. Merge the right end of - * the new runlist, if necessary. Adjust the size of the hole before the - * appended runlist. - * - * It is up to the caller to serialize access to the runlists @dst and @src. - * - * On success, return a pointer to the new, combined, runlist. Note, both - * runlists @dst and @src are deallocated before returning so you cannot use - * the pointers for anything any more. (Strictly speaking the returned runlist - * may be the same as @dst but this is irrelevant.) - * - * On error, return -errno. Both runlists are left unmodified. The following - * error codes are defined: - * -ENOMEM - Not enough memory to allocate runlist array. - * -EINVAL - Invalid parameters were passed in. - */ -static inline runlist_element *ntfs_rl_append(runlist_element *dst, - int dsize, runlist_element *src, int ssize, int loc) -{ - bool right = false; /* Right end of @src needs merging. */ - int marker; /* End of the inserted runs. */ - - BUG_ON(!dst); - BUG_ON(!src); - - /* First, check if the right hand end needs merging. */ - if ((loc + 1) < dsize) - right = ntfs_are_rl_mergeable(src + ssize - 1, dst + loc + 1); - - /* Space required: @dst size + @src size, less one if we merged. */ - dst = ntfs_rl_realloc(dst, dsize, dsize + ssize - right); - if (IS_ERR(dst)) - return dst; - /* - * We are guaranteed to succeed from here so can start modifying the - * original runlists. - */ - - /* First, merge the right hand end, if necessary. */ - if (right) - __ntfs_rl_merge(src + ssize - 1, dst + loc + 1); - - /* First run after the @src runs that have been inserted. */ - marker = loc + ssize + 1; - - /* Move the tail of @dst out of the way, then copy in @src. */ - ntfs_rl_mm(dst, marker, loc + 1 + right, dsize - (loc + 1 + right)); - ntfs_rl_mc(dst, loc + 1, src, 0, ssize); - - /* Adjust the size of the preceding hole. */ - dst[loc].length = dst[loc + 1].vcn - dst[loc].vcn; - - /* We may have changed the length of the file, so fix the end marker */ - if (dst[marker].lcn == LCN_ENOENT) - dst[marker].vcn = dst[marker - 1].vcn + dst[marker - 1].length; - - return dst; -} - -/** - * ntfs_rl_insert - insert a runlist into another - * @dst: original runlist to be worked on - * @dsize: number of elements in @dst (including end marker) - * @src: new runlist to be inserted - * @ssize: number of elements in @src (excluding end marker) - * @loc: insert the new runlist @src before this element in @dst - * - * Insert the runlist @src before element @loc in the runlist @dst. Merge the - * left end of the new runlist, if necessary. Adjust the size of the hole - * after the inserted runlist. - * - * It is up to the caller to serialize access to the runlists @dst and @src. - * - * On success, return a pointer to the new, combined, runlist. Note, both - * runlists @dst and @src are deallocated before returning so you cannot use - * the pointers for anything any more. (Strictly speaking the returned runlist - * may be the same as @dst but this is irrelevant.) - * - * On error, return -errno. Both runlists are left unmodified. The following - * error codes are defined: - * -ENOMEM - Not enough memory to allocate runlist array. - * -EINVAL - Invalid parameters were passed in. - */ -static inline runlist_element *ntfs_rl_insert(runlist_element *dst, - int dsize, runlist_element *src, int ssize, int loc) -{ - bool left = false; /* Left end of @src needs merging. */ - bool disc = false; /* Discontinuity between @dst and @src. */ - int marker; /* End of the inserted runs. */ - - BUG_ON(!dst); - BUG_ON(!src); - - /* - * disc => Discontinuity between the end of @dst and the start of @src. - * This means we might need to insert a "not mapped" run. - */ - if (loc == 0) - disc = (src[0].vcn > 0); - else { - s64 merged_length; - - left = ntfs_are_rl_mergeable(dst + loc - 1, src); - - merged_length = dst[loc - 1].length; - if (left) - merged_length += src->length; - - disc = (src[0].vcn > dst[loc - 1].vcn + merged_length); - } - /* - * Space required: @dst size + @src size, less one if we merged, plus - * one if there was a discontinuity. - */ - dst = ntfs_rl_realloc(dst, dsize, dsize + ssize - left + disc); - if (IS_ERR(dst)) - return dst; - /* - * We are guaranteed to succeed from here so can start modifying the - * original runlist. - */ - if (left) - __ntfs_rl_merge(dst + loc - 1, src); - /* - * First run after the @src runs that have been inserted. - * Nominally, @marker equals @loc + @ssize, i.e. location + number of - * runs in @src. However, if @left, then the first run in @src has - * been merged with one in @dst. And if @disc, then @dst and @src do - * not meet and we need an extra run to fill the gap. - */ - marker = loc + ssize - left + disc; - - /* Move the tail of @dst out of the way, then copy in @src. */ - ntfs_rl_mm(dst, marker, loc, dsize - loc); - ntfs_rl_mc(dst, loc + disc, src, left, ssize - left); - - /* Adjust the VCN of the first run after the insertion... */ - dst[marker].vcn = dst[marker - 1].vcn + dst[marker - 1].length; - /* ... and the length. */ - if (dst[marker].lcn == LCN_HOLE || dst[marker].lcn == LCN_RL_NOT_MAPPED) - dst[marker].length = dst[marker + 1].vcn - dst[marker].vcn; - - /* Writing beyond the end of the file and there is a discontinuity. */ - if (disc) { - if (loc > 0) { - dst[loc].vcn = dst[loc - 1].vcn + dst[loc - 1].length; - dst[loc].length = dst[loc + 1].vcn - dst[loc].vcn; - } else { - dst[loc].vcn = 0; - dst[loc].length = dst[loc + 1].vcn; - } - dst[loc].lcn = LCN_RL_NOT_MAPPED; - } - return dst; -} - -/** - * ntfs_rl_replace - overwrite a runlist element with another runlist - * @dst: original runlist to be worked on - * @dsize: number of elements in @dst (including end marker) - * @src: new runlist to be inserted - * @ssize: number of elements in @src (excluding end marker) - * @loc: index in runlist @dst to overwrite with @src - * - * Replace the runlist element @dst at @loc with @src. Merge the left and - * right ends of the inserted runlist, if necessary. - * - * It is up to the caller to serialize access to the runlists @dst and @src. - * - * On success, return a pointer to the new, combined, runlist. Note, both - * runlists @dst and @src are deallocated before returning so you cannot use - * the pointers for anything any more. (Strictly speaking the returned runlist - * may be the same as @dst but this is irrelevant.) - * - * On error, return -errno. Both runlists are left unmodified. The following - * error codes are defined: - * -ENOMEM - Not enough memory to allocate runlist array. - * -EINVAL - Invalid parameters were passed in. - */ -static inline runlist_element *ntfs_rl_replace(runlist_element *dst, - int dsize, runlist_element *src, int ssize, int loc) -{ - signed delta; - bool left = false; /* Left end of @src needs merging. */ - bool right = false; /* Right end of @src needs merging. */ - int tail; /* Start of tail of @dst. */ - int marker; /* End of the inserted runs. */ - - BUG_ON(!dst); - BUG_ON(!src); - - /* First, see if the left and right ends need merging. */ - if ((loc + 1) < dsize) - right = ntfs_are_rl_mergeable(src + ssize - 1, dst + loc + 1); - if (loc > 0) - left = ntfs_are_rl_mergeable(dst + loc - 1, src); - /* - * Allocate some space. We will need less if the left, right, or both - * ends get merged. The -1 accounts for the run being replaced. - */ - delta = ssize - 1 - left - right; - if (delta > 0) { - dst = ntfs_rl_realloc(dst, dsize, dsize + delta); - if (IS_ERR(dst)) - return dst; - } - /* - * We are guaranteed to succeed from here so can start modifying the - * original runlists. - */ - - /* First, merge the left and right ends, if necessary. */ - if (right) - __ntfs_rl_merge(src + ssize - 1, dst + loc + 1); - if (left) - __ntfs_rl_merge(dst + loc - 1, src); - /* - * Offset of the tail of @dst. This needs to be moved out of the way - * to make space for the runs to be copied from @src, i.e. the first - * run of the tail of @dst. - * Nominally, @tail equals @loc + 1, i.e. location, skipping the - * replaced run. However, if @right, then one of @dst's runs is - * already merged into @src. - */ - tail = loc + right + 1; - /* - * First run after the @src runs that have been inserted, i.e. where - * the tail of @dst needs to be moved to. - * Nominally, @marker equals @loc + @ssize, i.e. location + number of - * runs in @src. However, if @left, then the first run in @src has - * been merged with one in @dst. - */ - marker = loc + ssize - left; - - /* Move the tail of @dst out of the way, then copy in @src. */ - ntfs_rl_mm(dst, marker, tail, dsize - tail); - ntfs_rl_mc(dst, loc, src, left, ssize - left); - - /* We may have changed the length of the file, so fix the end marker. */ - if (dsize - tail > 0 && dst[marker].lcn == LCN_ENOENT) - dst[marker].vcn = dst[marker - 1].vcn + dst[marker - 1].length; - return dst; -} - -/** - * ntfs_rl_split - insert a runlist into the centre of a hole - * @dst: original runlist to be worked on - * @dsize: number of elements in @dst (including end marker) - * @src: new runlist to be inserted - * @ssize: number of elements in @src (excluding end marker) - * @loc: index in runlist @dst at which to split and insert @src - * - * Split the runlist @dst at @loc into two and insert @new in between the two - * fragments. No merging of runlists is necessary. Adjust the size of the - * holes either side. - * - * It is up to the caller to serialize access to the runlists @dst and @src. - * - * On success, return a pointer to the new, combined, runlist. Note, both - * runlists @dst and @src are deallocated before returning so you cannot use - * the pointers for anything any more. (Strictly speaking the returned runlist - * may be the same as @dst but this is irrelevant.) - * - * On error, return -errno. Both runlists are left unmodified. The following - * error codes are defined: - * -ENOMEM - Not enough memory to allocate runlist array. - * -EINVAL - Invalid parameters were passed in. - */ -static inline runlist_element *ntfs_rl_split(runlist_element *dst, int dsize, - runlist_element *src, int ssize, int loc) -{ - BUG_ON(!dst); - BUG_ON(!src); - - /* Space required: @dst size + @src size + one new hole. */ - dst = ntfs_rl_realloc(dst, dsize, dsize + ssize + 1); - if (IS_ERR(dst)) - return dst; - /* - * We are guaranteed to succeed from here so can start modifying the - * original runlists. - */ - - /* Move the tail of @dst out of the way, then copy in @src. */ - ntfs_rl_mm(dst, loc + 1 + ssize, loc, dsize - loc); - ntfs_rl_mc(dst, loc + 1, src, 0, ssize); - - /* Adjust the size of the holes either size of @src. */ - dst[loc].length = dst[loc+1].vcn - dst[loc].vcn; - dst[loc+ssize+1].vcn = dst[loc+ssize].vcn + dst[loc+ssize].length; - dst[loc+ssize+1].length = dst[loc+ssize+2].vcn - dst[loc+ssize+1].vcn; - - return dst; -} - -/** - * ntfs_runlists_merge - merge two runlists into one - * @drl: original runlist to be worked on - * @srl: new runlist to be merged into @drl - * - * First we sanity check the two runlists @srl and @drl to make sure that they - * are sensible and can be merged. The runlist @srl must be either after the - * runlist @drl or completely within a hole (or unmapped region) in @drl. - * - * It is up to the caller to serialize access to the runlists @drl and @srl. - * - * Merging of runlists is necessary in two cases: - * 1. When attribute lists are used and a further extent is being mapped. - * 2. When new clusters are allocated to fill a hole or extend a file. - * - * There are four possible ways @srl can be merged. It can: - * - be inserted at the beginning of a hole, - * - split the hole in two and be inserted between the two fragments, - * - be appended at the end of a hole, or it can - * - replace the whole hole. - * It can also be appended to the end of the runlist, which is just a variant - * of the insert case. - * - * On success, return a pointer to the new, combined, runlist. Note, both - * runlists @drl and @srl are deallocated before returning so you cannot use - * the pointers for anything any more. (Strictly speaking the returned runlist - * may be the same as @dst but this is irrelevant.) - * - * On error, return -errno. Both runlists are left unmodified. The following - * error codes are defined: - * -ENOMEM - Not enough memory to allocate runlist array. - * -EINVAL - Invalid parameters were passed in. - * -ERANGE - The runlists overlap and cannot be merged. - */ -runlist_element *ntfs_runlists_merge(runlist_element *drl, - runlist_element *srl) -{ - int di, si; /* Current index into @[ds]rl. */ - int sstart; /* First index with lcn > LCN_RL_NOT_MAPPED. */ - int dins; /* Index into @drl at which to insert @srl. */ - int dend, send; /* Last index into @[ds]rl. */ - int dfinal, sfinal; /* The last index into @[ds]rl with - lcn >= LCN_HOLE. */ - int marker = 0; - VCN marker_vcn = 0; - -#ifdef DEBUG - ntfs_debug("dst:"); - ntfs_debug_dump_runlist(drl); - ntfs_debug("src:"); - ntfs_debug_dump_runlist(srl); -#endif - - /* Check for silly calling... */ - if (unlikely(!srl)) - return drl; - if (IS_ERR(srl) || IS_ERR(drl)) - return ERR_PTR(-EINVAL); - - /* Check for the case where the first mapping is being done now. */ - if (unlikely(!drl)) { - drl = srl; - /* Complete the source runlist if necessary. */ - if (unlikely(drl[0].vcn)) { - /* Scan to the end of the source runlist. */ - for (dend = 0; likely(drl[dend].length); dend++) - ; - dend++; - drl = ntfs_rl_realloc(drl, dend, dend + 1); - if (IS_ERR(drl)) - return drl; - /* Insert start element at the front of the runlist. */ - ntfs_rl_mm(drl, 1, 0, dend); - drl[0].vcn = 0; - drl[0].lcn = LCN_RL_NOT_MAPPED; - drl[0].length = drl[1].vcn; - } - goto finished; - } - - si = di = 0; - - /* Skip any unmapped start element(s) in the source runlist. */ - while (srl[si].length && srl[si].lcn < LCN_HOLE) - si++; - - /* Can't have an entirely unmapped source runlist. */ - BUG_ON(!srl[si].length); - - /* Record the starting points. */ - sstart = si; - - /* - * Skip forward in @drl until we reach the position where @srl needs to - * be inserted. If we reach the end of @drl, @srl just needs to be - * appended to @drl. - */ - for (; drl[di].length; di++) { - if (drl[di].vcn + drl[di].length > srl[sstart].vcn) - break; - } - dins = di; - - /* Sanity check for illegal overlaps. */ - if ((drl[di].vcn == srl[si].vcn) && (drl[di].lcn >= 0) && - (srl[si].lcn >= 0)) { - ntfs_error(NULL, "Run lists overlap. Cannot merge!"); - return ERR_PTR(-ERANGE); - } - - /* Scan to the end of both runlists in order to know their sizes. */ - for (send = si; srl[send].length; send++) - ; - for (dend = di; drl[dend].length; dend++) - ; - - if (srl[send].lcn == LCN_ENOENT) - marker_vcn = srl[marker = send].vcn; - - /* Scan to the last element with lcn >= LCN_HOLE. */ - for (sfinal = send; sfinal >= 0 && srl[sfinal].lcn < LCN_HOLE; sfinal--) - ; - for (dfinal = dend; dfinal >= 0 && drl[dfinal].lcn < LCN_HOLE; dfinal--) - ; - - { - bool start; - bool finish; - int ds = dend + 1; /* Number of elements in drl & srl */ - int ss = sfinal - sstart + 1; - - start = ((drl[dins].lcn < LCN_RL_NOT_MAPPED) || /* End of file */ - (drl[dins].vcn == srl[sstart].vcn)); /* Start of hole */ - finish = ((drl[dins].lcn >= LCN_RL_NOT_MAPPED) && /* End of file */ - ((drl[dins].vcn + drl[dins].length) <= /* End of hole */ - (srl[send - 1].vcn + srl[send - 1].length))); - - /* Or we will lose an end marker. */ - if (finish && !drl[dins].length) - ss++; - if (marker && (drl[dins].vcn + drl[dins].length > srl[send - 1].vcn)) - finish = false; -#if 0 - ntfs_debug("dfinal = %i, dend = %i", dfinal, dend); - ntfs_debug("sstart = %i, sfinal = %i, send = %i", sstart, sfinal, send); - ntfs_debug("start = %i, finish = %i", start, finish); - ntfs_debug("ds = %i, ss = %i, dins = %i", ds, ss, dins); -#endif - if (start) { - if (finish) - drl = ntfs_rl_replace(drl, ds, srl + sstart, ss, dins); - else - drl = ntfs_rl_insert(drl, ds, srl + sstart, ss, dins); - } else { - if (finish) - drl = ntfs_rl_append(drl, ds, srl + sstart, ss, dins); - else - drl = ntfs_rl_split(drl, ds, srl + sstart, ss, dins); - } - if (IS_ERR(drl)) { - ntfs_error(NULL, "Merge failed."); - return drl; - } - ntfs_free(srl); - if (marker) { - ntfs_debug("Triggering marker code."); - for (ds = dend; drl[ds].length; ds++) - ; - /* We only need to care if @srl ended after @drl. */ - if (drl[ds].vcn <= marker_vcn) { - int slots = 0; - - if (drl[ds].vcn == marker_vcn) { - ntfs_debug("Old marker = 0x%llx, replacing " - "with LCN_ENOENT.", - (unsigned long long) - drl[ds].lcn); - drl[ds].lcn = LCN_ENOENT; - goto finished; - } - /* - * We need to create an unmapped runlist element in - * @drl or extend an existing one before adding the - * ENOENT terminator. - */ - if (drl[ds].lcn == LCN_ENOENT) { - ds--; - slots = 1; - } - if (drl[ds].lcn != LCN_RL_NOT_MAPPED) { - /* Add an unmapped runlist element. */ - if (!slots) { - drl = ntfs_rl_realloc_nofail(drl, ds, - ds + 2); - slots = 2; - } - ds++; - /* Need to set vcn if it isn't set already. */ - if (slots != 1) - drl[ds].vcn = drl[ds - 1].vcn + - drl[ds - 1].length; - drl[ds].lcn = LCN_RL_NOT_MAPPED; - /* We now used up a slot. */ - slots--; - } - drl[ds].length = marker_vcn - drl[ds].vcn; - /* Finally add the ENOENT terminator. */ - ds++; - if (!slots) - drl = ntfs_rl_realloc_nofail(drl, ds, ds + 1); - drl[ds].vcn = marker_vcn; - drl[ds].lcn = LCN_ENOENT; - drl[ds].length = (s64)0; - } - } - } - -finished: - /* The merge was completed successfully. */ - ntfs_debug("Merged runlist:"); - ntfs_debug_dump_runlist(drl); - return drl; -} - -/** - * ntfs_mapping_pairs_decompress - convert mapping pairs array to runlist - * @vol: ntfs volume on which the attribute resides - * @attr: attribute record whose mapping pairs array to decompress - * @old_rl: optional runlist in which to insert @attr's runlist - * - * It is up to the caller to serialize access to the runlist @old_rl. - * - * Decompress the attribute @attr's mapping pairs array into a runlist. On - * success, return the decompressed runlist. - * - * If @old_rl is not NULL, decompressed runlist is inserted into the - * appropriate place in @old_rl and the resultant, combined runlist is - * returned. The original @old_rl is deallocated. - * - * On error, return -errno. @old_rl is left unmodified in that case. - * - * The following error codes are defined: - * -ENOMEM - Not enough memory to allocate runlist array. - * -EIO - Corrupt runlist. - * -EINVAL - Invalid parameters were passed in. - * -ERANGE - The two runlists overlap. - * - * FIXME: For now we take the conceptionally simplest approach of creating the - * new runlist disregarding the already existing one and then splicing the - * two into one, if that is possible (we check for overlap and discard the new - * runlist if overlap present before returning ERR_PTR(-ERANGE)). - */ -runlist_element *ntfs_mapping_pairs_decompress(const ntfs_volume *vol, - const ATTR_RECORD *attr, runlist_element *old_rl) -{ - VCN vcn; /* Current vcn. */ - LCN lcn; /* Current lcn. */ - s64 deltaxcn; /* Change in [vl]cn. */ - runlist_element *rl; /* The output runlist. */ - u8 *buf; /* Current position in mapping pairs array. */ - u8 *attr_end; /* End of attribute. */ - int rlsize; /* Size of runlist buffer. */ - u16 rlpos; /* Current runlist position in units of - runlist_elements. */ - u8 b; /* Current byte offset in buf. */ - -#ifdef DEBUG - /* Make sure attr exists and is non-resident. */ - if (!attr || !attr->non_resident || sle64_to_cpu( - attr->data.non_resident.lowest_vcn) < (VCN)0) { - ntfs_error(vol->sb, "Invalid arguments."); - return ERR_PTR(-EINVAL); - } -#endif - /* Start at vcn = lowest_vcn and lcn 0. */ - vcn = sle64_to_cpu(attr->data.non_resident.lowest_vcn); - lcn = 0; - /* Get start of the mapping pairs array. */ - buf = (u8*)attr + le16_to_cpu( - attr->data.non_resident.mapping_pairs_offset); - attr_end = (u8*)attr + le32_to_cpu(attr->length); - if (unlikely(buf < (u8*)attr || buf > attr_end)) { - ntfs_error(vol->sb, "Corrupt attribute."); - return ERR_PTR(-EIO); - } - /* If the mapping pairs array is valid but empty, nothing to do. */ - if (!vcn && !*buf) - return old_rl; - /* Current position in runlist array. */ - rlpos = 0; - /* Allocate first page and set current runlist size to one page. */ - rl = ntfs_malloc_nofs(rlsize = PAGE_SIZE); - if (unlikely(!rl)) - return ERR_PTR(-ENOMEM); - /* Insert unmapped starting element if necessary. */ - if (vcn) { - rl->vcn = 0; - rl->lcn = LCN_RL_NOT_MAPPED; - rl->length = vcn; - rlpos++; - } - while (buf < attr_end && *buf) { - /* - * Allocate more memory if needed, including space for the - * not-mapped and terminator elements. ntfs_malloc_nofs() - * operates on whole pages only. - */ - if (((rlpos + 3) * sizeof(*old_rl)) > rlsize) { - runlist_element *rl2; - - rl2 = ntfs_malloc_nofs(rlsize + (int)PAGE_SIZE); - if (unlikely(!rl2)) { - ntfs_free(rl); - return ERR_PTR(-ENOMEM); - } - memcpy(rl2, rl, rlsize); - ntfs_free(rl); - rl = rl2; - rlsize += PAGE_SIZE; - } - /* Enter the current vcn into the current runlist element. */ - rl[rlpos].vcn = vcn; - /* - * Get the change in vcn, i.e. the run length in clusters. - * Doing it this way ensures that we signextend negative values. - * A negative run length doesn't make any sense, but hey, I - * didn't make up the NTFS specs and Windows NT4 treats the run - * length as a signed value so that's how it is... - */ - b = *buf & 0xf; - if (b) { - if (unlikely(buf + b > attr_end)) - goto io_error; - for (deltaxcn = (s8)buf[b--]; b; b--) - deltaxcn = (deltaxcn << 8) + buf[b]; - } else { /* The length entry is compulsory. */ - ntfs_error(vol->sb, "Missing length entry in mapping " - "pairs array."); - deltaxcn = (s64)-1; - } - /* - * Assume a negative length to indicate data corruption and - * hence clean-up and return NULL. - */ - if (unlikely(deltaxcn < 0)) { - ntfs_error(vol->sb, "Invalid length in mapping pairs " - "array."); - goto err_out; - } - /* - * Enter the current run length into the current runlist - * element. - */ - rl[rlpos].length = deltaxcn; - /* Increment the current vcn by the current run length. */ - vcn += deltaxcn; - /* - * There might be no lcn change at all, as is the case for - * sparse clusters on NTFS 3.0+, in which case we set the lcn - * to LCN_HOLE. - */ - if (!(*buf & 0xf0)) - rl[rlpos].lcn = LCN_HOLE; - else { - /* Get the lcn change which really can be negative. */ - u8 b2 = *buf & 0xf; - b = b2 + ((*buf >> 4) & 0xf); - if (buf + b > attr_end) - goto io_error; - for (deltaxcn = (s8)buf[b--]; b > b2; b--) - deltaxcn = (deltaxcn << 8) + buf[b]; - /* Change the current lcn to its new value. */ - lcn += deltaxcn; -#ifdef DEBUG - /* - * On NTFS 1.2-, apparently can have lcn == -1 to - * indicate a hole. But we haven't verified ourselves - * whether it is really the lcn or the deltaxcn that is - * -1. So if either is found give us a message so we - * can investigate it further! - */ - if (vol->major_ver < 3) { - if (unlikely(deltaxcn == (LCN)-1)) - ntfs_error(vol->sb, "lcn delta == -1"); - if (unlikely(lcn == (LCN)-1)) - ntfs_error(vol->sb, "lcn == -1"); - } -#endif - /* Check lcn is not below -1. */ - if (unlikely(lcn < (LCN)-1)) { - ntfs_error(vol->sb, "Invalid LCN < -1 in " - "mapping pairs array."); - goto err_out; - } - /* Enter the current lcn into the runlist element. */ - rl[rlpos].lcn = lcn; - } - /* Get to the next runlist element. */ - rlpos++; - /* Increment the buffer position to the next mapping pair. */ - buf += (*buf & 0xf) + ((*buf >> 4) & 0xf) + 1; - } - if (unlikely(buf >= attr_end)) - goto io_error; - /* - * If there is a highest_vcn specified, it must be equal to the final - * vcn in the runlist - 1, or something has gone badly wrong. - */ - deltaxcn = sle64_to_cpu(attr->data.non_resident.highest_vcn); - if (unlikely(deltaxcn && vcn - 1 != deltaxcn)) { -mpa_err: - ntfs_error(vol->sb, "Corrupt mapping pairs array in " - "non-resident attribute."); - goto err_out; - } - /* Setup not mapped runlist element if this is the base extent. */ - if (!attr->data.non_resident.lowest_vcn) { - VCN max_cluster; - - max_cluster = ((sle64_to_cpu( - attr->data.non_resident.allocated_size) + - vol->cluster_size - 1) >> - vol->cluster_size_bits) - 1; - /* - * A highest_vcn of zero means this is a single extent - * attribute so simply terminate the runlist with LCN_ENOENT). - */ - if (deltaxcn) { - /* - * If there is a difference between the highest_vcn and - * the highest cluster, the runlist is either corrupt - * or, more likely, there are more extents following - * this one. - */ - if (deltaxcn < max_cluster) { - ntfs_debug("More extents to follow; deltaxcn " - "= 0x%llx, max_cluster = " - "0x%llx", - (unsigned long long)deltaxcn, - (unsigned long long) - max_cluster); - rl[rlpos].vcn = vcn; - vcn += rl[rlpos].length = max_cluster - - deltaxcn; - rl[rlpos].lcn = LCN_RL_NOT_MAPPED; - rlpos++; - } else if (unlikely(deltaxcn > max_cluster)) { - ntfs_error(vol->sb, "Corrupt attribute. " - "deltaxcn = 0x%llx, " - "max_cluster = 0x%llx", - (unsigned long long)deltaxcn, - (unsigned long long) - max_cluster); - goto mpa_err; - } - } - rl[rlpos].lcn = LCN_ENOENT; - } else /* Not the base extent. There may be more extents to follow. */ - rl[rlpos].lcn = LCN_RL_NOT_MAPPED; - - /* Setup terminating runlist element. */ - rl[rlpos].vcn = vcn; - rl[rlpos].length = (s64)0; - /* If no existing runlist was specified, we are done. */ - if (!old_rl) { - ntfs_debug("Mapping pairs array successfully decompressed:"); - ntfs_debug_dump_runlist(rl); - return rl; - } - /* Now combine the new and old runlists checking for overlaps. */ - old_rl = ntfs_runlists_merge(old_rl, rl); - if (!IS_ERR(old_rl)) - return old_rl; - ntfs_free(rl); - ntfs_error(vol->sb, "Failed to merge runlists."); - return old_rl; -io_error: - ntfs_error(vol->sb, "Corrupt attribute."); -err_out: - ntfs_free(rl); - return ERR_PTR(-EIO); -} - -/** - * ntfs_rl_vcn_to_lcn - convert a vcn into a lcn given a runlist - * @rl: runlist to use for conversion - * @vcn: vcn to convert - * - * Convert the virtual cluster number @vcn of an attribute into a logical - * cluster number (lcn) of a device using the runlist @rl to map vcns to their - * corresponding lcns. - * - * It is up to the caller to serialize access to the runlist @rl. - * - * Since lcns must be >= 0, we use negative return codes with special meaning: - * - * Return code Meaning / Description - * ================================================== - * LCN_HOLE Hole / not allocated on disk. - * LCN_RL_NOT_MAPPED This is part of the runlist which has not been - * inserted into the runlist yet. - * LCN_ENOENT There is no such vcn in the attribute. - * - * Locking: - The caller must have locked the runlist (for reading or writing). - * - This function does not touch the lock, nor does it modify the - * runlist. - */ -LCN ntfs_rl_vcn_to_lcn(const runlist_element *rl, const VCN vcn) -{ - int i; - - BUG_ON(vcn < 0); - /* - * If rl is NULL, assume that we have found an unmapped runlist. The - * caller can then attempt to map it and fail appropriately if - * necessary. - */ - if (unlikely(!rl)) - return LCN_RL_NOT_MAPPED; - - /* Catch out of lower bounds vcn. */ - if (unlikely(vcn < rl[0].vcn)) - return LCN_ENOENT; - - for (i = 0; likely(rl[i].length); i++) { - if (unlikely(vcn < rl[i+1].vcn)) { - if (likely(rl[i].lcn >= (LCN)0)) - return rl[i].lcn + (vcn - rl[i].vcn); - return rl[i].lcn; - } - } - /* - * The terminator element is setup to the correct value, i.e. one of - * LCN_HOLE, LCN_RL_NOT_MAPPED, or LCN_ENOENT. - */ - if (likely(rl[i].lcn < (LCN)0)) - return rl[i].lcn; - /* Just in case... We could replace this with BUG() some day. */ - return LCN_ENOENT; -} - -#ifdef NTFS_RW - -/** - * ntfs_rl_find_vcn_nolock - find a vcn in a runlist - * @rl: runlist to search - * @vcn: vcn to find - * - * Find the virtual cluster number @vcn in the runlist @rl and return the - * address of the runlist element containing the @vcn on success. - * - * Return NULL if @rl is NULL or @vcn is in an unmapped part/out of bounds of - * the runlist. - * - * Locking: The runlist must be locked on entry. - */ -runlist_element *ntfs_rl_find_vcn_nolock(runlist_element *rl, const VCN vcn) -{ - BUG_ON(vcn < 0); - if (unlikely(!rl || vcn < rl[0].vcn)) - return NULL; - while (likely(rl->length)) { - if (unlikely(vcn < rl[1].vcn)) { - if (likely(rl->lcn >= LCN_HOLE)) - return rl; - return NULL; - } - rl++; - } - if (likely(rl->lcn == LCN_ENOENT)) - return rl; - return NULL; -} - -/** - * ntfs_get_nr_significant_bytes - get number of bytes needed to store a number - * @n: number for which to get the number of bytes for - * - * Return the number of bytes required to store @n unambiguously as - * a signed number. - * - * This is used in the context of the mapping pairs array to determine how - * many bytes will be needed in the array to store a given logical cluster - * number (lcn) or a specific run length. - * - * Return the number of bytes written. This function cannot fail. - */ -static inline int ntfs_get_nr_significant_bytes(const s64 n) -{ - s64 l = n; - int i; - s8 j; - - i = 0; - do { - l >>= 8; - i++; - } while (l != 0 && l != -1); - j = (n >> 8 * (i - 1)) & 0xff; - /* If the sign bit is wrong, we need an extra byte. */ - if ((n < 0 && j >= 0) || (n > 0 && j < 0)) - i++; - return i; -} - -/** - * ntfs_get_size_for_mapping_pairs - get bytes needed for mapping pairs array - * @vol: ntfs volume (needed for the ntfs version) - * @rl: locked runlist to determine the size of the mapping pairs of - * @first_vcn: first vcn which to include in the mapping pairs array - * @last_vcn: last vcn which to include in the mapping pairs array - * - * Walk the locked runlist @rl and calculate the size in bytes of the mapping - * pairs array corresponding to the runlist @rl, starting at vcn @first_vcn and - * finishing with vcn @last_vcn. - * - * A @last_vcn of -1 means end of runlist and in that case the size of the - * mapping pairs array corresponding to the runlist starting at vcn @first_vcn - * and finishing at the end of the runlist is determined. - * - * This for example allows us to allocate a buffer of the right size when - * building the mapping pairs array. - * - * If @rl is NULL, just return 1 (for the single terminator byte). - * - * Return the calculated size in bytes on success. On error, return -errno. - * The following error codes are defined: - * -EINVAL - Run list contains unmapped elements. Make sure to only pass - * fully mapped runlists to this function. - * -EIO - The runlist is corrupt. - * - * Locking: @rl must be locked on entry (either for reading or writing), it - * remains locked throughout, and is left locked upon return. - */ -int ntfs_get_size_for_mapping_pairs(const ntfs_volume *vol, - const runlist_element *rl, const VCN first_vcn, - const VCN last_vcn) -{ - LCN prev_lcn; - int rls; - bool the_end = false; - - BUG_ON(first_vcn < 0); - BUG_ON(last_vcn < -1); - BUG_ON(last_vcn >= 0 && first_vcn > last_vcn); - if (!rl) { - BUG_ON(first_vcn); - BUG_ON(last_vcn > 0); - return 1; - } - /* Skip to runlist element containing @first_vcn. */ - while (rl->length && first_vcn >= rl[1].vcn) - rl++; - if (unlikely((!rl->length && first_vcn > rl->vcn) || - first_vcn < rl->vcn)) - return -EINVAL; - prev_lcn = 0; - /* Always need the termining zero byte. */ - rls = 1; - /* Do the first partial run if present. */ - if (first_vcn > rl->vcn) { - s64 delta, length = rl->length; - - /* We know rl->length != 0 already. */ - if (unlikely(length < 0 || rl->lcn < LCN_HOLE)) - goto err_out; - /* - * If @stop_vcn is given and finishes inside this run, cap the - * run length. - */ - if (unlikely(last_vcn >= 0 && rl[1].vcn > last_vcn)) { - s64 s1 = last_vcn + 1; - if (unlikely(rl[1].vcn > s1)) - length = s1 - rl->vcn; - the_end = true; - } - delta = first_vcn - rl->vcn; - /* Header byte + length. */ - rls += 1 + ntfs_get_nr_significant_bytes(length - delta); - /* - * If the logical cluster number (lcn) denotes a hole and we - * are on NTFS 3.0+, we don't store it at all, i.e. we need - * zero space. On earlier NTFS versions we just store the lcn. - * Note: this assumes that on NTFS 1.2-, holes are stored with - * an lcn of -1 and not a delta_lcn of -1 (unless both are -1). - */ - if (likely(rl->lcn >= 0 || vol->major_ver < 3)) { - prev_lcn = rl->lcn; - if (likely(rl->lcn >= 0)) - prev_lcn += delta; - /* Change in lcn. */ - rls += ntfs_get_nr_significant_bytes(prev_lcn); - } - /* Go to next runlist element. */ - rl++; - } - /* Do the full runs. */ - for (; rl->length && !the_end; rl++) { - s64 length = rl->length; - - if (unlikely(length < 0 || rl->lcn < LCN_HOLE)) - goto err_out; - /* - * If @stop_vcn is given and finishes inside this run, cap the - * run length. - */ - if (unlikely(last_vcn >= 0 && rl[1].vcn > last_vcn)) { - s64 s1 = last_vcn + 1; - if (unlikely(rl[1].vcn > s1)) - length = s1 - rl->vcn; - the_end = true; - } - /* Header byte + length. */ - rls += 1 + ntfs_get_nr_significant_bytes(length); - /* - * If the logical cluster number (lcn) denotes a hole and we - * are on NTFS 3.0+, we don't store it at all, i.e. we need - * zero space. On earlier NTFS versions we just store the lcn. - * Note: this assumes that on NTFS 1.2-, holes are stored with - * an lcn of -1 and not a delta_lcn of -1 (unless both are -1). - */ - if (likely(rl->lcn >= 0 || vol->major_ver < 3)) { - /* Change in lcn. */ - rls += ntfs_get_nr_significant_bytes(rl->lcn - - prev_lcn); - prev_lcn = rl->lcn; - } - } - return rls; -err_out: - if (rl->lcn == LCN_RL_NOT_MAPPED) - rls = -EINVAL; - else - rls = -EIO; - return rls; -} - -/** - * ntfs_write_significant_bytes - write the significant bytes of a number - * @dst: destination buffer to write to - * @dst_max: pointer to last byte of destination buffer for bounds checking - * @n: number whose significant bytes to write - * - * Store in @dst, the minimum bytes of the number @n which are required to - * identify @n unambiguously as a signed number, taking care not to exceed - * @dest_max, the maximum position within @dst to which we are allowed to - * write. - * - * This is used when building the mapping pairs array of a runlist to compress - * a given logical cluster number (lcn) or a specific run length to the minimum - * size possible. - * - * Return the number of bytes written on success. On error, i.e. the - * destination buffer @dst is too small, return -ENOSPC. - */ -static inline int ntfs_write_significant_bytes(s8 *dst, const s8 *dst_max, - const s64 n) -{ - s64 l = n; - int i; - s8 j; - - i = 0; - do { - if (unlikely(dst > dst_max)) - goto err_out; - *dst++ = l & 0xffll; - l >>= 8; - i++; - } while (l != 0 && l != -1); - j = (n >> 8 * (i - 1)) & 0xff; - /* If the sign bit is wrong, we need an extra byte. */ - if (n < 0 && j >= 0) { - if (unlikely(dst > dst_max)) - goto err_out; - i++; - *dst = (s8)-1; - } else if (n > 0 && j < 0) { - if (unlikely(dst > dst_max)) - goto err_out; - i++; - *dst = (s8)0; - } - return i; -err_out: - return -ENOSPC; -} - -/** - * ntfs_mapping_pairs_build - build the mapping pairs array from a runlist - * @vol: ntfs volume (needed for the ntfs version) - * @dst: destination buffer to which to write the mapping pairs array - * @dst_len: size of destination buffer @dst in bytes - * @rl: locked runlist for which to build the mapping pairs array - * @first_vcn: first vcn which to include in the mapping pairs array - * @last_vcn: last vcn which to include in the mapping pairs array - * @stop_vcn: first vcn outside destination buffer on success or -ENOSPC - * - * Create the mapping pairs array from the locked runlist @rl, starting at vcn - * @first_vcn and finishing with vcn @last_vcn and save the array in @dst. - * @dst_len is the size of @dst in bytes and it should be at least equal to the - * value obtained by calling ntfs_get_size_for_mapping_pairs(). - * - * A @last_vcn of -1 means end of runlist and in that case the mapping pairs - * array corresponding to the runlist starting at vcn @first_vcn and finishing - * at the end of the runlist is created. - * - * If @rl is NULL, just write a single terminator byte to @dst. - * - * On success or -ENOSPC error, if @stop_vcn is not NULL, *@stop_vcn is set to - * the first vcn outside the destination buffer. Note that on error, @dst has - * been filled with all the mapping pairs that will fit, thus it can be treated - * as partial success, in that a new attribute extent needs to be created or - * the next extent has to be used and the mapping pairs build has to be - * continued with @first_vcn set to *@stop_vcn. - * - * Return 0 on success and -errno on error. The following error codes are - * defined: - * -EINVAL - Run list contains unmapped elements. Make sure to only pass - * fully mapped runlists to this function. - * -EIO - The runlist is corrupt. - * -ENOSPC - The destination buffer is too small. - * - * Locking: @rl must be locked on entry (either for reading or writing), it - * remains locked throughout, and is left locked upon return. - */ -int ntfs_mapping_pairs_build(const ntfs_volume *vol, s8 *dst, - const int dst_len, const runlist_element *rl, - const VCN first_vcn, const VCN last_vcn, VCN *const stop_vcn) -{ - LCN prev_lcn; - s8 *dst_max, *dst_next; - int err = -ENOSPC; - bool the_end = false; - s8 len_len, lcn_len; - - BUG_ON(first_vcn < 0); - BUG_ON(last_vcn < -1); - BUG_ON(last_vcn >= 0 && first_vcn > last_vcn); - BUG_ON(dst_len < 1); - if (!rl) { - BUG_ON(first_vcn); - BUG_ON(last_vcn > 0); - if (stop_vcn) - *stop_vcn = 0; - /* Terminator byte. */ - *dst = 0; - return 0; - } - /* Skip to runlist element containing @first_vcn. */ - while (rl->length && first_vcn >= rl[1].vcn) - rl++; - if (unlikely((!rl->length && first_vcn > rl->vcn) || - first_vcn < rl->vcn)) - return -EINVAL; - /* - * @dst_max is used for bounds checking in - * ntfs_write_significant_bytes(). - */ - dst_max = dst + dst_len - 1; - prev_lcn = 0; - /* Do the first partial run if present. */ - if (first_vcn > rl->vcn) { - s64 delta, length = rl->length; - - /* We know rl->length != 0 already. */ - if (unlikely(length < 0 || rl->lcn < LCN_HOLE)) - goto err_out; - /* - * If @stop_vcn is given and finishes inside this run, cap the - * run length. - */ - if (unlikely(last_vcn >= 0 && rl[1].vcn > last_vcn)) { - s64 s1 = last_vcn + 1; - if (unlikely(rl[1].vcn > s1)) - length = s1 - rl->vcn; - the_end = true; - } - delta = first_vcn - rl->vcn; - /* Write length. */ - len_len = ntfs_write_significant_bytes(dst + 1, dst_max, - length - delta); - if (unlikely(len_len < 0)) - goto size_err; - /* - * If the logical cluster number (lcn) denotes a hole and we - * are on NTFS 3.0+, we don't store it at all, i.e. we need - * zero space. On earlier NTFS versions we just write the lcn - * change. FIXME: Do we need to write the lcn change or just - * the lcn in that case? Not sure as I have never seen this - * case on NT4. - We assume that we just need to write the lcn - * change until someone tells us otherwise... (AIA) - */ - if (likely(rl->lcn >= 0 || vol->major_ver < 3)) { - prev_lcn = rl->lcn; - if (likely(rl->lcn >= 0)) - prev_lcn += delta; - /* Write change in lcn. */ - lcn_len = ntfs_write_significant_bytes(dst + 1 + - len_len, dst_max, prev_lcn); - if (unlikely(lcn_len < 0)) - goto size_err; - } else - lcn_len = 0; - dst_next = dst + len_len + lcn_len + 1; - if (unlikely(dst_next > dst_max)) - goto size_err; - /* Update header byte. */ - *dst = lcn_len << 4 | len_len; - /* Position at next mapping pairs array element. */ - dst = dst_next; - /* Go to next runlist element. */ - rl++; - } - /* Do the full runs. */ - for (; rl->length && !the_end; rl++) { - s64 length = rl->length; - - if (unlikely(length < 0 || rl->lcn < LCN_HOLE)) - goto err_out; - /* - * If @stop_vcn is given and finishes inside this run, cap the - * run length. - */ - if (unlikely(last_vcn >= 0 && rl[1].vcn > last_vcn)) { - s64 s1 = last_vcn + 1; - if (unlikely(rl[1].vcn > s1)) - length = s1 - rl->vcn; - the_end = true; - } - /* Write length. */ - len_len = ntfs_write_significant_bytes(dst + 1, dst_max, - length); - if (unlikely(len_len < 0)) - goto size_err; - /* - * If the logical cluster number (lcn) denotes a hole and we - * are on NTFS 3.0+, we don't store it at all, i.e. we need - * zero space. On earlier NTFS versions we just write the lcn - * change. FIXME: Do we need to write the lcn change or just - * the lcn in that case? Not sure as I have never seen this - * case on NT4. - We assume that we just need to write the lcn - * change until someone tells us otherwise... (AIA) - */ - if (likely(rl->lcn >= 0 || vol->major_ver < 3)) { - /* Write change in lcn. */ - lcn_len = ntfs_write_significant_bytes(dst + 1 + - len_len, dst_max, rl->lcn - prev_lcn); - if (unlikely(lcn_len < 0)) - goto size_err; - prev_lcn = rl->lcn; - } else - lcn_len = 0; - dst_next = dst + len_len + lcn_len + 1; - if (unlikely(dst_next > dst_max)) - goto size_err; - /* Update header byte. */ - *dst = lcn_len << 4 | len_len; - /* Position at next mapping pairs array element. */ - dst = dst_next; - } - /* Success. */ - err = 0; -size_err: - /* Set stop vcn. */ - if (stop_vcn) - *stop_vcn = rl->vcn; - /* Add terminator byte. */ - *dst = 0; - return err; -err_out: - if (rl->lcn == LCN_RL_NOT_MAPPED) - err = -EINVAL; - else - err = -EIO; - return err; -} - -/** - * ntfs_rl_truncate_nolock - truncate a runlist starting at a specified vcn - * @vol: ntfs volume (needed for error output) - * @runlist: runlist to truncate - * @new_length: the new length of the runlist in VCNs - * - * Truncate the runlist described by @runlist as well as the memory buffer - * holding the runlist elements to a length of @new_length VCNs. - * - * If @new_length lies within the runlist, the runlist elements with VCNs of - * @new_length and above are discarded. As a special case if @new_length is - * zero, the runlist is discarded and set to NULL. - * - * If @new_length lies beyond the runlist, a sparse runlist element is added to - * the end of the runlist @runlist or if the last runlist element is a sparse - * one already, this is extended. - * - * Note, no checking is done for unmapped runlist elements. It is assumed that - * the caller has mapped any elements that need to be mapped already. - * - * Return 0 on success and -errno on error. - * - * Locking: The caller must hold @runlist->lock for writing. - */ -int ntfs_rl_truncate_nolock(const ntfs_volume *vol, runlist *const runlist, - const s64 new_length) -{ - runlist_element *rl; - int old_size; - - ntfs_debug("Entering for new_length 0x%llx.", (long long)new_length); - BUG_ON(!runlist); - BUG_ON(new_length < 0); - rl = runlist->rl; - if (!new_length) { - ntfs_debug("Freeing runlist."); - runlist->rl = NULL; - if (rl) - ntfs_free(rl); - return 0; - } - if (unlikely(!rl)) { - /* - * Create a runlist consisting of a sparse runlist element of - * length @new_length followed by a terminator runlist element. - */ - rl = ntfs_malloc_nofs(PAGE_SIZE); - if (unlikely(!rl)) { - ntfs_error(vol->sb, "Not enough memory to allocate " - "runlist element buffer."); - return -ENOMEM; - } - runlist->rl = rl; - rl[1].length = rl->vcn = 0; - rl->lcn = LCN_HOLE; - rl[1].vcn = rl->length = new_length; - rl[1].lcn = LCN_ENOENT; - return 0; - } - BUG_ON(new_length < rl->vcn); - /* Find @new_length in the runlist. */ - while (likely(rl->length && new_length >= rl[1].vcn)) - rl++; - /* - * If not at the end of the runlist we need to shrink it. - * If at the end of the runlist we need to expand it. - */ - if (rl->length) { - runlist_element *trl; - bool is_end; - - ntfs_debug("Shrinking runlist."); - /* Determine the runlist size. */ - trl = rl + 1; - while (likely(trl->length)) - trl++; - old_size = trl - runlist->rl + 1; - /* Truncate the run. */ - rl->length = new_length - rl->vcn; - /* - * If a run was partially truncated, make the following runlist - * element a terminator. - */ - is_end = false; - if (rl->length) { - rl++; - if (!rl->length) - is_end = true; - rl->vcn = new_length; - rl->length = 0; - } - rl->lcn = LCN_ENOENT; - /* Reallocate memory if necessary. */ - if (!is_end) { - int new_size = rl - runlist->rl + 1; - rl = ntfs_rl_realloc(runlist->rl, old_size, new_size); - if (IS_ERR(rl)) - ntfs_warning(vol->sb, "Failed to shrink " - "runlist buffer. This just " - "wastes a bit of memory " - "temporarily so we ignore it " - "and return success."); - else - runlist->rl = rl; - } - } else if (likely(/* !rl->length && */ new_length > rl->vcn)) { - ntfs_debug("Expanding runlist."); - /* - * If there is a previous runlist element and it is a sparse - * one, extend it. Otherwise need to add a new, sparse runlist - * element. - */ - if ((rl > runlist->rl) && ((rl - 1)->lcn == LCN_HOLE)) - (rl - 1)->length = new_length - (rl - 1)->vcn; - else { - /* Determine the runlist size. */ - old_size = rl - runlist->rl + 1; - /* Reallocate memory if necessary. */ - rl = ntfs_rl_realloc(runlist->rl, old_size, - old_size + 1); - if (IS_ERR(rl)) { - ntfs_error(vol->sb, "Failed to expand runlist " - "buffer, aborting."); - return PTR_ERR(rl); - } - runlist->rl = rl; - /* - * Set @rl to the same runlist element in the new - * runlist as before in the old runlist. - */ - rl += old_size - 1; - /* Add a new, sparse runlist element. */ - rl->lcn = LCN_HOLE; - rl->length = new_length - rl->vcn; - /* Add a new terminator runlist element. */ - rl++; - rl->length = 0; - } - rl->vcn = new_length; - rl->lcn = LCN_ENOENT; - } else /* if (unlikely(!rl->length && new_length == rl->vcn)) */ { - /* Runlist already has same size as requested. */ - rl->lcn = LCN_ENOENT; - } - ntfs_debug("Done."); - return 0; -} - -/** - * ntfs_rl_punch_nolock - punch a hole into a runlist - * @vol: ntfs volume (needed for error output) - * @runlist: runlist to punch a hole into - * @start: starting VCN of the hole to be created - * @length: size of the hole to be created in units of clusters - * - * Punch a hole into the runlist @runlist starting at VCN @start and of size - * @length clusters. - * - * Return 0 on success and -errno on error, in which case @runlist has not been - * modified. - * - * If @start and/or @start + @length are outside the runlist return error code - * -ENOENT. - * - * If the runlist contains unmapped or error elements between @start and @start - * + @length return error code -EINVAL. - * - * Locking: The caller must hold @runlist->lock for writing. - */ -int ntfs_rl_punch_nolock(const ntfs_volume *vol, runlist *const runlist, - const VCN start, const s64 length) -{ - const VCN end = start + length; - s64 delta; - runlist_element *rl, *rl_end, *rl_real_end, *trl; - int old_size; - bool lcn_fixup = false; - - ntfs_debug("Entering for start 0x%llx, length 0x%llx.", - (long long)start, (long long)length); - BUG_ON(!runlist); - BUG_ON(start < 0); - BUG_ON(length < 0); - BUG_ON(end < 0); - rl = runlist->rl; - if (unlikely(!rl)) { - if (likely(!start && !length)) - return 0; - return -ENOENT; - } - /* Find @start in the runlist. */ - while (likely(rl->length && start >= rl[1].vcn)) - rl++; - rl_end = rl; - /* Find @end in the runlist. */ - while (likely(rl_end->length && end >= rl_end[1].vcn)) { - /* Verify there are no unmapped or error elements. */ - if (unlikely(rl_end->lcn < LCN_HOLE)) - return -EINVAL; - rl_end++; - } - /* Check the last element. */ - if (unlikely(rl_end->length && rl_end->lcn < LCN_HOLE)) - return -EINVAL; - /* This covers @start being out of bounds, too. */ - if (!rl_end->length && end > rl_end->vcn) - return -ENOENT; - if (!length) - return 0; - if (!rl->length) - return -ENOENT; - rl_real_end = rl_end; - /* Determine the runlist size. */ - while (likely(rl_real_end->length)) - rl_real_end++; - old_size = rl_real_end - runlist->rl + 1; - /* If @start is in a hole simply extend the hole. */ - if (rl->lcn == LCN_HOLE) { - /* - * If both @start and @end are in the same sparse run, we are - * done. - */ - if (end <= rl[1].vcn) { - ntfs_debug("Done (requested hole is already sparse)."); - return 0; - } -extend_hole: - /* Extend the hole. */ - rl->length = end - rl->vcn; - /* If @end is in a hole, merge it with the current one. */ - if (rl_end->lcn == LCN_HOLE) { - rl_end++; - rl->length = rl_end->vcn - rl->vcn; - } - /* We have done the hole. Now deal with the remaining tail. */ - rl++; - /* Cut out all runlist elements up to @end. */ - if (rl < rl_end) - memmove(rl, rl_end, (rl_real_end - rl_end + 1) * - sizeof(*rl)); - /* Adjust the beginning of the tail if necessary. */ - if (end > rl->vcn) { - delta = end - rl->vcn; - rl->vcn = end; - rl->length -= delta; - /* Only adjust the lcn if it is real. */ - if (rl->lcn >= 0) - rl->lcn += delta; - } -shrink_allocation: - /* Reallocate memory if the allocation changed. */ - if (rl < rl_end) { - rl = ntfs_rl_realloc(runlist->rl, old_size, - old_size - (rl_end - rl)); - if (IS_ERR(rl)) - ntfs_warning(vol->sb, "Failed to shrink " - "runlist buffer. This just " - "wastes a bit of memory " - "temporarily so we ignore it " - "and return success."); - else - runlist->rl = rl; - } - ntfs_debug("Done (extend hole)."); - return 0; - } - /* - * If @start is at the beginning of a run things are easier as there is - * no need to split the first run. - */ - if (start == rl->vcn) { - /* - * @start is at the beginning of a run. - * - * If the previous run is sparse, extend its hole. - * - * If @end is not in the same run, switch the run to be sparse - * and extend the newly created hole. - * - * Thus both of these cases reduce the problem to the above - * case of "@start is in a hole". - */ - if (rl > runlist->rl && (rl - 1)->lcn == LCN_HOLE) { - rl--; - goto extend_hole; - } - if (end >= rl[1].vcn) { - rl->lcn = LCN_HOLE; - goto extend_hole; - } - /* - * The final case is when @end is in the same run as @start. - * For this need to split the run into two. One run for the - * sparse region between the beginning of the old run, i.e. - * @start, and @end and one for the remaining non-sparse - * region, i.e. between @end and the end of the old run. - */ - trl = ntfs_rl_realloc(runlist->rl, old_size, old_size + 1); - if (IS_ERR(trl)) - goto enomem_out; - old_size++; - if (runlist->rl != trl) { - rl = trl + (rl - runlist->rl); - rl_end = trl + (rl_end - runlist->rl); - rl_real_end = trl + (rl_real_end - runlist->rl); - runlist->rl = trl; - } -split_end: - /* Shift all the runs up by one. */ - memmove(rl + 1, rl, (rl_real_end - rl + 1) * sizeof(*rl)); - /* Finally, setup the two split runs. */ - rl->lcn = LCN_HOLE; - rl->length = length; - rl++; - rl->vcn += length; - /* Only adjust the lcn if it is real. */ - if (rl->lcn >= 0 || lcn_fixup) - rl->lcn += length; - rl->length -= length; - ntfs_debug("Done (split one)."); - return 0; - } - /* - * @start is neither in a hole nor at the beginning of a run. - * - * If @end is in a hole, things are easier as simply truncating the run - * @start is in to end at @start - 1, deleting all runs after that up - * to @end, and finally extending the beginning of the run @end is in - * to be @start is all that is needed. - */ - if (rl_end->lcn == LCN_HOLE) { - /* Truncate the run containing @start. */ - rl->length = start - rl->vcn; - rl++; - /* Cut out all runlist elements up to @end. */ - if (rl < rl_end) - memmove(rl, rl_end, (rl_real_end - rl_end + 1) * - sizeof(*rl)); - /* Extend the beginning of the run @end is in to be @start. */ - rl->vcn = start; - rl->length = rl[1].vcn - start; - goto shrink_allocation; - } - /* - * If @end is not in a hole there are still two cases to distinguish. - * Either @end is or is not in the same run as @start. - * - * The second case is easier as it can be reduced to an already solved - * problem by truncating the run @start is in to end at @start - 1. - * Then, if @end is in the next run need to split the run into a sparse - * run followed by a non-sparse run (already covered above) and if @end - * is not in the next run switching it to be sparse, again reduces the - * problem to the already covered case of "@start is in a hole". - */ - if (end >= rl[1].vcn) { - /* - * If @end is not in the next run, reduce the problem to the - * case of "@start is in a hole". - */ - if (rl[1].length && end >= rl[2].vcn) { - /* Truncate the run containing @start. */ - rl->length = start - rl->vcn; - rl++; - rl->vcn = start; - rl->lcn = LCN_HOLE; - goto extend_hole; - } - trl = ntfs_rl_realloc(runlist->rl, old_size, old_size + 1); - if (IS_ERR(trl)) - goto enomem_out; - old_size++; - if (runlist->rl != trl) { - rl = trl + (rl - runlist->rl); - rl_end = trl + (rl_end - runlist->rl); - rl_real_end = trl + (rl_real_end - runlist->rl); - runlist->rl = trl; - } - /* Truncate the run containing @start. */ - rl->length = start - rl->vcn; - rl++; - /* - * @end is in the next run, reduce the problem to the case - * where "@start is at the beginning of a run and @end is in - * the same run as @start". - */ - delta = rl->vcn - start; - rl->vcn = start; - if (rl->lcn >= 0) { - rl->lcn -= delta; - /* Need this in case the lcn just became negative. */ - lcn_fixup = true; - } - rl->length += delta; - goto split_end; - } - /* - * The first case from above, i.e. @end is in the same run as @start. - * We need to split the run into three. One run for the non-sparse - * region between the beginning of the old run and @start, one for the - * sparse region between @start and @end, and one for the remaining - * non-sparse region, i.e. between @end and the end of the old run. - */ - trl = ntfs_rl_realloc(runlist->rl, old_size, old_size + 2); - if (IS_ERR(trl)) - goto enomem_out; - old_size += 2; - if (runlist->rl != trl) { - rl = trl + (rl - runlist->rl); - rl_end = trl + (rl_end - runlist->rl); - rl_real_end = trl + (rl_real_end - runlist->rl); - runlist->rl = trl; - } - /* Shift all the runs up by two. */ - memmove(rl + 2, rl, (rl_real_end - rl + 1) * sizeof(*rl)); - /* Finally, setup the three split runs. */ - rl->length = start - rl->vcn; - rl++; - rl->vcn = start; - rl->lcn = LCN_HOLE; - rl->length = length; - rl++; - delta = end - rl->vcn; - rl->vcn = end; - rl->lcn += delta; - rl->length -= delta; - ntfs_debug("Done (split both)."); - return 0; -enomem_out: - ntfs_error(vol->sb, "Not enough memory to extend runlist buffer."); - return -ENOMEM; -} - -#endif /* NTFS_RW */ diff --git a/fs/ntfs/runlist.h b/fs/ntfs/runlist.h deleted file mode 100644 index 38de0a375f59..000000000000 --- a/fs/ntfs/runlist.h +++ /dev/null @@ -1,88 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * runlist.h - Defines for runlist handling in NTFS Linux kernel driver. - * Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2005 Anton Altaparmakov - * Copyright (c) 2002 Richard Russon - */ - -#ifndef _LINUX_NTFS_RUNLIST_H -#define _LINUX_NTFS_RUNLIST_H - -#include "types.h" -#include "layout.h" -#include "volume.h" - -/** - * runlist_element - in memory vcn to lcn mapping array element - * @vcn: starting vcn of the current array element - * @lcn: starting lcn of the current array element - * @length: length in clusters of the current array element - * - * The last vcn (in fact the last vcn + 1) is reached when length == 0. - * - * When lcn == -1 this means that the count vcns starting at vcn are not - * physically allocated (i.e. this is a hole / data is sparse). - */ -typedef struct { /* In memory vcn to lcn mapping structure element. */ - VCN vcn; /* vcn = Starting virtual cluster number. */ - LCN lcn; /* lcn = Starting logical cluster number. */ - s64 length; /* Run length in clusters. */ -} runlist_element; - -/** - * runlist - in memory vcn to lcn mapping array including a read/write lock - * @rl: pointer to an array of runlist elements - * @lock: read/write spinlock for serializing access to @rl - * - */ -typedef struct { - runlist_element *rl; - struct rw_semaphore lock; -} runlist; - -static inline void ntfs_init_runlist(runlist *rl) -{ - rl->rl = NULL; - init_rwsem(&rl->lock); -} - -typedef enum { - LCN_HOLE = -1, /* Keep this as highest value or die! */ - LCN_RL_NOT_MAPPED = -2, - LCN_ENOENT = -3, - LCN_ENOMEM = -4, - LCN_EIO = -5, -} LCN_SPECIAL_VALUES; - -extern runlist_element *ntfs_runlists_merge(runlist_element *drl, - runlist_element *srl); - -extern runlist_element *ntfs_mapping_pairs_decompress(const ntfs_volume *vol, - const ATTR_RECORD *attr, runlist_element *old_rl); - -extern LCN ntfs_rl_vcn_to_lcn(const runlist_element *rl, const VCN vcn); - -#ifdef NTFS_RW - -extern runlist_element *ntfs_rl_find_vcn_nolock(runlist_element *rl, - const VCN vcn); - -extern int ntfs_get_size_for_mapping_pairs(const ntfs_volume *vol, - const runlist_element *rl, const VCN first_vcn, - const VCN last_vcn); - -extern int ntfs_mapping_pairs_build(const ntfs_volume *vol, s8 *dst, - const int dst_len, const runlist_element *rl, - const VCN first_vcn, const VCN last_vcn, VCN *const stop_vcn); - -extern int ntfs_rl_truncate_nolock(const ntfs_volume *vol, - runlist *const runlist, const s64 new_length); - -int ntfs_rl_punch_nolock(const ntfs_volume *vol, runlist *const runlist, - const VCN start, const s64 length); - -#endif /* NTFS_RW */ - -#endif /* _LINUX_NTFS_RUNLIST_H */ diff --git a/fs/ntfs/super.c b/fs/ntfs/super.c deleted file mode 100644 index 56a7d5bd33e4..000000000000 --- a/fs/ntfs/super.c +++ /dev/null @@ -1,3202 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * super.c - NTFS kernel super block handling. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2012 Anton Altaparmakov and Tuxera Inc. - * Copyright (c) 2001,2002 Richard Russon - */ -#define pr_fmt(fmt) KBUILD_MODNAME ": " fmt - -#include -#include -#include -#include -#include -#include /* For bdev_logical_block_size(). */ -#include -#include -#include -#include -#include - -#include "sysctl.h" -#include "logfile.h" -#include "quota.h" -#include "usnjrnl.h" -#include "dir.h" -#include "debug.h" -#include "index.h" -#include "inode.h" -#include "aops.h" -#include "layout.h" -#include "malloc.h" -#include "ntfs.h" - -/* Number of mounted filesystems which have compression enabled. */ -static unsigned long ntfs_nr_compression_users; - -/* A global default upcase table and a corresponding reference count. */ -static ntfschar *default_upcase; -static unsigned long ntfs_nr_upcase_users; - -/* Error constants/strings used in inode.c::ntfs_show_options(). */ -typedef enum { - /* One of these must be present, default is ON_ERRORS_CONTINUE. */ - ON_ERRORS_PANIC = 0x01, - ON_ERRORS_REMOUNT_RO = 0x02, - ON_ERRORS_CONTINUE = 0x04, - /* Optional, can be combined with any of the above. */ - ON_ERRORS_RECOVER = 0x10, -} ON_ERRORS_ACTIONS; - -const option_t on_errors_arr[] = { - { ON_ERRORS_PANIC, "panic" }, - { ON_ERRORS_REMOUNT_RO, "remount-ro", }, - { ON_ERRORS_CONTINUE, "continue", }, - { ON_ERRORS_RECOVER, "recover" }, - { 0, NULL } -}; - -/** - * simple_getbool - convert input string to a boolean value - * @s: input string to convert - * @setval: where to store the output boolean value - * - * Copied from old ntfs driver (which copied from vfat driver). - * - * "1", "yes", "true", or an empty string are converted to %true. - * "0", "no", and "false" are converted to %false. - * - * Return: %1 if the string is converted or was empty and *setval contains it; - * %0 if the string was not valid. - */ -static int simple_getbool(char *s, bool *setval) -{ - if (s) { - if (!strcmp(s, "1") || !strcmp(s, "yes") || !strcmp(s, "true")) - *setval = true; - else if (!strcmp(s, "0") || !strcmp(s, "no") || - !strcmp(s, "false")) - *setval = false; - else - return 0; - } else - *setval = true; - return 1; -} - -/** - * parse_options - parse the (re)mount options - * @vol: ntfs volume - * @opt: string containing the (re)mount options - * - * Parse the recognized options in @opt for the ntfs volume described by @vol. - */ -static bool parse_options(ntfs_volume *vol, char *opt) -{ - char *p, *v, *ov; - static char *utf8 = "utf8"; - int errors = 0, sloppy = 0; - kuid_t uid = INVALID_UID; - kgid_t gid = INVALID_GID; - umode_t fmask = (umode_t)-1, dmask = (umode_t)-1; - int mft_zone_multiplier = -1, on_errors = -1; - int show_sys_files = -1, case_sensitive = -1, disable_sparse = -1; - struct nls_table *nls_map = NULL, *old_nls; - - /* I am lazy... (-8 */ -#define NTFS_GETOPT_WITH_DEFAULT(option, variable, default_value) \ - if (!strcmp(p, option)) { \ - if (!v || !*v) \ - variable = default_value; \ - else { \ - variable = simple_strtoul(ov = v, &v, 0); \ - if (*v) \ - goto needs_val; \ - } \ - } -#define NTFS_GETOPT(option, variable) \ - if (!strcmp(p, option)) { \ - if (!v || !*v) \ - goto needs_arg; \ - variable = simple_strtoul(ov = v, &v, 0); \ - if (*v) \ - goto needs_val; \ - } -#define NTFS_GETOPT_UID(option, variable) \ - if (!strcmp(p, option)) { \ - uid_t uid_value; \ - if (!v || !*v) \ - goto needs_arg; \ - uid_value = simple_strtoul(ov = v, &v, 0); \ - if (*v) \ - goto needs_val; \ - variable = make_kuid(current_user_ns(), uid_value); \ - if (!uid_valid(variable)) \ - goto needs_val; \ - } -#define NTFS_GETOPT_GID(option, variable) \ - if (!strcmp(p, option)) { \ - gid_t gid_value; \ - if (!v || !*v) \ - goto needs_arg; \ - gid_value = simple_strtoul(ov = v, &v, 0); \ - if (*v) \ - goto needs_val; \ - variable = make_kgid(current_user_ns(), gid_value); \ - if (!gid_valid(variable)) \ - goto needs_val; \ - } -#define NTFS_GETOPT_OCTAL(option, variable) \ - if (!strcmp(p, option)) { \ - if (!v || !*v) \ - goto needs_arg; \ - variable = simple_strtoul(ov = v, &v, 8); \ - if (*v) \ - goto needs_val; \ - } -#define NTFS_GETOPT_BOOL(option, variable) \ - if (!strcmp(p, option)) { \ - bool val; \ - if (!simple_getbool(v, &val)) \ - goto needs_bool; \ - variable = val; \ - } -#define NTFS_GETOPT_OPTIONS_ARRAY(option, variable, opt_array) \ - if (!strcmp(p, option)) { \ - int _i; \ - if (!v || !*v) \ - goto needs_arg; \ - ov = v; \ - if (variable == -1) \ - variable = 0; \ - for (_i = 0; opt_array[_i].str && *opt_array[_i].str; _i++) \ - if (!strcmp(opt_array[_i].str, v)) { \ - variable |= opt_array[_i].val; \ - break; \ - } \ - if (!opt_array[_i].str || !*opt_array[_i].str) \ - goto needs_val; \ - } - if (!opt || !*opt) - goto no_mount_options; - ntfs_debug("Entering with mount options string: %s", opt); - while ((p = strsep(&opt, ","))) { - if ((v = strchr(p, '='))) - *v++ = 0; - NTFS_GETOPT_UID("uid", uid) - else NTFS_GETOPT_GID("gid", gid) - else NTFS_GETOPT_OCTAL("umask", fmask = dmask) - else NTFS_GETOPT_OCTAL("fmask", fmask) - else NTFS_GETOPT_OCTAL("dmask", dmask) - else NTFS_GETOPT("mft_zone_multiplier", mft_zone_multiplier) - else NTFS_GETOPT_WITH_DEFAULT("sloppy", sloppy, true) - else NTFS_GETOPT_BOOL("show_sys_files", show_sys_files) - else NTFS_GETOPT_BOOL("case_sensitive", case_sensitive) - else NTFS_GETOPT_BOOL("disable_sparse", disable_sparse) - else NTFS_GETOPT_OPTIONS_ARRAY("errors", on_errors, - on_errors_arr) - else if (!strcmp(p, "posix") || !strcmp(p, "show_inodes")) - ntfs_warning(vol->sb, "Ignoring obsolete option %s.", - p); - else if (!strcmp(p, "nls") || !strcmp(p, "iocharset")) { - if (!strcmp(p, "iocharset")) - ntfs_warning(vol->sb, "Option iocharset is " - "deprecated. Please use " - "option nls= in " - "the future."); - if (!v || !*v) - goto needs_arg; -use_utf8: - old_nls = nls_map; - nls_map = load_nls(v); - if (!nls_map) { - if (!old_nls) { - ntfs_error(vol->sb, "NLS character set " - "%s not found.", v); - return false; - } - ntfs_error(vol->sb, "NLS character set %s not " - "found. Using previous one %s.", - v, old_nls->charset); - nls_map = old_nls; - } else /* nls_map */ { - unload_nls(old_nls); - } - } else if (!strcmp(p, "utf8")) { - bool val = false; - ntfs_warning(vol->sb, "Option utf8 is no longer " - "supported, using option nls=utf8. Please " - "use option nls=utf8 in the future and " - "make sure utf8 is compiled either as a " - "module or into the kernel."); - if (!v || !*v) - val = true; - else if (!simple_getbool(v, &val)) - goto needs_bool; - if (val) { - v = utf8; - goto use_utf8; - } - } else { - ntfs_error(vol->sb, "Unrecognized mount option %s.", p); - if (errors < INT_MAX) - errors++; - } -#undef NTFS_GETOPT_OPTIONS_ARRAY -#undef NTFS_GETOPT_BOOL -#undef NTFS_GETOPT -#undef NTFS_GETOPT_WITH_DEFAULT - } -no_mount_options: - if (errors && !sloppy) - return false; - if (sloppy) - ntfs_warning(vol->sb, "Sloppy option given. Ignoring " - "unrecognized mount option(s) and continuing."); - /* Keep this first! */ - if (on_errors != -1) { - if (!on_errors) { - ntfs_error(vol->sb, "Invalid errors option argument " - "or bug in options parser."); - return false; - } - } - if (nls_map) { - if (vol->nls_map && vol->nls_map != nls_map) { - ntfs_error(vol->sb, "Cannot change NLS character set " - "on remount."); - return false; - } /* else (!vol->nls_map) */ - ntfs_debug("Using NLS character set %s.", nls_map->charset); - vol->nls_map = nls_map; - } else /* (!nls_map) */ { - if (!vol->nls_map) { - vol->nls_map = load_nls_default(); - if (!vol->nls_map) { - ntfs_error(vol->sb, "Failed to load default " - "NLS character set."); - return false; - } - ntfs_debug("Using default NLS character set (%s).", - vol->nls_map->charset); - } - } - if (mft_zone_multiplier != -1) { - if (vol->mft_zone_multiplier && vol->mft_zone_multiplier != - mft_zone_multiplier) { - ntfs_error(vol->sb, "Cannot change mft_zone_multiplier " - "on remount."); - return false; - } - if (mft_zone_multiplier < 1 || mft_zone_multiplier > 4) { - ntfs_error(vol->sb, "Invalid mft_zone_multiplier. " - "Using default value, i.e. 1."); - mft_zone_multiplier = 1; - } - vol->mft_zone_multiplier = mft_zone_multiplier; - } - if (!vol->mft_zone_multiplier) - vol->mft_zone_multiplier = 1; - if (on_errors != -1) - vol->on_errors = on_errors; - if (!vol->on_errors || vol->on_errors == ON_ERRORS_RECOVER) - vol->on_errors |= ON_ERRORS_CONTINUE; - if (uid_valid(uid)) - vol->uid = uid; - if (gid_valid(gid)) - vol->gid = gid; - if (fmask != (umode_t)-1) - vol->fmask = fmask; - if (dmask != (umode_t)-1) - vol->dmask = dmask; - if (show_sys_files != -1) { - if (show_sys_files) - NVolSetShowSystemFiles(vol); - else - NVolClearShowSystemFiles(vol); - } - if (case_sensitive != -1) { - if (case_sensitive) - NVolSetCaseSensitive(vol); - else - NVolClearCaseSensitive(vol); - } - if (disable_sparse != -1) { - if (disable_sparse) - NVolClearSparseEnabled(vol); - else { - if (!NVolSparseEnabled(vol) && - vol->major_ver && vol->major_ver < 3) - ntfs_warning(vol->sb, "Not enabling sparse " - "support due to NTFS volume " - "version %i.%i (need at least " - "version 3.0).", vol->major_ver, - vol->minor_ver); - else - NVolSetSparseEnabled(vol); - } - } - return true; -needs_arg: - ntfs_error(vol->sb, "The %s option requires an argument.", p); - return false; -needs_bool: - ntfs_error(vol->sb, "The %s option requires a boolean argument.", p); - return false; -needs_val: - ntfs_error(vol->sb, "Invalid %s option argument: %s", p, ov); - return false; -} - -#ifdef NTFS_RW - -/** - * ntfs_write_volume_flags - write new flags to the volume information flags - * @vol: ntfs volume on which to modify the flags - * @flags: new flags value for the volume information flags - * - * Internal function. You probably want to use ntfs_{set,clear}_volume_flags() - * instead (see below). - * - * Replace the volume information flags on the volume @vol with the value - * supplied in @flags. Note, this overwrites the volume information flags, so - * make sure to combine the flags you want to modify with the old flags and use - * the result when calling ntfs_write_volume_flags(). - * - * Return 0 on success and -errno on error. - */ -static int ntfs_write_volume_flags(ntfs_volume *vol, const VOLUME_FLAGS flags) -{ - ntfs_inode *ni = NTFS_I(vol->vol_ino); - MFT_RECORD *m; - VOLUME_INFORMATION *vi; - ntfs_attr_search_ctx *ctx; - int err; - - ntfs_debug("Entering, old flags = 0x%x, new flags = 0x%x.", - le16_to_cpu(vol->vol_flags), le16_to_cpu(flags)); - if (vol->vol_flags == flags) - goto done; - BUG_ON(!ni); - m = map_mft_record(ni); - if (IS_ERR(m)) { - err = PTR_ERR(m); - goto err_out; - } - ctx = ntfs_attr_get_search_ctx(ni, m); - if (!ctx) { - err = -ENOMEM; - goto put_unm_err_out; - } - err = ntfs_attr_lookup(AT_VOLUME_INFORMATION, NULL, 0, 0, 0, NULL, 0, - ctx); - if (err) - goto put_unm_err_out; - vi = (VOLUME_INFORMATION*)((u8*)ctx->attr + - le16_to_cpu(ctx->attr->data.resident.value_offset)); - vol->vol_flags = vi->flags = flags; - flush_dcache_mft_record_page(ctx->ntfs_ino); - mark_mft_record_dirty(ctx->ntfs_ino); - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(ni); -done: - ntfs_debug("Done."); - return 0; -put_unm_err_out: - if (ctx) - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(ni); -err_out: - ntfs_error(vol->sb, "Failed with error code %i.", -err); - return err; -} - -/** - * ntfs_set_volume_flags - set bits in the volume information flags - * @vol: ntfs volume on which to modify the flags - * @flags: flags to set on the volume - * - * Set the bits in @flags in the volume information flags on the volume @vol. - * - * Return 0 on success and -errno on error. - */ -static inline int ntfs_set_volume_flags(ntfs_volume *vol, VOLUME_FLAGS flags) -{ - flags &= VOLUME_FLAGS_MASK; - return ntfs_write_volume_flags(vol, vol->vol_flags | flags); -} - -/** - * ntfs_clear_volume_flags - clear bits in the volume information flags - * @vol: ntfs volume on which to modify the flags - * @flags: flags to clear on the volume - * - * Clear the bits in @flags in the volume information flags on the volume @vol. - * - * Return 0 on success and -errno on error. - */ -static inline int ntfs_clear_volume_flags(ntfs_volume *vol, VOLUME_FLAGS flags) -{ - flags &= VOLUME_FLAGS_MASK; - flags = vol->vol_flags & cpu_to_le16(~le16_to_cpu(flags)); - return ntfs_write_volume_flags(vol, flags); -} - -#endif /* NTFS_RW */ - -/** - * ntfs_remount - change the mount options of a mounted ntfs filesystem - * @sb: superblock of mounted ntfs filesystem - * @flags: remount flags - * @opt: remount options string - * - * Change the mount options of an already mounted ntfs filesystem. - * - * NOTE: The VFS sets the @sb->s_flags remount flags to @flags after - * ntfs_remount() returns successfully (i.e. returns 0). Otherwise, - * @sb->s_flags are not changed. - */ -static int ntfs_remount(struct super_block *sb, int *flags, char *opt) -{ - ntfs_volume *vol = NTFS_SB(sb); - - ntfs_debug("Entering with remount options string: %s", opt); - - sync_filesystem(sb); - -#ifndef NTFS_RW - /* For read-only compiled driver, enforce read-only flag. */ - *flags |= SB_RDONLY; -#else /* NTFS_RW */ - /* - * For the read-write compiled driver, if we are remounting read-write, - * make sure there are no volume errors and that no unsupported volume - * flags are set. Also, empty the logfile journal as it would become - * stale as soon as something is written to the volume and mark the - * volume dirty so that chkdsk is run if the volume is not umounted - * cleanly. Finally, mark the quotas out of date so Windows rescans - * the volume on boot and updates them. - * - * When remounting read-only, mark the volume clean if no volume errors - * have occurred. - */ - if (sb_rdonly(sb) && !(*flags & SB_RDONLY)) { - static const char *es = ". Cannot remount read-write."; - - /* Remounting read-write. */ - if (NVolErrors(vol)) { - ntfs_error(sb, "Volume has errors and is read-only%s", - es); - return -EROFS; - } - if (vol->vol_flags & VOLUME_IS_DIRTY) { - ntfs_error(sb, "Volume is dirty and read-only%s", es); - return -EROFS; - } - if (vol->vol_flags & VOLUME_MODIFIED_BY_CHKDSK) { - ntfs_error(sb, "Volume has been modified by chkdsk " - "and is read-only%s", es); - return -EROFS; - } - if (vol->vol_flags & VOLUME_MUST_MOUNT_RO_MASK) { - ntfs_error(sb, "Volume has unsupported flags set " - "(0x%x) and is read-only%s", - (unsigned)le16_to_cpu(vol->vol_flags), - es); - return -EROFS; - } - if (ntfs_set_volume_flags(vol, VOLUME_IS_DIRTY)) { - ntfs_error(sb, "Failed to set dirty bit in volume " - "information flags%s", es); - return -EROFS; - } -#if 0 - // TODO: Enable this code once we start modifying anything that - // is different between NTFS 1.2 and 3.x... - /* Set NT4 compatibility flag on newer NTFS version volumes. */ - if ((vol->major_ver > 1)) { - if (ntfs_set_volume_flags(vol, VOLUME_MOUNTED_ON_NT4)) { - ntfs_error(sb, "Failed to set NT4 " - "compatibility flag%s", es); - NVolSetErrors(vol); - return -EROFS; - } - } -#endif - if (!ntfs_empty_logfile(vol->logfile_ino)) { - ntfs_error(sb, "Failed to empty journal $LogFile%s", - es); - NVolSetErrors(vol); - return -EROFS; - } - if (!ntfs_mark_quotas_out_of_date(vol)) { - ntfs_error(sb, "Failed to mark quotas out of date%s", - es); - NVolSetErrors(vol); - return -EROFS; - } - if (!ntfs_stamp_usnjrnl(vol)) { - ntfs_error(sb, "Failed to stamp transaction log " - "($UsnJrnl)%s", es); - NVolSetErrors(vol); - return -EROFS; - } - } else if (!sb_rdonly(sb) && (*flags & SB_RDONLY)) { - /* Remounting read-only. */ - if (!NVolErrors(vol)) { - if (ntfs_clear_volume_flags(vol, VOLUME_IS_DIRTY)) - ntfs_warning(sb, "Failed to clear dirty bit " - "in volume information " - "flags. Run chkdsk."); - } - } -#endif /* NTFS_RW */ - - // TODO: Deal with *flags. - - if (!parse_options(vol, opt)) - return -EINVAL; - - ntfs_debug("Done."); - return 0; -} - -/** - * is_boot_sector_ntfs - check whether a boot sector is a valid NTFS boot sector - * @sb: Super block of the device to which @b belongs. - * @b: Boot sector of device @sb to check. - * @silent: If 'true', all output will be silenced. - * - * is_boot_sector_ntfs() checks whether the boot sector @b is a valid NTFS boot - * sector. Returns 'true' if it is valid and 'false' if not. - * - * @sb is only needed for warning/error output, i.e. it can be NULL when silent - * is 'true'. - */ -static bool is_boot_sector_ntfs(const struct super_block *sb, - const NTFS_BOOT_SECTOR *b, const bool silent) -{ - /* - * Check that checksum == sum of u32 values from b to the checksum - * field. If checksum is zero, no checking is done. We will work when - * the checksum test fails, since some utilities update the boot sector - * ignoring the checksum which leaves the checksum out-of-date. We - * report a warning if this is the case. - */ - if ((void*)b < (void*)&b->checksum && b->checksum && !silent) { - le32 *u; - u32 i; - - for (i = 0, u = (le32*)b; u < (le32*)(&b->checksum); ++u) - i += le32_to_cpup(u); - if (le32_to_cpu(b->checksum) != i) - ntfs_warning(sb, "Invalid boot sector checksum."); - } - /* Check OEMidentifier is "NTFS " */ - if (b->oem_id != magicNTFS) - goto not_ntfs; - /* Check bytes per sector value is between 256 and 4096. */ - if (le16_to_cpu(b->bpb.bytes_per_sector) < 0x100 || - le16_to_cpu(b->bpb.bytes_per_sector) > 0x1000) - goto not_ntfs; - /* Check sectors per cluster value is valid. */ - switch (b->bpb.sectors_per_cluster) { - case 1: case 2: case 4: case 8: case 16: case 32: case 64: case 128: - break; - default: - goto not_ntfs; - } - /* Check the cluster size is not above the maximum (64kiB). */ - if ((u32)le16_to_cpu(b->bpb.bytes_per_sector) * - b->bpb.sectors_per_cluster > NTFS_MAX_CLUSTER_SIZE) - goto not_ntfs; - /* Check reserved/unused fields are really zero. */ - if (le16_to_cpu(b->bpb.reserved_sectors) || - le16_to_cpu(b->bpb.root_entries) || - le16_to_cpu(b->bpb.sectors) || - le16_to_cpu(b->bpb.sectors_per_fat) || - le32_to_cpu(b->bpb.large_sectors) || b->bpb.fats) - goto not_ntfs; - /* Check clusters per file mft record value is valid. */ - if ((u8)b->clusters_per_mft_record < 0xe1 || - (u8)b->clusters_per_mft_record > 0xf7) - switch (b->clusters_per_mft_record) { - case 1: case 2: case 4: case 8: case 16: case 32: case 64: - break; - default: - goto not_ntfs; - } - /* Check clusters per index block value is valid. */ - if ((u8)b->clusters_per_index_record < 0xe1 || - (u8)b->clusters_per_index_record > 0xf7) - switch (b->clusters_per_index_record) { - case 1: case 2: case 4: case 8: case 16: case 32: case 64: - break; - default: - goto not_ntfs; - } - /* - * Check for valid end of sector marker. We will work without it, but - * many BIOSes will refuse to boot from a bootsector if the magic is - * incorrect, so we emit a warning. - */ - if (!silent && b->end_of_sector_marker != cpu_to_le16(0xaa55)) - ntfs_warning(sb, "Invalid end of sector marker."); - return true; -not_ntfs: - return false; -} - -/** - * read_ntfs_boot_sector - read the NTFS boot sector of a device - * @sb: super block of device to read the boot sector from - * @silent: if true, suppress all output - * - * Reads the boot sector from the device and validates it. If that fails, tries - * to read the backup boot sector, first from the end of the device a-la NT4 and - * later and then from the middle of the device a-la NT3.51 and before. - * - * If a valid boot sector is found but it is not the primary boot sector, we - * repair the primary boot sector silently (unless the device is read-only or - * the primary boot sector is not accessible). - * - * NOTE: To call this function, @sb must have the fields s_dev, the ntfs super - * block (u.ntfs_sb), nr_blocks and the device flags (s_flags) initialized - * to their respective values. - * - * Return the unlocked buffer head containing the boot sector or NULL on error. - */ -static struct buffer_head *read_ntfs_boot_sector(struct super_block *sb, - const int silent) -{ - const char *read_err_str = "Unable to read %s boot sector."; - struct buffer_head *bh_primary, *bh_backup; - sector_t nr_blocks = NTFS_SB(sb)->nr_blocks; - - /* Try to read primary boot sector. */ - if ((bh_primary = sb_bread(sb, 0))) { - if (is_boot_sector_ntfs(sb, (NTFS_BOOT_SECTOR*) - bh_primary->b_data, silent)) - return bh_primary; - if (!silent) - ntfs_error(sb, "Primary boot sector is invalid."); - } else if (!silent) - ntfs_error(sb, read_err_str, "primary"); - if (!(NTFS_SB(sb)->on_errors & ON_ERRORS_RECOVER)) { - if (bh_primary) - brelse(bh_primary); - if (!silent) - ntfs_error(sb, "Mount option errors=recover not used. " - "Aborting without trying to recover."); - return NULL; - } - /* Try to read NT4+ backup boot sector. */ - if ((bh_backup = sb_bread(sb, nr_blocks - 1))) { - if (is_boot_sector_ntfs(sb, (NTFS_BOOT_SECTOR*) - bh_backup->b_data, silent)) - goto hotfix_primary_boot_sector; - brelse(bh_backup); - } else if (!silent) - ntfs_error(sb, read_err_str, "backup"); - /* Try to read NT3.51- backup boot sector. */ - if ((bh_backup = sb_bread(sb, nr_blocks >> 1))) { - if (is_boot_sector_ntfs(sb, (NTFS_BOOT_SECTOR*) - bh_backup->b_data, silent)) - goto hotfix_primary_boot_sector; - if (!silent) - ntfs_error(sb, "Could not find a valid backup boot " - "sector."); - brelse(bh_backup); - } else if (!silent) - ntfs_error(sb, read_err_str, "backup"); - /* We failed. Cleanup and return. */ - if (bh_primary) - brelse(bh_primary); - return NULL; -hotfix_primary_boot_sector: - if (bh_primary) { - /* - * If we managed to read sector zero and the volume is not - * read-only, copy the found, valid backup boot sector to the - * primary boot sector. Note we only copy the actual boot - * sector structure, not the actual whole device sector as that - * may be bigger and would potentially damage the $Boot system - * file (FIXME: Would be nice to know if the backup boot sector - * on a large sector device contains the whole boot loader or - * just the first 512 bytes). - */ - if (!sb_rdonly(sb)) { - ntfs_warning(sb, "Hot-fix: Recovering invalid primary " - "boot sector from backup copy."); - memcpy(bh_primary->b_data, bh_backup->b_data, - NTFS_BLOCK_SIZE); - mark_buffer_dirty(bh_primary); - sync_dirty_buffer(bh_primary); - if (buffer_uptodate(bh_primary)) { - brelse(bh_backup); - return bh_primary; - } - ntfs_error(sb, "Hot-fix: Device write error while " - "recovering primary boot sector."); - } else { - ntfs_warning(sb, "Hot-fix: Recovery of primary boot " - "sector failed: Read-only mount."); - } - brelse(bh_primary); - } - ntfs_warning(sb, "Using backup boot sector."); - return bh_backup; -} - -/** - * parse_ntfs_boot_sector - parse the boot sector and store the data in @vol - * @vol: volume structure to initialise with data from boot sector - * @b: boot sector to parse - * - * Parse the ntfs boot sector @b and store all imporant information therein in - * the ntfs super block @vol. Return 'true' on success and 'false' on error. - */ -static bool parse_ntfs_boot_sector(ntfs_volume *vol, const NTFS_BOOT_SECTOR *b) -{ - unsigned int sectors_per_cluster_bits, nr_hidden_sects; - int clusters_per_mft_record, clusters_per_index_record; - s64 ll; - - vol->sector_size = le16_to_cpu(b->bpb.bytes_per_sector); - vol->sector_size_bits = ffs(vol->sector_size) - 1; - ntfs_debug("vol->sector_size = %i (0x%x)", vol->sector_size, - vol->sector_size); - ntfs_debug("vol->sector_size_bits = %i (0x%x)", vol->sector_size_bits, - vol->sector_size_bits); - if (vol->sector_size < vol->sb->s_blocksize) { - ntfs_error(vol->sb, "Sector size (%i) is smaller than the " - "device block size (%lu). This is not " - "supported. Sorry.", vol->sector_size, - vol->sb->s_blocksize); - return false; - } - ntfs_debug("sectors_per_cluster = 0x%x", b->bpb.sectors_per_cluster); - sectors_per_cluster_bits = ffs(b->bpb.sectors_per_cluster) - 1; - ntfs_debug("sectors_per_cluster_bits = 0x%x", - sectors_per_cluster_bits); - nr_hidden_sects = le32_to_cpu(b->bpb.hidden_sectors); - ntfs_debug("number of hidden sectors = 0x%x", nr_hidden_sects); - vol->cluster_size = vol->sector_size << sectors_per_cluster_bits; - vol->cluster_size_mask = vol->cluster_size - 1; - vol->cluster_size_bits = ffs(vol->cluster_size) - 1; - ntfs_debug("vol->cluster_size = %i (0x%x)", vol->cluster_size, - vol->cluster_size); - ntfs_debug("vol->cluster_size_mask = 0x%x", vol->cluster_size_mask); - ntfs_debug("vol->cluster_size_bits = %i", vol->cluster_size_bits); - if (vol->cluster_size < vol->sector_size) { - ntfs_error(vol->sb, "Cluster size (%i) is smaller than the " - "sector size (%i). This is not supported. " - "Sorry.", vol->cluster_size, vol->sector_size); - return false; - } - clusters_per_mft_record = b->clusters_per_mft_record; - ntfs_debug("clusters_per_mft_record = %i (0x%x)", - clusters_per_mft_record, clusters_per_mft_record); - if (clusters_per_mft_record > 0) - vol->mft_record_size = vol->cluster_size << - (ffs(clusters_per_mft_record) - 1); - else - /* - * When mft_record_size < cluster_size, clusters_per_mft_record - * = -log2(mft_record_size) bytes. mft_record_size normaly is - * 1024 bytes, which is encoded as 0xF6 (-10 in decimal). - */ - vol->mft_record_size = 1 << -clusters_per_mft_record; - vol->mft_record_size_mask = vol->mft_record_size - 1; - vol->mft_record_size_bits = ffs(vol->mft_record_size) - 1; - ntfs_debug("vol->mft_record_size = %i (0x%x)", vol->mft_record_size, - vol->mft_record_size); - ntfs_debug("vol->mft_record_size_mask = 0x%x", - vol->mft_record_size_mask); - ntfs_debug("vol->mft_record_size_bits = %i (0x%x)", - vol->mft_record_size_bits, vol->mft_record_size_bits); - /* - * We cannot support mft record sizes above the PAGE_SIZE since - * we store $MFT/$DATA, the table of mft records in the page cache. - */ - if (vol->mft_record_size > PAGE_SIZE) { - ntfs_error(vol->sb, "Mft record size (%i) exceeds the " - "PAGE_SIZE on your system (%lu). " - "This is not supported. Sorry.", - vol->mft_record_size, PAGE_SIZE); - return false; - } - /* We cannot support mft record sizes below the sector size. */ - if (vol->mft_record_size < vol->sector_size) { - ntfs_error(vol->sb, "Mft record size (%i) is smaller than the " - "sector size (%i). This is not supported. " - "Sorry.", vol->mft_record_size, - vol->sector_size); - return false; - } - clusters_per_index_record = b->clusters_per_index_record; - ntfs_debug("clusters_per_index_record = %i (0x%x)", - clusters_per_index_record, clusters_per_index_record); - if (clusters_per_index_record > 0) - vol->index_record_size = vol->cluster_size << - (ffs(clusters_per_index_record) - 1); - else - /* - * When index_record_size < cluster_size, - * clusters_per_index_record = -log2(index_record_size) bytes. - * index_record_size normaly equals 4096 bytes, which is - * encoded as 0xF4 (-12 in decimal). - */ - vol->index_record_size = 1 << -clusters_per_index_record; - vol->index_record_size_mask = vol->index_record_size - 1; - vol->index_record_size_bits = ffs(vol->index_record_size) - 1; - ntfs_debug("vol->index_record_size = %i (0x%x)", - vol->index_record_size, vol->index_record_size); - ntfs_debug("vol->index_record_size_mask = 0x%x", - vol->index_record_size_mask); - ntfs_debug("vol->index_record_size_bits = %i (0x%x)", - vol->index_record_size_bits, - vol->index_record_size_bits); - /* We cannot support index record sizes below the sector size. */ - if (vol->index_record_size < vol->sector_size) { - ntfs_error(vol->sb, "Index record size (%i) is smaller than " - "the sector size (%i). This is not " - "supported. Sorry.", vol->index_record_size, - vol->sector_size); - return false; - } - /* - * Get the size of the volume in clusters and check for 64-bit-ness. - * Windows currently only uses 32 bits to save the clusters so we do - * the same as it is much faster on 32-bit CPUs. - */ - ll = sle64_to_cpu(b->number_of_sectors) >> sectors_per_cluster_bits; - if ((u64)ll >= 1ULL << 32) { - ntfs_error(vol->sb, "Cannot handle 64-bit clusters. Sorry."); - return false; - } - vol->nr_clusters = ll; - ntfs_debug("vol->nr_clusters = 0x%llx", (long long)vol->nr_clusters); - /* - * On an architecture where unsigned long is 32-bits, we restrict the - * volume size to 2TiB (2^41). On a 64-bit architecture, the compiler - * will hopefully optimize the whole check away. - */ - if (sizeof(unsigned long) < 8) { - if ((ll << vol->cluster_size_bits) >= (1ULL << 41)) { - ntfs_error(vol->sb, "Volume size (%lluTiB) is too " - "large for this architecture. " - "Maximum supported is 2TiB. Sorry.", - (unsigned long long)ll >> (40 - - vol->cluster_size_bits)); - return false; - } - } - ll = sle64_to_cpu(b->mft_lcn); - if (ll >= vol->nr_clusters) { - ntfs_error(vol->sb, "MFT LCN (%lli, 0x%llx) is beyond end of " - "volume. Weird.", (unsigned long long)ll, - (unsigned long long)ll); - return false; - } - vol->mft_lcn = ll; - ntfs_debug("vol->mft_lcn = 0x%llx", (long long)vol->mft_lcn); - ll = sle64_to_cpu(b->mftmirr_lcn); - if (ll >= vol->nr_clusters) { - ntfs_error(vol->sb, "MFTMirr LCN (%lli, 0x%llx) is beyond end " - "of volume. Weird.", (unsigned long long)ll, - (unsigned long long)ll); - return false; - } - vol->mftmirr_lcn = ll; - ntfs_debug("vol->mftmirr_lcn = 0x%llx", (long long)vol->mftmirr_lcn); -#ifdef NTFS_RW - /* - * Work out the size of the mft mirror in number of mft records. If the - * cluster size is less than or equal to the size taken by four mft - * records, the mft mirror stores the first four mft records. If the - * cluster size is bigger than the size taken by four mft records, the - * mft mirror contains as many mft records as will fit into one - * cluster. - */ - if (vol->cluster_size <= (4 << vol->mft_record_size_bits)) - vol->mftmirr_size = 4; - else - vol->mftmirr_size = vol->cluster_size >> - vol->mft_record_size_bits; - ntfs_debug("vol->mftmirr_size = %i", vol->mftmirr_size); -#endif /* NTFS_RW */ - vol->serial_no = le64_to_cpu(b->volume_serial_number); - ntfs_debug("vol->serial_no = 0x%llx", - (unsigned long long)vol->serial_no); - return true; -} - -/** - * ntfs_setup_allocators - initialize the cluster and mft allocators - * @vol: volume structure for which to setup the allocators - * - * Setup the cluster (lcn) and mft allocators to the starting values. - */ -static void ntfs_setup_allocators(ntfs_volume *vol) -{ -#ifdef NTFS_RW - LCN mft_zone_size, mft_lcn; -#endif /* NTFS_RW */ - - ntfs_debug("vol->mft_zone_multiplier = 0x%x", - vol->mft_zone_multiplier); -#ifdef NTFS_RW - /* Determine the size of the MFT zone. */ - mft_zone_size = vol->nr_clusters; - switch (vol->mft_zone_multiplier) { /* % of volume size in clusters */ - case 4: - mft_zone_size >>= 1; /* 50% */ - break; - case 3: - mft_zone_size = (mft_zone_size + - (mft_zone_size >> 1)) >> 2; /* 37.5% */ - break; - case 2: - mft_zone_size >>= 2; /* 25% */ - break; - /* case 1: */ - default: - mft_zone_size >>= 3; /* 12.5% */ - break; - } - /* Setup the mft zone. */ - vol->mft_zone_start = vol->mft_zone_pos = vol->mft_lcn; - ntfs_debug("vol->mft_zone_pos = 0x%llx", - (unsigned long long)vol->mft_zone_pos); - /* - * Calculate the mft_lcn for an unmodified NTFS volume (see mkntfs - * source) and if the actual mft_lcn is in the expected place or even - * further to the front of the volume, extend the mft_zone to cover the - * beginning of the volume as well. This is in order to protect the - * area reserved for the mft bitmap as well within the mft_zone itself. - * On non-standard volumes we do not protect it as the overhead would - * be higher than the speed increase we would get by doing it. - */ - mft_lcn = (8192 + 2 * vol->cluster_size - 1) / vol->cluster_size; - if (mft_lcn * vol->cluster_size < 16 * 1024) - mft_lcn = (16 * 1024 + vol->cluster_size - 1) / - vol->cluster_size; - if (vol->mft_zone_start <= mft_lcn) - vol->mft_zone_start = 0; - ntfs_debug("vol->mft_zone_start = 0x%llx", - (unsigned long long)vol->mft_zone_start); - /* - * Need to cap the mft zone on non-standard volumes so that it does - * not point outside the boundaries of the volume. We do this by - * halving the zone size until we are inside the volume. - */ - vol->mft_zone_end = vol->mft_lcn + mft_zone_size; - while (vol->mft_zone_end >= vol->nr_clusters) { - mft_zone_size >>= 1; - vol->mft_zone_end = vol->mft_lcn + mft_zone_size; - } - ntfs_debug("vol->mft_zone_end = 0x%llx", - (unsigned long long)vol->mft_zone_end); - /* - * Set the current position within each data zone to the start of the - * respective zone. - */ - vol->data1_zone_pos = vol->mft_zone_end; - ntfs_debug("vol->data1_zone_pos = 0x%llx", - (unsigned long long)vol->data1_zone_pos); - vol->data2_zone_pos = 0; - ntfs_debug("vol->data2_zone_pos = 0x%llx", - (unsigned long long)vol->data2_zone_pos); - - /* Set the mft data allocation position to mft record 24. */ - vol->mft_data_pos = 24; - ntfs_debug("vol->mft_data_pos = 0x%llx", - (unsigned long long)vol->mft_data_pos); -#endif /* NTFS_RW */ -} - -#ifdef NTFS_RW - -/** - * load_and_init_mft_mirror - load and setup the mft mirror inode for a volume - * @vol: ntfs super block describing device whose mft mirror to load - * - * Return 'true' on success or 'false' on error. - */ -static bool load_and_init_mft_mirror(ntfs_volume *vol) -{ - struct inode *tmp_ino; - ntfs_inode *tmp_ni; - - ntfs_debug("Entering."); - /* Get mft mirror inode. */ - tmp_ino = ntfs_iget(vol->sb, FILE_MFTMirr); - if (IS_ERR(tmp_ino) || is_bad_inode(tmp_ino)) { - if (!IS_ERR(tmp_ino)) - iput(tmp_ino); - /* Caller will display error message. */ - return false; - } - /* - * Re-initialize some specifics about $MFTMirr's inode as - * ntfs_read_inode() will have set up the default ones. - */ - /* Set uid and gid to root. */ - tmp_ino->i_uid = GLOBAL_ROOT_UID; - tmp_ino->i_gid = GLOBAL_ROOT_GID; - /* Regular file. No access for anyone. */ - tmp_ino->i_mode = S_IFREG; - /* No VFS initiated operations allowed for $MFTMirr. */ - tmp_ino->i_op = &ntfs_empty_inode_ops; - tmp_ino->i_fop = &ntfs_empty_file_ops; - /* Put in our special address space operations. */ - tmp_ino->i_mapping->a_ops = &ntfs_mst_aops; - tmp_ni = NTFS_I(tmp_ino); - /* The $MFTMirr, like the $MFT is multi sector transfer protected. */ - NInoSetMstProtected(tmp_ni); - NInoSetSparseDisabled(tmp_ni); - /* - * Set up our little cheat allowing us to reuse the async read io - * completion handler for directories. - */ - tmp_ni->itype.index.block_size = vol->mft_record_size; - tmp_ni->itype.index.block_size_bits = vol->mft_record_size_bits; - vol->mftmirr_ino = tmp_ino; - ntfs_debug("Done."); - return true; -} - -/** - * check_mft_mirror - compare contents of the mft mirror with the mft - * @vol: ntfs super block describing device whose mft mirror to check - * - * Return 'true' on success or 'false' on error. - * - * Note, this function also results in the mft mirror runlist being completely - * mapped into memory. The mft mirror write code requires this and will BUG() - * should it find an unmapped runlist element. - */ -static bool check_mft_mirror(ntfs_volume *vol) -{ - struct super_block *sb = vol->sb; - ntfs_inode *mirr_ni; - struct page *mft_page, *mirr_page; - u8 *kmft, *kmirr; - runlist_element *rl, rl2[2]; - pgoff_t index; - int mrecs_per_page, i; - - ntfs_debug("Entering."); - /* Compare contents of $MFT and $MFTMirr. */ - mrecs_per_page = PAGE_SIZE / vol->mft_record_size; - BUG_ON(!mrecs_per_page); - BUG_ON(!vol->mftmirr_size); - mft_page = mirr_page = NULL; - kmft = kmirr = NULL; - index = i = 0; - do { - u32 bytes; - - /* Switch pages if necessary. */ - if (!(i % mrecs_per_page)) { - if (index) { - ntfs_unmap_page(mft_page); - ntfs_unmap_page(mirr_page); - } - /* Get the $MFT page. */ - mft_page = ntfs_map_page(vol->mft_ino->i_mapping, - index); - if (IS_ERR(mft_page)) { - ntfs_error(sb, "Failed to read $MFT."); - return false; - } - kmft = page_address(mft_page); - /* Get the $MFTMirr page. */ - mirr_page = ntfs_map_page(vol->mftmirr_ino->i_mapping, - index); - if (IS_ERR(mirr_page)) { - ntfs_error(sb, "Failed to read $MFTMirr."); - goto mft_unmap_out; - } - kmirr = page_address(mirr_page); - ++index; - } - /* Do not check the record if it is not in use. */ - if (((MFT_RECORD*)kmft)->flags & MFT_RECORD_IN_USE) { - /* Make sure the record is ok. */ - if (ntfs_is_baad_recordp((le32*)kmft)) { - ntfs_error(sb, "Incomplete multi sector " - "transfer detected in mft " - "record %i.", i); -mm_unmap_out: - ntfs_unmap_page(mirr_page); -mft_unmap_out: - ntfs_unmap_page(mft_page); - return false; - } - } - /* Do not check the mirror record if it is not in use. */ - if (((MFT_RECORD*)kmirr)->flags & MFT_RECORD_IN_USE) { - if (ntfs_is_baad_recordp((le32*)kmirr)) { - ntfs_error(sb, "Incomplete multi sector " - "transfer detected in mft " - "mirror record %i.", i); - goto mm_unmap_out; - } - } - /* Get the amount of data in the current record. */ - bytes = le32_to_cpu(((MFT_RECORD*)kmft)->bytes_in_use); - if (bytes < sizeof(MFT_RECORD_OLD) || - bytes > vol->mft_record_size || - ntfs_is_baad_recordp((le32*)kmft)) { - bytes = le32_to_cpu(((MFT_RECORD*)kmirr)->bytes_in_use); - if (bytes < sizeof(MFT_RECORD_OLD) || - bytes > vol->mft_record_size || - ntfs_is_baad_recordp((le32*)kmirr)) - bytes = vol->mft_record_size; - } - /* Compare the two records. */ - if (memcmp(kmft, kmirr, bytes)) { - ntfs_error(sb, "$MFT and $MFTMirr (record %i) do not " - "match. Run ntfsfix or chkdsk.", i); - goto mm_unmap_out; - } - kmft += vol->mft_record_size; - kmirr += vol->mft_record_size; - } while (++i < vol->mftmirr_size); - /* Release the last pages. */ - ntfs_unmap_page(mft_page); - ntfs_unmap_page(mirr_page); - - /* Construct the mft mirror runlist by hand. */ - rl2[0].vcn = 0; - rl2[0].lcn = vol->mftmirr_lcn; - rl2[0].length = (vol->mftmirr_size * vol->mft_record_size + - vol->cluster_size - 1) / vol->cluster_size; - rl2[1].vcn = rl2[0].length; - rl2[1].lcn = LCN_ENOENT; - rl2[1].length = 0; - /* - * Because we have just read all of the mft mirror, we know we have - * mapped the full runlist for it. - */ - mirr_ni = NTFS_I(vol->mftmirr_ino); - down_read(&mirr_ni->runlist.lock); - rl = mirr_ni->runlist.rl; - /* Compare the two runlists. They must be identical. */ - i = 0; - do { - if (rl2[i].vcn != rl[i].vcn || rl2[i].lcn != rl[i].lcn || - rl2[i].length != rl[i].length) { - ntfs_error(sb, "$MFTMirr location mismatch. " - "Run chkdsk."); - up_read(&mirr_ni->runlist.lock); - return false; - } - } while (rl2[i++].length); - up_read(&mirr_ni->runlist.lock); - ntfs_debug("Done."); - return true; -} - -/** - * load_and_check_logfile - load and check the logfile inode for a volume - * @vol: ntfs super block describing device whose logfile to load - * - * Return 'true' on success or 'false' on error. - */ -static bool load_and_check_logfile(ntfs_volume *vol, - RESTART_PAGE_HEADER **rp) -{ - struct inode *tmp_ino; - - ntfs_debug("Entering."); - tmp_ino = ntfs_iget(vol->sb, FILE_LogFile); - if (IS_ERR(tmp_ino) || is_bad_inode(tmp_ino)) { - if (!IS_ERR(tmp_ino)) - iput(tmp_ino); - /* Caller will display error message. */ - return false; - } - if (!ntfs_check_logfile(tmp_ino, rp)) { - iput(tmp_ino); - /* ntfs_check_logfile() will have displayed error output. */ - return false; - } - NInoSetSparseDisabled(NTFS_I(tmp_ino)); - vol->logfile_ino = tmp_ino; - ntfs_debug("Done."); - return true; -} - -#define NTFS_HIBERFIL_HEADER_SIZE 4096 - -/** - * check_windows_hibernation_status - check if Windows is suspended on a volume - * @vol: ntfs super block of device to check - * - * Check if Windows is hibernated on the ntfs volume @vol. This is done by - * looking for the file hiberfil.sys in the root directory of the volume. If - * the file is not present Windows is definitely not suspended. - * - * If hiberfil.sys exists and is less than 4kiB in size it means Windows is - * definitely suspended (this volume is not the system volume). Caveat: on a - * system with many volumes it is possible that the < 4kiB check is bogus but - * for now this should do fine. - * - * If hiberfil.sys exists and is larger than 4kiB in size, we need to read the - * hiberfil header (which is the first 4kiB). If this begins with "hibr", - * Windows is definitely suspended. If it is completely full of zeroes, - * Windows is definitely not hibernated. Any other case is treated as if - * Windows is suspended. This caters for the above mentioned caveat of a - * system with many volumes where no "hibr" magic would be present and there is - * no zero header. - * - * Return 0 if Windows is not hibernated on the volume, >0 if Windows is - * hibernated on the volume, and -errno on error. - */ -static int check_windows_hibernation_status(ntfs_volume *vol) -{ - MFT_REF mref; - struct inode *vi; - struct page *page; - u32 *kaddr, *kend; - ntfs_name *name = NULL; - int ret = 1; - static const ntfschar hiberfil[13] = { cpu_to_le16('h'), - cpu_to_le16('i'), cpu_to_le16('b'), - cpu_to_le16('e'), cpu_to_le16('r'), - cpu_to_le16('f'), cpu_to_le16('i'), - cpu_to_le16('l'), cpu_to_le16('.'), - cpu_to_le16('s'), cpu_to_le16('y'), - cpu_to_le16('s'), 0 }; - - ntfs_debug("Entering."); - /* - * Find the inode number for the hibernation file by looking up the - * filename hiberfil.sys in the root directory. - */ - inode_lock(vol->root_ino); - mref = ntfs_lookup_inode_by_name(NTFS_I(vol->root_ino), hiberfil, 12, - &name); - inode_unlock(vol->root_ino); - if (IS_ERR_MREF(mref)) { - ret = MREF_ERR(mref); - /* If the file does not exist, Windows is not hibernated. */ - if (ret == -ENOENT) { - ntfs_debug("hiberfil.sys not present. Windows is not " - "hibernated on the volume."); - return 0; - } - /* A real error occurred. */ - ntfs_error(vol->sb, "Failed to find inode number for " - "hiberfil.sys."); - return ret; - } - /* We do not care for the type of match that was found. */ - kfree(name); - /* Get the inode. */ - vi = ntfs_iget(vol->sb, MREF(mref)); - if (IS_ERR(vi) || is_bad_inode(vi)) { - if (!IS_ERR(vi)) - iput(vi); - ntfs_error(vol->sb, "Failed to load hiberfil.sys."); - return IS_ERR(vi) ? PTR_ERR(vi) : -EIO; - } - if (unlikely(i_size_read(vi) < NTFS_HIBERFIL_HEADER_SIZE)) { - ntfs_debug("hiberfil.sys is smaller than 4kiB (0x%llx). " - "Windows is hibernated on the volume. This " - "is not the system volume.", i_size_read(vi)); - goto iput_out; - } - page = ntfs_map_page(vi->i_mapping, 0); - if (IS_ERR(page)) { - ntfs_error(vol->sb, "Failed to read from hiberfil.sys."); - ret = PTR_ERR(page); - goto iput_out; - } - kaddr = (u32*)page_address(page); - if (*(le32*)kaddr == cpu_to_le32(0x72626968)/*'hibr'*/) { - ntfs_debug("Magic \"hibr\" found in hiberfil.sys. Windows is " - "hibernated on the volume. This is the " - "system volume."); - goto unm_iput_out; - } - kend = kaddr + NTFS_HIBERFIL_HEADER_SIZE/sizeof(*kaddr); - do { - if (unlikely(*kaddr)) { - ntfs_debug("hiberfil.sys is larger than 4kiB " - "(0x%llx), does not contain the " - "\"hibr\" magic, and does not have a " - "zero header. Windows is hibernated " - "on the volume. This is not the " - "system volume.", i_size_read(vi)); - goto unm_iput_out; - } - } while (++kaddr < kend); - ntfs_debug("hiberfil.sys contains a zero header. Windows is not " - "hibernated on the volume. This is the system " - "volume."); - ret = 0; -unm_iput_out: - ntfs_unmap_page(page); -iput_out: - iput(vi); - return ret; -} - -/** - * load_and_init_quota - load and setup the quota file for a volume if present - * @vol: ntfs super block describing device whose quota file to load - * - * Return 'true' on success or 'false' on error. If $Quota is not present, we - * leave vol->quota_ino as NULL and return success. - */ -static bool load_and_init_quota(ntfs_volume *vol) -{ - MFT_REF mref; - struct inode *tmp_ino; - ntfs_name *name = NULL; - static const ntfschar Quota[7] = { cpu_to_le16('$'), - cpu_to_le16('Q'), cpu_to_le16('u'), - cpu_to_le16('o'), cpu_to_le16('t'), - cpu_to_le16('a'), 0 }; - static ntfschar Q[3] = { cpu_to_le16('$'), - cpu_to_le16('Q'), 0 }; - - ntfs_debug("Entering."); - /* - * Find the inode number for the quota file by looking up the filename - * $Quota in the extended system files directory $Extend. - */ - inode_lock(vol->extend_ino); - mref = ntfs_lookup_inode_by_name(NTFS_I(vol->extend_ino), Quota, 6, - &name); - inode_unlock(vol->extend_ino); - if (IS_ERR_MREF(mref)) { - /* - * If the file does not exist, quotas are disabled and have - * never been enabled on this volume, just return success. - */ - if (MREF_ERR(mref) == -ENOENT) { - ntfs_debug("$Quota not present. Volume does not have " - "quotas enabled."); - /* - * No need to try to set quotas out of date if they are - * not enabled. - */ - NVolSetQuotaOutOfDate(vol); - return true; - } - /* A real error occurred. */ - ntfs_error(vol->sb, "Failed to find inode number for $Quota."); - return false; - } - /* We do not care for the type of match that was found. */ - kfree(name); - /* Get the inode. */ - tmp_ino = ntfs_iget(vol->sb, MREF(mref)); - if (IS_ERR(tmp_ino) || is_bad_inode(tmp_ino)) { - if (!IS_ERR(tmp_ino)) - iput(tmp_ino); - ntfs_error(vol->sb, "Failed to load $Quota."); - return false; - } - vol->quota_ino = tmp_ino; - /* Get the $Q index allocation attribute. */ - tmp_ino = ntfs_index_iget(vol->quota_ino, Q, 2); - if (IS_ERR(tmp_ino)) { - ntfs_error(vol->sb, "Failed to load $Quota/$Q index."); - return false; - } - vol->quota_q_ino = tmp_ino; - ntfs_debug("Done."); - return true; -} - -/** - * load_and_init_usnjrnl - load and setup the transaction log if present - * @vol: ntfs super block describing device whose usnjrnl file to load - * - * Return 'true' on success or 'false' on error. - * - * If $UsnJrnl is not present or in the process of being disabled, we set - * NVolUsnJrnlStamped() and return success. - * - * If the $UsnJrnl $DATA/$J attribute has a size equal to the lowest valid usn, - * i.e. transaction logging has only just been enabled or the journal has been - * stamped and nothing has been logged since, we also set NVolUsnJrnlStamped() - * and return success. - */ -static bool load_and_init_usnjrnl(ntfs_volume *vol) -{ - MFT_REF mref; - struct inode *tmp_ino; - ntfs_inode *tmp_ni; - struct page *page; - ntfs_name *name = NULL; - USN_HEADER *uh; - static const ntfschar UsnJrnl[9] = { cpu_to_le16('$'), - cpu_to_le16('U'), cpu_to_le16('s'), - cpu_to_le16('n'), cpu_to_le16('J'), - cpu_to_le16('r'), cpu_to_le16('n'), - cpu_to_le16('l'), 0 }; - static ntfschar Max[5] = { cpu_to_le16('$'), - cpu_to_le16('M'), cpu_to_le16('a'), - cpu_to_le16('x'), 0 }; - static ntfschar J[3] = { cpu_to_le16('$'), - cpu_to_le16('J'), 0 }; - - ntfs_debug("Entering."); - /* - * Find the inode number for the transaction log file by looking up the - * filename $UsnJrnl in the extended system files directory $Extend. - */ - inode_lock(vol->extend_ino); - mref = ntfs_lookup_inode_by_name(NTFS_I(vol->extend_ino), UsnJrnl, 8, - &name); - inode_unlock(vol->extend_ino); - if (IS_ERR_MREF(mref)) { - /* - * If the file does not exist, transaction logging is disabled, - * just return success. - */ - if (MREF_ERR(mref) == -ENOENT) { - ntfs_debug("$UsnJrnl not present. Volume does not " - "have transaction logging enabled."); -not_enabled: - /* - * No need to try to stamp the transaction log if - * transaction logging is not enabled. - */ - NVolSetUsnJrnlStamped(vol); - return true; - } - /* A real error occurred. */ - ntfs_error(vol->sb, "Failed to find inode number for " - "$UsnJrnl."); - return false; - } - /* We do not care for the type of match that was found. */ - kfree(name); - /* Get the inode. */ - tmp_ino = ntfs_iget(vol->sb, MREF(mref)); - if (IS_ERR(tmp_ino) || unlikely(is_bad_inode(tmp_ino))) { - if (!IS_ERR(tmp_ino)) - iput(tmp_ino); - ntfs_error(vol->sb, "Failed to load $UsnJrnl."); - return false; - } - vol->usnjrnl_ino = tmp_ino; - /* - * If the transaction log is in the process of being deleted, we can - * ignore it. - */ - if (unlikely(vol->vol_flags & VOLUME_DELETE_USN_UNDERWAY)) { - ntfs_debug("$UsnJrnl in the process of being disabled. " - "Volume does not have transaction logging " - "enabled."); - goto not_enabled; - } - /* Get the $DATA/$Max attribute. */ - tmp_ino = ntfs_attr_iget(vol->usnjrnl_ino, AT_DATA, Max, 4); - if (IS_ERR(tmp_ino)) { - ntfs_error(vol->sb, "Failed to load $UsnJrnl/$DATA/$Max " - "attribute."); - return false; - } - vol->usnjrnl_max_ino = tmp_ino; - if (unlikely(i_size_read(tmp_ino) < sizeof(USN_HEADER))) { - ntfs_error(vol->sb, "Found corrupt $UsnJrnl/$DATA/$Max " - "attribute (size is 0x%llx but should be at " - "least 0x%zx bytes).", i_size_read(tmp_ino), - sizeof(USN_HEADER)); - return false; - } - /* Get the $DATA/$J attribute. */ - tmp_ino = ntfs_attr_iget(vol->usnjrnl_ino, AT_DATA, J, 2); - if (IS_ERR(tmp_ino)) { - ntfs_error(vol->sb, "Failed to load $UsnJrnl/$DATA/$J " - "attribute."); - return false; - } - vol->usnjrnl_j_ino = tmp_ino; - /* Verify $J is non-resident and sparse. */ - tmp_ni = NTFS_I(vol->usnjrnl_j_ino); - if (unlikely(!NInoNonResident(tmp_ni) || !NInoSparse(tmp_ni))) { - ntfs_error(vol->sb, "$UsnJrnl/$DATA/$J attribute is resident " - "and/or not sparse."); - return false; - } - /* Read the USN_HEADER from $DATA/$Max. */ - page = ntfs_map_page(vol->usnjrnl_max_ino->i_mapping, 0); - if (IS_ERR(page)) { - ntfs_error(vol->sb, "Failed to read from $UsnJrnl/$DATA/$Max " - "attribute."); - return false; - } - uh = (USN_HEADER*)page_address(page); - /* Sanity check the $Max. */ - if (unlikely(sle64_to_cpu(uh->allocation_delta) > - sle64_to_cpu(uh->maximum_size))) { - ntfs_error(vol->sb, "Allocation delta (0x%llx) exceeds " - "maximum size (0x%llx). $UsnJrnl is corrupt.", - (long long)sle64_to_cpu(uh->allocation_delta), - (long long)sle64_to_cpu(uh->maximum_size)); - ntfs_unmap_page(page); - return false; - } - /* - * If the transaction log has been stamped and nothing has been written - * to it since, we do not need to stamp it. - */ - if (unlikely(sle64_to_cpu(uh->lowest_valid_usn) >= - i_size_read(vol->usnjrnl_j_ino))) { - if (likely(sle64_to_cpu(uh->lowest_valid_usn) == - i_size_read(vol->usnjrnl_j_ino))) { - ntfs_unmap_page(page); - ntfs_debug("$UsnJrnl is enabled but nothing has been " - "logged since it was last stamped. " - "Treating this as if the volume does " - "not have transaction logging " - "enabled."); - goto not_enabled; - } - ntfs_error(vol->sb, "$UsnJrnl has lowest valid usn (0x%llx) " - "which is out of bounds (0x%llx). $UsnJrnl " - "is corrupt.", - (long long)sle64_to_cpu(uh->lowest_valid_usn), - i_size_read(vol->usnjrnl_j_ino)); - ntfs_unmap_page(page); - return false; - } - ntfs_unmap_page(page); - ntfs_debug("Done."); - return true; -} - -/** - * load_and_init_attrdef - load the attribute definitions table for a volume - * @vol: ntfs super block describing device whose attrdef to load - * - * Return 'true' on success or 'false' on error. - */ -static bool load_and_init_attrdef(ntfs_volume *vol) -{ - loff_t i_size; - struct super_block *sb = vol->sb; - struct inode *ino; - struct page *page; - pgoff_t index, max_index; - unsigned int size; - - ntfs_debug("Entering."); - /* Read attrdef table and setup vol->attrdef and vol->attrdef_size. */ - ino = ntfs_iget(sb, FILE_AttrDef); - if (IS_ERR(ino) || is_bad_inode(ino)) { - if (!IS_ERR(ino)) - iput(ino); - goto failed; - } - NInoSetSparseDisabled(NTFS_I(ino)); - /* The size of FILE_AttrDef must be above 0 and fit inside 31 bits. */ - i_size = i_size_read(ino); - if (i_size <= 0 || i_size > 0x7fffffff) - goto iput_failed; - vol->attrdef = (ATTR_DEF*)ntfs_malloc_nofs(i_size); - if (!vol->attrdef) - goto iput_failed; - index = 0; - max_index = i_size >> PAGE_SHIFT; - size = PAGE_SIZE; - while (index < max_index) { - /* Read the attrdef table and copy it into the linear buffer. */ -read_partial_attrdef_page: - page = ntfs_map_page(ino->i_mapping, index); - if (IS_ERR(page)) - goto free_iput_failed; - memcpy((u8*)vol->attrdef + (index++ << PAGE_SHIFT), - page_address(page), size); - ntfs_unmap_page(page); - } - if (size == PAGE_SIZE) { - size = i_size & ~PAGE_MASK; - if (size) - goto read_partial_attrdef_page; - } - vol->attrdef_size = i_size; - ntfs_debug("Read %llu bytes from $AttrDef.", i_size); - iput(ino); - return true; -free_iput_failed: - ntfs_free(vol->attrdef); - vol->attrdef = NULL; -iput_failed: - iput(ino); -failed: - ntfs_error(sb, "Failed to initialize attribute definition table."); - return false; -} - -#endif /* NTFS_RW */ - -/** - * load_and_init_upcase - load the upcase table for an ntfs volume - * @vol: ntfs super block describing device whose upcase to load - * - * Return 'true' on success or 'false' on error. - */ -static bool load_and_init_upcase(ntfs_volume *vol) -{ - loff_t i_size; - struct super_block *sb = vol->sb; - struct inode *ino; - struct page *page; - pgoff_t index, max_index; - unsigned int size; - int i, max; - - ntfs_debug("Entering."); - /* Read upcase table and setup vol->upcase and vol->upcase_len. */ - ino = ntfs_iget(sb, FILE_UpCase); - if (IS_ERR(ino) || is_bad_inode(ino)) { - if (!IS_ERR(ino)) - iput(ino); - goto upcase_failed; - } - /* - * The upcase size must not be above 64k Unicode characters, must not - * be zero and must be a multiple of sizeof(ntfschar). - */ - i_size = i_size_read(ino); - if (!i_size || i_size & (sizeof(ntfschar) - 1) || - i_size > 64ULL * 1024 * sizeof(ntfschar)) - goto iput_upcase_failed; - vol->upcase = (ntfschar*)ntfs_malloc_nofs(i_size); - if (!vol->upcase) - goto iput_upcase_failed; - index = 0; - max_index = i_size >> PAGE_SHIFT; - size = PAGE_SIZE; - while (index < max_index) { - /* Read the upcase table and copy it into the linear buffer. */ -read_partial_upcase_page: - page = ntfs_map_page(ino->i_mapping, index); - if (IS_ERR(page)) - goto iput_upcase_failed; - memcpy((char*)vol->upcase + (index++ << PAGE_SHIFT), - page_address(page), size); - ntfs_unmap_page(page); - } - if (size == PAGE_SIZE) { - size = i_size & ~PAGE_MASK; - if (size) - goto read_partial_upcase_page; - } - vol->upcase_len = i_size >> UCHAR_T_SIZE_BITS; - ntfs_debug("Read %llu bytes from $UpCase (expected %zu bytes).", - i_size, 64 * 1024 * sizeof(ntfschar)); - iput(ino); - mutex_lock(&ntfs_lock); - if (!default_upcase) { - ntfs_debug("Using volume specified $UpCase since default is " - "not present."); - mutex_unlock(&ntfs_lock); - return true; - } - max = default_upcase_len; - if (max > vol->upcase_len) - max = vol->upcase_len; - for (i = 0; i < max; i++) - if (vol->upcase[i] != default_upcase[i]) - break; - if (i == max) { - ntfs_free(vol->upcase); - vol->upcase = default_upcase; - vol->upcase_len = max; - ntfs_nr_upcase_users++; - mutex_unlock(&ntfs_lock); - ntfs_debug("Volume specified $UpCase matches default. Using " - "default."); - return true; - } - mutex_unlock(&ntfs_lock); - ntfs_debug("Using volume specified $UpCase since it does not match " - "the default."); - return true; -iput_upcase_failed: - iput(ino); - ntfs_free(vol->upcase); - vol->upcase = NULL; -upcase_failed: - mutex_lock(&ntfs_lock); - if (default_upcase) { - vol->upcase = default_upcase; - vol->upcase_len = default_upcase_len; - ntfs_nr_upcase_users++; - mutex_unlock(&ntfs_lock); - ntfs_error(sb, "Failed to load $UpCase from the volume. Using " - "default."); - return true; - } - mutex_unlock(&ntfs_lock); - ntfs_error(sb, "Failed to initialize upcase table."); - return false; -} - -/* - * The lcn and mft bitmap inodes are NTFS-internal inodes with - * their own special locking rules: - */ -static struct lock_class_key - lcnbmp_runlist_lock_key, lcnbmp_mrec_lock_key, - mftbmp_runlist_lock_key, mftbmp_mrec_lock_key; - -/** - * load_system_files - open the system files using normal functions - * @vol: ntfs super block describing device whose system files to load - * - * Open the system files with normal access functions and complete setting up - * the ntfs super block @vol. - * - * Return 'true' on success or 'false' on error. - */ -static bool load_system_files(ntfs_volume *vol) -{ - struct super_block *sb = vol->sb; - MFT_RECORD *m; - VOLUME_INFORMATION *vi; - ntfs_attr_search_ctx *ctx; -#ifdef NTFS_RW - RESTART_PAGE_HEADER *rp; - int err; -#endif /* NTFS_RW */ - - ntfs_debug("Entering."); -#ifdef NTFS_RW - /* Get mft mirror inode compare the contents of $MFT and $MFTMirr. */ - if (!load_and_init_mft_mirror(vol) || !check_mft_mirror(vol)) { - static const char *es1 = "Failed to load $MFTMirr"; - static const char *es2 = "$MFTMirr does not match $MFT"; - static const char *es3 = ". Run ntfsfix and/or chkdsk."; - - /* If a read-write mount, convert it to a read-only mount. */ - if (!sb_rdonly(sb)) { - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=" - "continue nor on_errors=" - "remount-ro was specified%s", - !vol->mftmirr_ino ? es1 : es2, - es3); - goto iput_mirr_err_out; - } - sb->s_flags |= SB_RDONLY; - ntfs_error(sb, "%s. Mounting read-only%s", - !vol->mftmirr_ino ? es1 : es2, es3); - } else - ntfs_warning(sb, "%s. Will not be able to remount " - "read-write%s", - !vol->mftmirr_ino ? es1 : es2, es3); - /* This will prevent a read-write remount. */ - NVolSetErrors(vol); - } -#endif /* NTFS_RW */ - /* Get mft bitmap attribute inode. */ - vol->mftbmp_ino = ntfs_attr_iget(vol->mft_ino, AT_BITMAP, NULL, 0); - if (IS_ERR(vol->mftbmp_ino)) { - ntfs_error(sb, "Failed to load $MFT/$BITMAP attribute."); - goto iput_mirr_err_out; - } - lockdep_set_class(&NTFS_I(vol->mftbmp_ino)->runlist.lock, - &mftbmp_runlist_lock_key); - lockdep_set_class(&NTFS_I(vol->mftbmp_ino)->mrec_lock, - &mftbmp_mrec_lock_key); - /* Read upcase table and setup @vol->upcase and @vol->upcase_len. */ - if (!load_and_init_upcase(vol)) - goto iput_mftbmp_err_out; -#ifdef NTFS_RW - /* - * Read attribute definitions table and setup @vol->attrdef and - * @vol->attrdef_size. - */ - if (!load_and_init_attrdef(vol)) - goto iput_upcase_err_out; -#endif /* NTFS_RW */ - /* - * Get the cluster allocation bitmap inode and verify the size, no - * need for any locking at this stage as we are already running - * exclusively as we are mount in progress task. - */ - vol->lcnbmp_ino = ntfs_iget(sb, FILE_Bitmap); - if (IS_ERR(vol->lcnbmp_ino) || is_bad_inode(vol->lcnbmp_ino)) { - if (!IS_ERR(vol->lcnbmp_ino)) - iput(vol->lcnbmp_ino); - goto bitmap_failed; - } - lockdep_set_class(&NTFS_I(vol->lcnbmp_ino)->runlist.lock, - &lcnbmp_runlist_lock_key); - lockdep_set_class(&NTFS_I(vol->lcnbmp_ino)->mrec_lock, - &lcnbmp_mrec_lock_key); - - NInoSetSparseDisabled(NTFS_I(vol->lcnbmp_ino)); - if ((vol->nr_clusters + 7) >> 3 > i_size_read(vol->lcnbmp_ino)) { - iput(vol->lcnbmp_ino); -bitmap_failed: - ntfs_error(sb, "Failed to load $Bitmap."); - goto iput_attrdef_err_out; - } - /* - * Get the volume inode and setup our cache of the volume flags and - * version. - */ - vol->vol_ino = ntfs_iget(sb, FILE_Volume); - if (IS_ERR(vol->vol_ino) || is_bad_inode(vol->vol_ino)) { - if (!IS_ERR(vol->vol_ino)) - iput(vol->vol_ino); -volume_failed: - ntfs_error(sb, "Failed to load $Volume."); - goto iput_lcnbmp_err_out; - } - m = map_mft_record(NTFS_I(vol->vol_ino)); - if (IS_ERR(m)) { -iput_volume_failed: - iput(vol->vol_ino); - goto volume_failed; - } - if (!(ctx = ntfs_attr_get_search_ctx(NTFS_I(vol->vol_ino), m))) { - ntfs_error(sb, "Failed to get attribute search context."); - goto get_ctx_vol_failed; - } - if (ntfs_attr_lookup(AT_VOLUME_INFORMATION, NULL, 0, 0, 0, NULL, 0, - ctx) || ctx->attr->non_resident || ctx->attr->flags) { -err_put_vol: - ntfs_attr_put_search_ctx(ctx); -get_ctx_vol_failed: - unmap_mft_record(NTFS_I(vol->vol_ino)); - goto iput_volume_failed; - } - vi = (VOLUME_INFORMATION*)((char*)ctx->attr + - le16_to_cpu(ctx->attr->data.resident.value_offset)); - /* Some bounds checks. */ - if ((u8*)vi < (u8*)ctx->attr || (u8*)vi + - le32_to_cpu(ctx->attr->data.resident.value_length) > - (u8*)ctx->attr + le32_to_cpu(ctx->attr->length)) - goto err_put_vol; - /* Copy the volume flags and version to the ntfs_volume structure. */ - vol->vol_flags = vi->flags; - vol->major_ver = vi->major_ver; - vol->minor_ver = vi->minor_ver; - ntfs_attr_put_search_ctx(ctx); - unmap_mft_record(NTFS_I(vol->vol_ino)); - pr_info("volume version %i.%i.\n", vol->major_ver, - vol->minor_ver); - if (vol->major_ver < 3 && NVolSparseEnabled(vol)) { - ntfs_warning(vol->sb, "Disabling sparse support due to NTFS " - "volume version %i.%i (need at least version " - "3.0).", vol->major_ver, vol->minor_ver); - NVolClearSparseEnabled(vol); - } -#ifdef NTFS_RW - /* Make sure that no unsupported volume flags are set. */ - if (vol->vol_flags & VOLUME_MUST_MOUNT_RO_MASK) { - static const char *es1a = "Volume is dirty"; - static const char *es1b = "Volume has been modified by chkdsk"; - static const char *es1c = "Volume has unsupported flags set"; - static const char *es2a = ". Run chkdsk and mount in Windows."; - static const char *es2b = ". Mount in Windows."; - const char *es1, *es2; - - es2 = es2a; - if (vol->vol_flags & VOLUME_IS_DIRTY) - es1 = es1a; - else if (vol->vol_flags & VOLUME_MODIFIED_BY_CHKDSK) { - es1 = es1b; - es2 = es2b; - } else { - es1 = es1c; - ntfs_warning(sb, "Unsupported volume flags 0x%x " - "encountered.", - (unsigned)le16_to_cpu(vol->vol_flags)); - } - /* If a read-write mount, convert it to a read-only mount. */ - if (!sb_rdonly(sb)) { - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=" - "continue nor on_errors=" - "remount-ro was specified%s", - es1, es2); - goto iput_vol_err_out; - } - sb->s_flags |= SB_RDONLY; - ntfs_error(sb, "%s. Mounting read-only%s", es1, es2); - } else - ntfs_warning(sb, "%s. Will not be able to remount " - "read-write%s", es1, es2); - /* - * Do not set NVolErrors() because ntfs_remount() re-checks the - * flags which we need to do in case any flags have changed. - */ - } - /* - * Get the inode for the logfile, check it and determine if the volume - * was shutdown cleanly. - */ - rp = NULL; - if (!load_and_check_logfile(vol, &rp) || - !ntfs_is_logfile_clean(vol->logfile_ino, rp)) { - static const char *es1a = "Failed to load $LogFile"; - static const char *es1b = "$LogFile is not clean"; - static const char *es2 = ". Mount in Windows."; - const char *es1; - - es1 = !vol->logfile_ino ? es1a : es1b; - /* If a read-write mount, convert it to a read-only mount. */ - if (!sb_rdonly(sb)) { - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=" - "continue nor on_errors=" - "remount-ro was specified%s", - es1, es2); - if (vol->logfile_ino) { - BUG_ON(!rp); - ntfs_free(rp); - } - goto iput_logfile_err_out; - } - sb->s_flags |= SB_RDONLY; - ntfs_error(sb, "%s. Mounting read-only%s", es1, es2); - } else - ntfs_warning(sb, "%s. Will not be able to remount " - "read-write%s", es1, es2); - /* This will prevent a read-write remount. */ - NVolSetErrors(vol); - } - ntfs_free(rp); -#endif /* NTFS_RW */ - /* Get the root directory inode so we can do path lookups. */ - vol->root_ino = ntfs_iget(sb, FILE_root); - if (IS_ERR(vol->root_ino) || is_bad_inode(vol->root_ino)) { - if (!IS_ERR(vol->root_ino)) - iput(vol->root_ino); - ntfs_error(sb, "Failed to load root directory."); - goto iput_logfile_err_out; - } -#ifdef NTFS_RW - /* - * Check if Windows is suspended to disk on the target volume. If it - * is hibernated, we must not write *anything* to the disk so set - * NVolErrors() without setting the dirty volume flag and mount - * read-only. This will prevent read-write remounting and it will also - * prevent all writes. - */ - err = check_windows_hibernation_status(vol); - if (unlikely(err)) { - static const char *es1a = "Failed to determine if Windows is " - "hibernated"; - static const char *es1b = "Windows is hibernated"; - static const char *es2 = ". Run chkdsk."; - const char *es1; - - es1 = err < 0 ? es1a : es1b; - /* If a read-write mount, convert it to a read-only mount. */ - if (!sb_rdonly(sb)) { - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=" - "continue nor on_errors=" - "remount-ro was specified%s", - es1, es2); - goto iput_root_err_out; - } - sb->s_flags |= SB_RDONLY; - ntfs_error(sb, "%s. Mounting read-only%s", es1, es2); - } else - ntfs_warning(sb, "%s. Will not be able to remount " - "read-write%s", es1, es2); - /* This will prevent a read-write remount. */ - NVolSetErrors(vol); - } - /* If (still) a read-write mount, mark the volume dirty. */ - if (!sb_rdonly(sb) && ntfs_set_volume_flags(vol, VOLUME_IS_DIRTY)) { - static const char *es1 = "Failed to set dirty bit in volume " - "information flags"; - static const char *es2 = ". Run chkdsk."; - - /* Convert to a read-only mount. */ - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=continue nor " - "on_errors=remount-ro was specified%s", - es1, es2); - goto iput_root_err_out; - } - ntfs_error(sb, "%s. Mounting read-only%s", es1, es2); - sb->s_flags |= SB_RDONLY; - /* - * Do not set NVolErrors() because ntfs_remount() might manage - * to set the dirty flag in which case all would be well. - */ - } -#if 0 - // TODO: Enable this code once we start modifying anything that is - // different between NTFS 1.2 and 3.x... - /* - * If (still) a read-write mount, set the NT4 compatibility flag on - * newer NTFS version volumes. - */ - if (!(sb->s_flags & SB_RDONLY) && (vol->major_ver > 1) && - ntfs_set_volume_flags(vol, VOLUME_MOUNTED_ON_NT4)) { - static const char *es1 = "Failed to set NT4 compatibility flag"; - static const char *es2 = ". Run chkdsk."; - - /* Convert to a read-only mount. */ - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=continue nor " - "on_errors=remount-ro was specified%s", - es1, es2); - goto iput_root_err_out; - } - ntfs_error(sb, "%s. Mounting read-only%s", es1, es2); - sb->s_flags |= SB_RDONLY; - NVolSetErrors(vol); - } -#endif - /* If (still) a read-write mount, empty the logfile. */ - if (!sb_rdonly(sb) && !ntfs_empty_logfile(vol->logfile_ino)) { - static const char *es1 = "Failed to empty $LogFile"; - static const char *es2 = ". Mount in Windows."; - - /* Convert to a read-only mount. */ - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=continue nor " - "on_errors=remount-ro was specified%s", - es1, es2); - goto iput_root_err_out; - } - ntfs_error(sb, "%s. Mounting read-only%s", es1, es2); - sb->s_flags |= SB_RDONLY; - NVolSetErrors(vol); - } -#endif /* NTFS_RW */ - /* If on NTFS versions before 3.0, we are done. */ - if (unlikely(vol->major_ver < 3)) - return true; - /* NTFS 3.0+ specific initialization. */ - /* Get the security descriptors inode. */ - vol->secure_ino = ntfs_iget(sb, FILE_Secure); - if (IS_ERR(vol->secure_ino) || is_bad_inode(vol->secure_ino)) { - if (!IS_ERR(vol->secure_ino)) - iput(vol->secure_ino); - ntfs_error(sb, "Failed to load $Secure."); - goto iput_root_err_out; - } - // TODO: Initialize security. - /* Get the extended system files' directory inode. */ - vol->extend_ino = ntfs_iget(sb, FILE_Extend); - if (IS_ERR(vol->extend_ino) || is_bad_inode(vol->extend_ino) || - !S_ISDIR(vol->extend_ino->i_mode)) { - if (!IS_ERR(vol->extend_ino)) - iput(vol->extend_ino); - ntfs_error(sb, "Failed to load $Extend."); - goto iput_sec_err_out; - } -#ifdef NTFS_RW - /* Find the quota file, load it if present, and set it up. */ - if (!load_and_init_quota(vol)) { - static const char *es1 = "Failed to load $Quota"; - static const char *es2 = ". Run chkdsk."; - - /* If a read-write mount, convert it to a read-only mount. */ - if (!sb_rdonly(sb)) { - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=" - "continue nor on_errors=" - "remount-ro was specified%s", - es1, es2); - goto iput_quota_err_out; - } - sb->s_flags |= SB_RDONLY; - ntfs_error(sb, "%s. Mounting read-only%s", es1, es2); - } else - ntfs_warning(sb, "%s. Will not be able to remount " - "read-write%s", es1, es2); - /* This will prevent a read-write remount. */ - NVolSetErrors(vol); - } - /* If (still) a read-write mount, mark the quotas out of date. */ - if (!sb_rdonly(sb) && !ntfs_mark_quotas_out_of_date(vol)) { - static const char *es1 = "Failed to mark quotas out of date"; - static const char *es2 = ". Run chkdsk."; - - /* Convert to a read-only mount. */ - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=continue nor " - "on_errors=remount-ro was specified%s", - es1, es2); - goto iput_quota_err_out; - } - ntfs_error(sb, "%s. Mounting read-only%s", es1, es2); - sb->s_flags |= SB_RDONLY; - NVolSetErrors(vol); - } - /* - * Find the transaction log file ($UsnJrnl), load it if present, check - * it, and set it up. - */ - if (!load_and_init_usnjrnl(vol)) { - static const char *es1 = "Failed to load $UsnJrnl"; - static const char *es2 = ". Run chkdsk."; - - /* If a read-write mount, convert it to a read-only mount. */ - if (!sb_rdonly(sb)) { - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=" - "continue nor on_errors=" - "remount-ro was specified%s", - es1, es2); - goto iput_usnjrnl_err_out; - } - sb->s_flags |= SB_RDONLY; - ntfs_error(sb, "%s. Mounting read-only%s", es1, es2); - } else - ntfs_warning(sb, "%s. Will not be able to remount " - "read-write%s", es1, es2); - /* This will prevent a read-write remount. */ - NVolSetErrors(vol); - } - /* If (still) a read-write mount, stamp the transaction log. */ - if (!sb_rdonly(sb) && !ntfs_stamp_usnjrnl(vol)) { - static const char *es1 = "Failed to stamp transaction log " - "($UsnJrnl)"; - static const char *es2 = ". Run chkdsk."; - - /* Convert to a read-only mount. */ - if (!(vol->on_errors & (ON_ERRORS_REMOUNT_RO | - ON_ERRORS_CONTINUE))) { - ntfs_error(sb, "%s and neither on_errors=continue nor " - "on_errors=remount-ro was specified%s", - es1, es2); - goto iput_usnjrnl_err_out; - } - ntfs_error(sb, "%s. Mounting read-only%s", es1, es2); - sb->s_flags |= SB_RDONLY; - NVolSetErrors(vol); - } -#endif /* NTFS_RW */ - return true; -#ifdef NTFS_RW -iput_usnjrnl_err_out: - iput(vol->usnjrnl_j_ino); - iput(vol->usnjrnl_max_ino); - iput(vol->usnjrnl_ino); -iput_quota_err_out: - iput(vol->quota_q_ino); - iput(vol->quota_ino); - iput(vol->extend_ino); -#endif /* NTFS_RW */ -iput_sec_err_out: - iput(vol->secure_ino); -iput_root_err_out: - iput(vol->root_ino); -iput_logfile_err_out: -#ifdef NTFS_RW - iput(vol->logfile_ino); -iput_vol_err_out: -#endif /* NTFS_RW */ - iput(vol->vol_ino); -iput_lcnbmp_err_out: - iput(vol->lcnbmp_ino); -iput_attrdef_err_out: - vol->attrdef_size = 0; - if (vol->attrdef) { - ntfs_free(vol->attrdef); - vol->attrdef = NULL; - } -#ifdef NTFS_RW -iput_upcase_err_out: -#endif /* NTFS_RW */ - vol->upcase_len = 0; - mutex_lock(&ntfs_lock); - if (vol->upcase == default_upcase) { - ntfs_nr_upcase_users--; - vol->upcase = NULL; - } - mutex_unlock(&ntfs_lock); - if (vol->upcase) { - ntfs_free(vol->upcase); - vol->upcase = NULL; - } -iput_mftbmp_err_out: - iput(vol->mftbmp_ino); -iput_mirr_err_out: -#ifdef NTFS_RW - iput(vol->mftmirr_ino); -#endif /* NTFS_RW */ - return false; -} - -/** - * ntfs_put_super - called by the vfs to unmount a volume - * @sb: vfs superblock of volume to unmount - * - * ntfs_put_super() is called by the VFS (from fs/super.c::do_umount()) when - * the volume is being unmounted (umount system call has been invoked) and it - * releases all inodes and memory belonging to the NTFS specific part of the - * super block. - */ -static void ntfs_put_super(struct super_block *sb) -{ - ntfs_volume *vol = NTFS_SB(sb); - - ntfs_debug("Entering."); - -#ifdef NTFS_RW - /* - * Commit all inodes while they are still open in case some of them - * cause others to be dirtied. - */ - ntfs_commit_inode(vol->vol_ino); - - /* NTFS 3.0+ specific. */ - if (vol->major_ver >= 3) { - if (vol->usnjrnl_j_ino) - ntfs_commit_inode(vol->usnjrnl_j_ino); - if (vol->usnjrnl_max_ino) - ntfs_commit_inode(vol->usnjrnl_max_ino); - if (vol->usnjrnl_ino) - ntfs_commit_inode(vol->usnjrnl_ino); - if (vol->quota_q_ino) - ntfs_commit_inode(vol->quota_q_ino); - if (vol->quota_ino) - ntfs_commit_inode(vol->quota_ino); - if (vol->extend_ino) - ntfs_commit_inode(vol->extend_ino); - if (vol->secure_ino) - ntfs_commit_inode(vol->secure_ino); - } - - ntfs_commit_inode(vol->root_ino); - - down_write(&vol->lcnbmp_lock); - ntfs_commit_inode(vol->lcnbmp_ino); - up_write(&vol->lcnbmp_lock); - - down_write(&vol->mftbmp_lock); - ntfs_commit_inode(vol->mftbmp_ino); - up_write(&vol->mftbmp_lock); - - if (vol->logfile_ino) - ntfs_commit_inode(vol->logfile_ino); - - if (vol->mftmirr_ino) - ntfs_commit_inode(vol->mftmirr_ino); - ntfs_commit_inode(vol->mft_ino); - - /* - * If a read-write mount and no volume errors have occurred, mark the - * volume clean. Also, re-commit all affected inodes. - */ - if (!sb_rdonly(sb)) { - if (!NVolErrors(vol)) { - if (ntfs_clear_volume_flags(vol, VOLUME_IS_DIRTY)) - ntfs_warning(sb, "Failed to clear dirty bit " - "in volume information " - "flags. Run chkdsk."); - ntfs_commit_inode(vol->vol_ino); - ntfs_commit_inode(vol->root_ino); - if (vol->mftmirr_ino) - ntfs_commit_inode(vol->mftmirr_ino); - ntfs_commit_inode(vol->mft_ino); - } else { - ntfs_warning(sb, "Volume has errors. Leaving volume " - "marked dirty. Run chkdsk."); - } - } -#endif /* NTFS_RW */ - - iput(vol->vol_ino); - vol->vol_ino = NULL; - - /* NTFS 3.0+ specific clean up. */ - if (vol->major_ver >= 3) { -#ifdef NTFS_RW - if (vol->usnjrnl_j_ino) { - iput(vol->usnjrnl_j_ino); - vol->usnjrnl_j_ino = NULL; - } - if (vol->usnjrnl_max_ino) { - iput(vol->usnjrnl_max_ino); - vol->usnjrnl_max_ino = NULL; - } - if (vol->usnjrnl_ino) { - iput(vol->usnjrnl_ino); - vol->usnjrnl_ino = NULL; - } - if (vol->quota_q_ino) { - iput(vol->quota_q_ino); - vol->quota_q_ino = NULL; - } - if (vol->quota_ino) { - iput(vol->quota_ino); - vol->quota_ino = NULL; - } -#endif /* NTFS_RW */ - if (vol->extend_ino) { - iput(vol->extend_ino); - vol->extend_ino = NULL; - } - if (vol->secure_ino) { - iput(vol->secure_ino); - vol->secure_ino = NULL; - } - } - - iput(vol->root_ino); - vol->root_ino = NULL; - - down_write(&vol->lcnbmp_lock); - iput(vol->lcnbmp_ino); - vol->lcnbmp_ino = NULL; - up_write(&vol->lcnbmp_lock); - - down_write(&vol->mftbmp_lock); - iput(vol->mftbmp_ino); - vol->mftbmp_ino = NULL; - up_write(&vol->mftbmp_lock); - -#ifdef NTFS_RW - if (vol->logfile_ino) { - iput(vol->logfile_ino); - vol->logfile_ino = NULL; - } - if (vol->mftmirr_ino) { - /* Re-commit the mft mirror and mft just in case. */ - ntfs_commit_inode(vol->mftmirr_ino); - ntfs_commit_inode(vol->mft_ino); - iput(vol->mftmirr_ino); - vol->mftmirr_ino = NULL; - } - /* - * We should have no dirty inodes left, due to - * mft.c::ntfs_mft_writepage() cleaning all the dirty pages as - * the underlying mft records are written out and cleaned. - */ - ntfs_commit_inode(vol->mft_ino); - write_inode_now(vol->mft_ino, 1); -#endif /* NTFS_RW */ - - iput(vol->mft_ino); - vol->mft_ino = NULL; - - /* Throw away the table of attribute definitions. */ - vol->attrdef_size = 0; - if (vol->attrdef) { - ntfs_free(vol->attrdef); - vol->attrdef = NULL; - } - vol->upcase_len = 0; - /* - * Destroy the global default upcase table if necessary. Also decrease - * the number of upcase users if we are a user. - */ - mutex_lock(&ntfs_lock); - if (vol->upcase == default_upcase) { - ntfs_nr_upcase_users--; - vol->upcase = NULL; - } - if (!ntfs_nr_upcase_users && default_upcase) { - ntfs_free(default_upcase); - default_upcase = NULL; - } - if (vol->cluster_size <= 4096 && !--ntfs_nr_compression_users) - free_compression_buffers(); - mutex_unlock(&ntfs_lock); - if (vol->upcase) { - ntfs_free(vol->upcase); - vol->upcase = NULL; - } - - unload_nls(vol->nls_map); - - sb->s_fs_info = NULL; - kfree(vol); -} - -/** - * get_nr_free_clusters - return the number of free clusters on a volume - * @vol: ntfs volume for which to obtain free cluster count - * - * Calculate the number of free clusters on the mounted NTFS volume @vol. We - * actually calculate the number of clusters in use instead because this - * allows us to not care about partial pages as these will be just zero filled - * and hence not be counted as allocated clusters. - * - * The only particularity is that clusters beyond the end of the logical ntfs - * volume will be marked as allocated to prevent errors which means we have to - * discount those at the end. This is important as the cluster bitmap always - * has a size in multiples of 8 bytes, i.e. up to 63 clusters could be outside - * the logical volume and marked in use when they are not as they do not exist. - * - * If any pages cannot be read we assume all clusters in the erroring pages are - * in use. This means we return an underestimate on errors which is better than - * an overestimate. - */ -static s64 get_nr_free_clusters(ntfs_volume *vol) -{ - s64 nr_free = vol->nr_clusters; - struct address_space *mapping = vol->lcnbmp_ino->i_mapping; - struct page *page; - pgoff_t index, max_index; - - ntfs_debug("Entering."); - /* Serialize accesses to the cluster bitmap. */ - down_read(&vol->lcnbmp_lock); - /* - * Convert the number of bits into bytes rounded up, then convert into - * multiples of PAGE_SIZE, rounding up so that if we have one - * full and one partial page max_index = 2. - */ - max_index = (((vol->nr_clusters + 7) >> 3) + PAGE_SIZE - 1) >> - PAGE_SHIFT; - /* Use multiples of 4 bytes, thus max_size is PAGE_SIZE / 4. */ - ntfs_debug("Reading $Bitmap, max_index = 0x%lx, max_size = 0x%lx.", - max_index, PAGE_SIZE / 4); - for (index = 0; index < max_index; index++) { - unsigned long *kaddr; - - /* - * Read the page from page cache, getting it from backing store - * if necessary, and increment the use count. - */ - page = read_mapping_page(mapping, index, NULL); - /* Ignore pages which errored synchronously. */ - if (IS_ERR(page)) { - ntfs_debug("read_mapping_page() error. Skipping " - "page (index 0x%lx).", index); - nr_free -= PAGE_SIZE * 8; - continue; - } - kaddr = kmap_atomic(page); - /* - * Subtract the number of set bits. If this - * is the last page and it is partial we don't really care as - * it just means we do a little extra work but it won't affect - * the result as all out of range bytes are set to zero by - * ntfs_readpage(). - */ - nr_free -= bitmap_weight(kaddr, - PAGE_SIZE * BITS_PER_BYTE); - kunmap_atomic(kaddr); - put_page(page); - } - ntfs_debug("Finished reading $Bitmap, last index = 0x%lx.", index - 1); - /* - * Fixup for eventual bits outside logical ntfs volume (see function - * description above). - */ - if (vol->nr_clusters & 63) - nr_free += 64 - (vol->nr_clusters & 63); - up_read(&vol->lcnbmp_lock); - /* If errors occurred we may well have gone below zero, fix this. */ - if (nr_free < 0) - nr_free = 0; - ntfs_debug("Exiting."); - return nr_free; -} - -/** - * __get_nr_free_mft_records - return the number of free inodes on a volume - * @vol: ntfs volume for which to obtain free inode count - * @nr_free: number of mft records in filesystem - * @max_index: maximum number of pages containing set bits - * - * Calculate the number of free mft records (inodes) on the mounted NTFS - * volume @vol. We actually calculate the number of mft records in use instead - * because this allows us to not care about partial pages as these will be just - * zero filled and hence not be counted as allocated mft record. - * - * If any pages cannot be read we assume all mft records in the erroring pages - * are in use. This means we return an underestimate on errors which is better - * than an overestimate. - * - * NOTE: Caller must hold mftbmp_lock rw_semaphore for reading or writing. - */ -static unsigned long __get_nr_free_mft_records(ntfs_volume *vol, - s64 nr_free, const pgoff_t max_index) -{ - struct address_space *mapping = vol->mftbmp_ino->i_mapping; - struct page *page; - pgoff_t index; - - ntfs_debug("Entering."); - /* Use multiples of 4 bytes, thus max_size is PAGE_SIZE / 4. */ - ntfs_debug("Reading $MFT/$BITMAP, max_index = 0x%lx, max_size = " - "0x%lx.", max_index, PAGE_SIZE / 4); - for (index = 0; index < max_index; index++) { - unsigned long *kaddr; - - /* - * Read the page from page cache, getting it from backing store - * if necessary, and increment the use count. - */ - page = read_mapping_page(mapping, index, NULL); - /* Ignore pages which errored synchronously. */ - if (IS_ERR(page)) { - ntfs_debug("read_mapping_page() error. Skipping " - "page (index 0x%lx).", index); - nr_free -= PAGE_SIZE * 8; - continue; - } - kaddr = kmap_atomic(page); - /* - * Subtract the number of set bits. If this - * is the last page and it is partial we don't really care as - * it just means we do a little extra work but it won't affect - * the result as all out of range bytes are set to zero by - * ntfs_readpage(). - */ - nr_free -= bitmap_weight(kaddr, - PAGE_SIZE * BITS_PER_BYTE); - kunmap_atomic(kaddr); - put_page(page); - } - ntfs_debug("Finished reading $MFT/$BITMAP, last index = 0x%lx.", - index - 1); - /* If errors occurred we may well have gone below zero, fix this. */ - if (nr_free < 0) - nr_free = 0; - ntfs_debug("Exiting."); - return nr_free; -} - -/** - * ntfs_statfs - return information about mounted NTFS volume - * @dentry: dentry from mounted volume - * @sfs: statfs structure in which to return the information - * - * Return information about the mounted NTFS volume @dentry in the statfs structure - * pointed to by @sfs (this is initialized with zeros before ntfs_statfs is - * called). We interpret the values to be correct of the moment in time at - * which we are called. Most values are variable otherwise and this isn't just - * the free values but the totals as well. For example we can increase the - * total number of file nodes if we run out and we can keep doing this until - * there is no more space on the volume left at all. - * - * Called from vfs_statfs which is used to handle the statfs, fstatfs, and - * ustat system calls. - * - * Return 0 on success or -errno on error. - */ -static int ntfs_statfs(struct dentry *dentry, struct kstatfs *sfs) -{ - struct super_block *sb = dentry->d_sb; - s64 size; - ntfs_volume *vol = NTFS_SB(sb); - ntfs_inode *mft_ni = NTFS_I(vol->mft_ino); - pgoff_t max_index; - unsigned long flags; - - ntfs_debug("Entering."); - /* Type of filesystem. */ - sfs->f_type = NTFS_SB_MAGIC; - /* Optimal transfer block size. */ - sfs->f_bsize = PAGE_SIZE; - /* - * Total data blocks in filesystem in units of f_bsize and since - * inodes are also stored in data blocs ($MFT is a file) this is just - * the total clusters. - */ - sfs->f_blocks = vol->nr_clusters << vol->cluster_size_bits >> - PAGE_SHIFT; - /* Free data blocks in filesystem in units of f_bsize. */ - size = get_nr_free_clusters(vol) << vol->cluster_size_bits >> - PAGE_SHIFT; - if (size < 0LL) - size = 0LL; - /* Free blocks avail to non-superuser, same as above on NTFS. */ - sfs->f_bavail = sfs->f_bfree = size; - /* Serialize accesses to the inode bitmap. */ - down_read(&vol->mftbmp_lock); - read_lock_irqsave(&mft_ni->size_lock, flags); - size = i_size_read(vol->mft_ino) >> vol->mft_record_size_bits; - /* - * Convert the maximum number of set bits into bytes rounded up, then - * convert into multiples of PAGE_SIZE, rounding up so that if we - * have one full and one partial page max_index = 2. - */ - max_index = ((((mft_ni->initialized_size >> vol->mft_record_size_bits) - + 7) >> 3) + PAGE_SIZE - 1) >> PAGE_SHIFT; - read_unlock_irqrestore(&mft_ni->size_lock, flags); - /* Number of inodes in filesystem (at this point in time). */ - sfs->f_files = size; - /* Free inodes in fs (based on current total count). */ - sfs->f_ffree = __get_nr_free_mft_records(vol, size, max_index); - up_read(&vol->mftbmp_lock); - /* - * File system id. This is extremely *nix flavour dependent and even - * within Linux itself all fs do their own thing. I interpret this to - * mean a unique id associated with the mounted fs and not the id - * associated with the filesystem driver, the latter is already given - * by the filesystem type in sfs->f_type. Thus we use the 64-bit - * volume serial number splitting it into two 32-bit parts. We enter - * the least significant 32-bits in f_fsid[0] and the most significant - * 32-bits in f_fsid[1]. - */ - sfs->f_fsid = u64_to_fsid(vol->serial_no); - /* Maximum length of filenames. */ - sfs->f_namelen = NTFS_MAX_NAME_LEN; - return 0; -} - -#ifdef NTFS_RW -static int ntfs_write_inode(struct inode *vi, struct writeback_control *wbc) -{ - return __ntfs_write_inode(vi, wbc->sync_mode == WB_SYNC_ALL); -} -#endif - -/* - * The complete super operations. - */ -static const struct super_operations ntfs_sops = { - .alloc_inode = ntfs_alloc_big_inode, /* VFS: Allocate new inode. */ - .free_inode = ntfs_free_big_inode, /* VFS: Deallocate inode. */ -#ifdef NTFS_RW - .write_inode = ntfs_write_inode, /* VFS: Write dirty inode to - disk. */ -#endif /* NTFS_RW */ - .put_super = ntfs_put_super, /* Syscall: umount. */ - .statfs = ntfs_statfs, /* Syscall: statfs */ - .remount_fs = ntfs_remount, /* Syscall: mount -o remount. */ - .evict_inode = ntfs_evict_big_inode, /* VFS: Called when an inode is - removed from memory. */ - .show_options = ntfs_show_options, /* Show mount options in - proc. */ -}; - -/** - * ntfs_fill_super - mount an ntfs filesystem - * @sb: super block of ntfs filesystem to mount - * @opt: string containing the mount options - * @silent: silence error output - * - * ntfs_fill_super() is called by the VFS to mount the device described by @sb - * with the mount otions in @data with the NTFS filesystem. - * - * If @silent is true, remain silent even if errors are detected. This is used - * during bootup, when the kernel tries to mount the root filesystem with all - * registered filesystems one after the other until one succeeds. This implies - * that all filesystems except the correct one will quite correctly and - * expectedly return an error, but nobody wants to see error messages when in - * fact this is what is supposed to happen. - * - * NOTE: @sb->s_flags contains the mount options flags. - */ -static int ntfs_fill_super(struct super_block *sb, void *opt, const int silent) -{ - ntfs_volume *vol; - struct buffer_head *bh; - struct inode *tmp_ino; - int blocksize, result; - - /* - * We do a pretty difficult piece of bootstrap by reading the - * MFT (and other metadata) from disk into memory. We'll only - * release this metadata during umount, so the locking patterns - * observed during bootstrap do not count. So turn off the - * observation of locking patterns (strictly for this context - * only) while mounting NTFS. [The validator is still active - * otherwise, even for this context: it will for example record - * lock class registrations.] - */ - lockdep_off(); - ntfs_debug("Entering."); -#ifndef NTFS_RW - sb->s_flags |= SB_RDONLY; -#endif /* ! NTFS_RW */ - /* Allocate a new ntfs_volume and place it in sb->s_fs_info. */ - sb->s_fs_info = kmalloc(sizeof(ntfs_volume), GFP_NOFS); - vol = NTFS_SB(sb); - if (!vol) { - if (!silent) - ntfs_error(sb, "Allocation of NTFS volume structure " - "failed. Aborting mount..."); - lockdep_on(); - return -ENOMEM; - } - /* Initialize ntfs_volume structure. */ - *vol = (ntfs_volume) { - .sb = sb, - /* - * Default is group and other don't have any access to files or - * directories while owner has full access. Further, files by - * default are not executable but directories are of course - * browseable. - */ - .fmask = 0177, - .dmask = 0077, - }; - init_rwsem(&vol->mftbmp_lock); - init_rwsem(&vol->lcnbmp_lock); - - /* By default, enable sparse support. */ - NVolSetSparseEnabled(vol); - - /* Important to get the mount options dealt with now. */ - if (!parse_options(vol, (char*)opt)) - goto err_out_now; - - /* We support sector sizes up to the PAGE_SIZE. */ - if (bdev_logical_block_size(sb->s_bdev) > PAGE_SIZE) { - if (!silent) - ntfs_error(sb, "Device has unsupported sector size " - "(%i). The maximum supported sector " - "size on this architecture is %lu " - "bytes.", - bdev_logical_block_size(sb->s_bdev), - PAGE_SIZE); - goto err_out_now; - } - /* - * Setup the device access block size to NTFS_BLOCK_SIZE or the hard - * sector size, whichever is bigger. - */ - blocksize = sb_min_blocksize(sb, NTFS_BLOCK_SIZE); - if (blocksize < NTFS_BLOCK_SIZE) { - if (!silent) - ntfs_error(sb, "Unable to set device block size."); - goto err_out_now; - } - BUG_ON(blocksize != sb->s_blocksize); - ntfs_debug("Set device block size to %i bytes (block size bits %i).", - blocksize, sb->s_blocksize_bits); - /* Determine the size of the device in units of block_size bytes. */ - vol->nr_blocks = sb_bdev_nr_blocks(sb); - if (!vol->nr_blocks) { - if (!silent) - ntfs_error(sb, "Unable to determine device size."); - goto err_out_now; - } - /* Read the boot sector and return unlocked buffer head to it. */ - if (!(bh = read_ntfs_boot_sector(sb, silent))) { - if (!silent) - ntfs_error(sb, "Not an NTFS volume."); - goto err_out_now; - } - /* - * Extract the data from the boot sector and setup the ntfs volume - * using it. - */ - result = parse_ntfs_boot_sector(vol, (NTFS_BOOT_SECTOR*)bh->b_data); - brelse(bh); - if (!result) { - if (!silent) - ntfs_error(sb, "Unsupported NTFS filesystem."); - goto err_out_now; - } - /* - * If the boot sector indicates a sector size bigger than the current - * device block size, switch the device block size to the sector size. - * TODO: It may be possible to support this case even when the set - * below fails, we would just be breaking up the i/o for each sector - * into multiple blocks for i/o purposes but otherwise it should just - * work. However it is safer to leave disabled until someone hits this - * error message and then we can get them to try it without the setting - * so we know for sure that it works. - */ - if (vol->sector_size > blocksize) { - blocksize = sb_set_blocksize(sb, vol->sector_size); - if (blocksize != vol->sector_size) { - if (!silent) - ntfs_error(sb, "Unable to set device block " - "size to sector size (%i).", - vol->sector_size); - goto err_out_now; - } - BUG_ON(blocksize != sb->s_blocksize); - vol->nr_blocks = sb_bdev_nr_blocks(sb); - ntfs_debug("Changed device block size to %i bytes (block size " - "bits %i) to match volume sector size.", - blocksize, sb->s_blocksize_bits); - } - /* Initialize the cluster and mft allocators. */ - ntfs_setup_allocators(vol); - /* Setup remaining fields in the super block. */ - sb->s_magic = NTFS_SB_MAGIC; - /* - * Ntfs allows 63 bits for the file size, i.e. correct would be: - * sb->s_maxbytes = ~0ULL >> 1; - * But the kernel uses a long as the page cache page index which on - * 32-bit architectures is only 32-bits. MAX_LFS_FILESIZE is kernel - * defined to the maximum the page cache page index can cope with - * without overflowing the index or to 2^63 - 1, whichever is smaller. - */ - sb->s_maxbytes = MAX_LFS_FILESIZE; - /* Ntfs measures time in 100ns intervals. */ - sb->s_time_gran = 100; - /* - * Now load the metadata required for the page cache and our address - * space operations to function. We do this by setting up a specialised - * read_inode method and then just calling the normal iget() to obtain - * the inode for $MFT which is sufficient to allow our normal inode - * operations and associated address space operations to function. - */ - sb->s_op = &ntfs_sops; - tmp_ino = new_inode(sb); - if (!tmp_ino) { - if (!silent) - ntfs_error(sb, "Failed to load essential metadata."); - goto err_out_now; - } - tmp_ino->i_ino = FILE_MFT; - insert_inode_hash(tmp_ino); - if (ntfs_read_inode_mount(tmp_ino) < 0) { - if (!silent) - ntfs_error(sb, "Failed to load essential metadata."); - goto iput_tmp_ino_err_out_now; - } - mutex_lock(&ntfs_lock); - /* - * The current mount is a compression user if the cluster size is - * less than or equal 4kiB. - */ - if (vol->cluster_size <= 4096 && !ntfs_nr_compression_users++) { - result = allocate_compression_buffers(); - if (result) { - ntfs_error(NULL, "Failed to allocate buffers " - "for compression engine."); - ntfs_nr_compression_users--; - mutex_unlock(&ntfs_lock); - goto iput_tmp_ino_err_out_now; - } - } - /* - * Generate the global default upcase table if necessary. Also - * temporarily increment the number of upcase users to avoid race - * conditions with concurrent (u)mounts. - */ - if (!default_upcase) - default_upcase = generate_default_upcase(); - ntfs_nr_upcase_users++; - mutex_unlock(&ntfs_lock); - /* - * From now on, ignore @silent parameter. If we fail below this line, - * it will be due to a corrupt fs or a system error, so we report it. - */ - /* - * Open the system files with normal access functions and complete - * setting up the ntfs super block. - */ - if (!load_system_files(vol)) { - ntfs_error(sb, "Failed to load system files."); - goto unl_upcase_iput_tmp_ino_err_out_now; - } - - /* We grab a reference, simulating an ntfs_iget(). */ - ihold(vol->root_ino); - if ((sb->s_root = d_make_root(vol->root_ino))) { - ntfs_debug("Exiting, status successful."); - /* Release the default upcase if it has no users. */ - mutex_lock(&ntfs_lock); - if (!--ntfs_nr_upcase_users && default_upcase) { - ntfs_free(default_upcase); - default_upcase = NULL; - } - mutex_unlock(&ntfs_lock); - sb->s_export_op = &ntfs_export_ops; - lockdep_on(); - return 0; - } - ntfs_error(sb, "Failed to allocate root directory."); - /* Clean up after the successful load_system_files() call from above. */ - // TODO: Use ntfs_put_super() instead of repeating all this code... - // FIXME: Should mark the volume clean as the error is most likely - // -ENOMEM. - iput(vol->vol_ino); - vol->vol_ino = NULL; - /* NTFS 3.0+ specific clean up. */ - if (vol->major_ver >= 3) { -#ifdef NTFS_RW - if (vol->usnjrnl_j_ino) { - iput(vol->usnjrnl_j_ino); - vol->usnjrnl_j_ino = NULL; - } - if (vol->usnjrnl_max_ino) { - iput(vol->usnjrnl_max_ino); - vol->usnjrnl_max_ino = NULL; - } - if (vol->usnjrnl_ino) { - iput(vol->usnjrnl_ino); - vol->usnjrnl_ino = NULL; - } - if (vol->quota_q_ino) { - iput(vol->quota_q_ino); - vol->quota_q_ino = NULL; - } - if (vol->quota_ino) { - iput(vol->quota_ino); - vol->quota_ino = NULL; - } -#endif /* NTFS_RW */ - if (vol->extend_ino) { - iput(vol->extend_ino); - vol->extend_ino = NULL; - } - if (vol->secure_ino) { - iput(vol->secure_ino); - vol->secure_ino = NULL; - } - } - iput(vol->root_ino); - vol->root_ino = NULL; - iput(vol->lcnbmp_ino); - vol->lcnbmp_ino = NULL; - iput(vol->mftbmp_ino); - vol->mftbmp_ino = NULL; -#ifdef NTFS_RW - if (vol->logfile_ino) { - iput(vol->logfile_ino); - vol->logfile_ino = NULL; - } - if (vol->mftmirr_ino) { - iput(vol->mftmirr_ino); - vol->mftmirr_ino = NULL; - } -#endif /* NTFS_RW */ - /* Throw away the table of attribute definitions. */ - vol->attrdef_size = 0; - if (vol->attrdef) { - ntfs_free(vol->attrdef); - vol->attrdef = NULL; - } - vol->upcase_len = 0; - mutex_lock(&ntfs_lock); - if (vol->upcase == default_upcase) { - ntfs_nr_upcase_users--; - vol->upcase = NULL; - } - mutex_unlock(&ntfs_lock); - if (vol->upcase) { - ntfs_free(vol->upcase); - vol->upcase = NULL; - } - if (vol->nls_map) { - unload_nls(vol->nls_map); - vol->nls_map = NULL; - } - /* Error exit code path. */ -unl_upcase_iput_tmp_ino_err_out_now: - /* - * Decrease the number of upcase users and destroy the global default - * upcase table if necessary. - */ - mutex_lock(&ntfs_lock); - if (!--ntfs_nr_upcase_users && default_upcase) { - ntfs_free(default_upcase); - default_upcase = NULL; - } - if (vol->cluster_size <= 4096 && !--ntfs_nr_compression_users) - free_compression_buffers(); - mutex_unlock(&ntfs_lock); -iput_tmp_ino_err_out_now: - iput(tmp_ino); - if (vol->mft_ino && vol->mft_ino != tmp_ino) - iput(vol->mft_ino); - vol->mft_ino = NULL; - /* Errors at this stage are irrelevant. */ -err_out_now: - sb->s_fs_info = NULL; - kfree(vol); - ntfs_debug("Failed, returning -EINVAL."); - lockdep_on(); - return -EINVAL; -} - -/* - * This is a slab cache to optimize allocations and deallocations of Unicode - * strings of the maximum length allowed by NTFS, which is NTFS_MAX_NAME_LEN - * (255) Unicode characters + a terminating NULL Unicode character. - */ -struct kmem_cache *ntfs_name_cache; - -/* Slab caches for efficient allocation/deallocation of inodes. */ -struct kmem_cache *ntfs_inode_cache; -struct kmem_cache *ntfs_big_inode_cache; - -/* Init once constructor for the inode slab cache. */ -static void ntfs_big_inode_init_once(void *foo) -{ - ntfs_inode *ni = (ntfs_inode *)foo; - - inode_init_once(VFS_I(ni)); -} - -/* - * Slab caches to optimize allocations and deallocations of attribute search - * contexts and index contexts, respectively. - */ -struct kmem_cache *ntfs_attr_ctx_cache; -struct kmem_cache *ntfs_index_ctx_cache; - -/* Driver wide mutex. */ -DEFINE_MUTEX(ntfs_lock); - -static struct dentry *ntfs_mount(struct file_system_type *fs_type, - int flags, const char *dev_name, void *data) -{ - return mount_bdev(fs_type, flags, dev_name, data, ntfs_fill_super); -} - -static struct file_system_type ntfs_fs_type = { - .owner = THIS_MODULE, - .name = "ntfs", - .mount = ntfs_mount, - .kill_sb = kill_block_super, - .fs_flags = FS_REQUIRES_DEV, -}; -MODULE_ALIAS_FS("ntfs"); - -/* Stable names for the slab caches. */ -static const char ntfs_index_ctx_cache_name[] = "ntfs_index_ctx_cache"; -static const char ntfs_attr_ctx_cache_name[] = "ntfs_attr_ctx_cache"; -static const char ntfs_name_cache_name[] = "ntfs_name_cache"; -static const char ntfs_inode_cache_name[] = "ntfs_inode_cache"; -static const char ntfs_big_inode_cache_name[] = "ntfs_big_inode_cache"; - -static int __init init_ntfs_fs(void) -{ - int err = 0; - - /* This may be ugly but it results in pretty output so who cares. (-8 */ - pr_info("driver " NTFS_VERSION " [Flags: R/" -#ifdef NTFS_RW - "W" -#else - "O" -#endif -#ifdef DEBUG - " DEBUG" -#endif -#ifdef MODULE - " MODULE" -#endif - "].\n"); - - ntfs_debug("Debug messages are enabled."); - - ntfs_index_ctx_cache = kmem_cache_create(ntfs_index_ctx_cache_name, - sizeof(ntfs_index_context), 0 /* offset */, - SLAB_HWCACHE_ALIGN, NULL /* ctor */); - if (!ntfs_index_ctx_cache) { - pr_crit("Failed to create %s!\n", ntfs_index_ctx_cache_name); - goto ictx_err_out; - } - ntfs_attr_ctx_cache = kmem_cache_create(ntfs_attr_ctx_cache_name, - sizeof(ntfs_attr_search_ctx), 0 /* offset */, - SLAB_HWCACHE_ALIGN, NULL /* ctor */); - if (!ntfs_attr_ctx_cache) { - pr_crit("NTFS: Failed to create %s!\n", - ntfs_attr_ctx_cache_name); - goto actx_err_out; - } - - ntfs_name_cache = kmem_cache_create(ntfs_name_cache_name, - (NTFS_MAX_NAME_LEN+1) * sizeof(ntfschar), 0, - SLAB_HWCACHE_ALIGN, NULL); - if (!ntfs_name_cache) { - pr_crit("Failed to create %s!\n", ntfs_name_cache_name); - goto name_err_out; - } - - ntfs_inode_cache = kmem_cache_create(ntfs_inode_cache_name, - sizeof(ntfs_inode), 0, - SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD, NULL); - if (!ntfs_inode_cache) { - pr_crit("Failed to create %s!\n", ntfs_inode_cache_name); - goto inode_err_out; - } - - ntfs_big_inode_cache = kmem_cache_create(ntfs_big_inode_cache_name, - sizeof(big_ntfs_inode), 0, - SLAB_HWCACHE_ALIGN|SLAB_RECLAIM_ACCOUNT|SLAB_MEM_SPREAD| - SLAB_ACCOUNT, ntfs_big_inode_init_once); - if (!ntfs_big_inode_cache) { - pr_crit("Failed to create %s!\n", ntfs_big_inode_cache_name); - goto big_inode_err_out; - } - - /* Register the ntfs sysctls. */ - err = ntfs_sysctl(1); - if (err) { - pr_crit("Failed to register NTFS sysctls!\n"); - goto sysctl_err_out; - } - - err = register_filesystem(&ntfs_fs_type); - if (!err) { - ntfs_debug("NTFS driver registered successfully."); - return 0; /* Success! */ - } - pr_crit("Failed to register NTFS filesystem driver!\n"); - - /* Unregister the ntfs sysctls. */ - ntfs_sysctl(0); -sysctl_err_out: - kmem_cache_destroy(ntfs_big_inode_cache); -big_inode_err_out: - kmem_cache_destroy(ntfs_inode_cache); -inode_err_out: - kmem_cache_destroy(ntfs_name_cache); -name_err_out: - kmem_cache_destroy(ntfs_attr_ctx_cache); -actx_err_out: - kmem_cache_destroy(ntfs_index_ctx_cache); -ictx_err_out: - if (!err) { - pr_crit("Aborting NTFS filesystem driver registration...\n"); - err = -ENOMEM; - } - return err; -} - -static void __exit exit_ntfs_fs(void) -{ - ntfs_debug("Unregistering NTFS driver."); - - unregister_filesystem(&ntfs_fs_type); - - /* - * Make sure all delayed rcu free inodes are flushed before we - * destroy cache. - */ - rcu_barrier(); - kmem_cache_destroy(ntfs_big_inode_cache); - kmem_cache_destroy(ntfs_inode_cache); - kmem_cache_destroy(ntfs_name_cache); - kmem_cache_destroy(ntfs_attr_ctx_cache); - kmem_cache_destroy(ntfs_index_ctx_cache); - /* Unregister the ntfs sysctls. */ - ntfs_sysctl(0); -} - -MODULE_AUTHOR("Anton Altaparmakov "); -MODULE_DESCRIPTION("NTFS 1.2/3.x driver - Copyright (c) 2001-2014 Anton Altaparmakov and Tuxera Inc."); -MODULE_VERSION(NTFS_VERSION); -MODULE_LICENSE("GPL"); -#ifdef DEBUG -module_param(debug_msgs, bint, 0); -MODULE_PARM_DESC(debug_msgs, "Enable debug messages."); -#endif - -module_init(init_ntfs_fs) -module_exit(exit_ntfs_fs) diff --git a/fs/ntfs/sysctl.c b/fs/ntfs/sysctl.c deleted file mode 100644 index 174fe536a1c0..000000000000 --- a/fs/ntfs/sysctl.c +++ /dev/null @@ -1,59 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * sysctl.c - Code for sysctl handling in NTFS Linux kernel driver. Part of - * the Linux-NTFS project. Adapted from the old NTFS driver, - * Copyright (C) 1997 Martin von Löwis, Régis Duchesne - * - * Copyright (c) 2002-2005 Anton Altaparmakov - */ - -#ifdef DEBUG - -#include - -#ifdef CONFIG_SYSCTL - -#include -#include - -#include "sysctl.h" -#include "debug.h" - -/* Definition of the ntfs sysctl. */ -static struct ctl_table ntfs_sysctls[] = { - { - .procname = "ntfs-debug", - .data = &debug_msgs, /* Data pointer and size. */ - .maxlen = sizeof(debug_msgs), - .mode = 0644, /* Mode, proc handler. */ - .proc_handler = proc_dointvec - }, - {} -}; - -/* Storage for the sysctls header. */ -static struct ctl_table_header *sysctls_root_table; - -/** - * ntfs_sysctl - add or remove the debug sysctl - * @add: add (1) or remove (0) the sysctl - * - * Add or remove the debug sysctl. Return 0 on success or -errno on error. - */ -int ntfs_sysctl(int add) -{ - if (add) { - BUG_ON(sysctls_root_table); - sysctls_root_table = register_sysctl("fs", ntfs_sysctls); - if (!sysctls_root_table) - return -ENOMEM; - } else { - BUG_ON(!sysctls_root_table); - unregister_sysctl_table(sysctls_root_table); - sysctls_root_table = NULL; - } - return 0; -} - -#endif /* CONFIG_SYSCTL */ -#endif /* DEBUG */ diff --git a/fs/ntfs/sysctl.h b/fs/ntfs/sysctl.h deleted file mode 100644 index 96bb2299d2d5..000000000000 --- a/fs/ntfs/sysctl.h +++ /dev/null @@ -1,27 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * sysctl.h - Defines for sysctl handling in NTFS Linux kernel driver. Part of - * the Linux-NTFS project. Adapted from the old NTFS driver, - * Copyright (C) 1997 Martin von Löwis, Régis Duchesne - * - * Copyright (c) 2002-2004 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_SYSCTL_H -#define _LINUX_NTFS_SYSCTL_H - - -#if defined(DEBUG) && defined(CONFIG_SYSCTL) - -extern int ntfs_sysctl(int add); - -#else - -/* Just return success. */ -static inline int ntfs_sysctl(int add) -{ - return 0; -} - -#endif /* DEBUG && CONFIG_SYSCTL */ -#endif /* _LINUX_NTFS_SYSCTL_H */ diff --git a/fs/ntfs/time.h b/fs/ntfs/time.h deleted file mode 100644 index 6b63261300cc..000000000000 --- a/fs/ntfs/time.h +++ /dev/null @@ -1,89 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * time.h - NTFS time conversion functions. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2005 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_TIME_H -#define _LINUX_NTFS_TIME_H - -#include /* For current_kernel_time(). */ -#include /* For do_div(). */ - -#include "endian.h" - -#define NTFS_TIME_OFFSET ((s64)(369 * 365 + 89) * 24 * 3600 * 10000000) - -/** - * utc2ntfs - convert Linux UTC time to NTFS time - * @ts: Linux UTC time to convert to NTFS time - * - * Convert the Linux UTC time @ts to its corresponding NTFS time and return - * that in little endian format. - * - * Linux stores time in a struct timespec64 consisting of a time64_t tv_sec - * and a long tv_nsec where tv_sec is the number of 1-second intervals since - * 1st January 1970, 00:00:00 UTC and tv_nsec is the number of 1-nano-second - * intervals since the value of tv_sec. - * - * NTFS uses Microsoft's standard time format which is stored in a s64 and is - * measured as the number of 100-nano-second intervals since 1st January 1601, - * 00:00:00 UTC. - */ -static inline sle64 utc2ntfs(const struct timespec64 ts) -{ - /* - * Convert the seconds to 100ns intervals, add the nano-seconds - * converted to 100ns intervals, and then add the NTFS time offset. - */ - return cpu_to_sle64((s64)ts.tv_sec * 10000000 + ts.tv_nsec / 100 + - NTFS_TIME_OFFSET); -} - -/** - * get_current_ntfs_time - get the current time in little endian NTFS format - * - * Get the current time from the Linux kernel, convert it to its corresponding - * NTFS time and return that in little endian format. - */ -static inline sle64 get_current_ntfs_time(void) -{ - struct timespec64 ts; - - ktime_get_coarse_real_ts64(&ts); - return utc2ntfs(ts); -} - -/** - * ntfs2utc - convert NTFS time to Linux time - * @time: NTFS time (little endian) to convert to Linux UTC - * - * Convert the little endian NTFS time @time to its corresponding Linux UTC - * time and return that in cpu format. - * - * Linux stores time in a struct timespec64 consisting of a time64_t tv_sec - * and a long tv_nsec where tv_sec is the number of 1-second intervals since - * 1st January 1970, 00:00:00 UTC and tv_nsec is the number of 1-nano-second - * intervals since the value of tv_sec. - * - * NTFS uses Microsoft's standard time format which is stored in a s64 and is - * measured as the number of 100 nano-second intervals since 1st January 1601, - * 00:00:00 UTC. - */ -static inline struct timespec64 ntfs2utc(const sle64 time) -{ - struct timespec64 ts; - - /* Subtract the NTFS time offset. */ - u64 t = (u64)(sle64_to_cpu(time) - NTFS_TIME_OFFSET); - /* - * Convert the time to 1-second intervals and the remainder to - * 1-nano-second intervals. - */ - ts.tv_nsec = do_div(t, 10000000) * 100; - ts.tv_sec = t; - return ts; -} - -#endif /* _LINUX_NTFS_TIME_H */ diff --git a/fs/ntfs/types.h b/fs/ntfs/types.h deleted file mode 100644 index 9a47859e7a06..000000000000 --- a/fs/ntfs/types.h +++ /dev/null @@ -1,55 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * types.h - Defines for NTFS Linux kernel driver specific types. - * Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2005 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_TYPES_H -#define _LINUX_NTFS_TYPES_H - -#include - -typedef __le16 le16; -typedef __le32 le32; -typedef __le64 le64; -typedef __u16 __bitwise sle16; -typedef __u32 __bitwise sle32; -typedef __u64 __bitwise sle64; - -/* 2-byte Unicode character type. */ -typedef le16 ntfschar; -#define UCHAR_T_SIZE_BITS 1 - -/* - * Clusters are signed 64-bit values on NTFS volumes. We define two types, LCN - * and VCN, to allow for type checking and better code readability. - */ -typedef s64 VCN; -typedef sle64 leVCN; -typedef s64 LCN; -typedef sle64 leLCN; - -/* - * The NTFS journal $LogFile uses log sequence numbers which are signed 64-bit - * values. We define our own type LSN, to allow for type checking and better - * code readability. - */ -typedef s64 LSN; -typedef sle64 leLSN; - -/* - * The NTFS transaction log $UsnJrnl uses usn which are signed 64-bit values. - * We define our own type USN, to allow for type checking and better code - * readability. - */ -typedef s64 USN; -typedef sle64 leUSN; - -typedef enum { - CASE_SENSITIVE = 0, - IGNORE_CASE = 1, -} IGNORE_CASE_BOOL; - -#endif /* _LINUX_NTFS_TYPES_H */ diff --git a/fs/ntfs/unistr.c b/fs/ntfs/unistr.c deleted file mode 100644 index a6b6c64f14a9..000000000000 --- a/fs/ntfs/unistr.c +++ /dev/null @@ -1,384 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * unistr.c - NTFS Unicode string handling. Part of the Linux-NTFS project. - * - * Copyright (c) 2001-2006 Anton Altaparmakov - */ - -#include - -#include "types.h" -#include "debug.h" -#include "ntfs.h" - -/* - * IMPORTANT - * ========= - * - * All these routines assume that the Unicode characters are in little endian - * encoding inside the strings!!! - */ - -/* - * This is used by the name collation functions to quickly determine what - * characters are (in)valid. - */ -static const u8 legal_ansi_char_array[0x40] = { - 0x00, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, - 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, - - 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, - 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, 0x10, - - 0x17, 0x07, 0x18, 0x17, 0x17, 0x17, 0x17, 0x17, - 0x17, 0x17, 0x18, 0x16, 0x16, 0x17, 0x07, 0x00, - - 0x17, 0x17, 0x17, 0x17, 0x17, 0x17, 0x17, 0x17, - 0x17, 0x17, 0x04, 0x16, 0x18, 0x16, 0x18, 0x18, -}; - -/** - * ntfs_are_names_equal - compare two Unicode names for equality - * @s1: name to compare to @s2 - * @s1_len: length in Unicode characters of @s1 - * @s2: name to compare to @s1 - * @s2_len: length in Unicode characters of @s2 - * @ic: ignore case bool - * @upcase: upcase table (only if @ic == IGNORE_CASE) - * @upcase_size: length in Unicode characters of @upcase (if present) - * - * Compare the names @s1 and @s2 and return 'true' (1) if the names are - * identical, or 'false' (0) if they are not identical. If @ic is IGNORE_CASE, - * the @upcase table is used to performa a case insensitive comparison. - */ -bool ntfs_are_names_equal(const ntfschar *s1, size_t s1_len, - const ntfschar *s2, size_t s2_len, const IGNORE_CASE_BOOL ic, - const ntfschar *upcase, const u32 upcase_size) -{ - if (s1_len != s2_len) - return false; - if (ic == CASE_SENSITIVE) - return !ntfs_ucsncmp(s1, s2, s1_len); - return !ntfs_ucsncasecmp(s1, s2, s1_len, upcase, upcase_size); -} - -/** - * ntfs_collate_names - collate two Unicode names - * @name1: first Unicode name to compare - * @name2: second Unicode name to compare - * @err_val: if @name1 contains an invalid character return this value - * @ic: either CASE_SENSITIVE or IGNORE_CASE - * @upcase: upcase table (ignored if @ic is CASE_SENSITIVE) - * @upcase_len: upcase table size (ignored if @ic is CASE_SENSITIVE) - * - * ntfs_collate_names collates two Unicode names and returns: - * - * -1 if the first name collates before the second one, - * 0 if the names match, - * 1 if the second name collates before the first one, or - * @err_val if an invalid character is found in @name1 during the comparison. - * - * The following characters are considered invalid: '"', '*', '<', '>' and '?'. - */ -int ntfs_collate_names(const ntfschar *name1, const u32 name1_len, - const ntfschar *name2, const u32 name2_len, - const int err_val, const IGNORE_CASE_BOOL ic, - const ntfschar *upcase, const u32 upcase_len) -{ - u32 cnt, min_len; - u16 c1, c2; - - min_len = name1_len; - if (name1_len > name2_len) - min_len = name2_len; - for (cnt = 0; cnt < min_len; ++cnt) { - c1 = le16_to_cpu(*name1++); - c2 = le16_to_cpu(*name2++); - if (ic) { - if (c1 < upcase_len) - c1 = le16_to_cpu(upcase[c1]); - if (c2 < upcase_len) - c2 = le16_to_cpu(upcase[c2]); - } - if (c1 < 64 && legal_ansi_char_array[c1] & 8) - return err_val; - if (c1 < c2) - return -1; - if (c1 > c2) - return 1; - } - if (name1_len < name2_len) - return -1; - if (name1_len == name2_len) - return 0; - /* name1_len > name2_len */ - c1 = le16_to_cpu(*name1); - if (c1 < 64 && legal_ansi_char_array[c1] & 8) - return err_val; - return 1; -} - -/** - * ntfs_ucsncmp - compare two little endian Unicode strings - * @s1: first string - * @s2: second string - * @n: maximum unicode characters to compare - * - * Compare the first @n characters of the Unicode strings @s1 and @s2, - * The strings in little endian format and appropriate le16_to_cpu() - * conversion is performed on non-little endian machines. - * - * The function returns an integer less than, equal to, or greater than zero - * if @s1 (or the first @n Unicode characters thereof) is found, respectively, - * to be less than, to match, or be greater than @s2. - */ -int ntfs_ucsncmp(const ntfschar *s1, const ntfschar *s2, size_t n) -{ - u16 c1, c2; - size_t i; - - for (i = 0; i < n; ++i) { - c1 = le16_to_cpu(s1[i]); - c2 = le16_to_cpu(s2[i]); - if (c1 < c2) - return -1; - if (c1 > c2) - return 1; - if (!c1) - break; - } - return 0; -} - -/** - * ntfs_ucsncasecmp - compare two little endian Unicode strings, ignoring case - * @s1: first string - * @s2: second string - * @n: maximum unicode characters to compare - * @upcase: upcase table - * @upcase_size: upcase table size in Unicode characters - * - * Compare the first @n characters of the Unicode strings @s1 and @s2, - * ignoring case. The strings in little endian format and appropriate - * le16_to_cpu() conversion is performed on non-little endian machines. - * - * Each character is uppercased using the @upcase table before the comparison. - * - * The function returns an integer less than, equal to, or greater than zero - * if @s1 (or the first @n Unicode characters thereof) is found, respectively, - * to be less than, to match, or be greater than @s2. - */ -int ntfs_ucsncasecmp(const ntfschar *s1, const ntfschar *s2, size_t n, - const ntfschar *upcase, const u32 upcase_size) -{ - size_t i; - u16 c1, c2; - - for (i = 0; i < n; ++i) { - if ((c1 = le16_to_cpu(s1[i])) < upcase_size) - c1 = le16_to_cpu(upcase[c1]); - if ((c2 = le16_to_cpu(s2[i])) < upcase_size) - c2 = le16_to_cpu(upcase[c2]); - if (c1 < c2) - return -1; - if (c1 > c2) - return 1; - if (!c1) - break; - } - return 0; -} - -void ntfs_upcase_name(ntfschar *name, u32 name_len, const ntfschar *upcase, - const u32 upcase_len) -{ - u32 i; - u16 u; - - for (i = 0; i < name_len; i++) - if ((u = le16_to_cpu(name[i])) < upcase_len) - name[i] = upcase[u]; -} - -void ntfs_file_upcase_value(FILE_NAME_ATTR *file_name_attr, - const ntfschar *upcase, const u32 upcase_len) -{ - ntfs_upcase_name((ntfschar*)&file_name_attr->file_name, - file_name_attr->file_name_length, upcase, upcase_len); -} - -int ntfs_file_compare_values(FILE_NAME_ATTR *file_name_attr1, - FILE_NAME_ATTR *file_name_attr2, - const int err_val, const IGNORE_CASE_BOOL ic, - const ntfschar *upcase, const u32 upcase_len) -{ - return ntfs_collate_names((ntfschar*)&file_name_attr1->file_name, - file_name_attr1->file_name_length, - (ntfschar*)&file_name_attr2->file_name, - file_name_attr2->file_name_length, - err_val, ic, upcase, upcase_len); -} - -/** - * ntfs_nlstoucs - convert NLS string to little endian Unicode string - * @vol: ntfs volume which we are working with - * @ins: input NLS string buffer - * @ins_len: length of input string in bytes - * @outs: on return contains the allocated output Unicode string buffer - * - * Convert the input string @ins, which is in whatever format the loaded NLS - * map dictates, into a little endian, 2-byte Unicode string. - * - * This function allocates the string and the caller is responsible for - * calling kmem_cache_free(ntfs_name_cache, *@outs); when finished with it. - * - * On success the function returns the number of Unicode characters written to - * the output string *@outs (>= 0), not counting the terminating Unicode NULL - * character. *@outs is set to the allocated output string buffer. - * - * On error, a negative number corresponding to the error code is returned. In - * that case the output string is not allocated. Both *@outs and *@outs_len - * are then undefined. - * - * This might look a bit odd due to fast path optimization... - */ -int ntfs_nlstoucs(const ntfs_volume *vol, const char *ins, - const int ins_len, ntfschar **outs) -{ - struct nls_table *nls = vol->nls_map; - ntfschar *ucs; - wchar_t wc; - int i, o, wc_len; - - /* We do not trust outside sources. */ - if (likely(ins)) { - ucs = kmem_cache_alloc(ntfs_name_cache, GFP_NOFS); - if (likely(ucs)) { - for (i = o = 0; i < ins_len; i += wc_len) { - wc_len = nls->char2uni(ins + i, ins_len - i, - &wc); - if (likely(wc_len >= 0 && - o < NTFS_MAX_NAME_LEN)) { - if (likely(wc)) { - ucs[o++] = cpu_to_le16(wc); - continue; - } /* else if (!wc) */ - break; - } /* else if (wc_len < 0 || - o >= NTFS_MAX_NAME_LEN) */ - goto name_err; - } - ucs[o] = 0; - *outs = ucs; - return o; - } /* else if (!ucs) */ - ntfs_error(vol->sb, "Failed to allocate buffer for converted " - "name from ntfs_name_cache."); - return -ENOMEM; - } /* else if (!ins) */ - ntfs_error(vol->sb, "Received NULL pointer."); - return -EINVAL; -name_err: - kmem_cache_free(ntfs_name_cache, ucs); - if (wc_len < 0) { - ntfs_error(vol->sb, "Name using character set %s contains " - "characters that cannot be converted to " - "Unicode.", nls->charset); - i = -EILSEQ; - } else /* if (o >= NTFS_MAX_NAME_LEN) */ { - ntfs_error(vol->sb, "Name is too long (maximum length for a " - "name on NTFS is %d Unicode characters.", - NTFS_MAX_NAME_LEN); - i = -ENAMETOOLONG; - } - return i; -} - -/** - * ntfs_ucstonls - convert little endian Unicode string to NLS string - * @vol: ntfs volume which we are working with - * @ins: input Unicode string buffer - * @ins_len: length of input string in Unicode characters - * @outs: on return contains the (allocated) output NLS string buffer - * @outs_len: length of output string buffer in bytes - * - * Convert the input little endian, 2-byte Unicode string @ins, of length - * @ins_len into the string format dictated by the loaded NLS. - * - * If *@outs is NULL, this function allocates the string and the caller is - * responsible for calling kfree(*@outs); when finished with it. In this case - * @outs_len is ignored and can be 0. - * - * On success the function returns the number of bytes written to the output - * string *@outs (>= 0), not counting the terminating NULL byte. If the output - * string buffer was allocated, *@outs is set to it. - * - * On error, a negative number corresponding to the error code is returned. In - * that case the output string is not allocated. The contents of *@outs are - * then undefined. - * - * This might look a bit odd due to fast path optimization... - */ -int ntfs_ucstonls(const ntfs_volume *vol, const ntfschar *ins, - const int ins_len, unsigned char **outs, int outs_len) -{ - struct nls_table *nls = vol->nls_map; - unsigned char *ns; - int i, o, ns_len, wc; - - /* We don't trust outside sources. */ - if (ins) { - ns = *outs; - ns_len = outs_len; - if (ns && !ns_len) { - wc = -ENAMETOOLONG; - goto conversion_err; - } - if (!ns) { - ns_len = ins_len * NLS_MAX_CHARSET_SIZE; - ns = kmalloc(ns_len + 1, GFP_NOFS); - if (!ns) - goto mem_err_out; - } - for (i = o = 0; i < ins_len; i++) { -retry: wc = nls->uni2char(le16_to_cpu(ins[i]), ns + o, - ns_len - o); - if (wc > 0) { - o += wc; - continue; - } else if (!wc) - break; - else if (wc == -ENAMETOOLONG && ns != *outs) { - unsigned char *tc; - /* Grow in multiples of 64 bytes. */ - tc = kmalloc((ns_len + 64) & - ~63, GFP_NOFS); - if (tc) { - memcpy(tc, ns, ns_len); - ns_len = ((ns_len + 64) & ~63) - 1; - kfree(ns); - ns = tc; - goto retry; - } /* No memory so goto conversion_error; */ - } /* wc < 0, real error. */ - goto conversion_err; - } - ns[o] = 0; - *outs = ns; - return o; - } /* else (!ins) */ - ntfs_error(vol->sb, "Received NULL pointer."); - return -EINVAL; -conversion_err: - ntfs_error(vol->sb, "Unicode name contains characters that cannot be " - "converted to character set %s. You might want to " - "try to use the mount option nls=utf8.", nls->charset); - if (ns != *outs) - kfree(ns); - if (wc != -ENAMETOOLONG) - wc = -EILSEQ; - return wc; -mem_err_out: - ntfs_error(vol->sb, "Failed to allocate name!"); - return -ENOMEM; -} diff --git a/fs/ntfs/upcase.c b/fs/ntfs/upcase.c deleted file mode 100644 index 4ebe84a78dea..000000000000 --- a/fs/ntfs/upcase.c +++ /dev/null @@ -1,73 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * upcase.c - Generate the full NTFS Unicode upcase table in little endian. - * Part of the Linux-NTFS project. - * - * Copyright (c) 2001 Richard Russon - * Copyright (c) 2001-2006 Anton Altaparmakov - */ - -#include "malloc.h" -#include "ntfs.h" - -ntfschar *generate_default_upcase(void) -{ - static const int uc_run_table[][3] = { /* Start, End, Add */ - {0x0061, 0x007B, -32}, {0x0451, 0x045D, -80}, {0x1F70, 0x1F72, 74}, - {0x00E0, 0x00F7, -32}, {0x045E, 0x0460, -80}, {0x1F72, 0x1F76, 86}, - {0x00F8, 0x00FF, -32}, {0x0561, 0x0587, -48}, {0x1F76, 0x1F78, 100}, - {0x0256, 0x0258, -205}, {0x1F00, 0x1F08, 8}, {0x1F78, 0x1F7A, 128}, - {0x028A, 0x028C, -217}, {0x1F10, 0x1F16, 8}, {0x1F7A, 0x1F7C, 112}, - {0x03AC, 0x03AD, -38}, {0x1F20, 0x1F28, 8}, {0x1F7C, 0x1F7E, 126}, - {0x03AD, 0x03B0, -37}, {0x1F30, 0x1F38, 8}, {0x1FB0, 0x1FB2, 8}, - {0x03B1, 0x03C2, -32}, {0x1F40, 0x1F46, 8}, {0x1FD0, 0x1FD2, 8}, - {0x03C2, 0x03C3, -31}, {0x1F51, 0x1F52, 8}, {0x1FE0, 0x1FE2, 8}, - {0x03C3, 0x03CC, -32}, {0x1F53, 0x1F54, 8}, {0x1FE5, 0x1FE6, 7}, - {0x03CC, 0x03CD, -64}, {0x1F55, 0x1F56, 8}, {0x2170, 0x2180, -16}, - {0x03CD, 0x03CF, -63}, {0x1F57, 0x1F58, 8}, {0x24D0, 0x24EA, -26}, - {0x0430, 0x0450, -32}, {0x1F60, 0x1F68, 8}, {0xFF41, 0xFF5B, -32}, - {0} - }; - - static const int uc_dup_table[][2] = { /* Start, End */ - {0x0100, 0x012F}, {0x01A0, 0x01A6}, {0x03E2, 0x03EF}, {0x04CB, 0x04CC}, - {0x0132, 0x0137}, {0x01B3, 0x01B7}, {0x0460, 0x0481}, {0x04D0, 0x04EB}, - {0x0139, 0x0149}, {0x01CD, 0x01DD}, {0x0490, 0x04BF}, {0x04EE, 0x04F5}, - {0x014A, 0x0178}, {0x01DE, 0x01EF}, {0x04BF, 0x04BF}, {0x04F8, 0x04F9}, - {0x0179, 0x017E}, {0x01F4, 0x01F5}, {0x04C1, 0x04C4}, {0x1E00, 0x1E95}, - {0x018B, 0x018B}, {0x01FA, 0x0218}, {0x04C7, 0x04C8}, {0x1EA0, 0x1EF9}, - {0} - }; - - static const int uc_word_table[][2] = { /* Offset, Value */ - {0x00FF, 0x0178}, {0x01AD, 0x01AC}, {0x01F3, 0x01F1}, {0x0269, 0x0196}, - {0x0183, 0x0182}, {0x01B0, 0x01AF}, {0x0253, 0x0181}, {0x026F, 0x019C}, - {0x0185, 0x0184}, {0x01B9, 0x01B8}, {0x0254, 0x0186}, {0x0272, 0x019D}, - {0x0188, 0x0187}, {0x01BD, 0x01BC}, {0x0259, 0x018F}, {0x0275, 0x019F}, - {0x018C, 0x018B}, {0x01C6, 0x01C4}, {0x025B, 0x0190}, {0x0283, 0x01A9}, - {0x0192, 0x0191}, {0x01C9, 0x01C7}, {0x0260, 0x0193}, {0x0288, 0x01AE}, - {0x0199, 0x0198}, {0x01CC, 0x01CA}, {0x0263, 0x0194}, {0x0292, 0x01B7}, - {0x01A8, 0x01A7}, {0x01DD, 0x018E}, {0x0268, 0x0197}, - {0} - }; - - int i, r; - ntfschar *uc; - - uc = ntfs_malloc_nofs(default_upcase_len * sizeof(ntfschar)); - if (!uc) - return uc; - memset(uc, 0, default_upcase_len * sizeof(ntfschar)); - /* Generate the little endian Unicode upcase table used by ntfs. */ - for (i = 0; i < default_upcase_len; i++) - uc[i] = cpu_to_le16(i); - for (r = 0; uc_run_table[r][0]; r++) - for (i = uc_run_table[r][0]; i < uc_run_table[r][1]; i++) - le16_add_cpu(&uc[i], uc_run_table[r][2]); - for (r = 0; uc_dup_table[r][0]; r++) - for (i = uc_dup_table[r][0]; i < uc_dup_table[r][1]; i += 2) - le16_add_cpu(&uc[i + 1], -1); - for (r = 0; uc_word_table[r][0]; r++) - uc[uc_word_table[r][0]] = cpu_to_le16(uc_word_table[r][1]); - return uc; -} diff --git a/fs/ntfs/usnjrnl.c b/fs/ntfs/usnjrnl.c deleted file mode 100644 index 9097a0b4ef25..000000000000 --- a/fs/ntfs/usnjrnl.c +++ /dev/null @@ -1,70 +0,0 @@ -// SPDX-License-Identifier: GPL-2.0-or-later -/* - * usnjrnl.h - NTFS kernel transaction log ($UsnJrnl) handling. Part of the - * Linux-NTFS project. - * - * Copyright (c) 2005 Anton Altaparmakov - */ - -#ifdef NTFS_RW - -#include -#include -#include - -#include "aops.h" -#include "debug.h" -#include "endian.h" -#include "time.h" -#include "types.h" -#include "usnjrnl.h" -#include "volume.h" - -/** - * ntfs_stamp_usnjrnl - stamp the transaction log ($UsnJrnl) on an ntfs volume - * @vol: ntfs volume on which to stamp the transaction log - * - * Stamp the transaction log ($UsnJrnl) on the ntfs volume @vol and return - * 'true' on success and 'false' on error. - * - * This function assumes that the transaction log has already been loaded and - * consistency checked by a call to fs/ntfs/super.c::load_and_init_usnjrnl(). - */ -bool ntfs_stamp_usnjrnl(ntfs_volume *vol) -{ - ntfs_debug("Entering."); - if (likely(!NVolUsnJrnlStamped(vol))) { - sle64 stamp; - struct page *page; - USN_HEADER *uh; - - page = ntfs_map_page(vol->usnjrnl_max_ino->i_mapping, 0); - if (IS_ERR(page)) { - ntfs_error(vol->sb, "Failed to read from " - "$UsnJrnl/$DATA/$Max attribute."); - return false; - } - uh = (USN_HEADER*)page_address(page); - stamp = get_current_ntfs_time(); - ntfs_debug("Stamping transaction log ($UsnJrnl): old " - "journal_id 0x%llx, old lowest_valid_usn " - "0x%llx, new journal_id 0x%llx, new " - "lowest_valid_usn 0x%llx.", - (long long)sle64_to_cpu(uh->journal_id), - (long long)sle64_to_cpu(uh->lowest_valid_usn), - (long long)sle64_to_cpu(stamp), - i_size_read(vol->usnjrnl_j_ino)); - uh->lowest_valid_usn = - cpu_to_sle64(i_size_read(vol->usnjrnl_j_ino)); - uh->journal_id = stamp; - flush_dcache_page(page); - set_page_dirty(page); - ntfs_unmap_page(page); - /* Set the flag so we do not have to do it again on remount. */ - NVolSetUsnJrnlStamped(vol); - } - ntfs_debug("Done."); - return true; -} - -#endif /* NTFS_RW */ diff --git a/fs/ntfs/usnjrnl.h b/fs/ntfs/usnjrnl.h deleted file mode 100644 index 85f531b59395..000000000000 --- a/fs/ntfs/usnjrnl.h +++ /dev/null @@ -1,191 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * usnjrnl.h - Defines for NTFS kernel transaction log ($UsnJrnl) handling. - * Part of the Linux-NTFS project. - * - * Copyright (c) 2005 Anton Altaparmakov - */ - -#ifndef _LINUX_NTFS_USNJRNL_H -#define _LINUX_NTFS_USNJRNL_H - -#ifdef NTFS_RW - -#include "types.h" -#include "endian.h" -#include "layout.h" -#include "volume.h" - -/* - * Transaction log ($UsnJrnl) organization: - * - * The transaction log records whenever a file is modified in any way. So for - * example it will record that file "blah" was written to at a particular time - * but not what was written. If will record that a file was deleted or - * created, that a file was truncated, etc. See below for all the reason - * codes used. - * - * The transaction log is in the $Extend directory which is in the root - * directory of each volume. If it is not present it means transaction - * logging is disabled. If it is present it means transaction logging is - * either enabled or in the process of being disabled in which case we can - * ignore it as it will go away as soon as Windows gets its hands on it. - * - * To determine whether the transaction logging is enabled or in the process - * of being disabled, need to check the volume flags in the - * $VOLUME_INFORMATION attribute in the $Volume system file (which is present - * in the root directory and has a fixed mft record number, see layout.h). - * If the flag VOLUME_DELETE_USN_UNDERWAY is set it means the transaction log - * is in the process of being disabled and if this flag is clear it means the - * transaction log is enabled. - * - * The transaction log consists of two parts; the $DATA/$Max attribute as well - * as the $DATA/$J attribute. $Max is a header describing the transaction - * log whilst $J is the transaction log data itself as a sequence of variable - * sized USN_RECORDs (see below for all the structures). - * - * We do not care about transaction logging at this point in time but we still - * need to let windows know that the transaction log is out of date. To do - * this we need to stamp the transaction log. This involves setting the - * lowest_valid_usn field in the $DATA/$Max attribute to the usn to be used - * for the next added USN_RECORD to the $DATA/$J attribute as well as - * generating a new journal_id in $DATA/$Max. - * - * The journal_id is as of the current version (2.0) of the transaction log - * simply the 64-bit timestamp of when the journal was either created or last - * stamped. - * - * To determine the next usn there are two ways. The first is to parse - * $DATA/$J and to find the last USN_RECORD in it and to add its record_length - * to its usn (which is the byte offset in the $DATA/$J attribute). The - * second is simply to take the data size of the attribute. Since the usns - * are simply byte offsets into $DATA/$J, this is exactly the next usn. For - * obvious reasons we use the second method as it is much simpler and faster. - * - * As an aside, note that to actually disable the transaction log, one would - * need to set the VOLUME_DELETE_USN_UNDERWAY flag (see above), then go - * through all the mft records on the volume and set the usn field in their - * $STANDARD_INFORMATION attribute to zero. Once that is done, one would need - * to delete the transaction log file, i.e. \$Extent\$UsnJrnl, and finally, - * one would need to clear the VOLUME_DELETE_USN_UNDERWAY flag. - * - * Note that if a volume is unmounted whilst the transaction log is being - * disabled, the process will continue the next time the volume is mounted. - * This is why we can safely mount read-write when we see a transaction log - * in the process of being deleted. - */ - -/* Some $UsnJrnl related constants. */ -#define UsnJrnlMajorVer 2 -#define UsnJrnlMinorVer 0 - -/* - * $DATA/$Max attribute. This is (always?) resident and has a fixed size of - * 32 bytes. It contains the header describing the transaction log. - */ -typedef struct { -/*Ofs*/ -/* 0*/sle64 maximum_size; /* The maximum on-disk size of the $DATA/$J - attribute. */ -/* 8*/sle64 allocation_delta; /* Number of bytes by which to increase the - size of the $DATA/$J attribute. */ -/*0x10*/sle64 journal_id; /* Current id of the transaction log. */ -/*0x18*/leUSN lowest_valid_usn; /* Lowest valid usn in $DATA/$J for the - current journal_id. */ -/* sizeof() = 32 (0x20) bytes */ -} __attribute__ ((__packed__)) USN_HEADER; - -/* - * Reason flags (32-bit). Cumulative flags describing the change(s) to the - * file since it was last opened. I think the names speak for themselves but - * if you disagree check out the descriptions in the Linux NTFS project NTFS - * documentation: http://www.linux-ntfs.org/ - */ -enum { - USN_REASON_DATA_OVERWRITE = cpu_to_le32(0x00000001), - USN_REASON_DATA_EXTEND = cpu_to_le32(0x00000002), - USN_REASON_DATA_TRUNCATION = cpu_to_le32(0x00000004), - USN_REASON_NAMED_DATA_OVERWRITE = cpu_to_le32(0x00000010), - USN_REASON_NAMED_DATA_EXTEND = cpu_to_le32(0x00000020), - USN_REASON_NAMED_DATA_TRUNCATION= cpu_to_le32(0x00000040), - USN_REASON_FILE_CREATE = cpu_to_le32(0x00000100), - USN_REASON_FILE_DELETE = cpu_to_le32(0x00000200), - USN_REASON_EA_CHANGE = cpu_to_le32(0x00000400), - USN_REASON_SECURITY_CHANGE = cpu_to_le32(0x00000800), - USN_REASON_RENAME_OLD_NAME = cpu_to_le32(0x00001000), - USN_REASON_RENAME_NEW_NAME = cpu_to_le32(0x00002000), - USN_REASON_INDEXABLE_CHANGE = cpu_to_le32(0x00004000), - USN_REASON_BASIC_INFO_CHANGE = cpu_to_le32(0x00008000), - USN_REASON_HARD_LINK_CHANGE = cpu_to_le32(0x00010000), - USN_REASON_COMPRESSION_CHANGE = cpu_to_le32(0x00020000), - USN_REASON_ENCRYPTION_CHANGE = cpu_to_le32(0x00040000), - USN_REASON_OBJECT_ID_CHANGE = cpu_to_le32(0x00080000), - USN_REASON_REPARSE_POINT_CHANGE = cpu_to_le32(0x00100000), - USN_REASON_STREAM_CHANGE = cpu_to_le32(0x00200000), - USN_REASON_CLOSE = cpu_to_le32(0x80000000), -}; - -typedef le32 USN_REASON_FLAGS; - -/* - * Source info flags (32-bit). Information about the source of the change(s) - * to the file. For detailed descriptions of what these mean, see the Linux - * NTFS project NTFS documentation: - * http://www.linux-ntfs.org/ - */ -enum { - USN_SOURCE_DATA_MANAGEMENT = cpu_to_le32(0x00000001), - USN_SOURCE_AUXILIARY_DATA = cpu_to_le32(0x00000002), - USN_SOURCE_REPLICATION_MANAGEMENT = cpu_to_le32(0x00000004), -}; - -typedef le32 USN_SOURCE_INFO_FLAGS; - -/* - * $DATA/$J attribute. This is always non-resident, is marked as sparse, and - * is of variabled size. It consists of a sequence of variable size - * USN_RECORDS. The minimum allocated_size is allocation_delta as - * specified in $DATA/$Max. When the maximum_size specified in $DATA/$Max is - * exceeded by more than allocation_delta bytes, allocation_delta bytes are - * allocated and appended to the $DATA/$J attribute and an equal number of - * bytes at the beginning of the attribute are freed and made sparse. Note the - * making sparse only happens at volume checkpoints and hence the actual - * $DATA/$J size can exceed maximum_size + allocation_delta temporarily. - */ -typedef struct { -/*Ofs*/ -/* 0*/le32 length; /* Byte size of this record (8-byte - aligned). */ -/* 4*/le16 major_ver; /* Major version of the transaction log used - for this record. */ -/* 6*/le16 minor_ver; /* Minor version of the transaction log used - for this record. */ -/* 8*/leMFT_REF mft_reference;/* The mft reference of the file (or - directory) described by this record. */ -/*0x10*/leMFT_REF parent_directory;/* The mft reference of the parent - directory of the file described by this - record. */ -/*0x18*/leUSN usn; /* The usn of this record. Equals the offset - within the $DATA/$J attribute. */ -/*0x20*/sle64 time; /* Time when this record was created. */ -/*0x28*/USN_REASON_FLAGS reason;/* Reason flags (see above). */ -/*0x2c*/USN_SOURCE_INFO_FLAGS source_info;/* Source info flags (see above). */ -/*0x30*/le32 security_id; /* File security_id copied from - $STANDARD_INFORMATION. */ -/*0x34*/FILE_ATTR_FLAGS file_attributes; /* File attributes copied from - $STANDARD_INFORMATION or $FILE_NAME (not - sure which). */ -/*0x38*/le16 file_name_size; /* Size of the file name in bytes. */ -/*0x3a*/le16 file_name_offset; /* Offset to the file name in bytes from the - start of this record. */ -/*0x3c*/ntfschar file_name[0]; /* Use when creating only. When reading use - file_name_offset to determine the location - of the name. */ -/* sizeof() = 60 (0x3c) bytes */ -} __attribute__ ((__packed__)) USN_RECORD; - -extern bool ntfs_stamp_usnjrnl(ntfs_volume *vol); - -#endif /* NTFS_RW */ - -#endif /* _LINUX_NTFS_USNJRNL_H */ diff --git a/fs/ntfs/volume.h b/fs/ntfs/volume.h deleted file mode 100644 index 930a9ae8a053..000000000000 --- a/fs/ntfs/volume.h +++ /dev/null @@ -1,164 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-or-later */ -/* - * volume.h - Defines for volume structures in NTFS Linux kernel driver. Part - * of the Linux-NTFS project. - * - * Copyright (c) 2001-2006 Anton Altaparmakov - * Copyright (c) 2002 Richard Russon - */ - -#ifndef _LINUX_NTFS_VOLUME_H -#define _LINUX_NTFS_VOLUME_H - -#include -#include - -#include "types.h" -#include "layout.h" - -/* - * The NTFS in memory super block structure. - */ -typedef struct { - /* - * FIXME: Reorder to have commonly used together element within the - * same cache line, aiming at a cache line size of 32 bytes. Aim for - * 64 bytes for less commonly used together elements. Put most commonly - * used elements to front of structure. Obviously do this only when the - * structure has stabilized... (AIA) - */ - /* Device specifics. */ - struct super_block *sb; /* Pointer back to the super_block. */ - LCN nr_blocks; /* Number of sb->s_blocksize bytes - sized blocks on the device. */ - /* Configuration provided by user at mount time. */ - unsigned long flags; /* Miscellaneous flags, see below. */ - kuid_t uid; /* uid that files will be mounted as. */ - kgid_t gid; /* gid that files will be mounted as. */ - umode_t fmask; /* The mask for file permissions. */ - umode_t dmask; /* The mask for directory - permissions. */ - u8 mft_zone_multiplier; /* Initial mft zone multiplier. */ - u8 on_errors; /* What to do on filesystem errors. */ - /* NTFS bootsector provided information. */ - u16 sector_size; /* in bytes */ - u8 sector_size_bits; /* log2(sector_size) */ - u32 cluster_size; /* in bytes */ - u32 cluster_size_mask; /* cluster_size - 1 */ - u8 cluster_size_bits; /* log2(cluster_size) */ - u32 mft_record_size; /* in bytes */ - u32 mft_record_size_mask; /* mft_record_size - 1 */ - u8 mft_record_size_bits; /* log2(mft_record_size) */ - u32 index_record_size; /* in bytes */ - u32 index_record_size_mask; /* index_record_size - 1 */ - u8 index_record_size_bits; /* log2(index_record_size) */ - LCN nr_clusters; /* Volume size in clusters == number of - bits in lcn bitmap. */ - LCN mft_lcn; /* Cluster location of mft data. */ - LCN mftmirr_lcn; /* Cluster location of copy of mft. */ - u64 serial_no; /* The volume serial number. */ - /* Mount specific NTFS information. */ - u32 upcase_len; /* Number of entries in upcase[]. */ - ntfschar *upcase; /* The upcase table. */ - - s32 attrdef_size; /* Size of the attribute definition - table in bytes. */ - ATTR_DEF *attrdef; /* Table of attribute definitions. - Obtained from FILE_AttrDef. */ - -#ifdef NTFS_RW - /* Variables used by the cluster and mft allocators. */ - s64 mft_data_pos; /* Mft record number at which to - allocate the next mft record. */ - LCN mft_zone_start; /* First cluster of the mft zone. */ - LCN mft_zone_end; /* First cluster beyond the mft zone. */ - LCN mft_zone_pos; /* Current position in the mft zone. */ - LCN data1_zone_pos; /* Current position in the first data - zone. */ - LCN data2_zone_pos; /* Current position in the second data - zone. */ -#endif /* NTFS_RW */ - - struct inode *mft_ino; /* The VFS inode of $MFT. */ - - struct inode *mftbmp_ino; /* Attribute inode for $MFT/$BITMAP. */ - struct rw_semaphore mftbmp_lock; /* Lock for serializing accesses to the - mft record bitmap ($MFT/$BITMAP). */ -#ifdef NTFS_RW - struct inode *mftmirr_ino; /* The VFS inode of $MFTMirr. */ - int mftmirr_size; /* Size of mft mirror in mft records. */ - - struct inode *logfile_ino; /* The VFS inode of $LogFile. */ -#endif /* NTFS_RW */ - - struct inode *lcnbmp_ino; /* The VFS inode of $Bitmap. */ - struct rw_semaphore lcnbmp_lock; /* Lock for serializing accesses to the - cluster bitmap ($Bitmap/$DATA). */ - - struct inode *vol_ino; /* The VFS inode of $Volume. */ - VOLUME_FLAGS vol_flags; /* Volume flags. */ - u8 major_ver; /* Ntfs major version of volume. */ - u8 minor_ver; /* Ntfs minor version of volume. */ - - struct inode *root_ino; /* The VFS inode of the root - directory. */ - struct inode *secure_ino; /* The VFS inode of $Secure (NTFS3.0+ - only, otherwise NULL). */ - struct inode *extend_ino; /* The VFS inode of $Extend (NTFS3.0+ - only, otherwise NULL). */ -#ifdef NTFS_RW - /* $Quota stuff is NTFS3.0+ specific. Unused/NULL otherwise. */ - struct inode *quota_ino; /* The VFS inode of $Quota. */ - struct inode *quota_q_ino; /* Attribute inode for $Quota/$Q. */ - /* $UsnJrnl stuff is NTFS3.0+ specific. Unused/NULL otherwise. */ - struct inode *usnjrnl_ino; /* The VFS inode of $UsnJrnl. */ - struct inode *usnjrnl_max_ino; /* Attribute inode for $UsnJrnl/$Max. */ - struct inode *usnjrnl_j_ino; /* Attribute inode for $UsnJrnl/$J. */ -#endif /* NTFS_RW */ - struct nls_table *nls_map; -} ntfs_volume; - -/* - * Defined bits for the flags field in the ntfs_volume structure. - */ -typedef enum { - NV_Errors, /* 1: Volume has errors, prevent remount rw. */ - NV_ShowSystemFiles, /* 1: Return system files in ntfs_readdir(). */ - NV_CaseSensitive, /* 1: Treat file names as case sensitive and - create filenames in the POSIX namespace. - Otherwise be case insensitive but still - create file names in POSIX namespace. */ - NV_LogFileEmpty, /* 1: $LogFile journal is empty. */ - NV_QuotaOutOfDate, /* 1: $Quota is out of date. */ - NV_UsnJrnlStamped, /* 1: $UsnJrnl has been stamped. */ - NV_SparseEnabled, /* 1: May create sparse files. */ -} ntfs_volume_flags; - -/* - * Macro tricks to expand the NVolFoo(), NVolSetFoo(), and NVolClearFoo() - * functions. - */ -#define DEFINE_NVOL_BIT_OPS(flag) \ -static inline int NVol##flag(ntfs_volume *vol) \ -{ \ - return test_bit(NV_##flag, &(vol)->flags); \ -} \ -static inline void NVolSet##flag(ntfs_volume *vol) \ -{ \ - set_bit(NV_##flag, &(vol)->flags); \ -} \ -static inline void NVolClear##flag(ntfs_volume *vol) \ -{ \ - clear_bit(NV_##flag, &(vol)->flags); \ -} - -/* Emit the ntfs volume bitops functions. */ -DEFINE_NVOL_BIT_OPS(Errors) -DEFINE_NVOL_BIT_OPS(ShowSystemFiles) -DEFINE_NVOL_BIT_OPS(CaseSensitive) -DEFINE_NVOL_BIT_OPS(LogFileEmpty) -DEFINE_NVOL_BIT_OPS(QuotaOutOfDate) -DEFINE_NVOL_BIT_OPS(UsnJrnlStamped) -DEFINE_NVOL_BIT_OPS(SparseEnabled) - -#endif /* _LINUX_NTFS_VOLUME_H */ diff --git a/fs/ntfs3/bitmap.c b/fs/ntfs3/bitmap.c index 29b585f443f3..65d05e6a0566 100644 --- a/fs/ntfs3/bitmap.c +++ b/fs/ntfs3/bitmap.c @@ -710,20 +710,17 @@ int wnd_set_free(struct wnd_bitmap *wnd, size_t bit, size_t bits) { int err = 0; struct super_block *sb = wnd->sb; - size_t bits0 = bits; u32 wbits = 8 * sb->s_blocksize; size_t iw = bit >> (sb->s_blocksize_bits + 3); u32 wbit = bit & (wbits - 1); struct buffer_head *bh; + u32 op; - while (iw < wnd->nwnd && bits) { - u32 tail, op; - + for (; iw < wnd->nwnd && bits; iw++, bit += op, bits -= op, wbit = 0) { if (iw + 1 == wnd->nwnd) wbits = wnd->bits_last; - tail = wbits - wbit; - op = min_t(u32, tail, bits); + op = min_t(u32, wbits - wbit, bits); bh = wnd_map(wnd, iw); if (IS_ERR(bh)) { @@ -736,20 +733,15 @@ int wnd_set_free(struct wnd_bitmap *wnd, size_t bit, size_t bits) ntfs_bitmap_clear_le(bh->b_data, wbit, op); wnd->free_bits[iw] += op; + wnd->total_zeroes += op; set_buffer_uptodate(bh); mark_buffer_dirty(bh); unlock_buffer(bh); put_bh(bh); - wnd->total_zeroes += op; - bits -= op; - wbit = 0; - iw += 1; + wnd_add_free_ext(wnd, bit, op, false); } - - wnd_add_free_ext(wnd, bit, bits0, false); - return err; } @@ -760,20 +752,17 @@ int wnd_set_used(struct wnd_bitmap *wnd, size_t bit, size_t bits) { int err = 0; struct super_block *sb = wnd->sb; - size_t bits0 = bits; size_t iw = bit >> (sb->s_blocksize_bits + 3); u32 wbits = 8 * sb->s_blocksize; u32 wbit = bit & (wbits - 1); struct buffer_head *bh; + u32 op; - while (iw < wnd->nwnd && bits) { - u32 tail, op; - + for (; iw < wnd->nwnd && bits; iw++, bit += op, bits -= op, wbit = 0) { if (unlikely(iw + 1 == wnd->nwnd)) wbits = wnd->bits_last; - tail = wbits - wbit; - op = min_t(u32, tail, bits); + op = min_t(u32, wbits - wbit, bits); bh = wnd_map(wnd, iw); if (IS_ERR(bh)) { @@ -785,21 +774,16 @@ int wnd_set_used(struct wnd_bitmap *wnd, size_t bit, size_t bits) ntfs_bitmap_set_le(bh->b_data, wbit, op); wnd->free_bits[iw] -= op; + wnd->total_zeroes -= op; set_buffer_uptodate(bh); mark_buffer_dirty(bh); unlock_buffer(bh); put_bh(bh); - wnd->total_zeroes -= op; - bits -= op; - wbit = 0; - iw += 1; + if (!RB_EMPTY_ROOT(&wnd->start_tree)) + wnd_remove_free_ext(wnd, bit, op); } - - if (!RB_EMPTY_ROOT(&wnd->start_tree)) - wnd_remove_free_ext(wnd, bit, bits0); - return err; } @@ -852,15 +836,13 @@ static bool wnd_is_free_hlp(struct wnd_bitmap *wnd, size_t bit, size_t bits) size_t iw = bit >> (sb->s_blocksize_bits + 3); u32 wbits = 8 * sb->s_blocksize; u32 wbit = bit & (wbits - 1); + u32 op; - while (iw < wnd->nwnd && bits) { - u32 tail, op; - + for (; iw < wnd->nwnd && bits; iw++, bits -= op, wbit = 0) { if (unlikely(iw + 1 == wnd->nwnd)) wbits = wnd->bits_last; - tail = wbits - wbit; - op = min_t(u32, tail, bits); + op = min_t(u32, wbits - wbit, bits); if (wbits != wnd->free_bits[iw]) { bool ret; @@ -875,10 +857,6 @@ static bool wnd_is_free_hlp(struct wnd_bitmap *wnd, size_t bit, size_t bits) if (!ret) return false; } - - bits -= op; - wbit = 0; - iw += 1; } return true; @@ -928,6 +906,7 @@ bool wnd_is_used(struct wnd_bitmap *wnd, size_t bit, size_t bits) size_t iw = bit >> (sb->s_blocksize_bits + 3); u32 wbits = 8 * sb->s_blocksize; u32 wbit = bit & (wbits - 1); + u32 op; size_t end; struct rb_node *n; struct e_node *e; @@ -945,14 +924,11 @@ bool wnd_is_used(struct wnd_bitmap *wnd, size_t bit, size_t bits) return false; use_wnd: - while (iw < wnd->nwnd && bits) { - u32 tail, op; - + for (; iw < wnd->nwnd && bits; iw++, bits -= op, wbit = 0) { if (unlikely(iw + 1 == wnd->nwnd)) wbits = wnd->bits_last; - tail = wbits - wbit; - op = min_t(u32, tail, bits); + op = min_t(u32, wbits - wbit, bits); if (wnd->free_bits[iw]) { bool ret; @@ -966,10 +942,6 @@ bool wnd_is_used(struct wnd_bitmap *wnd, size_t bit, size_t bits) if (!ret) goto out; } - - bits -= op; - wbit = 0; - iw += 1; } ret = true; diff --git a/fs/ntfs3/file.c b/fs/ntfs3/file.c index 3eee810c69e2..7ef52eec6f8a 100644 --- a/fs/ntfs3/file.c +++ b/fs/ntfs3/file.c @@ -253,8 +253,7 @@ static int ntfs_zero_range(struct inode *inode, u64 vbo, u64 vbo_to) */ static int ntfs_file_mmap(struct file *file, struct vm_area_struct *vma) { - struct address_space *mapping = file->f_mapping; - struct inode *inode = mapping->host; + struct inode *inode = file_inode(file); struct ntfs_inode *ni = ntfs_i(inode); u64 from = ((u64)vma->vm_pgoff << PAGE_SHIFT); bool rw = vma->vm_flags & VM_WRITE; @@ -430,7 +429,7 @@ static int ntfs_truncate(struct inode *inode, loff_t new_size) */ static long ntfs_fallocate(struct file *file, int mode, loff_t vbo, loff_t len) { - struct inode *inode = file->f_mapping->host; + struct inode *inode = file_inode(file); struct address_space *mapping = inode->i_mapping; struct super_block *sb = inode->i_sb; struct ntfs_sb_info *sbi = sb->s_fs_info; @@ -733,10 +732,12 @@ int ntfs3_setattr(struct mnt_idmap *idmap, struct dentry *dentry, return err; } -static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) +/* + * check_read_restriction: + * common code for ntfs_file_read_iter and ntfs_file_splice_read + */ +static int check_read_restriction(struct inode *inode) { - struct file *file = iocb->ki_filp; - struct inode *inode = file->f_mapping->host; struct ntfs_inode *ni = ntfs_i(inode); if (unlikely(ntfs3_forced_shutdown(inode->i_sb))) @@ -747,11 +748,6 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) return -EOPNOTSUPP; } - if (is_compressed(ni) && (iocb->ki_flags & IOCB_DIRECT)) { - ntfs_inode_warn(inode, "direct i/o + compressed not supported"); - return -EOPNOTSUPP; - } - #ifndef CONFIG_NTFS3_LZX_XPRESS if (ni->ni_flags & NI_FLAG_COMPRESSED_MASK) { ntfs_inode_warn( @@ -766,37 +762,44 @@ static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) return -EOPNOTSUPP; } - return generic_file_read_iter(iocb, iter); + return 0; } -static ssize_t ntfs_file_splice_read(struct file *in, loff_t *ppos, - struct pipe_inode_info *pipe, size_t len, - unsigned int flags) +/* + * ntfs_file_read_iter - file_operations::read_iter + */ +static ssize_t ntfs_file_read_iter(struct kiocb *iocb, struct iov_iter *iter) { - struct inode *inode = in->f_mapping->host; + struct file *file = iocb->ki_filp; + struct inode *inode = file_inode(file); struct ntfs_inode *ni = ntfs_i(inode); + ssize_t err; - if (unlikely(ntfs3_forced_shutdown(inode->i_sb))) - return -EIO; + err = check_read_restriction(inode); + if (err) + return err; - if (is_encrypted(ni)) { - ntfs_inode_warn(inode, "encrypted i/o not supported"); + if (is_compressed(ni) && (iocb->ki_flags & IOCB_DIRECT)) { + ntfs_inode_warn(inode, "direct i/o + compressed not supported"); return -EOPNOTSUPP; } -#ifndef CONFIG_NTFS3_LZX_XPRESS - if (ni->ni_flags & NI_FLAG_COMPRESSED_MASK) { - ntfs_inode_warn( - inode, - "activate CONFIG_NTFS3_LZX_XPRESS to read external compressed files"); - return -EOPNOTSUPP; - } -#endif + return generic_file_read_iter(iocb, iter); +} - if (is_dedup(ni)) { - ntfs_inode_warn(inode, "read deduplicated not supported"); - return -EOPNOTSUPP; - } +/* + * ntfs_file_splice_read - file_operations::splice_read + */ +static ssize_t ntfs_file_splice_read(struct file *in, loff_t *ppos, + struct pipe_inode_info *pipe, size_t len, + unsigned int flags) +{ + struct inode *inode = file_inode(in); + ssize_t err; + + err = check_read_restriction(inode); + if (err) + return err; return filemap_splice_read(in, ppos, pipe, len, flags); } @@ -1066,15 +1069,11 @@ static ssize_t ntfs_compress_write(struct kiocb *iocb, struct iov_iter *from) } /* - * ntfs_file_write_iter - file_operations::write_iter + * check_write_restriction: + * common code for ntfs_file_write_iter and ntfs_file_splice_write */ -static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from) +static int check_write_restriction(struct inode *inode) { - struct file *file = iocb->ki_filp; - struct address_space *mapping = file->f_mapping; - struct inode *inode = mapping->host; - ssize_t ret; - int err; struct ntfs_inode *ni = ntfs_i(inode); if (unlikely(ntfs3_forced_shutdown(inode->i_sb))) @@ -1085,22 +1084,41 @@ static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from) return -EOPNOTSUPP; } - if (is_compressed(ni) && (iocb->ki_flags & IOCB_DIRECT)) { - ntfs_inode_warn(inode, "direct i/o + compressed not supported"); - return -EOPNOTSUPP; - } - if (is_dedup(ni)) { ntfs_inode_warn(inode, "write into deduplicated not supported"); return -EOPNOTSUPP; } + return 0; +} + +/* + * ntfs_file_write_iter - file_operations::write_iter + */ +static ssize_t ntfs_file_write_iter(struct kiocb *iocb, struct iov_iter *from) +{ + struct file *file = iocb->ki_filp; + struct inode *inode = file_inode(file); + struct ntfs_inode *ni = ntfs_i(inode); + ssize_t ret; + int err; + if (!inode_trylock(inode)) { if (iocb->ki_flags & IOCB_NOWAIT) return -EAGAIN; inode_lock(inode); } + ret = check_write_restriction(inode); + if (ret) + goto out; + + if (is_compressed(ni) && (iocb->ki_flags & IOCB_DIRECT)) { + ntfs_inode_warn(inode, "direct i/o + compressed not supported"); + ret = -EOPNOTSUPP; + goto out; + } + ret = generic_write_checks(iocb, from); if (ret <= 0) goto out; @@ -1258,6 +1276,23 @@ int ntfs_file_fsync(struct file *file, loff_t start, loff_t end, int datasync) return ret; } +/* + * ntfs_file_splice_write - file_operations::splice_write + */ +static ssize_t ntfs_file_splice_write(struct pipe_inode_info *pipe, + struct file *file, loff_t *ppos, + size_t len, unsigned int flags) +{ + ssize_t err; + struct inode *inode = file_inode(file); + + err = check_write_restriction(inode); + if (err) + return err; + + return iter_file_splice_write(pipe, file, ppos, len, flags); +} + // clang-format off const struct inode_operations ntfs_file_inode_operations = { .getattr = ntfs_getattr, @@ -1277,10 +1312,10 @@ const struct file_operations ntfs_file_operations = { .compat_ioctl = ntfs_compat_ioctl, #endif .splice_read = ntfs_file_splice_read, + .splice_write = ntfs_file_splice_write, .mmap = ntfs_file_mmap, .open = ntfs_file_open, .fsync = ntfs_file_fsync, - .splice_write = iter_file_splice_write, .fallocate = ntfs_fallocate, .release = ntfs_file_release, }; diff --git a/fs/ntfs3/frecord.c b/fs/ntfs3/frecord.c index 7952ef0c585b..00faacc33b08 100644 --- a/fs/ntfs3/frecord.c +++ b/fs/ntfs3/frecord.c @@ -1975,7 +1975,6 @@ int ni_fiemap(struct ntfs_inode *ni, struct fiemap_extent_info *fieinfo, if (end > alloc_size) end = alloc_size; - while (vbo < end) { if (idx == -1) { ok = run_lookup_entry(&run, vcn, &lcn, &clen, &idx); diff --git a/fs/ntfs3/fsntfs.c b/fs/ntfs3/fsntfs.c index 3909c95d9bed..513c04be1caf 100644 --- a/fs/ntfs3/fsntfs.c +++ b/fs/ntfs3/fsntfs.c @@ -2711,4 +2711,4 @@ int ntfs_set_label(struct ntfs_sb_info *sbi, u8 *label, int len) out: kfree(uni); return err; -} \ No newline at end of file +} diff --git a/fs/ntfs3/run.c b/fs/ntfs3/run.c index 535a094e822e..6c7f311f24c6 100644 --- a/fs/ntfs3/run.c +++ b/fs/ntfs3/run.c @@ -1131,14 +1131,17 @@ int run_unpack_ex(struct runs_tree *run, struct ntfs_sb_info *sbi, CLST ino, err = wnd_set_used_safe(wnd, lcn, len, &done); if (zone) { /* Restore zone. Lock mft run. */ - struct rw_semaphore *lock; - lock = is_mounted(sbi) ? &sbi->mft.ni->file.run_lock : - NULL; - if (lock) - down_read(lock); - ntfs_refresh_zone(sbi); - if (lock) - up_read(lock); + struct rw_semaphore *lock = + is_mounted(sbi) ? &sbi->mft.ni->file.run_lock : + NULL; + if (lock) { + if (down_read_trylock(lock)) { + ntfs_refresh_zone(sbi); + up_read(lock); + } + } else { + ntfs_refresh_zone(sbi); + } } up_write(&wnd->rw_lock); if (err) diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c index 37c5d462f599..48acd923e90f 100644 --- a/fs/ocfs2/cluster/heartbeat.c +++ b/fs/ocfs2/cluster/heartbeat.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include #include @@ -210,6 +211,7 @@ struct o2hb_region { /* protected by the hr_callback_sem */ struct task_struct *hr_task; + u8 hr_node_num; unsigned int hr_blocks; unsigned long long hr_start_block; @@ -264,6 +266,9 @@ struct o2hb_region { /* Message key for negotiate timeout message. */ unsigned int hr_key; struct list_head hr_handler_list; + /* Serializes timeout arming against failed-start and teardown. */ + struct mutex hr_arming_mutex; + bool hr_stopping; /* last hb status, 0 for success, other value for error. */ int hr_last_hb_status; @@ -323,9 +328,14 @@ static void o2hb_write_timeout(struct work_struct *work) static void o2hb_arm_timeout(struct o2hb_region *reg) { + mutex_lock(®->hr_arming_mutex); + + if (reg->hr_stopping) + goto out_unlock; + /* Arm writeout only after thread reaches steady state */ if (atomic_read(®->hr_steady_iterations) != 0) - return; + goto out_unlock; mlog(ML_HEARTBEAT, "Queue write timeout for %u ms\n", O2HB_MAX_WRITE_TIMEOUT_MS); @@ -344,6 +354,18 @@ static void o2hb_arm_timeout(struct o2hb_region *reg) schedule_delayed_work(®->hr_nego_timeout_work, msecs_to_jiffies(O2HB_NEGO_TIMEOUT_MS)); bitmap_zero(reg->hr_nego_node_bitmap, O2NM_MAX_NODES); + +out_unlock: + mutex_unlock(®->hr_arming_mutex); +} + +static void o2hb_queue_nego_timeout(struct o2hb_region *reg, + unsigned long delay) +{ + mutex_lock(®->hr_arming_mutex); + if (!reg->hr_stopping) + schedule_delayed_work(®->hr_nego_timeout_work, delay); + mutex_unlock(®->hr_arming_mutex); } static void o2hb_disarm_timeout(struct o2hb_region *reg) @@ -352,12 +374,25 @@ static void o2hb_disarm_timeout(struct o2hb_region *reg) cancel_delayed_work_sync(®->hr_nego_timeout_work); } -static int o2hb_send_nego_msg(int key, int type, u8 target) +static void o2hb_set_region_stopping(struct o2hb_region *reg, bool stopping) +{ + mutex_lock(®->hr_arming_mutex); + reg->hr_stopping = stopping; + mutex_unlock(®->hr_arming_mutex); +} + +static void o2hb_quiesce_timeout(struct o2hb_region *reg) +{ + o2hb_set_region_stopping(reg, true); + o2hb_disarm_timeout(reg); +} + +static int o2hb_send_nego_msg(int key, int type, u8 target, u8 node_num) { struct o2hb_nego_msg msg; int status, ret; - msg.node_num = o2nm_this_node(); + msg.node_num = node_num; again: ret = o2net_send_message(type, key, &msg, sizeof(msg), target, &status); @@ -375,8 +410,10 @@ static void o2hb_nego_timeout(struct work_struct *work) unsigned long live_node_bitmap[BITS_TO_LONGS(O2NM_MAX_NODES)]; int master_node, i, ret; struct o2hb_region *reg; + u8 node_num; reg = container_of(work, struct o2hb_region, hr_nego_timeout_work.work); + node_num = reg->hr_node_num; /* don't negotiate timeout if last hb failed since it is very * possible io failed. Should let write timeout fence self. */ @@ -387,10 +424,10 @@ static void o2hb_nego_timeout(struct work_struct *work) /* lowest node as master node to make negotiate decision. */ master_node = find_first_bit(live_node_bitmap, O2NM_MAX_NODES); - if (master_node == o2nm_this_node()) { + if (master_node == node_num) { if (!test_bit(master_node, reg->hr_nego_node_bitmap)) { printk(KERN_NOTICE "o2hb: node %d hb write hung for %ds on region %s (%pg).\n", - o2nm_this_node(), O2HB_NEGO_TIMEOUT_MS/1000, + node_num, O2HB_NEGO_TIMEOUT_MS / 1000, config_item_name(®->hr_item), reg->hr_bdev); set_bit(master_node, reg->hr_nego_node_bitmap); } @@ -399,8 +436,7 @@ static void o2hb_nego_timeout(struct work_struct *work) /* check negotiate bitmap every second to do timeout * approve decision. */ - schedule_delayed_work(®->hr_nego_timeout_work, - msecs_to_jiffies(1000)); + o2hb_queue_nego_timeout(reg, msecs_to_jiffies(1000)); return; } @@ -418,7 +454,7 @@ static void o2hb_nego_timeout(struct work_struct *work) mlog(ML_HEARTBEAT, "send NEGO_APPROVE msg to node %d\n", i); ret = o2hb_send_nego_msg(reg->hr_key, - O2HB_NEGO_APPROVE_MSG, i); + O2HB_NEGO_APPROVE_MSG, i, node_num); if (ret) mlog(ML_ERROR, "send NEGO_APPROVE msg to node %d fail %d\n", i, ret); @@ -426,10 +462,10 @@ static void o2hb_nego_timeout(struct work_struct *work) } else { /* negotiate timeout with master node. */ printk(KERN_NOTICE "o2hb: node %d hb write hung for %ds on region %s (%pg), negotiate timeout with node %d.\n", - o2nm_this_node(), O2HB_NEGO_TIMEOUT_MS/1000, config_item_name(®->hr_item), + node_num, O2HB_NEGO_TIMEOUT_MS / 1000, config_item_name(®->hr_item), reg->hr_bdev, master_node); ret = o2hb_send_nego_msg(reg->hr_key, O2HB_NEGO_TIMEOUT_MSG, - master_node); + master_node, node_num); if (ret) mlog(ML_ERROR, "send NEGO_TIMEOUT msg to node %d fail %d\n", master_node, ret); @@ -601,7 +637,9 @@ static int o2hb_issue_node_write(struct o2hb_region *reg, o2hb_bio_wait_init(write_wc); - slot = o2nm_this_node(); + slot = reg->hr_node_num; + if (slot >= O2NM_MAX_NODES) + return -EINVAL; bio = o2hb_setup_one_bio(reg, write_wc, &slot, slot+1, REQ_OP_WRITE | REQ_SYNC); @@ -670,8 +708,12 @@ static int o2hb_check_own_slot(struct o2hb_region *reg) struct o2hb_disk_slot *slot; struct o2hb_disk_heartbeat_block *hb_block; char *errstr; + u8 node_num = reg->hr_node_num; + + if (node_num >= O2NM_MAX_NODES) + return 0; - slot = ®->hr_slots[o2nm_this_node()]; + slot = ®->hr_slots[node_num]; /* Don't check on our 1st timestamp */ if (!slot->ds_last_time) return 0; @@ -712,7 +754,10 @@ static inline void o2hb_prepare_block(struct o2hb_region *reg, struct o2hb_disk_slot *slot; struct o2hb_disk_heartbeat_block *hb_block; - node_num = o2nm_this_node(); + node_num = reg->hr_node_num; + if (node_num >= O2NM_MAX_NODES) + return; + slot = ®->hr_slots[node_num]; hb_block = (struct o2hb_disk_heartbeat_block *)slot->ds_raw_block; @@ -1206,7 +1251,7 @@ static int o2hb_thread(void *data) set_user_nice(current, MIN_NICE); /* Pin node */ - ret = o2nm_depend_this_node(); + ret = o2nm_depend_node(reg->hr_node_num); if (ret) { mlog(ML_ERROR, "Node has been deleted, ret = %d\n", ret); reg->hr_node_deleted = 1; @@ -1215,7 +1260,8 @@ static int o2hb_thread(void *data) } while (!kthread_should_stop() && - !reg->hr_unclean_stop && !reg->hr_aborted_start) { + !reg->hr_unclean_stop && !reg->hr_aborted_start && + o2nm_this_node() == reg->hr_node_num) { /* We track the time spent inside * o2hb_do_disk_heartbeat so that we avoid more than * hr_timeout_ms between disk writes. On busy systems @@ -1264,7 +1310,7 @@ static int o2hb_thread(void *data) } /* Unpin node */ - o2nm_undepend_this_node(); + o2nm_undepend_node(reg->hr_node_num); mlog(ML_HEARTBEAT|ML_KTHREAD, "o2hb thread exiting\n"); @@ -1511,33 +1557,47 @@ static struct o2hb_region *to_o2hb_region(struct config_item *item) return item ? container_of(item, struct o2hb_region, hr_item) : NULL; } -/* drop_item only drops its ref after killing the thread, nothing should - * be using the region anymore. this has to clean up any state that - * attributes might have built up. */ -static void o2hb_region_release(struct config_item *item) +static void o2hb_unmap_slot_data(struct o2hb_region *reg) { int i; struct page *page; - struct o2hb_region *reg = to_o2hb_region(item); - - mlog(ML_HEARTBEAT, "hb region release (%pg)\n", reg->hr_bdev); - - kfree(reg->hr_tmp_block); if (reg->hr_slot_data) { for (i = 0; i < reg->hr_num_pages; i++) { page = reg->hr_slot_data[i]; - if (page) + if (page) { __free_page(page); + reg->hr_slot_data[i] = NULL; + } } kfree(reg->hr_slot_data); + reg->hr_slot_data = NULL; } + kfree(reg->hr_slots); + reg->hr_slots = NULL; + + kfree(reg->hr_tmp_block); + reg->hr_tmp_block = NULL; +} + +/* drop_item only drops its ref after killing the thread, nothing should + * be using the region anymore. this has to clean up any state that + * attributes might have built up. + */ +static void o2hb_region_release(struct config_item *item) +{ + struct o2hb_region *reg = to_o2hb_region(item); + + mlog(ML_HEARTBEAT, "hb region release (%pg)\n", reg->hr_bdev); + + o2hb_quiesce_timeout(reg); + o2net_unregister_and_flush_handler_list(®->hr_handler_list); + o2hb_unmap_slot_data(reg); + if (reg->hr_bdev) blkdev_put(reg->hr_bdev, NULL); - kfree(reg->hr_slots); - debugfs_remove_recursive(reg->hr_debug_dir); kfree(reg->hr_db_livenodes); kfree(reg->hr_db_regnum); @@ -1548,7 +1608,6 @@ static void o2hb_region_release(struct config_item *item) list_del(®->hr_all_item); spin_unlock(&o2hb_live_lock); - o2net_unregister_handler_list(®->hr_handler_list); kfree(reg); } @@ -1688,6 +1747,7 @@ static void o2hb_init_region_params(struct o2hb_region *reg) static int o2hb_map_slot_data(struct o2hb_region *reg) { int i, j; + int ret = -ENOMEM; unsigned int last_slot; unsigned int spp = reg->hr_slots_per_page; struct page *page; @@ -1695,15 +1755,15 @@ static int o2hb_map_slot_data(struct o2hb_region *reg) struct o2hb_disk_slot *slot; reg->hr_tmp_block = kmalloc(reg->hr_block_bytes, GFP_KERNEL); - if (reg->hr_tmp_block == NULL) - return -ENOMEM; + if (!reg->hr_tmp_block) + goto out; reg->hr_slots = kcalloc(reg->hr_blocks, sizeof(struct o2hb_disk_slot), GFP_KERNEL); - if (reg->hr_slots == NULL) - return -ENOMEM; + if (!reg->hr_slots) + goto out; - for(i = 0; i < reg->hr_blocks; i++) { + for (i = 0; i < reg->hr_blocks; i++) { slot = ®->hr_slots[i]; slot->ds_node_num = i; INIT_LIST_HEAD(&slot->ds_live_item); @@ -1718,12 +1778,12 @@ static int o2hb_map_slot_data(struct o2hb_region *reg) reg->hr_slot_data = kcalloc(reg->hr_num_pages, sizeof(struct page *), GFP_KERNEL); if (!reg->hr_slot_data) - return -ENOMEM; + goto out; - for(i = 0; i < reg->hr_num_pages; i++) { + for (i = 0; i < reg->hr_num_pages; i++) { page = alloc_page(GFP_KERNEL); if (!page) - return -ENOMEM; + goto out; reg->hr_slot_data[i] = page; @@ -1743,6 +1803,10 @@ static int o2hb_map_slot_data(struct o2hb_region *reg) } return 0; + +out: + o2hb_unmap_slot_data(reg); + return ret; } /* Read in all the slots available and populate the tracking @@ -1795,7 +1859,8 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, /* We can't heartbeat without having had our node number * configured yet. */ - if (o2nm_this_node() == O2NM_MAX_NODES) + reg->hr_node_num = o2nm_this_node(); + if (reg->hr_node_num == O2NM_MAX_NODES) goto out; fd = simple_strtol(p, &p, 0); @@ -1834,6 +1899,8 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, goto out3; } + reg->hr_aborted_start = 0; + reg->hr_node_deleted = 0; o2hb_init_region_params(reg); /* Generation of zero is invalid */ @@ -1854,9 +1921,6 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, goto out3; } - INIT_DELAYED_WORK(®->hr_write_timeout_work, o2hb_write_timeout); - INIT_DELAYED_WORK(®->hr_nego_timeout_work, o2hb_nego_timeout); - /* * A node is considered live after it has beat LIVE_THRESHOLD * times. We're not steady until we've given them a chance @@ -1876,6 +1940,7 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, atomic_set(®->hr_steady_iterations, live_threshold); /* unsteady_iterations is triple the steady_iterations */ atomic_set(®->hr_unsteady_iterations, (live_threshold * 3)); + o2hb_set_region_stopping(reg, false); hb_task = kthread_run(o2hb_thread, reg, "o2hb-%s", reg->hr_item.ci_name); @@ -1925,6 +1990,18 @@ static ssize_t o2hb_region_dev_store(struct config_item *item, out3: if (ret < 0) { + o2hb_quiesce_timeout(reg); + + spin_lock(&o2hb_live_lock); + hb_task = reg->hr_task; + reg->hr_task = NULL; + spin_unlock(&o2hb_live_lock); + + if (hb_task) + kthread_stop(hb_task); + + o2hb_unmap_slot_data(reg); + blkdev_put(reg->hr_bdev, NULL); reg->hr_bdev = NULL; } @@ -2032,6 +2109,7 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g ret = -ENAMETOOLONG; goto free; } + reg->hr_node_num = O2NM_MAX_NODES; spin_lock(&o2hb_live_lock); reg->hr_region_num = 0; @@ -2056,6 +2134,10 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g */ reg->hr_key = crc32_le(reg->hr_region_num + O2NM_MAX_REGIONS, name, strlen(name)); + mutex_init(®->hr_arming_mutex); + reg->hr_stopping = true; + INIT_DELAYED_WORK(®->hr_write_timeout_work, o2hb_write_timeout); + INIT_DELAYED_WORK(®->hr_nego_timeout_work, o2hb_nego_timeout); INIT_LIST_HEAD(®->hr_handler_list); ret = o2net_register_handler(O2HB_NEGO_TIMEOUT_MSG, reg->hr_key, sizeof(struct o2hb_nego_msg), @@ -2076,7 +2158,7 @@ static struct config_item *o2hb_heartbeat_group_make_item(struct config_group *g return ®->hr_item; unregister_handler: - o2net_unregister_handler_list(®->hr_handler_list); + o2net_unregister_and_flush_handler_list(®->hr_handler_list); remove_item: spin_lock(&o2hb_live_lock); list_del(®->hr_all_item); @@ -2095,6 +2177,8 @@ static void o2hb_heartbeat_group_drop_item(struct config_group *group, struct o2hb_region *reg = to_o2hb_region(item); int quorum_region = 0; + o2hb_quiesce_timeout(reg); + /* stop the thread when the user removes the region dir */ spin_lock(&o2hb_live_lock); hb_task = reg->hr_task; diff --git a/fs/ocfs2/cluster/nodemanager.c b/fs/ocfs2/cluster/nodemanager.c index 0edfd21be80e..b65fc6eeae6e 100644 --- a/fs/ocfs2/cluster/nodemanager.c +++ b/fs/ocfs2/cluster/nodemanager.c @@ -368,6 +368,7 @@ static ssize_t o2nm_node_local_store(struct config_item *item, const char *page, if (!tmp && cluster->cl_has_local && cluster->cl_local_node == node->nd_num) { o2net_stop_listening(node); + cluster->cl_has_local = 0; cluster->cl_local_node = O2NM_INVALID_NODE_NUM; } @@ -791,12 +792,12 @@ void o2nm_undepend_item(struct config_item *item) configfs_undepend_item(item); } -int o2nm_depend_this_node(void) +int o2nm_depend_node(u8 node_num) { int ret = 0; struct o2nm_node *local_node; - local_node = o2nm_get_node_by_num(o2nm_this_node()); + local_node = o2nm_get_node_by_num(node_num); if (!local_node) { ret = -EINVAL; goto out; @@ -809,17 +810,27 @@ int o2nm_depend_this_node(void) return ret; } -void o2nm_undepend_this_node(void) +void o2nm_undepend_node(u8 node_num) { struct o2nm_node *local_node; - local_node = o2nm_get_node_by_num(o2nm_this_node()); + local_node = o2nm_get_node_by_num(node_num); BUG_ON(!local_node); o2nm_undepend_item(&local_node->nd_item); o2nm_node_put(local_node); } +int o2nm_depend_this_node(void) +{ + return o2nm_depend_node(o2nm_this_node()); +} + +void o2nm_undepend_this_node(void) +{ + o2nm_undepend_node(o2nm_this_node()); +} + static void __exit exit_o2nm(void) { diff --git a/fs/ocfs2/cluster/nodemanager.h b/fs/ocfs2/cluster/nodemanager.h index 2f72f56996bd..ca3483fb5450 100644 --- a/fs/ocfs2/cluster/nodemanager.h +++ b/fs/ocfs2/cluster/nodemanager.h @@ -66,6 +66,8 @@ void o2nm_node_put(struct o2nm_node *node); int o2nm_depend_item(struct config_item *item); int o2nm_depend_item_unlocked(struct config_item *item); void o2nm_undepend_item(struct config_item *item); +int o2nm_depend_node(u8 node_num); +void o2nm_undepend_node(u8 node_num); int o2nm_depend_this_node(void); void o2nm_undepend_this_node(void); diff --git a/fs/ocfs2/cluster/tcp.c b/fs/ocfs2/cluster/tcp.c index 2008bd847986..cdfccff3cb77 100644 --- a/fs/ocfs2/cluster/tcp.c +++ b/fs/ocfs2/cluster/tcp.c @@ -38,6 +38,8 @@ */ #include +#include +#include #include #include #include @@ -101,10 +103,15 @@ static struct socket *o2net_listen_sock; * o2net_wq. teardown detaches the callbacks before destroying the workqueue. * quorum work is queued as sock containers are shutdown.. stop_listening * tears down all the node's sock containers, preventing future shutdowns - * and queued quroum work, before canceling delayed quorum work and - * destroying the work queue. + * and queued quorum work, before canceling delayed quorum work and + * destroying the work queue. Handler teardown can also race local listener + * shutdown, so keep a waitable destroying pointer until the old ordered + * queue has finished draining. */ static struct workqueue_struct *o2net_wq; +static struct workqueue_struct *o2net_wq_destroying; +static DEFINE_MUTEX(o2net_wq_mutex); +static DECLARE_COMPLETION(o2net_wq_destroyed); /* Heartbeat callbacks stay registered across local-node off/on. */ static bool o2net_listening; static struct work_struct o2net_listen_work; @@ -886,6 +893,27 @@ void o2net_unregister_handler_list(struct list_head *list) } EXPORT_SYMBOL_GPL(o2net_unregister_handler_list); +static void o2net_flush_wq(void) +{ + mutex_lock(&o2net_wq_mutex); + if (o2net_wq_destroying) { + mutex_unlock(&o2net_wq_mutex); + wait_for_completion(&o2net_wq_destroyed); + return; + } + + if (o2net_wq) + flush_workqueue(o2net_wq); + mutex_unlock(&o2net_wq_mutex); +} + +void o2net_unregister_and_flush_handler_list(struct list_head *list) +{ + o2net_unregister_handler_list(list); + o2net_flush_wq(); +} +EXPORT_SYMBOL_GPL(o2net_unregister_and_flush_handler_list); + static struct o2net_msg_handler *o2net_handler_get(u32 msg_type, u32 key) { struct o2net_msg_handler *nmh; @@ -1716,12 +1744,10 @@ void o2net_disconnect_node(struct o2nm_node *node) o2net_set_nn_state(nn, NULL, 0, -ENOTCONN); spin_unlock(&nn->nn_lock); - if (o2net_wq) { - cancel_delayed_work(&nn->nn_connect_expired); - cancel_delayed_work(&nn->nn_connect_work); - cancel_delayed_work(&nn->nn_still_up); - flush_workqueue(o2net_wq); - } + cancel_delayed_work(&nn->nn_connect_expired); + cancel_delayed_work(&nn->nn_connect_work); + cancel_delayed_work(&nn->nn_still_up); + o2net_flush_wq(); } static void o2net_hb_node_down_cb(struct o2nm_node *node, int node_num, @@ -2063,6 +2089,36 @@ static int o2net_open_listening_sock(__be32 addr, __be16 port) return ret; } +static void o2net_destroy_wq(void) +{ + struct workqueue_struct *wq; + + mutex_lock(&o2net_wq_mutex); + if (o2net_wq_destroying) { + mutex_unlock(&o2net_wq_mutex); + wait_for_completion(&o2net_wq_destroyed); + return; + } + + wq = o2net_wq; + if (!wq) { + mutex_unlock(&o2net_wq_mutex); + return; + } + + reinit_completion(&o2net_wq_destroyed); + o2net_wq_destroying = wq; + mutex_unlock(&o2net_wq_mutex); + + destroy_workqueue(wq); + + mutex_lock(&o2net_wq_mutex); + o2net_wq = NULL; + o2net_wq_destroying = NULL; + complete_all(&o2net_wq_destroyed); + mutex_unlock(&o2net_wq_mutex); +} + /* * called from node manager when we should bring up our network listening * socket. node manager handles all the serialization to only call this @@ -2073,24 +2129,44 @@ static int o2net_open_listening_sock(__be32 addr, __be16 port) int o2net_start_listening(struct o2nm_node *node) { int ret = 0; + struct workqueue_struct *wq; if (WARN_ON_ONCE(READ_ONCE(o2net_listening))) return -EBUSY; - BUG_ON(o2net_wq != NULL); + + mutex_lock(&o2net_wq_mutex); + if (o2net_wq_destroying) { + mutex_unlock(&o2net_wq_mutex); + return -EBUSY; + } + if (WARN_ON_ONCE(o2net_wq)) { + mutex_unlock(&o2net_wq_mutex); + return -EBUSY; + } + mutex_unlock(&o2net_wq_mutex); + BUG_ON(o2net_listen_sock != NULL); mlog(ML_KTHREAD, "starting o2net thread...\n"); - o2net_wq = alloc_ordered_workqueue("o2net", WQ_MEM_RECLAIM); - if (o2net_wq == NULL) { + wq = alloc_ordered_workqueue("o2net", WQ_MEM_RECLAIM); + if (!wq) { mlog(ML_ERROR, "unable to launch o2net thread\n"); return -ENOMEM; /* ? */ } + mutex_lock(&o2net_wq_mutex); + if (unlikely(o2net_wq_destroying || o2net_wq)) { + mutex_unlock(&o2net_wq_mutex); + destroy_workqueue(wq); + return -EBUSY; + } + o2net_wq = wq; + mutex_unlock(&o2net_wq_mutex); + ret = o2net_open_listening_sock(node->nd_ipv4_address, node->nd_ipv4_port); if (ret) { - destroy_workqueue(o2net_wq); - o2net_wq = NULL; + o2net_destroy_wq(); } else o2quo_conn_up(node->nd_num); @@ -2126,8 +2202,7 @@ void o2net_stop_listening(struct o2nm_node *node) /* finish all work and tear down the work queue */ mlog(ML_KTHREAD, "waiting for o2net thread to exit....\n"); - destroy_workqueue(o2net_wq); - o2net_wq = NULL; + o2net_destroy_wq(); sock_release(o2net_listen_sock); o2net_listen_sock = NULL; diff --git a/fs/ocfs2/cluster/tcp.h b/fs/ocfs2/cluster/tcp.h index 2e86d42b5faf..a11bcee28947 100644 --- a/fs/ocfs2/cluster/tcp.h +++ b/fs/ocfs2/cluster/tcp.h @@ -89,6 +89,7 @@ int o2net_register_handler(u32 msg_type, u32 key, u32 max_len, o2net_post_msg_handler_func *post_func, struct list_head *unreg_list); void o2net_unregister_handler_list(struct list_head *list); +void o2net_unregister_and_flush_handler_list(struct list_head *list); void o2net_fill_node_map(unsigned long *map, unsigned bytes); diff --git a/fs/ocfs2/dir.c b/fs/ocfs2/dir.c index 49eb03107caa..906c59783713 100644 --- a/fs/ocfs2/dir.c +++ b/fs/ocfs2/dir.c @@ -580,7 +580,7 @@ static int ocfs2_validate_dx_root(struct super_block *sb, mlog(ML_ERROR, "Checksum failed for dir index root block %llu\n", (unsigned long long)bh->b_blocknr); - return ret; + goto bail; } if (!OCFS2_IS_VALID_DX_ROOT(dx_root)) { @@ -588,8 +588,54 @@ static int ocfs2_validate_dx_root(struct super_block *sb, "Dir Index Root # %llu has bad signature %.*s\n", (unsigned long long)le64_to_cpu(dx_root->dr_blkno), 7, dx_root->dr_signature); + goto bail; } + if (!(dx_root->dr_flags & OCFS2_DX_FLAG_INLINE)) { + struct ocfs2_extent_list *el = &dx_root->dr_list; + + if (le16_to_cpu(el->l_count) != ocfs2_extent_recs_per_dx_root(sb)) { + ret = ocfs2_error(sb, + "Dir Index Root # %llu has invalid l_count %u (expected %u)\n", + (unsigned long long)le64_to_cpu(dx_root->dr_blkno), + le16_to_cpu(el->l_count), + ocfs2_extent_recs_per_dx_root(sb)); + goto bail; + } + + if (le16_to_cpu(el->l_next_free_rec) > le16_to_cpu(el->l_count)) { + ret = ocfs2_error(sb, + "Dir Index Root # %llu has invalid l_next_free_rec %u (l_count %u)\n", + (unsigned long long)le64_to_cpu(dx_root->dr_blkno), + le16_to_cpu(el->l_next_free_rec), + le16_to_cpu(el->l_count)); + goto bail; + } + } else { + struct ocfs2_dx_entry_list *dl_list = &dx_root->dr_entries; + + if (le16_to_cpu(dl_list->de_count) != + ocfs2_dx_entries_per_root(sb)) { + ret = ocfs2_error(sb, + "Dir Index Root # %llu has invalid de_count %u (expected %u)\n", + (unsigned long long)le64_to_cpu(dx_root->dr_blkno), + le16_to_cpu(dl_list->de_count), + ocfs2_dx_entries_per_root(sb)); + goto bail; + } + + if (le16_to_cpu(dl_list->de_num_used) > + le16_to_cpu(dl_list->de_count)) { + ret = ocfs2_error(sb, + "Dir Index Root # %llu has invalid de_num_used %u (de_count %u)\n", + (unsigned long long)le64_to_cpu(dx_root->dr_blkno), + le16_to_cpu(dl_list->de_num_used), + le16_to_cpu(dl_list->de_count)); + goto bail; + } + } + +bail: return ret; } @@ -626,10 +672,25 @@ static int ocfs2_validate_dx_leaf(struct super_block *sb, return ret; } - if (!OCFS2_IS_VALID_DX_LEAF(dx_leaf)) { - ret = ocfs2_error(sb, "Dir Index Leaf has bad signature %.*s\n", - 7, dx_leaf->dl_signature); - } + if (!OCFS2_IS_VALID_DX_LEAF(dx_leaf)) + return ocfs2_error(sb, "Dir Index Leaf has bad signature %.*s\n", + 7, dx_leaf->dl_signature); + + if (le16_to_cpu(dx_leaf->dl_list.de_count) != + ocfs2_dx_entries_per_leaf(sb)) + return ocfs2_error(sb, + "Dir Index Leaf # %llu has invalid de_count %u (expected %u)\n", + (unsigned long long)le64_to_cpu(dx_leaf->dl_blkno), + le16_to_cpu(dx_leaf->dl_list.de_count), + ocfs2_dx_entries_per_leaf(sb)); + + if (le16_to_cpu(dx_leaf->dl_list.de_num_used) > + le16_to_cpu(dx_leaf->dl_list.de_count)) + return ocfs2_error(sb, + "Dir Index Leaf # %llu has invalid de_num_used %u (de_count %u)\n", + (unsigned long long)le64_to_cpu(dx_leaf->dl_blkno), + le16_to_cpu(dx_leaf->dl_list.de_num_used), + le16_to_cpu(dx_leaf->dl_list.de_count)); return ret; } diff --git a/fs/ocfs2/inode.c b/fs/ocfs2/inode.c index bbf8dfeeac74..01e33782f4f6 100644 --- a/fs/ocfs2/inode.c +++ b/fs/ocfs2/inode.c @@ -233,6 +233,20 @@ struct inode *ocfs2_iget(struct ocfs2_super *osb, u64 blkno, unsigned flags, return inode; } +static int ocfs2_dinode_has_extents(struct ocfs2_dinode *di) +{ + /* inodes flagged with other stuff in id2 */ + if (di->i_flags & (OCFS2_SUPER_BLOCK_FL | OCFS2_LOCAL_ALLOC_FL | + OCFS2_CHAIN_FL | OCFS2_DEALLOC_FL)) + return 0; + /* i_flags doesn't indicate when id2 is a fast symlink */ + if (S_ISLNK(di->i_mode) && di->i_size && di->i_clusters == 0) + return 0; + if (di->i_dyn_features & OCFS2_INLINE_DATA_FL) + return 0; + + return 1; +} /* * here's how inodes get read from disk: @@ -1452,6 +1466,22 @@ int ocfs2_validate_inode_block(struct super_block *sb, goto bail; } + if ((le32_to_cpu(di->i_flags) & OCFS2_DIO_ORPHANED_FL) && + le16_to_cpu(di->i_dio_orphaned_slot) >= OCFS2_SB(sb)->max_slots) { + rc = ocfs2_error(sb, "Invalid dinode %llu: DIO orphaned slot %u\n", + (unsigned long long)bh->b_blocknr, + le16_to_cpu(di->i_dio_orphaned_slot)); + goto bail; + } + + if ((le32_to_cpu(di->i_flags) & OCFS2_ORPHANED_FL) && + le16_to_cpu(di->i_orphaned_slot) >= OCFS2_SB(sb)->max_slots) { + rc = ocfs2_error(sb, "Invalid dinode %llu: orphaned slot %u\n", + (unsigned long long)bh->b_blocknr, + le16_to_cpu(di->i_orphaned_slot)); + goto bail; + } + /* * Reject dinodes whose i_mode does not name one of the seven * canonical POSIX file types. ocfs2_populate_inode() copies @@ -1532,6 +1562,10 @@ int ocfs2_validate_inode_block(struct super_block *sb, goto bail; } + rc = ocfs2_validate_inode_xattr(sb, bh->b_blocknr, di); + if (rc) + goto bail; + if (le16_to_cpu(di->i_dyn_features) & OCFS2_INLINE_DATA_FL) { struct ocfs2_inline_data *data = &di->id2.i_data; @@ -1563,6 +1597,93 @@ int ocfs2_validate_inode_block(struct super_block *sb, } } + if (le16_to_cpu(di->i_suballoc_slot) != (u16)OCFS2_INVALID_SLOT && + (u32)le16_to_cpu(di->i_suballoc_slot) > OCFS2_SB(sb)->max_slots - 1) { + rc = ocfs2_error(sb, "Invalid dinode %llu: suballoc slot %u\n", + (unsigned long long)bh->b_blocknr, + le16_to_cpu(di->i_suballoc_slot)); + goto bail; + } + + if ((le16_to_cpu(di->i_dyn_features) & OCFS2_HAS_REFCOUNT_FL) && + !di->i_refcount_loc) { + rc = ocfs2_error(sb, "Inode #%llu has refcount flag but no i_refcount_loc\n", + (unsigned long long)bh->b_blocknr); + goto bail; + } + + if (ocfs2_dinode_has_extents(di)) { + struct ocfs2_extent_list *el = &di->id2.i_list; + u16 count = le16_to_cpu(el->l_count); + u16 next_free = le16_to_cpu(el->l_next_free_rec); + + if (count == 0) { + rc = ocfs2_error(sb, + "Invalid dinode %llu: extent list l_count is zero\n", + (unsigned long long)bh->b_blocknr); + goto bail; + } + /* + * The exact capacity depends on i_xattr_inline_size, another + * unvalidated on-disk field. Inline xattrs only shrink the + * list, so the no-xattr maximum is a safe upper bound that a + * valid l_count never exceeds. + */ + if (count > ocfs2_extent_recs_per_inode(sb)) { + rc = ocfs2_error(sb, + "Invalid dinode %llu: extent list l_count %u exceeds max %u\n", + (unsigned long long)bh->b_blocknr, count, + ocfs2_extent_recs_per_inode(sb)); + goto bail; + } + if (next_free > count) { + rc = ocfs2_error(sb, + "Invalid dinode %llu: extent list l_next_free_rec %u exceeds l_count %u\n", + (unsigned long long)bh->b_blocknr, next_free, count); + goto bail; + } + } + + if (S_ISLNK(le16_to_cpu(di->i_mode)) && + !le32_to_cpu(di->i_clusters)) { + int max_inline = ocfs2_fast_symlink_chars(sb); + u64 i_size = le64_to_cpu(di->i_size); + + if (i_size >= max_inline) { + rc = ocfs2_error(sb, + "Invalid dinode #%llu: fast symlink i_size %llu exceeds max %d\n", + (unsigned long long)bh->b_blocknr, + (unsigned long long)i_size, + max_inline - 1); + goto bail; + } + + if (strnlen((char *)di->id2.i_symlink, i_size + 1) != i_size) { + rc = ocfs2_error(sb, + "Invalid dinode #%llu: fast symlink is not NUL-terminated at i_size %llu\n", + (unsigned long long)bh->b_blocknr, + (unsigned long long)i_size); + goto bail; + } + } + + if (le32_to_cpu(di->i_flags) & OCFS2_CHAIN_FL) { + struct ocfs2_chain_list *cl = &di->id2.i_chain; + + if (le16_to_cpu(cl->cl_count) != ocfs2_chain_recs_per_inode(sb)) { + rc = ocfs2_error(sb, "Invalid dinode %llu: chain list count %u\n", + (unsigned long long)bh->b_blocknr, + le16_to_cpu(cl->cl_count)); + goto bail; + } + if (le16_to_cpu(cl->cl_next_free_rec) > le16_to_cpu(cl->cl_count)) { + rc = ocfs2_error(sb, "Invalid dinode %llu: chain list index %u\n", + (unsigned long long)bh->b_blocknr, + le16_to_cpu(cl->cl_next_free_rec)); + goto bail; + } + } + rc = 0; bail: @@ -1728,6 +1849,16 @@ static int ocfs2_filecheck_repair_inode_block(struct super_block *sb, le32_to_cpu(di->i_fs_generation)); } + if (ocfs2_dinode_has_extents(di) && + le16_to_cpu(di->id2.i_list.l_next_free_rec) > le16_to_cpu(di->id2.i_list.l_count)) { + di->id2.i_list.l_next_free_rec = di->id2.i_list.l_count; + changed = 1; + mlog(ML_ERROR, + "Filecheck: reset dinode #%llu: l_next_free_rec to %u\n", + (unsigned long long)bh->b_blocknr, + le16_to_cpu(di->id2.i_list.l_next_free_rec)); + } + if (changed || ocfs2_validate_meta_ecc(sb, bh->b_data, &di->i_check)) { ocfs2_compute_meta_ecc(sb, bh->b_data, &di->i_check); mark_buffer_dirty(bh); diff --git a/fs/ocfs2/namei.c b/fs/ocfs2/namei.c index 111e60ac19b9..886b0ddd883f 100644 --- a/fs/ocfs2/namei.c +++ b/fs/ocfs2/namei.c @@ -1342,7 +1342,7 @@ static int ocfs2_rename(struct mnt_idmap *idmap, goto bail; } - if (S_ISDIR(old_inode->i_mode)) { + if (S_ISDIR(old_inode->i_mode) && new_dir != old_dir) { u64 old_inode_parent; update_dot_dot = 1; @@ -1359,8 +1359,7 @@ static int ocfs2_rename(struct mnt_idmap *idmap, goto bail; } - if (!new_inode && new_dir != old_dir && - new_dir->i_nlink >= ocfs2_link_max(osb)) { + if (!new_inode && new_dir->i_nlink >= ocfs2_link_max(osb)) { status = -EMLINK; goto bail; } @@ -1603,6 +1602,9 @@ static int ocfs2_rename(struct mnt_idmap *idmap, if (update_dot_dot) { status = ocfs2_update_entry(old_inode, handle, &old_inode_dot_dot_res, new_dir); + } + + if (S_ISDIR(old_inode->i_mode)) { drop_nlink(old_dir); if (new_inode) { drop_nlink(new_inode); diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c index cc4bba65f152..e595c22282e8 100644 --- a/fs/ocfs2/super.c +++ b/fs/ocfs2/super.c @@ -2030,8 +2030,8 @@ static int ocfs2_initialize_super(struct super_block *sb, cbits = le32_to_cpu(di->id2.i_super.s_clustersize_bits); bbits = le32_to_cpu(di->id2.i_super.s_blocksize_bits); sb->s_maxbytes = ocfs2_max_file_offset(bbits, cbits); - memcpy(&sb->s_uuid, di->id2.i_super.s_uuid, - sizeof(di->id2.i_super.s_uuid)); + super_set_uuid(sb, di->id2.i_super.s_uuid, + sizeof(di->id2.i_super.s_uuid)); osb->osb_dx_mask = (1 << (cbits - bbits)) - 1; diff --git a/fs/ocfs2/xattr.c b/fs/ocfs2/xattr.c index dc3d88a09d10..72b6b73d8372 100644 --- a/fs/ocfs2/xattr.c +++ b/fs/ocfs2/xattr.c @@ -386,6 +386,12 @@ static int ocfs2_init_xattr_bucket(struct ocfs2_xattr_bucket *bucket, return rc; } +static int ocfs2_validate_xattr_entries_flat(struct super_block *sb, u64 blkno, + struct ocfs2_xattr_header *xh, + size_t region_size); +static int ocfs2_validate_xattr_bucket(struct ocfs2_xattr_bucket *bucket, + u64 blkno); + /* Read the xattr bucket at xb_blkno */ static int ocfs2_read_xattr_bucket(struct ocfs2_xattr_bucket *bucket, u64 xb_blkno) @@ -404,6 +410,8 @@ static int ocfs2_read_xattr_bucket(struct ocfs2_xattr_bucket *bucket, spin_unlock(&OCFS2_SB(bucket->bu_inode->i_sb)->osb_xattr_lock); if (rc) mlog_errno(rc); + else + rc = ocfs2_validate_xattr_bucket(bucket, xb_blkno); } if (rc) @@ -505,6 +513,22 @@ static int ocfs2_validate_xattr_block(struct super_block *sb, le32_to_cpu(xb->xb_fs_generation)); } + if (!(le16_to_cpu(xb->xb_flags) & OCFS2_XATTR_INDEXED)) { + size_t region_offset = + offsetof(struct ocfs2_xattr_block, xb_attrs.xb_header); + + if (bh->b_size < region_offset) + return ocfs2_error(sb, + "Invalid xattr block %llu: block size %zu is too small\n", + (unsigned long long)bh->b_blocknr, + bh->b_size); + + return ocfs2_validate_xattr_entries_flat(sb, bh->b_blocknr, + &xb->xb_attrs.xb_header, + bh->b_size - + region_offset); + } + return 0; } @@ -953,6 +977,197 @@ static int ocfs2_xattr_list_entries(struct inode *inode, return result; } +static int ocfs2_validate_xattr_entries_flat(struct super_block *sb, u64 blkno, + struct ocfs2_xattr_header *xh, + size_t region_size) +{ + u16 xattr_count = le16_to_cpu(xh->xh_count); + size_t entries_limit = region_size; + size_t nv_limit = region_size; + size_t max_entries; + int i; + + if (region_size < sizeof(*xh)) + return ocfs2_error(sb, + "Invalid xattr in block %llu: region size %zu is too small\n", + (unsigned long long)blkno, region_size); + + max_entries = (entries_limit - sizeof(*xh)) / + sizeof(struct ocfs2_xattr_entry); + + if (xattr_count > max_entries) + return ocfs2_error(sb, + "Invalid xattr in block %llu: entry count %u exceeds maximum %zu\n", + (unsigned long long)blkno, + xattr_count, max_entries); + + for (i = 0; i < xattr_count; i++) { + struct ocfs2_xattr_entry *xe = &xh->xh_entries[i]; + size_t name_offset = le16_to_cpu(xe->xe_name_offset); + size_t value_offset; + + if (name_offset > nv_limit || + xe->xe_name_len > nv_limit - name_offset) + return ocfs2_error(sb, + "Invalid xattr in block %llu: entry %d name is out of bounds\n", + (unsigned long long)blkno, i); + + value_offset = name_offset + OCFS2_XATTR_SIZE(xe->xe_name_len); + if (value_offset > nv_limit) + return ocfs2_error(sb, + "Invalid xattr in block %llu: entry %d value starts out of bounds\n", + (unsigned long long)blkno, i); + + if (ocfs2_xattr_is_local(xe)) { + if (le64_to_cpu(xe->xe_value_size) > + nv_limit - value_offset) + return ocfs2_error(sb, + "Invalid xattr in block %llu: entry %d value is out of bounds\n", + (unsigned long long)blkno, + i); + } else if (sizeof(struct ocfs2_xattr_value_root) > + nv_limit - value_offset) { + return ocfs2_error(sb, + "Invalid xattr in block %llu: entry %d value root is out of bounds\n", + (unsigned long long)blkno, i); + } + } + + return 0; +} + +static int ocfs2_xattr_ibody_lookup_header_raw(struct super_block *sb, + u64 blkno, + struct ocfs2_dinode *di, + struct ocfs2_xattr_header **header, + u16 *inline_size_ret) +{ + struct ocfs2_xattr_header *xh; + u16 xattr_count; + size_t max_entries; + u16 inline_size = le16_to_cpu(di->i_xattr_inline_size); + + if (inline_size > sb->s_blocksize || + inline_size < sizeof(struct ocfs2_xattr_header)) { + ocfs2_error(sb, + "Invalid inode %llu: xattr inline size %u\n", + (unsigned long long)blkno, inline_size); + return -EFSCORRUPTED; + } + + xh = (struct ocfs2_xattr_header *) + ((void *)di + sb->s_blocksize - inline_size); + + xattr_count = le16_to_cpu(xh->xh_count); + max_entries = (inline_size - sizeof(struct ocfs2_xattr_header)) / + sizeof(struct ocfs2_xattr_entry); + + if (xattr_count > max_entries) { + ocfs2_error(sb, + "xattr entry count %u exceeds maximum %zu in inode %llu\n", + xattr_count, max_entries, + (unsigned long long)blkno); + return -EFSCORRUPTED; + } + + *header = xh; + if (inline_size_ret) + *inline_size_ret = inline_size; + + return 0; +} + +int ocfs2_validate_inode_xattr(struct super_block *sb, u64 blkno, + struct ocfs2_dinode *di) +{ + struct ocfs2_xattr_header *xh; + u16 inline_size; + int ret; + + if (!(le16_to_cpu(di->i_dyn_features) & OCFS2_INLINE_XATTR_FL)) + return 0; + + ret = ocfs2_xattr_ibody_lookup_header_raw(sb, blkno, di, &xh, + &inline_size); + if (ret) + return ret; + + return ocfs2_validate_xattr_entries_flat(sb, blkno, xh, inline_size); +} + +static int ocfs2_validate_xattr_bucket(struct ocfs2_xattr_bucket *bucket, + u64 blkno) +{ + struct super_block *sb = bucket->bu_inode->i_sb; + struct ocfs2_xattr_header *xh = bucket_xh(bucket); + u16 xattr_count = le16_to_cpu(xh->xh_count); + size_t region_size = (size_t)sb->s_blocksize * bucket->bu_blocks; + size_t entries_limit = sb->s_blocksize; + size_t nv_limit = sb->s_blocksize; + size_t max_entries; + int i; + + if (region_size < sizeof(*xh)) + return ocfs2_error(sb, + "Invalid xattr bucket %llu: region size %zu is too small\n", + (unsigned long long)blkno, region_size); + + if (entries_limit < sizeof(*xh)) + return ocfs2_error(sb, + "Invalid xattr bucket %llu: entries limit %zu is too small\n", + (unsigned long long)blkno, + entries_limit); + + max_entries = (entries_limit - sizeof(*xh)) / + sizeof(struct ocfs2_xattr_entry); + + if (xattr_count > max_entries) + return ocfs2_error(sb, + "Invalid xattr bucket %llu: entry count %u exceeds maximum %zu\n", + (unsigned long long)blkno, + xattr_count, max_entries); + + for (i = 0; i < xattr_count; i++) { + struct ocfs2_xattr_entry *xe = &xh->xh_entries[i]; + size_t name_offset = le16_to_cpu(xe->xe_name_offset); + size_t block_off = name_offset >> sb->s_blocksize_bits; + size_t block_offset = name_offset % nv_limit; + size_t value_offset; + + if (name_offset >= region_size || block_off >= bucket->bu_blocks) + return ocfs2_error(sb, + "Invalid xattr bucket %llu: entry %d name is out of bounds\n", + (unsigned long long)blkno, i); + + if (xe->xe_name_len > nv_limit - block_offset) + return ocfs2_error(sb, + "Invalid xattr bucket %llu: entry %d name crosses block boundary\n", + (unsigned long long)blkno, i); + + value_offset = block_offset + OCFS2_XATTR_SIZE(xe->xe_name_len); + if (value_offset > nv_limit) + return ocfs2_error(sb, + "Invalid xattr bucket %llu: entry %d value starts out of bounds\n", + (unsigned long long)blkno, i); + + if (ocfs2_xattr_is_local(xe)) { + if (le64_to_cpu(xe->xe_value_size) > + nv_limit - value_offset) + return ocfs2_error(sb, + "Invalid xattr bucket %llu: entry %d value is out of bounds\n", + (unsigned long long)blkno, + i); + } else if (sizeof(struct ocfs2_xattr_value_root) > + nv_limit - value_offset) { + return ocfs2_error(sb, + "Invalid xattr bucket %llu: entry %d value root is out of bounds\n", + (unsigned long long)blkno, i); + } + } + + return 0; +} + int ocfs2_has_inline_xattr_value_outside(struct inode *inode, struct ocfs2_dinode *di) { diff --git a/fs/ocfs2/xattr.h b/fs/ocfs2/xattr.h index 016374e9a147..2c4d895ed5eb 100644 --- a/fs/ocfs2/xattr.h +++ b/fs/ocfs2/xattr.h @@ -43,6 +43,8 @@ int ocfs2_xattr_set_handle(handle_t *, struct inode *, struct buffer_head *, struct ocfs2_alloc_context *); int ocfs2_has_inline_xattr_value_outside(struct inode *inode, struct ocfs2_dinode *di); +int ocfs2_validate_inode_xattr(struct super_block *sb, u64 blkno, + struct ocfs2_dinode *di); int ocfs2_xattr_remove(struct inode *, struct buffer_head *); int ocfs2_init_security_get(struct inode *, struct inode *, const struct qstr *, diff --git a/fs/overlayfs/file.c b/fs/overlayfs/file.c index 07f00cf91977..a493eed7f605 100644 --- a/fs/overlayfs/file.c +++ b/fs/overlayfs/file.c @@ -615,6 +615,7 @@ static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in, struct file *file_out, loff_t pos_out, loff_t len, unsigned int flags, enum ovl_copyop op) { + struct inode *inode_in = file_inode(file_in); struct inode *inode_out = file_inode(file_out); struct fd real_in, real_out; const struct cred *old_cred; @@ -639,7 +640,23 @@ static loff_t ovl_copyfile(struct file *file_in, loff_t pos_in, goto out_unlock; } - old_cred = ovl_override_creds(file_inode(file_out)->i_sb); + /* + * For cross-sb copy, vfs_copy_file_range() will verify read access with + * the mounter creds of the dest fs mounter, so we need to explicitly + * verify read access with the source mounter creds. + */ + if (unlikely(inode_in->i_sb != inode_out->i_sb)) { + old_cred = ovl_override_creds(inode_in->i_sb); + ret = rw_verify_area(READ, real_in.file, &pos_in, len); + revert_creds(old_cred); + if (unlikely(ret)) { + fdput(real_in); + fdput(real_out); + goto out_unlock; + } + } + + old_cred = ovl_override_creds(inode_out->i_sb); switch (op) { case OVL_COPY: ret = vfs_copy_file_range(real_in.file, pos_in, diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index d0896bf8e12f..5c6214d8e093 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -724,8 +724,11 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, struct mem_size_stats *mss = walk->private; struct vm_area_struct *vma = walk->vma; struct page *page = NULL; - pte_t ptent = ptep_get(pte); + spinlock_t *ptl; + pte_t ptent; + ptl = huge_pte_lock(hstate_vma(vma), walk->mm, pte); + ptent = ptep_get(pte); if (pte_present(ptent)) { page = vm_normal_page(vma, addr, ptent); } else if (is_swap_pte(ptent)) { @@ -740,6 +743,7 @@ static int smaps_hugetlb_range(pte_t *pte, unsigned long hmask, else mss->private_hugetlb += huge_page_size(hstate_vma(vma)); } + spin_unlock(ptl); return 0; } #else @@ -865,7 +869,8 @@ static int show_smap(struct seq_file *m, void *v) __show_smap(m, &mss, false); seq_printf(m, "THPeligible: %8u\n", - hugepage_vma_check(vma, vma->vm_flags, true, false, true)); + hugepage_vma_check(vma, vma->vm_flags, + TVA_SMAPS | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER))); if (arch_pkeys_enabled()) seq_printf(m, "ProtectionKey: %8u\n", vma_pkey(vma)); diff --git a/fs/smb/client/cifsacl.c b/fs/smb/client/cifsacl.c index 0a7c98e76836..4ed7b6311af5 100644 --- a/fs/smb/client/cifsacl.c +++ b/fs/smb/client/cifsacl.c @@ -1024,13 +1024,13 @@ unsigned int setup_special_user_owner_ACE(struct smb_ace *pntace) static void populate_new_aces(char *nacl_base, struct smb_sid *pownersid, struct smb_sid *pgrpsid, - __u64 *pnmode, u16 *pnum_aces, u16 *pnsize, + __u64 *pnmode, u16 *pnum_aces, u32 *pnsize, bool modefromsid, bool posix) { __u64 nmode; u16 num_aces = 0; - u16 nsize = 0; + u32 nsize = 0; __u64 user_mode; __u64 group_mode; __u64 other_mode; @@ -1129,16 +1129,17 @@ static void populate_new_aces(char *nacl_base, *pnsize = nsize; } -static __u16 replace_sids_and_copy_aces(struct smb_acl *pdacl, struct smb_acl *pndacl, - struct smb_sid *pownersid, struct smb_sid *pgrpsid, - struct smb_sid *pnownersid, struct smb_sid *pngrpsid) +static int replace_sids_and_copy_aces(struct smb_acl *pdacl, struct smb_acl *pndacl, + struct smb_sid *pownersid, struct smb_sid *pgrpsid, + struct smb_sid *pnownersid, struct smb_sid *pngrpsid, + u16 *pnsize) { int i; u16 size = 0; struct smb_ace *pntace = NULL; char *acl_base = NULL; u16 src_num_aces = 0; - u16 nsize = 0; + u32 nsize = 0; struct smb_ace *pnntace = NULL; char *nacl_base = NULL; u16 ace_size = 0; @@ -1164,9 +1165,12 @@ static __u16 replace_sids_and_copy_aces(struct smb_acl *pdacl, struct smb_acl *p size += le16_to_cpu(pntace->size); nsize += ace_size; + if (nsize > U16_MAX) + return -EOVERFLOW; } - return nsize; + *pnsize = nsize; + return 0; } static int set_chmod_dacl(struct smb_acl *pdacl, struct smb_acl *pndacl, @@ -1178,7 +1182,7 @@ static int set_chmod_dacl(struct smb_acl *pdacl, struct smb_acl *pndacl, struct smb_ace *pntace = NULL; char *acl_base = NULL; u16 src_num_aces = 0; - u16 nsize = 0; + u32 nsize = 0; struct smb_ace *pnntace = NULL; char *nacl_base = NULL; u16 num_aces = 0; @@ -1229,6 +1233,8 @@ static int set_chmod_dacl(struct smb_acl *pdacl, struct smb_acl *pndacl, nsize += cifs_copy_ace(pnntace, pntace, NULL); num_aces++; + if (nsize > U16_MAX) + return -EOVERFLOW; next_ace: size += le16_to_cpu(pntace->size); @@ -1245,6 +1251,10 @@ static int set_chmod_dacl(struct smb_acl *pdacl, struct smb_acl *pndacl, } finalize_dacl: + /* The DACL size field is 16-bit on the wire, see MS-DTYP 2.4.5 */ + if (nsize > U16_MAX) + return -EOVERFLOW; + pndacl->num_aces = cpu_to_le16(num_aces); pndacl->size = cpu_to_le16(nsize); @@ -1302,6 +1312,7 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb, { int rc = 0; struct smb_sid *owner_sid_ptr, *group_sid_ptr; + unsigned int sbflags = cifs_sb->mnt_cifs_flags; struct smb_acl *dacl_ptr; /* no need for SACL ptr */ char *end_of_acl = ((char *)pntsd) + acl_len; __u32 dacloffset; @@ -1318,17 +1329,21 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb, pntsd->revision, pntsd->type, le32_to_cpu(pntsd->osidoffset), le32_to_cpu(pntsd->gsidoffset), le32_to_cpu(pntsd->sacloffset), dacloffset); -/* cifs_dump_mem("owner_sid: ", owner_sid_ptr, 64); */ + fattr->cf_uid = cifs_sb->ctx->linux_uid; + fattr->cf_gid = cifs_sb->ctx->linux_gid; + rc = parse_sid(owner_sid_ptr, end_of_acl); if (rc) { cifs_dbg(FYI, "%s: Error %d parsing Owner SID\n", __func__, rc); return rc; } - rc = sid_to_id(cifs_sb, owner_sid_ptr, fattr, SIDOWNER); - if (rc) { - cifs_dbg(FYI, "%s: Error %d mapping Owner SID to uid\n", - __func__, rc); - return rc; + if (!(sbflags & CIFS_MOUNT_OVERR_UID)) { + rc = sid_to_id(cifs_sb, owner_sid_ptr, fattr, SIDOWNER); + if (rc) { + cifs_dbg(FYI, "%s: Error %d mapping Owner SID to uid\n", + __func__, rc); + return rc; + } } rc = parse_sid(group_sid_ptr, end_of_acl); @@ -1337,11 +1352,13 @@ static int parse_sec_desc(struct cifs_sb_info *cifs_sb, __func__, rc); return rc; } - rc = sid_to_id(cifs_sb, group_sid_ptr, fattr, SIDGROUP); - if (rc) { - cifs_dbg(FYI, "%s: Error %d mapping Group SID to gid\n", - __func__, rc); - return rc; + if (!(sbflags & CIFS_MOUNT_OVERR_GID)) { + rc = sid_to_id(cifs_sb, group_sid_ptr, fattr, SIDGROUP); + if (rc) { + cifs_dbg(FYI, "%s: Error %d mapping Group SID to gid\n", + __func__, rc); + return rc; + } } if (dacloffset) { @@ -1405,6 +1422,8 @@ static int build_sec_desc(struct smb_ntsd *pntsd, struct smb_ntsd *pnntsd, rc = set_chmod_dacl(dacl_ptr, ndacl_ptr, owner_sid_ptr, group_sid_ptr, pnmode, mode_from_sid, posix); + if (rc) + return rc; sidsoffset = ndacloffset + le16_to_cpu(ndacl_ptr->size); /* copy the non-dacl portion of secdesc */ @@ -1480,9 +1499,12 @@ static int build_sec_desc(struct smb_ntsd *pntsd, struct smb_ntsd *pnntsd, if (dacloffset) { /* Replace ACEs for old owner with new one */ - size = replace_sids_and_copy_aces(dacl_ptr, ndacl_ptr, - owner_sid_ptr, group_sid_ptr, - nowner_sid_ptr, ngroup_sid_ptr); + rc = replace_sids_and_copy_aces(dacl_ptr, ndacl_ptr, + owner_sid_ptr, group_sid_ptr, + nowner_sid_ptr, ngroup_sid_ptr, + &size); + if (rc) + goto chown_chgrp_exit; ndacl_ptr->size = cpu_to_le16(size); } diff --git a/fs/smb/client/cifsencrypt.c b/fs/smb/client/cifsencrypt.c index 3d6b66321233..63712b83eb83 100644 --- a/fs/smb/client/cifsencrypt.c +++ b/fs/smb/client/cifsencrypt.c @@ -854,7 +854,6 @@ cifs_crypto_secmech_release(struct TCP_Server_Info *server) crypto_free_aead(server->secmech.enc); server->secmech.enc = NULL; } - if (server->secmech.dec) { crypto_free_aead(server->secmech.dec); server->secmech.dec = NULL; diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c index a1c5ef3b2f1e..a3dfeda8144d 100644 --- a/fs/smb/client/cifsfs.c +++ b/fs/smb/client/cifsfs.c @@ -401,6 +401,7 @@ cifs_alloc_inode(struct super_block *sb) return NULL; cifs_inode->cifsAttrs = 0x20; /* default */ cifs_inode->time = 0; + cifs_inode->time_last_write = 0; /* * Until the file is open and we have gotten oplock info back from the * server, can not assume caching of file data or metadata. diff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h index a4e0618c58ef..b6a89d051575 100644 --- a/fs/smb/client/cifsglob.h +++ b/fs/smb/client/cifsglob.h @@ -1417,11 +1417,11 @@ struct cifsFileInfo { struct dentry *dentry; struct tcon_link *tlink; unsigned int f_flags; - bool invalidHandle:1; /* file closed via session abend */ - bool swapfile:1; - bool oplock_break_cancelled:1; - bool status_file_deleted:1; /* file has been deleted */ - bool offload:1; /* offload final part of _put to a wq */ + bool invalidHandle; /* file closed via session abend */ + bool swapfile; + bool oplock_break_cancelled; + bool status_file_deleted; /* file has been deleted */ + bool offload; /* offload final part of _put to a wq */ __u16 oplock_epoch; /* epoch from the lease break */ __u32 oplock_level; /* oplock/lease level from the lease break */ int count; @@ -1575,6 +1575,7 @@ struct cifsInodeInfo { unsigned int writers; /* Number of writers on this inode */ unsigned long time; /* jiffies of last update of inode */ u64 server_eof; /* current file size on server -- protected by i_lock */ + unsigned long time_last_write; /* jiffies of last writable close or truncate */ u64 uniqueid; /* server inode number */ u64 createtime; /* creation time on server */ __u8 lease_key[SMB2_LEASE_KEY_SIZE]; /* lease key for this inode */ diff --git a/fs/smb/client/cifssmb.c b/fs/smb/client/cifssmb.c index e7df15011ae3..3ba39c375077 100644 --- a/fs/smb/client/cifssmb.c +++ b/fs/smb/client/cifssmb.c @@ -2793,7 +2793,7 @@ int cifs_query_reparse_point(const unsigned int xid, end = 2 + get_bcc(&io_rsp->hdr) + (__u8 *)&io_rsp->ByteCount; start = (__u8 *)&io_rsp->hdr.Protocol + data_offset; - if (start >= end) { + if (start >= end || (size_t)(end - start) < sizeof(*buf)) { rc = -EIO; goto error; } diff --git a/fs/smb/client/connect.c b/fs/smb/client/connect.c index 9da32fb95c9c..cbe8cb863c1c 100644 --- a/fs/smb/client/connect.c +++ b/fs/smb/client/connect.c @@ -3891,6 +3891,10 @@ CIFSTCon(const unsigned int xid, struct cifs_ses *ses, tcon->tid = smb_buffer_response->Tid; bcc_ptr = pByteArea(smb_buffer_response); bytes_left = get_bcc(smb_buffer_response); + if (bytes_left < 2) { + rc = -EIO; + goto out; + } length = strnlen(bcc_ptr, bytes_left - 2); if (smb_buffer->Flags2 & SMBFLG2_UNICODE) is_unicode = true; @@ -3946,6 +3950,7 @@ CIFSTCon(const unsigned int xid, struct cifs_ses *ses, reset_cifs_unix_caps(xid, tcon, NULL, NULL); } } +out: cifs_buf_release(smb_buffer); return rc; } diff --git a/fs/smb/client/dir.c b/fs/smb/client/dir.c index 6bb0e98462a9..66bf81d15962 100644 --- a/fs/smb/client/dir.c +++ b/fs/smb/client/dir.c @@ -176,7 +176,8 @@ check_name(struct dentry *direntry, struct cifs_tcon *tcon) static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned int xid, struct tcon_link *tlink, unsigned int oflags, umode_t mode, __u32 *oplock, - struct cifs_fid *fid, struct cifs_open_info_data *buf) + struct cifs_fid *fid, struct cifs_open_info_data *buf, + bool *opened) { int rc = -ENOENT; int create_options = CREATE_NOT_DIR; @@ -191,6 +192,7 @@ static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned struct cifs_open_parms oparms; int rdwr_for_fscache = 0; + *opened = false; *oplock = 0; if (tcon->ses->server->oplocks) *oplock = REQ_OPLOCK; @@ -213,6 +215,7 @@ static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned oflags, oplock, &fid->netfid, xid); switch (rc) { case 0: + *opened = true; if (newinode == NULL) { /* query inode info */ goto cifs_create_get_file_info; @@ -229,11 +232,9 @@ static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned /* * The server may allow us to open things like * FIFOs, but the client isn't set up to deal - * with that. If it's not a regular file, just - * close it and proceed as if it were a normal - * lookup. + * with that. Keep the handle until the caller + * can finish the lookup. */ - CIFSSMBClose(xid, tcon, fid->netfid); goto cifs_create_get_file_info; } /* success, no need to query */ @@ -333,6 +334,7 @@ static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned } goto out; } + *opened = true; if (rdwr_for_fscache == 2) cifs_invalidate_cache(inode, FSCACHE_INVAL_DIO_WRITE); @@ -423,7 +425,7 @@ static int cifs_do_create(struct inode *inode, struct dentry *direntry, unsigned return rc; out_err: - if (server->ops->close) + if (*opened && server->ops->close) server->ops->close(xid, tcon, fid); if (newinode) iput(newinode); @@ -442,6 +444,8 @@ cifs_atomic_open(struct inode *inode, struct dentry *direntry, struct cifs_fid fid = {}; struct cifs_pending_open open; __u32 oplock; + bool is_regular; + bool opened; struct cifsFileInfo *file_info; struct cifs_open_info_data buf = {}; @@ -501,15 +505,28 @@ cifs_atomic_open(struct inode *inode, struct dentry *direntry, cifs_add_pending_open(&fid, tlink, &open); rc = cifs_do_create(inode, direntry, xid, tlink, oflags, mode, - &oplock, &fid, &buf); + &oplock, &fid, &buf, &opened); if (rc) { cifs_del_pending_open(&open); goto out; } - if ((oflags & (O_CREAT | O_EXCL)) == (O_CREAT | O_EXCL)) + is_regular = d_is_reg(direntry); + if (!is_regular || !opened) { + if (opened && server->ops->close) + server->ops->close(xid, tcon, &fid); + cifs_del_pending_open(&open); + } + + if (is_regular && opened && + (oflags & (O_CREAT | O_EXCL)) == (O_CREAT | O_EXCL)) file->f_mode |= FMODE_CREATED; + if (!is_regular || !opened) { + rc = finish_no_open(file, NULL); + goto out; + } + rc = finish_open(file, direntry, generic_file_open); if (rc) { if (server->ops->close) @@ -564,6 +581,7 @@ int cifs_create(struct mnt_idmap *idmap, struct inode *inode, struct TCP_Server_Info *server; struct cifs_fid fid; __u32 oplock; + bool opened; struct cifs_open_info_data buf = {}; cifs_dbg(FYI, "cifs_create parent inode = 0x%p name is: %pd and dentry = 0x%p\n", @@ -585,8 +603,9 @@ int cifs_create(struct mnt_idmap *idmap, struct inode *inode, if (server->ops->new_lease_key) server->ops->new_lease_key(&fid); - rc = cifs_do_create(inode, direntry, xid, tlink, oflags, mode, &oplock, &fid, &buf); - if (!rc && server->ops->close) + rc = cifs_do_create(inode, direntry, xid, tlink, oflags, mode, + &oplock, &fid, &buf, &opened); + if (!rc && opened && server->ops->close) server->ops->close(xid, tcon, &fid); cifs_free_open_info(&buf); diff --git a/fs/smb/client/file.c b/fs/smb/client/file.c index a6be7903a67c..d329b8a3b756 100644 --- a/fs/smb/client/file.c +++ b/fs/smb/client/file.c @@ -1207,11 +1207,21 @@ void smb2_deferred_work_close(struct work_struct *work) { struct cifsFileInfo *cfile = container_of(work, struct cifsFileInfo, deferred.work); + struct cifsInodeInfo *cinode = CIFS_I(d_inode(cfile->dentry)); - spin_lock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock); + spin_lock(&cinode->deferred_lock); cifs_del_deferred_close(cfile); cfile->deferred_close_scheduled = false; - spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock); + spin_unlock(&cinode->deferred_lock); + /* + * Refresh time_last_write immediately before the actual server close + * so the protection window is anchored to the real close time, not + * the earlier userspace close time stored by cifs_close(). + */ + if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) { + /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */ + smp_store_release(&cinode->time_last_write, jiffies); + } _cifsFileInfo_put(cfile, true, false); } @@ -1240,6 +1250,10 @@ int cifs_close(struct inode *inode, struct file *file) if (file->private_data != NULL) { cfile = file->private_data; file->private_data = NULL; + if (file->f_mode & FMODE_WRITE) { + /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */ + smp_store_release(&cinode->time_last_write, jiffies); + } dclose = kmalloc(sizeof(struct cifs_deferred_close), GFP_KERNEL); if ((cfile->status_file_deleted == false) && (smb2_can_defer_close(inode, dclose))) { @@ -1260,11 +1274,18 @@ int cifs_close(struct inode *inode, struct file *file) cifsFileInfo_get(cfile); } else { /* Deferred close for files */ - queue_delayed_work(deferredclose_wq, - &cfile->deferred, cifs_sb->ctx->closetimeo); - cfile->deferred_close_scheduled = true; - spin_unlock(&cinode->deferred_lock); - return 0; + /* + * Each queued execution owns one reference. + * If nothing was queued, the reference of + * the closing file is dropped below. + */ + if (queue_delayed_work(deferredclose_wq, + &cfile->deferred, + cifs_sb->ctx->closetimeo)) { + cfile->deferred_close_scheduled = true; + spin_unlock(&cinode->deferred_lock); + return 0; + } } spin_unlock(&cinode->deferred_lock); _cifsFileInfo_put(cfile, true, false); @@ -4994,15 +5015,27 @@ static int is_inode_writable(struct cifsInodeInfo *cifs_inode) bool is_size_safe_to_change(struct cifsInodeInfo *cifsInode, __u64 end_of_file, bool from_readdir) { + struct cifs_sb_info *cifs_sb; + unsigned long tlw; + if (!cifsInode) return true; + cifs_sb = CIFS_SB(cifsInode->netfs.inode.i_sb); + if (is_inode_writable(cifsInode) || ((cifsInode->oplock & CIFS_CACHE_RW_FLG) != 0 && from_readdir)) { /* This inode is open for write at least once */ - struct cifs_sb_info *cifs_sb; - cifs_sb = CIFS_SB(cifsInode->netfs.inode.i_sb); + /* + * Readdir data is unreliable when we have writable handles or + * an exclusive lease -- never allow it to change i_size, even + * on direct-IO mounts where the server's directory metadata + * can still lag behind the actual file state. + */ + if (from_readdir) + return false; + if (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_DIRECT_IO) { /* since no page cache to corrupt on directio we can change size safely */ @@ -5013,8 +5046,40 @@ bool is_size_safe_to_change(struct cifsInodeInfo *cifsInode, __u64 end_of_file, return true; return false; - } else - return true; + } + + /* + * No writable handles open. Check whether we are within the attribute + * cache validity window of a recent local modification. + * + * For the close() path: cifs_close() calls smp_store_release() on + * time_last_write before _cifsFileInfo_put() removes the handle under + * open_file_lock. That spin_unlock() is a store-release that pairs + * with the spin_lock() (load-acquire) in is_inode_writable() above, + * so if is_inode_writable() returned false the smp_load_acquire() + * below is guaranteed to observe any time_last_write update from a + * concurrent close(). + * + * For the setattr/truncate paths: those callers use smp_store_release() + * directly; the smp_load_acquire() below pairs with that store. There + * is no shared lock between setattr and readdir, so this relies on + * acquire-release semantics alone. The store propagation latency on + * weakly-ordered architectures (nanoseconds) is negligible relative to + * the acregmax window (seconds) and the readdir RPC round-trip + * (milliseconds), making this a sound design choice in practice. + * + * time_last_write == 0 means the inode has never been written locally; + * skip the window check to avoid false positives near boot time when + * jiffies is still close to INITIAL_JIFFIES on 32-bit systems. + */ + if (from_readdir) { + /* Pairs with smp_store_release() at close and truncate sites. */ + tlw = smp_load_acquire(&cifsInode->time_last_write); + if (tlw && time_before(jiffies, tlw + cifs_sb->ctx->acregmax)) + return false; + } + + return true; } static int cifs_write_begin(struct file *file, struct address_space *mapping, @@ -5139,8 +5204,6 @@ void cifs_oplock_break(struct work_struct *work) struct cifsFileInfo *cfile = container_of(work, struct cifsFileInfo, oplock_break); struct inode *inode = d_inode(cfile->dentry); - struct super_block *sb = inode->i_sb; - struct cifs_sb_info *cifs_sb = CIFS_SB(sb); struct cifsInodeInfo *cinode = CIFS_I(inode); struct cifs_tcon *tcon; struct TCP_Server_Info *server; @@ -5153,8 +5216,8 @@ void cifs_oplock_break(struct work_struct *work) wait_on_bit(&cinode->flags, CIFS_INODE_PENDING_WRITERS, TASK_UNINTERRUPTIBLE); - tlink = cifs_sb_tlink(cifs_sb); - if (IS_ERR(tlink)) { + tlink = cifs_get_tlink(cfile->tlink); + if (IS_ERR_OR_NULL(tlink)) { /* drop the reference taken when the break was queued */ _cifsFileInfo_put(cfile, false /* do not wait for ourself */, false); goto out; diff --git a/fs/smb/client/inode.c b/fs/smb/client/inode.c index cca480ab7e64..8d201db93028 100644 --- a/fs/smb/client/inode.c +++ b/fs/smb/client/inode.c @@ -227,6 +227,8 @@ cifs_fattr_to_inode(struct inode *inode, struct cifs_fattr *fattr, * calculating num blocks. */ inode->i_blocks = (512 - 1 + fattr->cf_bytes) >> 9; + } else if (from_readdir && i_size_read(inode) != fattr->cf_eof) { + cifs_i->time = 0; } if (S_ISLNK(fattr->cf_mode) && fattr->cf_symlink_target) { @@ -770,6 +772,7 @@ static void smb311_posix_info_to_fattr(struct cifs_fattr *fattr, struct smb311_posix_qinfo *info = &data->posix_fi; struct cifs_sb_info *cifs_sb = CIFS_SB(sb); struct cifs_tcon *tcon = cifs_sb_master_tcon(cifs_sb); + unsigned int sbflags = cifs_sb->mnt_cifs_flags; memset(fattr, 0, sizeof(*fattr)); @@ -814,8 +817,12 @@ static void smb311_posix_info_to_fattr(struct cifs_fattr *fattr, fattr->cf_symlink_target = data->symlink_target; data->symlink_target = NULL; } - sid_to_id(cifs_sb, &data->posix_owner, fattr, SIDOWNER); - sid_to_id(cifs_sb, &data->posix_group, fattr, SIDGROUP); + fattr->cf_uid = cifs_sb->ctx->linux_uid; + fattr->cf_gid = cifs_sb->ctx->linux_gid; + if (!(sbflags & CIFS_MOUNT_OVERR_UID)) + sid_to_id(cifs_sb, &data->posix_owner, fattr, SIDOWNER); + if (!(sbflags & CIFS_MOUNT_OVERR_GID)) + sid_to_id(cifs_sb, &data->posix_group, fattr, SIDGROUP); cifs_dbg(FYI, "POSIX query info: mode 0x%x uniqueid 0x%llx nlink %d\n", fattr->cf_mode, fattr->cf_uniqueid, fattr->cf_nlink); @@ -2906,13 +2913,17 @@ int cifs_truncate_page(struct address_space *mapping, loff_t from) void cifs_setsize(struct inode *inode, loff_t offset) { struct cifsInodeInfo *cifs_i = CIFS_I(inode); + loff_t old_size; spin_lock(&inode->i_lock); + old_size = i_size_read(inode); i_size_write(inode, offset); spin_unlock(&inode->i_lock); /* Cached inode must be refreshed on truncate */ cifs_i->time = 0; + if (offset > old_size) + pagecache_isize_extended(inode, old_size, offset); truncate_pagecache(inode, offset); } @@ -3064,9 +3075,13 @@ cifs_setattr_unix(struct dentry *direntry, struct iattr *attrs) goto out; } - /* skip mode change if it's just for clearing setuid/setgid */ - if (attrs->ia_valid & (ATTR_KILL_SUID|ATTR_KILL_SGID)) - attrs->ia_valid &= ~ATTR_MODE; + /* + * This function is only called when Unix extensions are in effect, + * so the mode is always sent to and stored on the server. Do not + * skip the mode change when clearing setuid/setgid bits: dropping + * ATTR_MODE here would leave those bits set on the server after a + * write, which is a security issue. + */ args = kmalloc(sizeof(*args), GFP_KERNEL); if (args == NULL) { @@ -3131,6 +3146,8 @@ cifs_setattr_unix(struct dentry *direntry, struct iattr *attrs) if ((attrs->ia_valid & ATTR_SIZE) && attrs->ia_size != i_size_read(inode)) { + /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */ + smp_store_release(&cifsInode->time_last_write, jiffies); truncate_setsize(inode, attrs->ia_size); fscache_resize_cookie(cifs_inode_cookie(inode), attrs->ia_size); } @@ -3238,7 +3255,8 @@ cifs_setattr_nounix(struct dentry *direntry, struct iattr *attrs) gid = attrs->ia_gid; if ((cifs_sb->mnt_cifs_flags & CIFS_MOUNT_CIFS_ACL) || - (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_MODE_FROM_SID)) { + (cifs_sb->mnt_cifs_flags & CIFS_MOUNT_MODE_FROM_SID) || + cifs_sb_master_tcon(cifs_sb)->posix_extensions) { if (uid_valid(uid) || gid_valid(gid)) { mode = NO_CHANGE_64; rc = id_mode_to_cifs_acl(inode, full_path, &mode, @@ -3253,8 +3271,24 @@ cifs_setattr_nounix(struct dentry *direntry, struct iattr *attrs) if (!(cifs_sb->mnt_cifs_flags & CIFS_MOUNT_SET_UID)) attrs->ia_valid &= ~(ATTR_UID | ATTR_GID); - /* skip mode change if it's just for clearing setuid/setgid */ - if (attrs->ia_valid & (ATTR_KILL_SUID|ATTR_KILL_SGID)) + /* + * Skip the mode change if it is only being done to clear the + * setuid/setgid bits *and* the mode is emulated via the DOS + * read-only attribute (the default, non-ACL case), which cannot + * represent the setuid/setgid bits anyway. + * + * When the mode is instead stored on the server - i.e. with the + * cifsacl or modefromsid mount options (via an ACL) or with the + * SMB3.1.1 POSIX extensions - the cleared mode must be pushed to + * the server. Dropping ATTR_MODE here would leave the setuid/ + * setgid bit set on the server after a write, which is a security + * issue (the bits are not stripped as they are on local + * filesystems). + */ + if ((attrs->ia_valid & (ATTR_KILL_SUID|ATTR_KILL_SGID)) && + !((cifs_sb->mnt_cifs_flags & + (CIFS_MOUNT_CIFS_ACL | CIFS_MOUNT_MODE_FROM_SID)) || + posix)) attrs->ia_valid &= ~ATTR_MODE; if (attrs->ia_valid & ATTR_MODE) { @@ -3332,6 +3366,8 @@ cifs_setattr_nounix(struct dentry *direntry, struct iattr *attrs) if ((attrs->ia_valid & ATTR_SIZE) && attrs->ia_size != i_size_read(inode)) { + /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */ + smp_store_release(&cifsInode->time_last_write, jiffies); truncate_setsize(inode, attrs->ia_size); fscache_resize_cookie(cifs_inode_cookie(inode), attrs->ia_size); } diff --git a/fs/smb/client/link.c b/fs/smb/client/link.c index 007da0a699cb..a8664328701e 100644 --- a/fs/smb/client/link.c +++ b/fs/smb/client/link.c @@ -257,7 +257,7 @@ cifs_query_mf_symlink(unsigned int xid, struct cifs_tcon *tcon, struct cifs_open_parms oparms; struct cifs_io_parms io_parms = {0}; int buf_type = CIFS_NO_BUFFER; - struct cifs_open_info_data query_data; + struct cifs_open_info_data query_data = {}; oparms = (struct cifs_open_parms) { .tcon = tcon, @@ -343,7 +343,7 @@ smb3_query_mf_symlink(unsigned int xid, struct cifs_tcon *tcon, int buf_type = CIFS_NO_BUFFER; __le16 *utf16_path; __u8 oplock = SMB2_OPLOCK_LEVEL_NONE; - struct smb2_file_all_info *pfile_info = NULL; + struct cifs_open_info_data data = {}; oparms = (struct cifs_open_parms) { .tcon = tcon, @@ -359,20 +359,12 @@ smb3_query_mf_symlink(unsigned int xid, struct cifs_tcon *tcon, if (utf16_path == NULL) return -ENOMEM; - pfile_info = kzalloc(sizeof(struct smb2_file_all_info) + PATH_MAX * 2, - GFP_KERNEL); - - if (pfile_info == NULL) { - kfree(utf16_path); - return -ENOMEM; - } - - rc = SMB2_open(xid, &oparms, utf16_path, &oplock, pfile_info, NULL, + rc = SMB2_open(xid, &oparms, utf16_path, &oplock, &data, NULL, NULL, NULL); if (rc) goto qmf_out_open_fail; - if (pfile_info->EndOfFile != cpu_to_le64(CIFS_MF_SYMLINK_FILE_SIZE)) { + if (data.fi.EndOfFile != cpu_to_le64(CIFS_MF_SYMLINK_FILE_SIZE)) { /* it's not a symlink */ rc = -ENOENT; /* Is there a better rc to return? */ goto qmf_out; @@ -390,7 +382,6 @@ smb3_query_mf_symlink(unsigned int xid, struct cifs_tcon *tcon, SMB2_close(xid, tcon, fid.persistent_fid, fid.volatile_fid); qmf_out_open_fail: kfree(utf16_path); - kfree(pfile_info); return rc; } diff --git a/fs/smb/client/misc.c b/fs/smb/client/misc.c index 6f426986eec6..8937095184ca 100644 --- a/fs/smb/client/misc.c +++ b/fs/smb/client/misc.c @@ -772,7 +772,7 @@ cifs_del_deferred_close(struct cifsFileInfo *cfile) void cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode) { - struct cifsFileInfo *cfile = NULL; + struct cifsFileInfo *cfile = NULL, *failed_cfile = NULL; struct file_list *tmp_list, *tmp_next_list; struct list_head file_head; @@ -789,8 +789,10 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode) spin_unlock(&cifs_inode->deferred_lock); tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC); - if (tmp_list == NULL) + if (tmp_list == NULL) { + failed_cfile = cfile; break; + } tmp_list->cfile = cfile; list_add_tail(&tmp_list->list, &file_head); } @@ -798,8 +800,24 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode) } spin_unlock(&cifs_inode->open_file_lock); + if (failed_cfile) { + if (OPEN_FMODE(failed_cfile->f_flags) & FMODE_WRITE) { + /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */ + smp_store_release(&CIFS_I(d_inode(failed_cfile->dentry))->time_last_write, + jiffies); + } + _cifsFileInfo_put(failed_cfile, false, false); + } + list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) { - _cifsFileInfo_put(tmp_list->cfile, false, false); + struct cifsFileInfo *cfile = tmp_list->cfile; + + if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) { + /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */ + smp_store_release(&CIFS_I(d_inode(cfile->dentry))->time_last_write, + jiffies); + } + _cifsFileInfo_put(cfile, false, false); list_del(&tmp_list->list); kfree(tmp_list); } @@ -808,7 +826,7 @@ cifs_close_deferred_file(struct cifsInodeInfo *cifs_inode) void cifs_close_all_deferred_files(struct cifs_tcon *tcon) { - struct cifsFileInfo *cfile; + struct cifsFileInfo *cfile, *failed_cfile = NULL; struct file_list *tmp_list, *tmp_next_list; struct list_head file_head; @@ -822,8 +840,10 @@ cifs_close_all_deferred_files(struct cifs_tcon *tcon) spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock); tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC); - if (tmp_list == NULL) + if (tmp_list == NULL) { + failed_cfile = cfile; break; + } tmp_list->cfile = cfile; list_add_tail(&tmp_list->list, &file_head); } @@ -831,8 +851,24 @@ cifs_close_all_deferred_files(struct cifs_tcon *tcon) } spin_unlock(&tcon->open_file_lock); + if (failed_cfile) { + if (OPEN_FMODE(failed_cfile->f_flags) & FMODE_WRITE) { + /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */ + smp_store_release(&CIFS_I(d_inode(failed_cfile->dentry))->time_last_write, + jiffies); + } + _cifsFileInfo_put(failed_cfile, true, false); + } + list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) { - _cifsFileInfo_put(tmp_list->cfile, true, false); + struct cifsFileInfo *cfile = tmp_list->cfile; + + if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) { + /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */ + smp_store_release(&CIFS_I(d_inode(cfile->dentry))->time_last_write, + jiffies); + } + _cifsFileInfo_put(cfile, true, false); list_del(&tmp_list->list); kfree(tmp_list); } @@ -878,7 +914,7 @@ void cifs_close_all_deferred_files_sb(struct cifs_sb_info *cifs_sb) void cifs_close_deferred_file_under_dentry(struct cifs_tcon *tcon, const char *path) { - struct cifsFileInfo *cfile; + struct cifsFileInfo *cfile, *failed_cfile = NULL; struct file_list *tmp_list, *tmp_next_list; struct list_head file_head; void *page; @@ -897,8 +933,10 @@ cifs_close_deferred_file_under_dentry(struct cifs_tcon *tcon, const char *path) spin_unlock(&CIFS_I(d_inode(cfile->dentry))->deferred_lock); tmp_list = kmalloc(sizeof(struct file_list), GFP_ATOMIC); - if (tmp_list == NULL) + if (tmp_list == NULL) { + failed_cfile = cfile; break; + } tmp_list->cfile = cfile; list_add_tail(&tmp_list->list, &file_head); } @@ -907,8 +945,24 @@ cifs_close_deferred_file_under_dentry(struct cifs_tcon *tcon, const char *path) } spin_unlock(&tcon->open_file_lock); + if (failed_cfile) { + if (OPEN_FMODE(failed_cfile->f_flags) & FMODE_WRITE) { + /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */ + smp_store_release(&CIFS_I(d_inode(failed_cfile->dentry))->time_last_write, + jiffies); + } + _cifsFileInfo_put(failed_cfile, true, false); + } + list_for_each_entry_safe(tmp_list, tmp_next_list, &file_head, list) { - _cifsFileInfo_put(tmp_list->cfile, true, false); + struct cifsFileInfo *cfile = tmp_list->cfile; + + if (OPEN_FMODE(cfile->f_flags) & FMODE_WRITE) { + /* Pairs with smp_load_acquire() in is_size_safe_to_change(). */ + smp_store_release(&CIFS_I(d_inode(cfile->dentry))->time_last_write, + jiffies); + } + _cifsFileInfo_put(cfile, true, false); list_del(&tmp_list->list); kfree(tmp_list); } diff --git a/fs/smb/client/readdir.c b/fs/smb/client/readdir.c index d6f3f9a3c573..7b7942f0f53e 100644 --- a/fs/smb/client/readdir.c +++ b/fs/smb/client/readdir.c @@ -241,9 +241,11 @@ static void cifs_posix_to_fattr(struct cifs_fattr *fattr, struct smb2_posix_info *info, struct cifs_sb_info *cifs_sb) { + unsigned int sbflags = cifs_sb->mnt_cifs_flags; struct smb2_posix_info_parsed parsed; + int rc; - posix_info_parse(info, NULL, &parsed); + rc = posix_info_parse(info, NULL, &parsed); memset(fattr, 0, sizeof(*fattr)); fattr->cf_uniqueid = le64_to_cpu(info->Inode); @@ -280,8 +282,17 @@ cifs_posix_to_fattr(struct cifs_fattr *fattr, struct smb2_posix_info *info, le32_to_cpu(info->ReparseTag), le32_to_cpu(info->Mode)); - sid_to_id(cifs_sb, &parsed.owner, fattr, SIDOWNER); - sid_to_id(cifs_sb, &parsed.group, fattr, SIDGROUP); + fattr->cf_uid = cifs_sb->ctx->linux_uid; + fattr->cf_gid = cifs_sb->ctx->linux_gid; + if (rc < 0) { + cifs_dbg(VFS, "%s: failed to parse SIDs: %d\n", + __func__, rc); + return; + } + if (!(sbflags & CIFS_MOUNT_OVERR_UID)) + sid_to_id(cifs_sb, &parsed.owner, fattr, SIDOWNER); + if (!(sbflags & CIFS_MOUNT_OVERR_GID)) + sid_to_id(cifs_sb, &parsed.group, fattr, SIDGROUP); } static void __dir_info_to_fattr(struct cifs_fattr *fattr, const void *info) diff --git a/fs/smb/client/sess.c b/fs/smb/client/sess.c index 6fac7a107592..35d94589013b 100644 --- a/fs/smb/client/sess.c +++ b/fs/smb/client/sess.c @@ -187,9 +187,9 @@ int cifs_try_adding_channels(struct cifs_ses *ses) int old_chan_count, new_chan_count; int left; int rc = 0; - int tries = 0; + int tries = 0, attempts; size_t iface_weight = 0, iface_min_speed = 0; - struct cifs_server_iface *iface = NULL, *niface = NULL; + struct cifs_server_iface *iface = NULL, *candidate = NULL; struct cifs_server_iface *last_iface = NULL; spin_lock(&ses->chan_lock); @@ -235,67 +235,89 @@ int cifs_try_adding_channels(struct cifs_ses *ses) break; } - if (!iface) - iface = list_first_entry(&ses->iface_list, struct cifs_server_iface, - iface_head); last_iface = list_last_entry(&ses->iface_list, struct cifs_server_iface, iface_head); iface_min_speed = last_iface->speed; + spin_unlock(&ses->iface_lock); - list_for_each_entry_safe_from(iface, niface, &ses->iface_list, - iface_head) { - /* do not mix rdma and non-rdma interfaces */ - if (iface->rdma_capable != ses->server->rdma) - continue; - - /* skip ifaces that are unusable */ - if (!iface->is_active || - (is_ses_using_iface(ses, iface) && - !iface->rss_capable)) - continue; + attempts = 0; + while (left > 0) { + spin_lock(&ses->iface_lock); - /* check if we already allocated enough channels */ - iface_weight = iface->speed / iface_min_speed; + /* + * iface_lock must be dropped while opening a channel, + * and a concurrent interface refresh may remove and + * free entries during that window, so no list entry + * may be kept across it without a reference. Scan + * the list from the beginning each time and only pass + * a referenced candidate to cifs_ses_add_channel(); + * weight_fulfilled tracks the progress so that no + * iface is selected beyond its weight. + */ + candidate = NULL; + list_for_each_entry(iface, &ses->iface_list, iface_head) { + /* do not mix rdma and non-rdma interfaces */ + if (iface->rdma_capable != ses->server->rdma) + continue; + + /* skip ifaces that are unusable */ + if (!iface->is_active || + (is_ses_using_iface(ses, iface) && + !iface->rss_capable)) + continue; + + /* check if we already allocated enough channels */ + iface_weight = iface->speed / iface_min_speed; + + if (iface->weight_fulfilled >= iface_weight) + continue; + + /* take ref before unlock */ + kref_get(&iface->refcount); + candidate = iface; + break; + } - if (iface->weight_fulfilled >= iface_weight) - continue; + if (!candidate) { + /* no usable iface. reset weight_fulfilled and start over */ + list_for_each_entry(iface, &ses->iface_list, iface_head) + iface->weight_fulfilled = 0; + spin_unlock(&ses->iface_lock); + break; + } - /* take ref before unlock */ - kref_get(&iface->refcount); + attempts++; + if (attempts > 3 * ses->chan_max) { + kref_put(&candidate->refcount, release_iface); + spin_unlock(&ses->iface_lock); + break; + } spin_unlock(&ses->iface_lock); - rc = cifs_ses_add_channel(ses, iface); + rc = cifs_ses_add_channel(ses, candidate); spin_lock(&ses->iface_lock); if (rc) { cifs_dbg(VFS, "failed to open extra channel on iface:%pIS rc=%d\n", - &iface->sockaddr, + &candidate->sockaddr, rc); /* failure to add chan should increase weight */ - iface->weight_fulfilled++; - kref_put(&iface->refcount, release_iface); + candidate->weight_fulfilled++; + kref_put(&candidate->refcount, release_iface); + spin_unlock(&ses->iface_lock); continue; } - iface->num_channels++; - iface->weight_fulfilled++; - cifs_dbg(VFS, "successfully opened new channel on iface:%pIS\n", - &iface->sockaddr); - break; - } - - /* reached end of list. reset weight_fulfilled and start over */ - if (list_entry_is_head(iface, &ses->iface_list, iface_head)) { - list_for_each_entry(iface, &ses->iface_list, iface_head) - iface->weight_fulfilled = 0; + candidate->num_channels++; + candidate->weight_fulfilled++; + cifs_info("successfully opened new channel on iface:%pIS\n", + &candidate->sockaddr); spin_unlock(&ses->iface_lock); - iface = NULL; - continue; - } - spin_unlock(&ses->iface_lock); - left--; - new_chan_count++; + left--; + new_chan_count++; + break; + } } return new_chan_count - old_chan_count; diff --git a/fs/smb/client/smb1ops.c b/fs/smb/client/smb1ops.c index 6acadfcbadc8..c43b75128e0c 100644 --- a/fs/smb/client/smb1ops.c +++ b/fs/smb/client/smb1ops.c @@ -325,12 +325,31 @@ coalesce_t2(char *second_buf, struct smb_hdr *target_hdr) data_area_of_tgt = (char *)&pSMBt->hdr.Protocol + get_unaligned_le16(&pSMBt->t2_rsp.DataOffset); - /* validate target area */ data_area_of_src = (char *)&pSMBs->hdr.Protocol + get_unaligned_le16(&pSMBs->t2_rsp.DataOffset); data_area_of_tgt += total_in_tgt; + /* + * DataOffset fields are server-supplied and not validated against + * buffer bounds; check both data pointers before mutating the + * target header. + */ + if (data_area_of_tgt < (char *)target_hdr + + sizeof(struct smb_t2_rsp) + sizeof(__le16) || + data_area_of_tgt + total_in_src > + (char *)target_hdr + CIFSMaxBufSize + MAX_CIFS_HDR_SIZE) { + cifs_dbg(VFS, "%s: target data area out of bounds\n", __func__); + return -EPROTO; + } + if (data_area_of_src < second_buf + + sizeof(struct smb_t2_rsp) + sizeof(__le16) || + data_area_of_src + total_in_src > + second_buf + smbCalcSize((struct smb_hdr *)second_buf)) { + cifs_dbg(VFS, "%s: secondary data area out of bounds\n", __func__); + return -EPROTO; + } + total_in_tgt += total_in_src; /* is the result too big for the field? */ if (total_in_tgt > USHRT_MAX) { @@ -971,7 +990,7 @@ smb_set_file_info(struct inode *inode, const char *full_path, struct cifs_open_parms oparms; struct cifsFileInfo *open_file; FILE_BASIC_INFO new_buf; - struct cifs_open_info_data query_data; + struct cifs_open_info_data query_data = {}; __le64 write_time = buf->LastWriteTime; struct cifsInodeInfo *cinode = CIFS_I(inode); struct cifs_sb_info *cifs_sb = CIFS_SB(inode->i_sb); diff --git a/fs/smb/client/smb2file.c b/fs/smb/client/smb2file.c index 134155669a59..4444eea01e01 100644 --- a/fs/smb/client/smb2file.c +++ b/fs/smb/client/smb2file.c @@ -111,8 +111,6 @@ int smb2_open_file(const unsigned int xid, struct cifs_open_parms *oparms, __u32 __le16 *smb2_path; __u8 smb2_oplock; struct cifs_open_info_data *data = buf; - struct smb2_file_all_info file_info = {}; - struct smb2_file_all_info *smb2_data = data ? &file_info : NULL; struct kvec err_iov = {}; int err_buftype = CIFS_NO_BUFFER; struct cifs_fid *fid = oparms->fid; @@ -129,14 +127,14 @@ int smb2_open_file(const unsigned int xid, struct cifs_open_parms *oparms, __u32 } smb2_oplock = SMB2_OPLOCK_LEVEL_BATCH; - rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, smb2_data, NULL, &err_iov, + rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, data, NULL, &err_iov, &err_buftype); if (rc == -EACCES && retry_without_read_attributes) { free_rsp_buf(err_buftype, err_iov.iov_base); memset(&err_iov, 0, sizeof(err_iov)); err_buftype = CIFS_NO_BUFFER; oparms->desired_access &= ~FILE_READ_ATTRIBUTES; - rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, smb2_data, NULL, &err_iov, + rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, data, NULL, &err_iov, &err_buftype); } if (rc && data) { @@ -149,9 +147,9 @@ int smb2_open_file(const unsigned int xid, struct cifs_open_parms *oparms, __u32 oparms->path, &data->symlink_target); if (!rc) { - memset(smb2_data, 0, sizeof(*smb2_data)); + memset(&data->fi, 0, sizeof(data->fi)); oparms->create_options |= OPEN_REPARSE_POINT; - rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, smb2_data, + rc = SMB2_open(xid, oparms, smb2_path, &smb2_oplock, data, NULL, NULL, NULL); oparms->create_options &= ~OPEN_REPARSE_POINT; } @@ -180,23 +178,22 @@ int smb2_open_file(const unsigned int xid, struct cifs_open_parms *oparms, __u32 rc = 0; } - if (smb2_data) { + if (data) { /* if open response does not have IndexNumber field - get it */ - if (smb2_data->IndexNumber == 0) { + if (data->fi.IndexNumber == 0) { rc = SMB2_get_srv_num(xid, oparms->tcon, fid->persistent_fid, fid->volatile_fid, - &smb2_data->IndexNumber); + &data->fi.IndexNumber); if (rc) { /* * let get_inode_info disable server inode * numbers */ - smb2_data->IndexNumber = 0; + data->fi.IndexNumber = 0; rc = 0; } } - memcpy(&data->fi, smb2_data, sizeof(data->fi)); } *oplock = smb2_oplock; diff --git a/fs/smb/client/smb2inode.c b/fs/smb/client/smb2inode.c index db5475fe0071..2c7640939be3 100644 --- a/fs/smb/client/smb2inode.c +++ b/fs/smb/client/smb2inode.c @@ -42,7 +42,7 @@ static struct reparse_data_buffer *reparse_buf_ptr(struct kvec *iov) { struct reparse_data_buffer *buf; struct smb2_ioctl_rsp *io = iov->iov_base; - u32 off, count, len; + u32 off, count, len, rdlen; count = le32_to_cpu(io->OutputCount); off = le32_to_cpu(io->OutputOffset); @@ -51,7 +51,11 @@ static struct reparse_data_buffer *reparse_buf_ptr(struct kvec *iov) buf = (struct reparse_data_buffer *)((u8 *)io + off); len = sizeof(*buf); - if (count < len || count < le16_to_cpu(buf->ReparseDataLength) + len) + if (count < len) + return ERR_PTR(-EIO); + + rdlen = le16_to_cpu(buf->ReparseDataLength); + if (count < rdlen + len) return ERR_PTR(-EIO); return buf; } diff --git a/fs/smb/client/smb2misc.c b/fs/smb/client/smb2misc.c index 6fa23cf7a529..e37fe2d3b409 100644 --- a/fs/smb/client/smb2misc.c +++ b/fs/smb/client/smb2misc.c @@ -818,7 +818,8 @@ smb2_cancelled_close_fid(struct work_struct *work) */ static int __smb2_handle_cancelled_cmd(struct cifs_tcon *tcon, __u16 cmd, __u64 mid, - __u64 persistent_fid, __u64 volatile_fid) + __u64 persistent_fid, __u64 volatile_fid, + bool account_remote_open) { struct close_cancelled_open *cancelled; @@ -832,6 +833,8 @@ __smb2_handle_cancelled_cmd(struct cifs_tcon *tcon, __u16 cmd, __u64 mid, cancelled->cmd = cmd; cancelled->mid = mid; INIT_WORK(&cancelled->work, smb2_cancelled_close_fid); + if (account_remote_open) + atomic_inc(&tcon->num_remote_opens); WARN_ON(queue_work(cifsiod_wq, &cancelled->work) == false); return 0; @@ -868,7 +871,7 @@ smb2_handle_cancelled_close(struct cifs_tcon *tcon, __u64 persistent_fid, spin_unlock(&tcon->tc_lock); rc = __smb2_handle_cancelled_cmd(tcon, SMB2_CLOSE_HE, 0, - persistent_fid, volatile_fid); + persistent_fid, volatile_fid, false); if (rc) cifs_put_tcon(tcon, netfs_trace_tcon_ref_put_cancelled_close); @@ -896,7 +899,7 @@ smb2_handle_cancelled_mid(struct mid_q_entry *mid, struct TCP_Server_Info *serve le16_to_cpu(hdr->Command), le64_to_cpu(hdr->MessageId), rsp->PersistentFileId, - rsp->VolatileFileId); + rsp->VolatileFileId, true); if (rc) cifs_put_tcon(tcon, netfs_trace_tcon_ref_put_cancelled_mid); diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c index bc1fdeee9846..4da027ee4b99 100644 --- a/fs/smb/client/smb2ops.c +++ b/fs/smb/client/smb2ops.c @@ -1014,8 +1014,9 @@ move_smb2_ea_to_cifs(char *dst, size_t dst_size, char *name, *value; size_t buf_size = dst_size; size_t name_len, value_len, user_name_len; + u32 next_off; - while (src_size > 0) { + while (src_size >= sizeof(*src)) { name_len = (size_t)src->ea_name_length; value_len = (size_t)le16_to_cpu(src->ea_value_length); @@ -1070,14 +1071,21 @@ move_smb2_ea_to_cifs(char *dst, size_t dst_size, if (!src->next_entry_offset) break; - if (src_size < le32_to_cpu(src->next_entry_offset)) { - /* stop before overrun buffer */ - rc = -ERANGE; - break; + next_off = le32_to_cpu(src->next_entry_offset); + if (next_off < sizeof(*src) || src_size < next_off) { + cifs_dbg(FYI, "EA next_entry_offset %u out of range [%zu, %zu]\n", + next_off, sizeof(*src), src_size); + rc = -EIO; + goto out; + } + src_size -= next_off; + src = (void *)((char *)src + next_off); + if (src_size > 0 && src_size < sizeof(*src)) { + cifs_dbg(FYI, "EA next_entry_offset %u left truncated entry (%zu bytes)\n", + next_off, src_size); + rc = -EIO; + goto out; } - src_size -= le32_to_cpu(src->next_entry_offset); - src = (void *)((char *)src + - le32_to_cpu(src->next_entry_offset)); } /* didn't find the named attribute */ @@ -3260,6 +3268,14 @@ static long smb3_zero_range(struct file *file, struct cifs_tcon *tcon, ses->Suid, offset, len); inode_lock(inode); + + new_size = offset + len; + if (!keep_size && i_size_read(inode) < new_size) { + rc = inode_newsize_ok(inode, new_size); + if (rc) + goto out; + } + filemap_invalidate_lock(inode->i_mapping); i_size = i_size_read(inode); @@ -3275,14 +3291,20 @@ static long smb3_zero_range(struct file *file, struct cifs_tcon *tcon, /* * We zero the range through ioctl, so we need remove the page caches * first, otherwise the data may be inconsistent with the server. + * + * Start at the old EOF when extending so the folio straddling it, which + * may hold data written past EOF through an mmap, is dropped too. */ - truncate_pagecache_range(inode, offset, offset + len - 1); + truncate_pagecache_range(inode, min(offset, i_size), offset + len - 1); /* if file not oplocked can't be sure whether asking to extend size */ rc = -EOPNOTSUPP; if (keep_size == false && !CIFS_CACHE_READ(cifsi)) goto zero_range_exit; + fscache_invalidate(cifs_inode_cookie(inode), NULL, + i_size_read(inode), 0); + rc = smb3_zero_data(file, tcon, offset, len, xid); if (rc < 0) goto zero_range_exit; @@ -3290,7 +3312,6 @@ static long smb3_zero_range(struct file *file, struct cifs_tcon *tcon, /* * do we also need to change the size of the file? */ - new_size = offset + len; if (keep_size == false && (unsigned long long)i_size_read(inode) < new_size) { rc = SMB2_set_eof(xid, tcon, cfile->fid.persistent_fid, cfile->fid.volatile_fid, cfile->pid, new_size); @@ -3302,6 +3323,7 @@ static long smb3_zero_range(struct file *file, struct cifs_tcon *tcon, zero_range_exit: filemap_invalidate_unlock(inode->i_mapping); + out: inode_unlock(inode); free_xid(xid); if (rc) @@ -3348,6 +3370,8 @@ static long smb3_punch_hole(struct file *file, struct cifs_tcon *tcon, * caches first, otherwise the data may be inconsistent with the server. */ truncate_pagecache_range(inode, offset, offset + len - 1); + fscache_invalidate(cifs_inode_cookie(inode), NULL, + i_size_read(inode), 0); cifs_dbg(FYI, "Offset %lld len %lld\n", offset, len); @@ -3528,28 +3552,46 @@ static long smb3_simple_falloc(struct file *file, struct cifs_tcon *tcon, struct cifsFileInfo *cfile = file->private_data; long rc = -EOPNOTSUPP; unsigned int xid; - loff_t new_eof; + loff_t old_eof, new_eof; xid = get_xid(); inode = d_inode(cfile->dentry); cifsi = CIFS_I(inode); + old_eof = i_size_read(inode); trace_smb3_falloc_enter(xid, cfile->fid.persistent_fid, tcon->tid, tcon->ses->Suid, off, len); - /* if file not oplocked can't be sure whether asking to extend size */ - if (!CIFS_CACHE_READ(cifsi)) - if (keep_size == false) { + + if (!keep_size && !CIFS_CACHE_READ(cifsi)) { + struct smb2_file_all_info file_inf = {}; + unsigned long long server_eof; + + rc = filemap_write_and_wait(inode->i_mapping); + if (rc) { trace_smb3_falloc_err(xid, cfile->fid.persistent_fid, tcon->tid, tcon->ses->Suid, off, len, rc); free_xid(xid); return rc; } + rc = SMB2_query_info(xid, tcon, cfile->fid.persistent_fid, + cfile->fid.volatile_fid, &file_inf); + if (rc) { + trace_smb3_falloc_err(xid, cfile->fid.persistent_fid, + tcon->tid, tcon->ses->Suid, off, len, rc); + free_xid(xid); + return rc; + } + server_eof = le64_to_cpu(file_inf.EndOfFile); + /* Only use the larger EOF to decide whether we're extending. */ + old_eof = max_t(loff_t, old_eof, server_eof); + } + /* * Extending the file */ - if ((keep_size == false) && i_size_read(inode) < off + len) { + if (!keep_size && old_eof < off + len) { rc = inode_newsize_ok(inode, off + len); if (rc) goto out; @@ -3597,7 +3639,7 @@ static long smb3_simple_falloc(struct file *file, struct cifs_tcon *tcon, } } - if ((keep_size == true) || (i_size_read(inode) >= off + len)) { + if (keep_size || old_eof >= off + len) { /* * At this point, we are trying to fallocate an internal * regions of a sparse file. Since smb2 does not have a @@ -3626,7 +3668,7 @@ static long smb3_simple_falloc(struct file *file, struct cifs_tcon *tcon, * ie potentially making a few extra pages at the beginning * or end of the file non-sparse via set_sparse is harmless. */ - if ((off > 8192) || (off + len + 8192 < i_size_read(inode))) { + if (off > 8192 || off + len + 8192 < old_eof) { rc = -EOPNOTSUPP; goto out; } @@ -3669,11 +3711,18 @@ static long smb3_collapse_range(struct file *file, struct cifs_tcon *tcon, } filemap_invalidate_lock(inode->i_mapping); - rc = filemap_write_and_wait_range(inode->i_mapping, off, old_eof - 1); + rc = filemap_write_and_wait_range(inode->i_mapping, + round_down(off, PAGE_SIZE), + old_eof - 1); if (rc < 0) goto out_2; - truncate_pagecache_range(inode, off, old_eof); + /* + * Invalidate cached folios from the page containing off to EOF before + * moving data on the server, so subsequent reads do not see stale data. + */ + truncate_pagecache_range(inode, round_down(off, PAGE_SIZE), -1); + fscache_invalidate(cifs_inode_cookie(inode), NULL, old_eof, 0); rc = smb2_copychunk_range(xid, cfile, cfile, off + len, old_eof - off - len, off); @@ -3706,7 +3755,8 @@ static long smb3_insert_range(struct file *file, struct cifs_tcon *tcon, unsigned int xid; struct cifsFileInfo *cfile = file->private_data; struct inode *inode = file_inode(file); - __u64 count, old_eof, new_eof; + u64 count; + loff_t old_eof, new_eof; xid = get_xid(); @@ -3718,8 +3768,15 @@ static long smb3_insert_range(struct file *file, struct cifs_tcon *tcon, goto out; } + if (check_add_overflow(old_eof, len, &new_eof)) { + rc = -EFBIG; + goto out; + } + rc = inode_newsize_ok(inode, new_eof); + if (rc) + goto out; + count = old_eof - off; - new_eof = old_eof + len; /* SET_ZERO_DATA creates a hole only in a sparse file. */ rc = smb2_set_sparse(xid, tcon, cfile, inode, true); @@ -3727,10 +3784,17 @@ static long smb3_insert_range(struct file *file, struct cifs_tcon *tcon, goto out; filemap_invalidate_lock(inode->i_mapping); - rc = filemap_write_and_wait_range(inode->i_mapping, off, new_eof - 1); + rc = filemap_write_and_wait_range(inode->i_mapping, + round_down(off, PAGE_SIZE), + old_eof - 1); if (rc < 0) goto out_2; - truncate_pagecache_range(inode, off, old_eof); + /* + * Invalidate cached folios from the page containing off to EOF before + * moving data on the server, so subsequent reads do not see stale data. + */ + truncate_pagecache_range(inode, round_down(off, PAGE_SIZE), -1); + fscache_invalidate(cifs_inode_cookie(inode), NULL, old_eof, 0); rc = SMB2_set_eof(xid, tcon, cfile->fid.persistent_fid, cfile->fid.volatile_fid, cfile->pid, new_eof); @@ -4565,9 +4629,9 @@ decrypt_raw_data(struct TCP_Server_Info *server, char *buf, return rc; } } else { - if (unlikely(!server->secmech.dec)) - return -EIO; - + rc = smb3_crypto_aead_allocate(server); + if (unlikely(rc)) + return rc; tfm = server->secmech.dec; } diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c index 8c593d54fbe3..13c877a9bc97 100644 --- a/fs/smb/client/smb2pdu.c +++ b/fs/smb/client/smb2pdu.c @@ -1280,11 +1280,8 @@ SMB2_negotiate(const unsigned int xid, cifs_server_dbg(VFS, "Missing expected negotiate contexts\n"); } - if (server->cipher_type && !rc) { + if (server->cipher_type && !rc) rc = smb3_crypto_aead_allocate(server); - if (rc) - cifs_server_dbg(VFS, "%s: crypto alloc failed, rc=%d\n", __func__, rc); - } neg_exit: free_rsp_buf(resp_buftype, rsp); return rc; @@ -3218,7 +3215,7 @@ SMB2_open_free(struct smb_rqst *rqst) int SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path, - __u8 *oplock, struct smb2_file_all_info *buf, + __u8 *oplock, struct cifs_open_info_data *buf, struct create_posix_rsp *posix, struct kvec *err_iov, int *buftype) { @@ -3233,6 +3230,7 @@ SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path, int rc = 0; int flags = 0; int retries = 0, cur_sleep = 1; + struct smb2_file_all_info *file_info = buf ? &buf->fi : NULL; replay_again: /* reinitialize for possible replay */ @@ -3288,9 +3286,6 @@ SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path, goto creat_exit; } else if (rsp == NULL) /* unlikely to happen, but safer to check */ goto creat_exit; - else - trace_smb3_open_done(xid, rsp->PersistentFileId, tcon->tid, ses->Suid, - oparms->create_options, oparms->desired_access); atomic_inc(&tcon->num_remote_opens); oparms->fid->persistent_fid = rsp->PersistentFileId; @@ -3300,21 +3295,27 @@ SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path, oparms->fid->mid = le64_to_cpu(rsp->hdr.MessageId); #endif /* CIFS_DEBUG2 */ - if (buf) { - buf->CreationTime = rsp->CreationTime; - buf->LastAccessTime = rsp->LastAccessTime; - buf->LastWriteTime = rsp->LastWriteTime; - buf->ChangeTime = rsp->ChangeTime; - buf->AllocationSize = rsp->AllocationSize; - buf->EndOfFile = rsp->EndofFile; - buf->Attributes = rsp->FileAttributes; - buf->NumberOfLinks = cpu_to_le32(1); - buf->DeletePending = 0; + if (file_info) { + file_info->CreationTime = rsp->CreationTime; + file_info->LastAccessTime = rsp->LastAccessTime; + file_info->LastWriteTime = rsp->LastWriteTime; + file_info->ChangeTime = rsp->ChangeTime; + file_info->AllocationSize = rsp->AllocationSize; + file_info->EndOfFile = rsp->EndofFile; + file_info->Attributes = rsp->FileAttributes; + file_info->NumberOfLinks = cpu_to_le32(1); + file_info->DeletePending = 0; } rc = smb2_parse_contexts(server, &rsp_iov, &oparms->fid->epoch, - oparms->fid->lease_key, oplock, buf, posix); + oparms->fid->lease_key, oplock, file_info, posix); + if (rc) + SMB2_close(xid, tcon, oparms->fid->persistent_fid, + oparms->fid->volatile_fid); + + trace_smb3_open_done(xid, rsp->PersistentFileId, tcon->tid, ses->Suid, + oparms->create_options, oparms->desired_access); creat_exit: SMB2_open_free(&rqst); free_rsp_buf(resp_buftype, rsp); diff --git a/fs/smb/client/smb2proto.h b/fs/smb/client/smb2proto.h index f553e3edd54e..da0adcb6a06d 100644 --- a/fs/smb/client/smb2proto.h +++ b/fs/smb/client/smb2proto.h @@ -154,7 +154,7 @@ extern int SMB2_tcon(const unsigned int xid, struct cifs_ses *ses, extern int SMB2_tdis(const unsigned int xid, struct cifs_tcon *tcon); extern int SMB2_open(const unsigned int xid, struct cifs_open_parms *oparms, __le16 *path, __u8 *oplock, - struct smb2_file_all_info *buf, + struct cifs_open_info_data *buf, struct create_posix_rsp *posix, struct kvec *err_iov, int *resp_buftype); extern int SMB2_open_init(struct cifs_tcon *tcon, diff --git a/fs/smb/client/smbdirect.c b/fs/smb/client/smbdirect.c index 359ab64e17d9..5b99ea2f5728 100644 --- a/fs/smb/client/smbdirect.c +++ b/fs/smb/client/smbdirect.c @@ -473,7 +473,9 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc) struct smbd_response *response = container_of(wc->wr_cqe, struct smbd_response, cqe); struct smbd_connection *info = response->info; - int data_length = 0; + u32 data_offset = 0; + u32 data_length = 0; + u32 remaining_data_length = 0; log_rdma_recv(INFO, "response=0x%p type=%d wc status=%d wc opcode %d byte_len=%d pkey_index=%u\n", response, response->type, wc->status, wc->opcode, @@ -505,7 +507,22 @@ static void recv_done(struct ib_cq *cq, struct ib_wc *wc) /* SMBD data transfer packet */ case SMBD_TRANSFER_DATA: data_transfer = smbd_response_payload(response); + + if (wc->byte_len < + offsetof(struct smbd_data_transfer, padding)) + goto error; + + remaining_data_length = le32_to_cpu(data_transfer->remaining_data_length); + data_offset = le32_to_cpu(data_transfer->data_offset); data_length = le32_to_cpu(data_transfer->data_length); + if (wc->byte_len < data_offset || + (u64)wc->byte_len < (u64)data_offset + data_length) + goto error; + + if (remaining_data_length > info->max_fragmented_recv_size || + data_length > info->max_fragmented_recv_size || + (u64)remaining_data_length + (u64)data_length > (u64)info->max_fragmented_recv_size) + goto error; /* * If this is a packet with data playload place the data in diff --git a/fs/smb/client/transport.c b/fs/smb/client/transport.c index 8d28b594b417..facf4c6b180d 100644 --- a/fs/smb/client/transport.c +++ b/fs/smb/client/transport.c @@ -1080,6 +1080,7 @@ compound_send_recv(const unsigned int xid, struct cifs_ses *ses, int *resp_buf_type, struct kvec *resp_iov) { int i, j, optype, rc = 0; + int num_processed = 0; struct mid_q_entry *midQ[MAX_COMPOUND]; bool cancelled_mid[MAX_COMPOUND] = {false}; struct cifs_credits credits[MAX_COMPOUND] = { @@ -1222,6 +1223,17 @@ compound_send_recv(const unsigned int xid, struct cifs_ses *ses, break; } if (rc != 0) { + /* + * A completed CREATE earlier in the compound chain may have + * opened a remote handle even though a later wait was + * interrupted. Mark it cancelled so __release_mid() invokes + * the existing unmatched-open cleanup. + */ + spin_lock(&server->mid_lock); + for (j = 0; j < i; j++) + midQ[j]->mid_flags |= MID_WAIT_CANCELLED; + spin_unlock(&server->mid_lock); + for (; i < num_rqst; i++) { cifs_server_dbg(FYI, "Cancelling wait for mid %llu cmd: %d\n", midQ[i]->mid, le16_to_cpu(midQ[i]->command)); @@ -1244,6 +1256,17 @@ compound_send_recv(const unsigned int xid, struct cifs_ses *ses, rc = cifs_sync_mid_result(midQ[i], server); if (rc != 0) { + /* + * A previous CREATE may have completed before this + * response failed. Mark it cancelled so its remote + * handle is closed when the mid is released. + */ + spin_lock(&server->mid_lock); + for (j = 0; j < i; j++) + midQ[j]->mid_flags |= MID_WAIT_CANCELLED; + spin_unlock(&server->mid_lock); + /* Keep their response buffers for cancelled-mid cleanup. */ + num_processed = 0; /* mark this mid as cancelled to not free it below */ cancelled_mid[i] = true; goto out; @@ -1253,9 +1276,26 @@ compound_send_recv(const unsigned int xid, struct cifs_ses *ses, midQ[i]->mid_state != MID_RESPONSE_READY) { rc = -EIO; cifs_dbg(FYI, "Bad MID state?\n"); + spin_lock(&server->mid_lock); + for (j = 0; j < i; j++) + midQ[j]->mid_flags |= MID_WAIT_CANCELLED; + spin_unlock(&server->mid_lock); + num_processed = 0; goto out; } + rc = server->ops->check_receive(midQ[i], server, + flags & CIFS_LOG_ERROR); + num_processed = i + 1; + } + +out: + /* + * Delay moving response buffers out of their mids until response + * synchronization completes. This lets cancelled-mid cleanup inspect + * an earlier CREATE response if a later MID fails. + */ + for (i = 0; i < num_processed; i++) { buf = (char *)midQ[i]->resp_buf; resp_iov[i].iov_base = buf; resp_iov[i].iov_len = midQ[i]->resp_buf_size + @@ -1266,33 +1306,30 @@ compound_send_recv(const unsigned int xid, struct cifs_ses *ses, else resp_buf_type[i] = CIFS_SMALL_BUFFER; - rc = server->ops->check_receive(midQ[i], server, - flags & CIFS_LOG_ERROR); - /* mark it so buf will not be freed by delete_mid */ if ((flags & CIFS_NO_RSP_BUF) == 0) midQ[i]->resp_buf = NULL; - } /* * Compounding is never used during session establish. */ - spin_lock(&ses->ses_lock); - if ((ses->ses_status == SES_NEW) || (optype & CIFS_NEG_OP) || (optype & CIFS_SESS_OP)) { - struct kvec iov = { - .iov_base = resp_iov[0].iov_base, - .iov_len = resp_iov[0].iov_len - }; - spin_unlock(&ses->ses_lock); - cifs_server_lock(server); - smb311_update_preauth_hash(ses, server, &iov, 1); - cifs_server_unlock(server); + if (num_processed == num_rqst) { spin_lock(&ses->ses_lock); + if ((ses->ses_status == SES_NEW) || (optype & CIFS_NEG_OP) || (optype & CIFS_SESS_OP)) { + struct kvec iov = { + .iov_base = resp_iov[0].iov_base, + .iov_len = resp_iov[0].iov_len + }; + spin_unlock(&ses->ses_lock); + cifs_server_lock(server); + smb311_update_preauth_hash(ses, server, &iov, 1); + cifs_server_unlock(server); + spin_lock(&ses->ses_lock); + } + spin_unlock(&ses->ses_lock); } - spin_unlock(&ses->ses_lock); -out: /* * This will dequeue all mids. After this it is important that the * demultiplex_thread will not process any of these mids any futher. diff --git a/fs/smb/server/auth.c b/fs/smb/server/auth.c index e5b77d875478..c9892611b6e2 100644 --- a/fs/smb/server/auth.c +++ b/fs/smb/server/auth.c @@ -72,13 +72,12 @@ void ksmbd_copy_gss_neg_header(void *buf) /** * ksmbd_gen_sess_key() - function to generate session key - * @sess: session of connection + * @sess_key: output buffer for the generated session key * @hash: source hash value to be used for find session key * @hmac: source hmac value to be used for finding session key * */ -static int ksmbd_gen_sess_key(struct ksmbd_session *sess, char *hash, - char *hmac) +static int ksmbd_gen_sess_key(char *sess_key, char *hash, char *hmac) { struct ksmbd_crypto_ctx *ctx; int rc; @@ -111,7 +110,7 @@ static int ksmbd_gen_sess_key(struct ksmbd_session *sess, char *hash, goto out; } - rc = crypto_shash_final(CRYPTO_HMACMD5(ctx), sess->sess_key); + rc = crypto_shash_final(CRYPTO_HMACMD5(ctx), sess_key); if (rc) { ksmbd_debug(AUTH, "Could not generate hmacmd5 hash error %d\n", rc); goto out; @@ -224,6 +223,7 @@ int ksmbd_auth_ntlmv2(struct ksmbd_conn *conn, struct ksmbd_session *sess, { char ntlmv2_hash[CIFS_ENCPWD_SIZE]; char ntlmv2_rsp[CIFS_HMAC_MD5_HASH_SIZE]; + char sess_key[SMB2_NTLMV2_SESSKEY_SIZE]; struct ksmbd_crypto_ctx *ctx = NULL; char *construct = NULL; int rc, len; @@ -278,19 +278,26 @@ int ksmbd_auth_ntlmv2(struct ksmbd_conn *conn, struct ksmbd_session *sess, ksmbd_release_crypto_ctx(ctx); ctx = NULL; - rc = ksmbd_gen_sess_key(sess, ntlmv2_hash, ntlmv2_rsp); + rc = ksmbd_gen_sess_key(sess_key, ntlmv2_hash, ntlmv2_rsp); if (rc) { ksmbd_debug(AUTH, "Could not generate sess key\n"); goto out; } if (crypto_memneq(ntlmv2->ntlmv2_hash, ntlmv2_rsp, - CIFS_HMAC_MD5_HASH_SIZE)) + CIFS_HMAC_MD5_HASH_SIZE)) { rc = -EINVAL; + goto out; + } + + memcpy(sess->sess_key, sess_key, sizeof(sess_key)); out: if (ctx) ksmbd_release_crypto_ctx(ctx); kfree(construct); + memzero_explicit(ntlmv2_hash, sizeof(ntlmv2_hash)); + memzero_explicit(ntlmv2_rsp, sizeof(ntlmv2_rsp)); + memzero_explicit(sess_key, sizeof(sess_key)); return rc; } @@ -348,6 +355,8 @@ int ksmbd_decode_ntlmssp_auth_blob(struct authenticate_message *authblob, nt_len - CIFS_ENCPWD_SIZE, domain_name, conn->ntlmssp.cryptkey); kfree(domain_name); + if (ret) + return ret; /* The recovered secondary session key */ if (conn->ntlmssp.client_flags & NTLMSSP_NEGOTIATE_KEY_XCH) { diff --git a/fs/smb/server/connection.c b/fs/smb/server/connection.c index 8edae6eada3d..35971331a785 100644 --- a/fs/smb/server/connection.c +++ b/fs/smb/server/connection.c @@ -242,15 +242,37 @@ void ksmbd_conn_unlock(struct ksmbd_conn *conn) mutex_unlock(&conn->srv_mutex); } -void ksmbd_all_conn_set_status(u64 sess_id, u32 status) +static bool ksmbd_session_is_bound_to_conn(struct ksmbd_session *sess, + struct ksmbd_conn *conn) +{ + bool found; + + rcu_read_lock(); + found = xa_load(&conn->sessions, sess->id) == sess; + rcu_read_unlock(); + if (found) + return true; + + down_read(&sess->chann_lock); + found = xa_load(&sess->ksmbd_chann_list, (long)conn); + up_read(&sess->chann_lock); + return found; +} + +void ksmbd_all_conn_set_status(struct ksmbd_session *sess, u32 status) { struct ksmbd_conn *conn; int bkt; down_read(&conn_list_lock); hash_for_each(conn_list, bkt, conn, hlist) { - if (conn->binding || xa_load(&conn->sessions, sess_id)) - WRITE_ONCE(conn->status, status); + if (ksmbd_session_is_bound_to_conn(sess, conn)) { + spin_lock(&conn->request_lock); + if (!ksmbd_conn_exiting(conn) && + !ksmbd_conn_releasing(conn)) + WRITE_ONCE(conn->status, status); + spin_unlock(&conn->request_lock); + } } up_read(&conn_list_lock); } @@ -276,7 +298,8 @@ void ksmbd_conn_wait_idle(struct ksmbd_conn *conn) wait_event(conn->req_running_q, atomic_read(&conn->req_running) < 2); } -int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id) +int ksmbd_conn_wait_idle_sess(struct ksmbd_conn *curr_conn, + struct ksmbd_session *sess) { struct ksmbd_conn *conn; int rc, retry_count = 0, max_timeout = 120; @@ -288,7 +311,7 @@ int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id) down_read(&conn_list_lock); hash_for_each(conn_list, bkt, conn, hlist) { - if (conn->binding || xa_load(&conn->sessions, sess_id)) { + if (ksmbd_session_is_bound_to_conn(sess, conn)) { rcount = (conn == curr_conn) ? 2 : 1; if (atomic_read(&conn->req_running) >= rcount) { rc = wait_event_timeout(conn->req_running_q, @@ -589,7 +612,9 @@ static void stop_sessions(void) down_read(&conn_list_lock); hash_for_each(conn_list, bkt, conn, hlist) { t = conn->transport; + spin_lock(&conn->request_lock); ksmbd_conn_set_exiting(conn); + spin_unlock(&conn->request_lock); if (t->ops->shutdown) { up_read(&conn_list_lock); t->ops->shutdown(t); diff --git a/fs/smb/server/connection.h b/fs/smb/server/connection.h index 01b9020146a4..e7559ba2eabb 100644 --- a/fs/smb/server/connection.h +++ b/fs/smb/server/connection.h @@ -20,6 +20,8 @@ #include "smb_common.h" #include "ksmbd_work.h" +struct ksmbd_session; + #define KSMBD_SOCKET_BACKLOG 16 enum { @@ -160,7 +162,8 @@ extern struct rw_semaphore conn_list_lock; bool ksmbd_conn_alive(struct ksmbd_conn *conn); void ksmbd_conn_wait_idle(struct ksmbd_conn *conn); -int ksmbd_conn_wait_idle_sess_id(struct ksmbd_conn *curr_conn, u64 sess_id); +int ksmbd_conn_wait_idle_sess(struct ksmbd_conn *curr_conn, + struct ksmbd_session *sess); struct ksmbd_conn *ksmbd_conn_alloc(void); void ksmbd_conn_free(struct ksmbd_conn *conn); struct ksmbd_conn *ksmbd_conn_get(struct ksmbd_conn *conn); @@ -265,5 +268,5 @@ static inline void ksmbd_conn_set_releasing(struct ksmbd_conn *conn) WRITE_ONCE(conn->status, KSMBD_SESS_RELEASING); } -void ksmbd_all_conn_set_status(u64 sess_id, u32 status); +void ksmbd_all_conn_set_status(struct ksmbd_session *sess, u32 status); #endif /* __CONNECTION_H__ */ diff --git a/fs/smb/server/mgmt/share_config.c b/fs/smb/server/mgmt/share_config.c index 7db7345f1af9..cb89e54c4da8 100644 --- a/fs/smb/server/mgmt/share_config.c +++ b/fs/smb/server/mgmt/share_config.c @@ -161,6 +161,11 @@ static struct ksmbd_share_config *share_config_request(struct ksmbd_work *work, atomic_set(&share->refcount, 1); INIT_LIST_HEAD(&share->veto_list); share->name = kstrdup(name, GFP_KERNEL); + if (!share->name) { + kill_share(share); + share = NULL; + goto out; + } if (!test_share_config_flag(share, KSMBD_SHARE_FLAG_PIPE)) { size_t path_len; @@ -217,7 +222,7 @@ static struct ksmbd_share_config *share_config_request(struct ksmbd_work *work, share->path = NULL; } } - if (ret || !share->name) { + if (ret) { kill_share(share); share = NULL; goto out; diff --git a/fs/smb/server/mgmt/tree_connect.c b/fs/smb/server/mgmt/tree_connect.c index 9bde1b58f9c4..8f963602935c 100644 --- a/fs/smb/server/mgmt/tree_connect.c +++ b/fs/smb/server/mgmt/tree_connect.c @@ -76,7 +76,7 @@ ksmbd_tree_conn_connect(struct ksmbd_work *work, const char *share_name) tree_conn->share_conf = sc; tree_conn->t_state = TREE_NEW; status.tree_conn = tree_conn; - atomic_set(&tree_conn->refcount, 1); + atomic_set(&tree_conn->refcount, 2); ret = xa_err(xa_store(&sess->tree_conns, tree_conn->id, tree_conn, GFP_KERNEL)); @@ -108,6 +108,12 @@ int ksmbd_tree_conn_disconnect(struct ksmbd_session *sess, int ret; write_lock(&sess->tree_conns_lock); + if (tree_conn->t_state == TREE_DISCONNECTED || + xa_load(&sess->tree_conns, tree_conn->id) != tree_conn) { + write_unlock(&sess->tree_conns_lock); + return -ENOENT; + } + tree_conn->t_state = TREE_DISCONNECTED; xa_erase(&sess->tree_conns, tree_conn->id); write_unlock(&sess->tree_conns_lock); @@ -146,18 +152,8 @@ int ksmbd_tree_conn_session_logoff(struct ksmbd_session *sess) if (!sess) return -EINVAL; - xa_for_each(&sess->tree_conns, id, tc) { - write_lock(&sess->tree_conns_lock); - if (tc->t_state == TREE_DISCONNECTED) { - write_unlock(&sess->tree_conns_lock); - ret = -ENOENT; - continue; - } - tc->t_state = TREE_DISCONNECTED; - write_unlock(&sess->tree_conns_lock); - + xa_for_each(&sess->tree_conns, id, tc) ret |= ksmbd_tree_conn_disconnect(sess, tc); - } xa_destroy(&sess->tree_conns); return ret; } diff --git a/fs/smb/server/mgmt/user_config.c b/fs/smb/server/mgmt/user_config.c index 3d7c9b6b032e..6d51ff79ba0d 100644 --- a/fs/smb/server/mgmt/user_config.c +++ b/fs/smb/server/mgmt/user_config.c @@ -37,6 +37,17 @@ struct ksmbd_user *ksmbd_alloc_user(struct ksmbd_login_response *resp, { struct ksmbd_user *user; + /* + * resp->hash_sz is a __u16 taken from the mountd IPC login response but + * resp->hash[] is only KSMBD_REQ_MAX_HASH_SZ bytes. A malformed or + * malicious response can set hash_sz far beyond that (up to 65535), + * making the memcpy() below read past the response object + * (slab-out-of-bounds in ksmbd_alloc_user()). Reject any oversized + * hash rather than trust the length. + */ + if (resp->hash_sz > sizeof(resp->hash)) + return NULL; + user = kmalloc(sizeof(struct ksmbd_user), GFP_KERNEL); if (!user) return NULL; diff --git a/fs/smb/server/mgmt/user_session.c b/fs/smb/server/mgmt/user_session.c index 64f6ff2a7a8b..e55b95d22e16 100644 --- a/fs/smb/server/mgmt/user_session.c +++ b/fs/smb/server/mgmt/user_session.c @@ -32,12 +32,14 @@ static void free_channel_list(struct ksmbd_session *sess) struct channel *chann; unsigned long index; + down_write(&sess->chann_lock); xa_for_each(&sess->ksmbd_chann_list, index, chann) { xa_erase(&sess->ksmbd_chann_list, index); kfree(chann); } xa_destroy(&sess->ksmbd_chann_list); + up_write(&sess->chann_lock); } static void __session_rpc_close(struct ksmbd_session *sess, @@ -210,17 +212,29 @@ static void ksmbd_expire_session(struct ksmbd_conn *conn) int ksmbd_session_register(struct ksmbd_conn *conn, struct ksmbd_session *sess) { + int ret; + sess->dialect = conn->dialect; memcpy(sess->ClientGUID, conn->ClientGUID, SMB2_CLIENT_GUID_SIZE); ksmbd_expire_session(conn); - return xa_err(xa_store(&conn->sessions, sess->id, sess, GFP_KERNEL)); + ret = xa_err(xa_store(&conn->sessions, sess->id, sess, GFP_KERNEL)); + if (ret) { + down_write(&sessions_table_lock); + hash_del(&sess->hlist); + up_write(&sessions_table_lock); + ksmbd_user_session_put(sess); + } + + return ret; } static int ksmbd_chann_del(struct ksmbd_conn *conn, struct ksmbd_session *sess) { struct channel *chann; + down_write(&sess->chann_lock); chann = xa_erase(&sess->ksmbd_chann_list, (long)conn); + up_write(&sess->chann_lock); if (!chann) return -ENOENT; @@ -398,16 +412,16 @@ void destroy_previous_session(struct ksmbd_conn *conn, memcmp(user->passkey, prev_user->passkey, user->passkey_sz)) goto out; - ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_RECONNECT); - err = ksmbd_conn_wait_idle_sess_id(conn, id); + ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_RECONNECT); + err = ksmbd_conn_wait_idle_sess(conn, prev_sess); if (err) { - ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_SETUP); + ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_SETUP); goto out; } ksmbd_destroy_file_table(prev_sess); prev_sess->state = SMB2_SESSION_EXPIRED; - ksmbd_all_conn_set_status(id, KSMBD_SESS_NEED_SETUP); + ksmbd_all_conn_set_status(prev_sess, KSMBD_SESS_NEED_SETUP); ksmbd_launch_ksmbd_durable_scavenger(); out: up_write(&conn->session_lock); @@ -469,6 +483,7 @@ static struct ksmbd_session *__session_create(int protocol) rwlock_init(&sess->tree_conns_lock); atomic_set(&sess->refcnt, 2); init_rwsem(&sess->rpc_lock); + init_rwsem(&sess->chann_lock); ret = __init_smb2_session(sess); if (ret) diff --git a/fs/smb/server/mgmt/user_session.h b/fs/smb/server/mgmt/user_session.h index 523b68c4ca40..525650d6876c 100644 --- a/fs/smb/server/mgmt/user_session.h +++ b/fs/smb/server/mgmt/user_session.h @@ -49,6 +49,7 @@ struct ksmbd_session { char sess_key[CIFS_KEY_SIZE]; struct hlist_node hlist; + struct rw_semaphore chann_lock; struct xarray ksmbd_chann_list; struct xarray tree_conns; struct ida tree_conn_ida; diff --git a/fs/smb/server/misc.c b/fs/smb/server/misc.c index 7cc4b84a49b3..79de46a0283a 100644 --- a/fs/smb/server/misc.c +++ b/fs/smb/server/misc.c @@ -285,39 +285,6 @@ char *ksmbd_extract_sharename(struct unicode_map *um, const char *treename) return ksmbd_casefold_sharename(um, name); } -/** - * convert_to_unix_name() - convert windows name to unix format - * @share: ksmbd_share_config pointer - * @name: file name that is relative to share - * - * Return: converted name on success, otherwise NULL - */ -char *convert_to_unix_name(struct ksmbd_share_config *share, const char *name) -{ - int no_slash = 0, name_len, path_len; - char *new_name; - - if (name[0] == '/') - name++; - - path_len = share->path_sz; - name_len = strlen(name); - new_name = kmalloc(path_len + name_len + 2, GFP_KERNEL); - if (!new_name) - return new_name; - - memcpy(new_name, share->path, path_len); - if (new_name[path_len - 1] != '/') { - new_name[path_len] = '/'; - no_slash = 1; - } - - memcpy(new_name + path_len + no_slash, name, name_len); - path_len += name_len + no_slash; - new_name[path_len] = 0x00; - return new_name; -} - char *ksmbd_convert_dir_info_name(struct ksmbd_dir_info *d_info, const struct nls_table *local_nls, int *conv_len) diff --git a/fs/smb/server/misc.h b/fs/smb/server/misc.h index 1facfcd21200..53282a6d2436 100644 --- a/fs/smb/server/misc.h +++ b/fs/smb/server/misc.h @@ -22,7 +22,6 @@ void ksmbd_strip_last_slash(char *path); void ksmbd_conv_path_to_windows(char *path); char *ksmbd_casefold_sharename(struct unicode_map *um, const char *name); char *ksmbd_extract_sharename(struct unicode_map *um, const char *treename); -char *convert_to_unix_name(struct ksmbd_share_config *share, const char *name); #define KSMBD_DIR_INFO_ALIGNMENT 8 struct ksmbd_dir_info; diff --git a/fs/smb/server/oplock.c b/fs/smb/server/oplock.c index dc2ca423a1bf..7d85b9e338ec 100644 --- a/fs/smb/server/oplock.c +++ b/fs/smb/server/oplock.c @@ -56,6 +56,7 @@ static struct oplock_info *alloc_opinfo(struct ksmbd_work *work, opinfo->conn = ksmbd_conn_get(work->conn); opinfo->level = SMB2_OPLOCK_LEVEL_NONE; opinfo->op_state = OPLOCK_STATE_NONE; + spin_lock_init(&opinfo->state_lock); opinfo->pending_break = 0; opinfo->fid = id; opinfo->Tid = Tid; @@ -424,14 +425,23 @@ void close_id_del_oplock(struct ksmbd_file *fp) opinfo_del(opinfo); rcu_assign_pointer(fp->f_opinfo, NULL); - if (opinfo->op_state == OPLOCK_ACK_WAIT) { - opinfo->op_state = OPLOCK_CLOSING; - wake_up_interruptible_all(&opinfo->oplock_q); - if (opinfo->is_lease) { - atomic_set(&opinfo->breaking_cnt, 0); - wake_up_interruptible_all(&opinfo->oplock_brk); - } - } + spin_lock(&opinfo->state_lock); + if (opinfo->op_state == OPLOCK_ACK_WAIT && opinfo->is_lease) + atomic_set(&opinfo->breaking_cnt, 0); + /* + * An opinfo that has been removed from the inode list is terminal. Keep + * this transition and releasing pending_break under state_lock. a breaker + * takes the same lock before it acquires pending_break or sets ACK_WAIT. + */ + opinfo->op_state = OPLOCK_CLOSING; + clear_bit_unlock(0, &opinfo->pending_break); + spin_unlock(&opinfo->state_lock); + wake_up_interruptible_all(&opinfo->oplock_q); + if (opinfo->is_lease) + wake_up_interruptible_all(&opinfo->oplock_brk); + /* memory barrier is needed for wake_up_bit() */ + smp_mb__after_atomic(); + wake_up_bit(&opinfo->pending_break, 0); opinfo_count_dec(fp); atomic_dec(&opinfo->refcount); @@ -610,10 +620,16 @@ static void wait_for_break_ack(struct oplock_info *opinfo) /* is this a timeout ? */ if (!rc) { + spin_lock(&opinfo->state_lock); + if (opinfo->op_state == OPLOCK_CLOSING) { + spin_unlock(&opinfo->state_lock); + return; + } if (opinfo->is_lease) opinfo->o_lease->state = SMB2_LEASE_NONE_LE; opinfo->level = SMB2_OPLOCK_LEVEL_NONE; opinfo->op_state = OPLOCK_STATE_NONE; + spin_unlock(&opinfo->state_lock); } } @@ -625,17 +641,46 @@ static void wake_up_oplock_break(struct oplock_info *opinfo) wake_up_bit(&opinfo->pending_break, 0); } +static bool oplock_break_set_ack_wait(struct oplock_info *opinfo) +{ + bool ret = false; + + spin_lock(&opinfo->state_lock); + if (opinfo->op_state != OPLOCK_CLOSING) { + opinfo->op_state = OPLOCK_ACK_WAIT; + ret = true; + } + spin_unlock(&opinfo->state_lock); + + return ret; +} + static int oplock_break_pending(struct oplock_info *opinfo, int req_op_level) { - while (test_and_set_bit(0, &opinfo->pending_break)) { + for (;;) { + bool closing; + + spin_lock(&opinfo->state_lock); + closing = opinfo->op_state == OPLOCK_CLOSING; + if (!closing && !test_and_set_bit(0, &opinfo->pending_break)) { + spin_unlock(&opinfo->state_lock); + break; + } + spin_unlock(&opinfo->state_lock); + if (closing) + return -ENOENT; + wait_on_bit(&opinfo->pending_break, 0, TASK_UNINTERRUPTIBLE); /* Not immediately break to none. */ opinfo->open_trunc = 0; - if (opinfo->op_state == OPLOCK_CLOSING) + spin_lock(&opinfo->state_lock); + closing = opinfo->op_state == OPLOCK_CLOSING; + spin_unlock(&opinfo->state_lock); + if (closing) return -ENOENT; - else if (opinfo->level <= req_op_level) { + if (opinfo->level <= req_op_level) { if (opinfo->is_lease == false) return 1; @@ -999,13 +1044,17 @@ static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci, if (lease->state & (SMB2_LEASE_WRITE_CACHING_LE | SMB2_LEASE_HANDLE_CACHING_LE)) { + if (!oplock_break_set_ack_wait(brk_opinfo)) { + atomic_dec_if_positive(&brk_opinfo->breaking_cnt); + wake_up_oplock_break(brk_opinfo); + return -ENOENT; + } + if (in_work) { setup_async_work(in_work, NULL, NULL); smb2_send_interim_resp(in_work, STATUS_PENDING); release_async_work(in_work); } - - brk_opinfo->op_state = OPLOCK_ACK_WAIT; } else atomic_dec(&brk_opinfo->breaking_cnt); } else { @@ -1014,8 +1063,12 @@ static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci, return err < 0 ? err : 0; if (brk_opinfo->level == SMB2_OPLOCK_LEVEL_BATCH || - brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE) - brk_opinfo->op_state = OPLOCK_ACK_WAIT; + brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE) { + if (!oplock_break_set_ack_wait(brk_opinfo)) { + wake_up_oplock_break(brk_opinfo); + return -ENOENT; + } + } } if (brk_opinfo->is_lease) diff --git a/fs/smb/server/oplock.h b/fs/smb/server/oplock.h index f50366ef6637..4992c850280d 100644 --- a/fs/smb/server/oplock.h +++ b/fs/smb/server/oplock.h @@ -66,6 +66,7 @@ struct oplock_info { struct ksmbd_file *o_fp; int level; int op_state; + spinlock_t state_lock; unsigned long pending_break; u64 fid; atomic_t breaking_cnt; diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c index b9b0ea3701c2..ea552b1d4abb 100644 --- a/fs/smb/server/smb2pdu.c +++ b/fs/smb/server/smb2pdu.c @@ -79,7 +79,13 @@ static inline bool check_session_id(struct ksmbd_conn *conn, u64 id) struct channel *lookup_chann_list(struct ksmbd_session *sess, struct ksmbd_conn *conn) { - return xa_load(&sess->ksmbd_chann_list, (long)conn); + struct channel *chann; + + down_read(&sess->chann_lock); + chann = xa_load(&sess->ksmbd_chann_list, (long)conn); + up_read(&sess->chann_lock); + + return chann; } /** @@ -1563,7 +1569,9 @@ static int ntlm_authenticate(struct ksmbd_work *work, return -ENOMEM; chann->conn = conn; + down_write(&sess->chann_lock); xa_store(&sess->ksmbd_chann_list, (long)conn, chann, GFP_KERNEL); + up_write(&sess->chann_lock); } } @@ -1600,7 +1608,7 @@ static int krb5_authenticate(struct ksmbd_work *work, in_len = le16_to_cpu(req->SecurityBufferLength); out_blob = (char *)&rsp->hdr.ProtocolId + le16_to_cpu(rsp->SecurityBufferOffset); - out_len = work->response_sz - + out_len = work->response_sz - work->next_smb2_rsp_hdr_off - (le16_to_cpu(rsp->SecurityBufferOffset) + 4); retval = ksmbd_krb5_authenticate(sess, in_blob, in_len, @@ -1656,7 +1664,9 @@ static int krb5_authenticate(struct ksmbd_work *work, return -ENOMEM; chann->conn = conn; + down_write(&sess->chann_lock); xa_store(&sess->ksmbd_chann_list, (long)conn, chann, GFP_KERNEL); + up_write(&sess->chann_lock); } } @@ -1981,6 +1991,7 @@ int smb2_tree_connect(struct ksmbd_work *work) struct ksmbd_session *sess = work->sess; char *treename = NULL, *name = NULL; struct ksmbd_tree_conn_status status; + struct ksmbd_tree_connect *tree_conn = NULL; struct ksmbd_share_config *share = NULL; int rc = -EINVAL; @@ -2005,10 +2016,12 @@ int smb2_tree_connect(struct ksmbd_work *work) name, treename); status = ksmbd_tree_conn_connect(work, name); - if (status.ret == KSMBD_TREE_CONN_STATUS_OK) + if (status.ret == KSMBD_TREE_CONN_STATUS_OK) { + tree_conn = status.tree_conn; rsp->hdr.Id.SyncId.TreeId = cpu_to_le32(status.tree_conn->id); - else + } else { goto out_err1; + } share = status.tree_conn->share_conf; if (test_share_config_flag(share, KSMBD_SHARE_FLAG_PIPE)) { @@ -2039,8 +2052,13 @@ int smb2_tree_connect(struct ksmbd_work *work) status.tree_conn->posix_extensions = true; write_lock(&sess->tree_conns_lock); - status.tree_conn->t_state = TREE_CONNECTED; + if (status.tree_conn->t_state == TREE_DISCONNECTED) + status.ret = KSMBD_TREE_CONN_STATUS_ERROR; + else + status.tree_conn->t_state = TREE_CONNECTED; write_unlock(&sess->tree_conns_lock); + if (status.ret != KSMBD_TREE_CONN_STATUS_OK) + goto out_err1; rsp->StructureSize = cpu_to_le16(16); out_err1: /* @@ -2053,8 +2071,13 @@ int smb2_tree_connect(struct ksmbd_work *work) rsp->ShareFlags = SMB2_SHAREFLAG_MANUAL_CACHING; rc = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_tree_connect_rsp)); - if (rc) + if (rc) { + if (status.ret == KSMBD_TREE_CONN_STATUS_OK) { + ksmbd_tree_conn_disconnect(sess, status.tree_conn); + status.tree_conn = NULL; + } status.ret = KSMBD_TREE_CONN_STATUS_NOMEM; + } if (!IS_ERR(treename)) kfree(treename); @@ -2090,6 +2113,9 @@ int smb2_tree_connect(struct ksmbd_work *work) if (status.ret != KSMBD_TREE_CONN_STATUS_OK) smb2_set_err_rsp(work); + if (tree_conn) + ksmbd_tree_connect_put(tree_conn); + return rc; } @@ -2191,17 +2217,6 @@ int smb2_tree_disconnect(struct ksmbd_work *work) ksmbd_close_tree_conn_fds(work); - write_lock(&sess->tree_conns_lock); - if (tcon->t_state == TREE_DISCONNECTED) { - write_unlock(&sess->tree_conns_lock); - rsp->hdr.Status = STATUS_NETWORK_NAME_DELETED; - err = -ENOENT; - goto err_out; - } - - tcon->t_state = TREE_DISCONNECTED; - write_unlock(&sess->tree_conns_lock); - err = ksmbd_tree_conn_disconnect(sess, tcon); if (err) { rsp->hdr.Status = STATUS_NETWORK_NAME_DELETED; @@ -2236,7 +2251,6 @@ int smb2_session_logoff(struct ksmbd_work *work) struct ksmbd_session *sess = work->sess; struct smb2_logoff_req *req; struct smb2_logoff_rsp *rsp; - u64 sess_id; int err; WORK_BUFFERS(work, req, rsp); @@ -2250,8 +2264,7 @@ int smb2_session_logoff(struct ksmbd_work *work) smb2_set_err_rsp(work); return -ENOENT; } - sess_id = le64_to_cpu(req->hdr.SessionId); - ksmbd_all_conn_set_status(sess_id, KSMBD_SESS_NEED_RECONNECT); + ksmbd_all_conn_set_status(sess, KSMBD_SESS_NEED_RECONNECT); ksmbd_conn_unlock(conn); ksmbd_close_session_fds(work); @@ -2268,7 +2281,7 @@ int smb2_session_logoff(struct ksmbd_work *work) sess->state = SMB2_SESSION_EXPIRED; up_write(&conn->session_lock); - ksmbd_all_conn_set_status(sess_id, KSMBD_SESS_NEED_SETUP); + ksmbd_all_conn_set_status(sess, KSMBD_SESS_NEED_SETUP); rsp->StructureSize = cpu_to_le16(4); err = ksmbd_iov_pin_rsp(work, rsp, sizeof(struct smb2_logoff_rsp)); @@ -3244,6 +3257,7 @@ int smb2_open(struct ksmbd_work *work) if (file_present && !(req->CreateOptions & FILE_DELETE_ON_CLOSE_LE)) { rc = smb_check_perm_dacl(conn, &path, &daccess, + req->DesiredAccess, sess->user->uid); if (rc) goto err_out; @@ -3787,8 +3801,12 @@ int smb2_open(struct ksmbd_work *work) rsp->hdr.Status = STATUS_INVALID_PARAMETER; else if (rc == -EOPNOTSUPP) rsp->hdr.Status = STATUS_NOT_SUPPORTED; - else if (rc == -EACCES || rc == -ESTALE || rc == -EXDEV) - rsp->hdr.Status = STATUS_ACCESS_DENIED; + else if (rc == -EACCES || rc == -ESTALE || rc == -EXDEV) { + if (req->DesiredAccess & FILE_ACCESS_SYSTEM_SECURITY_LE) + rsp->hdr.Status = STATUS_PRIVILEGE_NOT_HELD; + else + rsp->hdr.Status = STATUS_ACCESS_DENIED; + } else if (rc == -ENOENT) rsp->hdr.Status = STATUS_OBJECT_NAME_INVALID; else if (rc == -EPERM) @@ -5533,7 +5551,6 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, struct smb2_query_info_req *req, struct smb2_query_info_rsp *rsp) { - struct ksmbd_session *sess = work->sess; struct ksmbd_conn *conn = work->conn; struct ksmbd_share_config *share = work->tcon->share_conf; int fsinfoclass = 0; @@ -5669,10 +5686,11 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info = (struct object_id_info *)(rsp->Buffer); - if (!user_guest(sess->user)) - memcpy(info->objid, user_passkey(sess->user), 16); + if (path.mnt->mnt_sb->s_uuid_len == 16) + memcpy(info->objid, path.mnt->mnt_sb->s_uuid.b, + path.mnt->mnt_sb->s_uuid_len); else - memset(info->objid, 0, 16); + memcpy(info->objid, &stfs.f_fsid, sizeof(stfs.f_fsid)); info->extended_info.magic = cpu_to_le32(EXTENDED_INFO_MAGIC); info->extended_info.version = cpu_to_le32(1); @@ -6788,13 +6806,18 @@ static noinline int smb2_read_pipe(struct ksmbd_work *work) } aux_payload_buf = - kvmalloc(rpc_resp->payload_sz, GFP_KERNEL); + kvmalloc(ALIGN(rpc_resp->payload_sz, 8), + GFP_KERNEL); if (!aux_payload_buf) { err = -ENOMEM; goto out; } memcpy(aux_payload_buf, rpc_resp->payload, rpc_resp->payload_sz); + if (rpc_resp->payload_sz & 7) + memset(aux_payload_buf + rpc_resp->payload_sz, 0, + ALIGN(rpc_resp->payload_sz, 8) - + rpc_resp->payload_sz); nbytes = rpc_resp->payload_sz; err = ksmbd_iov_pin_rsp_read(work, (void *)rsp, @@ -7493,6 +7516,19 @@ static void smb2_remove_blocked_lock(void **argv) wake_up(&flock->fl_wait); } +static void smb2_free_lock(struct file_lock *flock) +{ + ksmbd_vfs_posix_lock_unblock(flock); + locks_free_lock(flock); +} + +static void smb2_free_blocked_lock(struct file_lock *flock) +{ + ksmbd_vfs_posix_lock_unblock(flock); + locks_wake_up(flock); + locks_free_lock(flock); +} + static inline bool lock_defer_pending(struct file_lock *fl) { /* check pending lock waiters */ @@ -7646,14 +7682,14 @@ int smb2_lock(struct ksmbd_work *work) cmp_lock->end == smb_lock->end && !lock_defer_pending(cmp_lock->fl)) { nolock = 0; - list_del(&cmp_lock->flist); - list_del(&cmp_lock->clist); + list_del_init(&cmp_lock->flist); + list_del_init(&cmp_lock->clist); cmp_lock->conn = NULL; spin_unlock(&conn->llist_lock); up_read(&conn_list_lock); ksmbd_conn_put(conn); - locks_free_lock(cmp_lock->fl); + smb2_free_lock(cmp_lock->fl); kfree(cmp_lock); goto out_check_cl; } @@ -7735,11 +7771,12 @@ int smb2_lock(struct ksmbd_work *work) ksmbd_debug(SMB, "would have to wait for getting lock\n"); - list_add(&smb_lock->llist, &rollback_list); argv = kmalloc(sizeof(void *), GFP_KERNEL); if (!argv) { err = -ENOMEM; + smb2_free_blocked_lock(flock); + kfree(smb_lock); goto out; } argv[0] = flock; @@ -7750,8 +7787,11 @@ int smb2_lock(struct ksmbd_work *work) if (rc) { kfree(argv); err = -ENOMEM; + smb2_free_blocked_lock(flock); + kfree(smb_lock); goto out; } + list_add(&smb_lock->llist, &rollback_list); spin_lock(&fp->f_lock); list_add(&work->fp_entry, &fp->blocked_works); spin_unlock(&fp->f_lock); @@ -7784,16 +7824,10 @@ int smb2_lock(struct ksmbd_work *work) rsp->hdr.Status = STATUS_RANGE_NOT_LOCKED; kfree(smb_lock); - goto out2; + /* rollback_list may still hold earlier grants */ + goto out; } else if (!rc) { list_add(&smb_lock->llist, &rollback_list); - smb_lock->conn = ksmbd_conn_get(work->conn); - spin_lock(&work->conn->llist_lock); - list_add_tail(&smb_lock->clist, - &work->conn->lock_list); - list_add_tail(&smb_lock->flist, - &fp->lock_list); - spin_unlock(&work->conn->llist_lock); ksmbd_debug(SMB, "successful in taking lock\n"); } else { locks_free_lock(flock); @@ -7815,6 +7849,19 @@ int smb2_lock(struct ksmbd_work *work) if (err) goto out; + /* publish only once the whole batch has committed */ + if (!list_empty(&rollback_list)) { + spin_lock(&work->conn->llist_lock); + list_for_each_entry(smb_lock, &rollback_list, llist) { + smb_lock->conn = ksmbd_conn_get(work->conn); + list_add_tail(&smb_lock->clist, + &work->conn->lock_list); + list_add_tail(&smb_lock->flist, + &fp->lock_list); + } + spin_unlock(&work->conn->llist_lock); + } + ksmbd_fd_put(work, fp); return 0; @@ -7842,16 +7889,7 @@ int smb2_lock(struct ksmbd_work *work) } list_del(&smb_lock->llist); - conn = smb_lock->conn; - spin_lock(&conn->llist_lock); - if (!list_empty(&smb_lock->flist)) - list_del(&smb_lock->flist); - list_del(&smb_lock->clist); - smb_lock->conn = NULL; - spin_unlock(&conn->llist_lock); - ksmbd_conn_put(conn); - - locks_free_lock(smb_lock->fl); + smb2_free_lock(smb_lock->fl); if (rlock) locks_free_lock(rlock); kfree(smb_lock); @@ -8789,7 +8827,10 @@ static void smb20_oplock_break_ack(struct ksmbd_work *work) smb2_set_err_rsp(work); out: - opinfo->op_state = OPLOCK_STATE_NONE; + spin_lock(&opinfo->state_lock); + if (opinfo->op_state != OPLOCK_CLOSING) + opinfo->op_state = OPLOCK_STATE_NONE; + spin_unlock(&opinfo->state_lock); wake_up_interruptible_all(&opinfo->oplock_q); opinfo_put(opinfo); ksmbd_fd_put(work, fp); @@ -8875,9 +8916,12 @@ static void smb21_lease_break_ack(struct ksmbd_work *work) if (ret) goto err_out; - opinfo->op_state = OPLOCK_STATE_NONE; + spin_lock(&opinfo->state_lock); + if (opinfo->op_state != OPLOCK_CLOSING) + opinfo->op_state = OPLOCK_STATE_NONE; + spin_unlock(&opinfo->state_lock); wake_up_interruptible_all(&opinfo->oplock_q); - atomic_dec(&opinfo->breaking_cnt); + atomic_dec_if_positive(&opinfo->breaking_cnt); wake_up_interruptible_all(&opinfo->oplock_brk); opinfo_put(opinfo); return; diff --git a/fs/smb/server/smbacl.c b/fs/smb/server/smbacl.c index b68b63a3195f..da7bf23a2db0 100644 --- a/fs/smb/server/smbacl.c +++ b/fs/smb/server/smbacl.c @@ -380,10 +380,10 @@ void free_acl_state(struct posix_acl_state *state) kfree(state->groups); } -static void parse_dacl(struct mnt_idmap *idmap, - struct smb_acl *pdacl, char *end_of_acl, - struct smb_sid *pownersid, struct smb_sid *pgrpsid, - struct smb_fattr *fattr) +static int parse_dacl(struct mnt_idmap *idmap, + struct smb_acl *pdacl, char *end_of_acl, + struct smb_sid *pownersid, struct smb_sid *pgrpsid, + struct smb_fattr *fattr) { int i, ret; u16 num_aces = 0; @@ -397,13 +397,13 @@ static void parse_dacl(struct mnt_idmap *idmap, bool owner_found = false, group_found = false, others_found = false; if (!pdacl) - return; + return 0; /* validate that we do not go past end of acl */ if (end_of_acl < (char *)pdacl + sizeof(struct smb_acl) || end_of_acl < (char *)pdacl + le16_to_cpu(pdacl->size)) { pr_err("ACL too small to parse DACL\n"); - return; + return -EINVAL; } ksmbd_debug(SMB, "DACL revision %d size %d num aces %d\n", @@ -415,31 +415,31 @@ static void parse_dacl(struct mnt_idmap *idmap, num_aces = le16_to_cpu(pdacl->num_aces); if (num_aces <= 0) - return; + return 0; dacl_size = le16_to_cpu(pdacl->size); if (dacl_size < sizeof(struct smb_acl)) - return; + return -EINVAL; if (num_aces > (dacl_size - sizeof(struct smb_acl)) / (offsetof(struct smb_ace, sid) + offsetof(struct smb_sid, sub_auth) + sizeof(__le16))) - return; + return -EINVAL; ret = init_acl_state(&acl_state, num_aces); if (ret) - return; + return ret; ret = init_acl_state(&default_acl_state, num_aces); if (ret) { free_acl_state(&acl_state); - return; + return ret; } ppace = kmalloc_array(num_aces, sizeof(struct smb_ace *), GFP_KERNEL); if (!ppace) { free_acl_state(&default_acl_state); free_acl_state(&acl_state); - return; + return -ENOMEM; } /* @@ -448,8 +448,10 @@ static void parse_dacl(struct mnt_idmap *idmap, * user/group/other have no permissions */ for (i = 0; i < num_aces; ++i) { - if (end_of_acl - acl_base < acl_size) - break; + if (end_of_acl - acl_base < acl_size) { + ret = -EINVAL; + goto out; + } ppace[i] = (struct smb_ace *)(acl_base + acl_size); acl_base = (char *)ppace[i]; @@ -462,8 +464,10 @@ static void parse_dacl(struct mnt_idmap *idmap, (end_of_acl - acl_base < acl_size + sizeof(__le32) * ppace[i]->sid.num_subauth) || (le16_to_cpu(ppace[i]->size) < - acl_size + sizeof(__le32) * ppace[i]->sid.num_subauth)) - break; + acl_size + sizeof(__le32) * ppace[i]->sid.num_subauth)) { + ret = -EINVAL; + goto out; + } acl_size = le16_to_cpu(ppace[i]->size); ppace[i]->access_req = @@ -538,7 +542,6 @@ static void parse_dacl(struct mnt_idmap *idmap, ((acl_mode & 0700) >> 6) | 0004; } } - kfree(ppace); if (owner_found) { /* The owner must be set to at least read-only. */ @@ -581,10 +584,12 @@ static void parse_dacl(struct mnt_idmap *idmap, fattr->cf_acls = posix_acl_alloc(acl_state.users->n + acl_state.groups->n + 4, GFP_KERNEL); - if (fattr->cf_acls) { - cf_pace = fattr->cf_acls->a_entries; - posix_state_to_acl(&acl_state, cf_pace); + if (!fattr->cf_acls) { + ret = -ENOMEM; + goto out; } + cf_pace = fattr->cf_acls->a_entries; + posix_state_to_acl(&acl_state, cf_pace); } } @@ -595,14 +600,20 @@ static void parse_dacl(struct mnt_idmap *idmap, fattr->cf_dacls = posix_acl_alloc(default_acl_state.users->n + default_acl_state.groups->n + 4, GFP_KERNEL); - if (fattr->cf_dacls) { - cf_pdace = fattr->cf_dacls->a_entries; - posix_state_to_acl(&default_acl_state, cf_pdace); + if (!fattr->cf_dacls) { + ret = -ENOMEM; + goto out; } + cf_pdace = fattr->cf_dacls->a_entries; + posix_state_to_acl(&default_acl_state, cf_pdace); } } + ret = 0; +out: + kfree(ppace); free_acl_state(&acl_state); free_acl_state(&default_acl_state); + return ret; } static void set_posix_acl_entries_dacl(struct mnt_idmap *idmap, @@ -963,8 +974,10 @@ int parse_sec_desc(struct mnt_idmap *idmap, struct smb_ntsd *pntsd, if (dacloffset < sizeof(struct smb_ntsd)) return -EINVAL; - parse_dacl(idmap, dacl_ptr, end_of_acl, - owner_sid_ptr, group_sid_ptr, fattr); + rc = parse_dacl(idmap, dacl_ptr, end_of_acl, + owner_sid_ptr, group_sid_ptr, fattr); + if (rc) + return rc; } return 0; @@ -1136,6 +1149,40 @@ static int smb_append_inherited_ace(struct smb_ace **ace, int *nt_size, return 0; } +static int smb_validate_ntsd_sid(struct smb_ntsd *pntsd, size_t pntsd_size, + unsigned int sid_offset, struct smb_sid **sid, + size_t *sid_size) +{ + size_t sid_end; + + *sid = NULL; + *sid_size = 0; + + if (!sid_offset) + return 0; + + if (sid_offset < sizeof(struct smb_ntsd) || + check_add_overflow(sid_offset, (size_t)CIFS_SID_BASE_SIZE, + &sid_end) || + sid_end > pntsd_size) + return -EINVAL; + + *sid = (struct smb_sid *)((char *)pntsd + sid_offset); + if ((*sid)->num_subauth > SID_MAX_SUB_AUTHORITIES) + return -EINVAL; + + if (check_add_overflow((size_t)CIFS_SID_BASE_SIZE, + sizeof(__le32) * (size_t)(*sid)->num_subauth, + &sid_end)) + return -EINVAL; + + if (sid_offset > pntsd_size || sid_end > pntsd_size - sid_offset) + return -EINVAL; + + *sid_size = sid_end; + return 0; +} + int smb_inherit_dacl(struct ksmbd_conn *conn, const struct path *path, unsigned int uid, unsigned int gid) @@ -1148,28 +1195,28 @@ int smb_inherit_dacl(struct ksmbd_conn *conn, struct dentry *parent = path->dentry->d_parent; struct mnt_idmap *idmap = mnt_idmap(path->mnt); int inherited_flags = 0, flags = 0, i, nt_size = 0, pdacl_size; - int rc = 0, pntsd_type, pntsd_size, acl_len, aces_size; + int rc = 0, pntsd_type, ppntsd_size, acl_len, aces_size; unsigned int dacloffset; size_t dacl_struct_end; u16 num_aces, ace_cnt = 0; char *aces_base; bool is_dir = S_ISDIR(d_inode(path->dentry)->i_mode); - pntsd_size = ksmbd_vfs_get_sd_xattr(conn, idmap, + ppntsd_size = ksmbd_vfs_get_sd_xattr(conn, idmap, parent, &parent_pntsd); - if (pntsd_size <= 0) + if (ppntsd_size <= 0) return -ENOENT; dacloffset = le32_to_cpu(parent_pntsd->dacloffset); if (!dacloffset || check_add_overflow(dacloffset, sizeof(struct smb_acl), &dacl_struct_end) || - dacl_struct_end > (size_t)pntsd_size) { + dacl_struct_end > (size_t)ppntsd_size) { rc = -EINVAL; goto free_parent_pntsd; } parent_pdacl = (struct smb_acl *)((char *)parent_pntsd + dacloffset); - acl_len = pntsd_size - dacloffset; + acl_len = ppntsd_size - dacloffset; num_aces = le16_to_cpu(parent_pdacl->num_aces); pntsd_type = le16_to_cpu(parent_pntsd->type); pdacl_size = le16_to_cpu(parent_pdacl->size); @@ -1179,7 +1226,24 @@ int smb_inherit_dacl(struct ksmbd_conn *conn, goto free_parent_pntsd; } - aces_base = kmalloc(sizeof(struct smb_ace) * num_aces * 2, GFP_KERNEL); + aces_size = pdacl_size - sizeof(struct smb_acl); + + /* + * Validate num_aces against the DACL payload before allocating. + * Each ACE must be at least as large as its fixed-size header + * (up to the SID base), so num_aces cannot exceed the payload + * divided by the minimum ACE size. This mirrors the existing + * check in parse_dacl(). + */ + if (num_aces > aces_size / (offsetof(struct smb_ace, sid) + + offsetof(struct smb_sid, sub_auth) + + sizeof(__le16))) { + rc = -EINVAL; + goto free_parent_pntsd; + } + + aces_base = kmalloc_array(num_aces * 2, sizeof(struct smb_ace), + GFP_KERNEL); if (!aces_base) { rc = -ENOMEM; goto free_parent_pntsd; @@ -1188,7 +1252,6 @@ int smb_inherit_dacl(struct ksmbd_conn *conn, aces = (struct smb_ace *)aces_base; parent_aces = (struct smb_ace *)((char *)parent_pdacl + sizeof(struct smb_acl)); - aces_size = acl_len - sizeof(struct smb_acl); if (pntsd_type & DACL_AUTO_INHERITED) inherited_flags = INHERITED_ACE; @@ -1196,11 +1259,14 @@ int smb_inherit_dacl(struct ksmbd_conn *conn, for (i = 0; i < num_aces; i++) { int pace_size; - if (offsetof(struct smb_ace, access_req) > aces_size) + if (aces_size < offsetof(struct smb_ace, sid) + + CIFS_SID_BASE_SIZE) break; pace_size = le16_to_cpu(parent_aces->size); - if (pace_size > aces_size) + if (pace_size > aces_size || + pace_size < offsetof(struct smb_ace, sid) + + CIFS_SID_BASE_SIZE) break; if (parent_aces->sid.num_subauth > SID_MAX_SUB_AUTHORITIES || @@ -1264,19 +1330,19 @@ int smb_inherit_dacl(struct ksmbd_conn *conn, struct smb_ntsd *pntsd; struct smb_acl *pdacl; struct smb_sid *powner_sid = NULL, *pgroup_sid = NULL; - int powner_sid_size = 0, pgroup_sid_size = 0, pntsd_size; + size_t powner_sid_size = 0, pgroup_sid_size = 0, pntsd_size; size_t pntsd_alloc_size; - if (parent_pntsd->osidoffset) { - powner_sid = (struct smb_sid *)((char *)parent_pntsd + - le32_to_cpu(parent_pntsd->osidoffset)); - powner_sid_size = 1 + 1 + 6 + (powner_sid->num_subauth * 4); - } - if (parent_pntsd->gsidoffset) { - pgroup_sid = (struct smb_sid *)((char *)parent_pntsd + - le32_to_cpu(parent_pntsd->gsidoffset)); - pgroup_sid_size = 1 + 1 + 6 + (pgroup_sid->num_subauth * 4); - } + rc = smb_validate_ntsd_sid(parent_pntsd, ppntsd_size, + le32_to_cpu(parent_pntsd->osidoffset), + &powner_sid, &powner_sid_size); + if (rc) + goto free_aces_base; + rc = smb_validate_ntsd_sid(parent_pntsd, ppntsd_size, + le32_to_cpu(parent_pntsd->gsidoffset), + &pgroup_sid, &pgroup_sid_size); + if (rc) + goto free_aces_base; if (check_add_overflow(sizeof(struct smb_ntsd), (size_t)powner_sid_size, @@ -1379,7 +1445,7 @@ bool smb_inherit_flags(int flags, bool is_dir) } int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, - __le32 *pdaccess, int uid) + __le32 *pdaccess, __le32 raw_daccess, int uid) { struct mnt_idmap *idmap = mnt_idmap(path->mnt); struct smb_ntsd *pntsd = NULL; @@ -1389,10 +1455,11 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, unsigned int dacl_offset; size_t dacl_struct_end; struct smb_sid sid; - int granted = le32_to_cpu(*pdaccess & ~FILE_MAXIMAL_ACCESS_LE); + int requested = le32_to_cpu(*pdaccess & ~FILE_MAXIMAL_ACCESS_LE); + int granted = requested; struct smb_ace *ace; int i, found = 0; - unsigned int access_bits = 0; + unsigned int access_bits = 0, denied = 0; struct smb_ace *others_ace = NULL; struct posix_acl_entry *pa_entry; unsigned int sid_type = SIDOWNER; @@ -1426,10 +1493,13 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, goto err_out; } + if (!uid) + sid_type = SIDUNIX_USER; + id_to_sid(uid, sid_type, &sid); + if (*pdaccess & FILE_MAXIMAL_ACCESS_LE) { - granted = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES | + access_bits = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES | DELETE; - ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl)); aces_size = pdacl_size - sizeof(struct smb_acl); for (i = 0; i < le16_to_cpu(pdacl->num_aces); i++) { @@ -1442,18 +1512,41 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, CIFS_SID_BASE_SIZE) break; aces_size -= ace_size; - granted |= le32_to_cpu(ace->access_req); + + if (ace->sid.num_subauth > SID_MAX_SUB_AUTHORITIES || + ace_size < offsetof(struct smb_ace, sid) + + CIFS_SID_BASE_SIZE + + sizeof(__le32) * ace->sid.num_subauth) + break; + + if (ace->flags & INHERIT_ONLY_ACE || + (compare_sids(&sid, &ace->sid) && + compare_sids(&sid_unix_NFS_mode, &ace->sid) && + compare_sids(&sid_everyone, &ace->sid) && + compare_sids(&sid_authusers, &ace->sid))) + goto next_ace; + + switch (ace->type) { + case ACCESS_ALLOWED_ACE_TYPE: + access_bits |= le32_to_cpu(ace->access_req); + break; + case ACCESS_DENIED_ACE_TYPE: + case ACCESS_DENIED_CALLBACK_ACE_TYPE: + denied |= ~access_bits & + le32_to_cpu(ace->access_req); + break; + } +next_ace: ace = (struct smb_ace *)((char *)ace + le16_to_cpu(ace->size)); } - - if (!pdacl->num_aces) - granted = GENERIC_ALL_FLAGS; + access_bits &= ~denied; + if ((raw_daccess & FILE_GENERIC_EXECUTE_LE) && + S_ISREG(d_inode(path->dentry)->i_mode) && + (access_bits & GENERIC_READ_FLAGS) == GENERIC_READ_FLAGS) + access_bits |= FILE_EXECUTE; + granted = requested | access_bits; } - if (!uid) - sid_type = SIDUNIX_USER; - id_to_sid(uid, sid_type, &sid); - ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl)); aces_size = pdacl_size - sizeof(struct smb_acl); for (i = 0; i < le16_to_cpu(pdacl->num_aces); i++) { @@ -1477,25 +1570,16 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, found = 1; break; } - if (!compare_sids(&sid_everyone, &ace->sid)) + if (!compare_sids(&sid_everyone, &ace->sid) || + !compare_sids(&sid_authusers, &ace->sid)) others_ace = ace; ace = (struct smb_ace *)((char *)ace + le16_to_cpu(ace->size)); } - if (*pdaccess & FILE_MAXIMAL_ACCESS_LE && found) { - granted = READ_CONTROL | WRITE_DAC | FILE_READ_ATTRIBUTES | - DELETE; - - granted |= le32_to_cpu(ace->access_req); - - if (!pdacl->num_aces) - granted = GENERIC_ALL_FLAGS; - } - if (IS_ENABLED(CONFIG_FS_POSIX_ACL)) { posix_acls = get_inode_acl(d_inode(path->dentry), ACL_TYPE_ACCESS); - if (!IS_ERR_OR_NULL(posix_acls) && !found) { + if (!IS_ERR_OR_NULL(posix_acls) && !found && !others_ace) { unsigned int id = -1; pa_entry = posix_acls->a_entries; @@ -1533,14 +1617,16 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, } } - switch (ace->type) { - case ACCESS_ALLOWED_ACE_TYPE: - access_bits = le32_to_cpu(ace->access_req); - break; - case ACCESS_DENIED_ACE_TYPE: - case ACCESS_DENIED_CALLBACK_ACE_TYPE: - access_bits = le32_to_cpu(~ace->access_req); - break; + if (!(*pdaccess & FILE_MAXIMAL_ACCESS_LE)) { + switch (ace->type) { + case ACCESS_ALLOWED_ACE_TYPE: + access_bits = le32_to_cpu(ace->access_req); + break; + case ACCESS_DENIED_ACE_TYPE: + case ACCESS_DENIED_CALLBACK_ACE_TYPE: + access_bits = le32_to_cpu(~ace->access_req); + break; + } } check_access_bits: diff --git a/fs/smb/server/smbacl.h b/fs/smb/server/smbacl.h index ab21ba2cd4df..bf0312bfa040 100644 --- a/fs/smb/server/smbacl.h +++ b/fs/smb/server/smbacl.h @@ -95,7 +95,7 @@ bool smb_inherit_flags(int flags, bool is_dir); int smb_inherit_dacl(struct ksmbd_conn *conn, const struct path *path, unsigned int uid, unsigned int gid); int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, - __le32 *pdaccess, int uid); + __le32 *pdaccess, __le32 raw_daccess, int uid); int set_info_sec(struct ksmbd_conn *conn, struct ksmbd_tree_connect *tcon, const struct path *path, struct smb_ntsd *pntsd, int ntsd_len, bool type_check, bool get_write); diff --git a/fs/smb/server/transport_rdma.c b/fs/smb/server/transport_rdma.c index 10ceade59052..d522ace20840 100644 --- a/fs/smb/server/transport_rdma.c +++ b/fs/smb/server/transport_rdma.c @@ -1977,9 +1977,9 @@ static int smb_direct_prepare(struct ksmbd_transport *t) goto out; req = (struct smb_direct_negotiate_req *)recvmsg->packet; - st->max_recv_size = min_t(int, st->max_recv_size, + st->max_recv_size = min_t(u32, st->max_recv_size, le32_to_cpu(req->preferred_send_size)); - st->max_send_size = min_t(int, st->max_send_size, + st->max_send_size = min_t(u32, st->max_send_size, le32_to_cpu(req->max_receive_size)); st->max_fragmented_send_size = le32_to_cpu(req->max_fragmented_size); @@ -2206,7 +2206,8 @@ static int smb_direct_ib_client_add(struct ib_device *ib_dev) if (ib_dev->node_type != RDMA_NODE_IB_CA) smb_direct_port = SMB_DIRECT_PORT_IWARP; - if (!rdma_frwr_is_supported(&ib_dev->attrs)) + if (!ib_dev->ops.get_netdev || + !rdma_frwr_is_supported(&ib_dev->attrs)) return 0; smb_dev = kzalloc(sizeof(*smb_dev), GFP_KERNEL); @@ -2309,38 +2310,17 @@ bool ksmbd_rdma_capable_netdev(struct net_device *netdev) for (i = 0; i < smb_dev->ib_dev->phys_port_cnt; i++) { struct net_device *ndev; - if (smb_dev->ib_dev->ops.get_netdev) { - ndev = smb_dev->ib_dev->ops.get_netdev( - smb_dev->ib_dev, i + 1); - if (!ndev) - continue; + ndev = smb_dev->ib_dev->ops.get_netdev(smb_dev->ib_dev, + i + 1); + if (!ndev) + continue; - if (ndev == netdev) { - dev_put(ndev); - rdma_capable = true; - goto out; - } + if (ndev == netdev) { dev_put(ndev); - /* if ib_dev does not implement ops.get_netdev - * check for matching infiniband GUID in hw_addr - */ - } else if (netdev->type == ARPHRD_INFINIBAND) { - struct netdev_hw_addr *ha; - union ib_gid gid; - u32 port_num; - int ret; - - netdev_hw_addr_list_for_each( - ha, &netdev->dev_addrs) { - memcpy(&gid, ha->addr + 4, sizeof(gid)); - ret = ib_find_gid(smb_dev->ib_dev, &gid, - &port_num, NULL); - if (!ret) { - rdma_capable = true; - goto out; - } - } + rdma_capable = true; + goto out; } + dev_put(ndev); } } out: diff --git a/fs/smb/server/vfs.c b/fs/smb/server/vfs.c index 2bea89fc9ecd..fd1697c2d665 100644 --- a/fs/smb/server/vfs.c +++ b/fs/smb/server/vfs.c @@ -73,7 +73,7 @@ static int ksmbd_vfs_path_lookup_locked(struct ksmbd_share_config *share_conf, struct qstr last; struct filename *filename; struct path *root_share_path = &share_conf->vfs_path; - int err, type; + int err; struct dentry *d; if (pathname[0] == '\0') { @@ -88,19 +88,13 @@ static int ksmbd_vfs_path_lookup_locked(struct ksmbd_share_config *share_conf, return PTR_ERR(filename); err = vfs_path_parent_lookup(filename, flags, - parent_path, &last, &type, + parent_path, &last, root_share_path); if (err) { putname(filename); return err; } - if (unlikely(type != LAST_NORM)) { - path_put(parent_path); - putname(filename); - return -ENOENT; - } - err = mnt_want_write(parent_path->mnt); if (err) { path_put(parent_path); @@ -288,17 +282,20 @@ static ssize_t ksmbd_vfs_getcasexattr(struct mnt_idmap *idmap, static int ksmbd_vfs_stream_read(struct ksmbd_file *fp, char *buf, loff_t *pos, size_t count) { + const struct cred *saved_cred; ssize_t v_len; char *stream_buf = NULL; ksmbd_debug(VFS, "read stream data pos : %llu, count : %zd\n", *pos, count); + saved_cred = override_creds(fp->filp->f_cred); v_len = ksmbd_vfs_getcasexattr(file_mnt_idmap(fp->filp), fp->filp->f_path.dentry, fp->stream.name, fp->stream.size, &stream_buf); + revert_creds(saved_cred); if ((int)v_len <= 0) return (int)v_len; @@ -417,6 +414,7 @@ int ksmbd_vfs_read(struct ksmbd_work *work, struct ksmbd_file *fp, size_t count, static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos, size_t count) { + const struct cred *saved_cred; char *stream_buf = NULL, *wbuf; struct mnt_idmap *idmap = file_mnt_idmap(fp->filp); size_t size; @@ -437,6 +435,7 @@ static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos, count = XATTR_SIZE_MAX - *pos; } + saved_cred = override_creds(fp->filp->f_cred); v_len = ksmbd_vfs_getcasexattr(idmap, fp->filp->f_path.dentry, fp->stream.name, @@ -445,14 +444,14 @@ static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos, if (v_len < 0) { pr_err("not found stream in xattr : %zd\n", v_len); err = v_len; - goto out; + goto out_revert; } if (v_len < size) { wbuf = kvzalloc(size, GFP_KERNEL); if (!wbuf) { err = -ENOMEM; - goto out; + goto out_revert; } if (v_len > 0) @@ -470,6 +469,8 @@ static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos, size, 0, true); +out_revert: + revert_creds(saved_cred); if (err < 0) goto out; @@ -698,22 +699,16 @@ int ksmbd_vfs_rename(struct ksmbd_work *work, const struct path *old_path, struct filename *to; struct ksmbd_share_config *share_conf = work->tcon->share_conf; struct ksmbd_file *parent_fp; - int new_type; int err, lookup_flags = LOOKUP_NO_SYMLINKS; if (ksmbd_override_fsids(work)) return -ENOMEM; to = getname_kernel(newname); - if (IS_ERR(to)) { - err = PTR_ERR(to); - goto revert_fsids; - } retry: err = vfs_path_parent_lookup(to, lookup_flags | LOOKUP_BENEATH, - &new_path, &new_last, &new_type, - &share_conf->vfs_path); + &new_path, &new_last, &share_conf->vfs_path); if (err) goto out1; @@ -810,7 +805,6 @@ int ksmbd_vfs_rename(struct ksmbd_work *work, const struct path *old_path, } out1: putname(to); -revert_fsids: ksmbd_revert_fsids(work); return err; } @@ -1248,7 +1242,7 @@ int ksmbd_vfs_kern_path_locked(struct ksmbd_work *work, char *name, if (!err) return 0; - if (caseless) { + if (caseless && err == -ENOENT) { char *filepath; size_t path_len, remain_len; @@ -1279,7 +1273,7 @@ int ksmbd_vfs_kern_path_locked(struct ksmbd_work *work, char *name, err = vfs_path_lookup(share_conf->vfs_path.dentry, share_conf->vfs_path.mnt, filepath, - flags, + flags | LOOKUP_BENEATH, path); if (!is_last) next[0] = '/'; @@ -1330,15 +1324,39 @@ struct dentry *ksmbd_vfs_kern_path_create(struct ksmbd_work *work, unsigned int flags, struct path *path) { - char *abs_name; + struct ksmbd_share_config *share_conf = work->tcon->share_conf; + struct filename *filename; + struct qstr last; struct dentry *dent; + int err; - abs_name = convert_to_unix_name(work->tcon->share_conf, name); - if (!abs_name) - return ERR_PTR(-ENOMEM); + /* resolve the name beneath the share root so ".." cannot escape */ + filename = getname_kernel(name); + if (IS_ERR(filename)) + return ERR_CAST(filename); + + err = vfs_path_parent_lookup(filename, flags | LOOKUP_BENEATH, + path, &last, &share_conf->vfs_path); + if (err) { + putname(filename); + return ERR_PTR(err); + } - dent = kern_path_create(AT_FDCWD, abs_name, path, flags); - kfree(abs_name); + err = mnt_want_write(path->mnt); + if (err) { + path_put(path); + putname(filename); + return ERR_PTR(err); + } + + inode_lock_nested(path->dentry->d_inode, I_MUTEX_PARENT); + dent = lookup_one_qstr_excl(&last, path->dentry, LOOKUP_CREATE); + if (IS_ERR(dent)) { + inode_unlock(path->dentry->d_inode); + mnt_drop_write(path->mnt); + path_put(path); + } + putname(filename); return dent; } diff --git a/fs/smb/server/vfs_cache.c b/fs/smb/server/vfs_cache.c index 287858f6304e..c96016417e91 100644 --- a/fs/smb/server/vfs_cache.c +++ b/fs/smb/server/vfs_cache.c @@ -388,8 +388,9 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp) if (!IS_ERR_OR_NULL(filp)) fput(filp); - /* because the reference count of fp is 0, it is guaranteed that - * there are not accesses to fp->lock_list. + /* + * The zero fp reference count serializes access to fp->lock_list, but + * the VFS may still have blocked requests chained below these locks. */ list_for_each_entry_safe(smb_lock, tmp_lock, &fp->lock_list, flist) { struct ksmbd_conn *conn = smb_lock->conn; @@ -402,7 +403,8 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp) ksmbd_conn_put(conn); } - list_del(&smb_lock->flist); + list_del_init(&smb_lock->flist); + ksmbd_vfs_posix_lock_unblock(smb_lock->fl); locks_free_lock(smb_lock->fl); kfree(smb_lock); } diff --git a/fs/super.c b/fs/super.c index f8fe07501c9f..cb7adc567210 100644 --- a/fs/super.c +++ b/fs/super.c @@ -409,7 +409,10 @@ static struct super_block *alloc_super(struct file_system_type *type, int flags, static void __put_super(struct super_block *s) { if (!--s->s_count) { + struct file_system_type *type = s->s_type; + list_del_init(&s->s_list); + hlist_del_init(&s->s_instances); WARN_ON(s->s_dentry_lru.node); WARN_ON(s->s_inode_lru.node); WARN_ON(!list_empty(&s->s_mounts)); @@ -417,6 +420,8 @@ static void __put_super(struct super_block *s) put_user_ns(s->s_user_ns); kfree(s->s_subtype); call_rcu(&s->rcu, destroy_super_rcu); + /* The unlink above may touch type->fs_supers, so drop it last. */ + put_filesystem(type); } } @@ -442,24 +447,17 @@ static void kill_super_notify(struct super_block *sb) if (sb->s_flags & SB_DEAD) return; - /* - * Remove it from @fs_supers so it isn't found by new - * sget{_fc}() walkers anymore. Any concurrent mounter still - * managing to grab a temporary reference is guaranteed to - * already see SB_DYING and will wait until we notify them about - * SB_DEAD. - */ - spin_lock(&sb_lock); - hlist_del_init(&sb->s_instances); - spin_unlock(&sb_lock); - /* * Let concurrent mounts know that this thing is really dead. - * We don't need @sb->s_umount here as every concurrent caller - * will see SB_DYING and either discard the superblock or wait - * for SB_DEAD. + * sget{_fc}() skips SB_DEAD superblocks and calls test() under + * sb_lock, so set it under sb_lock: once we return no test() + * runs on this superblock anymore and none will start. Everyone + * else already saw SB_DYING and either discarded the superblock + * or waits for SB_DEAD. */ + spin_lock(&sb_lock); super_wake(sb, SB_DEAD); + spin_unlock(&sb_lock); } /** @@ -490,7 +488,6 @@ void deactivate_locked_super(struct super_block *s) list_lru_destroy(&s->s_dentry_lru); list_lru_destroy(&s->s_inode_lru); - put_filesystem(fs); put_super(s); } else { super_unlock_excl(s); @@ -713,12 +710,12 @@ void generic_shutdown_super(struct super_block *sb) } /* * Broadcast to everyone that grabbed a temporary reference to this - * superblock before we removed it from @fs_supers that the superblock - * is dying. Every walker of @fs_supers outside of sget{_fc}() will now - * discard this superblock and treat it as dead. + * superblock that it is dying. Every walker of @fs_supers outside + * of sget{_fc}() will now discard this superblock and treat it as + * dead. * - * We leave the superblock on @fs_supers so it can be found by - * sget{_fc}() until we passed sb->kill_sb(). + * sget{_fc}() keeps finding the superblock until SB_DEAD is set, so + * a concurrent mounter waits until we passed sb->kill_sb(). */ super_wake(sb, SB_DYING); super_unlock_excl(sb); @@ -796,6 +793,9 @@ struct super_block *sget_fc(struct fs_context *fc, spin_lock(&sb_lock); if (test) { hlist_for_each_entry(old, &fc->fs_type->fs_supers, s_instances) { + /* Only unlinked at the last passive reference. */ + if (smp_load_acquire(&old->s_flags) & SB_DEAD) + continue; if (test(old, fc)) goto share_extant_sb; } @@ -879,6 +879,9 @@ struct super_block *sget(struct file_system_type *type, spin_lock(&sb_lock); if (test) { hlist_for_each_entry(old, &type->fs_supers, s_instances) { + /* Only unlinked at the last passive reference. */ + if (smp_load_acquire(&old->s_flags) & SB_DEAD) + continue; if (!test(old, data)) continue; if (user_ns != old->s_user_ns) { diff --git a/fs/tracefs/event_inode.c b/fs/tracefs/event_inode.c index c4a9009cb56a..df1d571144d6 100644 --- a/fs/tracefs/event_inode.c +++ b/fs/tracefs/event_inode.c @@ -125,7 +125,17 @@ static inline void put_ei(struct eventfs_inode *ei) static inline void free_ei(struct eventfs_inode *ei) { if (ei) { + /* The ei should have no children if it is being freed. */ + WARN_ON_ONCE(!list_empty(&ei->children)); ei->is_freed = 1; + /* + * The SRCU iteration has a smp_rmb() to make sure it + * sees a child (that may have already been freed) + * before it reads is_free. If is_free is set, it must + * not use the child it acquired from ei->children, as + * the list may be used for SRCU. + */ + smp_wmb(); put_ei(ei); } } @@ -719,6 +729,20 @@ static int eventfs_iterate(struct file *file, struct dir_context *ctx) list_for_each_entry_srcu(ei_child, &ei->children, list, srcu_read_lock_held(&eventfs_srcu)) { + /* + * If the ei is being freed, then the ei->children may be + * being used as the rcu list, which means the next element + * may be garbage. The ei->is_free is set before switching + * the ei->children over to ei->rcu. The read memory barrier + * here makes sure the ei_child is read before is_free is + * updated. + * + * Matches the smp_wmb() in free_ei() + */ + smp_rmb(); + if (ei->is_freed) + return -EINVAL; + if (c > 0) { c--; continue; diff --git a/fs/tracefs/internal.h b/fs/tracefs/internal.h index a4a7f8431aff..c61481d04c8e 100644 --- a/fs/tracefs/internal.h +++ b/fs/tracefs/internal.h @@ -46,11 +46,11 @@ struct eventfs_attr { * @ino: The saved inode number */ struct eventfs_inode { + struct list_head list; union { - struct list_head list; + struct list_head children; struct rcu_head rcu; }; - struct list_head children; const struct eventfs_entry *entries; const char *name; struct eventfs_attr *entry_attrs; diff --git a/fs/ubifs/super.c b/fs/ubifs/super.c index 3409488d39ba..78f31bddff74 100644 --- a/fs/ubifs/super.c +++ b/fs/ubifs/super.c @@ -2246,7 +2246,7 @@ static int ubifs_fill_super(struct super_block *sb, void *data, int silent) goto out_umount; } - import_uuid(&sb->s_uuid, c->uuid); + super_set_uuid(sb, c->uuid, sizeof(c->uuid)); mutex_unlock(&c->umount_mutex); return 0; diff --git a/fs/udf/inode.c b/fs/udf/inode.c index 6e40689845bc..d1bed7927c2f 100644 --- a/fs/udf/inode.c +++ b/fs/udf/inode.c @@ -48,13 +48,25 @@ #define FE_DELETE_PERMS (FE_PERM_U_DELETE | FE_PERM_G_DELETE | \ FE_PERM_O_DELETE) -struct udf_map_rq; +#define UDF_MAP_CREATE 0x01 /* Mapping can allocate new blocks */ +#define UDF_MAP_NOPREALLOC 0x02 /* Do not preallocate blocks */ + +#define UDF_BLK_MAPPED 0x01 /* Block was successfully mapped */ +#define UDF_BLK_NEW 0x02 /* Block was freshly allocated */ + +struct udf_map_rq { + sector_t lblk; + udf_pblk_t pblk; + int iflags; /* UDF_MAP_ flags determining behavior */ + int oflags; /* UDF_BLK_ flags reporting results */ +}; static umode_t udf_convert_permissions(struct fileEntry *); static int udf_update_inode(struct inode *, int); static int udf_sync_inode(struct inode *inode); static int udf_alloc_i_data(struct inode *inode, size_t size); static int inode_getblk(struct inode *inode, struct udf_map_rq *map); +static int udf_map_block(struct inode *inode, struct udf_map_rq *map); static int udf_insert_aext(struct inode *, struct extent_position, struct kernel_lb_addr, uint32_t); static void udf_split_extents(struct inode *, int *, int, udf_pblk_t, @@ -343,6 +355,10 @@ int udf_expand_file_adinicb(struct inode *inode) { struct folio *folio; struct udf_inode_info *iinfo = UDF_I(inode); + struct udf_map_rq map = { + .lblk = 0, + .iflags = UDF_MAP_CREATE, + }; int err; WARN_ON_ONCE(!inode_is_locked(inode)); @@ -372,20 +388,27 @@ int udf_expand_file_adinicb(struct inode *inode) iinfo->i_alloc_type = ICBTAG_FLAG_AD_SHORT; else iinfo->i_alloc_type = ICBTAG_FLAG_AD_LONG; + up_write(&iinfo->i_data_sem); + + /* Allocate the block underlying the data */ + err = udf_map_block(inode, &map); + if (err < 0) + goto restore; + folio_mark_dirty(folio); folio_unlock(folio); - up_write(&iinfo->i_data_sem); err = filemap_fdatawrite(inode->i_mapping); if (err) { /* Restore everything back so that we don't lose data... */ folio_lock(folio); +restore: down_write(&iinfo->i_data_sem); memcpy_from_folio(iinfo->i_data + iinfo->i_lenEAttr, folio, 0, inode->i_size); - folio_unlock(folio); iinfo->i_alloc_type = ICBTAG_FLAG_AD_IN_ICB; iinfo->i_lenAlloc = inode->i_size; up_write(&iinfo->i_data_sem); + folio_unlock(folio); } folio_put(folio); mark_inode_dirty(inode); @@ -393,19 +416,6 @@ int udf_expand_file_adinicb(struct inode *inode) return err; } -#define UDF_MAP_CREATE 0x01 /* Mapping can allocate new blocks */ -#define UDF_MAP_NOPREALLOC 0x02 /* Do not preallocate blocks */ - -#define UDF_BLK_MAPPED 0x01 /* Block was successfully mapped */ -#define UDF_BLK_NEW 0x02 /* Block was freshly allocated */ - -struct udf_map_rq { - sector_t lblk; - udf_pblk_t pblk; - int iflags; /* UDF_MAP_ flags determining behavior */ - int oflags; /* UDF_BLK_ flags reporting results */ -}; - static int udf_map_block(struct inode *inode, struct udf_map_rq *map) { int ret; diff --git a/fs/userfaultfd.c b/fs/userfaultfd.c index 0ebdf3c2c9ee..bd64d6b9cf89 100644 --- a/fs/userfaultfd.c +++ b/fs/userfaultfd.c @@ -138,6 +138,13 @@ bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma) return ctx->features & UFFD_FEATURE_WP_UNPOPULATED; } +bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma) +{ + struct userfaultfd_ctx *uffd_ctx = vma->vm_userfaultfd_ctx.ctx; + + return uffd_ctx && (uffd_ctx->features & UFFD_FEATURE_EVENT_REMAP) == 0; +} + static void userfaultfd_set_vm_flags(struct vm_area_struct *vma, vm_flags_t flags) { diff --git a/fs/verity/Kconfig b/fs/verity/Kconfig index e1036e535352..5ff8d9a9052b 100644 --- a/fs/verity/Kconfig +++ b/fs/verity/Kconfig @@ -2,6 +2,9 @@ config FS_VERITY bool "FS Verity (read-only file-based authenticity protection)" + # Filesystems cache the Merkle tree at a 64K aligned offset in the + # pagecache. That approach assumes the page size is at most 64K. + depends on !PAGE_SIZE_256KB select CRYPTO select CRYPTO_HASH_INFO # SHA-256 is implied as it's intended to be the default hash algorithm. diff --git a/fs/verity/open.c b/fs/verity/open.c index 6c31a871b84b..5342a8e1f781 100644 --- a/fs/verity/open.c +++ b/fs/verity/open.c @@ -393,14 +393,6 @@ int __fsverity_file_open(struct inode *inode, struct file *filp) } EXPORT_SYMBOL_GPL(__fsverity_file_open); -int __fsverity_prepare_setattr(struct dentry *dentry, struct iattr *attr) -{ - if (attr->ia_valid & ATTR_SIZE) - return -EPERM; - return 0; -} -EXPORT_SYMBOL_GPL(__fsverity_prepare_setattr); - void __fsverity_cleanup_inode(struct inode *inode) { fsverity_free_info(inode->i_verity_info); diff --git a/fs/xfs/libxfs/xfs_alloc.c b/fs/xfs/libxfs/xfs_alloc.c index ad2fa3c26f8a..0383216e39dc 100644 --- a/fs/xfs/libxfs/xfs_alloc.c +++ b/fs/xfs/libxfs/xfs_alloc.c @@ -26,6 +26,7 @@ #include "xfs_ag.h" #include "xfs_ag_resv.h" #include "xfs_bmap.h" +#include "xfs_health.h" struct kmem_cache *xfs_extfree_item_cache; @@ -3240,18 +3241,43 @@ xfs_alloc_read_agf( set_bit(XFS_AGSTATE_AGF_INIT, &pag->pag_opstate); } + #ifdef DEBUG - else if (!xfs_is_shutdown(pag->pag_mount)) { - ASSERT(pag->pagf_freeblks == be32_to_cpu(agf->agf_freeblks)); - ASSERT(pag->pagf_btreeblks == be32_to_cpu(agf->agf_btreeblks)); - ASSERT(pag->pagf_flcount == be32_to_cpu(agf->agf_flcount)); - ASSERT(pag->pagf_longest == be32_to_cpu(agf->agf_longest)); - ASSERT(pag->pagf_levels[XFS_BTNUM_BNOi] == - be32_to_cpu(agf->agf_levels[XFS_BTNUM_BNOi])); - ASSERT(pag->pagf_levels[XFS_BTNUM_CNTi] == - be32_to_cpu(agf->agf_levels[XFS_BTNUM_CNTi])); + /* + * It's possible for the AGF to be out of sync if the block device is + * silently dropping writes. This can happen in fstests with dmflakey + * enabled, which allows the buffer to be cleaned and reclaimed by + * memory pressure and then re-read from disk here. We will get a + * stale version of the AGF from disk, and nothing good can happen from + * here. Hence if we detect this situation, immediately shut down the + * filesystem. + * + * This can also happen if we are already in the middle of a forced + * shutdown, so don't bother checking if we are already shut down. + */ + if (!xfs_is_shutdown(pag->pag_mount)) { + bool ok = true; + + ok &= pag->pagf_freeblks == be32_to_cpu(agf->agf_freeblks); + ok &= pag->pagf_freeblks == be32_to_cpu(agf->agf_freeblks); + ok &= pag->pagf_btreeblks == be32_to_cpu(agf->agf_btreeblks); + ok &= pag->pagf_flcount == be32_to_cpu(agf->agf_flcount); + ok &= pag->pagf_longest == be32_to_cpu(agf->agf_longest); + ok &= pag->pagf_levels[XFS_BTNUM_BNOi] == + be32_to_cpu(agf->agf_levels[XFS_BTNUM_BNOi]); + ok &= pag->pagf_levels[XFS_BTNUM_CNTi] == + be32_to_cpu(agf->agf_levels[XFS_BTNUM_CNTi]); + + if (XFS_IS_CORRUPT(pag->pag_mount, !ok)) { + xfs_ag_mark_sick(pag, XFS_SICK_AG_AGF); + xfs_trans_brelse(tp, agfbp); + xfs_force_shutdown(pag->pag_mount, + SHUTDOWN_CORRUPT_ONDISK); + return -EFSCORRUPTED; + } } -#endif +#endif /* DEBUG */ + if (agfbpp) *agfbpp = agfbp; else diff --git a/fs/xfs/libxfs/xfs_attr.c b/fs/xfs/libxfs/xfs_attr.c index 50172bb8026f..024e9734c0b6 100644 --- a/fs/xfs/libxfs/xfs_attr.c +++ b/fs/xfs/libxfs/xfs_attr.c @@ -49,7 +49,6 @@ STATIC int xfs_attr_shortform_addname(xfs_da_args_t *args); */ STATIC int xfs_attr_leaf_get(xfs_da_args_t *args); STATIC int xfs_attr_leaf_removename(xfs_da_args_t *args); -STATIC int xfs_attr_leaf_hasname(struct xfs_da_args *args, struct xfs_buf **bp); /* * Internal routines when attribute list is more than one block. @@ -917,11 +916,11 @@ xfs_attr_lookup( return xfs_attr_sf_findname(args, NULL, NULL); if (xfs_attr_is_leaf(dp)) { - error = xfs_attr_leaf_hasname(args, &bp); - - if (bp) - xfs_trans_brelse(args->trans, bp); - + error = xfs_attr3_leaf_read(args->trans, args->dp, 0, &bp); + if (error) + return error; + error = xfs_attr3_leaf_lookup_int(bp, args); + xfs_trans_brelse(args->trans, bp); return error; } @@ -1215,27 +1214,6 @@ xfs_attr_shortform_addname( * External routines when attribute list is one block *========================================================================*/ -/* - * Return EEXIST if attr is found, or ENOATTR if not - */ -STATIC int -xfs_attr_leaf_hasname( - struct xfs_da_args *args, - struct xfs_buf **bp) -{ - int error = 0; - - error = xfs_attr3_leaf_read(args->trans, args->dp, 0, bp); - if (error) - return error; - - error = xfs_attr3_leaf_lookup_int(*bp, args); - if (error != -ENOATTR && error != -EEXIST) - xfs_trans_brelse(args->trans, *bp); - - return error; -} - /* * Remove a name from the leaf attribute list structure * @@ -1246,25 +1224,22 @@ STATIC int xfs_attr_leaf_removename( struct xfs_da_args *args) { - struct xfs_inode *dp; - struct xfs_buf *bp; + struct xfs_inode *dp = args->dp; int error, forkoff; + struct xfs_buf *bp; trace_xfs_attr_leaf_removename(args); - /* - * Remove the attribute. - */ - dp = args->dp; - - error = xfs_attr_leaf_hasname(args, &bp); - if (error == -ENOATTR) { + error = xfs_attr3_leaf_read(args->trans, args->dp, 0, &bp); + if (error) + return error; + error = xfs_attr3_leaf_lookup_int(bp, args); + if (error != -EEXIST) { xfs_trans_brelse(args->trans, bp); - if (args->op_flags & XFS_DA_OP_RECOVERY) + if (error == -ENOATTR && (args->op_flags & XFS_DA_OP_RECOVERY)) return 0; return error; - } else if (error != -EEXIST) - return error; + } xfs_attr3_leaf_remove(bp, args); @@ -1288,23 +1263,20 @@ xfs_attr_leaf_removename( * Returns 0 on successful retrieval, otherwise an error. */ STATIC int -xfs_attr_leaf_get(xfs_da_args_t *args) +xfs_attr_leaf_get( + struct xfs_da_args *args) { - struct xfs_buf *bp; - int error; + struct xfs_buf *bp; + int error; trace_xfs_attr_leaf_get(args); - error = xfs_attr_leaf_hasname(args, &bp); - - if (error == -ENOATTR) { - xfs_trans_brelse(args->trans, bp); - return error; - } else if (error != -EEXIST) + error = xfs_attr3_leaf_read(args->trans, args->dp, 0, &bp); + if (error) return error; - - - error = xfs_attr3_leaf_getvalue(bp, args); + error = xfs_attr3_leaf_lookup_int(bp, args); + if (error == -EEXIST) + error = xfs_attr3_leaf_getvalue(bp, args); xfs_trans_brelse(args->trans, bp); return error; } diff --git a/fs/xfs/libxfs/xfs_ialloc.c b/fs/xfs/libxfs/xfs_ialloc.c index b83e54c70906..dee46010702d 100644 --- a/fs/xfs/libxfs/xfs_ialloc.c +++ b/fs/xfs/libxfs/xfs_ialloc.c @@ -27,6 +27,7 @@ #include "xfs_log.h" #include "xfs_rmap.h" #include "xfs_ag.h" +#include "xfs_health.h" /* * Lookup a record by ino in the btree given by cur. @@ -2634,12 +2635,35 @@ xfs_ialloc_read_agi( set_bit(XFS_AGSTATE_AGI_INIT, &pag->pag_opstate); } +#ifdef DEBUG /* - * It's possible for these to be out of sync if - * we are in the middle of a forced shutdown. + * It's possible for the AGF to be out of sync if the block device is + * silently dropping writes. This can happen in fstests with dmflakey + * enabled, which allows the buffer to be cleaned and reclaimed by + * memory pressure and then re-read from disk here. We will get a + * stale version of the AGF from disk, and nothing good can happen from + * here. Hence if we detect this situation, immediately shut down the + * filesystem. + * + * This can also happen if we are already in the middle of a forced + * shutdown, so don't bother checking if we are already shut down. */ - ASSERT(pag->pagi_freecount == be32_to_cpu(agi->agi_freecount) || - xfs_is_shutdown(pag->pag_mount)); + if (!xfs_is_shutdown(pag->pag_mount)) { + bool ok = true; + + ok &= pag->pagi_freecount == be32_to_cpu(agi->agi_freecount); + ok &= pag->pagi_count == be32_to_cpu(agi->agi_count); + + if (XFS_IS_CORRUPT(pag->pag_mount, !ok)) { + xfs_ag_mark_sick(pag, XFS_SICK_AG_AGI); + xfs_trans_brelse(tp, agibp); + xfs_force_shutdown(pag->pag_mount, + SHUTDOWN_CORRUPT_ONDISK); + return -EFSCORRUPTED; + } + } +#endif /* DEBUG */ + if (agibpp) *agibpp = agibp; else diff --git a/fs/xfs/xfs_buf.c b/fs/xfs/xfs_buf.c index 1181108f8074..4c197c366bc8 100644 --- a/fs/xfs/xfs_buf.c +++ b/fs/xfs/xfs_buf.c @@ -169,7 +169,7 @@ xfs_buf_stale( atomic_set(&bp->b_lru_ref, 0); if (!(bp->b_state & XFS_BSTATE_DISPOSE) && - (list_lru_del(&bp->b_target->bt_lru, &bp->b_lru))) + (list_lru_del_obj(&bp->b_target->bt_lru, &bp->b_lru))) atomic_dec(&bp->b_hold); ASSERT(atomic_read(&bp->b_hold) >= 1); @@ -1047,7 +1047,7 @@ xfs_buf_rele( * buffer for the LRU and clear the (now stale) dispose list * state flag */ - if (list_lru_add(&bp->b_target->bt_lru, &bp->b_lru)) { + if (list_lru_add_obj(&bp->b_target->bt_lru, &bp->b_lru)) { bp->b_state &= ~XFS_BSTATE_DISPOSE; atomic_inc(&bp->b_hold); } @@ -1060,7 +1060,7 @@ xfs_buf_rele( * was on was the disposal list */ if (!(bp->b_state & XFS_BSTATE_DISPOSE)) { - list_lru_del(&bp->b_target->bt_lru, &bp->b_lru); + list_lru_del_obj(&bp->b_target->bt_lru, &bp->b_lru); } else { ASSERT(list_empty(&bp->b_lru)); } diff --git a/fs/xfs/xfs_buf_item.c b/fs/xfs/xfs_buf_item.c index 72d6e2dd5ef5..a5f3408e213d 100644 --- a/fs/xfs/xfs_buf_item.c +++ b/fs/xfs/xfs_buf_item.c @@ -32,6 +32,63 @@ static inline struct xfs_buf_log_item *BUF_ITEM(struct xfs_log_item *lip) return container_of(lip, struct xfs_buf_log_item, bli_item); } +static void +xfs_buf_item_get_format( + struct xfs_buf_log_item *bip, + int count) +{ + ASSERT(bip->bli_formats == NULL); + bip->bli_format_count = count; + + if (count == 1) { + bip->bli_formats = &bip->__bli_format; + return; + } + + bip->bli_formats = kmem_zalloc(count * sizeof(struct xfs_buf_log_format), + 0); +} + +static void +xfs_buf_item_free_format( + struct xfs_buf_log_item *bip) +{ + if (bip->bli_formats != &bip->__bli_format) { + kmem_free(bip->bli_formats); + bip->bli_formats = NULL; + } +} + +static void +xfs_buf_item_free( + struct xfs_buf_log_item *bip) +{ + xfs_buf_item_free_format(bip); + kmem_free(bip->bli_item.li_lv_shadow); + kmem_cache_free(xfs_buf_item_cache, bip); +} + +/* + * xfs_buf_item_relse() is called when the buf log item is no longer needed. + */ +static void +xfs_buf_item_relse( + struct xfs_buf_log_item *bip) +{ + struct xfs_buf *bp = bip->bli_buf; + + trace_xfs_buf_item_relse(bp, _RET_IP_); + + ASSERT(!test_bit(XFS_LI_IN_AIL, &bip->bli_item.li_flags)); + + if (atomic_read(&bip->bli_refcount)) + return; + + bp->b_log_item = NULL; + xfs_buf_rele(bp); + xfs_buf_item_free(bip); +} + /* Is this log iovec plausibly large enough to contain the buffer log format? */ bool xfs_buf_log_check_iovec( @@ -484,6 +541,42 @@ xfs_buf_item_pin( atomic_inc(&bip->bli_buf->b_pin_count); } +/* + * For a stale BLI, process all the necessary completions that must be + * performed when the final BLI reference goes away. The buffer will be + * referenced and locked here - we return to the caller with the buffer still + * referenced and locked for them to finalise processing of the buffer. + */ +static void +xfs_buf_item_finish_stale( + struct xfs_buf_log_item *bip) +{ + struct xfs_buf *bp = bip->bli_buf; + struct xfs_log_item *lip = &bip->bli_item; + + ASSERT(bip->bli_flags & XFS_BLI_STALE); + ASSERT(xfs_buf_islocked(bp)); + ASSERT(bp->b_flags & XBF_STALE); + ASSERT(bip->__bli_format.blf_flags & XFS_BLF_CANCEL); + ASSERT(list_empty(&lip->li_trans)); + ASSERT(!bp->b_transp); + + if (bip->bli_flags & XFS_BLI_STALE_INODE) { + xfs_buf_item_done(bp); + xfs_buf_inode_iodone(bp); + ASSERT(list_empty(&bp->b_li_list)); + return; + } + + /* + * We may or may not be on the AIL here, xfs_trans_ail_delete() will do + * the right thing regardless of the situation in which we are called. + */ + xfs_trans_ail_delete(lip, SHUTDOWN_LOG_IO_ERROR); + xfs_buf_item_relse(bip); + ASSERT(bp->b_log_item == NULL); +} + /* * This is called to unpin the buffer associated with the buf log item which was * previously pinned with a call to xfs_buf_item_pin(). We enter this function @@ -533,13 +626,6 @@ xfs_buf_item_unpin( } if (stale) { - ASSERT(bip->bli_flags & XFS_BLI_STALE); - ASSERT(xfs_buf_islocked(bp)); - ASSERT(bp->b_flags & XBF_STALE); - ASSERT(bip->__bli_format.blf_flags & XFS_BLF_CANCEL); - ASSERT(list_empty(&lip->li_trans)); - ASSERT(!bp->b_transp); - trace_xfs_buf_item_unpin_stale(bip); /* @@ -550,22 +636,7 @@ xfs_buf_item_unpin( * processing is complete. */ xfs_buf_rele(bp); - - /* - * If we get called here because of an IO error, we may or may - * not have the item on the AIL. xfs_trans_ail_delete() will - * take care of that situation. xfs_trans_ail_delete() drops - * the AIL lock. - */ - if (bip->bli_flags & XFS_BLI_STALE_INODE) { - xfs_buf_item_done(bp); - xfs_buf_inode_iodone(bp); - ASSERT(list_empty(&bp->b_li_list)); - } else { - xfs_trans_ail_delete(lip, SHUTDOWN_LOG_IO_ERROR); - xfs_buf_item_relse(bp); - ASSERT(bp->b_log_item == NULL); - } + xfs_buf_item_finish_stale(bip); xfs_buf_relse(bp); return; } @@ -638,43 +709,42 @@ xfs_buf_item_push( * Drop the buffer log item refcount and take appropriate action. This helper * determines whether the bli must be freed or not, since a decrement to zero * does not necessarily mean the bli is unused. - * - * Return true if the bli is freed, false otherwise. */ -bool +void xfs_buf_item_put( struct xfs_buf_log_item *bip) { - struct xfs_log_item *lip = &bip->bli_item; - bool aborted; - bool dirty; + + ASSERT(xfs_buf_islocked(bip->bli_buf)); /* drop the bli ref and return if it wasn't the last one */ if (!atomic_dec_and_test(&bip->bli_refcount)) - return false; + return; - /* - * We dropped the last ref and must free the item if clean or aborted. - * If the bli is dirty and non-aborted, the buffer was clean in the - * transaction but still awaiting writeback from previous changes. In - * that case, the bli is freed on buffer writeback completion. - */ - aborted = test_bit(XFS_LI_ABORTED, &lip->li_flags) || - xlog_is_shutdown(lip->li_log); - dirty = bip->bli_flags & XFS_BLI_DIRTY; - if (dirty && !aborted) - return false; + /* If the BLI is in the AIL, then it is still dirty and in use */ + if (test_bit(XFS_LI_IN_AIL, &bip->bli_item.li_flags)) { + ASSERT(bip->bli_flags & XFS_BLI_DIRTY); + return; + } /* - * The bli is aborted or clean. An aborted item may be in the AIL - * regardless of dirty state. For example, consider an aborted - * transaction that invalidated a dirty bli and cleared the dirty - * state. + * In shutdown conditions, we can be asked to free a dirty BLI that + * isn't in the AIL. This can occur due to a checkpoint aborting a BLI + * instead of inserting it into the AIL at checkpoint IO completion. If + * there's another bli reference (e.g. a btree cursor holds a clean + * reference) and it is released via xfs_trans_brelse(), we can get here + * with that aborted, dirty BLI. In this case, it is safe to free the + * dirty BLI immediately, as it is not in the AIL and there are no + * other references to it. + * + * We should never get here with a stale BLI via that path as + * xfs_trans_brelse() specifically holds onto stale buffers rather than + * releasing them. */ - if (aborted) - xfs_trans_ail_delete(lip, 0); - xfs_buf_item_relse(bip->bli_buf); - return true; + ASSERT(!(bip->bli_flags & XFS_BLI_DIRTY) || + test_bit(XFS_LI_ABORTED, &bip->bli_item.li_flags)); + ASSERT(!(bip->bli_flags & XFS_BLI_STALE)); + xfs_buf_item_relse(bip); } /* @@ -695,6 +765,15 @@ xfs_buf_item_put( * if necessary but do not unlock the buffer. This is for support of * xfs_trans_bhold(). Make sure the XFS_BLI_HOLD field is cleared if we don't * free the item. + * + * If the XFS_BLI_STALE flag is set, the last reference to the BLI *must* + * perform a completion abort of any objects attached to the buffer for IO + * tracking purposes. This generally only happens in shutdown situations, + * normally xfs_buf_item_unpin() will drop the last BLI reference and perform + * completion processing. However, because transaction completion can race with + * checkpoint completion during a shutdown, this release context may end up + * being the last active reference to the BLI and so needs to perform this + * cleanup. */ STATIC void xfs_buf_item_release( @@ -702,18 +781,19 @@ xfs_buf_item_release( { struct xfs_buf_log_item *bip = BUF_ITEM(lip); struct xfs_buf *bp = bip->bli_buf; - bool released; bool hold = bip->bli_flags & XFS_BLI_HOLD; bool stale = bip->bli_flags & XFS_BLI_STALE; -#if defined(DEBUG) || defined(XFS_WARN) - bool ordered = bip->bli_flags & XFS_BLI_ORDERED; - bool dirty = bip->bli_flags & XFS_BLI_DIRTY; bool aborted = test_bit(XFS_LI_ABORTED, &lip->li_flags); + bool dirty = bip->bli_flags & XFS_BLI_DIRTY; +#if defined(DEBUG) || defined(XFS_WARN) + bool ordered = bip->bli_flags & XFS_BLI_ORDERED; #endif trace_xfs_buf_item_release(bip); + ASSERT(xfs_buf_islocked(bp)); + /* * The bli dirty state should match whether the blf has logged segments * except for ordered buffers, where only the bli should be dirty. @@ -729,16 +809,56 @@ xfs_buf_item_release( bp->b_transp = NULL; bip->bli_flags &= ~(XFS_BLI_LOGGED | XFS_BLI_HOLD | XFS_BLI_ORDERED); + /* If there are other references, then we have nothing to do. */ + if (!atomic_dec_and_test(&bip->bli_refcount)) + goto out_release; + + /* + * Stale buffer completion frees the BLI, unlocks and releases the + * buffer. Neither the BLI or buffer are safe to reference after this + * call, so there's nothing more we need to do here. + * + * If we get here with a stale buffer and references to the BLI remain, + * we must not unlock the buffer as the last BLI reference owns lock + * context, not us. + */ + if (stale) { + xfs_buf_item_finish_stale(bip); + xfs_buf_relse(bp); + ASSERT(!hold); + return; + } + + /* + * Dirty or clean, aborted items are done and need to be removed from + * the AIL and released. This frees the BLI, but leaves the buffer + * locked and referenced. + */ + if (aborted || xlog_is_shutdown(lip->li_log)) { + ASSERT(list_empty(&bip->bli_buf->b_li_list)); + xfs_buf_item_done(bp); + goto out_release; + } + + /* + * Clean, unreferenced BLIs can be immediately freed, leaving the buffer + * locked and referenced. + * + * Dirty, unreferenced BLIs *must* be in the AIL awaiting writeback. + */ + if (!dirty) + xfs_buf_item_relse(bip); + else + ASSERT(test_bit(XFS_LI_IN_AIL, &lip->li_flags)); + + /* Not safe to reference the BLI from here */ +out_release: /* - * Unref the item and unlock the buffer unless held or stale. Stale - * buffers remain locked until final unpin unless the bli is freed by - * the unref call. The latter implies shutdown because buffer - * invalidation dirties the bli and transaction. + * If we get here with a stale buffer, we must not unlock the + * buffer as the last BLI reference owns lock context, not us. */ - released = xfs_buf_item_put(bip); - if (hold || (stale && !released)) + if (stale || hold) return; - ASSERT(!stale || aborted); xfs_buf_relse(bp); } @@ -824,33 +944,6 @@ static const struct xfs_item_ops xfs_buf_item_ops = { .iop_push = xfs_buf_item_push, }; -STATIC void -xfs_buf_item_get_format( - struct xfs_buf_log_item *bip, - int count) -{ - ASSERT(bip->bli_formats == NULL); - bip->bli_format_count = count; - - if (count == 1) { - bip->bli_formats = &bip->__bli_format; - return; - } - - bip->bli_formats = kmem_zalloc(count * sizeof(struct xfs_buf_log_format), - 0); -} - -STATIC void -xfs_buf_item_free_format( - struct xfs_buf_log_item *bip) -{ - if (bip->bli_formats != &bip->__bli_format) { - kmem_free(bip->bli_formats); - bip->bli_formats = NULL; - } -} - /* * Allocate a new buf log item to go with the given buffer. * Set the buffer's b_log_item field to point to the new @@ -1072,34 +1165,6 @@ xfs_buf_item_dirty_format( return false; } -STATIC void -xfs_buf_item_free( - struct xfs_buf_log_item *bip) -{ - xfs_buf_item_free_format(bip); - kmem_free(bip->bli_item.li_lv_shadow); - kmem_cache_free(xfs_buf_item_cache, bip); -} - -/* - * xfs_buf_item_relse() is called when the buf log item is no longer needed. - */ -void -xfs_buf_item_relse( - struct xfs_buf *bp) -{ - struct xfs_buf_log_item *bip = bp->b_log_item; - - trace_xfs_buf_item_relse(bp, _RET_IP_); - ASSERT(!test_bit(XFS_LI_IN_AIL, &bip->bli_item.li_flags)); - - if (atomic_read(&bip->bli_refcount)) - return; - bp->b_log_item = NULL; - xfs_buf_rele(bp); - xfs_buf_item_free(bip); -} - void xfs_buf_item_done( struct xfs_buf *bp) @@ -1119,5 +1184,5 @@ xfs_buf_item_done( xfs_trans_ail_delete(&bp->b_log_item->bli_item, (bp->b_flags & _XBF_LOGRECOVERY) ? 0 : SHUTDOWN_CORRUPT_INCORE); - xfs_buf_item_relse(bp); + xfs_buf_item_relse(bp->b_log_item); } diff --git a/fs/xfs/xfs_buf_item.h b/fs/xfs/xfs_buf_item.h index 7ddda0aa9dac..141c3780a36b 100644 --- a/fs/xfs/xfs_buf_item.h +++ b/fs/xfs/xfs_buf_item.h @@ -49,8 +49,7 @@ struct xfs_buf_log_item { int xfs_buf_item_init(struct xfs_buf *, struct xfs_mount *); void xfs_buf_item_done(struct xfs_buf *bp); -void xfs_buf_item_relse(struct xfs_buf *); -bool xfs_buf_item_put(struct xfs_buf_log_item *); +void xfs_buf_item_put(struct xfs_buf_log_item *bip); void xfs_buf_item_log(struct xfs_buf_log_item *, uint, uint); bool xfs_buf_item_dirty_format(struct xfs_buf_log_item *); void xfs_buf_inode_iodone(struct xfs_buf *); diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c index 5fb8663b9093..dd91bdad5d72 100644 --- a/fs/xfs/xfs_dquot.c +++ b/fs/xfs/xfs_dquot.c @@ -1065,7 +1065,7 @@ xfs_qm_dqput( struct xfs_quotainfo *qi = dqp->q_mount->m_quotainfo; trace_xfs_dqput_free(dqp); - if (list_lru_add(&qi->qi_lru, &dqp->q_lru)) + if (list_lru_add_obj(&qi->qi_lru, &dqp->q_lru)) XFS_STATS_INC(dqp->q_mount, xs_qm_dquot_unused); } xfs_dqunlock(dqp); diff --git a/fs/xfs/xfs_icache.c b/fs/xfs/xfs_icache.c index be44c7477c16..ea743699ae6f 100644 --- a/fs/xfs/xfs_icache.c +++ b/fs/xfs/xfs_icache.c @@ -879,7 +879,15 @@ xfs_reclaim_inode( */ if (xlog_is_shutdown(ip->i_mount->m_log)) { xfs_iunpin_wait(ip); + /* + * Avoid a ABBA deadlock on the inode cluster buffer vs + * concurrent xfs_ifree_cluster() trying to mark the inode + * stale. We don't need the inode locked to run the flush abort + * code, but the flush abort needs to lock the cluster buffer. + */ + xfs_iunlock(ip, XFS_ILOCK_EXCL); xfs_iflush_shutdown_abort(ip); + xfs_ilock(ip, XFS_ILOCK_EXCL); goto reclaim; } if (xfs_ipincount(ip)) diff --git a/fs/xfs/xfs_inode.c b/fs/xfs/xfs_inode.c index e1adad91dba6..a9fa5b1fd208 100644 --- a/fs/xfs/xfs_inode.c +++ b/fs/xfs/xfs_inode.c @@ -2235,7 +2235,7 @@ xfs_ifree_mark_inode_stale( iip = ip->i_itemp; if (__xfs_iflags_test(ip, XFS_IFLUSHING)) { ASSERT(!list_empty(&iip->ili_item.li_bio_list)); - ASSERT(iip->ili_last_fields); + ASSERT(iip->ili_last_fields || xlog_is_shutdown(mp->m_log)); goto out_iunlock; } diff --git a/fs/xfs/xfs_log_recover.c b/fs/xfs/xfs_log_recover.c index 38ae42bbf9b5..f2f7e35c03cc 100644 --- a/fs/xfs/xfs_log_recover.c +++ b/fs/xfs/xfs_log_recover.c @@ -2701,12 +2701,13 @@ xlog_recover_iunlink_bucket( { struct xfs_mount *mp = pag->pag_mount; struct xfs_inode *prev_ip = NULL; - struct xfs_inode *ip; xfs_agino_t prev_agino, agino; int error = 0; agino = be32_to_cpu(agi->agi_unlinked[bucket]); while (agino != NULLAGINO) { + struct xfs_inode *ip; + error = xfs_iget(mp, NULL, XFS_AGINO_TO_INO(mp, pag->pag_agno, agino), 0, 0, &ip); @@ -2716,11 +2717,11 @@ xlog_recover_iunlink_bucket( ASSERT(VFS_I(ip)->i_nlink == 0); ASSERT(VFS_I(ip)->i_mode != 0); xfs_iflags_clear(ip, XFS_IRECOVERY); - agino = ip->i_next_unlinked; if (prev_ip) { ip->i_prev_unlinked = prev_agino; xfs_irele(prev_ip); + prev_ip = NULL; /* * Ensure the inode is removed from the unlinked list @@ -2732,18 +2733,20 @@ xlog_recover_iunlink_bucket( * complete. */ error = xfs_inodegc_flush(mp); - if (error) - break; + if (error) { + xfs_irele(ip); + return error; + } } prev_agino = agino; + agino = ip->i_next_unlinked; prev_ip = ip; } if (prev_ip) { int error2; - ip->i_prev_unlinked = prev_agino; xfs_irele(prev_ip); error2 = xfs_inodegc_flush(mp); diff --git a/fs/xfs/xfs_mount.c b/fs/xfs/xfs_mount.c index fca502690690..44c686d57e7b 100644 --- a/fs/xfs/xfs_mount.c +++ b/fs/xfs/xfs_mount.c @@ -62,7 +62,7 @@ xfs_uuid_mount( int hole, i; /* Publish UUID in struct super_block */ - uuid_copy(&mp->m_super->s_uuid, uuid); + super_set_uuid(mp->m_super, uuid->b, sizeof(*uuid)); if (xfs_has_nouuid(mp)) return 0; diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c index 086e78a6143a..5ef080c3c734 100644 --- a/fs/xfs/xfs_qm.c +++ b/fs/xfs/xfs_qm.c @@ -171,7 +171,7 @@ xfs_qm_dqpurge( * hits zero, so it really should be on the freelist here. */ ASSERT(!list_empty(&dqp->q_lru)); - list_lru_del(&qi->qi_lru, &dqp->q_lru); + list_lru_del_obj(&qi->qi_lru, &dqp->q_lru); XFS_STATS_DEC(dqp->q_mount, xs_qm_dquot_unused); xfs_qm_dqdestroy(dqp); diff --git a/include/acpi/acpi_bus.h b/include/acpi/acpi_bus.h index fea9ff235f07..36d5a61d04e9 100644 --- a/include/acpi/acpi_bus.h +++ b/include/acpi/acpi_bus.h @@ -568,6 +568,7 @@ void acpi_bus_unregister_driver(struct acpi_driver *driver); int acpi_bus_scan(acpi_handle handle); void acpi_bus_trim(struct acpi_device *start); acpi_status acpi_bus_get_ejd(acpi_handle handle, acpi_handle * ejd); +struct device *acpi_bus_get_primary_device(struct acpi_device *adev); int acpi_match_device_ids(struct acpi_device *device, const struct acpi_device_id *ids); void acpi_set_modalias(struct acpi_device *adev, const char *default_id, @@ -821,6 +822,11 @@ static inline void acpi_put_acpi_dev(struct acpi_device *adev) } #else /* CONFIG_ACPI */ +static inline struct device *acpi_bus_get_primary_device(struct acpi_device *adev) +{ + return NULL; +} + static inline int register_acpi_bus_type(void *bus) { return 0; } static inline int unregister_acpi_bus_type(void *bus) { return 0; } diff --git a/include/clocksource/samsung_pwm.h b/include/clocksource/samsung_pwm.h index 9b435caa95fe..36f6f246e559 100644 --- a/include/clocksource/samsung_pwm.h +++ b/include/clocksource/samsung_pwm.h @@ -15,7 +15,7 @@ * spinlock is not shared between both drivers. */ #ifdef CONFIG_CLKSRC_SAMSUNG_PWM -extern spinlock_t samsung_pwm_lock; +extern raw_spinlock_t samsung_pwm_lock; #endif struct samsung_pwm_variant { diff --git a/include/drm/drm_debugfs.h b/include/drm/drm_debugfs.h index cb2c1956a214..8c1f308ca33b 100644 --- a/include/drm/drm_debugfs.h +++ b/include/drm/drm_debugfs.h @@ -35,7 +35,7 @@ #include #include -#include +#include /** * DRM_DEBUGFS_GPUVA_INFO - &drm_info_list entry to dump a GPU VA space @@ -152,7 +152,7 @@ void drm_debugfs_add_files(struct drm_device *dev, const struct drm_debugfs_info *files, int count); int drm_debugfs_gpuva_info(struct seq_file *m, - struct drm_gpuva_manager *mgr); + struct drm_gpuvm *gpuvm); #else static inline void drm_debugfs_create_files(const struct drm_info_list *files, int count, struct dentry *root, @@ -176,7 +176,7 @@ static inline void drm_debugfs_add_files(struct drm_device *dev, {} static inline int drm_debugfs_gpuva_info(struct seq_file *m, - struct drm_gpuva_manager *mgr) + struct drm_gpuvm *gpuvm) { return 0; } diff --git a/include/drm/drm_gpuva_mgr.h b/include/drm/drm_gpuva_mgr.h deleted file mode 100644 index ed8d50200cc3..000000000000 --- a/include/drm/drm_gpuva_mgr.h +++ /dev/null @@ -1,706 +0,0 @@ -/* SPDX-License-Identifier: GPL-2.0-only */ - -#ifndef __DRM_GPUVA_MGR_H__ -#define __DRM_GPUVA_MGR_H__ - -/* - * Copyright (c) 2022 Red Hat. - * - * Permission is hereby granted, free of charge, to any person obtaining a - * copy of this software and associated documentation files (the "Software"), - * to deal in the Software without restriction, including without limitation - * the rights to use, copy, modify, merge, publish, distribute, sublicense, - * and/or sell copies of the Software, and to permit persons to whom the - * Software is furnished to do so, subject to the following conditions: - * - * The above copyright notice and this permission notice shall be included in - * all copies or substantial portions of the Software. - * - * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR - * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, - * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL - * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR - * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, - * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR - * OTHER DEALINGS IN THE SOFTWARE. - */ - -#include -#include -#include - -#include - -struct drm_gpuva_manager; -struct drm_gpuva_fn_ops; - -/** - * enum drm_gpuva_flags - flags for struct drm_gpuva - */ -enum drm_gpuva_flags { - /** - * @DRM_GPUVA_INVALIDATED: - * - * Flag indicating that the &drm_gpuva's backing GEM is invalidated. - */ - DRM_GPUVA_INVALIDATED = (1 << 0), - - /** - * @DRM_GPUVA_SPARSE: - * - * Flag indicating that the &drm_gpuva is a sparse mapping. - */ - DRM_GPUVA_SPARSE = (1 << 1), - - /** - * @DRM_GPUVA_USERBITS: user defined bits - */ - DRM_GPUVA_USERBITS = (1 << 2), -}; - -/** - * struct drm_gpuva - structure to track a GPU VA mapping - * - * This structure represents a GPU VA mapping and is associated with a - * &drm_gpuva_manager. - * - * Typically, this structure is embedded in bigger driver structures. - */ -struct drm_gpuva { - /** - * @mgr: the &drm_gpuva_manager this object is associated with - */ - struct drm_gpuva_manager *mgr; - - /** - * @flags: the &drm_gpuva_flags for this mapping - */ - enum drm_gpuva_flags flags; - - /** - * @va: structure containing the address and range of the &drm_gpuva - */ - struct { - /** - * @addr: the start address - */ - u64 addr; - - /* - * @range: the range - */ - u64 range; - } va; - - /** - * @gem: structure containing the &drm_gem_object and it's offset - */ - struct { - /** - * @offset: the offset within the &drm_gem_object - */ - u64 offset; - - /** - * @obj: the mapped &drm_gem_object - */ - struct drm_gem_object *obj; - - /** - * @entry: the &list_head to attach this object to a &drm_gem_object - */ - struct list_head entry; - } gem; - - /** - * @rb: structure containing data to store &drm_gpuvas in a rb-tree - */ - struct { - /** - * @rb: the rb-tree node - */ - struct rb_node node; - - /** - * @entry: The &list_head to additionally connect &drm_gpuvas - * in the same order they appear in the interval tree. This is - * useful to keep iterating &drm_gpuvas from a start node found - * through the rb-tree while doing modifications on the rb-tree - * itself. - */ - struct list_head entry; - - /** - * @__subtree_last: needed by the interval tree, holding last-in-subtree - */ - u64 __subtree_last; - } rb; -}; - -int drm_gpuva_insert(struct drm_gpuva_manager *mgr, struct drm_gpuva *va); -void drm_gpuva_remove(struct drm_gpuva *va); - -void drm_gpuva_link(struct drm_gpuva *va); -void drm_gpuva_unlink(struct drm_gpuva *va); - -struct drm_gpuva *drm_gpuva_find(struct drm_gpuva_manager *mgr, - u64 addr, u64 range); -struct drm_gpuva *drm_gpuva_find_first(struct drm_gpuva_manager *mgr, - u64 addr, u64 range); -struct drm_gpuva *drm_gpuva_find_prev(struct drm_gpuva_manager *mgr, u64 start); -struct drm_gpuva *drm_gpuva_find_next(struct drm_gpuva_manager *mgr, u64 end); - -bool drm_gpuva_interval_empty(struct drm_gpuva_manager *mgr, u64 addr, u64 range); - -static inline void drm_gpuva_init(struct drm_gpuva *va, u64 addr, u64 range, - struct drm_gem_object *obj, u64 offset) -{ - va->va.addr = addr; - va->va.range = range; - va->gem.obj = obj; - va->gem.offset = offset; -} - -/** - * drm_gpuva_invalidate() - sets whether the backing GEM of this &drm_gpuva is - * invalidated - * @va: the &drm_gpuva to set the invalidate flag for - * @invalidate: indicates whether the &drm_gpuva is invalidated - */ -static inline void drm_gpuva_invalidate(struct drm_gpuva *va, bool invalidate) -{ - if (invalidate) - va->flags |= DRM_GPUVA_INVALIDATED; - else - va->flags &= ~DRM_GPUVA_INVALIDATED; -} - -/** - * drm_gpuva_invalidated() - indicates whether the backing BO of this &drm_gpuva - * is invalidated - * @va: the &drm_gpuva to check - */ -static inline bool drm_gpuva_invalidated(struct drm_gpuva *va) -{ - return va->flags & DRM_GPUVA_INVALIDATED; -} - -/** - * struct drm_gpuva_manager - DRM GPU VA Manager - * - * The DRM GPU VA Manager keeps track of a GPU's virtual address space by using - * &maple_tree structures. Typically, this structure is embedded in bigger - * driver structures. - * - * Drivers can pass addresses and ranges in an arbitrary unit, e.g. bytes or - * pages. - * - * There should be one manager instance per GPU virtual address space. - */ -struct drm_gpuva_manager { - /** - * @name: the name of the DRM GPU VA space - */ - const char *name; - - /** - * @mm_start: start of the VA space - */ - u64 mm_start; - - /** - * @mm_range: length of the VA space - */ - u64 mm_range; - - /** - * @rb: structures to track &drm_gpuva entries - */ - struct { - /** - * @tree: the rb-tree to track GPU VA mappings - */ - struct rb_root_cached tree; - - /** - * @list: the &list_head to track GPU VA mappings - */ - struct list_head list; - } rb; - - /** - * @kernel_alloc_node: - * - * &drm_gpuva representing the address space cutout reserved for - * the kernel - */ - struct drm_gpuva kernel_alloc_node; - - /** - * @ops: &drm_gpuva_fn_ops providing the split/merge steps to drivers - */ - const struct drm_gpuva_fn_ops *ops; -}; - -void drm_gpuva_manager_init(struct drm_gpuva_manager *mgr, - const char *name, - u64 start_offset, u64 range, - u64 reserve_offset, u64 reserve_range, - const struct drm_gpuva_fn_ops *ops); -void drm_gpuva_manager_destroy(struct drm_gpuva_manager *mgr); - -static inline struct drm_gpuva * -__drm_gpuva_next(struct drm_gpuva *va) -{ - if (va && !list_is_last(&va->rb.entry, &va->mgr->rb.list)) - return list_next_entry(va, rb.entry); - - return NULL; -} - -/** - * drm_gpuva_for_each_va_range() - iterate over a range of &drm_gpuvas - * @va__: &drm_gpuva structure to assign to in each iteration step - * @mgr__: &drm_gpuva_manager to walk over - * @start__: starting offset, the first gpuva will overlap this - * @end__: ending offset, the last gpuva will start before this (but may - * overlap) - * - * This iterator walks over all &drm_gpuvas in the &drm_gpuva_manager that lie - * between @start__ and @end__. It is implemented similarly to list_for_each(), - * but is using the &drm_gpuva_manager's internal interval tree to accelerate - * the search for the starting &drm_gpuva, and hence isn't safe against removal - * of elements. It assumes that @end__ is within (or is the upper limit of) the - * &drm_gpuva_manager. This iterator does not skip over the &drm_gpuva_manager's - * @kernel_alloc_node. - */ -#define drm_gpuva_for_each_va_range(va__, mgr__, start__, end__) \ - for (va__ = drm_gpuva_find_first((mgr__), (start__), (end__) - (start__)); \ - va__ && (va__->va.addr < (end__)); \ - va__ = __drm_gpuva_next(va__)) - -/** - * drm_gpuva_for_each_va_range_safe() - safely iterate over a range of - * &drm_gpuvas - * @va__: &drm_gpuva to assign to in each iteration step - * @next__: another &drm_gpuva to use as temporary storage - * @mgr__: &drm_gpuva_manager to walk over - * @start__: starting offset, the first gpuva will overlap this - * @end__: ending offset, the last gpuva will start before this (but may - * overlap) - * - * This iterator walks over all &drm_gpuvas in the &drm_gpuva_manager that lie - * between @start__ and @end__. It is implemented similarly to - * list_for_each_safe(), but is using the &drm_gpuva_manager's internal interval - * tree to accelerate the search for the starting &drm_gpuva, and hence is safe - * against removal of elements. It assumes that @end__ is within (or is the - * upper limit of) the &drm_gpuva_manager. This iterator does not skip over the - * &drm_gpuva_manager's @kernel_alloc_node. - */ -#define drm_gpuva_for_each_va_range_safe(va__, next__, mgr__, start__, end__) \ - for (va__ = drm_gpuva_find_first((mgr__), (start__), (end__) - (start__)), \ - next__ = __drm_gpuva_next(va__); \ - va__ && (va__->va.addr < (end__)); \ - va__ = next__, next__ = __drm_gpuva_next(va__)) - -/** - * drm_gpuva_for_each_va() - iterate over all &drm_gpuvas - * @va__: &drm_gpuva to assign to in each iteration step - * @mgr__: &drm_gpuva_manager to walk over - * - * This iterator walks over all &drm_gpuva structures associated with the given - * &drm_gpuva_manager. - */ -#define drm_gpuva_for_each_va(va__, mgr__) \ - list_for_each_entry(va__, &(mgr__)->rb.list, rb.entry) - -/** - * drm_gpuva_for_each_va_safe() - safely iterate over all &drm_gpuvas - * @va__: &drm_gpuva to assign to in each iteration step - * @next__: another &drm_gpuva to use as temporary storage - * @mgr__: &drm_gpuva_manager to walk over - * - * This iterator walks over all &drm_gpuva structures associated with the given - * &drm_gpuva_manager. It is implemented with list_for_each_entry_safe(), and - * hence safe against the removal of elements. - */ -#define drm_gpuva_for_each_va_safe(va__, next__, mgr__) \ - list_for_each_entry_safe(va__, next__, &(mgr__)->rb.list, rb.entry) - -/** - * enum drm_gpuva_op_type - GPU VA operation type - * - * Operations to alter the GPU VA mappings tracked by the &drm_gpuva_manager. - */ -enum drm_gpuva_op_type { - /** - * @DRM_GPUVA_OP_MAP: the map op type - */ - DRM_GPUVA_OP_MAP, - - /** - * @DRM_GPUVA_OP_REMAP: the remap op type - */ - DRM_GPUVA_OP_REMAP, - - /** - * @DRM_GPUVA_OP_UNMAP: the unmap op type - */ - DRM_GPUVA_OP_UNMAP, - - /** - * @DRM_GPUVA_OP_PREFETCH: the prefetch op type - */ - DRM_GPUVA_OP_PREFETCH, -}; - -/** - * struct drm_gpuva_op_map - GPU VA map operation - * - * This structure represents a single map operation generated by the - * DRM GPU VA manager. - */ -struct drm_gpuva_op_map { - /** - * @va: structure containing address and range of a map - * operation - */ - struct { - /** - * @addr: the base address of the new mapping - */ - u64 addr; - - /** - * @range: the range of the new mapping - */ - u64 range; - } va; - - /** - * @gem: structure containing the &drm_gem_object and it's offset - */ - struct { - /** - * @offset: the offset within the &drm_gem_object - */ - u64 offset; - - /** - * @obj: the &drm_gem_object to map - */ - struct drm_gem_object *obj; - } gem; -}; - -/** - * struct drm_gpuva_op_unmap - GPU VA unmap operation - * - * This structure represents a single unmap operation generated by the - * DRM GPU VA manager. - */ -struct drm_gpuva_op_unmap { - /** - * @va: the &drm_gpuva to unmap - */ - struct drm_gpuva *va; - - /** - * @keep: - * - * Indicates whether this &drm_gpuva is physically contiguous with the - * original mapping request. - * - * Optionally, if &keep is set, drivers may keep the actual page table - * mappings for this &drm_gpuva, adding the missing page table entries - * only and update the &drm_gpuva_manager accordingly. - */ - bool keep; -}; - -/** - * struct drm_gpuva_op_remap - GPU VA remap operation - * - * This represents a single remap operation generated by the DRM GPU VA manager. - * - * A remap operation is generated when an existing GPU VA mmapping is split up - * by inserting a new GPU VA mapping or by partially unmapping existent - * mapping(s), hence it consists of a maximum of two map and one unmap - * operation. - * - * The @unmap operation takes care of removing the original existing mapping. - * @prev is used to remap the preceding part, @next the subsequent part. - * - * If either a new mapping's start address is aligned with the start address - * of the old mapping or the new mapping's end address is aligned with the - * end address of the old mapping, either @prev or @next is NULL. - * - * Note, the reason for a dedicated remap operation, rather than arbitrary - * unmap and map operations, is to give drivers the chance of extracting driver - * specific data for creating the new mappings from the unmap operations's - * &drm_gpuva structure which typically is embedded in larger driver specific - * structures. - */ -struct drm_gpuva_op_remap { - /** - * @prev: the preceding part of a split mapping - */ - struct drm_gpuva_op_map *prev; - - /** - * @next: the subsequent part of a split mapping - */ - struct drm_gpuva_op_map *next; - - /** - * @unmap: the unmap operation for the original existing mapping - */ - struct drm_gpuva_op_unmap *unmap; -}; - -/** - * struct drm_gpuva_op_prefetch - GPU VA prefetch operation - * - * This structure represents a single prefetch operation generated by the - * DRM GPU VA manager. - */ -struct drm_gpuva_op_prefetch { - /** - * @va: the &drm_gpuva to prefetch - */ - struct drm_gpuva *va; -}; - -/** - * struct drm_gpuva_op - GPU VA operation - * - * This structure represents a single generic operation. - * - * The particular type of the operation is defined by @op. - */ -struct drm_gpuva_op { - /** - * @entry: - * - * The &list_head used to distribute instances of this struct within - * &drm_gpuva_ops. - */ - struct list_head entry; - - /** - * @op: the type of the operation - */ - enum drm_gpuva_op_type op; - - union { - /** - * @map: the map operation - */ - struct drm_gpuva_op_map map; - - /** - * @remap: the remap operation - */ - struct drm_gpuva_op_remap remap; - - /** - * @unmap: the unmap operation - */ - struct drm_gpuva_op_unmap unmap; - - /** - * @prefetch: the prefetch operation - */ - struct drm_gpuva_op_prefetch prefetch; - }; -}; - -/** - * struct drm_gpuva_ops - wraps a list of &drm_gpuva_op - */ -struct drm_gpuva_ops { - /** - * @list: the &list_head - */ - struct list_head list; -}; - -/** - * drm_gpuva_for_each_op() - iterator to walk over &drm_gpuva_ops - * @op: &drm_gpuva_op to assign in each iteration step - * @ops: &drm_gpuva_ops to walk - * - * This iterator walks over all ops within a given list of operations. - */ -#define drm_gpuva_for_each_op(op, ops) list_for_each_entry(op, &(ops)->list, entry) - -/** - * drm_gpuva_for_each_op_safe() - iterator to safely walk over &drm_gpuva_ops - * @op: &drm_gpuva_op to assign in each iteration step - * @next: &next &drm_gpuva_op to store the next step - * @ops: &drm_gpuva_ops to walk - * - * This iterator walks over all ops within a given list of operations. It is - * implemented with list_for_each_safe(), so save against removal of elements. - */ -#define drm_gpuva_for_each_op_safe(op, next, ops) \ - list_for_each_entry_safe(op, next, &(ops)->list, entry) - -/** - * drm_gpuva_for_each_op_from_reverse() - iterate backwards from the given point - * @op: &drm_gpuva_op to assign in each iteration step - * @ops: &drm_gpuva_ops to walk - * - * This iterator walks over all ops within a given list of operations beginning - * from the given operation in reverse order. - */ -#define drm_gpuva_for_each_op_from_reverse(op, ops) \ - list_for_each_entry_from_reverse(op, &(ops)->list, entry) - -/** - * drm_gpuva_first_op() - returns the first &drm_gpuva_op from &drm_gpuva_ops - * @ops: the &drm_gpuva_ops to get the fist &drm_gpuva_op from - */ -#define drm_gpuva_first_op(ops) \ - list_first_entry(&(ops)->list, struct drm_gpuva_op, entry) - -/** - * drm_gpuva_last_op() - returns the last &drm_gpuva_op from &drm_gpuva_ops - * @ops: the &drm_gpuva_ops to get the last &drm_gpuva_op from - */ -#define drm_gpuva_last_op(ops) \ - list_last_entry(&(ops)->list, struct drm_gpuva_op, entry) - -/** - * drm_gpuva_prev_op() - previous &drm_gpuva_op in the list - * @op: the current &drm_gpuva_op - */ -#define drm_gpuva_prev_op(op) list_prev_entry(op, entry) - -/** - * drm_gpuva_next_op() - next &drm_gpuva_op in the list - * @op: the current &drm_gpuva_op - */ -#define drm_gpuva_next_op(op) list_next_entry(op, entry) - -struct drm_gpuva_ops * -drm_gpuva_sm_map_ops_create(struct drm_gpuva_manager *mgr, - u64 addr, u64 range, - struct drm_gem_object *obj, u64 offset); -struct drm_gpuva_ops * -drm_gpuva_sm_unmap_ops_create(struct drm_gpuva_manager *mgr, - u64 addr, u64 range); - -struct drm_gpuva_ops * -drm_gpuva_prefetch_ops_create(struct drm_gpuva_manager *mgr, - u64 addr, u64 range); - -struct drm_gpuva_ops * -drm_gpuva_gem_unmap_ops_create(struct drm_gpuva_manager *mgr, - struct drm_gem_object *obj); - -void drm_gpuva_ops_free(struct drm_gpuva_manager *mgr, - struct drm_gpuva_ops *ops); - -static inline void drm_gpuva_init_from_op(struct drm_gpuva *va, - struct drm_gpuva_op_map *op) -{ - drm_gpuva_init(va, op->va.addr, op->va.range, - op->gem.obj, op->gem.offset); -} - -/** - * struct drm_gpuva_fn_ops - callbacks for split/merge steps - * - * This structure defines the callbacks used by &drm_gpuva_sm_map and - * &drm_gpuva_sm_unmap to provide the split/merge steps for map and unmap - * operations to drivers. - */ -struct drm_gpuva_fn_ops { - /** - * @op_alloc: called when the &drm_gpuva_manager allocates - * a struct drm_gpuva_op - * - * Some drivers may want to embed struct drm_gpuva_op into driver - * specific structures. By implementing this callback drivers can - * allocate memory accordingly. - * - * This callback is optional. - */ - struct drm_gpuva_op *(*op_alloc)(void); - - /** - * @op_free: called when the &drm_gpuva_manager frees a - * struct drm_gpuva_op - * - * Some drivers may want to embed struct drm_gpuva_op into driver - * specific structures. By implementing this callback drivers can - * free the previously allocated memory accordingly. - * - * This callback is optional. - */ - void (*op_free)(struct drm_gpuva_op *op); - - /** - * @sm_step_map: called from &drm_gpuva_sm_map to finally insert the - * mapping once all previous steps were completed - * - * The &priv pointer matches the one the driver passed to - * &drm_gpuva_sm_map or &drm_gpuva_sm_unmap, respectively. - * - * Can be NULL if &drm_gpuva_sm_map is used. - */ - int (*sm_step_map)(struct drm_gpuva_op *op, void *priv); - - /** - * @sm_step_remap: called from &drm_gpuva_sm_map and - * &drm_gpuva_sm_unmap to split up an existent mapping - * - * This callback is called when existent mapping needs to be split up. - * This is the case when either a newly requested mapping overlaps or - * is enclosed by an existent mapping or a partial unmap of an existent - * mapping is requested. - * - * The &priv pointer matches the one the driver passed to - * &drm_gpuva_sm_map or &drm_gpuva_sm_unmap, respectively. - * - * Can be NULL if neither &drm_gpuva_sm_map nor &drm_gpuva_sm_unmap is - * used. - */ - int (*sm_step_remap)(struct drm_gpuva_op *op, void *priv); - - /** - * @sm_step_unmap: called from &drm_gpuva_sm_map and - * &drm_gpuva_sm_unmap to unmap an existent mapping - * - * This callback is called when existent mapping needs to be unmapped. - * This is the case when either a newly requested mapping encloses an - * existent mapping or an unmap of an existent mapping is requested. - * - * The &priv pointer matches the one the driver passed to - * &drm_gpuva_sm_map or &drm_gpuva_sm_unmap, respectively. - * - * Can be NULL if neither &drm_gpuva_sm_map nor &drm_gpuva_sm_unmap is - * used. - */ - int (*sm_step_unmap)(struct drm_gpuva_op *op, void *priv); -}; - -int drm_gpuva_sm_map(struct drm_gpuva_manager *mgr, void *priv, - u64 addr, u64 range, - struct drm_gem_object *obj, u64 offset); - -int drm_gpuva_sm_unmap(struct drm_gpuva_manager *mgr, void *priv, - u64 addr, u64 range); - -void drm_gpuva_map(struct drm_gpuva_manager *mgr, - struct drm_gpuva *va, - struct drm_gpuva_op_map *op); - -void drm_gpuva_remap(struct drm_gpuva *prev, - struct drm_gpuva *next, - struct drm_gpuva_op_remap *op); - -void drm_gpuva_unmap(struct drm_gpuva_op_unmap *op); - -#endif /* __DRM_GPUVA_MGR_H__ */ diff --git a/include/drm/drm_gpuvm.h b/include/drm/drm_gpuvm.h new file mode 100644 index 000000000000..c7ed6bf441d4 --- /dev/null +++ b/include/drm/drm_gpuvm.h @@ -0,0 +1,705 @@ +/* SPDX-License-Identifier: GPL-2.0-only */ + +#ifndef __DRM_GPUVM_H__ +#define __DRM_GPUVM_H__ + +/* + * Copyright (c) 2022 Red Hat. + * + * Permission is hereby granted, free of charge, to any person obtaining a + * copy of this software and associated documentation files (the "Software"), + * to deal in the Software without restriction, including without limitation + * the rights to use, copy, modify, merge, publish, distribute, sublicense, + * and/or sell copies of the Software, and to permit persons to whom the + * Software is furnished to do so, subject to the following conditions: + * + * The above copyright notice and this permission notice shall be included in + * all copies or substantial portions of the Software. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR + * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, + * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL + * THE COPYRIGHT HOLDER(S) OR AUTHOR(S) BE LIABLE FOR ANY CLAIM, DAMAGES OR + * OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, + * ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR + * OTHER DEALINGS IN THE SOFTWARE. + */ + +#include +#include +#include + +#include + +struct drm_gpuvm; +struct drm_gpuvm_ops; + +/** + * enum drm_gpuva_flags - flags for struct drm_gpuva + */ +enum drm_gpuva_flags { + /** + * @DRM_GPUVA_INVALIDATED: + * + * Flag indicating that the &drm_gpuva's backing GEM is invalidated. + */ + DRM_GPUVA_INVALIDATED = (1 << 0), + + /** + * @DRM_GPUVA_SPARSE: + * + * Flag indicating that the &drm_gpuva is a sparse mapping. + */ + DRM_GPUVA_SPARSE = (1 << 1), + + /** + * @DRM_GPUVA_USERBITS: user defined bits + */ + DRM_GPUVA_USERBITS = (1 << 2), +}; + +/** + * struct drm_gpuva - structure to track a GPU VA mapping + * + * This structure represents a GPU VA mapping and is associated with a + * &drm_gpuvm. + * + * Typically, this structure is embedded in bigger driver structures. + */ +struct drm_gpuva { + /** + * @vm: the &drm_gpuvm this object is associated with + */ + struct drm_gpuvm *vm; + + /** + * @flags: the &drm_gpuva_flags for this mapping + */ + enum drm_gpuva_flags flags; + + /** + * @va: structure containing the address and range of the &drm_gpuva + */ + struct { + /** + * @addr: the start address + */ + u64 addr; + + /* + * @range: the range + */ + u64 range; + } va; + + /** + * @gem: structure containing the &drm_gem_object and it's offset + */ + struct { + /** + * @offset: the offset within the &drm_gem_object + */ + u64 offset; + + /** + * @obj: the mapped &drm_gem_object + */ + struct drm_gem_object *obj; + + /** + * @entry: the &list_head to attach this object to a &drm_gem_object + */ + struct list_head entry; + } gem; + + /** + * @rb: structure containing data to store &drm_gpuvas in a rb-tree + */ + struct { + /** + * @rb: the rb-tree node + */ + struct rb_node node; + + /** + * @entry: The &list_head to additionally connect &drm_gpuvas + * in the same order they appear in the interval tree. This is + * useful to keep iterating &drm_gpuvas from a start node found + * through the rb-tree while doing modifications on the rb-tree + * itself. + */ + struct list_head entry; + + /** + * @__subtree_last: needed by the interval tree, holding last-in-subtree + */ + u64 __subtree_last; + } rb; +}; + +int drm_gpuva_insert(struct drm_gpuvm *gpuvm, struct drm_gpuva *va); +void drm_gpuva_remove(struct drm_gpuva *va); + +void drm_gpuva_link(struct drm_gpuva *va); +void drm_gpuva_unlink(struct drm_gpuva *va); + +struct drm_gpuva *drm_gpuva_find(struct drm_gpuvm *gpuvm, + u64 addr, u64 range); +struct drm_gpuva *drm_gpuva_find_first(struct drm_gpuvm *gpuvm, + u64 addr, u64 range); +struct drm_gpuva *drm_gpuva_find_prev(struct drm_gpuvm *gpuvm, u64 start); +struct drm_gpuva *drm_gpuva_find_next(struct drm_gpuvm *gpuvm, u64 end); + +static inline void drm_gpuva_init(struct drm_gpuva *va, u64 addr, u64 range, + struct drm_gem_object *obj, u64 offset) +{ + va->va.addr = addr; + va->va.range = range; + va->gem.obj = obj; + va->gem.offset = offset; +} + +/** + * drm_gpuva_invalidate() - sets whether the backing GEM of this &drm_gpuva is + * invalidated + * @va: the &drm_gpuva to set the invalidate flag for + * @invalidate: indicates whether the &drm_gpuva is invalidated + */ +static inline void drm_gpuva_invalidate(struct drm_gpuva *va, bool invalidate) +{ + if (invalidate) + va->flags |= DRM_GPUVA_INVALIDATED; + else + va->flags &= ~DRM_GPUVA_INVALIDATED; +} + +/** + * drm_gpuva_invalidated() - indicates whether the backing BO of this &drm_gpuva + * is invalidated + * @va: the &drm_gpuva to check + */ +static inline bool drm_gpuva_invalidated(struct drm_gpuva *va) +{ + return va->flags & DRM_GPUVA_INVALIDATED; +} + +/** + * struct drm_gpuvm - DRM GPU VA Manager + * + * The DRM GPU VA Manager keeps track of a GPU's virtual address space by using + * &maple_tree structures. Typically, this structure is embedded in bigger + * driver structures. + * + * Drivers can pass addresses and ranges in an arbitrary unit, e.g. bytes or + * pages. + * + * There should be one manager instance per GPU virtual address space. + */ +struct drm_gpuvm { + /** + * @name: the name of the DRM GPU VA space + */ + const char *name; + + /** + * @mm_start: start of the VA space + */ + u64 mm_start; + + /** + * @mm_range: length of the VA space + */ + u64 mm_range; + + /** + * @rb: structures to track &drm_gpuva entries + */ + struct { + /** + * @tree: the rb-tree to track GPU VA mappings + */ + struct rb_root_cached tree; + + /** + * @list: the &list_head to track GPU VA mappings + */ + struct list_head list; + } rb; + + /** + * @kernel_alloc_node: + * + * &drm_gpuva representing the address space cutout reserved for + * the kernel + */ + struct drm_gpuva kernel_alloc_node; + + /** + * @ops: &drm_gpuvm_ops providing the split/merge steps to drivers + */ + const struct drm_gpuvm_ops *ops; +}; + +void drm_gpuvm_init(struct drm_gpuvm *gpuvm, const char *name, + u64 start_offset, u64 range, + u64 reserve_offset, u64 reserve_range, + const struct drm_gpuvm_ops *ops); +void drm_gpuvm_destroy(struct drm_gpuvm *gpuvm); + +bool drm_gpuvm_interval_empty(struct drm_gpuvm *gpuvm, u64 addr, u64 range); + +static inline struct drm_gpuva * +__drm_gpuva_next(struct drm_gpuva *va) +{ + if (va && !list_is_last(&va->rb.entry, &va->vm->rb.list)) + return list_next_entry(va, rb.entry); + + return NULL; +} + +/** + * drm_gpuvm_for_each_va_range() - iterate over a range of &drm_gpuvas + * @va__: &drm_gpuva structure to assign to in each iteration step + * @gpuvm__: &drm_gpuvm to walk over + * @start__: starting offset, the first gpuva will overlap this + * @end__: ending offset, the last gpuva will start before this (but may + * overlap) + * + * This iterator walks over all &drm_gpuvas in the &drm_gpuvm that lie + * between @start__ and @end__. It is implemented similarly to list_for_each(), + * but is using the &drm_gpuvm's internal interval tree to accelerate + * the search for the starting &drm_gpuva, and hence isn't safe against removal + * of elements. It assumes that @end__ is within (or is the upper limit of) the + * &drm_gpuvm. This iterator does not skip over the &drm_gpuvm's + * @kernel_alloc_node. + */ +#define drm_gpuvm_for_each_va_range(va__, gpuvm__, start__, end__) \ + for (va__ = drm_gpuva_find_first((gpuvm__), (start__), (end__) - (start__)); \ + va__ && (va__->va.addr < (end__)); \ + va__ = __drm_gpuva_next(va__)) + +/** + * drm_gpuvm_for_each_va_range_safe() - safely iterate over a range of + * &drm_gpuvas + * @va__: &drm_gpuva to assign to in each iteration step + * @next__: another &drm_gpuva to use as temporary storage + * @gpuvm__: &drm_gpuvm to walk over + * @start__: starting offset, the first gpuva will overlap this + * @end__: ending offset, the last gpuva will start before this (but may + * overlap) + * + * This iterator walks over all &drm_gpuvas in the &drm_gpuvm that lie + * between @start__ and @end__. It is implemented similarly to + * list_for_each_safe(), but is using the &drm_gpuvm's internal interval + * tree to accelerate the search for the starting &drm_gpuva, and hence is safe + * against removal of elements. It assumes that @end__ is within (or is the + * upper limit of) the &drm_gpuvm. This iterator does not skip over the + * &drm_gpuvm's @kernel_alloc_node. + */ +#define drm_gpuvm_for_each_va_range_safe(va__, next__, gpuvm__, start__, end__) \ + for (va__ = drm_gpuva_find_first((gpuvm__), (start__), (end__) - (start__)), \ + next__ = __drm_gpuva_next(va__); \ + va__ && (va__->va.addr < (end__)); \ + va__ = next__, next__ = __drm_gpuva_next(va__)) + +/** + * drm_gpuvm_for_each_va() - iterate over all &drm_gpuvas + * @va__: &drm_gpuva to assign to in each iteration step + * @gpuvm__: &drm_gpuvm to walk over + * + * This iterator walks over all &drm_gpuva structures associated with the given + * &drm_gpuvm. + */ +#define drm_gpuvm_for_each_va(va__, gpuvm__) \ + list_for_each_entry(va__, &(gpuvm__)->rb.list, rb.entry) + +/** + * drm_gpuvm_for_each_va_safe() - safely iterate over all &drm_gpuvas + * @va__: &drm_gpuva to assign to in each iteration step + * @next__: another &drm_gpuva to use as temporary storage + * @gpuvm__: &drm_gpuvm to walk over + * + * This iterator walks over all &drm_gpuva structures associated with the given + * &drm_gpuvm. It is implemented with list_for_each_entry_safe(), and + * hence safe against the removal of elements. + */ +#define drm_gpuvm_for_each_va_safe(va__, next__, gpuvm__) \ + list_for_each_entry_safe(va__, next__, &(gpuvm__)->rb.list, rb.entry) + +/** + * enum drm_gpuva_op_type - GPU VA operation type + * + * Operations to alter the GPU VA mappings tracked by the &drm_gpuvm. + */ +enum drm_gpuva_op_type { + /** + * @DRM_GPUVA_OP_MAP: the map op type + */ + DRM_GPUVA_OP_MAP, + + /** + * @DRM_GPUVA_OP_REMAP: the remap op type + */ + DRM_GPUVA_OP_REMAP, + + /** + * @DRM_GPUVA_OP_UNMAP: the unmap op type + */ + DRM_GPUVA_OP_UNMAP, + + /** + * @DRM_GPUVA_OP_PREFETCH: the prefetch op type + */ + DRM_GPUVA_OP_PREFETCH, +}; + +/** + * struct drm_gpuva_op_map - GPU VA map operation + * + * This structure represents a single map operation generated by the + * DRM GPU VA manager. + */ +struct drm_gpuva_op_map { + /** + * @va: structure containing address and range of a map + * operation + */ + struct { + /** + * @addr: the base address of the new mapping + */ + u64 addr; + + /** + * @range: the range of the new mapping + */ + u64 range; + } va; + + /** + * @gem: structure containing the &drm_gem_object and it's offset + */ + struct { + /** + * @offset: the offset within the &drm_gem_object + */ + u64 offset; + + /** + * @obj: the &drm_gem_object to map + */ + struct drm_gem_object *obj; + } gem; +}; + +/** + * struct drm_gpuva_op_unmap - GPU VA unmap operation + * + * This structure represents a single unmap operation generated by the + * DRM GPU VA manager. + */ +struct drm_gpuva_op_unmap { + /** + * @va: the &drm_gpuva to unmap + */ + struct drm_gpuva *va; + + /** + * @keep: + * + * Indicates whether this &drm_gpuva is physically contiguous with the + * original mapping request. + * + * Optionally, if &keep is set, drivers may keep the actual page table + * mappings for this &drm_gpuva, adding the missing page table entries + * only and update the &drm_gpuvm accordingly. + */ + bool keep; +}; + +/** + * struct drm_gpuva_op_remap - GPU VA remap operation + * + * This represents a single remap operation generated by the DRM GPU VA manager. + * + * A remap operation is generated when an existing GPU VA mmapping is split up + * by inserting a new GPU VA mapping or by partially unmapping existent + * mapping(s), hence it consists of a maximum of two map and one unmap + * operation. + * + * The @unmap operation takes care of removing the original existing mapping. + * @prev is used to remap the preceding part, @next the subsequent part. + * + * If either a new mapping's start address is aligned with the start address + * of the old mapping or the new mapping's end address is aligned with the + * end address of the old mapping, either @prev or @next is NULL. + * + * Note, the reason for a dedicated remap operation, rather than arbitrary + * unmap and map operations, is to give drivers the chance of extracting driver + * specific data for creating the new mappings from the unmap operations's + * &drm_gpuva structure which typically is embedded in larger driver specific + * structures. + */ +struct drm_gpuva_op_remap { + /** + * @prev: the preceding part of a split mapping + */ + struct drm_gpuva_op_map *prev; + + /** + * @next: the subsequent part of a split mapping + */ + struct drm_gpuva_op_map *next; + + /** + * @unmap: the unmap operation for the original existing mapping + */ + struct drm_gpuva_op_unmap *unmap; +}; + +/** + * struct drm_gpuva_op_prefetch - GPU VA prefetch operation + * + * This structure represents a single prefetch operation generated by the + * DRM GPU VA manager. + */ +struct drm_gpuva_op_prefetch { + /** + * @va: the &drm_gpuva to prefetch + */ + struct drm_gpuva *va; +}; + +/** + * struct drm_gpuva_op - GPU VA operation + * + * This structure represents a single generic operation. + * + * The particular type of the operation is defined by @op. + */ +struct drm_gpuva_op { + /** + * @entry: + * + * The &list_head used to distribute instances of this struct within + * &drm_gpuva_ops. + */ + struct list_head entry; + + /** + * @op: the type of the operation + */ + enum drm_gpuva_op_type op; + + union { + /** + * @map: the map operation + */ + struct drm_gpuva_op_map map; + + /** + * @remap: the remap operation + */ + struct drm_gpuva_op_remap remap; + + /** + * @unmap: the unmap operation + */ + struct drm_gpuva_op_unmap unmap; + + /** + * @prefetch: the prefetch operation + */ + struct drm_gpuva_op_prefetch prefetch; + }; +}; + +/** + * struct drm_gpuva_ops - wraps a list of &drm_gpuva_op + */ +struct drm_gpuva_ops { + /** + * @list: the &list_head + */ + struct list_head list; +}; + +/** + * drm_gpuva_for_each_op() - iterator to walk over &drm_gpuva_ops + * @op: &drm_gpuva_op to assign in each iteration step + * @ops: &drm_gpuva_ops to walk + * + * This iterator walks over all ops within a given list of operations. + */ +#define drm_gpuva_for_each_op(op, ops) list_for_each_entry(op, &(ops)->list, entry) + +/** + * drm_gpuva_for_each_op_safe() - iterator to safely walk over &drm_gpuva_ops + * @op: &drm_gpuva_op to assign in each iteration step + * @next: &next &drm_gpuva_op to store the next step + * @ops: &drm_gpuva_ops to walk + * + * This iterator walks over all ops within a given list of operations. It is + * implemented with list_for_each_safe(), so save against removal of elements. + */ +#define drm_gpuva_for_each_op_safe(op, next, ops) \ + list_for_each_entry_safe(op, next, &(ops)->list, entry) + +/** + * drm_gpuva_for_each_op_from_reverse() - iterate backwards from the given point + * @op: &drm_gpuva_op to assign in each iteration step + * @ops: &drm_gpuva_ops to walk + * + * This iterator walks over all ops within a given list of operations beginning + * from the given operation in reverse order. + */ +#define drm_gpuva_for_each_op_from_reverse(op, ops) \ + list_for_each_entry_from_reverse(op, &(ops)->list, entry) + +/** + * drm_gpuva_first_op() - returns the first &drm_gpuva_op from &drm_gpuva_ops + * @ops: the &drm_gpuva_ops to get the fist &drm_gpuva_op from + */ +#define drm_gpuva_first_op(ops) \ + list_first_entry(&(ops)->list, struct drm_gpuva_op, entry) + +/** + * drm_gpuva_last_op() - returns the last &drm_gpuva_op from &drm_gpuva_ops + * @ops: the &drm_gpuva_ops to get the last &drm_gpuva_op from + */ +#define drm_gpuva_last_op(ops) \ + list_last_entry(&(ops)->list, struct drm_gpuva_op, entry) + +/** + * drm_gpuva_prev_op() - previous &drm_gpuva_op in the list + * @op: the current &drm_gpuva_op + */ +#define drm_gpuva_prev_op(op) list_prev_entry(op, entry) + +/** + * drm_gpuva_next_op() - next &drm_gpuva_op in the list + * @op: the current &drm_gpuva_op + */ +#define drm_gpuva_next_op(op) list_next_entry(op, entry) + +struct drm_gpuva_ops * +drm_gpuvm_sm_map_ops_create(struct drm_gpuvm *gpuvm, + u64 addr, u64 range, + struct drm_gem_object *obj, u64 offset); +struct drm_gpuva_ops * +drm_gpuvm_sm_unmap_ops_create(struct drm_gpuvm *gpuvm, + u64 addr, u64 range); + +struct drm_gpuva_ops * +drm_gpuvm_prefetch_ops_create(struct drm_gpuvm *gpuvm, + u64 addr, u64 range); + +struct drm_gpuva_ops * +drm_gpuvm_gem_unmap_ops_create(struct drm_gpuvm *gpuvm, + struct drm_gem_object *obj); + +void drm_gpuva_ops_free(struct drm_gpuvm *gpuvm, + struct drm_gpuva_ops *ops); + +static inline void drm_gpuva_init_from_op(struct drm_gpuva *va, + struct drm_gpuva_op_map *op) +{ + drm_gpuva_init(va, op->va.addr, op->va.range, + op->gem.obj, op->gem.offset); +} + +/** + * struct drm_gpuvm_ops - callbacks for split/merge steps + * + * This structure defines the callbacks used by &drm_gpuvm_sm_map and + * &drm_gpuvm_sm_unmap to provide the split/merge steps for map and unmap + * operations to drivers. + */ +struct drm_gpuvm_ops { + /** + * @op_alloc: called when the &drm_gpuvm allocates + * a struct drm_gpuva_op + * + * Some drivers may want to embed struct drm_gpuva_op into driver + * specific structures. By implementing this callback drivers can + * allocate memory accordingly. + * + * This callback is optional. + */ + struct drm_gpuva_op *(*op_alloc)(void); + + /** + * @op_free: called when the &drm_gpuvm frees a + * struct drm_gpuva_op + * + * Some drivers may want to embed struct drm_gpuva_op into driver + * specific structures. By implementing this callback drivers can + * free the previously allocated memory accordingly. + * + * This callback is optional. + */ + void (*op_free)(struct drm_gpuva_op *op); + + /** + * @sm_step_map: called from &drm_gpuvm_sm_map to finally insert the + * mapping once all previous steps were completed + * + * The &priv pointer matches the one the driver passed to + * &drm_gpuvm_sm_map or &drm_gpuvm_sm_unmap, respectively. + * + * Can be NULL if &drm_gpuvm_sm_map is used. + */ + int (*sm_step_map)(struct drm_gpuva_op *op, void *priv); + + /** + * @sm_step_remap: called from &drm_gpuvm_sm_map and + * &drm_gpuvm_sm_unmap to split up an existent mapping + * + * This callback is called when existent mapping needs to be split up. + * This is the case when either a newly requested mapping overlaps or + * is enclosed by an existent mapping or a partial unmap of an existent + * mapping is requested. + * + * The &priv pointer matches the one the driver passed to + * &drm_gpuvm_sm_map or &drm_gpuvm_sm_unmap, respectively. + * + * Can be NULL if neither &drm_gpuvm_sm_map nor &drm_gpuvm_sm_unmap is + * used. + */ + int (*sm_step_remap)(struct drm_gpuva_op *op, void *priv); + + /** + * @sm_step_unmap: called from &drm_gpuvm_sm_map and + * &drm_gpuvm_sm_unmap to unmap an existent mapping + * + * This callback is called when existent mapping needs to be unmapped. + * This is the case when either a newly requested mapping encloses an + * existent mapping or an unmap of an existent mapping is requested. + * + * The &priv pointer matches the one the driver passed to + * &drm_gpuvm_sm_map or &drm_gpuvm_sm_unmap, respectively. + * + * Can be NULL if neither &drm_gpuvm_sm_map nor &drm_gpuvm_sm_unmap is + * used. + */ + int (*sm_step_unmap)(struct drm_gpuva_op *op, void *priv); +}; + +int drm_gpuvm_sm_map(struct drm_gpuvm *gpuvm, void *priv, + u64 addr, u64 range, + struct drm_gem_object *obj, u64 offset); + +int drm_gpuvm_sm_unmap(struct drm_gpuvm *gpuvm, void *priv, + u64 addr, u64 range); + +void drm_gpuva_map(struct drm_gpuvm *gpuvm, + struct drm_gpuva *va, + struct drm_gpuva_op_map *op); + +void drm_gpuva_remap(struct drm_gpuva *prev, + struct drm_gpuva *next, + struct drm_gpuva_op_remap *op); + +void drm_gpuva_unmap(struct drm_gpuva_op_unmap *op); + +#endif /* __DRM_GPUVM_H__ */ diff --git a/include/drm/gpu_scheduler.h b/include/drm/gpu_scheduler.h index ac65f0626cfc..6591b77f178f 100644 --- a/include/drm/gpu_scheduler.h +++ b/include/drm/gpu_scheduler.h @@ -518,11 +518,32 @@ struct drm_gpu_scheduler { struct device *dev; }; +/** + * struct drm_sched_init_args - parameters for initializing a DRM GPU scheduler + * + * @ops: backend operations provided by the driver + * @timeout_wq: workqueue to use for timeout work. If NULL, the system_wq is used. + * @hw_submission: number of hw submissions that can be in flight + * @hang_limit: number of times to allow a job to hang before dropping it. + * This mechanism is DEPRECATED. Set it to 0. + * @timeout: timeout value in jiffies for submitted jobs. + * @score: score atomic shared with other schedulers. May be NULL. + * @name: name (typically the driver's name). Used for debugging + * @dev: associated device. Used for debugging + */ +struct drm_sched_init_args { + const struct drm_sched_backend_ops *ops; + struct workqueue_struct *timeout_wq; + uint32_t hw_submission; + unsigned int hang_limit; + long timeout; + atomic_t *score; + const char *name; + struct device *dev; +}; + int drm_sched_init(struct drm_gpu_scheduler *sched, - const struct drm_sched_backend_ops *ops, - uint32_t hw_submission, unsigned hang_limit, - long timeout, struct workqueue_struct *timeout_wq, - atomic_t *score, const char *name, struct device *dev); + const struct drm_sched_init_args *args); void drm_sched_fini(struct drm_gpu_scheduler *sched); int drm_sched_job_init(struct drm_sched_job *job, diff --git a/include/linux/bpf.h b/include/linux/bpf.h index 19aa8a525788..e3af0e06b4bf 100644 --- a/include/linux/bpf.h +++ b/include/linux/bpf.h @@ -1258,11 +1258,16 @@ int bpf_dynptr_check_size(u32 size); u32 __bpf_dynptr_size(const struct bpf_dynptr_kern *ptr); #ifdef CONFIG_BPF_JIT -int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr); -int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr); +int bpf_trampoline_link_prog(struct bpf_tramp_link *link, + struct bpf_trampoline *tr, + struct bpf_prog *tgt_prog); +int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, + struct bpf_trampoline *tr, + struct bpf_prog *tgt_prog); struct bpf_trampoline *bpf_trampoline_get(u64 key, struct bpf_attach_target_info *tgt_info); void bpf_trampoline_put(struct bpf_trampoline *tr); +void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags); int arch_prepare_bpf_dispatcher(void *image, void *buf, s64 *funcs, int num_funcs); /* @@ -1340,12 +1345,14 @@ void bpf_jit_uncharge_modmem(u32 size); bool bpf_prog_has_trampoline(const struct bpf_prog *prog); #else static inline int bpf_trampoline_link_prog(struct bpf_tramp_link *link, - struct bpf_trampoline *tr) + struct bpf_trampoline *tr, + struct bpf_prog *tgt_prog) { return -ENOTSUPP; } static inline int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, - struct bpf_trampoline *tr) + struct bpf_trampoline *tr, + struct bpf_prog *tgt_prog) { return -ENOTSUPP; } @@ -1355,6 +1362,7 @@ static inline struct bpf_trampoline *bpf_trampoline_get(u64 key, return NULL; } static inline void bpf_trampoline_put(struct bpf_trampoline *tr) {} +static inline void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags) {} #define DEFINE_BPF_DISPATCHER(name) #define DECLARE_BPF_DISPATCHER(name) #define BPF_DISPATCHER_FUNC(name) bpf_dispatcher_nop_func @@ -1444,6 +1452,9 @@ struct bpf_prog_aux { bool tail_call_reachable; bool xdp_has_frags; bool changes_pkt_data; + bool is_extended; /* true if extended by freplace program */ + u64 prog_array_member_cnt; /* counts how many times as member of prog_array */ + struct mutex ext_mutex; /* mutex for is_extended and prog_array_member_cnt */ /* BTF_KIND_FUNC_PROTO for valid attach_btf_id */ const struct btf_type *attach_func_proto; /* function name for valid attach_btf_id */ @@ -2464,7 +2475,7 @@ int btf_struct_access(struct bpf_verifier_log *log, bool btf_struct_ids_match(struct bpf_verifier_log *log, const struct btf *btf, u32 id, int off, const struct btf *need_btf, u32 need_type_id, - bool strict); + bool strict, bool walk_flex_arrays); int btf_distill_func_proto(struct bpf_verifier_log *log, struct btf *btf, @@ -2789,6 +2800,11 @@ static inline void bpf_dynptr_set_rdonly(struct bpf_dynptr_kern *ptr) } #endif /* CONFIG_BPF_SYSCALL */ +static inline bool bpf_net_capable(void) +{ + return capable(CAP_NET_ADMIN) || capable(CAP_SYS_ADMIN); +} + static __always_inline int bpf_probe_read_kernel_common(void *dst, u32 size, const void *unsafe_ptr) { @@ -3262,4 +3278,10 @@ static inline gfp_t bpf_memcg_flags(gfp_t flags) return flags; } +static inline bool bpf_prog_is_raw_tp(const struct bpf_prog *prog) +{ + return prog->type == BPF_PROG_TYPE_TRACING && + prog->expected_attach_type == BPF_TRACE_RAW_TP; +} + #endif /* _LINUX_BPF_H */ diff --git a/include/linux/bpf_mprog.h b/include/linux/bpf_mprog.h index 929225f7b095..0b9f4caeeb0a 100644 --- a/include/linux/bpf_mprog.h +++ b/include/linux/bpf_mprog.h @@ -340,4 +340,14 @@ static inline bool bpf_mprog_supported(enum bpf_prog_type type) return false; } } + +static inline bool bpf_mprog_detach_empty(enum bpf_prog_type type) +{ + switch (type) { + case BPF_PROG_TYPE_SCHED_CLS: + return bpf_net_capable(); + default: + return false; + } +} #endif /* __BPF_MPROG_H */ diff --git a/include/linux/bpf_verifier.h b/include/linux/bpf_verifier.h index 9a3b93c24f19..23eac97f8a7d 100644 --- a/include/linux/bpf_verifier.h +++ b/include/linux/bpf_verifier.h @@ -333,7 +333,11 @@ enum { INSN_F_SPI_MASK = 0x3f, /* 6 bits */ INSN_F_SPI_SHIFT = 3, /* shifted 3 bits to the left */ - INSN_F_STACK_ACCESS = BIT(9), /* we need 10 bits total */ + INSN_F_STACK_ACCESS = BIT(9), + + INSN_F_DST_REG_STACK = BIT(10), /* dst_reg is PTR_TO_STACK */ + INSN_F_SRC_REG_STACK = BIT(11), /* src_reg is PTR_TO_STACK */ + /* total 12 bits are used now. */ }; static_assert(INSN_F_FRAMENO_MASK + 1 >= MAX_CALL_FRAMES); @@ -342,9 +346,9 @@ static_assert(INSN_F_SPI_MASK + 1 >= MAX_BPF_STACK / 8); struct bpf_jmp_history_entry { u32 idx; /* insn idx can't be bigger than 1 million */ - u32 prev_idx : 22; - /* special flags, e.g., whether insn is doing register stack spill/load */ - u32 flags : 10; + u32 prev_idx : 20; + /* special INSN_F_xxx flags */ + u32 flags : 12; /* additional registers that need precision tracking when this * jump is backtracked, vector of six 10-bit records */ diff --git a/include/linux/can/skb.h b/include/linux/can/skb.h index 869ea574a40a..27d0ca115c6c 100644 --- a/include/linux/can/skb.h +++ b/include/linux/can/skb.h @@ -49,11 +49,13 @@ bool can_dropped_invalid_skb(struct net_device *dev, struct sk_buff *skb); /** * struct can_skb_priv - private additional data inside CAN sk_buffs * @ifindex: ifindex of the first interface the CAN frame appeared on + * @skbcnt: atomic counter to have an unique id together with skb pointer * @frame_len: length of CAN frame in data link layer * @cf: align to the following CAN frame at skb->data */ struct can_skb_priv { int ifindex; + u32 skbcnt; unsigned int frame_len; struct can_frame cf[]; }; diff --git a/include/linux/cgroup.h b/include/linux/cgroup.h index fa87b6a15082..bbfa1abf05d3 100644 --- a/include/linux/cgroup.h +++ b/include/linux/cgroup.h @@ -690,6 +690,7 @@ static inline void cgroup_path_from_kernfs_id(u64 id, char *buf, size_t buflen) /* * cgroup scalable recursive statistics. */ +void __cgroup_rstat_updated(struct cgroup *cgrp, int cpu); void cgroup_rstat_updated(struct cgroup *cgrp, int cpu); void cgroup_rstat_flush(struct cgroup *cgrp); void cgroup_rstat_flush_hold(struct cgroup *cgrp); diff --git a/include/linux/cpufreq.h b/include/linux/cpufreq.h index 1f94fe8559a9..445b68efe49f 100644 --- a/include/linux/cpufreq.h +++ b/include/linux/cpufreq.h @@ -388,7 +388,7 @@ struct cpufreq_driver { int (*online)(struct cpufreq_policy *policy); int (*offline)(struct cpufreq_policy *policy); - int (*exit)(struct cpufreq_policy *policy); + void (*exit)(struct cpufreq_policy *policy); int (*suspend)(struct cpufreq_policy *policy); int (*resume)(struct cpufreq_policy *policy); diff --git a/include/linux/device/bus.h b/include/linux/device/bus.h index d43ce072d747..2ade5f02afae 100644 --- a/include/linux/device/bus.h +++ b/include/linux/device/bus.h @@ -48,6 +48,7 @@ struct fwnode_handle; * will never get called until they do. * @remove: Called when a device removed from this bus. * @shutdown: Called at shut-down time to quiesce the device. + * @irq_get_affinity: Get IRQ affinity mask for the device on this bus. * * @online: Called to put the device back online (after offlining it). * @offline: Called to put the device offline for hot-removal. May fail. @@ -93,6 +94,8 @@ struct bus_type { void (*sync_state)(struct device *dev); void (*remove)(struct device *dev); void (*shutdown)(struct device *dev); + const struct cpumask *(*irq_get_affinity)(struct device *dev, + unsigned int irq_vec); int (*online)(struct device *dev); int (*offline)(struct device *dev); diff --git a/include/linux/dma-buf.h b/include/linux/dma-buf.h index 3f31baa3293f..e29dcab1d2d1 100644 --- a/include/linux/dma-buf.h +++ b/include/linux/dma-buf.h @@ -606,6 +606,7 @@ void dma_buf_unpin(struct dma_buf_attachment *attach); struct dma_buf *dma_buf_export(const struct dma_buf_export_info *exp_info); int dma_buf_fd(struct dma_buf *dmabuf, int flags); +void dma_buf_fd_install(struct dma_buf *dmabuf, int fd); struct dma_buf *dma_buf_get(int fd); void dma_buf_put(struct dma_buf *dmabuf); diff --git a/include/linux/dma/k3-udma-glue.h b/include/linux/dma/k3-udma-glue.h index e443be4d3b4b..a76f1c09a884 100644 --- a/include/linux/dma/k3-udma-glue.h +++ b/include/linux/dma/k3-udma-glue.h @@ -130,8 +130,7 @@ void k3_udma_glue_rx_put_irq(struct k3_udma_glue_rx_channel *rx_chn, u32 flow_num); void k3_udma_glue_reset_rx_chn(struct k3_udma_glue_rx_channel *rx_chn, u32 flow_num, void *data, - void (*cleanup)(void *data, dma_addr_t desc_dma), - bool skip_fdq); + void (*cleanup)(void *data, dma_addr_t desc_dma)); int k3_udma_glue_rx_flow_enable(struct k3_udma_glue_rx_channel *rx_chn, u32 flow_idx); int k3_udma_glue_rx_flow_disable(struct k3_udma_glue_rx_channel *rx_chn, diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h index cff3dba65820..ea476ea79f76 100644 --- a/include/linux/dmaengine.h +++ b/include/linux/dmaengine.h @@ -1490,6 +1490,7 @@ struct dma_chan *__dma_request_channel(const dma_cap_mask_t *mask, struct dma_chan *dma_request_chan(struct device *dev, const char *name); struct dma_chan *dma_request_chan_by_mask(const dma_cap_mask_t *mask); +struct dma_chan *devm_dma_request_chan(struct device *dev, const char *name); void dma_release_channel(struct dma_chan *chan); int dma_get_slave_caps(struct dma_chan *chan, struct dma_slave_caps *caps); @@ -1526,6 +1527,12 @@ static inline struct dma_chan *dma_request_chan_by_mask( { return ERR_PTR(-ENODEV); } + +static inline struct dma_chan *devm_dma_request_chan(struct device *dev, const char *name) +{ + return ERR_PTR(-ENODEV); +} + static inline void dma_release_channel(struct dma_chan *chan) { } diff --git a/include/linux/eventfd.h b/include/linux/eventfd.h index b9d83652c097..562089431551 100644 --- a/include/linux/eventfd.h +++ b/include/linux/eventfd.h @@ -35,7 +35,7 @@ void eventfd_ctx_put(struct eventfd_ctx *ctx); struct file *eventfd_fget(int fd); struct eventfd_ctx *eventfd_ctx_fdget(int fd); struct eventfd_ctx *eventfd_ctx_fileget(struct file *file); -__u64 eventfd_signal(struct eventfd_ctx *ctx, __u64 n); +__u64 eventfd_signal(struct eventfd_ctx *ctx); __u64 eventfd_signal_mask(struct eventfd_ctx *ctx, __u64 n, __poll_t mask); int eventfd_ctx_remove_wait_queue(struct eventfd_ctx *ctx, wait_queue_entry_t *wait, __u64 *cnt); @@ -58,7 +58,7 @@ static inline struct eventfd_ctx *eventfd_ctx_fdget(int fd) return ERR_PTR(-ENOSYS); } -static inline int eventfd_signal(struct eventfd_ctx *ctx, __u64 n) +static inline int eventfd_signal(struct eventfd_ctx *ctx) { return -ENOSYS; } diff --git a/include/linux/fb.h b/include/linux/fb.h index 8a9d949cc7e2..452bbb6c523c 100644 --- a/include/linux/fb.h +++ b/include/linux/fb.h @@ -527,6 +527,8 @@ extern int fb_set_var(struct fb_info *info, struct fb_var_screeninfo *var); extern int fb_pan_display(struct fb_info *info, struct fb_var_screeninfo *var); extern int fb_blank(struct fb_info *info, int blank); +int fb_set_var_from_user(struct fb_info *info, struct fb_var_screeninfo *var); + /* * Helpers for framebuffers in I/O memory */ diff --git a/include/linux/filelock.h b/include/linux/filelock.h index 95e868e09e29..4ae068f1522c 100644 --- a/include/linux/filelock.h +++ b/include/linux/filelock.h @@ -85,23 +85,43 @@ bool opens_in_grace(struct net *); * * Obviously, the last two criteria only matter for POSIX locks. */ -struct file_lock { - struct file_lock *fl_blocker; /* The lock, that is blocking us */ - struct list_head fl_list; /* link into file_lock_context */ - struct hlist_node fl_link; /* node in global lists */ - struct list_head fl_blocked_requests; /* list of requests with +struct file_lock_core { + struct file_lock *flc_blocker; /* The lock that is blocking us */ + struct list_head flc_list; /* link into file_lock_context */ + struct hlist_node flc_link; /* node in global lists */ + struct list_head flc_blocked_requests; /* list of requests with * ->fl_blocker pointing here */ - struct list_head fl_blocked_member; /* node in + struct list_head flc_blocked_member; /* node in * ->fl_blocker->fl_blocked_requests */ - fl_owner_t fl_owner; - unsigned int fl_flags; - unsigned char fl_type; - unsigned int fl_pid; - int fl_link_cpu; /* what cpu's list is this on? */ - wait_queue_head_t fl_wait; - struct file *fl_file; + fl_owner_t flc_owner; + unsigned int flc_flags; + unsigned char flc_type; + unsigned int flc_pid; + int flc_link_cpu; /* what cpu's list is this on? */ + wait_queue_head_t flc_wait; + struct file *flc_file; +}; + +struct file_lock { + union { + struct file_lock_core c; + struct { + struct file_lock *fl_blocker; + struct list_head fl_list; + struct hlist_node fl_link; + struct list_head fl_blocked_requests; + struct list_head fl_blocked_member; + fl_owner_t fl_owner; + unsigned int fl_flags; + unsigned char fl_type; + unsigned int fl_pid; + int fl_link_cpu; + wait_queue_head_t fl_wait; + struct file *fl_file; + }; + }; loff_t fl_start; loff_t fl_end; @@ -147,6 +167,29 @@ int fcntl_setlk64(unsigned int, struct file *, unsigned int, int fcntl_setlease(unsigned int fd, struct file *filp, int arg); int fcntl_getlease(struct file *filp); +static inline bool lock_is_unlock(struct file_lock *fl) +{ + return fl->fl_type == F_UNLCK; +} + +static inline bool lock_is_read(struct file_lock *fl) +{ + return fl->fl_type == F_RDLCK; +} + +static inline bool lock_is_write(struct file_lock *fl) +{ + return fl->fl_type == F_WRLCK; +} + +static inline void locks_wake_up(struct file_lock *fl) +{ + wake_up(&fl->fl_wait); +} + +/* for walking lists of file_locks linked by fl_list */ +#define for_each_file_lock(_fl, _head) list_for_each_entry(_fl, _head, fl_list) + /* fs/locks.c */ void locks_free_lock_context(struct inode *inode); void locks_free_lock(struct file_lock *fl); @@ -223,6 +266,27 @@ static inline int fcntl_getlease(struct file *filp) return F_UNLCK; } +static inline bool lock_is_unlock(struct file_lock *fl) +{ + return false; +} + +static inline bool lock_is_read(struct file_lock *fl) +{ + return false; +} + +static inline bool lock_is_write(struct file_lock *fl) +{ + return false; +} + +static inline void locks_wake_up(struct file_lock *fl) +{ +} + +#define for_each_file_lock(_fl, _head) while(false) + static inline void locks_free_lock_context(struct inode *inode) { diff --git a/include/linux/filter.h b/include/linux/filter.h index f2e351cc543d..bc6ea3317120 100644 --- a/include/linux/filter.h +++ b/include/linux/filter.h @@ -867,20 +867,22 @@ bpf_ctx_narrow_access_offset(u32 off, u32 size, u32 size_default) #define bpf_classic_proglen(fprog) (fprog->len * sizeof(fprog->filter[0])) -static inline void bpf_prog_lock_ro(struct bpf_prog *fp) +static inline int __must_check bpf_prog_lock_ro(struct bpf_prog *fp) { #ifndef CONFIG_BPF_JIT_ALWAYS_ON if (!fp->jited) { set_vm_flush_reset_perms(fp); - set_memory_ro((unsigned long)fp, fp->pages); + return set_memory_ro((unsigned long)fp, fp->pages); } #endif + return 0; } -static inline void bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) +static inline int __must_check +bpf_jit_binary_lock_ro(struct bpf_binary_header *hdr) { set_vm_flush_reset_perms(hdr); - set_memory_rox((unsigned long)hdr, hdr->size >> PAGE_SHIFT); + return set_memory_rox((unsigned long)hdr, hdr->size >> PAGE_SHIFT); } int sk_filter_trim_cap(struct sock *sk, struct sk_buff *skb, unsigned int cap); @@ -1081,8 +1083,7 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **ro_image, u8 **rw_image, bpf_jit_fill_hole_t bpf_fill_ill_insns, bool was_classic); -int bpf_jit_binary_pack_finalize(struct bpf_prog *prog, - struct bpf_binary_header *ro_header, +int bpf_jit_binary_pack_finalize(struct bpf_binary_header *ro_header, struct bpf_binary_header *rw_header); void bpf_jit_binary_pack_free(struct bpf_binary_header *ro_header, struct bpf_binary_header *rw_header); diff --git a/include/linux/fs.h b/include/linux/fs.h index 12ea9af49a89..326d51d5262c 100644 --- a/include/linux/fs.h +++ b/include/linux/fs.h @@ -1251,6 +1251,7 @@ struct super_block { char s_id[32]; /* Informational name */ uuid_t s_uuid; /* UUID */ + u8 s_uuid_len; /* Default 16, possibly smaller for weird filesystems */ unsigned int s_max_links; @@ -2453,6 +2454,14 @@ extern __printf(2, 3) int super_setup_bdi_name(struct super_block *sb, char *fmt, ...); extern int super_setup_bdi(struct super_block *sb); +static inline void super_set_uuid(struct super_block *sb, const u8 *uuid, unsigned len) +{ + if (WARN_ON(len > sizeof(sb->s_uuid))) + len = sizeof(sb->s_uuid); + sb->s_uuid_len = len; + memcpy(&sb->s_uuid, uuid, len); +} + extern int current_umask(void); extern void ihold(struct inode * inode); @@ -3231,7 +3240,7 @@ struct offset_ctx { void simple_offset_init(struct offset_ctx *octx); int simple_offset_add(struct offset_ctx *octx, struct dentry *dentry); void simple_offset_remove(struct offset_ctx *octx, struct dentry *dentry); -int simple_offset_rename(struct inode *old_dir, struct dentry *old_dentry, +void simple_offset_rename(struct inode *old_dir, struct dentry *old_dentry, struct inode *new_dir, struct dentry *new_dentry); int simple_offset_rename_exchange(struct inode *old_dir, struct dentry *old_dentry, diff --git a/include/linux/fsl/mc.h b/include/linux/fsl/mc.h index a1b3de87a3d1..7a19f5a1cbcf 100644 --- a/include/linux/fsl/mc.h +++ b/include/linux/fsl/mc.h @@ -178,9 +178,6 @@ struct fsl_mc_obj_desc { * @regions: pointer to array of MMIO region entries * @irqs: pointer to array of pointers to interrupts allocated to this device * @resource: generic resource associated with this MC object device, if any. - * @driver_override: driver name to force a match; do not set directly, - * because core frees it; use driver_set_override() to - * set or clear it. * * Generic device object for MC object devices that are "attached" to a * MC bus. @@ -214,7 +211,6 @@ struct fsl_mc_device { struct fsl_mc_device_irq **irqs; struct fsl_mc_resource *resource; struct device_link *consumer_link; - const char *driver_override; }; #define to_fsl_mc_device(_dev) \ diff --git a/include/linux/fsverity.h b/include/linux/fsverity.h index 1eb7eae580be..139a6f5c6217 100644 --- a/include/linux/fsverity.h +++ b/include/linux/fsverity.h @@ -149,7 +149,6 @@ int fsverity_get_digest(struct inode *inode, /* open.c */ int __fsverity_file_open(struct inode *inode, struct file *filp); -int __fsverity_prepare_setattr(struct dentry *dentry, struct iattr *attr); void __fsverity_cleanup_inode(struct inode *inode); /** @@ -214,12 +213,6 @@ static inline int __fsverity_file_open(struct inode *inode, struct file *filp) return -EOPNOTSUPP; } -static inline int __fsverity_prepare_setattr(struct dentry *dentry, - struct iattr *attr) -{ - return -EOPNOTSUPP; -} - static inline void fsverity_cleanup_inode(struct inode *inode) { } @@ -301,22 +294,4 @@ static inline int fsverity_file_open(struct inode *inode, struct file *filp) return 0; } -/** - * fsverity_prepare_setattr() - prepare to change a verity inode's attributes - * @dentry: dentry through which the inode is being changed - * @attr: attributes to change - * - * Verity files are immutable, so deny truncates. This isn't covered by the - * open-time check because sys_truncate() takes a path, not a file descriptor. - * - * Return: 0 on success, -errno on failure - */ -static inline int fsverity_prepare_setattr(struct dentry *dentry, - struct iattr *attr) -{ - if (IS_VERITY(d_inode(dentry))) - return __fsverity_prepare_setattr(dentry, attr); - return 0; -} - #endif /* _LINUX_FSVERITY_H */ diff --git a/include/linux/ftrace.h b/include/linux/ftrace.h index 5c3eaf9fc90c..594fda9702ab 100644 --- a/include/linux/ftrace.h +++ b/include/linux/ftrace.h @@ -673,8 +673,9 @@ unsigned long ftrace_get_addr_new(struct dyn_ftrace *rec); unsigned long ftrace_get_addr_curr(struct dyn_ftrace *rec); extern ftrace_func_t ftrace_trace_function; +struct trace_array; -int ftrace_regex_open(struct ftrace_ops *ops, int flag, +int ftrace_regex_open(struct trace_array *tr, struct ftrace_ops *ops, int flag, struct inode *inode, struct file *file); ssize_t ftrace_filter_write(struct file *file, const char __user *ubuf, size_t cnt, loff_t *ppos); @@ -883,7 +884,7 @@ static inline unsigned long ftrace_location(unsigned long ip) * have them defined when ftrace is not enabled, but these * functions may still be called. Use a macro instead of inline. */ -#define ftrace_regex_open(ops, flag, inod, file) ({ -ENODEV; }) +#define ftrace_regex_open(tr, ops, flag, inode, file) ({ -ENODEV; }) #define ftrace_set_early_filter(ops, buf, enable) do { } while (0) #define ftrace_set_filter_ip(ops, ip, remove, reset) ({ -ENODEV; }) #define ftrace_set_filter_ips(ops, ips, cnt, remove, reset) ({ -ENODEV; }) diff --git a/include/linux/gfp.h b/include/linux/gfp.h index a0803ed4b469..9794c6f5fcbf 100644 --- a/include/linux/gfp.h +++ b/include/linux/gfp.h @@ -303,6 +303,7 @@ extern void __free_pages(struct page *page, unsigned int order); extern void free_pages(unsigned long addr, unsigned int order); struct page_frag_cache; +void page_frag_cache_drain(struct page_frag_cache *nc); extern void __page_frag_cache_drain(struct page *page, unsigned int count); extern void *page_frag_alloc_align(struct page_frag_cache *nc, unsigned int fragsz, gfp_t gfp_mask, diff --git a/include/linux/hid.h b/include/linux/hid.h index f560ae891d7e..c7f04c9f8ff4 100644 --- a/include/linux/hid.h +++ b/include/linux/hid.h @@ -81,6 +81,8 @@ struct hid_item { #define HID_MAIN_ITEM_TAG_FEATURE 11 #define HID_MAIN_ITEM_TAG_BEGIN_COLLECTION 10 #define HID_MAIN_ITEM_TAG_END_COLLECTION 12 +#define HID_MAIN_ITEM_TAG_RESERVED_MIN 13 +#define HID_MAIN_ITEM_TAG_RESERVED_MAX 15 /* * HID report descriptor main item contents @@ -869,11 +871,11 @@ struct hid_driver { void (*feature_mapping)(struct hid_device *hdev, struct hid_field *field, struct hid_usage *usage); -#ifdef CONFIG_PM + int (*suspend)(struct hid_device *hdev, pm_message_t message); int (*resume)(struct hid_device *hdev); int (*reset_resume)(struct hid_device *hdev); -#endif + /* private: */ struct device_driver driver; }; @@ -976,6 +978,8 @@ extern void hidinput_disconnect(struct hid_device *); int hid_set_field(struct hid_field *, unsigned, __s32); int hid_input_report(struct hid_device *hid, enum hid_report_type type, u8 *data, u32 size, int interrupt); +int hid_safe_input_report(struct hid_device *hid, enum hid_report_type type, u8 *data, + size_t bufsize, u32 size, int interrupt); struct hid_field *hidinput_get_led_field(struct hid_device *hid); unsigned int hidinput_count_leds(struct hid_device *hid); __s32 hidinput_calc_abs_res(const struct hid_field *field, __u16 code); @@ -1265,6 +1269,8 @@ int hid_pidff_init_with_quirks(struct hid_device *hid, __u32 initial_quirks); dev_notice(&(hid)->dev, fmt, ##__VA_ARGS__) #define hid_warn(hid, fmt, ...) \ dev_warn(&(hid)->dev, fmt, ##__VA_ARGS__) +#define hid_warn_ratelimited(hid, fmt, ...) \ + dev_warn_ratelimited(&(hid)->dev, fmt, ##__VA_ARGS__) #define hid_info(hid, fmt, ...) \ dev_info(&(hid)->dev, fmt, ##__VA_ARGS__) #define hid_dbg(hid, fmt, ...) \ diff --git a/include/linux/huge_mm.h b/include/linux/huge_mm.h index fc789c0ac85b..a0f70cb8596b 100644 --- a/include/linux/huge_mm.h +++ b/include/linux/huge_mm.h @@ -64,6 +64,28 @@ ssize_t single_hugepage_flag_show(struct kobject *kobj, enum transparent_hugepage_flag flag); extern struct kobj_attribute shmem_enabled_attr; +#define THP_ORDERS_ALL_ANON BIT(PMD_ORDER) + +/* + * Mask of all large folio orders supported for file THP. Folios in a DAX + * file is never split and the MAX_PAGECACHE_ORDER limit does not apply to + * it. + */ +#define THP_ORDERS_ALL_FILE_DAX \ + (BIT(PMD_ORDER) | BIT(PUD_ORDER)) +#define THP_ORDERS_ALL_FILE_DEFAULT \ + ((BIT(MAX_PAGECACHE_ORDER + 1) - 1) & ~BIT(0)) + +/* + * Mask of all large folio orders supported for THP. + */ +#define THP_ORDERS_ALL \ + (THP_ORDERS_ALL_ANON | THP_ORDERS_ALL_FILE_DAX | THP_ORDERS_ALL_FILE_DEFAULT) + +#define TVA_SMAPS (1 << 0) /* Will be used for procfs */ +#define TVA_IN_PF (1 << 1) /* Page fault handler */ +#define TVA_ENFORCE_SYSFS (1 << 2) /* Obey sysfs configuration */ + #define HPAGE_PMD_ORDER (HPAGE_PMD_SHIFT-PAGE_SHIFT) #define HPAGE_PMD_NR (1< #include #include +#include /** * struct hwrng - Hardware Random Number Generator driver @@ -49,6 +50,7 @@ struct hwrng { /* internal. */ struct list_head list; struct kref ref; + struct work_struct cleanup_work; struct completion cleanup_done; struct completion dying; }; diff --git a/include/linux/hyperv.h b/include/linux/hyperv.h index e4ad9760774e..20f180a08123 100644 --- a/include/linux/hyperv.h +++ b/include/linux/hyperv.h @@ -1305,11 +1305,6 @@ struct hv_device { u16 device_id; struct device device; - /* - * Driver name to force a match. Do not set directly, because core - * frees it. Use driver_set_override() to set or clear it. - */ - const char *driver_override; struct vmbus_channel *channel; struct kset *channels_kset; diff --git a/include/linux/i3c/master.h b/include/linux/i3c/master.h index f2eb2452ffa5..5e689cec1d72 100644 --- a/include/linux/i3c/master.h +++ b/include/linux/i3c/master.h @@ -220,6 +220,8 @@ struct i3c_dev_desc { * every time the I3C device is rediscovered with a different dynamic * address assigned * @bus: I3C bus this device is attached to + * @node: unregistered device list node, only for use by + * i3c_master_register_new_i3c_devs(), it is not protected by a lock * * I3C device object exposed to I3C device drivers. The takes care of linking * this object to the relevant &struct_i3c_dev_desc one. @@ -230,6 +232,7 @@ struct i3c_device { struct device dev; struct i3c_dev_desc *desc; struct i3c_bus *bus; + struct list_head node; }; /* diff --git a/include/linux/ieee80211.h b/include/linux/ieee80211.h index 38e3e81b880f..8b73e37c604d 100644 --- a/include/linux/ieee80211.h +++ b/include/linux/ieee80211.h @@ -4945,7 +4945,7 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, size_t len) const struct ieee80211_multi_link_elem *mle = (const void *)data; u8 fixed = sizeof(*mle); u8 common = 0; - bool check_common_len = false; + u8 common_len; u16 control; if (!data || len < fixed) @@ -4956,7 +4956,6 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, size_t len) switch (u16_get_bits(control, IEEE80211_ML_CONTROL_TYPE)) { case IEEE80211_ML_CONTROL_TYPE_BASIC: common += sizeof(struct ieee80211_mle_basic_common_info); - check_common_len = true; if (control & IEEE80211_MLC_BASIC_PRES_LINK_ID) common += 1; if (control & IEEE80211_MLC_BASIC_PRES_BSS_PARAM_CH_CNT) @@ -4974,15 +4973,14 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, size_t len) common += sizeof(struct ieee80211_mle_preq_common_info); if (control & IEEE80211_MLC_PREQ_PRES_MLD_ID) common += 1; - check_common_len = true; break; case IEEE80211_ML_CONTROL_TYPE_RECONF: + common += 1; if (control & IEEE80211_MLC_RECONF_PRES_MLD_MAC_ADDR) common += ETH_ALEN; break; case IEEE80211_ML_CONTROL_TYPE_TDLS: common += sizeof(struct ieee80211_mle_tdls_common_info); - check_common_len = true; break; case IEEE80211_ML_CONTROL_TYPE_PRIO_ACCESS: common = ETH_ALEN + 1; @@ -4995,11 +4993,9 @@ static inline bool ieee80211_mle_size_ok(const u8 *data, size_t len) if (len < fixed + common) return false; - if (!check_common_len) - return true; + common_len = mle->variable[0]; - /* if present, common length is the first octet there */ - return mle->variable[0] >= common; + return common_len >= common && common_len <= len - fixed; } /** diff --git a/include/linux/if_team.h b/include/linux/if_team.h index cdc684e04a2f..06fec0a78db5 100644 --- a/include/linux/if_team.h +++ b/include/linux/if_team.h @@ -121,8 +121,7 @@ struct team_mode_ops { int (*port_enter)(struct team *team, struct team_port *port); void (*port_leave)(struct team *team, struct team_port *port); void (*port_change_dev_addr)(struct team *team, struct team_port *port); - void (*port_enabled)(struct team *team, struct team_port *port); - void (*port_disabled)(struct team *team, struct team_port *port); + void (*port_tx_disabled)(struct team *team, struct team_port *port); }; extern int team_modeop_port_enter(struct team *team, struct team_port *port); diff --git a/include/linux/if_vlan.h b/include/linux/if_vlan.h index 272d9ad73960..1cab62956589 100644 --- a/include/linux/if_vlan.h +++ b/include/linux/if_vlan.h @@ -356,6 +356,9 @@ static inline int __vlan_insert_inner_tag(struct sk_buff *skb, { struct vlan_ethhdr *veth; + if (unlikely(!pskb_may_pull(skb, mac_len))) + return -EINVAL; + if (skb_cow_head(skb, VLAN_HLEN) < 0) return -ENOMEM; diff --git a/include/linux/igmp.h b/include/linux/igmp.h index ebf4349a53af..a57989cfa54c 100644 --- a/include/linux/igmp.h +++ b/include/linux/igmp.h @@ -57,20 +57,21 @@ struct ip_mc_socklist { }; struct ip_sf_list { - struct ip_sf_list *sf_next; + struct ip_sf_list __rcu *sf_next; unsigned long sf_count[2]; /* include/exclude counts */ __be32 sf_inaddr; unsigned char sf_gsresp; /* include in g & s response? */ unsigned char sf_oldin; /* change state */ unsigned char sf_crcount; /* retrans. left to send */ + struct rcu_head rcu; }; struct ip_mc_list { struct in_device *interface; __be32 multiaddr; unsigned int sfmode; - struct ip_sf_list *sources; - struct ip_sf_list *tomb; + struct ip_sf_list __rcu *sources; + struct ip_sf_list __rcu *tomb; unsigned long sfcount[2]; union { struct ip_mc_list *next; diff --git a/include/linux/iio/iio.h b/include/linux/iio/iio.h index 202e55b0a28b..3da7f2331cc6 100644 --- a/include/linux/iio/iio.h +++ b/include/linux/iio/iio.h @@ -756,6 +756,21 @@ static inline struct dentry *iio_get_debugfs_dentry(struct iio_dev *indio_dev) } #endif +/** + * iio_get_masklength - Get length of the channels mask + * @indio_dev: the IIO device to get the masklength for + */ +#define iio_get_masklength(indio_dev) ((unsigned int)(indio_dev)->masklength) + +/** + * iio_for_each_active_channel - Iterated over active channels + * @indio_dev: the IIO device + * @chan: Holds the index of the enabled channel + */ +#define iio_for_each_active_channel(indio_dev, chan) \ + for_each_set_bit((chan), (indio_dev)->active_scan_mask, \ + iio_get_masklength(indio_dev)) + ssize_t iio_format_value(char *buf, unsigned int type, int size, int *vals); int iio_str_to_fixpoint(const char *str, int fract_mult, int *integer, diff --git a/include/linux/iommu.h b/include/linux/iommu.h index b6ef263e85c0..6e5257bf9aea 100644 --- a/include/linux/iommu.h +++ b/include/linux/iommu.h @@ -289,7 +289,8 @@ struct iommu_ops { struct iommu_page_response *msg); int (*def_domain_type)(struct device *dev); - void (*remove_dev_pasid)(struct device *dev, ioasid_t pasid); + void (*remove_dev_pasid)(struct device *dev, ioasid_t pasid, + struct iommu_domain *domain); const struct iommu_domain_ops *default_domain_ops; unsigned long pgsize_bitmap; diff --git a/include/linux/irq_work.h b/include/linux/irq_work.h index 136f2980cba3..c5afd053ae32 100644 --- a/include/linux/irq_work.h +++ b/include/linux/irq_work.h @@ -2,8 +2,9 @@ #ifndef _LINUX_IRQ_WORK_H #define _LINUX_IRQ_WORK_H -#include +#include #include +#include /* * An entry can be in one of four states: @@ -14,12 +15,6 @@ * busy NULL, 2 -> {free, claimed} : callback in progress, can be claimed */ -struct irq_work { - struct __call_single_node node; - void (*func)(struct irq_work *); - struct rcuwait irqwait; -}; - #define __IRQ_WORK_INIT(_func, _flags) (struct irq_work){ \ .node = { .u_flags = (_flags), }, \ .func = (_func), \ diff --git a/include/linux/irq_work_types.h b/include/linux/irq_work_types.h new file mode 100644 index 000000000000..73abec5bb06e --- /dev/null +++ b/include/linux/irq_work_types.h @@ -0,0 +1,14 @@ +/* SPDX-License-Identifier: GPL-2.0 */ +#ifndef _LINUX_IRQ_WORK_TYPES_H +#define _LINUX_IRQ_WORK_TYPES_H + +#include +#include + +struct irq_work { + struct __call_single_node node; + void (*func)(struct irq_work *); + struct rcuwait irqwait; +}; + +#endif diff --git a/include/linux/jbd2.h b/include/linux/jbd2.h index 7dad9b761cbc..9f498fef91cd 100644 --- a/include/linux/jbd2.h +++ b/include/linux/jbd2.h @@ -1451,7 +1451,9 @@ void jbd2_update_log_tail(journal_t *journal, tid_t tid, unsigned long block); extern void jbd2_journal_commit_transaction(journal_t *); /* Checkpoint list management */ -void __jbd2_journal_clean_checkpoint_list(journal_t *journal, bool destroy); +enum shrink_type {SHRINK_DESTROY, SHRINK_BUSY_STOP, SHRINK_BUSY_SKIP}; + +void __jbd2_journal_clean_checkpoint_list(journal_t *journal, enum shrink_type type); unsigned long jbd2_journal_shrink_checkpoint_list(journal_t *journal, unsigned long *nr_to_scan); int __jbd2_journal_remove_checkpoint(struct journal_head *); int jbd2_journal_try_remove_checkpoint(struct journal_head *jh); diff --git a/include/linux/kasan.h b/include/linux/kasan.h index 92bc6bfde603..d0664f6a8a15 100644 --- a/include/linux/kasan.h +++ b/include/linux/kasan.h @@ -382,7 +382,7 @@ static inline void kasan_init_hw_tags(void) { } #if defined(CONFIG_KASAN_GENERIC) || defined(CONFIG_KASAN_SW_TAGS) void kasan_populate_early_vm_area_shadow(void *start, unsigned long size); -int kasan_populate_vmalloc(unsigned long addr, unsigned long size); +int kasan_populate_vmalloc(unsigned long addr, unsigned long size, gfp_t gfp_mask); void kasan_release_vmalloc(unsigned long start, unsigned long end, unsigned long free_region_start, unsigned long free_region_end); @@ -393,7 +393,7 @@ static inline void kasan_populate_early_vm_area_shadow(void *start, unsigned long size) { } static inline int kasan_populate_vmalloc(unsigned long start, - unsigned long size) + unsigned long size, gfp_t gfp_mask) { return 0; } @@ -438,7 +438,7 @@ kasan_unpoison_vmap_areas(struct vm_struct **vms, int nr_vms, static inline void kasan_populate_early_vm_area_shadow(void *start, unsigned long size) { } static inline int kasan_populate_vmalloc(unsigned long start, - unsigned long size) + unsigned long size, gfp_t gfp_mask) { return 0; } diff --git a/include/linux/list_lru.h b/include/linux/list_lru.h index b35968ee9fb5..7675a48a0701 100644 --- a/include/linux/list_lru.h +++ b/include/linux/list_lru.h @@ -73,8 +73,10 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren /** * list_lru_add: add an element to the lru list's tail - * @list_lru: the lru pointer + * @lru: the lru pointer * @item: the item to be added. + * @nid: the node id of the sublist to add the item to. + * @memcg: the cgroup of the sublist to add the item to. * * If the element is already part of a list, this function returns doing * nothing. Therefore the caller does not need to keep state about whether or @@ -83,24 +85,54 @@ void memcg_reparent_list_lrus(struct mem_cgroup *memcg, struct mem_cgroup *paren * the caller organize itself in a way that elements can be in more than * one type of list, it is up to the caller to fully remove the item from * the previous list (with list_lru_del() for instance) before moving it - * to @list_lru + * to @lru. + * + * Return: true if the list was updated, false otherwise + */ +bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg); + +/** + * list_lru_add_obj: add an element to the lru list's tail + * @lru: the lru pointer + * @item: the item to be added. + * + * This function is similar to list_lru_add(), but the NUMA node and the + * memcg of the sublist is determined by @item list_head. This assumption is + * valid for slab objects LRU such as dentries, inodes, etc. * * Return value: true if the list was updated, false otherwise */ -bool list_lru_add(struct list_lru *lru, struct list_head *item); +bool list_lru_add_obj(struct list_lru *lru, struct list_head *item); /** - * list_lru_del: delete an element to the lru list - * @list_lru: the lru pointer + * list_lru_del: delete an element from the lru list + * @lru: the lru pointer * @item: the item to be deleted. + * @nid: the node id of the sublist to delete the item from. + * @memcg: the cgroup of the sublist to delete the item from. * - * This function works analogously as list_lru_add in terms of list + * This function works analogously as list_lru_add() in terms of list * manipulation. The comments about an element already pertaining to - * a list are also valid for list_lru_del. + * a list are also valid for list_lru_del(). * - * Return value: true if the list was updated, false otherwise + * Return: true if the list was updated, false otherwise */ -bool list_lru_del(struct list_lru *lru, struct list_head *item); +bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg); + +/** + * list_lru_del_obj: delete an element from the lru list + * @lru: the lru pointer + * @item: the item to be deleted. + * + * This function is similar to list_lru_del(), but the NUMA node and the + * memcg of the sublist is determined by @item list_head. This assumption is + * valid for slab objects LRU such as dentries, inodes, etc. + * + * Return value: true if the list was updated, false otherwise. + */ +bool list_lru_del_obj(struct list_lru *lru, struct list_head *item); /** * list_lru_count_one: return the number of objects currently held by @lru @@ -108,9 +140,11 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item); * @nid: the node id to count from. * @memcg: the cgroup to count from. * - * Always return a non-negative number, 0 for empty lists. There is no - * guarantee that the list is not updated while the count is being computed. - * Callers that want such a guarantee need to provide an outer lock. + * There is no guarantee that the list is not updated while the count is being + * computed. Callers that want such a guarantee need to provide an outer lock. + * + * Return: 0 for empty lists, otherwise the number of objects + * currently held by @lru. */ unsigned long list_lru_count_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg); @@ -136,12 +170,28 @@ static inline unsigned long list_lru_count(struct list_lru *lru) void list_lru_isolate(struct list_lru_one *list, struct list_head *item); void list_lru_isolate_move(struct list_lru_one *list, struct list_head *item, struct list_head *head); +/** + * list_lru_putback: undo list_lru_isolate + * @lru: the lru pointer. + * @item: the item to put back. + * @nid: the node id of the sublist to put the item back to. + * @memcg: the cgroup of the sublist to put the item back to. + * + * Put back an isolated item into its original LRU. Note that unlike + * list_lru_add, this does not increment the node LRU count (as + * list_lru_isolate does not originally decrement this count). + * + * Since we might have dropped the LRU lock in between, recompute list_lru_one + * from the node's id and memcg. + */ +void list_lru_putback(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg); typedef enum lru_status (*list_lru_walk_cb)(struct list_head *item, struct list_lru_one *list, spinlock_t *lock, void *cb_arg); /** - * list_lru_walk_one: walk a list_lru, isolating and disposing freeable items. + * list_lru_walk_one: walk a @lru, isolating and disposing freeable items. * @lru: the lru pointer. * @nid: the node id to scan from. * @memcg: the cgroup to scan from. @@ -150,24 +200,24 @@ typedef enum lru_status (*list_lru_walk_cb)(struct list_head *item, * @cb_arg: opaque type that will be passed to @isolate * @nr_to_walk: how many items to scan. * - * This function will scan all elements in a particular list_lru, calling the + * This function will scan all elements in a particular @lru, calling the * @isolate callback for each of those items, along with the current list * spinlock and a caller-provided opaque. The @isolate callback can choose to * drop the lock internally, but *must* return with the lock held. The callback - * will return an enum lru_status telling the list_lru infrastructure what to + * will return an enum lru_status telling the @lru infrastructure what to * do with the object being scanned. * - * Please note that nr_to_walk does not mean how many objects will be freed, + * Please note that @nr_to_walk does not mean how many objects will be freed, * just how many objects will be scanned. * - * Return value: the number of objects effectively removed from the LRU. + * Return: the number of objects effectively removed from the LRU. */ unsigned long list_lru_walk_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg, list_lru_walk_cb isolate, void *cb_arg, unsigned long *nr_to_walk); /** - * list_lru_walk_one_irq: walk a list_lru, isolating and disposing freeable items. + * list_lru_walk_one_irq: walk a @lru, isolating and disposing freeable items. * @lru: the lru pointer. * @nid: the node id to scan from. * @memcg: the cgroup to scan from. @@ -176,7 +226,7 @@ unsigned long list_lru_walk_one(struct list_lru *lru, * @cb_arg: opaque type that will be passed to @isolate * @nr_to_walk: how many items to scan. * - * Same as @list_lru_walk_one except that the spinlock is acquired with + * Same as list_lru_walk_one() except that the spinlock is acquired with * spin_lock_irq(). */ unsigned long list_lru_walk_one_irq(struct list_lru *lru, diff --git a/include/linux/llist.h b/include/linux/llist.h index 85bda2d02d65..dcb91e3bac1c 100644 --- a/include/linux/llist.h +++ b/include/linux/llist.h @@ -73,6 +73,33 @@ static inline void init_llist_head(struct llist_head *list) list->first = NULL; } +/** + * init_llist_node - initialize lock-less list node + * @node: the node to be initialised + * + * In cases where there is a need to test if a node is on + * a list or not, this initialises the node to clearly + * not be on any list. + */ +static inline void init_llist_node(struct llist_node *node) +{ + node->next = node; +} + +/** + * llist_on_list - test if a lock-list list node is on a list + * @node: the node to test + * + * When a node is on a list the ->next pointer will be NULL or + * some other node. It can never point to itself. We use that + * in init_llist_node() to record that a node is not on any list, + * and here to test whether it is on any list. + */ +static inline bool llist_on_list(const struct llist_node *node) +{ + return node->next != node; +} + /** * llist_entry - get the struct of this entry * @ptr: the &struct llist_node pointer. @@ -249,6 +276,21 @@ static inline struct llist_node *__llist_del_all(struct llist_head *head) extern struct llist_node *llist_del_first(struct llist_head *head); +/** + * llist_del_first_init - delete first entry from lock-list and mark is as being off-list + * @head: the head of lock-less list to delete from. + * + * This behave the same as llist_del_first() except that llist_init_node() is called + * on the returned node so that llist_on_list() will report false for the node. + */ +static inline struct llist_node *llist_del_first_init(struct llist_head *head) +{ + struct llist_node *n = llist_del_first(head); + + if (n) + init_llist_node(n); + return n; +} struct llist_node *llist_reverse_order(struct llist_node *head); #endif /* LLIST_H */ diff --git a/include/linux/lockd/bind.h b/include/linux/lockd/bind.h index c53c81242e72..e25943e485c5 100644 --- a/include/linux/lockd/bind.h +++ b/include/linux/lockd/bind.h @@ -21,11 +21,17 @@ struct svc_rqst; struct rpc_task; struct rpc_clnt; +struct module; -/* - * This is the set of functions for lockd->nfsd communication +/** + * struct nlmsvc_binding - lockd -> nfsd callback table + * @owner: module that provides this binding. + * @fopen: open a file by NFS file handle on behalf of an NLM request. + * @fclose: close a file that was previously opened via @fopen. + * Implementations MUST be semantically equivalent to fput(). */ struct nlmsvc_binding { + struct module *owner; __be32 (*fopen)(struct svc_rqst *, struct nfs_fh *, struct file **, @@ -33,7 +39,7 @@ struct nlmsvc_binding { void (*fclose)(struct file *); }; -extern const struct nlmsvc_binding *nlmsvc_ops; +extern const struct nlmsvc_binding __rcu *nlmsvc_ops; /* * Similar to nfs_client_initdata, but without the NFS-specific diff --git a/include/linux/lockd/lockd.h b/include/linux/lockd/lockd.h index 0f016d69c996..cf1f28a5329a 100644 --- a/include/linux/lockd/lockd.h +++ b/include/linux/lockd/lockd.h @@ -268,7 +268,7 @@ void nsm_release(struct nsm_handle *nsm); * This is used in garbage collection and resource reclaim * A return value != 0 means destroy the lock/block/share */ -typedef int (*nlm_host_match_fn_t)(void *cur, struct nlm_host *ref); +typedef int (*nlm_host_match_fn_t)(void *owner, struct nlm_host *ref); /* * Server-side lock handling @@ -375,12 +375,12 @@ static inline int nlm_privileged_requester(const struct svc_rqst *rqstp) static inline int nlm_compare_locks(const struct file_lock *fl1, const struct file_lock *fl2) { - return file_inode(fl1->fl_file) == file_inode(fl2->fl_file) - && fl1->fl_pid == fl2->fl_pid - && fl1->fl_owner == fl2->fl_owner + return file_inode(fl1->c.flc_file) == file_inode(fl2->c.flc_file) + && fl1->c.flc_pid == fl2->c.flc_pid + && fl1->c.flc_owner == fl2->c.flc_owner && fl1->fl_start == fl2->fl_start && fl1->fl_end == fl2->fl_end - &&(fl1->fl_type == fl2->fl_type || fl2->fl_type == F_UNLCK); + &&(fl1->c.flc_type == fl2->c.flc_type || fl2->c.flc_type == F_UNLCK); } extern const struct lock_manager_operations nlmsvc_lock_operations; diff --git a/include/linux/memory.h b/include/linux/memory.h index f53cfdaaaa41..11dbf013b099 100644 --- a/include/linux/memory.h +++ b/include/linux/memory.h @@ -19,6 +19,7 @@ #include #include #include +#include #define MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS) @@ -64,10 +65,20 @@ struct memory_group { }; }; +enum memory_block_state { + /* These states are exposed to userspace as text strings in sysfs */ + MEM_ONLINE = (1<<0), /* exposed to userspace */ + MEM_GOING_OFFLINE = (1<<1), /* exposed to userspace */ + MEM_OFFLINE = (1<<2), /* exposed to userspace */ + MEM_GOING_ONLINE = (1<<3), + MEM_CANCEL_ONLINE = (1<<4), + MEM_CANCEL_OFFLINE = (1<<5), +}; + struct memory_block { unsigned long start_section_nr; - unsigned long state; /* serialized by the dev->lock */ - int online_type; /* for passing data to online routine */ + enum memory_block_state state; /* serialized by the dev->lock */ + enum mmop online_type; /* for passing data to online routine */ int nid; /* NID for this memory block */ /* * The single zone of this memory block if all PFNs of this memory block @@ -89,14 +100,6 @@ int arch_get_memory_phys_device(unsigned long start_pfn); unsigned long memory_block_size_bytes(void); int set_memory_block_size_order(unsigned int order); -/* These states are exposed to userspace as text strings in sysfs */ -#define MEM_ONLINE (1<<0) /* exposed to userspace */ -#define MEM_GOING_OFFLINE (1<<1) /* exposed to userspace */ -#define MEM_OFFLINE (1<<2) /* exposed to userspace */ -#define MEM_GOING_ONLINE (1<<3) -#define MEM_CANCEL_ONLINE (1<<4) -#define MEM_CANCEL_OFFLINE (1<<5) - struct memory_notify { unsigned long start_pfn; unsigned long nr_pages; diff --git a/include/linux/memory_hotplug.h b/include/linux/memory_hotplug.h index 7d2076583494..41abb2624eb8 100644 --- a/include/linux/memory_hotplug.h +++ b/include/linux/memory_hotplug.h @@ -64,11 +64,8 @@ static inline void arch_refresh_nodedata(int nid, pg_data_t *pgdat) #endif /* CONFIG_NUMA */ #endif /* CONFIG_HAVE_ARCH_NODEDATA_EXTENSION */ -#ifdef CONFIG_MEMORY_HOTPLUG -struct page *pfn_to_online_page(unsigned long pfn); - /* Types for control the zone type of onlined and offlined memory */ -enum { +enum mmop { /* Offline the memory. */ MMOP_OFFLINE = 0, /* Online the memory. Zone depends, see default_zone_for_pfn(). */ @@ -79,6 +76,9 @@ enum { MMOP_ONLINE_MOVABLE, }; +#ifdef CONFIG_MEMORY_HOTPLUG +struct page *pfn_to_online_page(unsigned long pfn); + /* Flags for add_memory() and friends to specify memory hotplug details. */ typedef int __bitwise mhp_t; @@ -329,7 +329,7 @@ static inline void __remove_memory(u64 start, u64 size) {} #endif /* CONFIG_MEMORY_HOTREMOVE */ #ifdef CONFIG_MEMORY_HOTPLUG -extern void __ref free_area_init_core_hotplug(struct pglist_data *pgdat); +int __ref free_area_init_core_hotplug(struct pglist_data *pgdat); extern int __add_memory(int nid, u64 start, u64 size, mhp_t mhp_flags); extern int add_memory(int nid, u64 start, u64 size, mhp_t mhp_flags); extern int add_memory_resource(int nid, struct resource *resource, @@ -350,8 +350,8 @@ extern void sparse_remove_section(unsigned long pfn, unsigned long nr_pages, struct vmem_altmap *altmap); extern struct page *sparse_decode_mem_map(unsigned long coded_mem_map, unsigned long pnum); -extern struct zone *zone_for_pfn_range(int online_type, int nid, - struct memory_group *group, unsigned long start_pfn, +extern struct zone *zone_for_pfn_range(enum mmop online_type, + int nid, struct memory_group *group, unsigned long start_pfn, unsigned long nr_pages); extern int arch_create_linear_mapping(int nid, u64 start, u64 size, struct mhp_params *params); diff --git a/include/linux/mlx5/cq.h b/include/linux/mlx5/cq.h index cb15308b5cb0..804432fd0f27 100644 --- a/include/linux/mlx5/cq.h +++ b/include/linux/mlx5/cq.h @@ -183,6 +183,7 @@ static inline void mlx5_cq_put(struct mlx5_core_cq *cq) complete(&cq->free); } +void mlx5_add_cq_to_tasklet(struct mlx5_core_cq *cq, struct mlx5_eqe *eqe); int mlx5_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq, u32 *in, int inlen, u32 *out, int outlen); int mlx5_core_create_cq(struct mlx5_core_dev *dev, struct mlx5_core_cq *cq, diff --git a/include/linux/mlx5/mlx5_ifc.h b/include/linux/mlx5/mlx5_ifc.h index c59fd31719a1..573461c9dced 100644 --- a/include/linux/mlx5/mlx5_ifc.h +++ b/include/linux/mlx5/mlx5_ifc.h @@ -11320,18 +11320,7 @@ struct mlx5_ifc_mcia_reg_bits { u8 reserved_at_60[0x20]; - u8 dword_0[0x20]; - u8 dword_1[0x20]; - u8 dword_2[0x20]; - u8 dword_3[0x20]; - u8 dword_4[0x20]; - u8 dword_5[0x20]; - u8 dword_6[0x20]; - u8 dword_7[0x20]; - u8 dword_8[0x20]; - u8 dword_9[0x20]; - u8 dword_10[0x20]; - u8 dword_11[0x20]; + u8 dwords[0x400]; }; struct mlx5_ifc_dcbx_param_bits { diff --git a/include/linux/mm.h b/include/linux/mm.h index 09669ea23084..c365c33b9c3a 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -3313,7 +3313,7 @@ extern int insert_vm_struct(struct mm_struct *, struct vm_area_struct *); extern void unlink_file_vma(struct vm_area_struct *); extern struct vm_area_struct *copy_vma(struct vm_area_struct **, unsigned long addr, unsigned long len, pgoff_t pgoff, - bool *need_rmap_locks); + bool *need_rmap_locks, bool keep_source); extern void exit_mmap(struct mm_struct *); static inline int check_data_rlimit(unsigned long rlim, diff --git a/include/linux/mount.h b/include/linux/mount.h index 4f40b40306d0..d3425c44de81 100644 --- a/include/linux/mount.h +++ b/include/linux/mount.h @@ -51,7 +51,7 @@ struct path; #define MNT_INTERNAL_FLAGS (MNT_SHARED | MNT_WRITE_HOLD | MNT_INTERNAL | \ MNT_DOOMED | MNT_SYNC_UMOUNT | MNT_MARKED | \ - MNT_CURSOR) + MNT_CURSOR | MNT_UMOUNT) #define MNT_INTERNAL 0x4000 diff --git a/include/linux/namei.h b/include/linux/namei.h index c1ff032607dd..e95663f1e048 100644 --- a/include/linux/namei.h +++ b/include/linux/namei.h @@ -12,11 +12,6 @@ enum { MAX_NESTED_LINKS = 8 }; #define MAXSYMLINKS 40 -/* - * Type of the last component on LOOKUP_PARENT - */ -enum {LAST_NORM, LAST_ROOT, LAST_DOT, LAST_DOTDOT}; - /* pathwalk mode */ #define LOOKUP_FOLLOW 0x0001 /* follow links at the end */ #define LOOKUP_DIRECTORY 0x0002 /* require a directory */ @@ -68,7 +63,7 @@ extern struct dentry *user_path_create(int, const char __user *, struct path *, extern void done_path_create(struct path *, struct dentry *); extern struct dentry *kern_path_locked(const char *, struct path *); int vfs_path_parent_lookup(struct filename *filename, unsigned int flags, - struct path *parent, struct qstr *last, int *type, + struct path *parent, struct qstr *last, const struct path *root); int vfs_path_lookup(struct dentry *, struct vfsmount *, const char *, unsigned int, struct path *); diff --git a/include/linux/netdevice.h b/include/linux/netdevice.h index 3510177cd1ba..61f1a64b7229 100644 --- a/include/linux/netdevice.h +++ b/include/linux/netdevice.h @@ -1977,6 +1977,8 @@ enum netdev_reg_state { * @qdisc_hash: qdisc hash table * @watchdog_timeo: Represents the timeout that is used by * the watchdog (see dev_watchdog()) + * @watchdog_lock: protect watchdog_ref_held + * @watchdog_ref_held: True if the watchdog device ref is taken. * @watchdog_timer: List of timers * * @proto_down_reason: reason a netdev interface is held down @@ -2320,6 +2322,8 @@ struct net_device { /* These may be needed for future network-power-down code. */ struct timer_list watchdog_timer; int watchdog_timeo; + spinlock_t watchdog_lock; + bool watchdog_ref_held; u32 proto_down_reason; @@ -4027,12 +4031,13 @@ int dev_change_flags(struct net_device *dev, unsigned int flags, int dev_set_alias(struct net_device *, const char *, size_t); int dev_get_alias(const struct net_device *, char *, size_t); int __dev_change_net_namespace(struct net_device *dev, struct net *net, - const char *pat, int new_ifindex); + const char *pat, int new_ifindex, + struct netlink_ext_ack *extack); static inline int dev_change_net_namespace(struct net_device *dev, struct net *net, const char *pat) { - return __dev_change_net_namespace(dev, net, pat, 0); + return __dev_change_net_namespace(dev, net, pat, 0, NULL); } int __dev_set_mtu(struct net_device *, int); int dev_set_mtu(struct net_device *, int); diff --git a/include/linux/netfilter.h b/include/linux/netfilter.h index cc5a2a220af8..278efb0ab696 100644 --- a/include/linux/netfilter.h +++ b/include/linux/netfilter.h @@ -22,6 +22,16 @@ static inline int NF_DROP_GETERR(int verdict) return -(verdict >> NF_VERDICT_QBITS); } +static __always_inline int +NF_DROP_REASON(struct sk_buff *skb, enum skb_drop_reason reason, u32 err) +{ + BUILD_BUG_ON(err > 0xffff); + + kfree_skb_reason(skb, reason); + + return ((err << 16) | NF_STOLEN); +} + static inline int nf_inet_addr_cmp(const union nf_inet_addr *a1, const union nf_inet_addr *a2) { @@ -85,6 +95,8 @@ enum nf_hook_ops_type { }; struct nf_hook_ops { + struct rcu_head rcu; + /* User fills in from here down. */ nf_hookfn *hook; struct net_device *dev; diff --git a/include/linux/netfilter/x_tables.h b/include/linux/netfilter/x_tables.h index 5897f3dbaf7c..3716b42a4ad6 100644 --- a/include/linux/netfilter/x_tables.h +++ b/include/linux/netfilter/x_tables.h @@ -307,6 +307,7 @@ struct xt_counters *xt_counters_alloc(unsigned int counters); struct xt_table *xt_register_table(struct net *net, const struct xt_table *table, + const struct nf_hook_ops *template_ops, struct xt_table_info *bootstrap, struct xt_table_info *newinfo); void *xt_unregister_table(struct xt_table *table); diff --git a/include/linux/ntb.h b/include/linux/ntb.h index 191b524e5c0d..5c29e904a9c5 100644 --- a/include/linux/ntb.h +++ b/include/linux/ntb.h @@ -256,6 +256,7 @@ static inline int ntb_ctx_ops_is_valid(const struct ntb_ctx_ops *ops) * @msg_clear_mask: See ntb_msg_clear_mask(). * @msg_read: See ntb_msg_read(). * @peer_msg_write: See ntb_peer_msg_write(). + * @get_dma_dev: See ntb_get_dma_dev(). */ struct ntb_dev_ops { int (*port_number)(struct ntb_dev *ntb); @@ -329,6 +330,7 @@ struct ntb_dev_ops { int (*msg_clear_mask)(struct ntb_dev *ntb, u64 mask_bits); u32 (*msg_read)(struct ntb_dev *ntb, int *pidx, int midx); int (*peer_msg_write)(struct ntb_dev *ntb, int pidx, int midx, u32 msg); + struct device *(*get_dma_dev)(struct ntb_dev *ntb); }; static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops) @@ -391,6 +393,8 @@ static inline int ntb_dev_ops_is_valid(const struct ntb_dev_ops *ops) /* !ops->msg_clear_mask == !ops->msg_count && */ !ops->msg_read == !ops->msg_count && !ops->peer_msg_write == !ops->msg_count && + + /* ops->get_dma_dev is optional */ 1; } @@ -1563,6 +1567,26 @@ static inline int ntb_peer_msg_write(struct ntb_dev *ntb, int pidx, int midx, return ntb->ops->peer_msg_write(ntb, pidx, midx, msg); } +/** + * ntb_get_dma_dev() - get the device to use for DMA allocations/mappings + * @ntb: NTB device context. + * + * Return a struct device suitable for DMA API allocations and mappings. + * This is typically the parent of the NTB device, but may be overridden by a + * driver by implementing .get_dma_dev(). + * + * Drivers that implement .get_dma_dev() must return a non-NULL pointer. + * + * Return: device pointer to use for DMA operations. + */ +static inline struct device *ntb_get_dma_dev(struct ntb_dev *ntb) +{ + if (!ntb->ops->get_dma_dev) + return ntb->dev.parent; + + return ntb->ops->get_dma_dev(ntb); +} + /** * ntb_peer_resource_idx() - get a resource index for a given peer idx * @ntb: NTB device context. diff --git a/include/linux/nvme.h b/include/linux/nvme.h index 815850270be5..c50a9bd13ee1 100644 --- a/include/linux/nvme.h +++ b/include/linux/nvme.h @@ -1099,8 +1099,8 @@ enum { NVME_ZRASF_ZONE_STATE_IMP_OPEN = 0x02, NVME_ZRASF_ZONE_STATE_EXP_OPEN = 0x03, NVME_ZRASF_ZONE_STATE_CLOSED = 0x04, - NVME_ZRASF_ZONE_STATE_READONLY = 0x05, - NVME_ZRASF_ZONE_STATE_FULL = 0x06, + NVME_ZRASF_ZONE_STATE_FULL = 0x05, + NVME_ZRASF_ZONE_STATE_READONLY = 0x06, NVME_ZRASF_ZONE_STATE_OFFLINE = 0x07, NVME_REPORT_ZONE_PARTIAL = 1, }; diff --git a/include/linux/nvmem-consumer.h b/include/linux/nvmem-consumer.h index 526025561df1..39959d9d4465 100644 --- a/include/linux/nvmem-consumer.h +++ b/include/linux/nvmem-consumer.h @@ -242,7 +242,6 @@ struct nvmem_cell *of_nvmem_cell_get(struct device_node *np, const char *id); struct nvmem_device *of_nvmem_device_get(struct device_node *np, const char *name); -struct device_node *of_nvmem_layout_get_container(struct nvmem_device *nvmem); #else static inline struct nvmem_cell *of_nvmem_cell_get(struct device_node *np, const char *id) @@ -255,12 +254,6 @@ static inline struct nvmem_device *of_nvmem_device_get(struct device_node *np, { return ERR_PTR(-EOPNOTSUPP); } - -static inline struct device_node * -of_nvmem_layout_get_container(struct nvmem_device *nvmem) -{ - return NULL; -} #endif /* CONFIG_NVMEM && CONFIG_OF */ #endif /* ifndef _LINUX_NVMEM_CONSUMER_H */ diff --git a/include/linux/nvmem-provider.h b/include/linux/nvmem-provider.h index 9a015e4d428c..7164401d7474 100644 --- a/include/linux/nvmem-provider.h +++ b/include/linux/nvmem-provider.h @@ -9,6 +9,7 @@ #ifndef _LINUX_NVMEM_PROVIDER_H #define _LINUX_NVMEM_PROVIDER_H +#include #include #include #include @@ -160,12 +161,11 @@ struct nvmem_cell_table { /** * struct nvmem_layout - NVMEM layout definitions * - * @name: Layout name. - * @of_match_table: Open firmware match table. - * @add_cells: Called to populate the layout using - * nvmem_add_one_cell(). - * @owner: Pointer to struct module. - * @node: List node. + * @dev: Device-model layout device. + * @nvmem: The underlying NVMEM device + * @add_cells: Will be called if a nvmem device is found which + * has this layout. The function will add layout + * specific cells with nvmem_add_one_cell(). * * A nvmem device can hold a well defined structure which can just be * evaluated during runtime. For example a TLV list, or a list of "name=val" @@ -173,13 +173,15 @@ struct nvmem_cell_table { * cells. */ struct nvmem_layout { - const char *name; - const struct of_device_id *of_match_table; + struct device dev; + struct nvmem_device *nvmem; int (*add_cells)(struct device *dev, struct nvmem_device *nvmem); +}; - /* private */ - struct module *owner; - struct list_head node; +struct nvmem_layout_driver { + struct device_driver driver; + int (*probe)(struct nvmem_layout *layout); + void (*remove)(struct nvmem_layout *layout); }; #if IS_ENABLED(CONFIG_NVMEM) @@ -196,11 +198,15 @@ void nvmem_del_cell_table(struct nvmem_cell_table *table); int nvmem_add_one_cell(struct nvmem_device *nvmem, const struct nvmem_cell_info *info); -int __nvmem_layout_register(struct nvmem_layout *layout, struct module *owner); -#define nvmem_layout_register(layout) \ - __nvmem_layout_register(layout, THIS_MODULE) +int nvmem_layout_register(struct nvmem_layout *layout); void nvmem_layout_unregister(struct nvmem_layout *layout); +int nvmem_layout_driver_register(struct nvmem_layout_driver *drv); +void nvmem_layout_driver_unregister(struct nvmem_layout_driver *drv); +#define module_nvmem_layout_driver(__nvmem_layout_driver) \ + module_driver(__nvmem_layout_driver, nvmem_layout_driver_register, \ + nvmem_layout_driver_unregister) + const void *nvmem_layout_get_match_data(struct nvmem_device *nvmem, struct nvmem_layout *layout); @@ -243,8 +249,25 @@ nvmem_layout_get_match_data(struct nvmem_device *nvmem, #endif /* CONFIG_NVMEM */ -#define module_nvmem_layout_driver(__layout_driver) \ - module_driver(__layout_driver, nvmem_layout_register, \ - nvmem_layout_unregister) +#if IS_ENABLED(CONFIG_NVMEM) && IS_ENABLED(CONFIG_OF) + +/** + * of_nvmem_layout_get_container() - Get OF node of layout container + * + * @nvmem: nvmem device + * + * Return: a node pointer with refcount incremented or NULL if no + * container exists. Use of_node_put() on it when done. + */ +struct device_node *of_nvmem_layout_get_container(struct nvmem_device *nvmem); + +#else /* CONFIG_NVMEM && CONFIG_OF */ + +static inline struct device_node *of_nvmem_layout_get_container(struct nvmem_device *nvmem) +{ + return NULL; +} + +#endif /* CONFIG_NVMEM && CONFIG_OF */ #endif /* ifndef _LINUX_NVMEM_PROVIDER_H */ diff --git a/include/linux/of_device.h b/include/linux/of_device.h index 2c7a3d4bc775..a72661e47faa 100644 --- a/include/linux/of_device.h +++ b/include/linux/of_device.h @@ -40,6 +40,9 @@ static inline int of_dma_configure(struct device *dev, { return of_dma_configure_id(dev, np, force_dma, NULL); } + +void of_device_make_bus_id(struct device *dev); + #else /* CONFIG_OF */ static inline int of_driver_match_device(struct device *dev, @@ -82,6 +85,9 @@ static inline int of_dma_configure(struct device *dev, { return 0; } + +static inline void of_device_make_bus_id(struct device *dev) {} + #endif /* CONFIG_OF */ #endif /* _LINUX_OF_DEVICE_H */ diff --git a/include/linux/path.h b/include/linux/path.h index 475225a03d0d..ca073e70decd 100644 --- a/include/linux/path.h +++ b/include/linux/path.h @@ -24,4 +24,13 @@ static inline void path_put_init(struct path *path) *path = (struct path) { }; } +/* + * Cleanup macro for use with __free(path_put). Avoids dereference and + * copying @path unlike DEFINE_FREE(). path_put() will handle the empty + * path correctly just ensure @path is initialized: + * + * struct path path __free(path_put) = {}; + */ +#define __free_path_put path_put + #endif /* _LINUX_PATH_H */ diff --git a/include/linux/pci.h b/include/linux/pci.h index db4d82aae134..5c8f0a8e02ea 100644 --- a/include/linux/pci.h +++ b/include/linux/pci.h @@ -72,12 +72,24 @@ /* return bus from PCI devid = ((u16)bus_number) << 8) | devfn */ #define PCI_BUS_NUM(x) (((x) >> 8) & 0xff) +/* + * PCI_SLOT_ALL_DEVICES indicates a slot that covers all devices on the bus. + * Used for PCIe hotplug where the physical slot is the entire secondary bus, + * and, if ARI Forwarding is enabled, functions may appear to be on multiple + * devices. + */ +#define PCI_SLOT_ALL_DEVICES 0xfeff + +/* Used to identify a slot as a placeholder */ +#define PCI_SLOT_PLACEHOLDER 0xffff + /* pci_slot represents a physical slot */ struct pci_slot { struct pci_bus *bus; /* Bus this slot is on */ struct list_head list; /* Node in list of slots */ struct hotplug_slot *hotplug; /* Hotplug info (move here) */ - unsigned char number; /* PCI_SLOT(pci_dev->devfn) */ + u16 number; /* Device nr, or PCI_SLOT_ALL_DEVICES */ + unsigned int per_func_slot:1; /* Allow per function slot */ struct kobject kobj; }; @@ -529,12 +541,6 @@ struct pci_dev { u16 acs_cap; /* ACS Capability offset */ phys_addr_t rom; /* Physical address if not from BAR */ size_t romlen; /* Length if not from BAR */ - /* - * Driver name to force a match. Do not set directly, because core - * frees it. Use driver_set_override() to set or clear it. - */ - const char *driver_override; - unsigned long priv_flags; /* Private flags for the PCI driver */ /* These methods index pci_reset_fn_methods[] */ @@ -886,6 +892,7 @@ struct module; * function returns zero when the driver chooses to * take "ownership" of the device or an error code * (negative number) otherwise. + * The pci_device_id parameter is only valid during probe. * The probe function always gets called from process * context, so it can sleep. * @remove: The remove() function gets called whenever a device diff --git a/include/linux/perf/riscv_pmu.h b/include/linux/perf/riscv_pmu.h index 43282e22ebe1..b8146420425e 100644 --- a/include/linux/perf/riscv_pmu.h +++ b/include/linux/perf/riscv_pmu.h @@ -47,7 +47,7 @@ struct riscv_pmu { irqreturn_t (*handle_irq)(int irq_num, void *dev); - unsigned long cmask; + DECLARE_BITMAP(cmask, RISCV_MAX_COUNTERS); u64 (*ctr_read)(struct perf_event *event); int (*ctr_get_idx)(struct perf_event *event); int (*ctr_get_width)(int idx); diff --git a/include/linux/perf_event.h b/include/linux/perf_event.h index 3fdc58cc5783..72642a021964 100644 --- a/include/linux/perf_event.h +++ b/include/linux/perf_event.h @@ -453,7 +453,7 @@ struct pmu { * context-switches callback */ void (*sched_task) (struct perf_event_pmu_context *pmu_ctx, - bool sched_in); + struct task_struct *task, bool sched_in); /* * Kmem cache of PMU specific data @@ -786,7 +786,6 @@ struct perf_event { struct irq_work pending_irq; struct callback_head pending_task; unsigned int pending_work; - struct rcuwait pending_work_wait; atomic_t event_limit; diff --git a/include/linux/platform_data/x86/asus-wmi.h b/include/linux/platform_data/x86/asus-wmi.h index 8e48bdeb5549..5fa03fbe536a 100644 --- a/include/linux/platform_data/x86/asus-wmi.h +++ b/include/linux/platform_data/x86/asus-wmi.h @@ -111,6 +111,9 @@ /* Charging mode - 1=Barrel, 2=USB */ #define ASUS_WMI_DEVID_CHARGE_MODE 0x0012006C +/* MCU powersave mode */ +#define ASUS_WMI_DEVID_MCU_POWERSAVE 0x001200E2 + /* epu is connected? 1 == true */ #define ASUS_WMI_DEVID_EGPU_CONNECTED 0x00090018 /* egpu on/off */ diff --git a/include/linux/property.h b/include/linux/property.h index d32b8052e086..e91b4d759758 100644 --- a/include/linux/property.h +++ b/include/linux/property.h @@ -124,6 +124,18 @@ static inline bool device_is_compatible(const struct device *dev, const char *co return fwnode_device_is_compatible(dev_fwnode(dev), compat); } +int fwnode_property_match_property_string(const struct fwnode_handle *fwnode, + const char *propname, + const char * const *array, size_t n); + +static inline +int device_property_match_property_string(const struct device *dev, + const char *propname, + const char * const *array, size_t n) +{ + return fwnode_property_match_property_string(dev_fwnode(dev), propname, array, n); +} + int fwnode_property_get_reference_args(const struct fwnode_handle *fwnode, const char *prop, const char *nargs_prop, unsigned int nargs, unsigned int index, diff --git a/include/linux/rcupdate.h b/include/linux/rcupdate.h index 27531a0b3a6e..95f783de1fbb 100644 --- a/include/linux/rcupdate.h +++ b/include/linux/rcupdate.h @@ -908,6 +908,20 @@ static inline notrace void rcu_read_unlock_sched_notrace(void) preempt_enable_notrace(); } +static __always_inline void rcu_read_lock_dont_migrate(void) +{ + if (IS_ENABLED(CONFIG_PREEMPT_RCU)) + migrate_disable(); + rcu_read_lock(); +} + +static inline void rcu_read_unlock_migrate(void) +{ + rcu_read_unlock(); + if (IS_ENABLED(CONFIG_PREEMPT_RCU)) + migrate_enable(); +} + /** * RCU_INIT_POINTER() - initialize an RCU protected pointer * @p: The pointer to be initialized. diff --git a/include/linux/rcuwait.h b/include/linux/rcuwait.h index 27343424225c..9ad134a04b41 100644 --- a/include/linux/rcuwait.h +++ b/include/linux/rcuwait.h @@ -4,18 +4,7 @@ #include #include - -/* - * rcuwait provides a way of blocking and waking up a single - * task in an rcu-safe manner. - * - * The only time @task is non-nil is when a user is blocked (or - * checking if it needs to) on a condition, and reset as soon as we - * know that the condition has succeeded and are awoken. - */ -struct rcuwait { - struct task_struct __rcu *task; -}; +#include #define __RCUWAIT_INITIALIZER(name) \ { .task = NULL, } diff --git a/include/linux/sched.h b/include/linux/sched.h index 4be7af82f813..66b554f5fdb2 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1739,7 +1739,7 @@ static inline int is_global_init(struct task_struct *tsk) return task_tgid_nr(tsk) == 1; } -extern struct pid *cad_pid; +extern struct pid __rcu *cad_pid; /* * Per process flags diff --git a/include/linux/sched/signal.h b/include/linux/sched/signal.h index 0014d3adaf84..daa95d325189 100644 --- a/include/linux/sched/signal.h +++ b/include/linux/sched/signal.h @@ -568,10 +568,7 @@ static inline sigset_t *sigmask_to_save(void) return res; } -static inline int kill_cad_pid(int sig, int priv) -{ - return kill_pid(cad_pid, sig, priv); -} +int kill_cad_pid(int sig, int priv); /* These can be the second arg to send_sig_info/send_group_sig_info. */ #define SEND_SIG_NOINFO ((struct kernel_siginfo *) 0) diff --git a/include/linux/sched/user.h b/include/linux/sched/user.h index 4cc52698e214..8d7e5521f7cd 100644 --- a/include/linux/sched/user.h +++ b/include/linux/sched/user.h @@ -25,7 +25,8 @@ struct user_struct { #if defined(CONFIG_PERF_EVENTS) || defined(CONFIG_BPF_SYSCALL) || \ defined(CONFIG_NET) || defined(CONFIG_IO_URING) || \ - defined(CONFIG_VFIO_PCI_ZDEV_KVM) || IS_ENABLED(CONFIG_IOMMUFD) + defined(CONFIG_VFIO_PCI_ZDEV_KVM) || IS_ENABLED(CONFIG_IOMMUFD) || \ + defined(CONFIG_SECRETMEM) atomic_long_t locked_vm; #endif #ifdef CONFIG_WATCH_QUEUE diff --git a/include/linux/skbuff.h b/include/linux/skbuff.h index 60462d64c6e0..171775202bd9 100644 --- a/include/linux/skbuff.h +++ b/include/linux/skbuff.h @@ -1500,7 +1500,7 @@ __skb_set_sw_hash(struct sk_buff *skb, __u32 hash, bool is_l4) __skb_set_hash(skb, hash, true, is_l4); } -void __skb_get_hash(struct sk_buff *skb); +void __skb_get_hash_net(const struct net *net, struct sk_buff *skb); u32 __skb_get_hash_symmetric(const struct sk_buff *skb); u32 skb_get_poff(const struct sk_buff *skb); u32 __skb_get_poff(const struct sk_buff *skb, const void *data, @@ -1580,10 +1580,18 @@ void skb_flow_dissect_hash(const struct sk_buff *skb, struct flow_dissector *flow_dissector, void *target_container); +static inline __u32 skb_get_hash_net(const struct net *net, struct sk_buff *skb) +{ + if (!skb->l4_hash && !skb->sw_hash) + __skb_get_hash_net(net, skb); + + return skb->hash; +} + static inline __u32 skb_get_hash(struct sk_buff *skb) { if (!skb->l4_hash && !skb->sw_hash) - __skb_get_hash(skb); + __skb_get_hash_net(NULL, skb); return skb->hash; } diff --git a/include/linux/skmsg.h b/include/linux/skmsg.h index ed95128d29c6..285230488e90 100644 --- a/include/linux/skmsg.h +++ b/include/linux/skmsg.h @@ -549,20 +549,6 @@ static inline bool sk_psock_strp_enabled(struct sk_psock *psock) return !!psock->saved_data_ready; } -/* for tcp only, sk is locked */ -static inline ssize_t sk_psock_msg_inq(struct sock *sk) -{ - struct sk_psock *psock; - ssize_t inq = 0; - - psock = sk_psock_get(sk); - if (likely(psock)) { - inq = sk_psock_get_msg_len_nolock(psock); - sk_psock_put(sk, psock); - } - return inq; -} - /* for udp only, sk is not locked */ static inline ssize_t sk_msg_first_len(struct sock *sk) { diff --git a/include/linux/soc/qcom/geni-se.h b/include/linux/soc/qcom/geni-se.h index 29e06905bc1f..543feee0965e 100644 --- a/include/linux/soc/qcom/geni-se.h +++ b/include/linux/soc/qcom/geni-se.h @@ -287,17 +287,12 @@ struct geni_se { #define QUP_SE_VERSION_2_5 0x20050000 /* - * Define bandwidth thresholds that cause the underlying Core 2X interconnect - * clock to run at the named frequency. These baseline values are recommended - * by the hardware team, and are not dynamically scaled with GENI bandwidth - * beyond basic on/off. + * QUP Core 2X clock votes used by GENI clients through the "qup-core" ICC + * path. Values are in Bps and must be converted with Bps_to_icc() before + * setting avg_bw. */ -#define CORE_2X_19_2_MHZ 960 -#define CORE_2X_50_MHZ 2500 -#define CORE_2X_100_MHZ 5000 -#define CORE_2X_150_MHZ 7500 -#define CORE_2X_200_MHZ 10000 -#define CORE_2X_236_MHZ 16383 +#define CORE_2X_19_2_MHZ 9600000 +#define CORE_2X_50_MHZ 25000000 #define GENI_DEFAULT_BW Bps_to_icc(1000) diff --git a/include/linux/spi/spi.h b/include/linux/spi/spi.h index e5baf43bcfbb..3f6ca8052f63 100644 --- a/include/linux/spi/spi.h +++ b/include/linux/spi/spi.h @@ -153,10 +153,6 @@ extern void spi_transfer_cs_change_delay_exec(struct spi_message *msg, * @modalias: Name of the driver to use with this device, or an alias * for that name. This appears in the sysfs "modalias" attribute * for driver coldplugging, and in uevents used for hotplugging - * @driver_override: If the name of a driver is written to this attribute, then - * the device will bind to the named driver and only the named driver. - * Do not set directly, because core frees it; use driver_set_override() to - * set or clear it. * @cs_gpiod: GPIO descriptor of the chipselect line (optional, NULL when * not using a GPIO line) * @word_delay: delay to be inserted between consecutive @@ -212,7 +208,6 @@ struct spi_device { void *controller_state; void *controller_data; char modalias[SPI_NAME_SIZE]; - const char *driver_override; struct gpio_desc *cs_gpiod; /* Chip select GPIO descriptor */ struct spi_delay word_delay; /* Inter-word delay */ /* CS delays */ diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index 447133171d95..bad5fd0f1ddb 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -11,6 +11,7 @@ #ifndef _LINUX_SRCU_TINY_H #define _LINUX_SRCU_TINY_H +#include #include struct srcu_struct { @@ -24,18 +25,21 @@ struct srcu_struct { struct rcu_head *srcu_cb_head; /* Pending callbacks: Head. */ struct rcu_head **srcu_cb_tail; /* Pending callbacks: Tail. */ struct work_struct srcu_work; /* For driving grace periods. */ + struct irq_work srcu_irq_work; /* Defer schedule_work() to irq work. */ #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map; #endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ }; void srcu_drive_gp(struct work_struct *wp); +void srcu_tiny_irq_work(struct irq_work *irq_work); #define __SRCU_STRUCT_INIT(name, __ignored, ___ignored) \ { \ .srcu_wq = __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \ .srcu_cb_tail = &name.srcu_cb_head, \ .srcu_work = __WORK_INITIALIZER(name.srcu_work, srcu_drive_gp), \ + .srcu_irq_work = { .func = srcu_tiny_irq_work }, \ __SRCU_DEP_MAP_INIT(name) \ } diff --git a/include/linux/string.h b/include/linux/string.h index bbcaceb8fb6f..2722badba6ac 100644 --- a/include/linux/string.h +++ b/include/linux/string.h @@ -232,6 +232,7 @@ extern char *kstrndup(const char *s, size_t len, gfp_t gfp); extern void *kmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); extern void *kvmemdup(const void *src, size_t len, gfp_t gfp) __realloc_size(2); extern char *kmemdup_nul(const char *s, size_t len, gfp_t gfp); +extern void *kmemdup_array(const void *src, size_t element_size, size_t count, gfp_t gfp); extern char **argv_split(gfp_t gfp, const char *str, int *argcp); extern void argv_free(char **argv); diff --git a/include/linux/sunrpc/svc_rdma.h b/include/linux/sunrpc/svc_rdma.h index a5ee0af2a310..7f7605867778 100644 --- a/include/linux/sunrpc/svc_rdma.h +++ b/include/linux/sunrpc/svc_rdma.h @@ -137,6 +137,7 @@ struct svc_rdma_recv_ctxt { struct xdr_stream rc_stream; u32 rc_byte_len; unsigned int rc_page_count; + unsigned int rc_maxpages; u32 rc_inv_rkey; __be32 rc_msgtype; diff --git a/include/linux/tty.h b/include/linux/tty.h index f002d0f25db7..e4b17886d998 100644 --- a/include/linux/tty.h +++ b/include/linux/tty.h @@ -173,6 +173,7 @@ struct tty_operations; * @write_wait: concurrent writers are waiting in this queue until they are * allowed to write * @read_wait: readers wait for data in this queue + * @break_wait: wait queue for timed breaks * @hangup_work: normally a work to perform a hangup (do_tty_hangup()); while * freeing the tty, (re)used to release_one_tty() * @disc_data: pointer to @ldisc's private data (e.g. to &struct n_tty_data) @@ -237,6 +238,7 @@ struct tty_struct { struct fasync_struct *fasync; wait_queue_head_t write_wait; wait_queue_head_t read_wait; + wait_queue_head_t break_wait; struct work_struct hangup_work; void *disc_data; void *driver_data; diff --git a/include/linux/tty_driver.h b/include/linux/tty_driver.h index b4f99f6a5385..91aa2d5b15da 100644 --- a/include/linux/tty_driver.h +++ b/include/linux/tty_driver.h @@ -542,6 +542,11 @@ static inline void tty_set_operations(struct tty_driver *driver, * Do not create numbered ``/dev`` nodes. For example, create * ``/dev/ttyprintk`` and not ``/dev/ttyprintk0``. Applicable only when a * driver for a single tty device is being allocated. + * + * TTY_DRIVER_RESET_SAVED_TERMIOS + * Reset any saved termios settings on device registration when reusing a + * minor number. Must only be set by drivers that guarantee that the minor + * number is no longer in use. */ #define TTY_DRIVER_INSTALLED 0x0001 #define TTY_DRIVER_RESET_TERMIOS 0x0002 @@ -551,6 +556,7 @@ static inline void tty_set_operations(struct tty_driver *driver, #define TTY_DRIVER_HARDWARE_BREAK 0x0020 #define TTY_DRIVER_DYNAMIC_ALLOC 0x0040 #define TTY_DRIVER_UNNUMBERED_NODE 0x0080 +#define TTY_DRIVER_RESET_SAVED_TERMIOS 0x0200 /* tty driver types */ #define TTY_DRIVER_TYPE_SYSTEM 0x0001 diff --git a/include/linux/tty_port.h b/include/linux/tty_port.h index 0b97e562894b..6db28a4ec453 100644 --- a/include/linux/tty_port.h +++ b/include/linux/tty_port.h @@ -235,7 +235,8 @@ bool tty_port_carrier_raised(struct tty_port *port); void tty_port_raise_dtr_rts(struct tty_port *port); void tty_port_lower_dtr_rts(struct tty_port *port); void tty_port_hangup(struct tty_port *port); -void __tty_port_tty_hangup(struct tty_port *port, bool check_clocal, bool async); +void tty_port_tty_hangup(struct tty_port *port, bool check_clocal); +void tty_port_tty_vhangup(struct tty_port *port); void tty_port_tty_wakeup(struct tty_port *port); int tty_port_block_til_ready(struct tty_port *port, struct tty_struct *tty, struct file *filp); @@ -254,23 +255,4 @@ static inline int tty_port_users(struct tty_port *port) return port->count + port->blocked_open; } -/** - * tty_port_tty_hangup - helper to hang up a tty asynchronously - * @port: tty port - * @check_clocal: hang only ttys with %CLOCAL unset? - */ -static inline void tty_port_tty_hangup(struct tty_port *port, bool check_clocal) -{ - __tty_port_tty_hangup(port, check_clocal, true); -} - -/** - * tty_port_tty_vhangup - helper to hang up a tty synchronously - * @port: tty port - */ -static inline void tty_port_tty_vhangup(struct tty_port *port) -{ - __tty_port_tty_hangup(port, false, false); -} - #endif diff --git a/include/linux/types.h b/include/linux/types.h index 78d87c751ff5..ecacc613cfea 100644 --- a/include/linux/types.h +++ b/include/linux/types.h @@ -245,5 +245,17 @@ typedef void (*swap_func_t)(void *a, void *b, int size); typedef int (*cmp_r_func_t)(const void *a, const void *b, const void *priv); typedef int (*cmp_func_t)(const void *a, const void *b); +/* + * rcuwait provides a way of blocking and waking up a single + * task in an rcu-safe manner. + * + * The only time @task is non-nil is when a user is blocked (or + * checking if it needs to) on a condition, and reset as soon as we + * know that the condition has succeeded and are awoken. + */ +struct rcuwait { + struct task_struct __rcu *task; +}; + #endif /* __ASSEMBLY__ */ #endif /* _LINUX_TYPES_H */ diff --git a/include/linux/usb/gadget.h b/include/linux/usb/gadget.h index 1a13d0c01b0a..6ec7f94c5adc 100644 --- a/include/linux/usb/gadget.h +++ b/include/linux/usb/gadget.h @@ -54,6 +54,7 @@ struct usb_ep; * @short_not_ok: When reading data, makes short packets be * treated as errors (queue stops advancing till cleanup). * @dma_mapped: Indicates if request has been mapped to DMA (internal) + * @sg_was_mapped: Set if the scatterlist has been mapped before the request * @complete: Function called when request completes, so this request and * its buffer may be re-used. The function will always be called with * interrupts disabled, and it must not sleep. @@ -114,6 +115,7 @@ struct usb_request { unsigned zero:1; unsigned short_not_ok:1; unsigned dma_mapped:1; + unsigned sg_was_mapped:1; void (*complete)(struct usb_ep *ep, struct usb_request *req); diff --git a/include/linux/usb/serial.h b/include/linux/usb/serial.h index 1a0a4dc87980..40e5bc1da680 100644 --- a/include/linux/usb/serial.h +++ b/include/linux/usb/serial.h @@ -377,6 +377,7 @@ void usb_serial_handle_dcd_change(struct usb_serial_port *usb_port, int usb_serial_bus_register(struct usb_serial_driver *device); void usb_serial_bus_deregister(struct usb_serial_driver *device); +void usb_serial_bus_remove_new_id(struct usb_serial_driver *driver); extern const struct bus_type usb_serial_bus_type; extern struct tty_driver *usb_serial_tty_driver; diff --git a/include/linux/userfaultfd_k.h b/include/linux/userfaultfd_k.h index 67ff2889374f..c08e3b0da5e8 100644 --- a/include/linux/userfaultfd_k.h +++ b/include/linux/userfaultfd_k.h @@ -162,6 +162,8 @@ static inline bool userfaultfd_armed(struct vm_area_struct *vma) bool vma_can_userfault(struct vm_area_struct *vma, unsigned long vm_flags); +bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma); + extern int dup_userfaultfd(struct vm_area_struct *, struct list_head *); extern void dup_userfaultfd_complete(struct list_head *); @@ -280,6 +282,11 @@ static inline bool userfaultfd_wp_unpopulated(struct vm_area_struct *vma) return false; } +static inline bool vma_has_uffd_without_event_remap(struct vm_area_struct *vma) +{ + return false; +} + #endif /* CONFIG_USERFAULTFD */ static inline bool userfaultfd_wp_use_markers(struct vm_area_struct *vma) diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h index db15ac07f8a6..85e4331f1eb9 100644 --- a/include/linux/vdpa.h +++ b/include/linux/vdpa.h @@ -70,9 +70,6 @@ struct vdpa_mgmt_dev; * struct vdpa_device - representation of a vDPA device * @dev: underlying device * @dma_dev: the actual device that is performing DMA - * @driver_override: driver name to force a match; do not set directly, - * because core frees it; use driver_set_override() to - * set or clear it. * @config: the configuration ops for this device. * @cf_lock: Protects get and set access to configuration layout. * @index: device index @@ -87,7 +84,6 @@ struct vdpa_mgmt_dev; struct vdpa_device { struct device dev; struct device *dma_dev; - const char *driver_override; const struct vdpa_config_ops *config; struct rw_semaphore cf_lock; /* Protects get/set config */ unsigned int index; diff --git a/include/linux/virtio.h b/include/linux/virtio.h index 4cc614a38376..73b95651fe57 100644 --- a/include/linux/virtio.h +++ b/include/linux/virtio.h @@ -158,7 +158,7 @@ void virtio_reset_device(struct virtio_device *dev); size_t virtio_max_dma_size(const struct virtio_device *vdev); #define virtio_device_for_each_vq(vdev, vq) \ - list_for_each_entry(vq, &vdev->vqs, list) + list_for_each_entry(vq, &(vdev)->vqs, list) /** * struct virtio_driver - operations for a virtio I/O driver diff --git a/include/media/v4l2-mem2mem.h b/include/media/v4l2-mem2mem.h index 370c230ad3be..4a2649a4562a 100644 --- a/include/media/v4l2-mem2mem.h +++ b/include/media/v4l2-mem2mem.h @@ -537,6 +537,27 @@ v4l2_m2m_register_media_controller(struct v4l2_m2m_dev *m2m_dev, */ void v4l2_m2m_release(struct v4l2_m2m_dev *m2m_dev); +/** + * v4l2_m2m_get() - take a reference to the m2m_dev structure + * + * @m2m_dev: opaque pointer to the internal data to handle M2M context + * + * This is used to share the M2M device across multiple devices. This + * can be used to avoid scheduling two hardware nodes concurrently. + */ +void v4l2_m2m_get(struct v4l2_m2m_dev *m2m_dev); + +/** + * v4l2_m2m_put() - remove a reference to the m2m_dev structure + * + * @m2m_dev: opaque pointer to the internal data to handle M2M context + * + * Once the M2M device has no more references, v4l2_m2m_release() will be + * called automatically. Users of this method should never call + * v4l2_m2m_release() directly. See v4l2_m2m_get() for more details. + */ +void v4l2_m2m_put(struct v4l2_m2m_dev *m2m_dev); + /** * v4l2_m2m_ctx_init() - allocate and initialize a m2m context * diff --git a/include/net/amt.h b/include/net/amt.h index c881bc8b673b..ad844d65a64d 100644 --- a/include/net/amt.h +++ b/include/net/amt.h @@ -231,10 +231,6 @@ struct amt_relay_headers { }; } __packed; -struct amt_skb_cb { - struct amt_tunnel_list *tunnel; -}; - struct amt_tunnel_list { struct list_head list; /* Protect All resources under an amt_tunne_list */ diff --git a/include/net/ax25.h b/include/net/ax25.h index ef79023f1a28..df3515e43da5 100644 --- a/include/net/ax25.h +++ b/include/net/ax25.h @@ -416,7 +416,6 @@ void ax25_rt_device_down(struct net_device *); int ax25_rt_ioctl(unsigned int, void __user *); extern const struct seq_operations ax25_rt_seqops; ax25_route *ax25_get_route(ax25_address *addr, struct net_device *dev); -int ax25_rt_autobind(ax25_cb *, ax25_address *); struct sk_buff *ax25_rt_build_path(struct sk_buff *, ax25_address *, ax25_address *, ax25_digi *); void ax25_rt_free(void); diff --git a/include/net/bluetooth/hci_core.h b/include/net/bluetooth/hci_core.h index edd56d6fe76e..62804d98d3ee 100644 --- a/include/net/bluetooth/hci_core.h +++ b/include/net/bluetooth/hci_core.h @@ -64,7 +64,7 @@ struct inquiry_entry { NAME_PENDING, NAME_KNOWN, } name_state; - __u32 timestamp; + unsigned long timestamp; struct inquiry_data data; }; @@ -80,7 +80,7 @@ struct discovery_state { struct list_head all; /* All devices found during inquiry */ struct list_head unknown; /* Name state not known */ struct list_head resolve; /* Name needs to be resolved */ - __u32 timestamp; + unsigned long timestamp; bdaddr_t last_adv_addr; u8 last_adv_addr_type; s8 last_adv_rssi; @@ -555,6 +555,7 @@ struct hci_dev { struct list_head link_keys; struct list_head long_term_keys; struct list_head identity_resolving_keys; + struct mutex remote_oob_lock; struct list_head remote_oob_data; struct list_head le_accept_list; struct list_head le_resolv_list; @@ -744,6 +745,8 @@ struct hci_conn { struct dentry *debugfs; struct hci_dev *hdev; + + spinlock_t proto_lock; /* lock guarding protocol data */ void *l2cap_data; void *sco_data; void *iso_data; diff --git a/include/net/bluetooth/rfcomm.h b/include/net/bluetooth/rfcomm.h index ba8d3702853d..d483830188f3 100644 --- a/include/net/bluetooth/rfcomm.h +++ b/include/net/bluetooth/rfcomm.h @@ -210,6 +210,7 @@ struct rfcomm_dlc { #define RFCOMM_AUTH_REJECT 7 #define RFCOMM_DEFER_SETUP 8 #define RFCOMM_ENC_DROP 9 +#define RFCOMM_CLOSED 10 /* Scheduling flags and events */ #define RFCOMM_SCHED_WAKEUP 31 diff --git a/include/net/cfg80211.h b/include/net/cfg80211.h index cda0b2907f32..2343f4c0f1d8 100644 --- a/include/net/cfg80211.h +++ b/include/net/cfg80211.h @@ -1872,6 +1872,99 @@ struct cfg80211_tid_stats { #define IEEE80211_MAX_CHAINS 4 +/** + * struct link_station_info - link station information + * + * Link station information filled by driver for get_station() and + * dump_station(). + * @filled: bit flag of flags using the bits of &enum nl80211_sta_info to + * indicate the relevant values in this struct for them + * @connected_time: time(in secs) since a link of station is last connected + * @inactive_time: time since last activity for link station(tx/rx) + * in milliseconds + * @assoc_at: bootime (ns) of the last association of link of station + * @rx_bytes: bytes (size of MPDUs) received from this link of station + * @tx_bytes: bytes (size of MPDUs) transmitted to this link of station + * @signal: The signal strength, type depends on the wiphy's signal_type. + * For CFG80211_SIGNAL_TYPE_MBM, value is expressed in _dBm_. + * @signal_avg: Average signal strength, type depends on the wiphy's + * signal_type. For CFG80211_SIGNAL_TYPE_MBM, value is expressed in _dBm_ + * @chains: bitmask for filled values in @chain_signal, @chain_signal_avg + * @chain_signal: per-chain signal strength of last received packet in dBm + * @chain_signal_avg: per-chain signal strength average in dBm + * @txrate: current unicast bitrate from this link of station + * @rxrate: current unicast bitrate to this link of station + * @rx_packets: packets (MSDUs & MMPDUs) received from this link of station + * @tx_packets: packets (MSDUs & MMPDUs) transmitted to this link of station + * @tx_retries: cumulative retry counts (MPDUs) for this link of station + * @tx_failed: number of failed transmissions (MPDUs) (retries exceeded, no ACK) + * @rx_dropped_misc: Dropped for un-specified reason. + * @bss_param: current BSS parameters + * @beacon_loss_count: Number of times beacon loss event has triggered. + * @expected_throughput: expected throughput in kbps (including 802.11 headers) + * towards this station. + * @rx_beacon: number of beacons received from this peer + * @rx_beacon_signal_avg: signal strength average (in dBm) for beacons received + * from this peer + * @rx_duration: aggregate PPDU duration(usecs) for all the frames from a peer + * @tx_duration: aggregate PPDU duration(usecs) for all the frames to a peer + * @airtime_weight: current airtime scheduling weight + * @pertid: per-TID statistics, see &struct cfg80211_tid_stats, using the last + * (IEEE80211_NUM_TIDS) index for MSDUs not encapsulated in QoS-MPDUs. + * Note that this doesn't use the @filled bit, but is used if non-NULL. + * @ack_signal: signal strength (in dBm) of the last ACK frame. + * @avg_ack_signal: average rssi value of ack packet for the no of msdu's has + * been sent. + * @rx_mpdu_count: number of MPDUs received from this station + * @fcs_err_count: number of packets (MPDUs) received from this station with + * an FCS error. This counter should be incremented only when TA of the + * received packet with an FCS error matches the peer MAC address. + * @addr: For MLO STA connection, filled with address of the link of station. + */ +struct link_station_info { + u64 filled; + u32 connected_time; + u32 inactive_time; + u64 assoc_at; + u64 rx_bytes; + u64 tx_bytes; + s8 signal; + s8 signal_avg; + + u8 chains; + s8 chain_signal[IEEE80211_MAX_CHAINS]; + s8 chain_signal_avg[IEEE80211_MAX_CHAINS]; + + struct rate_info txrate; + struct rate_info rxrate; + u32 rx_packets; + u32 tx_packets; + u32 tx_retries; + u32 tx_failed; + u32 rx_dropped_misc; + struct sta_bss_parameters bss_param; + + u32 beacon_loss_count; + + u32 expected_throughput; + + u64 tx_duration; + u64 rx_duration; + u64 rx_beacon; + u8 rx_beacon_signal_avg; + + u16 airtime_weight; + + s8 ack_signal; + s8 avg_ack_signal; + struct cfg80211_tid_stats *pertid; + + u32 rx_mpdu_count; + u32 fcs_err_count; + + u8 addr[ETH_ALEN] __aligned(2); +}; + /** * struct station_info - station information * @@ -1956,6 +2049,11 @@ struct cfg80211_tid_stats { * dump_station() callbacks. User space needs this information to determine * the accepted and rejected affiliated links of the connected station. * @assoc_resp_ies_len: Length of @assoc_resp_ies buffer in octets. + * @valid_links: bitmap of valid links, or 0 for non-MLO. Drivers fill this + * information in cfg80211_new_sta(), cfg80211_del_sta_sinfo(), + * get_station() and dump_station() callbacks. + * @links: reference to Link sta entries for MLO STA, all link specific + * information is accessed through links[link_id]. */ struct station_info { u64 filled; @@ -2021,6 +2119,9 @@ struct station_info { u8 mld_addr[ETH_ALEN] __aligned(2); const u8 *assoc_resp_ies; size_t assoc_resp_ies_len; + + u16 valid_links; + struct link_station_info *links[IEEE80211_MLD_MAX_NUM_LINKS]; }; /** @@ -6000,7 +6101,7 @@ struct wireless_dev { enum ieee80211_bss_type conn_bss_type; u32 conn_owner_nlportid; - struct work_struct disconnect_wk; + struct wiphy_work disconnect_wk; u8 disconnect_bssid[ETH_ALEN]; struct list_head event_list; @@ -6042,7 +6143,7 @@ struct wireless_dev { struct list_head pmsr_list; spinlock_t pmsr_lock; - struct work_struct pmsr_free_wk; + struct wiphy_work pmsr_free_wk; unsigned long unprot_beacon_reported; diff --git a/include/net/dropreason-core.h b/include/net/dropreason-core.h index e88e785d221b..16c140e00cc8 100644 --- a/include/net/dropreason-core.h +++ b/include/net/dropreason-core.h @@ -56,6 +56,7 @@ FN(UNHANDLED_PROTO) \ FN(SKB_CSUM) \ FN(SKB_GSO_SEG) \ + FN(SKB_BAD_GSO) \ FN(SKB_UCOPY_FAULT) \ FN(DEV_HDR) \ FN(DEV_READY) \ @@ -262,6 +263,8 @@ enum skb_drop_reason { SKB_DROP_REASON_SKB_CSUM, /** @SKB_DROP_REASON_SKB_GSO_SEG: gso segmentation error */ SKB_DROP_REASON_SKB_GSO_SEG, + /** @SKB_DROP_REASON_SKB_BAD_GSO: malicious gso packet. */ + SKB_DROP_REASON_SKB_BAD_GSO, /** * @SKB_DROP_REASON_SKB_UCOPY_FAULT: failed to copy data from user space, * e.g., via zerocopy_sg_from_iter() or skb_orphan_frags_rx() diff --git a/include/net/inet_connection_sock.h b/include/net/inet_connection_sock.h index 4ed883cad65a..0a33e7cdcc8b 100644 --- a/include/net/inet_connection_sock.h +++ b/include/net/inet_connection_sock.h @@ -63,6 +63,9 @@ struct inet_connection_sock_af_ops { * @icsk_bind2_hash: Bind node in the bhash2 table * @icsk_timeout: Timeout * @icsk_retransmit_timer: Resend (no ack) + * @icsk_delack_timer: Delayed ACK timer + * @icsk_keepalive_timer: Keepalive timer + * @mptcp_tout_timer: MPTCP timeout timer * @icsk_rto: Retransmit timeout * @icsk_pmtu_cookie Last pmtu seen by socket * @icsk_ca_ops Pluggable congestion control hook @@ -89,8 +92,12 @@ struct inet_connection_sock { struct inet_bind_bucket *icsk_bind_hash; struct inet_bind2_bucket *icsk_bind2_hash; unsigned long icsk_timeout; - struct timer_list icsk_retransmit_timer; - struct timer_list icsk_delack_timer; + struct timer_list icsk_retransmit_timer; + struct timer_list icsk_delack_timer; + union { + struct timer_list icsk_keepalive_timer; + struct timer_list mptcp_tout_timer; + }; __u32 icsk_rto; __u32 icsk_rto_min; __u32 icsk_delack_max; diff --git a/include/net/ip.h b/include/net/ip.h index 32825e2ac43d..775185fe552f 100644 --- a/include/net/ip.h +++ b/include/net/ip.h @@ -465,9 +465,14 @@ static inline unsigned int ip_dst_mtu_maybe_forward(const struct dst_entry *dst, bool forwarding) { const struct rtable *rt = container_of(dst, struct rtable, dst); - struct net *net = dev_net(dst_dev(dst)); - unsigned int mtu; + const struct net_device *dev; + unsigned int mtu, res; + struct net *net; + + rcu_read_lock(); + dev = dst_dev_rcu(dst); + net = dev_net_rcu(dev); if (READ_ONCE(net->ipv4.sysctl_ip_fwd_use_pmtu) || ip_mtu_locked(dst) || !forwarding) { @@ -481,7 +486,7 @@ static inline unsigned int ip_dst_mtu_maybe_forward(const struct dst_entry *dst, if (mtu) goto out; - mtu = READ_ONCE(dst_dev(dst)->mtu); + mtu = READ_ONCE(dev->mtu); if (unlikely(ip_mtu_locked(dst))) { if (rt->rt_uses_gateway && mtu > 576) @@ -490,8 +495,11 @@ static inline unsigned int ip_dst_mtu_maybe_forward(const struct dst_entry *dst, out: mtu = min_t(unsigned int, mtu, IP_MAX_MTU); + res = mtu - lwtunnel_headroom(dst->lwtstate, mtu); - return mtu - lwtunnel_headroom(dst->lwtstate, mtu); + rcu_read_unlock(); + + return res; } static inline unsigned int ip_skb_dst_mtu(struct sock *sk, diff --git a/include/net/ip6_route.h b/include/net/ip6_route.h index 211d1ef8ab42..d5f3b842dd0a 100644 --- a/include/net/ip6_route.h +++ b/include/net/ip6_route.h @@ -333,7 +333,7 @@ static inline unsigned int ip6_dst_mtu_maybe_forward(const struct dst_entry *dst mtu = IPV6_MIN_MTU; rcu_read_lock(); - idev = __in6_dev_get(dst->dev); + idev = __in6_dev_get(dst_dev_rcu(dst)); if (idev) mtu = idev->cnf.mtu6; rcu_read_unlock(); diff --git a/include/net/ip6_tunnel.h b/include/net/ip6_tunnel.h index 74b369bddf49..b3453ee4523a 100644 --- a/include/net/ip6_tunnel.h +++ b/include/net/ip6_tunnel.h @@ -156,6 +156,16 @@ static inline void ip6tunnel_xmit(struct sock *sk, struct sk_buff *skb, { int pkt_len, err; + if (dev_recursion_level() > IP_TUNNEL_RECURSION_LIMIT) { + net_crit_ratelimited("Dead loop on virtual device %s, fix it urgently!\n", + dev->name); + DEV_STATS_INC(dev, tx_errors); + kfree_skb(skb); + return; + } + + dev_xmit_recursion_inc(); + memset(skb->cb, 0, sizeof(struct inet6_skb_parm)); pkt_len = skb->len - skb_inner_network_offset(skb); err = ip6_local_out(dev_net(skb_dst(skb)->dev), sk, skb); @@ -165,6 +175,8 @@ static inline void ip6tunnel_xmit(struct sock *sk, struct sk_buff *skb, pkt_len = -1; iptunnel_xmit_stats(dev, pkt_len); } + + dev_xmit_recursion_dec(); } #endif #endif diff --git a/include/net/ip_tunnels.h b/include/net/ip_tunnels.h index 1243ec414bcd..5e0965212c70 100644 --- a/include/net/ip_tunnels.h +++ b/include/net/ip_tunnels.h @@ -24,6 +24,13 @@ #include #endif +/* Recursion limit for tunnel xmit to detect routing loops. + * Unlike XMIT_RECURSION_LIMIT (8) used in the no-qdisc path, tunnel + * recursion involves route lookups and full IP output, consuming much + * more stack per level, so a lower limit is needed. + */ +#define IP_TUNNEL_RECURSION_LIMIT 4 + /* Keep error state on tunnel for 30 sec */ #define IPTUNNEL_ERR_TIMEO (30*HZ) @@ -535,6 +542,16 @@ static inline int iptunnel_pull_offloads(struct sk_buff *skb) static inline void iptunnel_xmit_stats(struct net_device *dev, int pkt_len) { if (pkt_len > 0) { + if (dev->pcpu_stat_type == NETDEV_PCPU_STAT_DSTATS) { + struct pcpu_dstats *dstats = get_cpu_ptr(dev->dstats); + + u64_stats_update_begin(&dstats->syncp); + u64_stats_add(&dstats->tx_bytes, pkt_len); + u64_stats_inc(&dstats->tx_packets); + u64_stats_update_end(&dstats->syncp); + put_cpu_ptr(dstats); + return; + } struct pcpu_sw_netstats *tstats = get_cpu_ptr(dev->tstats); u64_stats_update_begin(&tstats->syncp); diff --git a/include/net/ip_vs.h b/include/net/ip_vs.h index 35f012f62015..d2bfc9f67082 100644 --- a/include/net/ip_vs.h +++ b/include/net/ip_vs.h @@ -1713,7 +1713,7 @@ static inline char ip_vs_fwd_tag(struct ip_vs_conn *cp) return fwd; } -void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp, +bool ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp, struct ip_vs_conn *cp, int dir, unsigned int toff, bool has_ports, struct ip_vs_iphdr *ciph); diff --git a/include/net/mac80211.h b/include/net/mac80211.h index faf1fd70ff94..89fe0f0b69fe 100644 --- a/include/net/mac80211.h +++ b/include/net/mac80211.h @@ -1786,12 +1786,16 @@ enum ieee80211_vif_flags { * @IEEE80211_OFFLOAD_DECAP_ENABLED: rx encapsulation offload is enabled * The driver supports passing received 802.11 frames as 802.3 frames to * mac80211. + * @IEEE80211_OFFLOAD_ENCAP_MCAST: tx multicast encapsulation offload is enabled + * The driver supports sending multicast frames passed as 802.3 frames + * by mac80211. */ enum ieee80211_offload_flags { IEEE80211_OFFLOAD_ENCAP_ENABLED = BIT(0), IEEE80211_OFFLOAD_ENCAP_4ADDR = BIT(1), IEEE80211_OFFLOAD_DECAP_ENABLED = BIT(2), + IEEE80211_OFFLOAD_ENCAP_MCAST = BIT(3), }; /** diff --git a/include/net/netfilter/nf_conntrack.h b/include/net/netfilter/nf_conntrack.h index 3b41312cb0b0..45267aba724b 100644 --- a/include/net/netfilter/nf_conntrack.h +++ b/include/net/netfilter/nf_conntrack.h @@ -213,8 +213,7 @@ bool nf_ct_get_tuplepr(const struct sk_buff *skb, unsigned int nhoff, struct nf_conntrack_tuple *tuple); void __nf_ct_refresh_acct(struct nf_conn *ct, enum ip_conntrack_info ctinfo, - const struct sk_buff *skb, - u32 extra_jiffies, bool do_acct); + u32 extra_jiffies, unsigned int bytes); /* Refresh conntrack for this many jiffies and do accounting */ static inline void nf_ct_refresh_acct(struct nf_conn *ct, @@ -222,15 +221,14 @@ static inline void nf_ct_refresh_acct(struct nf_conn *ct, const struct sk_buff *skb, u32 extra_jiffies) { - __nf_ct_refresh_acct(ct, ctinfo, skb, extra_jiffies, true); + __nf_ct_refresh_acct(ct, ctinfo, extra_jiffies, skb->len); } /* Refresh conntrack for this many jiffies */ static inline void nf_ct_refresh(struct nf_conn *ct, - const struct sk_buff *skb, u32 extra_jiffies) { - __nf_ct_refresh_acct(ct, 0, skb, extra_jiffies, false); + __nf_ct_refresh_acct(ct, 0, extra_jiffies, 0); } /* kill conntrack and do accounting */ @@ -324,16 +322,6 @@ static inline bool nf_ct_should_gc(const struct nf_conn *ct) #define NF_CT_DAY (86400 * HZ) -/* Set an arbitrary timeout large enough not to ever expire, this save - * us a check for the IPS_OFFLOAD_BIT from the packet path via - * nf_ct_is_expired(). - */ -static inline void nf_ct_offload_timeout(struct nf_conn *ct) -{ - if (nf_ct_expires(ct) < NF_CT_DAY / 2) - WRITE_ONCE(ct->timeout, nfct_time_stamp + NF_CT_DAY); -} - struct kernel_param; int nf_conntrack_set_hashsize(const char *val, const struct kernel_param *kp); diff --git a/include/net/netfilter/nf_conntrack_expect.h b/include/net/netfilter/nf_conntrack_expect.h index 80f50fd0f7ad..be4a120d549e 100644 --- a/include/net/netfilter/nf_conntrack_expect.h +++ b/include/net/netfilter/nf_conntrack_expect.h @@ -54,8 +54,8 @@ struct nf_conntrack_expect { /* The conntrack of the master connection */ struct nf_conn *master; - /* Timer function; deletes the expectation. */ - struct timer_list timeout; + /* jiffies32 when this expectation expires */ + u32 timeout; #if IS_ENABLED(CONFIG_NF_NAT) union nf_inet_addr saved_addr; @@ -69,6 +69,14 @@ struct nf_conntrack_expect { struct rcu_head rcu; }; +static inline bool nf_ct_exp_is_expired(const struct nf_conntrack_expect *exp) +{ + if (READ_ONCE(exp->flags) & NF_CT_EXPECT_DEAD) + return true; + + return (__s32)(READ_ONCE(exp->timeout) - nfct_time_stamp) <= 0; +} + static inline struct net *nf_ct_exp_net(struct nf_conntrack_expect *exp) { return read_pnet(&exp->net); @@ -130,7 +138,6 @@ static inline void nf_ct_unlink_expect(struct nf_conntrack_expect *exp) void nf_ct_remove_expectations(struct nf_conn *ct); void nf_ct_unexpect_related(struct nf_conntrack_expect *exp); -bool nf_ct_remove_expect(struct nf_conntrack_expect *exp); void nf_ct_expect_iterate_destroy(bool (*iter)(struct nf_conntrack_expect *e, void *data), void *data); void nf_ct_expect_iterate_net(struct net *net, @@ -153,5 +160,8 @@ static inline int nf_ct_expect_related(struct nf_conntrack_expect *expect, return nf_ct_expect_related_report(expect, 0, 0, flags); } +struct nf_conn_help; +void nf_ct_expectation_gc(struct nf_conn_help *master_help); + #endif /*_NF_CONNTRACK_EXPECT_H*/ diff --git a/include/net/netfilter/nf_conntrack_timeout.h b/include/net/netfilter/nf_conntrack_timeout.h index 3a66d4abb6d6..d60aa86be019 100644 --- a/include/net/netfilter/nf_conntrack_timeout.h +++ b/include/net/netfilter/nf_conntrack_timeout.h @@ -12,6 +12,7 @@ #define CTNL_TIMEOUT_NAME_MAX 32 struct nf_ct_timeout { + refcount_t refcnt; __u16 l3num; const struct nf_conntrack_l4proto *l4proto; struct rcu_head rcu; @@ -22,6 +23,22 @@ struct nf_conn_timeout { struct nf_ct_timeout __rcu *timeout; }; +static inline void nf_ct_timeout_put(const struct nf_conn *ct) +{ +#ifdef CONFIG_NF_CONNTRACK_TIMEOUT + struct nf_conn_timeout *timeout_ext; + struct nf_ct_timeout *timeout; + + timeout_ext = nf_ct_ext_find(ct, NF_CT_EXT_TIMEOUT); + if (!timeout_ext) + return; + + timeout = rcu_dereference(timeout_ext->timeout); + if (timeout && refcount_dec_and_test(&timeout->refcnt)) + kfree_rcu(timeout, rcu); +#endif +} + static inline unsigned int * nf_ct_timeout_data(const struct nf_conn_timeout *t) { @@ -56,8 +73,14 @@ struct nf_conn_timeout *nf_ct_timeout_ext_add(struct nf_conn *ct, #ifdef CONFIG_NF_CONNTRACK_TIMEOUT struct nf_conn_timeout *timeout_ext; + if (!timeout) + return NULL; + timeout_ext = nf_ct_ext_add(ct, NF_CT_EXT_TIMEOUT, gfp); - if (timeout_ext == NULL) + if (!timeout_ext || timeout_ext->timeout) + return NULL; + + if (!refcount_inc_not_zero(&timeout->refcnt)) return NULL; rcu_assign_pointer(timeout_ext->timeout, timeout); @@ -75,7 +98,7 @@ static inline unsigned int *nf_ct_timeout_lookup(const struct nf_conn *ct) struct nf_conn_timeout *timeout_ext; timeout_ext = nf_ct_timeout_find(ct); - if (timeout_ext) + if (timeout_ext && rcu_access_pointer(timeout_ext->timeout)) timeouts = nf_ct_timeout_data(timeout_ext); #endif return timeouts; diff --git a/include/net/netfilter/nf_flow_table.h b/include/net/netfilter/nf_flow_table.h index 0097791e1eed..f07234df1d03 100644 --- a/include/net/netfilter/nf_flow_table.h +++ b/include/net/netfilter/nf_flow_table.h @@ -163,10 +163,10 @@ enum nf_flow_flags { NF_FLOW_SNAT, NF_FLOW_DNAT, NF_FLOW_TEARDOWN, + NF_FLOW_PENDING, NF_FLOW_HW, NF_FLOW_HW_DYING, NF_FLOW_HW_DEAD, - NF_FLOW_HW_PENDING, NF_FLOW_HW_BIDIRECTIONAL, NF_FLOW_HW_ESTABLISHED, }; diff --git a/include/net/netfilter/nf_queue.h b/include/net/netfilter/nf_queue.h index c576479b9aa5..4cf158b5ea69 100644 --- a/include/net/netfilter/nf_queue.h +++ b/include/net/netfilter/nf_queue.h @@ -21,6 +21,7 @@ struct nf_queue_entry { struct net_device *physout; #endif struct nf_hook_state state; + bool nf_ct_is_unconfirmed; u16 size; /* sizeof(entry) + saved route keys */ /* extra space to store route keys */ diff --git a/include/net/netfilter/nf_tables.h b/include/net/netfilter/nf_tables.h index 7cfc5b3d3b31..0a6edf2aec7d 100644 --- a/include/net/netfilter/nf_tables.h +++ b/include/net/netfilter/nf_tables.h @@ -462,7 +462,7 @@ struct nft_set_ops { int (*insert)(const struct net *net, const struct nft_set *set, const struct nft_set_elem *elem, - struct nft_set_ext **ext); + void **priv); void (*activate)(const struct net *net, const struct nft_set *set, const struct nft_set_elem *elem); @@ -855,8 +855,6 @@ void *nft_set_elem_init(const struct nft_set *set, const struct nft_set_ext_tmpl *tmpl, const u32 *key, const u32 *key_end, const u32 *data, u64 timeout, u64 expiration, gfp_t gfp); -int nft_set_elem_expr_clone(const struct nft_ctx *ctx, struct nft_set *set, - struct nft_expr *expr_array[]); void nft_set_elem_expr_destroy(const struct nft_ctx *ctx, struct nft_set_elem_expr *elem_expr); void nft_set_elem_destroy(const struct nft_set *set, void *elem, @@ -1267,6 +1265,7 @@ static inline void nft_use_inc_restore(u32 *use) * @sets: sets in the table * @objects: stateful objects in the table * @flowtables: flow tables in the table + * @objname_ht: hashtable for objects lookup by name * @hgenerator: handle generator state * @handle: table handle * @use: number of chain references to this table @@ -1283,6 +1282,7 @@ struct nft_table { struct list_head sets; struct list_head objects; struct list_head flowtables; + struct rhltable objname_ht; u64 hgenerator; u64 handle; u32 use; @@ -1364,7 +1364,7 @@ static inline void *nft_obj_data(const struct nft_object *obj) #define nft_expr_obj(expr) *((struct nft_object **)nft_expr_priv(expr)) struct nft_object *nft_obj_lookup(const struct net *net, - const struct nft_table *table, + struct nft_table *table, const struct nlattr *nla, u32 objtype, u8 genmask); diff --git a/include/net/netns/mpls.h b/include/net/netns/mpls.h index 19ad2574b267..9b83b7f43d83 100644 --- a/include/net/netns/mpls.h +++ b/include/net/netns/mpls.h @@ -7,6 +7,7 @@ #define __NETNS_MPLS_H__ #include +#include struct mpls_route; struct ctl_table_header; @@ -16,6 +17,7 @@ struct netns_mpls { int default_ttl; size_t platform_labels; struct mpls_route __rcu * __rcu *platform_label; + seqcount_t platform_label_seq; struct ctl_table_header *ctl; }; diff --git a/include/net/netns/xfrm.h b/include/net/netns/xfrm.h index 7922ec72eaa6..eb9edddd7309 100644 --- a/include/net/netns/xfrm.h +++ b/include/net/netns/xfrm.h @@ -23,6 +23,7 @@ struct xfrm_policy_hash { struct xfrm_policy_hthresh { struct work_struct work; seqlock_t lock; + bool work_disabled; u8 lbits4; u8 rbits4; u8 lbits6; diff --git a/include/net/nexthop.h b/include/net/nexthop.h index 2b12725de9c0..f37e96f93f20 100644 --- a/include/net/nexthop.h +++ b/include/net/nexthop.h @@ -97,7 +97,7 @@ struct nh_res_table { struct nh_grp_entry { struct nexthop *nh; - u8 weight; + u16 weight; union { struct { @@ -173,14 +173,14 @@ struct nh_notifier_single_info { __be32 ipv4; struct in6_addr ipv6; }; + u32 id; u8 is_reject:1, is_fdb:1, has_encap:1; }; struct nh_notifier_grp_entry_info { - u8 weight; - u32 id; + u16 weight; struct nh_notifier_single_info nh; }; diff --git a/include/net/pkt_cls.h b/include/net/pkt_cls.h index 325a8553853d..05890fc84abe 100644 --- a/include/net/pkt_cls.h +++ b/include/net/pkt_cls.h @@ -154,8 +154,20 @@ static inline int tcf_classify(struct sk_buff *skb, { return TC_ACT_UNSPEC; } - #endif +static inline int tcf_classify_qdisc(struct sk_buff *skb, + const struct tcf_proto *tp, + struct tcf_result *res, bool compat_mode) +{ + int ret = tcf_classify(skb, NULL, tp, res, compat_mode); + + /* TC_ACT_REDIRECT from qdisc filter chains is not supported. + * Use BPF via tcx or mirred redirect instead. + */ + if (unlikely(ret == TC_ACT_REDIRECT)) + ret = TC_ACT_SHOT; + return ret; +} static inline unsigned long __cls_set_class(unsigned long *clp, unsigned long cl) diff --git a/include/net/route.h b/include/net/route.h index d443de683628..7f37b8b96c92 100644 --- a/include/net/route.h +++ b/include/net/route.h @@ -359,7 +359,7 @@ static inline int ip4_dst_hoplimit(const struct dst_entry *dst) const struct net *net; rcu_read_lock(); - net = dev_net_rcu(dst_dev(dst)); + net = dst_dev_net_rcu(dst); hoplimit = READ_ONCE(net->ipv4.sysctl_ip_default_ttl); rcu_read_unlock(); } diff --git a/include/net/sch_generic.h b/include/net/sch_generic.h index 9d623fc8b1fc..79e029b5d1a2 100644 --- a/include/net/sch_generic.h +++ b/include/net/sch_generic.h @@ -595,6 +595,7 @@ static inline void sch_tree_unlock(struct Qdisc *q) extern struct Qdisc noop_qdisc; extern struct Qdisc_ops noop_qdisc_ops; extern struct Qdisc_ops pfifo_fast_ops; +extern const u8 sch_default_prio2band[TC_PRIO_MAX + 1]; extern struct Qdisc_ops mq_qdisc_ops; extern struct Qdisc_ops noqueue_qdisc_ops; extern const struct Qdisc_ops *default_qdisc_ops; @@ -1077,12 +1078,17 @@ static inline struct sk_buff *qdisc_dequeue_internal(struct Qdisc *sch, bool dir skb = __skb_dequeue(&sch->gso_skb); if (skb) { sch->q.qlen--; + qdisc_qstats_backlog_dec(sch, skb); return skb; } - if (direct) - return __qdisc_dequeue_head(&sch->q); - else + if (direct) { + skb = __qdisc_dequeue_head(&sch->q); + if (skb) + qdisc_qstats_backlog_dec(sch, skb); + return skb; + } else { return sch->dequeue(sch); + } } static inline struct sk_buff *qdisc_dequeue_head(struct Qdisc *sch) diff --git a/include/net/sctp/auth.h b/include/net/sctp/auth.h index d4b3b2dcd15b..c2399fb17889 100644 --- a/include/net/sctp/auth.h +++ b/include/net/sctp/auth.h @@ -23,6 +23,7 @@ struct sctp_association; struct sctp_authkey; struct sctp_hmacalgo; struct crypto_shash; +struct sctp_cookie; /* * Define a generic struct that will hold all the info @@ -77,6 +78,8 @@ struct sctp_shared_key *sctp_auth_get_shkey( int sctp_auth_asoc_copy_shkeys(const struct sctp_endpoint *ep, struct sctp_association *asoc, gfp_t gfp); +bool sctp_auth_verify_cookie_params(const struct sctp_endpoint *ep, + const struct sctp_cookie *cookie); int sctp_auth_init_hmacs(struct sctp_endpoint *ep, gfp_t gfp); void sctp_auth_destroy_hmacs(struct crypto_shash *auth_hmacs[]); struct sctp_hmac *sctp_auth_get_hmac(__u16 hmac_id); diff --git a/include/net/secure_seq.h b/include/net/secure_seq.h index 21e7fa2a1813..e4ad681a2a12 100644 --- a/include/net/secure_seq.h +++ b/include/net/secure_seq.h @@ -5,20 +5,51 @@ #include struct net; +extern struct net init_net; + +union tcp_seq_and_ts_off { + struct { + u32 seq; + u32 ts_off; + }; + u64 hash64; +}; u64 secure_ipv4_port_ephemeral(__be32 saddr, __be32 daddr, __be16 dport); u64 secure_ipv6_port_ephemeral(const __be32 *saddr, const __be32 *daddr, __be16 dport); -u32 secure_tcp_seq(__be32 saddr, __be32 daddr, - __be16 sport, __be16 dport); -u32 secure_tcp_ts_off(const struct net *net, __be32 saddr, __be32 daddr); -u32 secure_tcpv6_seq(const __be32 *saddr, const __be32 *daddr, - __be16 sport, __be16 dport); -u32 secure_tcpv6_ts_off(const struct net *net, - const __be32 *saddr, const __be32 *daddr); u64 secure_dccp_sequence_number(__be32 saddr, __be32 daddr, __be16 sport, __be16 dport); u64 secure_dccpv6_sequence_number(__be32 *saddr, __be32 *daddr, __be16 sport, __be16 dport); +union tcp_seq_and_ts_off +secure_tcp_seq_and_ts_off(const struct net *net, __be32 saddr, __be32 daddr, + __be16 sport, __be16 dport); + +static inline u32 secure_tcp_seq(__be32 saddr, __be32 daddr, + __be16 sport, __be16 dport) +{ + union tcp_seq_and_ts_off ts; + + ts = secure_tcp_seq_and_ts_off(&init_net, saddr, daddr, + sport, dport); + + return ts.seq; +} + +union tcp_seq_and_ts_off +secure_tcpv6_seq_and_ts_off(const struct net *net, const __be32 *saddr, + const __be32 *daddr, + __be16 sport, __be16 dport); + +static inline u32 secure_tcpv6_seq(const __be32 *saddr, const __be32 *daddr, + __be16 sport, __be16 dport) +{ + union tcp_seq_and_ts_off ts; + + ts = secure_tcpv6_seq_and_ts_off(&init_net, saddr, daddr, + sport, dport); + return ts.seq; +} #endif /* _NET_SECURE_SEQ */ diff --git a/include/net/sock.h b/include/net/sock.h index ca476d93e3a3..78e4bfbd3d7a 100644 --- a/include/net/sock.h +++ b/include/net/sock.h @@ -315,6 +315,7 @@ struct sk_filter; * @sk_txrehash: enable TX hash rethink * @sk_filter: socket filtering instructions * @sk_timer: sock cleanup timer + * @mptcp_retransmit_timer: MPTCP retransmit timer * @sk_stamp: time stamp of last packet received * @sk_stamp_seq: lock for accessing sk_stamp on 32 bit architectures only * @sk_tsflags: SO_TIMESTAMPING flags @@ -462,7 +463,10 @@ struct sock { __u32 sk_dst_pending_confirm; u32 sk_pacing_status; /* see enum sk_pacing */ long sk_sndtimeo; - struct timer_list sk_timer; + union { + struct timer_list sk_timer; + struct timer_list mptcp_retransmit_timer; + }; __u32 sk_priority; __u32 sk_mark; unsigned long sk_pacing_rate; /* bytes per second */ @@ -1955,6 +1959,8 @@ static inline struct sk_buff *sock_alloc_send_skb(struct sock *sk, } void *sock_kmalloc(struct sock *sk, int size, gfp_t priority); +void *sock_kmemdup(struct sock *sk, const void *src, + int size, gfp_t priority); void sock_kfree_s(struct sock *sk, void *mem, int size); void sock_kzfree_s(struct sock *sk, void *mem, int size); void sk_send_sigurg(struct sock *sk); diff --git a/include/net/strparser.h b/include/net/strparser.h index 0a83010b3a64..6db67c2526e3 100644 --- a/include/net/strparser.h +++ b/include/net/strparser.h @@ -91,11 +91,11 @@ static inline struct strp_msg *strp_msg(struct sk_buff *skb) struct strparser { struct sock *sk; - u32 stopped : 1; - u32 paused : 1; - u32 aborted : 1; - u32 interrupted : 1; - u32 unrecov_intr : 1; + u8 stopped; + u8 paused; + u8 aborted; + u8 interrupted; + u8 unrecov_intr; struct sk_buff **skb_nextp; struct sk_buff *skb_head; diff --git a/include/net/tcp.h b/include/net/tcp.h index c75d12e5156d..1df0e2d66bfc 100644 --- a/include/net/tcp.h +++ b/include/net/tcp.h @@ -42,6 +42,7 @@ #include #include #include +#include #include #include @@ -985,6 +986,7 @@ static inline bool tcp_skb_can_collapse(const struct sk_buff *to, { return likely(tcp_skb_can_collapse_to(to) && mptcp_skb_can_collapse(to, from) && + !skb_cmp_decrypted(to, from) && skb_pure_zcopy_same(to, from)); } @@ -1477,6 +1479,11 @@ static inline int tcp_full_space(const struct sock *sk) return tcp_win_from_space(sk, READ_ONCE(sk->sk_rcvbuf)); } +static inline u32 tcp_dst_advmss(const struct dst_entry *dst) +{ + return max_t(u32, dst_metric_advmss(dst), TCP_MIN_MSS); +} + static inline void __tcp_adjust_rcv_ssthresh(struct sock *sk, u32 new_ssthresh) { int unused_mem = sk_unused_reserved_mem(sk); @@ -2188,8 +2195,9 @@ struct tcp_request_sock_ops { struct sk_buff *skb, struct flowi *fl, struct request_sock *req); - u32 (*init_seq)(const struct sk_buff *skb); - u32 (*init_ts_off)(const struct net *net, const struct sk_buff *skb); + union tcp_seq_and_ts_off (*init_seq_and_ts_off)( + const struct net *net, + const struct sk_buff *skb); int (*send_synack)(const struct sock *sk, struct dst_entry *dst, struct flowi *fl, struct request_sock *req, struct tcp_fastopen_cookie *foc, diff --git a/include/net/xdp.h b/include/net/xdp.h index 3d8989096b5d..59c12ca58039 100644 --- a/include/net/xdp.h +++ b/include/net/xdp.h @@ -388,14 +388,22 @@ void xdp_attachment_setup(struct xdp_attachment_info *info, #define DEV_MAP_BULK_SIZE XDP_BULK_QUEUE_SIZE +/* Define the relationship between xdp-rx-metadata kfunc and + * various other entities: + * - xdp_rx_metadata enum + * - kfunc name + * - xdp_metadata_ops field + */ #define XDP_METADATA_KFUNC_xxx \ XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_TIMESTAMP, \ - bpf_xdp_metadata_rx_timestamp) \ + bpf_xdp_metadata_rx_timestamp, \ + xmo_rx_timestamp) \ XDP_METADATA_KFUNC(XDP_METADATA_KFUNC_RX_HASH, \ - bpf_xdp_metadata_rx_hash) \ + bpf_xdp_metadata_rx_hash, \ + xmo_rx_hash) \ -enum { -#define XDP_METADATA_KFUNC(name, _) name, +enum xdp_rx_metadata { +#define XDP_METADATA_KFUNC(name, _, __) name, XDP_METADATA_KFUNC_xxx #undef XDP_METADATA_KFUNC MAX_XDP_METADATA_KFUNC, diff --git a/include/scsi/scsi_host.h b/include/scsi/scsi_host.h index f9d5ce6170a7..8a3ea67a6af1 100644 --- a/include/scsi/scsi_host.h +++ b/include/scsi/scsi_host.h @@ -626,6 +626,10 @@ struct Scsi_Host { */ unsigned nr_hw_queues; unsigned nr_maps; + + /* Don't resume host in EH */ + bool eh_noresume; + unsigned active_mode:2; /* @@ -647,9 +651,6 @@ struct Scsi_Host { /* Asynchronous scan in progress */ unsigned async_scan:1; - /* Don't resume host in EH */ - unsigned eh_noresume:1; - /* The controller does not support WRITE SAME */ unsigned no_write_same:1; @@ -710,6 +711,9 @@ struct Scsi_Host { */ struct device *dma_dev; + /* Used for an rcu-synchronizing eh wakeup */ + struct work_struct eh_work; + /* * We should ensure that this is aligned, both for better performance * and also because some compilers (m68k) don't automatically force diff --git a/include/sound/core.h b/include/sound/core.h index fe955fb238e5..f9d85dd65306 100644 --- a/include/sound/core.h +++ b/include/sound/core.h @@ -316,6 +316,15 @@ static inline void snd_card_unref(struct snd_card *card) DEFINE_FREE(snd_card_unref, struct snd_card *, if (_T) snd_card_unref(_T)) +/* + * For automatic error handling at probe time, assign the card like: + * struct snd_card *card __free(snd_card_free) = NULL; + * then create and process as usual. + * But, don't forget to clear to NULL at successful return for avoiding + * the unexpected cleanup! + */ +DEFINE_FREE(snd_card_free, struct snd_card *, if (_T) snd_card_free(_T)) + #define snd_card_set_dev(card, devptr) ((card)->dev = (devptr)) /* device.c */ diff --git a/include/trace/events/rv.h b/include/trace/events/rv.h index 56592da9301c..9ac1e37bcc82 100644 --- a/include/trace/events/rv.h +++ b/include/trace/events/rv.h @@ -16,23 +16,23 @@ DECLARE_EVENT_CLASS(event_da_monitor, TP_ARGS(state, event, next_state, final_state), TP_STRUCT__entry( - __array( char, state, MAX_DA_NAME_LEN ) - __array( char, event, MAX_DA_NAME_LEN ) - __array( char, next_state, MAX_DA_NAME_LEN ) + __string( state, state ) + __string( event, event ) + __string( next_state, next_state ) __field( bool, final_state ) ), TP_fast_assign( - memcpy(__entry->state, state, MAX_DA_NAME_LEN); - memcpy(__entry->event, event, MAX_DA_NAME_LEN); - memcpy(__entry->next_state, next_state, MAX_DA_NAME_LEN); + __assign_str(state, state); + __assign_str(event, event); + __assign_str(next_state, next_state); __entry->final_state = final_state; ), TP_printk("%s x %s -> %s %s", - __entry->state, - __entry->event, - __entry->next_state, + __get_str(state), + __get_str(event), + __get_str(next_state), __entry->final_state ? "(final)" : "") ); @@ -43,18 +43,18 @@ DECLARE_EVENT_CLASS(error_da_monitor, TP_ARGS(state, event), TP_STRUCT__entry( - __array( char, state, MAX_DA_NAME_LEN ) - __array( char, event, MAX_DA_NAME_LEN ) + __string( state, state ) + __string( event, event ) ), TP_fast_assign( - memcpy(__entry->state, state, MAX_DA_NAME_LEN); - memcpy(__entry->event, event, MAX_DA_NAME_LEN); + __assign_str(state, state); + __assign_str(event, event); ), TP_printk("event %s not expected in the state %s", - __entry->event, - __entry->state) + __get_str(event), + __get_str(state)) ); #ifdef CONFIG_RV_MON_WIP @@ -77,25 +77,25 @@ DECLARE_EVENT_CLASS(event_da_monitor_id, TP_STRUCT__entry( __field( int, id ) - __array( char, state, MAX_DA_NAME_LEN ) - __array( char, event, MAX_DA_NAME_LEN ) - __array( char, next_state, MAX_DA_NAME_LEN ) + __string( state, state ) + __string( event, event ) + __string( next_state, next_state ) __field( bool, final_state ) ), TP_fast_assign( - memcpy(__entry->state, state, MAX_DA_NAME_LEN); - memcpy(__entry->event, event, MAX_DA_NAME_LEN); - memcpy(__entry->next_state, next_state, MAX_DA_NAME_LEN); + __assign_str(state, state); + __assign_str(event, event); + __assign_str(next_state, next_state); __entry->id = id; __entry->final_state = final_state; ), TP_printk("%d: %s x %s -> %s %s", __entry->id, - __entry->state, - __entry->event, - __entry->next_state, + __get_str(state), + __get_str(event), + __get_str(next_state), __entry->final_state ? "(final)" : "") ); @@ -107,20 +107,20 @@ DECLARE_EVENT_CLASS(error_da_monitor_id, TP_STRUCT__entry( __field( int, id ) - __array( char, state, MAX_DA_NAME_LEN ) - __array( char, event, MAX_DA_NAME_LEN ) + __string( state, state ) + __string( event, event ) ), TP_fast_assign( - memcpy(__entry->state, state, MAX_DA_NAME_LEN); - memcpy(__entry->event, event, MAX_DA_NAME_LEN); + __assign_str(state, state); + __assign_str(event, event); __entry->id = id; ), TP_printk("%d: event %s not expected in the state %s", __entry->id, - __entry->event, - __entry->state) + __get_str(event), + __get_str(state)) ); #ifdef CONFIG_RV_MON_WWNR diff --git a/include/uapi/linux/netfilter/nf_conntrack_common.h b/include/uapi/linux/netfilter/nf_conntrack_common.h index 56b6b60a814f..ee51045ae1d6 100644 --- a/include/uapi/linux/netfilter/nf_conntrack_common.h +++ b/include/uapi/linux/netfilter/nf_conntrack_common.h @@ -160,6 +160,7 @@ enum ip_conntrack_expect_events { #define NF_CT_EXPECT_USERSPACE 0x4 #ifdef __KERNEL__ +#define NF_CT_EXPECT_DEAD 0x8 #define NF_CT_EXPECT_MASK (NF_CT_EXPECT_PERMANENT | NF_CT_EXPECT_INACTIVE | \ NF_CT_EXPECT_USERSPACE) #endif diff --git a/include/uapi/linux/nexthop.h b/include/uapi/linux/nexthop.h index d8ffa8c9ca78..6b0de563beec 100644 --- a/include/uapi/linux/nexthop.h +++ b/include/uapi/linux/nexthop.h @@ -16,10 +16,15 @@ struct nhmsg { struct nexthop_grp { __u32 id; /* nexthop id - must exist */ __u8 weight; /* weight of this nexthop */ - __u8 resvd1; + __u8 weight_high; /* high order bits of weight */ __u16 resvd2; }; +static inline __u16 nexthop_grp_weight(const struct nexthop_grp *entry) +{ + return ((entry->weight_high << 8) | entry->weight) + 1; +} + enum { NEXTHOP_GRP_TYPE_MPATH, /* hash-threshold nexthop group * default type if not specified diff --git a/include/ufs/ufshcd.h b/include/ufs/ufshcd.h index 60942ed0040f..bf84b00168bc 100644 --- a/include/ufs/ufshcd.h +++ b/include/ufs/ufshcd.h @@ -1410,10 +1410,6 @@ void ufshcd_auto_hibern8_enable(struct ufs_hba *hba); void ufshcd_auto_hibern8_update(struct ufs_hba *hba, u32 ahit); void ufshcd_fixup_dev_quirks(struct ufs_hba *hba, const struct ufs_dev_quirk *fixups); -#define SD_ASCII_STD true -#define SD_RAW false -int ufshcd_read_string_desc(struct ufs_hba *hba, u8 desc_index, - u8 **buf, bool ascii); void ufshcd_hold(struct ufs_hba *hba); void ufshcd_release(struct ufs_hba *hba); diff --git a/init/main.c b/init/main.c index 62d8f7610e84..0598d6aa8be8 100644 --- a/init/main.c +++ b/init/main.c @@ -268,7 +268,8 @@ static void * __init get_boot_config_from_initrd(size_t *_size) u32 *hdr; int i; - if (!initrd_end) + if (!initrd_end || initrd_end < initrd_start || + initrd_end - initrd_start < BOOTCONFIG_MAGIC_LEN + 8) return NULL; data = (char *)initrd_end - BOOTCONFIG_MAGIC_LEN; @@ -285,16 +286,26 @@ static void * __init get_boot_config_from_initrd(size_t *_size) found: hdr = (u32 *)(data - 8); + if ((unsigned long)hdr < initrd_start) + return NULL; + size = le32_to_cpu(hdr[0]); csum = le32_to_cpu(hdr[1]); - data = ((void *)hdr) - size; - if ((unsigned long)data < initrd_start) { - pr_err("bootconfig size %d is greater than initrd size %ld\n", + if (size > XBC_DATA_MAX) { + pr_err("bootconfig size %u is greater than max size %d\n", + size, XBC_DATA_MAX); + return NULL; + } + + if (size > ((unsigned long)hdr - initrd_start)) { + pr_err("bootconfig size %u is greater than initrd size %lu\n", size, initrd_end - initrd_start); return NULL; } + data = ((void *)hdr) - size; + if (xbc_calc_checksum(data, size) != csum) { pr_err("bootconfig checksum failed\n"); return NULL; @@ -397,12 +408,6 @@ static void __init setup_boot_config(void) return; } - if (size >= XBC_DATA_MAX) { - pr_err("bootconfig size %ld greater than max size %d\n", - (long)size, XBC_DATA_MAX); - return; - } - ret = xbc_init(data, size, &msg, &pos); if (ret < 0) { if (pos < 0) @@ -1503,7 +1508,7 @@ static noinline void __init kernel_init_freeable(void) */ set_mems_allowed(node_states[N_MEMORY]); - cad_pid = get_pid(task_pid(current)); + rcu_assign_pointer(cad_pid, get_pid(task_pid(current))); smp_prepare_cpus(setup_max_cpus); diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 712c724a7444..d4973d0d4e43 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -539,7 +539,7 @@ static void io_eventfd_ops(struct rcu_head *rcu) call_rcu(&ev_fd->rcu, io_eventfd_free); } -static void io_eventfd_signal(struct io_ring_ctx *ctx) +static void io_eventfd_signal(struct io_ring_ctx *ctx, bool defer) { struct io_ev_fd *ev_fd = NULL; @@ -562,7 +562,7 @@ static void io_eventfd_signal(struct io_ring_ctx *ctx) if (ev_fd->eventfd_async && !io_wq_current_is_worker()) goto out; - if (likely(eventfd_signal_allowed())) { + if (!defer && likely(eventfd_signal_allowed())) { eventfd_signal_mask(ev_fd->cq_ev_fd, 1, EPOLL_URING_WAKE); } else { atomic_inc(&ev_fd->refs); @@ -596,7 +596,7 @@ static void io_eventfd_flush_signal(struct io_ring_ctx *ctx) if (skip) return; - io_eventfd_signal(ctx); + io_eventfd_signal(ctx, false); } void __io_commit_cqring_flush(struct io_ring_ctx *ctx) @@ -1310,7 +1310,7 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) if (ctx->flags & IORING_SETUP_TASKRUN_FLAG) atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); if (ctx->has_evfd) - io_eventfd_signal(ctx); + io_eventfd_signal(ctx, flags & IOU_F_TWQ_IN_WAKE); } nr_wait = atomic_read(&ctx->cq_wait_nr); @@ -2718,7 +2718,12 @@ struct page **io_pin_pages(unsigned long uaddr, unsigned long len, int *npages) struct page **pages; int ret; - end = (uaddr + len + PAGE_SIZE - 1) >> PAGE_SHIFT; + if (check_add_overflow(uaddr, len, &end)) + return ERR_PTR(-EOVERFLOW); + if (check_add_overflow(end, PAGE_SIZE - 1, &end)) + return ERR_PTR(-EOVERFLOW); + + end = end >> PAGE_SHIFT; start = uaddr >> PAGE_SHIFT; nr_pages = end - start; if (WARN_ON_ONCE(!nr_pages)) diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 0f6b47f55c24..f845bf3eef0b 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -25,6 +25,14 @@ enum { * It's also ignored unless IORING_SETUP_DEFER_TASKRUN is set. */ IOU_F_TWQ_LAZY_WAKE = 1, + + /* + * Set when task_work is queued from a waitqueue wakeup handler, where + * an arbitrary provider waitqueue lock is held. Signaling the CQ ring + * eventfd inline from there can recurse back into that lock through + * epoll, so the eventfd signal must be deferred. + */ + IOU_F_TWQ_IN_WAKE = 2, }; enum { diff --git a/io_uring/poll.c b/io_uring/poll.c index 66a0a9b9950b..c4cfc7a5236d 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -229,19 +229,21 @@ enum { IOU_POLL_REQUEUE = 4, }; -static void __io_poll_execute(struct io_kiocb *req, int mask) +static void __io_poll_execute(struct io_kiocb *req, int mask, + unsigned tw_flags) { io_req_set_res(req, mask, 0); req->io_task_work.func = io_poll_task_func; trace_io_uring_task_add(req, mask); - io_req_task_work_add(req); + __io_req_task_work_add(req, tw_flags); } -static inline void io_poll_execute(struct io_kiocb *req, int res) +static inline void io_poll_execute(struct io_kiocb *req, int res, + unsigned tw_flags) { if (io_poll_get_ownership(req)) - __io_poll_execute(req, res); + __io_poll_execute(req, res, tw_flags); } /* @@ -359,7 +361,7 @@ void io_poll_task_func(struct io_kiocb *req, struct io_tw_state *ts) return; } else if (ret == IOU_POLL_REQUEUE) { io_kbuf_recycle(req, 0); - __io_poll_execute(req, 0); + __io_poll_execute(req, 0, 0); return; } io_poll_remove_entries(req); @@ -397,7 +399,7 @@ static void io_poll_cancel_req(struct io_kiocb *req) { io_poll_mark_cancelled(req); /* kick tw, which should complete the request */ - io_poll_execute(req, 0); + io_poll_execute(req, 0, 0); } #define IO_ASYNC_POLL_COMMON (EPOLLONESHOT | EPOLLPRI) @@ -406,7 +408,7 @@ static __cold int io_pollfree_wake(struct io_kiocb *req, struct io_poll *poll) { io_poll_mark_cancelled(req); /* we have to kick tw in case it's not already */ - io_poll_execute(req, 0); + io_poll_execute(req, 0, IOU_F_TWQ_IN_WAKE); /* * If the waitqueue is being freed early but someone is already @@ -461,7 +463,7 @@ static int io_poll_wake(struct wait_queue_entry *wait, unsigned mode, int sync, else req->flags &= ~REQ_F_SINGLE_POLL; } - __io_poll_execute(req, mask); + __io_poll_execute(req, mask, IOU_F_TWQ_IN_WAKE); } return 1; } @@ -644,7 +646,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, if (mask && (poll->events & EPOLLET) && io_poll_can_finish_inline(req, ipt)) { - __io_poll_execute(req, mask); + __io_poll_execute(req, mask, 0); return 0; } @@ -654,7 +656,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, * poll was waken up, queue up a tw, it'll deal with it. */ if (atomic_cmpxchg(&req->poll_refs, 1, 0) != 1) - __io_poll_execute(req, 0); + __io_poll_execute(req, 0, 0); } return 0; } diff --git a/io_uring/rw.c b/io_uring/rw.c index f2ab967b9507..9c3932302cab 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -244,20 +244,15 @@ static void io_req_end_write(struct io_kiocb *req) } } -/* - * Trigger the notifications after having done some IO, and finish the write - * accounting, if any. - */ -static void io_req_io_end(struct io_kiocb *req) +/* Trigger the notifications after having done some IO. */ +static void io_req_io_notify(struct io_kiocb *req) { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); - if (rw->kiocb.ki_flags & IOCB_WRITE) { - io_req_end_write(req); + if (rw->kiocb.ki_flags & IOCB_WRITE) fsnotify_modify(req->file); - } else { + else fsnotify_access(req->file); - } } static bool __io_complete_rw_common(struct io_kiocb *req, long res) @@ -265,11 +260,8 @@ static bool __io_complete_rw_common(struct io_kiocb *req, long res) if (unlikely(res != req->cqe.res)) { if ((res == -EAGAIN || res == -EOPNOTSUPP) && io_rw_should_reissue(req)) { - /* - * Reissue will start accounting again, finish the - * current cycle. - */ - io_req_io_end(req); + /* Notify for the current cycle before reissuing. */ + io_req_io_notify(req); req->flags |= REQ_F_REISSUE | REQ_F_PARTIAL_IO; return true; } @@ -304,7 +296,7 @@ void io_req_rw_complete(struct io_kiocb *req, struct io_tw_state *ts) io_req_set_res(req, io_fixup_rw_res(req, res), 0); } - io_req_io_end(req); + io_req_io_notify(req); if (req->flags & (REQ_F_BUFFER_SELECTED|REQ_F_BUFFER_RING)) { unsigned issue_flags = ts->locked ? 0 : IO_URING_F_UNLOCKED; @@ -319,6 +311,10 @@ static void io_complete_rw(struct kiocb *kiocb, long res) struct io_rw *rw = container_of(kiocb, struct io_rw, kiocb); struct io_kiocb *req = cmd_to_io_kiocb(rw); + /* ring owner may block in freeze_super() before task_work runs */ + if (kiocb->ki_flags & IOCB_WRITE) + io_req_end_write(req); + if (!kiocb->dio_complete || !(kiocb->ki_flags & IOCB_DIO_CALLER_COMP)) { if (__io_complete_rw_common(req, res)) return; @@ -358,10 +354,12 @@ static int kiocb_done(struct io_kiocb *req, ssize_t ret, if (ret >= 0 && (rw->kiocb.ki_complete == io_complete_rw)) { if (!__io_complete_rw_common(req, ret)) { /* - * Safe to call io_end from here as we're inline + * Safe to notify from here as we're inline * from the submission path. */ - io_req_io_end(req); + if (rw->kiocb.ki_flags & IOCB_WRITE) + io_req_end_write(req); + io_req_io_notify(req); io_req_set_res(req, final_ret, io_put_kbuf(req, issue_flags)); return IOU_OK; diff --git a/kernel/audit_tree.c b/kernel/audit_tree.c index 17125ac3a573..e87c11345b7d 100644 --- a/kernel/audit_tree.c +++ b/kernel/audit_tree.c @@ -543,22 +543,38 @@ static void kill_rules(struct audit_context *context, struct audit_tree *tree) { struct audit_krule *rule, *next; struct audit_entry *entry; + bool need_sync = false; list_for_each_entry_safe(rule, next, &tree->rules, rlist) { entry = container_of(rule, struct audit_entry, rule); - list_del_init(&rule->rlist); if (rule->tree) { /* not a half-baked one */ audit_tree_log_remove_rule(context, rule); - if (entry->rule.exe) - audit_remove_mark(entry->rule.exe); rule->tree = NULL; list_del_rcu(&entry->list); list_del(&entry->rule.list); - call_rcu(&entry->rcu, audit_free_rule_rcu); + if (entry->rule.exe) + need_sync = true; + } else { + list_del_init(&rule->rlist); } } + + if (list_empty(&tree->rules)) + return; + + if (need_sync) + synchronize_rcu(); + + list_for_each_entry_safe(rule, next, &tree->rules, rlist) { + entry = container_of(rule, struct audit_entry, rule); + + list_del_init(&rule->rlist); + if (entry->rule.exe) + audit_remove_mark(entry->rule.exe); + call_rcu(&entry->rcu, audit_free_rule_rcu); + } } /* diff --git a/kernel/bpf/arraymap.c b/kernel/bpf/arraymap.c index ec5c489ed313..ca135fa407b8 100644 --- a/kernel/bpf/arraymap.c +++ b/kernel/bpf/arraymap.c @@ -909,22 +909,44 @@ static void *prog_fd_array_get_ptr(struct bpf_map *map, struct file *map_file, int fd) { struct bpf_prog *prog = bpf_prog_get(fd); + bool is_extended; if (IS_ERR(prog)) return prog; - if (!bpf_prog_map_compatible(map, prog)) { + if (prog->type == BPF_PROG_TYPE_EXT || + !bpf_prog_map_compatible(map, prog)) { bpf_prog_put(prog); return ERR_PTR(-EINVAL); } + mutex_lock(&prog->aux->ext_mutex); + is_extended = prog->aux->is_extended; + if (!is_extended) + prog->aux->prog_array_member_cnt++; + mutex_unlock(&prog->aux->ext_mutex); + if (is_extended) { + /* Extended prog can not be tail callee. It's to prevent a + * potential infinite loop like: + * tail callee prog entry -> tail callee prog subprog -> + * freplace prog entry --tailcall-> tail callee prog entry. + */ + bpf_prog_put(prog); + return ERR_PTR(-EBUSY); + } + return prog; } static void prog_fd_array_put_ptr(struct bpf_map *map, void *ptr, bool need_defer) { + struct bpf_prog *prog = ptr; + + mutex_lock(&prog->aux->ext_mutex); + prog->aux->prog_array_member_cnt--; + mutex_unlock(&prog->aux->ext_mutex); /* bpf_prog is freed after one RCU or tasks trace grace period */ - bpf_prog_put(ptr); + bpf_prog_put(prog); } static u32 prog_fd_array_sys_lookup_elem(void *ptr) diff --git a/kernel/bpf/btf.c b/kernel/bpf/btf.c index 60018b87589d..5cef2bb2ddfc 100644 --- a/kernel/bpf/btf.c +++ b/kernel/bpf/btf.c @@ -3424,7 +3424,15 @@ static int btf_get_field_type(const char *name, u32 field_mask, u32 *seen_mask, field_mask_test_name(BPF_LIST_NODE, "bpf_list_node"); field_mask_test_name(BPF_RB_ROOT, "bpf_rb_root"); field_mask_test_name(BPF_RB_NODE, "bpf_rb_node"); - field_mask_test_name(BPF_REFCOUNT, "bpf_refcount"); + if (field_mask & BPF_REFCOUNT) { + if (!strcmp(name, "bpf_refcount")) { + if (*seen_mask & BPF_REFCOUNT) + return -E2BIG; + *seen_mask |= BPF_REFCOUNT; + type = BPF_REFCOUNT; + goto end; + } + } /* Only return BPF_KPTR when all other types with matchable names fail */ if (field_mask & BPF_KPTR) { @@ -6099,6 +6107,10 @@ bool btf_ctx_access(int off, int size, enum bpf_access_type type, if (prog_args_trusted(prog)) info->reg_type |= PTR_TRUSTED; + /* Raw tracepoint arguments always get marked as maybe NULL */ + if (bpf_prog_is_raw_tp(prog)) + info->reg_type |= PTR_MAYBE_NULL; + if (tgt_prog) { enum bpf_prog_type tgt_type; @@ -6156,7 +6168,7 @@ enum bpf_struct_walk_result { static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf, const struct btf_type *t, int off, int size, u32 *next_btf_id, enum bpf_type_flag *flag, - const char **field_name) + const char **field_name, bool walk_flex_arrays) { u32 i, moff, mtrue_end, msize = 0, total_nelems = 0; const struct btf_type *mtype, *elem_type = NULL; @@ -6183,11 +6195,14 @@ static int btf_struct_walk(struct bpf_verifier_log *log, const struct btf *btf, *flag |= PTR_UNTRUSTED; if (off + size > t->size) { + struct btf_array *array_elem; + + if (!walk_flex_arrays) + goto error; + /* If the last element is a variable size array, we may * need to relax the rule. */ - struct btf_array *array_elem; - if (vlen == 0) goto error; @@ -6448,7 +6463,8 @@ int btf_struct_access(struct bpf_verifier_log *log, t = btf_type_by_id(btf, id); do { - err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag, field_name); + err = btf_struct_walk(log, btf, t, off, size, &id, &tmp_flag, + field_name, !type_is_alloc(reg->type)); switch (err) { case WALK_PTR: @@ -6504,7 +6520,7 @@ bool btf_types_are_same(const struct btf *btf1, u32 id1, bool btf_struct_ids_match(struct bpf_verifier_log *log, const struct btf *btf, u32 id, int off, const struct btf *need_btf, u32 need_type_id, - bool strict) + bool strict, bool walk_flex_arrays) { const struct btf_type *type; enum bpf_type_flag flag = 0; @@ -6523,7 +6539,8 @@ bool btf_struct_ids_match(struct bpf_verifier_log *log, type = btf_type_by_id(btf, id); if (!type) return false; - err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL); + err = btf_struct_walk(log, btf, type, off, 1, &id, &flag, NULL, + walk_flex_arrays); if (err != WALK_STRUCT) return false; diff --git a/kernel/bpf/cgroup.c b/kernel/bpf/cgroup.c index fa6f818c8eab..790de12ca57f 100644 --- a/kernel/bpf/cgroup.c +++ b/kernel/bpf/cgroup.c @@ -856,6 +856,20 @@ static void replace_effective_prog(struct cgroup *cgrp, } } +static bool cgroup_bpf_storages_compatible(struct bpf_prog *old_prog, + struct bpf_prog *new_prog) +{ + enum bpf_cgroup_storage_type stype; + + for_each_cgroup_storage_type(stype) { + if (old_prog->aux->cgroup_storage[stype] != + new_prog->aux->cgroup_storage[stype]) + return false; + } + + return true; +} + /** * __cgroup_bpf_replace() - Replace link's program and propagate the change * to descendants @@ -894,6 +908,9 @@ static int __cgroup_bpf_replace(struct cgroup *cgrp, if (!found) return -ENOENT; + if (!cgroup_bpf_storages_compatible(link->link.prog, new_prog)) + return -EINVAL; + old_prog = xchg(&link->link.prog, new_prog); replace_effective_prog(cgrp, atype, pl); bpf_prog_put(old_prog); @@ -1416,7 +1433,7 @@ int __cgroup_bpf_run_filter_skb(struct sock *sk, struct sk_buff *skb, enum cgroup_bpf_attach_type atype) { - unsigned int offset = skb->data - skb_network_header(skb); + unsigned int offset = -skb_network_offset(skb); struct sock *save_sk; void *saved_data_end; struct cgroup *cgrp; diff --git a/kernel/bpf/core.c b/kernel/bpf/core.c index 9fa984ee076b..175ef2a64cb1 100644 --- a/kernel/bpf/core.c +++ b/kernel/bpf/core.c @@ -123,6 +123,7 @@ struct bpf_prog *bpf_prog_alloc_no_stats(unsigned int size, gfp_t gfp_extra_flag INIT_LIST_HEAD_RCU(&fp->aux->ksym.lnode); mutex_init(&fp->aux->used_maps_mutex); + mutex_init(&fp->aux->ext_mutex); mutex_init(&fp->aux->dst_mutex); return fp; @@ -1190,8 +1191,7 @@ bpf_jit_binary_pack_alloc(unsigned int proglen, u8 **image_ptr, } /* Copy JITed text from rw_header to its final location, the ro_header. */ -int bpf_jit_binary_pack_finalize(struct bpf_prog *prog, - struct bpf_binary_header *ro_header, +int bpf_jit_binary_pack_finalize(struct bpf_binary_header *ro_header, struct bpf_binary_header *rw_header) { void *ptr; @@ -2484,7 +2484,9 @@ struct bpf_prog *bpf_prog_select_runtime(struct bpf_prog *fp, int *err) } finalize: - bpf_prog_lock_ro(fp); + *err = bpf_prog_lock_ro(fp); + if (*err) + return fp; /* The tail call compatibility check can only be done at * this late stage as we need to determine, if we deal @@ -2959,7 +2961,10 @@ EXPORT_SYMBOL_GPL(bpf_event_output); /* Always built-in helper functions. */ const struct bpf_func_proto bpf_tail_call_proto = { - .func = NULL, + /* func is unused for tail_call, we set it to pass the + * get_helper_proto check + */ + .func = BPF_PTR_POISON, .gpl_only = false, .ret_type = RET_VOID, .arg1_type = ARG_PTR_TO_CTX, diff --git a/kernel/bpf/hashtab.c b/kernel/bpf/hashtab.c index e14a0caa07bf..fb7f64c494b0 100644 --- a/kernel/bpf/hashtab.c +++ b/kernel/bpf/hashtab.c @@ -197,12 +197,12 @@ static bool htab_is_percpu(const struct bpf_htab *htab) static inline void htab_elem_set_ptr(struct htab_elem *l, u32 key_size, void __percpu *pptr) { - *(void __percpu **)(l->key + key_size) = pptr; + *(void __percpu **)(l->key + roundup(key_size, 8)) = pptr; } static inline void __percpu *htab_elem_get_ptr(struct htab_elem *l, u32 key_size) { - return *(void __percpu **)(l->key + key_size); + return *(void __percpu **)(l->key + roundup(key_size, 8)); } static void *fd_htab_map_get_ptr(const struct bpf_map *map, struct htab_elem *l) diff --git a/kernel/bpf/helpers.c b/kernel/bpf/helpers.c index 1fa140915b0c..4236e381d280 100644 --- a/kernel/bpf/helpers.c +++ b/kernel/bpf/helpers.c @@ -1533,10 +1533,24 @@ void bpf_timer_cancel_and_free(void *val) * To avoid these issues, punt to workqueue context when we are in a * timer callback. */ - if (this_cpu_read(hrtimer_running)) + if (this_cpu_read(hrtimer_running)) { queue_work(system_unbound_wq, &t->cb.delete_work); - else + return; + } + + if (IS_ENABLED(CONFIG_PREEMPT_RT)) { + /* If the timer is running on other CPU, also use a kworker to + * wait for the completion of the timer instead of trying to + * acquire a sleepable lock in hrtimer_cancel() to wait for its + * completion. + */ + if (hrtimer_try_to_cancel(&t->timer) >= 0) + kfree_rcu(t, cb.rcu); + else + queue_work(system_unbound_wq, &t->cb.delete_work); + } else { bpf_timer_delete_work(&t->cb.delete_work); + } } BPF_CALL_2(bpf_kptr_xchg, void *, map_value, void *, ptr) @@ -2005,6 +2019,7 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, struct bpf_spin_lock *spin_lock) { struct rb_root_cached orig_root, *root = rb_root; + struct bpf_rb_node_kern *node; struct rb_node *pos, *n; void *obj; @@ -2013,14 +2028,20 @@ void bpf_rb_root_free(const struct btf_field *field, void *rb_root, __bpf_spin_lock_irqsave(spin_lock); orig_root = *root; + bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) { + node = rb_entry(pos, struct bpf_rb_node_kern, rb_node); + WRITE_ONCE(node->owner, BPF_PTR_POISON); + } *root = RB_ROOT_CACHED; __bpf_spin_unlock_irqrestore(spin_lock); bpf_rbtree_postorder_for_each_entry_safe(pos, n, &orig_root.rb_root) { obj = pos; obj -= field->graph_root.node_offset; - - + node = rb_entry(pos, struct bpf_rb_node_kern, rb_node); + RB_CLEAR_NODE(pos); + /* Ensure __bpf_rbtree_add() sees the node as unlinked. */ + smp_store_release(&node->owner, NULL); migrate_disable(); __bpf_obj_drop_impl(obj, field->graph_root.value_rec); migrate_enable(); diff --git a/kernel/bpf/mmap_unlock_work.h b/kernel/bpf/mmap_unlock_work.h index 5d18d7d85bef..1834db20b861 100644 --- a/kernel/bpf/mmap_unlock_work.h +++ b/kernel/bpf/mmap_unlock_work.h @@ -4,12 +4,15 @@ #ifndef __MMAP_UNLOCK_WORK_H__ #define __MMAP_UNLOCK_WORK_H__ +#include +#include #include /* irq_work to run mmap_read_unlock() in irq_work */ struct mmap_unlock_irq_work { struct irq_work irq_work; struct mm_struct *mm; + atomic_t active; }; DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work); @@ -18,32 +21,36 @@ DECLARE_PER_CPU(struct mmap_unlock_irq_work, mmap_unlock_work); * We cannot do mmap_read_unlock() when the irq is disabled, because of * risk to deadlock with rq_lock. To look up vma when the irqs are * disabled, we need to run mmap_read_unlock() in irq_work. We use a - * percpu variable to do the irq_work. If the irq_work is already used - * by another lookup, we fall over. + * percpu variable to do the irq_work. The active flag reserves the slot + * before mmap_read_trylock() and until the irq_work callback consumes mm. */ -static inline bool bpf_mmap_unlock_get_irq_work(struct mmap_unlock_irq_work **work_ptr) +static inline struct mmap_unlock_irq_work *bpf_mmap_unlock_guard_get(void) { - struct mmap_unlock_irq_work *work = NULL; - bool irq_work_busy = false; + struct mmap_unlock_irq_work *work; - if (irqs_disabled()) { - if (!IS_ENABLED(CONFIG_PREEMPT_RT)) { - work = this_cpu_ptr(&mmap_unlock_work); - if (irq_work_is_busy(&work->irq_work)) { - /* cannot queue more up_read, fallback */ - irq_work_busy = true; - } - } else { - /* - * PREEMPT_RT does not allow to trylock mmap sem in - * interrupt disabled context. Force the fallback code. - */ - irq_work_busy = true; - } - } + if (!irqs_disabled()) + return NULL; + + /* + * PREEMPT_RT does not allow to trylock mmap sem in interrupt + * disabled context. Force the fallback code. + */ + if (IS_ENABLED(CONFIG_PREEMPT_RT)) + return ERR_PTR(-EBUSY); + + work = this_cpu_ptr(&mmap_unlock_work); + if (irq_work_is_busy(&work->irq_work) || + atomic_cmpxchg_acquire(&work->active, 0, 1)) + return ERR_PTR(-EBUSY); - *work_ptr = work; - return irq_work_busy; + return work; +} + +static inline void +bpf_mmap_unlock_guard_put(struct mmap_unlock_irq_work *work) +{ + if (work) + atomic_set_release(&work->active, 0); } static inline void bpf_mmap_unlock_mm(struct mmap_unlock_irq_work *work, struct mm_struct *mm) diff --git a/kernel/bpf/offload.c b/kernel/bpf/offload.c index 3345de2da15d..ae0c8e789458 100644 --- a/kernel/bpf/offload.c +++ b/kernel/bpf/offload.c @@ -847,10 +847,11 @@ void *bpf_dev_bound_resolve_kfunc(struct bpf_prog *prog, u32 func_id) if (!ops) goto out; - if (func_id == bpf_xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_TIMESTAMP)) - p = ops->xmo_rx_timestamp; - else if (func_id == bpf_xdp_metadata_kfunc_id(XDP_METADATA_KFUNC_RX_HASH)) - p = ops->xmo_rx_hash; +#define XDP_METADATA_KFUNC(name, _, xmo) \ + if (func_id == bpf_xdp_metadata_kfunc_id(name)) p = ops->xmo; + XDP_METADATA_KFUNC_xxx +#undef XDP_METADATA_KFUNC + out: up_read(&bpf_devs_lock); diff --git a/kernel/bpf/queue_stack_maps.c b/kernel/bpf/queue_stack_maps.c index d869f51ea93a..36384342d7d8 100644 --- a/kernel/bpf/queue_stack_maps.c +++ b/kernel/bpf/queue_stack_maps.c @@ -99,8 +99,10 @@ static long __queue_map_get(struct bpf_map *map, void *value, bool delete) void *ptr; if (in_nmi()) { - if (!raw_spin_trylock_irqsave(&qs->lock, flags)) + if (!raw_spin_trylock_irqsave(&qs->lock, flags)) { + memset(value, 0, qs->map.value_size); return -EBUSY; + } } else { raw_spin_lock_irqsave(&qs->lock, flags); } @@ -134,8 +136,10 @@ static long __stack_map_get(struct bpf_map *map, void *value, bool delete) u32 index; if (in_nmi()) { - if (!raw_spin_trylock_irqsave(&qs->lock, flags)) + if (!raw_spin_trylock_irqsave(&qs->lock, flags)) { + memset(value, 0, qs->map.value_size); return -EBUSY; + } } else { raw_spin_lock_irqsave(&qs->lock, flags); } diff --git a/kernel/bpf/stackmap.c b/kernel/bpf/stackmap.c index a330f38ae733..4db255334617 100644 --- a/kernel/bpf/stackmap.c +++ b/kernel/bpf/stackmap.c @@ -128,24 +128,20 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, u64 *ips, u32 trace_nr, bool user) { int i; - struct mmap_unlock_irq_work *work = NULL; - bool irq_work_busy = bpf_mmap_unlock_get_irq_work(&work); + struct mmap_unlock_irq_work *work; struct vm_area_struct *vma, *prev_vma = NULL; const char *prev_build_id; - /* If the irq_work is in use, fall back to report ips. Same - * fallback is used for kernel stack (!user) on a stackmap with - * build_id. - */ - if (!user || !current || !current->mm || irq_work_busy || - !mmap_read_trylock(current->mm)) { - /* cannot access current->mm, fall back to ips */ - for (i = 0; i < trace_nr; i++) { - id_offs[i].status = BPF_STACK_BUILD_ID_IP; - id_offs[i].ip = ips[i]; - memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX); - } - return; + if (!user || !current || !current->mm) + goto fallback; + + work = bpf_mmap_unlock_guard_get(); + if (IS_ERR(work)) + goto fallback; + + if (!mmap_read_trylock(current->mm)) { + bpf_mmap_unlock_guard_put(work); + goto fallback; } for (i = 0; i < trace_nr; i++) { @@ -171,6 +167,15 @@ static void stack_map_get_build_id_offset(struct bpf_stack_build_id *id_offs, prev_build_id = id_offs[i].build_id; } bpf_mmap_unlock_mm(work, current->mm); + return; + +fallback: + /* cannot access current->mm, fall back to ips */ + for (i = 0; i < trace_nr; i++) { + id_offs[i].status = BPF_STACK_BUILD_ID_IP; + id_offs[i].ip = ips[i]; + memset(id_offs[i].build_id, 0, BUILD_ID_SIZE_MAX); + } } static struct perf_callchain_entry * @@ -216,7 +221,7 @@ static long __bpf_get_stackid(struct bpf_map *map, struct bpf_stack_map *smap = container_of(map, struct bpf_stack_map, map); struct stack_map_bucket *bucket, *new_bucket, *old_bucket; u32 skip = flags & BPF_F_SKIP_FIELD_MASK; - u32 hash, id, trace_nr, trace_len; + u32 hash, id, trace_nr, trace_len, max_depth; bool user = flags & BPF_F_USER_STACK; u64 *ips; bool hash_matches; @@ -225,7 +230,10 @@ static long __bpf_get_stackid(struct bpf_map *map, /* skipping more than usable stack trace */ return -EFAULT; - trace_nr = trace->nr - skip; + max_depth = map->value_size / stack_map_data_size(map) + skip; + if (max_depth > sysctl_perf_event_max_stack) + max_depth = sysctl_perf_event_max_stack; + trace_nr = min_t(u32, trace->nr - skip, max_depth - skip); trace_len = trace_nr * sizeof(u64); ips = trace->ip + skip; hash = jhash2((u32 *)ips, trace_len / sizeof(u32), 0); @@ -297,6 +305,7 @@ BPF_CALL_3(bpf_get_stackid, struct pt_regs *, regs, struct bpf_map *, map, if (max_depth > sysctl_perf_event_max_stack) max_depth = sysctl_perf_event_max_stack; + guard(preempt)(); trace = get_perf_callchain(regs, 0, kernel, user, max_depth, false, false); @@ -354,15 +363,11 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, return -EFAULT; nr_kernel = count_kernel_ip(trace); + __u64 nr = trace->nr; /* save original */ if (kernel) { - __u64 nr = trace->nr; - trace->nr = nr_kernel; ret = __bpf_get_stackid(map, trace, flags); - - /* restore nr */ - trace->nr = nr; } else { /* user */ u64 skip = flags & BPF_F_SKIP_FIELD_MASK; @@ -373,6 +378,10 @@ BPF_CALL_3(bpf_get_stackid_pe, struct bpf_perf_event_data_kern *, ctx, flags = (flags & ~BPF_F_SKIP_FIELD_MASK) | skip; ret = __bpf_get_stackid(map, trace, flags); } + + /* restore nr */ + trace->nr = nr; + return ret; } @@ -427,6 +436,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, if (sysctl_perf_event_max_stack < max_depth) max_depth = sysctl_perf_event_max_stack; + preempt_disable(); if (trace_in) trace = trace_in; else if (kernel && task) @@ -434,11 +444,10 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, else trace = get_perf_callchain(regs, 0, kernel, user, max_depth, crosstask, false); - if (unlikely(!trace)) - goto err_fault; - - if (trace->nr < skip) + if (unlikely(!trace) || trace->nr < skip) { + preempt_enable(); goto err_fault; + } trace_nr = trace->nr - skip; trace_nr = (trace_nr <= num_elem) ? trace_nr : num_elem; @@ -449,6 +458,7 @@ static long __bpf_get_stack(struct pt_regs *regs, struct task_struct *task, stack_map_get_build_id_offset(buf, ips, trace_nr, user); else memcpy(buf, ips, copy_len); + preempt_enable(); if (size > copy_len) memset(buf + copy_len, 0, size - copy_len); @@ -483,14 +493,17 @@ BPF_CALL_4(bpf_get_task_stack, struct task_struct *, task, void *, buf, struct pt_regs *regs; long res = -EINVAL; - if (!try_get_task_stack(task)) + if (!try_get_task_stack(task)) { + memset(buf, 0, size); return -EFAULT; + } regs = task_pt_regs(task); if (regs) res = __bpf_get_stack(regs, task, NULL, buf, size, flags); + else + memset(buf, 0, size); put_task_stack(task); - return res; } diff --git a/kernel/bpf/syscall.c b/kernel/bpf/syscall.c index 1710391f386a..599ae0b13591 100644 --- a/kernel/bpf/syscall.c +++ b/kernel/bpf/syscall.c @@ -3125,7 +3125,8 @@ static void bpf_tracing_link_release(struct bpf_link *link) container_of(link, struct bpf_tracing_link, link.link); WARN_ON_ONCE(bpf_trampoline_unlink_prog(&tr_link->link, - tr_link->trampoline)); + tr_link->trampoline, + tr_link->tgt_prog)); bpf_trampoline_put(tr_link->trampoline); @@ -3260,7 +3261,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, * in prog->aux * * - if prog->aux->dst_trampoline is NULL, the program has already been - * attached to a target and its initial target was cleared (below) + * attached to a target and its initial target was cleared (below) * * - if tgt_prog != NULL, the caller specified tgt_prog_fd + * target_btf_id using the link_create API. @@ -3335,7 +3336,7 @@ static int bpf_tracing_prog_attach(struct bpf_prog *prog, if (err) goto out_unlock; - err = bpf_trampoline_link_prog(&link->link, tr); + err = bpf_trampoline_link_prog(&link->link, tr, tgt_prog); if (err) { bpf_link_cleanup(&link_primer); link = NULL; @@ -4017,6 +4018,8 @@ static int bpf_prog_detach(const union bpf_attr *attr) prog = bpf_prog_get_type(attr->attach_bpf_fd, ptype); if (IS_ERR(prog)) return PTR_ERR(prog); + } else if (!bpf_mprog_detach_empty(ptype)) { + return -EPERM; } } else if (attr->attach_flags || attr->relative_fd || @@ -5682,6 +5685,7 @@ EXPORT_SYMBOL(kern_sys_bpf); static const struct bpf_func_proto bpf_sys_bpf_proto = { .func = bpf_sys_bpf, .gpl_only = false, + .might_sleep = true, .ret_type = RET_INTEGER, .arg1_type = ARG_ANYTHING, .arg2_type = ARG_PTR_TO_MEM | MEM_RDONLY, @@ -5707,6 +5711,7 @@ BPF_CALL_1(bpf_sys_close, u32, fd) static const struct bpf_func_proto bpf_sys_close_proto = { .func = bpf_sys_close, .gpl_only = false, + .might_sleep = true, .ret_type = RET_INTEGER, .arg1_type = ARG_ANYTHING, }; diff --git a/kernel/bpf/task_iter.c b/kernel/bpf/task_iter.c index 30dd2b31a38d..481e2b43f0ed 100644 --- a/kernel/bpf/task_iter.c +++ b/kernel/bpf/task_iter.c @@ -780,9 +780,8 @@ static struct bpf_iter_reg task_vma_reg_info = { BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, bpf_callback_t, callback_fn, void *, callback_ctx, u64, flags) { - struct mmap_unlock_irq_work *work = NULL; + struct mmap_unlock_irq_work *work; struct vm_area_struct *vma; - bool irq_work_busy = false; bool __maybe_unused mmput_needed = false; struct mm_struct *mm; int ret = -ENOENT; @@ -819,9 +818,14 @@ BPF_CALL_5(bpf_find_vma, struct task_struct *, task, u64, start, if (!mm) return -ENOENT; - irq_work_busy = bpf_mmap_unlock_get_irq_work(&work); + work = bpf_mmap_unlock_guard_get(); + if (IS_ERR(work)) { + ret = PTR_ERR(work); + goto out; + } - if (irq_work_busy || !mmap_read_trylock(mm)) { + if (!mmap_read_trylock(mm)) { + bpf_mmap_unlock_guard_put(work); ret = -EBUSY; goto out; } @@ -864,6 +868,8 @@ static void do_mmap_read_unlock(struct irq_work *entry) work = container_of(entry, struct mmap_unlock_irq_work, irq_work); mmap_read_unlock_non_owner(work->mm); + work->mm = NULL; + bpf_mmap_unlock_guard_put(work); } static int __init task_iter_init(void) diff --git a/kernel/bpf/trampoline.c b/kernel/bpf/trampoline.c index 6f7968d3704e..4eebb9e22d26 100644 --- a/kernel/bpf/trampoline.c +++ b/kernel/bpf/trampoline.c @@ -392,6 +392,13 @@ static struct bpf_tramp_image *bpf_tramp_image_alloc(u64 key) return ERR_PTR(err); } +void bpf_trampoline_set_flags(struct bpf_trampoline *tr, u32 flags) +{ + mutex_lock(&tr->mutex); + tr->flags |= flags; + mutex_unlock(&tr->mutex); +} + static int bpf_trampoline_update(struct bpf_trampoline *tr, bool lock_direct_mutex) { struct bpf_tramp_image *im; @@ -508,7 +515,27 @@ static enum bpf_tramp_prog_type bpf_attach_type_to_tramp(struct bpf_prog *prog) } } -static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr) +static int bpf_freplace_check_tgt_prog(struct bpf_prog *tgt_prog) +{ + struct bpf_prog_aux *aux = tgt_prog->aux; + + guard(mutex)(&aux->ext_mutex); + if (aux->prog_array_member_cnt) + /* Program extensions can not extend target prog when the target + * prog has been updated to any prog_array map as tail callee. + * It's to prevent a potential infinite loop like: + * tgt prog entry -> tgt prog subprog -> freplace prog entry + * --tailcall-> tgt prog entry. + */ + return -EBUSY; + + aux->is_extended = true; + return 0; +} + +static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, + struct bpf_trampoline *tr, + struct bpf_prog *tgt_prog) { enum bpf_tramp_prog_type kind; struct bpf_tramp_link *link_exiting; @@ -529,6 +556,9 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_tr /* Cannot attach extension if fentry/fexit are in use. */ if (cnt) return -EBUSY; + err = bpf_freplace_check_tgt_prog(tgt_prog); + if (err) + return err; tr->extension_prog = link->link.prog; return bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, NULL, link->link.prog->bpf_func); @@ -555,17 +585,21 @@ static int __bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_tr return err; } -int bpf_trampoline_link_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr) +int bpf_trampoline_link_prog(struct bpf_tramp_link *link, + struct bpf_trampoline *tr, + struct bpf_prog *tgt_prog) { int err; mutex_lock(&tr->mutex); - err = __bpf_trampoline_link_prog(link, tr); + err = __bpf_trampoline_link_prog(link, tr, tgt_prog); mutex_unlock(&tr->mutex); return err; } -static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr) +static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, + struct bpf_trampoline *tr, + struct bpf_prog *tgt_prog) { enum bpf_tramp_prog_type kind; int err; @@ -576,6 +610,8 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_ err = bpf_arch_text_poke(tr->func.addr, BPF_MOD_JUMP, tr->extension_prog->bpf_func, NULL); tr->extension_prog = NULL; + guard(mutex)(&tgt_prog->aux->ext_mutex); + tgt_prog->aux->is_extended = false; return err; } hlist_del_init(&link->tramp_hlist); @@ -584,12 +620,14 @@ static int __bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_ } /* bpf_trampoline_unlink_prog() should never fail. */ -int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, struct bpf_trampoline *tr) +int bpf_trampoline_unlink_prog(struct bpf_tramp_link *link, + struct bpf_trampoline *tr, + struct bpf_prog *tgt_prog) { int err; mutex_lock(&tr->mutex); - err = __bpf_trampoline_unlink_prog(link, tr); + err = __bpf_trampoline_unlink_prog(link, tr, tgt_prog); mutex_unlock(&tr->mutex); return err; } @@ -604,7 +642,7 @@ static void bpf_shim_tramp_link_release(struct bpf_link *link) if (!shim_link->trampoline) return; - WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link, shim_link->trampoline)); + WARN_ON_ONCE(bpf_trampoline_unlink_prog(&shim_link->link, shim_link->trampoline, NULL)); bpf_trampoline_put(shim_link->trampoline); } @@ -716,7 +754,7 @@ int bpf_trampoline_link_cgroup_shim(struct bpf_prog *prog, goto err; } - err = __bpf_trampoline_link_prog(&shim_link->link, tr); + err = __bpf_trampoline_link_prog(&shim_link->link, tr, NULL); if (err) goto err; diff --git a/kernel/bpf/verifier.c b/kernel/bpf/verifier.c index 778a68bfb91d..d020de130185 100644 --- a/kernel/bpf/verifier.c +++ b/kernel/bpf/verifier.c @@ -453,9 +453,18 @@ static bool reg_not_null(const struct bpf_reg_state *reg) if (type_may_be_null(type)) return false; + /* + * The types below guarantee a non-NULL base, an unbounded offset can + * still wrap base + offset to zero. + */ + if (reg->smin_value <= -BPF_MAX_VAR_OFF || reg->smax_value >= BPF_MAX_VAR_OFF) + return false; + type = base_type(type); return type == PTR_TO_SOCKET || type == PTR_TO_TCP_SOCK || + type == PTR_TO_XDP_SOCK || + type == PTR_TO_BUF || type == PTR_TO_MAP_VALUE || type == PTR_TO_MAP_KEY || type == PTR_TO_SOCK_COMMON || @@ -489,6 +498,25 @@ static struct btf_record *reg_btf_record(const struct bpf_reg_state *reg) return rec; } +static bool mask_raw_tp_reg_cond(const struct bpf_verifier_env *env, struct bpf_reg_state *reg) { + return reg->type == (PTR_TO_BTF_ID | PTR_TRUSTED | PTR_MAYBE_NULL) && + bpf_prog_is_raw_tp(env->prog) && !reg->ref_obj_id; +} + +static bool mask_raw_tp_reg(const struct bpf_verifier_env *env, struct bpf_reg_state *reg) +{ + if (!mask_raw_tp_reg_cond(env, reg)) + return false; + reg->type &= ~PTR_MAYBE_NULL; + return true; +} + +static void unmask_raw_tp_reg(struct bpf_reg_state *reg, bool result) +{ + if (result) + reg->type |= PTR_MAYBE_NULL; +} + static bool subprog_is_global(const struct bpf_verifier_env *env, int subprog) { struct bpf_func_info_aux *aux = env->prog->aux->func_info_aux; @@ -677,6 +705,13 @@ static void print_liveness(struct bpf_verifier_env *env, verbose(env, "D"); } +static bool is_atomic_fetch_insn(const struct bpf_insn *insn) +{ + return BPF_CLASS(insn->code) == BPF_STX && + BPF_MODE(insn->code) == BPF_ATOMIC && + (insn->imm & BPF_FETCH); +} + static int __get_spi(s32 off) { return (-off - 1) / BPF_REG_SIZE; @@ -1804,6 +1839,7 @@ static int copy_verifier_state(struct bpf_verifier_state *dst_state, dst_state->dfs_depth = src->dfs_depth; dst_state->callback_unroll_depth = src->callback_unroll_depth; dst_state->used_as_loop_entry = src->used_as_loop_entry; + dst_state->loop_entry = src->loop_entry; for (i = 0; i <= src->curframe; i++) { dst = dst_state->frame[i]; if (!dst) { @@ -3924,10 +3960,22 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, * dreg still needs precision before this insn */ } - } else if (class == BPF_LDX) { - if (!bt_is_reg_set(bt, dreg)) + } else if (class == BPF_LDX || is_atomic_fetch_insn(insn)) { + u32 load_reg = dreg; + + /* + * Atomic fetch operation writes the old value into + * a register (sreg or r0) and if it was tracked for + * precision, propagate to the stack slot like we do + * in regular ldx. + */ + if (is_atomic_fetch_insn(insn)) + load_reg = insn->imm == BPF_CMPXCHG ? + BPF_REG_0 : sreg; + + if (!bt_is_reg_set(bt, load_reg)) return 0; - bt_clear_reg(bt, dreg); + bt_clear_reg(bt, load_reg); /* scalars can only be spilled into stack w/o losing precision. * Load from any other memory can be zero extended. @@ -4063,37 +4111,34 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, return -EFAULT; } } else if (opcode == BPF_EXIT) { - bool r0_precise; + bool from_subprog_call, r0_precise; + + /* BPF_EXIT in subprog or callback always returns + * right after the call instruction, so by checking + * whether the instruction at subseq_idx-1 is subprog + * call or not we can distinguish actual exit from + * *subprog* from exit from *callback*. In the former + * case, we need to propagate r0 precision, if + * necessary. In the former we never do that. + */ + from_subprog_call = subseq_idx - 1 >= 0 && + bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]); + + r0_precise = from_subprog_call && bt_is_reg_set(bt, BPF_REG_0); /* Backtracking to a nested function call, 'idx' is a part of * the inner frame 'subseq_idx' is a part of the outer frame. * In case of a regular function call, instructions giving * precision to registers R1-R5 should have been found already. - * In case of a callback, it is ok to have R1-R5 marked for - * backtracking, as these registers are set by the function - * invoking callback. + * In case of a callback from bpf_loop(), R{1,4} in the calling + * frame would be set as precise and that is correct. */ - if (subseq_idx >= 0 && calls_callback(env, subseq_idx)) - for (i = BPF_REG_1; i <= BPF_REG_5; i++) - bt_clear_reg(bt, i); - if (bt_reg_mask(bt) & BPF_REGMASK_ARGS) { + if (from_subprog_call && (bt_reg_mask(bt) & BPF_REGMASK_ARGS)) { verbose(env, "BUG regs %x\n", bt_reg_mask(bt)); WARN_ONCE(1, "verifier backtracking bug"); return -EFAULT; } - /* BPF_EXIT in subprog or callback always returns - * right after the call instruction, so by checking - * whether the instruction at subseq_idx-1 is subprog - * call or not we can distinguish actual exit from - * *subprog* from exit from *callback*. In the former - * case, we need to propagate r0 precision, if - * necessary. In the former we never do that. - */ - r0_precise = subseq_idx - 1 >= 0 && - bpf_pseudo_call(&env->prog->insnsi[subseq_idx - 1]) && - bt_is_reg_set(bt, BPF_REG_0); - bt_clear_reg(bt, BPF_REG_0); if (bt_subprog_enter(bt)) return -EFAULT; @@ -4113,8 +4158,10 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, * before it would be equally necessary to * propagate it to dreg. */ - bt_set_reg(bt, dreg); - bt_set_reg(bt, sreg); + if (!hist || !(hist->flags & INSN_F_SRC_REG_STACK)) + bt_set_reg(bt, sreg); + if (!hist || !(hist->flags & INSN_F_DST_REG_STACK)) + bt_set_reg(bt, dreg); } else if (BPF_SRC(insn->code) == BPF_K) { /* dreg K * Only dreg still needs precision before @@ -4123,16 +4170,29 @@ static int backtrack_insn(struct bpf_verifier_env *env, int idx, int subseq_idx, */ } } else if (class == BPF_LD) { - if (!bt_is_reg_set(bt, dreg)) - return 0; - bt_clear_reg(bt, dreg); /* It's ld_imm64 or ld_abs or ld_ind. * For ld_imm64 no further tracking of precision * into parent is necessary */ - if (mode == BPF_IND || mode == BPF_ABS) - /* to be analyzed */ - return -ENOTSUPP; + if (mode == BPF_IMM) { + bt_clear_reg(bt, dreg); + return 0; + } + /* + * BPF_{IND,ABS} are modelled as two branches: + * - fallthrough; + * - implicit subprogram exit. + * It is necessary to switch current frame if + * implicit subprogram exit branch is backtracked. + */ + if (mode == BPF_IND || mode == BPF_ABS) { + if (bt_is_reg_set(bt, dreg)) + return -ENOTSUPP; + if (subseq_idx != idx + 1) + if (bt_subprog_enter(bt)) + return -EFAULT; + return 0; + } } /* Propagate precision marks to linked registers, to account for * registers marked as precise in this function. @@ -5411,7 +5471,8 @@ static int map_kptr_match_type(struct bpf_verifier_env *env, */ if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->off, kptr_field->kptr.btf, kptr_field->kptr.btf_id, - kptr_field->type == BPF_KPTR_REF)) + kptr_field->type == BPF_KPTR_REF, + !type_is_alloc(reg->type))) goto bad_type; return 0; bad_type: @@ -5498,8 +5559,15 @@ static int check_map_kptr_access(struct bpf_verifier_env *env, u32 regno, PTR_MAYBE_NULL | PTR_UNTRUSTED); } else if (class == BPF_STX) { val_reg = reg_state(env, value_regno); - if (!register_is_null(val_reg) && - map_kptr_match_type(env, kptr_field, val_reg, value_regno)) + if (register_is_null(val_reg)) { + /* + * This store is valid only because the scalar is known to be + * zero. Mark it precise so another scalar cannot be pruned + * against this state. + */ + return mark_chain_precision(env, value_regno); + } + if (map_kptr_match_type(env, kptr_field, val_reg, value_regno)) return -EACCES; } else if (class == BPF_ST) { if (insn->imm) { @@ -6513,6 +6581,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, const char *field_name = NULL; enum bpf_type_flag flag = 0; u32 btf_id = 0; + bool mask; int ret; if (!env->allow_ptr_leaks) { @@ -6557,6 +6626,11 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, return -EACCES; } + if (atype != BPF_READ && (type_flag(reg->type) & PTR_UNTRUSTED)) { + verbose(env, "only read is supported\n"); + return -EACCES; + } + if (env->ops->btf_struct_access && !type_is_alloc(reg->type) && atype == BPF_WRITE) { if (!btf_is_kernel(reg->btf)) { verbose(env, "verifier internal error: reg->btf must be kernel btf\n"); @@ -6565,8 +6639,7 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, ret = env->ops->btf_struct_access(&env->log, reg, off, size); } else { /* Writes are permitted with default btf_struct_access for - * program allocated objects (which always have ref_obj_id > 0), - * but not for untrusted PTR_TO_BTF_ID | MEM_ALLOC. + * program allocated objects (which always have ref_obj_id > 0). */ if (atype != BPF_READ && !type_is_ptr_alloc_obj(reg->type)) { verbose(env, "only read is supported\n"); @@ -6584,7 +6657,21 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, if (ret < 0) return ret; - + /* For raw_tp progs, we allow dereference of PTR_MAYBE_NULL + * trusted PTR_TO_BTF_ID, these are the ones that are possibly + * arguments to the raw_tp. Since internal checks in for trusted + * reg in check_ptr_to_btf_access would consider PTR_MAYBE_NULL + * modifier as problematic, mask it out temporarily for the + * check. Don't apply this to pointers with ref_obj_id > 0, as + * those won't be raw_tp args. + * + * We may end up applying this relaxation to other trusted + * PTR_TO_BTF_ID with maybe null flag, since we cannot + * distinguish PTR_MAYBE_NULL tagged for arguments vs normal + * tagging, but that should expand allowed behavior, and not + * cause regression for existing behavior. + */ + mask = mask_raw_tp_reg(env, reg); if (ret != PTR_TO_BTF_ID) { /* just mark; */ @@ -6645,8 +6732,13 @@ static int check_ptr_to_btf_access(struct bpf_verifier_env *env, clear_trusted_flags(&flag); } - if (atype == BPF_READ && value_regno >= 0) + if (atype == BPF_READ && value_regno >= 0) { mark_btf_ld_reg(env, regs, value_regno, ret, reg->btf, btf_id, flag); + /* We've assigned a new type to regno, so don't undo masking. */ + if (regno == value_regno) + mask = false; + } + unmask_raw_tp_reg(reg, mask); return 0; } @@ -6998,7 +7090,7 @@ static int check_mem_access(struct bpf_verifier_env *env, int insn_idx, u32 regn if (!err && t == BPF_READ && value_regno >= 0) mark_reg_unknown(env, regs, value_regno); } else if (base_type(reg->type) == PTR_TO_BTF_ID && - !type_may_be_null(reg->type)) { + (mask_raw_tp_reg_cond(env, reg) || !type_may_be_null(reg->type))) { err = check_ptr_to_btf_access(env, regs, regno, off, size, t, value_regno); } else if (reg->type == CONST_PTR_TO_MAP) { @@ -8374,7 +8466,7 @@ static int check_reg_type(struct bpf_verifier_env *env, u32 regno, if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, reg->off, btf_vmlinux, *arg_btf_id, - strict_type_match)) { + strict_type_match, !type_is_alloc(reg->type))) { verbose(env, "R%d is of type %s but %s is expected\n", regno, btf_type_name(reg->btf, reg->btf_id), btf_type_name(btf_vmlinux, *arg_btf_id)); @@ -8569,6 +8661,7 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, enum bpf_reg_type type = reg->type; u32 *arg_btf_id = NULL; int err = 0; + bool mask; if (arg_type == ARG_DONTCARE) return 0; @@ -8598,22 +8691,26 @@ static int check_func_arg(struct bpf_verifier_env *env, u32 arg, return err; } - if (register_is_null(reg) && type_may_be_null(arg_type)) + if (register_is_null(reg) && type_may_be_null(arg_type)) { /* A NULL register has a SCALAR_VALUE type, so skip * type checking. */ + err = mark_chain_precision(env, regno); + if (err) + return err; goto skip_type_check; + } /* arg_btf_id and arg_size are in a union. */ if (base_type(arg_type) == ARG_PTR_TO_BTF_ID || base_type(arg_type) == ARG_PTR_TO_SPIN_LOCK) arg_btf_id = fn->arg_btf_id[arg]; + mask = mask_raw_tp_reg(env, reg); err = check_reg_type(env, regno, arg_type, arg_btf_id, meta); - if (err) - return err; - err = check_func_arg_reg_off(env, reg, regno, arg_type); + err = err ?: check_func_arg_reg_off(env, reg, regno, arg_type); + unmask_raw_tp_reg(reg, mask); if (err) return err; @@ -9517,10 +9614,12 @@ int map_set_for_each_callback_args(struct bpf_verifier_env *env, callee->regs[BPF_REG_2].type = PTR_TO_MAP_KEY; __mark_reg_known_zero(&callee->regs[BPF_REG_2]); callee->regs[BPF_REG_2].map_ptr = caller->regs[BPF_REG_1].map_ptr; + callee->regs[BPF_REG_2].map_uid = caller->regs[BPF_REG_1].map_uid; callee->regs[BPF_REG_3].type = PTR_TO_MAP_VALUE; __mark_reg_known_zero(&callee->regs[BPF_REG_3]); callee->regs[BPF_REG_3].map_ptr = caller->regs[BPF_REG_1].map_ptr; + callee->regs[BPF_REG_3].map_uid = caller->regs[BPF_REG_1].map_uid; /* pointer to stack or null */ callee->regs[BPF_REG_4] = caller->regs[BPF_REG_3]; @@ -9602,6 +9701,7 @@ static int set_timer_callback_state(struct bpf_verifier_env *env, int insn_idx) { struct bpf_map *map_ptr = caller->regs[BPF_REG_1].map_ptr; + u32 map_uid = caller->regs[BPF_REG_1].map_uid; /* bpf_timer_set_callback(struct bpf_timer *timer, void *callback_fn); * callback_fn(struct bpf_map *map, void *key, void *value); @@ -9609,14 +9709,17 @@ static int set_timer_callback_state(struct bpf_verifier_env *env, callee->regs[BPF_REG_1].type = CONST_PTR_TO_MAP; __mark_reg_known_zero(&callee->regs[BPF_REG_1]); callee->regs[BPF_REG_1].map_ptr = map_ptr; + callee->regs[BPF_REG_1].map_uid = map_uid; callee->regs[BPF_REG_2].type = PTR_TO_MAP_KEY; __mark_reg_known_zero(&callee->regs[BPF_REG_2]); callee->regs[BPF_REG_2].map_ptr = map_ptr; + callee->regs[BPF_REG_2].map_uid = map_uid; callee->regs[BPF_REG_3].type = PTR_TO_MAP_VALUE; __mark_reg_known_zero(&callee->regs[BPF_REG_3]); callee->regs[BPF_REG_3].map_ptr = map_ptr; + callee->regs[BPF_REG_3].map_uid = map_uid; /* unused */ __mark_reg_not_init(env, &callee->regs[BPF_REG_4]); @@ -10068,6 +10171,19 @@ static void update_loop_inline_state(struct bpf_verifier_env *env, u32 subprogno state->callback_subprogno == subprogno); } +static int get_helper_proto(struct bpf_verifier_env *env, int func_id, + const struct bpf_func_proto **ptr) +{ + if (func_id < 0 || func_id >= __BPF_FUNC_MAX_ID) + return -ERANGE; + + if (!env->ops->get_func_proto) + return -EINVAL; + + *ptr = env->ops->get_func_proto(func_id, env->prog); + return *ptr && (*ptr)->func ? 0 : -EINVAL; +} + static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn, int *insn_idx_p) { @@ -10083,18 +10199,16 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn /* find function prototype */ func_id = insn->imm; - if (func_id < 0 || func_id >= __BPF_FUNC_MAX_ID) { - verbose(env, "invalid func %s#%d\n", func_id_name(func_id), - func_id); + err = get_helper_proto(env, insn->imm, &fn); + if (err == -ERANGE) { + verbose(env, "invalid func %s#%d\n", func_id_name(func_id), func_id); return -EINVAL; } - if (env->ops->get_func_proto) - fn = env->ops->get_func_proto(func_id, env->prog); - if (!fn) { - verbose(env, "unknown func %s#%d\n", func_id_name(func_id), - func_id); - return -EINVAL; + if (err) { + verbose(env, "program of this type cannot use helper %s#%d\n", + func_id_name(func_id), func_id); + return err; } /* eBPF programs must be GPL compatible to use GPL-ed functions */ @@ -10213,6 +10327,9 @@ static int check_helper_call(struct bpf_verifier_env *env, struct bpf_insn *insn verbose(env, "get_local_storage() doesn't support non-zero flags\n"); return -EINVAL; } + err = mark_chain_precision(env, BPF_REG_2); + if (err) + return err; break; case BPF_FUNC_for_each_map_elem: err = push_callback_call(env, insn, insn_idx, meta.subprogno, @@ -11017,7 +11134,8 @@ static int process_kf_arg_ptr_to_btf_id(struct bpf_verifier_env *env, reg_ref_t = btf_type_skip_modifiers(reg_btf, reg_ref_id, ®_ref_id); reg_ref_tname = btf_name_by_offset(reg_btf, reg_ref_t->name_off); - if (!btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match)) { + if (!btf_struct_ids_match(&env->log, reg_btf, reg_ref_id, reg->off, meta->btf, ref_id, strict_type_match, + !type_is_alloc(reg->type))) { verbose(env, "kernel function %s args#%d expected pointer to %s %s but R%d has a pointer to %s %s\n", meta->func_name, argno, btf_type_str(ref_t), ref_tname, argno + 1, btf_type_str(reg_ref_t), reg_ref_tname); @@ -11345,7 +11463,8 @@ __process_kf_arg_ptr_to_graph_node(struct bpf_verifier_env *env, et = btf_type_by_id(field->graph_root.btf, field->graph_root.value_btf_id); t = btf_type_by_id(reg->btf, reg->btf_id); if (!btf_struct_ids_match(&env->log, reg->btf, reg->btf_id, 0, field->graph_root.btf, - field->graph_root.value_btf_id, true)) { + field->graph_root.value_btf_id, true, + !type_is_alloc(reg->type))) { verbose(env, "operation on %s expects arg#1 %s at offset=%d " "in struct %s, but arg is at offset=%d in struct %s\n", btf_field_type_name(head_field_type), @@ -11414,6 +11533,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ enum bpf_arg_type arg_type = ARG_DONTCARE; u32 regno = i + 1, ref_id, type_size; bool is_ret_buf_sz = false; + bool mask = false; int kf_arg_type; t = btf_type_skip_modifiers(btf, args[i].type, NULL); @@ -11460,6 +11580,10 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ } meta->r0_size = reg->var_off.value; + if (meta->r0_size > U32_MAX) { + verbose(env, "R%d rdonly/rdwr_buf_size exceeds u32 max\n", regno); + return -EINVAL; + } ret = mark_chain_precision(env, regno); if (ret) return ret; @@ -11472,11 +11596,14 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ return -EINVAL; } + mask = mask_raw_tp_reg(env, reg); if ((is_kfunc_trusted_args(meta) || is_kfunc_rcu(meta)) && (register_is_null(reg) || type_may_be_null(reg->type))) { verbose(env, "Possibly NULL pointer passed to trusted arg%d\n", i); + unmask_raw_tp_reg(reg, mask); return -EACCES; } + unmask_raw_tp_reg(reg, mask); if (reg->ref_obj_id) { if (is_kfunc_release(meta) && meta->ref_obj_id) { @@ -11503,17 +11630,24 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ if (!is_kfunc_trusted_args(meta) && !is_kfunc_rcu(meta)) break; + /* Allow passing maybe NULL raw_tp arguments to + * kfuncs for compatibility. Don't apply this to + * arguments with ref_obj_id > 0. + */ + mask = mask_raw_tp_reg(env, reg); if (!is_trusted_reg(reg)) { if (!is_kfunc_rcu(meta)) { verbose(env, "R%d must be referenced or trusted\n", regno); + unmask_raw_tp_reg(reg, mask); return -EINVAL; } if (!is_rcu_reg(reg)) { verbose(env, "R%d must be a rcu pointer\n", regno); + unmask_raw_tp_reg(reg, mask); return -EINVAL; } } - + unmask_raw_tp_reg(reg, mask); fallthrough; case KF_ARG_PTR_TO_CTX: /* Trusted arguments have the same offset checks as release arguments */ @@ -11538,7 +11672,9 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ if (is_kfunc_release(meta) && reg->ref_obj_id) arg_type |= OBJ_RELEASE; + mask = mask_raw_tp_reg(env, reg); ret = check_func_arg_reg_off(env, reg, regno, arg_type); + unmask_raw_tp_reg(reg, mask); if (ret < 0) return ret; @@ -11699,6 +11835,7 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ return ret; break; case KF_ARG_PTR_TO_BTF_ID: + mask = mask_raw_tp_reg(env, reg); /* Only base_type is checked, further checks are done here */ if ((base_type(reg->type) != PTR_TO_BTF_ID || (bpf_type_has_unsafe_modifiers(reg->type) && !is_rcu_reg(reg))) && @@ -11707,9 +11844,11 @@ static int check_kfunc_args(struct bpf_verifier_env *env, struct bpf_kfunc_call_ verbose(env, "expected %s or socket\n", reg_type_str(env, base_type(reg->type) | (type_flag(reg->type) & BPF_REG_TRUSTED_MODIFIERS))); + unmask_raw_tp_reg(reg, mask); return -EINVAL; } ret = process_kf_arg_ptr_to_btf_id(env, reg, ref_t, ref_tname, ref_id, meta, i); + unmask_raw_tp_reg(reg, mask); if (ret < 0) return ret; break; @@ -12596,7 +12735,7 @@ static int sanitize_check_bounds(struct bpf_verifier_env *env, */ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_insn *insn, - const struct bpf_reg_state *ptr_reg, + struct bpf_reg_state *ptr_reg, const struct bpf_reg_state *off_reg) { struct bpf_verifier_state *vstate = env->cur_state; @@ -12610,6 +12749,7 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, struct bpf_sanitize_info info = {}; u8 opcode = BPF_OP(insn->code); u32 dst = insn->dst_reg; + bool mask; int ret; dst_reg = ®s[dst]; @@ -12636,11 +12776,14 @@ static int adjust_ptr_min_max_vals(struct bpf_verifier_env *env, return -EACCES; } + mask = mask_raw_tp_reg(env, ptr_reg); if (ptr_reg->type & PTR_MAYBE_NULL) { verbose(env, "R%d pointer arithmetic on %s prohibited, null-check it first\n", dst, reg_type_str(env, ptr_reg->type)); + unmask_raw_tp_reg(ptr_reg, mask); return -EACCES; } + unmask_raw_tp_reg(ptr_reg, mask); switch (base_type(ptr_reg->type)) { case PTR_TO_FLOW_KEYS: @@ -13952,153 +14095,119 @@ static void find_good_pkt_pointers(struct bpf_verifier_state *vstate, })); } -static int is_branch32_taken(struct bpf_reg_state *reg, u32 val, u8 opcode) +/* check if register is a constant scalar value */ +static bool is_reg_const(struct bpf_reg_state *reg, bool subreg32) { - struct tnum subreg = tnum_subreg(reg->var_off); - s32 sval = (s32)val; - - switch (opcode) { - case BPF_JEQ: - if (tnum_is_const(subreg)) - return !!tnum_equals_const(subreg, val); - else if (val < reg->u32_min_value || val > reg->u32_max_value) - return 0; - break; - case BPF_JNE: - if (tnum_is_const(subreg)) - return !tnum_equals_const(subreg, val); - else if (val < reg->u32_min_value || val > reg->u32_max_value) - return 1; - break; - case BPF_JSET: - if ((~subreg.mask & subreg.value) & val) - return 1; - if (!((subreg.mask | subreg.value) & val)) - return 0; - break; - case BPF_JGT: - if (reg->u32_min_value > val) - return 1; - else if (reg->u32_max_value <= val) - return 0; - break; - case BPF_JSGT: - if (reg->s32_min_value > sval) - return 1; - else if (reg->s32_max_value <= sval) - return 0; - break; - case BPF_JLT: - if (reg->u32_max_value < val) - return 1; - else if (reg->u32_min_value >= val) - return 0; - break; - case BPF_JSLT: - if (reg->s32_max_value < sval) - return 1; - else if (reg->s32_min_value >= sval) - return 0; - break; - case BPF_JGE: - if (reg->u32_min_value >= val) - return 1; - else if (reg->u32_max_value < val) - return 0; - break; - case BPF_JSGE: - if (reg->s32_min_value >= sval) - return 1; - else if (reg->s32_max_value < sval) - return 0; - break; - case BPF_JLE: - if (reg->u32_max_value <= val) - return 1; - else if (reg->u32_min_value > val) - return 0; - break; - case BPF_JSLE: - if (reg->s32_max_value <= sval) - return 1; - else if (reg->s32_min_value > sval) - return 0; - break; - } - - return -1; + return reg->type == SCALAR_VALUE && + tnum_is_const(subreg32 ? tnum_subreg(reg->var_off) : reg->var_off); } - -static int is_branch64_taken(struct bpf_reg_state *reg, u64 val, u8 opcode) +/* assuming is_reg_const() is true, return constant value of a register */ +static u64 reg_const_value(struct bpf_reg_state *reg, bool subreg32) { - s64 sval = (s64)val; + return subreg32 ? tnum_subreg(reg->var_off).value : reg->var_off.value; +} + +/* + * , currently assuming reg2 is a constant + */ +static int is_scalar_branch_taken(struct bpf_reg_state *reg1, struct bpf_reg_state *reg2, + u8 opcode, bool is_jmp32) +{ + struct tnum t1 = is_jmp32 ? tnum_subreg(reg1->var_off) : reg1->var_off; + struct tnum t2 = is_jmp32 ? tnum_subreg(reg2->var_off) : reg2->var_off; + u64 umin1 = is_jmp32 ? (u64)reg1->u32_min_value : reg1->umin_value; + u64 umax1 = is_jmp32 ? (u64)reg1->u32_max_value : reg1->umax_value; + s64 smin1 = is_jmp32 ? (s64)reg1->s32_min_value : reg1->smin_value; + s64 smax1 = is_jmp32 ? (s64)reg1->s32_max_value : reg1->smax_value; + u64 umin2 = is_jmp32 ? (u64)reg2->u32_min_value : reg2->umin_value; + u64 umax2 = is_jmp32 ? (u64)reg2->u32_max_value : reg2->umax_value; + s64 smin2 = is_jmp32 ? (s64)reg2->s32_min_value : reg2->smin_value; + s64 smax2 = is_jmp32 ? (s64)reg2->s32_max_value : reg2->smax_value; switch (opcode) { case BPF_JEQ: - if (tnum_is_const(reg->var_off)) - return !!tnum_equals_const(reg->var_off, val); - else if (val < reg->umin_value || val > reg->umax_value) + /* constants, umin/umax and smin/smax checks would be + * redundant in this case because they all should match + */ + if (tnum_is_const(t1) && tnum_is_const(t2)) + return t1.value == t2.value; + /* non-overlapping ranges */ + if (umin1 > umax2 || umax1 < umin2) + return 0; + if (smin1 > smax2 || smax1 < smin2) return 0; break; case BPF_JNE: - if (tnum_is_const(reg->var_off)) - return !tnum_equals_const(reg->var_off, val); - else if (val < reg->umin_value || val > reg->umax_value) + /* constants, umin/umax and smin/smax checks would be + * redundant in this case because they all should match + */ + if (tnum_is_const(t1) && tnum_is_const(t2)) + return t1.value != t2.value; + /* non-overlapping ranges */ + if (umin1 > umax2 || umax1 < umin2) + return 1; + if (smin1 > smax2 || smax1 < smin2) return 1; break; case BPF_JSET: - if ((~reg->var_off.mask & reg->var_off.value) & val) + if (!is_reg_const(reg2, is_jmp32)) { + swap(reg1, reg2); + swap(t1, t2); + } + if (!is_reg_const(reg2, is_jmp32)) + return -1; + if ((~t1.mask & t1.value) & t2.value) return 1; - if (!((reg->var_off.mask | reg->var_off.value) & val)) + if (!((t1.mask | t1.value) & t2.value)) return 0; break; case BPF_JGT: - if (reg->umin_value > val) + if (umin1 > umax2) return 1; - else if (reg->umax_value <= val) + else if (umax1 <= umin2) return 0; break; case BPF_JSGT: - if (reg->smin_value > sval) + if (smin1 > smax2) return 1; - else if (reg->smax_value <= sval) + else if (smax1 <= smin2) return 0; break; case BPF_JLT: - if (reg->umax_value < val) + if (umax1 < umin2) return 1; - else if (reg->umin_value >= val) + else if (umin1 >= umax2) return 0; break; case BPF_JSLT: - if (reg->smax_value < sval) + if (smax1 < smin2) return 1; - else if (reg->smin_value >= sval) + else if (smin1 >= smax2) return 0; break; case BPF_JGE: - if (reg->umin_value >= val) + if (umin1 >= umax2) return 1; - else if (reg->umax_value < val) + else if (umax1 < umin2) return 0; break; case BPF_JSGE: - if (reg->smin_value >= sval) + if (smin1 >= smax2) return 1; - else if (reg->smax_value < sval) + else if (smax1 < smin2) return 0; break; case BPF_JLE: - if (reg->umax_value <= val) + if (umax1 <= umin2) return 1; - else if (reg->umin_value > val) + else if (umin1 > umax2) return 0; break; case BPF_JSLE: - if (reg->smax_value <= sval) + if (smax1 <= smin2) return 1; - else if (reg->smin_value > sval) + else if (smin1 > smax2) return 0; break; } @@ -14106,41 +14215,6 @@ static int is_branch64_taken(struct bpf_reg_state *reg, u64 val, u8 opcode) return -1; } -/* compute branch direction of the expression "if (reg opcode val) goto target;" - * and return: - * 1 - branch will be taken and "goto target" will be executed - * 0 - branch will not be taken and fall-through to next insn - * -1 - unknown. Example: "if (reg < 5)" is unknown when register value - * range [0,10] - */ -static int is_branch_taken(struct bpf_reg_state *reg, u64 val, u8 opcode, - bool is_jmp32) -{ - if (__is_pointer_value(false, reg)) { - if (!reg_not_null(reg)) - return -1; - - /* If pointer is valid tests against zero will fail so we can - * use this to direct branch taken. - */ - if (val != 0) - return -1; - - switch (opcode) { - case BPF_JEQ: - return 0; - case BPF_JNE: - return 1; - default: - return -1; - } - } - - if (is_jmp32) - return is_branch32_taken(reg, val, opcode); - return is_branch64_taken(reg, val, opcode); -} - static int flip_opcode(u32 opcode) { /* How can we transform "a b" into "b a"? */ @@ -14202,6 +14276,62 @@ static int is_pkt_ptr_branch_taken(struct bpf_reg_state *dst_reg, return -1; } +/* compute branch direction of the expression "if ( opcode ) goto target;" + * and return: + * 1 - branch will be taken and "goto target" will be executed + * 0 - branch will not be taken and fall-through to next insn + * -1 - unknown. Example: "if (reg1 < 5)" is unknown when register value + * range [0,10] + */ +static int is_branch_taken(struct bpf_reg_state *reg1, struct bpf_reg_state *reg2, + u8 opcode, bool is_jmp32) +{ + if (reg_is_pkt_pointer_any(reg1) && reg_is_pkt_pointer_any(reg2) && !is_jmp32) + return is_pkt_ptr_branch_taken(reg1, reg2, opcode); + + if (__is_pointer_value(false, reg1) || __is_pointer_value(false, reg2)) { + u64 val; + + /* + * The low 32 bits of a valid pointer may well be zero, hence + * nothing below applies to a 32-bit comparison. + */ + if (is_jmp32) + return -1; + + /* arrange that reg2 is a scalar, and reg1 is a pointer */ + if (!is_reg_const(reg2, is_jmp32)) { + opcode = flip_opcode(opcode); + swap(reg1, reg2); + } + /* and ensure that reg2 is a constant */ + if (!is_reg_const(reg2, is_jmp32)) + return -1; + + if (!reg_not_null(reg1)) + return -1; + + /* If pointer is valid tests against zero will fail so we can + * use this to direct branch taken. + */ + val = reg_const_value(reg2, is_jmp32); + if (val != 0) + return -1; + + switch (opcode) { + case BPF_JEQ: + return 0; + case BPF_JNE: + return 1; + default: + return -1; + } + } + + /* now deal with two scalars, but not necessarily constants */ + return is_scalar_branch_taken(reg1, reg2, opcode, is_jmp32); +} + /* Adjusts the register min/max values in the case that the dst_reg is the * variable register that we are working on, and src_reg is a constant or we're * simply doing a BPF_K check. @@ -14209,15 +14339,15 @@ static int is_pkt_ptr_branch_taken(struct bpf_reg_state *dst_reg, */ static void reg_set_min_max(struct bpf_reg_state *true_reg, struct bpf_reg_state *false_reg, - u64 val, u32 val32, + u64 uval, u32 uval32, u8 opcode, bool is_jmp32) { struct tnum false_32off = tnum_subreg(false_reg->var_off); struct tnum false_64off = false_reg->var_off; struct tnum true_32off = tnum_subreg(true_reg->var_off); struct tnum true_64off = true_reg->var_off; - s64 sval = (s64)val; - s32 sval32 = (s32)val32; + s64 sval = (s64)uval; + s32 sval32 = (s32)uval32; /* If the dst_reg is a pointer, we can't learn anything about its * variable offset from the compare (unless src_reg were a pointer into @@ -14240,19 +14370,19 @@ static void reg_set_min_max(struct bpf_reg_state *true_reg, */ case BPF_JEQ: if (is_jmp32) { - __mark_reg32_known(true_reg, val32); + __mark_reg32_known(true_reg, uval32); true_32off = tnum_subreg(true_reg->var_off); } else { - ___mark_reg_known(true_reg, val); + ___mark_reg_known(true_reg, uval); true_64off = true_reg->var_off; } break; case BPF_JNE: if (is_jmp32) { - __mark_reg32_known(false_reg, val32); + __mark_reg32_known(false_reg, uval32); false_32off = tnum_subreg(false_reg->var_off); } else { - ___mark_reg_known(false_reg, val); + ___mark_reg_known(false_reg, uval); false_64off = false_reg->var_off; } break; @@ -14262,31 +14392,31 @@ static void reg_set_min_max(struct bpf_reg_state *true_reg, */ __mark_reg_unbounded(false_reg); if (is_jmp32) { - false_32off = tnum_and(false_32off, tnum_const(~val32)); - if (is_power_of_2(val32)) + false_32off = tnum_and(false_32off, tnum_const(~uval32)); + if (is_power_of_2(uval32)) true_32off = tnum_or(true_32off, - tnum_const(val32)); + tnum_const(uval32)); } else { - false_64off = tnum_and(false_64off, tnum_const(~val)); - if (is_power_of_2(val)) + false_64off = tnum_and(false_64off, tnum_const(~uval)); + if (is_power_of_2(uval)) true_64off = tnum_or(true_64off, - tnum_const(val)); + tnum_const(uval)); } break; case BPF_JGE: case BPF_JGT: { if (is_jmp32) { - u32 false_umax = opcode == BPF_JGT ? val32 : val32 - 1; - u32 true_umin = opcode == BPF_JGT ? val32 + 1 : val32; + u32 false_umax = opcode == BPF_JGT ? uval32 : uval32 - 1; + u32 true_umin = opcode == BPF_JGT ? uval32 + 1 : uval32; false_reg->u32_max_value = min(false_reg->u32_max_value, false_umax); true_reg->u32_min_value = max(true_reg->u32_min_value, true_umin); } else { - u64 false_umax = opcode == BPF_JGT ? val : val - 1; - u64 true_umin = opcode == BPF_JGT ? val + 1 : val; + u64 false_umax = opcode == BPF_JGT ? uval : uval - 1; + u64 true_umin = opcode == BPF_JGT ? uval + 1 : uval; false_reg->umax_value = min(false_reg->umax_value, false_umax); true_reg->umin_value = max(true_reg->umin_value, true_umin); @@ -14315,16 +14445,16 @@ static void reg_set_min_max(struct bpf_reg_state *true_reg, case BPF_JLT: { if (is_jmp32) { - u32 false_umin = opcode == BPF_JLT ? val32 : val32 + 1; - u32 true_umax = opcode == BPF_JLT ? val32 - 1 : val32; + u32 false_umin = opcode == BPF_JLT ? uval32 : uval32 + 1; + u32 true_umax = opcode == BPF_JLT ? uval32 - 1 : uval32; false_reg->u32_min_value = max(false_reg->u32_min_value, false_umin); true_reg->u32_max_value = min(true_reg->u32_max_value, true_umax); } else { - u64 false_umin = opcode == BPF_JLT ? val : val + 1; - u64 true_umax = opcode == BPF_JLT ? val - 1 : val; + u64 false_umin = opcode == BPF_JLT ? uval : uval + 1; + u64 true_umax = opcode == BPF_JLT ? uval - 1 : uval; false_reg->umin_value = max(false_reg->umin_value, false_umin); true_reg->umax_value = min(true_reg->umax_value, true_umax); @@ -14373,7 +14503,7 @@ static void reg_set_min_max(struct bpf_reg_state *true_reg, */ static void reg_set_min_max_inv(struct bpf_reg_state *true_reg, struct bpf_reg_state *false_reg, - u64 val, u32 val32, + u64 uval, u32 uval32, u8 opcode, bool is_jmp32) { opcode = flip_opcode(opcode); @@ -14381,7 +14511,7 @@ static void reg_set_min_max_inv(struct bpf_reg_state *true_reg, * BPF_JA, can't get here. */ if (opcode) - reg_set_min_max(true_reg, false_reg, val, val32, opcode, is_jmp32); + reg_set_min_max(true_reg, false_reg, uval, uval32, opcode, is_jmp32); } /* Regs are known to be equal, so intersect their min/max/var_off */ @@ -14673,7 +14803,9 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, struct bpf_reg_state *dst_reg, *other_branch_regs, *src_reg = NULL; struct bpf_reg_state *eq_branch_regs; struct linked_regs linked_regs = {}; + struct bpf_reg_state fake_reg = {}; u8 opcode = BPF_OP(insn->code); + int insn_flags = 0; bool is_jmp32; int pred = -1; int err; @@ -14708,47 +14840,29 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, insn->src_reg); return -EACCES; } + + if (src_reg->type == PTR_TO_STACK) + insn_flags |= INSN_F_SRC_REG_STACK; } else { if (insn->src_reg != BPF_REG_0) { verbose(env, "BPF_JMP/JMP32 uses reserved fields\n"); return -EINVAL; } + src_reg = &fake_reg; + src_reg->type = SCALAR_VALUE; + __mark_reg_known(src_reg, insn->imm); } - is_jmp32 = BPF_CLASS(insn->code) == BPF_JMP32; - - if (BPF_SRC(insn->code) == BPF_K) { - pred = is_branch_taken(dst_reg, insn->imm, opcode, is_jmp32); - } else if (src_reg->type == SCALAR_VALUE && - is_jmp32 && tnum_is_const(tnum_subreg(src_reg->var_off))) { - pred = is_branch_taken(dst_reg, - tnum_subreg(src_reg->var_off).value, - opcode, - is_jmp32); - } else if (src_reg->type == SCALAR_VALUE && - !is_jmp32 && tnum_is_const(src_reg->var_off)) { - pred = is_branch_taken(dst_reg, - src_reg->var_off.value, - opcode, - is_jmp32); - } else if (dst_reg->type == SCALAR_VALUE && - is_jmp32 && tnum_is_const(tnum_subreg(dst_reg->var_off))) { - pred = is_branch_taken(src_reg, - tnum_subreg(dst_reg->var_off).value, - flip_opcode(opcode), - is_jmp32); - } else if (dst_reg->type == SCALAR_VALUE && - !is_jmp32 && tnum_is_const(dst_reg->var_off)) { - pred = is_branch_taken(src_reg, - dst_reg->var_off.value, - flip_opcode(opcode), - is_jmp32); - } else if (reg_is_pkt_pointer_any(dst_reg) && - reg_is_pkt_pointer_any(src_reg) && - !is_jmp32) { - pred = is_pkt_ptr_branch_taken(dst_reg, src_reg, opcode); + if (dst_reg->type == PTR_TO_STACK) + insn_flags |= INSN_F_DST_REG_STACK; + if (insn_flags) { + err = push_jmp_history(env, this_branch, insn_flags, 0); + if (err) + return err; } + is_jmp32 = BPF_CLASS(insn->code) == BPF_JMP32; + pred = is_branch_taken(dst_reg, src_reg, opcode, is_jmp32); if (pred >= 0) { /* If we get here with a dst_reg pointer type it is because * above is_branch_taken() special cased the 0 comparison. @@ -14880,7 +14994,6 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, */ if (!is_jmp32 && BPF_SRC(insn->code) == BPF_X && __is_pointer_value(false, src_reg) && __is_pointer_value(false, dst_reg) && - type_may_be_null(src_reg->type) != type_may_be_null(dst_reg->type) && base_type(src_reg->type) != PTR_TO_BTF_ID && base_type(dst_reg->type) != PTR_TO_BTF_ID) { eq_branch_regs = NULL; @@ -14896,9 +15009,11 @@ static int check_cond_jmp_op(struct bpf_verifier_env *env, break; } if (eq_branch_regs) { - if (type_may_be_null(src_reg->type)) + /* src == dst && dst != NULL => src != NULL */ + if (reg_not_null(dst_reg) && type_may_be_null(src_reg->type)) mark_ptr_not_null_reg(&eq_branch_regs[insn->src_reg]); - else + /* src == dst && src != NULL => dst != NULL */ + if (reg_not_null(src_reg) && type_may_be_null(dst_reg->type)) mark_ptr_not_null_reg(&eq_branch_regs[insn->dst_reg]); } } @@ -17561,6 +17676,10 @@ static int do_check(struct bpf_verifier_env *env) return err; break; } else { + if (WARN_ON_ONCE(env->cur_state->loop_entry)) { + verbose(env, "verifier bug: env->cur_state->loop_entry != NULL\n"); + return -EFAULT; + } do_print_state = true; continue; } @@ -18637,6 +18756,7 @@ static int convert_ctx_accesses(struct bpf_verifier_env *env) * for this case. */ case PTR_TO_BTF_ID | MEM_ALLOC | PTR_UNTRUSTED: + case PTR_TO_BTF_ID | PTR_TRUSTED | PTR_MAYBE_NULL: if (type == BPF_READ) { if (BPF_MODE(insn->code) == BPF_MEM) insn->code = BPF_LDX | BPF_PROBE_MEM | @@ -18900,10 +19020,14 @@ static int jit_subprogs(struct bpf_verifier_env *env) * bpf_prog_load will add the kallsyms for the main program. */ for (i = 1; i < env->subprog_cnt; i++) { - bpf_prog_lock_ro(func[i]); - bpf_prog_kallsyms_add(func[i]); + err = bpf_prog_lock_ro(func[i]); + if (err) + goto out_free; } + for (i = 1; i < env->subprog_cnt; i++) + bpf_prog_kallsyms_add(func[i]); + /* Last step: make now unused interpreter insns from main * prog consistent for later dump requests, so they can * later look the same as if they were interpreted only. @@ -19169,13 +19293,46 @@ static int do_misc_fixups(struct bpf_verifier_env *env) int i, ret, cnt, delta = 0; for (i = 0; i < insn_cnt; i++, insn++) { - /* Make divide-by-zero exceptions impossible. */ + /* Make sdiv/smod divide-by-minus-one exceptions impossible. */ + if ((insn->code == (BPF_ALU64 | BPF_MOD | BPF_K) || + insn->code == (BPF_ALU64 | BPF_DIV | BPF_K) || + insn->code == (BPF_ALU | BPF_MOD | BPF_K) || + insn->code == (BPF_ALU | BPF_DIV | BPF_K)) && + insn->off == 1 && insn->imm == -1) { + bool is64 = BPF_CLASS(insn->code) == BPF_ALU64; + bool isdiv = BPF_OP(insn->code) == BPF_DIV; + struct bpf_insn *patchlet; + struct bpf_insn chk_and_sdiv[] = { + BPF_RAW_INSN((is64 ? BPF_ALU64 : BPF_ALU) | + BPF_NEG | BPF_K, insn->dst_reg, + 0, 0, 0), + }; + struct bpf_insn chk_and_smod[] = { + BPF_MOV32_IMM(insn->dst_reg, 0), + }; + + patchlet = isdiv ? chk_and_sdiv : chk_and_smod; + cnt = isdiv ? ARRAY_SIZE(chk_and_sdiv) : ARRAY_SIZE(chk_and_smod); + + new_prog = bpf_patch_insn_data(env, i + delta, patchlet, cnt); + if (!new_prog) + return -ENOMEM; + + delta += cnt - 1; + env->prog = prog = new_prog; + insn = new_prog->insnsi + i + delta; + continue; + } + + /* Make divide-by-zero and divide-by-minus-one exceptions impossible. */ if (insn->code == (BPF_ALU64 | BPF_MOD | BPF_X) || insn->code == (BPF_ALU64 | BPF_DIV | BPF_X) || insn->code == (BPF_ALU | BPF_MOD | BPF_X) || insn->code == (BPF_ALU | BPF_DIV | BPF_X)) { bool is64 = BPF_CLASS(insn->code) == BPF_ALU64; bool isdiv = BPF_OP(insn->code) == BPF_DIV; + bool is_sdiv = isdiv && insn->off == 1; + bool is_smod = !isdiv && insn->off == 1; struct bpf_insn *patchlet; struct bpf_insn chk_and_div[] = { /* [R,W]x div 0 -> 0 */ @@ -19195,10 +19352,62 @@ static int do_misc_fixups(struct bpf_verifier_env *env) BPF_JMP_IMM(BPF_JA, 0, 0, 1), BPF_MOV32_REG(insn->dst_reg, insn->dst_reg), }; + struct bpf_insn chk_and_sdiv[] = { + /* [R,W]x sdiv 0 -> 0 + * LLONG_MIN sdiv -1 -> LLONG_MIN + * INT_MIN sdiv -1 -> INT_MIN + */ + BPF_MOV64_REG(BPF_REG_AX, insn->src_reg), + BPF_RAW_INSN((is64 ? BPF_ALU64 : BPF_ALU) | + BPF_ADD | BPF_K, BPF_REG_AX, + 0, 0, 1), + BPF_RAW_INSN((is64 ? BPF_JMP : BPF_JMP32) | + BPF_JGT | BPF_K, BPF_REG_AX, + 0, 4, 1), + BPF_RAW_INSN((is64 ? BPF_JMP : BPF_JMP32) | + BPF_JEQ | BPF_K, BPF_REG_AX, + 0, 1, 0), + BPF_RAW_INSN((is64 ? BPF_ALU64 : BPF_ALU) | + BPF_MOV | BPF_K, insn->dst_reg, + 0, 0, 0), + /* BPF_NEG(LLONG_MIN) == -LLONG_MIN == LLONG_MIN */ + BPF_RAW_INSN((is64 ? BPF_ALU64 : BPF_ALU) | + BPF_NEG | BPF_K, insn->dst_reg, + 0, 0, 0), + BPF_JMP_IMM(BPF_JA, 0, 0, 1), + *insn, + }; + struct bpf_insn chk_and_smod[] = { + /* [R,W]x mod 0 -> [R,W]x */ + /* [R,W]x mod -1 -> 0 */ + BPF_MOV64_REG(BPF_REG_AX, insn->src_reg), + BPF_RAW_INSN((is64 ? BPF_ALU64 : BPF_ALU) | + BPF_ADD | BPF_K, BPF_REG_AX, + 0, 0, 1), + BPF_RAW_INSN((is64 ? BPF_JMP : BPF_JMP32) | + BPF_JGT | BPF_K, BPF_REG_AX, + 0, 3, 1), + BPF_RAW_INSN((is64 ? BPF_JMP : BPF_JMP32) | + BPF_JEQ | BPF_K, BPF_REG_AX, + 0, 3 + (is64 ? 0 : 1), 1), + BPF_MOV32_IMM(insn->dst_reg, 0), + BPF_JMP_IMM(BPF_JA, 0, 0, 1), + *insn, + BPF_JMP_IMM(BPF_JA, 0, 0, 1), + BPF_MOV32_REG(insn->dst_reg, insn->dst_reg), + }; - patchlet = isdiv ? chk_and_div : chk_and_mod; - cnt = isdiv ? ARRAY_SIZE(chk_and_div) : - ARRAY_SIZE(chk_and_mod) - (is64 ? 2 : 0); + if (is_sdiv) { + patchlet = chk_and_sdiv; + cnt = ARRAY_SIZE(chk_and_sdiv); + } else if (is_smod) { + patchlet = chk_and_smod; + cnt = ARRAY_SIZE(chk_and_smod) - (is64 ? 2 : 0); + } else { + patchlet = isdiv ? chk_and_div : chk_and_mod; + cnt = isdiv ? ARRAY_SIZE(chk_and_div) : + ARRAY_SIZE(chk_and_mod) - (is64 ? 2 : 0); + } new_prog = bpf_patch_insn_data(env, i + delta, patchlet, cnt); if (!new_prog) @@ -20426,8 +20635,10 @@ static bool can_be_sleepable(struct bpf_prog *prog) return false; } } - return prog->type == BPF_PROG_TYPE_LSM || - prog->type == BPF_PROG_TYPE_KPROBE /* only for uprobes */ || + if (prog->type == BPF_PROG_TYPE_LSM) + return prog->expected_attach_type != BPF_LSM_CGROUP; + + return prog->type == BPF_PROG_TYPE_KPROBE /* only for uprobes */ || prog->type == BPF_PROG_TYPE_STRUCT_OPS; } @@ -20514,7 +20725,7 @@ static int check_attach_btf_id(struct bpf_verifier_env *env) return -ENOMEM; if (tgt_prog && tgt_prog->aux->tail_call_reachable) - tr->flags = BPF_TRAMP_F_TAIL_CALL_CTX; + bpf_trampoline_set_flags(tr, BPF_TRAMP_F_TAIL_CALL_CTX); prog->aux->dst_trampoline = tr; return 0; diff --git a/kernel/cgroup/Makefile b/kernel/cgroup/Makefile index 12f8457ad1f9..005ac4c675cb 100644 --- a/kernel/cgroup/Makefile +++ b/kernel/cgroup/Makefile @@ -4,6 +4,6 @@ obj-y := cgroup.o rstat.o namespace.o cgroup-v1.o freezer.o obj-$(CONFIG_CGROUP_FREEZER) += legacy_freezer.o obj-$(CONFIG_CGROUP_PIDS) += pids.o obj-$(CONFIG_CGROUP_RDMA) += rdma.o -obj-$(CONFIG_CPUSETS) += cpuset.o +obj-$(CONFIG_CPUSETS) += cpuset.o cpuset-v1.o obj-$(CONFIG_CGROUP_MISC) += misc.o obj-$(CONFIG_CGROUP_DEBUG) += debug.o diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index e787dc0eacd7..97812c0b04dd 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -3775,33 +3775,41 @@ static int cgroup_cpu_pressure_show(struct seq_file *seq, void *v) static ssize_t pressure_write(struct kernfs_open_file *of, char *buf, size_t nbytes, enum psi_res res) { - struct cgroup_file_ctx *ctx = of->priv; + struct cgroup_file_ctx *ctx; struct psi_trigger *new; struct cgroup *cgrp; struct psi_group *psi; + ssize_t ret = 0; cgrp = cgroup_kn_lock_live(of->kn, false); if (!cgrp) return -ENODEV; - cgroup_get(cgrp); - cgroup_kn_unlock(of->kn); + ctx = of->priv; + if (!ctx) { + ret = -ENODEV; + goto out_unlock; + } /* Allow only one trigger per file descriptor */ if (ctx->psi.trigger) { - cgroup_put(cgrp); - return -EBUSY; + ret = -EBUSY; + goto out_unlock; } psi = cgroup_psi(cgrp); new = psi_trigger_create(psi, buf, res, of->file, of); if (IS_ERR(new)) { - cgroup_put(cgrp); - return PTR_ERR(new); + ret = PTR_ERR(new); + goto out_unlock; } smp_store_release(&ctx->psi.trigger, new); - cgroup_put(cgrp); + +out_unlock: + cgroup_kn_unlock(of->kn); + if (ret) + return ret; return nbytes; } diff --git a/kernel/cgroup/cpuset-internal.h b/kernel/cgroup/cpuset-internal.h new file mode 100644 index 000000000000..8e5c275d730d --- /dev/null +++ b/kernel/cgroup/cpuset-internal.h @@ -0,0 +1,222 @@ +/* SPDX-License-Identifier: GPL-2.0-or-later */ + +#ifndef __CPUSET_INTERNAL_H +#define __CPUSET_INTERNAL_H + +#include +#include +#include +#include + +/* See "Frequency meter" comments, below. */ + +struct fmeter { + int cnt; /* unprocessed events count */ + int val; /* most recent output value */ + time64_t time; /* clock (secs) when val computed */ + spinlock_t lock; /* guards read or write of above */ +}; + +/* + * Invalid partition error code + */ +enum prs_errcode { + PERR_NONE = 0, + PERR_INVCPUS, + PERR_INVPARENT, + PERR_NOTPART, + PERR_NOTEXCL, + PERR_NOCPUS, + PERR_HOTPLUG, + PERR_CPUSEMPTY, +}; + +/* bits in struct cpuset flags field */ +typedef enum { + CS_ONLINE, + CS_CPU_EXCLUSIVE, + CS_MEM_EXCLUSIVE, + CS_MEM_HARDWALL, + CS_MEMORY_MIGRATE, + CS_SCHED_LOAD_BALANCE, + CS_SPREAD_PAGE, + CS_SPREAD_SLAB, +} cpuset_flagbits_t; + +/* The various types of files and directories in a cpuset file system */ + +typedef enum { + FILE_MEMORY_MIGRATE, + FILE_CPULIST, + FILE_MEMLIST, + FILE_EFFECTIVE_CPULIST, + FILE_EFFECTIVE_MEMLIST, + FILE_SUBPARTS_CPULIST, + FILE_CPU_EXCLUSIVE, + FILE_MEM_EXCLUSIVE, + FILE_MEM_HARDWALL, + FILE_SCHED_LOAD_BALANCE, + FILE_PARTITION_ROOT, + FILE_SCHED_RELAX_DOMAIN_LEVEL, + FILE_MEMORY_PRESSURE_ENABLED, + FILE_MEMORY_PRESSURE, + FILE_SPREAD_PAGE, + FILE_SPREAD_SLAB, +} cpuset_filetype_t; + +struct cpuset { + struct cgroup_subsys_state css; + + unsigned long flags; /* "unsigned long" so bitops work */ + + /* + * On default hierarchy: + * + * The user-configured masks can only be changed by writing to + * cpuset.cpus and cpuset.mems, and won't be limited by the + * parent masks. + * + * The effective masks is the real masks that apply to the tasks + * in the cpuset. They may be changed if the configured masks are + * changed or hotplug happens. + * + * effective_mask == configured_mask & parent's effective_mask, + * and if it ends up empty, it will inherit the parent's mask. + * + * + * On legacy hierarchy: + * + * The user-configured masks are always the same with effective masks. + */ + + /* user-configured CPUs and Memory Nodes allow to tasks */ + cpumask_var_t cpus_allowed; + nodemask_t mems_allowed; + + /* effective CPUs and Memory Nodes allow to tasks */ + cpumask_var_t effective_cpus; + nodemask_t effective_mems; + + /* + * CPUs allocated to child sub-partitions (default hierarchy only) + * - CPUs granted by the parent = effective_cpus U subparts_cpus + * - effective_cpus and subparts_cpus are mutually exclusive. + * + * effective_cpus contains only onlined CPUs, but subparts_cpus + * may have offlined ones. + */ + cpumask_var_t subparts_cpus; + + /* + * This is old Memory Nodes tasks took on. + * + * - top_cpuset.old_mems_allowed is initialized to mems_allowed. + * - A new cpuset's old_mems_allowed is initialized when some + * task is moved into it. + * - old_mems_allowed is used in cpuset_migrate_mm() when we change + * cpuset.mems_allowed and have tasks' nodemask updated, and + * then old_mems_allowed is updated to mems_allowed. + */ + nodemask_t old_mems_allowed; + + struct fmeter fmeter; /* memory_pressure filter */ + + /* + * Tasks are being attached to this cpuset. Used to prevent + * zeroing cpus/mems_allowed between ->can_attach() and ->attach(). + */ + int attach_in_progress; + + /* partition number for rebuild_sched_domains() */ + int pn; + + /* for custom sched domain */ + int relax_domain_level; + + /* number of CPUs in subparts_cpus */ + int nr_subparts_cpus; + + /* partition root state */ + int partition_root_state; + + /* + * Default hierarchy only: + * use_parent_ecpus - set if using parent's effective_cpus + * child_ecpus_count - # of children with use_parent_ecpus set + */ + int use_parent_ecpus; + int child_ecpus_count; + + /* + * number of SCHED_DEADLINE tasks attached to this cpuset, so that we + * know when to rebuild associated root domain bandwidth information. + */ + atomic_t nr_deadline_tasks; + int nr_migrate_dl_tasks; + u64 sum_migrate_dl_bw; + + /* Invalid partition error code, not lock protected */ + enum prs_errcode prs_err; + + /* Handle for cpuset.cpus.partition */ + struct cgroup_file partition_file; +}; + +static inline struct cpuset *css_cs(struct cgroup_subsys_state *css) +{ + return css ? container_of(css, struct cpuset, css) : NULL; +} + +/* Retrieve the cpuset for a task */ +static inline struct cpuset *task_cs(struct task_struct *task) +{ + return css_cs(task_css(task, cpuset_cgrp_id)); +} + +static inline struct cpuset *parent_cs(struct cpuset *cs) +{ + return css_cs(cs->css.parent); +} + +/* convenient tests for these bits */ +static inline bool is_cpuset_online(struct cpuset *cs) +{ + return test_bit(CS_ONLINE, &cs->flags) && !css_is_dying(&cs->css); +} + +static inline int is_cpu_exclusive(const struct cpuset *cs) +{ + return test_bit(CS_CPU_EXCLUSIVE, &cs->flags); +} + +static inline int is_mem_exclusive(const struct cpuset *cs) +{ + return test_bit(CS_MEM_EXCLUSIVE, &cs->flags); +} + +static inline int is_mem_hardwall(const struct cpuset *cs) +{ + return test_bit(CS_MEM_HARDWALL, &cs->flags); +} + +static inline int is_sched_load_balance(const struct cpuset *cs) +{ + return test_bit(CS_SCHED_LOAD_BALANCE, &cs->flags); +} + +static inline int is_memory_migrate(const struct cpuset *cs) +{ + return test_bit(CS_MEMORY_MIGRATE, &cs->flags); +} + +static inline int is_spread_page(const struct cpuset *cs) +{ + return test_bit(CS_SPREAD_PAGE, &cs->flags); +} + +static inline int is_spread_slab(const struct cpuset *cs) +{ + return test_bit(CS_SPREAD_SLAB, &cs->flags); +} + +#endif /* __CPUSET_INTERNAL_H */ diff --git a/kernel/cgroup/cpuset-v1.c b/kernel/cgroup/cpuset-v1.c new file mode 100644 index 000000000000..bdec4b196986 --- /dev/null +++ b/kernel/cgroup/cpuset-v1.c @@ -0,0 +1,3 @@ +// SPDX-License-Identifier: GPL-2.0-or-later + +#include "cpuset-internal.h" diff --git a/kernel/cgroup/cpuset.c b/kernel/cgroup/cpuset.c index bb413045d174..f59c0922122d 100644 --- a/kernel/cgroup/cpuset.c +++ b/kernel/cgroup/cpuset.c @@ -22,10 +22,9 @@ * distribution for more details. */ #include "cgroup-internal.h" +#include "cpuset-internal.h" #include -#include -#include #include #include #include @@ -39,10 +38,8 @@ #include #include #include -#include #include #include -#include #include DEFINE_STATIC_KEY_FALSE(cpusets_pre_enable_key); @@ -55,29 +52,6 @@ DEFINE_STATIC_KEY_FALSE(cpusets_enabled_key); */ DEFINE_STATIC_KEY_FALSE(cpusets_insane_config_key); -/* See "Frequency meter" comments, below. */ - -struct fmeter { - int cnt; /* unprocessed events count */ - int val; /* most recent output value */ - time64_t time; /* clock (secs) when val computed */ - spinlock_t lock; /* guards read or write of above */ -}; - -/* - * Invalid partition error code - */ -enum prs_errcode { - PERR_NONE = 0, - PERR_INVCPUS, - PERR_INVPARENT, - PERR_NOTPART, - PERR_NOTEXCL, - PERR_NOCPUS, - PERR_HOTPLUG, - PERR_CPUSEMPTY, -}; - static const char * const perr_strings[] = { [PERR_INVCPUS] = "Invalid cpu list in cpuset.cpus", [PERR_INVPARENT] = "Parent is an invalid partition root", @@ -88,104 +62,6 @@ static const char * const perr_strings[] = { [PERR_CPUSEMPTY] = "cpuset.cpus is empty", }; -struct cpuset { - struct cgroup_subsys_state css; - - unsigned long flags; /* "unsigned long" so bitops work */ - - /* - * On default hierarchy: - * - * The user-configured masks can only be changed by writing to - * cpuset.cpus and cpuset.mems, and won't be limited by the - * parent masks. - * - * The effective masks is the real masks that apply to the tasks - * in the cpuset. They may be changed if the configured masks are - * changed or hotplug happens. - * - * effective_mask == configured_mask & parent's effective_mask, - * and if it ends up empty, it will inherit the parent's mask. - * - * - * On legacy hierarchy: - * - * The user-configured masks are always the same with effective masks. - */ - - /* user-configured CPUs and Memory Nodes allow to tasks */ - cpumask_var_t cpus_allowed; - nodemask_t mems_allowed; - - /* effective CPUs and Memory Nodes allow to tasks */ - cpumask_var_t effective_cpus; - nodemask_t effective_mems; - - /* - * CPUs allocated to child sub-partitions (default hierarchy only) - * - CPUs granted by the parent = effective_cpus U subparts_cpus - * - effective_cpus and subparts_cpus are mutually exclusive. - * - * effective_cpus contains only onlined CPUs, but subparts_cpus - * may have offlined ones. - */ - cpumask_var_t subparts_cpus; - - /* - * This is old Memory Nodes tasks took on. - * - * - top_cpuset.old_mems_allowed is initialized to mems_allowed. - * - A new cpuset's old_mems_allowed is initialized when some - * task is moved into it. - * - old_mems_allowed is used in cpuset_migrate_mm() when we change - * cpuset.mems_allowed and have tasks' nodemask updated, and - * then old_mems_allowed is updated to mems_allowed. - */ - nodemask_t old_mems_allowed; - - struct fmeter fmeter; /* memory_pressure filter */ - - /* - * Tasks are being attached to this cpuset. Used to prevent - * zeroing cpus/mems_allowed between ->can_attach() and ->attach(). - */ - int attach_in_progress; - - /* partition number for rebuild_sched_domains() */ - int pn; - - /* for custom sched domain */ - int relax_domain_level; - - /* number of CPUs in subparts_cpus */ - int nr_subparts_cpus; - - /* partition root state */ - int partition_root_state; - - /* - * Default hierarchy only: - * use_parent_ecpus - set if using parent's effective_cpus - * child_ecpus_count - # of children with use_parent_ecpus set - */ - int use_parent_ecpus; - int child_ecpus_count; - - /* - * number of SCHED_DEADLINE tasks attached to this cpuset, so that we - * know when to rebuild associated root domain bandwidth information. - */ - int nr_deadline_tasks; - int nr_migrate_dl_tasks; - u64 sum_migrate_dl_bw; - - /* Invalid partition error code, not lock protected */ - enum prs_errcode prs_err; - - /* Handle for cpuset.cpus.partition */ - struct cgroup_file partition_file; -}; - /* * Partition root states: * @@ -215,87 +91,18 @@ struct tmpmasks { cpumask_var_t new_cpus; /* For update_cpumasks_hier() */ }; -static inline struct cpuset *css_cs(struct cgroup_subsys_state *css) -{ - return css ? container_of(css, struct cpuset, css) : NULL; -} - -/* Retrieve the cpuset for a task */ -static inline struct cpuset *task_cs(struct task_struct *task) -{ - return css_cs(task_css(task, cpuset_cgrp_id)); -} - -static inline struct cpuset *parent_cs(struct cpuset *cs) -{ - return css_cs(cs->css.parent); -} - void inc_dl_tasks_cs(struct task_struct *p) { struct cpuset *cs = task_cs(p); - cs->nr_deadline_tasks++; + atomic_inc(&cs->nr_deadline_tasks); } void dec_dl_tasks_cs(struct task_struct *p) { struct cpuset *cs = task_cs(p); - cs->nr_deadline_tasks--; -} - -/* bits in struct cpuset flags field */ -typedef enum { - CS_ONLINE, - CS_CPU_EXCLUSIVE, - CS_MEM_EXCLUSIVE, - CS_MEM_HARDWALL, - CS_MEMORY_MIGRATE, - CS_SCHED_LOAD_BALANCE, - CS_SPREAD_PAGE, - CS_SPREAD_SLAB, -} cpuset_flagbits_t; - -/* convenient tests for these bits */ -static inline bool is_cpuset_online(struct cpuset *cs) -{ - return test_bit(CS_ONLINE, &cs->flags) && !css_is_dying(&cs->css); -} - -static inline int is_cpu_exclusive(const struct cpuset *cs) -{ - return test_bit(CS_CPU_EXCLUSIVE, &cs->flags); -} - -static inline int is_mem_exclusive(const struct cpuset *cs) -{ - return test_bit(CS_MEM_EXCLUSIVE, &cs->flags); -} - -static inline int is_mem_hardwall(const struct cpuset *cs) -{ - return test_bit(CS_MEM_HARDWALL, &cs->flags); -} - -static inline int is_sched_load_balance(const struct cpuset *cs) -{ - return test_bit(CS_SCHED_LOAD_BALANCE, &cs->flags); -} - -static inline int is_memory_migrate(const struct cpuset *cs) -{ - return test_bit(CS_MEMORY_MIGRATE, &cs->flags); -} - -static inline int is_spread_page(const struct cpuset *cs) -{ - return test_bit(CS_SPREAD_PAGE, &cs->flags); -} - -static inline int is_spread_slab(const struct cpuset *cs) -{ - return test_bit(CS_SPREAD_SLAB, &cs->flags); + atomic_dec(&cs->nr_deadline_tasks); } static inline int is_partition_valid(const struct cpuset *cs) @@ -1091,7 +898,7 @@ static void dl_update_tasks_root_domain(struct cpuset *cs) struct css_task_iter it; struct task_struct *task; - if (cs->nr_deadline_tasks == 0) + if (atomic_read(&cs->nr_deadline_tasks) == 0) return; css_task_iter_start(&cs->css, 0, &it); @@ -2757,8 +2564,8 @@ static void cpuset_attach(struct cgroup_taskset *tset) cs->old_mems_allowed = cpuset_attach_nodemask_to; if (cs->nr_migrate_dl_tasks) { - cs->nr_deadline_tasks += cs->nr_migrate_dl_tasks; - oldcs->nr_deadline_tasks -= cs->nr_migrate_dl_tasks; + atomic_add(cs->nr_migrate_dl_tasks, &cs->nr_deadline_tasks); + atomic_sub(cs->nr_migrate_dl_tasks, &oldcs->nr_deadline_tasks); reset_migrate_dl_data(cs); } @@ -2769,27 +2576,6 @@ static void cpuset_attach(struct cgroup_taskset *tset) mutex_unlock(&cpuset_mutex); } -/* The various types of files and directories in a cpuset file system */ - -typedef enum { - FILE_MEMORY_MIGRATE, - FILE_CPULIST, - FILE_MEMLIST, - FILE_EFFECTIVE_CPULIST, - FILE_EFFECTIVE_MEMLIST, - FILE_SUBPARTS_CPULIST, - FILE_CPU_EXCLUSIVE, - FILE_MEM_EXCLUSIVE, - FILE_MEM_HARDWALL, - FILE_SCHED_LOAD_BALANCE, - FILE_PARTITION_ROOT, - FILE_SCHED_RELAX_DOMAIN_LEVEL, - FILE_MEMORY_PRESSURE_ENABLED, - FILE_MEMORY_PRESSURE, - FILE_SPREAD_PAGE, - FILE_SPREAD_SLAB, -} cpuset_filetype_t; - static int cpuset_write_u64(struct cgroup_subsys_state *css, struct cftype *cft, u64 val) { diff --git a/kernel/cgroup/rstat.c b/kernel/cgroup/rstat.c index c32439b855f5..c78d3c4c652e 100644 --- a/kernel/cgroup/rstat.c +++ b/kernel/cgroup/rstat.c @@ -1,6 +1,7 @@ // SPDX-License-Identifier: GPL-2.0-only #include "cgroup-internal.h" +#include #include #include @@ -18,7 +19,7 @@ static struct cgroup_rstat_cpu *cgroup_rstat_cpu(struct cgroup *cgrp, int cpu) } /** - * cgroup_rstat_updated - keep track of updated rstat_cpu + * __cgroup_rstat_updated - keep track of updated rstat_cpu * @cgrp: target cgroup * @cpu: cpu on which rstat_cpu was updated * @@ -26,7 +27,7 @@ static struct cgroup_rstat_cpu *cgroup_rstat_cpu(struct cgroup *cgrp, int cpu) * rstat_cpu->updated_children list. See the comment on top of * cgroup_rstat_cpu definition for details. */ -__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) +void __cgroup_rstat_updated(struct cgroup *cgrp, int cpu) { raw_spinlock_t *cpu_lock = per_cpu_ptr(&cgroup_rstat_cpu_lock, cpu); unsigned long flags; @@ -73,6 +74,18 @@ __bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) raw_spin_unlock_irqrestore(cpu_lock, flags); } +/* + * BPF-facing wrapper for __cgroup_rstat_updated(). Validate the caller-provided + * CPU before passing it to the internal rstat updater. + */ +__bpf_kfunc void cgroup_rstat_updated(struct cgroup *cgrp, int cpu) +{ + if (unlikely(cpu < 0 || cpu >= nr_cpu_ids || !cpu_possible(cpu))) + return; + + __cgroup_rstat_updated(cgrp, cpu); +} + /** * cgroup_rstat_cpu_pop_updated - iterate and dismantle rstat_cpu updated tree * @pos: current position @@ -394,7 +407,7 @@ static void cgroup_base_stat_cputime_account_end(struct cgroup *cgrp, unsigned long flags) { u64_stats_update_end_irqrestore(&rstatc->bsync, flags); - cgroup_rstat_updated(cgrp, smp_processor_id()); + __cgroup_rstat_updated(cgrp, smp_processor_id()); put_cpu_ptr(rstatc); } diff --git a/kernel/dma/direct.c b/kernel/dma/direct.c index 91915d351e12..d04464d7e230 100644 --- a/kernel/dma/direct.c +++ b/kernel/dma/direct.c @@ -168,22 +168,21 @@ static bool dma_direct_use_pool(struct device *dev, gfp_t gfp) return !gfpflags_allow_blocking(gfp) && !is_swiotlb_for_alloc(dev); } -static void *dma_direct_alloc_from_pool(struct device *dev, size_t size, - dma_addr_t *dma_handle, gfp_t gfp) +static struct page *dma_direct_alloc_from_pool(struct device *dev, size_t size, + dma_addr_t *dma_handle, void **cpu_addr, gfp_t gfp) { struct page *page; u64 phys_limit; - void *ret; if (WARN_ON_ONCE(!IS_ENABLED(CONFIG_DMA_COHERENT_POOL))) return NULL; gfp |= dma_direct_optimal_gfp_mask(dev, &phys_limit); - page = dma_alloc_from_pool(dev, size, &ret, gfp, dma_coherent_ok); + page = dma_alloc_from_pool(dev, size, cpu_addr, gfp, dma_coherent_ok); if (!page) return NULL; *dma_handle = phys_to_dma_direct(dev, page_to_phys(page)); - return ret; + return page; } static void *dma_direct_alloc_no_mapping(struct device *dev, size_t size, @@ -234,28 +233,26 @@ void *dma_direct_alloc(struct device *dev, size_t size, dma_handle); /* - * Otherwise remap if the architecture is asking for it. But - * given that remapping memory is a blocking operation we'll - * instead have to dip into the atomic pools. + * Otherwise we require the architecture to either be able to + * mark arbitrary parts of the kernel direct mapping uncached, + * or remapped it uncached. */ + set_uncached = IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED); remap = IS_ENABLED(CONFIG_DMA_DIRECT_REMAP); - if (remap) { - if (dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, - dma_handle, gfp); - } else { - if (!IS_ENABLED(CONFIG_ARCH_HAS_DMA_SET_UNCACHED)) - return NULL; - set_uncached = true; - } + if (!set_uncached && !remap) + return NULL; } /* - * Decrypting memory may block, so allocate the memory from the atomic - * pools if we can't block. + * Remapping or decrypting memory may block, allocate the memory from + * the atomic pools instead if we aren't allowed block. */ - if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); + if ((remap || force_dma_unencrypted(dev)) && + dma_direct_use_pool(dev, gfp)) { + page = dma_direct_alloc_from_pool(dev, size, dma_handle, + &ret, gfp); + return page ? ret : NULL; + } /* we always manually zero the memory once we are done */ page = __dma_direct_alloc_pages(dev, size, gfp & ~__GFP_ZERO, true); @@ -364,7 +361,7 @@ struct page *dma_direct_alloc_pages(struct device *dev, size_t size, void *ret; if (force_dma_unencrypted(dev) && dma_direct_use_pool(dev, gfp)) - return dma_direct_alloc_from_pool(dev, size, dma_handle, gfp); + return dma_direct_alloc_from_pool(dev, size, dma_handle, &ret, gfp); page = __dma_direct_alloc_pages(dev, size, gfp, false); if (!page) diff --git a/kernel/dma/swiotlb.c b/kernel/dma/swiotlb.c index 0968155a638e..669fd550338d 100644 --- a/kernel/dma/swiotlb.c +++ b/kernel/dma/swiotlb.c @@ -1594,10 +1594,10 @@ phys_addr_t default_swiotlb_base(void) phys_addr_t default_swiotlb_limit(void) { #ifdef CONFIG_SWIOTLB_DYNAMIC - return io_tlb_default_mem.phys_limit; -#else - return io_tlb_default_mem.defpool.end - 1; + if (io_tlb_default_mem.can_grow) + return io_tlb_default_mem.phys_limit; #endif + return io_tlb_default_mem.defpool.end - 1; } #ifdef CONFIG_DEBUG_FS diff --git a/kernel/events/core.c b/kernel/events/core.c index 95743e6c5b0c..a226f908490f 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -2312,6 +2312,7 @@ event_sched_out(struct perf_event *event, struct perf_event_context *ctx) !event->pending_work && !task_work_add(current, &event->pending_task, TWA_RESUME)) { event->pending_work = 1; + WARN_ON_ONCE(!atomic_long_inc_not_zero(&event->refcount)); } else { local_dec(&event->ctx->nr_pending); } @@ -3524,7 +3525,8 @@ static void perf_event_swap_task_ctx_data(struct perf_event_context *prev_ctx, } } -static void perf_ctx_sched_task_cb(struct perf_event_context *ctx, bool sched_in) +static void perf_ctx_sched_task_cb(struct perf_event_context *ctx, + struct task_struct *task, bool sched_in) { struct perf_event_pmu_context *pmu_ctx; struct perf_cpu_pmu_context *cpc; @@ -3532,8 +3534,11 @@ static void perf_ctx_sched_task_cb(struct perf_event_context *ctx, bool sched_in list_for_each_entry(pmu_ctx, &ctx->pmu_ctx_list, pmu_ctx_entry) { cpc = this_cpu_ptr(pmu_ctx->pmu->cpu_pmu_context); + if (cpc->task_epc != pmu_ctx) + continue; + if (cpc->sched_cb_usage && pmu_ctx->pmu->sched_task) - pmu_ctx->pmu->sched_task(pmu_ctx, sched_in); + pmu_ctx->pmu->sched_task(pmu_ctx, task, sched_in); } } @@ -3591,7 +3596,7 @@ perf_event_context_sched_out(struct task_struct *task, struct task_struct *next) WRITE_ONCE(ctx->task, next); WRITE_ONCE(next_ctx->task, task); - perf_ctx_sched_task_cb(ctx, false); + perf_ctx_sched_task_cb(ctx, task, false); perf_event_swap_task_ctx_data(ctx, next_ctx); perf_ctx_enable(ctx, false); @@ -3621,7 +3626,7 @@ perf_event_context_sched_out(struct task_struct *task, struct task_struct *next) perf_ctx_disable(ctx, false); inside_switch: - perf_ctx_sched_task_cb(ctx, false); + perf_ctx_sched_task_cb(ctx, task, false); task_ctx_sched_out(ctx, EVENT_ALL); perf_ctx_enable(ctx, false); @@ -3663,7 +3668,8 @@ void perf_sched_cb_inc(struct pmu *pmu) * PEBS requires this to provide PID/TID information. This requires we flush * all queued PEBS records before we context switch to a new task. */ -static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc, bool sched_in) +static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc, + struct task_struct *task, bool sched_in) { struct perf_cpu_context *cpuctx = this_cpu_ptr(&perf_cpu_context); struct pmu *pmu; @@ -3677,7 +3683,7 @@ static void __perf_pmu_sched_task(struct perf_cpu_pmu_context *cpc, bool sched_i perf_ctx_lock(cpuctx, cpuctx->task_ctx); perf_pmu_disable(pmu); - pmu->sched_task(cpc->task_epc, sched_in); + pmu->sched_task(&cpc->epc, task, sched_in); perf_pmu_enable(pmu); perf_ctx_unlock(cpuctx, cpuctx->task_ctx); @@ -3687,15 +3693,17 @@ static void perf_pmu_sched_task(struct task_struct *prev, struct task_struct *next, bool sched_in) { - struct perf_cpu_context *cpuctx = this_cpu_ptr(&perf_cpu_context); - struct perf_cpu_pmu_context *cpc; + struct perf_cpu_pmu_context *cpc, *cpc2; - /* cpuctx->task_ctx will be handled in perf_event_context_sched_in/out */ - if (prev == next || cpuctx->task_ctx) + if (prev == next) return; - list_for_each_entry(cpc, this_cpu_ptr(&sched_cb_list), sched_cb_entry) - __perf_pmu_sched_task(cpc, sched_in); + list_for_each_entry_safe(cpc, cpc2, this_cpu_ptr(&sched_cb_list), sched_cb_entry) { + if (cpc->task_epc) + continue; + + __perf_pmu_sched_task(cpc, sched_in ? next : prev, sched_in); + } } static void perf_event_switch(struct task_struct *task, @@ -3999,7 +4007,7 @@ static void perf_event_context_sched_in(struct task_struct *task) perf_ctx_lock(cpuctx, ctx); perf_ctx_disable(ctx, false); - perf_ctx_sched_task_cb(ctx, true); + perf_ctx_sched_task_cb(ctx, task, true); perf_ctx_enable(ctx, false); perf_ctx_unlock(cpuctx, ctx); @@ -4030,7 +4038,7 @@ static void perf_event_context_sched_in(struct task_struct *task) perf_event_sched_in(cpuctx, ctx); - perf_ctx_sched_task_cb(cpuctx->task_ctx, true); + perf_ctx_sched_task_cb(cpuctx->task_ctx, task, true); if (!RB_EMPTY_ROOT(&ctx->pinned_groups.tree)) perf_ctx_enable(&cpuctx->ctx, false); @@ -5014,6 +5022,7 @@ static void free_event_rcu(struct rcu_head *head) if (event->ns) put_pid_ns(event->ns); perf_event_free_filter(event); + kfree(event->addr_filter_ranges); kmem_cache_free(perf_event_cache, event); } @@ -5219,35 +5228,9 @@ static bool exclusive_event_installable(struct perf_event *event, static void perf_addr_filters_splice(struct perf_event *event, struct list_head *head); -static void perf_pending_task_sync(struct perf_event *event) -{ - struct callback_head *head = &event->pending_task; - - if (!event->pending_work) - return; - /* - * If the task is queued to the current task's queue, we - * obviously can't wait for it to complete. Simply cancel it. - */ - if (task_work_cancel(current, head)) { - event->pending_work = 0; - local_dec(&event->ctx->nr_pending); - return; - } - - /* - * All accesses related to the event are within the same - * non-preemptible section in perf_pending_task(). The RCU - * grace period before the event is freed will make sure all - * those accesses are complete by then. - */ - rcuwait_wait_event(&event->pending_work_wait, !event->pending_work, TASK_UNINTERRUPTIBLE); -} - static void _free_event(struct perf_event *event) { irq_work_sync(&event->pending_irq); - perf_pending_task_sync(event); unaccount_event(event); @@ -5275,7 +5258,6 @@ static void _free_event(struct perf_event *event) perf_event_free_bpf_prog(event); perf_addr_filters_splice(event, NULL); - kfree(event->addr_filter_ranges); if (event->destroy) event->destroy(event); @@ -5372,10 +5354,17 @@ static void perf_remove_from_owner(struct perf_event *event) static void put_event(struct perf_event *event) { + struct perf_event *parent; + if (!atomic_long_dec_and_test(&event->refcount)) return; + parent = event->parent; _free_event(event); + + /* Matches the refcount bump in inherit_event() */ + if (parent) + put_event(parent); } /* @@ -5459,11 +5448,6 @@ int perf_event_release_kernel(struct perf_event *event) if (tmp == child) { perf_remove_from_context(child, DETACH_GROUP); list_move(&child->child_list, &free_list); - /* - * This matches the refcount bump in inherit_event(); - * this can't be the last reference. - */ - put_event(event); } else { var = &ctx->refcount; } @@ -5489,7 +5473,8 @@ int perf_event_release_kernel(struct perf_event *event) void *var = &child->ctx->refcount; list_del(&child->child_list); - free_event(child); + /* Last reference unless ->pending_task work is pending */ + put_event(child); /* * Wake any perf_event_free_task() waiting for this event to be @@ -5500,7 +5485,11 @@ int perf_event_release_kernel(struct perf_event *event) } no_ctx: - put_event(event); /* Must be the 'last' reference */ + /* + * Last reference unless ->pending_task work is pending on this event + * or any of its children. + */ + put_event(event); return 0; } EXPORT_SYMBOL_GPL(perf_event_release_kernel); @@ -6801,18 +6790,20 @@ void perf_event_wakeup(struct perf_event *event) static void perf_sigtrap(struct perf_event *event) { /* - * We'd expect this to only occur if the irq_work is delayed and either - * ctx->task or current has changed in the meantime. This can be the - * case on architectures that do not implement arch_irq_work_raise(). + * Both perf_pending_task() and perf_pending_irq() can race with the + * task exiting or exec-ing. We can determine if such a race has + * occurred by checking if perf_event_exit_task(), which will set + * ctx->task to TASK_TOMBSTONE, has already been called. */ - if (WARN_ON_ONCE(event->ctx->task != current)) + if (event->ctx->task == TASK_TOMBSTONE) return; /* - * Both perf_pending_task() and perf_pending_irq() can race with the - * task exiting. + * We'd expect this to only occur if the irq_work is delayed and either + * ctx->task or current has changed in the meantime. This can be the + * case on architectures that do not implement arch_irq_work_raise(). */ - if (current->flags & PF_EXITING) + if (WARN_ON_ONCE(event->ctx->task != current)) return; send_sig_perf((void __user *)event->pending_addr, @@ -6903,11 +6894,6 @@ static void perf_pending_task(struct callback_head *head) struct perf_event *event = container_of(head, struct perf_event, pending_task); int rctx; - /* - * All accesses to the event must belong to the same implicit RCU read-side - * critical section as the ->pending_work reset. See comment in - * perf_pending_task_sync(). - */ preempt_disable_notrace(); /* * If we 'fail' here, that's OK, it means recursion is already disabled @@ -6919,12 +6905,13 @@ static void perf_pending_task(struct callback_head *head) event->pending_work = 0; perf_sigtrap(event); local_dec(&event->ctx->nr_pending); - rcuwait_wake_up(&event->pending_work_wait); } if (rctx >= 0) perf_swevent_put_recursion_context(rctx); preempt_enable_notrace(); + + put_event(event); } #ifdef CONFIG_GUEST_PERF_EVENTS @@ -12124,7 +12111,6 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu, init_waitqueue_head(&event->waitq); init_irq_work(&event->pending_irq, perf_pending_irq); init_task_work(&event->pending_task, perf_pending_task); - rcuwait_init(&event->pending_work_wait); mutex_init(&event->mmap_mutex); raw_spin_lock_init(&event->addr_filters.lock); @@ -12290,8 +12276,6 @@ perf_event_alloc(struct perf_event_attr *attr, int cpu, put_callchain_buffers(); } err_addr_filters: - kfree(event->addr_filter_ranges); - err_per_task: exclusive_event_destroy(event); @@ -12638,7 +12622,7 @@ SYSCALL_DEFINE5(perf_event_open, if (err) return err; - if (!attr.exclude_kernel) { + if (!attr.exclude_kernel || attr.text_poke) { err = perf_allow_kernel(&attr); if (err) return err; @@ -13283,8 +13267,7 @@ perf_event_exit_event(struct perf_event *event, * Kick perf_poll() for is_event_hup(); */ perf_event_wakeup(parent_event); - free_event(event); - put_event(parent_event); + put_event(event); return; } diff --git a/kernel/irq/proc.c b/kernel/irq/proc.c index 623b8136e9af..1dccdbbd7697 100644 --- a/kernel/irq/proc.c +++ b/kernel/irq/proc.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include @@ -333,7 +334,7 @@ void register_handler_proc(unsigned int irq, struct irqaction *action) #undef MAX_NAMELEN -#define MAX_NAMELEN 10 +#define MAX_NAMELEN 11 void register_irq_proc(unsigned int irq, struct irq_desc *desc) { @@ -354,7 +355,7 @@ void register_irq_proc(unsigned int irq, struct irq_desc *desc) if (desc->dir) goto out_unlock; - sprintf(name, "%d", irq); + snprintf(name, MAX_NAMELEN, "%u", irq); /* create /proc/irq/1234 */ desc->dir = proc_mkdir(name, root_irq_dir); @@ -408,7 +409,7 @@ void unregister_irq_proc(unsigned int irq, struct irq_desc *desc) #endif remove_proc_entry("spurious", desc->dir); - sprintf(name, "%u", irq); + snprintf(name, MAX_NAMELEN, "%u", irq); remove_proc_entry(name, root_irq_dir); } diff --git a/kernel/kprobes.c b/kernel/kprobes.c index fbe7c34fcd87..f05688f3acc5 100644 --- a/kernel/kprobes.c +++ b/kernel/kprobes.c @@ -491,14 +491,16 @@ static bool kprobe_queued(struct kprobe *p) static struct kprobe *get_optimized_kprobe(kprobe_opcode_t *addr) { int i; - struct kprobe *p = NULL; + struct kprobe *p; struct optimized_kprobe *op; /* Don't check i == 0, since that is a breakpoint case. */ - for (i = 1; !p && i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++) + for (i = 1; i < MAX_OPTIMIZED_LENGTH / sizeof(kprobe_opcode_t); i++) { p = get_kprobe(addr - i); + /* A disabled probe can have prepared, but inactive, optinsns. */ + if (!p || !kprobe_optready(p) || kprobe_disarmed(p)) + continue; - if (p && kprobe_optready(p)) { op = container_of(p, struct optimized_kprobe, kp); if (arch_within_optimized_kprobe(op, addr)) return p; diff --git a/kernel/module/dups.c b/kernel/module/dups.c index f3d7ea1e96d8..019545cef7e4 100644 --- a/kernel/module/dups.c +++ b/kernel/module/dups.c @@ -31,6 +31,7 @@ #include #include #include +#include #include "internal.h" @@ -39,13 +40,12 @@ static bool enable_dups_trace = IS_ENABLED(CONFIG_MODULE_DEBUG_AUTOLOAD_DUPS_TRACE); module_param(enable_dups_trace, bool_enable_only, 0644); -/* - * Protects dup_kmod_reqs list, adds / removals with RCU. - */ +/* A mutex-protected list of active kmod requests. */ static DEFINE_MUTEX(kmod_dup_mutex); static LIST_HEAD(dup_kmod_reqs); struct kmod_dup_req { + refcount_t refcount; struct list_head list; char name[MODULE_NAME_LEN]; struct completion first_req_done; @@ -54,12 +54,24 @@ struct kmod_dup_req { int dup_ret; }; +static void get_kmod_req(struct kmod_dup_req *kmod_req) +{ + refcount_inc(&kmod_req->refcount); +} + +static void put_kmod_req(struct kmod_dup_req *kmod_req) +{ + if (refcount_dec_and_test(&kmod_req->refcount)) + kfree(kmod_req); +} + static struct kmod_dup_req *kmod_dup_request_lookup(char *module_name) { struct kmod_dup_req *kmod_req; - list_for_each_entry_rcu(kmod_req, &dup_kmod_reqs, list, - lockdep_is_held(&kmod_dup_mutex)) { + lockdep_assert_held(&kmod_dup_mutex); + + list_for_each_entry(kmod_req, &dup_kmod_reqs, list) { if (strlen(kmod_req->name) == strlen(module_name) && !memcmp(kmod_req->name, module_name, strlen(module_name))) { return kmod_req; @@ -88,10 +100,10 @@ static void kmod_dup_request_delete(struct work_struct *work) * just returning 0. */ mutex_lock(&kmod_dup_mutex); - list_del_rcu(&kmod_req->list); - synchronize_rcu(); + list_del(&kmod_req->list); mutex_unlock(&kmod_dup_mutex); - kfree(kmod_req); + + put_kmod_req(kmod_req); } static void kmod_dup_request_complete(struct work_struct *work) @@ -100,6 +112,9 @@ static void kmod_dup_request_complete(struct work_struct *work) kmod_req = container_of(work, struct kmod_dup_req, complete_work); + if (completion_done(&kmod_req->first_req_done)) + goto out; + /* * This will ensure that the kernel will let all the waiters get * informed its time to check the return value. It's time to @@ -115,6 +130,9 @@ static void kmod_dup_request_complete(struct work_struct *work) * possible abuses of vmalloc() incurred by finit_module() thrashing. */ queue_delayed_work(system_wq, &kmod_req->delete_work, 60 * HZ); + +out: + put_kmod_req(kmod_req); } bool kmod_dup_request_exists_wait(char *module_name, bool wait, int *dup_ret) @@ -130,6 +148,7 @@ bool kmod_dup_request_exists_wait(char *module_name, bool wait, int *dup_ret) if (!new_kmod_req) return false; + refcount_set(&new_kmod_req->refcount, 1); memcpy(new_kmod_req->name, module_name, strlen(module_name)); INIT_WORK(&new_kmod_req->complete_work, kmod_dup_request_complete); INIT_DELAYED_WORK(&new_kmod_req->delete_work, kmod_dup_request_delete); @@ -162,10 +181,12 @@ bool kmod_dup_request_exists_wait(char *module_name, bool wait, int *dup_ret) * keep tab on duplicates later. */ pr_debug("New request_module() for %s\n", module_name); - list_add_rcu(&new_kmod_req->list, &dup_kmod_reqs); + list_add(&new_kmod_req->list, &dup_kmod_reqs); mutex_unlock(&kmod_dup_mutex); return false; } + + get_kmod_req(kmod_req); mutex_unlock(&kmod_dup_mutex); /* We are dealing with a duplicate request now */ @@ -195,7 +216,7 @@ bool kmod_dup_request_exists_wait(char *module_name, bool wait, int *dup_ret) * calls bail out right away. */ *dup_ret = 0; - return true; + goto out; } /* @@ -210,12 +231,14 @@ bool kmod_dup_request_exists_wait(char *module_name, bool wait, int *dup_ret) TASK_UNINTERRUPTIBLE | TASK_KILLABLE); if (ret) { *dup_ret = ret; - return true; + goto out; } /* Now the duplicate request has the same exact return value as the first request */ *dup_ret = kmod_req->dup_ret; +out: + put_kmod_req(kmod_req); return true; } @@ -225,12 +248,23 @@ void kmod_dup_request_announce(char *module_name, int ret) mutex_lock(&kmod_dup_mutex); + /* + * Look for a kmod_dup_req previously added in + * kmod_dup_request_exists_wait(). Note that a request_module_nowait() + * without its own kmod_dup_req entry can announce a result of + * a concurrent request_module() call. + */ kmod_req = kmod_dup_request_lookup(module_name); - if (!kmod_req) - goto out; + if (!kmod_req || completion_done(&kmod_req->first_req_done)) { + mutex_unlock(&kmod_dup_mutex); + return; + } kmod_req->dup_ret = ret; + get_kmod_req(kmod_req); + mutex_unlock(&kmod_dup_mutex); + /* * If we complete() here we may allow duplicate threads * to continue before the first one that submitted the @@ -241,8 +275,6 @@ void kmod_dup_request_announce(char *module_name, int ret) * There is no rush. But we also don't want to hold the * caller up forever or introduce any boot delays. */ - queue_work(system_wq, &kmod_req->complete_work); - -out: - mutex_unlock(&kmod_dup_mutex); + if (!queue_work(system_wq, &kmod_req->complete_work)) + put_kmod_req(kmod_req); } diff --git a/kernel/params.c b/kernel/params.c index 9d09ea99363f..12fb187ebf63 100644 --- a/kernel/params.c +++ b/kernel/params.c @@ -1,19 +1,20 @@ // SPDX-License-Identifier: GPL-2.0-or-later -/* Helpers for initial module or kernel cmdline parsing - Copyright (C) 2001 Rusty Russell. - -*/ +/* + * Helpers for initial module or kernel cmdline parsing + * Copyright (C) 2001 Rusty Russell. + */ +#include +#include +#include +#include #include #include -#include -#include #include #include -#include -#include -#include -#include +#include #include +#include +#include #ifdef CONFIG_SYSFS /* Protects all built-in parameters, modules use their own param_lock */ @@ -48,7 +49,7 @@ static void *kmalloc_parameter(unsigned int size) { struct kmalloced_param *p; - p = kmalloc(sizeof(*p) + size, GFP_KERNEL); + p = kmalloc(size_add(sizeof(*p), size), GFP_KERNEL); if (!p) return NULL; @@ -264,22 +265,29 @@ EXPORT_SYMBOL_GPL(param_set_uint_minmax); int param_set_charp(const char *val, const struct kernel_param *kp) { - if (strlen(val) > 1024) { + char *tmp; + size_t len, maxlen = 1024; + + len = strnlen(val, maxlen + 1); + if (len == maxlen + 1) { pr_err("%s: string parameter too long\n", kp->name); return -ENOSPC; } - maybe_kfree_parameter(*(char **)kp->arg); - - /* This is a hack. We can't kmalloc in early boot, and we - * don't need to; this mangled commandline is preserved. */ + /* + * This is a hack. We can't kmalloc() in early boot, and we + * don't need to; this mangled commandline is preserved. + */ if (slab_is_available()) { - *(char **)kp->arg = kmalloc_parameter(strlen(val)+1); - if (!*(char **)kp->arg) + tmp = kmalloc_parameter(len + 1); + if (!tmp) return -ENOMEM; - strcpy(*(char **)kp->arg, val); + memcpy(tmp, val, len + 1); } else - *(const char **)kp->arg = val; + tmp = (char *)val; + + maybe_kfree_parameter(*(char **)kp->arg); + *(char **)kp->arg = tmp; return 0; } @@ -512,7 +520,7 @@ int param_set_copystring(const char *val, const struct kernel_param *kp) { const struct kparam_string *kps = kp->str; - if (strlen(val)+1 > kps->maxlen) { + if (strnlen(val, kps->maxlen) == kps->maxlen) { pr_err("%s: string doesn't fit in %u chars.\n", kp->name, kps->maxlen-1); return -ENOSPC; @@ -737,8 +745,10 @@ void module_param_sysfs_remove(struct module *mod) { if (mod->mkobj.mp) { sysfs_remove_group(&mod->mkobj.kobj, &mod->mkobj.mp->grp); - /* We are positive that no one is using any param - * attrs at this point. Deallocate immediately. */ + /* + * We are positive that no one is using any param + * attrs at this point. Deallocate immediately. + */ free_module_param_attrs(&mod->mkobj); } } diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c index c38e5933a5d6..24aa302b3269 100644 --- a/kernel/rcu/srcutiny.c +++ b/kernel/rcu/srcutiny.c @@ -9,6 +9,7 @@ */ #include +#include #include #include #include @@ -37,6 +38,7 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp) ssp->srcu_idx_max = 0; INIT_WORK(&ssp->srcu_work, srcu_drive_gp); INIT_LIST_HEAD(&ssp->srcu_work.entry); + init_irq_work(&ssp->srcu_irq_work, srcu_tiny_irq_work); return 0; } @@ -80,6 +82,7 @@ EXPORT_SYMBOL_GPL(init_srcu_struct); void cleanup_srcu_struct(struct srcu_struct *ssp) { WARN_ON(ssp->srcu_lock_nesting[0] || ssp->srcu_lock_nesting[1]); + irq_work_sync(&ssp->srcu_irq_work); flush_work(&ssp->srcu_work); WARN_ON(ssp->srcu_gp_running); WARN_ON(ssp->srcu_gp_waiting); @@ -156,6 +159,20 @@ void srcu_drive_gp(struct work_struct *wp) } EXPORT_SYMBOL_GPL(srcu_drive_gp); +/* + * Use an irq_work to defer schedule_work() to avoid acquiring the workqueue + * pool->lock while the caller might hold scheduler locks, causing lockdep + * splats due to workqueue_init() doing a wakeup. + */ +void srcu_tiny_irq_work(struct irq_work *irq_work) +{ + struct srcu_struct *ssp; + + ssp = container_of(irq_work, struct srcu_struct, srcu_irq_work); + schedule_work(&ssp->srcu_work); +} +EXPORT_SYMBOL_GPL(srcu_tiny_irq_work); + static void srcu_gp_start_if_needed(struct srcu_struct *ssp) { unsigned long cookie; @@ -166,7 +183,7 @@ static void srcu_gp_start_if_needed(struct srcu_struct *ssp) WRITE_ONCE(ssp->srcu_idx_max, cookie); if (!READ_ONCE(ssp->srcu_gp_running)) { if (likely(srcu_init_done)) - schedule_work(&ssp->srcu_work); + irq_work_queue(&ssp->srcu_irq_work); else if (list_empty(&ssp->srcu_work.entry)) list_add(&ssp->srcu_work.entry, &srcu_boot_list); } diff --git a/kernel/reboot.c b/kernel/reboot.c index 6ebef11c8876..153a7588354f 100644 --- a/kernel/reboot.c +++ b/kernel/reboot.c @@ -24,8 +24,7 @@ */ static int C_A_D = 1; -struct pid *cad_pid; -EXPORT_SYMBOL(cad_pid); +struct pid __rcu *cad_pid; #if defined(CONFIG_ARM) #define DEFAULT_REBOOT_MODE = REBOOT_HARD diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 797b81dcf69a..ae0638510904 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -366,6 +366,17 @@ static void __sched_core_flip(bool enabled) sched_core_lock(cpu, &flags); + /* + * A core-wide selection may have the shared rq lock temporarily + * released by a lock-dropping ->pick_task(). Flipping would + * rebind rq_lockp() under it. Wait it out. + */ + while (cpu_rq(cpu)->core->core_pick_in_flight) { + sched_core_unlock(cpu, &flags); + cpu_relax(); + sched_core_lock(cpu, &flags); + } + for_each_cpu(t, smt_mask) cpu_rq(t)->core_enabled = enabled; @@ -6094,6 +6105,8 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) return __pick_next_task(rq, prev, rf); } + rq->core->core_pick_in_flight++; + /* * If there were no {en,de}queues since we picked (IOW, the task * pointers are all still valid), and we haven't scheduled the last @@ -6287,6 +6300,7 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) out_set_next: set_next_task(rq, next); out: + rq->core->core_pick_in_flight--; if (rq->core->core_forceidle_count && next == rq->idle) queue_core_balance(rq); @@ -6483,6 +6497,13 @@ static void sched_core_cpu_deactivate(unsigned int cpu) core_rq->core_forceidle_seq = rq->core_forceidle_seq; core_rq->core_forceidle_occupation = rq->core_forceidle_occupation; + /* + * A stale leftover would bias the count forever if this CPU later + * returns as its own leader. Move, don't copy. + */ + core_rq->core_pick_in_flight = rq->core_pick_in_flight; + rq->core_pick_in_flight = 0; + /* * Accounting edge for forced idle is handled in pick_next_task(). * Don't need another one here, since the hotplug thread shouldn't @@ -7536,6 +7557,35 @@ static void __setscheduler_params(struct task_struct *p, set_load_weight(p, true); } +#ifdef CONFIG_RT_MUTEXES +static inline void __setscheduler_dl_pi(int newprio, int policy, + struct task_struct *p, + int *queue_flags) +{ + /* + * In case a DEADLINE task (either proper or boosted) gets + * setscheduled to a lower priority class, check if it neeeds to + * inherit parameters from a potential pi_task. In that case make + * sure replenishment happens with the next enqueue. + */ + + if (dl_prio(newprio) && !dl_policy(policy)) { + struct task_struct *pi_task = rt_mutex_get_top_task(p); + + if (pi_task) { + p->dl.pi_se = pi_task->dl.pi_se; + *queue_flags |= ENQUEUE_REPLENISH; + } + } +} +#else /* !CONFIG_RT_MUTEXES */ +static inline void __setscheduler_dl_pi(int newprio, int policy, + struct task_struct *p, + int *queue_flags) +{ +} +#endif /* !CONFIG_RT_MUTEXES */ + /* * Check the target process has a UID that matches the current process's: */ @@ -7804,6 +7854,7 @@ static int __sched_setscheduler(struct task_struct *p, if (!(attr->sched_flags & SCHED_FLAG_KEEP_PARAMS)) { __setscheduler_params(p, attr); __setscheduler_prio(p, newprio); + __setscheduler_dl_pi(newprio, policy, p, &queue_flags); } __setscheduler_uclamp(p, attr); @@ -8346,7 +8397,7 @@ __sched_setaffinity(struct task_struct *p, struct affinity_context *ctx) bool empty = !cpumask_and(new_mask, new_mask, ctx->user_mask); - if (WARN_ON_ONCE(empty)) + if (empty) cpumask_copy(new_mask, cpus_allowed); } __set_cpus_allowed_ptr(p, ctx); @@ -10074,6 +10125,7 @@ void __init sched_init(void) rq->core_forceidle_count = 0; rq->core_forceidle_occupation = 0; rq->core_forceidle_start = 0; + rq->core_pick_in_flight = 0; rq->core_cookie = 0UL; #endif diff --git a/kernel/sched/deadline.c b/kernel/sched/deadline.c index 91459b664014..4546d15419c2 100644 --- a/kernel/sched/deadline.c +++ b/kernel/sched/deadline.c @@ -1727,8 +1727,12 @@ static void enqueue_task_dl(struct rq *rq, struct task_struct *p, int flags) * The replenish timer needs to be canceled. No * problem if it fires concurrently: boosted threads * are ignored in dl_task_timer(). + * + * If the timer callback was running (hrtimer_try_to_cancel == -1), + * it will eventually call put_task_struct(). */ - hrtimer_try_to_cancel(&p->dl.dl_timer); + if (hrtimer_try_to_cancel(&p->dl.dl_timer) == 1) + put_task_struct(p); p->dl.dl_throttled = 0; } } else if (!dl_prio(p->normal_prio)) { diff --git a/kernel/sched/sched.h b/kernel/sched/sched.h index 65ff0254659a..47aebf155c3a 100644 --- a/kernel/sched/sched.h +++ b/kernel/sched/sched.h @@ -1158,6 +1158,7 @@ struct rq { unsigned int core_forceidle_seq; unsigned int core_forceidle_occupation; u64 core_forceidle_start; + unsigned int core_pick_in_flight; #endif /* Scratch cpumask to be temporarily used under rq_lock */ diff --git a/kernel/signal.c b/kernel/signal.c index 6a51bd690092..cfc419613190 100644 --- a/kernel/signal.c +++ b/kernel/signal.c @@ -1950,6 +1950,18 @@ int kill_pid(struct pid *pid, int sig, int priv) } EXPORT_SYMBOL(kill_pid); +int kill_cad_pid(int sig, int priv) +{ + int ret; + + rcu_read_lock(); + ret = kill_pid(rcu_dereference(cad_pid), sig, priv); + rcu_read_unlock(); + + return ret; +} +EXPORT_SYMBOL(kill_cad_pid); + /* * These functions support sending signals using preallocated sigqueue * structures. This is needed "because realtime applications cannot diff --git a/kernel/sysctl.c b/kernel/sysctl.c index f8f07c049238..0f0c2fafcfb2 100644 --- a/kernel/sysctl.c +++ b/kernel/sysctl.c @@ -1331,10 +1331,13 @@ static int proc_do_cad_pid(struct ctl_table *table, int write, void *buffer, size_t *lenp, loff_t *ppos) { struct pid *new_pid; + struct pid *old_pid; pid_t tmp; int r; - tmp = pid_vnr(cad_pid); + rcu_read_lock(); + tmp = pid_vnr(rcu_dereference(cad_pid)); + rcu_read_unlock(); r = __do_proc_dointvec(&tmp, table, write, buffer, lenp, ppos, NULL, NULL); @@ -1345,7 +1348,13 @@ static int proc_do_cad_pid(struct ctl_table *table, int write, void *buffer, if (!new_pid) return -ESRCH; - put_pid(xchg(&cad_pid, new_pid)); + old_pid = unrcu_pointer(xchg(&cad_pid, RCU_INITIALIZER(new_pid))); + /* + * Wait for cad_pid readers before put_pid(). We cannot use + * call_rcu() here because free_pid() already owns pid->rcu. + */ + synchronize_rcu(); + put_pid(old_pid); return 0; } diff --git a/kernel/time/namespace.c b/kernel/time/namespace.c index 0775b9ec952a..95a18cae6647 100644 --- a/kernel/time/namespace.c +++ b/kernel/time/namespace.c @@ -416,6 +416,11 @@ int proc_timens_set_offset(struct file *file, struct task_struct *p, off->val.tv_sec < -KTIME_SEC_MAX) goto out; + if (off->val.tv_nsec < 0 || off->val.tv_nsec >= NSEC_PER_SEC) { + err = -EINVAL; + goto out; + } + tp = timespec64_add(tp, off->val); /* * KTIME_SEC_MAX is divided by 2 to be sure that KTIME_MAX is diff --git a/kernel/trace/fgraph.c b/kernel/trace/fgraph.c index c83c005e654e..f7e0ade90500 100644 --- a/kernel/trace/fgraph.c +++ b/kernel/trace/fgraph.c @@ -648,6 +648,7 @@ int register_ftrace_graph(struct fgraph_ops *gops) ret = start_graph_tracing(); if (ret) { ftrace_graph_active--; + unregister_pm_notifier(&ftrace_suspend_notifier); goto out; } diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index a5ddc3f7aefa..ad61bd1cba55 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -75,6 +75,8 @@ #define INIT_OPS_HASH(opsname) \ .func_hash = &opsname.local_hash, \ .local_hash.regex_lock = __MUTEX_INITIALIZER(opsname.local_hash.regex_lock), +/* Used only to synchronize the initialization of ftrace_ops */ +static DEFINE_MUTEX(ops_mutex); #else #define INIT_OPS_HASH(opsname) #endif @@ -159,10 +161,17 @@ const struct ftrace_ops ftrace_nop_ops = { static inline void ftrace_ops_init(struct ftrace_ops *ops) { #ifdef CONFIG_DYNAMIC_FTRACE - if (!(ops->flags & FTRACE_OPS_FL_INITIALIZED)) { + unsigned long flags = smp_load_acquire(&ops->flags); + + if (!(flags & FTRACE_OPS_FL_INITIALIZED)) { + guard(mutex)(&ops_mutex); + /* Could have been initialized before lock taken */ + if (unlikely(ops->flags & FTRACE_OPS_FL_INITIALIZED)) + return; mutex_init(&ops->local_hash.regex_lock); ops->func_hash = &ops->local_hash; - ops->flags |= FTRACE_OPS_FL_INITIALIZED; + flags = ops->flags | FTRACE_OPS_FL_INITIALIZED; + smp_store_release(&ops->flags, flags); } #endif } @@ -4017,7 +4026,8 @@ ftrace_avail_addrs_open(struct inode *inode, struct file *file) /** * ftrace_regex_open - initialize function tracer filter files - * @ops: The ftrace_ops that hold the hash filters + * @tr: The trace_array that holds the ftrace_ops [optional] + * @ops: The ftrace_ops that hold the hash filters [optional] * @flag: The type of filter to process * @inode: The inode, usually passed in to your open routine * @file: The file, usually passed in to your open routine @@ -4030,25 +4040,46 @@ ftrace_avail_addrs_open(struct inode *inode, struct file *file) * ftrace_notrace_write() if @flag has FTRACE_ITER_NOTRACE set. * tracing_lseek() should be used as the lseek routine, and * release must call ftrace_regex_release(). + * + * Note, If @tr is not NULL, its reference has to be taken before + * @ops may be referenced. + * If @ops is NULL and @tr is not, then @tr->ops is used. + * If @tr is NULL and @ops is not then @ops->private is uesd for @tr. + * If both @tr and @ops are NULL, then the &global_ops is + * to be used, and @tr will be the global_ops.private pointer. + * + * Returns: 0 on success or a negative errno value on failure */ int -ftrace_regex_open(struct ftrace_ops *ops, int flag, +ftrace_regex_open(struct trace_array *tr, struct ftrace_ops *ops, int flag, struct inode *inode, struct file *file) { - struct ftrace_iterator *iter; + struct ftrace_iterator *iter = NULL; struct ftrace_hash *hash; struct list_head *mod_head; - struct trace_array *tr = ops->private; - int ret = -ENOMEM; - - ftrace_ops_init(ops); + int ret = -ENODEV; if (unlikely(ftrace_disabled)) return -ENODEV; + if (!tr) { + if (!ops) + ops = &global_ops; + tr = ops->private; + } + if (tracing_check_open_get_tr(tr)) return -ENODEV; + if (!ops) + ops = tr->ops; + + if (WARN_ON_ONCE(!ops)) + goto out; + + ftrace_ops_init(ops); + + ret = -ENOMEM; iter = kzalloc(sizeof(*iter), GFP_KERNEL); if (!iter) goto out; @@ -4126,21 +4157,19 @@ ftrace_regex_open(struct ftrace_ops *ops, int flag, static int ftrace_filter_open(struct inode *inode, struct file *file) { - struct ftrace_ops *ops = inode->i_private; + struct trace_array *tr = inode->i_private; - /* Checks for tracefs lockdown */ - return ftrace_regex_open(ops, - FTRACE_ITER_FILTER | FTRACE_ITER_DO_PROBES, - inode, file); + return ftrace_regex_open(tr, NULL, + FTRACE_ITER_FILTER | FTRACE_ITER_DO_PROBES, + inode, file); } static int ftrace_notrace_open(struct inode *inode, struct file *file) { - struct ftrace_ops *ops = inode->i_private; + struct trace_array *tr = inode->i_private; - /* Checks for tracefs lockdown */ - return ftrace_regex_open(ops, FTRACE_ITER_NOTRACE, + return ftrace_regex_open(tr, NULL, FTRACE_ITER_NOTRACE, inode, file); } @@ -6437,15 +6466,15 @@ static const struct file_operations ftrace_graph_notrace_fops = { }; #endif /* CONFIG_FUNCTION_GRAPH_TRACER */ -void ftrace_create_filter_files(struct ftrace_ops *ops, +void ftrace_create_filter_files(struct trace_array *tr, struct dentry *parent) { trace_create_file("set_ftrace_filter", TRACE_MODE_WRITE, parent, - ops, &ftrace_filter_fops); + tr, &ftrace_filter_fops); trace_create_file("set_ftrace_notrace", TRACE_MODE_WRITE, parent, - ops, &ftrace_notrace_fops); + tr, &ftrace_notrace_fops); } /* @@ -6470,7 +6499,6 @@ void ftrace_destroy_filter_files(struct ftrace_ops *ops) static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer) { - trace_create_file("available_filter_functions", TRACE_MODE_READ, d_tracer, NULL, &ftrace_avail_fops); @@ -6483,7 +6511,7 @@ static __init int ftrace_init_dyn_tracefs(struct dentry *d_tracer) trace_create_file("touched_functions", TRACE_MODE_READ, d_tracer, NULL, &ftrace_touched_fops); - ftrace_create_filter_files(&global_ops, d_tracer); + ftrace_create_filter_files(NULL, d_tracer); #ifdef CONFIG_FUNCTION_GRAPH_TRACER trace_create_file("set_graph_function", TRACE_MODE_WRITE, d_tracer, diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 47f1e8f47d47..9ab20e1a936f 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -8902,12 +8902,57 @@ trace_options_write(struct file *filp, const char __user *ubuf, size_t cnt, return cnt; } +static bool tr_option_match(struct trace_array *tr, void *topt) +{ + for (int i = 0; i < tr->nr_topts; i++) { + struct trace_options *tr_topts = &tr->topts[i]; + + if (topt >= (void *)&tr_topts->topts[0] && + topt < (void *)&tr_topts->topts[tr_topts->nr_topts]) + return true; + } + return false; +} + +/* + * The topt is the address of a trace_array->topts[] element that holds the + * the tracer options descriptor. But since the trace_array reference has not + * been taken yet, it cannot be dereferenced as it could have been freed by + * a rmdir of the instance the trace_array represents. + * + * Search the list of trace_arrays and compare the topt to the address of + * the entire trace_array topts array for each trace_array in the list. + * If one is matched, then take the reference and return it. If not, the + * trace_array no longer exits. + */ +static int trace_array_tracer_options_get(void *topt) +{ + struct trace_array *tr; + int ret; + + ret = security_locked_down(LOCKDOWN_TRACEFS); + if (ret) + return ret; + + if (tracing_disabled) + return -ENODEV; + + guard(mutex)(&trace_types_lock); + list_for_each_entry(tr, &ftrace_trace_arrays, list) { + if (tr_option_match(tr, topt)) { + tr->ref++; + return 0; + } + } + return -ENODEV; +} + static int tracing_open_options(struct inode *inode, struct file *filp) { struct trace_option_dentry *topt = inode->i_private; int ret; - ret = tracing_check_open_get_tr(topt->tr); + ret = trace_array_tracer_options_get(topt); if (ret) return ret; @@ -9016,11 +9061,72 @@ trace_options_core_write(struct file *filp, const char __user *ubuf, size_t cnt, return cnt; } +/* + * The tr_index is the address of a trace_array->trace_flags_index[] + * element that holds the index of the trace flag. But since the + * trace_array reference has not been taken yet, it cannot be referenced + * as it could have been freed by a rmdir of the instance the trace_array + * represents. + * + * Search the list of trace_arrays and compare the tr_index to the + * address of the entire trace_array trace_flags_index array for each + * trace_array in the list. If one is matched, then take the reference + * and return it. If not, the trace_array no longer exits. + */ +static int trace_array_options_get(void *tr_index) +{ + struct trace_array *tr; + int ret; + + ret = security_locked_down(LOCKDOWN_TRACEFS); + if (ret) + return ret; + + if (tracing_disabled) + return -ENODEV; + + guard(mutex)(&trace_types_lock); + list_for_each_entry(tr, &ftrace_trace_arrays, list) { + if (tr_index >= (void *)&tr->trace_flags_index[0] && + tr_index < (void *)&tr->trace_flags_index[TRACE_FLAGS_MAX_SIZE]) { + tr->ref++; + return 0; + } + } + return -ENODEV; +} + +static int trace_options_open(struct inode *inode, struct file *filp) +{ + void *tr_index = inode->i_private; + + if (trace_array_options_get(tr_index) < 0) + return -ENODEV; + + filp->private_data = tr_index; + + return 0; +} + +static int trace_options_release(struct inode *inode, struct file *filp) +{ + void *tr_index = filp->private_data; + struct trace_array *tr; + unsigned int index; + + get_tr_index(tr_index, &tr, &index); + + trace_array_put(tr); + + return 0; +} + static const struct file_operations trace_options_core_fops = { - .open = tracing_open_generic, - .read = trace_options_core_read, - .write = trace_options_core_write, - .llseek = generic_file_llseek, + .open = trace_options_open, + .read = trace_options_core_read, + .write = trace_options_core_write, + .llseek = generic_file_llseek, + .release = trace_options_release, }; struct dentry *trace_create_file(const char *name, @@ -9130,6 +9236,7 @@ create_trace_option_files(struct trace_array *tr, struct tracer *tracer) tr->topts = tr_topts; tr->topts[tr->nr_topts].tracer = tracer; tr->topts[tr->nr_topts].topts = topts; + tr->topts[tr->nr_topts].nr_topts = cnt; tr->nr_topts++; for (cnt = 0; opts[cnt].name; cnt++) { diff --git a/kernel/trace/trace.h b/kernel/trace/trace.h index 05ac2b5c8e20..640257a8405d 100644 --- a/kernel/trace/trace.h +++ b/kernel/trace/trace.h @@ -223,6 +223,7 @@ struct array_buffer { struct trace_options { struct tracer *tracer; struct trace_option_dentry *topts; + int nr_topts; }; struct trace_pid_list *trace_pid_list_alloc(void); @@ -1114,7 +1115,7 @@ extern void clear_ftrace_function_probes(struct trace_array *tr); int register_ftrace_command(struct ftrace_func_command *cmd); int unregister_ftrace_command(struct ftrace_func_command *cmd); -void ftrace_create_filter_files(struct ftrace_ops *ops, +void ftrace_create_filter_files(struct trace_array *tr, struct dentry *parent); void ftrace_destroy_filter_files(struct ftrace_ops *ops); @@ -1137,11 +1138,11 @@ static inline void clear_ftrace_function_probes(struct trace_array *tr) { } +#define ftrace_create_filter_files(tr, parent) do { } while (0) /* * The ops parameter passed in is usually undefined. * This must be a macro. */ -#define ftrace_create_filter_files(ops, parent) do { } while (0) #define ftrace_destroy_filter_files(ops) do { } while (0) #endif /* CONFIG_FUNCTION_TRACER && CONFIG_DYNAMIC_FTRACE */ diff --git a/kernel/trace/trace_btf.c b/kernel/trace/trace_btf.c index 5bbdbcbbde3c..40e66eb59abb 100644 --- a/kernel/trace/trace_btf.c +++ b/kernel/trace/trace_btf.c @@ -74,24 +74,24 @@ const struct btf_member *btf_find_struct_member(struct btf *btf, { struct btf_anon_stack *anon_stack; const struct btf_member *member; + const struct btf_type *mtype; u32 tid, cur_offset = 0; const char *name; int i, top = 0; + if (!btf_type_is_struct(type)) + return ERR_PTR(-EINVAL); + anon_stack = kcalloc(BTF_ANON_STACK_MAX, sizeof(*anon_stack), GFP_KERNEL); if (!anon_stack) return ERR_PTR(-ENOMEM); retry: - if (!btf_type_is_struct(type)) { - member = ERR_PTR(-EINVAL); - goto out; - } - for_each_member(i, type, member) { if (!member->name_off) { /* Anonymous union/struct: push it for later use */ - if (btf_type_skip_modifiers(btf, member->type, &tid) && + mtype = btf_type_skip_modifiers(btf, member->type, &tid); + if (mtype && btf_type_is_struct(mtype) && top < BTF_ANON_STACK_MAX) { anon_stack[top].tid = tid; anon_stack[top++].offset = diff --git a/kernel/trace/trace_events_hist.c b/kernel/trace/trace_events_hist.c index 3ea423c62448..60ec8c7c6b67 100644 --- a/kernel/trace/trace_events_hist.c +++ b/kernel/trace/trace_events_hist.c @@ -2379,6 +2379,7 @@ parse_field(struct hist_trigger_data *hist_data, struct trace_event_file *file, *flags |= HIST_FIELD_FL_CPU; } else if (field && field->filter_type == FILTER_STACKTRACE) { *flags |= HIST_FIELD_FL_STACKTRACE; + field = NULL; } else { hist_err(tr, HIST_ERR_FIELD_NOT_FOUND, errpos(field_name)); @@ -6332,11 +6333,17 @@ static struct event_trigger_ops event_hist_trigger_ops = { static int event_hist_trigger_named_init(struct event_trigger_data *data) { + int ret; + + ret = event_hist_trigger_init(data->named_data); + if (ret < 0) + return ret; + data->ref++; save_named_trigger(data->named_data->name, data); - return event_hist_trigger_init(data->named_data); + return 0; } static void event_hist_trigger_named_free(struct event_trigger_data *data) @@ -6564,12 +6571,6 @@ static int hist_register_trigger(char *glob, data->ops = &event_hist_trigger_named_ops; } - if (data->ops->init) { - ret = data->ops->init(data); - if (ret < 0) - goto out; - } - if (hist_data->enable_timestamps) { char *clock = hist_data->attrs->clock; @@ -6582,6 +6583,15 @@ static int hist_register_trigger(char *glob, tracing_set_filter_buffering(file->tr, true); } + if (data->ops->init) { + ret = data->ops->init(data); + if (ret < 0) { + if (hist_data->enable_timestamps) + tracing_set_filter_buffering(file->tr, false); + goto out; + } + } + if (named_data) { remove_hist_vars(hist_data); destroy_hist_data(hist_data); @@ -6599,11 +6609,12 @@ static int hist_trigger_enable(struct event_trigger_data *data, update_cond_flag(file); - if (trace_event_trigger_enable_disable(file, 1) < 0) { - list_del_rcu(&data->list); - update_cond_flag(file); + /* + * On failure the caller undoes the registration, and + * hist_unregister_trigger() can only find the trigger here. + */ + if (trace_event_trigger_enable_disable(file, 1) < 0) ret--; - } return ret; } @@ -6681,13 +6692,13 @@ static void hist_unregister_trigger(char *glob, } } - if (test && test->ops->free) - test->ops->free(test); - if (hist_data->enable_timestamps) { if (!hist_data->remove || test) tracing_set_filter_buffering(file->tr, false); } + + if (test && test->ops->free) + test->ops->free(test); } static bool hist_file_check_refs(struct trace_event_file *file) @@ -6892,6 +6903,8 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops, return ret; out_unreg: event_trigger_unregister(cmd_ops, file, glob+1, trigger_data); + /* The unregister frees trigger_data, skip out_free */ + goto out; out_free: event_trigger_reset_filter(cmd_ops, trigger_data); diff --git a/kernel/trace/trace_functions.c b/kernel/trace/trace_functions.c index 44ae51d1dc45..eecac19a2bcb 100644 --- a/kernel/trace/trace_functions.c +++ b/kernel/trace/trace_functions.c @@ -90,7 +90,7 @@ int ftrace_create_function_files(struct trace_array *tr, if (!tr->ops) return -EINVAL; - ftrace_create_filter_files(tr->ops, parent); + ftrace_create_filter_files(tr, parent); return 0; } diff --git a/kernel/trace/trace_probe.c b/kernel/trace/trace_probe.c index afae842009d1..a43d974fbccc 100644 --- a/kernel/trace/trace_probe.c +++ b/kernel/trace/trace_probe.c @@ -620,7 +620,7 @@ static int parse_btf_arg(char *varname, int i, is_ptr, ret; u32 tid; - if (WARN_ON_ONCE(!ctx->funcname)) + if (!ctx->funcname) return -EINVAL; is_ptr = split_next_field(varname, &field, ctx); diff --git a/kernel/trace/trace_stack.c b/kernel/trace/trace_stack.c index 5a48dba912ea..c6b2f622ffa1 100644 --- a/kernel/trace/trace_stack.c +++ b/kernel/trace/trace_stack.c @@ -499,7 +499,7 @@ stack_trace_filter_open(struct inode *inode, struct file *file) struct ftrace_ops *ops = inode->i_private; /* Checks for tracefs lockdown */ - return ftrace_regex_open(ops, FTRACE_ITER_FILTER, + return ftrace_regex_open(NULL, ops, FTRACE_ITER_FILTER, inode, file); } diff --git a/kernel/workqueue.c b/kernel/workqueue.c index c2de440b7589..66baaf1babe4 100644 --- a/kernel/workqueue.c +++ b/kernel/workqueue.c @@ -1692,7 +1692,7 @@ static bool is_chained_work(struct workqueue_struct *wq) * Return %true iff I'm a worker executing a work item on @wq. If * I'm @worker, it's safe to dereference it without locking. */ - return worker && worker->current_pwq->wq == wq; + return worker && worker->current_pwq && worker->current_pwq->wq == wq; } /* diff --git a/mm/damon/core.c b/mm/damon/core.c index fe91b296f4c6..b5b6d00d7a3b 100644 --- a/mm/damon/core.c +++ b/mm/damon/core.c @@ -871,49 +871,52 @@ static bool damos_valid_target(struct damon_ctx *c, struct damon_target *t, * This function checks if a given region should be skipped or not for the * reason. If only the starting part of the region has previously charged, * this function splits the region into two so that the second one covers the - * area that not charged in the previous charge widnow and saves the second - * region in *rp and returns false, so that the caller can apply DAMON action - * to the second one. + * area that not charged in the previous charge widnow, and return true. The + * caller can see the second one on the next iteration of the region walk. + * Note that this means the caller should use damon_for_each_region() instead + * of damon_for_each_region_safe(). If damon_for_each_region_safe() is used, + * the second region will just be ignored. * - * Return: true if the region should be entirely skipped, false otherwise. + * Return: true if the region should be skipped, false otherwise. */ static bool damos_skip_charged_region(struct damon_target *t, - struct damon_region **rp, struct damos *s) + struct damon_region *r, struct damos *s) { - struct damon_region *r = *rp; struct damos_quota *quota = &s->quota; unsigned long sz_to_skip; + bool skip = false; /* Skip previously charged regions */ if (quota->charge_target_from) { if (t != quota->charge_target_from) return true; - if (r == damon_last_region(t)) { - quota->charge_target_from = NULL; - quota->charge_addr_from = 0; - return true; - } if (quota->charge_addr_from && - r->ar.end <= quota->charge_addr_from) - return true; + r->ar.end <= quota->charge_addr_from) { + skip = true; + goto out; + } if (quota->charge_addr_from && r->ar.start < quota->charge_addr_from) { sz_to_skip = ALIGN_DOWN(quota->charge_addr_from - r->ar.start, DAMON_MIN_REGION); if (!sz_to_skip) { - if (damon_sz_region(r) <= DAMON_MIN_REGION) - return true; + if (damon_sz_region(r) <= DAMON_MIN_REGION) { + skip = true; + goto out; + } sz_to_skip = DAMON_MIN_REGION; } damon_split_region_at(t, r, sz_to_skip); - r = damon_next_region(r); - *rp = r; + skip = true; } + } +out: + if (r == damon_last_region(t)) { quota->charge_target_from = NULL; quota->charge_addr_from = 0; } - return false; + return skip; } static void damos_update_stat(struct damos *s, @@ -1045,7 +1048,7 @@ static void damon_do_apply_schemes(struct damon_ctx *c, if (quota->esz && quota->charged_sz >= quota->esz) continue; - if (damos_skip_charged_region(t, &r, s)) + if (damos_skip_charged_region(t, r, s)) continue; if (!damos_valid_target(c, t, r, s)) @@ -1087,7 +1090,7 @@ static void damos_adjust_quota(struct damon_ctx *c, struct damos *s) unsigned long cumulated_sz; unsigned int score, max_score = 0; - if (!quota->ms && !quota->sz) + if (!quota->esz && !quota->ms && !quota->sz) return; /* First charge window */ @@ -1134,7 +1137,7 @@ static void damos_adjust_quota(struct damon_ctx *c, struct damos *s) static void kdamond_apply_schemes(struct damon_ctx *c) { struct damon_target *t; - struct damon_region *r, *next_r; + struct damon_region *r; struct damos *s; unsigned long sample_interval = c->attrs.sample_interval ? c->attrs.sample_interval : 1; @@ -1156,7 +1159,7 @@ static void kdamond_apply_schemes(struct damon_ctx *c) return; damon_for_each_target(t, c) { - damon_for_each_region_safe(r, next_r, t) + damon_for_each_region(r, t) damon_do_apply_schemes(c, t, r); } diff --git a/mm/damon/vaddr.c b/mm/damon/vaddr.c index c74b3fd1d227..0a9812eb1047 100644 --- a/mm/damon/vaddr.c +++ b/mm/damon/vaddr.c @@ -347,6 +347,7 @@ static void damon_hugetlb_mkold(pte_t *pte, struct mm_struct *mm, if (pte_young(entry)) { referenced = true; + entry = huge_ptep_get_and_clear(mm, addr, pte, psize); entry = pte_mkold(entry); set_huge_pte_at(mm, addr, pte, entry, psize); } diff --git a/mm/filemap.c b/mm/filemap.c index 5a340cad44ed..7ad7202ad65e 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -2704,15 +2704,6 @@ ssize_t filemap_read(struct kiocb *iocb, struct iov_iter *iter, goto put_folios; end_offset = min_t(loff_t, isize, iocb->ki_pos + iter->count); - /* - * Pairs with a barrier in - * block_write_end()->mark_buffer_dirty() or other page - * dirtying routines like iomap_write_end() to ensure - * changes to page contents are visible before we see - * increased inode size. - */ - smp_rmb(); - /* * Once we start copying data, we don't want to be touching any * cachelines that might be contended: diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 8df057bbe1bc..28502c539cab 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -75,8 +75,25 @@ struct page *huge_zero_page __read_mostly; unsigned long huge_zero_pfn __read_mostly = ~0UL; bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, - bool smaps, bool in_pf, bool enforce_sysfs) -{ + unsigned long tva_flags, unsigned long orders) +{ + bool smaps = tva_flags & TVA_SMAPS; + bool in_pf = tva_flags & TVA_IN_PF; + bool enforce_sysfs = tva_flags & TVA_ENFORCE_SYSFS; + unsigned long supported_orders; + + /* Check the intersection of requested and supported orders. */ + if (vma_is_anonymous(vma)) + supported_orders = THP_ORDERS_ALL_ANON; + else if (vma_is_dax(vma)) + supported_orders = THP_ORDERS_ALL_FILE_DAX; + else + supported_orders = THP_ORDERS_ALL_FILE_DEFAULT; + + orders &= supported_orders; + if (!orders) + return 0; + if (!vma->vm_mm) /* vdso */ return false; @@ -88,11 +105,11 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, return in_pf; /* - * Special VMA and hugetlb VMA. + * khugepaged special VMA and hugetlb VMA. * Must be checked after dax since some dax mappings may have * VM_MIXEDMAP set. */ - if (vm_flags & VM_NO_KHUGEPAGED) + if (!in_pf && !smaps && (vm_flags & VM_NO_KHUGEPAGED)) return false; /* @@ -120,12 +137,18 @@ bool hugepage_vma_check(struct vm_area_struct *vma, unsigned long vm_flags, !hugepage_flags_always()))) return false; - /* Only regular file is valid */ - if (!in_pf && file_thp_enabled(vma)) - return true; - - if (!vma_is_anonymous(vma)) + if (!vma_is_anonymous(vma)) { + /* + * Trust that ->huge_fault() handlers know what they are doing + * in fault path. + */ + if (((in_pf || smaps)) && vma->vm_ops->huge_fault) + return true; + /* Only regular file is valid in collapse path */ + if (((!in_pf || smaps)) && file_thp_enabled(vma)) + return true; return false; + } if (vma_is_temporary_stack(vma)) return false; @@ -1714,6 +1737,9 @@ int zap_huge_pmd(struct mmu_gather *tlb, struct vm_area_struct *vma, if (arch_needs_pgtable_deposit()) zap_deposited_table(tlb->mm, pmd); add_mm_counter(tlb->mm, mm_counter_file(page), -HPAGE_PMD_NR); + + if (pmd_present(orig_pmd) && pmd_dirty(orig_pmd)) + folio_mark_dirty(page_folio(page)); } spin_unlock(ptl); @@ -1749,6 +1775,16 @@ static pmd_t move_soft_dirty_pmd(pmd_t pmd) return pmd; } +static pmd_t clear_uffd_wp_pmd(pmd_t pmd) +{ + if (pmd_present(pmd)) + pmd = pmd_clear_uffd_wp(pmd); + else if (is_swap_pmd(pmd)) + pmd = pmd_swp_clear_uffd_wp(pmd); + + return pmd; +} + bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, unsigned long new_addr, pmd_t *old_pmd, pmd_t *new_pmd) { @@ -1787,6 +1823,8 @@ bool move_huge_pmd(struct vm_area_struct *vma, unsigned long old_addr, pgtable_trans_huge_deposit(mm, new_pmd, pgtable); } pmd = move_soft_dirty_pmd(pmd); + if (vma_has_uffd_without_event_remap(vma)) + pmd = clear_uffd_wp_pmd(pmd); set_pmd_at(mm, new_addr, new_pmd, pmd); if (force_flush) flush_pmd_tlb_range(vma, old_addr, old_addr + PMD_SIZE); diff --git a/mm/hugetlb.c b/mm/hugetlb.c index bd8752a5356b..8bfaaa84bb3f 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -1398,8 +1398,7 @@ static unsigned long available_huge_pages(struct hstate *h) static struct folio *dequeue_hugetlb_folio_vma(struct hstate *h, struct vm_area_struct *vma, - unsigned long address, int avoid_reserve, - long chg) + unsigned long address, long chg) { struct folio *folio = NULL; struct mempolicy *mpol; @@ -1415,10 +1414,6 @@ static struct folio *dequeue_hugetlb_folio_vma(struct hstate *h, if (!vma_has_reserves(vma, chg) && !available_huge_pages(h)) goto err; - /* If reserves cannot be used, ensure enough pages are in the pool */ - if (avoid_reserve && !available_huge_pages(h)) - goto err; - gfp_mask = htlb_alloc_mask(h); nid = huge_node(vma, address, gfp_mask, &mpol, &nodemask); @@ -1434,7 +1429,7 @@ static struct folio *dequeue_hugetlb_folio_vma(struct hstate *h, folio = dequeue_hugetlb_folio_nodemask(h, gfp_mask, nid, nodemask); - if (folio && !avoid_reserve && vma_has_reserves(vma, chg)) { + if (folio && vma_has_reserves(vma, chg)) { folio_set_hugetlb_restore_reserve(folio); h->resv_huge_pages--; } @@ -3063,6 +3058,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, long gbl_chg; int ret, idx; struct hugetlb_cgroup *h_cg = NULL; + struct hugetlb_cgroup *h_cg_rsvd = NULL; bool deferred_reserve; idx = hstate_index(h); @@ -3088,17 +3084,6 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, vma_end_reservation(h, vma, addr); return ERR_PTR(-ENOSPC); } - - /* - * Even though there was no reservation in the region/reserve - * map, there could be reservations associated with the - * subpool that can be used. This would be indicated if the - * return value of hugepage_subpool_get_pages() is zero. - * However, if avoid_reserve is specified we still avoid even - * the subpool reservations. - */ - if (avoid_reserve) - gbl_chg = 1; } /* If this allocation is not consuming a reservation, charge it now. @@ -3106,7 +3091,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, deferred_reserve = map_chg || avoid_reserve; if (deferred_reserve) { ret = hugetlb_cgroup_charge_cgroup_rsvd( - idx, pages_per_huge_page(h), &h_cg); + idx, pages_per_huge_page(h), &h_cg_rsvd); if (ret) goto out_subpool_put; } @@ -3121,7 +3106,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, * from the global free pool (global change). gbl_chg == 0 indicates * a reservation exists for the allocation. */ - folio = dequeue_hugetlb_folio_vma(h, vma, addr, avoid_reserve, gbl_chg); + folio = dequeue_hugetlb_folio_vma(h, vma, addr, gbl_chg); if (!folio) { spin_unlock_irq(&hugetlb_lock); folio = alloc_buddy_hugetlb_folio_with_mpol(h, vma, addr); @@ -3143,7 +3128,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, */ if (deferred_reserve) { hugetlb_cgroup_commit_charge_rsvd(idx, pages_per_huge_page(h), - h_cg, folio); + h_cg_rsvd, folio); } spin_unlock_irq(&hugetlb_lock); @@ -3179,7 +3164,7 @@ struct folio *alloc_hugetlb_folio(struct vm_area_struct *vma, out_uncharge_cgroup_reservation: if (deferred_reserve) hugetlb_cgroup_uncharge_cgroup_rsvd(idx, pages_per_huge_page(h), - h_cg); + h_cg_rsvd); out_subpool_put: if (map_chg || avoid_reserve) hugepage_subpool_put_pages(spool, 1); @@ -5241,6 +5226,7 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, unsigned long new_addr, pte_t *src_pte, pte_t *dst_pte, unsigned long sz) { + bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma); struct hstate *h = hstate_vma(vma); struct mm_struct *mm = vma->vm_mm; spinlock_t *src_ptl, *dst_ptl; @@ -5257,7 +5243,18 @@ static void move_huge_pte(struct vm_area_struct *vma, unsigned long old_addr, spin_lock_nested(src_ptl, SINGLE_DEPTH_NESTING); pte = huge_ptep_get_and_clear(mm, old_addr, src_pte, sz); - set_huge_pte_at(mm, new_addr, dst_pte, pte, sz); + + if (need_clear_uffd_wp && pte_marker_uffd_wp(pte)) + huge_pte_clear(mm, new_addr, dst_pte, sz); + else { + if (need_clear_uffd_wp) { + if (pte_present(pte)) + pte = huge_pte_clear_uffd_wp(pte); + else if (is_swap_pte(pte)) + pte = pte_swp_clear_uffd_wp(pte); + } + set_huge_pte_at(mm, new_addr, dst_pte, pte, sz); + } if (src_ptl != dst_ptl) spin_unlock(src_ptl); @@ -5295,18 +5292,21 @@ int move_hugetlb_page_tables(struct vm_area_struct *vma, hugetlb_vma_lock_write(vma); i_mmap_lock_write(mapping); for (; old_addr < old_end; old_addr += sz, new_addr += sz) { + const unsigned long offset_to_last_entry = + (old_addr | last_addr_mask) - old_addr; + src_pte = hugetlb_walk(vma, old_addr, sz); if (!src_pte) { - old_addr |= last_addr_mask; - new_addr |= last_addr_mask; + old_addr += offset_to_last_entry; + new_addr += offset_to_last_entry; continue; } if (huge_pte_none(huge_ptep_get(src_pte))) continue; if (huge_pmd_unshare(&tlb, vma, old_addr, src_pte)) { - old_addr |= last_addr_mask; - new_addr |= last_addr_mask; + old_addr += offset_to_last_entry; + new_addr += offset_to_last_entry; continue; } diff --git a/mm/kasan/shadow.c b/mm/kasan/shadow.c index d687f09a7ae3..d4e568ecd183 100644 --- a/mm/kasan/shadow.c +++ b/mm/kasan/shadow.c @@ -311,34 +311,117 @@ void __init __weak kasan_populate_early_vm_area_shadow(void *start, { } +struct vmalloc_populate_data { + unsigned long start; + struct page **pages; +}; + static int kasan_populate_vmalloc_pte(pte_t *ptep, unsigned long addr, - void *unused) + void *_data) { - unsigned long page; + struct vmalloc_populate_data *data = _data; + struct page *page; pte_t pte; + int index; if (likely(!pte_none(ptep_get(ptep)))) return 0; - page = __get_free_page(GFP_KERNEL); - if (!page) - return -ENOMEM; - - __memset((void *)page, KASAN_VMALLOC_INVALID, PAGE_SIZE); - pte = pfn_pte(PFN_DOWN(__pa(page)), PAGE_KERNEL); + index = PFN_DOWN(addr - data->start); + page = data->pages[index]; + __memset(page_to_virt(page), KASAN_VMALLOC_INVALID, PAGE_SIZE); + pte = pfn_pte(page_to_pfn(page), PAGE_KERNEL); spin_lock(&init_mm.page_table_lock); if (likely(pte_none(ptep_get(ptep)))) { set_pte_at(&init_mm, addr, ptep, pte); - page = 0; + data->pages[index] = NULL; } spin_unlock(&init_mm.page_table_lock); - if (page) - free_page(page); + + return 0; +} + +static void ___free_pages_bulk(struct page **pages, int nr_pages) +{ + int i; + + for (i = 0; i < nr_pages; i++) { + if (pages[i]) { + __free_pages(pages[i], 0); + pages[i] = NULL; + } + } +} + +static int ___alloc_pages_bulk(struct page **pages, int nr_pages, gfp_t gfp_mask) +{ + unsigned long nr_populated, nr_total = nr_pages; + struct page **page_array = pages; + + while (nr_pages) { + nr_populated = alloc_pages_bulk_array(gfp_mask, nr_pages, pages); + if (!nr_populated) { + ___free_pages_bulk(page_array, nr_total - nr_pages); + return -ENOMEM; + } + pages += nr_populated; + nr_pages -= nr_populated; + } + return 0; } -int kasan_populate_vmalloc(unsigned long addr, unsigned long size) +static int __kasan_populate_vmalloc(unsigned long start, unsigned long end, gfp_t gfp_mask) +{ + unsigned long nr_pages, nr_total = PFN_UP(end - start); + struct vmalloc_populate_data data; + unsigned int flags; + int ret = 0; + + data.pages = (struct page **)__get_free_page(gfp_mask | __GFP_ZERO); + if (!data.pages) + return -ENOMEM; + + while (nr_total) { + nr_pages = min(nr_total, PAGE_SIZE / sizeof(data.pages[0])); + ret = ___alloc_pages_bulk(data.pages, nr_pages, gfp_mask); + if (ret) + break; + + data.start = start; + + /* + * page tables allocations ignore external gfp mask, enforce it + * by the scope API + */ + if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) + flags = memalloc_nofs_save(); + else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == 0) + flags = memalloc_noio_save(); + + ret = apply_to_page_range(&init_mm, start, nr_pages * PAGE_SIZE, + kasan_populate_vmalloc_pte, &data); + + if ((gfp_mask & (__GFP_FS | __GFP_IO)) == __GFP_IO) + memalloc_nofs_restore(flags); + else if ((gfp_mask & (__GFP_FS | __GFP_IO)) == 0) + memalloc_noio_restore(flags); + + ___free_pages_bulk(data.pages, nr_pages); + if (ret) + break; + + start += nr_pages * PAGE_SIZE; + nr_total -= nr_pages; + } + + free_page((unsigned long)data.pages); + + return ret; +} + +int kasan_populate_vmalloc(unsigned long addr, unsigned long size, gfp_t gfp_mask) { unsigned long shadow_start, shadow_end; int ret; @@ -367,9 +450,7 @@ int kasan_populate_vmalloc(unsigned long addr, unsigned long size) shadow_start = PAGE_ALIGN_DOWN(shadow_start); shadow_end = PAGE_ALIGN(shadow_end); - ret = apply_to_page_range(&init_mm, shadow_start, - shadow_end - shadow_start, - kasan_populate_vmalloc_pte, NULL); + ret = __kasan_populate_vmalloc(shadow_start, shadow_end, gfp_mask); if (ret) return ret; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 16d29ee602c7..ead6b0001524 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -446,7 +446,8 @@ void khugepaged_enter_vma(struct vm_area_struct *vma, { if (!test_bit(MMF_VM_HUGEPAGE, &vma->vm_mm->flags) && hugepage_flags_enabled()) { - if (hugepage_vma_check(vma, vm_flags, false, false, true)) + if (hugepage_vma_check(vma, vm_flags, TVA_ENFORCE_SYSFS, + BIT(PMD_ORDER))) __khugepaged_enter(vma->vm_mm); } } @@ -899,6 +900,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, struct collapse_control *cc) { struct vm_area_struct *vma; + unsigned long tva_flags = cc->is_khugepaged ? TVA_ENFORCE_SYSFS : 0; if (unlikely(hpage_collapse_test_exit(mm))) return SCAN_ANY_PROCESS; @@ -909,8 +911,7 @@ static int hugepage_vma_revalidate(struct mm_struct *mm, unsigned long address, if (!transhuge_vma_suitable(vma, address)) return SCAN_ADDRESS_RANGE; - if (!hugepage_vma_check(vma, vma->vm_flags, false, false, - cc->is_khugepaged)) + if (!hugepage_vma_check(vma, vma->vm_flags, tva_flags, BIT(PMD_ORDER))) return SCAN_VMA_CHECK; /* * Anon VMA expected, the address may be unmapped then @@ -1492,7 +1493,7 @@ int collapse_pte_mapped_thp(struct mm_struct *mm, unsigned long addr, * and map it by a PMD, regardless of sysfs THP settings. As such, let's * analogously elide sysfs THP settings here. */ - if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false)) + if (!hugepage_vma_check(vma, vma->vm_flags, 0, BIT(PMD_ORDER))) return SCAN_VMA_CHECK; /* Keep pmd pgtable for uffd-wp; see comment in retract_page_tables() */ @@ -2364,7 +2365,8 @@ static unsigned int khugepaged_scan_mm_slot(unsigned int pages, int *result, progress++; break; } - if (!hugepage_vma_check(vma, vma->vm_flags, false, false, true)) { + if (!hugepage_vma_check(vma, vma->vm_flags, + TVA_ENFORCE_SYSFS, BIT(PMD_ORDER))) { skip: progress++; continue; @@ -2701,7 +2703,7 @@ int madvise_collapse(struct vm_area_struct *vma, struct vm_area_struct **prev, *prev = vma; - if (!hugepage_vma_check(vma, vma->vm_flags, false, false, false)) + if (!hugepage_vma_check(vma, vma->vm_flags, 0, BIT(PMD_ORDER))) return -EINVAL; cc = kmalloc(sizeof(*cc), GFP_KERNEL); diff --git a/mm/kmemleak.c b/mm/kmemleak.c index e2e41de55c02..fceb28690f2f 100644 --- a/mm/kmemleak.c +++ b/mm/kmemleak.c @@ -1516,6 +1516,42 @@ static void kmemleak_cond_resched(struct kmemleak_object *object) put_object(object); } +/* + * Scan all task kernel stacks, rescheduling between tasks. Each task is looked + * up and pinned within its own RCU read-side section, so no lock is held across + * the scan and the walk cannot trip the soft lockup watchdog. + */ +static void kmemleak_scan_task_stacks(void) +{ + struct pid *pid; + int nr = 1; + + do { + struct task_struct *p = NULL; + + rcu_read_lock(); + pid = find_ge_pid(nr, &init_pid_ns); + if (pid) { + nr = pid_nr(pid) + 1; + p = pid_task(pid, PIDTYPE_PID); + if (p) + get_task_struct(p); + } + rcu_read_unlock(); + + if (p) { + void *stack = try_get_task_stack(p); + + if (stack) { + scan_block(stack, stack + THREAD_SIZE, NULL); + put_task_stack(p); + } + put_task_struct(p); + } + cond_resched(); + } while (pid && !scan_should_stop()); +} + /* * Scan data sections and all the referenced memory blocks allocated via the * kernel's standard allocators. This function must be called with the @@ -1607,19 +1643,8 @@ static void kmemleak_scan(void) /* * Scanning the task stacks (may introduce false negatives). */ - if (kmemleak_stack_scan) { - struct task_struct *p, *g; - - rcu_read_lock(); - for_each_process_thread(g, p) { - void *stack = try_get_task_stack(p); - if (stack) { - scan_block(stack, stack + THREAD_SIZE, NULL); - put_task_stack(p); - } - } - rcu_read_unlock(); - } + if (kmemleak_stack_scan) + kmemleak_scan_task_stacks(); /* * Scan the objects already referenced from the sections scanned diff --git a/mm/list_lru.c b/mm/list_lru.c index a05e5bef3b40..fcca67ac26ec 100644 --- a/mm/list_lru.c +++ b/mm/list_lru.c @@ -116,21 +116,19 @@ list_lru_from_kmem(struct list_lru *lru, int nid, void *ptr, } #endif /* CONFIG_MEMCG_KMEM */ -bool list_lru_add(struct list_lru *lru, struct list_head *item) +bool list_lru_add(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg) { - int nid = page_to_nid(virt_to_page(item)); struct list_lru_node *nlru = &lru->node[nid]; - struct mem_cgroup *memcg; struct list_lru_one *l; spin_lock(&nlru->lock); if (list_empty(item)) { - l = list_lru_from_kmem(lru, nid, item, &memcg); + l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); list_add_tail(item, &l->list); /* Set shrinker bit if the first element was added */ if (!l->nr_items++) - set_shrinker_bit(memcg, nid, - lru_shrinker_id(lru)); + set_shrinker_bit(memcg, nid, lru_shrinker_id(lru)); nlru->nr_items++; spin_unlock(&nlru->lock); return true; @@ -140,15 +138,25 @@ bool list_lru_add(struct list_lru *lru, struct list_head *item) } EXPORT_SYMBOL_GPL(list_lru_add); -bool list_lru_del(struct list_lru *lru, struct list_head *item) +bool list_lru_add_obj(struct list_lru *lru, struct list_head *item) { int nid = page_to_nid(virt_to_page(item)); + struct mem_cgroup *memcg = list_lru_memcg_aware(lru) ? + mem_cgroup_from_slab_obj(item) : NULL; + + return list_lru_add(lru, item, nid, memcg); +} +EXPORT_SYMBOL_GPL(list_lru_add_obj); + +bool list_lru_del(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg) +{ struct list_lru_node *nlru = &lru->node[nid]; struct list_lru_one *l; spin_lock(&nlru->lock); if (!list_empty(item)) { - l = list_lru_from_kmem(lru, nid, item, NULL); + l = list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); list_del_init(item); l->nr_items--; nlru->nr_items--; @@ -160,6 +168,16 @@ bool list_lru_del(struct list_lru *lru, struct list_head *item) } EXPORT_SYMBOL_GPL(list_lru_del); +bool list_lru_del_obj(struct list_lru *lru, struct list_head *item) +{ + int nid = page_to_nid(virt_to_page(item)); + struct mem_cgroup *memcg = list_lru_memcg_aware(lru) ? + mem_cgroup_from_slab_obj(item) : NULL; + + return list_lru_del(lru, item, nid, memcg); +} +EXPORT_SYMBOL_GPL(list_lru_del_obj); + void list_lru_isolate(struct list_lru_one *list, struct list_head *item) { list_del_init(item); @@ -175,6 +193,20 @@ void list_lru_isolate_move(struct list_lru_one *list, struct list_head *item, } EXPORT_SYMBOL_GPL(list_lru_isolate_move); +void list_lru_putback(struct list_lru *lru, struct list_head *item, int nid, + struct mem_cgroup *memcg) +{ + struct list_lru_one *list = + list_lru_from_memcg_idx(lru, nid, memcg_kmem_id(memcg)); + + if (list_empty(item)) { + list_add_tail(item, &list->list); + if (!list->nr_items++) + set_shrinker_bit(memcg, nid, lru_shrinker_id(lru)); + } +} +EXPORT_SYMBOL_GPL(list_lru_putback); + unsigned long list_lru_count_one(struct list_lru *lru, int nid, struct mem_cgroup *memcg) { diff --git a/mm/memcontrol.c b/mm/memcontrol.c index 5c654a395fff..18b5b8cd7ed2 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -715,7 +715,7 @@ static inline void memcg_rstat_updated(struct mem_cgroup *memcg, int val) if (!val) return; - cgroup_rstat_updated(memcg->css.cgroup, cpu); + __cgroup_rstat_updated(memcg->css.cgroup, cpu); statc = this_cpu_ptr(memcg->vmstats_percpu); for (; statc; statc = statc->parent) { stats_updates = READ_ONCE(statc->stats_updates) + abs(val); @@ -3721,9 +3721,32 @@ enum { RES_LIMIT, RES_MAX_USAGE, RES_FAILCNT, - RES_SOFT_LIMIT, }; +static u64 mem_cgroup_soft_limit_read(struct cgroup_subsys_state *css, + struct cftype *cft) +{ + return (u64)PAGE_COUNTER_MAX * PAGE_SIZE; +} + +static ssize_t mem_cgroup_soft_limit_write(struct kernfs_open_file *of, + char *buf, size_t nbytes, loff_t off) +{ + unsigned long nr_pages; + int ret; + + ret = page_counter_memparse(strstrip(buf), "-1", &nr_pages); + if (ret) + return ret; + + pr_warn_once("soft_limit_in_bytes is deprecated and will be removed. " + "Writing any value to this file has no effect. " + "Please report your usecase to linux-mm@kvack.org if you " + "depend on this functionality.\n"); + + return nbytes; +} + static u64 mem_cgroup_read_u64(struct cgroup_subsys_state *css, struct cftype *cft) { @@ -3760,8 +3783,6 @@ static u64 mem_cgroup_read_u64(struct cgroup_subsys_state *css, return (u64)counter->watermark * PAGE_SIZE; case RES_FAILCNT: return counter->failcnt; - case RES_SOFT_LIMIT: - return (u64)READ_ONCE(memcg->soft_limit) * PAGE_SIZE; default: BUG(); } @@ -3912,14 +3933,6 @@ static ssize_t mem_cgroup_write(struct kernfs_open_file *of, break; } break; - case RES_SOFT_LIMIT: - if (IS_ENABLED(CONFIG_PREEMPT_RT)) { - ret = -EOPNOTSUPP; - } else { - WRITE_ONCE(memcg->soft_limit, nr_pages); - ret = 0; - } - break; } return ret ?: nbytes; } @@ -4260,7 +4273,7 @@ static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap) * only one element of the array here. */ for (; i >= 0 && unlikely(t->entries[i].threshold > usage); i--) - eventfd_signal(t->entries[i].eventfd, 1); + eventfd_signal(t->entries[i].eventfd); /* i = current_threshold + 1 */ i++; @@ -4272,7 +4285,7 @@ static void __mem_cgroup_threshold(struct mem_cgroup *memcg, bool swap) * only one element of the array here. */ for (; i < t->size && unlikely(t->entries[i].threshold <= usage); i++) - eventfd_signal(t->entries[i].eventfd, 1); + eventfd_signal(t->entries[i].eventfd); /* Update current_threshold */ t->current_threshold = i - 1; @@ -4312,7 +4325,7 @@ static int mem_cgroup_oom_notify_cb(struct mem_cgroup *memcg) spin_lock(&memcg_oom_lock); list_for_each_entry(ev, &memcg->oom_notify, list) - eventfd_signal(ev->eventfd, 1); + eventfd_signal(ev->eventfd); spin_unlock(&memcg_oom_lock); return 0; @@ -4531,7 +4544,7 @@ static int mem_cgroup_oom_register_event(struct mem_cgroup *memcg, /* already in OOM ? */ if (memcg->under_oom) - eventfd_signal(eventfd, 1); + eventfd_signal(eventfd); spin_unlock(&memcg_oom_lock); return 0; @@ -4823,7 +4836,7 @@ static void memcg_event_remove(struct work_struct *work) event->unregister_event(memcg, event->eventfd); /* Notify userspace the event is going away. */ - eventfd_signal(event->eventfd, 1); + eventfd_signal(event->eventfd); eventfd_ctx_put(event->eventfd); kfree(event); @@ -5067,9 +5080,8 @@ static struct cftype mem_cgroup_legacy_files[] = { }, { .name = "soft_limit_in_bytes", - .private = MEMFILE_PRIVATE(_MEM, RES_SOFT_LIMIT), - .write = mem_cgroup_write, - .read_u64 = mem_cgroup_read_u64, + .write = mem_cgroup_soft_limit_write, + .read_u64 = mem_cgroup_soft_limit_read, }, { .name = "failcnt", diff --git a/mm/memory.c b/mm/memory.c index 2157c5c9139f..a64919cf0fb0 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -5121,7 +5121,8 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, return VM_FAULT_OOM; retry_pud: if (pud_none(*vmf.pud) && - hugepage_vma_check(vma, vm_flags, false, true, true)) { + hugepage_vma_check(vma, vm_flags, + TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PUD_ORDER))) { ret = create_huge_pud(&vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; @@ -5155,7 +5156,8 @@ static vm_fault_t __handle_mm_fault(struct vm_area_struct *vma, goto retry_pud; if (pmd_none(*vmf.pmd) && - hugepage_vma_check(vma, vm_flags, false, true, true)) { + hugepage_vma_check(vma, vm_flags, + TVA_IN_PF | TVA_ENFORCE_SYSFS, BIT(PMD_ORDER))) { ret = create_huge_pmd(&vmf); if (!(ret & VM_FAULT_FALLBACK)) return ret; diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 4498855daed8..08444c40ed65 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1044,7 +1044,7 @@ static inline struct zone *default_zone_for_pfn(int nid, unsigned long start_pfn return movable_node_enabled ? movable_zone : kernel_zone; } -struct zone *zone_for_pfn_range(int online_type, int nid, +struct zone *zone_for_pfn_range(enum mmop online_type, int nid, struct memory_group *group, unsigned long start_pfn, unsigned long nr_pages) { @@ -1237,7 +1237,8 @@ static pg_data_t __ref *hotadd_init_pgdat(int nid) pgdat = NODE_DATA(nid); /* init node's zones as empty zones, we don't have any present pages.*/ - free_area_init_core_hotplug(pgdat); + if (free_area_init_core_hotplug(pgdat)) + return NULL; /* * The node we allocated has no zone fallback lists. For avoiding @@ -2267,7 +2268,7 @@ EXPORT_SYMBOL_GPL(remove_memory); static int try_offline_memory_block(struct memory_block *mem, void *arg) { - uint8_t online_type = MMOP_ONLINE_KERNEL; + enum mmop online_type = MMOP_ONLINE_KERNEL; uint8_t **online_types = arg; struct page *page; int rc; @@ -2300,7 +2301,7 @@ static int try_reonline_memory_block(struct memory_block *mem, void *arg) int rc; if (**online_types != MMOP_OFFLINE) { - mem->online_type = **online_types; + mem->online_type = (enum mmop)**online_types; rc = device_online(&mem->dev); if (rc < 0) pr_warn("%s: Failed to re-online memory: %d", diff --git a/mm/migrate_device.c b/mm/migrate_device.c index d4ee7ab735a3..1ec278a529b4 100644 --- a/mm/migrate_device.c +++ b/mm/migrate_device.c @@ -745,6 +745,12 @@ static void __migrate_device_pages(unsigned long *src_pfns, src_pfns[i] &= ~MIGRATE_PFN_MIGRATE; continue; } + + /* + * folio_free_swap() removed the folio from the swap + * cache. Refresh the saved mapping before migration. + */ + mapping = folio_mapping(folio); } } else if (is_zone_device_page(newpage)) { /* diff --git a/mm/mm_init.c b/mm/mm_init.c index 45c5a7ca9609..27451dfc2fc0 100644 --- a/mm/mm_init.c +++ b/mm/mm_init.c @@ -1509,7 +1509,7 @@ void __init set_pageblock_order(void) * NOTE: this function is only called during memory hotplug */ #ifdef CONFIG_MEMORY_HOTPLUG -void __ref free_area_init_core_hotplug(struct pglist_data *pgdat) +int __ref free_area_init_core_hotplug(struct pglist_data *pgdat) { int nid = pgdat->node_id; enum zone_type z; @@ -1517,8 +1517,14 @@ void __ref free_area_init_core_hotplug(struct pglist_data *pgdat) pgdat_init_internals(pgdat); - if (pgdat->per_cpu_nodestats == &boot_nodestats) - pgdat->per_cpu_nodestats = alloc_percpu(struct per_cpu_nodestat); + if (pgdat->per_cpu_nodestats == &boot_nodestats) { + struct per_cpu_nodestat __percpu *p; + + p = alloc_percpu(struct per_cpu_nodestat); + if (!p) + return -ENOMEM; + pgdat->per_cpu_nodestats = p; + } /* * Reset the nr_zones, order and highest_zoneidx before reuse. @@ -1556,6 +1562,8 @@ void __ref free_area_init_core_hotplug(struct pglist_data *pgdat) zone->present_pages = 0; zone_init_internals(zone, z, nid, 0); } + + return 0; } #endif diff --git a/mm/mmap.c b/mm/mmap.c index 8cf23a07ae50..4fb3f7156dc3 100644 --- a/mm/mmap.c +++ b/mm/mmap.c @@ -1081,6 +1081,13 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct * * acceptable for merging, so we can do all of this optimistically. But * we do that READ_ONCE() to make sure that we never re-load the pointer. * + * The READ_ONCE() establishes an address dependency between anon_vma and + * any access to its fields, which pairs with the assignment to + * vma->anon_vma performed with release semantics in __anon_vma_prepare(). + * + * This is especially important as anon_vma's are SLAB_TYPESAFE_BY_RCU so + * accessing an uninitialised anon_vma's fields may result in a UAF. + * * IOW: that the "list_is_singular()" test on the anon_vma_chain only * matters for the 'stable anon_vma' case (ie the thing we want to avoid * is to return an anon_vma that is "complex" due to having gone through @@ -1093,6 +1100,7 @@ static int anon_vma_compatible(struct vm_area_struct *a, struct vm_area_struct * static struct anon_vma *reusable_anon_vma(struct vm_area_struct *old, struct vm_area_struct *a, struct vm_area_struct *b) { if (anon_vma_compatible(a, b)) { + /* Paired with a memory barrier in __anon_vma_prepare(). */ struct anon_vma *anon_vma = READ_ONCE(old->anon_vma); if (anon_vma && list_is_singular(&old->anon_vma_chain)) @@ -3383,13 +3391,14 @@ int insert_vm_struct(struct mm_struct *mm, struct vm_area_struct *vma) */ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, unsigned long addr, unsigned long len, pgoff_t pgoff, - bool *need_rmap_locks) + bool *need_rmap_locks, bool keep_source) { struct vm_area_struct *vma = *vmap; unsigned long vma_start = vma->vm_start; struct mm_struct *mm = vma->vm_mm; struct vm_area_struct *new_vma, *prev; bool faulted_in_anon_vma = true; + bool can_self_merge = false; VMA_ITERATOR(vmi, mm, addr); /* @@ -3399,15 +3408,29 @@ struct vm_area_struct *copy_vma(struct vm_area_struct **vmap, if (unlikely(vma_is_anonymous(vma) && !vma->anon_vma)) { pgoff = addr >> PAGE_SHIFT; faulted_in_anon_vma = false; + can_self_merge = true; } new_vma = find_vma_prev(mm, addr, &prev); if (new_vma && new_vma->vm_start < addr + len) return NULL; /* should never get here */ - new_vma = vma_merge(&vmi, mm, prev, addr, addr + len, vma->vm_flags, - vma->anon_vma, vma->vm_file, pgoff, vma_policy(vma), - vma->vm_userfaultfd_ctx, anon_vma_name(vma)); + /* + * If the original VMA is kept (MREMAP_DONTUNMAP), the source and + * destination VMA must be treated distinctly. + * + * A merge violates this, so in this case disallow a self-merge. A + * self-merge can only occur if the new range is immediately adjacent to + * the source VMA, so simply do not attempt to merge in that case. + */ + if (can_self_merge && keep_source && + (addr == vma->vm_end || vma->vm_start == addr + len)) + new_vma = NULL; + else + new_vma = vma_merge(&vmi, mm, prev, addr, addr + len, + vma->vm_flags, vma->anon_vma, vma->vm_file, + pgoff, vma_policy(vma), + vma->vm_userfaultfd_ctx, anon_vma_name(vma)); if (new_vma) { /* * Source vma may have been merged into new_vma diff --git a/mm/mremap.c b/mm/mremap.c index d458ef218a37..0739a31b54ee 100644 --- a/mm/mremap.c +++ b/mm/mremap.c @@ -138,6 +138,7 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, struct vm_area_struct *new_vma, pmd_t *new_pmd, unsigned long new_addr, bool need_rmap_locks) { + bool need_clear_uffd_wp = vma_has_uffd_without_event_remap(vma); struct mm_struct *mm = vma->vm_mm; pte_t *old_pte, *new_pte, pte; spinlock_t *old_ptl, *new_ptl; @@ -207,7 +208,18 @@ static int move_ptes(struct vm_area_struct *vma, pmd_t *old_pmd, force_flush = true; pte = move_pte(pte, new_vma->vm_page_prot, old_addr, new_addr); pte = move_soft_dirty_pte(pte); - set_pte_at(mm, new_addr, new_pte, pte); + + if (need_clear_uffd_wp && pte_marker_uffd_wp(pte)) + pte_clear(mm, new_addr, new_pte); + else { + if (need_clear_uffd_wp) { + if (pte_present(pte)) + pte = pte_clear_uffd_wp(pte); + else if (is_swap_pte(pte)) + pte = pte_swp_clear_uffd_wp(pte); + } + set_pte_at(mm, new_addr, new_pte, pte); + } } arch_leave_lazy_mmu_mode(); @@ -269,6 +281,15 @@ static bool move_normal_pmd(struct vm_area_struct *vma, unsigned long old_addr, if (WARN_ON_ONCE(!pmd_none(*new_pmd))) return false; + /* If this pmd belongs to a uffd vma with remap events disabled, we need + * to ensure that the uffd-wp state is cleared from all pgtables. This + * means recursing into lower page tables in move_page_tables(), and we + * can reuse the existing code if we simply treat the entry as "not + * moved". + */ + if (vma_has_uffd_without_event_remap(vma)) + return false; + /* * We don't have to worry about the ordering of src and dst * ptlocks because exclusive mmap_lock prevents deadlock. @@ -324,6 +345,15 @@ static bool move_normal_pud(struct vm_area_struct *vma, unsigned long old_addr, if (WARN_ON_ONCE(!pud_none(*new_pud))) return false; + /* If this pud belongs to a uffd vma with remap events disabled, we need + * to ensure that the uffd-wp state is cleared from all pgtables. This + * means recursing into lower page tables in move_page_tables(), and we + * can reuse the existing code if we simply treat the entry as "not + * moved". + */ + if (vma_has_uffd_without_event_remap(vma)) + return false; + /* * We don't have to worry about the ordering of src and dst * ptlocks because exclusive mmap_lock prevents deadlock. @@ -605,6 +635,8 @@ static unsigned long move_vma(struct vm_area_struct *vma, int err = 0; bool need_rmap_locks; struct vma_iterator vmi; + const bool split_before = vma->vm_start != old_addr; + const bool split_after = vma->vm_end != old_addr + old_len; /* * We'd prefer to avoid failure later on in do_munmap: @@ -617,9 +649,9 @@ static unsigned long move_vma(struct vm_area_struct *vma, to_account = new_len; if (vma->vm_ops && vma->vm_ops->may_split) { - if (vma->vm_start != old_addr) + if (split_before) err = vma->vm_ops->may_split(vma, old_addr); - if (!err && vma->vm_end != old_addr + old_len) + if (!err && split_after) err = vma->vm_ops->may_split(vma, old_addr + old_len); if (err) return err; @@ -637,6 +669,21 @@ static unsigned long move_vma(struct vm_area_struct *vma, if (err) return err; + /* + * To account mlock()'d pages correctly in the MREMAP_DONTUNMAP + * case perform any split ahead of time for an mlock()'d VMA. + */ + if ((flags & MREMAP_DONTUNMAP) && (vma->vm_flags & VM_LOCKED)) { + vma_iter_init(&vmi, mm, old_addr); + + if (split_before) + err = split_vma(&vmi, vma, old_addr, 1); + if (!err && split_after) + err = split_vma(&vmi, vma, old_addr + old_len, 0); + if (err) + return err; + } + if (vm_flags & VM_ACCOUNT) { if (security_vm_enough_memory_mm(mm, to_account >> PAGE_SHIFT)) return -ENOMEM; @@ -645,7 +692,7 @@ static unsigned long move_vma(struct vm_area_struct *vma, vma_start_write(vma); new_pgoff = vma->vm_pgoff + ((old_addr - vma->vm_start) >> PAGE_SHIFT); new_vma = copy_vma(&vma, new_addr, new_len, new_pgoff, - &need_rmap_locks); + &need_rmap_locks, flags & MREMAP_DONTUNMAP); if (!new_vma) { if (vm_flags & VM_ACCOUNT) vm_unacct_memory(to_account >> PAGE_SHIFT); @@ -706,6 +753,9 @@ static unsigned long move_vma(struct vm_area_struct *vma, untrack_pfn_clear(vma); if (unlikely(!err && (flags & MREMAP_DONTUNMAP))) { + /* Self-merge is disallowed. */ + VM_WARN_ON_ONCE(new_vma == vma); + /* We always clear VM_LOCKED[ONFAULT] on the old vma */ vm_flags_clear(vma, VM_LOCKED_MASK); @@ -713,9 +763,19 @@ static unsigned long move_vma(struct vm_area_struct *vma, * anon_vma links of the old vma is no longer needed after its page * table has been moved. */ - if (new_vma != vma && vma->vm_start == old_addr && - vma->vm_end == (old_addr + old_len)) + if (vma->vm_start == old_addr && + vma->vm_end == (old_addr + old_len)) { + const pgoff_t pgoff_unfaulted = vma->vm_start >> PAGE_SHIFT; + unlink_anon_vmas(vma); + /* + * The VMA is now unfaulted and it is an invariant that + * unfaulted anonymous VMAs have page offset equal to + * vma->vm_start >> PAGE_SHIFT. + */ + if (vma_is_anonymous(vma) && !vma->vm_file) + vma->vm_pgoff = pgoff_unfaulted; + } /* Because we won't unmap we don't need to touch locked_vm */ return new_addr; diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ffdc71103c2a..06c493b908ed 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -4633,6 +4633,16 @@ static struct page *__page_frag_cache_refill(struct page_frag_cache *nc, return page; } +void page_frag_cache_drain(struct page_frag_cache *nc) +{ + if (!nc->va) + return; + + __page_frag_cache_drain(virt_to_head_page(nc->va), nc->pagecnt_bias); + nc->va = NULL; +} +EXPORT_SYMBOL(page_frag_cache_drain); + void __page_frag_cache_drain(struct page *page, unsigned int count) { VM_BUG_ON_PAGE(page_ref_count(page) == 0, page); diff --git a/mm/page_owner.c b/mm/page_owner.c index 4e2723e1b300..3f539928324a 100644 --- a/mm/page_owner.c +++ b/mm/page_owner.c @@ -363,6 +363,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, { #ifdef CONFIG_MEMCG unsigned long memcg_data; + struct obj_cgroup *objcg; struct mem_cgroup *memcg; bool online; char name[80]; @@ -372,11 +373,18 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, if (!memcg_data) goto out_unlock; - if (memcg_data & MEMCG_DATA_OBJCGS) + if (memcg_data & MEMCG_DATA_OBJCGS) { ret += scnprintf(kbuf + ret, count - ret, "Slab cache page\n"); + goto out_unlock; + } - memcg = page_memcg_check(page); + if (memcg_data & MEMCG_DATA_KMEM) { + objcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + memcg = obj_cgroup_memcg(objcg); + } else { + memcg = (void *)(memcg_data & ~MEMCG_DATA_FLAGS_MASK); + } if (!memcg) goto out_unlock; @@ -384,7 +392,7 @@ static inline int print_page_owner_memcg(char *kbuf, size_t count, int ret, cgroup_name(memcg->css.cgroup, name, sizeof(name)); ret += scnprintf(kbuf + ret, count - ret, "Charged %sto %smemcg %s\n", - PageMemcgKmem(page) ? "(via objcg) " : "", + (memcg_data & MEMCG_DATA_KMEM) ? "(via objcg) " : "", online ? "" : "offline ", name); out_unlock: diff --git a/mm/page_table_check.c b/mm/page_table_check.c index 509c6ef8de40..75269bc47547 100644 --- a/mm/page_table_check.c +++ b/mm/page_table_check.c @@ -165,18 +165,29 @@ void __page_table_check_pte_clear(struct mm_struct *mm, pte_t pte) if (&init_mm == mm) return; - if (pte_user_accessible_page(pte)) { + if (pte_user_accessible_page(pte) && !pte_special(pte)) { page_table_check_clear(pte_pfn(pte), PAGE_SIZE >> PAGE_SHIFT); } } EXPORT_SYMBOL(__page_table_check_pte_clear); +static inline bool page_table_check_huge_zero_pmd(pmd_t pmd) +{ + unsigned long pfn = pmd_pfn(pmd); + + if (!pfn_valid(pfn)) + return false; + + return is_huge_zero_page(pfn_to_page(pfn)); +} + void __page_table_check_pmd_clear(struct mm_struct *mm, pmd_t pmd) { if (&init_mm == mm) return; - if (pmd_user_accessible_page(pmd)) { + if (pmd_user_accessible_page(pmd) && + !page_table_check_huge_zero_pmd(pmd)) { page_table_check_clear(pmd_pfn(pmd), PMD_SIZE >> PAGE_SHIFT); } } @@ -221,7 +232,7 @@ void __page_table_check_ptes_set(struct mm_struct *mm, pte_t *ptep, pte_t pte, for (i = 0; i < nr; i++) __page_table_check_pte_clear(mm, ptep_get(ptep + i)); - if (pte_user_accessible_page(pte)) + if (pte_user_accessible_page(pte) && !pte_special(pte)) page_table_check_set(pte_pfn(pte), nr, pte_write(pte)); } EXPORT_SYMBOL(__page_table_check_ptes_set); @@ -242,7 +253,8 @@ void __page_table_check_pmd_set(struct mm_struct *mm, pmd_t *pmdp, pmd_t pmd) page_table_check_pmd_flags(pmd); __page_table_check_pmd_clear(mm, *pmdp); - if (pmd_user_accessible_page(pmd)) { + if (pmd_user_accessible_page(pmd) && + !page_table_check_huge_zero_pmd(pmd)) { page_table_check_set(pmd_pfn(pmd), PMD_SIZE >> PAGE_SHIFT, pmd_write(pmd)); } diff --git a/mm/rmap.c b/mm/rmap.c index 0a88928c410a..531d6c813027 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -211,7 +211,11 @@ int __anon_vma_prepare(struct vm_area_struct *vma) /* page_table_lock to protect against threads */ spin_lock(&mm->page_table_lock); if (likely(!vma->anon_vma)) { - vma->anon_vma = anon_vma; + /* + * Make anon_vma fields visible before anon_vma is published. + * Paired with an address dependency in reusable_anon_vma(). + */ + smp_store_release(&vma->anon_vma, anon_vma); anon_vma_chain_link(vma, avc, anon_vma); anon_vma->num_active_vmas++; allocated = NULL; diff --git a/mm/secretmem.c b/mm/secretmem.c index f64ea1cde2f9..ef507d68f3e1 100644 --- a/mm/secretmem.c +++ b/mm/secretmem.c @@ -18,6 +18,8 @@ #include #include #include +#include +#include #include @@ -47,10 +49,69 @@ bool secretmem_active(void) return !!atomic_read(&secretmem_users); } +struct secretmem_inode_state { + struct user_struct *user; + atomic_long_t nr_pages_accounted; +}; + +static bool __secretmem_account_pages(struct user_struct *user, + unsigned long nr_pages) +{ + unsigned long page_limit, cur_pages, new_pages; + + if (!nr_pages) + return true; + + page_limit = rlimit(RLIMIT_MEMLOCK) >> PAGE_SHIFT; + + cur_pages = atomic_long_read(&user->locked_vm); + do { + new_pages = cur_pages + nr_pages; + if (new_pages > page_limit) + return false; + } while (!atomic_long_try_cmpxchg(&user->locked_vm, + &cur_pages, new_pages)); + return true; +} + +static bool secretmem_account_folio(struct secretmem_inode_state *state, + struct folio *folio) +{ + const unsigned long nr_pages = folio_nr_pages(folio); + + if (!__secretmem_account_pages(state->user, nr_pages)) + return false; + + atomic_long_add(nr_pages, &state->nr_pages_accounted); + return true; +} + +static void __secretmem_unaccount_pages(struct secretmem_inode_state *state, + unsigned long nr_pages) +{ + atomic_long_sub(nr_pages, &state->user->locked_vm); + atomic_long_sub(nr_pages, &state->nr_pages_accounted); +} + +static void secretmem_unaccount_folio(struct secretmem_inode_state *state, + struct folio *folio) +{ + __secretmem_unaccount_pages(state, folio_nr_pages(folio)); +} + +static void secretmem_unaccount_all_folios(struct secretmem_inode_state *state) +{ + const unsigned long nr_pages_accounted = + atomic_long_read(&state->nr_pages_accounted); + + __secretmem_unaccount_pages(state, nr_pages_accounted); +} + static vm_fault_t secretmem_fault(struct vm_fault *vmf) { struct address_space *mapping = vmf->vma->vm_file->f_mapping; struct inode *inode = file_inode(vmf->vma->vm_file); + struct secretmem_inode_state *state = inode->i_private; pgoff_t offset = vmf->pgoff; gfp_t gfp = vmf->gfp_mask; unsigned long addr; @@ -73,9 +134,16 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) goto out; } + if (!secretmem_account_folio(state, folio)) { + folio_put(folio); + ret = VM_FAULT_SIGBUS; + goto out; + } + page = &folio->page; err = set_direct_map_invalid_noflush(page); if (err) { + secretmem_unaccount_folio(state, folio); folio_put(folio); ret = vmf_error(err); goto out; @@ -84,6 +152,7 @@ static vm_fault_t secretmem_fault(struct vm_fault *vmf) __folio_mark_uptodate(folio); err = filemap_add_folio(mapping, folio, offset, gfp); if (unlikely(err)) { + secretmem_unaccount_folio(state, folio); /* * If a split of large page was required, it * already happened when we marked the page invalid @@ -114,23 +183,30 @@ static const struct vm_operations_struct secretmem_vm_ops = { .fault = secretmem_fault, }; +static void secretmem_destroy_inode_priv(struct inode *inode) +{ + struct secretmem_inode_state *state = inode->i_private; + + secretmem_unaccount_all_folios(state); + free_uid(state->user); + kfree(state); + inode->i_private = NULL; +} + static int secretmem_release(struct inode *inode, struct file *file) { atomic_dec(&secretmem_users); + secretmem_destroy_inode_priv(inode); + return 0; } static int secretmem_mmap(struct file *file, struct vm_area_struct *vma) { - unsigned long len = vma->vm_end - vma->vm_start; - if ((vma->vm_flags & (VM_SHARED | VM_MAYSHARE)) == 0) return -EINVAL; - if (!mlock_future_ok(vma->vm_mm, vma->vm_flags | VM_LOCKED, len)) - return -EAGAIN; - - vm_flags_set(vma, VM_LOCKED | VM_DONTDUMP); + vm_flags_set(vma, VM_DONTDUMP); vma->vm_ops = &secretmem_vm_ops; return 0; @@ -190,19 +266,40 @@ static const struct inode_operations secretmem_iops = { static struct vfsmount *secretmem_mnt; +static int secretmem_init_inode_priv(struct inode *inode) +{ + struct secretmem_inode_state *state; + + state = kzalloc(sizeof(*state), GFP_KERNEL); + if (!state) + return -ENOMEM; + + state->user = get_uid(current_user()); + inode->i_private = state; + return 0; +} + static struct file *secretmem_file_create(unsigned long flags) { struct file *file; struct inode *inode; const char *anon_name = "[secretmem]"; + int err; inode = anon_inode_make_secure_inode(secretmem_mnt->mnt_sb, anon_name, NULL); if (IS_ERR(inode)) return ERR_CAST(inode); + + err = secretmem_init_inode_priv(inode); + if (err) + goto err_free_inode; + file = alloc_file_pseudo(inode, secretmem_mnt, "secretmem", O_RDWR, &secretmem_fops); - if (IS_ERR(file)) - goto err_free_inode; + if (IS_ERR(file)) { + err = PTR_ERR(file); + goto err_free_priv; + } mapping_set_gfp_mask(inode->i_mapping, GFP_HIGHUSER); mapping_set_unevictable(inode->i_mapping); @@ -215,10 +312,11 @@ static struct file *secretmem_file_create(unsigned long flags) inode->i_size = 0; return file; - +err_free_priv: + secretmem_destroy_inode_priv(inode); err_free_inode: iput(inode); - return file; + return ERR_PTR(err); } SYSCALL_DEFINE1(memfd_secret, unsigned int, flags) diff --git a/mm/shmem.c b/mm/shmem.c index d24398f56660..f31f6ffa2876 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -3415,6 +3415,7 @@ static int shmem_rename2(struct mnt_idmap *idmap, { struct inode *inode = d_inode(old_dentry); int they_are_dirs = S_ISDIR(inode->i_mode); + bool had_offset = false; int error; if (flags & ~(RENAME_NOREPLACE | RENAME_EXCHANGE | RENAME_WHITEOUT)) @@ -3427,16 +3428,23 @@ static int shmem_rename2(struct mnt_idmap *idmap, if (!simple_empty(new_dentry)) return -ENOTEMPTY; + error = simple_offset_add(shmem_get_offset_ctx(new_dir), new_dentry); + if (error == -EBUSY) + had_offset = true; + else if (unlikely(error)) + return error; + if (flags & RENAME_WHITEOUT) { error = shmem_whiteout(idmap, old_dir, old_dentry); - if (error) + if (error) { + if (!had_offset) + simple_offset_remove(shmem_get_offset_ctx(new_dir), + new_dentry); return error; + } } - error = simple_offset_rename(old_dir, old_dentry, new_dir, new_dentry); - if (error) - return error; - + simple_offset_rename(old_dir, old_dentry, new_dir, new_dentry); if (d_really_is_positive(new_dentry)) { (void) shmem_unlink(new_dir, new_dentry); if (they_are_dirs) { @@ -4351,7 +4359,9 @@ static int shmem_fill_super(struct super_block *sb, struct fs_context *fc) #ifdef CONFIG_TMPFS_POSIX_ACL sb->s_flags |= SB_POSIXACL; #endif - uuid_gen(&sb->s_uuid); + uuid_t uuid; + uuid_gen(&uuid); + super_set_uuid(sb, uuid.b, sizeof(uuid)); #ifdef CONFIG_TMPFS_QUOTA if (ctx->seen & SHMEM_SEEN_QUOTA) { diff --git a/mm/slub.c b/mm/slub.c index 400563c45266..eeb0aa9e6ce8 100644 --- a/mm/slub.c +++ b/mm/slub.c @@ -452,6 +452,26 @@ static inline void set_freepointer(struct kmem_cache *s, void *object, void *fp) *(freeptr_t *)freeptr_addr = freelist_ptr_encode(s, fp, freeptr_addr); } +/* + * See comment in calculate_sizes(). + */ +static inline bool freeptr_outside_object(struct kmem_cache *s) +{ + return s->offset >= s->inuse; +} + +/* + * Return offset of the end of info block which is inuse + free pointer if + * not overlapping with object. + */ +static inline unsigned int get_info_end(struct kmem_cache *s) +{ + if (freeptr_outside_object(s)) + return s->inuse + sizeof(void *); + else + return s->inuse; +} + /* Loop over all objects in a slab */ #define for_each_object(__p, __s, __addr, __objects) \ for (__p = fixup_red_left(__s, __addr); \ @@ -627,6 +647,50 @@ static inline bool slab_update_freelist(struct kmem_cache *s, struct slab *slab, return false; } +/* + * kmalloc caches has fixed sizes (mostly power of 2), and kmalloc() API + * family will round up the real request size to these fixed ones, so + * there could be an extra area than what is requested. Save the original + * request size in the meta data area, for better debug and sanity check. + */ +static inline void set_orig_size(struct kmem_cache *s, + void *object, unsigned int orig_size) +{ + void *p = kasan_reset_tag(object); + unsigned int kasan_meta_size; + + if (!slub_debug_orig_size(s)) + return; + + /* + * KASAN can save its free meta data inside of the object at offset 0. + * If this meta data size is larger than 'orig_size', it will overlap + * the data redzone in [orig_size+1, object_size]. Thus, we adjust + * 'orig_size' to be as at least as big as KASAN's meta data. + */ + kasan_meta_size = kasan_metadata_size(s, true); + if (kasan_meta_size > orig_size) + orig_size = kasan_meta_size; + + p += get_info_end(s); + p += sizeof(struct track) * 2; + + *(unsigned int *)p = orig_size; +} + +static inline unsigned int get_orig_size(struct kmem_cache *s, void *object) +{ + void *p = kasan_reset_tag(object); + + if (!slub_debug_orig_size(s)) + return s->object_size; + + p += get_info_end(s); + p += sizeof(struct track) * 2; + + return *(unsigned int *)p; +} + #ifdef CONFIG_SLUB_DEBUG static unsigned long object_map[BITS_TO_LONGS(MAX_OBJS_PER_PAGE)]; static DEFINE_SPINLOCK(object_map_lock); @@ -740,26 +804,6 @@ static void print_section(char *level, char *text, u8 *addr, metadata_access_disable(); } -/* - * See comment in calculate_sizes(). - */ -static inline bool freeptr_outside_object(struct kmem_cache *s) -{ - return s->offset >= s->inuse; -} - -/* - * Return offset of the end of info block which is inuse + free pointer if - * not overlapping with object. - */ -static inline unsigned int get_info_end(struct kmem_cache *s) -{ - if (freeptr_outside_object(s)) - return s->inuse + sizeof(void *); - else - return s->inuse; -} - static struct track *get_track(struct kmem_cache *s, void *object, enum track_item alloc) { @@ -860,50 +904,6 @@ static void print_slab_info(const struct slab *slab) folio_flags(folio, 0)); } -/* - * kmalloc caches has fixed sizes (mostly power of 2), and kmalloc() API - * family will round up the real request size to these fixed ones, so - * there could be an extra area than what is requested. Save the original - * request size in the meta data area, for better debug and sanity check. - */ -static inline void set_orig_size(struct kmem_cache *s, - void *object, unsigned int orig_size) -{ - void *p = kasan_reset_tag(object); - - if (!slub_debug_orig_size(s)) - return; - -#ifdef CONFIG_KASAN_GENERIC - /* - * KASAN could save its free meta data in object's data area at - * offset 0, if the size is larger than 'orig_size', it will - * overlap the data redzone in [orig_size+1, object_size], and - * the check should be skipped. - */ - if (kasan_metadata_size(s, true) > orig_size) - orig_size = s->object_size; -#endif - - p += get_info_end(s); - p += sizeof(struct track) * 2; - - *(unsigned int *)p = orig_size; -} - -static inline unsigned int get_orig_size(struct kmem_cache *s, void *object) -{ - void *p = kasan_reset_tag(object); - - if (!slub_debug_orig_size(s)) - return s->object_size; - - p += get_info_end(s); - p += sizeof(struct track) * 2; - - return *(unsigned int *)p; -} - void skip_orig_size_check(struct kmem_cache *s, const void *object) { set_orig_size(s, (void *)object, s->object_size); @@ -1197,7 +1197,7 @@ static int check_object(struct kmem_cache *s, struct slab *slab, { u8 *p = object; u8 *endobject = object + s->object_size; - unsigned int orig_size; + unsigned int orig_size, kasan_meta_size; if (s->flags & SLAB_RED_ZONE) { if (!check_bytes_and_report(s, slab, object, "Left Redzone", @@ -1227,12 +1227,23 @@ static int check_object(struct kmem_cache *s, struct slab *slab, } if (s->flags & SLAB_POISON) { - if (val != SLUB_RED_ACTIVE && (s->flags & __OBJECT_POISON) && - (!check_bytes_and_report(s, slab, p, "Poison", p, - POISON_FREE, s->object_size - 1) || - !check_bytes_and_report(s, slab, p, "End Poison", - p + s->object_size - 1, POISON_END, 1))) - return 0; + if (val != SLUB_RED_ACTIVE && (s->flags & __OBJECT_POISON)) { + /* + * KASAN can save its free meta data inside of the + * object at offset 0. Thus, skip checking the part of + * the redzone that overlaps with the meta data. + */ + kasan_meta_size = kasan_metadata_size(s, true); + if (kasan_meta_size < s->object_size - 1 && + !check_bytes_and_report(s, slab, p, "Poison", + p + kasan_meta_size, POISON_FREE, + s->object_size - kasan_meta_size - 1)) + return 0; + if (kasan_meta_size < s->object_size && + !check_bytes_and_report(s, slab, p, "End Poison", + p + s->object_size - 1, POISON_END, 1)) + return 0; + } /* * check_pad_bytes cleans up on its own. */ @@ -1760,7 +1771,6 @@ static inline void inc_slabs_node(struct kmem_cache *s, int node, int objects) {} static inline void dec_slabs_node(struct kmem_cache *s, int node, int objects) {} - #ifndef CONFIG_SLUB_TINY static bool freelist_corrupted(struct kmem_cache *s, struct slab *slab, void **freelist, void *nextfree) @@ -1797,15 +1807,27 @@ static __always_inline bool slab_free_hook(struct kmem_cache *s, * * The initialization memset's clear the object and the metadata, * but don't touch the SLAB redzone. + * + * The object's freepointer is also avoided if stored outside the + * object. */ if (init) { int rsize; + unsigned int inuse, orig_size; + inuse = get_info_end(s); + orig_size = get_orig_size(s, x); if (!kasan_has_integrated_init()) - memset(kasan_reset_tag(x), 0, s->object_size); + memset(kasan_reset_tag(x), 0, orig_size); rsize = (s->flags & SLAB_RED_ZONE) ? s->red_left_pad : 0; - memset((char *)kasan_reset_tag(x) + s->inuse, 0, - s->size - s->inuse - rsize); + memset((char *)kasan_reset_tag(x) + inuse, 0, + s->size - inuse - rsize); + /* + * Restore orig_size, otherwize kmalloc redzone overwritten + * would be reported + */ + set_orig_size(s, x, orig_size); + } /* KASAN might put x into memory quarantine, delaying its reuse. */ return kasan_slab_free(s, x, init); @@ -3452,7 +3474,8 @@ static void *__slab_alloc_node(struct kmem_cache *s, static __always_inline void maybe_wipe_obj_freeptr(struct kmem_cache *s, void *obj) { - if (unlikely(slab_want_init_on_free(s)) && obj) + if (unlikely(slab_want_init_on_free(s)) && obj && + !freeptr_outside_object(s)) memset((void *)((char *)kasan_reset_tag(obj) + s->offset), 0, sizeof(void *)); } diff --git a/mm/util.c b/mm/util.c index 08d494896552..b8a7ec447192 100644 --- a/mm/util.c +++ b/mm/util.c @@ -135,6 +135,23 @@ void *kmemdup(const void *src, size_t len, gfp_t gfp) } EXPORT_SYMBOL(kmemdup); +/** + * kmemdup_array - duplicate a given array. + * + * @src: array to duplicate. + * @element_size: size of each element of array. + * @count: number of elements to duplicate from array. + * @gfp: GFP mask to use. + * + * Return: duplicated array of @src or %NULL in case of error, + * result is physically contiguous. Use kfree() to free. + */ +void *kmemdup_array(const void *src, size_t element_size, size_t count, gfp_t gfp) +{ + return kmemdup(src, size_mul(element_size, count), gfp); +} +EXPORT_SYMBOL(kmemdup_array); + /** * kvmemdup - duplicate region of memory * diff --git a/mm/vmalloc.c b/mm/vmalloc.c index 3681f0207b21..11ae4bb25fbb 100644 --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -1677,7 +1677,7 @@ static struct vmap_area *alloc_vmap_area(unsigned long size, BUG_ON(va->va_start < vstart); BUG_ON(va->va_end > vend); - ret = kasan_populate_vmalloc(addr, size); + ret = kasan_populate_vmalloc(addr, size, gfp_mask); if (ret) { free_vmap_area(va); return ERR_PTR(ret); @@ -4207,7 +4207,7 @@ struct vm_struct **pcpu_get_vm_areas(const unsigned long *offsets, /* populate the kasan shadow space */ for (area = 0; area < nr_vms; area++) { - if (kasan_populate_vmalloc(vas[area]->va_start, sizes[area])) + if (kasan_populate_vmalloc(vas[area]->va_start, sizes[area], GFP_KERNEL)) goto err_free_shadow; } diff --git a/mm/vmpressure.c b/mm/vmpressure.c index 22c6689d9302..bd5183dfd879 100644 --- a/mm/vmpressure.c +++ b/mm/vmpressure.c @@ -169,7 +169,7 @@ static bool vmpressure_event(struct vmpressure *vmpr, continue; if (level < ev->level) continue; - eventfd_signal(ev->efd, 1); + eventfd_signal(ev->efd); ret = true; } mutex_unlock(&vmpr->events_lock); diff --git a/mm/vmscan.c b/mm/vmscan.c index b6ec592c3f14..4197e10a27b6 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -4923,15 +4923,9 @@ static bool sort_folio(struct lruvec *lruvec, struct folio *folio, struct scan_c VM_WARN_ON_ONCE_FOLIO(gen >= MAX_NR_GENS, folio); - /* unevictable */ - if (!folio_evictable(folio)) { - success = lru_gen_del_folio(lruvec, folio, true); - VM_WARN_ON_ONCE_FOLIO(!success, folio); - folio_set_unevictable(folio); - lruvec_add_folio(lruvec, folio); - __count_vm_events(UNEVICTABLE_PGCULLED, delta); - return true; - } + /* unevictable: let it through and the generic path will cull it */ + if (!folio_evictable(folio)) + return false; /* dirty lazyfree */ if (type == LRU_GEN_FILE && folio_test_anon(folio) && folio_test_dirty(folio)) { @@ -5212,11 +5206,9 @@ static int evict_folios(struct lruvec *lruvec, struct scan_control *sc, int swap sc->nr_reclaimed += reclaimed; list_for_each_entry_safe_reverse(folio, next, &list, lru) { - if (!folio_evictable(folio)) { - list_del(&folio->lru); - folio_putback_lru(folio); + /* move_folios_to_lru() culls unevictable folios via folio_putback_lru() */ + if (!folio_evictable(folio)) continue; - } if (folio_test_reclaim(folio) && (folio_test_dirty(folio) || folio_test_writeback(folio))) { diff --git a/mm/workingset.c b/mm/workingset.c index 1cdbc1fb5873..e879a13fec31 100644 --- a/mm/workingset.c +++ b/mm/workingset.c @@ -639,12 +639,12 @@ void workingset_update_node(struct xa_node *node) if (node->count && node->count == node->nr_values) { if (list_empty(&node->private_list)) { - list_lru_add(&shadow_nodes, &node->private_list); + list_lru_add_obj(&shadow_nodes, &node->private_list); __inc_lruvec_kmem_state(node, WORKINGSET_NODES); } } else { if (!list_empty(&node->private_list)) { - list_lru_del(&shadow_nodes, &node->private_list); + list_lru_del_obj(&shadow_nodes, &node->private_list); __dec_lruvec_kmem_state(node, WORKINGSET_NODES); } } diff --git a/net/9p/trans_xen.c b/net/9p/trans_xen.c index 6387ee924a2d..7709be0fa541 100644 --- a/net/9p/trans_xen.c +++ b/net/9p/trans_xen.c @@ -275,45 +275,52 @@ static void xen_9pfs_front_free(struct xen_9pfs_front_priv *priv) { int i, j; - write_lock(&xen_9pfs_lock); - list_del(&priv->list); - write_unlock(&xen_9pfs_lock); - - for (i = 0; i < priv->num_rings; i++) { - struct xen_9pfs_dataring *ring = &priv->rings[i]; - - cancel_work_sync(&ring->work); - - if (!priv->rings[i].intf) - break; - if (priv->rings[i].irq > 0) - unbind_from_irqhandler(priv->rings[i].irq, ring); - if (priv->rings[i].data.in) { - for (j = 0; - j < (1 << priv->rings[i].intf->ring_order); - j++) { - grant_ref_t ref; - - ref = priv->rings[i].intf->ref[j]; - gnttab_end_foreign_access(ref, NULL); - } - free_pages_exact(priv->rings[i].data.in, + if (priv->rings) { + for (i = 0; i < priv->num_rings; i++) { + struct xen_9pfs_dataring *ring = &priv->rings[i]; + + cancel_work_sync(&ring->work); + + if (!priv->rings[i].intf) + break; + if (priv->rings[i].irq > 0) + unbind_from_irqhandler(priv->rings[i].irq, ring); + if (priv->rings[i].data.in) { + for (j = 0; + j < (1 << priv->rings[i].intf->ring_order); + j++) { + grant_ref_t ref; + + ref = priv->rings[i].intf->ref[j]; + gnttab_end_foreign_access(ref, NULL); + } + free_pages_exact(priv->rings[i].data.in, 1UL << (priv->rings[i].intf->ring_order + XEN_PAGE_SHIFT)); + } + gnttab_end_foreign_access(priv->rings[i].ref, NULL); + free_page((unsigned long)priv->rings[i].intf); } - gnttab_end_foreign_access(priv->rings[i].ref, NULL); - free_page((unsigned long)priv->rings[i].intf); + kfree(priv->rings); } - kfree(priv->rings); kfree(priv->tag); kfree(priv); } static void xen_9pfs_front_remove(struct xenbus_device *dev) { - struct xen_9pfs_front_priv *priv = dev_get_drvdata(&dev->dev); + struct xen_9pfs_front_priv *priv; + write_lock(&xen_9pfs_lock); + priv = dev_get_drvdata(&dev->dev); + if (priv == NULL) { + write_unlock(&xen_9pfs_lock); + return; + } dev_set_drvdata(&dev->dev, NULL); + list_del(&priv->list); + write_unlock(&xen_9pfs_lock); + xen_9pfs_front_free(priv); } @@ -380,7 +387,7 @@ static int xen_9pfs_front_init(struct xenbus_device *dev) { int ret, i; struct xenbus_transaction xbt; - struct xen_9pfs_front_priv *priv = dev_get_drvdata(&dev->dev); + struct xen_9pfs_front_priv *priv; char *versions, *v; unsigned int max_rings, max_ring_order, len = 0; @@ -408,6 +415,10 @@ static int xen_9pfs_front_init(struct xenbus_device *dev) if (p9_xen_trans.maxsize > XEN_FLEX_RING_SIZE(max_ring_order)) p9_xen_trans.maxsize = XEN_FLEX_RING_SIZE(max_ring_order) / 2; + priv = kzalloc(sizeof(*priv), GFP_KERNEL); + if (!priv) + return -ENOMEM; + priv->dev = dev; priv->num_rings = XEN_9PFS_NUM_RINGS; priv->rings = kcalloc(priv->num_rings, sizeof(*priv->rings), GFP_KERNEL); @@ -466,6 +477,11 @@ static int xen_9pfs_front_init(struct xenbus_device *dev) goto error; } + write_lock(&xen_9pfs_lock); + dev_set_drvdata(&dev->dev, priv); + list_add_tail(&priv->list, &xen_9pfs_devs); + write_unlock(&xen_9pfs_lock); + xenbus_switch_state(dev, XenbusStateInitialised); return 0; @@ -480,19 +496,6 @@ static int xen_9pfs_front_init(struct xenbus_device *dev) static int xen_9pfs_front_probe(struct xenbus_device *dev, const struct xenbus_device_id *id) { - struct xen_9pfs_front_priv *priv = NULL; - - priv = kzalloc(sizeof(*priv), GFP_KERNEL); - if (!priv) - return -ENOMEM; - - priv->dev = dev; - dev_set_drvdata(&dev->dev, priv); - - write_lock(&xen_9pfs_lock); - list_add_tail(&priv->list, &xen_9pfs_devs); - write_unlock(&xen_9pfs_lock); - return 0; } diff --git a/net/ax25/af_ax25.c b/net/ax25/af_ax25.c index 4a16142ac58a..8b86f05c1dea 100644 --- a/net/ax25/af_ax25.c +++ b/net/ax25/af_ax25.c @@ -1270,28 +1270,18 @@ static int __must_check ax25_connect(struct socket *sock, } } - /* - * Must bind first - autobinding in this may or may not work. If - * the socket is already bound, check to see if the device has - * been filled in, error if it hasn't. - */ + /* Must bind first - autobinding does not work. */ if (sock_flag(sk, SOCK_ZAPPED)) { - /* check if we can remove this feature. It is broken. */ - printk(KERN_WARNING "ax25_connect(): %s uses autobind, please contact jreuter@yaina.de\n", - current->comm); - if ((err = ax25_rt_autobind(ax25, &fsa->fsa_ax25.sax25_call)) < 0) { - kfree(digi); - goto out_release; - } + kfree(digi); + err = -EINVAL; + goto out_release; + } - ax25_fillin_cb(ax25, ax25->ax25_dev); - ax25_cb_add(ax25); - } else { - if (ax25->ax25_dev == NULL) { - kfree(digi); - err = -EHOSTUNREACH; - goto out_release; - } + /* Check to see if the device has been filled in, error if it hasn't. */ + if (ax25->ax25_dev == NULL) { + kfree(digi); + err = -EHOSTUNREACH; + goto out_release; } if (sk->sk_type == SOCK_SEQPACKET && diff --git a/net/ax25/ax25_route.c b/net/ax25/ax25_route.c index 69de75db0c9c..10577434f40b 100644 --- a/net/ax25/ax25_route.c +++ b/net/ax25/ax25_route.c @@ -373,80 +373,6 @@ ax25_route *ax25_get_route(ax25_address *addr, struct net_device *dev) return ax25_rt; } -/* - * Adjust path: If you specify a default route and want to connect - * a target on the digipeater path but w/o having a special route - * set before, the path has to be truncated from your target on. - */ -static inline void ax25_adjust_path(ax25_address *addr, ax25_digi *digipeat) -{ - int k; - - for (k = 0; k < digipeat->ndigi; k++) { - if (ax25cmp(addr, &digipeat->calls[k]) == 0) - break; - } - - digipeat->ndigi = k; -} - - -/* - * Find which interface to use. - */ -int ax25_rt_autobind(ax25_cb *ax25, ax25_address *addr) -{ - ax25_uid_assoc *user; - ax25_route *ax25_rt; - int err = 0; - - ax25_route_lock_use(); - ax25_rt = ax25_get_route(addr, NULL); - if (!ax25_rt) { - ax25_route_lock_unuse(); - return -EHOSTUNREACH; - } - rcu_read_lock(); - if ((ax25->ax25_dev = ax25_dev_ax25dev(ax25_rt->dev)) == NULL) { - err = -EHOSTUNREACH; - goto put; - } - - user = ax25_findbyuid(current_euid()); - if (user) { - ax25->source_addr = user->call; - ax25_uid_put(user); - } else { - if (ax25_uid_policy && !capable(CAP_NET_BIND_SERVICE)) { - err = -EPERM; - goto put; - } - ax25->source_addr = *(ax25_address *)ax25->ax25_dev->dev->dev_addr; - } - - if (ax25_rt->digipeat != NULL) { - ax25->digipeat = kmemdup(ax25_rt->digipeat, sizeof(ax25_digi), - GFP_ATOMIC); - if (ax25->digipeat == NULL) { - err = -ENOMEM; - goto put; - } - ax25_adjust_path(addr, ax25->digipeat); - } - - if (ax25->sk != NULL) { - local_bh_disable(); - bh_lock_sock(ax25->sk); - sock_reset_flag(ax25->sk, SOCK_ZAPPED); - bh_unlock_sock(ax25->sk); - local_bh_enable(); - } - -put: - rcu_read_unlock(); - ax25_route_lock_unuse(); - return err; -} struct sk_buff *ax25_rt_build_path(struct sk_buff *skb, ax25_address *src, ax25_address *dest, ax25_digi *digi) diff --git a/net/bluetooth/6lowpan.c b/net/bluetooth/6lowpan.c index 8625d3db2442..53b09639cba5 100644 --- a/net/bluetooth/6lowpan.c +++ b/net/bluetooth/6lowpan.c @@ -1065,23 +1065,15 @@ static void disconnect_all_peers(void) } while (nchans); } -struct set_enable { - struct work_struct work; - bool flag; -}; - -static void do_enable_set(struct work_struct *work) +static void do_enable_set(bool flag) { - struct set_enable *set_enable = container_of(work, - struct set_enable, work); - - if (!set_enable->flag || enable_6lowpan != set_enable->flag) + if (!flag || enable_6lowpan != flag) /* Disconnect existing connections if 6lowpan is * disabled */ disconnect_all_peers(); - enable_6lowpan = set_enable->flag; + enable_6lowpan = flag; mutex_lock(&set_lock); if (listen_chan) { @@ -1091,22 +1083,11 @@ static void do_enable_set(struct work_struct *work) listen_chan = bt_6lowpan_listen(); mutex_unlock(&set_lock); - - kfree(set_enable); } static int lowpan_enable_set(void *data, u64 val) { - struct set_enable *set_enable; - - set_enable = kzalloc(sizeof(*set_enable), GFP_KERNEL); - if (!set_enable) - return -ENOMEM; - - set_enable->flag = !!val; - INIT_WORK(&set_enable->work, do_enable_set); - - schedule_work(&set_enable->work); + do_enable_set(!!val); return 0; } diff --git a/net/bluetooth/eir.c b/net/bluetooth/eir.c index 7a28baa2b71f..ee0136bfae40 100644 --- a/net/bluetooth/eir.c +++ b/net/bluetooth/eir.c @@ -242,7 +242,7 @@ u8 eir_create_per_adv_data(struct hci_dev *hdev, u8 instance, u8 *ptr) return ad_len; } -u8 eir_create_adv_data(struct hci_dev *hdev, u8 instance, u8 *ptr) +u8 eir_create_adv_data(struct hci_dev *hdev, u8 instance, u8 *ptr, u8 size) { struct adv_info *adv = NULL; u8 ad_len = 0, flags = 0; @@ -283,10 +283,12 @@ u8 eir_create_adv_data(struct hci_dev *hdev, u8 instance, u8 *ptr) if (!flags) flags |= mgmt_get_adv_discov_flags(hdev); - /* If flags would still be empty, then there is no need to - * include the "Flags" AD field". + /* Only add the "Flags" if it fits together with the instance + * advertising data; drop it rather than overflow the buffer. */ - if (flags) { + if (flags && + (ad_len + eir_precalc_len(1) + + (adv ? adv->adv_data_len : 0) <= size)) { ptr[0] = 0x02; ptr[1] = EIR_FLAGS; ptr[2] = flags; @@ -316,7 +318,8 @@ u8 eir_create_adv_data(struct hci_dev *hdev, u8 instance, u8 *ptr) } /* Provide Tx Power only if we can provide a valid value for it */ - if (adv_tx_power != HCI_TX_POWER_INVALID) { + if (adv_tx_power != HCI_TX_POWER_INVALID && + (ad_len + eir_precalc_len(1) <= size)) { ptr[0] = 0x02; ptr[1] = EIR_TX_POWER; ptr[2] = (u8)adv_tx_power; diff --git a/net/bluetooth/eir.h b/net/bluetooth/eir.h index 0df19f2f4af9..4497f8fd5fef 100644 --- a/net/bluetooth/eir.h +++ b/net/bluetooth/eir.h @@ -9,7 +9,7 @@ void eir_create(struct hci_dev *hdev, u8 *data); -u8 eir_create_adv_data(struct hci_dev *hdev, u8 instance, u8 *ptr); +u8 eir_create_adv_data(struct hci_dev *hdev, u8 instance, u8 *ptr, u8 size); u8 eir_create_scan_rsp(struct hci_dev *hdev, u8 instance, u8 *ptr); u8 eir_create_per_adv_data(struct hci_dev *hdev, u8 instance, u8 *ptr); diff --git a/net/bluetooth/hci_conn.c b/net/bluetooth/hci_conn.c index 644c751e41c6..7e209176d7db 100644 --- a/net/bluetooth/hci_conn.c +++ b/net/bluetooth/hci_conn.c @@ -307,11 +307,13 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data) cp.tx_bandwidth = cpu_to_le32(0x00001f40); cp.rx_bandwidth = cpu_to_le32(0x00001f40); + hci_dev_lock(hdev); + switch (conn->codec.id) { case BT_CODEC_MSBC: if (!find_next_esco_param(conn, esco_param_msbc, ARRAY_SIZE(esco_param_msbc))) - return -EINVAL; + goto unlock; param = &esco_param_msbc[conn->attempt - 1]; cp.tx_coding_format.id = 0x05; @@ -337,7 +339,7 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data) case BT_CODEC_TRANSPARENT: if (!find_next_esco_param(conn, esco_param_msbc, ARRAY_SIZE(esco_param_msbc))) - return -EINVAL; + goto unlock; param = &esco_param_msbc[conn->attempt - 1]; cp.tx_coding_format.id = 0x03; @@ -364,11 +366,11 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data) if (conn->parent && lmp_esco_capable(conn->parent)) { if (!find_next_esco_param(conn, esco_param_cvsd, ARRAY_SIZE(esco_param_cvsd))) - return -EINVAL; + goto unlock; param = &esco_param_cvsd[conn->attempt - 1]; } else { if (conn->attempt > ARRAY_SIZE(sco_param_cvsd)) - return -EINVAL; + goto unlock; param = &sco_param_cvsd[conn->attempt - 1]; } cp.tx_coding_format.id = 2; @@ -391,9 +393,11 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data) cp.out_transport_unit_size = 16; break; default: - return -EINVAL; + goto unlock; } + hci_dev_unlock(hdev); + cp.retrans_effort = param->retrans_effort; cp.pkt_type = __cpu_to_le16(param->pkt_type); cp.max_latency = __cpu_to_le16(param->max_latency); @@ -402,6 +406,10 @@ static int hci_enhanced_setup_sync(struct hci_dev *hdev, void *data) return -EIO; return 0; + +unlock: + hci_dev_unlock(hdev); + return -EINVAL; } static bool hci_setup_sync_conn(struct hci_conn *conn, __u16 handle) @@ -1011,6 +1019,8 @@ static struct hci_conn *__hci_conn_add(struct hci_dev *hdev, int type, bdaddr_t INIT_DELAYED_WORK(&conn->idle_work, hci_conn_idle); INIT_DELAYED_WORK(&conn->le_conn_timeout, le_conn_timeout); + spin_lock_init(&conn->proto_lock); + atomic_set(&conn->refcnt, 0); hci_dev_hold(hdev); diff --git a/net/bluetooth/hci_core.c b/net/bluetooth/hci_core.c index efc23bed5b9f..84d9a445e08d 100644 --- a/net/bluetooth/hci_core.c +++ b/net/bluetooth/hci_core.c @@ -1521,8 +1521,10 @@ int hci_remove_remote_oob_data(struct hci_dev *hdev, bdaddr_t *bdaddr, BT_DBG("%s removing %pMR (%u)", hdev->name, bdaddr, bdaddr_type); + mutex_lock(&hdev->remote_oob_lock); list_del(&data->list); kfree(data); + mutex_unlock(&hdev->remote_oob_lock); return 0; } @@ -1531,10 +1533,12 @@ void hci_remote_oob_data_clear(struct hci_dev *hdev) { struct oob_data *data, *n; + mutex_lock(&hdev->remote_oob_lock); list_for_each_entry_safe(data, n, &hdev->remote_oob_data, list) { list_del(&data->list); kfree(data); } + mutex_unlock(&hdev->remote_oob_lock); } int hci_add_remote_oob_data(struct hci_dev *hdev, bdaddr_t *bdaddr, @@ -1543,11 +1547,14 @@ int hci_add_remote_oob_data(struct hci_dev *hdev, bdaddr_t *bdaddr, { struct oob_data *data; + mutex_lock(&hdev->remote_oob_lock); data = hci_find_remote_oob_data(hdev, bdaddr, bdaddr_type); if (!data) { data = kmalloc(sizeof(*data), GFP_KERNEL); - if (!data) + if (!data) { + mutex_unlock(&hdev->remote_oob_lock); return -ENOMEM; + } bacpy(&data->bdaddr, bdaddr); data->bdaddr_type = bdaddr_type; @@ -1580,6 +1587,8 @@ int hci_add_remote_oob_data(struct hci_dev *hdev, bdaddr_t *bdaddr, BT_DBG("%s for %pMR", hdev->name, bdaddr); + mutex_unlock(&hdev->remote_oob_lock); + return 0; } @@ -2514,6 +2523,7 @@ struct hci_dev *hci_alloc_dev_priv(int sizeof_priv) mutex_init(&hdev->lock); mutex_init(&hdev->req_lock); mutex_init(&hdev->mgmt_pending_lock); + mutex_init(&hdev->remote_oob_lock); ida_init(&hdev->unset_handle_ida); @@ -2587,8 +2597,10 @@ int hci_register_dev(struct hci_dev *hdev) return id; error = dev_set_name(&hdev->dev, "hci%u", id); - if (error) + if (error) { + ida_free(&hci_index_ida, id); return error; + } hdev->name = dev_name(&hdev->dev); hdev->id = id; @@ -2701,6 +2713,8 @@ void hci_unregister_dev(struct hci_dev *hdev) cancel_work_sync(&hdev->tx_work); cancel_work_sync(&hdev->power_on); cancel_work_sync(&hdev->error_reset); + cancel_delayed_work_sync(&hdev->cmd_timer); + cancel_delayed_work_sync(&hdev->ncmd_timer); hci_cmd_sync_clear(hdev); @@ -3221,6 +3235,17 @@ static void hci_queue_acl(struct hci_chan *chan, struct sk_buff_head *queue, } } +/* Queue hdev->tx_work, unless hdev->workqueue is being drained by + * hci_dev_close_sync(), which would otherwise WARN and drop the work. + */ +static void hci_sched_tx(struct hci_dev *hdev) +{ + rcu_read_lock(); + if (!hci_dev_test_flag(hdev, HCI_CMD_DRAIN_WORKQUEUE)) + queue_work(hdev->workqueue, &hdev->tx_work); + rcu_read_unlock(); +} + void hci_send_acl(struct hci_chan *chan, struct sk_buff *skb, __u16 flags) { struct hci_dev *hdev = chan->conn->hdev; @@ -3229,7 +3254,7 @@ void hci_send_acl(struct hci_chan *chan, struct sk_buff *skb, __u16 flags) hci_queue_acl(chan, &chan->data_q, skb, flags); - queue_work(hdev->workqueue, &hdev->tx_work); + hci_sched_tx(hdev); } /* Send SCO data */ @@ -3250,7 +3275,7 @@ void hci_send_sco(struct hci_conn *conn, struct sk_buff *skb) hci_skb_pkt_type(skb) = HCI_SCODATA_PKT; skb_queue_tail(&conn->data_q, skb); - queue_work(hdev->workqueue, &hdev->tx_work); + hci_sched_tx(hdev); } /* Send ISO data */ @@ -3294,6 +3319,8 @@ static void hci_queue_iso(struct hci_conn *conn, struct sk_buff_head *queue, skb_shinfo(skb)->frag_list = NULL; + spin_lock_bh(&queue->lock); + __skb_queue_tail(queue, skb); do { @@ -3308,6 +3335,8 @@ static void hci_queue_iso(struct hci_conn *conn, struct sk_buff_head *queue, __skb_queue_tail(queue, skb); } while (list); + + spin_unlock_bh(&queue->lock); } } @@ -3319,7 +3348,7 @@ void hci_send_iso(struct hci_conn *conn, struct sk_buff *skb) hci_queue_iso(conn, &conn->data_q, skb); - queue_work(hdev->workqueue, &hdev->tx_work); + hci_sched_tx(hdev); } /* ---- HCI TX task (outgoing data) ---- */ @@ -3369,9 +3398,6 @@ static struct hci_conn *hci_low_sent(struct hci_dev *hdev, __u8 type, struct hci_conn *conn = NULL, *c; unsigned int num = 0, min = ~0; - /* We don't have to lock device here. Connections are always - * added and removed with TX task disabled. */ - rcu_read_lock(); list_for_each_entry_rcu(c, &h->list, list) { @@ -3566,6 +3592,8 @@ static void hci_sched_sco(struct hci_dev *hdev, __u8 type) int quote, *cnt; unsigned int pkts = hdev->sco_pkts; + lockdep_assert_held(&hdev->lock); + bt_dev_dbg(hdev, "type %u", type); if (!hci_conn_num(hdev, type) || !pkts) @@ -3609,6 +3637,8 @@ static void hci_sched_acl_pkt(struct hci_dev *hdev) __check_timeout(hdev, cnt, ACL_LINK); + hci_dev_lock(hdev); + while (hdev->acl_cnt && (chan = hci_chan_sent(hdev, ACL_LINK, "e))) { u32 priority = (skb_peek(&chan->data_q))->priority; @@ -3640,6 +3670,8 @@ static void hci_sched_acl_pkt(struct hci_dev *hdev) if (cnt != hdev->acl_cnt) hci_prio_recalculate(hdev, ACL_LINK); + + hci_dev_unlock(hdev); } static void hci_sched_acl(struct hci_dev *hdev) @@ -3668,6 +3700,8 @@ static void hci_sched_le(struct hci_dev *hdev) __check_timeout(hdev, *cnt, LE_LINK); + hci_dev_lock(hdev); + tmp = *cnt; while (*cnt && (chan = hci_chan_sent(hdev, LE_LINK, "e))) { u32 priority = (skb_peek(&chan->data_q))->priority; @@ -3696,6 +3730,8 @@ static void hci_sched_le(struct hci_dev *hdev) if (*cnt != tmp) hci_prio_recalculate(hdev, LE_LINK); + + hci_dev_unlock(hdev); } /* Schedule CIS */ @@ -3713,6 +3749,9 @@ static void hci_sched_iso(struct hci_dev *hdev) cnt = hdev->iso_pkts ? &hdev->iso_cnt : hdev->le_pkts ? &hdev->le_cnt : &hdev->acl_cnt; + + hci_dev_lock(hdev); + while (*cnt && (conn = hci_low_sent(hdev, CIS_LINK, BIS_LINK, "e))) { while (quote-- && (skb = skb_dequeue(&conn->data_q))) { @@ -3725,6 +3764,8 @@ static void hci_sched_iso(struct hci_dev *hdev) (*cnt)--; } } + + hci_dev_unlock(hdev); } static void hci_tx_work(struct work_struct *work) @@ -3737,8 +3778,10 @@ static void hci_tx_work(struct work_struct *work) if (!hci_dev_test_flag(hdev, HCI_USER_CHANNEL)) { /* Schedule queues and send stuff to HCI driver */ + hci_dev_lock(hdev); hci_sched_sco(hdev, SCO_LINK); hci_sched_sco(hdev, ESCO_LINK); + hci_dev_unlock(hdev); hci_sched_iso(hdev); hci_sched_acl(hdev); hci_sched_le(hdev); @@ -4045,9 +4088,11 @@ static void hci_send_cmd_sync(struct hci_dev *hdev, struct sk_buff *skb) bt_dev_dbg(hdev, "skb %p", skb); + hci_dev_lock(hdev); kfree_skb(hdev->sent_cmd); hdev->sent_cmd = skb_clone(skb, GFP_KERNEL); + hci_dev_unlock(hdev); if (!hdev->sent_cmd) { skb_queue_head(&hdev->cmd_q, skb); queue_work(hdev->workqueue, &hdev->cmd_work); @@ -4062,8 +4107,10 @@ static void hci_send_cmd_sync(struct hci_dev *hdev, struct sk_buff *skb) if (READ_ONCE(hdev->req_status) == HCI_REQ_PEND && !hci_dev_test_and_set_flag(hdev, HCI_CMD_PENDING)) { + hci_dev_lock(hdev); kfree_skb(hdev->req_skb); hdev->req_skb = skb_get(hdev->sent_cmd); + hci_dev_unlock(hdev); } atomic_dec(&hdev->cmd_cnt); diff --git a/net/bluetooth/hci_debugfs.c b/net/bluetooth/hci_debugfs.c index ce3ff2fa72e5..2b07efb770d4 100644 --- a/net/bluetooth/hci_debugfs.c +++ b/net/bluetooth/hci_debugfs.c @@ -368,7 +368,7 @@ static int inquiry_cache_show(struct seq_file *f, void *p) list_for_each_entry(e, &cache->all, all) { struct inquiry_data *data = &e->data; - seq_printf(f, "%pMR %d %d %d 0x%.2x%.2x%.2x 0x%.4x %d %d %u\n", + seq_printf(f, "%pMR %d %d %d 0x%.2x%.2x%.2x 0x%.4x %d %d %lu\n", &data->bdaddr, data->pscan_rep_mode, data->pscan_period_mode, data->pscan_mode, data->dev_class[2], diff --git a/net/bluetooth/hci_event.c b/net/bluetooth/hci_event.c index 1a6fb75f2c1e..4c3fd5631c64 100644 --- a/net/bluetooth/hci_event.c +++ b/net/bluetooth/hci_event.c @@ -81,6 +81,10 @@ static void *hci_le_ev_skb_pull(struct hci_dev *hdev, struct sk_buff *skb, return data; } +static void hci_store_wake_reason(struct hci_dev *hdev, + const bdaddr_t *bdaddr, u8 addr_type) + __must_hold(&hdev->lock); + static u8 hci_cc_inquiry_cancel(struct hci_dev *hdev, void *data, struct sk_buff *skb) { @@ -1754,6 +1758,13 @@ static void le_set_scan_enable_complete(struct hci_dev *hdev, u8 enable) hci_dev_clear_flag(hdev, HCI_LE_SCAN); + if (hdev->discovery.type == DISCOV_TYPE_INTERLEAVED && + test_bit(HCI_QUIRK_SIMULTANEOUS_DISCOVERY, &hdev->quirks) && + !test_bit(HCI_INQUIRY, &hdev->flags) && + hdev->discovery.state == DISCOVERY_FINDING) { + hci_discovery_set_state(hdev, DISCOVERY_STOPPED); + } + /* The HCI_LE_SCAN_INTERRUPTED flag indicates that we * interrupted scanning due to a connect request. Mark * therefore discovery as stopped. @@ -3051,6 +3062,7 @@ static void hci_conn_complete_evt(struct hci_dev *hdev, void *data, bt_dev_dbg(hdev, "status 0x%2.2x", status); hci_dev_lock(hdev); + hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR); /* Check for existing connection: * @@ -3225,6 +3237,10 @@ static void hci_conn_request_evt(struct hci_dev *hdev, void *data, bt_dev_dbg(hdev, "bdaddr %pMR type 0x%x", &ev->bdaddr, ev->link_type); + hci_dev_lock(hdev); + hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR); + hci_dev_unlock(hdev); + /* Reject incoming connection from device with same BD ADDR against * CVE-2020-26555 */ @@ -4944,6 +4960,7 @@ static void hci_sync_conn_complete_evt(struct hci_dev *hdev, void *data, bt_dev_dbg(hdev, "status 0x%2.2x", status); hci_dev_lock(hdev); + hci_store_wake_reason(hdev, &ev->bdaddr, BDADDR_BREDR); conn = hci_conn_hash_lookup_ba(hdev, ev->link_type, &ev->bdaddr); if (!conn) { @@ -5645,6 +5662,7 @@ static void le_conn_complete_evt(struct hci_dev *hdev, u8 status, u8 addr_type; hci_dev_lock(hdev); + hci_store_wake_reason(hdev, bdaddr, bdaddr_type); /* Advertising stops when a connection is created. On a failed * connection it keeps running, so leave the state bit alone. @@ -6263,6 +6281,8 @@ static void hci_le_adv_report_evt(struct hci_dev *hdev, void *data, info->length + 1)) break; + hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type); + if (info->length <= max_adv_len(hdev)) { rssi = info->data[info->length]; process_adv_report(hdev, info->type, &info->bdaddr, @@ -6350,6 +6370,8 @@ static void hci_le_ext_adv_report_evt(struct hci_dev *hdev, void *data, info->length)) break; + hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type); + evt_type = __le16_to_cpu(info->type) & LE_EXT_ADV_EVT_TYPE_MASK; legacy_evt_type = ext_evt_type_to_legacy(hdev, evt_type); if (legacy_evt_type != LE_ADV_INVALID) { @@ -6385,6 +6407,7 @@ static void hci_le_pa_sync_estabilished_evt(struct hci_dev *hdev, void *data, bt_dev_dbg(hdev, "status 0x%2.2x", ev->status); hci_dev_lock(hdev); + hci_store_wake_reason(hdev, &ev->bdaddr, ev->bdaddr_type); hci_dev_clear_flag(hdev, HCI_PA_SYNC); @@ -6642,6 +6665,8 @@ static void hci_le_direct_adv_report_evt(struct hci_dev *hdev, void *data, for (i = 0; i < ev->num; i++) { struct hci_ev_le_direct_adv_info *info = &ev->info[i]; + hci_store_wake_reason(hdev, &info->bdaddr, info->bdaddr_type); + process_adv_report(hdev, info->type, &info->bdaddr, info->bdaddr_type, &info->direct_addr, info->direct_addr_type, info->rssi, NULL, 0, @@ -7233,73 +7258,29 @@ static bool hci_get_cmd_complete(struct hci_dev *hdev, u16 opcode, return true; } -static void hci_store_wake_reason(struct hci_dev *hdev, u8 event, - struct sk_buff *skb) +static void hci_store_wake_reason(struct hci_dev *hdev, + const bdaddr_t *bdaddr, u8 addr_type) + __must_hold(&hdev->lock) { - struct hci_ev_le_advertising_info *adv; - struct hci_ev_le_direct_adv_info *direct_adv; - struct hci_ev_le_ext_adv_info *ext_adv; - const struct hci_ev_conn_complete *conn_complete = (void *)skb->data; - const struct hci_ev_conn_request *conn_request = (void *)skb->data; - - hci_dev_lock(hdev); + lockdep_assert_held(&hdev->lock); /* If we are currently suspended and this is the first BT event seen, * save the wake reason associated with the event. */ if (!hdev->suspended || hdev->wake_reason) - goto unlock; + return; + + if (!bdaddr) { + hdev->wake_reason = MGMT_WAKE_REASON_UNEXPECTED; + return; + } /* Default to remote wake. Values for wake_reason are documented in the * Bluez mgmt api docs. */ hdev->wake_reason = MGMT_WAKE_REASON_REMOTE_WAKE; - - /* Once configured for remote wakeup, we should only wake up for - * reconnections. It's useful to see which device is waking us up so - * keep track of the bdaddr of the connection event that woke us up. - */ - if (event == HCI_EV_CONN_REQUEST) { - bacpy(&hdev->wake_addr, &conn_request->bdaddr); - hdev->wake_addr_type = BDADDR_BREDR; - } else if (event == HCI_EV_CONN_COMPLETE) { - bacpy(&hdev->wake_addr, &conn_complete->bdaddr); - hdev->wake_addr_type = BDADDR_BREDR; - } else if (event == HCI_EV_LE_META) { - struct hci_ev_le_meta *le_ev = (void *)skb->data; - u8 subevent = le_ev->subevent; - u8 *ptr = &skb->data[sizeof(*le_ev)]; - u8 num_reports = *ptr; - - if ((subevent == HCI_EV_LE_ADVERTISING_REPORT || - subevent == HCI_EV_LE_DIRECT_ADV_REPORT || - subevent == HCI_EV_LE_EXT_ADV_REPORT) && - num_reports) { - adv = (void *)(ptr + 1); - direct_adv = (void *)(ptr + 1); - ext_adv = (void *)(ptr + 1); - - switch (subevent) { - case HCI_EV_LE_ADVERTISING_REPORT: - bacpy(&hdev->wake_addr, &adv->bdaddr); - hdev->wake_addr_type = adv->bdaddr_type; - break; - case HCI_EV_LE_DIRECT_ADV_REPORT: - bacpy(&hdev->wake_addr, &direct_adv->bdaddr); - hdev->wake_addr_type = direct_adv->bdaddr_type; - break; - case HCI_EV_LE_EXT_ADV_REPORT: - bacpy(&hdev->wake_addr, &ext_adv->bdaddr); - hdev->wake_addr_type = ext_adv->bdaddr_type; - break; - } - } - } else { - hdev->wake_reason = MGMT_WAKE_REASON_UNEXPECTED; - } - -unlock: - hci_dev_unlock(hdev); + bacpy(&hdev->wake_addr, bdaddr); + hdev->wake_addr_type = addr_type; } #define HCI_EV_VL(_op, _func, _min_len, _max_len) \ @@ -7544,14 +7525,15 @@ void hci_event_packet(struct hci_dev *hdev, struct sk_buff *skb) skb_pull(skb, HCI_EVENT_HDR_SIZE); - /* Store wake reason if we're suspended */ - hci_store_wake_reason(hdev, event, skb); - bt_dev_dbg(hdev, "event 0x%2.2x", event); hci_event_func(hdev, event, skb, &opcode, &status, &req_complete, &req_complete_skb); + hci_dev_lock(hdev); + hci_store_wake_reason(hdev, NULL, 0); + hci_dev_unlock(hdev); + if (req_complete) { req_complete(hdev, status, opcode); } else if (req_complete_skb) { diff --git a/net/bluetooth/hci_sock.c b/net/bluetooth/hci_sock.c index dbce6d63be18..a0ea56dbcc99 100644 --- a/net/bluetooth/hci_sock.c +++ b/net/bluetooth/hci_sock.c @@ -1218,8 +1218,10 @@ static int hci_sock_bind(struct socket *sock, struct sockaddr *addr, */ hdev = hci_pi(sk)->hdev; if (hdev && hci_dev_test_flag(hdev, HCI_UNREGISTER)) { + write_lock(&hci_sk_list.lock); hci_pi(sk)->hdev = NULL; sk->sk_state = BT_OPEN; + write_unlock(&hci_sk_list.lock); hci_dev_put(hdev); } hdev = NULL; diff --git a/net/bluetooth/hci_sync.c b/net/bluetooth/hci_sync.c index e3aa6c982e06..4c6779575878 100644 --- a/net/bluetooth/hci_sync.c +++ b/net/bluetooth/hci_sync.c @@ -32,8 +32,10 @@ static void hci_cmd_sync_complete(struct hci_dev *hdev, u8 result, u16 opcode, WRITE_ONCE(hdev->req_status, HCI_REQ_DONE); /* Free the request command so it is not used as response */ + hci_dev_lock(hdev); kfree_skb(hdev->req_skb); hdev->req_skb = NULL; + hci_dev_unlock(hdev); if (skb) { struct sock *sk = hci_skb_sk(skb); @@ -313,6 +315,10 @@ static void hci_cmd_sync_work(struct work_struct *work) while (1) { struct hci_cmd_sync_work_entry *entry; + /* Leave the backlog to hci_cmd_sync_clear() */ + if (hci_dev_test_flag(hdev, HCI_UNREGISTER)) + break; + mutex_lock(&hdev->cmd_sync_work_lock); entry = list_first_entry_or_null(&hdev->cmd_sync_work_list, struct hci_cmd_sync_work_entry, @@ -682,6 +688,8 @@ void hci_cmd_sync_clear(struct hci_dev *hdev) { struct hci_cmd_sync_work_entry *entry, *tmp; + /* cmd_work is disabled, the pending request can only time out */ + hci_cmd_sync_cancel_sync(hdev, ENODEV); cancel_work_sync(&hdev->cmd_sync_work); cancel_work_sync(&hdev->reenable_adv_work); @@ -810,7 +818,7 @@ int hci_cmd_sync_queue_once(struct hci_dev *hdev, hci_cmd_sync_work_func_t func, void *data, hci_cmd_sync_work_destroy_t destroy) { if (hci_cmd_sync_lookup_entry(hdev, func, data, destroy)) - return 0; + return -EEXIST; return hci_cmd_sync_queue(hdev, func, data, destroy); } @@ -1328,7 +1336,8 @@ static int hci_set_ext_adv_data_sync(struct hci_dev *hdev, u8 instance) } } - len = eir_create_adv_data(hdev, instance, pdu.data); + len = eir_create_adv_data(hdev, instance, pdu.data, + sizeof(pdu.data)); pdu.cp.length = len; pdu.cp.handle = instance; @@ -1370,7 +1379,7 @@ static int hci_set_adv_data_sync(struct hci_dev *hdev, u8 instance) memset(&cp, 0, sizeof(cp)); - len = eir_create_adv_data(hdev, instance, cp.data); + len = eir_create_adv_data(hdev, instance, cp.data, sizeof(cp.data)); /* There's nothing to do if the data hasn't changed */ if (hdev->adv_data_len == len && @@ -3044,13 +3053,19 @@ static int hci_le_set_ext_scan_param_sync(struct hci_dev *hdev, u8 type, */ if (hci_dev_test_flag(hdev, HCI_PA_SYNC)) { struct hci_cp_le_add_to_accept_list *sent; + bdaddr_t bdaddr; + hci_dev_lock(hdev); sent = hci_sent_cmd_data(hdev, HCI_OP_LE_ADD_TO_ACCEPT_LIST); + if (sent) + bacpy(&bdaddr, &sent->bdaddr); + hci_dev_unlock(hdev); + if (sent) { struct hci_conn *conn; conn = hci_conn_hash_lookup_ba(hdev, BIS_LINK, - &sent->bdaddr); + &bdaddr); if (conn) { struct bt_iso_qos *qos = &conn->iso_qos; @@ -3361,6 +3376,8 @@ static int update_passive_scan_sync(struct hci_dev *hdev, void *data) int hci_update_passive_scan(struct hci_dev *hdev) { + int err; + /* Only queue if it would have any effect */ if (!test_bit(HCI_UP, &hdev->flags) || test_bit(HCI_INIT, &hdev->flags) || @@ -3370,8 +3387,9 @@ int hci_update_passive_scan(struct hci_dev *hdev) hci_dev_test_flag(hdev, HCI_UNREGISTER)) return 0; - return hci_cmd_sync_queue_once(hdev, update_passive_scan_sync, NULL, - NULL); + err = hci_cmd_sync_queue_once(hdev, update_passive_scan_sync, NULL, + NULL); + return (err == -EEXIST) ? 0 : err; } int hci_write_sc_support_sync(struct hci_dev *hdev, u8 val) @@ -5343,6 +5361,12 @@ int hci_dev_close_sync(struct hci_dev *hdev) bt_dev_dbg(hdev, ""); + /* Set HCI_DRAIN_WORKQUEUE flag to prevent queuing work during + * reset/close. See hci_cmd_work() and handle_cmd_cnt_and_timer(). + */ + hci_dev_set_flag(hdev, HCI_CMD_DRAIN_WORKQUEUE); + synchronize_rcu(); + cancel_delayed_work(&hdev->power_off); cancel_delayed_work(&hdev->ncmd_timer); cancel_delayed_work(&hdev->le_scan_disable); @@ -5361,6 +5385,7 @@ int hci_dev_close_sync(struct hci_dev *hdev) if (!test_and_clear_bit(HCI_UP, &hdev->flags)) { cancel_delayed_work_sync(&hdev->cmd_timer); + hci_dev_clear_flag(hdev, HCI_CMD_DRAIN_WORKQUEUE); return err; } @@ -5461,6 +5486,7 @@ int hci_dev_close_sync(struct hci_dev *hdev) /* Clear flags */ hdev->flags &= BIT(HCI_RAW); hci_dev_clear_volatile_flags(hdev); + hci_dev_clear_flag(hdev, HCI_CMD_DRAIN_WORKQUEUE); memset(hdev->eir, 0, sizeof(hdev->eir)); memset(hdev->dev_class, 0, sizeof(hdev->dev_class)); @@ -5556,6 +5582,7 @@ int hci_stop_discovery_sync(struct hci_dev *hdev) { struct discovery_state *d = &hdev->discovery; struct inquiry_entry *e; + bdaddr_t addr; int err; bt_dev_dbg(hdev, "state %u", hdev->discovery.state); @@ -5592,15 +5619,21 @@ int hci_stop_discovery_sync(struct hci_dev *hdev) return 0; if (d->state == DISCOVERY_RESOLVING || d->state == DISCOVERY_STOPPING) { + hci_dev_lock(hdev); e = hci_inquiry_cache_lookup_resolve(hdev, BDADDR_ANY, NAME_PENDING); - if (!e) + if (!e) { + hci_dev_unlock(hdev); return 0; + } + + bacpy(&addr, &e->data.bdaddr); + hci_dev_unlock(hdev); /* Ignore cancel errors since it should interfere with stopping * of the discovery. */ - hci_remote_name_cancel_sync(hdev, &e->data.bdaddr); + hci_remote_name_cancel_sync(hdev, &addr); } return 0; @@ -7041,6 +7074,7 @@ static int hci_acl_create_conn_sync(struct hci_dev *hdev, void *data) bacpy(&cp.bdaddr, &conn->dst); cp.pscan_rep_mode = 0x02; + hci_dev_lock(hdev); ie = hci_inquiry_cache_lookup(hdev, &conn->dst); if (ie) { if (inquiry_entry_age(ie) <= INQUIRY_ENTRY_AGE_MAX) { @@ -7052,6 +7086,7 @@ static int hci_acl_create_conn_sync(struct hci_dev *hdev, void *data) memcpy(conn->dev_class, ie->data.dev_class, 3); } + hci_dev_unlock(hdev); cp.pkt_type = cpu_to_le16(conn->pkt_type); if (lmp_rswitch_capable(hdev) && !(hdev->link_mode & HCI_LM_MASTER)) @@ -7080,10 +7115,24 @@ static int hci_acl_create_conn_sync(struct hci_dev *hdev, void *data) return err; } +static void hci_acl_create_conn_sync_complete(struct hci_dev *hdev, void *data, + int err) +{ + struct hci_conn *conn = data; + + hci_conn_put(conn); +} + int hci_connect_acl_sync(struct hci_dev *hdev, struct hci_conn *conn) { - return hci_cmd_sync_queue_once(hdev, hci_acl_create_conn_sync, conn, - NULL); + int err; + + err = hci_cmd_sync_queue_once(hdev, hci_acl_create_conn_sync, + hci_conn_get(conn), + hci_acl_create_conn_sync_complete); + if (err) + hci_conn_put(conn); + return (err == -EEXIST) ? 0 : err; } static void create_le_conn_complete(struct hci_dev *hdev, void *data, int err) @@ -7093,34 +7142,47 @@ static void create_le_conn_complete(struct hci_dev *hdev, void *data, int err) bt_dev_dbg(hdev, "err %d", err); if (err == -ECANCELED) - return; + goto done; hci_dev_lock(hdev); if (!hci_conn_valid(hdev, conn)) - goto done; + goto unlock; if (!err) { hci_connect_le_scan_cleanup(conn, 0x00); - goto done; + goto unlock; } - /* Check if connection is still pending */ - if (conn != hci_lookup_le_connect(hdev)) - goto done; + /* Check if this connection is still pending. + * + * hci_lookup_le_connect() returns only the first LE connection + * in BT_CONNECT, which is not necessarily this one when two are + * pending at once, so ask the connection itself. + */ + if (conn->state != BT_CONNECT) + goto unlock; /* Flush to make sure we send create conn cancel command if needed */ flush_delayed_work(&conn->le_conn_timeout); hci_conn_failed(conn, bt_status(err)); -done: +unlock: hci_dev_unlock(hdev); +done: + hci_conn_put(conn); } int hci_connect_le_sync(struct hci_dev *hdev, struct hci_conn *conn) { - return hci_cmd_sync_queue_once(hdev, hci_le_create_conn_sync, conn, - create_le_conn_complete); + int err; + + err = hci_cmd_sync_queue_once(hdev, hci_le_create_conn_sync, + hci_conn_get(conn), + create_le_conn_complete); + if (err) + hci_conn_put(conn); + return (err == -EEXIST) ? 0 : err; } static int hci_acl_cancel_create_conn_sync(struct hci_dev *hdev, diff --git a/net/bluetooth/hci_sysfs.c b/net/bluetooth/hci_sysfs.c index 60350c6723cb..feb4f1302b37 100644 --- a/net/bluetooth/hci_sysfs.c +++ b/net/bluetooth/hci_sysfs.c @@ -13,7 +13,10 @@ static const struct class bt_class = { static void bt_link_release(struct device *dev) { struct hci_conn *conn = to_hci_conn(dev); + struct device *parent = dev->parent; + kfree(conn); + put_device(parent); } static const struct device_type bt_link = { @@ -21,6 +24,16 @@ static const struct device_type bt_link = { .release = bt_link_release, }; +/* + * The rfcomm tty device will possibly retain even when conn + * is down, and sysfs doesn't support move zombie device, + * so we should move the device before conn device is destroyed. + */ +static int __match_tty(struct device *dev, void *data) +{ + return !strncmp(dev_name(dev), "rfcomm", 6); +} + void hci_conn_init_sysfs(struct hci_conn *conn) { struct hci_dev *hdev = conn->hdev; @@ -29,7 +42,7 @@ void hci_conn_init_sysfs(struct hci_conn *conn) conn->dev.type = &bt_link; conn->dev.class = &bt_class; - conn->dev.parent = &hdev->dev; + conn->dev.parent = get_device(&hdev->dev); device_initialize(&conn->dev); } @@ -69,7 +82,7 @@ void hci_conn_del_sysfs(struct hci_conn *conn) while (1) { struct device *dev; - dev = device_find_any_child(&conn->dev); + dev = device_find_child(&conn->dev, NULL, __match_tty); if (!dev) break; device_move(dev, NULL, DPM_ORDER_DEV_LAST); diff --git a/net/bluetooth/iso.c b/net/bluetooth/iso.c index 23bd4d8c397f..217ba98b053b 100644 --- a/net/bluetooth/iso.c +++ b/net/bluetooth/iso.c @@ -55,6 +55,7 @@ enum { BT_SK_BIG_SYNC, BT_SK_PA_SYNC, BT_SK_PA_SYNC_TERM, + BT_SK_CONNECTING, }; struct iso_pinfo { @@ -277,6 +278,9 @@ static int __iso_chan_add(struct iso_conn *conn, struct sock *sk, return -EBUSY; } + if (iso_pi(sk)->conn) + return -EISCONN; + iso_pi(sk)->conn = conn; conn->sk = sk; @@ -321,6 +325,12 @@ static int iso_connect_bis(struct sock *sk) return -EHOSTUNREACH; hci_dev_lock(hdev); + lock_sock(sk); + + if (iso_pi(sk)->conn) { + err = -EISCONN; + goto unlock; + } if (!bis_capable(hdev)) { err = -EOPNOTSUPP; @@ -367,13 +377,9 @@ static int iso_connect_bis(struct sock *sk) goto unlock; } - lock_sock(sk); - err = iso_chan_add(conn, sk, NULL); - if (err) { - release_sock(sk); + if (err) goto unlock; - } /* Update source addr of the socket */ bacpy(&iso_pi(sk)->src, &hcon->src); @@ -389,9 +395,8 @@ static int iso_connect_bis(struct sock *sk) iso_sock_set_timer(sk, sk->sk_sndtimeo); } - release_sock(sk); - unlock: + release_sock(sk); hci_dev_unlock(hdev); hci_dev_put(hdev); return err; @@ -412,6 +417,7 @@ static int iso_connect_cis(struct sock *sk) return -EHOSTUNREACH; hci_dev_lock(hdev); + lock_sock(sk); if (!cis_central_capable(hdev)) { err = -EOPNOTSUPP; @@ -450,6 +456,13 @@ static int iso_connect_cis(struct sock *sk) } } + /* The socket lock keeps the current attachment and its hcon stable. */ + if (iso_pi(sk)->conn && iso_pi(sk)->conn->hcon != hcon) { + hci_conn_drop(hcon); + err = -EISCONN; + goto unlock; + } + conn = iso_conn_add(hcon); if (!conn) { hci_conn_drop(hcon); @@ -457,13 +470,9 @@ static int iso_connect_cis(struct sock *sk) goto unlock; } - lock_sock(sk); - err = iso_chan_add(conn, sk, NULL); - if (err) { - release_sock(sk); + if (err) goto unlock; - } /* Update source addr of the socket */ bacpy(&iso_pi(sk)->src, &hcon->src); @@ -479,9 +488,8 @@ static int iso_connect_cis(struct sock *sk) iso_sock_set_timer(sk, sk->sk_sndtimeo); } - release_sock(sk); - unlock: + release_sock(sk); hci_dev_unlock(hdev); hci_dev_put(hdev); return err; @@ -727,11 +735,13 @@ static void iso_sock_kill(struct sock *sk) BT_DBG("sk %p state %d", sk, sk->sk_state); /* Sock is dead, so set conn->sk to NULL to avoid possible UAF */ + lock_sock(sk); if (iso_pi(sk)->conn) { iso_conn_lock(iso_pi(sk)->conn); iso_pi(sk)->conn->sk = NULL; iso_conn_unlock(iso_pi(sk)->conn); } + release_sock(sk); /* Kill poor orphan */ bt_sock_unlink(&iso_sk_list, sk); @@ -1000,17 +1010,26 @@ static int iso_sock_connect(struct socket *sock, struct sockaddr *addr, addr->sa_family != AF_BLUETOOTH) return -EINVAL; - if (sk->sk_state != BT_OPEN && sk->sk_state != BT_BOUND) - return -EBADFD; + lock_sock(sk); - if (sk->sk_type != SOCK_SEQPACKET) - return -EINVAL; + if ((sk->sk_state != BT_OPEN && sk->sk_state != BT_BOUND) || + test_bit(BT_SK_CONNECTING, &iso_pi(sk)->flags)) { + err = -EBADFD; + goto done; + } + + if (sk->sk_type != SOCK_SEQPACKET) { + err = -EINVAL; + goto done; + } /* Check if the address type is of LE type */ - if (!bdaddr_type_is_le(sa->iso_bdaddr_type)) - return -EINVAL; + if (!bdaddr_type_is_le(sa->iso_bdaddr_type)) { + err = -EINVAL; + goto done; + } - lock_sock(sk); + set_bit(BT_SK_CONNECTING, &iso_pi(sk)->flags); bacpy(&iso_pi(sk)->dst, &sa->iso_bdaddr); iso_pi(sk)->dst_type = sa->iso_bdaddr_type; @@ -1022,16 +1041,18 @@ static int iso_sock_connect(struct socket *sock, struct sockaddr *addr, else err = iso_connect_bis(sk); - if (err) - return err; - lock_sock(sk); + clear_bit(BT_SK_CONNECTING, &iso_pi(sk)->flags); + if (err) + goto done; + if (!test_bit(BT_SK_DEFER_SETUP, &bt_sk(sk)->flags)) { err = bt_sock_wait_state(sk, BT_CONNECTED, sock_sndtimeo(sk, flags & O_NONBLOCK)); } +done: release_sock(sk); return err; } @@ -1200,6 +1221,7 @@ static int iso_sock_getname(struct socket *sock, struct sockaddr *addr, BT_DBG("sock %p, sk %p", sock, sk); + memset(sa, 0, sizeof(struct sockaddr_iso)); addr->sa_family = AF_BLUETOOTH; if (peer) { @@ -1677,14 +1699,11 @@ static void iso_sock_ready(struct sock *sk) { BT_DBG("sk %p", sk); - if (!sk) - return; + lockdep_assert(lockdep_sock_is_held(sk)); - lock_sock(sk); iso_sock_clear_timer(sk); sk->sk_state = BT_CONNECTED; sk->sk_state_change(sk); - release_sock(sk); } struct iso_list_data { @@ -1707,7 +1726,7 @@ static bool iso_match_pa_sync_flag(struct sock *sk, void *data) static void iso_conn_ready(struct iso_conn *conn) { struct sock *parent = NULL; - struct sock *sk = conn->sk; + struct sock *sk; struct hci_ev_le_big_sync_estabilished *ev = NULL; struct hci_ev_le_pa_sync_established *ev2 = NULL; struct hci_evt_le_big_info_adv_report *ev3 = NULL; @@ -1715,8 +1734,26 @@ static void iso_conn_ready(struct iso_conn *conn) BT_DBG("conn %p", conn); + iso_conn_lock(conn); + sk = iso_sock_hold(conn); + iso_conn_unlock(conn); + if (sk) { - iso_sock_ready(conn->sk); + lock_sock(sk); + + /* conn->sk may have become NULL if racing with sk close, but + * due to held hdev->lock, it can't become different sk. + */ + if (!conn->sk) { + release_sock(sk); + sock_put(sk); + return; + } + + iso_sock_ready(sk); + + release_sock(sk); + sock_put(sk); } else { hcon = conn->hcon; if (!hcon) diff --git a/net/bluetooth/l2cap_core.c b/net/bluetooth/l2cap_core.c index 0b4b7fcca987..c2bd2b25a809 100644 --- a/net/bluetooth/l2cap_core.c +++ b/net/bluetooth/l2cap_core.c @@ -1325,7 +1325,7 @@ static void l2cap_le_connect(struct l2cap_chan *chan) struct l2cap_ecred_conn_data { struct { struct l2cap_ecred_conn_req req; - __le16 scid[5]; + __le16 scid[L2CAP_ECRED_CONN_SCID_MAX]; } __packed pdu; struct l2cap_chan *chan; struct pid *pid; @@ -1356,6 +1356,10 @@ static void l2cap_ecred_defer_connect(struct l2cap_chan *chan, void *data) if (test_and_set_bit(FLAG_ECRED_CONN_REQ_SENT, &chan->flags)) return; + /* Unreachable, checked in l2cap_connect (+timer drops it if reached) */ + if (WARN_ON_ONCE(conn->count >= ARRAY_SIZE(conn->pdu.scid))) + return; + l2cap_ecred_init(chan, 0); /* Set the same ident so we can match on the rsp */ @@ -1824,7 +1828,10 @@ static void l2cap_conn_del(struct hci_conn *hcon, int err) hci_chan_del(conn->hchan); conn->hchan = NULL; + spin_lock(&hcon->proto_lock); hcon->l2cap_data = NULL; + spin_unlock(&hcon->proto_lock); + mutex_unlock(&conn->lock); l2cap_conn_put(conn); } @@ -7105,8 +7112,6 @@ static struct l2cap_conn *l2cap_conn_add(struct hci_conn *hcon) } kref_init(&conn->ref); - hcon->l2cap_data = conn; - conn->hcon = hci_conn_get(hcon); conn->hchan = hchan; BT_DBG("hcon %p conn %p hchan %p", hcon, conn, hchan); @@ -7135,6 +7140,11 @@ static struct l2cap_conn *l2cap_conn_add(struct hci_conn *hcon) conn->disc_reason = HCI_ERROR_REMOTE_USER_TERM; + spin_lock(&hcon->proto_lock); + conn->hcon = hci_conn_get(hcon); + hcon->l2cap_data = conn; + spin_unlock(&hcon->proto_lock); + return conn; } @@ -7298,6 +7308,9 @@ int l2cap_chan_connect(struct l2cap_chan *chan, __le16 psm, u16 cid, goto done; } + mutex_lock(&conn->lock); + l2cap_chan_lock(chan); + if (chan->mode == L2CAP_MODE_EXT_FLOWCTL) { struct l2cap_chan_data data; @@ -7305,19 +7318,20 @@ int l2cap_chan_connect(struct l2cap_chan *chan, __le16 psm, u16 cid, data.pid = chan->ops->get_peer_pid(chan); data.count = 1; - l2cap_chan_list(conn, l2cap_chan_by_pid, &data); + __l2cap_chan_list(conn, l2cap_chan_by_pid, &data); + + /* Leave room for non-deferred channel that ends the group. */ + if (test_bit(FLAG_DEFER_SETUP, &chan->flags)) + data.count += 1; /* Check if there isn't too many channels being connected */ if (data.count > L2CAP_ECRED_CONN_SCID_MAX) { hci_conn_drop(hcon); err = -EPROTO; - goto done; + goto chan_unlock; } } - mutex_lock(&conn->lock); - l2cap_chan_lock(chan); - if (cid && __l2cap_get_chan_by_dcid(conn, cid)) { hci_conn_drop(hcon); err = -EBUSY; @@ -7524,13 +7538,18 @@ static void l2cap_connect_cfm(struct hci_conn *hcon, u8 status) int l2cap_disconn_ind(struct hci_conn *hcon) { - struct l2cap_conn *conn = hcon->l2cap_data; + struct l2cap_conn *conn; + int ret = HCI_ERROR_REMOTE_USER_TERM; BT_DBG("hcon %p", hcon); - if (!conn) - return HCI_ERROR_REMOTE_USER_TERM; - return conn->disc_reason; + spin_lock(&hcon->proto_lock); + conn = hcon->l2cap_data; + if (conn) + ret = conn->disc_reason; + spin_unlock(&hcon->proto_lock); + + return ret; } static void l2cap_disconn_cfm(struct hci_conn *hcon, u8 reason) diff --git a/net/bluetooth/l2cap_sock.c b/net/bluetooth/l2cap_sock.c index d09a0b0ffd0c..c0da2b1168a1 100644 --- a/net/bluetooth/l2cap_sock.c +++ b/net/bluetooth/l2cap_sock.c @@ -437,11 +437,26 @@ static int l2cap_get_mode(struct l2cap_chan *chan) return -EINVAL; } +static struct l2cap_conn *l2cap_chan_conn(struct l2cap_chan *chan) +{ + lockdep_assert_held(&chan->lock); + + /* l2cap_conn_del() sets FLAG_DEL while holding chan->lock before + * conn->hcon is deleted. If not set and conn is non-NULL, conn->hcon + * remains alive during this chan->lock critical section. + */ + if (test_bit(FLAG_DEL, &chan->flags)) + return NULL; + + return chan->conn; +} + static int l2cap_sock_getsockopt_old(struct socket *sock, int optname, char __user *optval, int __user *optlen) { struct sock *sk = sock->sk; struct l2cap_chan *chan = l2cap_pi(sk)->chan; + struct l2cap_conn *conn; struct l2cap_options opts; struct l2cap_conninfo cinfo; int err = 0; @@ -453,6 +468,7 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname, if (get_user(len, optlen)) return -EFAULT; + l2cap_chan_lock(chan); lock_sock(sk); switch (optname) { @@ -538,9 +554,15 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname, break; } + conn = l2cap_chan_conn(chan); + if (!conn) { + err = -ENOTCONN; + break; + } + memset(&cinfo, 0, sizeof(cinfo)); - cinfo.hci_handle = chan->conn->hcon->handle; - memcpy(cinfo.dev_class, chan->conn->hcon->dev_class, 3); + cinfo.hci_handle = conn->hcon->handle; + memcpy(cinfo.dev_class, conn->hcon->dev_class, 3); len = min(len, sizeof(cinfo)); if (copy_to_user(optval, (char *) &cinfo, len)) @@ -554,6 +576,8 @@ static int l2cap_sock_getsockopt_old(struct socket *sock, int optname, } release_sock(sk); + l2cap_chan_unlock(chan); + return err; } @@ -562,6 +586,7 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname, { struct sock *sk = sock->sk; struct l2cap_chan *chan = l2cap_pi(sk)->chan; + struct l2cap_conn *conn; struct bt_security sec; struct bt_power pwr; u32 phys; @@ -578,6 +603,7 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname, if (get_user(len, optlen)) return -EFAULT; + l2cap_chan_lock(chan); lock_sock(sk); switch (optname) { @@ -589,12 +615,14 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname, break; } + conn = l2cap_chan_conn(chan); + memset(&sec, 0, sizeof(sec)); - if (chan->conn) { - sec.level = chan->conn->hcon->sec_level; + if (conn) { + sec.level = conn->hcon->sec_level; if (sk->sk_state == BT_CONNECTED) - sec.key_size = chan->conn->hcon->enc_key_size; + sec.key_size = conn->hcon->enc_key_size; } else { sec.level = chan->sec_level; } @@ -670,12 +698,14 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname, break; case BT_PHY: - if (sk->sk_state != BT_CONNECTED) { + conn = l2cap_chan_conn(chan); + + if (sk->sk_state != BT_CONNECTED || !conn) { err = -ENOTCONN; break; } - phys = hci_conn_get_phy(chan->conn->hcon); + phys = hci_conn_get_phy(conn->hcon); if (put_user(phys, (u32 __user *) optval)) err = -EFAULT; @@ -708,6 +738,7 @@ static int l2cap_sock_getsockopt(struct socket *sock, int level, int optname, } release_sock(sk); + l2cap_chan_unlock(chan); return err; } @@ -738,6 +769,7 @@ static int l2cap_sock_setsockopt_old(struct socket *sock, int optname, BT_DBG("sk %p", sk); + l2cap_chan_lock(chan); lock_sock(sk); switch (optname) { @@ -838,6 +870,7 @@ static int l2cap_sock_setsockopt_old(struct socket *sock, int optname, } release_sock(sk); + l2cap_chan_unlock(chan); return err; } @@ -901,6 +934,7 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname, if (level != SOL_BLUETOOTH) return -ENOPROTOOPT; + l2cap_chan_lock(chan); lock_sock(sk); switch (optname) { @@ -926,11 +960,10 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname, chan->sec_level = sec.level; - if (!chan->conn) + conn = l2cap_chan_conn(chan); + if (!conn) break; - conn = chan->conn; - /* change security for LE channels */ if (chan->scid == L2CAP_CID_ATT) { if (smp_conn_security(conn->hcon, sec.level)) { @@ -985,7 +1018,8 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname, } if (opt == BT_FLUSHABLE_OFF) { - conn = chan->conn; + conn = l2cap_chan_conn(chan); + /* proceed further only when we have l2cap_conn and No Flush support in the LM */ if (!conn || !lmp_no_flush_capable(conn->hcon->hdev)) { @@ -1108,6 +1142,7 @@ static int l2cap_sock_setsockopt(struct socket *sock, int level, int optname, } release_sock(sk); + l2cap_chan_unlock(chan); return err; } diff --git a/net/bluetooth/mgmt.c b/net/bluetooth/mgmt.c index a678a4a57ce6..20984813a145 100644 --- a/net/bluetooth/mgmt.c +++ b/net/bluetooth/mgmt.c @@ -1462,7 +1462,7 @@ static void cmd_complete_rsp(struct mgmt_pending_cmd *cmd, void *data) return; } - cmd_status_rsp(cmd, data); + cmd_status_rsp(cmd, &match->mgmt_status); } static int generic_cmd_complete(struct mgmt_pending_cmd *cmd, u8 status) @@ -2162,19 +2162,25 @@ static void set_mesh_complete(struct hci_dev *hdev, void *data, int err) sk = cmd->sk; if (status) { - mgmt_pending_foreach(MGMT_OP_SET_MESH_RECEIVER, hdev, true, - cmd_status_rsp, &status); - return; + mgmt_cmd_status(cmd->sk, hdev->id, cmd->opcode, status); + goto done; } - mgmt_pending_remove(cmd); mgmt_cmd_complete(sk, hdev->id, MGMT_OP_SET_MESH_RECEIVER, 0, NULL, 0); + +done: + mgmt_pending_free(cmd); } static int set_mesh_sync(struct hci_dev *hdev, void *data) { struct mgmt_pending_cmd *cmd = data; - struct mgmt_cp_set_mesh cp; + union { + struct mgmt_cp_set_mesh cp; + u8 buf[sizeof(struct mgmt_cp_set_mesh) + + sizeof(hdev->mesh_ad_types)]; + } u = {}; + struct mgmt_cp_set_mesh *cp = &u.cp; size_t len; mutex_lock(&hdev->mgmt_pending_lock); @@ -2184,27 +2190,26 @@ static int set_mesh_sync(struct hci_dev *hdev, void *data) return -ECANCELED; } - memcpy(&cp, cmd->param, sizeof(cp)); + len = cmd->param_len; + memcpy(u.buf, cmd->param, min(sizeof(u.buf), len)); mutex_unlock(&hdev->mgmt_pending_lock); - len = cmd->param_len; - memset(hdev->mesh_ad_types, 0, sizeof(hdev->mesh_ad_types)); - if (cp.enable) + if (cp->enable) hci_dev_set_flag(hdev, HCI_MESH); else hci_dev_clear_flag(hdev, HCI_MESH); - hdev->le_scan_interval = __le16_to_cpu(cp.period); - hdev->le_scan_window = __le16_to_cpu(cp.window); + hdev->le_scan_interval = __le16_to_cpu(cp->period); + hdev->le_scan_window = __le16_to_cpu(cp->window); - len -= sizeof(cp); + len -= sizeof(struct mgmt_cp_set_mesh); /* If filters don't fit, forward all adv pkts */ if (len <= sizeof(hdev->mesh_ad_types)) - memcpy(hdev->mesh_ad_types, cp.ad_types, len); + memcpy(hdev->mesh_ad_types, cp->ad_types, len); hci_update_passive_scan_sync(hdev); return 0; @@ -5426,7 +5431,7 @@ static void mgmt_add_adv_patterns_monitor_complete(struct hci_dev *hdev, mgmt_cmd_complete(cmd->sk, cmd->hdev->id, cmd->opcode, mgmt_status(status), &rp, sizeof(rp)); - mgmt_pending_remove(cmd); + mgmt_pending_free(cmd); hci_dev_unlock(hdev); bt_dev_dbg(hdev, "add monitor %d complete, status %d", diff --git a/net/bluetooth/rfcomm/core.c b/net/bluetooth/rfcomm/core.c index 791283d1c1f7..199e70d37663 100644 --- a/net/bluetooth/rfcomm/core.c +++ b/net/bluetooth/rfcomm/core.c @@ -65,10 +65,9 @@ static void rfcomm_make_uih(struct sk_buff *skb, u8 addr); static void rfcomm_process_connect(struct rfcomm_session *s); -static struct rfcomm_session *rfcomm_session_create(bdaddr_t *src, - bdaddr_t *dst, - u8 sec_level, - int *err); +static struct socket *rfcomm_session_connect(bdaddr_t *src, bdaddr_t *dst, + u8 sec_level, int *err); +static struct rfcomm_session *rfcomm_session_add(struct socket *sock, int state); static struct rfcomm_session *rfcomm_session_get(bdaddr_t *src, bdaddr_t *dst); static struct rfcomm_session *rfcomm_session_del(struct rfcomm_session *s); @@ -368,28 +367,17 @@ static int rfcomm_check_channel(u8 channel) return channel < 1 || channel > 30; } -static int __rfcomm_dlc_open(struct rfcomm_dlc *d, bdaddr_t *src, bdaddr_t *dst, u8 channel) +static int __rfcomm_dlc_open(struct rfcomm_dlc *d, struct rfcomm_session *s, + u8 channel) { - struct rfcomm_session *s; - int err = 0; u8 dlci; - BT_DBG("dlc %p state %ld %pMR -> %pMR channel %d", - d, d->state, src, dst, channel); - - if (rfcomm_check_channel(channel)) - return -EINVAL; + BT_DBG("dlc %p state %ld session %p channel %d", + d, d->state, s, channel); if (d->state != BT_OPEN && d->state != BT_CLOSED) return 0; - s = rfcomm_session_get(src, dst); - if (!s) { - s = rfcomm_session_create(src, dst, d->sec_level, &err); - if (!s) - return err; - } - dlci = __dlci(__session_dir(s), channel); /* Check if DLCI already exists */ @@ -424,14 +412,84 @@ static int __rfcomm_dlc_open(struct rfcomm_dlc *d, bdaddr_t *src, bdaddr_t *dst, int rfcomm_dlc_open(struct rfcomm_dlc *d, bdaddr_t *src, bdaddr_t *dst, u8 channel) { - int r; + struct rfcomm_session *s; + struct socket *sock; + int err; + + BT_DBG("dlc %p state %ld %pMR -> %pMR channel %d", + d, d->state, src, dst, channel); + + if (rfcomm_check_channel(channel)) + return -EINVAL; rfcomm_lock(); - r = __rfcomm_dlc_open(d, src, dst, channel); + clear_bit(RFCOMM_CLOSED, &d->flags); + /* Do not page the remote device for a DLC that cannot be opened + * anyway. __rfcomm_dlc_open() looks at the state again once the + * lock has been re-acquired below. + */ + if (d->state != BT_OPEN && d->state != BT_CLOSED) { + rfcomm_unlock(); + return 0; + } + + s = rfcomm_session_get(src, dst); + if (s) { + err = __rfcomm_dlc_open(d, s, channel); + rfcomm_unlock(); + return err; + } rfcomm_unlock(); - return r; + + /* There is no session for this pair yet. kernel_connect() ends up in + * l2cap_chan_connect(), which takes hdev->lock, and the HCI event + * path takes rfcomm_mutex while holding hdev->lock, so the socket has + * to be connected with rfcomm_mutex released. + */ + sock = rfcomm_session_connect(src, dst, d->sec_level, &err); + if (!sock) + return err; + + rfcomm_lock(); + + /* The DLC may have been closed while the socket was connecting. It + * was not on a session, so rfcomm_dlc_close() could only mark it; + * attaching it now would leave it with no owner. + */ + if (test_bit(RFCOMM_CLOSED, &d->flags)) { + rfcomm_unlock(); + sock_release(sock); + return -ECONNRESET; + } + + /* Another opener may have added a session for the same pair in the + * meantime; that one is used and this socket is dropped. + */ + s = rfcomm_session_get(src, dst); + if (!s) { + s = rfcomm_session_add(sock, BT_BOUND); + if (s) { + s->initiator = 1; + sock = NULL; + } + } + + err = s ? __rfcomm_dlc_open(d, s, channel) : -ENOMEM; + + rfcomm_unlock(); + + if (sock) + sock_release(sock); + + /* Over an existing ACL link the connection can complete before the + * session reaches the list, and that wakeup is then lost, so let + * krfcommd look at the socket state now. + */ + rfcomm_schedule(); + + return err; } static void __rfcomm_dlc_disconn(struct rfcomm_dlc *d) @@ -511,8 +569,14 @@ int rfcomm_dlc_close(struct rfcomm_dlc *d, int err) rfcomm_lock(); s = d->session; - if (!s) + if (!s) { + /* Not on a session yet: rfcomm_dlc_open() may be connecting a + * socket for it with rfcomm_mutex released. Leave a mark so + * that it does not attach the DLC once it holds the lock again. + */ + set_bit(RFCOMM_CLOSED, &d->flags); goto no_session; + } /* after waiting on the mutex check the session still exists * then check the dlc still exists @@ -760,12 +824,12 @@ static struct rfcomm_session *rfcomm_session_close(struct rfcomm_session *s, return rfcomm_session_del(s); } -static struct rfcomm_session *rfcomm_session_create(bdaddr_t *src, - bdaddr_t *dst, - u8 sec_level, - int *err) +/* Creates the L2CAP socket a new session will run on and starts connecting + * it. Must be called with rfcomm_mutex released. + */ +static struct socket *rfcomm_session_connect(bdaddr_t *src, bdaddr_t *dst, + u8 sec_level, int *err) { - struct rfcomm_session *s = NULL; struct sockaddr_l2 addr; struct socket *sock; struct sock *sk; @@ -795,24 +859,16 @@ static struct rfcomm_session *rfcomm_session_create(bdaddr_t *src, l2cap_pi(sk)->chan->mode = L2CAP_MODE_ERTM; release_sock(sk); - s = rfcomm_session_add(sock, BT_BOUND); - if (!s) { - *err = -ENOMEM; - goto failed; - } - - s->initiator = 1; - bacpy(&addr.l2_bdaddr, dst); addr.l2_family = AF_BLUETOOTH; addr.l2_psm = cpu_to_le16(L2CAP_PSM_RFCOMM); addr.l2_cid = 0; addr.l2_bdaddr_type = BDADDR_BREDR; *err = kernel_connect(sock, (struct sockaddr *) &addr, sizeof(addr), O_NONBLOCK); - if (*err == 0 || *err == -EINPROGRESS) - return s; + if (*err && *err != -EINPROGRESS) + goto failed; - return rfcomm_session_del(s); + return sock; failed: sock_release(sock); diff --git a/net/bluetooth/rfcomm/sock.c b/net/bluetooth/rfcomm/sock.c index 3c5cdd42622a..8e61991d97c0 100644 --- a/net/bluetooth/rfcomm/sock.c +++ b/net/bluetooth/rfcomm/sock.c @@ -50,8 +50,11 @@ static void rfcomm_sock_kill(struct sock *sk); static void rfcomm_sk_data_ready(struct rfcomm_dlc *d, struct sk_buff *skb) { struct sock *sk = d->owner; - if (!sk) + + if (!sk) { + kfree_skb(skb); return; + } atomic_add(skb->len, &sk->sk_rmem_alloc); skb_queue_tail(&sk->sk_receive_queue, skb); diff --git a/net/bluetooth/rfcomm/tty.c b/net/bluetooth/rfcomm/tty.c index 3f732b5fdeec..31e42219d586 100644 --- a/net/bluetooth/rfcomm/tty.c +++ b/net/bluetooth/rfcomm/tty.c @@ -131,7 +131,7 @@ static void rfcomm_dev_shutdown(struct tty_port *port) { struct rfcomm_dev *dev = container_of(port, struct rfcomm_dev, port); - if (dev->tty_dev->parent) + if (dev->tty_dev && dev->tty_dev->parent) device_move(dev->tty_dev, NULL, DPM_ORDER_DEV_LAST); /* close the dlc */ @@ -465,7 +465,7 @@ static int __rfcomm_release_dev(void __user *arg) /* Shut down TTY synchronously before freeing rfcomm_dev */ tty_port_tty_vhangup(&dev->port); - if (!test_bit(RFCOMM_TTY_OWNED, &dev->status)) + if (!test_and_set_bit(RFCOMM_TTY_OWNED, &dev->status)) tty_port_put(&dev->port); tty_port_put(&dev->port); @@ -730,10 +730,9 @@ static int rfcomm_tty_install(struct tty_driver *driver, struct tty_struct *tty) * when the last process closes the tty. The behaviour is expected by * userspace. */ - if (test_bit(RFCOMM_RELEASE_ONHUP, &dev->flags)) { - set_bit(RFCOMM_TTY_OWNED, &dev->status); + if (test_bit(RFCOMM_RELEASE_ONHUP, &dev->flags) && + !test_and_set_bit(RFCOMM_TTY_OWNED, &dev->status)) tty_port_put(&dev->port); - } return 0; } diff --git a/net/bluetooth/smp.c b/net/bluetooth/smp.c index c87b39380de2..83eef7a02e0d 100644 --- a/net/bluetooth/smp.c +++ b/net/bluetooth/smp.c @@ -684,6 +684,7 @@ static void build_pairing_cmd(struct l2cap_conn *conn, else bdaddr_type = BDADDR_LE_RANDOM; + mutex_lock(&hdev->remote_oob_lock); oob_data = hci_find_remote_oob_data(hdev, &hcon->dst, bdaddr_type); if (oob_data && oob_data->present) { @@ -694,6 +695,7 @@ static void build_pairing_cmd(struct l2cap_conn *conn, SMP_DBG("OOB Remote Confirmation: %16phN", smp->pcnf); SMP_DBG("OOB Remote Random: %16phN", smp->rr); } + mutex_unlock(&hdev->remote_oob_lock); } else { authreq &= ~SMP_AUTH_SC; diff --git a/net/bridge/br_arp_nd_proxy.c b/net/bridge/br_arp_nd_proxy.c index f2d6643d2c0a..e1c6a54e9a0c 100644 --- a/net/bridge/br_arp_nd_proxy.c +++ b/net/bridge/br_arp_nd_proxy.c @@ -191,13 +191,15 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br, n = neigh_lookup(&arp_tbl, &tip, vlandev); if (n) { struct net_bridge_fdb_entry *f; + u8 ha[ETH_ALEN] __aligned(2); if (!(READ_ONCE(n->nud_state) & NUD_VALID)) { neigh_release(n); return; } - f = br_fdb_find_rcu(br, n->ha, vid); + neigh_ha_snapshot(ha, n, n->dev); + f = br_fdb_find_rcu(br, ha, vid); if (f) { const struct net_bridge_port *dst = READ_ONCE(f->dst); bool replied = false; @@ -207,10 +209,10 @@ void br_do_proxy_suppress_arp(struct sk_buff *skb, struct net_bridge *br, br_is_neigh_suppress_enabled(dst, vid)) { if (!vid) br_arp_send(br, p, skb->dev, sip, tip, - sha, n->ha, sha, 0, 0); + sha, ha, sha, 0, 0); else br_arp_send(br, p, skb->dev, sip, tip, - sha, n->ha, sha, + sha, ha, sha, skb->vlan_proto, skb_vlan_tag_get(skb)); replied = true; @@ -248,7 +250,7 @@ struct nd_msg *br_is_nd_neigh_msg(struct sk_buff *skb, struct nd_msg *msg) } static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p, - struct sk_buff *request, struct neighbour *n, + struct sk_buff *request, struct neighbour *n, u8 *ha, __be16 vlan_proto, u16 vlan_tci) { struct net_device *dev = request->dev; @@ -306,7 +308,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p, ipv6_eth_mc_map(&in6addr_linklocal_allnodes, eth_hdr(reply)->h_dest); else ether_addr_copy(eth_hdr(reply)->h_dest, daddr); - ether_addr_copy(eth_hdr(reply)->h_source, n->ha); + ether_addr_copy(eth_hdr(reply)->h_source, ha); eth_hdr(reply)->h_proto = htons(ETH_P_IPV6); reply->protocol = htons(ETH_P_IPV6); @@ -336,7 +338,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p, na->icmph.icmp6_override = 1; na->icmph.icmp6_solicited = dad ? 0 : 1; na->target = ns->target; - ether_addr_copy(&na->opt[2], n->ha); + ether_addr_copy(&na->opt[2], ha); na->opt[0] = ND_OPT_TARGET_LL_ADDR; na->opt[1] = na_olen >> 3; @@ -365,7 +367,7 @@ static void br_nd_send(struct net_bridge *br, struct net_bridge_port *p, __vlan_hwaccel_put_tag(reply, vlan_proto, vlan_tci); netdev_dbg(dev, "nd send dev %s dst %pI6 dst_hw %pM src %pI6 src_hw %pM\n", - dev->name, &pip6->daddr, daddr, &pip6->saddr, n->ha); + dev->name, &pip6->daddr, daddr, &pip6->saddr, ha); if (p) { dev_queue_xmit(reply); @@ -462,24 +464,26 @@ void br_do_suppress_nd(struct sk_buff *skb, struct net_bridge *br, n = neigh_lookup(ipv6_stub->nd_tbl, &msg->target, vlandev); if (n) { struct net_bridge_fdb_entry *f; + u8 ha[ETH_ALEN] __aligned(2); if (!(READ_ONCE(n->nud_state) & NUD_VALID)) { neigh_release(n); return; } - f = br_fdb_find_rcu(br, n->ha, vid); + neigh_ha_snapshot(ha, n, n->dev); + f = br_fdb_find_rcu(br, ha, vid); if (f) { const struct net_bridge_port *dst = READ_ONCE(f->dst); bool replied = false; if (br_is_neigh_suppress_enabled(dst, vid)) { if (vid != 0) - br_nd_send(br, p, skb, n, + br_nd_send(br, p, skb, n, ha, skb->vlan_proto, skb_vlan_tag_get(skb)); else - br_nd_send(br, p, skb, n, 0, 0); + br_nd_send(br, p, skb, n, ha, 0, 0); replied = true; } diff --git a/net/bridge/br_cfm.c b/net/bridge/br_cfm.c index a89aec831416..b9a3546fb4b4 100644 --- a/net/bridge/br_cfm.c +++ b/net/bridge/br_cfm.c @@ -367,7 +367,7 @@ static u32 ccm_tlv_extract(struct sk_buff *skb, u32 index, } /* note: already called with rcu_read_lock */ -static int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb) +int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb) { u32 mdlevel, interval, size, index, max; const struct br_cfm_common_hdr *hdr; @@ -489,11 +489,6 @@ static int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb) return 1; } -static struct br_frame_type cfm_frame_type __read_mostly = { - .type = cpu_to_be16(ETH_P_CFM), - .frame_handler = br_cfm_frame_rx, -}; - int br_cfm_mep_create(struct net_bridge *br, const u32 instance, struct br_cfm_mep_create *const create, @@ -559,7 +554,7 @@ int br_cfm_mep_create(struct net_bridge *br, INIT_DELAYED_WORK(&mep->ccm_tx_dwork, ccm_tx_work_expired); if (hlist_empty(&br->mep_list)) - br_add_frame(br, &cfm_frame_type); + br_opt_toggle(br, BROPT_CFM_ENABLED, true); hlist_add_tail_rcu(&mep->head, &br->mep_list); @@ -588,7 +583,7 @@ static void mep_delete_implementation(struct net_bridge *br, kfree_rcu(mep, rcu); if (hlist_empty(&br->mep_list)) - br_del_frame(br, &cfm_frame_type); + br_opt_toggle(br, BROPT_CFM_ENABLED, false); } int br_cfm_mep_delete(struct net_bridge *br, diff --git a/net/bridge/br_device.c b/net/bridge/br_device.c index a4f6e56ef872..19d2021926f3 100644 --- a/net/bridge/br_device.c +++ b/net/bridge/br_device.c @@ -512,7 +512,6 @@ void br_dev_setup(struct net_device *dev) spin_lock_init(&br->lock); INIT_LIST_HEAD(&br->port_list); INIT_HLIST_HEAD(&br->fdb_list); - INIT_HLIST_HEAD(&br->frame_type_list); #if IS_ENABLED(CONFIG_BRIDGE_MRP) INIT_HLIST_HEAD(&br->mrp_list); #endif diff --git a/net/bridge/br_input.c b/net/bridge/br_input.c index 7f9bb05048d9..6d6bedd32211 100644 --- a/net/bridge/br_input.c +++ b/net/bridge/br_input.c @@ -303,17 +303,25 @@ static int nf_hook_bridge_pre(struct sk_buff *skb, struct sk_buff **pskb) return RX_HANDLER_CONSUMED; } +#define BR_CFM_MRP_OPTS \ + ((IS_ENABLED(CONFIG_BRIDGE_CFM) ? BIT(BROPT_CFM_ENABLED) : 0UL) | \ + (IS_ENABLED(CONFIG_BRIDGE_MRP) ? BIT(BROPT_MRP_ENABLED) : 0UL)) + /* Return 0 if the frame was not processed otherwise 1 * note: already called with rcu_read_lock */ static int br_process_frame_type(struct net_bridge_port *p, struct sk_buff *skb) { - struct br_frame_type *tmp; + struct net_bridge *br = p->br; + + if (skb->protocol == htons(ETH_P_CFM) && + br_opt_get(br, BROPT_CFM_ENABLED)) + return br_cfm_frame_rx(p, skb); - hlist_for_each_entry_rcu(tmp, &p->br->frame_type_list, list) - if (unlikely(tmp->type == skb->protocol)) - return tmp->frame_handler(p, skb); + if (skb->protocol == htons(ETH_P_MRP) && + br_opt_get(br, BROPT_MRP_ENABLED)) + return br_mrp_process(p, skb); return 0; } @@ -407,7 +415,8 @@ static rx_handler_result_t br_handle_frame(struct sk_buff **pskb) } } - if (unlikely(br_process_frame_type(p, skb))) + if (unlikely((READ_ONCE(p->br->options) & BR_CFM_MRP_OPTS) && + br_process_frame_type(p, skb))) return RX_HANDLER_PASS; forward: @@ -448,19 +457,3 @@ rx_handler_func_t *br_get_rx_handler(const struct net_device *dev) return br_handle_frame; } - -void br_add_frame(struct net_bridge *br, struct br_frame_type *ft) -{ - hlist_add_head_rcu(&ft->list, &br->frame_type_list); -} - -void br_del_frame(struct net_bridge *br, struct br_frame_type *ft) -{ - struct br_frame_type *tmp; - - hlist_for_each_entry(tmp, &br->frame_type_list, list) - if (ft == tmp) { - hlist_del_rcu(&ft->list); - return; - } -} diff --git a/net/bridge/br_mrp.c b/net/bridge/br_mrp.c index a837ff7d6f36..79d6bc62179e 100644 --- a/net/bridge/br_mrp.c +++ b/net/bridge/br_mrp.c @@ -6,13 +6,6 @@ static const u8 mrp_test_dmac[ETH_ALEN] = { 0x1, 0x15, 0x4e, 0x0, 0x0, 0x1 }; static const u8 mrp_in_test_dmac[ETH_ALEN] = { 0x1, 0x15, 0x4e, 0x0, 0x0, 0x3 }; -static int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb); - -static struct br_frame_type mrp_frame_type __read_mostly = { - .type = cpu_to_be16(ETH_P_MRP), - .frame_handler = br_mrp_process, -}; - static bool br_mrp_is_ring_port(struct net_bridge_port *p_port, struct net_bridge_port *s_port, struct net_bridge_port *port) @@ -486,7 +479,7 @@ static void br_mrp_del_impl(struct net_bridge *br, struct br_mrp *mrp) kfree_rcu(mrp, rcu); if (hlist_empty(&br->mrp_list)) - br_del_frame(br, &mrp_frame_type); + br_opt_toggle(br, BROPT_MRP_ENABLED, false); } /* Adds a new MRP instance. @@ -536,7 +529,7 @@ int br_mrp_add(struct net_bridge *br, struct br_mrp_instance *instance) rcu_assign_pointer(mrp->s_port, p); if (hlist_empty(&br->mrp_list)) - br_add_frame(br, &mrp_frame_type); + br_opt_toggle(br, BROPT_MRP_ENABLED, true); INIT_DELAYED_WORK(&mrp->test_work, br_mrp_test_work_expired); INIT_DELAYED_WORK(&mrp->in_test_work, br_mrp_in_test_work_expired); @@ -1241,7 +1234,7 @@ static int br_mrp_rcv(struct net_bridge_port *p, * normal forwarding. * note: already called with rcu_read_lock */ -static int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb) +int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb) { /* If there is no MRP instance do normal forwarding */ if (likely(!(p->flags & BR_MRP_AWARE))) diff --git a/net/bridge/br_multicast.c b/net/bridge/br_multicast.c index b563c97cc709..821112260a91 100644 --- a/net/bridge/br_multicast.c +++ b/net/bridge/br_multicast.c @@ -81,6 +81,10 @@ __br_multicast_add_group(struct net_bridge_mcast *brmctx, bool blocked); static void br_multicast_find_del_pg(struct net_bridge *br, struct net_bridge_port_group *pg); +static void __br_multicast_del_pg(struct net_bridge_mdb_entry *mp, + struct net_bridge_port_group *pg, + struct net_bridge_port_group __rcu **pp, + bool sg_del_exclude_ports); static void __br_multicast_stop(struct net_bridge_mcast *brmctx); static int br_mc_disabled_update(struct net_device *dev, bool value, @@ -457,7 +461,7 @@ static void br_multicast_sg_del_exclude_ports(struct net_bridge_mdb_entry *sgmp) for (pp = &sgmp->ports; (p = mlock_dereference(*pp, sgmp->br)) != NULL;) { if (!(p->flags & MDB_PG_FLAGS_PERMANENT)) - br_multicast_del_pg(sgmp, p, pp); + __br_multicast_del_pg(sgmp, p, pp, false); else pp = &p->next; } @@ -798,9 +802,10 @@ static void br_multicast_destroy_port_group(struct net_bridge_mcast_gc *gc) kfree_rcu(pg, rcu); } -void br_multicast_del_pg(struct net_bridge_mdb_entry *mp, - struct net_bridge_port_group *pg, - struct net_bridge_port_group __rcu **pp) +static void __br_multicast_del_pg(struct net_bridge_mdb_entry *mp, + struct net_bridge_port_group *pg, + struct net_bridge_port_group __rcu **pp, + bool sg_del_exclude_ports) { struct net_bridge *br = pg->key.port->br; struct net_bridge_group_src *ent; @@ -815,7 +820,8 @@ void br_multicast_del_pg(struct net_bridge_mdb_entry *mp, if (!br_multicast_is_star_g(&mp->addr)) { rhashtable_remove_fast(&br->sg_port_tbl, &pg->rhnode, br_sg_port_rht_params); - br_multicast_sg_del_exclude_ports(mp); + if (sg_del_exclude_ports) + br_multicast_sg_del_exclude_ports(mp); } else { br_multicast_star_g_handle_mode(pg, MCAST_INCLUDE); } @@ -827,6 +833,13 @@ void br_multicast_del_pg(struct net_bridge_mdb_entry *mp, mod_timer(&mp->timer, jiffies); } +void br_multicast_del_pg(struct net_bridge_mdb_entry *mp, + struct net_bridge_port_group *pg, + struct net_bridge_port_group __rcu **pp) +{ + __br_multicast_del_pg(mp, pg, pp, true); +} + static void br_multicast_find_del_pg(struct net_bridge *br, struct net_bridge_port_group *pg) { diff --git a/net/bridge/br_private.h b/net/bridge/br_private.h index 4867464bde32..958d12ebb660 100644 --- a/net/bridge/br_private.h +++ b/net/bridge/br_private.h @@ -483,12 +483,13 @@ enum net_bridge_opts { BROPT_VLAN_BRIDGE_BINDING, BROPT_MCAST_VLAN_SNOOPING_ENABLED, BROPT_MST_ENABLED, + BROPT_CFM_ENABLED, + BROPT_MRP_ENABLED, }; struct net_bridge { spinlock_t lock; spinlock_t hash_lock; - struct hlist_head frame_type_list; struct net_device *dev; unsigned long options; /* These fields are accessed on each packet */ @@ -914,16 +915,6 @@ int nbp_backup_change(struct net_bridge_port *p, struct net_device *backup_dev); int br_handle_frame_finish(struct net *net, struct sock *sk, struct sk_buff *skb); rx_handler_func_t *br_get_rx_handler(const struct net_device *dev); -struct br_frame_type { - __be16 type; - int (*frame_handler)(struct net_bridge_port *port, - struct sk_buff *skb); - struct hlist_node list; -}; - -void br_add_frame(struct net_bridge *br, struct br_frame_type *ft); -void br_del_frame(struct net_bridge *br, struct br_frame_type *ft); - static inline bool br_rx_handler_check_rcu(const struct net_device *dev) { return rcu_dereference(dev->rx_handler) == br_get_rx_handler(dev); @@ -2009,6 +2000,7 @@ int br_mrp_parse(struct net_bridge *br, struct net_bridge_port *p, bool br_mrp_enabled(struct net_bridge *br); void br_mrp_port_del(struct net_bridge *br, struct net_bridge_port *p); int br_mrp_fill_info(struct sk_buff *skb, struct net_bridge *br); +int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb); #else static inline int br_mrp_parse(struct net_bridge *br, struct net_bridge_port *p, struct nlattr *attr, int cmd, @@ -2032,6 +2024,11 @@ static inline int br_mrp_fill_info(struct sk_buff *skb, struct net_bridge *br) return 0; } +static inline int br_mrp_process(struct net_bridge_port *p, struct sk_buff *skb) +{ + return 0; +} + #endif /* br_cfm.c */ @@ -2040,6 +2037,7 @@ int br_cfm_parse(struct net_bridge *br, struct net_bridge_port *p, struct nlattr *attr, int cmd, struct netlink_ext_ack *extack); bool br_cfm_created(struct net_bridge *br); void br_cfm_port_del(struct net_bridge *br, struct net_bridge_port *p); +int br_cfm_frame_rx(struct net_bridge_port *port, struct sk_buff *skb); int br_cfm_config_fill_info(struct sk_buff *skb, struct net_bridge *br); int br_cfm_status_fill_info(struct sk_buff *skb, struct net_bridge *br, @@ -2064,6 +2062,12 @@ static inline void br_cfm_port_del(struct net_bridge *br, { } +static inline int br_cfm_frame_rx(struct net_bridge_port *port, + struct sk_buff *skb) +{ + return 0; +} + static inline int br_cfm_config_fill_info(struct sk_buff *skb, struct net_bridge *br) { return -EOPNOTSUPP; diff --git a/net/bridge/netfilter/ebt_dnat.c b/net/bridge/netfilter/ebt_dnat.c index 3fda71a8579d..73f185cccd63 100644 --- a/net/bridge/netfilter/ebt_dnat.c +++ b/net/bridge/netfilter/ebt_dnat.c @@ -39,7 +39,9 @@ ebt_dnat_tg(struct sk_buff *skb, const struct xt_action_param *par) dev = xt_in(par); break; case NF_BR_PRE_ROUTING: - dev = br_port_get_rcu(xt_in(par))->br->dev; + dev = netdev_master_upper_dev_get_rcu(xt_in(par)); + if (!dev) /* bridge port removed? */ + return EBT_DROP; break; default: dev = NULL; diff --git a/net/bridge/netfilter/ebt_redirect.c b/net/bridge/netfilter/ebt_redirect.c index 307790562b49..83486cd4d564 100644 --- a/net/bridge/netfilter/ebt_redirect.c +++ b/net/bridge/netfilter/ebt_redirect.c @@ -24,12 +24,18 @@ ebt_redirect_tg(struct sk_buff *skb, const struct xt_action_param *par) if (skb_ensure_writable(skb, 0)) return EBT_DROP; - if (xt_hooknum(par) != NF_BR_BROUTING) - /* rcu_read_lock()ed by nf_hook_thresh */ - ether_addr_copy(eth_hdr(skb)->h_dest, - br_port_get_rcu(xt_in(par))->br->dev->dev_addr); - else + if (xt_hooknum(par) != NF_BR_BROUTING) { + const struct net_device *dev; + + dev = netdev_master_upper_dev_get_rcu(xt_in(par)); + if (!dev) + return EBT_DROP; + + ether_addr_copy(eth_hdr(skb)->h_dest, dev->dev_addr); + } else { ether_addr_copy(eth_hdr(skb)->h_dest, xt_in(par)->dev_addr); + } + skb->pkt_type = PACKET_HOST; return info->target; } diff --git a/net/bridge/netfilter/ebtable_broute.c b/net/bridge/netfilter/ebtable_broute.c index 8f19253024b0..33d8640d21ac 100644 --- a/net/bridge/netfilter/ebtable_broute.c +++ b/net/bridge/netfilter/ebtable_broute.c @@ -128,8 +128,8 @@ static int __init ebtable_broute_init(void) static void __exit ebtable_broute_fini(void) { - unregister_pernet_subsys(&broute_net_ops); ebt_unregister_template(&broute_table); + unregister_pernet_subsys(&broute_net_ops); } module_init(ebtable_broute_init); diff --git a/net/bridge/netfilter/ebtable_filter.c b/net/bridge/netfilter/ebtable_filter.c index 278f324e6752..fdb988c24916 100644 --- a/net/bridge/netfilter/ebtable_filter.c +++ b/net/bridge/netfilter/ebtable_filter.c @@ -109,8 +109,8 @@ static int __init ebtable_filter_init(void) static void __exit ebtable_filter_fini(void) { - unregister_pernet_subsys(&frame_filter_net_ops); ebt_unregister_template(&frame_filter); + unregister_pernet_subsys(&frame_filter_net_ops); } module_init(ebtable_filter_init); diff --git a/net/bridge/netfilter/ebtable_nat.c b/net/bridge/netfilter/ebtable_nat.c index 9066f7f376d5..8b981b2041b5 100644 --- a/net/bridge/netfilter/ebtable_nat.c +++ b/net/bridge/netfilter/ebtable_nat.c @@ -109,8 +109,8 @@ static int __init ebtable_nat_init(void) static void __exit ebtable_nat_fini(void) { - unregister_pernet_subsys(&frame_nat_net_ops); ebt_unregister_template(&frame_nat); + unregister_pernet_subsys(&frame_nat_net_ops); } module_init(ebtable_nat_init); diff --git a/net/bridge/netfilter/ebtables.c b/net/bridge/netfilter/ebtables.c index 32e73f04d5b1..b5db2d08e006 100644 --- a/net/bridge/netfilter/ebtables.c +++ b/net/bridge/netfilter/ebtables.c @@ -42,6 +42,7 @@ struct ebt_pernet { struct list_head tables; + struct list_head dead_tables; }; struct ebt_template { @@ -1164,11 +1165,6 @@ static int do_replace(struct net *net, sockptr_t arg, unsigned int len) static void __ebt_unregister_table(struct net *net, struct ebt_table *table) { - mutex_lock(&ebt_mutex); - list_del(&table->list); - mutex_unlock(&ebt_mutex); - audit_log_nfcfg(table->name, AF_BRIDGE, table->private->nentries, - AUDIT_XT_OP_UNREGISTER, GFP_KERNEL); EBT_ENTRY_ITERATE(table->private->entries, table->private->entries_size, ebt_cleanup_entry, net, NULL); if (table->private->nentries) @@ -1258,7 +1254,7 @@ int ebt_register_table(struct net *net, const struct ebt_table *input_table, goto free_unlock; } - ops = kmemdup(template_ops, sizeof(*ops) * num_ops, GFP_KERNEL); + ops = kmemdup_array(template_ops, num_ops, sizeof(*ops), GFP_KERNEL); if (!ops) { ret = -ENOMEM; if (newinfo->nentries) @@ -1269,13 +1265,15 @@ int ebt_register_table(struct net *net, const struct ebt_table *input_table, for (i = 0; i < num_ops; i++) ops[i].priv = table; - list_add(&table->list, &ebt_net->tables); - mutex_unlock(&ebt_mutex); - table->ops = ops; ret = nf_register_net_hooks(net, ops, num_ops); - if (ret) + if (ret) { + synchronize_rcu(); __ebt_unregister_table(net, table); + } else { + list_add(&table->list, &ebt_net->tables); + } + mutex_unlock(&ebt_mutex); audit_log_nfcfg(repl->name, AF_BRIDGE, repl->nentries, AUDIT_XT_OP_REGISTER, GFP_KERNEL); @@ -1341,7 +1339,7 @@ void ebt_unregister_template(const struct ebt_table *t) } EXPORT_SYMBOL(ebt_unregister_template); -static struct ebt_table *__ebt_find_table(struct net *net, const char *name) +void ebt_unregister_table_pre_exit(struct net *net, const char *name) { struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id); struct ebt_table *t; @@ -1350,30 +1348,36 @@ static struct ebt_table *__ebt_find_table(struct net *net, const char *name) list_for_each_entry(t, &ebt_net->tables, list) { if (strcmp(t->name, name) == 0) { + list_move(&t->list, &ebt_net->dead_tables); mutex_unlock(&ebt_mutex); - return t; + nf_unregister_net_hooks(net, t->ops, hweight32(t->valid_hooks)); + return; } } mutex_unlock(&ebt_mutex); - return NULL; -} - -void ebt_unregister_table_pre_exit(struct net *net, const char *name) -{ - struct ebt_table *table = __ebt_find_table(net, name); - - if (table) - nf_unregister_net_hooks(net, table->ops, hweight32(table->valid_hooks)); } EXPORT_SYMBOL(ebt_unregister_table_pre_exit); void ebt_unregister_table(struct net *net, const char *name) { - struct ebt_table *table = __ebt_find_table(net, name); + struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id); + struct ebt_table *t; - if (table) - __ebt_unregister_table(net, table); + mutex_lock(&ebt_mutex); + + list_for_each_entry(t, &ebt_net->dead_tables, list) { + if (strcmp(t->name, name) == 0) { + list_del(&t->list); + audit_log_nfcfg(t->name, AF_BRIDGE, t->private->nentries, + AUDIT_XT_OP_UNREGISTER, GFP_KERNEL); + __ebt_unregister_table(net, t); + mutex_unlock(&ebt_mutex); + return; + } + } + + mutex_unlock(&ebt_mutex); } /* userspace just supplied us with counters */ @@ -2594,11 +2598,21 @@ static int __net_init ebt_pernet_init(struct net *net) struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id); INIT_LIST_HEAD(&ebt_net->tables); + INIT_LIST_HEAD(&ebt_net->dead_tables); return 0; } +static void __net_exit ebt_pernet_exit(struct net *net) +{ + struct ebt_pernet *ebt_net = net_generic(net, ebt_pernet_id); + + WARN_ON_ONCE(!list_empty(&ebt_net->tables)); + WARN_ON_ONCE(!list_empty(&ebt_net->dead_tables)); +} + static struct pernet_operations ebt_net_ops = { .init = ebt_pernet_init, + .exit = ebt_pernet_exit, .id = &ebt_pernet_id, .size = sizeof(struct ebt_pernet), }; @@ -2607,19 +2621,20 @@ static int __init ebtables_init(void) { int ret; - ret = xt_register_target(&ebt_standard_target); + ret = register_pernet_subsys(&ebt_net_ops); if (ret < 0) return ret; - ret = nf_register_sockopt(&ebt_sockopts); + + ret = xt_register_target(&ebt_standard_target); if (ret < 0) { - xt_unregister_target(&ebt_standard_target); + unregister_pernet_subsys(&ebt_net_ops); return ret; } - ret = register_pernet_subsys(&ebt_net_ops); + ret = nf_register_sockopt(&ebt_sockopts); if (ret < 0) { - nf_unregister_sockopt(&ebt_sockopts); xt_unregister_target(&ebt_standard_target); + unregister_pernet_subsys(&ebt_net_ops); return ret; } diff --git a/net/can/af_can.c b/net/can/af_can.c index 54877f06cd78..69f794e82ca6 100644 --- a/net/can/af_can.c +++ b/net/can/af_can.c @@ -642,10 +642,8 @@ static int can_rcv_filter(struct can_dev_rcv_lists *dev_rcv_lists, struct sk_buf void can_set_skb_uid(struct sk_buff *skb) { /* create non-zero unique skb identifier together with *skb */ - while (!(skb->hash)) - skb->hash = atomic_inc_return(&skbcounter); - - skb->sw_hash = 1; + while (!(can_skb_prv(skb)->skbcnt)) + can_skb_prv(skb)->skbcnt = atomic_inc_return(&skbcounter); } EXPORT_SYMBOL(can_set_skb_uid); diff --git a/net/can/bcm.c b/net/can/bcm.c index 491495803167..fcd923c223f4 100644 --- a/net/can/bcm.c +++ b/net/can/bcm.c @@ -338,6 +338,7 @@ static void bcm_can_tx(struct bcm_op *op, struct canfd_frame *cf) can_skb_reserve(skb); can_skb_prv(skb)->ifindex = dev->ifindex; + can_skb_prv(skb)->skbcnt = 0; skb_put_data(skb, cf, op->cfsiz); @@ -1581,6 +1582,7 @@ static int bcm_tx_send(struct msghdr *msg, int ifindex, struct sock *sk, } can_skb_prv(skb)->ifindex = dev->ifindex; + can_skb_prv(skb)->skbcnt = 0; skb->dev = dev; can_skb_set_owner(skb, sk); err = can_send(skb, 1); /* send with loopback */ diff --git a/net/can/isotp.c b/net/can/isotp.c index fded419bc34f..e448cfdb1f07 100644 --- a/net/can/isotp.c +++ b/net/can/isotp.c @@ -300,6 +300,7 @@ static int isotp_send_fc(struct sock *sk, int ae, u8 flowstatus) can_skb_reserve(nskb); can_skb_prv(nskb)->ifindex = dev->ifindex; + can_skb_prv(nskb)->skbcnt = 0; nskb->dev = dev; can_skb_set_owner(nskb, sk); @@ -871,6 +872,7 @@ static void isotp_send_cframe(struct isotp_sock *so) can_skb_reserve(skb); can_skb_prv(skb)->ifindex = dev->ifindex; + can_skb_prv(skb)->skbcnt = 0; /* set uid in tx skb to identify CF echo frames */ can_set_skb_uid(skb); @@ -899,7 +901,7 @@ static void isotp_send_cframe(struct isotp_sock *so) pr_notice_once("can-isotp: cfecho is %08X != 0\n", old_cfecho); /* set consecutive frame echo tag */ - WRITE_ONCE(so->cfecho, skb->hash); + WRITE_ONCE(so->cfecho, can_skb_prv(skb)->skbcnt); /* send frame with local echo enabled */ can_send_ret = can_send(skb, 1); @@ -962,7 +964,7 @@ static void isotp_rcv_echo(struct sk_buff *skb, void *data) spin_lock(&so->rx_lock); /* so->cfecho may since belong to a new transfer; recheck under lock */ - if (READ_ONCE(so->cfecho) != skb->hash) + if (READ_ONCE(so->cfecho) != can_skb_prv(skb)->skbcnt) goto out_unlock; /* cancel local echo timeout */ @@ -1194,6 +1196,7 @@ static int isotp_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) can_skb_reserve(skb); can_skb_prv(skb)->ifindex = dev->ifindex; + can_skb_prv(skb)->skbcnt = 0; /* set uid in tx skb to identify CF echo frames */ can_set_skb_uid(skb); @@ -1235,7 +1238,7 @@ static int isotp_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) cf->data[ae] |= size; /* set CF echo tag for isotp_rcv_echo() (SF-mode) */ - WRITE_ONCE(so->cfecho, skb->hash); + WRITE_ONCE(so->cfecho, can_skb_prv(skb)->skbcnt); } else { /* send first frame */ @@ -1252,7 +1255,7 @@ static int isotp_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) so->txfc.bs = 0; /* set CF echo tag for isotp_rcv_echo() (CF-mode) */ - WRITE_ONCE(so->cfecho, skb->hash); + WRITE_ONCE(so->cfecho, can_skb_prv(skb)->skbcnt); } else { /* standard flow control check */ new_state = ISOTP_WAIT_FIRST_FC; diff --git a/net/can/j1939/socket.c b/net/can/j1939/socket.c index 4841aae2fcaa..e1d6d57c921b 100644 --- a/net/can/j1939/socket.c +++ b/net/can/j1939/socket.c @@ -895,6 +895,7 @@ static struct sk_buff *j1939_sk_alloc_skb(struct net_device *ndev, can_skb_reserve(skb); can_skb_prv(skb)->ifindex = ndev->ifindex; + can_skb_prv(skb)->skbcnt = 0; skb_reserve(skb, offsetof(struct can_frame, data)); ret = memcpy_from_msg(skb_put(skb, size), msg, size); diff --git a/net/can/j1939/transport.c b/net/can/j1939/transport.c index a262a58588c2..61f0c6b68138 100644 --- a/net/can/j1939/transport.c +++ b/net/can/j1939/transport.c @@ -613,6 +613,7 @@ sk_buff *j1939_tp_tx_dat_new(struct j1939_priv *priv, skb->dev = priv->ndev; can_skb_reserve(skb); can_skb_prv(skb)->ifindex = priv->ndev->ifindex; + can_skb_prv(skb)->skbcnt = 0; /* reserve CAN header */ skb_reserve(skb, offsetof(struct can_frame, data)); @@ -1552,6 +1553,7 @@ j1939_session *j1939_session_fresh_new(struct j1939_priv *priv, skb->dev = priv->ndev; can_skb_reserve(skb); can_skb_prv(skb)->ifindex = priv->ndev->ifindex; + can_skb_prv(skb)->skbcnt = 0; skcb = j1939_skb_to_cb(skb); memcpy(skcb, rel_skcb, sizeof(*skcb)); diff --git a/net/can/raw.c b/net/can/raw.c index ad4b16476743..517f075a0aec 100644 --- a/net/can/raw.c +++ b/net/can/raw.c @@ -75,8 +75,8 @@ MODULE_ALIAS("can-proto-1"); */ struct uniqframe { + u32 skbcnt; const struct sk_buff *skb; - u32 hash; unsigned int join_rx_count; }; @@ -140,7 +140,7 @@ static void raw_rcv(struct sk_buff *oskb, void *data) /* eliminate multiple filter matches for the same skb */ if (this_cpu_ptr(ro->uniq)->skb == oskb && - this_cpu_ptr(ro->uniq)->hash == oskb->hash) { + this_cpu_ptr(ro->uniq)->skbcnt == can_skb_prv(oskb)->skbcnt) { if (!ro->join_filters) return; @@ -150,7 +150,7 @@ static void raw_rcv(struct sk_buff *oskb, void *data) return; } else { this_cpu_ptr(ro->uniq)->skb = oskb; - this_cpu_ptr(ro->uniq)->hash = oskb->hash; + this_cpu_ptr(ro->uniq)->skbcnt = can_skb_prv(oskb)->skbcnt; this_cpu_ptr(ro->uniq)->join_rx_count = 1; /* drop first frame to check all enabled filters? */ if (ro->join_filters && ro->count > 1) @@ -871,6 +871,7 @@ static int raw_sendmsg(struct socket *sock, struct msghdr *msg, size_t size) can_skb_reserve(skb); can_skb_prv(skb)->ifindex = dev->ifindex; + can_skb_prv(skb)->skbcnt = 0; /* fill the skb before testing for valid CAN frames */ err = memcpy_from_msg(skb_put(skb, size), msg, size); diff --git a/net/ceph/auth_x.c b/net/ceph/auth_x.c index c83559f015e4..f805a813308f 100644 --- a/net/ceph/auth_x.c +++ b/net/ceph/auth_x.c @@ -86,6 +86,11 @@ static int __ceph_x_decrypt(struct ceph_crypto_key *secret, void *p, if (ret) return ret; + if (plaintext_len < sizeof(*hdr)) { + pr_err("%s plaintext too small %d\n", __func__, plaintext_len); + return -EINVAL; + } + if (le64_to_cpu(hdr->magic) != CEPHX_ENC_MAGIC) { pr_err("%s bad magic\n", __func__); return -EINVAL; diff --git a/net/core/bpf_sk_storage.c b/net/core/bpf_sk_storage.c index cca7594be92e..e4cdcf8a0e3b 100644 --- a/net/core/bpf_sk_storage.c +++ b/net/core/bpf_sk_storage.c @@ -50,15 +50,14 @@ void bpf_sk_storage_free(struct sock *sk) { struct bpf_local_storage *sk_storage; - rcu_read_lock(); + rcu_read_lock_dont_migrate(); sk_storage = rcu_dereference(sk->sk_bpf_storage); - if (!sk_storage) { - rcu_read_unlock(); - return; - } + if (!sk_storage) + goto out; bpf_local_storage_destroy(sk_storage); - rcu_read_unlock(); +out: + rcu_read_unlock_migrate(); } static void bpf_sk_storage_map_free(struct bpf_map *map) @@ -158,9 +157,7 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk) struct bpf_local_storage_elem *selem; int ret = 0; - RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL); - - rcu_read_lock(); + rcu_read_lock_dont_migrate(); sk_storage = rcu_dereference(sk->sk_bpf_storage); if (!sk_storage || hlist_empty(&sk_storage->list)) @@ -211,7 +208,7 @@ int bpf_sk_storage_clone(const struct sock *sk, struct sock *newsk) } out: - rcu_read_unlock(); + rcu_read_unlock_migrate(); /* In case of an error, don't free anything explicitly here, the * caller is responsible to call bpf_sk_storage_free. diff --git a/net/core/dev.c b/net/core/dev.c index 39ae67377358..f9436563a2e8 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3541,20 +3541,36 @@ static netdev_features_t dflt_features_check(struct sk_buff *skb, return vlan_features_check(skb, features); } -static bool skb_gso_has_extension_hdr(const struct sk_buff *skb) -{ - if (!skb->encapsulation) - return ((skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6 || - (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_L4 && - vlan_get_protocol(skb) == htons(ETH_P_IPV6))) && - skb_transport_header_was_set(skb) && - skb_network_header_len(skb) != sizeof(struct ipv6hdr)); - else - return (!skb_inner_network_header_was_set(skb) || - ((skb_shinfo(skb)->gso_type & SKB_GSO_TCPV6 || - (skb_shinfo(skb)->gso_type & SKB_GSO_UDP_L4 && - inner_ip_hdr(skb)->version == 6)) && - skb_inner_network_header_len(skb) != sizeof(struct ipv6hdr))); +static bool __skb_has_ipv6_ext_hdr(const struct sk_buff *skb, int nhoff) +{ + const struct ipv6hdr *ip6h; + struct ipv6hdr _ip6h; + + ip6h = skb_header_pointer(skb, nhoff, sizeof(_ip6h), &_ip6h); + return ip6h && ip6h->version == 6 && ipv6_ext_hdr(ip6h->nexthdr); +} + +static bool skb_has_ipv6_extension_hdr(const struct sk_buff *skb) +{ + if (vlan_get_protocol(skb) == htons(ETH_P_IPV6)) { + if (__skb_has_ipv6_ext_hdr(skb, skb_network_offset(skb))) + return true; + + /* Catch tunnels without skb->encapsulation (e.g., virtio). */ + if (!skb->encapsulation && + skb_transport_header_was_set(skb) && + skb_network_header_len(skb) != sizeof(struct ipv6hdr)) + return true; + } + + /* Tunnels without an inner network header, such as SCTP-in-UDP or + * PSP, have no inner IP header and thus no inner extension header. + */ + if (skb->encapsulation && skb_inner_network_header_was_set(skb) && + __skb_has_ipv6_ext_hdr(skb, skb_inner_network_offset(skb))) + return true; + + return false; } static netdev_features_t gso_features_check(const struct sk_buff *skb, @@ -3587,10 +3603,15 @@ static netdev_features_t gso_features_check(const struct sk_buff *skb, * IPv4 header has the potential to be fragmented. */ if (skb_shinfo(skb)->gso_type & SKB_GSO_TCPV4) { - struct iphdr *iph = skb->encapsulation ? - inner_ip_hdr(skb) : ip_hdr(skb); + const struct iphdr *iph; + struct iphdr _iph; + int nhoff = skb->encapsulation ? + skb_inner_network_offset(skb) : + skb_network_offset(skb); - if (!(iph->frag_off & htons(IP_DF))) + iph = skb_header_pointer(skb, nhoff, sizeof(_iph), &_iph); + + if (!iph || !(iph->frag_off & htons(IP_DF))) features &= ~NETIF_F_TSO_MANGLEID; } @@ -3598,7 +3619,7 @@ static netdev_features_t gso_features_check(const struct sk_buff *skb, * so neither does TSO that depends on it. */ if (features & NETIF_F_IPV6_CSUM && - skb_gso_has_extension_hdr(skb) && + skb_has_ipv6_extension_hdr(skb) && !ipv6_has_hopopt_jumbo(skb)) features &= ~(NETIF_F_IPV6_CSUM | NETIF_F_TSO6 | NETIF_F_GSO_UDP_L4); @@ -3701,8 +3722,7 @@ int skb_csum_hwoffload_help(struct sk_buff *skb, return 0; if (features & (NETIF_F_IP_CSUM | NETIF_F_IPV6_CSUM)) { - if (vlan_get_protocol(skb) == htons(ETH_P_IPV6) && - skb_network_header_len(skb) != sizeof(struct ipv6hdr) && + if (skb_has_ipv6_extension_hdr(skb) && !ipv6_has_hopopt_jumbo(skb)) goto sw_checksum; @@ -3801,49 +3821,62 @@ struct sk_buff *validate_xmit_skb_list(struct sk_buff *skb, struct net_device *d } EXPORT_SYMBOL_GPL(validate_xmit_skb_list); -static void qdisc_pkt_len_init(struct sk_buff *skb) +static enum skb_drop_reason qdisc_pkt_len_init(struct sk_buff *skb) { - const struct skb_shared_info *shinfo = skb_shinfo(skb); + struct skb_shared_info *shinfo = skb_shinfo(skb); + unsigned int hdr_len, tlen; + u16 gso_segs; qdisc_skb_cb(skb)->pkt_len = skb->len; + if (!shinfo->gso_size) + return SKB_NOT_DROPPED_YET; + + gso_segs = shinfo->gso_segs; /* To get more precise estimation of bytes sent on wire, * we add to pkt_len the headers size of all segments */ - if (shinfo->gso_size && skb_transport_header_was_set(skb)) { - u16 gso_segs = shinfo->gso_segs; - unsigned int hdr_len; - /* mac layer + network layer */ + /* mac layer + network layer */ + if (!skb->encapsulation) { + if (unlikely(!skb_transport_header_was_set(skb))) + return SKB_NOT_DROPPED_YET; hdr_len = skb_transport_offset(skb); + } else { + hdr_len = skb_inner_transport_offset(skb); + } + /* + transport layer */ + if (likely(shinfo->gso_type & (SKB_GSO_TCPV4 | SKB_GSO_TCPV6))) { + const struct tcphdr *th; - /* + transport layer */ - if (likely(shinfo->gso_type & (SKB_GSO_TCPV4 | SKB_GSO_TCPV6))) { - const struct tcphdr *th; - struct tcphdr _tcphdr; - - th = skb_header_pointer(skb, hdr_len, - sizeof(_tcphdr), &_tcphdr); - if (likely(th)) - hdr_len += __tcp_hdrlen(th); - } else if (shinfo->gso_type & SKB_GSO_UDP_L4) { - struct udphdr _udphdr; - - if (skb_header_pointer(skb, hdr_len, - sizeof(_udphdr), &_udphdr)) - hdr_len += sizeof(struct udphdr); - } + if (!pskb_may_pull(skb, hdr_len + sizeof(struct tcphdr))) + return SKB_DROP_REASON_SKB_BAD_GSO; - if (unlikely(shinfo->gso_type & SKB_GSO_DODGY)) { - int payload = skb->len - hdr_len; + th = (const struct tcphdr *)(skb->data + hdr_len); + tlen = __tcp_hdrlen(th); + if (tlen < sizeof(*th)) + return SKB_DROP_REASON_SKB_BAD_GSO; + hdr_len += tlen; + if (!pskb_may_pull(skb, hdr_len)) + return SKB_DROP_REASON_SKB_BAD_GSO; + } else if (shinfo->gso_type & SKB_GSO_UDP_L4) { + if (!pskb_may_pull(skb, hdr_len + sizeof(struct udphdr))) + return SKB_DROP_REASON_SKB_BAD_GSO; + hdr_len += sizeof(struct udphdr); + } - /* Malicious packet. */ - if (payload <= 0) - return; - gso_segs = DIV_ROUND_UP(payload, shinfo->gso_size); - } - qdisc_skb_cb(skb)->pkt_len += (gso_segs - 1) * hdr_len; + /* prior pskb_may_pull() might have changed skb->head. */ + shinfo = skb_shinfo(skb); + if (unlikely(shinfo->gso_type & SKB_GSO_DODGY)) { + int payload = skb->len - hdr_len; + + /* Malicious packet. */ + if (payload <= 0) + return SKB_DROP_REASON_SKB_BAD_GSO; + gso_segs = DIV_ROUND_UP(payload, shinfo->gso_size); } + qdisc_skb_cb(skb)->pkt_len += (gso_segs - 1) * hdr_len; + return SKB_NOT_DROPPED_YET; } static int dev_qdisc_enqueue(struct sk_buff *skb, struct Qdisc *q, @@ -4387,6 +4420,7 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) struct Qdisc *q; int rc = -ENOMEM; bool again = false; + enum skb_drop_reason reason; skb_reset_mac_header(skb); skb_assert_len(skb); @@ -4394,6 +4428,12 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) if (unlikely(skb_shinfo(skb)->tx_flags & SKBTX_SCHED_TSTAMP)) __skb_tstamp_tx(skb, NULL, NULL, skb->sk, SCM_TSTAMP_SCHED); + reason = qdisc_pkt_len_init(skb); + if (unlikely(reason)) { + dev_core_stats_tx_dropped_inc(dev); + kfree_skb_reason(skb, reason); + return -EINVAL; + } /* Disable soft irqs for various locks below. Also * stops preemption for RCU. */ @@ -4401,7 +4441,6 @@ int __dev_queue_xmit(struct sk_buff *skb, struct net_device *sb_dev) skb_update_prio(skb); - qdisc_pkt_len_init(skb); tcx_set_ingress(skb, false); #ifdef CONFIG_NET_EGRESS if (static_branch_unlikely(&egress_needed_key)) { @@ -9304,6 +9343,37 @@ static int dev_xdp_install(struct net_device *dev, enum bpf_xdp_mode mode, struct netdev_bpf xdp; int err; + if (prog) { + enum bpf_xdp_mode other_mode = mode == XDP_MODE_SKB + ? XDP_MODE_DRV : XDP_MODE_SKB; + bool offload = mode == XDP_MODE_HW; + + if (!offload && dev_xdp_prog(dev, other_mode)) { + NL_SET_ERR_MSG(extack, "Native and generic XDP can't be active at the same time"); + return -EEXIST; + } + if (!offload && bpf_prog_is_offloaded(prog->aux)) { + NL_SET_ERR_MSG(extack, "Using offloaded program without HW_MODE flag is not supported"); + return -EINVAL; + } + if (bpf_prog_is_dev_bound(prog->aux) && !bpf_offload_dev_match(prog, dev)) { + NL_SET_ERR_MSG(extack, "Program bound to different device"); + return -EINVAL; + } + if (bpf_prog_is_dev_bound(prog->aux) && mode == XDP_MODE_SKB) { + NL_SET_ERR_MSG(extack, "Can't attach device-bound programs in generic mode"); + return -EINVAL; + } + if (prog->expected_attach_type == BPF_XDP_DEVMAP) { + NL_SET_ERR_MSG(extack, "BPF_XDP_DEVMAP programs can not be attached to a device"); + return -EINVAL; + } + if (prog->expected_attach_type == BPF_XDP_CPUMAP) { + NL_SET_ERR_MSG(extack, "BPF_XDP_CPUMAP programs can not be attached to a device"); + return -EINVAL; + } + } + memset(&xdp, 0, sizeof(xdp)); xdp.command = mode == XDP_MODE_HW ? XDP_SETUP_PROG_HW : XDP_SETUP_PROG; xdp.extack = extack; @@ -9432,38 +9502,10 @@ static int dev_xdp_attach(struct net_device *dev, struct netlink_ext_ack *extack new_prog = link->link.prog; if (new_prog) { - bool offload = mode == XDP_MODE_HW; - enum bpf_xdp_mode other_mode = mode == XDP_MODE_SKB - ? XDP_MODE_DRV : XDP_MODE_SKB; - if ((flags & XDP_FLAGS_UPDATE_IF_NOEXIST) && cur_prog) { NL_SET_ERR_MSG(extack, "XDP program already attached"); return -EBUSY; } - if (!offload && dev_xdp_prog(dev, other_mode)) { - NL_SET_ERR_MSG(extack, "Native and generic XDP can't be active at the same time"); - return -EEXIST; - } - if (!offload && bpf_prog_is_offloaded(new_prog->aux)) { - NL_SET_ERR_MSG(extack, "Using offloaded program without HW_MODE flag is not supported"); - return -EINVAL; - } - if (bpf_prog_is_dev_bound(new_prog->aux) && !bpf_offload_dev_match(new_prog, dev)) { - NL_SET_ERR_MSG(extack, "Program bound to different device"); - return -EINVAL; - } - if (bpf_prog_is_dev_bound(new_prog->aux) && mode == XDP_MODE_SKB) { - NL_SET_ERR_MSG(extack, "Can't attach device-bound programs in generic mode"); - return -EINVAL; - } - if (new_prog->expected_attach_type == BPF_XDP_DEVMAP) { - NL_SET_ERR_MSG(extack, "BPF_XDP_DEVMAP programs can not be attached to a device"); - return -EINVAL; - } - if (new_prog->expected_attach_type == BPF_XDP_CPUMAP) { - NL_SET_ERR_MSG(extack, "BPF_XDP_CPUMAP programs can not be attached to a device"); - return -EINVAL; - } } /* don't call drivers if the effective program didn't change */ @@ -10160,7 +10202,8 @@ static int netif_alloc_netdev_queues(struct net_device *dev) netdev_for_each_tx_queue(dev, netdev_init_one_queue, NULL); spin_lock_init(&dev->tx_global_lock); - + spin_lock_init(&dev->watchdog_lock); + dev->watchdog_ref_held = false; return 0; } @@ -10997,6 +11040,7 @@ struct net_device *alloc_netdev_mqs(int sizeof_priv, const char *name, free_percpu(dev->pcpu_refcnt); free_dev: #endif + ref_tracker_dir_exit(&dev->refcnt_tracker); netdev_freemem(dev); return NULL; } @@ -11265,6 +11309,7 @@ EXPORT_SYMBOL(unregister_netdev); * is already taken in the destination network namespace. * @new_ifindex: If not zero, specifies device index in the target * namespace. + * @extack: netlink extended ack * * This function shuts down a device interface and moves it * to a new network namespace. On success 0 is returned, on @@ -11274,9 +11319,10 @@ EXPORT_SYMBOL(unregister_netdev); */ int __dev_change_net_namespace(struct net_device *dev, struct net *net, - const char *pat, int new_ifindex) + const char *pat, int new_ifindex, + struct netlink_ext_ack *extack) { - struct netdev_name_node *name_node; + struct netdev_name_node *name_node, *tmp; struct net *net_old = dev_net(dev); char new_name[IFNAMSIZ] = {}; int err, new_nsid; @@ -11285,12 +11331,16 @@ int __dev_change_net_namespace(struct net_device *dev, struct net *net, /* Don't allow namespace local devices to be moved. */ err = -EINVAL; - if (dev->features & NETIF_F_NETNS_LOCAL) + if (dev->features & NETIF_F_NETNS_LOCAL) { + NL_SET_ERR_MSG(extack, "The interface netns is immutable"); goto out; + } /* Ensure the device has been registrered */ - if (dev->reg_state != NETREG_REGISTERED) + if (dev->reg_state != NETREG_REGISTERED) { + NL_SET_ERR_MSG(extack, "The interface isn't registered"); goto out; + } /* Get out if there is nothing todo */ err = 0; @@ -11303,30 +11353,57 @@ int __dev_change_net_namespace(struct net_device *dev, struct net *net, err = -EEXIST; if (netdev_name_in_use(net, dev->name)) { /* We get here if we can't use the current device name */ - if (!pat) + if (!pat) { + NL_SET_ERR_MSG(extack, + "An interface with the same name exists in the target netns"); goto out; + } err = dev_prep_valid_name(net, dev, pat, new_name); - if (err < 0) + if (err < 0) { + NL_SET_ERR_MSG_FMT(extack, + "Unable to use '%s' for the new interface name in the target netns", + pat); goto out; + } } /* Check that none of the altnames conflicts. */ err = -EEXIST; - netdev_for_each_altname(dev, name_node) - if (netdev_name_in_use(net, name_node->name)) - goto out; + netdev_for_each_altname_safe(dev, name_node, tmp) { + if (!netdev_name_in_use(net, name_node->name)) + continue; + + if (!check_net(net_old)) { + netdev_name_node_del(name_node); + synchronize_rcu(); + __netdev_name_node_alt_destroy(name_node); + continue; + } + + NL_SET_ERR_MSG_FMT(extack, + "An interface with the altname %s exists in the target netns", + name_node->name); + goto out; + } /* Check that new_ifindex isn't used yet. */ if (new_ifindex) { err = dev_index_reserve(net, new_ifindex); - if (err < 0) + if (err < 0) { + NL_SET_ERR_MSG_FMT(extack, + "The ifindex %d is not available in the target netns", + new_ifindex); goto out; + } } else { /* If there is an ifindex conflict assign a new one */ err = dev_index_reserve(net, dev->ifindex); if (err == -EBUSY) err = dev_index_reserve(net, 0); - if (err < 0) + if (err < 0) { + NL_SET_ERR_MSG(extack, + "Unable to allocate a new ifindex in the target netns"); goto out; + } new_ifindex = err; } @@ -11735,7 +11812,6 @@ static struct pernet_operations __net_initdata netdev_net_ops = { static void __net_exit default_device_exit_net(struct net *net) { - struct netdev_name_node *name_node, *tmp; struct net_device *dev, *aux; /* * Push all migratable network devices back to the @@ -11759,13 +11835,6 @@ static void __net_exit default_device_exit_net(struct net *net) if (netdev_name_in_use(&init_net, fb_name)) snprintf(fb_name, IFNAMSIZ, "dev%%d"); - netdev_for_each_altname_safe(dev, name_node, tmp) - if (netdev_name_in_use(&init_net, name_node->name)) { - netdev_name_node_del(name_node); - synchronize_rcu(); - __netdev_name_node_alt_destroy(name_node); - } - err = dev_change_net_namespace(dev, &init_net, fb_name); if (err) { pr_emerg("%s: failed to move %s to init_net: %d\n", diff --git a/net/core/filter.c b/net/core/filter.c index 3f5c21f78377..531e7a9f42b0 100644 --- a/net/core/filter.c +++ b/net/core/filter.c @@ -7410,7 +7410,7 @@ BPF_CALL_5(bpf_tcp_check_syncookie, struct sock *, sk, void *, iph, u32, iph_len return -EINVAL; /* sk_listener() allows TCP_NEW_SYN_RECV, which makes no sense here. */ - if (sk->sk_protocol != IPPROTO_TCP || sk->sk_state != TCP_LISTEN) + if (sk->sk_state != TCP_LISTEN || sk->sk_protocol != IPPROTO_TCP) return -EINVAL; if (!READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_syncookies)) @@ -7485,7 +7485,7 @@ BPF_CALL_5(bpf_tcp_gen_syncookie, struct sock *, sk, void *, iph, u32, iph_len, if (unlikely(!sk || th_len < sizeof(*th) || th_len != th->doff * 4)) return -EINVAL; - if (sk->sk_protocol != IPPROTO_TCP || sk->sk_state != TCP_LISTEN) + if (sk->sk_state != TCP_LISTEN || sk->sk_protocol != IPPROTO_TCP) return -EINVAL; if (!READ_ONCE(sock_net(sk)->ipv4.sysctl_tcp_syncookies)) diff --git a/net/core/flow_dissector.c b/net/core/flow_dissector.c index 5ac4cd4f3c68..57fb95186238 100644 --- a/net/core/flow_dissector.c +++ b/net/core/flow_dissector.c @@ -999,6 +999,8 @@ u32 bpf_flow_dissect(struct bpf_prog *prog, struct bpf_flow_dissector *ctx, result = bpf_prog_run_pin_on_cpu(prog, ctx); + /* bpf_flow_keys offsets are u16: do not let @hlen be truncated. */ + hlen = min_t(int, hlen, U16_MAX); flow_keys->nhoff = clamp_t(u16, flow_keys->nhoff, nhoff, hlen); flow_keys->thoff = clamp_t(u16, flow_keys->thoff, flow_keys->nhoff, hlen); @@ -1047,6 +1049,7 @@ bool __skb_flow_dissect(const struct net *net, int mpls_lse = 0; int num_hdrs = 0; u8 ip_proto = 0; + u32 thoff; bool ret; if (!data) { @@ -1668,7 +1671,13 @@ bool __skb_flow_dissect(const struct net *net, ret = true; out: - key_control->thoff = min_t(u16, nhoff, skb ? skb->len : hlen); + thoff = min_t(u32, nhoff, skb ? skb->len : hlen); + if (unlikely(thoff > U16_MAX)) { + /* Cannot be represented in key_control->thoff. */ + thoff = U16_MAX; + ret = false; + } + key_control->thoff = thoff; key_basic->n_proto = proto; key_basic->ip_proto = ip_proto; @@ -1853,7 +1862,8 @@ u32 __skb_get_hash_symmetric(const struct sk_buff *skb) EXPORT_SYMBOL_GPL(__skb_get_hash_symmetric); /** - * __skb_get_hash: calculate a flow hash + * __skb_get_hash_net: calculate a flow hash + * @net: associated network namespace, derived from @skb if NULL * @skb: sk_buff to calculate flow hash from * * This function calculates a flow hash based on src/dst addresses @@ -1861,18 +1871,24 @@ EXPORT_SYMBOL_GPL(__skb_get_hash_symmetric); * on success, zero indicates no valid hash. Also, sets l4_hash in skb * if hash is a canonical 4-tuple hash over transport ports. */ -void __skb_get_hash(struct sk_buff *skb) +void __skb_get_hash_net(const struct net *net, struct sk_buff *skb) { struct flow_keys keys; u32 hash; + memset(&keys, 0, sizeof(keys)); + + __skb_flow_dissect(net, skb, &flow_keys_dissector, + &keys, NULL, 0, 0, 0, + FLOW_DISSECTOR_F_STOP_AT_FLOW_LABEL); + __flow_hash_secret_init(); - hash = ___skb_get_hash(skb, &keys, &hashrnd); + hash = __flow_hash_from_keys(&keys, &hashrnd); __skb_set_sw_hash(skb, hash, flow_keys_have_l4(&keys)); } -EXPORT_SYMBOL(__skb_get_hash); +EXPORT_SYMBOL(__skb_get_hash_net); __u32 skb_get_hash_perturb(const struct sk_buff *skb, const siphash_key_t *perturb) diff --git a/net/core/pktgen.c b/net/core/pktgen.c index cbb916f9edca..f1385e41c0ae 100644 --- a/net/core/pktgen.c +++ b/net/core/pktgen.c @@ -128,6 +128,7 @@ #include #include #include +#include #include #include #include @@ -3613,7 +3614,7 @@ static int pktgen_thread_worker(void *arg) struct pktgen_dev *pkt_dev = NULL; int cpu = t->cpu; - WARN_ON(smp_processor_id() != cpu); + WARN_ON_ONCE(smp_processor_id() != cpu); init_waitqueue_head(&t->queue); complete(&t->start_done); @@ -3949,6 +3950,7 @@ static int __net_init pg_net_init(struct net *net) goto remove; } + cpus_read_lock(); for_each_online_cpu(cpu) { int err; @@ -3957,6 +3959,7 @@ static int __net_init pg_net_init(struct net *net) pr_warn("Cannot create thread for cpu %d (%d)\n", cpu, err); } + cpus_read_unlock(); if (list_empty(&pn->pktgen_threads)) { pr_err("Initialization failed for all threads\n"); diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c index 09a3ca7e64c5..5e2df9b8aaf9 100644 --- a/net/core/rtnetlink.c +++ b/net/core/rtnetlink.c @@ -1962,6 +1962,11 @@ static int rtnl_fill_ifinfo(struct sk_buff *skb, return -EMSGSIZE; } +static struct netlink_range_validation txqlen_range = { + .min = 0, + .max = S16_MAX, +}; + static const struct nla_policy ifla_policy[IFLA_MAX+1] = { [IFLA_IFNAME] = { .type = NLA_STRING, .len = IFNAMSIZ-1 }, [IFLA_ADDRESS] = { .type = NLA_BINARY, .len = MAX_ADDR_LEN }, @@ -1971,7 +1976,7 @@ static const struct nla_policy ifla_policy[IFLA_MAX+1] = { [IFLA_LINK] = { .type = NLA_U32 }, [IFLA_MASTER] = { .type = NLA_U32 }, [IFLA_CARRIER] = { .type = NLA_U8 }, - [IFLA_TXQLEN] = { .type = NLA_U32 }, + [IFLA_TXQLEN] = NLA_POLICY_FULL_RANGE(NLA_U32, &txqlen_range), [IFLA_WEIGHT] = { .type = NLA_U32 }, [IFLA_OPERSTATE] = { .type = NLA_U8 }, [IFLA_LINKMODE] = { .type = NLA_U8 }, @@ -2819,7 +2824,7 @@ static int do_setlink(const struct sk_buff *skb, else new_ifindex = 0; - err = __dev_change_net_namespace(dev, net, pat, new_ifindex); + err = __dev_change_net_namespace(dev, net, pat, new_ifindex, extack); put_net(net); if (err) goto errout; diff --git a/net/core/secure_seq.c b/net/core/secure_seq.c index b0ff6153be62..740642aeaf76 100644 --- a/net/core/secure_seq.c +++ b/net/core/secure_seq.c @@ -20,7 +20,6 @@ #include static siphash_aligned_key_t net_secret; -static siphash_aligned_key_t ts_secret; #define EPHEMERAL_PORT_SHUFFLE_PERIOD (10 * HZ) @@ -28,11 +27,6 @@ static __always_inline void net_secret_init(void) { net_get_random_once(&net_secret, sizeof(net_secret)); } - -static __always_inline void ts_secret_init(void) -{ - net_get_random_once(&ts_secret, sizeof(ts_secret)); -} #endif #ifdef CONFIG_INET @@ -53,28 +47,9 @@ static u32 seq_scale(u32 seq) #endif #if IS_ENABLED(CONFIG_IPV6) -u32 secure_tcpv6_ts_off(const struct net *net, - const __be32 *saddr, const __be32 *daddr) -{ - const struct { - struct in6_addr saddr; - struct in6_addr daddr; - } __aligned(SIPHASH_ALIGNMENT) combined = { - .saddr = *(struct in6_addr *)saddr, - .daddr = *(struct in6_addr *)daddr, - }; - - if (READ_ONCE(net->ipv4.sysctl_tcp_timestamps) != 1) - return 0; - - ts_secret_init(); - return siphash(&combined, offsetofend(typeof(combined), daddr), - &ts_secret); -} -EXPORT_SYMBOL(secure_tcpv6_ts_off); - -u32 secure_tcpv6_seq(const __be32 *saddr, const __be32 *daddr, - __be16 sport, __be16 dport) +union tcp_seq_and_ts_off +secure_tcpv6_seq_and_ts_off(const struct net *net, const __be32 *saddr, + const __be32 *daddr, __be16 sport, __be16 dport) { const struct { struct in6_addr saddr; @@ -87,14 +62,20 @@ u32 secure_tcpv6_seq(const __be32 *saddr, const __be32 *daddr, .sport = sport, .dport = dport }; - u32 hash; + union tcp_seq_and_ts_off st; net_secret_init(); - hash = siphash(&combined, offsetofend(typeof(combined), dport), - &net_secret); - return seq_scale(hash); + + st.hash64 = siphash(&combined, offsetofend(typeof(combined), dport), + &net_secret); + + if (READ_ONCE(net->ipv4.sysctl_tcp_timestamps) != 1) + st.ts_off = 0; + + st.seq = seq_scale(st.seq); + return st; } -EXPORT_SYMBOL(secure_tcpv6_seq); +EXPORT_SYMBOL(secure_tcpv6_seq_and_ts_off); u64 secure_ipv6_port_ephemeral(const __be32 *saddr, const __be32 *daddr, __be16 dport) @@ -118,33 +99,30 @@ EXPORT_SYMBOL(secure_ipv6_port_ephemeral); #endif #ifdef CONFIG_INET -u32 secure_tcp_ts_off(const struct net *net, __be32 saddr, __be32 daddr) -{ - if (READ_ONCE(net->ipv4.sysctl_tcp_timestamps) != 1) - return 0; - - ts_secret_init(); - return siphash_2u32((__force u32)saddr, (__force u32)daddr, - &ts_secret); -} - /* secure_tcp_seq_and_tsoff(a, b, 0, d) == secure_ipv4_port_ephemeral(a, b, d), * but fortunately, `sport' cannot be 0 in any circumstances. If this changes, * it would be easy enough to have the former function use siphash_4u32, passing * the arguments as separate u32. */ -u32 secure_tcp_seq(__be32 saddr, __be32 daddr, - __be16 sport, __be16 dport) +union tcp_seq_and_ts_off +secure_tcp_seq_and_ts_off(const struct net *net, __be32 saddr, __be32 daddr, + __be16 sport, __be16 dport) { - u32 hash; + u32 ports = (__force u32)sport << 16 | (__force u32)dport; + union tcp_seq_and_ts_off st; net_secret_init(); - hash = siphash_3u32((__force u32)saddr, (__force u32)daddr, - (__force u32)sport << 16 | (__force u32)dport, - &net_secret); - return seq_scale(hash); + + st.hash64 = siphash_3u32((__force u32)saddr, (__force u32)daddr, + ports, &net_secret); + + if (READ_ONCE(net->ipv4.sysctl_tcp_timestamps) != 1) + st.ts_off = 0; + + st.seq = seq_scale(st.seq); + return st; } -EXPORT_SYMBOL_GPL(secure_tcp_seq); +EXPORT_SYMBOL_GPL(secure_tcp_seq_and_ts_off); u64 secure_ipv4_port_ephemeral(__be32 saddr, __be32 daddr, __be16 dport) { diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 3f7c4e1d7c76..cf5a3a64b882 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -6919,7 +6919,8 @@ nodefer: __kfree_skb(skb); } static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, - size_t offset, size_t len) + size_t offset, size_t len, + unsigned int csum_offset) { const char *kaddr; __wsum csum; @@ -6927,7 +6928,7 @@ static void skb_splice_csum_page(struct sk_buff *skb, struct page *page, kaddr = kmap_local_page(page); csum = csum_partial(kaddr + offset, len, 0); kunmap_local(kaddr); - skb->csum = csum_block_add(skb->csum, csum, skb->len); + skb->csum = csum_block_add(skb->csum, csum, csum_offset); } /** @@ -6988,7 +6989,8 @@ ssize_t skb_splice_from_iter(struct sk_buff *skb, struct iov_iter *iter, } if (skb->ip_summed == CHECKSUM_NONE) - skb_splice_csum_page(skb, page, off, part); + skb_splice_csum_page(skb, page, off, part, + skb->len + spliced); off = 0; spliced += part; diff --git a/net/core/sock.c b/net/core/sock.c index ef5f69a4b3ab..74c54043e64a 100644 --- a/net/core/sock.c +++ b/net/core/sock.c @@ -2327,6 +2327,9 @@ struct sock *sk_clone_lock(const struct sock *sk, const gfp_t priority) sock_copy(newsk, sk); newsk->sk_prot_creator = prot; +#ifdef CONFIG_BPF_SYSCALL + RCU_INIT_POINTER(newsk->sk_bpf_storage, NULL); +#endif /* SANITY */ if (likely(newsk->sk_net_refcnt)) { @@ -2444,7 +2447,7 @@ void sk_free_unlock_clone(struct sock *sk) } EXPORT_SYMBOL_GPL(sk_free_unlock_clone); -static u32 sk_dst_gso_max_size(struct sock *sk, struct dst_entry *dst) +static u32 sk_dst_gso_max_size(struct sock *sk, const struct net_device *dev) { bool is_ipv6 = false; u32 max_size; @@ -2454,8 +2457,8 @@ static u32 sk_dst_gso_max_size(struct sock *sk, struct dst_entry *dst) !ipv6_addr_v4mapped(&sk->sk_v6_rcv_saddr)); #endif /* pairs with the WRITE_ONCE() in netif_set_gso(_ipv4)_max_size() */ - max_size = is_ipv6 ? READ_ONCE(dst_dev(dst)->gso_max_size) : - READ_ONCE(dst_dev(dst)->gso_ipv4_max_size); + max_size = is_ipv6 ? READ_ONCE(dev->gso_max_size) : + READ_ONCE(dev->gso_ipv4_max_size); if (max_size > GSO_LEGACY_MAX_SIZE && !sk_is_tcp(sk)) max_size = GSO_LEGACY_MAX_SIZE; @@ -2464,9 +2467,12 @@ static u32 sk_dst_gso_max_size(struct sock *sk, struct dst_entry *dst) void sk_setup_caps(struct sock *sk, struct dst_entry *dst) { + const struct net_device *dev; u32 max_segs = 1; - sk->sk_route_caps = dst_dev(dst)->features; + rcu_read_lock(); + dev = dst_dev_rcu(dst); + sk->sk_route_caps = dev->features; if (sk_is_tcp(sk)) sk->sk_route_caps |= NETIF_F_GSO; if (sk->sk_route_caps & NETIF_F_GSO) @@ -2478,13 +2484,14 @@ void sk_setup_caps(struct sock *sk, struct dst_entry *dst) sk->sk_route_caps &= ~NETIF_F_GSO_MASK; } else { sk->sk_route_caps |= NETIF_F_SG | NETIF_F_HW_CSUM; - sk->sk_gso_max_size = sk_dst_gso_max_size(sk, dst); + sk->sk_gso_max_size = sk_dst_gso_max_size(sk, dev); /* pairs with the WRITE_ONCE() in netif_set_gso_max_segs() */ - max_segs = max_t(u32, READ_ONCE(dst_dev(dst)->gso_max_segs), 1); + max_segs = max_t(u32, READ_ONCE(dev->gso_max_segs), 1); } } sk->sk_gso_max_segs = max_segs; sk_dst_set(sk, dst); + rcu_read_unlock(); } EXPORT_SYMBOL_GPL(sk_setup_caps); @@ -2736,6 +2743,22 @@ void *sock_kmalloc(struct sock *sk, int size, gfp_t priority) } EXPORT_SYMBOL(sock_kmalloc); +/* + * Duplicate the input "src" memory block using the socket's + * option memory buffer. + */ +void *sock_kmemdup(struct sock *sk, const void *src, + int size, gfp_t priority) +{ + void *mem; + + mem = sock_kmalloc(sk, size, priority); + if (mem) + memcpy(mem, src, size); + return mem; +} +EXPORT_SYMBOL(sock_kmemdup); + /* Free an option memory block. Note, we actually want the inline * here as this allows gcc to detect the nullify and fold away the * condition entirely. diff --git a/net/core/xdp.c b/net/core/xdp.c index 672d971ce4ea..0b312aa7adf5 100644 --- a/net/core/xdp.c +++ b/net/core/xdp.c @@ -737,7 +737,7 @@ __bpf_kfunc int bpf_xdp_metadata_rx_hash(const struct xdp_md *ctx, u32 *hash, __diag_pop(); BTF_SET8_START(xdp_metadata_kfunc_ids) -#define XDP_METADATA_KFUNC(_, name) BTF_ID_FLAGS(func, name, KF_TRUSTED_ARGS) +#define XDP_METADATA_KFUNC(_, name, __) BTF_ID_FLAGS(func, name, KF_TRUSTED_ARGS) XDP_METADATA_KFUNC_xxx #undef XDP_METADATA_KFUNC BTF_SET8_END(xdp_metadata_kfunc_ids) @@ -748,7 +748,7 @@ static const struct btf_kfunc_id_set xdp_metadata_kfunc_set = { }; BTF_ID_LIST(xdp_metadata_kfunc_ids_unsorted) -#define XDP_METADATA_KFUNC(name, str) BTF_ID(func, str) +#define XDP_METADATA_KFUNC(name, str, _) BTF_ID(func, str) XDP_METADATA_KFUNC_xxx #undef XDP_METADATA_KFUNC diff --git a/net/dccp/timer.c b/net/dccp/timer.c index b3255e87cc7e..b4fb33eb02fb 100644 --- a/net/dccp/timer.c +++ b/net/dccp/timer.c @@ -160,7 +160,9 @@ static void dccp_write_timer(struct timer_list *t) static void dccp_keepalive_timer(struct timer_list *t) { - struct sock *sk = from_timer(sk, t, sk_timer); + struct inet_connection_sock *icsk = + from_timer(icsk, t, icsk_keepalive_timer); + struct sock *sk = &icsk->icsk_inet.sk; pr_err("dccp should not use a keepalive timer !\n"); sock_put(sk); diff --git a/net/ethtool/channels.c b/net/ethtool/channels.c index 7b4bbd674bae..cee188da54f8 100644 --- a/net/ethtool/channels.c +++ b/net/ethtool/channels.c @@ -171,11 +171,9 @@ ethnl_set_channels(struct ethnl_req_info *req_info, struct genl_info *info) */ if (ethtool_get_max_rxnfc_channel(dev, &max_rxnfc_in_use)) max_rxnfc_in_use = 0; - if (!netif_is_rxfh_configured(dev) || - ethtool_get_max_rxfh_channel(dev, &max_rxfh_in_use)) - max_rxfh_in_use = 0; + max_rxfh_in_use = ethtool_get_max_rxfh_channel(dev); if (channels.combined_count + channels.rx_count <= max_rxfh_in_use) { - GENL_SET_ERR_MSG(info, "requested channel counts are too low for existing indirection table settings"); + GENL_SET_ERR_MSG_FMT(info, "requested channel counts are too low for existing indirection table (%d)", max_rxfh_in_use); return -EINVAL; } if (channels.combined_count + channels.rx_count <= max_rxnfc_in_use) { diff --git a/net/ethtool/common.c b/net/ethtool/common.c index f5598c5f50de..30fa329fec05 100644 --- a/net/ethtool/common.c +++ b/net/ethtool/common.c @@ -587,35 +587,39 @@ int ethtool_get_max_rxnfc_channel(struct net_device *dev, u64 *max) return err; } -int ethtool_get_max_rxfh_channel(struct net_device *dev, u32 *max) +u32 ethtool_get_max_rxfh_channel(struct net_device *dev) { - u32 dev_size, current_max = 0; + u32 dev_size, current_max; u32 *indir; int ret; + if (!netif_is_rxfh_configured(dev)) + return 0; + if (!dev->ethtool_ops->get_rxfh_indir_size || !dev->ethtool_ops->get_rxfh) - return -EOPNOTSUPP; + return 0; dev_size = dev->ethtool_ops->get_rxfh_indir_size(dev); if (dev_size == 0) - return -EOPNOTSUPP; + return 0; indir = kcalloc(dev_size, sizeof(indir[0]), GFP_USER); if (!indir) - return -ENOMEM; + return U32_MAX; ret = dev->ethtool_ops->get_rxfh(dev, indir, NULL, NULL); - if (ret) + if (ret) { + current_max = U32_MAX; goto out; + } + current_max = 0; while (dev_size--) current_max = max(current_max, indir[dev_size]); - *max = current_max; - out: kfree(indir); - return ret; + return current_max; } int ethtool_check_ops(const struct ethtool_ops *ops) diff --git a/net/ethtool/common.h b/net/ethtool/common.h index 28b8aaaf9bcb..b55705a9ad5a 100644 --- a/net/ethtool/common.h +++ b/net/ethtool/common.h @@ -42,7 +42,7 @@ int __ethtool_get_link(struct net_device *dev); bool convert_legacy_settings_to_link_ksettings( struct ethtool_link_ksettings *link_ksettings, const struct ethtool_cmd *legacy_settings); -int ethtool_get_max_rxfh_channel(struct net_device *dev, u32 *max); +u32 ethtool_get_max_rxfh_channel(struct net_device *dev); int ethtool_get_max_rxnfc_channel(struct net_device *dev, u64 *max); int __ethtool_get_ts_info(struct net_device *dev, struct ethtool_ts_info *info); diff --git a/net/ethtool/ioctl.c b/net/ethtool/ioctl.c index eaeb514b7e5f..8eaf786eb0c7 100644 --- a/net/ethtool/ioctl.c +++ b/net/ethtool/ioctl.c @@ -1847,9 +1847,7 @@ static noinline_for_stack int ethtool_set_channels(struct net_device *dev, * indirection table/rxnfc settings */ if (ethtool_get_max_rxnfc_channel(dev, &max_rxnfc_in_use)) max_rxnfc_in_use = 0; - if (!netif_is_rxfh_configured(dev) || - ethtool_get_max_rxfh_channel(dev, &max_rxfh_in_use)) - max_rxfh_in_use = 0; + max_rxfh_in_use = ethtool_get_max_rxfh_channel(dev); if (channels.combined_count + channels.rx_count <= max_t(u64, max_rxnfc_in_use, max_rxfh_in_use)) return -EINVAL; diff --git a/net/ethtool/rss.c b/net/ethtool/rss.c index 5764202e6cb6..8387cbda3373 100644 --- a/net/ethtool/rss.c +++ b/net/ethtool/rss.c @@ -94,8 +94,10 @@ rss_prepare_data(const struct ethnl_req_info *req_base, ret = ops->get_rxfh(dev, data->indir_table, data->hkey, &dev_hfunc); - if (ret) + if (ret) { + kfree(rss_config); goto out_ops; + } data->hfunc = dev_hfunc; out_ops: diff --git a/net/handshake/handshake-test.c b/net/handshake/handshake-test.c index 34fd1d9b2db8..6eeb53da6e2d 100644 --- a/net/handshake/handshake-test.c +++ b/net/handshake/handshake-test.c @@ -375,6 +375,10 @@ static void handshake_req_cancel_test2(struct kunit *test) /* Pretend to accept this request */ next = handshake_req_next(hn, HANDSHAKE_HANDLER_CLASS_TLSHD); KUNIT_ASSERT_PTR_EQ(test, req, next); + /* Simulate FD_PREPARE() consuming the file reference handed + * off by handshake_req_next(); see handshake_nl_accept_doit(). + */ + fput(filp); /* Act */ result = handshake_req_cancel(sock->sk); @@ -417,6 +421,10 @@ static void handshake_req_cancel_test3(struct kunit *test) /* Pretend to accept this request */ next = handshake_req_next(hn, HANDSHAKE_HANDLER_CLASS_TLSHD); KUNIT_ASSERT_PTR_EQ(test, req, next); + /* Simulate FD_PREPARE() consuming the file reference handed + * off by handshake_req_next(); see handshake_nl_accept_doit(). + */ + fput(filp); /* Pretend to complete this request */ handshake_complete(next, -ETIMEDOUT, NULL); diff --git a/net/handshake/handshake.h b/net/handshake/handshake.h index a48163765a7a..d35f677d7f3e 100644 --- a/net/handshake/handshake.h +++ b/net/handshake/handshake.h @@ -24,6 +24,7 @@ enum hn_flags_bits { HANDSHAKE_F_NET_DRAINING, }; +struct file; struct handshake_proto; /* One handshake request */ @@ -32,6 +33,7 @@ struct handshake_req { struct rhash_head hr_rhash; unsigned long hr_flags; const struct handshake_proto *hr_proto; + struct file *hr_file; struct sock *hr_sk; void (*hr_odestruct)(struct sock *sk); diff --git a/net/handshake/netlink.c b/net/handshake/netlink.c index 80c7302692c7..1afa7d488038 100644 --- a/net/handshake/netlink.c +++ b/net/handshake/netlink.c @@ -92,7 +92,6 @@ int handshake_nl_accept_doit(struct sk_buff *skb, struct genl_info *info) struct net *net = sock_net(skb->sk); struct handshake_net *hn = handshake_pernet(net); struct handshake_req *req = NULL; - struct socket *sock; int class, fd, err; err = -EOPNOTSUPP; @@ -109,9 +108,9 @@ int handshake_nl_accept_doit(struct sk_buff *skb, struct genl_info *info) if (!req) goto out_status; - sock = req->hr_sk->sk_socket; fd = get_unused_fd_flags(O_CLOEXEC); if (fd < 0) { + fput(req->hr_file); /* drop ref from handshake_req_next() */ err = fd; goto out_complete; } @@ -119,10 +118,11 @@ int handshake_nl_accept_doit(struct sk_buff *skb, struct genl_info *info) err = req->hr_proto->hp_accept(req, info, fd); if (err) { put_unused_fd(fd); + fput(req->hr_file); /* drop ref from handshake_req_next() */ goto out_complete; } - fd_install(fd, get_file(sock->file)); + fd_install(fd, req->hr_file); trace_handshake_cmd_accept(net, req, req->hr_sk, fd); return 0; @@ -205,19 +205,20 @@ static void __net_exit handshake_net_exit(struct net *net) */ spin_lock(&hn->hn_lock); set_bit(HANDSHAKE_F_NET_DRAINING, &hn->hn_flags); - list_splice_init(&requests, &hn->hn_requests); + list_splice_init(&hn->hn_requests, &requests); + list_for_each_entry(req, &requests, hr_list) + sock_hold(req->hr_sk); spin_unlock(&hn->hn_lock); while (!list_empty(&requests)) { - req = list_first_entry(&requests, struct handshake_req, hr_list); - list_del(&req->hr_list); + struct sock *sk; - /* - * Requests on this list have not yet been - * accepted, so they do not have an fd to put. - */ + req = list_first_entry(&requests, struct handshake_req, hr_list); + sk = req->hr_sk; + list_del_init(&req->hr_list); handshake_complete(req, -ETIMEDOUT, NULL); + sock_put(sk); } } diff --git a/net/handshake/request.c b/net/handshake/request.c index 5df102534a59..b8eb7e8e0ec4 100644 --- a/net/handshake/request.c +++ b/net/handshake/request.c @@ -14,6 +14,7 @@ #include #include #include +#include #include #include @@ -163,13 +164,16 @@ static void __remove_pending_locked(struct handshake_net *hn, * otherwise %false. * * If @req was on a pending list, it has not yet been accepted. + * Returns %false when the net namespace is draining; the drain + * loop has taken ownership of the pending list. */ static bool remove_pending(struct handshake_net *hn, struct handshake_req *req) { bool ret = false; spin_lock(&hn->hn_lock); - if (!list_empty(&req->hr_list)) { + if (!test_bit(HANDSHAKE_F_NET_DRAINING, &hn->hn_flags) && + !list_empty(&req->hr_list)) { __remove_pending_locked(hn, req); ret = true; } @@ -178,6 +182,17 @@ static bool remove_pending(struct handshake_net *hn, struct handshake_req *req) return ret; } +/** + * handshake_req_next - Return the next queued handshake request + * @hn: per-net handshake state + * @class: handler class to match + * + * On a non-NULL return, the caller owns an extra reference + * on @req->hr_file. FD_PREPARE() consumes it on success; on + * the FD_PREPARE() failure path the caller must fput() it. + * + * Return: pointer to a removed handshake_req, or NULL. + */ struct handshake_req *handshake_req_next(struct handshake_net *hn, int class) { struct handshake_req *req, *pos; @@ -188,6 +203,13 @@ struct handshake_req *handshake_req_next(struct handshake_net *hn, int class) if (pos->hr_proto->hp_handler_class != class) continue; __remove_pending_locked(hn, pos); + /* Hand off a file reference to the accept side under + * hn_lock. A concurrent handshake_req_cancel() can drop + * hr_file before accept reaches FD_PREPARE(); this extra + * reference keeps the file alive until FD_PREPARE() takes + * ownership. + */ + get_file(pos->hr_file); req = pos; break; } @@ -216,9 +238,16 @@ EXPORT_SYMBOL_IF_KUNIT(handshake_req_next); * A zero return value from handshake_req_submit() means that * exactly one subsequent completion callback is guaranteed. * - * A negative return value from handshake_req_submit() means that - * no completion callback will be done and that @req has been - * destroyed. + * A negative return value from handshake_req_submit() guarantees that + * no completion callback will occur and that @req is no longer owned by + * the caller. If cancellation wins the completion race after the request + * has been published, final destruction is deferred until socket teardown. + * + * The caller must hold a reference on @sock->file for the duration + * of this call. Once the request is published to the accept side, a + * concurrent completion or cancellation may release the request's pin on + * @sock->file; the caller's reference is what keeps @sock->sk valid until + * handshake_req_submit() returns. */ int handshake_req_submit(struct socket *sock, struct handshake_req *req, gfp_t flags) @@ -237,6 +266,14 @@ int handshake_req_submit(struct socket *sock, struct handshake_req *req, kfree(req); return -EINVAL; } + + /* + * Pin sock->file for the lifetime of the request so the + * accept side does not race a consumer that releases the + * socket while a handshake is pending. + */ + req->hr_file = get_file(sock->file); + req->hr_odestruct = req->hr_sk->sk_destruct; req->hr_sk->sk_destruct = handshake_sk_destruct; @@ -259,28 +296,37 @@ int handshake_req_submit(struct socket *sock, struct handshake_req *req, goto out_unlock; if (!__add_pending_locked(hn, req)) goto out_unlock; + + /* + * Pin struct sock so sk_destruct does not run until the + * handshake completion path releases it; struct socket is + * held separately via hr_file above. + */ + sock_hold(req->hr_sk); spin_unlock(&hn->hn_lock); ret = handshake_genl_notify(net, req->hr_proto, flags); if (ret) { trace_handshake_notify_err(net, req, req->hr_sk, ret); - if (remove_pending(hn, req)) + if (remove_pending(hn, req)) { + sock_put(req->hr_sk); goto out_err; + } } - /* Prevent socket release while a handshake request is pending */ - sock_hold(req->hr_sk); - trace_handshake_submit(net, req, req->hr_sk); return 0; out_unlock: spin_unlock(&hn->hn_lock); out_err: - /* Restore original destructor so socket teardown still runs on failure */ - req->hr_sk->sk_destruct = req->hr_odestruct; trace_handshake_submit_err(net, req, req->hr_sk, ret); - handshake_req_destroy(req); + if (!test_and_set_bit(HANDSHAKE_F_REQ_COMPLETED, &req->hr_flags)) { + /* Restore original destructor so socket teardown still runs. */ + req->hr_sk->sk_destruct = req->hr_odestruct; + fput(req->hr_file); + handshake_req_destroy(req); + } return ret; } EXPORT_SYMBOL(handshake_req_submit); @@ -292,11 +338,15 @@ void handshake_complete(struct handshake_req *req, unsigned int status, struct net *net = sock_net(sk); if (!test_and_set_bit(HANDSHAKE_F_REQ_COMPLETED, &req->hr_flags)) { + struct file *file = req->hr_file; + trace_handshake_complete(net, req, sk, status); req->hr_proto->hp_done(req, status, info); /* Handshake request is no longer pending */ sock_put(sk); + + fput(file); } } EXPORT_SYMBOL_IF_KUNIT(handshake_complete); @@ -345,6 +395,7 @@ bool handshake_req_cancel(struct sock *sk) /* Handshake request is no longer pending */ sock_put(sk); + fput(req->hr_file); return true; } EXPORT_SYMBOL(handshake_req_cancel); diff --git a/net/ieee802154/core.c b/net/ieee802154/core.c index cdf4b41c54b2..73744459e5e0 100644 --- a/net/ieee802154/core.c +++ b/net/ieee802154/core.c @@ -209,8 +209,10 @@ int cfg802154_switch_netns(struct cfg802154_registered_device *rdev, continue; wpan_dev->netdev->features &= ~NETIF_F_NETNS_LOCAL; err = dev_change_net_namespace(wpan_dev->netdev, net, "wpan%d"); - if (err) + if (err) { + WARN_ON(err && err != -ENOMEM); break; + } wpan_dev->netdev->features |= NETIF_F_NETNS_LOCAL; } @@ -226,7 +228,7 @@ int cfg802154_switch_netns(struct cfg802154_registered_device *rdev, wpan_dev->netdev->features &= ~NETIF_F_NETNS_LOCAL; err = dev_change_net_namespace(wpan_dev->netdev, net, "wpan%d"); - WARN_ON(err); + WARN_ON(err && err != -ENOMEM); wpan_dev->netdev->features |= NETIF_F_NETNS_LOCAL; } @@ -236,7 +238,7 @@ int cfg802154_switch_netns(struct cfg802154_registered_device *rdev, wpan_phy_net_set(&rdev->wpan_phy, net); err = device_rename(&rdev->wpan_phy.dev, dev_name(&rdev->wpan_phy.dev)); - WARN_ON(err); + WARN_ON(err && err != -ENOMEM); return 0; } diff --git a/net/ife/ife.c b/net/ife/ife.c index 7a75947a31e3..2ddf725d3389 100644 --- a/net/ife/ife.c +++ b/net/ife/ife.c @@ -37,7 +37,7 @@ void *ife_encode(struct sk_buff *skb, u16 metalen) * where ORIGDATA = original ethernet header ... */ int hdrm = metalen + IFE_METAHDRLEN; - int total_push = hdrm + skb->dev->hard_header_len; + int total_push = hdrm + ETH_HLEN; struct ifeheadr *ifehdr; struct ethhdr *iethh; /* inner ether header */ int skboff = 0; @@ -50,9 +50,9 @@ void *ife_encode(struct sk_buff *skb, u16 metalen) iethh = (struct ethhdr *) skb->data; __skb_push(skb, total_push); - memcpy(skb->data, iethh, skb->dev->hard_header_len); + memcpy(skb->data, iethh, ETH_HLEN); skb_reset_mac_header(skb); - skboff += skb->dev->hard_header_len; + skboff += ETH_HLEN; /* total metadata length */ ifehdr = (struct ifeheadr *) (skb->data + skboff); @@ -69,12 +69,12 @@ void *ife_decode(struct sk_buff *skb, u16 *metalen) int total_pull; u16 ifehdrln; - if (!pskb_may_pull(skb, skb->dev->hard_header_len + IFE_METAHDRLEN)) + if (!pskb_may_pull(skb, ETH_HLEN + IFE_METAHDRLEN)) return NULL; - ifehdr = (struct ifeheadr *) (skb->data + skb->dev->hard_header_len); + ifehdr = (struct ifeheadr *)(skb->data + ETH_HLEN); ifehdrln = ntohs(ifehdr->metalen); - total_pull = skb->dev->hard_header_len + ifehdrln; + total_pull = ETH_HLEN + ifehdrln; if (unlikely(ifehdrln < 2)) return NULL; @@ -82,7 +82,7 @@ void *ife_decode(struct sk_buff *skb, u16 *metalen) if (unlikely(!pskb_may_pull(skb, total_pull + ETH_HLEN))) return NULL; - ifehdr = (struct ifeheadr *)(skb->data + skb->dev->hard_header_len); + ifehdr = (struct ifeheadr *)(skb->data + ETH_HLEN); skb_set_mac_header(skb, total_pull); __skb_pull(skb, total_pull); *metalen = ifehdrln - IFE_METAHDRLEN; diff --git a/net/ipv4/esp4.c b/net/ipv4/esp4.c index 8850a1dc8c33..ec7415e3dba0 100644 --- a/net/ipv4/esp4.c +++ b/net/ipv4/esp4.c @@ -103,6 +103,13 @@ static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb) struct aead_request *req; struct scatterlist *sg; + /* Managed frags are owned by the zerocopy ubuf; the skb holds no + * per-frag page reference, so we must not drop one here. Mirrors + * the SKBFL_MANAGED_FRAG_REFS handling in skb_release_data(). + */ + if (skb_zcopy_managed(skb)) + return; + if (x->props.flags & XFRM_STATE_ESN) extralen += sizeof(struct esp_output_extra); diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c index facbda1ff51d..454e2544820a 100644 --- a/net/ipv4/fib_semantics.c +++ b/net/ipv4/fib_semantics.c @@ -514,6 +514,9 @@ size_t fib_nlmsg_size(struct fib_info *fi) struct fib_nh_common *nhc = fib_info_nhc(fi, i); size_t nhsize; + if (!nhc) + break; + nhsize = fib_nexthop_nlmsg_size(nhc, nhs != 1); if (nhs != 1) diff --git a/net/ipv4/fib_trie.c b/net/ipv4/fib_trie.c index f40a774686eb..29c84154a0d7 100644 --- a/net/ipv4/fib_trie.c +++ b/net/ipv4/fib_trie.c @@ -1084,8 +1084,6 @@ void fib_alias_hw_flags_set(struct net *net, const struct fib_rt_info *fri) err = fib_dump_info(skb, 0, 0, RTM_NEWROUTE, fri, 0); if (err < 0) { - /* -EMSGSIZE implies BUG in fib_nlmsg_size() */ - WARN_ON(err == -EMSGSIZE); kfree_skb(skb); goto errout; } diff --git a/net/ipv4/igmp.c b/net/ipv4/igmp.c index 8509e9473834..6647aa93276c 100644 --- a/net/ipv4/igmp.c +++ b/net/ipv4/igmp.c @@ -172,6 +172,10 @@ static void ip_ma_put(struct ip_mc_list *im) } } +#define pmc_dereference(e, pmc) \ + rcu_dereference_protected(e, lockdep_is_held(&(pmc)->lock) || \ + lockdep_is_held(&(pmc)->interface->mc_tomb_lock)) + #define for_each_pmc_rcu(in_dev, pmc) \ for (pmc = rcu_dereference(in_dev->mc_list); \ pmc != NULL; \ @@ -182,13 +186,28 @@ static void ip_ma_put(struct ip_mc_list *im) pmc != NULL; \ pmc = rtnl_dereference(pmc->next_rcu)) +#define for_each_psf_mclock(pmc, psf) \ + for (psf = pmc_dereference((pmc)->sources, pmc); \ + psf; \ + psf = pmc_dereference(psf->sf_next, pmc)) + +#define for_each_psf_rcu(im, psf) \ + for (psf = rcu_dereference((im)->sources); \ + psf; \ + psf = rcu_dereference(psf->sf_next)) + +#define for_each_psf_tomb(pmc, psf) \ + for (psf = pmc_dereference((pmc)->tomb, pmc); \ + psf; \ + psf = pmc_dereference(psf->sf_next, pmc)) + static void ip_sf_list_clear_all(struct ip_sf_list *psf) { struct ip_sf_list *next; while (psf) { - next = psf->sf_next; - kfree(psf); + next = rcu_dereference_protected(psf->sf_next, 1); + kfree_rcu(psf, rcu); psf = next; } } @@ -201,13 +220,18 @@ static void ip_sf_list_clear_all(struct ip_sf_list *psf) static void igmp_stop_timer(struct ip_mc_list *im) { + bool put = false; + spin_lock_bh(&im->lock); if (del_timer(&im->timer)) - refcount_dec(&im->refcnt); + put = true; im->tm_running = 0; im->reporter = 0; im->unsolicit_count = 0; spin_unlock_bh(&im->lock); + + if (put) + ip_ma_put(im); } /* It must be called with locked im->lock */ @@ -250,6 +274,8 @@ static void igmp_ifc_start_timer(struct in_device *in_dev, int delay) static void igmp_mod_timer(struct ip_mc_list *im, int max_delay) { + bool put = false; + spin_lock_bh(&im->lock); im->unsolicit_count = 0; if (del_timer(&im->timer)) { @@ -259,10 +285,13 @@ static void igmp_mod_timer(struct ip_mc_list *im, int max_delay) spin_unlock_bh(&im->lock); return; } - refcount_dec(&im->refcnt); + put = true; } igmp_start_timer(im, max_delay); spin_unlock_bh(&im->lock); + + if (put) + ip_ma_put(im); } @@ -323,7 +352,7 @@ igmp_scount(struct ip_mc_list *pmc, int type, int gdeleted, int sdeleted) struct ip_sf_list *psf; int scount = 0; - for (psf = pmc->sources; psf; psf = psf->sf_next) { + for_each_psf_mclock(pmc, psf) { if (!is_in(pmc, psf, type, gdeleted, sdeleted)) continue; scount++; @@ -468,7 +497,8 @@ static struct sk_buff *add_grec(struct sk_buff *skb, struct ip_mc_list *pmc, struct net *net = dev_net(dev); struct igmpv3_report *pih; struct igmpv3_grec *pgr = NULL; - struct ip_sf_list *psf, *psf_next, *psf_prev, **psf_list; + struct ip_sf_list *psf, *psf_next, *psf_prev; + struct ip_sf_list __rcu **psf_list; int scount, stotal, first, isquery, truncate; unsigned int mtu; @@ -491,7 +521,7 @@ static struct sk_buff *add_grec(struct sk_buff *skb, struct ip_mc_list *pmc, psf_list = sdeleted ? &pmc->tomb : &pmc->sources; - if (!*psf_list) + if (!rcu_access_pointer(*psf_list)) goto empty_source; pih = skb ? igmpv3_report_hdr(skb) : NULL; @@ -507,10 +537,12 @@ static struct sk_buff *add_grec(struct sk_buff *skb, struct ip_mc_list *pmc, } first = 1; psf_prev = NULL; - for (psf = *psf_list; psf; psf = psf_next) { + for (psf = pmc_dereference(*psf_list, pmc); + psf; + psf = psf_next) { __be32 *psrc; - psf_next = psf->sf_next; + psf_next = pmc_dereference(psf->sf_next, pmc); if (!is_in(pmc, psf, type, gdeleted, sdeleted)) { psf_prev = psf; @@ -557,10 +589,12 @@ static struct sk_buff *add_grec(struct sk_buff *skb, struct ip_mc_list *pmc, psf->sf_crcount--; if ((sdeleted || gdeleted) && psf->sf_crcount == 0) { if (psf_prev) - psf_prev->sf_next = psf->sf_next; + rcu_assign_pointer(psf_prev->sf_next, + psf_next); else - *psf_list = psf->sf_next; - kfree(psf); + rcu_assign_pointer(*psf_list, + psf_next); + kfree_rcu(psf, rcu); continue; } } @@ -629,28 +663,29 @@ static int igmpv3_send_report(struct in_device *in_dev, struct ip_mc_list *pmc) /* * remove zero-count source records from a source filter list */ -static void igmpv3_clear_zeros(struct ip_sf_list **ppsf) +static void igmpv3_clear_zeros(struct ip_sf_list __rcu **ppsf) { struct ip_sf_list *psf_prev, *psf_next, *psf; psf_prev = NULL; - for (psf = *ppsf; psf; psf = psf_next) { - psf_next = psf->sf_next; + for (psf = rcu_dereference_protected(*ppsf, 1); psf; psf = psf_next) { + psf_next = rcu_dereference_protected(psf->sf_next, 1); if (psf->sf_crcount == 0) { if (psf_prev) - psf_prev->sf_next = psf->sf_next; + rcu_assign_pointer(psf_prev->sf_next, psf_next); else - *ppsf = psf->sf_next; - kfree(psf); - } else + rcu_assign_pointer(*ppsf, psf_next); + kfree_rcu(psf, rcu); + } else { psf_prev = psf; + } } } static void kfree_pmc(struct ip_mc_list *pmc) { - ip_sf_list_clear_all(pmc->sources); - ip_sf_list_clear_all(pmc->tomb); + ip_sf_list_clear_all(rcu_dereference_protected(pmc->sources, 1)); + ip_sf_list_clear_all(rcu_dereference_protected(pmc->tomb, 1)); kfree(pmc); } @@ -684,7 +719,8 @@ static void igmpv3_send_cr(struct in_device *in_dev) igmpv3_clear_zeros(&pmc->sources); } } - if (pmc->crcount == 0 && !pmc->tomb && !pmc->sources) { + if (pmc->crcount == 0 && !rcu_access_pointer(pmc->tomb) && + !rcu_access_pointer(pmc->sources)) { if (pmc_prev) pmc_prev->next = pmc_next; else @@ -870,7 +906,7 @@ static int igmp_xmarksources(struct ip_mc_list *pmc, int nsrcs, __be32 *srcs) int i, scount; scount = 0; - for (psf = pmc->sources; psf; psf = psf->sf_next) { + for_each_psf_mclock(pmc, psf) { if (scount == nsrcs) break; for (i = 0; i < nsrcs; i++) { @@ -901,7 +937,7 @@ static int igmp_marksources(struct ip_mc_list *pmc, int nsrcs, __be32 *srcs) /* mark INCLUDE-mode sources */ scount = 0; - for (psf = pmc->sources; psf; psf = psf->sf_next) { + for_each_psf_mclock(pmc, psf) { if (scount == nsrcs) break; for (i = 0; i < nsrcs; i++) @@ -1201,11 +1237,12 @@ static void igmpv3_add_delrec(struct in_device *in_dev, struct ip_mc_list *im, if (pmc->sfmode == MCAST_INCLUDE) { struct ip_sf_list *psf; + for_each_psf_mclock(im, psf) + psf->sf_crcount = pmc->crcount; pmc->tomb = im->tomb; pmc->sources = im->sources; - im->tomb = im->sources = NULL; - for (psf = pmc->sources; psf; psf = psf->sf_next) - psf->sf_crcount = pmc->crcount; + RCU_INIT_POINTER(im->tomb, NULL); + RCU_INIT_POINTER(im->sources, NULL); } spin_unlock_bh(&im->lock); @@ -1244,9 +1281,18 @@ static void igmpv3_del_delrec(struct in_device *in_dev, struct ip_mc_list *im) if (pmc) { im->interface = pmc->interface; if (im->sfmode == MCAST_INCLUDE) { - swap(im->tomb, pmc->tomb); - swap(im->sources, pmc->sources); - for (psf = im->sources; psf; psf = psf->sf_next) + struct ip_sf_list *sources, *tomb; + + tomb = rcu_replace_pointer(im->tomb, + rcu_dereference_protected(pmc->tomb, 1), + lockdep_is_held(&im->lock)); + rcu_assign_pointer(pmc->tomb, tomb); + + sources = rcu_replace_pointer(im->sources, + rcu_dereference_protected(pmc->sources, 1), + lockdep_is_held(&im->lock)); + rcu_assign_pointer(pmc->sources, sources); + for_each_psf_mclock(im, psf) psf->sf_crcount = in_dev->mr_qrv ?: READ_ONCE(net->ipv4.sysctl_igmp_qrv); } else { @@ -1283,8 +1329,8 @@ static void igmpv3_clear_delrec(struct in_device *in_dev) struct ip_sf_list *psf; spin_lock_bh(&pmc->lock); - psf = pmc->tomb; - pmc->tomb = NULL; + psf = pmc_dereference(pmc->tomb, pmc); + RCU_INIT_POINTER(pmc->tomb, NULL); spin_unlock_bh(&pmc->lock); ip_sf_list_clear_all(psf); } @@ -1871,7 +1917,7 @@ static int ip_mc_del1_src(struct ip_mc_list *pmc, int sfmode, int rv = 0; psf_prev = NULL; - for (psf = pmc->sources; psf; psf = psf->sf_next) { + for_each_psf_mclock(pmc, psf) { if (psf->sf_inaddr == *psfsrc) break; psf_prev = psf; @@ -1880,7 +1926,7 @@ static int ip_mc_del1_src(struct ip_mc_list *pmc, int sfmode, /* source filter not found, or count wrong => bug */ return -ESRCH; } - psf->sf_count[sfmode]--; + WRITE_ONCE(psf->sf_count[sfmode], psf->sf_count[sfmode] - 1); if (psf->sf_count[sfmode] == 0) { ip_rt_multicast_event(pmc->interface); } @@ -1892,19 +1938,28 @@ static int ip_mc_del1_src(struct ip_mc_list *pmc, int sfmode, /* no more filters for this source */ if (psf_prev) - psf_prev->sf_next = psf->sf_next; + rcu_assign_pointer(psf_prev->sf_next, + pmc_dereference(psf->sf_next, pmc)); else - pmc->sources = psf->sf_next; + rcu_assign_pointer(pmc->sources, + pmc_dereference(psf->sf_next, pmc)); #ifdef CONFIG_IP_MULTICAST if (psf->sf_oldin && !IGMP_V1_SEEN(in_dev) && !IGMP_V2_SEEN(in_dev)) { - psf->sf_crcount = in_dev->mr_qrv ?: READ_ONCE(net->ipv4.sysctl_igmp_qrv); - psf->sf_next = pmc->tomb; - pmc->tomb = psf; - rv = 1; - } else + struct ip_sf_list *dpsf = kmalloc(sizeof(*dpsf), GFP_ATOMIC); + + if (dpsf) { + *dpsf = *psf; + dpsf->sf_crcount = in_dev->mr_qrv ?: + READ_ONCE(net->ipv4.sysctl_igmp_qrv); + rcu_assign_pointer(dpsf->sf_next, + pmc_dereference(pmc->tomb, pmc)); + rcu_assign_pointer(pmc->tomb, dpsf); + rv = 1; + } + } #endif - kfree(psf); + kfree_rcu(psf, rcu); } return rv; } @@ -1941,7 +1996,7 @@ static int ip_mc_del_src(struct in_device *in_dev, __be32 *pmca, int sfmode, err = -EINVAL; if (!pmc->sfcount[sfmode]) goto out_unlock; - pmc->sfcount[sfmode]--; + WRITE_ONCE(pmc->sfcount[sfmode], pmc->sfcount[sfmode] - 1); } err = 0; for (i = 0; i < sfcount; i++) { @@ -1964,7 +2019,7 @@ static int ip_mc_del_src(struct in_device *in_dev, __be32 *pmca, int sfmode, #ifdef CONFIG_IP_MULTICAST pmc->crcount = in_dev->mr_qrv ?: READ_ONCE(net->ipv4.sysctl_igmp_qrv); WRITE_ONCE(in_dev->mr_ifc_count, pmc->crcount); - for (psf = pmc->sources; psf; psf = psf->sf_next) + for_each_psf_mclock(pmc, psf) psf->sf_crcount = 0; igmp_ifc_event(pmc->interface); } else if (sf_setstate(pmc) || changerec) { @@ -1985,7 +2040,7 @@ static int ip_mc_add1_src(struct ip_mc_list *pmc, int sfmode, struct ip_sf_list *psf, *psf_prev; psf_prev = NULL; - for (psf = pmc->sources; psf; psf = psf->sf_next) { + for_each_psf_mclock(pmc, psf) { if (psf->sf_inaddr == *psfsrc) break; psf_prev = psf; @@ -1995,12 +2050,12 @@ static int ip_mc_add1_src(struct ip_mc_list *pmc, int sfmode, if (!psf) return -ENOBUFS; psf->sf_inaddr = *psfsrc; - if (psf_prev) { - psf_prev->sf_next = psf; - } else - pmc->sources = psf; + if (psf_prev) + rcu_assign_pointer(psf_prev->sf_next, psf); + else + rcu_assign_pointer(pmc->sources, psf); } - psf->sf_count[sfmode]++; + WRITE_ONCE(psf->sf_count[sfmode], psf->sf_count[sfmode] + 1); if (psf->sf_count[sfmode] == 1) { ip_rt_multicast_event(pmc->interface); } @@ -2013,13 +2068,15 @@ static void sf_markstate(struct ip_mc_list *pmc) struct ip_sf_list *psf; int mca_xcount = pmc->sfcount[MCAST_EXCLUDE]; - for (psf = pmc->sources; psf; psf = psf->sf_next) + for_each_psf_mclock(pmc, psf) { if (pmc->sfcount[MCAST_EXCLUDE]) { psf->sf_oldin = mca_xcount == psf->sf_count[MCAST_EXCLUDE] && !psf->sf_count[MCAST_INCLUDE]; - } else + } else { psf->sf_oldin = psf->sf_count[MCAST_INCLUDE] != 0; + } + } } static int sf_setstate(struct ip_mc_list *pmc) @@ -2030,27 +2087,31 @@ static int sf_setstate(struct ip_mc_list *pmc) int new_in, rv; rv = 0; - for (psf = pmc->sources; psf; psf = psf->sf_next) { + for_each_psf_mclock(pmc, psf) { if (pmc->sfcount[MCAST_EXCLUDE]) { new_in = mca_xcount == psf->sf_count[MCAST_EXCLUDE] && !psf->sf_count[MCAST_INCLUDE]; - } else + } else { new_in = psf->sf_count[MCAST_INCLUDE] != 0; + } if (new_in) { if (!psf->sf_oldin) { struct ip_sf_list *prev = NULL; - for (dpsf = pmc->tomb; dpsf; dpsf = dpsf->sf_next) { + for_each_psf_tomb(pmc, dpsf) { if (dpsf->sf_inaddr == psf->sf_inaddr) break; prev = dpsf; } if (dpsf) { + struct ip_sf_list *dpsf_next; + + dpsf_next = pmc_dereference(dpsf->sf_next, pmc); if (prev) - prev->sf_next = dpsf->sf_next; + rcu_assign_pointer(prev->sf_next, dpsf_next); else - pmc->tomb = dpsf->sf_next; - kfree(dpsf); + rcu_assign_pointer(pmc->tomb, dpsf_next); + kfree_rcu(dpsf, rcu); } psf->sf_crcount = qrv; rv++; @@ -2062,17 +2123,19 @@ static int sf_setstate(struct ip_mc_list *pmc) * add or update "delete" records if an active filter * is now inactive */ - for (dpsf = pmc->tomb; dpsf; dpsf = dpsf->sf_next) + for_each_psf_tomb(pmc, dpsf) { if (dpsf->sf_inaddr == psf->sf_inaddr) break; + } if (!dpsf) { dpsf = kmalloc(sizeof(*dpsf), GFP_ATOMIC); if (!dpsf) continue; *dpsf = *psf; /* pmc->lock held by callers */ - dpsf->sf_next = pmc->tomb; - pmc->tomb = dpsf; + rcu_assign_pointer(dpsf->sf_next, + pmc_dereference(pmc->tomb, pmc)); + rcu_assign_pointer(pmc->tomb, dpsf); } dpsf->sf_crcount = qrv; rv++; @@ -2112,7 +2175,7 @@ static int ip_mc_add_src(struct in_device *in_dev, __be32 *pmca, int sfmode, #endif isexclude = pmc->sfmode == MCAST_EXCLUDE; if (!delta) - pmc->sfcount[sfmode]++; + WRITE_ONCE(pmc->sfcount[sfmode], pmc->sfcount[sfmode] + 1); err = 0; for (i = 0; i < sfcount; i++) { err = ip_mc_add1_src(pmc, sfmode, &psfsrc[i]); @@ -2123,7 +2186,7 @@ static int ip_mc_add_src(struct in_device *in_dev, __be32 *pmca, int sfmode, int j; if (!delta) - pmc->sfcount[sfmode]--; + WRITE_ONCE(pmc->sfcount[sfmode], pmc->sfcount[sfmode] - 1); for (j = 0; j < i; j++) (void) ip_mc_del1_src(pmc, sfmode, &psfsrc[j]); } else if (isexclude != (pmc->sfcount[MCAST_EXCLUDE] != 0)) { @@ -2143,7 +2206,7 @@ static int ip_mc_add_src(struct in_device *in_dev, __be32 *pmca, int sfmode, pmc->crcount = in_dev->mr_qrv ?: READ_ONCE(net->ipv4.sysctl_igmp_qrv); WRITE_ONCE(in_dev->mr_ifc_count, pmc->crcount); - for (psf = pmc->sources; psf; psf = psf->sf_next) + for_each_psf_mclock(pmc, psf) psf->sf_crcount = 0; igmp_ifc_event(in_dev); } else if (sf_setstate(pmc)) { @@ -2159,13 +2222,13 @@ static void ip_mc_clear_src(struct ip_mc_list *pmc) struct ip_sf_list *tomb, *sources; spin_lock_bh(&pmc->lock); - tomb = pmc->tomb; - pmc->tomb = NULL; - sources = pmc->sources; - pmc->sources = NULL; + tomb = pmc_dereference(pmc->tomb, pmc); + RCU_INIT_POINTER(pmc->tomb, NULL); + sources = pmc_dereference(pmc->sources, pmc); + RCU_INIT_POINTER(pmc->sources, NULL); pmc->sfmode = MCAST_EXCLUDE; - pmc->sfcount[MCAST_INCLUDE] = 0; - pmc->sfcount[MCAST_EXCLUDE] = 1; + WRITE_ONCE(pmc->sfcount[MCAST_INCLUDE], 0); + WRITE_ONCE(pmc->sfcount[MCAST_EXCLUDE], 1); spin_unlock_bh(&pmc->lock); ip_sf_list_clear_all(tomb); @@ -2747,20 +2810,19 @@ int ip_check_mc_rcu(struct in_device *in_dev, __be32 mc_addr, __be32 src_addr, u rv = 1; } else if (im) { if (src_addr) { - spin_lock_bh(&im->lock); - for (psf = im->sources; psf; psf = psf->sf_next) { + for_each_psf_rcu(im, psf) { if (psf->sf_inaddr == src_addr) break; } if (psf) - rv = psf->sf_count[MCAST_INCLUDE] || - psf->sf_count[MCAST_EXCLUDE] != - im->sfcount[MCAST_EXCLUDE]; + rv = READ_ONCE(psf->sf_count[MCAST_INCLUDE]) || + READ_ONCE(psf->sf_count[MCAST_EXCLUDE]) != + READ_ONCE(im->sfcount[MCAST_EXCLUDE]); else - rv = im->sfcount[MCAST_EXCLUDE] != 0; - spin_unlock_bh(&im->lock); - } else + rv = READ_ONCE(im->sfcount[MCAST_EXCLUDE]) != 0; + } else { rv = 1; /* unspecified source; tentatively allow */ + } } return rv; } @@ -2920,7 +2982,7 @@ static inline struct ip_sf_list *igmp_mcf_get_first(struct seq_file *seq) im = rcu_dereference(idev->mc_list); if (likely(im)) { spin_lock_bh(&im->lock); - psf = im->sources; + psf = pmc_dereference(im->sources, im); if (likely(psf)) { state->im = im; state->idev = idev; @@ -2936,7 +2998,7 @@ static struct ip_sf_list *igmp_mcf_get_next(struct seq_file *seq, struct ip_sf_l { struct igmp_mcf_iter_state *state = igmp_mcf_seq_private(seq); - psf = psf->sf_next; + psf = pmc_dereference(psf->sf_next, state->im); while (!psf) { spin_unlock_bh(&state->im->lock); state->im = state->im->next; @@ -2954,7 +3016,7 @@ static struct ip_sf_list *igmp_mcf_get_next(struct seq_file *seq, struct ip_sf_l if (!state->im) break; spin_lock_bh(&state->im->lock); - psf = state->im->sources; + psf = pmc_dereference(state->im->sources, state->im); } out: return psf; diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index 43184b9c7f4f..eace2be308dc 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -752,7 +752,7 @@ void inet_csk_init_xmit_timers(struct sock *sk, timer_setup(&icsk->icsk_retransmit_timer, retransmit_handler, 0); timer_setup(&icsk->icsk_delack_timer, delack_handler, 0); - timer_setup(&sk->sk_timer, keepalive_handler, 0); + timer_setup(&icsk->icsk_keepalive_timer, keepalive_handler, 0); icsk->icsk_pending = icsk->icsk_ack.pending = 0; } EXPORT_SYMBOL(inet_csk_init_xmit_timers); @@ -765,7 +765,7 @@ void inet_csk_clear_xmit_timers(struct sock *sk) sk_stop_timer(sk, &icsk->icsk_retransmit_timer); sk_stop_timer(sk, &icsk->icsk_delack_timer); - sk_stop_timer(sk, &sk->sk_timer); + sk_stop_timer(sk, &icsk->icsk_keepalive_timer); } EXPORT_SYMBOL(inet_csk_clear_xmit_timers); @@ -780,18 +780,18 @@ void inet_csk_clear_xmit_timers_sync(struct sock *sk) sk_stop_timer_sync(sk, &icsk->icsk_retransmit_timer); sk_stop_timer_sync(sk, &icsk->icsk_delack_timer); - sk_stop_timer_sync(sk, &sk->sk_timer); + sk_stop_timer_sync(sk, &icsk->icsk_keepalive_timer); } void inet_csk_delete_keepalive_timer(struct sock *sk) { - sk_stop_timer(sk, &sk->sk_timer); + sk_stop_timer(sk, &inet_csk(sk)->icsk_keepalive_timer); } EXPORT_SYMBOL(inet_csk_delete_keepalive_timer); void inet_csk_reset_keepalive_timer(struct sock *sk, unsigned long len) { - sk_reset_timer(sk, &sk->sk_timer, jiffies + len); + sk_reset_timer(sk, &inet_csk(sk)->icsk_keepalive_timer, jiffies + len); } EXPORT_SYMBOL(inet_csk_reset_keepalive_timer); diff --git a/net/ipv4/inet_diag.c b/net/ipv4/inet_diag.c index 5d09ab3ed735..2128f28e8248 100644 --- a/net/ipv4/inet_diag.c +++ b/net/ipv4/inet_diag.c @@ -310,11 +310,11 @@ int inet_sk_diag_fill(struct sock *sk, struct inet_connection_sock *icsk, r->idiag_retrans = icsk->icsk_probes_out; r->idiag_expires = jiffies_delta_to_msecs(icsk->icsk_timeout - jiffies); - } else if (timer_pending(&sk->sk_timer)) { + } else if (timer_pending(&icsk->icsk_keepalive_timer)) { r->idiag_timer = 2; r->idiag_retrans = icsk->icsk_probes_out; r->idiag_expires = - jiffies_delta_to_msecs(sk->sk_timer.expires - jiffies); + jiffies_delta_to_msecs(icsk->icsk_keepalive_timer.expires - jiffies); } if ((ext & (1 << (INET_DIAG_INFO - 1))) && handler->idiag_info_size) { diff --git a/net/ipv4/ip_tunnel_core.c b/net/ipv4/ip_tunnel_core.c index 3a1ab7ab171c..87933ed7d904 100644 --- a/net/ipv4/ip_tunnel_core.c +++ b/net/ipv4/ip_tunnel_core.c @@ -57,6 +57,17 @@ void iptunnel_xmit(struct sock *sk, struct rtable *rt, struct sk_buff *skb, struct iphdr *iph; int err; + if (dev_recursion_level() > IP_TUNNEL_RECURSION_LIMIT) { + net_crit_ratelimited("Dead loop on virtual device %s, fix it urgently!\n", + dev->name); + DEV_STATS_INC(dev, tx_errors); + ip_rt_put(rt); + kfree_skb(skb); + return; + } + + dev_xmit_recursion_inc(); + skb_scrub_packet(skb, xnet); skb_clear_hash_if_not_l4(skb); @@ -86,6 +97,8 @@ void iptunnel_xmit(struct sock *sk, struct rtable *rt, struct sk_buff *skb, pkt_len = 0; iptunnel_xmit_stats(dev, pkt_len); } + + dev_xmit_recursion_dec(); } EXPORT_SYMBOL_GPL(iptunnel_xmit); diff --git a/net/ipv4/ipconfig.c b/net/ipv4/ipconfig.c index c56b6fe6f0d7..74a770ea2288 100644 --- a/net/ipv4/ipconfig.c +++ b/net/ipv4/ipconfig.c @@ -676,11 +676,40 @@ static const u8 ic_bootp_cookie[4] = { 99, 130, 83, 99 }; #ifdef IPCONFIG_DHCP +static bool __init +ic_dhcp_add_option(u8 **options, const u8 *end, u8 type, const void *value, + int len) +{ + u8 *e = *options; + + /* leave room for the option header and the END marker */ + if (len > U8_MAX || end - e < len + 3) + return false; + + *e++ = type; + *e++ = len; + memcpy(e, value, len); + *options = e + len; + + return true; +} + static void __init ic_dhcp_init_options(u8 *options, struct ic_device *d) { - u8 mt = ((ic_servaddr == NONE) - ? DHCPDISCOVER : DHCPREQUEST); + static const u8 ic_req_params[] = { + 1, /* Subnet mask */ + 3, /* Default gateway */ + 6, /* DNS server */ + 12, /* Host name */ + 15, /* Domain name */ + 17, /* Boot path */ + 26, /* MTU */ + 40, /* NIS domain name */ + 42, /* NTP servers */ + }; + u8 mt = (ic_servaddr == NONE) ? DHCPDISCOVER : DHCPREQUEST; + u8 *end = options + sizeof(((struct bootp_pkt *)0)->exten); u8 *e = options; int len; @@ -705,52 +734,25 @@ ic_dhcp_init_options(u8 *options, struct ic_device *d) e += 4; } - /* always? */ - { - static const u8 ic_req_params[] = { - 1, /* Subnet mask */ - 3, /* Default gateway */ - 6, /* DNS server */ - 12, /* Host name */ - 15, /* Domain name */ - 17, /* Boot path */ - 26, /* MTU */ - 40, /* NIS domain name */ - 42, /* NTP servers */ - }; - - *e++ = 55; /* Parameter request list */ - *e++ = sizeof(ic_req_params); - memcpy(e, ic_req_params, sizeof(ic_req_params)); - e += sizeof(ic_req_params); - - if (ic_host_name_set) { - *e++ = 12; /* host-name */ - len = strlen(utsname()->nodename); - *e++ = len; - memcpy(e, utsname()->nodename, len); - e += len; - } - if (*vendor_class_identifier) { + *e++ = 55; /* Parameter request list */ + *e++ = sizeof(ic_req_params); + memcpy(e, ic_req_params, sizeof(ic_req_params)); + e += sizeof(ic_req_params); + + if (ic_host_name_set) { + len = strlen(utsname()->nodename); + ic_dhcp_add_option(&e, end, 12, utsname()->nodename, len); + } + if (*vendor_class_identifier) { + len = strlen(vendor_class_identifier); + if (ic_dhcp_add_option(&e, end, 60, vendor_class_identifier, len)) pr_info("DHCP: sending class identifier \"%s\"\n", vendor_class_identifier); - *e++ = 60; /* Class-identifier */ - len = strlen(vendor_class_identifier); - *e++ = len; - memcpy(e, vendor_class_identifier, len); - e += len; - } - len = strlen(dhcp_client_identifier + 1); - /* the minimum length of identifier is 2, include 1 byte type, - * and can not be larger than the length of options - */ - if (len >= 1 && len < 312 - (e - options) - 1) { - *e++ = 61; - *e++ = len + 1; - memcpy(e, dhcp_client_identifier, len + 1); - e += len + 1; - } } + len = strlen(dhcp_client_identifier + 1); + /* the minimum length of identifier is 2, include 1 byte type */ + if (len >= 1) + ic_dhcp_add_option(&e, end, 61, dhcp_client_identifier, len + 1); *e++ = 255; /* End of the list */ } diff --git a/net/ipv4/ipmr.c b/net/ipv4/ipmr.c index f39cd565569a..e07e90717c50 100644 --- a/net/ipv4/ipmr.c +++ b/net/ipv4/ipmr.c @@ -2139,6 +2139,9 @@ int ip_mr_input(struct sk_buff *skb) if (IPCB(skb)->flags & IPSKB_FORWARDED) goto dont_forward; + if (!local) + skb_orphan(skb); + mrt = ipmr_rt_fib_lookup(net, skb); if (IS_ERR(mrt)) { kfree_skb(skb); @@ -3159,7 +3162,8 @@ int __init ip_mr_init(void) mrt_cachep = kmem_cache_create("ip_mrt_cache", sizeof(struct mfc_cache), - 0, SLAB_HWCACHE_ALIGN | SLAB_PANIC, + 0, SLAB_HWCACHE_ALIGN | SLAB_PANIC | + SLAB_ACCOUNT, NULL); err = register_pernet_subsys(&ipmr_net_ops); diff --git a/net/ipv4/ipmr_base.c b/net/ipv4/ipmr_base.c index 28d77d454d44..66fe1a253b9f 100644 --- a/net/ipv4/ipmr_base.c +++ b/net/ipv4/ipmr_base.c @@ -38,7 +38,7 @@ mr_table_alloc(struct net *net, u32 id, struct mr_table *mrt; int err; - mrt = kzalloc(sizeof(*mrt), GFP_KERNEL); + mrt = kzalloc(sizeof(*mrt), GFP_KERNEL_ACCOUNT); if (!mrt) return ERR_PTR(-ENOMEM); mrt->id = id; diff --git a/net/ipv4/netfilter/arp_tables.c b/net/ipv4/netfilter/arp_tables.c index eeb48265208a..480a7e89a0f1 100644 --- a/net/ipv4/netfilter/arp_tables.c +++ b/net/ipv4/netfilter/arp_tables.c @@ -1519,13 +1519,11 @@ int arpt_register_table(struct net *net, const struct arpt_replace *repl, const struct nf_hook_ops *template_ops) { - struct nf_hook_ops *ops; - unsigned int num_ops; - int ret, i; - struct xt_table_info *newinfo; struct xt_table_info bootstrap = {0}; - void *loc_cpu_entry; + struct xt_table_info *newinfo; struct xt_table *new_table; + void *loc_cpu_entry; + int ret; newinfo = xt_alloc_table_info(repl->size); if (!newinfo) @@ -1540,7 +1538,7 @@ int arpt_register_table(struct net *net, return ret; } - new_table = xt_register_table(net, table, &bootstrap, newinfo); + new_table = xt_register_table(net, table, template_ops, &bootstrap, newinfo); if (IS_ERR(new_table)) { struct arpt_entry *iter; @@ -1550,31 +1548,6 @@ int arpt_register_table(struct net *net, return PTR_ERR(new_table); } - num_ops = hweight32(table->valid_hooks); - if (num_ops == 0) { - ret = -EINVAL; - goto out_free; - } - - ops = kmemdup(template_ops, sizeof(*ops) * num_ops, GFP_KERNEL); - if (!ops) { - ret = -ENOMEM; - goto out_free; - } - - for (i = 0; i < num_ops; i++) - ops[i].priv = new_table; - - new_table->ops = ops; - - ret = nf_register_net_hooks(net, ops, num_ops); - if (ret != 0) - goto out_free; - - return ret; - -out_free: - __arpt_unregister_table(net, new_table); return ret; } diff --git a/net/ipv4/netfilter/ip_tables.c b/net/ipv4/netfilter/ip_tables.c index 74840604e8d9..e01f040b7eef 100644 --- a/net/ipv4/netfilter/ip_tables.c +++ b/net/ipv4/netfilter/ip_tables.c @@ -1721,13 +1721,11 @@ int ipt_register_table(struct net *net, const struct xt_table *table, const struct ipt_replace *repl, const struct nf_hook_ops *template_ops) { - struct nf_hook_ops *ops; - unsigned int num_ops; - int ret, i; - struct xt_table_info *newinfo; struct xt_table_info bootstrap = {0}; - void *loc_cpu_entry; + struct xt_table_info *newinfo; struct xt_table *new_table; + void *loc_cpu_entry; + int ret; newinfo = xt_alloc_table_info(repl->size); if (!newinfo) @@ -1742,7 +1740,7 @@ int ipt_register_table(struct net *net, const struct xt_table *table, return ret; } - new_table = xt_register_table(net, table, &bootstrap, newinfo); + new_table = xt_register_table(net, table, template_ops, &bootstrap, newinfo); if (IS_ERR(new_table)) { struct ipt_entry *iter; @@ -1752,37 +1750,6 @@ int ipt_register_table(struct net *net, const struct xt_table *table, return PTR_ERR(new_table); } - /* No template? No need to do anything. This is used by 'nat' table, it registers - * with the nat core instead of the netfilter core. - */ - if (!template_ops) - return 0; - - num_ops = hweight32(table->valid_hooks); - if (num_ops == 0) { - ret = -EINVAL; - goto out_free; - } - - ops = kmemdup(template_ops, sizeof(*ops) * num_ops, GFP_KERNEL); - if (!ops) { - ret = -ENOMEM; - goto out_free; - } - - for (i = 0; i < num_ops; i++) - ops[i].priv = new_table; - - new_table->ops = ops; - - ret = nf_register_net_hooks(net, ops, num_ops); - if (ret != 0) - goto out_free; - - return ret; - -out_free: - __ipt_unregister_table(net, new_table); return ret; } diff --git a/net/ipv4/netfilter/iptable_nat.c b/net/ipv4/netfilter/iptable_nat.c index 12ca666d6e2c..a8fe357bc6ec 100644 --- a/net/ipv4/netfilter/iptable_nat.c +++ b/net/ipv4/netfilter/iptable_nat.c @@ -79,7 +79,7 @@ static int ipt_nat_register_lookups(struct net *net) while (i) nf_nat_ipv4_unregister_fn(net, &ops[--i]); - kfree(ops); + kfree_rcu(ops, rcu); return ret; } } @@ -100,7 +100,7 @@ static void ipt_nat_unregister_lookups(struct net *net) for (i = 0; i < ARRAY_SIZE(nf_nat_ipv4_ops); i++) nf_nat_ipv4_unregister_fn(net, &ops[i]); - kfree(ops); + kfree_rcu(ops, rcu); } static int iptable_nat_table_init(struct net *net) diff --git a/net/ipv4/nexthop.c b/net/ipv4/nexthop.c index 7a00acde974a..16a90d88940c 100644 --- a/net/ipv4/nexthop.c +++ b/net/ipv4/nexthop.c @@ -92,6 +92,7 @@ __nh_notifier_single_info_init(struct nh_notifier_single_info *nh_info, else if (nh_info->gw_family == AF_INET6) nh_info->ipv6 = nhi->fib_nhc.nhc_gw.ipv6; + nh_info->id = nhi->nh_parent->id; nh_info->is_reject = nhi->reject_nh; nh_info->is_fdb = nhi->fdb_nh; nh_info->has_encap = !!nhi->fib_nhc.nhc_lwtstate; @@ -137,7 +138,6 @@ static int nh_notifier_mpath_info_init(struct nh_notifier_info *info, struct nh_info *nhi; nhi = rtnl_dereference(nhge->nh->nh_info); - info->nh_grp->nh_entries[i].id = nhge->nh->id; info->nh_grp->nh_entries[i].weight = nhge->weight; __nh_notifier_single_info_init(&info->nh_grp->nh_entries[i].nh, nhi); @@ -660,6 +660,7 @@ static int nla_put_nh_group(struct sk_buff *skb, struct nh_group *nhg) size_t len = nhg->num_nh * sizeof(*p); struct nlattr *nla; u16 group_type = 0; + u16 weight; int i; if (nhg->hash_threshold) @@ -676,9 +677,12 @@ static int nla_put_nh_group(struct sk_buff *skb, struct nh_group *nhg) p = nla_data(nla); for (i = 0; i < nhg->num_nh; ++i) { + weight = nhg->nh_entries[i].weight - 1; + *p++ = (struct nexthop_grp) { .id = nhg->nh_entries[i].nh->id, - .weight = nhg->nh_entries[i].weight - 1, + .weight = weight, + .weight_high = weight >> 8, }; } @@ -786,7 +790,8 @@ static size_t nh_nlmsg_size_grp(struct nexthop *nh) struct nh_group *nhg = rtnl_dereference(nh->nh_grp); size_t sz = sizeof(struct nexthop_grp) * nhg->num_nh; size_t tot = nla_total_size(sz) + - nla_total_size(2); /* NHA_GROUP_TYPE */ + nla_total_size(2) + /* NHA_GROUP_TYPE */ + nla_total_size(0); /* NHA_FDB */ if (nhg->resilient) tot += nh_nlmsg_size_grp_res(nhg); @@ -1062,11 +1067,14 @@ static int nh_check_attr_group(struct net *net, nhg = nla_data(tb[NHA_GROUP]); for (i = 0; i < len; ++i) { - if (nhg[i].resvd1 || nhg[i].resvd2) { - NL_SET_ERR_MSG(extack, "Reserved fields in nexthop_grp must be 0"); + if (nhg[i].resvd2) { + NL_SET_ERR_MSG(extack, "Reserved field in nexthop_grp must be 0"); return -EINVAL; } - if (nhg[i].weight > 254) { + if (nexthop_grp_weight(&nhg[i]) == 0) { + /* 0xffff got passed in, representing weight of 0x10000, + * which is too heavy. + */ NL_SET_ERR_MSG(extack, "Invalid value for weight"); return -EINVAL; } @@ -1655,9 +1663,9 @@ static void nh_res_table_cancel_upkeep(struct nh_res_table *res_table) static void nh_res_group_rebalance(struct nh_group *nhg, struct nh_res_table *res_table) { - int prev_upper_bound = 0; - int total = 0; - int w = 0; + u16 prev_upper_bound = 0; + u32 total = 0; + u32 w = 0; int i; INIT_LIST_HEAD(&res_table->uw_nh_entries); @@ -1667,11 +1675,12 @@ static void nh_res_group_rebalance(struct nh_group *nhg, for (i = 0; i < nhg->num_nh; ++i) { struct nh_grp_entry *nhge = &nhg->nh_entries[i]; - int upper_bound; + u16 upper_bound; + u64 btw; w += nhge->weight; - upper_bound = DIV_ROUND_CLOSEST(res_table->num_nh_buckets * w, - total); + btw = ((u64)res_table->num_nh_buckets) * w; + upper_bound = DIV_ROUND_CLOSEST_ULL(btw, total); nhge->res.wants_buckets = upper_bound - prev_upper_bound; prev_upper_bound = upper_bound; @@ -1737,8 +1746,8 @@ static void replace_nexthop_grp_res(struct nh_group *oldg, static void nh_hthr_group_rebalance(struct nh_group *nhg) { - int total = 0; - int w = 0; + u32 total = 0; + u32 w = 0; int i; for (i = 0; i < nhg->num_nh; ++i) @@ -1746,7 +1755,7 @@ static void nh_hthr_group_rebalance(struct nh_group *nhg) for (i = 0; i < nhg->num_nh; ++i) { struct nh_grp_entry *nhge = &nhg->nh_entries[i]; - int upper_bound; + u32 upper_bound; w += nhge->weight; upper_bound = DIV_ROUND_CLOSEST_ULL((u64)w << 31, total) - 1; @@ -2492,7 +2501,8 @@ static struct nexthop *nexthop_create_group(struct net *net, nhg->has_v4 = true; nhg->nh_entries[i].nh = nhe; - nhg->nh_entries[i].weight = entry[i].weight + 1; + nhg->nh_entries[i].weight = nexthop_grp_weight(&entry[i]); + list_add(&nhg->nh_entries[i].nh_list, &nhe->grp_list); nhg->nh_entries[i].nh_parent = nh; } @@ -3061,15 +3071,15 @@ static int rtm_get_nexthop(struct sk_buff *in_skb, struct nlmsghdr *nlh, if (err) return err; - err = -ENOBUFS; - skb = alloc_skb(NLMSG_GOODSIZE, GFP_KERNEL); - if (!skb) - goto out; - err = -ENOENT; nh = nexthop_find_by_id(net, id); if (!nh) - goto errout_free; + goto out; + + err = -ENOBUFS; + skb = nlmsg_new(nh_nlmsg_size(nh), GFP_KERNEL); + if (!skb) + goto out; err = nh_fill_node(skb, nh, RTM_NEWNEXTHOP, NETLINK_CB(in_skb).portid, nlh->nlmsg_seq, 0); diff --git a/net/ipv4/raw.c b/net/ipv4/raw.c index b670364630a1..ac49dac9b4fd 100644 --- a/net/ipv4/raw.c +++ b/net/ipv4/raw.c @@ -310,7 +310,7 @@ int raw_rcv(struct sock *sk, struct sk_buff *skb) } nf_reset_ct(skb); - skb_push(skb, skb->data - skb_network_header(skb)); + skb_push(skb, -skb_network_offset(skb)); raw_rcv_skb(sk, skb); return 0; diff --git a/net/ipv4/route.c b/net/ipv4/route.c index b175a8292eb3..0b63f10bed8d 100644 --- a/net/ipv4/route.c +++ b/net/ipv4/route.c @@ -1089,6 +1089,8 @@ static void __ip_rt_update_pmtu(struct rtable *rt, struct flowi4 *fl4, u32 mtu) for (nhsel = 0; nhsel < fib_info_num_path(res.fi); nhsel++) { nhc = fib_info_nhc(res.fi, nhsel); + if (!nhc) + break; update_or_create_fnhe(nhc, fl4->daddr, 0, mtu, lock, jiffies + net->ipv4.ip_rt_mtu_expires); } diff --git a/net/ipv4/syncookies.c b/net/ipv4/syncookies.c index 9deeb3f8215e..349e96d029c3 100644 --- a/net/ipv4/syncookies.c +++ b/net/ipv4/syncookies.c @@ -353,9 +353,14 @@ struct sock *cookie_v4_check(struct sock *sk, struct sk_buff *skb) tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL); if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) { - tsoff = secure_tcp_ts_off(sock_net(sk), - ip_hdr(skb)->daddr, - ip_hdr(skb)->saddr); + union tcp_seq_and_ts_off st; + + st = secure_tcp_seq_and_ts_off(sock_net(sk), + ip_hdr(skb)->daddr, + ip_hdr(skb)->saddr, + tcp_hdr(skb)->dest, + tcp_hdr(skb)->source); + tsoff = st.ts_off; tcp_opt.rcv_tsecr -= tsoff; } diff --git a/net/ipv4/tcp_bpf.c b/net/ipv4/tcp_bpf.c index a4a4434171a0..46f5ed7a163e 100644 --- a/net/ipv4/tcp_bpf.c +++ b/net/ipv4/tcp_bpf.c @@ -335,6 +335,7 @@ static int tcp_bpf_recvmsg_parser(struct sock *sk, static int tcp_bpf_ioctl(struct sock *sk, int cmd, int *karg) { + struct sk_psock *psock; bool slow; if (cmd != SIOCINQ) @@ -345,7 +346,21 @@ static int tcp_bpf_ioctl(struct sock *sk, int cmd, int *karg) return -EINVAL; slow = lock_sock_fast(sk); - *karg = sk_psock_msg_inq(sk); + psock = sk_psock_get(sk); + if (unlikely(!psock)) { + unlock_sock_fast(sk, slow); + return tcp_ioctl(sk, cmd, karg); + } + *karg = sk_psock_get_msg_len_nolock(psock); + /* Without a verdict program, ingress data is never diverted to + * ingress_msg: it stays in sk_receive_queue and is read through + * the fallback to tcp_recvmsg(), so account for it like + * tcp_ioctl() does. + */ + if (!READ_ONCE(psock->progs.stream_verdict) && + !READ_ONCE(psock->progs.skb_verdict)) + *karg += tcp_inq(sk); + sk_psock_put(sk, psock); unlock_sock_fast(sk, slow); return 0; diff --git a/net/ipv4/tcp_fastopen.c b/net/ipv4/tcp_fastopen.c index e874ec597014..d36a1ceb7510 100644 --- a/net/ipv4/tcp_fastopen.c +++ b/net/ipv4/tcp_fastopen.c @@ -575,11 +575,12 @@ void tcp_fastopen_active_disable_ofo_check(struct sock *sk) } } else if (tp->syn_fastopen_ch && atomic_read(&sock_net(sk)->ipv4.tfo_active_disable_times)) { - dst = sk_dst_get(sk); - dev = dst ? dst_dev(dst) : NULL; + rcu_read_lock(); + dst = __sk_dst_get(sk); + dev = dst ? dst_dev_rcu(dst) : NULL; if (!(dev && (dev->flags & IFF_LOOPBACK))) atomic_set(&sock_net(sk)->ipv4.tfo_active_disable_times, 0); - dst_release(dst); + rcu_read_unlock(); } } diff --git a/net/ipv4/tcp_input.c b/net/ipv4/tcp_input.c index 89ef6ed7f750..8a6d7dac50b2 100644 --- a/net/ipv4/tcp_input.c +++ b/net/ipv4/tcp_input.c @@ -4087,6 +4087,12 @@ static int tcp_ack(struct sock *sk, const struct sk_buff *skb, int flag) return 1; old_ack: + /* An old ACK can carry new data. Update TS.Recent before SACK + * processing can trigger a retransmission. + */ + if (flag & FLAG_UPDATE_TS_RECENT) + tcp_replace_ts_recent(tp, TCP_SKB_CB(skb)->seq); + /* If data was SACKed, tag it and see if we should send more data. * If data was DSACKed, see if we can undo a cwnd reduction. */ @@ -7140,6 +7146,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops, struct tcp_sock *tp = tcp_sk(sk); struct net *net = sock_net(sk); struct sock *fastopen_sk = NULL; + union tcp_seq_and_ts_off st; struct request_sock *req; bool want_cookie = false; struct dst_entry *dst; @@ -7197,8 +7204,11 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops, if (!dst) goto drop_and_free; + if (tmp_opt.tstamp_ok || (!want_cookie && !isn)) + st = af_ops->init_seq_and_ts_off(net, skb); + if (tmp_opt.tstamp_ok) - tcp_rsk(req)->ts_off = af_ops->init_ts_off(net, skb); + tcp_rsk(req)->ts_off = st.ts_off; if (!want_cookie && !isn) { int max_syn_backlog = READ_ONCE(net->ipv4.sysctl_max_syn_backlog); @@ -7220,7 +7230,7 @@ int tcp_conn_request(struct request_sock_ops *rsk_ops, goto drop_and_release; } - isn = af_ops->init_seq(skb); + isn = st.seq; } tcp_ecn_create_request(req, skb, sk, dst); diff --git a/net/ipv4/tcp_ipv4.c b/net/ipv4/tcp_ipv4.c index 5120db6c5f13..0b055eaacf3a 100644 --- a/net/ipv4/tcp_ipv4.c +++ b/net/ipv4/tcp_ipv4.c @@ -98,17 +98,14 @@ static DEFINE_PER_CPU(struct sock *, ipv4_tcp_sk); static DEFINE_MUTEX(tcp_exit_batch_mutex); -static u32 tcp_v4_init_seq(const struct sk_buff *skb) +static union tcp_seq_and_ts_off +tcp_v4_init_seq_and_ts_off(const struct net *net, const struct sk_buff *skb) { - return secure_tcp_seq(ip_hdr(skb)->daddr, - ip_hdr(skb)->saddr, - tcp_hdr(skb)->dest, - tcp_hdr(skb)->source); -} - -static u32 tcp_v4_init_ts_off(const struct net *net, const struct sk_buff *skb) -{ - return secure_tcp_ts_off(net, ip_hdr(skb)->daddr, ip_hdr(skb)->saddr); + return secure_tcp_seq_and_ts_off(net, + ip_hdr(skb)->daddr, + ip_hdr(skb)->saddr, + tcp_hdr(skb)->dest, + tcp_hdr(skb)->source); } int tcp_twsk_unique(struct sock *sk, struct sock *sktw, void *twp) @@ -315,15 +312,16 @@ int tcp_v4_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len) rt = NULL; if (likely(!tp->repair)) { + union tcp_seq_and_ts_off st; + + st = secure_tcp_seq_and_ts_off(net, + inet->inet_saddr, + inet->inet_daddr, + inet->inet_sport, + usin->sin_port); if (!tp->write_seq) - WRITE_ONCE(tp->write_seq, - secure_tcp_seq(inet->inet_saddr, - inet->inet_daddr, - inet->inet_sport, - usin->sin_port)); - WRITE_ONCE(tp->tsoffset, - secure_tcp_ts_off(net, inet->inet_saddr, - inet->inet_daddr)); + WRITE_ONCE(tp->write_seq, st.seq); + WRITE_ONCE(tp->tsoffset, st.ts_off); } atomic_set(&inet->inet_id, get_random_u16()); @@ -1538,8 +1536,7 @@ const struct tcp_request_sock_ops tcp_request_sock_ipv4_ops = { .cookie_init_seq = cookie_v4_init_sequence, #endif .route_req = tcp_v4_route_req, - .init_seq = tcp_v4_init_seq, - .init_ts_off = tcp_v4_init_ts_off, + .init_seq_and_ts_off = tcp_v4_init_seq_and_ts_off, .send_synack = tcp_v4_send_synack, }; @@ -1632,7 +1629,7 @@ struct sock *tcp_v4_syn_recv_sock(const struct sock *sk, struct sk_buff *skb, tcp_ca_openreq_child(newsk, dst); tcp_sync_mss(newsk, dst_mtu(dst)); - newtp->advmss = tcp_mss_clamp(tcp_sk(sk), dst_metric_advmss(dst)); + newtp->advmss = tcp_mss_clamp(tcp_sk(sk), tcp_dst_advmss(dst)); tcp_initialize_rcv_mss(newsk); @@ -2713,9 +2710,9 @@ static void get_tcp4_sock(struct sock *sk, struct seq_file *f, int i) } else if (icsk->icsk_pending == ICSK_TIME_PROBE0) { timer_active = 4; timer_expires = icsk->icsk_timeout; - } else if (timer_pending(&sk->sk_timer)) { + } else if (timer_pending(&icsk->icsk_keepalive_timer)) { timer_active = 2; - timer_expires = sk->sk_timer.expires; + timer_expires = icsk->icsk_keepalive_timer.expires; } else { timer_active = 0; timer_expires = jiffies; diff --git a/net/ipv4/tcp_metrics.c b/net/ipv4/tcp_metrics.c index 49983d864e4f..ca88ba261ea1 100644 --- a/net/ipv4/tcp_metrics.c +++ b/net/ipv4/tcp_metrics.c @@ -170,7 +170,7 @@ static struct tcp_metrics_block *tcpm_new(struct dst_entry *dst, struct net *net; spin_lock_bh(&tcp_metrics_lock); - net = dev_net(dst_dev(dst)); + net = dst_dev_net_rcu(dst); /* While waiting for the spin-lock the cache might have been populated * with this entry and so we have to check again. @@ -273,7 +273,7 @@ static struct tcp_metrics_block *__tcp_get_metrics_req(struct request_sock *req, return NULL; } - net = dev_net(dst_dev(dst)); + net = dst_dev_net_rcu(dst); hash ^= net_hash_mix(net); hash = hash_32(hash, tcp_metrics_hash_log); @@ -318,7 +318,7 @@ static struct tcp_metrics_block *tcp_get_metrics(struct sock *sk, else return NULL; - net = dev_net(dst_dev(dst)); + net = dst_dev_net_rcu(dst); hash ^= net_hash_mix(net); hash = hash_32(hash, tcp_metrics_hash_log); diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c index d6d64697ce89..b89fd89c3c95 100644 --- a/net/ipv4/tcp_minisocks.c +++ b/net/ipv4/tcp_minisocks.c @@ -403,7 +403,7 @@ void tcp_openreq_init_rwin(struct request_sock *req, u32 rcv_wnd; int mss; - mss = tcp_mss_clamp(tp, dst_metric_advmss(dst)); + mss = tcp_mss_clamp(tp, tcp_dst_advmss(dst)); window_clamp = READ_ONCE(tp->window_clamp); /* Set this up on the first call only */ req->rsk_window_clamp = window_clamp ? : dst_metric(dst, RTAX_WINDOW); diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 7d446106cbcf..088f28fa3b75 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -125,7 +125,7 @@ static __u16 tcp_advertise_mss(struct sock *sk) int mss = tp->advmss; if (dst) { - unsigned int metric = dst_metric_advmss(dst); + unsigned int metric = tcp_dst_advmss(dst); if (metric < mss) { mss = metric; @@ -3701,7 +3701,7 @@ struct sk_buff *tcp_make_synack(const struct sock *sk, struct dst_entry *dst, } skb_dst_set(skb, dst); - mss = tcp_mss_clamp(tp, dst_metric_advmss(dst)); + mss = tcp_mss_clamp(tp, tcp_dst_advmss(dst)); memset(&opts, 0, sizeof(opts)); now = tcp_clock_ns(); @@ -3813,7 +3813,7 @@ static void tcp_connect_init(struct sock *sk) if (!tp->window_clamp) WRITE_ONCE(tp->window_clamp, dst_metric(dst, RTAX_WINDOW)); - tp->advmss = tcp_mss_clamp(tp, dst_metric_advmss(dst)); + tp->advmss = tcp_mss_clamp(tp, tcp_dst_advmss(dst)); tcp_initialize_rcv_mss(sk); diff --git a/net/ipv4/tcp_timer.c b/net/ipv4/tcp_timer.c index b65cd417b0f7..1205fcb138af 100644 --- a/net/ipv4/tcp_timer.c +++ b/net/ipv4/tcp_timer.c @@ -739,8 +739,9 @@ EXPORT_SYMBOL_GPL(tcp_set_keepalive); static void tcp_keepalive_timer (struct timer_list *t) { - struct sock *sk = from_timer(sk, t, sk_timer); - struct inet_connection_sock *icsk = inet_csk(sk); + struct inet_connection_sock *icsk = + from_timer(icsk, t, icsk_keepalive_timer); + struct sock *sk = &icsk->icsk_inet.sk; struct tcp_sock *tp = tcp_sk(sk); u32 elapsed; diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index 339d9db972e5..86a1b2c9e71f 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -421,6 +421,49 @@ u32 udp_ehashfn(const struct net *net, const __be32 laddr, const __u16 lport, udp_ehash_secret + net_hash_mix(net)); } +/** + * udp4_lib_lookup1() - Simplified lookup using primary hash (destination port) + * @net: Network namespace + * @saddr: Source address, network order + * @sport: Source port, network order + * @daddr: Destination address, network order + * @hnum: Destination port, host order + * @dif: Destination interface index + * @sdif: Destination bridge port index, if relevant + * @udptable: Set of UDP hash tables + * + * Simplified lookup to be used as fallback if no sockets are found due to a + * potential race between (receive) address change, and lookup happening before + * the rehash operation. This function ignores SO_REUSEPORT groups while scoring + * result sockets, because if we have one, we don't need the fallback at all. + * + * Called under rcu_read_lock(). + * + * Return: socket with highest matching score if any, NULL if none + */ +static struct sock *udp4_lib_lookup1(struct net *net, + __be32 saddr, __be16 sport, + __be32 daddr, unsigned int hnum, + int dif, int sdif, + const struct udp_table *udptable) +{ + unsigned int slot = udp_hashfn(net, hnum, udptable->mask); + struct udp_hslot *hslot = &udptable->hash[slot]; + struct sock *sk, *result = NULL; + int score, badness = 0; + + sk_for_each_rcu(sk, &hslot->head) { + score = compute_score(sk, net, + saddr, sport, daddr, hnum, dif, sdif); + if (score > badness) { + result = sk; + badness = score; + } + } + + return result; +} + /* called with rcu_read_lock() */ static struct sock *udp4_lib_lookup2(struct net *net, __be32 saddr, __be16 sport, @@ -526,6 +569,19 @@ struct sock *__udp4_lib_lookup(struct net *net, __be32 saddr, result = udp4_lib_lookup2(net, saddr, sport, htonl(INADDR_ANY), hnum, dif, sdif, hslot2, skb); + if (!IS_ERR_OR_NULL(result)) + goto done; + + /* Primary hash (destination port) lookup as fallback for this race: + * 1. __ip4_datagram_connect() sets sk_rcv_saddr + * 2. lookup (this function): new sk_rcv_saddr, hashes not updated yet + * 3. rehash operation updating _secondary and four-tuple_ hashes + * The primary hash doesn't need an update after 1., so, thanks to this + * further step, 1. and 3. don't need to be atomic against the lookup. + */ + result = udp4_lib_lookup1(net, saddr, sport, daddr, hnum, dif, sdif, + udptable); + done: if (IS_ERR(result)) return NULL; diff --git a/net/ipv4/udp_bpf.c b/net/ipv4/udp_bpf.c index 779a3a03762f..b07774034a91 100644 --- a/net/ipv4/udp_bpf.c +++ b/net/ipv4/udp_bpf.c @@ -52,7 +52,9 @@ static int udp_msg_wait_data(struct sock *sk, struct sk_psock *psock, sk_set_bit(SOCKWQ_ASYNC_WAITDATA, sk); ret = udp_msg_has_data(sk, psock); if (!ret) { + release_sock(sk); wait_woken(&wait, TASK_INTERRUPTIBLE, timeo); + lock_sock(sk); ret = udp_msg_has_data(sk, psock); } sk_clear_bit(SOCKWQ_ASYNC_WAITDATA, sk); @@ -81,6 +83,7 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, goto out; } + lock_sock(sk); msg_bytes_ready: copied = sk_msg_recvmsg(sk, psock, msg, len, flags); if (!copied) { @@ -92,11 +95,17 @@ static int udp_bpf_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, if (data) { if (psock_has_data(psock)) goto msg_bytes_ready; + + release_sock(sk); + ret = sk_udp_recvmsg(sk, msg, len, flags, addr_len); goto out; } copied = -EAGAIN; } + + release_sock(sk); + ret = copied; out: sk_psock_put(sk, psock); diff --git a/net/ipv4/udp_tunnel_nic.c b/net/ipv4/udp_tunnel_nic.c index a08b0b6e0727..27543fe015b1 100644 --- a/net/ipv4/udp_tunnel_nic.c +++ b/net/ipv4/udp_tunnel_nic.c @@ -31,19 +31,17 @@ struct udp_tunnel_nic_table_entry { * @dev: netdev pointer * @need_sync: at least one port start changed * @need_replay: space was freed, we need a replay of all ports - * @work_pending: @work is currently scheduled * @n_tables: number of tables under @entries * @missed: bitmap of tables which overflown * @entries: table of tables of ports currently offloaded */ struct udp_tunnel_nic { - struct work_struct work; + struct delayed_work work; struct net_device *dev; u8 need_sync:1; u8 need_replay:1; - u8 work_pending:1; unsigned int n_tables; unsigned long missed; @@ -311,8 +309,7 @@ udp_tunnel_nic_device_sync(struct net_device *dev, struct udp_tunnel_nic *utn) if (!may_sleep) __udp_tunnel_nic_device_sync(dev, utn); if (may_sleep || utn->need_replay) { - queue_work(udp_tunnel_nic_workqueue, &utn->work); - utn->work_pending = 1; + queue_delayed_work(udp_tunnel_nic_workqueue, &utn->work, 0); } } @@ -707,10 +704,16 @@ udp_tunnel_nic_replay(struct net_device *dev, struct udp_tunnel_nic *utn) static void udp_tunnel_nic_device_sync_work(struct work_struct *work) { struct udp_tunnel_nic *utn = - container_of(work, struct udp_tunnel_nic, work); + container_of(work, struct udp_tunnel_nic, work.work); - rtnl_lock(); - utn->work_pending = 0; + /* We cannot block on RTNL here, otherwise we would deadlock with + * udp_tunnel_nic_unregister() calling cancel_delayed_work_sync() + * while holding RTNL. Requeue with 1 jiffy delay if RTNL is contended. + */ + if (!rtnl_trylock()) { + queue_delayed_work(udp_tunnel_nic_workqueue, &utn->work, 1); + return; + } __udp_tunnel_nic_device_sync(utn->dev, utn); if (utn->need_replay) @@ -729,7 +732,7 @@ udp_tunnel_nic_alloc(const struct udp_tunnel_nic_info *info, if (!utn) return NULL; utn->n_tables = n_tables; - INIT_WORK(&utn->work, udp_tunnel_nic_device_sync_work); + INIT_DELAYED_WORK(&utn->work, udp_tunnel_nic_device_sync_work); utn->entries = kmalloc_array(n_tables, sizeof(void *), GFP_KERNEL); if (!utn->entries) @@ -870,11 +873,11 @@ udp_tunnel_nic_unregister(struct net_device *dev, struct udp_tunnel_nic *utn) */ udp_tunnel_nic_flush(dev, utn); - /* Wait for the work to be done using the state, netdev core will - * retry unregister until we give up our reference on this device. + /* Make sure no work is running or queued before freeing @utn. + * The work handler uses rtnl_trylock(), so it will not deadlock + * against the RTNL we are holding here. */ - if (utn->work_pending) - return; + cancel_delayed_work_sync(&utn->work); udp_tunnel_nic_free(utn); release_dev: diff --git a/net/ipv4/xfrm4_input.c b/net/ipv4/xfrm4_input.c index f6e90ba50b63..94fbcf4879fe 100644 --- a/net/ipv4/xfrm4_input.c +++ b/net/ipv4/xfrm4_input.c @@ -48,6 +48,7 @@ int xfrm4_transport_finish(struct sk_buff *skb, int async) { struct xfrm_offload *xo = xfrm_offload(skb); struct iphdr *iph = ip_hdr(skb); + struct net_device *dev = skb->dev; iph->protocol = XFRM_MODE_SKB_CB(skb)->protocol; @@ -56,7 +57,7 @@ int xfrm4_transport_finish(struct sk_buff *skb, int async) return -iph->protocol; #endif - __skb_push(skb, skb->data - skb_network_header(skb)); + __skb_push(skb, -skb_network_offset(skb)); iph->tot_len = htons(skb->len); ip_send_check(iph); @@ -71,8 +72,10 @@ int xfrm4_transport_finish(struct sk_buff *skb, int async) } NF_HOOK(NFPROTO_IPV4, NF_INET_PRE_ROUTING, - dev_net(skb->dev), NULL, skb, skb->dev, NULL, + dev_net(dev), NULL, skb, dev, NULL, xfrm4_rcv_encap_finish); + if (async) + dev_put(dev); return 0; } diff --git a/net/ipv6/addrconf.c b/net/ipv6/addrconf.c index 24111053ebce..47ea00e96443 100644 --- a/net/ipv6/addrconf.c +++ b/net/ipv6/addrconf.c @@ -4837,7 +4837,7 @@ static int modify_prefix_route(struct net *net, struct inet6_ifaddr *ifp, fib6_clean_expires(f6i); fib6_may_remove_gc_list(net, f6i); } else { - fib6_set_expires(f6i, expires); + fib6_set_expires(f6i, jiffies + clock_t_to_jiffies(expires)); fib6_add_gc_list(f6i); } diff --git a/net/ipv6/esp6.c b/net/ipv6/esp6.c index 66a4d37bc0f9..de29dd1489f7 100644 --- a/net/ipv6/esp6.c +++ b/net/ipv6/esp6.c @@ -120,6 +120,13 @@ static void esp_ssg_unref(struct xfrm_state *x, void *tmp, struct sk_buff *skb) struct aead_request *req; struct scatterlist *sg; + /* Managed frags are owned by the zerocopy ubuf; the skb holds no + * per-frag page reference, so we must not drop one here. Mirrors + * the SKBFL_MANAGED_FRAG_REFS handling in skb_release_data(). + */ + if (skb_zcopy_managed(skb)) + return; + if (x->props.flags & XFRM_STATE_ESN) extralen += sizeof(struct esp_output_extra); diff --git a/net/ipv6/exthdrs.c b/net/ipv6/exthdrs.c index 6b4cdc7745e8..a8b26b4f9bf2 100644 --- a/net/ipv6/exthdrs.c +++ b/net/ipv6/exthdrs.c @@ -370,23 +370,16 @@ static void seg6_update_csum(struct sk_buff *skb) (__be32 *)addr); } -static int ipv6_srh_rcv(struct sk_buff *skb) +static int ipv6_srh_rcv(struct sk_buff *skb, struct inet6_dev *idev) { struct inet6_skb_parm *opt = IP6CB(skb); struct net *net = dev_net(skb->dev); struct ipv6_sr_hdr *hdr; - struct inet6_dev *idev; struct in6_addr *addr; int accept_seg6; hdr = (struct ipv6_sr_hdr *)skb_transport_header(skb); - idev = __in6_dev_get(skb->dev); - if (!idev) { - kfree_skb(skb); - return -1; - } - accept_seg6 = min(READ_ONCE(net->ipv6.devconf_all->seg6_enabled), READ_ONCE(idev->cnf.seg6_enabled)); @@ -487,12 +480,11 @@ static int ipv6_srh_rcv(struct sk_buff *skb) return -1; } -static int ipv6_rpl_srh_rcv(struct sk_buff *skb) +static int ipv6_rpl_srh_rcv(struct sk_buff *skb, struct inet6_dev *idev) { struct ipv6_rpl_sr_hdr *hdr, *ohdr, *chdr; struct inet6_skb_parm *opt = IP6CB(skb); struct net *net = dev_net(skb->dev); - struct inet6_dev *idev; struct ipv6hdr *oldhdr; unsigned int chdr_len; unsigned char *buf; @@ -501,12 +493,8 @@ static int ipv6_rpl_srh_rcv(struct sk_buff *skb) u64 n = 0; u32 r; - idev = __in6_dev_get(skb->dev); - - accept_rpl_seg = net->ipv6.devconf_all->rpl_seg_enabled; - if (accept_rpl_seg > idev->cnf.rpl_seg_enabled) - accept_rpl_seg = idev->cnf.rpl_seg_enabled; - + accept_rpl_seg = min(READ_ONCE(net->ipv6.devconf_all->rpl_seg_enabled), + READ_ONCE(idev->cnf.rpl_seg_enabled)); if (!accept_rpl_seg) { kfree_skb(skb); return -1; @@ -693,10 +681,14 @@ static int ipv6_rthdr_rcv(struct sk_buff *skb) switch (hdr->type) { case IPV6_SRCRT_TYPE_4: /* segment routing */ - return ipv6_srh_rcv(skb); + if (!idev) + goto disabled; + return ipv6_srh_rcv(skb, idev); case IPV6_SRCRT_TYPE_3: /* rpl segment routing */ - return ipv6_rpl_srh_rcv(skb); + if (!idev) + goto disabled; + return ipv6_rpl_srh_rcv(skb, idev); default: break; } @@ -815,7 +807,7 @@ static int ipv6_rthdr_rcv(struct sk_buff *skb) ip6_route_input(skb); if (skb_dst(skb)->error) { - skb_push(skb, skb->data - skb_network_header(skb)); + skb_push(skb, -skb_network_offset(skb)); dst_input(skb); return -1; } @@ -832,7 +824,7 @@ static int ipv6_rthdr_rcv(struct sk_buff *skb) goto looped_back; } - skb_push(skb, skb->data - skb_network_header(skb)); + skb_push(skb, -skb_network_offset(skb)); dst_input(skb); return -1; @@ -841,6 +833,10 @@ static int ipv6_rthdr_rcv(struct sk_buff *skb) icmpv6_param_prob(skb, ICMPV6_HDR_FIELD, (&hdr->type) - skb_network_header(skb)); return -1; + +disabled: + kfree_skb_reason(skb, SKB_DROP_REASON_IPV6DISABLED); + return -1; } static const struct inet6_protocol rthdr_protocol = { diff --git a/net/ipv6/ioam6.c b/net/ipv6/ioam6.c index fee1e23c3ae1..05d8d9a5b353 100644 --- a/net/ipv6/ioam6.c +++ b/net/ipv6/ioam6.c @@ -650,6 +650,9 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb, struct ioam6_schema *sc, unsigned int sclen, bool is_input) { + /* Note: skb_dst_dev_rcu() can't be NULL at this point. */ + struct net_device *dev = skb_dst_dev_rcu(skb); + struct inet6_dev *i_skb_dev, *idev; struct timespec64 ts; ktime_t tstamp; u64 raw64; @@ -660,13 +663,16 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb, data = trace->data + trace->remlen * 4 - trace->nodelen * 4 - sclen * 4; + i_skb_dev = skb->dev ? __in6_dev_get(skb->dev) : NULL; + idev = __in6_dev_get(dev); + /* hop_lim and node_id */ if (trace->type.bit0) { byte = ipv6_hdr(skb)->hop_limit; if (is_input) byte--; - raw32 = dev_net(skb_dst(skb)->dev)->ipv6.sysctl.ioam6_id; + raw32 = READ_ONCE(dev_net(dev)->ipv6.sysctl.ioam6_id); *(__be32 *)data = cpu_to_be32((byte << 24) | raw32); data += sizeof(__be32); @@ -674,18 +680,18 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb, /* ingress_if_id and egress_if_id */ if (trace->type.bit1) { - if (!skb->dev) + if (!i_skb_dev) raw16 = IOAM6_U16_UNAVAILABLE; else - raw16 = (__force u16)READ_ONCE(__in6_dev_get(skb->dev)->cnf.ioam6_id); + raw16 = (__force u16)READ_ONCE(i_skb_dev->cnf.ioam6_id); *(__be16 *)data = cpu_to_be16(raw16); data += sizeof(__be16); - if (skb_dst(skb)->dev->flags & IFF_LOOPBACK) + if ((dev->flags & IFF_LOOPBACK) || !idev) raw16 = IOAM6_U16_UNAVAILABLE; else - raw16 = (__force u16)READ_ONCE(__in6_dev_get(skb_dst(skb)->dev)->cnf.ioam6_id); + raw16 = (__force u16)READ_ONCE(idev->cnf.ioam6_id); *(__be16 *)data = cpu_to_be16(raw16); data += sizeof(__be16); @@ -737,12 +743,16 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb, struct Qdisc *qdisc; __u32 qlen, backlog; - if (skb_dst(skb)->dev->flags & IFF_LOOPBACK) { + if (skb_dst(skb)->dev->flags & IFF_LOOPBACK || + skb_get_queue_mapping(skb) >= skb_dst(skb)->dev->num_tx_queues) { *(__be32 *)data = cpu_to_be32(IOAM6_U32_UNAVAILABLE); } else { queue = skb_get_tx_queue(skb_dst(skb)->dev, skb); qdisc = rcu_dereference(queue->qdisc); + + spin_lock_bh(qdisc_lock(qdisc)); qdisc_qstats_qlen_backlog(qdisc, &qlen, &backlog); + spin_unlock_bh(qdisc_lock(qdisc)); *(__be32 *)data = cpu_to_be32(backlog); } @@ -761,7 +771,7 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb, if (is_input) byte--; - raw64 = dev_net(skb_dst(skb)->dev)->ipv6.sysctl.ioam6_id_wide; + raw64 = READ_ONCE(dev_net(dev)->ipv6.sysctl.ioam6_id_wide); *(__be64 *)data = cpu_to_be64(((u64)byte << 56) | raw64); data += sizeof(__be64); @@ -769,18 +779,18 @@ static void __ioam6_fill_trace_data(struct sk_buff *skb, /* ingress_if_id and egress_if_id (wide) */ if (trace->type.bit9) { - if (!skb->dev) + if (!i_skb_dev) raw32 = IOAM6_U32_UNAVAILABLE; else - raw32 = READ_ONCE(__in6_dev_get(skb->dev)->cnf.ioam6_id_wide); + raw32 = READ_ONCE(i_skb_dev->cnf.ioam6_id_wide); *(__be32 *)data = cpu_to_be32(raw32); data += sizeof(__be32); - if (skb_dst(skb)->dev->flags & IFF_LOOPBACK) + if ((dev->flags & IFF_LOOPBACK) || !idev) raw32 = IOAM6_U32_UNAVAILABLE; else - raw32 = READ_ONCE(__in6_dev_get(skb_dst(skb)->dev)->cnf.ioam6_id_wide); + raw32 = READ_ONCE(idev->cnf.ioam6_id_wide); *(__be32 *)data = cpu_to_be32(raw32); data += sizeof(__be32); diff --git a/net/ipv6/ioam6_iptunnel.c b/net/ipv6/ioam6_iptunnel.c index 3a6065faaf9b..b536eae433d7 100644 --- a/net/ipv6/ioam6_iptunnel.c +++ b/net/ipv6/ioam6_iptunnel.c @@ -365,9 +365,12 @@ static int ioam6_output(struct net *net, struct sock *sk, struct sk_buff *skb) goto drop; } - local_bh_disable(); - dst_cache_set_ip6(&ilwt->cache, dst, &fl6.saddr); - local_bh_enable(); + /* cache only if we don't create a dst reference loop */ + if (skb_dst(skb)->lwtstate != dst->lwtstate) { + local_bh_disable(); + dst_cache_set_ip6(&ilwt->cache, dst, &fl6.saddr); + local_bh_enable(); + } } skb_dst_drop(skb); diff --git a/net/ipv6/ip6_flowlabel.c b/net/ipv6/ip6_flowlabel.c index bf909fa69a97..f6f2e25215c5 100644 --- a/net/ipv6/ip6_flowlabel.c +++ b/net/ipv6/ip6_flowlabel.c @@ -463,6 +463,22 @@ fl_create(struct net *net, struct sock *sk, struct in6_flowlabel_req *freq, return NULL; } +static bool fl_sock_at_lease_limit(const struct sock *sk) +{ + const struct ipv6_pinfo *np = inet6_sk(sk); + const struct ipv6_fl_socklist *sfl; + int count = 0; + + rcu_read_lock(); + for_each_sk_fl_rcu(np, sfl) { + if (++count >= FL_MAX_PER_SOCK) + break; + } + rcu_read_unlock(); + + return count >= FL_MAX_PER_SOCK; +} + static int mem_check(struct sock *sk) { struct ipv6_pinfo *np = inet6_sk(sk); @@ -673,6 +689,10 @@ static int ipv6_flowlabel_get(struct sock *sk, struct in6_flowlabel_req *freq, err = -ENOMEM; if (!sfl1) goto release; + err = -ENOBUFS; + if (fl_sock_at_lease_limit(sk) && + !capable(CAP_NET_ADMIN)) + goto release; if (fl->linger > fl1->linger) fl1->linger = fl->linger; if ((long)(fl->expires - fl1->expires) > 0) diff --git a/net/ipv6/ip6_gre.c b/net/ipv6/ip6_gre.c index 06a165775538..5e8c7071c638 100644 --- a/net/ipv6/ip6_gre.c +++ b/net/ipv6/ip6_gre.c @@ -942,7 +942,6 @@ static netdev_tx_t ip6erspan_tunnel_xmit(struct sk_buff *skb, { struct ip_tunnel_info *tun_info = NULL; struct ip6_tnl *t = netdev_priv(dev); - struct dst_entry *dst = skb_dst(skb); bool truncate = false; int encap_limit = -1; __u8 dsfield = false; @@ -1074,10 +1073,6 @@ static netdev_tx_t ip6erspan_tunnel_xmit(struct sk_buff *skb, /* Push GRE header. */ gre_build_header(skb, 8, TUNNEL_SEQ, proto, 0, htonl(atomic_fetch_inc(&t->o_seqno))); - /* TooBig packet may have updated dst->dev's mtu */ - if (!t->parms.collect_md && dst && dst_mtu(dst) > dst->dev->mtu) - dst->ops->update_pmtu(dst, NULL, skb, dst->dev->mtu, false); - err = ip6_tnl_xmit(skb, dev, dsfield, &fl6, encap_limit, &mtu, NEXTHDR_GRE); if (err != 0) { diff --git a/net/ipv6/ip6_input.c b/net/ipv6/ip6_input.c index 133610a49da6..9f8f17b3934f 100644 --- a/net/ipv6/ip6_input.c +++ b/net/ipv6/ip6_input.c @@ -572,6 +572,7 @@ int ip6_mc_input(struct sk_buff *skb) if (deliver) skb2 = skb_clone(skb, GFP_ATOMIC); else { + skb_orphan(skb); skb2 = skb; skb = NULL; } diff --git a/net/ipv6/ip6_output.c b/net/ipv6/ip6_output.c index 056c95f8f6d3..512a2a348609 100644 --- a/net/ipv6/ip6_output.c +++ b/net/ipv6/ip6_output.c @@ -60,7 +60,7 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff *skb) { struct dst_entry *dst = skb_dst(skb); - struct net_device *dev = dst->dev; + struct net_device *dev = dst_dev_rcu(dst); struct inet6_dev *idev = ip6_dst_idev(dst); unsigned int hh_len = LL_RESERVED_SPACE(dev); const struct in6_addr *daddr, *nexthop; @@ -70,15 +70,12 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff * /* Be paranoid, rather than too clever. */ if (unlikely(hh_len > skb_headroom(skb)) && dev->header_ops) { - /* Make sure idev stays alive */ - rcu_read_lock(); + /* idev stays alive because we hold rcu_read_lock(). */ skb = skb_expand_head(skb, hh_len); if (!skb) { IP6_INC_STATS(net, idev, IPSTATS_MIB_OUTDISCARDS); - rcu_read_unlock(); return -ENOMEM; } - rcu_read_unlock(); } hdr = ipv6_hdr(skb); @@ -125,7 +122,6 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff * IP6_UPD_PO_STATS(net, idev, IPSTATS_MIB_OUT, skb->len); - rcu_read_lock(); nexthop = rt6_nexthop(dst_rt6_info(dst), daddr); neigh = __ipv6_neigh_lookup_noref(dev, nexthop); @@ -133,7 +129,6 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff * if (unlikely(!neigh)) neigh = __neigh_create(&nd_tbl, nexthop, dev, false); if (IS_ERR(neigh)) { - rcu_read_unlock(); IP6_INC_STATS(net, idev, IPSTATS_MIB_OUTNOROUTES); kfree_skb_reason(skb, SKB_DROP_REASON_NEIGH_CREATEFAIL); return -EINVAL; @@ -141,7 +136,6 @@ static int ip6_finish_output2(struct net *net, struct sock *sk, struct sk_buff * } sock_confirm_neigh(skb, neigh); ret = neigh_output(neigh, skb, false); - rcu_read_unlock(); return ret; } @@ -227,22 +221,30 @@ static int ip6_finish_output(struct net *net, struct sock *sk, struct sk_buff *s int ip6_output(struct net *net, struct sock *sk, struct sk_buff *skb) { - struct net_device *dev = skb_dst(skb)->dev, *indev = skb->dev; - struct inet6_dev *idev = ip6_dst_idev(skb_dst(skb)); + struct dst_entry *dst = skb_dst(skb); + struct net_device *dev, *indev = skb->dev; + struct inet6_dev *idev; + int ret; skb->protocol = htons(ETH_P_IPV6); + rcu_read_lock(); + dev = dst_dev_rcu(dst); + idev = ip6_dst_idev(dst); skb->dev = dev; if (unlikely(!idev || READ_ONCE(idev->cnf.disable_ipv6))) { IP6_INC_STATS(net, idev, IPSTATS_MIB_OUTDISCARDS); + rcu_read_unlock(); kfree_skb_reason(skb, SKB_DROP_REASON_IPV6DISABLED); return 0; } - return NF_HOOK_COND(NFPROTO_IPV6, NF_INET_POST_ROUTING, - net, sk, skb, indev, dev, - ip6_finish_output, - !(IP6CB(skb)->flags & IP6SKB_REROUTED)); + ret = NF_HOOK_COND(NFPROTO_IPV6, NF_INET_POST_ROUTING, + net, sk, skb, indev, dev, + ip6_finish_output, + !(IP6CB(skb)->flags & IP6SKB_REROUTED)); + rcu_read_unlock(); + return ret; } EXPORT_SYMBOL(ip6_output); diff --git a/net/ipv6/ip6mr.c b/net/ipv6/ip6mr.c index 18451dfeae68..1724d50c50fd 100644 --- a/net/ipv6/ip6mr.c +++ b/net/ipv6/ip6mr.c @@ -1128,7 +1128,6 @@ static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt, msg->im6_src = ipv6_hdr(pkt)->saddr; msg->im6_dst = ipv6_hdr(pkt)->daddr; - skb_dst_set(skb, dst_clone(skb_dst(pkt))); skb->ip_summed = CHECKSUM_UNNECESSARY; } @@ -1138,6 +1137,7 @@ static int ip6mr_cache_report(const struct mr_table *mrt, struct sk_buff *pkt, return -EINVAL; } + skb_dst_drop(skb); mrt6msg_netlink_event(mrt, skb); /* Deliver to user space multicast routing algorithms */ @@ -1217,6 +1217,7 @@ static int ip6mr_cache_unresolved(struct mr_table *mrt, mifi_t mifi, skb->dev = dev; skb->skb_iif = dev->ifindex; } + skb_dst_drop(skb); skb_queue_tail(&c->_c.mfc_un.unres.unresolved, skb); err = 0; } @@ -1389,7 +1390,7 @@ int __init ip6_mr_init(void) mrt_cachep = kmem_cache_create("ip6_mrt_cache", sizeof(struct mfc6_cache), - 0, SLAB_HWCACHE_ALIGN, + 0, SLAB_HWCACHE_ALIGN | SLAB_ACCOUNT, NULL); if (!mrt_cachep) return -ENOMEM; diff --git a/net/ipv6/mcast.c b/net/ipv6/mcast.c index 9d8908c7cfa7..971603ee8979 100644 --- a/net/ipv6/mcast.c +++ b/net/ipv6/mcast.c @@ -229,7 +229,8 @@ static int __ipv6_sock_mc_join(struct sock *sk, int ifindex, return err; } - mc_lst->next = np->ipv6_mc_list; + rcu_assign_pointer(mc_lst->next, + sock_dereference(np->ipv6_mc_list, sk)); rcu_assign_pointer(np->ipv6_mc_list, mc_lst); return 0; @@ -289,7 +290,8 @@ int ipv6_sock_mc_drop(struct sock *sk, int ifindex, const struct in6_addr *addr) lnk = &mc_lst->next) { if ((ifindex == 0 || mc_lst->ifindex == ifindex) && ipv6_addr_equal(&mc_lst->addr, addr)) { - *lnk = mc_lst->next; + rcu_assign_pointer(*lnk, + sock_dereference(mc_lst->next, sk)); __ipv6_sock_mc_drop(sk, mc_lst); return 0; } @@ -343,7 +345,8 @@ void __ipv6_sock_mc_close(struct sock *sk) while ((mc_lst = sock_dereference(np->ipv6_mc_list, sk)) != NULL) { struct net_device *dev; - np->ipv6_mc_list = mc_lst->next; + rcu_assign_pointer(np->ipv6_mc_list, + sock_dereference(mc_lst->next, sk)); dev = __dev_get_by_index(net, mc_lst->ifindex); if (dev) { @@ -825,9 +828,11 @@ static void mld_del_delrec(struct inet6_dev *idev, struct ifmcaddr6 *im) } if (pmc) { if (pmc_prev) - rcu_assign_pointer(pmc_prev->next, pmc->next); + rcu_assign_pointer(pmc_prev->next, + mc_dereference(pmc->next, idev)); else - rcu_assign_pointer(idev->mc_tomb, pmc->next); + rcu_assign_pointer(idev->mc_tomb, + mc_dereference(pmc->next, idev)); } if (pmc) { @@ -985,7 +990,7 @@ static int __ipv6_dev_mc_inc(struct net_device *dev, return -ENOMEM; } - rcu_assign_pointer(mc->next, idev->mc_list); + rcu_assign_pointer(mc->next, mc_dereference(idev->mc_list, idev)); rcu_assign_pointer(idev->mc_list, mc); mca_get(mc); @@ -1017,7 +1022,8 @@ int __ipv6_dev_mc_dec(struct inet6_dev *idev, const struct in6_addr *addr) map = &ma->next) { if (ipv6_addr_equal(&ma->mca_addr, addr)) { if (--ma->mca_users == 0) { - *map = ma->next; + rcu_assign_pointer(*map, + mc_dereference(ma->next, idev)); igmp6_group_dropped(ma); ip6_mc_clear_src(ma); diff --git a/net/ipv6/netfilter/ip6_tables.c b/net/ipv6/netfilter/ip6_tables.c index 3f08d4600d69..551d3ca1036d 100644 --- a/net/ipv6/netfilter/ip6_tables.c +++ b/net/ipv6/netfilter/ip6_tables.c @@ -1735,13 +1735,11 @@ int ip6t_register_table(struct net *net, const struct xt_table *table, const struct ip6t_replace *repl, const struct nf_hook_ops *template_ops) { - struct nf_hook_ops *ops; - unsigned int num_ops; - int ret, i; - struct xt_table_info *newinfo; struct xt_table_info bootstrap = {0}; - void *loc_cpu_entry; + struct xt_table_info *newinfo; struct xt_table *new_table; + void *loc_cpu_entry; + int ret; newinfo = xt_alloc_table_info(repl->size); if (!newinfo) @@ -1756,7 +1754,7 @@ int ip6t_register_table(struct net *net, const struct xt_table *table, return ret; } - new_table = xt_register_table(net, table, &bootstrap, newinfo); + new_table = xt_register_table(net, table, template_ops, &bootstrap, newinfo); if (IS_ERR(new_table)) { struct ip6t_entry *iter; @@ -1766,34 +1764,6 @@ int ip6t_register_table(struct net *net, const struct xt_table *table, return PTR_ERR(new_table); } - if (!template_ops) - return 0; - - num_ops = hweight32(table->valid_hooks); - if (num_ops == 0) { - ret = -EINVAL; - goto out_free; - } - - ops = kmemdup(template_ops, sizeof(*ops) * num_ops, GFP_KERNEL); - if (!ops) { - ret = -ENOMEM; - goto out_free; - } - - for (i = 0; i < num_ops; i++) - ops[i].priv = new_table; - - new_table->ops = ops; - - ret = nf_register_net_hooks(net, ops, num_ops); - if (ret != 0) - goto out_free; - - return ret; - -out_free: - __ip6t_unregister_table(net, new_table); return ret; } diff --git a/net/ipv6/netfilter/ip6table_nat.c b/net/ipv6/netfilter/ip6table_nat.c index 52d597b16b65..21957d53e9c9 100644 --- a/net/ipv6/netfilter/ip6table_nat.c +++ b/net/ipv6/netfilter/ip6table_nat.c @@ -81,7 +81,7 @@ static int ip6t_nat_register_lookups(struct net *net) while (i) nf_nat_ipv6_unregister_fn(net, &ops[--i]); - kfree(ops); + kfree_rcu(ops, rcu); return ret; } } @@ -102,7 +102,7 @@ static void ip6t_nat_unregister_lookups(struct net *net) for (i = 0; i < ARRAY_SIZE(nf_nat_ipv6_ops); i++) nf_nat_ipv6_unregister_fn(net, &ops[i]); - kfree(ops); + kfree_rcu(ops, rcu); } static int ip6table_nat_table_init(struct net *net) diff --git a/net/ipv6/netfilter/nf_conntrack_reasm.c b/net/ipv6/netfilter/nf_conntrack_reasm.c index 662e3c78b741..0d19974e85cf 100644 --- a/net/ipv6/netfilter/nf_conntrack_reasm.c +++ b/net/ipv6/netfilter/nf_conntrack_reasm.c @@ -335,9 +335,9 @@ static int nf_ct_frag6_reasm(struct frag_queue *fq, struct sk_buff *skb, if (!reasm_data) goto err; - payload_len = ((skb->data - skb_network_header(skb)) - + payload_len = -skb_network_offset(skb) - sizeof(struct ipv6hdr) + fq->q.len - - sizeof(struct frag_hdr)); + sizeof(struct frag_hdr); if (payload_len > IPV6_MAXPLEN) { net_dbg_ratelimited("nf_ct_frag6_reasm: payload len = %d\n", payload_len); diff --git a/net/ipv6/reassembly.c b/net/ipv6/reassembly.c index 359b0b4ec1bb..a429620d06ed 100644 --- a/net/ipv6/reassembly.c +++ b/net/ipv6/reassembly.c @@ -274,9 +274,9 @@ static int ip6_frag_reasm(struct frag_queue *fq, struct sk_buff *skb, if (!reasm_data) goto out_oom; - payload_len = ((skb->data - skb_network_header(skb)) - + payload_len = -skb_network_offset(skb) - sizeof(struct ipv6hdr) + fq->q.len - - sizeof(struct frag_hdr)); + sizeof(struct frag_hdr); if (payload_len > IPV6_MAXPLEN) goto out_oversize; diff --git a/net/ipv6/route.c b/net/ipv6/route.c index 3193e6150539..baa43c7f0300 100644 --- a/net/ipv6/route.c +++ b/net/ipv6/route.c @@ -6340,8 +6340,6 @@ void fib6_info_hw_flags_set(struct net *net, struct fib6_info *f6i, err = rt6_fill_node(net, skb, f6i, NULL, NULL, NULL, 0, RTM_NEWROUTE, 0, 0, 0); if (err < 0) { - /* -EMSGSIZE implies BUG in rt6_nlmsg_size() */ - WARN_ON(err == -EMSGSIZE); kfree_skb(skb); goto errout; } diff --git a/net/ipv6/rpl_iptunnel.c b/net/ipv6/rpl_iptunnel.c index 702509dffd7c..e0970440b6ee 100644 --- a/net/ipv6/rpl_iptunnel.c +++ b/net/ipv6/rpl_iptunnel.c @@ -236,9 +236,12 @@ static int rpl_output(struct net *net, struct sock *sk, struct sk_buff *skb) goto drop; } - local_bh_disable(); - dst_cache_set_ip6(&rlwt->cache, dst, &fl6.saddr); - local_bh_enable(); + /* cache only if we don't create a dst reference loop */ + if (orig_dst->lwtstate != dst->lwtstate) { + local_bh_disable(); + dst_cache_set_ip6(&rlwt->cache, dst, &fl6.saddr); + local_bh_enable(); + } err = skb_cow_head(skb, LL_RESERVED_SPACE(dst->dev)); if (unlikely(err)) diff --git a/net/ipv6/seg6_iptunnel.c b/net/ipv6/seg6_iptunnel.c index 6fb1f99c5412..2a4caae36541 100644 --- a/net/ipv6/seg6_iptunnel.c +++ b/net/ipv6/seg6_iptunnel.c @@ -116,7 +116,7 @@ static __be32 seg6_make_flowlabel(struct net *net, struct sk_buff *skb, u32 hash; if (do_flowlabel > 0) { - hash = skb_get_hash(skb); + hash = skb_get_hash_net(net, skb); hash = rol32(hash, 16); flowlabel = (__force __be32)hash & IPV6_FLOWLABEL_MASK; } else if (!do_flowlabel && skb->protocol == htons(ETH_P_IPV6)) { diff --git a/net/ipv6/syncookies.c b/net/ipv6/syncookies.c index 593ead8a45d7..acacd758505e 100644 --- a/net/ipv6/syncookies.c +++ b/net/ipv6/syncookies.c @@ -161,9 +161,14 @@ struct sock *cookie_v6_check(struct sock *sk, struct sk_buff *skb) tcp_parse_options(sock_net(sk), skb, &tcp_opt, 0, NULL); if (tcp_opt.saw_tstamp && tcp_opt.rcv_tsecr) { - tsoff = secure_tcpv6_ts_off(sock_net(sk), - ipv6_hdr(skb)->daddr.s6_addr32, - ipv6_hdr(skb)->saddr.s6_addr32); + union tcp_seq_and_ts_off st; + + st = secure_tcpv6_seq_and_ts_off(sock_net(sk), + ipv6_hdr(skb)->daddr.s6_addr32, + ipv6_hdr(skb)->saddr.s6_addr32, + tcp_hdr(skb)->dest, + tcp_hdr(skb)->source); + tsoff = st.ts_off; tcp_opt.rcv_tsecr -= tsoff; } diff --git a/net/ipv6/tcp_ipv6.c b/net/ipv6/tcp_ipv6.c index 4a51015ef4ea..7adeb623cfb2 100644 --- a/net/ipv6/tcp_ipv6.c +++ b/net/ipv6/tcp_ipv6.c @@ -108,18 +108,14 @@ static void inet6_sk_rx_dst_set(struct sock *sk, const struct sk_buff *skb) } } -static u32 tcp_v6_init_seq(const struct sk_buff *skb) +static union tcp_seq_and_ts_off +tcp_v6_init_seq_and_ts_off(const struct net *net, const struct sk_buff *skb) { - return secure_tcpv6_seq(ipv6_hdr(skb)->daddr.s6_addr32, - ipv6_hdr(skb)->saddr.s6_addr32, - tcp_hdr(skb)->dest, - tcp_hdr(skb)->source); -} - -static u32 tcp_v6_init_ts_off(const struct net *net, const struct sk_buff *skb) -{ - return secure_tcpv6_ts_off(net, ipv6_hdr(skb)->daddr.s6_addr32, - ipv6_hdr(skb)->saddr.s6_addr32); + return secure_tcpv6_seq_and_ts_off(net, + ipv6_hdr(skb)->daddr.s6_addr32, + ipv6_hdr(skb)->saddr.s6_addr32, + tcp_hdr(skb)->dest, + tcp_hdr(skb)->source); } static int tcp_v6_pre_connect(struct sock *sk, struct sockaddr *uaddr, @@ -319,14 +315,16 @@ static int tcp_v6_connect(struct sock *sk, struct sockaddr *uaddr, sk_set_txhash(sk); if (likely(!tp->repair)) { + union tcp_seq_and_ts_off st; + + st = secure_tcpv6_seq_and_ts_off(net, + np->saddr.s6_addr32, + sk->sk_v6_daddr.s6_addr32, + inet->inet_sport, + inet->inet_dport); if (!tp->write_seq) - WRITE_ONCE(tp->write_seq, - secure_tcpv6_seq(np->saddr.s6_addr32, - sk->sk_v6_daddr.s6_addr32, - inet->inet_sport, - inet->inet_dport)); - tp->tsoffset = secure_tcpv6_ts_off(net, np->saddr.s6_addr32, - sk->sk_v6_daddr.s6_addr32); + WRITE_ONCE(tp->write_seq, st.seq); + tp->tsoffset = st.ts_off; } if (tcp_fastopen_defer_connect(sk, &err)) @@ -826,8 +824,7 @@ const struct tcp_request_sock_ops tcp_request_sock_ipv6_ops = { .cookie_init_seq = cookie_v6_init_sequence, #endif .route_req = tcp_v6_route_req, - .init_seq = tcp_v6_init_seq, - .init_ts_off = tcp_v6_init_ts_off, + .init_seq_and_ts_off = tcp_v6_init_seq_and_ts_off, .send_synack = tcp_v6_send_synack, }; @@ -1327,7 +1324,7 @@ static struct sock *tcp_v6_syn_recv_sock(const struct sock *sk, struct sk_buff * tcp_ca_openreq_child(newsk, dst); tcp_sync_mss(newsk, dst_mtu(dst)); - newtp->advmss = tcp_mss_clamp(tcp_sk(sk), dst_metric_advmss(dst)); + newtp->advmss = tcp_mss_clamp(tcp_sk(sk), tcp_dst_advmss(dst)); tcp_initialize_rcv_mss(newsk); @@ -1992,9 +1989,9 @@ static void get_tcp6_sock(struct seq_file *seq, struct sock *sp, int i) } else if (icsk->icsk_pending == ICSK_TIME_PROBE0) { timer_active = 4; timer_expires = icsk->icsk_timeout; - } else if (timer_pending(&sp->sk_timer)) { + } else if (timer_pending(&icsk->icsk_keepalive_timer)) { timer_active = 2; - timer_expires = sp->sk_timer.expires; + timer_expires = icsk->icsk_keepalive_timer.expires; } else { timer_active = 0; timer_expires = jiffies; diff --git a/net/ipv6/udp.c b/net/ipv6/udp.c index 6df2459f2561..8cb969b4d862 100644 --- a/net/ipv6/udp.c +++ b/net/ipv6/udp.c @@ -162,6 +162,49 @@ static int compute_score(struct sock *sk, struct net *net, return score; } +/** + * udp6_lib_lookup1() - Simplified lookup using primary hash (destination port) + * @net: Network namespace + * @saddr: Source address, network order + * @sport: Source port, network order + * @daddr: Destination address, network order + * @hnum: Destination port, host order + * @dif: Destination interface index + * @sdif: Destination bridge port index, if relevant + * @udptable: Set of UDP hash tables + * + * Simplified lookup to be used as fallback if no sockets are found due to a + * potential race between (receive) address change, and lookup happening before + * the rehash operation. This function ignores SO_REUSEPORT groups while scoring + * result sockets, because if we have one, we don't need the fallback at all. + * + * Called under rcu_read_lock(). + * + * Return: socket with highest matching score if any, NULL if none + */ +static struct sock *udp6_lib_lookup1(struct net *net, + const struct in6_addr *saddr, __be16 sport, + const struct in6_addr *daddr, + unsigned int hnum, int dif, int sdif, + const struct udp_table *udptable) +{ + unsigned int slot = udp_hashfn(net, hnum, udptable->mask); + struct udp_hslot *hslot = &udptable->hash[slot]; + struct sock *sk, *result = NULL; + int score, badness = 0; + + sk_for_each_rcu(sk, &hslot->head) { + score = compute_score(sk, net, + saddr, sport, daddr, hnum, dif, sdif); + if (score > badness) { + result = sk; + badness = score; + } + } + + return result; +} + /* called with rcu_read_lock() */ static struct sock *udp6_lib_lookup2(struct net *net, const struct in6_addr *saddr, __be16 sport, @@ -266,6 +309,13 @@ struct sock *__udp6_lib_lookup(struct net *net, result = udp6_lib_lookup2(net, saddr, sport, &in6addr_any, hnum, dif, sdif, hslot2, skb); + if (!IS_ERR_OR_NULL(result)) + goto done; + + /* Cover address change/lookup/rehash race: see __udp4_lib_lookup() */ + result = udp6_lib_lookup1(net, saddr, sport, daddr, hnum, dif, sdif, + udptable); + done: if (IS_ERR(result)) return NULL; diff --git a/net/ipv6/xfrm6_input.c b/net/ipv6/xfrm6_input.c index 00d2a1b8b679..d637478adedb 100644 --- a/net/ipv6/xfrm6_input.c +++ b/net/ipv6/xfrm6_input.c @@ -41,7 +41,8 @@ static int xfrm6_transport_finish2(struct net *net, struct sock *sk, int xfrm6_transport_finish(struct sk_buff *skb, int async) { struct xfrm_offload *xo = xfrm_offload(skb); - int nhlen = skb->data - skb_network_header(skb); + struct net_device *dev = skb->dev; + int nhlen = -skb_network_offset(skb); skb_network_header(skb)[IP6CB(skb)->nhoff] = XFRM_MODE_SKB_CB(skb)->protocol; @@ -66,8 +67,10 @@ int xfrm6_transport_finish(struct sk_buff *skb, int async) } NF_HOOK(NFPROTO_IPV6, NF_INET_PRE_ROUTING, - dev_net(skb->dev), NULL, skb, skb->dev, NULL, + dev_net(dev), NULL, skb, dev, NULL, xfrm6_transport_finish2); + if (async) + dev_put(dev); return 0; } diff --git a/net/kcm/kcmsock.c b/net/kcm/kcmsock.c index 93b1c499d76d..f8a960c7d168 100644 --- a/net/kcm/kcmsock.c +++ b/net/kcm/kcmsock.c @@ -630,7 +630,7 @@ static int kcm_write_msgs(struct kcm_sock *kcm) skb = txm->frag_skb; } - if (WARN_ON(!skb_shinfo(skb)->nr_frags)) { + if (WARN_ON_ONCE(!skb_shinfo(skb)->nr_frags)) { ret = -EINVAL; goto out; } @@ -750,7 +750,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) { struct sock *sk = sock->sk; struct kcm_sock *kcm = kcm_sk(sk); - struct sk_buff *skb = NULL, *head = NULL; + struct sk_buff *skb = NULL, *head = NULL, *frag_prev = NULL; size_t copy, copied = 0; long timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); int eor = (sock->type == SOCK_DGRAM) ? @@ -825,6 +825,7 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) else skb->next = tskb; + frag_prev = skb; skb = tskb; skb->ip_summed = CHECKSUM_UNNECESSARY; continue; @@ -935,6 +936,22 @@ static int kcm_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) out_error: kcm_push(kcm); + /* When MAX_SKB_FRAGS was reached, a new skb was allocated and + * linked into the frag_list before data copy. If the copy + * subsequently failed, this skb has zero frags. Remove it from + * the frag_list to prevent kcm_write_msgs from later hitting + * WARN_ON(!skb_shinfo(skb)->nr_frags). + */ + if (frag_prev && !skb_shinfo(skb)->nr_frags) { + if (head == frag_prev) + skb_shinfo(head)->frag_list = NULL; + else + frag_prev->next = NULL; + kfree_skb(skb); + /* Update skb as it may be saved in partial_message via goto */ + skb = frag_prev; + } + if (sock->type == SOCK_SEQPACKET) { /* Wrote some bytes before encountering an * error, return partial success. diff --git a/net/l2tp/l2tp_core.c b/net/l2tp/l2tp_core.c index 3c701795fa10..717559ead628 100644 --- a/net/l2tp/l2tp_core.c +++ b/net/l2tp/l2tp_core.c @@ -380,11 +380,13 @@ int l2tp_session_register(struct l2tp_session *session, } l2tp_tunnel_inc_refcount(tunnel); + WRITE_ONCE(session->tunnel, tunnel); hlist_add_head_rcu(&session->global_hlist, g_head); spin_unlock_bh(&pn->l2tp_session_hlist_lock); } else { l2tp_tunnel_inc_refcount(tunnel); + WRITE_ONCE(session->tunnel, tunnel); } hlist_add_head_rcu(&session->hlist, head); @@ -698,7 +700,8 @@ void l2tp_recv_common(struct l2tp_session *session, struct sk_buff *skb, if (!session->lns_mode && !session->send_seq) { trace_session_seqnum_lns_enable(session); session->send_seq = 1; - l2tp_session_set_header_len(session, tunnel->version); + l2tp_session_set_header_len(session, tunnel->version, + tunnel->encap); } } else { /* No sequence numbers. @@ -719,7 +722,8 @@ void l2tp_recv_common(struct l2tp_session *session, struct sk_buff *skb, if (!session->lns_mode && session->send_seq) { trace_session_seqnum_lns_disable(session); session->send_seq = 0; - l2tp_session_set_header_len(session, tunnel->version); + l2tp_session_set_header_len(session, tunnel->version, + tunnel->encap); } else if (session->send_seq) { pr_debug_ratelimited("%s: recv data has no seq numbers when required. Discarding.\n", session->name); @@ -1579,7 +1583,8 @@ EXPORT_SYMBOL_GPL(l2tp_session_delete); /* We come here whenever a session's send_seq, cookie_len or * l2specific_type parameters are set. */ -void l2tp_session_set_header_len(struct l2tp_session *session, int version) +void l2tp_session_set_header_len(struct l2tp_session *session, int version, + enum l2tp_encap_type encap) { if (version == L2TP_HDR_VER_2) { session->hdr_len = 6; @@ -1588,7 +1593,7 @@ void l2tp_session_set_header_len(struct l2tp_session *session, int version) } else { session->hdr_len = 4 + session->cookie_len; session->hdr_len += l2tp_get_l2specific_len(session); - if (session->tunnel->encap == L2TP_ENCAPTYPE_UDP) + if (encap == L2TP_ENCAPTYPE_UDP) session->hdr_len += 4; } } @@ -1602,7 +1607,6 @@ struct l2tp_session *l2tp_session_create(int priv_size, struct l2tp_tunnel *tunn session = kzalloc(sizeof(*session) + priv_size, GFP_KERNEL); if (session) { session->magic = L2TP_SESSION_MAGIC; - session->tunnel = tunnel; session->session_id = session_id; session->peer_session_id = peer_session_id; @@ -1638,7 +1642,7 @@ struct l2tp_session *l2tp_session_create(int priv_size, struct l2tp_tunnel *tunn memcpy(&session->peer_cookie[0], &cfg->peer_cookie[0], cfg->peer_cookie_len); } - l2tp_session_set_header_len(session, tunnel->version); + l2tp_session_set_header_len(session, tunnel->version, tunnel->encap); refcount_set(&session->ref_count, 1); diff --git a/net/l2tp/l2tp_core.h b/net/l2tp/l2tp_core.h index 91ebf0a3f499..61f402c3e7f4 100644 --- a/net/l2tp/l2tp_core.h +++ b/net/l2tp/l2tp_core.h @@ -261,7 +261,8 @@ void l2tp_recv_common(struct l2tp_session *session, struct sk_buff *skb, int l2tp_udp_encap_recv(struct sock *sk, struct sk_buff *skb); /* Transmit path helpers for sending packets over the tunnel socket. */ -void l2tp_session_set_header_len(struct l2tp_session *session, int version); +void l2tp_session_set_header_len(struct l2tp_session *session, int version, + enum l2tp_encap_type encap); int l2tp_xmit_skb(struct l2tp_session *session, struct sk_buff *skb); /* Pseudowire management. diff --git a/net/l2tp/l2tp_debugfs.c b/net/l2tp/l2tp_debugfs.c index 4595b56d175d..fe56dbae43be 100644 --- a/net/l2tp/l2tp_debugfs.c +++ b/net/l2tp/l2tp_debugfs.c @@ -307,6 +307,10 @@ static int l2tp_dfs_seq_release(struct inode *inode, struct file *file) seq = file->private_data; pd = seq->private; + if (pd->session) + l2tp_session_dec_refcount(pd->session); + if (pd->tunnel) + l2tp_tunnel_dec_refcount(pd->tunnel); if (pd->net) put_net_track(pd->net, &pd->ns_tracker); kfree(pd); diff --git a/net/l2tp/l2tp_netlink.c b/net/l2tp/l2tp_netlink.c index e27e00cb16c6..25800de8b482 100644 --- a/net/l2tp/l2tp_netlink.c +++ b/net/l2tp/l2tp_netlink.c @@ -247,8 +247,8 @@ static int l2tp_nl_cmd_tunnel_create(struct sk_buff *skb, struct genl_info *info kfree(tunnel); goto out; } - ret = l2tp_tunnel_notify(&l2tp_nl_family, info, tunnel, - L2TP_CMD_TUNNEL_CREATE); + l2tp_tunnel_notify(&l2tp_nl_family, info, tunnel, + L2TP_CMD_TUNNEL_CREATE); l2tp_tunnel_dec_refcount(tunnel); out: @@ -304,8 +304,8 @@ static int l2tp_nl_cmd_tunnel_modify(struct sk_buff *skb, struct genl_info *info goto out; } - ret = l2tp_tunnel_notify(&l2tp_nl_family, info, - tunnel, L2TP_CMD_TUNNEL_MODIFY); + l2tp_tunnel_notify(&l2tp_nl_family, info, + tunnel, L2TP_CMD_TUNNEL_MODIFY); l2tp_tunnel_dec_refcount(tunnel); @@ -637,8 +637,8 @@ static int l2tp_nl_cmd_session_create(struct sk_buff *skb, struct genl_info *inf if (ret >= 0) { session = l2tp_tunnel_get_session(tunnel, session_id); if (session) { - ret = l2tp_session_notify(&l2tp_nl_family, info, session, - L2TP_CMD_SESSION_CREATE); + l2tp_session_notify(&l2tp_nl_family, info, session, + L2TP_CMD_SESSION_CREATE); l2tp_session_dec_refcount(session); } } @@ -690,8 +690,10 @@ static int l2tp_nl_cmd_session_modify(struct sk_buff *skb, struct genl_info *inf session->recv_seq = nla_get_u8(info->attrs[L2TP_ATTR_RECV_SEQ]); if (info->attrs[L2TP_ATTR_SEND_SEQ]) { + struct l2tp_tunnel *tunnel = session->tunnel; + session->send_seq = nla_get_u8(info->attrs[L2TP_ATTR_SEND_SEQ]); - l2tp_session_set_header_len(session, session->tunnel->version); + l2tp_session_set_header_len(session, tunnel->version, tunnel->encap); } if (info->attrs[L2TP_ATTR_LNS_MODE]) @@ -700,8 +702,8 @@ static int l2tp_nl_cmd_session_modify(struct sk_buff *skb, struct genl_info *inf if (info->attrs[L2TP_ATTR_RECV_TIMEOUT]) session->reorder_timeout = nla_get_msecs(info->attrs[L2TP_ATTR_RECV_TIMEOUT]); - ret = l2tp_session_notify(&l2tp_nl_family, info, - session, L2TP_CMD_SESSION_MODIFY); + l2tp_session_notify(&l2tp_nl_family, info, + session, L2TP_CMD_SESSION_MODIFY); l2tp_session_dec_refcount(session); diff --git a/net/l2tp/l2tp_ppp.c b/net/l2tp/l2tp_ppp.c index 2e856a83a750..21b1016020f3 100644 --- a/net/l2tp/l2tp_ppp.c +++ b/net/l2tp/l2tp_ppp.c @@ -1056,64 +1056,76 @@ static int pppol2tp_ioctl(struct socket *sock, unsigned int cmd, { struct pppol2tp_ioc_stats stats; struct l2tp_session *session; + int err = 0; + + session = pppol2tp_sock_to_session(sock->sk); + /* Validate session presence and magic integrity ONLY for commands + * that belong to L2TP and require a valid session. + */ switch (cmd) { case PPPIOCGMRU: case PPPIOCGFLAGS: - session = sock->sk->sk_user_data; + case PPPIOCSMRU: + case PPPIOCSFLAGS: + case PPPIOCGL2TPSTATS: if (!session) return -ENOTCONN; - if (WARN_ON(session->magic != L2TP_SESSION_MAGIC)) + if (session->magic != L2TP_SESSION_MAGIC) { + sock_put(sock->sk); return -EBADF; + } + break; + default: + break; + } + switch (cmd) { + case PPPIOCGMRU: + case PPPIOCGFLAGS: /* Not defined for tunnels */ - if (!session->session_id && !session->peer_session_id) - return -ENOSYS; + if (!session->session_id && !session->peer_session_id) { + err = -ENOSYS; + break; + } - if (put_user(0, (int __user *)arg)) - return -EFAULT; + if (put_user(0, (int __user *)arg)) { + err = -EFAULT; + break; + } break; case PPPIOCSMRU: case PPPIOCSFLAGS: - session = sock->sk->sk_user_data; - if (!session) - return -ENOTCONN; - - if (WARN_ON(session->magic != L2TP_SESSION_MAGIC)) - return -EBADF; - /* Not defined for tunnels */ - if (!session->session_id && !session->peer_session_id) - return -ENOSYS; + if (!session->session_id && !session->peer_session_id) { + err = -ENOSYS; + break; + } - if (!access_ok((int __user *)arg, sizeof(int))) - return -EFAULT; + if (!access_ok((int __user *)arg, sizeof(int))) { + err = -EFAULT; + break; + } break; case PPPIOCGL2TPSTATS: - session = sock->sk->sk_user_data; - if (!session) - return -ENOTCONN; - - if (WARN_ON(session->magic != L2TP_SESSION_MAGIC)) - return -EBADF; - /* Session 0 represents the parent tunnel */ if (!session->session_id && !session->peer_session_id) { u32 session_id; - int err; if (copy_from_user(&stats, (void __user *)arg, - sizeof(stats))) - return -EFAULT; + sizeof(stats))) { + err = -EFAULT; + break; + } session_id = stats.session_id; err = pppol2tp_tunnel_copy_stats(&stats, session->tunnel); if (err < 0) - return err; + break; stats.session_id = session_id; } else { @@ -1123,15 +1135,21 @@ static int pppol2tp_ioctl(struct socket *sock, unsigned int cmd, stats.tunnel_id = session->tunnel->tunnel_id; stats.using_ipsec = l2tp_tunnel_uses_xfrm(session->tunnel); - if (copy_to_user((void __user *)arg, &stats, sizeof(stats))) - return -EFAULT; + if (copy_to_user((void __user *)arg, &stats, sizeof(stats))) { + err = -EFAULT; + break; + } break; default: - return -ENOIOCTLCMD; + err = -ENOIOCTLCMD; + break; } - return 0; + if (session) + sock_put(sock->sk); + + return err; } /***************************************************************************** @@ -1194,7 +1212,8 @@ static int pppol2tp_session_setsockopt(struct sock *sk, po->chan.hdrlen = val ? PPPOL2TP_L2TP_HDR_SIZE_SEQ : PPPOL2TP_L2TP_HDR_SIZE_NOSEQ; } - l2tp_session_set_header_len(session, session->tunnel->version); + l2tp_session_set_header_len(session, session->tunnel->version, + session->tunnel->encap); break; case PPPOL2TP_SO_LNSMODE: @@ -1601,7 +1620,53 @@ static const struct seq_operations pppol2tp_seq_ops = { .stop = pppol2tp_seq_stop, .show = pppol2tp_seq_show, }; -#endif /* CONFIG_PROC_FS */ + +static int pppol2tp_proc_open(struct inode *inode, struct file *file) +{ + struct net *net = pde_data(inode); + struct pppol2tp_seq_data *pd; + + net = maybe_get_net(net); + if (!net) + return -ENXIO; + + pd = __seq_open_private(file, &pppol2tp_seq_ops, sizeof(*pd)); + if (!pd) { + put_net(net); + return -ENOMEM; + } + +#ifdef CONFIG_NET_NS + pd->p.net = net; + netns_tracker_alloc(net, &pd->p.ns_tracker, GFP_KERNEL); +#endif + return 0; +} + +static int pppol2tp_proc_release(struct inode *inode, struct file *file) +{ + struct seq_file *seq = file->private_data; + struct pppol2tp_seq_data *pd = seq->private; + + if (pd->session) + l2tp_session_dec_refcount(pd->session); + if (pd->tunnel) + l2tp_tunnel_dec_refcount(pd->tunnel); + +#ifdef CONFIG_NET_NS + put_net_track(pd->p.net, &pd->p.ns_tracker); +#else + put_net(&init_net); +#endif + return seq_release_private(inode, file); +} + +static const struct proc_ops pppol2tp_proc_ops = { + .proc_open = pppol2tp_proc_open, + .proc_read = seq_read, + .proc_lseek = seq_lseek, + .proc_release = pppol2tp_proc_release, +}; /***************************************************************************** * Network namespace @@ -1612,8 +1677,8 @@ static __net_init int pppol2tp_init_net(struct net *net) struct proc_dir_entry *pde; int err = 0; - pde = proc_create_net("pppol2tp", 0444, net->proc_net, - &pppol2tp_seq_ops, sizeof(struct pppol2tp_seq_data)); + pde = proc_create_data("pppol2tp", 0444, net->proc_net, + &pppol2tp_proc_ops, net); if (!pde) { err = -ENOMEM; goto out; @@ -1628,9 +1693,13 @@ static __net_exit void pppol2tp_exit_net(struct net *net) remove_proc_entry("pppol2tp", net->proc_net); } +#endif /* CONFIG_PROC_FS */ + static struct pernet_operations pppol2tp_net_ops = { +#ifdef CONFIG_PROC_FS .init = pppol2tp_init_net, .exit = pppol2tp_exit_net, +#endif .id = &pppol2tp_net_id, }; diff --git a/net/mac80211/agg-rx.c b/net/mac80211/agg-rx.c index c6fa53230450..32c1be3096b3 100644 --- a/net/mac80211/agg-rx.c +++ b/net/mac80211/agg-rx.c @@ -103,7 +103,7 @@ void ___ieee80211_stop_rx_ba_session(struct sta_info *sta, u16 tid, if (!tid_rx) return; - del_timer_sync(&tid_rx->session_timer); + timer_shutdown_sync(&tid_rx->session_timer); /* make sure ieee80211_sta_reorder_release() doesn't re-arm the timer */ spin_lock_bh(&tid_rx->reorder_lock); diff --git a/net/mac80211/chan.c b/net/mac80211/chan.c index 4a21e53afa72..29a6ad65c9a8 100644 --- a/net/mac80211/chan.c +++ b/net/mac80211/chan.c @@ -1067,6 +1067,7 @@ int ieee80211_link_unreserve_chanctx(struct ieee80211_link_data *link) { struct ieee80211_sub_if_data *sdata = link->sdata; struct ieee80211_chanctx *ctx = link->reserved_chanctx; + struct ieee80211_chanctx *old_ctx = NULL; lockdep_assert_held(&sdata->local->chanctx_mtx); @@ -1081,6 +1082,7 @@ int ieee80211_link_unreserve_chanctx(struct ieee80211_link_data *link) if (WARN_ON(!ctx->replace_ctx)) return -EINVAL; + old_ctx = ctx->replace_ctx; WARN_ON(ctx->replace_ctx->replace_state != IEEE80211_CHANCTX_WILL_BE_REPLACED); WARN_ON(ctx->replace_ctx->replace_ctx != ctx); @@ -1091,6 +1093,10 @@ int ieee80211_link_unreserve_chanctx(struct ieee80211_link_data *link) list_del_rcu(&ctx->list); kfree_rcu(ctx, rcu_head); + + if (ieee80211_chanctx_refcount(sdata->local, + old_ctx) == 0) + ieee80211_free_chanctx(sdata->local, old_ctx); } else { ieee80211_free_chanctx(sdata->local, ctx); } @@ -1322,7 +1328,9 @@ ieee80211_link_use_reserved_reassign(struct ieee80211_link_data *link) ieee80211_check_fast_xmit_iface(sdata); - if (ieee80211_chanctx_refcount(local, old_ctx) == 0) + if (ieee80211_chanctx_refcount(local, old_ctx) == 0 && + !(old_ctx->replace_state == IEEE80211_CHANCTX_WILL_BE_REPLACED && + old_ctx->replace_ctx)) ieee80211_free_chanctx(local, old_ctx); ieee80211_recalc_chanctx_min_def(local, new_ctx, NULL); @@ -1386,7 +1394,8 @@ ieee80211_link_use_reserved_assign(struct ieee80211_link_data *link) } static bool -ieee80211_link_has_in_place_reservation(struct ieee80211_link_data *link) +ieee80211_link_has_in_place_reservation(struct ieee80211_link_data *link, + struct ieee80211_chanctx *ctx) { struct ieee80211_sub_if_data *sdata = link->sdata; struct ieee80211_chanctx *old_ctx, *new_ctx; @@ -1396,6 +1405,9 @@ ieee80211_link_has_in_place_reservation(struct ieee80211_link_data *link) new_ctx = link->reserved_chanctx; old_ctx = ieee80211_link_get_chanctx(link); + if (new_ctx != ctx) + return false; + if (!old_ctx) return false; @@ -1408,6 +1420,12 @@ ieee80211_link_has_in_place_reservation(struct ieee80211_link_data *link) if (new_ctx->replace_state != IEEE80211_CHANCTX_REPLACES_OTHER) return false; + if (new_ctx->replace_ctx != old_ctx) + return false; + + if (old_ctx->replace_ctx != new_ctx) + return false; + return true; } @@ -1457,7 +1475,7 @@ static int ieee80211_chsw_switch_vifs(struct ieee80211_local *local, list_for_each_entry(link, &ctx->reserved_links, reserved_chanctx_list) { - if (!ieee80211_link_has_in_place_reservation(link)) + if (!ieee80211_link_has_in_place_reservation(link, ctx)) continue; old_ctx = ieee80211_link_get_chanctx(link); @@ -1566,7 +1584,7 @@ static int ieee80211_vif_use_reserved_switch(struct ieee80211_local *local) list_for_each_entry(link, &ctx->replace_ctx->assigned_links, assigned_chanctx_list) { n_assigned++; - if (link->reserved_chanctx) { + if (ieee80211_link_has_in_place_reservation(link, ctx)) { n_reserved++; if (link->reserved_ready) n_ready++; @@ -1587,7 +1605,7 @@ static int ieee80211_vif_use_reserved_switch(struct ieee80211_local *local) ctx->conf.radar_enabled = false; list_for_each_entry(link, &ctx->reserved_links, reserved_chanctx_list) { - if (ieee80211_link_has_in_place_reservation(link) && + if (ieee80211_link_has_in_place_reservation(link, ctx) && !link->reserved_ready) return -EAGAIN; @@ -1661,7 +1679,7 @@ static int ieee80211_vif_use_reserved_switch(struct ieee80211_local *local) struct ieee80211_bss_conf *link_conf = link->conf; u64 changed = 0; - if (!ieee80211_link_has_in_place_reservation(link)) + if (!ieee80211_link_has_in_place_reservation(link, ctx)) continue; rcu_assign_pointer(link_conf->chanctx_conf, @@ -1714,7 +1732,8 @@ static int ieee80211_vif_use_reserved_switch(struct ieee80211_local *local) */ list_for_each_entry_safe(link, link_tmp, &ctx->reserved_links, reserved_chanctx_list) { - if (WARN_ON(ieee80211_link_has_in_place_reservation(link))) + if (WARN_ON(ieee80211_link_has_in_place_reservation(link, + ctx))) continue; if (WARN_ON(link->reserved_chanctx != ctx)) @@ -1802,7 +1821,9 @@ static void __ieee80211_link_release_channel(struct ieee80211_link_data *link) } ieee80211_assign_link_chanctx(link, NULL); - if (ieee80211_chanctx_refcount(local, ctx) == 0) + if (ieee80211_chanctx_refcount(local, ctx) == 0 && + !(ctx->replace_state == IEEE80211_CHANCTX_WILL_BE_REPLACED && + ctx->replace_ctx)) ieee80211_free_chanctx(local, ctx); link->radar_required = false; diff --git a/net/mac80211/fils_aead.c b/net/mac80211/fils_aead.c index 912c46f74d24..622e8ee7ab0d 100644 --- a/net/mac80211/fils_aead.c +++ b/net/mac80211/fils_aead.c @@ -95,6 +95,14 @@ static int aes_siv_encrypt(const u8 *key, size_t key_len, if (res) return res; + /* With an empty plaintext there is no ciphertext to produce and + * the S2V result is the complete output. + */ + if (!plain_len) { + memcpy(out, v, AES_BLOCK_SIZE); + return 0; + } + /* Use a temporary buffer of the plaintext to handle need for * overwriting this during AES-CTR. */ diff --git a/net/mac80211/ibss.c b/net/mac80211/ibss.c index 5542c93edfba..2474feb149b3 100644 --- a/net/mac80211/ibss.c +++ b/net/mac80211/ibss.c @@ -695,7 +695,9 @@ static void ieee80211_ibss_disconnect(struct ieee80211_sub_if_data *sdata) ifibss->state = IEEE80211_IBSS_MLME_SEARCH; - sta_info_flush(sdata); + netif_carrier_off(sdata->dev); + if (!sta_info_flush(sdata)) + synchronize_net(); spin_lock_bh(&ifibss->incomplete_lock); while (!list_empty(&ifibss->incomplete_stations)) { @@ -709,8 +711,6 @@ static void ieee80211_ibss_disconnect(struct ieee80211_sub_if_data *sdata) } spin_unlock_bh(&ifibss->incomplete_lock); - netif_carrier_off(sdata->dev); - sdata->vif.cfg.ibss_joined = false; sdata->vif.cfg.ibss_creator = false; sdata->vif.bss_conf.enable_beacon = false; @@ -741,7 +741,6 @@ static void ieee80211_csa_connection_drop_work(struct wiphy *wiphy, sdata_lock(sdata); ieee80211_ibss_disconnect(sdata); - synchronize_rcu(); skb_queue_purge(&sdata->skb_queue); /* trigger a scan to find another IBSS network to join */ @@ -1882,8 +1881,6 @@ int ieee80211_ibss_leave(struct ieee80211_sub_if_data *sdata) memset(&ifibss->ht_capa, 0, sizeof(ifibss->ht_capa)); memset(&ifibss->ht_capa_mask, 0, sizeof(ifibss->ht_capa_mask)); - synchronize_rcu(); - skb_queue_purge(&sdata->skb_queue); del_timer_sync(&sdata->u.ibss.timer); diff --git a/net/mac80211/ieee80211_i.h b/net/mac80211/ieee80211_i.h index fba2e9109300..b156e2e991d1 100644 --- a/net/mac80211/ieee80211_i.h +++ b/net/mac80211/ieee80211_i.h @@ -2270,7 +2270,7 @@ void __ieee80211_tx_skb_tid_band(struct ieee80211_sub_if_data *sdata, /* sta_out needs to be checked for ERR_PTR() before using */ int ieee80211_lookup_ra_sta(struct ieee80211_sub_if_data *sdata, struct sk_buff *skb, - struct sta_info **sta_out); + struct sta_info **sta_out, bool bss); static inline void ieee80211_tx_skb_tid_band(struct ieee80211_sub_if_data *sdata, diff --git a/net/mac80211/mesh_pathtbl.c b/net/mac80211/mesh_pathtbl.c index 530581ba812b..7cbc54b41828 100644 --- a/net/mac80211/mesh_pathtbl.c +++ b/net/mac80211/mesh_pathtbl.c @@ -572,6 +572,12 @@ void mesh_fast_tx_cache(struct ieee80211_sub_if_data *sdata, goto unlock_sta; spin_lock(&cache->walk_lock); + if ((READ_ONCE(mpath->flags) & MESH_PATH_DELETED) || + (mppath && (READ_ONCE(mppath->flags) & MESH_PATH_DELETED))) { + kfree(entry); + goto unlock_cache; + } + prev = rhashtable_lookup_get_insert_fast(&cache->rht, &entry->rhash, fast_tx_rht_params); @@ -689,8 +695,10 @@ struct mesh_path *mesh_path_add(struct ieee80211_sub_if_data *sdata, return ERR_PTR(-ENOSPC); new_mpath = mesh_path_new(sdata, dst, GFP_ATOMIC); - if (!new_mpath) + if (!new_mpath) { + atomic_dec(&sdata->u.mesh.mpaths); return ERR_PTR(-ENOMEM); + } tbl = &sdata->u.mesh.mesh_paths; spin_lock_bh(&tbl->walk_lock); @@ -703,6 +711,7 @@ struct mesh_path *mesh_path_add(struct ieee80211_sub_if_data *sdata, if (mpath) { kfree(new_mpath); + atomic_dec(&sdata->u.mesh.mpaths); if (IS_ERR(mpath)) return mpath; @@ -728,10 +737,15 @@ int mpp_path_add(struct ieee80211_sub_if_data *sdata, if (is_multicast_ether_addr(dst)) return -ENOTSUPP; + if (!atomic_add_unless(&sdata->u.mesh.mpaths, 1, MESH_MAX_MPATHS)) + return -ENOSPC; + new_mpath = mesh_path_new(sdata, dst, GFP_ATOMIC); - if (!new_mpath) + if (!new_mpath) { + atomic_dec(&sdata->u.mesh.mpaths); return -ENOMEM; + } memcpy(new_mpath->mpp, mpp, ETH_ALEN); tbl = &sdata->u.mesh.mpp_paths; @@ -744,10 +758,12 @@ int mpp_path_add(struct ieee80211_sub_if_data *sdata, hlist_add_head_rcu(&new_mpath->walk_list, &tbl->walk_head); spin_unlock_bh(&tbl->walk_lock); - if (ret) + if (ret) { kfree(new_mpath); - else + atomic_dec(&sdata->u.mesh.mpaths); + } else { mesh_fast_tx_flush_addr(sdata, dst); + } sdata->u.mesh.mpp_paths_generation++; return ret; @@ -793,7 +809,8 @@ static void mesh_path_free_rcu(struct mesh_table *tbl, struct ieee80211_sub_if_data *sdata = mpath->sdata; spin_lock_bh(&mpath->state_lock); - mpath->flags |= MESH_PATH_RESOLVING | MESH_PATH_DELETED; + WRITE_ONCE(mpath->flags, + mpath->flags | MESH_PATH_RESOLVING | MESH_PATH_DELETED); mesh_gate_del(tbl, mpath); spin_unlock_bh(&mpath->state_lock); timer_shutdown_sync(&mpath->timer); @@ -807,6 +824,9 @@ static void __mesh_path_del(struct mesh_table *tbl, struct mesh_path *mpath) { hlist_del_rcu(&mpath->walk_list); rhashtable_remove_fast(&tbl->rhead, &mpath->rhash, mesh_rht_params); + spin_lock_bh(&mpath->state_lock); + WRITE_ONCE(mpath->flags, mpath->flags | MESH_PATH_DELETED); + spin_unlock_bh(&mpath->state_lock); if (tbl == &mpath->sdata->u.mesh.mpp_paths) mesh_fast_tx_flush_addr(mpath->sdata, mpath->dst); else diff --git a/net/mac80211/mlme.c b/net/mac80211/mlme.c index ac3c2c3334fd..c085cfa5273e 100644 --- a/net/mac80211/mlme.c +++ b/net/mac80211/mlme.c @@ -452,7 +452,10 @@ ieee80211_determine_chantype(struct ieee80211_sub_if_data *sdata, while (!cfg80211_chandef_usable(sdata->local->hw.wiphy, chandef, tracking ? 0 : IEEE80211_CHAN_DISABLED)) { - if (WARN_ON(chandef->width == NL80211_CHAN_WIDTH_20_NOHT)) { + if (chandef->width == NL80211_CHAN_WIDTH_20_NOHT) { + sdata_info(sdata, + "unusable channel (%d MHz) for connection\n", + chandef->chan->center_freq); ret = IEEE80211_CONN_DISABLE_HT | IEEE80211_CONN_DISABLE_VHT | IEEE80211_CONN_DISABLE_HE | @@ -4026,6 +4029,7 @@ static bool ieee80211_assoc_config_link(struct ieee80211_link_data *link, bool is_6ghz = cbss->channel->band == NL80211_BAND_6GHZ; bool is_s1g = cbss->channel->band == NL80211_BAND_S1GHZ; const struct cfg80211_bss_ies *bss_ies = NULL; + struct ieee802_11_elems *bss_elems = NULL; struct ieee80211_supported_band *sband; struct ieee802_11_elems *elems; const __le16 prof_bss_param_ch_present = @@ -4103,7 +4107,6 @@ static bool ieee80211_assoc_config_link(struct ieee80211_link_data *link, (!(link->u.mgd.conn_flags & IEEE80211_CONN_DISABLE_VHT) && (!elems->vht_cap_elem || !elems->vht_operation)))) { const struct cfg80211_bss_ies *ies; - struct ieee802_11_elems *bss_elems; rcu_read_lock(); ies = rcu_dereference(cbss->ies); @@ -4160,8 +4163,6 @@ static bool ieee80211_assoc_config_link(struct ieee80211_link_data *link, sdata_info(sdata, "AP bug: VHT operation missing from AssocResp\n"); } - - kfree(bss_elems); } /* @@ -4389,6 +4390,7 @@ static bool ieee80211_assoc_config_link(struct ieee80211_link_data *link, ret = true; out: kfree(elems); + kfree(bss_elems); kfree(bss_ies); return ret; } diff --git a/net/mac80211/rc80211_minstrel_ht.c b/net/mac80211/rc80211_minstrel_ht.c index b34c80522047..afedbcdcd174 100644 --- a/net/mac80211/rc80211_minstrel_ht.c +++ b/net/mac80211/rc80211_minstrel_ht.c @@ -7,6 +7,7 @@ #include #include #include +#include #include #include #include @@ -1968,14 +1969,39 @@ minstrel_ht_alloc(struct ieee80211_hw *hw) } #ifdef CONFIG_MAC80211_DEBUGFS +static int minstrel_ht_fixed_rate_idx_get(void *data, u64 *val) +{ + *val = *(u32 *)data; + return 0; +} + +static int minstrel_ht_fixed_rate_idx_set(void *data, u64 val) +{ + u32 idx = val; + + /* U32_MAX is the default and keeps fixed rate processing disabled */ + if (val != U32_MAX && + (val > U16_MAX || + MI_RATE_GROUP(idx) >= ARRAY_SIZE(minstrel_mcs_groups) || + MI_RATE_IDX(idx) >= MCS_GROUP_RATES)) + return -EINVAL; + + *(u32 *)data = idx; + return 0; +} + +DEFINE_DEBUGFS_ATTRIBUTE(minstrel_ht_fixed_rate_idx_fops, + minstrel_ht_fixed_rate_idx_get, + minstrel_ht_fixed_rate_idx_set, "%llu\n"); + static void minstrel_ht_add_debugfs(struct ieee80211_hw *hw, void *priv, struct dentry *debugfsdir) { struct minstrel_priv *mp = priv; mp->fixed_rate_idx = (u32) -1; - debugfs_create_u32("fixed_rate_idx", S_IRUGO | S_IWUGO, debugfsdir, - &mp->fixed_rate_idx); + debugfs_create_file("fixed_rate_idx", S_IRUGO | S_IWUGO, debugfsdir, + &mp->fixed_rate_idx, &minstrel_ht_fixed_rate_idx_fops); } #endif diff --git a/net/mac80211/rx.c b/net/mac80211/rx.c index b0f975dbdde0..378d58395532 100644 --- a/net/mac80211/rx.c +++ b/net/mac80211/rx.c @@ -4835,6 +4835,7 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx, u8 sa[ETH_ALEN]; } addrs __aligned(2); struct ieee80211_sta_rx_stats *stats; + u32 encoded_rate; /* for parallel-rx, we need to have DUP_VALIDATED, otherwise we write * to a common data structure; drivers can implement that per queue @@ -4906,6 +4907,11 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx, /* after this point, don't punt to the slowpath! */ + if (fast_rx->uses_rss) + stats = this_cpu_ptr(rx->link_sta->pcpu_rx_stats); + else + stats = &rx->link_sta->rx_stats; + if (rx->key && !(status->flag & RX_FLAG_MIC_STRIPPED) && pskb_trim(skb, skb->len - fast_rx->icv_len)) goto drop; @@ -4937,9 +4943,14 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx, /* push the addresses in front */ memcpy(skb_push(skb, sizeof(addrs)), &addrs, sizeof(addrs)); + /* capture before mesh forward may memset or free skb->cb */ + encoded_rate = sta_stats_encode_rate(status); + res = ieee80211_rx_mesh_data(rx->sdata, rx->sta, rx->skb); switch (res) { case RX_QUEUED: + stats->last_rx = jiffies; + stats->last_rate = encoded_rate; return true; case RX_CONTINUE: break; @@ -4953,11 +4964,6 @@ static bool ieee80211_invoke_fast_rx(struct ieee80211_rx_data *rx, drop: dev_kfree_skb(skb); - if (fast_rx->uses_rss) - stats = this_cpu_ptr(rx->link_sta->pcpu_rx_stats); - else - stats = &rx->link_sta->rx_stats; - stats->dropped++; return true; } diff --git a/net/mac80211/sta_info.c b/net/mac80211/sta_info.c index 9d7d7ee9d7ce..44019b515bf7 100644 --- a/net/mac80211/sta_info.c +++ b/net/mac80211/sta_info.c @@ -122,6 +122,8 @@ static void __cleanup_single_sta(struct sta_info *sta) struct ieee80211_local *local = sdata->local; struct ps_data *ps; + cancel_work_sync(&sta->drv_deliver_wk); + if (test_sta_flag(sta, WLAN_STA_PS_STA) || test_sta_flag(sta, WLAN_STA_PS_DRIVER) || test_sta_flag(sta, WLAN_STA_PS_DELIVER)) { @@ -160,8 +162,6 @@ static void __cleanup_single_sta(struct sta_info *sta) if (ieee80211_vif_is_mesh(&sdata->vif)) mesh_sta_cleanup(sta); - cancel_work_sync(&sta->drv_deliver_wk); - /* * Destroy aggregation state here. It would be nice to wait for the * driver to finish aggregation stop and then clean up, but for now @@ -349,6 +349,15 @@ static void sta_info_free_link(struct link_sta_info *link_sta) free_percpu(link_sta->pcpu_rx_stats); } +static void sta_link_free_rcu(struct rcu_head *head) +{ + struct sta_link_alloc *alloc = + container_of(head, struct sta_link_alloc, rcu_head); + + sta_info_free_link(&alloc->info); + kfree(alloc); +} + static void sta_remove_link(struct sta_info *sta, unsigned int link_id, bool unhash) { @@ -374,10 +383,8 @@ static void sta_remove_link(struct sta_info *sta, unsigned int link_id, sta->sta.valid_links &= ~BIT(link_id); RCU_INIT_POINTER(sta->link[link_id], NULL); RCU_INIT_POINTER(sta->sta.link[link_id], NULL); - if (alloc) { - sta_info_free_link(&alloc->info); - kfree_rcu(alloc, rcu_head); - } + if (alloc) + call_rcu(&alloc->rcu_head, sta_link_free_rcu); ieee80211_sta_recalc_aggregates(&sta->sta); } @@ -1466,6 +1473,8 @@ static void __sta_info_destroy_part2(struct sta_info *sta, bool recalc) sta->dead = true; + cancel_work_sync(&sta->drv_deliver_wk); + local->num_sta--; local->sta_generation++; @@ -2587,6 +2596,39 @@ static inline u64 sta_get_stats_bytes(struct ieee80211_sta_rx_stats *rxstats) return value; } +#ifdef CONFIG_MAC80211_MESH +static void sta_set_mesh_sinfo(struct sta_info *sta, + struct station_info *sinfo) +{ + struct ieee80211_local *local = sta->sdata->local; + + sinfo->filled |= BIT_ULL(NL80211_STA_INFO_LLID) | + BIT_ULL(NL80211_STA_INFO_PLID) | + BIT_ULL(NL80211_STA_INFO_PLINK_STATE) | + BIT_ULL(NL80211_STA_INFO_LOCAL_PM) | + BIT_ULL(NL80211_STA_INFO_PEER_PM) | + BIT_ULL(NL80211_STA_INFO_NONPEER_PM) | + BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_GATE) | + BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_AS); + + sinfo->llid = sta->mesh->llid; + sinfo->plid = sta->mesh->plid; + sinfo->plink_state = sta->mesh->plink_state; + if (test_sta_flag(sta, WLAN_STA_TOFFSET_KNOWN)) { + sinfo->filled |= BIT_ULL(NL80211_STA_INFO_T_OFFSET); + sinfo->t_offset = sta->mesh->t_offset; + } + sinfo->local_pm = sta->mesh->local_pm; + sinfo->peer_pm = sta->mesh->peer_pm; + sinfo->nonpeer_pm = sta->mesh->nonpeer_pm; + sinfo->connected_to_gate = sta->mesh->connected_to_gate; + sinfo->connected_to_as = sta->mesh->connected_to_as; + + sinfo->filled |= BIT_ULL(NL80211_STA_INFO_AIRTIME_LINK_METRIC); + sinfo->airtime_link_metric = airtime_link_metric_get(local, sta); +} +#endif + void sta_set_sinfo(struct sta_info *sta, struct station_info *sinfo, bool tidstats) { @@ -2770,31 +2812,10 @@ void sta_set_sinfo(struct sta_info *sta, struct station_info *sinfo, sta_set_tidstats(sta, &sinfo->pertid[i], i); } - if (ieee80211_vif_is_mesh(&sdata->vif)) { #ifdef CONFIG_MAC80211_MESH - sinfo->filled |= BIT_ULL(NL80211_STA_INFO_LLID) | - BIT_ULL(NL80211_STA_INFO_PLID) | - BIT_ULL(NL80211_STA_INFO_PLINK_STATE) | - BIT_ULL(NL80211_STA_INFO_LOCAL_PM) | - BIT_ULL(NL80211_STA_INFO_PEER_PM) | - BIT_ULL(NL80211_STA_INFO_NONPEER_PM) | - BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_GATE) | - BIT_ULL(NL80211_STA_INFO_CONNECTED_TO_AS); - - sinfo->llid = sta->mesh->llid; - sinfo->plid = sta->mesh->plid; - sinfo->plink_state = sta->mesh->plink_state; - if (test_sta_flag(sta, WLAN_STA_TOFFSET_KNOWN)) { - sinfo->filled |= BIT_ULL(NL80211_STA_INFO_T_OFFSET); - sinfo->t_offset = sta->mesh->t_offset; - } - sinfo->local_pm = sta->mesh->local_pm; - sinfo->peer_pm = sta->mesh->peer_pm; - sinfo->nonpeer_pm = sta->mesh->nonpeer_pm; - sinfo->connected_to_gate = sta->mesh->connected_to_gate; - sinfo->connected_to_as = sta->mesh->connected_to_as; + if (ieee80211_vif_is_mesh(&sdata->vif)) + sta_set_mesh_sinfo(sta, sinfo); #endif - } sinfo->bss_param.flags = 0; if (sdata->vif.bss_conf.use_cts_prot) @@ -2850,12 +2871,6 @@ void sta_set_sinfo(struct sta_info *sta, struct station_info *sinfo, sinfo->filled |= BIT_ULL(NL80211_STA_INFO_ACK_SIGNAL_AVG); } - - if (ieee80211_vif_is_mesh(&sdata->vif)) { - sinfo->filled |= BIT_ULL(NL80211_STA_INFO_AIRTIME_LINK_METRIC); - sinfo->airtime_link_metric = - airtime_link_metric_get(local, sta); - } } u32 sta_get_expected_throughput(struct sta_info *sta) diff --git a/net/mac80211/tx.c b/net/mac80211/tx.c index 7acf34206a40..cc7507ffaf00 100644 --- a/net/mac80211/tx.c +++ b/net/mac80211/tx.c @@ -2467,7 +2467,7 @@ static inline bool ieee80211_is_tdls_setup(struct sk_buff *skb) int ieee80211_lookup_ra_sta(struct ieee80211_sub_if_data *sdata, struct sk_buff *skb, - struct sta_info **sta_out) + struct sta_info **sta_out, bool bss) { struct sta_info *sta; @@ -2488,7 +2488,10 @@ int ieee80211_lookup_ra_sta(struct ieee80211_sub_if_data *sdata, *sta_out = ERR_PTR(-ENOENT); return 0; } - sta = sta_info_get_bss(sdata, skb->data); + if (bss) + sta = sta_info_get_bss(sdata, skb->data); + else + sta = sta_info_get(sdata, skb->data); break; #ifdef CONFIG_MAC80211_MESH case NL80211_IFTYPE_MESH_POINT: @@ -4308,7 +4311,8 @@ void __ieee80211_subif_start_xmit(struct sk_buff *skb, ieee80211_mesh_xmit_fast(sdata, skb, ctrl_flags)) goto out; - if (ieee80211_lookup_ra_sta(sdata, skb, &sta)) + if (ieee80211_lookup_ra_sta(sdata, skb, &sta, + skb->protocol == sdata->control_port_protocol)) goto out_free; if (IS_ERR(sta)) @@ -4619,7 +4623,8 @@ static void ieee80211_8023_xmit(struct ieee80211_sub_if_data *sdata, { struct ieee80211_tx_info *info; struct ieee80211_local *local = sdata->local; - struct tid_ampdu_tx *tid_tx; + struct ieee80211_chanctx_conf *chanctx_conf; + struct tid_ampdu_tx *tid_tx = NULL; struct sk_buff *seg, *next; unsigned int skbs = 0, len = 0; u16 queue; @@ -4639,7 +4644,9 @@ static void ieee80211_8023_xmit(struct ieee80211_sub_if_data *sdata, ieee80211_aggr_check(sdata, sta, skb); tid = skb->priority & IEEE80211_QOS_CTL_TAG1D_MASK; - tid_tx = rcu_dereference(sta->ampdu_mlme.tid_tx[tid]); + + if (sta) + tid_tx = rcu_dereference(sta->ampdu_mlme.tid_tx[tid]); if (tid_tx) { if (!test_bit(HT_AGG_STATE_OPERATIONAL, &tid_tx->state)) { /* fall back to non-offload slow path */ @@ -4666,6 +4673,16 @@ static void ieee80211_8023_xmit(struct ieee80211_sub_if_data *sdata, sdata = container_of(sdata->bss, struct ieee80211_sub_if_data, u.ap); + /* MLD transmissions must not rely on the band */ + if (!ieee80211_vif_is_mld(&sdata->vif)) { + chanctx_conf = rcu_dereference(sdata->vif.bss_conf.chanctx_conf); + if (unlikely(!chanctx_conf)) { + kfree_skb_list(skb); + return; + } + info->band = chanctx_conf->def.chan->band; + } + info->flags |= IEEE80211_TX_CTL_HW_80211_ENCAP; info->control.vif = &sdata->vif; @@ -4688,8 +4705,11 @@ static void ieee80211_8023_xmit(struct ieee80211_sub_if_data *sdata, } dev_sw_netstats_tx_add(dev, skbs, len); - sta->deflink.tx_stats.packets[queue] += skbs; - sta->deflink.tx_stats.bytes[queue] += len; + + if (sta) { + sta->deflink.tx_stats.packets[queue] += skbs; + sta->deflink.tx_stats.bytes[queue] += len; + } ieee80211_tpt_led_trig_tx(local, len); @@ -4701,34 +4721,68 @@ static void ieee80211_8023_xmit(struct ieee80211_sub_if_data *sdata, kfree_skb(skb); } -netdev_tx_t ieee80211_subif_start_xmit_8023(struct sk_buff *skb, - struct net_device *dev) +static bool ieee80211_check_mcast_offload(struct ieee80211_sub_if_data *sdata, + struct sk_buff *skb) +{ + struct ethhdr *ehdr = (struct ethhdr *)skb->data; + + if ((ieee80211_vif_is_mld(&sdata->vif) && + (sdata->vif.type == NL80211_IFTYPE_AP && + !ieee80211_hw_check(&sdata->local->hw, MLO_MCAST_MULTI_LINK_TX))) || + (sdata->vif.type == NL80211_IFTYPE_AP_VLAN && + !sdata->wdev.use_4addr)) + return false; + + if (!is_multicast_ether_addr(skb->data) || + !(sdata->vif.offload_flags & IEEE80211_OFFLOAD_ENCAP_MCAST) || + sdata->control_port_protocol == ehdr->h_proto) + return false; + + return true; +} + +static void __ieee80211_subif_start_xmit_8023(struct sk_buff *skb, + struct net_device *dev) { struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev); struct ethhdr *ehdr = (struct ethhdr *)skb->data; + struct ieee80211_link_data *link; struct ieee80211_key *key; struct sta_info *sta; - if (unlikely(!ieee80211_sdata_running(sdata) || skb->len < ETH_HLEN)) { - kfree_skb(skb); - return NETDEV_TX_OK; - } - rcu_read_lock(); - if (ieee80211_lookup_ra_sta(sdata, skb, &sta)) { + if (unlikely(sdata->control_port_protocol == ehdr->h_proto)) + goto skip_offload; + + if (ieee80211_lookup_ra_sta(sdata, skb, &sta, false)) { kfree_skb(skb); goto out; } - if (unlikely(IS_ERR_OR_NULL(sta) || !sta->uploaded || - !test_sta_flag(sta, WLAN_STA_AUTHORIZED) || - sdata->control_port_protocol == ehdr->h_proto)) + /* + * If STA is invalid, use the multicast offload path when applicable. + * In AP mode, drop the frame if there are no associated stations; + * otherwise use the default link and multicast key for transmission. + */ + if (unlikely(IS_ERR_OR_NULL(sta) && + ieee80211_check_mcast_offload(sdata, skb))) { + sta = NULL; + if (ieee80211_vif_get_num_mcast_if(sdata) <= 0) { + /* No associated STAs - no need to send multicast frames. */ + kfree_skb(skb); + goto out; + } + link = &sdata->deflink; + key = rcu_dereference(link->default_multicast_key); + } else if (unlikely(IS_ERR_OR_NULL(sta) || !sta->uploaded || + !test_sta_flag(sta, WLAN_STA_AUTHORIZED))) { goto skip_offload; - - key = rcu_dereference(sta->ptk[sta->ptk_idx]); - if (!key) - key = rcu_dereference(sdata->default_unicast_key); + } else { + key = rcu_dereference(sta->ptk[sta->ptk_idx]); + if (!key) + key = rcu_dereference(sdata->default_unicast_key); + } if (key && (!(key->flags & KEY_FLAG_UPLOADED_TO_HARDWARE) || key->conf.cipher == WLAN_CIPHER_SUITE_TKIP)) @@ -4742,6 +4796,30 @@ netdev_tx_t ieee80211_subif_start_xmit_8023(struct sk_buff *skb, ieee80211_subif_start_xmit(skb, dev); out: rcu_read_unlock(); +} + +netdev_tx_t ieee80211_subif_start_xmit_8023(struct sk_buff *skb, + struct net_device *dev) +{ + struct ieee80211_sub_if_data *sdata = IEEE80211_DEV_TO_SUB_IF(dev); + struct ethhdr *ehdr = (struct ethhdr *)skb->data; + + if (unlikely(!ieee80211_sdata_running(sdata) || skb->len < ETH_HLEN)) { + kfree_skb(skb); + return NETDEV_TX_OK; + } + + if (unlikely(is_multicast_ether_addr(ehdr->h_dest) && + ieee80211_multicast_to_unicast(skb, dev))) { + struct sk_buff_head queue; + + __skb_queue_head_init(&queue); + ieee80211_convert_to_unicast(skb, dev, &queue); + while ((skb = __skb_dequeue(&queue))) + __ieee80211_subif_start_xmit_8023(skb, dev); + } else { + __ieee80211_subif_start_xmit_8023(skb, dev); + } return NETDEV_TX_OK; } @@ -4759,7 +4837,7 @@ ieee80211_build_data_template(struct ieee80211_sub_if_data *sdata, rcu_read_lock(); - if (ieee80211_lookup_ra_sta(sdata, skb, &sta)) { + if (ieee80211_lookup_ra_sta(sdata, skb, &sta, false)) { kfree_skb(skb); skb = ERR_PTR(-EINVAL); goto out; @@ -4830,7 +4908,7 @@ static bool ieee80211_tx_pending_skb(struct ieee80211_local *local, } result = ieee80211_tx(sdata, NULL, skb, true); } else if (info->flags & IEEE80211_TX_CTL_HW_80211_ENCAP) { - if (ieee80211_lookup_ra_sta(sdata, skb, &sta)) { + if (ieee80211_lookup_ra_sta(sdata, skb, &sta, true)) { dev_kfree_skb(skb); return true; } @@ -6218,7 +6296,7 @@ int ieee80211_tx_control_port(struct wiphy *wiphy, struct net_device *dev, * AF_PACKET */ rcu_read_lock(); - err = ieee80211_lookup_ra_sta(sdata, skb, &sta); + err = ieee80211_lookup_ra_sta(sdata, skb, &sta, true); if (err) { dev_kfree_skb(skb); rcu_read_unlock(); diff --git a/net/mac80211/util.c b/net/mac80211/util.c index 3a3cd09bdab6..28d2a7fafc31 100644 --- a/net/mac80211/util.c +++ b/net/mac80211/util.c @@ -2814,12 +2814,7 @@ int ieee80211_reconfig(struct ieee80211_local *local) } break; case NL80211_IFTYPE_NAN: - res = ieee80211_reconfig_nan(sdata); - if (res < 0) { - sdata_unlock(sdata); - ieee80211_handle_reconfig_failure(local); - return res; - } + WARN_ON(ieee80211_reconfig_nan(sdata)); break; case NL80211_IFTYPE_AP_VLAN: case NL80211_IFTYPE_MONITOR: diff --git a/net/mctp/device.c b/net/mctp/device.c index 67576cb2728e..ee430b73b097 100644 --- a/net/mctp/device.c +++ b/net/mctp/device.c @@ -344,13 +344,13 @@ static struct mctp_dev *mctp_add_dev(struct net_device *dev) mdev->net = mctp_default_net(dev_net(dev)); - /* associate to net_device */ refcount_set(&mdev->refs, 1); - rcu_assign_pointer(dev->mctp_ptr, mdev); - dev_hold(dev); mdev->dev = dev; + /* Associate to net_device, publish the mdev. */ + rcu_assign_pointer(dev->mctp_ptr, mdev); + return mdev; } diff --git a/net/mctp/route.c b/net/mctp/route.c index 6d6e19c04939..acd3cecc8a36 100644 --- a/net/mctp/route.c +++ b/net/mctp/route.c @@ -298,8 +298,14 @@ static int mctp_frag_queue(struct mctp_sk_key *key, struct sk_buff *skb) & MCTP_HDR_SEQ_MASK; if (!key->reasm_head) { - key->reasm_head = skb; - key->reasm_tailp = &(skb_shinfo(skb)->frag_list); + /* Since we're manipulating the shared frag_list, ensure it isn't + * shared with any other SKBs. + */ + key->reasm_head = skb_unshare(skb, GFP_ATOMIC); + if (!key->reasm_head) + return -ENOMEM; + + key->reasm_tailp = &(skb_shinfo(key->reasm_head)->frag_list); key->last_seq = this_seq; return 0; } diff --git a/net/mctp/test/route-test.c b/net/mctp/test/route-test.c index a944490a724d..ef6a3d6115ca 100644 --- a/net/mctp/test/route-test.c +++ b/net/mctp/test/route-test.c @@ -665,6 +665,117 @@ static void mctp_route_input_sk_keys_to_desc( KUNIT_ARRAY_PARAM(mctp_route_input_sk_keys, mctp_route_input_sk_keys_tests, mctp_route_input_sk_keys_to_desc); +/* Input route to socket, using a fragmented message created from clones. + */ +static void mctp_test_route_input_cloned_frag(struct kunit *test) +{ + /* 5 packet fragments, forming 2 complete messages */ + const struct mctp_hdr hdrs[5] = { + RX_FRAG(FL_S, 0), + RX_FRAG(0, 1), + RX_FRAG(FL_E, 2), + RX_FRAG(FL_S, 0), + RX_FRAG(FL_E, 1), + }; + struct mctp_test_route *rt; + struct mctp_test_dev *dev; + struct sk_buff *skb[5]; + struct sk_buff *rx_skb; + struct socket *sock; + size_t data_len; + u8 compare[100]; + u8 flat[100]; + size_t total; + void *p; + int rc; + + /* Arbitrary length */ + data_len = 3; + total = data_len + sizeof(struct mctp_hdr); + + __mctp_route_test_init(test, &dev, &rt, &sock); + + /* Create a single skb initially with concatenated packets */ + skb[0] = mctp_test_create_skb(&hdrs[0], 5 * total); + skb[0]->dev = dev->ndev; + __mctp_cb(skb[0]); + memset(skb[0]->data, 0 * 0x11, skb[0]->len); + memcpy(skb[0]->data, &hdrs[0], sizeof(struct mctp_hdr)); + + /* Extract and populate packets */ + for (int i = 1; i < 5; i++) { + skb[i] = skb_clone(skb[i - 1], GFP_ATOMIC); + KUNIT_ASSERT_TRUE(test, skb[i]); + p = skb_pull(skb[i], total); + KUNIT_ASSERT_TRUE(test, p); + skb_reset_network_header(skb[i]); + skb[i]->dev = dev->ndev; + __mctp_cb(skb[i]); + memcpy(skb[i]->data, &hdrs[i], sizeof(struct mctp_hdr)); + memset(&skb[i]->data[sizeof(struct mctp_hdr)], i * 0x11, data_len); + } + for (int i = 0; i < 5; i++) + skb_trim(skb[i], total); + + /* SOM packets have a type byte to match the socket */ + skb[0]->data[4] = 0; + skb[3]->data[4] = 0; + + skb_dump("pkt1 ", skb[0], false); + skb_dump("pkt2 ", skb[1], false); + skb_dump("pkt3 ", skb[2], false); + skb_dump("pkt4 ", skb[3], false); + skb_dump("pkt5 ", skb[4], false); + + for (int i = 0; i < 5; i++) { + KUNIT_EXPECT_EQ(test, refcount_read(&skb[i]->users), 1); + /* Take a reference so we can check refcounts at the end */ + skb_get(skb[i]); + } + + /* Feed the fragments into MCTP core */ + for (int i = 0; i < 5; i++) { + rc = mctp_route_input(&rt->rt, skb[i]); + KUNIT_EXPECT_EQ(test, rc, 0); + } + + /* Receive first reassembled message */ + rx_skb = skb_recv_datagram(sock->sk, MSG_DONTWAIT, &rc); + KUNIT_EXPECT_EQ(test, rc, 0); + KUNIT_EXPECT_EQ(test, rx_skb->len, 3 * data_len); + rc = skb_copy_bits(rx_skb, 0, flat, rx_skb->len); + for (int i = 0; i < rx_skb->len; i++) + compare[i] = (i / data_len) * 0x11; + /* Set type byte */ + compare[0] = 0; + + KUNIT_EXPECT_MEMEQ(test, flat, compare, rx_skb->len); + KUNIT_EXPECT_EQ(test, refcount_read(&rx_skb->users), 1); + kfree_skb(rx_skb); + + /* Receive second reassembled message */ + rx_skb = skb_recv_datagram(sock->sk, MSG_DONTWAIT, &rc); + KUNIT_EXPECT_EQ(test, rc, 0); + KUNIT_EXPECT_EQ(test, rx_skb->len, 2 * data_len); + rc = skb_copy_bits(rx_skb, 0, flat, rx_skb->len); + for (int i = 0; i < rx_skb->len; i++) + compare[i] = (i / data_len + 3) * 0x11; + /* Set type byte */ + compare[0] = 0; + + KUNIT_EXPECT_MEMEQ(test, flat, compare, rx_skb->len); + KUNIT_EXPECT_EQ(test, refcount_read(&rx_skb->users), 1); + kfree_skb(rx_skb); + + /* Check input skb refcounts */ + for (int i = 0; i < 5; i++) { + KUNIT_EXPECT_EQ(test, refcount_read(&skb[i]->users), 1); + kfree_skb(skb[i]); + } + + __mctp_route_test_fini(test, dev, rt, sock); +} + static struct kunit_case mctp_test_cases[] = { KUNIT_CASE_PARAM(mctp_test_fragment, mctp_frag_gen_params), KUNIT_CASE_PARAM(mctp_test_rx_input, mctp_rx_input_gen_params), @@ -673,6 +784,7 @@ static struct kunit_case mctp_test_cases[] = { mctp_route_input_sk_reasm_gen_params), KUNIT_CASE_PARAM(mctp_test_route_input_sk_keys, mctp_route_input_sk_keys_gen_params), + KUNIT_CASE(mctp_test_route_input_cloned_frag), {} }; diff --git a/net/mpls/af_mpls.c b/net/mpls/af_mpls.c index 9685405d744b..b94ea43b965d 100644 --- a/net/mpls/af_mpls.c +++ b/net/mpls/af_mpls.c @@ -75,16 +75,31 @@ static void rtmsg_lfib(int event, u32 label, struct mpls_route *rt, struct nlmsghdr *nlh, struct net *net, u32 portid, unsigned int nlm_flags); +static struct mpls_route __rcu **mpls_platform_label_rcu(struct net *net, size_t *platform_labels) +{ + struct mpls_route __rcu **platform_label; + unsigned int sequence; + + do { + sequence = read_seqcount_begin(&net->mpls.platform_label_seq); + platform_label = rcu_dereference_rtnl(net->mpls.platform_label); + *platform_labels = net->mpls.platform_labels; + } while (read_seqcount_retry(&net->mpls.platform_label_seq, sequence)); + + return platform_label; +} + static struct mpls_route *mpls_route_input_rcu(struct net *net, unsigned index) { - struct mpls_route *rt = NULL; + struct mpls_route __rcu **platform_label; + size_t platform_labels; - if (index < net->mpls.platform_labels) { - struct mpls_route __rcu **platform_label = - rcu_dereference_rtnl(net->mpls.platform_label); - rt = rcu_dereference_rtnl(platform_label[index]); - } - return rt; + platform_label = mpls_platform_label_rcu(net, &platform_labels); + + if (index >= platform_labels) + return NULL; + + return rcu_dereference_rtnl(platform_label[index]); } bool mpls_output_possible(const struct net_device *dev) @@ -2583,8 +2598,12 @@ static int resize_platform_label_table(struct net *net, size_t limit) } /* Update the global pointers */ + local_bh_disable(); + write_seqcount_begin(&net->mpls.platform_label_seq); net->mpls.platform_labels = limit; rcu_assign_pointer(net->mpls.platform_label, labels); + write_seqcount_end(&net->mpls.platform_label_seq); + local_bh_enable(); rtnl_unlock(); @@ -2665,6 +2684,8 @@ static int mpls_net_init(struct net *net) struct ctl_table *table; int i; + seqcount_init(&net->mpls.platform_label_seq); + net->mpls.platform_labels = 0; net->mpls.platform_label = NULL; net->mpls.ip_ttl_propagate = 1; diff --git a/net/mptcp/pm.c b/net/mptcp/pm.c index 0d21f4e69390..ee2ef0171969 100644 --- a/net/mptcp/pm.c +++ b/net/mptcp/pm.c @@ -155,6 +155,9 @@ void mptcp_pm_fully_established(struct mptcp_sock *msk, const struct sock *ssk) void mptcp_pm_connection_closed(struct mptcp_sock *msk) { pr_debug("msk=%p\n", msk); + + if (msk->token) + mptcp_event(MPTCP_EVENT_CLOSED, msk, NULL, GFP_KERNEL); } void mptcp_pm_subflow_established(struct mptcp_sock *msk) diff --git a/net/mptcp/pm_netlink.c b/net/mptcp/pm_netlink.c index 2be19e1877c8..8eb7fed89d6f 100644 --- a/net/mptcp/pm_netlink.c +++ b/net/mptcp/pm_netlink.c @@ -418,10 +418,10 @@ bool mptcp_pm_alloc_anno_list(struct mptcp_sock *msk, add_entry->addr = *addr; add_entry->sock = msk; - add_entry->retrans_times = 0; timer_setup(&add_entry->add_timer, mptcp_pm_add_timer, 0); reset_timer: + add_entry->retrans_times = 0; add_entry->timer_done = false; timeout = mptcp_get_add_addr_timeout(net); if (timeout) @@ -1725,6 +1725,8 @@ static int mptcp_nl_remove_id_zero_address(struct net *net, while ((msk = mptcp_token_iter_next(net, &s_slot, &s_num)) != NULL) { struct sock *sk = (struct sock *)msk; struct mptcp_addr_info msk_local; + struct mptcp_addr_info anno_addr; + bool announced; if (list_empty(&msk->conn_list) || mptcp_pm_is_userspace(msk)) goto next; @@ -1734,7 +1736,13 @@ static int mptcp_nl_remove_id_zero_address(struct net *net, goto next; lock_sock(sk); + /* Drop a possibly pending ADD_ADDR for this address. */ + anno_addr = msk_local; + anno_addr.port = 0; + announced = remove_anno_list_by_saddr(msk, &anno_addr); spin_lock_bh(&msk->pm.lock); + if (announced) + msk->pm.add_addr_signaled--; mptcp_pm_remove_addr(msk, &list); mptcp_pm_nl_rm_subflow_received(msk, &list); __mark_subflow_endp_available(msk, 0); diff --git a/net/mptcp/pm_userspace.c b/net/mptcp/pm_userspace.c index 12dd30d63c71..335ef95da403 100644 --- a/net/mptcp/pm_userspace.c +++ b/net/mptcp/pm_userspace.c @@ -40,7 +40,6 @@ static int mptcp_userspace_pm_append_new_local_addr(struct mptcp_sock *msk, bool needs_id) { DECLARE_BITMAP(id_bitmap, MPTCP_PM_MAX_ADDR_ID + 1); - struct mptcp_pm_addr_entry *match = NULL; struct sock *sk = (struct sock *)msk; struct mptcp_pm_addr_entry *e; bool addr_match = false; @@ -59,34 +58,39 @@ static int mptcp_userspace_pm_append_new_local_addr(struct mptcp_sock *msk, if (addr_match && entry->addr.id == 0 && needs_id) entry->addr.id = e->addr.id; id_match = (e->addr.id == entry->addr.id); - if (addr_match && id_match) { - match = e; + if (addr_match || id_match) break; - } else if (addr_match || id_match) { - break; - } __set_bit(e->addr.id, id_bitmap); } - if (!match && !addr_match && !id_match) { + if (!addr_match && !id_match) { + unsigned int id; + + if (!entry->addr.id && needs_id) { + id = find_next_zero_bit(id_bitmap, + MPTCP_PM_MAX_ADDR_ID + 1, 1); + if (id > MPTCP_PM_MAX_ADDR_ID) { + ret = -ENOSPC; + goto append_err; + } + } else { + id = entry->addr.id; + } + /* Memory for the entry is allocated from the * sock option buffer. */ - e = sock_kmalloc(sk, sizeof(*e), GFP_ATOMIC); + e = sock_kmemdup(sk, entry, sizeof(*entry), GFP_ATOMIC); if (!e) { ret = -ENOMEM; goto append_err; } - *e = *entry; - if (!e->addr.id && needs_id) - e->addr.id = find_next_zero_bit(id_bitmap, - MPTCP_PM_MAX_ADDR_ID + 1, - 1); + e->addr.id = id; list_add_tail_rcu(&e->list, &msk->pm.userspace_pm_local_addr_list); msk->pm.local_addr_used++; ret = e->addr.id; - } else if (match) { + } else if (addr_match && id_match) { ret = entry->addr.id; } diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 48b6e9ed04ee..3e8e5aae4dc4 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -416,9 +416,7 @@ static bool __mptcp_move_skb(struct sock *sk, struct sk_buff *skb) static void mptcp_stop_rtx_timer(struct sock *sk) { - struct inet_connection_sock *icsk = inet_csk(sk); - - sk_stop_timer(sk, &icsk->icsk_retransmit_timer); + sk_stop_timer(sk, &sk->mptcp_retransmit_timer); mptcp_sk(sk)->timer_ival = 0; } @@ -971,20 +969,20 @@ static void __mptcp_flush_join_list(struct sock *sk, struct list_head *join_list static bool mptcp_rtx_timer_pending(struct sock *sk) { - return timer_pending(&inet_csk(sk)->icsk_retransmit_timer); + return timer_pending(&sk->mptcp_retransmit_timer); } static void mptcp_reset_rtx_timer(struct sock *sk) { - struct inet_connection_sock *icsk = inet_csk(sk); + struct mptcp_sock *msk = mptcp_sk(sk); unsigned long tout; - /* prevent rescheduling on close */ - if (unlikely(inet_sk_state_load(sk) == TCP_CLOSE)) + /* Prevent rescheduling on close and in case of fallback. */ + if (!test_bit(MPTCP_RTX_ENABLED, &msk->flags)) return; - tout = mptcp_sk(sk)->timer_ival; - sk_reset_timer(sk, &icsk->icsk_retransmit_timer, jiffies + tout); + tout = msk->timer_ival; + sk_reset_timer(sk, &sk->mptcp_retransmit_timer, jiffies + tout); } bool mptcp_schedule_work(struct sock *sk) @@ -2356,9 +2354,7 @@ static int mptcp_recvmsg(struct sock *sk, struct msghdr *msg, size_t len, static void mptcp_retransmit_timer(struct timer_list *t) { - struct inet_connection_sock *icsk = from_timer(icsk, t, - icsk_retransmit_timer); - struct sock *sk = &icsk->icsk_inet.sk; + struct sock *sk = from_timer(sk, t, mptcp_retransmit_timer); struct mptcp_sock *msk = mptcp_sk(sk); bh_lock_sock(sk); @@ -2376,7 +2372,9 @@ static void mptcp_retransmit_timer(struct timer_list *t) static void mptcp_tout_timer(struct timer_list *t) { - struct sock *sk = from_timer(sk, t, sk_timer); + struct inet_connection_sock *icsk = + from_timer(icsk, t, mptcp_tout_timer); + struct sock *sk = &icsk->icsk_inet.sk; mptcp_schedule_work(sk); sock_put(sk); @@ -2819,7 +2817,7 @@ void mptcp_reset_tout_timer(struct mptcp_sock *msk, unsigned long fail_tout) */ timeout = inet_csk(sk)->icsk_mtup.probe_timestamp ? close_timeout : fail_tout; - sk_reset_timer(sk, &sk->sk_timer, timeout); + sk_reset_timer(sk, &inet_csk(sk)->mptcp_tout_timer, timeout); } static void mptcp_mp_fail_no_response(struct mptcp_sock *msk) @@ -2934,7 +2932,8 @@ static void __mptcp_init_sock(struct sock *sk) msk->first_pending = NULL; msk->rmem_fwd_alloc = 0; WRITE_ONCE(msk->rmem_released, 0); - msk->timer_ival = TCP_RTO_MIN; + msk->sk.icsk_rto_min = TCP_RTO_MIN; + msk->timer_ival = msk->sk.icsk_rto_min; msk->scaling_ratio = TCP_DEFAULT_SCALING_RATIO; WRITE_ONCE(msk->first, NULL); @@ -2948,9 +2947,8 @@ static void __mptcp_init_sock(struct sock *sk) mptcp_pm_data_init(msk); spin_lock_init(&msk->fallback_lock); - /* re-use the csk retrans timer for MPTCP-level retrans */ - timer_setup(&msk->sk.icsk_retransmit_timer, mptcp_retransmit_timer, 0); - timer_setup(&sk->sk_timer, mptcp_tout_timer, 0); + timer_setup(&sk->mptcp_retransmit_timer, mptcp_retransmit_timer, 0); + timer_setup(&msk->sk.mptcp_tout_timer, mptcp_tout_timer, 0); } static void mptcp_ca_reset(struct sock *sk) @@ -3067,6 +3065,9 @@ void mptcp_set_state(struct sock *sk, int state) * transition from TCP_SYN_RECV to TCP_CLOSE_WAIT. */ break; + case TCP_CLOSE: + clear_bit(MPTCP_RTX_ENABLED, &mptcp_sk(sk)->flags); + fallthrough; default: if (oldstate == TCP_ESTABLISHED || oldstate == TCP_CLOSE_WAIT) MPTCP_DEC_STATS(sock_net(sk), MPTCP_MIB_CURRESTAB); @@ -3151,7 +3152,7 @@ static void __mptcp_destroy_sock(struct sock *sk) might_sleep(); mptcp_stop_rtx_timer(sk); - sk_stop_timer(sk, &sk->sk_timer); + sk_stop_timer(sk, &inet_csk(sk)->mptcp_tout_timer); msk->pm.status = 0; mptcp_release_sched(msk); @@ -3308,6 +3309,7 @@ static void mptcp_copy_inaddrs(struct sock *msk, const struct sock *ssk) static int mptcp_disconnect(struct sock *sk, int flags) { + struct inet_connection_sock *icsk = inet_csk(sk); struct mptcp_sock *msk = mptcp_sk(sk); /* We are on the fastopen error path. We can't call straight into the @@ -3320,11 +3322,15 @@ static int mptcp_disconnect(struct sock *sk, int flags) mptcp_check_listen_stop(sk); mptcp_set_state(sk, TCP_CLOSE); - mptcp_stop_rtx_timer(sk); - mptcp_stop_tout_timer(sk); + /* The later subflow close can not kick again the tout timer, + * as the msk is already in closed status. + */ + msk->timer_ival = TCP_RTO_MIN; + sk_stop_timer_sync(sk, &sk->mptcp_retransmit_timer); + icsk->icsk_mtup.probe_timestamp = 0; + sk_stop_timer_sync(sk, &icsk->mptcp_tout_timer); - if (msk->token) - mptcp_event(MPTCP_EVENT_CLOSED, msk, NULL, GFP_KERNEL); + mptcp_pm_connection_closed(msk); /* msk->subflow is still intact, the following will not free the first * subflow @@ -3900,6 +3906,7 @@ static int mptcp_connect(struct sock *sk, struct sockaddr *uaddr, int addr_len) if (IS_ERR(ssk)) return PTR_ERR(ssk); + set_bit(MPTCP_RTX_ENABLED, &msk->flags); mptcp_set_state(sk, TCP_SYN_SENT); subflow = mptcp_subflow_ctx(ssk); #ifdef CONFIG_TCP_MD5SIG @@ -4043,6 +4050,7 @@ static int mptcp_listen(struct socket *sock, int backlog) goto unlock; } + set_bit(MPTCP_RTX_ENABLED, &msk->flags); mptcp_set_state(sk, TCP_LISTEN); sock_set_flag(sk, SOCK_RCU_FREE); diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index 825f77019092..64c6a6cb3d99 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -116,6 +116,7 @@ #define MPTCP_WORK_RTX 2 #define MPTCP_FALLBACK_DONE 4 #define MPTCP_WORK_CLOSE_SUBFLOW 5 +#define MPTCP_RTX_ENABLED 6 /* MPTCP socket release cb flags */ #define MPTCP_PUSH_PENDING 1 @@ -765,7 +766,7 @@ static inline void mptcp_stop_tout_timer(struct sock *sk) if (!inet_csk(sk)->icsk_mtup.probe_timestamp) return; - sk_stop_timer(sk, &sk->sk_timer); + sk_stop_timer(sk, &inet_csk(sk)->mptcp_tout_timer); inet_csk(sk)->icsk_mtup.probe_timestamp = 0; } @@ -1134,6 +1135,7 @@ static inline bool __mptcp_try_fallback(struct mptcp_sock *msk) msk->allow_subflows = false; set_bit(MPTCP_FALLBACK_DONE, &msk->flags); + clear_bit(MPTCP_RTX_ENABLED, &msk->flags); spin_unlock_bh(&msk->fallback_lock); return true; } diff --git a/net/ncsi/ncsi-manage.c b/net/ncsi/ncsi-manage.c index a5d3615d7996..16a44079cda1 100644 --- a/net/ncsi/ncsi-manage.c +++ b/net/ncsi/ncsi-manage.c @@ -1967,6 +1967,8 @@ void ncsi_unregister_dev(struct ncsi_dev *nd) list_del_rcu(&ndp->node); spin_unlock_irqrestore(&ncsi_dev_lock, flags); + cancel_work_sync(&ndp->work); + kfree(ndp); } EXPORT_SYMBOL_GPL(ncsi_unregister_dev); diff --git a/net/netfilter/core.c b/net/netfilter/core.c index 7bae43b00ebb..b00fc285b334 100644 --- a/net/netfilter/core.c +++ b/net/netfilter/core.c @@ -639,10 +639,10 @@ int nf_hook_slow(struct sk_buff *skb, struct nf_hook_state *state, if (ret == 1) continue; return ret; + case NF_STOLEN: + return NF_DROP_GETERR(verdict); default: - /* Implicit handling for NF_STOLEN, as well as any other - * non conventional verdicts. - */ + WARN_ON_ONCE(1); return 0; } } diff --git a/net/netfilter/ipset/ip_set_bitmap_gen.h b/net/netfilter/ipset/ip_set_bitmap_gen.h index 605d54d2bd7e..3dc2ebbe4800 100644 --- a/net/netfilter/ipset/ip_set_bitmap_gen.h +++ b/net/netfilter/ipset/ip_set_bitmap_gen.h @@ -157,7 +157,7 @@ mtype_add(struct ip_set *set, void *value, const struct ip_set_ext *ext, if (SET_WITH_COUNTER(set)) ip_set_init_counter(ext_counter(x, set), ext); - if (SET_WITH_COMMENT(set)) + if (SET_WITH_COMMENT(set) && !ext->target) ip_set_init_comment(set, ext_comment(x, set), ext); if (SET_WITH_SKBINFO(set)) ip_set_init_skbinfo(ext_skbinfo(x, set), ext); diff --git a/net/netfilter/ipvs/ip_vs_conn.c b/net/netfilter/ipvs/ip_vs_conn.c index f0a00e902450..252fb49c36d0 100644 --- a/net/netfilter/ipvs/ip_vs_conn.c +++ b/net/netfilter/ipvs/ip_vs_conn.c @@ -590,6 +590,9 @@ ip_vs_bind_dest(struct ip_vs_conn *cp, struct ip_vs_dest *dest) if (cp->protocol != IPPROTO_UDP) conn_flags &= ~IP_VS_CONN_F_ONE_PACKET; flags = cp->flags; + /* Only visible templates can control multiple connections */ + if (flags & IP_VS_CONN_F_TEMPLATE) + conn_flags &= ~IP_VS_CONN_F_ONE_PACKET; /* Bind with the destination and its corresponding transmitter */ if (flags & IP_VS_CONN_F_SYNC) { /* Synced conns are hashed, so they can not get this flag */ diff --git a/net/netfilter/ipvs/ip_vs_core.c b/net/netfilter/ipvs/ip_vs_core.c index a7b495bf1985..81aa3e07a761 100644 --- a/net/netfilter/ipvs/ip_vs_core.c +++ b/net/netfilter/ipvs/ip_vs_core.c @@ -745,7 +745,7 @@ static int ip_vs_route_me_harder(struct netns_ipvs *ipvs, int af, * Packet has been made sufficiently writable in caller * - inout: 1=in->out, 0=out->in */ -void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp, +bool ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp, struct ip_vs_conn *cp, int inout, unsigned int toff, bool has_ports, struct ip_vs_iphdr *ciph) { @@ -753,6 +753,11 @@ void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp, struct icmphdr *icmph = (struct icmphdr *)(skb->data + toff); struct iphdr *cih = (struct iphdr *)(icmph + 1); + /* Before now we may used ihl from skb frag, revalidate it after + * copying it into skb head to prevent out-of-bounds access + */ + if (cih->ihl * 4 != ciph->len - ciph->off) + return false; if (inout) { iph->saddr = cp->vaddr.ip; ip_send_check(iph); @@ -786,6 +791,7 @@ void ip_vs_nat_icmp(struct sk_buff *skb, struct ip_vs_protocol *pp, else IP_VS_DBG_PKT(11, AF_INET, pp, skb, ciph->off, "Forwarding altered incoming ICMP"); + return true; } #ifdef CONFIG_IP_VS_IPV6 @@ -877,7 +883,8 @@ static int handle_response_icmp(int af, struct sk_buff *skb, ip_vs_nat_icmp_v6(skb, pp, cp, 1, toff, has_ports, ciph); else #endif - ip_vs_nat_icmp(skb, pp, cp, 1, toff, has_ports, ciph); + if (!ip_vs_nat_icmp(skb, pp, cp, 1, toff, has_ports, ciph)) + goto out; if (ip_vs_route_me_harder(cp->ipvs, af, skb, hooknum)) goto out; diff --git a/net/netfilter/ipvs/ip_vs_lblc.c b/net/netfilter/ipvs/ip_vs_lblc.c index 70014c89955d..cfe0c0a3e2cf 100644 --- a/net/netfilter/ipvs/ip_vs_lblc.c +++ b/net/netfilter/ipvs/ip_vs_lblc.c @@ -205,7 +205,11 @@ ip_vs_lblc_new(struct ip_vs_lblc_table *tbl, const union nf_inet_addr *daddr, if (en->dest == dest) return en; ip_vs_lblc_del(en); + atomic_dec(&tbl->entries); } + if (atomic_read(&tbl->entries) >= tbl->max_size * 3 / 2) + return NULL; + en = kmalloc(sizeof(*en), GFP_ATOMIC); if (!en) return NULL; diff --git a/net/netfilter/ipvs/ip_vs_lblcr.c b/net/netfilter/ipvs/ip_vs_lblcr.c index 6b582be08778..9c471a14e3d0 100644 --- a/net/netfilter/ipvs/ip_vs_lblcr.c +++ b/net/netfilter/ipvs/ip_vs_lblcr.c @@ -365,6 +365,9 @@ ip_vs_lblcr_new(struct ip_vs_lblcr_table *tbl, const union nf_inet_addr *daddr, en = ip_vs_lblcr_get(af, tbl, daddr); if (!en) { + if (atomic_read(&tbl->entries) >= tbl->max_size * 3 / 2) + return NULL; + en = kmalloc(sizeof(*en), GFP_ATOMIC); if (!en) return NULL; diff --git a/net/netfilter/ipvs/ip_vs_sync.c b/net/netfilter/ipvs/ip_vs_sync.c index e88dc9077ee3..60428e15659a 100644 --- a/net/netfilter/ipvs/ip_vs_sync.c +++ b/net/netfilter/ipvs/ip_vs_sync.c @@ -950,6 +950,21 @@ static void ip_vs_proc_conn(struct netns_ipvs *ipvs, struct ip_vs_conn_param *pa ip_vs_conn_put(cp); } +/* Check for incompatible flags */ +static bool ip_vs_sync_validate_flags(u32 flags) +{ + /* We do not expect NO_CPORT, especially to allow lookups + * to hit templates + */ + if (flags & IP_VS_CONN_F_NO_CPORT) { + if (flags & IP_VS_CONN_F_TEMPLATE) + return false; + } + if (flags & IP_VS_CONN_F_ONE_PACKET) + return false; + return true; +} + /* * Process received multicast message for Version 0 */ @@ -973,8 +988,7 @@ static void ip_vs_process_message_v0(struct netns_ipvs *ipvs, const char *buffer return; } s = (struct ip_vs_sync_conn_v0 *) p; - flags = ntohs(s->flags) | IP_VS_CONN_F_SYNC; - flags &= ~IP_VS_CONN_F_HASHED; + flags = ntohs(s->flags); if (flags & IP_VS_CONN_F_SEQ_MASK) { opt = (struct ip_vs_sync_conn_options *)&s[1]; p += FULL_CONN_SIZE; @@ -987,6 +1001,13 @@ static void ip_vs_process_message_v0(struct netns_ipvs *ipvs, const char *buffer p += SIMPLE_CONN_SIZE; } + if (!ip_vs_sync_validate_flags(flags)) { + IP_VS_DBG(2, "BACKUP v0, Invalid flags 0x%X\n", flags); + continue; + } + flags &= IP_VS_CONN_F_BACKUP_MASK; + flags |= IP_VS_CONN_F_SYNC; + state = ntohs(s->state); if (!(flags & IP_VS_CONN_F_TEMPLATE)) { pp = ip_vs_proto_get(s->protocol); @@ -1142,7 +1163,13 @@ static inline int ip_vs_proc_sync_conn(struct netns_ipvs *ipvs, __u8 *p, __u8 *m } /* Get flags and Mask off unsupported */ - flags = ntohl(s->v4.flags) & IP_VS_CONN_F_BACKUP_MASK; + flags = ntohl(s->v4.flags); + if (!ip_vs_sync_validate_flags(flags)) { + IP_VS_DBG(3, "BACKUP, Invalid flags 0x%X\n", flags); + retc = 25; + goto out; + } + flags &= IP_VS_CONN_F_BACKUP_MASK; flags |= IP_VS_CONN_F_SYNC; state = ntohs(s->v4.state); diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c index 5b37e92df02c..863d48624c6a 100644 --- a/net/netfilter/ipvs/ip_vs_xmit.c +++ b/net/netfilter/ipvs/ip_vs_xmit.c @@ -306,6 +306,12 @@ static inline bool decrement_ttl(struct netns_ipvs *ipvs, return true; } +/* rt has device that is down */ +static bool rt_dev_is_down(const struct net_device *dev) +{ + return dev && !netif_running(dev); +} + /* Get route to destination or remote server */ static int __ip_vs_get_out_rt(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb, @@ -321,9 +327,11 @@ __ip_vs_get_out_rt(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb, if (dest) { dest_dst = __ip_vs_dst_check(dest); - if (likely(dest_dst)) + if (likely(dest_dst)) { rt = (struct rtable *) dest_dst->dst_cache; - else { + if (ret_saddr) + *ret_saddr = dest_dst->dst_saddr.ip; + } else { dest_dst = ip_vs_dest_dst_alloc(); spin_lock_bh(&dest->dst_lock); if (!dest_dst) { @@ -339,14 +347,22 @@ __ip_vs_get_out_rt(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb, ip_vs_dest_dst_free(dest_dst); goto err_unreach; } - __ip_vs_dst_set(dest, dest_dst, &rt->dst, 0); + /* It is forbidden to attach dest->dest_dst if + * device is going down. + */ + if (!rt_dev_is_down(dst_dev_rcu(&rt->dst))) + __ip_vs_dst_set(dest, dest_dst, &rt->dst, 0); + else + noref = 0; spin_unlock_bh(&dest->dst_lock); IP_VS_DBG(10, "new dst %pI4, src %pI4, refcnt=%d\n", &dest->addr.ip, &dest_dst->dst_saddr.ip, rcuref_read(&rt->dst.__rcuref)); + if (ret_saddr) + *ret_saddr = dest_dst->dst_saddr.ip; + if (!noref) + ip_vs_dest_dst_free(dest_dst); } - if (ret_saddr) - *ret_saddr = dest_dst->dst_saddr.ip; } else { noref = 0; @@ -483,9 +499,11 @@ __ip_vs_get_out_rt_v6(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb, if (dest) { dest_dst = __ip_vs_dst_check(dest); - if (likely(dest_dst)) + if (likely(dest_dst)) { rt = dst_rt6_info(dest_dst->dst_cache); - else { + if (ret_saddr) + *ret_saddr = dest_dst->dst_saddr.in6; + } else { u32 cookie; dest_dst = ip_vs_dest_dst_alloc(); @@ -506,14 +524,22 @@ __ip_vs_get_out_rt_v6(struct netns_ipvs *ipvs, int skb_af, struct sk_buff *skb, } rt = dst_rt6_info(dst); cookie = rt6_get_cookie(rt); - __ip_vs_dst_set(dest, dest_dst, &rt->dst, cookie); + /* It is forbidden to attach dest->dest_dst if + * device is going down. + */ + if (!rt_dev_is_down(dst_dev_rcu(&rt->dst))) + __ip_vs_dst_set(dest, dest_dst, &rt->dst, cookie); + else + noref = 0; spin_unlock_bh(&dest->dst_lock); IP_VS_DBG(10, "new dst %pI6, src %pI6, refcnt=%d\n", &dest->addr.in6, &dest_dst->dst_saddr.in6, rcuref_read(&rt->dst.__rcuref)); + if (ret_saddr) + *ret_saddr = dest_dst->dst_saddr.in6; + if (!noref) + ip_vs_dest_dst_free(dest_dst); } - if (ret_saddr) - *ret_saddr = dest_dst->dst_saddr.in6; } else { noref = 0; dst = __ip_vs_route_output_v6(net, daddr, ret_saddr, do_xfrm, @@ -1551,7 +1577,8 @@ ip_vs_icmp_xmit(struct sk_buff *skb, struct ip_vs_conn *cp, if (skb_cow(skb, rt->dst.dev->hard_header_len)) goto tx_error; - ip_vs_nat_icmp(skb, pp, cp, 0, toff, has_ports, ciph); + if (!ip_vs_nat_icmp(skb, pp, cp, 0, toff, has_ports, ciph)) + goto tx_error; /* Another hack: avoid icmp_send in ip_fragment */ skb->ignore_df = 1; diff --git a/net/netfilter/nf_conntrack_amanda.c b/net/netfilter/nf_conntrack_amanda.c index d011d2eb0848..7be4c35e4795 100644 --- a/net/netfilter/nf_conntrack_amanda.c +++ b/net/netfilter/nf_conntrack_amanda.c @@ -106,7 +106,7 @@ static int amanda_help(struct sk_buff *skb, /* increase the UDP timeout of the master connection as replies from * Amanda clients to the server can be quite delayed */ - nf_ct_refresh(ct, skb, master_timeout * HZ); + nf_ct_refresh(ct, master_timeout * HZ); /* No data? */ dataoff = protoff + sizeof(struct udphdr); diff --git a/net/netfilter/nf_conntrack_bpf.c b/net/netfilter/nf_conntrack_bpf.c index b21799d468d2..89609f422bd6 100644 --- a/net/netfilter/nf_conntrack_bpf.c +++ b/net/netfilter/nf_conntrack_bpf.c @@ -58,6 +58,15 @@ enum { NF_BPF_CT_OPTS_SZ = 12, }; +static void *bpf_ct_opts_result(struct bpf_ct_opts *opts, u32 opts__sz, void *ret) +{ + if (!IS_ERR(ret)) + return ret; + if (opts__sz >= offsetofend(struct bpf_ct_opts, error)) + opts->error = PTR_ERR(ret); + return NULL; +} + static int bpf_nf_ct_tuple_parse(struct bpf_sock_tuple *bpf_tuple, u32 tuple_len, u8 protonum, u8 dir, struct nf_conntrack_tuple *tuple) @@ -105,27 +114,32 @@ __bpf_nf_ct_alloc_entry(struct net *net, struct bpf_sock_tuple *bpf_tuple, { struct nf_conntrack_tuple otuple, rtuple; struct nf_conn *ct; + s32 netns_id; + u8 l4proto; int err; if (!opts || !bpf_tuple || opts->reserved[0] || opts->reserved[1] || opts_len != NF_BPF_CT_OPTS_SZ) return ERR_PTR(-EINVAL); - if (unlikely(opts->netns_id < BPF_F_CURRENT_NETNS)) + netns_id = READ_ONCE(opts->netns_id); + l4proto = READ_ONCE(opts->l4proto); + + if (unlikely(netns_id < BPF_F_CURRENT_NETNS)) return ERR_PTR(-EINVAL); - err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto, + err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto, IP_CT_DIR_ORIGINAL, &otuple); if (err < 0) return ERR_PTR(err); - err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto, + err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto, IP_CT_DIR_REPLY, &rtuple); if (err < 0) return ERR_PTR(err); - if (opts->netns_id >= 0) { - net = get_net_ns_by_id(net, opts->netns_id); + if (netns_id >= 0) { + net = get_net_ns_by_id(net, netns_id); if (unlikely(!net)) return ERR_PTR(-ENONET); } @@ -139,7 +153,7 @@ __bpf_nf_ct_alloc_entry(struct net *net, struct bpf_sock_tuple *bpf_tuple, __nf_ct_set_timeout(ct, timeout * HZ); out: - if (opts->netns_id >= 0) + if (netns_id >= 0) put_net(net); return ct; @@ -153,29 +167,35 @@ static struct nf_conn *__bpf_nf_ct_lookup(struct net *net, struct nf_conntrack_tuple_hash *hash; struct nf_conntrack_tuple tuple; struct nf_conn *ct; + s32 netns_id; + u8 l4proto; int err; if (!opts || !bpf_tuple || opts->reserved[0] || opts->reserved[1] || opts_len != NF_BPF_CT_OPTS_SZ) return ERR_PTR(-EINVAL); - if (unlikely(opts->l4proto != IPPROTO_TCP && opts->l4proto != IPPROTO_UDP)) + + netns_id = READ_ONCE(opts->netns_id); + l4proto = READ_ONCE(opts->l4proto); + + if (unlikely(l4proto != IPPROTO_TCP && l4proto != IPPROTO_UDP)) return ERR_PTR(-EPROTO); - if (unlikely(opts->netns_id < BPF_F_CURRENT_NETNS)) + if (unlikely(netns_id < BPF_F_CURRENT_NETNS)) return ERR_PTR(-EINVAL); - err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, opts->l4proto, + err = bpf_nf_ct_tuple_parse(bpf_tuple, tuple_len, l4proto, IP_CT_DIR_ORIGINAL, &tuple); if (err < 0) return ERR_PTR(err); - if (opts->netns_id >= 0) { - net = get_net_ns_by_id(net, opts->netns_id); + if (netns_id >= 0) { + net = get_net_ns_by_id(net, netns_id); if (unlikely(!net)) return ERR_PTR(-ENONET); } hash = nf_conntrack_find_get(net, &nf_ct_zone_dflt, &tuple); - if (opts->netns_id >= 0) + if (netns_id >= 0) put_net(net); if (!hash) return ERR_PTR(-ENOENT); @@ -256,15 +276,12 @@ bpf_xdp_ct_alloc(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple, struct xdp_buff *ctx = (struct xdp_buff *)xdp_ctx; struct nf_conn *nfct; - nfct = __bpf_nf_ct_alloc_entry(dev_net(ctx->rxq->dev), bpf_tuple, tuple__sz, - opts, opts__sz, 10); - if (IS_ERR(nfct)) { - if (opts) - opts->error = PTR_ERR(nfct); + if (!opts) return NULL; - } - return (struct nf_conn___init *)nfct; + nfct = __bpf_nf_ct_alloc_entry(dev_net(ctx->rxq->dev), bpf_tuple, tuple__sz, + opts, opts__sz, 10); + return (struct nf_conn___init *)bpf_ct_opts_result(opts, opts__sz, nfct); } /* bpf_xdp_ct_lookup - Lookup CT entry for the given tuple, and acquire a @@ -291,14 +308,12 @@ bpf_xdp_ct_lookup(struct xdp_md *xdp_ctx, struct bpf_sock_tuple *bpf_tuple, struct net *caller_net; struct nf_conn *nfct; + if (!opts) + return NULL; + caller_net = dev_net(ctx->rxq->dev); nfct = __bpf_nf_ct_lookup(caller_net, bpf_tuple, tuple__sz, opts, opts__sz); - if (IS_ERR(nfct)) { - if (opts) - opts->error = PTR_ERR(nfct); - return NULL; - } - return nfct; + return bpf_ct_opts_result(opts, opts__sz, nfct); } /* bpf_skb_ct_alloc - Allocate a new CT entry @@ -324,15 +339,12 @@ bpf_skb_ct_alloc(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple, struct nf_conn *nfct; struct net *net; - net = skb->dev ? dev_net(skb->dev) : sock_net(skb->sk); - nfct = __bpf_nf_ct_alloc_entry(net, bpf_tuple, tuple__sz, opts, opts__sz, 10); - if (IS_ERR(nfct)) { - if (opts) - opts->error = PTR_ERR(nfct); + if (!opts) return NULL; - } - return (struct nf_conn___init *)nfct; + net = skb->dev ? dev_net(skb->dev) : sock_net(skb->sk); + nfct = __bpf_nf_ct_alloc_entry(net, bpf_tuple, tuple__sz, opts, opts__sz, 10); + return (struct nf_conn___init *)bpf_ct_opts_result(opts, opts__sz, nfct); } /* bpf_skb_ct_lookup - Lookup CT entry for the given tuple, and acquire a @@ -359,14 +371,12 @@ bpf_skb_ct_lookup(struct __sk_buff *skb_ctx, struct bpf_sock_tuple *bpf_tuple, struct net *caller_net; struct nf_conn *nfct; + if (!opts) + return NULL; + caller_net = skb->dev ? dev_net(skb->dev) : sock_net(skb->sk); nfct = __bpf_nf_ct_lookup(caller_net, bpf_tuple, tuple__sz, opts, opts__sz); - if (IS_ERR(nfct)) { - if (opts) - opts->error = PTR_ERR(nfct); - return NULL; - } - return nfct; + return bpf_ct_opts_result(opts, opts__sz, nfct); } /* bpf_ct_insert_entry - Add the provided entry into a CT map diff --git a/net/netfilter/nf_conntrack_broadcast.c b/net/netfilter/nf_conntrack_broadcast.c index ef8a7ca8c116..f5e24f1902b3 100644 --- a/net/netfilter/nf_conntrack_broadcast.c +++ b/net/netfilter/nf_conntrack_broadcast.c @@ -80,7 +80,7 @@ int nf_conntrack_broadcast_help(struct sk_buff *skb, nf_ct_expect_related(exp, 0); nf_ct_expect_put(exp); - nf_ct_refresh(ct, skb, timeout * HZ); + nf_ct_refresh(ct, timeout * HZ); out: return NF_ACCEPT; } diff --git a/net/netfilter/nf_conntrack_core.c b/net/netfilter/nf_conntrack_core.c index bed9c370fd70..8969d8abe4cb 100644 --- a/net/netfilter/nf_conntrack_core.c +++ b/net/netfilter/nf_conntrack_core.c @@ -1423,6 +1423,31 @@ static bool gc_worker_can_early_drop(const struct nf_conn *ct) return false; } +static void nf_ct_help_gc(struct nf_conn *ct) +{ + struct nf_conn_help *help; + + if (!refcount_inc_not_zero(&ct->ct_general.use)) + return; + + /* load ->status after refcount increase */ + smp_acquire__after_ctrl_dep(); + + if (!nf_ct_is_confirmed(ct) || nf_ct_is_dying(ct)) { + nf_ct_put(ct); + return; + } + + /* re-check helper due to SLAB_TYPESAFE_BY_RCU */ + if (test_bit(IPS_HELPER_BIT, &ct->status)) { + help = nfct_help(ct); + if (help) + nf_ct_expectation_gc(help); + } + + nf_ct_put(ct); +} + static void gc_worker(struct work_struct *work) { unsigned int i, hashsz, nf_conntrack_max95 = 0; @@ -1471,12 +1496,6 @@ static void gc_worker(struct work_struct *work) tmp = nf_ct_tuplehash_to_ctrack(h); - if (test_bit(IPS_OFFLOAD_BIT, &tmp->status)) { - nf_ct_offload_timeout(tmp); - if (!nf_conntrack_max95) - continue; - } - if (expired_count > GC_SCAN_EXPIRED_MAX) { rcu_read_unlock(); @@ -1501,7 +1520,13 @@ static void gc_worker(struct work_struct *work) expires = (expires - (long)next_run) / ++count; next_run += expires; - if (nf_conntrack_max95 == 0 || gc_worker_skip_ct(tmp)) + if (gc_worker_skip_ct(tmp)) + continue; + + if (test_bit(IPS_HELPER_BIT, &tmp->status)) + nf_ct_help_gc(tmp); + + if (nf_conntrack_max95 == 0) continue; net = nf_ct_net(tmp); @@ -1648,16 +1673,18 @@ void nf_conntrack_free(struct nf_conn *ct) */ WARN_ON(refcount_read(&ct->ct_general.use) != 0); + rcu_read_lock(); if (ct->status & IPS_SRC_NAT_DONE) { const struct nf_nat_hook *nat_hook; - rcu_read_lock(); nat_hook = rcu_dereference(nf_nat_hook); if (nat_hook) nat_hook->remove_nat_bysrc(ct); - rcu_read_unlock(); } + nf_ct_timeout_put(ct); + rcu_read_unlock(); + kfree(ct->ext); kmem_cache_free(nf_conntrack_cachep, ct); cnet = nf_ct_pernet(net); @@ -2037,9 +2064,8 @@ EXPORT_SYMBOL_GPL(nf_conntrack_alter_reply); /* Refresh conntrack for this many jiffies and do accounting if do_acct is 1 */ void __nf_ct_refresh_acct(struct nf_conn *ct, enum ip_conntrack_info ctinfo, - const struct sk_buff *skb, u32 extra_jiffies, - bool do_acct) + unsigned int bytes) { /* Only update if this is not a fixed timeout */ if (test_bit(IPS_FIXED_TIMEOUT_BIT, &ct->status)) @@ -2052,8 +2078,8 @@ void __nf_ct_refresh_acct(struct nf_conn *ct, if (READ_ONCE(ct->timeout) != extra_jiffies) WRITE_ONCE(ct->timeout, extra_jiffies); acct: - if (do_acct) - nf_ct_acct_update(ct, CTINFO2DIR(ctinfo), skb->len); + if (bytes) + nf_ct_acct_update(ct, CTINFO2DIR(ctinfo), bytes); } EXPORT_SYMBOL_GPL(__nf_ct_refresh_acct); diff --git a/net/netfilter/nf_conntrack_expect.c b/net/netfilter/nf_conntrack_expect.c index 70d06ae0a379..f908e64eb814 100644 --- a/net/netfilter/nf_conntrack_expect.c +++ b/net/netfilter/nf_conntrack_expect.c @@ -43,6 +43,24 @@ unsigned int nf_ct_expect_max __read_mostly; static struct kmem_cache *nf_ct_expect_cachep __read_mostly; static siphash_aligned_key_t nf_ct_expect_hashrnd; +void nf_ct_expectation_gc(struct nf_conn_help *master_help) +{ + struct nf_conntrack_expect *exp; + struct hlist_node *next; + + if (hlist_empty(&master_help->expectations)) + return; + + spin_lock_bh(&nf_conntrack_expect_lock); + hlist_for_each_entry_safe(exp, next, &master_help->expectations, lnode) { + if (!nf_ct_exp_is_expired(exp)) + continue; + + nf_ct_unlink_expect(exp); + } + spin_unlock_bh(&nf_conntrack_expect_lock); +} + /* nf_conntrack_expect helper functions */ void nf_ct_unlink_expect_report(struct nf_conntrack_expect *exp, u32 portid, int report) @@ -53,7 +71,6 @@ void nf_ct_unlink_expect_report(struct nf_conntrack_expect *exp, lockdep_nfct_expect_lock_held(); WARN_ON(!master_help); - WARN_ON(timer_pending(&exp->timeout)); hlist_del_rcu(&exp->hnode); @@ -70,16 +87,6 @@ void nf_ct_unlink_expect_report(struct nf_conntrack_expect *exp, } EXPORT_SYMBOL_GPL(nf_ct_unlink_expect_report); -static void nf_ct_expectation_timed_out(struct timer_list *t) -{ - struct nf_conntrack_expect *exp = from_timer(exp, t, timeout); - - spin_lock_bh(&nf_conntrack_expect_lock); - nf_ct_unlink_expect(exp); - spin_unlock_bh(&nf_conntrack_expect_lock); - nf_ct_expect_put(exp); -} - static unsigned int nf_ct_expect_dst_hash(const struct net *n, const struct nf_conntrack_tuple *tuple) { struct { @@ -117,19 +124,6 @@ nf_ct_exp_equal(const struct nf_conntrack_tuple *tuple, nf_ct_exp_zone_equal_any(i, zone); } -bool nf_ct_remove_expect(struct nf_conntrack_expect *exp) -{ - lockdep_nfct_expect_lock_held(); - - if (del_timer(&exp->timeout)) { - nf_ct_unlink_expect(exp); - nf_ct_expect_put(exp); - return true; - } - return false; -} -EXPORT_SYMBOL_GPL(nf_ct_remove_expect); - struct nf_conntrack_expect * __nf_ct_expect_find(struct net *net, const struct nf_conntrack_zone *zone, @@ -144,6 +138,8 @@ __nf_ct_expect_find(struct net *net, h = nf_ct_expect_dst_hash(net, tuple); hlist_for_each_entry_rcu(i, &nf_ct_expect_hash[h], hnode) { + if (nf_ct_exp_is_expired(i)) + continue; if (nf_ct_exp_equal(tuple, i, zone, net)) return i; } @@ -178,6 +174,7 @@ nf_ct_find_expectation(struct net *net, { struct nf_conntrack_net *cnet = nf_ct_pernet(net); struct nf_conntrack_expect *i, *exp = NULL; + struct hlist_node *next; unsigned int h; lockdep_nfct_expect_lock_held(); @@ -186,7 +183,11 @@ nf_ct_find_expectation(struct net *net, return NULL; h = nf_ct_expect_dst_hash(net, tuple); - hlist_for_each_entry(i, &nf_ct_expect_hash[h], hnode) { + hlist_for_each_entry_safe(i, next, &nf_ct_expect_hash[h], hnode) { + if (nf_ct_exp_is_expired(i)) { + nf_ct_unlink_expect(i); + continue; + } if (!(i->flags & NF_CT_EXPECT_INACTIVE) && nf_ct_exp_equal(tuple, i, zone, net)) { exp = i; @@ -196,13 +197,16 @@ nf_ct_find_expectation(struct net *net, if (!exp) return NULL; + if (!refcount_inc_not_zero(&exp->use)) + return NULL; + /* If master is not in hash table yet (ie. packet hasn't left this machine yet), how can other end know about expected? Hence these are not the droids you are looking for (if master ct never got confirmed, we'd hold a reference to it and weird things would happen to future packets). */ if (!nf_ct_is_confirmed(exp->master)) - return NULL; + goto err_release_exp; /* Avoid race with other CPUs, that for exp->master ct, is * about to invoke ->destroy(), or nf_ct_delete() via timeout @@ -214,18 +218,17 @@ nf_ct_find_expectation(struct net *net, */ if (unlikely(nf_ct_is_dying(exp->master) || !refcount_inc_not_zero(&exp->master->ct_general.use))) - return NULL; + goto err_release_exp; - if (exp->flags & NF_CT_EXPECT_PERMANENT || !unlink) { - refcount_inc(&exp->use); - return exp; - } else if (del_timer(&exp->timeout)) { - nf_ct_unlink_expect(exp); + if (exp->flags & NF_CT_EXPECT_PERMANENT || !unlink) return exp; - } - /* Undo exp->master refcnt increase, if del_timer() failed */ - nf_ct_put(exp->master); + nf_ct_unlink_expect(exp); + + return exp; + +err_release_exp: + nf_ct_expect_put(exp); return NULL; } @@ -241,9 +244,8 @@ void nf_ct_remove_expectations(struct nf_conn *ct) return; spin_lock_bh(&nf_conntrack_expect_lock); - hlist_for_each_entry_safe(exp, next, &help->expectations, lnode) { - nf_ct_remove_expect(exp); - } + hlist_for_each_entry_safe(exp, next, &help->expectations, lnode) + nf_ct_unlink_expect(exp); spin_unlock_bh(&nf_conntrack_expect_lock); } EXPORT_SYMBOL_GPL(nf_ct_remove_expectations); @@ -292,7 +294,7 @@ static bool master_matches(const struct nf_conntrack_expect *a, void nf_ct_unexpect_related(struct nf_conntrack_expect *exp) { spin_lock_bh(&nf_conntrack_expect_lock); - nf_ct_remove_expect(exp); + WRITE_ONCE(exp->flags, exp->flags | NF_CT_EXPECT_DEAD); spin_unlock_bh(&nf_conntrack_expect_lock); } EXPORT_SYMBOL_GPL(nf_ct_unexpect_related); @@ -308,6 +310,7 @@ struct nf_conntrack_expect *nf_ct_expect_alloc(struct nf_conn *me) if (!new) return NULL; + new->timeout = nfct_time_stamp; new->master = me; refcount_set(&new->use, 1); return new; @@ -414,17 +417,12 @@ static void nf_ct_expect_insert(struct nf_conntrack_expect *exp) struct net *net = nf_ct_exp_net(exp); unsigned int h = nf_ct_expect_dst_hash(net, &exp->tuple); - /* two references : one for hash insert, one for the timer */ - refcount_add(2, &exp->use); + refcount_inc(&exp->use); - timer_setup(&exp->timeout, nf_ct_expectation_timed_out, 0); helper = rcu_dereference_protected(master_help->helper, lockdep_is_held(&nf_conntrack_expect_lock)); - if (helper) { - exp->timeout.expires = jiffies + - helper->expect_policy[exp->class].timeout * HZ; - } - add_timer(&exp->timeout); + if (helper) + exp->timeout += helper->expect_policy[exp->class].timeout * HZ; hlist_add_head_rcu(&exp->lnode, &master_help->expectations); master_help->expecting[exp->class]++; @@ -436,20 +434,27 @@ static void nf_ct_expect_insert(struct nf_conntrack_expect *exp) NF_CT_STAT_INC(net, expect_create); } -/* Race with expectations being used means we could have none to find; OK. */ static void evict_oldest_expect(struct nf_conn *master, - struct nf_conntrack_expect *new) + struct nf_conntrack_expect *new, + const struct nf_conntrack_expect_policy *p) { struct nf_conn_help *master_help = nfct_help(master); struct nf_conntrack_expect *exp, *last = NULL; + struct hlist_node *next; - hlist_for_each_entry(exp, &master_help->expectations, lnode) { + hlist_for_each_entry_safe(exp, next, &master_help->expectations, lnode) { + if (nf_ct_exp_is_expired(exp)) { + nf_ct_unlink_expect(exp); + continue; + } if (exp->class == new->class) last = exp; } - if (last) - nf_ct_remove_expect(last); + /* Still worth to evict oldest expectation after garbage collection? */ + if (last && + master_help->expecting[last->class] >= p->max_expected) + nf_ct_unlink_expect(last); } static inline int __nf_ct_expect_check(struct nf_conntrack_expect *expect, @@ -474,14 +479,18 @@ static inline int __nf_ct_expect_check(struct nf_conntrack_expect *expect, } h = nf_ct_expect_dst_hash(net, &expect->tuple); hlist_for_each_entry_safe(i, next, &nf_ct_expect_hash[h], hnode) { + if (nf_ct_exp_is_expired(i)) { + nf_ct_unlink_expect(i); + continue; + } if (master_matches(i, expect, flags) && expect_matches(i, expect)) { if (i->class != expect->class || i->master != expect->master) return -EALREADY; - if (nf_ct_remove_expect(i)) - break; + nf_ct_unlink_expect(i); + break; } else if (expect_clash(i, expect)) { ret = -EBUSY; goto out; @@ -493,14 +502,8 @@ static inline int __nf_ct_expect_check(struct nf_conntrack_expect *expect, if (helper) { p = &helper->expect_policy[expect->class]; if (p->max_expected && - master_help->expecting[expect->class] >= p->max_expected) { - evict_oldest_expect(master, expect); - if (master_help->expecting[expect->class] - >= p->max_expected) { - ret = -EMFILE; - goto out; - } - } + master_help->expecting[expect->class] >= p->max_expected) + evict_oldest_expect(master, expect, p); } cnet = nf_ct_pernet(net); @@ -547,10 +550,8 @@ void nf_ct_expect_iterate_destroy(bool (*iter)(struct nf_conntrack_expect *e, vo hlist_for_each_entry_safe(exp, next, &nf_ct_expect_hash[i], hnode) { - if (iter(exp, data) && del_timer(&exp->timeout)) { + if (iter(exp, data)) nf_ct_unlink_expect(exp); - nf_ct_expect_put(exp); - } } } @@ -577,10 +578,8 @@ void nf_ct_expect_iterate_net(struct net *net, if (!net_eq(nf_ct_exp_net(exp), net)) continue; - if (iter(exp, data) && del_timer(&exp->timeout)) { + if (iter(exp, data)) nf_ct_unlink_expect_report(exp, portid, report); - nf_ct_expect_put(exp); - } } } @@ -657,17 +656,17 @@ static int exp_seq_show(struct seq_file *s, void *v) struct net *net = seq_file_net(s); struct hlist_node *n = v; char *delim = ""; + __s32 timeout; expect = hlist_entry(n, struct nf_conntrack_expect, hnode); if (!net_eq(nf_ct_exp_net(expect), net)) return 0; + if (nf_ct_exp_is_expired(expect)) + return 0; - if (expect->timeout.function) - seq_printf(s, "%ld ", timer_pending(&expect->timeout) - ? (long)(expect->timeout.expires - jiffies)/HZ : 0); - else - seq_puts(s, "- "); + timeout = (__s32)(READ_ONCE(expect->timeout) - nfct_time_stamp) / HZ; + seq_printf(s, "%d ", timeout > 0 ? timeout : 0); seq_printf(s, "l3proto = %u proto=%u ", expect->tuple.src.l3num, expect->tuple.dst.protonum); diff --git a/net/netfilter/nf_conntrack_h323_main.c b/net/netfilter/nf_conntrack_h323_main.c index c42547284f35..279201f8a18f 100644 --- a/net/netfilter/nf_conntrack_h323_main.c +++ b/net/netfilter/nf_conntrack_h323_main.c @@ -1385,7 +1385,7 @@ static int process_rcf(struct sk_buff *skb, struct nf_conn *ct, if (info->timeout > 0) { pr_debug("nf_ct_ras: set RAS connection timeout to " "%u seconds\n", info->timeout); - nf_ct_refresh(ct, skb, info->timeout * HZ); + nf_ct_refresh(ct, info->timeout * HZ); /* Set expect timeout */ spin_lock_bh(&nf_conntrack_expect_lock); @@ -1396,8 +1396,8 @@ static int process_rcf(struct sk_buff *skb, struct nf_conn *ct, "timeout to %u seconds for", info->timeout); nf_ct_dump_tuple(&exp->tuple); - mod_timer_pending(&exp->timeout, - jiffies + info->timeout * HZ); + WRITE_ONCE(exp->timeout, + nfct_time_stamp + (info->timeout * HZ)); } spin_unlock_bh(&nf_conntrack_expect_lock); } @@ -1433,7 +1433,7 @@ static int process_urq(struct sk_buff *skb, struct nf_conn *ct, info->sig_port[!dir] = 0; /* Give it 30 seconds for UCF or URJ */ - nf_ct_refresh(ct, skb, 30 * HZ); + nf_ct_refresh(ct, 30 * HZ); return 0; } diff --git a/net/netfilter/nf_conntrack_helper.c b/net/netfilter/nf_conntrack_helper.c index 7174369e5aa6..2caf728408a7 100644 --- a/net/netfilter/nf_conntrack_helper.c +++ b/net/netfilter/nf_conntrack_helper.c @@ -180,10 +180,10 @@ nf_ct_helper_ext_add(struct nf_conn *ct, gfp_t gfp) struct nf_conn_help *help; help = nf_ct_ext_add(ct, NF_CT_EXT_HELPER, gfp); - if (help) + if (help) { + __set_bit(IPS_HELPER_BIT, &ct->status); INIT_HLIST_HEAD(&help->expectations); - else - pr_debug("failed to add helper extension area"); + } return help; } EXPORT_SYMBOL_GPL(nf_ct_helper_ext_add); @@ -207,10 +207,8 @@ int __nf_ct_try_assign_helper(struct nf_conn *ct, struct nf_conn *tmpl, return 0; help = nfct_help(tmpl); - if (help != NULL) { + if (help) helper = rcu_dereference(help->helper); - set_bit(IPS_HELPER_BIT, &ct->status); - } help = nfct_help(ct); diff --git a/net/netfilter/nf_conntrack_netlink.c b/net/netfilter/nf_conntrack_netlink.c index 537a44967bc8..f4627a8ab6e7 100644 --- a/net/netfilter/nf_conntrack_netlink.c +++ b/net/netfilter/nf_conntrack_netlink.c @@ -2995,8 +2995,8 @@ static int ctnetlink_exp_dump_expect(struct sk_buff *skb, const struct nf_conntrack_expect *exp) { + __s32 timeout = (__s32)(READ_ONCE(exp->timeout) - nfct_time_stamp) / HZ; struct nf_conn *master = exp->master; - long timeout = ((long)exp->timeout.expires - (long)jiffies) / HZ; struct nf_conntrack_helper *helper; #if IS_ENABLED(CONFIG_NF_NAT) struct nlattr *nest_parms; @@ -3159,6 +3159,9 @@ ctnetlink_exp_dump_table(struct sk_buff *skb, struct netlink_callback *cb) restart: hlist_for_each_entry_rcu(exp, &nf_ct_expect_hash[cb->args[0]], hnode) { + if (nf_ct_exp_is_expired(exp)) + continue; + if (l3proto && exp->tuple.src.l3num != l3proto) continue; @@ -3438,11 +3441,8 @@ static int ctnetlink_del_expect(struct sk_buff *skb, } /* after list removal, usage count == 1 */ - if (del_timer(&exp->timeout)) { - nf_ct_unlink_expect_report(exp, NETLINK_CB(skb).portid, - nlmsg_report(info->nlh)); - nf_ct_expect_put(exp); - } + nf_ct_unlink_expect_report(exp, NETLINK_CB(skb).portid, + nlmsg_report(info->nlh)); spin_unlock_bh(&nf_conntrack_expect_lock); /* have to put what we 'get' above. * after this line usage count == 0 */ @@ -3466,14 +3466,10 @@ static int ctnetlink_change_expect(struct nf_conntrack_expect *x, const struct nlattr * const cda[]) { - if (cda[CTA_EXPECT_TIMEOUT]) { - if (!del_timer(&x->timeout)) - return -ETIME; + if (cda[CTA_EXPECT_TIMEOUT]) + WRITE_ONCE(x->timeout, nfct_time_stamp + + ntohl(nla_get_be32(cda[CTA_EXPECT_TIMEOUT])) * HZ); - x->timeout.expires = jiffies + - ntohl(nla_get_be32(cda[CTA_EXPECT_TIMEOUT])) * HZ; - add_timer(&x->timeout); - } return 0; } diff --git a/net/netfilter/nf_conntrack_sip.c b/net/netfilter/nf_conntrack_sip.c index 2b5f807436e2..449a6e53feae 100644 --- a/net/netfilter/nf_conntrack_sip.c +++ b/net/netfilter/nf_conntrack_sip.c @@ -894,11 +894,10 @@ static int refresh_signalling_expectation(struct nf_conn *ct, exp->tuple.dst.protonum != proto || exp->tuple.dst.u.udp.port != port) continue; - if (mod_timer_pending(&exp->timeout, jiffies + expires * HZ)) { - exp->flags &= ~NF_CT_EXPECT_INACTIVE; - found = 1; - break; - } + WRITE_ONCE(exp->timeout, nfct_time_stamp + (expires * HZ)); + WRITE_ONCE(exp->flags, exp->flags & ~NF_CT_EXPECT_INACTIVE); + found = 1; + break; } spin_unlock_bh(&nf_conntrack_expect_lock); return found; @@ -914,8 +913,7 @@ static void flush_expectations(struct nf_conn *ct, bool media) hlist_for_each_entry_safe(exp, next, &help->expectations, lnode) { if ((exp->class != SIP_EXPECT_SIGNALLING) ^ media) continue; - if (!nf_ct_remove_expect(exp)) - continue; + nf_ct_unlink_expect(exp); if (!media) break; } @@ -1385,7 +1383,6 @@ static int process_register_request(struct sk_buff *skb, unsigned int protoff, nf_ct_expect_init(exp, SIP_EXPECT_SIGNALLING, nf_ct_l3num(ct), saddr, &daddr, proto, NULL, &port); - exp->timeout.expires = sip_timeout * HZ; rcu_assign_pointer(exp->assign_helper, helper); exp->flags = NF_CT_EXPECT_PERMANENT | NF_CT_EXPECT_INACTIVE; @@ -1646,7 +1643,7 @@ static int sip_help_tcp(struct sk_buff *skb, unsigned int protoff, if (dataoff >= skb->len) return NF_ACCEPT; - nf_ct_refresh(ct, skb, sip_timeout * HZ); + nf_ct_refresh(ct, sip_timeout * HZ); if (unlikely(skb_linearize(skb))) return NF_DROP; @@ -1720,7 +1717,7 @@ static int sip_help_udp(struct sk_buff *skb, unsigned int protoff, if (dataoff >= skb->len) return NF_ACCEPT; - nf_ct_refresh(ct, skb, sip_timeout * HZ); + nf_ct_refresh(ct, sip_timeout * HZ); if (unlikely(skb_linearize(skb))) return NF_DROP; diff --git a/net/netfilter/nf_conntrack_timeout.c b/net/netfilter/nf_conntrack_timeout.c index 0cc584d3dbb1..c81becde2afa 100644 --- a/net/netfilter/nf_conntrack_timeout.c +++ b/net/netfilter/nf_conntrack_timeout.c @@ -25,17 +25,32 @@ const struct nf_ct_timeout_hooks __rcu *nf_ct_timeout_hook __read_mostly; EXPORT_SYMBOL_GPL(nf_ct_timeout_hook); +/* nf_ct_iterate_cleanup() holds the conntrack lock. */ static int untimeout(struct nf_conn *ct, void *timeout) { struct nf_conn_timeout *timeout_ext = nf_ct_timeout_find(ct); if (timeout_ext) { - const struct nf_ct_timeout *t; + struct nf_ct_timeout *t; - t = rcu_access_pointer(timeout_ext->timeout); + rcu_read_lock(); + t = rcu_dereference(timeout_ext->timeout); + if (!t) { + rcu_read_unlock(); + return 0; + } - if (!timeout || t == timeout) + if (!timeout || t == timeout) { RCU_INIT_POINTER(timeout_ext->timeout, NULL); + + /* No race with nf_conntrack_free() which is called + * only after the conntrack has been removed from + * the hashes. + */ + if (refcount_dec_and_test(&t->refcnt)) + kfree_rcu(t, rcu); + } + rcu_read_unlock(); } /* We are not intended to delete this conntrack. */ @@ -70,6 +85,8 @@ int nf_ct_set_timeout(struct net *net, struct nf_conn *ct, const char *errmsg = NULL; int ret = 0; + WARN_ON_ONCE(!nf_ct_is_template(ct)); + rcu_read_lock(); h = rcu_dereference(nf_ct_timeout_hook); if (!h) { @@ -127,6 +144,8 @@ void nf_ct_destroy_timeout(struct nf_conn *ct) struct nf_conn_timeout *timeout_ext; const struct nf_ct_timeout_hooks *h; + WARN_ON_ONCE(!nf_ct_is_template(ct)); + rcu_read_lock(); h = rcu_dereference(nf_ct_timeout_hook); @@ -139,6 +158,8 @@ void nf_ct_destroy_timeout(struct nf_conn *ct) if (t) h->timeout_put(t); RCU_INIT_POINTER(timeout_ext->timeout, NULL); + if (t && refcount_dec_and_test(&t->refcnt)) + kfree_rcu(t, rcu); } } rcu_read_unlock(); diff --git a/net/netfilter/nf_flow_table_core.c b/net/netfilter/nf_flow_table_core.c index e3afc1f4f168..a0788039811c 100644 --- a/net/netfilter/nf_flow_table_core.c +++ b/net/netfilter/nf_flow_table_core.c @@ -302,7 +302,7 @@ int flow_offload_add(struct nf_flowtable *flow_table, struct flow_offload *flow) return err; } - nf_ct_offload_timeout(flow->ct); + nf_ct_refresh(flow->ct, NF_CT_DAY); if (nf_flowtable_hw_offload(flow_table)) { __set_bit(NF_FLOW_HW, &flow->flags); @@ -351,8 +351,8 @@ static void flow_offload_del(struct nf_flowtable *flow_table, void flow_offload_teardown(struct flow_offload *flow) { clear_bit(IPS_OFFLOAD_BIT, &flow->ct->status); - set_bit(NF_FLOW_TEARDOWN, &flow->flags); - flow_offload_fixup_ct(flow->ct); + if (!test_and_set_bit(NF_FLOW_TEARDOWN, &flow->flags)) + flow_offload_fixup_ct(flow->ct); } EXPORT_SYMBOL_GPL(flow_offload_teardown); @@ -422,15 +422,128 @@ static bool nf_flow_custom_gc(struct nf_flowtable *flow_table, return flow_table->type->gc && flow_table->type->gc(flow); } +/** + * nf_flow_table_tcp_timeout() - new timeout of offloaded tcp entry + * @ct: Flowtable offloaded tcp ct + * + * Return: number of seconds when ct entry should expire. + */ +static u32 nf_flow_table_tcp_timeout(const struct nf_conn *ct) +{ + u8 state = READ_ONCE(ct->proto.tcp.state); + + switch (state) { + case TCP_CONNTRACK_SYN_SENT: + case TCP_CONNTRACK_SYN_RECV: + return 0; + case TCP_CONNTRACK_ESTABLISHED: + return NF_CT_DAY; + case TCP_CONNTRACK_FIN_WAIT: + case TCP_CONNTRACK_CLOSE_WAIT: + case TCP_CONNTRACK_LAST_ACK: + case TCP_CONNTRACK_TIME_WAIT: + return 5 * 60 * HZ; + case TCP_CONNTRACK_CLOSE: + return 0; + } + + return 0; +} + +/** + * nf_flow_table_extend_ct_timeout() - Extend ct timeout of offloaded conntrack entry + * @ct: Flowtable offloaded ct + * + * Datapath lookups in the conntrack table will evict nf_conn entries + * if they have expired. + * + * Once nf_conn entries have been offloaded, nf_conntrack might not see any + * packets anymore. Thus ct->timeout is no longer refreshed and ct can + * be evicted. + * + * To avoid the need for an additional check on the offload bit for every + * packet processed via nf_conntrack_in(), set an arbitrary timeout large + * enough not to ever expire, this save us a check for the IPS_OFFLOAD_BIT + * from the packet path via nf_ct_is_expired(). + */ +static void nf_flow_table_extend_ct_timeout(struct nf_conn *ct) +{ + static const s32 min_timeout = 5 * 60 * HZ; + u32 ct_timeout = READ_ONCE(ct->timeout); + s32 expires; + + expires = ct_timeout - nfct_time_stamp; + if (expires <= 0) /* already expired */ + return; + + /* normal case: large enough timeout, nothing to do. */ + if (likely(expires >= min_timeout)) + return; + + /* must check offload bit after this, we do not hold any locks. + * flowtable and ct entries could have been removed on another CPU. + */ + if (!refcount_inc_not_zero(&ct->ct_general.use)) + return; + + /* load ct->status after refcount increase */ + smp_acquire__after_ctrl_dep(); + + if (nf_ct_is_confirmed(ct) && + test_bit(IPS_OFFLOAD_BIT, &ct->status)) { + u8 l4proto = nf_ct_protonum(ct); + u32 new_timeout = 1; + + switch (l4proto) { + case IPPROTO_UDP: + new_timeout = NF_CT_DAY; + break; + case IPPROTO_TCP: + new_timeout = nf_flow_table_tcp_timeout(ct); + break; + default: + WARN_ON_ONCE(1); + break; + } + + /* Update to ct->timeout from nf_conntrack happens + * without holding ct->lock. + * + * Use cmpxchg to ensure timeout extension doesn't + * happen when we race with conntrack datapath. + * + * The inverse -- datapath updating ->timeout right + * after this -- is fine, datapath is authoritative. + */ + if (new_timeout) { + new_timeout += nfct_time_stamp; + cmpxchg(&ct->timeout, ct_timeout, new_timeout); + } + } + + nf_ct_put(ct); +} + static void nf_flow_offload_gc_step(struct nf_flowtable *flow_table, struct flow_offload *flow, void *data) { + bool teardown; + + if (test_bit(NF_FLOW_PENDING, &flow->flags)) + return; + + teardown = test_bit(NF_FLOW_TEARDOWN, &flow->flags); + if (nf_flow_has_expired(flow) || nf_ct_is_dying(flow->ct) || - nf_flow_custom_gc(flow_table, flow)) + nf_flow_custom_gc(flow_table, flow)) { flow_offload_teardown(flow); + teardown = true; + } else if (!teardown) { + nf_flow_table_extend_ct_timeout(flow->ct); + } - if (test_bit(NF_FLOW_TEARDOWN, &flow->flags)) { + if (teardown) { if (test_bit(NF_FLOW_HW, &flow->flags)) { if (!test_bit(NF_FLOW_HW_DYING, &flow->flags)) nf_flow_offload_del(flow_table, flow); diff --git a/net/netfilter/nf_flow_table_ip.c b/net/netfilter/nf_flow_table_ip.c index 20dd4eeac78b..6e6d036fe139 100644 --- a/net/netfilter/nf_flow_table_ip.c +++ b/net/netfilter/nf_flow_table_ip.c @@ -268,6 +268,7 @@ static unsigned int nf_flow_xmit_xfrm(struct sk_buff *skb, struct dst_entry *dst) { skb_orphan(skb); + skb_dst_drop(skb); skb_dst_set_noref(skb, dst); dst_output(state->net, state->sk, skb); return NF_STOLEN; @@ -453,6 +454,7 @@ nf_flow_offload_ip_hook(void *priv, struct sk_buff *skb, outdev = rt->dst.dev; skb->dev = outdev; nexthop = rt_nexthop(rt, flow->tuplehash[!dir].tuple.src_v4.s_addr); + skb_dst_drop(skb); skb_dst_set_noref(skb, &rt->dst); neigh_xmit(NEIGH_ARP_TABLE, outdev, &nexthop, skb); ret = NF_STOLEN; @@ -748,6 +750,7 @@ nf_flow_offload_ipv6_hook(void *priv, struct sk_buff *skb, outdev = rt->dst.dev; skb->dev = outdev; nexthop = rt6_nexthop(rt, &flow->tuplehash[!dir].tuple.src_v6); + skb_dst_drop(skb); skb_dst_set_noref(skb, &rt->dst); neigh_xmit(NEIGH_ND_TABLE, outdev, nexthop, skb); ret = NF_STOLEN; diff --git a/net/netfilter/nf_flow_table_offload.c b/net/netfilter/nf_flow_table_offload.c index 0a49a6d72dcd..1f3469d45267 100644 --- a/net/netfilter/nf_flow_table_offload.c +++ b/net/netfilter/nf_flow_table_offload.c @@ -993,6 +993,7 @@ static void flow_offload_work_del(struct flow_offload_work *offload) flow_offload_tuple_del(offload, FLOW_OFFLOAD_DIR_ORIGINAL); if (test_bit(NF_FLOW_HW_BIDIRECTIONAL, &offload->flow->flags)) flow_offload_tuple_del(offload, FLOW_OFFLOAD_DIR_REPLY); + set_bit(NF_FLOW_HW_DEAD, &offload->flow->flags); } static void flow_offload_tuple_stats(struct flow_offload_work *offload, @@ -1054,13 +1055,8 @@ static void flow_offload_work_handler(struct work_struct *work) default: WARN_ON_ONCE(1); } - - clear_bit(NF_FLOW_HW_PENDING, &offload->flow->flags); - if (offload->cmd == FLOW_CLS_DESTROY) { - /* Publish after the worker's last flow access. */ - smp_mb__before_atomic(); - set_bit(NF_FLOW_HW_DEAD, &offload->flow->flags); - } + smp_mb__before_atomic(); + clear_bit(NF_FLOW_PENDING, &offload->flow->flags); kfree(offload); } @@ -1087,12 +1083,12 @@ nf_flow_offload_work_alloc(struct nf_flowtable *flowtable, { struct flow_offload_work *offload; - if (test_and_set_bit(NF_FLOW_HW_PENDING, &flow->flags)) + if (test_and_set_bit(NF_FLOW_PENDING, &flow->flags)) return NULL; offload = kmalloc(sizeof(struct flow_offload_work), GFP_ATOMIC); if (!offload) { - clear_bit(NF_FLOW_HW_PENDING, &flow->flags); + clear_bit(NF_FLOW_PENDING, &flow->flags); return NULL; } diff --git a/net/netfilter/nf_nat_bpf.c b/net/netfilter/nf_nat_bpf.c index 141ee7783223..2fd62678e775 100644 --- a/net/netfilter/nf_nat_bpf.c +++ b/net/netfilter/nf_nat_bpf.c @@ -41,6 +41,9 @@ __bpf_kfunc int bpf_ct_set_nat_info(struct nf_conn___init *nfct, if (proto != NFPROTO_IPV4 && proto != NFPROTO_IPV6) return -EINVAL; + if (manip != NF_NAT_MANIP_SRC && manip != NF_NAT_MANIP_DST) + return -EINVAL; + memset(&range, 0, sizeof(struct nf_nat_range2)); range.flags = NF_NAT_RANGE_MAP_IPS; range.min_addr = *addr; diff --git a/net/netfilter/nf_nat_core.c b/net/netfilter/nf_nat_core.c index d29767646ffa..649dede4ace0 100644 --- a/net/netfilter/nf_nat_core.c +++ b/net/netfilter/nf_nat_core.c @@ -794,8 +794,9 @@ nf_nat_setup_info(struct nf_conn *ct, if (nf_ct_is_confirmed(ct)) return NF_ACCEPT; - WARN_ON(maniptype != NF_NAT_MANIP_SRC && - maniptype != NF_NAT_MANIP_DST); + if (WARN_ON(maniptype != NF_NAT_MANIP_SRC && + maniptype != NF_NAT_MANIP_DST)) + return NF_DROP; if (WARN_ON(nf_nat_initialized(ct, maniptype))) return NF_DROP; @@ -1233,7 +1234,7 @@ int nf_nat_register_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops, if (!nat_proto_net->nat_hook_ops) { WARN_ON(nat_proto_net->users != 0); - nat_ops = kmemdup(orig_nat_ops, sizeof(*orig_nat_ops) * ops_count, GFP_KERNEL); + nat_ops = kmemdup_array(orig_nat_ops, ops_count, sizeof(*orig_nat_ops), GFP_KERNEL); if (!nat_ops) { mutex_unlock(&nf_nat_proto_mutex); return -ENOMEM; @@ -1255,9 +1256,11 @@ int nf_nat_register_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops, ret = nf_register_net_hooks(net, nat_ops, ops_count); if (ret < 0) { mutex_unlock(&nf_nat_proto_mutex); - for (i = 0; i < ops_count; i++) - kfree(nat_ops[i].priv); - kfree(nat_ops); + for (i = 0; i < ops_count; i++) { + priv = nat_ops[i].priv; + kfree_rcu(priv, rcu_head); + } + kfree_rcu(nat_ops, rcu); return ret; } @@ -1321,7 +1324,7 @@ void nf_nat_unregister_fn(struct net *net, u8 pf, const struct nf_hook_ops *ops, } nat_proto_net->nat_hook_ops = NULL; - kfree(nat_ops); + kfree_rcu(nat_ops, rcu); } unlock: mutex_unlock(&nf_nat_proto_mutex); diff --git a/net/netfilter/nf_synproxy_core.c b/net/netfilter/nf_synproxy_core.c index 500a90311ed5..35c38c3268a6 100644 --- a/net/netfilter/nf_synproxy_core.c +++ b/net/netfilter/nf_synproxy_core.c @@ -183,23 +183,23 @@ synproxy_check_timestamp_cookie(struct synproxy_options *opts) opts->options |= opts->tsecr & (1 << 5) ? NF_SYNPROXY_OPT_ECN : 0; } -static unsigned int +static bool synproxy_tstamp_adjust(struct sk_buff *skb, unsigned int protoff, struct tcphdr *th, struct nf_conn *ct, enum ip_conntrack_info ctinfo, const struct nf_conn_synproxy *synproxy) { unsigned int optoff, optend; - __be32 *ptr, old; + u32 new, old; if (synproxy->tsoff == 0) - return 1; + return true; optoff = protoff + sizeof(struct tcphdr); optend = protoff + th->doff * 4; if (skb_ensure_writable(skb, optend)) - return 0; + return false; th = (struct tcphdr *)(skb->data + protoff); @@ -208,7 +208,7 @@ synproxy_tstamp_adjust(struct sk_buff *skb, unsigned int protoff, switch (op[0]) { case TCPOPT_EOL: - return 1; + return true; case TCPOPT_NOP: optoff++; continue; @@ -216,28 +216,26 @@ synproxy_tstamp_adjust(struct sk_buff *skb, unsigned int protoff, if (optoff + 1 == optend || optoff + op[1] > optend || op[1] < 2) - return 0; + return true; if (op[0] == TCPOPT_TIMESTAMP && op[1] == TCPOLEN_TIMESTAMP) { if (CTINFO2DIR(ctinfo) == IP_CT_DIR_REPLY) { - ptr = (__be32 *)&op[2]; - old = *ptr; - *ptr = htonl(ntohl(*ptr) - - synproxy->tsoff); + old = get_unaligned_be32(&op[2]); + new = old - synproxy->tsoff; + put_unaligned_be32(new, &op[2]); } else { - ptr = (__be32 *)&op[6]; - old = *ptr; - *ptr = htonl(ntohl(*ptr) + - synproxy->tsoff); + old = get_unaligned_be32(&op[6]); + new = old + synproxy->tsoff; + put_unaligned_be32(new, &op[6]); } inet_proto_csum_replace4(&th->check, skb, - old, *ptr, false); - return 1; + cpu_to_be32(old), + cpu_to_be32(new), false); } optoff += op[1]; } } - return 1; + return true; } #ifdef CONFIG_PROC_FS @@ -748,7 +746,9 @@ ipv4_synproxy_hook(void *priv, struct sk_buff *skb, break; } - synproxy_tstamp_adjust(skb, thoff, th, ct, ctinfo, synproxy); + if (!synproxy_tstamp_adjust(skb, thoff, th, ct, ctinfo, synproxy)) + return NF_DROP_REASON(skb, SKB_DROP_REASON_NETFILTER_DROP, ENOMEM); + return NF_ACCEPT; } EXPORT_SYMBOL_GPL(ipv4_synproxy_hook); @@ -1176,7 +1176,9 @@ ipv6_synproxy_hook(void *priv, struct sk_buff *skb, break; } - synproxy_tstamp_adjust(skb, thoff, th, ct, ctinfo, synproxy); + if (!synproxy_tstamp_adjust(skb, thoff, th, ct, ctinfo, synproxy)) + return NF_DROP_REASON(skb, SKB_DROP_REASON_NETFILTER_DROP, ENOMEM); + return NF_ACCEPT; } EXPORT_SYMBOL_GPL(ipv6_synproxy_hook); diff --git a/net/netfilter/nf_tables_api.c b/net/netfilter/nf_tables_api.c index c8a3c344cfaf..a036deb93781 100644 --- a/net/netfilter/nf_tables_api.c +++ b/net/netfilter/nf_tables_api.c @@ -42,8 +42,6 @@ enum { NFT_VALIDATE_DO, }; -static struct rhltable nft_objname_ht; - static u32 nft_chain_hash(const void *data, u32 len, u32 seed); static u32 nft_chain_hash_obj(const void *data, u32 len, u32 seed); static int nft_chain_hash_cmp(struct rhashtable_compare_arg *, const void *); @@ -334,7 +332,7 @@ static void nft_netdev_unregister_hooks(struct net *net, list_for_each_entry_safe(hook, next, hook_list, list) { nf_unregister_net_hook(net, &hook->ops); if (release_netdev) { - list_del(&hook->list); + list_del_rcu(&hook->list); kfree_rcu(hook, rcu); } } @@ -1397,6 +1395,10 @@ static int nf_tables_newtable(struct sk_buff *skb, const struct nfnl_info *info, if (err) goto err_chain_ht; + err = rhltable_init(&table->objname_ht, &nft_objname_ht_params); + if (err < 0) + goto err_obj_ht; + INIT_LIST_HEAD(&table->chains); INIT_LIST_HEAD(&table->sets); INIT_LIST_HEAD(&table->objects); @@ -1415,6 +1417,8 @@ static int nf_tables_newtable(struct sk_buff *skb, const struct nfnl_info *info, list_add_tail_rcu(&table->list, &nft_net->tables); return 0; err_trans: + rhltable_destroy(&table->objname_ht); +err_obj_ht: rhltable_destroy(&table->chains_ht); err_chain_ht: kfree(table->udata); @@ -1581,6 +1585,7 @@ static void nf_tables_table_destroy(struct nft_ctx *ctx) return; rhltable_destroy(&ctx->table->chains_ht); + rhltable_destroy(&ctx->table->objname_ht); kfree(ctx->table->name); kfree(ctx->table->udata); kfree(ctx->table); @@ -6461,39 +6466,11 @@ void nf_tables_set_elem_destroy(const struct nft_ctx *ctx, kfree(elem); } -int nft_set_elem_expr_clone(const struct nft_ctx *ctx, struct nft_set *set, - struct nft_expr *expr_array[]) -{ - struct nft_expr *expr; - int err, i, k; - - for (i = 0; i < set->num_exprs; i++) { - expr = kzalloc(set->exprs[i]->ops->size, GFP_KERNEL_ACCOUNT); - if (!expr) - goto err_expr; - - err = nft_expr_clone(expr, set->exprs[i], GFP_KERNEL_ACCOUNT); - if (err < 0) { - kfree(expr); - goto err_expr; - } - expr_array[i] = expr; - } - - return 0; - -err_expr: - for (k = i - 1; k >= 0; k--) - nft_expr_destroy(ctx, expr_array[k]); - - return -ENOMEM; -} - static int nft_set_elem_expr_setup(struct nft_ctx *ctx, const struct nft_set_ext_tmpl *tmpl, const struct nft_set_ext *ext, struct nft_expr *expr_array[], - u32 num_exprs) + u32 num_exprs, bool override_exprs) { struct nft_set_elem_expr *elem_expr = nft_set_ext_expr(ext); u32 len = sizeof(struct nft_set_elem_expr); @@ -6516,7 +6493,8 @@ static int nft_set_elem_expr_setup(struct nft_ctx *ctx, goto err_elem_expr_setup; elem_expr->size += expr_array[i]->ops->size; - nft_expr_destroy(ctx, expr_array[i]); + if (override_exprs) + nft_expr_destroy(ctx, expr_array[i]); expr_array[i] = NULL; } @@ -6524,7 +6502,9 @@ static int nft_set_elem_expr_setup(struct nft_ctx *ctx, err_elem_expr_setup: for (; i < num_exprs; i++) { - nft_expr_destroy(ctx, expr_array[i]); + if (override_exprs) + nft_expr_destroy(ctx, expr_array[i]); + expr_array[i] = NULL; } @@ -6553,16 +6533,19 @@ EXPORT_SYMBOL_GPL(nft_set_catchall_lookup); static int nft_setelem_catchall_insert(const struct net *net, struct nft_set *set, const struct nft_set_elem *elem, - struct nft_set_ext **pext) + void **priv) { struct nft_set_elem_catchall *catchall; u8 genmask = nft_genmask_next(net); + u64 tstamp = nft_net_tstamp(net); struct nft_set_ext *ext; list_for_each_entry(catchall, &set->catchall_list, list) { ext = nft_set_elem_ext(set, catchall->elem); - if (nft_set_elem_active(ext, genmask)) { - *pext = ext; + if (nft_set_elem_active(ext, genmask) && + !__nft_set_elem_expired(ext, tstamp) && + !nft_set_elem_is_dead(ext)) { + *priv = catchall->elem; return -EEXIST; } } @@ -6580,14 +6563,15 @@ static int nft_setelem_catchall_insert(const struct net *net, static int nft_setelem_insert(const struct net *net, struct nft_set *set, const struct nft_set_elem *elem, - struct nft_set_ext **ext, unsigned int flags) + void **elem_priv, + unsigned int flags) { int ret; if (flags & NFT_SET_ELEM_CATCHALL) - ret = nft_setelem_catchall_insert(net, set, elem, ext); + ret = nft_setelem_catchall_insert(net, set, elem, elem_priv); else - ret = set->ops->insert(net, set, elem, ext); + ret = set->ops->insert(net, set, elem, elem_priv); return ret; } @@ -6621,11 +6605,14 @@ static int nft_setelem_catchall_deactivate(const struct net *net, struct nft_set_elem *elem) { struct nft_set_elem_catchall *catchall; + u64 tstamp = nft_net_tstamp(net); struct nft_set_ext *ext; list_for_each_entry(catchall, &set->catchall_list, list) { ext = nft_set_elem_ext(set, catchall->elem); - if (!nft_is_active_next(net, ext)) + if (!nft_is_active_next(net, ext) || + __nft_set_elem_expired(ext, tstamp) || + nft_set_elem_is_dead(ext)) continue; kfree(elem->priv); @@ -6750,11 +6737,13 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set, struct nft_set_elem elem; struct nft_set_binding *binding; struct nft_object *obj = NULL; + bool override_exprs = false; struct nft_userdata *udata; struct nft_data_desc desc; enum nft_registers dreg; struct nft_trans *trans; bool set_full = false; + void *elem_priv; u64 timeout; u64 expiration; int err, i; @@ -6855,6 +6844,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set, expr_array[0] = expr; num_exprs = 1; + override_exprs = true; if (set->num_exprs && set->exprs[0]->ops != expr->ops) { err = -EOPNOTSUPP; @@ -6883,6 +6873,7 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set, } expr_array[i] = expr; num_exprs++; + override_exprs = true; if (set->num_exprs && expr->ops != set->exprs[i]->ops) { err = -EOPNOTSUPP; @@ -6896,9 +6887,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set, } } else if (set->num_exprs > 0 && !(flags & NFT_SET_ELEM_INTERVAL_END)) { - err = nft_set_elem_expr_clone(ctx, set, expr_array); - if (err < 0) - goto err_set_elem_expr_clone; + for (i = 0; i < set->num_exprs; i++) + expr_array[i] = set->exprs[i]; num_exprs = set->num_exprs; } @@ -7043,7 +7033,8 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set, udata->len = ulen - 1; nla_memcpy(&udata->data, nla[NFTA_SET_ELEM_USERDATA], ulen); } - err = nft_set_elem_expr_setup(ctx, &tmpl, ext, expr_array, num_exprs); + err = nft_set_elem_expr_setup(ctx, &tmpl, ext, expr_array, num_exprs, + override_exprs); if (err < 0) goto err_elem_free; @@ -7063,9 +7054,10 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set, ext->genmask = nft_genmask_cur(ctx->net); - err = nft_setelem_insert(ctx->net, set, &elem, &ext2, flags); + err = nft_setelem_insert(ctx->net, set, &elem, &elem_priv, flags); if (err) { if (err == -EEXIST) { + ext2 = nft_set_elem_ext(set, elem_priv); if (nft_set_ext_exists(ext, NFT_SET_EXT_DATA) ^ nft_set_ext_exists(ext2, NFT_SET_EXT_DATA) || nft_set_ext_exists(ext, NFT_SET_EXT_OBJREF) ^ @@ -7113,9 +7105,11 @@ static int nft_add_set_elem(struct nft_ctx *ctx, struct nft_set *set, err_parse_key: nft_data_release(&elem.key.val, NFT_DATA_VALUE); err_set_elem_expr: - for (i = 0; i < num_exprs && expr_array[i]; i++) - nft_expr_destroy(ctx, expr_array[i]); -err_set_elem_expr_clone: + if (override_exprs) { + for (i = 0; i < num_exprs && expr_array[i]; i++) + nft_expr_destroy(ctx, expr_array[i]); + } + return err; } @@ -7506,7 +7500,7 @@ void nft_unregister_obj(struct nft_object_type *obj_type) EXPORT_SYMBOL_GPL(nft_unregister_obj); struct nft_object *nft_obj_lookup(const struct net *net, - const struct nft_table *table, + struct nft_table *table, const struct nlattr *nla, u32 objtype, u8 genmask) { @@ -7522,7 +7516,7 @@ struct nft_object *nft_obj_lookup(const struct net *net, !lockdep_commit_lock_is_held(net)); rcu_read_lock(); - list = rhltable_lookup(&nft_objname_ht, &k, nft_objname_ht_params); + list = rhltable_lookup(&table->objname_ht, &k, nft_objname_ht_params); if (!list) goto out; @@ -7801,7 +7795,7 @@ static int nf_tables_newobj(struct sk_buff *skb, const struct nfnl_info *info, if (err < 0) goto err_trans; - err = rhltable_insert(&nft_objname_ht, &obj->rhlhead, + err = rhltable_insert(&table->objname_ht, &obj->rhlhead, nft_objname_ht_params); if (err < 0) goto err_obj_ht; @@ -7941,19 +7935,6 @@ static int nf_tables_dump_obj(struct sk_buff *skb, struct netlink_callback *cb) return skb->len; } -static int nf_tables_dumpreset_obj(struct sk_buff *skb, - struct netlink_callback *cb) -{ - struct nftables_pernet *nft_net = nft_pernet(sock_net(skb->sk)); - int ret; - - mutex_lock(&nft_net->commit_mutex); - ret = nf_tables_dump_obj(skb, cb); - mutex_unlock(&nft_net->commit_mutex); - - return ret; -} - static int nf_tables_dump_obj_start(struct netlink_callback *cb) { struct nft_obj_dump_ctx *ctx = (void *)cb->ctx; @@ -7970,16 +7951,10 @@ static int nf_tables_dump_obj_start(struct netlink_callback *cb) if (nla[NFTA_OBJ_TYPE]) ctx->type = ntohl(nla_get_be32(nla[NFTA_OBJ_TYPE])); - return 0; -} - -static int nf_tables_dumpreset_obj_start(struct netlink_callback *cb) -{ - struct nft_obj_dump_ctx *ctx = (void *)cb->ctx; - - ctx->reset = true; + if (NFNL_MSG_TYPE(cb->nlh->nlmsg_type) == NFT_MSG_GETOBJ_RESET) + ctx->reset = true; - return nf_tables_dump_obj_start(cb); + return 0; } static int nf_tables_dump_obj_done(struct netlink_callback *cb) @@ -7999,8 +7974,8 @@ nf_tables_getobj_single(u32 portid, const struct nfnl_info *info, struct netlink_ext_ack *extack = info->extack; u8 genmask = nft_genmask_cur(info->net); u8 family = info->nfmsg->nfgen_family; - const struct nft_table *table; struct net *net = info->net; + struct nft_table *table; struct nft_object *obj; struct sk_buff *skb2; u32 objtype; @@ -8041,42 +8016,16 @@ nf_tables_getobj_single(u32 portid, const struct nfnl_info *info, static int nf_tables_getobj(struct sk_buff *skb, const struct nfnl_info *info, const struct nlattr * const nla[]) { - u32 portid = NETLINK_CB(skb).portid; - struct sk_buff *skb2; - - if (info->nlh->nlmsg_flags & NLM_F_DUMP) { - struct netlink_dump_control c = { - .start = nf_tables_dump_obj_start, - .dump = nf_tables_dump_obj, - .done = nf_tables_dump_obj_done, - .module = THIS_MODULE, - .data = (void *)nla, - }; - - return nft_netlink_dump_start_rcu(info->sk, skb, info->nlh, &c); - } - - skb2 = nf_tables_getobj_single(portid, info, nla, false); - if (IS_ERR(skb2)) - return PTR_ERR(skb2); - - return nfnetlink_unicast(skb2, info->net, portid); -} - -static int nf_tables_getobj_reset(struct sk_buff *skb, - const struct nfnl_info *info, - const struct nlattr * const nla[]) -{ - struct nftables_pernet *nft_net = nft_pernet(info->net); u32 portid = NETLINK_CB(skb).portid; struct net *net = info->net; struct sk_buff *skb2; + bool reset = false; char *buf; if (info->nlh->nlmsg_flags & NLM_F_DUMP) { struct netlink_dump_control c = { - .start = nf_tables_dumpreset_obj_start, - .dump = nf_tables_dumpreset_obj, + .start = nf_tables_dump_obj_start, + .dump = nf_tables_dump_obj, .done = nf_tables_dump_obj_done, .module = THIS_MODULE, .data = (void *)nla, @@ -8085,18 +8034,16 @@ static int nf_tables_getobj_reset(struct sk_buff *skb, return nft_netlink_dump_start_rcu(info->sk, skb, info->nlh, &c); } - if (!try_module_get(THIS_MODULE)) - return -EINVAL; - rcu_read_unlock(); - mutex_lock(&nft_net->commit_mutex); - skb2 = nf_tables_getobj_single(portid, info, nla, true); - mutex_unlock(&nft_net->commit_mutex); - rcu_read_lock(); - module_put(THIS_MODULE); + if (NFNL_MSG_TYPE(info->nlh->nlmsg_type) == NFT_MSG_GETOBJ_RESET) + reset = true; + skb2 = nf_tables_getobj_single(portid, info, nla, reset); if (IS_ERR(skb2)) return PTR_ERR(skb2); + if (!reset) + return nfnetlink_unicast(skb2, net, NETLINK_CB(skb).portid); + buf = kasprintf(GFP_ATOMIC, "%.*s:%u", nla_len(nla[NFTA_OBJ_TABLE]), (char *)nla_data(nla[NFTA_OBJ_TABLE]), @@ -8454,7 +8401,7 @@ static void __nft_unregister_flowtable_net_hooks(struct net *net, flowtable->data.type->setup(&flowtable->data, hook->ops.dev, FLOW_BLOCK_UNBIND); if (release_netdev) { - list_del(&hook->list); + list_del_rcu(&hook->list); kfree_rcu(hook, rcu); } } @@ -9404,7 +9351,7 @@ static const struct nfnl_callback nf_tables_cb[NFT_MSG_MAX] = { .policy = nft_obj_policy, }, [NFT_MSG_GETOBJ_RESET] = { - .call = nf_tables_getobj_reset, + .call = nf_tables_getobj, .type = NFNL_CB_RCU, .attr_count = NFTA_OBJ_MAX, .policy = nft_obj_policy, @@ -9751,9 +9698,9 @@ static void nf_tables_commit_chain(struct net *net, struct nft_chain *chain) nf_tables_commit_chain_free_rules_old(g0); } -static void nft_obj_del(struct nft_object *obj) +static void nft_obj_del(struct nft_table *table, struct nft_object *obj) { - rhltable_remove(&nft_objname_ht, &obj->rhlhead, nft_objname_ht_params); + rhltable_remove(&table->objname_ht, &obj->rhlhead, nft_objname_ht_params); list_del_rcu(&obj->list); } @@ -10435,7 +10382,7 @@ static int nf_tables_commit(struct net *net, struct sk_buff *skb) break; case NFT_MSG_DELOBJ: case NFT_MSG_DESTROYOBJ: - nft_obj_del(nft_trans_obj(trans)); + nft_obj_del(trans->ctx.table, nft_trans_obj(trans)); nf_tables_obj_notify(&trans->ctx, nft_trans_obj(trans), trans->msg_type); break; @@ -10716,7 +10663,7 @@ static int __nf_tables_abort(struct net *net, enum nfnl_abort_action action) nft_trans_destroy(trans); } else { nft_use_dec_restore(&trans->ctx.table->use); - nft_obj_del(nft_trans_obj(trans)); + nft_obj_del(trans->ctx.table, nft_trans_obj(trans)); } break; case NFT_MSG_DELOBJ: @@ -11340,7 +11287,7 @@ static void __nft_release_table(struct net *net, struct nft_table *table) nft_set_destroy(&ctx, set); } list_for_each_entry_safe(obj, ne, &table->objects, list) { - nft_obj_del(obj); + nft_obj_del(table, obj); nft_use_dec(&table->use); nft_obj_destroy(&ctx, obj); } @@ -11503,10 +11450,6 @@ static int __init nf_tables_module_init(void) if (err < 0) goto err_netdev_notifier; - err = rhltable_init(&nft_objname_ht, &nft_objname_ht_params); - if (err < 0) - goto err_rht_objname; - err = nft_offload_init(); if (err < 0) goto err_offload; @@ -11529,8 +11472,6 @@ static int __init nf_tables_module_init(void) err_netlink_notifier: nft_offload_exit(); err_offload: - rhltable_destroy(&nft_objname_ht); -err_rht_objname: unregister_netdevice_notifier(&nf_tables_flowtable_notifier); err_netdev_notifier: nf_tables_core_module_exit(); @@ -11554,7 +11495,6 @@ static void __exit nf_tables_module_exit(void) cancel_work_sync(&trans_gc_work); cancel_work_sync(&trans_destroy_work); rcu_barrier(); - rhltable_destroy(&nft_objname_ht); nf_tables_core_module_exit(); } diff --git a/net/netfilter/nf_tables_core.c b/net/netfilter/nf_tables_core.c index 711c22ab701d..4b0d2b3a7bca 100644 --- a/net/netfilter/nf_tables_core.c +++ b/net/netfilter/nf_tables_core.c @@ -308,10 +308,11 @@ nft_do_chain(struct nft_pktinfo *pkt, void *priv) switch (regs.verdict.code & NF_VERDICT_MASK) { case NF_ACCEPT: - case NF_DROP: case NF_QUEUE: case NF_STOLEN: return regs.verdict.code; + case NF_DROP: + return NF_DROP_REASON(pkt->skb, SKB_DROP_REASON_NETFILTER_DROP, EPERM); } switch (regs.verdict.code) { @@ -342,6 +343,9 @@ nft_do_chain(struct nft_pktinfo *pkt, void *priv) if (static_branch_unlikely(&nft_counters_enabled)) nft_update_chain_stats(basechain, pkt); + if (nft_base_chain(basechain)->policy == NF_DROP) + return NF_DROP_REASON(pkt->skb, SKB_DROP_REASON_NETFILTER_DROP, EPERM); + return nft_base_chain(basechain)->policy; } EXPORT_SYMBOL_GPL(nft_do_chain); diff --git a/net/netfilter/nf_tables_trace.c b/net/netfilter/nf_tables_trace.c index 6d41c0bd3d78..6eedb84b552e 100644 --- a/net/netfilter/nf_tables_trace.c +++ b/net/netfilter/nf_tables_trace.c @@ -313,7 +313,7 @@ void nft_trace_init(struct nft_traceinfo *info, const struct nft_pktinfo *pkt, net_get_random_once(&trace_key, sizeof(trace_key)); info->skbid = (u32)siphash_3u32(hash32_ptr(skb), - skb_get_hash(skb), + skb_get_hash_net(nft_net(pkt), skb), skb->skb_iif, &trace_key); } diff --git a/net/netfilter/nfnetlink_cttimeout.c b/net/netfilter/nfnetlink_cttimeout.c index f466af4f8531..6067e99830e4 100644 --- a/net/netfilter/nfnetlink_cttimeout.c +++ b/net/netfilter/nfnetlink_cttimeout.c @@ -37,11 +37,8 @@ struct ctnl_timeout { struct list_head head; struct list_head free_head; struct rcu_head rcu_head; - refcount_t refcnt; char name[CTNL_TIMEOUT_NAME_MAX]; - - /* must be at the end */ - struct nf_ct_timeout timeout; + struct nf_ct_timeout *timeout; }; struct nfct_timeout_pernet { @@ -133,12 +130,12 @@ static int cttimeout_new_timeout(struct sk_buff *skb, /* You cannot replace one timeout policy by another of * different kind, sorry. */ - if (matching->timeout.l3num != l3num || - matching->timeout.l4proto->l4proto != l4num) + if (matching->timeout->l3num != l3num || + matching->timeout->l4proto->l4proto != l4num) return -EINVAL; - return ctnl_timeout_parse_policy(&matching->timeout.data, - matching->timeout.l4proto, + return ctnl_timeout_parse_policy(&matching->timeout->data, + matching->timeout->l4proto, info->net, cda[CTA_TIMEOUT_DATA]); } @@ -154,26 +151,35 @@ static int cttimeout_new_timeout(struct sk_buff *skb, goto err_proto_put; } - timeout = kzalloc(sizeof(struct ctnl_timeout) + - l4proto->ctnl_timeout.obj_size, GFP_KERNEL); + timeout = kzalloc(sizeof(*timeout), GFP_KERNEL); if (timeout == NULL) { ret = -ENOMEM; goto err_proto_put; } - ret = ctnl_timeout_parse_policy(&timeout->timeout.data, l4proto, + timeout->timeout = kzalloc(sizeof(*timeout->timeout) + + l4proto->ctnl_timeout.obj_size, GFP_KERNEL); + if (!timeout->timeout) { + ret = -ENOMEM; + goto err; + } + + ret = ctnl_timeout_parse_policy(&timeout->timeout->data, l4proto, info->net, cda[CTA_TIMEOUT_DATA]); if (ret < 0) - goto err; + goto err_free_timeout_policy; strcpy(timeout->name, nla_data(cda[CTA_TIMEOUT_NAME])); - timeout->timeout.l3num = l3num; - timeout->timeout.l4proto = l4proto; - refcount_set(&timeout->refcnt, 1); + timeout->timeout->l3num = l3num; + timeout->timeout->l4proto = l4proto; + refcount_set(&timeout->timeout->refcnt, 1); __module_get(THIS_MODULE); list_add_tail_rcu(&timeout->head, &pernet->nfct_timeout_list); return 0; + +err_free_timeout_policy: + kfree(timeout->timeout); err: kfree(timeout); err_proto_put: @@ -186,7 +192,7 @@ ctnl_timeout_fill_info(struct sk_buff *skb, u32 portid, u32 seq, u32 type, { struct nlmsghdr *nlh; unsigned int flags = portid ? NLM_F_MULTI : 0; - const struct nf_conntrack_l4proto *l4proto = timeout->timeout.l4proto; + const struct nf_conntrack_l4proto *l4proto = timeout->timeout->l4proto; struct nlattr *nest_parms; int ret; @@ -198,17 +204,17 @@ ctnl_timeout_fill_info(struct sk_buff *skb, u32 portid, u32 seq, u32 type, if (nla_put_string(skb, CTA_TIMEOUT_NAME, timeout->name) || nla_put_be16(skb, CTA_TIMEOUT_L3PROTO, - htons(timeout->timeout.l3num)) || + htons(timeout->timeout->l3num)) || nla_put_u8(skb, CTA_TIMEOUT_L4PROTO, l4proto->l4proto) || nla_put_be32(skb, CTA_TIMEOUT_USE, - htonl(refcount_read(&timeout->refcnt)))) + htonl(refcount_read(&timeout->timeout->refcnt)))) goto nla_put_failure; nest_parms = nla_nest_start(skb, CTA_TIMEOUT_DATA); if (!nest_parms) goto nla_put_failure; - ret = l4proto->ctnl_timeout.obj_to_nlattr(skb, &timeout->timeout.data); + ret = l4proto->ctnl_timeout.obj_to_nlattr(skb, &timeout->timeout->data); if (ret < 0) goto nla_put_failure; @@ -308,23 +314,17 @@ static int cttimeout_get_timeout(struct sk_buff *skb, return ret; } -/* try to delete object, fail if it is still in use. */ -static int ctnl_timeout_try_del(struct net *net, struct ctnl_timeout *timeout) +static void ctnl_timeout_del(struct net *net, struct ctnl_timeout *timeout) { - int ret = 0; + /* We are protected by nfnl mutex. */ + list_del_rcu(&timeout->head); + nf_ct_untimeout(net, timeout->timeout); - /* We want to avoid races with ctnl_timeout_put. So only when the - * current refcnt is 1, we decrease it to 0. - */ - if (refcount_dec_if_one(&timeout->refcnt)) { - /* We are protected by nfnl mutex. */ - list_del_rcu(&timeout->head); - nf_ct_untimeout(net, &timeout->timeout); - kfree_rcu(timeout, rcu_head); - } else { - ret = -EBUSY; - } - return ret; + if (refcount_dec_and_test(&timeout->timeout->refcnt)) + kfree_rcu(timeout->timeout, rcu); + + kfree_rcu(timeout, rcu_head); + module_put(THIS_MODULE); } static int cttimeout_del_timeout(struct sk_buff *skb, @@ -339,7 +339,7 @@ static int cttimeout_del_timeout(struct sk_buff *skb, if (!cda[CTA_TIMEOUT_NAME]) { list_for_each_entry_safe(cur, tmp, &pernet->nfct_timeout_list, head) - ctnl_timeout_try_del(info->net, cur); + ctnl_timeout_del(info->net, cur); return 0; } @@ -349,10 +349,8 @@ static int cttimeout_del_timeout(struct sk_buff *skb, if (strncmp(cur->name, name, CTNL_TIMEOUT_NAME_MAX) != 0) continue; - ret = ctnl_timeout_try_del(info->net, cur); - if (ret < 0) - return ret; - + ctnl_timeout_del(info->net, cur); + ret = 0; break; } return ret; @@ -519,24 +517,22 @@ static struct nf_ct_timeout *ctnl_timeout_find_get(struct net *net, if (strncmp(timeout->name, name, CTNL_TIMEOUT_NAME_MAX) != 0) continue; - if (!refcount_inc_not_zero(&timeout->refcnt)) + if (!refcount_inc_not_zero(&timeout->timeout->refcnt)) goto err; matching = timeout; + __module_get(THIS_MODULE); break; } err: - return matching ? &matching->timeout : NULL; + return matching ? matching->timeout : NULL; } -static void ctnl_timeout_put(struct nf_ct_timeout *t) +static void ctnl_timeout_put(struct nf_ct_timeout *timeout) { - struct ctnl_timeout *timeout = - container_of(t, struct ctnl_timeout, timeout); + if (refcount_dec_and_test(&timeout->refcnt)) + kfree_rcu(timeout, rcu); - if (refcount_dec_and_test(&timeout->refcnt)) { - kfree_rcu(timeout, rcu_head); - module_put(THIS_MODULE); - } + module_put(THIS_MODULE); } static const struct nfnl_callback cttimeout_cb[IPCTNL_MSG_TIMEOUT_MAX] = { @@ -617,8 +613,11 @@ static void __net_exit cttimeout_net_exit(struct net *net) list_for_each_entry_safe(cur, tmp, &pernet->nfct_timeout_freelist, free_head) { list_del(&cur->free_head); - if (refcount_dec_and_test(&cur->refcnt)) - kfree_rcu(cur, rcu_head); + if (refcount_dec_and_test(&cur->timeout->refcnt)) + kfree_rcu(cur->timeout, rcu); + + kfree_rcu(cur, rcu_head); + module_put(THIS_MODULE); } } @@ -657,24 +656,13 @@ static int __init cttimeout_init(void) return ret; } -static int untimeout(struct nf_conn *ct, void *timeout) -{ - struct nf_conn_timeout *timeout_ext = nf_ct_timeout_find(ct); - - if (timeout_ext) - RCU_INIT_POINTER(timeout_ext->timeout, NULL); - - return 0; -} - static void __exit cttimeout_exit(void) { nfnetlink_subsys_unregister(&cttimeout_subsys); - unregister_pernet_subsys(&cttimeout_ops); RCU_INIT_POINTER(nf_ct_timeout_hook, NULL); - - nf_ct_iterate_destroy(untimeout, NULL); + synchronize_net(); + unregister_pernet_subsys(&cttimeout_ops); } module_init(cttimeout_init); diff --git a/net/netfilter/nfnetlink_log.c b/net/netfilter/nfnetlink_log.c index abae76685a04..c0c8935dbcbb 100644 --- a/net/netfilter/nfnetlink_log.c +++ b/net/netfilter/nfnetlink_log.c @@ -455,6 +455,23 @@ static int nfulnl_put_bridge(struct nfulnl_instance *inst, const struct sk_buff return -1; } +#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER) +static int nflog_put_master_ifindex(struct sk_buff *nlskb, int attr, + const struct net_device *dev) +{ + const struct net_device *upper; + + if (dev && !netif_is_bridge_port(dev)) + return 0; + + upper = netdev_master_upper_dev_get_rcu((struct net_device *)dev); + if (upper && nla_put_be32(nlskb, attr, htonl(upper->ifindex))) + return -EMSGSIZE; + + return 0; +} +#endif + /* This is an inline function, we don't really care about a long * list of arguments */ static inline int @@ -509,8 +526,7 @@ __build_packet_message(struct nfnl_log_net *log, /* rcu_read_lock()ed by nf_hook_thresh or * nf_log_packet. */ - nla_put_be32(inst->skb, NFULA_IFINDEX_INDEV, - htonl(br_port_get_rcu(indev)->br->dev->ifindex))) + nflog_put_master_ifindex(inst->skb, NFULA_IFINDEX_INDEV, indev)) goto nla_put_failure; } else { int physinif; @@ -546,8 +562,7 @@ __build_packet_message(struct nfnl_log_net *log, /* rcu_read_lock()ed by nf_hook_thresh or * nf_log_packet. */ - nla_put_be32(inst->skb, NFULA_IFINDEX_OUTDEV, - htonl(br_port_get_rcu(outdev)->br->dev->ifindex))) + nflog_put_master_ifindex(inst->skb, NFULA_IFINDEX_OUTDEV, outdev)) goto nla_put_failure; } else { struct net_device *physoutdev; diff --git a/net/netfilter/nfnetlink_queue.c b/net/netfilter/nfnetlink_queue.c index 1944fba91b02..719f7b009a68 100644 --- a/net/netfilter/nfnetlink_queue.c +++ b/net/netfilter/nfnetlink_queue.c @@ -225,11 +225,76 @@ find_dequeue_entry(struct nfqnl_instance *queue, unsigned int id) return entry; } +/* return true if the entry has an unconfirmed conntrack attached that isn't owned by us + * exclusively. + */ +static bool nf_ct_drop_unconfirmed(const struct nf_queue_entry *entry, bool *is_unconfirmed) +{ +#if IS_ENABLED(CONFIG_NF_CONNTRACK) + struct nf_conn *ct = (void *)skb_nfct(entry->skb); + + if (!ct || nf_ct_is_confirmed(ct)) + return false; + + if (is_unconfirmed) + *is_unconfirmed = true; + + /* in some cases skb_clone() can occur after initial conntrack + * pickup, but conntrack assumes exclusive skb->_nfct ownership for + * unconfirmed entries. + * + * This happens for br_netfilter and with ip multicast routing. + * This can't be solved with serialization here because one clone + * could have been queued for local delivery or could be transmitted + * in parallel on another CPU. + */ + return refcount_read(&ct->ct_general.use) > 1; +#endif + return false; +} + +static bool nf_bridge_port_valid(const struct net_device *dev) +{ + if (!dev) + return true; + + return netif_is_bridge_port(dev); +} + +/* queued skbs leave rcu protection. We bump device refcount so that + * the device cannot go away. However, while packet was out the port + * could have been removed from the bridge. + * + * Ensure in+outdev are still part of a bridge at reinject time. + * + * The device rx_handler_data could even be pointing at data that is + * not a net_bridge_port structure. + */ +static bool nf_bridge_ports_valid(const struct nf_queue_entry *entry) +{ +#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER) + if (!nf_bridge_port_valid(entry->physin) || + !nf_bridge_port_valid(entry->physout)) + return false; +#endif + if (entry->state.pf != PF_BRIDGE) + return true; + + if (!nf_bridge_port_valid(entry->state.in) || + !nf_bridge_port_valid(entry->state.out)) + return false; + + return true; +} + static void nfqnl_reinject(struct nf_queue_entry *entry, unsigned int verdict) { const struct nf_ct_hook *ct_hook; int err; + if (!nf_bridge_ports_valid(entry)) + verdict = NF_DROP; + if (verdict == NF_ACCEPT || verdict == NF_REPEAT || verdict == NF_STOP) { @@ -242,6 +307,24 @@ static void nfqnl_reinject(struct nf_queue_entry *entry, unsigned int verdict) } rcu_read_unlock(); } + + if (verdict != NF_DROP && entry->nf_ct_is_unconfirmed) { + /* If first queued segment was already reinjected then + * there is a good chance the ct entry is now confirmed. + * + * Handle the rare cases: + * - out-of-order verdict + * - threaded userspace reinjecting in parallel + * - first segment was dropped + * + * In all of those cases we can't handle this packet + * because we can't be sure that another CPU won't modify + * nf_conn->ext in parallel which isn't allowed. + */ + if (nf_ct_drop_unconfirmed(entry, NULL)) + verdict = NF_DROP; + } + nf_reinject(entry, verdict); } @@ -400,6 +483,23 @@ static int nf_queue_checksum_help(struct sk_buff *entskb) return skb_checksum_help(entskb); } +#if IS_ENABLED(CONFIG_BRIDGE_NETFILTER) +static int nfqnl_put_master_ifindex(struct sk_buff *nlskb, int attr, + const struct net_device *dev) +{ + const struct net_device *upper; + + if (dev && !netif_is_bridge_port(dev)) + return 0; + + upper = netdev_master_upper_dev_get_rcu((struct net_device *)dev); + if (upper && nla_put_be32(nlskb, attr, htonl(upper->ifindex))) + return -EMSGSIZE; + + return 0; +} +#endif + static struct sk_buff * nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue, struct nf_queue_entry *entry, @@ -533,10 +633,7 @@ nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue, * netfilter_bridge) */ if (nla_put_be32(skb, NFQA_IFINDEX_PHYSINDEV, htonl(indev->ifindex)) || - /* this is the bridge group "brX" */ - /* rcu_read_lock()ed by __nf_queue */ - nla_put_be32(skb, NFQA_IFINDEX_INDEV, - htonl(br_port_get_rcu(indev)->br->dev->ifindex))) + nfqnl_put_master_ifindex(skb, NFQA_IFINDEX_INDEV, indev)) goto nla_put_failure; } else { int physinif; @@ -567,10 +664,7 @@ nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue, * netfilter_bridge) */ if (nla_put_be32(skb, NFQA_IFINDEX_PHYSOUTDEV, htonl(outdev->ifindex)) || - /* this is the bridge group "brX" */ - /* rcu_read_lock()ed by __nf_queue */ - nla_put_be32(skb, NFQA_IFINDEX_OUTDEV, - htonl(br_port_get_rcu(outdev)->br->dev->ifindex))) + nfqnl_put_master_ifindex(skb, NFQA_IFINDEX_OUTDEV, outdev)) goto nla_put_failure; } else { int physoutif; @@ -676,49 +770,6 @@ nfqnl_build_packet_message(struct net *net, struct nfqnl_instance *queue, return NULL; } -static bool nf_ct_drop_unconfirmed(const struct nf_queue_entry *entry) -{ -#if IS_ENABLED(CONFIG_NF_CONNTRACK) - static const unsigned long flags = IPS_CONFIRMED | IPS_DYING; - struct nf_conn *ct = (void *)skb_nfct(entry->skb); - unsigned long status; - unsigned int use; - - if (!ct) - return false; - - status = READ_ONCE(ct->status); - if ((status & flags) == IPS_DYING) - return true; - - if (status & IPS_CONFIRMED) - return false; - - /* in some cases skb_clone() can occur after initial conntrack - * pickup, but conntrack assumes exclusive skb->_nfct ownership for - * unconfirmed entries. - * - * This happens for br_netfilter and with ip multicast routing. - * We can't be solved with serialization here because one clone could - * have been queued for local delivery. - */ - use = refcount_read(&ct->ct_general.use); - if (likely(use == 1)) - return false; - - /* Can't decrement further? Exclusive ownership. */ - if (!refcount_dec_not_one(&ct->ct_general.use)) - return false; - - skb_set_nfct(entry->skb, 0); - /* No nf_ct_put(): we already decremented .use and it cannot - * drop down to 0. - */ - return true; -#endif - return false; -} - static int __nfqnl_enqueue_packet(struct net *net, struct nfqnl_instance *queue, struct nf_queue_entry *entry) @@ -735,9 +786,6 @@ __nfqnl_enqueue_packet(struct net *net, struct nfqnl_instance *queue, } spin_lock_bh(&queue->lock); - if (nf_ct_drop_unconfirmed(entry)) - goto err_out_free_nskb; - if (queue->queue_total >= queue->queue_maxlen) { if (queue->flags & NFQA_CFG_F_FAIL_OPEN) { failopen = 1; @@ -848,9 +896,10 @@ __nfqnl_enqueue_packet_gso(struct net *net, struct nfqnl_instance *queue, static int nfqnl_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum) { - unsigned int queued; - struct nfqnl_instance *queue; struct sk_buff *skb, *segs, *nskb; + bool ct_is_unconfirmed = false; + struct nfqnl_instance *queue; + unsigned int queued; int err = -ENOBUFS; struct net *net = entry->state.net; struct nfnl_queue_net *q = nfnl_queue_pernet(net); @@ -874,6 +923,15 @@ nfqnl_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum) break; } + /* Check if someone already holds another reference to + * unconfirmed ct. If so, we cannot queue the skb: + * concurrent modifications of nf_conn->ext are not + * allowed and we can't know if another CPU isn't + * processing the same nf_conn entry in parallel. + */ + if (nf_ct_drop_unconfirmed(entry, &ct_is_unconfirmed)) + return -EINVAL; + if (!skb_is_gso(skb) || ((queue->flags & NFQA_CFG_F_GSO) && !skb_is_gso_sctp(skb))) return __nfqnl_enqueue_packet(net, queue, entry); @@ -887,7 +945,23 @@ nfqnl_enqueue_packet(struct nf_queue_entry *entry, unsigned int queuenum) goto out_err; queued = 0; err = 0; + skb_list_walk_safe(segs, segs, nskb) { + if (ct_is_unconfirmed && queued > 0) { + /* skb_gso_segment() increments the ct refcount. + * This is a problem for unconfirmed (not in hash) + * entries, those can race when reinjections happen + * in parallel. + * + * Annotate this for all queued entries except the + * first one. + * + * As long as the first one is reinjected first it + * will do the confirmation for us. + */ + entry->nf_ct_is_unconfirmed = ct_is_unconfirmed; + } + if (err == 0) err = __nfqnl_enqueue_packet_gso(net, queue, segs, entry); diff --git a/net/netfilter/nft_ct.c b/net/netfilter/nft_ct.c index 1d1673f49ff7..3a4f5c4da0e3 100644 --- a/net/netfilter/nft_ct.c +++ b/net/netfilter/nft_ct.c @@ -879,14 +879,12 @@ static void nft_ct_timeout_obj_eval(struct nft_object *obj, } } - rcu_assign_pointer(timeout->timeout, priv->timeout); - /* adjust the timeout as per 'new' state. ct is unconfirmed, * so the current timestamp must not be added. */ values = nf_ct_timeout_data(timeout); if (values) - nf_ct_refresh(ct, pkt->skb, values[0]); + nf_ct_refresh(ct, values[0]); } static int nft_ct_timeout_obj_init(const struct nft_ctx *ctx, @@ -931,6 +929,7 @@ static int nft_ct_timeout_obj_init(const struct nft_ctx *ctx, timeout->l3num = l3num; timeout->l4proto = l4proto; + refcount_set(&timeout->refcnt, 1); ret = nf_ct_netns_get(ctx->net, ctx->family); if (ret < 0) @@ -951,10 +950,10 @@ static void nft_ct_timeout_obj_destroy(const struct nft_ctx *ctx, struct nft_ct_timeout_obj *priv = nft_obj_data(obj); struct nf_ct_timeout *timeout = priv->timeout; - nf_queue_nf_hook_drop(ctx->net); nf_ct_untimeout(ctx->net, timeout); nf_ct_netns_put(ctx->net, ctx->family); - kfree_rcu(priv->timeout, rcu); + if (refcount_dec_and_test(&timeout->refcnt)) + kfree_rcu(priv->timeout, rcu); } static int nft_ct_timeout_obj_dump(struct sk_buff *skb, @@ -1129,7 +1128,6 @@ static void nft_ct_helper_obj_eval(struct nft_object *obj, help = nf_ct_helper_ext_add(ct, GFP_ATOMIC); if (help) { rcu_assign_pointer(help->helper, to_assign); - set_bit(IPS_HELPER_BIT, &ct->status); if ((ct->status & IPS_NAT_MASK) && !nfct_seqadj(ct)) if (!nfct_seqadj_ext_add(ct)) @@ -1311,7 +1309,7 @@ static void nft_ct_expect_obj_eval(struct nft_object *obj, &ct->tuplehash[!dir].tuple.src.u3, &ct->tuplehash[!dir].tuple.dst.u3, priv->l4proto, NULL, &priv->dport); - exp->timeout.expires = jiffies + priv->timeout * HZ; + exp->timeout += priv->timeout * HZ; if (nf_ct_expect_related(exp, 0) != 0) regs->verdict.code = NF_DROP; diff --git a/net/netfilter/nft_dynset.c b/net/netfilter/nft_dynset.c index b1c6bb490d46..41a385cc334d 100644 --- a/net/netfilter/nft_dynset.c +++ b/net/netfilter/nft_dynset.c @@ -19,7 +19,8 @@ struct nft_dynset { u8 sreg_key; u8 sreg_data; bool invert; - bool expr; + bool expr:1, + override_exprs:1; u8 num_exprs; u64 timeout; struct nft_expr *expr_array[NFT_SET_EXPR_MAX]; @@ -254,6 +255,7 @@ static int nft_dynset_init(const struct nft_ctx *ctx, priv->num_exprs++; priv->expr_array[0] = dynset_expr; + priv->override_exprs = true; if (set->num_exprs > 1 || (set->num_exprs == 1 && @@ -286,6 +288,7 @@ static int nft_dynset_init(const struct nft_ctx *ctx, } priv->expr_array[i] = dynset_expr; priv->num_exprs++; + priv->override_exprs = true; if (set->num_exprs) { if (i >= set->num_exprs) { @@ -304,9 +307,8 @@ static int nft_dynset_init(const struct nft_ctx *ctx, goto err_expr_free; } } else if (set->num_exprs > 0) { - err = nft_set_elem_expr_clone(ctx, set, priv->expr_array); - if (err < 0) - return err; + for (i = 0; i < set->num_exprs; i++) + priv->expr_array[i] = set->exprs[i]; priv->num_exprs = set->num_exprs; } @@ -339,8 +341,10 @@ static int nft_dynset_init(const struct nft_ctx *ctx, return 0; err_expr_free: - for (i = 0; i < priv->num_exprs; i++) - nft_expr_destroy(ctx, priv->expr_array[i]); + if (priv->override_exprs) { + for (i = 0; i < priv->num_exprs; i++) + nft_expr_destroy(ctx, priv->expr_array[i]); + } return err; } @@ -367,9 +371,10 @@ static void nft_dynset_destroy(const struct nft_ctx *ctx, struct nft_dynset *priv = nft_expr_priv(expr); int i; - for (i = 0; i < priv->num_exprs; i++) - nft_expr_destroy(ctx, priv->expr_array[i]); - + if (priv->override_exprs) { + for (i = 0; i < priv->num_exprs; i++) + nft_expr_destroy(ctx, priv->expr_array[i]); + } nf_tables_destroy_set(ctx, priv->set); } @@ -393,7 +398,7 @@ static int nft_dynset_dump(struct sk_buff *skb, nf_jiffies64_to_msecs(priv->timeout), NFTA_DYNSET_PAD)) goto nla_put_failure; - if (priv->set->num_exprs == 0) { + if (priv->set->num_exprs == 0 || priv->override_exprs) { if (priv->num_exprs == 1) { if (nft_expr_dump(skb, NFTA_DYNSET_EXPR, priv->expr_array[0], reset)) diff --git a/net/netfilter/nft_flow_offload.c b/net/netfilter/nft_flow_offload.c index 2df9178d423a..b2d8e8237354 100644 --- a/net/netfilter/nft_flow_offload.c +++ b/net/netfilter/nft_flow_offload.c @@ -149,10 +149,11 @@ static void nft_dev_path_info(const struct net_device_path_stack *stack, info->num_encaps++; break; case DEV_PATH_BR_VLAN_UNTAG: - if (WARN_ON_ONCE(info->num_encaps-- == 0)) { + if (info->num_encaps == 0) { info->indev = NULL; break; } + info->num_encaps--; break; case DEV_PATH_BR_VLAN_KEEP: break; @@ -374,6 +375,7 @@ static void nft_flow_offload_eval(const struct nft_expr *expr, if (tcph) flow_offload_ct_tcp(ct); + __set_bit(NF_FLOW_HW_BIDIRECTIONAL, &flow->flags); ret = flow_offload_add(flowtable, flow); if (ret < 0) goto err_flow_add; @@ -416,6 +418,7 @@ static int nft_flow_offload_init(const struct nft_ctx *ctx, struct nft_flow_offload *priv = nft_expr_priv(expr); u8 genmask = nft_genmask_next(ctx->net); struct nft_flowtable *flowtable; + int err; if (!tb[NFTA_FLOW_TABLE_NAME]) return -EINVAL; @@ -430,7 +433,11 @@ static int nft_flow_offload_init(const struct nft_ctx *ctx, priv->flowtable = flowtable; - return nf_ct_netns_get(ctx->net, ctx->family); + err = nf_ct_netns_get(ctx->net, ctx->family); + if (err < 0) + nft_use_dec(&flowtable->use); + + return err; } static void nft_flow_offload_deactivate(const struct nft_ctx *ctx, diff --git a/net/netfilter/nft_lookup.c b/net/netfilter/nft_lookup.c index c56cc6f3f8fa..fed08c103ae0 100644 --- a/net/netfilter/nft_lookup.c +++ b/net/netfilter/nft_lookup.c @@ -65,14 +65,16 @@ void nft_lookup_eval(const struct nft_expr *expr, const struct net *net = nft_net(pkt); bool found; - found = nft_set_do_lookup(net, set, ®s->data[priv->sreg], &ext) ^ - priv->invert; + found = nft_set_do_lookup(net, set, ®s->data[priv->sreg], &ext); if (!found) { ext = nft_set_catchall_lookup(net, set); - if (!ext) { - regs->verdict.code = NFT_BREAK; - return; - } + found = !!ext; + } + + found ^= priv->invert; + if (!found) { + regs->verdict.code = NFT_BREAK; + return; } if (ext) { diff --git a/net/netfilter/nft_set_bitmap.c b/net/netfilter/nft_set_bitmap.c index cbf7f7825f1b..06b77100bef7 100644 --- a/net/netfilter/nft_set_bitmap.c +++ b/net/netfilter/nft_set_bitmap.c @@ -123,7 +123,7 @@ static void *nft_bitmap_get(const struct net *net, const struct nft_set *set, static int nft_bitmap_insert(const struct net *net, const struct nft_set *set, const struct nft_set_elem *elem, - struct nft_set_ext **ext) + void **elem_priv) { struct nft_bitmap *priv = nft_set_priv(set); struct nft_bitmap_elem *new = elem->priv, *be; @@ -132,7 +132,7 @@ static int nft_bitmap_insert(const struct net *net, const struct nft_set *set, be = nft_bitmap_elem_find(set, new, genmask); if (be) { - *ext = &be->ext; + *elem_priv = be; return -EEXIST; } diff --git a/net/netfilter/nft_set_hash.c b/net/netfilter/nft_set_hash.c index 5a74ee4b7dfb..60c0d72fb98b 100644 --- a/net/netfilter/nft_set_hash.c +++ b/net/netfilter/nft_set_hash.c @@ -168,7 +168,7 @@ static bool nft_rhash_update(struct nft_set *set, const u32 *key, static int nft_rhash_insert(const struct net *net, const struct nft_set *set, const struct nft_set_elem *elem, - struct nft_set_ext **ext) + void **elem_priv) { struct nft_rhash *priv = nft_set_priv(set); struct nft_rhash_elem *he = elem->priv; @@ -185,7 +185,7 @@ static int nft_rhash_insert(const struct net *net, const struct nft_set *set, if (IS_ERR(prev)) return PTR_ERR(prev); if (prev) { - *ext = &prev->ext; + *elem_priv = prev; return -EEXIST; } return 0; @@ -588,7 +588,7 @@ static u32 nft_jhash(const struct nft_set *set, const struct nft_hash *priv, static int nft_hash_insert(const struct net *net, const struct nft_set *set, const struct nft_set_elem *elem, - struct nft_set_ext **ext) + void **elem_priv) { struct nft_hash_elem *this = elem->priv, *he; struct nft_hash *priv = nft_set_priv(set); @@ -600,7 +600,7 @@ static int nft_hash_insert(const struct net *net, const struct nft_set *set, if (!memcmp(nft_set_ext_key(&this->ext), nft_set_ext_key(&he->ext), set->klen) && nft_set_elem_active(&he->ext, genmask)) { - *ext = &he->ext; + *elem_priv = he; return -EEXIST; } } diff --git a/net/netfilter/nft_set_pipapo.c b/net/netfilter/nft_set_pipapo.c index 7c8d28a031ad..cfdea0e65af6 100644 --- a/net/netfilter/nft_set_pipapo.c +++ b/net/netfilter/nft_set_pipapo.c @@ -1260,13 +1260,13 @@ static struct nft_pipapo_match *pipapo_maybe_clone(const struct nft_set *set) * @net: Network namespace * @set: nftables API set representation * @elem: nftables API element representation containing key data - * @ext2: Filled with pointer to &struct nft_set_ext in inserted element + * @elem_priv: Filled with pointer to private element data * * Return: 0 on success, error pointer on failure. */ static int nft_pipapo_insert(const struct net *net, const struct nft_set *set, const struct nft_set_elem *elem, - struct nft_set_ext **ext2) + void **elem_priv) { const struct nft_set_ext *ext = nft_set_elem_ext(set, elem->priv); union nft_pipapo_map_bucket rulemap[NFT_PIPAPO_MAX_FIELDS]; @@ -1301,7 +1301,7 @@ static int nft_pipapo_insert(const struct net *net, const struct nft_set *set, if (!memcmp(start, dup_key->data, sizeof(*dup_key->data)) && !memcmp(end, dup_end->data, sizeof(*dup_end->data))) { - *ext2 = &dup->ext; + *elem_priv = dup; return -EEXIST; } @@ -1317,7 +1317,7 @@ static int nft_pipapo_insert(const struct net *net, const struct nft_set *set, if (PTR_ERR(dup) != -ENOENT) { if (IS_ERR(dup)) return PTR_ERR(dup); - *ext2 = &dup->ext; + *elem_priv = dup; return -ENOTEMPTY; } @@ -1379,7 +1379,7 @@ static int nft_pipapo_insert(const struct net *net, const struct nft_set *set, put_cpu_ptr(m->scratch); } - *ext2 = &e->ext; + *elem_priv = e; pipapo_map(m, rulemap, e); @@ -1465,7 +1465,7 @@ static struct nft_pipapo_match *pipapo_clone(struct nft_pipapo_match *old) goto out_mt; dst->mt = kvmalloc_array(src->rules, sizeof(*src->mt), - GFP_KERNEL); + GFP_KERNEL_ACCOUNT); if (!dst->mt) goto out_mt; diff --git a/net/netfilter/nft_set_rbtree.c b/net/netfilter/nft_set_rbtree.c index 91d677b692f7..4ea4b65a4d79 100644 --- a/net/netfilter/nft_set_rbtree.c +++ b/net/netfilter/nft_set_rbtree.c @@ -319,7 +319,7 @@ static struct nft_rbtree_elem *nft_rbtree_prev_active(struct nft_rbtree_elem *rb static int __nft_rbtree_insert(const struct net *net, const struct nft_set *set, struct nft_rbtree_elem *new, - struct nft_set_ext **ext) + void **elem_priv) { struct nft_rbtree_elem *rbe, *rbe_le = NULL, *rbe_ge = NULL, *rbe_prev; struct rb_node *node, *next, *parent, **p, *first = NULL; @@ -437,7 +437,7 @@ static int __nft_rbtree_insert(const struct net *net, const struct nft_set *set, */ if (rbe_ge && !nft_rbtree_cmp(set, new, rbe_ge) && nft_rbtree_interval_start(rbe_ge) == nft_rbtree_interval_start(new)) { - *ext = &rbe_ge->ext; + *elem_priv = rbe_ge; return -EEXIST; } @@ -446,7 +446,7 @@ static int __nft_rbtree_insert(const struct net *net, const struct nft_set *set, */ if (rbe_le && !nft_rbtree_cmp(set, new, rbe_le) && nft_rbtree_interval_end(rbe_le) == nft_rbtree_interval_end(new)) { - *ext = &rbe_le->ext; + *elem_priv = rbe_le; return -EEXIST; } @@ -506,7 +506,7 @@ static int __nft_rbtree_insert(const struct net *net, const struct nft_set *set, static int nft_rbtree_insert(const struct net *net, const struct nft_set *set, const struct nft_set_elem *elem, - struct nft_set_ext **ext) + void **elem_priv) { struct nft_rbtree *priv = nft_set_priv(set); struct nft_rbtree_elem *rbe = elem->priv; @@ -520,7 +520,7 @@ static int nft_rbtree_insert(const struct net *net, const struct nft_set *set, write_lock_bh(&priv->lock); write_seqcount_begin(&priv->count); - err = __nft_rbtree_insert(net, set, rbe, ext); + err = __nft_rbtree_insert(net, set, rbe, elem_priv); write_seqcount_end(&priv->count); write_unlock_bh(&priv->lock); } while (err == -EAGAIN); diff --git a/net/netfilter/x_tables.c b/net/netfilter/x_tables.c index c1ab85fb8c46..a5af6082d99a 100644 --- a/net/netfilter/x_tables.c +++ b/net/netfilter/x_tables.c @@ -1403,11 +1403,9 @@ struct xt_counters *xt_counters_alloc(unsigned int counters) } EXPORT_SYMBOL(xt_counters_alloc); -struct xt_table_info * -xt_replace_table(struct xt_table *table, - unsigned int num_counters, - struct xt_table_info *newinfo, - int *error) +static struct xt_table_info * +do_replace_table(struct xt_table *table, unsigned int num_counters, + struct xt_table_info *newinfo, int *error) { struct xt_table_info *private; unsigned int cpu; @@ -1462,30 +1460,54 @@ xt_replace_table(struct xt_table *table, } } - audit_log_nfcfg(table->name, table->af, private->number, - !private->number ? AUDIT_XT_OP_REGISTER : - AUDIT_XT_OP_REPLACE, - GFP_KERNEL); + return private; +} + +struct xt_table_info * +xt_replace_table(struct xt_table *table, unsigned int num_counters, + struct xt_table_info *newinfo, + int *error) +{ + struct xt_table_info *private; + + private = do_replace_table(table, num_counters, newinfo, error); + if (private) + audit_log_nfcfg(table->name, table->af, private->number, + AUDIT_XT_OP_REPLACE, + GFP_KERNEL); + return private; } EXPORT_SYMBOL_GPL(xt_replace_table); struct xt_table *xt_register_table(struct net *net, const struct xt_table *input_table, + const struct nf_hook_ops *template_ops, struct xt_table_info *bootstrap, struct xt_table_info *newinfo) { struct xt_pernet *xt_net = net_generic(net, xt_pernet_id); + struct xt_table *t, *table = NULL; + struct nf_hook_ops *ops = NULL; struct xt_table_info *private; - struct xt_table *t, *table; - int ret; + unsigned int num_ops; + int ret = -EINVAL; + + num_ops = hweight32(input_table->valid_hooks); + if (num_ops == 0) + goto out; + + ret = -ENOMEM; + if (template_ops) { + ops = kmemdup_array(template_ops, num_ops, sizeof(*ops), GFP_KERNEL); + if (!ops) + goto out; + } /* Don't add one object to multiple lists. */ table = kmemdup(input_table, sizeof(struct xt_table), GFP_KERNEL); - if (!table) { - ret = -ENOMEM; + if (!table) goto out; - } mutex_lock(&xt[table->af].mutex); /* Don't autoload: we'd eat our tail... */ @@ -1499,7 +1521,7 @@ struct xt_table *xt_register_table(struct net *net, /* Simplifies replace_table code. */ table->private = bootstrap; - if (!xt_replace_table(table, 0, newinfo, &ret)) + if (!do_replace_table(table, 0, newinfo, &ret)) goto unlock; private = table->private; @@ -1508,14 +1530,37 @@ struct xt_table *xt_register_table(struct net *net, /* save number of initial entries */ private->initial_entries = private->number; + if (ops) { + int i; + + for (i = 0; i < num_ops; i++) + ops[i].priv = table; + + ret = nf_register_net_hooks(net, ops, num_ops); + if (ret != 0) { + mutex_unlock(&xt[table->af].mutex); + /* nf_register_net_hooks() might have published a + * base chain before internal error unwind. + */ + synchronize_rcu(); + goto out; + } + + table->ops = ops; + } + + audit_log_nfcfg(table->name, table->af, private->number, + AUDIT_XT_OP_REGISTER, GFP_KERNEL); + list_add(&table->list, &xt_net->tables[table->af]); mutex_unlock(&xt[table->af].mutex); return table; unlock: mutex_unlock(&xt[table->af].mutex); - kfree(table); out: + kfree(table); + kfree(ops); return ERR_PTR(ret); } EXPORT_SYMBOL_GPL(xt_register_table); diff --git a/net/netfilter/xt_CT.c b/net/netfilter/xt_CT.c index 498f5871c84a..90a295cf5ec6 100644 --- a/net/netfilter/xt_CT.c +++ b/net/netfilter/xt_CT.c @@ -284,7 +284,7 @@ static void xt_ct_tg_destroy(const struct xt_tgdtor_param *par, struct nf_conn_help *help; if (ct) { - if (info->helper[0] || info->timeout[0]) + if (info->helper[0]) nf_queue_nf_hook_drop(par->net); help = nfct_help(ct); diff --git a/net/netlink/genetlink.c b/net/netlink/genetlink.c index 015b14b852f8..563ede127b6b 100644 --- a/net/netlink/genetlink.c +++ b/net/netlink/genetlink.c @@ -1368,6 +1368,7 @@ struct ctrl_dump_policy_ctx { struct netlink_policy_dump_state *state; const struct genl_family *rt; struct genl_op_iter *op_iter; + struct module *owner; u32 op; u16 fam_id; u8 dump_map:1, @@ -1410,6 +1411,9 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb) return -ENOENT; ctx->rt = rt; + ctx->owner = rt->module; + if (!try_module_get(ctx->owner)) + return -ENOENT; if (tb[CTRL_ATTR_OP]) { struct genl_split_ops doit, dump; @@ -1420,7 +1424,7 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb) err = genl_get_cmd_both(ctx->op, rt, &doit, &dump); if (err) { NL_SET_BAD_ATTR(cb->extack, tb[CTRL_ATTR_OP]); - return err; + goto err_put_owner; } if (doit.policy) { @@ -1438,16 +1442,20 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb) goto err_free_state; } - if (!ctx->state) - return -ENODATA; + if (!ctx->state) { + err = -ENODATA; + goto err_put_owner; + } ctx->dump_map = 1; return 0; } ctx->op_iter = kmalloc(sizeof(*ctx->op_iter), GFP_KERNEL); - if (!ctx->op_iter) - return -ENOMEM; + if (!ctx->op_iter) { + err = -ENOMEM; + goto err_put_owner; + } genl_op_iter_init(rt, ctx->op_iter); ctx->dump_map = genl_op_iter_next(ctx->op_iter); @@ -1479,6 +1487,8 @@ static int ctrl_dumppolicy_start(struct netlink_callback *cb) netlink_policy_dump_free(ctx->state); err_free_op_iter: kfree(ctx->op_iter); +err_put_owner: + module_put(ctx->owner); return err; } @@ -1617,6 +1627,7 @@ static int ctrl_dumppolicy_done(struct netlink_callback *cb) kfree(ctx->op_iter); netlink_policy_dump_free(ctx->state); + module_put(ctx->owner); return 0; } diff --git a/net/openvswitch/actions.c b/net/openvswitch/actions.c index 4cdf0f09d488..c525eb456609 100644 --- a/net/openvswitch/actions.c +++ b/net/openvswitch/actions.c @@ -382,7 +382,12 @@ static void set_ip_addr(struct sk_buff *skb, struct iphdr *nh, static void update_ipv6_checksum(struct sk_buff *skb, u8 l4_proto, __be32 addr[4], const __be32 new_addr[4]) { - int transport_len = skb->len - skb_transport_offset(skb); + int transport_len; + + if (l4_proto == NEXTHDR_FRAGMENT) + return; + + transport_len = skb->len - skb_transport_offset(skb); if (l4_proto == NEXTHDR_TCP) { if (likely(transport_len >= sizeof(struct tcphdr))) diff --git a/net/openvswitch/conntrack.c b/net/openvswitch/conntrack.c index 45a7ab66868f..cdbd6ad2b8d7 100644 --- a/net/openvswitch/conntrack.c +++ b/net/openvswitch/conntrack.c @@ -679,6 +679,8 @@ static int ovs_ct_nat(struct net *net, struct sw_flow_key *key, action |= BIT(NF_NAT_MANIP_DST); err = nf_ct_nat(skb, ct, ctinfo, &action, &info->range, info->commit); + if (err != NF_ACCEPT) + return err; if (action & BIT(NF_NAT_MANIP_SRC)) ovs_nat_update_key(key, skb, NF_NAT_MANIP_SRC); @@ -697,6 +699,22 @@ static int ovs_ct_nat(struct net *net, struct sw_flow_key *key, } #endif +static int verdict_to_errno(unsigned int verdict) +{ + switch (verdict & NF_VERDICT_MASK) { + case NF_ACCEPT: + return 0; + case NF_DROP: + return -EINVAL; + case NF_STOLEN: + return -EINPROGRESS; + default: + break; + } + + return -EINVAL; +} + /* Pass 'skb' through conntrack in 'net', using zone configured in 'info', if * not done already. Update key with new CT state after passing the packet * through conntrack. @@ -747,7 +765,7 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key, err = nf_conntrack_in(skb, &state); if (err != NF_ACCEPT) - return -ENOENT; + return verdict_to_errno(err); /* Clear CT state NAT flags to mark that we have not yet done * NAT after the nf_conntrack_in() call. We can actually clear @@ -761,8 +779,6 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key, ct = nf_ct_get(skb, &ctinfo); if (ct) { - bool add_helper = false; - /* Packets starting a new connection must be NATted before the * helper, so that the helper knows about the NAT. We enforce * this by delaying both NAT and helper calls for unconfirmed @@ -774,9 +790,12 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key, * the key->ct_state. */ if (info->nat && !(key->ct_state & OVS_CS_F_NAT_MASK) && - (nf_ct_is_confirmed(ct) || info->commit) && - ovs_ct_nat(net, key, info, skb, ct, ctinfo) != NF_ACCEPT) { - return -EINVAL; + (nf_ct_is_confirmed(ct) || info->commit)) { + int err = ovs_ct_nat(net, key, info, skb, ct, ctinfo); + + err = verdict_to_errno(err); + if (err) + return err; } /* Userspace may decide to perform a ct lookup without a helper @@ -791,7 +810,6 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key, GFP_ATOMIC); if (err) return err; - add_helper = true; /* helper installed, add seqadj if NAT is required */ if (info->nat && !nfct_seqadj(ct)) { @@ -800,16 +818,19 @@ static int __ovs_ct_lookup(struct net *net, struct sw_flow_key *key, } } - /* Call the helper only if: - * - nf_conntrack_in() was executed above ("!cached") or a - * helper was just attached ("add_helper") for a confirmed - * connection, or - * - When committing an unconfirmed connection. + /* Call the helper only if nf_conntrack_in() was executed + * above ("!cached"). + * + * For unconfirmed connections it will be called later during + * commit as we need to have all the other extensions allocated + * before the call. */ - if ((nf_ct_is_confirmed(ct) ? !cached || add_helper : - info->commit) && - nf_ct_helper(skb, ct, ctinfo, info->family) != NF_ACCEPT) { - return -EINVAL; + if (nf_ct_is_confirmed(ct) && !cached) { + int err = nf_ct_helper(skb, ct, ctinfo, info->family); + + err = verdict_to_errno(err); + if (err) + return err; } if (nf_ct_protonum(ct) == IPPROTO_TCP && @@ -1007,6 +1028,14 @@ static int ovs_ct_commit(struct net *net, struct sw_flow_key *key, return err; nf_conn_act_ct_ext_add(skb, ct, ctinfo); + + /* Call the helpers now. We couldn't do this before as + * all the extensions must be allocated before the call. + */ + err = nf_ct_helper(skb, ct, ctinfo, info->family); + err = verdict_to_errno(err); + if (err) + return err; } else if (IS_ENABLED(CONFIG_NF_CONNTRACK_LABELS) && labels_nonzero(&info->labels.mask)) { err = ovs_ct_set_labels(ct, key, &info->labels.value, @@ -1017,10 +1046,9 @@ static int ovs_ct_commit(struct net *net, struct sw_flow_key *key, /* This will take care of sending queued events even if the connection * is already confirmed. */ - if (nf_conntrack_confirm(skb) != NF_ACCEPT) - return -EINVAL; + err = nf_conntrack_confirm(skb); - return 0; + return verdict_to_errno(err); } /* Returns 0 on success, -EINPROGRESS if 'skb' is stolen, or other nonzero @@ -1055,6 +1083,10 @@ int ovs_ct_execute(struct net *net, struct sk_buff *skb, else err = ovs_ct_lookup(net, key, info, skb); + /* conntrack core returned NF_STOLEN */ + if (err == -EINPROGRESS) + return err; + skb_push_rcsum(skb, nh_ofs); if (err) ovs_kfree_skb_reason(skb, OVS_DROP_CONNTRACK); diff --git a/net/openvswitch/datapath.c b/net/openvswitch/datapath.c index a43d3af3218f..e786a7576907 100644 --- a/net/openvswitch/datapath.c +++ b/net/openvswitch/datapath.c @@ -1500,6 +1500,12 @@ static int ovs_flow_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb) return -ENODEV; } + /* + * Not needed for safety. Stops every spin_unlock_bh() in + * ovs_flow_stats_get() from running the softirq backlog. + */ + local_bh_disable(); + ti = rcu_dereference(dp->table.ti); for (;;) { struct sw_flow *flow; @@ -1520,6 +1526,8 @@ static int ovs_flow_cmd_dump(struct sk_buff *skb, struct netlink_callback *cb) cb->args[0] = bucket; cb->args[1] = obj; } + + local_bh_enable(); rcu_read_unlock(); return skb->len; } diff --git a/net/packet/af_packet.c b/net/packet/af_packet.c index f268a0c7ed1f..baf39b937fb5 100644 --- a/net/packet/af_packet.c +++ b/net/packet/af_packet.c @@ -1963,7 +1963,8 @@ static int packet_rcv_spkt(struct sk_buff *skb, struct net_device *dev, spkt = &PACKET_SKB_CB(skb)->sa.pkt; - skb_push(skb, skb->data - skb_mac_header(skb)); + if (dev_has_header(dev)) + skb_push(skb, skb->data - skb_mac_header(skb)); /* * The SOCK_PACKET socket receives _all_ frames. @@ -2917,6 +2918,9 @@ static int tpacket_snd(struct packet_sock *po, struct msghdr *msg) vnet_hdr.hdr_len); has_vnet_hdr = true; } + /* Prevent the byte count from wrapping into a negative errno */ + if (unlikely(tp_len > INT_MAX - len_sum)) + break; copylen = max_t(int, copylen, hard_header_len); skb = sock_alloc_send_skb(&po->sk, hlen + tlen + sizeof(struct sockaddr_ll) + diff --git a/net/phonet/socket.c b/net/phonet/socket.c index 1018340d89a7..0b93de85802c 100644 --- a/net/phonet/socket.c +++ b/net/phonet/socket.c @@ -208,9 +208,15 @@ static int pn_socket_autobind(struct socket *sock) sa.spn_family = AF_PHONET; err = pn_socket_bind(sock, (struct sockaddr *)&sa, sizeof(struct sockaddr_pn)); - if (err != -EINVAL) + /* + * pn_socket_bind() also returns -EINVAL when sk_state != TCP_CLOSE + * without a prior bind, so -EINVAL alone is not sufficient to infer + * that the socket was already bound. Only treat it as "already + * bound" when the port is non-zero; otherwise propagate the error + * instead of crashing the kernel. + */ + if (err != -EINVAL || unlikely(!pn_port(pn_sk(sock->sk)->sobject))) return err; - BUG_ON(!pn_port(pn_sk(sock->sk)->sobject)); return 0; /* socket was already bound */ } diff --git a/net/qrtr/mhi.c b/net/qrtr/mhi.c index 9ced13c0627a..aa9cd496b549 100644 --- a/net/qrtr/mhi.c +++ b/net/qrtr/mhi.c @@ -24,13 +24,25 @@ static void qcom_mhi_qrtr_dl_callback(struct mhi_device *mhi_dev, struct qrtr_mhi_dev *qdev = dev_get_drvdata(&mhi_dev->dev); int rc; - if (!qdev || mhi_res->transaction_status) + if (!qdev || (mhi_res->transaction_status && mhi_res->transaction_status != -ENOTCONN)) return; + /* Channel got reset. So just free the buffer */ + if (mhi_res->transaction_status == -ENOTCONN) { + devm_kfree(&mhi_dev->dev, mhi_res->buf_addr); + return; + } + rc = qrtr_endpoint_post(&qdev->ep, mhi_res->buf_addr, mhi_res->bytes_xferd); if (rc == -EINVAL) dev_err(qdev->dev, "invalid ipcrouter packet\n"); + + /* Done with the buffer, now recycle it for future use */ + rc = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, mhi_res->buf_addr, + mhi_dev->mhi_cntrl->buffer_len, MHI_EOT); + if (rc) + dev_err(&mhi_dev->dev, "Failed to recycle the buffer: %d\n", rc); } /* From QRTR to MHI */ @@ -72,6 +84,29 @@ static int qcom_mhi_qrtr_send(struct qrtr_endpoint *ep, struct sk_buff *skb) return rc; } +static int qcom_mhi_qrtr_queue_dl_buffers(struct mhi_device *mhi_dev) +{ + u32 free_desc; + void *buf; + int ret; + + free_desc = mhi_get_free_desc_count(mhi_dev, DMA_FROM_DEVICE); + while (free_desc--) { + buf = devm_kmalloc(&mhi_dev->dev, mhi_dev->mhi_cntrl->buffer_len, GFP_KERNEL); + if (!buf) + return -ENOMEM; + + ret = mhi_queue_buf(mhi_dev, DMA_FROM_DEVICE, buf, mhi_dev->mhi_cntrl->buffer_len, + MHI_EOT); + if (ret) { + dev_err(&mhi_dev->dev, "Failed to queue buffer: %d\n", ret); + return ret; + } + } + + return 0; +} + static int qcom_mhi_qrtr_probe(struct mhi_device *mhi_dev, const struct mhi_device_id *id) { @@ -87,20 +122,30 @@ static int qcom_mhi_qrtr_probe(struct mhi_device *mhi_dev, qdev->ep.xmit = qcom_mhi_qrtr_send; dev_set_drvdata(&mhi_dev->dev, qdev); - rc = qrtr_endpoint_register(&qdev->ep, QRTR_EP_NID_AUTO); - if (rc) - return rc; /* start channels */ - rc = mhi_prepare_for_transfer_autoqueue(mhi_dev); - if (rc) { - qrtr_endpoint_unregister(&qdev->ep); + rc = mhi_prepare_for_transfer(mhi_dev); + if (rc) return rc; - } + + rc = qrtr_endpoint_register(&qdev->ep, QRTR_EP_NID_AUTO); + if (rc) + goto err_unprepare; + + rc = qcom_mhi_qrtr_queue_dl_buffers(mhi_dev); + if (rc) + goto err_unregister; dev_dbg(qdev->dev, "Qualcomm MHI QRTR driver probed\n"); return 0; + +err_unregister: + qrtr_endpoint_unregister(&qdev->ep); +err_unprepare: + mhi_unprepare_from_transfer(mhi_dev); + + return rc; } static void qcom_mhi_qrtr_remove(struct mhi_device *mhi_dev) diff --git a/net/rds/tcp.c b/net/rds/tcp.c index 8f3d09c541db..69ebc4334cc2 100644 --- a/net/rds/tcp.c +++ b/net/rds/tcp.c @@ -53,8 +53,10 @@ static atomic_t rds_tcp_unloading = ATOMIC_INIT(0); static struct kmem_cache *rds_tcp_conn_slab; -static int rds_tcp_skbuf_handler(struct ctl_table *ctl, int write, - void *buffer, size_t *lenp, loff_t *fpos); +static int rds_tcp_sndbuf_handler(struct ctl_table *ctl, int write, + void *buffer, size_t *lenp, loff_t *fpos); +static int rds_tcp_rcvbuf_handler(struct ctl_table *ctl, int write, + void *buffer, size_t *lenp, loff_t *fpos); static int rds_tcp_min_sndbuf = SOCK_MIN_SNDBUF; static int rds_tcp_min_rcvbuf = SOCK_MIN_RCVBUF; @@ -66,7 +68,7 @@ static struct ctl_table rds_tcp_sysctl_table[] = { /* data is per-net pointer */ .maxlen = sizeof(int), .mode = 0644, - .proc_handler = rds_tcp_skbuf_handler, + .proc_handler = rds_tcp_sndbuf_handler, .extra1 = &rds_tcp_min_sndbuf, }, #define RDS_TCP_RCVBUF 1 @@ -75,7 +77,7 @@ static struct ctl_table rds_tcp_sysctl_table[] = { /* data is per-net pointer */ .maxlen = sizeof(int), .mode = 0644, - .proc_handler = rds_tcp_skbuf_handler, + .proc_handler = rds_tcp_rcvbuf_handler, .extra1 = &rds_tcp_min_rcvbuf, }, { } @@ -231,16 +233,19 @@ void rds_tcp_set_callbacks(struct socket *sock, struct rds_conn_path *cp) rdsdebug("setting sock %p callbacks to tc %p\n", sock, tc); write_lock_bh(&sock->sk->sk_callback_lock); - /* done under the callback_lock to serialize with write_space */ + /* done under the callback_lock to serialize with write_space. + * Set t_sock inside rds_tcp_tc_list_lock so readers walking + * rds_tcp_tc_list under the same lock cannot observe an + * entry whose t_sock is NULL. + */ spin_lock(&rds_tcp_tc_list_lock); + tc->t_sock = sock; list_add_tail(&tc->t_list_item, &rds_tcp_tc_list); spin_unlock(&rds_tcp_tc_list_lock); /* accepted sockets need our listen data ready undone */ if (sock->sk->sk_data_ready == rds_tcp_listen_data_ready) sock->sk->sk_data_ready = sock->sk->sk_user_data; - - tc->t_sock = sock; tc->t_cpath = cp; tc->t_orig_data_ready = sock->sk->sk_data_ready; tc->t_orig_write_space = sock->sk->sk_write_space; @@ -755,10 +760,10 @@ static void rds_tcp_sysctl_reset(struct net *net) spin_unlock_irq(&rds_tcp_conn_lock); } -static int rds_tcp_skbuf_handler(struct ctl_table *ctl, int write, +static int rds_tcp_skbuf_handler(struct rds_tcp_net *rtn, + struct ctl_table *ctl, int write, void *buffer, size_t *lenp, loff_t *fpos) { - struct net *net = current->nsproxy->net_ns; int err; err = proc_dointvec_minmax(ctl, write, buffer, lenp, fpos); @@ -767,11 +772,34 @@ static int rds_tcp_skbuf_handler(struct ctl_table *ctl, int write, *(int *)(ctl->extra1)); return err; } - if (write) + + if (write && rtn->rds_tcp_listen_sock && rtn->rds_tcp_listen_sock->sk) { + struct net *net = sock_net(rtn->rds_tcp_listen_sock->sk); + rds_tcp_sysctl_reset(net); + } + return 0; } +static int rds_tcp_sndbuf_handler(struct ctl_table *ctl, int write, + void *buffer, size_t *lenp, loff_t *fpos) +{ + struct rds_tcp_net *rtn = container_of(ctl->data, struct rds_tcp_net, + sndbuf_size); + + return rds_tcp_skbuf_handler(rtn, ctl, write, buffer, lenp, fpos); +} + +static int rds_tcp_rcvbuf_handler(struct ctl_table *ctl, int write, + void *buffer, size_t *lenp, loff_t *fpos) +{ + struct rds_tcp_net *rtn = container_of(ctl->data, struct rds_tcp_net, + rcvbuf_size); + + return rds_tcp_skbuf_handler(rtn, ctl, write, buffer, lenp, fpos); +} + static void rds_tcp_exit(void) { rds_tcp_set_unloading(); diff --git a/net/rxrpc/proc.c b/net/rxrpc/proc.c index 55a95f064df0..f698796976c5 100644 --- a/net/rxrpc/proc.c +++ b/net/rxrpc/proc.c @@ -10,6 +10,10 @@ #include #include "ar-internal.h" +#define RXRPC_PROC_ADDRBUF_SIZE \ + (sizeof("[xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:255.255.255.255]") + \ + sizeof(":12345")) + static const char *const rxrpc_conn_states[RXRPC_CONN__NR_STATES] = { [RXRPC_CONN_UNUSED] = "Unused ", [RXRPC_CONN_CLIENT_UNSECURED] = "ClUnsec ", @@ -54,7 +58,7 @@ static int rxrpc_call_seq_show(struct seq_file *seq, void *v) enum rxrpc_call_state state; unsigned long timeout = 0; rxrpc_seq_t acks_hard_ack; - char lbuff[50], rbuff[50]; + char lbuff[RXRPC_PROC_ADDRBUF_SIZE], rbuff[RXRPC_PROC_ADDRBUF_SIZE]; if (v == &rxnet->calls) { seq_puts(seq, @@ -69,11 +73,11 @@ static int rxrpc_call_seq_show(struct seq_file *seq, void *v) local = call->local; if (local) - sprintf(lbuff, "%pISpc", &local->srx.transport); + scnprintf(lbuff, sizeof(lbuff), "%pISpc", &local->srx.transport); else strcpy(lbuff, "no_local"); - sprintf(rbuff, "%pISpc", &call->dest_srx.transport); + scnprintf(rbuff, sizeof(rbuff), "%pISpc", &call->dest_srx.transport); state = rxrpc_call_state(call); if (state != RXRPC_CALL_SERVER_PREALLOC) { @@ -144,7 +148,7 @@ static int rxrpc_connection_seq_show(struct seq_file *seq, void *v) struct rxrpc_connection *conn; struct rxrpc_net *rxnet = rxrpc_net(seq_file_net(seq)); const char *state; - char lbuff[50], rbuff[50]; + char lbuff[RXRPC_PROC_ADDRBUF_SIZE], rbuff[RXRPC_PROC_ADDRBUF_SIZE]; if (v == &rxnet->conn_proc_list) { seq_puts(seq, @@ -163,8 +167,8 @@ static int rxrpc_connection_seq_show(struct seq_file *seq, void *v) goto print; } - sprintf(lbuff, "%pISpc", &conn->local->srx.transport); - sprintf(rbuff, "%pISpc", &conn->peer->srx.transport); + scnprintf(lbuff, sizeof(lbuff), "%pISpc", &conn->local->srx.transport); + scnprintf(rbuff, sizeof(rbuff), "%pISpc", &conn->peer->srx.transport); print: state = rxrpc_is_conn_aborted(conn) ? rxrpc_call_completions[conn->completion] : @@ -205,7 +209,7 @@ static int rxrpc_peer_seq_show(struct seq_file *seq, void *v) { struct rxrpc_peer *peer; time64_t now; - char lbuff[50], rbuff[50]; + char lbuff[RXRPC_PROC_ADDRBUF_SIZE], rbuff[RXRPC_PROC_ADDRBUF_SIZE]; if (v == SEQ_START_TOKEN) { seq_puts(seq, @@ -218,9 +222,9 @@ static int rxrpc_peer_seq_show(struct seq_file *seq, void *v) peer = list_entry(v, struct rxrpc_peer, hash_link); - sprintf(lbuff, "%pISpc", &peer->local->srx.transport); + scnprintf(lbuff, sizeof(lbuff), "%pISpc", &peer->local->srx.transport); - sprintf(rbuff, "%pISpc", &peer->srx.transport); + scnprintf(rbuff, sizeof(rbuff), "%pISpc", &peer->srx.transport); now = ktime_get_seconds(); seq_printf(seq, @@ -330,7 +334,7 @@ const struct seq_operations rxrpc_peer_seq_ops = { static int rxrpc_local_seq_show(struct seq_file *seq, void *v) { struct rxrpc_local *local; - char lbuff[50]; + char lbuff[RXRPC_PROC_ADDRBUF_SIZE]; if (v == SEQ_START_TOKEN) { seq_puts(seq, @@ -341,7 +345,7 @@ static int rxrpc_local_seq_show(struct seq_file *seq, void *v) local = hlist_entry(v, struct rxrpc_local, link); - sprintf(lbuff, "%pISpc", &local->srx.transport); + scnprintf(lbuff, sizeof(lbuff), "%pISpc", &local->srx.transport); seq_printf(seq, "UDP %-47.47s %3u %3u %3u\n", diff --git a/net/sched/act_api.c b/net/sched/act_api.c index b00d0b0215f8..2954324f1230 100644 --- a/net/sched/act_api.c +++ b/net/sched/act_api.c @@ -1144,19 +1144,14 @@ EXPORT_SYMBOL(tcf_action_exec); int tcf_action_destroy(struct tc_action *actions[], int bind) { - const struct tc_action_ops *ops; struct tc_action *a; int ret = 0, i; for (i = 0; i < TCA_ACT_MAX_PRIO && actions[i]; i++) { a = actions[i]; actions[i] = NULL; - ops = a->ops; - ret = __tcf_idr_release(a, bind, true); - if (ret == ACT_P_DELETED) - module_put(ops->owner); - else if (ret < 0) - return ret; + /* Drop our reference even if the action is still bound to a filter. */ + ret = tcf_idr_release(a, bind); } return ret; } diff --git a/net/sched/act_ct.c b/net/sched/act_ct.c index 096adcf2209a..d0fd70106767 100644 --- a/net/sched/act_ct.c +++ b/net/sched/act_ct.c @@ -289,7 +289,7 @@ static bool tcf_ct_flow_is_outdated(const struct flow_offload *flow) { return test_bit(IPS_SEEN_REPLY_BIT, &flow->ct->status) && test_bit(IPS_HW_OFFLOAD_BIT, &flow->ct->status) && - !test_bit(NF_FLOW_HW_PENDING, &flow->flags) && + !test_bit(NF_FLOW_PENDING, &flow->flags) && !test_bit(NF_FLOW_HW_ESTABLISHED, &flow->flags); } @@ -973,11 +973,11 @@ TC_INDIRECT_SCOPE int tcf_ct_act(struct sk_buff *skb, const struct tc_action *a, struct tcf_result *res) { struct net *net = dev_net(skb->dev); + bool cached, commit, clear, nat; enum ip_conntrack_info ctinfo; struct tcf_ct *c = to_ct(a); struct nf_conn *tmpl = NULL; struct nf_hook_state state; - bool cached, commit, clear; int nh_ofs, err, retval; struct tcf_ct_params *p; bool add_helper = false; @@ -992,6 +992,7 @@ TC_INDIRECT_SCOPE int tcf_ct_act(struct sk_buff *skb, const struct tc_action *a, retval = READ_ONCE(c->tcf_action); commit = p->ct_action & TCA_CT_ACT_COMMIT; clear = p->ct_action & TCA_CT_ACT_CLEAR; + nat = p->ct_action & TCA_CT_ACT_NAT; tmpl = p->tmpl; tcf_lastuse_update(&c->tcf_tm); @@ -1040,6 +1041,19 @@ TC_INDIRECT_SCOPE int tcf_ct_act(struct sk_buff *skb, const struct tc_action *a, * different zone. */ cached = tcf_ct_skb_nfct_cached(net, skb, p); + + /* If the ct entry is not confirmed and shared with some other skb, + * e.g., a cloned one, we can't just modify it with a commit or nat + * as we must not modify the extension set. Reset. + */ + if (cached && (commit || nat)) { + ct = nf_ct_get(skb, &ctinfo); + if (ct && !nf_ct_is_confirmed(ct) && nf_ct_shared(ct)) { + nf_reset_ct(skb); + cached = false; + } + } + if (!cached) { if (tcf_ct_flow_table_lookup(p, skb, family)) { skip_add = true; @@ -1077,24 +1091,30 @@ TC_INDIRECT_SCOPE int tcf_ct_act(struct sk_buff *skb, const struct tc_action *a, if (err) goto drop; add_helper = true; - if (p->ct_action & TCA_CT_ACT_NAT && !nfct_seqadj(ct)) { + if (nat && !nfct_seqadj(ct)) { if (!nfct_seqadj_ext_add(ct)) goto drop; } } - if (nf_ct_is_confirmed(ct) ? ((!cached && !skip_add) || add_helper) : commit) { - if (nf_ct_helper(skb, ct, ctinfo, family) != NF_ACCEPT) - goto drop; - } - if (commit) { tcf_ct_act_set_mark(ct, p->mark, p->mark_mask); tcf_ct_act_set_labels(ct, p->labels, p->labels_mask); if (!nf_ct_is_confirmed(ct)) nf_conn_act_ct_ext_add(skb, ct, ctinfo); + } + /* Run helpers for the connection if nf_conntrack_in() was executed + * or if we're about to commit. This has to be done after all the + * extensions are already added. + */ + if (nf_ct_is_confirmed(ct) ? ((!cached && !skip_add) || add_helper) : commit) { + if (nf_ct_helper(skb, ct, ctinfo, family) != NF_ACCEPT) + goto drop; + } + + if (commit) { /* This will take care of sending queued events * even if the connection is already confirmed. */ diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c index bf772401b1f4..3adcd507c6ee 100644 --- a/net/sched/act_ife.c +++ b/net/sched/act_ife.c @@ -28,6 +28,7 @@ #include #include #include +#include #include #include @@ -723,7 +724,7 @@ static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a, tcf_lastuse_update(&ife->tcf_tm); if (skb_at_tc_ingress(skb)) - skb_push(skb, skb->dev->hard_header_len); + skb_push(skb, ETH_HLEN); tlv_data = ife_decode(skb, &metalen); if (unlikely(!tlv_data)) { @@ -795,7 +796,7 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a, where ORIGDATA = original ethernet header ... */ u16 metalen = ife_get_sz(skb, p); - int hdrm = metalen + skb->dev->hard_header_len + IFE_METAHDRLEN; + int hdrm = metalen + ETH_HLEN + IFE_METAHDRLEN; unsigned int skboff = 0; int new_len = skb->len + hdrm; bool exceed_mtu = false; @@ -826,7 +827,7 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a, } if (skb_at_tc_ingress(skb)) - skb_push(skb, skb->dev->hard_header_len); + skb_push(skb, ETH_HLEN); ife_meta = ife_encode(skb, metalen); if (!ife_meta) @@ -856,11 +857,27 @@ static int tcf_ife_encode(struct sk_buff *skb, const struct tc_action *a, oethh->h_proto = htons(p->eth_type); if (skb_at_tc_ingress(skb)) - skb_pull(skb, skb->dev->hard_header_len); + skb_pull(skb, ETH_HLEN); return action; } +/* IFE encapsulates the original Ethernet header and, on decode, expects to + * find one, so it can only ever work on skbs that carry one. Loopback carries + * Ethernet header as well, so it qualifies here. + * At ingress, also verify that the L2 header about to be pushed back really + * is an Ethernet header because the skb could've been redirected with mirred + * from a non-Ethernet device. + */ +static bool tcf_ife_is_eth_skb(const struct sk_buff *skb) +{ + if (skb->dev->type != ARPHRD_ETHER && + skb->dev->type != ARPHRD_LOOPBACK) + return false; + + return !skb_at_tc_ingress(skb) || skb->mac_len == ETH_HLEN; +} + TC_INDIRECT_SCOPE int tcf_ife_act(struct sk_buff *skb, const struct tc_action *a, struct tcf_result *res) @@ -869,6 +886,13 @@ TC_INDIRECT_SCOPE int tcf_ife_act(struct sk_buff *skb, struct tcf_ife_params *p; int ret; + if (unlikely(!tcf_ife_is_eth_skb(skb))) { + bstats_update(this_cpu_ptr(ife->common.cpu_bstats), skb); + tcf_lastuse_update(&ife->tcf_tm); + qstats_drop_inc(this_cpu_ptr(ife->common.cpu_qstats)); + return TC_ACT_SHOT; + } + p = rcu_dereference_bh(ife->params); if (p->flags & IFE_ENCODE) { ret = tcf_ife_encode(skb, a, res, p); diff --git a/net/sched/cls_api.c b/net/sched/cls_api.c index 539228eb06c6..5eb337c1531f 100644 --- a/net/sched/cls_api.c +++ b/net/sched/cls_api.c @@ -2436,7 +2436,20 @@ static int tc_new_tfilter(struct sk_buff *skb, struct nlmsghdr *n, } errout: - if (err && tp_state == TP_CREATED) + if (err && !IS_ERR_OR_NULL(tp) && + (tp_state == TP_CREATED || tp->ops->delete_empty)) + /* + * The request is dropping its reference to tp. If it was + * the last user (the idr is empty now), reclaim the proto. + * A tp this request created is reclaimed unconditionally: + * it is the only owner, so marking it for deletion is + * safe. Otherwise only classifiers with a delete_empty + * callback are reclaimed -- the callback admits an empty + * proto only, so a live shared proto is never unlinked. + * Classifiers without one (deleting is set + * unconditionally) are rtnl-serialized, so the raced + * window this guard closes cannot arise for them. + */ tcf_chain_tp_delete_empty(chain, tp, rtnl_held, NULL); errout_tp: if (chain) { @@ -3343,7 +3356,7 @@ int tcf_exts_init_ex(struct tcf_exts *exts, struct net *net, int action, */ exts->net = net; exts->actions = kcalloc(TCA_ACT_MAX_PRIO, sizeof(struct tc_action *), - GFP_KERNEL); + GFP_KERNEL_ACCOUNT); if (!exts->actions) return -ENOMEM; #endif diff --git a/net/sched/cls_basic.c b/net/sched/cls_basic.c index 1b92c33b5f81..ee09ff574f96 100644 --- a/net/sched/cls_basic.c +++ b/net/sched/cls_basic.c @@ -193,7 +193,7 @@ static int basic_change(struct net *net, struct sk_buff *in_skb, return -EINVAL; } - fnew = kzalloc(sizeof(*fnew), GFP_KERNEL); + fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT); if (!fnew) return -ENOBUFS; @@ -212,9 +212,11 @@ static int basic_change(struct net *net, struct sk_buff *in_skb, if (err) goto errout; fnew->handle = handle; - fnew->pf = alloc_percpu(struct tc_basic_pcnt); + fnew->pf = alloc_percpu_gfp(struct tc_basic_pcnt, GFP_KERNEL_ACCOUNT); if (!fnew->pf) { err = -ENOMEM; + if (!fold) + idr_remove(&head->handle_idr, fnew->handle); goto errout; } diff --git a/net/sched/cls_bpf.c b/net/sched/cls_bpf.c index 8ca716bf9e42..b0ae488bc533 100644 --- a/net/sched/cls_bpf.c +++ b/net/sched/cls_bpf.c @@ -352,7 +352,7 @@ static int cls_bpf_prog_from_ops(struct nlattr **tb, struct cls_bpf_prog *prog) if (bpf_size != nla_len(tb[TCA_BPF_OPS])) return -EINVAL; - bpf_ops = kmemdup(nla_data(tb[TCA_BPF_OPS]), bpf_size, GFP_KERNEL); + bpf_ops = kmemdup(nla_data(tb[TCA_BPF_OPS]), bpf_size, GFP_KERNEL_ACCOUNT); if (bpf_ops == NULL) return -ENOMEM; @@ -403,7 +403,7 @@ static int cls_bpf_prog_from_efd(struct nlattr **tb, struct cls_bpf_prog *prog, } if (tb[TCA_BPF_NAME]) { - name = nla_memdup(tb[TCA_BPF_NAME], GFP_KERNEL); + name = nla_memdup(tb[TCA_BPF_NAME], GFP_KERNEL_ACCOUNT); if (!name) { bpf_prog_put(fp); return -ENOMEM; @@ -443,7 +443,7 @@ static int cls_bpf_change(struct net *net, struct sk_buff *in_skb, if (ret < 0) return ret; - prog = kzalloc(sizeof(*prog), GFP_KERNEL); + prog = kzalloc(sizeof(*prog), GFP_KERNEL_ACCOUNT); if (!prog) return -ENOBUFS; diff --git a/net/sched/cls_cgroup.c b/net/sched/cls_cgroup.c index bd9322d71910..f05e654e0b99 100644 --- a/net/sched/cls_cgroup.c +++ b/net/sched/cls_cgroup.c @@ -95,7 +95,7 @@ static int cls_cgroup_change(struct net *net, struct sk_buff *in_skb, if (head && handle != head->handle) return -ENOENT; - new = kzalloc(sizeof(*head), GFP_KERNEL); + new = kzalloc(sizeof(*head), GFP_KERNEL_ACCOUNT); if (!new) return -ENOBUFS; diff --git a/net/sched/cls_flow.c b/net/sched/cls_flow.c index 9ba13953d3a5..e2883f0e4048 100644 --- a/net/sched/cls_flow.c +++ b/net/sched/cls_flow.c @@ -438,7 +438,7 @@ static int flow_change(struct net *net, struct sk_buff *in_skb, return -EOPNOTSUPP; } - fnew = kzalloc(sizeof(*fnew), GFP_KERNEL); + fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT); if (!fnew) return -ENOBUFS; diff --git a/net/sched/cls_flower.c b/net/sched/cls_flower.c index 4fc59af6696e..8b4dc3930906 100644 --- a/net/sched/cls_flower.c +++ b/net/sched/cls_flower.c @@ -2109,7 +2109,7 @@ static struct fl_flow_mask *fl_create_new_mask(struct cls_fl_head *head, struct fl_flow_mask *newmask; int err; - newmask = kzalloc(sizeof(*newmask), GFP_KERNEL); + newmask = kzalloc(sizeof(*newmask), GFP_KERNEL_ACCOUNT); if (!newmask) return ERR_PTR(-ENOMEM); @@ -2269,7 +2269,7 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, goto errout_tb; } - fnew = kzalloc(sizeof(*fnew), GFP_KERNEL); + fnew = kzalloc(sizeof(*fnew), GFP_KERNEL_ACCOUNT); if (!fnew) { err = -ENOBUFS; goto errout_tb; diff --git a/net/sched/cls_fw.c b/net/sched/cls_fw.c index bc007bb47432..1d5427757c69 100644 --- a/net/sched/cls_fw.c +++ b/net/sched/cls_fw.c @@ -276,7 +276,7 @@ static int fw_change(struct net *net, struct sk_buff *in_skb, if (f->id != handle && handle) return -EINVAL; - fnew = kzalloc(sizeof(struct fw_filter), GFP_KERNEL); + fnew = kzalloc(sizeof(struct fw_filter), GFP_KERNEL_ACCOUNT); if (!fnew) return -ENOBUFS; @@ -330,7 +330,7 @@ static int fw_change(struct net *net, struct sk_buff *in_skb, rcu_assign_pointer(tp->root, head); } - f = kzalloc(sizeof(struct fw_filter), GFP_KERNEL); + f = kzalloc(sizeof(struct fw_filter), GFP_KERNEL_ACCOUNT); if (f == NULL) return -ENOBUFS; diff --git a/net/sched/cls_matchall.c b/net/sched/cls_matchall.c index e8353b27c8c4..c7060d8e54a2 100644 --- a/net/sched/cls_matchall.c +++ b/net/sched/cls_matchall.c @@ -189,7 +189,7 @@ static int mall_change(struct net *net, struct sk_buff *in_skb, return -EINVAL; } - new = kzalloc(sizeof(*new), GFP_KERNEL); + new = kzalloc(sizeof(*new), GFP_KERNEL_ACCOUNT); if (!new) return -ENOBUFS; @@ -201,7 +201,7 @@ static int mall_change(struct net *net, struct sk_buff *in_skb, handle = 1; new->handle = handle; new->flags = userflags; - new->pf = alloc_percpu(struct tc_matchall_pcnt); + new->pf = alloc_percpu_gfp(struct tc_matchall_pcnt, GFP_KERNEL_ACCOUNT); if (!new->pf) { err = -ENOMEM; goto err_alloc_percpu; diff --git a/net/sched/cls_route.c b/net/sched/cls_route.c index 233bfba3bc81..592cd27cfaba 100644 --- a/net/sched/cls_route.c +++ b/net/sched/cls_route.c @@ -434,6 +434,12 @@ static int route4_set_parms(struct net *net, struct tcf_proto *tp, if (handle && (!fold || nhandle == (handle & ~0x7F00))) nhandle |= handle & 0x7F00; + if (fold && !tb[TCA_ROUTE4_TO] && !tb[TCA_ROUTE4_FROM] && + !tb[TCA_ROUTE4_IIF] && nhandle != fold->handle) { + NL_SET_ERR_MSG(extack, "Routing attributes required"); + return -EINVAL; + } + if (handle && !fold && nhandle != handle) { NL_SET_ERR_MSG_FMT(extack, "Handle mismatch constructed: %x (expected: %x)", @@ -449,7 +455,7 @@ static int route4_set_parms(struct net *net, struct tcf_proto *tp, h1 = to_hash(nhandle); b = rtnl_dereference(head->table[h1]); if (!b) { - b = kzalloc(sizeof(struct route4_bucket), GFP_KERNEL); + b = kzalloc(sizeof(struct route4_bucket), GFP_KERNEL_ACCOUNT); if (b == NULL) return -ENOBUFS; @@ -524,7 +530,7 @@ static int route4_change(struct net *net, struct sk_buff *in_skb, return -EINVAL; err = -ENOBUFS; - f = kzalloc(sizeof(struct route4_filter), GFP_KERNEL); + f = kzalloc(sizeof(struct route4_filter), GFP_KERNEL_ACCOUNT); if (!f) goto errout; diff --git a/net/sched/cls_u32.c b/net/sched/cls_u32.c index 30e703fa22a7..69c6ccd8f091 100644 --- a/net/sched/cls_u32.c +++ b/net/sched/cls_u32.c @@ -841,7 +841,7 @@ static struct tc_u_knode *u32_init_knode(struct net *net, struct tcf_proto *tp, struct tc_u32_sel *s = &n->sel; struct tc_u_knode *new; - new = kzalloc(struct_size(new, sel.keys, s->nkeys), GFP_KERNEL); + new = kzalloc(struct_size(new, sel.keys, s->nkeys), GFP_KERNEL_ACCOUNT); if (!new) return NULL; @@ -1132,15 +1132,23 @@ static int u32_change(struct net *net, struct sk_buff *in_skb, goto erridr; } - n = kzalloc(struct_size(n, sel.keys, s->nkeys), GFP_KERNEL); + if (s->offshift >= 16) { + NL_SET_ERR_MSG_MOD(extack, + "offshift must be less than 16"); + err = -EINVAL; + goto erridr; + } + + n = kzalloc(struct_size(n, sel.keys, s->nkeys), GFP_KERNEL_ACCOUNT); if (n == NULL) { err = -ENOBUFS; goto erridr; } #ifdef CONFIG_CLS_U32_PERF - n->pf = __alloc_percpu(struct_size(n->pf, kcnts, s->nkeys), - __alignof__(struct tc_u32_pcnt)); + n->pf = __alloc_percpu_gfp(struct_size(n->pf, kcnts, s->nkeys), + __alignof__(struct tc_u32_pcnt), + GFP_KERNEL_ACCOUNT); if (!n->pf) { err = -ENOBUFS; goto errfree; @@ -1162,7 +1170,7 @@ static int u32_change(struct net *net, struct sk_buff *in_skb, goto errout; #ifdef CONFIG_CLS_U32_MARK - n->pcpu_success = alloc_percpu(u32); + n->pcpu_success = alloc_percpu_gfp(u32, GFP_KERNEL_ACCOUNT); if (!n->pcpu_success) { err = -ENOMEM; goto errout; diff --git a/net/sched/em_text.c b/net/sched/em_text.c index 32aae8a9deda..d612ad4e5c12 100644 --- a/net/sched/em_text.c +++ b/net/sched/em_text.c @@ -58,6 +58,9 @@ static int em_text_change(struct net *net, void *data, int len, if (len < sizeof(*conf) || len < (sizeof(*conf) + conf->pattern_len)) return -EINVAL; + if (!memchr(conf->algo, '\0', sizeof(conf->algo))) + return -EINVAL; + if (conf->from_layer > conf->to_layer) return -EINVAL; diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c index 4331097f813e..a25b45ab6fb0 100644 --- a/net/sched/sch_cake.c +++ b/net/sched/sch_cake.c @@ -1721,7 +1721,7 @@ static u32 cake_classify(struct Qdisc *sch, struct cake_tin_data **t, goto hash; *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS; - result = tcf_classify(skb, NULL, filter, &res, false); + result = tcf_classify_qdisc(skb, filter, &res, false); if (result >= 0) { #ifdef CONFIG_NET_CLS_ACT diff --git a/net/sched/sch_codel.c b/net/sched/sch_codel.c index c05be1eeea56..655eee9e629b 100644 --- a/net/sched/sch_codel.c +++ b/net/sched/sch_codel.c @@ -168,9 +168,9 @@ static const struct nla_policy codel_policy[TCA_CODEL_MAX + 1] = { static int codel_change(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { + unsigned int dropped_pkts = 0, dropped_bytes = 0; struct codel_sched_data *q = qdisc_priv(sch); struct nlattr *tb[TCA_CODEL_MAX + 1]; - unsigned int qlen, dropped = 0; int err; err = nla_parse_nested_deprecated(tb, TCA_CODEL_MAX, opt, @@ -204,15 +204,17 @@ static int codel_change(struct Qdisc *sch, struct nlattr *opt, if (tb[TCA_CODEL_ECN]) q->params.ecn = !!nla_get_u32(tb[TCA_CODEL_ECN]); - qlen = sch->q.qlen; while (sch->q.qlen > sch->limit) { struct sk_buff *skb = qdisc_dequeue_internal(sch, true); - dropped += qdisc_pkt_len(skb); - qdisc_qstats_backlog_dec(sch, skb); + if (!skb) + break; + + dropped_pkts++; + dropped_bytes += qdisc_pkt_len(skb); rtnl_qdisc_drop(skb, sch); } - qdisc_tree_reduce_backlog(sch, qlen - sch->q.qlen, dropped); + qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes); sch_tree_unlock(sch); return 0; @@ -228,7 +230,7 @@ static int codel_init(struct Qdisc *sch, struct nlattr *opt, codel_params_init(&q->params); codel_vars_init(&q->vars); codel_stats_init(&q->stats); - q->params.mtu = clamp_t(u32, psched_mtu(qdisc_dev(sch)), 256, 1 << 20); + q->params.mtu = min_t(u32, psched_mtu(qdisc_dev(sch)), 1 << 20); if (opt) { int err = codel_change(sch, opt, extack); diff --git a/net/sched/sch_drr.c b/net/sched/sch_drr.c index 6469b39c9161..843dbe346904 100644 --- a/net/sched/sch_drr.c +++ b/net/sched/sch_drr.c @@ -313,7 +313,7 @@ static struct drr_class *drr_classify(struct sk_buff *skb, struct Qdisc *sch, *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS; fl = rcu_dereference_bh(q->filter_list); - result = tcf_classify(skb, NULL, fl, &res, false); + result = tcf_classify_qdisc(skb, fl, &res, false); if (result >= 0) { #ifdef CONFIG_NET_CLS_ACT switch (result) { diff --git a/net/sched/sch_ets.c b/net/sched/sch_ets.c index 2f1e8a5d5b37..f65dae86c618 100644 --- a/net/sched/sch_ets.c +++ b/net/sched/sch_ets.c @@ -389,7 +389,7 @@ static struct ets_class *ets_classify(struct sk_buff *skb, struct Qdisc *sch, *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS; if (TC_H_MAJ(skb->priority) != sch->handle) { fl = rcu_dereference_bh(q->filter_list); - err = tcf_classify(skb, NULL, fl, &res, false); + err = tcf_classify_qdisc(skb, fl, &res, false); #ifdef CONFIG_NET_CLS_ACT switch (err) { case TC_ACT_STOLEN: diff --git a/net/sched/sch_fifo.c b/net/sched/sch_fifo.c index 1080d89f9178..f3c5e22eb529 100644 --- a/net/sched/sch_fifo.c +++ b/net/sched/sch_fifo.c @@ -19,7 +19,8 @@ static int bfifo_enqueue(struct sk_buff *skb, struct Qdisc *sch, struct sk_buff **to_free) { - if (likely(sch->qstats.backlog + qdisc_pkt_len(skb) <= sch->limit)) + if (likely((u64)sch->qstats.backlog + qdisc_pkt_len(skb) <= + READ_ONCE(sch->limit))) return qdisc_enqueue_tail(skb, sch); return qdisc_drop(skb, sch, to_free); @@ -28,7 +29,7 @@ static int bfifo_enqueue(struct sk_buff *skb, struct Qdisc *sch, static int pfifo_enqueue(struct sk_buff *skb, struct Qdisc *sch, struct sk_buff **to_free) { - if (likely(sch->q.qlen < sch->limit)) + if (likely(sch->q.qlen < READ_ONCE(sch->limit))) return qdisc_enqueue_tail(skb, sch); return qdisc_drop(skb, sch, to_free); @@ -38,11 +39,12 @@ static int pfifo_tail_enqueue(struct sk_buff *skb, struct Qdisc *sch, struct sk_buff **to_free) { unsigned int prev_backlog; + u32 limit = READ_ONCE(sch->limit); - if (unlikely(sch->limit == 0)) + if (unlikely(limit == 0)) return qdisc_drop(skb, sch, to_free); - if (likely(sch->q.qlen < sch->limit)) + if (likely(sch->q.qlen < limit)) return qdisc_enqueue_tail(skb, sch); prev_backlog = sch->qstats.backlog; @@ -108,14 +110,14 @@ static int __fifo_init(struct Qdisc *sch, struct nlattr *opt, if (is_bfifo) limit *= psched_mtu(qdisc_dev(sch)); - sch->limit = limit; + WRITE_ONCE(sch->limit, limit); } else { struct tc_fifo_qopt *ctl = nla_data(opt); if (nla_len(opt) < sizeof(*ctl)) return -EINVAL; - sch->limit = ctl->limit; + WRITE_ONCE(sch->limit, ctl->limit); } if (is_bfifo) @@ -157,7 +159,7 @@ static void fifo_destroy(struct Qdisc *sch) static int __fifo_dump(struct Qdisc *sch, struct sk_buff *skb) { - struct tc_fifo_qopt opt = { .limit = sch->limit }; + struct tc_fifo_qopt opt = { .limit = READ_ONCE(sch->limit) }; if (nla_put(skb, TCA_OPTIONS, sizeof(opt), &opt)) goto nla_put_failure; diff --git a/net/sched/sch_fq.c b/net/sched/sch_fq.c index 91f5ef6be0f2..edabe21ef303 100644 --- a/net/sched/sch_fq.c +++ b/net/sched/sch_fq.c @@ -104,6 +104,9 @@ struct fq_sched_data { unsigned long unthrottle_latency_ns; struct fq_flow internal; /* for non classified or high prio packets */ + +/* Read mostly cache line */ + u32 quantum; u32 initial_quantum; u32 flow_refill_delay; @@ -117,22 +120,27 @@ struct fq_sched_data { u8 rate_enable; u8 fq_trees_log; u8 horizon_drop; + u32 timer_slack; /* hrtimer slack in ns */ + +/* Read/Write fields. */ + u32 flows; - u32 inactive_flows; + u32 inactive_flows; /* Flows with no packet to send. */ u32 throttled_flows; + u64 stat_throttled; + struct qdisc_watchdog watchdog; u64 stat_gc_flows; + +/* Seldom used fields. */ + u64 stat_internal_packets; - u64 stat_throttled; u64 stat_ce_mark; u64 stat_horizon_drops; u64 stat_horizon_caps; u64 stat_flows_plimit; u64 stat_pkts_too_long; u64 stat_allocation_errors; - - u32 timer_slack; /* hrtimer slack in ns */ - struct qdisc_watchdog watchdog; }; /* @@ -394,9 +402,12 @@ static void fq_erase_head(struct Qdisc *sch, struct fq_flow *flow, static void fq_dequeue_skb(struct Qdisc *sch, struct fq_flow *flow, struct sk_buff *skb) { + struct fq_sched_data *q = qdisc_priv(sch); + fq_erase_head(sch, flow, skb); skb_mark_not_on_list(skb); - flow->qlen--; + if (--flow->qlen == 0) + q->inactive_flows++; qdisc_qstats_backlog_dec(sch, skb); sch->q.qlen--; } @@ -476,13 +487,13 @@ static int fq_enqueue(struct sk_buff *skb, struct Qdisc *sch, return qdisc_drop(skb, sch, to_free); } - f->qlen++; + if (f->qlen++ == 0) + q->inactive_flows--; qdisc_qstats_backlog_inc(sch, skb); if (fq_flow_is_detached(f)) { fq_flow_add_tail(&q->new_flows, f); if (time_after(jiffies, f->age + q->flow_refill_delay)) f->credit = max_t(u32, f->credit, q->quantum); - q->inactive_flows--; } /* Note: this overwrites f->age */ @@ -589,7 +600,6 @@ static struct sk_buff *fq_dequeue(struct Qdisc *sch) fq_flow_add_tail(&q->old_flows, f); } else { fq_flow_set_detached(f); - q->inactive_flows++; } goto begin; } @@ -806,11 +816,11 @@ static const struct nla_policy fq_policy[TCA_FQ_MAX + 1] = { static int fq_change(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { + unsigned int dropped_pkts = 0, dropped_bytes = 0; struct fq_sched_data *q = qdisc_priv(sch); struct nlattr *tb[TCA_FQ_MAX + 1]; - int err, drop_count = 0; - unsigned drop_len = 0; u32 fq_log; + int err; err = nla_parse_nested_deprecated(tb, TCA_FQ_MAX, opt, fq_policy, NULL); @@ -900,16 +910,18 @@ static int fq_change(struct Qdisc *sch, struct nlattr *opt, err = fq_resize(sch, fq_log); sch_tree_lock(sch); } + while (sch->q.qlen > sch->limit) { struct sk_buff *skb = qdisc_dequeue_internal(sch, false); if (!skb) break; - drop_len += qdisc_pkt_len(skb); + + dropped_pkts++; + dropped_bytes += qdisc_pkt_len(skb); rtnl_kfree_skbs(skb, skb); - drop_count++; } - qdisc_tree_reduce_backlog(sch, drop_count, drop_len); + qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes); sch_tree_unlock(sch); return err; @@ -928,12 +940,14 @@ static int fq_init(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { struct fq_sched_data *q = qdisc_priv(sch); + u32 mtu; int err; sch->limit = 10000; q->flow_plimit = 100; - q->quantum = 2 * psched_mtu(qdisc_dev(sch)); - q->initial_quantum = 10 * psched_mtu(qdisc_dev(sch)); + mtu = clamp_t(u32, psched_mtu(qdisc_dev(sch)), 1, 1 << 20); + q->quantum = min_t(u32, 2 * mtu, 1 << 20); + q->initial_quantum = min_t(u32, 10 * mtu, 1 << 20); q->flow_refill_delay = msecs_to_jiffies(40); q->flow_max_rate = ~0UL; q->time_next_delayed_flow = ~0ULL; diff --git a/net/sched/sch_fq_codel.c b/net/sched/sch_fq_codel.c index ac2f643dd842..8a4f4c496370 100644 --- a/net/sched/sch_fq_codel.c +++ b/net/sched/sch_fq_codel.c @@ -91,7 +91,7 @@ static unsigned int fq_codel_classify(struct sk_buff *skb, struct Qdisc *sch, return fq_codel_hash(q, skb) + 1; *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS; - result = tcf_classify(skb, NULL, filter, &res, false); + result = tcf_classify_qdisc(skb, filter, &res, false); if (result >= 0) { #ifdef CONFIG_NET_CLS_ACT switch (result) { @@ -402,6 +402,7 @@ static const struct nla_policy fq_codel_policy[TCA_FQ_CODEL_MAX + 1] = { static int fq_codel_change(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { + unsigned int dropped_pkts = 0, dropped_bytes = 0; struct fq_codel_sched_data *q = qdisc_priv(sch); struct nlattr *tb[TCA_FQ_CODEL_MAX + 1]; u32 quantum = 0; @@ -470,13 +471,14 @@ static int fq_codel_change(struct Qdisc *sch, struct nlattr *opt, q->memory_usage > q->memory_limit) { struct sk_buff *skb = qdisc_dequeue_internal(sch, false); - q->cstats.drop_len += qdisc_pkt_len(skb); + if (!skb) + break; + + dropped_pkts++; + dropped_bytes += qdisc_pkt_len(skb); rtnl_kfree_skbs(skb, skb); - q->cstats.drop_count++; } - qdisc_tree_reduce_backlog(sch, q->cstats.drop_count, q->cstats.drop_len); - q->cstats.drop_count = 0; - q->cstats.drop_len = 0; + qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes); sch_tree_unlock(sch); return 0; @@ -503,8 +505,8 @@ static int fq_codel_init(struct Qdisc *sch, struct nlattr *opt, q->flows_cnt = 1024; q->memory_limit = 32 << 20; /* 32 MBytes */ q->drop_batch_size = 64; - mtu = clamp_t(u32, psched_mtu(qdisc_dev(sch)), 256, FQ_CODEL_QUANTUM_MAX); - q->quantum = mtu; + mtu = min_t(u32, psched_mtu(qdisc_dev(sch)), FQ_CODEL_QUANTUM_MAX); + q->quantum = clamp_t(u32, mtu, 256, FQ_CODEL_QUANTUM_MAX); INIT_LIST_HEAD(&q->new_flows); INIT_LIST_HEAD(&q->old_flows); codel_params_init(&q->cparams); diff --git a/net/sched/sch_fq_pie.c b/net/sched/sch_fq_pie.c index 75d6e7716ae8..1bf86b65dacf 100644 --- a/net/sched/sch_fq_pie.c +++ b/net/sched/sch_fq_pie.c @@ -95,7 +95,7 @@ static unsigned int fq_pie_classify(struct sk_buff *skb, struct Qdisc *sch, return fq_pie_hash(q, skb) + 1; *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS; - result = tcf_classify(skb, NULL, filter, &res, false); + result = tcf_classify_qdisc(skb, filter, &res, false); if (result >= 0) { #ifdef CONFIG_NET_CLS_ACT switch (result) { @@ -284,10 +284,9 @@ static struct sk_buff *fq_pie_qdisc_dequeue(struct Qdisc *sch) static int fq_pie_change(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { + unsigned int dropped_pkts = 0, dropped_bytes = 0; struct fq_pie_sched_data *q = qdisc_priv(sch); struct nlattr *tb[TCA_FQ_PIE_MAX + 1]; - unsigned int len_dropped = 0; - unsigned int num_dropped = 0; int err; err = nla_parse_nested(tb, TCA_FQ_PIE_MAX, opt, fq_pie_policy, extack); @@ -366,11 +365,14 @@ static int fq_pie_change(struct Qdisc *sch, struct nlattr *opt, while (sch->q.qlen > sch->limit) { struct sk_buff *skb = qdisc_dequeue_internal(sch, false); - len_dropped += qdisc_pkt_len(skb); - num_dropped += 1; + if (!skb) + break; + + dropped_pkts++; + dropped_bytes += qdisc_pkt_len(skb); rtnl_kfree_skbs(skb, skb); } - qdisc_tree_reduce_backlog(sch, num_dropped, len_dropped); + qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes); sch_tree_unlock(sch); return 0; diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c index 3d695ed25d32..d42b38c86f61 100644 --- a/net/sched/sch_generic.c +++ b/net/sched/sch_generic.c @@ -541,28 +541,43 @@ static void dev_watchdog(struct timer_list *t) dev->netdev_ops->ndo_tx_timeout(dev, i); netif_unfreeze_queues(dev); } - if (!mod_timer(&dev->watchdog_timer, - round_jiffies(oldest_start + - dev->watchdog_timeo))) - release = false; + spin_lock(&dev->watchdog_lock); + mod_timer(&dev->watchdog_timer, + round_jiffies(oldest_start + + dev->watchdog_timeo)); + release = false; + spin_unlock(&dev->watchdog_lock); } } spin_unlock(&dev->tx_global_lock); - if (release) + spin_lock(&dev->watchdog_lock); + if (timer_pending(&dev->watchdog_timer)) + release = false; + if (release && dev->watchdog_ref_held) { netdev_put(dev, &dev->watchdog_dev_tracker); + dev->watchdog_ref_held = false; + } + spin_unlock(&dev->watchdog_lock); } void __netdev_watchdog_up(struct net_device *dev) { - if (dev->netdev_ops->ndo_tx_timeout) { - if (dev->watchdog_timeo <= 0) - dev->watchdog_timeo = 5*HZ; - if (!mod_timer(&dev->watchdog_timer, - round_jiffies(jiffies + dev->watchdog_timeo))) + if (!dev->netdev_ops->ndo_tx_timeout) + return; + if (dev->watchdog_timeo <= 0) + dev->watchdog_timeo = 5*HZ; + + spin_lock_bh(&dev->watchdog_lock); + if (!mod_timer(&dev->watchdog_timer, + round_jiffies(jiffies + dev->watchdog_timeo))) { + if (!dev->watchdog_ref_held) { netdev_hold(dev, &dev->watchdog_dev_tracker, GFP_ATOMIC); + dev->watchdog_ref_held = true; + } } + spin_unlock_bh(&dev->watchdog_lock); } EXPORT_SYMBOL_GPL(__netdev_watchdog_up); @@ -574,8 +589,14 @@ static void dev_watchdog_up(struct net_device *dev) static void dev_watchdog_down(struct net_device *dev) { netif_tx_lock_bh(dev); - if (del_timer(&dev->watchdog_timer)) + + spin_lock(&dev->watchdog_lock); + if (timer_delete(&dev->watchdog_timer)) { netdev_put(dev, &dev->watchdog_dev_tracker); + dev->watchdog_ref_held = false; + } + spin_unlock(&dev->watchdog_lock); + netif_tx_unlock_bh(dev); } @@ -592,8 +613,6 @@ void netif_carrier_on(struct net_device *dev) return; atomic_inc(&dev->carrier_up_count); linkwatch_fire_event(dev); - if (netif_running(dev)) - __netdev_watchdog_up(dev); } } EXPORT_SYMBOL(netif_carrier_on); @@ -708,9 +727,10 @@ struct Qdisc_ops noqueue_qdisc_ops __read_mostly = { .owner = THIS_MODULE, }; -static const u8 prio2band[TC_PRIO_MAX + 1] = { - 1, 2, 2, 2, 1, 2, 0, 0 , 1, 1, 1, 1, 1, 1, 1, 1 +const u8 sch_default_prio2band[TC_PRIO_MAX + 1] = { + 1, 2, 2, 2, 1, 2, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1 }; +EXPORT_SYMBOL(sch_default_prio2band); /* 3-band FIFO queue: old style, but should be a bit faster than generic prio+fifo combination. @@ -735,7 +755,7 @@ static inline struct skb_array *band2list(struct pfifo_fast_priv *priv, static int pfifo_fast_enqueue(struct sk_buff *skb, struct Qdisc *qdisc, struct sk_buff **to_free) { - int band = prio2band[skb->priority & TC_PRIO_MAX]; + int band = sch_default_prio2band[skb->priority & TC_PRIO_MAX]; struct pfifo_fast_priv *priv = qdisc_priv(qdisc); struct skb_array *q = band2list(priv, band); unsigned int pkt_len = qdisc_pkt_len(skb); @@ -844,7 +864,7 @@ static int pfifo_fast_dump(struct Qdisc *qdisc, struct sk_buff *skb) { struct tc_prio_qopt opt = { .bands = PFIFO_FAST_BANDS }; - memcpy(&opt.priomap, prio2band, TC_PRIO_MAX + 1); + memcpy(&opt.priomap, sch_default_prio2band, TC_PRIO_MAX + 1); if (nla_put(skb, TCA_OPTIONS, sizeof(opt), &opt)) goto nla_put_failure; return skb->len; diff --git a/net/sched/sch_gred.c b/net/sched/sch_gred.c index fa7a1b69c0f3..4f44a268286f 100644 --- a/net/sched/sch_gred.c +++ b/net/sched/sch_gred.c @@ -179,7 +179,7 @@ static int gred_enqueue(struct sk_buff *skb, struct Qdisc *sch, * if no default DP has been configured. This * allows for DP flows to be left untouched. */ - if (likely(sch->qstats.backlog + qdisc_pkt_len(skb) <= + if (likely((u64)sch->qstats.backlog + qdisc_pkt_len(skb) <= sch->limit)) return qdisc_enqueue_tail(skb, sch); else @@ -244,7 +244,7 @@ static int gred_enqueue(struct sk_buff *skb, struct Qdisc *sch, break; } - if (gred_backlog(t, q, sch) + qdisc_pkt_len(skb) <= q->limit) { + if ((u64)gred_backlog(t, q, sch) + qdisc_pkt_len(skb) <= q->limit) { q->backlog += qdisc_pkt_len(skb); return qdisc_enqueue_tail(skb, sch); } diff --git a/net/sched/sch_hfsc.c b/net/sched/sch_hfsc.c index 51dd4243db46..d0249c45394d 100644 --- a/net/sched/sch_hfsc.c +++ b/net/sched/sch_hfsc.c @@ -1154,7 +1154,7 @@ hfsc_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr) head = &q->root; drift = HFSC_CLASSIFY_MAX_DRIFT; tcf = rcu_dereference_bh(q->root.filter_list); - while (tcf && (result = tcf_classify(skb, NULL, tcf, &res, false)) >= 0) { + while (tcf && (result = tcf_classify_qdisc(skb, tcf, &res, false)) >= 0) { #ifdef CONFIG_NET_CLS_ACT switch (result) { case TC_ACT_QUEUED: diff --git a/net/sched/sch_hhf.c b/net/sched/sch_hhf.c index e1e8527e4519..17027d1f4d11 100644 --- a/net/sched/sch_hhf.c +++ b/net/sched/sch_hhf.c @@ -536,9 +536,9 @@ static const struct nla_policy hhf_policy[TCA_HHF_MAX + 1] = { static int hhf_change(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { + unsigned int dropped_pkts = 0, dropped_bytes = 0; struct hhf_sched_data *q = qdisc_priv(sch); struct nlattr *tb[TCA_HHF_MAX + 1]; - unsigned int qlen, prev_backlog; int err; u64 non_hh_quantum; u32 new_quantum = q->quantum; @@ -585,15 +585,17 @@ static int hhf_change(struct Qdisc *sch, struct nlattr *opt, q->hhf_evict_timeout = usecs_to_jiffies(us); } - qlen = sch->q.qlen; - prev_backlog = sch->qstats.backlog; while (sch->q.qlen > sch->limit) { struct sk_buff *skb = qdisc_dequeue_internal(sch, false); + if (!skb) + break; + + dropped_pkts++; + dropped_bytes += qdisc_pkt_len(skb); rtnl_kfree_skbs(skb, skb); } - qdisc_tree_reduce_backlog(sch, qlen - sch->q.qlen, - prev_backlog - sch->qstats.backlog); + qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes); sch_tree_unlock(sch); return 0; diff --git a/net/sched/sch_htb.c b/net/sched/sch_htb.c index ed1bf8406ae6..6edd9a82aaeb 100644 --- a/net/sched/sch_htb.c +++ b/net/sched/sch_htb.c @@ -243,7 +243,7 @@ static struct htb_class *htb_classify(struct sk_buff *skb, struct Qdisc *sch, } *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS; - while (tcf && (result = tcf_classify(skb, NULL, tcf, &res, false)) >= 0) { + while (tcf && (result = tcf_classify_qdisc(skb, tcf, &res, false)) >= 0) { #ifdef CONFIG_NET_CLS_ACT switch (result) { case TC_ACT_QUEUED: diff --git a/net/sched/sch_multiq.c b/net/sched/sch_multiq.c index 62cc1a21a603..3dff4b1d4aa9 100644 --- a/net/sched/sch_multiq.c +++ b/net/sched/sch_multiq.c @@ -36,7 +36,7 @@ multiq_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr) int err; *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS; - err = tcf_classify(skb, NULL, fl, &res, false); + err = tcf_classify_qdisc(skb, fl, &res, false); #ifdef CONFIG_NET_CLS_ACT switch (err) { case TC_ACT_STOLEN: diff --git a/net/sched/sch_pie.c b/net/sched/sch_pie.c index fba806f0acd4..9b41c68f9b97 100644 --- a/net/sched/sch_pie.c +++ b/net/sched/sch_pie.c @@ -138,9 +138,9 @@ static const struct nla_policy pie_policy[TCA_PIE_MAX + 1] = { static int pie_change(struct Qdisc *sch, struct nlattr *opt, struct netlink_ext_ack *extack) { + unsigned int dropped_pkts = 0, dropped_bytes = 0; struct pie_sched_data *q = qdisc_priv(sch); struct nlattr *tb[TCA_PIE_MAX + 1]; - unsigned int qlen, dropped = 0; int err; err = nla_parse_nested_deprecated(tb, TCA_PIE_MAX, opt, pie_policy, @@ -188,15 +188,17 @@ static int pie_change(struct Qdisc *sch, struct nlattr *opt, nla_get_u32(tb[TCA_PIE_DQ_RATE_ESTIMATOR]); /* Drop excess packets if new limit is lower */ - qlen = sch->q.qlen; while (sch->q.qlen > sch->limit) { struct sk_buff *skb = qdisc_dequeue_internal(sch, true); - dropped += qdisc_pkt_len(skb); - qdisc_qstats_backlog_dec(sch, skb); + if (!skb) + break; + + dropped_pkts++; + dropped_bytes += qdisc_pkt_len(skb); rtnl_qdisc_drop(skb, sch); } - qdisc_tree_reduce_backlog(sch, qlen - sch->q.qlen, dropped); + qdisc_tree_reduce_backlog(sch, dropped_pkts, dropped_bytes); sch_tree_unlock(sch); return 0; diff --git a/net/sched/sch_plug.c b/net/sched/sch_plug.c index 35f49edf63db..6638343ca671 100644 --- a/net/sched/sch_plug.c +++ b/net/sched/sch_plug.c @@ -89,7 +89,7 @@ static int plug_enqueue(struct sk_buff *skb, struct Qdisc *sch, { struct plug_sched_data *q = qdisc_priv(sch); - if (likely(sch->qstats.backlog + skb->len <= q->limit)) { + if (likely((u64)sch->qstats.backlog + skb->len <= q->limit)) { if (!q->unplug_indefinite) q->pkts_current_epoch++; return qdisc_enqueue_tail(skb, sch); diff --git a/net/sched/sch_prio.c b/net/sched/sch_prio.c index fdd9caa41e80..7ef77ead388e 100644 --- a/net/sched/sch_prio.c +++ b/net/sched/sch_prio.c @@ -39,7 +39,7 @@ prio_classify(struct sk_buff *skb, struct Qdisc *sch, int *qerr) *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS; if (TC_H_MAJ(skb->priority) != sch->handle) { fl = rcu_dereference_bh(q->filter_list); - err = tcf_classify(skb, NULL, fl, &res, false); + err = tcf_classify_qdisc(skb, fl, &res, false); #ifdef CONFIG_NET_CLS_ACT switch (err) { case TC_ACT_STOLEN: diff --git a/net/sched/sch_qfq.c b/net/sched/sch_qfq.c index a17f7c31378e..e3b1921aa7dd 100644 --- a/net/sched/sch_qfq.c +++ b/net/sched/sch_qfq.c @@ -712,7 +712,7 @@ static struct qfq_class *qfq_classify(struct sk_buff *skb, struct Qdisc *sch, *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS; fl = rcu_dereference_bh(q->filter_list); - result = tcf_classify(skb, NULL, fl, &res, false); + result = tcf_classify_qdisc(skb, fl, &res, false); if (result >= 0) { #ifdef CONFIG_NET_CLS_ACT switch (result) { diff --git a/net/sched/sch_sfb.c b/net/sched/sch_sfb.c index 58bf4c803f7f..693f562bd864 100644 --- a/net/sched/sch_sfb.c +++ b/net/sched/sch_sfb.c @@ -260,7 +260,7 @@ static bool sfb_classify(struct sk_buff *skb, struct tcf_proto *fl, struct tcf_result res; int result; - result = tcf_classify(skb, NULL, fl, &res, false); + result = tcf_classify_qdisc(skb, fl, &res, false); if (result >= 0) { #ifdef CONFIG_NET_CLS_ACT switch (result) { diff --git a/net/sched/sch_sfq.c b/net/sched/sch_sfq.c index 779e632ede19..31b5aeb416ce 100644 --- a/net/sched/sch_sfq.c +++ b/net/sched/sch_sfq.c @@ -171,7 +171,7 @@ static unsigned int sfq_classify(struct sk_buff *skb, struct Qdisc *sch, return sfq_hash(q, skb) + 1; *qerr = NET_XMIT_SUCCESS | __NET_XMIT_BYPASS; - result = tcf_classify(skb, NULL, fl, &res, false); + result = tcf_classify_qdisc(skb, fl, &res, false); if (result >= 0) { #ifdef CONFIG_NET_CLS_ACT switch (result) { diff --git a/net/sctp/auth.c b/net/sctp/auth.c index 8320764b16b9..5fcfcdce0694 100644 --- a/net/sctp/auth.c +++ b/net/sctp/auth.c @@ -376,6 +376,82 @@ int sctp_auth_asoc_copy_shkeys(const struct sctp_endpoint *ep, return -ENOMEM; } +static bool sctp_auth_chunk_id_forbidden(__u8 chunk_id) +{ + switch (chunk_id) { + case SCTP_CID_INIT: + case SCTP_CID_INIT_ACK: + case SCTP_CID_SHUTDOWN_COMPLETE: + case SCTP_CID_AUTH: + return true; + default: + return false; + } +} + +/* Verify AUTH parameters copied from a state cookie before they are restored + * into an association. When cookie authentication is disabled these fields + * are peer-controlled, so they must satisfy the same constraints as locally + * generated AUTH parameters. + */ +bool sctp_auth_verify_cookie_params(const struct sctp_endpoint *ep, + const struct sctp_cookie *cookie) +{ + const struct sctp_paramhdr *random; + const struct sctp_hmac_algo_param *hmacs; + const struct sctp_chunks_param *chunks; + u16 hmacs_len, chunks_len; + u16 n_hmacs, n_chunks, i; + bool has_sha1 = false; + + if (sctp_sk(ep->base.sk)->hmac || !ep->auth_enable) + return true; + + random = (const struct sctp_paramhdr *)cookie->auth_random; + if (random->type != SCTP_PARAM_RANDOM || + ntohs(random->length) != sizeof(*random) + SCTP_AUTH_RANDOM_LENGTH) + return false; + + hmacs = (const struct sctp_hmac_algo_param *)cookie->auth_hmacs; + hmacs_len = ntohs(hmacs->param_hdr.length); + if (hmacs->param_hdr.type != SCTP_PARAM_HMAC_ALGO || + hmacs_len < sizeof(struct sctp_paramhdr) + + sizeof(hmacs->hmac_ids[0]) || + hmacs_len > sizeof(cookie->auth_hmacs) || + (hmacs_len - sizeof(struct sctp_paramhdr)) % + sizeof(hmacs->hmac_ids[0])) + return false; + + n_hmacs = (hmacs_len - sizeof(struct sctp_paramhdr)) / + sizeof(hmacs->hmac_ids[0]); + for (i = 0; i < n_hmacs; i++) { + u16 hmac_id = ntohs(hmacs->hmac_ids[i]); + + if (hmac_id >= SCTP_AUTH_NUM_HMACS || + !sctp_hmac_list[hmac_id].hmac_name) + return false; + if (hmac_id == SCTP_AUTH_HMAC_ID_SHA1) + has_sha1 = true; + } + if (!has_sha1) + return false; + + chunks = (const struct sctp_chunks_param *)cookie->auth_chunks; + chunks_len = ntohs(chunks->param_hdr.length); + if (chunks->param_hdr.type != SCTP_PARAM_CHUNKS || + chunks_len < sizeof(struct sctp_paramhdr) || + chunks_len > sizeof(cookie->auth_chunks)) + return false; + + n_chunks = chunks_len - sizeof(struct sctp_paramhdr); + for (i = 0; i < n_chunks; i++) { + if (sctp_auth_chunk_id_forbidden(chunks->chunks[i])) + return false; + } + + return true; +} + /* Public interface to create the association shared key. * See code above for the algorithm. diff --git a/net/sctp/bind_addr.c b/net/sctp/bind_addr.c index 6b95d3ba8fe1..0947b276d1e0 100644 --- a/net/sctp/bind_addr.c +++ b/net/sctp/bind_addr.c @@ -275,6 +275,16 @@ int sctp_raw_to_bind_addrs(struct sctp_bind_addr *bp, __u8 *raw_addr_list, param = (struct sctp_paramhdr *)raw_addr_list; rawaddr = (union sctp_addr_param *)raw_addr_list; + if (addrs_len < sizeof(*param)) { + retval = -EINVAL; + goto out_err; + } + len = ntohs(param->length); + if (addrs_len < len) { + retval = -EINVAL; + goto out_err; + } + af = sctp_get_af_specific(param_type2af(param->type)); if (unlikely(!af) || !af->from_addr_param(&addr, rawaddr, htons(port), 0)) { @@ -291,7 +301,6 @@ int sctp_raw_to_bind_addrs(struct sctp_bind_addr *bp, __u8 *raw_addr_list, goto out_err; next: - len = ntohs(param->length); addrs_len -= len; raw_addr_list += len; } diff --git a/net/sctp/ipv6.c b/net/sctp/ipv6.c index 02989e5612df..4f0647e60dae 100644 --- a/net/sctp/ipv6.c +++ b/net/sctp/ipv6.c @@ -227,6 +227,7 @@ static int sctp_v6_xmit(struct sk_buff *skb, struct sctp_transport *t) struct ipv6_pinfo *np = inet6_sk(sk); __u8 tclass = np->tclass; __be32 label; + int err; pr_debug("%s: skb:%p, len:%d, src:%pI6 dst:%pI6\n", __func__, skb, skb->len, &fl6->saddr, &fl6->daddr); @@ -263,9 +264,12 @@ static int sctp_v6_xmit(struct sk_buff *skb, struct sctp_transport *t) skb_set_inner_ipproto(skb, IPPROTO_SCTP); label = ip6_make_flowlabel(sock_net(sk), skb, fl6->flowlabel, true, fl6); - return udp_tunnel6_xmit_skb(dst, sk, skb, NULL, &fl6->saddr, - &fl6->daddr, tclass, ip6_dst_hoplimit(dst), - label, sctp_sk(sk)->udp_port, t->encap_port, false); + local_bh_disable(); + err = udp_tunnel6_xmit_skb(dst, sk, skb, NULL, &fl6->saddr, + &fl6->daddr, tclass, ip6_dst_hoplimit(dst), + label, sctp_sk(sk)->udp_port, t->encap_port, false); + local_bh_enable(); + return err; } /* Returns the dst cache entry for the given source and destination ip diff --git a/net/sctp/protocol.c b/net/sctp/protocol.c index ee598e319d0e..f91448f35b0c 100644 --- a/net/sctp/protocol.c +++ b/net/sctp/protocol.c @@ -692,8 +692,9 @@ static void sctp_free_addr_wq(struct net *net) struct sctp_sockaddr_entry *addrw; struct sctp_sockaddr_entry *temp; + timer_shutdown_sync(&net->sctp.addr_wq_timer); + spin_lock_bh(&net->sctp.addr_wq_lock); - del_timer(&net->sctp.addr_wq_timer); list_for_each_entry_safe(addrw, temp, &net->sctp.addr_waitq, list) { list_del(&addrw->list); kfree(addrw); @@ -1085,9 +1086,11 @@ static inline int sctp_v4_xmit(struct sk_buff *skb, struct sctp_transport *t) skb_reset_inner_mac_header(skb); skb_reset_inner_transport_header(skb); skb_set_inner_ipproto(skb, IPPROTO_SCTP); + local_bh_disable(); udp_tunnel_xmit_skb((struct rtable *)dst, sk, skb, fl4->saddr, fl4->daddr, dscp, ip4_dst_hoplimit(dst), df, sctp_sk(sk)->udp_port, t->encap_port, false, false); + local_bh_enable(); return 0; } diff --git a/net/sctp/sm_make_chunk.c b/net/sctp/sm_make_chunk.c index ec3a9363d22c..4d663210f825 100644 --- a/net/sctp/sm_make_chunk.c +++ b/net/sctp/sm_make_chunk.c @@ -1750,8 +1750,9 @@ struct sctp_association *sctp_unpack_cookie( struct sk_buff *skb = chunk->skb; struct sctp_cookie *bear_cookie; __u8 *digest = ep->digest; + struct sctp_chunkhdr *ch; enum sctp_scope scope; - unsigned int len; + unsigned int len, chlen; ktime_t kt; /* Header size is static data prior to the actual cookie, including @@ -1779,6 +1780,17 @@ struct sctp_association *sctp_unpack_cookie( cookie = chunk->subh.cookie_hdr; bear_cookie = &cookie->c; + ch = (struct sctp_chunkhdr *)(bear_cookie + 1); + if (ch->type != SCTP_CID_INIT) + goto malformed; + chlen = ntohs(ch->length); + if (chlen < sizeof(struct sctp_init_chunk)) + goto malformed; + if (chlen > len - fixed_size) + goto malformed; + if (bear_cookie->raw_addr_list_len > len - fixed_size - chlen) + goto malformed; + if (!sctp_sk(ep->base.sk)->hmac) goto no_hmac; @@ -1871,6 +1883,9 @@ struct sctp_association *sctp_unpack_cookie( /* Set up our peer's port number. */ retval->peer.port = ntohs(chunk->sctp_hdr->source); + if (!sctp_auth_verify_cookie_params(ep, bear_cookie)) + goto malformed; + /* Populate the association from the cookie. */ memcpy(&retval->c, bear_cookie, sizeof(*bear_cookie)); @@ -2328,7 +2343,8 @@ int sctp_verify_init(struct net *net, const struct sctp_endpoint *ep, * VIOLATION error. We build the ERROR chunk here and let the normal * error handling code build and send the packet. */ - if (param.v != (void *)chunk->chunk_end) + if (param.v != (void *)peer_init + + SCTP_PAD4(ntohs(peer_init->chunk_hdr.length))) return sctp_process_inv_paramlength(asoc, param.p, chunk, errp); /* The only missing mandatory param possible today is diff --git a/net/sctp/sm_statefuns.c b/net/sctp/sm_statefuns.c index a2f24e56c626..0b54e45f241c 100644 --- a/net/sctp/sm_statefuns.c +++ b/net/sctp/sm_statefuns.c @@ -707,11 +707,12 @@ enum sctp_disposition sctp_sf_do_5_1D_ce(struct net *net, struct sctp_cmd_seq *commands) { struct sctp_ulpevent *ev, *ai_ev = NULL, *auth_ev = NULL; + struct sctp_chunk *err_chk_p = NULL; struct sctp_association *new_asoc; struct sctp_init_chunk *peer_init; struct sctp_chunk *chunk = arg; - struct sctp_chunk *err_chk_p; struct sctp_chunk *repl; + enum sctp_cid cid; struct sock *sk; int error = 0; @@ -785,6 +786,19 @@ enum sctp_disposition sctp_sf_do_5_1D_ce(struct net *net, } } + peer_init = (struct sctp_init_chunk *)(chunk->subh.cookie_hdr + 1); + cid = peer_init->chunk_hdr.type; + if (!sctp_sk(sk)->hmac && + !sctp_verify_init(net, ep, asoc, cid, peer_init, chunk, + &err_chk_p)) { + sctp_association_free(new_asoc); + if (err_chk_p) + sctp_chunk_free(err_chk_p); + return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands); + } + if (err_chk_p) + sctp_chunk_free(err_chk_p); + if (security_sctp_assoc_request(new_asoc, chunk->head_skb ?: chunk->skb)) { sctp_association_free(new_asoc); return sctp_sf_pdiscard(net, ep, asoc, type, arg, commands); @@ -798,7 +812,6 @@ enum sctp_disposition sctp_sf_do_5_1D_ce(struct net *net, /* This is a brand-new association, so these are not yet side * effects--it is safe to run them here. */ - peer_init = (struct sctp_init_chunk *)(chunk->subh.cookie_hdr + 1); if (!sctp_process_init(new_asoc, chunk, &chunk->subh.cookie_hdr->c.peer_addr, peer_init, GFP_ATOMIC)) @@ -2215,10 +2228,12 @@ enum sctp_disposition sctp_sf_do_5_2_4_dupcook( void *arg, struct sctp_cmd_seq *commands) { + struct sctp_chunk *err_chk_p = NULL; struct sctp_association *new_asoc; + struct sctp_init_chunk *peer_init; struct sctp_chunk *chunk = arg; enum sctp_disposition retval; - struct sctp_chunk *err_chk_p; + enum sctp_cid cid; int error = 0; char action; @@ -2287,6 +2302,21 @@ enum sctp_disposition sctp_sf_do_5_2_4_dupcook( switch (action) { case 'A': /* Association restart. */ case 'B': /* Collision case B. */ + peer_init = (struct sctp_init_chunk *) + (chunk->subh.cookie_hdr + 1); + cid = peer_init->chunk_hdr.type; + if (!sctp_sk(ep->base.sk)->hmac && + !sctp_verify_init(net, ep, asoc, cid, peer_init, chunk, + &err_chk_p)) { + sctp_association_free(new_asoc); + if (err_chk_p) + sctp_chunk_free(err_chk_p); + return sctp_sf_pdiscard(net, ep, asoc, type, arg, + commands); + } + if (err_chk_p) + sctp_chunk_free(err_chk_p); + fallthrough; case 'D': /* Collision case D. */ /* Update socket peer label if first association. */ if (security_sctp_assoc_request((struct sctp_association *)asoc, diff --git a/net/sctp/socket.c b/net/sctp/socket.c index 4cc73e7b3c2a..113e02e89403 100644 --- a/net/sctp/socket.c +++ b/net/sctp/socket.c @@ -1843,7 +1843,7 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc, if (asoc->ep->intl_enable) { timeo = sock_sndtimeo(sk, 0); err = sctp_wait_for_connect(asoc, &timeo); - if (err) { + if (err || sk->sk_shutdown & RCV_SHUTDOWN) { err = -ESRCH; goto err; } @@ -7523,11 +7523,7 @@ static int sctp_getsockopt_pr_streamstatus(struct sock *sk, int len, /* Not allocated yet, means all stats are 0 */ params.sprstat_abandoned_unsent = 0; params.sprstat_abandoned_sent = 0; - retval = 0; - goto out; - } - - if (policy == SCTP_PR_SCTP_ALL) { + } else if (policy == SCTP_PR_SCTP_ALL) { params.sprstat_abandoned_unsent = 0; params.sprstat_abandoned_sent = 0; for (policy = 0; policy <= SCTP_PR_INDEX(MAX); policy++) { diff --git a/net/smc/af_smc.c b/net/smc/af_smc.c index e5b5490da251..db380048d06b 100644 --- a/net/smc/af_smc.c +++ b/net/smc/af_smc.c @@ -2777,6 +2777,7 @@ static int smc_getname(struct socket *sock, struct sockaddr *addr, int peer) { struct smc_sock *smc; + int rc = -EBADF; if (peer && (sock->sk->sk_state != SMC_ACTIVE) && (sock->sk->sk_state != SMC_APPCLOSEWAIT1)) @@ -2784,7 +2785,11 @@ static int smc_getname(struct socket *sock, struct sockaddr *addr, smc = smc_sk(sock->sk); - return smc->clcsock->ops->getname(smc->clcsock, addr, peer); + mutex_lock(&smc->clcsock_release_lock); + if (smc->clcsock) + rc = smc->clcsock->ops->getname(smc->clcsock, addr, peer); + mutex_unlock(&smc->clcsock_release_lock); + return rc; } static int smc_sendmsg(struct socket *sock, struct msghdr *msg, size_t len) @@ -3365,11 +3370,10 @@ int smc_create_clcsk(struct net *net, struct sock *sk, int family) return 0; } -static int __smc_create(struct net *net, struct socket *sock, int protocol, - int kern, struct socket *clcsock) +static int smc_create(struct net *net, struct socket *sock, int protocol, + int kern) { int family = (protocol == SMCPROTO_SMC6) ? PF_INET6 : PF_INET; - struct smc_sock *smc; struct sock *sk; int rc; @@ -3388,88 +3392,17 @@ static int __smc_create(struct net *net, struct socket *sock, int protocol, if (!sk) goto out; - /* create internal TCP socket for CLC handshake and fallback */ - smc = smc_sk(sk); - - rc = 0; - if (clcsock) - smc->clcsock = clcsock; - else - rc = smc_create_clcsk(net, sk, family); + rc = smc_create_clcsk(net, sk, family); out: return rc; } -static int smc_create(struct net *net, struct socket *sock, int protocol, - int kern) -{ - return __smc_create(net, sock, protocol, kern, NULL); -} - static const struct net_proto_family smc_sock_family_ops = { .family = PF_SMC, .owner = THIS_MODULE, .create = smc_create, }; -static int smc_ulp_init(struct sock *sk) -{ - struct socket *tcp = sk->sk_socket; - struct net *net = sock_net(sk); - struct socket *smcsock; - int protocol, ret; - - /* only TCP can be replaced */ - if (tcp->type != SOCK_STREAM || sk->sk_protocol != IPPROTO_TCP || - (sk->sk_family != AF_INET && sk->sk_family != AF_INET6)) - return -ESOCKTNOSUPPORT; - /* don't handle wq now */ - if (tcp->state != SS_UNCONNECTED || !tcp->file || tcp->wq.fasync_list) - return -ENOTCONN; - - if (sk->sk_family == AF_INET) - protocol = SMCPROTO_SMC; - else - protocol = SMCPROTO_SMC6; - - smcsock = sock_alloc(); - if (!smcsock) - return -ENFILE; - - smcsock->type = SOCK_STREAM; - __module_get(THIS_MODULE); /* tried in __tcp_ulp_find_autoload */ - ret = __smc_create(net, smcsock, protocol, 1, tcp); - if (ret) { - sock_release(smcsock); /* module_put() which ops won't be NULL */ - return ret; - } - - /* replace tcp socket to smc */ - smcsock->file = tcp->file; - smcsock->file->private_data = smcsock; - smcsock->file->f_inode = SOCK_INODE(smcsock); /* replace inode when sock_close */ - smcsock->file->f_path.dentry->d_inode = SOCK_INODE(smcsock); /* dput() in __fput */ - tcp->file = NULL; - - return ret; -} - -static void smc_ulp_clone(const struct request_sock *req, struct sock *newsk, - const gfp_t priority) -{ - struct inet_connection_sock *icsk = inet_csk(newsk); - - /* don't inherit ulp ops to child when listen */ - icsk->icsk_ulp_ops = NULL; -} - -static struct tcp_ulp_ops smc_ulp_ops __read_mostly = { - .name = "smc", - .owner = THIS_MODULE, - .init = smc_ulp_init, - .clone = smc_ulp_clone, -}; - unsigned int smc_net_id; static __net_init int smc_net_init(struct net *net) @@ -3590,18 +3523,8 @@ static int __init smc_init(void) pr_err("%s: ib_register fails with %d\n", __func__, rc); goto out_sock; } - - rc = tcp_register_ulp(&smc_ulp_ops); - if (rc) { - pr_err("%s: tcp_ulp_register fails with %d\n", __func__, rc); - goto out_ib; - } - static_branch_enable(&tcp_have_smc); return 0; - -out_ib: - smc_ib_unregister_client(); out_sock: sock_unregister(PF_SMC); out_proto6: @@ -3634,7 +3557,6 @@ static int __init smc_init(void) static void __exit smc_exit(void) { static_branch_disable(&tcp_have_smc); - tcp_unregister_ulp(&smc_ulp_ops); sock_unregister(PF_SMC); smc_core_exit(); smc_ib_unregister_client(); @@ -3659,5 +3581,4 @@ MODULE_AUTHOR("Ursula Braun "); MODULE_DESCRIPTION("smc socket address family"); MODULE_LICENSE("GPL"); MODULE_ALIAS_NETPROTO(PF_SMC); -MODULE_ALIAS_TCP_ULP("smc"); MODULE_ALIAS_GENL_FAMILY(SMC_GENL_FAMILY_NAME); diff --git a/net/smc/smc.h b/net/smc/smc.h index 49bf6971610d..b19f41e7e117 100644 --- a/net/smc/smc.h +++ b/net/smc/smc.h @@ -238,9 +238,9 @@ struct smc_connection { * 0 for SMC-R, 32 for SMC-D */ u64 peer_token; /* SMC-D token of peer */ - u8 killed : 1; /* abnormal termination */ - u8 freed : 1; /* normal termiation */ - u8 out_of_sync : 1; /* out of sync with peer */ + u8 killed; /* abnormal termination */ + u8 freed; /* normal termination */ + u8 out_of_sync; /* out of sync with peer */ }; struct smc_sock { /* smc sock container */ diff --git a/net/smc/smc_clc.h b/net/smc/smc_clc.h index 6e24d44de4a7..fc31eeb586b8 100644 --- a/net/smc/smc_clc.h +++ b/net/smc/smc_clc.h @@ -150,7 +150,7 @@ struct smc_clc_msg_proposal_prefix { /* prefix part of clc proposal message*/ } __aligned(4); struct smc_clc_msg_smcd { /* SMC-D GID information */ - struct smc_clc_smcd_gid_chid ism; /* ISM native GID+CHID of requestor */ + struct smc_clc_smcd_gid_chid ism; /* ISM native GID+CHID of requester */ __be16 v2_ext_offset; /* SMC Version 2 Extension Offset */ u8 vendor_oui[3]; /* vendor organizationally unique identifier */ u8 vendor_exp_options[5]; diff --git a/net/smc/smc_core.c b/net/smc/smc_core.c index 3694a5a5918a..5a39031611b6 100644 --- a/net/smc/smc_core.c +++ b/net/smc/smc_core.c @@ -2275,7 +2275,7 @@ static struct smc_buf_desc *smcr_new_buf_create(struct smc_link_group *lgr, } if (lgr->buf_type == SMCR_PHYS_CONT_BUFS) goto out; - fallthrough; // try virtually continguous buf + fallthrough; // try virtually contiguous buf case SMCR_VIRT_CONT_BUFS: buf_desc->order = get_order(bufsize); buf_desc->cpu_addr = vzalloc(PAGE_SIZE << buf_desc->order); diff --git a/net/smc/smc_core.h b/net/smc/smc_core.h index de001f4b46c7..3694949b243d 100644 --- a/net/smc/smc_core.h +++ b/net/smc/smc_core.h @@ -30,7 +30,7 @@ */ #define SMC_CONN_PER_LGR_PREFER 255 /* Preferred connections per link group used for * SMC-R v2.1 and later negotiation, vendors or - * distrubutions may modify it to a value between + * distributions may modify it to a value between * 16-255 as needed. */ @@ -181,7 +181,7 @@ struct smc_link { */ #define SMC_LINKS_PER_LGR_MAX_PREFER 2 /* Preferred max links per link group used for * SMC-R v2.1 and later negotiation, vendors or - * distrubutions may modify it to a value between + * distributions may modify it to a value between * 1-2 as needed. */ diff --git a/net/strparser/strparser.c b/net/strparser/strparser.c index 2a805c964210..be422d077ec7 100644 --- a/net/strparser/strparser.c +++ b/net/strparser/strparser.c @@ -518,12 +518,16 @@ void strp_unpause(struct strparser *strp) EXPORT_SYMBOL_GPL(strp_unpause); /* strp must already be stopped so that strp_recv will no longer be called. - * Note that strp_done is not called with the lower socket held. + * Note that strp_done must not be called with the lower socket held. */ void strp_done(struct strparser *strp) { WARN_ON(!strp->stopped); + lock_sock(strp->sk); + /* sync with pending strp_recv */ + release_sock(strp->sk); + cancel_delayed_work_sync(&strp->msg_timer_work); cancel_work_sync(&strp->work); diff --git a/net/sunrpc/auth_gss/gss_krb5_unseal.c b/net/sunrpc/auth_gss/gss_krb5_unseal.c index 4fbc50a0a2c4..1a98060dae2c 100644 --- a/net/sunrpc/auth_gss/gss_krb5_unseal.c +++ b/net/sunrpc/auth_gss/gss_krb5_unseal.c @@ -88,6 +88,9 @@ gss_krb5_verify_mic_v2(struct krb5_ctx *ctx, struct xdr_buf *message_buffer, dprintk("RPC: %s\n", __func__); + if (read_token->len < GSS_KRB5_TOK_HDR_LEN + ctx->gk5e->cksumlength) + return GSS_S_DEFECTIVE_TOKEN; + memcpy(&be16_ptr, (char *) ptr, 2); if (be16_to_cpu(be16_ptr) != KG2_TOK_MIC) return GSS_S_DEFECTIVE_TOKEN; diff --git a/net/sunrpc/auth_gss/gss_rpc_xdr.c b/net/sunrpc/auth_gss/gss_rpc_xdr.c index ee91f4d641e6..967943ae61c6 100644 --- a/net/sunrpc/auth_gss/gss_rpc_xdr.c +++ b/net/sunrpc/auth_gss/gss_rpc_xdr.c @@ -222,7 +222,8 @@ static int gssx_dec_linux_creds(struct xdr_stream *xdr, return 0; out_free_groups: - groups_free(creds->cr_group_info); + put_group_info(creds->cr_group_info); + creds->cr_group_info = NULL; return err; } @@ -230,6 +231,7 @@ static int gssx_dec_option_array(struct xdr_stream *xdr, struct gssx_option_array *oa) { struct svc_cred *creds; + bool creds_decoded = false; u32 count, i; __be32 *p; int err; @@ -242,12 +244,12 @@ static int gssx_dec_option_array(struct xdr_stream *xdr, return 0; /* we recognize only 1 currently: CREDS_VALUE */ - oa->count = 1; - oa->data = kmalloc(sizeof(struct gssx_option), GFP_KERNEL); if (!oa->data) return -ENOMEM; + oa->count = 1; + creds = kzalloc(sizeof(struct svc_cred), GFP_KERNEL); if (!creds) { err = -ENOMEM; @@ -280,9 +282,14 @@ static int gssx_dec_option_array(struct xdr_stream *xdr, if (length == sizeof(CREDS_VALUE) && memcmp(p, CREDS_VALUE, sizeof(CREDS_VALUE)) == 0) { /* We have creds here. parse them */ + if (creds_decoded) { + err = -EINVAL; + goto free_creds; + } err = gssx_dec_linux_creds(xdr, creds); if (err) goto free_creds; + creds_decoded = true; oa->data[0].value.len = 1; /* presence */ } else { /* consume uninteresting buffer */ @@ -294,8 +301,10 @@ static int gssx_dec_option_array(struct xdr_stream *xdr, return 0; free_creds: + free_svc_cred(creds); kfree(creds); free_oa: + oa->count = 0; kfree(oa->data); oa->data = NULL; return err; diff --git a/net/sunrpc/clnt.c b/net/sunrpc/clnt.c index 85cfd7b7c212..29b73519220f 100644 --- a/net/sunrpc/clnt.c +++ b/net/sunrpc/clnt.c @@ -101,7 +101,10 @@ static void rpc_unregister_client(struct rpc_clnt *clnt) static void __rpc_clnt_remove_pipedir(struct rpc_clnt *clnt) { - rpc_remove_client_dir(clnt); + if (clnt->pipefs_sb) { + rpc_remove_client_dir(clnt); + clnt->pipefs_sb = NULL; + } } static void rpc_clnt_remove_pipedir(struct rpc_clnt *clnt) @@ -183,26 +186,33 @@ static int rpc_clnt_skip_event(struct rpc_clnt *clnt, unsigned long event) } static int __rpc_clnt_handle_event(struct rpc_clnt *clnt, unsigned long event, - struct super_block *sb) + struct super_block *sb) { struct dentry *dentry; + int err = 0; switch (event) { case RPC_PIPEFS_MOUNT: + clnt->pipefs_sb = sb; dentry = rpc_setup_pipedir_sb(sb, clnt); if (!dentry) - return -ENOENT; - if (IS_ERR(dentry)) - return PTR_ERR(dentry); + err = -ENOENT; + else if (IS_ERR(dentry)) + err = PTR_ERR(dentry); + if (err) + clnt->pipefs_sb = NULL; break; case RPC_PIPEFS_UMOUNT: - __rpc_clnt_remove_pipedir(clnt); + if (clnt->pipefs_sb == sb) { + __rpc_clnt_remove_pipedir(clnt); + clnt->pipefs_sb = NULL; + } break; default: printk(KERN_ERR "%s: unknown event: %ld\n", __func__, event); return -ENOTSUPP; } - return 0; + return err; } static int __rpc_pipefs_event(struct rpc_clnt *clnt, unsigned long event, diff --git a/net/sunrpc/xprtrdma/svc_rdma_pcl.c b/net/sunrpc/xprtrdma/svc_rdma_pcl.c index b63cfeaa2923..a1c61311714d 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_pcl.c +++ b/net/sunrpc/xprtrdma/svc_rdma_pcl.c @@ -213,7 +213,6 @@ bool pcl_alloc_write(struct svc_rdma_recv_ctxt *rctxt, chunk = pcl_alloc_chunk(segcount, 0); if (!chunk) return false; - list_add_tail(&chunk->ch_list, &pcl->cl_chunks); for (j = 0; j < segcount; j++) { segment = &chunk->ch_segments[j]; @@ -225,6 +224,7 @@ bool pcl_alloc_write(struct svc_rdma_recv_ctxt *rctxt, chunk->ch_length += segment->rs_length; chunk->ch_segcount++; } + list_add_tail(&chunk->ch_list, &pcl->cl_chunks); } return true; } diff --git a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c index 9cec7bcb8a97..0cdef62a160f 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c +++ b/net/sunrpc/xprtrdma/svc_rdma_recvfrom.c @@ -411,11 +411,14 @@ static void svc_rdma_build_arg_xdr(struct svc_rqst *rqstp, * to the first byte past the Read list. rc_read_pcl and * rc_call_pcl cl_count fields are set to the number of * Read segments in the list. - * %false: Read list is corrupt. @rctxt's xdr_stream is left in an - * unknown state. + * %false: Read list is corrupt or exceeds the page budget. @rctxt's + * xdr_stream is left in an unknown state. */ static bool xdr_count_read_segments(struct svc_rdma_recv_ctxt *rctxt, __be32 *p) { + unsigned int maxlen = rctxt->rc_maxpages << PAGE_SHIFT; + unsigned int total_len = 0; + rctxt->rc_call_pcl.cl_count = 0; rctxt->rc_read_pcl.cl_count = 0; while (xdr_item_is_present(p)) { @@ -429,6 +432,11 @@ static bool xdr_count_read_segments(struct svc_rdma_recv_ctxt *rctxt, __be32 *p) xdr_decode_read_segment(p, &position, &handle, &length, &offset); + if (length > maxlen) + return false; + total_len += length; + if (PAGE_ALIGN(total_len) > maxlen) + return false; if (position) { if (position & 3) return false; @@ -479,10 +487,13 @@ static bool xdr_check_write_chunk(struct svc_rdma_recv_ctxt *rctxt) return false; /* Before trusting the segcount value enough to use it in - * a computation, perform a simple range check. This is an - * arbitrary but sensible limit (ie, not architectural). + * a computation, perform a simple range check. A zero + * segcount describes no remote buffer and is rejected so + * downstream consumers never see a degenerate ch_segcount==0 + * chunk. The upper bound is an arbitrary but sensible limit + * (ie, not architectural). */ - if (unlikely(segcount > RPCSVC_MAXPAGES)) + if (segcount == 0 || unlikely(segcount > RPCSVC_MAXPAGES)) return false; p = xdr_inline_decode(&rctxt->rc_stream, @@ -824,6 +835,7 @@ int svc_rdma_recvfrom(struct svc_rqst *rqstp) ctxt->rc_recv_sge.addr, ctxt->rc_byte_len, DMA_FROM_DEVICE); svc_rdma_build_arg_xdr(rqstp, ctxt); + ctxt->rc_maxpages = rqstp->rq_page_end - rqstp->rq_pages; ret = svc_rdma_xdr_decode_req(&rqstp->rq_arg, ctxt); if (ret < 0) diff --git a/net/sunrpc/xprtrdma/svc_rdma_rw.c b/net/sunrpc/xprtrdma/svc_rdma_rw.c index 347db3b7e0bc..e9edbcfec28c 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_rw.c +++ b/net/sunrpc/xprtrdma/svc_rdma_rw.c @@ -790,7 +790,7 @@ static int svc_rdma_build_read_chunk(struct svc_rdma_read_info *info, /** * svc_rdma_copy_inline_range - Copy part of the inline content into pages * @info: context for RDMA Reads - * @offset: offset into the Receive buffer of region to copy + * @offset: offset into the inline content of region to copy * @remaining: length of region to copy * * Take a page at a time from rqstp->rq_pages and copy the inline @@ -807,10 +807,14 @@ static int svc_rdma_copy_inline_range(struct svc_rdma_read_info *info, unsigned int remaining) { struct svc_rdma_recv_ctxt *head = info->ri_readctxt; - unsigned char *dst, *src = head->rc_recv_buf; struct svc_rqst *rqstp = info->ri_rqst; + unsigned char *dst, *src = rqstp->rq_arg.head[0].iov_base; + unsigned int inline_len = rqstp->rq_arg.head[0].iov_len; unsigned int page_no, numpages; + if (offset > inline_len || remaining > inline_len - offset) + return -EINVAL; + numpages = PAGE_ALIGN(info->ri_pageoff + remaining) >> PAGE_SHIFT; for (page_no = 0; page_no < numpages; page_no++) { unsigned int page_len; @@ -860,9 +864,10 @@ static noinline int svc_rdma_read_multiple_chunks(struct svc_rdma_read_info *inf const struct svc_rdma_pcl *pcl = &head->rc_read_pcl; struct xdr_buf *buf = &info->ri_rqst->rq_arg; struct svc_rdma_chunk *chunk, *next; - unsigned int start, length; + unsigned int inline_len, start, length; int ret; + inline_len = buf->head[0].iov_len; start = 0; chunk = pcl_first_chunk(pcl); length = chunk->ch_position; @@ -880,6 +885,8 @@ static noinline int svc_rdma_read_multiple_chunks(struct svc_rdma_read_info *inf break; start += length; + if (info->ri_totalbytes > next->ch_position) + return -EINVAL; length = next->ch_position - info->ri_totalbytes; ret = svc_rdma_copy_inline_range(info, start, length); if (ret < 0) @@ -887,7 +894,9 @@ static noinline int svc_rdma_read_multiple_chunks(struct svc_rdma_read_info *inf } start += length; - length = head->rc_byte_len - start; + if (start > inline_len) + return -EINVAL; + length = inline_len - start; ret = svc_rdma_copy_inline_range(info, start, length); if (ret < 0) return ret; @@ -928,6 +937,9 @@ static int svc_rdma_read_data_item(struct svc_rdma_read_info *info) int ret; chunk = pcl_first_chunk(&head->rc_read_pcl); + if (chunk->ch_position > buf->head[0].iov_len) + return -EINVAL; + ret = svc_rdma_build_read_chunk(info, chunk); if (ret < 0) goto out; @@ -1028,14 +1040,17 @@ static int svc_rdma_read_call_chunk(struct svc_rdma_read_info *info) pcl_first_chunk(&head->rc_call_pcl); const struct svc_rdma_pcl *pcl = &head->rc_read_pcl; struct svc_rdma_chunk *chunk, *next; - unsigned int start, length; + unsigned int call_len, start, length; int ret; if (pcl_is_empty(pcl)) return svc_rdma_build_read_chunk(info, call_chunk); + call_len = call_chunk->ch_length; start = 0; chunk = pcl_first_chunk(pcl); + if (chunk->ch_position > call_len) + return -EINVAL; length = chunk->ch_position; ret = svc_rdma_read_chunk_range(info, call_chunk, start, length); if (ret < 0) @@ -1051,6 +1066,8 @@ static int svc_rdma_read_call_chunk(struct svc_rdma_read_info *info) break; start += length; + if (info->ri_totalbytes > next->ch_position) + return -EINVAL; length = next->ch_position - info->ri_totalbytes; ret = svc_rdma_read_chunk_range(info, call_chunk, start, length); @@ -1059,7 +1076,9 @@ static int svc_rdma_read_call_chunk(struct svc_rdma_read_info *info) } start += length; - length = call_chunk->ch_length - start; + if (start > call_len) + return -EINVAL; + length = call_len - start; return svc_rdma_read_chunk_range(info, call_chunk, start, length); } diff --git a/net/sunrpc/xprtrdma/svc_rdma_sendto.c b/net/sunrpc/xprtrdma/svc_rdma_sendto.c index c6644cca52c5..c7800779ac93 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_sendto.c +++ b/net/sunrpc/xprtrdma/svc_rdma_sendto.c @@ -650,20 +650,21 @@ static int svc_rdma_xb_count_sges(const struct xdr_buf *xdr, } /** - * svc_rdma_pull_up_needed - Determine whether to use pull-up + * svc_rdma_check_pull_up - Determine whether to use pull-up * @rdma: controlling transport * @sctxt: send_ctxt for the Send WR * @rctxt: Write and Reply chunks provided by client * @xdr: xdr_buf containing RPC message to transmit * * Returns: - * %true if pull-up must be used - * %false otherwise + * %1 if pull-up must be used + * %0 if pull-up is not needed + * %-E2BIG if the reply is too large to be pulled up */ -static bool svc_rdma_pull_up_needed(const struct svcxprt_rdma *rdma, - const struct svc_rdma_send_ctxt *sctxt, - const struct svc_rdma_recv_ctxt *rctxt, - const struct xdr_buf *xdr) +static int svc_rdma_check_pull_up(const struct svcxprt_rdma *rdma, + const struct svc_rdma_send_ctxt *sctxt, + const struct svc_rdma_recv_ctxt *rctxt, + const struct xdr_buf *xdr) { /* Resources needed for the transport header */ struct svc_rdma_pullup_data args = { @@ -675,11 +676,22 @@ static bool svc_rdma_pull_up_needed(const struct svcxprt_rdma *rdma, ret = pcl_process_nonpayloads(&rctxt->rc_write_pcl, xdr, svc_rdma_xb_count_sges, &args); if (ret < 0) - return false; + return 0; if (args.pd_length < RPCRDMA_PULLUP_THRESH) - return true; - return args.pd_num_sges >= rdma->sc_max_send_sges; + return 1; + if (args.pd_num_sges < rdma->sc_max_send_sges) + return 0; + + /* + * The reply has too many SGEs to Send inline, so it has to be + * linearized into sc_xprt_buf. That buffer holds only + * sc_max_req_size bytes, so a larger reply cannot be pulled up. + * RFC 8166 Section 4.5.3 requires responding with ERR_CHUNK. + */ + if (args.pd_length > rdma->sc_max_req_size) + return -E2BIG; + return 1; } /** @@ -735,7 +747,7 @@ static int svc_rdma_xb_linearize(const struct xdr_buf *xdr, * Assemble the elements of @xdr into the transport header buffer. * * Assumptions: - * pull_up_needed has determined that @xdr will fit in the buffer. + * check_pull_up has determined that @xdr will fit in the buffer. * * Returns: * %0 if pull-up was successful @@ -769,6 +781,7 @@ static int svc_rdma_pull_up_reply_msg(const struct svcxprt_rdma *rdma, * * Returns: * %0 if DMA mapping was successful. + * %-E2BIG if the reply is too large to be pulled up * %-EMSGSIZE if a buffer manipulation problem occurred * %-EIO if DMA mapping failed * @@ -783,6 +796,7 @@ int svc_rdma_map_reply_msg(struct svcxprt_rdma *rdma, .md_rdma = rdma, .md_ctxt = sctxt, }; + int ret; /* Set up the (persistently-mapped) transport header SGE. */ sctxt->sc_send_wr.num_sge = 1; @@ -797,7 +811,10 @@ int svc_rdma_map_reply_msg(struct svcxprt_rdma *rdma, /* For pull-up, svc_rdma_send() will sync the transport header. * No additional DMA mapping is necessary. */ - if (svc_rdma_pull_up_needed(rdma, sctxt, rctxt, xdr)) + ret = svc_rdma_check_pull_up(rdma, sctxt, rctxt, xdr); + if (ret < 0) + return ret; + if (ret) return svc_rdma_pull_up_reply_msg(rdma, sctxt, rctxt, xdr); return pcl_process_nonpayloads(&rctxt->rc_write_pcl, xdr, @@ -974,7 +991,7 @@ int svc_rdma_sendto(struct svc_rqst *rqstp) ret = svc_rdma_send_reply_chunk(rdma, rctxt, &rqstp->rq_res); if (ret < 0) - goto reply_chunk; + goto send_err; rc_size = ret; *p++ = *rdma_argp; @@ -994,10 +1011,10 @@ int svc_rdma_sendto(struct svc_rqst *rqstp) ret = svc_rdma_send_reply_msg(rdma, sctxt, rctxt, rqstp); if (ret < 0) - goto put_ctxt; + goto send_err; return 0; -reply_chunk: +send_err: if (ret != -E2BIG && ret != -EINVAL) goto put_ctxt; diff --git a/net/sunrpc/xprtrdma/svc_rdma_transport.c b/net/sunrpc/xprtrdma/svc_rdma_transport.c index b8d13b522298..85867dba0c01 100644 --- a/net/sunrpc/xprtrdma/svc_rdma_transport.c +++ b/net/sunrpc/xprtrdma/svc_rdma_transport.c @@ -65,6 +65,8 @@ static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv, struct net *net, int node); +static int svc_rdma_listen_handler(struct rdma_cm_id *cma_id, + struct rdma_cm_event *event); static struct svc_xprt *svc_rdma_create(struct svc_serv *serv, struct net *net, struct sockaddr *sa, int salen, @@ -122,6 +124,41 @@ static void qp_event_handler(struct ib_event *event, void *context) } } +static struct rdma_cm_id * +svc_rdma_create_listen_id(struct net *net, struct sockaddr *sap, + void *context) +{ + struct rdma_cm_id *listen_id; + int ret; + + listen_id = rdma_create_id(net, svc_rdma_listen_handler, context, + RDMA_PS_TCP, IB_QPT_RC); + if (IS_ERR(listen_id)) + return listen_id; + + /* Allow both IPv4 and IPv6 sockets to bind a single port + * at the same time. + */ +#if IS_ENABLED(CONFIG_IPV6) + ret = rdma_set_afonly(listen_id, 1); + if (ret) + goto out_destroy; +#endif + ret = rdma_bind_addr(listen_id, sap); + if (ret) + goto out_destroy; + + ret = rdma_listen(listen_id, RPCRDMA_LISTEN_BACKLOG); + if (ret) + goto out_destroy; + + return listen_id; + +out_destroy: + rdma_destroy_id(listen_id); + return ERR_PTR(ret); +} + static struct svcxprt_rdma *svc_rdma_create_xprt(struct svc_serv *serv, struct net *net, int node) { @@ -182,12 +219,16 @@ svc_rdma_parse_connect_private(struct svcxprt_rdma *newxprt, * structure for the listening endpoint. * * This function creates a new xprt for the new connection and enqueues it on - * the accept queue for the listent xprt. When the listen thread is kicked, it + * the accept queue for the listen xprt. When the listen thread is kicked, it * will call the recvfrom method on the listen xprt which will accept the new * connection. + * + * Return values: + * %0: Do not destroy @new_cma_id + * %1: Destroy @new_cma_id (allocation failure) */ -static void handle_connect_req(struct rdma_cm_id *new_cma_id, - struct rdma_conn_param *param) +static int handle_connect_req(struct rdma_cm_id *new_cma_id, + struct rdma_conn_param *param) { struct svcxprt_rdma *listen_xprt = new_cma_id->context; struct svcxprt_rdma *newxprt; @@ -197,7 +238,7 @@ static void handle_connect_req(struct rdma_cm_id *new_cma_id, listen_xprt->sc_xprt.xpt_net, ibdev_to_node(new_cma_id->device)); if (!newxprt) - return; + return 1; newxprt->sc_cm_id = new_cma_id; new_cma_id->context = newxprt; svc_rdma_parse_connect_private(newxprt, param); @@ -231,6 +272,7 @@ static void handle_connect_req(struct rdma_cm_id *new_cma_id, set_bit(XPT_CONN, &listen_xprt->sc_xprt.xpt_flags); svc_xprt_enqueue(&listen_xprt->sc_xprt); + return 0; } /** @@ -240,17 +282,30 @@ static void handle_connect_req(struct rdma_cm_id *new_cma_id, * * Return values: * %0: Do not destroy @cma_id - * %1: Destroy @cma_id (never returned here) + * %1: Destroy @cma_id * * NB: There is never a DEVICE_REMOVAL event for INADDR_ANY listeners. */ static int svc_rdma_listen_handler(struct rdma_cm_id *cma_id, struct rdma_cm_event *event) { + struct sockaddr *sap = (struct sockaddr *)&cma_id->route.addr.src_addr; + struct svcxprt_rdma *cma_xprt = cma_id->context; + struct svc_xprt *cma_rdma = &cma_xprt->sc_xprt; + struct rdma_cm_id *listen_id; + switch (event->event) { case RDMA_CM_EVENT_CONNECT_REQUEST: - handle_connect_req(cma_id, &event->param.conn); - break; + return handle_connect_req(cma_id, &event->param.conn); + case RDMA_CM_EVENT_ADDR_CHANGE: + listen_id = svc_rdma_create_listen_id(cma_rdma->xpt_net, + sap, cma_xprt); + if (IS_ERR(listen_id)) { + pr_err("Listener dead, address change failed for device %s\n", + cma_id->device->name); + } else + cma_xprt->sc_cm_id = listen_id; + return 1; default: break; } @@ -300,7 +355,6 @@ static struct svc_xprt *svc_rdma_create(struct svc_serv *serv, { struct rdma_cm_id *listen_id; struct svcxprt_rdma *cma_xprt; - int ret; if (sa->sa_family != AF_INET && sa->sa_family != AF_INET6) return ERR_PTR(-EAFNOSUPPORT); @@ -310,30 +364,19 @@ static struct svc_xprt *svc_rdma_create(struct svc_serv *serv, set_bit(XPT_LISTENER, &cma_xprt->sc_xprt.xpt_flags); strcpy(cma_xprt->sc_xprt.xpt_remotebuf, "listener"); - listen_id = rdma_create_id(net, svc_rdma_listen_handler, cma_xprt, - RDMA_PS_TCP, IB_QPT_RC); + listen_id = svc_rdma_create_listen_id(net, sa, cma_xprt); if (IS_ERR(listen_id)) { - ret = PTR_ERR(listen_id); - goto err0; + /* _svc_xprt_create() acquired one module reference and + * puts it on xpo_create failure. svc_xprt_free() puts + * a second one when the kref drops to zero. Take a + * compensating reference so both puts are balanced. + */ + __module_get(cma_xprt->sc_xprt.xpt_class->xcl_owner); + svc_xprt_put(&cma_xprt->sc_xprt); + return ERR_CAST(listen_id); } - - /* Allow both IPv4 and IPv6 sockets to bind a single port - * at the same time. - */ -#if IS_ENABLED(CONFIG_IPV6) - ret = rdma_set_afonly(listen_id, 1); - if (ret) - goto err1; -#endif - ret = rdma_bind_addr(listen_id, sa); - if (ret) - goto err1; cma_xprt->sc_cm_id = listen_id; - ret = rdma_listen(listen_id, RPCRDMA_LISTEN_BACKLOG); - if (ret) - goto err1; - /* * We need to use the address from the cm_id in case the * caller specified 0 for the port number. @@ -342,12 +385,6 @@ static struct svc_xprt *svc_rdma_create(struct svc_serv *serv, svc_xprt_set_local(&cma_xprt->sc_xprt, sa, salen); return &cma_xprt->sc_xprt; - - err1: - rdma_destroy_id(listen_id); - err0: - kfree(cma_xprt); - return ERR_PTR(ret); } /* diff --git a/net/sunrpc/xprtrdma/verbs.c b/net/sunrpc/xprtrdma/verbs.c index 34a3c08a82fb..4990e37bcdc0 100644 --- a/net/sunrpc/xprtrdma/verbs.c +++ b/net/sunrpc/xprtrdma/verbs.c @@ -256,8 +256,17 @@ rpcrdma_cm_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event) } return 0; case RDMA_CM_EVENT_ADDR_CHANGE: - ep->re_connect_status = -ENODEV; - goto disconnected; + switch (xchg(&ep->re_connect_status, -ENODEV)) { + case 0: + goto wake_connect_worker; + case 1: + /* The later DISCONNECTED event balances the + * ESTABLISHED get; do not put here. + */ + rpcrdma_force_disconnect(ep); + return 0; + } + return 0; case RDMA_CM_EVENT_ESTABLISHED: rpcrdma_ep_get(ep); ep->re_connect_status = 1; diff --git a/net/tipc/core.c b/net/tipc/core.c index 1ddecea1df6e..315975c3be81 100644 --- a/net/tipc/core.c +++ b/net/tipc/core.c @@ -45,6 +45,7 @@ #include "crypto.h" #include +#include /* configurable TIPC parameters */ unsigned int tipc_net_id __read_mostly; @@ -118,8 +119,7 @@ static void __net_exit tipc_exit_net(struct net *net) #ifdef CONFIG_TIPC_CRYPTO tipc_crypto_stop(&tipc_net(net)->crypto_tx); #endif - while (atomic_read(&tn->wq_count)) - cond_resched(); + wait_var_event(&tn->wq_count, atomic_read(&tn->wq_count) == 0); } static void __net_exit tipc_pernet_pre_exit(struct net *net) diff --git a/net/tipc/crypto.c b/net/tipc/crypto.c index d58db40d6c03..9c3ef67eb5f4 100644 --- a/net/tipc/crypto.c +++ b/net/tipc/crypto.c @@ -144,7 +144,7 @@ struct tipc_tfm { * @rcu: struct rcu_head * @key: the aead key * @gen: the key's generation - * @seqno: the key seqno (cluster scope) + * @seqno: the per-key TX nonce counter * @refcnt: the key reference counter */ struct tipc_aead { @@ -190,7 +190,6 @@ struct tipc_crypto_stats { * @rekeying_intv: rekeying interval (in minutes) * @stats: the crypto statistics * @name: the crypto name - * @sndnxt: the per-peer sndnxt (TX) * @timer1: general timer 1 (jiffies) * @timer2: general timer 2 (jiffies) * @working: the crypto is working or not @@ -219,7 +218,6 @@ struct tipc_crypto { struct tipc_crypto_stats __percpu *stats; char name[48]; - atomic64_t sndnxt ____cacheline_aligned; unsigned long timer1; unsigned long timer2; union { @@ -1060,14 +1058,11 @@ static int tipc_ehdr_build(struct net *net, struct tipc_aead *aead, WARN_ON(skb_headroom(skb) < ehsz); ehdr = (struct tipc_ehdr *)skb_push(skb, ehsz); - /* Obtain a seqno first: - * Use the key seqno (= cluster wise) if dest is unknown or we're in - * cluster key mode, otherwise it's better for a per-peer seqno! + /* + * Keep the nonce unique for the lifetime of the TX key, + * including key state changes and peer reconnection. */ - if (!__rx || aead->mode == CLUSTER_KEY) - seqno = atomic64_inc_return(&aead->seqno); - else - seqno = atomic64_inc_return(&__rx->sndnxt); + seqno = atomic64_inc_return(&aead->seqno); /* Revoke the key if seqno is wrapped around */ if (unlikely(!seqno)) @@ -1246,7 +1241,6 @@ void tipc_crypto_key_flush(struct tipc_crypto *c) tipc_crypto_key_set_state(c, 0, 0, 0); for (k = KEY_MIN; k <= KEY_MAX; k++) tipc_crypto_key_detach(c->aead[k], &c->lock); - atomic64_set(&c->sndnxt, 0); spin_unlock_bh(&c->lock); } @@ -1393,8 +1387,6 @@ static struct tipc_aead *tipc_crypto_key_pick_tx(struct tipc_crypto *tx, * It also considers if peer has no key, then we need to make own master key * (if any) taking over i.e. starting grace period and also trigger key * distributing process. - * - * The "per-peer" sndnxt is also reset when the peer key has switched. */ static void tipc_crypto_key_synch(struct tipc_crypto *rx, struct sk_buff *skb) { @@ -1445,7 +1437,6 @@ static void tipc_crypto_key_synch(struct tipc_crypto *rx, struct sk_buff *skb) if (cur) tipc_aead_users_dec(tx->aead[cur], 0); - atomic64_set(&rx->sndnxt, 0); /* Mark the point TX key users changed */ tx->timer1 = jiffies; @@ -1510,7 +1501,6 @@ int tipc_crypto_start(struct tipc_crypto **crypto, struct net *net, tipc_crypto_key_set_state(c, 0, 0, 0); atomic_set(&c->key_distr, 0); atomic_set(&c->peer_rx_active, 0); - atomic64_set(&c->sndnxt, 0); c->timer1 = jiffies; c->timer2 = jiffies; c->rekeying_intv = TIPC_REKEYING_INTV_DEF; diff --git a/net/tipc/link.c b/net/tipc/link.c index 6c6d8546c578..9bdb054eafaa 100644 --- a/net/tipc/link.c +++ b/net/tipc/link.c @@ -528,7 +528,7 @@ bool tipc_link_create(struct net *net, char *if_name, int bearer_id, snprintf(l->name, sizeof(l->name), "%s:%s-%s:unknown", self_str, if_name, peer_str); - strcpy(l->if_name, if_name); + strscpy(l->if_name, if_name, sizeof(l->if_name)); l->addr = peer; l->peer_caps = peer_caps; l->net = net; @@ -599,7 +599,7 @@ bool tipc_link_bc_create(struct net *net, u32 ownnode, u32 peer, u8 *peer_id, snprintf(l->name, sizeof(l->name), "%s:%s", tipc_bclink_name, peer_str); } else { - strcpy(l->name, tipc_bclink_name); + strscpy(l->name, tipc_bclink_name, sizeof(l->name)); } trace_tipc_link_reset(l, TIPC_DUMP_ALL, "bclink created!"); tipc_link_reset(l); @@ -1923,7 +1923,7 @@ static void tipc_link_build_proto_msg(struct tipc_link *l, int mtyp, bool probe, msg_set_dest_session(hdr, l->peer_session); } msg_set_max_pkt(hdr, l->advertised_mtu); - strcpy(data, l->if_name); + memcpy(data, l->if_name, TIPC_MAX_IF_NAME); msg_set_size(hdr, INT_H_SIZE + TIPC_MAX_IF_NAME); skb_trim(skb, INT_H_SIZE + TIPC_MAX_IF_NAME); } diff --git a/net/tipc/node.c b/net/tipc/node.c index f4a25ae7dc1b..85f431ce9ea5 100644 --- a/net/tipc/node.c +++ b/net/tipc/node.c @@ -2421,8 +2421,11 @@ static struct tipc_node *tipc_node_find_by_name(struct net *net, } } tipc_node_read_unlock(n); - if (found_node) + if (found_node) { + if (!kref_get_unless_zero(&found_node->kref)) + found_node = NULL; break; + } } rcu_read_unlock(); @@ -2507,6 +2510,7 @@ int tipc_nl_node_set_link(struct sk_buff *skb, struct genl_info *info) tipc_node_read_unlock(node); tipc_bearer_xmit(net, bearer_id, &xmitq, &node->links[bearer_id].maddr, NULL); + tipc_node_put(node); return res; } @@ -2558,12 +2562,14 @@ int tipc_nl_node_get_link(struct sk_buff *skb, struct genl_info *info) link = node->links[bearer_id].link; if (!link) { tipc_node_read_unlock(node); + tipc_node_put(node); err = -EINVAL; goto err_free; } err = __tipc_nl_add_link(net, &msg, link, 0); tipc_node_read_unlock(node); + tipc_node_put(node); if (err) goto err_free; } @@ -2634,11 +2640,13 @@ int tipc_nl_node_reset_link_stats(struct sk_buff *skb, struct genl_info *info) if (!link) { spin_unlock_bh(&le->lock); tipc_node_read_unlock(node); + tipc_node_put(node); return -EINVAL; } tipc_link_reset_stats(link); spin_unlock_bh(&le->lock); tipc_node_read_unlock(node); + tipc_node_put(node); return 0; } diff --git a/net/tipc/topsrv.c b/net/tipc/topsrv.c index ffe577bf6b51..c41bdae2b6b2 100644 --- a/net/tipc/topsrv.c +++ b/net/tipc/topsrv.c @@ -55,7 +55,7 @@ /** * struct tipc_topsrv - TIPC server structure * @conn_idr: identifier set of connection - * @idr_lock: protect the connection identifier set + * @idr_lock: protect the connection identifier set and listener * @idr_in_use: amount of allocated identifier entry * @net: network namspace instance * @awork: accept work item @@ -218,6 +218,10 @@ static struct tipc_conn *tipc_conn_lookup(struct tipc_topsrv *s, int conid) struct tipc_conn *con; spin_lock_bh(&s->idr_lock); + if (!s->listener) { + spin_unlock_bh(&s->idr_lock); + return NULL; + } con = idr_find(&s->conn_idr, conid); if (!connected(con) || !kref_get_unless_zero(&con->kref)) con = NULL; @@ -301,10 +305,20 @@ static void tipc_conn_send_to_sock(struct tipc_conn *con) static void tipc_conn_send_work(struct work_struct *work) { struct tipc_conn *con = container_of(work, struct tipc_conn, swork); + struct tipc_topsrv *srv; + + srv = con->server; + spin_lock_bh(&srv->idr_lock); + if (!srv->listener) { + spin_unlock_bh(&srv->idr_lock); + goto out; + } + spin_unlock_bh(&srv->idr_lock); if (connected(con)) tipc_conn_send_to_sock(con); +out: conn_put(con); } @@ -334,8 +348,14 @@ void tipc_topsrv_queue_evt(struct net *net, int conid, list_add_tail(&e->list, &con->outqueue); spin_unlock_bh(&con->outqueue_lock); - if (queue_work(srv->send_wq, &con->swork)) - return; + spin_lock_bh(&srv->idr_lock); + if (srv->listener) { + if (queue_work(srv->send_wq, &con->swork)) { + spin_unlock_bh(&srv->idr_lock); + return; + } + } + spin_unlock_bh(&srv->idr_lock); err: conn_put(con); } @@ -346,14 +366,20 @@ void tipc_topsrv_queue_evt(struct net *net, int conid, */ static void tipc_conn_write_space(struct sock *sk) { + struct tipc_topsrv *srv; struct tipc_conn *con; read_lock_bh(&sk->sk_callback_lock); con = sk->sk_user_data; if (connected(con)) { - conn_get(con); - if (!queue_work(con->server->send_wq, &con->swork)) - conn_put(con); + srv = con->server; + spin_lock_bh(&srv->idr_lock); + if (srv->listener) { + conn_get(con); + if (!queue_work(srv->send_wq, &con->swork)) + conn_put(con); + } + spin_unlock_bh(&srv->idr_lock); } read_unlock_bh(&sk->sk_callback_lock); } @@ -418,8 +444,17 @@ static int tipc_conn_rcv_from_sock(struct tipc_conn *con) static void tipc_conn_recv_work(struct work_struct *work) { struct tipc_conn *con = container_of(work, struct tipc_conn, rwork); + struct tipc_topsrv *srv; int count = 0; + srv = con->server; + spin_lock_bh(&srv->idr_lock); + if (!srv->listener) { + spin_unlock_bh(&srv->idr_lock); + goto out; + } + spin_unlock_bh(&srv->idr_lock); + while (connected(con)) { if (tipc_conn_rcv_from_sock(con)) break; @@ -430,6 +465,7 @@ static void tipc_conn_recv_work(struct work_struct *work) count = 0; } } +out: conn_put(con); } @@ -438,6 +474,7 @@ static void tipc_conn_recv_work(struct work_struct *work) */ static void tipc_conn_data_ready(struct sock *sk) { + struct tipc_topsrv *srv; struct tipc_conn *con; trace_sk_data_ready(sk); @@ -445,9 +482,14 @@ static void tipc_conn_data_ready(struct sock *sk) read_lock_bh(&sk->sk_callback_lock); con = sk->sk_user_data; if (connected(con)) { - conn_get(con); - if (!queue_work(con->server->rcv_wq, &con->rwork)) - conn_put(con); + srv = con->server; + spin_lock_bh(&srv->idr_lock); + if (srv->listener) { + conn_get(con); + if (!queue_work(srv->rcv_wq, &con->rwork)) + conn_put(con); + } + spin_unlock_bh(&srv->idr_lock); } read_unlock_bh(&sk->sk_callback_lock); } @@ -503,8 +545,12 @@ static void tipc_topsrv_listener_data_ready(struct sock *sk) read_lock_bh(&sk->sk_callback_lock); srv = sk->sk_user_data; - if (srv) - queue_work(srv->rcv_wq, &srv->awork); + if (srv) { + spin_lock_bh(&srv->idr_lock); + if (srv->listener) + queue_work(srv->rcv_wq, &srv->awork); + spin_unlock_bh(&srv->idr_lock); + } read_unlock_bh(&sk->sk_callback_lock); } @@ -700,23 +746,27 @@ static void tipc_topsrv_stop(struct net *net) struct tipc_conn *con; int id; + spin_lock_bh(&srv->idr_lock); + srv->listener = NULL; + spin_unlock_bh(&srv->idr_lock); + tipc_topsrv_work_stop(srv); + spin_lock_bh(&srv->idr_lock); for (id = 0; srv->idr_in_use; id++) { con = idr_find(&srv->conn_idr, id); if (con) { - conn_get(con); spin_unlock_bh(&srv->idr_lock); tipc_conn_close(con); - conn_put(con); spin_lock_bh(&srv->idr_lock); + continue; } + spin_unlock_bh(&srv->idr_lock); + spin_lock_bh(&srv->idr_lock); } __module_get(lsock->ops->owner); __module_get(lsock->sk->sk_prot_creator->owner); - srv->listener = NULL; spin_unlock_bh(&srv->idr_lock); - tipc_topsrv_work_stop(srv); sock_release(lsock); idr_destroy(&srv->conn_idr); kfree(srv); diff --git a/net/tipc/udp_media.c b/net/tipc/udp_media.c index e993bd6ed7c2..8af3349e154d 100644 --- a/net/tipc/udp_media.c +++ b/net/tipc/udp_media.c @@ -40,6 +40,7 @@ #include #include #include +#include #include #include #include @@ -94,6 +95,8 @@ struct udp_replicast { * @ifindex: local address scope * @work: used to schedule deferred work on a bearer * @rcast: associated udp_replicast container + * @rcast_lock: serialize updates to @rcast.list against concurrent updaters + * @disabled: bearer is being torn down; reject further @rcast.list additions */ struct udp_bearer { struct tipc_bearer __rcu *bearer; @@ -101,6 +104,8 @@ struct udp_bearer { u32 ifindex; struct work_struct work; struct udp_replicast rcast; + spinlock_t rcast_lock; + bool disabled; }; static int tipc_udp_is_mcast_addr(struct udp_media_addr *addr) @@ -279,26 +284,6 @@ static int tipc_udp_send_msg(struct net *net, struct sk_buff *skb, return err; } -static bool tipc_udp_is_known_peer(struct tipc_bearer *b, - struct udp_media_addr *addr) -{ - struct udp_replicast *rcast, *tmp; - struct udp_bearer *ub; - - ub = rcu_dereference_rtnl(b->media_ptr); - if (!ub) { - pr_err_ratelimited("UDP bearer instance not found\n"); - return false; - } - - list_for_each_entry_safe(rcast, tmp, &ub->rcast.list, list) { - if (!memcmp(&rcast->addr, addr, sizeof(struct udp_media_addr))) - return true; - } - - return false; -} - static int tipc_udp_rcast_add(struct tipc_bearer *b, struct udp_media_addr *addr) { @@ -309,16 +294,34 @@ static int tipc_udp_rcast_add(struct tipc_bearer *b, if (!ub) return -ENODEV; + spin_lock_bh(&ub->rcast_lock); + if (ub->disabled) { + spin_unlock_bh(&ub->rcast_lock); + return 0; + } + list_for_each_entry(rcast, &ub->rcast.list, list) { + if (!memcmp(&rcast->addr, addr, sizeof(*addr))) { + spin_unlock_bh(&ub->rcast_lock); + return 0; + } + } + rcast = kmalloc(sizeof(*rcast), GFP_ATOMIC); - if (!rcast) + if (!rcast) { + spin_unlock_bh(&ub->rcast_lock); return -ENOMEM; + } if (dst_cache_init(&rcast->dst_cache, GFP_ATOMIC)) { + spin_unlock_bh(&ub->rcast_lock); kfree(rcast); return -ENOMEM; } memcpy(&rcast->addr, addr, sizeof(struct udp_media_addr)); + list_add_rcu(&rcast->list, &ub->rcast.list); + b->bcast_addr.broadcast = TIPC_REPLICAST_SUPPORT; + spin_unlock_bh(&ub->rcast_lock); if (ntohs(addr->proto) == ETH_P_IP) pr_info("New replicast peer: %pI4\n", &rcast->addr.ipv4); @@ -326,8 +329,6 @@ static int tipc_udp_rcast_add(struct tipc_bearer *b, else if (ntohs(addr->proto) == ETH_P_IPV6) pr_info("New replicast peer: %pI6\n", &rcast->addr.ipv6); #endif - b->bcast_addr.broadcast = TIPC_REPLICAST_SUPPORT; - list_add_rcu(&rcast->list, &ub->rcast.list); return 0; } @@ -362,9 +363,6 @@ static int tipc_udp_rcast_disc(struct tipc_bearer *b, struct sk_buff *skb) return 0; } - if (likely(tipc_udp_is_known_peer(b, &src))) - return 0; - return tipc_udp_rcast_add(b, &src); } @@ -646,9 +644,6 @@ int tipc_udp_nl_bearer_add(struct tipc_bearer *b, struct nlattr *attr) return -EINVAL; } - if (tipc_udp_is_known_peer(b, &addr)) - return 0; - return tipc_udp_rcast_add(b, &addr); } @@ -680,6 +675,7 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b, return -ENOMEM; INIT_LIST_HEAD(&ub->rcast.list); + spin_lock_init(&ub->rcast_lock); if (!attrs[TIPC_NLA_BEARER_UDP_OPTS]) goto err; @@ -802,6 +798,14 @@ static int tipc_udp_enable(struct net *net, struct tipc_bearer *b, return err; } +static void rcast_free_rcu(struct rcu_head *rcu) +{ + struct udp_replicast *rcast = container_of(rcu, struct udp_replicast, rcu); + + dst_cache_destroy(&rcast->dst_cache); + kfree(rcast); +} + /* cleanup_bearer - break the socket/bearer association */ static void cleanup_bearer(struct work_struct *work) { @@ -809,20 +813,23 @@ static void cleanup_bearer(struct work_struct *work) struct udp_replicast *rcast, *tmp; struct tipc_net *tn; + spin_lock_bh(&ub->rcast_lock); list_for_each_entry_safe(rcast, tmp, &ub->rcast.list, list) { - dst_cache_destroy(&rcast->dst_cache); list_del_rcu(&rcast->list); - kfree_rcu(rcast, rcu); + call_rcu_hurry(&rcast->rcu, rcast_free_rcu); } + ub->disabled = true; + spin_unlock_bh(&ub->rcast_lock); tn = tipc_net(sock_net(ub->ubsock->sk)); - dst_cache_destroy(&ub->rcast.dst_cache); udp_tunnel_sock_release(ub->ubsock); - /* Note: could use a call_rcu() to avoid another synchronize_net() */ synchronize_net(); - atomic_dec(&tn->wq_count); + + dst_cache_destroy(&ub->rcast.dst_cache); + if (atomic_dec_and_test(&tn->wq_count)) + wake_up_var(&tn->wq_count); kfree(ub); } diff --git a/net/tls/tls_device.c b/net/tls/tls_device.c index 34d321ff1198..bb7057aa828e 100644 --- a/net/tls/tls_device.c +++ b/net/tls/tls_device.c @@ -239,7 +239,7 @@ static void tls_device_resync_tx(struct sock *sk, struct tls_context *tls_ctx, int err = 0; u8 *rcd_sn; - skb = tcp_write_queue_tail(sk); + skb = tcp_write_queue_tail(sk) ?: tcp_rtx_queue_tail(sk); if (skb) TCP_SKB_CB(skb)->eor = 1; @@ -1157,7 +1157,7 @@ int tls_set_device_offload(struct sock *sk, struct tls_context *ctx) * SKBs where only part of the payload needs to be encrypted. * So mark the last skb in the write queue as end of record. */ - skb = tcp_write_queue_tail(sk); + skb = tcp_write_queue_tail(sk) ?: tcp_rtx_queue_tail(sk); if (skb) TCP_SKB_CB(skb)->eor = 1; diff --git a/net/tls/tls_sw.c b/net/tls/tls_sw.c index fbea00967c14..f797c740dcea 100644 --- a/net/tls/tls_sw.c +++ b/net/tls/tls_sw.c @@ -1753,6 +1753,8 @@ tls_decrypt_sw(struct sock *sk, struct tls_context *tls_ctx, /* If opportunistic TLS 1.3 ZC failed retry without ZC */ if (unlikely(darg->zc && prot->version == TLS_1_3_VERSION && darg->tail != TLS_RECORD_TYPE_DATA)) { + iov_iter_revert(&msg->msg_iter, strp_msg(darg->skb)->full_len - + prot->overhead_size); darg->zc = false; if (!darg->tail) TLS_INC_STATS(sock_net(sk), LINUX_MIB_TLSRXNOPADVIOL); diff --git a/net/unix/af_unix.c b/net/unix/af_unix.c index e1f798af1a8a..8aa73ccdf67a 100644 --- a/net/unix/af_unix.c +++ b/net/unix/af_unix.c @@ -2641,8 +2641,8 @@ static int unix_stream_recv_urg(struct unix_stream_read_state *state) return 1; } -static struct sk_buff *manage_oob(struct sk_buff *skb, struct sock *sk, - int flags, int copied) +static struct sk_buff *manage_oob(struct sk_buff *skb, struct sk_buff **last, + struct sock *sk, int flags, int copied) { struct unix_sock *u = unix_sk(sk); @@ -2665,8 +2665,10 @@ static struct sk_buff *manage_oob(struct sk_buff *skb, struct sock *sk, __skb_unlink(skb, &sk->sk_receive_queue); unlinked_skb = skb; skb = skb_peek(&sk->sk_receive_queue); + *last = skb; } } else if (!sock_flag(sk, SOCK_URGINLINE)) { + *last = skb; skb = skb_peek_next(skb, &sk->sk_receive_queue); } } @@ -2792,7 +2794,7 @@ static int unix_stream_read_generic(struct unix_stream_read_state *state, again: #if IS_ENABLED(CONFIG_AF_UNIX_OOB) if (skb) { - skb = manage_oob(skb, sk, flags, copied); + skb = manage_oob(skb, &last, sk, flags, copied); if (!skb && copied) { unix_state_unlock(sk); break; diff --git a/net/vmw_vsock/af_vsock.c b/net/vmw_vsock/af_vsock.c index 1392fcde7776..6b7bdd6166c0 100644 --- a/net/vmw_vsock/af_vsock.c +++ b/net/vmw_vsock/af_vsock.c @@ -1649,8 +1649,7 @@ static int vsock_accept(struct socket *sock, struct socket *newsock, int flags, timeout = sock_rcvtimeo(listener, flags & O_NONBLOCK); prepare_to_wait(sk_sleep(listener), &wait, TASK_INTERRUPTIBLE); - while ((connected = vsock_dequeue_accept(listener)) == NULL && - listener->sk_err == 0) { + while ((connected = vsock_dequeue_accept(listener)) == NULL) { release_sock(listener); timeout = schedule_timeout(timeout); finish_wait(sk_sleep(listener), &wait); @@ -1668,9 +1667,6 @@ static int vsock_accept(struct socket *sock, struct socket *newsock, int flags, } finish_wait(sk_sleep(listener), &wait); - if (listener->sk_err) - err = -listener->sk_err; - if (connected) { sk_acceptq_removed(listener); diff --git a/net/wireless/core.c b/net/wireless/core.c index 800fbfd59194..da355e5975ca 100644 --- a/net/wireless/core.c +++ b/net/wireless/core.c @@ -237,6 +237,9 @@ void cfg80211_stop_p2p_device(struct cfg80211_registered_device *rdev, if (!wdev_running(wdev)) return; + wdev_lock(wdev); + cfg80211_pmsr_wdev_down(wdev); + wdev_unlock(wdev); rdev_stop_p2p_device(rdev, wdev); wdev->is_running = false; @@ -262,6 +265,10 @@ void cfg80211_stop_nan(struct cfg80211_registered_device *rdev, if (!wdev_running(wdev)) return; + wdev_lock(wdev); + cfg80211_pmsr_wdev_down(wdev); + wdev_unlock(wdev); + rdev_stop_nan(rdev, wdev); wdev->is_running = false; @@ -690,6 +697,24 @@ static int wiphy_verify_combinations(struct wiphy *wiphy) return 0; } +static bool wiphy_cipher_suites_valid(const struct wiphy *wiphy) +{ + int i, j; + + if (wiphy->n_cipher_suites && !wiphy->cipher_suites) + return false; + + for (i = 0; i < wiphy->n_cipher_suites; i++) { + for (j = 0; j < i; j++) { + if (wiphy->cipher_suites[i] == + wiphy->cipher_suites[j]) + return false; + } + } + + return true; +} + int wiphy_register(struct wiphy *wiphy) { struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy); @@ -821,6 +846,9 @@ int wiphy_register(struct wiphy *wiphy) if (res) return res; + if (!wiphy_cipher_suites_valid(wiphy)) + return -EINVAL; + /* sanity check supported bands/channels */ for (band = 0; band < NUM_NL80211_BANDS; band++) { u16 types = 0; @@ -1220,6 +1248,7 @@ static void _cfg80211_unregister_wdev(struct wireless_dev *wdev, list_del_rcu(&wdev->list); synchronize_net(); rdev->devlist_generation++; + wiphy_work_cancel(wdev->wiphy, &wdev->disconnect_wk); cfg80211_mlme_purge_registrations(wdev); @@ -1386,7 +1415,7 @@ void cfg80211_init_wdev(struct wireless_dev *wdev) INIT_LIST_HEAD(&wdev->mgmt_registrations); INIT_LIST_HEAD(&wdev->pmsr_list); spin_lock_init(&wdev->pmsr_lock); - INIT_WORK(&wdev->pmsr_free_wk, cfg80211_pmsr_free_wk); + wiphy_work_init(&wdev->pmsr_free_wk, cfg80211_pmsr_free_wk); #ifdef CONFIG_CFG80211_WEXT wdev->wext.default_key = -1; @@ -1408,7 +1437,7 @@ void cfg80211_init_wdev(struct wireless_dev *wdev) wdev->iftype == NL80211_IFTYPE_ADHOC) && !wdev->use_4addr) wdev->netdev->priv_flags |= IFF_DONT_BRIDGE; - INIT_WORK(&wdev->disconnect_wk, cfg80211_autodisconnect_wk); + wiphy_work_init(&wdev->disconnect_wk, cfg80211_autodisconnect_wk); } void cfg80211_register_wdev(struct cfg80211_registered_device *rdev, @@ -1516,10 +1545,9 @@ static int cfg80211_netdev_notifier_call(struct notifier_block *nb, wiphy_lock(&rdev->wiphy); cfg80211_leave(rdev, wdev); cfg80211_remove_links(wdev); - wiphy_unlock(&rdev->wiphy); /* since we just did cfg80211_leave() nothing to do there */ - cancel_work_sync(&wdev->disconnect_wk); - cancel_work_sync(&wdev->pmsr_free_wk); + wiphy_work_cancel(wdev->wiphy, &wdev->disconnect_wk); + wiphy_unlock(&rdev->wiphy); break; case NETDEV_DOWN: wiphy_lock(&rdev->wiphy); diff --git a/net/wireless/core.h b/net/wireless/core.h index f0a3a2317638..3ba403698f41 100644 --- a/net/wireless/core.h +++ b/net/wireless/core.h @@ -427,7 +427,7 @@ void __cfg80211_port_authorized(struct wireless_dev *wdev, const u8 *bssid, const u8 *td_bitmap, u8 td_bitmap_len); int cfg80211_mgd_wext_connect(struct cfg80211_registered_device *rdev, struct wireless_dev *wdev); -void cfg80211_autodisconnect_wk(struct work_struct *work); +void cfg80211_autodisconnect_wk(struct wiphy *wiphy, struct wiphy_work *work); /* SME implementation */ void cfg80211_conn_work(struct work_struct *work); @@ -575,7 +575,7 @@ cfg80211_bss_update(struct cfg80211_registered_device *rdev, void cfg80211_release_pmsr(struct wireless_dev *wdev, u32 portid); void cfg80211_pmsr_wdev_down(struct wireless_dev *wdev); -void cfg80211_pmsr_free_wk(struct work_struct *work); +void cfg80211_pmsr_free_wk(struct wiphy *wiphy, struct wiphy_work *work); void cfg80211_remove_link(struct wireless_dev *wdev, unsigned int link_id); void cfg80211_remove_links(struct wireless_dev *wdev); diff --git a/net/wireless/lib80211_crypt_tkip.c b/net/wireless/lib80211_crypt_tkip.c index 1b4d6c87a5c5..823274728711 100644 --- a/net/wireless/lib80211_crypt_tkip.c +++ b/net/wireless/lib80211_crypt_tkip.c @@ -38,6 +38,7 @@ MODULE_DESCRIPTION("lib80211 crypt: TKIP"); MODULE_LICENSE("GPL"); #define TKIP_HDR_LEN 8 +#define MICHAEL_MIC_LEN 8 struct lib80211_tkip_data { #define TKIP_KEY_LEN 32 @@ -585,16 +586,18 @@ static int lib80211_michael_mic_verify(struct sk_buff *skb, int keyidx, int hdr_len, void *priv) { struct lib80211_tkip_data *tkey = priv; - u8 mic[8]; + u8 mic[MICHAEL_MIC_LEN]; - if (!tkey->key_set) + if (!tkey->key_set || skb->len < hdr_len + MICHAEL_MIC_LEN) return -1; michael_mic_hdr(skb, tkey->rx_hdr); if (michael_mic(tkey->rx_tfm_michael, &tkey->key[24], tkey->rx_hdr, - skb->data + hdr_len, skb->len - 8 - hdr_len, mic)) + skb->data + hdr_len, + skb->len - MICHAEL_MIC_LEN - hdr_len, mic)) return -1; - if (memcmp(mic, skb->data + skb->len - 8, 8) != 0) { + if (memcmp(mic, skb->data + skb->len - MICHAEL_MIC_LEN, + MICHAEL_MIC_LEN) != 0) { struct ieee80211_hdr *hdr; hdr = (struct ieee80211_hdr *)skb->data; printk(KERN_DEBUG "%s: Michael MIC verification failed for " @@ -612,7 +615,7 @@ static int lib80211_michael_mic_verify(struct sk_buff *skb, int keyidx, tkey->rx_iv32 = tkey->rx_iv32_new; tkey->rx_iv16 = tkey->rx_iv16_new; - skb_trim(skb, skb->len - 8); + skb_trim(skb, skb->len - MICHAEL_MIC_LEN); return 0; } diff --git a/net/wireless/nl80211.c b/net/wireless/nl80211.c index e4cd5503ae11..3eccc83fa79b 100644 --- a/net/wireless/nl80211.c +++ b/net/wireless/nl80211.c @@ -16090,6 +16090,7 @@ static int nl80211_color_change(struct sk_buff *skb, struct genl_info *info) struct net_device *dev = info->user_ptr[1]; struct wireless_dev *wdev = dev->ieee80211_ptr; struct nlattr **tb; + unsigned int link_id; u16 offset; int err; @@ -16111,15 +16112,19 @@ static int nl80211_color_change(struct sk_buff *skb, struct genl_info *info) params.count = nla_get_u8(info->attrs[NL80211_ATTR_COLOR_CHANGE_COUNT]); params.color = nla_get_u8(info->attrs[NL80211_ATTR_COLOR_CHANGE_COLOR]); - err = nl80211_parse_beacon(rdev, info->attrs, ¶ms.beacon_next, - info->extack); - if (err) - return err; + link_id = nl80211_link_id(info->attrs); + if (!wdev->links[link_id].ap.beacon_interval) + return -EINVAL; tb = kcalloc(NL80211_ATTR_MAX + 1, sizeof(*tb), GFP_KERNEL); if (!tb) return -ENOMEM; + err = nl80211_parse_beacon(rdev, info->attrs, ¶ms.beacon_next, + info->extack); + if (err) + goto out; + err = nla_parse_nested(tb, NL80211_ATTR_MAX, info->attrs[NL80211_ATTR_COLOR_CHANGE_ELEMS], nl80211_policy, info->extack); @@ -20029,7 +20034,8 @@ static int nl80211_netlink_notify(struct notifier_block * nb, wdev->nl_owner_dead = true; schedule_work(&rdev->destroy_work); } else if (wdev->conn_owner_nlportid == notify->portid) { - schedule_work(&wdev->disconnect_wk); + wiphy_work_queue(wdev->wiphy, + &wdev->disconnect_wk); } cfg80211_release_pmsr(wdev, notify->portid); diff --git a/net/wireless/pmsr.c b/net/wireless/pmsr.c index 678bbe5bf455..88aade01cd06 100644 --- a/net/wireless/pmsr.c +++ b/net/wireless/pmsr.c @@ -634,16 +634,14 @@ static void cfg80211_pmsr_process_abort(struct wireless_dev *wdev) } } -void cfg80211_pmsr_free_wk(struct work_struct *work) +void cfg80211_pmsr_free_wk(struct wiphy *wiphy, struct wiphy_work *work) { struct wireless_dev *wdev = container_of(work, struct wireless_dev, pmsr_free_wk); - wiphy_lock(wdev->wiphy); wdev_lock(wdev); cfg80211_pmsr_process_abort(wdev); wdev_unlock(wdev); - wiphy_unlock(wdev->wiphy); } void cfg80211_pmsr_wdev_down(struct wireless_dev *wdev) @@ -658,7 +656,7 @@ void cfg80211_pmsr_wdev_down(struct wireless_dev *wdev) } spin_unlock_bh(&wdev->pmsr_lock); - cancel_work_sync(&wdev->pmsr_free_wk); + wiphy_work_cancel(wdev->wiphy, &wdev->pmsr_free_wk); if (found) cfg80211_pmsr_process_abort(wdev); @@ -673,7 +671,7 @@ void cfg80211_release_pmsr(struct wireless_dev *wdev, u32 portid) list_for_each_entry(req, &wdev->pmsr_list, list) { if (req->nl_portid == portid) { req->nl_portid = 0; - schedule_work(&wdev->pmsr_free_wk); + wiphy_work_queue(wdev->wiphy, &wdev->pmsr_free_wk); } } spin_unlock_bh(&wdev->pmsr_lock); diff --git a/net/wireless/scan.c b/net/wireless/scan.c index 0766367bda6c..d10d2dfd8b8d 100644 --- a/net/wireless/scan.c +++ b/net/wireless/scan.c @@ -1028,6 +1028,7 @@ int cfg80211_scan(struct cfg80211_registered_device *rdev) struct cfg80211_scan_request *request; struct cfg80211_scan_request *rdev_req = rdev->scan_req; u32 n_channels = 0, idx, i; + int err; if (!(rdev->wiphy.flags & WIPHY_FLAG_SPLIT_SCAN_6GHZ)) return rdev_scan(rdev, rdev_req); @@ -1054,8 +1055,14 @@ int cfg80211_scan(struct cfg80211_registered_device *rdev) } rdev_req->scan_6ghz = false; + err = rdev_scan(rdev, request); + if (err) { + kfree(request); + return err; + } + rdev->int_scan_req = request; - return rdev_scan(rdev, request); + return 0; } void ___cfg80211_scan_done(struct cfg80211_registered_device *rdev, @@ -1812,20 +1819,22 @@ cfg80211_update_known_bss(struct cfg80211_registered_device *rdev, return false; } - old = rcu_access_pointer(known->pub.beacon_ies); + if (!new->pub.hidden_beacon_bss) { + old = rcu_access_pointer(known->pub.beacon_ies); - rcu_assign_pointer(known->pub.beacon_ies, new->pub.beacon_ies); + rcu_assign_pointer(known->pub.beacon_ies, new->pub.beacon_ies); - /* Override IEs if they were from a beacon before */ - if (old == rcu_access_pointer(known->pub.ies)) - rcu_assign_pointer(known->pub.ies, new->pub.beacon_ies); + /* Override IEs if they were from a beacon before */ + if (old == rcu_access_pointer(known->pub.ies)) + rcu_assign_pointer(known->pub.ies, new->pub.beacon_ies); - cfg80211_update_hidden_bsses(known, - rcu_access_pointer(new->pub.beacon_ies), - old); + cfg80211_update_hidden_bsses(known, + rcu_access_pointer(new->pub.beacon_ies), + old); - if (old) - kfree_rcu((struct cfg80211_bss_ies *)old, rcu_head); + if (old) + kfree_rcu((struct cfg80211_bss_ies *)old, rcu_head); + } } known->pub.beacon_interval = new->pub.beacon_interval; @@ -3162,10 +3171,9 @@ void cfg80211_update_assoc_bss_entry(struct wireless_dev *wdev, if (new) { /* to save time, update IEs for transmitting bss only */ - if (cfg80211_update_known_bss(rdev, cbss, new, false)) { - new->pub.proberesp_ies = NULL; - new->pub.beacon_ies = NULL; - } + cfg80211_update_known_bss(rdev, cbss, new, false); + new->pub.proberesp_ies = NULL; + new->pub.beacon_ies = NULL; list_for_each_entry_safe(nontrans_bss, tmp, &new->pub.nontrans_list, diff --git a/net/wireless/sme.c b/net/wireless/sme.c index 6f116ae5442b..eb0434783f9f 100644 --- a/net/wireless/sme.c +++ b/net/wireless/sme.c @@ -1584,13 +1584,12 @@ int cfg80211_disconnect(struct cfg80211_registered_device *rdev, * Used to clean up after the connection / connection attempt owner socket * disconnects */ -void cfg80211_autodisconnect_wk(struct work_struct *work) +void cfg80211_autodisconnect_wk(struct wiphy *wiphy, struct wiphy_work *work) { struct wireless_dev *wdev = container_of(work, struct wireless_dev, disconnect_wk); - struct cfg80211_registered_device *rdev = wiphy_to_rdev(wdev->wiphy); + struct cfg80211_registered_device *rdev = wiphy_to_rdev(wiphy); - wiphy_lock(wdev->wiphy); wdev_lock(wdev); if (wdev->conn_owner_nlportid) { @@ -1629,5 +1628,4 @@ void cfg80211_autodisconnect_wk(struct work_struct *work) } wdev_unlock(wdev); - wiphy_unlock(wdev->wiphy); } diff --git a/net/wireless/wext-sme.c b/net/wireless/wext-sme.c index cfa903eb0e82..0e7c314bdeef 100644 --- a/net/wireless/wext-sme.c +++ b/net/wireless/wext-sme.c @@ -354,8 +354,10 @@ int cfg80211_wext_siwgenie(struct net_device *dev, /* nothing */ } - if (!for_each_element_completed(elem, extra, ie_len)) - return -EINVAL; + if (!for_each_element_completed(elem, extra, ie_len)) { + err = -EINVAL; + goto out; + } ie = kmemdup(extra, ie_len, GFP_KERNEL); if (!ie) { diff --git a/net/xdp/xsk.c b/net/xdp/xsk.c index 3caeac3a5f16..95944ad2eaae 100644 --- a/net/xdp/xsk.c +++ b/net/xdp/xsk.c @@ -614,8 +614,11 @@ static void xsk_consume_skb(struct sk_buff *skb) static void xsk_drop_skb(struct sk_buff *skb) { - xdp_sk(skb->sk)->tx->invalid_descs += xsk_get_num_desc(skb); - xsk_consume_skb(skb); + struct xdp_sock *xs = xdp_sk(skb->sk); + + xs->tx->invalid_descs += xsk_get_num_desc(skb); + consume_skb(skb); + xs->skb = NULL; } static struct sk_buff *xsk_build_skb_zerocopy(struct xdp_sock *xs, diff --git a/net/xfrm/xfrm_device.c b/net/xfrm/xfrm_device.c index a73b3c9d7e4d..49529d95afa3 100644 --- a/net/xfrm/xfrm_device.c +++ b/net/xfrm/xfrm_device.c @@ -226,6 +226,14 @@ struct sk_buff *validate_xmit_xfrm(struct sk_buff *skb, netdev_features_t featur pskb = skb2; } + /* skb_gso_segment() set skb->prev to the last segment, but async + * crypto may have stolen it above without updating ->prev. Repoint + * it at the last retained segment so validate_xmit_skb_list() does + * not chain onto a segment now owned by the crypto engine. + */ + if (skb) + skb->prev = pskb; + return skb; } EXPORT_SYMBOL_GPL(validate_xmit_xfrm); diff --git a/net/xfrm/xfrm_input.c b/net/xfrm/xfrm_input.c index 3100c5ef1862..da015efb4541 100644 --- a/net/xfrm/xfrm_input.c +++ b/net/xfrm/xfrm_input.c @@ -627,18 +627,21 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) XFRM_SKB_CB(skb)->seq.input.low = seq; XFRM_SKB_CB(skb)->seq.input.hi = seq_hi; - dev_hold(skb->dev); - - if (crypto_done) + if (crypto_done) { nexthdr = x->type_offload->input_tail(x, skb); - else + } else { + dev_hold(skb->dev); + nexthdr = x->type->input(x, skb); + if (nexthdr == -EINPROGRESS) { + if (async) + dev_put(skb->dev); + return 0; + } - if (nexthdr == -EINPROGRESS) - return 0; + dev_put(skb->dev); + } resume: - dev_put(skb->dev); - spin_lock(&x->lock); if (nexthdr < 0) { if (nexthdr == -EBADMSG) { @@ -705,6 +708,8 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) sp->olen = 0; if (skb_valid_dst(skb)) skb_dst_drop(skb); + if (async) + dev_put(skb->dev); gro_cells_receive(&gro_cells, skb); return 0; } else { @@ -724,6 +729,8 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) sp->olen = 0; if (skb_valid_dst(skb)) skb_dst_drop(skb); + if (async) + dev_put(skb->dev); gro_cells_receive(&gro_cells, skb); return err; } @@ -734,6 +741,8 @@ int xfrm_input(struct sk_buff *skb, int nexthdr, __be32 spi, int encap_type) drop_unlock: spin_unlock(&x->lock); drop: + if (async) + dev_put(skb->dev); xfrm_rcv_cb(skb, family, x && x->type ? x->type->proto : nexthdr, -1); kfree_skb(skb); return 0; diff --git a/net/xfrm/xfrm_policy.c b/net/xfrm/xfrm_policy.c index 00d9693c13ae..cc8f4d4b7087 100644 --- a/net/xfrm/xfrm_policy.c +++ b/net/xfrm/xfrm_policy.c @@ -1384,7 +1384,10 @@ static void xfrm_hash_rebuild(struct work_struct *work) void xfrm_policy_hash_rebuild(struct net *net) { - schedule_work(&net->xfrm.policy_hthresh.work); + write_seqlock(&net->xfrm.policy_hthresh.lock); + if (!net->xfrm.policy_hthresh.work_disabled) + schedule_work(&net->xfrm.policy_hthresh.work); + write_sequnlock(&net->xfrm.policy_hthresh.lock); } EXPORT_SYMBOL(xfrm_policy_hash_rebuild); @@ -4181,6 +4184,7 @@ static int __net_init xfrm_policy_init(struct net *net) net->xfrm.policy_hthresh.rbits6 = 128; seqlock_init(&net->xfrm.policy_hthresh.lock); + net->xfrm.policy_hthresh.work_disabled = false; INIT_LIST_HEAD(&net->xfrm.policy_all); INIT_LIST_HEAD(&net->xfrm.inexact_bins); @@ -4206,6 +4210,19 @@ static void xfrm_policy_fini(struct net *net) unsigned int sz; int dir; + /* Prevent new policy hash rebuilds before draining the work item. + * + * The upstream fix uses disable_work_sync(), which is unavailable + * in v6.6. Protecting work_disabled and schedule_work() with the + * same seqlock closes the check-to-queue race, while the subsequent + * cancel_work_sync() drains work that was queued or running before + * teardown disabled it. + */ + write_seqlock(&net->xfrm.policy_hthresh.lock); + net->xfrm.policy_hthresh.work_disabled = true; + write_sequnlock(&net->xfrm.policy_hthresh.lock); + cancel_work_sync(&net->xfrm.policy_hthresh.work); + flush_work(&net->xfrm.policy_hash_work); #ifdef CONFIG_XFRM_SUB_POLICY xfrm_policy_flush(net, XFRM_POLICY_TYPE_SUB, false); diff --git a/net/xfrm/xfrm_user.c b/net/xfrm/xfrm_user.c index b926d4651c06..2f84d8ddc46d 100644 --- a/net/xfrm/xfrm_user.c +++ b/net/xfrm/xfrm_user.c @@ -2404,7 +2404,8 @@ static inline unsigned int xfrm_aevent_msgsize(struct xfrm_state *x) + nla_total_size_64bit(sizeof(struct xfrm_lifetime_cur)) + nla_total_size(sizeof(struct xfrm_mark)) + nla_total_size(4) /* XFRM_AE_RTHR */ - + nla_total_size(4); /* XFRM_AE_ETHR */ + + nla_total_size(4) /* XFRM_AE_ETHR */ + + nla_total_size(sizeof(x->if_id)); /* XFRMA_IF_ID */ } static int build_aevent(struct sk_buff *skb, struct xfrm_state *x, const struct km_event *c) @@ -2505,7 +2506,12 @@ static int xfrm_get_ae(struct sk_buff *skb, struct nlmsghdr *nlh, c.portid = nlh->nlmsg_pid; err = build_aevent(r_skb, x, &c); - BUG_ON(err < 0); + if (err < 0) { + spin_unlock_bh(&x->lock); + xfrm_state_put(x); + kfree_skb(r_skb); + return err; + } err = nlmsg_unicast(xfrm_net_nlsk(net, skb), r_skb, NETLINK_CB(skb).portid); spin_unlock_bh(&x->lock); diff --git a/samples/vfio-mdev/mtty.c b/samples/vfio-mdev/mtty.c index 245db52bedf2..34e333560a73 100644 --- a/samples/vfio-mdev/mtty.c +++ b/samples/vfio-mdev/mtty.c @@ -203,10 +203,10 @@ static void mtty_trigger_interrupt(struct mdev_state *mdev_state) if (is_msi(mdev_state)) { if (mdev_state->msi_evtfd) - eventfd_signal(mdev_state->msi_evtfd, 1); + eventfd_signal(mdev_state->msi_evtfd); } else if (is_intx(mdev_state)) { if (mdev_state->intx_evtfd && !mdev_state->intx_mask) { - eventfd_signal(mdev_state->intx_evtfd, 1); + eventfd_signal(mdev_state->intx_evtfd); mdev_state->intx_mask = true; } } diff --git a/security/apparmor/domain.c b/security/apparmor/domain.c index f14f8a4e8594..5baf694d6a5e 100644 --- a/security/apparmor/domain.c +++ b/security/apparmor/domain.c @@ -1029,8 +1029,8 @@ static struct aa_label *build_change_hat(const struct cred *subj_cred, if (!hat) { error = -ENOENT; if (COMPLAIN_MODE(profile)) { - hat = aa_new_learning_profile(profile, true, name, - GFP_KERNEL); + hat = __aa_new_learning_profile(profile, true, name, + GFP_KERNEL); if (!hat) { info = "failed null profile create"; error = -ENOMEM; @@ -1068,6 +1068,7 @@ static struct aa_label *change_hat(const struct cred *subj_cred, bool sibling = false; const char *name, *info = NULL; int i, error; + bool needput = false; AA_BUG(!label); AA_BUG(!hats); @@ -1080,7 +1081,6 @@ static struct aa_label *change_hat(const struct cred *subj_cred, * the profiles and label, we can rely on the namespaces being live * and avoid incrementing their refcounts while grabbing the lock. */ - label = aa_get_label(label); ns = labels_ns(label); retry: @@ -1088,15 +1088,19 @@ static struct aa_label *change_hat(const struct cred *subj_cred, if (label_is_stale(label)) { new = aa_get_newest_label(label); new_ns = labels_ns(new); + + if (needput) + /* aa_put_label() is safe to call when under lock */ + aa_put_label(label); + label = new; + needput = true; + /* check if replaced with label in parent ns, and lock there */ if (new_ns != ns) { - aa_put_label(new); mutex_unlock(&ns->lock); ns = new_ns; - label = new; + /* retry will bottom out at the root of the tree */ goto retry; } - aa_put_label(label); - label = new; } if (PROFILE_IS_HAT(labels_profile(label))) @@ -1107,7 +1111,8 @@ static struct aa_label *change_hat(const struct cred *subj_cred, name = hats[i]; label_for_each_in_ns(it, labels_ns(label), label, profile) { if (sibling && PROFILE_IS_HAT(profile)) { - root = aa_get_profile(profile->parent); + root = aa_get_profile(rcu_dereference_protected(profile->parent, + mutex_is_locked(&ns->lock))); } else if (!sibling && !PROFILE_IS_HAT(profile)) { root = aa_get_profile(profile); } else { /* conflicting change type */ @@ -1168,6 +1173,8 @@ static struct aa_label *change_hat(const struct cred *subj_cred, } } mutex_unlock(&ns->lock); + if (needput) + aa_put_label(label); return ERR_PTR(error); build: @@ -1181,6 +1188,8 @@ static struct aa_label *change_hat(const struct cred *subj_cred, goto fail; } /* else if (IS_ERR) build_change_hat has logged error so return new */ mutex_unlock(&ns->lock); + if (needput) + aa_put_label(label); return new; } diff --git a/security/apparmor/include/cred.h b/security/apparmor/include/cred.h index 58fdc72af664..3fea4da22d08 100644 --- a/security/apparmor/include/cred.h +++ b/security/apparmor/include/cred.h @@ -148,13 +148,9 @@ static inline struct aa_label *begin_current_label_crit_section(void) { struct aa_label *label = aa_current_raw_label(); - might_sleep(); - if (label_is_stale(label)) { label = aa_get_newest_label(label); - if (aa_replace_current_label(label) == 0) - /* task cred will keep the reference */ - aa_put_label(label); + aa_schedule_stale_label_replacement(); } return label; diff --git a/security/apparmor/include/label.h b/security/apparmor/include/label.h index 5aca0f612f8f..55b5372ecf24 100644 --- a/security/apparmor/include/label.h +++ b/security/apparmor/include/label.h @@ -22,7 +22,7 @@ struct aa_ns; #define LOCAL_VEC_ENTRIES 8 #define DEFINE_VEC(T, V) \ - struct aa_ ## T *(_ ## V ## _localtmp)[LOCAL_VEC_ENTRIES]; \ + struct aa_ ## T *(_ ## V ## _localtmp)[LOCAL_VEC_ENTRIES + 1]; \ struct aa_ ## T **(V) #define vec_setup(T, V, N, GFP) \ @@ -30,10 +30,10 @@ struct aa_ns; if ((N) <= LOCAL_VEC_ENTRIES) { \ typeof(N) i; \ (V) = (_ ## V ## _localtmp); \ - for (i = 0; i < (N); i++) \ + for (i = 0; i <= (N); i++) \ (V)[i] = NULL; \ } else \ - (V) = kzalloc(sizeof(struct aa_ ## T *) * (N), (GFP)); \ + (V) = kcalloc((N) + 1, sizeof(struct aa_ ## T *), (GFP)); \ (V) ? 0 : -ENOMEM; \ }) diff --git a/security/apparmor/include/match.h b/security/apparmor/include/match.h index a86f74b59360..14c0401f97c1 100644 --- a/security/apparmor/include/match.h +++ b/security/apparmor/include/match.h @@ -102,16 +102,18 @@ struct aa_dfa { struct table_header *tables[YYTD_ID_TSIZE]; }; -#define byte_to_byte(X) (X) - #define UNPACK_ARRAY(TABLE, BLOB, LEN, TTYPE, BTYPE, NTOHX) \ do { \ typeof(LEN) __i; \ TTYPE *__t = (TTYPE *) TABLE; \ BTYPE *__b = (BTYPE *) BLOB; \ - for (__i = 0; __i < LEN; __i++) { \ - __t[__i] = NTOHX(__b[__i]); \ - } \ + BUILD_BUG_ON(sizeof(TTYPE) != sizeof(BTYPE)); \ + if (IS_ENABLED(CONFIG_CPU_BIG_ENDIAN)) \ + memcpy(__t, __b, (LEN) * sizeof(BTYPE)); \ + else /* copy & convert from big-endian */ \ + for (__i = 0; __i < LEN; __i++) { \ + __t[__i] = NTOHX(&__b[__i]); \ + } \ } while (0) static inline size_t table_size(size_t len, size_t el_size) diff --git a/security/apparmor/include/policy.h b/security/apparmor/include/policy.h index e486238f2d33..0d100f26b746 100644 --- a/security/apparmor/include/policy.h +++ b/security/apparmor/include/policy.h @@ -261,6 +261,9 @@ struct aa_profile *aa_alloc_profile(const char *name, struct aa_proxy *proxy, gfp_t gfp); struct aa_profile *aa_alloc_null(struct aa_profile *parent, const char *name, gfp_t gfp); +struct aa_profile *__aa_new_learning_profile(struct aa_profile *parent, + bool hat, const char *base, + gfp_t gfp); struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, const char *base, gfp_t gfp); void aa_free_profile(struct aa_profile *profile); diff --git a/security/apparmor/include/task.h b/security/apparmor/include/task.h index 29ba55107b7d..9bf4aaa0aa13 100644 --- a/security/apparmor/include/task.h +++ b/security/apparmor/include/task.h @@ -21,15 +21,22 @@ static inline struct aa_task_ctx *task_ctx(struct task_struct *task) * @onexec: profile to transition to on next exec (MAY BE NULL) * @previous: profile the task may return to (MAY BE NULL) * @token: magic value the task must know for returning to @previous_profile + * @label_replacement_tw: for aa_schedule_stale_label_replacement() + * @label_replacement_pending: is @label_replacement_tw pending? + * + * When changing this, check if aa_dup_task_ctx() needs to be updated. */ struct aa_task_ctx { struct aa_label *nnp; struct aa_label *onexec; struct aa_label *previous; u64 token; + struct callback_head label_replacement_tw; + bool label_replacement_pending; }; int aa_replace_current_label(struct aa_label *label); +void aa_schedule_stale_label_replacement(void); int aa_set_current_onexec(struct aa_label *label, bool stack); int aa_set_current_hat(struct aa_label *label, u64 token); int aa_restore_previous_label(u64 cookie); @@ -56,10 +63,10 @@ static inline void aa_free_task_ctx(struct aa_task_ctx *ctx) static inline void aa_dup_task_ctx(struct aa_task_ctx *new, const struct aa_task_ctx *old) { - *new = *old; - aa_get_label(new->nnp); - aa_get_label(new->previous); - aa_get_label(new->onexec); + new->nnp = aa_get_label(old->nnp); + new->onexec = aa_get_label(old->onexec); + new->previous = aa_get_label(old->previous); + new->token = old->token; } /** diff --git a/security/apparmor/match.c b/security/apparmor/match.c index 29e728f6fbcf..0a3307cd4107 100644 --- a/security/apparmor/match.c +++ b/security/apparmor/match.c @@ -15,6 +15,7 @@ #include #include #include +#include #include "include/lib.h" #include "include/match.h" @@ -42,11 +43,11 @@ static struct table_header *unpack_table(char *blob, size_t bsize) /* loaded td_id's start at 1, subtract 1 now to avoid doing * it every time we use td_id as an index */ - th.td_id = be16_to_cpu(*(__be16 *) (blob)) - 1; + th.td_id = get_unaligned_be16(blob) - 1; if (th.td_id > YYTD_ID_MAX) goto out; - th.td_flags = be16_to_cpu(*(__be16 *) (blob + 2)); - th.td_lolen = be32_to_cpu(*(__be32 *) (blob + 8)); + th.td_flags = get_unaligned_be16(blob + 2); + th.td_lolen = get_unaligned_be32(blob + 8); blob += sizeof(struct table_header); if (!(th.td_flags == YYTD_DATA16 || th.td_flags == YYTD_DATA32 || @@ -66,14 +67,13 @@ static struct table_header *unpack_table(char *blob, size_t bsize) table->td_flags = th.td_flags; table->td_lolen = th.td_lolen; if (th.td_flags == YYTD_DATA8) - UNPACK_ARRAY(table->td_data, blob, th.td_lolen, - u8, u8, byte_to_byte); + memcpy(table->td_data, blob, th.td_lolen); else if (th.td_flags == YYTD_DATA16) UNPACK_ARRAY(table->td_data, blob, th.td_lolen, - u16, __be16, be16_to_cpu); + u16, __be16, get_unaligned_be16); else if (th.td_flags == YYTD_DATA32) UNPACK_ARRAY(table->td_data, blob, th.td_lolen, - u32, __be32, be32_to_cpu); + u32, __be32, get_unaligned_be32); else goto fail; /* if table was vmalloced make sure the page tables are synced @@ -293,14 +293,14 @@ struct aa_dfa *aa_dfa_unpack(void *blob, size_t size, int flags) if (size < sizeof(struct table_set_header)) goto fail; - if (ntohl(*(__be32 *) data) != YYTH_MAGIC) + if (get_unaligned_be32(data) != YYTH_MAGIC) goto fail; - hsize = ntohl(*(__be32 *) (data + 4)); + hsize = get_unaligned_be32(data + 4); if (size < hsize) goto fail; - dfa->flags = ntohs(*(__be16 *) (data + 12)); + dfa->flags = get_unaligned_be16(data + 12); if (dfa->flags & ~(YYTH_FLAGS)) goto fail; @@ -309,7 +309,7 @@ struct aa_dfa *aa_dfa_unpack(void *blob, size_t size, int flags) * if (dfa->flags & YYTH_FLAGS_OOB_TRANS) { * if (hsize < 16 + 4) * goto fail; - * dfa->max_oob = ntol(*(__be32 *) (data + 16)); + * dfa->max_oob = get_unaligned_be32(data + 16); * if (dfa->max <= MAX_OOB_SUPPORTED) { * pr_err("AppArmor DFA OOB greater than supported\n"); * goto fail; diff --git a/security/apparmor/policy.c b/security/apparmor/policy.c index a113afe79fde..5c7182c4192a 100644 --- a/security/apparmor/policy.c +++ b/security/apparmor/policy.c @@ -665,7 +665,7 @@ struct aa_profile *aa_alloc_null(struct aa_profile *parent, const char *name, } /** - * aa_new_learning_profile - create or find a null-X learning profile + * __aa_new_learning_profile - create or find a null-X learning profile * @parent: profile that caused this profile to be created (NOT NULL) * @hat: true if the null- learning profile is a hat * @base: name to base the null profile off of @@ -682,14 +682,16 @@ struct aa_profile *aa_alloc_null(struct aa_profile *parent, const char *name, * * Returns: new refcounted profile else NULL on failure */ -struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, - const char *base, gfp_t gfp) +struct aa_profile *__aa_new_learning_profile(struct aa_profile *parent, + bool hat, const char *base, + gfp_t gfp) { struct aa_profile *p, *profile; const char *bname; char *name = NULL; AA_BUG(!parent); + AA_BUG(!mutex_is_locked(&parent->ns->lock)); if (base) { name = kmalloc(strlen(parent->base.hname) + 8 + strlen(base), @@ -721,7 +723,6 @@ struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, if (hat) profile->label.flags |= FLAG_HAT; - mutex_lock_nested(&profile->ns->lock, profile->ns->level); p = __find_child(&parent->base.profiles, bname); if (p) { aa_free_profile(profile); @@ -729,7 +730,6 @@ struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, } else { __add_profile(&parent->base.profiles, profile); } - mutex_unlock(&profile->ns->lock); /* refcount released by caller */ out: @@ -743,6 +743,18 @@ struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, return NULL; } +struct aa_profile *aa_new_learning_profile(struct aa_profile *parent, bool hat, + const char *base, gfp_t gfp) +{ + struct aa_profile *profile; + + mutex_lock_nested(&parent->ns->lock, parent->ns->level); + profile = __aa_new_learning_profile(parent, hat, base, gfp); + mutex_unlock(&parent->ns->lock); + + return profile; +} + /** * replacement_allowed - test to see if replacement is allowed * @profile: profile to test if it can be replaced (MAYBE NULL) diff --git a/security/apparmor/task.c b/security/apparmor/task.c index 0d7af707cccd..3dd62624e88f 100644 --- a/security/apparmor/task.c +++ b/security/apparmor/task.c @@ -14,6 +14,7 @@ #include #include +#include #include "include/audit.h" #include "include/cred.h" @@ -88,6 +89,32 @@ int aa_replace_current_label(struct aa_label *label) return 0; } +static void aa_replace_stale_label_tw_func(struct callback_head *tw) +{ + struct aa_task_ctx *ctx = task_ctx(current); + struct aa_label *label; + + ctx->label_replacement_pending = false; + label = aa_current_raw_label(); + if (!label_is_stale(label)) + return; + label = aa_get_newest_label(label); + aa_replace_current_label(label); + aa_put_label(label); +} + +/* replace the current task's stale label on syscall return */ +void aa_schedule_stale_label_replacement(void) +{ + struct aa_task_ctx *ctx = task_ctx(current); + + if (ctx->label_replacement_pending) + return; + init_task_work(&ctx->label_replacement_tw, aa_replace_stale_label_tw_func); + if (task_work_add(current, &ctx->label_replacement_tw, TWA_RESUME) == 0) + ctx->label_replacement_pending = true; +} + /** * aa_set_current_onexec - set the tasks change_profile to happen onexec diff --git a/security/keys/key.c b/security/keys/key.c index 35db23d05302..4e49d2e3b858 100644 --- a/security/keys/key.c +++ b/security/keys/key.c @@ -825,9 +825,6 @@ static key_ref_t __key_create_or_update(key_ref_t keyring_ref, key_check(keyring); - if (!(flags & KEY_ALLOC_BYPASS_RESTRICTION)) - restrict_link = keyring->restrict_link; - key_ref = ERR_PTR(-ENOTDIR); if (keyring->type != &key_type_keyring) goto error_put_type; @@ -865,6 +862,9 @@ static key_ref_t __key_create_or_update(key_ref_t keyring_ref, goto error_link_end; } + if (!(flags & KEY_ALLOC_BYPASS_RESTRICTION)) + restrict_link = keyring->restrict_link; + if (restrict_link && restrict_link->check) { ret = restrict_link->check(keyring, index_key.type, &prep.payload, restrict_link->key); diff --git a/security/keys/persistent.c b/security/keys/persistent.c index 97af230aa4b2..bdab99539326 100644 --- a/security/keys/persistent.c +++ b/security/keys/persistent.c @@ -33,25 +33,31 @@ static int key_create_persistent_register(struct user_namespace *ns) } /* - * Create the persistent keyring for the specified user. + * Get or create the persistent keyring for the specified user. * * Called with the namespace's sem locked for writing. + * + * Returns a boolean indicating whether the keyring already existed, + * or a negative error. On success, *persistent_ref holds a reference + * to the keyring. */ -static key_ref_t key_create_persistent(struct user_namespace *ns, kuid_t uid, - struct keyring_index_key *index_key) +static int key_get_or_create_persistent(struct user_namespace *ns, kuid_t uid, + struct keyring_index_key *index_key, + key_ref_t *persistent_ref) { struct key *persistent; - key_ref_t reg_ref, persistent_ref; + key_ref_t reg_ref; if (!ns->persistent_keyring_register) { - long err = key_create_persistent_register(ns); + int err = key_create_persistent_register(ns); + if (err < 0) - return ERR_PTR(err); + return err; } else { reg_ref = make_key_ref(ns->persistent_keyring_register, true); - persistent_ref = find_key_to_update(reg_ref, index_key); - if (persistent_ref) - return persistent_ref; + *persistent_ref = find_key_to_update(reg_ref, index_key); + if (*persistent_ref) + return 1; } persistent = keyring_alloc(index_key->description, @@ -61,9 +67,10 @@ static key_ref_t key_create_persistent(struct user_namespace *ns, kuid_t uid, KEY_ALLOC_NOT_IN_QUOTA, NULL, ns->persistent_keyring_register); if (IS_ERR(persistent)) - return ERR_CAST(persistent); + return PTR_ERR(persistent); - return make_key_ref(persistent, true); + *persistent_ref = make_key_ref(persistent, true); + return 0; } /* @@ -78,6 +85,7 @@ static long key_get_persistent(struct user_namespace *ns, kuid_t uid, key_ref_t reg_ref, persistent_ref; char buf[32]; long ret; + bool created = false; /* Look in the register if it exists */ memset(&index_key, 0, sizeof(index_key)); @@ -100,23 +108,24 @@ static long key_get_persistent(struct user_namespace *ns, kuid_t uid, * also need to create the register. */ down_write(&ns->keyring_sem); - persistent_ref = key_create_persistent(ns, uid, &index_key); + ret = key_get_or_create_persistent(ns, uid, &index_key, + &persistent_ref); up_write(&ns->keyring_sem); - if (!IS_ERR(persistent_ref)) - goto found; - - return PTR_ERR(persistent_ref); + if (ret < 0) + return ret; + created = ret == 0; found: + persistent = key_ref_to_ptr(persistent_ref); ret = key_task_permission(persistent_ref, current_cred(), KEY_NEED_LINK); - if (ret == 0) { - persistent = key_ref_to_ptr(persistent_ref); + if (ret == 0) ret = key_link(key_ref_to_ptr(dest_ref), persistent); - if (ret == 0) { - key_set_timeout(persistent, persistent_keyring_expiry); - ret = persistent->serial; - } - } + + if (ret == 0 || created) + key_set_timeout(persistent, persistent_keyring_expiry); + + if (ret == 0) + ret = persistent->serial; key_ref_put(persistent_ref); return ret; diff --git a/security/keys/trusted-keys/trusted_tpm1.c b/security/keys/trusted-keys/trusted_tpm1.c index ddcbfaede684..3b1ee879419c 100644 --- a/security/keys/trusted-keys/trusted_tpm1.c +++ b/security/keys/trusted-keys/trusted_tpm1.c @@ -1058,10 +1058,10 @@ static int __init trusted_tpm_init(void) static void trusted_tpm_exit(void) { if (chip) { + unregister_key_type(&key_type_trusted); put_device(&chip->dev); kfree(digests); trusted_shash_release(); - unregister_key_type(&key_type_trusted); } } diff --git a/security/landlock/fs.c b/security/landlock/fs.c index 9c1cee4c4c3c..91773a7f27e7 100644 --- a/security/landlock/fs.c +++ b/security/landlock/fs.c @@ -908,11 +908,12 @@ static int current_check_refer_path(struct dentry *const old_dentry, /* * old_dentry may be the root of the common mount point and * !IS_ROOT(old_dentry) at the same time (e.g. with open_tree() and - * OPEN_TREE_CLONE). We do not need to call dget(old_parent) because - * we keep a reference to old_dentry. + * OPEN_TREE_CLONE). Pin the dentry used as old_parent in either case. + * Otherwise, dget_parent() safely fetches and pins the current parent + * against a concurrent rename(2). */ - old_parent = (old_dentry == mnt_dir.dentry) ? old_dentry : - old_dentry->d_parent; + old_parent = (old_dentry == mnt_dir.dentry) ? dget(old_dentry) : + dget_parent(old_dentry); /* new_dir->dentry is equal to new_dentry->d_parent */ allow_parent1 = collect_domain_accesses(dom, mnt_dir.dentry, old_parent, @@ -920,8 +921,10 @@ static int current_check_refer_path(struct dentry *const old_dentry, allow_parent2 = collect_domain_accesses( dom, mnt_dir.dentry, new_dir->dentry, &layer_masks_parent2); - if (allow_parent1 && allow_parent2) + if (allow_parent1 && allow_parent2) { + dput(old_parent); return 0; + } /* * To be able to compare source and destination domain access rights, @@ -932,8 +935,12 @@ static int current_check_refer_path(struct dentry *const old_dentry, if (is_access_to_paths_allowed( dom, &mnt_dir, access_request_parent1, &layer_masks_parent1, old_dentry, access_request_parent2, &layer_masks_parent2, - exchange ? new_dentry : NULL)) + exchange ? new_dentry : NULL)) { + dput(old_parent); return 0; + } + + dput(old_parent); /* * This prioritizes EACCES over EXDEV for all actions, including diff --git a/security/selinux/avc.c b/security/selinux/avc.c index d1e3c22a4393..e752013984f9 100644 --- a/security/selinux/avc.c +++ b/security/selinux/avc.c @@ -581,7 +581,7 @@ static int avc_latest_notif_update(u32 seqno, int is_insert) } } else { if (seqno > selinux_avc.avc_cache.latest_notif) - selinux_avc.avc_cache.latest_notif = seqno; + WRITE_ONCE(selinux_avc.avc_cache.latest_notif, seqno); } spin_unlock_irqrestore(¬if_lock, flag); @@ -1208,5 +1208,6 @@ int avc_has_perm(u32 ssid, u32 tsid, u16 tclass, u32 avc_policy_seqno(void) { - return selinux_avc.avc_cache.latest_notif; + /* Writers hold notif_lock; readers only sample the sequence number. */ + return READ_ONCE(selinux_avc.avc_cache.latest_notif); } diff --git a/security/selinux/hooks.c b/security/selinux/hooks.c index 7c57cdc42614..9f1bc322bde6 100644 --- a/security/selinux/hooks.c +++ b/security/selinux/hooks.c @@ -670,8 +670,12 @@ static int selinux_set_mnt_opts(struct super_block *sb, * will be used for both mounts) */ if ((sbsec->flags & SE_SBINITIALIZED) && (sb->s_type->fs_flags & FS_BINARY_MOUNTDATA) - && !opts) + && !opts) { + if ((kern_flags & SECURITY_LSM_NATIVE_LABELS) && + sbsec->behavior == SECURITY_FS_USE_NATIVE) + *set_kern_flags |= SECURITY_LSM_NATIVE_LABELS; goto out; + } root_isec = backing_inode_security_novalidate(root); @@ -715,6 +719,9 @@ static int selinux_set_mnt_opts(struct super_block *sb, /* previously mounted with options, but not on this attempt? */ if ((sbsec->flags & SE_MNTMASK) && !opts) goto out_double_mount; + if ((kern_flags & SECURITY_LSM_NATIVE_LABELS) && + sbsec->behavior == SECURITY_FS_USE_NATIVE) + *set_kern_flags |= SECURITY_LSM_NATIVE_LABELS; rc = 0; goto out; } diff --git a/security/selinux/selinuxfs.c b/security/selinux/selinuxfs.c index 3e28fa2444fb..d425137c0d1b 100644 --- a/security/selinux/selinuxfs.c +++ b/security/selinux/selinuxfs.c @@ -74,7 +74,6 @@ struct selinux_fs_info { int *bool_pending_values; struct dentry *class_dir; unsigned long last_class_ino; - bool policy_opened; struct dentry *policycap_dir; unsigned long last_ino; struct super_block *sb; @@ -343,44 +342,31 @@ struct policy_load_memory { static int sel_open_policy(struct inode *inode, struct file *filp) { - struct selinux_fs_info *fsi = inode->i_sb->s_fs_info; struct policy_load_memory *plm = NULL; int rc; - BUG_ON(filp->private_data); - - mutex_lock(&selinux_state.policy_mutex); - rc = avc_has_perm(current_sid(), SECINITSID_SECURITY, SECCLASS_SECURITY, SECURITY__READ_POLICY, NULL); if (rc) - goto err; - - rc = -EBUSY; - if (fsi->policy_opened) - goto err; + return rc; - rc = -ENOMEM; plm = kzalloc(sizeof(*plm), GFP_KERNEL); if (!plm) - goto err; + return -ENOMEM; + mutex_lock(&selinux_state.policy_mutex); rc = security_read_policy(&plm->data, &plm->len); if (rc) goto err; - if ((size_t)i_size_read(inode) != plm->len) { inode_lock(inode); i_size_write(inode, plm->len); inode_unlock(inode); } - - fsi->policy_opened = 1; + mutex_unlock(&selinux_state.policy_mutex); filp->private_data = plm; - mutex_unlock(&selinux_state.policy_mutex); - return 0; err: mutex_unlock(&selinux_state.policy_mutex); @@ -393,13 +379,8 @@ static int sel_open_policy(struct inode *inode, struct file *filp) static int sel_release_policy(struct inode *inode, struct file *filp) { - struct selinux_fs_info *fsi = inode->i_sb->s_fs_info; struct policy_load_memory *plm = filp->private_data; - BUG_ON(!plm); - - fsi->policy_opened = 0; - vfree(plm->data); kfree(plm); diff --git a/security/selinux/ss/policydb.c b/security/selinux/ss/policydb.c index 81e910aaf22a..9e55c2c53243 100644 --- a/security/selinux/ss/policydb.c +++ b/security/selinux/ss/policydb.c @@ -1127,6 +1127,10 @@ static int perm_read(struct policydb *p, struct symtab *s, void *fp) len = le32_to_cpu(buf[0]); perdatum->value = le32_to_cpu(buf[1]); + rc = -EINVAL; + /* indexes an nprim-sized array in security_get_permissions() */ + if (perdatum->value > s->nprim) + goto bad; rc = str_read(&key, GFP_KERNEL, fp, len); if (rc) diff --git a/sound/core/control_led.c b/sound/core/control_led.c index 15acb07c6515..639d86e4dd16 100644 --- a/sound/core/control_led.c +++ b/sound/core/control_led.c @@ -253,6 +253,10 @@ static int snd_ctl_led_set_id(int card_number, struct snd_ctl_elem_id *id, down_write(&card->controls_rwsem); kctl = snd_ctl_find_id_locked(card, id); if (kctl) { + if (!kctl->info || !kctl->get) { + err = -EINVAL; + goto unlock; + } ioff = snd_ctl_get_ioff(kctl, id); vd = &kctl->vd[ioff]; access = vd->access & SNDRV_CTL_ELEM_ACCESS_LED_MASK; diff --git a/sound/core/oss/mixer_oss.c b/sound/core/oss/mixer_oss.c index abc99ae33328..ecb4af670959 100644 --- a/sound/core/oss/mixer_oss.c +++ b/sound/core/oss/mixer_oss.c @@ -540,6 +540,10 @@ static void snd_mixer_oss_get_volume1_vol(struct snd_mixer_oss_file *fmixer, if (numid == ID_UNKNOWN) return; down_read(&card->controls_rwsem); + if (card->shutdown) { + up_read(&card->controls_rwsem); + return; + } kctl = snd_ctl_find_numid_locked(card, numid); if (!kctl) { up_read(&card->controls_rwsem); @@ -579,6 +583,10 @@ static void snd_mixer_oss_get_volume1_sw(struct snd_mixer_oss_file *fmixer, if (numid == ID_UNKNOWN) return; down_read(&card->controls_rwsem); + if (card->shutdown) { + up_read(&card->controls_rwsem); + return; + } kctl = snd_ctl_find_numid_locked(card, numid); if (!kctl) { up_read(&card->controls_rwsem); @@ -645,6 +653,10 @@ static void snd_mixer_oss_put_volume1_vol(struct snd_mixer_oss_file *fmixer, if (numid == ID_UNKNOWN) return; down_read(&card->controls_rwsem); + if (card->shutdown) { + up_read(&card->controls_rwsem); + return; + } kctl = snd_ctl_find_numid_locked(card, numid); if (!kctl) { up_read(&card->controls_rwsem); @@ -688,6 +700,10 @@ static void snd_mixer_oss_put_volume1_sw(struct snd_mixer_oss_file *fmixer, if (numid == ID_UNKNOWN) return; down_read(&card->controls_rwsem); + if (card->shutdown) { + up_read(&card->controls_rwsem); + return; + } kctl = snd_ctl_find_numid_locked(card, numid); if (!kctl) { up_read(&card->controls_rwsem); @@ -833,6 +849,10 @@ static int snd_mixer_oss_get_recsrc2(struct snd_mixer_oss_file *fmixer, unsigned goto __free_only; } down_read(&card->controls_rwsem); + if (card->shutdown) { + err = -ENODEV; + goto __unlock; + } kctl = snd_mixer_oss_test_id(mixer, "Capture Source", 0); if (! kctl) { err = -ENOENT; @@ -886,6 +906,10 @@ static int snd_mixer_oss_put_recsrc2(struct snd_mixer_oss_file *fmixer, unsigned goto __free_only; } down_read(&card->controls_rwsem); + if (card->shutdown) { + err = -ENODEV; + goto __unlock; + } kctl = snd_mixer_oss_test_id(mixer, "Capture Source", 0); if (! kctl) { err = -ENOENT; @@ -937,6 +961,10 @@ static int snd_mixer_oss_build_test(struct snd_mixer_oss *mixer, struct slot *sl int err; down_read(&card->controls_rwsem); + if (card->shutdown) { + up_read(&card->controls_rwsem); + return -ENODEV; + } kcontrol = snd_mixer_oss_test_id(mixer, name, index); if (kcontrol == NULL) { up_read(&card->controls_rwsem); @@ -1069,6 +1097,10 @@ static int snd_mixer_oss_build_input(struct snd_mixer_oss *mixer, if (snd_mixer_oss_build_test_all(mixer, ptr, &slot)) return 0; down_read(&mixer->card->controls_rwsem); + if (mixer->card->shutdown) { + up_read(&mixer->card->controls_rwsem); + return -ENODEV; + } kctl = NULL; if (!ptr->index) kctl = snd_mixer_oss_test_id(mixer, "Capture Source", 0); diff --git a/sound/core/oss/pcm_oss.c b/sound/core/oss/pcm_oss.c index 23ecbd70ccf6..15683237c7ee 100644 --- a/sound/core/oss/pcm_oss.c +++ b/sound/core/oss/pcm_oss.c @@ -2161,10 +2161,18 @@ static int snd_pcm_oss_get_trigger(struct snd_pcm_oss_file *pcm_oss_file) psubstream = pcm_oss_file->streams[SNDRV_PCM_STREAM_PLAYBACK]; csubstream = pcm_oss_file->streams[SNDRV_PCM_STREAM_CAPTURE]; - if (psubstream && psubstream->runtime && psubstream->runtime->oss.trigger) - result |= PCM_ENABLE_OUTPUT; - if (csubstream && csubstream->runtime && csubstream->runtime->oss.trigger) - result |= PCM_ENABLE_INPUT; + if (psubstream && psubstream->runtime) { + mutex_lock(&psubstream->runtime->oss.params_lock); + if (psubstream->runtime->oss.trigger) + result |= PCM_ENABLE_OUTPUT; + mutex_unlock(&psubstream->runtime->oss.params_lock); + } + if (csubstream && csubstream->runtime) { + mutex_lock(&csubstream->runtime->oss.params_lock); + if (csubstream->runtime->oss.trigger) + result |= PCM_ENABLE_INPUT; + mutex_unlock(&csubstream->runtime->oss.params_lock); + } return result; } @@ -2839,6 +2847,18 @@ static int snd_pcm_oss_capture_ready(struct snd_pcm_substream *substream) runtime->oss.period_frames; } +static bool need_input_retrigger(struct snd_pcm_runtime *runtime) +{ + bool ret; + + mutex_lock(&runtime->oss.params_lock); + ret = runtime->oss.trigger; + if (ret) + runtime->oss.trigger = 0; + mutex_unlock(&runtime->oss.params_lock); + return ret; +} + static __poll_t snd_pcm_oss_poll(struct file *file, poll_table * wait) { struct snd_pcm_oss_file *pcm_oss_file; @@ -2871,11 +2891,11 @@ static __poll_t snd_pcm_oss_poll(struct file *file, poll_table * wait) snd_pcm_oss_capture_ready(csubstream)) mask |= EPOLLIN | EPOLLRDNORM; snd_pcm_stream_unlock_irq(csubstream); - if (ostate != SNDRV_PCM_STATE_RUNNING && runtime->oss.trigger) { + if (ostate != SNDRV_PCM_STATE_RUNNING && + need_input_retrigger(runtime)) { struct snd_pcm_oss_file ofile; memset(&ofile, 0, sizeof(ofile)); ofile.streams[SNDRV_PCM_STREAM_CAPTURE] = pcm_oss_file->streams[SNDRV_PCM_STREAM_CAPTURE]; - runtime->oss.trigger = 0; snd_pcm_oss_set_trigger(&ofile, PCM_ENABLE_INPUT); } } @@ -2964,8 +2984,10 @@ static void snd_pcm_oss_proc_read(struct snd_info_entry *entry, struct snd_info_buffer *buffer) { struct snd_pcm_str *pstr = entry->private_data; - struct snd_pcm_oss_setup *setup = pstr->oss.setup_list; + struct snd_pcm_oss_setup *setup; + mutex_lock(&pstr->oss.setup_mutex); + setup = pstr->oss.setup_list; while (setup) { snd_iprintf(buffer, "%s %u %u%s%s%s%s%s%s\n", setup->task_name, @@ -3053,6 +3075,14 @@ static void snd_pcm_oss_proc_write(struct snd_info_entry *entry, mutex_unlock(&pstr->oss.setup_mutex); return; } + template.task_name = kstrdup(task_name, GFP_KERNEL); + if (!template.task_name) { + kfree(setup); + buffer->error = -ENOMEM; + mutex_unlock(&pstr->oss.setup_mutex); + return; + } + *setup = template; if (pstr->oss.setup_list == NULL) pstr->oss.setup_list = setup; else { @@ -3060,13 +3090,8 @@ static void snd_pcm_oss_proc_write(struct snd_info_entry *entry, setup1->next; setup1 = setup1->next); setup1->next = setup; } - template.task_name = kstrdup(task_name, GFP_KERNEL); - if (! template.task_name) { - kfree(setup); - buffer->error = -ENOMEM; - mutex_unlock(&pstr->oss.setup_mutex); - return; - } + mutex_unlock(&pstr->oss.setup_mutex); + continue; } *setup = template; mutex_unlock(&pstr->oss.setup_mutex); diff --git a/sound/core/seq/oss/seq_oss_event.c b/sound/core/seq/oss/seq_oss_event.c index 7b7c925dd3aa..4699c084d018 100644 --- a/sound/core/seq/oss/seq_oss_event.c +++ b/sound/core/seq/oss/seq_oss_event.c @@ -39,8 +39,10 @@ static int set_echo_event(struct seq_oss_devinfo *dp, union evrec *rec, struct s */ int -snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, struct snd_seq_event *ev) +snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, + struct snd_seq_event *ev, snd_use_lock_t **lockp) { + *lockp = NULL; switch (q->s.code) { case SEQ_EXTENDED: return extended_event(dp, q, ev); @@ -69,7 +71,7 @@ snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, struct snd if (snd_seq_oss_midi_open(dp, q->s.dev, SNDRV_SEQ_OSS_FILE_WRITE)) break; if (snd_seq_oss_midi_filemode(dp, q->s.dev) & SNDRV_SEQ_OSS_FILE_WRITE) - return snd_seq_oss_midi_putc(dp, q->s.dev, q->s.parm1, ev); + return snd_seq_oss_midi_putc(dp, q->s.dev, q->s.parm1, ev, lockp); break; case SEQ_ECHO: diff --git a/sound/core/seq/oss/seq_oss_event.h b/sound/core/seq/oss/seq_oss_event.h index b4f723949a17..a4524e51d0e9 100644 --- a/sound/core/seq/oss/seq_oss_event.h +++ b/sound/core/seq/oss/seq_oss_event.h @@ -91,7 +91,8 @@ union evrec { #define ev_is_long(ev) ((ev)->s.code >= 128) #define ev_length(ev) ((ev)->s.code >= 128 ? LONG_EVENT_SIZE : SHORT_EVENT_SIZE) -int snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, struct snd_seq_event *ev); +int snd_seq_oss_process_event(struct seq_oss_devinfo *dp, union evrec *q, + struct snd_seq_event *ev, snd_use_lock_t **lockp); int snd_seq_oss_process_timer_event(struct seq_oss_timer *rec, union evrec *q); int snd_seq_oss_event_input(struct snd_seq_event *ev, int direct, void *private_data, int atomic, int hop); diff --git a/sound/core/seq/oss/seq_oss_ioctl.c b/sound/core/seq/oss/seq_oss_ioctl.c index ccf682689ec9..ce7a69d52b30 100644 --- a/sound/core/seq/oss/seq_oss_ioctl.c +++ b/sound/core/seq/oss/seq_oss_ioctl.c @@ -45,14 +45,17 @@ static int snd_seq_oss_oob_user(struct seq_oss_devinfo *dp, void __user *arg) { unsigned char ev[8]; struct snd_seq_event tmpev; + snd_use_lock_t *lock = NULL; if (copy_from_user(ev, arg, 8)) return -EFAULT; memset(&tmpev, 0, sizeof(tmpev)); snd_seq_oss_fill_addr(dp, &tmpev, dp->addr.client, dp->addr.port); tmpev.time.tick = 0; - if (! snd_seq_oss_process_event(dp, (union evrec *)ev, &tmpev)) { + if (!snd_seq_oss_process_event(dp, (union evrec *)ev, &tmpev, &lock)) { snd_seq_oss_dispatch(dp, &tmpev, 0, 0); + if (lock) + snd_use_lock_free(lock); } return 0; } diff --git a/sound/core/seq/oss/seq_oss_midi.c b/sound/core/seq/oss/seq_oss_midi.c index f2940b29595f..15ee97f3e907 100644 --- a/sound/core/seq/oss/seq_oss_midi.c +++ b/sound/core/seq/oss/seq_oss_midi.c @@ -639,7 +639,8 @@ send_midi_event(struct seq_oss_devinfo *dp, struct snd_seq_event *ev, struct seq * non-zero : invalid - ignored */ int -snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c, struct snd_seq_event *ev) +snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c, + struct snd_seq_event *ev, snd_use_lock_t **lockp) { struct seq_oss_midi *mdev; @@ -648,7 +649,8 @@ snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c, stru return -ENODEV; if (snd_midi_event_encode_byte(mdev->coder, c, ev)) { snd_seq_oss_fill_addr(dp, ev, mdev->client, mdev->port); - snd_use_lock_free(&mdev->use_lock); + /* the caller must release this later */ + *lockp = &mdev->use_lock; return 0; } snd_use_lock_free(&mdev->use_lock); diff --git a/sound/core/seq/oss/seq_oss_midi.h b/sound/core/seq/oss/seq_oss_midi.h index bcc1683773df..4819d4170bf6 100644 --- a/sound/core/seq/oss/seq_oss_midi.h +++ b/sound/core/seq/oss/seq_oss_midi.h @@ -26,7 +26,7 @@ void snd_seq_oss_midi_open_all(struct seq_oss_devinfo *dp, int file_mode); int snd_seq_oss_midi_close(struct seq_oss_devinfo *dp, int dev); void snd_seq_oss_midi_reset(struct seq_oss_devinfo *dp, int dev); int snd_seq_oss_midi_putc(struct seq_oss_devinfo *dp, int dev, unsigned char c, - struct snd_seq_event *ev); + struct snd_seq_event *ev, snd_use_lock_t **lockp); int snd_seq_oss_midi_input(struct snd_seq_event *ev, int direct, void *private); int snd_seq_oss_midi_filemode(struct seq_oss_devinfo *dp, int dev); int snd_seq_oss_midi_make_info(struct seq_oss_devinfo *dp, int dev, struct midi_info *inf); diff --git a/sound/core/seq/oss/seq_oss_readq.c b/sound/core/seq/oss/seq_oss_readq.c index f0db5d3dcba4..b7cdb1c61f26 100644 --- a/sound/core/seq/oss/seq_oss_readq.c +++ b/sound/core/seq/oss/seq_oss_readq.c @@ -73,13 +73,17 @@ snd_seq_oss_readq_delete(struct seq_oss_readq *q) void snd_seq_oss_readq_clear(struct seq_oss_readq *q) { - if (q->qlen) { - q->qlen = 0; - q->head = q->tail = 0; + scoped_guard(spinlock_irqsave, &q->lock) { + if (q->qlen) { + q->qlen = 0; + q->head = 0; + q->tail = 0; + } + q->input_time = (unsigned long)-1; } + /* if someone sleeping, wake'em up */ wake_up(&q->midi_sleep); - q->input_time = (unsigned long)-1; } /* @@ -136,28 +140,37 @@ int snd_seq_oss_readq_sysex(struct seq_oss_readq *q, int dev, /* * copy an event to input queue: * return zero if enqueued + * caller must hold lock */ -int -snd_seq_oss_readq_put_event(struct seq_oss_readq *q, union evrec *ev) +static int snd_seq_oss_readq_put_event_locked(struct seq_oss_readq *q, + union evrec *ev) { - unsigned long flags; - - spin_lock_irqsave(&q->lock, flags); - if (q->qlen >= q->maxlen - 1) { - spin_unlock_irqrestore(&q->lock, flags); + if (q->qlen >= q->maxlen - 1) return -ENOMEM; - } memcpy(&q->q[q->tail], ev, sizeof(*ev)); q->tail = (q->tail + 1) % q->maxlen; q->qlen++; - /* wake up sleeper */ - wake_up(&q->midi_sleep); + return 0; +} - spin_unlock_irqrestore(&q->lock, flags); +/* + * copy an event to input queue: + * return zero if enqueued + */ +int +snd_seq_oss_readq_put_event(struct seq_oss_readq *q, union evrec *ev) +{ + int rc; - return 0; + scoped_guard(spinlock_irqsave, &q->lock) { + rc = snd_seq_oss_readq_put_event_locked(q, ev); + if (!rc) + wake_up(&q->midi_sleep); + } + + return rc; } @@ -215,23 +228,31 @@ snd_seq_oss_readq_poll(struct seq_oss_readq *q, struct file *file, poll_table *w int snd_seq_oss_readq_put_timestamp(struct seq_oss_readq *q, unsigned long curt, int seq_mode) { - if (curt != q->input_time) { - union evrec rec; - memset(&rec, 0, sizeof(rec)); - switch (seq_mode) { - case SNDRV_SEQ_OSS_MODE_SYNTH: - rec.echo = (curt << 8) | SEQ_WAIT; - snd_seq_oss_readq_put_event(q, &rec); - break; - case SNDRV_SEQ_OSS_MODE_MUSIC: - rec.t.code = EV_TIMING; - rec.t.cmd = TMR_WAIT_ABS; - rec.t.time = curt; - snd_seq_oss_readq_put_event(q, &rec); - break; + int queued = 0; + + scoped_guard(spinlock_irqsave, &q->lock) { + if (curt != q->input_time) { + union evrec rec; + + memset(&rec, 0, sizeof(rec)); + switch (seq_mode) { + case SNDRV_SEQ_OSS_MODE_SYNTH: + rec.echo = (curt << 8) | SEQ_WAIT; + queued = !snd_seq_oss_readq_put_event_locked(q, &rec); + break; + case SNDRV_SEQ_OSS_MODE_MUSIC: + rec.t.code = EV_TIMING; + rec.t.cmd = TMR_WAIT_ABS; + rec.t.time = curt; + queued = !snd_seq_oss_readq_put_event_locked(q, &rec); + break; + } + q->input_time = curt; } - q->input_time = curt; } + if (queued) + wake_up(&q->midi_sleep); + return 0; } diff --git a/sound/core/seq/oss/seq_oss_rw.c b/sound/core/seq/oss/seq_oss_rw.c index 45bb45859770..b7147ac78ee8 100644 --- a/sound/core/seq/oss/seq_oss_rw.c +++ b/sound/core/seq/oss/seq_oss_rw.c @@ -154,6 +154,7 @@ insert_queue(struct seq_oss_devinfo *dp, union evrec *rec, struct file *opt) { int rc = 0; struct snd_seq_event event; + snd_use_lock_t *lock = NULL; /* if this is a timing event, process the current time */ if (snd_seq_oss_process_timer_event(dp->timer, rec)) @@ -165,7 +166,7 @@ insert_queue(struct seq_oss_devinfo *dp, union evrec *rec, struct file *opt) event.type = SNDRV_SEQ_EVENT_NOTEOFF; snd_seq_oss_fill_addr(dp, &event, dp->addr.client, dp->addr.port); - if (snd_seq_oss_process_event(dp, rec, &event)) + if (snd_seq_oss_process_event(dp, rec, &event, &lock)) return 0; /* invalid event - no need to insert queue */ event.time.tick = snd_seq_oss_timer_cur_tick(dp->timer); @@ -174,6 +175,8 @@ insert_queue(struct seq_oss_devinfo *dp, union evrec *rec, struct file *opt) else rc = snd_seq_kernel_client_enqueue(dp->cseq, &event, opt, !is_nonblock_mode(dp->file_mode)); + if (lock) + snd_use_lock_free(lock); return rc; } diff --git a/sound/core/seq/seq_clientmgr.c b/sound/core/seq/seq_clientmgr.c index 1fddeb91c9f4..f039fd6ee3a7 100644 --- a/sound/core/seq/seq_clientmgr.c +++ b/sound/core/seq/seq_clientmgr.c @@ -70,7 +70,7 @@ static int bounce_error_event(struct snd_seq_client *client, int err, int atomic, int hop); static int snd_seq_deliver_single_event(struct snd_seq_client *client, struct snd_seq_event *event, - int filter, int atomic, int hop); + int atomic, int hop); #if IS_ENABLED(CONFIG_SND_SEQ_UMP) static void free_ump_info(struct snd_seq_client *client); @@ -463,7 +463,7 @@ static ssize_t snd_seq_read(struct file *file, char __user *buf, size_t count, cell = NULL; err = 0; - snd_seq_fifo_lock(fifo); + guard(snd_seq_fifo)(fifo); if (IS_ENABLED(CONFIG_SND_SEQ_UMP) && client->midi_version > 0) aligned_size = sizeof(struct snd_seq_ump_event); @@ -522,7 +522,6 @@ static ssize_t snd_seq_read(struct file *file, char __user *buf, size_t count, if (err == -EAGAIN && result > 0) err = 0; } - snd_seq_fifo_unlock(fifo); return (err < 0) ? err : result; } @@ -540,10 +539,8 @@ static int check_port_perm(struct snd_seq_client_port *port, unsigned int flags) /* * check if the destination client is available, and return the pointer - * if filter is non-zero, client filter bitmap is tested. */ -static struct snd_seq_client *get_event_dest_client(struct snd_seq_event *event, - int filter) +static struct snd_seq_client *get_event_dest_client(struct snd_seq_event *event) { struct snd_seq_client *dest; @@ -558,8 +555,6 @@ static struct snd_seq_client *get_event_dest_client(struct snd_seq_event *event, if ((dest->filter & SNDRV_SEQ_FILTER_USE_EVENT) && ! test_bit(event->type, dest->event_filter)) goto __not_avail; - if (filter && !(dest->filter & filter)) - goto __not_avail; return dest; /* ok - accessible */ __not_avail: @@ -593,17 +588,39 @@ static int bounce_error_event(struct snd_seq_client *client, /* set up quoted error */ memset(&bounce_ev, 0, sizeof(bounce_ev)); - bounce_ev.type = SNDRV_SEQ_EVENT_KERNEL_ERROR; - bounce_ev.flags = SNDRV_SEQ_EVENT_LENGTH_FIXED; + + if (client->type == USER_CLIENT) { + /* + * For user clients, send SNDRV_SEQ_EVENT_BOUNCE with the + * original event embedded as variable-length data. This + * avoids exposing data.quote.event (a kernel pointer) to + * userspace. The variable-length path in snd_seq_event_dup() + * copies the event data from data.ext.ptr into chained cells, + * and snd_seq_expand_var_event() copies only the data content + * -- never the pointer -- to userspace. + */ + bounce_ev.type = SNDRV_SEQ_EVENT_BOUNCE; + bounce_ev.flags = SNDRV_SEQ_EVENT_LENGTH_VARIABLE; + bounce_ev.data.ext.len = sizeof(struct snd_seq_event); + bounce_ev.data.ext.ptr = (char *)event; + } else { + /* + * For kernel clients, quote the event pointer directly. + * Kernel consumers can safely dereference the pointer. + */ + bounce_ev.type = SNDRV_SEQ_EVENT_KERNEL_ERROR; + bounce_ev.flags = SNDRV_SEQ_EVENT_LENGTH_FIXED; + bounce_ev.data.quote.origin = event->dest; + bounce_ev.data.quote.event = event; + bounce_ev.data.quote.value = -err; /* use positive value */ + } + bounce_ev.queue = SNDRV_SEQ_QUEUE_DIRECT; bounce_ev.source.client = SNDRV_SEQ_CLIENT_SYSTEM; bounce_ev.source.port = SNDRV_SEQ_PORT_SYSTEM_ANNOUNCE; bounce_ev.dest.client = client->number; bounce_ev.dest.port = event->source.port; - bounce_ev.data.quote.origin = event->dest; - bounce_ev.data.quote.event = event; - bounce_ev.data.quote.value = -err; /* use positive value */ - result = snd_seq_deliver_single_event(NULL, &bounce_ev, 0, atomic, hop + 1); + result = snd_seq_deliver_single_event(NULL, &bounce_ev, atomic, hop + 1); if (result < 0) { client->event_lost++; return result; @@ -670,7 +687,7 @@ int __snd_seq_deliver_single_event(struct snd_seq_client *dest, */ static int snd_seq_deliver_single_event(struct snd_seq_client *client, struct snd_seq_event *event, - int filter, int atomic, int hop) + int atomic, int hop) { struct snd_seq_client *dest = NULL; struct snd_seq_client_port *dest_port = NULL; @@ -679,7 +696,7 @@ static int snd_seq_deliver_single_event(struct snd_seq_client *client, direct = snd_seq_ev_is_direct(event); - dest = get_event_dest_client(event, filter); + dest = get_event_dest_client(event); if (dest == NULL) goto __skip; dest_port = snd_seq_port_use_ptr(dest, event->dest.port); @@ -771,8 +788,7 @@ static int __deliver_to_subscribers(struct snd_seq_client *client, /* convert time according to flag with subscription */ update_timestamp_of_queue(event, subs->info.queue, subs->info.flags & SNDRV_SEQ_PORT_SUBS_TIME_REAL); - err = snd_seq_deliver_single_event(client, event, - 0, atomic, hop); + err = snd_seq_deliver_single_event(client, event, atomic, hop); if (err < 0) { /* save first error that occurs and continue */ if (!result) @@ -853,7 +869,7 @@ static int snd_seq_deliver_event(struct snd_seq_client *client, struct snd_seq_e event->dest.client == SNDRV_SEQ_ADDRESS_SUBSCRIBERS) result = deliver_to_subscribers(client, event, atomic, hop); else - result = snd_seq_deliver_single_event(client, event, 0, atomic, hop); + result = snd_seq_deliver_single_event(client, event, atomic, hop); return result; } diff --git a/sound/core/seq/seq_compat.c b/sound/core/seq/seq_compat.c index 1e35bf086a51..e26200fb351e 100644 --- a/sound/core/seq/seq_compat.c +++ b/sound/core/seq/seq_compat.c @@ -44,7 +44,9 @@ static int snd_seq_call_port_info_ioctl(struct snd_seq_client *client, unsigned goto error; data->kernel = NULL; - err = snd_seq_kernel_client_ctl(client->number, cmd, data); + scoped_guard(mutex, &client->ioctl_mutex) { + err = snd_seq_kernel_client_ctl(client->number, cmd, data); + } if (err < 0) goto error; diff --git a/sound/core/seq/seq_fifo.c b/sound/core/seq/seq_fifo.c index f8e02e98709a..c3dfe04c3883 100644 --- a/sound/core/seq/seq_fifo.c +++ b/sound/core/seq/seq_fifo.c @@ -88,12 +88,11 @@ void snd_seq_fifo_clear(struct snd_seq_fifo *f) atomic_set(&f->overflow, 0); snd_use_lock_sync(&f->use_lock); - spin_lock_irq(&f->lock); + guard(spinlock_irq)(&f->lock); /* drain the fifo */ while ((cell = fifo_cell_out(f)) != NULL) { snd_seq_cell_free(cell); } - spin_unlock_irq(&f->lock); } @@ -102,38 +101,48 @@ int snd_seq_fifo_event_in(struct snd_seq_fifo *f, struct snd_seq_event *event) { struct snd_seq_event_cell *cell; - unsigned long flags; + struct snd_seq_pool *pool; + bool linked; int err; if (snd_BUG_ON(!f)) return -EINVAL; - snd_use_lock_use(&f->use_lock); - err = snd_seq_event_dup(f->pool, event, &cell, 1, NULL, NULL); /* always non-blocking */ + guard(snd_seq_fifo)(f); +retry: + pool = READ_ONCE(f->pool); + err = snd_seq_event_dup(pool, event, &cell, 1, NULL, NULL); /* always non-blocking */ if (err < 0) { if ((err == -ENOMEM) || (err == -EAGAIN)) atomic_inc(&f->overflow); - snd_use_lock_free(&f->use_lock); return err; } /* append new cells to fifo */ - spin_lock_irqsave(&f->lock, flags); - if (f->tail != NULL) - f->tail->next = cell; - f->tail = cell; - if (f->head == NULL) - f->head = cell; - cell->next = NULL; - f->cells++; - spin_unlock_irqrestore(&f->lock, flags); + linked = false; + scoped_guard(spinlock_irqsave, &f->lock) { + if (cell->pool == f->pool) { + if (f->tail) + f->tail->next = cell; + f->tail = cell; + if (!f->head) + f->head = cell; + cell->next = NULL; + f->cells++; + linked = true; + } + } + + if (!linked) { + /* Retry against the replacement pool after resize publishes it. */ + snd_seq_cell_free(cell); + goto retry; + } /* wakeup client */ if (waitqueue_active(&f->input_sleep)) wake_up(&f->input_sleep); - snd_use_lock_free(&f->use_lock); - return 0; /* success */ } @@ -199,16 +208,21 @@ int snd_seq_fifo_cell_out(struct snd_seq_fifo *f, void snd_seq_fifo_cell_putback(struct snd_seq_fifo *f, struct snd_seq_event_cell *cell) { - unsigned long flags; + bool linked = false; if (cell) { - spin_lock_irqsave(&f->lock, flags); - cell->next = f->head; - f->head = cell; - if (!f->tail) - f->tail = cell; - f->cells++; - spin_unlock_irqrestore(&f->lock, flags); + scoped_guard(spinlock_irqsave, &f->lock) { + if (cell->pool == f->pool) { + cell->next = f->head; + f->head = cell; + if (!f->tail) + f->tail = cell; + f->cells++; + linked = true; + } + } + if (!linked) + snd_seq_cell_free(cell); } } @@ -239,17 +253,17 @@ int snd_seq_fifo_resize(struct snd_seq_fifo *f, int poolsize) return -ENOMEM; } - spin_lock_irq(&f->lock); - /* remember old pool */ - oldpool = f->pool; - oldhead = f->head; - /* exchange pools */ - f->pool = newpool; - f->head = NULL; - f->tail = NULL; - f->cells = 0; - /* NOTE: overflow flag is not cleared */ - spin_unlock_irq(&f->lock); + scoped_guard(spinlock_irq, &f->lock) { + /* remember old pool */ + oldpool = f->pool; + oldhead = f->head; + /* exchange pools */ + WRITE_ONCE(f->pool, newpool); + f->head = NULL; + f->tail = NULL; + f->cells = 0; + /* NOTE: overflow flag is not cleared */ + } /* close the old pool and wait until all users are gone */ snd_seq_pool_mark_closing(oldpool); @@ -268,16 +282,10 @@ int snd_seq_fifo_resize(struct snd_seq_fifo *f, int poolsize) /* get the number of unused cells safely */ int snd_seq_fifo_unused_cells(struct snd_seq_fifo *f) { - unsigned long flags; - int cells; - if (!f) return 0; - snd_use_lock_use(&f->use_lock); - spin_lock_irqsave(&f->lock, flags); - cells = snd_seq_unused_cells(f->pool); - spin_unlock_irqrestore(&f->lock, flags); - snd_use_lock_free(&f->use_lock); - return cells; + guard(snd_seq_fifo)(f); + guard(spinlock_irqsave)(&f->lock); + return snd_seq_unused_cells(f->pool); } diff --git a/sound/core/seq/seq_fifo.h b/sound/core/seq/seq_fifo.h index b56a7b897c9c..4c9c49127746 100644 --- a/sound/core/seq/seq_fifo.h +++ b/sound/core/seq/seq_fifo.h @@ -37,6 +37,7 @@ int snd_seq_fifo_event_in(struct snd_seq_fifo *f, struct snd_seq_event *event); /* lock fifo from release */ #define snd_seq_fifo_lock(fifo) snd_use_lock_use(&(fifo)->use_lock) #define snd_seq_fifo_unlock(fifo) snd_use_lock_free(&(fifo)->use_lock) +DEFINE_GUARD(snd_seq_fifo, struct snd_seq_fifo *, snd_seq_fifo_lock(_T), snd_seq_fifo_unlock(_T)) /* get a cell from fifo - fifo should be locked */ int snd_seq_fifo_cell_out(struct snd_seq_fifo *f, struct snd_seq_event_cell **cellp, int nonblock); diff --git a/sound/drivers/mtpav.c b/sound/drivers/mtpav.c index f212f233ea61..1fb6797fa45f 100644 --- a/sound/drivers/mtpav.c +++ b/sound/drivers/mtpav.c @@ -663,12 +663,8 @@ static int snd_mtpav_get_RAWMIDI(struct mtpav *mcard) static void snd_mtpav_free(struct snd_card *card) { struct mtpav *crd = card->private_data; - unsigned long flags; - spin_lock_irqsave(&crd->spinlock, flags); - if (crd->istimer > 0) - snd_mtpav_remove_output_timer(crd); - spin_unlock_irqrestore(&crd->spinlock, flags); + timer_shutdown_sync(&crd->timer); } /* diff --git a/sound/pci/hda/cs35l41_hda.c b/sound/pci/hda/cs35l41_hda.c index cbc350ec03cc..c8de2cb4e90a 100644 --- a/sound/pci/hda/cs35l41_hda.c +++ b/sound/pci/hda/cs35l41_hda.c @@ -1072,6 +1072,44 @@ static int cs35l41_fw_type_ctl_info(struct snd_kcontrol *kcontrol, struct snd_ct return snd_ctl_enum_info(uinfo, 1, ARRAY_SIZE(hda_cs_dsp_fw_ids), hda_cs_dsp_fw_ids); } +static void cs35l41_remove_controls(struct cs35l41_hda *cs35l41) +{ + guard(mutex)(&cs35l41->ctl_lock); + + if (!cs35l41->codec) + return; + + if (cs35l41->fw_load_ctl) + snd_ctl_remove(cs35l41->codec->card, cs35l41->fw_load_ctl); + cs35l41->fw_load_ctl = NULL; + + if (cs35l41->fw_type_ctl) + snd_ctl_remove(cs35l41->codec->card, cs35l41->fw_type_ctl); + cs35l41->fw_type_ctl = NULL; +} + +static int cs35l41_add_control(struct cs35l41_hda *cs35l41, + struct snd_kcontrol_new *ctl, + struct snd_kcontrol **kctl) +{ + int ret; + + *kctl = snd_ctl_new1(ctl, cs35l41); + if (!*kctl) + return -ENOMEM; + + ret = snd_ctl_add(cs35l41->codec->card, *kctl); + if (ret) { + dev_err(cs35l41->dev, "Failed to add KControl %s = %d\n", ctl->name, ret); + *kctl = NULL; + return ret; + } + + dev_dbg(cs35l41->dev, "Added Control %s\n", ctl->name); + + return 0; +} + static int cs35l41_create_controls(struct cs35l41_hda *cs35l41) { char fw_type_ctl_name[SNDRV_CTL_ELEM_ID_NAME_MAXLEN]; @@ -1097,23 +1135,19 @@ static int cs35l41_create_controls(struct cs35l41_hda *cs35l41) scnprintf(fw_load_ctl_name, SNDRV_CTL_ELEM_ID_NAME_MAXLEN, "%s DSP1 Firmware Load", cs35l41->amp_name); - ret = snd_ctl_add(cs35l41->codec->card, snd_ctl_new1(&fw_type_ctl, cs35l41)); - if (ret) { - dev_err(cs35l41->dev, "Failed to add KControl %s = %d\n", fw_type_ctl.name, ret); - return ret; - } - - dev_dbg(cs35l41->dev, "Added Control %s\n", fw_type_ctl.name); - - ret = snd_ctl_add(cs35l41->codec->card, snd_ctl_new1(&fw_load_ctl, cs35l41)); - if (ret) { - dev_err(cs35l41->dev, "Failed to add KControl %s = %d\n", fw_load_ctl.name, ret); - return ret; - } + ret = cs35l41_add_control(cs35l41, &fw_type_ctl, &cs35l41->fw_type_ctl); + if (ret) + goto err; - dev_dbg(cs35l41->dev, "Added Control %s\n", fw_load_ctl.name); + ret = cs35l41_add_control(cs35l41, &fw_load_ctl, &cs35l41->fw_load_ctl); + if (ret) + goto err; return 0; + +err: + cs35l41_remove_controls(cs35l41); + return ret; } static int cs35l41_hda_bind(struct device *dev, struct device *master, void *master_data) @@ -1189,6 +1223,13 @@ static void cs35l41_hda_unbind(struct device *dev, struct device *master, void * sleep_flags = lock_system_sleep(); device_link_remove(&cs35l41->codec->core.dev, cs35l41->dev); unlock_system_sleep(sleep_flags); + + cs35l41_remove_controls(cs35l41); + cancel_work_sync(&cs35l41->fw_load_work); + + mutex_lock(&cs35l41->ctl_lock); + cs35l41->codec = NULL; + mutex_unlock(&cs35l41->ctl_lock); } } @@ -1653,6 +1694,7 @@ int cs35l41_hda_probe(struct device *dev, const char *device_name, int id, int i INIT_WORK(&cs35l41->fw_load_work, cs35l41_fw_load_work); mutex_init(&cs35l41->fw_mutex); + mutex_init(&cs35l41->ctl_lock); pm_runtime_set_autosuspend_delay(cs35l41->dev, 3000); pm_runtime_use_autosuspend(cs35l41->dev); @@ -1696,6 +1738,9 @@ void cs35l41_hda_remove(struct device *dev) { struct cs35l41_hda *cs35l41 = dev_get_drvdata(dev); + cs35l41_remove_controls(cs35l41); + cancel_work_sync(&cs35l41->fw_load_work); + pm_runtime_get_sync(cs35l41->dev); pm_runtime_dont_use_autosuspend(cs35l41->dev); pm_runtime_disable(cs35l41->dev); diff --git a/sound/pci/hda/cs35l41_hda.h b/sound/pci/hda/cs35l41_hda.h index b93bf762976e..545c7200cc7c 100644 --- a/sound/pci/hda/cs35l41_hda.h +++ b/sound/pci/hda/cs35l41_hda.h @@ -45,6 +45,8 @@ enum cs35l41_hda_gpio_function { CS35l41_SYNC, }; +struct snd_kcontrol; + struct cs35l41_hda { struct device *dev; struct regmap *regmap; @@ -61,7 +63,10 @@ struct cs35l41_hda { int firmware_type; int speaker_id; struct mutex fw_mutex; + struct mutex ctl_lock; struct work_struct fw_load_work; + struct snd_kcontrol *fw_type_ctl; + struct snd_kcontrol *fw_load_ctl; struct regmap_irq_chip_data *irq_data; bool firmware_running; diff --git a/sound/pci/hda/hda_tegra.c b/sound/pci/hda/hda_tegra.c index 12a144a269ee..de47ea7b9ee1 100644 --- a/sound/pci/hda/hda_tegra.c +++ b/sound/pci/hda/hda_tegra.c @@ -376,7 +376,8 @@ static int hda_tegra_first_init(struct azx *chip, struct platform_device *pdev) * in powers of 2, next available ratio is 16 which can be * used as a limiting factor here. */ - if (of_device_is_compatible(np, "nvidia,tegra30-hda")) + if (of_device_is_compatible(np, "nvidia,tegra30-hda") || + of_device_is_compatible(np, "nvidia,tegra194-hda")) chip->bus.core.sdo_limit = 16; /* codec detection */ diff --git a/sound/pci/hda/patch_realtek.c b/sound/pci/hda/patch_realtek.c index 99e266f01748..c7f392c0dd73 100644 --- a/sound/pci/hda/patch_realtek.c +++ b/sound/pci/hda/patch_realtek.c @@ -7511,6 +7511,7 @@ enum { ALC286_FIXUP_ACER_AIO_MIC_NO_PRESENCE, ALC294_FIXUP_ASUS_MIC, ALC294_FIXUP_ASUS_HEADSET_MIC, + ALC294_FIXUP_ASUS_I2C_HEADSET_MIC, ALC294_FIXUP_ASUS_SPK, ALC293_FIXUP_SYSTEM76_MIC_NO_PRESENCE, ALC285_FIXUP_LENOVO_PC_BEEP_IN_NOISE, @@ -8824,6 +8825,15 @@ static const struct hda_fixup alc269_fixups[] = { .chained = true, .chain_id = ALC269_FIXUP_HEADSET_MIC }, + [ALC294_FIXUP_ASUS_I2C_HEADSET_MIC] = { + .type = HDA_FIXUP_PINS, + .v.pins = (const struct hda_pintbl[]) { + { 0x19, 0x03a19020 }, /* use as headset mic */ + { } + }, + .chained = true, + .chain_id = ALC287_FIXUP_CS35L41_I2C_2 + }, [ALC294_FIXUP_ASUS_SPK] = { .type = HDA_FIXUP_VERBS, .v.verbs = (const struct hda_verb[]) { @@ -10359,7 +10369,7 @@ static const struct hda_quirk alc269_fixup_tbl[] = { SND_PCI_QUIRK(0x1043, 0x1b13, "Asus U41SV", ALC269_FIXUP_INV_DMIC), SND_PCI_QUIRK(0x1043, 0x1b93, "ASUS G614JVR/JIR", ALC245_FIXUP_CS35L41_SPI_2), SND_PCI_QUIRK(0x1043, 0x1bbd, "ASUS Z550MA", ALC255_FIXUP_ASUS_MIC_NO_PRESENCE), - SND_PCI_QUIRK(0x1043, 0x1c03, "ASUS UM3406HA", ALC287_FIXUP_CS35L41_I2C_2), + SND_PCI_QUIRK(0x1043, 0x1c03, "ASUS UM3406HA", ALC294_FIXUP_ASUS_I2C_HEADSET_MIC), SND_PCI_QUIRK(0x1043, 0x1c23, "Asus X55U", ALC269_FIXUP_LIMIT_INT_MIC_BOOST), SND_PCI_QUIRK(0x1043, 0x1c33, "ASUS UX5304MA", ALC245_FIXUP_CS35L41_SPI_2), SND_PCI_QUIRK(0x1043, 0x1c43, "ASUS UX8406MA", ALC245_FIXUP_CS35L41_SPI_2), diff --git a/sound/pci/ice1712/ice1712.c b/sound/pci/ice1712/ice1712.c index 45cb3e7b8d21..ce1758ad7200 100644 --- a/sound/pci/ice1712/ice1712.c +++ b/sound/pci/ice1712/ice1712.c @@ -2549,7 +2549,7 @@ static int snd_ice1712_probe(struct pci_dev *pci, const struct pci_device_id *pci_id) { static int dev; - struct snd_card *card __free(snd_card_unref) = NULL; + struct snd_card *card __free(snd_card_free) = NULL; struct snd_ice1712 *ice; int pcm_dev = 0, err; const struct snd_ice1712_card_info * const *tbl, *c; diff --git a/sound/soc/amd/acp/acp-mach-common.c b/sound/soc/amd/acp/acp-mach-common.c index fc4e91535578..ecfabdda2fd0 100644 --- a/sound/soc/amd/acp/acp-mach-common.c +++ b/sound/soc/amd/acp/acp-mach-common.c @@ -126,8 +126,13 @@ static int acp_card_rt5682_init(struct snd_soc_pcm_runtime *rtd) if (drvdata->hs_codec_id != RT5682) return -EINVAL; - drvdata->wclk = clk_get(component->dev, "rt5682-dai-wclk"); - drvdata->bclk = clk_get(component->dev, "rt5682-dai-bclk"); + drvdata->wclk = devm_clk_get(component->dev, "rt5682-dai-wclk"); + if (IS_ERR(drvdata->wclk)) + return PTR_ERR(drvdata->wclk); + + drvdata->bclk = devm_clk_get(component->dev, "rt5682-dai-bclk"); + if (IS_ERR(drvdata->bclk)) + return PTR_ERR(drvdata->bclk); ret = snd_soc_dapm_new_controls(&card->dapm, rt5682_widgets, ARRAY_SIZE(rt5682_widgets)); @@ -352,8 +357,13 @@ static int acp_card_rt5682s_init(struct snd_soc_pcm_runtime *rtd) return -EINVAL; if (!drvdata->soc_mclk) { - drvdata->wclk = clk_get(component->dev, "rt5682-dai-wclk"); - drvdata->bclk = clk_get(component->dev, "rt5682-dai-bclk"); + drvdata->wclk = devm_clk_get(component->dev, "rt5682-dai-wclk"); + if (IS_ERR(drvdata->wclk)) + return PTR_ERR(drvdata->wclk); + + drvdata->bclk = devm_clk_get(component->dev, "rt5682-dai-bclk"); + if (IS_ERR(drvdata->bclk)) + return PTR_ERR(drvdata->bclk); } ret = snd_soc_dapm_new_controls(&card->dapm, rt5682s_widgets, diff --git a/sound/soc/amd/yc/acp6x-mach.c b/sound/soc/amd/yc/acp6x-mach.c index 429c2566add5..39d5edf8d321 100644 --- a/sound/soc/amd/yc/acp6x-mach.c +++ b/sound/soc/amd/yc/acp6x-mach.c @@ -696,6 +696,13 @@ static const struct dmi_system_id yc_acp_quirk_table[] = { DMI_MATCH(DMI_BOARD_NAME, "Herbag_MDU"), } }, + { + .driver_data = &acp6x_card, + .matches = { + DMI_MATCH(DMI_BOARD_VENDOR, "MDC"), + DMI_MATCH(DMI_BOARD_NAME, "Herbag2_MDU"), + } + }, { .driver_data = &acp6x_card, .matches = { diff --git a/sound/soc/codecs/adau1372.c b/sound/soc/codecs/adau1372.c index 7dc0b5c79e39..5f85edb33f66 100644 --- a/sound/soc/codecs/adau1372.c +++ b/sound/soc/codecs/adau1372.c @@ -798,7 +798,7 @@ static int adau1372_set_power(struct adau1372 *adau1372, bool enable) if (ret) return ret; if (adau1372->pd_gpio) - gpiod_set_value(adau1372->pd_gpio, 0); + gpiod_set_value_cansleep(adau1372->pd_gpio, 0); if (adau1372->switch_mode) adau1372->switch_mode(adau1372->dev); @@ -818,8 +818,7 @@ static int adau1372_set_power(struct adau1372 *adau1372, bool enable) ADAU1372_CLK_CTRL_PLL_EN, 0); regcache_cache_only(adau1372->regmap, true); - if (adau1372->pd_gpio) - gpiod_set_value(adau1372->pd_gpio, 1); + gpiod_set_value_cansleep(adau1372->pd_gpio, 1); clk_disable_unprepare(adau1372->mclk); return ret; } @@ -836,7 +835,7 @@ static int adau1372_set_power(struct adau1372 *adau1372, bool enable) * map. No need to do any register writes to manually * turn things off. */ - gpiod_set_value(adau1372->pd_gpio, 1); + gpiod_set_value_cansleep(adau1372->pd_gpio, 1); regcache_mark_dirty(adau1372->regmap); } else { regmap_update_bits(adau1372->regmap, ADAU1372_REG_CLK_CTRL, @@ -1003,6 +1002,17 @@ int adau1372_probe(struct device *dev, struct regmap *regmap, regmap_write(regmap, ADAU1372_REG_OP_STAGE_MUTE, 0x0); + /* + * Set sane default values for the Output ASRC and DAC input muxes, + * since the power-on reset defaults are invalid "Reserved" states. + */ + regmap_write(regmap, ADAU1372_REG_ASRCO_SOURCE_0_1, + 0x54); /* Decimator0, Decimator1 */ + regmap_write(regmap, ADAU1372_REG_ASRCO_SOURCE_2_3, + 0x76); /* Decimator2, Decimator3 */ + regmap_write(regmap, ADAU1372_REG_DAC_SOURCE, + 0xdc); /* Input ASRC0, Input ASRC1 */ + regmap_write(regmap, 0x7, 0x01); /* CLOCK OUT */ return devm_snd_soc_register_component(dev, &adau1372_driver, &adau1372_dai_driver, 1); diff --git a/sound/soc/codecs/aw88261.c b/sound/soc/codecs/aw88261.c index eed05b2508dd..7fe486b707b2 100644 --- a/sound/soc/codecs/aw88261.c +++ b/sound/soc/codecs/aw88261.c @@ -151,7 +151,7 @@ static int aw88261_dev_get_iis_status(struct aw_device *aw_dev) if (ret) return ret; if ((reg_val & AW88261_BIT_PLL_CHECK) != AW88261_BIT_PLL_CHECK) { - dev_err(aw_dev->dev, "check pll lock fail,reg_val:0x%04x", reg_val); + dev_dbg(aw_dev->dev, "check pll lock fail,reg_val:0x%04x", reg_val); return -EINVAL; } @@ -165,7 +165,7 @@ static int aw88261_dev_check_mode1_pll(struct aw_device *aw_dev) for (i = 0; i < AW88261_DEV_SYSST_CHECK_MAX; i++) { ret = aw88261_dev_get_iis_status(aw_dev); if (ret) { - dev_err(aw_dev->dev, "mode1 iis signal check error"); + dev_dbg(aw_dev->dev, "mode1 iis signal check error"); usleep_range(AW88261_2000_US, AW88261_2000_US + 10); } else { return ret; @@ -254,10 +254,10 @@ static int aw88261_dev_check_sysst(struct aw_device *aw_dev) return ret; check_val = reg_val & (~AW88261_BIT_SYSST_CHECK_MASK) - & AW88261_BIT_SYSST_CHECK; - if (check_val != AW88261_BIT_SYSST_CHECK) { - dev_err(aw_dev->dev, "check sysst fail, reg_val=0x%04x, check:0x%x", - reg_val, AW88261_BIT_SYSST_CHECK); + & AW88261_BIT_PLL_CHECK; + if (check_val != AW88261_BIT_PLL_CHECK) { + dev_dbg(aw_dev->dev, "check sysst fail, reg_val=0x%04x, check:0x%x", + reg_val, AW88261_BIT_PLL_CHECK); usleep_range(AW88261_2000_US, AW88261_2000_US + 10); } else { return 0; @@ -548,7 +548,7 @@ static int aw88261_dev_start(struct aw88261 *aw88261) int ret; if (aw_dev->status == AW88261_DEV_PW_ON) { - dev_info(aw_dev->dev, "already power on"); + dev_dbg(aw_dev->dev, "already power on"); return 0; } @@ -558,7 +558,7 @@ static int aw88261_dev_start(struct aw88261 *aw88261) ret = aw88261_dev_check_syspll(aw_dev); if (ret) { - dev_err(aw_dev->dev, "pll check failed cannot start"); + dev_dbg(aw_dev->dev, "pll check failed"); goto pll_check_fail; } @@ -569,7 +569,7 @@ static int aw88261_dev_start(struct aw88261 *aw88261) /* check i2s status */ ret = aw88261_dev_check_sysst(aw_dev); if (ret) { - dev_err(aw_dev->dev, "sysst check failed"); + dev_dbg(aw_dev->dev, "sysst check failed"); goto sysst_check_fail; } @@ -670,18 +670,22 @@ static void aw88261_start_pa(struct aw88261 *aw88261) for (i = 0; i < AW88261_START_RETRIES; i++) { ret = aw88261_reg_update(aw88261, aw88261->phase_sync); if (ret) { - dev_err(aw88261->aw_pa->dev, "fw update failed, cnt:%d\n", i); + dev_dbg(aw88261->aw_pa->dev, + "aw88261_reg_update failed, cnt:%d, ret:%d\n", i, ret); continue; } ret = aw88261_dev_start(aw88261); if (ret) { - dev_err(aw88261->aw_pa->dev, "aw88261 device start failed. retry = %d", i); + dev_dbg(aw88261->aw_pa->dev, + "aw88261_dev_start failed, cnt:%d, ret:%d\n", i, ret); continue; } else { - dev_info(aw88261->aw_pa->dev, "start success\n"); + dev_dbg(aw88261->aw_pa->dev, "start success\n"); break; } } + if (ret != 0) + dev_err(aw88261->aw_pa->dev, "start failure (%d)\n", ret); } static void aw88261_startup_work(struct work_struct *work) @@ -1210,7 +1214,7 @@ static int aw88261_init(struct aw88261 **aw88261, struct i2c_client *i2c, struct return ret; } if (chip_id != AW88261_CHIP_ID) { - dev_err(&i2c->dev, "unsupported device"); + dev_err(&i2c->dev, "unsupported device id = %x", chip_id); return -ENXIO; } diff --git a/sound/soc/codecs/aw88261.h b/sound/soc/codecs/aw88261.h index 4f3dbf438510..79ce52f60283 100644 --- a/sound/soc/codecs/aw88261.h +++ b/sound/soc/codecs/aw88261.h @@ -181,12 +181,6 @@ AW88261_OTHS_OT_VALUE | \ AW88261_PLLS_LOCKED_VALUE)) -#define AW88261_BIT_SYSST_CHECK \ - (AW88261_BSTS_FINISHED_VALUE | \ - AW88261_SWS_SWITCHING_VALUE | \ - AW88261_CLKS_STABLE_VALUE | \ - AW88261_PLLS_LOCKED_VALUE) - #define AW88261_ULS_HMUTE_START_BIT (14) #define AW88261_ULS_HMUTE_BITS_LEN (1) #define AW88261_ULS_HMUTE_MASK \ diff --git a/sound/soc/codecs/hdac_hda.c b/sound/soc/codecs/hdac_hda.c index d59d38ce5657..c497354251cb 100644 --- a/sound/soc/codecs/hdac_hda.c +++ b/sound/soc/codecs/hdac_hda.c @@ -612,10 +612,8 @@ static int hdac_hda_dev_probe(struct hdac_device *hdev) &hdac_hda_codec, hdac_hda_dais, ARRAY_SIZE(hdac_hda_dais)); - if (ret < 0) { + if (ret < 0) dev_err(&hdev->dev, "failed to register HDA codec %d\n", ret); - return ret; - } snd_hdac_ext_bus_link_put(hdev->bus, hlink); diff --git a/sound/soc/codecs/max98363.c b/sound/soc/codecs/max98363.c index a2cca3436c68..4fc46dd8f007 100644 --- a/sound/soc/codecs/max98363.c +++ b/sound/soc/codecs/max98363.c @@ -205,7 +205,7 @@ static int max98363_sdw_dai_hw_params(struct snd_pcm_substream *substream, struct max98363_priv *max98363 = snd_soc_component_get_drvdata(component); - struct sdw_stream_config stream_config; + struct sdw_stream_config stream_config = {0}; struct sdw_port_config port_config; enum sdw_data_direction direction; struct sdw_stream_runtime *stream; diff --git a/sound/soc/codecs/rt1017-sdca-sdw.c b/sound/soc/codecs/rt1017-sdca-sdw.c index 7295f44c77eb..70f121883d7e 100644 --- a/sound/soc/codecs/rt1017-sdca-sdw.c +++ b/sound/soc/codecs/rt1017-sdca-sdw.c @@ -16,6 +16,7 @@ #include #include #include +#include #include #include @@ -575,11 +576,10 @@ static int rt1017_sdca_pcm_hw_params(struct snd_pcm_substream *substream, { struct snd_soc_component *component = dai->component; struct rt1017_sdca_priv *rt1017 = snd_soc_component_get_drvdata(component); - struct sdw_stream_config stream_config; + struct sdw_stream_config stream_config = {0}; struct sdw_port_config port_config; - enum sdw_data_direction direction; struct sdw_stream_runtime *sdw_stream; - int retval, port, num_channels, ch_mask; + int retval, port; unsigned int sampling_rate; dev_dbg(dai->dev, "%s %s", __func__, dai->name); @@ -593,28 +593,17 @@ static int rt1017_sdca_pcm_hw_params(struct snd_pcm_substream *substream, /* SoundWire specific configuration */ /* port 1 for playback */ - if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) { - direction = SDW_DATA_DIR_RX; + if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) port = 1; - } else { - direction = SDW_DATA_DIR_TX; + else port = 2; - } - - num_channels = params_channels(params); - ch_mask = (1 << num_channels) - 1; - - stream_config.frame_rate = params_rate(params); - stream_config.ch_count = num_channels; - stream_config.bps = snd_pcm_format_width(params_format(params)); - stream_config.direction = direction; - port_config.ch_mask = ch_mask; + snd_sdw_params_to_config(substream, params, &stream_config, &port_config); port_config.num = port; dev_dbg(dai->dev, "frame_rate %d, ch_count %d, bps %d, direction %d, ch_mask %d, port: %d\n", - params_rate(params), num_channels, snd_pcm_format_width(params_format(params)), - direction, ch_mask, port); + stream_config.frame_rate, stream_config.ch_count, stream_config.bps, + stream_config.direction, port_config.ch_mask, port); retval = sdw_stream_add_slave(rt1017->sdw_slave, &stream_config, &port_config, 1, sdw_stream); diff --git a/sound/soc/codecs/rt274.c b/sound/soc/codecs/rt274.c index 9a33e3776b55..d29ddd501510 100644 --- a/sound/soc/codecs/rt274.c +++ b/sound/soc/codecs/rt274.c @@ -619,9 +619,11 @@ static int rt274_hw_params(struct snd_pcm_substream *substream, unsigned int val = 0; int d_len_code = 0, c_len_code = 0; + /* bit 14 Sample Rate Base, 0: 48kHz 1: 44.1kHz */ switch (params_rate(params)) { - /* bit 14 0:48K 1:44.1K */ case 44100: + val |= BIT(14); + break; case 48000: break; default: diff --git a/sound/soc/codecs/rt286.c b/sound/soc/codecs/rt286.c index 981155b046fd..ec3d450736da 100644 --- a/sound/soc/codecs/rt286.c +++ b/sound/soc/codecs/rt286.c @@ -951,7 +951,7 @@ static int rt286_probe(struct snd_soc_component *component) if (rt286->i2c->irq) schedule_delayed_work(&rt286->jack_detect_work, - msecs_to_jiffies(50)); + msecs_to_jiffies(1250)); return 0; } diff --git a/sound/soc/codecs/wm8903.c b/sound/soc/codecs/wm8903.c index 84ae1102ac88..1c6b22d78f57 100644 --- a/sound/soc/codecs/wm8903.c +++ b/sound/soc/codecs/wm8903.c @@ -677,7 +677,7 @@ SOC_ENUM("DRC Decay Rate", drc_decay), SOC_ENUM("DRC FF Delay", drc_ff_delay), SOC_SINGLE("DRC Anticlip Switch", WM8903_DRC_0, 1, 1, 0), SOC_SINGLE("DRC QR Switch", WM8903_DRC_0, 2, 1, 0), -SOC_SINGLE_TLV("DRC QR Threshold Volume", WM8903_DRC_0, 6, 3, 0, drc_tlv_max), +SOC_SINGLE_TLV("DRC QR Threshold Volume", WM8903_DRC_1, 6, 3, 0, drc_tlv_max), SOC_ENUM("DRC QR Decay Rate", drc_qr_decay), SOC_SINGLE("DRC Smoothing Switch", WM8903_DRC_0, 3, 1, 0), SOC_SINGLE("DRC Smoothing Hysteresis Switch", WM8903_DRC_0, 0, 1, 0), diff --git a/sound/soc/intel/avs/pcm.c b/sound/soc/intel/avs/pcm.c index 9251c38cf9d1..12e31027d99c 100644 --- a/sound/soc/intel/avs/pcm.c +++ b/sound/soc/intel/avs/pcm.c @@ -1178,7 +1178,10 @@ int avs_soc_component_register(struct device *dev, const char *name, return ret; /* force name change after ASoC is done with its init */ - acomp->base.name = name; + acomp->base.name = devm_kstrdup(dev, name, GFP_KERNEL); + if (!acomp->base.name) + return -ENOMEM; + INIT_LIST_HEAD(&acomp->node); return snd_soc_add_component(&acomp->base, cpu_dais, num_cpu_dais); diff --git a/sound/soc/qcom/lpass-cpu.c b/sound/soc/qcom/lpass-cpu.c index bdb5e0c740a9..ceefdfa1dc38 100644 --- a/sound/soc/qcom/lpass-cpu.c +++ b/sound/soc/qcom/lpass-cpu.c @@ -1238,8 +1238,8 @@ int asoc_qcom_lpass_cpu_platform_probe(struct platform_device *pdev) } if (drvdata->mi2s_playback_sd_mode[dai_id] == LPAIF_I2SCTL_MODE_QUAD01) { - variant->dai_driver[dai_id].playback.channels_min = 4; - variant->dai_driver[dai_id].playback.channels_max = 4; + variant->dai_driver[i].playback.channels_min = 4; + variant->dai_driver[i].playback.channels_max = 4; } } diff --git a/sound/soc/qcom/qdsp6/q6asm.c b/sound/soc/qcom/qdsp6/q6asm.c index 9f53cffe5184..b6bdb977029d 100644 --- a/sound/soc/qcom/qdsp6/q6asm.c +++ b/sound/soc/qcom/qdsp6/q6asm.c @@ -653,6 +653,11 @@ static int32_t q6asm_stream_callback(struct apr_device *adev, goto done; } break; + case ASM_DATA_CMD_EOS: + case ASM_DATA_CMD_READ_V2: + case ASM_DATA_CMD_WRITE_V2: + /* response as result of close stream */ + goto done; default: dev_err(ac->dev, "command[0x%x] not expecting rsp\n", result->opcode); diff --git a/sound/soc/samsung/i2s.c b/sound/soc/samsung/i2s.c index 925782e67673..6dae7f06234a 100644 --- a/sound/soc/samsung/i2s.c +++ b/sound/soc/samsung/i2s.c @@ -1547,9 +1547,16 @@ static int samsung_i2s_probe(struct platform_device *pdev) goto err_disable_pm; priv->op_clk = clk_get_parent(priv->clk_table[CLK_I2S_RCLK_SRC]); + ret = clk_prepare_enable(priv->op_clk); + if (ret < 0) { + priv->op_clk = NULL; + goto err_unregister_clk; + } return 0; +err_unregister_clk: + i2s_unregister_clock_provider(priv); err_disable_pm: pm_runtime_disable(&pdev->dev); err_del_sec: @@ -1572,6 +1579,7 @@ static void samsung_i2s_remove(struct platform_device *pdev) i2s_unregister_clock_provider(priv); i2s_delete_secondary_device(priv); + clk_disable_unprepare(priv->op_clk); clk_disable_unprepare(priv->clk); pm_runtime_put_noidle(&pdev->dev); diff --git a/sound/soc/sof/intel/hda-dai.c b/sound/soc/sof/intel/hda-dai.c index a3d0b6d721aa..f541cd0fe78a 100644 --- a/sound/soc/sof/intel/hda-dai.c +++ b/sound/soc/sof/intel/hda-dai.c @@ -454,6 +454,12 @@ int sdw_hda_dai_hw_params(struct snd_pcm_substream *substream, struct snd_sof_dev *sdev; int ret; + if (!w) { + dev_err(cpu_dai->dev, "%s widget not found, check amp link num in the topology\n", + cpu_dai->name); + return -EINVAL; + } + ret = non_hda_dai_hw_params(substream, params, cpu_dai); if (ret < 0) { dev_err(cpu_dai->dev, "%s: non_hda_dai_hw_params failed %d\n", __func__, ret); diff --git a/sound/soc/sof/intel/hda.c b/sound/soc/sof/intel/hda.c index 15e6779efaa3..aa3f9c961173 100644 --- a/sound/soc/sof/intel/hda.c +++ b/sound/soc/sof/intel/hda.c @@ -102,6 +102,11 @@ static int sdw_params_stream(struct device *dev, struct snd_soc_dapm_widget *w = snd_soc_dai_get_widget(d, params_data->substream->stream); struct snd_sof_dai_config_data data = { 0 }; + if (!w) { + dev_err(dev, "%s widget not found, check amp link num in the topology\n", + d->name); + return -EINVAL; + } data.dai_index = (params_data->link_id << 8) | d->id; data.dai_data = params_data->alh_stream_id; diff --git a/sound/soc/sof/ipc4-topology.c b/sound/soc/sof/ipc4-topology.c index 39e4f0fdd5e4..8911c075251c 100644 --- a/sound/soc/sof/ipc4-topology.c +++ b/sound/soc/sof/ipc4-topology.c @@ -534,6 +534,7 @@ static int sof_ipc4_widget_setup_comp_dai(struct snd_sof_widget *swidget) dev_dbg(scomp->dev, "dai %s node_type %u dai_type %u dai_index %d\n", swidget->widget->name, node_type, ipc4_copier->dai_type, ipc4_copier->dai_index); + dai->type = ipc4_copier->dai_type; ipc4_copier->data.gtw_cfg.node_id = SOF_IPC4_NODE_TYPE(node_type); pipe_widget = swidget->spipe->pipe_widget; @@ -576,10 +577,16 @@ static int sof_ipc4_widget_setup_comp_dai(struct snd_sof_widget *swidget) } list_for_each_entry(w, &sdev->widget_list, list) { - if (w->widget->sname && + struct snd_sof_dai *alh_dai; + + if (!WIDGET_IS_DAI(w->id) || !w->widget->sname || strcmp(w->widget->sname, swidget->widget->sname)) continue; + alh_dai = w->private; + if (alh_dai->type != SOF_DAI_INTEL_ALH) + continue; + blob->alh_cfg.device_count++; } @@ -1691,11 +1698,13 @@ sof_ipc4_prepare_copier_module(struct snd_sof_widget *swidget, */ i = 0; list_for_each_entry(w, &sdev->widget_list, list) { - if (w->widget->sname && + if (!WIDGET_IS_DAI(w->id) || !w->widget->sname || strcmp(w->widget->sname, swidget->widget->sname)) continue; dai = w->private; + if (dai->type != SOF_DAI_INTEL_ALH) + continue; alh_copier = (struct sof_ipc4_copier *)dai->private; alh_data = &alh_copier->data; blob->alh_cfg.mapping[i].device = alh_data->gtw_cfg.node_id; diff --git a/sound/soc/sof/sof-audio.h b/sound/soc/sof/sof-audio.h index 7620596ead07..13b9026ffd4d 100644 --- a/sound/soc/sof/sof-audio.h +++ b/sound/soc/sof/sof-audio.h @@ -514,6 +514,7 @@ struct snd_sof_route { struct snd_sof_dai { struct snd_soc_component *scomp; const char *name; + u32 type; int number_configs; int current_config; diff --git a/sound/soc/sof/topology.c b/sound/soc/sof/topology.c index 7eb8eb35b137..011095722a80 100644 --- a/sound/soc/sof/topology.c +++ b/sound/soc/sof/topology.c @@ -2364,7 +2364,10 @@ static int sof_dspless_widget_ready(struct snd_soc_component *scomp, int index, struct snd_soc_tplg_dapm_widget *tw) { if (WIDGET_IS_DAI(w->id)) { + static const struct sof_topology_token dai_tokens[] = { + {SOF_TKN_DAI_TYPE, SND_SOC_TPLG_TUPLE_TYPE_STRING, get_token_dai_type, 0}}; struct snd_sof_dev *sdev = snd_soc_component_get_drvdata(scomp); + struct snd_soc_tplg_private *priv = &tw->priv; struct snd_sof_widget *swidget; struct snd_sof_dai dai; int ret; @@ -2375,6 +2378,14 @@ static int sof_dspless_widget_ready(struct snd_soc_component *scomp, int index, memset(&dai, 0, sizeof(dai)); + ret = sof_parse_tokens(scomp, &dai.type, dai_tokens, ARRAY_SIZE(dai_tokens), + priv->array, le32_to_cpu(priv->size)); + if (ret < 0) { + dev_err(scomp->dev, "Failed to parse DAI tokens for %s\n", tw->name); + kfree(swidget); + return ret; + } + ret = sof_connect_dai_widget(scomp, w, tw, &dai); if (ret) { kfree(swidget); diff --git a/sound/soc/tegra/tegra186_asrc.c b/sound/soc/tegra/tegra186_asrc.c index 208e2fcefcf2..798842c4a65f 100644 --- a/sound/soc/tegra/tegra186_asrc.c +++ b/sound/soc/tegra/tegra186_asrc.c @@ -831,7 +831,7 @@ static const struct snd_kcontrol_new tegra186_asrc_controls[] = { tegra186_asrc_put_input_threshold), SOC_SINGLE_EXT("Stream6 Input Threshold", - ASRC_STREAM_REG(TEGRA186_ASRC_RX_THRESHOLD, 4), 0, 3, 0, + ASRC_STREAM_REG(TEGRA186_ASRC_RX_THRESHOLD, 5), 0, 3, 0, tegra186_asrc_get_input_threshold, tegra186_asrc_put_input_threshold), diff --git a/sound/usb/caiaq/device.c b/sound/usb/caiaq/device.c index 3d821fde4582..f604846ec6e6 100644 --- a/sound/usb/caiaq/device.c +++ b/sound/usb/caiaq/device.c @@ -562,6 +562,9 @@ static int snd_probe(struct usb_interface *intf, ret = init_card(caiaqdev(card)); if (ret < 0) { dev_err(&usb_dev->dev, "unable to init card! (ret=%d)\n", ret); +#ifdef CONFIG_SND_USB_CAIAQ_INPUT + snd_usb_caiaq_input_disconnect(caiaqdev(card)); +#endif snd_card_free(card); return ret; } diff --git a/sound/usb/card.h b/sound/usb/card.h index 6ec95b2edf86..0d860c57d9d9 100644 --- a/sound/usb/card.h +++ b/sound/usb/card.h @@ -89,7 +89,7 @@ struct snd_usb_endpoint { struct snd_urb_ctx urb[MAX_URBS]; struct snd_usb_packet_info { - uint32_t packet_size[MAX_PACKS_HS]; + int packet_size[MAX_PACKS_HS]; int packets; } next_packet[MAX_URBS]; unsigned int next_packet_head; /* ring buffer offset to read */ @@ -115,6 +115,7 @@ struct snd_usb_endpoint { unsigned int phase; /* phase accumulator */ unsigned int maxpacksize; /* max packet size in bytes */ unsigned int maxframesize; /* max packet size in frames */ + unsigned int max_urb_packs; /* packets allocated per data URB */ unsigned int max_urb_frames; /* max URB size in frames */ unsigned int curpacksize; /* current packet size in bytes (for capture) */ unsigned int curframesize; /* current packet size in frames (for capture) */ diff --git a/sound/usb/endpoint.c b/sound/usb/endpoint.c index c393a9125b9b..9736a7a72e77 100644 --- a/sound/usb/endpoint.c +++ b/sound/usb/endpoint.c @@ -478,7 +478,7 @@ int snd_usb_queue_pending_output_urbs(struct snd_usb_endpoint *ep, unsigned long flags; struct snd_usb_packet_info *packet; struct snd_urb_ctx *ctx = NULL; - int err, i; + int err; spin_lock_irqsave(&ep->lock, flags); if ((!implicit_fb || ep->next_packet_queued > 0) && @@ -497,9 +497,10 @@ int snd_usb_queue_pending_output_urbs(struct snd_usb_endpoint *ep, /* copy over the length information */ if (implicit_fb) { - ctx->packets = packet->packets; - for (i = 0; i < packet->packets; i++) - ctx->packet_size[i] = packet->packet_size[i]; + ctx->packets = min_t(int, packet->packets, + ep->max_urb_packs); + memcpy(ctx->packet_size, packet->packet_size, + ctx->packets * sizeof(packet->packet_size[0])); } /* call the data handler to fill in playback data */ @@ -1256,15 +1257,16 @@ static int data_ep_set_params(struct snd_usb_endpoint *ep) ep->nurbs = min(max_urbs, urbs_per_period * ep->cur_buffer_periods); } + if (fmt->fmt_type == UAC_FORMAT_TYPE_II) + urb_packs++; /* for transfer delimiter */ + ep->max_urb_packs = urb_packs; + /* allocate and initialize data urbs */ for (i = 0; i < ep->nurbs; i++) { struct snd_urb_ctx *u = &ep->urb[i]; u->index = i; u->ep = ep; u->packets = urb_packs; - - if (fmt->fmt_type == UAC_FORMAT_TYPE_II) - u->packets++; /* for transfer delimiter */ u->buffer_size = maxsize * u->packets; u->urb = usb_alloc_urb(u->packets, GFP_KERNEL); if (!u->urb) diff --git a/sound/usb/line6/playback.c b/sound/usb/line6/playback.c index 8233c61e23f1..0da1998dd5ad 100644 --- a/sound/usb/line6/playback.c +++ b/sound/usb/line6/playback.c @@ -182,6 +182,13 @@ static int submit_audio_out_urb(struct snd_line6_pcm *line6pcm) fsize *= bytes_per_frame; + if (fsize > line6pcm->max_packet_size_out) { + dev_err(line6pcm->line6->ifcdev, + "playback packet too large: %d > %d\n", + fsize, line6pcm->max_packet_size_out); + return -EMSGSIZE; + } + fout->offset = urb_size; fout->length = fsize; urb_size += fsize; diff --git a/sound/usb/misc/ua101.c b/sound/usb/misc/ua101.c index 4d23ec97475d..c43f4e4a8d67 100644 --- a/sound/usb/misc/ua101.c +++ b/sound/usb/misc/ua101.c @@ -1022,6 +1022,15 @@ static int detect_usb_format(struct ua101 *ua) } ua->playback.usb_pipe = usb_sndisocpipe(ua->dev, usb_endpoint_num(epd)); ua->playback.max_packet_bytes = usb_endpoint_maxp(epd); + + if (ua->capture.max_packet_bytes / ua->capture.frame_bytes * + ua->playback.frame_bytes > ua->playback.max_packet_bytes) { + dev_err(&ua->dev->dev, + "playback packet size %u too small for %u capture frames\n", + ua->playback.max_packet_bytes, + ua->capture.max_packet_bytes / ua->capture.frame_bytes); + return -ENXIO; + } return 0; } diff --git a/sound/usb/pcm.c b/sound/usb/pcm.c index 08bf535ed163..5aadd1221f72 100644 --- a/sound/usb/pcm.c +++ b/sound/usb/pcm.c @@ -1505,7 +1505,8 @@ static int prepare_playback_urb(struct snd_usb_substream *subs, for (i = 0; i < ctx->packets; i++) { counts = snd_usb_endpoint_next_packet_size(ep, ctx, i, avail); - if (counts < 0) + if (counts < 0 || + (frames + counts) * stride > ctx->buffer_size) break; /* set up descriptor */ urb->iso_frame_desc[i].offset = frames * stride; diff --git a/sound/usb/quirks.c b/sound/usb/quirks.c index edfdf1625a67..598ca38be967 100644 --- a/sound/usb/quirks.c +++ b/sound/usb/quirks.c @@ -2271,8 +2271,6 @@ static const struct usb_audio_quirk_flags_table quirk_flags_table[] = { QUIRK_FLAG_SET_IFACE_FIRST), DEVICE_FLG(0x262a, 0x9302, /* ddHiFi TC44C */ QUIRK_FLAG_DSD_RAW), - DEVICE_FLG(0x2708, 0x0002, /* Audient iD14 */ - QUIRK_FLAG_IGNORE_CTL_ERROR), DEVICE_FLG(0x2772, 0x0502, /* Musical Fidelity M6s DAC */ 0), /* for avoiding QUIRK_FLAG_DSD_RAW with vendor match */ DEVICE_FLG(0x2912, 0x30c8, /* Audioengine D1 */ @@ -2340,6 +2338,8 @@ static const struct usb_audio_quirk_flags_table quirk_flags_table[] = { QUIRK_FLAG_DSD_RAW), VENDOR_FLG(0x2622, /* IAG Limited devices */ QUIRK_FLAG_DSD_RAW), + VENDOR_FLG(0x2708, /* Audient devices */ + QUIRK_FLAG_IGNORE_CTL_ERROR), VENDOR_FLG(0x2772, /* Musical Fidelity devices */ QUIRK_FLAG_DSD_RAW), VENDOR_FLG(0x278b, /* Rotel? */ diff --git a/sound/usb/usx2y/us122l.c b/sound/usb/usx2y/us122l.c index 36167300b8d0..2a107803af88 100644 --- a/sound/usb/usx2y/us122l.c +++ b/sound/usb/usx2y/us122l.c @@ -84,12 +84,9 @@ static int us144_create_usbmidi(struct snd_card *card) static void pt_info_set(struct usb_device *dev, u8 v) { - int ret; - - ret = usb_control_msg_send(dev, 0, 'I', - USB_DIR_OUT | USB_TYPE_VENDOR | USB_RECIP_DEVICE, - v, 0, NULL, 0, 1000, GFP_NOIO); - snd_printdd(KERN_DEBUG "%i\n", ret); + usb_control_msg_send(dev, 0, 'I', + USB_DIR_OUT | USB_TYPE_VENDOR | USB_RECIP_DEVICE, + v, 0, NULL, 0, 1000, GFP_NOIO); } static void usb_stream_hwdep_vm_open(struct vm_area_struct *area) @@ -97,7 +94,6 @@ static void usb_stream_hwdep_vm_open(struct vm_area_struct *area) struct us122l *us122l = area->vm_private_data; atomic_inc(&us122l->mmap_count); - snd_printdd(KERN_DEBUG "%i\n", atomic_read(&us122l->mmap_count)); } static vm_fault_t usb_stream_hwdep_vm_fault(struct vm_fault *vmf) @@ -141,7 +137,6 @@ static void usb_stream_hwdep_vm_close(struct vm_area_struct *area) struct us122l *us122l = area->vm_private_data; atomic_dec(&us122l->mmap_count); - snd_printdd(KERN_DEBUG "%i\n", atomic_read(&us122l->mmap_count)); } static const struct vm_operations_struct usb_stream_hwdep_vm_ops = { @@ -155,7 +150,6 @@ static int usb_stream_hwdep_open(struct snd_hwdep *hw, struct file *file) struct us122l *us122l = hw->private_data; struct usb_interface *iface; - snd_printdd(KERN_DEBUG "%p %p\n", hw, file); if (hw->used >= 2) return -EBUSY; @@ -176,8 +170,6 @@ static int usb_stream_hwdep_release(struct snd_hwdep *hw, struct file *file) struct us122l *us122l = hw->private_data; struct usb_interface *iface; - snd_printdd(KERN_DEBUG "%p %p\n", hw, file); - if (us122l->is_us144) { iface = usb_ifnum_to_if(us122l->dev, 0); usb_autopm_put_interface(iface); @@ -216,12 +208,10 @@ static int usb_stream_hwdep_mmap(struct snd_hwdep *hw, } vm_flags_clear(area, VM_MAYWRITE); } - snd_printdd(KERN_DEBUG "%lu %u\n", size, - read ? s->read_size : s->write_size); /* if userspace tries to mmap beyond end of our buffer, fail */ if (size > PAGE_ALIGN(read ? s->read_size : s->write_size)) { - snd_printk(KERN_WARNING "%lu > %u\n", size, - read ? s->read_size : s->write_size); + dev_warn(hw->card->dev, "%s: size %lu > %u\n", __func__, + size, read ? s->read_size : s->write_size); err = -EINVAL; goto out; } @@ -292,8 +282,8 @@ static int us122l_set_sample_rate(struct usb_device *dev, int rate) UAC_EP_CS_ATTR_SAMPLE_RATE << 8, ep, data, 3, 1000, GFP_NOIO); if (err) - snd_printk(KERN_ERR "%d: cannot set freq %d to ep 0x%x\n", - dev->devnum, rate, ep); + dev_err(&dev->dev, "%d: cannot set freq %d to ep 0x%x\n", + dev->devnum, rate, ep); return err; } @@ -329,13 +319,13 @@ static bool us122l_start(struct us122l *us122l, err = us122l_set_sample_rate(us122l->dev, rate); if (err < 0) { us122l_stop(us122l); - snd_printk(KERN_ERR "us122l_set_sample_rate error\n"); + dev_err(&us122l->dev->dev, "us122l_set_sample_rate error\n"); goto out; } err = usb_stream_start(&us122l->sk); if (err < 0) { us122l_stop(us122l); - snd_printk(KERN_ERR "%s error %i\n", __func__, err); + dev_err(&us122l->dev->dev, "%s error %i\n", __func__, err); goto out; } list_for_each(p, &us122l->midi_list) @@ -448,13 +438,13 @@ static bool us122l_create_card(struct snd_card *card) if (us122l->is_us144) { err = usb_set_interface(us122l->dev, 0, 1); if (err) { - snd_printk(KERN_ERR "usb_set_interface error\n"); + dev_err(card->dev, "usb_set_interface error\n"); return false; } } err = usb_set_interface(us122l->dev, 1, 1); if (err) { - snd_printk(KERN_ERR "usb_set_interface error\n"); + dev_err(card->dev, "usb_set_interface error\n"); return false; } @@ -469,7 +459,7 @@ static bool us122l_create_card(struct snd_card *card) else err = us122l_create_usbmidi(card); if (err < 0) { - snd_printk(KERN_ERR "us122l_create_usbmidi error %i\n", err); + dev_err(card->dev, "us122l_create_usbmidi error %i\n", err); goto stop; } err = usb_stream_hwdep_new(card); @@ -520,6 +510,7 @@ static int usx2y_create_card(struct usb_device *device, card->private_free = snd_us122l_free; US122L(card)->dev = device; mutex_init(&US122L(card)->mutex); + US122L(card)->sk.dev = device; init_waitqueue_head(&US122L(card)->sk.sleep); US122L(card)->is_us144 = flags & US122L_FLAG_US144; INIT_LIST_HEAD(&US122L(card)->midi_list); @@ -575,12 +566,10 @@ static int snd_us122l_probe(struct usb_interface *intf, if (id->driver_info & US122L_FLAG_US144 && device->speed == USB_SPEED_HIGH) { - snd_printk(KERN_ERR "disable ehci-hcd to run US-144\n"); + dev_err(&device->dev, "disable ehci-hcd to run US-144\n"); return -ENODEV; } - snd_printdd(KERN_DEBUG"%p:%i\n", - intf, intf->cur_altsetting->desc.bInterfaceNumber); if (intf->cur_altsetting->desc.bInterfaceNumber != 1) return 0; @@ -668,13 +657,13 @@ static int snd_us122l_resume(struct usb_interface *intf) if (us122l->is_us144) { err = usb_set_interface(us122l->dev, 0, 1); if (err) { - snd_printk(KERN_ERR "usb_set_interface error\n"); + dev_err(&us122l->dev->dev, "usb_set_interface error\n"); goto unlock; } } err = usb_set_interface(us122l->dev, 1, 1); if (err) { - snd_printk(KERN_ERR "usb_set_interface error\n"); + dev_err(&us122l->dev->dev, "usb_set_interface error\n"); goto unlock; } @@ -684,7 +673,7 @@ static int snd_us122l_resume(struct usb_interface *intf) err = us122l_set_sample_rate(us122l->dev, us122l->sk.s->cfg.sample_rate); if (err < 0) { - snd_printk(KERN_ERR "us122l_set_sample_rate error\n"); + dev_err(&us122l->dev->dev, "us122l_set_sample_rate error\n"); goto unlock; } err = usb_stream_start(&us122l->sk); diff --git a/sound/usb/usx2y/usX2Yhwdep.c b/sound/usb/usx2y/usX2Yhwdep.c index 34421b85838a..2cc0f90a23be 100644 --- a/sound/usb/usx2y/usX2Yhwdep.c +++ b/sound/usb/usx2y/usX2Yhwdep.c @@ -24,10 +24,6 @@ static vm_fault_t snd_us428ctls_vm_fault(struct vm_fault *vmf) struct page *page; void *vaddr; - snd_printdd("ENTER, start %lXh, pgoff %ld\n", - vmf->vma->vm_start, - vmf->pgoff); - offset = vmf->pgoff << PAGE_SHIFT; if (offset >= US428_SHAREDMEM_PAGES) return VM_FAULT_SIGBUS; @@ -36,9 +32,6 @@ static vm_fault_t snd_us428ctls_vm_fault(struct vm_fault *vmf) get_page(page); vmf->page = page; - snd_printdd("vaddr=%p made us428ctls_vm_fault() page %p\n", - vaddr, page); - return 0; } @@ -58,7 +51,8 @@ static int snd_us428ctls_mmap(struct snd_hwdep *hw, struct file *filp, struct vm /* if userspace tries to mmap beyond end of our buffer, fail */ if (size > US428_SHAREDMEM_PAGES) { - snd_printd("%lu > %lu\n", size, (unsigned long)US428_SHAREDMEM_PAGES); + dev_dbg(hw->card->dev, "%s: mmap size %lu > %lu\n", __func__, + size, (unsigned long)US428_SHAREDMEM_PAGES); return -EINVAL; } @@ -152,7 +146,6 @@ static int usx2y_create_usbmidi(struct snd_card *card) le16_to_cpu(dev->descriptor.idProduct) == USB_ID_US428 ? &quirk_2 : &quirk_1; - snd_printdd("%s\n", __func__); return snd_usbmidi_create(card, iface, &usx2y(card)->midi_list, quirk); } @@ -162,7 +155,8 @@ static int usx2y_create_alsa_devices(struct snd_card *card) err = usx2y_create_usbmidi(card); if (err < 0) { - snd_printk(KERN_ERR "%s: usx2y_create_usbmidi error %i\n", __func__, err); + dev_err(card->dev, "%s: usx2y_create_usbmidi error %i\n", + __func__, err); return err; } err = usx2y_audio_create(card); @@ -185,15 +179,13 @@ static int snd_usx2y_hwdep_dsp_load(struct snd_hwdep *hw, int lret, err; char *buf; - snd_printdd("dsp_load %s\n", dsp->name); - buf = memdup_user(dsp->image, dsp->length); if (IS_ERR(buf)) return PTR_ERR(buf); err = usb_set_interface(dev, 0, 1); if (err) - snd_printk(KERN_ERR "usb_set_interface error\n"); + dev_err(&dev->dev, "usb_set_interface error\n"); else err = usb_bulk_msg(dev, usb_sndbulkpipe(dev, 2), buf, dsp->length, &lret, 6000); kfree(buf); @@ -203,21 +195,20 @@ static int snd_usx2y_hwdep_dsp_load(struct snd_hwdep *hw, msleep(250); // give the device some time err = usx2y_async_seq04_init(priv); if (err) { - snd_printk(KERN_ERR "usx2y_async_seq04_init error\n"); + dev_err(&dev->dev, "usx2y_async_seq04_init error\n"); return err; } err = usx2y_in04_init(priv); if (err) { - snd_printk(KERN_ERR "usx2y_in04_init error\n"); + dev_err(&dev->dev, "usx2y_in04_init error\n"); return err; } err = usx2y_create_alsa_devices(hw->card); if (err) { - snd_printk(KERN_ERR "usx2y_create_alsa_devices error %i\n", err); + dev_err(&dev->dev, "usx2y_create_alsa_devices error %i\n", err); return err; } priv->chip_status |= USX2Y_STAT_CHIP_INIT; - snd_printdd("%s: alsa all started\n", hw->name); } return err; } diff --git a/sound/usb/usx2y/usb_stream.c b/sound/usb/usx2y/usb_stream.c index a4d32e8a1d36..66ea610aa433 100644 --- a/sound/usb/usx2y/usb_stream.c +++ b/sound/usb/usx2y/usb_stream.c @@ -34,14 +34,11 @@ static void playback_prep_freqn(struct usb_stream_kernel *sk, struct urb *urb) urb->iso_frame_desc[pack].length = l; lb += l; } - snd_printdd(KERN_DEBUG "%i\n", lb); check: urb->number_of_packets = pack; urb->transfer_buffer_length = lb; s->idle_outsize += lb - s->period_size; - snd_printdd(KERN_DEBUG "idle=%i ul=%i ps=%i\n", s->idle_outsize, - lb, s->period_size); } static int init_pipe_urbs(struct usb_stream_kernel *sk, @@ -191,14 +188,14 @@ struct usb_stream *usb_stream_new(struct usb_stream_kernel *sk, write_size = max_packsize * packets * USB_STREAM_URBDEPTH; if (read_size >= 256*PAGE_SIZE || write_size >= 256*PAGE_SIZE) { - snd_printk(KERN_WARNING "a size exceeds 128*PAGE_SIZE\n"); + dev_warn(&dev->dev, "%s: a size exceeds 128*PAGE_SIZE\n", __func__); goto out; } sk->s = alloc_pages_exact(read_size, GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN); if (!sk->s) { - pr_warn("us122l: couldn't allocate read buffer\n"); + dev_warn(&dev->dev, "us122l: couldn't allocate read buffer\n"); goto out; } sk->s->cfg.version = USB_STREAM_INTERFACE_VERSION; @@ -217,7 +214,7 @@ struct usb_stream *usb_stream_new(struct usb_stream_kernel *sk, sk->write_page = alloc_pages_exact(write_size, GFP_KERNEL | __GFP_ZERO | __GFP_NOWARN); if (!sk->write_page) { - pr_warn("us122l: couldn't allocate write buffer\n"); + dev_warn(&dev->dev, "us122l: couldn't allocate write buffer\n"); usb_stream_free(sk); return NULL; } @@ -246,7 +243,8 @@ static bool balance_check(struct usb_stream_kernel *sk, struct urb *urb) if (unlikely(urb->status)) { if (urb->status != -ESHUTDOWN && urb->status != -ENOENT) - snd_printk(KERN_WARNING "status=%i\n", urb->status); + dev_warn(&sk->dev->dev, "%s: status=%i\n", __func__, + urb->status); sk->iso_frame_balance = 0x7FFFFFFF; return false; } @@ -318,17 +316,18 @@ static int usb_stream_prepare_playback(struct usb_stream_kernel *sk, check_ok: s->sync_packet -= inurb->number_of_packets; if (unlikely(s->sync_packet < -2 || s->sync_packet > 0)) { - snd_printk(KERN_WARNING "invalid sync_packet = %i;" - " p=%i nop=%i %i %x %x %x > %x\n", - s->sync_packet, p, inurb->number_of_packets, - s->idle_outsize + lb + l, - s->idle_outsize, lb, l, - s->period_size); + dev_warn(&sk->dev->dev, + "%s: invalid sync_packet = %i; p=%i nop=%i %i %x %x %x > %x\n", + __func__, + s->sync_packet, p, inurb->number_of_packets, + s->idle_outsize + lb + l, + s->idle_outsize, lb, l, + s->period_size); return -1; } if (unlikely(lb % s->cfg.frame_size)) { - snd_printk(KERN_WARNING"invalid outsize = %i\n", - lb); + dev_warn(&sk->dev->dev, "%s: invalid outsize = %i\n", + __func__, lb); return -1; } s->idle_outsize += lb - s->period_size; @@ -337,7 +336,7 @@ static int usb_stream_prepare_playback(struct usb_stream_kernel *sk, if (s->idle_outsize <= 0) return 0; - snd_printk(KERN_WARNING "idle=%i\n", s->idle_outsize); + dev_warn(&sk->dev->dev, "%s: idle=%i\n", __func__, s->idle_outsize); return -1; } @@ -377,7 +376,7 @@ static int submit_urbs(struct usb_stream_kernel *sk, return 0; report_failure: - snd_printk(KERN_ERR "%i\n", err); + dev_err(&sk->dev->dev, "%s error: %i\n", __func__, err); return err; } @@ -424,8 +423,8 @@ static void loop_back(struct usb_stream *s) } if (iu == sk->completed_inurb) { if (l != s->period_size) - printk(KERN_DEBUG"%s:%i %i\n", __func__, __LINE__, - l/(int)s->cfg.frame_size); + dev_dbg(&sk->dev->dev, "%s:%i %i\n", __func__, __LINE__, + l/(int)s->cfg.frame_size); return; } @@ -460,8 +459,8 @@ static void stream_idle(struct usb_stream_kernel *sk, l = id[p].actual_length; if (unlikely(l == 0 || id[p].status)) { - snd_printk(KERN_WARNING "underrun, status=%u\n", - id[p].status); + dev_warn(&sk->dev->dev, "%s: underrun, status=%u\n", + __func__, id[p].status); goto err_out; } s->inpacket_head++; @@ -482,8 +481,8 @@ static void stream_idle(struct usb_stream_kernel *sk, } s->idle_insize += urb_size - s->period_size; if (s->idle_insize < 0) { - snd_printk(KERN_WARNING "%i\n", - (s->idle_insize)/(int)s->cfg.frame_size); + dev_warn(&sk->dev->dev, "%s error: %i\n", __func__, + (s->idle_insize)/(int)s->cfg.frame_size); goto err_out; } s->insize_done += urb_size; @@ -558,19 +557,15 @@ static void stream_start(struct usb_stream_kernel *sk, min_frames += frames_per_packet; diff = urb_size - (min_frames >> 8) * s->cfg.frame_size; - if (diff < max_diff) { - snd_printdd(KERN_DEBUG "%i %i %i %i\n", - s->insize_done, - urb_size / (int)s->cfg.frame_size, - inurb->number_of_packets, diff); + if (diff < max_diff) max_diff = diff; - } } s->idle_insize -= max_diff - max_diff_0; s->idle_insize += urb_size - s->period_size; if (s->idle_insize < 0) { - snd_printk(KERN_WARNING "%i %i %i\n", - s->idle_insize, urb_size, s->period_size); + dev_warn(&sk->dev->dev, "%s idle_insize: %i %i %i\n", + __func__, + s->idle_insize, urb_size, s->period_size); return; } else if (s->idle_insize == 0) { s->next_inpacket_split = @@ -620,7 +615,7 @@ static void i_capture_start(struct urb *urb) int empty = 0; if (urb->status) { - snd_printk(KERN_WARNING "status=%i\n", urb->status); + dev_warn(&sk->dev->dev, "%s: status=%i\n", __func__, urb->status); return; } @@ -630,7 +625,7 @@ static void i_capture_start(struct urb *urb) if (l < s->cfg.frame_size) { ++empty; if (s->state >= usb_stream_sync0) { - snd_printk(KERN_WARNING "%i\n", l); + dev_warn(&sk->dev->dev, "%s: length %i\n", __func__, l); return; } } @@ -642,8 +637,8 @@ static void i_capture_start(struct urb *urb) } #ifdef SHOW_EMPTY if (empty) { - printk(KERN_DEBUG"%s:%i: %i", __func__, __LINE__, - urb->iso_frame_desc[0].actual_length); + dev_dbg(&sk->dev->dev, "%s:%i: %i", __func__, __LINE__, + urb->iso_frame_desc[0].actual_length); for (pack = 1; pack < urb->number_of_packets; ++pack) { int l = urb->iso_frame_desc[pack].actual_length; @@ -710,38 +705,38 @@ int usb_stream_start(struct usb_stream_kernel *sk) } err = usb_submit_urb(inurb, GFP_ATOMIC); if (err < 0) { - snd_printk(KERN_ERR - "usb_submit_urb(sk->inurb[%i]) returned %i\n", - u, err); + dev_err(&sk->dev->dev, + "%s: usb_submit_urb(sk->inurb[%i]) returned %i\n", + __func__, u, err); return err; } err = usb_submit_urb(outurb, GFP_ATOMIC); if (err < 0) { - snd_printk(KERN_ERR - "usb_submit_urb(sk->outurb[%i]) returned %i\n", - u, err); + dev_err(&sk->dev->dev, + "%s: usb_submit_urb(sk->outurb[%i]) returned %i\n", + __func__, u, err); return err; } if (inurb->start_frame != outurb->start_frame) { - snd_printd(KERN_DEBUG - "u[%i] start_frames differ in:%u out:%u\n", - u, inurb->start_frame, outurb->start_frame); + dev_dbg(&sk->dev->dev, + "%s: u[%i] start_frames differ in:%u out:%u\n", + __func__, u, inurb->start_frame, outurb->start_frame); goto check_retry; } } - snd_printdd(KERN_DEBUG "%i %i\n", frame, iters); try = 0; check_retry: if (try) { usb_stream_stop(sk); if (try < 5) { msleep(1500); - snd_printd(KERN_DEBUG "goto dotry;\n"); + dev_dbg(&sk->dev->dev, "goto dotry;\n"); goto dotry; } - snd_printk(KERN_WARNING - "couldn't start all urbs on the same start_frame.\n"); + dev_warn(&sk->dev->dev, + "%s: couldn't start all urbs on the same start_frame.\n", + __func__); return -EFAULT; } @@ -755,7 +750,6 @@ int usb_stream_start(struct usb_stream_kernel *sk) int wait_ms = 3000; while (s->state != usb_stream_ready && wait_ms > 0) { - snd_printdd(KERN_DEBUG "%i\n", s->state); msleep(200); wait_ms -= 200; } diff --git a/sound/usb/usx2y/usb_stream.h b/sound/usb/usx2y/usb_stream.h index 73e57b341adc..2ed10add49f1 100644 --- a/sound/usb/usx2y/usb_stream.h +++ b/sound/usb/usx2y/usb_stream.h @@ -9,6 +9,7 @@ struct usb_stream_kernel { struct usb_stream *s; + struct usb_device *dev; void *write_page; diff --git a/sound/usb/usx2y/usbusx2y.c b/sound/usb/usx2y/usbusx2y.c index f6ab927490b4..1ccb01bf85f3 100644 --- a/sound/usb/usx2y/usbusx2y.c +++ b/sound/usb/usx2y/usbusx2y.c @@ -169,7 +169,7 @@ static void i_usx2y_out04_int(struct urb *urb) for (i = 0; i < 10 && usx2y->as04.urb[i] != urb; i++) ; - snd_printdd("%s urb %i status=%i\n", __func__, i, urb->status); + dev_dbg(&urb->dev->dev, "%s urb %i status=%i\n", __func__, i, urb->status); } #endif } @@ -185,11 +185,13 @@ static void i_usx2y_in04_int(struct urb *urb) usx2y->in04_int_calls++; if (urb->status) { - snd_printdd("Interrupt Pipe 4 came back with status=%i\n", urb->status); + dev_dbg(&urb->dev->dev, "Interrupt Pipe 4 came back with status=%i\n", urb->status); return; } - // printk("%i:0x%02X ", 8, (int)((unsigned char*)usx2y->in04_buf)[8]); Master volume shows 0 here if fader is at max during boot ?!? + if (urb->actual_length < USX2Y_IN04_SIZE) + goto resubmit; + if (us428ctls) { diff = -1; if (us428ctls->ctl_snapshot_last == -2) { @@ -252,8 +254,9 @@ static void i_usx2y_in04_int(struct urb *urb) } if (err) - snd_printk(KERN_ERR "in04_int() usb_submit_urb err=%i\n", err); + dev_err(&urb->dev->dev, "in04_int() usb_submit_urb err=%i\n", err); +resubmit: urb->dev = usx2y->dev; usb_submit_urb(urb, GFP_ATOMIC); } @@ -306,7 +309,7 @@ int usx2y_in04_init(struct usx2ydev *usx2y) goto error; } - usx2y->in04_buf = kmalloc(USX2Y_IN04_SIZE, GFP_KERNEL); + usx2y->in04_buf = kzalloc(USX2Y_IN04_SIZE, GFP_KERNEL); if (!usx2y->in04_buf) { err = -ENOMEM; goto error; diff --git a/sound/usb/usx2y/usbusx2yaudio.c b/sound/usb/usx2y/usbusx2yaudio.c index 98d0e8edc983..e8c7a9ba4e51 100644 --- a/sound/usb/usx2y/usbusx2yaudio.c +++ b/sound/usb/usx2y/usbusx2yaudio.c @@ -40,14 +40,15 @@ static int usx2y_urb_capt_retire(struct snd_usx2y_substream *subs) for (i = 0; i < nr_of_packs(); i++) { cp = (unsigned char *)urb->transfer_buffer + urb->iso_frame_desc[i].offset; if (urb->iso_frame_desc[i].status) { /* active? hmm, skip this */ - snd_printk(KERN_ERR - "active frame status %i. Most probably some hardware problem.\n", - urb->iso_frame_desc[i].status); + dev_err(&usx2y->dev->dev, + "%s: active frame status %i. Most probably some hardware problem.\n", + __func__, + urb->iso_frame_desc[i].status); return urb->iso_frame_desc[i].status; } len = urb->iso_frame_desc[i].actual_length / usx2y->stride; if (!len) { - snd_printd("0 == len ERROR!\n"); + dev_dbg(&usx2y->dev->dev, "%s: 0 == len ERROR!\n", __func__); continue; } @@ -101,7 +102,8 @@ static int usx2y_urb_play_prepare(struct snd_usx2y_substream *subs, counts = cap_urb->iso_frame_desc[pack].actual_length / usx2y->stride; count += counts; if (counts < 43 || counts > 50) { - snd_printk(KERN_ERR "should not be here with counts=%i\n", counts); + dev_err(&usx2y->dev->dev, "%s: should not be here with counts=%i\n", + __func__, counts); return -EPIPE; } /* set up descriptor */ @@ -169,7 +171,8 @@ static int usx2y_urb_submit(struct snd_usx2y_substream *subs, struct urb *urb, i urb->dev = subs->usx2y->dev; /* we need to set this at each time */ err = usb_submit_urb(urb, GFP_ATOMIC); if (err < 0) { - snd_printk(KERN_ERR "usb_submit_urb() returned %i\n", err); + dev_err(&urb->dev->dev, "%s: usb_submit_urb() returned %i\n", + __func__, err); return err; } return 0; @@ -237,7 +240,8 @@ static void usx2y_clients_stop(struct usx2ydev *usx2y) for (s = 0; s < 4; s++) { subs = usx2y->subs[s]; if (subs) { - snd_printdd("%i %p state=%i\n", s, subs, atomic_read(&subs->state)); + dev_dbg(&usx2y->dev->dev, "%s: %i %p state=%i\n", + __func__, s, subs, atomic_read(&subs->state)); atomic_set(&subs->state, STATE_STOPPED); } } @@ -249,8 +253,9 @@ static void usx2y_clients_stop(struct usx2ydev *usx2y) for (u = 0; u < NRURBS; u++) { urb = subs->urb[u]; if (urb) - snd_printdd("%i status=%i start_frame=%i\n", - u, urb->status, urb->start_frame); + dev_dbg(&usx2y->dev->dev, + "%s: %i status=%i start_frame=%i\n", + __func__, u, urb->status, urb->start_frame); } } } @@ -261,7 +266,8 @@ static void usx2y_clients_stop(struct usx2ydev *usx2y) static void usx2y_error_urb_status(struct usx2ydev *usx2y, struct snd_usx2y_substream *subs, struct urb *urb) { - snd_printk(KERN_ERR "ep=%i stalled with status=%i\n", subs->endpoint, urb->status); + dev_err(&usx2y->dev->dev, "%s: ep=%i stalled with status=%i\n", + __func__, subs->endpoint, urb->status); urb->status = 0; usx2y_clients_stop(usx2y); } @@ -273,10 +279,12 @@ static void i_usx2y_urb_complete(struct urb *urb) struct snd_usx2y_substream *capsubs, *playbacksubs; if (unlikely(atomic_read(&subs->state) < STATE_PREPARED)) { - snd_printdd("hcd_frame=%i ep=%i%s status=%i start_frame=%i\n", - usb_get_current_frame_number(usx2y->dev), - subs->endpoint, usb_pipein(urb->pipe) ? "in" : "out", - urb->status, urb->start_frame); + dev_dbg(&usx2y->dev->dev, + "%s: hcd_frame=%i ep=%i%s status=%i start_frame=%i\n", + __func__, + usb_get_current_frame_number(usx2y->dev), + subs->endpoint, usb_pipein(urb->pipe) ? "in" : "out", + urb->status, urb->start_frame); return; } if (unlikely(urb->status)) { @@ -296,7 +304,6 @@ static void i_usx2y_urb_complete(struct urb *urb) if (!usx2y_usbframe_complete(capsubs, playbacksubs, urb->start_frame)) { usx2y->wait_iso_frame += nr_of_packs(); } else { - snd_printdd("\n"); usx2y_clients_stop(usx2y); } } @@ -346,8 +353,9 @@ static void i_usx2y_subs_startup(struct urb *urb) static void usx2y_subs_prepare(struct snd_usx2y_substream *subs) { - snd_printdd("usx2y_substream_prepare(%p) ep=%i urb0=%p urb1=%p\n", - subs, subs->endpoint, subs->urb[0], subs->urb[1]); + dev_dbg(&subs->usx2y->dev->dev, + "%s(%p) ep=%i urb0=%p urb1=%p\n", + __func__, subs, subs->endpoint, subs->urb[0], subs->urb[1]); /* reset the pointer */ subs->hwptr = 0; subs->hwptr_done = 0; @@ -372,7 +380,7 @@ static void usx2y_urbs_release(struct snd_usx2y_substream *subs) { int i; - snd_printdd("%s %i\n", __func__, subs->endpoint); + dev_dbg(&subs->usx2y->dev->dev, "%s %i\n", __func__, subs->endpoint); for (i = 0; i < NRURBS; i++) usx2y_urb_release(subs->urb + i, subs != subs->usx2y->subs[SNDRV_PCM_STREAM_PLAYBACK]); @@ -478,7 +486,8 @@ static int usx2y_urbs_start(struct snd_usx2y_substream *subs) urb->transfer_buffer_length = subs->maxpacksize * nr_of_packs(); err = usb_submit_urb(urb, GFP_ATOMIC); if (err < 0) { - snd_printk(KERN_ERR "cannot submit datapipe for urb %d, err = %d\n", i, err); + dev_err(&urb->dev->dev, "%s: cannot submit datapipe for urb %d, err = %d\n", + __func__, i, err); err = -EPIPE; goto cleanup; } else { @@ -523,17 +532,16 @@ static int snd_usx2y_pcm_trigger(struct snd_pcm_substream *substream, int cmd) switch (cmd) { case SNDRV_PCM_TRIGGER_START: - snd_printdd("%s(START)\n", __func__); + dev_dbg(&subs->usx2y->dev->dev, "%s(START)\n", __func__); if (atomic_read(&subs->state) == STATE_PREPARED && atomic_read(&subs->usx2y->subs[SNDRV_PCM_STREAM_CAPTURE]->state) >= STATE_PREPARED) { atomic_set(&subs->state, STATE_PRERUNNING); } else { - snd_printdd("\n"); return -EPIPE; } break; case SNDRV_PCM_TRIGGER_STOP: - snd_printdd("%s(STOP)\n", __func__); + dev_dbg(&subs->usx2y->dev->dev, "%s(STOP)\n", __func__); if (atomic_read(&subs->state) >= STATE_PRERUNNING) atomic_set(&subs->state, STATE_PREPARED); break; @@ -634,7 +642,8 @@ static void i_usx2y_04int(struct urb *urb) struct usx2ydev *usx2y = urb->context; if (urb->status) - snd_printk(KERN_ERR "snd_usx2y_04int() urb->status=%i\n", urb->status); + dev_err(&urb->dev->dev, "%s() urb->status=%i\n", + __func__, urb->status); if (!--usx2y->us04->len) wake_up(&usx2y->in04_wait_queue); } @@ -724,7 +733,8 @@ static int usx2y_format_set(struct usx2ydev *usx2y, snd_pcm_format_t format) usb_kill_urb(usx2y->in04_urb); err = usb_set_interface(usx2y->dev, 0, alternate); if (err) { - snd_printk(KERN_ERR "usb_set_interface error\n"); + dev_err(&usx2y->dev->dev, "%s: usb_set_interface error\n", + __func__); return err; } usx2y->in04_urb->dev = usx2y->dev; @@ -751,7 +761,7 @@ static int snd_usx2y_pcm_hw_params(struct snd_pcm_substream *substream, int i; mutex_lock(&usx2y(card)->pcm_mutex); - snd_printdd("snd_usx2y_hw_params(%p, %p)\n", substream, hw_params); + dev_dbg(&dev->dev->dev, "%s(%p, %p)\n", __func__, substream, hw_params); /* all pcm substreams off one usx2y have to operate at the same * rate & format */ @@ -787,7 +797,7 @@ static int snd_usx2y_pcm_hw_free(struct snd_pcm_substream *substream) struct snd_usx2y_substream *cap_subs, *playback_subs; mutex_lock(&subs->usx2y->pcm_mutex); - snd_printdd("snd_usx2y_hw_free(%p)\n", substream); + dev_dbg(&subs->usx2y->dev->dev, "%s(%p)\n", __func__, substream); if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) { cap_subs = subs->usx2y->subs[SNDRV_PCM_STREAM_CAPTURE]; @@ -823,7 +833,7 @@ static int snd_usx2y_pcm_prepare(struct snd_pcm_substream *substream) struct snd_usx2y_substream *capsubs = subs->usx2y->subs[SNDRV_PCM_STREAM_CAPTURE]; int err = 0; - snd_printdd("%s(%p)\n", __func__, substream); + dev_dbg(&usx2y->dev->dev, "%s(%p)\n", __func__, substream); mutex_lock(&usx2y->pcm_mutex); usx2y_subs_prepare(subs); @@ -840,7 +850,8 @@ static int snd_usx2y_pcm_prepare(struct snd_pcm_substream *substream) if (err < 0) goto up_prepare_mutex; } - snd_printdd("starting capture pipe for %s\n", subs == capsubs ? "self" : "playpipe"); + dev_dbg(&usx2y->dev->dev, "%s: starting capture pipe for %s\n", + __func__, subs == capsubs ? "self" : "playpipe"); err = usx2y_urbs_start(capsubs); if (err < 0) goto up_prepare_mutex; diff --git a/sound/usb/usx2y/usx2yhwdeppcm.c b/sound/usb/usx2y/usx2yhwdeppcm.c index 54290caa9fad..0b8dd9e3f984 100644 --- a/sound/usb/usx2y/usx2yhwdeppcm.c +++ b/sound/usb/usx2y/usx2yhwdeppcm.c @@ -59,13 +59,13 @@ static int usx2y_usbpcm_urb_capt_retire(struct snd_usx2y_substream *subs) if (head >= ARRAY_SIZE(usx2y->hwdep_pcm_shm->captured_iso)) head = 0; usx2y->hwdep_pcm_shm->capture_iso_start = head; - snd_printdd("cap start %i\n", head); + dev_dbg(&usx2y->dev->dev, "cap start %i\n", head); } for (i = 0; i < nr_of_packs(); i++) { if (urb->iso_frame_desc[i].status) { /* active? hmm, skip this */ - snd_printk(KERN_ERR - "active frame status %i. Most probably some hardware problem.\n", - urb->iso_frame_desc[i].status); + dev_err(&usx2y->dev->dev, + "active frame status %i. Most probably some hardware problem.\n", + urb->iso_frame_desc[i].status); return urb->iso_frame_desc[i].status; } lens += urb->iso_frame_desc[i].actual_length / usx2y->stride; @@ -120,7 +120,7 @@ static int usx2y_hwdep_urb_play_prepare(struct snd_usx2y_substream *subs, /* calculate the size of a packet */ counts = shm->captured_iso[shm->playback_iso_head].length / usx2y->stride; if (counts < 43 || counts > 50) { - snd_printk(KERN_ERR "should not be here with counts=%i\n", counts); + dev_err(&usx2y->dev->dev, "should not be here with counts=%i\n", counts); return -EPIPE; } /* set up descriptor */ @@ -234,10 +234,11 @@ static void i_usx2y_usbpcm_urb_complete(struct urb *urb) struct snd_usx2y_substream *capsubs, *capsubs2, *playbacksubs; if (unlikely(atomic_read(&subs->state) < STATE_PREPARED)) { - snd_printdd("hcd_frame=%i ep=%i%s status=%i start_frame=%i\n", - usb_get_current_frame_number(usx2y->dev), - subs->endpoint, usb_pipein(urb->pipe) ? "in" : "out", - urb->status, urb->start_frame); + dev_dbg(&usx2y->dev->dev, + "hcd_frame=%i ep=%i%s status=%i start_frame=%i\n", + usb_get_current_frame_number(usx2y->dev), + subs->endpoint, usb_pipein(urb->pipe) ? "in" : "out", + urb->status, urb->start_frame); return; } if (unlikely(urb->status)) { @@ -255,7 +256,6 @@ static void i_usx2y_usbpcm_urb_complete(struct urb *urb) if (!usx2y_usbpcm_usbframe_complete(capsubs, capsubs2, playbacksubs, urb->start_frame)) { usx2y->wait_iso_frame += nr_of_packs(); } else { - snd_printdd("\n"); usx2y_clients_stop(usx2y); } } @@ -275,7 +275,8 @@ static void usx2y_usbpcm_urbs_release(struct snd_usx2y_substream *subs) { int i; - snd_printdd("snd_usx2y_urbs_release() %i\n", subs->endpoint); + dev_dbg(&subs->usx2y->dev->dev, + "snd_usx2y_urbs_release() %i\n", subs->endpoint); for (i = 0; i < NRURBS; i++) usx2y_hwdep_urb_release(subs->urb + i); } @@ -365,7 +366,7 @@ static int snd_usx2y_usbpcm_hw_free(struct snd_pcm_substream *substream) struct snd_usx2y_substream *cap_subs2; mutex_lock(&subs->usx2y->pcm_mutex); - snd_printdd("%s(%p)\n", __func__, substream); + dev_dbg(&subs->usx2y->dev->dev, "%s(%p)\n", __func__, substream); cap_subs2 = subs->usx2y->subs[SNDRV_PCM_STREAM_CAPTURE + 2]; if (substream->stream == SNDRV_PCM_STREAM_PLAYBACK) { @@ -456,11 +457,12 @@ static int usx2y_usbpcm_urbs_start(struct snd_usx2y_substream *subs) urb->transfer_buffer_length = subs->maxpacksize * nr_of_packs(); err = usb_submit_urb(urb, GFP_KERNEL); if (err < 0) { - snd_printk(KERN_ERR "cannot usb_submit_urb() for urb %d, err = %d\n", u, err); + dev_err(&urb->dev->dev, + "cannot usb_submit_urb() for urb %d, err = %d\n", + u, err); err = -EPIPE; goto cleanup; } else { - snd_printdd("%i\n", urb->start_frame); if (!u) usx2y->wait_iso_frame = urb->start_frame; } @@ -500,7 +502,7 @@ static int snd_usx2y_usbpcm_prepare(struct snd_pcm_substream *substream) struct snd_usx2y_substream *capsubs = subs->usx2y->subs[SNDRV_PCM_STREAM_CAPTURE]; int err = 0; - snd_printdd("snd_usx2y_pcm_prepare(%p)\n", substream); + dev_dbg(&usx2y->dev->dev, "snd_usx2y_pcm_prepare(%p)\n", substream); mutex_lock(&usx2y->pcm_mutex); @@ -528,8 +530,9 @@ static int snd_usx2y_usbpcm_prepare(struct snd_pcm_substream *substream) if (err < 0) goto up_prepare_mutex; } - snd_printdd("starting capture pipe for %s\n", subs == capsubs ? - "self" : "playpipe"); + dev_dbg(&usx2y->dev->dev, + "starting capture pipe for %s\n", subs == capsubs ? + "self" : "playpipe"); err = usx2y_usbpcm_urbs_start(capsubs); if (err < 0) goto up_prepare_mutex; @@ -540,9 +543,10 @@ static int snd_usx2y_usbpcm_prepare(struct snd_pcm_substream *substream) if (atomic_read(&subs->state) < STATE_PREPARED) { while (usx2y_iso_frames_per_buffer(runtime, usx2y) > usx2y->hwdep_pcm_shm->captured_iso_frames) { - snd_printdd("Wait: iso_frames_per_buffer=%i,captured_iso_frames=%i\n", - usx2y_iso_frames_per_buffer(runtime, usx2y), - usx2y->hwdep_pcm_shm->captured_iso_frames); + dev_dbg(&usx2y->dev->dev, + "Wait: iso_frames_per_buffer=%i,captured_iso_frames=%i\n", + usx2y_iso_frames_per_buffer(runtime, usx2y), + usx2y->hwdep_pcm_shm->captured_iso_frames); if (msleep_interruptible(10)) { err = -ERESTARTSYS; goto up_prepare_mutex; @@ -552,9 +556,10 @@ static int snd_usx2y_usbpcm_prepare(struct snd_pcm_substream *substream) if (err < 0) goto up_prepare_mutex; } - snd_printdd("Ready: iso_frames_per_buffer=%i,captured_iso_frames=%i\n", - usx2y_iso_frames_per_buffer(runtime, usx2y), - usx2y->hwdep_pcm_shm->captured_iso_frames); + dev_dbg(&usx2y->dev->dev, + "Ready: iso_frames_per_buffer=%i,captured_iso_frames=%i\n", + usx2y_iso_frames_per_buffer(runtime, usx2y), + usx2y->hwdep_pcm_shm->captured_iso_frames); } else { usx2y->hwdep_pcm_shm->capture_iso_start = -1; } @@ -700,7 +705,8 @@ static int snd_usx2y_hwdep_pcm_mmap(struct snd_hwdep *hw, struct file *filp, str /* if userspace tries to mmap beyond end of our buffer, fail */ if (size > USX2Y_HWDEP_PCM_PAGES) { - snd_printd("%lu > %lu\n", size, (unsigned long)USX2Y_HWDEP_PCM_PAGES); + dev_dbg(hw->card->dev, "%s: %lu > %lu\n", __func__, + size, (unsigned long)USX2Y_HWDEP_PCM_PAGES); return -EINVAL; } diff --git a/tools/bpf/bpftool/feature.c b/tools/bpf/bpftool/feature.c index edda4fc2c4d0..e36ace0d4c00 100644 --- a/tools/bpf/bpftool/feature.c +++ b/tools/bpf/bpftool/feature.c @@ -668,7 +668,8 @@ probe_helper_ifindex(enum bpf_func_id id, enum bpf_prog_type prog_type, probe_prog_load_ifindex(prog_type, insns, ARRAY_SIZE(insns), buf, sizeof(buf), ifindex); - res = !grep(buf, "invalid func ") && !grep(buf, "unknown func "); + res = !grep(buf, "invalid func ") && !grep(buf, "unknown func ") && + !grep(buf, "program of this type cannot use helper "); switch (get_vendor_id(ifindex)) { case 0x19ee: /* Netronome specific */ diff --git a/tools/lib/bpf/libbpf_probes.c b/tools/lib/bpf/libbpf_probes.c index 9c4db90b92b6..8cb1691017d3 100644 --- a/tools/lib/bpf/libbpf_probes.c +++ b/tools/lib/bpf/libbpf_probes.c @@ -435,7 +435,8 @@ int libbpf_probe_bpf_helper(enum bpf_prog_type prog_type, enum bpf_func_id helpe /* If BPF verifier doesn't recognize BPF helper ID (enum bpf_func_id) * at all, it will emit something like "invalid func unknown#181". * If BPF verifier recognizes BPF helper but it's not supported for - * given BPF program type, it will emit "unknown func bpf_sys_bpf#166". + * given BPF program type, it will emit "unknown func bpf_sys_bpf#166" + * or "program of this type cannot use helper bpf_sys_bpf#166". * In both cases, provided combination of BPF program type and BPF * helper is not supported by the kernel. * In all other cases, probe_prog_load() above will either succeed (e.g., @@ -444,7 +445,8 @@ int libbpf_probe_bpf_helper(enum bpf_prog_type prog_type, enum bpf_func_id helpe * that), or we'll get some more specific BPF verifier error about * some unsatisfied conditions. */ - if (ret == 0 && (strstr(buf, "invalid func ") || strstr(buf, "unknown func "))) + if (ret == 0 && (strstr(buf, "invalid func ") || strstr(buf, "unknown func ") || + strstr(buf, "program of this type cannot use helper "))) return 0; return 1; /* assume supported */ } diff --git a/tools/net/ynl/generated/netdev-user.c b/tools/net/ynl/generated/netdev-user.c index 68b408ca0f7f..b5ffe8cd1144 100644 --- a/tools/net/ynl/generated/netdev-user.c +++ b/tools/net/ynl/generated/netdev-user.c @@ -45,12 +45,26 @@ const char *netdev_xdp_act_str(enum netdev_xdp_act value) return netdev_xdp_act_strmap[value]; } +static const char * const netdev_xdp_rx_metadata_strmap[] = { + [0] = "timestamp", + [1] = "hash", +}; + +const char *netdev_xdp_rx_metadata_str(enum netdev_xdp_rx_metadata value) +{ + value = ffs(value) - 1; + if (value < 0 || value >= (int)MNL_ARRAY_SIZE(netdev_xdp_rx_metadata_strmap)) + return NULL; + return netdev_xdp_rx_metadata_strmap[value]; +} + /* Policies */ struct ynl_policy_attr netdev_dev_policy[NETDEV_A_DEV_MAX + 1] = { [NETDEV_A_DEV_IFINDEX] = { .name = "ifindex", .type = YNL_PT_U32, }, [NETDEV_A_DEV_PAD] = { .name = "pad", .type = YNL_PT_IGNORE, }, [NETDEV_A_DEV_XDP_FEATURES] = { .name = "xdp-features", .type = YNL_PT_U64, }, [NETDEV_A_DEV_XDP_ZC_MAX_SEGS] = { .name = "xdp-zc-max-segs", .type = YNL_PT_U32, }, + [NETDEV_A_DEV_XDP_RX_METADATA_FEATURES] = { .name = "xdp-rx-metadata-features", .type = YNL_PT_U64, }, }; struct ynl_policy_nest netdev_dev_nest = { @@ -97,6 +111,11 @@ int netdev_dev_get_rsp_parse(const struct nlmsghdr *nlh, void *data) return MNL_CB_ERROR; dst->_present.xdp_zc_max_segs = 1; dst->xdp_zc_max_segs = mnl_attr_get_u32(attr); + } else if (type == NETDEV_A_DEV_XDP_RX_METADATA_FEATURES) { + if (ynl_attr_validate(yarg, attr)) + return MNL_CB_ERROR; + dst->_present.xdp_rx_metadata_features = 1; + dst->xdp_rx_metadata_features = mnl_attr_get_u64(attr); } } diff --git a/tools/net/ynl/generated/netdev-user.h b/tools/net/ynl/generated/netdev-user.h index 0952d3261f4d..b4351ff34595 100644 --- a/tools/net/ynl/generated/netdev-user.h +++ b/tools/net/ynl/generated/netdev-user.h @@ -18,6 +18,7 @@ extern const struct ynl_family ynl_netdev_family; /* Enums */ const char *netdev_op_str(int op); const char *netdev_xdp_act_str(enum netdev_xdp_act value); +const char *netdev_xdp_rx_metadata_str(enum netdev_xdp_rx_metadata value); /* Common nested types */ /* ============== NETDEV_CMD_DEV_GET ============== */ @@ -48,11 +49,13 @@ struct netdev_dev_get_rsp { __u32 ifindex:1; __u32 xdp_features:1; __u32 xdp_zc_max_segs:1; + __u32 xdp_rx_metadata_features:1; } _present; __u32 ifindex; __u64 xdp_features; __u32 xdp_zc_max_segs; + __u64 xdp_rx_metadata_features; }; void netdev_dev_get_rsp_free(struct netdev_dev_get_rsp *rsp); diff --git a/tools/net/ynl/samples/Makefile b/tools/net/ynl/samples/Makefile index f2db8bb78309..32abbc0af39e 100644 --- a/tools/net/ynl/samples/Makefile +++ b/tools/net/ynl/samples/Makefile @@ -4,7 +4,7 @@ include ../Makefile.deps CC=gcc CFLAGS=-std=gnu11 -O2 -W -Wall -Wextra -Wno-unused-parameter -Wshadow \ - -I../lib/ -I../generated/ -idirafter $(UAPI_PATH) + -I../../../include/uapi -I../lib/ -I../generated/ -idirafter $(UAPI_PATH) ifeq ("$(DEBUG)","1") CFLAGS += -g -fsanitize=address -fsanitize=leak -static-libasan endif diff --git a/tools/net/ynl/samples/netdev.c b/tools/net/ynl/samples/netdev.c index 06433400dddd..b828225daad0 100644 --- a/tools/net/ynl/samples/netdev.c +++ b/tools/net/ynl/samples/netdev.c @@ -32,12 +32,18 @@ static void netdev_print_device(struct netdev_dev_get_rsp *d, unsigned int op) if (!d->_present.xdp_features) return; - printf("%llx:", d->xdp_features); + printf("xdp-features (%llx):", d->xdp_features); for (int i = 0; d->xdp_features > 1U << i; i++) { if (d->xdp_features & (1U << i)) printf(" %s", netdev_xdp_act_str(1 << i)); } + printf(" xdp-rx-metadata-features (%llx):", d->xdp_rx_metadata_features); + for (int i = 0; d->xdp_rx_metadata_features > 1U << i; i++) { + if (d->xdp_rx_metadata_features & (1U << i)) + printf(" %s", netdev_xdp_rx_metadata_str(1 << i)); + } + printf(" xdp-zc-max-segs=%u", d->xdp_zc_max_segs); name = netdev_op_str(op); diff --git a/tools/objtool/check.c b/tools/objtool/check.c index 2c7b09c8415f..c54d531f73d9 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -1236,7 +1236,7 @@ static const char *uaccess_safe_builtin[] = { "copy_mc_enhanced_fast_string", "rep_stos_alternative", "rep_movs_alternative", - "__copy_user_nocache", + "copy_to_nontemporal", NULL }; diff --git a/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c b/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c index d0d9a0241545..1202b33627f2 100644 --- a/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c +++ b/tools/testing/selftests/bpf/prog_tests/bpf_tcp_ca.c @@ -319,7 +319,7 @@ static void test_rel_setsockopt(void) struct bpf_dctcp_release *rel_skel; libbpf_print_fn_t old_print_fn; - err_str = "unknown func bpf_setsockopt"; + err_str = "program of this type cannot use helper bpf_setsockopt"; found = false; old_print_fn = libbpf_set_print(libbpf_debug_print); diff --git a/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c b/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c index 92267abb462f..be540970fcc1 100644 --- a/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c +++ b/tools/testing/selftests/bpf/progs/bpf_iter_tcp4.c @@ -103,9 +103,9 @@ static int dump_tcp_sock(struct seq_file *seq, struct tcp_sock *tp, } else if (icsk->icsk_pending == ICSK_TIME_PROBE0) { timer_active = 4; timer_expires = icsk->icsk_timeout; - } else if (timer_pending(&sp->sk_timer)) { + } else if (timer_pending(&icsk->icsk_keepalive_timer)) { timer_active = 2; - timer_expires = sp->sk_timer.expires; + timer_expires = icsk->icsk_keepalive_timer.expires; } else { timer_active = 0; timer_expires = bpf_jiffies64(); diff --git a/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c b/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c index 943f7bba180e..9f72398fc3cc 100644 --- a/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c +++ b/tools/testing/selftests/bpf/progs/bpf_iter_tcp6.c @@ -103,9 +103,9 @@ static int dump_tcp6_sock(struct seq_file *seq, struct tcp6_sock *tp, } else if (icsk->icsk_pending == ICSK_TIME_PROBE0) { timer_active = 4; timer_expires = icsk->icsk_timeout; - } else if (timer_pending(&sp->sk_timer)) { + } else if (timer_pending(&icsk->icsk_keepalive_timer)) { timer_active = 2; - timer_expires = sp->sk_timer.expires; + timer_expires = icsk->icsk_keepalive_timer.expires; } else { timer_active = 0; timer_expires = bpf_jiffies64(); diff --git a/tools/testing/selftests/bpf/progs/verifier_helper_restricted.c b/tools/testing/selftests/bpf/progs/verifier_helper_restricted.c index 0ede0ccd090c..059aa716e3d0 100644 --- a/tools/testing/selftests/bpf/progs/verifier_helper_restricted.c +++ b/tools/testing/selftests/bpf/progs/verifier_helper_restricted.c @@ -30,7 +30,7 @@ struct { SEC("kprobe") __description("bpf_ktime_get_coarse_ns is forbidden in BPF_PROG_TYPE_KPROBE") -__failure __msg("unknown func bpf_ktime_get_coarse_ns") +__failure __msg("program of this type cannot use helper bpf_ktime_get_coarse_ns") __naked void in_bpf_prog_type_kprobe_1(void) { asm volatile (" \ @@ -44,7 +44,7 @@ __naked void in_bpf_prog_type_kprobe_1(void) SEC("tracepoint") __description("bpf_ktime_get_coarse_ns is forbidden in BPF_PROG_TYPE_TRACEPOINT") -__failure __msg("unknown func bpf_ktime_get_coarse_ns") +__failure __msg("program of this type cannot use helper bpf_ktime_get_coarse_ns") __naked void in_bpf_prog_type_tracepoint_1(void) { asm volatile (" \ @@ -58,7 +58,7 @@ __naked void in_bpf_prog_type_tracepoint_1(void) SEC("perf_event") __description("bpf_ktime_get_coarse_ns is forbidden in BPF_PROG_TYPE_PERF_EVENT") -__failure __msg("unknown func bpf_ktime_get_coarse_ns") +__failure __msg("program of this type cannot use helper bpf_ktime_get_coarse_ns") __naked void bpf_prog_type_perf_event_1(void) { asm volatile (" \ @@ -72,7 +72,7 @@ __naked void bpf_prog_type_perf_event_1(void) SEC("raw_tracepoint") __description("bpf_ktime_get_coarse_ns is forbidden in BPF_PROG_TYPE_RAW_TRACEPOINT") -__failure __msg("unknown func bpf_ktime_get_coarse_ns") +__failure __msg("program of this type cannot use helper bpf_ktime_get_coarse_ns") __naked void bpf_prog_type_raw_tracepoint_1(void) { asm volatile (" \ diff --git a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c index a188e26f04da..54d9861194e6 100644 --- a/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c +++ b/tools/testing/selftests/bpf/progs/verifier_subprog_precision.c @@ -150,9 +150,9 @@ __msg("17: (b7) r0 = 0") __msg("18: (95) exit") __msg("returning from callee:") __msg("to caller at 9:") -__msg("frame 0: propagating r1,r4") +__msg("frame 0: propagating r1,r3,r4") __msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1") -__msg("mark_precise: frame0: regs=r1,r4 stack= before 18: (95) exit") +__msg("mark_precise: frame0: regs=r1,r3,r4 stack= before 18: (95) exit") __msg("from 18 to 9: safe") __naked int callback_result_precise(void) { @@ -282,9 +282,9 @@ __msg("to caller at 9:") /* r1, r4 are always precise for bpf_loop(), * r6 was marked before backtracking to callback body. */ -__msg("frame 0: propagating r1,r4,r6") +__msg("frame 0: propagating r1,r3,r4,r6") __msg("mark_precise: frame0: last_idx 9 first_idx 9 subseq_idx -1") -__msg("mark_precise: frame0: regs=r1,r4,r6 stack= before 16: (95) exit") +__msg("mark_precise: frame0: regs=r1,r3,r4,r6 stack= before 16: (95) exit") __msg("mark_precise: frame1: regs= stack= before 15: (b7) r0 = 0") __msg("mark_precise: frame1: regs= stack= before 9: (85) call bpf_loop") __msg("mark_precise: frame0: parent state regs= stack=:") @@ -440,9 +440,9 @@ __msg("to caller at 10:") /* r1, r4 are always precise for bpf_loop(), * fp-8 was marked before backtracking to callback body. */ -__msg("frame 0: propagating r1,r4,fp-8") +__msg("frame 0: propagating r1,r3,r4,fp-8") __msg("mark_precise: frame0: last_idx 10 first_idx 10 subseq_idx -1") -__msg("mark_precise: frame0: regs=r1,r4 stack=-8 before 18: (95) exit") +__msg("mark_precise: frame0: regs=r1,r3,r4 stack=-8 before 18: (95) exit") __msg("mark_precise: frame1: regs= stack= before 17: (b7) r0 = 0") __msg("mark_precise: frame1: regs= stack= before 10: (85) call bpf_loop#181") __msg("mark_precise: frame0: parent state regs= stack=:") diff --git a/tools/testing/selftests/mm/memfd_secret.c b/tools/testing/selftests/mm/memfd_secret.c index 9b298f6a04b3..348cc976726b 100644 --- a/tools/testing/selftests/mm/memfd_secret.c +++ b/tools/testing/selftests/mm/memfd_secret.c @@ -56,33 +56,6 @@ static void test_file_apis(int fd) pass("file IO is blocked as expected\n"); } -static void test_mlock_limit(int fd) -{ - size_t len; - char *mem; - - len = mlock_limit_cur; - if (len % page_size != 0) - len = (len/page_size) * page_size; - - mem = mmap(NULL, len, prot, mode, fd, 0); - if (mem == MAP_FAILED) { - fail("unable to mmap secret memory\n"); - return; - } - munmap(mem, len); - - len = mlock_limit_max * 2; - mem = mmap(NULL, len, prot, mode, fd, 0); - if (mem != MAP_FAILED) { - fail("unexpected mlock limit violation\n"); - munmap(mem, len); - return; - } - - pass("mlock limit is respected\n"); -} - static void try_process_vm_read(int fd, int pipefd[2]) { struct iovec liov, riov; @@ -258,7 +231,7 @@ static void prepare(void) strerror(errno)); } -#define NUM_TESTS 4 +#define NUM_TESTS 3 int main(int argc, char *argv[]) { @@ -278,7 +251,6 @@ int main(int argc, char *argv[]) strerror(errno)); } - test_mlock_limit(fd); test_file_apis(fd); test_process_vm_read(fd); test_ptrace(fd); diff --git a/tools/testing/selftests/net/tls.c b/tools/testing/selftests/net/tls.c index bc36c91c4480..73a777347227 100644 --- a/tools/testing/selftests/net/tls.c +++ b/tools/testing/selftests/net/tls.c @@ -1570,6 +1570,555 @@ TEST_F(tls, getsockopt) EXPECT_EQ(errno, EINVAL); } +TEST_F(tls, recv_efault) +{ + char *rec1 = "1111111111"; + char *rec2 = "2222222222"; + struct msghdr hdr = {}; + struct iovec iov[2]; + char recv_mem[12]; + int ret; + + if (self->notls) + SKIP(return, "no TLS support"); + + EXPECT_EQ(send(self->fd, rec1, 10, 0), 10); + EXPECT_EQ(send(self->fd, rec2, 10, 0), 10); + + iov[0].iov_base = recv_mem; + iov[0].iov_len = sizeof(recv_mem); + iov[1].iov_base = NULL; /* broken iov to make process_rx_list fail */ + iov[1].iov_len = 1; + + hdr.msg_iovlen = 2; + hdr.msg_iov = iov; + + EXPECT_EQ(recv(self->cfd, recv_mem, 1, 0), 1); + EXPECT_EQ(recv_mem[0], rec1[0]); + + ret = recvmsg(self->cfd, &hdr, 0); + EXPECT_LE(ret, sizeof(recv_mem)); + EXPECT_GE(ret, 9); + EXPECT_EQ(memcmp(rec1, recv_mem, 9), 0); + if (ret > 9) + EXPECT_EQ(memcmp(rec2, recv_mem + 9, ret - 9), 0); +} + +#define TLS_RECORD_TYPE_HANDSHAKE 0x16 + +TEST_F(tls_basic, recvmsg_nopad_retry_iov) +{ + char payload[32]; + char first_iov[sizeof(payload)]; + char later_iov[sizeof(payload) * 2]; + char expected_later_iov[sizeof(later_iov)]; + char cbuf[CMSG_SPACE(sizeof(char))]; + struct tls_crypto_info_keys tls13; + struct iovec iov[] = { + { .iov_base = first_iov, .iov_len = sizeof(first_iov) }, + { .iov_base = later_iov, .iov_len = sizeof(later_iov) }, + }; + struct msghdr msg = { + .msg_iov = iov, + .msg_iovlen = ARRAY_SIZE(iov), + .msg_control = cbuf, + .msg_controllen = sizeof(cbuf), + }; + int one = 1; + int ret; + int i; + + if (self->notls) + SKIP(return, "no TLS support"); + + tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128, + &tls13, 0); + + ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls13, tls13.len); + ASSERT_EQ(ret, 0); + + ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls13, tls13.len); + ASSERT_EQ(ret, 0); + + ret = setsockopt(self->cfd, SOL_TLS, TLS_RX_EXPECT_NO_PAD, + &one, sizeof(one)); + ASSERT_EQ(ret, 0); + + for (i = 0; i < sizeof(payload); i++) + payload[i] = 0x40 + i; + memset(first_iov, 0xa5, sizeof(first_iov)); + memset(later_iov, 0x5a, sizeof(later_iov)); + memset(expected_later_iov, 0x5a, sizeof(expected_later_iov)); + + /* A control record forces optimistic TLS 1.3 RX to retry. */ + ret = tls_send_cmsg(self->fd, TLS_RECORD_TYPE_HANDSHAKE, + payload, sizeof(payload), 0); + ASSERT_EQ(ret, sizeof(payload)); + + ret = recvmsg(self->cfd, &msg, 0); + ASSERT_EQ(ret, sizeof(payload)); + EXPECT_EQ(memcmp(first_iov, payload, sizeof(payload)), 0); + EXPECT_EQ(memcmp(later_iov, expected_later_iov, + sizeof(later_iov)), 0); +} + +/* key_update, length 1, update_not_requested */ +static const char key_update_msg[] = "\x18\x00\x00\x01\x00"; +static void tls_send_keyupdate(struct __test_metadata *_metadata, int fd) +{ + size_t len = sizeof(key_update_msg); + + EXPECT_EQ(tls_send_cmsg(fd, TLS_RECORD_TYPE_HANDSHAKE, + (char *)key_update_msg, len, 0), + len); +} + +static void tls_recv_keyupdate(struct __test_metadata *_metadata, int fd, int flags) +{ + char buf[100]; + + EXPECT_EQ(tls_recv_cmsg(_metadata, fd, TLS_RECORD_TYPE_HANDSHAKE, buf, sizeof(buf), flags), + sizeof(key_update_msg)); + EXPECT_EQ(memcmp(buf, key_update_msg, sizeof(key_update_msg)), 0); +} + +/* set the key to 0 then 1 for RX, immediately to 1 for TX */ +TEST_F(tls_basic, rekey_rx) +{ + struct tls_crypto_info_keys tls12_0, tls12_1; + char const *test_str = "test_message"; + int send_len = strlen(test_str) + 1; + char buf[20]; + int ret; + + if (self->notls) + return; + + tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128, + &tls12_0, 0); + tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128, + &tls12_1, 1); + + ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_1, tls12_1.len); + ASSERT_EQ(ret, 0); + + ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_0, tls12_0.len); + ASSERT_EQ(ret, 0); + + ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_1, tls12_1.len); + EXPECT_EQ(ret, 0); + + EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len); + EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len); + EXPECT_EQ(memcmp(buf, test_str, send_len), 0); +} + +/* set the key to 0 then 1 for TX, immediately to 1 for RX */ +TEST_F(tls_basic, rekey_tx) +{ + struct tls_crypto_info_keys tls12_0, tls12_1; + char const *test_str = "test_message"; + int send_len = strlen(test_str) + 1; + char buf[20]; + int ret; + + if (self->notls) + return; + + tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128, + &tls12_0, 0); + tls_crypto_info_init(TLS_1_3_VERSION, TLS_CIPHER_AES_GCM_128, + &tls12_1, 1); + + ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_0, tls12_0.len); + ASSERT_EQ(ret, 0); + + ret = setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_1, tls12_1.len); + ASSERT_EQ(ret, 0); + + ret = setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_1, tls12_1.len); + EXPECT_EQ(ret, 0); + + EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len); + EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len); + EXPECT_EQ(memcmp(buf, test_str, send_len), 0); +} + +TEST_F(tls, rekey) +{ + char const *test_str_1 = "test_message_before_rekey"; + char const *test_str_2 = "test_message_after_rekey"; + struct tls_crypto_info_keys tls12; + int send_len; + char buf[100]; + + if (variant->tls_version != TLS_1_3_VERSION) + return; + + /* initial send/recv */ + send_len = strlen(test_str_1) + 1; + EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len); + EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len); + EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0); + + /* update TX key */ + tls_send_keyupdate(_metadata, self->fd); + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0); + + /* send after rekey */ + send_len = strlen(test_str_2) + 1; + EXPECT_EQ(send(self->fd, test_str_2, send_len, 0), send_len); + + /* can't receive the KeyUpdate without a control message */ + EXPECT_EQ(recv(self->cfd, buf, send_len, 0), -1); + + /* get KeyUpdate */ + tls_recv_keyupdate(_metadata, self->cfd, 0); + + /* recv blocking -> -EKEYEXPIRED */ + EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), 0), -1); + EXPECT_EQ(errno, EKEYEXPIRED); + + /* recv non-blocking -> -EKEYEXPIRED */ + EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_DONTWAIT), -1); + EXPECT_EQ(errno, EKEYEXPIRED); + + /* update RX key */ + EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0); + + /* recv after rekey */ + EXPECT_NE(recv(self->cfd, buf, send_len, 0), -1); + EXPECT_EQ(memcmp(buf, test_str_2, send_len), 0); +} + +TEST_F(tls, rekey_fail) +{ + char const *test_str_1 = "test_message_before_rekey"; + char const *test_str_2 = "test_message_after_rekey"; + struct tls_crypto_info_keys tls12; + int send_len; + char buf[100]; + + /* initial send/recv */ + send_len = strlen(test_str_1) + 1; + EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len); + EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len); + EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0); + + /* update TX key */ + tls_send_keyupdate(_metadata, self->fd); + + if (variant->tls_version != TLS_1_3_VERSION) { + /* just check that rekey is not supported and return */ + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), -1); + EXPECT_EQ(errno, EBUSY); + return; + } + + /* successful update */ + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0); + + /* invalid update: change of version */ + tls_crypto_info_init(TLS_1_2_VERSION, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), -1); + EXPECT_EQ(errno, EINVAL); + + /* invalid update (RX socket): change of version */ + tls_crypto_info_init(TLS_1_2_VERSION, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), -1); + EXPECT_EQ(errno, EINVAL); + + /* invalid update: change of cipher */ + if (variant->cipher_type == TLS_CIPHER_AES_GCM_256) + tls_crypto_info_init(variant->tls_version, TLS_CIPHER_CHACHA20_POLY1305, &tls12, 1); + else + tls_crypto_info_init(variant->tls_version, TLS_CIPHER_AES_GCM_256, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), -1); + EXPECT_EQ(errno, EINVAL); + + /* send after rekey, the invalid updates shouldn't have an effect */ + send_len = strlen(test_str_2) + 1; + EXPECT_EQ(send(self->fd, test_str_2, send_len, 0), send_len); + + /* can't receive the KeyUpdate without a control message */ + EXPECT_EQ(recv(self->cfd, buf, send_len, 0), -1); + + /* get KeyUpdate */ + tls_recv_keyupdate(_metadata, self->cfd, 0); + + /* recv blocking -> -EKEYEXPIRED */ + EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), 0), -1); + EXPECT_EQ(errno, EKEYEXPIRED); + + /* recv non-blocking -> -EKEYEXPIRED */ + EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_DONTWAIT), -1); + EXPECT_EQ(errno, EKEYEXPIRED); + + /* update RX key */ + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0); + + /* recv after rekey */ + EXPECT_NE(recv(self->cfd, buf, send_len, 0), -1); + EXPECT_EQ(memcmp(buf, test_str_2, send_len), 0); +} + +TEST_F(tls, rekey_peek) +{ + char const *test_str_1 = "test_message_before_rekey"; + struct tls_crypto_info_keys tls12; + int send_len; + char buf[100]; + + if (variant->tls_version != TLS_1_3_VERSION) + return; + + send_len = strlen(test_str_1) + 1; + EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len); + + /* update TX key */ + tls_send_keyupdate(_metadata, self->fd); + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0); + + EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_PEEK), send_len); + EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0); + + EXPECT_EQ(recv(self->cfd, buf, send_len, 0), send_len); + EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0); + + /* can't receive the KeyUpdate without a control message */ + EXPECT_EQ(recv(self->cfd, buf, send_len, MSG_PEEK), -1); + + /* peek KeyUpdate */ + tls_recv_keyupdate(_metadata, self->cfd, MSG_PEEK); + + /* get KeyUpdate */ + tls_recv_keyupdate(_metadata, self->cfd, 0); + + /* update RX key */ + EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0); +} + +TEST_F(tls, splice_rekey) +{ + int send_len = TLS_PAYLOAD_MAX_LEN / 2; + char mem_send[TLS_PAYLOAD_MAX_LEN]; + char mem_recv[TLS_PAYLOAD_MAX_LEN]; + struct tls_crypto_info_keys tls12; + int p[2]; + + if (variant->tls_version != TLS_1_3_VERSION) + return; + + memrnd(mem_send, sizeof(mem_send)); + + ASSERT_GE(pipe(p), 0); + EXPECT_EQ(send(self->fd, mem_send, send_len, 0), send_len); + + /* update TX key */ + tls_send_keyupdate(_metadata, self->fd); + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0); + + EXPECT_EQ(send(self->fd, mem_send, send_len, 0), send_len); + + EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), send_len); + EXPECT_EQ(read(p[0], mem_recv, send_len), send_len); + EXPECT_EQ(memcmp(mem_send, mem_recv, send_len), 0); + + /* can't splice the KeyUpdate */ + EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), -1); + EXPECT_EQ(errno, EINVAL); + + /* peek KeyUpdate */ + tls_recv_keyupdate(_metadata, self->cfd, MSG_PEEK); + + /* get KeyUpdate */ + tls_recv_keyupdate(_metadata, self->cfd, 0); + + /* can't splice before updating the key */ + EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), -1); + EXPECT_EQ(errno, EKEYEXPIRED); + + /* update RX key */ + EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0); + + EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), send_len); + EXPECT_EQ(read(p[0], mem_recv, send_len), send_len); + EXPECT_EQ(memcmp(mem_send, mem_recv, send_len), 0); +} + +TEST_F(tls, rekey_peek_splice) +{ + char const *test_str_1 = "test_message_before_rekey"; + struct tls_crypto_info_keys tls12; + int send_len; + char buf[100]; + char mem_recv[TLS_PAYLOAD_MAX_LEN]; + int p[2]; + + if (variant->tls_version != TLS_1_3_VERSION) + return; + + ASSERT_GE(pipe(p), 0); + + send_len = strlen(test_str_1) + 1; + EXPECT_EQ(send(self->fd, test_str_1, send_len, 0), send_len); + + /* update TX key */ + tls_send_keyupdate(_metadata, self->fd); + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0); + + EXPECT_EQ(recv(self->cfd, buf, sizeof(buf), MSG_PEEK), send_len); + EXPECT_EQ(memcmp(buf, test_str_1, send_len), 0); + + EXPECT_EQ(splice(self->cfd, NULL, p[1], NULL, TLS_PAYLOAD_MAX_LEN, 0), send_len); + EXPECT_EQ(read(p[0], mem_recv, send_len), send_len); + EXPECT_EQ(memcmp(mem_recv, test_str_1, send_len), 0); +} + +TEST_F(tls, rekey_getsockopt) +{ + struct tls_crypto_info_keys tls12; + struct tls_crypto_info_keys tls12_get; + socklen_t len; + + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 0); + + len = tls12.len; + EXPECT_EQ(getsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_get, &len), 0); + EXPECT_EQ(len, tls12.len); + EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0); + + len = tls12.len; + EXPECT_EQ(getsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_get, &len), 0); + EXPECT_EQ(len, tls12.len); + EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0); + + if (variant->tls_version != TLS_1_3_VERSION) + return; + + tls_send_keyupdate(_metadata, self->fd); + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0); + + tls_recv_keyupdate(_metadata, self->cfd, 0); + EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0); + + len = tls12.len; + EXPECT_EQ(getsockopt(self->fd, SOL_TLS, TLS_TX, &tls12_get, &len), 0); + EXPECT_EQ(len, tls12.len); + EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0); + + len = tls12.len; + EXPECT_EQ(getsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12_get, &len), 0); + EXPECT_EQ(len, tls12.len); + EXPECT_EQ(memcmp(&tls12_get, &tls12, tls12.len), 0); +} + +TEST_F(tls, rekey_poll_pending) +{ + char const *test_str = "test_message_after_rekey"; + struct tls_crypto_info_keys tls12; + struct pollfd pfd = { }; + int send_len; + int ret; + + if (variant->tls_version != TLS_1_3_VERSION) + return; + + /* update TX key */ + tls_send_keyupdate(_metadata, self->fd); + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0); + + /* get KeyUpdate */ + tls_recv_keyupdate(_metadata, self->cfd, 0); + + /* send immediately after rekey */ + send_len = strlen(test_str) + 1; + EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len); + + /* key hasn't been updated, expect cfd to be non-readable */ + pfd.fd = self->cfd; + pfd.events = POLLIN; + EXPECT_EQ(poll(&pfd, 1, 0), 0); + + ret = fork(); + ASSERT_GE(ret, 0); + + if (ret) { + int pid2, status; + + /* wait before installing the new key */ + sleep(1); + + /* update RX key while poll() is sleeping */ + EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0); + + pid2 = wait(&status); + EXPECT_EQ(pid2, ret); + EXPECT_EQ(status, 0); + } else { + pfd.fd = self->cfd; + pfd.events = POLLIN; + EXPECT_EQ(poll(&pfd, 1, 5000), 1); + + exit(!__test_passed(_metadata)); + } +} + +TEST_F(tls, rekey_poll_delay) +{ + char const *test_str = "test_message_after_rekey"; + struct tls_crypto_info_keys tls12; + struct pollfd pfd = { }; + int send_len; + int ret; + + if (variant->tls_version != TLS_1_3_VERSION) + return; + + /* update TX key */ + tls_send_keyupdate(_metadata, self->fd); + tls_crypto_info_init(variant->tls_version, variant->cipher_type, &tls12, 1); + EXPECT_EQ(setsockopt(self->fd, SOL_TLS, TLS_TX, &tls12, tls12.len), 0); + + /* get KeyUpdate */ + tls_recv_keyupdate(_metadata, self->cfd, 0); + + ret = fork(); + ASSERT_GE(ret, 0); + + if (ret) { + int pid2, status; + + /* wait before installing the new key */ + sleep(1); + + /* update RX key while poll() is sleeping */ + EXPECT_EQ(setsockopt(self->cfd, SOL_TLS, TLS_RX, &tls12, tls12.len), 0); + + sleep(1); + send_len = strlen(test_str) + 1; + EXPECT_EQ(send(self->fd, test_str, send_len, 0), send_len); + + pid2 = wait(&status); + EXPECT_EQ(pid2, ret); + EXPECT_EQ(status, 0); + } else { + pfd.fd = self->cfd; + pfd.events = POLLIN; + EXPECT_EQ(poll(&pfd, 1, 5000), 1); + exit(!__test_passed(_metadata)); + } +} + FIXTURE(tls_err) { int fd, cfd; diff --git a/virt/kvm/eventfd.c b/virt/kvm/eventfd.c index 1ff3da1930a9..c0a8fa1aa9d6 100644 --- a/virt/kvm/eventfd.c +++ b/virt/kvm/eventfd.c @@ -62,7 +62,7 @@ static void irqfd_resampler_notify(struct kvm_kernel_irqfd_resampler *resampler) list_for_each_entry_srcu(irqfd, &resampler->list, resampler_link, srcu_read_lock_held(&resampler->kvm->irq_srcu)) - eventfd_signal(irqfd->resamplefd, 1); + eventfd_signal(irqfd->resamplefd); } /* @@ -788,7 +788,7 @@ ioeventfd_write(struct kvm_vcpu *vcpu, struct kvm_io_device *this, gpa_t addr, if (!ioeventfd_in_range(p, addr, len, val)) return -EOPNOTSUPP; - eventfd_signal(p->eventfd, 1); + eventfd_signal(p->eventfd); return 0; }