From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793985512; b=hCT4IIMbSvYS03/Q0zsTLpcMaeLsk85U0EyI0YOfgj3uhhULhwKJoDpXPM1VEWfzgXUS KQJZQzUTkgv3vfTTDE93IjUWE0NQawzHzxDEOEG+juJsTavjYL47vGgyctIqVZ5I5SHkZ KFAelutK8XIxC4+X/BdZgtE8AHetwDJuEUtYhH+OqA+iAVqmRzox+LnCKo+S11FaareW+ uK+CzAvJ0yYIFwwUFaTf7n1HUQ4q2BpO8P+j0LelA0O8SNqt6xg23I4mcObqMNw20jx3i R2Y9I8cgXXWdR+A0Rq2xKkBMY62C+nz9OMINFNHVVSeyATWkI/IBopXu1jQx/AjwPNQ== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793985512; h=Received:DKIM-Signature:From:To:Cc:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:Content-Transfer-Encoding; bh=lJo25QMD8+USL/07Bj7XaXTmEVPyrU7J1krNytTaoIA=; b=CVaAP2yjBkKRIfcxu8LiFpCu/IjUa+0+3Byy3vq7yUYnTpdFY+hVxtTPolAWRojDqQnI sI5maMZS3txD5SJoOPjKGhCi5080cHU65XV1ljqf4357uBc5Vm1h2JuJJbyeHthwvHInb +FQeWnk9/v5LZS+6vVDn4W0IOTv7LDFmHJdJ/uATbZBWc3/nEpxorfrKm96DHIqtMrPbq CFxaoy60GIVo1FI84ysHpj6Bp1GHpSC5am7m1YyV7e5KwNV8JxnBEuNqEkKXybzeUQ/1D hyGIdMTImSi3XF/X0+trCXm+F12MnfVAP8yUhPtvgbBmAQyrykEIEVsEvYcN6fGDRWA== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=o4TgOMJR; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D7A41606A33C; Fri, 6 Nov 2026 17:18:29 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=o4TgOMJR Received: by smtp.kernel.org (Postfix) with ESMTPSA id 3B0741F008A1; Fri, 6 Nov 2026 17:18:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=korg; t=1793985509; bh=lJo25QMD8+USL/07Bj7XaXTmEVPyrU7J1krNytTaoIA=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=o4TgOMJRURZb2KhpITNsbvBDwEtd2La0kAFPv44/E5nB3WWjvucygfpZXyJi0ESXI No+rkIfrnL3UbhP2d6wS0cK1xZd/aEW3yiR7XaM5SgEGQ6NHpo1xoBDVzL2G8kyOMN VAlnuBuuq9Aw4jiNuicx8ieA8OrAu7R3S5TBexTA= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, torvalds@linux-foundation.org, stable@vger.kernel.org Cc: lwn@lwn.net, jslaby@suse.cz, Greg Kroah-Hartman Subject: Re: Linux 7.3.2 Date: Fri, 6 Nov 2026 12:17:46 -0500 Message-ID: <2026110646-angler-cadet-5f61@gregkh> X-Mailer: git-send-email 2.56.0 In-Reply-To: <2026110646-extruding-blitz-b2bc@gregkh> References: <2026110646-extruding-blitz-b2bc@gregkh> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Transfer-Encoding: 8bit diff --git a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v5.yaml b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v5.yaml index 86ca7f3ac281..b901e4288816 100644 --- a/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v5.yaml +++ b/Documentation/devicetree/bindings/interrupt-controller/arm,gic-v5.yaml @@ -120,13 +120,14 @@ patternProperties: cpus: description: - CPUs managed by the IRS. + CPUs managed by the IRS. Omit it if the IRS has no associated CPUs. arm,iaffids: $ref: /schemas/types.yaml#/definitions/uint16-array description: Interrupt AFFinity ID (IAFFID) associated with the CPU whose CPU node phandle is at the same index in the cpus array. + Omit it if the IRS has no associated CPUs. patternProperties: "^its@[0-9a-f]+$": @@ -210,12 +211,14 @@ patternProperties: - reg - reg-names + dependencies: + cpus: [ "arm,iaffids" ] + arm,iaffids: [ cpus ] + required: - compatible - reg - reg-names - - cpus - - arm,iaffids additionalProperties: false diff --git a/Documentation/networking/rxrpc.rst b/Documentation/networking/rxrpc.rst index 8926dab8e2e6..eca055a536aa 100644 --- a/Documentation/networking/rxrpc.rst +++ b/Documentation/networking/rxrpc.rst @@ -870,7 +870,6 @@ The kernel interface functions are as follows: int rxrpc_kernel_send_data(struct socket *sock, struct rxrpc_call *call, struct msghdr *msg, - size_t len, rxrpc_notify_end_tx_t notify_end_rx); This is used to supply either the request part of a client call or the @@ -879,14 +878,17 @@ The kernel interface functions are as follows: exclusively to in-kernel virtual addresses. msg.msg_flags may be given MSG_MORE if there will be subsequent data sends for this call. - The msg must not specify a destination address, control data or any flags - other than MSG_MORE. len is the total amount of data to transmit. + msg must not specify a destination address, control data or any flags + other than MSG_MORE or MSG_WAITALL. notify_end_rx can be NULL or it can be used to specify a function to be called when the call changes state to end the Tx phase. This function is called with a spinlock held to prevent the last DATA packet from being transmitted until the function returns. + It returns 0 if all the provided data has been buffered and a negative + error code on failure. + (#) Receive data from a call:: int rxrpc_kernel_recv_data(struct socket *sock, @@ -907,10 +909,15 @@ The kernel interface functions are as follows: want_more should be true if further data will be required after this is satisfied and false if this is the last item of the receive phase. - There are three normal returns: 0 if the buffer was filled and want_more - was true; 1 if the buffer was filled, the last DATA packet has been - emptied and want_more was false; and -EAGAIN if the function needs to be - called again. + For client calls, there are three normal returns: 0 if the buffer was + filled and want_more was true; 1 if the buffer was filled, the last DATA + packet has been emptied and want_more was false; and -EAGAIN if the + function needs to be called again. + + For service calls, there are four normal returns: 0 and -EAGAIN are the + same as for client calls; 2 indicates that the last DATA packet of the + request has been received, want_more was false and the call is still in + progress; and 1 indicates that the call is now successfully complete. If the last DATA packet is processed but the buffer contains less than the amount requested, EBADMSG is returned. If want_more wasn't set, but diff --git a/Makefile b/Makefile index 677b43beb9c6..692cda0dfe09 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 VERSION = 7 PATCHLEVEL = 3 -SUBLEVEL = 1 +SUBLEVEL = 2 EXTRAVERSION = NAME = Baby Opossum Posse diff --git a/arch/arm/mm/mmap.c b/arch/arm/mm/mmap.c index 3dbb383c26d5..dbb3f07a7785 100644 --- a/arch/arm/mm/mmap.c +++ b/arch/arm/mm/mmap.c @@ -11,6 +11,7 @@ #include #include #include +#include #include #define COLOUR_ALIGN(addr,pgoff) \ @@ -72,7 +73,10 @@ arch_get_unmapped_area(struct file *filp, unsigned long addr, info.length = len; info.low_limit = mm->mmap_base; info.high_limit = TASK_SIZE; - info.align_mask = do_align ? (PAGE_MASK & (SHMLBA - 1)) : 0; + if (filp && is_file_hugepages(filp)) + info.align_mask = huge_page_mask_align(filp); + else + info.align_mask = do_align ? (PAGE_MASK & (SHMLBA - 1)) : 0; info.align_offset = pgoff << PAGE_SHIFT; return vm_unmapped_area(&info); } @@ -123,7 +127,10 @@ arch_get_unmapped_area_topdown(struct file *filp, const unsigned long addr0, info.length = len; info.low_limit = FIRST_USER_ADDRESS; info.high_limit = mm->mmap_base; - info.align_mask = do_align ? (PAGE_MASK & (SHMLBA - 1)) : 0; + if (filp && is_file_hugepages(filp)) + info.align_mask = huge_page_mask_align(filp); + else + info.align_mask = do_align ? (PAGE_MASK & (SHMLBA - 1)) : 0; info.align_offset = pgoff << PAGE_SHIFT; addr = vm_unmapped_area(&info); diff --git a/arch/powerpc/kvm/book3s_hv.c b/arch/powerpc/kvm/book3s_hv.c index de05d721edc4..140eadb76dfe 100644 --- a/arch/powerpc/kvm/book3s_hv.c +++ b/arch/powerpc/kvm/book3s_hv.c @@ -1159,6 +1159,38 @@ static long kvmppc_h_rpt_invalidate(struct kvm_vcpu *vcpu, return H_SUCCESS; } +static long kvmppc_pseries_do_hpt_hcall(struct kvm_vcpu *vcpu, unsigned long req) +{ + switch (req) { + case H_REMOVE: + return kvmppc_h_remove(vcpu, kvmppc_get_gpr(vcpu, 4), + kvmppc_get_gpr(vcpu, 5), + kvmppc_get_gpr(vcpu, 6)); + case H_ENTER: + return kvmppc_h_enter(vcpu, kvmppc_get_gpr(vcpu, 4), + kvmppc_get_gpr(vcpu, 5), + kvmppc_get_gpr(vcpu, 6), + kvmppc_get_gpr(vcpu, 7)); + case H_READ: + return kvmppc_h_read(vcpu, kvmppc_get_gpr(vcpu, 4), + kvmppc_get_gpr(vcpu, 5)); + case H_CLEAR_MOD: + return kvmppc_h_clear_mod(vcpu, kvmppc_get_gpr(vcpu, 4), + kvmppc_get_gpr(vcpu, 5)); + case H_CLEAR_REF: + return kvmppc_h_clear_ref(vcpu, kvmppc_get_gpr(vcpu, 4), + kvmppc_get_gpr(vcpu, 5)); + case H_PROTECT: + return kvmppc_h_protect(vcpu, kvmppc_get_gpr(vcpu, 4), + kvmppc_get_gpr(vcpu, 5), + kvmppc_get_gpr(vcpu, 6)); + case H_BULK_REMOVE: + return kvmppc_h_bulk_remove(vcpu); + } + + return H_FUNCTION; +} + int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu) { struct kvm *kvm = vcpu->kvm; @@ -1174,47 +1206,17 @@ int kvmppc_pseries_do_hcall(struct kvm_vcpu *vcpu) switch (req) { case H_REMOVE: - ret = kvmppc_h_remove(vcpu, kvmppc_get_gpr(vcpu, 4), - kvmppc_get_gpr(vcpu, 5), - kvmppc_get_gpr(vcpu, 6)); - if (ret == H_TOO_HARD) - return RESUME_HOST; - break; case H_ENTER: - ret = kvmppc_h_enter(vcpu, kvmppc_get_gpr(vcpu, 4), - kvmppc_get_gpr(vcpu, 5), - kvmppc_get_gpr(vcpu, 6), - kvmppc_get_gpr(vcpu, 7)); - if (ret == H_TOO_HARD) - return RESUME_HOST; - break; case H_READ: - ret = kvmppc_h_read(vcpu, kvmppc_get_gpr(vcpu, 4), - kvmppc_get_gpr(vcpu, 5)); - if (ret == H_TOO_HARD) - return RESUME_HOST; - break; case H_CLEAR_MOD: - ret = kvmppc_h_clear_mod(vcpu, kvmppc_get_gpr(vcpu, 4), - kvmppc_get_gpr(vcpu, 5)); - if (ret == H_TOO_HARD) - return RESUME_HOST; - break; case H_CLEAR_REF: - ret = kvmppc_h_clear_ref(vcpu, kvmppc_get_gpr(vcpu, 4), - kvmppc_get_gpr(vcpu, 5)); - if (ret == H_TOO_HARD) - return RESUME_HOST; - break; case H_PROTECT: - ret = kvmppc_h_protect(vcpu, kvmppc_get_gpr(vcpu, 4), - kvmppc_get_gpr(vcpu, 5), - kvmppc_get_gpr(vcpu, 6)); - if (ret == H_TOO_HARD) - return RESUME_HOST; - break; case H_BULK_REMOVE: - ret = kvmppc_h_bulk_remove(vcpu); + preempt_disable(); + idx = srcu_read_lock(&kvm->srcu); + ret = kvmppc_pseries_do_hpt_hcall(vcpu, req); + srcu_read_unlock(&kvm->srcu, idx); + preempt_enable(); if (ret == H_TOO_HARD) return RESUME_HOST; break; @@ -1834,8 +1836,10 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu, else vsid = vcpu->arch.fault_gpa; + preempt_disable(); err = kvmppc_hpte_hv_fault(vcpu, vcpu->arch.fault_dar, vsid, vcpu->arch.fault_dsisr, true); + preempt_enable(); if (err == 0) { r = RESUME_GUEST; } else if (err == -1 || err == -2) { @@ -1881,8 +1885,10 @@ static int kvmppc_handle_exit_hv(struct kvm_vcpu *vcpu, else vsid = vcpu->arch.fault_gpa; + preempt_disable(); err = kvmppc_hpte_hv_fault(vcpu, vcpu->arch.fault_dar, vsid, vcpu->arch.fault_dsisr, false); + preempt_enable(); if (err == 0) { r = RESUME_GUEST; } else if (err == -1) { diff --git a/arch/powerpc/kvm/book3s_paired_singles.c b/arch/powerpc/kvm/book3s_paired_singles.c index bc39c76c9d9f..532f96293de0 100644 --- a/arch/powerpc/kvm/book3s_paired_singles.c +++ b/arch/powerpc/kvm/book3s_paired_singles.c @@ -479,12 +479,7 @@ static bool kvmppc_inst_is_paired_single(struct kvm_vcpu *vcpu, u32 inst) static int get_d_signext(u32 inst) { - int d = inst & 0x8ff; - - if (d & 0x800) - return -(d & 0x7ff); - - return (d & 0x7ff); + return sign_extend32(inst & 0xfff, 11); } static int kvmppc_ps_three_in(struct kvm_vcpu *vcpu, bool rc, diff --git a/arch/s390/kernel/kprobes.c b/arch/s390/kernel/kprobes.c index c450120b4474..e806b124e4ba 100644 --- a/arch/s390/kernel/kprobes.c +++ b/arch/s390/kernel/kprobes.c @@ -85,6 +85,9 @@ static bool can_probe(unsigned long paddr) if (!kallsyms_lookup_size_offset(paddr, NULL, &offset)) return false; + if (s390_search_extables(paddr)) + return false; + /* Decode instructions */ addr = paddr - offset; while (addr < paddr) { diff --git a/arch/s390/kernel/perf_cpum_cf.c b/arch/s390/kernel/perf_cpum_cf.c index 69fa303cd479..1d63dc3b0ab9 100644 --- a/arch/s390/kernel/perf_cpum_cf.c +++ b/arch/s390/kernel/perf_cpum_cf.c @@ -1797,6 +1797,9 @@ static int cfset_offline_cpu(unsigned int cpu) if (!list_empty(&cfset_session.head)) { list_for_each_entry(rp, &cfset_session.head, node) { + if (!cpumask_test_cpu(cpu, &rp->mask)) + continue; + p.sets = rp->ctrset; cfset_ioctl_off(&p); cpumask_clear_cpu(cpu, &rp->mask); diff --git a/arch/s390/kernel/process.c b/arch/s390/kernel/process.c index 416650ae4871..4676a6a8257e 100644 --- a/arch/s390/kernel/process.c +++ b/arch/s390/kernel/process.c @@ -131,6 +131,12 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.hardirq_timer = 0; p->thread.softirq_timer = 0; p->thread.last_break = 1; + /* + * s390 stores the svc return address in arch_data when calling + * sigreturn()/restart_syscall() via vdso. 1 means no valid address + * stored. + */ + p->restart_block.arch_data = 1; frame->sf.back_chain = 0; frame->sf.gprs[11 - 6] = (unsigned long)&frame->childregs; @@ -168,12 +174,6 @@ int copy_thread(struct task_struct *p, const struct kernel_clone_args *args) p->thread.acrs[0] = (unsigned int)(tls >> 32); p->thread.acrs[1] = (unsigned int)tls; } - /* - * s390 stores the svc return address in arch_data when calling - * sigreturn()/restart_syscall() via vdso. 1 means no valid address - * stored. - */ - p->restart_block.arch_data = 1; return 0; } diff --git a/arch/s390/kernel/signal.c b/arch/s390/kernel/signal.c index 4874de5edea0..b59b0c49a1aa 100644 --- a/arch/s390/kernel/signal.c +++ b/arch/s390/kernel/signal.c @@ -475,7 +475,11 @@ void arch_do_signal_or_restart(struct pt_regs *regs) } /* No longer in a system call */ clear_pt_regs_flag(regs, PIF_SYSCALL); - + if (!(current->restart_block.arch_data & 1)) { + regs->psw.addr = current->restart_block.arch_data; + current->restart_block.arch_data = 1; + regs->gprs[2] = -EINTR; + } rseq_signal_deliver(&ksig, regs); handle_signal(&ksig, oldset, regs); return; diff --git a/arch/s390/lib/probes.c b/arch/s390/lib/probes.c index 1e184a03442d..049300b68ddd 100644 --- a/arch/s390/lib/probes.c +++ b/arch/s390/lib/probes.c @@ -99,8 +99,8 @@ int probe_get_fixup_type(u16 *insn) break; case 0xec: switch (insn[2] & 0xff) { + case 0xe4: /* cgrb */ case 0xe5: /* clgrb */ - case 0xe6: /* cgrb */ case 0xf6: /* crb */ case 0xf7: /* clrb */ case 0xfc: /* cgib */ diff --git a/arch/s390/mm/extable.c b/arch/s390/mm/extable.c index 7498e858c401..848855da9a70 100644 --- a/arch/s390/mm/extable.c +++ b/arch/s390/mm/extable.c @@ -106,9 +106,9 @@ static bool ex_handler_ua_mvcos(const struct exception_table_entry *ex, regs->psw.addr = extable_fixup(ex); insn = (struct insn_ssf *)regs->psw.addr; if (from) - uaddr = regs->gprs[insn->b2] + insn->d2; + uaddr = (insn->b2 ? regs->gprs[insn->b2] : 0) + insn->d2; else - uaddr = regs->gprs[insn->b1] + insn->d1; + uaddr = (insn->b1 ? regs->gprs[insn->b1] : 0) + insn->d1; remainder = PAGE_SIZE - (uaddr & (PAGE_SIZE - 1)); if (regs->gprs[insn->r3] <= remainder) remainder = 0; diff --git a/arch/s390/pci/pci_irq.c b/arch/s390/pci/pci_irq.c index 9c9ed3d8d959..bcf11585df8d 100644 --- a/arch/s390/pci/pci_irq.c +++ b/arch/s390/pci/pci_irq.c @@ -22,12 +22,11 @@ static enum {FLOATING, DIRECTED} irq_delivery; */ static struct airq_iv *zpci_sbv; -/* - * interrupt bit vectors - * FLOATING - interrupt bit vector per function - * DIRECTED - interrupt bit vector per cpu - */ -static struct airq_iv **zpci_ibv; +/* FLOATING - interrupt bit vector per function */ +static struct airq_iv __rcu **zpci_ibv; + +/* DIRECTED - interrupt bit vector per cpu */ +static struct airq_iv **zpci_dibv; /* Modify PCI: Register floating adapter interruptions */ static int zpci_set_airq(struct zpci_dev *zdev) @@ -169,7 +168,7 @@ static struct irq_chip zpci_irq_chip = { static void zpci_handle_cpu_local_irq(bool rescan) { - struct airq_iv *dibv = zpci_ibv[smp_processor_id()]; + struct airq_iv *dibv = zpci_dibv[smp_processor_id()]; union zpci_sic_iib iib = {{0}}; struct irq_domain *msi_domain; irq_hw_number_t hwirq; @@ -279,7 +278,9 @@ static void zpci_floating_irq_handler(struct airq_struct *airq, } /* Scan the adapter interrupt vector for this device. */ - aibv = zpci_ibv[si]; + aibv = rcu_dereference(zpci_ibv[si]); + if (!aibv) + continue; for (ai = 0;;) { ai = airq_iv_scan(aibv, ai, airq_iv_end(aibv)); if (ai == -1UL) @@ -299,7 +300,7 @@ static int __alloc_airq(struct zpci_dev *zdev, int msi_vecs, { if (irq_delivery == DIRECTED) { /* Allocate cpu vector bits */ - *bit = airq_iv_alloc(zpci_ibv[0], msi_vecs); + *bit = airq_iv_alloc(zpci_dibv[0], msi_vecs); if (*bit == -1UL) return -EIO; } else { @@ -313,11 +314,14 @@ static int __alloc_airq(struct zpci_dev *zdev, int msi_vecs, zdev->aibv = airq_iv_create(msi_vecs, AIRQ_IV_PTR | AIRQ_IV_DATA | AIRQ_IV_BITLOCK, NULL); - if (!zdev->aibv) + if (!zdev->aibv) { + airq_iv_free_bit(zpci_sbv, *bit); + zdev->aisb = -1UL; return -ENOMEM; + } /* Wire up shortcut pointer */ - zpci_ibv[*bit] = zdev->aibv; + rcu_assign_pointer(zpci_ibv[*bit], zdev->aibv); /* Each function has its own interrupt vector */ *bit = 0; } @@ -339,13 +343,16 @@ static struct airq_struct zpci_airq = { static void zpci_msi_teardown_directed(struct zpci_dev *zdev) { - airq_iv_free(zpci_ibv[0], zdev->msi_first_bit, zdev->max_msi); + airq_iv_free(zpci_dibv[0], zdev->msi_first_bit, zdev->msi_nr_irqs); zdev->msi_first_bit = -1U; zdev->msi_nr_irqs = 0; } static void zpci_msi_teardown_floating(struct zpci_dev *zdev) { + rcu_assign_pointer(zpci_ibv[zdev->aisb], NULL); + synchronize_rcu(); + airq_iv_release(zdev->aibv); zdev->aibv = NULL; airq_iv_free_bit(zpci_sbv, zdev->aisb); @@ -425,9 +432,9 @@ static int zpci_msi_domain_alloc(struct irq_domain *domain, unsigned int virq, if (irq_delivery == DIRECTED) { for_each_possible_cpu(cpu) { - airq_iv_set_ptr(zpci_ibv[cpu], bit + i, + airq_iv_set_ptr(zpci_dibv[cpu], bit + i, (unsigned long)zbus->msi_parent_domain); - airq_iv_set_data(zpci_ibv[cpu], bit + i, hwirq + i); + airq_iv_set_data(zpci_dibv[cpu], bit + i, hwirq + i); } } else { airq_iv_set_ptr(zdev->aibv, bit + i, @@ -452,8 +459,8 @@ static void zpci_msi_clear_airq(struct irq_data *d, int i) if (irq_delivery == DIRECTED) { for_each_possible_cpu(cpu) { - airq_iv_set_ptr(zpci_ibv[cpu], bit + i, 0); - airq_iv_set_data(zpci_ibv[cpu], bit + i, 0); + airq_iv_set_ptr(zpci_dibv[cpu], bit + i, 0); + airq_iv_set_data(zpci_dibv[cpu], bit + i, 0); } } else { airq_iv_set_ptr(zdev->aibv, bit + i, 0); @@ -533,9 +540,13 @@ void zpci_remove_parent_msi_domain(struct zpci_bus *zbus) { struct fwnode_handle *fn; + if (!zbus->msi_parent_domain) + return; + fn = zbus->msi_parent_domain->fwnode; irq_domain_remove(zbus->msi_parent_domain); irq_domain_free_fwnode(fn); + zbus->msi_parent_domain = NULL; } static void __init cpu_enable_directed_irq(void *unused) @@ -543,7 +554,7 @@ static void __init cpu_enable_directed_irq(void *unused) union zpci_sic_iib iib = {{0}}; union zpci_sic_iib ziib = {{0}}; - iib.cdiib.dibv_addr = virt_to_phys(zpci_ibv[smp_processor_id()]->vector); + iib.cdiib.dibv_addr = virt_to_phys(zpci_dibv[smp_processor_id()]->vector); zpci_set_irq_ctrl(SIC_IRQ_MODE_SET_CPU, 0, &iib); zpci_set_irq_ctrl(SIC_IRQ_MODE_D_SINGLE, PCI_ISC, &ziib); @@ -558,33 +569,45 @@ static int __init zpci_directed_irq_init(void) if (!zpci_sbv) return -ENOMEM; - iib.diib.isc = PCI_ISC; - iib.diib.nr_cpus = num_possible_cpus(); - iib.diib.disb_addr = virt_to_phys(zpci_sbv->vector); - zpci_set_irq_ctrl(SIC_IRQ_MODE_DIRECT, 0, &iib); - - zpci_ibv = kzalloc_objs(*zpci_ibv, num_possible_cpus()); - if (!zpci_ibv) - return -ENOMEM; + zpci_dibv = kzalloc_objs(*zpci_dibv, num_possible_cpus()); + if (!zpci_dibv) + goto out_free_sbv; for_each_possible_cpu(cpu) { /* * Per CPU IRQ vectors look the same but bit-allocation * is only done on the first vector. */ - zpci_ibv[cpu] = airq_iv_create(cache_line_size() * BITS_PER_BYTE, - AIRQ_IV_PTR | - AIRQ_IV_DATA | - AIRQ_IV_CACHELINE | - (!cpu ? AIRQ_IV_ALLOC : 0), NULL); - if (!zpci_ibv[cpu]) - return -ENOMEM; + zpci_dibv[cpu] = airq_iv_create(cache_line_size() * BITS_PER_BYTE, + AIRQ_IV_PTR | + AIRQ_IV_DATA | + AIRQ_IV_CACHELINE | + (!cpu ? AIRQ_IV_ALLOC : 0), NULL); + if (!zpci_dibv[cpu]) + goto out_free_dibv; } + + iib.diib.isc = PCI_ISC; + iib.diib.nr_cpus = num_possible_cpus(); + iib.diib.disb_addr = virt_to_phys(zpci_sbv->vector); + zpci_set_irq_ctrl(SIC_IRQ_MODE_DIRECT, 0, &iib); on_each_cpu(cpu_enable_directed_irq, NULL, 1); zpci_irq_chip.irq_set_affinity = zpci_set_irq_affinity; return 0; + +out_free_dibv: + for_each_possible_cpu(cpu) { + if (zpci_dibv[cpu]) + airq_iv_release(zpci_dibv[cpu]); + } + kfree(zpci_dibv); + zpci_dibv = NULL; +out_free_sbv: + airq_iv_release(zpci_sbv); + zpci_sbv = NULL; + return -ENOMEM; } static int __init zpci_floating_irq_init(void) @@ -653,10 +676,12 @@ void __init zpci_irq_exit(void) if (irq_delivery == DIRECTED) { for_each_possible_cpu(cpu) { - airq_iv_release(zpci_ibv[cpu]); + airq_iv_release(zpci_dibv[cpu]); } + kfree(zpci_dibv); + } else { + kfree(zpci_ibv); } - kfree(zpci_ibv); if (zpci_sbv) airq_iv_release(zpci_sbv); unregister_adapter_interrupt(&zpci_airq); diff --git a/arch/x86/include/asm/cpufeatures.h b/arch/x86/include/asm/cpufeatures.h index fe84cf4e97d3..e713b5818c25 100644 --- a/arch/x86/include/asm/cpufeatures.h +++ b/arch/x86/include/asm/cpufeatures.h @@ -430,6 +430,7 @@ #define X86_FEATURE_SUCCOR (17*32+ 1) /* "succor" Uncorrectable error containment and recovery */ #define X86_FEATURE_CPPC_PERF_PRIO (17*32+ 2) /* CPPC Floor Perf support */ #define X86_FEATURE_SMCA (17*32+ 3) /* "smca" Scalable MCA */ +#define X86_FEATURE_BTB_CTX_ISOLATION (17*32+ 4) /* AMD: Branch predictions contexts isolated */ /* Intel-defined CPU features, CPUID level 0x00000007:0 (EDX), word 18 */ #define X86_FEATURE_AVX512_4VNNIW (18*32+ 2) /* "avx512_4vnniw" AVX-512 Neural Network Instructions */ diff --git a/arch/x86/include/asm/cpuid/api.h b/arch/x86/include/asm/cpuid/api.h index 82eddfa2347b..25b22b173f2a 100644 --- a/arch/x86/include/asm/cpuid/api.h +++ b/arch/x86/include/asm/cpuid/api.h @@ -181,6 +181,7 @@ static __always_inline bool cpuid_function_is_indexed(u32 function) case 0x1d: case 0x1e: case 0x1f: + case 0x23: case 0x24: case 0x8000001d: return true; diff --git a/arch/x86/include/asm/kvm-x86-ops.h b/arch/x86/include/asm/kvm-x86-ops.h index e213c9ae3e30..dd9026c1071c 100644 --- a/arch/x86/include/asm/kvm-x86-ops.h +++ b/arch/x86/include/asm/kvm-x86-ops.h @@ -82,6 +82,7 @@ KVM_X86_OP(interrupt_allowed) KVM_X86_OP(nmi_allowed) KVM_X86_OP(get_nmi_mask) KVM_X86_OP(set_nmi_mask) +KVM_X86_OP_OPTIONAL(clear_hlt) KVM_X86_OP(enable_nmi_window) KVM_X86_OP(enable_irq_window) KVM_X86_OP_OPTIONAL(update_cr8_intercept) diff --git a/arch/x86/include/asm/kvm_host.h b/arch/x86/include/asm/kvm_host.h index 683bb8bf43a9..e97f2ef6d278 100644 --- a/arch/x86/include/asm/kvm_host.h +++ b/arch/x86/include/asm/kvm_host.h @@ -1607,6 +1607,13 @@ struct kvm_x86_ops { int (*nmi_allowed)(struct kvm_vcpu *vcpu, bool for_injection); bool (*get_nmi_mask)(struct kvm_vcpu *vcpu); void (*set_nmi_mask)(struct kvm_vcpu *vcpu, bool masked); + /* + * Force the vCPU out of any hardware-tracked halted/inactive state so + * that it will fetch and execute instructions on the next VM-Enter. + * Only needed by VMX, where VMCS.GUEST_ACTIVITY_STATE persists across + * VM-Exit/VM-Enter; SVM has no equivalent VMCB field. + */ + void (*clear_hlt)(struct kvm_vcpu *vcpu); /* Whether or not a virtual NMI is pending in hardware. */ bool (*is_vnmi_pending)(struct kvm_vcpu *vcpu); /* diff --git a/arch/x86/include/asm/vmx.h b/arch/x86/include/asm/vmx.h index 3f1b3096ff04..e49b836c7313 100644 --- a/arch/x86/include/asm/vmx.h +++ b/arch/x86/include/asm/vmx.h @@ -20,6 +20,14 @@ #include #include +/* + * UNDEFINED is a KVM-defined value used to indicate that the basic exit reason + * is undefined/invalid, e.g. on VM-Fail or if TDX never attempted VM-Enter. + * The value itself is arbitrary, it just needs to not collide with a real exit + * reason. + */ +#define EXIT_REASON_UNDEFINED 0xdead + struct vmcs_hdr { u32 revision_id:31; u32 shadow_vmcs:1; diff --git a/arch/x86/kernel/cpu/bugs.c b/arch/x86/kernel/cpu/bugs.c index 56eac5611c31..1b2381da4d83 100644 --- a/arch/x86/kernel/cpu/bugs.c +++ b/arch/x86/kernel/cpu/bugs.c @@ -1175,6 +1175,7 @@ enum srso_mitigation { SRSO_MITIGATION_IBPB, SRSO_MITIGATION_IBPB_ON_VMEXIT, SRSO_MITIGATION_BP_SPEC_REDUCE, + SRSO_MITIGATION_USER_IBPB, }; static enum srso_mitigation srso_mitigation __ro_after_init = SRSO_MITIGATION_AUTO; @@ -2908,7 +2909,8 @@ static const char * const srso_strings[] = { [SRSO_MITIGATION_SAFE_RET] = "Mitigation: Safe RET", [SRSO_MITIGATION_IBPB] = "Mitigation: IBPB", [SRSO_MITIGATION_IBPB_ON_VMEXIT] = "Mitigation: IBPB on VMEXIT only", - [SRSO_MITIGATION_BP_SPEC_REDUCE] = "Mitigation: Reduced Speculation" + [SRSO_MITIGATION_BP_SPEC_REDUCE] = "Mitigation: Reduced Speculation", + [SRSO_MITIGATION_USER_IBPB] = "Mitigation: IBPB on context switch", }; static int __init srso_parse_cmdline(char *str) @@ -2948,7 +2950,9 @@ static void __init srso_select_mitigation(void) * required. Otherwise the 'microcode' mitigation is sufficient * to protect the user->user and guest->guest vectors. */ - if (cpu_attack_vector_mitigated(CPU_MITIGATE_GUEST_HOST) || + if ((cpu_attack_vector_mitigated(CPU_MITIGATE_GUEST_HOST) && + !boot_cpu_has(X86_FEATURE_BTB_CTX_ISOLATION)) + || (cpu_attack_vector_mitigated(CPU_MITIGATE_USER_KERNEL) && !boot_cpu_has(X86_FEATURE_SRSO_USER_KERNEL_NO))) { srso_mitigation = SRSO_MITIGATION_SAFE_RET; @@ -3024,6 +3028,16 @@ static void __init srso_update_mitigation(void) boot_cpu_has(X86_FEATURE_IBPB_BRTYPE)) srso_mitigation = SRSO_MITIGATION_IBPB; + /* + * See if IBPB on context switch is the only thing needed to address + * GUEST/GUEST and USER/USER vectors. + */ + if (srso_mitigation == SRSO_MITIGATION_MICROCODE && + boot_cpu_has(X86_FEATURE_SRSO_USER_KERNEL_NO) && + boot_cpu_has(X86_FEATURE_BTB_CTX_ISOLATION) && + spectre_v2_user_ibpb != SPECTRE_V2_USER_NONE) + srso_mitigation = SRSO_MITIGATION_USER_IBPB; + pr_info("%s\n", srso_strings[srso_mitigation]); } diff --git a/arch/x86/kernel/cpu/scattered.c b/arch/x86/kernel/cpu/scattered.c index 8665a6474806..41b4880be7ed 100644 --- a/arch/x86/kernel/cpu/scattered.c +++ b/arch/x86/kernel/cpu/scattered.c @@ -64,6 +64,7 @@ static const struct cpuid_bit cpuid_bits[] = { { X86_FEATURE_AMD_WORKLOAD_CLASS, CPUID_EAX, 22, 0x80000021, 0 }, { X86_FEATURE_TSA_SQ_NO, CPUID_ECX, 1, 0x80000021, 0 }, { X86_FEATURE_TSA_L1_NO, CPUID_ECX, 2, 0x80000021, 0 }, + { X86_FEATURE_BTB_CTX_ISOLATION, CPUID_ECX, 8, 0x80000021, 0 }, { X86_FEATURE_PERFMON_V2, CPUID_EAX, 0, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_V2, CPUID_EAX, 1, 0x80000022, 0 }, { X86_FEATURE_AMD_LBR_PMC_FREEZE, CPUID_EAX, 2, 0x80000022, 0 }, diff --git a/arch/x86/kvm/hyperv.c b/arch/x86/kvm/hyperv.c index 604651cb2739..8d2669d8ef34 100644 --- a/arch/x86/kvm/hyperv.c +++ b/arch/x86/kvm/hyperv.c @@ -802,7 +802,9 @@ static int synic_deliver_msg(struct kvm_vcpu_hv_synic *synic, u32 sint, if (no_retry) return 0; + hv_hdr.message_flags.asu8 = 0; hv_hdr.message_flags.msg_pending = 1; + r = kvm_vcpu_write_guest_page(vcpu, msg_page_gfn, &hv_hdr.message_flags, msg_off + diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index a0d608e3fceb..e56497089b59 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -5916,6 +5916,9 @@ static union kvm_cpu_role kvm_calc_cpu_role(struct kvm_vcpu *vcpu, return role; } + if (KVM_BUG_ON(____is_efer_lma(regs) && !____is_cr4_pae(regs), vcpu->kvm)) + *(u64 *)®s->efer &= ~EFER_LMA; + role.base.efer_nx = ____is_efer_nx(regs); role.base.cr0_wp = ____is_cr0_wp(regs); role.base.cr4_smep = ____is_cr4_smep(regs); @@ -7123,6 +7126,8 @@ void kvm_zap_gfn_range(struct kvm *kvm, gfn_t gfn_start, gfn_t gfn_end) if (WARN_ON_ONCE(gfn_end <= gfn_start)) return; + guard(srcu)(&kvm->srcu); + write_lock(&kvm->mmu_lock); kvm_mmu_invalidate_start(kvm); diff --git a/arch/x86/kvm/mmu/paging_tmpl.h b/arch/x86/kvm/mmu/paging_tmpl.h index e5450faa669e..6e0153f23136 100644 --- a/arch/x86/kvm/mmu/paging_tmpl.h +++ b/arch/x86/kvm/mmu/paging_tmpl.h @@ -367,13 +367,14 @@ static int FNAME(walk_addr_generic)(struct guest_walker *walker, pte_access = ~0; /* - * Queue a page fault for injection if this assertion fails, as callers - * assume that walker.fault contains sane info on a walk failure. I.e. - * avoid making the situation worse by inducing even worse badness - * between when the assertion fails and when KVM kicks the vCPU out to - * userspace (because the VM is bugged). + * Queue a page fault for injection if any of the below assertions fail, + * as callers assume that walker.fault contains sane info on a walk + * failure. I.e. avoid making the situation worse by inducing even + * worse badness between when the assertion fails and when KVM kicks + * the vCPU out to userspace (because the VM is bugged). */ - if (KVM_BUG_ON(is_long_mode(vcpu) && !is_pae(vcpu), vcpu->kvm)) + if (KVM_BUG_ON(is_long_mode(vcpu) && !is_pae(vcpu), vcpu->kvm) || + KVM_BUG_ON(walker->max_level > PT_MAX_FULL_LEVELS, vcpu->kvm)) goto error; ++walker->level; diff --git a/arch/x86/kvm/svm/nested.c b/arch/x86/kvm/svm/nested.c index f9090b601efa..2aa87ae6b8f3 100644 --- a/arch/x86/kvm/svm/nested.c +++ b/arch/x86/kvm/svm/nested.c @@ -789,6 +789,10 @@ static void nested_vmcb02_prepare_save(struct vcpu_svm *svm) kvm_set_rflags(vcpu, save->rflags | X86_EFLAGS_FIXED); + /* SVM ignores EFER.LMA if EFER.LME=0 (instead of failing VMRUN). */ + if (!(svm->nested.save.efer & EFER_LME)) + svm->nested.save.efer &= ~EFER_LMA; + svm_set_efer(vcpu, svm->nested.save.efer); svm_set_cr0(vcpu, svm->nested.save.cr0); @@ -1494,6 +1498,7 @@ int svm_allocate_nested(struct vcpu_svm *svm) if (!svm->nested.msrpm) goto err_free_vmcb02; + svm->nested.vmcb02.cpu = -1; svm->nested.initialized = true; return 0; @@ -2028,6 +2033,7 @@ static int svm_set_nested_state(struct kvm_vcpu *vcpu, if (!(save->cr0 & X86_CR0_PG) || !(save->cr0 & X86_CR0_PE) || (save->rflags & X86_EFLAGS_VM) || + ((save->efer & EFER_LMA) && !(save->efer & EFER_LME)) || !nested_vmcb_check_save(vcpu, &save_cached, false)) goto out_free; diff --git a/arch/x86/kvm/svm/svm.c b/arch/x86/kvm/svm/svm.c index c054803d4673..13c44d619651 100644 --- a/arch/x86/kvm/svm/svm.c +++ b/arch/x86/kvm/svm/svm.c @@ -3759,14 +3759,25 @@ static int pre_svm_run(struct kvm_vcpu *vcpu) struct vcpu_svm *svm = to_svm(vcpu); /* - * If the previous vmrun of the vmcb occurred on a different physical - * cpu, then mark the vmcb dirty and assign a new asid. Hardware's - * vmcb clean bits are per logical CPU, as are KVM's asid assignments. + * If the previous VMRUN of the VMCB occurred on a different physical + * cpu, then mark the VMCB dirty as hardware's clean bits are per pCPU. + * + * Reset the ASID generation in both VMCBs. This will lead to assigning + * a new ASID now, and then again when switching to the other VMCB. + * However, this is needed as the ASID is shared between the VMCBs, and + * otherwise it would be possible to use an ASID allocated on one pCPU + * on another, for example: + * - vCPU migrates from pCPU A to pCPU B, allocates a new ASID. + * - vCPU migrates back to pCPU A, and then switches the VMCB. + * - The new VMCB does not detect a pCPU change and runs on pCPU A with + * the new ASID allocated on pCPU B, which is potentially used by + * another vCPU/VM. */ if (unlikely(svm->current_vmcb->cpu != vcpu->cpu)) { - svm->current_vmcb->asid_generation = 0; vmcb_mark_all_dirty(svm->vmcb); svm->current_vmcb->cpu = vcpu->cpu; + svm->vmcb01.asid_generation = 0; + svm->nested.vmcb02.asid_generation = 0; } if (is_sev_guest(vcpu)) diff --git a/arch/x86/kvm/vmx/common.h b/arch/x86/kvm/vmx/common.h index 08005676702c..bbeec197a507 100644 --- a/arch/x86/kvm/vmx/common.h +++ b/arch/x86/kvm/vmx/common.h @@ -4,6 +4,7 @@ #include #include +#include #include "mmu.h" @@ -183,6 +184,25 @@ static inline void __vmx_deliver_posted_interrupt(struct kvm_vcpu *vcpu, kvm_vcpu_trigger_posted_interrupt(vcpu, POSTED_INTR_VECTOR); } +static inline int __vt_handle_notify(struct kvm_vcpu *vcpu, + unsigned long exit_qual) +{ + bool context_invalid = exit_qual & NOTIFY_VM_CONTEXT_INVALID; + + ++vcpu->stat.notify_window_exits; + + if (vcpu->kvm->arch.notify_vmexit_flags & KVM_X86_NOTIFY_VMEXIT_USER || + context_invalid) { + vcpu->run->exit_reason = KVM_EXIT_NOTIFY; + vcpu->run->notify.flags = context_invalid ? + KVM_NOTIFY_CONTEXT_INVALID : 0; + return 0; + } + + return 1; +} + noinstr void vmx_handle_nmi(struct kvm_vcpu *vcpu); +int vt_handle_bus_lock_vmexit(struct kvm_vcpu *vcpu); #endif /* __KVM_X86_VMX_COMMON_H */ diff --git a/arch/x86/kvm/vmx/main.c b/arch/x86/kvm/vmx/main.c index 4c52ab8d0786..75e5ca47a57b 100644 --- a/arch/x86/kvm/vmx/main.c +++ b/arch/x86/kvm/vmx/main.c @@ -10,6 +10,8 @@ #include "tdx.h" #include "tdx_arch.h" +#pragma GCC poison to_vmx + #ifdef CONFIG_KVM_INTEL_TDX static_assert(offsetof(struct vcpu_vmx, vt) == offsetof(struct vcpu_tdx, vt)); @@ -601,6 +603,18 @@ static void vt_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked) vmx_set_nmi_mask(vcpu, masked); } +static void vt_clear_hlt(struct kvm_vcpu *vcpu) +{ + /* + * TDX doesn't support disabling HLT-exiting, and KVM can't access a + * TD's VMCS, so there is never any hardware halted state to clear. + */ + if (is_td_vcpu(vcpu)) + return; + + vmx_clear_hlt(vcpu); +} + static void vt_enable_nmi_window(struct kvm_vcpu *vcpu) { /* Refer to the comments in tdx_inject_nmi(). */ @@ -876,6 +890,17 @@ static int vt_gmem_max_mapping_level(struct kvm *kvm, kvm_pfn_t pfn, #define vt_op_tdx_only(name) NULL #endif /* CONFIG_KVM_INTEL_TDX */ +int vt_handle_bus_lock_vmexit(struct kvm_vcpu *vcpu) +{ + /* + * Hardware may or may not set the BUS_LOCK_DETECTED flag on BUS_LOCK + * VM-Exits. Unconditionally set the flag here and leave the handling + * to .handle_exit() callback. + */ + to_vt(vcpu)->exit_reason.bus_lock_detected = true; + return 1; +} + #define VMX_REQUIRED_APICV_INHIBITS \ (BIT(APICV_INHIBIT_REASON_DISABLED) | \ BIT(APICV_INHIBIT_REASON_ABSENT) | \ @@ -964,6 +989,7 @@ struct kvm_x86_ops vt_x86_ops __initdata = { .nmi_allowed = vt_op(nmi_allowed), .get_nmi_mask = vt_op(get_nmi_mask), .set_nmi_mask = vt_op(set_nmi_mask), + .clear_hlt = vt_op(clear_hlt), .enable_nmi_window = vt_op(enable_nmi_window), .enable_irq_window = vt_op(enable_irq_window), .update_cr8_intercept = vt_op(update_cr8_intercept), diff --git a/arch/x86/kvm/vmx/posted_intr.c b/arch/x86/kvm/vmx/posted_intr.c index 4a6d9a17da23..24221ba553be 100644 --- a/arch/x86/kvm/vmx/posted_intr.c +++ b/arch/x86/kvm/vmx/posted_intr.c @@ -14,6 +14,8 @@ #include "vmx.h" #include "tdx.h" +#pragma GCC poison to_vmx + /* * Maintain a per-CPU list of vCPUs that need to be awakened by wakeup_handler() * when a WAKEUP_VECTOR interrupted is posted. vCPUs are added to the list when diff --git a/arch/x86/kvm/vmx/tdx.c b/arch/x86/kvm/vmx/tdx.c index 00850a3a77d6..b2993a2a27f6 100644 --- a/arch/x86/kvm/vmx/tdx.c +++ b/arch/x86/kvm/vmx/tdx.c @@ -921,36 +921,51 @@ static __always_inline u32 tdcall_to_vmx_exit_reason(struct kvm_vcpu *vcpu) return EXIT_REASON_TDCALL; } -static __always_inline u32 tdx_to_vmx_exit_reason(struct kvm_vcpu *vcpu) +static __always_inline bool tdx_is_exit_reason_valid(u64 vp_enter_ret) { - struct vcpu_tdx *tdx = to_tdx(vcpu); - u32 exit_reason; - - switch (tdx->vp_enter_ret & TDX_SEAMCALL_STATUS_MASK) { + switch (vp_enter_ret & TDX_SEAMCALL_STATUS_MASK) { case TDX_SUCCESS: case TDX_NON_RECOVERABLE_VCPU: case TDX_NON_RECOVERABLE_TD: case TDX_NON_RECOVERABLE_TD_NON_ACCESSIBLE: case TDX_NON_RECOVERABLE_TD_WRONG_APIC_MODE: - break; + return true; default: - return -1u; + return false; } +} - exit_reason = tdx->vp_enter_ret; +static __always_inline union vmx_exit_reason tdx_to_vmx_exit_reason(struct kvm_vcpu *vcpu) +{ + struct vcpu_tdx *tdx = to_tdx(vcpu); + union vmx_exit_reason exit_reason; - switch (exit_reason) { + /* + * Return the synthesized invalid Exit Reason, as the TDX module + * never attempted to run the vCPU, i.e. the Exit Reason is undefined, + * but this is NOT a failed VM-Enter. + */ + if (!tdx_is_exit_reason_valid(tdx->vp_enter_ret)) + return (union vmx_exit_reason) { + .basic = EXIT_REASON_UNDEFINED, + }; + + exit_reason.full = (u32)tdx->vp_enter_ret; + + switch (exit_reason.basic) { case EXIT_REASON_TDCALL: if (tdvmcall_exit_type(vcpu)) - return EXIT_REASON_VMCALL; - - return tdcall_to_vmx_exit_reason(vcpu); + exit_reason.basic = EXIT_REASON_VMCALL; + else + exit_reason.basic = tdcall_to_vmx_exit_reason(vcpu); + break; case EXIT_REASON_EPT_MISCONFIG: /* * Defer KVM_BUG_ON() until tdx_handle_exit() because this is in * non-instrumentable code with interrupts disabled. */ - return -1u; + exit_reason.basic = EXIT_REASON_UNDEFINED; + break; default: break; } @@ -967,7 +982,7 @@ static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu) tdx->vp_enter_ret = tdh_vp_enter(&tdx->vp, &tdx->vp_enter_args); - vt->exit_reason.full = tdx_to_vmx_exit_reason(vcpu); + vt->exit_reason = tdx_to_vmx_exit_reason(vcpu); vt->exit_qualification = tdx->vp_enter_args.rcx; tdx->ext_exit_qualification = tdx->vp_enter_args.rdx; @@ -981,8 +996,7 @@ static noinstr void tdx_vcpu_enter_exit(struct kvm_vcpu *vcpu) static bool tdx_failed_vmentry(struct kvm_vcpu *vcpu) { - return vmx_get_exit_reason(vcpu).failed_vmentry && - vmx_get_exit_reason(vcpu).full != -1u; + return vmx_get_exit_reason(vcpu).failed_vmentry; } static fastpath_t tdx_exit_handlers_fastpath(struct kvm_vcpu *vcpu) @@ -1071,8 +1085,16 @@ fastpath_t tdx_vcpu_run(struct kvm_vcpu *vcpu, u64 run_flags) * allowing vCPU entry to avoid contention with tdh_vp_enter() and * TDCALLs. */ - if (unlikely(READ_ONCE(to_kvm_tdx(vcpu->kvm)->wait_for_sept_zap))) + if (unlikely(READ_ONCE(to_kvm_tdx(vcpu->kvm)->wait_for_sept_zap))) { + /* + * The vCPU never entered the guest, but this looks like a + * handled exit to the caller. Synthesize an invalid exit + * reason so the previous exit's stale value isn't consumed + * a second time. + */ + vt->exit_reason.full = EXIT_REASON_UNDEFINED; return EXIT_FASTPATH_EXIT_HANDLED; + } trace_kvm_entry(vcpu, run_flags & KVM_RUN_FORCE_IMMEDIATE_EXIT); @@ -2028,7 +2050,7 @@ int tdx_complete_emulated_msr(struct kvm_vcpu *vcpu, int err) } -int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) +static int __tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) { struct vcpu_tdx *tdx = to_tdx(vcpu); u64 vp_enter_ret = tdx->vp_enter_ret; @@ -2126,6 +2148,11 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) * - If it's not an MSMI, no need to do anything here. */ return 1; + case EXIT_REASON_NOTIFY: + /* NMI blocking state is handled by TDX module */ + return __vt_handle_notify(vcpu, vmx_get_exit_qual(vcpu)); + case EXIT_REASON_BUS_LOCK: + return vt_handle_bus_lock_vmexit(vcpu); default: break; } @@ -2135,13 +2162,29 @@ int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) return 0; } +int tdx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t fastpath) +{ + int ret = __tdx_handle_exit(vcpu, fastpath); + + /* Exit to user space when bus lock was detected */ + if (vmx_get_exit_reason(vcpu).bus_lock_detected) { + if (ret > 0) { + vcpu->run->exit_reason = KVM_EXIT_X86_BUS_LOCK; + ret = 0; + } + + vcpu->run->flags |= KVM_RUN_X86_BUS_LOCK; + } + return ret; +} + void tdx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code) { struct vcpu_tdx *tdx = to_tdx(vcpu); *reason = tdx->vt.exit_reason.full; - if (*reason != -1u) { + if (tdx_is_exit_reason_valid(tdx->vp_enter_ret)) { *info1 = vmx_get_exit_qual(vcpu); *info2 = tdx->ext_exit_qualification; *intr_info = vmx_get_intr_info(vcpu); @@ -3149,6 +3192,17 @@ static int tdx_vcpu_init(struct kvm_vcpu *vcpu, struct kvm_tdx_cmd *cmd) td_vmcs_write64(tdx, POSTED_INTR_DESC_ADDR, __pa(&tdx->vt.pi_desc)); td_vmcs_setbit32(tdx, PIN_BASED_VM_EXEC_CONTROL, PIN_BASED_POSTED_INTR); + if (kvm_notify_vmexit_enabled(vcpu->kvm)) { + td_vmcs_setbit32(tdx, SECONDARY_VM_EXEC_CONTROL, + SECONDARY_EXEC_NOTIFY_VM_EXITING); + td_vmcs_write32(tdx, NOTIFY_WINDOW, + vcpu->kvm->arch.notify_window); + } + + if (vcpu->kvm->arch.bus_lock_detection_enabled) + td_vmcs_setbit32(tdx, SECONDARY_VM_EXEC_CONTROL, + SECONDARY_EXEC_BUS_LOCK_DETECTION); + tdx->state = VCPU_TD_STATE_INITIALIZED; return 0; diff --git a/arch/x86/kvm/vmx/vmx.c b/arch/x86/kvm/vmx/vmx.c index 612ab07d4100..3d96c3481a78 100644 --- a/arch/x86/kvm/vmx/vmx.c +++ b/arch/x86/kvm/vmx/vmx.c @@ -1909,7 +1909,7 @@ int vmx_skip_emulated_instruction(struct kvm_vcpu *vcpu) return skip_emulated_instruction(vcpu); } -static void vmx_clear_hlt(struct kvm_vcpu *vcpu) +void vmx_clear_hlt(struct kvm_vcpu *vcpu) { /* * Ensure that we clear the HLT state in the VMCS. We don't need to @@ -6267,23 +6267,9 @@ static int handle_encls(struct kvm_vcpu *vcpu) } #endif /* CONFIG_X86_SGX_KVM */ -static int handle_bus_lock_vmexit(struct kvm_vcpu *vcpu) -{ - /* - * Hardware may or may not set the BUS_LOCK_DETECTED flag on BUS_LOCK - * VM-Exits. Unconditionally set the flag here and leave the handling to - * vmx_handle_exit(). - */ - to_vt(vcpu)->exit_reason.bus_lock_detected = true; - return 1; -} - static int handle_notify(struct kvm_vcpu *vcpu) { unsigned long exit_qual = vmx_get_exit_qual(vcpu); - bool context_invalid = exit_qual & NOTIFY_VM_CONTEXT_INVALID; - - ++vcpu->stat.notify_window_exits; /* * Notify VM exit happened while executing iret from NMI, @@ -6293,15 +6279,7 @@ static int handle_notify(struct kvm_vcpu *vcpu) vmcs_set_bits(GUEST_INTERRUPTIBILITY_INFO, GUEST_INTR_STATE_NMI); - if (vcpu->kvm->arch.notify_vmexit_flags & KVM_X86_NOTIFY_VMEXIT_USER || - context_invalid) { - vcpu->run->exit_reason = KVM_EXIT_NOTIFY; - vcpu->run->notify.flags = context_invalid ? - KVM_NOTIFY_CONTEXT_INVALID : 0; - return 0; - } - - return 1; + return __vt_handle_notify(vcpu, exit_qual); } static int vmx_get_msr_imm_reg(struct kvm_vcpu *vcpu) @@ -6377,7 +6355,7 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = { [EXIT_REASON_VMFUNC] = handle_vmx_instruction, [EXIT_REASON_PREEMPTION_TIMER] = handle_preemption_timer, [EXIT_REASON_ENCLS] = handle_encls, - [EXIT_REASON_BUS_LOCK] = handle_bus_lock_vmexit, + [EXIT_REASON_BUS_LOCK] = vt_handle_bus_lock_vmexit, [EXIT_REASON_NOTIFY] = handle_notify, [EXIT_REASON_SEAMCALL] = handle_tdx_instruction, [EXIT_REASON_TDCALL] = handle_tdx_instruction, @@ -6388,6 +6366,8 @@ static int (*kvm_vmx_exit_handlers[])(struct kvm_vcpu *vcpu) = { static const int kvm_vmx_max_exit_handlers = ARRAY_SIZE(kvm_vmx_exit_handlers); +static_assert(EXIT_REASON_UNDEFINED >= ARRAY_SIZE(kvm_vmx_exit_handlers)); + void vmx_get_exit_info(struct kvm_vcpu *vcpu, u32 *reason, u64 *info1, u64 *info2, u32 *intr_info, u32 *error_code) { @@ -6873,11 +6853,12 @@ int vmx_handle_exit(struct kvm_vcpu *vcpu, fastpath_t exit_fastpath) * a bus lock in guest. */ if (vmx_get_exit_reason(vcpu).bus_lock_detected) { - if (ret > 0) + if (ret > 0) { vcpu->run->exit_reason = KVM_EXIT_X86_BUS_LOCK; + ret = 0; + } vcpu->run->flags |= KVM_RUN_X86_BUS_LOCK; - return 0; } return ret; } @@ -7474,7 +7455,7 @@ static noinstr void vmx_vcpu_enter_exit(struct kvm_vcpu *vcpu, vmx_enable_fb_clear(vmx); if (unlikely(vmx->fail)) { - vmx->vt.exit_reason.full = 0xdead; + vmx->vt.exit_reason.full = EXIT_REASON_UNDEFINED; goto out; } diff --git a/arch/x86/kvm/vmx/x86_ops.h b/arch/x86/kvm/vmx/x86_ops.h index 054fd14bb2e1..0b8f364621ab 100644 --- a/arch/x86/kvm/vmx/x86_ops.h +++ b/arch/x86/kvm/vmx/x86_ops.h @@ -95,6 +95,7 @@ int vmx_interrupt_allowed(struct kvm_vcpu *vcpu, bool for_injection); int vmx_nmi_allowed(struct kvm_vcpu *vcpu, bool for_injection); bool vmx_get_nmi_mask(struct kvm_vcpu *vcpu); void vmx_set_nmi_mask(struct kvm_vcpu *vcpu, bool masked); +void vmx_clear_hlt(struct kvm_vcpu *vcpu); void vmx_enable_nmi_window(struct kvm_vcpu *vcpu); void vmx_enable_irq_window(struct kvm_vcpu *vcpu); void vmx_update_cr8_intercept(struct kvm_vcpu *vcpu, int tpr, int irr); diff --git a/arch/x86/kvm/x86.c b/arch/x86/kvm/x86.c index aad065d035fb..c76498d1a162 100644 --- a/arch/x86/kvm/x86.c +++ b/arch/x86/kvm/x86.c @@ -399,30 +399,37 @@ void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu, switch (ex->vector) { case DB_VECTOR: /* - * "Certain debug exceptions may clear bit 0-3. The - * remaining contents of the DR6 register are never - * cleared by the processor". - */ - vcpu->arch.dr6 &= ~DR_TRAP_BITS; - /* - * In order to reflect the #DB exception payload in guest - * dr6, three components need to be considered: active low - * bit, FIXED_1 bits and active high bits (e.g. DR6_BD, - * DR6_BS and DR6_BT) - * DR6_ACTIVE_LOW contains the FIXED_1 and active low bits. - * In the target guest dr6: - * FIXED_1 bits should always be set. - * Active low bits should be cleared if 1-setting in payload. - * Active high bits should be set if 1-setting in payload. + * DR6 is a mess. Reserved/unused bits are fixed-to-1, and so + * to maintain backwards compatibility with existing software, + * features that use previously-reserved bits have active-low + * semantics, i.e. clear the bit when the feature is present in + * the payload. + * + * Further complicating matters, some DR6 bits are preserved by + * hardware, while others are explicitly modified on every #DB. + * The trap bits are always set based on the payload, as is the + * RTM flag (but it's active low). All other bits are modified + * if and only if a relevant debug exception occurs, e.g. BD, + * BS, and BT are never cleared by hardware, and BLD is never + * set by hardware (when supported, excepting RESET). * - * Note, the payload is compatible with the pending debug - * exceptions/exit qualification under VMX, that active_low bits - * are active high in payload. - * So they need to be flipped for DR6. + * Lastly, the payload does NOT have active-low semantics, e.g. + * so that it's compatible VMX's pending debug exceptions and + * qualification fields, and to avoid bleeding the DR6 madness + * into other KVM code. + * + * To compute DR6: + * + * 1. "Reset" the bits that are modified on all #DBs + * 2. Clear active-low bits that are present in the payload. + * 3. Set active-high bits that are present in the payload. + * 4. Clear fixed-0 bits. + * 5. Set fixed-1 bits. */ - vcpu->arch.dr6 |= DR6_ACTIVE_LOW; - vcpu->arch.dr6 |= ex->payload; - vcpu->arch.dr6 ^= ex->payload & DR6_ACTIVE_LOW; + vcpu->arch.dr6 &= ~DR_TRAP_BITS; + vcpu->arch.dr6 |= DR6_RTM; + vcpu->arch.dr6 &= ~(ex->payload & DR6_ACTIVE_LOW); + vcpu->arch.dr6 |= (ex->payload & ~DR6_ACTIVE_LOW); /* * The #DB payload is defined as compatible with the 'pending @@ -431,6 +438,7 @@ void kvm_deliver_exception_payload(struct kvm_vcpu *vcpu, * breakpoint), it is reserved and must be zero in DR6. */ vcpu->arch.dr6 &= ~BIT(12); + vcpu->arch.dr6 |= DR6_FIXED_1; break; case PF_VECTOR: vcpu->arch.cr2 = ex->payload; @@ -448,6 +456,8 @@ static void kvm_queue_exception_vmexit(struct kvm_vcpu *vcpu, unsigned int vecto { struct kvm_queued_exception *ex = &vcpu->arch.exception_vmexit; + kvm_make_request(KVM_REQ_EVENT, vcpu); + ex->vector = vector; ex->injected = false; ex->pending = true; @@ -7925,10 +7935,8 @@ void __kvm_set_or_clear_apicv_inhibit(struct kvm *kvm, kvm->arch.apicv_inhibit_reasons = new; if (new) { unsigned long gfn = gpa_to_gfn(APIC_DEFAULT_PHYS_BASE); - int idx = srcu_read_lock(&kvm->srcu); kvm_zap_gfn_range(kvm, gfn, gfn+1); - srcu_read_unlock(&kvm->srcu, idx); } } else { kvm->arch.apicv_inhibit_reasons = new; @@ -9062,6 +9070,19 @@ int kvm_arch_vcpu_ioctl_set_mpstate(struct kvm_vcpu *vcpu, } kvm_set_mp_state(vcpu, mp_state->mp_state); + + /* + * Force the vCPU out of any hardware-tracked halted state, e.g. VMX's + * GUEST_ACTIVITY_STATE=HLT, when userspace puts the vCPU into a state + * other than HALTED. The hardware state is sticky across VM-Exit and + * VM-Enter and is not touched by any other ioctl, so a vCPU that halted + * with HLT-exiting disabled stays wedged even after userspace rewrites + * its registers, e.g. when a VMM emulates a machine reset. + */ + if (kvm_hlt_in_guest(vcpu->kvm) && + mp_state->mp_state != KVM_MP_STATE_HALTED) + kvm_x86_call(clear_hlt)(vcpu); + kvm_make_request(KVM_REQ_EVENT, vcpu); ret = 0; diff --git a/arch/x86/kvm/xen.c b/arch/x86/kvm/xen.c index d113bd0f1c3d..6d142813dda8 100644 --- a/arch/x86/kvm/xen.c +++ b/arch/x86/kvm/xen.c @@ -728,6 +728,16 @@ int __kvm_xen_has_interrupt(struct kvm_vcpu *v) BUILD_BUG_ON(sizeof(rc) != sizeof_field(struct compat_vcpu_info, evtchn_upcall_pending)); + /* + * kvm_gpc_check() checks the memslot generation, so kvm->srcu must be + * held. Most callers hold it already, but this is also reached from + * kvm_emulate_halt() on the VM-Exit path and from kvm_vcpu_block(), + * where vcpu_enter_guest() has already dropped the vCPU's SRCU lock. + * Taking SRCU does not sleep, so it is safe even in the atomic case + * which is handled below. + */ + guard(srcu)(&v->kvm->srcu); + read_lock_irqsave(&gpc->lock, flags); while (!kvm_gpc_check(gpc, sizeof(struct vcpu_info))) { read_unlock_irqrestore(&gpc->lock, flags); @@ -941,6 +951,10 @@ int kvm_xen_vcpu_set_attr(struct kvm_vcpu *vcpu, struct kvm_xen_vcpu_attr *data) break; } + r = -ENXIO; + if (!IS_ALIGNED(data->u.gpa, sizeof(u32))) + break; + r = kvm_gpc_activate(&vcpu->arch.xen.vcpu_info_cache, data->u.gpa, sizeof(struct vcpu_info)); } else { @@ -950,6 +964,10 @@ int kvm_xen_vcpu_set_attr(struct kvm_vcpu *vcpu, struct kvm_xen_vcpu_attr *data) break; } + r = -ENXIO; + if (!IS_ALIGNED(data->u.hva, sizeof(u32))) + break; + r = kvm_gpc_activate_hva(&vcpu->arch.xen.vcpu_info_cache, data->u.hva, sizeof(struct vcpu_info)); } diff --git a/arch/x86/mm/pat/set_memory.c b/arch/x86/mm/pat/set_memory.c index 4652487b5572..226660973d51 100644 --- a/arch/x86/mm/pat/set_memory.c +++ b/arch/x86/mm/pat/set_memory.c @@ -754,7 +754,7 @@ pte_t *lookup_address_in_pgd_attr(pgd_t *pgd, unsigned long address, *level = PG_LEVEL_512G; *nx |= pgd_flags(*pgd) & _PAGE_NX; - *rw &= pgd_flags(*pgd) & _PAGE_RW; + *rw &= !!(pgd_flags(*pgd) & _PAGE_RW); p4d = p4d_offset(pgd, address); if (p4d_none(*p4d)) @@ -765,7 +765,7 @@ pte_t *lookup_address_in_pgd_attr(pgd_t *pgd, unsigned long address, *level = PG_LEVEL_1G; *nx |= p4d_flags(*p4d) & _PAGE_NX; - *rw &= p4d_flags(*p4d) & _PAGE_RW; + *rw &= !!(p4d_flags(*p4d) & _PAGE_RW); pud = pud_offset(p4d, address); if (pud_none(*pud)) @@ -776,7 +776,7 @@ pte_t *lookup_address_in_pgd_attr(pgd_t *pgd, unsigned long address, *level = PG_LEVEL_2M; *nx |= pud_flags(*pud) & _PAGE_NX; - *rw &= pud_flags(*pud) & _PAGE_RW; + *rw &= !!(pud_flags(*pud) & _PAGE_RW); pmd = pmd_offset(pud, address); if (pmd_none(*pmd)) @@ -787,7 +787,7 @@ pte_t *lookup_address_in_pgd_attr(pgd_t *pgd, unsigned long address, *level = PG_LEVEL_4K; *nx |= pmd_flags(*pmd) & _PAGE_NX; - *rw &= pmd_flags(*pmd) & _PAGE_RW; + *rw &= !!(pmd_flags(*pmd) & _PAGE_RW); return pte_offset_kernel(pmd, address); } diff --git a/block/blk-crypto-fallback.c b/block/blk-crypto-fallback.c index 2a8f40a65158..12e0b487e23a 100644 --- a/block/blk-crypto-fallback.c +++ b/block/blk-crypto-fallback.c @@ -524,6 +524,12 @@ bool blk_crypto_fallback_bio_prep(struct bio *bio) int blk_crypto_fallback_evict_key(const struct blk_crypto_key *key) { + /* + * Pairs with smp_store_release() in + * blk_crypto_fallback_start_using_mode() + */ + if (!smp_load_acquire(&tfms_inited[key->crypto_cfg.crypto_mode])) + return 0; return __blk_crypto_evict_key(blk_crypto_fallback_profile, key); } diff --git a/drivers/accel/amdxdna/aie2_message.c b/drivers/accel/amdxdna/aie2_message.c index b4c49259a1a2..f658760c3d48 100644 --- a/drivers/accel/amdxdna/aie2_message.c +++ b/drivers/accel/amdxdna/aie2_message.c @@ -277,6 +277,7 @@ int aie2_create_context(struct amdxdna_dev_hdl *ndev, struct amdxdna_hwctx *hwct free_channel: xdna_mailbox_free_channel(hwctx->priv->mbox_chann); + hwctx->priv->mbox_chann = NULL; del_ctx_req: aie2_destroy_context_req(ndev, hwctx->fw_ctx_id); return ret; diff --git a/drivers/accel/amdxdna/aie2_msg_priv.h b/drivers/accel/amdxdna/aie2_msg_priv.h index fd65a4236d49..8ed2b5701dcc 100644 --- a/drivers/accel/amdxdna/aie2_msg_priv.h +++ b/drivers/accel/amdxdna/aie2_msg_priv.h @@ -530,6 +530,7 @@ enum aie2_dev_revision { AIE2_DEV_REVISION_GPT1, AIE2_DEV_REVISION_GPT2, AIE2_DEV_REVISION_GPT3, + AIE2_DEV_REVISION_GRGH, AIE2_DEV_REVISION_UNKN, }; diff --git a/drivers/accel/amdxdna/npu4_regs.c b/drivers/accel/amdxdna/npu4_regs.c index 15a161384625..c9648c8032ec 100644 --- a/drivers/accel/amdxdna/npu4_regs.c +++ b/drivers/accel/amdxdna/npu4_regs.c @@ -153,6 +153,7 @@ const struct amdxdna_rev_vbnv npu4_rev_vbnv_tbl[] = { { AIE2_DEV_REVISION_GPT1, "NPU Gorgon Point 1" }, { AIE2_DEV_REVISION_GPT2, "NPU Gorgon Point 2" }, { AIE2_DEV_REVISION_GPT3, "NPU Gorgon Point 3" }, + { AIE2_DEV_REVISION_GRGH, "NPU Gorgon Halo" }, { 0 } }; diff --git a/drivers/android/binder/thread.rs b/drivers/android/binder/thread.rs index 24f7b5be1b0e..a1bef151c2c5 100644 --- a/drivers/android/binder/thread.rs +++ b/drivers/android/binder/thread.rs @@ -344,13 +344,13 @@ fn push_reply_work(&mut self, code: u32) -> PushWorkRes { } } - fn push_return_work(&mut self, reply: u32) { + fn push_return_work(&mut self, reply: u32) -> PushWorkRes { if let Ok(work) = ListArc::try_from_arc(self.return_work.clone()) { work.set_error_code(reply); - // Not notifying: Reply to current thread. - let _ = self.push_work(work); + self.push_work(work) } else { pr_warn!("Thread return work is already in use."); + PushWorkRes::Ok } } @@ -669,8 +669,12 @@ pub(crate) fn push_work_inner( pub(crate) fn push_work_if_looper(&self, work: DLArc) -> BinderResult { let mut inner = self.inner.lock(); if inner.is_looper() && !inner.is_dead { - // Not notifying: Reply to current thread. - let _ = inner.push_work(work); + // Not notifying work_condvar: Reply to current thread. + let res = inner.push_work(work); + drop(inner); + if matches!(res, PushWorkRes::OkNotifyPoll) { + self.process.notify_poll(false); + } Ok(()) } else { drop(inner); @@ -683,7 +687,11 @@ pub(crate) fn push_work_deferred(&self, work: DLArc) { } pub(crate) fn push_return_work(&self, reply: u32) { - self.inner.lock().push_return_work(reply); + // Not notifying work_condvar: Reply to current thread. + let res = self.inner.lock().push_return_work(reply); + if matches!(res, PushWorkRes::OkNotifyPoll) { + self.process.notify_poll(false); + } } pub(crate) fn pop_work_even_if_deferred(&self) -> Option> { @@ -1416,8 +1424,11 @@ fn reply_inner(self: &Arc, info: &mut TransactionInfo) -> BinderResult { let process = orig.from.process.clone(); let allow_fds = orig.flags.contains(TransactionFlag::AcceptFds); let reply = Transaction::new_reply(self, process, info, allow_fds)?; - // Not notifying: Reply to current thread. - let _ = self.inner.lock().push_work(completion); + // Not notifying work_condvar: Reply to current thread. + let res = self.inner.lock().push_work(completion); + if matches!(res, PushWorkRes::OkNotifyPoll) { + self.process.notify_poll(false); + } orig.from.deliver_reply(Ok(reply), &orig, None); Ok(()) })() @@ -1456,10 +1467,15 @@ fn oneway_transaction_inner(self: &Arc, info: &mut TransactionInfo) -> Bin let list_completion = DTRWrap::arc_try_new(DeliverCode::new(code, self.process.task.pid()))?; let completion = list_completion.clone_arc(); - // Not notifying: Reply to current thread. - let _ = self.inner.lock().push_work(list_completion); + // Not notifying work_condvar: Reply to current thread. + let res = self.inner.lock().push_work(list_completion); match transaction.submit(info) { - Ok(()) => Ok(()), + Ok(()) => { + if matches!(res, PushWorkRes::OkNotifyPoll) { + self.process.notify_poll(false); + } + Ok(()) + } Err(err) => { completion.skip(); Err(err) diff --git a/drivers/ata/sata_fsl.c b/drivers/ata/sata_fsl.c index 70b210afd291..c6df55886fe1 100644 --- a/drivers/ata/sata_fsl.c +++ b/drivers/ata/sata_fsl.c @@ -1500,7 +1500,7 @@ static int sata_fsl_probe(struct platform_device *ofdev) host_priv->intr_coalescing.attr.mode = S_IRUGO | S_IWUSR; retval = device_create_file(host->dev, &host_priv->intr_coalescing); if (retval) - goto error_exit_with_cleanup; + goto error_exit_detach; host_priv->rx_watermark.show = fsl_sata_rx_watermark_show; host_priv->rx_watermark.store = fsl_sata_rx_watermark_store; @@ -1510,16 +1510,24 @@ static int sata_fsl_probe(struct platform_device *ofdev) retval = device_create_file(host->dev, &host_priv->rx_watermark); if (retval) { device_remove_file(&ofdev->dev, &host_priv->intr_coalescing); - goto error_exit_with_cleanup; + goto error_exit_detach; } return 0; -error_exit_with_cleanup: +error_exit_detach: + /* + * Once the host has been activated, hcr_base and host_priv are + * released by sata_fsl_host_stop(), which is called by the driver core + * through the ata_host_stop() devres action registered by + * ata_host_start(). Releasing them here as well would result in a + * double iounmap() and a use-after-free. + */ + ata_host_detach(host); - if (host) - ata_host_detach(host); + return retval; +error_exit_with_cleanup: if (hcr_base) iounmap(hcr_base); kfree(host_priv); diff --git a/drivers/base/base.h b/drivers/base/base.h index a5b7abc10ff0..3c36a2128f88 100644 --- a/drivers/base/base.h +++ b/drivers/base/base.h @@ -154,6 +154,7 @@ struct kobject *virtual_device_parent(void); int bus_add_device(struct device *dev); void bus_probe_device(struct device *dev); +void bus_probe_device_deferred(struct device *dev); void bus_remove_device(struct device *dev); void bus_notify(struct device *dev, enum bus_notifier_event value); bool bus_is_registered(const struct bus_type *bus); diff --git a/drivers/base/bus.c b/drivers/base/bus.c index d17bd91490ee..43d131fd8085 100644 --- a/drivers/base/bus.c +++ b/drivers/base/bus.c @@ -603,13 +603,7 @@ int bus_add_device(struct device *dev) return error; } -/** - * bus_probe_device - probe drivers for a new device - * @dev: device to probe - * - * - Automatically probe for a driver if the bus allows it. - */ -void bus_probe_device(struct device *dev) +static void __bus_probe_device(struct device *dev) { struct subsys_private *sp = bus_to_subsys(dev->bus); struct subsys_interface *sif; @@ -627,6 +621,26 @@ void bus_probe_device(struct device *dev) subsys_put(sp); } +/** + * bus_probe_device - probe drivers for a new device + * @dev: device to probe + * + * - Automatically probe for a driver if the bus allows it. + */ +void bus_probe_device(struct device *dev) +{ + __bus_probe_device(dev); +} + +void bus_probe_device_deferred(struct device *dev) +{ + if (dev->type && dev->type->need_parent_lock) + device_lock(dev->parent); + __bus_probe_device(dev); + if (dev->type && dev->type->need_parent_lock) + device_unlock(dev->parent); +} + /** * bus_remove_device - remove device from bus * @dev: device to be removed @@ -849,10 +863,10 @@ static int __must_check bus_rescan_devices_helper(struct device *dev, int ret = 0; if (!dev->driver) { - if (dev->parent && dev->bus->need_parent_lock) + if (dev->type && dev->type->need_parent_lock) device_lock(dev->parent); ret = device_attach(dev); - if (dev->parent && dev->bus->need_parent_lock) + if (dev->type && dev->type->need_parent_lock) device_unlock(dev->parent); } return ret < 0 ? ret : 0; diff --git a/drivers/base/core.c b/drivers/base/core.c index 4c0c373998a1..5378a2579a2c 100644 --- a/drivers/base/core.c +++ b/drivers/base/core.c @@ -2847,7 +2847,9 @@ static ssize_t uevent_store(struct device *dev, const struct device_attribute *a rc = kobject_synth_uevent(&dev->kobj, buf, count); if (rc) { - dev_err(dev, "uevent: failed to send synthetic uevent: %d\n", rc); + dev_err_ratelimited(dev, + "uevent: failed to send synthetic uevent: %d\n", + rc); return rc; } diff --git a/drivers/base/dd.c b/drivers/base/dd.c index f6525a7ee8c5..52123251965d 100644 --- a/drivers/base/dd.c +++ b/drivers/base/dd.c @@ -86,8 +86,9 @@ static void deferred_probe_work_func(struct work_struct *work) /* * This block processes every device in the deferred 'active' list. * Each device is removed from the active list and passed to - * bus_probe_device() to re-attempt the probe. The loop continues - * until every device in the active list is removed and retried. + * bus_probe_device_deferred() to re-attempt the probe. The loop + * continues until every device in the active list is removed and + * retried. * * Note: Once the device is removed from the list and the mutex is * released, it is possible for the device get freed by another thread @@ -121,7 +122,7 @@ static void deferred_probe_work_func(struct work_struct *work) device_pm_move_to_tail(dev); dev_dbg(dev, "Retrying from deferred list\n"); - bus_probe_device(dev); + bus_probe_device_deferred(dev); mutex_lock(&deferred_probe_mutex); put_device(dev); @@ -1029,6 +1030,38 @@ static int __device_attach_driver(struct device_driver *drv, void *_data) return ret == 0; } +/* + * __device_driver_lock - acquire locks needed to manipulate dev->drv + * @dev: Device we will update driver info for + * @parent: Parent device. Needed if the device type requires parent lock + * + * This function will take the required locks for manipulating dev->drv. + * Normally this will just be the @dev lock, but when called for a USB + * interface, @parent lock will be held as well. + */ +static void __device_driver_lock(struct device *dev, struct device *parent) +{ + if (dev->type && dev->type->need_parent_lock && parent) + device_lock(parent); + device_lock(dev); +} + +/* + * __device_driver_unlock - release locks needed to manipulate dev->drv + * @dev: Device we will update driver info for + * @parent: Parent device. Needed if the device type requires parent lock + * + * This function will release the required locks for manipulating dev->drv. + * Normally this will just be the @dev lock, but when called for a + * USB interface, @parent lock will be released as well. + */ +static void __device_driver_unlock(struct device *dev, struct device *parent) +{ + device_unlock(dev); + if (dev->type && dev->type->need_parent_lock && parent) + device_unlock(parent); +} + static void __device_attach_async_helper(void *_dev, async_cookie_t cookie) { struct device *dev = _dev; @@ -1038,7 +1071,7 @@ static void __device_attach_async_helper(void *_dev, async_cookie_t cookie) .want_async = true, }; - device_lock(dev); + __device_driver_lock(dev, dev->parent); /* * Check if device has already been removed or claimed. This may @@ -1060,7 +1093,7 @@ static void __device_attach_async_helper(void *_dev, async_cookie_t cookie) if (dev->parent) pm_runtime_put(dev->parent); out_unlock: - device_unlock(dev); + __device_driver_unlock(dev, dev->parent); put_device(dev); } @@ -1117,8 +1150,11 @@ static int __device_attach(struct device *dev, bool allow_async) } out_unlock: device_unlock(dev); - if (async) - async_schedule_dev(__device_attach_async_helper, dev); + if (async && !async_schedule_dev_nocall(__device_attach_async_helper, dev)) { + put_device(dev); + ret = 2; /* retry synchronously */ + } + return ret; } @@ -1145,48 +1181,20 @@ EXPORT_SYMBOL_GPL(device_attach); void device_initial_probe(struct device *dev) { struct subsys_private *sp = bus_to_subsys(dev->bus); + int ret; if (!sp) return; - if (sp->drivers_autoprobe) - __device_attach(dev, true); + if (sp->drivers_autoprobe) { + ret = __device_attach(dev, true); + if (ret == 2) + __device_attach(dev, false); + } subsys_put(sp); } -/* - * __device_driver_lock - acquire locks needed to manipulate dev->drv - * @dev: Device we will update driver info for - * @parent: Parent device. Needed if the bus requires parent lock - * - * This function will take the required locks for manipulating dev->drv. - * Normally this will just be the @dev lock, but when called for a USB - * interface, @parent lock will be held as well. - */ -static void __device_driver_lock(struct device *dev, struct device *parent) -{ - if (parent && dev->bus->need_parent_lock) - device_lock(parent); - device_lock(dev); -} - -/* - * __device_driver_unlock - release locks needed to manipulate dev->drv - * @dev: Device we will update driver info for - * @parent: Parent device. Needed if the bus requires parent lock - * - * This function will release the required locks for manipulating dev->drv. - * Normally this will just be the @dev lock, but when called for a - * USB interface, @parent lock will be released as well. - */ -static void __device_driver_unlock(struct device *dev, struct device *parent) -{ - device_unlock(dev); - if (parent && dev->bus->need_parent_lock) - device_unlock(parent); -} - /** * device_driver_attach - attach a specific driver to a specific device * @drv: Driver to attach diff --git a/drivers/block/xen-blkfront.c b/drivers/block/xen-blkfront.c index 8dad7bf5f664..86f5dd3aced1 100644 --- a/drivers/block/xen-blkfront.c +++ b/drivers/block/xen-blkfront.c @@ -1210,6 +1210,14 @@ static void blkif_free_ring(struct blkfront_ring_info *rinfo) struct blkfront_info *info = rinfo->dev_info; int i, j, segs; + /* + * Interrupt teardown must be done ahead of freeing queue-related + * data, otherwise the interrupt handler can race with the cleanup. + */ + if (rinfo->irq) + unbind_from_irqhandler(rinfo->irq, rinfo); + rinfo->evtchn = rinfo->irq = 0; + /* * Remove indirect pages, this only happens when using indirect * descriptors but not persistent grants @@ -1292,10 +1300,6 @@ static void blkif_free_ring(struct blkfront_ring_info *rinfo) /* Free resources associated with old device channel. */ xenbus_teardown_ring((void **)&rinfo->ring.sring, info->nr_ring_pages, rinfo->ring_ref); - - if (rinfo->irq) - unbind_from_irqhandler(rinfo->irq, rinfo); - rinfo->evtchn = rinfo->irq = 0; } static void blkif_free(struct blkfront_info *info, int suspend) diff --git a/drivers/bluetooth/hci_vhci.c b/drivers/bluetooth/hci_vhci.c index eddb32b30539..8f7db0102b25 100644 --- a/drivers/bluetooth/hci_vhci.c +++ b/drivers/bluetooth/hci_vhci.c @@ -155,6 +155,7 @@ static ssize_t force_suspend_write(struct file *file, } static const struct file_operations force_suspend_fops = { + .owner = THIS_MODULE, .open = simple_open, .read = force_suspend_read, .write = force_suspend_write, @@ -194,6 +195,7 @@ static ssize_t force_wakeup_write(struct file *file, } static const struct file_operations force_wakeup_fops = { + .owner = THIS_MODULE, .open = simple_open, .read = force_wakeup_read, .write = force_wakeup_write, @@ -263,6 +265,7 @@ static ssize_t aosp_capable_write(struct file *file, } static const struct file_operations aosp_capable_fops = { + .owner = THIS_MODULE, .open = simple_open, .read = aosp_capable_read, .write = aosp_capable_write, @@ -386,6 +389,7 @@ static ssize_t force_devcd_write(struct file *file, const char __user *user_buf, } static const struct file_operations force_devcoredump_fops = { + .owner = THIS_MODULE, .open = simple_open, .write = force_devcd_write, }; diff --git a/drivers/clk/renesas/r9a08g046-cpg.c b/drivers/clk/renesas/r9a08g046-cpg.c index 4488bd1988e8..2e350fa7fce2 100644 --- a/drivers/clk/renesas/r9a08g046-cpg.c +++ b/drivers/clk/renesas/r9a08g046-cpg.c @@ -277,10 +277,12 @@ static const struct cpg_core_clk r9a08g046_core_clks[] __initconst = { DEF_MUX(".sel_rspi0", CLK_SEL_RSPI0, G3L_SEL_RSPI0, sel_rsci_rspi), DEF_MUX(".sel_rspi1", CLK_SEL_RSPI1, G3L_SEL_RSPI1, sel_rsci_rspi), DEF_MUX(".sel_rspi2", CLK_SEL_RSPI2, G3L_SEL_RSPI2, sel_rsci_rspi), - DEF_MUX(".sel_eth0_tx", CLK_SEL_ETH0_TX, G3L_SEL_ETH0_TX, sel_eth0_tx), + DEF_MUX_FLAGS(".sel_eth0_tx", CLK_SEL_ETH0_TX, G3L_SEL_ETH0_TX, sel_eth0_tx, + CLK_SET_RATE_PARENT), DEF_MUX(".sel_eth0_rx", CLK_SEL_ETH0_RX, G3L_SEL_ETH0_RX, sel_eth0_rx), DEF_MUX(".sel_eth0_rm", CLK_SEL_ETH0_RM, G3L_SEL_ETH0_RM, sel_eth0_rm), - DEF_MUX(".sel_eth1_tx", CLK_SEL_ETH1_TX, G3L_SEL_ETH1_TX, sel_eth1_tx), + DEF_MUX_FLAGS(".sel_eth1_tx", CLK_SEL_ETH1_TX, G3L_SEL_ETH1_TX, sel_eth1_tx, + CLK_SET_RATE_PARENT), DEF_MUX(".sel_eth1_rx", CLK_SEL_ETH1_RX, G3L_SEL_ETH1_RX, sel_eth1_rx), DEF_MUX(".sel_eth1_rm", CLK_SEL_ETH1_RM, G3L_SEL_ETH1_RM, sel_eth1_rm), DEF_DIV(".div_eth0_tr", CLK_ETH0_TR, CLK_PLL6, G3L_SDIV_ETH_A, dtable_4_200), diff --git a/drivers/cpufreq/rcpufreq_dt.rs b/drivers/cpufreq/rcpufreq_dt.rs index d7ead60bf8c2..2b1f18b7a1e7 100644 --- a/drivers/cpufreq/rcpufreq_dt.rs +++ b/drivers/cpufreq/rcpufreq_dt.rs @@ -225,4 +225,5 @@ fn probe<'bound>( authors: ["Viresh Kumar "], description: "Generic CPUFreq DT driver", license: "GPL v2", + alias: ["platform:cpufreq-dt"], } diff --git a/drivers/gpu/drm/msm/msm_gem_shrinker.c b/drivers/gpu/drm/msm/msm_gem_shrinker.c index 9d2788f79ace..a626fbb869db 100644 --- a/drivers/gpu/drm/msm/msm_gem_shrinker.c +++ b/drivers/gpu/drm/msm/msm_gem_shrinker.c @@ -51,6 +51,7 @@ with_vm_locks(void (*fn)(struct drm_gem_object *obj), * success paths */ struct drm_gpuvm_bo *vm_bo, *last_locked = NULL; + struct drm_gpuvm_bo *next; bool locked = true; drm_gem_for_each_gpuvm_bo (vm_bo, obj) { @@ -82,7 +83,7 @@ with_vm_locks(void (*fn)(struct drm_gem_object *obj), out_unlock: if (last_locked) { - drm_gem_for_each_gpuvm_bo (vm_bo, obj) { + drm_gem_for_each_gpuvm_bo_safe(vm_bo, next, obj) { struct dma_resv *resv = drm_gpuvm_resv(vm_bo->vm); if (resv == obj->resv) diff --git a/drivers/gpu/drm/panthor/panthor_sched.c b/drivers/gpu/drm/panthor/panthor_sched.c index 653c6755896c..073fdb42ab7c 100644 --- a/drivers/gpu/drm/panthor/panthor_sched.c +++ b/drivers/gpu/drm/panthor/panthor_sched.c @@ -3794,12 +3794,18 @@ int panthor_group_destroy(struct panthor_file *pfile, u32 group_handle) struct panthor_group_pool *gpool = pfile->groups; struct panthor_device *ptdev = pfile->ptdev; struct panthor_scheduler *sched = ptdev->scheduler; - struct panthor_group *group; + struct panthor_group *group = NULL; - if (!xa_get_mark(&gpool->xa, group_handle, GROUP_REGISTERED)) - return -EINVAL; + /* + * Check the mark and erase the entry atomically, so a concurrent + * destroy + create can't make us erase a group that's still being + * initialized and happens to reuse the same handle. + */ + xa_lock(&gpool->xa); + if (xa_get_mark(&gpool->xa, group_handle, GROUP_REGISTERED)) + group = __xa_erase(&gpool->xa, group_handle); + xa_unlock(&gpool->xa); - group = xa_erase(&gpool->xa, group_handle); if (!group) return -EINVAL; diff --git a/drivers/hid/amd-sfh-hid/amd_sfh_client.c b/drivers/hid/amd-sfh-hid/amd_sfh_client.c index ae6add0b9ce3..a68b275f0456 100644 --- a/drivers/hid/amd-sfh-hid/amd_sfh_client.c +++ b/drivers/hid/amd-sfh-hid/amd_sfh_client.c @@ -95,7 +95,7 @@ void amd_sfh_work(struct work_struct *work) cli_data->feature_report[current_index]); if (report_size) hid_input_report(cli_data->hid_sensor_hubs[current_index], - cli_data->report_type[current_index], + node_type, cli_data->feature_report[current_index], report_size, 0); else pr_err("AMDSFH: Invalid report size\n"); @@ -104,7 +104,7 @@ void amd_sfh_work(struct work_struct *work) report_size = mp2_ops->get_in_rep(current_index, sensor_index, report_id, in_data); if (report_size) hid_input_report(cli_data->hid_sensor_hubs[current_index], - cli_data->report_type[current_index], + node_type, in_data->input_report[current_index], report_size, 0); else pr_err("AMDSFH: Invalid report size\n"); diff --git a/drivers/hid/amd-sfh-hid/hid_descriptor/amd_sfh_hid_desc.h b/drivers/hid/amd-sfh-hid/hid_descriptor/amd_sfh_hid_desc.h index 882434b1501f..230fb69cacfd 100644 --- a/drivers/hid/amd-sfh-hid/hid_descriptor/amd_sfh_hid_desc.h +++ b/drivers/hid/amd-sfh-hid/hid_descriptor/amd_sfh_hid_desc.h @@ -48,8 +48,6 @@ struct accel3_input_report { int in_accel_x_value; int in_accel_y_value; int in_accel_z_value; - /* include if required to support the "shake" event */ - u8 in_accel_shake_detection; } __packed; struct gyro_feature_report { diff --git a/drivers/hid/amd-sfh-hid/hid_descriptor/amd_sfh_hid_report_desc.h b/drivers/hid/amd-sfh-hid/hid_descriptor/amd_sfh_hid_report_desc.h index 67ec2d6a417d..64cff842e5f2 100644 --- a/drivers/hid/amd-sfh-hid/hid_descriptor/amd_sfh_hid_report_desc.h +++ b/drivers/hid/amd-sfh-hid/hid_descriptor/amd_sfh_hid_report_desc.h @@ -169,13 +169,6 @@ static const u8 accel3_report_descriptor[] = { 0x95, 1, /* HID report count (1) */ 0x55, 0x0E, /* HID unit exponent(0x0E) */ 0X81, 0x02, /* HID Input (Data_Arr_Abs) */ - -0x0A, 0x51, 0x04, /* HID usage sensor data motion state */ -0x15, 0, /* HID logical Min_8(0) False = Still*/ -0x25, 1, /* HID logical Min_8(1) True = In motion */ -0x75, 8, /* HID report size(8) */ -0x95, 1, /* HID report count (1) */ -0X81, 0x02, /* HID Input (Data_Arr_Abs) */ 0xC0 /* HID end collection */ }; diff --git a/drivers/hid/hid-asus.c b/drivers/hid/hid-asus.c index c38a2b3b33d9..7dc6417fe262 100644 --- a/drivers/hid/hid-asus.c +++ b/drivers/hid/hid-asus.c @@ -1493,12 +1493,14 @@ static int asus_probe(struct hid_device *hdev, const struct hid_device_id *id) return ret; } - for (int r = 0; r < ARRAY_SIZE(asus_report_id_init); r++) { - if (asus_has_report_id(hdev, asus_report_id_init[r])) { - ret = asus_kbd_init(hdev, asus_report_id_init[r]); - if (ret < 0) - hid_warn(hdev, "Failed to initialize 0x%x: %d.\n", - asus_report_id_init[r], ret); + if (!drvdata->tp) { + for (int r = 0; r < ARRAY_SIZE(asus_report_id_init); r++) { + if (asus_has_report_id(hdev, asus_report_id_init[r])) { + ret = asus_kbd_init(hdev, asus_report_id_init[r]); + if (ret < 0) + hid_warn(hdev, "Failed to initialize 0x%x: %d.\n", + asus_report_id_init[r], ret); + } } } diff --git a/drivers/hid/hid-core.c b/drivers/hid/hid-core.c index a3ff0514f9cd..a71ddaa810a0 100644 --- a/drivers/hid/hid-core.c +++ b/drivers/hid/hid-core.c @@ -71,8 +71,8 @@ static u32 s32ton(__s32 value, unsigned int n) if (!value || !n) return 0; - if (n > 32) - n = 32; + if (n >= 32) + return (u32) value; a = value >> (n - 1); if (a && a != -1) diff --git a/drivers/hid/hid-cougar.c b/drivers/hid/hid-cougar.c index ad027c45f162..7156658166f5 100644 --- a/drivers/hid/hid-cougar.c +++ b/drivers/hid/hid-cougar.c @@ -270,6 +270,9 @@ static int cougar_raw_event(struct hid_device *hdev, struct hid_report *report, if (!shared->enabled || !shared->input) return -EPERM; + if (size <= COUGAR_FIELD_ACTION) + return -EPERM; + code = data[COUGAR_FIELD_CODE]; action = data[COUGAR_FIELD_ACTION]; for (i = 0; cougar_mapping[i][0]; i++) { diff --git a/drivers/hid/hid-cp2112.c b/drivers/hid/hid-cp2112.c index 04379db93571..6b938f75e72f 100644 --- a/drivers/hid/hid-cp2112.c +++ b/drivers/hid/hid-cp2112.c @@ -1430,6 +1430,9 @@ static int cp2112_raw_event(struct hid_device *hdev, struct hid_report *report, switch (data[0]) { case CP2112_TRANSFER_STATUS_RESPONSE: + if (size < sizeof(*xfer)) + return 0; + hid_dbg(hdev, "xfer status: %02x %02x %04x %04x\n", xfer->status0, xfer->status1, be16_to_cpu(xfer->retries), be16_to_cpu(xfer->length)); @@ -1463,11 +1466,16 @@ static int cp2112_raw_event(struct hid_device *hdev, struct hid_report *report, atomic_set(&dev->xfer_avail, 1); break; case CP2112_DATA_READ_RESPONSE: + if (size < 3) + return 0; + hid_dbg(hdev, "read response: %02x %02x\n", data[1], data[2]); dev->read_length = data[2]; if (dev->read_length > sizeof(dev->read_data)) dev->read_length = sizeof(dev->read_data); + if (dev->read_length > size - 3) + return 0; memcpy(dev->read_data, &data[3], dev->read_length); atomic_set(&dev->read_avail, 1); @@ -1494,4 +1502,3 @@ module_hid_driver(cp2112_driver); MODULE_DESCRIPTION("Silicon Labs HID USB to SMBus master bridge"); MODULE_AUTHOR("David Barksdale "); MODULE_LICENSE("GPL"); - diff --git a/drivers/hid/hid-elo.c b/drivers/hid/hid-elo.c index b8f5f3eb53a4..86e8729e1975 100644 --- a/drivers/hid/hid-elo.c +++ b/drivers/hid/hid-elo.c @@ -85,6 +85,9 @@ static int elo_raw_event(struct hid_device *hdev, struct hid_report *report, if (!(hdev->claimed & HID_CLAIMED_INPUT) || list_empty(&hdev->inputs)) return 0; + if (size < ELO_SMARTSET_PACKET_SIZE) + return 0; + hidinput = list_first_entry(&hdev->inputs, struct hid_input, list); switch (report->id) { diff --git a/drivers/hid/hid-ids.h b/drivers/hid/hid-ids.h index cb7f16d75fce..3b19b742d4dd 100644 --- a/drivers/hid/hid-ids.h +++ b/drivers/hid/hid-ids.h @@ -747,6 +747,7 @@ #define USB_DEVICE_ID_HOLTEK_ALT_MOUSE_A081 0xa081 #define USB_DEVICE_ID_HOLTEK_ALT_MOUSE_A0C2 0xa0c2 #define USB_DEVICE_ID_HOLTEK_ALT_KEYBOARD_A096 0xa096 +#define USB_DEVICE_ID_HOLTEK_ALT_KEYBOARD_A292 0xa292 #define USB_DEVICE_ID_HOLTEK_ALT_KEYBOARD_A293 0xa293 #define USB_VENDOR_ID_IMATION 0x0718 diff --git a/drivers/hid/hid-lenovo-go-s.c b/drivers/hid/hid-lenovo-go-s.c index a72f7f748cb5..7102786ce124 100644 --- a/drivers/hid/hid-lenovo-go-s.c +++ b/drivers/hid/hid-lenovo-go-s.c @@ -10,6 +10,7 @@ #include #include #include +#include #include #include #include @@ -32,33 +33,42 @@ #define GO_S_CFG_INTF_IN 0x84 #define GO_S_PACKET_SIZE 64 -static struct hid_gos_cfg { - struct delayed_work gos_cfg_setup; +struct gos_cfg_drvdata { + /* MCU General Variables */ struct completion send_cmd_complete; - struct led_classdev *led_cdev; + struct delayed_work gos_cfg_setup; struct hid_device *hdev; + bool orphan_ack_pending; struct mutex cfg_mutex; /*ensure single synchronous output report*/ + int cmd_status; + + /* Gamepad Variables */ u8 gp_auto_sleep_time; - u8 gp_dpad_mode; - u8 gp_mode; - u8 gp_poll_rate; - u8 imu_bypass_en; u8 imu_manufacturer; + bool gp_registered; + u8 tp_windows_mode; + u8 tp_manufacturer; + u8 tp_linux_mode; + u8 imu_bypass_en; u8 imu_sensor_en; + u8 gp_dpad_mode; + u8 gp_poll_rate; u8 mcu_id[12]; u8 mouse_step; + u8 tp_version; + u8 gp_mode; u8 os_mode; - u8 rgb_effect; - u8 rgb_en; - u8 rgb_mode; + u8 tp_en; + + /* RGB Variables */ + struct led_classdev_mc led_mc; + bool rgb_registered; u8 rgb_profile; + u8 rgb_effect; u8 rgb_speed; - u8 tp_en; - u8 tp_linux_mode; - u8 tp_windows_mode; - u8 tp_version; - u8 tp_manufacturer; -} drvdata; + u8 rgb_mode; + u8 rgb_en; +}; struct gos_cfg_attr { u8 index; @@ -234,56 +244,57 @@ static const char *const imu_manufacturer_text[] = { [IMU_ST] = "ST", }; -static int hid_gos_version_event(u8 *data) +static int hid_gos_version_event(struct gos_cfg_drvdata *drvdata, u8 *data) { struct version_report *ver_rep = (struct version_report *)data; - drvdata.hdev->firmware_version = get_unaligned_le32(&ver_rep->version); + drvdata->hdev->firmware_version = get_unaligned_le32(&ver_rep->version); return 0; } -static int hid_gos_mcu_id_event(struct command_report *cmd_rep) +static int hid_gos_mcu_id_event(struct gos_cfg_drvdata *drvdata, struct command_report *cmd_rep) { - drvdata.mcu_id[0] = cmd_rep->sub_cmd; - memcpy(&drvdata.mcu_id[1], cmd_rep->data, 11); + drvdata->mcu_id[0] = cmd_rep->sub_cmd; + memcpy(&drvdata->mcu_id[1], cmd_rep->data, 11); return 0; } -static int hid_gos_gamepad_cfg_event(struct command_report *cmd_rep) +static int hid_gos_gamepad_cfg_event(struct gos_cfg_drvdata *drvdata, + struct command_report *cmd_rep) { int ret = 0; switch (cmd_rep->sub_cmd) { case FEATURE_GAMEPAD_MODE: - drvdata.gp_mode = cmd_rep->data[0]; + drvdata->gp_mode = cmd_rep->data[0]; break; case FEATURE_AUTO_SLEEP_TIME: - drvdata.gp_auto_sleep_time = cmd_rep->data[0]; + drvdata->gp_auto_sleep_time = cmd_rep->data[0]; break; case FEATURE_IMU_BYPASS: - drvdata.imu_bypass_en = cmd_rep->data[0]; + drvdata->imu_bypass_en = cmd_rep->data[0]; break; case FEATURE_RGB_ENABLE: - drvdata.rgb_en = cmd_rep->data[0]; + drvdata->rgb_en = cmd_rep->data[0]; break; case FEATURE_IMU_ENABLE: - drvdata.imu_sensor_en = cmd_rep->data[0]; + drvdata->imu_sensor_en = cmd_rep->data[0]; break; case FEATURE_TOUCHPAD_ENABLE: - drvdata.tp_en = cmd_rep->data[0]; + drvdata->tp_en = cmd_rep->data[0]; break; case FEATURE_OS_MODE: - drvdata.os_mode = cmd_rep->data[0]; + drvdata->os_mode = cmd_rep->data[0]; break; case FEATURE_POLL_RATE: - drvdata.gp_poll_rate = cmd_rep->data[0]; + drvdata->gp_poll_rate = cmd_rep->data[0]; break; case FEATURE_DPAD_MODE: - drvdata.gp_dpad_mode = cmd_rep->data[0]; + drvdata->gp_dpad_mode = cmd_rep->data[0]; break; case FEATURE_MOUSE_WHEEL_STEP: - drvdata.mouse_step = cmd_rep->data[0]; + drvdata->mouse_step = cmd_rep->data[0]; break; default: ret = -EINVAL; @@ -293,16 +304,16 @@ static int hid_gos_gamepad_cfg_event(struct command_report *cmd_rep) return ret; } -static int hid_gos_touchpad_event(struct command_report *cmd_rep) +static int hid_gos_touchpad_event(struct gos_cfg_drvdata *drvdata, struct command_report *cmd_rep) { int ret = 0; switch (cmd_rep->sub_cmd) { case CFG_LINUX_MODE: - drvdata.tp_linux_mode = cmd_rep->data[0]; + drvdata->tp_linux_mode = cmd_rep->data[0]; break; case CFG_WINDOWS_MODE: - drvdata.tp_windows_mode = cmd_rep->data[0]; + drvdata->tp_windows_mode = cmd_rep->data[0]; break; default: ret = -EINVAL; @@ -312,21 +323,21 @@ static int hid_gos_touchpad_event(struct command_report *cmd_rep) return ret; } -static int hid_gos_pl_test_event(struct command_report *cmd_rep) +static int hid_gos_pl_test_event(struct gos_cfg_drvdata *drvdata, struct command_report *cmd_rep) { int ret = 0; switch (cmd_rep->sub_cmd) { case TEST_TP_MFR: - drvdata.tp_manufacturer = cmd_rep->data[0]; + drvdata->tp_manufacturer = cmd_rep->data[0]; ret = 0; break; case TEST_IMU_MFR: - drvdata.imu_manufacturer = cmd_rep->data[0]; + drvdata->imu_manufacturer = cmd_rep->data[0]; ret = 0; break; case TEST_TP_VER: - drvdata.tp_version = cmd_rep->data[0]; + drvdata->tp_version = cmd_rep->data[0]; ret = 0; break; default: @@ -336,30 +347,30 @@ static int hid_gos_pl_test_event(struct command_report *cmd_rep) return ret; } -static int hid_gos_light_event(struct command_report *cmd_rep) +static int hid_gos_light_event(struct gos_cfg_drvdata *drvdata, struct command_report *cmd_rep) { - struct led_classdev_mc *mc_cdev; + struct led_classdev_mc *led_mc; int ret = 0; switch (cmd_rep->sub_cmd) { case LIGHT_MODE_SEL: - drvdata.rgb_mode = cmd_rep->data[0]; + drvdata->rgb_mode = cmd_rep->data[0]; ret = 0; break; case LIGHT_PROFILE_SEL: - drvdata.rgb_profile = cmd_rep->data[0]; + drvdata->rgb_profile = cmd_rep->data[0]; ret = 0; break; case USR_LIGHT_PROFILE_1: case USR_LIGHT_PROFILE_2: case USR_LIGHT_PROFILE_3: - mc_cdev = lcdev_to_mccdev(drvdata.led_cdev); - drvdata.rgb_effect = cmd_rep->data[0]; - mc_cdev->subled_info[0].intensity = cmd_rep->data[1]; - mc_cdev->subled_info[1].intensity = cmd_rep->data[2]; - mc_cdev->subled_info[2].intensity = cmd_rep->data[3]; - drvdata.led_cdev->brightness = cmd_rep->data[4]; - drvdata.rgb_speed = cmd_rep->data[5]; + led_mc = &drvdata->led_mc; + drvdata->rgb_effect = cmd_rep->data[0]; + led_mc->subled_info[0].intensity = cmd_rep->data[1]; + led_mc->subled_info[1].intensity = cmd_rep->data[2]; + led_mc->subled_info[2].intensity = cmd_rep->data[3]; + led_mc->led_cdev.brightness = cmd_rep->data[4]; + drvdata->rgb_speed = cmd_rep->data[5]; ret = 0; break; default: @@ -392,6 +403,7 @@ static int get_endpoint_address(struct hid_device *hdev) static int hid_gos_raw_event(struct hid_device *hdev, struct hid_report *report, u8 *data, int size) { + struct gos_cfg_drvdata *drvdata = hid_get_drvdata(hdev); struct command_report *cmd_rep; int ep, ret; @@ -399,6 +411,9 @@ static int hid_gos_raw_event(struct hid_device *hdev, struct hid_report *report, if (ep != GO_S_CFG_INTF_IN) return 0; + if (!drvdata) + return -ENODEV; + if (size != GO_S_PACKET_SIZE) return -EINVAL; @@ -406,22 +421,22 @@ static int hid_gos_raw_event(struct hid_device *hdev, struct hid_report *report, switch (cmd_rep->cmd) { case GET_VERSION: - ret = hid_gos_version_event(data); + ret = hid_gos_version_event(drvdata, data); break; case GET_MCU_ID: - ret = hid_gos_mcu_id_event(cmd_rep); + ret = hid_gos_mcu_id_event(drvdata, cmd_rep); break; case GET_GAMEPAD_CFG: - ret = hid_gos_gamepad_cfg_event(cmd_rep); + ret = hid_gos_gamepad_cfg_event(drvdata, cmd_rep); break; case GET_TP_PARAM: - ret = hid_gos_touchpad_event(cmd_rep); + ret = hid_gos_touchpad_event(drvdata, cmd_rep); break; case GET_PL_TEST: - ret = hid_gos_pl_test_event(cmd_rep); + ret = hid_gos_pl_test_event(drvdata, cmd_rep); break; case GET_RGB_CFG: - ret = hid_gos_light_event(cmd_rep); + ret = hid_gos_light_event(drvdata, cmd_rep); break; case SET_GAMEPAD_CFG: case SET_RGB_CFG: @@ -435,22 +450,41 @@ static int hid_gos_raw_event(struct hid_device *hdev, struct hid_report *report, dev_dbg(&hdev->dev, "Rx data as raw input report: [%*ph]\n", GO_S_PACKET_SIZE, data); - complete(&drvdata.send_cmd_complete); + drvdata->cmd_status = ret; + complete(&drvdata->send_cmd_complete); + return ret; } static int mcu_property_out(struct hid_device *hdev, u8 command, u8 index, u8 *data, size_t len) { + struct gos_cfg_drvdata *drvdata = hid_get_drvdata(hdev); unsigned char *dmabuf __free(kfree) = NULL; u8 header[] = { command, index }; size_t header_size = ARRAY_SIZE(header); int timeout, ret; + if (!drvdata) + return -ENODEV; + if (header_size + len > GO_S_PACKET_SIZE) return -EINVAL; - guard(mutex)(&drvdata.cfg_mutex); + guard(mutex)(&drvdata->cfg_mutex); + + /* + * A reply to the previous command may still be in flight. Give it a + * short window to arrive and be consumed before this call reinits the + * completion, so a late reply can't be mistaken for this command's. + */ + if (drvdata->orphan_ack_pending) { + wait_for_completion_timeout(&drvdata->send_cmd_complete, msecs_to_jiffies(25)); + drvdata->orphan_ack_pending = false; + drvdata->cmd_status = -ETIMEDOUT; + } + reinit_completion(&drvdata->send_cmd_complete); + /* We can't use a devm_alloc reusable buffer without side effects during suspend */ dmabuf = kzalloc(GO_S_PACKET_SIZE, GFP_KERNEL); if (!dmabuf) @@ -472,14 +506,14 @@ static int mcu_property_out(struct hid_device *hdev, u8 command, u8 index, /* PL_TEST commands can take longer because they go out to another device */ timeout = (command == GET_PL_TEST) ? 200 : 5; - ret = wait_for_completion_interruptible_timeout(&drvdata.send_cmd_complete, + ret = wait_for_completion_interruptible_timeout(&drvdata->send_cmd_complete, msecs_to_jiffies(timeout)); + ret = ret > 0 ? drvdata->cmd_status : ret ?: -EBUSY; - if (ret == 0) /* timeout occurred */ - ret = -EBUSY; + if (ret) + drvdata->orphan_ack_pending = true; - reinit_completion(&drvdata.send_cmd_complete); - return 0; + return ret; } static ssize_t gamepad_property_store(struct device *dev, @@ -487,10 +521,34 @@ static ssize_t gamepad_property_store(struct device *dev, const char *buf, size_t count, enum feature_status_index index) { + struct gos_cfg_drvdata *drvdata; + struct led_classdev_mc *led_mc; + struct led_classdev *led_cdev; + struct hid_device *hdev; + bool dev_registered; size_t size = 1; u8 val = 0; int ret; + /* rgb_enabled is attached to led_classdev, not hid_device */ + if (index == FEATURE_RGB_ENABLE) { + led_cdev = dev_get_drvdata(dev); + led_mc = lcdev_to_mccdev(led_cdev); + drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); + hdev = drvdata->hdev; + /* Pairs with smp_store_release from gos_cfg_setup */ + dev_registered = smp_load_acquire(&drvdata->rgb_registered); + + } else { + hdev = to_hid_device(dev); + drvdata = hid_get_drvdata(hdev); + /* Pairs with smp_store_release from gos_cfg_setup */ + dev_registered = smp_load_acquire(&drvdata->gp_registered); + } + + if (!dev_registered) + return -ENODEV; + switch (index) { case FEATURE_GAMEPAD_MODE: ret = sysfs_match_string(gamepad_mode_text, buf); @@ -559,7 +617,7 @@ static ssize_t gamepad_property_store(struct device *dev, if (!val) size = 0; - ret = mcu_property_out(drvdata.hdev, SET_GAMEPAD_CFG, index, &val, + ret = mcu_property_out(hdev, SET_GAMEPAD_CFG, index, &val, size); if (ret < 0) return ret; @@ -571,67 +629,92 @@ static ssize_t gamepad_property_show(struct device *dev, struct device_attribute *attr, char *buf, enum feature_status_index index) { + struct gos_cfg_drvdata *drvdata; + struct led_classdev_mc *led_mc; + struct led_classdev *led_cdev; + struct hid_device *hdev; + bool dev_registered; + ssize_t count = 0; u8 i; - count = mcu_property_out(drvdata.hdev, GET_GAMEPAD_CFG, index, NULL, 0); + /* rgb_enabled is attached to led_classdev, not hid_device */ + if (index == FEATURE_RGB_ENABLE) { + led_cdev = dev_get_drvdata(dev); + led_mc = lcdev_to_mccdev(led_cdev); + drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); + hdev = drvdata->hdev; + /* Pairs with smp_store_release from gos_cfg_setup */ + dev_registered = smp_load_acquire(&drvdata->rgb_registered); + + } else { + hdev = to_hid_device(dev); + drvdata = hid_get_drvdata(hdev); + /* Pairs with smp_store_release from gos_cfg_setup */ + dev_registered = smp_load_acquire(&drvdata->gp_registered); + } + + if (!dev_registered) + return -ENODEV; + + count = mcu_property_out(hdev, GET_GAMEPAD_CFG, index, NULL, 0); if (count < 0) return count; switch (index) { case FEATURE_GAMEPAD_MODE: - i = drvdata.gp_mode; + i = drvdata->gp_mode; if (i >= ARRAY_SIZE(gamepad_mode_text)) return -EINVAL; count = sysfs_emit(buf, "%s\n", gamepad_mode_text[i]); break; case FEATURE_AUTO_SLEEP_TIME: - count = sysfs_emit(buf, "%u\n", drvdata.gp_auto_sleep_time); + count = sysfs_emit(buf, "%u\n", drvdata->gp_auto_sleep_time); break; case FEATURE_IMU_ENABLE: - i = drvdata.imu_sensor_en; + i = drvdata->imu_sensor_en; if (i >= ARRAY_SIZE(feature_enabled_text)) return -EINVAL; count = sysfs_emit(buf, "%s\n", feature_enabled_text[i]); break; case FEATURE_IMU_BYPASS: - i = drvdata.imu_bypass_en; + i = drvdata->imu_bypass_en; if (i >= ARRAY_SIZE(feature_enabled_text)) return -EINVAL; count = sysfs_emit(buf, "%s\n", feature_enabled_text[i]); break; case FEATURE_RGB_ENABLE: - i = drvdata.rgb_en; + i = drvdata->rgb_en; if (i >= ARRAY_SIZE(feature_enabled_text)) return -EINVAL; count = sysfs_emit(buf, "%s\n", feature_enabled_text[i]); break; case FEATURE_TOUCHPAD_ENABLE: - i = drvdata.tp_en; + i = drvdata->tp_en; if (i >= ARRAY_SIZE(feature_enabled_text)) return -EINVAL; count = sysfs_emit(buf, "%s\n", feature_enabled_text[i]); break; case FEATURE_OS_MODE: - i = drvdata.os_mode; + i = drvdata->os_mode; if (i >= ARRAY_SIZE(os_type_text)) return -EINVAL; count = sysfs_emit(buf, "%s\n", os_type_text[i]); break; case FEATURE_POLL_RATE: - i = drvdata.gp_poll_rate; + i = drvdata->gp_poll_rate; if (i >= ARRAY_SIZE(poll_rate_text)) return -EINVAL; count = sysfs_emit(buf, "%s\n", poll_rate_text[i]); break; case FEATURE_DPAD_MODE: - i = drvdata.gp_dpad_mode; + i = drvdata->gp_dpad_mode; if (i >= ARRAY_SIZE(dpad_mode_text)) return -EINVAL; count = sysfs_emit(buf, "%s\n", dpad_mode_text[i]); break; case FEATURE_MOUSE_WHEEL_STEP: - i = drvdata.mouse_step; + i = drvdata->mouse_step; if (i < 1 || i > 127) return -EINVAL; count = sysfs_emit(buf, "%u\n", i); @@ -709,10 +792,21 @@ static ssize_t touchpad_property_store(struct device *dev, const char *buf, size_t count, enum touchpad_config_index index) { + struct hid_device *hdev = to_hid_device(dev); + struct gos_cfg_drvdata *drvdata = hid_get_drvdata(hdev); + bool gp_registered; size_t size = 1; u8 val = 0; int ret; + if (!drvdata) + return -ENODEV; + +/* Pairs with smp_store_release from gos_cfg_setup */ + gp_registered = smp_load_acquire(&drvdata->gp_registered); + if (!gp_registered) + return -ENODEV; + switch (index) { case CFG_WINDOWS_MODE: ret = sysfs_match_string(touchpad_mode_text, buf); @@ -732,7 +826,7 @@ static ssize_t touchpad_property_store(struct device *dev, if (!val) size = 0; - ret = mcu_property_out(drvdata.hdev, SET_TP_PARAM, index, &val, size); + ret = mcu_property_out(hdev, SET_TP_PARAM, index, &val, size); if (ret < 0) return ret; @@ -743,19 +837,30 @@ static ssize_t touchpad_property_show(struct device *dev, struct device_attribute *attr, char *buf, enum touchpad_config_index index) { + struct hid_device *hdev = to_hid_device(dev); + struct gos_cfg_drvdata *drvdata = hid_get_drvdata(hdev); + bool gp_registered; int ret = 0; u8 i; - ret = mcu_property_out(drvdata.hdev, GET_TP_PARAM, index, NULL, 0); + if (!drvdata) + return -ENODEV; + +/* Pairs with smp_store_release from gos_cfg_setup */ + gp_registered = smp_load_acquire(&drvdata->gp_registered); + if (!gp_registered) + return -ENODEV; + + ret = mcu_property_out(hdev, GET_TP_PARAM, index, NULL, 0); if (ret < 0) return ret; switch (index) { case CFG_WINDOWS_MODE: - i = drvdata.tp_windows_mode; + i = drvdata->tp_windows_mode; break; case CFG_LINUX_MODE: - i = drvdata.tp_linux_mode; + i = drvdata->tp_linux_mode; break; default: return -EINVAL; @@ -797,24 +902,35 @@ static ssize_t test_property_show(struct device *dev, struct device_attribute *attr, char *buf, enum test_command_index index) { + struct hid_device *hdev = to_hid_device(dev); + struct gos_cfg_drvdata *drvdata = hid_get_drvdata(hdev); + bool gp_registered; size_t count = 0; u8 i; + if (!drvdata) + return -ENODEV; + +/* Pairs with smp_store_release from gos_cfg_setup */ + gp_registered = smp_load_acquire(&drvdata->gp_registered); + if (!gp_registered) + return -ENODEV; + switch (index) { case TEST_TP_MFR: - i = drvdata.tp_manufacturer; + i = drvdata->tp_manufacturer; if (i >= ARRAY_SIZE(touchpad_manufacturer_text)) return -EINVAL; count = sysfs_emit(buf, "%s\n", touchpad_manufacturer_text[i]); break; case TEST_IMU_MFR: - i = drvdata.imu_manufacturer; + i = drvdata->imu_manufacturer; if (i >= ARRAY_SIZE(imu_manufacturer_text)) return -EINVAL; count = sysfs_emit(buf, "%s\n", imu_manufacturer_text[i]); break; case TEST_TP_VER: - count = sysfs_emit(buf, "%u\n", drvdata.tp_version); + count = sysfs_emit(buf, "%u\n", drvdata->tp_version); break; default: count = -EINVAL; @@ -827,7 +943,13 @@ static ssize_t test_property_show(struct device *dev, static ssize_t mcu_id_show(struct device *dev, struct device_attribute *attr, char *buf) { - return sysfs_emit(buf, "%*phN\n", 12, &drvdata.mcu_id); + struct hid_device *hdev = to_hid_device(dev); + struct gos_cfg_drvdata *drvdata = hid_get_drvdata(hdev); + + if (!drvdata) + return -ENODEV; + + return sysfs_emit(buf, "%*phN\n", 12, &drvdata->mcu_id); } static int rgb_cfg_call(struct hid_device *hdev, enum mcu_command_index cmd, @@ -842,58 +964,75 @@ static int rgb_cfg_call(struct hid_device *hdev, enum mcu_command_index cmd, return mcu_property_out(hdev, cmd, index, val, size); } -static int rgb_attr_show(void) +static int rgb_attr_show(struct gos_cfg_drvdata *drvdata) { enum rgb_config_index index; - index = drvdata.rgb_profile + 2; + index = drvdata->rgb_profile + 2; - return rgb_cfg_call(drvdata.hdev, GET_RGB_CFG, index, NULL, 0); + return rgb_cfg_call(drvdata->hdev, GET_RGB_CFG, index, NULL, 0); }; static ssize_t rgb_effect_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) { - struct led_classdev_mc *mc_cdev = lcdev_to_mccdev(drvdata.led_cdev); + struct led_classdev *led_cdev = dev_get_drvdata(dev); + struct led_classdev_mc *led_mc = lcdev_to_mccdev(led_cdev); + struct gos_cfg_drvdata *drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); enum rgb_config_index index; + bool rgb_registered; u8 effect; int ret; +/* Pairs with smp_store_release from gos_cfg_setup */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (!rgb_registered) + return -ENODEV; + ret = sysfs_match_string(rgb_effect_text, buf); if (ret < 0) return ret; effect = ret; - index = drvdata.rgb_profile + 2; + index = drvdata->rgb_profile + 2; u8 rgb_profile[6] = { effect, - mc_cdev->subled_info[0].intensity, - mc_cdev->subled_info[1].intensity, - mc_cdev->subled_info[2].intensity, - drvdata.led_cdev->brightness, - drvdata.rgb_speed }; + led_mc->subled_info[0].intensity, + led_mc->subled_info[1].intensity, + led_mc->subled_info[2].intensity, + led_mc->led_cdev.brightness, + drvdata->rgb_speed }; - ret = rgb_cfg_call(drvdata.hdev, SET_RGB_CFG, index, rgb_profile, 6); + ret = rgb_cfg_call(drvdata->hdev, SET_RGB_CFG, index, rgb_profile, 6); if (ret) return ret; - drvdata.rgb_effect = effect; + drvdata->rgb_effect = effect; return count; }; static ssize_t rgb_effect_show(struct device *dev, struct device_attribute *attr, char *buf) { + struct led_classdev *led_cdev = dev_get_drvdata(dev); + struct led_classdev_mc *led_mc = lcdev_to_mccdev(led_cdev); + struct gos_cfg_drvdata *drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); + bool rgb_registered; int ret; - ret = rgb_attr_show(); +/* Pairs with smp_store_release from gos_cfg_setup */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (!rgb_registered) + return -ENODEV; + + ret = rgb_attr_show(drvdata); if (ret) return ret; - if (drvdata.rgb_effect >= ARRAY_SIZE(rgb_effect_text)) + if (drvdata->rgb_effect >= ARRAY_SIZE(rgb_effect_text)) return -EINVAL; - return sysfs_emit(buf, "%s\n", rgb_effect_text[drvdata.rgb_effect]); + return sysfs_emit(buf, "%s\n", rgb_effect_text[drvdata->rgb_effect]); } static ssize_t rgb_effect_index_show(struct device *dev, @@ -915,11 +1054,19 @@ static ssize_t rgb_speed_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) { - struct led_classdev_mc *mc_cdev = lcdev_to_mccdev(drvdata.led_cdev); + struct led_classdev *led_cdev = dev_get_drvdata(dev); + struct led_classdev_mc *led_mc = lcdev_to_mccdev(led_cdev); + struct gos_cfg_drvdata *drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); enum rgb_config_index index; + bool rgb_registered; int val = 0; int ret; +/* Pairs with smp_store_release from gos_cfg_setup */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (!rgb_registered) + return -ENODEV; + ret = kstrtoint(buf, 10, &val); if (ret) return ret; @@ -927,19 +1074,19 @@ static ssize_t rgb_speed_store(struct device *dev, if (val < 0 || val > 100) return -EINVAL; - index = drvdata.rgb_profile + 2; - u8 rgb_profile[6] = { drvdata.rgb_effect, - mc_cdev->subled_info[0].intensity, - mc_cdev->subled_info[1].intensity, - mc_cdev->subled_info[2].intensity, - drvdata.led_cdev->brightness, + index = drvdata->rgb_profile + 2; + u8 rgb_profile[6] = { drvdata->rgb_effect, + led_mc->subled_info[0].intensity, + led_mc->subled_info[1].intensity, + led_mc->subled_info[2].intensity, + led_mc->led_cdev.brightness, val }; - ret = rgb_cfg_call(drvdata.hdev, SET_RGB_CFG, index, rgb_profile, 6); + ret = rgb_cfg_call(drvdata->hdev, SET_RGB_CFG, index, rgb_profile, 6); if (ret) return ret; - drvdata.rgb_speed = val; + drvdata->rgb_speed = val; return count; }; @@ -947,16 +1094,25 @@ static ssize_t rgb_speed_store(struct device *dev, static ssize_t rgb_speed_show(struct device *dev, struct device_attribute *attr, char *buf) { + struct led_classdev *led_cdev = dev_get_drvdata(dev); + struct led_classdev_mc *led_mc = lcdev_to_mccdev(led_cdev); + struct gos_cfg_drvdata *drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); + bool rgb_registered; int ret; - ret = rgb_attr_show(); +/* Pairs with smp_store_release from gos_cfg_setup */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (!rgb_registered) + return -ENODEV; + + ret = rgb_attr_show(drvdata); if (ret) return ret; - if (drvdata.rgb_speed > 100) + if (drvdata->rgb_speed > 100) return -EINVAL; - return sysfs_emit(buf, "%hhu\n", drvdata.rgb_speed); + return sysfs_emit(buf, "%hhu\n", drvdata->rgb_speed); } static ssize_t rgb_speed_range_show(struct device *dev, @@ -968,21 +1124,30 @@ static ssize_t rgb_speed_range_show(struct device *dev, static ssize_t rgb_mode_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) { + struct led_classdev *led_cdev = dev_get_drvdata(dev); + struct led_classdev_mc *led_mc = lcdev_to_mccdev(led_cdev); + struct gos_cfg_drvdata *drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); + bool rgb_registered; int ret; u8 val; +/* Pairs with smp_store_release from gos_cfg_setup */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (!rgb_registered) + return -ENODEV; + ret = sysfs_match_string(rgb_mode_text, buf); if (ret <= 0) return ret; val = ret; - ret = rgb_cfg_call(drvdata.hdev, SET_RGB_CFG, LIGHT_MODE_SEL, &val, + ret = rgb_cfg_call(drvdata->hdev, SET_RGB_CFG, LIGHT_MODE_SEL, &val, 1); if (ret) return ret; - drvdata.rgb_mode = val; + drvdata->rgb_mode = val; return count; }; @@ -990,16 +1155,25 @@ static ssize_t rgb_mode_store(struct device *dev, struct device_attribute *attr, static ssize_t rgb_mode_show(struct device *dev, struct device_attribute *attr, char *buf) { + struct led_classdev *led_cdev = dev_get_drvdata(dev); + struct led_classdev_mc *led_mc = lcdev_to_mccdev(led_cdev); + struct gos_cfg_drvdata *drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); + bool rgb_registered; int ret; - ret = rgb_cfg_call(drvdata.hdev, GET_RGB_CFG, LIGHT_MODE_SEL, NULL, 0); +/* Pairs with smp_store_release from gos_cfg_setup */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (!rgb_registered) + return -ENODEV; + + ret = rgb_cfg_call(drvdata->hdev, GET_RGB_CFG, LIGHT_MODE_SEL, NULL, 0); if (ret) return ret; - if (drvdata.rgb_mode >= ARRAY_SIZE(rgb_mode_text)) + if (drvdata->rgb_mode >= ARRAY_SIZE(rgb_mode_text)) return -EINVAL; - return sysfs_emit(buf, "%s\n", rgb_mode_text[drvdata.rgb_mode]); + return sysfs_emit(buf, "%s\n", rgb_mode_text[drvdata->rgb_mode]); }; static ssize_t rgb_mode_index_show(struct device *dev, @@ -1021,10 +1195,19 @@ static ssize_t rgb_profile_store(struct device *dev, struct device_attribute *attr, const char *buf, size_t count) { + struct led_classdev *led_cdev = dev_get_drvdata(dev); + struct led_classdev_mc *led_mc = lcdev_to_mccdev(led_cdev); + struct gos_cfg_drvdata *drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); + bool rgb_registered; size_t size = 1; int ret; u8 val; +/* Pairs with smp_store_release from gos_cfg_setup */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (!rgb_registered) + return -ENODEV; + ret = kstrtou8(buf, 10, &val); if (ret < 0) return ret; @@ -1032,11 +1215,11 @@ static ssize_t rgb_profile_store(struct device *dev, if (val < 1 || val > 3) return -EINVAL; - ret = rgb_cfg_call(drvdata.hdev, SET_RGB_CFG, LIGHT_PROFILE_SEL, &val, size); + ret = rgb_cfg_call(drvdata->hdev, SET_RGB_CFG, LIGHT_PROFILE_SEL, &val, size); if (ret) return ret; - drvdata.rgb_profile = val; + drvdata->rgb_profile = val; return count; }; @@ -1044,16 +1227,25 @@ static ssize_t rgb_profile_store(struct device *dev, static ssize_t rgb_profile_show(struct device *dev, struct device_attribute *attr, char *buf) { + struct led_classdev *led_cdev = dev_get_drvdata(dev); + struct led_classdev_mc *led_mc = lcdev_to_mccdev(led_cdev); + struct gos_cfg_drvdata *drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); + bool rgb_registered; int ret; - ret = rgb_cfg_call(drvdata.hdev, GET_RGB_CFG, LIGHT_PROFILE_SEL, NULL, 0); +/* Pairs with smp_store_release from gos_cfg_setup */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (!rgb_registered) + return -ENODEV; + + ret = rgb_cfg_call(drvdata->hdev, GET_RGB_CFG, LIGHT_PROFILE_SEL, NULL, 0); if (ret) return ret; - if (drvdata.rgb_profile < 1 || drvdata.rgb_profile > 3) + if (drvdata->rgb_profile < 1 || drvdata->rgb_profile > 3) return -EINVAL; - return sysfs_emit(buf, "%hhu\n", drvdata.rgb_profile); + return sysfs_emit(buf, "%hhu\n", drvdata->rgb_profile); }; static ssize_t rgb_profile_range_show(struct device *dev, @@ -1065,24 +1257,31 @@ static ssize_t rgb_profile_range_show(struct device *dev, static void hid_gos_brightness_set(struct led_classdev *led_cdev, enum led_brightness brightness) { - struct led_classdev_mc *mc_cdev = lcdev_to_mccdev(drvdata.led_cdev); + struct led_classdev_mc *led_mc = lcdev_to_mccdev(led_cdev); + struct gos_cfg_drvdata *drvdata = container_of(led_mc, struct gos_cfg_drvdata, led_mc); enum rgb_config_index index; + bool rgb_registered; int ret; +/* Pairs with smp_store_release from gos_cfg_setup */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (!rgb_registered) + return; + if (brightness > led_cdev->max_brightness) { dev_err(led_cdev->dev, "Invalid argument\n"); return; } - index = drvdata.rgb_profile + 2; - u8 rgb_profile[6] = { drvdata.rgb_effect, - mc_cdev->subled_info[0].intensity, - mc_cdev->subled_info[1].intensity, - mc_cdev->subled_info[2].intensity, + index = drvdata->rgb_profile + 2; + u8 rgb_profile[6] = { drvdata->rgb_effect, + led_mc->subled_info[0].intensity, + led_mc->subled_info[1].intensity, + led_mc->subled_info[2].intensity, brightness, - drvdata.rgb_speed }; + drvdata->rgb_speed }; - ret = rgb_cfg_call(drvdata.hdev, SET_RGB_CFG, index, rgb_profile, 6); + ret = rgb_cfg_call(drvdata->hdev, SET_RGB_CFG, index, rgb_profile, 6); switch (ret) { case 0: led_cdev->brightness = brightness; @@ -1318,103 +1517,140 @@ static struct mc_subled gos_rgb_subled_info[] = { }, }; -static struct led_classdev_mc gos_cdev_rgb = { - .led_cdev = { - .name = "go_s:rgb:joystick_rings", - .brightness = 0x50, - .max_brightness = 0x64, - .brightness_set = hid_gos_brightness_set, - }, - .num_colors = ARRAY_SIZE(gos_rgb_subled_info), - .subled_info = gos_rgb_subled_info, -}; - static void cfg_setup(struct work_struct *work) { + struct delayed_work *dwork = container_of(work, struct delayed_work, work); + struct gos_cfg_drvdata *drvdata = container_of(dwork, struct gos_cfg_drvdata, + gos_cfg_setup); + bool gp_registered, rgb_registered; int ret; + if (!drvdata) + return; + /* MCU */ - ret = mcu_property_out(drvdata.hdev, GET_MCU_ID, FEATURE_NONE, NULL, 0); + ret = mcu_property_out(drvdata->hdev, GET_MCU_ID, FEATURE_NONE, NULL, 0); if (ret) { - dev_err(&drvdata.hdev->dev, "Failed to retrieve MCU ID: %i\n", + dev_err(&drvdata->hdev->dev, "Failed to retrieve MCU ID: %i\n", ret); return; } - ret = mcu_property_out(drvdata.hdev, GET_VERSION, FEATURE_NONE, NULL, 0); + ret = mcu_property_out(drvdata->hdev, GET_VERSION, FEATURE_NONE, NULL, 0); if (ret) { - dev_err(&drvdata.hdev->dev, "Failed to retrieve MCU Version: %i\n", ret); + dev_err(&drvdata->hdev->dev, "Failed to retrieve MCU Version: %i\n", ret); return; } - ret = mcu_property_out(drvdata.hdev, GET_PL_TEST, TEST_TP_MFR, NULL, 0); + ret = mcu_property_out(drvdata->hdev, GET_PL_TEST, TEST_TP_MFR, NULL, 0); if (ret) { - dev_err(&drvdata.hdev->dev, + dev_err(&drvdata->hdev->dev, "Failed to retrieve Touchpad Manufacturer: %i\n", ret); return; } - ret = mcu_property_out(drvdata.hdev, GET_PL_TEST, TEST_TP_VER, NULL, 0); + ret = mcu_property_out(drvdata->hdev, GET_PL_TEST, TEST_TP_VER, NULL, 0); if (ret) { - dev_err(&drvdata.hdev->dev, + dev_err(&drvdata->hdev->dev, "Failed to retrieve Touchpad Firmware Version: %i\n", ret); return; } - ret = mcu_property_out(drvdata.hdev, GET_PL_TEST, TEST_IMU_MFR, NULL, 0); + ret = mcu_property_out(drvdata->hdev, GET_PL_TEST, TEST_IMU_MFR, NULL, 0); if (ret) { - dev_err(&drvdata.hdev->dev, + dev_err(&drvdata->hdev->dev, "Failed to retrieve IMU Manufacturer: %i\n", ret); return; } - ret = mcu_property_out(drvdata.hdev, GET_GAMEPAD_CFG, FEATURE_OS_MODE, + ret = mcu_property_out(drvdata->hdev, GET_GAMEPAD_CFG, FEATURE_OS_MODE, NULL, 0); if (ret) { - dev_err(&drvdata.hdev->dev, + dev_err(&drvdata->hdev->dev, "Failed to retrieve OS Mode: %i\n", ret); return; } -} - -static int hid_gos_cfg_probe(struct hid_device *hdev, - const struct hid_device_id *_id) -{ - int ret; - hid_set_drvdata(hdev, &drvdata); - drvdata.hdev = hdev; - mutex_init(&drvdata.cfg_mutex); +/* Pairs with smp_store_release from below */ + gp_registered = smp_load_acquire(&drvdata->gp_registered); + if (gp_registered) + goto try_rgb; - ret = sysfs_create_groups(&hdev->dev.kobj, top_level_attr_groups); + ret = sysfs_create_groups(&drvdata->hdev->dev.kobj, top_level_attr_groups); if (ret) { - dev_err_probe(&hdev->dev, ret, - "Failed to create gamepad configuration attributes\n"); - return ret; + dev_err(&drvdata->hdev->dev, + "Failed to create gamepad configuration attributes: %i\n", ret); + goto try_rgb; } - ret = devm_led_classdev_multicolor_register(&hdev->dev, &gos_cdev_rgb); +/* Pairs with smp_load_acquire in attribute show/store functions */ + smp_store_release(&drvdata->gp_registered, true); + gp_registered = true; + +try_rgb: +/* Pairs with smp_store_release from below */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (rgb_registered) + goto update_kobjects; + + ret = devm_led_classdev_multicolor_register(&drvdata->hdev->dev, &drvdata->led_mc); if (ret) { - dev_err_probe(&hdev->dev, ret, "Failed to create RGB device\n"); - return ret; + dev_err(&drvdata->hdev->dev, + "Failed to create RGB device: %i\n", ret); + goto update_kobjects; } - ret = devm_device_add_group(gos_cdev_rgb.led_cdev.dev, &rgb_attr_group); + ret = devm_device_add_group(drvdata->led_mc.led_cdev.dev, &rgb_attr_group); if (ret) { - dev_err_probe(&hdev->dev, ret, - "Failed to create RGB configuration attributes\n"); - return ret; + dev_err(&drvdata->hdev->dev, + "Failed to create RGB configuration attributes: %i\n", ret); + goto update_kobjects; } - drvdata.led_cdev = &gos_cdev_rgb.led_cdev; +/* Pairs with smp_load_acquire in attribute show/store functions */ + smp_store_release(&drvdata->rgb_registered, true); + rgb_registered = true; + +update_kobjects: + if (gp_registered) + kobject_uevent(&drvdata->hdev->dev.kobj, KOBJ_CHANGE); + if (rgb_registered) + kobject_uevent(&drvdata->led_mc.led_cdev.dev->kobj, KOBJ_CHANGE); +} + +static int hid_gos_cfg_probe(struct hid_device *hdev, + const struct hid_device_id *_id) +{ + struct gos_cfg_drvdata *drvdata; + int ret; - init_completion(&drvdata.send_cmd_complete); + drvdata = devm_kzalloc(&hdev->dev, sizeof(struct gos_cfg_drvdata), GFP_KERNEL); + if (!drvdata) + return -ENOMEM; + + hid_set_drvdata(hdev, drvdata); + drvdata->hdev = hdev; + + mutex_init(&drvdata->cfg_mutex); + init_completion(&drvdata->send_cmd_complete); + + /* Device is hardwired and name is guaranteed to be unique */ + drvdata->led_mc.led_cdev.name = "go_s:rgb:joystick_rings"; + drvdata->led_mc.led_cdev.brightness = 0x50; + drvdata->led_mc.led_cdev.max_brightness = 0x64; + drvdata->led_mc.led_cdev.color = LED_COLOR_ID_RGB; + drvdata->led_mc.led_cdev.brightness_set = hid_gos_brightness_set; + drvdata->led_mc.num_colors = 3; + drvdata->led_mc.subled_info = devm_kmemdup(&hdev->dev, gos_rgb_subled_info, + sizeof(gos_rgb_subled_info), GFP_KERNEL); + if (!drvdata->led_mc.subled_info) + return -ENOMEM; /* Executing calls prior to returning from probe will lock the MCU. Schedule * initial data call after probe has completed and MCU can accept calls. */ - INIT_DELAYED_WORK(&drvdata.gos_cfg_setup, &cfg_setup); - ret = schedule_delayed_work(&drvdata.gos_cfg_setup, msecs_to_jiffies(2)); + INIT_DELAYED_WORK(&drvdata->gos_cfg_setup, &cfg_setup); + ret = schedule_delayed_work(&drvdata->gos_cfg_setup, msecs_to_jiffies(2)); if (!ret) { dev_err(&hdev->dev, "Failed to schedule startup delayed work\n"); return -ENODEV; @@ -1425,8 +1661,12 @@ static int hid_gos_cfg_probe(struct hid_device *hdev, static void hid_gos_cfg_remove(struct hid_device *hdev) { - guard(mutex)(&drvdata.cfg_mutex); - cancel_delayed_work_sync(&drvdata.gos_cfg_setup); + struct gos_cfg_drvdata *drvdata = hid_get_drvdata(hdev); + + if (!drvdata) + return; + + disable_delayed_work_sync(&drvdata->gos_cfg_setup); sysfs_remove_groups(&hdev->dev.kobj, top_level_attr_groups); hid_hw_close(hdev); hid_hw_stop(hdev); @@ -1435,22 +1675,43 @@ static void hid_gos_cfg_remove(struct hid_device *hdev) static int hid_gos_cfg_reset_resume(struct hid_device *hdev) { - u8 os_mode = drvdata.os_mode; + struct gos_cfg_drvdata *drvdata = hid_get_drvdata(hdev); + bool gp_registered, rgb_registered; + u8 os_mode; int ret; - ret = mcu_property_out(drvdata.hdev, SET_GAMEPAD_CFG, + if (!drvdata) + return -ENODEV; + + os_mode = drvdata->os_mode; +/* Pairs with smp_store_release from gos_cfg_setup */ + gp_registered = smp_load_acquire(&drvdata->gp_registered); +/* Pairs with smp_store_release from gos_cfg_setup */ + rgb_registered = smp_load_acquire(&drvdata->rgb_registered); + if (!gp_registered || !rgb_registered) { + ret = schedule_delayed_work(&drvdata->gos_cfg_setup, msecs_to_jiffies(2)); + if (!ret) { + dev_err(&hdev->dev, "Failed to schedule startup delayed work\n"); + return -ENODEV; + } + + return 0; + } + + ret = mcu_property_out(drvdata->hdev, SET_GAMEPAD_CFG, FEATURE_OS_MODE, &os_mode, 1); if (ret < 0) return ret; - ret = mcu_property_out(drvdata.hdev, GET_GAMEPAD_CFG, + ret = mcu_property_out(drvdata->hdev, GET_GAMEPAD_CFG, FEATURE_OS_MODE, NULL, 0); if (ret < 0) return ret; - if (drvdata.os_mode != os_mode) + if (drvdata->os_mode != os_mode) return -ENODEV; + kobject_uevent(&drvdata->hdev->dev.kobj, KOBJ_CHANGE); return 0; } @@ -1525,6 +1786,35 @@ static int hid_gos_reset_resume(struct hid_device *hdev) return 0; } +static int hid_gos_cfg_suspend(struct hid_device *hdev) +{ + struct gos_cfg_drvdata *drvdata = hid_get_drvdata(hdev); + + if (!drvdata) + return -ENODEV; + + disable_delayed_work_sync(&drvdata->gos_cfg_setup); + + return 0; +} + +static int hid_gos_suspend(struct hid_device *hdev, pm_message_t msg) +{ + int ret; + u8 ep; + + /* Safe assumption. SET_INTERFACE ioctl can't be used while driver is bound */ + ret = get_endpoint_address(hdev); + if (ret <= 0) + return 0; + + ep = ret; + if (ep == GO_S_CFG_INTF_IN) + return hid_gos_cfg_suspend(hdev); + + return 0; +} + static const struct hid_device_id hid_gos_devices[] = { { HID_USB_DEVICE(USB_VENDOR_ID_QHE, USB_DEVICE_ID_LENOVO_LEGION_GO_S_XINPUT) }, @@ -1540,7 +1830,8 @@ static struct hid_driver hid_lenovo_go_s = { .probe = hid_gos_probe, .remove = hid_gos_remove, .raw_event = hid_gos_raw_event, - .reset_resume = hid_gos_reset_resume, + .reset_resume = pm_ptr(hid_gos_reset_resume), + .suspend = pm_ptr(hid_gos_suspend), }; module_hid_driver(hid_lenovo_go_s); diff --git a/drivers/hid/hid-logitech-dj.c b/drivers/hid/hid-logitech-dj.c index 390d106d5dc1..6e8db01d27ec 100644 --- a/drivers/hid/hid-logitech-dj.c +++ b/drivers/hid/hid-logitech-dj.c @@ -1790,6 +1790,9 @@ static int logi_dj_raw_event(struct hid_device *hdev, if (!hdev->report_enum[HID_INPUT_REPORT].numbered) { if (djrcv_dev->unnumbered_application == HID_GD_KEYBOARD) { + if (size < 2) + return false; + /* * For the keyboard, we can reuse the same report by * using the second byte which is constant in the USB diff --git a/drivers/hid/hid-mcp2200.c b/drivers/hid/hid-mcp2200.c index dafdd5b4a079..c6fd5fb5d578 100644 --- a/drivers/hid/hid-mcp2200.c +++ b/drivers/hid/hid-mcp2200.c @@ -301,6 +301,11 @@ static int mcp2200_raw_event(struct hid_device *hdev, struct hid_report *report, switch (data[0]) { case READ_ALL: + if (size < sizeof(*all_resp)) { + mcp->status = -EPROTO; + break; + } + all_resp = (struct mcp_read_all_resp *) data; mcp->status = 0; mcp->gpio_inval = all_resp->io_port_val_bmap; diff --git a/drivers/hid/hid-mcp2221.c b/drivers/hid/hid-mcp2221.c index d52ce3531ab7..053377dbced2 100644 --- a/drivers/hid/hid-mcp2221.c +++ b/drivers/hid/hid-mcp2221.c @@ -865,7 +865,7 @@ static int mcp2221_raw_event(struct hid_device *hdev, u8 *buf; struct mcp2221 *mcp = hid_get_drvdata(hdev); - if (size < 4) + if (size < sizeof(mcp->txbuf)) return 0; switch (data[0]) { @@ -1072,8 +1072,7 @@ static void mcp2221_remove(struct hid_device *hdev) #if IS_REACHABLE(CONFIG_IIO) struct mcp2221 *mcp = hid_get_drvdata(hdev); - if (!gpio_mode_enforce) - cancel_delayed_work_sync(&mcp->init_work); + cancel_delayed_work_sync(&mcp->init_work); #endif } @@ -1347,10 +1346,9 @@ static int mcp2221_probe(struct hid_device *hdev, #endif #if IS_REACHABLE(CONFIG_IIO) - if (!gpio_mode_enforce) { - INIT_DELAYED_WORK(&mcp->init_work, mcp_init_work); + INIT_DELAYED_WORK(&mcp->init_work, mcp_init_work); + if (!gpio_mode_enforce) schedule_delayed_work(&mcp->init_work, msecs_to_jiffies(100)); - } #endif return 0; diff --git a/drivers/hid/hid-picolcd_fb.c b/drivers/hid/hid-picolcd_fb.c index 8c28e982e09d..a016dd9ca847 100644 --- a/drivers/hid/hid-picolcd_fb.c +++ b/drivers/hid/hid-picolcd_fb.c @@ -531,17 +531,23 @@ int picolcd_init_framebuffer(struct picolcd_data *data) goto err_cleanup; } - fb_deferred_io_init(info); + error = fb_deferred_io_init(info); + if (error) { + dev_err(dev, "failed to initialize deferred I/O\n"); + goto err_sysfs; + } + error = register_framebuffer(info); if (error) { dev_err(dev, "failed to register framebuffer\n"); - goto err_sysfs; + goto err_defio; } return 0; +err_defio: + fb_deferred_io_cleanup(info); err_sysfs: device_remove_file(dev, &dev_attr_fb_update_rate); - fb_deferred_io_cleanup(info); err_cleanup: data->fb_info = NULL; diff --git a/drivers/hid/hid-prodikeys.c b/drivers/hid/hid-prodikeys.c index fba01e4fcab1..bf5add40d3e1 100644 --- a/drivers/hid/hid-prodikeys.c +++ b/drivers/hid/hid-prodikeys.c @@ -772,8 +772,13 @@ static int pk_raw_event(struct hid_device *hdev, struct hid_report *report, if (report->id == data[0]) switch (report->id) { case 0x01: /* midi keys (qwerty)*/ - case 0x03: /* midi keyboard (musical)*/ case 0x04: /* extra/midi keys (qwerty)*/ + if (size < 4) + break; + ret = pcmidi_handle_report(pm, report->id, + data, size); + break; + case 0x03: /* midi keyboard (musical)*/ ret = pcmidi_handle_report(pm, report->id, data, size); break; diff --git a/drivers/hid/hid-pxrc.c b/drivers/hid/hid-pxrc.c index 71fe0c06ddcd..e3755d8b85c2 100644 --- a/drivers/hid/hid-pxrc.c +++ b/drivers/hid/hid-pxrc.c @@ -55,6 +55,9 @@ static int pxrc_raw_event(struct hid_device *hdev, struct hid_report *report, { struct pxrc_priv *priv = hid_get_drvdata(hdev); + if (size < 8) + return 0; + if (priv->alternate) priv->slider = data[7]; else diff --git a/drivers/hid/hid-quirks.c b/drivers/hid/hid-quirks.c index 96a36c5ba04f..d15a587f10c5 100644 --- a/drivers/hid/hid-quirks.c +++ b/drivers/hid/hid-quirks.c @@ -102,6 +102,7 @@ static const struct hid_device_id hid_quirks[] = { { HID_USB_DEVICE(USB_VENDOR_ID_HAPP, USB_DEVICE_ID_UGCI_FIGHTING), HID_QUIRK_BADPAD | HID_QUIRK_MULTI_INPUT }, { HID_USB_DEVICE(USB_VENDOR_ID_HAPP, USB_DEVICE_ID_UGCI_FLYING), HID_QUIRK_BADPAD | HID_QUIRK_MULTI_INPUT }, { HID_USB_DEVICE(USB_VENDOR_ID_HOLTEK_ALT, USB_DEVICE_ID_HOLTEK_ALT_KEYBOARD_A096), HID_QUIRK_NO_INIT_REPORTS }, + { HID_USB_DEVICE(USB_VENDOR_ID_HOLTEK_ALT, USB_DEVICE_ID_HOLTEK_ALT_KEYBOARD_A292), HID_QUIRK_ALWAYS_POLL }, { HID_USB_DEVICE(USB_VENDOR_ID_HOLTEK_ALT, USB_DEVICE_ID_HOLTEK_ALT_KEYBOARD_A293), HID_QUIRK_ALWAYS_POLL }, { HID_USB_DEVICE(USB_VENDOR_ID_HP, USB_PRODUCT_ID_HP_LOGITECH_OEM_USB_OPTICAL_MOUSE_0A4A), HID_QUIRK_ALWAYS_POLL }, { HID_BLUETOOTH_DEVICE(USB_VENDOR_ID_HP, USB_PRODUCT_ID_HP_ELITE_PRESENTER_MOUSE_464A), HID_QUIRK_MULTI_INPUT }, diff --git a/drivers/hid/hid-roccat-isku.c b/drivers/hid/hid-roccat-isku.c index 93a49c93ae8c..ef1ec9ccaa82 100644 --- a/drivers/hid/hid-roccat-isku.c +++ b/drivers/hid/hid-roccat-isku.c @@ -411,6 +411,9 @@ static int isku_raw_event(struct hid_device *hdev, if (isku == NULL) return 0; + if (size < sizeof(struct isku_report_button)) + return 0; + isku_keep_values_up_to_date(isku, data); if (isku->roccat_claimed) diff --git a/drivers/hid/hid-roccat-koneplus.c b/drivers/hid/hid-roccat-koneplus.c index f80a60539a96..df39868d7e6a 100644 --- a/drivers/hid/hid-roccat-koneplus.c +++ b/drivers/hid/hid-roccat-koneplus.c @@ -523,6 +523,9 @@ static int koneplus_raw_event(struct hid_device *hdev, if (koneplus == NULL) return 0; + if (size < sizeof(struct koneplus_mouse_report_button)) + return 0; + koneplus_keep_values_up_to_date(koneplus, data); if (koneplus->roccat_claimed) diff --git a/drivers/hid/hid-roccat-konepure.c b/drivers/hid/hid-roccat-konepure.c index 7f753dfc2a10..529466c33bf8 100644 --- a/drivers/hid/hid-roccat-konepure.c +++ b/drivers/hid/hid-roccat-konepure.c @@ -181,6 +181,9 @@ static int konepure_raw_event(struct hid_device *hdev, != USB_INTERFACE_PROTOCOL_MOUSE) return 0; + if (size < sizeof(struct konepure_mouse_report_button)) + return 0; + if (data[0] != KONEPURE_MOUSE_REPORT_NUMBER_BUTTON) return 0; diff --git a/drivers/hid/hid-roccat-kovaplus.c b/drivers/hid/hid-roccat-kovaplus.c index 9ec42c218ef9..26832b279313 100644 --- a/drivers/hid/hid-roccat-kovaplus.c +++ b/drivers/hid/hid-roccat-kovaplus.c @@ -614,6 +614,9 @@ static int kovaplus_raw_event(struct hid_device *hdev, if (kovaplus == NULL) return 0; + if (size < sizeof(struct kovaplus_mouse_report_button)) + return 0; + kovaplus_keep_values_up_to_date(kovaplus, data); if (kovaplus->roccat_claimed) diff --git a/drivers/hid/hid-roccat-pyra.c b/drivers/hid/hid-roccat-pyra.c index 0d515995bb9d..44c2b416f2db 100644 --- a/drivers/hid/hid-roccat-pyra.c +++ b/drivers/hid/hid-roccat-pyra.c @@ -557,6 +557,9 @@ static int pyra_raw_event(struct hid_device *hdev, struct hid_report *report, if (pyra == NULL) return 0; + if (size < sizeof(struct pyra_mouse_event_button)) + return 0; + pyra_keep_values_up_to_date(pyra, data); if (pyra->roccat_claimed) diff --git a/drivers/hid/hid-roccat-ryos.c b/drivers/hid/hid-roccat-ryos.c index db83f42457da..85df7955b87c 100644 --- a/drivers/hid/hid-roccat-ryos.c +++ b/drivers/hid/hid-roccat-ryos.c @@ -189,6 +189,9 @@ static int ryos_raw_event(struct hid_device *hdev, != RYOS_USB_INTERFACE_PROTOCOL) return 0; + if (size < sizeof(struct ryos_report_special)) + return 0; + if (data[0] != RYOS_REPORT_NUMBER_SPECIAL) return 0; diff --git a/drivers/hid/hid-roccat-savu.c b/drivers/hid/hid-roccat-savu.c index 679136933560..cfd86267e35b 100644 --- a/drivers/hid/hid-roccat-savu.c +++ b/drivers/hid/hid-roccat-savu.c @@ -182,6 +182,9 @@ static int savu_raw_event(struct hid_device *hdev, if (savu == NULL) return 0; + if (size < sizeof(struct savu_mouse_report_special)) + return 0; + if (savu->roccat_claimed) savu_report_to_chrdev(savu, data); diff --git a/drivers/hid/hid-zydacron.c b/drivers/hid/hid-zydacron.c index 1aae80f848f5..02de153d5628 100644 --- a/drivers/hid/hid-zydacron.c +++ b/drivers/hid/hid-zydacron.c @@ -114,6 +114,9 @@ static int zc_raw_event(struct hid_device *hdev, struct hid_report *report, unsigned key; unsigned short index; + if (size < 2) + return 0; + if (report->id == data[0] && (hdev->claimed & HID_CLAIMED_INPUT)) { /* break keys */ diff --git a/drivers/hid/usbhid/hid-core.c b/drivers/hid/usbhid/hid-core.c index 96b0181cf819..d7cf2b56e117 100644 --- a/drivers/hid/usbhid/hid-core.c +++ b/drivers/hid/usbhid/hid-core.c @@ -688,7 +688,14 @@ static int usbhid_open(struct hid_device *hid) set_bit(HID_OPENED, &usbhid->iofl); - if (hid->quirks & HID_QUIRK_ALWAYS_POLL) { + /* + * ALWAYS_POLL devices are already polled from usbhid_start(), and a + * device with no input reports has nothing to send on the interrupt + * IN endpoint. In some cases polling is harmful: on the ASUS ROG + * N-Key keyboards it makes the sibling interface drop keypresses. + */ + if ((hid->quirks & HID_QUIRK_ALWAYS_POLL) || + list_empty(&hid->report_enum[HID_INPUT_REPORT].report_list)) { res = 0; goto Done; } diff --git a/drivers/hid/usbhid/hiddev.c b/drivers/hid/usbhid/hiddev.c index 6378801b22c6..21396481995b 100644 --- a/drivers/hid/usbhid/hiddev.c +++ b/drivers/hid/usbhid/hiddev.c @@ -46,6 +46,12 @@ struct hiddev_list { struct mutex thread_lock; }; +static void hiddev_put(struct hiddev *hiddev) +{ + if (refcount_dec_and_test(&hiddev->refcount)) + kfree(hiddev); +} + /* * Find a report, given the report's type and ID. The ID can be specified * indirectly by REPORT_ID_FIRST (which returns the first report of the given @@ -216,26 +222,21 @@ static int hiddev_fasync(int fd, struct file *file, int on) static int hiddev_release(struct inode * inode, struct file * file) { struct hiddev_list *list = file->private_data; + struct hiddev *hiddev = list->hiddev; unsigned long flags; - spin_lock_irqsave(&list->hiddev->list_lock, flags); + spin_lock_irqsave(&hiddev->list_lock, flags); list_del(&list->node); - spin_unlock_irqrestore(&list->hiddev->list_lock, flags); + spin_unlock_irqrestore(&hiddev->list_lock, flags); - mutex_lock(&list->hiddev->existancelock); - if (!--list->hiddev->open) { - if (list->hiddev->exist) { - hid_hw_close(list->hiddev->hid); - hid_hw_power(list->hiddev->hid, PM_HINT_NORMAL); - } else { - mutex_unlock(&list->hiddev->existancelock); - kfree(list->hiddev); - vfree(list); - return 0; - } + mutex_lock(&hiddev->existancelock); + if (!--hiddev->open && hiddev->exist) { + hid_hw_close(hiddev->hid); + hid_hw_power(hiddev->hid, PM_HINT_NORMAL); } - mutex_unlock(&list->hiddev->existancelock); + mutex_unlock(&hiddev->existancelock); + hiddev_put(hiddev); vfree(list); return 0; @@ -270,6 +271,7 @@ static int __hiddev_open(struct hiddev *hiddev, struct file *file) spin_unlock_irq(&hiddev->list_lock); file->private_data = list; + refcount_inc(&hiddev->refcount); return 0; @@ -897,6 +899,7 @@ int hiddev_connect(struct hid_device *hid, unsigned int force) INIT_LIST_HEAD(&hiddev->list); spin_lock_init(&hiddev->list_lock); mutex_init(&hiddev->existancelock); + refcount_set(&hiddev->refcount, 1); hid->hiddev = hiddev; hiddev->hid = hid; hiddev->exist = 1; @@ -937,9 +940,7 @@ void hiddev_disconnect(struct hid_device *hid) if (hiddev->open) { hid_hw_close(hiddev->hid); wake_up_interruptible(&hiddev->wait); - mutex_unlock(&hiddev->existancelock); - } else { - mutex_unlock(&hiddev->existancelock); - kfree(hiddev); } + mutex_unlock(&hiddev->existancelock); + hiddev_put(hiddev); } diff --git a/drivers/hid/wacom_sys.c b/drivers/hid/wacom_sys.c index 40770affdbde..628372ab818a 100644 --- a/drivers/hid/wacom_sys.c +++ b/drivers/hid/wacom_sys.c @@ -758,27 +758,47 @@ static void wacom_retrieve_hid_descriptor(struct hid_device *hdev, struct wacom_hdev_data { struct list_head list; struct kref kref; - struct hid_device *dev; + char phys[64]; + __u32 vendor; + __u32 product; + __u32 device_type; struct wacom_shared shared; }; +static bool wacom_compare_device_paths(struct hid_device *hdev_a, + const char *phys_b, char separator) +{ + const char *p1 = strrchr(hdev_a->phys, separator); + const char *p2 = strrchr(phys_b, separator); + int n1, n2; + + if (!p1 || !p2) + return false; + + n1 = p1 - hdev_a->phys; + n2 = p2 - phys_b; + + if (n1 != n2 || n1 <= 0 || n2 <= 0) + return false; + + return !strncmp(hdev_a->phys, phys_b, n1); +} + static LIST_HEAD(wacom_udev_list); static DEFINE_MUTEX(wacom_udev_list_lock); static bool wacom_are_sibling(struct hid_device *hdev, - struct hid_device *sibling) + struct wacom_hdev_data *data) { struct wacom *wacom = hid_get_drvdata(hdev); struct wacom_features *features = &wacom->wacom_wac.features; - struct wacom *sibling_wacom = hid_get_drvdata(sibling); - struct wacom_features *sibling_features = &sibling_wacom->wacom_wac.features; __u32 oVid = features->oVid ? features->oVid : hdev->vendor; __u32 oPid = features->oPid ? features->oPid : hdev->product; /* The defined oVid/oPid must match that of the sibling */ - if (features->oVid != HID_ANY_ID && sibling->vendor != oVid) + if (features->oVid != HID_ANY_ID && data->vendor != oVid) return false; - if (features->oPid != HID_ANY_ID && sibling->product != oPid) + if (features->oPid != HID_ANY_ID && data->product != oPid) return false; /* @@ -786,11 +806,11 @@ static bool wacom_are_sibling(struct hid_device *hdev, * device path, while those with different VID/PID must share * the same physical parent device path. */ - if (hdev->vendor == sibling->vendor && hdev->product == sibling->product) { - if (!hid_compare_device_paths(hdev, sibling, '/')) + if (hdev->vendor == data->vendor && hdev->product == data->product) { + if (!wacom_compare_device_paths(hdev, data->phys, '/')) return false; } else { - if (!hid_compare_device_paths(hdev, sibling, '.')) + if (!wacom_compare_device_paths(hdev, data->phys, '.')) return false; } @@ -803,7 +823,7 @@ static bool wacom_are_sibling(struct hid_device *hdev, * devices. */ if ((features->device_type & WACOM_DEVICETYPE_DIRECT) && - !(sibling_features->device_type & WACOM_DEVICETYPE_DIRECT)) + !(data->device_type & WACOM_DEVICETYPE_DIRECT)) return false; /* @@ -811,17 +831,17 @@ static bool wacom_are_sibling(struct hid_device *hdev, * devices. */ if (!(features->device_type & WACOM_DEVICETYPE_DIRECT) && - (sibling_features->device_type & WACOM_DEVICETYPE_DIRECT)) + (data->device_type & WACOM_DEVICETYPE_DIRECT)) return false; /* Pen devices may only be siblings of touch devices */ if ((features->device_type & WACOM_DEVICETYPE_PEN) && - !(sibling_features->device_type & WACOM_DEVICETYPE_TOUCH)) + !(data->device_type & WACOM_DEVICETYPE_TOUCH)) return false; /* Touch devices may only be siblings of pen devices */ if ((features->device_type & WACOM_DEVICETYPE_TOUCH) && - !(sibling_features->device_type & WACOM_DEVICETYPE_PEN)) + !(data->device_type & WACOM_DEVICETYPE_PEN)) return false; /* @@ -837,7 +857,7 @@ static struct wacom_hdev_data *wacom_get_hdev_data(struct hid_device *hdev) /* Try to find an already-probed interface from the same device */ list_for_each_entry(data, &wacom_udev_list, list) { - if (hid_compare_device_paths(hdev, data->dev, '/')) { + if (wacom_compare_device_paths(hdev, data->phys, '/')) { kref_get(&data->kref); return data; } @@ -845,7 +865,7 @@ static struct wacom_hdev_data *wacom_get_hdev_data(struct hid_device *hdev) /* Fallback to finding devices that appear to be "siblings" */ list_for_each_entry(data, &wacom_udev_list, list) { - if (wacom_are_sibling(hdev, data->dev)) { + if (wacom_are_sibling(hdev, data)) { kref_get(&data->kref); return data; } @@ -859,29 +879,41 @@ static void wacom_release_shared_data(struct kref *kref) struct wacom_hdev_data *data = container_of(kref, struct wacom_hdev_data, kref); - mutex_lock(&wacom_udev_list_lock); list_del(&data->list); - mutex_unlock(&wacom_udev_list_lock); - kfree(data); } static void wacom_remove_shared_data(void *res) { - struct wacom *wacom = res; + struct wacom *res_wacom = res; struct wacom_hdev_data *data; - struct wacom_wac *wacom_wac = &wacom->wacom_wac; + struct wacom_wac *wacom_wac = &res_wacom->wacom_wac; if (wacom_wac->shared) { data = container_of(wacom_wac->shared, struct wacom_hdev_data, shared); - if (wacom_wac->shared->touch == wacom->hdev) - wacom_wac->shared->touch = NULL; - else if (wacom_wac->shared->pen == wacom->hdev) - wacom_wac->shared->pen = NULL; + scoped_guard(mutex, &wacom_udev_list_lock) { + struct hid_device *touch = + rcu_dereference_protected(wacom_wac->shared->touch, + lockdep_is_held(&wacom_udev_list_lock)); + struct hid_device *pen = + rcu_dereference_protected(wacom_wac->shared->pen, + lockdep_is_held(&wacom_udev_list_lock)); + + if (touch == res_wacom->hdev) { + rcu_assign_pointer(wacom_wac->shared->touch, NULL); + rcu_assign_pointer(wacom_wac->shared->touch_input, NULL); + } else if (pen == res_wacom->hdev) { + rcu_assign_pointer(wacom_wac->shared->pen, NULL); + } + } + + synchronize_rcu(); + + if (kref_put_mutex(&data->kref, wacom_release_shared_data, &wacom_udev_list_lock)) + mutex_unlock(&wacom_udev_list_lock); - kref_put(&data->kref, wacom_release_shared_data); wacom_wac->shared = NULL; } } @@ -904,22 +936,21 @@ static int wacom_add_shared_data(struct hid_device *hdev) } kref_init(&data->kref); - data->dev = hdev; + strscpy(data->phys, hdev->phys, sizeof(data->phys)); + data->vendor = hdev->vendor; + data->product = hdev->product; + data->device_type = wacom_wac->features.device_type; list_add_tail(&data->list, &wacom_udev_list); } - mutex_unlock(&wacom_udev_list_lock); - wacom_wac->shared = &data->shared; - retval = devm_add_action_or_reset(&hdev->dev, wacom_remove_shared_data, wacom); - if (retval) - return retval; + if (wacom_wac->has_mute_touch_switch) + WRITE_ONCE(wacom_wac->shared->has_mute_touch_switch, true); - if (wacom_wac->features.device_type & WACOM_DEVICETYPE_TOUCH) - wacom_wac->shared->touch = hdev; - else if (wacom_wac->features.device_type & WACOM_DEVICETYPE_PEN) - wacom_wac->shared->pen = hdev; + mutex_unlock(&wacom_udev_list_lock); + + retval = devm_add_action_or_reset(&hdev->dev, wacom_remove_shared_data, wacom); return retval; } @@ -2344,28 +2375,86 @@ static void wacom_release_resources(struct wacom *wacom) static void wacom_set_shared_values(struct wacom_wac *wacom_wac) { - if (wacom_wac->features.device_type & WACOM_DEVICETYPE_TOUCH) { - wacom_wac->shared->type = wacom_wac->features.type; - wacom_wac->shared->touch_input = wacom_wac->touch_input; - } + struct wacom *wacom = container_of(wacom_wac, struct wacom, wacom_wac); + + guard(mutex)(&wacom_udev_list_lock); if (wacom_wac->has_mute_touch_switch) { - wacom_wac->shared->has_mute_touch_switch = true; + WRITE_ONCE(wacom_wac->shared->has_mute_touch_switch, true); /* Hardware touch switch may be off. Wait until * we know the switch state to decide is_touch_on. * Softkey state should be initialized to "on" to * match historic default. */ if (wacom_wac->is_soft_touch_switch) - wacom_wac->shared->is_touch_on = true; + WRITE_ONCE(wacom_wac->shared->is_touch_on, true); + } + + if (wacom_wac->features.device_type & WACOM_DEVICETYPE_TOUCH) { + struct hid_device *touch = + rcu_dereference_protected(wacom_wac->shared->touch, + lockdep_is_held(&wacom_udev_list_lock)); + + if (touch == wacom->hdev || !touch) { + wacom_wac->shared->type = wacom_wac->features.type; + rcu_assign_pointer(wacom_wac->shared->touch, wacom->hdev); + rcu_assign_pointer(wacom_wac->shared->touch_input, wacom_wac->touch_input); + } + } else if (wacom_wac->features.device_type & WACOM_DEVICETYPE_PEN) { + rcu_assign_pointer(wacom_wac->shared->pen, wacom->hdev); } +} + +static bool wacom_sibling_pending(struct wacom *wacom) +{ + const struct wacom_features *features = &wacom->wacom_wac.features; + struct hid_device *hdev = wacom->hdev; + struct usb_interface *sibling_intf; + int ifnum; + + if (features->type != HID_GENERIC || + !(features->device_type & WACOM_DEVICETYPE_TOUCH)) + return false; - if (wacom_wac->shared->has_mute_touch_switch && - wacom_wac->shared->touch_input) { - set_bit(EV_SW, wacom_wac->shared->touch_input->evbit); - input_set_capability(wacom_wac->shared->touch_input, EV_SW, - SW_MUTE_DEVICE); + if (wacom->wacom_wac.shared) { + if (rcu_access_pointer(wacom->wacom_wac.shared->pen)) + return false; } + + if (!hid_is_usb(hdev) || !wacom->usbdev || !wacom->intf || + !wacom->intf->cur_altsetting) + return false; + + ifnum = wacom->intf->cur_altsetting->desc.bInterfaceNumber; + + /* + * On composite Wacom devices, the Pen interface is always interface 0. + * If the Touch interface is interface 0, there is no sibling Pen + * interface on this device (standalone touch device). + */ + if (ifnum == 0) + return false; + + /* Look for the sibling Pen interface at interface 0 */ + sibling_intf = usb_ifnum_to_if(wacom->usbdev, 0); + if (!sibling_intf || !sibling_intf->cur_altsetting) + return false; + + if (sibling_intf->cur_altsetting->desc.bInterfaceClass != + USB_INTERFACE_CLASS_HID) + return false; + + if (sibling_intf->cur_altsetting->desc.bInterfaceSubClass == 1 && + (sibling_intf->cur_altsetting->desc.bInterfaceProtocol == 1 || + sibling_intf->cur_altsetting->desc.bInterfaceProtocol == 2)) + return false; + + /* + * Interface 0 is a candidate HID interface on this composite device + * whose probe has not completed yet (shared->pen is NULL). Defer until + * interface 0 finishes probing and registers shared values. + */ + return true; } static int wacom_parse_and_register(struct wacom *wacom, bool wireless) @@ -2445,6 +2534,11 @@ static int wacom_parse_and_register(struct wacom *wacom, bool wireless) if (error) goto fail; + if (wacom_sibling_pending(wacom)) { + error = -EPROBE_DEFER; + goto fail; + } + error = wacom_setup_inputs(wacom); if (error) goto fail; @@ -2502,6 +2596,7 @@ static int wacom_parse_and_register(struct wacom *wacom, bool wireless) fail_hw_stop: hid_hw_stop(hdev); + cancel_delayed_work_sync(&wacom->init_work); fail: wacom_release_resources(wacom); return error; @@ -2789,16 +2884,19 @@ static void wacom_mode_change_work(struct work_struct *work) bool is_direct = wacom->wacom_wac.is_direct_mode; int error = 0; - if (shared->pen) { - wacom1 = hid_get_drvdata(shared->pen); + struct hid_device *pen = rcu_access_pointer(shared->pen); + struct hid_device *touch = rcu_access_pointer(shared->touch); + + if (pen) { + wacom1 = hid_get_drvdata(pen); wacom_release_resources(wacom1); hid_hw_stop(wacom1->hdev); wacom1->wacom_wac.has_mode_change = true; wacom1->wacom_wac.is_direct_mode = is_direct; } - if (shared->touch) { - wacom2 = hid_get_drvdata(shared->touch); + if (touch) { + wacom2 = hid_get_drvdata(touch); wacom_release_resources(wacom2); hid_hw_stop(wacom2->hdev); wacom2->wacom_wac.has_mode_change = true; diff --git a/drivers/hid/wacom_wac.c b/drivers/hid/wacom_wac.c index 8feb8027be95..cf459c16b39a 100644 --- a/drivers/hid/wacom_wac.c +++ b/drivers/hid/wacom_wac.c @@ -510,6 +510,18 @@ static void wacom_intuos_schedule_prox_event(struct wacom_wac *wacom_wac) } } +static void wacom_report_touch_mute(struct wacom_wac *wacom_wac, bool mute) +{ + struct input_dev *touch_input; + + guard(rcu)(); + touch_input = rcu_dereference(wacom_wac->shared->touch_input); + if (touch_input) { + input_report_switch(touch_input, SW_MUTE_DEVICE, mute); + input_sync(touch_input); + } +} + static int wacom_intuos_pad(struct wacom_wac *wacom) { struct wacom_features *features = &wacom->features; @@ -650,12 +662,8 @@ static int wacom_intuos_pad(struct wacom_wac *wacom) input_report_key(input, KEY_CONTROLPANEL, menu); input_report_key(input, KEY_INFO, info); - if (wacom->shared && wacom->shared->touch_input) { - input_report_switch(wacom->shared->touch_input, - SW_MUTE_DEVICE, - !wacom->shared->is_touch_on); - input_sync(wacom->shared->touch_input); - } + if (wacom->shared) + wacom_report_touch_mute(wacom, !wacom->shared->is_touch_on); input_report_abs(input, ABS_RX, strip1); input_report_abs(input, ABS_RY, strip2); @@ -2168,7 +2176,7 @@ static void wacom_wac_pad_event(struct hid_device *hdev, struct hid_field *field */ if ((equivalent_usage == WACOM_HID_WD_MUTE_DEVICE) || (equivalent_usage == WACOM_HID_WD_TOUCHONOFF)) { - if (wacom_wac->shared->touch_input) { + if (wacom_wac->shared) { bool *is_touch_on = &wacom_wac->shared->is_touch_on; if (equivalent_usage == WACOM_HID_WD_MUTE_DEVICE && value) @@ -2176,9 +2184,7 @@ static void wacom_wac_pad_event(struct hid_device *hdev, struct hid_field *field else if (equivalent_usage == WACOM_HID_WD_TOUCHONOFF) *is_touch_on = value; - input_report_switch(wacom_wac->shared->touch_input, - SW_MUTE_DEVICE, !(*is_touch_on)); - input_sync(wacom_wac->shared->touch_input); + wacom_report_touch_mute(wacom_wac, !(*is_touch_on)); } return; } @@ -3301,6 +3307,7 @@ static int wacom_bpt_irq(struct wacom_wac *wacom, size_t len) static void wacom_bamboo_pad_pen_event(struct wacom_wac *wacom, unsigned char *data) { + struct hid_device *pen; unsigned char prefix; /* @@ -3313,13 +3320,11 @@ static void wacom_bamboo_pad_pen_event(struct wacom_wac *wacom, prefix = data[0]; data[0] = WACOM_REPORT_BPAD_PEN; - /* - * actually reroute the event. - * No need to check if wacom->shared->pen is valid, hid_input_report() - * will check for us. - */ - hid_input_report(wacom->shared->pen, HID_INPUT_REPORT, data, - WACOM_PKGLEN_PENABLED, 1); + guard(rcu)(); + pen = rcu_dereference(wacom->shared->pen); + if (pen) + hid_input_report(pen, HID_INPUT_REPORT, data, + WACOM_PKGLEN_PENABLED, 1); data[0] = prefix; } @@ -3396,11 +3401,8 @@ static int wacom_wireless_irq(struct wacom_wac *wacom, size_t len) if ((wacom->shared->type == INTUOSHT || wacom->shared->type == INTUOSHT2) && - wacom->shared->touch_input && wacom->shared->touch_max) { - input_report_switch(wacom->shared->touch_input, - SW_MUTE_DEVICE, data[5] & 0x40); - input_sync(wacom->shared->touch_input); + wacom_report_touch_mute(wacom, data[5] & 0x40); } pid = get_unaligned_be16(&data[6]); @@ -3435,11 +3437,8 @@ static int wacom_status_irq(struct wacom_wac *wacom_wac, size_t len) if ((features->type == INTUOSHT || features->type == INTUOSHT2) && - wacom_wac->shared->touch_input && features->touch_max) { - input_report_switch(wacom_wac->shared->touch_input, - SW_MUTE_DEVICE, data[8] & 0x40); - input_sync(wacom_wac->shared->touch_input); + wacom_report_touch_mute(wacom_wac, data[8] & 0x40); } if (data[9] & 0x02) { /* wireless module is attached */ @@ -3966,8 +3965,8 @@ int wacom_setup_pen_input_capabilities(struct input_dev *input_dev, int wacom_setup_touch_input_capabilities(struct input_dev *input_dev, struct wacom_wac *wacom_wac) { + struct hid_device *hdev = container_of(wacom_wac, struct wacom, wacom_wac)->hdev; struct wacom_features *features = &wacom_wac->features; - if (!(features->device_type & WACOM_DEVICETYPE_TOUCH)) return -ENODEV; @@ -3976,9 +3975,24 @@ int wacom_setup_touch_input_capabilities(struct input_dev *input_dev, else __set_bit(INPUT_PROP_POINTER, input_dev->propbit); - if (features->type == HID_GENERIC) + if (features->type == HID_GENERIC) { + hid_dbg(hdev, "generic touch setup\n"); + + /* + * This is to setup the touch switch for those devices that report + * pen interface first. The setup can only be done here since this + * routine is only called once. When we are here, the shared + * has_mute_touch_switch has been set by the pad. + + * There are two ways of reporting inputs from firmware: stylus first + * or touch first. This path covers the stylus input first case. + */ + if (wacom_wac->shared && READ_ONCE(wacom_wac->shared->has_mute_touch_switch)) { + input_set_capability(input_dev, EV_SW, SW_MUTE_DEVICE); + } /* setup has already been done */ return 0; + } input_dev->evbit[0] |= BIT_MASK(EV_KEY) | BIT_MASK(EV_ABS); __set_bit(BTN_TOUCH, input_dev->keybit); @@ -4010,19 +4024,17 @@ int wacom_setup_touch_input_capabilities(struct input_dev *input_dev, input_dev->evbit[0] |= BIT_MASK(EV_SW); __set_bit(SW_MUTE_DEVICE, input_dev->swbit); - if (wacom_wac->shared->touch->product == 0x361) { + if (hdev->product == 0x361) { input_set_abs_params(input_dev, ABS_MT_POSITION_X, 0, 12440, 4, 0); input_set_abs_params(input_dev, ABS_MT_POSITION_Y, 0, 8640, 4, 0); - } - else if (wacom_wac->shared->touch->product == 0x360) { + } else if (hdev->product == 0x360) { input_set_abs_params(input_dev, ABS_MT_POSITION_X, 0, 8960, 4, 0); input_set_abs_params(input_dev, ABS_MT_POSITION_Y, 0, 5920, 4, 0); - } - else if (wacom_wac->shared->touch->product == 0x393) { + } else if (hdev->product == 0x393) { input_set_abs_params(input_dev, ABS_MT_POSITION_X, 0, 6400, 4, 0); input_set_abs_params(input_dev, ABS_MT_POSITION_Y, @@ -4052,8 +4064,8 @@ int wacom_setup_touch_input_capabilities(struct input_dev *input_dev, fallthrough; case WACOM_27QHDT: - if (wacom_wac->shared->touch->product == 0x32C || - wacom_wac->shared->touch->product == 0xF6) { + if (hdev->product == 0x32C || + hdev->product == 0xF6) { input_dev->evbit[0] |= BIT_MASK(EV_SW); __set_bit(SW_MUTE_DEVICE, input_dev->swbit); wacom_wac->has_mute_touch_switch = true; @@ -4399,7 +4411,26 @@ int wacom_setup_pad_input_capabilities(struct input_dev *input_dev, break; case INTUOSHT3_BT: + break; + case HID_GENERIC: + if (wacom_wac->shared) { + /* + * The setup of touch switch capability can only be done + * here for those devices, such as Cintiq Pro 13 and 16, + * that have touch input registered before stylus and pad + * inputs. + * + * There are two ways of reporting inputs from firmware: + * stylus first or touch first. This path covers the touch + * input first case. + */ + if (wacom_wac->has_mute_touch_switch && + wacom_wac->shared->touch_input) { + input_set_capability(wacom_wac->shared->touch_input, + EV_SW, SW_MUTE_DEVICE); + } + } break; default: diff --git a/drivers/hid/wacom_wac.h b/drivers/hid/wacom_wac.h index 126bec6e5c0c..170d6adbe02a 100644 --- a/drivers/hid/wacom_wac.h +++ b/drivers/hid/wacom_wac.h @@ -285,9 +285,9 @@ struct wacom_shared { /* for wireless device to access USB interfaces */ unsigned touch_max; int type; - struct input_dev *touch_input; - struct hid_device *pen; - struct hid_device *touch; + struct input_dev __rcu *touch_input; + struct hid_device __rcu *pen; + struct hid_device __rcu *touch; bool has_mute_touch_switch; bool is_touch_on; }; diff --git a/drivers/hv/mshv_vtl_main.c b/drivers/hv/mshv_vtl_main.c index d8225242c73f..c6f1e2157d25 100644 --- a/drivers/hv/mshv_vtl_main.c +++ b/drivers/hv/mshv_vtl_main.c @@ -17,6 +17,7 @@ #include #include #include +#include #include #include #include @@ -448,12 +449,35 @@ static void mshv_vtl_cancel(int cpu) static int mshv_vtl_poll_file_wake(wait_queue_entry_t *wait, unsigned int mode, int sync, void *key) { struct mshv_vtl_poll_file *poll_file = container_of(wait, struct mshv_vtl_poll_file, wait); + __poll_t pollflags = key_to_poll(key); + + if (pollflags & POLLFREE) + list_del_init(&wait->entry); mshv_vtl_cancel(poll_file->cpu); + if (pollflags & POLLFREE) { + /* Publish teardown last; pair with acquire in the removal path. */ + smp_store_release(&poll_file->wqh, NULL); + } + return 0; } +static void mshv_vtl_poll_file_remove_wait_queue(struct mshv_vtl_poll_file *poll_file) +{ + wait_queue_head_t *wqh; + + /* POLLFREE providers RCU-delay the waitqueue free. */ + rcu_read_lock(); + /* Pair with the release store in the POLLFREE callback. */ + wqh = smp_load_acquire(&poll_file->wqh); + if (wqh) + remove_wait_queue(wqh, &poll_file->wait); + rcu_read_unlock(); + WRITE_ONCE(poll_file->wqh, NULL); +} + static void mshv_vtl_ptable_queue_proc(struct file *file, wait_queue_head_t *wqh, poll_table *pt) { struct mshv_vtl_poll_file *poll_file = container_of(pt, struct mshv_vtl_poll_file, pt); @@ -490,9 +514,7 @@ static int mshv_vtl_ioctl_set_poll_file(struct mshv_vtl_set_poll_file __user *us mutex_lock(&mshv_vtl_poll_file_lock); - if (poll_file->wqh) - remove_wait_queue(poll_file->wqh, &poll_file->wait); - poll_file->wqh = NULL; + mshv_vtl_poll_file_remove_wait_queue(poll_file); old_file = poll_file->file; poll_file->file = file; diff --git a/drivers/hv/ring_buffer.c b/drivers/hv/ring_buffer.c index 592a9601faaa..a18b309c0f4e 100644 --- a/drivers/hv/ring_buffer.c +++ b/drivers/hv/ring_buffer.c @@ -70,15 +70,6 @@ static void hv_signal_on_write(u32 old_write, struct vmbus_channel *channel) } } -/* Get the next write location for the specified ring buffer. */ -static inline u32 -hv_get_next_write_location(struct hv_ring_buffer_info *ring_info) -{ - u32 next = ring_info->ring_buffer->write_index; - - return next; -} - /* Set the next write location for the specified ring buffer. */ static inline void hv_set_next_write_location(struct hv_ring_buffer_info *ring_info, @@ -281,6 +272,7 @@ int hv_ringbuffer_write(struct vmbus_channel *channel, u32 totalbytes_towrite = sizeof(u64); u32 next_write_location; u32 old_write; + u32 read_index; u64 prev_indices; unsigned long flags; struct hv_ring_buffer_info *outring_info = &channel->outbound; @@ -295,7 +287,20 @@ int hv_ringbuffer_write(struct vmbus_channel *channel, spin_lock_irqsave(&outring_info->ring_lock, flags); - bytes_avail_towrite = hv_get_bytes_to_write(outring_info); + read_index = READ_ONCE(outring_info->ring_buffer->read_index); + old_write = READ_ONCE(outring_info->ring_buffer->write_index); + if (unlikely(read_index >= outring_info->ring_datasize || + old_write >= outring_info->ring_datasize)) { + spin_unlock_irqrestore(&outring_info->ring_lock, flags); + pr_err_ratelimited("outbound ring indices out of range: relid %u read %u write %u size %u\n", + channel->offermsg.child_relid, read_index, + old_write, outring_info->ring_datasize); + return -EIO; + } + + bytes_avail_towrite = old_write >= read_index ? + outring_info->ring_datasize - (old_write - read_index) : + read_index - old_write; /* * If there is only room for the packet, assume it is full. @@ -317,9 +322,7 @@ int hv_ringbuffer_write(struct vmbus_channel *channel, channel->out_full_flag = false; /* Write to the ring buffer */ - next_write_location = hv_get_next_write_location(outring_info); - - old_write = next_write_location; + next_write_location = old_write; for (i = 0; i < kv_count; i++) { next_write_location = hv_copyto_ringbuffer(outring_info, diff --git a/drivers/infiniband/core/ib_core_uverbs.c b/drivers/infiniband/core/ib_core_uverbs.c index 41c84ffe8c09..2bdb6972fe1a 100644 --- a/drivers/infiniband/core/ib_core_uverbs.c +++ b/drivers/infiniband/core/ib_core_uverbs.c @@ -244,16 +244,8 @@ void rdma_user_mmap_entry_remove(struct rdma_user_mmap_entry *entry) entry->driver_removed = true; xa_unlock(&entry->ucontext->mmap_xa); list_for_each_entry_safe(uverbs_dmabuf, tmp, &entry->dmabufs, dmabufs_elm) { - dma_resv_lock(uverbs_dmabuf->dmabuf->resv, NULL); list_del(&uverbs_dmabuf->dmabufs_elm); - uverbs_dmabuf->revoked = true; - dma_buf_invalidate_mappings(uverbs_dmabuf->dmabuf); - dma_resv_wait_timeout(uverbs_dmabuf->dmabuf->resv, - DMA_RESV_USAGE_BOOKKEEP, false, - MAX_SCHEDULE_TIMEOUT); - dma_resv_unlock(uverbs_dmabuf->dmabuf->resv); - kref_put(&uverbs_dmabuf->kref, ib_uverbs_dmabuf_done); - wait_for_completion(&uverbs_dmabuf->comp); + ib_uverbs_dmabuf_revoke(uverbs_dmabuf); } mutex_unlock(&entry->dmabufs_lock); diff --git a/drivers/infiniband/core/uverbs.h b/drivers/infiniband/core/uverbs.h index c64dd6b94e10..728e09b41992 100644 --- a/drivers/infiniband/core/uverbs.h +++ b/drivers/infiniband/core/uverbs.h @@ -42,6 +42,8 @@ #include #include #include +#include +#include #include #include @@ -135,6 +137,13 @@ struct ib_uverbs_completion_event_file { struct ib_uverbs_dmabuf_file { struct ib_uobject uobj; + /* + * Serializes users of dmabuf with the dma_buf's release, which sets it + * to NULL. The uobject can outlive the dma_buf while it waits to be + * destroyed. The map and unmap callbacks use attachment->dmabuf + * instead of taking this lock. + */ + struct mutex dmabuf_release_lock; struct dma_buf *dmabuf; struct list_head dmabufs_elm; struct rdma_user_mmap_entry *mmap_entry; @@ -359,6 +368,34 @@ static inline void ib_uverbs_dmabuf_done(struct kref *kref) complete(&priv->comp); } +/* + * Revoke the dmabuf and wait for all of its DMA mappings to be gone. Must be + * called with the mmap entry's dmabufs_lock held, after the dmabuf has been + * removed from the entry's dmabufs list, so it runs once per dmabuf. Lock + * order is dmabufs_lock, dmabuf_release_lock, then the dma_resv lock. + */ +static inline void ib_uverbs_dmabuf_revoke(struct ib_uverbs_dmabuf_file *priv) +{ + mutex_lock(&priv->dmabuf_release_lock); + if (priv->dmabuf) { + dma_resv_lock(priv->dmabuf->resv, NULL); + priv->revoked = true; + dma_buf_invalidate_mappings(priv->dmabuf); + dma_resv_wait_timeout(priv->dmabuf->resv, + DMA_RESV_USAGE_BOOKKEEP, false, + MAX_SCHEDULE_TIMEOUT); + dma_resv_unlock(priv->dmabuf->resv); + } else { + /* A released dma_buf has no attachments left to invalidate */ + priv->revoked = true; + } + mutex_unlock(&priv->dmabuf_release_lock); + + kref_put(&priv->kref, ib_uverbs_dmabuf_done); + /* Wait till all DMA unmaps are completed */ + wait_for_completion(&priv->comp); +} + int __uverbs_cleanup_ufile(struct ib_uverbs_file *ufile, enum rdma_remove_reason reason); diff --git a/drivers/infiniband/core/uverbs_std_types_dmabuf.c b/drivers/infiniband/core/uverbs_std_types_dmabuf.c index 2411ebee69e2..33bafadc695d 100644 --- a/drivers/infiniband/core/uverbs_std_types_dmabuf.c +++ b/drivers/infiniband/core/uverbs_std_types_dmabuf.c @@ -28,7 +28,7 @@ uverbs_dmabuf_map(struct dma_buf_attachment *attachment, struct ib_uverbs_dmabuf_file *priv = attachment->dmabuf->priv; struct sg_table *ret; - dma_resv_assert_held(priv->dmabuf->resv); + dma_resv_assert_held(attachment->dmabuf->resv); if (priv->revoked) return ERR_PTR(-ENODEV); @@ -49,7 +49,7 @@ static void uverbs_dmabuf_unmap(struct dma_buf_attachment *attachment, { struct ib_uverbs_dmabuf_file *priv = attachment->dmabuf->priv; - dma_resv_assert_held(priv->dmabuf->resv); + dma_resv_assert_held(attachment->dmabuf->resv); dma_buf_free_sgt(attachment, sgt, dir); kref_put(&priv->kref, ib_uverbs_dmabuf_done); } @@ -73,7 +73,25 @@ static void uverbs_dmabuf_release(struct dma_buf *dmabuf) if (!priv->uobj.context) return; + /* + * uverbs_uobject_release() may drop the last reference to the uobject, + * keep it around until the dmabuf pointer below has been cleared. + */ + uverbs_uobject_get(&priv->uobj); uverbs_uobject_release(&priv->uobj); + + /* + * The dma_buf is freed once this function returns. If the uobject was + * not destroyed above, because the ufile is concurrently being cleaned + * up, it is still reachable through the ufile's uobjects list and the + * mmap entry's dmabufs list. Make sure those paths are not in the + * middle of using the dma_buf and cannot use it afterwards. + */ + mutex_lock(&priv->dmabuf_release_lock); + priv->dmabuf = NULL; + mutex_unlock(&priv->dmabuf_release_lock); + + uverbs_uobject_put(&priv->uobj); } static const struct dma_buf_ops uverbs_dmabuf_ops = { @@ -99,6 +117,8 @@ static int UVERBS_HANDLER(UVERBS_METHOD_DMABUF_ALLOC)( off_t pg_off; int ret; + mutex_init(&uverbs_dmabuf->dmabuf_release_lock); + ret = uverbs_get_const(&pg_off, attrs, UVERBS_ATTR_ALLOC_DMABUF_PGOFF); if (ret) return ret; @@ -162,24 +182,11 @@ static void uverbs_dmabuf_fd_destroy_uobj(struct ib_uobject *uobj, { struct ib_uverbs_dmabuf_file *uverbs_dmabuf = container_of(uobj, struct ib_uverbs_dmabuf_file, uobj); - bool wait_for_comp = false; mutex_lock(&uverbs_dmabuf->mmap_entry->dmabufs_lock); - dma_resv_lock(uverbs_dmabuf->dmabuf->resv, NULL); if (!uverbs_dmabuf->revoked) { - uverbs_dmabuf->revoked = true; list_del(&uverbs_dmabuf->dmabufs_elm); - dma_buf_invalidate_mappings(uverbs_dmabuf->dmabuf); - dma_resv_wait_timeout(uverbs_dmabuf->dmabuf->resv, - DMA_RESV_USAGE_BOOKKEEP, false, - MAX_SCHEDULE_TIMEOUT); - wait_for_comp = true; - } - dma_resv_unlock(uverbs_dmabuf->dmabuf->resv); - if (wait_for_comp) { - kref_put(&uverbs_dmabuf->kref, ib_uverbs_dmabuf_done); - /* Let's wait till all DMA unmap are completed. */ - wait_for_completion(&uverbs_dmabuf->comp); + ib_uverbs_dmabuf_revoke(uverbs_dmabuf); } mutex_unlock(&uverbs_dmabuf->mmap_entry->dmabufs_lock); diff --git a/drivers/infiniband/hw/irdma/icrdma_if.c b/drivers/infiniband/hw/irdma/icrdma_if.c index 4b451d8482a4..f405ef761f84 100644 --- a/drivers/infiniband/hw/irdma/icrdma_if.c +++ b/drivers/infiniband/hw/irdma/icrdma_if.c @@ -318,14 +318,15 @@ static void icrdma_remove(struct auxiliary_device *aux_dev) container_of(aux_dev, struct iidc_rdma_core_auxiliary_dev, adev); struct iidc_rdma_core_dev_info *cdev_info = idc_adev->cdev_info; struct irdma_device *iwdev = auxiliary_get_drvdata(aux_dev); - u8 rdma_ver = iwdev->rf->rdma_ver; + struct irdma_pci_f *rf = iwdev->rf; + u8 rdma_ver = rf->rdma_ver; ice_rdma_update_vsi_filter(cdev_info, iwdev->vsi_num, false); irdma_ib_unregister_device(iwdev); - icrdma_deinit_interrupts(iwdev->rf, cdev_info); - mutex_destroy(&iwdev->rf->ah_tbl_lock); + icrdma_deinit_interrupts(rf, cdev_info); + mutex_destroy(&rf->ah_tbl_lock); - kfree(iwdev->rf); + kfree(rf); pr_debug("INIT: Gen[%d] func[%d] device remove success\n", rdma_ver, PCI_FUNC(cdev_info->pdev->devfn)); diff --git a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c index 5732f3ba0122..81baafaffc41 100644 --- a/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c +++ b/drivers/iommu/arm/arm-smmu-v3/arm-smmu-v3.c @@ -5633,7 +5633,7 @@ static void arm_smmu_device_shutdown(struct platform_device *pdev) { struct arm_smmu_device *smmu = platform_get_drvdata(pdev); - arm_smmu_device_disable(smmu); + arm_smmu_disable_action(smmu); } static const struct of_device_id arm_smmu_of_match[] = { diff --git a/drivers/iommu/iommufd/eventq.c b/drivers/iommu/iommufd/eventq.c index 747dd5155121..b8a54454dd17 100644 --- a/drivers/iommu/iommufd/eventq.c +++ b/drivers/iommu/iommufd/eventq.c @@ -396,9 +396,11 @@ static __poll_t iommufd_eventq_fops_poll(struct file *filep, static int iommufd_eventq_fops_release(struct inode *inode, struct file *filep) { struct iommufd_eventq *eventq = filep->private_data; + /* eventq may be freed once the users reference is dropped. */ + struct iommufd_ctx *ictx = eventq->ictx; refcount_dec(&eventq->obj.users); - iommufd_ctx_put(eventq->ictx); + iommufd_ctx_put(ictx); return 0; } diff --git a/drivers/irqchip/irq-gic-v5-irs.c b/drivers/irqchip/irq-gic-v5-irs.c index 4486645a4b6f..fb1cecef613b 100644 --- a/drivers/irqchip/irq-gic-v5-irs.c +++ b/drivers/irqchip/irq-gic-v5-irs.c @@ -609,6 +609,10 @@ static int __init gicv5_irs_of_init_affinity(struct device_node *node, u16 iaffid_mask = GENMASK(iaffid_bits - 1, 0); int ret, i, ncpus, niaffids; + /* An IRS is architecturally allowed to have no cpus associated with it */ + if (!of_property_present(node, "cpus") && !of_property_present(node, "arm,iaffids")) + return 0; + ncpus = of_count_phandle_with_args(node, "cpus", NULL); if (ncpus < 0) return -EINVAL; diff --git a/drivers/media/pci/cx23885/cx23885-core.c b/drivers/media/pci/cx23885/cx23885-core.c index 5fb26285e4af..7498091176b7 100644 --- a/drivers/media/pci/cx23885/cx23885-core.c +++ b/drivers/media/pci/cx23885/cx23885-core.c @@ -2246,6 +2246,10 @@ static void cx23885_finidev(struct pci_dev *pci_dev) /* unregister stuff */ free_irq(pci_dev->irq, dev); + cancel_work_sync(&dev->cx25840_work); + cancel_work_sync(&dev->ir_rx_work); + cancel_work_sync(&dev->ir_tx_work); + pci_disable_device(pci_dev); cx23885_dev_unregister(dev); diff --git a/drivers/media/pci/cx23885/cx23885-dvb.c b/drivers/media/pci/cx23885/cx23885-dvb.c index f240ccda40ed..348abde0ba26 100644 --- a/drivers/media/pci/cx23885/cx23885-dvb.c +++ b/drivers/media/pci/cx23885/cx23885-dvb.c @@ -1158,8 +1158,12 @@ static int dvb_register_ci_mac(struct cx23885_tsport *port) info.platform_data = &sp2_config; request_module(info.type); client_ci = i2c_new_client_device(&i2c_bus->i2c_adap, &info); - if (!i2c_client_has_driver(client_ci)) + if (IS_ERR(client_ci)) + return PTR_ERR(client_ci); + if (!client_ci->dev.driver) { + i2c_unregister_device(client_ci); return -ENODEV; + } if (!try_module_get(client_ci->dev.driver->owner)) { i2c_unregister_device(client_ci); return -ENODEV; @@ -1202,7 +1206,7 @@ static int dvb_register(struct cx23885_tsport *port) int (*p_set_voltage)(struct dvb_frontend *fe, enum fe_sec_voltage voltage) = NULL; int mfe_shared = 0; /* bus not shared by default */ - int ret; + int ret = -EINVAL; /* Get the first frontend */ fe0 = vb2_dvb_get_frontend(&port->frontends, 1); @@ -2586,8 +2590,10 @@ static int dvb_register(struct cx23885_tsport *port) goto frontend_detach; ret = dvb_register_ci_mac(port); - if (ret) + if (ret) { + vb2_dvb_unregister_bus(&port->frontends); goto frontend_detach; + } return 0; @@ -2618,7 +2624,7 @@ static int dvb_register(struct cx23885_tsport *port) port->gate_ctrl = NULL; vb2_dvb_dealloc_frontends(&port->frontends); - return -EINVAL; + return ret; } int cx23885_dvb_register(struct cx23885_tsport *port) diff --git a/drivers/media/pci/cx88/cx88-mpeg.c b/drivers/media/pci/cx88/cx88-mpeg.c index 0c07ad335799..cd2366768f4a 100644 --- a/drivers/media/pci/cx88/cx88-mpeg.c +++ b/drivers/media/pci/cx88/cx88-mpeg.c @@ -392,7 +392,8 @@ static int cx8802_init_common(struct cx8802_dev *dev) err = dma_set_mask(&dev->pci->dev, DMA_BIT_MASK(32)); if (err) { pr_err("Oops: no 32bit PCI DMA ???\n"); - return -EIO; + err = -EIO; + goto fail_disable_device; } dev->pci_rev = dev->pci->revision; @@ -413,13 +414,17 @@ static int cx8802_init_common(struct cx8802_dev *dev) IRQF_SHARED, dev->core->name, dev); if (err < 0) { pr_err("can't get IRQ %d\n", dev->pci->irq); - return err; + goto fail_disable_device; } cx_set(MO_PCI_INTMSK, core->pci_irqmask); /* everything worked */ pci_set_drvdata(dev->pci, dev); return 0; + +fail_disable_device: + pci_disable_device(dev->pci); + return err; } static void cx8802_fini_common(struct cx8802_dev *dev) diff --git a/drivers/media/pci/intel/ipu-bridge.c b/drivers/media/pci/intel/ipu-bridge.c index bd64c0400c0d..4d292ee2be59 100644 --- a/drivers/media/pci/intel/ipu-bridge.c +++ b/drivers/media/pci/intel/ipu-bridge.c @@ -326,8 +326,8 @@ static int ipu_bridge_check_ivsc_dev(struct ipu_sensor *sensor, if (adev) { csi_dev = ipu_bridge_get_ivsc_csi_dev(adev); if (!csi_dev) { - acpi_dev_put(adev); dev_err(ADEV_DEV(adev), "Failed to find MEI or CVS CSI dev\n"); + acpi_dev_put(adev); return -ENODEV; } diff --git a/drivers/media/platform/marvell/mcam-core.c b/drivers/media/platform/marvell/mcam-core.c index b8360d37000a..c9605317e8a7 100644 --- a/drivers/media/platform/marvell/mcam-core.c +++ b/drivers/media/platform/marvell/mcam-core.c @@ -406,6 +406,8 @@ static void mcam_free_dma_bufs(struct mcam_camera *cam) { int i; + cancel_work_sync(&cam->s_bh_work); + for (i = 0; i < cam->nbufs; i++) { dma_free_coherent(cam->dev, cam->dma_buf_size, cam->dma_bufs[i], cam->dma_handles[i]); @@ -1306,7 +1308,6 @@ static int mcam_setup_vb2(struct mcam_camera *cam) break; case B_vmalloc: #ifdef MCAM_MODE_VMALLOC - INIT_WORK(&cam->s_bh_work, mcam_frame_work); vq->ops = &mcam_vb2_ops; vq->mem_ops = &vb2_vmalloc_memops; cam->dma_setup = mcam_ctlr_dma_vmalloc; @@ -1864,6 +1865,12 @@ int mccic_register(struct mcam_camera *cam) goto out; } +#ifdef MCAM_MODE_VMALLOC + /* Init before sensor bind: armed by IRQ, cancelled on probe-error paths. */ + if (cam->buffer_mode == B_vmalloc) + INIT_WORK(&cam->s_bh_work, mcam_frame_work); +#endif + mutex_init(&cam->s_mutex); cam->state = S_NOTREADY; mcam_set_config_needed(cam, 1); @@ -1922,10 +1929,15 @@ void mccic_shutdown(struct mcam_camera *cam) * take it down again will wedge the machine, which is frowned * upon. */ + mutex_lock(&cam->s_mutex); if (!list_empty(&cam->vdev.fh_list)) { cam_warn(cam, "Removing a device with users!\n"); + /* Stop so the IRQ can't re-arm s_bh_work after the buffers are freed. */ + if (cam->state == S_STREAMING) + mcam_ctlr_stop_dma(cam); sensor_call(cam, core, s_power, 0); } + mutex_unlock(&cam->s_mutex); if (cam->buffer_mode == B_vmalloc) mcam_free_dma_bufs(cam); v4l2_ctrl_handler_free(&cam->ctrl_handler); diff --git a/drivers/media/platform/microchip/microchip-csi2dc.c b/drivers/media/platform/microchip/microchip-csi2dc.c index e69292f3b2a9..b2c17ff2322f 100644 --- a/drivers/media/platform/microchip/microchip-csi2dc.c +++ b/drivers/media/platform/microchip/microchip-csi2dc.c @@ -735,6 +735,7 @@ static int csi2dc_probe(struct platform_device *pdev) return 0; csi2dc_probe_cleanup_notifier: + v4l2_async_nf_unregister(&csi2dc->notifier); v4l2_async_nf_cleanup(&csi2dc->notifier); csi2dc_probe_cleanup_entity: media_entity_cleanup(&csi2dc->csi2dc_sd.entity); diff --git a/drivers/media/platform/microchip/microchip-isc-base.c b/drivers/media/platform/microchip/microchip-isc-base.c index a7cdc743fda7..32bdeae04ad0 100644 --- a/drivers/media/platform/microchip/microchip-isc-base.c +++ b/drivers/media/platform/microchip/microchip-isc-base.c @@ -1860,6 +1860,12 @@ void microchip_isc_subdev_cleanup(struct isc_device *isc) list_for_each_entry(subdev_entity, &isc->subdev_entities, list) { v4l2_async_nf_unregister(&subdev_entity->notifier); v4l2_async_nf_cleanup(&subdev_entity->notifier); + /* + * Release the endpoint reference taken while parsing. It is + * NULL for entities the bind loop already consumed, so this + * only drops the ones left over on an early exit. + */ + of_node_put(subdev_entity->epn); } INIT_LIST_HEAD(&isc->subdev_entities); diff --git a/drivers/media/platform/microchip/microchip-sama5d2-isc.c b/drivers/media/platform/microchip/microchip-sama5d2-isc.c index 66d3d7891991..88e6d8e1c6c9 100644 --- a/drivers/media/platform/microchip/microchip-sama5d2-isc.c +++ b/drivers/media/platform/microchip/microchip-sama5d2-isc.c @@ -356,28 +356,30 @@ static int isc_parse_dt(struct device *dev, struct isc_device *isc) struct device_node *epn; struct isc_subdev_entity *subdev_entity; unsigned int flags; + int ret; INIT_LIST_HEAD(&isc->subdev_entities); for_each_endpoint_of_node(np, epn) { struct v4l2_fwnode_endpoint v4l2_epn = { .bus_type = 0 }; - int ret; ret = v4l2_fwnode_endpoint_parse(of_fwnode_handle(epn), &v4l2_epn); if (ret) { - of_node_put(epn); dev_err(dev, "Could not parse the endpoint\n"); - return -EINVAL; + of_node_put(epn); + ret = -EINVAL; + goto err_cleanup; } subdev_entity = devm_kzalloc(dev, sizeof(*subdev_entity), GFP_KERNEL); if (!subdev_entity) { of_node_put(epn); - return -ENOMEM; + ret = -ENOMEM; + goto err_cleanup; } - subdev_entity->epn = epn; + subdev_entity->epn = of_node_get(epn); flags = v4l2_epn.bus.parallel.flags; @@ -398,6 +400,11 @@ static int isc_parse_dt(struct device *dev, struct isc_device *isc) } return 0; + +err_cleanup: + list_for_each_entry(subdev_entity, &isc->subdev_entities, list) + of_node_put(subdev_entity->epn); + return ret; } static int microchip_isc_probe(struct platform_device *pdev) diff --git a/drivers/media/platform/microchip/microchip-sama7g5-isc.c b/drivers/media/platform/microchip/microchip-sama7g5-isc.c index 7383341ec51d..77c6c183451b 100644 --- a/drivers/media/platform/microchip/microchip-sama7g5-isc.c +++ b/drivers/media/platform/microchip/microchip-sama7g5-isc.c @@ -340,6 +340,7 @@ static int xisc_parse_dt(struct device *dev, struct isc_device *isc) struct isc_subdev_entity *subdev_entity; unsigned int flags; bool mipi_mode; + int ret; INIT_LIST_HEAD(&isc->subdev_entities); @@ -347,23 +348,24 @@ static int xisc_parse_dt(struct device *dev, struct isc_device *isc) for_each_endpoint_of_node(np, epn) { struct v4l2_fwnode_endpoint v4l2_epn = { .bus_type = 0 }; - int ret; ret = v4l2_fwnode_endpoint_parse(of_fwnode_handle(epn), &v4l2_epn); if (ret) { - of_node_put(epn); dev_err(dev, "Could not parse the endpoint\n"); - return -EINVAL; + of_node_put(epn); + ret = -EINVAL; + goto err_cleanup; } subdev_entity = devm_kzalloc(dev, sizeof(*subdev_entity), GFP_KERNEL); if (!subdev_entity) { of_node_put(epn); - return -ENOMEM; + ret = -ENOMEM; + goto err_cleanup; } - subdev_entity->epn = epn; + subdev_entity->epn = of_node_get(epn); flags = v4l2_epn.bus.parallel.flags; @@ -387,6 +389,11 @@ static int xisc_parse_dt(struct device *dev, struct isc_device *isc) } return 0; + +err_cleanup: + list_for_each_entry(subdev_entity, &isc->subdev_entities, list) + of_node_put(subdev_entity->epn); + return ret; } static int microchip_xisc_probe(struct platform_device *pdev) diff --git a/drivers/media/platform/nxp/imx-jpeg/mxc-jpeg.c b/drivers/media/platform/nxp/imx-jpeg/mxc-jpeg.c index 725e94152884..ea2fad5dd646 100644 --- a/drivers/media/platform/nxp/imx-jpeg/mxc-jpeg.c +++ b/drivers/media/platform/nxp/imx-jpeg/mxc-jpeg.c @@ -1798,6 +1798,8 @@ static void mxc_jpeg_stop_streaming(struct vb2_queue *q) dev_dbg(ctx->mxc_jpeg->dev, "Stop streaming ctx=%p", ctx); + cancel_delayed_work_sync(&ctx->task_timer); + /* Release all active buffers */ for (;;) { if (V4L2_TYPE_IS_OUTPUT(q->type)) diff --git a/drivers/media/platform/samsung/s5p-mfc/s5p_mfc.c b/drivers/media/platform/samsung/s5p-mfc/s5p_mfc.c index 32eb402d439c..03e7eac8d1f7 100644 --- a/drivers/media/platform/samsung/s5p-mfc/s5p_mfc.c +++ b/drivers/media/platform/samsung/s5p-mfc/s5p_mfc.c @@ -1474,8 +1474,8 @@ static void s5p_mfc_remove(struct platform_device *pdev) } mutex_unlock(&dev->mfc_mutex); - timer_delete_sync(&dev->watchdog_timer); - flush_work(&dev->watchdog_work); + timer_shutdown_sync(&dev->watchdog_timer); + cancel_work_sync(&dev->watchdog_work); video_unregister_device(dev->vfd_enc); video_unregister_device(dev->vfd_dec); diff --git a/drivers/media/rc/bpf-lirc.c b/drivers/media/rc/bpf-lirc.c index 2f7564f26445..14ab611e7445 100644 --- a/drivers/media/rc/bpf-lirc.c +++ b/drivers/media/rc/bpf-lirc.c @@ -148,12 +148,13 @@ static int lirc_bpf_attach(struct rc_dev *rcdev, struct bpf_prog *prog) if (ret) return ret; - raw = rcdev->raw; - if (!raw) { + if (!rcdev->registered) { ret = -ENODEV; goto unlock; } + raw = rcdev->raw; + old_array = lirc_rcu_dereference(raw->progs); if (old_array && bpf_prog_array_length(old_array) >= BPF_MAX_PROGS) { ret = -E2BIG; @@ -186,12 +187,13 @@ static int lirc_bpf_detach(struct rc_dev *rcdev, struct bpf_prog *prog) if (ret) return ret; - raw = rcdev->raw; - if (!raw) { + if (!rcdev->registered) { ret = -ENODEV; goto unlock; } + raw = rcdev->raw; + old_array = lirc_rcu_dereference(raw->progs); ret = bpf_prog_array_copy(old_array, prog, NULL, 0, &new_array); /* @@ -235,7 +237,8 @@ void lirc_bpf_free(struct rc_dev *rcdev) struct bpf_prog_array_item *item; struct bpf_prog_array *array; - array = lirc_rcu_dereference(rcdev->raw->progs); + array = rcu_replace_pointer(rcdev->raw->progs, NULL, + lockdep_is_held(&ir_raw_handler_lock)); if (!array) return; @@ -316,6 +319,11 @@ int lirc_prog_query(const union bpf_attr *attr, union bpf_attr __user *uattr) if (ret) goto put; + if (!rcdev->registered) { + ret = -ENODEV; + goto unlock; + } + progs = lirc_rcu_dereference(rcdev->raw->progs); cnt = progs ? bpf_prog_array_length(progs) : 0; diff --git a/drivers/media/rc/fintek-cir.c b/drivers/media/rc/fintek-cir.c index c196ee923ecd..ce9c4eb4514e 100644 --- a/drivers/media/rc/fintek-cir.c +++ b/drivers/media/rc/fintek-cir.c @@ -340,6 +340,10 @@ static void fintek_get_rx_ir_data(struct fintek_dev *fintek, u8 rx_irqs) sample = fintek_cir_reg_read(fintek, CIR_RX_DATA); fit_dbg("%s: sample: 0x%02x", __func__, sample); + /* Process a full buffer before storing the next sample. */ + if (fintek->pkts >= RX_BUF_LEN) + fintek_process_rx_ir_data(fintek); + fintek->buf[fintek->pkts] = sample; fintek->pkts++; diff --git a/drivers/media/rc/imon.c b/drivers/media/rc/imon.c index 049a73b5f882..e54b588ff61c 100644 --- a/drivers/media/rc/imon.c +++ b/drivers/media/rc/imon.c @@ -492,16 +492,12 @@ MODULE_PARM_DESC(pad_thresh, "Threshold at which a pad push registers as an arro static void free_imon_context(struct imon_context *ictx) { - struct device *dev = ictx->dev; - usb_free_urb(ictx->tx_urb); WARN_ON(ictx->dev_present_intf0); usb_free_urb(ictx->rx_urb_intf0); WARN_ON(ictx->dev_present_intf1); usb_free_urb(ictx->rx_urb_intf1); kfree_rcu(ictx, rcu); - - dev_dbg(dev, "%s: iMON context freed\n", __func__); } /* @@ -577,7 +573,6 @@ static int display_close(struct inode *inode, struct file *file) retval = -EIO; } else { ictx->display_isopen = false; - dev_dbg(ictx->dev, "display port closed\n"); } mutex_unlock(&ictx->lock); diff --git a/drivers/media/rc/ir-mce_kbd-decoder.c b/drivers/media/rc/ir-mce_kbd-decoder.c index bb2d7c37c263..d1c9263511b0 100644 --- a/drivers/media/rc/ir-mce_kbd-decoder.c +++ b/drivers/media/rc/ir-mce_kbd-decoder.c @@ -219,6 +219,7 @@ static int ir_mce_kbd_decode(struct rc_dev *dev, struct ir_raw_event ev) struct mce_kbd_dec *data = &dev->raw->mce_kbd; u32 scancode; unsigned long delay; + unsigned long flags; struct lirc_scancode lsc = {}; if (!is_timing_event(ev)) { @@ -319,7 +320,7 @@ static int ir_mce_kbd_decode(struct rc_dev *dev, struct ir_raw_event ev) scancode = data->body & 0xffffff; dev_dbg(&dev->dev, "keyboard data 0x%08x\n", data->body); - spin_lock(&data->keylock); + spin_lock_irqsave(&data->keylock, flags); if (scancode) { delay = usecs_to_jiffies(dev->timeout) + msecs_to_jiffies(100); @@ -329,7 +330,7 @@ static int ir_mce_kbd_decode(struct rc_dev *dev, struct ir_raw_event ev) } /* Pass data to keyboard buffer parser */ ir_mce_kbd_process_keyboard_data(dev, scancode); - spin_unlock(&data->keylock); + spin_unlock_irqrestore(&data->keylock, flags); lsc.rc_proto = RC_PROTO_MCIR2_KBD; break; case MCIR2_MOUSE_NBITS: diff --git a/drivers/media/rc/lirc_dev.c b/drivers/media/rc/lirc_dev.c index 183f1939b941..9a97383dd9b5 100644 --- a/drivers/media/rc/lirc_dev.c +++ b/drivers/media/rc/lirc_dev.c @@ -775,6 +775,9 @@ void lirc_unregister(struct rc_dev *dev) unsigned long flags; struct lirc_fh *fh; + if (!device_is_registered(&dev->lirc_dev)) + return; + dev_dbg(&dev->dev, "lirc_dev: driver %s unregistered from minor = %d\n", dev->driver_name, MINOR(dev->lirc_dev.devt)); diff --git a/drivers/media/rc/rc-ir-raw.c b/drivers/media/rc/rc-ir-raw.c index ba24c2f22d39..75dee4a483d4 100644 --- a/drivers/media/rc/rc-ir-raw.c +++ b/drivers/media/rc/rc-ir-raw.c @@ -10,7 +10,8 @@ #include #include "rc-core-priv.h" -/* Used to keep track of IR raw clients, protected by ir_raw_handler_lock */ +/* Used to keep track of IR raw clients, protected by ir_raw_client_lock */ +static DEFINE_MUTEX(ir_raw_client_lock); static LIST_HEAD(ir_raw_client_list); /* Used to handle IR raw handler extensions */ @@ -71,9 +72,6 @@ static int ir_raw_event_thread(void *data) */ int ir_raw_event_store(struct rc_dev *dev, struct ir_raw_event *ev) { - if (!dev->raw) - return -EINVAL; - dev_dbg(&dev->dev, "sample: (%05dus %s)\n", ev->duration, TO_STR(ev->pulse)); @@ -102,9 +100,6 @@ int ir_raw_event_store_edge(struct rc_dev *dev, bool pulse) ktime_t now; struct ir_raw_event ev = {}; - if (!dev->raw) - return -EINVAL; - now = ktime_get(); ev.duration = ktime_to_us(ktime_sub(now, dev->raw->last_event)); ev.pulse = !pulse; @@ -129,9 +124,6 @@ int ir_raw_event_store_with_timeout(struct rc_dev *dev, struct ir_raw_event *ev) ktime_t now; int rc = 0; - if (!dev->raw) - return -EINVAL; - now = ktime_get(); spin_lock(&dev->raw->edge_spinlock); @@ -166,9 +158,6 @@ EXPORT_SYMBOL_GPL(ir_raw_event_store_with_timeout); */ int ir_raw_event_store_with_filter(struct rc_dev *dev, struct ir_raw_event *ev) { - if (!dev->raw) - return -EINVAL; - /* Ignore spaces in idle mode */ if (dev->idle && !ev->pulse) return 0; @@ -200,9 +189,6 @@ EXPORT_SYMBOL_GPL(ir_raw_event_store_with_filter); */ void ir_raw_event_set_idle(struct rc_dev *dev, bool idle) { - if (!dev->raw) - return; - dev_dbg(&dev->dev, "%s idle mode\n", idle ? "enter" : "leave"); if (idle) { @@ -226,7 +212,7 @@ EXPORT_SYMBOL_GPL(ir_raw_event_set_idle); */ void ir_raw_event_handle(struct rc_dev *dev) { - if (!dev->raw || !dev->raw->thread) + if (!dev->raw->thread) return; wake_up_process(dev->raw->thread); @@ -288,13 +274,6 @@ static int change_protocol(struct rc_dev *dev, u64 *rc_proto) return 0; } -static void ir_raw_disable_protocols(struct rc_dev *dev, u64 protocols) -{ - mutex_lock(&dev->lock); - dev->enabled_protocols &= ~protocols; - mutex_unlock(&dev->lock); -} - /** * ir_raw_gen_manchester() - Encode data with Manchester (bi-phase) modulation. * @ev: Pointer to pointer to next free event. *@ev is incremented for @@ -612,9 +591,6 @@ EXPORT_SYMBOL(ir_raw_encode_carrier); */ int ir_raw_event_prepare(struct rc_dev *dev) { - if (!dev) - return -EINVAL; - dev->raw = kzalloc_obj(*dev->raw); if (!dev->raw) return -ENOMEM; @@ -633,50 +609,63 @@ int ir_raw_event_register(struct rc_dev *dev) { struct task_struct *thread; + /* Holding dev->lock could result in a dead-lock */ + lockdep_assert_not_held(&dev->lock); + thread = kthread_run(ir_raw_event_thread, dev->raw, "rc%u", dev->minor); if (IS_ERR(thread)) return PTR_ERR(thread); dev->raw->thread = thread; - mutex_lock(&ir_raw_handler_lock); + mutex_lock(&ir_raw_client_lock); list_add_tail(&dev->raw->list, &ir_raw_client_list); - mutex_unlock(&ir_raw_handler_lock); + mutex_unlock(&ir_raw_client_lock); return 0; } void ir_raw_event_free(struct rc_dev *dev) { - kfree(dev->raw); - dev->raw = NULL; + if (dev->raw) { + timer_delete_sync(&dev->raw->edge_handle); + mutex_lock(&ir_raw_handler_lock); + if (dev->raw->thread) + put_task_struct(dev->raw->thread); + lirc_bpf_free(dev); + mutex_unlock(&ir_raw_handler_lock); + kfree(dev->raw); + dev->raw = NULL; + } } void ir_raw_event_unregister(struct rc_dev *dev) { struct ir_raw_handler *handler; - if (!dev || !dev->raw) - return; - + /* + * After ir_raw_event_unregister() is called, an sync + * call to ir_raw_event_handle() can still arrive. This function + * may call wake_up_process(dev->raw->thread). Ensure this memory + * is not freed by kthread_stop(). + */ + get_task_struct(dev->raw->thread); kthread_stop(dev->raw->thread); timer_delete_sync(&dev->raw->edge_handle); - mutex_lock(&ir_raw_handler_lock); + mutex_lock(&ir_raw_client_lock); list_del(&dev->raw->list); + + mutex_lock(&ir_raw_handler_lock); list_for_each_entry(handler, &ir_raw_handler_list, list) if (handler->raw_unregister && (handler->protocols & dev->enabled_protocols)) handler->raw_unregister(dev); lirc_bpf_free(dev); - - /* - * A user can be calling bpf(BPF_PROG_{QUERY|ATTACH|DETACH}), so - * ensure that the raw member is null on unlock; this is how - * "device gone" is checked. - */ mutex_unlock(&ir_raw_handler_lock); + + mutex_unlock(&ir_raw_client_lock); } /* @@ -699,15 +688,24 @@ void ir_raw_handler_unregister(struct ir_raw_handler *ir_raw_handler) struct ir_raw_event_ctrl *raw; u64 protocols = ir_raw_handler->protocols; + mutex_lock(&ir_raw_client_lock); + mutex_lock(&ir_raw_handler_lock); list_del(&ir_raw_handler->list); + atomic64_andnot(protocols, &available_protocols); + mutex_unlock(&ir_raw_handler_lock); + list_for_each_entry(raw, &ir_raw_client_list, list) { + mutex_lock(&raw->dev->lock); + mutex_lock(&ir_raw_handler_lock); if (ir_raw_handler->raw_unregister && (raw->dev->enabled_protocols & protocols)) ir_raw_handler->raw_unregister(raw->dev); - ir_raw_disable_protocols(raw->dev, protocols); + raw->dev->enabled_protocols &= ~protocols; + mutex_unlock(&ir_raw_handler_lock); + mutex_unlock(&raw->dev->lock); } - atomic64_andnot(protocols, &available_protocols); - mutex_unlock(&ir_raw_handler_lock); + + mutex_unlock(&ir_raw_client_lock); } EXPORT_SYMBOL(ir_raw_handler_unregister); diff --git a/drivers/media/rc/rc-main.c b/drivers/media/rc/rc-main.c index dda3479ea3ad..85c9be43b912 100644 --- a/drivers/media/rc/rc-main.c +++ b/drivers/media/rc/rc-main.c @@ -1701,14 +1701,24 @@ static const struct device_type rc_dev_type = { struct rc_dev *rc_allocate_device(enum rc_driver_type type) { struct rc_dev *dev; + int ret; dev = kzalloc_obj(*dev); if (!dev) return NULL; + if (type == RC_DRIVER_IR_RAW) { + ret = ir_raw_event_prepare(dev); + if (ret < 0) { + kfree(dev); + return NULL; + } + } + if (type != RC_DRIVER_IR_RAW_TX) { dev->input_dev = input_allocate_device(); if (!dev->input_dev) { + ir_raw_event_free(dev); kfree(dev); return NULL; } @@ -1724,6 +1734,7 @@ struct rc_dev *rc_allocate_device(enum rc_driver_type type) spin_lock_init(&dev->rc_map.lock); spin_lock_init(&dev->keylock); } + mutex_init(&dev->lock); dev->dev.type = &rc_dev_type; @@ -1742,7 +1753,12 @@ void rc_free_device(struct rc_dev *dev) if (!dev) return; - input_free_device(dev->input_dev); + if (dev->input_dev) { + timer_delete_sync(&dev->timer_keyup); + timer_delete_sync(&dev->timer_repeat); + } + + input_put_device(dev->input_dev); put_device(&dev->dev); @@ -1854,6 +1870,8 @@ static int rc_setup_rx_device(struct rc_dev *dev) if (rc) return rc; + input_get_device(dev->input_dev); + /* * Default delay of 250ms is too short for some protocols, especially * since the timeout is currently set to 250ms. Increase it to 500ms, @@ -1880,10 +1898,8 @@ static void rc_free_rx_device(struct rc_dev *dev) if (!dev) return; - if (dev->input_dev) { + if (dev->input_dev) input_unregister_device(dev->input_dev); - dev->input_dev = NULL; - } ir_free_table(&dev->rc_map); } @@ -1917,19 +1933,14 @@ int rc_register_device(struct rc_dev *dev) dev->sysfs_groups[attr++] = &rc_dev_wakeup_filter_attr_grp; dev->sysfs_groups[attr++] = NULL; - if (dev->driver_type == RC_DRIVER_IR_RAW) { - rc = ir_raw_event_prepare(dev); - if (rc < 0) - goto out_minor; - } - if (dev->driver_type != RC_DRIVER_IR_RAW_TX) { rc = rc_prepare_rx_device(dev); if (rc) goto out_raw; } - dev->registered = true; + scoped_guard(mutex, &dev->lock) + dev->registered = true; rc = device_add(&dev->dev); if (rc) @@ -1949,7 +1960,7 @@ int rc_register_device(struct rc_dev *dev) if (dev->allowed_protocols != RC_PROTO_BIT_CEC) { rc = lirc_register(dev); if (rc < 0) - goto out_dev; + goto out_lirc; } if (dev->driver_type != RC_DRIVER_IR_RAW_TX) { @@ -1972,15 +1983,19 @@ int rc_register_device(struct rc_dev *dev) out_rx: rc_free_rx_device(dev); out_lirc: - if (dev->allowed_protocols != RC_PROTO_BIT_CEC) - lirc_unregister(dev); -out_dev: + scoped_guard(mutex, &dev->lock) + dev->registered = false; + + lirc_unregister(dev); device_del(&dev->dev); + /* registered already cleared above */ + goto out_free_table; out_rx_free: + scoped_guard(mutex, &dev->lock) + dev->registered = false; +out_free_table: ir_free_table(&dev->rc_map); out_raw: - ir_raw_event_free(dev); -out_minor: ida_free(&rc_ida, minor); return rc; } @@ -2018,18 +2033,18 @@ void rc_unregister_device(struct rc_dev *dev) if (!dev) return; + mutex_lock(&dev->lock); + dev->registered = false; + if (dev->users && dev->close) + dev->close(dev); + mutex_unlock(&dev->lock); + if (dev->driver_type == RC_DRIVER_IR_RAW) ir_raw_event_unregister(dev); timer_delete_sync(&dev->timer_keyup); timer_delete_sync(&dev->timer_repeat); - mutex_lock(&dev->lock); - if (dev->users && dev->close) - dev->close(dev); - dev->registered = false; - mutex_unlock(&dev->lock); - rc_free_rx_device(dev); /* diff --git a/drivers/media/rc/redrat3.c b/drivers/media/rc/redrat3.c index 3f828a564e19..565c490b7051 100644 --- a/drivers/media/rc/redrat3.c +++ b/drivers/media/rc/redrat3.c @@ -975,6 +975,7 @@ static int redrat3_dev_probe(struct usb_interface *intf, struct device *dev = &intf->dev; struct usb_host_interface *uhi; struct redrat3_dev *rr3; + struct rc_dev *rc; struct usb_endpoint_descriptor *ep; struct usb_endpoint_descriptor *ep_narrow = NULL; struct usb_endpoint_descriptor *ep_wide = NULL; @@ -1119,9 +1120,12 @@ static int redrat3_dev_probe(struct usb_interface *intf, return 0; led_free: + rc_unregister_device(rr3->rc); led_classdev_unregister(&rr3->led); redrat_free: + rc = rr3->rc; redrat3_delete(rr3, rr3->udev); + rc_free_device(rc); no_endpoints: return retval; diff --git a/drivers/media/usb/au0828/au0828-dvb.c b/drivers/media/usb/au0828/au0828-dvb.c index 9c95b7ceaecd..d3fff47f0edf 100644 --- a/drivers/media/usb/au0828/au0828-dvb.c +++ b/drivers/media/usb/au0828/au0828-dvb.c @@ -537,6 +537,7 @@ void au0828_dvb_unregister(struct au0828_dev *dev) if (dvb->frontend == NULL) return; + timer_shutdown_sync(&dev->bulk_timeout); cancel_work_sync(&dev->restart_streaming); dvb_net_release(&dvb->net); diff --git a/drivers/media/usb/em28xx/em28xx-audio.c b/drivers/media/usb/em28xx/em28xx-audio.c index ccff6be5599e..7f0c5826ce5e 100644 --- a/drivers/media/usb/em28xx/em28xx-audio.c +++ b/drivers/media/usb/em28xx/em28xx-audio.c @@ -693,6 +693,13 @@ static int em28xx_audio_urb_init(struct em28xx *dev) } ep_size = em28xx_audio_ep_packet_size(udev, ep); + if (ep_size == 0) { + dev_err(&dev->intf->dev, + "Audio endpoint 0x%02x has a zero packet size\n", + EM28XX_EP_AUDIO); + return -ENODEV; + } + interval = 1 << (ep->bInterval - 1); dev_info(&dev->intf->dev, diff --git a/drivers/media/usb/go7007/go7007-driver.c b/drivers/media/usb/go7007/go7007-driver.c index 453ab5c3aa03..7db2440886b0 100644 --- a/drivers/media/usb/go7007/go7007-driver.c +++ b/drivers/media/usb/go7007/go7007-driver.c @@ -306,10 +306,9 @@ int go7007_register_encoder(struct go7007 *go, unsigned num_i2c_devs) if (ret < 0) goto err_free_controls; - if (go->board_info->flags & GO7007_BOARD_HAS_AUDIO) { + if ((go->board_info->flags & GO7007_BOARD_HAS_AUDIO) && + go7007_snd_init(go) == 0) go->audio_enabled = 1; - go7007_snd_init(go); - } return 0; err_free_controls: diff --git a/drivers/media/usb/gspca/gspca.c b/drivers/media/usb/gspca/gspca.c index 594d73e50b9f..4185c9047241 100644 --- a/drivers/media/usb/gspca/gspca.c +++ b/drivers/media/usb/gspca/gspca.c @@ -1636,9 +1636,8 @@ void gspca_disconnect(struct usb_interface *intf) #endif v4l2_device_disconnect(&gspca_dev->v4l2_dev); - video_unregister_device(&gspca_dev->vdev); - mutex_unlock(&gspca_dev->usb_lock); + vb2_video_unregister_device(&gspca_dev->vdev); /* (this will call gspca_release() immediately or on last close) */ v4l2_device_put(&gspca_dev->v4l2_dev); diff --git a/drivers/media/usb/hackrf/hackrf.c b/drivers/media/usb/hackrf/hackrf.c index a15829a60e88..70fd95f3e97d 100644 --- a/drivers/media/usb/hackrf/hackrf.c +++ b/drivers/media/usb/hackrf/hackrf.c @@ -665,7 +665,7 @@ static int hackrf_free_urbs(struct hackrf_dev *dev) static int hackrf_alloc_urbs(struct hackrf_dev *dev, bool rcv) { - int i, j; + int i; unsigned int pipe; usb_complete_t complete; @@ -681,11 +681,8 @@ static int hackrf_alloc_urbs(struct hackrf_dev *dev, bool rcv) for (i = 0; i < MAX_BULK_BUFS; i++) { dev_dbg(dev->dev, "alloc urb=%d\n", i); dev->urb_list[i] = usb_alloc_urb(0, GFP_KERNEL); - if (!dev->urb_list[i]) { - for (j = 0; j < i; j++) - usb_free_urb(dev->urb_list[j]); + if (!dev->urb_list[i]) return -ENOMEM; - } usb_fill_bulk_urb(dev->urb_list[i], dev->udev, pipe, diff --git a/drivers/media/usb/usbtv/usbtv-core.c b/drivers/media/usb/usbtv/usbtv-core.c index 4f10f6613bc4..89c3424e7685 100644 --- a/drivers/media/usb/usbtv/usbtv-core.c +++ b/drivers/media/usb/usbtv/usbtv-core.c @@ -139,8 +139,6 @@ static void usbtv_disconnect(struct usb_interface *intf) usbtv_audio_free(usbtv); usbtv_video_free(usbtv); - usbtv->udev = NULL; - /* the usbtv structure will be deallocated when v4l2 will be done using it */ v4l2_device_put(&usbtv->v4l2_dev); diff --git a/drivers/media/usb/usbtv/usbtv-video.c b/drivers/media/usb/usbtv/usbtv-video.c index 92bc7a2509c3..2a91af8d1e30 100644 --- a/drivers/media/usb/usbtv/usbtv-video.c +++ b/drivers/media/usb/usbtv/usbtv-video.c @@ -966,5 +966,9 @@ void usbtv_video_free(struct usbtv *usbtv) vb2_video_unregister_device(&usbtv->vdev); v4l2_device_disconnect(&usbtv->v4l2_dev); + mutex_lock(&usbtv->v4l2_lock); + usbtv->udev = NULL; + mutex_unlock(&usbtv->v4l2_lock); + v4l2_device_put(&usbtv->v4l2_dev); } diff --git a/drivers/mtd/mtd_virt_concat.c b/drivers/mtd/mtd_virt_concat.c index 25cf33fe1ec1..40b300021dda 100644 --- a/drivers/mtd/mtd_virt_concat.c +++ b/drivers/mtd/mtd_virt_concat.c @@ -130,6 +130,7 @@ int mtd_virt_concat_destroy(struct mtd_info *mtd) mtd_concat_destroy(&concat->mtd); } + list_del(&item->head); for (idx = 0; idx < item->count; idx++) of_node_put(item->nodes[idx]); @@ -201,6 +202,7 @@ void mtd_virt_concat_destroy_items(void) int i; list_for_each_entry_safe(item, temp, &concat_node_list, head) { + list_del(&item->head); for (i = 0; i < item->count; i++) of_node_put(item->nodes[i]); diff --git a/drivers/net/tun.c b/drivers/net/tun.c index 242899f7fd07..7cca1a6f72ff 100644 --- a/drivers/net/tun.c +++ b/drivers/net/tun.c @@ -513,14 +513,18 @@ static u16 tun_select_queue(struct net_device *dev, struct sk_buff *skb, return ret; } -static inline bool tun_not_capable(struct tun_struct *tun) +static inline bool tun_permitted(struct tun_struct *tun) { const struct cred *cred = current_cred(); struct net *net = dev_net(tun->dev); - return ((uid_valid(tun->owner) && !uid_eq(cred->euid, tun->owner)) || - (gid_valid(tun->group) && !in_egroup_p(tun->group))) && - !ns_capable(net->user_ns, CAP_NET_ADMIN); + /* If both uid and gid are set, *both* must match */ + if ((uid_valid(tun->owner) || gid_valid(tun->group)) && + (!uid_valid(tun->owner) || uid_eq(cred->euid, tun->owner)) && + (!gid_valid(tun->group) || in_egroup_p(tun->group))) + return true; + + return ns_capable(net->user_ns, CAP_NET_ADMIN); } static void tun_set_real_num_queues(struct tun_struct *tun) @@ -2877,7 +2881,7 @@ static int tun_set_iff(struct net *net, struct file *file, struct ifreq *ifr) !!(tun->flags & IFF_MULTI_QUEUE)) return -EINVAL; - if (tun_not_capable(tun)) + if (!tun_permitted(tun)) return -EPERM; err = security_tun_dev_open(tun->security); if (err < 0) diff --git a/drivers/net/usb/pegasus.c b/drivers/net/usb/pegasus.c index aba1a640fc26..7c7d6b69998f 100644 --- a/drivers/net/usb/pegasus.c +++ b/drivers/net/usb/pegasus.c @@ -461,6 +461,7 @@ static void read_bulk_callback(struct urb *urb) u8 *buf = urb->transfer_buffer; int rx_status, count = urb->actual_length; int status = urb->status; + int rx_offset = 0; __u16 pkt_len; if (!pegasus) @@ -510,7 +511,7 @@ static void read_bulk_callback(struct urb *urb) if (pegasus->chip == 0x8513) { pkt_len = le32_to_cpu(*(__le32 *)urb->transfer_buffer); pkt_len &= 0x0fff; - pegasus->rx_skb->data += 2; + rx_offset = 2; } else { pkt_len = buf[count - 3] << 8; pkt_len += buf[count - 4]; @@ -522,13 +523,14 @@ static void read_bulk_callback(struct urb *urb) * If the packet is unreasonably long, quietly drop it rather than * kernel panicing by calling skb_put. */ - if (pkt_len > PEGASUS_MTU) + if (pkt_len > PEGASUS_MTU - rx_offset) goto goon; /* * at this point we are sure pegasus->rx_skb != NULL * so we go ahead and pass up the packet. */ + skb_reserve(pegasus->rx_skb, rx_offset); skb_put(pegasus->rx_skb, pkt_len); pegasus->rx_skb->protocol = eth_type_trans(pegasus->rx_skb, net); netif_rx(pegasus->rx_skb); diff --git a/drivers/net/virtio_net.c b/drivers/net/virtio_net.c index daab43ac92ce..4e8a792cec0e 100644 --- a/drivers/net/virtio_net.c +++ b/drivers/net/virtio_net.c @@ -441,9 +441,6 @@ struct virtnet_info { /* Work struct for setting rx mode */ struct work_struct rx_mode_work; - /* OK to queue work setting RX mode? */ - bool rx_mode_work_enabled; - /* Does the affinity hint is set for virtqueues? */ bool affinity_hint_set; @@ -710,16 +707,13 @@ static void virtnet_rq_free_buf(struct virtnet_info *vi, static void enable_rx_mode_work(struct virtnet_info *vi) { - rtnl_lock(); - vi->rx_mode_work_enabled = true; - rtnl_unlock(); + if (enable_work(&vi->rx_mode_work) && netif_running(vi->dev)) + schedule_work(&vi->rx_mode_work); } static void disable_rx_mode_work(struct virtnet_info *vi) { - rtnl_lock(); - vi->rx_mode_work_enabled = false; - rtnl_unlock(); + disable_work_sync(&vi->rx_mode_work); } static void virtqueue_napi_schedule(struct napi_struct *napi, @@ -3925,8 +3919,7 @@ static void virtnet_set_rx_mode(struct net_device *dev) { struct virtnet_info *vi = netdev_priv(dev); - if (vi->rx_mode_work_enabled) - schedule_work(&vi->rx_mode_work); + schedule_work(&vi->rx_mode_work); } static int virtnet_vlan_rx_add_vid(struct net_device *dev, @@ -5742,7 +5735,6 @@ static void virtnet_freeze_down(struct virtio_device *vdev) /* Make sure no work handler is accessing the device */ flush_work(&vi->config_work); disable_rx_mode_work(vi); - flush_work(&vi->rx_mode_work); if (netif_running(vi->dev)) { rtnl_lock(); @@ -5772,8 +5764,6 @@ static int virtnet_restore_up(struct virtio_device *vdev) virtio_device_ready(vdev); - enable_rx_mode_work(vi); - if (netif_running(vi->dev)) { rtnl_lock(); err = virtnet_open(vi->dev); @@ -5785,6 +5775,8 @@ static int virtnet_restore_up(struct virtio_device *vdev) netif_tx_lock_bh(vi->dev); netif_device_attach(vi->dev); netif_tx_unlock_bh(vi->dev); + + enable_rx_mode_work(vi); return 0; err_destroy_pools: @@ -6880,6 +6872,7 @@ static int virtnet_probe(struct virtio_device *vdev) INIT_WORK(&vi->config_work, virtnet_config_changed_work); INIT_WORK(&vi->rx_mode_work, virtnet_rx_mode_work); + disable_work(&vi->rx_mode_work); if (virtio_has_feature(vdev, VIRTIO_NET_F_MRG_RXBUF)) { vi->mergeable_rx_bufs = true; @@ -7038,8 +7031,6 @@ static int virtnet_probe(struct virtio_device *vdev) if (vi->has_rss || vi->has_rss_hash_report) virtnet_init_default_rss(vi); - enable_rx_mode_work(vi); - for (i = 0; i < ARRAY_SIZE(guest_offloads); i++) { unsigned int fbit; @@ -7143,6 +7134,8 @@ static int virtnet_probe(struct virtio_device *vdev) goto free_unregister_netdev; } + enable_rx_mode_work(vi); + pr_debug("virtnet: registered device %s with %d RX and TX vq's\n", dev->name, max_queue_pairs); @@ -7195,7 +7188,6 @@ static void virtnet_remove(struct virtio_device *vdev) /* Make sure no work handler is accessing the device. */ flush_work(&vi->config_work); disable_rx_mode_work(vi); - flush_work(&vi->rx_mode_work); virtnet_free_irq_moder(vi); diff --git a/drivers/net/wireless/ath/ath12k/core.c b/drivers/net/wireless/ath/ath12k/core.c index b0107cde99ff..b25202effb00 100644 --- a/drivers/net/wireless/ath/ath12k/core.c +++ b/drivers/net/wireless/ath/ath12k/core.c @@ -54,6 +54,7 @@ ath12k_mem_profile_based_param ath12k_mem_profile_based_param[] = { .num_pool_tx_desc = 32768, .rx_desc_count = 12288, .rx_release_ring_size = 16384, + .rxdma_buf_ring_size = 8192, }, }, [ATH12K_QMI_MEMORY_MODE_LOW_512_M] = { @@ -68,6 +69,7 @@ ath12k_mem_profile_based_param ath12k_mem_profile_based_param[] = { .num_pool_tx_desc = 16384, .rx_desc_count = 6144, .rx_release_ring_size = 8192, + .rxdma_buf_ring_size = 4096, }, }, }; diff --git a/drivers/net/wireless/ath/ath12k/core.h b/drivers/net/wireless/ath/ath12k/core.h index 93fd227b4d19..99d71dab7c04 100644 --- a/drivers/net/wireless/ath/ath12k/core.h +++ b/drivers/net/wireless/ath/ath12k/core.h @@ -927,15 +927,6 @@ struct ath12k_wsi_info { u32 hw_link_id_base; }; -struct ath12k_dp_profile_params { - u32 tx_comp_ring_size; - u32 rxdma_monitor_buf_ring_size; - u32 rxdma_monitor_dst_ring_size; - u32 num_pool_tx_desc; - u32 rx_desc_count; - u32 rx_release_ring_size; -}; - struct ath12k_mem_profile_based_param { u32 num_vdevs; u32 max_client_single; diff --git a/drivers/net/wireless/ath/ath12k/dp.c b/drivers/net/wireless/ath/ath12k/dp.c index f9b37d75956d..779529d189f8 100644 --- a/drivers/net/wireless/ath/ath12k/dp.c +++ b/drivers/net/wireless/ath/ath12k/dp.c @@ -439,6 +439,7 @@ static void ath12k_dp_srng_common_cleanup(struct ath12k_base *ab) static int ath12k_dp_srng_common_setup(struct ath12k_base *ab) { + const struct ath12k_dp_profile_params *dp_params = &ab->profile_param->dp_params; struct ath12k_dp *dp = ath12k_ab_to_dp(ab); const struct ath12k_hal_tcl_to_wbm_rbm_map *map; struct hal_srng *srng; @@ -469,7 +470,7 @@ static int ath12k_dp_srng_common_setup(struct ath12k_base *ab) ret = ath12k_dp_srng_setup(ab, &dp->tx_ring[i].tcl_comp_ring, HAL_WBM2SW_RELEASE, tx_comp_ring_num, 0, - DP_TX_COMP_RING_SIZE(ab)); + ath12k_dp_tx_comp_ring_size(dp_params)); if (ret) { ath12k_warn(ab, "failed to set up tcl_comp ring (%d) :%d\n", tx_comp_ring_num, ret); @@ -487,7 +488,7 @@ static int ath12k_dp_srng_common_setup(struct ath12k_base *ab) ret = ath12k_dp_srng_setup(ab, &dp->rx_rel_ring, HAL_WBM2SW_RELEASE, HAL_WBM2SW_REL_ERR_RING_NUM, 0, - DP_RX_RELEASE_RING_SIZE(ab)); + ath12k_dp_rx_release_ring_size(dp_params)); if (ret) { ath12k_warn(ab, "failed to set up rx_rel ring :%d\n", ret); goto err; @@ -966,14 +967,15 @@ void ath12k_dp_vdev_tx_attach(struct ath12k *ar, struct ath12k_link_vif *arvif) static void ath12k_dp_cc_cleanup(struct ath12k_base *ab) { - struct ath12k_rx_desc_info *desc_info; + const struct ath12k_dp_profile_params *dp_params = &ab->profile_param->dp_params; + u32 pool_id, tx_spt_page, tx_spt_pages_per_pool, num_rx_spt_pages; struct ath12k_tx_desc_info *tx_desc_info, *tmp1; struct ath12k_dp *dp = ath12k_ab_to_dp(ab); + struct ath12k_rx_desc_info *desc_info; struct ath12k_skb_cb *skb_cb; struct sk_buff *skb; struct ath12k *ar; int i, j; - u32 pool_id, tx_spt_page; if (!dp->spt_info) return; @@ -981,8 +983,10 @@ static void ath12k_dp_cc_cleanup(struct ath12k_base *ab) /* RX Descriptor cleanup */ spin_lock_bh(&dp->rx_desc_lock); + num_rx_spt_pages = ath12k_dp_num_rx_spt_pages(dp_params); + if (dp->rxbaddr) { - for (i = 0; i < ATH12K_NUM_RX_SPT_PAGES(ab); i++) { + for (i = 0; i < num_rx_spt_pages; i++) { if (!dp->rxbaddr[i]) continue; @@ -1055,12 +1059,13 @@ static void ath12k_dp_cc_cleanup(struct ath12k_base *ab) } if (dp->txbaddr) { + tx_spt_pages_per_pool = ath12k_dp_tx_spt_pages_per_pool(dp_params); + for (pool_id = 0; pool_id < ATH12K_HW_MAX_QUEUES; pool_id++) { spin_lock_bh(&dp->tx_desc_lock[pool_id]); - for (i = 0; i < ATH12K_TX_SPT_PAGES_PER_POOL(ab); i++) { - tx_spt_page = i + pool_id * - ATH12K_TX_SPT_PAGES_PER_POOL(ab); + for (i = 0; i < tx_spt_pages_per_pool; i++) { + tx_spt_page = i + pool_id * tx_spt_pages_per_pool; if (!dp->txbaddr[tx_spt_page]) continue; @@ -1154,14 +1159,16 @@ static void *ath12k_dp_cc_get_desc_addr_ptr(struct ath12k_dp *dp, struct ath12k_rx_desc_info *ath12k_dp_get_rx_desc(struct ath12k_dp *dp, u32 cookie) { + const struct ath12k_dp_profile_params *dp_params; struct ath12k_rx_desc_info **desc_addr_ptr; u16 start_ppt_idx, end_ppt_idx, ppt_idx, spt_idx; ppt_idx = u32_get_bits(cookie, ATH12K_DP_CC_COOKIE_PPT); spt_idx = u32_get_bits(cookie, ATH12K_DP_CC_COOKIE_SPT); + dp_params = &dp->ab->profile_param->dp_params; - start_ppt_idx = dp->rx_ppt_base + ATH12K_RX_SPT_PAGE_OFFSET(dp->ab); - end_ppt_idx = start_ppt_idx + ATH12K_NUM_RX_SPT_PAGES(dp->ab); + start_ppt_idx = dp->rx_ppt_base + ath12k_dp_rx_spt_page_offset(dp_params); + end_ppt_idx = start_ppt_idx + ath12k_dp_num_rx_spt_pages(dp_params); if (ppt_idx < start_ppt_idx || ppt_idx >= end_ppt_idx || @@ -1178,15 +1185,16 @@ EXPORT_SYMBOL(ath12k_dp_get_rx_desc); struct ath12k_tx_desc_info *ath12k_dp_get_tx_desc(struct ath12k_dp *dp, u32 cookie) { + const struct ath12k_dp_profile_params *dp_params; struct ath12k_tx_desc_info **desc_addr_ptr; u16 start_ppt_idx, end_ppt_idx, ppt_idx, spt_idx; ppt_idx = u32_get_bits(cookie, ATH12K_DP_CC_COOKIE_PPT); spt_idx = u32_get_bits(cookie, ATH12K_DP_CC_COOKIE_SPT); + dp_params = &dp->ab->profile_param->dp_params; start_ppt_idx = ATH12K_TX_SPT_PAGE_OFFSET; - end_ppt_idx = start_ppt_idx + - (ATH12K_TX_SPT_PAGES_PER_POOL(dp->ab) * ATH12K_HW_MAX_QUEUES); + end_ppt_idx = start_ppt_idx + ath12k_dp_num_tx_spt_pages(dp_params); if (ppt_idx < start_ppt_idx || ppt_idx >= end_ppt_idx || @@ -1201,12 +1209,13 @@ EXPORT_SYMBOL(ath12k_dp_get_tx_desc); static int ath12k_dp_cc_desc_init(struct ath12k_base *ab) { - struct ath12k_dp *dp = ath12k_ab_to_dp(ab); + const struct ath12k_dp_profile_params *dp_params = &ab->profile_param->dp_params; + u32 num_rx_spt_pages = ath12k_dp_num_rx_spt_pages(dp_params); struct ath12k_rx_desc_info *rx_descs, **rx_desc_addr; struct ath12k_tx_desc_info *tx_descs, **tx_desc_addr; - u32 num_rx_spt_pages = ATH12K_NUM_RX_SPT_PAGES(ab); - u32 i, j, pool_id, tx_spt_page; - u32 ppt_idx, cookie_ppt_idx; + u32 i, j, pool_id, tx_spt_page, tx_spt_pages_per_pool; + u32 ppt_idx, cookie_ppt_idx, rx_spt_page_offset; + struct ath12k_dp *dp = ath12k_ab_to_dp(ab); spin_lock_bh(&dp->rx_desc_lock); @@ -1218,9 +1227,9 @@ static int ath12k_dp_cc_desc_init(struct ath12k_base *ab) return -ENOMEM; } - /* First ATH12K_NUM_RX_SPT_PAGES(ab) of allocated SPT pages are used for - * RX - */ + rx_spt_page_offset = ath12k_dp_rx_spt_page_offset(dp_params); + + /* First num_rx_spt_pages of allocated SPT pages are used for RX */ for (i = 0; i < num_rx_spt_pages; i++) { rx_descs = kzalloc_objs(*rx_descs, ATH12K_MAX_SPT_ENTRIES, GFP_ATOMIC); @@ -1230,7 +1239,7 @@ static int ath12k_dp_cc_desc_init(struct ath12k_base *ab) return -ENOMEM; } - ppt_idx = ATH12K_RX_SPT_PAGE_OFFSET(ab) + i; + ppt_idx = rx_spt_page_offset + i; cookie_ppt_idx = dp->rx_ppt_base + ppt_idx; dp->rxbaddr[i] = &rx_descs[0]; @@ -1249,14 +1258,17 @@ static int ath12k_dp_cc_desc_init(struct ath12k_base *ab) spin_unlock_bh(&dp->rx_desc_lock); dp->txbaddr = kzalloc_objs(struct ath12k_tx_desc_info *, - ATH12K_NUM_TX_SPT_PAGES(ab), GFP_ATOMIC); + ath12k_dp_num_tx_spt_pages(dp_params), + GFP_ATOMIC); if (!dp->txbaddr) return -ENOMEM; + tx_spt_pages_per_pool = ath12k_dp_tx_spt_pages_per_pool(dp_params); + for (pool_id = 0; pool_id < ATH12K_HW_MAX_QUEUES; pool_id++) { spin_lock_bh(&dp->tx_desc_lock[pool_id]); - for (i = 0; i < ATH12K_TX_SPT_PAGES_PER_POOL(ab); i++) { + for (i = 0; i < tx_spt_pages_per_pool; i++) { tx_descs = kzalloc_objs(*tx_descs, ATH12K_MAX_SPT_ENTRIES, GFP_ATOMIC); @@ -1267,8 +1279,7 @@ static int ath12k_dp_cc_desc_init(struct ath12k_base *ab) return -ENOMEM; } - tx_spt_page = i + pool_id * - ATH12K_TX_SPT_PAGES_PER_POOL(ab); + tx_spt_page = i + pool_id * tx_spt_pages_per_pool; ppt_idx = ATH12K_TX_SPT_PAGE_OFFSET + tx_spt_page; dp->txbaddr[tx_spt_page] = &tx_descs[0]; @@ -1294,6 +1305,7 @@ static int ath12k_dp_cmem_init(struct ath12k_base *ab, struct ath12k_dp *dp, enum ath12k_dp_desc_type type) { + const struct ath12k_dp_profile_params *dp_params = &ab->profile_param->dp_params; u32 cmem_base; int i, start, end; @@ -1302,12 +1314,12 @@ static int ath12k_dp_cmem_init(struct ath12k_base *ab, switch (type) { case ATH12K_DP_TX_DESC: start = ATH12K_TX_SPT_PAGE_OFFSET; - end = start + ATH12K_NUM_TX_SPT_PAGES(ab); + end = start + ath12k_dp_num_tx_spt_pages(dp_params); break; case ATH12K_DP_RX_DESC: cmem_base += ATH12K_PPT_ADDR_OFFSET(dp->rx_ppt_base); - start = ATH12K_RX_SPT_PAGE_OFFSET(ab); - end = start + ATH12K_NUM_RX_SPT_PAGES(ab); + start = ath12k_dp_rx_spt_page_offset(dp_params); + end = start + ath12k_dp_num_rx_spt_pages(dp_params); break; default: ath12k_err(ab, "invalid descriptor type %d in cmem init\n", type); @@ -1337,7 +1349,10 @@ void ath12k_dp_partner_cc_init(struct ath12k_base *ab) static u32 ath12k_dp_get_num_spt_pages(struct ath12k_base *ab) { - return ATH12K_NUM_RX_SPT_PAGES(ab) + ATH12K_NUM_TX_SPT_PAGES(ab); + const struct ath12k_dp_profile_params *dp_params = &ab->profile_param->dp_params; + + return ath12k_dp_num_rx_spt_pages(dp_params) + + ath12k_dp_num_tx_spt_pages(dp_params); } static int ath12k_dp_cc_init(struct ath12k_base *ab) @@ -1365,7 +1380,8 @@ static int ath12k_dp_cc_init(struct ath12k_base *ab) return -ENOMEM; } - dp->rx_ppt_base = ab->device_id * ATH12K_NUM_RX_SPT_PAGES(ab); + dp->rx_ppt_base = ab->device_id * + ath12k_dp_num_rx_spt_pages(&ab->profile_param->dp_params); for (i = 0; i < dp->num_spt_pages; i++) { dp->spt_info[i].vaddr = dma_alloc_coherent(ab->dev, @@ -1465,6 +1481,7 @@ static int ath12k_dp_reoq_lut_setup(struct ath12k_base *ab) static int ath12k_dp_setup(struct ath12k_base *ab) { + const struct ath12k_dp_profile_params *dp_params; struct ath12k_dp *dp; struct hal_srng *srng = NULL; size_t size = 0; @@ -1474,6 +1491,7 @@ static int ath12k_dp_setup(struct ath12k_base *ab) dp = ath12k_ab_to_dp(ab); dp->ab = ab; + dp_params = &ab->profile_param->dp_params; INIT_LIST_HEAD(&dp->reo_cmd_list); INIT_LIST_HEAD(&dp->reo_cmd_cache_flush_list); @@ -1528,7 +1546,7 @@ static int ath12k_dp_setup(struct ath12k_base *ab) goto fail_dp_bank_profiles_cleanup; size = ab->hal.hal_wbm_release_ring_tx_size * - DP_TX_COMP_RING_SIZE(ab); + ath12k_dp_tx_comp_ring_size(dp_params); ret = ath12k_dp_reoq_lut_setup(ab); if (ret) { @@ -1540,7 +1558,8 @@ static int ath12k_dp_setup(struct ath12k_base *ab) dp->tx_ring[i].tcl_data_ring_id = i; dp->tx_ring[i].tx_status_head = 0; - dp->tx_ring[i].tx_status_tail = DP_TX_COMP_RING_SIZE(ab) - 1; + dp->tx_ring[i].tx_status_tail = + ath12k_dp_tx_comp_ring_size(dp_params) - 1; dp->tx_ring[i].tx_status = kmalloc(size, GFP_KERNEL); if (!dp->tx_ring[i].tx_status) { ret = -ENOMEM; diff --git a/drivers/net/wireless/ath/ath12k/dp.h b/drivers/net/wireless/ath/ath12k/dp.h index bef0f2ba0560..842d8cdbaeb5 100644 --- a/drivers/net/wireless/ath/ath12k/dp.h +++ b/drivers/net/wireless/ath/ath12k/dp.h @@ -23,6 +23,16 @@ struct ath12k_ext_irq_grp; struct ath12k_dp_rx_tid; struct ath12k_dp_rx_tid_rxq; +struct ath12k_dp_profile_params { + u32 tx_comp_ring_size; + u32 rxdma_monitor_buf_ring_size; + u32 rxdma_monitor_dst_ring_size; + u32 num_pool_tx_desc; + u32 rx_desc_count; + u32 rx_release_ring_size; + u32 rxdma_buf_ring_size; +}; + #define DP_MON_PURGE_TIMEOUT_MS 100 #define DP_MON_SERVICE_BUDGET 128 @@ -52,7 +62,7 @@ struct dp_rxdma_ring { int bufs_max; }; -#define ATH12K_TX_COMPL_NEXT(ab, x) (((x) + 1) % DP_TX_COMP_RING_SIZE(ab)) +#define ATH12K_TX_COMPL_NEXT(ring_size, x) (((x) + 1) % (ring_size)) struct dp_tx_ring { u8 tcl_data_ring_id; @@ -192,28 +202,18 @@ struct ath12k_pdev_dp { #define DP_WBM_RELEASE_RING_SIZE 64 #define DP_TCL_DATA_RING_SIZE 512 -#define DP_TX_COMP_RING_SIZE(ab) \ - ((ab)->profile_param->dp_params.tx_comp_ring_size) -#define DP_TX_IDR_SIZE(ab) DP_TX_COMP_RING_SIZE(ab) #define DP_TCL_CMD_RING_SIZE 32 #define DP_TCL_STATUS_RING_SIZE 32 #define DP_REO_DST_RING_MAX 8 #define DP_REO_DST_RING_SIZE 2048 #define DP_REO_REINJECT_RING_SIZE 32 -#define DP_RX_RELEASE_RING_SIZE(ab) \ - ((ab)->profile_param->dp_params.rx_release_ring_size) #define DP_REO_EXCEPTION_RING_SIZE 128 #define DP_REO_CMD_RING_SIZE 256 #define DP_REO_STATUS_RING_SIZE 2048 -#define DP_RXDMA_BUF_RING_SIZE 4096 #define DP_RX_MAC_BUF_RING_SIZE 4096 #define DP_RXDMA_REFILL_RING_SIZE 2048 #define DP_RXDMA_ERR_DST_RING_SIZE 1024 #define DP_RXDMA_MON_STATUS_RING_SIZE 1024 -#define DP_RXDMA_MONITOR_BUF_RING_SIZE(ab) \ - ((ab)->profile_param->dp_params.rxdma_monitor_buf_ring_size) -#define DP_RXDMA_MONITOR_DST_RING_SIZE(ab) \ - ((ab)->profile_param->dp_params.rxdma_monitor_dst_ring_size) #define DP_RXDMA_MONITOR_DESC_RING_SIZE 4096 #define DP_TX_MONITOR_BUF_RING_SIZE 4096 #define DP_TX_MONITOR_DEST_RING_SIZE 2048 @@ -247,12 +247,6 @@ struct ath12k_pdev_dp { #define ATH12K_SHADOW_DP_TIMER_INTERVAL 20 #define ATH12K_SHADOW_CTRL_TIMER_INTERVAL 10 -#define ATH12K_NUM_POOL_TX_DESC(ab) \ - ((ab)->profile_param->dp_params.num_pool_tx_desc) -/* TODO: revisit this count during testing */ -#define ATH12K_RX_DESC_COUNT(ab) \ - ((ab)->profile_param->dp_params.rx_desc_count) - #define ATH12K_PAGE_SIZE PAGE_SIZE /* Total 1024 entries in PPT, i.e 4K/4 considering 4K aligned @@ -263,21 +257,7 @@ struct ath12k_pdev_dp { /* Total 512 entries in a SPT, i.e 4K Page/8 */ #define ATH12K_MAX_SPT_ENTRIES 512 -#define ATH12K_NUM_RX_SPT_PAGES(ab) ((ATH12K_RX_DESC_COUNT(ab)) / \ - ATH12K_MAX_SPT_ENTRIES) - -#define ATH12K_TX_SPT_PAGES_PER_POOL(ab) (ATH12K_NUM_POOL_TX_DESC(ab) / \ - ATH12K_MAX_SPT_ENTRIES) -#define ATH12K_NUM_TX_SPT_PAGES(ab) (ATH12K_TX_SPT_PAGES_PER_POOL(ab) * \ - ATH12K_HW_MAX_QUEUES) - #define ATH12K_TX_SPT_PAGE_OFFSET 0 -#define ATH12K_RX_SPT_PAGE_OFFSET(ab) ATH12K_NUM_TX_SPT_PAGES(ab) - -/* The SPT pages are divided for RX and TX, first block for RX - * and remaining for TX - */ -#define ATH12K_NUM_TX_SPT_PAGE_START(ab) ATH12K_NUM_RX_SPT_PAGES(ab) #define ATH12K_DP_RX_DESC_MAGIC 0xBABABABA @@ -680,6 +660,72 @@ ath12k_dp_to_pdev_dp(struct ath12k_dp *dp, u8 pdev_idx) return rcu_dereference(dp->dp_pdevs[pdev_idx]); } +static inline u32 +ath12k_dp_tx_comp_ring_size(const struct ath12k_dp_profile_params *p) +{ + return p->tx_comp_ring_size; +} + +static inline u32 +ath12k_dp_rxdma_monitor_buf_ring_size(const struct ath12k_dp_profile_params *p) +{ + return p->rxdma_monitor_buf_ring_size; +} + +static inline u32 +ath12k_dp_rxdma_monitor_dst_ring_size(const struct ath12k_dp_profile_params *p) +{ + return p->rxdma_monitor_dst_ring_size; +} + +static inline u32 +ath12k_dp_num_pool_tx_desc(const struct ath12k_dp_profile_params *p) +{ + return p->num_pool_tx_desc; +} + +static inline u32 +ath12k_dp_tx_spt_pages_per_pool(const struct ath12k_dp_profile_params *p) +{ + return ath12k_dp_num_pool_tx_desc(p) / ATH12K_MAX_SPT_ENTRIES; +} + +static inline u32 +ath12k_dp_num_tx_spt_pages(const struct ath12k_dp_profile_params *p) +{ + return ath12k_dp_tx_spt_pages_per_pool(p) * ATH12K_HW_MAX_QUEUES; +} + +static inline u32 +ath12k_dp_rx_spt_page_offset(const struct ath12k_dp_profile_params *p) +{ + return ath12k_dp_num_tx_spt_pages(p); +} + +static inline u32 +ath12k_dp_rx_desc_count(const struct ath12k_dp_profile_params *p) +{ + return p->rx_desc_count; +} + +static inline u32 +ath12k_dp_num_rx_spt_pages(const struct ath12k_dp_profile_params *p) +{ + return ath12k_dp_rx_desc_count(p) / ATH12K_MAX_SPT_ENTRIES; +} + +static inline u32 +ath12k_dp_rx_release_ring_size(const struct ath12k_dp_profile_params *p) +{ + return p->rx_release_ring_size; +} + +static inline u32 +ath12k_dp_rxdma_buf_ring_size(const struct ath12k_dp_profile_params *p) +{ + return p->rxdma_buf_ring_size; +} + void ath12k_dp_vdev_tx_attach(struct ath12k *ar, struct ath12k_link_vif *arvif); void ath12k_dp_partner_cc_init(struct ath12k_base *ab); int ath12k_dp_pdev_alloc(struct ath12k_base *ab); diff --git a/drivers/net/wireless/ath/ath12k/dp_rx.c b/drivers/net/wireless/ath/ath12k/dp_rx.c index 8fa0e90b4531..3628344f3593 100644 --- a/drivers/net/wireless/ath/ath12k/dp_rx.c +++ b/drivers/net/wireless/ath/ath12k/dp_rx.c @@ -330,16 +330,20 @@ static int ath12k_dp_rx_pdev_srng_alloc(struct ath12k *ar) { struct ath12k_pdev_dp *dp = &ar->dp; struct ath12k_base *ab = ar->ab; + u32 monitor_dst_ring_size; + u32 mac_id = dp->mac_id; int i; int ret; - u32 mac_id = dp->mac_id; + + monitor_dst_ring_size = + ath12k_dp_rxdma_monitor_dst_ring_size(&ab->profile_param->dp_params); for (i = 0; i < ab->hw_params->num_rxdma_per_pdev; i++) { ret = ath12k_dp_srng_setup(ar->ab, &dp->rxdma_mon_dst_ring[i], HAL_RXDMA_MONITOR_DST, 0, mac_id + i, - DP_RXDMA_MONITOR_DST_RING_SIZE(ab)); + monitor_dst_ring_size); if (ret) { ath12k_warn(ar->ab, "failed to setup HAL_RXDMA_MONITOR_DST\n"); @@ -1692,6 +1696,8 @@ int ath12k_dp_rx_htt_setup(struct ath12k_base *ab) int ath12k_dp_rx_alloc(struct ath12k_base *ab) { + const struct ath12k_dp_profile_params *dp_params = &ab->profile_param->dp_params; + u32 monitor_buf_ring_size = ath12k_dp_rxdma_monitor_buf_ring_size(dp_params); struct ath12k_dp *dp = ath12k_ab_to_dp(ab); struct dp_srng *srng; int i, ret; @@ -1702,7 +1708,7 @@ int ath12k_dp_rx_alloc(struct ath12k_base *ab) ret = ath12k_dp_srng_setup(ab, &dp->rx_refill_buf_ring.refill_buf_ring, HAL_RXDMA_BUF, 0, 0, - DP_RXDMA_BUF_RING_SIZE); + ath12k_dp_rxdma_buf_ring_size(dp_params)); if (ret) { ath12k_warn(ab, "failed to setup rx_refill_buf_ring\n"); return ret; @@ -1736,7 +1742,7 @@ int ath12k_dp_rx_alloc(struct ath12k_base *ab) ret = ath12k_dp_srng_setup(ab, &dp->rxdma_mon_buf_ring.refill_buf_ring, HAL_RXDMA_MONITOR_BUF, 0, 0, - DP_RXDMA_MONITOR_BUF_RING_SIZE(ab)); + monitor_buf_ring_size); if (ret) { ath12k_warn(ab, "failed to setup HAL_RXDMA_MONITOR_BUF\n"); return ret; diff --git a/drivers/net/wireless/ath/ath12k/mac.c b/drivers/net/wireless/ath/ath12k/mac.c index 99bf5cf79d10..a997e888feb5 100644 --- a/drivers/net/wireless/ath/ath12k/mac.c +++ b/drivers/net/wireless/ath/ath12k/mac.c @@ -9732,6 +9732,7 @@ static int ath12k_mac_config_mon_status_default(struct ath12k *ar, bool enable) static int ath12k_mac_start(struct ath12k *ar) { + const struct ath12k_dp_profile_params *dp_params; struct ath12k_hw *ah = ar->ah; struct ath12k_base *ab = ar->ab; struct ath12k_pdev *pdev = ar->pdev; @@ -9740,6 +9741,8 @@ static int ath12k_mac_start(struct ath12k *ar) lockdep_assert_held(&ah->hw_mutex); lockdep_assert_wiphy(ath12k_ar_to_hw(ar)->wiphy); + dp_params = &ab->profile_param->dp_params; + ret = ath12k_wmi_pdev_set_param(ar, WMI_PDEV_PARAM_PMF_QOS, 1, pdev->pdev_id); @@ -9788,7 +9791,7 @@ static int ath12k_mac_start(struct ath12k *ar) if (ab->hw_params->supports_cong_ctrl_max_msdus) { ret = ath12k_wmi_pdev_set_param(ar, WMI_PDEV_PARAM_SET_CONG_CTRL_MAX_MSDUS, - ATH12K_NUM_POOL_TX_DESC(ab), + ath12k_dp_num_pool_tx_desc(dp_params), pdev->pdev_id); if (ret) { ath12k_err(ab, diff --git a/drivers/net/wireless/ath/ath12k/wifi7/dp_tx.c b/drivers/net/wireless/ath/ath12k/wifi7/dp_tx.c index 587d58eeccfa..e14f1d190ba8 100644 --- a/drivers/net/wireless/ath/ath12k/wifi7/dp_tx.c +++ b/drivers/net/wireless/ath/ath12k/wifi7/dp_tx.c @@ -922,12 +922,14 @@ void ath12k_wifi7_dp_tx_completion_handler(struct ath12k_dp *dp, int ring_id) u64 desc_va; enum hal_wbm_rel_src_module buf_rel_source; enum hal_wbm_tqm_rel_reason rel_status; + u32 ring_size; spin_lock_bh(&status_ring->lock); + ring_size = ath12k_dp_tx_comp_ring_size(&ab->profile_param->dp_params); ath12k_hal_srng_access_begin(ab, status_ring); - while (ATH12K_TX_COMPL_NEXT(ab, tx_ring->tx_status_head) != + while (ATH12K_TX_COMPL_NEXT(ring_size, tx_ring->tx_status_head) != tx_ring->tx_status_tail) { desc = ath12k_hal_srng_dst_get_next_entry(ab, status_ring); if (!desc) @@ -936,11 +938,11 @@ void ath12k_wifi7_dp_tx_completion_handler(struct ath12k_dp *dp, int ring_id) memcpy(&tx_ring->tx_status[tx_ring->tx_status_head], desc, sizeof(*desc)); tx_ring->tx_status_head = - ATH12K_TX_COMPL_NEXT(ab, tx_ring->tx_status_head); + ATH12K_TX_COMPL_NEXT(ring_size, tx_ring->tx_status_head); } if (ath12k_hal_srng_dst_peek(ab, status_ring) && - (ATH12K_TX_COMPL_NEXT(ab, tx_ring->tx_status_head) == + (ATH12K_TX_COMPL_NEXT(ring_size, tx_ring->tx_status_head) == tx_ring->tx_status_tail)) { /* TODO: Process pending tx_status messages when kfifo_is_full() */ ath12k_warn(ab, "Unable to process some of the tx_status ring desc because status_fifo is full\n"); @@ -950,13 +952,13 @@ void ath12k_wifi7_dp_tx_completion_handler(struct ath12k_dp *dp, int ring_id) spin_unlock_bh(&status_ring->lock); - while (ATH12K_TX_COMPL_NEXT(ab, tx_ring->tx_status_tail) != + while (ATH12K_TX_COMPL_NEXT(ring_size, tx_ring->tx_status_tail) != tx_ring->tx_status_head) { struct hal_wbm_completion_ring_tx *tx_status; u32 desc_id; tx_ring->tx_status_tail = - ATH12K_TX_COMPL_NEXT(ab, tx_ring->tx_status_tail); + ATH12K_TX_COMPL_NEXT(ring_size, tx_ring->tx_status_tail); tx_status = &tx_ring->tx_status[tx_ring->tx_status_tail]; ath12k_wifi7_dp_tx_status_parse(dp, tx_status, &ts); diff --git a/drivers/net/wireless/ath/ath9k/channel.c b/drivers/net/wireless/ath/ath9k/channel.c index 8b27d8cc086a..cfb1c90eb4b6 100644 --- a/drivers/net/wireless/ath/ath9k/channel.c +++ b/drivers/net/wireless/ath/ath9k/channel.c @@ -1363,6 +1363,8 @@ void ath9k_init_channel_context(struct ath_softc *sc) void ath9k_deinit_channel_context(struct ath_softc *sc) { + timer_shutdown_sync(&sc->sched.timer); + timer_shutdown_sync(&sc->offchannel.timer); cancel_work_sync(&sc->chanctx_work); } diff --git a/drivers/net/wireless/broadcom/b43/main.c b/drivers/net/wireless/broadcom/b43/main.c index 817448e58e7f..3825b6ad9f4f 100644 --- a/drivers/net/wireless/broadcom/b43/main.c +++ b/drivers/net/wireless/broadcom/b43/main.c @@ -5677,6 +5677,7 @@ static void b43_bcma_remove(struct bcma_device *core) { struct b43_wldev *wldev = bcma_get_drvdata(core); struct b43_wl *wl = wldev->wl; + struct b43_bus_dev *dev = wldev->dev; /* We must cancel any work here before unregistering from ieee80211, * as the ieee80211 unreg will destroy the workqueue. */ @@ -5691,14 +5692,14 @@ static void b43_bcma_remove(struct bcma_device *core) ieee80211_unregister_hw(wl->hw); } - b43_one_core_detach(wldev->dev); + b43_one_core_detach(dev); /* Unregister HW RNG driver */ b43_rng_exit(wl); b43_leds_unregister(wl); ieee80211_free_hw(wl->hw); - kfree(wldev->dev); + kfree(dev); } static struct bcma_driver b43_bcma_driver = { diff --git a/drivers/net/wireless/mediatek/mt76/mt7925/pci.c b/drivers/net/wireless/mediatek/mt76/mt7925/pci.c index 02ef09dd797d..d2c0a78ae9f2 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7925/pci.c +++ b/drivers/net/wireless/mediatek/mt76/mt7925/pci.c @@ -493,7 +493,6 @@ static const struct mt792x_irq_map mt7925_irq_map = { .all_complete_mask = MT7925_INT_RX_DONE_ALL, .data_complete_mask = MT7925_INT_RX_DONE_DATA, .wm_complete_mask = MT7925_INT_RX_DONE_WM, - .wm2_complete_mask = MT_INT_RX_DONE_WM2, }, }; diff --git a/drivers/net/wireless/mediatek/mt76/mt7925/regs.h b/drivers/net/wireless/mediatek/mt76/mt7925/regs.h index ab6c33b4a180..e040dff321dd 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7925/regs.h +++ b/drivers/net/wireless/mediatek/mt76/mt7925/regs.h @@ -48,8 +48,7 @@ #define MT7925_INT_RX_DONE_DATA HOST_RX_DONE_INT_ENA2 #define MT7925_INT_RX_DONE_WM HOST_RX_DONE_INT_ENA0 #define MT7925_INT_RX_DONE_ALL (MT7925_INT_RX_DONE_DATA | \ - MT7925_INT_RX_DONE_WM | \ - MT_INT_RX_DONE_WM2) + MT7925_INT_RX_DONE_WM) #define MT7927_INT_RX_DONE_DATA MT7927_RX_DONE_INT_ENA4 #define MT7927_INT_RX_DONE_WM MT7927_RX_DONE_INT_ENA6 diff --git a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c index 1cba381a78b8..a9558381ad91 100644 --- a/drivers/net/wireless/mediatek/mt76/mt7996/mac.c +++ b/drivers/net/wireless/mediatek/mt76/mt7996/mac.c @@ -3285,8 +3285,10 @@ void mt7996_mac_add_twt_setup(struct ieee80211_hw *hw, flow->tsf = le64_to_cpu(twt_agrt->twt); if (mt7996_mcu_twt_agrt_update(dev, &msta->vif->deflink, flow, - MCU_TWT_AGRT_ADD)) + MCU_TWT_AGRT_ADD)) { + list_del(&flow->list); goto unlock; + } setup_cmd = TWT_SETUP_CMD_ACCEPT; dev->twt.table_mask |= BIT(table_id); diff --git a/drivers/net/wwan/wwan_hwsim.c b/drivers/net/wwan/wwan_hwsim.c index c3c2d2a68767..34eea3edc824 100644 --- a/drivers/net/wwan/wwan_hwsim.c +++ b/drivers/net/wwan/wwan_hwsim.c @@ -500,6 +500,7 @@ static ssize_t wwan_hwsim_debugfs_portdestroy_write(struct file *file, } static const struct file_operations wwan_hwsim_debugfs_portdestroy_fops = { + .owner = THIS_MODULE, .write = wwan_hwsim_debugfs_portdestroy_write, .open = simple_open, .llseek = noop_llseek, @@ -524,6 +525,7 @@ static ssize_t wwan_hwsim_debugfs_portcreate_write(struct file *file, } static const struct file_operations wwan_hwsim_debugfs_portcreate_fops = { + .owner = THIS_MODULE, .write = wwan_hwsim_debugfs_portcreate_write, .open = simple_open, .llseek = noop_llseek, @@ -545,6 +547,7 @@ static ssize_t wwan_hwsim_debugfs_devdestroy_write(struct file *file, } static const struct file_operations wwan_hwsim_debugfs_devdestroy_fops = { + .owner = THIS_MODULE, .write = wwan_hwsim_debugfs_devdestroy_write, .open = simple_open, .llseek = noop_llseek, @@ -568,6 +571,7 @@ static ssize_t wwan_hwsim_debugfs_devcreate_write(struct file *file, } static const struct file_operations wwan_hwsim_debugfs_devcreate_fops = { + .owner = THIS_MODULE, .write = wwan_hwsim_debugfs_devcreate_write, .open = simple_open, .llseek = noop_llseek, diff --git a/drivers/nvme/target/auth.c b/drivers/nvme/target/auth.c index a55319bcdbd1..1680e5e6839c 100644 --- a/drivers/nvme/target/auth.c +++ b/drivers/nvme/target/auth.c @@ -229,6 +229,7 @@ u8 nvmet_setup_auth(struct nvmet_ctrl *ctrl, struct nvmet_sq *sq, bool reset) void nvmet_auth_sq_free(struct nvmet_sq *sq) { + sq->auth_expiry_active = false; cancel_delayed_work(&sq->auth_expired_work); kfree(sq->dhchap_c1); sq->dhchap_c1 = NULL; diff --git a/drivers/nvme/target/fabrics-cmd-auth.c b/drivers/nvme/target/fabrics-cmd-auth.c index dccdb55a367c..f30c2539a6fd 100644 --- a/drivers/nvme/target/fabrics-cmd-auth.c +++ b/drivers/nvme/target/fabrics-cmd-auth.c @@ -6,6 +6,7 @@ */ #define pr_fmt(fmt) KBUILD_MODNAME ": " fmt #include +#include #include #include #include @@ -17,16 +18,27 @@ static void nvmet_auth_expired_work(struct work_struct *work) struct nvmet_sq *sq = container_of(to_delayed_work(work), struct nvmet_sq, auth_expired_work); + mutex_lock(&sq->auth_lock); + /* A command may have cancelled or rearmed this timeout. */ + if (!sq->auth_expiry_active || + delayed_work_pending(&sq->auth_expired_work)) + goto out_unlock; + sq->auth_expiry_active = false; + pr_debug("%s: ctrl %d qid %d transaction %u expired, resetting\n", __func__, sq->ctrl->cntlid, sq->qid, sq->dhchap_tid); sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE; sq->dhchap_tid = -1; +out_unlock: + mutex_unlock(&sq->auth_lock); } void nvmet_auth_sq_init(struct nvmet_sq *sq) { /* Initialize in-band authentication */ + mutex_init(&sq->auth_lock); INIT_DELAYED_WORK(&sq->auth_expired_work, nvmet_auth_expired_work); + sq->auth_expiry_active = false; sq->authenticated = false; sq->dhchap_step = NVME_AUTH_DHCHAP_MESSAGE_NEGOTIATE; } @@ -256,6 +268,8 @@ void nvmet_execute_auth_send(struct nvmet_req *req) u16 status = 0; u8 dhchap_status; + mutex_lock(&req->sq->auth_lock); + if (req->cmd->auth_send.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) { status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR; req->error_loc = @@ -281,8 +295,10 @@ void nvmet_execute_auth_send(struct nvmet_req *req) offsetof(struct nvmf_auth_send_command, tl); goto done; } - if (!nvmet_check_transfer_len(req, tl)) { + if (unlikely(tl != req->transfer_len)) { pr_debug("%s: transfer length mismatch (%u)\n", __func__, tl); + mutex_unlock(&req->sq->auth_lock); + nvmet_check_transfer_len(req, tl); return; } @@ -406,10 +422,12 @@ void nvmet_execute_auth_send(struct nvmet_req *req) status, req->error_loc); if (req->sq->dhchap_step != NVME_AUTH_DHCHAP_MESSAGE_SUCCESS2 && req->sq->dhchap_step != NVME_AUTH_DHCHAP_MESSAGE_FAILURE2) { - unsigned long auth_expire_secs = ctrl->kato ? ctrl->kato : 120; + unsigned long auth_expire = min_t(u64, MAX_JIFFY_OFFSET, + (u64)(ctrl->kato ? ctrl->kato : 120) * HZ); + req->sq->auth_expiry_active = true; mod_delayed_work(system_percpu_wq, &req->sq->auth_expired_work, - auth_expire_secs * HZ); + auth_expire); goto complete; } /* Final states, clear up variables */ @@ -418,6 +436,7 @@ void nvmet_execute_auth_send(struct nvmet_req *req) nvmet_ctrl_fatal_error(ctrl); complete: + mutex_unlock(&req->sq->auth_lock); nvmet_req_complete(req, status); } @@ -536,6 +555,8 @@ void nvmet_execute_auth_receive(struct nvmet_req *req) u32 al; u16 status = 0; + mutex_lock(&req->sq->auth_lock); + if (req->cmd->auth_receive.secp != NVME_AUTH_DHCHAP_PROTOCOL_IDENTIFIER) { status = NVME_SC_INVALID_FIELD | NVME_STATUS_DNR; req->error_loc = @@ -561,8 +582,10 @@ void nvmet_execute_auth_receive(struct nvmet_req *req) offsetof(struct nvmf_auth_receive_command, al); goto done; } - if (!nvmet_check_transfer_len(req, al)) { + if (unlikely(al != req->transfer_len)) { pr_debug("%s: transfer length mismatch (%u)\n", __func__, al); + mutex_unlock(&req->sq->auth_lock); + nvmet_check_transfer_len(req, al); return; } @@ -621,5 +644,6 @@ void nvmet_execute_auth_receive(struct nvmet_req *req) nvmet_auth_sq_free(req->sq); nvmet_ctrl_fatal_error(ctrl); } + mutex_unlock(&req->sq->auth_lock); nvmet_req_complete(req, status); } diff --git a/drivers/nvme/target/fc.c b/drivers/nvme/target/fc.c index 1b557775e033..f3e80095d7d1 100644 --- a/drivers/nvme/target/fc.c +++ b/drivers/nvme/target/fc.c @@ -1796,9 +1796,6 @@ nvmet_fc_ls_create_connection(struct nvmet_fc_tgtport *tgtport, be16_to_cpu(rqst->connect_cmd.sqsize)); if (!queue) ret = VERR_QUEUE_ALLOC_FAIL; - - /* release get taken in nvmet_fc_find_target_assoc */ - nvmet_fc_tgt_a_put(iod->assoc); } } @@ -1812,6 +1809,8 @@ nvmet_fc_ls_create_connection(struct nvmet_fc_tgtport *tgtport, FCNVME_RJT_RC_INV_ASSOC : FCNVME_RJT_RC_LOGIC, FCNVME_RJT_EXP_NONE, 0); + if (iod->assoc) + nvmet_fc_tgt_a_put(iod->assoc); return; } @@ -1833,6 +1832,9 @@ nvmet_fc_ls_create_connection(struct nvmet_fc_tgtport *tgtport, acc->connectid.connection_id = cpu_to_be64(nvmet_fc_makeconnid(iod->assoc, be16_to_cpu(rqst->connect_cmd.qid))); + + /* release get taken in nvmet_fc_find_target_assoc */ + nvmet_fc_tgt_a_put(iod->assoc); } /* diff --git a/drivers/nvme/target/nvmet.h b/drivers/nvme/target/nvmet.h index 162e2fdd848e..2032db380f99 100644 --- a/drivers/nvme/target/nvmet.h +++ b/drivers/nvme/target/nvmet.h @@ -161,8 +161,10 @@ struct nvmet_sq { u32 sqhd; bool sqhd_disabled; #ifdef CONFIG_NVME_TARGET_AUTH + struct mutex auth_lock; bool authenticated; struct delayed_work auth_expired_work; + bool auth_expiry_active; u16 dhchap_tid; u8 sc_c; u8 dhchap_status; diff --git a/drivers/nvme/target/pr.c b/drivers/nvme/target/pr.c index 81f48cb00aee..ad13a93afb78 100644 --- a/drivers/nvme/target/pr.c +++ b/drivers/nvme/target/pr.c @@ -593,7 +593,7 @@ static u16 nvmet_pr_preempt(struct nvmet_req *req, status = nvmet_pr_update_reg_attr(pr, holder, nvmet_pr_update_holder_rtype, &rtype); if (!status && original_rtype != rtype) - nvmet_pr_resv_released(pr, ®->hostid); + nvmet_pr_resv_released(pr, &ctrl->hostid); return status; } diff --git a/drivers/nvmem/core.c b/drivers/nvmem/core.c index 0556d140170a..b4e5069d9e38 100644 --- a/drivers/nvmem/core.c +++ b/drivers/nvmem/core.c @@ -1598,12 +1598,14 @@ static void nvmem_shift_read_buffer_in_place(struct nvmem_cell_entry *cell, void *p = *b++ >> bit_offset; /* setup rest of the bytes if any */ - for (i = 1; i < cell->bytes; i++) { + for (i = 1; i < (cell->bytes - bytes_offset); i++) { /* Get bits from next byte and shift them towards msb */ *p++ |= *b << (BITS_PER_BYTE - bit_offset); *p = *b++ >> bit_offset; } + /* point to end of the buffer unused bits will be cleared */ + p = buf + cell->bytes - 1; } else if (p != b) { memmove(p, b, cell->bytes - bytes_offset); p += cell->bytes - 1; diff --git a/drivers/phy/phy-core.c b/drivers/phy/phy-core.c index 21aaf2f76e53..89addd732bff 100644 --- a/drivers/phy/phy-core.c +++ b/drivers/phy/phy-core.c @@ -124,13 +124,13 @@ static struct phy *phy_find(struct device *dev, const char *con_id) const char *dev_id = dev_name(dev); struct phy_lookup *p, *pl = NULL; - mutex_lock(&phy_provider_mutex); + lockdep_assert_held(&phy_provider_mutex); + list_for_each_entry(p, &phys, node) if (!strcmp(p->dev_id, dev_id) && !strcmp(p->con_id, con_id)) { pl = p; break; } - mutex_unlock(&phy_provider_mutex); return pl ? pl->phy : ERR_PTR(-ENODEV); } @@ -624,6 +624,8 @@ static struct phy *_of_phy_get(struct device_node *np, int index) struct phy *phy = NULL; struct of_phandle_args args; + lockdep_assert_held(&phy_provider_mutex); + ret = of_parse_phandle_with_args(np, "phys", "#phy-cells", index, &args); if (ret) @@ -635,11 +637,10 @@ static struct phy *_of_phy_get(struct device_node *np, int index) goto out_put_node; } - mutex_lock(&phy_provider_mutex); phy_provider = of_phy_provider_lookup(args.np); if (IS_ERR(phy_provider) || !try_module_get(phy_provider->owner)) { phy = ERR_PTR(-EPROBE_DEFER); - goto out_unlock; + goto out_put_node; } if (!of_device_is_available(args.np)) { @@ -653,8 +654,6 @@ static struct phy *_of_phy_get(struct device_node *np, int index) out_put_module: module_put(phy_provider->owner); -out_unlock: - mutex_unlock(&phy_provider_mutex); out_put_node: of_node_put(args.np); @@ -678,15 +677,21 @@ struct phy *of_phy_get(struct device_node *np, const char *con_id) if (con_id) index = of_property_match_string(np, "phy-names", con_id); + mutex_lock(&phy_provider_mutex); + phy = _of_phy_get(np, index); if (IS_ERR(phy)) - return phy; + goto out_unlock; - if (!try_module_get(phy->ops->owner)) - return ERR_PTR(-EPROBE_DEFER); + if (!try_module_get(phy->ops->owner)) { + phy = ERR_PTR(-EPROBE_DEFER); + goto out_unlock; + } get_device(&phy->dev); +out_unlock: + mutex_unlock(&phy_provider_mutex); return phy; } EXPORT_SYMBOL_GPL(of_phy_get); @@ -786,6 +791,7 @@ struct phy *phy_get(struct device *dev, const char *string) struct phy *phy; struct device_link *link; + mutex_lock(&phy_provider_mutex); if (dev->of_node) { if (string) index = of_property_match_string(dev->of_node, "phy-names", @@ -796,15 +802,18 @@ struct phy *phy_get(struct device *dev, const char *string) } else { if (string == NULL) { dev_WARN(dev, "missing string\n"); - return ERR_PTR(-EINVAL); + phy = ERR_PTR(-EINVAL); + goto out_unlock; } phy = phy_find(dev, string); } if (IS_ERR(phy)) - return phy; + goto out_unlock; - if (!try_module_get(phy->ops->owner)) - return ERR_PTR(-EPROBE_DEFER); + if (!try_module_get(phy->ops->owner)) { + phy = ERR_PTR(-EPROBE_DEFER); + goto out_unlock; + } get_device(&phy->dev); @@ -813,6 +822,8 @@ struct phy *phy_get(struct device *dev, const char *string) dev_dbg(dev, "failed to create device link to %s\n", dev_name(phy->dev.parent)); +out_unlock: + mutex_unlock(&phy_provider_mutex); return phy; } EXPORT_SYMBOL_GPL(phy_get); @@ -961,15 +972,17 @@ struct phy *devm_of_phy_get_by_index(struct device *dev, struct device_node *np, if (!ptr) return ERR_PTR(-ENOMEM); + mutex_lock(&phy_provider_mutex); phy = _of_phy_get(np, index); if (IS_ERR(phy)) { devres_free(ptr); - return phy; + goto out_unlock; } if (!try_module_get(phy->ops->owner)) { devres_free(ptr); - return ERR_PTR(-EPROBE_DEFER); + phy = ERR_PTR(-EPROBE_DEFER); + goto out_unlock; } get_device(&phy->dev); @@ -982,6 +995,8 @@ struct phy *devm_of_phy_get_by_index(struct device *dev, struct device_node *np, dev_dbg(dev, "failed to create device link to %s\n", dev_name(phy->dev.parent)); +out_unlock: + mutex_unlock(&phy_provider_mutex); return phy; } EXPORT_SYMBOL_GPL(devm_of_phy_get_by_index); diff --git a/drivers/platform/cznic/turris-omnia-mcu-base.c b/drivers/platform/cznic/turris-omnia-mcu-base.c index e8fc0d7b3343..f0416302a1f3 100644 --- a/drivers/platform/cznic/turris-omnia-mcu-base.c +++ b/drivers/platform/cznic/turris-omnia-mcu-base.c @@ -403,6 +403,7 @@ static const struct of_device_id of_omnia_mcu_match[] = { { .compatible = "cznic,turris-omnia-mcu" }, {} }; +MODULE_DEVICE_TABLE(of, of_omnia_mcu_match); static struct i2c_driver omnia_mcu_driver = { .probe = omnia_mcu_probe, diff --git a/drivers/platform/x86/amd/hfi/hfi.c b/drivers/platform/x86/amd/hfi/hfi.c index e0ebcb0c4acd..e1c776b71f3e 100644 --- a/drivers/platform/x86/amd/hfi/hfi.c +++ b/drivers/platform/x86/amd/hfi/hfi.c @@ -76,6 +76,8 @@ struct amd_hfi_data { struct amd_shmem_info *shmem; struct dentry *dbgfs_dir; + + int cpuhp_state; }; /** @@ -244,6 +246,8 @@ static void amd_hfi_remove(struct platform_device *pdev) { struct amd_hfi_data *dev = platform_get_drvdata(pdev); + cpuhp_remove_state(dev->cpuhp_state); + cancel_work_sync(&sched_amd_hfi_itmt_work); debugfs_remove_recursive(dev->dbgfs_dir); } @@ -494,6 +498,7 @@ static int amd_hfi_probe(struct platform_device *pdev) amd_hfi_online, amd_hfi_offline); if (ret < 0) return ret; + amd_hfi_data->cpuhp_state = ret; schedule_work(&sched_amd_hfi_itmt_work); diff --git a/drivers/platform/x86/asus-wmi.c b/drivers/platform/x86/asus-wmi.c index a65090429ca7..daf8bd17e638 100644 --- a/drivers/platform/x86/asus-wmi.c +++ b/drivers/platform/x86/asus-wmi.c @@ -2088,6 +2088,18 @@ static void asus_wmi_led_exit(struct asus_wmi *asus) if (asus->led_workqueue) destroy_workqueue(asus->led_workqueue); + + /* + * kbd_led is registered lazily by kbd_led_work: now that the + * workqueue is destroyed and asus_ref.asus is NULL, the work can + * neither run nor be queued anymore, furthermore leaving it to + * devres would run the unregister from devres_release_all(), + * after .remove() returned and the struct asus_wmi embedding + * kbd_led has been freed. + */ + if (asus->kbd_led_registered) + devm_led_classdev_unregister(&asus->platform_device->dev, + &asus->kbd_led); } static int asus_wmi_led_init(struct asus_wmi *asus) diff --git a/drivers/power/supply/pf1550-charger.c b/drivers/power/supply/pf1550-charger.c index 2ead1df60e65..e3d4e9817f91 100644 --- a/drivers/power/supply/pf1550-charger.c +++ b/drivers/power/supply/pf1550-charger.c @@ -566,6 +566,22 @@ static int pf1550_charger_probe(struct platform_device *pdev) platform_set_drvdata(pdev, chg); + psy_cfg.drv_data = chg; + + chg->charger = devm_power_supply_register(&pdev->dev, + &pf1550_charger_desc, + &psy_cfg); + if (IS_ERR(chg->charger)) + return dev_err_probe(&pdev->dev, PTR_ERR(chg->charger), + "failed: power supply register\n"); + + chg->battery = devm_power_supply_register(&pdev->dev, + &pf1550_battery_desc, + &psy_cfg); + if (IS_ERR(chg->battery)) + return dev_err_probe(&pdev->dev, PTR_ERR(chg->battery), + "failed: power supply register\n"); + ret = devm_delayed_work_autocancel(chg->dev, &chg->vbus_sense_work, pf1550_chg_vbus_work); if (ret) @@ -584,22 +600,6 @@ static int pf1550_charger_probe(struct platform_device *pdev) return dev_err_probe(chg->dev, ret, "failed to add battery sense work\n"); - psy_cfg.drv_data = chg; - - chg->charger = devm_power_supply_register(&pdev->dev, - &pf1550_charger_desc, - &psy_cfg); - if (IS_ERR(chg->charger)) - return dev_err_probe(&pdev->dev, PTR_ERR(chg->charger), - "failed: power supply register\n"); - - chg->battery = devm_power_supply_register(&pdev->dev, - &pf1550_battery_desc, - &psy_cfg); - if (IS_ERR(chg->battery)) - return dev_err_probe(&pdev->dev, PTR_ERR(chg->battery), - "failed: power supply register\n"); - for (i = 0; i < PF1550_CHARGER_IRQ_NR; i++) { irq = platform_get_irq(pdev, i); if (irq < 0) diff --git a/drivers/power/supply/sbs-battery.c b/drivers/power/supply/sbs-battery.c index 501c8e069755..6714313bb3a3 100644 --- a/drivers/power/supply/sbs-battery.c +++ b/drivers/power/supply/sbs-battery.c @@ -1068,6 +1068,13 @@ static void sbs_alert(struct i2c_client *client, enum i2c_alert_protocol prot, sbs_supply_changed(i2c_get_clientdata(client)); } +static void sbs_disable_work(void *data) +{ + struct sbs_info *chip = data; + + disable_delayed_work_sync(&chip->work); +} + static void sbs_external_power_changed(struct power_supply *psy) { struct sbs_info *chip = power_supply_get_drvdata(psy); @@ -1208,6 +1215,10 @@ static int sbs_probe(struct i2c_client *client) return dev_err_probe(&client->dev, PTR_ERR(chip->power_supply), "Failed to register power supply\n"); + rc = devm_add_action_or_reset(&client->dev, sbs_disable_work, chip); + if (rc) + return rc; + if (!chip->gpio_detect) goto out; diff --git a/drivers/pps/generators/pps_gen.c b/drivers/pps/generators/pps_gen.c index 5e207c75e340..d80e28dc31dc 100644 --- a/drivers/pps/generators/pps_gen.c +++ b/drivers/pps/generators/pps_gen.c @@ -63,23 +63,34 @@ static long pps_gen_cdev_ioctl(struct file *file, switch (cmd) { case PPS_GEN_SETENABLE: - dev_dbg(pps_gen->dev, "PPS_GEN_SETENABLE\n"); + dev_dbg(&pps_gen->dev, "PPS_GEN_SETENABLE\n"); ret = get_user(status, uiuarg); if (ret) return -EFAULT; - ret = pps_gen->info->enable(pps_gen, status); - if (ret) - return ret; - pps_gen->enabled = status; + scoped_guard(mutex, &pps_gen->info_lock) { + if (!pps_gen->info) + return -ENODEV; + + ret = pps_gen->info->enable(pps_gen, status); + if (ret) + return ret; + pps_gen->enabled = status; + } break; case PPS_GEN_USESYSTEMCLOCK: - dev_dbg(pps_gen->dev, "PPS_GEN_USESYSTEMCLOCK\n"); + dev_dbg(&pps_gen->dev, "PPS_GEN_USESYSTEMCLOCK\n"); - ret = put_user(pps_gen->info->use_system_clock, uiuarg); + scoped_guard(mutex, &pps_gen->info_lock) { + if (!pps_gen->info) + return -ENODEV; + status = pps_gen->info->use_system_clock; + } + + ret = put_user(status, uiuarg); if (ret) return -EFAULT; @@ -89,12 +100,12 @@ static long pps_gen_cdev_ioctl(struct file *file, struct pps_gen_event info; unsigned int ev = pps_gen->last_ev; - dev_dbg(pps_gen->dev, "PPS_GEN_FETCHEVENT\n"); + dev_dbg(&pps_gen->dev, "PPS_GEN_FETCHEVENT\n"); ret = wait_event_interruptible(pps_gen->queue, ev != pps_gen->last_ev); if (ret == -ERESTARTSYS) { - dev_dbg(pps_gen->dev, "pending signal caught\n"); + dev_dbg(&pps_gen->dev, "pending signal caught\n"); return -EINTR; } @@ -121,7 +132,7 @@ static int pps_gen_cdev_open(struct inode *inode, struct file *file) struct pps_gen_device *pps_gen = container_of(inode->i_cdev, struct pps_gen_device, cdev); - get_device(pps_gen->dev); + get_device(&pps_gen->dev); file->private_data = pps_gen; return 0; } @@ -130,7 +141,7 @@ static int pps_gen_cdev_release(struct inode *inode, struct file *file) { struct pps_gen_device *pps_gen = file->private_data; - put_device(pps_gen->dev); + put_device(&pps_gen->dev); return 0; } @@ -151,19 +162,15 @@ static void pps_gen_device_destruct(struct device *dev) { struct pps_gen_device *pps_gen = dev_get_drvdata(dev); - cdev_del(&pps_gen->cdev); - pr_debug("deallocating pps-gen%d\n", pps_gen->id); ida_free(&pps_gen_ida, pps_gen->id); - kfree(dev); kfree(pps_gen); } static int pps_gen_register_cdev(struct pps_gen_device *pps_gen) { int err; - dev_t devt; err = ida_alloc_max(&pps_gen_ida, PPS_GEN_MAX_SOURCES - 1, GFP_KERNEL); if (err < 0) { @@ -171,46 +178,61 @@ static int pps_gen_register_cdev(struct pps_gen_device *pps_gen) pr_err("too many PPS sources in the system\n"); err = -EBUSY; } + kfree(pps_gen); return err; } pps_gen->id = err; - devt = MKDEV(MAJOR(pps_gen_devt), pps_gen->id); + /* + * From here on pps_gen belongs to its device and is freed by + * pps_gen_device_destruct(). The cdev holds a reference to the + * device, so pps_gen stays around until the last file is closed. + */ + device_initialize(&pps_gen->dev); + pps_gen->dev.class = &pps_gen_class; + pps_gen->dev.parent = pps_gen->info->parent; + pps_gen->dev.devt = MKDEV(MAJOR(pps_gen_devt), pps_gen->id); + pps_gen->dev.release = pps_gen_device_destruct; + dev_set_drvdata(&pps_gen->dev, pps_gen); cdev_init(&pps_gen->cdev, &pps_gen_cdev_fops); pps_gen->cdev.owner = pps_gen->info->owner; - err = cdev_add(&pps_gen->cdev, devt, 1); + err = dev_set_name(&pps_gen->dev, "pps-gen%d", pps_gen->id); + if (err) + goto put_dev; + + err = cdev_device_add(&pps_gen->cdev, &pps_gen->dev); if (err) { pr_err("failed to add char device %d:%d\n", MAJOR(pps_gen_devt), pps_gen->id); - goto free_ida; + goto put_dev; } - pps_gen->dev = device_create(&pps_gen_class, pps_gen->info->parent, devt, - pps_gen, "pps-gen%d", pps_gen->id); - if (IS_ERR(pps_gen->dev)) { - err = PTR_ERR(pps_gen->dev); - goto del_cdev; - } - pps_gen->dev->release = pps_gen_device_destruct; - dev_set_drvdata(pps_gen->dev, pps_gen); pr_debug("generator got cdev (%d:%d)\n", MAJOR(pps_gen_devt), pps_gen->id); return 0; -del_cdev: - cdev_del(&pps_gen->cdev); -free_ida: - ida_free(&pps_gen_ida, pps_gen->id); +put_dev: + put_device(&pps_gen->dev); return err; } static void pps_gen_unregister_cdev(struct pps_gen_device *pps_gen) { pr_debug("unregistering pps-gen%d\n", pps_gen->id); - device_destroy(&pps_gen_class, pps_gen->dev->devt); + cdev_device_del(&pps_gen->cdev, &pps_gen->dev); + + /* + * An open file keeps pps_gen around, but the driver may free info as + * soon as we return. The sysfs files are gone now, so wait for the + * ioctls using info and make later ones fail. + */ + scoped_guard(mutex, &pps_gen->info_lock) + pps_gen->info = NULL; + + put_device(&pps_gen->dev); } /* @@ -241,21 +263,19 @@ struct pps_gen_device *pps_gen_register_source(const struct pps_gen_source_info pps_gen->info = info; pps_gen->enabled = false; + mutex_init(&pps_gen->info_lock); init_waitqueue_head(&pps_gen->queue); spin_lock_init(&pps_gen->lock); - /* Create the char device */ + /* Create the char device, this frees pps_gen on failure */ err = pps_gen_register_cdev(pps_gen); if (err < 0) { pr_err(" unable to create char device\n"); - goto kfree_pps_gen; + goto pps_gen_register_source_exit; } return pps_gen; -kfree_pps_gen: - kfree(pps_gen); - pps_gen_register_source_exit: pr_err("unable to register generator\n"); @@ -289,7 +309,7 @@ void pps_gen_event(struct pps_gen_device *pps_gen, { unsigned long flags; - dev_dbg(pps_gen->dev, "PPS generator event %u\n", event); + dev_dbg(&pps_gen->dev, "PPS generator event %u\n", event); spin_lock_irqsave(&pps_gen->lock, flags); diff --git a/drivers/rapidio/devices/rio_mport_cdev.c b/drivers/rapidio/devices/rio_mport_cdev.c index ad82c2108a56..47ba34b4afb2 100644 --- a/drivers/rapidio/devices/rio_mport_cdev.c +++ b/drivers/rapidio/devices/rio_mport_cdev.c @@ -2167,11 +2167,12 @@ static void mport_mm_open(struct vm_area_struct *vma) static void mport_mm_close(struct vm_area_struct *vma) { struct rio_mport_mapping *map = vma->vm_private_data; + struct mport_dev *md = map->md; rmcd_debug(MMAP, "%pad", &map->phys_addr); - mutex_lock(&map->md->buf_mutex); + mutex_lock(&md->buf_mutex); kref_put(&map->ref, mport_release_mapping); - mutex_unlock(&map->md->buf_mutex); + mutex_unlock(&md->buf_mutex); } static const struct vm_operations_struct vm_ops = { diff --git a/drivers/s390/char/vmcp.c b/drivers/s390/char/vmcp.c index b277f9d0c321..2cf94f23af2a 100644 --- a/drivers/s390/char/vmcp.c +++ b/drivers/s390/char/vmcp.c @@ -104,13 +104,11 @@ static int vmcp_open(struct inode *inode, struct file *file) if (!capable(CAP_SYS_ADMIN)) return -EPERM; - session = kmalloc_obj(*session); + session = kzalloc_obj(*session); if (!session) return -ENOMEM; session->bufsize = PAGE_SIZE; - session->response = NULL; - session->resp_size = 0; mutex_init(&session->mutex); file->private_data = session; return nonseekable_open(inode, file); diff --git a/drivers/s390/crypto/zcrypt_api.c b/drivers/s390/crypto/zcrypt_api.c index ec6a4c2f9f04..80435fd8e6ad 100644 --- a/drivers/s390/crypto/zcrypt_api.c +++ b/drivers/s390/crypto/zcrypt_api.c @@ -854,7 +854,7 @@ static long _zcrypt_send_cprb(u32 xflags, struct ap_perms *perms, struct ica_xcRB *xcrb) { bool userspace = xflags & ZCRYPT_XFLAG_USERSPACE; - unsigned int card, domain, func_code = 0; + unsigned int card, domain, _dom, func_code = 0; unsigned int wgt = 0, pref_wgt = 0; struct zcrypt_queue *zq, *pref_zq; struct zcrypt_card *zc, *pref_zc; @@ -877,10 +877,17 @@ static long _zcrypt_send_cprb(u32 xflags, struct ap_perms *perms, print_hex_dump_debug("ccareq: ", DUMP_PREFIX_ADDRESS, 16, 1, ap_msg.msg, ap_msg.len, false); + /* Check domain for either AUTOSEL_DOM or in range 0...AP_DOMAINS-1 */ + if (domain != AUTOSEL_DOM && domain >= AP_DOMAINS) { + rc = -EINVAL; + goto out; + } + /* Spectre-v1: sanitize domain unconditionally for later use */ + _dom = array_index_nospec(domain, AP_DOMAINS); + if (perms != &ap_perms && domain < AP_DOMAINS) { if (ap_msg.flags & AP_MSG_FLAG_ADMIN) { - domain = array_index_nospec(domain, AP_DOMAINS); - if (!test_bit_inv(domain, perms->adm)) { + if (!test_bit_inv(_dom, perms->adm)) { rc = -ENODEV; goto out; } @@ -893,10 +900,11 @@ static long _zcrypt_send_cprb(u32 xflags, struct ap_perms *perms, * If a valid target domain is set and this domain is NOT a usage * domain but a control only domain, autoselect target domain. */ - if (domain < AP_DOMAINS && - !ap_test_config_usage_domain(domain) && - ap_test_config_ctrl_domain(domain)) - domain = AUTOSEL_DOM; + if (domain < AP_DOMAINS) { + if (!ap_test_config_usage_domain(_dom) && + ap_test_config_ctrl_domain(_dom)) + domain = AUTOSEL_DOM; + } pref_zc = NULL; pref_zq = NULL; @@ -1041,7 +1049,7 @@ static long _zcrypt_send_ep11_cprb(u32 xflags, struct ap_perms *perms, struct ep11_target_dev *targets = NULL; unsigned short target_num; unsigned int wgt = 0, pref_wgt = 0; - unsigned int func_code = 0, domain; + unsigned int func_code = 0, domain, _dom; struct ap_message ap_msg; int cpen, qpen, qid = 0, rc; struct module *mod; @@ -1078,10 +1086,17 @@ static long _zcrypt_send_ep11_cprb(u32 xflags, struct ap_perms *perms, print_hex_dump_debug("ep11req: ", DUMP_PREFIX_ADDRESS, 16, 1, ap_msg.msg, ap_msg.len, false); + /* Check domain for either AUTOSEL_DOM or in range 0...AP_DOMAINS-1 */ + if (domain != AUTOSEL_DOM && domain >= AP_DOMAINS) { + rc = -EINVAL; + goto out; + } + /* Spectre-v1: sanitize domain unconditionally for later use */ + _dom = array_index_nospec(domain, AP_DOMAINS); + if (perms != &ap_perms && domain < AP_DOMAINS) { if (ap_msg.flags & AP_MSG_FLAG_ADMIN) { - domain = array_index_nospec(domain, AP_DOMAINS); - if (!test_bit_inv(domain, perms->adm)) { + if (!test_bit_inv(_dom, perms->adm)) { rc = -ENODEV; goto out; } diff --git a/drivers/s390/crypto/zcrypt_ccamisc.c b/drivers/s390/crypto/zcrypt_ccamisc.c index 19909bf43dc9..862947431164 100644 --- a/drivers/s390/crypto/zcrypt_ccamisc.c +++ b/drivers/s390/crypto/zcrypt_ccamisc.c @@ -1627,6 +1627,7 @@ int cca_query_crypto_facility(u16 cardnr, u16 domain, u8 subfunc_code[2]; u8 lvdata[]; } __packed * prepparm; + size_t datalen; /* get already prepared memory for 2 cprbs with param block each */ rc = alloc_and_prep_cprbmem(parmbsize, &mem, @@ -1673,27 +1674,44 @@ int cca_query_crypto_facility(u16 cardnr, u16 domain, prepcblk->rpl_parmb = (u8 __user *)ptr; prepparm = (struct fqrepparm *)ptr; ptr = prepparm->lvdata; + datalen = parmbsize - 2 * sizeof(u8); /* check and possibly copy reply rule array */ len = *((u16 *)ptr); + if (len > datalen) { + ZCRYPT_DBF_ERR("%s reply rule array len %u exceeds datalen %zu\n", + __func__, len, datalen); + rc = -EIO; + goto out; + } + datalen -= sizeof(u16); + ptr += sizeof(u16); if (len > sizeof(u16)) { - ptr += sizeof(u16); len -= sizeof(u16); if (rarray && rarraylen && *rarraylen > 0) { *rarraylen = (len > *rarraylen ? *rarraylen : len); memcpy(rarray, ptr, *rarraylen); } + datalen -= len; ptr += len; } /* check and possible copy reply var array */ len = *((u16 *)ptr); + if (len > datalen) { + ZCRYPT_DBF_ERR("%s reply var array len %u exceeds datalen %zu\n", + __func__, len, datalen); + rc = -EIO; + goto out; + } + datalen -= sizeof(u16); + ptr += sizeof(u16); if (len > sizeof(u16)) { - ptr += sizeof(u16); len -= sizeof(u16); if (varray && varraylen && *varraylen > 0) { *varraylen = (len > *varraylen ? *varraylen : len); memcpy(varray, ptr, *varraylen); } + datalen -= len; ptr += len; } diff --git a/drivers/s390/crypto/zcrypt_msgtype50.c b/drivers/s390/crypto/zcrypt_msgtype50.c index d6fc2d8e7fad..5907fb2a6920 100644 --- a/drivers/s390/crypto/zcrypt_msgtype50.c +++ b/drivers/s390/crypto/zcrypt_msgtype50.c @@ -152,6 +152,11 @@ struct type80_hdr { unsigned char reserved3[8]; } __packed; +struct type80_reply { + struct type80_hdr hdr; + char data[]; +} __packed; + int get_rsa_modex_fc(struct ica_rsa_modexpo *mex, int *fcode) { if (!mex->inputdatalength) @@ -340,32 +345,43 @@ static int ICACRT_msg_to_type50CRT_msg(struct zcrypt_queue *zq, * @data: pointer to user output data * @length: size of user output data * - * Returns 0 on success or -EFAULT. + * Returns 0 on success or neg. errno value on failure. */ static int convert_type80(struct zcrypt_queue *zq, struct ap_message *reply, char __user *outputdata, unsigned int outputdatalength) { - struct type80_hdr *t80h = reply->msg; - unsigned char *data; + struct type80_reply *msg = reply->msg; + size_t payload_len; + + /* + * reply->len is always >= sizeof(struct type80_reply) here and + * outputdatalength (the modulus size) is guaranteed to be equal to + * inputdatalength and 0 < outputdatalength <= CEX3A_MAX_MOD_SIZE, + * also payload size may be >= outputdatalength. + */ + + payload_len = reply->len - sizeof(msg->hdr); - if (t80h->len < sizeof(*t80h) + outputdatalength) { - /* The result is too short, the CEXxA card may not do that.. */ + if (outputdatalength > payload_len) { + /* We expect at least outputdatalength bytes, broken card ? */ zq->online = 0; pr_err("Crypto dev=%02x.%04x code=0x%02x => online=0 rc=EAGAIN\n", AP_QID_CARD(zq->queue->qid), - AP_QID_QUEUE(zq->queue->qid), t80h->code); + AP_QID_QUEUE(zq->queue->qid), msg->hdr.code); ZCRYPT_DBF_ERR("%s dev=%02x.%04x code=0x%02x => online=0 rc=EAGAIN\n", __func__, AP_QID_CARD(zq->queue->qid), - AP_QID_QUEUE(zq->queue->qid), t80h->code); + AP_QID_QUEUE(zq->queue->qid), msg->hdr.code); ap_send_online_uevent(&zq->queue->ap_dev, zq->online); return -EAGAIN; } - BUG_ON(t80h->len > CEX3A_MAX_RESPONSE_SIZE); - data = reply->msg + t80h->len - outputdatalength; - if (copy_to_user(outputdata, data, outputdatalength)) + + if (copy_to_user(outputdata, + msg->data + payload_len - outputdatalength, + outputdatalength)) return -EFAULT; + return 0; } @@ -374,14 +390,19 @@ static int convert_response(struct zcrypt_queue *zq, char __user *outputdata, unsigned int outputdatalength) { - /* Response type byte is the second byte in the response. */ - unsigned char rtype = ((unsigned char *)reply->msg)[1]; + struct type80_reply *msg = reply->msg; + + /* reply->len is always >= sizeof(struct error_hdr) here */ - switch (rtype) { + switch (msg->hdr.type) { case TYPE82_RSP_CODE: case TYPE88_RSP_CODE: return convert_error(zq, reply); case TYPE80_RSP_CODE: + if (msg->hdr.code) + return convert_error(zq, reply); + if (reply->len < sizeof(struct type80_reply)) + return -EINVAL; return convert_type80(zq, reply, outputdata, outputdatalength); default: /* Unknown response type, this should NEVER EVER happen */ @@ -389,11 +410,11 @@ static int convert_response(struct zcrypt_queue *zq, pr_err("Crypto dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n", AP_QID_CARD(zq->queue->qid), AP_QID_QUEUE(zq->queue->qid), - (int)rtype); + (int)msg->hdr.type); ZCRYPT_DBF_ERR( "%s dev=%02x.%04x unknown response type 0x%02x => online=0 rc=EAGAIN\n", __func__, AP_QID_CARD(zq->queue->qid), - AP_QID_QUEUE(zq->queue->qid), (int)rtype); + AP_QID_QUEUE(zq->queue->qid), (int)msg->hdr.type); ap_send_online_uevent(&zq->queue->ap_dev, zq->online); return -EAGAIN; } @@ -416,26 +437,46 @@ static void zcrypt_msgtype50_receive(struct ap_queue *aq, .reply_code = REP82_ERROR_MACHINE_FAILURE, }; struct type80_hdr *t80h; - int len; + size_t len; /* Copy the reply message to the request message buffer. */ if (!reply) goto out; /* ap_msg->rc indicates the error */ + t80h = reply->msg; - if (t80h->type == TYPE80_RSP_CODE) { - len = t80h->len; - if (len > reply->bufsize || len > msg->bufsize || - len != reply->len) { - pr_debug("len mismatch => EMSGSIZE\n"); - msg->rc = -EMSGSIZE; + + if (reply->len < sizeof(*t80h) || + t80h->type != TYPE80_RSP_CODE) { + if (reply->len < sizeof(error_reply)) { + /* total broken reply, use static error reply instead */ + memcpy(msg->msg, &error_reply, sizeof(error_reply)); + msg->len = sizeof(error_reply); goto out; + } else { + /* malformed reply, convert function will handle this */ + len = reply->len; + goto copy_len_and_out; } - memcpy(msg->msg, reply->msg, len); - msg->len = len; - } else { - memcpy(msg->msg, reply->msg, sizeof(error_reply)); - msg->len = sizeof(error_reply); } + + len = t80h->len; + +copy_len_and_out: + if (len != reply->len) { + msg->rc = -EMSGSIZE; + pr_debug("len %zu rpl.len %zu mismatch, msg.rc=%d\n", + len, reply->len, msg->rc); + goto out; + } + if (len > reply->bufsize || len > msg->bufsize) { + msg->rc = -EMSGSIZE; + pr_debug("len %zu exceeds buf %zu/%zu, msg.rc=%d\n", + len, reply->bufsize, msg->bufsize, msg->rc); + goto out; + } + memcpy(msg->msg, reply->msg, len); + msg->len = len; + out: complete(&msg->response.work); } diff --git a/drivers/s390/crypto/zcrypt_msgtype6.c b/drivers/s390/crypto/zcrypt_msgtype6.c index 3df1d676de5d..5962224bfb8b 100644 --- a/drivers/s390/crypto/zcrypt_msgtype6.c +++ b/drivers/s390/crypto/zcrypt_msgtype6.c @@ -766,6 +766,15 @@ static int convert_type86_rng(struct zcrypt_queue *zq, if (msg->cprbx.ccp_rtcode != 0 || msg->cprbx.ccp_rscode != 0) return -EINVAL; + /* + * Note that offset2 and count2 have already been checked in + * zcrypt_msgtype6_receive(). So only check for valid count2 + * and for not exceeding the hard coded rng buffer size. + */ + if (!msg->fmt2.count2) + return -EINVAL; + if (msg->fmt2.count2 > ZCRYPT_RNG_BUFFER_SIZE) + return -EMSGSIZE; memcpy(buffer, data + msg->fmt2.offset2, msg->fmt2.count2); return msg->fmt2.count2; } @@ -777,11 +786,17 @@ static int convert_response_ica(struct zcrypt_queue *zq, { struct type86x_reply *msg = reply->msg; + /* reply->len is always >= sizeof(struct error_hdr) here */ + switch (msg->hdr.type) { case TYPE82_RSP_CODE: case TYPE88_RSP_CODE: return convert_error(zq, reply); case TYPE86_RSP_CODE: + if (msg->hdr.reply_code) + return convert_error(zq, reply); + if (reply->len < sizeof(struct type86x_reply)) + return -EINVAL; if (msg->cprbx.ccp_rtcode && msg->cprbx.ccp_rscode == 0x14f && outputdatalength > 256) { @@ -792,8 +807,6 @@ static int convert_response_ica(struct zcrypt_queue *zq, return -EINVAL; } } - if (msg->hdr.reply_code) - return convert_error(zq, reply); if (msg->cprbx.cprb_ver_id == 0x02) return convert_type86_ica(zq, reply, outputdata, outputdatalength); @@ -820,6 +833,8 @@ static int convert_response_xcrb(bool userspace, struct zcrypt_queue *zq, { struct type86x_reply *msg = reply->msg; + /* reply->len is always >= sizeof(struct error_hdr) here */ + switch (msg->hdr.type) { case TYPE82_RSP_CODE: case TYPE88_RSP_CODE: @@ -827,9 +842,16 @@ static int convert_response_xcrb(bool userspace, struct zcrypt_queue *zq, return convert_error(zq, reply); case TYPE86_RSP_CODE: if (msg->hdr.reply_code) { - xcrb->status = msg->fmt2.apfs; + if (reply->len < sizeof(struct type86_fmt2_msg)) + xcrb->status = 0x0008044DL; + else + xcrb->status = msg->fmt2.apfs; return convert_error(zq, reply); } + if (reply->len < sizeof(struct type86_fmt2_msg) + + offsetof(struct CPRBX, cprb_ver_id) + + sizeof(msg->cprbx.cprb_ver_id)) + return -EINVAL; if (msg->cprbx.cprb_ver_id == 0x02) return convert_type86_xcrb(userspace, zq, reply, xcrb); fallthrough; /* wrong cprb version is an unknown response */ @@ -854,6 +876,8 @@ static int convert_response_ep11_xcrb(bool userspace, struct zcrypt_queue *zq, { struct type86_ep11_reply *msg = reply->msg; + /* reply->len is always >= sizeof(struct error_hdr) here */ + switch (msg->hdr.type) { case TYPE82_RSP_CODE: case TYPE87_RSP_CODE: @@ -861,6 +885,8 @@ static int convert_response_ep11_xcrb(bool userspace, struct zcrypt_queue *zq, case TYPE86_RSP_CODE: if (msg->hdr.reply_code) return convert_error(zq, reply); + if (reply->len < sizeof(struct type86_ep11_reply)) + return -EINVAL; if (msg->cprbx.cprb_ver_id == 0x04) return convert_type86_ep11_xcrb(userspace, zq, reply, xcrb); fallthrough; /* wrong cprb version is an unknown resp */ @@ -885,6 +911,8 @@ static int convert_response_rng(struct zcrypt_queue *zq, { struct type86x_reply *msg = reply->msg; + /* reply->len is always >= sizeof(struct error_hdr) here */ + switch (msg->hdr.type) { case TYPE82_RSP_CODE: case TYPE88_RSP_CODE: @@ -892,6 +920,8 @@ static int convert_response_rng(struct zcrypt_queue *zq, case TYPE86_RSP_CODE: if (msg->hdr.reply_code) return -EINVAL; + if (reply->len < sizeof(struct type86x_reply)) + return -EINVAL; if (msg->cprbx.cprb_ver_id == 0x02) return convert_type86_rng(zq, reply, data); fallthrough; /* wrong cprb version is an unknown response */ @@ -928,48 +958,85 @@ static void zcrypt_msgtype6_receive(struct ap_queue *aq, }; struct ap_response_type *resp_type = &msg->response; struct type86x_reply *t86r; - int len; + size_t minlen, len; /* Copy the reply message to the request message buffer. */ if (!reply) goto out; /* ap_msg->rc indicates the error */ + t86r = reply->msg; - if (t86r->hdr.type == TYPE86_RSP_CODE && - t86r->cprbx.cprb_ver_id == 0x02) { - switch (resp_type->type) { - case CEXXC_RESPONSE_TYPE_ICA: - len = sizeof(struct type86x_reply) + t86r->length; - if (len > reply->bufsize || len > msg->bufsize || - len != reply->len) { - pr_debug("len mismatch => EMSGSIZE\n"); - msg->rc = -EMSGSIZE; - goto out; - } - memcpy(msg->msg, reply->msg, len); - msg->len = len; - break; - case CEXXC_RESPONSE_TYPE_XCRB: - if (t86r->fmt2.count2) - len = t86r->fmt2.offset2 + t86r->fmt2.count2; - else - len = t86r->fmt2.offset1 + t86r->fmt2.count1; - if (len > reply->bufsize || len > msg->bufsize || - len != reply->len) { - pr_debug("len mismatch => EMSGSIZE\n"); + minlen = sizeof(t86r->hdr) + sizeof(t86r->fmt2) + + offsetof(struct CPRBX, cprb_ver_id) + + sizeof(t86r->cprbx.cprb_ver_id); + + if (reply->len < minlen || + t86r->hdr.type != TYPE86_RSP_CODE || + t86r->cprbx.cprb_ver_id != 0x02) { + if (reply->len < sizeof(error_reply)) { + /* total broken reply, use static error reply instead */ + memcpy(msg->msg, &error_reply, sizeof(error_reply)); + msg->len = sizeof(error_reply); + goto out; + } else { + /* malformed reply, convert function will handle this */ + len = reply->len; + goto copy_len_and_out; + } + } + + switch (resp_type->type) { + case CEXXC_RESPONSE_TYPE_ICA: + if (reply->len < sizeof(struct type86x_reply)) { + msg->rc = -EMSGSIZE; + pr_debug("rpl.len %zu < struct type86_reply, msg.rc=%d\n", + reply->len, msg->rc); + goto out; + } + len = sizeof(struct type86x_reply) + (size_t)t86r->length; + break; + case CEXXC_RESPONSE_TYPE_XCRB: + len = (size_t)t86r->fmt2.offset1 + (size_t)t86r->fmt2.count1; + if (len > reply->len) { + msg->rc = -EMSGSIZE; + pr_debug("offset1 %u count1 %u rpl.len %zu mismatch, msg.rc=%d\n", + t86r->fmt2.offset1, t86r->fmt2.count1, + reply->len, msg->rc); + goto out; + } + if (t86r->fmt2.count2) { + len = (size_t)t86r->fmt2.offset2 + + (size_t)t86r->fmt2.count2; + if (len > reply->len) { msg->rc = -EMSGSIZE; + pr_debug("offset2 %u count2 %u rpl.len %zu mismatch, msg.rc=%d\n", + t86r->fmt2.offset2, t86r->fmt2.count2, + reply->len, msg->rc); goto out; } - memcpy(msg->msg, reply->msg, len); - msg->len = len; - break; - default: - memcpy(msg->msg, &error_reply, sizeof(error_reply)); - msg->len = sizeof(error_reply); } - } else { - memcpy(msg->msg, reply->msg, sizeof(error_reply)); + break; + default: + memcpy(msg->msg, &error_reply, sizeof(error_reply)); msg->len = sizeof(error_reply); + goto out; + } + +copy_len_and_out: + if (len != reply->len) { + msg->rc = -EMSGSIZE; + pr_debug("len %zu rpl.len %zu mismatch, msg.rc=%d\n", + len, reply->len, msg->rc); + goto out; + } + if (len > reply->bufsize || len > msg->bufsize) { + msg->rc = -EMSGSIZE; + pr_debug("len %zu exceeds buf %zu/%zu, msg.rc=%d\n", + len, reply->bufsize, msg->bufsize, msg->rc); + goto out; } + memcpy(msg->msg, reply->msg, len); + msg->len = len; + out: complete(&resp_type->work); } @@ -992,34 +1059,65 @@ static void zcrypt_msgtype6_receive_ep11(struct ap_queue *aq, }; struct ap_response_type *resp_type = &msg->response; struct type86_ep11_reply *t86r; - int len; + size_t minlen, len; /* Copy the reply message to the request message buffer. */ if (!reply) goto out; /* ap_msg->rc indicates the error */ + t86r = reply->msg; - if (t86r->hdr.type == TYPE86_RSP_CODE && - t86r->cprbx.cprb_ver_id == 0x04) { - switch (resp_type->type) { - case CEXXC_RESPONSE_TYPE_EP11: - len = t86r->fmt2.offset1 + t86r->fmt2.count1; - if (len > reply->bufsize || len > msg->bufsize || - len != reply->len) { - pr_debug("len mismatch => EMSGSIZE\n"); - msg->rc = -EMSGSIZE; - goto out; - } - memcpy(msg->msg, reply->msg, len); - msg->len = len; - break; - default: + minlen = sizeof(t86r->hdr) + sizeof(t86r->fmt2) + + offsetof(struct ep11_cprb, cprb_ver_id) + + sizeof(t86r->cprbx.cprb_ver_id); + + if (reply->len < minlen || + t86r->hdr.type != TYPE86_RSP_CODE || + t86r->cprbx.cprb_ver_id != 0x04) { + if (reply->len < sizeof(error_reply)) { + /* total broken reply, use static error reply instead */ memcpy(msg->msg, &error_reply, sizeof(error_reply)); msg->len = sizeof(error_reply); + goto out; + } else { + /* malformed reply, convert function will handle this */ + len = reply->len; + goto copy_len_and_out; } - } else { - memcpy(msg->msg, reply->msg, sizeof(error_reply)); + } + + switch (resp_type->type) { + case CEXXC_RESPONSE_TYPE_EP11: + len = (size_t)t86r->fmt2.offset1 + (size_t)t86r->fmt2.count1; + if (len > reply->len) { + msg->rc = -EMSGSIZE; + pr_debug("offset1 %u count1 %u rpl.len %zu mismatch, msg.rc=%d\n", + t86r->fmt2.offset1, t86r->fmt2.count1, + reply->len, msg->rc); + goto out; + } + break; + default: + memcpy(msg->msg, &error_reply, sizeof(error_reply)); msg->len = sizeof(error_reply); + goto out; + } + +copy_len_and_out: + if (len != reply->len) { + msg->rc = -EMSGSIZE; + pr_debug("len %zu rpl.len %zu mismatch, msg.rc=%d\n", + len, reply->len, msg->rc); + goto out; + } + if (len > reply->bufsize || len > msg->bufsize) { + msg->rc = -EMSGSIZE; + pr_debug("len %zu exceeds buf %zu/%zu, msg.rc=%d\n", + len, reply->bufsize, msg->bufsize, msg->rc); + goto out; } + memcpy(msg->msg, reply->msg, len); + msg->len = len; + out: complete(&resp_type->work); } diff --git a/drivers/scsi/dc395x.c b/drivers/scsi/dc395x.c index ba490174a417..72911c7d3dd1 100644 --- a/drivers/scsi/dc395x.c +++ b/drivers/scsi/dc395x.c @@ -3808,13 +3808,11 @@ static void adapter_uninit_chip(struct AdapterCtlBlk *acb) static void adapter_uninit(struct AdapterCtlBlk *acb) { unsigned long flags; - DC395x_LOCK_IO(acb->scsi_host, flags); - /* remove timers */ - if (timer_pending(&acb->waiting_timer)) - timer_delete(&acb->waiting_timer); - if (timer_pending(&acb->selto_timer)) - timer_delete(&acb->selto_timer); + /* Drain the self-rearming timer; must not run under host_lock. */ + timer_shutdown_sync(&acb->waiting_timer); + + DC395x_LOCK_IO(acb->scsi_host, flags); adapter_uninit_chip(acb); adapter_remove_and_free_all_devices(acb); diff --git a/drivers/spi/spi-pci1xxxx.c b/drivers/spi/spi-pci1xxxx.c index af6ed78493e3..c50e6841de52 100644 --- a/drivers/spi/spi-pci1xxxx.c +++ b/drivers/spi/spi-pci1xxxx.c @@ -542,6 +542,8 @@ static int pci1xxxx_spi_transfer_with_dma(struct spi_controller *spi_ctlr, struct pci1xxxx_spi_internal *p = spi_controller_get_devdata(spi_ctlr); struct pci1xxxx_spi *par = p->parent; dma_addr_t tx_dma_addr = 0; + unsigned long flags; + bool engines_stopped; int ret = 0; u32 regval; @@ -618,12 +620,72 @@ static int pci1xxxx_spi_transfer_with_dma(struct spi_controller *spi_ctlr, p->dma_aborted_wr = false; ret = -ECANCELED; } + + /* + * A timed out transfer leaves the DMA engines running and + * the done/abort interrupt status armed with p->xfer, + * p->tx_sgl and p->rx_sgl still registered. Stop the + * engines, drain the pending status and synchronize the + * IRQs before dropping the transfer state, otherwise a + * late interrupt makes the ISRs reprogram the engines + * from or signal completion through memory which the SPI + * core is about to unmap or free. + */ + writel(SPI_DMA_ENGINE_DIS, + par->dma_offset_bar + SPI_DMA_GLOBAL_WR_ENGINE_EN); + writel(SPI_DMA_ENGINE_DIS, + par->dma_offset_bar + SPI_DMA_GLOBAL_RD_ENGINE_EN); + engines_stopped = + !(readl_poll_timeout(par->dma_offset_bar + + SPI_DMA_GLOBAL_WR_ENGINE_EN, + regval, regval == 0x0, 0, + USEC_PER_MSEC) || + readl_poll_timeout(par->dma_offset_bar + + SPI_DMA_GLOBAL_RD_ENGINE_EN, + regval, regval == 0x0, 0, + USEC_PER_MSEC)); + if (!engines_stopped) + dev_warn(&par->dev->dev, + "SPI DMA engines failed to stop\n"); + + spin_lock_irqsave(&par->dma_wr_reg_lock, flags); + writel((SPI_DMA_DONE_INT_MASK(p->hw_inst) | + SPI_DMA_ABORT_INT_MASK(p->hw_inst)), + par->dma_offset_bar + SPI_DMA_INTR_WR_CLR); + spin_unlock_irqrestore(&par->dma_wr_reg_lock, flags); + spin_lock_irqsave(&par->dma_rd_reg_lock, flags); + writel((SPI_DMA_DONE_INT_MASK(p->hw_inst) | + SPI_DMA_ABORT_INT_MASK(p->hw_inst)), + par->dma_offset_bar + SPI_DMA_INTR_RD_CLR); + spin_unlock_irqrestore(&par->dma_rd_reg_lock, flags); + + p->spi_xfer_in_progress = false; + synchronize_irq(p->irq[1]); + synchronize_irq(p->irq[2]); + + /* + * Turn the engines back on only once both are confirmed + * to have stopped: re-enabling an engine that failed to + * stop would cancel the pending stop request and let it + * run into buffers the SPI core is about to unmap or + * free. An engine that failed to stop is left disabled, + * as the abort paths already do. + */ + if (engines_stopped) { + writel(SPI_DMA_ENGINE_EN, + par->dma_offset_bar + SPI_DMA_GLOBAL_WR_ENGINE_EN); + writel(SPI_DMA_ENGINE_EN, + par->dma_offset_bar + SPI_DMA_GLOBAL_RD_ENGINE_EN); + } goto error; } ret = 0; error: p->spi_xfer_in_progress = false; + p->xfer = NULL; + p->tx_sgl = NULL; + p->rx_sgl = NULL; return ret; } @@ -695,6 +757,9 @@ static irqreturn_t pci1xxxx_spi_isr_dma_rd(int irq, void *dev) unsigned long flags; u32 regval; + if (!p->spi_xfer_in_progress) + return IRQ_NONE; + /* Clear the DMA RD INT and start spi xfer*/ regval = readl(p->parent->dma_offset_bar + SPI_DMA_INTR_RD_STS); if (regval) { @@ -723,6 +788,9 @@ static irqreturn_t pci1xxxx_spi_isr_dma_wr(int irq, void *dev) unsigned long flags; u32 regval; + if (!p->spi_xfer_in_progress) + return IRQ_NONE; + /* Clear the DMA WR INT */ regval = readl(p->parent->dma_offset_bar + SPI_DMA_INTR_WR_STS); if (regval) { @@ -755,6 +823,9 @@ static irqreturn_t pci1xxxx_spi_isr_dma(int irq, void *dev) irqreturn_t spi_int_fired = IRQ_NONE; u32 regval; + if (!p->spi_xfer_in_progress) + return IRQ_NONE; + /* Clear the SPI GO_BIT Interrupt */ regval = readl(p->parent->reg_base + SPI_MST_EVENT_REG_OFFSET(p->hw_inst)); if (regval & SPI_INTR) { diff --git a/drivers/spmi/spmi.c b/drivers/spmi/spmi.c index 57b7c0cb4240..f6d2f4bddfcd 100644 --- a/drivers/spmi/spmi.c +++ b/drivers/spmi/spmi.c @@ -24,6 +24,7 @@ static void spmi_dev_release(struct device *dev) { struct spmi_device *sdev = to_spmi_device(dev); + of_node_put(dev->of_node); kfree(sdev); } @@ -517,7 +518,7 @@ static void of_spmi_register_devices(struct spmi_controller *ctrl) if (!sdev) continue; - device_set_node(&sdev->dev, of_fwnode_handle(node)); + device_set_node(&sdev->dev, of_fwnode_handle(of_node_get(node))); sdev->usid = (u8)reg[0]; err = spmi_device_add(sdev); diff --git a/drivers/staging/rtl8723bs/core/rtw_recv.c b/drivers/staging/rtl8723bs/core/rtw_recv.c index 7568fc514d7c..4756e0fedd46 100644 --- a/drivers/staging/rtl8723bs/core/rtw_recv.c +++ b/drivers/staging/rtl8723bs/core/rtw_recv.c @@ -426,8 +426,21 @@ static union recv_frame *decryptor(struct adapter *padapter, union recv_frame *p u32 res = _SUCCESS; if (prxattrib->encrypt > 0) { - u8 *iv = precv_frame->u.hdr.rx_data + prxattrib->hdrlen; + u8 *iv; + u32 min_len = prxattrib->hdrlen + prxattrib->iv_len + prxattrib->icv_len; + /* TKIP appends an 8-byte Michael MIC that icv_len doesn't account for */ + if (prxattrib->encrypt == _TKIP_) + min_len += 8; + + /* a protected frame must be long enough to hold the IV and ICV/MIC */ + if (precv_frame->u.hdr.len < min_len) { + rtw_free_recvframe(precv_frame, + &padapter->recvpriv.free_recv_queue); + return NULL; + } + + iv = precv_frame->u.hdr.rx_data + prxattrib->hdrlen; prxattrib->key_index = (((iv[3]) >> 6) & 0x3); if (prxattrib->key_index > WEP_KEYS) { @@ -1395,6 +1408,10 @@ static signed int validate_80211w_mgmt(struct adapter *adapter, union recv_frame if (!mgmt_DATA) goto validate_80211w_fail; precv_frame = decryptor(adapter, precv_frame); + if (!precv_frame) { + kfree(mgmt_DATA); + goto validate_80211w_fail; + } /* save actual management data frame body */ memcpy(mgmt_DATA, ptr + pattrib->hdrlen + pattrib->iv_len, data_len); /* overwrite the iv field */ @@ -1402,8 +1419,6 @@ static signed int validate_80211w_mgmt(struct adapter *adapter, union recv_frame /* remove the iv and icv length */ pattrib->pkt_len = pattrib->pkt_len - pattrib->iv_len - pattrib->icv_len; kfree(mgmt_DATA); - if (!precv_frame) - goto validate_80211w_fail; } else if (is_multicast_ether_addr(GetAddr1Ptr(ptr)) && (subtype == WIFI_DEAUTH || subtype == WIFI_DISASSOC)) { signed int BIP_ret = _SUCCESS; diff --git a/drivers/thunderbolt/icm.c b/drivers/thunderbolt/icm.c index 669807f0eaf8..a295a5e6cc9c 100644 --- a/drivers/thunderbolt/icm.c +++ b/drivers/thunderbolt/icm.c @@ -2072,7 +2072,7 @@ static int complete_rpm(struct device *dev, void *data) { struct tb_switch *sw = tb_to_switch(dev); - if (sw) + if (sw && sw->tb == data) complete(&sw->rpm_complete); return 0; } @@ -2088,8 +2088,8 @@ static void remove_unplugged_switch(struct tb_switch *sw) * tb_switch_remove() calls pm_runtime_get_sync() that then waits * for it. */ - complete_rpm(&sw->dev, NULL); - bus_for_each_dev(&tb_bus_type, &sw->dev, NULL, complete_rpm); + complete_rpm(&sw->dev, sw->tb); + bus_for_each_dev(&tb_bus_type, &sw->dev, sw->tb, complete_rpm); tb_switch_remove(sw); pm_runtime_mark_last_busy(parent); diff --git a/drivers/tty/serial/serial_core.c b/drivers/tty/serial/serial_core.c index 6332ed545c89..d873df2f1529 100644 --- a/drivers/tty/serial/serial_core.c +++ b/drivers/tty/serial/serial_core.c @@ -896,7 +896,11 @@ static int uart_set_info(struct tty_struct *tty, struct tty_port *port, upf_t old_flags, new_flags; int retval; - if (!uport || tty_io_error(tty)) + /* + * TTY_IO_ERROR is also set on a port that could not be started, and + * this is how such a port is configured. Only refuse a hung up port. + */ + if (!uport || !tty_port_active(port)) return -EIO; new_port = new_info->port; @@ -1144,7 +1148,8 @@ static int uart_do_autoconfig(struct tty_struct *tty, struct uart_state *state) */ scoped_cond_guard(mutex_intr, return -ERESTARTSYS, &port->mutex) { uport = uart_port_check(state); - if (!uport || tty_io_error(tty)) + /* As in uart_set_info(), only refuse a hung up port. */ + if (!uport || !tty_port_active(port)) return -EIO; if (tty_port_users(port) != 1) @@ -3236,6 +3241,13 @@ static void serial_core_remove_one_port(struct uart_driver *drv, if (uart_console(uport)) unregister_console(uport->cons); + scoped_guard(mutex, &port->mutex) { + port->console = false; + + if (tty_port_initialized(port)) + uart_shutdown(NULL, state); + } + /* * Free the port IO and memory resources, if any. */ diff --git a/drivers/usb/core/driver.c b/drivers/usb/core/driver.c index 7f33fe5ba03b..bf660b15a0f6 100644 --- a/drivers/usb/core/driver.c +++ b/drivers/usb/core/driver.c @@ -2081,5 +2081,4 @@ const struct bus_type usb_bus_type = { .name = "usb", .match = usb_device_match, .uevent = usb_uevent, - .need_parent_lock = true, }; diff --git a/drivers/usb/core/message.c b/drivers/usb/core/message.c index 0cd6dd2b6334..276768f47558 100644 --- a/drivers/usb/core/message.c +++ b/drivers/usb/core/message.c @@ -1922,6 +1922,7 @@ const struct device_type usb_if_device_type = { .name = "usb_interface", .release = usb_release_interface, .uevent = usb_if_uevent, + .need_parent_lock = true, }; static struct usb_interface_assoc_descriptor *find_iad(struct usb_device *dev, diff --git a/drivers/usb/gadget/legacy/inode.c b/drivers/usb/gadget/legacy/inode.c index 67c6ffaf4f72..ce61348d9f2b 100644 --- a/drivers/usb/gadget/legacy/inode.c +++ b/drivers/usb/gadget/legacy/inode.c @@ -1692,6 +1692,8 @@ static int gadgetfs_bind(struct usb_gadget *gadget, dev->gadget = gadget; gadget->ep0->driver_data = dev; + get_dev (dev); + /* preallocate control response and buffer */ dev->req = usb_ep_alloc_request (gadget->ep0, GFP_KERNEL); if (!dev->req) @@ -1706,7 +1708,6 @@ static int gadgetfs_bind(struct usb_gadget *gadget, spin_lock_irq(&dev->lock); dev->state = STATE_DEV_UNCONNECTED; spin_unlock_irq(&dev->lock); - get_dev (dev); return 0; enomem: diff --git a/drivers/vfio/pci/nvgrace-gpu/main.c b/drivers/vfio/pci/nvgrace-gpu/main.c index 963fd8ded20d..208463c21808 100644 --- a/drivers/vfio/pci/nvgrace-gpu/main.c +++ b/drivers/vfio/pci/nvgrace-gpu/main.c @@ -1083,6 +1083,21 @@ static const struct vfio_device_ops nvgrace_gpu_pci_ops = { .detach_ioas = vfio_iommufd_physical_detach_ioas, }; +static int nvgrace_gpu_core_open_device(struct vfio_device *core_vdev) +{ + struct vfio_pci_core_device *vdev = + container_of(core_vdev, struct vfio_pci_core_device, vdev); + int ret; + + ret = vfio_pci_core_enable(vdev); + if (ret) + return ret; + + vfio_pci_core_finish_enable(vdev); + + return 0; +} + static const struct vfio_pci_device_ops nvgrace_gpu_pci_dev_core_ops = { .get_dmabuf_phys = vfio_pci_core_get_dmabuf_phys, }; @@ -1091,7 +1106,7 @@ static const struct vfio_device_ops nvgrace_gpu_pci_core_ops = { .name = "nvgrace-gpu-vfio-pci-core", .init = vfio_pci_core_init_dev, .release = vfio_pci_core_release_dev, - .open_device = nvgrace_gpu_open_device, + .open_device = nvgrace_gpu_core_open_device, .close_device = vfio_pci_core_close_device, .ioctl = vfio_pci_core_ioctl, .get_region_info_caps = vfio_pci_ioctl_get_region_info, diff --git a/drivers/video/backlight/lm3630a_bl.c b/drivers/video/backlight/lm3630a_bl.c index 8f49e59ce374..8ef8163c7fef 100644 --- a/drivers/video/backlight/lm3630a_bl.c +++ b/drivers/video/backlight/lm3630a_bl.c @@ -591,6 +591,7 @@ static void lm3630a_remove(struct i2c_client *client) if (pchip->irq) { free_irq(pchip->irq, pchip); + cancel_delayed_work_sync(&pchip->work); destroy_workqueue(pchip->irqthread); } } diff --git a/drivers/video/backlight/qcom-wled.c b/drivers/video/backlight/qcom-wled.c index 650dd95f06ef..3ce3bade3b8d 100644 --- a/drivers/video/backlight/qcom-wled.c +++ b/drivers/video/backlight/qcom-wled.c @@ -59,7 +59,7 @@ #define WLED3_SINK_REG_CURR_SINK_SHFT 5 /* WLED3 specific per-'string' registers below */ -#define WLED3_SINK_REG_BRIGHT(n) (0x40 + n) +#define WLED3_SINK_REG_BRIGHT(n) (0x40 + (n * 0x2)) #define WLED3_SINK_REG_STR_MOD_EN(n) (0x60 + (n * 0x10)) #define WLED3_SINK_REG_STR_MOD_MASK BIT(7) @@ -1627,7 +1627,6 @@ static int wled_configure_ovp_irq(struct wled *wled, struct platform_device *pdev) { int rc; - u32 val; wled->ovp_irq = platform_get_irq_byname(pdev, "ovp"); if (wled->ovp_irq < 0) { @@ -1643,13 +1642,8 @@ static int wled_configure_ovp_irq(struct wled *wled, return 0; } - rc = regmap_read(wled->regmap, wled->ctrl_addr + - WLED3_CTRL_REG_MOD_EN, &val); - if (rc < 0) - return rc; - - /* Keep OVP irq disabled until module is enabled */ - if (!(val & WLED3_CTRL_REG_MOD_EN_MASK)) + /* Keep the OVP irq disabled until the module is enabled */ + if (!wled->brightness) disable_irq(wled->ovp_irq); return 0; @@ -1665,6 +1659,7 @@ static int wled_probe(struct platform_device *pdev) struct backlight_device *bl; struct wled *wled; struct regmap *regmap; + u32 mod_en; u32 val; int rc; @@ -1680,6 +1675,7 @@ static int wled_probe(struct platform_device *pdev) wled->regmap = regmap; wled->dev = &pdev->dev; + platform_set_drvdata(pdev, wled); wled->version = (uintptr_t)of_device_get_match_data(&pdev->dev); if (!wled->version) { @@ -1734,6 +1730,24 @@ static int wled_probe(struct platform_device *pdev) INIT_DELAYED_WORK(&wled->ovp_work, wled_ovp_work); + val = WLED_DEFAULT_BRIGHTNESS; + of_property_read_u32(pdev->dev.of_node, "default-brightness", &val); + + /* + * The module may already be enabled, either by a bootloader that left + * the backlight lit or by the setup above. Record that, so that the + * first brightness update does not enable an already enabled module, + * and so that the OVP irq is armed from probe rather than from that + * first update. + */ + rc = regmap_read(wled->regmap, wled->ctrl_addr + WLED3_CTRL_REG_MOD_EN, + &mod_en); + if (rc < 0) + return rc; + + if (mod_en & WLED3_CTRL_REG_MOD_EN_MASK) + wled->brightness = val; + rc = wled_configure_short_irq(wled, pdev); if (rc < 0) return rc; @@ -1742,9 +1756,6 @@ static int wled_probe(struct platform_device *pdev) if (rc < 0) return rc; - val = WLED_DEFAULT_BRIGHTNESS; - of_property_read_u32(pdev->dev.of_node, "default-brightness", &val); - memset(&props, 0, sizeof(struct backlight_properties)); props.type = BACKLIGHT_RAW; props.brightness = val; diff --git a/drivers/video/fbdev/atmel_lcdfb.c b/drivers/video/fbdev/atmel_lcdfb.c index 53f0156992e6..9a0cc09004b7 100644 --- a/drivers/video/fbdev/atmel_lcdfb.c +++ b/drivers/video/fbdev/atmel_lcdfb.c @@ -1146,6 +1146,9 @@ static int atmel_lcdfb_probe(struct platform_device *pdev) /* Initialize PWM for contrast or backlight ("off") */ init_contrast(sinfo); + /* Some LCDC recovery operations require process context. */ + INIT_WORK(&sinfo->task, atmel_lcdfb_task); + /* interrupt */ ret = request_irq(sinfo->irq_base, atmel_lcdfb_interrupt, 0, pdev->name, info); if (ret) { @@ -1153,10 +1156,6 @@ static int atmel_lcdfb_probe(struct platform_device *pdev) goto unmap_mmio; } - /* Some operations on the LCDC might sleep and - * require a preemptible task context */ - INIT_WORK(&sinfo->task, atmel_lcdfb_task); - ret = atmel_lcdfb_init_fbinfo(sinfo); if (ret < 0) { dev_err(dev, "init fbinfo failed: %d\n", ret); @@ -1192,8 +1191,8 @@ static int atmel_lcdfb_probe(struct platform_device *pdev) dev_set_drvdata(dev, NULL); fb_dealloc_cmap(&info->cmap); unregister_irqs: - cancel_work_sync(&sinfo->task); free_irq(sinfo->irq_base, info); + cancel_work_sync(&sinfo->task); unmap_mmio: exit_backlight(sinfo); iounmap(sinfo->mmio); @@ -1230,6 +1229,7 @@ static void atmel_lcdfb_remove(struct platform_device *pdev) return; sinfo = info->par; + free_irq(sinfo->irq_base, info); cancel_work_sync(&sinfo->task); exit_backlight(sinfo); atmel_lcdfb_power_control(sinfo, 0); @@ -1238,7 +1238,6 @@ static void atmel_lcdfb_remove(struct platform_device *pdev) clk_put(sinfo->lcdc_clk); clk_put(sinfo->bus_clk); fb_dealloc_cmap(&info->cmap); - free_irq(sinfo->irq_base, info); iounmap(sinfo->mmio); release_mem_region(info->fix.mmio_start, info->fix.mmio_len); if (platform_get_resource(pdev, IORESOURCE_MEM, 1)) { diff --git a/drivers/video/fbdev/udlfb.c b/drivers/video/fbdev/udlfb.c index e78d6f95c9c5..92b352bc7e96 100644 --- a/drivers/video/fbdev/udlfb.c +++ b/drivers/video/fbdev/udlfb.c @@ -1293,6 +1293,11 @@ static int dlfb_setup_modes(struct dlfb_data *dlfb, goto error; } + if (dlfb->fb_count) { + result = -EINVAL; + goto error; + } + fb_destroy_modelist(&info->modelist); memset(&info->monspecs, 0, sizeof(info->monspecs)); diff --git a/fs/afs/rxrpc.c b/fs/afs/rxrpc.c index d82916657a3d..768b26820dea 100644 --- a/fs/afs/rxrpc.c +++ b/fs/afs/rxrpc.c @@ -412,8 +412,7 @@ void afs_make_call(struct afs_call *call, gfp_t gfp) msg.msg_controllen = 0; msg.msg_flags = MSG_WAITALL | (call->write_iter ? MSG_MORE : 0); - ret = rxrpc_kernel_send_data(call->net->socket, rxcall, - &msg, call->request_size, + ret = rxrpc_kernel_send_data(call->net->socket, rxcall, &msg, afs_notify_end_request_tx); if (ret < 0) goto error_do_abort; @@ -425,7 +424,6 @@ void afs_make_call(struct afs_call *call, gfp_t gfp) ret = rxrpc_kernel_send_data(call->net->socket, call->rxcall, &msg, - iov_iter_count(&msg.msg_iter), afs_notify_end_request_tx); *call->write_iter = msg.msg_iter; @@ -443,10 +441,8 @@ void afs_make_call(struct afs_call *call, gfp_t gfp) return; error_do_abort: - if (ret != -ECONNABORTED) - rxrpc_kernel_abort_call(call->net->socket, rxcall, - RX_USER_ABORT, ret, - afs_abort_send_data_error); + rxrpc_kernel_abort_call(call->net->socket, rxcall, + RX_USER_ABORT, ret, afs_abort_send_data_error); if (call->async) { afs_see_call(call, afs_call_trace_async_abort); return; @@ -545,7 +541,7 @@ void afs_deliver_to_call(struct afs_call *call) &call->service_id); trace_afs_receive_data(call, &call->def_iter, false, ret); - if (ret == -EINPROGRESS || ret == -EAGAIN) + if (ret == -EAGAIN || ret == 2) return; if (ret < 0 || ret == 1) { if (ret == 1) @@ -859,6 +855,7 @@ void afs_send_empty_reply(struct afs_call *call) { struct afs_net *net = call->net; struct msghdr msg; + int ret; _enter(""); @@ -871,21 +868,13 @@ void afs_send_empty_reply(struct afs_call *call) msg.msg_controllen = 0; msg.msg_flags = 0; - switch (rxrpc_kernel_send_data(net->socket, call->rxcall, &msg, 0, - afs_notify_end_reply_tx)) { - case 0: - _leave(" [replied]"); - return; - - case -ENOMEM: - _debug("oom"); + ret = rxrpc_kernel_send_data(net->socket, call->rxcall, &msg, + afs_notify_end_reply_tx); + if (ret < 0) { rxrpc_kernel_abort_call(net->socket, call->rxcall, - RXGEN_SS_MARSHAL, -ENOMEM, - afs_abort_oom); - fallthrough; - default: - _leave(" [error]"); - return; + RXGEN_SS_MARSHAL, ret, + afs_abort_send_error); + afs_set_call_complete(call, ret, RXGEN_SS_MARSHAL); } } @@ -897,7 +886,7 @@ void afs_send_simple_reply(struct afs_call *call, const void *buf, size_t len) struct afs_net *net = call->net; struct msghdr msg; struct kvec iov[1]; - int n; + int ret; _enter(""); @@ -912,21 +901,15 @@ void afs_send_simple_reply(struct afs_call *call, const void *buf, size_t len) msg.msg_controllen = 0; msg.msg_flags = 0; - n = rxrpc_kernel_send_data(net->socket, call->rxcall, &msg, len, - afs_notify_end_reply_tx); - if (n >= 0) { - /* Success */ - _leave(" [replied]"); - return; - } - - if (n == -ENOMEM) { - _debug("oom"); + ret = rxrpc_kernel_send_data(net->socket, call->rxcall, &msg, + afs_notify_end_reply_tx); + if (ret < 0) { rxrpc_kernel_abort_call(net->socket, call->rxcall, - RXGEN_SS_MARSHAL, -ENOMEM, - afs_abort_oom); + RXGEN_SS_MARSHAL, ret, + afs_abort_send_error); + afs_set_call_complete(call, ret, RXGEN_SS_MARSHAL); + _leave(" [error]"); } - _leave(" [error]"); } /* @@ -951,7 +934,7 @@ int afs_extract_data(struct afs_call *call, bool want_more) return ret; state = READ_ONCE(call->state); - if (ret == 1) { + if (ret == 1 || ret == 2) { switch (state) { case AFS_CALL_CL_AWAIT_REPLY: afs_set_call_state(call, state, AFS_CALL_CL_PROC_REPLY); diff --git a/fs/btrfs/verity.c b/fs/btrfs/verity.c index 600337a84fbe..59ee70bd2374 100644 --- a/fs/btrfs/verity.c +++ b/fs/btrfs/verity.c @@ -489,8 +489,6 @@ static int rollback_verity(struct btrfs_inode *inode) inode->vfs_inode.i_ino); goto out; } - inode->ro_flags &= ~BTRFS_INODE_RO_VERITY; - btrfs_sync_inode_flags_to_i_flags(inode); ret = btrfs_update_inode(trans, inode); if (unlikely(ret)) { btrfs_abort_transaction(trans, ret); @@ -555,16 +553,19 @@ static int finish_verity(struct btrfs_inode *inode, const void *desc, if (IS_ERR(trans)) return PTR_ERR(trans); inode->ro_flags |= BTRFS_INODE_RO_VERITY; - btrfs_sync_inode_flags_to_i_flags(inode); ret = btrfs_update_inode(trans, inode); if (ret) goto end_trans; ret = del_orphan(trans, inode); if (ret) goto end_trans; + /* Now that rollback is no longer possible, set S_VERITY. */ + btrfs_sync_inode_flags_to_i_flags(inode); clear_bit(BTRFS_INODE_VERITY_IN_PROGRESS, &inode->runtime_flags); btrfs_set_fs_compat_ro(root->fs_info, VERITY); end_trans: + if (ret) + inode->ro_flags &= ~BTRFS_INODE_RO_VERITY; btrfs_end_transaction(trans); return ret; diff --git a/fs/ceph/dir.c b/fs/ceph/dir.c index 2e5c0ccb1b34..d7d48d3c3baf 100644 --- a/fs/ceph/dir.c +++ b/fs/ceph/dir.c @@ -466,6 +466,10 @@ static int ceph_readdir(struct file *file, struct dir_context *ctx) (int)req->r_reply_info.hash_order); rinfo = &req->r_reply_info; + if (WARN_ON_ONCE(!rinfo->dir_dir)) { + ceph_mdsc_put_request(req); + return -EIO; + } if (le32_to_cpu(rinfo->dir_dir->frag) != frag) { frag = le32_to_cpu(rinfo->dir_dir->frag); if (!rinfo->hash_order) { diff --git a/fs/exec.c b/fs/exec.c index a5269b5e00df..e738144cee6b 100644 --- a/fs/exec.c +++ b/fs/exec.c @@ -1160,6 +1160,13 @@ int begin_new_exec(struct linux_binprm * bprm) */ bprm->point_of_no_return = true; + /* + * Cancel any io_uring activity across execve. This runs task work + * that may still create an io-wq worker, so do it while de_thread() + * can still zap it. + */ + io_uring_task_cancel(); + /* Make this the only thread in the thread group */ retval = de_thread(me); if (retval) @@ -1176,10 +1183,6 @@ int begin_new_exec(struct linux_binprm * bprm) /* see the comment in check_unsafe_exec() */ current->fs->in_exec = 0; - /* - * Cancel any io_uring activity across execve - */ - io_uring_task_cancel(); /* Ensure the files table is not shared. */ retval = unshare_files(); diff --git a/fs/ext4/verity.c b/fs/ext4/verity.c index ca61da53f313..dec1cac1b58e 100644 --- a/fs/ext4/verity.c +++ b/fs/ext4/verity.c @@ -242,17 +242,19 @@ static int ext4_end_enable_verity(struct file *filp, const void *desc, goto stop_and_cleanup; ext4_set_inode_flag(inode, EXT4_INODE_VERITY); - ext4_set_inode_flags(inode, false); err = ext4_mark_iloc_dirty(handle, inode, &iloc); if (err) goto stop_and_cleanup; ext4_journal_stop(handle); + /* Now that rollback is no longer possible, set S_VERITY. */ + ext4_set_inode_flags(inode, false); ext4_clear_inode_state(inode, EXT4_STATE_VERITY_IN_PROGRESS); return 0; stop_and_cleanup: + ext4_clear_inode_flag(inode, EXT4_INODE_VERITY); ext4_journal_stop(handle); cleanup: /* diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c index 4fec31fc0b84..4148e4488d77 100644 --- a/fs/fuse/dev.c +++ b/fs/fuse/dev.c @@ -1061,7 +1061,8 @@ static int fuse_copy_fill(struct fuse_copy_state *cs) err = iov_iter_get_pages2(cs->iter, &page, PAGE_SIZE, 1, &off); if (err < 0) return err; - BUG_ON(!err); + if (!err) + return -EIO; cs->len = err; cs->offset = off; cs->pg = page; diff --git a/fs/iomap/ioend.c b/fs/iomap/ioend.c index 7bbbb417f915..32ae292a84cb 100644 --- a/fs/iomap/ioend.c +++ b/fs/iomap/ioend.c @@ -246,8 +246,16 @@ ssize_t iomap_add_to_ioend(struct iomap_writepage_ctx *wpc, struct folio *folio, new_ioend: if (ioend) { error = wpc->ops->writeback_submit(wpc, 0); - if (error) + if (error) { + /* + * ->writeback_submit() completed the ioend + * with an error, so drop the stale context; + * iomap_writepages() would otherwise submit + * it a second time. + */ + wpc->wb_ctx = NULL; return error; + } } wpc->wb_ctx = ioend = iomap_alloc_ioend(wpc, pos, ioend_flags); } diff --git a/fs/kernfs/mount.c b/fs/kernfs/mount.c index a57399021c8b..a0be784bfb06 100644 --- a/fs/kernfs/mount.c +++ b/fs/kernfs/mount.c @@ -124,22 +124,32 @@ static struct dentry *__kernfs_fh_to_dentry(struct super_block *sb, return NULL; } - kn = kernfs_find_and_get_node_by_id(info->root, id); - if (!kn) - return ERR_PTR(-ESTALE); + /* + * Hold kernfs_rwsem across the lookup as well as kernfs_get_inode(). + * __kernfs_remove() deactivates the subtree and clears i_nlink on its + * inodes under the write lock, so under the read lock either + * kernfs_find_and_get_node_by_id() refuses the node, or the inode is + * in the inode hash before the ilookup() pass goes looking for it. + */ + scoped_guard(rwsem_read, &info->root->kernfs_rwsem) { + kn = kernfs_find_and_get_node_by_id(info->root, id); + if (!kn) + return ERR_PTR(-ESTALE); - if (get_parent) { - struct kernfs_node *parent; + if (get_parent) { + struct kernfs_node *parent; - parent = kernfs_get_parent(kn); + parent = kernfs_get_parent(kn); + kernfs_put(kn); + kn = parent; + if (!kn) + return ERR_PTR(-ESTALE); + } + + inode = kernfs_get_inode(sb, kn); kernfs_put(kn); - kn = parent; - if (!kn) - return ERR_PTR(-ESTALE); } - inode = kernfs_get_inode(sb, kn); - kernfs_put(kn); return d_obtain_alias(inode); } diff --git a/fs/lockd/svclock.c b/fs/lockd/svclock.c index e628b5d35507..8d83283d3e21 100644 --- a/fs/lockd/svclock.c +++ b/fs/lockd/svclock.c @@ -1023,6 +1023,7 @@ nlmsvc_retry_blocked(struct svc_rqst *rqstp) timeout = block->b_when - jiffies; break; } + kref_get(&block->b_count); spin_unlock(&nlm_blocked_lock); dprintk("nlmsvc_retry_blocked(%p, when=%ld)\n", @@ -1033,6 +1034,7 @@ nlmsvc_retry_blocked(struct svc_rqst *rqstp) retry_deferred_block(block); } else nlmsvc_grant_blocked(block); + nlmsvc_release_block(block); spin_lock(&nlm_blocked_lock); } spin_unlock(&nlm_blocked_lock); diff --git a/fs/nfs/blocklayout/blocklayout.h b/fs/nfs/blocklayout/blocklayout.h index 6da40ca19570..e242b1d5b4bd 100644 --- a/fs/nfs/blocklayout/blocklayout.h +++ b/fs/nfs/blocklayout/blocklayout.h @@ -161,11 +161,6 @@ BLK_LSEG2EXT(struct pnfs_layout_segment *lseg) return BLK_LO2EXT(lseg->pls_layout); } -struct bl_pipe_msg { - struct rpc_pipe_msg msg; - wait_queue_head_t *bl_wq; -}; - struct bl_msg_hdr { u8 type; u16 totallen; /* length of entire message, including hdr itself */ diff --git a/fs/nfs/blocklayout/rpc_pipefs.c b/fs/nfs/blocklayout/rpc_pipefs.c index d526f5ba7887..f0004a8249e7 100644 --- a/fs/nfs/blocklayout/rpc_pipefs.c +++ b/fs/nfs/blocklayout/rpc_pipefs.c @@ -55,17 +55,15 @@ bl_resolve_deviceid(struct nfs_server *server, struct pnfs_block_volume *b, struct net *net = server->nfs_client->cl_net; struct nfs_net *nn = net_generic(net, nfs_net_id); struct bl_dev_msg *reply = &nn->bl_mount_reply; - struct bl_pipe_msg bl_pipe_msg; - struct rpc_pipe_msg *msg = &bl_pipe_msg.msg; + struct rpc_pipe_msg *msg = &nn->bl_pipe_msg; + struct rpc_pipe *pipe = nn->bl_device_pipe; struct bl_msg_hdr *bl_msg; - DECLARE_WAITQUEUE(wq, current); dev_t dev = 0; int rc; dprintk("%s CREATING PIPEFS MESSAGE\n", __func__); mutex_lock(&nn->bl_mutex); - bl_pipe_msg.bl_wq = &nn->bl_wq; b->simple.len += 4; /* single volume */ if (b->simple.len > PAGE_SIZE) @@ -83,17 +81,15 @@ bl_resolve_deviceid(struct nfs_server *server, struct pnfs_block_volume *b, nfs4_encode_simple(msg->data + sizeof(*bl_msg), b); dprintk("%s CALLING USERSPACE DAEMON\n", __func__); - add_wait_queue(&nn->bl_wq, &wq); - rc = rpc_queue_upcall(nn->bl_device_pipe, msg); - if (rc < 0) { - remove_wait_queue(&nn->bl_wq, &wq); + reinit_completion(&nn->bl_done); + rc = rpc_queue_upcall(pipe, msg); + if (rc < 0) goto out_free_data; - } - set_current_state(TASK_UNINTERRUPTIBLE); - schedule(); - remove_wait_queue(&nn->bl_wq, &wq); + wait_for_completion(&nn->bl_done); + if (msg->errno < 0) + goto out_free_data; if (reply->status != BL_DEVICE_REQUEST_PROC) { printk(KERN_WARNING "%s failed to decode device: %d\n", __func__, reply->status); @@ -113,32 +109,73 @@ static ssize_t bl_pipe_downcall(struct file *filp, const char __user *src, { struct nfs_net *nn = net_generic(file_inode(filp)->i_sb->s_fs_info, nfs_net_id); + struct rpc_pipe *pipe = nn->bl_device_pipe; + struct bl_dev_msg reply; + bool accepted; - if (mlen != sizeof (struct bl_dev_msg)) + if (mlen != sizeof(reply)) return -EINVAL; - - if (copy_from_user(&nn->bl_mount_reply, src, mlen) != 0) + if (copy_from_user(&reply, src, mlen) != 0) return -EFAULT; - wake_up(&nn->bl_wq); - + /* + * Accept a reply only after blkmapd has read the whole upcall. + * Completion retires the message, here and in + * bl_pipe_destroy_msg(), so a later reply finds nothing in + * flight. + */ + spin_lock(&pipe->lock); + accepted = rpc_msg_is_inflight(&nn->bl_pipe_msg); + if (accepted) { + nn->bl_pipe_msg.copied = 0; + nn->bl_mount_reply = reply; + complete(&nn->bl_done); + } + spin_unlock(&pipe->lock); + if (!accepted) + return -EINVAL; return mlen; } static void bl_pipe_destroy_msg(struct rpc_pipe_msg *msg) { - struct bl_pipe_msg *bl_pipe_msg = - container_of(msg, struct bl_pipe_msg, msg); + struct nfs_net *nn = container_of(msg, struct nfs_net, bl_pipe_msg); + struct rpc_pipe *pipe = nn->bl_device_pipe; if (msg->errno >= 0) return; - wake_up(bl_pipe_msg->bl_wq); + + spin_lock(&pipe->lock); + msg->copied = 0; + spin_unlock(&pipe->lock); + complete(&nn->bl_done); +} + +/* + * An upcall blkmapd has consumed is off every pipe list, so the + * purges in rpc_pipe_release() and rpc_close_pipes() cannot reach it. + * No downcall can arrive once the pipe is closed; release its waiter + * here. + */ +static void bl_release_pipe(struct inode *inode) +{ + struct nfs_net *nn = net_generic(inode->i_sb->s_fs_info, nfs_net_id); + struct rpc_pipe *pipe = nn->bl_device_pipe; + + spin_lock(&pipe->lock); + if (rpc_msg_is_inflight(&nn->bl_pipe_msg)) { + nn->bl_pipe_msg.copied = 0; + nn->bl_pipe_msg.errno = -EPIPE; + complete(&nn->bl_done); + } + spin_unlock(&pipe->lock); } static const struct rpc_pipe_ops bl_upcall_ops = { .upcall = rpc_pipe_generic_upcall, .downcall = bl_pipe_downcall, .destroy_msg = bl_pipe_destroy_msg, + .release_pipe = bl_release_pipe, }; static int nfs4blocklayout_register_sb(struct super_block *sb, @@ -221,7 +258,7 @@ static int nfs4blocklayout_net_init(struct net *net) int err; mutex_init(&nn->bl_mutex); - init_waitqueue_head(&nn->bl_wq); + init_completion(&nn->bl_done); nn->bl_device_pipe = rpc_mkpipe_data(&bl_upcall_ops, 0); if (IS_ERR(nn->bl_device_pipe)) return PTR_ERR(nn->bl_device_pipe); diff --git a/fs/nfs/callback_xdr.c b/fs/nfs/callback_xdr.c index 88e1af0fda01..5a1f05469268 100644 --- a/fs/nfs/callback_xdr.c +++ b/fs/nfs/callback_xdr.c @@ -271,6 +271,13 @@ __be32 decode_devicenotify_args(struct svc_rqst *rqstp, if (n == 0) goto out; + /* sanity check the count against the remaining stream */ + if (n > xdr_stream_remaining(xdr) / + ((4 * sizeof(uint32_t)) + NFS4_DEVICEID4_SIZE)) { + status = htonl(NFS4ERR_BADXDR); + goto out; + } + args->devs = kmalloc_objs(*args->devs, n); if (!args->devs) { status = htonl(NFS4ERR_DELAY); @@ -312,7 +319,7 @@ __be32 decode_devicenotify_args(struct svc_rqst *rqstp, memcpy(dev->cbd_dev_id.data, p, NFS4_DEVICEID4_SIZE); p += XDR_QUADLEN(NFS4_DEVICEID4_SIZE); - if (dev->cbd_layout_type == NOTIFY_DEVICEID4_CHANGE) { + if (dev->cbd_notify_type == NOTIFY_DEVICEID4_CHANGE) { p = xdr_inline_decode(xdr, sizeof(uint32_t)); if (unlikely(p == NULL)) { status = htonl(NFS4ERR_BADXDR); diff --git a/fs/nfs/delegation.c b/fs/nfs/delegation.c index ab3e441464a6..ad774316f716 100644 --- a/fs/nfs/delegation.c +++ b/fs/nfs/delegation.c @@ -934,8 +934,7 @@ nfs_mark_return_if_closed_delegation(struct nfs_server *server, { struct inode *inode; - if (!list_empty_careful(&server->delegations_return) || - test_bit(NFS_DELEGATION_RETURN_IF_CLOSED, &delegation->flags)) + if (test_bit(NFS_DELEGATION_RETURN_IF_CLOSED, &delegation->flags)) return; spin_lock(&delegation->lock); inode = delegation->inode; diff --git a/fs/nfs/dir.c b/fs/nfs/dir.c index 49394123bd09..a26c459ffb04 100644 --- a/fs/nfs/dir.c +++ b/fs/nfs/dir.c @@ -2490,6 +2490,8 @@ struct dentry *nfs_mkdir(struct mnt_idmap *idmap, struct inode *dir, trace_nfs_mkdir_enter(dir, dentry); ret = NFS_PROTO(dir)->mkdir(dir, dentry, &attr); trace_nfs_mkdir_exit(dir, dentry, PTR_ERR_OR_ZERO(ret)); + if (IS_ERR(ret)) + d_drop(dentry); return ret; } EXPORT_SYMBOL_GPL(nfs_mkdir); diff --git a/fs/nfs/filelayout/filelayout.c b/fs/nfs/filelayout/filelayout.c index 72e20b56fbc7..0d53277f972e 100644 --- a/fs/nfs/filelayout/filelayout.c +++ b/fs/nfs/filelayout/filelayout.c @@ -796,7 +796,7 @@ filelayout_pg_test(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, unsigned int size; u64 p_stripe, r_stripe; u32 stripe_offset; - u64 segment_offset = pgio->pg_lseg->pls_range.offset; + u64 pattern_offset = FILELAYOUT_LSEG(pgio->pg_lseg)->pattern_offset; u32 stripe_unit = FILELAYOUT_LSEG(pgio->pg_lseg)->stripe_unit; /* calls nfs_generic_pg_test */ @@ -808,8 +808,8 @@ filelayout_pg_test(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, /* see if req and prev are in the same stripe */ if (prev) { - p_stripe = (u64)req_offset(prev) - segment_offset; - r_stripe = (u64)req_offset(req) - segment_offset; + p_stripe = (u64)req_offset(prev) - pattern_offset; + r_stripe = (u64)req_offset(req) - pattern_offset; do_div(p_stripe, stripe_unit); do_div(r_stripe, stripe_unit); @@ -818,7 +818,7 @@ filelayout_pg_test(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, } /* calculate remaining bytes in the current stripe */ - div_u64_rem((u64)req_offset(req) - segment_offset, + div_u64_rem((u64)req_offset(req) - pattern_offset, stripe_unit, &stripe_offset); WARN_ON_ONCE(stripe_offset > stripe_unit); diff --git a/fs/nfs/filelayout/filelayoutdev.c b/fs/nfs/filelayout/filelayoutdev.c index 88bc79ec3459..57e0654cd98c 100644 --- a/fs/nfs/filelayout/filelayoutdev.c +++ b/fs/nfs/filelayout/filelayoutdev.c @@ -177,27 +177,14 @@ nfs4_fl_alloc_deviceid_node(struct nfs_server *server, struct pnfs_device *pdev, trace_fl_getdevinfo(server, &pdev->dev_id, dsaddr->ds_list[i]->ds_remotestr); /* If DS was already in cache, free ds addrs */ - while (!list_empty(&dsaddrs)) { - da = list_first_entry(&dsaddrs, - struct nfs4_pnfs_ds_addr, - da_node); - list_del_init(&da->da_node); - kfree(da->da_remotestr); - kfree(da); - } + nfs4_pnfs_ds_addr_list_free(&dsaddrs); } folio_put(scratch); return dsaddr; out_err_drain_dsaddrs: - while (!list_empty(&dsaddrs)) { - da = list_first_entry(&dsaddrs, struct nfs4_pnfs_ds_addr, - da_node); - list_del_init(&da->da_node); - kfree(da->da_remotestr); - kfree(da); - } + nfs4_pnfs_ds_addr_list_free(&dsaddrs); out_err_free_deviceid: nfs4_fl_free_deviceid(dsaddr); /* stripe_indicies was part of dsaddr */ diff --git a/fs/nfs/flexfilelayout/flexfilelayout.c b/fs/nfs/flexfilelayout/flexfilelayout.c index 7fe8b91fa47c..15980d9dad8e 100644 --- a/fs/nfs/flexfilelayout/flexfilelayout.c +++ b/fs/nfs/flexfilelayout/flexfilelayout.c @@ -878,7 +878,7 @@ ff_layout_mark_ds_reachable(struct pnfs_layout_segment *lseg, u32 idx, u32 dss_i static struct nfs4_pnfs_ds * ff_layout_choose_ds_for_read(struct pnfs_layout_segment *lseg, u32 start_idx, u32 *best_idx, - u32 offset, u32 *dss_id, + u64 offset, u32 *dss_id, bool check_device) { struct nfs4_ff_layout_segment *fls = FF_LAYOUT_LSEG(lseg); @@ -914,7 +914,7 @@ ff_layout_choose_ds_for_read(struct pnfs_layout_segment *lseg, static struct nfs4_pnfs_ds * ff_layout_choose_any_ds_for_read(struct pnfs_layout_segment *lseg, u32 start_idx, u32 *best_idx, - u32 offset, u32 *dss_id) + u64 offset, u32 *dss_id) { return ff_layout_choose_ds_for_read(lseg, start_idx, best_idx, offset, dss_id, false); @@ -923,7 +923,7 @@ ff_layout_choose_any_ds_for_read(struct pnfs_layout_segment *lseg, static struct nfs4_pnfs_ds * ff_layout_choose_valid_ds_for_read(struct pnfs_layout_segment *lseg, u32 start_idx, u32 *best_idx, - u32 offset, u32 *dss_id) + u64 offset, u32 *dss_id) { return ff_layout_choose_ds_for_read(lseg, start_idx, best_idx, offset, dss_id, true); @@ -932,7 +932,7 @@ ff_layout_choose_valid_ds_for_read(struct pnfs_layout_segment *lseg, static struct nfs4_pnfs_ds * ff_layout_choose_best_ds_for_read(struct pnfs_layout_segment *lseg, u32 start_idx, u32 *best_idx, - u32 offset, u32 *dss_id) + u64 offset, u32 *dss_id) { struct nfs4_pnfs_ds *ds; @@ -947,7 +947,7 @@ ff_layout_choose_best_ds_for_read(struct pnfs_layout_segment *lseg, static struct nfs4_pnfs_ds * ff_layout_get_ds_for_read(struct nfs_pageio_descriptor *pgio, u32 *best_idx, - u32 offset, + u64 offset, u32 *dss_id) { struct pnfs_layout_segment *lseg = pgio->pg_lseg; @@ -995,9 +995,8 @@ ff_layout_pg_test(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, { unsigned int size; u64 p_stripe, r_stripe; - u32 stripe_offset; - u64 segment_offset = pgio->pg_lseg->pls_range.offset; - u32 stripe_unit = FF_LAYOUT_LSEG(pgio->pg_lseg)->stripe_unit; + u64 stripe_offset; + u64 stripe_unit = FF_LAYOUT_LSEG(pgio->pg_lseg)->stripe_unit; /* calls nfs_generic_pg_test */ size = pnfs_generic_pg_test(pgio, prev, req); @@ -1008,23 +1007,23 @@ ff_layout_pg_test(struct nfs_pageio_descriptor *pgio, struct nfs_page *prev, /* see if req and prev are in the same stripe */ if (prev) { - p_stripe = (u64)req_offset(prev) - segment_offset; - r_stripe = (u64)req_offset(req) - segment_offset; - do_div(p_stripe, stripe_unit); - do_div(r_stripe, stripe_unit); + p_stripe = (u64)req_offset(prev); + r_stripe = (u64)req_offset(req); + p_stripe = div64_u64(p_stripe, stripe_unit); + r_stripe = div64_u64(r_stripe, stripe_unit); if (p_stripe != r_stripe) return 0; } /* calculate remaining bytes in the current stripe */ - div_u64_rem((u64)req_offset(req) - segment_offset, + div64_u64_rem((u64)req_offset(req), stripe_unit, &stripe_offset); WARN_ON_ONCE(stripe_offset > stripe_unit); if (stripe_offset >= stripe_unit) return 0; - return min(stripe_unit - (unsigned int)stripe_offset, size); + return min_t(u64, stripe_unit - stripe_offset, size); } static void @@ -1597,6 +1596,15 @@ static void ff_layout_io_track_ds_error(struct pnfs_layout_segment *lseg, dprintk("%s: err %d op %d status %u\n", __func__, err, opnum, status); } +static bool ff_layout_io_state_lost(const struct nfs_pgio_header *hdr) +{ + struct nfs_open_context *ctx = hdr->args.context; + + return test_bit(NFS_CONTEXT_BAD, &ctx->flags) || + !nfs4_valid_open_stateid(ctx->state) || + nfs4_lock_context_lost(ctx->state, hdr->args.lock_context); +} + /* NFS_PROTO call done callback routines */ static int ff_layout_read_done_cb(struct rpc_task *task, struct nfs_pgio_header *hdr) @@ -1608,6 +1616,12 @@ static int ff_layout_read_done_cb(struct rpc_task *task, hdr->args.offset); int err; + if (task->tk_status < 0 && !RPC_WAS_SENT(task) && + ff_layout_io_state_lost(hdr)) { + task->tk_status = -EIO; + return 0; + } + if (task->tk_status < 0) { ff_layout_io_track_ds_error(hdr->lseg, hdr->pgio_mirror_idx, dss_id, @@ -1724,7 +1738,7 @@ static void ff_layout_read_record_layoutstats_done(struct rpc_task *task, static int ff_layout_read_prepare_common(struct rpc_task *task, struct nfs_pgio_header *hdr) { - if (unlikely(test_bit(NFS_CONTEXT_BAD, &hdr->args.context->flags))) { + if (unlikely(ff_layout_io_state_lost(hdr))) { rpc_exit(task, -EIO); return -EIO; } @@ -1813,6 +1827,12 @@ static int ff_layout_write_done_cb(struct rpc_task *task, loff_t end_offs = 0; int err; + if (task->tk_status < 0 && !RPC_WAS_SENT(task) && + ff_layout_io_state_lost(hdr)) { + task->tk_status = -EIO; + return 0; + } + if (task->tk_status < 0) { ff_layout_io_track_ds_error(hdr->lseg, hdr->pgio_mirror_idx, dss_id, @@ -1950,7 +1970,7 @@ static void ff_layout_write_record_layoutstats_done(struct rpc_task *task, static int ff_layout_write_prepare_common(struct rpc_task *task, struct nfs_pgio_header *hdr) { - if (unlikely(test_bit(NFS_CONTEXT_BAD, &hdr->args.context->flags))) { + if (unlikely(ff_layout_io_state_lost(hdr))) { rpc_exit(task, -EIO); return -EIO; } diff --git a/fs/nfs/flexfilelayout/flexfilelayout.h b/fs/nfs/flexfilelayout/flexfilelayout.h index a5bd00f69e82..ec69cd1c3ae9 100644 --- a/fs/nfs/flexfilelayout/flexfilelayout.h +++ b/fs/nfs/flexfilelayout/flexfilelayout.h @@ -220,7 +220,7 @@ nfs4_ff_layout_calc_dss_id(const u64 stripe_unit, const u32 dss_count, const lof if (dss_count == 1 || stripe_unit == 0) return 0; - do_div(tmp, stripe_unit); + tmp = div64_u64(tmp, stripe_unit); return do_div(tmp, dss_count); } diff --git a/fs/nfs/flexfilelayout/flexfilelayoutdev.c b/fs/nfs/flexfilelayout/flexfilelayoutdev.c index 5c0216bd5fce..6cd3859d4a24 100644 --- a/fs/nfs/flexfilelayout/flexfilelayoutdev.c +++ b/fs/nfs/flexfilelayout/flexfilelayoutdev.c @@ -159,26 +159,13 @@ nfs4_ff_alloc_deviceid_node(struct nfs_server *server, struct pnfs_device *pdev, goto out_err_drain_dsaddrs; /* If DS was already in cache, free ds addrs */ - while (!list_empty(&dsaddrs)) { - da = list_first_entry(&dsaddrs, - struct nfs4_pnfs_ds_addr, - da_node); - list_del_init(&da->da_node); - kfree(da->da_remotestr); - kfree(da); - } + nfs4_pnfs_ds_addr_list_free(&dsaddrs); folio_put(scratch); return new_ds; out_err_drain_dsaddrs: - while (!list_empty(&dsaddrs)) { - da = list_first_entry(&dsaddrs, struct nfs4_pnfs_ds_addr, - da_node); - list_del_init(&da->da_node); - kfree(da->da_remotestr); - kfree(da); - } + nfs4_pnfs_ds_addr_list_free(&dsaddrs); kfree(ds_versions); out_scratch: diff --git a/fs/nfs/localio.c b/fs/nfs/localio.c index f42b6112a613..cdada6cbb13b 100644 --- a/fs/nfs/localio.c +++ b/fs/nfs/localio.c @@ -446,7 +446,7 @@ nfs_local_iters_setup_dio(struct nfs_local_kiocb *iocb, int rw, if (unlikely(!iocb->iter_is_dio_aligned[n_iters])) { trace_nfs_local_dio_misaligned(iocb->hdr->inode, - local_dio->start_len, local_dio->middle_len, local_dio); + local_dio->middle_offset, local_dio->middle_len, local_dio); return 0; /* no DIO-aligned IO possible */ } iocb->end_iter_index = n_iters; @@ -674,6 +674,8 @@ static void nfs_local_call_read(struct work_struct *work) n_iters = atomic_read(&iocb->n_iters); for (int i = 0; i < n_iters ; i++) { + size_t expected; + if (iocb->iter_is_dio_aligned[i]) { iocb->kiocb.ki_flags |= IOCB_DIRECT; /* Only use AIO completion if DIO-aligned segment is last */ @@ -684,6 +686,8 @@ static void nfs_local_call_read(struct work_struct *work) } else iocb->kiocb.ki_flags &= ~IOCB_DIRECT; + /* read_iter() advances the iterator: measure it beforehand */ + expected = iov_iter_count(&iocb->iters[i]); scoped_with_creds(filp->f_cred) status = filp->f_op->read_iter(&iocb->kiocb, &iocb->iters[i]); @@ -691,7 +695,7 @@ static void nfs_local_call_read(struct work_struct *work) continue; /* Break on completion, errors, or short reads */ if (nfs_local_pgio_done(iocb, status) || status < 0 || - (size_t)status < iov_iter_count(&iocb->iters[i])) { + (size_t)status < expected) { nfs_local_read_iocb_done(iocb); break; } @@ -890,7 +894,7 @@ static void nfs_local_call_write(struct work_struct *work) file_start_write(filp); n_iters = atomic_read(&iocb->n_iters); for (int i = 0; i < n_iters ; i++) { - size_t icount; + size_t expected; if (iocb->iter_is_dio_aligned[i]) { iocb->kiocb.ki_flags |= IOCB_DIRECT; @@ -902,16 +906,17 @@ static void nfs_local_call_write(struct work_struct *work) } else iocb->kiocb.ki_flags &= ~IOCB_DIRECT; + /* write_iter() advances the iterator: measure it beforehand */ + expected = iov_iter_count(&iocb->iters[i]); scoped_with_creds(filp->f_cred) status = filp->f_op->write_iter(&iocb->kiocb, &iocb->iters[i]); if (status == -EIOCBQUEUED) continue; /* Break on completion, errors, or short writes */ - icount = iov_iter_count(&iocb->iters[i]); if (nfs_local_pgio_done(iocb, status) || status < 0 || - (size_t)status < icount) { - if ((size_t)status < icount) { + (size_t)status < expected) { + if ((size_t)status < expected) { struct nfs_lock_context *ctx = iocb->hdr->req->wb_lock_context; @@ -931,6 +936,7 @@ static void nfs_local_do_write(struct nfs_local_kiocb *iocb, const struct rpc_call_ops *call_ops) { struct nfs_pgio_header *hdr = iocb->hdr; + enum nfs3_stable_how committed = hdr->args.stable; dprintk("%s: vfs_write count=%u pos=%llu %s\n", __func__, hdr->args.count, hdr->args.offset, @@ -949,9 +955,20 @@ static void nfs_local_do_write(struct nfs_local_kiocb *iocb, iocb->kiocb.ki_flags |= IOCB_DSYNC|IOCB_SYNC; } + /* + * Report the stability the write will actually have. A DIO WRITE + * is persisted before it completes whatever was asked for, see + * nfs_local_iters_init(), and a caller told so has no reason to + * COMMIT data that is already on stable storage. + */ + if (iocb->kiocb.ki_flags & IOCB_SYNC) + committed = NFS_FILE_SYNC; + else if (iocb->kiocb.ki_flags & IOCB_DSYNC) + committed = NFS_DATA_SYNC; + nfs_local_pgio_init(hdr, call_ops); - nfs_set_local_verifier(hdr->inode, hdr->res.verf, hdr->args.stable); + nfs_set_local_verifier(hdr->inode, hdr->res.verf, committed); INIT_WORK(&iocb->work, nfs_local_call_write); if (nfs_local_defer_io()) diff --git a/fs/nfs/netns.h b/fs/nfs/netns.h index 36658579100d..e1decff366d4 100644 --- a/fs/nfs/netns.h +++ b/fs/nfs/netns.h @@ -10,6 +10,8 @@ #include #include #include +#include +#include struct bl_dev_msg { int32_t status; @@ -21,8 +23,9 @@ struct nfs_netns_client; struct nfs_net { struct cache_detail *nfs_dns_resolve; struct rpc_pipe *bl_device_pipe; + struct rpc_pipe_msg bl_pipe_msg; struct bl_dev_msg bl_mount_reply; - wait_queue_head_t bl_wq; + struct completion bl_done; struct mutex bl_mutex; struct list_head nfs_client_list; struct list_head nfs_volume_list; diff --git a/fs/nfs/nfs4_fs.h b/fs/nfs/nfs4_fs.h index b48e5b87cb2a..29e2fb695953 100644 --- a/fs/nfs/nfs4_fs.h +++ b/fs/nfs/nfs4_fs.h @@ -526,6 +526,8 @@ extern int nfs4_select_rw_stateid(struct nfs4_state *, fmode_t, const struct cred **); extern bool nfs4_copy_open_stateid(nfs4_stateid *dst, struct nfs4_state *state); +extern bool nfs4_lock_context_lost(struct nfs4_state *state, + const struct nfs_lock_context *l_ctx); extern struct nfs_seqid *nfs_alloc_seqid(struct nfs_seqid_counter *counter, gfp_t gfp_mask); extern int nfs_wait_on_sequence(struct nfs_seqid *seqid, struct rpc_task *task); diff --git a/fs/nfs/nfs4proc.c b/fs/nfs/nfs4proc.c index 04b1987115d5..e41c792a2725 100644 --- a/fs/nfs/nfs4proc.c +++ b/fs/nfs/nfs4proc.c @@ -9680,6 +9680,14 @@ nfs4_layoutget_handle_exception(struct rpc_task *task, case -NFS4ERR_BADLAYOUT: status = -EOVERFLOW; goto out; + /* + * NFS4ERR_TOOSMALL means the layout for the requested range + * exceeds what the client advertised in loga_maxcount (see + * RFC8881 section 18.43.3). + */ + case -ETOOSMALL: + status = -EMSGSIZE; + goto out; /* * NFS4ERR_LAYOUTTRYLATER is a conflict with another client * (or clients) writing to the same RAID stripe except when diff --git a/fs/nfs/nfs4state.c b/fs/nfs/nfs4state.c index a5dec0473e22..78270037059c 100644 --- a/fs/nfs/nfs4state.c +++ b/fs/nfs/nfs4state.c @@ -812,10 +812,13 @@ void nfs4_close_sync(struct nfs4_state *state, fmode_t fmode) * be made. Failing that, a "consistent" choice is preferable. The * consistent choice we make is to prefer the first owner, that of a * Posix lock. + * + * Called with state->state_lock held. No reference is taken: a lock + * state is freed only after it is unlinked under state->state_lock. */ static struct nfs4_lock_state * -__nfs4_find_lock_state(struct nfs4_state *state, - fl_owner_t owner, fl_owner_t owner2) +__nfs4_lookup_lock_state(struct nfs4_state *state, + fl_owner_t owner, fl_owner_t owner2) { struct nfs4_lock_state *pos, *ret = NULL; list_for_each_entry(pos, &state->lock_states, ls_locks) { @@ -826,6 +829,16 @@ __nfs4_find_lock_state(struct nfs4_state *state, if (pos->ls_owner == owner2) ret = pos; } + return ret; +} + +static struct nfs4_lock_state * +__nfs4_find_lock_state(struct nfs4_state *state, + fl_owner_t owner, fl_owner_t owner2) +{ + struct nfs4_lock_state *ret; + + ret = __nfs4_lookup_lock_state(state, owner, owner2); if (ret) refcount_inc(&ret->ls_count); return ret; @@ -968,7 +981,7 @@ static int nfs4_copy_lock_stateid(nfs4_stateid *dst, fl_flock_owner = l_ctx->open_context->flock_owner; spin_lock(&state->state_lock); - lsp = __nfs4_find_lock_state(state, owner, fl_flock_owner); + lsp = __nfs4_lookup_lock_state(state, owner, fl_flock_owner); if (lsp && test_bit(NFS_LOCK_LOST, &lsp->ls_flags)) ret = -EIO; else if (lsp != NULL && test_bit(NFS_LOCK_INITIALIZED, &lsp->ls_flags) != 0) { @@ -976,11 +989,28 @@ static int nfs4_copy_lock_stateid(nfs4_stateid *dst, ret = 0; } spin_unlock(&state->state_lock); - nfs4_put_lock_state(lsp); out: return ret; } +bool nfs4_lock_context_lost(struct nfs4_state *state, + const struct nfs_lock_context *l_ctx) +{ + struct nfs4_lock_state *lsp; + bool ret; + + if (!test_bit(LK_STATE_IN_USE, &state->flags)) + return false; + + spin_lock(&state->state_lock); + lsp = __nfs4_lookup_lock_state(state, l_ctx->lockowner, + l_ctx->open_context->flock_owner); + ret = lsp && test_bit(NFS_LOCK_LOST, &lsp->ls_flags); + spin_unlock(&state->state_lock); + return ret; +} +EXPORT_SYMBOL_GPL(nfs4_lock_context_lost); + bool nfs4_copy_open_stateid(nfs4_stateid *dst, struct nfs4_state *state) { bool ret; diff --git a/fs/nfs/nfstrace.h b/fs/nfs/nfstrace.h index b15c1732c869..a32c76df72db 100644 --- a/fs/nfs/nfstrace.h +++ b/fs/nfs/nfstrace.h @@ -1772,7 +1772,10 @@ DECLARE_EVENT_CLASS(nfs_local_dio_class, __entry->count = count; __entry->mem_align = local_dio->mem_align; __entry->offset_align = local_dio->offset_align; - __entry->start = offset; + if (local_dio->start_len) + __entry->start = local_dio->middle_offset - local_dio->start_len; + else + __entry->start = 0; __entry->start_len = local_dio->start_len; __entry->middle = local_dio->middle_offset; __entry->middle_len = local_dio->middle_len; diff --git a/fs/nfs/pnfs.c b/fs/nfs/pnfs.c index 4f9c0f639014..c5d1951d5400 100644 --- a/fs/nfs/pnfs.c +++ b/fs/nfs/pnfs.c @@ -2331,6 +2331,8 @@ pnfs_update_layout(struct inode *ino, break; case -ENODATA: /* The server returned NFS4ERR_LAYOUTUNAVAILABLE */ + case -EMSGSIZE: + /* The layout exceeded loga_maxcount (NFS4ERR_TOOSMALL) */ pnfs_layout_set_fail_bit( lo, pnfs_iomode_to_fail_bit(iomode)); lseg = NULL; diff --git a/fs/nfs/pnfs.h b/fs/nfs/pnfs.h index 70d20f779678..5725ea06efc2 100644 --- a/fs/nfs/pnfs.h +++ b/fs/nfs/pnfs.h @@ -427,6 +427,7 @@ int nfs4_pnfs_ds_connect(struct nfs_server *mds_srv, struct nfs4_pnfs_ds *ds, struct nfs4_pnfs_ds_addr *nfs4_decode_mp_ds_addr(struct net *net, struct xdr_stream *xdr, gfp_t gfp_flags); +void nfs4_pnfs_ds_addr_list_free(struct list_head *dsaddrs); void pnfs_layout_mark_request_commit(struct nfs_page *req, struct pnfs_layout_segment *lseg, struct nfs_commit_info *cinfo, diff --git a/fs/nfs/pnfs_nfs.c b/fs/nfs/pnfs_nfs.c index 93d63f75a355..5bd014b981bf 100644 --- a/fs/nfs/pnfs_nfs.c +++ b/fs/nfs/pnfs_nfs.c @@ -633,23 +633,28 @@ static void nfs4_pnfs_ds_addr_free(struct nfs4_pnfs_ds_addr *da) kfree(da); } -static void destroy_ds(struct nfs4_pnfs_ds *ds) +void nfs4_pnfs_ds_addr_list_free(struct list_head *dsaddrs) { struct nfs4_pnfs_ds_addr *da; + while (!list_empty(dsaddrs)) { + da = list_first_entry(dsaddrs, struct nfs4_pnfs_ds_addr, + da_node); + list_del_init(&da->da_node); + nfs4_pnfs_ds_addr_free(da); + } +} +EXPORT_SYMBOL_GPL(nfs4_pnfs_ds_addr_list_free); + +static void destroy_ds(struct nfs4_pnfs_ds *ds) +{ dprintk("--> %s\n", __func__); ifdebug(FACILITY) print_ds(ds); nfs_put_client(ds->ds_clp); - while (!list_empty(&ds->ds_addrs)) { - da = list_first_entry(&ds->ds_addrs, - struct nfs4_pnfs_ds_addr, - da_node); - list_del_init(&da->da_node); - nfs4_pnfs_ds_addr_free(da); - } + nfs4_pnfs_ds_addr_list_free(&ds->ds_addrs); kfree(ds->ds_remotestr); kfree(ds); diff --git a/fs/nfsd/filecache.c b/fs/nfsd/filecache.c index b9548eb17c77..e80f88787ef7 100644 --- a/fs/nfsd/filecache.c +++ b/fs/nfsd/filecache.c @@ -1285,9 +1285,9 @@ nfsd_file_do_acquire(struct svc_rqst *rqstp, struct net *net, } status = nfserrno(ret); trace_nfsd_file_open(nf, status); - if (status == nfs_ok) - status = nfsd_file_get_dio_attrs(fhp, nf); } + if (status == nfs_ok) + status = nfsd_file_get_dio_attrs(fhp, nf); } else status = nfserr_jukebox; /* diff --git a/fs/nfsd/flexfilelayout.c b/fs/nfsd/flexfilelayout.c index 6d531285ab43..c73169e78d59 100644 --- a/fs/nfsd/flexfilelayout.c +++ b/fs/nfsd/flexfilelayout.c @@ -14,6 +14,7 @@ #include #include "flexfilelayoutxdr.h" +#include "auth.h" #include "pnfs.h" #include "vfs.h" @@ -23,10 +24,10 @@ static __be32 nfsd4_ff_proc_layoutget(struct svc_rqst *rqstp, struct inode *inode, const struct svc_fh *fhp, struct nfsd4_layoutget *args) { + struct user_namespace *userns = nfsd_user_namespace(rqstp); struct nfsd4_layout_seg *seg = &args->lg_seg; u32 device_generation = 0; int error; - uid_t u; struct pnfs_ff_layout *fl; @@ -49,13 +50,16 @@ nfsd4_ff_proc_layoutget(struct svc_rqst *rqstp, struct inode *inode, fl->flags = FF_FLAGS_NO_LAYOUTCOMMIT | FF_FLAGS_NO_IO_THRU_MDS | FF_FLAGS_NO_READ_IO; - /* Do not allow a IOMODE_READ segment to have write pemissions */ - if (seg->iomode == IOMODE_READ) { - u = from_kuid(&init_user_ns, inode->i_uid) + 1; - fl->uid = make_kuid(&init_user_ns, u); - } else - fl->uid = inode->i_uid; - fl->gid = inode->i_gid; + fl->uid = from_kuid_munged(userns, inode->i_uid); + fl->gid = from_kgid_munged(userns, inode->i_gid); + + /* + * Do not allow an IOMODE_READ segment to have write permissions. + * The group is left intact so group-readable files stay readable; + * nfsd_setuser() squashes an unmapped uid to the export's anon ID. + */ + if (seg->iomode == IOMODE_READ) + fl->uid++; error = nfsd4_set_deviceid(&fl->deviceid, fhp, device_generation); if (error) diff --git a/fs/nfsd/flexfilelayoutxdr.c b/fs/nfsd/flexfilelayoutxdr.c index 374e52d3064a..b80352f41c0d 100644 --- a/fs/nfsd/flexfilelayoutxdr.c +++ b/fs/nfsd/flexfilelayoutxdr.c @@ -32,8 +32,8 @@ nfsd4_ff_encode_layoutget(struct xdr_stream *xdr, fh_len = 4 + xdr_align_size(fl->fh.size); - uid.len = sprintf(uid.buf, "%u", from_kuid(&init_user_ns, fl->uid)); - gid.len = sprintf(gid.buf, "%u", from_kgid(&init_user_ns, fl->gid)); + uid.len = sprintf(uid.buf, "%u", fl->uid); + gid.len = sprintf(gid.buf, "%u", fl->gid); /* data server entry: deviceid + efficiency + stateid + fh list + * user + group + flags + stats_collect_hint diff --git a/fs/nfsd/flexfilelayoutxdr.h b/fs/nfsd/flexfilelayoutxdr.h index 6d5a1066a903..3e1876d49db2 100644 --- a/fs/nfsd/flexfilelayoutxdr.h +++ b/fs/nfsd/flexfilelayoutxdr.h @@ -35,8 +35,9 @@ struct pnfs_ff_device_addr { struct pnfs_ff_layout { u32 flags; u32 stats_collect_hint; - kuid_t uid; - kgid_t gid; + /* Values to encode; nfsd4_ff_proc_layoutget() has mapped these */ + u32 uid; + u32 gid; struct nfsd4_deviceid deviceid; stateid_t stateid; struct nfs_fh fh; diff --git a/fs/nfsd/nfs3acl.c b/fs/nfsd/nfs3acl.c index 6b6b289db636..f350a1403d22 100644 --- a/fs/nfsd/nfs3acl.c +++ b/fs/nfsd/nfs3acl.c @@ -13,6 +13,19 @@ #include "xdr3.h" #include "vfs.h" +static __be32 nfsd3_map_status(__be32 status) +{ + switch (status) { + case nfserr_nofilehandle: + status = nfserr_badhandle; + break; + case nfserr_wrongsec: + status = nfserr_acces; + break; + } + return status; +} + /* * NULL call. */ @@ -71,6 +84,7 @@ static __be32 nfsd3_proc_getacl(struct svc_rqst *rqstp) /* resp->acl_{access,default} are released in nfs3svc_release_getacl. */ out: + resp->status = nfsd3_map_status(resp->status); return rpc_success; fail: @@ -124,6 +138,7 @@ static __be32 nfsd3_proc_setacl(struct svc_rqst *rqstp) resp->status = nfserrno(error); out: /* argp->acl_{access,default} are released in nfs3svc_release_setacl. */ + resp->status = nfsd3_map_status(resp->status); return rpc_success; } diff --git a/fs/nfsd/nfs4idmap.c b/fs/nfsd/nfs4idmap.c index e9faf8b78f74..3514475c7323 100644 --- a/fs/nfsd/nfs4idmap.c +++ b/fs/nfsd/nfs4idmap.c @@ -253,8 +253,10 @@ idtoname_parse(struct cache_detail *cd, char *buf, int buflen) /* Name */ error = -EINVAL; len = qword_get(&buf, buf1, PAGE_SIZE); - if (len < 0 || len >= IDMAP_NAMESZ) + if (len < 0 || len >= IDMAP_NAMESZ) { + cache_put(&res->h, cd); goto out; + } if (len == 0) set_bit(CACHE_NEGATIVE, &ent.h.flags); else diff --git a/fs/nfsd/nfs4layouts.c b/fs/nfsd/nfs4layouts.c index 22bcb6d09f70..cef4d40c89a2 100644 --- a/fs/nfsd/nfs4layouts.c +++ b/fs/nfsd/nfs4layouts.c @@ -127,10 +127,12 @@ nfsd4_set_deviceid(struct nfsd4_deviceid *id, const struct svc_fh *fhp, void nfsd4_setup_layout_type(struct svc_export *exp) { + struct nfsd_net *nn = net_generic(exp->cd->net, nfsd_net_id); struct super_block *sb = exp->ex_path.mnt->mnt_sb; expfs_block_layouts_t block_supported = exportfs_layouts_supported(sb); - if (IS_ENABLED(CONFIG_NFSD_FLEXFILELAYOUT)) + if (IS_ENABLED(CONFIG_NFSD_FLEXFILELAYOUT) && + nfsd_vers(nn, 3, NFSD_TEST)) exp->ex_layout_types |= 1 << LAYOUT_FLEX_FILES; if (IS_ENABLED(CONFIG_NFSD_BLOCKLAYOUT) && (block_supported & EXPFS_BLOCK_IN_BAND_ID)) diff --git a/fs/nfsd/nfs4proc.c b/fs/nfsd/nfs4proc.c index 50c07561e31f..f3f7f14a9368 100644 --- a/fs/nfsd/nfs4proc.c +++ b/fs/nfsd/nfs4proc.c @@ -394,8 +394,8 @@ nfsd4_create_file(struct svc_rqst *rqstp, struct svc_fh *fhp, if ((iap->ia_valid & ATTR_SIZE) && (iap->ia_size == 0)) iap->ia_valid &= ~ATTR_SIZE; if (nfsd4_create_is_exclusive(open->op_createmode)) { - iap->ia_valid = ATTR_MTIME | ATTR_ATIME | - ATTR_MTIME_SET|ATTR_ATIME_SET; + iap->ia_valid |= ATTR_MTIME | ATTR_ATIME | + ATTR_MTIME_SET|ATTR_ATIME_SET; iap->ia_mtime.tv_sec = v_mtime; iap->ia_atime.tv_sec = v_atime; iap->ia_mtime.tv_nsec = 0; diff --git a/fs/nfsd/nfs4recover.c b/fs/nfsd/nfs4recover.c index d513971fb119..6441e54283a3 100644 --- a/fs/nfsd/nfs4recover.c +++ b/fs/nfsd/nfs4recover.c @@ -647,6 +647,8 @@ struct cld_upcall { struct list_head cu_list; struct cld_net *cu_net; struct completion cu_done; + /* daemon has read the whole upcall; protected by cn_lock */ + bool cu_inflight; union { struct cld_msg_hdr cu_hdr; struct cld_msg cu_msg; @@ -726,6 +728,11 @@ __cld_pipe_inprogress_downcall(const struct cld_msg_v2 __user *cmsg, name.len = namelen; if (get_user(princhashlen, &ci->cc_princhash.cp_len)) return -EFAULT; + if (princhashlen > SHA256_DIGEST_SIZE) { + dprintk("%s: invalid princhashlen (%u)", + __func__, princhashlen); + return -EINVAL; + } if (princhashlen > 0) { princhashcopy = memdup_user( &ci->cc_princhash.cp_data, @@ -807,6 +814,13 @@ cld_pipe_downcall(struct file *filp, const char __user *src, size_t mlen) spin_lock(&cn->cn_lock); list_for_each_entry(tmp, &cn->cn_list, cu_list) { if (get_unaligned(&tmp->cu_u.cu_hdr.cm_xid) == xid) { + /* + * Completing now would return the waiter + * while its message is still queued on the + * pipe. + */ + if (!tmp->cu_inflight) + break; cup = tmp; if (status != -EINPROGRESS) list_del_init(&cup->cu_list); @@ -815,9 +829,9 @@ cld_pipe_downcall(struct file *filp, const char __user *src, size_t mlen) } spin_unlock(&cn->cn_lock); - /* couldn't find upcall? */ + /* no upcall to complete? */ if (!cup) { - dprintk("%s: couldn't find upcall -- xid=%u\n", __func__, xid); + dprintk("%s: no completable upcall -- xid=%u\n", __func__, xid); return -EINVAL; } @@ -837,8 +851,16 @@ cld_pipe_destroy_msg(struct rpc_pipe_msg *msg) struct cld_msg *cmsg = msg->data; struct cld_upcall *cup = container_of(cmsg, struct cld_upcall, cu_u.cu_msg); + struct cld_net *cn = cup->cu_net; + + /* + * errno >= 0 means the daemon read the whole message and a + * downcall will complete the upcall. + */ + spin_lock(&cn->cn_lock); + cup->cu_inflight = msg->errno >= 0; + spin_unlock(&cn->cn_lock); - /* errno >= 0 means we got a downcall */ if (msg->errno >= 0) return; @@ -915,18 +937,24 @@ __nfsd4_init_cld_pipe(struct net *net) } spin_lock_init(&cn->cn_lock); INIT_LIST_HEAD(&cn->cn_list); +#ifdef CONFIG_NFSD_LEGACY_CLIENT_TRACKING + cn->cn_has_legacy = false; +#endif + + /* + * The pipe's methods reach @cn through nn->cld_net, so set + * it before the pipe can be opened. + */ + nn->cld_net = cn; ret = nfsd4_cld_register_net(net, cn->cn_pipe); if (unlikely(ret)) goto err_destroy_data; -#ifdef CONFIG_NFSD_LEGACY_CLIENT_TRACKING - cn->cn_has_legacy = false; -#endif - nn->cld_net = cn; return 0; err_destroy_data: + nn->cld_net = NULL; rpc_destroy_pipe_data(cn->cn_pipe); err: kfree(cn); diff --git a/fs/nfsd/nfs4state.c b/fs/nfsd/nfs4state.c index 9c4adf3110ae..3bdde4c1bb8e 100644 --- a/fs/nfsd/nfs4state.c +++ b/fs/nfsd/nfs4state.c @@ -92,6 +92,8 @@ static void nfsd4_end_grace(struct nfsd_net *nn); static void _free_cpntf_state_locked(struct nfsd_net *nn, struct nfs4_cpntf_state *cps); static void nfsd4_file_hash_remove(struct nfs4_file *fi); static void deleg_reaper(struct nfsd_net *nn); +static void nfsd4_drop_revoked_stid(struct nfs4_stid *s) + __releases(&s->sc_client->cl_lock); static const struct lease_manager_operations nfsd_lease_mng_ops; @@ -1279,6 +1281,9 @@ __alloc_init_deleg(struct nfs4_client *clp, struct nfs4_file *fp, dp->dl_type = dl_type; dp->dl_retries = 1; dp->dl_recalled = false; + dp->dl_recall_rejected = false; + dp->dl_recall_grant.valid = false; + dp->dl_recall_grant.retired_at_send = false; get_nfs4_file(fp); dp->dl_stid.sc_file = fp; nfsd4_init_cb(&dp->dl_recall, dp->dl_stid.sc_client, @@ -1563,27 +1568,22 @@ static void destroy_delegation(struct nfs4_delegation *dp) } /** - * revoke_delegation - perform nfs4 delegation structure cleanup - * @dp: pointer to the delegation + * revoke_delegation - dispose of a delegation the server has revoked + * @dp: delegation to dispose of + * + * The caller holds a reference on @dp, which this function consumes. + * On NFSv4.1 and newer, @dp's sc_status must already carry + * SC_STATUS_REVOKED or SC_STATUS_ADMIN_REVOKED. * - * This function assumes that it's called either from the administrative - * interface (nfsd4_revoke_states()) that's revoking a specific delegation - * stateid or it's called from a laundromat thread (nfsd4_landromat()) that - * determined that this specific state has expired and needs to be revoked - * (both mark state with the appropriate stid sc_status mode). It is also - * assumed that a reference was taken on the @dp state. This function - * consumes that reference. + * @dp is parked on the client's cl_revoked list to await a FREE_STATEID. + * Where none can arrive, @dp is destroyed here instead: FREE_STATEID has + * already freed it, or the client rejected the recall with + * NFS4ERR_BADHANDLE or NFS4ERR_BAD_STATEID and holds no record of the + * delegation. NFS4ERR_ADMIN_REVOKED still prompts one, so an + * administrative revoke waits on cl_revoked. * - * If this function finds that the @dp state is SC_STATUS_FREED it means - * that a FREE_STATEID operation for this stateid has been processed and - * we can proceed to removing it from recalled list. However, if @dp state - * isn't marked SC_STATUS_FREED, it means we need place it on the cl_revoked - * list and wait for the FREE_STATEID to arrive from the client. At the same - * time, we need to mark it as SC_STATUS_FREEABLE to indicate to the - * nfsd4_free_stateid() function that this stateid has already been added - * to the cl_revoked list and that nfsd4_free_stateid() is now responsible - * for removing it from the list. Inspection of where the delegation state - * in the revocation process is protected by the clp->cl_lock. + * Context: Takes and releases the client's cl_lock; may sleep after + * dropping it. */ static void revoke_delegation(struct nfs4_delegation *dp) { @@ -1601,6 +1601,19 @@ static void revoke_delegation(struct nfs4_delegation *dp) list_del_init(&dp->dl_recall_lru); goto out; } + if (dp->dl_recall_rejected && + !(dp->dl_stid.sc_status & SC_STATUS_ADMIN_REVOKED)) { + /* + * SC_STATUS_CLOSED, set under cl_lock, makes a racing + * FREE_STATEID bail out rather than drop this reference + * too. The put releases what cl_revoked would have held. + */ + dp->dl_stid.sc_status |= SC_STATUS_CLOSED; + spin_unlock(&clp->cl_lock); + nfs4_put_stid(&dp->dl_stid); + destroy_unhashed_deleg(dp); + return; + } list_add(&dp->dl_recall_lru, &clp->cl_revoked); dp->dl_stid.sc_status |= SC_STATUS_FREEABLE; out: @@ -2789,10 +2802,16 @@ void nfsd4_put_client(struct nfs4_client *clp) static void free_client(struct nfs4_client *clp) { - while (!list_empty(&clp->cl_sessions)) { + LIST_HEAD(reaplist); + + /* client_info_show() walks cl_sessions under cl_lock */ + spin_lock(&clp->cl_lock); + list_splice_init(&clp->cl_sessions, &reaplist); + spin_unlock(&clp->cl_lock); + while (!list_empty(&reaplist)) { struct nfsd4_session *ses; - ses = list_entry(clp->cl_sessions.next, struct nfsd4_session, - se_perclnt); + ses = list_entry(reaplist.next, struct nfsd4_session, + se_perclnt); list_del(&ses->se_perclnt); WARN_ON_ONCE(atomic_read(&ses->se_ref)); free_session(ses); @@ -2889,11 +2908,18 @@ __destroy_client(struct nfs4_client *clp) list_del_init(&dp->dl_recall_lru); destroy_unhashed_deleg(dp); } + /* + * A CB_RECALL reply can release revoked delegations concurrently: + * nfsd4_shutdown_callback() has not run yet. + */ + spin_lock(&clp->cl_lock); while (!list_empty(&clp->cl_revoked)) { dp = list_entry(clp->cl_revoked.next, struct nfs4_delegation, dl_recall_lru); - list_del_init(&dp->dl_recall_lru); - nfs4_put_stid(&dp->dl_stid); + /* this function drops ->cl_lock */ + nfsd4_drop_revoked_stid(&dp->dl_stid); + spin_lock(&clp->cl_lock); } + spin_unlock(&clp->cl_lock); while (!list_empty(&clp->cl_openowners)) { oo = list_entry(clp->cl_openowners.next, struct nfs4_openowner, oo_perclient); nfs4_get_stateowner(&oo->oo_owner); @@ -4216,12 +4242,18 @@ nfsd4_set_ex_flags(struct nfs4_client *new, struct nfsd4_exchange_id *clid) static bool client_has_openowners(struct nfs4_client *clp) { struct nfs4_openowner *oo; + bool found = false; + spin_lock(&clp->cl_lock); list_for_each_entry(oo, &clp->cl_openowners, oo_perclient) { - if (!list_empty(&oo->oo_owner.so_stateids)) - return true; + if (!list_empty(&oo->oo_owner.so_stateids)) { + found = true; + break; + } } - return false; + spin_unlock(&clp->cl_lock); + + return found; } static bool client_has_state(struct nfs4_client *clp) @@ -5016,6 +5048,7 @@ __be32 nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, union nfsd4_op_u *u) { + struct nfsd4_compoundargs *args = rqstp->rq_argp; struct nfsd4_sequence *seq = &u->sequence; struct nfsd4_compoundres *resp = rqstp->rq_resp; struct xdr_stream *xdr = resp->xdr; @@ -5025,6 +5058,7 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, struct nfsd4_conn *conn; __be32 status; int buflen; + u32 maxlen, respsize; struct net *net = SVC_NET(rqstp); struct nfsd_net *nn = net_generic(net, nfsd_net_id); @@ -5102,7 +5136,22 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, session->se_fchannel.maxresp_sz; status = (seq->cachethis) ? nfserr_rep_too_big_to_cache : nfserr_rep_too_big; - if (xdr_restrict_buflen(xdr, buflen - rqstp->rq_auth_slack)) + if (buflen < rqstp->rq_auth_slack) + goto out_put_session; + maxlen = buflen - rqstp->rq_auth_slack; + + /* + * A SEQUENCE result too large for maxlen never reaches + * nfsd4_encode_sequence(), so cstate.data_offset stays zero and + * the reply cache overruns the slot. + */ + respsize = nfsd4_max_reply(rqstp, &args->ops[0]); + if (!nfsd4_last_compound_op(rqstp)) + respsize += COMPOUND_ERR_SLACK_SPACE; + if (xdr->buf->len + respsize > maxlen) + goto out_put_session; + + if (xdr_restrict_buflen(xdr, maxlen)) goto out_put_session; svc_reserve_auth(rqstp, buflen); @@ -5118,6 +5167,7 @@ nfsd4_sequence(struct svc_rqst *rqstp, struct nfsd4_compound_state *cstate, slot->sl_flags &= ~NFSD4_SLOT_CACHETHIS; cstate->slot = slot; + cstate->slot_owned = true; cstate->session = session; cstate->clp = clp; @@ -5192,7 +5242,11 @@ nfsd4_sequence_done(struct nfsd4_compoundres *resp) struct nfsd4_compound_state *cs = &resp->cstate; if (nfsd4_has_session(cs)) { - if (cs->status != nfserr_replay_cache) { + /* + * Only the request that claimed the slot may update its + * cached reply and clear NFSD4_SLOT_INUSE. + */ + if (cs->slot_owned) { nfsd4_store_cache_entry(resp); cs->slot->sl_flags &= ~NFSD4_SLOT_INUSE; } @@ -6063,6 +6117,60 @@ bool nfsd_wait_for_delegreturn(struct svc_rqst *rqstp, struct inode *inode) return timeo > 0; } +static bool nfsd4_recall_grant_slot_retired(struct nfs4_delegation *dp) +{ + struct nfs4_client *clp = dp->dl_stid.sc_client; + struct nfsd_net *nn = net_generic(clp->net, nfsd_net_id); + struct nfsd4_session *ses; + struct nfsd4_sessionid sid; + bool retired = false; + void *entry; + + if (!dp->dl_recall_grant.valid) + return false; + + /* + * gen_sessionid() composes a sessionid from the client's clientid + * and a sequence counter, so the sequence alone identifies the + * granting session. + */ + sid.clientid = clp->cl_clientid; + sid.sequence = dp->dl_recall_grant.sessionid_seq; + sid.reserved = 0; + + /* + * A missing session does not prove the client saw the grant: a + * DESTROY_SESSION unhashes its own session before the reply to + * that compound is encoded. + */ + spin_lock(&nn->client_lock); + ses = __find_in_sessionid_hashtbl((struct nfs4_sessionid *)&sid, + clp->net); + entry = ses ? xa_load(&ses->se_slots, dp->dl_recall_grant.slotid) : NULL; + if (xa_is_value(entry)) { + /* + * A slot is freed only once the client has acknowledged + * the smaller slot table, which it cannot do while a + * request on that slot is outstanding. + */ + retired = true; + } else if (entry) { + struct nfsd4_slot *slot = entry; + + /* + * A reactivated slot was freed and rebuilt, so the same + * acknowledgment applies. The seqid test errs toward + * revoking: a rebuilt slot restarting at seqid 1 matches + * an old grant. + */ + retired = (slot->sl_flags & NFSD4_SLOT_REUSED) || + ((slot->sl_flags & NFSD4_SLOT_INITIALIZED) && + slot->sl_seqid != dp->dl_recall_grant.seqid); + } + spin_unlock(&nn->client_lock); + return retired; +} + static bool nfsd4_cb_recall_prepare(struct nfsd4_callback *cb) { struct nfs4_delegation *dp = cb_to_delegation(cb); @@ -6084,9 +6192,37 @@ static bool nfsd4_cb_recall_prepare(struct nfsd4_callback *cb) list_add_tail(&dp->dl_recall_lru, &nn->del_recall_lru); } spin_unlock(&nn->deleg_lock); + + dp->dl_recall_grant.retired_at_send = + nfsd4_recall_grant_slot_retired(dp); return true; } +/* + * cl_lock orders this against a laundromat reaping @dp: either + * revoke_delegation() observes dl_recall_rejected and destroys @dp, or + * it reached cl_revoked first and @dp is released here instead. + */ +static void nfsd4_deleg_recall_rejected(struct nfs4_delegation *dp) +{ + struct nfs4_client *clp = dp->dl_stid.sc_client; + + spin_lock(&clp->cl_lock); + if (dp->dl_stid.sc_status & (SC_STATUS_CLOSED | SC_STATUS_FREED | + SC_STATUS_ADMIN_REVOKED)) { + spin_unlock(&clp->cl_lock); + return; + } + if (dp->dl_stid.sc_status & SC_STATUS_FREEABLE) { + dp->dl_stid.sc_status |= SC_STATUS_CLOSED; + /* this function drops ->cl_lock */ + nfsd4_drop_revoked_stid(&dp->dl_stid); + return; + } + dp->dl_recall_rejected = true; + spin_unlock(&clp->cl_lock); +} + static int nfsd4_cb_recall_done(struct nfsd4_callback *cb, struct rpc_task *task) { @@ -6094,27 +6230,33 @@ static int nfsd4_cb_recall_done(struct nfsd4_callback *cb, trace_nfsd_cb_recall_done(&dp->dl_stid.sc_stateid, task); - if (dp->dl_stid.sc_status) - /* CLOSED or REVOKED */ - return 1; - switch (task->tk_status) { case 0: return 1; case -NFS4ERR_DELAY: + if (dp->dl_stid.sc_status) + /* CLOSED or REVOKED */ + return 1; rpc_delay(task, 2 * HZ); return 0; case -EBADHANDLE: case -NFS4ERR_BAD_STATEID: /* - * Race: client probably got cb_recall before open reply - * granting delegation. + * Retirement of the granting slot proves the client saw + * the grant. Trust the rejection only if the slot had + * retired when this recall was sent. */ - if (dp->dl_retries--) { + if (dp->dl_recall_grant.retired_at_send) { + nfsd4_deleg_recall_rejected(dp); + return 1; + } + if (!dp->dl_stid.sc_status && dp->dl_retries--) { + dp->dl_recall_grant.retired_at_send = + nfsd4_recall_grant_slot_retired(dp); rpc_delay(task, 2 * HZ); return 0; } - fallthrough; + return 1; default: return 1; } @@ -6708,9 +6850,25 @@ static bool nfsd4_want_deleg_timestamps(const struct nfsd4_open *open) return open->op_deleg_want & OPEN4_SHARE_ACCESS_WANT_DELEG_TIMESTAMPS; } +static void +nfs4_delegation_record_grant_slot(struct nfs4_delegation *dp, + const struct nfsd4_compound_state *cstate) +{ + const struct nfsd4_sessionid *sid; + + if (!cstate->session) + return; + sid = (struct nfsd4_sessionid *)cstate->session->se_sessionid.data; + dp->dl_recall_grant.sessionid_seq = sid->sequence; + dp->dl_recall_grant.slotid = cstate->slot->sl_index; + dp->dl_recall_grant.seqid = cstate->slot->sl_seqid; + dp->dl_recall_grant.valid = true; +} + static struct nfs4_delegation * -nfs4_set_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp, - struct svc_fh *parent) +nfs4_set_delegation(struct nfsd4_open *open, + const struct nfsd4_compound_state *cstate, + struct nfs4_ol_stateid *stp, struct svc_fh *parent) { bool deleg_ts = nfsd4_want_deleg_timestamps(open); struct nfs4_client *clp = stp->st_stid.sc_client; @@ -6800,6 +6958,14 @@ nfs4_set_delegation(struct nfsd4_open *open, struct nfs4_ol_stateid *stp, dp = alloc_init_deleg(clp, fp, odstate, dl_type); if (!dp) goto out_delegees; + + /* + * Record the granting slot before kernel_setlease() makes @dp + * visible to lease breakers. A conflicting open can drive + * CB_RECALL to completion from that point on. + */ + nfs4_delegation_record_grant_slot(dp, cstate); + if (stp->st_stid.sc_export) dp->dl_stid.sc_export = exp_get(stp->st_stid.sc_export); @@ -6964,6 +7130,7 @@ nfs4_open_delegation(struct svc_rqst *rqstp, struct nfsd4_open *open, struct nfs4_ol_stateid *stp, struct svc_fh *currentfh, struct svc_fh *fh) { + struct nfsd4_compoundres *resp = rqstp->rq_resp; struct nfs4_openowner *oo = openowner(stp->st_stateowner); bool deleg_ts = nfsd4_want_deleg_timestamps(open); struct nfs4_client *clp = stp->st_stid.sc_client; @@ -7000,7 +7167,7 @@ nfs4_open_delegation(struct svc_rqst *rqstp, struct nfsd4_open *open, default: goto out_no_deleg; } - dp = nfs4_set_delegation(open, stp, parent); + dp = nfs4_set_delegation(open, &resp->cstate, stp, parent); if (IS_ERR(dp)) goto out_no_deleg; @@ -7776,6 +7943,8 @@ deleg_reaper(struct nfsd_net *nn) list_for_each_safe(pos, next, &nn->client_lru) { clp = list_entry(pos, struct nfs4_client, cl_lru); + if (clp->cl_minorversion == 0) + continue; if (clp->cl_state != NFSD4_ACTIVE) continue; if (list_empty(&clp->cl_delegations)) @@ -10294,6 +10463,7 @@ nfsd_get_dir_deleg(struct nfsd4_compound_state *cstate, dp = alloc_init_dir_deleg(clp, fp); if (!dp) goto out_delegees; + nfs4_delegation_record_grant_slot(dp, cstate); if (cstate->current_fh.fh_export) dp->dl_stid.sc_export = exp_get(cstate->current_fh.fh_export); diff --git a/fs/nfsd/nfs4xdr.c b/fs/nfsd/nfs4xdr.c index 606ddcb085c0..ebd0dc8f00fc 100644 --- a/fs/nfsd/nfs4xdr.c +++ b/fs/nfsd/nfs4xdr.c @@ -2320,10 +2320,8 @@ static __be32 nfsd4_vbuf_from_vector(struct nfsd4_compoundargs *argp, struct xdr_buf *xdr, char **bufp, size_t buflen) { - struct page **pages = xdr->pages; struct kvec *head = xdr->head; - char *tmp, *dp; - u32 len; + char *tmp; if (buflen <= head->iov_len) { /* @@ -2338,19 +2336,8 @@ nfsd4_vbuf_from_vector(struct nfsd4_compoundargs *argp, struct xdr_buf *xdr, if (tmp == NULL) return nfserr_jukebox; - dp = tmp; - memcpy(dp, head->iov_base, head->iov_len); - buflen -= head->iov_len; - dp += head->iov_len; - - while (buflen > 0) { - len = min_t(u32, buflen, PAGE_SIZE); - memcpy(dp, page_address(*pages), len); - - buflen -= len; - dp += len; - pages++; - } + if (read_bytes_from_xdr_buf(xdr, 0, tmp, buflen)) + return nfserr_bad_xdr; *bufp = tmp; return 0; @@ -2467,7 +2454,7 @@ nfsd4_decode_setxattr(struct nfsd4_compoundargs *argp, &setxattr->setxa_buf, size); } - return nfs_ok; + return status; } static __be32 @@ -6274,7 +6261,7 @@ nfsd4_vbuf_to_stream(struct xdr_stream *xdr, char *buf, u32 buflen) if (!p) return nfserr_resource; - memcpy(p, buf, cplen); + xdr_encode_opaque_fixed(p, buf, cplen); buf += cplen; buflen -= cplen; @@ -6801,8 +6788,7 @@ nfs4svc_encode_compoundres(struct svc_rqst *rqstp, struct xdr_stream *xdr) *p++ = resp->cstate.status; *p++ = htonl(resp->taglen); - memcpy(p, resp->tag, resp->taglen); - p += XDR_QUADLEN(resp->taglen); + p = xdr_encode_opaque_fixed(p, resp->tag, resp->taglen); *p++ = htonl(resp->opcnt); nfsd4_sequence_done(resp); diff --git a/fs/nfsd/nfsctl.c b/fs/nfsd/nfsctl.c index 5abb2d4274c9..86aa8d202b4b 100644 --- a/fs/nfsd/nfsctl.c +++ b/fs/nfsd/nfsctl.c @@ -477,7 +477,7 @@ static ssize_t write_pool_threads(struct file *file, char *buf, size_t size) char *mesg = buf; int i; int rv; - int len; + size_t len; int npools; int *nthreads; struct net *net = netns(file); @@ -531,9 +531,13 @@ static ssize_t write_pool_threads(struct file *file, char *buf, size_t size) mesg = buf; size = SIMPLE_TRANSACTION_LIMIT; - for (i = 0; i < npools && size > 0; i++) { - snprintf(mesg, size, "%d%c", nthreads[i], (i == npools-1 ? '\n' : ' ')); - len = strlen(mesg); + for (i = 0; i < npools; i++) { + len = snprintf(mesg, size, "%d%c", nthreads[i], + (i == npools - 1 ? '\n' : ' ')); + if (len >= size) { + rv = -ENAMETOOLONG; + goto out_free; + } size -= len; mesg += len; } @@ -1581,14 +1585,29 @@ int nfsd_nl_rpc_status_get_dumpit(struct sk_buff *skb, rqstp->rq_proc == NFSPROC4_COMPOUND) { /* NFSv4 compound */ struct nfsd4_compoundargs *args; + struct nfsd4_op *ops; + u32 opcnt; int j; args = rqstp->rq_argp; - genl_rqstp.rq_opcnt = min_t(u32, args->opcnt, + opcnt = READ_ONCE(args->opcnt); + ops = READ_ONCE(args->ops); + + /* + * Finish the seqcount retry before + * dereferencing ops. An unchanged counter means + * opcnt and ops came from the same COMPOUND, + * where opcnt cannot exceed what ops holds. + */ + smp_rmb(); + if (READ_ONCE(rqstp->rq_status_counter) != + status_counter) + continue; + + genl_rqstp.rq_opcnt = min_t(u32, opcnt, ARRAY_SIZE(genl_rqstp.rq_opnum)); for (j = 0; j < genl_rqstp.rq_opcnt; j++) - genl_rqstp.rq_opnum[j] = - args->ops[j].opnum; + genl_rqstp.rq_opnum[j] = ops[j].opnum; } #endif /* CONFIG_NFSD_V4 */ @@ -1973,21 +1992,39 @@ int nfsd_nl_version_get_doit(struct sk_buff *skb, struct genl_info *info) return err; } +/* + * Transport classes NFSD knows how to instantiate. Vetting the name here + * keeps a bogus string from reaching svc_xprt_create_from_sa(), where an + * unknown name triggers a request_module("svc%s", name) upcall under + * nfsd_mutex. + */ +static bool nfsd_nl_transport_supported(const char *name) +{ + static const char * const supported[] = { "tcp", "udp", "rdma" }; + int i; + + for (i = 0; i < ARRAY_SIZE(supported); i++) + if (!strcmp(name, supported[i])) + return true; + return false; +} + +/* Upper bound on the number of listeners a single request may carry. */ +#define NFSD_NL_LISTENER_MAX 1024 + /** * nfsd_nl_validate_listeners - sanity-check the listener list from userland * @info: netlink metadata and command arguments * - * Walk every NFSD_A_SERVER_SOCK_ADDR attribute and confirm that each entry - * is well-formed: it parses against the policy, carries both an address and - * a transport name, and the address is long enough for its family. Doing - * this up front lets the callers below assume every entry is valid and - * guarantees we make no changes when the request is malformed. + * Walk every NFSD_A_SERVER_SOCK_ADDR attribute and confirm that the list is + * not oversized and that each entry is well-formed. * * Return: 0 if every entry is valid, or a negative errno otherwise. */ static int nfsd_nl_validate_listeners(struct genl_info *info) { const struct nlattr *attr; + unsigned int count = 0; int rem; nlmsg_for_each_attr_type(attr, NFSD_A_SERVER_SOCK_ADDR, info->nlhdr, @@ -1996,6 +2033,11 @@ static int nfsd_nl_validate_listeners(struct genl_info *info) struct sockaddr *sa; int err; + if (++count > NFSD_NL_LISTENER_MAX) { + NL_SET_ERR_MSG(info->extack, "too many listeners"); + return -E2BIG; + } + err = nla_parse_nested(tb, NFSD_A_SOCK_MAX, attr, nfsd_sock_nl_policy, info->extack); if (err < 0) @@ -2004,6 +2046,13 @@ static int nfsd_nl_validate_listeners(struct genl_info *info) if (!tb[NFSD_A_SOCK_ADDR] || !tb[NFSD_A_SOCK_TRANSPORT_NAME]) return -EINVAL; + if (!nfsd_nl_transport_supported(nla_data(tb[NFSD_A_SOCK_TRANSPORT_NAME]))) { + NL_SET_ERR_MSG_ATTR(info->extack, + tb[NFSD_A_SOCK_TRANSPORT_NAME], + "unsupported transport name"); + return -EPROTONOSUPPORT; + } + sa = nla_data(tb[NFSD_A_SOCK_ADDR]); if (nla_len(tb[NFSD_A_SOCK_ADDR]) < sizeof(sa->sa_family)) return -EINVAL; diff --git a/fs/nfsd/nfsproc.c b/fs/nfsd/nfsproc.c index e2b5f8a241be..aaeb222b62ba 100644 --- a/fs/nfsd/nfsproc.c +++ b/fs/nfsd/nfsproc.c @@ -39,6 +39,21 @@ static __be32 nfsd_map_status(__be32 status) return status; } +/* + * Because NFSv2 does not have an NFSERR_SYMLINK, Solaris returns + * NFSERR_ISDIR when the target of a READ or WRITE is any object + * that is not a regular file. + */ +static __be32 nfsd_map_io_status(__be32 status) +{ + switch (status) { + case nfserr_symlink: + case nfserr_wrong_type: + return nfserr_isdir; + } + return nfsd_map_status(status); +} + static __be32 nfsd_proc_null(struct svc_rqst *rqstp) { @@ -237,7 +252,7 @@ nfsd_proc_read(struct svc_rqst *rqstp) resp->status = fh_getattr(&resp->fh, &resp->stat); else if (resp->status == nfserr_jukebox) set_bit(RQ_DROPME, &rqstp->rq_flags); - resp->status = nfsd_map_status(resp->status); + resp->status = nfsd_map_io_status(resp->status); return rpc_success; } @@ -270,7 +285,7 @@ nfsd_proc_write(struct svc_rqst *rqstp) resp->status = fh_getattr(&resp->fh, &resp->stat); else if (resp->status == nfserr_jukebox) set_bit(RQ_DROPME, &rqstp->rq_flags); - resp->status = nfsd_map_status(resp->status); + resp->status = nfsd_map_io_status(resp->status); return rpc_success; } diff --git a/fs/nfsd/state.h b/fs/nfsd/state.h index 2d00a411c663..9661c08b879e 100644 --- a/fs/nfsd/state.h +++ b/fs/nfsd/state.h @@ -270,7 +270,8 @@ struct nfsd4_cb_notify { * If the server attempts to recall a delegation and the client doesn't do so * before a timeout, the server may also revoke the delegation. In that case, * the object will either be destroyed (v4.0) or moved to a per-client list of - * revoked delegations (v4.1+). + * revoked delegations (v4.1+). A v4.1+ client that rejects the recall holds + * no record of the delegation, so the object is destroyed rather than listed. * * This object is a superset of the nfs4_stid. */ @@ -286,9 +287,19 @@ struct nfs4_delegation { int dl_retries; struct nfsd4_callback dl_recall; bool dl_recalled; + bool dl_recall_rejected; bool dl_written; bool dl_setattr; + /* Forward-channel slot that carried the granting request */ + struct { + u32 sessionid_seq; + u32 slotid; + u32 seqid; + bool valid; + bool retired_at_send; + } dl_recall_grant; + union { /* for CB_GETATTR */ struct nfs4_cb_fattr dl_cb_fattr; diff --git a/fs/nfsd/xdr4.h b/fs/nfsd/xdr4.h index c7eda5bc833b..ca1c2063eea2 100644 --- a/fs/nfsd/xdr4.h +++ b/fs/nfsd/xdr4.h @@ -188,6 +188,7 @@ struct nfsd4_compound_state { struct nfsd4_slot *slot; int data_offset; bool spo_must_allowed; + bool slot_owned; size_t iovlen; u32 minorversion; __be32 status; diff --git a/fs/ocfs2/cluster/heartbeat.c b/fs/ocfs2/cluster/heartbeat.c index 1c3def99bb07..602f8b913de7 100644 --- a/fs/ocfs2/cluster/heartbeat.c +++ b/fs/ocfs2/cluster/heartbeat.c @@ -880,8 +880,11 @@ static void o2hb_shutdown_slot(struct o2hb_disk_slot *slot) int queued = 0; node = o2nm_get_node_by_num(slot->ds_node_num); - if (!node) - return; + /* + * The node may have been removed from the node table while this + * region's slot is still linked. The list member must still be + * removed before the region releases its slot array. + */ spin_lock(&o2hb_live_lock); if (!list_empty(&slot->ds_live_item)) { @@ -903,7 +906,8 @@ static void o2hb_shutdown_slot(struct o2hb_disk_slot *slot) if (queued) o2hb_run_event_list(&event); - o2nm_node_put(node); + if (node) + o2nm_node_put(node); } static void o2hb_set_quorum_device(struct o2hb_region *reg) @@ -1038,7 +1042,7 @@ static int o2hb_check_slot(struct o2hb_region *reg, fire_callbacks: /* dead nodes only come to life after some number of * changes at any time during their dead time */ - if (list_empty(&slot->ds_live_item) && + if (node && list_empty(&slot->ds_live_item) && slot->ds_changed_samples >= O2HB_LIVE_THRESHOLD) { mlog(ML_HEARTBEAT, "Node %d (id 0x%llx) joined my region\n", slot->ds_node_num, (long long)slot->ds_last_generation); @@ -1443,6 +1447,7 @@ static ssize_t o2hb_debug_read(struct file *file, char __user *buf, #endif /* CONFIG_DEBUG_FS */ static const struct file_operations o2hb_debug_fops = { + .owner = THIS_MODULE, .open = o2hb_debug_open, .release = o2hb_debug_release, .read = o2hb_debug_read, diff --git a/fs/ocfs2/cluster/netdebug.c b/fs/ocfs2/cluster/netdebug.c index f7ab66e70799..5d373002297d 100644 --- a/fs/ocfs2/cluster/netdebug.c +++ b/fs/ocfs2/cluster/netdebug.c @@ -183,6 +183,7 @@ static int nst_fop_release(struct inode *inode, struct file *file) } static const struct file_operations nst_seq_fops = { + .owner = THIS_MODULE, .open = nst_fop_open, .read = seq_read, .llseek = seq_lseek, @@ -415,6 +416,7 @@ static int stats_fop_open(struct inode *inode, struct file *file) } static const struct file_operations stats_seq_fops = { + .owner = THIS_MODULE, .open = stats_fop_open, .read = seq_read, .llseek = seq_lseek, @@ -427,6 +429,7 @@ static int sc_fop_open(struct inode *inode, struct file *file) } static const struct file_operations sc_seq_fops = { + .owner = THIS_MODULE, .open = sc_fop_open, .read = seq_read, .llseek = seq_lseek, @@ -476,6 +479,7 @@ static ssize_t o2net_debug_read(struct file *file, char __user *buf, } static const struct file_operations nodes_fops = { + .owner = THIS_MODULE, .open = nodes_fop_open, .release = o2net_debug_release, .read = o2net_debug_read, diff --git a/fs/ocfs2/cluster/nodemanager.c b/fs/ocfs2/cluster/nodemanager.c index e08850a5d736..ced4fe6d3b59 100644 --- a/fs/ocfs2/cluster/nodemanager.c +++ b/fs/ocfs2/cluster/nodemanager.c @@ -12,6 +12,7 @@ #include "tcp.h" #include "nodemanager.h" #include "heartbeat.h" +#include "quorum.h" #include "masklog.h" #include "sys.h" @@ -740,6 +741,8 @@ static void o2nm_cluster_group_drop_item(struct config_group *group, struct conf struct o2nm_cluster *cluster = to_o2nm_cluster(item); BUG_ON(o2nm_single_cluster != cluster); + /* Drain quorum work before it loses access to the cluster. */ + o2quo_flush_work(); o2nm_single_cluster = NULL; configfs_remove_default_groups(&cluster->cl_group); diff --git a/fs/ocfs2/cluster/quorum.c b/fs/ocfs2/cluster/quorum.c index bfb8b456876c..ab5b3d4fd0e8 100644 --- a/fs/ocfs2/cluster/quorum.c +++ b/fs/ocfs2/cluster/quorum.c @@ -318,9 +318,14 @@ void o2quo_init(void) INIT_WORK(&qs->qs_work, o2quo_make_decision); } -void o2quo_exit(void) +void o2quo_flush_work(void) { struct o2quo_state *qs = &o2quo_state; flush_work(&qs->qs_work); } + +void o2quo_exit(void) +{ + o2quo_flush_work(); +} diff --git a/fs/ocfs2/cluster/quorum.h b/fs/ocfs2/cluster/quorum.h index d64bf4482a4a..1ce5fc2a8ca9 100644 --- a/fs/ocfs2/cluster/quorum.h +++ b/fs/ocfs2/cluster/quorum.h @@ -8,6 +8,7 @@ void o2quo_init(void); void o2quo_exit(void); +void o2quo_flush_work(void); void o2quo_hb_up(u8 node); void o2quo_hb_down(u8 node); diff --git a/fs/ocfs2/dlm/dlmconvert.c b/fs/ocfs2/dlm/dlmconvert.c index 450d46eefab3..8a828448ba95 100644 --- a/fs/ocfs2/dlm/dlmconvert.c +++ b/fs/ocfs2/dlm/dlmconvert.c @@ -534,15 +534,19 @@ int dlm_convert_lock_handler(struct o2net_msg *msg, u32 len, void *data, } leave: - if (lock) - dlm_lock_put(lock); - /* either queue the ast or release it, if reserved */ if (call_ast) dlm_queue_ast(dlm, lock); else if (ast_reserved) dlm_lockres_release_ast(dlm, res); + /* + * Keep the lock alive through AST enqueue: a concurrent + * dlm_free_dead_locks() drops the list and base references. + */ + if (lock) + dlm_lock_put(lock); + if (kick_thread) dlm_kick_thread(dlm, res); diff --git a/fs/ocfs2/dlm/dlmdebug.c b/fs/ocfs2/dlm/dlmdebug.c index a91f167ed683..a064591801e2 100644 --- a/fs/ocfs2/dlm/dlmdebug.c +++ b/fs/ocfs2/dlm/dlmdebug.c @@ -7,6 +7,7 @@ * Copyright (C) 2004, 2008 Oracle. All rights reserved. */ +#include #include #include #include @@ -339,6 +340,7 @@ static int debug_purgelist_open(struct inode *inode, struct file *file) } static const struct file_operations debug_purgelist_fops = { + .owner = THIS_MODULE, .open = debug_purgelist_open, .release = debug_release, .read = debug_read, @@ -394,6 +396,7 @@ static int debug_mle_open(struct inode *inode, struct file *file) } static const struct file_operations debug_mle_fops = { + .owner = THIS_MODULE, .open = debug_mle_open, .release = debug_release, .read = debug_read, @@ -601,6 +604,7 @@ static int debug_lockres_release(struct inode *inode, struct file *file) } static const struct file_operations debug_lockres_fops = { + .owner = THIS_MODULE, .open = debug_lockres_open, .release = debug_lockres_release, .read = seq_read, @@ -794,6 +798,7 @@ static int debug_state_open(struct inode *inode, struct file *file) } static const struct file_operations debug_state_fops = { + .owner = THIS_MODULE, .open = debug_state_open, .release = debug_release, .read = debug_read, diff --git a/fs/ocfs2/dlm/dlmdomain.c b/fs/ocfs2/dlm/dlmdomain.c index 97bb9400e24b..a7dc3da6dfc7 100644 --- a/fs/ocfs2/dlm/dlmdomain.c +++ b/fs/ocfs2/dlm/dlmdomain.c @@ -2215,7 +2215,8 @@ static LIST_HEAD(dlm_join_handlers); static void dlm_unregister_net_handlers(void) { - o2net_unregister_handler_list(&dlm_join_handlers); + if (!list_empty(&dlm_join_handlers)) + o2net_unregister_and_flush_handler_list(&dlm_join_handlers); } static int dlm_register_net_handlers(void) diff --git a/fs/ocfs2/dlm/dlmlock.c b/fs/ocfs2/dlm/dlmlock.c index 46120eb75d90..660237c0d7cd 100644 --- a/fs/ocfs2/dlm/dlmlock.c +++ b/fs/ocfs2/dlm/dlmlock.c @@ -138,6 +138,8 @@ static enum dlm_status dlmlock_master(struct dlm_ctxt *dlm, res->lockname.len)) { kick_thread = 1; call_ast = 1; + /* Keep the lock alive through AST enqueue. */ + dlm_lock_get(lock); } else { mlog(0, "%s: returning DLM_NORMAL to " "node %u for reco lock\n", dlm->name, @@ -166,10 +168,12 @@ static enum dlm_status dlmlock_master(struct dlm_ctxt *dlm, wake_up(&res->wq); /* either queue the ast or release it */ - if (call_ast) + if (call_ast) { dlm_queue_ast(dlm, lock); - else + dlm_lock_put(lock); + } else { dlm_lockres_release_ast(dlm, res); + } dlm_lockres_calc_usage(dlm, res); if (kick_thread) diff --git a/fs/ocfs2/dlmglue.c b/fs/ocfs2/dlmglue.c index a23dd8f86c89..54e8f81119fb 100644 --- a/fs/ocfs2/dlmglue.c +++ b/fs/ocfs2/dlmglue.c @@ -7,6 +7,7 @@ * Copyright (C) 2003, 2004 Oracle. All rights reserved. */ +#include #include #include #include @@ -3311,6 +3312,7 @@ static int ocfs2_dlm_debug_open(struct inode *inode, struct file *file) } static const struct file_operations ocfs2_dlm_debug_fops = { + .owner = THIS_MODULE, .open = ocfs2_dlm_debug_open, .release = ocfs2_dlm_debug_release, .read = seq_read, diff --git a/fs/ocfs2/file.c b/fs/ocfs2/file.c index d6e977ba6565..c6e7606ff9e1 100644 --- a/fs/ocfs2/file.c +++ b/fs/ocfs2/file.c @@ -85,10 +85,17 @@ static int ocfs2_file_open(struct inode *inode, struct file *file) int mode = file->f_flags; struct ocfs2_inode_info *oi = OCFS2_I(inode); - trace_ocfs2_file_open(inode, file, file->f_path.dentry, - (unsigned long long)oi->ip_blkno, - file->f_path.dentry->d_name.len, - file->f_path.dentry->d_name.name, mode); + if (trace_ocfs2_file_open_enabled()) { + struct dentry *dentry = file->f_path.dentry; + struct name_snapshot name; + + take_dentry_name_snapshot(&name, dentry); + trace_ocfs2_file_open(inode, file, dentry, + oi->ip_blkno, + name.name.len, name.name.name, + mode); + release_dentry_name_snapshot(&name); + } if (file->f_mode & FMODE_WRITE) { status = dquot_initialize(inode); @@ -174,11 +181,17 @@ static int ocfs2_sync_file(struct file *file, loff_t start, loff_t end, tid_t commit_tid; bool needs_barrier = false; - trace_ocfs2_sync_file(inode, file, file->f_path.dentry, - oi->ip_blkno, - file->f_path.dentry->d_name.len, - file->f_path.dentry->d_name.name, - (unsigned long long)datasync); + if (trace_ocfs2_sync_file_enabled()) { + struct dentry *dentry = file->f_path.dentry; + struct name_snapshot name; + + take_dentry_name_snapshot(&name, dentry); + trace_ocfs2_sync_file(inode, file, dentry, + oi->ip_blkno, + name.name.len, name.name.name, + (unsigned long long)datasync); + release_dentry_name_snapshot(&name); + } if (unlikely(ocfs2_emergency_state(osb))) return -EROFS; @@ -1132,15 +1145,19 @@ int ocfs2_setattr(struct mnt_idmap *idmap, struct dentry *dentry, int had_lock; struct ocfs2_lock_holder oh; - trace_ocfs2_setattr(inode, dentry, - (unsigned long long)OCFS2_I(inode)->ip_blkno, - dentry->d_name.len, dentry->d_name.name, - attr->ia_valid, - attr->ia_valid & ATTR_MODE ? attr->ia_mode : 0, - attr->ia_valid & ATTR_UID ? + if (trace_ocfs2_setattr_enabled()) { + struct name_snapshot name; + + take_dentry_name_snapshot(&name, dentry); + trace_ocfs2_setattr(inode, dentry, OCFS2_I(inode)->ip_blkno, + name.name.len, name.name.name, attr->ia_valid, + attr->ia_valid & ATTR_MODE ? attr->ia_mode : 0, + attr->ia_valid & ATTR_UID ? from_kuid(&init_user_ns, attr->ia_uid) : 0, - attr->ia_valid & ATTR_GID ? + attr->ia_valid & ATTR_GID ? from_kgid(&init_user_ns, attr->ia_gid) : 0); + release_dentry_name_snapshot(&name); + } status = ocfs2_emergency_state(osb); if (unlikely(status)) { @@ -2392,11 +2409,17 @@ static ssize_t ocfs2_file_write_iter(struct kiocb *iocb, int direct_io = iocb->ki_flags & IOCB_DIRECT ? 1 : 0; int nowait = iocb->ki_flags & IOCB_NOWAIT ? 1 : 0; - trace_ocfs2_file_write_iter(inode, file, file->f_path.dentry, - (unsigned long long)OCFS2_I(inode)->ip_blkno, - file->f_path.dentry->d_name.len, - file->f_path.dentry->d_name.name, - (unsigned int)from->nr_segs); /* GRRRRR */ + if (trace_ocfs2_file_write_iter_enabled()) { + struct dentry *dentry = file->f_path.dentry; + struct name_snapshot name; + + take_dentry_name_snapshot(&name, dentry); + trace_ocfs2_file_write_iter(inode, file, dentry, + OCFS2_I(inode)->ip_blkno, + name.name.len, name.name.name, + (unsigned int)from->nr_segs); + release_dentry_name_snapshot(&name); + } if (!direct_io && nowait) return -EOPNOTSUPP; @@ -2542,17 +2565,16 @@ static ssize_t ocfs2_file_read_iter(struct kiocb *iocb, int direct_io = iocb->ki_flags & IOCB_DIRECT ? 1 : 0; int nowait = iocb->ki_flags & IOCB_NOWAIT ? 1 : 0; - trace_ocfs2_file_read_iter(inode, filp, filp->f_path.dentry, - (unsigned long long)OCFS2_I(inode)->ip_blkno, - filp->f_path.dentry->d_name.len, - filp->f_path.dentry->d_name.name, - to->nr_segs); /* GRRRRR */ - + if (trace_ocfs2_file_read_iter_enabled()) { + struct dentry *dentry = filp->f_path.dentry; + struct name_snapshot name; - if (!inode) { - ret = -EINVAL; - mlog_errno(ret); - goto bail; + take_dentry_name_snapshot(&name, dentry); + trace_ocfs2_file_read_iter(inode, filp, dentry, + OCFS2_I(inode)->ip_blkno, + name.name.len, name.name.name, + to->nr_segs); + release_dentry_name_snapshot(&name); } if (!direct_io && nowait) @@ -2624,11 +2646,17 @@ static ssize_t ocfs2_file_splice_read(struct file *in, loff_t *ppos, ssize_t ret = 0; int lock_level = 0; - trace_ocfs2_file_splice_read(inode, in, in->f_path.dentry, - (unsigned long long)OCFS2_I(inode)->ip_blkno, - in->f_path.dentry->d_name.len, - in->f_path.dentry->d_name.name, - flags); + if (trace_ocfs2_file_splice_read_enabled()) { + struct dentry *dentry = in->f_path.dentry; + struct name_snapshot name; + + take_dentry_name_snapshot(&name, dentry); + trace_ocfs2_file_splice_read(inode, in, dentry, + OCFS2_I(inode)->ip_blkno, + name.name.len, name.name.name, + flags); + release_dentry_name_snapshot(&name); + } /* * We're fine letting folks race truncates and extending writes with diff --git a/fs/ocfs2/filecheck.c b/fs/ocfs2/filecheck.c index be713d7d24a7..75334e4bd082 100644 --- a/fs/ocfs2/filecheck.c +++ b/fs/ocfs2/filecheck.c @@ -179,17 +179,20 @@ int ocfs2_filecheck_create_sysfs(struct ocfs2_super *osb) fcheck->fc_size = 0; fcheck->fc_done = 0; + /* Publish the initialized owner before activating its sysfs callbacks. */ + entry->fs_fcheck = fcheck; entry->fs_kobj.kset = osb->osb_dev_kset; init_completion(&entry->fs_kobj_unregister); ret = kobject_init_and_add(&entry->fs_kobj, &ocfs2_ktype_filecheck, NULL, "filecheck"); if (ret) { + /* Mount failure still calls ocfs2_filecheck_remove_sysfs(). */ + entry->fs_fcheck = NULL; kobject_put(&entry->fs_kobj); kfree(fcheck); return ret; } - entry->fs_fcheck = fcheck; return 0; } diff --git a/fs/ocfs2/quota_local.c b/fs/ocfs2/quota_local.c index f55810c59b1b..28c1c9f9403c 100644 --- a/fs/ocfs2/quota_local.c +++ b/fs/ocfs2/quota_local.c @@ -816,6 +816,9 @@ static int ocfs2_local_free_info(struct super_block *sb, int type) int mark_clean = 1, len; int status = 0; + /* Quota-core error paths can bypass ocfs2_disable_quotas(). */ + cancel_delayed_work_sync(&oinfo->dqi_sync_work); + iput(oinfo->dqi_gqinode); ocfs2_simple_drop_lockres(OCFS2_SB(sb), &oinfo->dqi_gqlock); ocfs2_lock_res_free(&oinfo->dqi_gqlock); @@ -1071,10 +1074,13 @@ static struct ocfs2_quota_chunk *ocfs2_local_quota_add_chunk( goto out; } + if (list_empty(&oinfo->dqi_chunk)) + chunk->qc_num = 0; + else + chunk->qc_num = list_entry(oinfo->dqi_chunk.prev, + struct ocfs2_quota_chunk, + qc_chunk)->qc_num + 1; list_add_tail(&chunk->qc_chunk, &oinfo->dqi_chunk); - chunk->qc_num = list_entry(chunk->qc_chunk.prev, - struct ocfs2_quota_chunk, - qc_chunk)->qc_num + 1; chunk->qc_headerbh = bh; *offset = 0; return chunk; diff --git a/fs/ocfs2/refcounttree.c b/fs/ocfs2/refcounttree.c index d9f22b4a2654..790e788ca64b 100644 --- a/fs/ocfs2/refcounttree.c +++ b/fs/ocfs2/refcounttree.c @@ -348,8 +348,10 @@ ocfs2_allocate_refcount_tree(struct ocfs2_super *osb, u64 rf_blkno) return new; } +/* Callers without the refcount tree lock must request a reference. */ static int ocfs2_get_refcount_tree(struct ocfs2_super *osb, u64 rf_blkno, - struct ocfs2_refcount_tree **ret_tree) + struct ocfs2_refcount_tree **ret_tree, + bool get_ref) { int ret = 0; struct ocfs2_refcount_tree *tree, *new = NULL; @@ -406,6 +408,8 @@ static int ocfs2_get_refcount_tree(struct ocfs2_super *osb, u64 rf_blkno, new = NULL; out: + if (get_ref) + ocfs2_refcount_tree_get(tree); *ret_tree = tree; osb->osb_ref_tree_lru = tree; @@ -479,14 +483,12 @@ int ocfs2_lock_refcount_tree(struct ocfs2_super *osb, struct ocfs2_refcount_block *rb; again: - ret = ocfs2_get_refcount_tree(osb, ref_blkno, &tree); + ret = ocfs2_get_refcount_tree(osb, ref_blkno, &tree, true); if (ret) { mlog_errno(ret); return ret; } - ocfs2_refcount_tree_get(tree); - ret = __ocfs2_lock_refcount_tree(osb, tree, rw); if (ret) { mlog_errno(ret); @@ -2328,7 +2330,8 @@ int ocfs2_decrease_refcount(struct inode *inode, goto out; } - ret = ocfs2_get_refcount_tree(OCFS2_SB(inode->i_sb), ref_blkno, &tree); + ret = ocfs2_get_refcount_tree(OCFS2_SB(inode->i_sb), ref_blkno, + &tree, false); if (ret) { mlog_errno(ret); goto out; @@ -2563,7 +2566,7 @@ int ocfs2_prepare_refcount_change_for_del(struct inode *inode, BUG_ON(!ocfs2_is_refcount_inode(inode)); ret = ocfs2_get_refcount_tree(OCFS2_SB(inode->i_sb), - refcount_loc, &tree); + refcount_loc, &tree, false); if (ret) { mlog_errno(ret); goto out; diff --git a/fs/ocfs2/super.c b/fs/ocfs2/super.c index c62e389d4dd6..acce465f45b0 100644 --- a/fs/ocfs2/super.c +++ b/fs/ocfs2/super.c @@ -387,6 +387,7 @@ static ssize_t ocfs2_debug_read(struct file *file, char __user *buf, #endif /* CONFIG_DEBUG_FS */ static const struct file_operations ocfs2_osb_debug_fops = { + .owner = THIS_MODULE, .open = ocfs2_osb_debug_open, .release = ocfs2_debug_release, .read = ocfs2_debug_read, diff --git a/fs/overlayfs/copy_up.c b/fs/overlayfs/copy_up.c index e963701b4c87..c147ad155b79 100644 --- a/fs/overlayfs/copy_up.c +++ b/fs/overlayfs/copy_up.c @@ -72,6 +72,18 @@ static int ovl_copy_acl(struct ovl_fs *ofs, const struct path *path, return err; } +static ssize_t ovl_getxattr_value(const struct path *path, char *name, + char **value, ssize_t *value_size); + +/* + * Any user can store an empty security.capability, which fails to read with + * -EINVAL. It grants no capability, so treat it as none, like no xattr. + */ +static bool ovl_no_fscap(ssize_t err) +{ + return err == -ENODATA || err == -EOPNOTSUPP || err == -EINVAL; +} + int ovl_copy_xattr(struct super_block *sb, const struct path *oldpath, struct dentry *new) { struct dentry *old = oldpath->dentry; @@ -129,30 +141,16 @@ int ovl_copy_xattr(struct super_block *sb, const struct path *oldpath, struct de break; } -retry: - size = ovl_do_getxattr(oldpath, name, value, value_size); - if (size == -ERANGE) - size = ovl_do_getxattr(oldpath, name, NULL, 0); - + size = ovl_getxattr_value(oldpath, name, &value, &value_size); if (size < 0) { + if (!strcmp(name, XATTR_NAME_CAPS) && ovl_no_fscap(size)) { + error = 0; + continue; + } error = size; break; } - if (size > value_size) { - void *new; - - new = kvmalloc(size, GFP_KERNEL); - if (!new) { - error = -ENOMEM; - break; - } - kvfree(value); - value = new; - value_size = size; - goto retry; - } - error = ovl_do_setxattr(OVL_FS(sb), new, name, value, size, 0); if (error) { if (error != -EOPNOTSUPP || ovl_must_copy_xattr(name)) @@ -476,8 +474,12 @@ struct ovl_fh *ovl_get_origin_fh(struct ovl_fs *ofs, struct dentry *origin) * When lower layer doesn't support export operations store a 'null' fh, * so we can use the overlay.origin xattr to distignuish between a copy * up and a pure upper inode. + * + * With index enabled, mount already found that all layers can decode + * file handles. Do not ask again, because an LSM may change its answer + * about CAP_DAC_READ_SEARCH and index users cannot cope with a 'null' fh. */ - if (!ovl_can_decode_fh(origin->d_sb)) + if (!ofs->config.index && !ovl_can_decode_fh(origin->d_sb)) return NULL; return ovl_encode_real_fh(ofs, d_inode(origin), false); @@ -808,9 +810,10 @@ static int ovl_copy_up_workdir(struct ovl_copy_up_ctx *c) rd.old_parent = c->workdir; rd.new_parent = c->destdir; rd.flags = 0; - err = start_renaming_dentry(&rd, 0, temp, + /* The name may have been taken since lookup, never replace it */ + err = start_renaming_dentry(&rd, LOOKUP_EXCL, temp, &QSTR_LEN(c->destname.name, c->destname.len)); - if (err) { + if (err && err != -EEXIST) { /* temp or workdir moved underneath us? map to -EIO */ err = -EIO; } @@ -834,9 +837,9 @@ static int ovl_copy_up_workdir(struct ovl_copy_up_ctx *c) if (!c->metacopy) ovl_set_upperdata(inode); - ovl_inode_update(inode, temp); if (S_ISDIR(inode->i_mode)) ovl_set_flag(OVL_WHITEOUTS, inode); + ovl_inode_update(inode, temp); out: ovl_end_write(c->dentry); @@ -935,6 +938,7 @@ static int ovl_do_copy_up(struct ovl_copy_up_ctx *c) struct ovl_fs *ofs = OVL_FS(c->dentry->d_sb); struct dentry *origin = c->lowerpath.dentry; struct ovl_fh *fh = NULL; + struct name_snapshot name; bool to_index = false; /* @@ -972,12 +976,15 @@ static int ovl_do_copy_up(struct ovl_copy_up_ctx *c) goto out_free_fh; } else { /* - * c->dentry->d_name is stabilzed by ovl_copy_up_start(), - * because if we got here, it means that c->dentry has no upper - * alias and changing ->d_name means going through ovl_rename() - * that will call ovl_copy_up() on source and target dentry. + * ovl_copy_up_start() keeps ovl_rename() away from c->dentry, + * because ovl_rename() calls ovl_copy_up() on source and target + * dentry. It does not stop d_splice_alias() from moving a + * directory that a lookup found under another name after a + * change to the lower layer. That move frees an external name, + * so take a snapshot. */ - c->destname = c->dentry->d_name; + take_dentry_name_snapshot(&name, c->dentry); + c->destname = name.name; /* * Mark parent "impure" because it may now contain non-pure * upper @@ -986,7 +993,7 @@ static int ovl_do_copy_up(struct ovl_copy_up_ctx *c) err = ovl_set_impure(c->parent, c->destdir); ovl_end_write(c->dentry); if (err) - goto out_free_fh; + goto out; } /* Should we copyup with O_TMPFILE or with workdir? */ @@ -1020,6 +1027,8 @@ static int ovl_do_copy_up(struct ovl_copy_up_ctx *c) out: if (to_index) kfree(c->destname.name); + else + release_dentry_name_snapshot(&name); out_free_fh: kfree(fh); return err; @@ -1055,26 +1064,39 @@ static bool ovl_need_meta_copy_up(struct dentry *dentry, umode_t mode, return true; } -static ssize_t ovl_getxattr_value(const struct path *path, char *name, char **value) +static ssize_t ovl_getxattr_value(const struct path *path, char *name, + char **value, ssize_t *value_size) { ssize_t res; char *buf; - res = ovl_do_getxattr(path, name, NULL, 0); - if (res == -ENODATA || res == -EOPNOTSUPP) - res = 0; + res = ovl_do_getxattr(path, name, *value, *value_size); + if (res == -ERANGE) { + res = ovl_do_getxattr(path, name, NULL, 0); + /* Xattr changed underneath us? */ + if (res >= 0 && res <= *value_size) + return -EIO; + } - if (res > 0) { - buf = kzalloc(res, GFP_KERNEL); + if (res > *value_size) { + buf = kvmalloc(res, GFP_KERNEL); if (!buf) return -ENOMEM; res = ovl_do_getxattr(path, name, buf, res); - if (res < 0) - kfree(buf); - else - *value = buf; + if (res < 0) { + kvfree(buf); + /* Xattr changed underneath us? */ + if (res == -ERANGE) + return -EIO; + return res; + } + + kvfree(*value); + *value = buf; + *value_size = res; } + return res; } @@ -1085,7 +1107,7 @@ static int ovl_copy_up_meta_inode_data(struct ovl_copy_up_ctx *c) struct path upperpath; int err; char *capability = NULL; - ssize_t cap_size; + ssize_t cap_size = 0; ovl_path_upper(c->dentry, &upperpath); if (WARN_ON(upperpath.dentry == NULL)) @@ -1093,8 +1115,10 @@ static int ovl_copy_up_meta_inode_data(struct ovl_copy_up_ctx *c) if (c->stat.size) { err = cap_size = ovl_getxattr_value(&upperpath, XATTR_NAME_CAPS, - &capability); - if (cap_size < 0) + &capability, &cap_size); + if (ovl_no_fscap(cap_size)) + cap_size = 0; + else if (cap_size < 0) goto out; } @@ -1123,7 +1147,7 @@ static int ovl_copy_up_meta_inode_data(struct ovl_copy_up_ctx *c) ovl_clear_flag(OVL_VERIFIED_DIGEST, d_inode(c->dentry)); ovl_set_upperdata(d_inode(c->dentry)); out_free: - kfree(capability); + kvfree(capability); out: return err; } diff --git a/fs/overlayfs/dir.c b/fs/overlayfs/dir.c index 7beb0af26498..f716ac13badf 100644 --- a/fs/overlayfs/dir.c +++ b/fs/overlayfs/dir.c @@ -304,7 +304,6 @@ static int ovl_instantiate(struct dentry *dentry, struct inode *inode, }; ovl_dentry_set_upper_alias(dentry); - ovl_dentry_init_reval(dentry, newdentry, NULL); if (!hardlink) { /* @@ -335,6 +334,13 @@ static int ovl_instantiate(struct dentry *dentry, struct inode *inode, d_mark_tmpfile(tmpfile, inode); d_instantiate(dentry, inode); + /* + * Unless it is a tmpfile, the dentry is hashed and a path walk can find + * it at any time. Set the revalidate flags only now, because + * ovl_dentry_revalidate_common() does not expect them on a negative + * dentry. + */ + ovl_dentry_init_reval(dentry, ovl_dentry_upper(dentry), NULL); if (inode != oip.newinode) { pr_warn_ratelimited("newly created inode found in cache (%pd2)\n", dentry); @@ -1027,7 +1033,19 @@ static char *ovl_get_redirect(struct dentry *dentry, bool abs_redirect) const char *name; int thislen; + /* + * A rename can move d to another directory at any time. Hold + * d_lock to copy the name that d has in this very parent, or + * else the path mixes the old name with the new parent. + */ + tmp = dget_parent(d); spin_lock(&d->d_lock); + if (unlikely(tmp != d->d_parent)) { + spin_unlock(&d->d_lock); + dput(tmp); + continue; + } + name = ovl_dentry_get_redirect(d); if (name) { thislen = strlen(name); @@ -1040,13 +1058,13 @@ static char *ovl_get_redirect(struct dentry *dentry, bool abs_redirect) if (thislen + (name[0] != '/') > buflen) { ret = ERR_PTR(-EXDEV); spin_unlock(&d->d_lock); + dput(tmp); goto out_put; } buflen -= thislen; memcpy(&buf[buflen], name, thislen); spin_unlock(&d->d_lock); - tmp = dget_parent(d); dput(d); d = tmp; diff --git a/fs/overlayfs/inode.c b/fs/overlayfs/inode.c index 401cb8c75520..7e1ac70c5d97 100644 --- a/fs/overlayfs/inode.c +++ b/fs/overlayfs/inode.c @@ -999,7 +999,7 @@ int ovl_set_nlink_lower(struct dentry *dentry) } unsigned int ovl_get_nlink(struct ovl_fs *ofs, struct dentry *lowerdentry, - struct dentry *upperdentry, + struct dentry *upperdentry, bool indexed, unsigned int fallback) { int nlink_diff; @@ -1007,7 +1007,15 @@ unsigned int ovl_get_nlink(struct ovl_fs *ofs, struct dentry *lowerdentry, char buf[13]; int err; - if (!lowerdentry || !upperdentry || d_inode(lowerdentry)->i_nlink == 1) + if (!lowerdentry || !upperdentry) + return fallback; + + /* + * A lower inode with one link is not indexed on copy up, unless all + * files are indexed for NFS export. If the caller found an index entry, + * then the xattr holds the union nlink, whatever the lower nlink is. + */ + if (!indexed && d_inode(lowerdentry)->i_nlink == 1) return fallback; err = ovl_getxattr_upper(ofs, upperdentry, OVL_XATTR_NLINK, @@ -1264,7 +1272,7 @@ struct inode *ovl_get_inode(struct super_block *sb, /* Recalculate nlink for non-dir due to indexing */ if (!is_dir) nlink = ovl_get_nlink(ofs, lowerdentry, upperdentry, - nlink); + oip->index, nlink); set_nlink(inode, nlink); ino = key->i_ino; } else { @@ -1279,7 +1287,7 @@ struct inode *ovl_get_inode(struct super_block *sb, } ovl_fill_inode(inode, realinode->i_mode, realinode->i_rdev); ovl_inode_init(inode, oip, ino, fsid); - WARN_ON_ONCE(!!IS_CASEFOLDED(inode) != ofs->casefold); + WARN_ON_ONCE(is_dir && !!IS_CASEFOLDED(inode) != ofs->casefold); if (upperdentry && ovl_is_impuredir(sb, upperdentry)) ovl_set_flag(OVL_IMPURE, inode); diff --git a/fs/overlayfs/namei.c b/fs/overlayfs/namei.c index ca899fdfaafd..c6127edb9be9 100644 --- a/fs/overlayfs/namei.c +++ b/fs/overlayfs/namei.c @@ -266,7 +266,8 @@ static int ovl_lookup_single(struct dentry *base, struct ovl_lookup_data *d, goto out_err; } - if (ofs->casefold != ovl_dentry_casefolded(this)) { + /* Filesystems set the casefold flag only on directories */ + if (d_is_dir(this) && ofs->casefold != ovl_dentry_casefolded(this)) { warn = "child wrong casefold"; err = -EREMOTE; goto out_warn; @@ -712,7 +713,7 @@ int ovl_verify_index(struct ovl_fs *ofs, struct dentry *index) if (err) goto fail; - if (ovl_get_nlink(ofs, origin.dentry, index, 0) == 0) + if (ovl_get_nlink(ofs, origin.dentry, index, true, 0) == 0) goto orphan; } @@ -942,8 +943,10 @@ static int ovl_fix_origin(struct ovl_fs *ofs, struct dentry *dentry, goto out; err = ovl_set_origin_fh(ofs, fh, upper); - if (!err) - err = ovl_set_impure(dentry->d_parent, upper->d_parent); + if (!err) { + /* upper->d_parent is not stable, use the overlay parent */ + err = ovl_set_impure_dir(dentry->d_parent); + } ovl_drop_write(dentry); out: diff --git a/fs/overlayfs/overlayfs.h b/fs/overlayfs/overlayfs.h index 7f3558372c59..91cdd7e15696 100644 --- a/fs/overlayfs/overlayfs.h +++ b/fs/overlayfs/overlayfs.h @@ -572,6 +572,12 @@ int ovl_check_setxattr(struct ovl_fs *ofs, struct dentry *upperdentry, enum ovl_xattr ox, const void *value, size_t size, int xerr); int ovl_set_impure(struct dentry *dentry, struct dentry *upperdentry); + +static inline int ovl_set_impure_dir(struct dentry *dentry) +{ + return ovl_set_impure(dentry, ovl_dentry_upper(dentry)); +} + bool ovl_inuse_trylock(struct dentry *dentry); void ovl_inuse_unlock(struct dentry *dentry); bool ovl_is_inuse(struct dentry *dentry); @@ -802,7 +808,7 @@ static inline bool ovl_dir_is_real(struct inode *dir) int ovl_set_nlink_upper(struct dentry *dentry); int ovl_set_nlink_lower(struct dentry *dentry); unsigned int ovl_get_nlink(struct ovl_fs *ofs, struct dentry *lowerdentry, - struct dentry *upperdentry, + struct dentry *upperdentry, bool indexed, unsigned int fallback); int ovl_permission(struct mnt_idmap *idmap, struct inode *inode, int mask); diff --git a/fs/overlayfs/readdir.c b/fs/overlayfs/readdir.c index 7d6f7f6022eb..80366424b3dc 100644 --- a/fs/overlayfs/readdir.c +++ b/fs/overlayfs/readdir.c @@ -81,13 +81,13 @@ static int ovl_casefold(struct ovl_readdir_data *rdd, const char *str, int len, name_is_dot_dotdot(str, len)) return 0; - cf_name = kmalloc(NAME_MAX, GFP_KERNEL); + cf_name = kmalloc(NAME_MAX + 1, GFP_KERNEL); if (!cf_name) { rdd->err = -ENOMEM; return -ENOMEM; } - cf_len = utf8_casefold(rdd->map, &qstr, cf_name, NAME_MAX); + cf_len = utf8_casefold(rdd->map, &qstr, cf_name, NAME_MAX + 1); if (cf_len > 0) *dst = cf_name; else @@ -568,8 +568,8 @@ static int ovl_cache_update(const struct path *path, struct ovl_cache_entry *p, goto get; } if (p->len == 2) { - /* we shall not be moved */ - this = dget(dir->d_parent); + /* d_splice_alias() can move even a locked dir */ + this = dget_parent(dir); goto get; } } @@ -797,6 +797,7 @@ static int ovl_iterate_real(struct file *file, struct dir_context *ctx) struct dentry *dir = file->f_path.dentry; struct ovl_fs *ofs = OVL_FS(dir->d_sb); const struct ovl_layer *lower_layer = ovl_layer_lower(dir); + struct dentry *parent; struct ovl_readdir_translate rdt = { .ctx.actor = ovl_fill_real, .ctx.count = ctx->count, @@ -808,18 +809,26 @@ static int ovl_iterate_real(struct file *file, struct dir_context *ctx) if (rdt.xinobits && lower_layer) rdt.fsid = lower_layer->fsid; - if (OVL_TYPE_MERGE(ovl_path_type(dir->d_parent))) { + /* + * The lock on dir does not stop d_splice_alias() from moving it to + * another parent, which drops the reference dir holds on this one. + */ + parent = dget_parent(dir); + if (OVL_TYPE_MERGE(ovl_path_type(parent))) { struct kstat stat; struct path statpath = file->f_path; - statpath.dentry = dir->d_parent; + statpath.dentry = parent; err = vfs_getattr(&statpath, &stat, STATX_INO, 0); - if (err) + if (err) { + dput(parent); return err; + } WARN_ON_ONCE(dir->d_sb->s_dev != stat.dev); rdt.parent_ino = stat.ino; } + dput(parent); if (ovl_is_impure_dir(file)) { rdt.cache = ovl_cache_get_impure(&file->f_path); @@ -875,8 +884,10 @@ static bool ovl_need_adjust_d_ino(struct file *file) { struct dentry *dentry = file->f_path.dentry; struct ovl_fs *ofs = OVL_FS(dentry->d_sb); + struct dentry *parent; + enum ovl_path_type type; - /* If parent is merge, then need to adjust d_ino for '..' */ + /* With xino, need to map real d_ino to the unified overlay ino space */ if (ovl_xino_bits(ofs)) return true; @@ -885,8 +896,14 @@ static bool ovl_need_adjust_d_ino(struct file *file) return false; /* If dir is impure then need to adjust d_ino for copied up entries */ - if (ovl_is_impure_dir(file) || - OVL_TYPE_MERGE(ovl_path_type(dentry->d_parent))) + if (ovl_is_impure_dir(file)) + return true; + + /* If parent is merge, then need to adjust d_ino for '..' */ + parent = dget_parent(dentry); + type = ovl_path_type(parent); + dput(parent); + if (OVL_TYPE_MERGE(type)) return true; /* Pure: no need to adjust d_ino */ @@ -1055,7 +1072,8 @@ static int ovl_dir_open(struct inode *inode, struct file *file) return PTR_ERR(realfile); } od->realfile = realfile; - od->is_real = ovl_dir_is_real(inode); + od->is_real = !OVL_TYPE_MERGE(type) && + !ovl_test_flag(OVL_WHITEOUTS, inode); od->is_upper = OVL_TYPE_UPPER(type); file->private_data = od; diff --git a/fs/overlayfs/super.c b/fs/overlayfs/super.c index bd0a3f9039d2..861b0775f9dd 100644 --- a/fs/overlayfs/super.c +++ b/fs/overlayfs/super.c @@ -497,6 +497,20 @@ static int ovl_report_in_use(struct ovl_fs *ofs, const char *name) } } +/* + * clone_private_mount() checks CAP_SYS_ADMIN in current->cred. + * ovl_fill_super_creds() runs under the stashed creator cred, which + * override_creds may have dropped. + * The task that issued FSCONFIG_CMD_CREATE is still current->real_cred, so + * use that only for the clone. creator_cred stays in effect for every other + * backing-fs operation. + */ +static struct vfsmount *ovl_clone_private_mount(const struct path *path) +{ + scoped_with_creds(current_real_cred()) + return clone_private_mount(path); +} + static int ovl_get_upper(struct super_block *sb, struct ovl_fs *ofs, struct ovl_layer *upper_layer, const struct path *upperpath) @@ -520,7 +534,7 @@ static int ovl_get_upper(struct super_block *sb, struct ovl_fs *ofs, if (err) goto out; - upper_mnt = clone_private_mount(upperpath); + upper_mnt = ovl_clone_private_mount(upperpath); err = PTR_ERR(upper_mnt); if (IS_ERR(upper_mnt)) { pr_err("failed to clone upperpath\n"); @@ -621,16 +635,21 @@ static int ovl_check_rename_whiteout(struct ovl_fs *ofs) static struct dentry *ovl_lookup_or_create(struct ovl_fs *ofs, struct dentry *parent, - const char *name, umode_t mode) + const char *name, bool is_dir) { struct dentry *child; child = ovl_start_creating_upper(ofs, parent, &QSTR(name)); if (!IS_ERR(child)) { - if (!child->d_inode) + if (!child->d_inode) { child = ovl_create_real(ofs, parent, child, &QSTR(name), - OVL_CATTR(mode)); + OVL_CATTR(is_dir ? S_IFDIR : + S_IFREG)); + } else if (is_dir && !d_can_lookup(child)) { + end_creating(child); + child = ERR_PTR(-ENOTDIR); + } end_creating_keep(child); } dput(parent); @@ -652,7 +671,7 @@ static int ovl_create_volatile_dirty(struct ovl_fs *ofs) const char *const *name = volatile_path; for (ctr = ARRAY_SIZE(volatile_path); ctr; ctr--, name++) { - d = ovl_lookup_or_create(ofs, d, *name, ctr > 1 ? S_IFDIR : S_IFREG); + d = ovl_lookup_or_create(ofs, d, *name, ctr > 1); if (IS_ERR(d)) return PTR_ERR(d); } @@ -1093,7 +1112,7 @@ static int ovl_get_layers(struct super_block *sb, struct ovl_fs *ofs, } } - mnt = clone_private_mount(&l->path); + mnt = ovl_clone_private_mount(&l->path); err = PTR_ERR(mnt); if (IS_ERR(mnt)) { pr_err("failed to clone lowerpath\n"); diff --git a/fs/proc/task_mmu.c b/fs/proc/task_mmu.c index 5c54aebe2118..dd025474b0fc 100644 --- a/fs/proc/task_mmu.c +++ b/fs/proc/task_mmu.c @@ -3231,15 +3231,18 @@ static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) size_t n_ranges_out = 0; int ret; + if (!mmget_not_zero(mm)) + return -ESRCH; + ret = pagemap_scan_get_args(&p.arg, uarg); if (ret) - return ret; + goto out_mmput; p.masks_of_interest = p.arg.category_mask | p.arg.category_anyof_mask | p.arg.return_mask; ret = pagemap_scan_init_bounce_buffer(&p); if (ret) - return ret; + goto out_mmput; for (walk_start = p.arg.start; walk_start < p.arg.end; walk_start = p.arg.walk_end) { @@ -3294,6 +3297,8 @@ static long do_pagemap_scan(struct mm_struct *mm, unsigned long uarg) ret = -EFAULT; kfree(p.vec_buf); +out_mmput: + mmput(mm); return ret; } diff --git a/fs/smb/client/cached_dir.c b/fs/smb/client/cached_dir.c index 647fa26da4d2..50217f47a0ed 100644 --- a/fs/smb/client/cached_dir.c +++ b/fs/smb/client/cached_dir.c @@ -649,6 +649,17 @@ void invalidate_all_cached_dirs(struct cifs_tcon *tcon, bool sync) flush_delayed_work(&cfids->laundromat_work); } +static bool cached_dir_get_tcon(struct cifs_tcon *tcon, + enum smb3_tcon_ref_trace trace) +{ + guard(spinlock)(&tcon->tc_lock); + if (tcon->status == TID_EXITING || tcon->tc_count <= 0) + return false; + ++tcon->tc_count; + trace_smb3_tcon_ref(tcon->debug_id, tcon->tc_count, trace); + return true; +} + static void cached_dir_offload_close(struct work_struct *work) { @@ -693,6 +704,9 @@ bool cached_dir_lease_break(struct cifs_tcon *tcon, __u8 lease_key[16]) !memcmp(lease_key, cfid->fid.lease_key, SMB2_LEASE_KEY_SIZE)) { + if (!cached_dir_get_tcon(tcon, + netfs_trace_tcon_ref_get_cached_lease_break)) + break; cfid->has_lease = false; cfid->time = 0; /* @@ -703,9 +717,6 @@ bool cached_dir_lease_break(struct cifs_tcon *tcon, __u8 lease_key[16]) cfid->on_list = false; cfids->num_entries--; - ++tcon->tc_count; - trace_smb3_tcon_ref(tcon->debug_id, tcon->tc_count, - netfs_trace_tcon_ref_get_cached_lease_break); queue_work(cfid_put_wq, &cfid->put_work); spin_unlock(&cfids->cfid_list_lock); return true; @@ -813,12 +824,13 @@ static void cfids_laundromat_worker(struct work_struct *work) cfid->dentry = NULL; if (cfid->is_open) { - spin_lock(&cfid->tcon->tc_lock); - ++cfid->tcon->tc_count; - trace_smb3_tcon_ref(cfid->tcon->debug_id, cfid->tcon->tc_count, - netfs_trace_tcon_ref_get_cached_laundromat); - spin_unlock(&cfid->tcon->tc_lock); - queue_work(serverclose_wq, &cfid->close_work); + if (cached_dir_get_tcon(cfid->tcon, + netfs_trace_tcon_ref_get_cached_laundromat)) { + queue_work(serverclose_wq, &cfid->close_work); + continue; + } + scoped_guard(spinlock, &cfids->cfid_list_lock) + list_add_tail(&cfid->entry, &cfids->dying); } else /* * Drop the ref-count from above, either the lease-ref (if there diff --git a/fs/smb/client/cifsacl.c b/fs/smb/client/cifsacl.c index c5e47a835f99..e37cf5043f8f 100644 --- a/fs/smb/client/cifsacl.c +++ b/fs/smb/client/cifsacl.c @@ -1826,12 +1826,15 @@ id_mode_to_cifs_acl(struct inode *inode, const char *path, __u64 *pnmode, /* Potentially, five new ACEs can be added to the ACL for U,G,O mapping */ if (pnmode && *pnmode != NO_CHANGE_64) { /* chmod */ + nsecdesclen = sizeof(struct smb_acl) + + 2 * sizeof(struct smb_sid); if (posix) - nsecdesclen = 1 * sizeof(struct smb_ace); + nsecdesclen += sizeof(struct smb_ntsd) + + sizeof(struct smb_ace); else if (mode_from_sid) - nsecdesclen = secdesclen + (2 * sizeof(struct smb_ace)); + nsecdesclen += secdesclen + 2 * sizeof(struct smb_ace); else /* cifsacl */ - nsecdesclen = secdesclen + (5 * sizeof(struct smb_ace)); + nsecdesclen += secdesclen + 5 * sizeof(struct smb_ace); } else { /* chown */ /* When ownership changes, changes new owner sid length could be different */ nsecdesclen = sizeof(struct smb_ntsd) + (sizeof(struct smb_sid) * 2); diff --git a/fs/smb/client/cifsfs.c b/fs/smb/client/cifsfs.c index 98b610b2a114..c1f99d7007b1 100644 --- a/fs/smb/client/cifsfs.c +++ b/fs/smb/client/cifsfs.c @@ -2021,7 +2021,6 @@ init_cifs(void) if (rc) return rc; - cifs_proc_init(); INIT_LIST_HEAD(&cifs_tcp_ses_list); /* * Initialize Global counters @@ -2070,10 +2069,8 @@ init_cifs(void) cifsiod_wq = alloc_workqueue("cifsiod", WQ_FREEZABLE | WQ_MEM_RECLAIM | WQ_PERCPU, 0); - if (!cifsiod_wq) { - rc = -ENOMEM; - goto out_clean_proc; - } + if (!cifsiod_wq) + return -ENOMEM; /* * Consider in future setting limit!=0 maybe to min(num_of_cores - 1, 3) @@ -2164,19 +2161,22 @@ init_cifs(void) if (rc) goto out_cifs_swn_init; + cifs_proc_init(); + rc = register_filesystem(&cifs_fs_type); if (rc) - goto out_init_cifs_idmap; + goto out_clean_proc; rc = register_filesystem(&smb3_fs_type); if (rc) { unregister_filesystem(&cifs_fs_type); - goto out_init_cifs_idmap; + goto out_clean_proc; } return 0; -out_init_cifs_idmap: +out_clean_proc: + cifs_proc_clean(); exit_cifs_idmap(); out_cifs_swn_init: #ifdef CONFIG_CIFS_SWN_UPCALL @@ -2212,8 +2212,6 @@ init_cifs(void) destroy_workqueue(decrypt_wq); out_destroy_cifsiod_wq: destroy_workqueue(cifsiod_wq); -out_clean_proc: - cifs_proc_clean(); return rc; } @@ -2223,6 +2221,7 @@ exit_cifs(void) cifs_dbg(NOISY, "exit_smb3\n"); unregister_filesystem(&cifs_fs_type); unregister_filesystem(&smb3_fs_type); + cifs_proc_clean(); cifs_release_automount_timer(); exit_cifs_idmap(); #ifdef CONFIG_CIFS_SWN_UPCALL @@ -2245,7 +2244,6 @@ exit_cifs(void) destroy_workqueue(serverclose_wq); destroy_workqueue(cfid_put_wq); destroy_workqueue(cifsiod_wq); - cifs_proc_clean(); } MODULE_AUTHOR("Steve French"); diff --git a/fs/smb/client/cifsglob.h b/fs/smb/client/cifsglob.h index 5c5b76a9e9fe..92174c4483a2 100644 --- a/fs/smb/client/cifsglob.h +++ b/fs/smb/client/cifsglob.h @@ -16,6 +16,7 @@ #include #include #include +#include #include #include #include @@ -679,6 +680,7 @@ struct TCP_Server_Info { spinlock_t srv_lock; /* protect anything here that is not protected */ __u64 conn_id; /* connection identifier (useful for debugging) */ int srv_count; /* reference counter */ + refcount_t obj_refs; /* struct lifetime: srv_count users + cifsd */ int rfc1001_sessinit; /* whether to estasblish netbios session */ bool with_rfc1001; /* if netbios session is used */ /* 15 character server name + 0x20 16th byte indicating type = srv */ diff --git a/fs/smb/client/cifsproto.h b/fs/smb/client/cifsproto.h index 00168839c123..2a8f830dff08 100644 --- a/fs/smb/client/cifsproto.h +++ b/fs/smb/client/cifsproto.h @@ -131,6 +131,7 @@ void cifs_signal_cifsd_for_reconnect(struct TCP_Server_Info *server, void cifs_mark_tcp_ses_conns_for_reconnect(struct TCP_Server_Info *server, bool mark_smb_session); int cifs_reconnect(struct TCP_Server_Info *server, bool mark_smb_session); +void smb2_add_credits_from_hdr(char *buffer, struct TCP_Server_Info *server); bool backup_cred(struct cifs_sb_info *cifs_sb); bool is_size_safe_to_change(struct cifsInodeInfo *cifsInode, __u64 end_of_file, bool from_readdir); diff --git a/fs/smb/client/cifssmb.c b/fs/smb/client/cifssmb.c index 6dddbd84b93b..8283a7ac44c0 100644 --- a/fs/smb/client/cifssmb.c +++ b/fs/smb/client/cifssmb.c @@ -2380,6 +2380,7 @@ CIFSSMBPosixLock(const unsigned int xid, struct cifs_tcon *tcon, cifs_dbg(FYI, "Send error in Posix Lock = %d\n", rc); } else if (pLockData) { /* lock structure can be returned on get */ + unsigned int rsp_len; __u16 data_offset; __u16 data_count; rc = validate_t2((struct smb_t2_rsp *)pSMBr); @@ -2391,11 +2392,18 @@ CIFSSMBPosixLock(const unsigned int xid, struct cifs_tcon *tcon, } data_offset = le16_to_cpu(pSMBr->t2.DataOffset); data_count = le16_to_cpu(pSMBr->t2.DataCount); + rsp_len = smbCalcSize(&pSMBr->hdr); if (data_count < sizeof(struct cifs_posix_lock)) { rc = smb_EIO2(smb_eio_trace_lock_data_too_small, data_count, sizeof(struct cifs_posix_lock)); goto plk_err_exit; } + if (data_offset > rsp_len || + data_count > rsp_len - data_offset) { + rc = smb_EIO2(smb_eio_trace_lock_data_area, + data_offset, data_count); + goto plk_err_exit; + } parm_data = (struct cifs_posix_lock *) ((char *)&pSMBr->hdr.Protocol + data_offset); if (parm_data->lock_type == cpu_to_le16(CIFS_UNLCK)) @@ -3791,9 +3799,10 @@ static int validate_ntransact(char *buf, char **ppparm, char **ppdata, __u32 *pparmlen, __u32 *pdatalen) { - char *end_of_smb; __u32 data_count, data_offset, parm_count, parm_offset; struct smb_com_ntransact_rsp *pSMBr; + size_t buflen; + char *end_buf; u16 bcc; *pdatalen = 0; @@ -3804,37 +3813,42 @@ validate_ntransact(char *buf, char **ppparm, char **ppdata, pSMBr = (struct smb_com_ntransact_rsp *)buf; + if (pSMBr->hdr.WordCount != 18) { + cifs_dbg(FYI, "invalid NT transaction WordCount %u\n", + pSMBr->hdr.WordCount); + return -EINVAL; + } + bcc = get_bcc(&pSMBr->hdr); - end_of_smb = 2 /* sizeof byte count */ + bcc + - (char *)&pSMBr->ByteCount; + end_buf = 2 /* sizeof byte count */ + bcc + (char *)&pSMBr->ByteCount; + buflen = end_buf - buf; data_offset = le32_to_cpu(pSMBr->DataOffset); data_count = le32_to_cpu(pSMBr->DataCount); parm_offset = le32_to_cpu(pSMBr->ParameterOffset); parm_count = le32_to_cpu(pSMBr->ParameterCount); - *ppparm = (char *)&pSMBr->hdr.Protocol + parm_offset; - *ppdata = (char *)&pSMBr->hdr.Protocol + data_offset; - /* should we also check that parm and data areas do not overlap? */ - if (*ppparm > end_of_smb) { + if (parm_offset > buflen) { cifs_dbg(FYI, "parms start after end of smb\n"); return -EINVAL; - } else if (parm_count + *ppparm > end_of_smb) { + } else if (parm_count > buflen - parm_offset) { cifs_dbg(FYI, "parm end after end of smb\n"); return -EINVAL; - } else if (*ppdata > end_of_smb) { + } else if (data_offset > buflen) { cifs_dbg(FYI, "data starts after end of smb\n"); return -EINVAL; - } else if (data_count + *ppdata > end_of_smb) { - cifs_dbg(FYI, "data %p + count %d (%p) past smb end %p start %p\n", - *ppdata, data_count, (data_count + *ppdata), - end_of_smb, pSMBr); + } else if (data_count > buflen - data_offset) { + cifs_dbg(FYI, "data offset %u + count %u past smb length %zu\n", + data_offset, data_count, buflen); return -EINVAL; } else if (parm_count + data_count > bcc) { cifs_dbg(FYI, "parm count and data count larger than SMB\n"); return -EINVAL; } + + *ppparm = buf + parm_offset; + *ppdata = buf + data_offset; *pdatalen = data_count; *pparmlen = parm_count; return 0; @@ -4408,10 +4422,11 @@ CIFSFindFirst(const unsigned int xid, struct cifs_tcon *tcon, struct cifs_search_info *psrch_inf, bool msearch) { /* level 257 SMB_ */ + unsigned int parm_off, parm_count, data_off, data_count; TRANSACTION2_FFIRST_REQ *pSMB = NULL; TRANSACTION2_FFIRST_RSP *pSMBr = NULL; - T2_FFIRST_RSP_PARMS *parms; struct nls_table *nls_codepage; + T2_FFIRST_RSP_PARMS *parms; unsigned int in_len, lnoff; __u16 params, byte_count; int bytes_returned = 0; @@ -4530,27 +4545,52 @@ CIFSFindFirst(const unsigned int xid, struct cifs_tcon *tcon, return rc; } + parm_off = le16_to_cpu(pSMBr->t2.ParameterOffset); + parm_count = le16_to_cpu(pSMBr->t2.ParameterCount); + if (parm_count < sizeof(*parms) || + parm_off + parm_count > (unsigned int)bytes_returned) { + cifs_dbg(FYI, "%s: bad parameter area offset %u count %u for response of %d\n", + __func__, parm_off, parm_count, bytes_returned); + rc = smb_EIO2(smb_eio_trace_ffirst_param_area, + parm_off, parm_count); + cifs_buf_release(pSMB); + return rc; + } + + data_off = le16_to_cpu(pSMBr->t2.DataOffset); + data_count = le16_to_cpu(pSMBr->t2.DataCount); + if (data_off + data_count > (unsigned int)bytes_returned) { + cifs_dbg(VFS, "%s: data area offset %u count %u past response length %d\n", + __func__, data_off, data_count, bytes_returned); + cifs_buf_release(pSMB); + return -EINVAL; + } + psrch_inf->unicode = !!(pSMBr->hdr.Flags2 & SMBFLG2_UNICODE); psrch_inf->ntwrk_buf_start = (char *)pSMBr; psrch_inf->smallBuf = false; - psrch_inf->srch_entries_start = (char *)&pSMBr->hdr.Protocol + - le16_to_cpu(pSMBr->t2.DataOffset); + psrch_inf->srch_entries_start = (char *)&pSMBr->hdr.Protocol + data_off; - parms = (T2_FFIRST_RSP_PARMS *)((char *)&pSMBr->hdr.Protocol + - le16_to_cpu(pSMBr->t2.ParameterOffset)); + parms = (T2_FFIRST_RSP_PARMS *)((char *)&pSMBr->hdr.Protocol + parm_off); psrch_inf->endOfSearch = !!parms->EndofSearch; psrch_inf->entries_in_buffer = le16_to_cpu(parms->SearchCount); psrch_inf->index_of_last_entry = 2 /* skip . and .. */ + psrch_inf->entries_in_buffer; + psrch_inf->presume_name = NULL; + psrch_inf->resume_name_len = 0; + psrch_inf->resume_key = 0; + if (pnetfid) + *pnetfid = parms->SearchHandle; lnoff = le16_to_cpu(parms->LastNameOffset); - if (CIFSMaxBufSize < lnoff) { + if (!psrch_inf->entries_in_buffer) { + psrch_inf->last_entry = NULL; + } else if (psrch_inf->srch_entries_start + lnoff >= + psrch_inf->ntwrk_buf_start + bytes_returned) { cifs_dbg(VFS, "ignoring corrupt resume name\n"); psrch_inf->last_entry = NULL; } else { psrch_inf->last_entry = psrch_inf->srch_entries_start + lnoff; - if (pnetfid) - *pnetfid = parms->SearchHandle; } return 0; } @@ -4559,10 +4599,11 @@ int CIFSFindNext(const unsigned int xid, struct cifs_tcon *tcon, __u16 searchHandle, __u16 search_flags, struct cifs_search_info *psrch_inf) { + unsigned int parm_off, parm_count, data_off, data_count; TRANSACTION2_FNEXT_REQ *pSMB = NULL; TRANSACTION2_FNEXT_RSP *pSMBr = NULL; - T2_FNEXT_RSP_PARMS *parms; unsigned int name_len, in_len; + T2_FNEXT_RSP_PARMS *parms; unsigned int lnoff; __u16 params, byte_count; char *response_data; @@ -4607,7 +4648,10 @@ int CIFSFindNext(const unsigned int xid, struct cifs_tcon *tcon, name_len = psrch_inf->resume_name_len; params += name_len; if (name_len < PATH_MAX) { - memcpy(pSMB->ResumeFileName, psrch_inf->presume_name, name_len); + if (name_len) { + memcpy(pSMB->ResumeFileName, + psrch_inf->presume_name, name_len); + } byte_count += name_len; /* 14 byte parm len above enough for 2 byte null terminator */ pSMB->ResumeFileName[name_len] = 0; @@ -4643,13 +4687,33 @@ int CIFSFindNext(const unsigned int xid, struct cifs_tcon *tcon, cifs_buf_release(pSMB); return rc; } + + parm_off = le16_to_cpu(pSMBr->t2.ParameterOffset); + parm_count = le16_to_cpu(pSMBr->t2.ParameterCount); + if (parm_count < sizeof(*parms) || + parm_off + parm_count > (unsigned int)bytes_returned) { + cifs_dbg(FYI, "%s: bad parameter area offset %u count %u for response of %d\n", + __func__, parm_off, parm_count, bytes_returned); + rc = smb_EIO2(smb_eio_trace_fnext_param_area, + parm_off, parm_count); + cifs_buf_release(pSMB); + return rc; + } + + data_off = le16_to_cpu(pSMBr->t2.DataOffset); + data_count = le16_to_cpu(pSMBr->t2.DataCount); + if (data_off + data_count > (unsigned int)bytes_returned) { + cifs_dbg(VFS, "%s: data area offset %u count %u past response length %d\n", + __func__, data_off, data_count, bytes_returned); + cifs_buf_release(pSMB); + return -EINVAL; + } + /* BB fixme add lock for file (srch_info) struct here */ psrch_inf->unicode = !!(pSMBr->hdr.Flags2 & SMBFLG2_UNICODE); - response_data = (char *)&pSMBr->hdr.Protocol + - le16_to_cpu(pSMBr->t2.ParameterOffset); + response_data = (char *)&pSMBr->hdr.Protocol + parm_off; parms = (T2_FNEXT_RSP_PARMS *)response_data; - response_data = (char *)&pSMBr->hdr.Protocol + - le16_to_cpu(pSMBr->t2.DataOffset); + response_data = (char *)&pSMBr->hdr.Protocol + data_off; if (psrch_inf->smallBuf) cifs_small_buf_release(psrch_inf->ntwrk_buf_start); @@ -4662,8 +4726,14 @@ int CIFSFindNext(const unsigned int xid, struct cifs_tcon *tcon, psrch_inf->endOfSearch = !!parms->EndofSearch; psrch_inf->entries_in_buffer = le16_to_cpu(parms->SearchCount); psrch_inf->index_of_last_entry += psrch_inf->entries_in_buffer; + psrch_inf->presume_name = NULL; + psrch_inf->resume_name_len = 0; + psrch_inf->resume_key = 0; lnoff = le16_to_cpu(parms->LastNameOffset); - if (CIFSMaxBufSize < lnoff) { + if (!psrch_inf->entries_in_buffer) { + psrch_inf->last_entry = NULL; + } else if (psrch_inf->srch_entries_start + lnoff >= + psrch_inf->ntwrk_buf_start + bytes_returned) { cifs_dbg(VFS, "ignoring corrupt resume name\n"); psrch_inf->last_entry = NULL; } else { @@ -6161,9 +6231,11 @@ CIFSSMBQAllEAs(const unsigned int xid, struct cifs_tcon *tcon, int rc = 0; int bytes_returned; int list_len; + u32 ea_list_len; struct fealist *ea_response_data; struct fea *temp_fea; char *temp_ptr; + char *byte_area; char *end_of_smb; __u16 params, byte_count, data_offset; unsigned int ea_name_len = ea_name ? strlen(ea_name) : 0; @@ -6242,9 +6314,19 @@ CIFSSMBQAllEAs(const unsigned int xid, struct cifs_tcon *tcon, ea_response_data = (struct fealist *) (((char *) &pSMBr->hdr.Protocol) + data_offset); - list_len = le32_to_cpu(ea_response_data->list_len); - cifs_dbg(FYI, "ea length %d\n", list_len); - if (list_len <= 8) { + byte_area = pByteArea(&pSMBr->hdr); + end_of_smb = byte_area + get_bcc(&pSMBr->hdr); + if ((char *)ea_response_data < byte_area || + (char *)ea_response_data > + end_of_smb - sizeof(ea_response_data->list_len)) { + rc = smb_EIO2(smb_eio_trace_qalleas_overlong, + data_offset, get_bcc(&pSMBr->hdr)); + goto QAllEAsOut; + } + + ea_list_len = le32_to_cpu(ea_response_data->list_len); + cifs_dbg(FYI, "%s: ea length: %u\n", __func__, ea_list_len); + if (ea_list_len <= 8) { cifs_dbg(FYI, "empty EA list returned from server\n"); /* didn't find the named attribute */ if (ea_name) @@ -6253,14 +6335,14 @@ CIFSSMBQAllEAs(const unsigned int xid, struct cifs_tcon *tcon, } /* make sure list_len doesn't go past end of SMB */ - end_of_smb = (char *)pByteArea(&pSMBr->hdr) + get_bcc(&pSMBr->hdr); - if ((char *)ea_response_data + list_len > end_of_smb) { + if (ea_list_len > end_of_smb - (char *)ea_response_data) { cifs_dbg(FYI, "EA list appears to go beyond SMB\n"); rc = smb_EIO2(smb_eio_trace_qalleas_overlong, - (unsigned long)ea_response_data + list_len - (unsigned long)pSMBr, - (unsigned long)end_of_smb - (unsigned long)pSMBr); + ea_list_len, + end_of_smb - (char *)ea_response_data); goto QAllEAsOut; } + list_len = ea_list_len; /* account for ea list len */ list_len -= 4; diff --git a/fs/smb/client/connect.c b/fs/smb/client/connect.c index 28e1ddeb6182..9795aa0661d5 100644 --- a/fs/smb/client/connect.c +++ b/fs/smb/client/connect.c @@ -1062,6 +1062,19 @@ cifs_enable_signing(struct TCP_Server_Info *server, bool mnt_sign_required) return 0; } +static void cifs_tcp_server_put(struct TCP_Server_Info *server) +{ + if (!refcount_dec_and_test(&server->obj_refs)) + return; + + cifs_crypto_secmech_release(server); + kfree_sensitive(server->session_key.response); + put_net(cifs_net_ns(server)); + kfree(server->leaf_fullpath); + kfree(server->hostname); + kfree_sensitive(server); +} + static noinline_for_stack void clean_demultiplex_info(struct TCP_Server_Info *server) { @@ -1147,14 +1160,11 @@ clean_demultiplex_info(struct TCP_Server_Info *server) */ } - put_net(cifs_net_ns(server)); - kfree(server->leaf_fullpath); - kfree(server->hostname); - kfree_sensitive(server); - length = atomic_dec_return(&tcpSesAllocCount); if (length > 0) mempool_resize(cifs_req_poolp, length + cifs_min_rcv); + + cifs_tcp_server_put(server); } static int @@ -1228,8 +1238,7 @@ cifs_handle_standard(struct TCP_Server_Info *server, struct mid_q_entry *mid) return 0; } -static void -smb2_add_credits_from_hdr(char *buffer, struct TCP_Server_Info *server) +void smb2_add_credits_from_hdr(char *buffer, struct TCP_Server_Info *server) { struct smb2_hdr *shdr = (struct smb2_hdr *)buffer; int scredits, in_flight; @@ -1401,12 +1410,14 @@ cifs_demultiplex_thread(void *p) mid_execute_callback(server, mids[i]); release_mid(server, mids[i]); - } else if (server->ops->is_oplock_break && + } else if (bufs[i] && server->ops->is_oplock_break && server->ops->is_oplock_break(bufs[i], server)) { smb2_add_credits_from_hdr(bufs[i], server); cifs_dbg(FYI, "Received oplock break\n"); } else { + if (!bufs[i]) + continue; cifs_server_dbg(VFS, "No task to wake, unknown frame received! NumMids %d\n", atomic_read(&mid_count)); cifs_dump_mem("Received Data is: ", bufs[i], @@ -1729,7 +1740,7 @@ cifs_put_tcp_session(struct TCP_Server_Info *server, int from_reconnect) else cancel_delayed_work_sync(&server->reconnect); - /* For secondary channels, we pick up ref-count on the primary server */ + /* For secondary channels, drop the reference on the primary server */ if (SERVER_IS_CHAN(server)) cifs_put_tcp_session(server->primary_server, from_reconnect); @@ -1737,15 +1748,11 @@ cifs_put_tcp_session(struct TCP_Server_Info *server, int from_reconnect) server->tcpStatus = CifsExiting; spin_unlock(&server->srv_lock); - cifs_crypto_secmech_release(server); - - kfree_sensitive(server->session_key.response); - server->session_key.response = NULL; - server->session_key.len = 0; - task = xchg(&server->tsk, NULL); if (task) send_sig(SIGKILL, task, 1); + + cifs_tcp_server_put(server); } struct TCP_Server_Info * @@ -1767,6 +1774,7 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx, rc = -ENOMEM; goto out_err; } + refcount_set(&tcp_ses->obj_refs, 1); tcp_ses->hostname = kstrdup(ctx->server_hostname, GFP_KERNEL); if (!tcp_ses->hostname) { @@ -1905,6 +1913,8 @@ cifs_get_tcp_session(struct smb3_fs_context *ctx, module_put(THIS_MODULE); goto out_err_crypto_release; } + /* Keep the server alive until the demultiplex thread has exited */ + refcount_inc(&tcp_ses->obj_refs); /* thread created, put it on the list */ spin_lock(&cifs_tcp_ses_lock); list_add(&tcp_ses->tcp_ses_list, &cifs_tcp_ses_list); diff --git a/fs/smb/client/inode.c b/fs/smb/client/inode.c index f29555f3c7d0..a2215f1abf52 100644 --- a/fs/smb/client/inode.c +++ b/fs/smb/client/inode.c @@ -88,7 +88,9 @@ static void cifs_set_ops(struct inode *inode) else inode->i_data.a_ops = &cifs_addr_ops; mapping_set_large_folios(inode->i_mapping); - if (tcon->ses->server->sign) + if (tcon->ses->server->sign || + (tcon->ses->server->compression.enabled && + (tcon->share_flags & SMB2_SHAREFLAG_COMPRESS_DATA))) mapping_set_stable_writes(inode->i_mapping); break; case S_IFDIR: diff --git a/fs/smb/client/readdir.c b/fs/smb/client/readdir.c index 9530e5b01564..1667bba40e44 100644 --- a/fs/smb/client/readdir.c +++ b/fs/smb/client/readdir.c @@ -456,17 +456,20 @@ initiate_cifs_search(const unsigned int xid, struct file *file, } /* return length of unicode string in bytes */ -static int cifs_unicode_bytelen(const char *str) +static size_t cifs_unicode_bytelen(const char *str, size_t maxlen) { - int len; const __le16 *ustr = (const __le16 *)str; + size_t len; - for (len = 0; len <= PATH_MAX; len++) { + maxlen = umin(PATH_MAX + 1, maxlen / sizeof(__le16)); + for (len = 0; len < maxlen; len++) { if (ustr[len] == 0) - return len << 1; + return len * sizeof(__le16); } - cifs_dbg(FYI, "Unicode string longer than PATH_MAX found\n"); - return len << 1; + len *= sizeof(__le16); + cifs_dbg(FYI, "%s: Unicode string not terminated within %zu bytes\n", + __func__, len); + return len; } static char *nxt_dir_entry(char *old_entry, char *end_of_smb, int level) @@ -532,13 +535,13 @@ static void cifs_fill_dirent_posix(struct cifs_dirent *de, } static void cifs_fill_dirent_unix(struct cifs_dirent *de, - const FILE_UNIX_INFO *info, bool is_unicode) + const FILE_UNIX_INFO *info, bool is_unicode, size_t maxlen) { de->name = &info->FileName[0]; if (is_unicode) - de->namelen = cifs_unicode_bytelen(de->name); + de->namelen = cifs_unicode_bytelen(de->name, maxlen); else - de->namelen = strnlen(de->name, PATH_MAX); + de->namelen = strnlen(de->name, min_t(size_t, maxlen, PATH_MAX)); de->resume_key = info->ResumeKey; de->ino = le64_to_cpu(info->basic.UniqueId); } @@ -586,30 +589,49 @@ static void cifs_fill_dirent_std(struct cifs_dirent *de, } static int cifs_fill_dirent(struct cifs_dirent *de, const void *info, - u16 level, bool is_unicode) + const char *end, u16 level, bool is_unicode) { + size_t len = end > (const char *)info ? end - (const char *)info : 0; + size_t nul_len = 0; + memset(de, 0, sizeof(*de)); switch (level) { case SMB_FIND_FILE_POSIX_INFO: + if (len < sizeof(struct smb2_posix_info)) + goto too_short; cifs_fill_dirent_posix(de, info); break; case SMB_FIND_FILE_UNIX: - cifs_fill_dirent_unix(de, info, is_unicode); + if (len < offsetof(FILE_UNIX_INFO, FileName)) + goto too_short; + cifs_fill_dirent_unix(de, info, is_unicode, + len - offsetof(FILE_UNIX_INFO, FileName)); + nul_len = is_unicode ? sizeof(__le16) : sizeof(__u8); break; case SMB_FIND_FILE_DIRECTORY_INFO: + if (len < offsetof(FILE_DIRECTORY_INFO, FileName)) + goto too_short; cifs_fill_dirent_dir(de, info); break; case SMB_FIND_FILE_FULL_DIRECTORY_INFO: + if (len < offsetof(FILE_FULL_DIRECTORY_INFO, FileName)) + goto too_short; cifs_fill_dirent_full(de, info); break; case SMB_FIND_FILE_ID_FULL_DIR_INFO: + if (len < offsetof(FILE_ID_FULL_DIR_INFO, FileName)) + goto too_short; cifs_fill_dirent_search(de, info); break; case SMB_FIND_FILE_BOTH_DIRECTORY_INFO: + if (len < offsetof(FILE_BOTH_DIRECTORY_INFO, FileName)) + goto too_short; cifs_fill_dirent_both(de, info); break; case SMB_FIND_FILE_INFO_STANDARD: + if (len < offsetof(FIND_FILE_STANDARD_INFO, FileName)) + goto too_short; cifs_fill_dirent_std(de, info); break; default: @@ -617,7 +639,20 @@ static int cifs_fill_dirent(struct cifs_dirent *de, const void *info, return -EINVAL; } + if (de->name && + (end < de->name || + (size_t)(end - de->name) < de->namelen + nul_len)) { + cifs_dbg(VFS, "%s: search entry name extends past end of SMB\n", + __func__); + return -EINVAL; + } + return 0; + +too_short: + cifs_dbg(VFS, "%s: search entry extends past end of SMB\n", + __func__); + return -EINVAL; } #define UNICODE_DOT cpu_to_le16(0x2e) @@ -672,10 +707,14 @@ static int is_dir_changed(struct file *file) static int cifs_save_resume_key(const char *current_entry, struct cifsFileInfo *file_info) { + struct TCP_Server_Info *server = tlink_tcon(file_info->tlink)->ses->server; + char *buf = file_info->srch_inf.ntwrk_buf_start; + const char *end = buf + server->ops->calc_smb_size(buf); struct cifs_dirent de; int rc; - rc = cifs_fill_dirent(&de, current_entry, file_info->srch_inf.info_level, + rc = cifs_fill_dirent(&de, current_entry, end, + file_info->srch_inf.info_level, file_info->srch_inf.unicode); if (!rc) { file_info->srch_inf.presume_name = de.name; @@ -752,6 +791,9 @@ find_cifs_entry(const unsigned int xid, struct cifs_tcon *tcon, loff_t pos, cfile->srch_inf.ntwrk_buf_start = NULL; cfile->srch_inf.srch_entries_start = NULL; cfile->srch_inf.last_entry = NULL; + cfile->srch_inf.presume_name = NULL; + cfile->srch_inf.resume_name_len = 0; + cfile->srch_inf.resume_key = 0; } rc = initiate_cifs_search(xid, file, full_path); if (rc) { @@ -974,7 +1016,8 @@ static int cifs_filldir(char *find_entry, struct file *file, struct qstr name; int rc = 0; - rc = cifs_fill_dirent(&de, find_entry, file_info->srch_inf.info_level, + rc = cifs_fill_dirent(&de, find_entry, end_of_smb, + file_info->srch_inf.info_level, file_info->srch_inf.unicode); if (rc) return rc; diff --git a/fs/smb/client/sess.c b/fs/smb/client/sess.c index e095f41b5882..0ab3f37d4809 100644 --- a/fs/smb/client/sess.c +++ b/fs/smb/client/sess.c @@ -626,12 +626,13 @@ cifs_ses_add_channel(struct cifs_ses *ses, out: if (rc && chan->server) { - cifs_put_tcp_session(chan->server, 0); + struct TCP_Server_Info *pserver; spin_lock(&ses->chan_lock); + pserver = chan->server; /* we rely on all bits beyond chan_count to be clear */ - cifs_chan_clear_need_reconnect(ses, chan->server); + cifs_chan_clear_need_reconnect(ses, pserver); ses->chan_count--; /* * chan_count should never reach 0 as at least the primary @@ -639,6 +640,7 @@ cifs_ses_add_channel(struct cifs_ses *ses, */ WARN_ON(ses->chan_count < 1); spin_unlock(&ses->chan_lock); + cifs_put_tcp_session(pserver, 0); } kfree(ctx->UNC); diff --git a/fs/smb/client/smb2misc.c b/fs/smb/client/smb2misc.c index 5e5cf92d1eb3..5f13bd7ce73c 100644 --- a/fs/smb/client/smb2misc.c +++ b/fs/smb/client/smb2misc.c @@ -225,6 +225,22 @@ smb2_check_message(char *buf, unsigned int pdu_len, unsigned int len, return 1; } + /* + * A SESSION_SETUP response that asks for more processing carries a + * security blob and is parsed as a normal reply. Do not let the + * short-error path below accept a frame that cannot hold the fixed + * response fields; the minimum-PDU check later is unreachable once + * the early return has accepted the frame. + */ + if (command == SMB2_SESSION_SETUP_HE && + shdr->Status == STATUS_MORE_PROCESSING_REQUIRED && + len < smb2_min_pdu_len[command]) { + cifs_server_dbg(VFS, + "%s: session setup response too short: %u < %zu\n", + __func__, len, smb2_min_pdu_len[command]); + return 1; + } + if (len < pdu_size) { if ((len >= hdr_size) && (shdr->Status != 0)) { diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c index 583ab4c4d773..a6da016ce7b9 100644 --- a/fs/smb/client/smb2ops.c +++ b/fs/smb/client/smb2ops.c @@ -4360,6 +4360,15 @@ static int smb3_fiemap(struct cifs_tcon *tcon, goto out; } + /* + * A truncated range blob must contain at least one range, otherwise + * there is no offset to resume from and out_data is NULL. + */ + if (!last_blob && out_data_len == 0) { + rc = -EINVAL; + goto out; + } + num = out_data_len / sizeof(struct file_allocated_range_buffer); for (i = 0; i < num; i++) { flags = 0; @@ -4380,10 +4389,25 @@ static int smb3_fiemap(struct cifs_tcon *tcon, } if (!last_blob) { - next = le64_to_cpu(out_data[num - 1].file_offset) + - le64_to_cpu(out_data[num - 1].length); - len = len - (next - start); + u64 last_start, last_len; + + last_start = le64_to_cpu(out_data[num - 1].file_offset); + last_len = le64_to_cpu(out_data[num - 1].length); + + if (check_add_overflow(last_start, last_len, &next) || + next <= start) { + rc = -EINVAL; + goto out; + } + if (next - start >= len) { + rc = 0; + goto out; + } + + len -= next - start; start = next; + kfree(out_data); + out_data = NULL; goto again; } @@ -5264,9 +5288,10 @@ static void smb2_decrypt_offload(struct work_struct *work) dw->server->lstrp = jiffies; mid = smb2_find_dequeue_mid(dw->server, dw->buf); - if (mid == NULL) + if (mid == NULL) { cifs_dbg(FYI, "mid not found\n"); - else { + smb2_add_credits_from_hdr(dw->buf, dw->server); + } else { mid->decrypted = true; rc = handle_read_data(dw->server, mid, dw->buf, dw->server->vals->read_rsp_size, @@ -5397,6 +5422,8 @@ receive_encrypted_read(struct TCP_Server_Info *server, struct mid_q_entry **mid, *mid = smb2_find_mid(server, buf); if (*mid == NULL) { cifs_dbg(FYI, "mid not found\n"); + smb2_add_credits_from_hdr(buf, server); + rc = -1; } else { cifs_dbg(FYI, "mid found\n"); (*mid)->decrypted = true; diff --git a/fs/smb/client/smb2pdu.c b/fs/smb/client/smb2pdu.c index cfe3c4b4abe7..78ced1f51044 100644 --- a/fs/smb/client/smb2pdu.c +++ b/fs/smb/client/smb2pdu.c @@ -1860,14 +1860,20 @@ SMB2_sess_auth_rawntlmssp_negotiate(struct SMB2_sess_data *sess_data) goto out; u16 boff = le16_to_cpu(rsp->SecurityBufferOffset); + u16 blen = le16_to_cpu(rsp->SecurityBufferLength); if (offsetof(struct smb2_sess_setup_rsp, Buffer) != boff) { cifs_dbg(VFS, "Invalid security buffer offset %d\n", boff); rc = smb_EIO1(smb_eio_trace_sess_buf_off, boff); goto out; } - rc = decode_ntlmssp_challenge(rsp->Buffer, - le16_to_cpu(rsp->SecurityBufferLength), ses); + if ((u32)boff + blen > sess_data->iov[0].iov_len) { + cifs_dbg(VFS, "%s: invalid security buffer length: %u\n", + __func__, blen); + rc = -EINVAL; + goto out; + } + rc = decode_ntlmssp_challenge(rsp->Buffer, blen, ses); if (rc) goto out; @@ -2118,8 +2124,6 @@ static inline void cifs_stats_fail_inc(struct cifs_tcon *tcon, uint16_t code) cifs_stats_inc(&tcon->stats.smb2_stats.smb2_com_failed[code]); } -#define MAX_SHARENAME_LENGTH (255 /* server */ + 80 /* share */ + 1 /* NULL */) - /* These are similar values to what Windows uses */ static inline void init_copy_chunk_defaults(struct cifs_tcon *tcon) { @@ -2150,7 +2154,15 @@ SMB2_tcon(const unsigned int xid, struct cifs_ses *ses, const char *tree, if (!server || !tree) return smb_EIO(smb_eio_trace_null_pointers); - unc_path = kmalloc(MAX_SHARENAME_LENGTH * 2, GFP_KERNEL); + total_len = strlen(tree); + if (total_len > MAX_TREE_SIZE) { + cifs_server_dbg(VFS | ONCE, + "%s: tree name exceeded %u characters\n", + __func__, MAX_TREE_SIZE); + return -EINVAL; + } + + unc_path = kmalloc((total_len + 1) * sizeof(*unc_path), GFP_KERNEL); if (unc_path == NULL) return -ENOMEM; @@ -4950,6 +4962,7 @@ SMB2_read(const unsigned int xid, struct cifs_io_parms *io_parms, struct kvec iov[1]; struct kvec rsp_iov; unsigned int total_len; + u16 data_offset; int flags = CIFS_LOG_ERROR; struct cifs_ses *ses = io_parms->tcon->ses; @@ -5009,8 +5022,29 @@ SMB2_read(const unsigned int xid, struct cifs_io_parms *io_parms, *nbytes = 0; } + data_offset = rsp->DataOffset; + if (data_offset < io_parms->server->vals->read_rsp_size) { + /* + * win2k8 sometimes sends an offset of 0 when the read is beyond + * the EOF. Treat it as if the data starts just after the header, + * as handle_read_data() does. + */ + cifs_dbg(FYI, "data offset %u inside read response header\n", + data_offset); + data_offset = io_parms->server->vals->read_rsp_size; + } + + /* The data area must lie within the response actually received */ + if (*nbytes && (size_t)data_offset + *nbytes > rsp_iov.iov_len) { + cifs_dbg(FYI, "bad data offset %u length %u for response of %zu\n", + data_offset, *nbytes, rsp_iov.iov_len); + rc = smb_EIO2(smb_eio_trace_read_bad_offset, + data_offset, *nbytes); + *nbytes = 0; + } + if (*buf) { - memcpy(*buf, (char *)rsp + rsp->DataOffset, *nbytes); + memcpy(*buf, (char *)rsp + data_offset, *nbytes); free_rsp_buf(resp_buftype, rsp_iov.iov_base); } else if (resp_buftype != CIFS_NO_BUFFER) { *buf = rsp_iov.iov_base; diff --git a/fs/smb/client/trace.h b/fs/smb/client/trace.h index bb8d0197cb54..5331ce6dc8b3 100644 --- a/fs/smb/client/trace.h +++ b/fs/smb/client/trace.h @@ -30,6 +30,8 @@ EM(smb_eio_trace_ea_next_offset, "ea_next_offset") \ EM(smb_eio_trace_ea_overrun, "ea_overrun") \ EM(smb_eio_trace_extract_will_pin, "extract_will_pin") \ + EM(smb_eio_trace_ffirst_param_area, "ffirst_param_area") \ + EM(smb_eio_trace_fnext_param_area, "fnext_param_area") \ EM(smb_eio_trace_forced_shutdown, "forced_shutdown") \ EM(smb_eio_trace_getacl_bcc_too_small, "getacl_bcc_too_small") \ EM(smb_eio_trace_getcifsacl_param_count, "getcifsacl_param_count") \ @@ -42,6 +44,7 @@ EM(smb_eio_trace_ioctl_no_rsp, "ioctl_no_rsp") \ EM(smb_eio_trace_ioctl_out_off, "ioctl_out_off") \ EM(smb_eio_trace_lock_bcc_too_small, "lock_bcc_too_small") \ + EM(smb_eio_trace_lock_data_area, "lock_data_area") \ EM(smb_eio_trace_lock_data_too_small, "lock_data_too_small") \ EM(smb_eio_trace_malformed_ksid_key, "malformed_ksid_key") \ EM(smb_eio_trace_malformed_sid_key, "malformed_sid_key") \ diff --git a/fs/smb/client/transport.c b/fs/smb/client/transport.c index 6e21b5f8754a..1c4aaa4b5ac8 100644 --- a/fs/smb/client/transport.c +++ b/fs/smb/client/transport.c @@ -1317,6 +1317,8 @@ cifs_readv_receive(struct TCP_Server_Info *server, struct mid_q_entry *mid) #endif length = cifs_read_iter_from_socket(server, &rdata->subreq.io_iter, data_len); + if (length < 0) + return length; if (length > 0) rdata->got_bytes += length; server->total_read += length; diff --git a/fs/smb/common/smbglob.h b/fs/smb/common/smbglob.h index d9c7e6e7af29..fdf840888062 100644 --- a/fs/smb/common/smbglob.h +++ b/fs/smb/common/smbglob.h @@ -40,6 +40,7 @@ struct smb_version_values { size_t create_disk_id_size; size_t create_posix_size; size_t create_aapl_size; + size_t create_rsp_size; }; static inline unsigned int get_rfc1002_len(void *buf) diff --git a/fs/smb/server/auth.c b/fs/smb/server/auth.c index 1b24eb3f3998..0c51ca0cb14d 100644 --- a/fs/smb/server/auth.c +++ b/fs/smb/server/auth.c @@ -336,8 +336,7 @@ ksmbd_build_ntlmssp_challenge_blob(struct challenge_message *chgblob, if (cflags & NTLMSSP_REQUEST_TARGET) flags |= NTLMSSP_REQUEST_TARGET; - if (conn->use_spnego && - (cflags & NTLMSSP_NEGOTIATE_EXTENDED_SEC)) + if (cflags & NTLMSSP_NEGOTIATE_EXTENDED_SEC) flags |= NTLMSSP_NEGOTIATE_EXTENDED_SEC; if (cflags & NTLMSSP_NEGOTIATE_KEY_XCH) @@ -464,7 +463,8 @@ int ksmbd_krb5_authenticate(struct ksmbd_session *sess, char *in_blob, memcpy(out_blob, resp->payload + resp->session_key_len, resp->spnego_blob_len); *out_len = resp->spnego_blob_len; - sess->kerberos_expiry = resp->session_expiry; + sess->kerberos_expiry = (u64)resp->login_response.session_expiry_hi << 32 | + resp->login_response.session_expiry_lo; retval = 0; out: kvfree_sensitive(resp, sizeof(*resp) + resp->session_key_len + diff --git a/fs/smb/server/compress.c b/fs/smb/server/compress.c index 5162fb84c755..3fcb3e44302e 100644 --- a/fs/smb/server/compress.c +++ b/fs/smb/server/compress.c @@ -65,6 +65,7 @@ static int __ksmbd_decompress_request(struct ksmbd_conn *conn, return -ENOMEM; *(__be32 *)out = cpu_to_be32(out_size); + out[out_size + 4] = 0; rc = smb_compression_decompress(conn->compress_algorithm, conn->compress_chained, conn->compress_pattern, diff --git a/fs/smb/server/connection.c b/fs/smb/server/connection.c index d211861ff86f..b519376ecc17 100644 --- a/fs/smb/server/connection.c +++ b/fs/smb/server/connection.c @@ -306,6 +306,7 @@ struct ksmbd_conn *ksmbd_conn_alloc(void) atomic_set(&conn->req_running, 0); atomic_set(&conn->r_count, 0); atomic_set(&conn->refcnt, 1); + atomic64_set(&conn->session_count, 0); conn->total_credits = 1; conn->outstanding_credits = 0; @@ -457,17 +458,13 @@ void ksmbd_all_conn_set_status(struct ksmbd_session *sess, u32 status) void ksmbd_conn_abort(struct ksmbd_conn *conn) { - bool shutdown = false; - spin_lock(&conn->request_lock); - if (!ksmbd_conn_exiting(conn) && !ksmbd_conn_releasing(conn)) { + if (!ksmbd_conn_exiting(conn) && !ksmbd_conn_releasing(conn)) ksmbd_conn_set_exiting(conn); - shutdown = true; - } spin_unlock(&conn->request_lock); wake_up_all(&conn->req_running_q); - if (shutdown && conn->transport->ops->shutdown) + if (conn->transport->ops->shutdown) conn->transport->ops->shutdown(conn->transport); } @@ -534,12 +531,19 @@ static int __ksmbd_conn_write(struct ksmbd_work *work, tx->need_invalidate_rkey = work->need_invalidate_rkey; tx->remote_key = work->remote_key; - ksmbd_conn_lock(conn); + if (tx->tracking) { + /* A busy writer must not consume the break's retry budget. */ + if (!mutex_trylock(&conn->srv_mutex)) + return -EAGAIN; + } else { + ksmbd_conn_lock(conn); + } sent = conn->transport->ops->writev(conn->transport, tx); ksmbd_conn_unlock(conn); if (sent < 0) { - pr_err("Failed to send message: %d\n", sent); + if (!tx->tracking || sent != -EAGAIN) + pr_err("Failed to send message: %d\n", sent); return sent; } @@ -553,6 +557,20 @@ int ksmbd_conn_write(struct ksmbd_work *work) return __ksmbd_conn_write(work, &tx); } +int ksmbd_conn_write_tracked(struct ksmbd_work *work, + struct ksmbd_tx_tracking *tracking) +{ + struct ksmbd_transport_write tx = {}; + + tracking->valid = false; + /* Transports without delivery tracking keep their synchronous send path. */ + if (work->conn->transport->ops->is_acked) { + tx.msg_flags = MSG_DONTWAIT; + tx.tracking = tracking; + } + return __ksmbd_conn_write(work, &tx); +} + int ksmbd_conn_write_eor(struct ksmbd_work *work) { struct ksmbd_transport_write tx = { @@ -641,6 +659,7 @@ static void ksmbd_session_expiration_worker(struct work_struct *work) goto reschedule; ksmbd_expire_sessions(); + ksmbd_reclaim_ended_sessions(); /* * An old connection without a Valid or Expired session must be @@ -752,6 +771,7 @@ int ksmbd_conn_handler_loop(void *p) if (!conn->request_buf) break; + conn->request_buf[pdu_size + 4] = 0; memcpy(conn->request_buf, hdr_buf, sizeof(hdr_buf)); /* @@ -804,7 +824,12 @@ int ksmbd_conn_handler_loop(void *p) } } + /* Exclude new break senders before draining their work references. */ + spin_lock(&conn->request_lock); ksmbd_conn_set_releasing(conn); + spin_unlock(&conn->request_lock); + /* TCP writers waiting for queue space must release their work references. */ + wake_up_all(&conn->req_running_q); ksmbd_conn_cancel_async_requests(conn); /* Wait till all reference dropped to the Server object*/ ksmbd_debug(CONN, "Wait for all pending requests(%d)\n", atomic_read(&conn->r_count)); @@ -831,6 +856,20 @@ void ksmbd_conn_r_count_inc(struct ksmbd_conn *conn) atomic_inc(&conn->r_count); } +/* The caller already owns a connection reference. */ +bool ksmbd_conn_r_count_inc_if_good(struct ksmbd_conn *conn) +{ + bool ret = false; + + spin_lock(&conn->request_lock); + if (ksmbd_conn_good(conn)) { + atomic_inc(&conn->r_count); + ret = true; + } + spin_unlock(&conn->request_lock); + return ret; +} + void ksmbd_conn_r_count_dec(struct ksmbd_conn *conn) { /* diff --git a/fs/smb/server/connection.h b/fs/smb/server/connection.h index 371f17b4f02a..a4f5a64b36b4 100644 --- a/fs/smb/server/connection.h +++ b/fs/smb/server/connection.h @@ -77,6 +77,8 @@ struct ksmbd_conn { struct rw_semaphore session_lock; /* smb session 1 per user */ struct xarray sessions; + /* New sessions and SMB3 channels each reserve one connection slot. */ + atomic64_t session_count; unsigned long creation_time; unsigned long last_active; /* How many request are running currently */ @@ -100,6 +102,8 @@ struct ksmbd_conn { wait_queue_head_t r_count_q; /* Lock to protect requests list*/ spinlock_t request_lock; + /* Set under request_lock at teardown start and kept set thereafter. */ + bool releasing; struct list_head requests; struct list_head async_requests; int connection_type; @@ -159,6 +163,13 @@ struct ksmbd_conn_ops { int (*terminate_fn)(struct ksmbd_conn *conn); }; +struct ksmbd_tx_tracking { + /* Required TCP byte count, including transport and socket queues. */ + u64 end; + unsigned long retry_after; + bool valid; +}; + struct ksmbd_transport_write { struct kvec *iov; int iov_cnt; @@ -166,6 +177,7 @@ struct ksmbd_transport_write { bool need_invalidate_rkey; unsigned int remote_key; int msg_flags; + struct ksmbd_tx_tracking *tracking; }; struct ksmbd_transport_ops { @@ -173,8 +185,12 @@ struct ksmbd_transport_ops { void (*shutdown)(struct ksmbd_transport *t); int (*read)(struct ksmbd_transport *t, char *buf, unsigned int size, int max_retries); + /* Retained bytes must be privately owned before a full result is returned. */ int (*writev)(struct ksmbd_transport *t, const struct ksmbd_transport_write *tx); + /* Optional delivery tracking for nonblocking break sends. */ + bool (*is_acked)(struct ksmbd_transport *t, + const struct ksmbd_tx_tracking *tracking); int (*rdma_read)(struct ksmbd_transport *t, void *buf, unsigned int len, struct smbdirect_buffer_descriptor_v1 *desc, @@ -214,6 +230,8 @@ int ksmbd_conn_wq_init(void); void ksmbd_conn_wq_destroy(void); bool ksmbd_conn_lookup_dialect(struct ksmbd_conn *c); int ksmbd_conn_write(struct ksmbd_work *work); +int ksmbd_conn_write_tracked(struct ksmbd_work *work, + struct ksmbd_tx_tracking *tracking); int ksmbd_conn_write_eor(struct ksmbd_work *work); int ksmbd_conn_rdma_read(struct ksmbd_conn *conn, void *buf, unsigned int buflen, @@ -232,6 +250,7 @@ void ksmbd_conn_transport_destroy(void); void ksmbd_conn_lock(struct ksmbd_conn *conn); void ksmbd_conn_unlock(struct ksmbd_conn *conn); void ksmbd_conn_r_count_inc(struct ksmbd_conn *conn); +bool ksmbd_conn_r_count_inc_if_good(struct ksmbd_conn *conn); void ksmbd_conn_r_count_dec(struct ksmbd_conn *conn); /* @@ -247,7 +266,8 @@ static inline bool ksmbd_conn_new(struct ksmbd_conn *conn) static inline bool ksmbd_conn_good(struct ksmbd_conn *conn) { - return READ_ONCE(conn->status) == KSMBD_SESS_GOOD; + return !READ_ONCE(conn->releasing) && + READ_ONCE(conn->status) == KSMBD_SESS_GOOD; } static inline unsigned int @@ -281,7 +301,8 @@ static inline bool ksmbd_conn_exiting(struct ksmbd_conn *conn) static inline bool ksmbd_conn_releasing(struct ksmbd_conn *conn) { - return READ_ONCE(conn->status) == KSMBD_SESS_RELEASING; + return READ_ONCE(conn->releasing) || + READ_ONCE(conn->status) == KSMBD_SESS_RELEASING; } static inline void ksmbd_conn_set_new(struct ksmbd_conn *conn) @@ -316,6 +337,7 @@ static inline void ksmbd_conn_set_exiting(struct ksmbd_conn *conn) static inline void ksmbd_conn_set_releasing(struct ksmbd_conn *conn) { + WRITE_ONCE(conn->releasing, true); WRITE_ONCE(conn->status, KSMBD_SESS_RELEASING); } diff --git a/fs/smb/server/ksmbd_netlink.h b/fs/smb/server/ksmbd_netlink.h index 2673522c76bc..ad050988659e 100644 --- a/fs/smb/server/ksmbd_netlink.h +++ b/fs/smb/server/ksmbd_netlink.h @@ -114,7 +114,8 @@ struct ksmbd_startup_request { __s8 bind_interfaces_only; __u32 max_ip_connections; /* Number of maximum connection per ip address */ __s8 aapl_model[32]; /* AAPL model string for Finder icon, e.g. "Xserve" */ - __s8 reserved[467]; /* Reserved room */ + __u32 max_sessions_per_connection; /* Maximum sessions per connection */ + __s8 reserved[463]; /* Reserved room */ __u32 ifc_list_sz; /* interfaces list size */ __s8 ____payload[]; } __packed; @@ -148,7 +149,9 @@ struct ksmbd_login_response { __u16 status; __u16 hash_sz; /* hash size */ __s8 hash[KSMBD_REQ_MAX_HASH_SZ]; /* password hash */ - __u32 reserved[16]; /* Reserved room */ + __u32 session_expiry_lo; /* Kerberos ticket expiry, SPNEGO only */ + __u32 session_expiry_hi; + __u32 reserved[14]; /* Reserved room */ }; /* @@ -286,7 +289,6 @@ struct ksmbd_spnego_authen_response { * stored in SecurityBuffer of SMB2 SESSION * SETUP response */ - __u64 session_expiry; /* Kerberos ticket expiry time */ __u8 payload[]; /* session key + AP_REP */ }; diff --git a/fs/smb/server/ksmbd_work.c b/fs/smb/server/ksmbd_work.c index d307aefe0aec..b0285eb02f35 100644 --- a/fs/smb/server/ksmbd_work.c +++ b/fs/smb/server/ksmbd_work.c @@ -16,6 +16,7 @@ static struct kmem_cache *work_cache; static struct workqueue_struct *ksmbd_wq; +static struct workqueue_struct *ksmbd_break_wq; static int ksmbd_reserve_iov(struct ksmbd_work *work, int need_iov_cnt) { @@ -113,6 +114,12 @@ int ksmbd_workqueue_init(void) ksmbd_wq = alloc_workqueue("ksmbd-io", WQ_PERCPU, 0); if (!ksmbd_wq) return -ENOMEM; + ksmbd_break_wq = alloc_workqueue("ksmbd-break", WQ_UNBOUND, 0); + if (!ksmbd_break_wq) { + destroy_workqueue(ksmbd_wq); + ksmbd_wq = NULL; + return -ENOMEM; + } return 0; } @@ -120,6 +127,8 @@ void ksmbd_workqueue_destroy(void) { destroy_workqueue(ksmbd_wq); ksmbd_wq = NULL; + destroy_workqueue(ksmbd_break_wq); + ksmbd_break_wq = NULL; } bool ksmbd_queue_work(struct ksmbd_work *work) @@ -127,6 +136,16 @@ bool ksmbd_queue_work(struct ksmbd_work *work) return queue_work(ksmbd_wq, &work->work); } +void ksmbd_queue_io_work(struct work_struct *work) +{ + queue_work(ksmbd_wq, work); +} + +void ksmbd_queue_break_work(struct work_struct *work) +{ + queue_work(ksmbd_break_wq, work); +} + static inline void __ksmbd_iov_pin(struct ksmbd_work *work, void *ib, unsigned int ib_len) { diff --git a/fs/smb/server/ksmbd_work.h b/fs/smb/server/ksmbd_work.h index 0844aa929f55..6877f6838118 100644 --- a/fs/smb/server/ksmbd_work.h +++ b/fs/smb/server/ksmbd_work.h @@ -156,6 +156,8 @@ int ksmbd_work_pool_init(void); int ksmbd_workqueue_init(void); void ksmbd_workqueue_destroy(void); bool ksmbd_queue_work(struct ksmbd_work *work); +void ksmbd_queue_io_work(struct work_struct *work); +void ksmbd_queue_break_work(struct work_struct *work); int ksmbd_iov_pin_rsp_read(struct ksmbd_work *work, void *ib, int len, void *aux_buf, unsigned int aux_size); int ksmbd_iov_pin_rsp(struct ksmbd_work *work, void *ib, int len); diff --git a/fs/smb/server/mgmt/user_session.c b/fs/smb/server/mgmt/user_session.c index 44dc3f800cd4..1b30e432ebfd 100644 --- a/fs/smb/server/mgmt/user_session.c +++ b/fs/smb/server/mgmt/user_session.c @@ -3,6 +3,7 @@ * Copyright (C) 2018 Samsung Electronics Co., Ltd. */ +#include #include #include #include @@ -18,6 +19,7 @@ #include "../vfs_cache.h" #include "../misc.h" #include "../stats.h" +#include "../server.h" static DEFINE_IDA(session_ida); @@ -26,6 +28,21 @@ static DEFINE_IDA(session_ida); static DEFINE_HASHTABLE(sessions_table, SESSION_HASH_BITS); static DECLARE_RWSEM(sessions_table_lock); +int ksmbd_session_reserve_slot(struct ksmbd_conn *conn) +{ + if (atomic64_inc_return(&conn->session_count) > + READ_ONCE(server_conf.max_sessions_per_connection)) { + atomic64_dec(&conn->session_count); + return -ENOSPC; + } + return 0; +} + +void ksmbd_session_release_slot(struct ksmbd_conn *conn) +{ + atomic64_dec(&conn->session_count); +} + struct ksmbd_session_rpc { int id; unsigned int method; @@ -95,8 +112,10 @@ static unsigned int session_open_file_count(struct ksmbd_session *session) unsigned int id; read_lock(&session->file_table.lock); - idr_for_each_entry(session->file_table.idr, fp, id) - count++; + if (session->file_table.idr) { + idr_for_each_entry(session->file_table.idr, fp, id) + count++; + } read_unlock(&session->file_table.lock); return count; } @@ -212,8 +231,10 @@ static int create_proc_session(struct ksmbd_session *sess) static void delete_proc_session(struct ksmbd_session *sess) { - if (sess->proc_entry) + if (sess->proc_entry) { proc_remove(sess->proc_entry); + sess->proc_entry = NULL; + } } static int show_proc_sessions(struct seq_file *m, void *v) @@ -271,6 +292,8 @@ static void free_channel_list(struct ksmbd_session *sess) down_write(&sess->chann_lock); xa_for_each(&sess->ksmbd_chann_list, index, chann) { xa_erase(&sess->ksmbd_chann_list, index); + ksmbd_session_release_slot(chann->conn); + ksmbd_conn_put(chann->conn); kfree_sensitive(chann); } @@ -397,9 +420,12 @@ int ksmbd_session_rpc_method(struct ksmbd_session *sess, int id) void ksmbd_session_destroy(struct ksmbd_session *sess) { + struct ksmbd_conn *setup_conn; + if (!sess) return; + setup_conn = sess->setup_conn; delete_proc_session(sess); ksmbd_tree_conn_session_logoff(sess); ksmbd_destroy_file_table(sess); @@ -409,9 +435,14 @@ void ksmbd_session_destroy(struct ksmbd_session *sess) ksmbd_session_rpc_clear_list(sess); free_channel_list(sess); kfree_sensitive(sess->Preauth_HashValue); - ksmbd_release_id(&session_ida, sess->id); + if (sess->id) + ksmbd_release_id(&session_ida, sess->id); ida_destroy(&sess->tree_conn_ida); kfree_sensitive(sess); + if (setup_conn) { + ksmbd_session_release_slot(setup_conn); + ksmbd_conn_put(setup_conn); + } } static void ksmbd_session_remove_from_table(struct ksmbd_session *sess) @@ -420,6 +451,17 @@ static void ksmbd_session_remove_from_table(struct ksmbd_session *sess) ksmbd_counter_dec(KSMBD_COUNTER_SESSIONS); } +static void ksmbd_dispose_sessions(struct list_head *dispose) +{ + struct ksmbd_session *sess, *tmp; + + list_for_each_entry_safe(sess, tmp, dispose, dispose_entry) { + list_del_init(&sess->dispose_entry); + delete_proc_session(sess); + ksmbd_user_session_put(sess); + } +} + struct ksmbd_session *__session_lookup(unsigned long long id) { struct ksmbd_session *sess; @@ -433,8 +475,9 @@ struct ksmbd_session *__session_lookup(unsigned long long id) return NULL; } -static bool ksmbd_too_many_session_setups(struct ksmbd_conn *conn) +bool ksmbd_too_many_session_setups(struct ksmbd_conn *conn) { + LIST_HEAD(dispose); unsigned long id; struct ksmbd_session *sess; unsigned int pending = 0; @@ -445,18 +488,20 @@ static bool ksmbd_too_many_session_setups(struct ksmbd_conn *conn) if (READ_ONCE(sess->state) != SMB2_SESSION_IN_PROGRESS) continue; - if (atomic_read(&sess->refcnt) <= 1 && + /* Keep response signing keys until every request has dropped its reference. */ + if (atomic_read(&sess->refcnt) == 1 && time_after(jiffies, sess->last_active + KSMBD_UNAUTHENTICATED_CONN_TIMEOUT)) { xa_erase(&conn->sessions, sess->id); ksmbd_session_remove_from_table(sess); - ksmbd_session_destroy(sess); + list_add_tail(&sess->dispose_entry, &dispose); continue; } pending++; } up_write(&conn->session_lock); up_write(&sessions_table_lock); + ksmbd_dispose_sessions(&dispose); return pending >= KSMBD_MAX_PENDING_SESSIONS; } @@ -467,24 +512,22 @@ int ksmbd_session_register(struct ksmbd_conn *conn, sess->dialect = conn->dialect; memcpy(sess->ClientGUID, conn->ClientGUID, SMB2_CLIENT_GUID_SIZE); - /* Bound abandoned SessionId-zero authentication exchanges. */ - if (ksmbd_too_many_session_setups(conn)) - ret = -ENOSPC; - else - ret = xa_err(xa_store(&conn->sessions, sess->id, sess, - KSMBD_DEFAULT_GFP)); + ret = xa_err(xa_store(&conn->sessions, sess->id, sess, + KSMBD_DEFAULT_GFP)); if (ret) { down_write(&sessions_table_lock); ksmbd_session_remove_from_table(sess); up_write(&sessions_table_lock); + delete_proc_session(sess); ksmbd_user_session_put(sess); } return ret; } -void ksmbd_session_unregister(struct ksmbd_conn *conn, - struct ksmbd_session *sess) +/* The caller holds sessions_table_lock for writing. */ +static bool __ksmbd_session_unregister(struct ksmbd_conn *conn, + struct ksmbd_session *sess) { struct ksmbd_conn *session_conns[KSMBD_MAX_CHANNELS]; struct channel *chann; @@ -492,10 +535,13 @@ void ksmbd_session_unregister(struct ksmbd_conn *conn, unsigned int nr_conns = 0, i; bool removed = false; - down_write(&sessions_table_lock); + lockdep_assert_held_write(&sessions_table_lock); if (!hlist_unhashed(&sess->hlist)) { - /* Keep each channel connection stable under sessions_table_lock. */ + /* Existing references stay valid under sessions_table_lock. */ down_read(&sess->chann_lock); + /* SMB2.0/2.1 sessions have no channel holding their connection. */ + if (sess->setup_conn) + session_conns[nr_conns++] = sess->setup_conn; xa_for_each(&sess->ksmbd_chann_list, index, chann) { if (nr_conns == ARRAY_SIZE(session_conns)) break; @@ -507,10 +553,12 @@ void ksmbd_session_unregister(struct ksmbd_conn *conn, removed = true; } - down_write(&conn->session_lock); - if (xa_load(&conn->sessions, sess->id) == sess) - xa_erase(&conn->sessions, sess->id); - up_write(&conn->session_lock); + if (conn) { + down_write(&conn->session_lock); + if (xa_load(&conn->sessions, sess->id) == sess) + xa_erase(&conn->sessions, sess->id); + up_write(&conn->session_lock); + } for (i = 0; i < nr_conns; i++) { if (session_conns[i] == conn) continue; @@ -519,10 +567,47 @@ void ksmbd_session_unregister(struct ksmbd_conn *conn, xa_erase(&session_conns[i]->sessions, sess->id); up_write(&session_conns[i]->session_lock); } + return removed; +} + +void ksmbd_session_unregister(struct ksmbd_conn *conn, + struct ksmbd_session *sess) +{ + bool removed; + + down_write(&sessions_table_lock); + removed = __ksmbd_session_unregister(conn, sess); up_write(&sessions_table_lock); - if (removed) + if (removed) { + /* + * Keep the table reference until procfs readers have finished, + * so none can remove its own proc entry on the last put. + */ + delete_proc_session(sess); ksmbd_user_session_put(sess); + } +} + +void ksmbd_reclaim_ended_sessions(void) +{ + LIST_HEAD(dispose); + struct ksmbd_session *sess; + struct hlist_node *tmp; + int bkt; + + down_write(&sessions_table_lock); + hash_for_each_safe(sessions_table, bkt, tmp, sess, hlist) { + if (READ_ONCE(sess->state) != SMB2_SESSION_EXPIRED || + READ_ONCE(sess->kerberos_expiry) || + atomic_read(&sess->refcnt) != 1) + continue; + + __ksmbd_session_unregister(NULL, sess); + list_add_tail(&sess->dispose_entry, &dispose); + } + up_write(&sessions_table_lock); + ksmbd_dispose_sessions(&dispose); } bool ksmbd_conn_has_valid_or_expired_session(struct ksmbd_conn *conn) @@ -532,7 +617,12 @@ bool ksmbd_conn_has_valid_or_expired_session(struct ksmbd_conn *conn) int state, bkt; bool found = false; - down_read(&conn->session_lock); + /* + * The expiration worker holds conn_list_lock, which session teardown + * takes after the session locks. Defer scavenging if either is busy. + */ + if (!down_read_trylock(&conn->session_lock)) + return true; xa_for_each(&conn->sessions, id, sess) { state = READ_ONCE(sess->state); if (state == SMB2_SESSION_VALID || @@ -546,7 +636,8 @@ bool ksmbd_conn_has_valid_or_expired_session(struct ksmbd_conn *conn) return true; /* A session bound through SMB3 multichannel is not in conn->sessions. */ - down_read(&sessions_table_lock); + if (!down_read_trylock(&sessions_table_lock)) + return true; hash_for_each(sessions_table, bkt, sess, hlist) { state = READ_ONCE(sess->state); if (state != SMB2_SESSION_VALID && @@ -592,12 +683,15 @@ static int ksmbd_chann_del(struct ksmbd_conn *conn, struct ksmbd_session *sess) if (!chann) return -ENOENT; + ksmbd_session_release_slot(chann->conn); + ksmbd_conn_put(chann->conn); kfree_sensitive(chann); return 0; } void ksmbd_conn_sessions_cleanup(struct ksmbd_conn *conn) { + LIST_HEAD(dispose); struct ksmbd_session *sess; unsigned long id; struct hlist_node *tmp; @@ -611,8 +705,7 @@ void ksmbd_conn_sessions_cleanup(struct ksmbd_conn *conn) down_write(&conn->session_lock); xa_erase(&conn->sessions, sess->id); up_write(&conn->session_lock); - if (atomic_dec_and_test(&sess->refcnt)) - ksmbd_session_destroy(sess); + list_add_tail(&sess->dispose_entry, &dispose); } } @@ -622,12 +715,12 @@ void ksmbd_conn_sessions_cleanup(struct ksmbd_conn *conn) if (xa_empty(&sess->ksmbd_chann_list)) { xa_erase(&conn->sessions, sess->id); ksmbd_session_remove_from_table(sess); - if (atomic_dec_and_test(&sess->refcnt)) - ksmbd_session_destroy(sess); + list_add_tail(&sess->dispose_entry, &dispose); } } up_write(&conn->session_lock); up_write(&sessions_table_lock); + ksmbd_dispose_sessions(&dispose); } bool is_ksmbd_session_in_connection(struct ksmbd_conn *conn, @@ -674,6 +767,37 @@ struct ksmbd_session *ksmbd_session_lookup_slowpath(unsigned long long id) return sess; } +/* The caller holds a session reference and must put each returned connection. */ +unsigned int ksmbd_session_get_channels(struct ksmbd_session *sess, + struct ksmbd_conn **conns, + unsigned int max_conns) +{ + struct channel *chann; + unsigned long index; + unsigned int nr_conns = 0; + + /* + * Channel connections are borrowed. Unregister can leave the channel + * list in a pinned session after removing it from the registry, so + * chann_lock alone does not keep those connections alive. Only collect + * registered sessions, excluding both unregister and connection cleanup + * until the connection references have been taken. + */ + down_read(&sessions_table_lock); + if (hlist_unhashed(&sess->hlist)) + goto out; + down_read(&sess->chann_lock); + xa_for_each(&sess->ksmbd_chann_list, index, chann) { + if (nr_conns == max_conns) + break; + conns[nr_conns++] = ksmbd_conn_get(chann->conn); + } + up_read(&sess->chann_lock); +out: + up_read(&sessions_table_lock); + return nr_conns; +} + struct ksmbd_session *ksmbd_session_lookup_all_states(struct ksmbd_conn *conn, unsigned long long id) { @@ -830,7 +954,7 @@ static int __init_smb2_session(struct ksmbd_session *sess) return 0; } -static struct ksmbd_session *__session_create(int protocol) +static struct ksmbd_session *__session_create(struct ksmbd_conn *conn, int protocol) { struct ksmbd_session *sess; int ret; @@ -838,27 +962,33 @@ static struct ksmbd_session *__session_create(int protocol) if (protocol != CIFDS_SESSION_FLAG_SMB2) return NULL; + if (ksmbd_session_reserve_slot(conn)) + return NULL; + sess = kzalloc_obj(struct ksmbd_session, KSMBD_DEFAULT_GFP); - if (!sess) + if (!sess) { + ksmbd_session_release_slot(conn); return NULL; + } + sess->setup_conn = ksmbd_conn_get(conn); ida_init(&sess->tree_conn_ida); - - if (ksmbd_init_file_table(&sess->file_table)) - goto error; - sess->last_active = jiffies; sess->state = SMB2_SESSION_IN_PROGRESS; set_session_flag(sess, protocol); xa_init(&sess->tree_conns); xa_init(&sess->ksmbd_chann_list); xa_init(&sess->rpc_handle_list); + INIT_LIST_HEAD(&sess->dispose_entry); sess->sequence_number = 1; atomic_set(&sess->refcnt, 2); init_rwsem(&sess->tree_conns_lock); init_rwsem(&sess->rpc_lock); init_rwsem(&sess->chann_lock); + if (ksmbd_init_file_table(&sess->file_table)) + goto error; + ret = __init_smb2_session(sess); if (ret) goto error; @@ -877,9 +1007,9 @@ static struct ksmbd_session *__session_create(int protocol) return NULL; } -struct ksmbd_session *ksmbd_smb2_session_create(void) +struct ksmbd_session *ksmbd_smb2_session_create(struct ksmbd_conn *conn) { - return __session_create(CIFDS_SESSION_FLAG_SMB2); + return __session_create(conn, CIFDS_SESSION_FLAG_SMB2); } int ksmbd_acquire_tree_conn_id(struct ksmbd_session *sess) diff --git a/fs/smb/server/mgmt/user_session.h b/fs/smb/server/mgmt/user_session.h index 217258551d6d..8e4ddb04f0e6 100644 --- a/fs/smb/server/mgmt/user_session.h +++ b/fs/smb/server/mgmt/user_session.h @@ -37,6 +37,8 @@ struct ksmbd_session { char ClientGUID[SMB2_CLIENT_GUID_SIZE]; struct ksmbd_user *user; + /* Holds the connection slot until an SMB3 channel takes it over. */ + struct ksmbd_conn *setup_conn; unsigned int sequence_number; unsigned int flags; @@ -51,6 +53,7 @@ struct ksmbd_session { u64 kerberos_expiry; struct hlist_node hlist; + struct list_head dispose_entry; struct rw_semaphore chann_lock; struct xarray ksmbd_chann_list; struct xarray tree_conns; @@ -89,15 +92,21 @@ static inline void clear_session_flag(struct ksmbd_session *sess, int bit) sess->flags &= ~bit; } -struct ksmbd_session *ksmbd_smb2_session_create(void); +struct ksmbd_session *ksmbd_smb2_session_create(struct ksmbd_conn *conn); +int ksmbd_session_reserve_slot(struct ksmbd_conn *conn); +void ksmbd_session_release_slot(struct ksmbd_conn *conn); void ksmbd_session_destroy(struct ksmbd_session *sess); struct ksmbd_session *ksmbd_session_lookup_slowpath(unsigned long long id); +unsigned int ksmbd_session_get_channels(struct ksmbd_session *sess, + struct ksmbd_conn **conns, + unsigned int max_conns); struct ksmbd_session *ksmbd_session_lookup(struct ksmbd_conn *conn, unsigned long long id); bool is_ksmbd_session_in_connection(struct ksmbd_conn *conn, unsigned long long id); +bool ksmbd_too_many_session_setups(struct ksmbd_conn *conn); int ksmbd_session_register(struct ksmbd_conn *conn, struct ksmbd_session *sess); void ksmbd_session_unregister(struct ksmbd_conn *conn, @@ -105,6 +114,7 @@ void ksmbd_session_unregister(struct ksmbd_conn *conn, void ksmbd_conn_sessions_cleanup(struct ksmbd_conn *conn); bool ksmbd_conn_has_valid_or_expired_session(struct ksmbd_conn *conn); void ksmbd_expire_sessions(void); +void ksmbd_reclaim_ended_sessions(void); struct ksmbd_session *__session_lookup(unsigned long long id); struct ksmbd_session *ksmbd_session_lookup_all(struct ksmbd_conn *conn, unsigned long long id); diff --git a/fs/smb/server/misc.c b/fs/smb/server/misc.c index 2dd91c9e956a..5037f37c9226 100644 --- a/fs/smb/server/misc.c +++ b/fs/smb/server/misc.c @@ -296,12 +296,13 @@ char *ksmbd_convert_dir_info_name(struct ksmbd_dir_info *d_info, int *conv_len) { char *conv; - int sz = min(4 * d_info->name_len, PATH_MAX); - if (!sz) + if (d_info->name_len <= 0) return NULL; - conv = kmalloc(sz, KSMBD_DEFAULT_GFP); + /* One UTF-16 code unit per input byte, plus the terminating NUL. */ + conv = kmalloc_array((size_t)d_info->name_len + 1, + sizeof(__le16), KSMBD_DEFAULT_GFP); if (!conv) return NULL; @@ -310,7 +311,6 @@ char *ksmbd_convert_dir_info_name(struct ksmbd_dir_info *d_info, d_info->name_len, local_nls, 0); *conv_len *= 2; - /* We allocate buffer twice bigger than needed. */ conv[*conv_len] = 0x00; conv[*conv_len + 1] = 0x00; return conv; diff --git a/fs/smb/server/oplock.c b/fs/smb/server/oplock.c index 1b8c3482d1e4..313bf01ad350 100644 --- a/fs/smb/server/oplock.c +++ b/fs/smb/server/oplock.c @@ -87,6 +87,7 @@ static struct oplock_info *alloc_opinfo(struct ksmbd_work *work, return NULL; opinfo->sess = sess; + opinfo->session_id = sess->id; opinfo->conn = ksmbd_conn_get(work->conn); opinfo->level = SMB2_OPLOCK_LEVEL_NONE; opinfo->op_state = OPLOCK_STATE_NONE; @@ -302,15 +303,9 @@ static struct oplock_info *opinfo_get_list(struct ksmbd_inode *ci, opinfo = list_first_entry_or_null(&ci->m_op_list, struct oplock_info, op_entry); if (opinfo) { - if (opinfo->conn == NULL || - !atomic_inc_not_zero(&opinfo->refcount)) + /* A live open may still be bound to a disconnected channel. */ + if (!atomic_inc_not_zero(&opinfo->refcount)) opinfo = NULL; - else { - if (ksmbd_conn_releasing(opinfo->conn)) { - atomic_dec(&opinfo->refcount); - opinfo = NULL; - } - } if (opinfo && snapshot && opinfo->o_fp && opinfo->o_fp != skip_fp && @@ -367,7 +362,7 @@ static void opinfo_del(struct oplock_info *opinfo) lease_del_open(opinfo); down_write(&ci->m_lock); - list_del(&opinfo->op_entry); + list_del_init(&opinfo->op_entry); up_write(&ci->m_lock); } @@ -639,19 +634,27 @@ static void grant_none_oplock(struct oplock_info *opinfo_new, static inline int compare_guid_key(struct oplock_info *opinfo, const char *guid1, const char *key1) { - const char *guid2, *key2; + struct lease *lease = opinfo->o_lease; + struct lease_table *lb; struct ksmbd_conn *conn; + int ret = 0; - conn = READ_ONCE(opinfo->conn); - if (!conn) + if (memcmp(key1, lease->lease_key, SMB2_LEASE_KEY_SIZE)) return 0; - guid2 = conn->ClientGUID; - key2 = opinfo->o_lease->lease_key; - if (!memcmp(guid1, guid2, SMB2_CLIENT_GUID_SIZE) && - !memcmp(key1, key2, SMB2_LEASE_KEY_SIZE)) - return 1; - return 0; + /* The caller holds ci->m_lock, which protects opinfo->conn. */ + conn = READ_ONCE(opinfo->conn); + if (conn) + return !memcmp(guid1, conn->ClientGUID, SMB2_CLIENT_GUID_SIZE); + + /* A detached durable lease retains its client identity in the table. */ + read_lock(&lease_list_lock); + lb = lease->l_lb; + if (lb) + ret = !memcmp(guid1, lb->client_guid, SMB2_CLIENT_GUID_SIZE); + read_unlock(&lease_list_lock); + + return ret; } /** @@ -681,7 +684,7 @@ static struct oplock_info *same_client_has_lease(struct ksmbd_inode *ci, */ down_read(&ci->m_lock); list_for_each_entry(opinfo, &ci->m_op_list, op_entry) { - if (!opinfo->is_lease || !opinfo->conn) + if (!opinfo->is_lease) continue; lease = opinfo->o_lease; @@ -734,43 +737,6 @@ static struct oplock_info *same_client_has_lease(struct ksmbd_inode *ci, return m_opinfo; } -static bool wait_for_break_ack(struct oplock_info *opinfo) -{ - int rc = 0; - - rc = wait_event_interruptible_timeout(opinfo->oplock_q, - opinfo->op_state == OPLOCK_STATE_NONE || - opinfo->op_state == OPLOCK_CLOSING, - OPLOCK_WAIT_TIME); - - /* is this a timeout ? */ - if (!rc) { - spin_lock(&opinfo->state_lock); - if (opinfo->op_state == OPLOCK_CLOSING) { - spin_unlock(&opinfo->state_lock); - return false; - } - if (opinfo->is_lease) { - opinfo->o_lease->state = SMB2_LEASE_NONE_LE; - lease_update_oplock_levels(opinfo->o_lease); - } - opinfo->level = SMB2_OPLOCK_LEVEL_NONE; - opinfo->op_state = OPLOCK_STATE_NONE; - spin_unlock(&opinfo->state_lock); - return true; - } - - return false; -} - -static void wake_up_oplock_break(struct oplock_info *opinfo) -{ - clear_bit_unlock(0, &opinfo->pending_break); - /* memory barrier is needed for wake_up_bit() */ - smp_mb__after_atomic(); - wake_up_bit(&opinfo->pending_break, 0); -} - static bool oplock_break_set_ack_wait(struct oplock_info *opinfo) { bool ret = false; @@ -826,13 +792,13 @@ static int oplock_break_pending(struct oplock_info *opinfo, int req_op_level) if (opinfo->level <= req_op_level) { if (opinfo->is_lease == false) { - wake_up_oplock_break(opinfo); + clear_and_wake_up_bit(0, &opinfo->pending_break); return 1; } if (opinfo->o_lease->state != (SMB2_LEASE_HANDLE_CACHING_LE | SMB2_LEASE_READ_CACHING_LE)) { - wake_up_oplock_break(opinfo); + clear_and_wake_up_bit(0, &opinfo->pending_break); return 1; } } @@ -850,330 +816,492 @@ static bool lease_break_needed(struct oplock_info *opinfo, int req_op_level, return opinfo->level > req_op_level; } -/** - * __smb2_oplock_break_noti() - send smb2 oplock break cmd from conn - * to client - * @wk: smb work object - * - * There are two ways this function can be called. 1- while file open we break - * from exclusive/batch lock to levelII oplock and 2- while file write/truncate - * we break from levelII oplock no oplock. - * work->request_buf contains oplock_info. - */ -static void __smb2_oplock_break_noti(struct work_struct *wk) +struct smb2_break_channel { + struct list_head list; + struct ksmbd_conn *conn; + struct ksmbd_tx_tracking tracking; + bool queued; + bool delivered; + bool failed; +}; + +static bool smb2_break_complete(struct oplock_info *opinfo, bool expect_ack) { - struct smb2_oplock_break *rsp = NULL; - struct ksmbd_work *work = container_of(wk, struct ksmbd_work, work); - struct ksmbd_conn *conn = work->conn; - struct oplock_break_info *br_info = work->request_buf; - struct smb2_hdr *rsp_hdr; - struct ksmbd_file *fp; + bool complete; - fp = ksmbd_lookup_global_fd(br_info->fid); - if (!fp) - goto out; + spin_lock(&opinfo->state_lock); + complete = opinfo->op_state == OPLOCK_CLOSING || + (expect_ack && opinfo->op_state == OPLOCK_STATE_NONE); + spin_unlock(&opinfo->state_lock); + return complete; +} - if (allocate_interim_rsp_buf(work)) { - pr_err("smb2_allocate_rsp_buf failed! "); - ksmbd_fd_put(work, fp); - goto out; +static int smb2_break_failed(struct oplock_info *opinfo, bool expect_ack) +{ + spin_lock(&opinfo->state_lock); + /* An ACK or close which won the race has already completed the break. */ + if (opinfo->op_state == OPLOCK_CLOSING || + (expect_ack && opinfo->op_state == OPLOCK_STATE_NONE)) { + spin_unlock(&opinfo->state_lock); + return 0; } + if (opinfo->is_lease) { + opinfo->o_lease->state = SMB2_LEASE_NONE_LE; + opinfo->o_lease->new_state = SMB2_LEASE_NONE_LE; + lease_update_oplock_levels(opinfo->o_lease); + atomic_set(&opinfo->breaking_cnt, 0); + } + opinfo->level = SMB2_OPLOCK_LEVEL_NONE; + opinfo->op_state = OPLOCK_STATE_NONE; + spin_unlock(&opinfo->state_lock); + wake_up_interruptible_all(&opinfo->oplock_q); + wake_up_interruptible_all(&opinfo->oplock_brk); + return ksmbd_invalidate_durable_fd(opinfo->fid); +} - rsp_hdr = smb_get_msg(work->response_buf); - memset(rsp_hdr, 0, sizeof(struct smb2_hdr) + 2); - rsp_hdr->ProtocolId = SMB2_PROTO_NUMBER; - rsp_hdr->StructureSize = SMB2_HEADER_STRUCTURE_SIZE; - rsp_hdr->CreditRequest = cpu_to_le16(0); - rsp_hdr->Command = SMB2_OPLOCK_BREAK; - rsp_hdr->Flags = (SMB2_FLAGS_SERVER_TO_REDIR); - rsp_hdr->NextCommand = 0; - rsp_hdr->MessageId = cpu_to_le64(-1); - rsp_hdr->Id.SyncId.ProcessId = 0; - rsp_hdr->Id.SyncId.TreeId = 0; - rsp_hdr->SessionId = 0; - memset(rsp_hdr->Signature, 0, 16); - - rsp = smb_get_msg(work->response_buf); - - rsp->StructureSize = cpu_to_le16(24); - if (!br_info->open_trunc && - (br_info->level == SMB2_OPLOCK_LEVEL_BATCH || - br_info->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE)) - rsp->OplockLevel = SMB2_OPLOCK_LEVEL_II; - else - rsp->OplockLevel = SMB2_OPLOCK_LEVEL_NONE; - rsp->Reserved = 0; - rsp->Reserved2 = 0; - rsp->PersistentFid = fp->persistent_id; - rsp->VolatileFid = fp->volatile_id; - - ksmbd_fd_put(work, fp); - if (ksmbd_iov_pin_rsp(work, (void *)rsp, - sizeof(struct smb2_oplock_break))) - goto out; - - ksmbd_debug(OPLOCK, - "sending oplock break v_id %llu p_id = %llu lock level = %d\n", - rsp->VolatileFid, rsp->PersistentFid, rsp->OplockLevel); - - ksmbd_conn_write(work); +static int smb2_break_add_channel(struct list_head *head, struct ksmbd_conn *conn) +{ + struct smb2_break_channel *channel; -out: - ksmbd_free_work_struct(work); - ksmbd_conn_r_count_dec(conn); - ksmbd_conn_put(conn); + channel = kzalloc_obj(*channel, KSMBD_DEFAULT_GFP); + if (!channel) + return -ENOMEM; + channel->conn = ksmbd_conn_get(conn); + list_add_tail(&channel->list, head); + return 0; } /* - * Select and pin the connection used for an oplock break before doing any - * allocations which may sleep. The caller of oplock_break() holds a live - * reference on ci (a file being opened, a file being operated on, or an - * explicit ksmbd_inode_lookup_lock() reference in the parent lease break - * paths), so the inode cannot be freed during the call and its lock is - * reachable without dereferencing opinfo->o_fp, which is not pinned by - * the oplock reference and may be freed by a concurrent close. - * - * opinfo->conn is cleared under ci->m_lock by session_fd_check() when the - * durable handle owning the oplock is disconnected, reassigned by - * ksmbd_reopen_durable_fd() under the same lock, and the last - * ksmbd_conn_put() of the old connection frees it. Holding the read lock - * excludes both writers, so the connection cannot be freed while it is - * selected. + * Snapshot connections before sending. References keep their transports alive, + * while r_count is taken separately for each write, under request_lock. */ -static struct ksmbd_conn *smb2_oplock_break_conn_get(struct oplock_info *opinfo, - struct ksmbd_inode *ci) -{ +static int smb2_break_collect_channels(struct oplock_info *opinfo, + struct ksmbd_inode *ci, + struct smb2_break_channel *owner, + struct list_head *head, + struct smb2_oplock_break *rsp) +{ + struct ksmbd_session *sess = NULL; + struct ksmbd_conn *session_conns[KSMBD_MAX_CHANNELS]; + struct ksmbd_file_table *ft; + struct ksmbd_file *fp; struct ksmbd_conn *conn; + struct lease_table *lb; + char guid[SMB2_CLIENT_GUID_SIZE]; + u64 session_id = 0; + unsigned int nr_conns, i; + bool have_guid = false; + int bkt, ret = 0; + /* m_lock protects membership and binding, but not the session's lifetime. */ down_read(&ci->m_lock); - conn = READ_ONCE(opinfo->conn); - if (conn && !ksmbd_conn_releasing(conn)) - conn = ksmbd_conn_get(conn); - else - conn = NULL; + if (!list_empty(&opinfo->op_entry)) { + if (!opinfo->is_lease) { + rsp->PersistentFid = READ_ONCE(opinfo->o_fp->persistent_id); + rsp->VolatileFid = READ_ONCE(opinfo->o_fp->volatile_id); + if (!has_file_id(rsp->PersistentFid) || + !has_file_id(rsp->VolatileFid)) { + /* A detached oplock must take the failed-break path. */ + ret = opinfo->conn ? -ENOENT : -ENOTCONN; + goto unlock; + } + session_id = opinfo->session_id; + } + conn = opinfo->conn; + if (conn) { + owner->conn = ksmbd_conn_get(conn); + memcpy(guid, conn->ClientGUID, sizeof(guid)); + have_guid = true; + } + } else { + ret = -ENOENT; + } +unlock: up_read(&ci->m_lock); + if (ret) + return ret; + + if (opinfo->is_lease && !have_guid) { + read_lock(&lease_list_lock); + lb = opinfo->o_lease->l_lb; + if (lb) { + memcpy(guid, lb->client_guid, sizeof(guid)); + have_guid = true; + if (lb->conn) + owner->conn = ksmbd_conn_get(lb->conn); + } + read_unlock(&lease_list_lock); + } + if (owner->conn) + list_add_tail(&owner->list, head); + + if (!opinfo->is_lease) { + /* Session teardown takes m_lock while holding the session table lock. */ + sess = ksmbd_session_lookup_slowpath(session_id); + if (!sess) + return -ENOENT; - return conn; + /* Revalidate the binding and IDs after taking a safe session reference. */ + down_read(&ci->m_lock); + if (list_empty(&opinfo->op_entry) || opinfo->sess != sess) { + ret = -ENOENT; + } else { + ft = &sess->file_table; + read_lock(&ft->lock); + fp = opinfo->o_fp; + if (fp->f_state == FP_CLOSED || + !has_file_id(fp->volatile_id) || + idr_find(ft->idr, fp->volatile_id) != fp) { + ret = -ENOENT; + } else { + rsp->PersistentFid = READ_ONCE(fp->persistent_id); + rsp->VolatileFid = fp->volatile_id; + } + read_unlock(&ft->lock); + } + up_read(&ci->m_lock); + if (ret) { + ksmbd_user_session_put(sess); + return ret; + } + } + + if (opinfo->is_lease && have_guid) { + down_read(&conn_list_lock); + hash_for_each(conn_list, bkt, conn, hlist) { + if (conn == owner->conn || !ksmbd_conn_good(conn) || + memcmp(conn->ClientGUID, guid, sizeof(guid))) + continue; + ret = smb2_break_add_channel(head, conn); + if (ret) + break; + } + up_read(&conn_list_lock); + } else if (!opinfo->is_lease && sess && sess->dialect >= SMB30_PROT_ID) { + nr_conns = ksmbd_session_get_channels(sess, session_conns, + ARRAY_SIZE(session_conns)); + for (i = 0; i < nr_conns; i++) { + conn = session_conns[i]; + if (!ret && conn && conn != owner->conn && ksmbd_conn_good(conn)) + ret = smb2_break_add_channel(head, conn); + /* Drop all snapshot references, also on allocation failure. */ + ksmbd_conn_put(conn); + } + } + ksmbd_user_session_put(sess); + + /* Allocation pressure may limit retries, but must not discard the owner. */ + return list_empty(head) ? (ret ?: -ENOTCONN) : 0; } -/** - * smb2_oplock_break_noti() - send smb2 exclusive/batch to level2 oplock - * break command from server to client - * @opinfo: oplock info object - * @ci: inode owning the break target's oplock list, pinned by - * the caller - * - * Return: 0 on success, otherwise error - */ -static int smb2_oplock_break_noti(struct oplock_info *opinfo, - struct ksmbd_inode *ci) +static bool smb2_break_delivered(struct list_head *head, bool require_live) { + struct smb2_break_channel *channel; struct ksmbd_conn *conn; - struct oplock_break_info *br_info; - int ret = 0; - struct ksmbd_work *work; - conn = smb2_oplock_break_conn_get(opinfo, ci); - if (!conn) - return ksmbd_invalidate_durable_fd(opinfo->fid); - - work = ksmbd_alloc_work_struct(); - if (!work) { - ksmbd_conn_put(conn); - return -ENOMEM; + list_for_each_entry(channel, head, list) { + conn = channel->conn; + if (!channel->queued || (require_live && !ksmbd_conn_good(conn))) + continue; + /* A synchronous send needs no separate delivery probe. */ + if (channel->delivered || !channel->tracking.valid || + conn->transport->ops->is_acked(conn->transport, + &channel->tracking)) { + channel->delivered = true; + return true; + } } + return false; +} - br_info = kmalloc_obj(struct oplock_break_info, KSMBD_DEFAULT_GFP); - if (!br_info) { - ksmbd_free_work_struct(work); - ksmbd_conn_put(conn); - return -ENOMEM; - } +static bool smb2_break_has_unsent_channels(struct list_head *head) +{ + struct smb2_break_channel *channel; - br_info->level = opinfo->level; - br_info->fid = opinfo->fid; - br_info->open_trunc = opinfo->open_trunc; + list_for_each_entry(channel, head, list) { + if (!channel->queued && !channel->failed) + return true; + } + return false; +} - work->request_buf = (char *)br_info; - /* Transfer the reference acquired by smb2_oplock_break_conn_get(). */ - work->conn = conn; - work->sess = opinfo->sess; +static int smb2_break_wait(struct oplock_info *opinfo, struct list_head *head, + struct smb2_break_channel *sent_channel, + bool expect_ack, bool wait_ack, bool can_retry, + unsigned long retry_at, unsigned long deadline) +{ + unsigned long remaining; + long ret; - ksmbd_conn_r_count_inc(conn); - if (opinfo->op_state == OPLOCK_ACK_WAIT) { - INIT_WORK(&work->work, __smb2_oplock_break_noti); - ksmbd_queue_work(work); + for (;;) { + if (smb2_break_complete(opinfo, expect_ack)) + return 0; + if (!wait_ack && smb2_break_delivered(head, false)) + return 0; + if (time_after_eq(jiffies, deadline)) + return -ETIMEDOUT; + /* TCP delivery followed by a slow SMB ACK is not a send failure. */ + if (can_retry && !smb2_break_delivered(head, true) && + ((sent_channel && !ksmbd_conn_good(sent_channel->conn)) || + time_after_eq(jiffies, retry_at))) + return -EAGAIN; - if (wait_for_break_ack(opinfo)) - ret = ksmbd_invalidate_durable_fd(opinfo->fid); - } else { - __smb2_oplock_break_noti(&work->work); - if (opinfo->level == SMB2_OPLOCK_LEVEL_II) - opinfo->level = SMB2_OPLOCK_LEVEL_NONE; + remaining = min_t(unsigned long, deadline - jiffies, + max_t(unsigned long, 1, HZ / 10)); + ret = wait_event_interruptible_timeout(opinfo->oplock_q, + smb2_break_complete(opinfo, expect_ack), + remaining); + if (ret < 0) + return ret; } - return ret; } -/** - * __smb2_lease_break_noti() - send lease break command from server - * to client - * @wk: smb work object - */ -static void __smb2_lease_break_noti(struct work_struct *wk) +static void smb2_break_init_header(struct smb2_hdr *hdr) { - struct smb2_lease_break *rsp = NULL; - struct ksmbd_work *work = container_of(wk, struct ksmbd_work, work); - struct ksmbd_conn *conn = work->conn; - struct lease_break_info *br_info = work->request_buf; - struct smb2_hdr *rsp_hdr; + memset(hdr, 0, sizeof(*hdr)); + hdr->ProtocolId = SMB2_PROTO_NUMBER; + hdr->StructureSize = SMB2_HEADER_STRUCTURE_SIZE; + hdr->Command = SMB2_OPLOCK_BREAK; + hdr->Flags = SMB2_FLAGS_SERVER_TO_REDIR; + hdr->MessageId = cpu_to_le64(-1); +} - if (allocate_interim_rsp_buf(work)) { - ksmbd_debug(OPLOCK, "smb2_allocate_rsp_buf failed! "); - goto out; +/* Build one notification and reuse its payload and epoch on each connection. */ +static int smb2_break_notify(struct oplock_info *opinfo, struct ksmbd_inode *ci, + bool expect_ack, bool wait_ack, bool inc_epoch, + struct work_struct *interim_work) +{ + struct smb2_break_channel owner = {}, *channel, *tmp, *last = NULL; + struct ksmbd_work *work; + struct smb2_oplock_break *oplock_rsp; + struct smb2_lease_break *lease_rsp; + struct lease *lease = opinfo->o_lease; + unsigned long deadline, retry_at; + int ret, size; + LIST_HEAD(channels); + + if (smb2_break_complete(opinfo, expect_ack)) { + ret = 0; + goto out_interim; + } + wait_ack = wait_ack && expect_ack; + work = ksmbd_alloc_work_struct(); + if (!work) { + ret = smb2_break_failed(opinfo, expect_ack); + goto out_interim; } + ret = allocate_interim_rsp_buf(work); + if (ret) + goto failed; + oplock_rsp = smb_get_msg(work->response_buf); + smb2_break_init_header(&oplock_rsp->hdr); - rsp_hdr = smb_get_msg(work->response_buf); - memset(rsp_hdr, 0, sizeof(struct smb2_hdr) + 2); - rsp_hdr->ProtocolId = SMB2_PROTO_NUMBER; - rsp_hdr->StructureSize = SMB2_HEADER_STRUCTURE_SIZE; - rsp_hdr->CreditRequest = cpu_to_le16(0); - rsp_hdr->Command = SMB2_OPLOCK_BREAK; - rsp_hdr->Flags = (SMB2_FLAGS_SERVER_TO_REDIR); - rsp_hdr->NextCommand = 0; - rsp_hdr->MessageId = cpu_to_le64(-1); - rsp_hdr->Id.SyncId.ProcessId = 0; - rsp_hdr->Id.SyncId.TreeId = 0; - rsp_hdr->SessionId = 0; - memset(rsp_hdr->Signature, 0, 16); - - rsp = smb_get_msg(work->response_buf); - rsp->StructureSize = cpu_to_le16(44); - rsp->Epoch = br_info->epoch; - rsp->Flags = 0; - - if (br_info->curr_state & (SMB2_LEASE_WRITE_CACHING_LE | - SMB2_LEASE_HANDLE_CACHING_LE)) - rsp->Flags = SMB2_NOTIFY_BREAK_LEASE_FLAG_ACK_REQUIRED; - - memcpy(rsp->LeaseKey, br_info->lease_key, SMB2_LEASE_KEY_SIZE); - rsp->CurrentLeaseState = br_info->curr_state; - rsp->NewLeaseState = br_info->new_state; - rsp->BreakReason = 0; - rsp->AccessMaskHint = 0; - rsp->ShareMaskHint = 0; - - if (ksmbd_iov_pin_rsp(work, (void *)rsp, - sizeof(struct smb2_lease_break))) + ret = smb2_break_collect_channels(opinfo, ci, &owner, &channels, oplock_rsp); + if (ret == -ENOENT) goto out; + if (ret) + goto failed; - ksmbd_conn_write(work); - + spin_lock(&opinfo->state_lock); + if (opinfo->op_state == OPLOCK_CLOSING || + (expect_ack && opinfo->op_state == OPLOCK_STATE_NONE)) { + spin_unlock(&opinfo->state_lock); + ret = 0; + goto out; + } + if (opinfo->is_lease) { + lease_rsp = smb_get_msg(work->response_buf); + lease_rsp->StructureSize = cpu_to_le16(44); + if (lease->version == 2) { + if (inc_epoch) + lease->epoch++; + lease_rsp->Epoch = cpu_to_le16(lease->epoch); + } + if (lease->state & (SMB2_LEASE_WRITE_CACHING_LE | + SMB2_LEASE_HANDLE_CACHING_LE)) + lease_rsp->Flags = SMB2_NOTIFY_BREAK_LEASE_FLAG_ACK_REQUIRED; + memcpy(lease_rsp->LeaseKey, lease->lease_key, SMB2_LEASE_KEY_SIZE); + lease_rsp->CurrentLeaseState = lease->state; + lease_rsp->NewLeaseState = lease->new_state; + size = sizeof(*lease_rsp); + } else { + oplock_rsp->StructureSize = cpu_to_le16(24); + oplock_rsp->OplockLevel = !opinfo->open_trunc && + (opinfo->level == SMB2_OPLOCK_LEVEL_BATCH || + opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE) ? + SMB2_OPLOCK_LEVEL_II : SMB2_OPLOCK_LEVEL_NONE; + /* + * A level II break needs no ACK. Publish NONE before sending so + * an unsolicited ACK cannot observe II while delivery is pending. + */ + if (!expect_ack) + opinfo->level = SMB2_OPLOCK_LEVEL_NONE; + size = sizeof(*oplock_rsp); + } + spin_unlock(&opinfo->state_lock); + ret = ksmbd_iov_pin_rsp(work, oplock_rsp, size); + if (ret) + goto failed; + + deadline = jiffies + OPLOCK_WAIT_TIME; +retry: + list_for_each_entry(channel, &channels, list) { + if (channel->queued || channel->failed) + continue; + if (smb2_break_complete(opinfo, expect_ack)) { + ret = 0; + goto out; + } + if (time_after_eq(jiffies, deadline)) { + ret = -ETIMEDOUT; + break; + } + if (!ksmbd_conn_r_count_inc_if_good(channel->conn)) { + channel->failed = true; + ret = -ENOTCONN; + continue; + } + work->conn = channel->conn; + ret = ksmbd_conn_write_tracked(work, &channel->tracking); + ksmbd_conn_r_count_dec(channel->conn); + if (ret) { + /* Only a nonblocking send guarantees no bytes on -EAGAIN. */ + if (ret != -EAGAIN || !channel->conn->transport->ops->is_acked) + channel->failed = true; + continue; + } + channel->queued = true; + if (interim_work) { + ksmbd_queue_io_work(interim_work); + interim_work = NULL; + } + last = channel; + retry_at = jiffies + channel->tracking.retry_after; + ret = smb2_break_wait(opinfo, &channels, channel, expect_ack, + wait_ack, smb2_break_has_unsent_channels(&channels), + retry_at, deadline); + if (ret != -EAGAIN) + goto done; + } + if (smb2_break_has_unsent_channels(&channels)) { + /* Allow busy writers to make progress before another nonblocking pass. */ + retry_at = jiffies + max_t(unsigned long, 1, HZ / 10); + ret = smb2_break_wait(opinfo, &channels, NULL, expect_ack, + wait_ack, true, retry_at, deadline); + if (ret == -EAGAIN) + goto retry; + } else if (last) { + /* Give already queued notifications the rest of the shared wait. */ + ret = smb2_break_wait(opinfo, &channels, last, expect_ack, + wait_ack, false, deadline, deadline); + } + +done: + if (ret) + goto failed; + if (!expect_ack && opinfo->is_lease) { + spin_lock(&opinfo->state_lock); + if (opinfo->op_state != OPLOCK_CLOSING) { + lease->state = lease->new_state; + lease_update_oplock_levels(lease); + } + spin_unlock(&opinfo->state_lock); + } + goto out; +failed: + ret = smb2_break_failed(opinfo, expect_ack); out: ksmbd_free_work_struct(work); - ksmbd_conn_r_count_dec(conn); - ksmbd_conn_put(conn); + list_for_each_entry_safe(channel, tmp, &channels, list) { + list_del(&channel->list); + ksmbd_conn_put(channel->conn); + if (channel != &owner) + kfree(channel); + } +out_interim: + if (interim_work) + ksmbd_queue_io_work(interim_work); + return ret; } -/* - * Select and pin the connection used for a lease break before doing any - * allocations which may sleep. opinfo->conn is cleared under ci->m_lock, - * while lease->l_lb and the lease table lifetime are protected by - * lease_list_lock. - */ -static struct ksmbd_conn *smb2_lease_break_conn_get(struct oplock_info *opinfo) +struct smb2_interim_work { + struct work_struct work; + struct ksmbd_work *in_work; +}; + +static void smb2_break_interim_work(struct work_struct *work) { - struct lease *lease = opinfo->o_lease; - struct lease_table *lb; - struct ksmbd_conn *conn; + struct smb2_interim_work *iw = container_of(work, struct smb2_interim_work, work); - /* Keep the connection which owns the open, when it is still active. */ - down_read(&lease->ci->m_lock); - conn = READ_ONCE(opinfo->conn); - if (conn && !ksmbd_conn_releasing(conn)) - conn = ksmbd_conn_get(conn); - else - conn = NULL; - up_read(&lease->ci->m_lock); + if (!setup_async_work(iw->in_work, NULL, NULL)) { + smb2_send_interim_resp(iw->in_work, STATUS_PENDING); + release_async_work(iw->in_work); + } +} - if (conn || lease->version != 2) - return conn; +struct smb2_break_work { + struct work_struct work; + struct oplock_info *opinfo; + struct ksmbd_inode *ci; + bool expect_ack; + bool wait_ack; + bool inc_epoch; + int ret; +}; - /* Otherwise route v2 lease breaks through the shared lease channel. */ - read_lock(&lease_list_lock); - lb = lease->l_lb; - if (lb && lb->conn && !ksmbd_conn_releasing(lb->conn)) - conn = ksmbd_conn_get(lb->conn); - read_unlock(&lease_list_lock); +static void smb2_break_notify_work(struct work_struct *work) +{ + struct smb2_break_work *bw = container_of(work, struct smb2_break_work, work); - return conn; + bw->ret = smb2_break_notify(bw->opinfo, bw->ci, bw->expect_ack, + bw->wait_ack, bw->inc_epoch, NULL); } -/** - * smb2_lease_break_noti() - break lease when a new client request - * write lease - * @opinfo: contains lease state information - * @sync: send the lease break notification synchronously - * @inc_epoch: increment the lease epoch before sending the break - * - * Return: 0 on success, otherwise error - */ -static int smb2_lease_break_noti(struct oplock_info *opinfo, bool sync, - bool inc_epoch) +static int smb2_break_notify_with_interim(struct oplock_info *opinfo, + struct ksmbd_inode *ci, + bool expect_ack, bool wait_ack, + bool inc_epoch, struct ksmbd_work *in_work, + bool notify_first) { - struct ksmbd_conn *conn; - struct ksmbd_work *work; - struct lease_break_info *br_info; - struct lease *lease = opinfo->o_lease; + struct smb2_break_work bw = { + .opinfo = opinfo, + .ci = ci, + .expect_ack = expect_ack, + .wait_ack = wait_ack, + .inc_epoch = inc_epoch, + }; - conn = smb2_lease_break_conn_get(opinfo); - if (!conn) - return ksmbd_invalidate_durable_fd(opinfo->fid); + if (!in_work) + return smb2_break_notify(opinfo, ci, expect_ack, wait_ack, inc_epoch, NULL); - work = ksmbd_alloc_work_struct(); - if (!work) { - ksmbd_conn_put(conn); - return -ENOMEM; - } + if (notify_first) { + struct smb2_interim_work iw = { .in_work = in_work }; + int ret; - br_info = kmalloc_obj(struct lease_break_info, KSMBD_DEFAULT_GFP); - if (!br_info) { - ksmbd_free_work_struct(work); - ksmbd_conn_put(conn); - return -ENOMEM; + /* + * Rename sends the first notification in the request context. + * Queue the interim writer after that send so a blocked response + * cannot stop notification retries or the break ACK wait. + */ + INIT_WORK_ONSTACK(&iw.work, smb2_break_interim_work); + ret = smb2_break_notify(opinfo, ci, expect_ack, wait_ack, + inc_epoch, &iw.work); + flush_work(&iw.work); + destroy_work_on_stack(&iw.work); + return ret; } - br_info->curr_state = lease->state; - br_info->new_state = lease->new_state; - if (lease->version == 2) { - if (inc_epoch) - lease->epoch++; - br_info->epoch = cpu_to_le16(lease->epoch); - } else { - br_info->epoch = 0; + /* A blocked interim writer must not stop notification sends or retries. */ + INIT_WORK_ONSTACK(&bw.work, smb2_break_notify_work); + ksmbd_queue_break_work(&bw.work); + if (!setup_async_work(in_work, NULL, NULL)) { + smb2_send_interim_resp(in_work, STATUS_PENDING); + release_async_work(in_work); } - memcpy(br_info->lease_key, lease->lease_key, SMB2_LEASE_KEY_SIZE); - - work->request_buf = (char *)br_info; - /* Transfer the reference acquired by smb2_lease_break_conn_get(). */ - work->conn = conn; - work->sess = opinfo->sess; - - ksmbd_conn_r_count_inc(conn); - if (opinfo->op_state == OPLOCK_ACK_WAIT) { - if (sync) { - __smb2_lease_break_noti(&work->work); - } else { - INIT_WORK(&work->work, __smb2_lease_break_noti); - ksmbd_queue_work(work); - } - } else { - __smb2_lease_break_noti(&work->work); - if (opinfo->o_lease->new_state == SMB2_LEASE_NONE_LE) { - opinfo->o_lease->state = SMB2_LEASE_NONE_LE; - lease_update_oplock_levels(opinfo->o_lease); - } - } - return 0; + /* The caller keeps opinfo and ci alive until the notifier has finished. */ + flush_work(&bw.work); + destroy_work_on_stack(&bw.work); + return bw.ret; } static void wait_lease_breaking(struct oplock_info *opinfo) @@ -1195,10 +1323,11 @@ static void wait_lease_breaking(struct oplock_info *opinfo) static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci, int req_op_level, struct ksmbd_work *in_work, - bool share_break, bool sync_lease_break) + bool share_break, bool notify_first) { int err = 0; bool sent_interim = false; + bool expect_ack = false; /* Need to break exclusive/batch oplock, write lease or overwrite_if */ ksmbd_debug(OPLOCK, @@ -1256,11 +1385,12 @@ static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci, } } - if (lease->state & (SMB2_LEASE_WRITE_CACHING_LE | - SMB2_LEASE_HANDLE_CACHING_LE)) { + expect_ack = lease->state & (SMB2_LEASE_WRITE_CACHING_LE | + SMB2_LEASE_HANDLE_CACHING_LE); + if (expect_ack) { if (!oplock_break_set_ack_wait(brk_opinfo)) { atomic_dec_if_positive(&brk_opinfo->breaking_cnt); - wake_up_oplock_break(brk_opinfo); + clear_and_wake_up_bit(0, &brk_opinfo->pending_break); return -ENOENT; } } else @@ -1273,16 +1403,12 @@ static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci, inc_epoch = false; lease->reuse_epoch = false; } - err = smb2_lease_break_noti(brk_opinfo, sync_lease_break, inc_epoch); + err = smb2_break_notify_with_interim(brk_opinfo, ci, expect_ack, + wait_ack, inc_epoch, + sent_interim ? NULL : in_work, + notify_first); + sent_interim = true; inc_epoch = false; - if (in_work && !sent_interim) { - setup_async_work(in_work, NULL, NULL); - smb2_send_interim_resp(in_work, STATUS_PENDING); - release_async_work(in_work); - sent_interim = true; - } - if (wait_ack && !err && wait_for_break_ack(brk_opinfo)) - err = ksmbd_invalidate_durable_fd(brk_opinfo->fid); ksmbd_debug(OPLOCK, "oplock granted = %d\n", brk_opinfo->level); if (brk_opinfo->op_state == OPLOCK_CLOSING) @@ -1309,7 +1435,7 @@ static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci, lease->state != SMB2_LEASE_NONE_LE))) goto again; - wake_up_oplock_break(brk_opinfo); + clear_and_wake_up_bit(0, &brk_opinfo->pending_break); return err; } else { err = oplock_break_pending(brk_opinfo, req_op_level); @@ -1318,31 +1444,21 @@ static int oplock_break(struct oplock_info *brk_opinfo, struct ksmbd_inode *ci, if (brk_opinfo->level == SMB2_OPLOCK_LEVEL_BATCH || brk_opinfo->level == SMB2_OPLOCK_LEVEL_EXCLUSIVE) { + expect_ack = true; if (!oplock_break_set_ack_wait(brk_opinfo)) { - wake_up_oplock_break(brk_opinfo); + clear_and_wake_up_bit(0, &brk_opinfo->pending_break); return -ENOENT; } } - - /* - * Keep a conflicting CREATE asynchronous while waiting for an - * oplock-break acknowledgement. Besides avoiding a blocked client - * request, this lets a replay arrive while the original CREATE is - * still pending and be rejected with FILE_NOT_AVAILABLE. - */ - if (in_work) { - setup_async_work(in_work, NULL, NULL); - smb2_send_interim_resp(in_work, STATUS_PENDING); - release_async_work(in_work); - } } - err = smb2_oplock_break_noti(brk_opinfo, ci); + err = smb2_break_notify_with_interim(brk_opinfo, ci, expect_ack, + true, false, in_work, false); ksmbd_debug(OPLOCK, "oplock granted = %d\n", brk_opinfo->level); if (brk_opinfo->op_state == OPLOCK_CLOSING) err = -EAGAIN; - wake_up_oplock_break(brk_opinfo); + clear_and_wake_up_bit(0, &brk_opinfo->pending_break); return err; } @@ -1499,7 +1615,7 @@ void smb_send_parent_lease_break_noti(struct ksmbd_file *fp, down_read(&p_ci->m_lock); list_for_each_entry(opinfo, &p_ci->m_op_list, op_entry) { - if (opinfo->conn == NULL || !opinfo->is_lease) + if (!opinfo->is_lease) continue; if (opinfo->o_lease->state != SMB2_OPLOCK_LEVEL_NONE && @@ -1510,11 +1626,6 @@ void smb_send_parent_lease_break_noti(struct ksmbd_file *fp, if (!atomic_inc_not_zero(&opinfo->refcount)) continue; - if (ksmbd_conn_releasing(opinfo->conn)) { - opinfo_put(opinfo); - continue; - } - if (oplock_break_add(&brk_list, opinfo)) opinfo_put(opinfo); } @@ -1547,18 +1658,13 @@ void smb_lazy_parent_lease_break_close(struct ksmbd_file *fp) down_read(&p_ci->m_lock); list_for_each_entry(opinfo, &p_ci->m_op_list, op_entry) { - if (opinfo->conn == NULL || !opinfo->is_lease) + if (!opinfo->is_lease) continue; if (opinfo->o_lease->state != SMB2_OPLOCK_LEVEL_NONE) { if (!atomic_inc_not_zero(&opinfo->refcount)) continue; - if (ksmbd_conn_releasing(opinfo->conn)) { - opinfo_put(opinfo); - continue; - } - if (oplock_break_add(&brk_list, opinfo)) opinfo_put(opinfo); } @@ -1707,7 +1813,11 @@ int smb_grant_oplock(struct ksmbd_work *work, int req_op_level, u64 pid, err = oplock_break(prev_opinfo, ci, break_level, work, share_ret < 0 && prev_opinfo->is_lease, false); - if (prev_durable_detached || (prev_durable_open && err == -ENOENT)) + /* Keep a detached durable lease if its break retains handle caching. */ + if ((prev_durable_detached && + (!prev_op_has_lease || + !(prev_opinfo->o_lease->state & SMB2_LEASE_HANDLE_CACHING_LE))) || + (prev_durable_open && err == -ENOENT)) ksmbd_invalidate_durable_fd(prev_fid); opinfo_put(prev_opinfo); if (err == -EAGAIN) { @@ -1811,8 +1921,7 @@ static bool smb_break_all_write_oplock(struct ksmbd_work *work, } brk_opinfo->open_trunc = is_trunc; - oplock_break(brk_opinfo, fp->f_ci, SMB2_OPLOCK_LEVEL_II, work, false, - false); + oplock_break(brk_opinfo, fp->f_ci, SMB2_OPLOCK_LEVEL_II, work, false, false); sent_break = true; opinfo_put(brk_opinfo); @@ -1827,12 +1936,12 @@ static bool smb_break_all_write_oplock(struct ksmbd_work *work, * @is_trunc: truncate on open * @send_interim: send interim response to the client * @send_oplock_break: send oplock break notification to the client - * @sync_lease_break: send the lease break notification synchronously + * @notify_first: send the lease break notification before the interim response */ static void __smb_break_all_levII_oplock(struct ksmbd_work *work, struct ksmbd_file *fp, int is_trunc, bool send_interim, bool send_oplock_break, - bool sync_lease_break) + bool notify_first) { struct oplock_info *op, *brk_op; struct oplock_break_entry *ent, *tmp; @@ -1850,17 +1959,9 @@ static void __smb_break_all_levII_oplock(struct ksmbd_work *work, down_read(&ci->m_lock); list_for_each_entry(brk_op, &ci->m_op_list, op_entry) { - if (brk_op->conn == NULL) - continue; - if (!atomic_inc_not_zero(&brk_op->refcount)) continue; - if (ksmbd_conn_releasing(brk_op->conn)) { - opinfo_put(brk_op); - continue; - } - if (!brk_op->is_lease && brk_op->level != SMB2_OPLOCK_LEVEL_II) { ksmbd_debug(OPLOCK, "unexpected oplock(0x%x)\n", @@ -1875,10 +1976,8 @@ static void __smb_break_all_levII_oplock(struct ksmbd_work *work, goto next; if (op && op->is_lease && brk_op->is_lease && - !memcmp(conn->ClientGUID, brk_op->conn->ClientGUID, - SMB2_CLIENT_GUID_SIZE) && - !memcmp(op->o_lease->lease_key, brk_op->o_lease->lease_key, - SMB2_LEASE_KEY_SIZE)) + compare_guid_key(brk_op, conn->ClientGUID, + op->o_lease->lease_key)) goto next; brk_op->open_trunc = is_trunc; @@ -1908,7 +2007,7 @@ static void __smb_break_all_levII_oplock(struct ksmbd_work *work, brk_op->is_lease && !is_trunc ? SMB2_OPLOCK_LEVEL_II : SMB2_OPLOCK_LEVEL_NONE, send_interim && !sent_interim ? work : NULL, - false, sync_lease_break); + false, notify_first); } sent_interim = true; list_del(&ent->list); diff --git a/fs/smb/server/oplock.h b/fs/smb/server/oplock.h index b08d21758e07..e114e2148c30 100644 --- a/fs/smb/server/oplock.h +++ b/fs/smb/server/oplock.h @@ -62,6 +62,7 @@ struct lease { struct oplock_info { struct ksmbd_conn *conn; struct ksmbd_session *sess; + u64 session_id; struct ksmbd_work *work; struct ksmbd_file *o_fp; int level; @@ -82,19 +83,6 @@ struct oplock_info { struct rcu_head rcu; }; -struct lease_break_info { - __le32 curr_state; - __le32 new_state; - __le16 epoch; - char lease_key[SMB2_LEASE_KEY_SIZE]; -}; - -struct oplock_break_info { - int level; - int open_trunc; - int fid; -}; - int smb_grant_oplock(struct ksmbd_work *work, int req_op_level, u64 pid, struct ksmbd_file *fp, __u16 tid, struct lease_ctx_info *lctx, int share_ret, bool replay); diff --git a/fs/smb/server/proc.c b/fs/smb/server/proc.c index 19f0f2cfbf54..e5a9158c09bd 100644 --- a/fs/smb/server/proc.c +++ b/fs/smb/server/proc.c @@ -161,6 +161,8 @@ static int proc_show_ksmbd_stats(struct seq_file *m, void *v) seq_printf(m, "max_connections:\t%u\n", server_conf.max_connections); seq_printf(m, "max_connections_per_ip:\t%u\n", server_conf.max_ip_connections); + seq_printf(m, "max_sessions_per_connection:\t%u\n", + server_conf.max_sessions_per_connection); seq_printf(m, "max_inflight_requests:\t%u\n", server_conf.max_inflight_req); seq_printf(m, "deadtime_seconds:\t%lu\n", server_conf.deadtime / HZ); diff --git a/fs/smb/server/server.c b/fs/smb/server/server.c index 0827c8c51006..0593094f5fd0 100644 --- a/fs/smb/server/server.c +++ b/fs/smb/server/server.c @@ -187,6 +187,16 @@ static void __handle_ksmbd_work(struct ksmbd_work *work, if (conn->ops->is_transform_hdr && conn->ops->is_transform_hdr(work->request_buf)) { + u64 tr_sess_id; + + if (get_rfc1002_len(work->request_buf) < + sizeof(struct smb2_transform_hdr)) { + ksmbd_conn_abort(conn); + return; + } + tr_sess_id = le64_to_cpu(((struct smb2_transform_hdr *) + smb_get_msg(work->request_buf))->SessionId); + rc = conn->ops->decrypt_req(work); if (rc < 0) { ksmbd_conn_abort(conn); @@ -215,6 +225,12 @@ static void __handle_ksmbd_work(struct ksmbd_work *work, ksmbd_conn_abort(conn); return; } + + rc = ksmbd_check_transform_session(work, tr_sess_id); + if (rc < 0) { + ksmbd_conn_abort(conn); + return; + } } if (conn->ops->allocate_rsp_buf(work)) @@ -382,6 +398,7 @@ static void handle_ksmbd_work(struct work_struct *wk) static int queue_ksmbd_work(struct ksmbd_conn *conn) { struct ksmbd_work *work; + bool wait_for_neg = ksmbd_conn_new(conn) || ksmbd_conn_need_negotiate(conn); int err; err = ksmbd_init_smb_server(conn); @@ -404,6 +421,10 @@ static int queue_ksmbd_work(struct ksmbd_conn *conn) conn->last_active = jiffies; INIT_WORK(&work->work, handle_ksmbd_work); ksmbd_queue_work(work); + /* Negotiation can change conn->ops, finish it before reading another PDU. */ + if (wait_for_neg) + wait_event(conn->req_running_q, + atomic_read(&conn->req_running) == 0); return 0; } @@ -451,6 +472,8 @@ static int server_conf_init(void) KSMBD_AUTH_MSKRB5; #endif server_conf.max_inflight_req = SMB2_MAX_CREDITS; + server_conf.max_sessions_per_connection = + KSMBD_DEFAULT_MAX_SESSIONS_PER_CONNECTION; return 0; } diff --git a/fs/smb/server/server.h b/fs/smb/server/server.h index 4d4d268b59d5..7b7df3fafe5b 100644 --- a/fs/smb/server/server.h +++ b/fs/smb/server/server.h @@ -8,6 +8,8 @@ #include "smbacl.h" +#define KSMBD_DEFAULT_MAX_SESSIONS_PER_CONNECTION 64 + /* * Server state type */ @@ -44,6 +46,7 @@ struct ksmbd_server_config { unsigned int max_connections; unsigned int max_inflight_req; unsigned int max_ip_connections; + unsigned int max_sessions_per_connection; char *conf[SERVER_CONF_WORK_GROUP + 1]; struct task_struct *dh_task; diff --git a/fs/smb/server/smb2misc.c b/fs/smb/server/smb2misc.c index 532dea7be0b3..4103a33e6f84 100644 --- a/fs/smb/server/smb2misc.c +++ b/fs/smb/server/smb2misc.c @@ -501,6 +501,14 @@ int ksmbd_smb2_check_message(struct ksmbd_work *work) if (req_struct_size > len + 1) return 1; + if (command == SMB2_SESSION_SETUP_HE && + (work->next_smb2_rcv_hdr_off || next_cmd)) { + struct smb2_sess_setup_req *req = (void *)hdr; + + if (req->Flags & SMB2_SESSION_REQ_FLAG_BINDING) + return 1; + } + if (smb2_calc_size(hdr, &clc_len)) return 1; diff --git a/fs/smb/server/smb2ops.c b/fs/smb/server/smb2ops.c index 4578291fb172..747276b171bd 100644 --- a/fs/smb/server/smb2ops.c +++ b/fs/smb/server/smb2ops.c @@ -13,6 +13,35 @@ #include "server.h" #include "stats.h" +/* + * work->response_sz includes the RFC1002 length field while smb_get_msg() + * skips over it. Durable v1 and v2 response contexts are mutually exclusive, + * and POSIX CREATE contexts are only negotiated for SMB3.1.1. + */ +#define SMB2_CREATE_RSP_SIZE(lease_size, durable_size) \ + (sizeof(__be32) + offsetof(struct smb2_create_rsp, Buffer) + \ + (lease_size) + (durable_size) + \ + sizeof(struct create_mxac_rsp) + \ + sizeof(struct create_disk_id_rsp) + \ + AAPL_RSP_MAX_SIZE) + +#define SMB21_CREATE_RSP_SIZE \ + SMB2_CREATE_RSP_SIZE(sizeof(struct create_lease), \ + sizeof(struct create_durable_rsp)) + +#define SMB3_CREATE_DURABLE_RSP_SIZE \ + ((sizeof(struct create_durable_rsp) > \ + sizeof(struct create_durable_rsp_v2)) ? \ + sizeof(struct create_durable_rsp) : \ + sizeof(struct create_durable_rsp_v2)) + +#define SMB3_CREATE_RSP_SIZE \ + SMB2_CREATE_RSP_SIZE(sizeof(struct create_lease_v2), \ + SMB3_CREATE_DURABLE_RSP_SIZE) + +#define SMB311_CREATE_RSP_SIZE \ + (SMB3_CREATE_RSP_SIZE + sizeof(struct create_posix_rsp)) + static struct smb_version_values smb21_server_values = { .version_string = SMB21_VERSION_STRING, .protocol_id = SMB21_PROT_ID, @@ -38,6 +67,7 @@ static struct smb_version_values smb21_server_values = { .create_disk_id_size = sizeof(struct create_disk_id_rsp), .create_posix_size = sizeof(struct create_posix_rsp), .create_aapl_size = AAPL_RSP_MAX_SIZE, + .create_rsp_size = SMB21_CREATE_RSP_SIZE, }; static struct smb_version_values smb30_server_values = { @@ -66,6 +96,7 @@ static struct smb_version_values smb30_server_values = { .create_disk_id_size = sizeof(struct create_disk_id_rsp), .create_posix_size = sizeof(struct create_posix_rsp), .create_aapl_size = AAPL_RSP_MAX_SIZE, + .create_rsp_size = SMB3_CREATE_RSP_SIZE, }; static struct smb_version_values smb302_server_values = { @@ -94,6 +125,7 @@ static struct smb_version_values smb302_server_values = { .create_disk_id_size = sizeof(struct create_disk_id_rsp), .create_posix_size = sizeof(struct create_posix_rsp), .create_aapl_size = AAPL_RSP_MAX_SIZE, + .create_rsp_size = SMB3_CREATE_RSP_SIZE, }; static struct smb_version_values smb311_server_values = { @@ -122,6 +154,7 @@ static struct smb_version_values smb311_server_values = { .create_disk_id_size = sizeof(struct create_disk_id_rsp), .create_posix_size = sizeof(struct create_posix_rsp), .create_aapl_size = AAPL_RSP_MAX_SIZE, + .create_rsp_size = SMB311_CREATE_RSP_SIZE, }; static struct smb_version_ops smb2_0_server_ops = { diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c index 6b8809f67b92..3f1ab9a20307 100644 --- a/fs/smb/server/smb2pdu.c +++ b/fs/smb/server/smb2pdu.c @@ -116,13 +116,28 @@ static int register_session_channel(struct ksmbd_session *sess, goto out; } - chann->conn = conn; + if (sess->setup_conn != conn) { + rc = ksmbd_session_reserve_slot(conn); + if (rc) { + kfree_sensitive(chann); + goto out; + } + } + + chann->conn = ksmbd_conn_get(conn); memcpy(chann->sess_key, sess_key, sizeof(chann->sess_key)); old = xa_store(&sess->ksmbd_chann_list, (long)conn, chann, KSMBD_DEFAULT_GFP); if (xa_is_err(old)) { + if (sess->setup_conn != conn) + ksmbd_session_release_slot(conn); + ksmbd_conn_put(chann->conn); kfree_sensitive(chann); rc = xa_err(old); + } else if (sess->setup_conn == conn) { + /* Transfer the initial reservation and reference to this channel. */ + sess->setup_conn = NULL; + ksmbd_conn_put(conn); } out: up_write(&sess->chann_lock); @@ -858,14 +873,18 @@ static void smb2_update_lock_sequence(struct ksmbd_work *work, int smb2_allocate_rsp_buf(struct ksmbd_work *work) { struct smb2_hdr *hdr = smb_get_msg(work->request_buf); + struct smb_version_values *vals = work->conn->vals; size_t small_sz = MAX_CIFS_SMALL_BUFFER_SIZE; - size_t large_sz = small_sz + work->conn->vals->max_trans_size; - size_t sz = small_sz; + size_t large_sz = small_sz + vals->max_trans_size; + size_t sz = small_sz, pad_sz = 0; int cmd = le16_to_cpu(hdr->Command); if (cmd == SMB2_IOCTL_HE || cmd == SMB2_QUERY_DIRECTORY_HE) sz = large_sz; + if (cmd == SMB2_CREATE_HE) + sz = max_t(size_t, sz, vals->create_rsp_size); + if (cmd == SMB2_QUERY_INFO_HE) { struct smb2_query_info_req *req; @@ -883,10 +902,17 @@ int smb2_allocate_rsp_buf(struct ksmbd_work *work) } /* allocate large response buf for chained commands */ - if (le32_to_cpu(hdr->NextCommand) > 0) + if (le32_to_cpu(hdr->NextCommand) > 0) { sz = large_sz; + /* + * is_chained_smb2_message() pads the last response of a + * compound to 8 bytes, which can take it past response_sz. + * Leave room for the padding. kvzalloc() zeroes it. + */ + pad_sz = ALIGN(sz - sizeof(__be32), 8) - (sz - sizeof(__be32)); + } - work->response_buf = kvzalloc(sz, KSMBD_DEFAULT_GFP); + work->response_buf = kvzalloc(sz + pad_sz, KSMBD_DEFAULT_GFP); if (!work->response_buf) return -ENOMEM; @@ -2438,6 +2464,8 @@ int smb2_sess_setup(struct ksmbd_work *work) struct smb2_sess_setup_req *req; struct smb2_sess_setup_rsp *rsp; struct ksmbd_session *sess = NULL; + struct ksmbd_session *prev_sess = work->sess; + bool installed = false; struct negotiate_message *negblob; unsigned int negblob_len, negblob_off; int rc = 0; @@ -2458,7 +2486,12 @@ int smb2_sess_setup(struct ksmbd_work *work) ksmbd_conn_lock(conn); if (!req->hdr.SessionId) { - sess = ksmbd_smb2_session_create(); + /* Bound abandoned SessionId-zero authentication exchanges. */ + if (ksmbd_too_many_session_setups(conn)) { + rc = -ENOSPC; + goto out_err; + } + sess = ksmbd_smb2_session_create(conn); if (!sess) { rc = -ENOMEM; goto out_err; @@ -2547,6 +2580,7 @@ int smb2_sess_setup(struct ksmbd_work *work) int sign_ret; work->sess = sess; + installed = true; if (sess->dialect >= SMB30_PROT_ID) sign_ret = smb3_check_sign_req(work); else @@ -2599,6 +2633,7 @@ int smb2_sess_setup(struct ksmbd_work *work) conn->binding = false; } work->sess = sess; + installed = true; negblob_off = le16_to_cpu(req->SecurityBufferOffset); negblob_len = le16_to_cpu(req->SecurityBufferLength); @@ -2786,6 +2821,14 @@ int smb2_sess_setup(struct ksmbd_work *work) } ksmbd_conn_unlock(conn); + + /* + * Later in a compound, work->sess holds the reference that an + * earlier command took. Drop it once this request has installed + * its own session there or cleared it. + */ + if (prev_sess && (installed || !work->sess)) + ksmbd_user_session_put(prev_sess); return rc; } @@ -3146,6 +3189,8 @@ int smb2_session_logoff(struct ksmbd_work *work) up_write(&conn->session_lock); ksmbd_all_conn_set_status(sess, KSMBD_SESS_NEED_SETUP); + /* Drop the table reference while work->sess pins the response keys. */ + ksmbd_session_unregister(conn, sess); if (err) return err; @@ -3322,7 +3367,7 @@ static int smb2_set_ea(struct smb2_ea_info *eabuf, unsigned int buf_len, path->dentry, attr_name, XATTR_USER_PREFIX_LEN + - eabuf->EaNameLength); + eabuf->EaNameLength, NULL); /* delete the EA only when it exits */ if (rc > 0) { @@ -3394,13 +3439,15 @@ static noinline int smb2_set_stream_name_xattr(const struct path *path, fp->stream.name = xattr_stream_name; fp->stream.size = xattr_stream_size; - /* Check if there is stream prefix in xattr space */ + /* Keep the existing xattr's case for subsequent writes and removal. */ rc = ksmbd_vfs_casexattr_len(idmap, path->dentry, xattr_stream_name, - xattr_stream_size); + xattr_stream_size, xattr_stream_name); if (rc >= 0) return 0; + if (rc != -ENOENT && rc != -ENODATA) + return rc; if (fp->cdoption == FILE_OPEN_LE) { if (!strcmp(stream_name, "AFP_AfpInfo") && @@ -3449,7 +3496,7 @@ static loff_t ksmbd_stream_eof(struct ksmbd_file *fp) ssize_t slen = ksmbd_vfs_casexattr_len(file_mnt_idmap(fp->filp), fp->filp->f_path.dentry, fp->stream.name, - fp->stream.size); + fp->stream.size, NULL); return slen < 0 ? 0 : (loff_t)slen; } @@ -3647,10 +3694,11 @@ static int smb2_create_sd_buffer(struct ksmbd_work *work, le32_to_cpu(sd_buf->ccontext.DataLength), true, false); } -static void ksmbd_acls_fattr(struct smb_fattr *fattr, - struct mnt_idmap *idmap, - struct inode *inode) +static int ksmbd_acls_fattr(struct smb_fattr *fattr, + struct mnt_idmap *idmap, + struct inode *inode) { + struct posix_acl *acl; vfsuid_t vfsuid = i_uid_into_vfsuid(idmap, inode); vfsgid_t vfsgid = i_gid_into_vfsgid(idmap, inode); @@ -3661,10 +3709,28 @@ static void ksmbd_acls_fattr(struct smb_fattr *fattr, fattr->cf_dacls = NULL; if (IS_ENABLED(CONFIG_FS_POSIX_ACL)) { - fattr->cf_acls = get_inode_acl(inode, ACL_TYPE_ACCESS); - if (S_ISDIR(inode->i_mode)) - fattr->cf_dacls = get_inode_acl(inode, ACL_TYPE_DEFAULT); + acl = get_inode_acl(inode, ACL_TYPE_ACCESS); + if (IS_ERR(acl)) { + if (acl != ERR_PTR(-EOPNOTSUPP)) + return PTR_ERR(acl); + acl = NULL; + } + fattr->cf_acls = acl; + + if (S_ISDIR(inode->i_mode)) { + acl = get_inode_acl(inode, ACL_TYPE_DEFAULT); + if (IS_ERR(acl)) { + if (acl != ERR_PTR(-EOPNOTSUPP)) { + posix_acl_release(fattr->cf_acls); + return PTR_ERR(acl); + } + acl = NULL; + } + fattr->cf_dacls = acl; + } } + + return 0; } enum { @@ -4080,18 +4146,76 @@ static int parse_app_instance_version(struct smb2_create_req *req, return 0; } -static int smb2_handle_app_instance_id(struct smb2_create_rsp *rsp, - struct durable_info *dh_info) +static int smb2_app_instance_stream_matches(struct ksmbd_file *fp, + char *stream_name, int s_type, + bool *matches) { + char *xattr_stream_name; + size_t xattr_stream_size; + int rc; + + if (!stream_name) { + *matches = !ksmbd_stream_fd(fp); + return 0; + } + + rc = ksmbd_vfs_xattr_stream_name(stream_name, &xattr_stream_name, + &xattr_stream_size, s_type); + if (rc) + return rc; + + *matches = fp->stream.name && fp->stream.size >= 0 && + (size_t)fp->stream.size == xattr_stream_size && + !strncasecmp(fp->stream.name, xattr_stream_name, + xattr_stream_size); + kfree(xattr_stream_name); + return 0; +} + +static int smb2_handle_app_instance_id(struct ksmbd_work *work, + struct smb2_create_rsp *rsp, + struct durable_info *dh_info, + const struct path *path, + char *stream_name, int s_type, + bool *closed) +{ + struct ksmbd_conn *conn = work->conn; struct ksmbd_file *old_fp; - bool reject = false; + const struct path *old_path; + char *app_instance_id; + __le32 maximal_access = FILE_MAXIMAL_ACCESS_LE; + bool reject = false, stream_matches; + int cursor = 0, rc; + *closed = false; if (!dh_info->app_instance_id) return 0; + app_instance_id = dh_info->AppInstanceId; - old_fp = ksmbd_lookup_fd_app_instance_id(dh_info->AppInstanceId); - if (!old_fp) - return 0; + while (true) { + old_fp = ksmbd_lookup_fd_app_instance_id_next(app_instance_id, + &cursor); + if (!old_fp) + return 0; + + if (!ksmbd_app_instance_id_matches(work, old_fp) || + !path_equal(&old_fp->filp->f_path, path)) { + ksmbd_put_durable_fd(old_fp); + continue; + } + + rc = smb2_app_instance_stream_matches(old_fp, stream_name, + s_type, &stream_matches); + if (rc) { + ksmbd_put_durable_fd(old_fp); + return rc; + } + if (!stream_matches) { + ksmbd_put_durable_fd(old_fp); + continue; + } + break; + } if (dh_info->app_instance_version_valid) { if (old_fp->app_instance_version_valid && @@ -4106,13 +4230,34 @@ static int smb2_handle_app_instance_id(struct smb2_create_rsp *rsp, reject = true; } - ksmbd_put_durable_fd(old_fp); if (reject) { + ksmbd_put_durable_fd(old_fp); rsp->hdr.Status = STATUS_FILE_FORCED_CLOSED; return -EIO; } - return ksmbd_close_fd_app_instance_id(dh_info->AppInstanceId); + /* Authorize the referenced open even if its name changes meanwhile. */ + old_path = &old_fp->filp->f_path; + rc = smb_check_perm_dacl(conn, old_path, &maximal_access, 0, + work->sess->user->uid, false); + if (rc) { + /* Access denial prevents takeover, not the CREATE itself. */ + ksmbd_put_durable_fd(old_fp); + return rc == -EACCES ? 0 : rc; + } + if (maximal_access == FILE_MAXIMAL_ACCESS_LE) + ksmbd_vfs_query_maximal_access(mnt_idmap(old_path->mnt), + old_path->dentry, &maximal_access); + /* Also require read access with the effective VFS credentials. */ + if ((le32_to_cpu(maximal_access) & GENERIC_READ_FLAGS) != + GENERIC_READ_FLAGS || + inode_permission(mnt_idmap(old_path->mnt), + d_inode(old_path->dentry), MAY_OPEN | MAY_READ)) { + ksmbd_put_durable_fd(old_fp); + return 0; + } + + return ksmbd_close_fd_app_instance_id(work, old_fp, closed); } /** @@ -4152,6 +4297,7 @@ int smb2_open(struct ksmbd_work *work) char *name = NULL; char *stream_name = NULL; bool file_present = false, created = false, already_permitted = false; + bool app_instance_id_checked = false, app_instance_closed; int share_ret, need_truncate = 0; u64 time, alloc_size = 0; umode_t posix_mode = 0; @@ -4326,13 +4472,6 @@ int smb2_open(struct ksmbd_work *work) } } - if (dh_info.app_instance_id && !dh_info.reconnected && - !dh_info.replay) { - rc = smb2_handle_app_instance_id(rsp, &dh_info); - if (rc) - goto err_out2; - } - if (le32_to_cpu(req->ImpersonationLevel) > le32_to_cpu(IL_DELEGATE)) { pr_err("Invalid impersonationlevel : 0x%x\n", le32_to_cpu(req->ImpersonationLevel)); @@ -4445,9 +4584,29 @@ int smb2_open(struct ksmbd_work *work) goto err_out2; } +resolve_path: + file_present = false; + maximal_access = 0; rc = ksmbd_vfs_kern_path(work, name, LOOKUP_NO_SYMLINKS, &path, 1); + /* Validate takeover before disconnected delete-on-close cleanup. */ + if (!rc && dh_info.app_instance_id && !dh_info.reconnected && + !dh_info.replay && !app_instance_id_checked) { + app_instance_id_checked = true; + rc = smb2_handle_app_instance_id(work, rsp, &dh_info, &path, + stream_name, s_type, + &app_instance_closed); + if (rc) { + path_put(&path); + goto err_out1; + } + if (app_instance_closed) { + path_put(&path); + goto resolve_path; + } + } + /* * A durable handle opened with delete-on-close is preserved across a * disconnect so it can be reclaimed by a durable reconnect. When a new @@ -4791,7 +4950,10 @@ int smb2_open(struct ksmbd_work *work) int pntsd_size; size_t scratch_len; - ksmbd_acls_fattr(&fattr, idmap, inode); + rc = ksmbd_acls_fattr(&fattr, idmap, inode); + if (rc) + goto err_out; + scratch_len = smb_acl_sec_desc_scratch_len(&fattr, NULL, 0, OWNER_SECINFO | GROUP_SECINFO | @@ -5358,8 +5520,16 @@ int smb2_open(struct ksmbd_work *work) if (!rsp->hdr.Status) rsp->hdr.Status = STATUS_UNEXPECTED_IO_ERROR; - if (fp && !dh_info.replay) - ksmbd_fd_put(work, fp); + if (fp && !dh_info.replay) { + if (work->request_open == fp) { + ksmbd_close_fd(work, fp->volatile_id); + smb2_complete_request_open(work); + } else { + if (dh_info.reconnected && fp == dh_info.fp) + ksmbd_unpublish_reopened_fd(work, fp); + ksmbd_fd_put(work, fp); + } + } smb2_set_err_rsp(work); ksmbd_debug(SMB, "Error response: %x\n", rsp->hdr.Status); } @@ -5370,10 +5540,9 @@ int smb2_open(struct ksmbd_work *work) if (dh_info.reconnected) { /* * If reconnect succeeded, fp was republished in the - * session file table. On a later error, ksmbd_fd_put() - * above drops the session reference; drop the durable - * lookup reference through the same session-aware path so - * final close removes the volatile id before freeing fp. + * session file table, and the error path above has + * already removed its volatile id. Drop the durable + * lookup reference through the same session-aware path. */ if (rc && fp == dh_info.fp) ksmbd_fd_put(work, dh_info.fp); @@ -6900,7 +7069,8 @@ static int get_file_stream_info(struct ksmbd_work *work, ":%s", &stream_name[XATTR_NAME_STREAM_LEN]); next = sizeof(struct smb2_file_stream_info) + streamlen * 2; - if (next > buf_free_len) { + /* smbConvertToUTF16() also writes a terminating UTF-16 unit. */ + if (next + sizeof(__le16) > buf_free_len) { kfree(stream_buf); break; } @@ -6922,7 +7092,7 @@ static int get_file_stream_info(struct ksmbd_work *work, */ slen = ksmbd_vfs_casexattr_len(file_mnt_idmap(fp->filp), path->dentry, stream_name, - strlen(stream_name) + 1); + strlen(stream_name) + 1, NULL); ssize = slen < 0 ? 0 : (loff_t)slen; file_info->StreamSize = cpu_to_le64(ssize); file_info->StreamAllocationSize = cpu_to_le64(ssize); @@ -6934,7 +7104,7 @@ static int get_file_stream_info(struct ksmbd_work *work, out: if (!S_ISDIR(stat.mode) && - buf_free_len >= sizeof(struct smb2_file_stream_info) + 7 * 2) { + buf_free_len >= sizeof(struct smb2_file_stream_info) + (7 + 1) * 2) { file_info = (struct smb2_file_stream_info *) &rsp->Buffer[nbytes]; streamlen = smbConvertToUTF16((__le16 *)file_info->StreamName, @@ -7652,7 +7822,11 @@ static int smb2_get_info_sec(struct ksmbd_work *work, idmap = file_mnt_idmap(fp->filp); inode = file_inode(fp->filp); - ksmbd_acls_fattr(&fattr, idmap, inode); + rc = ksmbd_acls_fattr(&fattr, idmap, inode); + if (rc) { + ksmbd_fd_put(work, fp); + return rc; + } if (test_share_config_flag(work->tcon->share_conf, KSMBD_SHARE_FLAG_ACL_XATTR)) @@ -8241,24 +8415,17 @@ static int set_file_allocation_info(struct ksmbd_work *work, pr_err("vfs_fallocate is failed : %d\n", rc); return rc; } - } else if (alloc_blks < stat.blocks) { - loff_t size; - + } else if (alloc_blks < stat.blocks && + alloc_blks * 512 <= i_size_read(inode)) { /* - * Allocation size could be smaller than original one - * which means allocated blocks in file should be - * deallocated. use truncate to cut out it, but inode - * size is also updated with truncate offset. - * inode size is retained by backup inode size. + * Truncate at or below EOF to discard excess allocation. + * Truncating above EOF would extend the file on disk. */ - size = i_size_read(inode); rc = ksmbd_vfs_truncate(work, fp, alloc_blks * 512); if (rc) { pr_err("truncate failed!, err %d\n", rc); return rc; } - if (size < alloc_blks * 512) - i_size_write(inode, size); } fp->allocation_size = le64_to_cpu(file_alloc_info->AllocationSize); @@ -8278,6 +8445,8 @@ static int set_end_of_file_info(struct ksmbd_work *work, struct ksmbd_file *fp, newsize = le64_to_cpu(file_eof_info->EndOfFile); inode = file_inode(fp->filp); + if (ksmbd_stream_fd(fp)) + return ksmbd_vfs_stream_truncate(fp, newsize); /* * If FILE_END_OF_FILE_INFORMATION of set_info_file is called @@ -8286,8 +8455,7 @@ static int set_end_of_file_info(struct ksmbd_work *work, struct ksmbd_file *fp, * truncate of some filesystem like FAT32 fill zero data in * truncated range. */ - if (inode->i_sb->s_magic != MSDOS_SUPER_MAGIC && - ksmbd_stream_fd(fp) == false) { + if (inode->i_sb->s_magic != MSDOS_SUPER_MAGIC) { ksmbd_debug(SMB, "truncated to newsize %lld\n", newsize); rc = ksmbd_vfs_truncate(work, fp, newsize); if (rc) { @@ -8324,6 +8492,7 @@ static int set_file_disposition_info(struct ksmbd_work *work, struct smb2_file_disposition_info *file_info) { struct inode *inode; + int err; if (!(fp->daccess & FILE_DELETE_LE)) { pr_err("no right to delete : 0x%x\n", fp->daccess); @@ -8338,9 +8507,13 @@ static int set_file_disposition_info(struct ksmbd_work *work, if (ksmbd_has_stream_without_delete_share(fp)) return -ESHARE; - if (S_ISDIR(inode->i_mode) && !ksmbd_stream_fd(fp) && - ksmbd_vfs_empty_dir(fp) == -ENOTEMPTY) - return -EBUSY; + if (S_ISDIR(inode->i_mode) && !ksmbd_stream_fd(fp)) { + mutex_lock(&fp->readdir_lock); + err = ksmbd_vfs_empty_dir(fp); + mutex_unlock(&fp->readdir_lock); + if (err == -ENOTEMPTY) + return -EBUSY; + } smb_break_all_levII_oplock_for_delete(work, fp); ksmbd_fd_set_delete_pending(fp); } else { @@ -9276,23 +9449,25 @@ static noinline int smb2_write_pipe(struct ksmbd_work *work) int err = 0, ret = 0; char *data_buf; size_t length; + unsigned int data_offset, req_len; WORK_BUFFERS(work, req, rsp); length = le32_to_cpu(req->Length); id = req->VolatileFileId; + data_offset = le16_to_cpu(req->DataOffset); + req_len = smb2_current_req_len(work, &req->hdr); - if ((u64)le16_to_cpu(req->DataOffset) + length > - get_rfc1002_len(work->request_buf)) { - pr_err("invalid write data offset %u, smb_len %u\n", - le16_to_cpu(req->DataOffset), - get_rfc1002_len(work->request_buf)); + if (data_offset < offsetof(struct smb2_write_req, Buffer) || + data_offset > req_len || length > req_len - data_offset) { + pr_err("invalid write data offset %u, length %zu, req_len %u\n", + data_offset, length, req_len); err = -EINVAL; goto out; } data_buf = (char *)(((char *)&req->hdr.ProtocolId) + - le16_to_cpu(req->DataOffset)); + data_offset); rpc_resp = ksmbd_rpc_write(work->sess, id, data_buf, length); if (rpc_resp) { @@ -9579,14 +9754,21 @@ int smb2_write(struct ksmbd_work *work) writethrough = true; if (is_rdma_channel == false) { - if (le16_to_cpu(req->DataOffset) < - offsetof(struct smb2_write_req, Buffer)) { + unsigned int data_offset = le16_to_cpu(req->DataOffset); + unsigned int req_len = smb2_current_req_len(work, &req->hdr); + + if (data_offset < offsetof(struct smb2_write_req, Buffer) || + data_offset > req_len || + length > req_len - data_offset) { + ksmbd_debug(SMB, + "invalid write data offset %u, length %zu, req_len %u\n", + data_offset, length, req_len); err = -EINVAL; goto out; } data_buf = (char *)(((char *)&req->hdr.ProtocolId) + - le16_to_cpu(req->DataOffset)); + data_offset); ksmbd_debug(SMB, "filename %pD, offset %lld, len %zu\n", fp->filp, offset, length); @@ -11444,7 +11626,7 @@ static void smb20_oplock_break_ack(struct ksmbd_work *work) __le32 status = STATUS_SUCCESS; int ret; u64 volatile_id, persistent_id; - char req_oplevel = 0, rsp_oplevel = 0; + u8 req_oplevel = 0, rsp_oplevel = 0; WORK_BUFFERS(work, req, rsp); @@ -12346,6 +12528,60 @@ bool smb3_is_transform_hdr(void *buf) return trhdr->ProtocolId == SMB2_TRANSFORM_PROTO_NUM; } +/* + * The decryption key is selected by the transform SessionId, but + * the request is authorized under the session named in the + * decrypted SMB2 header. Per MS-SMB2 the two must match, so check + * the decrypted message and its compound chain before dispatch. + */ +int ksmbd_check_transform_session(struct ksmbd_work *work, u64 tr_sess_id) +{ + struct smb2_hdr *hdr = smb_get_msg(work->request_buf); + size_t msg_len = get_rfc1002_len(work->request_buf); + size_t off = 0; + bool first = true; + + if (msg_len < sizeof(*hdr)) { + pr_err_ratelimited("Decrypted message is smaller than SMB2 header\n"); + return -ECONNABORTED; + } + + for (;;) { + u64 sid = le64_to_cpu(hdr->SessionId); + u32 next; + + if (first) { + if (hdr->Flags & SMB2_FLAGS_RELATED_OPERATIONS) { + pr_err_ratelimited("RELATED_OPERATIONS set on first operation\n"); + return -ECONNABORTED; + } + if (sid != tr_sess_id) { + pr_err_ratelimited("SessionId mismatch between transform and inner header\n"); + return -ECONNABORTED; + } + first = false; + } else if (!(hdr->Flags & SMB2_FLAGS_RELATED_OPERATIONS) && + sid != tr_sess_id) { + pr_err_ratelimited("SessionId mismatch in compound chain\n"); + return -ECONNABORTED; + } + + next = le32_to_cpu(hdr->NextCommand); + if (!next) + return 0; + if (next % 8) { + pr_err_ratelimited("NextCommand %u is not 8-byte aligned\n", next); + return -ECONNABORTED; + } + if (next > msg_len - off - sizeof(*hdr)) { + pr_err_ratelimited("NextCommand %u is out of the message\n", next); + return -ECONNABORTED; + } + off += next; + hdr = (struct smb2_hdr *)((u8 *)smb_get_msg(work->request_buf) + off); + } +} + int smb3_decrypt_req(struct ksmbd_work *work) { char *buf = work->request_buf; diff --git a/fs/smb/server/smb2pdu.h b/fs/smb/server/smb2pdu.h index ca8e27f7b712..20a5337459ab 100644 --- a/fs/smb/server/smb2pdu.h +++ b/fs/smb/server/smb2pdu.h @@ -420,6 +420,7 @@ struct channel *lookup_chann_list(struct ksmbd_session *sess, void smb3_preauth_hash_rsp(struct ksmbd_work *work); bool smb3_is_transform_hdr(void *buf); int smb3_decrypt_req(struct ksmbd_work *work); +int ksmbd_check_transform_session(struct ksmbd_work *work, u64 tr_sess_id); int smb3_encrypt_resp(struct ksmbd_work *work); bool smb3_11_final_sess_setup_resp(struct ksmbd_work *work); int smb2_set_rsp_credits(struct ksmbd_work *work); diff --git a/fs/smb/server/smb_common.c b/fs/smb/server/smb_common.c index 4c2da65510bc..21ba0ce5abbf 100644 --- a/fs/smb/server/smb_common.c +++ b/fs/smb/server/smb_common.c @@ -478,16 +478,17 @@ int ksmbd_populate_dot_dotdot_entries(struct ksmbd_work *work, int info_level, if (i == 0) { d_info->name = "."; d_info->name_len = 1; - dentry = dir->filp->f_path.dentry; + dentry = dget(dir->filp->f_path.dentry); } else { d_info->name = ".."; d_info->name_len = 2; - dentry = dir->filp->f_path.dentry->d_parent; + dentry = dget_parent(dir->filp->f_path.dentry); } if (!match_pattern(d_info->name, d_info->name_len, search_pattern)) { dir->dot_dotdot[i] = 1; + dput(dentry); continue; } @@ -496,6 +497,7 @@ int ksmbd_populate_dot_dotdot_entries(struct ksmbd_work *work, int info_level, idmap, dentry, &ksmbd_kstat); + dput(dentry); if (rc) break; diff --git a/fs/smb/server/smbacl.c b/fs/smb/server/smbacl.c index 1fad6ccf3a72..f8bd8e2ed33e 100644 --- a/fs/smb/server/smbacl.c +++ b/fs/smb/server/smbacl.c @@ -899,7 +899,7 @@ int parse_sec_desc(struct mnt_idmap *idmap, struct smb_ntsd *pntsd, int acl_len, struct smb_fattr *fattr) { int rc = 0; - struct smb_sid *owner_sid_ptr, *group_sid_ptr; + struct smb_sid *owner_sid_ptr = NULL, *group_sid_ptr = NULL; struct smb_acl *dacl_ptr; /* no need for SACL ptr */ char *end_of_acl = ((char *)pntsd) + acl_len; __u32 dacloffset; @@ -911,12 +911,7 @@ int parse_sec_desc(struct mnt_idmap *idmap, struct smb_ntsd *pntsd, if (acl_len < sizeof(struct smb_ntsd)) return -EINVAL; - owner_sid_ptr = (struct smb_sid *)((char *)pntsd + - le32_to_cpu(pntsd->osidoffset)); - group_sid_ptr = (struct smb_sid *)((char *)pntsd + - le32_to_cpu(pntsd->gsidoffset)); dacloffset = le32_to_cpu(pntsd->dacloffset); - dacl_ptr = (struct smb_acl *)((char *)pntsd + dacloffset); ksmbd_debug(SMB, "revision %d type 0x%x ooffset 0x%x goffset 0x%x sacloffset 0x%x dacloffset 0x%x\n", pntsd->revision, pntsd->type, le32_to_cpu(pntsd->osidoffset), @@ -932,9 +927,12 @@ int parse_sec_desc(struct mnt_idmap *idmap, struct smb_ntsd *pntsd, pntsd->type = cpu_to_le16(DACL_PRESENT); if (pntsd->osidoffset) { - if (le32_to_cpu(pntsd->osidoffset) < sizeof(struct smb_ntsd)) + if (le32_to_cpu(pntsd->osidoffset) < sizeof(struct smb_ntsd) || + le32_to_cpu(pntsd->osidoffset) >= acl_len) return -EINVAL; + owner_sid_ptr = (struct smb_sid *)((char *)pntsd + + le32_to_cpu(pntsd->osidoffset)); rc = parse_sid(owner_sid_ptr, end_of_acl); if (rc) { pr_err("%s: Error %d parsing Owner SID\n", __func__, rc); @@ -950,9 +948,12 @@ int parse_sec_desc(struct mnt_idmap *idmap, struct smb_ntsd *pntsd, } if (pntsd->gsidoffset) { - if (le32_to_cpu(pntsd->gsidoffset) < sizeof(struct smb_ntsd)) + if (le32_to_cpu(pntsd->gsidoffset) < sizeof(struct smb_ntsd) || + le32_to_cpu(pntsd->gsidoffset) >= acl_len) return -EINVAL; + group_sid_ptr = (struct smb_sid *)((char *)pntsd + + le32_to_cpu(pntsd->gsidoffset)); rc = parse_sid(group_sid_ptr, end_of_acl); if (rc) { pr_err("%s: Error %d mapping Owner SID to gid\n", @@ -974,9 +975,10 @@ int parse_sec_desc(struct mnt_idmap *idmap, struct smb_ntsd *pntsd, pntsd->type |= cpu_to_le16(DACL_PROTECTED); if (dacloffset) { - if (dacloffset < sizeof(struct smb_ntsd)) + if (dacloffset < sizeof(struct smb_ntsd) || dacloffset >= acl_len) return -EINVAL; + dacl_ptr = (struct smb_acl *)((char *)pntsd + dacloffset); rc = parse_dacl(idmap, dacl_ptr, end_of_acl, owner_sid_ptr, group_sid_ptr, fattr); if (rc) @@ -1096,9 +1098,12 @@ int build_sec_desc(struct mnt_idmap *idmap, struct smb_acl *ppdacl_ptr; unsigned int dacl_offset = le32_to_cpu(ppntsd->dacloffset); int ppdacl_size, ntacl_size = ppntsd_size - dacl_offset; + size_t dacl_struct_end; if (!dacl_offset || - (dacl_offset + sizeof(struct smb_acl) > ppntsd_size)) + check_add_overflow(dacl_offset, sizeof(struct smb_acl), + &dacl_struct_end) || + dacl_struct_end > (size_t)ppntsd_size) goto out; ppdacl_ptr = (struct smb_acl *)((char *)ppntsd + dacl_offset); @@ -1490,21 +1495,27 @@ int smb_check_perm_dacl(struct ksmbd_conn *conn, const struct path *path, if (pdacl_size > acl_size || pdacl_size < sizeof(struct smb_acl)) goto err_out; - if (!pdacl->num_aces) { - if (!(pdacl_size - sizeof(struct smb_acl)) && - *pdaccess & ~(FILE_READ_CONTROL_LE | FILE_WRITE_DAC_LE)) { - rc = -EACCES; - goto err_out; - } - goto err_out; - } - if (!uid) sid_type = SIDUNIX_USER; id_to_sid(uid, sid_type, &sid); vfsuid = i_uid_into_vfsuid(idmap, d_inode(path->dentry)); is_owner = uid == from_kuid(&init_user_ns, vfsuid_into_kuid(vfsuid)); + if (!pdacl->num_aces) { + /* + * An empty (present, zero-ACE) DACL grants no access to + * anyone except the object owner's implicit READ_CONTROL + * and WRITE_DAC (MS-DTYP 2.4.5). Deny every other caller, + * deny the owner any access beyond those two bits, and do + * not let trailing bytes after a zero-ACE DACL become an + * implicit grant. + */ + if (!is_owner || + (*pdaccess & ~(FILE_READ_CONTROL_LE | FILE_WRITE_DAC_LE))) + rc = -EACCES; + goto err_out; + } + if (*pdaccess & FILE_MAXIMAL_ACCESS_LE) { ace = (struct smb_ace *)((char *)pdacl + sizeof(struct smb_acl)); aces_size = pdacl_size - sizeof(struct smb_acl); diff --git a/fs/smb/server/transport_ipc.c b/fs/smb/server/transport_ipc.c index e550aa41ad2c..eb07cdf17410 100644 --- a/fs/smb/server/transport_ipc.c +++ b/fs/smb/server/transport_ipc.c @@ -353,6 +353,13 @@ static int ipc_server_config_on_startup(struct ksmbd_startup_request *req) if (req->max_ip_connections) server_conf.max_ip_connections = req->max_ip_connections; + if (req->max_sessions_per_connection) + server_conf.max_sessions_per_connection = + req->max_sessions_per_connection; + else + server_conf.max_sessions_per_connection = + KSMBD_DEFAULT_MAX_SESSIONS_PER_CONNECTION; + ret = ksmbd_set_netbios_name(req->netbios_name); ret |= ksmbd_set_server_string(req->server_string); ret |= ksmbd_set_work_group(req->work_group); @@ -532,11 +539,25 @@ static int ipc_validate_msg(struct ipc_msg_table_entry *entry) if (entry->msg_sz < sizeof(struct ksmbd_share_config_response)) return -EINVAL; + /* + * ksmbd-tools releases before 3.5.2 leave payload_sz zero. + * Derive it from the size of the message actually received so + * the checks below bound the payload for those releases too. + */ + if (!resp->payload_sz) + resp->payload_sz = entry->msg_sz - + sizeof(struct ksmbd_share_config_response); + if (strnlen(resp->share_name, sizeof(resp->share_name)) == sizeof(resp->share_name)) return -EINVAL; - if (resp->veto_list_sz > resp->payload_sz) + /* + * Pipe shares carry no veto list, but ksmbd-tools before 3.5.3 + * copy a global "veto files" setting into the IPC$ response. + */ + if (!(resp->flags & KSMBD_SHARE_FLAG_PIPE) && + resp->veto_list_sz > resp->payload_sz) return -EINVAL; if (resp->flags != KSMBD_SHARE_FLAG_INVALID && diff --git a/fs/smb/server/transport_tcp.c b/fs/smb/server/transport_tcp.c index 4968cfc1a572..af3b6a01862b 100644 --- a/fs/smb/server/transport_tcp.c +++ b/fs/smb/server/transport_tcp.c @@ -5,6 +5,7 @@ */ #include +#include #include "smb_common.h" #include "server.h" @@ -29,11 +30,30 @@ static LIST_HEAD(iface_list); static int bind_additional_ifaces; +struct tcp_tx { + struct list_head list; + size_t len; + size_t offset; + unsigned long deadline; + int msg_flags; + u8 data[]; +}; + struct tcp_transport { struct ksmbd_transport transport; struct socket *sock; struct kvec *iov; unsigned int nr_iov; + /* Serializes queue access and socket writes. Never taken by callbacks. */ + struct mutex tx_lock; + struct list_head tx_queue; + struct delayed_work tx_work; + size_t tx_queued; + /* TCP byte positions for accepted PDUs and bytes passed to the socket. */ + u64 tx_end; + u64 tx_sent; + bool tx_stopped; + void (*write_space)(struct sock *sk); }; static const struct ksmbd_transport_ops ksmbd_tcp_transport_ops; @@ -42,6 +62,10 @@ static void tcp_stop_kthread(struct task_struct *kthread); static void ksmbd_tcp_stop_listener(struct interface *iface); static struct interface *alloc_iface(char *ifname); static void ksmbd_tcp_disconnect(struct ksmbd_transport *t); +static void ksmbd_tcp_shutdown(struct ksmbd_transport *t); +static void ksmbd_tcp_tx_work(struct work_struct *work); +static void ksmbd_tcp_write_space(struct sock *sk); +static void ksmbd_tcp_stop_tx(struct tcp_transport *t); #define KSMBD_TRANS(t) (&(t)->transport) #define TCP_TRANS(t) ((struct tcp_transport *)container_of(t, \ @@ -66,6 +90,9 @@ static struct tcp_transport *alloc_transport(struct socket *client_sk) if (!t) return NULL; t->sock = client_sk; + mutex_init(&t->tx_lock); + INIT_LIST_HEAD(&t->tx_queue); + INIT_DELAYED_WORK(&t->tx_work, ksmbd_tcp_tx_work); conn = ksmbd_conn_alloc(); if (!conn) { @@ -85,13 +112,26 @@ static struct tcp_transport *alloc_transport(struct socket *client_sk) conn->inet_addr = inet_sk(client_sk->sk)->inet_daddr; conn->inet_hash = ipv4_addr_hash(inet_sk(client_sk->sk)->inet_daddr); #endif - down_write(&conn_list_lock); - hash_add(conn_list, &conn->hlist, conn->inet_hash); - up_write(&conn_list_lock); - conn->transport = KSMBD_TRANS(t); KSMBD_TRANS(t)->conn = conn; KSMBD_TRANS(t)->ops = &ksmbd_tcp_transport_ops; + + lock_sock(client_sk->sk); + t->tx_end = tcp_sk(client_sk->sk)->bytes_acked + + (u32)(tcp_sk(client_sk->sk)->write_seq - + tcp_sk(client_sk->sk)->snd_una); + t->tx_sent = t->tx_end; + release_sock(client_sk->sk); + + write_lock_bh(&client_sk->sk->sk_callback_lock); + t->write_space = client_sk->sk->sk_write_space; + client_sk->sk->sk_user_data = t; + client_sk->sk->sk_write_space = ksmbd_tcp_write_space; + write_unlock_bh(&client_sk->sk->sk_callback_lock); + + down_write(&conn_list_lock); + hash_add(conn_list, &conn->hlist, conn->inet_hash); + up_write(&conn_list_lock); return t; } @@ -99,6 +139,7 @@ static void ksmbd_tcp_free_transport(struct ksmbd_transport *kt) { struct tcp_transport *t = TCP_TRANS(kt); + ksmbd_tcp_stop_tx(t); sock_release(t->sock); kfree(t->iov); kfree(t); @@ -106,7 +147,8 @@ static void ksmbd_tcp_free_transport(struct ksmbd_transport *kt) static void free_transport(struct tcp_transport *t) { - kernel_sock_shutdown(t->sock, SHUT_RDWR); + ksmbd_tcp_shutdown(KSMBD_TRANS(t)); + ksmbd_tcp_stop_tx(t); ksmbd_conn_free(KSMBD_TRANS(t)->conn); } @@ -431,15 +473,251 @@ static int ksmbd_tcp_read(struct ksmbd_transport *t, char *buf, return ksmbd_tcp_readv(TCP_TRANS(t), &iov, 1, to_read, max_retries); } +static bool ksmbd_tcp_tx_has_space(struct tcp_transport *t, size_t size) +{ + size_t limit = max_t(size_t, READ_ONCE(t->sock->sk->sk_sndbuf), size); + + /* Allow one large response, but bound the retained response buffers. */ + return READ_ONCE(t->tx_queued) <= limit - size; +} + +static struct tcp_tx *ksmbd_tcp_alloc_tx(const struct ksmbd_transport_write *tx, + size_t offset) +{ + struct tcp_tx *msg; + struct iov_iter iter; + size_t remaining = tx->size - offset; + + msg = kvmalloc(struct_size(msg, data, remaining), KSMBD_DEFAULT_GFP); + if (!msg) + return NULL; + + iov_iter_kvec(&iter, ITER_SOURCE, tx->iov, tx->iov_cnt, tx->size); + iov_iter_advance(&iter, offset); + if (!copy_from_iter_full(msg->data, remaining, &iter)) { + kvfree(msg); + return NULL; + } + msg->len = remaining; + msg->offset = 0; + msg->msg_flags = tx->msg_flags; + return msg; +} + +static void ksmbd_tcp_purge_tx(struct tcp_transport *t) +{ + struct tcp_tx *msg, *next; + + list_for_each_entry_safe(msg, next, &t->tx_queue, list) { + list_del(&msg->list); + kvfree(msg); + } + WRITE_ONCE(t->tx_queued, 0); +} + +static bool ksmbd_tcp_send_retry(int ret) +{ + return ret == -EAGAIN || ret == -EINTR || ret == -ERESTARTSYS || + ret == -ENOMEM; +} + +static void ksmbd_tcp_tx_work(struct work_struct *work) +{ + struct tcp_transport *t = container_of(to_delayed_work(work), + struct tcp_transport, tx_work); + unsigned long delay = 0; + unsigned int budget = 16; + bool failed = false; + int ret; + + mutex_lock(&t->tx_lock); + while (!READ_ONCE(t->tx_stopped) && !list_empty(&t->tx_queue) && budget--) { + struct tcp_tx *msg = list_first_entry(&t->tx_queue, struct tcp_tx, list); + struct msghdr hdr = { + .msg_flags = MSG_NOSIGNAL | MSG_DONTWAIT | msg->msg_flags, + }; + struct kvec iov = { + .iov_base = msg->data + msg->offset, + .iov_len = msg->len - msg->offset, + }; + + /* MSG_DONTWAIT bypasses sk_sndtimeo, including for retained PDUs. */ + if (time_after_eq(jiffies, msg->deadline)) { + WRITE_ONCE(t->tx_stopped, true); + failed = true; + break; + } + ret = kernel_sendmsg(t->sock, &hdr, &iov, 1, iov.iov_len); + if (ret > 0) { + msg->offset += ret; + t->tx_sent += ret; + if (msg->offset == msg->len) { + WRITE_ONCE(t->tx_queued, t->tx_queued - msg->len); + list_del(&msg->list); + kvfree(msg); + continue; + } + } else if (!ksmbd_tcp_send_retry(ret)) { + WRITE_ONCE(t->tx_stopped, true); + failed = true; + break; + } + /* + * Write-space notifications normally resume the head of the queue. + * Also retry after memory pressure, which may produce no notification. + */ + delay = max_t(unsigned long, 1, HZ / 10); + break; + } + if (READ_ONCE(t->tx_stopped)) + ksmbd_tcp_purge_tx(t); + else if (!list_empty(&t->tx_queue)) + queue_delayed_work(system_unbound_wq, &t->tx_work, delay); + mutex_unlock(&t->tx_lock); + wake_up_all(&t->transport.conn->req_running_q); + + if (failed) + ksmbd_conn_abort(t->transport.conn); +} + +static void ksmbd_tcp_write_space(struct sock *sk) +{ + struct tcp_transport *t; + + read_lock_bh(&sk->sk_callback_lock); + t = sk->sk_user_data; + if (t) { + t->write_space(sk); + if (!READ_ONCE(t->tx_stopped) && sk_stream_is_writeable(sk)) + mod_delayed_work(system_unbound_wq, &t->tx_work, 0); + } + read_unlock_bh(&sk->sk_callback_lock); +} + +static void ksmbd_tcp_stop_tx(struct tcp_transport *t) +{ + struct sock *sk = t->sock->sk; + + WRITE_ONCE(t->tx_stopped, true); + write_lock_bh(&sk->sk_callback_lock); + if (sk->sk_user_data == t) { + sk->sk_write_space = t->write_space; + sk->sk_user_data = NULL; + } + write_unlock_bh(&sk->sk_callback_lock); + cancel_delayed_work_sync(&t->tx_work); + mutex_lock(&t->tx_lock); + ksmbd_tcp_purge_tx(t); + mutex_unlock(&t->tx_lock); + wake_up_all(&t->transport.conn->req_running_q); +} + static int ksmbd_tcp_writev(struct ksmbd_transport *t, const struct ksmbd_transport_write *tx) { - struct msghdr smb_msg = { - .msg_flags = MSG_NOSIGNAL | tx->msg_flags, + struct tcp_transport *tt = TCP_TRANS(t); + struct sock *sk = tt->sock->sk; + struct msghdr hdr = { + .msg_flags = MSG_NOSIGNAL | MSG_DONTWAIT | tx->msg_flags, }; + struct tcp_tx *msg; + unsigned long deadline = jiffies + KSMBD_TCP_SEND_TIMEOUT; + long timeout; + int sent = 0; + + for (;;) { + if (tx->msg_flags & MSG_DONTWAIT) { + if (!mutex_trylock(&tt->tx_lock)) + return -EAGAIN; + } else { + mutex_lock(&tt->tx_lock); + } + if (READ_ONCE(tt->tx_stopped) || ksmbd_conn_exiting(t->conn) || + ksmbd_conn_releasing(t->conn)) { + mutex_unlock(&tt->tx_lock); + return -EPIPE; + } + if (ksmbd_tcp_tx_has_space(tt, tx->size)) + break; + mutex_unlock(&tt->tx_lock); + if (tx->msg_flags & MSG_DONTWAIT) + return -EAGAIN; + timeout = deadline - jiffies; + if (time_after_eq(jiffies, deadline) || + !wait_event_timeout(t->conn->req_running_q, + READ_ONCE(tt->tx_stopped) || + ksmbd_conn_exiting(t->conn) || + ksmbd_conn_releasing(t->conn) || + ksmbd_tcp_tx_has_space(tt, tx->size), + timeout)) { + /* A queued PDU may already have a prefix in the socket. */ + ksmbd_conn_abort(t->conn); + return -EAGAIN; + } + } + + /* Avoid an extra copy when the socket accepts the whole PDU at once. */ + if (list_empty(&tt->tx_queue)) { + sent = kernel_sendmsg(tt->sock, &hdr, tx->iov, tx->iov_cnt, tx->size); + if (sent > 0) { + tt->tx_sent += sent; + if (sent == tx->size) + goto accepted; + } else if (ksmbd_tcp_send_retry(sent)) { + sent = 0; + } else { + WRITE_ONCE(tt->tx_stopped, true); + mutex_unlock(&tt->tx_lock); + ksmbd_conn_abort(t->conn); + return sent ?: -EPIPE; + } + } - return kernel_sendmsg(TCP_TRANS(t)->sock, &smb_msg, tx->iov, - tx->iov_cnt, tx->size); + /* The request buffers can be freed as soon as writev returns. */ + msg = ksmbd_tcp_alloc_tx(tx, sent); + if (!msg) { + if (sent) + WRITE_ONCE(tt->tx_stopped, true); + mutex_unlock(&tt->tx_lock); + if (sent) + ksmbd_conn_abort(t->conn); + return -ENOMEM; + } + msg->deadline = deadline; + list_add_tail(&msg->list, &tt->tx_queue); + WRITE_ONCE(tt->tx_queued, tt->tx_queued + msg->len); + queue_delayed_work(system_unbound_wq, &tt->tx_work, 0); + +accepted: + tt->tx_end += tx->size; + if (tx->tracking) { + /* Include retained data, without retaining the caller's tracking object. */ + tx->tracking->end = tt->tx_end; + lock_sock(sk); + tx->tracking->retry_after = + clamp_t(unsigned long, 6UL * inet_csk(sk)->icsk_rto, + DIV_ROUND_UP(1200 * HZ, MSEC_PER_SEC), 6 * HZ); + release_sock(sk); + tx->tracking->valid = true; + } + mutex_unlock(&tt->tx_lock); + return tx->size; +} + +static bool ksmbd_tcp_is_acked(struct ksmbd_transport *t, + const struct ksmbd_tx_tracking *tracking) +{ + struct tcp_transport *tt = TCP_TRANS(t); + struct sock *sk = tt->sock->sk; + bool acked; + + mutex_lock(&tt->tx_lock); + lock_sock(sk); + acked = tt->tx_sent >= tracking->end && + tcp_sk(sk)->bytes_acked >= tracking->end; + release_sock(sk); + mutex_unlock(&tt->tx_lock); + return acked; } static void ksmbd_tcp_disconnect(struct ksmbd_transport *t) @@ -451,7 +729,9 @@ static void ksmbd_tcp_disconnect(struct ksmbd_transport *t) static void ksmbd_tcp_shutdown(struct ksmbd_transport *t) { + WRITE_ONCE(TCP_TRANS(t)->tx_stopped, true); kernel_sock_shutdown(TCP_TRANS(t)->sock, SHUT_RDWR); + wake_up_all(&t->conn->req_running_q); } static void tcp_destroy_socket(struct socket *ksmbd_socket) @@ -577,6 +857,15 @@ static int ksmbd_netdev_event(struct notifier_block *nb, unsigned long event, struct interface *iface; int ret; + /* + * ksmbd serves the initial namespace only. create_socket() listens in + * the namespace of the task raising the event, so a container bringing + * up its own interface would get a listener on its port 445, and the + * name-only interface lookup would let its interfaces stop ours. + */ + if (!net_eq(dev_net(netdev), &init_net)) + return NOTIFY_DONE; + switch (event) { case NETDEV_UP: if (netif_is_bridge_port(netdev)) @@ -715,6 +1004,7 @@ int ksmbd_tcp_set_interfaces(char *ifc_list, int ifc_list_sz) static const struct ksmbd_transport_ops ksmbd_tcp_transport_ops = { .read = ksmbd_tcp_read, .writev = ksmbd_tcp_writev, + .is_acked = ksmbd_tcp_is_acked, .disconnect = ksmbd_tcp_disconnect, .shutdown = ksmbd_tcp_shutdown, .free_transport = ksmbd_tcp_free_transport, diff --git a/fs/smb/server/vfs.c b/fs/smb/server/vfs.c index c2c9aaa5de1b..7453b6e72bd4 100644 --- a/fs/smb/server/vfs.c +++ b/fs/smb/server/vfs.c @@ -61,6 +61,10 @@ static int ksmbd_vfs_path_lookup(struct ksmbd_share_config *share_conf, struct dentry *d; if (pathname[0] == '\0') { + /* The share root must never be removed. */ + if (for_remove) + return -EINVAL; + pathname = share_conf->path; root_share_path = NULL; } else { @@ -118,10 +122,11 @@ static int ksmbd_vfs_path_lookup(struct ksmbd_share_config *share_conf, void ksmbd_vfs_query_maximal_access(struct mnt_idmap *idmap, struct dentry *dentry, __le32 *daccess) { - *daccess = cpu_to_le32(FILE_READ_ATTRIBUTES | READ_CONTROL); + *daccess = cpu_to_le32(FILE_READ_ATTRIBUTES | READ_CONTROL | + SYNCHRONIZE); if (!inode_permission(idmap, d_inode(dentry), MAY_OPEN | MAY_WRITE)) - *daccess |= cpu_to_le32(WRITE_DAC | WRITE_OWNER | SYNCHRONIZE | + *daccess |= cpu_to_le32(WRITE_DAC | WRITE_OWNER | FILE_WRITE_DATA | FILE_APPEND_DATA | FILE_WRITE_EA | FILE_WRITE_ATTRIBUTES | FILE_DELETE_CHILD); @@ -268,7 +273,7 @@ static int ksmbd_vfs_stream_read(struct ksmbd_file *fp, char *buf, loff_t *pos, return (int)v_len; if (v_len <= *pos) { - count = -EINVAL; + count = 0; goto free_buf; } @@ -387,23 +392,21 @@ static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos, const struct cred *saved_cred; char *stream_buf = NULL, *wbuf; struct mnt_idmap *idmap = file_mnt_idmap(fp->filp); - size_t size; + size_t size, write_end; ssize_t v_len; int err = 0; ksmbd_debug(VFS, "write stream data pos : %llu, count : %zd\n", *pos, count); - if (*pos >= XATTR_SIZE_MAX) { + if (*pos < 0 || *pos >= XATTR_SIZE_MAX) { pr_err("stream write position %lld is out of bounds\n", *pos); return -EINVAL; } - size = *pos + count; - if (size > XATTR_SIZE_MAX) { - size = XATTR_SIZE_MAX; - count = XATTR_SIZE_MAX - *pos; - } + if (count > XATTR_SIZE_MAX - *pos) + return -EFBIG; + write_end = *pos + count; saved_cred = override_creds(fp->filp->f_cred); v_len = ksmbd_vfs_getcasexattr(idmap, @@ -416,6 +419,8 @@ static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos, err = v_len; goto out_revert; } + /* Preserve the tail of an existing stream on an in-place write. */ + size = max_t(size_t, v_len, write_end); if (v_len < size) { wbuf = kvzalloc(size, KSMBD_DEFAULT_GFP); @@ -444,13 +449,69 @@ static int ksmbd_vfs_stream_write(struct ksmbd_file *fp, char *buf, loff_t *pos, if (err < 0) goto out; else - fp->stream.pos = size; + fp->stream.pos = write_end; err = 0; out: kvfree(stream_buf); return err; } +/** + * ksmbd_vfs_stream_truncate() - change the length of a named stream + * @fp: stream file handle + * @newsize: new stream length + * + * Resize the stream xattr without changing the base inode size. + * + * Return: 0 on success, otherwise a negative error code + */ +int ksmbd_vfs_stream_truncate(struct ksmbd_file *fp, loff_t newsize) +{ + const struct cred *saved_cred; + struct mnt_idmap *idmap = file_mnt_idmap(fp->filp); + char *stream_buf = NULL, *new_buf = NULL; + ssize_t v_len; + int err; + + if (newsize < 0) + return -EINVAL; + if (newsize > XATTR_SIZE_MAX) + return -EFBIG; + + saved_cred = override_creds(fp->filp->f_cred); + v_len = ksmbd_vfs_getcasexattr(idmap, fp->filp->f_path.dentry, + fp->stream.name, fp->stream.size, + &stream_buf); + if (v_len < 0) { + err = v_len; + goto out; + } + if (v_len == newsize) { + err = 0; + goto out; + } + + new_buf = stream_buf; + if (newsize > v_len) { + new_buf = kvzalloc(newsize, KSMBD_DEFAULT_GFP); + if (!new_buf) { + err = -ENOMEM; + goto out; + } + if (v_len) + memcpy(new_buf, stream_buf, v_len); + } + + err = ksmbd_vfs_setxattr(idmap, &fp->filp->f_path, fp->stream.name, + new_buf, newsize, 0, true); + if (new_buf != stream_buf) + kvfree(new_buf); +out: + kvfree(stream_buf); + revert_creds(saved_cred); + return err; +} + /** * ksmbd_vfs_write() - vfs helper for smb file write * @work: work @@ -955,6 +1016,15 @@ int ksmbd_vfs_zero_data(struct ksmbd_work *work, struct ksmbd_file *fp, char *zero_buf = NULL; int err; + /* + * A directory handle can hold FILE_WRITE_DATA (the add-file right), + * so the caller access check does not exclude it. Writing zeroes to a + * directory makes no sense and kernel_write() would warn on the + * directory file (no FMODE_WRITE), so reject it here. + */ + if (S_ISDIR(file_inode(fp->filp)->i_mode)) + return -EINVAL; + smb_break_all_levII_oplock(work, fp, 1); if (!work->tcon->posix_extensions) { loff_t size = i_size_read(file_inode(fp->filp)); @@ -1897,13 +1967,17 @@ int ksmbd_vfs_fill_dentry_attrs(struct ksmbd_work *work, ssize_t ksmbd_vfs_casexattr_len(struct mnt_idmap *idmap, struct dentry *dentry, char *attr_name, - int attr_name_len) + int attr_name_len, char *actual_name) { char *name, *xattr_list = NULL; ssize_t value_len = -ENOENT, xattr_list_len; xattr_list_len = ksmbd_vfs_listxattr(dentry, &xattr_list); - if (xattr_list_len <= 0) + if (xattr_list_len < 0) { + value_len = xattr_list_len; + goto out; + } + if (!xattr_list_len) goto out; for (name = xattr_list; name - xattr_list < xattr_list_len; @@ -1911,8 +1985,13 @@ ssize_t ksmbd_vfs_casexattr_len(struct mnt_idmap *idmap, ksmbd_debug(VFS, "%s, len %zd\n", name, strlen(name)); if (strncasecmp(attr_name, name, attr_name_len)) continue; + if (actual_name && strlen(name) + 1 != attr_name_len) + continue; value_len = ksmbd_vfs_xattr_len(idmap, dentry, name); + /* The caller provides attr_name_len bytes for the actual name. */ + if (value_len >= 0 && actual_name) + memcpy(actual_name, name, attr_name_len); break; } @@ -2067,7 +2146,7 @@ int ksmbd_vfs_copy_file_ranges(struct ksmbd_work *work, src_file_size = ksmbd_vfs_casexattr_len( file_mnt_idmap(src_fp->filp), src_fp->filp->f_path.dentry, - src_fp->stream.name, src_fp->stream.size); + src_fp->stream.name, src_fp->stream.size, NULL); revert_creds(saved_cred); if (src_file_size < 0) return src_file_size; diff --git a/fs/smb/server/vfs.h b/fs/smb/server/vfs.h index 55d099de71f5..53044549571a 100644 --- a/fs/smb/server/vfs.h +++ b/fs/smb/server/vfs.h @@ -83,6 +83,7 @@ int ksmbd_vfs_read(struct ksmbd_work *work, struct ksmbd_file *fp, size_t count, int ksmbd_vfs_write(struct ksmbd_work *work, struct ksmbd_file *fp, char *buf, size_t count, loff_t *pos, bool sync, ssize_t *written); +int ksmbd_vfs_stream_truncate(struct ksmbd_file *fp, loff_t newsize); int ksmbd_vfs_fsync(struct ksmbd_work *work, u64 fid, u64 p_id); int ksmbd_vfs_remove_file(struct ksmbd_work *work, const struct path *path); int ksmbd_vfs_link(struct ksmbd_work *work, @@ -110,7 +111,7 @@ ssize_t ksmbd_vfs_getxattr(struct mnt_idmap *idmap, char **xattr_buf); ssize_t ksmbd_vfs_casexattr_len(struct mnt_idmap *idmap, struct dentry *dentry, char *attr_name, - int attr_name_len); + int attr_name_len, char *actual_name); int ksmbd_vfs_setxattr(struct mnt_idmap *idmap, const struct path *path, const char *attr_name, void *attr_value, size_t attr_size, int flags, diff --git a/fs/smb/server/vfs_cache.c b/fs/smb/server/vfs_cache.c index fd2c595f0486..1efa9f814560 100644 --- a/fs/smb/server/vfs_cache.c +++ b/fs/smb/server/vfs_cache.c @@ -605,7 +605,7 @@ static void __ksmbd_remove_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp list_del_init(&fp->node); up_write(&fp->f_ci->m_lock); - if (!has_file_id(fp->volatile_id)) + if (!ft || !has_file_id(fp->volatile_id)) return; write_lock(&ft->lock); @@ -621,8 +621,7 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp) fd_limit_close(); ksmbd_remove_durable_fd(fp); - if (ft) - __ksmbd_remove_fd(ft, fp); + __ksmbd_remove_fd(ft, fp); close_id_del_oplock(fp); filp = fp->filp; @@ -705,12 +704,14 @@ static void __ksmbd_close_fd(struct ksmbd_file_table *ft, struct ksmbd_file *fp) * in which case there is nothing to drop here. */ if (fp->conn) { + atomic_dec(&fp->conn->stats.open_files_count); ksmbd_conn_put(fp->conn); fp->conn = NULL; } if (ksmbd_stream_fd(fp)) kfree(fp->stream.name); + ksmbd_share_config_put(fp->share_conf); kfree(fp->owner.name); kmem_cache_free(filp_cache, fp); @@ -821,22 +822,7 @@ static struct ksmbd_file *__ksmbd_lookup_fd(struct ksmbd_file_table *ft, static void __put_fd_final(struct ksmbd_work *work, struct ksmbd_file *fp) { - /* - * Detached durable fp -- session_fd_check() cleared fp->conn at - * preserve, so this fp is no longer tracked by any conn's - * stats.open_files_count. This happens when - * ksmbd_scavenger_dispose_dh() hands the final close off to an - * m_fp_list walker (e.g. ksmbd_lookup_fd_inode()) whose work->conn - * is unrelated to the conn that originally opened the handle; close - * via the NULL-ft path so we do not underflow that unrelated - * counter. - */ - if (!fp->conn) { - __ksmbd_close_fd(NULL, fp); - return; - } - __ksmbd_close_fd(&work->sess->file_table, fp); - atomic_dec(&work->conn->stats.open_files_count); + __ksmbd_close_fd(fp->conn ? &work->sess->file_table : NULL, fp); } static void set_close_state_blocked_works(struct ksmbd_file *fp) @@ -865,6 +851,34 @@ void ksmbd_wake_session_blocked_works(struct ksmbd_session *sess) read_unlock(&ft->lock); } +/** + * ksmbd_unpublish_reopened_fd() - remove a reopened durable handle from the + * session file table after its CREATE failed + * @work: smb work of the failed CREATE + * @fp: durable handle republished by ksmbd_reopen_durable_fd() + * + * The reopen publishes @fp in this session's table without taking a reference + * for the entry, so the CREATE error path's puts need not be the last ones. + * Another holder of a global-table reference, such as a concurrent reconnect + * of the same handle, can drop the last one through ksmbd_put_durable_fd(), + * which frees @fp without touching this table. Remove the id here, while + * the caller still holds its references. + * + * Every path that removes the id before the final put also clears + * @fp->volatile_id under ft->lock, so a valid id is still @fp's entry. + */ +void ksmbd_unpublish_reopened_fd(struct ksmbd_work *work, struct ksmbd_file *fp) +{ + struct ksmbd_file_table *ft = &work->sess->file_table; + + write_lock(&ft->lock); + if (has_file_id(fp->volatile_id)) { + idr_remove(ft->idr, fp->volatile_id); + fp->volatile_id = KSMBD_NO_FID; + } + write_unlock(&ft->lock); +} + int ksmbd_close_fd(struct ksmbd_work *work, u64 id) { struct ksmbd_file *fp; @@ -1009,37 +1023,56 @@ bool ksmbd_has_other_active_fd(struct ksmbd_file *fp) return ret; } -struct ksmbd_file *ksmbd_lookup_fd_app_instance_id(char *app_instance_id) +bool ksmbd_app_instance_id_matches(struct ksmbd_work *work, + struct ksmbd_file *fp) { - struct ksmbd_file *fp = NULL; - unsigned int id; + bool matches; + + /* Keep the current connection and share alive across the comparison. */ + spin_lock(&fp->f_lock); + matches = READ_ONCE(fp->f_state) == FP_INITED && + fp->conn && fp->share_conf && + !strcmp(fp->share_conf->name, work->tcon->share_conf->name) && + memcmp(fp->conn->ClientGUID, work->conn->ClientGUID, + SMB2_CLIENT_GUID_SIZE); + spin_unlock(&fp->f_lock); + + return matches; +} + +struct ksmbd_file * +ksmbd_lookup_fd_app_instance_id_next(char *app_instance_id, int *cursor) +{ + struct ksmbd_file *fp, *found = NULL; + int id = *cursor; read_lock(&global_ft.lock); - idr_for_each_entry(global_ft.idr, fp, id) { - if (!fp->has_app_instance_id) - continue; - if (!memcmp(fp->app_instance_id, app_instance_id, + while ((fp = idr_get_next(global_ft.idr, &id))) { + *cursor = id + 1; + if (fp->has_app_instance_id && + !memcmp(fp->app_instance_id, app_instance_id, SMB2_CREATE_GUID_SIZE)) { - fp = ksmbd_fp_get(fp); - break; + found = ksmbd_fp_get(fp); + if (found) + break; } + id++; } read_unlock(&global_ft.lock); - return fp; + return found; } -int ksmbd_close_fd_app_instance_id(char *app_instance_id) +int ksmbd_close_fd_app_instance_id(struct ksmbd_work *work, + struct ksmbd_file *fp, bool *closed) { struct ksmbd_file_table *ft; - struct ksmbd_file *fp; + struct ksmbd_session *sess; struct oplock_info *opinfo; + u64 session_id; int n_to_drop = 0; - fp = ksmbd_lookup_fd_app_instance_id(app_instance_id); - if (!fp) - return 0; - + *closed = false; opinfo = opinfo_get(fp); if (!opinfo) goto out; @@ -1049,16 +1082,43 @@ int ksmbd_close_fd_app_instance_id(char *app_instance_id) up_read(&fp->f_ci->m_lock); goto out; } + /* + * opinfo->sess is not refcounted, and m_lock does not keep it alive. + * The owner can be torn down while we hold only a reference to fp. + * Look the owner up by id to take a real reference instead. + */ + session_id = opinfo->session_id; + up_read(&fp->f_ci->m_lock); - ft = &opinfo->sess->file_table; + /* destroy_previous_session() takes m_lock under sessions_table_lock. */ + sess = ksmbd_session_lookup_slowpath(session_id); + if (!sess) + goto out; + + down_read(&fp->f_ci->m_lock); + if (opinfo->sess != sess) { + up_read(&fp->f_ci->m_lock); + ksmbd_user_session_put(sess); + goto out; + } + ft = &sess->file_table; write_lock(&ft->lock); - if (fp->f_state == FP_INITED && has_file_id(fp->volatile_id)) { + /* + * The table may already be detached by destroy_previous_session(). + * A new session could also reuse the old id and address and pass the + * check above. Only remove fp if this table still publishes it. + */ + if (ft->idr && fp->f_state == FP_INITED && + has_file_id(fp->volatile_id) && + idr_find(ft->idr, fp->volatile_id) == fp && + ksmbd_app_instance_id_matches(work, fp)) { idr_remove(ft->idr, fp->volatile_id); fp->volatile_id = KSMBD_NO_FID; n_to_drop = ksmbd_mark_fp_closed(fp); } write_unlock(&ft->lock); up_read(&fp->f_ci->m_lock); + ksmbd_user_session_put(sess); opinfo_put(opinfo); opinfo = NULL; @@ -1069,11 +1129,9 @@ int ksmbd_close_fd_app_instance_id(char *app_instance_id) list_del_init(&fp->node); up_write(&fp->f_ci->m_lock); - if (atomic_sub_and_test(n_to_drop, &fp->refcount)) { - if (fp->conn) - atomic_dec(&fp->conn->stats.open_files_count); + *closed = true; + if (atomic_sub_and_test(n_to_drop, &fp->refcount)) __ksmbd_close_fd(NULL, fp); - } return 0; out: @@ -1281,6 +1339,8 @@ struct ksmbd_file *ksmbd_open_fd(struct ksmbd_work *work, struct file *filp) */ fp->conn = ksmbd_conn_get(work->conn); fp->tcon = work->tcon; + fp->share_conf = work->tcon->share_conf; + atomic_inc(&fp->share_conf->refcount); fp->volatile_id = KSMBD_NO_FID; fp->persistent_id = KSMBD_NO_FID; fp->f_state = FP_NEW; @@ -1303,6 +1363,7 @@ struct ksmbd_file *ksmbd_open_fd(struct ksmbd_work *work, struct file *filp) err_out: /* fp->conn was set and refcounted before every branch here. */ ksmbd_conn_put(fp->conn); + ksmbd_share_config_put(fp->share_conf); kmem_cache_free(filp_cache, fp); return ERR_PTR(ret); } @@ -1361,7 +1422,7 @@ static int ksmbd_mark_fp_closed(struct ksmbd_file *fp) return 1; } -static int +static void __close_file_table_ids(struct ksmbd_session *sess, struct ksmbd_tree_connect *tcon, bool (*skip)(struct ksmbd_tree_connect *tcon, @@ -1372,7 +1433,6 @@ __close_file_table_ids(struct ksmbd_session *sess, struct ksmbd_file_table *ft = &sess->file_table; struct ksmbd_file *fp; unsigned int id = 0; - int num = 0; while (1) { int n_to_drop; @@ -1456,10 +1516,8 @@ __close_file_table_ids(struct ksmbd_session *sess, } /* - * fp->volatile_id is already cleared to prevent stale idr - * removal from a deferred final close. Remove fp from - * m_fp_list here because __ksmbd_remove_fd() will skip the - * list unlink when volatile_id is KSMBD_NO_FID. + * Stop inode walkers from finding the open after close is committed. + * Final close also unlinks fp if a global lookup owns the last put. */ down_write(&fp->f_ci->m_lock); list_del_init(&fp->node); @@ -1478,22 +1536,13 @@ __close_file_table_ids(struct ksmbd_session *sess, * the original reference until ksmbd_update_fstate() * observes the cleared volatile id. * - * If we end up as the final putter, finalize fp and - * account the open_files_count decrement via the caller's - * atomic_sub(num, ...). Otherwise the remaining user's - * ksmbd_fd_put() reaches __put_fd_final(), which does its - * own atomic_dec(&open_files_count), so we must not count - * this fp here -- doing so would double-decrement the - * connection-wide counter. + * The final close accounts the open against fp->conn, + * including when an in-flight user drops the last reference. */ - if (atomic_sub_and_test(n_to_drop, &fp->refcount)) { + if (atomic_sub_and_test(n_to_drop, &fp->refcount)) __ksmbd_close_fd(NULL, fp); - num++; - } id++; } - - return num; } static inline bool is_reconnectable(struct ksmbd_file *fp) @@ -1782,19 +1831,18 @@ static bool session_fd_check(struct ksmbd_tree_connect *tcon, * / ksmbd_reopen_durable_fd()), so conn stays valid for the whole * body of this function regardless of any op->conn puts below. */ - conn = fp->conn; ci = fp->f_ci; down_write(&ci->m_lock); + conn = fp->conn; list_for_each_entry_rcu(op, &ci->m_op_list, op_entry, lockdep_is_held(&ci->m_lock)) { - if (op->conn != conn) + if (op->o_fp != fp) continue; ksmbd_conn_put(op->conn); op->conn = NULL; op->sess = NULL; + op->session_id = 0; } - up_write(&ci->m_lock); - list_for_each_entry_safe(smb_lock, tmp_lock, &fp->lock_list, flist) { struct ksmbd_conn *lock_conn = smb_lock->conn; @@ -1807,6 +1855,7 @@ static bool session_fd_check(struct ksmbd_tree_connect *tcon, ksmbd_conn_put(lock_conn); } + write_lock(&global_ft.lock); fp->conn = NULL; fp->tcon = NULL; fp->volatile_id = KSMBD_NO_FID; @@ -1814,6 +1863,9 @@ static bool session_fd_check(struct ksmbd_tree_connect *tcon, if (fp->durable_timeout) fp->durable_scavenger_timeout = jiffies_to_msecs(jiffies) + fp->durable_timeout; + atomic_dec(&conn->stats.open_files_count); + write_unlock(&global_ft.lock); + up_write(&ci->m_lock); /* Drop fp's own reference on conn. */ ksmbd_conn_put(conn); @@ -1822,22 +1874,12 @@ static bool session_fd_check(struct ksmbd_tree_connect *tcon, void ksmbd_close_tree_conn_fds(struct ksmbd_work *work) { - int num = __close_file_table_ids(work->sess, - work->tcon, - tree_conn_fd_check, - false); - - atomic_sub(num, &work->conn->stats.open_files_count); + __close_file_table_ids(work->sess, work->tcon, tree_conn_fd_check, false); } void ksmbd_close_session_fds(struct ksmbd_work *work) { - int num = __close_file_table_ids(work->sess, - work->tcon, - session_fd_check, - true); - - atomic_sub(num, &work->conn->stats.open_files_count); + __close_file_table_ids(work->sess, work->tcon, session_fd_check, true); } int ksmbd_init_global_file_table(void) @@ -1877,9 +1919,21 @@ int ksmbd_validate_name_reconnect(struct ksmbd_share_config *share, return -EACCES; } - if (name && strcmp(&ab_pathname[share->path_sz + 1], name)) { - ksmbd_debug(SMB, "invalid name reconnect %s\n", name); - ret = -EINVAL; + if (name) { + size_t len = strlen(ab_pathname); + + if (len == share->path_sz && !strncmp(ab_pathname, share->path, len)) { + /* the durable fp is the share root itself */ + if (name[0]) + ret = -EINVAL; + } else if (len <= share->path_sz || + strncmp(ab_pathname, share->path, share->path_sz) || + ab_pathname[share->path_sz] != '/' || + strcmp(&ab_pathname[share->path_sz + 1], name)) { + ret = -EINVAL; + } + if (ret) + ksmbd_debug(SMB, "invalid name reconnect %s\n", name); } kfree(pathname); @@ -1889,23 +1943,27 @@ int ksmbd_validate_name_reconnect(struct ksmbd_share_config *share, int ksmbd_reopen_durable_fd(struct ksmbd_work *work, struct ksmbd_file *fp) { - struct ksmbd_inode *ci; + struct ksmbd_inode *ci = fp->f_ci; struct oplock_info *op; struct ksmbd_conn *conn = work->conn; + struct ksmbd_share_config *old_share; struct ksmbd_lock *smb_lock; unsigned int old_f_state; + int ret = -EBADF; + down_write(&ci->m_lock); write_lock(&global_ft.lock); - if ((!fp->is_durable && !fp->is_persistent) || fp->conn || fp->tcon) { + if ((!fp->is_durable && !fp->is_persistent) || fp->conn || fp->tcon || + fp->f_state != FP_INITED) { write_unlock(&global_ft.lock); pr_err("Invalid durable fd [%p:%p]\n", fp->conn, fp->tcon); - return -EBADF; + goto out; } if (has_file_id(fp->volatile_id)) { write_unlock(&global_ft.lock); pr_err("Still in use durable fd: %llu\n", fp->volatile_id); - return -EBADF; + goto out; } /* @@ -1916,22 +1974,32 @@ int ksmbd_reopen_durable_fd(struct ksmbd_work *work, struct ksmbd_file *fp) * partially-initialized fp. fp owns a strong reference on the new * conn (see ksmbd_open_fd()); undo it on __open_id() failure. */ + spin_lock(&fp->f_lock); + old_f_state = fp->f_state; + fp->f_state = FP_NEW; + old_share = fp->share_conf; + fp->share_conf = work->tcon->share_conf; + atomic_inc(&fp->share_conf->refcount); fp->conn = ksmbd_conn_get(conn); fp->tcon = work->tcon; + spin_unlock(&fp->f_lock); + atomic_inc(&conn->stats.open_files_count); write_unlock(&global_ft.lock); - old_f_state = fp->f_state; - fp->f_state = FP_NEW; - __open_id(&work->sess->file_table, fp, OPEN_ID_TYPE_VOLATILE_ID); if (!has_file_id(fp->volatile_id)) { write_lock(&global_ft.lock); + spin_lock(&fp->f_lock); fp->conn = NULL; fp->tcon = NULL; + fp->share_conf = old_share; + fp->f_state = old_f_state; + spin_unlock(&fp->f_lock); + atomic_dec(&conn->stats.open_files_count); write_unlock(&global_ft.lock); ksmbd_conn_put(conn); - fp->f_state = old_f_state; - return -EBADF; + ksmbd_share_config_put(work->tcon->share_conf); + goto out; } list_for_each_entry(smb_lock, &fp->lock_list, flist) { @@ -1941,24 +2009,26 @@ int ksmbd_reopen_durable_fd(struct ksmbd_work *work, struct ksmbd_file *fp) spin_unlock(&conn->llist_lock); } - ci = fp->f_ci; - down_write(&ci->m_lock); list_for_each_entry_rcu(op, &ci->m_op_list, op_entry, lockdep_is_held(&ci->m_lock)) { if (op->conn || op->o_fp != fp) continue; op->conn = ksmbd_conn_get(fp->conn); op->sess = work->sess; + op->session_id = work->sess->id; } - up_write(&ci->m_lock); spin_lock(&fp->f_lock); fp->owner.uid = fp->owner.gid = 0; kfree(fp->owner.name); fp->owner.name = NULL; spin_unlock(&fp->f_lock); + ksmbd_share_config_put(old_share); - return 0; + ret = 0; +out: + up_write(&ci->m_lock); + return ret; } int ksmbd_init_file_table(struct ksmbd_file_table *ft) @@ -1975,14 +2045,19 @@ int ksmbd_init_file_table(struct ksmbd_file_table *ft) void ksmbd_destroy_file_table(struct ksmbd_session *sess) { struct ksmbd_file_table *ft = &sess->file_table; + struct idr *idr; if (!ft->idr) return; __close_file_table_ids(sess, NULL, session_fd_check, true); - idr_destroy(ft->idr); - kfree(ft->idr); + /* Stop procfs readers before freeing the detached file table. */ + write_lock(&ft->lock); + idr = ft->idr; ft->idr = NULL; + write_unlock(&ft->lock); + idr_destroy(idr); + kfree(idr); } int ksmbd_init_file_cache(void) diff --git a/fs/smb/server/vfs_cache.h b/fs/smb/server/vfs_cache.h index 1884f6deb9d0..84ca73f48314 100644 --- a/fs/smb/server/vfs_cache.h +++ b/fs/smb/server/vfs_cache.h @@ -102,6 +102,8 @@ struct ksmbd_file { struct oplock_info __rcu *f_opinfo; struct ksmbd_conn *conn; struct ksmbd_tree_connect *tcon; + /* Own a share reference independently of tcon; rebind under f_lock. */ + struct ksmbd_share_config *share_conf; atomic_t refcount; __le32 daccess; @@ -196,6 +198,8 @@ static inline bool ksmbd_stream_fd(struct ksmbd_file *fp) int ksmbd_init_file_table(struct ksmbd_file_table *ft); void ksmbd_destroy_file_table(struct ksmbd_session *sess); int ksmbd_close_fd(struct ksmbd_work *work, u64 id); +void ksmbd_unpublish_reopened_fd(struct ksmbd_work *work, + struct ksmbd_file *fp); struct ksmbd_file *ksmbd_lookup_fd_fast(struct ksmbd_work *work, u64 id); struct ksmbd_file *ksmbd_lookup_foreign_fd(struct ksmbd_work *work, u64 id); struct ksmbd_file *ksmbd_lookup_fd_slow(struct ksmbd_work *work, u64 id, @@ -213,8 +217,12 @@ void ksmbd_put_durable_fd(struct ksmbd_file *fp); int ksmbd_invalidate_durable_fd(unsigned long long id); bool ksmbd_has_other_active_fd(struct ksmbd_file *fp); bool ksmbd_has_stream_without_delete_share(struct ksmbd_file *fp); -struct ksmbd_file *ksmbd_lookup_fd_app_instance_id(char *app_instance_id); -int ksmbd_close_fd_app_instance_id(char *app_instance_id); +bool ksmbd_app_instance_id_matches(struct ksmbd_work *work, + struct ksmbd_file *fp); +struct ksmbd_file * +ksmbd_lookup_fd_app_instance_id_next(char *app_instance_id, int *cursor); +int ksmbd_close_fd_app_instance_id(struct ksmbd_work *work, + struct ksmbd_file *fp, bool *closed); struct ksmbd_file *ksmbd_lookup_fd_cguid(char *cguid); struct ksmbd_file *ksmbd_lookup_fd_inode(struct dentry *dentry); bool ksmbd_has_other_nonposix_open(struct dentry *dentry); diff --git a/fs/smb/smbdirect/accept.c b/fs/smb/smbdirect/accept.c index 1b30ca8476c1..e7e5eb75a5ae 100644 --- a/fs/smb/smbdirect/accept.c +++ b/fs/smb/smbdirect/accept.c @@ -19,15 +19,33 @@ int smbdirect_accept_connect_request(struct smbdirect_socket *sc, const struct rdma_conn_param *param) { struct smbdirect_socket_parameters *sp = &sc->parameters; - struct smbdirect_recv_io *recv_io; + struct smbdirect_recv_io *recv_io = NULL; u8 peer_initiator_depth; u8 peer_responder_resources; struct rdma_conn_param conn_param; __be32 ird_ord_hdr[2]; int ret; - if (SMBDIRECT_CHECK_STATUS_WARN(sc, SMBDIRECT_SOCKET_CREATED)) - return -EINVAL; + /* + * Install the real event handler (and the event we expect next) + * before anything that can fail. On an early error the caller + * (smbdirect_listen_connect_request()) only schedules our deferred + * cleanup and returns 0 to the rdma_cm core, which keeps the cm_id, so + * the cm_id must already carry smbdirect_accept_rdma_event_handler(): + * it turns any unexpected event (e.g. RDMA_CM_EVENT_DEVICE_REMOVAL) + * into a clean smbdirect_socket_schedule_cleanup(). Otherwise the + * placeholder smbdirect_socket_rdma_event_handler() installed by + * smbdirect_socket_init_accepting() stays in place and returns -ESTALE, + * which makes the rdma_cm core destroy the cm_id behind the back of our + * pending teardown. + */ + sc->rdma.expected_event = RDMA_CM_EVENT_ESTABLISHED; + sc->rdma.cm_id->event_handler = smbdirect_accept_rdma_event_handler; + + if (SMBDIRECT_CHECK_STATUS_WARN(sc, SMBDIRECT_SOCKET_CREATED)) { + ret = -EINVAL; + goto cleanup; + } /* * First set what the we as server are able to support @@ -48,7 +66,7 @@ int smbdirect_accept_connect_request(struct smbdirect_socket *sc, smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_ERR, "smbdirect_accept_init_params() failed %1pe\n", SMBDIRECT_DEBUG_ERR_PTR(ret)); - goto init_params_failed; + goto cleanup; } ret = smbdirect_connection_create_qp(sc); @@ -56,7 +74,7 @@ int smbdirect_accept_connect_request(struct smbdirect_socket *sc, smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_ERR, "smbdirect_connection_create_qp() failed %1pe\n", SMBDIRECT_DEBUG_ERR_PTR(ret)); - goto create_qp_failed; + goto cleanup; } ret = smbdirect_connection_create_mem_pools(sc); @@ -64,7 +82,7 @@ int smbdirect_accept_connect_request(struct smbdirect_socket *sc, smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_ERR, "smbdirect_connection_create_mem_pools() failed %1pe\n", SMBDIRECT_DEBUG_ERR_PTR(ret)); - goto create_mem_failed; + goto cleanup; } recv_io = smbdirect_connection_get_recv_io(sc); @@ -73,7 +91,7 @@ int smbdirect_accept_connect_request(struct smbdirect_socket *sc, smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_ERR, "smbdirect_connection_get_recv_io() failed %1pe\n", SMBDIRECT_DEBUG_ERR_PTR(ret)); - goto get_recv_io_failed; + goto cleanup; } recv_io->cqe.done = smbdirect_accept_negotiate_recv_done; @@ -87,7 +105,7 @@ int smbdirect_accept_connect_request(struct smbdirect_socket *sc, smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_ERR, "smbdirect_connection_post_recv_io() failed %1pe\n", SMBDIRECT_DEBUG_ERR_PTR(ret)); - goto post_recv_io_failed; + goto cleanup; } /* * From here recv_io is known to the RDMA QP and needs ib_drain_qp and @@ -123,14 +141,12 @@ int smbdirect_accept_connect_request(struct smbdirect_socket *sc, /* explicitly set above */ WARN_ON_ONCE(sc->status != SMBDIRECT_SOCKET_RDMA_CONNECT_NEEDED); sc->status = SMBDIRECT_SOCKET_RDMA_CONNECT_RUNNING; - sc->rdma.expected_event = RDMA_CM_EVENT_ESTABLISHED; - sc->rdma.cm_id->event_handler = smbdirect_accept_rdma_event_handler; ret = rdma_accept(sc->rdma.cm_id, &conn_param); if (ret) { smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_ERR, "rdma_accept() failed %1pe\n", SMBDIRECT_DEBUG_ERR_PTR(ret)); - goto rdma_accept_failed; + goto cleanup; } /* @@ -144,31 +160,25 @@ int smbdirect_accept_connect_request(struct smbdirect_socket *sc, return 0; -rdma_accept_failed: +cleanup: /* - * The recv_io posted above is now owned by the QP (recv_io was set to - * NULL after a successful post). smbdirect_connection_destroy_qp() - * calls ib_drain_qp(), whose completion - * (smbdirect_accept_negotiate_recv_done) returns the recv_io to the - * free list via smbdirect_connection_put_recv_io(). It therefore MUST - * run BEFORE smbdirect_connection_destroy_mem_pools(): otherwise the - * posted recv_io is still outstanding when kmem_cache_destroy() runs - * ("Slab cache still has objects") and is later freed into an - * already-destroyed mempool (mempool_free_bulk NULL-ptr-deref). + * No RDMA state is torn down here, and in particular sc is not + * released: the caller (smbdirect_listen_connect_request()) owns that + * decision and schedules sc's teardown via + * smbdirect_socket_schedule_cleanup() on a non-zero return. All of the + * RDMA state - the QP, the memory pools and the cm_id itself - is then + * torn down asynchronously by smbdirect_socket_cleanup_work() -> + * smbdirect_socket_release() -> smbdirect_socket_destroy(), which + * drains the QP (returning any posted recv_io to the pool) before + * destroying the memory pools and destroys the cm_id with + * rdma_destroy_id(), all in the correct order. + * + * The only resource not reachable by that teardown is a recv_io that + * was taken from the pool but not yet posted to the QP (recv_io is set + * to NULL after a successful post), so hand it back here. */ - smbdirect_connection_destroy_qp(sc); - smbdirect_connection_destroy_mem_pools(sc); - return ret; -post_recv_io_failed: - /* post failed: recv_io was not accepted by the QP, still in hand */ if (recv_io) smbdirect_connection_put_recv_io(recv_io); -get_recv_io_failed: - smbdirect_connection_destroy_mem_pools(sc); -create_mem_failed: - smbdirect_connection_destroy_qp(sc); -create_qp_failed: -init_params_failed: return ret; } @@ -302,6 +312,7 @@ static void smbdirect_accept_negotiate_recv_work(struct work_struct *work) struct smbdirect_socket *sc = container_of(work, struct smbdirect_socket, connect.work); struct smbdirect_socket_parameters *sp = &sc->parameters; + struct smbdirect_socket *lsc; struct smbdirect_recv_io *recv_io; struct smbdirect_negotiate_req *nreq; unsigned long flags; @@ -464,29 +475,55 @@ static void smbdirect_accept_negotiate_recv_work(struct work_struct *work) */ sp->max_fragmented_send_size = max_fragmented_size; - if (sc->accept.listener) { - struct smbdirect_socket *lsc = sc->accept.listener; - unsigned long flags; + /* + * Every accepting socket was created by + * smbdirect_listen_connect_request() and has a + * listener. If it's already cleared, the listener + * (or its purge_orphaned_work) is responsible for + * releasing us, so we must not send a negotiate + * response. + * + * The memory of the listener is freed via kfree_rcu(), + * see smbdirect_listen_orphan_socket(). + */ + rcu_read_lock(); + lsc = READ_ONCE(sc->accept.listener); + if (!lsc) { + rcu_read_unlock(); + return; + } - spin_lock_irqsave(&lsc->listen.lock, flags); - list_del(&sc->accept.list); - list_add_tail(&sc->accept.list, &lsc->listen.ready); + spin_lock_irqsave(&lsc->listen.lock, flags); + /* + * The listener (or its purge_orphaned_work) + * may have cleared sc->accept.listener in the + * meantime and is responsible for releasing us. + * + * If we already failed, we're either + * already on the orphaned list or + * smbdirect_socket_cleanup_work() will + * move us there. + * + * In both cases we must not move us + * to the ready list. + */ + if (sc->accept.listener == lsc && !READ_ONCE(sc->first_error)) { + list_move_tail(&sc->accept.list, &lsc->listen.ready); wake_up(&lsc->listen.wait_queue); - spin_unlock_irqrestore(&lsc->listen.lock, flags); - - /* - * smbdirect_socket_accept() will call - * smbdirect_accept_negotiate_finish(nsc, 0); - * - * So that we don't send the negotiation - * response that grants credits to the peer - * before the socket is accepted by the - * application. - */ - return; } + spin_unlock_irqrestore(&lsc->listen.lock, flags); + rcu_read_unlock(); - ntstatus = le32_to_cpu(STATUS_SUCCESS); + /* + * smbdirect_socket_accept() will call + * smbdirect_accept_negotiate_finish(nsc, 0); + * + * So that we don't send the negotiation + * response that grants credits to the peer + * before the socket is accepted by the + * application. + */ + return; not_supported: smbdirect_accept_negotiate_finish(sc, ntstatus); @@ -799,7 +836,11 @@ static long smbdirect_socket_wait_for_accept(struct smbdirect_socket *lsc, long if (ret < 0) return ret; - return 0; + /* + * Return the remaining timeout, so the caller can carry it + * over to the next smbdirect_socket_wait_for_accept() call. + */ + return ret; } struct smbdirect_socket *smbdirect_socket_accept(struct smbdirect_socket *lsc, @@ -809,6 +850,7 @@ struct smbdirect_socket *smbdirect_socket_accept(struct smbdirect_socket *lsc, struct smbdirect_socket *nsc; unsigned long flags; +again: if (lsc->status != SMBDIRECT_SOCKET_LISTENING) { arg->err = -EINVAL; return NULL; @@ -820,7 +862,7 @@ struct smbdirect_socket *smbdirect_socket_accept(struct smbdirect_socket *lsc, } if (list_empty_careful(&lsc->listen.ready)) { - int ret; + long ret; if (timeo == 0) { arg->err = -EAGAIN; @@ -828,25 +870,78 @@ struct smbdirect_socket *smbdirect_socket_accept(struct smbdirect_socket *lsc, } ret = smbdirect_socket_wait_for_accept(lsc, timeo); - if (ret) { + if (ret < 0) { arg->err = ret; return NULL; } + /* + * Carry the remaining timeout over, so that a stream of + * failed connections that we orphan and skip (goto again) + * can't reset the caller's timeout and wait forever. + */ + timeo = ret; } spin_lock_irqsave(&lsc->listen.lock, flags); - nsc = list_first_entry_or_null(&lsc->listen.ready, - struct smbdirect_socket, - accept.list); + /* + * Recheck under the lock. + */ + if (lsc->status != SMBDIRECT_SOCKET_LISTENING) { + arg->err = -EINVAL; + spin_unlock_irqrestore(&lsc->listen.lock, flags); + return NULL; + } + if (lsc->first_error) { + arg->err = lsc->first_error; + spin_unlock_irqrestore(&lsc->listen.lock, flags); + return NULL; + } + + while ((nsc = list_first_entry_or_null(&lsc->listen.ready, + struct smbdirect_socket, + accept.list))) { + /* + * nsc may have failed after it was moved + * to the ready list, e.g. RDMA_CM_EVENT_DISCONNECTED + * already moved it to SMBDIRECT_SOCKET_DISCONNECTED. + * We must not overwrite that with + * SMBDIRECT_SOCKET_CONNECTED and hand + * out an already disconnected socket. + * + * Doing this under listen.lock means nsc + * still belongs to us, so we can just move + * a failed socket to the orphaned list, + * like smbdirect_listen_orphan_socket() does, + * and try the next one. + */ + if (!READ_ONCE(nsc->first_error) && + cmpxchg(&nsc->status, + SMBDIRECT_SOCKET_NEGOTIATE_RUNNING, + SMBDIRECT_SOCKET_CONNECTED) == + SMBDIRECT_SOCKET_NEGOTIATE_RUNNING) + break; + + smbdirect_log_rdma_event(nsc, SMBDIRECT_LOG_INFO, + "orphaning failed socket status=%s first_error=%1pe\n", + smbdirect_socket_status_string(nsc->status), + SMBDIRECT_DEBUG_ERR_PTR(nsc->first_error)); + list_move_tail(&nsc->accept.list, &lsc->listen.orphaned); + queue_work(lsc->workqueues.cleanup, &lsc->listen.purge_orphaned_work); + } if (nsc) { - nsc->accept.listener = NULL; + WRITE_ONCE(nsc->accept.listener, NULL); list_del_init_careful(&nsc->accept.list); arg->is_empty = list_empty_careful(&lsc->listen.ready); } spin_unlock_irqrestore(&lsc->listen.lock, flags); if (!nsc) { - arg->err = -EAGAIN; - return NULL; + /* + * If we only found failed sockets or no socket, + * we wait for the next one. + * + * A possible -EAGAIN/-ETIMEOUT is handled above. + */ + goto again; } /* @@ -855,12 +950,14 @@ struct smbdirect_socket *smbdirect_socket_accept(struct smbdirect_socket *lsc, * so it didn't grant any credits to us. * * The caller expects a connected socket - * now as there are no credits anyway. + * now as there are no credits anyway, + * above we already changed to SMBDIRECT_SOCKET_CONNECTED + * under the lsc->listen.lock and with cmpxchg. * - * Then we send the negotiation response in - * order to grant credits to the peer. + * Now we send the negotiation response in + * order to grant credits to the peer, + * as the socket is now visible to the application layer. */ - nsc->status = SMBDIRECT_SOCKET_CONNECTED; smbdirect_accept_negotiate_finish(nsc, 0); return nsc; diff --git a/fs/smb/smbdirect/connection.c b/fs/smb/smbdirect/connection.c index afd31fa12a36..f48c857bda5a 100644 --- a/fs/smb/smbdirect/connection.c +++ b/fs/smb/smbdirect/connection.c @@ -83,9 +83,9 @@ static int smbdirect_connection_rdma_event_handler(struct rdma_cm_id *id, * smbdirect_socket_schedule_cleanup[_status]() => * smbdirect_socket_cleanup_work(). * - * As otherwise we'd set SMBDIRECT_SOCKET_DISCONNECTING, - * but never ever get RDMA_CM_EVENT_DISCONNECTED and - * never reach SMBDIRECT_SOCKET_DISCONNECTED. + * As otherwise we'd set SMBDIRECT_SOCKET_DISCONNECTING + * and call rdma_disconnect(), but never ever get + * RDMA_CM_EVENT_DISCONNECTED. */ if (event->event == RDMA_CM_EVENT_DEVICE_REMOVAL) smbdirect_socket_schedule_cleanup_status(sc, @@ -113,9 +113,9 @@ static int smbdirect_connection_rdma_event_handler(struct rdma_cm_id *id, * smbdirect_socket_schedule_cleanup_status() => * smbdirect_socket_cleanup_work(). * - * As otherwise we'd set SMBDIRECT_SOCKET_DISCONNECTING, - * but never ever get RDMA_CM_EVENT_DISCONNECTED and - * never reach SMBDIRECT_SOCKET_DISCONNECTED. + * As otherwise we'd set SMBDIRECT_SOCKET_DISCONNECTING + * and call rdma_disconnect(), but never ever get + * RDMA_CM_EVENT_DISCONNECTED. * * This is also a normal disconnect so * SMBDIRECT_LOG_INFO should be good enough diff --git a/fs/smb/smbdirect/internal.h b/fs/smb/smbdirect/internal.h index e9959e6dc13a..69d0c9f899de 100644 --- a/fs/smb/smbdirect/internal.h +++ b/fs/smb/smbdirect/internal.h @@ -74,6 +74,8 @@ void __smbdirect_socket_schedule_cleanup(struct smbdirect_socket *sc, void smbdirect_socket_destroy_sync(struct smbdirect_socket *sc); +void smbdirect_listen_orphan_socket(struct smbdirect_socket *sc); + int smbdirect_socket_wait_for_credits(struct smbdirect_socket *sc, enum smbdirect_socket_status expected_status, int unexpected_errno, diff --git a/fs/smb/smbdirect/listen.c b/fs/smb/smbdirect/listen.c index 2f78bcaedbf8..3c91f00d95ce 100644 --- a/fs/smb/smbdirect/listen.c +++ b/fs/smb/smbdirect/listen.c @@ -10,6 +10,78 @@ static int smbdirect_listen_rdma_event_handler(struct rdma_cm_id *id, struct rdma_cm_event *event); +/* + * This is called by a socket that failed while it + * is still on the pending or ready list of its + * listener, typically at the end of + * smbdirect_socket_cleanup_work(), when the + * disconnect was already started. + * + * It moves itself to the orphaned list and + * lets smbdirect_listen_purge_orphaned_work() + * release it. Otherwise it would stay on the + * pending or ready list until the listener is + * destroyed and fill up the backlog, so that + * no new connections would be accepted. + * + * It's fine to call this more than once. + */ +void smbdirect_listen_orphan_socket(struct smbdirect_socket *sc) +{ + struct smbdirect_socket *lsc; + unsigned long flags; + + /* + * The memory of the listener is freed via + * kfree_rcu(), so it's safe to dereference it + * under rcu_read_lock(), even if sc->accept.listener + * is cleared and the listener is released concurrently. + */ + rcu_read_lock(); + lsc = READ_ONCE(sc->accept.listener); + if (!lsc) { + rcu_read_unlock(); + return; + } + + spin_lock_irqsave(&lsc->listen.lock, flags); + if (sc->accept.listener == lsc) { + list_move_tail(&sc->accept.list, &lsc->listen.orphaned); + queue_work(lsc->workqueues.cleanup, &lsc->listen.purge_orphaned_work); + } + spin_unlock_irqrestore(&lsc->listen.lock, flags); + rcu_read_unlock(); +} + +static void smbdirect_listen_purge_orphaned_work(struct work_struct *work) +{ + struct smbdirect_socket *lsc = + container_of(work, struct smbdirect_socket, listen.purge_orphaned_work); + struct smbdirect_socket *psc, *tsc; + LIST_HEAD(orphaned_list); + unsigned long flags; + + /* + * Clearing accept.listener under listen.lock + * makes us responsible for releasing them. + */ + spin_lock_irqsave(&lsc->listen.lock, flags); + list_splice_tail_init(&lsc->listen.orphaned, &orphaned_list); + list_for_each_entry(psc, &orphaned_list, accept.list) + WRITE_ONCE(psc->accept.listener, NULL); + spin_unlock_irqrestore(&lsc->listen.lock, flags); + + /* + * We don't hold the listener's rdma_lock_handler() + * lock here, see smbdirect_socket_destroy() + * for why that's important. + */ + list_for_each_entry_safe(psc, tsc, &orphaned_list, accept.list) { + list_del_init(&psc->accept.list); + smbdirect_socket_release(psc); + } +} + int smbdirect_socket_listen(struct smbdirect_socket *sc, int backlog) { int ret; @@ -47,6 +119,9 @@ int smbdirect_socket_listen(struct smbdirect_socket *sc, int backlog) sc->rdma.cm_id->event_handler = smbdirect_listen_rdma_event_handler; rdma_unlock_handler(sc->rdma.cm_id); + INIT_WORK(&sc->listen.purge_orphaned_work, + smbdirect_listen_purge_orphaned_work); + ret = rdma_listen(sc->rdma.cm_id, backlog); if (ret) { sc->first_error = ret; @@ -214,6 +289,7 @@ static int smbdirect_listen_connect_request(struct smbdirect_socket *lsc, size_t backlog = max_t(size_t, 1, lsc->listen.backlog); size_t psockets; size_t rsockets; + size_t osockets; int ret; if (!smbdirect_frwr_is_supported(&new_id->device->attrs)) { @@ -254,14 +330,21 @@ static int smbdirect_listen_connect_request(struct smbdirect_socket *lsc, spin_lock_irqsave(&lsc->listen.lock, flags); psockets = list_count_nodes(&lsc->listen.pending); rsockets = list_count_nodes(&lsc->listen.ready); + /* + * Orphaned sockets still count against + * the backlog until they are released by + * smbdirect_listen_purge_orphaned_work(). + */ + osockets = list_count_nodes(&lsc->listen.orphaned); spin_unlock_irqrestore(&lsc->listen.lock, flags); if (psockets > backlog || rsockets > backlog || - (psockets + rsockets) > backlog) { + osockets > backlog || + (psockets + rsockets + osockets) > backlog) { smbdirect_log_rdma_event(lsc, SMBDIRECT_LOG_ERR, - "Backlog[%d][%zu] full pending[%zu] ready[%zu]\n", - lsc->listen.backlog, backlog, psockets, rsockets); + "Backlog[%d][%zu] full pending[%zu] ready[%zu] orphaned[%zu]\n", + lsc->listen.backlog, backlog, psockets, rsockets, osockets); return -EBUSY; } @@ -279,22 +362,54 @@ static int smbdirect_listen_connect_request(struct smbdirect_socket *lsc, if (ret) goto set_settings_failed; + /* + * Publish nsc on the pending list with accept.listener set before + * smbdirect_accept_connect_request() calls rdma_accept(). Once the + * connection can establish, smbdirect_accept_negotiate_recv_work() + * may run and it must observe accept.listener, otherwise nsc would + * be left stranded on the pending list forever. + * + * The listener's handler_mutex is held while we're called, so + * smbdirect_socket_destroy() of the listener can't reach nsc on the + * pending list before we're done. + * + * From here nsc is published on the pending list: on failure of + * smbdirect_accept_connect_request() below we schedule nsc's teardown, + * which orphans nsc off the listener so + * smbdirect_listen_purge_orphaned_work() -> smbdirect_socket_release() + * releases it. + */ spin_lock_irqsave(&lsc->listen.lock, flags); list_add_tail(&nsc->accept.list, &lsc->listen.pending); - nsc->accept.listener = lsc; + WRITE_ONCE(nsc->accept.listener, lsc); spin_unlock_irqrestore(&lsc->listen.lock, flags); ret = smbdirect_accept_connect_request(nsc, &event->param.conn); - if (ret) - goto accept_connect_failed; + if (ret) { + /* + * The rdma_cm core holds both the listener's and nsc's + * id_priv->handler_mutex across this CONNECT_REQUEST handler, + * so neither can go away under us here and nsc cannot receive + * any rdma event: it is safe to hand nsc to its teardown. + * + * That teardown must be deferred though: nsc->rdma.cm_id + * (= new_id) has its handler_mutex held by us, so it cannot be + * destroyed synchronously from here. + * smbdirect_socket_schedule_cleanup() only queues work; + * smbdirect_socket_cleanup_work() then orphans nsc off the + * listener and smbdirect_listen_purge_orphaned_work() releases + * it, destroying nsc's cm_id with rdma_destroy_id(). + */ + smbdirect_socket_schedule_cleanup(nsc, ret); + } + /* + * Always return 0 so the rdma_cm core keeps new_id: nsc owns it now. + * On success it stays connected; on failure its deferred teardown + * above destroys it. + */ return 0; -accept_connect_failed: - spin_lock_irqsave(&lsc->listen.lock, flags); - list_del_init(&nsc->accept.list); - nsc->accept.listener = NULL; - spin_unlock_irqrestore(&lsc->listen.lock, flags); set_settings_failed: set_params_failed: /* diff --git a/fs/smb/smbdirect/rw.c b/fs/smb/smbdirect/rw.c index 6fe38042cfb9..cb142c3c50b2 100644 --- a/fs/smb/smbdirect/rw.c +++ b/fs/smb/smbdirect/rw.c @@ -143,7 +143,6 @@ int smbdirect_connection_rdma_xmit(struct smbdirect_socket *sc, if (desc_buf_len > buf_len) { desc_buf_len = buf_len; desc[i].length = cpu_to_le32(desc_buf_len); - buf_len = 0; } credits_needed += smbdirect_connection_calc_rw_credits(sc, diff --git a/fs/smb/smbdirect/socket.c b/fs/smb/smbdirect/socket.c index bb02df6158b9..e270e36b3145 100644 --- a/fs/smb/smbdirect/socket.c +++ b/fs/smb/smbdirect/socket.c @@ -319,6 +319,15 @@ void __smbdirect_socket_schedule_cleanup(struct smbdirect_socket *sc, * (smbdirect_socket_destroy) to reap. */ if (sc->listen.backlog != -1) { /* was a listener */ + /* + * We don't move them to the orphaned list here, + * each child does that itself at the end of its + * smbdirect_socket_cleanup_work(), see + * smbdirect_listen_orphan_socket(). Only that checks + * accept.listener, which is still NULL for a child + * that smbdirect_listen_connect_request() is still + * setting up and will release itself on failure. + */ spin_lock_irqsave(&sc->listen.lock, flags); list_splice_init(&sc->listen.ready, &sc->listen.pending); list_for_each_entry_safe(psc, tsc, &sc->listen.pending, accept.list) @@ -427,6 +436,15 @@ static void smbdirect_socket_cleanup_work(struct work_struct *work) * instances of one class -- harmless, but lockdep cannot tell). */ if (sc->listen.backlog != -1) { /* was a listener */ + /* + * We don't move them to the orphaned list here, + * each child does that itself at the end of its + * smbdirect_socket_cleanup_work(), see + * smbdirect_listen_orphan_socket(). Only that checks + * accept.listener, which is still NULL for a child + * that smbdirect_listen_connect_request() is still + * setting up and will release itself on failure. + */ spin_lock_irqsave(&sc->listen.lock, flags); list_splice_init(&sc->listen.ready, &sc->listen.pending); list_for_each_entry_safe(psc, tsc, &sc->listen.pending, accept.list) @@ -486,6 +504,18 @@ static void smbdirect_socket_cleanup_work(struct work_struct *work) * in order to notice the broken connection. */ smbdirect_socket_wake_up_all(sc); + + /* + * If we're still on the pending or ready list + * of a listener, we started the disconnect + * as far as possible above, so we move ourself + * to the orphaned list of the listener, + * which will release us. + * + * This is a no-op internally if + * sc->accept.listener is NULL. + */ + smbdirect_listen_orphan_socket(sc); } static void smbdirect_socket_destroy(struct smbdirect_socket *sc) @@ -511,7 +541,13 @@ static void smbdirect_socket_destroy(struct smbdirect_socket *sc) if (sc->status == SMBDIRECT_SOCKET_DESTROYED) return; - WARN_ONCE(sc->status != SMBDIRECT_SOCKET_DISCONNECTED, + /* + * smbdirect_socket_destroy_sync() doesn't wait + * for RDMA_CM_EVENT_DISCONNECTED, so we may still + * be in SMBDIRECT_SOCKET_DISCONNECTING + * (or already reached SMBDIRECT_SOCKET_DISCONNECTED) + */ + WARN_ONCE(sc->status < SMBDIRECT_SOCKET_DISCONNECTING, "status=%s first_error=%1pe", smbdirect_socket_status_string(sc->status), SMBDIRECT_DEBUG_ERR_PTR(sc->first_error)); @@ -538,10 +574,37 @@ static void smbdirect_socket_destroy(struct smbdirect_socket *sc) disable_work_sync(&sc->recv_io.posted.refill_work); disable_work_sync(&sc->idle.immediate_work); disable_delayed_work_sync(&sc->idle.timer_work); + disable_work_sync(&sc->listen.purge_orphaned_work); if (sc->rdma.cm_id) rdma_lock_handler(sc->rdma.cm_id); + /* + * We hold the handler lock, so the rdma event + * handlers can't change the status anymore. + * + * If RDMA_CM_EVENT_DISCONNECTED didn't arrive yet, + * we just stop waiting for it here. + * + * We already disabled disconnect_work above + * and before we call rdma_unlock_handler below + * we call smbdirect_connection_destroy_qp which + * sets sc->ib.qp = NULL. + * + * Between rdma_unlock_handler() and + * rdma_destroy_id() below there's a small + * windows where RDMA_CM_EVENT_DISCONNECTED + * could still arrive. + * + * But smbdirect_connection_rdma_event_handler + * will be a noop when calling smbdirect_socket_schedule_cleanup* + * ib_drain_qp() also won't be called. + */ + if (sc->status < SMBDIRECT_SOCKET_DISCONNECTED) { + sc->status = SMBDIRECT_SOCKET_DISCONNECTED; + smbdirect_socket_wake_up_all(sc); + } + if (sc->ib.qp) { smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_INFO, "drain qp\n"); @@ -575,6 +638,19 @@ static void smbdirect_socket_destroy(struct smbdirect_socket *sc) spin_lock_irqsave(&sc->listen.lock, flags); list_splice_tail_init(&sc->listen.ready, &pending_list); list_splice_tail_init(&sc->listen.pending, &pending_list); + /* + * purge_orphaned_work is already disabled above, + * so we also need to release the orphaned sockets. + * + * Clearing accept.listener under listen.lock + * makes us responsible for releasing them and + * prevents them from moving themselves to + * the orphaned list via + * smbdirect_listen_orphan_socket(). + */ + list_splice_tail_init(&sc->listen.orphaned, &pending_list); + list_for_each_entry(psc, &pending_list, accept.list) + WRITE_ONCE(psc->accept.listener, NULL); spin_unlock_irqrestore(&sc->listen.lock, flags); /* It's not possible for upper layer to get to reassembly */ @@ -618,7 +694,6 @@ static void smbdirect_socket_destroy(struct smbdirect_socket *sc) "release %zu pending sockets\n", psockets); list_for_each_entry_safe(psc, tsc, &pending_list, accept.list) { list_del_init(&psc->accept.list); - psc->accept.listener = NULL; smbdirect_socket_release(psc); } if (sc->listen.backlog != -1) /* was a listener */ @@ -679,17 +754,21 @@ void smbdirect_socket_destroy_sync(struct smbdirect_socket *sc) "destroying rdma session\n"); if (sc->status < SMBDIRECT_SOCKET_DISCONNECTING) smbdirect_socket_cleanup_work(&sc->disconnect_work); - if (sc->status < SMBDIRECT_SOCKET_DISCONNECTED) { - smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_INFO, - "wait for transport being disconnected\n"); - wait_event(sc->status_wait, sc->status == SMBDIRECT_SOCKET_DISCONNECTED); - smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_INFO, - "waited for transport being disconnected\n"); - } /* - * Once we reached SMBDIRECT_SOCKET_DISCONNECTED, - * we should call smbdirect_socket_destroy() + * We don't wait for RDMA_CM_EVENT_DISCONNECTED, + * rdma_disconnect() was already called by + * smbdirect_socket_cleanup_work() if needed + * and smbdirect_socket_destroy() drains the qp, + * destroys it and calls rdma_destroy_id(), which + * only waits for a currently running event handler + * and makes sure no further events are delivered. + * The rest of the disconnect protocol is handled + * by the rdma core asynchronously. + * + * Waiting for RDMA_CM_EVENT_DISCONNECTED could + * take very long or forever, e.g. if the peer + * just disappeared. */ smbdirect_socket_destroy(sc); smbdirect_log_rdma_event(sc, SMBDIRECT_LOG_INFO, @@ -741,7 +820,17 @@ static void smbdirect_socket_release_destroy(struct kref *kref) * in DESTROYED state, before we free the memory. */ smbdirect_socket_destroy_sync(sc); - kfree(sc); + + /* + * Only a listener (backlog != -1) is ever dereferenced + * via sc->accept.listener under rcu_read_lock(), see + * smbdirect_listen_orphan_socket(). Other sockets can be + * freed immediately. + */ + if (sc->listen.backlog != -1) /* was a listener */ + kfree_rcu(sc, refs.rcu); + else + kfree(sc); } void smbdirect_socket_release(struct smbdirect_socket *sc) diff --git a/fs/smb/smbdirect/socket.h b/fs/smb/smbdirect/socket.h index c09eddd8ad16..14114a2567a2 100644 --- a/fs/smb/smbdirect/socket.h +++ b/fs/smb/smbdirect/socket.h @@ -151,6 +151,12 @@ struct smbdirect_socket { * the disconnect refcount. */ struct kref destroy; + + /* + * smbdirect_socket_release_destroy() uses + * kfree_rcu(), see accept.listener. + */ + struct rcu_head rcu; } refs; /* RDMA related */ @@ -216,6 +222,16 @@ struct smbdirect_socket { * only be > 0. */ int backlog; + /* + * Sockets on pending or ready that failed + * move themselves to orphaned and queue + * purge_orphaned_work, which releases them. + * So that they are freed while the listener + * is still alive. They still count against + * the backlog until they are released. + */ + struct list_head orphaned; + struct work_struct purge_orphaned_work; } listen; /* @@ -226,7 +242,30 @@ struct smbdirect_socket { * connection. */ struct { + /* + * This is only set, protected by + * listener->listen.lock, while the socket + * is owned by the listener (on its pending, + * ready or orphaned list). The one who clears + * it is responsible for releasing the socket. + * + * The memory of a struct smbdirect_socket is + * freed via kfree_rcu(), so the listener + * can be dereferenced under rcu_read_lock(), + * even if accept.listener is cleared and + * the listener is released concurrently. + */ struct smbdirect_socket *listener; + /* + * Linkage on one of the listener's + * listen.{pending,ready,orphaned} lists. + * + * accept.list and accept.listener are set + * together under listener->listen.lock before + * smbdirect_accept_connect_request() is called, + * so whenever the socket is on one of those lists + * accept.listener is that listener (never NULL). + */ struct list_head list; } accept; @@ -588,6 +627,9 @@ static __always_inline void smbdirect_socket_init(struct smbdirect_socket *sc) spin_lock_init(&sc->listen.lock); INIT_LIST_HEAD(&sc->listen.pending); INIT_LIST_HEAD(&sc->listen.ready); + INIT_LIST_HEAD(&sc->listen.orphaned); + INIT_WORK(&sc->listen.purge_orphaned_work, __smbdirect_socket_disabled_work); + disable_work_sync(&sc->listen.purge_orphaned_work); sc->listen.backlog = -1; /* not a listener */ init_waitqueue_head(&sc->listen.wait_queue); diff --git a/fs/udf/inode.c b/fs/udf/inode.c index e45e546a739a..2f04c4f0e1b8 100644 --- a/fs/udf/inode.c +++ b/fs/udf/inode.c @@ -372,8 +372,8 @@ static int udf_map_block(struct inode *inode, struct udf_map_rq *map) map->pblk = udf_get_lb_pblock(inode->i_sb, &eloc, offset); map->oflags |= UDF_BLK_MAPPED; - ret = 0; } + ret = 0; out_read: up_read(&iinfo->i_data_sem); brelse(epos.bh); diff --git a/fs/ufs/dir.c b/fs/ufs/dir.c index ce43cf20b07c..e96174b738b4 100644 --- a/fs/ufs/dir.c +++ b/fs/ufs/dir.c @@ -213,7 +213,7 @@ static void *ufs_get_folio(struct inode *dir, unsigned long n, static unsigned ufs_last_byte(struct inode *inode, unsigned long page_nr) { - unsigned last_byte = inode->i_size; + u64 last_byte = inode->i_size; last_byte -= page_nr << PAGE_SHIFT; if (last_byte > PAGE_SIZE) diff --git a/fs/verity/enable.c b/fs/verity/enable.c index 42dfed1ce0ce..64fccb526ccd 100644 --- a/fs/verity/enable.c +++ b/fs/verity/enable.c @@ -274,8 +274,8 @@ static int enable_verity(struct file *filp, * initialization so that we don't have to undo the enabling when memory * allocation for the hash table fails. This is safe because looking up * the fsverity_info always first checks the S_VERITY flag on the inode, - * which will only be set at the very end of the ->end_enable_verity - * method. + * which will only be set at the very end of a successful call to + * ->end_enable_verity(). */ err = fsverity_set_info(vi); if (err) { @@ -285,10 +285,7 @@ static int enable_verity(struct file *filp, /* * Tell the filesystem to finish enabling verity on the file. - * Serialized with ->begin_enable_verity() by the inode lock. The file - * system needs to set the S_VERITY flag on the inode at the very end of - * the method, at which point the fsverity information can be accessed - * by other threads. + * Serialized with ->begin_enable_verity() by the inode lock. */ inode_lock(inode); err = vops->end_enable_verity(filp, desc, desc_size, params.tree_size); @@ -296,7 +293,8 @@ static int enable_verity(struct file *filp, if (err) { fsverity_err(inode, "%ps() failed with err %d", vops->end_enable_verity, err); - fsverity_remove_info(vi); + if (!WARN_ON_ONCE(IS_VERITY(inode))) + fsverity_remove_info(vi); } else if (WARN_ON_ONCE(!IS_VERITY(inode))) { fsverity_remove_info(vi); err = -EINVAL; diff --git a/fs/xfs/libxfs/xfs_attr_leaf.c b/fs/xfs/libxfs/xfs_attr_leaf.c index 2c80f4fd0b78..02ca50442b46 100644 --- a/fs/xfs/libxfs/xfs_attr_leaf.c +++ b/fs/xfs/libxfs/xfs_attr_leaf.c @@ -353,6 +353,8 @@ xfs_attr3_leaf_verify_entry( if (!(ent->flags & XFS_ATTR_INCOMPLETE) && rentry->valueblk == 0) return __this_address; + if (be32_to_cpu(rentry->valuelen) > XFS_XATTR_SIZE_MAX) + return __this_address; } if (name_end > buf_end) @@ -642,6 +644,17 @@ xfs_attr_copy_value( unsigned char *value, int valuelen) { + /* + * A value length that is negative or larger than the maximum xattr + * size is on-disk corruption. The remote value length is an on-disk + * __be32 stored into the signed args->rmtvaluelen, so a crafted value + * such as 0x80000000 becomes negative and would slip past the + * "args->valuelen < valuelen" check below and be used as a copy + * length. Reject it before that can happen. + */ + if (valuelen < 0 || valuelen > XFS_XATTR_SIZE_MAX) + return -EFSCORRUPTED; + /* * Parent pointer lookups require the caller to specify the name and * value, so don't copy anything. diff --git a/fs/xfs/libxfs/xfs_da_btree.c b/fs/xfs/libxfs/xfs_da_btree.c index 3d02a0d7ba44..faf88488265c 100644 --- a/fs/xfs/libxfs/xfs_da_btree.c +++ b/fs/xfs/libxfs/xfs_da_btree.c @@ -1787,6 +1787,20 @@ xfs_da3_node_lookup_int( } else expected_level--; + /* + * The node verifier cannot tell whether this block belongs to + * the directory or the attribute tree, so it only bounds the + * entry count against the larger of the two geometries. Here + * args->geo is the geometry of the fork we are actually + * searching, so reject a count that would walk btree[] off the + * end of this node buffer. + */ + if (nodehdr.count > args->geo->node_ents) { + xfs_buf_mark_corrupt(blk->bp); + xfs_da_mark_sick(args); + return -EFSCORRUPTED; + } + max = nodehdr.count; blk->hashval = be32_to_cpu(btree[max - 1].hashval); diff --git a/fs/xfs/libxfs/xfs_inode_fork.c b/fs/xfs/libxfs/xfs_inode_fork.c index 606a36526ce2..486fe7ab8b61 100644 --- a/fs/xfs/libxfs/xfs_inode_fork.c +++ b/fs/xfs/libxfs/xfs_inode_fork.c @@ -683,9 +683,7 @@ xfs_ifork_verify_local_data( break; } case S_IFLNK: { - struct xfs_ifork *ifp = xfs_ifork_ptr(ip, XFS_DATA_FORK); - - fa = xfs_symlink_shortform_verify(ifp->if_data, ifp->if_bytes); + fa = xfs_symlink_shortform_verify(ip); break; } default: diff --git a/fs/xfs/libxfs/xfs_symlink_remote.c b/fs/xfs/libxfs/xfs_symlink_remote.c index b0dc3888bf1b..0201a3d59b1a 100644 --- a/fs/xfs/libxfs/xfs_symlink_remote.c +++ b/fs/xfs/libxfs/xfs_symlink_remote.c @@ -208,11 +208,15 @@ xfs_symlink_local_to_remote( */ xfs_failaddr_t xfs_symlink_shortform_verify( - void *sfp, - int64_t size) + struct xfs_inode *ip) { + struct xfs_ifork *ifp = xfs_ifork_ptr(ip, XFS_DATA_FORK); + char *sfp = (char *)ifp->if_data; + int size = ifp->if_bytes; char *endp = sfp + size; + ASSERT(ifp->if_format == XFS_DINODE_FMT_LOCAL); + /* * Zero length symlinks should never occur in memory as they are * never allowed to exist on disk. diff --git a/fs/xfs/libxfs/xfs_symlink_remote.h b/fs/xfs/libxfs/xfs_symlink_remote.h index c1672fe1f17b..3f6590602473 100644 --- a/fs/xfs/libxfs/xfs_symlink_remote.h +++ b/fs/xfs/libxfs/xfs_symlink_remote.h @@ -18,7 +18,7 @@ bool xfs_symlink_hdr_ok(xfs_ino_t ino, uint32_t offset, void xfs_symlink_local_to_remote(struct xfs_trans *tp, struct xfs_buf *bp, struct xfs_inode *ip, struct xfs_ifork *ifp, void *priv); -xfs_failaddr_t xfs_symlink_shortform_verify(void *sfp, int64_t size); +xfs_failaddr_t xfs_symlink_shortform_verify(struct xfs_inode *ip); int xfs_symlink_remote_read(struct xfs_inode *ip, char *link); int xfs_symlink_write_target(struct xfs_trans *tp, struct xfs_inode *ip, xfs_ino_t owner, const char *target_path, int pathlen, diff --git a/fs/xfs/libxfs/xfs_trans_space.c b/fs/xfs/libxfs/xfs_trans_space.c index c4cd547033e5..7edd0d86f0bd 100644 --- a/fs/xfs/libxfs/xfs_trans_space.c +++ b/fs/xfs/libxfs/xfs_trans_space.c @@ -127,10 +127,10 @@ xfs_rename_space_res( if (has_whiteout) ret += xfs_parent_calc_space_res(mp, src_namelen); ret += 2 * xfs_parent_calc_space_res(mp, target_namelen); - } - if (target_exists) - ret += xfs_parent_calc_space_res(mp, target_namelen); + if (target_exists) + ret += xfs_parent_calc_space_res(mp, target_namelen); + } return ret; } diff --git a/fs/xfs/scrub/agheader_repair.c b/fs/xfs/scrub/agheader_repair.c index ff1b4b361cf2..07e907051e87 100644 --- a/fs/xfs/scrub/agheader_repair.c +++ b/fs/xfs/scrub/agheader_repair.c @@ -1081,12 +1081,9 @@ xrep_iunlink_reload_next( if (VFS_I(ip)->i_nlink != 0) { error = xrep_iunlink_store_next(ragi, agino, NULLAGINO); if (error) - return error; + goto rele; error = xrep_iunlink_store_prev(ragi, agino, LINKED_AGINO); - if (error) - return error; - goto rele; } @@ -1100,7 +1097,7 @@ xrep_iunlink_reload_next( */ rele: xchk_irele(sc, ip); - return 0; + return error; } /* @@ -1325,8 +1322,12 @@ xrep_iunlink_mark_ondisk_rec( */ error = xchk_iget(ragi->sc, xfs_agino_to_ino(sc->sa.pag, agino), &ip); - if (error) + if (error == -ENOMEM) + return error; + if (error) { + error = 0; continue; + } trace_xrep_iunlink_reload_ondisk(ip); diff --git a/fs/xfs/scrub/alloc_repair.c b/fs/xfs/scrub/alloc_repair.c index 2398e3819597..4401281cf6f1 100644 --- a/fs/xfs/scrub/alloc_repair.c +++ b/fs/xfs/scrub/alloc_repair.c @@ -227,7 +227,10 @@ xrep_abt_walk_rmap( void *priv) { struct xrep_abt *ra = priv; - int error; + int error = 0; + + if (xchk_should_terminate(ra->sc, &error)) + return error; /* Record all the OWN_AG blocks... */ if (rec->rm_owner == XFS_RMAP_OWN_AG) { @@ -660,15 +663,11 @@ xrep_abt_reset_counters( /* * Compute the contribution to agf_btreeblks for the new free space - * btrees. This is the computed btree size minus anything we didn't - * use. + * btrees. This is the computed btree size minus the root block. */ freesp_btreeblks += ra->new_bnobt.bload.nr_blocks - 1; freesp_btreeblks += ra->new_cntbt.bload.nr_blocks - 1; - freesp_btreeblks -= xrep_newbt_unused_blocks(&ra->new_bnobt); - freesp_btreeblks -= xrep_newbt_unused_blocks(&ra->new_cntbt); - /* * The AGF header contains extra information related to the free space * btrees, so we must update those fields here. diff --git a/fs/xfs/scrub/attr.c b/fs/xfs/scrub/attr.c index d5226939c810..842d65834afb 100644 --- a/fs/xfs/scrub/attr.c +++ b/fs/xfs/scrub/attr.c @@ -216,6 +216,12 @@ xchk_xattr_actor( return -ECANCELED; } + /* Don't even try to handle a bizarrely large xattr. */ + if (valuelen > XATTR_SIZE_MAX) { + xchk_fblock_set_corrupt(sc, XFS_ATTR_FORK, args.blkno); + return -ECANCELED; + } + /* * Try to allocate enough memory to extract the attr value. If that * doesn't work, return -EDEADLOCK as a signal to try again with a @@ -590,12 +596,17 @@ xchk_xattr_check_sf( } for (i = 0; i < sf->count; i++) { - unsigned char *name = sfe->nameval; - unsigned char *value = &sfe->nameval[sfe->namelen]; + unsigned char *name; + unsigned char *value; if (xchk_should_terminate(sc, &error)) return error; + if ((unsigned char *)(sfe + 1) >= end) { + xchk_fblock_set_corrupt(sc, XFS_ATTR_FORK, 0); + break; + } + next = xfs_attr_sf_nextentry(sfe); if ((unsigned char *)next > end) { xchk_fblock_set_corrupt(sc, XFS_ATTR_FORK, 0); @@ -618,6 +629,7 @@ xchk_xattr_check_sf( break; } + name = sfe->nameval; if (!xchk_xattr_set_map(sc, ab->usedmap, (char *)name - (char *)sf, sfe->namelen)) { @@ -625,6 +637,7 @@ xchk_xattr_check_sf( break; } + value = &sfe->nameval[sfe->namelen]; if (!xchk_xattr_set_map(sc, ab->usedmap, (char *)value - (char *)sf, sfe->valuelen)) { diff --git a/fs/xfs/scrub/attr_repair.c b/fs/xfs/scrub/attr_repair.c index 28f92e9ba72b..c7695d4cc8e7 100644 --- a/fs/xfs/scrub/attr_repair.c +++ b/fs/xfs/scrub/attr_repair.c @@ -477,6 +477,9 @@ xrep_xattr_recover_sf( if (xchk_should_terminate(sc, &error)) return error; + if ((unsigned char *)(sfe + 1) >= end) + break; + next = xfs_attr_sf_nextentry(sfe); if ((unsigned char *)next > end) break; @@ -837,11 +840,11 @@ xrep_xattr_full_reset( ASSERT(ifp->if_bytes == 0); ifp->if_format = XFS_DINODE_FMT_LOCAL; - xfs_idata_realloc(sc->tempip, sizeof(*hdr), XFS_ATTR_FORK); } /* Reinitialize the attr fork to an empty shortform structure. */ - hdr = ifp->if_data; + hdr = xfs_idata_realloc(sc->tempip, + (int64_t)sizeof(*hdr) - ifp->if_bytes, XFS_ATTR_FORK); memset(hdr, 0, sizeof(*hdr)); hdr->totsize = cpu_to_be16(sizeof(*hdr)); xfs_trans_log_inode(sc->tp, sc->tempip, XFS_ILOG_CORE | XFS_ILOG_ADATA); @@ -967,7 +970,7 @@ xrep_xattr_fork_remove( ifp->if_format = XFS_DINODE_FMT_LOCAL; hdr = xfs_idata_realloc(ip, (int)sizeof(*hdr) - ifp->if_bytes, XFS_ATTR_FORK); - hdr->count = 0; + memset(hdr, 0, sizeof(*hdr)); hdr->totsize = cpu_to_be16(sizeof(*hdr)); xfs_trans_log_inode(sc->tp, ip, XFS_ILOG_CORE | XFS_ILOG_ADATA); @@ -1474,6 +1477,9 @@ xrep_xattr_rebuild_tree( if (error) return error; + if (rx->live_update_aborted) + return -EIO; + /* * Exchange the blocks mapped by the tempfile's attr fork with the file * being repaired. The old attr blocks will then be attached to the @@ -1485,7 +1491,7 @@ xrep_xattr_rebuild_tree( error = xrep_xattr_reset_tempfile_fork(sc); if (error) - return error; + goto forget_acls; /* * Roll to get a transaction without any inodes joined to it. Then we @@ -1494,7 +1500,7 @@ xrep_xattr_rebuild_tree( */ error = xfs_trans_roll(&sc->tp); if (error) - return error; + goto forget_acls; xrep_tempfile_iunlock(sc); xrep_tempfile_iounlock(sc); @@ -1503,7 +1509,7 @@ xrep_xattr_rebuild_tree( /* Invalidate cached ACLs now that we've reloaded all the xattrs. */ xfs_forget_acl(VFS_I(sc->ip), SGI_ACL_FILE); xfs_forget_acl(VFS_I(sc->ip), SGI_ACL_DEFAULT); - return 0; + return error; } /* Tear down all the incore scan stuff we created. */ diff --git a/fs/xfs/scrub/bitmap.c b/fs/xfs/scrub/bitmap.c index 08f216d26ec6..1da99e179baa 100644 --- a/fs/xfs/scrub/bitmap.c +++ b/fs/xfs/scrub/bitmap.c @@ -77,6 +77,9 @@ xbitmap64_clear( struct xbitmap64_node *new_bn; uint64_t last = start + len - 1; + if (!len) + return 0; + while ((bn = xbitmap64_tree_iter_first(&bitmap->xb_root, start, last))) { if (bn->bn_start < start && bn->bn_last > last) { uint64_t old_last = bn->bn_last; @@ -127,6 +130,9 @@ xbitmap64_set( uint64_t last = start + len - 1; int error; + if (!len) + return 0; + /* Is this whole range already set? */ left = xbitmap64_tree_iter_first(&bitmap->xb_root, start, last); if (left && left->bn_start <= start && left->bn_last >= last) @@ -223,6 +229,8 @@ xbitmap64_disunion( struct xbitmap64_node *bn; int error; + ASSERT(bitmap != sub); + if (xbitmap64_empty(bitmap) || xbitmap64_empty(sub)) return 0; @@ -287,6 +295,8 @@ xbitmap64_test( struct xbitmap64_node *bn; uint64_t last = start + *len - 1; + ASSERT(*len > 0); + bn = xbitmap64_tree_iter_first(&bitmap->xb_root, start, last); if (!bn) return false; @@ -357,6 +367,9 @@ xbitmap32_clear( struct xbitmap32_node *new_bn; uint32_t last = start + len - 1; + if (!len) + return 0; + while ((bn = xbitmap32_tree_iter_first(&bitmap->xb_root, start, last))) { if (bn->bn_start < start && bn->bn_last > last) { uint32_t old_last = bn->bn_last; @@ -407,6 +420,9 @@ xbitmap32_set( uint32_t last = start + len - 1; int error; + if (!len) + return 0; + /* Is this whole range already set? */ left = xbitmap32_tree_iter_first(&bitmap->xb_root, start, last); if (left && left->bn_start <= start && left->bn_last >= last) @@ -503,6 +519,8 @@ xbitmap32_disunion( struct xbitmap32_node *bn; int error; + ASSERT(bitmap != sub); + if (xbitmap32_empty(bitmap) || xbitmap32_empty(sub)) return 0; @@ -567,6 +585,8 @@ xbitmap32_test( struct xbitmap32_node *bn; uint32_t last = start + *len - 1; + ASSERT(*len > 0); + bn = xbitmap32_tree_iter_first(&bitmap->xb_root, start, last); if (!bn) return false; diff --git a/fs/xfs/scrub/bmap.c b/fs/xfs/scrub/bmap.c index 4f3c7f681bd9..428be9ecdcd4 100644 --- a/fs/xfs/scrub/bmap.c +++ b/fs/xfs/scrub/bmap.c @@ -626,6 +626,10 @@ xchk_bmap_check_rmap( struct xfs_ifork *ifp; struct xfs_scrub *sc = sbcri->sc; bool have_map; + int error = 0; + + if (xchk_should_terminate(sc, &error)) + return error; /* Is this even the right fork? */ if (rec->rm_owner != I_INO(sc->ip)) @@ -657,6 +661,9 @@ xchk_bmap_check_rmap( */ check_rec = *rec; while (have_map) { + if (xchk_should_terminate(sc, &error)) + return error; + if (irec.br_startoff != check_rec.rm_offset) xchk_fblock_set_corrupt(sc, sbcri->whichfork, check_rec.rm_offset); diff --git a/fs/xfs/scrub/btree.c b/fs/xfs/scrub/btree.c index 786b1e7c4dc3..43a7ea7308ee 100644 --- a/fs/xfs/scrub/btree.c +++ b/fs/xfs/scrub/btree.c @@ -46,8 +46,14 @@ __xchk_btree_process_error( case -ENODATA: /* Note the badness but don't abort. */ sc->sm->sm_flags |= errflag; + if (cur->bc_ops->type == XFS_BTREE_TYPE_INODE) + trace_xchk_ifork_btree_op_error(sc, cur, level, + *error, ret_ip); + else + trace_xchk_btree_op_error(sc, cur, level, + *error, ret_ip); *error = 0; - fallthrough; + break; default: if (cur->bc_ops->type == XFS_BTREE_TYPE_INODE) trace_xchk_ifork_btree_op_error(sc, cur, level, @@ -602,6 +608,14 @@ xchk_btree_get_block( xfs_btree_get_block(bs->cur, level, pbp); if (__xfs_btree_check_block(bs->cur, *pblock, level, *pbp)) { xchk_btree_set_corrupt(bs->sc, bs->cur, level); + /* + * The btree block fails basic checks, so we know it's corrupt. + * There's no need to continue looking for corruptions, so we + * null out pblock. If *pbp is non-null then it's also + * attached to the transaction so we won't leak it despite the + * 0 return code. + */ + *pblock = NULL; return 0; } if (*pbp) diff --git a/fs/xfs/scrub/common.c b/fs/xfs/scrub/common.c index e5233e31abb7..a61895872072 100644 --- a/fs/xfs/scrub/common.c +++ b/fs/xfs/scrub/common.c @@ -107,8 +107,9 @@ __xchk_process_error( case -ENODATA: /* Note the badness but don't abort. */ sc->sm->sm_flags |= errflag; + trace_xchk_op_error(sc, agno, bno, *error, ret_ip); *error = 0; - fallthrough; + break; default: trace_xchk_op_error(sc, agno, bno, *error, ret_ip); break; @@ -183,8 +184,10 @@ __xchk_fblock_process_error( case -ENODATA: /* Note the badness but don't abort. */ sc->sm->sm_flags |= errflag; + trace_xchk_file_op_error(sc, whichfork, offset, *error, + ret_ip); *error = 0; - fallthrough; + break; default: trace_xchk_file_op_error(sc, whichfork, offset, *error, ret_ip); diff --git a/fs/xfs/scrub/cow_repair.c b/fs/xfs/scrub/cow_repair.c index 8dd9c0266e21..e49d9396edc5 100644 --- a/fs/xfs/scrub/cow_repair.c +++ b/fs/xfs/scrub/cow_repair.c @@ -200,29 +200,35 @@ xrep_cow_mark_missing_staging_rmap( void *priv) { struct xrep_cow *xc = priv; - xfs_agblock_t rec_bno; - xfs_extlen_t rec_len; - unsigned int adj; - - if (rec->rm_owner == XFS_RMAP_OWN_COW) - return 0; + const xfs_agblock_t rec_end = + rec->rm_startblock + rec->rm_blockcount; + int error = 0; - rec_bno = rec->rm_startblock; - rec_len = rec->rm_blockcount; - if (rec_bno < xc->irec_startbno) { - adj = xc->irec_startbno - rec_bno; - rec_len -= adj; - rec_bno += adj; + /* Was there a gap since the last rmap? */ + if (xc->next_bno < rec->rm_startblock) { + error = xrep_cow_mark_file_range(xc, + xfs_gbno_to_fsb(cur->bc_group, xc->next_bno), + rec->rm_startblock - xc->next_bno); + if (error) + return error; } - if (rec_bno + rec_len > xc->irec_startbno + xc->irec.br_blockcount) { - adj = (rec_bno + rec_len) - - (xc->irec_startbno + xc->irec.br_blockcount); - rec_len -= adj; + if (rec->rm_owner != XFS_RMAP_OWN_COW) { + const xfs_agblock_t bad_bno = + max(xc->irec_startbno, rec->rm_startblock); + const xfs_agblock_t irec_end = + xc->irec_startbno + xc->irec.br_blockcount; + const xfs_extlen_t bad_end = min(irec_end, rec_end); + + error = xrep_cow_mark_file_range(xc, + xfs_gbno_to_fsb(cur->bc_group, bad_bno), + bad_end - bad_bno); + if (error) + return error; } - return xrep_cow_mark_file_range(xc, - xfs_gbno_to_fsb(cur->bc_group, rec_bno), rec_len); + xc->next_bno = max(xc->next_bno, rec_end); + return error; } /* @@ -263,10 +269,12 @@ xrep_cow_find_bad( struct xfs_perag *pag; struct xfs_scrub *sc = xc->sc; xfs_agnumber_t agno; + xfs_agblock_t irec_end; int error; agno = XFS_FSB_TO_AGNO(sc->mp, xc->irec.br_startblock); xc->irec_startbno = XFS_FSB_TO_AGBNO(sc->mp, xc->irec.br_startblock); + irec_end = xc->irec_startbno + xc->irec.br_blockcount; pag = xfs_perag_get(sc->mp, agno); if (!pag) @@ -278,7 +286,7 @@ xrep_cow_find_bad( /* Mark any CoW fork extents that are shared. */ rc_low.rc_startblock = xc->irec_startbno; - rc_high.rc_startblock = xc->irec_startbno + xc->irec.br_blockcount - 1; + rc_high.rc_startblock = irec_end - 1; rc_low.rc_domain = rc_high.rc_domain = XFS_REFC_DOMAIN_SHARED; error = xfs_refcount_query_range(sc->sa.refc_cur, &rc_low, &rc_high, xrep_cow_mark_shared_staging, xc); @@ -287,7 +295,7 @@ xrep_cow_find_bad( /* Make sure there are CoW staging extents for the whole mapping. */ rc_low.rc_startblock = xc->irec_startbno; - rc_high.rc_startblock = xc->irec_startbno + xc->irec.br_blockcount - 1; + rc_high.rc_startblock = irec_end - 1; rc_low.rc_domain = rc_high.rc_domain = XFS_REFC_DOMAIN_COW; xc->next_bno = xc->irec_startbno; error = xfs_refcount_query_range(sc->sa.refc_cur, &rc_low, &rc_high, @@ -295,11 +303,10 @@ xrep_cow_find_bad( if (error) goto out_sa; - if (xc->next_bno < xc->irec_startbno + xc->irec.br_blockcount) { + if (xc->next_bno < irec_end) { error = xrep_cow_mark_file_range(xc, xfs_agbno_to_fsb(pag, xc->next_bno), - xc->irec_startbno + xc->irec.br_blockcount - - xc->next_bno); + irec_end - xc->next_bno); if (error) goto out_sa; } @@ -308,11 +315,20 @@ xrep_cow_find_bad( rm_low.rm_startblock = xc->irec_startbno; memset(&rm_high, 0xFF, sizeof(rm_high)); rm_high.rm_startblock = xc->irec_startbno + xc->irec.br_blockcount - 1; + xc->next_bno = xc->irec_startbno; error = xfs_rmap_query_range(sc->sa.rmap_cur, &rm_low, &rm_high, xrep_cow_mark_missing_staging_rmap, xc); if (error) goto out_sa; + if (xc->next_bno < irec_end) { + error = xrep_cow_mark_file_range(xc, + xfs_agbno_to_fsb(pag, xc->next_bno), + irec_end - xc->next_bno); + if (error) + goto out_sa; + } + /* * If userspace is forcing us to rebuild the CoW fork or someone turned * on the debugging knob, replace everything in the CoW fork. @@ -344,9 +360,11 @@ xrep_cow_find_bad_rt( struct xfs_rmap_irec rm_high = { 0 }; struct xfs_scrub *sc = xc->sc; struct xfs_rtgroup *rtg; + xfs_agblock_t irec_end; int error = 0; xc->irec_startbno = xfs_rtb_to_rgbno(sc->mp, xc->irec.br_startblock); + irec_end = xc->irec_startbno + xc->irec.br_blockcount; rtg = xfs_rtgroup_get(sc->mp, xfs_rtb_to_rgno(sc->mp, xc->irec.br_startblock)); @@ -360,7 +378,7 @@ xrep_cow_find_bad_rt( /* Mark any CoW fork extents that are shared. */ rc_low.rc_startblock = xc->irec_startbno; - rc_high.rc_startblock = xc->irec_startbno + xc->irec.br_blockcount - 1; + rc_high.rc_startblock = irec_end - 1; rc_low.rc_domain = rc_high.rc_domain = XFS_REFC_DOMAIN_SHARED; error = xfs_refcount_query_range(sc->sr.refc_cur, &rc_low, &rc_high, xrep_cow_mark_shared_staging, xc); @@ -369,7 +387,7 @@ xrep_cow_find_bad_rt( /* Make sure there are CoW staging extents for the whole mapping. */ rc_low.rc_startblock = xc->irec_startbno; - rc_high.rc_startblock = xc->irec_startbno + xc->irec.br_blockcount - 1; + rc_high.rc_startblock = irec_end - 1; rc_low.rc_domain = rc_high.rc_domain = XFS_REFC_DOMAIN_COW; xc->next_bno = xc->irec_startbno; error = xfs_refcount_query_range(sc->sr.refc_cur, &rc_low, &rc_high, @@ -377,11 +395,10 @@ xrep_cow_find_bad_rt( if (error) goto out_sr; - if (xc->next_bno < xc->irec_startbno + xc->irec.br_blockcount) { + if (xc->next_bno < irec_end) { error = xrep_cow_mark_file_range(xc, xfs_rgbno_to_rtb(rtg, xc->next_bno), - xc->irec_startbno + xc->irec.br_blockcount - - xc->next_bno); + irec_end - xc->next_bno); if (error) goto out_sr; } @@ -389,12 +406,21 @@ xrep_cow_find_bad_rt( /* Mark any area has an rmap that isn't a COW staging extent. */ rm_low.rm_startblock = xc->irec_startbno; memset(&rm_high, 0xFF, sizeof(rm_high)); - rm_high.rm_startblock = xc->irec_startbno + xc->irec.br_blockcount - 1; + rm_high.rm_startblock = irec_end - 1; + xc->next_bno = xc->irec_startbno; error = xfs_rmap_query_range(sc->sr.rmap_cur, &rm_low, &rm_high, xrep_cow_mark_missing_staging_rmap, xc); if (error) goto out_sr; + if (xc->next_bno < irec_end) { + error = xrep_cow_mark_file_range(xc, + xfs_rgbno_to_rtb(rtg, xc->next_bno), + irec_end - xc->next_bno); + if (error) + goto out_sr; + } + /* * If userspace is forcing us to rebuild the CoW fork or someone * turned on the debugging knob, replace everything in the diff --git a/fs/xfs/scrub/dabtree.c b/fs/xfs/scrub/dabtree.c index d4e5afbde856..02c8544d8918 100644 --- a/fs/xfs/scrub/dabtree.c +++ b/fs/xfs/scrub/dabtree.c @@ -49,8 +49,12 @@ xchk_da_process_error( case -ENODATA: /* Note the badness but don't abort. */ sc->sm->sm_flags |= XFS_SCRUB_OFLAG_CORRUPT; + trace_xchk_file_op_error(sc, ds->dargs.whichfork, + xfs_dir2_da_to_db(ds->dargs.geo, + ds->state->path.blk[level].blkno), + *error, __return_address); *error = 0; - fallthrough; + break; default: trace_xchk_file_op_error(sc, ds->dargs.whichfork, xfs_dir2_da_to_db(ds->dargs.geo, @@ -253,9 +257,10 @@ xchk_da_btree_block_check_sibling( if (sibling == 0) { error = xfs_da3_path_shift(ds->state, altpath, direction, false, &retval); - if (error == 0 && retval == 0) + if (!xchk_da_process_error(ds, level, &error)) + goto out; + if (retval == 0) xchk_da_set_corrupt(ds, level); - error = 0; goto out; } @@ -410,6 +415,10 @@ xchk_da_btree_block( XFS_BLFT_ATTR_LEAF_BUF); blk->magic = XFS_ATTR_LEAF_MAGIC; blk->hashval = xfs_attr_leaf_lasthash(blk->bp, pmaxrecs); + if (dargs->whichfork != XFS_ATTR_FORK) { + xchk_da_set_corrupt(ds, level); + goto out_freebp; + } if (ds->tree_level != 0) xchk_da_set_corrupt(ds, level); break; @@ -419,6 +428,10 @@ xchk_da_btree_block( XFS_BLFT_DIR_LEAFN_BUF); blk->magic = XFS_DIR2_LEAFN_MAGIC; blk->hashval = xfs_dir2_leaf_lasthash(ip, blk->bp, pmaxrecs); + if (dargs->whichfork != XFS_DATA_FORK) { + xchk_da_set_corrupt(ds, level); + goto out_freebp; + } if (ds->tree_level != 0) xchk_da_set_corrupt(ds, level); break; @@ -428,6 +441,10 @@ xchk_da_btree_block( XFS_BLFT_DIR_LEAF1_BUF); blk->magic = XFS_DIR2_LEAF1_MAGIC; blk->hashval = xfs_dir2_leaf_lasthash(ip, blk->bp, pmaxrecs); + if (dargs->whichfork != XFS_DATA_FORK) { + xchk_da_set_corrupt(ds, level); + goto out_freebp; + } if (ds->tree_level != 0) xchk_da_set_corrupt(ds, level); break; @@ -439,6 +456,10 @@ xchk_da_btree_block( node = blk->bp->b_addr; xfs_da3_node_hdr_from_disk(ip->i_mount, &nodehdr, node); btree = nodehdr.btree; + if (nodehdr.count > dargs->geo->node_ents) { + xchk_da_set_corrupt(ds, level); + goto out_freebp; + } *pmaxrecs = nodehdr.count; blk->hashval = be32_to_cpu(btree[*pmaxrecs - 1].hashval); if (level == 0) { diff --git a/fs/xfs/scrub/dir.c b/fs/xfs/scrub/dir.c index 19d974c7e2b7..4ab7e6469248 100644 --- a/fs/xfs/scrub/dir.c +++ b/fs/xfs/scrub/dir.c @@ -155,6 +155,8 @@ xchk_dir_parent_pointer( &sd->pptr_args); if (error == -ENOATTR) xchk_fblock_xref_set_corrupt(sc, XFS_DATA_FORK, 0); + else + xchk_fblock_xref_process_error(sc, XFS_DATA_FORK, 0, &error); return 0; } diff --git a/fs/xfs/scrub/dir_repair.c b/fs/xfs/scrub/dir_repair.c index 2cfcf1c35679..bb3f2f462bde 100644 --- a/fs/xfs/scrub/dir_repair.c +++ b/fs/xfs/scrub/dir_repair.c @@ -1189,7 +1189,9 @@ xrep_dir_want_scan( struct xrep_dir *rd, const struct xfs_inode *ip) { - return ip != rd->sc->ip && !xrep_is_tempfile(ip); + return ip != rd->sc->ip && + xfs_is_metadir_inode(ip) == xfs_is_metadir_inode(rd->sc->ip) && + !xrep_is_tempfile(ip); } /* @@ -1672,7 +1674,10 @@ xrep_dir_swap( ASSERT(sc->ilock_flags & XFS_ILOCK_EXCL); if (rd->pscan.parent_ino == NULLFSINO) { rd->needs_adoption = true; - rd->pscan.parent_ino = rd->sc->mp->m_sb.sb_rootino; + if (xfs_is_metadir_inode(rd->sc->ip)) + rd->pscan.parent_ino = rd->sc->mp->m_sb.sb_metadirino; + else + rd->pscan.parent_ino = rd->sc->mp->m_sb.sb_rootino; } /* diff --git a/fs/xfs/scrub/dirtree.c b/fs/xfs/scrub/dirtree.c index 887383d2e941..11a9dea03d87 100644 --- a/fs/xfs/scrub/dirtree.c +++ b/fs/xfs/scrub/dirtree.c @@ -860,7 +860,8 @@ xchk_dirtree_find_paths_to_root( * A parent pointer of @sc->ip is bad, don't * bother continuing. */ - break; + xchk_set_incomplete(sc); + return error; } if (error == -ESTALE) { /* This had better be an invalidation. */ @@ -981,6 +982,7 @@ xchk_dirtree( * parent pointers are corrupt; this scan cannot be completed * without full information. */ + xchk_set_incomplete(sc); xchk_ip_xref_set_corrupt(sc, sc->ip); error = 0; goto out_scanlock; diff --git a/fs/xfs/scrub/dirtree_repair.c b/fs/xfs/scrub/dirtree_repair.c index 1d1eafcf6eb5..1de4d9f1b616 100644 --- a/fs/xfs/scrub/dirtree_repair.c +++ b/fs/xfs/scrub/dirtree_repair.c @@ -94,7 +94,7 @@ xrep_dirtree_delete_all_paths( } /* Since this is the surviving path, set the dotdot entry to this value. */ -STATIC void +STATIC int xrep_dirpath_retain_parent( struct xchk_dirtree *dl, struct xchk_dirpath *path) @@ -104,18 +104,20 @@ xrep_dirpath_retain_parent( error = xfarray_load(dl->path_steps, path->first_step, &step); if (error) - return; + return error; dl->parent_ino = be64_to_cpu(step.pptr_rec.p_ino); + return 0; } /* Find the one surviving path so we know how to set dotdot. */ -STATIC void +STATIC int xrep_dirtree_find_surviving_path( struct xchk_dirtree *dl, struct xchk_dirtree_outcomes *oc) { struct xchk_dirpath *path; + int error; bool foundit = false; xchk_dirtree_for_each_path(dl, path) { @@ -124,7 +126,10 @@ xrep_dirtree_find_surviving_path( case XCHK_DIRPATH_LOOP: case XCHK_DIRPATH_OK: if (!foundit) { - xrep_dirpath_retain_parent(dl, path); + error = xrep_dirpath_retain_parent(dl, path); + if (error) + return error; + foundit = true; continue; } @@ -136,16 +141,19 @@ xrep_dirtree_find_surviving_path( } ASSERT(oc->suspect + oc->good == 1); + + return 0; } /* Delete all paths except for the one good one. */ -STATIC void +STATIC int xrep_dirtree_keep_one_good_path( struct xchk_dirtree *dl, struct xchk_dirtree_outcomes *oc) { struct xchk_dirpath *path; bool foundit = false; + int error; xchk_dirtree_for_each_path(dl, path) { switch (path->outcome) { @@ -157,7 +165,10 @@ xrep_dirtree_keep_one_good_path( break; case XCHK_DIRPATH_OK: if (!foundit) { - xrep_dirpath_retain_parent(dl, path); + error = xrep_dirpath_retain_parent(dl, path); + if (error) + return error; + foundit = true; continue; } @@ -172,23 +183,28 @@ xrep_dirtree_keep_one_good_path( ASSERT(oc->suspect == 0); ASSERT(oc->good < 2); + return 0; } /* Delete all paths except for one suspect one. */ -STATIC void +STATIC int xrep_dirtree_keep_one_suspect_path( struct xchk_dirtree *dl, struct xchk_dirtree_outcomes *oc) { struct xchk_dirpath *path; bool foundit = false; + int error = 0; xchk_dirtree_for_each_path(dl, path) { switch (path->outcome) { case XCHK_DIRPATH_CORRUPT: case XCHK_DIRPATH_LOOP: if (!foundit) { - xrep_dirpath_retain_parent(dl, path); + error = xrep_dirpath_retain_parent(dl, path); + if (error) + return error; + foundit = true; continue; } @@ -206,13 +222,14 @@ xrep_dirtree_keep_one_suspect_path( ASSERT(oc->suspect == 1); ASSERT(oc->good == 0); + return error; } /* * Figure out what to do with the paths we tried to find. Returns -EDEADLOCK * if the scan results have become stale. */ -STATIC void +STATIC int xrep_dirtree_decide_fate( struct xchk_dirtree *dl, struct xchk_dirtree_outcomes *oc) @@ -222,36 +239,32 @@ xrep_dirtree_decide_fate( /* Parentless directories should not have any paths at all. */ if (xchk_dirtree_parentless(dl)) { xrep_dirtree_delete_all_paths(dl, oc); - return; + return 0; } /* One path is exactly the number of paths we want. */ - if (oc->good + oc->suspect == 1) { - xrep_dirtree_find_surviving_path(dl, oc); - return; - } + if (oc->good + oc->suspect == 1) + return xrep_dirtree_find_surviving_path(dl, oc); /* Zero paths means we should reattach the subdir to the orphanage. */ if (oc->good + oc->suspect == 0) { if (dl->sc->orphanage) oc->needs_adoption = true; - return; + return 0; } /* * Otherwise, this subdirectory has too many parents. If there's at * least one good path, keep it and delete the others. */ - if (oc->good > 0) { - xrep_dirtree_keep_one_good_path(dl, oc); - return; - } + if (oc->good > 0) + return xrep_dirtree_keep_one_good_path(dl, oc); /* * There are no good paths and there are too many suspect paths. * Keep the first suspect path and delete the rest. */ - xrep_dirtree_keep_one_suspect_path(dl, oc); + return xrep_dirtree_keep_one_suspect_path(dl, oc); } /* @@ -427,6 +440,11 @@ xrep_dirtree_unlink( error = -ESTALE; goto out_trans_cancel; } + if (dl->aborted) { + mutex_unlock(&dl->lock); + error = -EIO; + goto out_trans_cancel; + } xrep_dirpath_set_outcome(dl, path, XREP_DIRPATH_DELETING); mutex_unlock(&dl->lock); @@ -557,6 +575,8 @@ xrep_dirtree_delete_path( if (!error && dl->stale) error = -ESTALE; + if (!error && dl->aborted) + error = -EIO; return error; } @@ -702,6 +722,11 @@ xrep_dirtree_adopt( error = -ESTALE; goto out_trans; } + if (dl->aborted) { + mutex_unlock(&dl->lock); + error = -EIO; + goto out_trans; + } error = xrep_dirtree_create_adoption_path(dl); mutex_unlock(&dl->lock); if (error) @@ -762,6 +787,8 @@ xrep_dirtree_move_to_orphanage( if (!error && dl->stale) error = -ESTALE; + if (!error && dl->aborted) + error = -EIO; return error; } @@ -813,13 +840,20 @@ xrep_dirtree( */ mutex_lock(&dl->lock); do { + if (dl->aborted) { + error = -EIO; + break; + } + /* * Decide what we're going to do, then do it. An -ESTALE * return here means the scan results are invalid and we have * to walk again. */ if (!dl->stale) { - xrep_dirtree_decide_fate(dl, &oc); + error = xrep_dirtree_decide_fate(dl, &oc); + if (error) + break; trace_xrep_dirtree_decided_fate(dl, &oc); diff --git a/fs/xfs/scrub/dqiterate.c b/fs/xfs/scrub/dqiterate.c index 079dc4e691a0..76393c888484 100644 --- a/fs/xfs/scrub/dqiterate.c +++ b/fs/xfs/scrub/dqiterate.c @@ -58,6 +58,7 @@ xchk_dquot_iter_revalidate_bmap( * no need to reread the bmbt. */ if (cursor->bmap.br_startoff != NULLFILEOFF && + cursor->bmap.br_startoff <= fileoff && cursor->if_seq == ifp->if_seq && cursor->bmap.br_startoff + cursor->bmap.br_blockcount > fileoff) return 0; diff --git a/fs/xfs/scrub/fscounters.c b/fs/xfs/scrub/fscounters.c index 916699f3da7b..67e159f39951 100644 --- a/fs/xfs/scrub/fscounters.c +++ b/fs/xfs/scrub/fscounters.c @@ -297,7 +297,7 @@ xchk_fscount_aggregate_agcounts( { struct xfs_mount *mp = sc->mp; struct xfs_perag *pag = NULL; - uint64_t delayed; + int64_t delayed; int tries = 8; int error = 0; @@ -359,6 +359,11 @@ xchk_fscount_aggregate_agcounts( * of the computation. */ delayed = percpu_counter_sum(&mp->m_delalloc_blks); + if (delayed < 0) { + ASSERT(delayed >= 0); + xchk_set_incomplete(sc); + return -EFSCORRUPTED; + } fsc->fdblocks -= delayed; trace_xchk_fscounters_calc(mp, fsc->icount, fsc->ifree, fsc->fdblocks, @@ -368,8 +373,10 @@ xchk_fscount_aggregate_agcounts( /* Bail out if the values we compute are totally nonsense. */ if (fsc->icount < fsc->icount_min || fsc->icount > fsc->icount_max || fsc->fdblocks > mp->m_sb.sb_dblocks || - fsc->ifree > fsc->icount_max) + fsc->ifree > fsc->icount_max) { + xchk_set_incomplete(sc); return -EFSCORRUPTED; + } /* * If ifree > icount then we probably had some perturbation in the @@ -377,6 +384,11 @@ xchk_fscount_aggregate_agcounts( * to maintain ifree <= icount before giving up. */ if (fsc->ifree > fsc->icount) { + if (fsc->frozen) { + xchk_set_incomplete(sc); + return -EFSCORRUPTED; + } + if (tries--) goto retry; return -EDEADLOCK; @@ -410,6 +422,7 @@ xchk_fscount_count_frextents( { struct xfs_mount *mp = sc->mp; struct xfs_rtgroup *rtg = NULL; + int64_t delayed; int error; fsc->frextents = 0; @@ -435,7 +448,14 @@ xchk_fscount_count_frextents( } } - fsc->frextents_delayed = percpu_counter_sum(&mp->m_delalloc_rtextents); + delayed = percpu_counter_sum(&mp->m_delalloc_rtextents); + if (delayed < 0) { + ASSERT(delayed >= 0); + xchk_set_incomplete(sc); + return -EFSCORRUPTED; + } + fsc->frextents_delayed = delayed; + return 0; } #else @@ -541,7 +561,10 @@ xchk_fscounters( return -EDEADLOCK; xchk_set_corrupt(sc); - return 0; + + /* Need to compute all the fields in fsc for a repair */ + if (!xchk_could_repair(sc)) + return 0; } /* See if icount is obviously wrong. */ @@ -604,6 +627,13 @@ xchk_fscounters( try_again = true; } + if (!xfs_has_zoned(mp) && fsc->frextents < fsc->frextents_delayed) { + if (fsc->frozen) + xchk_set_incomplete(sc); + else + try_again = true; + } + if (!xfs_has_zoned(mp) && !xchk_fscount_within_range(sc, frextents, &mp->m_free[XC_FREE_RTEXTENTS].count, diff --git a/fs/xfs/scrub/ialloc_repair.c b/fs/xfs/scrub/ialloc_repair.c index 46b1c8ac5543..43846b3bbf4e 100644 --- a/fs/xfs/scrub/ialloc_repair.c +++ b/fs/xfs/scrub/ialloc_repair.c @@ -293,12 +293,8 @@ xrep_ibt_process_cluster( struct xfs_ino_geometry *igeo = M_IGEO(mp); xfs_agino_t cluster_ag_base; xfs_agino_t irec_index; - unsigned int nr_inodes; int error; - nr_inodes = min_t(unsigned int, igeo->inodes_per_cluster, - XFS_INODES_PER_CHUNK); - /* * Grab the inode cluster buffer. This is safe to do with a broken * inobt because imap_to_bp directly maps the buffer without touching @@ -317,6 +313,10 @@ xrep_ibt_process_cluster( for (irec_index = 0; irec_index < igeo->inodes_per_cluster; irec_index += XFS_INODES_PER_CHUNK) { + unsigned int nr_inodes = + min_t(unsigned int, XFS_INODES_PER_CHUNK, + igeo->inodes_per_cluster - irec_index); + error = xrep_ibt_cluster_record(ri, cluster_ag_base + irec_index, cluster_bp, nr_inodes); diff --git a/fs/xfs/scrub/inode.c b/fs/xfs/scrub/inode.c index 46e9bf4a4317..ac42867acbc4 100644 --- a/fs/xfs/scrub/inode.c +++ b/fs/xfs/scrub/inode.c @@ -212,7 +212,7 @@ xchk_setup_inode( * make repairs to the ondisk inode buffer. */ if (xchk_could_repair(sc)) - xrep_setup_inode(sc, &imap); + return xrep_setup_inode(sc, &imap); return 0; out_cancel: diff --git a/fs/xfs/scrub/inode_repair.c b/fs/xfs/scrub/inode_repair.c index b87c22146233..25e71e7d472f 100644 --- a/fs/xfs/scrub/inode_repair.c +++ b/fs/xfs/scrub/inode_repair.c @@ -41,6 +41,8 @@ #include "xfs_rtgroup.h" #include "xfs_rtrmap_btree.h" #include "xfs_rtrefcount_btree.h" +#include "xfs_dquot_item.h" +#include "xfs_dquot.h" #include "scrub/xfs_scrub.h" #include "scrub/scrub.h" #include "scrub/common.h" @@ -563,6 +565,8 @@ xrep_dinode_flags( */ if (xfs_has_reflink(mp) && S_ISREG(mode)) flags2 |= XFS_DIFLAG2_REFLINK; + else if (xfs_has_reflink(mp) && S_ISDIR(mode)) + flags2 &= ~XFS_DIFLAG2_REFLINK; else flags2 &= ~(XFS_DIFLAG2_REFLINK | XFS_DIFLAG2_COWEXTSIZE); if (!xfs_has_bigtime(mp)) @@ -579,8 +583,10 @@ xrep_dinode_flags( fa = xfs_dinode_verify_metadir(sc->mp, dip, mode, flags, flags2); - if (fa) + if (fa) { flags2 &= ~XFS_DIFLAG2_METADATA; + dip->di_metatype = cpu_to_be16(XFS_METAFILE_UNKNOWN); + } } dip->di_flags = cpu_to_be16(flags); @@ -1012,6 +1018,11 @@ xrep_dinode_bad_metabt_fork( if (whichfork != XFS_DATA_FORK) return true; + if (!xfs_has_metadir(sc->mp)) + return true; + if (!(dip->di_flags2 & cpu_to_be64(XFS_DIFLAG2_METADATA))) + return true; + switch (be16_to_cpu(dip->di_metatype)) { case XFS_METAFILE_RTRMAP: return xrep_dinode_bad_rtrmapbt_fork(sc, dip, dfork_size); @@ -1024,6 +1035,30 @@ xrep_dinode_bad_metabt_fork( return false; } +static xfs_failaddr_t +xrep_symlink_shortform_verify( + void *sfp, + int64_t size) +{ + /* + * Zero length symlinks should never occur in memory as they are + * never allowed to exist on disk. + */ + if (!size) + return __this_address; + + /* No negative sizes or overly long symlink targets. */ + if (size < 0 || size > XFS_SYMLINK_MAXLEN) + return __this_address; + + /* No NULLs in the target either. */ + if (memchr(sfp, 0, size)) + return __this_address; + + /* ondisk symlink target isn't null terminated, unlike incore */ + return NULL; +} + /* * Check the data fork for things that will fail the ifork verifiers or the * ifork formatters. @@ -1099,7 +1134,7 @@ xrep_dinode_check_dfork( return true; /* symlink structure must pass verification. */ if (S_ISLNK(mode) && - xfs_symlink_shortform_verify(dfork_ptr, data_size) != NULL) + xrep_symlink_shortform_verify(dfork_ptr, data_size) != NULL) return true; break; case XFS_DINODE_FMT_EXTENTS: @@ -1210,6 +1245,8 @@ xrep_dinode_check_afork( if (XFS_DFORK_BOFF(dip) == 0) return dip->di_aformat != XFS_DINODE_FMT_EXTENTS || xfs_dfork_attr_extents(dip) != 0; + if (XFS_DFORK_BOFF(dip) >= XFS_LITINO(sc->mp)) + return true; afork_size = XFS_DFORK_SIZE(dip, sc->mp, XFS_ATTR_FORK); afork_ptr = XFS_DFORK_PTR(dip, XFS_ATTR_FORK); @@ -1276,7 +1313,12 @@ xrep_dinode_zap_afork( */ if (dip->di_format != XFS_DINODE_FMT_BTREE) dip->di_forkoff = 0; - dip->di_mode = cpu_to_be16(mode & ~0777); + + /* + * Zapping the attr fork could remove ACLs, so we clear suid/sgid and + * all access mode bits. + */ + dip->di_mode = cpu_to_be16(mode & ~(S_ISUID | S_ISGID | 0777)); dip->di_uid = 0; dip->di_gid = 0; } @@ -1684,7 +1726,7 @@ xrep_inode_blockcounts( */ ; } else if (XFS_IS_REALTIME_INODE(sc->ip)) { - if (count >= sc->mp->m_sb.sb_rblocks) + if (count >= sc->mp->m_sb.sb_rblocks + sc->mp->m_sb.sb_dblocks) return -EFSCORRUPTED; } else { if (count >= sc->mp->m_sb.sb_dblocks) @@ -1726,6 +1768,20 @@ xrep_inode_ids( trace_xrep_inode_ids(sc); + /* + * We changed one of the file IDs, so we must detach the dquots to + * prevent further changes from being accounted to them, and try to + * attach the dquot for the new id to absorb any other block usage + * changes. We can't do that right now because the existing dquots + * might be attached to the scrub transaction and we're in the middle + * of fixing the inode, so we'll switch the incore dquot pointers + * later. + * + * Schedule a quotacheck so the accounting discrepancies will get fixed + * eventually. We'll have to repeat this later when we do the actual + * switch. + */ + if (!uid_valid(VFS_I(sc->ip)->i_uid)) { i_uid_write(VFS_I(sc->ip), 0); dirty = true; @@ -2035,6 +2091,113 @@ xrep_inode_unlinked( return 0; } +#ifdef CONFIG_XFS_QUOTA +static inline bool +xrep_inode_need_reset_dquots( + struct xfs_scrub *sc) +{ + struct xfs_mount *mp = sc->mp; + struct xfs_inode *ip = sc->ip; + + if (!XFS_IS_QUOTA_ON(mp)) + return false; + if (xfs_is_quota_inode(&mp->m_sb, I_INO(ip))) + return false; + if (xfs_is_metadir_inode(ip)) + return false; + + if (XFS_IS_UQUOTA_ON(sc->mp) && + (!ip->i_udquot || ip->i_udquot->q_id != i_uid_read(VFS_I(ip)))) + return true; + if (XFS_IS_GQUOTA_ON(sc->mp) && + (!ip->i_gdquot || ip->i_gdquot->q_id != i_gid_read(VFS_I(ip)))) + return true; + if (XFS_IS_PQUOTA_ON(sc->mp) && + (!ip->i_pdquot || ip->i_pdquot->q_id != ip->i_projid)) + return true; + + return false; +} + +/* Switch dquot, return dqtype if we need quotacheck */ +static xfs_dqtype_t +xrep_inode_switch_dquot( + struct xfs_scrub *sc, + xfs_dqtype_t type, + struct xfs_dquot **dqp) +{ + xfs_dqtype_t ret = 0; + xfs_dqid_t id = xfs_qm_id_for_quotatype(sc->ip, type); + + if (*dqp != NULL && (*dqp)->q_id != id) { + xfs_qm_dqrele(*dqp); + *dqp = NULL; + } + + if (!*dqp) { + xfs_qm_dqget_inode(sc->ip, type, true, dqp); + ret = type; + } + return ret; +} + +static int +xrep_inode_reset_dquots( + struct xfs_scrub *sc) +{ + xfs_dqtype_t need = 0; + int error; + + /* + * If we changed the file IDs, then we need to drop the transaction so + * that we can detach the dquots and attach new ones. + * + * Commit the scrub transaction so that there are no dangling + * references to any dquots that might get detached, and retain the + * ILOCK, as required by the dqattach code. + * + * Note that dqattach can cycle the ILOCK when creating a new dquot, so + * this must be the very last thing that the repair function does. + */ + error = xrep_trans_commit(sc); + if (error) + return error; + + if (XFS_IS_UQUOTA_ON(sc->mp)) + need |= xrep_inode_switch_dquot(sc, XFS_DQTYPE_USER, + &sc->ip->i_udquot); + if (XFS_IS_GQUOTA_ON(sc->mp)) + need |= xrep_inode_switch_dquot(sc, XFS_DQTYPE_GROUP, + &sc->ip->i_gdquot); + if (XFS_IS_PQUOTA_ON(sc->mp)) + need |= xrep_inode_switch_dquot(sc, XFS_DQTYPE_PROJ, + &sc->ip->i_pdquot); + + xchk_iunlock(sc, XFS_ILOCK_EXCL); + + /* + * Now that we've updated the dquots, force quotacheck once again. + * In theory we could be racing against a concurrent quotacheck. + * Note that we can't hold the ILOCK when allocating a transaction. + */ + error = xchk_trans_alloc(sc, 0); + if (error) + return error; + + if (need & XFS_DQTYPE_USER) + xrep_force_quotacheck(sc, XFS_DQTYPE_USER); + if (need & XFS_DQTYPE_GROUP) + xrep_force_quotacheck(sc, XFS_DQTYPE_GROUP); + if (need & XFS_DQTYPE_PROJ) + xrep_force_quotacheck(sc, XFS_DQTYPE_PROJ); + + return 0; +} +#else +# define xrep_inode_need_reset_dquots(sc) (false) +# define xrep_inode_reset_dquots(sc) (0) +#endif + /* Repair an inode's fields. */ int xrep_inode( @@ -2089,5 +2252,13 @@ xrep_inode( if (error) return error; - return xrep_defer_finish(sc); + error = xrep_defer_finish(sc); + if (error) + return error; + + /* Must come last */ + if (xrep_inode_need_reset_dquots(sc)) + return xrep_inode_reset_dquots(sc); + + return 0; } diff --git a/fs/xfs/scrub/iscan.c b/fs/xfs/scrub/iscan.c index 03f72ba9e7a3..33cfce603b5b 100644 --- a/fs/xfs/scrub/iscan.c +++ b/fs/xfs/scrub/iscan.c @@ -545,7 +545,7 @@ xchk_iscan_finish_batch( mutex_lock(&iscan->lock); - if (iscan->__batch_ino != NULLFSINO) { + if (iscan->__batch_ino != NULLFSINO && iscan->__skipped_inomask) { highest_skipped = iscan->__batch_ino + xfs_highbit64(iscan->__skipped_inomask); iscan->__visited_ino = max(iscan->__visited_ino, diff --git a/fs/xfs/scrub/metapath.c b/fs/xfs/scrub/metapath.c index e0ee7d9b903f..eb987f8c994d 100644 --- a/fs/xfs/scrub/metapath.c +++ b/fs/xfs/scrub/metapath.c @@ -129,6 +129,8 @@ static int xchk_setup_metapath_rtdir( struct xfs_scrub *sc) { + if (sc->sm->sm_agno) + return -EINVAL; if (!sc->mp->m_rtdirip) return -ENOENT; @@ -176,6 +178,8 @@ xchk_setup_metapath_quotadir( { struct xfs_quotainfo *qi = sc->mp->m_quotainfo; + if (sc->sm->sm_agno) + return -EINVAL; if (!qi || !qi->qi_dirip) return -ENOENT; @@ -192,6 +196,8 @@ xchk_setup_metapath_dqinode( struct xfs_quotainfo *qi = sc->mp->m_quotainfo; struct xfs_inode *ip = NULL; + if (sc->sm->sm_agno) + return -EINVAL; if (!qi) return -ENOENT; @@ -319,6 +325,13 @@ xchk_metapath( return 0; } + /* Callers should not set the parent to the child. */ + if (mpath->dp == sc->ip) { + ASSERT(mpath->dp != sc->ip); + xchk_set_incomplete(sc); + return -EIO; + } + xchk_trans_alloc_empty(sc); error = xchk_metapath_ilock_both(mpath); @@ -509,7 +522,7 @@ xrep_metapath_try_link( xfs_ino_t *alleged_child) { struct xfs_scrub *sc = mpath->sc; - xfs_ino_t ino; + xfs_ino_t ino = NULLFSINO; int error; /* Allocate transaction, lock inodes, join to transaction. */ @@ -608,7 +621,7 @@ xrep_metapath_try_unlink( { struct xfs_scrub *sc = mpath->sc; struct xfs_inode *ip = NULL; - xfs_ino_t ino; + xfs_ino_t ino = NULLFSINO; int error; ASSERT(*alleged_child != I_INO(sc->ip)); diff --git a/fs/xfs/scrub/newbt.c b/fs/xfs/scrub/newbt.c index 584076b2a6ee..2bb5c9f2d97e 100644 --- a/fs/xfs/scrub/newbt.c +++ b/fs/xfs/scrub/newbt.c @@ -71,7 +71,7 @@ xrep_newbt_estimate_slack( free = sc->sa.pag->pagf_freeblks; sz = xfs_ag_block_count(sc->mp, pag_agno(sc->sa.pag)); } else { - free = xfs_sum_freecounter_raw(sc->mp, XC_FREE_BLOCKS); + free = xfs_sum_freecounter(sc->mp, XC_FREE_BLOCKS); sz = sc->mp->m_sb.sb_dblocks; } @@ -598,16 +598,3 @@ xrep_newbt_claim_block( /* Relog all the EFIs. */ return xrep_defer_finish(xnr->sc); } - -/* How many reserved blocks are unused? */ -unsigned int -xrep_newbt_unused_blocks( - struct xrep_newbt *xnr) -{ - struct xrep_newbt_resv *resv; - unsigned int unused = 0; - - list_for_each_entry(resv, &xnr->resv_list, list) - unused += resv->len - resv->used; - return unused; -} diff --git a/fs/xfs/scrub/newbt.h b/fs/xfs/scrub/newbt.h index 5ce785599287..e5ebd3200c18 100644 --- a/fs/xfs/scrub/newbt.h +++ b/fs/xfs/scrub/newbt.h @@ -71,6 +71,5 @@ void xrep_newbt_cancel(struct xrep_newbt *xnr); int xrep_newbt_commit(struct xrep_newbt *xnr); int xrep_newbt_claim_block(struct xfs_btree_cur *cur, struct xrep_newbt *xnr, union xfs_btree_ptr *ptr); -unsigned int xrep_newbt_unused_blocks(struct xrep_newbt *xnr); #endif /* __XFS_SCRUB_NEWBT_H__ */ diff --git a/fs/xfs/scrub/parent_repair.c b/fs/xfs/scrub/parent_repair.c index f643724910f9..6a3c0a190ff9 100644 --- a/fs/xfs/scrub/parent_repair.c +++ b/fs/xfs/scrub/parent_repair.c @@ -910,6 +910,10 @@ xrep_parent_fetch_xattr_remote( }; int error; + /* Don't even try to handle a bizarrely large xattr. */ + if (valuelen > XATTR_SIZE_MAX) + return -EFSCORRUPTED; + /* * If we need a larger value buffer, try to allocate one. If that * fails, return with -EDEADLOCK to try harder. diff --git a/fs/xfs/scrub/quota.c b/fs/xfs/scrub/quota.c index 222812fe202c..7a1704b0c9d8 100644 --- a/fs/xfs/scrub/quota.c +++ b/fs/xfs/scrub/quota.c @@ -182,21 +182,21 @@ xchk_quota_item( * suspect, which is why we flag it for review. * * Complain about corruption if the soft limit is greater than - * the hard limit. + * the hard limit and there's a hard limit set. */ if (dq->q_blk.hardlimit > mp->m_sb.sb_dblocks) xchk_fblock_set_warning(sc, XFS_DATA_FORK, offset); - if (dq->q_blk.softlimit > dq->q_blk.hardlimit) + if (dq->q_blk.hardlimit && dq->q_blk.softlimit > dq->q_blk.hardlimit) xchk_fblock_set_corrupt(sc, XFS_DATA_FORK, offset); if (dq->q_ino.hardlimit > M_IGEO(mp)->maxicount) xchk_fblock_set_warning(sc, XFS_DATA_FORK, offset); - if (dq->q_ino.softlimit > dq->q_ino.hardlimit) + if (dq->q_ino.hardlimit && dq->q_ino.softlimit > dq->q_ino.hardlimit) xchk_fblock_set_corrupt(sc, XFS_DATA_FORK, offset); if (dq->q_rtb.hardlimit > mp->m_sb.sb_rblocks) xchk_fblock_set_warning(sc, XFS_DATA_FORK, offset); - if (dq->q_rtb.softlimit > dq->q_rtb.hardlimit) + if (dq->q_rtb.hardlimit && dq->q_rtb.softlimit > dq->q_rtb.hardlimit) xchk_fblock_set_corrupt(sc, XFS_DATA_FORK, offset); /* Check the resource counts. */ diff --git a/fs/xfs/scrub/quota_repair.c b/fs/xfs/scrub/quota_repair.c index 59302e8afc7e..b4f33f432b12 100644 --- a/fs/xfs/scrub/quota_repair.c +++ b/fs/xfs/scrub/quota_repair.c @@ -193,17 +193,17 @@ xrep_quota_item( goto out_unlock_dquot; /* Check the limits. */ - if (dq->q_blk.softlimit > dq->q_blk.hardlimit) { + if (dq->q_blk.hardlimit && dq->q_blk.softlimit > dq->q_blk.hardlimit) { dq->q_blk.softlimit = dq->q_blk.hardlimit; dirty = true; } - if (dq->q_ino.softlimit > dq->q_ino.hardlimit) { + if (dq->q_ino.hardlimit && dq->q_ino.softlimit > dq->q_ino.hardlimit) { dq->q_ino.softlimit = dq->q_ino.hardlimit; dirty = true; } - if (dq->q_rtb.softlimit > dq->q_rtb.hardlimit) { + if (dq->q_rtb.hardlimit && dq->q_rtb.softlimit > dq->q_rtb.hardlimit) { dq->q_rtb.softlimit = dq->q_rtb.hardlimit; dirty = true; } @@ -248,10 +248,7 @@ xrep_quota_item( dq->q_flags |= XFS_DQFLAG_DIRTY; xfs_trans_dqjoin(sc->tp, dq); - if (dq->q_id) { - xfs_qm_adjust_dqlimits(dq); - xfs_qm_adjust_dqtimers(dq); - } + xfs_qm_adjust_dqenforcement(dq); xfs_trans_log_dquot(sc->tp, dq); return xfs_trans_roll(&sc->tp); @@ -275,6 +272,9 @@ xrep_quota_fix_timer( time64_t new_timer; uint32_t t; + /* never reset the default quota timer */ + if (ddq->d_id == 0) + return; if (!soft || count <= soft || *timer != 0) return; @@ -419,7 +419,9 @@ xrep_quota_data_fork( /* Check for data fork problems that apply only to quota files. */ max_dqid_off = XFS_DQ_ID_MAX / qi->qi_dqperchunk; ifp = xfs_ifork_ptr(sc->ip, XFS_DATA_FORK); - for_each_xfs_iext(ifp, &icur, &irec) { + xfs_iext_first(ifp, &icur); + + while (xfs_iext_get_extent(ifp, &icur, &irec)) { if (isnullstartblock(irec.br_startblock)) { error = -EFSCORRUPTED; goto out; @@ -443,16 +445,35 @@ xrep_quota_data_fork( error = xfs_bmapi_write(sc->tp, sc->ip, irec.br_startoff, irec.br_blockcount, - XFS_BMAPI_CONVERT, 0, &nrec, &nmap); + XFS_BMAPI_CONVERT | XFS_BMAPI_ZERO, 0, + &nrec, &nmap); if (error) goto out; ASSERT(nrec.br_startoff == irec.br_startoff); ASSERT(nrec.br_blockcount == irec.br_blockcount); + /* + * We just zeroed the ondisk dquot, so counts could be + * wrong. Schedule a quotacheck to fix that. + */ + xrep_force_quotacheck(sc, dqtype); error = xfs_defer_finish(&sc->tp); if (error) goto out; + + /* + * Set cursor to the next block past the extent we just + * converted because bmapi write could have merged + * records. + */ + if (!xfs_iext_lookup_extent(sc->ip, ifp, + irec.br_startoff + irec.br_blockcount, + &icur, &irec)) + break; + continue; } + + xfs_iext_next(ifp, &icur); } if (!joined) { diff --git a/fs/xfs/scrub/quotacheck_repair.c b/fs/xfs/scrub/quotacheck_repair.c index dbb522e1513b..b8334edf380c 100644 --- a/fs/xfs/scrub/quotacheck_repair.c +++ b/fs/xfs/scrub/quotacheck_repair.c @@ -39,6 +39,54 @@ * dquot is locked. */ +static bool +xqcheck_dqres_force_dirty( + const struct xfs_dquot_res *res, + const struct xfs_quota_limits *qlim) +{ + /* zero limits mean that we should set the default limits */ + if (res->softlimit == 0 && qlim->soft != 0) + return true; + if (res->hardlimit == 0 && qlim->hard != 0) + return true; + + /* do we need to adjust the timer setting? */ + if ((res->softlimit && res->count > res->softlimit) || + (res->hardlimit && res->count > res->hardlimit)) { + if (!res->timer) + return true; + } else { + if (res->timer) + return true; + } + + return false; +} + +/* Decide if we need to adjust the dquot limits or timers */ +static bool +xqcheck_dquot_force_dirty( + const struct xfs_dquot *dq) +{ + struct xfs_quotainfo *qi = dq->q_mount->m_quotainfo; + struct xfs_def_quota *defq; + + /* root dquot does not enforce limits */ + if (dq->q_id == 0) + return false; + + defq = xfs_get_defquota(qi, xfs_dquot_type(dq)); + + if (xqcheck_dqres_force_dirty(&dq->q_blk, &defq->blk)) + return true; + if (xqcheck_dqres_force_dirty(&dq->q_ino, &defq->ino)) + return true; + if (xqcheck_dqres_force_dirty(&dq->q_rtb, &defq->rtb)) + return true; + + return false; +} + /* Commit new counters to a dquot. */ static int xqcheck_commit_dquot( @@ -91,6 +139,9 @@ xqcheck_commit_dquot( dirty = true; } + if (!dirty && xqcheck_dquot_force_dirty(dq)) + dirty = true; + xcdq.flags |= (XQCHECK_DQUOT_REPAIR_SCANNED | XQCHECK_DQUOT_WRITTEN); error = xfarray_store(counts, dq->q_id, &xcdq); if (error == -EFBIG) { @@ -110,8 +161,7 @@ xqcheck_commit_dquot( /* Commit the dirty dquot to disk. */ dq->q_flags |= XFS_DQFLAG_DIRTY; - if (dq->q_id) - xfs_qm_adjust_dqtimers(dq); + xfs_qm_adjust_dqenforcement(dq); xfs_trans_log_dquot(xqc->sc->tp, dq); return xrep_trans_commit(xqc->sc); diff --git a/fs/xfs/scrub/readdir.c b/fs/xfs/scrub/readdir.c index d68bf20b4a14..970d43030704 100644 --- a/fs/xfs/scrub/readdir.c +++ b/fs/xfs/scrub/readdir.c @@ -238,7 +238,7 @@ xchk_dir_walk_leaf( dep = bp->b_addr + offset; length = xfs_dir2_data_entsize(mp, dep->namelen); - dapos = xfs_dir2_byte_to_dataptr(curoff) & 0x7fffffff; + dapos = xfs_dir2_byte_to_dataptr(curoff); ino = be64_to_cpu(dep->inumber); name.name = dep->name; name.len = dep->namelen; diff --git a/fs/xfs/scrub/reap.c b/fs/xfs/scrub/reap.c index 0dfe61bafc6a..a0bee8c7abc0 100644 --- a/fs/xfs/scrub/reap.c +++ b/fs/xfs/scrub/reap.c @@ -796,9 +796,9 @@ xreap_fsmeta_extent( xfs_agblock_t agbno_next = agbno + len; int error = 0; - ASSERT(len <= XFS_MAX_BMBT_EXTLEN); ASSERT(sc->ip != NULL); ASSERT(!sc->sa.pag); + ASSERT(agno == XFS_FSB_TO_AGNO(sc->mp, fsbno + len - 1)); /* * We're reaping blocks after repairing file metadata, which means that @@ -1080,26 +1080,23 @@ xreap_configure_rgcow_limits( XFS_RTGLOCK_RMAP | \ XFS_RTGLOCK_REFCOUNT) -/* - * Break a rt file metadata extent into sub-extents by fate (crosslinked, not - * crosslinked), and dispose of each sub-extent separately. The extent must - * be aligned to a realtime extent. - */ -STATIC int -xreap_rtmeta_extent( - uint64_t rtbno, +static int +xreap_rtgroup_extent( + struct xreap_state *rs, + xfs_rtblock_t rtbno, uint64_t len, - void *priv) + xfs_rgblock_t *done) { - struct xreap_state *rs = priv; struct xfs_scrub *sc = rs->sc; xfs_rgblock_t rgbno = xfs_rtb_to_rgbno(sc->mp, rtbno); - xfs_rgblock_t rgbno_next = rgbno + len; + xfs_rgblock_t rgbno_next; int error = 0; ASSERT(sc->ip != NULL); ASSERT(!sc->sr.rtg); + *done = 0; + /* * We're reaping blocks after repairing file metadata, which means that * we have to init the xchk_ag structure ourselves. @@ -1110,6 +1107,7 @@ xreap_rtmeta_extent( xfs_rtgroup_lock(sc->sr.rtg, XREAP_RTGLOCK_ALL); + rgbno_next = rgbno + min(len, (uint64_t)rtg_blocks(sc->sr.rtg) - rgbno); while (rgbno < rgbno_next) { xfs_extlen_t rglen; bool crosslinked; @@ -1136,6 +1134,7 @@ xreap_rtmeta_extent( } rgbno += rglen; + *done += rglen; } out_unlock: @@ -1145,6 +1144,44 @@ xreap_rtmeta_extent( return error; } +/* + * Break a rt file metadata extent into sub-extents by fate (crosslinked, not + * crosslinked), and dispose of each sub-extent separately. The extent must + * be aligned to a realtime extent. + */ +STATIC int +xreap_rtmeta_extent( + uint64_t rtbno, + uint64_t len, + void *priv) +{ + struct xreap_state *rs = priv; + int error; + + /* + * Space from adjacent rtgroups can be stored as a single bitmap extent + * because rtgroups do not have fixed-location metadata such as a + * per-group superblock that cannot be reaped. Handle these overlaps + * by only reaping within a single rtgroup at a time. + * + * In the somewhat unlikely event that an rtgroup size is not a power + * of two, the segmented nature of xfs_rtblock_t prevents space in + * adjacent rtgroups from merging in the bitmap. + */ + while (len > 0) { + xfs_rgblock_t done = 0; + + error = xreap_rtgroup_extent(rs, rtbno, len, &done); + if (error) + return error; + + rtbno += done; + len -= done; + } + + return 0; +} + /* * Dispose of every block of every rt metadata extent in the bitmap. * Do not use this to dispose of the mappings in an ondisk inode fork. @@ -1443,12 +1480,12 @@ xrep_reap_bmapi_iter( /* * t0: Schedule removal of the mapping from the fork. We use * deferred log intents in this function to control the exact - * sequence of metadata updates. + * sequence of metadata updates. The BUI queues a RUI to + * remove the reverse mapping. */ xfs_bmap_unmap_extent(sc->tp, rs->ip, rs->whichfork, imap); xfs_trans_mod_dquot_byino(sc->tp, rs->ip, XFS_TRANS_DQ_BCOUNT, -(int64_t)imap->br_blockcount); - xfs_rmap_unmap_extent(sc->tp, rs->ip, rs->whichfork, imap); return 0; } diff --git a/fs/xfs/scrub/refcount.c b/fs/xfs/scrub/refcount.c index f8c51d8fbb3d..883c273cb988 100644 --- a/fs/xfs/scrub/refcount.c +++ b/fs/xfs/scrub/refcount.c @@ -129,6 +129,18 @@ xchk_refcountbt_rmap_check( return 0; } + /* + * The refcount btree should only have records for written file data + * blocks or for COW staging blocks. + */ + if ((XFS_RMAP_NON_INODE_OWNER(rec->rm_owner) && + rec->rm_owner != XFS_RMAP_OWN_COW) || + (rec->rm_flags & (XFS_RMAP_ATTR_FORK | XFS_RMAP_BMBT_BLOCK | + XFS_RMAP_UNWRITTEN))) { + xchk_btree_xref_set_corrupt(refchk->sc, cur, 0); + return 0; + } + if (rec->rm_startblock <= refchk->bno && rm_last >= rc_last) { /* * The rmap overlaps the refcount record, so we can confirm @@ -173,8 +185,11 @@ xchk_refcountbt_process_rmap_fragments( xfs_nlink_t target_nr; target_nr = refchk->refcount - refchk->seen; - if (target_nr == 0) + if (target_nr == 0) { + if (!list_empty(&refchk->fragments)) + refchk->seen = 0; return; + } /* * There are (refchk->rc.rc_refcount - refchk->nr refcount) diff --git a/fs/xfs/scrub/repair.c b/fs/xfs/scrub/repair.c index c2a437416227..956aa75218aa 100644 --- a/fs/xfs/scrub/repair.c +++ b/fs/xfs/scrub/repair.c @@ -1377,12 +1377,14 @@ xrep_reset_metafile_resv( { struct xfs_mount *mp = sc->mp; int64_t delta; - int error; + int error = 0; + + mutex_lock(&mp->m_metafile_resv_lock); delta = mp->m_metafile_resv_used + mp->m_metafile_resv_avail - mp->m_metafile_resv_target; if (delta == 0) - return 0; + goto out_resv_lock; /* * Too many blocks have been reserved, transfer some from the incore @@ -1398,7 +1400,7 @@ xrep_reset_metafile_resv( mp->m_metafile_resv_avail -= give_back; } - return 0; + goto out_resv_lock; } /* @@ -1412,14 +1414,18 @@ xrep_reset_metafile_resv( if (delta == 0) { xfs_warn(sc->mp, "Insufficient free space to reset metabtree reservation after repair."); - return 0; + error = 0; + goto out_resv_lock; } error = xfs_dec_fdblocks(mp, delta, true); } if (error) - return error; + goto out_resv_lock; xfs_mod_sb_delalloc(mp, delta); mp->m_metafile_resv_avail += delta; - return 0; + +out_resv_lock: + mutex_unlock(&mp->m_metafile_resv_lock); + return error; } diff --git a/fs/xfs/scrub/repair.h b/fs/xfs/scrub/repair.h index 2bb125c4f9bf..1af568651e20 100644 --- a/fs/xfs/scrub/repair.h +++ b/fs/xfs/scrub/repair.h @@ -239,7 +239,7 @@ xrep_setup_nothing( #define xrep_setup_rtrmapbt xrep_setup_nothing #define xrep_setup_rtrefcountbt xrep_setup_nothing -#define xrep_setup_inode(sc, imap) ((void)0) +#define xrep_setup_inode(sc, imap) (0) static inline int xrep_setup_symlink(struct xfs_scrub *sc, unsigned int *x) { diff --git a/fs/xfs/scrub/rtbitmap_repair.c b/fs/xfs/scrub/rtbitmap_repair.c index 442a17bf9720..f63aaaff783d 100644 --- a/fs/xfs/scrub/rtbitmap_repair.c +++ b/fs/xfs/scrub/rtbitmap_repair.c @@ -276,7 +276,7 @@ xrep_rtbitmap_mark_free( xrep_wordoff_t rem; xrep_wordcnt_t wordcnt; - wordcnt = min_t(xrep_wordcnt_t, nextwordoff - wordoff, + wordcnt = min_t(xrep_wordoff_t, nextwordoff - wordoff, bufwsize); /* diff --git a/fs/xfs/scrub/rtrefcount.c b/fs/xfs/scrub/rtrefcount.c index 3d916d71a135..460e3f094adf 100644 --- a/fs/xfs/scrub/rtrefcount.c +++ b/fs/xfs/scrub/rtrefcount.c @@ -379,6 +379,8 @@ xchk_rtrefcount_mergeable( if (r1->rc_blockcount == 0) return false; + if (r1->rc_domain != r2->rc_domain) + return false; if (r1->rc_startblock + r1->rc_blockcount != r2->rc_startblock) return false; if (r1->rc_refcount != r2->rc_refcount) diff --git a/fs/xfs/scrub/rtsummary.c b/fs/xfs/scrub/rtsummary.c index 546b335ade13..fa358746beaf 100644 --- a/fs/xfs/scrub/rtsummary.c +++ b/fs/xfs/scrub/rtsummary.c @@ -212,13 +212,20 @@ xchk_rtsum_compute( { struct xfs_mount *mp = sc->mp; struct xfs_rtgroup *rtg = sc->sr.rtg; + struct xchk_rtsummary *rts = sc->buf; + int error; /* If the bitmap size doesn't match the computed size, bail. */ if (XFS_FSB_TO_B(mp, xfs_rtbitmap_blockcount(mp)) != rtg_bitmap(rtg)->i_disk_size) return -EFSCORRUPTED; - return xfs_rtalloc_query_all(rtg, sc->tp, xchk_rtsum_record_free, sc); + error = xfs_rtalloc_query_all(rtg, sc->tp, xchk_rtsum_record_free, sc); + if (error) + return error; + + rts->summary_computed = true; + return 0; } /* Compare the rtsummary file against the one we computed. */ diff --git a/fs/xfs/scrub/rtsummary.h b/fs/xfs/scrub/rtsummary.h index e44b04cb6e2d..af1ab64c3f21 100644 --- a/fs/xfs/scrub/rtsummary.h +++ b/fs/xfs/scrub/rtsummary.h @@ -17,6 +17,7 @@ struct xchk_rtsummary { xfs_filblks_t rsumblocks; unsigned int rsumlevels; unsigned int resblks; + bool summary_computed; /* suminfo position of xfile as we write buffers to disk. */ xfs_rtsumoff_t prep_wordoff; diff --git a/fs/xfs/scrub/rtsummary_repair.c b/fs/xfs/scrub/rtsummary_repair.c index ed763290aec1..45bdefaeab01 100644 --- a/fs/xfs/scrub/rtsummary_repair.c +++ b/fs/xfs/scrub/rtsummary_repair.c @@ -122,7 +122,7 @@ xrep_rtsummary( return -EOPNOTSUPP; /* Walk away if we disagree on the size of the rt bitmap. */ - if (rts->rbmblocks != mp->m_sb.sb_rbmblocks) + if (rts->rbmblocks != mp->m_sb.sb_rbmblocks || !rts->summary_computed) return 0; /* Make sure any problems with the fork are fixed. */ diff --git a/fs/xfs/scrub/stats.c b/fs/xfs/scrub/stats.c index 3339cae4b39d..6934d76cfd15 100644 --- a/fs/xfs/scrub/stats.c +++ b/fs/xfs/scrub/stats.c @@ -168,7 +168,8 @@ xchk_stats_estimate_bufsize( ret += field_width + 1; } - return ret; + /* null byte for scnprintf */ + return ret + 1; } /* Clear all counters. */ @@ -300,6 +301,7 @@ xchk_scrub_stats_read( } static const struct file_operations scrub_stats_fops = { + .owner = THIS_MODULE, .open = simple_open, .read = xchk_scrub_stats_read, }; @@ -327,6 +329,7 @@ xchk_clear_scrub_stats_write( } static const struct file_operations clear_scrub_stats_fops = { + .owner = THIS_MODULE, .open = simple_open, .write = xchk_clear_scrub_stats_write, }; diff --git a/fs/xfs/scrub/symlink.c b/fs/xfs/scrub/symlink.c index 91d40b9fb5c6..ed9ca7f737c3 100644 --- a/fs/xfs/scrub/symlink.c +++ b/fs/xfs/scrub/symlink.c @@ -70,21 +70,20 @@ xchk_symlink( return 0; } - /* Inline symlink? */ if (ifp->if_format == XFS_DINODE_FMT_LOCAL) { + /* Inline symlink? */ if (len > xfs_inode_data_fork_size(ip) || len > strnlen(ifp->if_data, xfs_inode_data_fork_size(ip))) xchk_fblock_set_corrupt(sc, XFS_DATA_FORK, 0); - return 0; + } else { + /* Remote symlink; must read the contents. */ + error = xfs_symlink_remote_read(sc->ip, sc->buf); + if (!xchk_fblock_process_error(sc, XFS_DATA_FORK, 0, &error)) + return error; + if (strnlen(sc->buf, XFS_SYMLINK_MAXLEN) < len) + xchk_fblock_set_corrupt(sc, XFS_DATA_FORK, 0); } - /* Remote symlink; must read the contents. */ - error = xfs_symlink_remote_read(sc->ip, sc->buf); - if (!xchk_fblock_process_error(sc, XFS_DATA_FORK, 0, &error)) - return error; - if (strnlen(sc->buf, XFS_SYMLINK_MAXLEN) < len) - xchk_fblock_set_corrupt(sc, XFS_DATA_FORK, 0); - /* If a remote symlink is clean, it is clearly not zapped. */ xchk_mark_healthy_if_clean(sc, XFS_SICK_INO_SYMLINK_ZAPPED); return 0; diff --git a/fs/xfs/scrub/trace.c b/fs/xfs/scrub/trace.c index e5efc680651f..92a70e4b19d9 100644 --- a/fs/xfs/scrub/trace.c +++ b/fs/xfs/scrub/trace.c @@ -42,6 +42,9 @@ xchk_btree_cur_fsbno( struct xfs_btree_cur *cur, int level) { + if (level < 0) + return NULLFSBLOCK; + if (level < cur->bc_nlevels && cur->bc_levels[level].bp) return XFS_DADDR_TO_FSB(cur->bc_mp, xfs_buf_daddr(cur->bc_levels[level].bp)); @@ -53,6 +56,16 @@ xchk_btree_cur_fsbno( return NULLFSBLOCK; } +static inline int +xchk_btree_cur_ptr( + const struct xfs_btree_cur *cur, + int level) +{ + if (level >= 0 && level < cur->bc_nlevels) + return cur->bc_levels[level].ptr; + return -1; +} + /* * We include this last to have the helpers above available for the trace * event implementations. diff --git a/fs/xfs/scrub/trace.h b/fs/xfs/scrub/trace.h index cb85f75ce101..6b14b149b600 100644 --- a/fs/xfs/scrub/trace.h +++ b/fs/xfs/scrub/trace.h @@ -577,7 +577,7 @@ TRACE_EVENT(xchk_btree_op_error, __entry->level = level; __entry->agno = XFS_FSB_TO_AGNO(cur->bc_mp, fsbno); __entry->bno = XFS_FSB_TO_AGBNO(cur->bc_mp, fsbno); - __entry->ptr = cur->bc_levels[level].ptr; + __entry->ptr = xchk_btree_cur_ptr(cur, level); __entry->error = error; __entry->ret_ip = ret_ip; ), @@ -618,7 +618,7 @@ TRACE_EVENT(xchk_ifork_btree_op_error, __entry->type = sc->sm->sm_type; __assign_str(name); __entry->level = level; - __entry->ptr = cur->bc_levels[level].ptr; + __entry->ptr = xchk_btree_cur_ptr(cur, level); __entry->agno = XFS_FSB_TO_AGNO(cur->bc_mp, fsbno); __entry->bno = XFS_FSB_TO_AGBNO(cur->bc_mp, fsbno); __entry->error = error; @@ -660,7 +660,7 @@ TRACE_EVENT(xchk_btree_error, __entry->level = level; __entry->agno = XFS_FSB_TO_AGNO(cur->bc_mp, fsbno); __entry->bno = XFS_FSB_TO_AGBNO(cur->bc_mp, fsbno); - __entry->ptr = cur->bc_levels[level].ptr; + __entry->ptr = xchk_btree_cur_ptr(cur, level); __entry->ret_ip = ret_ip; ), TP_printk("dev %d:%d type %s %sbt level %d ptr %d agno 0x%x agbno 0x%x ret_ip %pS", @@ -700,7 +700,7 @@ TRACE_EVENT(xchk_ifork_btree_error, __entry->level = level; __entry->agno = XFS_FSB_TO_AGNO(cur->bc_mp, fsbno); __entry->bno = XFS_FSB_TO_AGBNO(cur->bc_mp, fsbno); - __entry->ptr = cur->bc_levels[level].ptr; + __entry->ptr = xchk_btree_cur_ptr(cur, level); __entry->ret_ip = ret_ip; ), TP_printk("dev %d:%d ino 0x%llx fork %s type %s %sbt level %d ptr %d agno 0x%x agbno 0x%x ret_ip %pS", @@ -740,7 +740,7 @@ DECLARE_EVENT_CLASS(xchk_sbtree_class, __entry->bno = XFS_FSB_TO_AGBNO(cur->bc_mp, fsbno); __entry->level = level; __entry->nlevels = cur->bc_nlevels; - __entry->ptr = cur->bc_levels[level].ptr; + __entry->ptr = xchk_btree_cur_ptr(cur, level); ), TP_printk("dev %d:%d type %s %sbt agno 0x%x agbno 0x%x level %d nlevels %d ptr %d", MAJOR(__entry->dev), MINOR(__entry->dev), diff --git a/fs/xfs/scrub/xfarray.c b/fs/xfs/scrub/xfarray.c index 2ce24bfe4c0f..dc18c6310cbd 100644 --- a/fs/xfs/scrub/xfarray.c +++ b/fs/xfs/scrub/xfarray.c @@ -943,8 +943,10 @@ xfarray_sort( * problems, we're done. */ ret = xfarray_foliosort(si, lo, hi); - if (ret < 0) + if (ret < 0) { + error = ret; goto out_free; + } if (ret == 1) { si->stack_depth--; continue; diff --git a/fs/xfs/xfs_dquot.c b/fs/xfs/xfs_dquot.c index e696ee36c2e8..6d22ead562da 100644 --- a/fs/xfs/xfs_dquot.c +++ b/fs/xfs/xfs_dquot.c @@ -115,18 +115,15 @@ xfs_qm_dqdestroy( * We overwrite the dquot limits only if they are zero and this * is not the root dquot. */ -void +static void xfs_qm_adjust_dqlimits( struct xfs_dquot *dq) { struct xfs_mount *mp = dq->q_mount; struct xfs_quotainfo *q = mp->m_quotainfo; - struct xfs_def_quota *defq; + struct xfs_def_quota *defq = xfs_get_defquota(q, xfs_dquot_type(dq)); int prealloc = 0; - ASSERT(dq->q_id); - defq = xfs_get_defquota(q, xfs_dquot_type(dq)); - if (!dq->q_blk.softlimit) { dq->q_blk.softlimit = defq->blk.soft; prealloc = 1; @@ -223,6 +220,19 @@ xfs_qm_adjust_dqtimers( xfs_qm_adjust_res_timer(dq->q_mount, &dq->q_rtb, &defq->rtb); } +/* Adjust enforcement limits and timers after a change in usage. */ +void +xfs_qm_adjust_dqenforcement( + struct xfs_dquot *dq) +{ + if (dq->q_id == 0) + return; + + xfs_qm_adjust_dqlimits(dq); + xfs_qm_adjust_dqtimers(dq); + dq->q_flags |= XFS_DQFLAG_DIRTY; +} + /* * initialize a buffer full of dquots and log the whole thing */ diff --git a/fs/xfs/xfs_dquot.h b/fs/xfs/xfs_dquot.h index bbb824adca82..28c09704acc2 100644 --- a/fs/xfs/xfs_dquot.h +++ b/fs/xfs/xfs_dquot.h @@ -205,7 +205,7 @@ void xfs_qm_dqdestroy(struct xfs_dquot *dqp); int xfs_qm_dqflush(struct xfs_dquot *dqp, struct xfs_buf *bp); void xfs_qm_dqunpin_wait(struct xfs_dquot *dqp); void xfs_qm_adjust_dqtimers(struct xfs_dquot *d); -void xfs_qm_adjust_dqlimits(struct xfs_dquot *d); +void xfs_qm_adjust_dqenforcement(struct xfs_dquot *d); xfs_dqid_t xfs_qm_id_for_quotatype(struct xfs_inode *ip, xfs_dqtype_t type); int xfs_qm_dqget(struct xfs_mount *mp, xfs_dqid_t id, diff --git a/fs/xfs/xfs_handle.c b/fs/xfs/xfs_handle.c index 0689cade8f74..fd9d4d8258ff 100644 --- a/fs/xfs/xfs_handle.c +++ b/fs/xfs/xfs_handle.c @@ -409,7 +409,7 @@ xfs_ioc_attr_list( void *buffer; int error; - if (bufsize < sizeof(struct xfs_attrlist) || + if (bufsize < struct_size(alist, al_offset, 1) || bufsize > XFS_XATTR_LIST_MAX) return -EINVAL; diff --git a/fs/xfs/xfs_inode_item_recover.c b/fs/xfs/xfs_inode_item_recover.c index 1d2319ad15c5..8da95131ab55 100644 --- a/fs/xfs/xfs_inode_item_recover.c +++ b/fs/xfs/xfs_inode_item_recover.c @@ -507,6 +507,25 @@ xlog_recover_inode_commit_pass2( ASSERT(!(fields & XFS_ILOG_DFORK) || (len == xlog_calc_iovec_len(in_f->ilf_dsize))); + /* + * The recovered inode is verified only after the fork data has been + * copied into it, so bound each logged fork region against the size of + * its fork now, before the memcpy below can overrun the on-disk inode. + * The DBROOT/ABROOT cases already bound their copies against the fork + * size. + */ + if ((fields & (XFS_ILOG_DDATA | XFS_ILOG_DEXT)) && + item->ri_buf[2].iov_len > XFS_DFORK_DSIZE(dip, mp)) { + error = -EFSCORRUPTED; + goto out_release; + } + if ((fields & (XFS_ILOG_ADATA | XFS_ILOG_AEXT)) && + item->ri_buf[(fields & XFS_ILOG_DFORK) ? 3 : 2].iov_len > + XFS_DFORK_ASIZE(dip, mp)) { + error = -EFSCORRUPTED; + goto out_release; + } + switch (fields & XFS_ILOG_DFORK) { case XFS_ILOG_DDATA: case XFS_ILOG_DEXT: @@ -546,7 +565,6 @@ xlog_recover_inode_commit_pass2( case XFS_ILOG_ADATA: case XFS_ILOG_AEXT: dest = XFS_DFORK_APTR(dip); - ASSERT(len <= XFS_DFORK_ASIZE(dip, mp)); memcpy(dest, src, len); break; diff --git a/fs/xfs/xfs_mount.c b/fs/xfs/xfs_mount.c index be90c7b03994..8838eff1f054 100644 --- a/fs/xfs/xfs_mount.c +++ b/fs/xfs/xfs_mount.c @@ -446,6 +446,8 @@ int xfs_mount_reset_sbqflags( struct xfs_mount *mp) { + int error; + mp->m_qflags = 0; /* It is OK to look at sb_qflags in the mount path without m_sb_lock. */ @@ -458,7 +460,12 @@ xfs_mount_reset_sbqflags( if (!xfs_fs_writable(mp, SB_FREEZE_WRITE)) return 0; - return xfs_sync_sb(mp, false); + error = xfs_sync_sb(mp, false); + if (error) + return error; + + xfs_qm_update_secondary_sb(mp); + return 0; } static const char *const xfs_free_pool_name[] = { diff --git a/fs/xfs/xfs_pnfs.c b/fs/xfs/xfs_pnfs.c index f8535ecde21a..ab385617003a 100644 --- a/fs/xfs/xfs_pnfs.c +++ b/fs/xfs/xfs_pnfs.c @@ -183,13 +183,28 @@ xfs_fs_map_blocks( offset_fsb = XFS_B_TO_FSBT(mp, offset); lock_flags = xfs_ilock_data_map_shared(ip); - /* request mappings for the specified range only */ + /* + * Map to the end of the extent that covers the start of the range, + * so that a client doing I/O in pieces gets a layout it can use for + * the pieces that follow. Never map anything before the start of + * the range: nfsd calls in here once per extent of a LAYOUTGET, for + * the range that is left after the previous extent, and the mapping + * can change in between, so a mapping that reaches back can overlap + * one already in the layout. Don't extend the mapping past EOF + * beyond the range either: xfs_free_eofblocks() can free blocks past + * EOF without breaking the layout. + */ error = xfs_bmapi_read(ip, offset_fsb, end_fsb - offset_fsb, - &imap, &nimaps, 0); + &imap, &nimaps, XFS_BMAPI_ENTIRE); if (error) { xfs_iunlock(ip, lock_flags); goto out_unlock; } + if (nimaps) + xfs_trim_extent(&imap, offset_fsb, + max_t(xfs_fileoff_t, end_fsb, + XFS_B_TO_FSB(mp, XFS_ISIZE(ip))) - + offset_fsb); seq = xfs_iomap_inode_sequence(ip, 0); ASSERT(!nimaps || imap.br_startblock != DELAYSTARTBLOCK); diff --git a/fs/xfs/xfs_qm.c b/fs/xfs/xfs_qm.c index 54d00d543b51..dd5011f578a6 100644 --- a/fs/xfs/xfs_qm.c +++ b/fs/xfs/xfs_qm.c @@ -693,6 +693,32 @@ xfs_qm_create_metadir_qinos( return 0; } +/* + * For metadir filesystems, quota flag state persists across mounts, so update + * secondary superblocks in case xfs_repair ever has to recover the primary + * from a secondary. growfs also updates backup supers so lock against that. + * Note that the secondary backups are opportunistic, so we log but do not + * return write errors. + */ +void +__xfs_qm_update_secondary_sb( + struct xfs_mount *mp, + const char *func) +{ + int error; + + if (!xfs_has_metadir(mp)) + return; + + mutex_lock(&mp->m_growlock); + error = xfs_update_secondary_sbs(mp); + mutex_unlock(&mp->m_growlock); + + if (error) + xfs_alert(mp, "%s: Secondary super update failed, err=%d!", + func, error); +} + /* * Add QUOTABIT to sb_versionnum and initialize qflags in preparation for * creating quota files on a metadir filesystem. @@ -718,7 +744,12 @@ xfs_qm_prep_metadir_sb( spin_unlock(&mp->m_sb_lock); xfs_log_sb(tp); - return xfs_trans_commit(tp); + error = xfs_trans_commit(tp); + if (error) + return error; + + xfs_qm_update_secondary_sb(mp); + return 0; } /* @@ -1294,11 +1325,7 @@ xfs_qm_quotacheck_dqadjust( * * There are no timers for the default values set in the root dquot. */ - if (dqp->q_id) { - xfs_qm_adjust_dqlimits(dqp); - xfs_qm_adjust_dqtimers(dqp); - } - + xfs_qm_adjust_dqenforcement(dqp); dqp->q_flags |= XFS_DQFLAG_DIRTY; out_unlock: mutex_unlock(&dqp->q_qlock); @@ -1687,6 +1714,9 @@ xfs_qm_mount_quotas( ASSERT(!(XFS_IS_QUOTA_ON(mp))); xfs_alert(mp, "%s: Superblock update failed!", __func__); + } else { + /* Primary update succeeded, try to write secondary */ + xfs_qm_update_secondary_sb(mp); } } diff --git a/fs/xfs/xfs_quota.h b/fs/xfs/xfs_quota.h index 105e6eb57620..b323a72dafbc 100644 --- a/fs/xfs/xfs_quota.h +++ b/fs/xfs/xfs_quota.h @@ -151,6 +151,8 @@ void xfs_dqtrx_hook_setup(struct xfs_dqtrx_hook *hook, notifier_fn_t mod_fn, xfs_trans_mod_dquot((tp), (dqp), (field), (delta)) # endif /* CONFIG_XFS_LIVE_HOOKS */ +void __xfs_qm_update_secondary_sb(struct xfs_mount *mp, + const char *func); #else static inline int xfs_qm_vop_dqalloc(struct xfs_inode *ip, kuid_t kuid, kgid_t kgid, @@ -216,8 +218,15 @@ static inline int xfs_quota_reserve_blkres(struct xfs_inode *ip, int64_t blocks) # define xfs_dqtrx_hook_disable() ((void)0) # endif /* CONFIG_XFS_LIVE_HOOKS */ +static inline void __xfs_qm_update_secondary_sb(struct xfs_mount *mp, + const char *func) +{ +} #endif /* CONFIG_XFS_QUOTA */ +#define xfs_qm_update_secondary_sb(mp) \ + __xfs_qm_update_secondary_sb((mp), __func__) + static inline void xfs_quota_unreserve_blkres(struct xfs_inode *ip, uint64_t blocks) { diff --git a/fs/xfs/xfs_trans_dquot.c b/fs/xfs/xfs_trans_dquot.c index 1606c614f205..93ec876792cc 100644 --- a/fs/xfs/xfs_trans_dquot.c +++ b/fs/xfs/xfs_trans_dquot.c @@ -566,11 +566,7 @@ xfs_trans_apply_dquot_deltas( * Get any default limits in use. * Start/reset the timer(s) if needed. */ - if (dqp->q_id) { - xfs_qm_adjust_dqlimits(dqp); - xfs_qm_adjust_dqtimers(dqp); - } - + xfs_qm_adjust_dqenforcement(dqp); dqp->q_flags |= XFS_DQFLAG_DIRTY; /* * add this to the list of items to get logged diff --git a/include/linux/device.h b/include/linux/device.h index aee79fd6b32b..ae05205742c0 100644 --- a/include/linux/device.h +++ b/include/linux/device.h @@ -83,6 +83,10 @@ int subsys_virtual_register(const struct bus_type *subsys, * information, equivalent to the kobj_type of a kobject. * If "name" is specified, the uevent will contain it in * the DEVTYPE variable. + * + * @need_parent_lock: When probing or removing a device of this type, + * driver core should lock the device's parent. Note that + * the device must hold a reference to its parent. */ struct device_type { const char *name; @@ -93,6 +97,7 @@ struct device_type { void (*release)(struct device *dev); const struct dev_pm_ops *pm; + bool need_parent_lock; }; /** diff --git a/include/linux/device/bus.h b/include/linux/device/bus.h index a38f7229b8f4..526407a371e2 100644 --- a/include/linux/device/bus.h +++ b/include/linux/device/bus.h @@ -68,8 +68,6 @@ struct fwnode_handle; * @driver_override: Set to true if this bus supports the driver_override * mechanism, which allows userspace to force a specific * driver to bind to a device via a sysfs attribute. - * @need_parent_lock: When probing or removing a device on this bus, the - * device core should lock the device's parent. * * A bus is a channel between the processor and one or more devices. For the * purposes of the device model, all devices are connected via a bus, even if @@ -110,7 +108,6 @@ struct bus_type { const struct dev_pm_ops *pm; bool driver_override; - bool need_parent_lock; }; int __must_check bus_register(const struct bus_type *bus); diff --git a/include/linux/filelock.h b/include/linux/filelock.h index ec11cc6b4c58..e909db319067 100644 --- a/include/linux/filelock.h +++ b/include/linux/filelock.h @@ -530,8 +530,8 @@ static inline bool is_delegated(struct delegated_inode *di) * * Break delegations in a non-blocking fashion. If there are * outstanding delegations and @di is set, then an extra reference - * will be taken on @inode and @di->di_inode will be populated so - * that it may be waited upon. + * will be taken on @inode and its superblock, and @di->di_inode will + * be populated so that it may be waited upon. * * Returns 0 if there is no need to wait or an error. If -EWOULDBLOCK * is returned, then @di will be populated (if non-NULL). @@ -545,16 +545,21 @@ static inline int try_break_deleg(struct inode *inode, unsigned int flags, if (ret == -EWOULDBLOCK && di) { di->di_inode = inode; ihold(inode); + /* The caller may drop its mount before break_deleg_wait(). */ + atomic_inc(&inode->i_sb->s_active); } return ret; } static inline int break_deleg_wait(struct delegated_inode *di) { + struct inode *inode = di->di_inode; + struct super_block *sb = inode->i_sb; int ret; - ret = break_deleg(di->di_inode, 0); - iput(di->di_inode); + ret = break_deleg(inode, 0); + iput(inode); + deactivate_super(sb); di->di_inode = NULL; return ret; } diff --git a/include/linux/fsverity.h b/include/linux/fsverity.h index 6c467ded9751..a9a45f3f85dc 100644 --- a/include/linux/fsverity.h +++ b/include/linux/fsverity.h @@ -54,11 +54,13 @@ struct fsverity_operations { * @merkle_tree_size: total bytes the Merkle tree took up * * If desc == NULL, then enabling verity failed and the filesystem only - * must do any necessary cleanups. Else, it must also store the given - * verity descriptor to a fs-specific location associated with the inode - * and do any fs-specific actions needed to mark the inode as a verity - * inode, e.g. setting a bit in the on-disk inode. The filesystem is - * also responsible for setting the S_VERITY flag in the VFS inode. + * must do any necessary cleanups. + * + * Else, the filesystem must store the given verity descriptor to a + * fs-specific location associated with the inode and do any fs-specific + * actions needed to mark the inode as a verity inode, e.g. set a bit in + * the on-disk inode. Finally, on success (only when rollback is no + * longer possible!) it must set S_VERITY in the VFS inode. * * i_rwsem is held for write, but it may have been dropped between * ->begin_enable_verity() and ->end_enable_verity(). diff --git a/include/linux/hiddev.h b/include/linux/hiddev.h index 2164c03d2c72..8e9f8a33e359 100644 --- a/include/linux/hiddev.h +++ b/include/linux/hiddev.h @@ -13,6 +13,7 @@ #ifndef _HIDDEV_H #define _HIDDEV_H +#include #include @@ -24,6 +25,7 @@ struct hiddev { int minor; int exist; int open; + refcount_t refcount; struct mutex existancelock; wait_queue_head_t wait; struct hid_device *hid; diff --git a/include/linux/interrupt.h b/include/linux/interrupt.h index 3bf969ad8fe0..50745608dc15 100644 --- a/include/linux/interrupt.h +++ b/include/linux/interrupt.h @@ -119,6 +119,7 @@ typedef irqreturn_t (*irq_handler_t)(int, void *); * @thread_flags: flags related to @thread * @thread_mask: bitmask for keeping track of @thread activity * @dir: pointer to the proc/irq/NN/name entry + * @thread_dtor: task_work for @thread irregular exit cleanup */ struct irqaction { irq_handler_t handler; @@ -137,6 +138,7 @@ struct irqaction { unsigned long thread_mask; const char *name; struct proc_dir_entry *dir; + struct callback_head thread_dtor; } ____cacheline_internodealigned_in_smp; extern irqreturn_t no_action(int cpl, void *dev_id); diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index e90e5b5dd70a..1c85ce0e471d 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -161,6 +161,9 @@ struct io_uring_task { /* drains ->task_list once the task can no longer run task_work */ struct work_struct fallback_work; + /* the tctx is freed after a grace period, see io_put_task() */ + struct rcu_head rcu; + struct { /* task_work */ struct mpscq task_list; struct callback_head task_work; @@ -519,6 +522,9 @@ struct io_ring_ctx { u32 iowq_limits[2]; struct callback_head poll_wq_task_work; + /* deferred ->poll_wq wake up for local task_work */ + struct rcu_head poll_wq_rcu; + atomic_t poll_wq_rcu_pending; struct list_head defer_list; unsigned nr_drained; /* drain request dispatched from ->defer_list, until it completes */ diff --git a/include/linux/kvm_host.h b/include/linux/kvm_host.h index c844def6dd95..375c2a3b12c5 100644 --- a/include/linux/kvm_host.h +++ b/include/linux/kvm_host.h @@ -854,6 +854,8 @@ struct kvm { gfn_t mmu_invalidate_range_start; gfn_t mmu_invalidate_range_end; + unsigned long gpc_invalidate_seq; + struct list_head devices; u64 manual_dirty_log_protect; struct dentry *debugfs_dentry; diff --git a/include/linux/pps_gen_kernel.h b/include/linux/pps_gen_kernel.h index 6214c8aa2e02..c4d22b1af118 100644 --- a/include/linux/pps_gen_kernel.h +++ b/include/linux/pps_gen_kernel.h @@ -11,6 +11,7 @@ #include #include #include +#include /* * Global defines @@ -44,6 +45,7 @@ struct pps_gen_source_info { /* The main struct */ struct pps_gen_device { const struct pps_gen_source_info *info; /* PSS generator info */ + struct mutex info_lock; /* protects info */ bool enabled; /* PSS generator status */ unsigned int event; @@ -54,7 +56,7 @@ struct pps_gen_device { unsigned int id; /* PPS generator unique ID */ struct cdev cdev; - struct device *dev; + struct device dev; struct fasync_struct *async_queue; /* fasync method */ spinlock_t lock; }; diff --git a/include/net/af_rxrpc.h b/include/net/af_rxrpc.h index 0fb4c41c9bbf..f3980348ed34 100644 --- a/include/net/af_rxrpc.h +++ b/include/net/af_rxrpc.h @@ -64,9 +64,8 @@ struct rxrpc_call *rxrpc_kernel_begin_call(struct socket *sock, bool upgrade, enum rxrpc_interruptibility interruptibility, unsigned int debug_id); -int rxrpc_kernel_send_data(struct socket *, struct rxrpc_call *, - struct msghdr *, size_t, - rxrpc_notify_end_tx_t); +int rxrpc_kernel_send_data(struct socket *sock, struct rxrpc_call *call, + struct msghdr *msg, rxrpc_notify_end_tx_t notify_end_tx); int rxrpc_kernel_recv_data(struct socket *, struct rxrpc_call *, struct iov_iter *, size_t *, bool, u32 *, u16 *); bool rxrpc_kernel_abort_call(struct socket *, struct rxrpc_call *, diff --git a/include/net/mptcp.h b/include/net/mptcp.h index 485d55b66ea6..ad7005b574bb 100644 --- a/include/net/mptcp.h +++ b/include/net/mptcp.h @@ -219,6 +219,9 @@ int mptcp_subflow_init_cookie_req(struct request_sock *req, struct request_sock *mptcp_subflow_reqsk_alloc(const struct request_sock_ops *ops, struct sock *sk_listener, bool attach_listener); +/* Caller must hold an RCU read-side lock. */ +void mptcp_subflow_reqsk_clone(struct request_sock *req, + struct request_sock *new_req); __be32 mptcp_get_reset_option(const struct sk_buff *skb); @@ -305,6 +308,12 @@ static inline struct request_sock *mptcp_subflow_reqsk_alloc(const struct reques return NULL; } +/* Caller must hold an RCU read-side lock. */ +static inline void mptcp_subflow_reqsk_clone(struct request_sock *req, + struct request_sock *new_req) +{ +} + static inline __be32 mptcp_reset_option(const struct sk_buff *skb) { return htonl(0u); } static inline void mptcp_active_detect_blackhole(struct sock *sk, bool expired) { } diff --git a/include/trace/events/huge_memory.h b/include/trace/events/huge_memory.h index 5a48c5406cce..5fb4d92cfd84 100644 --- a/include/trace/events/huge_memory.h +++ b/include/trace/events/huge_memory.h @@ -55,10 +55,10 @@ SCAN_STATUS TRACE_EVENT(mm_khugepaged_scan_pmd, - TP_PROTO(struct mm_struct *mm, struct folio *folio, + TP_PROTO(struct mm_struct *mm, unsigned long pfn, int referenced, int none_or_zero, int status, int unmapped), - TP_ARGS(mm, folio, referenced, none_or_zero, status, unmapped), + TP_ARGS(mm, pfn, referenced, none_or_zero, status, unmapped), TP_STRUCT__entry( __field(struct mm_struct *, mm) @@ -71,7 +71,7 @@ TRACE_EVENT(mm_khugepaged_scan_pmd, TP_fast_assign( __entry->mm = mm; - __entry->pfn = folio ? folio_pfn(folio) : -1; + __entry->pfn = pfn; __entry->referenced = referenced; __entry->none_or_zero = none_or_zero; __entry->status = status; @@ -178,10 +178,10 @@ TRACE_EVENT(mm_collapse_huge_page_swapin, TRACE_EVENT(mm_khugepaged_scan_file, - TP_PROTO(struct mm_struct *mm, struct folio *folio, struct file *file, + TP_PROTO(struct mm_struct *mm, unsigned long pfn, struct file *file, int present, int swap, int result), - TP_ARGS(mm, folio, file, present, swap, result), + TP_ARGS(mm, pfn, file, present, swap, result), TP_STRUCT__entry( __field(struct mm_struct *, mm) @@ -194,7 +194,7 @@ TRACE_EVENT(mm_khugepaged_scan_file, TP_fast_assign( __entry->mm = mm; - __entry->pfn = folio ? folio_pfn(folio) : -1; + __entry->pfn = pfn; __assign_str(filename); __entry->present = present; __entry->swap = swap; @@ -211,10 +211,10 @@ TRACE_EVENT(mm_khugepaged_scan_file, ); TRACE_EVENT(mm_khugepaged_collapse_file, - TP_PROTO(struct mm_struct *mm, struct folio *new_folio, pgoff_t index, + TP_PROTO(struct mm_struct *mm, unsigned long new_pfn, pgoff_t index, unsigned long addr, bool is_shmem, struct file *file, int nr, int result), - TP_ARGS(mm, new_folio, index, addr, is_shmem, file, nr, result), + TP_ARGS(mm, new_pfn, index, addr, is_shmem, file, nr, result), TP_STRUCT__entry( __field(struct mm_struct *, mm) __field(unsigned long, hpfn) @@ -228,7 +228,7 @@ TRACE_EVENT(mm_khugepaged_collapse_file, TP_fast_assign( __entry->mm = mm; - __entry->hpfn = new_folio ? folio_pfn(new_folio) : -1; + __entry->hpfn = new_pfn; __entry->index = index; __entry->addr = addr; __entry->is_shmem = is_shmem; diff --git a/include/trace/events/rxrpc.h b/include/trace/events/rxrpc.h index 704a10de6670..554dfb777b93 100644 --- a/include/trace/events/rxrpc.h +++ b/include/trace/events/rxrpc.h @@ -20,10 +20,10 @@ /* AFS errors */ \ EM(afs_abort_general_error, "afs-error") \ EM(afs_abort_interrupted, "afs-intr") \ - EM(afs_abort_oom, "afs-oom") \ EM(afs_abort_op_not_supported, "afs-op-notsupp") \ EM(afs_abort_probeuuid_negative, "afs-probeuuid-neg") \ EM(afs_abort_send_data_error, "afs-send-data") \ + EM(afs_abort_send_error, "afs-send-error") \ EM(afs_abort_unmarshal_error, "afs-unmarshal") \ EM(afs_abort_unsupported_sec_class, "afs-unsup-sec-class") \ /* rxperf errors */ \ diff --git a/include/trace/misc/nfs.h b/include/trace/misc/nfs.h index a394b4d38e18..f796db58c5a2 100644 --- a/include/trace/misc/nfs.h +++ b/include/trace/misc/nfs.h @@ -358,6 +358,16 @@ TRACE_DEFINE_ENUM(IOMODE_ANY); { IOMODE_RW, "RW" }, \ { IOMODE_ANY, "ANY" }) +TRACE_DEFINE_ENUM(RCA4_TYPE_MASK_RDATA_DLG); +TRACE_DEFINE_ENUM(RCA4_TYPE_MASK_WDATA_DLG); +TRACE_DEFINE_ENUM(RCA4_TYPE_MASK_DIR_DLG); +TRACE_DEFINE_ENUM(RCA4_TYPE_MASK_FILE_LAYOUT); +TRACE_DEFINE_ENUM(RCA4_TYPE_MASK_BLK_LAYOUT); +TRACE_DEFINE_ENUM(RCA4_TYPE_MASK_OBJ_LAYOUT_MIN); +TRACE_DEFINE_ENUM(RCA4_TYPE_MASK_OBJ_LAYOUT_MAX); +TRACE_DEFINE_ENUM(RCA4_TYPE_MASK_OTHER_LAYOUT_MIN); +TRACE_DEFINE_ENUM(RCA4_TYPE_MASK_OTHER_LAYOUT_MAX); + #define show_rca_mask(x) \ __print_flags(x, "|", \ { BIT(RCA4_TYPE_MASK_RDATA_DLG), "RDATA_DLG" }, \ diff --git a/io_uring/bpf-ops.c b/io_uring/bpf-ops.c index cf2bd068e331..6c2a587fb1bf 100644 --- a/io_uring/bpf-ops.c +++ b/io_uring/bpf-ops.c @@ -17,8 +17,22 @@ __bpf_kfunc_start_defs(); __bpf_kfunc int bpf_io_uring_submit_sqes(struct iou_ctx *loop_ctx, u32 nr) { struct io_ring_ctx *ctx = io_loop_demangle_ctx(loop_ctx); + struct bpf_run_ctx *old_run_ctx; + int ret; - return io_submit_sqes(ctx, nr); + /* + * The requests come from the SQ, which user space can write, so they + * are issued for the task and not for the BPF program. Hide the run + * context of the program from them. The users of has_current_bpf_ctx(), + * such as sockopt_lock_sock() and sockopt_capable(), take a context for + * a call from a BPF helper, whose caller holds the socket lock already + * and needs no capability. + */ + old_run_ctx = bpf_set_run_ctx(NULL); + ret = io_submit_sqes(ctx, nr); + bpf_reset_run_ctx(old_run_ctx); + + return ret; } __bpf_kfunc diff --git a/io_uring/cancel.c b/io_uring/cancel.c index 4f706aff14fc..75a21ecfe69c 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -72,7 +72,16 @@ static bool io_cancel_cb(struct io_wq_work *work, void *data) struct io_kiocb *req = container_of(work, struct io_kiocb, work); struct io_cancel_data *cd = data; - return io_cancel_req_match(req, cd); + if (!io_cancel_req_match(req, cd)) + return false; + /* + * Pending work is gone once it has matched. Work that is running may + * park the request where the same cancellation looks next. Don't make + * it skip the request as one that it has dealt with. + */ + if (cd->flags & (IORING_ASYNC_CANCEL_ALL | IORING_ASYNC_CANCEL_ANY)) + req->cancel_seq_set = false; + return true; } static int io_async_cancel_one(struct io_uring_task *tctx, @@ -117,6 +126,8 @@ int io_try_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd, */ if (!ret) return 0; + if (ret == -EALREADY) + cd->running = true; ret = io_poll_cancel(ctx, cd, issue_flags); if (ret != -ENOENT) @@ -180,6 +191,7 @@ static int __io_async_cancel(struct io_cancel_data *cd, struct io_tctx_node *node; int ret, nr = 0; + cd->running = false; do { ret = io_try_cancel(tctx, cd, issue_flags); if (ret == -ENOENT) @@ -195,6 +207,9 @@ static int __io_async_cancel(struct io_cancel_data *cd, mutex_lock(&ctx->tctx_lock); ret = -ENOENT; list_for_each_entry(node, &ctx->tctx_list, ctx_node) { + /* has been done above, don't count what is running there */ + if (all && node->task->io_uring == tctx) + continue; ret = io_async_cancel_one(node->task->io_uring, cd); if (ret != -ENOENT) { if (!all) @@ -204,7 +219,15 @@ static int __io_async_cancel(struct io_cancel_data *cd, } mutex_unlock(&ctx->tctx_lock); io_ring_submit_unlock(ctx, issue_flags); - return all ? nr : ret; + if (all) + return nr; + /* + * The worker may be done with the work that was found running, and + * flagged, above. That isn't the same as not having found it. + */ + if (ret == -ENOENT && cd->running) + ret = -EALREADY; + return ret; } int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags) @@ -308,7 +331,7 @@ int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg) if (ret != -EALREADY) goto out; - if (sc.timeout.tv_sec != -1UL || sc.timeout.tv_nsec != -1UL) { + if (sc.timeout.tv_sec != -1 || sc.timeout.tv_nsec != -1) { struct timespec64 ts = { .tv_sec = sc.timeout.tv_sec, .tv_nsec = sc.timeout.tv_nsec @@ -636,8 +659,18 @@ __cold void io_uring_cancel_generic(bool cancel_all, struct io_sq_data *sqd) io_uring_drop_tctx_refs(current); /* only skip the sleep for local work this task may run */ xa_for_each(&tctx->xa, index, node) { - if (io_local_work_pending(node->ctx) && - io_allowed_defer_tw_run(node->ctx)) + if (!(node->ctx->flags & IORING_SETUP_DEFER_TASKRUN) || + !io_allowed_defer_tw_run(node->ctx)) + continue; + /* + * The wake up io_uring_try_cancel_requests() asked for + * may have been issued while we were still running. Ask + * again now that it can't be missed. + */ + atomic_set(&node->ctx->cq_wait_nr, 1); + /* pairs with the xchg() in mpscq_push() */ + smp_mb(); + if (io_local_work_pending(node->ctx)) goto end_wait; } /* diff --git a/io_uring/cancel.h b/io_uring/cancel.h index 1b201a094303..e03a6fb0b7f9 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -11,6 +11,8 @@ struct io_cancel_data { u8 opcode; u32 flags; int seq; + /* io_try_cancel() has found it being run by io-wq, and flagged it */ + bool running; }; int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); diff --git a/io_uring/fdinfo.c b/io_uring/fdinfo.c index 3ae4804765b9..30027fd46045 100644 --- a/io_uring/fdinfo.c +++ b/io_uring/fdinfo.c @@ -154,11 +154,17 @@ static void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, struct seq_file *m) cq_entries = min(cq_tail - cq_head, ctx->cq_entries); for (i = 0; i < cq_entries; i++) { struct io_uring_cqe *cqe; - bool cqe32 = false; + bool cqe32 = false, two_entries = false; - cqe = &r->cqes[(cq_head & cq_mask)]; - if (cqe->flags & IORING_CQE_F_32 || ctx->flags & IORING_SETUP_CQE32) + if (ctx->flags & IORING_SETUP_CQE32) { + /* all entries have twice the size */ + cqe = &r->cqes[(cq_head & cq_mask) << 1]; cqe32 = true; + } else { + cqe = &r->cqes[(cq_head & cq_mask)]; + two_entries = cqe->flags & IORING_CQE_F_32; + cqe32 = two_entries; + } seq_printf(m, "%5u: user_data:%llu, res:%d, flags:%x", cq_head & cq_mask, cqe->user_data, cqe->res, cqe->flags); @@ -167,7 +173,7 @@ static void __io_uring_show_fdinfo(struct io_ring_ctx *ctx, struct seq_file *m) cqe->big_cqe[0], cqe->big_cqe[1]); seq_printf(m, "\n"); cq_head++; - if (cqe32) { + if (two_entries) { cq_head++; i++; } diff --git a/io_uring/futex.c b/io_uring/futex.c index eaee14242a3a..9203998654ff 100644 --- a/io_uring/futex.c +++ b/io_uring/futex.c @@ -152,6 +152,24 @@ int io_futex_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return 0; } +/* + * A wait on a private futex does not block the submitter. Even a single + * threaded task can go on and have the mm private hash installed by clone() + * or prctl(), which does not pick up waiters from the global hash. Make + * sure the private hash exists before the first waiter is queued. + */ +static int io_futex_prep_private(struct io_kiocb *req) +{ + int ret = futex_hash_allocate_default(); + + if (unlikely(ret)) + return ret; + + /* inflight tracking only needed for mm private hash */ + io_req_track_inflight(req); + return 0; +} + int io_futex_wait_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) { struct io_futex *iof = io_kiocb_to_cmd(req, struct io_futex); @@ -161,9 +179,8 @@ int io_futex_wait_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) if (unlikely(ret)) return ret; - /* inflight tracking only needed for mm private hash */ if (!(iof->futex_flags & FLAGS_SHARED)) - io_req_track_inflight(req); + return io_futex_prep_private(req); return 0; } @@ -213,10 +230,13 @@ int io_futexv_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) return ret; } - /* inflight tracking only needed for mm private hash */ for (i = 0; i < iof->futex_nr; i++) { if (!(ifd->futexv[i].w.flags & FLAGS_SHARED)) { - io_req_track_inflight(req); + ret = io_futex_prep_private(req); + if (ret) { + kfree(ifd); + return ret; + } break; } } @@ -283,6 +303,8 @@ int io_futexv_wait(struct io_kiocb *req, unsigned int issue_flags) */ __set_current_state(TASK_RUNNING); hlist_add_head(&req->hash_node, &ctx->futex_list); + if (io_wq_cancel_seen(req, issue_flags)) + __io_futex_cancel(req); } else { iof->futexv_unqueued = 1; if (woken != -1) @@ -323,6 +345,8 @@ int io_futex_wait(struct io_kiocb *req, unsigned int issue_flags) &ifd->q, NULL, NULL); if (!ret) { hlist_add_head(&req->hash_node, &ctx->futex_list); + if (io_wq_cancel_seen(req, issue_flags)) + __io_futex_cancel(req); io_ring_submit_unlock(ctx, issue_flags); return IOU_ISSUE_SKIP_COMPLETE; diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index 2a980e86dd94..6b9395060262 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -56,6 +56,8 @@ struct io_worker { struct io_wq_acct *acct; struct io_wq_work *cur_work; + /* rest of the hashed chain ->cur_work was taken off the list with */ + struct io_wq_work *next_work; raw_spinlock_t lock; struct completion ref_done; @@ -467,11 +469,9 @@ static void io_wq_dec_running(struct io_worker *worker) if (!atomic_dec_and_test(&acct->nr_running)) return; - if (!worker->cur_work) - return; if (!io_acct_run_queue(acct)) return; - if (io_wq_hash_defer(worker->cur_work, acct)) { + if (worker->cur_work && io_wq_hash_defer(worker->cur_work, acct)) { raw_spin_unlock(&acct->lock); return; } @@ -583,6 +583,17 @@ static struct io_wq_work *io_get_next_work(struct io_wq_acct *acct, return NULL; } +static void __io_assign_current_work(struct io_worker *worker, + struct io_wq_work *work) +{ + raw_spin_lock(&worker->lock); + worker->cur_work = work; + /* stays valid while in between two items of a chain */ + if (work) + worker->next_work = wq_next_work(work); + raw_spin_unlock(&worker->lock); +} + static void io_assign_current_work(struct io_worker *worker, struct io_wq_work *work) { @@ -591,9 +602,7 @@ static void io_assign_current_work(struct io_worker *worker, cond_resched(); } - raw_spin_lock(&worker->lock); - worker->cur_work = work; - raw_spin_unlock(&worker->lock); + __io_assign_current_work(worker, work); } /* @@ -624,9 +633,7 @@ static void io_worker_handle_work(struct io_wq_acct *acct, * work list, but isn't yet discoverable as the * current work item for this worker. */ - raw_spin_lock(&worker->lock); - worker->cur_work = work; - raw_spin_unlock(&worker->lock); + __io_assign_current_work(worker, work); } raw_spin_unlock(&acct->lock); @@ -642,25 +649,32 @@ static void io_worker_handle_work(struct io_wq_acct *acct, /* handle a whole dependent link */ do { bool do_kill = test_bit(IO_WQ_BIT_EXIT, &wq->state); - struct io_wq_work *next_hashed, *linked; + struct io_wq_work *next_hashed, *linked, *done = work; unsigned int work_flags = atomic_read(&work->flags); unsigned int hash = __io_wq_is_hashed(work_flags) ? __io_get_work_hash(work_flags) : -1U; struct io_kiocb *req; + bool last; next_hashed = wq_next_work(work); - if (do_kill && - (work_flags & IO_WQ_WORK_UNBOUND)) + /* + * io_wq_exit_workers() signals us once, so that we + * don't sleep for ever. If that was before this work + * became the current one, the signal is gone. + */ + if (do_kill && (work_flags & IO_WQ_WORK_UNBOUND)) atomic_or(IO_WQ_WORK_CANCEL, &work->flags); + else if (do_kill) + set_notify_signal(current); req = container_of(work, struct io_kiocb, work); kcov_remote_start_common(req->ctx->kcov_handle); io_wq_submit_work(work); kcov_remote_stop(); io_assign_current_work(worker, NULL); - linked = io_wq_free_work(work); + last = io_wq_put_work(work, &linked); work = next_hashed; if (!work && linked && !io_wq_is_hashed(linked)) { work = linked; @@ -669,6 +683,13 @@ static void io_worker_handle_work(struct io_wq_acct *acct, io_assign_current_work(worker, work); if (linked) io_wq_enqueue(wq, linked); + /* + * Not before the link is current or pending work again. + * This may be the last time that a task is woken up + * which exits and looks for requests to cancel. + */ + if (last) + io_wq_free_work(done); if (hash != -1U && !next_hashed) { /* serialize hash clear with wake_up() */ @@ -687,6 +708,35 @@ static void io_worker_handle_work(struct io_wq_acct *acct, } while (1); } +/* + * The worker has decided to exit, and has given up its ->nr_workers slot and + * its place on the free list. io_wq_enqueue() may have picked it from that + * list, or seen it running, after it last looked at the work list. Leave + * ->nr_running as well and check again. Returns false if the worker has to + * stay. + */ +static bool io_worker_retire(struct io_wq_acct *acct, struct io_worker *worker) +{ + /* turns io_wq_dec_running() and the scheduler hooks into no-ops */ + clear_bit(IO_WORKER_F_UP, &worker->flags); + atomic_dec(&acct->nr_running); + if (!io_acct_run_queue(acct)) + return true; + raw_spin_unlock(&acct->lock); + + raw_spin_lock(&acct->workers_lock); + if (acct->nr_workers >= acct->max_workers) { + /* a worker was created in our place, the work is its business */ + raw_spin_unlock(&acct->workers_lock); + return true; + } + acct->nr_workers++; + raw_spin_unlock(&acct->workers_lock); + atomic_inc(&acct->nr_running); + set_bit(IO_WORKER_F_UP, &worker->flags); + return false; +} + static int io_wq_worker(void *data) { struct io_worker *worker = data; @@ -720,19 +770,41 @@ static int io_wq_worker(void *data) * idle-exit, drop the worker as well. This is used to avoid * keeping io-wq workers around for tasks that no longer have * any active io_uring instances. + * Not as long as a request of ours for another worker is + * pending, though. It is accounted as a running worker, which + * others rely on, and io_worker_exit() would cancel it. */ - if ((last_timeout && (exit_mask || acct->nr_workers > 1)) || - test_bit(IO_WQ_BIT_EXIT_ON_IDLE, &wq->state)) { + if (((last_timeout && (exit_mask || acct->nr_workers > 1)) || + test_bit(IO_WQ_BIT_EXIT_ON_IDLE, &wq->state)) && + !test_bit(0, &worker->create_state)) { acct->nr_workers--; + if (test_and_clear_bit(IO_WORKER_F_FREE, &worker->flags)) + hlist_nulls_del_init_rcu(&worker->nulls_node); raw_spin_unlock(&acct->workers_lock); __set_current_state(TASK_RUNNING); - break; + if (io_worker_retire(acct, worker)) + break; + last_timeout = false; + continue; } last_timeout = false; __io_worker_idle(acct, worker); raw_spin_unlock(&acct->workers_lock); if (io_run_task_work()) continue; + /* + * io_wq_enqueue() leaves new work to us for as long as we are + * accounted as running. Get out of ->nr_running before taking + * the last look at the work list, rather than when scheduling + * out. No need to ask for another worker then, we can do it. + */ + clear_bit(IO_WORKER_F_RUNNING, &worker->flags); + atomic_dec(&acct->nr_running); + if (io_acct_run_queue(acct)) { + io_wq_worker_running(current); + io_worker_handle_work(acct, worker); + continue; + } ret = schedule_timeout(WORKER_IDLE_TIMEOUT); if (signal_pending(current)) { struct ksignal ksig; @@ -998,9 +1070,13 @@ void io_wq_set_exit_on_idle(struct io_wq *wq, bool enable) static void io_run_cancel(struct io_wq_work *work, struct io_wq *wq) { do { + struct io_wq_work *linked; + atomic_or(IO_WQ_WORK_CANCEL, &work->flags); io_wq_submit_work(work); - work = io_wq_free_work(work); + if (io_wq_put_work(work, &linked)) + io_wq_free_work(work); + work = linked; } while (work); } @@ -1108,6 +1184,7 @@ static bool __io_wq_worker_cancel(struct io_worker *worker, static bool io_wq_worker_cancel(struct io_worker *worker, void *data) { struct io_cb_cancel_data *match = data; + struct io_wq_work *work; /* * Hold the lock to avoid ->cur_work going out of scope, caller @@ -1116,6 +1193,19 @@ static bool io_wq_worker_cancel(struct io_worker *worker, void *data) raw_spin_lock(&worker->lock); if (__io_wq_worker_cancel(worker, match, worker->cur_work)) match->nr_running++; + /* + * Hashed work is taken off the work list a chain at a time, what the + * worker hasn't started yet can only be found here. It doesn't move on + * without the lock. Just mark the work, it fails once it is its turn. + */ + for (work = worker->next_work; work; work = wq_next_work(work)) { + if (match->nr_running && !match->cancel_all) + break; + if (match->fn(work, match->data)) { + atomic_or(IO_WQ_WORK_CANCEL, &work->flags); + match->nr_running++; + } + } raw_spin_unlock(&worker->lock); return match->nr_running && !match->cancel_all; @@ -1326,6 +1416,12 @@ void io_wq_exit_start(struct io_wq *wq) set_bit(IO_WQ_BIT_EXIT, &wq->state); /* Pairs with task_work_add() in io_queue_worker_create(). */ smp_mb__after_atomic(); + /* + * No more workers from here on. The caller is going to run its task + * work, and on exec a thread that is created now would still be + * around, exiting, by the time that the caller must be on its own. + */ + io_wq_cancel_tw_create(wq); } static void io_wq_cancel_tw_create(struct io_wq *wq) @@ -1334,14 +1430,16 @@ static void io_wq_cancel_tw_create(struct io_wq *wq) while ((cb = task_work_cancel_match(wq->task, io_task_work_match, wq)) != NULL) { struct io_worker *worker; + bool free_worker; worker = container_of(cb, struct io_worker, create_work); - io_worker_cancel_cb(worker); /* * Only the worker continuation helper has worker allocated and - * hence needs freeing. + * hence needs freeing. Save this before dropping its reference. */ - if (cb->func == create_worker_cont) + free_worker = cb->func == create_worker_cont; + io_worker_cancel_cb(worker); + if (free_worker) kfree(worker); } } diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index 91178f2fca09..41a03979cb8b 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -271,7 +271,7 @@ static __cold struct io_ring_ctx *io_ring_ctx_alloc(struct io_uring_params *p) if (ret) goto free_ref; init_completion(&ctx->ref_comp); - xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1); + xa_init_flags(&ctx->personalities, XA_FLAGS_ALLOC1 | XA_FLAGS_ACCOUNT); mutex_init(&ctx->uring_lock); init_waitqueue_head(&ctx->cq_wait); init_waitqueue_head(&ctx->poll_wq); @@ -407,16 +407,63 @@ static void io_prep_async_link(struct io_kiocb *req) } } +/* Called for every request that is issued: keep it inline */ +static __always_inline bool io_assign_file(struct io_kiocb *req, + const struct io_issue_def *def, + unsigned int issue_flags) +{ + if (req->file || !def->needs_file) + return true; + + if (req->flags & REQ_F_FIXED_FILE) + req->file = io_file_get_fixed(req, req->cqe.fd, issue_flags); + else + req->file = io_file_get_normal(req, req->cqe.fd); + + return !!req->file; +} + +static void io_req_queue_iowq_tw(struct io_tw_req tw_req, io_tw_token_t tw) +{ + struct io_kiocb *req = tw_req.req; + + io_tw_lock(req->ctx, tw); + if (unlikely(tw.cancel)) + io_req_defer_failed(req, -ECANCELED); + else + io_queue_iowq(req); +} + void io_queue_iowq(struct io_kiocb *req) { struct io_uring_task *tctx = req->tctx; BUG_ON(!tctx); + lockdep_assert_held(&req->ctx->uring_lock); - if ((current->flags & PF_KTHREAD) || !tctx->io_wq) { - io_req_task_queue_fail(req, -ECANCELED); + if (current->flags & PF_KTHREAD) + goto cancel; + + /* Only the submitter can safely access its io-wq during task exit. */ + if (unlikely(current != tctx->task)) { + req->io_task_work.func = io_req_queue_iowq_tw; + io_req_task_work_add(req); return; } + if (!tctx->io_wq) + goto cancel; + + /* + * The file is looked up as late as possible, as it may be the result + * of an earlier request of the link. This one is up now, and what kind + * of work it is depends on the file. If that fails, leave the error + * handling to the worker. + * + * Not for a splice: whatever it writes to, it may wait for its input + * for ever. It has to remain unbound work that isn't hashed. + */ + if (req->opcode != IORING_OP_SPLICE) + io_assign_file(req, &io_issue_defs[req->opcode], 0); /* init ->work of the whole link before punting */ io_prep_async_link(req); @@ -433,6 +480,9 @@ void io_queue_iowq(struct io_kiocb *req) trace_io_uring_queue_async_work(req, io_wq_is_hashed(&req->work)); io_wq_enqueue(tctx->io_wq, &req->work); + return; +cancel: + io_req_task_queue_fail(req, -ECANCELED); } unsigned io_linked_nr(struct io_kiocb *req) @@ -445,9 +495,9 @@ unsigned io_linked_nr(struct io_kiocb *req) return nr; } -static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx) +static __cold noinline bool io_queue_deferred(struct io_ring_ctx *ctx) { - bool drain_seen = false, first = true; + bool drain_seen = false, first = true, drain_queued = false; lockdep_assert_held(&ctx->uring_lock); __io_req_caches_free(ctx); @@ -461,17 +511,21 @@ static __cold noinline void io_queue_deferred(struct io_ring_ctx *ctx) drain_seen |= de->req->flags & REQ_F_IO_DRAIN; if ((drain_seen || first) && ctx->nr_req_allocated != ctx->nr_drained) - return; + return drain_queued; list_del_init(&de->list); ctx->nr_drained -= io_linked_nr(de->req); - /* it runs alone, track its completion unless it's a link */ - if ((de->req->flags & REQ_F_IO_DRAIN) && !de->req->link) - ctx->drain_req = de->req; + if (de->req->flags & REQ_F_IO_DRAIN) { + drain_queued = true; + /* it runs alone, track its completion unless it's a link */ + if (!de->req->link) + ctx->drain_req = de->req; + } io_req_task_queue(de->req); kfree(de); first = false; } + return drain_queued; } void __io_commit_cqring_flush(struct io_ring_ctx *ctx) @@ -601,6 +655,8 @@ static inline void io_put_task(struct io_kiocb *req) if (likely(tctx->task == current)) { tctx->cached_refs++; } else { + guard(rcu)(); + percpu_counter_sub(&tctx->inflight, 1); if (unlikely(atomic_read(&tctx->in_cancel))) wake_up(&tctx->wait); @@ -735,6 +791,13 @@ bool io_cqe_cache_refill(struct io_ring_ctx *ctx, bool overflow, bool cqe32) if (!overflow && (ctx->check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT))) return false; + /* + * A single entry may be left if a 32b CQE is asked for. It must not be + * handed out later on if this fails, as the CQE is going to overflow. + */ + ctx->cqe_cached = NULL; + ctx->cqe_sentinel = NULL; + /* * Post dummy CQE if a 32b CQE is needed and there's only room for a * 16b CQE before the ring wraps. @@ -1191,7 +1254,9 @@ void __io_submit_flush_completions(struct io_ring_ctx *ctx) } if (unlikely(ctx->int_flags & IO_RING_F_DRAIN_ACTIVE)) { - io_queue_deferred(ctx); + /* a drain request started from the list keeps the drain going */ + if (io_queue_deferred(ctx)) + drain_done = false; if (drain_done && list_empty(&ctx->defer_list)) ctx->int_flags &= ~IO_RING_F_DRAIN_ACTIVE; } @@ -1387,20 +1452,6 @@ static __cold void io_drain_req(struct io_kiocb *req) ctx->int_flags &= ~IO_RING_F_DRAIN_ACTIVE; } -static bool io_assign_file(struct io_kiocb *req, const struct io_issue_def *def, - unsigned int issue_flags) -{ - if (req->file || !def->needs_file) - return true; - - if (req->flags & REQ_F_FIXED_FILE) - req->file = io_file_get_fixed(req, req->cqe.fd, issue_flags); - else - req->file = io_file_get_normal(req, req->cqe.fd); - - return !!req->file; -} - #define REQ_ISSUE_SLOW_FLAGS (REQ_F_CREDS | REQ_F_ARM_LTIMEOUT) static inline int __io_issue_sqe(struct io_kiocb *req, @@ -1447,6 +1498,9 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) ret = __io_issue_sqe(req, issue_flags, def); if (ret == IOU_COMPLETE) { + /* not all handlers take care of the internal restart codes */ + if (unlikely(req->cqe.res <= -ERESTARTSYS)) + req->cqe.res = io_fixup_restart_res(req->cqe.res); if (issue_flags & IO_URING_F_COMPLETE_DEFER) io_req_complete_defer(req); else @@ -1461,6 +1515,8 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) if (req->flags & REQ_F_IOPOLL) io_iopoll_req_issued(req, issue_flags); } + if (unlikely(ret <= -ERESTARTSYS)) + ret = io_fixup_restart_res(ret); return ret; } @@ -1483,22 +1539,35 @@ int io_poll_issue(struct io_kiocb *req, io_tw_token_t tw) return ret; } -struct io_wq_work *io_wq_free_work(struct io_wq_work *work) +/* + * io-wq is done with @work. Returns true if it held the last reference, and + * hands out the next request of the link then. The caller has to make sure + * that cancellations can find the latter before it calls io_wq_free_work(). + */ +bool io_wq_put_work(struct io_wq_work *work, struct io_wq_work **linked) { struct io_kiocb *req = container_of(work, struct io_kiocb, work); struct io_kiocb *nxt = NULL; - if (req_ref_put_and_test_atomic(req)) { - if (req->flags & IO_REQ_LINK_FLAGS) { - struct io_ring_ctx *ctx = req->ctx; + *linked = NULL; + if (!req_ref_put_and_test_atomic(req)) + return false; - mutex_lock(&ctx->uring_lock); - nxt = io_req_find_next(req); - mutex_unlock(&ctx->uring_lock); - } - io_free_req(req); + if (req->flags & IO_REQ_LINK_FLAGS) { + struct io_ring_ctx *ctx = req->ctx; + + mutex_lock(&ctx->uring_lock); + nxt = io_req_find_next(req); + mutex_unlock(&ctx->uring_lock); } - return nxt ? &nxt->work : NULL; + if (nxt) + *linked = &nxt->work; + return true; +} + +void io_wq_free_work(struct io_wq_work *work) +{ + io_free_req(container_of(work, struct io_kiocb, work)); } void io_wq_submit_work(struct io_wq_work *work) @@ -1509,7 +1578,7 @@ void io_wq_submit_work(struct io_wq_work *work) bool needs_poll = false; int ret = 0, err = -ECANCELED; - /* one will be dropped by io_wq_free_work() after returning to io-wq */ + /* one will be dropped by io_wq_put_work() after returning to io-wq */ if (!(req->flags & REQ_F_REFCOUNT)) __io_req_set_refcount(req, 2); else @@ -1787,17 +1856,27 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, * A 128b op on a non-128b SQ requires mixed SQE support as * well as 2 contiguous entries. */ - if (!(ctx->flags & IORING_SETUP_SQE_MIXED) || *left < 2 || + if (!(ctx->flags & IORING_SETUP_SQE_MIXED) || (unsigned)(sqe - ctx->sq_sqes) >= ctx->sq_entries - 1) return io_init_fail_req(req, -EINVAL); + if (*left >= 2) { + /* + * A 128b operation on a mixed SQ uses two entries, so + * we have to increment the head and cached refs, and + * decrement what's left. + */ + current->io_uring->cached_refs++; + (*left)--; + } else if (!(ctx->flags & IORING_SETUP_SQPOLL) || + !__io_sqring_entries(ctx)) { + return io_init_fail_req(req, -EINVAL); + } /* - * A 128b operation on a mixed SQ uses two entries, so we have - * to increment the head and cached refs, and decrement what's - * left. + * else: the SQPOLL thread caps the number of entries it submits + * in one go, which is none of the application's business. Take + * the 2nd entry on top. */ - current->io_uring->cached_refs++; ctx->cached_sq_head++; - (*left)--; } if (unlikely(sqe_flags & ~SQE_COMMON_FLAGS)) { @@ -2241,6 +2320,7 @@ static __cold void io_ring_ctx_free(struct io_ring_ctx *ctx) kvfree(ctx->cancel_table.hbs); xa_destroy(&ctx->io_bl_xa); xa_destroy(&ctx->hpage_acct); + xa_destroy(&ctx->personalities); kfree(ctx); } @@ -2326,6 +2406,7 @@ struct io_tctx_exit { struct callback_head task_work; struct completion completion; struct io_ring_ctx *ctx; + bool in_cancel; }; static __cold void io_tctx_exit_cb(struct callback_head *cb) @@ -2340,7 +2421,8 @@ static __cold void io_tctx_exit_cb(struct callback_head *cb) * tctx can be NULL if the queueing of this task_work raced with * work cancelation off the exec path. */ - if (tctx && !atomic_read(&tctx->in_cancel)) + work->in_cancel = tctx && atomic_read(&tctx->in_cancel); + if (tctx && !work->in_cancel) io_uring_del_tctx_node((unsigned long)work->ctx); complete(&work->completion); } @@ -2434,6 +2516,14 @@ static __cold void io_ring_exit_work(struct work_struct *work) * wait is marked as interruptible. */ wait_for_completion_interruptible(&exit.completion); + /* + * The task removes the node when it is done with its + * cancellation. Give it the time for that: it doesn't get back + * to it as long as it finds new task_work each time it has run + * the old one, and we may be one of many who send it. + */ + if (exit.in_cancel) + schedule_timeout_idle(HZ / 20); mutex_lock(&ctx->uring_lock); mutex_lock(&ctx->tctx_lock); } diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index 896aab1ed026..2d6b2158605a 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -204,7 +204,8 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin); __cold void io_iopoll_try_reap_events(struct io_ring_ctx *ctx); void __io_submit_flush_completions(struct io_ring_ctx *ctx); -struct io_wq_work *io_wq_free_work(struct io_wq_work *work); +bool io_wq_put_work(struct io_wq_work *work, struct io_wq_work **linked); +void io_wq_free_work(struct io_wq_work *work); void io_wq_submit_work(struct io_wq_work *work); void io_free_req(struct io_kiocb *req); @@ -349,6 +350,24 @@ static inline void io_req_set_res32(struct io_kiocb *req, s32 res, u32 cflags, req->big_cqe.extra2 = extra2; } +static inline ssize_t io_fixup_restart_res(ssize_t ret) +{ + switch (ret) { + case -ERESTARTSYS: + case -ERESTARTNOINTR: + case -ERESTARTNOHAND: + case -ERESTART_RESTARTBLOCK: + /* + * We can't just restart the syscall, since previously + * submitted sqes may already be in progress. Just fail + * this IO with EINTR. + */ + return -EINTR; + default: + return ret; + } +} + static inline void *io_uring_alloc_async_data(struct io_alloc_cache *cache, struct io_kiocb *req) { @@ -411,6 +430,29 @@ static inline void io_ring_submit_lock(struct io_ring_ctx *ctx, lockdep_assert_held(&ctx->uring_lock); } +/* + * For a handler that is run by io-wq and parks the request on a list where + * cancellations find it, rather than completing it. Whoever tried to cancel + * the request before that found it running. It has flagged the work item, + * has searched the list in vain, and won't come back. To be called with + * the lock the search takes held: if the search came first, the flag is + * visible. For most lists that is ->uring_lock, timeouts use ->timeout_lock. + */ +static inline bool __io_wq_cancel_seen(struct io_kiocb *req, + unsigned int issue_flags) +{ + return (issue_flags & IO_URING_F_IOWQ) && + (atomic_read(&req->work.flags) & IO_WQ_WORK_CANCEL); +} + +static inline bool io_wq_cancel_seen(struct io_kiocb *req, + unsigned int issue_flags) +{ + lockdep_assert_held(&req->ctx->uring_lock); + + return __io_wq_cancel_seen(req, issue_flags); +} + static inline void io_commit_cqring(struct io_ring_ctx *ctx) { /* order cqe stores with ring update */ diff --git a/io_uring/kbuf.c b/io_uring/kbuf.c index 6e2facf34bfa..abf44f5dba26 100644 --- a/io_uring/kbuf.c +++ b/io_uring/kbuf.c @@ -204,7 +204,7 @@ static struct io_br_sel io_ring_buffer_select(struct io_kiocb *req, size_t *len, if (unlikely(tail == head)) return sel; - if (head + 1 == tail) + if ((__u16)(head + 1) == tail) req->flags |= REQ_F_BL_EMPTY; buf = io_ring_head_to_buf(br, head, bl->mask); @@ -326,8 +326,10 @@ static int io_ring_buffers_peek(struct io_kiocb *req, struct buf_sel_arg *arg, arg->out_len += len; arg->max_len -= len; - if (!arg->max_len) + if (!arg->max_len) { + head++; break; + } buf = io_ring_head_to_buf(br, ++head, bl->mask); } while (--nr_iovs); diff --git a/io_uring/loop.c b/io_uring/loop.c index 3d472b7827fa..2340459164a1 100644 --- a/io_uring/loop.c +++ b/io_uring/loop.c @@ -10,12 +10,6 @@ static inline int io_loop_nr_cqes(const struct io_ring_ctx *ctx, return lp->cq_wait_idx - READ_ONCE(ctx->rings->cq.tail); } -static inline void io_loop_wait_start(struct io_ring_ctx *ctx, unsigned nr_wait) -{ - atomic_set(&ctx->cq_wait_nr, nr_wait); - set_current_state(TASK_INTERRUPTIBLE); -} - static inline void io_loop_wait_finish(struct io_ring_ctx *ctx) { __set_current_state(TASK_RUNNING); @@ -25,7 +19,7 @@ static inline void io_loop_wait_finish(struct io_ring_ctx *ctx) static void io_loop_wait(struct io_ring_ctx *ctx, struct iou_loop_params *lp, unsigned nr_wait) { - io_loop_wait_start(ctx, nr_wait); + io_cq_wait_arm(ctx, nr_wait); if (unlikely(io_local_work_pending(ctx) || io_loop_nr_cqes(ctx, lp) <= 0) || diff --git a/io_uring/mpscq.h b/io_uring/mpscq.h index f910526766fd..da9474497cc8 100644 --- a/io_uring/mpscq.h +++ b/io_uring/mpscq.h @@ -25,9 +25,10 @@ * its predecessor. In between, the tail end of the queue is not yet reachable * from the head. mpscq_pop() detects this and returns NULL, while mpscq_empty() * reports false. The consumer must not treat such a NULL as "queue empty" - it - * should retry later. The window is two instructions wide, but a producer can - * be preempted inside it, so the consumer must not spin on it while holding - * resources the producer might need to make progress. + * should retry later. The window is two instructions wide and the producer + * can't be preempted inside it, but it can be interrupted, so the consumer must + * not spin on it while holding resources the producer might need to make + * progress. * * The consumer side only supports a single consumer at a time, callers must * provide their own serialization for it. The stub node is what allows the @@ -69,6 +70,12 @@ static inline bool mpscq_push(struct mpscq *q, struct llist_node *node) struct llist_node *prev; node->next = NULL; + /* + * All the consumer can do about a node that isn't linked yet is to try + * again. If it is what preempted us, and we don't get to run as long + * as it does, that never ends. + */ + preempt_disable(); /* * xchg() implies a full barrier, so the initialization of the * entry (including ->next above) is visible before the node can @@ -77,6 +84,7 @@ static inline bool mpscq_push(struct mpscq *q, struct llist_node *node) */ prev = xchg(&q->tail, node); WRITE_ONCE(prev->next, node); + preempt_enable(); return prev == &q->stub; } diff --git a/io_uring/napi.c b/io_uring/napi.c index ca1c814fa73a..7e698720cd51 100644 --- a/io_uring/napi.c +++ b/io_uring/napi.c @@ -6,7 +6,7 @@ #ifdef CONFIG_NET_RX_BUSY_POLL /* Timeout for cleanout of stale entries. */ -#define NAPI_TIMEOUT (60 * SEC_CONVERSION) +#define NAPI_TIMEOUT (60 * HZ) struct io_napi_entry { unsigned int napi_id; @@ -378,6 +378,8 @@ void __io_napi_busy_loop(struct io_ring_ctx *ctx, struct io_wait_queue *iowq) if (iowq->timeout != KTIME_MAX) { ktime_t dt = ktime_sub(iowq->timeout, io_get_time(ctx)); + /* an absolute timeout may have passed already */ + dt = max_t(ktime_t, dt, 0); iowq->napi_busy_poll_dt = min_t(u64, iowq->napi_busy_poll_dt, dt); } diff --git a/io_uring/net.c b/io_uring/net.c index c82ea46cf53f..4b36883e101b 100644 --- a/io_uring/net.c +++ b/io_uring/net.c @@ -480,6 +480,24 @@ static int io_bundle_nbufs(struct io_async_msghdr *kmsg, int ret) return nbufs; } +/* + * A receive that is retried after a partial transfer keeps writing to what it + * has mapped. io_net_kbuf_recyle() only commits what was filled so far, which + * takes a buffer out of circulation if it's a single one that is consumed as a + * whole. The tail of an incrementally consumed buffer, or the yet unused + * buffers of a bundle, would be up for grabs. They can't be committed in + * advance either, the CQE has no way to tell if they end up not being used. + */ +static bool io_net_kbuf_can_retry(struct io_kiocb *req, + struct io_buffer_list *bl, + struct io_async_msghdr *kmsg) +{ + /* not ring provided, or committed at selection time */ + if (!(req->flags & REQ_F_BUFFERS_COMMIT)) + return true; + return !(bl->flags & IOBL_INC) && iter_is_ubuf(&kmsg->msg.msg_iter); +} + static int io_net_kbuf_recyle(struct io_kiocb *req, struct io_buffer_list *bl, struct io_async_msghdr *kmsg, int len) { @@ -863,12 +881,15 @@ static inline bool io_recv_finish(struct io_kiocb *req, cflags |= IORING_CQE_F_SOCK_NONEMPTY; if (sel->val > 0 && sr->flags & IORING_RECV_MSHOT_LIM) { + unsigned int this_len = sel->val - sr->done_io; + /* - * If sr->len hits zero, the limit has been reached. Mark - * mshot as finished, and flag MSHOT_DONE as well to prevent - * a potential bundle from being retried. + * If sr->mshot_total_len hits zero, the limit has been reached. + * Mark mshot as finished, and flag MSHOT_DONE as well to + * prevent a potential bundle from being retried. */ - sr->mshot_total_len -= min_t(int, sel->val, sr->mshot_total_len); + sr->mshot_total_len -= min_t(unsigned int, this_len, + sr->mshot_total_len); if (!sr->mshot_total_len) { sr->flags |= IORING_RECV_MSHOT_DONE; mshot_finished = true; @@ -1050,6 +1071,7 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) sel = io_buffer_select(req, &len, sr->buf_group, issue_flags); if (!sel.addr) return -ENOBUFS; + len = min_t(size_t, len, MAX_RW_COUNT); if (req->flags & REQ_F_APOLL_MULTISHOT) { ret = io_recvmsg_prep_multishot(kmsg, sr, &sel.addr, &len); @@ -1091,7 +1113,8 @@ int io_recvmsg(struct io_kiocb *req, unsigned int issue_flags) io_kbuf_recycle(req, sel.buf_list, issue_flags); return IOU_RETRY; } - if (ret > 0 && io_net_retry(sock, flags)) { + if (ret > 0 && io_net_retry(sock, flags) && + io_net_kbuf_can_retry(req, sel.buf_list, kmsg)) { sr->done_io += ret; return io_net_kbuf_recyle(req, sel.buf_list, kmsg, ret); } @@ -1254,7 +1277,8 @@ int io_recv(struct io_kiocb *req, unsigned int issue_flags) io_kbuf_recycle(req, sel.buf_list, issue_flags); return IOU_RETRY; } - if (ret > 0 && io_net_retry(sock, flags)) { + if (ret > 0 && io_net_retry(sock, flags) && + io_net_kbuf_can_retry(req, sel.buf_list, kmsg)) { sr->len -= ret; sr->buf += ret; sr->done_io += ret; @@ -1713,7 +1737,7 @@ void io_socket_bpf_populate(struct io_uring_bpf_ctx *bctx, struct io_kiocb *req) struct io_socket *sock = io_kiocb_to_cmd(req, struct io_socket); bctx->socket.family = sock->domain; - bctx->socket.type = sock->type; + bctx->socket.type = sock->type & SOCK_TYPE_MASK; bctx->socket.protocol = sock->protocol; } @@ -1857,6 +1881,17 @@ int io_connect(struct io_kiocb *req, unsigned int issue_flags) if (connect->seen_econnaborted) goto out; connect->seen_econnaborted = true; + } else if (req->file->f_flags & O_NONBLOCK) { + /* + * __sys_connect_file() adds O_NONBLOCK from the socket + * to every attempt, so a retry from io-wq fails with + * -EAGAIN as well. A poll on the socket doesn't wait + * until a retry can succeed. For example, an AF_UNIX + * client reports EPOLLOUT right away while the + * listener's backlog is full. Complete the request + * with -EAGAIN, like connect(2) does. + */ + goto out; } return -EAGAIN; } diff --git a/io_uring/openclose.c b/io_uring/openclose.c index c71242915dad..24cf3ce2e2bc 100644 --- a/io_uring/openclose.c +++ b/io_uring/openclose.c @@ -375,13 +375,16 @@ static int io_pipe_fixed(struct io_kiocb *req, struct file **files, fds[1] = alloc_slot ? ret : slot - 1; files[1] = NULL; - io_ring_submit_unlock(ctx, issue_flags); - - if (!copy_to_user(p->fds, fds, sizeof(fds))) + /* + * Hang on to the lock. If this fails, the slots have to be cleared, and + * they must still hold what was put there. + */ + if (!copy_to_user(p->fds, fds, sizeof(fds))) { + io_ring_submit_unlock(ctx, issue_flags); return 0; + } ret = -EFAULT; - io_ring_submit_lock(ctx, issue_flags); err: if (fds[0] != -1) io_fixed_fd_remove(ctx, fds[0]); diff --git a/io_uring/poll.c b/io_uring/poll.c index 0ad9c00efdea..4f04bda42f7f 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -626,6 +626,18 @@ static int __io_arm_poll_handler(struct io_kiocb *req, io_poll_add_hash(req, issue_flags); + /* + * Whoever tried to cancel the request while io-wq was running it has + * flagged the work item. If it searched the hash table before the + * request got there it found nothing, and it won't come back. Both + * the search and the insertion are done with ->uring_lock held, and + * the flag is set before the search. If the search came first, + * the flag is visible by now. + */ + if (ipt->owning && + (atomic_read(&req->work.flags) & IO_WQ_WORK_CANCEL)) + io_poll_mark_cancelled(req); + if (mask && (poll->events & EPOLLET) && io_poll_can_finish_inline(req, ipt)) { __io_poll_execute(req, mask, 0); @@ -915,6 +927,15 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) ipt.pt._qproc = io_poll_queue_proc; + /* + * A poll update or a spurious wake up arms the request again. Forget + * the entries of the previous round like io_arm_apoll() does, a stale + * second entry would be taken for a third wait queue. + */ + if (req_has_async_data(req)) + io_req_async_data_free(req); + req->flags &= ~(REQ_F_SINGLE_POLL | REQ_F_DOUBLE_POLL); + ret = __io_arm_poll_handler(req, poll, &ipt, poll->events, issue_flags); if (ret > 0) { io_req_set_res(req, ipt.result_mask, 0); diff --git a/io_uring/register.c b/io_uring/register.c index dab3eb4258df..62f492e6472f 100644 --- a/io_uring/register.c +++ b/io_uring/register.c @@ -496,6 +496,17 @@ static void io_register_free_rings(struct io_ring_ctx *ctx, IORING_SETUP_CQE32 | IORING_SETUP_NO_MMAP | \ IORING_SETUP_CQE_MIXED | IORING_SETUP_SQE_MIXED) +static bool io_sqe_is_128(struct io_ring_ctx *ctx, + const struct io_uring_sqe *sqe) +{ + u8 opcode = READ_ONCE(sqe->opcode); + + if (!(ctx->flags & IORING_SETUP_SQE_MIXED) || opcode >= IORING_OP_LAST) + return false; + opcode = array_index_nospec(opcode, IORING_OP_LAST); + return io_issue_defs[opcode].is_128; +} + static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) { struct io_ctx_config config; @@ -505,6 +516,8 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) struct io_uring_params *p = &config.p; struct io_rings_layout *rl = &config.layout; u32 *o_sq_array, *n_sq_array = NULL; + bool cqe32_tail = false; + unsigned int sqe128_src = 0; int ret; memset(&config, 0, sizeof(config)); @@ -603,13 +616,20 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) old_head = READ_ONCE(o.rings->sq.head); if (tail - old_head > p->sq_entries) goto overflow; - for (i = old_head; i < tail; i++) { + for (i = old_head; i != tail; i++) { + bool sqe128_tail = sqe128_src; unsigned int dst, src; size_t sq_size; dst = i & (p->sq_entries - 1); src = i & (ctx->sq_entries - 1); - if (n_sq_array) { + if (sqe128_tail) { + /* it follows the first half, whatever the array says */ + src = sqe128_src; + sqe128_src = 0; + if (n_sq_array) + WRITE_ONCE(n_sq_array[dst], dst); + } else if (n_sq_array) { src = READ_ONCE(o_sq_array[src]); if (unlikely(src >= ctx->sq_entries)) { WRITE_ONCE(n_sq_array[dst], UINT_MAX); @@ -618,6 +638,19 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) WRITE_ONCE(n_sq_array[dst], dst); } + /* + * A 128b SQE on a SQE_MIXED ring takes two contiguous entries + * and must not wrap. A smaller ring may place it on the last + * entry, fail the resize if that happens. + */ + if (!sqe128_tail && i + 1 != tail && + src < ctx->sq_entries - 1 && + io_sqe_is_128(ctx, &o.sq_sqes[src])) { + if (dst == p->sq_entries - 1) + goto overflow; + sqe128_src = src + 1; + } + sq_size = sizeof(struct io_uring_sqe); if (ctx->flags & IORING_SETUP_SQE128) { dst <<= 1; @@ -640,7 +673,7 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) ret = -EOVERFLOW; goto out; } - for (i = old_head; i < tail; i++) { + for (i = old_head; i != tail; i++) { unsigned index, dst_mask, src_mask; size_t cq_size; @@ -654,6 +687,21 @@ static int io_register_resize_rings(struct io_ring_ctx *ctx, void __user *arg) src_mask = (ctx->cq_entries << 1) - 1; dst_mask = (p->cq_entries << 1) - 1; } + /* + * A 32b CQE on a CQE_MIXED ring takes two contiguous entries + * and must not wrap. A smaller ring may place it on the last + * entry, fail the resize if that happens. + */ + if (ctx->flags & IORING_SETUP_CQE_MIXED) { + if (cqe32_tail) { + cqe32_tail = false; + } else if (READ_ONCE(o.rings->cqes[index & src_mask].flags) & + IORING_CQE_F_32) { + if ((index & dst_mask) == dst_mask) + goto overflow; + cqe32_tail = true; + } + } memcpy(&n.rings->cqes[index & dst_mask], &o.rings->cqes[index & src_mask], cq_size); } WRITE_ONCE(n.rings->cq.head, old_head); @@ -1010,6 +1058,13 @@ static int io_uring_register_send_msg_ring(void __user *arg, unsigned int nr_arg static int io_uring_register_blind(unsigned int opcode, void __user *arg, unsigned int nr_args) { + if (current->io_uring_restrict && + current->io_uring_restrict->reg_registered) { + opcode = array_index_nospec(opcode, IORING_REGISTER_LAST); + if (!test_bit(opcode, current->io_uring_restrict->register_op)) + return -EACCES; + } + switch (opcode) { case IORING_REGISTER_SEND_MSG_RING: return io_uring_register_send_msg_ring(arg, nr_args); diff --git a/io_uring/rsrc.c b/io_uring/rsrc.c index 51b46e624ddd..d69baa47c7e7 100644 --- a/io_uring/rsrc.c +++ b/io_uring/rsrc.c @@ -162,8 +162,9 @@ static struct io_mapped_ubuf *io_alloc_imu(struct io_ring_ctx *ctx, int nr_bvecs) { if (nr_bvecs <= IO_CACHED_BVECS_SEGS) - return io_cache_alloc(&ctx->imu_cache, GFP_KERNEL); - return kvmalloc_flex(struct io_mapped_ubuf, bvec, nr_bvecs); + return io_cache_alloc(&ctx->imu_cache, GFP_KERNEL_ACCOUNT); + return kvmalloc_flex(struct io_mapped_ubuf, bvec, nr_bvecs, + GFP_KERNEL_ACCOUNT); } static void io_free_imu(struct io_ring_ctx *ctx, struct io_mapped_ubuf *imu) @@ -229,7 +230,7 @@ struct io_rsrc_node *io_rsrc_node_alloc(struct io_ring_ctx *ctx, int type) { struct io_rsrc_node *node; - node = io_cache_alloc(&ctx->node_cache, GFP_KERNEL); + node = io_cache_alloc(&ctx->node_cache, GFP_KERNEL_ACCOUNT); if (node) { node->type = type; node->refs = 1; diff --git a/io_uring/rw.c b/io_uring/rw.c index 432820f86251..b21daa3cfa5c 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -610,24 +610,6 @@ static void io_complete_rw_iopoll(struct kiocb *kiocb, long res) smp_store_release(&req->iopoll_completed, 1); } -static inline ssize_t io_fixup_restart_res(ssize_t ret) -{ - switch (ret) { - case -ERESTARTSYS: - case -ERESTARTNOINTR: - case -ERESTARTNOHAND: - case -ERESTART_RESTARTBLOCK: - /* - * We can't just restart the syscall, since previously - * submitted sqes may already be in progress. Just fail - * this IO with EINTR. - */ - return -EINTR; - default: - return ret; - } -} - static inline void io_rw_done(struct io_kiocb *req, ssize_t ret) { struct io_rw *rw = io_kiocb_to_cmd(req, struct io_rw); diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c index ad42e8eb1002..49c52284ecdd 100644 --- a/io_uring/sqpoll.c +++ b/io_uring/sqpoll.c @@ -63,6 +63,24 @@ void io_sq_thread_park(struct io_sq_data *sqd) } } +/* + * The sqd has two holders: its rings, as one, and its thread. Neither waits + * for the other to let go. The one that lets go last frees the sqd. + */ +static void io_sq_data_release(struct io_sq_data *sqd) +{ + if (refcount_dec_and_test(&sqd->free_refs)) + kfree(sqd); +} + +/* + * Ask the thread to exit, and don't wait for it. A thread that is stopped or + * frozen exits only when a user lets it run again, and the exit work of the + * ring would keep its place in the workqueue until then. All rings have left + * ctx_list by now and have no request left, so the thread can reach none of + * them. What it still uses is its own or counted: the sqd by free_refs, the + * hash of its io-wq by hash->refs. + */ void io_sq_thread_stop(struct io_sq_data *sqd) { struct task_struct *tsk; @@ -77,7 +95,6 @@ void io_sq_thread_stop(struct io_sq_data *sqd) wake_up_process(tsk); } mutex_unlock(&sqd->lock); - wait_for_completion(&sqd->exited); } void io_put_sq_data(struct io_sq_data *sqd) @@ -86,7 +103,7 @@ void io_put_sq_data(struct io_sq_data *sqd) WARN_ON_ONCE(atomic_read(&sqd->park_pending)); io_sq_thread_stop(sqd); - kfree(sqd); + io_sq_data_release(sqd); } } @@ -160,10 +177,10 @@ static struct io_sq_data *io_get_sq_data(struct io_uring_params *p, atomic_set(&sqd->park_pending, 0); refcount_set(&sqd->refs, 1); + refcount_set(&sqd->free_refs, 2); INIT_LIST_HEAD(&sqd->ctx_list); mutex_init(&sqd->lock); init_waitqueue_head(&sqd->wait); - init_completion(&sqd->exited); return sqd; } @@ -385,6 +402,13 @@ static int io_sq_thread(void *data) */ smp_mb__after_atomic(); + /* + * Nothing gets submitted before the ring is + * enabled, which comes with a wake up. + */ + if (READ_ONCE(ctx->flags) & IORING_SETUP_R_DISABLED) + continue; + if (io_sqring_entries(ctx)) { needs_sched = false; break; @@ -417,7 +441,7 @@ static int io_sq_thread(void *data) io_run_task_work(); mutex_unlock(&sqd->lock); err_out: - complete(&sqd->exited); + io_sq_data_release(sqd); do_exit(0); } @@ -537,7 +561,8 @@ __cold int io_sq_offload_create(struct io_ring_ctx *ctx, } return 0; err_sqpoll: - complete(&ctx->sq_data->exited); + /* no thread was made: let go of its share */ + io_sq_data_release(ctx->sq_data); err: io_sq_thread_finish(ctx); return ret; diff --git a/io_uring/sqpoll.h b/io_uring/sqpoll.h index fd2f6f29b516..024a51dc3069 100644 --- a/io_uring/sqpoll.h +++ b/io_uring/sqpoll.h @@ -2,6 +2,8 @@ struct io_sq_data { refcount_t refs; + /* one for all the rings, one for the thread: the last put frees */ + refcount_t free_refs; atomic_t park_pending; struct mutex lock; @@ -18,7 +20,6 @@ struct io_sq_data { u64 work_time; unsigned long state; - struct completion exited; }; int io_sq_offload_create(struct io_ring_ctx *ctx, struct io_uring_params *p); diff --git a/io_uring/tctx.c b/io_uring/tctx.c index 466b7300e208..b4fecea3ea47 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -64,7 +64,7 @@ void io_uring_free_tctx(struct task_struct *tsk) WARN_ON_ONCE(tctx->cached_refs); percpu_counter_destroy(&tctx->inflight); - kfree(tctx); + kfree_rcu(tctx, rcu); tsk->io_uring = NULL; } } @@ -104,7 +104,7 @@ __cold struct io_uring_task *io_uring_alloc_task_context(struct task_struct *tas } tctx->task = task; - xa_init(&tctx->xa); + xa_init_flags(&tctx->xa, XA_FLAGS_ACCOUNT); init_waitqueue_head(&tctx->wait); atomic_set(&tctx->in_cancel, 0); atomic_set(&tctx->inflight_tracked, 0); @@ -123,7 +123,7 @@ static int io_tctx_install_node(struct io_ring_ctx *ctx, if (xa_load(&tctx->xa, (unsigned long)ctx)) return 0; - node = kmalloc_obj(*node); + node = kmalloc_obj(*node, GFP_KERNEL_ACCOUNT); if (!node) return -ENOMEM; node->ctx = ctx; @@ -189,6 +189,7 @@ int __io_uring_add_tctx_node(struct io_ring_ctx *ctx) io_wq_exit_start(tctx->io_wq); io_wq_put_and_exit(tctx->io_wq); } + xa_destroy(&tctx->xa); percpu_counter_destroy(&tctx->inflight); kfree(tctx); } @@ -250,6 +251,8 @@ __cold void io_uring_clean_tctx(struct io_uring_task *tctx) io_uring_del_tctx_node(index); cond_resched(); } + /* A failed xa_store() leaves nodes with no entry behind. */ + xa_destroy(&tctx->xa); if (wq) { /* * Must be after io_uring_del_tctx_node() (removes nodes under diff --git a/io_uring/timeout.c b/io_uring/timeout.c index c4dd26cf342d..8e7a4bd7e514 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -171,6 +171,14 @@ __cold void io_flush_timeouts(struct io_ring_ctx *ctx) raw_spin_lock_irq(&ctx->timeout_lock); seq = READ_ONCE(ctx->cached_cq_tail) - atomic_read(&ctx->cq_timeouts); + /* + * A fired timeout bumps ->cq_timeouts before its CQE is posted, which + * leaves the sequence behind the last flush until it is. No event has + * happened then, see io_timeout(). + */ + if ((s32)(seq - ctx->cq_last_tm_flush) < 0) + seq = ctx->cq_last_tm_flush; + list_for_each_entry_safe(timeout, tmp, &ctx->timeout_list, list) { struct io_kiocb *req = cmd_to_io_kiocb(timeout); u32 events_needed, events_got; @@ -321,22 +329,29 @@ static enum hrtimer_restart io_timeout_fn(struct hrtimer *timer) return HRTIMER_NORESTART; } -static struct io_kiocb *io_timeout_extract(struct io_ring_ctx *ctx, - struct io_cancel_data *cd) +static struct io_kiocb *io_timeout_find(struct io_ring_ctx *ctx, + struct io_cancel_data *cd) __must_hold(&ctx->timeout_lock) { struct io_timeout *timeout; - struct io_timeout_data *io; - struct io_kiocb *req = NULL; list_for_each_entry(timeout, &ctx->timeout_list, list) { - struct io_kiocb *tmp = cmd_to_io_kiocb(timeout); + struct io_kiocb *req = cmd_to_io_kiocb(timeout); - if (io_cancel_req_match(tmp, cd)) { - req = tmp; - break; - } + if (io_cancel_req_match(req, cd)) + return req; } + return NULL; +} + +static struct io_kiocb *io_timeout_extract(struct io_ring_ctx *ctx, + struct io_cancel_data *cd) + __must_hold(&ctx->timeout_lock) +{ + struct io_kiocb *req = io_timeout_find(ctx, cd); + struct io_timeout *timeout; + struct io_timeout_data *io; + if (!req) return ERR_PTR(-ENOENT); @@ -355,6 +370,9 @@ int io_timeout_cancel(struct io_ring_ctx *ctx, struct io_cancel_data *cd) raw_spin_lock_irq(&ctx->timeout_lock); req = io_timeout_extract(ctx, cd); + if (!IS_ERR(req)) + atomic_set(&ctx->cq_timeouts, + atomic_read(&ctx->cq_timeouts) + 1); raw_spin_unlock_irq(&ctx->timeout_lock); if (IS_ERR(req)) @@ -386,6 +404,9 @@ static void io_req_task_link_timeout(struct io_tw_req tw_req, io_tw_token_t tw) }; ret = io_try_cancel(req->tctx, &cd, 0); + /* a number of requests for futex and waitid */ + if (ret > 0) + ret = 0; } else { ret = -ECANCELED; } @@ -458,6 +479,8 @@ static int io_linked_timeout_update(struct io_ring_ctx *ctx, __u64 user_data, io = req->async_data; if (hrtimer_try_to_cancel(&io->timer) == -1) return -EALREADY; + if (mode == HRTIMER_MODE_ABS) + ts = timens_ktime_to_host(io_timeout_get_clock(io), ts); hrtimer_setup(&io->timer, io_link_timeout_fn, io_timeout_get_clock(io), mode); hrtimer_start(&io->timer, ts, mode); return 0; @@ -468,15 +491,26 @@ static int io_timeout_update(struct io_ring_ctx *ctx, __u64 user_data, __must_hold(&ctx->timeout_lock) { struct io_cancel_data cd = { .ctx = ctx, .data = user_data, }; - struct io_kiocb *req = io_timeout_extract(ctx, &cd); + struct io_kiocb *req = io_timeout_find(ctx, &cd); struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout); struct io_timeout_data *data; - if (IS_ERR(req)) - return PTR_ERR(req); + if (!req) + return -ENOENT; - timeout->off = 0; /* noseq */ data = req->async_data; + /* multishot requests only make sense with rel values */ + if ((data->flags & IORING_TIMEOUT_MULTISHOT) && mode == HRTIMER_MODE_ABS) + return -EINVAL; + + req = io_timeout_extract(ctx, &cd); + if (IS_ERR(req)) + return PTR_ERR(req); + /* for a multishot request this isn't a sequence, keep the repeats */ + if (!(data->flags & IORING_TIMEOUT_MULTISHOT)) + timeout->off = 0; /* noseq */ + if (mode == HRTIMER_MODE_ABS) + time = timens_ktime_to_host(io_timeout_get_clock(data), time); data->time = time; list_add_tail(&timeout->list, &ctx->timeout_list); @@ -509,7 +543,12 @@ int io_timeout_remove_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe) IORING_TIMEOUT_ABS | IORING_TIMEOUT_IMMEDIATE_ARG)) return -EINVAL; - ret = io_parse_user_time(&tr->time, READ_ONCE(sqe->addr2), tr->flags); + /* + * The clock is that of the timeout to update, an absolute time + * is converted to the host's view once that has been found. + */ + ret = io_parse_user_time(&tr->time, READ_ONCE(sqe->addr2), + tr->flags & ~IORING_TIMEOUT_ABS); if (ret) return ret; } else if (tr->flags) { @@ -645,11 +684,19 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags) struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout); struct io_ring_ctx *ctx = req->ctx; struct io_timeout_data *data = req->async_data; + struct io_timeout *first; struct list_head *entry; - u32 tail, off = timeout->off; + u32 tail, base, off = timeout->off; raw_spin_lock_irq(&ctx->timeout_lock); + /* the list is searched under ->timeout_lock, see io_wq_cancel_seen() */ + if (__io_wq_cancel_seen(req, issue_flags)) { + raw_spin_unlock_irq(&ctx->timeout_lock); + io_req_task_queue_fail(req, -ECANCELED); + return IOU_ISSUE_SKIP_COMPLETE; + } + /* * sqe->off holds how many events that need to occur for this * timeout event to be satisfied. If it isn't set, then this is @@ -661,13 +708,27 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags) } tail = data_race(ctx->cached_cq_tail) - atomic_read(&ctx->cq_timeouts); - timeout->target_seq = tail + off; - /* Update the last seq here in case io_flush_timeouts() hasn't. - * This is safe because ->completion_lock is held, and submissions - * and completions are never mixed in the same ->completion_lock section. + /* + * Update the last seq here in case io_flush_timeouts() hasn't, it only + * gets called once sequences are in use. Leave it alone if there are + * timeouts waiting for one. CQEs may have been posted since the last + * flush, and a target that has been reached must not fall behind it. */ - ctx->cq_last_tm_flush = tail; + first = list_first_entry_or_null(&ctx->timeout_list, struct io_timeout, + list); + if (!first || io_is_timeout_noseq(cmd_to_io_kiocb(first))) + ctx->cq_last_tm_flush = tail; + base = ctx->cq_last_tm_flush; + + /* + * A fired timeout bumps ->cq_timeouts before its CQE is posted, which + * leaves the sequence behind the last flush until it is. Nothing else + * separates the two then, so count from the flush. + */ + if ((s32)(tail - base) < 0) + tail = base; + timeout->target_seq = tail + off; /* * Insertion sort, ensuring the first entry in the list is always @@ -679,8 +740,8 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags) if (io_is_timeout_noseq(nxt)) continue; - /* nxt.seq is behind @tail, otherwise would've been completed */ - if (off >= nextt->target_seq - tail) + /* no target is behind the last flush, see io_flush_timeouts() */ + if (timeout->target_seq - base >= nextt->target_seq - base) break; } add: diff --git a/io_uring/tw.c b/io_uring/tw.c index f9affd68dd26..0b5066f97224 100644 --- a/io_uring/tw.c +++ b/io_uring/tw.c @@ -148,6 +148,37 @@ static void io_ctx_mark_taskrun(struct io_ring_ctx *ctx) } } +static void io_poll_wq_wake_rcu(struct rcu_head *rcu) +{ + struct io_ring_ctx *ctx = container_of(rcu, struct io_ring_ctx, + poll_wq_rcu); + + atomic_set(&ctx->poll_wq_rcu_pending, 0); + io_poll_wq_wake(ctx); + percpu_ref_put(&ctx->refs); +} + +/* + * io_uring_poll() reports pending local work as EPOLLIN, as it takes a trip to + * the kernel to turn it into CQEs. Let those know who are asleep already. + */ +static void io_local_work_wake_poll(struct io_ring_ctx *ctx, unsigned int flags) +{ + if (!(flags & IOU_F_TWQ_IN_WAKE)) { + io_poll_wq_wake(ctx); + return; + } + /* + * Called from a wake up handler with some wait queue lock held, this + * one could lead back to it. Do it from a clean context. + */ + if (!wq_has_sleeper(&ctx->poll_wq) || + atomic_xchg(&ctx->poll_wq_rcu_pending, 1)) + return; + percpu_ref_get(&ctx->refs); + call_rcu_hurry(&ctx->poll_wq_rcu, io_poll_wq_wake_rcu); +} + void io_req_local_work_add(struct io_kiocb *req, unsigned flags) { struct io_ring_ctx *ctx = req->ctx; @@ -165,7 +196,7 @@ void io_req_local_work_add(struct io_kiocb *req, unsigned flags) /* * The xchg() in mpscq_push() implies a full barrier, which pairs with - * the barrier in set_current_state() on the io_cqring_wait() side. This + * the barrier in io_cq_wait_arm() on the io_cqring_wait() side. This * ensures that either we see the updated ->cq_wait_nr, or waiters going * to sleep will observe the work added to the list, which is similar to * the wait/wake task state sync. @@ -174,6 +205,9 @@ void io_req_local_work_add(struct io_kiocb *req, unsigned flags) io_ctx_mark_taskrun(ctx); if (data_race(ctx->int_flags) & IO_RING_F_HAS_EVFD) io_eventfd_signal(ctx, false, flags & IOU_F_TWQ_IN_WAKE); + /* whoever sets the flag wakes ->poll_wq afterwards */ + if (data_race(ctx->int_flags) & IO_RING_F_POLL_ACTIVATED) + io_local_work_wake_poll(ctx, flags); } /* diff --git a/io_uring/uring_cmd.c b/io_uring/uring_cmd.c index 726a659f38c3..3baf6459c1c8 100644 --- a/io_uring/uring_cmd.c +++ b/io_uring/uring_cmd.c @@ -87,7 +87,7 @@ static void io_uring_cmd_del_cancelable(struct io_uring_cmd *cmd, cmd->flags &= ~IORING_URING_CMD_CANCELABLE; io_ring_submit_lock(ctx, issue_flags); - hlist_del(&req->hash_node); + hlist_del_init(&req->hash_node); io_ring_submit_unlock(ctx, issue_flags); } diff --git a/io_uring/wait.c b/io_uring/wait.c index 8fc175f10c54..fb1c082993a7 100644 --- a/io_uring/wait.c +++ b/io_uring/wait.c @@ -27,7 +27,8 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode, */ if (io_should_wake(iowq) || io_has_work(iowq->ctx)) return autoremove_wake_function(curr, mode, wake_flags, key); - return -1; + /* keep going, a waiter further down may be content with less */ + return 0; } int io_run_task_work_sig(struct io_ring_ctx *ctx) @@ -77,9 +78,16 @@ static enum hrtimer_restart io_cqring_min_timer_wakeup(struct hrtimer *timer) if (iowq->timeout == KTIME_MAX || ktime_compare(iowq->min_timeout, iowq->timeout) >= 0) goto out_wake; - /* work we may need to run, wake function will see if we need to wake */ - if (io_has_work(ctx)) - goto out_wake; + /* + * Work we need to run. It isn't known if that is going to post a CQE, + * so this isn't a timeout. Carry on with the normal one if it doesn't. + */ + if (io_has_work(ctx)) { + iowq->min_timeout = 0; + iowq->cq_tail = iowq->cq_min_tail + 1; + wake_up_process(iowq->wq.private); + return HRTIMER_NORESTART; + } /* got events since we started waiting, min timeout is done */ scoped_guard(rcu) { struct io_rings *rings = io_get_rings(ctx); @@ -267,8 +275,7 @@ int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags, unsigned long check_cq; if (ctx->flags & IORING_SETUP_DEFER_TASKRUN) { - atomic_set(&ctx->cq_wait_nr, nr_wait); - set_current_state(TASK_INTERRUPTIBLE); + io_cq_wait_arm(ctx, nr_wait); } else { prepare_to_wait_exclusive(&ctx->cq_wait, &iowq.wq, TASK_INTERRUPTIBLE); @@ -301,9 +308,18 @@ int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, u32 flags, check_cq = READ_ONCE(ctx->check_cq); if (unlikely(check_cq)) { - /* let the caller flush overflows, retry */ - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) { io_cqring_do_overflow_flush(ctx); + /* + * No room for what has overflown. It takes the + * application to change that, so return to it. + */ + if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT, + &ctx->check_cq)) { + ret = 0; + break; + } + } if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) { ret = -EBADR; break; diff --git a/io_uring/wait.h b/io_uring/wait.h index b7b9c46b1b01..eb5ae8a6fc94 100644 --- a/io_uring/wait.h +++ b/io_uring/wait.h @@ -14,6 +14,21 @@ */ #define IO_CQ_WAKE_INIT (-1) +/* + * Start a wait cycle of the current task. The add that uses up ->cq_wait_nr + * may be one that queued its entry long ago and only now gets to look at the + * count. As nobody else wakes us after that, it has to find us sleeping, hence + * the task state must be visible before the count is. The barrier pairs with + * the xchg() in mpscq_push(), see io_req_local_work_add(). + */ +static inline void io_cq_wait_arm(struct io_ring_ctx *ctx, int nr_wait) +{ + __set_current_state(TASK_INTERRUPTIBLE); + atomic_set_release(&ctx->cq_wait_nr, nr_wait); + /* order the above against looking at the work list */ + smp_mb(); +} + struct ext_arg { size_t argsz; struct timespec64 ts; diff --git a/io_uring/waitid.c b/io_uring/waitid.c index 9ad8eadc5b8f..172cf770b29b 100644 --- a/io_uring/waitid.c +++ b/io_uring/waitid.c @@ -268,12 +268,18 @@ static int io_waitid_wait(struct wait_queue_entry *wait, unsigned mode, return 0; list_del_init(&wait->entry); - smp_store_release(&iw->head, NULL); - /* cancel is in progress */ + /* + * Cancel or issue is in progress. Leave ->head alone, so that whoever + * owns the request has to take the wait queue lock, which we hold, + * before it can complete and free it. + */ if (atomic_fetch_inc(&iw->refs) & IO_WAITID_REF_MASK) return 1; + /* pairs with the smp_load_acquire() in io_waitid_remove_wq() */ + smp_store_release(&iw->head, NULL); + req->io_task_work.func = io_waitid_cb; __io_req_task_work_add(req, IOU_F_TWQ_IN_WAKE); return 1; @@ -346,6 +352,8 @@ int io_waitid(struct io_kiocb *req, unsigned int issue_flags) * a waitqueue callback, or if someone cancels it. */ if (!io_waitid_drop_issue_ref(req)) { + if (io_wq_cancel_seen(req, issue_flags)) + __io_waitid_cancel(req, true); io_ring_submit_unlock(ctx, issue_flags); return IOU_ISSUE_SKIP_COMPLETE; } diff --git a/io_uring/zcrx.c b/io_uring/zcrx.c index cc9df7e64a40..6d7abb1636e7 100644 --- a/io_uring/zcrx.c +++ b/io_uring/zcrx.c @@ -1645,7 +1645,7 @@ static bool io_zcrx_queue_cqe(struct io_kiocb *req, struct net_iov *niov, cqe->flags |= IORING_CQE_F_32; area = io_zcrx_iov_to_area(niov); - offset = off + (net_iov_idx(niov) << ifq->niov_shift); + offset = off + ((u64)net_iov_idx(niov) << ifq->niov_shift); rcqe = (struct io_uring_zcrx_cqe *)(cqe + 1); rcqe->off = offset + zcrx_area_id_to_token(area->area_id); rcqe->__pad = 0; diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 227d09704ca5..42cc77dfa1c8 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -6844,6 +6844,31 @@ static struct cgroup *cgroup_get_from_file(struct file *f) return cgrp; } +/* + * A new threadgroup gets its own cgroup_threadgroup_rwsem. Once its PID is + * visible, a cgroup.procs writer can find the child and lock that rwsem even + * though cgroup_post_fork() has not linked the child to its css_set yet. + * + * cgroup_can_fork() already holds the global read lock, which keeps + * cgroup_enable_per_threadgroup_rwsem stable. Lock the child's rwsem as a + * nested instance until cgroup_post_fork() finishes. Threads share their + * parent's signal_struct and are covered by the existing read lock. + */ +static void cgroup_fork_child_lock(struct task_struct *child) +{ + if (cgroup_enable_per_threadgroup_rwsem && + child->signal != current->signal) + down_read_nested(&child->signal->cgroup_threadgroup_rwsem, + SINGLE_DEPTH_NESTING); +} + +static void cgroup_fork_child_unlock(struct task_struct *child) +{ + if (cgroup_enable_per_threadgroup_rwsem && + child->signal != current->signal) + up_read(&child->signal->cgroup_threadgroup_rwsem); +} + /** * cgroup_css_set_fork - find or create a css_set for a child process * @kargs: the arguments passed to create the child process @@ -7009,6 +7034,7 @@ int cgroup_can_fork(struct task_struct *child, struct kernel_clone_args *kargs) goto out_revert; } while_each_subsys_mask(); + cgroup_fork_child_lock(child); return 0; out_revert: @@ -7043,6 +7069,7 @@ void cgroup_cancel_fork(struct task_struct *child, if (ss->cancel_fork) ss->cancel_fork(child, kargs->cset); + cgroup_fork_child_unlock(child); cgroup_css_set_put_fork(kargs); } @@ -7140,6 +7167,7 @@ void cgroup_post_fork(struct task_struct *child, if (unlikely(kill)) do_send_sig_info(SIGKILL, SEND_SIG_NOINFO, child, PIDTYPE_TGID); + cgroup_fork_child_unlock(child); cgroup_css_set_put_fork(kargs); } diff --git a/kernel/fork.c b/kernel/fork.c index efe6157fedb4..26b666e98bb8 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -2718,6 +2718,10 @@ struct task_struct *create_io_thread(int (*fn)(void *), void *arg, int node) .user_worker = 1, }; + /* A creator past its fatal signal or its coredump point gets no thread. */ + if (current->flags & (PF_SIGNALED | PF_POSTCOREDUMP)) + return ERR_PTR(-EINTR); + return copy_process(NULL, 0, node, &args); } diff --git a/kernel/irq/internals.h b/kernel/irq/internals.h index 0ce21dd45404..2c30e7537987 100644 --- a/kernel/irq/internals.h +++ b/kernel/irq/internals.h @@ -36,6 +36,7 @@ extern struct irqaction chained_action; * IRQTF_AFFINITY - irq thread is requested to adjust affinity * IRQTF_FORCED_THREAD - irq action is force threaded * IRQTF_READY - signals that irq thread is ready + * IRQTF_EXIT_WORK - irq thread exit task_work is pending */ enum { IRQTF_RUNTHREAD, @@ -43,6 +44,7 @@ enum { IRQTF_AFFINITY, IRQTF_FORCED_THREAD, IRQTF_READY, + IRQTF_EXIT_WORK, }; /* diff --git a/kernel/irq/manage.c b/kernel/irq/manage.c index 57eff26fa646..3b647d9b458a 100644 --- a/kernel/irq/manage.c +++ b/kernel/irq/manage.c @@ -21,6 +21,7 @@ #include #include #include +#include #include "internals.h" @@ -1176,17 +1177,14 @@ void wake_threads_waitq(struct irq_desc *desc) wake_up(&desc->wait_for_threads); } -static void irq_thread_dtor(struct callback_head *unused) +static void __irq_thread_dtor(struct irqaction *action) { struct task_struct *tsk = current; struct irq_desc *desc; - struct irqaction *action; if (WARN_ON_ONCE(!(current->flags & PF_EXITING))) return; - action = kthread_data(tsk); - pr_err("exiting task \"%s\" (%d) is an active IRQ thread (irq %d)\n", tsk->comm, tsk->pid, action->irq); @@ -1203,6 +1201,14 @@ static void irq_thread_dtor(struct callback_head *unused) irq_finalize_oneshot(desc, action); } +static void irq_thread_dtor(struct callback_head *work) +{ + struct irqaction *action = container_of(work, struct irqaction, thread_dtor); + + __irq_thread_dtor(action); + clear_and_wake_up_bit(IRQTF_EXIT_WORK, &action->thread_flags); +} + static void irq_wake_secondary(struct irq_desc *desc, struct irqaction *action) { struct irqaction *secondary = action->secondary; @@ -1244,7 +1250,6 @@ static void wake_up_and_wait_for_irq_thread_ready(struct irq_desc *desc, */ static int irq_thread(void *data) { - struct callback_head on_exit_work; struct irqaction *action = data; struct irq_desc *desc = irq_to_desc(action->irq); irqreturn_t (*handler_fn)(struct irq_desc *desc, @@ -1263,8 +1268,9 @@ static int irq_thread(void *data) else handler_fn = irq_thread_fn; - init_task_work(&on_exit_work, irq_thread_dtor); - task_work_add(current, &on_exit_work, TWA_NONE); + set_bit(IRQTF_EXIT_WORK, &action->thread_flags); + init_task_work(&action->thread_dtor, irq_thread_dtor); + task_work_add(current, &action->thread_dtor, TWA_NONE); while (!irq_wait_for_interrupt(desc, action)) { irqreturn_t action_ret; @@ -1283,6 +1289,7 @@ static int irq_thread(void *data) * oneshot mask bit can be set. */ task_work_cancel_func(current, irq_thread_dtor); + clear_and_wake_up_bit(IRQTF_EXIT_WORK, &action->thread_flags); return 0; } @@ -1853,6 +1860,12 @@ __setup_irq(unsigned int irq, struct irq_desc *desc, struct irqaction *new) return ret; } +static void irq_stop_thread(struct irqaction *action) +{ + kthread_stop_put(action->thread); + wait_on_bit(&action->thread_flags, IRQTF_EXIT_WORK, TASK_UNINTERRUPTIBLE); +} + /* * Internal function to unregister an irqaction - used to free * regular and special interrupts that are part of the architecture. @@ -1958,9 +1971,9 @@ static struct irqaction *__free_irq(struct irq_desc *desc, void *dev_id) * the same bit to a newly requested action. */ if (action->thread) { - kthread_stop_put(action->thread); + irq_stop_thread(action); if (action->secondary && action->secondary->thread) - kthread_stop_put(action->secondary->thread); + irq_stop_thread(action->secondary); } /* Last action releases resources */ diff --git a/kernel/kcov.c b/kernel/kcov.c index 35420f0ac524..54eaae97bfd6 100644 --- a/kernel/kcov.c +++ b/kernel/kcov.c @@ -253,6 +253,14 @@ static void notrace write_comp_data(u64 type, u64 arg1, u64 arg2, u64 ip) count = READ_ONCE(area[0]); + /* + * area[0] is writable by the collecting process, so count cannot be + * trusted. Bound it to the records that fit, as kcov_move_area() + * does, so the end_pos multiply below cannot wrap past its check. + */ + if (count >= max_pos / (sizeof(u64) * KCOV_WORDS_PER_CMP)) + return; + /* Every record is KCOV_WORDS_PER_CMP 64-bit words. */ start_index = 1 + count * KCOV_WORDS_PER_CMP; end_pos = (start_index + KCOV_WORDS_PER_CMP) * sizeof(u64); diff --git a/kernel/kcsan/core.c b/kernel/kcsan/core.c index 2db82661cd60..a3aa5e5b039d 100644 --- a/kernel/kcsan/core.c +++ b/kernel/kcsan/core.c @@ -290,8 +290,9 @@ should_watch(struct kcsan_ctx *ctx, const volatile void *ptr, size_t size, int t } /* - * Returns a pseudo-random number in interval [0, ep_ro). Simple linear - * congruential generator, using constants from "Numerical Recipes". + * Returns a pseudo-random number in interval [0, ep_ro), or 0 if ep_ro is 0, + * which the runtime-tunable callers may pass. Simple linear congruential + * generator, using constants from "Numerical Recipes". */ static u32 kcsan_prandom_u32_max(u32 ep_ro) { @@ -300,7 +301,7 @@ static u32 kcsan_prandom_u32_max(u32 ep_ro) state = 1664525 * state + 1013904223; this_cpu_write(kcsan_rand_state, state); - return state % ep_ro; + return reciprocal_scale(state, ep_ro); } static inline void reset_kcsan_skip(void) diff --git a/kernel/power/em_netlink.c b/kernel/power/em_netlink.c index 4d4fd29bd2be..1f68d456a713 100644 --- a/kernel/power/em_netlink.c +++ b/kernel/power/em_netlink.c @@ -95,41 +95,62 @@ static int __em_nl_get_pd_for_dump(struct em_perf_domain *pd, void *data) return ret; } +struct em_nl_doit_ctx { + struct genl_info *info; + int cmd; + struct sk_buff *msg; +}; + +static int __em_nl_get_pd_doit_fill(struct em_perf_domain *pd, void *data) +{ + struct em_nl_doit_ctx *ctx = data; + void *hdr; + + hdr = genlmsg_put_reply(ctx->msg, ctx->info, &dev_energymodel_nl_family, 0, + ctx->cmd); + if (!hdr) + return -EMSGSIZE; + + if (__em_nl_get_pd(pd, ctx->msg)) { + genlmsg_cancel(ctx->msg, hdr); + return -EMSGSIZE; + } + + genlmsg_end(ctx->msg, hdr); + return 0; +} + int dev_energymodel_nl_get_perf_domains_doit(struct sk_buff *skb, - struct genl_info *info) + struct genl_info *info) { - int id, ret = -EMSGSIZE, msg_sz = 0; - int cmd = info->genlhdr->cmd; - struct em_perf_domain *pd; + struct em_nl_doit_ctx ctx = { + .info = info, + .cmd = info->genlhdr->cmd, + }; struct sk_buff *msg; - void *hdr; + int id, ret, msg_sz = 0; if (!info->attrs[DEV_ENERGYMODEL_A_PERF_DOMAIN_PERF_DOMAIN_ID]) return -EINVAL; id = nla_get_u32(info->attrs[DEV_ENERGYMODEL_A_PERF_DOMAIN_PERF_DOMAIN_ID]); - pd = em_perf_domain_get_by_id(id); - if (!pd) - return -EINVAL; - __em_nl_get_pd_size(pd, &msg_sz); + /* Encode under em_pd_list_mutex, like the dumpit path. */ + ret = em_perf_domain_for_id(id, __em_nl_get_pd_size, &msg_sz); + if (ret) + return ret; + msg = genlmsg_new(msg_sz, GFP_KERNEL); if (!msg) return -ENOMEM; - hdr = genlmsg_put_reply(msg, info, &dev_energymodel_nl_family, 0, cmd); - if (!hdr) - goto out_free_msg; - - ret = __em_nl_get_pd(pd, msg); + ctx.msg = msg; + ret = em_perf_domain_for_id(id, __em_nl_get_pd_doit_fill, &ctx); if (ret) - goto out_cancel_msg; - genlmsg_end(msg, hdr); + goto out_free_msg; return genlmsg_reply(msg, info); -out_cancel_msg: - genlmsg_cancel(msg, hdr); out_free_msg: nlmsg_free(msg); return ret; @@ -148,19 +169,6 @@ int dev_energymodel_nl_get_perf_domains_dumpit(struct sk_buff *skb, return for_each_em_perf_domain(__em_nl_get_pd_for_dump, &ctx); } -static struct em_perf_domain *__em_nl_get_pd_table_id(struct nlattr **attrs) -{ - struct em_perf_domain *pd; - int id; - - if (!attrs[DEV_ENERGYMODEL_A_PERF_TABLE_PERF_DOMAIN_ID]) - return NULL; - - id = nla_get_u32(attrs[DEV_ENERGYMODEL_A_PERF_TABLE_PERF_DOMAIN_ID]); - pd = em_perf_domain_get_by_id(id); - return pd; -} - static int __em_nl_get_pd_table_size(const struct em_perf_domain *pd) { int id_sz, ps_sz; @@ -245,34 +253,58 @@ int __em_nl_get_pd_table(struct sk_buff *msg, const struct em_perf_domain *pd) return -EMSGSIZE; } +static int __em_nl_get_pd_table_size_cb(struct em_perf_domain *pd, void *data) +{ + *(int *)data = __em_nl_get_pd_table_size(pd); + return 0; +} + +static int __em_nl_get_pd_table_doit_fill(struct em_perf_domain *pd, + void *data) +{ + struct em_nl_doit_ctx *ctx = data; + void *hdr; + + hdr = genlmsg_put_reply(ctx->msg, ctx->info, &dev_energymodel_nl_family, 0, + ctx->cmd); + if (!hdr) + return -EMSGSIZE; + + if (__em_nl_get_pd_table(ctx->msg, pd)) + return -EMSGSIZE; + + genlmsg_end(ctx->msg, hdr); + return 0; +} + int dev_energymodel_nl_get_perf_table_doit(struct sk_buff *skb, - struct genl_info *info) + struct genl_info *info) { - int cmd = info->genlhdr->cmd; - int msg_sz, ret = -EMSGSIZE; - struct em_perf_domain *pd; + struct em_nl_doit_ctx ctx = { + .info = info, + .cmd = info->genlhdr->cmd, + }; struct sk_buff *msg; - void *hdr; + int id, ret, msg_sz; - pd = __em_nl_get_pd_table_id(info->attrs); - if (!pd) + if (!info->attrs[DEV_ENERGYMODEL_A_PERF_TABLE_PERF_DOMAIN_ID]) return -EINVAL; - msg_sz = __em_nl_get_pd_table_size(pd); + id = nla_get_u32(info->attrs[DEV_ENERGYMODEL_A_PERF_TABLE_PERF_DOMAIN_ID]); + + ret = em_perf_domain_for_id(id, __em_nl_get_pd_table_size_cb, &msg_sz); + if (ret) + return ret; msg = genlmsg_new(msg_sz, GFP_KERNEL); if (!msg) return -ENOMEM; - hdr = genlmsg_put_reply(msg, info, &dev_energymodel_nl_family, 0, cmd); - if (!hdr) - goto out_free_msg; - - ret = __em_nl_get_pd_table(msg, pd); + ctx.msg = msg; + ret = em_perf_domain_for_id(id, __em_nl_get_pd_table_doit_fill, &ctx); if (ret) goto out_free_msg; - genlmsg_end(msg, hdr); return genlmsg_reply(msg, info); out_free_msg: diff --git a/kernel/power/em_netlink.h b/kernel/power/em_netlink.h index 583d7f1c3939..bc98a3c278b9 100644 --- a/kernel/power/em_netlink.h +++ b/kernel/power/em_netlink.h @@ -12,7 +12,8 @@ #if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_NET) int for_each_em_perf_domain(int (*cb)(struct em_perf_domain*, void *), void *data); -struct em_perf_domain *em_perf_domain_get_by_id(int id); +int em_perf_domain_for_id(int id, int (*cb)(struct em_perf_domain *, void *), + void *data); void em_notify_pd_created(const struct em_perf_domain *pd); void em_notify_pd_deleted(const struct em_perf_domain *pd); void em_notify_pd_updated(const struct em_perf_domain *pd); @@ -24,9 +25,10 @@ int for_each_em_perf_domain(int (*cb)(struct em_perf_domain*, void *), return -EINVAL; } static inline -struct em_perf_domain *em_perf_domain_get_by_id(int id) +int em_perf_domain_for_id(int id, int (*cb)(struct em_perf_domain *, void *), + void *data) { - return NULL; + return -EINVAL; } static inline void em_notify_pd_created(const struct em_perf_domain *pd) {} diff --git a/kernel/power/energy_model.c b/kernel/power/energy_model.c index e610cf8e9a06..a76089e2ce8c 100644 --- a/kernel/power/energy_model.c +++ b/kernel/power/energy_model.c @@ -1031,7 +1031,9 @@ int for_each_em_perf_domain(int (*cb)(struct em_perf_domain*, void *), return 0; } -struct em_perf_domain *em_perf_domain_get_by_id(int id) +/* Run @cb on the matching domain with em_pd_list_mutex held. */ +int em_perf_domain_for_id(int id, int (*cb)(struct em_perf_domain *, void *), + void *data) { struct em_perf_domain *pd; @@ -1040,9 +1042,9 @@ struct em_perf_domain *em_perf_domain_get_by_id(int id) list_for_each_entry(pd, &em_pd_list, node) { if (pd->id == id) - return pd; + return cb(pd, data); } - return NULL; + return -EINVAL; } #endif diff --git a/kernel/resource.c b/kernel/resource.c index e60539a55541..08bd7579cf40 100644 --- a/kernel/resource.c +++ b/kernel/resource.c @@ -1695,10 +1695,15 @@ int devm_request_resource(struct device *dev, struct resource *root, *ptr = new; - conflict = request_resource_conflict(root, new); - if (conflict) { + /* Report a conflict while the lock still keeps it in the tree. */ + write_lock(&resource_lock); + conflict = __request_resource(root, new); + if (conflict) dev_err(dev, "resource collision: %pR conflicts with %s %pR\n", new, conflict->name, conflict); + write_unlock(&resource_lock); + + if (conflict) { devres_free(ptr); return -EBUSY; } diff --git a/kernel/sched/ext/ext.c b/kernel/sched/ext/ext.c index f1b9d52f624a..04167ed9b580 100644 --- a/kernel/sched/ext/ext.c +++ b/kernel/sched/ext/ext.c @@ -1102,6 +1102,18 @@ void schedule_dsq_reenq(struct scx_sched *sch, struct scx_dispatch_q *dsq, if (dsq->id == SCX_DSQ_LOCAL) { rq = container_of(dsq, struct rq, scx.local_dsq); + /* + * An inactive cpu's local DSQ holds only tasks that are bound + * to it or are being moved off, so a reenq of it is pointless. + * Once balance push is installed, queue_balance_callback() also + * drops the callback that would run a reenq requested from + * dispatch, and irq_work_queue_on() can't reach an offline cpu. + * The reenq's node would then stay on this rq's list after + * @sch's pcpu is freed. + */ + if (unlikely(!cpu_active(cpu_of(rq)))) + return; + /* * A sub-sched lacking baseline access on the target cid has no * business triggering IPIs. The lockless test is fine: slipping @@ -10020,7 +10032,8 @@ __bpf_kfunc struct task_struct *scx_bpf_dsq_peek(u64 dsq_id, * - Local DSQs (%SCX_DSQ_LOCAL or %SCX_DSQ_LOCAL_ON | $cpu) * - User DSQs * - * Re-enqueues are performed asynchronously. Can be called from anywhere. + * Re-enqueues are performed asynchronously. Can be called from anywhere. A + * re-enqueue of an inactive cpu's local DSQ is ignored. * * %SCX_DSQ_LOCAL resolves to the local DSQ of the rq the current scheduler * operation is locked to - e.g. the rq being dispatched for in ops.dispatch() - diff --git a/lib/klist.c b/lib/klist.c index 332a4fbf18ff..f133740b1c2c 100644 --- a/lib/klist.c +++ b/lib/klist.c @@ -36,6 +36,7 @@ #include #include #include +#include /* * Use the lowest bit of n_klist to mark deleted nodes and exclude @@ -187,18 +188,24 @@ static void klist_release(struct kref *kref) WARN_ON(!knode_dead(n)); list_del(&n->n_node); + knode_set_klist(n, NULL); spin_lock(&klist_remove_lock); list_for_each_entry_safe(waiter, tmp, &klist_remove_waiters, list) { + struct task_struct *p; + if (waiter->node != n) continue; + p = waiter->process; + get_task_struct(p); list_del(&waiter->list); - waiter->woken = 1; + /* Publish only after the final waiter and n accesses */ + smp_store_release(&waiter->woken, 1); mb(); - wake_up_process(waiter->process); + wake_up_process(p); + put_task_struct(p); } spin_unlock(&klist_remove_lock); - knode_set_klist(n, NULL); } static int klist_dec_and_del(struct klist_node *n) @@ -250,7 +257,8 @@ void klist_remove(struct klist_node *n) for (;;) { set_current_state(TASK_UNINTERRUPTIBLE); - if (waiter.woken) + /* Pairs with the release store in klist_release() */ + if (smp_load_acquire(&waiter.woken)) break; schedule(); } diff --git a/lib/kobject_uevent.c b/lib/kobject_uevent.c index ddbc4d7482d2..6ecd39e4b704 100644 --- a/lib/kobject_uevent.c +++ b/lib/kobject_uevent.c @@ -223,9 +223,9 @@ int kobject_synth_uevent(struct kobject *kobj, const char *buf, size_t count) out: if (r) { devpath = kobject_get_path(kobj, GFP_KERNEL); - pr_warn("synth uevent: %s: %s\n", - devpath ?: "unknown device", - msg ?: "failed to send uevent"); + pr_warn_ratelimited("synth uevent: %s: %s\n", + devpath ?: "unknown device", + msg ?: "failed to send uevent"); kfree(devpath); } return r; diff --git a/mm/hugetlb.c b/mm/hugetlb.c index cea25773a6c9..86928d76ef12 100644 --- a/mm/hugetlb.c +++ b/mm/hugetlb.c @@ -5949,8 +5949,6 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, ret = hugetlb_wp(vmf); } - spin_unlock(vmf->ptl); - /* * Only set hugetlb_migratable in newly allocated pages. Existing pages * found in the pagecache may not have hugetlb_migratable if they have @@ -5961,6 +5959,12 @@ static vm_fault_t hugetlb_no_page(struct address_space *mapping, if (folio_locked) folio_unlock(folio); + + /* + * Hold the PTL until we're done with the folio: the page table owns the + * folio reference now. + */ + spin_unlock(vmf->ptl); out: hugetlb_vma_unlock_read(vma); @@ -6474,11 +6478,16 @@ int hugetlb_mfill_atomic_pte(pte_t *dst_pte, /* No need to invalidate - it was non-present before */ update_mmu_cache(dst_vma, dst_addr, dst_pte); - spin_unlock(ptl); if (!is_continue) folio_set_hugetlb_migratable(folio); if (vm_shared || is_continue) folio_unlock(folio); + + /* + * Hold the PTL until we're done with the folio: the page table owns the + * folio reference now. + */ + spin_unlock(ptl); ret = 0; out: return ret; diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 75639298efc2..792166950bba 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -1612,6 +1612,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, enum scan_result result = SCAN_FAIL; struct page *page = NULL; struct folio *folio = NULL; + unsigned long failed_pfn = -1; unsigned long addr; unsigned long enabled_orders; spinlock_t *ptl; @@ -1706,11 +1707,13 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, if (cc->is_khugepaged && !(vma->vm_flags & VM_DROPPABLE) && folio_test_lazyfree(folio) && !pte_dirty(pteval)) { result = SCAN_PAGE_LAZYFREE; + failed_pfn = folio_pfn(folio); goto out_unmap; } if (!folio_test_anon(folio)) { result = SCAN_PAGE_ANON; + failed_pfn = folio_pfn(folio); goto out_unmap; } @@ -1721,6 +1724,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, if (folio_maybe_mapped_shared(folio)) { if (++shared > max_ptes_shared) { result = SCAN_EXCEED_SHARED_PTE; + failed_pfn = folio_pfn(folio); count_collapse_event(HPAGE_PMD_ORDER, THP_SCAN_EXCEED_SHARED_PTE, MTHP_STAT_COLLAPSE_EXCEED_SHARED); goto out_unmap; @@ -1738,15 +1742,18 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, node = folio_nid(folio); if (collapse_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; + failed_pfn = folio_pfn(folio); goto out_unmap; } cc->node_load[node]++; if (!folio_test_lru(folio)) { result = SCAN_PAGE_LRU; + failed_pfn = folio_pfn(folio); goto out_unmap; } if (folio_test_locked(folio)) { result = SCAN_PAGE_LOCK; + failed_pfn = folio_pfn(folio); goto out_unmap; } @@ -1759,6 +1766,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, */ if (folio_expected_ref_count(folio) != folio_ref_count(folio)) { result = SCAN_PAGE_COUNT; + failed_pfn = folio_pfn(folio); goto out_unmap; } @@ -1784,7 +1792,7 @@ static enum scan_result collapse_scan_pmd(struct mm_struct *mm, *lock_dropped = true; } out: - trace_mm_khugepaged_scan_pmd(mm, folio, referenced, + trace_mm_khugepaged_scan_pmd(mm, failed_pfn, referenced, none_or_zero, result, unmapped); return result; } @@ -2245,6 +2253,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, struct address_space *mapping = file->f_mapping; struct page *dst; struct folio *folio, *tmp, *new_folio; + unsigned long new_pfn = -1; pgoff_t index = 0, end = start + HPAGE_PMD_NR; LIST_HEAD(pagelist); XA_STATE_ORDER(xas, &mapping->i_pages, start, HPAGE_PMD_ORDER); @@ -2264,6 +2273,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, result = alloc_charge_folio(&new_folio, mm, cc, HPAGE_PMD_ORDER); if (result != SCAN_SUCCEED) goto out; + new_pfn = folio_pfn(new_folio); mapping_set_update(&xas, mapping); @@ -2667,7 +2677,7 @@ static enum scan_result collapse_file(struct mm_struct *mm, unsigned long addr, folio_put(new_folio); out: VM_BUG_ON(!list_empty(&pagelist)); - trace_mm_khugepaged_collapse_file(mm, new_folio, index, addr, is_shmem, file, HPAGE_PMD_NR, result); + trace_mm_khugepaged_collapse_file(mm, new_pfn, index, addr, is_shmem, file, HPAGE_PMD_NR, result); return result; } @@ -2683,6 +2693,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, int present, swap; int node = NUMA_NO_NODE; enum scan_result result = SCAN_SUCCEED; + unsigned long failed_pfn = -1; present = 0; swap = 0; @@ -2715,6 +2726,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, if (is_pmd_order(folio_order(folio))) { result = SCAN_PTE_MAPPED_HUGEPAGE; + failed_pfn = folio_pfn(folio); /* * PMD-sized THP implies that we can only try * retracting the PTE table. @@ -2726,6 +2738,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, node = folio_nid(folio); if (collapse_scan_abort(node, cc)) { result = SCAN_SCAN_ABORT; + failed_pfn = folio_pfn(folio); folio_put(folio); break; } @@ -2733,12 +2746,14 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, if (!folio_test_lru(folio)) { result = SCAN_PAGE_LRU; + failed_pfn = folio_pfn(folio); folio_put(folio); break; } if (folio_expected_ref_count(folio) + 1 != folio_ref_count(folio)) { result = SCAN_PAGE_COUNT; + failed_pfn = folio_pfn(folio); folio_put(folio); break; } @@ -2773,7 +2788,7 @@ static enum scan_result collapse_scan_file(struct mm_struct *mm, } } - trace_mm_khugepaged_scan_file(mm, folio, file, present, swap, result); + trace_mm_khugepaged_scan_file(mm, failed_pfn, file, present, swap, result); return result; } diff --git a/mm/rmap.c b/mm/rmap.c index f3b21aaa34ee..6b3480330b92 100644 --- a/mm/rmap.c +++ b/mm/rmap.c @@ -2275,17 +2275,6 @@ static bool try_to_unmap_one(struct folio *folio, struct vm_area_struct *vma, } if (!pvmw.pte) { - if (folio_test_lazyfree(folio)) { - if (unmap_huge_pmd_locked(vma, pvmw.address, pvmw.pmd, folio)) - goto walk_done; - /* - * unmap_huge_pmd_locked has either already marked - * the folio as swap-backed or decided to retain it - * due to GUP or speculative references. - */ - goto walk_abort; - } - if (flags & TTU_SPLIT_HUGE_PMD) { /* * We temporarily have to drop the PTL and diff --git a/mm/slab_common.c b/mm/slab_common.c index 7223a7596dab..de11edfd1a0b 100644 --- a/mm/slab_common.c +++ b/mm/slab_common.c @@ -515,10 +515,10 @@ EXPORT_SYMBOL(kmem_buckets_create); * and release of the kobject does not need slab_mutex or cpu_hotplug_lock * protection. So they are now done without holding those locks. */ -static void kmem_cache_release(struct kmem_cache *s) +static void kmem_cache_release(struct kmem_cache *s, bool sysfs_ready) { kfence_shutdown_cache(s); - if (__is_defined(SLAB_SUPPORTS_SYSFS) && slab_state >= FULL) + if (__is_defined(SLAB_SUPPORTS_SYSFS) && sysfs_ready) sysfs_slab_release(s); else slab_kmem_cache_release(s); @@ -533,6 +533,7 @@ void slab_kmem_cache_release(struct kmem_cache *s) void kmem_cache_destroy(struct kmem_cache *s) { + bool sysfs_ready; int err; if (unlikely(!s) || !kasan_check_byte(s)) @@ -580,10 +581,18 @@ void kmem_cache_destroy(struct kmem_cache *s) list_del(&s->list); + /* + * slab_late_init() sets slab_state to FULL under slab_mutex and adds + * sysfs entries only for caches still on the list. Sample the state + * here, so that a cache unlinked before that point is not handed to + * sysfs_slab_release() with an uninitialized kobject. + */ + sysfs_ready = slab_state >= FULL; + mutex_unlock(&slab_mutex); cpus_read_unlock(); - if (slab_state >= FULL) + if (sysfs_ready) sysfs_slab_unlink(s); debugfs_slab_release(s); @@ -593,7 +602,7 @@ void kmem_cache_destroy(struct kmem_cache *s) if (s->flags & SLAB_TYPESAFE_BY_RCU) rcu_barrier(); - kmem_cache_release(s); + kmem_cache_release(s, sysfs_ready); } EXPORT_SYMBOL(kmem_cache_destroy); diff --git a/mm/truncate.c b/mm/truncate.c index b58ba940be47..7912d3123170 100644 --- a/mm/truncate.c +++ b/mm/truncate.c @@ -259,30 +259,32 @@ bool truncate_inode_partial_folio(struct folio *folio, loff_t start, loff_t end) * for shmem truncate */ struct folio *folio2; + pgoff_t end_idx; if (offset + length == size) goto no_split; - split_at2 = folio_page(folio, - PAGE_ALIGN_DOWN(offset + length) / PAGE_SIZE); - folio2 = page_folio(split_at2); - - if (!folio_try_get(folio2)) + /* + * After the first split at the start edge, the folio at the + * end edge may be freed and reused concurrently. + * __filemap_get_folio() looks up the straddler at end_idx + * and returns it locked and ref'd with the mapping + * validated. + */ + end_idx = (pos + offset + length) >> PAGE_SHIFT; + folio2 = __filemap_get_folio(folio->mapping, end_idx, + FGP_LOCK | FGP_NOWAIT, 0); + if (IS_ERR(folio2)) goto no_split; + /* make sure folio2 is large */ if (!folio_test_large(folio2)) goto out; - if (!folio_trylock(folio2)) - goto out; - - /* make sure folio2 is large and does not change its mapping */ - if (folio_test_large(folio2) && - folio2->mapping == folio->mapping) - folio_split_or_unmap(folio2, split_at2, min_order); - - folio_unlock(folio2); + split_at2 = folio_page(folio2, (end_idx - folio2->index)); + folio_split_or_unmap(folio2, split_at2, min_order); out: + folio_unlock(folio2); folio_put(folio2); no_split: return true; diff --git a/net/8021q/vlan_dev.c b/net/8021q/vlan_dev.c index c949c6a82945..2d775b4c84ce 100644 --- a/net/8021q/vlan_dev.c +++ b/net/8021q/vlan_dev.c @@ -110,6 +110,8 @@ static netdev_tx_t vlan_dev_hard_start_xmit(struct sk_buff *skb, unsigned int len; int ret; + len = skb->len; + /* Handle non-VLAN frames if they are sent to us, for example by DHCP. * * NOTE: THIS ASSUMES DIX ETHERNET, SPECIFICALLY NOT SUPPORTING @@ -121,10 +123,39 @@ static netdev_tx_t vlan_dev_hard_start_xmit(struct sk_buff *skb, vlan_tci = vlan->vlan_id; vlan_tci |= vlan_dev_get_egress_qos_mask(dev, skb->priority); __vlan_hwaccel_put_tag(skb, vlan->vlan_proto, vlan_tci); + } else if (skb_headlen(skb) >= VLAN_ETH_HLEN + sizeof(__be16) && + !skb_shared(skb) && !skb_vlan_tag_present(skb) && + !skb_metadata_len(skb) && + (skb->ip_summed != CHECKSUM_PARTIAL || + skb_checksum_start_offset(skb) >= VLAN_ETH_HLEN) && + vlan_hw_offload_capable(READ_ONCE(vlan->real_dev->features), + vlan->vlan_proto)) { + u16 vlan_tci; + + /* The frame carries its outer tag in-band, either inserted + * by vlan_dev_hard_header() or provided by the sender. + * Let the real device insert it: some NICs can not offload + * checksum or TSO for frames carrying an in-band tag. + * Not all ndo_start_xmit() callers set the mac header. + * vlan_set_encap_proto() may read two bytes after the vlan + * header, so frames with a shorter linear part are sent + * unchanged, as are shared skbs (e.g. from pktgen), skbs + * already carrying a hwaccel tag (e.g. from generic XDP + * redirect), skbs carrying XDP metadata, and CHECKSUM_PARTIAL + * skbs whose checksum starts within the header to be popped. + * On allocation failure, drop the frame rather than handing + * it to the real device with an in-band tag. + */ + skb_reset_mac_header(skb); + if (unlikely(__skb_vlan_pop(skb, &vlan_tci))) { + dev_kfree_skb_any(skb); + this_cpu_inc(vlan->vlan_pcpu_stats->tx_dropped); + return NETDEV_TX_OK; + } + __vlan_hwaccel_put_tag(skb, vlan->vlan_proto, vlan_tci); } skb->dev = vlan->real_dev; - len = skb->len; if (unlikely(netpoll_tx_running(dev))) return vlan_netpoll_send_skb(vlan, skb); diff --git a/net/bluetooth/hci_event.c b/net/bluetooth/hci_event.c index 2f5e21ff9752..03b805207ab0 100644 --- a/net/bluetooth/hci_event.c +++ b/net/bluetooth/hci_event.c @@ -6311,6 +6311,15 @@ static void process_adv_report(struct hci_dev *hdev, u8 type, bdaddr_t *bdaddr, if (irk) { bdaddr = &irk->bdaddr; bdaddr_type = irk->addr_type; + } else { + /* The peer is on air with its identity address, so whatever + * RPA is cached for it has been abandoned. Drop it, or + * hci_connect_le() would swap it back in and dial an address + * the peer no longer answers. + */ + irk = hci_find_irk_by_addr(hdev, bdaddr, bdaddr_type); + if (irk) + bacpy(&irk->rpa, BDADDR_ANY); } bdaddr_type = ev_bdaddr_type(hdev, bdaddr_type, &bdaddr_resolved); diff --git a/net/ceph/mon_client.c b/net/ceph/mon_client.c index c56457378d00..59b304eca273 100644 --- a/net/ceph/mon_client.c +++ b/net/ceph/mon_client.c @@ -943,7 +943,7 @@ static void handle_command_ack(struct ceph_mon_client *monc, { struct ceph_mon_generic_request *req; void *p = msg->front.iov_base; - void *const end = p + msg->front_alloc_len; + void *const end = p + msg->front.iov_len; u64 tid = le64_to_cpu(msg->hdr.tid); dout("%s msg %p tid %llu\n", __func__, msg, tid); diff --git a/net/core/dev.c b/net/core/dev.c index e76762e29360..3f39d953eb48 100644 --- a/net/core/dev.c +++ b/net/core/dev.c @@ -3440,6 +3440,9 @@ static void __netif_reschedule(struct Qdisc *q) void __netif_schedule(struct Qdisc *q) { + if (unlikely(test_bit(__QDISC_STATE_DEACTIVATED, &q->state))) + return; + /* If q->defer_list is not empty, at least one thread is * in __dev_xmit_skb() before llist_del_all(&q->defer_list). * This thread will attempt to run the queue. diff --git a/net/core/rtnetlink.c b/net/core/rtnetlink.c index be9d1625bac3..580101b5ab58 100644 --- a/net/core/rtnetlink.c +++ b/net/core/rtnetlink.c @@ -707,6 +707,10 @@ void rtnl_link_unregister(struct rtnl_link_ops *ops) list_del_rcu(&ops->list); mutex_unlock(&link_ops_mutex); + /* Wait for rtnl_link_ops_get() readers that found ops but have + * not yet taken ops->srcu. + */ + synchronize_rcu(); synchronize_srcu(&ops->srcu); cleanup_srcu_struct(&ops->srcu); diff --git a/net/core/skbuff.c b/net/core/skbuff.c index 41beaf625421..2bfabde31ea9 100644 --- a/net/core/skbuff.c +++ b/net/core/skbuff.c @@ -3942,6 +3942,8 @@ skb_zerocopy(struct sk_buff *to, struct sk_buff *from, int len, int hlen) } skb_shinfo(to)->nr_frags = j; + skb_shinfo(to)->flags |= skb_shinfo(from)->flags & SKBFL_SHARED_FRAG; + if (i > 0 && from->unreadable) to->unreadable = 1; diff --git a/net/ipv4/inet_connection_sock.c b/net/ipv4/inet_connection_sock.c index 6a30f1138454..c6f8011200a0 100644 --- a/net/ipv4/inet_connection_sock.c +++ b/net/ipv4/inet_connection_sock.c @@ -21,6 +21,7 @@ #include #include #include +#include #include #include @@ -961,6 +962,9 @@ static struct request_sock *inet_reqsk_clone(struct request_sock *req, rcu_assign_pointer(tcp_sk(nreq->sk)->fastopen_rsk, nreq); } + if (rsk_is_mptcp(req)) + mptcp_subflow_reqsk_clone(req, nreq); + return nreq; } diff --git a/net/ipv4/metrics.c b/net/ipv4/metrics.c index ad40762a8b38..4052a90249c6 100644 --- a/net/ipv4/metrics.c +++ b/net/ipv4/metrics.c @@ -60,6 +60,11 @@ static int ip_metrics_convert(struct nlattr *fc_mx, if (ecn_ca) metrics[RTAX_FEATURES - 1] |= DST_FEATURE_ECN_CA; + if ((metrics[RTAX_LOCK - 1] & (1 << RTAX_CWND)) && !metrics[RTAX_CWND - 1]) { + NL_SET_ERR_MSG(extack, "Locked cwnd metric must be non zero"); + return -EINVAL; + } + return 0; } diff --git a/net/mptcp/subflow.c b/net/mptcp/subflow.c index f0a6725d2c37..3139a9ab2853 100644 --- a/net/mptcp/subflow.c +++ b/net/mptcp/subflow.c @@ -47,6 +47,39 @@ static void subflow_req_destructor(struct request_sock *req) mptcp_token_destroy_request(req); } +void mptcp_subflow_reqsk_clone(struct request_sock *req, + struct request_sock *new_req) +{ + struct mptcp_subflow_request_sock *subflow_req = mptcp_subflow_rsk(req); + struct mptcp_subflow_request_sock *new_subflow_req; + struct mptcp_sock *msk; + + new_subflow_req = mptcp_subflow_rsk(new_req); + RCU_LOCKDEP_WARN(!rcu_read_lock_any_held(), + "MPTCP reqsk clone called without RCU"); + + /* A non-NULL ->msk means the request owns one reference. The clone + * copied only the pointer, while the original request can concurrently + * transfer its reference to the child. Acquire a reference for the + * clone, then verify that the original request still owns the same msk. + * MPTCP sockets use SLAB_TYPESAFE_BY_RCU and all clone callers run in + * an RCU read-side critical section, keeping the memory stable here. + */ + msk = READ_ONCE(subflow_req->msk); + if (msk) { + struct sock *sk = (struct sock *)msk; + + if (!refcount_inc_not_zero_acquire(&sk->sk_refcnt)) { + msk = NULL; + } else if (READ_ONCE(subflow_req->msk) != msk) { + sock_put(sk); + msk = NULL; + } + } + + new_subflow_req->msk = msk; +} + static void subflow_generate_hmac(u64 key1, u64 key2, u32 nonce1, u32 nonce2, void *hmac) { @@ -927,7 +960,7 @@ static struct sock *subflow_syn_recv_sock(const struct sock *sk, } /* move the msk reference ownership to the subflow */ - subflow_req->msk = NULL; + WRITE_ONCE(subflow_req->msk, NULL); ctx->conn = (struct sock *)owner; if (subflow_use_different_sport(owner, sk)) { diff --git a/net/netfilter/nft_payload.c b/net/netfilter/nft_payload.c index e315d35f73d4..5b76af68b704 100644 --- a/net/netfilter/nft_payload.c +++ b/net/netfilter/nft_payload.c @@ -20,6 +20,7 @@ #include #include #include +#include #include #include #include @@ -1008,11 +1009,14 @@ static bool nft_payload_validate_inet_csum_offset(const struct nft_ctx *ctx, if (priv->csum_flags) /* makes no sense, asks for "re-update" of L4 checksum */ return false; - /* no further check here; offset can't be negative so bogus - * offsets can corrupt L4 or payload but not l3 headers. - * We already allow arbitrary l4/inner payload writes. - */ - return true; + /* Validate csum_offset is one of the supported transport header checksums */ + if (priv->csum_offset == offsetof(struct tcphdr, check) || + priv->csum_offset == offsetof(struct udphdr, check) || + priv->csum_offset == offsetof(struct icmphdr, checksum) || + priv->csum_offset == offsetof(struct icmp6hdr, icmp6_cksum)) + return true; + + return false; case NFT_PAYLOAD_INNER_HEADER: return true; case NFT_PAYLOAD_TUN_HEADER: @@ -1046,6 +1050,27 @@ static bool nft_payload_csum_nh_write_ok(const struct nft_payload_set *priv, return false; } +static bool nft_payload_csum_th_write_ok(const struct nft_payload_set *priv, + const struct nft_pktinfo *pkt) +{ + if (!(pkt->flags & NFT_PKTINFO_L4PROTO)) + return false; + + switch (pkt->tprot) { + case IPPROTO_TCP: + return priv->csum_offset == offsetof(struct tcphdr, check); + case IPPROTO_UDP: + case IPPROTO_UDPLITE: + return priv->csum_offset == offsetof(struct udphdr, check); + case IPPROTO_ICMPV6: + return priv->csum_offset == offsetof(struct icmp6hdr, icmp6_cksum); + case IPPROTO_ICMP: + return priv->csum_offset == offsetof(struct icmphdr, checksum); + } + + return false; +} + static bool nft_payload_csum_write_ok(const struct nft_pktinfo *pkt, const struct nft_payload_set *priv) { @@ -1055,9 +1080,10 @@ static bool nft_payload_csum_write_ok(const struct nft_pktinfo *pkt, case NFT_PAYLOAD_NETWORK_HEADER: return nft_payload_csum_nh_write_ok(priv, pkt); case NFT_PAYLOAD_TRANSPORT_HEADER: + return nft_payload_csum_th_write_ok(priv, pkt); case NFT_PAYLOAD_INNER_HEADER: - /* neither offsets are validated, offsets cannot be - * negative so real l3 headers cannot be mangled. + /* offset is not validated, offset cannot be + * negative so real l3/l4 headers cannot be mangled. */ return true; case NFT_PAYLOAD_TUN_HEADER: diff --git a/net/rxrpc/recvmsg.c b/net/rxrpc/recvmsg.c index efcba4b2e74f..0c960f13b5fc 100644 --- a/net/rxrpc/recvmsg.c +++ b/net/rxrpc/recvmsg.c @@ -24,6 +24,7 @@ void rxrpc_notify_socket(struct rxrpc_call *call) { struct rxrpc_sock *rx; struct sock *sk; + unsigned long flags; _enter("%d", call->debug_id); @@ -38,16 +39,16 @@ void rxrpc_notify_socket(struct rxrpc_call *call) sk = &rx->sk; if (rx && sk->sk_state < RXRPC_CLOSE) { if (call->notify_rx) { - spin_lock_irq(&call->notify_lock); + spin_lock_irqsave(&call->notify_lock, flags); call->notify_rx(sk, call, call->user_call_ID); - spin_unlock_irq(&call->notify_lock); + spin_unlock_irqrestore(&call->notify_lock, flags); } else { - spin_lock_irq(&rx->recvmsg_lock); + spin_lock_irqsave(&rx->recvmsg_lock, flags); if (list_empty(&call->recvmsg_link)) { rxrpc_get_call(call, rxrpc_call_get_notify_socket); list_add_tail(&call->recvmsg_link, &rx->recvmsg_q); } - spin_unlock_irq(&rx->recvmsg_lock); + spin_unlock_irqrestore(&rx->recvmsg_lock, flags); if (!sock_flag(sk, SOCK_DEAD)) { _debug("call %ps", sk->sk_data_ready); @@ -637,9 +638,11 @@ int rxrpc_recvmsg(struct socket *sock, struct msghdr *msg, size_t len, * Note that we may return %-EAGAIN to drain empty packets at the end * of the data, even if we've already copied over the requested data. * - * Return: %0 if got what was asked for and there's more available, %1 - * if we got what was asked for and we're at the end of the data and - * %-EAGAIN if we need more data. + * Return: %0 if got what was asked for and there's more available, %1 if we + * got what was asked for and we're at the end of the call, %2 if a service + * call received all of the request but is still in progress and %-EAGAIN if we + * need more data. A variety of other errors can be returned if the call + * completed with failure. */ int rxrpc_kernel_recv_data(struct socket *sock, struct rxrpc_call *call, struct iov_iter *iter, size_t *_len, @@ -678,6 +681,11 @@ int rxrpc_kernel_recv_data(struct socket *sock, struct rxrpc_call *call, read_phase_complete: ret = 1; + if (rxrpc_is_service_call(call)) { + if (rxrpc_call_is_complete(call)) + goto call_failed; + ret = 2; + } out: if (_service) *_service = call->dest_srx.srx_service; diff --git a/net/rxrpc/rxperf.c b/net/rxrpc/rxperf.c index b8df6d22314d..6a815a4bb314 100644 --- a/net/rxrpc/rxperf.c +++ b/net/rxrpc/rxperf.c @@ -293,8 +293,20 @@ static void rxperf_deliver_to_call(struct work_struct *work) state == RXPERF_CALL_SV_AWAIT_ACK ) { if (state == RXPERF_CALL_SV_AWAIT_ACK) { - if (!rxrpc_kernel_check_life(rxperf_socket, call->rxcall)) + size_t len = 0; + iov_iter_kvec(&call->iter, ITER_DEST, NULL, 0, 0); + ret = rxrpc_kernel_recv_data(rxperf_socket, + call->rxcall, &call->iter, + &len, false, &remote_abort, + &call->service_id); + + if (ret == -EAGAIN || ret == 2) + return; + if (ret < 0 || ret == 1) { + if (ret == 1) + ret = 0; goto call_complete; + } return; } @@ -369,7 +381,7 @@ static int rxperf_extract_data(struct rxperf_call *call, bool want_more) if (ret == 0 || ret == -EAGAIN) return ret; - if (ret == 1) { + if (ret == 1 || ret == 2) { switch (call->state) { case RXPERF_CALL_SV_AWAIT_REQUEST: rxperf_set_call_state(call, RXPERF_CALL_SV_REPLYING); @@ -513,8 +525,8 @@ static int rxperf_process_call(struct rxperf_call *call) struct msghdr msg = {}; struct bio_vec bv; struct kvec iov[1]; - ssize_t n; size_t reply_len = call->reply_len, len; + int ret; rxrpc_kernel_set_tx_length(rxperf_socket, call->rxcall, reply_len + sizeof(rxperf_magic_cookie)); @@ -524,13 +536,11 @@ static int rxperf_process_call(struct rxperf_call *call) bvec_set_page(&bv, ZERO_PAGE(0), len, 0); iov_iter_bvec(&msg.msg_iter, WRITE, &bv, 1, len); msg.msg_flags = MSG_MORE; - n = rxrpc_kernel_send_data(rxperf_socket, call->rxcall, &msg, - len, rxperf_notify_end_reply_tx); - if (n < 0) - return n; - if (n == 0) - return -EIO; - reply_len -= n; + ret = rxrpc_kernel_send_data(rxperf_socket, call->rxcall, &msg, + rxperf_notify_end_reply_tx); + if (ret < 0) + return ret; + reply_len -= len; } len = sizeof(rxperf_magic_cookie); @@ -538,16 +548,13 @@ static int rxperf_process_call(struct rxperf_call *call) iov[0].iov_len = len; iov_iter_kvec(&msg.msg_iter, WRITE, iov, 1, len); msg.msg_flags = 0; - n = rxrpc_kernel_send_data(rxperf_socket, call->rxcall, &msg, len, - rxperf_notify_end_reply_tx); - if (n >= 0) - return 0; /* Success */ - - if (n == -ENOMEM) + ret = rxrpc_kernel_send_data(rxperf_socket, call->rxcall, &msg, + rxperf_notify_end_reply_tx); + if (ret == -ENOMEM) rxrpc_kernel_abort_call(rxperf_socket, call->rxcall, RXGEN_SS_MARSHAL, -ENOMEM, rxperf_abort_oom); - return n; + return ret; } /* diff --git a/net/rxrpc/sendmsg.c b/net/rxrpc/sendmsg.c index ed2c9a51005a..393a2dcfda07 100644 --- a/net/rxrpc/sendmsg.c +++ b/net/rxrpc/sendmsg.c @@ -320,10 +320,9 @@ static int rxrpc_alloc_txqueue(struct sock *sk, struct rxrpc_call *call) static int rxrpc_send_data(struct rxrpc_sock *rx, struct rxrpc_call *call, struct msghdr *msg, size_t len, - rxrpc_notify_end_tx_t notify_end_tx, - bool *_dropped_lock) + rxrpc_notify_end_tx_t notify_end_tx) + __releases(&call->user_mutex) { - struct rxrpc_txbuf *txb; struct sock *sk = &rx->sk; enum rxrpc_call_state state; long timeo; @@ -334,30 +333,26 @@ static int rxrpc_send_data(struct rxrpc_sock *rx, trace_rxrpc_abort(call->debug_id, rxrpc_sendmsg_late_send, call->cid, call->call_id, call->rx_consumed, 0, -EPROTO); - return -EPROTO; + ret = -EPROTO; + goto out_unlock; } timeo = sock_sndtimeo(sk, msg->msg_flags & MSG_DONTWAIT); ret = rxrpc_wait_to_be_connected(call, &timeo); if (ret < 0) - return ret; + goto out_unlock; if (call->conn->state == RXRPC_CONN_CLIENT_UNSECURED) { ret = rxrpc_init_client_conn_security(call->conn); if (ret < 0) - return ret; + goto out_unlock; } /* this should be in poll */ sk_clear_bit(SOCKWQ_ASYNC_NOSPACE, sk); reload: - txb = call->tx_pending; - call->tx_pending = NULL; - if (txb) - rxrpc_see_txbuf(txb, rxrpc_txbuf_see_send_more); - ret = -EPIPE; if (sk->sk_shutdown & SEND_SHUTDOWN) goto maybe_error; @@ -386,6 +381,8 @@ static int rxrpc_send_data(struct rxrpc_sock *rx, } do { + struct rxrpc_txbuf *txb = call->tx_pending; + if (!txb) { size_t remain; @@ -411,6 +408,9 @@ static int rxrpc_send_data(struct rxrpc_sock *rx, ret = -ENOMEM; goto maybe_error; } + call->tx_pending = txb; + } else { + rxrpc_see_txbuf(txb, rxrpc_txbuf_see_send_more); } _debug("append"); @@ -445,9 +445,9 @@ static int rxrpc_send_data(struct rxrpc_sock *rx, ret = call->security->secure_packet(call, txb); if (ret < 0) - goto out; + goto out_unlock; rxrpc_queue_packet(rx, call, txb, notify_end_tx); - txb = NULL; + call->tx_pending = NULL; } } while (msg_data_left(msg) > 0); @@ -456,45 +456,46 @@ static int rxrpc_send_data(struct rxrpc_sock *rx, if (rxrpc_call_is_complete(call) && call->error < 0) ret = call->error; -out: - call->tx_pending = txb; +out_unlock: + mutex_unlock(&call->user_mutex); _leave(" = %d", ret); return ret; call_terminated: - rxrpc_put_txbuf(txb, rxrpc_txbuf_put_send_aborted); - _leave(" = %d", call->error); - return call->error; + ret = call->error; + goto out_unlock; maybe_error: if (copied) goto success; - goto out; + goto out_unlock; efault: ret = -EFAULT; - goto out; + goto out_unlock; wait_for_space: ret = -EAGAIN; if (msg->msg_flags & MSG_DONTWAIT) goto maybe_error; mutex_unlock(&call->user_mutex); - *_dropped_lock = true; + ret = rxrpc_wait_for_tx_window(rx, call, &timeo, msg->msg_flags & MSG_WAITALL); if (ret < 0) - goto maybe_error; + goto out_nolock; if (call->interruptibility == RXRPC_INTERRUPTIBLE) { if (mutex_lock_interruptible(&call->user_mutex) < 0) { ret = sock_intr_errno(timeo); - goto maybe_error; + goto out_nolock; } } else { mutex_lock(&call->user_mutex); } - *_dropped_lock = false; goto reload; +out_nolock: + _leave(" = %d [intr]", ret); + return copied ?: ret; } /* @@ -660,7 +661,6 @@ rxrpc_new_client_call_for_sendmsg(struct rxrpc_sock *rx, struct msghdr *msg, int rxrpc_do_sendmsg(struct rxrpc_sock *rx, struct msghdr *msg, size_t len) { struct rxrpc_call *call; - bool dropped_lock = false; int ret; struct rxrpc_send_params p = { @@ -769,16 +769,15 @@ int rxrpc_do_sendmsg(struct rxrpc_sock *rx, struct msghdr *msg, size_t len) ret = 0; break; case RXRPC_CMD_SEND_DATA: - ret = rxrpc_send_data(rx, call, msg, len, NULL, &dropped_lock); - break; + ret = rxrpc_send_data(rx, call, msg, len, NULL); + goto error_put; default: ret = -EINVAL; break; } out_put_unlock: - if (!dropped_lock) - mutex_unlock(&call->user_mutex); + mutex_unlock(&call->user_mutex); error_put: rxrpc_put_call(call, rxrpc_call_put_sendmsg); _leave(" = %d", ret); @@ -794,7 +793,6 @@ int rxrpc_do_sendmsg(struct rxrpc_sock *rx, struct msghdr *msg, size_t len) * @sock: The socket the call is on * @call: The call to send data through * @msg: The data to send - * @len: The amount of data to send * @notify_end_tx: Notification that the last packet is queued. * * Allow a kernel service to send data on a call. The call must be in an state @@ -805,10 +803,8 @@ int rxrpc_do_sendmsg(struct rxrpc_sock *rx, struct msghdr *msg, size_t len) * Return: %0 if successful and a negative error code otherwise. */ int rxrpc_kernel_send_data(struct socket *sock, struct rxrpc_call *call, - struct msghdr *msg, size_t len, - rxrpc_notify_end_tx_t notify_end_tx) + struct msghdr *msg, rxrpc_notify_end_tx_t notify_end_tx) { - bool dropped_lock = false; int ret; _enter("{%d},", call->debug_id); @@ -816,15 +812,25 @@ int rxrpc_kernel_send_data(struct socket *sock, struct rxrpc_call *call, ASSERTCMP(msg->msg_name, ==, NULL); ASSERTCMP(msg->msg_control, ==, NULL); - mutex_lock(&call->user_mutex); + for (;;) { + mutex_lock(&call->user_mutex); - ret = rxrpc_send_data(rxrpc_sk(sock->sk), call, msg, len, - notify_end_tx, &dropped_lock); - if (ret == -ESHUTDOWN) - ret = call->error; + ret = rxrpc_send_data(rxrpc_sk(sock->sk), call, msg, + msg_data_left(msg), notify_end_tx); + if (ret == -ESHUTDOWN) + ret = call->error; + if (ret < 0) + break; + if (msg_data_left(msg) == 0) { + ret = 0; + break; + } + if (ret == 0) { + ret = -EIO; + break; + } + } - if (!dropped_lock) - mutex_unlock(&call->user_mutex); _leave(" = %d", ret); return ret; } diff --git a/net/sched/act_ife.c b/net/sched/act_ife.c index 2afd68983ece..3946a9589bb8 100644 --- a/net/sched/act_ife.c +++ b/net/sched/act_ife.c @@ -769,6 +769,10 @@ static int tcf_ife_decode(struct sk_buff *skb, const struct tc_action *a, skb->protocol = eth_type_trans(skb, skb->dev); skb_reset_network_header(skb); + if (!skb_at_tc_ingress(skb)) { + skb_push(skb, ETH_HLEN); + skb_reset_mac_len(skb); + } return action; } diff --git a/net/sched/cls_bpf.c b/net/sched/cls_bpf.c index 188cf0f949dd..07ad9556889d 100644 --- a/net/sched/cls_bpf.c +++ b/net/sched/cls_bpf.c @@ -392,11 +392,31 @@ static int cls_bpf_prog_from_efd(struct nlattr **tb, struct cls_bpf_prog *prog, if (bpf_prog_is_dev_bound(fp->aux)) { struct tcf_block *block = tp->chain->block; struct net_device *dev; + unsigned long ifindex; + bool match = false; + + /* A shared block has no qdisc of its own, but it may bind + * several netdevs and the program is offloaded to all of them; + * require a match with every member. + */ + if (!tcf_block_shared(block)) { + dev = qdisc_dev(tcf_block_q(block)); + match = bpf_offload_dev_match(fp, dev); + if (!match) + NL_SET_ERR_MSG(extack, + "Program is bound to a different device"); + } else { + xa_for_each(&block->ports, ifindex, dev) { + match = bpf_offload_dev_match(fp, dev); + if (!match) + break; + } + if (!match) + NL_SET_ERR_MSG(extack, + "Program is not bound to every device in the block"); + } - dev = block->q ? qdisc_dev(block->q) : NULL; - if (!dev || !bpf_offload_dev_match(fp, dev)) { - NL_SET_ERR_MSG(extack, - "Program is bound to a different device"); + if (!match) { bpf_prog_put(fp); return -EINVAL; } diff --git a/net/sched/cls_flower.c b/net/sched/cls_flower.c index 1cefea571efd..92a7f2d6efc0 100644 --- a/net/sched/cls_flower.c +++ b/net/sched/cls_flower.c @@ -518,15 +518,22 @@ static void fl_hw_update_stats(struct tcf_proto *tp, struct cls_fl_filter *f, tcf_exts_hw_stats_update(&f->exts, &cls_flower.stats, cls_flower.use_act_stats); } -static void __fl_put(struct cls_fl_filter *f) +static bool __fl_put(struct tcf_proto *tp, struct cls_fl_filter *f) { + bool last = false; + if (!refcount_dec_and_test(&f->refcnt)) - return; + return false; + + if (f->mask) + last = fl_mask_put(fl_head_dereference(tp), f->mask); if (tcf_exts_get_net(&f->exts)) tcf_queue_work(&f->rwork, fl_destroy_filter_work); else __fl_destroy_filter(f); + + return last; } static struct cls_fl_filter *__fl_get(struct cls_fl_head *head, u32 handle) @@ -572,11 +579,10 @@ static int __fl_delete(struct tcf_proto *tp, struct cls_fl_filter *f, list_del_rcu(&f->list); spin_unlock(&tp->lock); - *last = fl_mask_put(head, f->mask); if (!tc_skip_hw(f->flags)) fl_hw_destroy_filter(tp, f, rtnl_held, extack); tcf_unbind_filter(tp, &f->res); - __fl_put(f); + *last = __fl_put(tp, f); return 0; } @@ -617,7 +623,7 @@ static void fl_put(struct tcf_proto *tp, void *arg) { struct cls_fl_filter *f = arg; - __fl_put(f); + __fl_put(tp, f); } static void *fl_get(struct tcf_proto *tp, u32 handle) @@ -2488,12 +2494,14 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, } err = fl_check_assign_mask(head, fnew, fold, mask); - if (err) + if (err) { + fnew->mask = NULL; goto unbind_filter; + } err = fl_ht_insert_unique(fnew, fold, &in_ht); if (err) - goto errout_mask; + goto unbind_filter; if (!tc_skip_hw(fnew->flags)) { err = fl_hw_replace_filter(tp, fnew, rtnl_held, extack); @@ -2547,7 +2555,6 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, spin_unlock(&tp->lock); - fl_mask_put(head, fold->mask); if (!tc_skip_hw(fold->flags)) fl_hw_destroy_filter(tp, fold, rtnl_held, NULL); tcf_unbind_filter(tp, &fold->res); @@ -2555,7 +2562,7 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, * after this. */ refcount_dec(&fold->refcnt); - __fl_put(fold); + __fl_put(tp, fold); } else { idr_replace(&head->handle_idr, fnew, fnew->handle); @@ -2580,9 +2587,6 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, if (in_ht) rhashtable_remove_fast(&fnew->mask->ht, &fnew->ht_node, fnew->mask->filter_ht_params); -errout_mask: - fl_mask_put(head, fnew->mask); - unbind_filter: if (bound_to_filter) { if (flags & TCA_ACT_FLAGS_NO_RTNL) @@ -2598,14 +2602,14 @@ static int fl_change(struct net *net, struct sk_buff *in_skb, idr_remove(&head->handle_idr, fnew->handle); spin_unlock(&tp->lock); } - __fl_put(fnew); + __fl_put(tp, fnew); errout_tb: kfree(tb); errout_mask_alloc: tcf_queue_work(&mask->rwork, fl_uninit_mask_free_work); errout_fold: if (fold) - __fl_put(fold); + __fl_put(tp, fold); return err; } @@ -2618,8 +2622,10 @@ static int fl_delete(struct tcf_proto *tp, void *arg, bool *last, int err = 0; err = __fl_delete(tp, f, &last_on_mask, rtnl_held, extack); - *last = list_empty(&head->masks); - __fl_put(f); + spin_lock(&tp->lock); + *last = idr_is_empty(&head->handle_idr); + spin_unlock(&tp->lock); + __fl_put(tp, f); return err; } @@ -2641,12 +2647,12 @@ static void fl_walk(struct tcf_proto *tp, struct tcf_walker *arg, rcu_read_unlock(); if (arg->fn(tp, f, arg) < 0) { - __fl_put(f); + __fl_put(tp, f); arg->stop = 1; rcu_read_lock(); break; } - __fl_put(f); + __fl_put(tp, f); arg->count++; rcu_read_lock(); } @@ -2697,7 +2703,7 @@ static int fl_reoffload(struct tcf_proto *tp, bool add, flow_setup_cb_t *cb, cls_flower.rule = flow_rule_alloc(tcf_exts_num_actions(&f->exts)); if (!cls_flower.rule) { - __fl_put(f); + __fl_put(tp, f); return -ENOMEM; } @@ -2715,7 +2721,7 @@ static int fl_reoffload(struct tcf_proto *tp, bool add, flow_setup_cb_t *cb, if (err) { kfree(cls_flower.rule); if (tc_skip_sw(f->flags)) { - __fl_put(f); + __fl_put(tp, f); return err; } goto next_flow; @@ -2731,11 +2737,11 @@ static int fl_reoffload(struct tcf_proto *tp, bool add, flow_setup_cb_t *cb, kfree(cls_flower.rule); if (err) { - __fl_put(f); + __fl_put(tp, f); return err; } next_flow: - __fl_put(f); + __fl_put(tp, f); } return 0; diff --git a/net/sched/sch_generic.c b/net/sched/sch_generic.c index 6f6a6f0d5eb0..3e8ee4a691c3 100644 --- a/net/sched/sch_generic.c +++ b/net/sched/sch_generic.c @@ -168,8 +168,11 @@ static inline void qdisc_enqueue_skb_bad_txq(struct Qdisc *q, static inline void dev_requeue_skb(struct sk_buff *skb, struct Qdisc *q) { + struct sk_buff_head requeue; spinlock_t *lock = NULL; + __skb_queue_head_init(&requeue); + if (q->flags & TCQ_F_NOLOCK) { lock = qdisc_lock(q); spin_lock(lock); @@ -178,7 +181,7 @@ static inline void dev_requeue_skb(struct sk_buff *skb, struct Qdisc *q) while (skb) { struct sk_buff *next = skb->next; - __skb_queue_tail(&q->gso_skb, skb); + __skb_queue_tail(&requeue, skb); /* it's still part of the queue */ if (qdisc_is_percpu_stats(q)) { @@ -194,6 +197,11 @@ static inline void dev_requeue_skb(struct sk_buff *skb, struct Qdisc *q) skb = next; } + /* the skbs may come from the head of gso_skb, so put them back there: + * appending them would let the ones still parked overtake them + */ + skb_queue_splice(&requeue, &q->gso_skb); + if (lock) { spin_unlock(lock); set_bit(__QDISC_STATE_MISSED, &q->state); @@ -1280,6 +1288,10 @@ static void transition_one_qdisc(struct net_device *dev, if (need_watchdog_p) { WRITE_ONCE(dev_queue->trans_start, jiffies); *need_watchdog_p = 1; + + /* Resume backlog left by a suppressed schedule while deactivated. */ + if (!qdisc_is_empty(new_qdisc)) + netif_schedule_queue(dev_queue); } } diff --git a/net/sched/sch_teql.c b/net/sched/sch_teql.c index 409ce50cc0db..b60ab79a6572 100644 --- a/net/sched/sch_teql.c +++ b/net/sched/sch_teql.c @@ -53,7 +53,7 @@ struct teql_master { struct Qdisc_ops qops; struct net_device *dev; struct Qdisc __rcu *slaves; - spinlock_t slaves_lock; /* serializes writes to ->slaves */ + raw_spinlock_t slaves_lock; /* serializes writes to ->slaves */ struct list_head master_list; unsigned long tx_bytes; unsigned long tx_packets; @@ -92,6 +92,7 @@ teql_dequeue(struct Qdisc *sch) { struct teql_sched_data *dat = qdisc_priv(sch); struct netdev_queue *dat_queue; + unsigned long flags; struct sk_buff *skb; struct Qdisc *q; @@ -102,9 +103,9 @@ teql_dequeue(struct Qdisc *sch) if (skb == NULL) { struct net_device *m = qdisc_dev(q); if (m) { - spin_lock_bh(&dat->m->slaves_lock); + raw_spin_lock_irqsave(&dat->m->slaves_lock, flags); rcu_assign_pointer(dat->m->slaves, sch); - spin_unlock_bh(&dat->m->slaves_lock); + raw_spin_unlock_irqrestore(&dat->m->slaves_lock, flags); netif_wake_queue(m); } } else { @@ -137,11 +138,13 @@ teql_destroy(struct Qdisc *sch) struct teql_master *master = dat->m; struct netdev_queue *txq = NULL; bool reset_master_queue = false; + unsigned long flags; + bool purge = false; if (!master) return; - spin_lock_bh(&master->slaves_lock); + raw_spin_lock_irqsave(&master->slaves_lock, flags); prev = rcu_dereference_protected(master->slaves, lockdep_is_held(&master->slaves_lock)); if (prev) { @@ -169,12 +172,18 @@ teql_destroy(struct Qdisc *sch) reset_master_queue = true; } } - skb_queue_purge(&dat->q); + purge = true; break; } while (prev != rcu_dereference_protected(master->slaves, lockdep_is_held(&master->slaves_lock))); } - spin_unlock_bh(&master->slaves_lock); + raw_spin_unlock_irqrestore(&master->slaves_lock, flags); + + /* skb_queue_purge() takes the queue's spinlock_t, which cannot nest + * inside a raw lock on PREEMPT_RT; purge after the raw section. + */ + if (purge) + skb_queue_purge(&dat->q); if (reset_master_queue) dev_reset_queue(master->dev, txq, NULL); @@ -187,6 +196,7 @@ static int teql_qdisc_init(struct Qdisc *sch, struct nlattr *opt, struct teql_master *m = (struct teql_master *)sch->ops; struct teql_sched_data *q = qdisc_priv(sch); struct Qdisc *first; + unsigned long flags; if (dev->hard_header_len > m->dev->hard_header_len) return -EINVAL; @@ -203,7 +213,7 @@ static int teql_qdisc_init(struct Qdisc *sch, struct nlattr *opt, skb_queue_head_init(&q->q); - spin_lock_bh(&m->slaves_lock); + raw_spin_lock_irqsave(&m->slaves_lock, flags); first = rcu_dereference_protected(m->slaves, lockdep_is_held(&m->slaves_lock)); if (first) { if (m->dev->flags & IFF_UP) { @@ -214,7 +224,7 @@ static int teql_qdisc_init(struct Qdisc *sch, struct nlattr *opt, (m->dev->flags & IFF_MULTICAST && !(dev->flags & IFF_MULTICAST)) || dev->mtu < m->dev->mtu) { - spin_unlock_bh(&m->slaves_lock); + raw_spin_unlock_irqrestore(&m->slaves_lock, flags); return -EINVAL; } } else { @@ -237,7 +247,7 @@ static int teql_qdisc_init(struct Qdisc *sch, struct nlattr *opt, m->dev->mtu = dev->mtu; m->dev->flags = (m->dev->flags&~FMASK)|(dev->flags&FMASK); } - spin_unlock_bh(&m->slaves_lock); + raw_spin_unlock_irqrestore(&m->slaves_lock, flags); return 0; } @@ -302,11 +312,12 @@ static inline int teql_resolve(struct sk_buff *skb, static netdev_tx_t teql_master_xmit(struct sk_buff *skb, struct net_device *dev) { struct teql_master *master = netdev_priv(dev); - struct Qdisc *start, *q; - int busy; - int nores; int subq = skb_get_queue_mapping(skb); struct sk_buff *skb_res = NULL; + struct Qdisc *start, *q; + unsigned long flags; + int nores; + int busy; restart: nores = 0; @@ -336,38 +347,43 @@ static netdev_tx_t teql_master_xmit(struct sk_buff *skb, struct net_device *dev) case 0: if (__netif_tx_trylock(slave_txq)) { unsigned int length = qdisc_pkt_len(skb); + int rc = NETDEV_TX_BUSY; skb->dev = slave; - if (!netif_xmit_frozen_or_stopped(slave_txq) && - netdev_start_xmit(skb, slave, slave_txq, false) == - NETDEV_TX_OK) { - __netif_tx_unlock(slave_txq); - spin_lock(&master->slaves_lock); + if (!netif_xmit_frozen_or_stopped(slave_txq)) + rc = netdev_start_xmit(skb, slave, slave_txq, false); + __netif_tx_unlock(slave_txq); + if (dev_xmit_complete(rc)) { + if (rc != NETDEV_TX_OK && rc != NET_XMIT_CN) { + master->tx_dropped++; + rcu_read_unlock(); + return NETDEV_TX_OK; + } + raw_spin_lock_irqsave(&master->slaves_lock, flags); if (rcu_dereference_protected(master->slaves, lockdep_is_held(&master->slaves_lock)) == q) rcu_assign_pointer(master->slaves, rcu_dereference_protected(NEXT_SLAVE(q), lockdep_is_held(&master->slaves_lock))); - spin_unlock(&master->slaves_lock); + raw_spin_unlock_irqrestore(&master->slaves_lock, flags); netif_wake_queue(dev); master->tx_packets++; master->tx_bytes += length; rcu_read_unlock(); return NETDEV_TX_OK; } - __netif_tx_unlock(slave_txq); } if (netif_xmit_stopped(netdev_get_tx_queue(dev, 0))) busy = 1; break; case 1: - spin_lock(&master->slaves_lock); + raw_spin_lock_irqsave(&master->slaves_lock, flags); if (rcu_dereference_protected(master->slaves, lockdep_is_held(&master->slaves_lock)) == q) rcu_assign_pointer(master->slaves, rcu_dereference_protected(NEXT_SLAVE(q), lockdep_is_held(&master->slaves_lock))); - spin_unlock(&master->slaves_lock); + raw_spin_unlock_irqrestore(&master->slaves_lock, flags); rcu_read_unlock(); return NETDEV_TX_OK; default: @@ -489,7 +505,7 @@ static __init void teql_master_setup(struct net_device *dev) struct teql_master *master = netdev_priv(dev); struct Qdisc_ops *ops = &master->qops; - spin_lock_init(&master->slaves_lock); + raw_spin_lock_init(&master->slaves_lock); master->dev = dev; ops->priv_size = sizeof(struct teql_sched_data); diff --git a/net/sunrpc/auth_gss/svcauth_gss.c b/net/sunrpc/auth_gss/svcauth_gss.c index 967e9d53080d..43839e805531 100644 --- a/net/sunrpc/auth_gss/svcauth_gss.c +++ b/net/sunrpc/auth_gss/svcauth_gss.c @@ -824,12 +824,18 @@ svcauth_gss_register_pseudoflavor(u32 pseudoflavor, char * name) new->h.flavour = &svcauthops_gss; new->pseudoflavor = pseudoflavor; + if (!try_module_get(svcauthops_gss.owner)) { + stat = -ENODEV; + goto out_free_name; + } + test = auth_domain_lookup(name, &new->h); if (test != &new->h) { pr_warn("svc: duplicate registration of gss pseudo flavour %s.\n", name); stat = -EADDRINUSE; auth_domain_put(test); + module_put(svcauthops_gss.owner); goto out_free_name; } return test; @@ -1085,10 +1091,11 @@ static int gss_read_proxy_verf(struct svc_rqst *rqstp, } length = min_t(unsigned int, inlen, (char *)xdr->end - (char *)xdr->p); - if (length) - memcpy(page_address(in_token->pages[0]), xdr->p, length); + for (to_offs = 0; to_offs < length; to_offs += PAGE_SIZE) + memcpy(page_address(in_token->pages[to_offs >> PAGE_SHIFT]), + (char *)xdr->p + to_offs, + min_t(unsigned int, length - to_offs, PAGE_SIZE)); inlen -= length; - to_offs = length; from_offs = rqstp->rq_arg.page_base; while (inlen) { @@ -1420,10 +1427,18 @@ static ssize_t write_gssp(struct file *file, const char __user *buf, return res; if (i != 1) return -EINVAL; + + /* + * Neither the proc entry nor an open descriptor pins @net, so it + * might already be dying when rpc_create() takes its reference. + */ + if (!maybe_get_net(net)) + return -ENXIO; + res = set_gssp_clnt(net); - if (res) - return res; - res = set_gss_proxy(net, 1); + if (!res) + res = set_gss_proxy(net, 1); + put_net(net); if (res) return res; return count; @@ -1995,9 +2010,11 @@ svcauth_gss_domain_release_rcu(struct rcu_head *head) { struct auth_domain *dom = container_of(head, struct auth_domain, rcu_head); struct gss_domain *gd = container_of(dom, struct gss_domain, h); + struct module *owner = dom->flavour->owner; kfree(dom->name); kfree(gd); + module_put(owner); } static void diff --git a/net/sunrpc/svc.c b/net/sunrpc/svc.c index 8297bad2b177..4f402bbf97ba 100644 --- a/net/sunrpc/svc.c +++ b/net/sunrpc/svc.c @@ -1208,13 +1208,16 @@ int svc_register(const struct svc_serv *serv, struct net *net, struct svc_program *progp = &serv->sv_programs[p]; for (i = 0; i < progp->pg_nvers; i++) { + int ret; - error = progp->pg_rpcbind_set(net, progp, i, + ret = progp->pg_rpcbind_set(net, progp, i, family, proto, port); - if (error < 0) { + if (ret < 0) { printk(KERN_WARNING "svc: failed to register " "%sv%u RPC service (errno %d).\n", - progp->pg_name, i, -error); + progp->pg_name, i, -ret); + if (!error) + error = ret; break; } } diff --git a/net/sunrpc/svc_xprt.c b/net/sunrpc/svc_xprt.c index 40040af588fb..730450034dff 100644 --- a/net/sunrpc/svc_xprt.c +++ b/net/sunrpc/svc_xprt.c @@ -852,8 +852,8 @@ static void svc_handle_xprt(struct svc_rqst *rqstp, struct svc_xprt *xprt) newxpt = xprt->xpt_ops->xpo_accept(xprt); if (newxpt) { newxpt->xpt_cred = get_cred(xprt->xpt_cred); - svc_add_new_temp_xprt(serv, newxpt); trace_svc_xprt_accept(newxpt, serv->sv_name); + svc_add_new_temp_xprt(serv, newxpt); } else { module_put(xprt->xpt_class->xcl_owner); } @@ -1268,13 +1268,8 @@ static void svc_revisit(struct cache_deferred_req *dreq, int too_many) } /* - * Save the request off for later processing. The request buffer looks - * like this: - * - * - * - * This code can only handle requests that consist of an xprt-header - * and rpc-header. + * Save the request off for later processing. Only a Call that fits + * entirely in rq_arg.head[0] can be deferred. */ static struct cache_deferred_req *svc_defer(struct cache_req *req) { @@ -1287,8 +1282,11 @@ static struct cache_deferred_req *svc_defer(struct cache_req *req) dr = rqstp->rq_deferred; rqstp->rq_deferred = NULL; } else { - size_t skip; size_t size; + + if (rqstp->rq_arg.len > rqstp->rq_arg.head[0].iov_len) + return NULL; + /* FIXME maybe discard if size too large */ size = sizeof(struct svc_deferred_req) + rqstp->rq_arg.len; dr = kmalloc(size, GFP_KERNEL); @@ -1302,9 +1300,7 @@ static struct cache_deferred_req *svc_defer(struct cache_req *req) dr->daddr = rqstp->rq_daddr; dr->argslen = rqstp->rq_arg.len >> 2; - /* back up head to the start of the buffer and copy */ - skip = rqstp->rq_arg.len - rqstp->rq_arg.head[0].iov_len; - memcpy(dr->args, rqstp->rq_arg.head[0].iov_base - skip, + memcpy(dr->args, rqstp->rq_arg.head[0].iov_base, dr->argslen << 2); } dr->xprt_ctxt = rqstp->rq_xprt_ctxt; @@ -1327,17 +1323,13 @@ static noinline int svc_deferred_recv(struct svc_rqst *rqstp) trace_svc_defer_recv(dr); - /* setup iov_base past transport header */ rqstp->rq_arg.head[0].iov_base = dr->args; - /* The iov_len does not include the transport header bytes */ rqstp->rq_arg.head[0].iov_len = dr->argslen << 2; rqstp->rq_arg.page_len = 0; - /* The rq_arg.len includes the transport header bytes */ rqstp->rq_arg.len = dr->argslen << 2; rqstp->rq_prot = dr->prot; memcpy(&rqstp->rq_addr, &dr->addr, dr->addrlen); rqstp->rq_addrlen = dr->addrlen; - /* Save off transport header len in case we get deferred again */ rqstp->rq_daddr = dr->daddr; rqstp->rq_xprt_ctxt = dr->xprt_ctxt; diff --git a/net/sunrpc/svcauth_unix.c b/net/sunrpc/svcauth_unix.c index 31a1bc60a5f6..a68fe44d1f62 100644 --- a/net/sunrpc/svcauth_unix.c +++ b/net/sunrpc/svcauth_unix.c @@ -540,6 +540,13 @@ static int unix_gid_parse(struct cache_detail *cd, if (ugp) { struct cache_head *ch; ug.h.flags = 0; + /* + * mountd sends at least the user's primary group on + * success, so an empty list can only mean the lookup + * failed. Keep the credential's own groups instead. + */ + if (gids == 0) + set_bit(CACHE_NEGATIVE, &ug.h.flags); ug.h.expiry_time = expiry; ch = sunrpc_cache_update(cd, &ug.h, &ugp->h, @@ -730,6 +737,8 @@ static int sunrpc_nl_parse_one_unix_gid(struct cache_detail *cd, boot.tv_sec; if (tb[SUNRPC_A_UNIX_GID_NEGATIVE]) { + /* failed lookup: keep the credential's own groups */ + set_bit(CACHE_NEGATIVE, &ug.h.flags); ug.gi = groups_alloc(0); if (!ug.gi) return -ENOMEM; diff --git a/net/sunrpc/svcsock.c b/net/sunrpc/svcsock.c index 50e5e7f5b762..5a2d52284d75 100644 --- a/net/sunrpc/svcsock.c +++ b/net/sunrpc/svcsock.c @@ -238,6 +238,39 @@ static int svc_one_sock_name(struct svc_sock *svsk, char *buf, int remaining) return len; } +/* + * kTLS delivers a record only up to the caller's buffer and keeps + * the remainder on its receive list, where no further data_ready + * announces it. Consume the whole record. + */ +static void +svc_tcp_sock_drain_record(struct socket *sock) +{ + union { + struct cmsghdr cmsg; + u8 buf[CMSG_SPACE(sizeof(u8))]; + } u; + u8 discard[64]; + struct kvec discard_kvec = { + .iov_base = discard, + .iov_len = sizeof(discard), + }; + + for (;;) { + struct msghdr msg = { + .msg_control = &u, + .msg_controllen = sizeof(u), + }; + + iov_iter_kvec(&msg.msg_iter, ITER_DEST, &discard_kvec, 1, + discard_kvec.iov_len); + if (sock_recvmsg(sock, &msg, MSG_DONTWAIT) <= 0) + break; + if (msg.msg_flags & MSG_EOR) + break; + } +} + static int svc_tcp_sock_process_cmsg(struct socket *sock, struct msghdr *msg, struct cmsghdr *cmsg, int ret) @@ -257,8 +290,17 @@ svc_tcp_sock_process_cmsg(struct socket *sock, struct msghdr *msg, break; case TLS_RECORD_TYPE_ALERT: tls_alert_recv(sock->sk, msg, &level, &description); - ret = (level == TLS_ALERT_LEVEL_FATAL) ? - -ENOTCONN : -EAGAIN; + /* RFC 8446 Section 6: every alert but a closure alert is + * an error alert. + */ + switch (description) { + case TLS_ALERT_DESC_CLOSE_NOTIFY: + case TLS_ALERT_DESC_USER_CANCELED: + ret = -EAGAIN; + break; + default: + ret = -ENOTCONN; + } break; default: /* discard this record type */ @@ -289,8 +331,31 @@ svc_tcp_sock_recv_cmsg(struct socket *sock, unsigned int *msg_flags) iov_iter_kvec(&msg.msg_iter, ITER_DEST, &alert_kvec, 1, alert_kvec.iov_len); ret = sock_recvmsg(sock, &msg, MSG_DONTWAIT); - if (ret > 0 && - tls_get_record_type(sock->sk, &u.cmsg) == TLS_RECORD_TYPE_ALERT) { + /* put_cmsg() shrinks msg_controllen, so a short one means + * kTLS filled in u.cmsg. + */ + if (ret >= 0 && msg.msg_controllen < sizeof(u)) { + u8 content_type = tls_get_record_type(sock->sk, &u.cmsg); + + /* Returning the count would credit the RPC stream with + * octets that never reached the caller's buffer. + */ + if (content_type != TLS_RECORD_TYPE_ALERT) { + /* An application data record carries RPC payload. + * Draining one breaks RPC fragment framing. + */ + if (content_type != TLS_RECORD_TYPE_DATA && + !(msg.msg_flags & MSG_EOR)) + svc_tcp_sock_drain_record(sock); + return -EAGAIN; + } + /* An Alert record carries exactly one two-octet message + * (RFC 8446 Section 5.1). alert_kvec caps the receive at two, + * so a longer record produces the same count. MSG_EOR appears + * only once kTLS has drained the whole record. + */ + if (ret != sizeof(alert) || !(msg.msg_flags & MSG_EOR)) + return -EBADMSG; iov_iter_revert(&msg.msg_iter, ret); ret = svc_tcp_sock_process_cmsg(sock, &msg, &u.cmsg, -EAGAIN); } @@ -306,8 +371,16 @@ svc_tcp_sock_recvmsg(struct svc_sock *svsk, struct msghdr *msg) ret = sock_recvmsg(sock, msg, MSG_DONTWAIT); if (msg->msg_flags & MSG_CTRUNC) { msg->msg_flags &= ~(MSG_CTRUNC | MSG_EOR); - if (ret == 0 || ret == -EIO) + if (ret == 0 || ret == -EIO) { ret = svc_tcp_sock_recv_cmsg(sock, &msg->msg_flags); + /* A control record delivers nothing to the caller, + * and kTLS announces no data_ready for records it + * already holds. Mark the transport ready so that + * the records behind this one can be received. + */ + if (ret == -EAGAIN) + set_bit(XPT_DATA, &svsk->sk_xprt.xpt_flags); + } } return ret; } @@ -1541,6 +1614,9 @@ int svc_addsock(struct svc_serv *serv, struct net *net, const int fd, err = -EISCONN; if (so->state > SS_UNCONNECTED) goto out; + err = -EBUSY; + if (so->sk->sk_user_data) + goto out; err = -ENOENT; if (!try_module_get(THIS_MODULE)) goto out; diff --git a/rust/bindgen_parameters b/rust/bindgen_parameters index 8402b0c93545..39aae62dca66 100644 --- a/rust/bindgen_parameters +++ b/rust/bindgen_parameters @@ -27,6 +27,8 @@ --opaque-type spinlock # enums that appear in indirect function calls should specify a cfi type +--newtype-enum fwctl_rpc_scope +--with-attribute-custom-enum=fwctl_rpc_scope='#[cfi_encoding="15fwctl_rpc_scope"]' --newtype-enum lru_status --with-attribute-custom-enum=lru_status='#[cfi_encoding="10lru_status"]' @@ -50,7 +52,7 @@ --blocklist-item VM_MAYREAD --blocklist-item VM_MAYWRITE --blocklist-item VM_MAYEXEC ---blocklist-item VM_MAYEXEC +--blocklist-item VM_MAYSHARE --blocklist-item VM_PFNMAP --blocklist-item VM_IO --blocklist-item VM_DONTCOPY diff --git a/rust/bindings/bindings_helper.h b/rust/bindings/bindings_helper.h index 4b31aa7f432f..c706dda5b5ca 100644 --- a/rust/bindings/bindings_helper.h +++ b/rust/bindings/bindings_helper.h @@ -138,7 +138,7 @@ const vm_flags_t RUST_CONST_HELPER_VM_SHARED = VM_SHARED; const vm_flags_t RUST_CONST_HELPER_VM_MAYREAD = VM_MAYREAD; const vm_flags_t RUST_CONST_HELPER_VM_MAYWRITE = VM_MAYWRITE; const vm_flags_t RUST_CONST_HELPER_VM_MAYEXEC = VM_MAYEXEC; -const vm_flags_t RUST_CONST_HELPER_VM_MAYSHARE = VM_MAYEXEC; +const vm_flags_t RUST_CONST_HELPER_VM_MAYSHARE = VM_MAYSHARE; const vm_flags_t RUST_CONST_HELPER_VM_PFNMAP = VM_PFNMAP; const vm_flags_t RUST_CONST_HELPER_VM_IO = VM_IO; const vm_flags_t RUST_CONST_HELPER_VM_DONTCOPY = VM_DONTCOPY; diff --git a/rust/kernel/bitmap.rs b/rust/kernel/bitmap.rs index b27e0ec80d64..053821fff74b 100644 --- a/rust/kernel/bitmap.rs +++ b/rust/kernel/bitmap.rs @@ -17,24 +17,59 @@ /// # Invariants /// /// Must reference a `[c_ulong]` long enough to fit `data.len()` bits. +/// Must not be longer than [`i32::MAX`] bits, so offsets and lengths used with +/// [`Bitmap`] functions fit in the `int` and `unsigned int` arguments of the C bitmap API. +/// This also matches [`BitmapVec::MAX_LEN`]. #[cfg_attr(CONFIG_64BIT, repr(align(8)))] #[cfg_attr(not(CONFIG_64BIT), repr(align(4)))] pub struct Bitmap { data: [()], } +macro_rules! bitmap_assert { + ($cond:expr, $($arg:tt)+) => { + #[cfg(CONFIG_RUST_BITMAP_HARDENED)] + assert!($cond, $($arg)*); + } +} + +macro_rules! bitmap_assert_return { + ($cond:expr, $($arg:tt)+) => { + #[cfg(CONFIG_RUST_BITMAP_HARDENED)] + assert!($cond, $($arg)*); + + #[cfg(not(CONFIG_RUST_BITMAP_HARDENED))] + if !($cond) { + pr_err!($($arg)*); + return + } + } +} + impl Bitmap { /// Borrows a C bitmap. /// + /// # Panics + /// + /// Panics if CONFIG_RUST_BITMAP_HARDENED is enabled and `nbits` exceeds [`i32::MAX`]. + /// /// # Safety /// /// * `ptr` holds a non-null address of an initialized array of `unsigned long` /// that is large enough to hold `nbits` bits. + /// * `nbits` must not exceed [`i32::MAX`]. /// * the array must not be freed for the lifetime of this [`Bitmap`] /// * concurrent access only happens through atomic operations pub unsafe fn from_raw<'a>(ptr: *const usize, nbits: usize) -> &'a Bitmap { + bitmap_assert!( + nbits <= i32::MAX as usize, + "`nbits` must be <= {}, was {}", + i32::MAX, + nbits + ); let data: *const [()] = core::ptr::slice_from_raw_parts(ptr.cast(), nbits); // INVARIANT: `data` references an initialized array that can hold `nbits` bits. + // INVARIANT: The caller guarantees that `nbits` does not exceed `i32::MAX`. // SAFETY: // The caller guarantees that `data` (derived from `ptr` and `nbits`) // points to a valid, initialized, and appropriately sized memory region @@ -51,15 +86,27 @@ pub unsafe fn from_raw<'a>(ptr: *const usize, nbits: usize) -> &'a Bitmap { /// Borrows a C bitmap exclusively. /// + /// # Panics + /// + /// Panics if CONFIG_RUST_BITMAP_HARDENED is enabled and `nbits` exceeds [`i32::MAX`]. + /// /// # Safety /// /// * `ptr` holds a non-null address of an initialized array of `unsigned long` /// that is large enough to hold `nbits` bits. + /// * `nbits` must not exceed [`i32::MAX`]. /// * the array must not be freed for the lifetime of this [`Bitmap`] /// * no concurrent access may happen. pub unsafe fn from_raw_mut<'a>(ptr: *mut usize, nbits: usize) -> &'a mut Bitmap { + bitmap_assert!( + nbits <= i32::MAX as usize, + "`nbits` must be <= {}, was {}", + i32::MAX, + nbits + ); let data: *mut [()] = core::ptr::slice_from_raw_parts_mut(ptr.cast(), nbits); // INVARIANT: `data` references an initialized array that can hold `nbits` bits. + // INVARIANT: The caller guarantees that `nbits` does not exceed `i32::MAX`. // SAFETY: // The caller guarantees that `data` (derived from `ptr` and `nbits`) // points to a valid, initialized, and appropriately sized memory region @@ -96,26 +143,6 @@ union BitmapRepr { ptr: NonNull, } -macro_rules! bitmap_assert { - ($cond:expr, $($arg:tt)+) => { - #[cfg(CONFIG_RUST_BITMAP_HARDENED)] - assert!($cond, $($arg)*); - } -} - -macro_rules! bitmap_assert_return { - ($cond:expr, $($arg:tt)+) => { - #[cfg(CONFIG_RUST_BITMAP_HARDENED)] - assert!($cond, $($arg)*); - - #[cfg(not(CONFIG_RUST_BITMAP_HARDENED))] - if !($cond) { - pr_err!($($arg)*); - return - } - } -} - /// Represents an owned bitmap. /// /// Wraps underlying C bitmap API. See [`Bitmap`] for available @@ -415,7 +442,8 @@ pub fn clear_bit_atomic(&self, index: usize) { #[inline] pub fn copy_and_extend(&mut self, src: &Bitmap) { let len = core::cmp::min(src.len(), self.len()); - // SAFETY: access to `self` and `src` is within bounds. + // SAFETY: Access to `self` and `src` is within bounds. Both lengths fit in `u32` + // because a `Bitmap` is at most `i32::MAX` bits, so the casts are lossless. unsafe { bindings::bitmap_copy_and_extend( self.as_mut_ptr(), diff --git a/rust/kernel/block/mq/gen_disk.rs b/rust/kernel/block/mq/gen_disk.rs index fc97dd873974..4d49f33c6fab 100644 --- a/rust/kernel/block/mq/gen_disk.rs +++ b/rust/kernel/block/mq/gen_disk.rs @@ -212,8 +212,14 @@ pub struct GenDisk { } // SAFETY: `GenDisk` is an owned pointer to a `struct gendisk` and an `Arc` to a -// `TagSet` It is safe to send this to other threads as long as T is Send. -unsafe impl Send for GenDisk {} +// `TagSet`. It is safe to send this to other threads as long as these two are `Send`. +unsafe impl Send for GenDisk +where + T: Operations, + T::QueueData: Send, + Arc>: Send, +{ +} impl Drop for GenDisk { fn drop(&mut self) { diff --git a/rust/kernel/block/mq/operations.rs b/rust/kernel/block/mq/operations.rs index 861903e18fbf..cbd1815ef99e 100644 --- a/rust/kernel/block/mq/operations.rs +++ b/rust/kernel/block/mq/operations.rs @@ -30,7 +30,7 @@ pub trait Operations: Sized { /// Data associated with the `struct request_queue` that is allocated for /// the `GenDisk` associated with this `Operations` implementation. - type QueueData: ForeignOwnable; + type QueueData: ForeignOwnable + Sync; /// Called by the kernel to queue a request with the driver. If `is_last` is /// `false`, the driver is allowed to defer committing the request. diff --git a/rust/kernel/bug.rs b/rust/kernel/bug.rs index 3566f0234ca4..6e048908da8a 100644 --- a/rust/kernel/bug.rs +++ b/rust/kernel/bug.rs @@ -85,15 +85,18 @@ macro_rules! warn_flags { ($file:expr, $flags:expr) => { if false { _ = $file; + _ = $flags; } - // SAFETY: It is always safe to call `warn_slowpath_fmt()` - // with a valid null-terminated string. + // SAFETY: + // - The file name is a null-terminated string with `'static` lifetime. + // - `TAINT_WARN` is a valid taint number. + // - `fmt` can be NULL, and NULL is passed. unsafe { $crate::bindings::warn_slowpath_fmt( $crate::str::CStrExt::as_char_ptr($crate::c_str!(::core::file!())), line!() as $crate::ffi::c_int, - $flags as $crate::ffi::c_uint, + $crate::bindings::TAINT_WARN, ::core::ptr::null(), ); } diff --git a/rust/kernel/configfs.rs b/rust/kernel/configfs.rs index cd082b83e9e7..2b3325cd9e1f 100644 --- a/rust/kernel/configfs.rs +++ b/rust/kernel/configfs.rs @@ -297,26 +297,39 @@ unsafe fn container_of(group: *const bindings::config_group) -> *const Self { /// /// `this` must be a valid pointer. /// -/// If `this` does not represent the root group of a configfs subsystem, -/// `this` must be a pointer to a `bindings::config_group` embedded in a -/// `Group`. +/// If `this` is the `su_group` field of a `bindings::configfs_subsystem`, that +/// `configfs_subsystem` must be embedded in a `Subsystem`. /// -/// Otherwise, `this` must be a pointer to a `bindings::config_group` that -/// is embedded in a `bindings::configfs_subsystem` that is embedded in a -/// `Subsystem`. +/// Otherwise, `this` must be a pointer to a `bindings::config_group` embedded +/// in a `Group`. unsafe fn get_group_data<'a, Parent>(this: *mut bindings::config_group) -> &'a Parent { // SAFETY: `this` is a valid pointer. - let is_root = unsafe { (*this).cg_subsys.is_null() }; + let subsys = unsafe { (*this).cg_subsys }; + + // `link_group()` in `fs/configfs/dir.c` assigns `cg_subsys` for every + // `config_group` attached anywhere in a registered subsystem, including + // the subsystem's own `su_group` (which gets a pointer to itself). The + // only `config_group` with a NULL `cg_subsys` is the configfs root, and + // userspace cannot trigger callbacks on it. The group is therefore the + // subsystem's `su_group` iff it equals `&cg_subsys->su_group`. + let is_root = !subsys.is_null() + && core::ptr::eq( + this.cast_const(), + // SAFETY: For every `config_group` the configfs core dispatches a + // callback on, `cg_subsys` was set by `link_group()` at registration time + // and points to a valid `configfs_subsystem` that outlives the callback. + unsafe { &raw const (*subsys).su_group }, + ); - if !is_root { - // SAFETY: By C API contact,`this` was returned from a call to - // `make_group`. The pointer is known to be embedded within a - // `Group`. - unsafe { &(*Group::::container_of(this)).data } - } else { - // SAFETY: By C API contract, `this` is a pointer to the - // `bindings::config_group` field within a `Subsystem`. + if is_root { + // SAFETY: By the above, `this` is the `su_group` field of a + // `configfs_subsystem` that, by function safety requirements, is + // embedded in a `Subsystem`. unsafe { &(*Subsystem::container_of(this)).data } + } else { + // SAFETY: By function safety requirements, `this` is a + // `config_group` embedded in a `Group`. + unsafe { &(*Group::::container_of(this)).data } } } diff --git a/rust/kernel/cpufreq.rs b/rust/kernel/cpufreq.rs index affa2b9490ef..b75d692846a2 100644 --- a/rust/kernel/cpufreq.rs +++ b/rust/kernel/cpufreq.rs @@ -14,7 +14,13 @@ cpumask, device::{Bound, Device}, devres, - error::{code::*, from_err_ptr, from_result, to_result, Result, VTABLE_DEFAULT_ERROR}, + error::{ + code::*, + from_result, + to_result, + Result, + VTABLE_DEFAULT_ERROR, // + }, ffi::{c_char, c_ulong}, prelude::*, types::ForeignOwnable, @@ -29,7 +35,10 @@ marker::PhantomData, ops::{Deref, DerefMut}, pin::Pin, - ptr, + ptr::{ + self, + NonNull, // + }, }; use macros::vtable; @@ -687,12 +696,13 @@ fn clear_data(&mut self) -> Option { impl<'a> PolicyCpu<'a> { fn from_cpu(cpu: CpuId) -> Result { // SAFETY: It is safe to call `cpufreq_cpu_get` for any valid CPU. - let ptr = from_err_ptr(unsafe { bindings::cpufreq_cpu_get(u32::from(cpu)) })?; + let ptr = + NonNull::new(unsafe { bindings::cpufreq_cpu_get(u32::from(cpu)) }).ok_or(ENODEV)?; Ok(Self( - // SAFETY: The `ptr` is guaranteed to be valid and remains valid for the lifetime of - // the returned reference. - unsafe { Policy::from_raw_mut(ptr) }, + // SAFETY: `ptr` is non-NULL and `cpufreq_cpu_get()` took a reference on it, so it is + // valid for writing and remains valid for the lifetime of the returned reference. + unsafe { Policy::from_raw_mut(ptr.as_ptr()) }, )) } } diff --git a/rust/kernel/fwctl.rs b/rust/kernel/fwctl.rs index f29244fb0d1d..b7778e4d5ee0 100644 --- a/rust/kernel/fwctl.rs +++ b/rust/kernel/fwctl.rs @@ -53,13 +53,13 @@ pub enum DeviceType { #[derive(Copy, Clone, Debug, Eq, PartialEq)] pub enum RpcScope { /// Read/write access to device configuration. - Configuration = bindings::fwctl_rpc_scope_FWCTL_RPC_CONFIGURATION, + Configuration = bindings::fwctl_rpc_scope::FWCTL_RPC_CONFIGURATION.0, /// Read-only access to debug information. - DebugReadOnly = bindings::fwctl_rpc_scope_FWCTL_RPC_DEBUG_READ_ONLY, + DebugReadOnly = bindings::fwctl_rpc_scope::FWCTL_RPC_DEBUG_READ_ONLY.0, /// Write access to lockdown-compatible debug information. - DebugWrite = bindings::fwctl_rpc_scope_FWCTL_RPC_DEBUG_WRITE, + DebugWrite = bindings::fwctl_rpc_scope::FWCTL_RPC_DEBUG_WRITE.0, /// Full read/write access to all debug information (requires `CAP_SYS_RAWIO`). - DebugWriteFull = bindings::fwctl_rpc_scope_FWCTL_RPC_DEBUG_WRITE_FULL, + DebugWriteFull = bindings::fwctl_rpc_scope::FWCTL_RPC_DEBUG_WRITE_FULL.0, } impl TryFrom for RpcScope { @@ -539,12 +539,12 @@ impl VTable { /// - `out_len` must be valid for reading and writing an initialised `usize`. unsafe extern "C" fn fw_rpc_callback( uctx: *mut bindings::fwctl_uctx, - scope: u32, + scope: bindings::fwctl_rpc_scope, rpc_in: *mut ffi::c_void, in_len: usize, out_len: *mut usize, ) -> *mut ffi::c_void { - let scope = match RpcScope::try_from(scope) { + let scope = match RpcScope::try_from(scope.0) { Ok(s) => s, Err(e) => return Error::to_ptr(e), }; diff --git a/rust/kernel/module_param.rs b/rust/kernel/module_param.rs index f9a14765a926..1a2fcbf4a5b3 100644 --- a/rust/kernel/module_param.rs +++ b/rust/kernel/module_param.rs @@ -74,8 +74,13 @@ pub trait ModuleParam: Sized + Copy { crate::error::from_result(|| { let new_value = T::try_from_param_arg(arg)?; - // SAFETY: By function safety requirements, this access is safe. - let container = unsafe { &*((*param).__bindgen_anon_1.arg.cast::>()) }; + // SAFETY: By function safety requirements, `param` is valid for read. + let arg = unsafe { (*param).__bindgen_anon_1.arg }; + let param_access_ptr = arg.cast::>(); + // SAFETY: The `arg` field is initialized with a pointer to a `static ModuleAccessParam` by + // the rust module macro. Thus, the pointer is valid for use as a reference. + let param_access = unsafe { &*(param_access_ptr) }; + let container = ¶m_access.value; container .populate(new_value) diff --git a/security/apparmor/include/policy_ns.h b/security/apparmor/include/policy_ns.h index cc6e84151812..4f8359875476 100644 --- a/security/apparmor/include/policy_ns.h +++ b/security/apparmor/include/policy_ns.h @@ -117,6 +117,22 @@ static inline struct aa_ns *aa_get_ns(struct aa_ns *ns) return ns; } +/** + * aa_get_ns_not0 - increment references count on @ns found via lookup + * @ns: namespace to increment reference count of (MAYBE NULL) + * + * Returns: pointer to @ns, if @ns is NULL returns NULL, + * NULL if @ns is being freed + * Requires: @ns must be held with valid refcount when called + */ +static inline struct aa_ns *aa_get_ns_not0(struct aa_ns *ns) +{ + if (ns && aa_get_profile_not0(ns->unconfined)) + return ns; + + return NULL; +} + /** * aa_put_ns - decrement refcount on @ns * @ns: namespace to put reference of diff --git a/security/apparmor/policy_ns.c b/security/apparmor/policy_ns.c index 5a907a875d8f..ae9168c1a85e 100644 --- a/security/apparmor/policy_ns.c +++ b/security/apparmor/policy_ns.c @@ -207,7 +207,7 @@ struct aa_ns *aa_lookupn_ns(struct aa_ns *view, const char *name, size_t n) struct aa_ns *ns = NULL; rcu_read_lock(); - ns = aa_get_ns(__aa_lookupn_ns(view, name, n)); + ns = aa_get_ns_not0(__aa_lookupn_ns(view, name, n)); rcu_read_unlock(); return ns; diff --git a/security/landlock/fs.c b/security/landlock/fs.c index cab43892ec2f..0959e9f487e6 100644 --- a/security/landlock/fs.c +++ b/security/landlock/fs.c @@ -61,7 +61,7 @@ static void release_inode(struct landlock_object *const object) __releases(object->lock) { struct inode *const inode = object->underobj; - struct super_block *sb; + atomic_long_t *inode_refs; if (!inode) { spin_unlock(&object->lock); @@ -77,8 +77,8 @@ static void release_inode(struct landlock_object *const object) * Makes sure that if the filesystem is concurrently unmounted, * hook_sb_delete() will wait for us to finish iput(). */ - sb = inode->i_sb; - atomic_long_inc(&landlock_superblock(sb)->inode_refs); + inode_refs = &landlock_superblock(inode->i_sb)->inode_refs; + atomic_long_inc(inode_refs); spin_unlock(&object->lock); /* * Because object->underobj was not NULL, hook_sb_delete() and @@ -92,8 +92,8 @@ static void release_inode(struct landlock_object *const object) */ iput(inode); - if (atomic_long_dec_and_test(&landlock_superblock(sb)->inode_refs)) - wake_up_var(&landlock_superblock(sb)->inode_refs); + if (atomic_long_dec_and_test(inode_refs)) + wake_up_var(inode_refs); } static const struct landlock_object_underops landlock_fs_underops = { diff --git a/sound/core/control.c b/sound/core/control.c index 4199342d4ffe..535ceba294ac 100644 --- a/sound/core/control.c +++ b/sound/core/control.c @@ -1802,6 +1802,8 @@ static int snd_ctl_elem_add(struct snd_ctl_file *file, alloc_size = compute_user_elem_size(private_size, count); guard(rwsem_write)(&card->controls_rwsem); + if (card->shutdown) + return -ENODEV; if (check_user_elem_overflow(card, alloc_size)) return -ENOMEM; diff --git a/sound/drivers/pcmtest.c b/sound/drivers/pcmtest.c index 5d5281e4deb7..186e982d42e1 100644 --- a/sound/drivers/pcmtest.c +++ b/sound/drivers/pcmtest.c @@ -280,39 +280,35 @@ static void fill_block_pattern(struct pcmtst_buf_iter *v_iter, struct snd_pcm_ru fill_block_pattern_n(v_iter, runtime); } +static void fill_random_block(char *buf, size_t buf_size, size_t pos, + size_t count) +{ + while (count) { + size_t chunk = min(count, buf_size - pos); + + get_random_bytes(buf + pos, chunk); + count -= chunk; + pos = 0; + } +} + static void fill_block_rand_n(struct pcmtst_buf_iter *v_iter, struct snd_pcm_runtime *runtime) { unsigned int channels = runtime->channels; - // Remaining space in all channel buffers - size_t bytes_remain = runtime->dma_bytes - v_iter->buf_pos; + size_t pos = v_iter->buf_pos / channels; unsigned int i; - for (i = 0; i < channels; i++) { - if (v_iter->b_rw <= bytes_remain) { - //b_rw - count of bytes must be written for all channels at each timer tick - get_random_bytes(runtime->dma_area + buf_pos_n(v_iter, channels, i), - v_iter->b_rw / channels); - } else { - // Write to the end of buffer and start from the beginning of it - get_random_bytes(runtime->dma_area + buf_pos_n(v_iter, channels, i), - bytes_remain / channels); - get_random_bytes(runtime->dma_area + v_iter->chan_block * i, - (v_iter->b_rw - bytes_remain) / channels); - } - } + for (i = 0; i < channels; i++) + fill_random_block(runtime->dma_area + v_iter->chan_block * i, + v_iter->chan_block, pos, + v_iter->b_rw / channels); inc_buf_pos(v_iter, v_iter->b_rw, runtime->dma_bytes); } static void fill_block_rand_i(struct pcmtst_buf_iter *v_iter, struct snd_pcm_runtime *runtime) { - size_t in_cur_block = runtime->dma_bytes - v_iter->buf_pos; - - if (v_iter->b_rw <= in_cur_block) { - get_random_bytes(&runtime->dma_area[v_iter->buf_pos], v_iter->b_rw); - } else { - get_random_bytes(&runtime->dma_area[v_iter->buf_pos], in_cur_block); - get_random_bytes(runtime->dma_area, v_iter->b_rw - in_cur_block); - } + fill_random_block(runtime->dma_area, runtime->dma_bytes, + v_iter->buf_pos, v_iter->b_rw); inc_buf_pos(v_iter, v_iter->b_rw, runtime->dma_bytes); } diff --git a/sound/soc/amd/ps/acp63.h b/sound/soc/amd/ps/acp63.h index 62cb6bef17ab..d0cfbd4dafc1 100644 --- a/sound/soc/amd/ps/acp63.h +++ b/sound/soc/amd/ps/acp63.h @@ -294,6 +294,7 @@ struct acp63_dev_data; * struct acp_hw_ops - ACP PCI driver platform specific ops * @acp_init: ACP initialization * @acp_deinit: ACP de-initialization + * @disable_interrupts: disable ACP interrupt sources * @acp_get_config: function to read the acp pin configuration * @acp_sdw_dma_irq_thread: ACP SoundWire DMA interrupt thread * acp_suspend: ACP system level suspend callback @@ -304,6 +305,7 @@ struct acp63_dev_data; struct acp_hw_ops { int (*acp_init)(void __iomem *acp_base, struct device *dev); int (*acp_deinit)(void __iomem *acp_base, struct device *dev); + void (*disable_interrupts)(void __iomem *acp_base); void (*acp_get_config)(struct pci_dev *pci, struct acp63_dev_data *acp_data); void (*acp_sdw_dma_irq_thread)(struct acp63_dev_data *acp_data); int (*acp_suspend)(struct device *dev); @@ -397,6 +399,12 @@ static inline int acp_hw_deinit(struct acp63_dev_data *adata, struct device *dev return -EOPNOTSUPP; } +static inline void acp_hw_disable_interrupts(struct acp63_dev_data *adata) +{ + if (adata && adata->hw_ops && adata->hw_ops->disable_interrupts) + ACP_HW_OPS(adata, disable_interrupts)(adata->acp63_base); +} + static inline void acp_hw_get_config(struct pci_dev *pci, struct acp63_dev_data *adata) { if (adata && adata->hw_ops && adata->hw_ops->acp_get_config) diff --git a/sound/soc/amd/ps/pci-ps.c b/sound/soc/amd/ps/pci-ps.c index 729f9aaba69e..b604db4942dd 100644 --- a/sound/soc/amd/ps/pci-ps.c +++ b/sound/soc/amd/ps/pci-ps.c @@ -738,6 +738,8 @@ static void snd_acp63_remove(struct pci_dev *pci) int ret; adata = pci_get_drvdata(pci); + acp_hw_disable_interrupts(adata); + devm_free_irq(&pci->dev, pci->irq, adata); if (adata->sdw) { amd_sdw_exit(adata); platform_device_unregister(adata->sdw_dma_dev); diff --git a/sound/soc/amd/ps/ps-common.c b/sound/soc/amd/ps/ps-common.c index 7b4966b75dc6..ea71cdf19c1e 100644 --- a/sound/soc/amd/ps/ps-common.c +++ b/sound/soc/amd/ps/ps-common.c @@ -248,6 +248,7 @@ void acp63_hw_init_ops(struct acp_hw_ops *hw_ops) { hw_ops->acp_init = acp63_init; hw_ops->acp_deinit = acp63_deinit; + hw_ops->disable_interrupts = acp63_disable_interrupts; hw_ops->acp_get_config = acp63_get_config; hw_ops->acp_sdw_dma_irq_thread = acp63_sdw_dma_irq_thread; hw_ops->acp_suspend = snd_acp63_suspend; @@ -484,6 +485,7 @@ void acp70_hw_init_ops(struct acp_hw_ops *hw_ops) { hw_ops->acp_init = acp70_init; hw_ops->acp_deinit = acp70_deinit; + hw_ops->disable_interrupts = acp70_disable_interrupts; hw_ops->acp_get_config = acp70_get_config; hw_ops->acp_sdw_dma_irq_thread = acp70_sdw_dma_irq_thread; hw_ops->acp_suspend = snd_acp70_suspend; diff --git a/sound/soc/generic/audio-graph-card2.c b/sound/soc/generic/audio-graph-card2.c index 9fb3d3df5cf6..87fb4ca32619 100644 --- a/sound/soc/generic/audio-graph-card2.c +++ b/sound/soc/generic/audio-graph-card2.c @@ -969,7 +969,8 @@ int audio_graph2_link_c2c(struct simple_util_priv *priv, struct snd_soc_dai_link *dai_link = simple_priv_to_link(priv, li->link); struct device_node *port0 = lnk; struct device_node *ports __free(device_node) = port_to_ports(port0); - struct device_node *port1 __free(device_node) = of_graph_get_next_port(ports, port0); + struct device_node *port1 __free(device_node) = + of_graph_get_next_port(ports, of_node_get(port0)); u32 val = 0; int ret = -EINVAL; diff --git a/sound/synth/emux/emux_synth.c b/sound/synth/emux/emux_synth.c index 6982e1317da5..2cbf12364a1e 100644 --- a/sound/synth/emux/emux_synth.c +++ b/sound/synth/emux/emux_synth.c @@ -60,6 +60,9 @@ snd_emux_note_on(void *p, int note, int vel, struct snd_midi_channel *chan) return; key = note; /* remember the original note */ + + /* hold sflist->lock for protecting against concurrent free */ + guard(spinlock_irqsave)(&emu->sflist->lock); nvoices = get_zone(emu, port, ¬e, vel, chan, table); if (! nvoices) return; diff --git a/sound/synth/emux/soundfont.c b/sound/synth/emux/soundfont.c index c172b53b17a4..039357d96e07 100644 --- a/sound/synth/emux/soundfont.c +++ b/sound/synth/emux/soundfont.c @@ -1225,6 +1225,9 @@ delete_preset(struct snd_sf_list *sflist, struct snd_sf_zone *zp) * The found zones are stored on 'table' array. max_layers defines * the maximum number of elements in this array. * This function returns the number of found zones. 0 if not found. + * + * The caller must hold sflist->lock, and keep holding it as long as + * the returned zones are accessed. */ int snd_soundfont_search_zone(struct snd_sf_list *sflist, int *notep, int vel, @@ -1234,11 +1237,11 @@ snd_soundfont_search_zone(struct snd_sf_list *sflist, int *notep, int vel, { int nvoices; - /* this function is supposed to be called atomically, - * so we check the lock. if it's busy, just returns 0 to + lockdep_assert_held(&sflist->lock); + + /* if the presets are being modified, just returns 0 to * tell the caller the busy state */ - guard(spinlock_irqsave)(&sflist->lock); if (sflist->presets_locked) return 0; nvoices = search_zones(sflist, notep, vel, preset, bank, diff --git a/sound/usb/line6/pcm.c b/sound/usb/line6/pcm.c index 2932eaf157f4..b4c594add1ae 100644 --- a/sound/usb/line6/pcm.c +++ b/sound/usb/line6/pcm.c @@ -173,6 +173,14 @@ static void line6_buffer_release(struct snd_line6_pcm *line6pcm, clear_bit(type, &pstr->opened); if (!pstr->opened) { line6_wait_clear_audio_urbs(line6pcm, pstr); + if (pstr == &line6pcm->in) { + /* prev_fbuf points into the capture buffer we are about + * to free; drop it while no capture URB is in flight. + */ + guard(spinlock_irqsave)(&line6pcm->in.lock); + line6pcm->prev_fbuf = NULL; + line6pcm->prev_fsize = 0; + } kfree(pstr->buffer); pstr->buffer = NULL; } diff --git a/sound/x86/intel_hdmi_audio.c b/sound/x86/intel_hdmi_audio.c index 91dfcf47e979..f2377d40c5d1 100644 --- a/sound/x86/intel_hdmi_audio.c +++ b/sound/x86/intel_hdmi_audio.c @@ -475,7 +475,8 @@ static void had_build_channel_allocation_map(struct snd_intelhad *intelhaddata) kfree(intelhaddata->chmap->chmap); intelhaddata->chmap->chmap = NULL; - chmap = kzalloc_obj(*chmap); + /* info->chmap must be zero-terminated; allocate a spare terminator entry */ + chmap = kzalloc_objs(*chmap, 2); if (!chmap) return; @@ -575,6 +576,12 @@ static int had_register_chmap_ctls(struct snd_intelhad *intelhaddata, intelhaddata->chmap->kctl->info = had_chmap_ctl_info; intelhaddata->chmap->kctl->get = had_chmap_ctl_get; intelhaddata->chmap->chmap = NULL; + + /* drop the inherited TLV read: it walks chmap without our mutex */ + intelhaddata->chmap->kctl->tlv.c = NULL; + intelhaddata->chmap->kctl->vd[0].access &= + ~(SNDRV_CTL_ELEM_ACCESS_TLV_READ | SNDRV_CTL_ELEM_ACCESS_TLV_CALLBACK); + return 0; } diff --git a/tools/objtool/check.c b/tools/objtool/check.c index 464f6c9d9ff0..931c5c412f15 100644 --- a/tools/objtool/check.c +++ b/tools/objtool/check.c @@ -201,6 +201,7 @@ static bool is_rust_noreturn(const struct symbol *func) str_ends_with(func->name, "_4core6option13expect_failed") || str_ends_with(func->name, "_4core6option13unwrap_failed") || str_ends_with(func->name, "_4core6result13unwrap_failed") || + str_ends_with(func->name, "_4core6result13unwrap_failed19unwrap_failed_inner") || str_ends_with(func->name, "_4core9panicking5panic") || str_ends_with(func->name, "_4core9panicking9panic_fmt") || str_ends_with(func->name, "_4core9panicking14panic_explicit") || diff --git a/virt/kvm/guest_memfd.c b/virt/kvm/guest_memfd.c index 625e62e1a031..dcb0cd16c4aa 100644 --- a/virt/kvm/guest_memfd.c +++ b/virt/kvm/guest_memfd.c @@ -603,19 +603,21 @@ int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args) return __kvm_gmem_create(kvm, size, flags); } -int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, - unsigned int fd, uoff_t offset) +int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot, + unsigned int fd, uoff_t offset) { uoff_t size = slot->npages << PAGE_SHIFT; - unsigned long start, end; struct gmem_file *f; struct inode *inode; struct file *file; - int r = -EINVAL; + BUILD_BUG_ON(sizeof(gpa_t) != sizeof(offset)); BUILD_BUG_ON(sizeof(gfn_t) != sizeof(slot->gmem.pgoff)); + if (WARN_ON_ONCE(slot->flags & KVM_MEMSLOT_GMEM_ONLY)) + return -EINVAL; + file = fget(fd); if (!file) return -EBADF; @@ -632,84 +634,92 @@ int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, if (!PAGE_ALIGNED(offset) || offset + size > i_size_read(inode)) goto err; - filemap_invalidate_lock(inode->i_mapping); - - start = offset >> PAGE_SHIFT; - end = start + slot->npages; - - if (!xa_empty(&f->bindings) && - xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) { - r = -EEXIST; - filemap_invalidate_unlock(inode->i_mapping); - goto err; - } - /* * memslots of flag KVM_MEM_GUEST_MEMFD are immutable to change, so * kvm_gmem_bind() must occur on a new memslot. Because the memslot * is not visible yet, kvm_gmem_get_pfn() is guaranteed to see the file. */ WRITE_ONCE(slot->gmem.file, file); - slot->gmem.pgoff = start; + slot->gmem.pgoff = offset >> PAGE_SHIFT; if (kvm_gmem_supports_mmap(inode)) slot->flags |= KVM_MEMSLOT_GMEM_ONLY; - xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL); - filemap_invalidate_unlock(inode->i_mapping); - /* - * Drop the reference to the file, even on success. The file pins KVM, - * not the other way 'round. Active bindings are invalidated if the - * file is closed before memslots are destroyed. + * Gift the caller a reference to the file. The reference will be + * dropped after bindings are established, or if installing the new + * memslot ultimately fails. */ - r = 0; + return 0; + err: fput(file); - return r; + return -EINVAL; } -static void __kvm_gmem_unbind(struct kvm_memory_slot *slot, struct gmem_file *f) +int kvm_gmem_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot) { - unsigned long start = slot->gmem.pgoff; - unsigned long end = start + slot->npages; + struct gmem_file *f = slot->gmem.file->private_data; + struct inode *inode = file_inode(slot->gmem.file); + unsigned long start, end; + int r; - xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL); + if (WARN_ON_ONCE(slot->gmem.file->f_op != &kvm_gmem_fops)) + return -EIO; - /* - * synchronize_srcu(&kvm->srcu) ensured that kvm_gmem_get_pfn() - * cannot see this memslot. - */ - WRITE_ONCE(slot->gmem.file, NULL); + filemap_invalidate_lock(inode->i_mapping); + + start = slot->gmem.pgoff; + end = start + slot->npages; + + if (!xa_empty(&f->bindings) && + xa_find(&f->bindings, &start, end - 1, XA_PRESENT)) { + filemap_invalidate_unlock(inode->i_mapping); + return -EEXIST; + } + + r = xa_err(xa_store_range(&f->bindings, start, end - 1, slot, GFP_KERNEL)); + if (r) + xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL); + + filemap_invalidate_unlock(inode->i_mapping); + + return r; } void kvm_gmem_unbind(struct kvm_memory_slot *slot) { + struct file *file = slot->gmem.file; + unsigned long start = slot->gmem.pgoff; + unsigned long end = start + slot->npages; + struct gmem_file *f; + /* * Nothing to do if the underlying file was _already_ closed, as * kvm_gmem_release() invalidates and nullifies all bindings. */ - if (!slot->gmem.file) + if (!file) return; - CLASS(gmem_get_file, file)(slot); - /* * However, if the file is _being_ closed, then the bindings need to be * removed as kvm_gmem_release() might not run until after the memslot - * is freed. Note, modifying the bindings is safe even though the file - * is dying as kvm_gmem_release() nullifies slot->gmem.file under - * slots_lock, and only puts its reference to KVM after destroying all - * bindings. I.e. reaching this point means kvm_gmem_release() hasn't - * yet destroyed the bindings or freed the gmem_file, and can't do so - * until the caller drops slots_lock. + * is freed. Modifying the bindings is safe even if the file is dying + * as kvm_gmem_release() nullifies slot->gmem.file under slots_lock, + * and only puts its reference to KVM after destroying all bindings. + * I.e. reaching this point means kvm_gmem_release() hasn't destroyed + * the bindings or freed the gmem_file and can't do so until the caller + * drops slots_lock, so there's no need to verify the file is live. */ - if (!file) { - __kvm_gmem_unbind(slot, slot->gmem.file->private_data); - return; - } + f = file->private_data; filemap_invalidate_lock(file->f_mapping); - __kvm_gmem_unbind(slot, file->private_data); + xa_store_range(&f->bindings, start, end - 1, NULL, GFP_KERNEL); + + /* + * synchronize_srcu(&kvm->srcu) ensured that kvm_gmem_get_pfn() + * cannot see this memslot. + */ + WRITE_ONCE(slot->gmem.file, NULL); filemap_invalidate_unlock(file->f_mapping); } diff --git a/virt/kvm/guest_memfd.h b/virt/kvm/guest_memfd.h index 0f9c6f840838..01bd359d27e3 100644 --- a/virt/kvm/guest_memfd.h +++ b/virt/kvm/guest_memfd.h @@ -8,8 +8,9 @@ int kvm_gmem_init(struct module *module); void kvm_gmem_exit(void); int kvm_gmem_create(struct kvm *kvm, struct kvm_create_guest_memfd *args); -int kvm_gmem_bind(struct kvm *kvm, struct kvm_memory_slot *slot, - unsigned int fd, uoff_t offset); +int kvm_gmem_prepare_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot, + unsigned int fd, uoff_t offset); +int kvm_gmem_commit_memory_region(struct kvm *kvm, struct kvm_memory_slot *slot); void kvm_gmem_unbind(struct kvm_memory_slot *slot); #else static inline int kvm_gmem_init(struct module *module) @@ -17,9 +18,17 @@ static inline int kvm_gmem_init(struct module *module) return 0; } static inline void kvm_gmem_exit(void) {}; -static inline int kvm_gmem_bind(struct kvm *kvm, - struct kvm_memory_slot *slot, - unsigned int fd, uoff_t offset) + +static inline int kvm_gmem_prepare_memory_region(struct kvm *kvm, + struct kvm_memory_slot *slot, + unsigned int fd, uoff_t offset) +{ + WARN_ON_ONCE(1); + return -EIO; +} + +static inline int kvm_gmem_commit_memory_region(struct kvm *kvm, + struct kvm_memory_slot *slot) { WARN_ON_ONCE(1); return -EIO; diff --git a/virt/kvm/kvm_main.c b/virt/kvm/kvm_main.c index 5a782849521c..6bd1c427fc03 100644 --- a/virt/kvm/kvm_main.c +++ b/virt/kvm/kvm_main.c @@ -813,6 +813,16 @@ static void kvm_mmu_notifier_invalidate_range_end(struct mmu_notifier *mn, /* Pairs with the increment in range_start(). */ spin_lock(&kvm->mn_invalidate_lock); + kvm->gpc_invalidate_seq++; + + /* + * As with the MMU sequence counter and mmu_invalidate_in_progress, the + * GPC sequence increase must be visible before the invalidate count + * goes to zero. Pairs with the smp_rmb() in + * mmu_notifier_retry_cache(). + */ + smp_wmb(); + if (!WARN_ON_ONCE(!kvm->mn_active_invalidate_count)) --kvm->mn_active_invalidate_count; wake = !kvm->mn_active_invalidate_count; @@ -1748,10 +1758,10 @@ static void kvm_commit_memory_region(struct kvm *kvm, kvm_destroy_dirty_bitmap(old); /* - * Unbind the guest_memfd instance as needed; the @new slot has - * already created its own binding. TODO: Drop the WARN when - * dirty logging guest_memfd memslots is supported. Until then, - * flags-only changes on guest_memfd slots should be impossible. + * TODO: Drop the WARN and do the unbind() call only for MOVE + * when dirty logging guest_memfd memslots is supported. Until + * then, flags-only changes on guest_memfd slots should also be + * impossible; unbind the old memslot for defense-in-depth. */ if (WARN_ON_ONCE(old->flags & KVM_MEM_GUEST_MEMFD)) kvm_gmem_unbind(old); @@ -1937,20 +1947,16 @@ static int kvm_set_memslot(struct kvm *kvm, } r = kvm_prepare_memory_region(kvm, old, new, change); - if (r) { - /* - * For DELETE/MOVE, revert the above INVALID change. No - * modifications required since the original slot was preserved - * in the inactive slots. Changing the active memslots also - * release slots_arch_lock. - */ - if (change == KVM_MR_DELETE || change == KVM_MR_MOVE) { - kvm_activate_memslot(kvm, invalid_slot, old); - kfree(invalid_slot); - } else { - mutex_unlock(&kvm->slots_arch_lock); + if (r) + goto err; + + if (change == KVM_MR_CREATE && (new->flags & KVM_MEM_GUEST_MEMFD)) { + r = kvm_gmem_commit_memory_region(kvm, new); + if (r) { + kvm_arch_free_memslot(kvm, new); + kvm_destroy_dirty_bitmap(new); + goto err; } - return r; } /* @@ -1983,6 +1989,20 @@ static int kvm_set_memslot(struct kvm *kvm, kvm_commit_memory_region(kvm, old, new, change); return 0; + +err: + /* + * For DELETE/MOVE, revert the above INVALID change. No modifications + * required since the original slot was preserved in the inactive slots. + * Changing the active memslots also release slots_arch_lock. + */ + if (change == KVM_MR_DELETE || change == KVM_MR_MOVE) { + kvm_activate_memslot(kvm, invalid_slot, old); + kfree(invalid_slot); + } else { + mutex_unlock(&kvm->slots_arch_lock); + } + return r; } static bool kvm_check_memslot_overlap(struct kvm_memslots *slots, int id, @@ -2111,21 +2131,30 @@ static int kvm_set_memory_region(struct kvm *kvm, new->npages = npages; new->flags = mem->flags; new->userspace_addr = mem->userspace_addr; - if (mem->flags & KVM_MEM_GUEST_MEMFD) { - r = kvm_gmem_bind(kvm, new, mem->guest_memfd, mem->guest_memfd_offset); + if (change == KVM_MR_CREATE && (mem->flags & KVM_MEM_GUEST_MEMFD)) { + r = kvm_gmem_prepare_memory_region(kvm, new, mem->guest_memfd, + mem->guest_memfd_offset); if (r) goto out; } r = kvm_set_memslot(kvm, old, new, change); + + /* + * Drop the reference to the gmem file, even on success. The file pins + * KVM, not the other way 'round. Active bindings are invalidated if + * the file is closed before memslots are destroyed. + */ +#ifdef CONFIG_KVM_GUEST_MEMFD + if (change == KVM_MR_CREATE && (mem->flags & KVM_MEM_GUEST_MEMFD)) + fput(new->gmem.file); +#endif + if (r) - goto out_unbind; + goto out; return 0; -out_unbind: - if (mem->flags & KVM_MEM_GUEST_MEMFD) - kvm_gmem_unbind(new); out: kfree(new); return r; diff --git a/virt/kvm/pfncache.c b/virt/kvm/pfncache.c index 728d2c1b488a..3659686b97c2 100644 --- a/virt/kvm/pfncache.c +++ b/virt/kvm/pfncache.c @@ -124,7 +124,7 @@ static void gpc_unmap(kvm_pfn_t pfn, void *khva) #endif } -static inline bool mmu_notifier_retry_cache(struct kvm *kvm, unsigned long mmu_seq) +static inline bool mmu_notifier_retry_cache(struct kvm *kvm, unsigned long gpc_seq) { /* * mn_active_invalidate_count acts for all intents and purposes @@ -136,20 +136,20 @@ static inline bool mmu_notifier_retry_cache(struct kvm *kvm, unsigned long mmu_s * Note, it does not matter that mn_active_invalidate_count * is not protected by gpc->lock. It is guaranteed to * be elevated before the mmu_notifier acquires gpc->lock, and - * isn't dropped until after mmu_invalidate_seq is updated. + * isn't dropped until after gpc_invalidate_seq is updated. */ if (kvm->mn_active_invalidate_count) return true; /* * Ensure mn_active_invalidate_count is read before - * mmu_invalidate_seq. This pairs with the smp_wmb() in - * mmu_notifier_invalidate_range_end() to guarantee either the + * gpc_invalidate_seq. This pairs with the smp_wmb() in + * kvm_mmu_notifier_invalidate_range_end() to guarantee either the * old (non-zero) value of mn_active_invalidate_count or the - * new (incremented) value of mmu_invalidate_seq is observed. + * new (incremented) value of gpc_invalidate_seq is observed. */ smp_rmb(); - return kvm->mmu_invalidate_seq != mmu_seq; + return kvm->gpc_invalidate_seq != gpc_seq; } static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc) @@ -158,7 +158,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc) void *old_khva = (void *)PAGE_ALIGN_DOWN((uintptr_t)gpc->khva); kvm_pfn_t new_pfn = KVM_PFN_ERR_FAULT; void *new_khva = NULL; - unsigned long mmu_seq; + unsigned long gpc_seq; struct page *page; struct kvm_follow_pfn kfp = { @@ -181,7 +181,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc) gpc->valid = false; do { - mmu_seq = gpc->kvm->mmu_invalidate_seq; + gpc_seq = gpc->kvm->gpc_invalidate_seq; smp_rmb(); write_unlock_irq(&gpc->lock); @@ -232,7 +232,7 @@ static kvm_pfn_t hva_to_pfn_retry(struct gfn_to_pfn_cache *gpc) * attempting to refresh. */ WARN_ON_ONCE(gpc->valid); - } while (mmu_notifier_retry_cache(gpc->kvm, mmu_seq)); + } while (mmu_notifier_retry_cache(gpc->kvm, gpc_seq)); gpc->valid = true; gpc->pfn = new_pfn;