From mboxrd@z Thu Jan 1 00:00:00 1970 Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.remote-ip=100.103.45.18 ARC-Seal: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; cv=none; t=1793365084; b=WHAvLcneIjbUnjOryWXDWMWBL1koYblR77Z2tnNjJozDskATtIMSbi2NwbsUTAQQ6OtT cLkZxCLxJ7NszKT3Cyc6M28k6M0AXHWynrA6KJt8WIrYicjV1qjJlbWMWRvhlKxQx9vWB SAbb7oXTSmwK6qbLRxZHUC4CXB3/AVkn1EDWMgvazkso+BFM1sx06Q/WCX3ujW0Fd1jBX sgsGfv16zoKRiu1okqGq6tZ/yFU830xedeWyL8rl8/TzBqpfYoSCDuJ4AU2PqOIv93WO4 xLFJJcMWfvDgLN0+SaKzejmWPS6oIlBC4RETygenm4QKLsVY/sE8nKI7B1184xhrseA== ARC-Message-Signature: i=1; d=subspace.kernel.org; s=arc20261015; a=rsa-sha256; c=relaxed/relaxed; t=1793365084; h=Received:DKIM-Signature:From:To:Cc:Subject:Date:Message-ID:X-Mailer: In-Reply-To:References:MIME-Version:Content-Type: Content-Transfer-Encoding; bh=gYaP/mcyxGo8g3EF+bY8GEQBj4FjERNaEK/EoysL7hI=; b=i9qz3xbT4WGzK77L0iAWtQNnNl96WDtgizVVU/V5TpnCkrh1dBIA0c1IFtEBnByU19vK 32HwMll5eixn8sWTNQ+iN+TqcQictoZa4zRqFmT8XjDsWU4OF3Icey2ni0hs+++dwHgvq 586Q816PxSchijEulnmrT35nzYScWSlSPDXd1YUxOOsehqknVSq/apms7BHqGxKhrCMs0 tr1e2EbozwlY+tqpYlbZVqIjG4JOsdjy3BJSup3lslcRNBhC73yNJYN/OPW8s9LI+m2d8 p2wQzdZXW+jjFL5gCsp61VQXuMSbyQCFxnm2PoeiIT89d/loZIcL87Goq96y6mFxPTQ== ARC-Authentication-Results: i=1; smtp.subspace.kernel.org; dkim=pass header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=bF6LJEHM; arc=none smtp.remote-ip=100.103.45.18 Received: from smtp.kernel.org (quasi.space.kernel.org [100.103.45.18]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (prime256v1)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 5D565606A320; Fri, 30 Oct 2026 12:58:03 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; dkim=pass (1024-bit key, unprotected) header.d=linuxfoundation.org header.i=@linuxfoundation.org header.a=rsa-sha256 header.s=korg header.b=bF6LJEHM Received: by smtp.kernel.org (Postfix) with ESMTPSA id D6FD01F000C1; Fri, 30 Oct 2026 12:58:01 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linuxfoundation.org; s=korg; t=1793365082; bh=gYaP/mcyxGo8g3EF+bY8GEQBj4FjERNaEK/EoysL7hI=; h=From:To:Cc:Subject:Date:In-Reply-To:References; b=bF6LJEHMEUKdI5fPrUceaXbYpKYPttWJO5VOCJGFVDdZdtPH1q3OrHopxXuKUX+wu zIswYMI6yiDzWsMC8vGQ5EQmDr0YBzu9dMKbtcq3eZH9sA5lgBu2n+e480CInhmht/ 0WZfZx6quNaaalPdyOzX7iUPueB6QoxwC4ZUPrL4= From: Greg Kroah-Hartman To: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, torvalds@linux-foundation.org, stable@vger.kernel.org Cc: lwn@lwn.net, jslaby@suse.cz, Greg Kroah-Hartman Subject: Re: Linux 6.6.161 Date: Fri, 30 Oct 2026 13:57:50 +0100 Message-ID: <2026103050-negligee-rectal-aa47@gregkh> X-Mailer: git-send-email 2.56.0 In-Reply-To: <2026103050-gutter-cobbler-4279@gregkh> References: <2026103050-gutter-cobbler-4279@gregkh> X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Post: List-Help: List-Subscribe: List-Unsubscribe: List-Owner: Precedence: list MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit diff --git a/Makefile b/Makefile index 0b151a07ee79..05e06ca0dac1 100644 --- a/Makefile +++ b/Makefile @@ -1,7 +1,7 @@ # SPDX-License-Identifier: GPL-2.0 VERSION = 6 PATCHLEVEL = 6 -SUBLEVEL = 160 +SUBLEVEL = 161 EXTRAVERSION = NAME = Pinguïn Aangedreven diff --git a/arch/nios2/include/asm/checksum.h b/arch/nios2/include/asm/checksum.h index 69004e07a1ba..b6b415ae17a0 100644 --- a/arch/nios2/include/asm/checksum.h +++ b/arch/nios2/include/asm/checksum.h @@ -27,7 +27,7 @@ static inline __sum16 csum_fold(__wsum sum) "add %0, %0, r8\n" "nor %0, %0, %0\n" : "=r" (sum) - : "r" (sum << 16), "0" (sum) + : "r" ((__force u32)sum << 16), "0" (sum) : "r8"); return (__force __sum16) sum; } diff --git a/arch/s390/pci/pci_sysfs.c b/arch/s390/pci/pci_sysfs.c index cae280e5c047..f2b615c597bc 100644 --- a/arch/s390/pci/pci_sysfs.c +++ b/arch/s390/pci/pci_sysfs.c @@ -146,6 +146,9 @@ static ssize_t report_error_write(struct file *filp, struct kobject *kobj, if (off || (count < sizeof(*report))) return -EINVAL; + if (count < (report->length + sizeof(*report))) + return -EINVAL; + ret = sclp_pci_report(report, zdev->fh, zdev->fid); return ret ? ret : count; diff --git a/arch/x86/kvm/mmu/mmu.c b/arch/x86/kvm/mmu/mmu.c index b243cb405808..5bff73562cbd 100644 --- a/arch/x86/kvm/mmu/mmu.c +++ b/arch/x86/kvm/mmu/mmu.c @@ -2508,7 +2508,9 @@ static int mmu_page_zap_pte(struct kvm *kvm, struct kvm_mmu_page *sp, * avoids retaining a large number of stale nested SPs. */ if (tdp_enabled && invalid_list && - child->role.guest_mode && !child->parent_ptes.val) + child->role.guest_mode && + !child->root_count && + !child->parent_ptes.val) return kvm_mmu_prepare_zap_page(kvm, child, invalid_list); } diff --git a/arch/x86/kvm/vmx/nested.c b/arch/x86/kvm/vmx/nested.c index 2d9cfdfa7806..0087daad441d 100644 --- a/arch/x86/kvm/vmx/nested.c +++ b/arch/x86/kvm/vmx/nested.c @@ -3568,8 +3568,10 @@ enum nvmx_vmentry_status nested_vmx_enter_non_root_mode(struct kvm_vcpu *vcpu, vmentry_fail_vmexit: vmx_switch_vmcs(vcpu, &vmx->vmcs01); - if (!from_vmentry) + if (!from_vmentry) { + nested_ept_uninit_mmu_context(vcpu); return NVMX_VMENTRY_VMEXIT; + } kvm_vcpu_unmap(vcpu, &vmx->nested.apic_access_page_map, false); kvm_vcpu_unmap(vcpu, &vmx->nested.virtual_apic_map, true); diff --git a/block/blk-mq.c b/block/blk-mq.c index 4c1c5a261d0c..e3fb9222c5ee 100644 --- a/block/blk-mq.c +++ b/block/blk-mq.c @@ -2984,17 +2984,7 @@ static bool blk_mq_can_use_cached_rq(struct request *rq, struct blk_plug *plug, if (op_is_flush(rq->cmd_flags) != op_is_flush(bio->bi_opf)) return false; - /* - * If any qos ->throttle() end up blocking, we will have flushed the - * plug and hence killed the cached_rq list as well. Pop this entry - * before we throttle. - */ plug->cached_rq = rq_list_next(rq); - rq_qos_throttle(rq->q, bio); - - blk_mq_rq_time_init(rq, 0); - rq->cmd_flags = bio->bi_opf; - INIT_LIST_HEAD(&rq->queuelist); return true; } @@ -3027,20 +3017,30 @@ void blk_mq_submit_bio(struct bio *bio) rq = rq_list_peek(&plug->cached_rq); if (rq && rq->q != q) rq = NULL; + if (rq && !blk_mq_can_use_cached_rq(rq, plug, bio)) + rq = NULL; } if (rq) { if (unlikely(bio_may_exceed_limits(bio, &q->limits))) { bio = __bio_split_to_limits(bio, &q->limits, &nr_segs); - if (!bio) + if (!bio) { + rq_list_add(&plug->cached_rq, rq); return; + } } - if (!bio_integrity_prep(bio)) + if (!bio_integrity_prep(bio)) { + rq_list_add(&plug->cached_rq, rq); return; - if (blk_mq_attempt_bio_merge(q, bio, nr_segs)) + } + if (blk_mq_attempt_bio_merge(q, bio, nr_segs)) { + rq_list_add(&plug->cached_rq, rq); return; - if (blk_mq_can_use_cached_rq(rq, plug, bio)) - goto done; - percpu_ref_get(&q->q_usage_counter); + } + rq_qos_throttle(rq->q, bio); + blk_mq_rq_time_init(rq, 0); + rq->cmd_flags = bio->bi_opf; + INIT_LIST_HEAD(&rq->queuelist); + goto done; } else { if (unlikely(bio_queue_enter(bio))) return; diff --git a/block/bsg-lib.c b/block/bsg-lib.c index b3acdbdb6e7e..5df816a0a58e 100644 --- a/block/bsg-lib.c +++ b/block/bsg-lib.c @@ -23,11 +23,14 @@ struct bsg_set { struct bsg_device *bd; bsg_job_fn *job_fn; bsg_timeout_fn *timeout_fn; + unsigned int reply_len; }; static int bsg_transport_sg_io_fn(struct request_queue *q, struct sg_io_v4 *hdr, bool open_for_write, unsigned int timeout) { + struct bsg_set *bset = + container_of(q->tag_set, struct bsg_set, tag_set); struct bsg_job *job; struct request *rq; struct bio *bio; @@ -50,7 +53,7 @@ static int bsg_transport_sg_io_fn(struct request_queue *q, struct sg_io_v4 *hdr, reply = job->reply; memset(job, 0, sizeof(*job)); job->reply = reply; - job->reply_len = SCSI_SENSE_BUFFERSIZE; + job->reply_len = bset->reply_len; job->dd_data = job + 1; job->request_len = hdr->request_len; @@ -301,9 +304,10 @@ static blk_status_t bsg_queue_rq(struct blk_mq_hw_ctx *hctx, static int bsg_init_rq(struct blk_mq_tag_set *set, struct request *req, unsigned int hctx_idx, unsigned int numa_node) { + struct bsg_set *bset = container_of(set, struct bsg_set, tag_set); struct bsg_job *job = blk_mq_rq_to_pdu(req); - job->reply = kzalloc(SCSI_SENSE_BUFFERSIZE, GFP_KERNEL); + job->reply = kzalloc(bset->reply_len, GFP_KERNEL); if (!job->reply) return -ENOMEM; return 0; @@ -357,9 +361,11 @@ static const struct blk_mq_ops bsg_mq_ops = { * @job_fn: bsg job handler * @timeout: timeout handler function pointer * @dd_job_size: size of LLD data needed for each job + * @reply_len: size of the reply buffer needed for each job */ struct request_queue *bsg_setup_queue(struct device *dev, const char *name, - bsg_job_fn *job_fn, bsg_timeout_fn *timeout, int dd_job_size) + bsg_job_fn *job_fn, bsg_timeout_fn *timeout, int dd_job_size, + unsigned int reply_len) { struct bsg_set *bset; struct blk_mq_tag_set *set; @@ -372,6 +378,7 @@ struct request_queue *bsg_setup_queue(struct device *dev, const char *name, bset->job_fn = job_fn; bset->timeout_fn = timeout; + bset->reply_len = reply_len; set = &bset->tag_set; set->ops = &bsg_mq_ops; diff --git a/crypto/asymmetric_keys/restrict.c b/crypto/asymmetric_keys/restrict.c index 6b69ea40da23..7323a1af6d27 100644 --- a/crypto/asymmetric_keys/restrict.c +++ b/crypto/asymmetric_keys/restrict.c @@ -236,10 +236,14 @@ static int key_or_keyring_common(struct key *dest_keyring, if (IS_ERR(key)) key = NULL; } else if (trusted->type == &key_type_asymmetric) { + const struct asymmetric_key_ids *kids; const struct asymmetric_key_id **signer_ids; - signer_ids = (const struct asymmetric_key_id **) - asymmetric_key_ids(trusted)->id; + kids = asymmetric_key_ids(trusted); + if (!kids) + return -ENOKEY; + + signer_ids = (const struct asymmetric_key_id **)kids->id; /* * The auth_ids come from the candidate key (the diff --git a/drivers/gpio/gpio-mxc.c b/drivers/gpio/gpio-mxc.c index a8ab78ae7fa3..5feac844c70b 100644 --- a/drivers/gpio/gpio-mxc.c +++ b/drivers/gpio/gpio-mxc.c @@ -7,6 +7,8 @@ // Authors: Daniel Mack, Juergen Beisert. // Copyright (C) 2004-2010 Freescale Semiconductor, Inc. All Rights Reserved. +#include +#include #include #include #include @@ -66,11 +68,11 @@ struct mxc_gpio_port { struct irq_domain *domain; struct gpio_chip gc; struct device *dev; - u32 both_edges; + unsigned long both_edges; struct mxc_gpio_reg_saved gpio_saved_reg; bool power_off; - u32 wakeup_pads; bool is_pad_wakeup; + unsigned long wakeup_pads; u32 pad_type[32]; const struct mxc_gpio_hwdata *hwdata; }; @@ -161,13 +163,12 @@ static int gpio_set_irq_type(struct irq_data *d, u32 type) { struct irq_chip_generic *gc = irq_data_get_irq_chip_data(d); struct mxc_gpio_port *port = gc->private; - unsigned long flags; u32 bit, val; u32 gpio_idx = d->hwirq; int edge; void __iomem *reg = port->base; - port->both_edges &= ~(1 << gpio_idx); + clear_bit(gpio_idx, &port->both_edges); switch (type) { case IRQ_TYPE_EDGE_RISING: edge = GPIO_INT_RISE_EDGE; @@ -187,7 +188,7 @@ static int gpio_set_irq_type(struct irq_data *d, u32 type) edge = GPIO_INT_HIGH_LEV; pr_debug("mxc: set GPIO %d to high trigger\n", gpio_idx); } - port->both_edges |= 1 << gpio_idx; + set_bit(gpio_idx, &port->both_edges); } break; case IRQ_TYPE_LEVEL_LOW: @@ -200,41 +201,38 @@ static int gpio_set_irq_type(struct irq_data *d, u32 type) return -EINVAL; } - raw_spin_lock_irqsave(&port->gc.bgpio_lock, flags); + scoped_guard(raw_spinlock_irqsave, &port->gc.bgpio_lock) { + if (GPIO_EDGE_SEL >= 0) { + val = readl(port->base + GPIO_EDGE_SEL); + if (edge == GPIO_INT_BOTH_EDGES) + writel(val | (1 << gpio_idx), + port->base + GPIO_EDGE_SEL); + else + writel(val & ~(1 << gpio_idx), + port->base + GPIO_EDGE_SEL); + } - if (GPIO_EDGE_SEL >= 0) { - val = readl(port->base + GPIO_EDGE_SEL); - if (edge == GPIO_INT_BOTH_EDGES) - writel(val | (1 << gpio_idx), - port->base + GPIO_EDGE_SEL); - else - writel(val & ~(1 << gpio_idx), - port->base + GPIO_EDGE_SEL); - } + if (edge != GPIO_INT_BOTH_EDGES) { + reg += GPIO_ICR1 + ((gpio_idx & 0x10) >> 2); /* lower or upper register */ + bit = gpio_idx & 0xf; + val = readl(reg) & ~(0x3 << (bit << 1)); + writel(val | (edge << (bit << 1)), reg); + } - if (edge != GPIO_INT_BOTH_EDGES) { - reg += GPIO_ICR1 + ((gpio_idx & 0x10) >> 2); /* lower or upper register */ - bit = gpio_idx & 0xf; - val = readl(reg) & ~(0x3 << (bit << 1)); - writel(val | (edge << (bit << 1)), reg); + writel(1 << gpio_idx, port->base + GPIO_ISR); + port->pad_type[gpio_idx] = type; } - writel(1 << gpio_idx, port->base + GPIO_ISR); - port->pad_type[gpio_idx] = type; - - raw_spin_unlock_irqrestore(&port->gc.bgpio_lock, flags); - return port->gc.direction_input(&port->gc, gpio_idx); } static void mxc_flip_edge(struct mxc_gpio_port *port, u32 gpio) { void __iomem *reg = port->base; - unsigned long flags; u32 bit, val; int edge; - raw_spin_lock_irqsave(&port->gc.bgpio_lock, flags); + guard(raw_spinlock_irqsave)(&port->gc.bgpio_lock); reg += GPIO_ICR1 + ((gpio & 0x10) >> 2); /* lower or upper register */ bit = gpio & 0xf; @@ -250,12 +248,9 @@ static void mxc_flip_edge(struct mxc_gpio_port *port, u32 gpio) } else { pr_err("mxc: invalid configuration for GPIO %d: %x\n", gpio, edge); - goto unlock; + return; } writel(val | (edge << (bit << 1)), reg); - -unlock: - raw_spin_unlock_irqrestore(&port->gc.bgpio_lock, flags); } /* handle 32 interrupts in one status register */ @@ -264,7 +259,7 @@ static void mxc_gpio_irq_handler(struct mxc_gpio_port *port, u32 irq_stat) while (irq_stat != 0) { int irqoffset = fls(irq_stat) - 1; - if (port->both_edges & (1 << irqoffset)) + if (test_bit(irqoffset, &port->both_edges)) mxc_flip_edge(port, irqoffset); generic_handle_domain_irq(port->domain, irqoffset); @@ -335,16 +330,18 @@ static int gpio_set_wake_irq(struct irq_data *d, u32 enable) ret = enable_irq_wake(port->irq_high); else ret = enable_irq_wake(port->irq); - port->wakeup_pads |= (1 << gpio_idx); } else { if (port->irq_high && (gpio_idx >= 16)) ret = disable_irq_wake(port->irq_high); else ret = disable_irq_wake(port->irq); - port->wakeup_pads &= ~(1 << gpio_idx); } + if (ret) + return ret; - return ret; + assign_bit(gpio_idx, &port->wakeup_pads, enable); + + return 0; } static int mxc_gpio_init_gc(struct mxc_gpio_port *port, int irq_base) diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c index df7a5775d361..6709041c4b94 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c @@ -1440,6 +1440,7 @@ static void amdgpu_cs_parser_fini(struct amdgpu_cs_parser *parser) int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) { struct amdgpu_device *adev = drm_to_adev(dev); + struct amdgpu_bo_list_entry *e; struct amdgpu_cs_parser parser; int r; @@ -1495,6 +1496,11 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp) return 0; error_backoff: + /* amdgpu_cs_submit() only drops the ranges once it got that far. */ + amdgpu_bo_list_for_each_userptr_entry(e, parser.bo_list) { + amdgpu_ttm_tt_get_user_pages_done(e->bo->tbo.ttm, e->range); + e->range = NULL; + } mutex_unlock(&parser.bo_list->bo_list_mutex); error_fini: diff --git a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c index 7c304c1c9c6f..e6fe4aa8ceac 100644 --- a/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c +++ b/drivers/gpu/drm/amd/amdgpu/amdgpu_debugfs.c @@ -1157,7 +1157,6 @@ static ssize_t amdgpu_debugfs_gpr_read(struct file *f, char __user *buf, { struct amdgpu_device *adev = f->f_inode->i_private; int r; - ssize_t result = 0; uint32_t offset, se, sh, cu, wave, simd, thread, bank, *data; if (size > 4096 || size & 3 || *pos & 3) @@ -1202,25 +1201,11 @@ static ssize_t amdgpu_debugfs_gpr_read(struct file *f, char __user *buf, pm_runtime_mark_last_busy(adev_to_drm(adev)->dev); pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); + amdgpu_virt_disable_access_debugfs(adev); - while (size) { - uint32_t value; - - value = data[result >> 2]; - r = put_user(value, (uint32_t *)buf); - if (r) { - amdgpu_virt_disable_access_debugfs(adev); - goto err; - } - - result += 4; - buf += 4; - size -= 4; - } - + r = copy_to_user(buf, data, size) ? -EFAULT : size; kfree(data); - amdgpu_virt_disable_access_debugfs(adev); - return result; + return r; err: pm_runtime_put_autosuspend(adev_to_drm(adev)->dev); diff --git a/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c b/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c index 344202870aeb..f8f42b67c6b2 100644 --- a/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c +++ b/drivers/gpu/drm/amd/amdgpu/sdma_v3_0.c @@ -454,6 +454,9 @@ static void sdma_v3_0_ring_emit_hdp_flush(struct amdgpu_ring *ring) else ref_and_mask = REG_SET_FIELD(ref_and_mask, GPU_HDP_FLUSH_DONE, SDMA1, 1); + /* Hang workaround */ + amdgpu_ring_write(ring, SDMA_PKT_NOP_HEADER_OP(SDMA_OP_NOP)); + amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_POLL_REGMEM) | SDMA_PKT_POLL_REGMEM_HEADER_HDP_FLUSH(1) | SDMA_PKT_POLL_REGMEM_HEADER_FUNC(3)); /* == */ @@ -481,6 +484,10 @@ static void sdma_v3_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se unsigned flags) { bool write64bit = flags & AMDGPU_FENCE_FLAG_64BIT; + + /* Hang workaround */ + amdgpu_ring_write(ring, SDMA_PKT_NOP_HEADER_OP(SDMA_OP_NOP)); + /* write the fence */ amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_FENCE)); amdgpu_ring_write(ring, lower_32_bits(addr)); @@ -489,7 +496,11 @@ static void sdma_v3_0_ring_emit_fence(struct amdgpu_ring *ring, u64 addr, u64 se /* optionally write high bits as well */ if (write64bit) { + /* Hang workaround */ + amdgpu_ring_write(ring, SDMA_PKT_NOP_HEADER_OP(SDMA_OP_NOP)); + addr += 4; + amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_FENCE)); amdgpu_ring_write(ring, lower_32_bits(addr)); amdgpu_ring_write(ring, upper_32_bits(addr)); @@ -1035,6 +1046,9 @@ static void sdma_v3_0_ring_emit_pipeline_sync(struct amdgpu_ring *ring) uint32_t seq = ring->fence_drv.sync_seq; uint64_t addr = ring->fence_drv.gpu_addr; + /* Hang workaround */ + amdgpu_ring_write(ring, SDMA_PKT_NOP_HEADER_OP(SDMA_OP_NOP)); + /* wait for idle */ amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_POLL_REGMEM) | SDMA_PKT_POLL_REGMEM_HEADER_HDP_FLUSH(0) | @@ -1063,6 +1077,9 @@ static void sdma_v3_0_ring_emit_vm_flush(struct amdgpu_ring *ring, { amdgpu_gmc_emit_flush_gpu_tlb(ring, vmid, pd_addr); + /* Hang workaround */ + amdgpu_ring_write(ring, SDMA_PKT_NOP_HEADER_OP(SDMA_OP_NOP)); + /* wait for flush */ amdgpu_ring_write(ring, SDMA_PKT_HEADER_OP(SDMA_OP_POLL_REGMEM) | SDMA_PKT_POLL_REGMEM_HEADER_HDP_FLUSH(0) | @@ -1571,11 +1588,11 @@ static const struct amdgpu_ring_funcs sdma_v3_0_ring_funcs = { .get_wptr = sdma_v3_0_ring_get_wptr, .set_wptr = sdma_v3_0_ring_set_wptr, .emit_frame_size = - 6 + /* sdma_v3_0_ring_emit_hdp_flush */ + 6 + 1 + /* sdma_v3_0_ring_emit_hdp_flush */ 3 + /* hdp invalidate */ - 6 + /* sdma_v3_0_ring_emit_pipeline_sync */ - VI_FLUSH_GPU_TLB_NUM_WREG * 3 + 6 + /* sdma_v3_0_ring_emit_vm_flush */ - 10 + 10 + 10, /* sdma_v3_0_ring_emit_fence x3 for user fence, vm fence */ + 6 + 1 + /* sdma_v3_0_ring_emit_pipeline_sync */ + VI_FLUSH_GPU_TLB_NUM_WREG * 3 + 6 + 1 + /* sdma_v3_0_ring_emit_vm_flush */ + (10 + 1) * 3, /* sdma_v3_0_ring_emit_fence x3 for user fence, vm fence */ .emit_ib_size = 7 + 6, /* sdma_v3_0_ring_emit_ib */ .emit_ib = sdma_v3_0_ring_emit_ib, .emit_fence = sdma_v3_0_ring_emit_fence, diff --git a/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c b/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c index 1b5afa08f52b..18df0735ef77 100644 --- a/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c +++ b/drivers/gpu/drm/amd/amdgpu/vce_v3_0.c @@ -959,7 +959,7 @@ static const struct amdgpu_ring_funcs vce_v3_0_ring_phys_funcs = { static const struct amdgpu_ring_funcs vce_v3_0_ring_vm_funcs = { .type = AMDGPU_RING_TYPE_VCE, - .align_mask = 0xf, + .align_mask = 0x1f, .nop = VCE_CMD_NO_OP, .support_64bit_ptrs = false, .no_user_fence = true, diff --git a/drivers/gpu/drm/drm_exec.c b/drivers/gpu/drm/drm_exec.c index d22d487a8356..4947447598ef 100644 --- a/drivers/gpu/drm/drm_exec.c +++ b/drivers/gpu/drm/drm_exec.c @@ -50,6 +50,16 @@ /* Dummy value used to initially enter the retry loop */ #define DRM_EXEC_DUMMY ((void *)~0) +/* Unlock and put the prelocked obj */ +static void drm_exec_drop_prelocked(struct drm_exec *exec) +{ + if (exec->prelocked) { + dma_resv_unlock(exec->prelocked->resv); + drm_gem_object_put(exec->prelocked); + exec->prelocked = NULL; + } +} + /* Unlock all objects and drop references */ static void drm_exec_unlock_all(struct drm_exec *exec) { @@ -61,8 +71,7 @@ static void drm_exec_unlock_all(struct drm_exec *exec) drm_gem_object_put(obj); } - drm_gem_object_put(exec->prelocked); - exec->prelocked = NULL; + drm_exec_drop_prelocked(exec); } /** @@ -114,6 +123,14 @@ EXPORT_SYMBOL(drm_exec_fini); bool drm_exec_cleanup(struct drm_exec *exec) { if (likely(!exec->contended)) { + + /* + * The contended and prelocked object might never be passed to + * drm_exec_lock_obj() again. This can happen when the object + * is about to be cleaned up and we now hold the last reference + * to it. Make sure that we don't unnecessarily keep it around. + */ + drm_exec_drop_prelocked(exec); ww_acquire_done(&exec->ticket); return false; } @@ -172,16 +189,16 @@ static int drm_exec_lock_contended(struct drm_exec *exec) dma_resv_lock_slow(obj->resv, &exec->ticket); } - ret = drm_exec_obj_locked(exec, obj); - if (unlikely(ret)) - goto error_unlock; + /* + * It is perfectly possible that we see another contention before the + * first one is fully handled. + */ + drm_exec_drop_prelocked(exec); + /* We move the reference from contended to prelocked here */ exec->prelocked = obj; return 0; -error_unlock: - dma_resv_unlock(obj->resv); - error_dropref: drm_gem_object_put(obj); return ret; @@ -209,35 +226,31 @@ int drm_exec_lock_obj(struct drm_exec *exec, struct drm_gem_object *obj) if (exec->prelocked == obj) { drm_gem_object_put(exec->prelocked); exec->prelocked = NULL; - return 0; - } + } else { + if (exec->flags & DRM_EXEC_INTERRUPTIBLE_WAIT) + ret = dma_resv_lock_interruptible(obj->resv, + &exec->ticket); + else + ret = dma_resv_lock(obj->resv, &exec->ticket); + + if (unlikely(ret == -EDEADLK)) { + drm_gem_object_get(obj); + exec->contended = obj; + return -EDEADLK; + } - if (exec->flags & DRM_EXEC_INTERRUPTIBLE_WAIT) - ret = dma_resv_lock_interruptible(obj->resv, &exec->ticket); - else - ret = dma_resv_lock(obj->resv, &exec->ticket); + if (unlikely(ret == -EALREADY) && + exec->flags & DRM_EXEC_IGNORE_DUPLICATES) + return 0; - if (unlikely(ret == -EDEADLK)) { - drm_gem_object_get(obj); - exec->contended = obj; - return -EDEADLK; + if (unlikely(ret)) + return ret; } - if (unlikely(ret == -EALREADY) && - exec->flags & DRM_EXEC_IGNORE_DUPLICATES) - return 0; - - if (unlikely(ret)) - return ret; - ret = drm_exec_obj_locked(exec, obj); if (ret) - goto error_unlock; - - return 0; + dma_resv_unlock(obj->resv); -error_unlock: - dma_resv_unlock(obj->resv); return ret; } EXPORT_SYMBOL(drm_exec_lock_obj); diff --git a/drivers/gpu/drm/i915/display/intel_dmc.c b/drivers/gpu/drm/i915/display/intel_dmc.c index 8751973b5730..8d42ae60d862 100644 --- a/drivers/gpu/drm/i915/display/intel_dmc.c +++ b/drivers/gpu/drm/i915/display/intel_dmc.c @@ -770,7 +770,7 @@ static u32 parse_dmc_fw_header(struct intel_dmc *dmc, } /* Cache the dmc header info. */ - if (mmio_count > mmio_count_max) { + if (!mmio_count || mmio_count > mmio_count_max) { drm_err(&i915->drm, "DMC firmware has wrong mmio count %u\n", mmio_count); return 0; } @@ -785,13 +785,21 @@ static u32 parse_dmc_fw_header(struct intel_dmc *dmc, dmc_info->mmioaddr[i] = _MMIO(mmioaddr[i]); dmc_info->mmiodata[i] = mmiodata[i]; } - dmc_info->mmio_count = mmio_count; - dmc_info->start_mmioaddr = start_mmioaddr; rem_size -= header_len_bytes; /* fw_size is in dwords, so multiplied by 4 to convert into bytes. */ - payload_size = dmc_header->fw_size * 4; + if (check_mul_overflow(dmc_header->fw_size, 4u, &payload_size)) { + drm_err(&i915->drm, "DMC fw_size too large (%u dwords)\n", + dmc_header->fw_size); + return 0; + } + + if (!payload_size) { + drm_err(&i915->drm, "DMC firmware has empty payload\n"); + return 0; + } + if (rem_size < payload_size) goto error_truncated; @@ -799,12 +807,15 @@ static u32 parse_dmc_fw_header(struct intel_dmc *dmc, drm_err(&i915->drm, "DMC FW too big (%u bytes)\n", payload_size); return 0; } - dmc_info->dmc_fw_size = dmc_header->fw_size; dmc_info->payload = kmalloc(payload_size, GFP_KERNEL); if (!dmc_info->payload) return 0; + dmc_info->mmio_count = mmio_count; + dmc_info->start_mmioaddr = start_mmioaddr; + dmc_info->dmc_fw_size = dmc_header->fw_size; + payload = (u8 *)(dmc_header) + header_len_bytes; memcpy(dmc_info->payload, payload, payload_size); diff --git a/drivers/hv/channel_mgmt.c b/drivers/hv/channel_mgmt.c index 3c6011a48dab..914d479d7872 100644 --- a/drivers/hv/channel_mgmt.c +++ b/drivers/hv/channel_mgmt.c @@ -743,13 +743,14 @@ static void init_vp_index(struct vmbus_channel *channel) u32 i, ncpu = num_online_cpus(); cpumask_var_t available_mask; struct cpumask *allocated_mask; + const struct cpumask *node_mask; const struct cpumask *hk_mask = housekeeping_cpumask(HK_TYPE_MANAGED_IRQ); u32 target_cpu; int numa_node; if (!perf_chn || !alloc_cpumask_var(&available_mask, GFP_KERNEL) || - cpumask_empty(hk_mask)) { + !cpumask_intersects(hk_mask, cpu_online_mask)) { /* * If the channel is not a performance critical * channel, bind it to VMBUS_CONNECT_CPU. @@ -771,14 +772,19 @@ static void init_vp_index(struct vmbus_channel *channel) next_numa_node_id = 0; continue; } - if (cpumask_empty(cpumask_of_node(numa_node))) + /* + * Try next NUMA node if current NUMA node has no CPU + * or doesn't contain any housekeeping CPU. + */ + node_mask = cpumask_of_node(numa_node); + if (!cpumask_intersects(node_mask, hk_mask)) continue; break; } allocated_mask = &hv_context.hv_numa_map[numa_node]; retry: - cpumask_xor(available_mask, allocated_mask, cpumask_of_node(numa_node)); + cpumask_xor(available_mask, allocated_mask, node_mask); cpumask_and(available_mask, available_mask, hk_mask); if (cpumask_empty(available_mask)) { diff --git a/drivers/hv/connection.c b/drivers/hv/connection.c index 3cdba13b132f..722c07ec05be 100644 --- a/drivers/hv/connection.c +++ b/drivers/hv/connection.c @@ -340,7 +340,6 @@ int vmbus_connect(void) cleanup: pr_err("Unable to connect to host\n"); - vmbus_connection.conn_state = DISCONNECTED; vmbus_disconnect(); kfree(msginfo); diff --git a/drivers/hv/vmbus_drv.c b/drivers/hv/vmbus_drv.c index 9df19a69feb6..44af1c455bbf 100644 --- a/drivers/hv/vmbus_drv.c +++ b/drivers/hv/vmbus_drv.c @@ -2707,7 +2707,6 @@ static void __exit vmbus_exit(void) hv_remove_kexec_handler(); hv_remove_crash_handler(); - vmbus_connection.conn_state = DISCONNECTED; hv_stimer_global_cleanup(); vmbus_disconnect(); if (vmbus_irq == -1) { diff --git a/drivers/i2c/busses/i2c-qcom-cci.c b/drivers/i2c/busses/i2c-qcom-cci.c index 85de659bb362..7050b01f3228 100644 --- a/drivers/i2c/busses/i2c-qcom-cci.c +++ b/drivers/i2c/busses/i2c-qcom-cci.c @@ -168,12 +168,12 @@ static irqreturn_t cci_isr(int irq, void *dev) } if (unlikely(val & CCI_IRQ_STATUS_0_I2C_M0_Q0Q1_HALT_ACK)) { - reset = CCI_RESET_CMD_M0_MASK; + reset |= CCI_RESET_CMD_M0_MASK; ret = IRQ_HANDLED; } if (unlikely(val & CCI_IRQ_STATUS_0_I2C_M1_Q0Q1_HALT_ACK)) { - reset = CCI_RESET_CMD_M1_MASK; + reset |= CCI_RESET_CMD_M1_MASK; ret = IRQ_HANDLED; } diff --git a/drivers/i2c/busses/i2c-wmt.c b/drivers/i2c/busses/i2c-wmt.c index 76118abc6e10..8682143e15ac 100644 --- a/drivers/i2c/busses/i2c-wmt.c +++ b/drivers/i2c/busses/i2c-wmt.c @@ -389,7 +389,7 @@ static int wmt_i2c_probe(struct platform_device *pdev) return -EINVAL; } - i2c_dev->clk = of_clk_get(np, 0); + i2c_dev->clk = devm_clk_get(&pdev->dev, NULL); if (IS_ERR(i2c_dev->clk)) { dev_err(&pdev->dev, "unable to request clock\n"); return PTR_ERR(i2c_dev->clk); diff --git a/drivers/md/dm-cache-target.c b/drivers/md/dm-cache-target.c index bbc5c8bb2b85..0d36b3a5e50a 100644 --- a/drivers/md/dm-cache-target.c +++ b/drivers/md/dm-cache-target.c @@ -340,8 +340,6 @@ struct cache { struct list_head invalidation_requests; sector_t migration_threshold; - wait_queue_head_t migration_wait; - atomic_t nr_allocated_migrations; /* * The number of in flight migrations that are performing @@ -397,7 +395,11 @@ struct cache { bool loaded_mappings:1; bool loaded_discards:1; - struct rw_semaphore background_work_lock; + /* background work management */ + bool background_work_allowed; + unsigned background_work_nr; + spinlock_t background_work_lock; + wait_queue_head_t background_work_wait; struct batcher committer; struct work_struct commit_ws; @@ -488,19 +490,13 @@ static struct dm_cache_migration *alloc_migration(struct cache *cache) memset(mg, 0, sizeof(*mg)); mg->cache = cache; - atomic_inc(&cache->nr_allocated_migrations); return mg; } static void free_migration(struct dm_cache_migration *mg) { - struct cache *cache = mg->cache; - - if (atomic_dec_and_test(&cache->nr_allocated_migrations)) - wake_up(&cache->migration_wait); - - mempool_free(mg, &cache->migration_pool); + mempool_free(mg, &mg->cache->migration_pool); } /*----------------------------------------------------------------*/ @@ -1031,34 +1027,39 @@ static void calc_discard_block_range(struct cache *cache, struct bio *bio, static void prevent_background_work(struct cache *cache) { - lockdep_off(); - down_write(&cache->background_work_lock); - lockdep_on(); + spin_lock_irq(&cache->background_work_lock); + cache->background_work_allowed = false; + wait_event_lock_irq(cache->background_work_wait, + cache->background_work_nr == 0, + cache->background_work_lock); + spin_unlock_irq(&cache->background_work_lock); } static void allow_background_work(struct cache *cache) { - lockdep_off(); - up_write(&cache->background_work_lock); - lockdep_on(); + spin_lock_irq(&cache->background_work_lock); + cache->background_work_allowed = true; + spin_unlock_irq(&cache->background_work_lock); } static bool background_work_begin(struct cache *cache) { bool r; - lockdep_off(); - r = down_read_trylock(&cache->background_work_lock); - lockdep_on(); - + spin_lock_irq(&cache->background_work_lock); + r = cache->background_work_allowed; + if (r) + cache->background_work_nr++; + spin_unlock_irq(&cache->background_work_lock); return r; } static void background_work_end(struct cache *cache) { - lockdep_off(); - up_read(&cache->background_work_lock); - lockdep_on(); + spin_lock_irq(&cache->background_work_lock); + if (--cache->background_work_nr == 0) + wake_up(&cache->background_work_wait); + spin_unlock_irq(&cache->background_work_lock); } /*----------------------------------------------------------------*/ @@ -1463,6 +1464,9 @@ static void invalidate_complete(struct dm_cache_migration *mg, bool success) struct bio_list bios; struct cache *cache = mg->cache; + if (success) + atomic_inc(&cache->stats.demotion); + bio_list_init(&bios); if (mg->cell) { if (dm_cell_unlock_v2(cache->prison, mg->cell, &bios)) @@ -1734,7 +1738,6 @@ static int map_bio(struct cache *cache, struct bio *bio, dm_oblock_t block, if (passthrough_mode(cache)) { if (bio_data_dir(bio) == WRITE) { bio_drop_shared_lock(cache, bio); - atomic_inc(&cache->stats.demotion); invalidate_start(cache, cblock, block, bio); return DM_MAPIO_SUBMITTED; } else @@ -2517,9 +2520,7 @@ static int cache_create(struct cache_args *ca, struct cache **result) spin_lock_init(&cache->lock); bio_list_init(&cache->deferred_bios); - atomic_set(&cache->nr_allocated_migrations, 0); atomic_set(&cache->nr_io_migrations, 0); - init_waitqueue_head(&cache->migration_wait); r = -ENOMEM; atomic_set(&cache->nr_dirty, 0); @@ -2601,8 +2602,10 @@ static int cache_create(struct cache_args *ca, struct cache **result) issue_op, cache, cache->wq); dm_iot_init(&cache->tracker); - init_rwsem(&cache->background_work_lock); - prevent_background_work(cache); + init_waitqueue_head(&cache->background_work_wait); + spin_lock_init(&cache->background_work_lock); + cache->background_work_allowed = false; + cache->background_work_nr = 0; *result = cache; return 0; diff --git a/drivers/md/dm-crypt.c b/drivers/md/dm-crypt.c index 834478059220..db5f689e58cc 100644 --- a/drivers/md/dm-crypt.c +++ b/drivers/md/dm-crypt.c @@ -1794,7 +1794,6 @@ static void crypt_dec_pending(struct dm_crypt_io *io) { struct crypt_config *cc = io->cc; struct bio *base_bio = io->base_bio; - blk_status_t error = io->error; if (!atomic_dec_and_test(&io->io_pending)) return; @@ -1816,7 +1815,7 @@ static void crypt_dec_pending(struct dm_crypt_io *io) else kfree(io->integrity_metadata); - base_bio->bi_status = error; + base_bio->bi_status = io->error; bio_endio(base_bio); } diff --git a/drivers/md/dm-ebs-target.c b/drivers/md/dm-ebs-target.c index 3a609ef22168..45c74aa87bdc 100644 --- a/drivers/md/dm-ebs-target.c +++ b/drivers/md/dm-ebs-target.c @@ -265,8 +265,7 @@ static int ebs_ctr(struct dm_target *ti, unsigned int argc, char **argv) r = -EINVAL; if (sscanf(argv[1], "%llu%c", &tmp, &dummy) != 1 || - tmp != (sector_t)tmp || - (sector_t)tmp >= ti->len) { + tmp != (sector_t)tmp) { ti->error = "Invalid device offset sector"; goto bad; } diff --git a/drivers/md/dm-integrity.c b/drivers/md/dm-integrity.c index aa6f3e71c4fa..a10143b51c98 100644 --- a/drivers/md/dm-integrity.c +++ b/drivers/md/dm-integrity.c @@ -1814,7 +1814,7 @@ static void integrity_metadata(struct work_struct *w) if (unlikely(dio->op == REQ_OP_DISCARD)) { unsigned int bi_size = dio->bio_details.bi_iter.bi_size; - unsigned int max_size = likely(checksums != checksums_onstack) ? PAGE_SIZE : HASH_MAX_DIGESTSIZE; + unsigned int max_size = likely(checksums != checksums_onstack) ? PAGE_SIZE : sizeof(checksums_onstack); unsigned int max_blocks = max_size / ic->tag_size; memset(checksums, DISCARD_FILLER, max_size); @@ -3515,6 +3515,9 @@ static void dm_integrity_io_hints(struct dm_target *ti, struct queue_limits *lim limits->io_min = ic->sectors_per_block << SECTOR_SHIFT; limits->dma_alignment = limits->logical_block_size - 1; } + + if (ic->internal_hash && (ic->mode == 'D' || ic->mode == 'B')) + ti->needs_stable_writes = true; } static void calculate_journal_section_size(struct dm_integrity_c *ic) diff --git a/drivers/md/dm-table.c b/drivers/md/dm-table.c index ed0a5e91968d..6d4ce9790a56 100644 --- a/drivers/md/dm-table.c +++ b/drivers/md/dm-table.c @@ -1944,7 +1944,7 @@ static int device_requires_stable_pages(struct dm_target *ti, struct dm_dev *dev, sector_t start, sector_t len, void *data) { - return bdev_stable_writes(dev->bdev); + return ti->needs_stable_writes || bdev_stable_writes(dev->bdev); } int dm_table_set_restrictions(struct dm_table *t, struct request_queue *q, diff --git a/drivers/net/bonding/bond_main.c b/drivers/net/bonding/bond_main.c index e799791daa45..e480c692513f 100644 --- a/drivers/net/bonding/bond_main.c +++ b/drivers/net/bonding/bond_main.c @@ -2314,7 +2314,9 @@ int bond_enslave(struct net_device *bond_dev, struct net_device *slave_dev, bpf_prog_inc(bond->xdp_prog); } - if (bond_mode_can_use_xmit_hash(bond)) + /* broadcast mode uses the all_slaves to loop through slaves. */ + if (bond_mode_can_use_xmit_hash(bond) || + BOND_MODE(bond) == BOND_MODE_BROADCAST) bond_update_slave_arr(bond, NULL); bond_xdp_set_features(bond_dev); @@ -2457,7 +2459,8 @@ static int __bond_release_one(struct net_device *bond_dev, bond_upper_dev_unlink(bond, slave); - if (bond_mode_can_use_xmit_hash(bond)) + if (bond_mode_can_use_xmit_hash(bond) || + BOND_MODE(bond) == BOND_MODE_BROADCAST) bond_update_slave_arr(bond, slave); slave_info(bond_dev, slave_dev, "Releasing %s interface\n", @@ -5109,13 +5112,18 @@ static void bond_set_slave_arr(struct bonding *bond, { struct bond_up_slave *usable, *all; - usable = rtnl_dereference(bond->usable_slaves); - rcu_assign_pointer(bond->usable_slaves, usable_slaves); - kfree_rcu(usable, rcu); - all = rtnl_dereference(bond->all_slaves); rcu_assign_pointer(bond->all_slaves, all_slaves); kfree_rcu(all, rcu); + + if (BOND_MODE(bond) == BOND_MODE_BROADCAST) { + kfree_rcu(usable_slaves, rcu); + return; + } + + usable = rtnl_dereference(bond->usable_slaves); + rcu_assign_pointer(bond->usable_slaves, usable_slaves); + kfree_rcu(usable, rcu); } static void bond_reset_slave_arr(struct bonding *bond) diff --git a/drivers/net/ethernet/broadcom/genet/bcmgenet.c b/drivers/net/ethernet/broadcom/genet/bcmgenet.c index affac9887f16..37f42e8787a0 100644 --- a/drivers/net/ethernet/broadcom/genet/bcmgenet.c +++ b/drivers/net/ethernet/broadcom/genet/bcmgenet.c @@ -2104,6 +2104,16 @@ static netdev_tx_t bcmgenet_xmit(struct sk_buff *skb, struct net_device *dev) goto out; } + /* We store Tx state in the control block, so the skb must not be + * shared, but ether_setup() leaves IFF_TX_SKB_SHARING set. + */ + skb = skb_share_check(skb, GFP_ATOMIC); + if (!skb) { + BCMGENET_STATS64_INC((&ring->stats64), dropped); + ret = NETDEV_TX_OK; + goto out; + } + /* Retain how many bytes will be sent on the wire, without TSB inserted * by transmit checksum offload */ diff --git a/drivers/net/gtp.c b/drivers/net/gtp.c index f0ee46d3c873..39515aba86e6 100644 --- a/drivers/net/gtp.c +++ b/drivers/net/gtp.c @@ -1827,6 +1827,7 @@ static int gtp_genl_send_echo_req(struct sk_buff *skb, struct genl_info *info) return -ENODEV; } + local_bh_disable(); udp_tunnel_xmit_skb(rt, sk, skb_to_send, fl4.saddr, fl4.daddr, fl4.flowi4_tos, @@ -1836,6 +1837,7 @@ static int gtp_genl_send_echo_req(struct sk_buff *skb, struct genl_info *info) !net_eq(sock_net(sk), dev_net(gtp->dev)), false); + local_bh_enable(); return 0; } diff --git a/drivers/net/wireless/intersil/p54/fwio.c b/drivers/net/wireless/intersil/p54/fwio.c index 06e5df90b43e..34988cba5753 100644 --- a/drivers/net/wireless/intersil/p54/fwio.c +++ b/drivers/net/wireless/intersil/p54/fwio.c @@ -52,6 +52,13 @@ int p54_parse_firmware(struct ieee80211_hw *dev, const struct firmware *fw) u32 code = le32_to_cpu(bootrec->code); switch (code) { case BR_CODE_COMPONENT_ID: + if (len < sizeof(struct bootrec_comp_id) / + sizeof(*bootrec->data)) { + wiphy_err(priv->hw->wiphy, + "firmware component ID is too short\n"); + return -EINVAL; + } + priv->fw_interface = be32_to_cpup((__be32 *) bootrec->data); switch (priv->fw_interface) { @@ -72,16 +79,41 @@ int p54_parse_firmware(struct ieee80211_hw *dev, const struct firmware *fw) } break; case BR_CODE_COMPONENT_VERSION: + if (len < DIV_ROUND_UP(sizeof(struct bootrec_comp_ver), + sizeof(*bootrec->data))) { + wiphy_err(priv->hw->wiphy, + "firmware component version is too short\n"); + return -EINVAL; + } + /* 24 bytes should be enough for all firmwares */ - if (strnlen((unsigned char *) bootrec->data, 24) < 24) - fw_version = (unsigned char *) bootrec->data; + if (strnlen((unsigned char *)bootrec->data, + sizeof(struct bootrec_comp_ver)) < + sizeof(struct bootrec_comp_ver)) + fw_version = (unsigned char *)bootrec->data; break; case BR_CODE_DESCR: { struct bootrec_desc *desc = (struct bootrec_desc *)bootrec->data; - priv->rx_start = le32_to_cpu(desc->rx_start); - /* FIXME add sanity checking */ - priv->rx_end = le32_to_cpu(desc->rx_end) - 0x3500; + u32 rx_start, rx_end; + + /* 0xa is the shortest descriptor in supported firmware. */ + if (len < 0xa) { + wiphy_err(priv->hw->wiphy, + "firmware descriptor is too short\n"); + return -EINVAL; + } + + rx_start = le32_to_cpu(desc->rx_start); + rx_end = le32_to_cpu(desc->rx_end); + if (rx_end < 0x3500 || rx_end - 0x3500 <= rx_start) { + wiphy_err(priv->hw->wiphy, + "firmware descriptor has invalid RX range\n"); + return -EINVAL; + } + + priv->rx_start = rx_start; + priv->rx_end = rx_end - 0x3500; priv->headroom = desc->headroom; priv->tailroom = desc->tailroom; priv->privacy_caps = desc->privacy_caps; diff --git a/drivers/pinctrl/pinctrl-amd.c b/drivers/pinctrl/pinctrl-amd.c index 9c937f6cf525..64dc6b1a1c7e 100644 --- a/drivers/pinctrl/pinctrl-amd.c +++ b/drivers/pinctrl/pinctrl-amd.c @@ -871,6 +871,15 @@ static void amd_gpio_irq_init(struct amd_gpio *gpio_dev) mask = BIT(WAKE_CNTRL_OFF_S0I3) | BIT(WAKE_CNTRL_OFF_S3); +#ifdef CONFIG_ACPI + /* + * Platforms without _AEI leave GPIO wake sources to firmware, so + * preserve their S4 wake bits. + */ + if (acpi_has_method(ACPI_HANDLE(&gpio_dev->pdev->dev), "_AEI")) + mask |= BIT(WAKE_CNTRL_OFF_S4); +#endif + for (i = 0; i < desc->npins; i++) { int pin = desc->pins[i].number; const struct pin_desc *pd = pin_desc_get(gpio_dev->pctrl, pin); diff --git a/drivers/pinctrl/qcom/pinctrl-ipq5018.c b/drivers/pinctrl/qcom/pinctrl-ipq5018.c index 68f65b57003e..2bbc409924f8 100644 --- a/drivers/pinctrl/qcom/pinctrl-ipq5018.c +++ b/drivers/pinctrl/qcom/pinctrl-ipq5018.c @@ -363,7 +363,7 @@ static const char * const blsp1_uart0_groups[] = { }; static const char * const gcc_plltest_groups[] = { - "gpio10", "gpio12", + "gpio10", }; static const char * const gcc_tlmm_groups[] = { @@ -557,7 +557,7 @@ static const char * const qdss_cti_trig_in_b0_groups[] = { }; static const char * const pwm2_groups[] = { - "gpio44", + "gpio12", "gpio44", }; static const char * const qdss_cti_trig_out_b1_groups[] = { @@ -565,7 +565,7 @@ static const char * const qdss_cti_trig_out_b1_groups[] = { }; static const char * const pwm3_groups[] = { - "gpio45", + "gpio45", "gpio13", }; static const char * const qdss_cti_trig_in_b1_groups[] = { @@ -690,8 +690,8 @@ static const struct msm_pingroup ipq5018_groups[] = { PINGROUP(9, sdc1_clk, qspi_clk, _, qdss_tracedata_a, _, _, _, _, _), PINGROUP(10, blsp0_spi, blsp1_uart0, led0, gcc_plltest, qdss_tracedata_a, _, _, _, _), PINGROUP(11, blsp0_spi, blsp1_uart0, _, gcc_tlmm, qdss_tracedata_a, _, _, _, _), - PINGROUP(12, blsp0_spi, blsp0_i2c, blsp1_uart0, _, gcc_plltest, qdss_tracedata_a, _, _, _), - PINGROUP(13, blsp0_spi, blsp0_i2c, blsp1_uart0, _, qdss_tracedata_a, _, _, _, _), + PINGROUP(12, blsp0_spi, blsp0_i2c, blsp1_uart0, _, pwm2, qdss_tracedata_a, _, _, _), + PINGROUP(13, blsp0_spi, blsp0_i2c, blsp1_uart0, pwm3, qdss_tracedata_a, _, _, _, _), PINGROUP(14, pcie0_clk, _, _, cri_trng0, qdss_tracedata_a, _, _, _, _), PINGROUP(15, _, _, cri_trng1, qdss_tracedata_a, _, _, _, _, _), PINGROUP(16, pcie0_wake, _, _, cri_trng, qdss_tracedata_a, _, _, _, _), diff --git a/drivers/power/supply/ds2780_battery.c b/drivers/power/supply/ds2780_battery.c index 1e7f297f6cb1..7219ee3def5a 100644 --- a/drivers/power/supply/ds2780_battery.c +++ b/drivers/power/supply/ds2780_battery.c @@ -304,7 +304,7 @@ static int ds2780_get_capacity(struct ds2780_device_info *dev_info, return ret; *capacity = raw; - return raw; + return 0; } static int ds2780_get_status(struct ds2780_device_info *dev_info, int *status) diff --git a/drivers/power/supply/generic-adc-battery.c b/drivers/power/supply/generic-adc-battery.c index 7bdc6b263609..aebc824aa2ed 100644 --- a/drivers/power/supply/generic-adc-battery.c +++ b/drivers/power/supply/generic-adc-battery.c @@ -90,12 +90,17 @@ static int gab_read_channel(struct gab *adc_bat, enum gab_chan_type channel, int ret; ret = iio_read_channel_processed(adc_bat->channel[channel], result); - if (ret < 0) + if (ret < 0) { dev_err(&adc_bat->psy->dev, "read channel error: %d\n", ret); + return ret; + } + + if (channel == GAB_TEMP) + *result /= 100; /* Convert from milli- to deci-degree */ else *result *= 1000; - return ret; + return 0; } static int gab_get_property(struct power_supply *psy, diff --git a/drivers/s390/net/ctcm_mpc.c b/drivers/s390/net/ctcm_mpc.c index f8b3e3f7b8ce..4df434e4b8f0 100644 --- a/drivers/s390/net/ctcm_mpc.c +++ b/drivers/s390/net/ctcm_mpc.c @@ -826,7 +826,7 @@ static void mpc_action_go_ready(fsm_instance *fsm, int event, void *arg) fsm_deltimer(&grp->timer); - if (grp->saved_xid2->xid2_flag2 == 0x40) { + if (priv->xid->xid2_flag2 == 0x40) { priv->xid->xid2_flag2 = 0x00; if (grp->estconnfunc) { grp->estconnfunc(grp->port_num, 1, @@ -1636,7 +1636,6 @@ static int mpc_validate_xid(struct mpcg_info *mpcginfo) "The XID used in the MPC protocol is not valid, " "rc = %d\n", rc); priv->xid->xid2_flag2 = 0x40; - grp->saved_xid2->xid2_flag2 = 0x40; } return rc; diff --git a/drivers/scsi/libiscsi.c b/drivers/scsi/libiscsi.c index 213aea93120d..23748976b984 100644 --- a/drivers/scsi/libiscsi.c +++ b/drivers/scsi/libiscsi.c @@ -1142,13 +1142,15 @@ static int iscsi_handle_reject(struct iscsi_conn *conn, struct iscsi_hdr *hdr, spin_unlock(&conn->session->frwd_lock); spin_lock(&conn->session->back_lock); } else { - struct iscsi_task *task; + struct iscsi_task *task = NULL; /* * Our nop as ping got dropped. We know the target * and transport are ok so just clean up */ - task = iscsi_itt_to_task(conn, rejected_pdu.itt); - if (!task) { + if (!iscsi_verify_itt(conn, rejected_pdu.itt)) + task = iscsi_itt_to_task(conn, rejected_pdu.itt); + if (!task || task->state == ISCSI_TASK_FREE || + task->sc) { iscsi_conn_printk(KERN_ERR, conn, "Invalid pdu reject. Could " "not lookup rejected task.\n"); @@ -1191,7 +1193,7 @@ struct iscsi_task *iscsi_itt_to_task(struct iscsi_conn *conn, itt_t itt) session->tt->parse_pdu_itt(conn, itt, &i, NULL); else i = get_itt(itt); - if (i >= session->cmds_max) + if (i < 0 || i >= session->cmds_max) return NULL; return session->cmds[i]; @@ -1284,7 +1286,7 @@ int __iscsi_complete_pdu(struct iscsi_conn *conn, struct iscsi_hdr *hdr, case ISCSI_OP_SCSI_TMFUNC_RSP: case ISCSI_OP_NOOP_IN: task = iscsi_itt_to_task(conn, hdr->itt); - if (!task) + if (!task || task->state == ISCSI_TASK_FREE || task->sc) return ISCSI_ERR_BAD_ITT; break; default: @@ -1384,7 +1386,7 @@ int iscsi_verify_itt(struct iscsi_conn *conn, itt_t itt) return ISCSI_ERR_BAD_ITT; } - if (i >= session->cmds_max) { + if (i < 0 || i >= session->cmds_max) { iscsi_conn_printk(KERN_ERR, conn, "received invalid itt index %u (max cmds " "%u.\n", i, session->cmds_max); diff --git a/drivers/scsi/libsas/sas_expander.c b/drivers/scsi/libsas/sas_expander.c index e97f4e01a865..16cc1c14b9f4 100644 --- a/drivers/scsi/libsas/sas_expander.c +++ b/drivers/scsi/libsas/sas_expander.c @@ -40,7 +40,11 @@ static int smp_execute_task_sg(struct domain_device *dev, to_sas_internal(dev->port->ha->shost->transportt); struct sas_ha_struct *ha = dev->port->ha; - pm_runtime_get_sync(ha->dev); + /* + * Non-blocking: a sync resume here would deadlock against + * sas_drain_work() during HA resume. + */ + pm_runtime_get_noresume(ha->dev); mutex_lock(&dev->ex_dev.cmd_mutex); for (retry = 0; retry < 3; retry++) { if (test_bit(SAS_DEV_GONE, &dev->state)) { @@ -113,7 +117,7 @@ static int smp_execute_task_sg(struct domain_device *dev, } } mutex_unlock(&dev->ex_dev.cmd_mutex); - pm_runtime_put_sync(ha->dev); + pm_runtime_put(ha->dev); BUG_ON(retry == 3 && task != NULL); sas_free_task(task); @@ -2142,6 +2146,11 @@ void sas_smp_handler(struct bsg_job *job, struct Scsi_Host *shost, goto out; } + /* The host may have autosuspended, resume it here. */ + ret = pm_runtime_resume_and_get(dev->port->ha->dev); + if (ret) + goto out; + ret = smp_execute_task_sg(dev, job->request_payload.sg_list, job->reply_payload.sg_list); if (ret >= 0) { @@ -2149,6 +2158,7 @@ void sas_smp_handler(struct bsg_job *job, struct Scsi_Host *shost, rcvlen = job->reply_payload.payload_len - ret; ret = 0; } + pm_runtime_put(dev->port->ha->dev); out: bsg_job_done(job, ret, rcvlen); diff --git a/drivers/scsi/mpi3mr/mpi3mr_app.c b/drivers/scsi/mpi3mr/mpi3mr_app.c index 27fcffa94713..fdec01d4bcee 100644 --- a/drivers/scsi/mpi3mr/mpi3mr_app.c +++ b/drivers/scsi/mpi3mr/mpi3mr_app.c @@ -1551,7 +1551,7 @@ void mpi3mr_bsg_init(struct mpi3mr_ioc *mrioc) } q = bsg_setup_queue(bsg_dev, dev_name(bsg_dev), - mpi3mr_bsg_request, NULL, 0); + mpi3mr_bsg_request, NULL, 0, SCSI_SENSE_BUFFERSIZE); if (IS_ERR(q)) { ioc_err(mrioc, "%s: bsg registration failed\n", dev_name(bsg_dev)); diff --git a/drivers/scsi/scsi_transport_fc.c b/drivers/scsi/scsi_transport_fc.c index fd827e30556b..8d53a6717da8 100644 --- a/drivers/scsi/scsi_transport_fc.c +++ b/drivers/scsi/scsi_transport_fc.c @@ -4293,7 +4293,7 @@ fc_bsg_hostadd(struct Scsi_Host *shost, struct fc_host_attrs *fc_host) "fc_host%d", shost->host_no); q = bsg_setup_queue(dev, bsg_name, fc_bsg_dispatch, fc_bsg_job_timeout, - i->f->dd_bsg_size); + i->f->dd_bsg_size, SCSI_SENSE_BUFFERSIZE); if (IS_ERR(q)) { dev_err(dev, "fc_host%d: bsg interface failed to initialize - setup queue\n", @@ -4324,7 +4324,8 @@ fc_bsg_rportadd(struct Scsi_Host *shost, struct fc_rport *rport) return -ENOTSUPP; q = bsg_setup_queue(dev, dev_name(dev), fc_bsg_dispatch_prep, - fc_bsg_job_timeout, i->f->dd_bsg_size); + fc_bsg_job_timeout, i->f->dd_bsg_size, + SCSI_SENSE_BUFFERSIZE); if (IS_ERR(q)) { dev_err(dev, "failed to setup bsg queue\n"); return PTR_ERR(q); diff --git a/drivers/scsi/scsi_transport_iscsi.c b/drivers/scsi/scsi_transport_iscsi.c index 22bec2453bac..7cc99d213ce2 100644 --- a/drivers/scsi/scsi_transport_iscsi.c +++ b/drivers/scsi/scsi_transport_iscsi.c @@ -1542,7 +1542,8 @@ iscsi_bsg_host_add(struct Scsi_Host *shost, struct iscsi_cls_host *ihost) return -ENOTSUPP; snprintf(bsg_name, sizeof(bsg_name), "iscsi_host%d", shost->host_no); - q = bsg_setup_queue(dev, bsg_name, iscsi_bsg_host_dispatch, NULL, 0); + q = bsg_setup_queue(dev, bsg_name, iscsi_bsg_host_dispatch, NULL, 0, + SCSI_SENSE_BUFFERSIZE); if (IS_ERR(q)) { shost_printk(KERN_ERR, shost, "bsg interface failed to " "initialize - no request queue\n"); diff --git a/drivers/scsi/scsi_transport_sas.c b/drivers/scsi/scsi_transport_sas.c index ce69240accb2..edb9b853c938 100644 --- a/drivers/scsi/scsi_transport_sas.c +++ b/drivers/scsi/scsi_transport_sas.c @@ -199,7 +199,8 @@ static int sas_bsg_initialize(struct Scsi_Host *shost, struct sas_rphy *rphy) if (rphy) { q = bsg_setup_queue(&rphy->dev, dev_name(&rphy->dev), - sas_smp_dispatch, NULL, 0); + sas_smp_dispatch, NULL, 0, + SCSI_SENSE_BUFFERSIZE); if (IS_ERR(q)) return PTR_ERR(q); rphy->q = q; @@ -208,7 +209,8 @@ static int sas_bsg_initialize(struct Scsi_Host *shost, struct sas_rphy *rphy) snprintf(name, sizeof(name), "sas_host%d", shost->host_no); q = bsg_setup_queue(&shost->shost_gendev, name, - sas_smp_dispatch, NULL, 0); + sas_smp_dispatch, NULL, 0, + SCSI_SENSE_BUFFERSIZE); if (IS_ERR(q)) return PTR_ERR(q); to_sas_host_attrs(shost)->q = q; diff --git a/drivers/scsi/st.c b/drivers/scsi/st.c index f9ab45c4bb40..e302e366fa27 100644 --- a/drivers/scsi/st.c +++ b/drivers/scsi/st.c @@ -3195,14 +3195,20 @@ static int set_location(struct scsi_tape *STp, unsigned int block, int partition partition >= ST_NBR_PARTITIONS) return (-EINVAL); if (partition != STp->partition) { - if (get_location(STp, &blk, &p, 1)) + /* + * Save the location for the partition the drive reports, + * not for STp->partition: after a reset STp->partition may + * not match the partition the drive is in. + */ + if (get_location(STp, &blk, &p, 1) || + p < 0 || p >= ST_NBR_PARTITIONS) STps->last_block_valid = 0; else { - STps->last_block_valid = 1; - STps->last_block_visited = blk; + STp->ps[p].last_block_valid = 1; + STp->ps[p].last_block_visited = blk; DEBC_printk(STp, "Visited block %d for " "partition %d saved.\n", - blk, STp->partition); + blk, p); } } @@ -4443,11 +4449,11 @@ static int st_remove(struct device *dev) remove_cdevs(tpnt); mutex_lock(&st_ref_mutex); - kref_put(&tpnt->kref, scsi_tape_release); - mutex_unlock(&st_ref_mutex); spin_lock(&st_index_lock); idr_remove(&st_index_idr, index); spin_unlock(&st_index_lock); + kref_put(&tpnt->kref, scsi_tape_release); + mutex_unlock(&st_ref_mutex); return 0; } diff --git a/drivers/scsi/xen-scsifront.c b/drivers/scsi/xen-scsifront.c index 9ec55ddc1204..659e3e82da25 100644 --- a/drivers/scsi/xen-scsifront.c +++ b/drivers/scsi/xen-scsifront.c @@ -367,6 +367,7 @@ static void scsifront_do_response(struct vscsifrnt_info *info, struct vscsifrnt_shadow *shadow; if (ring_rsp->rqid >= VSCSIIF_MAX_REQS || + !info->shadow[ring_rsp->rqid] || !info->shadow[ring_rsp->rqid]->inflight) { scsifront_set_error(info, "illegal rqid returned by backend!"); return; diff --git a/drivers/spi/spi-tegra114.c b/drivers/spi/spi-tegra114.c index ba2c9feab1cd..2b7fc3e0c244 100644 --- a/drivers/spi/spi-tegra114.c +++ b/drivers/spi/spi-tegra114.c @@ -859,6 +859,13 @@ static u32 tegra_spi_setup_transfer_one(struct spi_device *spi, command1 |= SPI_BIT_LENGTH(bits_per_word - 1); } + /* + * Re-assert the CS GPIO for every transfer: transfer_end() drops it + * on cs_change, and the assert above only covers is_first_of_msg. + */ + if (spi_get_csgpiod(spi, 0)) + gpiod_set_value(spi_get_csgpiod(spi, 0), 1); + return command1; } diff --git a/drivers/target/iscsi/iscsi_target_configfs.c b/drivers/target/iscsi/iscsi_target_configfs.c index 1cff6052e820..16d91195c08f 100644 --- a/drivers/target/iscsi/iscsi_target_configfs.c +++ b/drivers/target/iscsi/iscsi_target_configfs.c @@ -1306,17 +1306,24 @@ static ssize_t iscsi_disc_enforce_discovery_auth_store(struct config_item *item, return -EINVAL; } + if (iscsit_get_tpg(discovery_tpg) < 0) + return -EINVAL; + param = iscsi_find_param_from_key(AUTHMETHOD, discovery_tpg->param_list); - if (!param) - return -EINVAL; + if (!param) { + err = -EINVAL; + goto out; + } if (op) { /* * Reset the AuthMethod key to CHAP. */ - if (iscsi_update_param_value(param, CHAP) < 0) - return -EINVAL; + if (iscsi_update_param_value(param, CHAP) < 0) { + err = -EINVAL; + goto out; + } discovery_tpg->tpg_attrib.authentication = 1; iscsit_global->discovery_acl.node_auth.enforce_discovery_auth = 1; @@ -1327,8 +1334,10 @@ static ssize_t iscsi_disc_enforce_discovery_auth_store(struct config_item *item, /* * Reset the AuthMethod key to CHAP,None */ - if (iscsi_update_param_value(param, "CHAP,None") < 0) - return -EINVAL; + if (iscsi_update_param_value(param, "CHAP,None") < 0) { + err = -EINVAL; + goto out; + } discovery_tpg->tpg_attrib.authentication = 0; iscsit_global->discovery_acl.node_auth.enforce_discovery_auth = 0; @@ -1337,7 +1346,10 @@ static ssize_t iscsi_disc_enforce_discovery_auth_store(struct config_item *item, " Discovery TPG\n"); } - return count; + err = count; +out: + iscsit_put_tpg(discovery_tpg); + return err; } CONFIGFS_ATTR(iscsi_disc_, enforce_discovery_auth); diff --git a/drivers/ufs/core/ufs_bsg.c b/drivers/ufs/core/ufs_bsg.c index 0043d69077b6..df31f89b9666 100644 --- a/drivers/ufs/core/ufs_bsg.c +++ b/drivers/ufs/core/ufs_bsg.c @@ -256,7 +256,8 @@ int ufs_bsg_probe(struct ufs_hba *hba) if (ret) goto out; - q = bsg_setup_queue(bsg_dev, dev_name(bsg_dev), ufs_bsg_request, NULL, 0); + q = bsg_setup_queue(bsg_dev, dev_name(bsg_dev), ufs_bsg_request, NULL, 0, + sizeof(struct ufs_rpmb_reply)); if (IS_ERR(q)) { ret = PTR_ERR(q); device_del(bsg_dev); diff --git a/fs/autofs/waitq.c b/fs/autofs/waitq.c index 33dd4660d82f..cab26ca65873 100644 --- a/fs/autofs/waitq.c +++ b/fs/autofs/waitq.c @@ -12,15 +12,13 @@ */ static autofs_wqt_t autofs_next_wait_queue = 1; -void autofs_catatonic_mode(struct autofs_sb_info *sbi) +static void autofs_catatonic_mode_locked(struct autofs_sb_info *sbi) { struct autofs_wait_queue *wq, *nwq; - mutex_lock(&sbi->wq_mutex); - if (sbi->flags & AUTOFS_SBI_CATATONIC) { - mutex_unlock(&sbi->wq_mutex); + lockdep_assert_held(&sbi->wq_mutex); + if (sbi->flags & AUTOFS_SBI_CATATONIC) return; - } pr_debug("entering catatonic mode\n"); @@ -40,6 +38,21 @@ void autofs_catatonic_mode(struct autofs_sb_info *sbi) fput(sbi->pipe); /* Close the pipe */ sbi->pipe = NULL; sbi->pipefd = -1; +} + +void autofs_catatonic_mode(struct autofs_sb_info *sbi) +{ + mutex_lock(&sbi->wq_mutex); + autofs_catatonic_mode_locked(sbi); + mutex_unlock(&sbi->wq_mutex); +} + +static void autofs_catatonic_mode_for_pipe(struct autofs_sb_info *sbi, + struct file *pipe) +{ + mutex_lock(&sbi->wq_mutex); + if (sbi->pipe == pipe) + autofs_catatonic_mode_locked(sbi); mutex_unlock(&sbi->wq_mutex); } @@ -170,7 +183,7 @@ static void autofs_notify_daemon(struct autofs_sb_info *sbi, autofs_wait_release(sbi, wq->wait_queue_token, ret); break; default: - autofs_catatonic_mode(sbi); + autofs_catatonic_mode_for_pipe(sbi, pipe); break; } fput(pipe); diff --git a/fs/coredump.c b/fs/coredump.c index 2f1f66f42fab..326b30421eec 100644 --- a/fs/coredump.c +++ b/fs/coredump.c @@ -416,7 +416,9 @@ static int zap_threads(struct task_struct *tsk, int nr = -EAGAIN; spin_lock_irq(&tsk->sighand->siglock); - if (!(signal->flags & SIGNAL_GROUP_EXIT) && !signal->group_exec_task) { + /* A freeze requested before the dump would be lost with TIF_SIGPENDING. */ + if (!(signal->flags & SIGNAL_GROUP_EXIT) && !signal->group_exec_task && + !freezing(tsk) && !(tsk->jobctl & JOBCTL_TRAP_FREEZE)) { signal->core_state = core_state; nr = zap_process(tsk, exit_code); clear_tsk_thread_flag(tsk, TIF_SIGPENDING); @@ -469,6 +471,8 @@ static void coredump_finish(bool core_dumped) current->signal->core_state = NULL; spin_unlock_irq(¤t->sighand->siglock); + /* A released thread may exit and be freed before it is woken. */ + guard(rcu)(); while ((curr = next) != NULL) { next = curr->next; task = curr->task; @@ -477,6 +481,7 @@ static void coredump_finish(bool core_dumped) * ->task == NULL before we read ->next. */ smp_mb(); + /* Any wakeup now lets the thread exit, rcu keeps it alive. */ curr->task = NULL; wake_up_process(task); } @@ -484,13 +489,8 @@ static void coredump_finish(bool core_dumped) static bool dump_interrupted(void) { - /* - * SIGKILL or freezing() interrupt the coredumping. Perhaps we - * can do try_to_freeze() and check __fatal_signal_pending(), - * but then we need to teach dump_write() to restart and clear - * TIF_SIGPENDING. - */ - return fatal_signal_pending(current) || freezing(current); + /* Only SIGKILL and the freezers set it after zap_threads(). */ + return task_sigpending(current); } static void wait_for_dump_helpers(struct file *file) diff --git a/fs/ext4/ext4.h b/fs/ext4/ext4.h index 5654a1cd2b4a..58746c5130ac 100644 --- a/fs/ext4/ext4.h +++ b/fs/ext4/ext4.h @@ -1669,6 +1669,8 @@ struct ext4_sb_info { struct list_head s_es_list; /* List of inodes with reclaimable extents */ long s_es_nr_inode; struct ext4_es_stats s_es_stats; + /* ext4_get_link() reads the sbi in rcu pathwalk */ + struct rcu_head s_rcu; struct mb_cache *s_ea_block_cache; struct mb_cache *s_ea_inode_cache; spinlock_t s_es_lock ____cacheline_aligned_in_smp; diff --git a/fs/ext4/extents_status.c b/fs/ext4/extents_status.c index d9d5cfb9c951..cd5160e72d5a 100644 --- a/fs/ext4/extents_status.c +++ b/fs/ext4/extents_status.c @@ -1749,12 +1749,17 @@ int ext4_es_register_shrinker(struct ext4_sb_info *sbi) } void ext4_es_unregister_shrinker(struct ext4_sb_info *sbi) +{ + unregister_shrinker(&sbi->s_es_shrinker); +} + +/* ext4_get_link() bumps the hit and miss counters in rcu pathwalk */ +void ext4_es_destroy_stats(struct ext4_sb_info *sbi) { percpu_counter_destroy(&sbi->s_es_stats.es_stats_cache_hits); percpu_counter_destroy(&sbi->s_es_stats.es_stats_cache_misses); percpu_counter_destroy(&sbi->s_es_stats.es_stats_all_cnt); percpu_counter_destroy(&sbi->s_es_stats.es_stats_shk_cnt); - unregister_shrinker(&sbi->s_es_shrinker); } /* diff --git a/fs/ext4/extents_status.h b/fs/ext4/extents_status.h index d9847a4a25db..b2c6579b2c9d 100644 --- a/fs/ext4/extents_status.h +++ b/fs/ext4/extents_status.h @@ -241,6 +241,7 @@ static inline void ext4_es_store_pblock_status(struct extent_status *es, extern int ext4_es_register_shrinker(struct ext4_sb_info *sbi); extern void ext4_es_unregister_shrinker(struct ext4_sb_info *sbi); +extern void ext4_es_destroy_stats(struct ext4_sb_info *sbi); extern int ext4_seq_es_shrinker_info_show(struct seq_file *seq, void *v); diff --git a/fs/ext4/mballoc.c b/fs/ext4/mballoc.c index 4ec4388932ea..646f55ba0dd1 100644 --- a/fs/ext4/mballoc.c +++ b/fs/ext4/mballoc.c @@ -2954,14 +2954,19 @@ ext4_mb_regular_allocator(struct ext4_allocation_context *ac) ac->ac_groups_scanned++; if (cr == CR_POWER2_ALIGNED) ext4_mb_simple_scan_group(ac, &e4b); - else if ((cr == CR_GOAL_LEN_FAST || - cr == CR_BEST_AVAIL_LEN) && - sbi->s_stripe && - !(ac->ac_g_ex.fe_len % - EXT4_B2C(sbi, sbi->s_stripe))) - ext4_mb_scan_aligned(ac, &e4b); - else - ext4_mb_complex_scan_group(ac, &e4b); + else { + bool is_stripe_aligned = sbi->s_stripe && + !(ac->ac_g_ex.fe_len % + EXT4_B2C(sbi, sbi->s_stripe)); + + if ((cr == CR_GOAL_LEN_FAST || + cr == CR_BEST_AVAIL_LEN) && + is_stripe_aligned) + ext4_mb_scan_aligned(ac, &e4b); + + if (ac->ac_status == AC_STATUS_CONTINUE) + ext4_mb_complex_scan_group(ac, &e4b); + } ext4_unlock_group(sb, group); ext4_mb_unload_buddy(&e4b); diff --git a/fs/ext4/super.c b/fs/ext4/super.c index 349e2b087769..94d40e03ff2f 100644 --- a/fs/ext4/super.c +++ b/fs/ext4/super.c @@ -1289,6 +1289,14 @@ static void ext4_flex_groups_free(struct ext4_sb_info *sbi) } } +static void ext4_put_sbi_rcu(struct rcu_head *head) +{ + struct ext4_sb_info *sbi = container_of(head, struct ext4_sb_info, s_rcu); + + ext4_es_destroy_stats(sbi); + kfree(sbi); +} + static void ext4_put_super(struct super_block *sb) { struct ext4_sb_info *sbi = EXT4_SB(sb); @@ -1378,7 +1386,6 @@ static void ext4_put_super(struct super_block *sb) ext4_stop_mmpd(sbi); brelse(sbi->s_sbh); - sb->s_fs_info = NULL; /* * Now that we are completely done shutting down the * superblock, we need to actually destroy the kobject. @@ -1393,7 +1400,8 @@ static void ext4_put_super(struct super_block *sb) #if IS_ENABLED(CONFIG_UNICODE) utf8_unload(sb->s_encoding); #endif - kfree(sbi); + /* ext4_get_link() reads the sbi and the counters in rcu pathwalk */ + call_rcu(&sbi->s_rcu, ext4_put_sbi_rcu); } static struct kmem_cache *ext4_inode_cachep; @@ -5688,6 +5696,7 @@ failed_mount8: __maybe_unused } failed_mount3a: ext4_es_unregister_shrinker(sbi); + ext4_es_destroy_stats(sbi); failed_mount3: /* flush s_sb_upd_work before sbi destroy */ flush_work(&sbi->s_sb_upd_work); diff --git a/fs/notify/mark.c b/fs/notify/mark.c index 951d7de29f82..465724aafd49 100644 --- a/fs/notify/mark.c +++ b/fs/notify/mark.c @@ -719,6 +719,11 @@ static int fsnotify_add_mark_list(struct fsnotify_mark *mark, goto out_err; } + if (conn->flags & FSNOTIFY_CONN_FLAG_DESTROYING) { + err = -ENOENT; + goto out_err; + } + /* is mark the first mark? */ if (hlist_empty(&conn->list)) { hlist_add_head_rcu(&mark->obj_list, &conn->list); @@ -901,6 +906,12 @@ void fsnotify_destroy_marks(fsnotify_connp_t *connp) conn = fsnotify_grab_connector(connp); if (!conn) return; + /* + * New marks could be inserted before our cursor and survive the final + * detachment of the connector from the object. Prevent additions to + * this connector before dropping its lock. + */ + conn->flags |= FSNOTIFY_CONN_FLAG_DESTROYING; /* * We have to be careful since we can race with e.g. * fsnotify_clear_marks_by_group() and once we drop the conn->lock, the diff --git a/fs/ntfs3/ntfs_fs.h b/fs/ntfs3/ntfs_fs.h index 5e7536ad0b94..8850ee2b75b1 100644 --- a/fs/ntfs3/ntfs_fs.h +++ b/fs/ntfs3/ntfs_fs.h @@ -209,6 +209,7 @@ struct ntfs_index { /* Ntfs file system in-core superblock data. */ struct ntfs_sb_info { struct super_block *sb; + struct rcu_head rcu; u32 discard_granularity; u64 discard_granularity_mask_inv; // ~(discard_granularity_mask_inv-1) diff --git a/fs/ntfs3/super.c b/fs/ntfs3/super.c index 15b1f2027b4e..3b7731b18830 100644 --- a/fs/ntfs3/super.c +++ b/fs/ntfs3/super.c @@ -183,7 +183,7 @@ void *ntfs_set_shared(void *ptr, u32 bytes) void *ret = NULL; int i, j = -1; - spin_lock(&s_shared_lock); + spin_lock_bh(&s_shared_lock); for (i = 0; i < ARRAY_SIZE(s_shared); i++) { if (!s_shared[i].cnt) { j = i; @@ -201,7 +201,7 @@ void *ntfs_set_shared(void *ptr, u32 bytes) s_shared[j].cnt = 1; ret = ptr; } - spin_unlock(&s_shared_lock); + spin_unlock_bh(&s_shared_lock); return ret; } @@ -218,7 +218,7 @@ void *ntfs_put_shared(void *ptr) void *ret = ptr; int i; - spin_lock(&s_shared_lock); + spin_lock_bh(&s_shared_lock); for (i = 0; i < ARRAY_SIZE(s_shared); i++) { if (s_shared[i].cnt && s_shared[i].ptr == ptr) { if (--s_shared[i].cnt) @@ -226,7 +226,7 @@ void *ntfs_put_shared(void *ptr) break; } } - spin_unlock(&s_shared_lock); + spin_unlock_bh(&s_shared_lock); return ret; } @@ -624,6 +624,8 @@ static noinline void ntfs3_put_sbi(struct ntfs_sb_info *sbi) static void ntfs3_free_sbi(struct ntfs_sb_info *sbi) { + if (sbi->options) + put_mount_options(sbi->options); kfree(sbi->new_rec); kvfree(ntfs_put_shared(sbi->upcase)); kvfree(sbi->def_table); @@ -635,6 +637,11 @@ static void ntfs3_free_sbi(struct ntfs_sb_info *sbi) kfree(sbi); } +static void ntfs3_free_sbi_rcu(struct rcu_head *head) +{ + ntfs3_free_sbi(container_of(head, struct ntfs_sb_info, rcu)); +} + static void ntfs_put_super(struct super_block *sb) { struct ntfs_sb_info *sbi = sb->s_fs_info; @@ -651,6 +658,12 @@ static void ntfs_put_super(struct super_block *sb) /* Mark rw ntfs as clear, if possible. */ ntfs_set_state(sbi, NTFS_DIRTY_CLEAR); + + /* + * The mount options stay until the sbi is freed: ->d_hash() and + * ->d_compare() of a pathwalk in rcu mode read the nls table through + * them. + */ ntfs3_put_sbi(sbi); } @@ -1807,9 +1820,11 @@ static void ntfs3_kill_sb(struct super_block *sb) kill_block_super(sb); - if (sbi->options) - put_mount_options(sbi->options); - ntfs3_free_sbi(sbi); + /* + * A pathwalk in rcu mode may still be in ->d_hash() or ->d_compare() + * and read the upcase table and the nls table through the sbi. + */ + call_rcu(&sbi->rcu, ntfs3_free_sbi_rcu); } // clang-format off diff --git a/fs/ocfs2/dlm/dlmcommon.h b/fs/ocfs2/dlm/dlmcommon.h index 20f790a47484..58b28dfa8d34 100644 --- a/fs/ocfs2/dlm/dlmcommon.h +++ b/fs/ocfs2/dlm/dlmcommon.h @@ -435,6 +435,8 @@ enum { DLM_DEREF_LOCKRES_DONE = 522, }; +extern spinlock_t dlm_reco_state_lock; + struct dlm_reco_node_data { int state; diff --git a/fs/ocfs2/dlm/dlmdebug.c b/fs/ocfs2/dlm/dlmdebug.c index e62ed98df7de..afaea6396b6f 100644 --- a/fs/ocfs2/dlm/dlmdebug.c +++ b/fs/ocfs2/dlm/dlmdebug.c @@ -792,8 +792,11 @@ static int debug_state_print(struct dlm_ctxt *dlm, char *buf, int len) buf + out, len - out); out += scnprintf(buf + out, len - out, "\n"); + spin_unlock(&dlm->spinlock); + /* Recovery Node State: */ out += scnprintf(buf + out, len - out, "Recovery Node State:\n"); + spin_lock(&dlm_reco_state_lock); list_for_each_entry(node, &dlm->reco.node_data, list) { switch (node->state) { case DLM_RECO_NODE_DATA_INIT: @@ -825,7 +828,7 @@ static int debug_state_print(struct dlm_ctxt *dlm, char *buf, int len) node->node_num, state); } - spin_unlock(&dlm->spinlock); + spin_unlock(&dlm_reco_state_lock); return out; } diff --git a/fs/ocfs2/dlm/dlmrecovery.c b/fs/ocfs2/dlm/dlmrecovery.c index 739da8bd7dbf..591dd17ef4f5 100644 --- a/fs/ocfs2/dlm/dlmrecovery.c +++ b/fs/ocfs2/dlm/dlmrecovery.c @@ -80,7 +80,7 @@ static int dlm_lockres_master_requery(struct dlm_ctxt *dlm, static u64 dlm_get_next_mig_cookie(void); -static DEFINE_SPINLOCK(dlm_reco_state_lock); +DEFINE_SPINLOCK(dlm_reco_state_lock); static DEFINE_SPINLOCK(dlm_mig_cookie_lock); static u64 dlm_mig_cookie = 1; diff --git a/fs/pipe.c b/fs/pipe.c index ba4376341ddd..2ad7eba72b35 100644 --- a/fs/pipe.c +++ b/fs/pipe.c @@ -751,9 +751,11 @@ pipe_fasync(int fd, struct file *filp, int on) if (filp->f_mode & FMODE_READ) retval = fasync_helper(fd, filp, on, &pipe->fasync_readers); if ((filp->f_mode & FMODE_WRITE) && retval >= 0) { + int read_ret = retval; + retval = fasync_helper(fd, filp, on, &pipe->fasync_writers); - if (retval < 0 && (filp->f_mode & FMODE_READ)) - /* this can happen only if on == T */ + /* Undo only a reader entry added by this call. */ + if (retval < 0 && read_ret > 0) fasync_helper(-1, filp, 0, &pipe->fasync_readers); } __pipe_unlock(pipe); diff --git a/fs/smb/client/cifssmb.c b/fs/smb/client/cifssmb.c index 3ba39c375077..bc5de4d3d346 100644 --- a/fs/smb/client/cifssmb.c +++ b/fs/smb/client/cifssmb.c @@ -3152,6 +3152,7 @@ int cifs_do_set_acl(const unsigned int xid, struct cifs_tcon *tcon, int rc = 0; int bytes_returned = 0; __u16 params, byte_count, data_count, param_offset, offset; + size_t cifs_acl_size, bytes_available; cifs_dbg(FYI, "In SetPosixACL (Unix) for path %s\n", fileName); setAclRetry: @@ -3170,8 +3171,7 @@ int cifs_do_set_acl(const unsigned int xid, struct cifs_tcon *tcon, } params = 6 + name_len; pSMB->MaxParameterCount = cpu_to_le16(2); - /* BB find max SMB size from sess */ - pSMB->MaxDataCount = cpu_to_le16(1000); + pSMB->MaxDataCount = cpu_to_le16(min_t(unsigned int, CIFSMaxBufSize, USHRT_MAX)); pSMB->MaxSetupCount = 0; pSMB->Reserved = 0; pSMB->Flags = 0; @@ -3183,6 +3183,15 @@ int cifs_do_set_acl(const unsigned int xid, struct cifs_tcon *tcon, parm_data = ((char *)pSMB) + sizeof(pSMB->hdr.smb_buf_length) + offset; pSMB->ParameterOffset = cpu_to_le16(param_offset); + /* make sure we can fit the larger cifs_posix_aces in the buffer */ + cifs_acl_size = sizeof(struct cifs_posix_acl) + + (acl->a_count * sizeof(struct cifs_posix_ace)); + bytes_available = (CIFSMaxBufSize + MAX_HEADER_SIZE(tcon->ses->server)) - offset; + if (cifs_acl_size > bytes_available || cifs_acl_size > USHRT_MAX) { + rc = -E2BIG; + goto setACLerrorExit; + } + /* convert to on the wire format for POSIX ACL */ data_count = posix_acl_to_cifs(parm_data, acl, acl_type); diff --git a/fs/smb/client/connect.c b/fs/smb/client/connect.c index cbe8cb863c1c..1c21ae3d61e2 100644 --- a/fs/smb/client/connect.c +++ b/fs/smb/client/connect.c @@ -4147,14 +4147,25 @@ cifs_setup_session(const unsigned int xid, struct cifs_ses *ses, return rc; } -static int -cifs_set_vol_auth(struct smb3_fs_context *ctx, struct cifs_ses *ses) +static int set_fs_context_auth(struct smb3_fs_context *ctx, + struct cifs_ses *ses) { ctx->sectype = ses->sectype; - /* krb5 is special, since we don't need username or pw */ - if (ctx->sectype == Kerberos) + /* + * krb5 is special as we might need to pass username (passwordless) down + * to cifs.upcall(8) for keytab. + */ + if (ctx->sectype == Kerberos) { + if (ses->user_name && ses->user_name[0]) { + ctx->username = kstrndup(ses->user_name, + CIFS_MAX_USERNAME_LEN, + GFP_KERNEL); + if (!ctx->username) + return -ENOMEM; + } return 0; + } return cifs_set_cifscreds(ctx, ses); } @@ -4193,7 +4204,7 @@ cifs_construct_tcon(struct cifs_sb_info *cifs_sb, kuid_t fsuid) ctx->witness = master_tcon->use_witness; ctx->dfs_root_ses = master_tcon->ses->dfs_root_ses; - rc = cifs_set_vol_auth(ctx, master_tcon->ses); + rc = set_fs_context_auth(ctx, master_tcon->ses); if (rc) { tcon = ERR_PTR(rc); goto out; diff --git a/fs/smb/client/smb2ops.c b/fs/smb/client/smb2ops.c index 4da027ee4b99..21a2b265ae7c 100644 --- a/fs/smb/client/smb2ops.c +++ b/fs/smb/client/smb2ops.c @@ -2070,7 +2070,7 @@ smb2_duplicate_extents(const unsigned int xid, src_off, dest_off, len); inode = d_inode(trgtfile->dentry); - if (inode->i_size < dest_off + len) { + if (i_size_read(inode) < dest_off + len) { rc = smb2_set_file_size(xid, tcon, trgtfile, dest_off + len, false); if (rc) goto duplicate_extents_out; @@ -2095,6 +2095,11 @@ smb2_duplicate_extents(const unsigned int xid, if (ret_data_len > 0) cifs_dbg(FYI, "Non-zero response length in duplicate extents\n"); + if (rc) { + CIFS_I(inode)->time = 0; /* force reval */ + cifs_invalidate_cache(inode, 0); + } + duplicate_extents_out: return rc; } diff --git a/fs/smb/server/smb2pdu.c b/fs/smb/server/smb2pdu.c index ea552b1d4abb..6910b9d9346f 100644 --- a/fs/smb/server/smb2pdu.c +++ b/fs/smb/server/smb2pdu.c @@ -5685,6 +5685,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, struct object_id_info *info; info = (struct object_id_info *)(rsp->Buffer); + memset(info, 0, sizeof(*info)); if (path.mnt->mnt_sb->s_uuid_len == 16) memcpy(info->objid, path.mnt->mnt_sb->s_uuid.b, @@ -5739,6 +5740,7 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->FreeSpaceStopFiltering = 0; info->DefaultQuotaThreshold = cpu_to_le64(SMB2_NO_FID); info->DefaultQuotaLimit = cpu_to_le64(SMB2_NO_FID); + info->FileSystemControlFlags = 0; info->Padding = 0; rsp->OutputBufferLength = cpu_to_le32(48); fixed_len = 48; @@ -5761,6 +5763,9 @@ static int smb2_get_info_filesystem(struct ksmbd_work *work, info->UserBlocksAvail = cpu_to_le64(stfs.f_bavail); info->TotalFileNodes = cpu_to_le64(stfs.f_files); info->FreeFileNodes = cpu_to_le64(stfs.f_ffree); + info->FileSysIdentifier = + cpu_to_le64((u64)(u32)stfs.f_fsid.val[1] << 32 | + (u32)stfs.f_fsid.val[0]); rsp->OutputBufferLength = cpu_to_le32(56); fixed_len = 56; } diff --git a/fs/tracefs/event_inode.c b/fs/tracefs/event_inode.c index df1d571144d6..ca55828e7446 100644 --- a/fs/tracefs/event_inode.c +++ b/fs/tracefs/event_inode.c @@ -92,6 +92,14 @@ static void free_ei_rcu(struct rcu_head *rcu) } } +/* read under eventfs_srcu and, by tracefs_d_revalidate(), in rcu pathwalk */ +static void free_ei_srcu(struct rcu_head *rcu) +{ + struct eventfs_inode *ei = container_of(rcu, struct eventfs_inode, rcu); + + call_rcu(&ei->rcu, free_ei_rcu); +} + /* * eventfs_inode reference count management. * @@ -113,7 +121,7 @@ static void release_ei(struct kref *ref) entry->release(entry->name, ei->data); } - call_srcu(&eventfs_srcu, &ei->rcu, free_ei_rcu); + call_srcu(&eventfs_srcu, &ei->rcu, free_ei_srcu); } static inline void put_ei(struct eventfs_inode *ei) @@ -740,8 +748,10 @@ static int eventfs_iterate(struct file *file, struct dir_context *ctx) * Matches the smp_wmb() in free_ei() */ smp_rmb(); - if (ei->is_freed) - return -EINVAL; + if (ei->is_freed) { + ret = -EINVAL; + goto out; + } if (c > 0) { c--; diff --git a/fs/unicode/utf8-core.c b/fs/unicode/utf8-core.c index 0400824ef493..a82673af2e66 100644 --- a/fs/unicode/utf8-core.c +++ b/fs/unicode/utf8-core.c @@ -3,6 +3,7 @@ #include #include #include +#include #include #include #include @@ -205,12 +206,24 @@ struct unicode_map *utf8_load(unsigned int version) } EXPORT_SYMBOL(utf8_load); +static void utf8_unload_rcu(struct rcu_head *head) +{ + struct unicode_map *um = container_of(head, struct unicode_map, rcu); + + symbol_put(utf8_data_table); + kfree(um); +} + +/* + * A pathwalk in rcu mode may still be in ->d_hash() or ->d_compare() of a + * filesystem that is being torn down and read the map through its + * superblock, so the map and the tables it points to have to outlive the + * grace period. + */ void utf8_unload(struct unicode_map *um) { - if (um) { - symbol_put(utf8_data_table); - kfree(um); - } + if (um) + call_rcu(&um->rcu, utf8_unload_rcu); } EXPORT_SYMBOL(utf8_unload); diff --git a/fs/xfs/libxfs/xfs_sb.c b/fs/xfs/libxfs/xfs_sb.c index f4656d17121e..279e174fba8a 100644 --- a/fs/xfs/libxfs/xfs_sb.c +++ b/fs/xfs/libxfs/xfs_sb.c @@ -1150,26 +1150,23 @@ xfs_sync_sb_buf( struct xfs_mount *mp) { struct xfs_trans *tp; - struct xfs_buf *bp; int error; error = xfs_trans_alloc(mp, &M_RES(mp)->tr_sb, 0, 0, 0, &tp); if (error) return error; - bp = xfs_trans_getsb(tp); xfs_log_sb(tp); - xfs_trans_bhold(tp, bp); xfs_trans_set_sync(tp); error = xfs_trans_commit(tp); if (error) - goto out; - /* - * write out the sb buffer to get the changes to disk - */ - error = xfs_bwrite(bp); -out: - xfs_buf_relse(bp); + return error; + + /* Re-acquire and write the sb to disk. */ + xfs_buf_lock(mp->m_sb_bp); + error = xfs_bwrite(mp->m_sb_bp); + xfs_buf_unlock(mp->m_sb_bp); + return error; } diff --git a/fs/xfs/xfs_fsmap.c b/fs/xfs/xfs_fsmap.c index 7754d51e1c27..98bd5078da5b 100644 --- a/fs/xfs/xfs_fsmap.c +++ b/fs/xfs/xfs_fsmap.c @@ -902,7 +902,7 @@ xfs_getfsmap( return -EINVAL; use_rmap = xfs_has_rmapbt(mp) && - has_capability_noaudit(current, CAP_SYS_ADMIN); + ns_capable_noaudit(&init_user_ns, CAP_SYS_ADMIN); head->fmh_entries = 0; /* Set up our device handlers. */ diff --git a/fs/xfs/xfs_ioctl.c b/fs/xfs/xfs_ioctl.c index d22285a74539..1d0c52925566 100644 --- a/fs/xfs/xfs_ioctl.c +++ b/fs/xfs/xfs_ioctl.c @@ -1218,7 +1218,7 @@ xfs_ioctl_setattr_get_trans( goto out_error; error = xfs_trans_alloc_ichange(ip, NULL, NULL, pdqp, - has_capability_noaudit(current, CAP_FOWNER), &tp); + ns_capable_noaudit(&init_user_ns, CAP_FOWNER), &tp); if (error) goto out_error; diff --git a/fs/xfs/xfs_iops.c b/fs/xfs/xfs_iops.c index f9466311dfea..982c1590a579 100644 --- a/fs/xfs/xfs_iops.c +++ b/fs/xfs/xfs_iops.c @@ -712,7 +712,7 @@ xfs_setattr_nonsize( } error = xfs_trans_alloc_ichange(ip, udqp, gdqp, NULL, - has_capability_noaudit(current, CAP_FOWNER), &tp); + ns_capable_noaudit(&init_user_ns, CAP_FOWNER), &tp); if (error) goto out_dqrele; diff --git a/include/drm/drm_exec.h b/include/drm/drm_exec.h index b5bf0b6da791..fe2f21f4f066 100644 --- a/include/drm/drm_exec.h +++ b/include/drm/drm_exec.h @@ -71,7 +71,8 @@ drm_exec_obj(struct drm_exec *exec, unsigned long index) * @index: unsigned long index for the iteration * @obj: the current GEM object * - * Iterate over all the locked GEM objects inside the drm_exec object. + * Iterate over all the explicitly locked GEM objects inside the drm_exec + * container, except for the contended and implicit prelocked one. */ #define drm_exec_for_each_locked_object(exec, index, obj) \ for ((index) = 0; ((obj) = drm_exec_obj(exec, index)); ++(index)) @@ -83,9 +84,9 @@ drm_exec_obj(struct drm_exec *exec, unsigned long index) * @index: unsigned long index for the iteration * @obj: the current GEM object * - * Iterate over all the locked GEM objects inside the drm_exec object in - * reverse locking order. Note that @index may go below zero and wrap, - * but that will be caught by drm_exec_obj(), returning a NULL object. + * Iterate over all the explicitly locked GEM objects inside the drm_exec + * container in reverse locking order. Note that @index may go below zero and wrap + * around, but that will be caught by drm_exec_obj(), returning a NULL object. */ #define drm_exec_for_each_locked_object_reverse(exec, index, obj) \ for ((index) = (exec)->num_objects - 1; \ diff --git a/include/linux/bsg-lib.h b/include/linux/bsg-lib.h index 9e97ced2896d..1c6b057c6412 100644 --- a/include/linux/bsg-lib.h +++ b/include/linux/bsg-lib.h @@ -65,7 +65,8 @@ struct bsg_job { void bsg_job_done(struct bsg_job *job, int result, unsigned int reply_payload_rcv_len); struct request_queue *bsg_setup_queue(struct device *dev, const char *name, - bsg_job_fn *job_fn, bsg_timeout_fn *timeout, int dd_job_size); + bsg_job_fn *job_fn, bsg_timeout_fn *timeout, int dd_job_size, + unsigned int reply_len); void bsg_remove_queue(struct request_queue *q); void bsg_job_put(struct bsg_job *job); int __must_check bsg_job_get(struct bsg_job *job); diff --git a/include/linux/cgroup-defs.h b/include/linux/cgroup-defs.h index c0c2b26725d0..d47ffa2d551f 100644 --- a/include/linux/cgroup-defs.h +++ b/include/linux/cgroup-defs.h @@ -435,7 +435,10 @@ struct cgroup { int nr_threaded_children; /* # of live threaded child cgroups */ - /* sequence number for cgroup.kill, serialized by css_set_lock. */ + /* + * Sequence number for cgroup.kill. Incremented with both cgroup_mutex + * and css_set_lock held. Readers hold either one. + */ unsigned int kill_seq; struct kernfs_node *kn; /* cgroup kernfs entry */ diff --git a/include/linux/device-mapper.h b/include/linux/device-mapper.h index 69d0435c7ebb..f61daeeecee6 100644 --- a/include/linux/device-mapper.h +++ b/include/linux/device-mapper.h @@ -356,6 +356,11 @@ struct dm_target { */ bool discards_supported:1; + /* + * Set if this target requires stable data during writeback. + */ + bool needs_stable_writes:1; + /* * Set if this target requires that discards be split on * 'max_discard_sectors' boundaries. diff --git a/include/linux/fsnotify_backend.h b/include/linux/fsnotify_backend.h index 9bc585a29b78..d81b9b263412 100644 --- a/include/linux/fsnotify_backend.h +++ b/include/linux/fsnotify_backend.h @@ -474,6 +474,7 @@ struct fsnotify_mark_connector { unsigned short type; /* Type of object [lock] */ #define FSNOTIFY_CONN_FLAG_HAS_FSID 0x01 #define FSNOTIFY_CONN_FLAG_HAS_IREF 0x02 +#define FSNOTIFY_CONN_FLAG_DESTROYING 0x04 unsigned short flags; /* flags [lock] */ __kernel_fsid_t fsid; /* fsid of filesystem containing object */ union { diff --git a/include/linux/futex.h b/include/linux/futex.h index b70df27d7e85..09f757f14d01 100644 --- a/include/linux/futex.h +++ b/include/linux/futex.h @@ -23,26 +23,30 @@ struct task_struct; * 01 : Shared futex (PTHREAD_PROCESS_SHARED) * mapped on a file (reference on the underlying inode) * 10 : Shared futex (PTHREAD_PROCESS_SHARED) - * (but private mapping on an mm, and reference taken on it) + * (but private mapping on an mm) */ -#define FUT_OFF_INODE 1 /* We set bit 0 if key has a reference on inode */ -#define FUT_OFF_MMSHARED 2 /* We set bit 1 if key has a reference on mm */ +#define FUT_OFF_INODE 1 /* We set bit 0 if shared key identifies an inode */ +#define FUT_OFF_MMSHARED 2 /* We set bit 1 if shared key identifies an mm */ union futex_key { + /* For FUT_OFF_INODE */ struct { u64 i_seq; unsigned long pgoff; unsigned int offset; } shared; + + /* For FUT_OFF_MMSHARED or private */ struct { union { - struct mm_struct *mm; - u64 __tmp; + struct mm_struct *mm; /* for private */ + u64 mm_seq; /* for FUT_OFF_MMSHARED */ }; unsigned long address; unsigned int offset; } private; + struct { u64 ptr; unsigned long word; diff --git a/include/linux/io_uring_types.h b/include/linux/io_uring_types.h index dad2f9e4d53d..d05ed8c7ae1e 100644 --- a/include/linux/io_uring_types.h +++ b/include/linux/io_uring_types.h @@ -362,6 +362,9 @@ struct io_ring_ctx { bool iowq_limits_set; struct callback_head poll_wq_task_work; + /* deferred ->poll_wq wake up for local task_work */ + struct rcu_head poll_wq_rcu; + atomic_t poll_wq_rcu_pending; struct list_head defer_list; unsigned sq_thread_idle; /* protected by ->completion_lock */ @@ -576,7 +579,7 @@ struct io_kiocb { __poll_t apoll_events; }; atomic_t refs; - atomic_t poll_refs; + bool cancel_seq_set; struct io_task_work io_task_work; unsigned nr_tw; /* for polled requests, i.e. IORING_OP_POLL_ADD and async armed poll */ @@ -586,6 +589,7 @@ struct io_kiocb { /* opcode allocated if it needs to store data for async defer */ void *async_data; /* linked requests, IFF REQ_F_HARDLINK or REQ_F_LINK are set */ + atomic_t poll_refs; struct io_kiocb *link; /* custom credentials, valid IFF REQ_F_CREDS is set */ const struct cred *creds; diff --git a/include/linux/ipv6.h b/include/linux/ipv6.h index 9cc278b5e4f4..915548699da8 100644 --- a/include/linux/ipv6.h +++ b/include/linux/ipv6.h @@ -117,6 +117,28 @@ static inline unsigned int ipv6_transport_len(const struct sk_buff *skb) skb_network_header_len(skb); } +static inline unsigned int +ipv6_payload_len(const struct sk_buff *skb, const struct ipv6hdr *ip6) +{ + u32 len = ntohs(ip6->payload_len); + + return (len || !skb_is_gso(skb) || !skb_is_gso_tcp(skb)) ? + len : + skb->len - skb_network_offset(skb) - sizeof(struct ipv6hdr); +} + +static inline unsigned int skb_ipv6_payload_len(const struct sk_buff *skb) +{ + return ipv6_payload_len(skb, ipv6_hdr(skb)); +} + +#define IPV6_MAXPLEN 65535 + +static inline void ipv6_set_payload_len(struct ipv6hdr *ip6, unsigned int len) +{ + ip6->payload_len = len <= IPV6_MAXPLEN ? htons(len) : 0; +} + /* This structure contains results of exthdrs parsing as offsets from skb->nh. diff --git a/include/linux/key-type.h b/include/linux/key-type.h index 5caf3ce82373..1336c14138ab 100644 --- a/include/linux/key-type.h +++ b/include/linux/key-type.h @@ -158,6 +158,7 @@ struct key_type { const void *in, const void *in2); /* internal fields */ + atomic64_t index_id; /* Opaque keyring index ID */ struct list_head link; /* link in types list */ struct lock_class_key lock_class; /* key->sem lock class */ } __randomize_layout; diff --git a/include/linux/key.h b/include/linux/key.h index 938d7ecfb495..1d0732dd5c53 100644 --- a/include/linux/key.h +++ b/include/linux/key.h @@ -107,6 +107,7 @@ struct keyring_name; struct key_tag { struct rcu_head rcu; + atomic64_t index_id; /* Opaque keyring index ID */ refcount_t usage; bool removed; /* T when subject removed */ }; @@ -129,6 +130,8 @@ struct keyring_index_key { struct key_type *type; struct key_tag *domain_tag; /* Domain of operation */ const char *description; + u64 type_id; /* Cached index IDs */ + u64 domain_id; }; union key_payload { @@ -250,6 +253,8 @@ struct key { struct key_type *type; /* type of key */ struct key_tag *domain_tag; /* Domain of operation */ char *description; + u64 type_id; + u64 domain_id; }; }; diff --git a/include/linux/mm_types.h b/include/linux/mm_types.h index ce3370823f9c..91c10d8b00c0 100644 --- a/include/linux/mm_types.h +++ b/include/linux/mm_types.h @@ -849,6 +849,9 @@ struct mm_struct { */ int mm_lock_seq; #endif +#ifdef CONFIG_FUTEX + atomic64_t futex_unique_id; +#endif unsigned long hiwater_rss; /* High-watermark of RSS usage */ diff --git a/include/linux/power_supply.h b/include/linux/power_supply.h index 85b86768c0b9..aa20bf8c3366 100644 --- a/include/linux/power_supply.h +++ b/include/linux/power_supply.h @@ -365,9 +365,9 @@ struct power_supply_vbat_ri_table { * @charge_voltage_max_uv: maintenance charging voltage that is usually a bit * lower than the constant_charge_voltage_max_uv. We can apply this settings * charge_current_max_ua until we get back up to this voltage. - * @safety_timer_minutes: maintenance charging safety timer, with an expiry - * time in minutes. We will only use maintenance charging in this setting - * for a certain amount of time, then we will first move to the next + * @charge_safety_timer_minutes: maintenance charging safety timer, with an + * expiry time in minutes. We will only use maintenance charging in this + * setting for a certain amount of time, then we will first move to the next * maintenance charge current and voltage pair in respective array and wait * for the next safety timer timeout, or, if we reached the last maintencance * charging setting, disable charging until we reach diff --git a/include/linux/sched.h b/include/linux/sched.h index 66b554f5fdb2..344cb874e3f8 100644 --- a/include/linux/sched.h +++ b/include/linux/sched.h @@ -1768,7 +1768,7 @@ extern struct pid __rcu *cad_pid; * I am cleaning dirty pages from some other bdi. */ #define PF_KTHREAD 0x00200000 /* I am a kernel thread */ #define PF_RANDOMIZE 0x00400000 /* Randomize virtual address space */ -#define PF__HOLE__00800000 0x00800000 +#define PF_NO_NOTIFY_SIGNAL 0x00800000 /* see no_notify_signal_save() */ #define PF__HOLE__01000000 0x01000000 #define PF__HOLE__02000000 0x02000000 #define PF_NO_SETAFFINITY 0x04000000 /* Userland is not allowed to meddle with cpus_mask */ diff --git a/include/linux/sched/signal.h b/include/linux/sched/signal.h index daa95d325189..193b364e57eb 100644 --- a/include/linux/sched/signal.h +++ b/include/linux/sched/signal.h @@ -2,6 +2,7 @@ #ifndef _LINUX_SCHED_SIGNAL_H #define _LINUX_SCHED_SIGNAL_H +#include #include #include #include @@ -389,14 +390,36 @@ static inline int task_sigpending(struct task_struct *p) return unlikely(test_tsk_thread_flag(p,TIF_SIGPENDING)); } +/* Prevent TIF_NOTIFY_SIGNAL from interrupting this task. */ +static inline unsigned int no_notify_signal_save(void) +{ + unsigned int flags = current->flags; + + current->flags |= PF_NO_NOTIFY_SIGNAL; + return flags; +} + +/* Restore the previous PF_NO_NOTIFY_SIGNAL state. */ +static inline void no_notify_signal_restore(unsigned int flags) +{ + current_restore_flags(flags, PF_NO_NOTIFY_SIGNAL); +} + +DEFINE_LOCK_GUARD_0(no_notify_signal, + _T->flags = no_notify_signal_save(), + no_notify_signal_restore(_T->flags), + unsigned int flags) + static inline int signal_pending(struct task_struct *p) { /* * TIF_NOTIFY_SIGNAL isn't really a signal, but it requires the same * behavior in terms of ensuring that we break out of wait loops - * so that notify signal callbacks can be processed. + * so that notify signal callbacks can be processed. Not for a task + * that asked not to be interrupted by it, see no_notify_signal_save(). */ - if (unlikely(test_tsk_thread_flag(p, TIF_NOTIFY_SIGNAL))) + if (unlikely(test_tsk_thread_flag(p, TIF_NOTIFY_SIGNAL)) && + likely(!(READ_ONCE(p->flags) & PF_NO_NOTIFY_SIGNAL))) return 1; return task_sigpending(p); } diff --git a/include/linux/sched/user.h b/include/linux/sched/user.h index 8d7e5521f7cd..ec262b605437 100644 --- a/include/linux/sched/user.h +++ b/include/linux/sched/user.h @@ -23,11 +23,13 @@ struct user_struct { struct hlist_node uidhash_node; kuid_t uid; -#if defined(CONFIG_PERF_EVENTS) || defined(CONFIG_BPF_SYSCALL) || \ - defined(CONFIG_NET) || defined(CONFIG_IO_URING) || \ +#if defined(CONFIG_NET) || defined(CONFIG_IO_URING) || \ defined(CONFIG_VFIO_PCI_ZDEV_KVM) || IS_ENABLED(CONFIG_IOMMUFD) || \ defined(CONFIG_SECRETMEM) - atomic_long_t locked_vm; + atomic_long_t locked_vm; /* pinned pages charged to RLIMIT_MEMLOCK */ +#endif +#ifdef CONFIG_PERF_EVENTS + atomic_long_t perf_mlock; /* perf buffer pages within perf_event_mlock_kb */ #endif #ifdef CONFIG_WATCH_QUEUE atomic_t nr_watches; /* The number of watches this user currently has */ diff --git a/include/linux/srcutiny.h b/include/linux/srcutiny.h index bad5fd0f1ddb..447133171d95 100644 --- a/include/linux/srcutiny.h +++ b/include/linux/srcutiny.h @@ -11,7 +11,6 @@ #ifndef _LINUX_SRCU_TINY_H #define _LINUX_SRCU_TINY_H -#include #include struct srcu_struct { @@ -25,21 +24,18 @@ struct srcu_struct { struct rcu_head *srcu_cb_head; /* Pending callbacks: Head. */ struct rcu_head **srcu_cb_tail; /* Pending callbacks: Tail. */ struct work_struct srcu_work; /* For driving grace periods. */ - struct irq_work srcu_irq_work; /* Defer schedule_work() to irq work. */ #ifdef CONFIG_DEBUG_LOCK_ALLOC struct lockdep_map dep_map; #endif /* #ifdef CONFIG_DEBUG_LOCK_ALLOC */ }; void srcu_drive_gp(struct work_struct *wp); -void srcu_tiny_irq_work(struct irq_work *irq_work); #define __SRCU_STRUCT_INIT(name, __ignored, ___ignored) \ { \ .srcu_wq = __SWAIT_QUEUE_HEAD_INITIALIZER(name.srcu_wq), \ .srcu_cb_tail = &name.srcu_cb_head, \ .srcu_work = __WORK_INITIALIZER(name.srcu_work, srcu_drive_gp), \ - .srcu_irq_work = { .func = srcu_tiny_irq_work }, \ __SRCU_DEP_MAP_INIT(name) \ } diff --git a/include/linux/unicode.h b/include/linux/unicode.h index 4d39e6e11a95..f5ee53dc4dff 100644 --- a/include/linux/unicode.h +++ b/include/linux/unicode.h @@ -51,6 +51,7 @@ struct unicode_map { unsigned int version; const struct utf8data *ntab[UTF8_NMAX]; const struct utf8data_table *tables; + struct rcu_head rcu; }; int utf8_validate(const struct unicode_map *um, const struct qstr *str); diff --git a/include/net/af_vsock.h b/include/net/af_vsock.h index 1820b87b8b7f..d00f017948ed 100644 --- a/include/net/af_vsock.h +++ b/include/net/af_vsock.h @@ -210,6 +210,9 @@ void vsock_remove_connected(struct vsock_sock *vsk); struct sock *vsock_find_bound_socket(struct sockaddr_vm *addr); struct sock *vsock_find_connected_socket(struct sockaddr_vm *src, struct sockaddr_vm *dst); +bool vsock_check_source(const struct vsock_sock *vsk, + const struct vsock_transport *transport, + const struct sockaddr_vm *src); void vsock_remove_sock(struct vsock_sock *vsk); void vsock_for_each_connected_socket(struct vsock_transport *transport, void (*fn)(struct sock *sk)); diff --git a/include/net/codel_impl.h b/include/net/codel_impl.h index 78a27ac73070..b2c359c6dd1b 100644 --- a/include/net/codel_impl.h +++ b/include/net/codel_impl.h @@ -158,6 +158,7 @@ static struct sk_buff *codel_dequeue(void *ctx, bool drop; if (!skb) { + vars->first_above_time = 0; vars->dropping = false; return skb; } diff --git a/include/net/ipv6.h b/include/net/ipv6.h index 9d72128ea902..60d95434a203 100644 --- a/include/net/ipv6.h +++ b/include/net/ipv6.h @@ -25,8 +25,6 @@ struct ip_tunnel_info; #define SIN6_LEN_RFC2133 24 -#define IPV6_MAXPLEN 65535 - /* * NextHeader field of IPv6 header */ diff --git a/include/net/netfilter/nf_tables_ipv6.h b/include/net/netfilter/nf_tables_ipv6.h index a0633eeaec97..c53ac00bb974 100644 --- a/include/net/netfilter/nf_tables_ipv6.h +++ b/include/net/netfilter/nf_tables_ipv6.h @@ -42,7 +42,7 @@ static inline int __nft_set_pktinfo_ipv6_validate(struct nft_pktinfo *pkt) if (ip6h->version != 6) return -1; - pkt_len = ntohs(ip6h->payload_len); + pkt_len = ipv6_payload_len(pkt->skb, ip6h); skb_len = pkt->skb->len - skb_network_offset(pkt->skb); if (pkt_len + sizeof(*ip6h) > skb_len) return -1; @@ -86,7 +86,7 @@ static inline int nft_set_pktinfo_ipv6_ingress(struct nft_pktinfo *pkt) if (ip6h->version != 6) goto inhdr_error; - pkt_len = ntohs(ip6h->payload_len); + pkt_len = ipv6_payload_len(pkt->skb, ip6h); if (pkt_len + sizeof(*ip6h) > pkt->skb->len) { idev = __in6_dev_get(nft_in(pkt)); __IP6_INC_STATS(nft_net(pkt), idev, IPSTATS_MIB_INTRUNCATEDPKTS); diff --git a/io_uring/cancel.c b/io_uring/cancel.c index a5d51471feeb..5663b8bf4ccb 100644 --- a/io_uring/cancel.c +++ b/io_uring/cancel.c @@ -56,9 +56,8 @@ bool io_cancel_req_match(struct io_kiocb *req, struct io_cancel_data *cd) return false; if (cd->flags & IORING_ASYNC_CANCEL_ALL) { check_seq: - if (cd->seq == req->work.cancel_seq) + if (io_cancel_match_sequence(req, cd->seq)) return false; - req->work.cancel_seq = cd->seq; } return true; @@ -69,7 +68,16 @@ static bool io_cancel_cb(struct io_wq_work *work, void *data) struct io_kiocb *req = container_of(work, struct io_kiocb, work); struct io_cancel_data *cd = data; - return io_cancel_req_match(req, cd); + if (!io_cancel_req_match(req, cd)) + return false; + /* + * Pending work is gone once it has matched. Work that is running may + * park the request where the same cancellation looks next. Don't make + * it skip the request as one that it has dealt with. + */ + if (cd->flags & (IORING_ASYNC_CANCEL_ALL | IORING_ASYNC_CANCEL_ANY)) + req->cancel_seq_set = false; + return true; } static int io_async_cancel_one(struct io_uring_task *tctx, @@ -114,6 +122,8 @@ int io_try_cancel(struct io_uring_task *tctx, struct io_cancel_data *cd, */ if (!ret) return 0; + if (ret == -EALREADY) + cd->running = true; ret = io_poll_cancel(ctx, cd, issue_flags); if (ret != -ENOENT) @@ -162,6 +172,7 @@ static int __io_async_cancel(struct io_cancel_data *cd, struct io_tctx_node *node; int ret, nr = 0; + cd->running = false; do { ret = io_try_cancel(tctx, cd, issue_flags); if (ret == -ENOENT) @@ -175,9 +186,10 @@ static int __io_async_cancel(struct io_cancel_data *cd, io_ring_submit_lock(ctx, issue_flags); ret = -ENOENT; list_for_each_entry(node, &ctx->tctx_list, ctx_node) { - struct io_uring_task *tctx = node->task->io_uring; - - ret = io_async_cancel_one(tctx, cd); + /* has been done above, don't count what is running there */ + if (all && node->task->io_uring == tctx) + continue; + ret = io_async_cancel_one(node->task->io_uring, cd); if (ret != -ENOENT) { if (!all) break; @@ -185,7 +197,15 @@ static int __io_async_cancel(struct io_cancel_data *cd, } } io_ring_submit_unlock(ctx, issue_flags); - return all ? nr : ret; + if (all) + return nr; + /* + * The worker may be done with the work that was found running, and + * flagged, above. That isn't the same as not having found it. + */ + if (ret == -ENOENT && cd->running) + ret = -EALREADY; + return ret; } int io_async_cancel(struct io_kiocb *req, unsigned int issue_flags) @@ -297,7 +317,7 @@ int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg) if (ret != -EALREADY) goto out; - if (sc.timeout.tv_sec != -1UL || sc.timeout.tv_nsec != -1UL) { + if (sc.timeout.tv_sec != -1 || sc.timeout.tv_nsec != -1) { struct timespec64 ts = { .tv_sec = sc.timeout.tv_sec, .tv_nsec = sc.timeout.tv_nsec diff --git a/io_uring/cancel.h b/io_uring/cancel.h index 7e6d0fca7db2..9bda1ed5209b 100644 --- a/io_uring/cancel.h +++ b/io_uring/cancel.h @@ -9,6 +9,8 @@ struct io_cancel_data { u8 opcode; u32 flags; int seq; + /* io_try_cancel() has found it being run by io-wq, and flagged it */ + bool running; }; int io_async_cancel_prep(struct io_kiocb *req, const struct io_uring_sqe *sqe); @@ -20,3 +22,13 @@ void init_hash_table(struct io_hash_table *table, unsigned size); int io_sync_cancel(struct io_ring_ctx *ctx, void __user *arg); bool io_cancel_req_match(struct io_kiocb *req, struct io_cancel_data *cd); + +static inline bool io_cancel_match_sequence(struct io_kiocb *req, int sequence) +{ + if (req->cancel_seq_set && sequence == req->work.cancel_seq) + return true; + + req->cancel_seq_set = true; + req->work.cancel_seq = sequence; + return false; +} diff --git a/io_uring/io-wq.c b/io_uring/io-wq.c index a20be1a43aaf..49cc69204f3d 100644 --- a/io_uring/io-wq.c +++ b/io_uring/io-wq.c @@ -54,6 +54,7 @@ struct io_worker { struct io_wq *wq; struct io_wq_work *cur_work; + /* rest of the hashed chain ->cur_work was taken off the list with */ struct io_wq_work *next_work; raw_spinlock_t lock; @@ -99,6 +100,7 @@ enum { struct io_wq { unsigned long state; + put_work_fn *put_work; free_work_fn *free_work; io_wq_work_fn *do_work; @@ -535,6 +537,17 @@ static struct io_wq_work *io_get_next_work(struct io_wq_acct *acct, return NULL; } +static void __io_assign_current_work(struct io_worker *worker, + struct io_wq_work *work) +{ + raw_spin_lock(&worker->lock); + worker->cur_work = work; + /* stays valid while in between two items of a chain */ + if (work) + worker->next_work = wq_next_work(work); + raw_spin_unlock(&worker->lock); +} + static void io_assign_current_work(struct io_worker *worker, struct io_wq_work *work) { @@ -543,10 +556,7 @@ static void io_assign_current_work(struct io_worker *worker, cond_resched(); } - raw_spin_lock(&worker->lock); - worker->cur_work = work; - worker->next_work = NULL; - raw_spin_unlock(&worker->lock); + __io_assign_current_work(worker, work); } /* @@ -577,9 +587,7 @@ static void io_worker_handle_work(struct io_wq_acct *acct, * work list, but isn't yet discoverable as the * current work item for this worker. */ - raw_spin_lock(&worker->lock); - worker->next_work = work; - raw_spin_unlock(&worker->lock); + __io_assign_current_work(worker, work); } raw_spin_unlock(&acct->lock); @@ -595,17 +603,25 @@ static void io_worker_handle_work(struct io_wq_acct *acct, /* handle a whole dependent link */ do { bool do_kill = test_bit(IO_WQ_BIT_EXIT, &wq->state); - struct io_wq_work *next_hashed, *linked; + struct io_wq_work *next_hashed, *linked, *done = work; unsigned int hash = io_get_work_hash(work); + bool last; next_hashed = wq_next_work(work); + /* + * io_wq_exit_workers() signals us once, so that we + * don't sleep for ever. If that was before this work + * became the current one, the signal is gone. + */ if (unlikely(do_kill) && (work->flags & IO_WQ_WORK_UNBOUND)) work->flags |= IO_WQ_WORK_CANCEL; + else if (unlikely(do_kill)) + set_notify_signal(current); wq->do_work(work); io_assign_current_work(worker, NULL); - linked = wq->free_work(work); + last = wq->put_work(work, &linked); work = next_hashed; if (!work && linked && !io_wq_is_hashed(linked)) { work = linked; @@ -614,6 +630,13 @@ static void io_worker_handle_work(struct io_wq_acct *acct, io_assign_current_work(worker, work); if (linked) io_wq_enqueue(wq, linked); + /* + * Not before the link is current or pending work again. + * This may be the last time that a task is woken up + * which exits and looks for requests to cancel. + */ + if (last) + wq->free_work(done); if (hash != -1U && !next_hashed) { /* serialize hash clear with wake_up() */ @@ -632,6 +655,36 @@ static void io_worker_handle_work(struct io_wq_acct *acct, } while (1); } +/* + * The worker has decided to exit, and has given up its ->nr_workers slot and + * its place on the free list. io_wq_enqueue() may have picked it from that + * list, or seen it running, after it last looked at the work list. Leave + * ->nr_running as well and check again. Returns false if the worker has to + * stay. + */ +static bool io_worker_retire(struct io_wq *wq, struct io_wq_acct *acct, + struct io_worker *worker) +{ + /* turns io_wq_dec_running() and the scheduler hooks into no-ops */ + clear_bit(IO_WORKER_F_UP, &worker->flags); + atomic_dec(&acct->nr_running); + if (!io_acct_run_queue(acct)) + return true; + raw_spin_unlock(&acct->lock); + + raw_spin_lock(&wq->lock); + if (acct->nr_workers >= acct->max_workers) { + /* a worker was created in our place, the work is its business */ + raw_spin_unlock(&wq->lock); + return true; + } + acct->nr_workers++; + raw_spin_unlock(&wq->lock); + atomic_inc(&acct->nr_running); + set_bit(IO_WORKER_F_UP, &worker->flags); + return false; +} + static int io_wq_worker(void *data) { struct io_worker *worker = data; @@ -662,12 +715,21 @@ static int io_wq_worker(void *data) /* * Last sleep timed out. Exit if we're not the last worker, * or if someone modified our affinity. + * Not as long as a request of ours for another worker is + * pending, though. It is accounted as a running worker, which + * others rely on, and io_worker_exit() would cancel it. */ - if (last_timeout && (exit_mask || acct->nr_workers > 1)) { + if (last_timeout && (exit_mask || acct->nr_workers > 1) && + !test_bit(0, &worker->create_state)) { acct->nr_workers--; + if (test_and_clear_bit(IO_WORKER_F_FREE, &worker->flags)) + hlist_nulls_del_init_rcu(&worker->nulls_node); raw_spin_unlock(&wq->lock); __set_current_state(TASK_RUNNING); - break; + if (io_worker_retire(wq, acct, worker)) + break; + last_timeout = false; + continue; } last_timeout = false; __io_worker_idle(wq, worker); @@ -914,9 +976,13 @@ static bool io_wq_worker_wake(struct io_worker *worker, void *data) static void io_run_cancel(struct io_wq_work *work, struct io_wq *wq) { do { + struct io_wq_work *linked; + work->flags |= IO_WQ_WORK_CANCEL; wq->do_work(work); - work = wq->free_work(work); + if (wq->put_work(work, &linked)) + wq->free_work(work); + work = linked; } while (work); } @@ -1024,15 +1090,28 @@ static bool __io_wq_worker_cancel(struct io_worker *worker, static bool io_wq_worker_cancel(struct io_worker *worker, void *data) { struct io_cb_cancel_data *match = data; + struct io_wq_work *work; /* * Hold the lock to avoid ->cur_work going out of scope, caller * may dereference the passed in work. */ raw_spin_lock(&worker->lock); - if (__io_wq_worker_cancel(worker, match, worker->cur_work) || - __io_wq_worker_cancel(worker, match, worker->next_work)) + if (__io_wq_worker_cancel(worker, match, worker->cur_work)) match->nr_running++; + /* + * Hashed work is taken off the work list a chain at a time, what the + * worker hasn't started yet can only be found here. It doesn't move on + * without the lock. Just mark the work, it fails once it is its turn. + */ + for (work = worker->next_work; work; work = wq_next_work(work)) { + if (match->nr_running && !match->cancel_all) + break; + if (match->fn(work, match->data)) { + work->flags |= IO_WQ_WORK_CANCEL; + match->nr_running++; + } + } raw_spin_unlock(&worker->lock); return match->nr_running && !match->cancel_all; @@ -1169,7 +1248,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data) int ret, i; struct io_wq *wq; - if (WARN_ON_ONCE(!data->free_work || !data->do_work)) + if (WARN_ON_ONCE(!data->put_work || !data->free_work || !data->do_work)) return ERR_PTR(-EINVAL); if (WARN_ON_ONCE(!bounded)) return ERR_PTR(-EINVAL); @@ -1180,6 +1259,7 @@ struct io_wq *io_wq_create(unsigned bounded, struct io_wq_data *data) refcount_inc(&data->hash->refs); wq->hash = data->hash; + wq->put_work = data->put_work; wq->free_work = data->free_work; wq->do_work = data->do_work; @@ -1236,6 +1316,8 @@ static bool io_task_work_match(struct callback_head *cb, void *data) void io_wq_exit_start(struct io_wq *wq) { set_bit(IO_WQ_BIT_EXIT, &wq->state); + /* Pairs with task_work_add() in io_queue_worker_create(). */ + smp_mb__after_atomic(); } static void io_wq_cancel_tw_create(struct io_wq *wq) diff --git a/io_uring/io-wq.h b/io_uring/io-wq.h index 2b2a6406dd8e..8bd6645e7463 100644 --- a/io_uring/io-wq.h +++ b/io_uring/io-wq.h @@ -21,7 +21,8 @@ enum io_wq_cancel { IO_WQ_CANCEL_NOTFOUND, /* work not found */ }; -typedef struct io_wq_work *(free_work_fn)(struct io_wq_work *); +typedef bool (put_work_fn)(struct io_wq_work *, struct io_wq_work **); +typedef void (free_work_fn)(struct io_wq_work *); typedef void (io_wq_work_fn)(struct io_wq_work *); struct io_wq_hash { @@ -40,6 +41,7 @@ struct io_wq_data { struct io_wq_hash *hash; struct task_struct *task; io_wq_work_fn *do_work; + put_work_fn *put_work; free_work_fn *free_work; }; diff --git a/io_uring/io_uring.c b/io_uring/io_uring.c index d4973d0d4e43..bc47a7d745b5 100644 --- a/io_uring/io_uring.c +++ b/io_uring/io_uring.c @@ -434,7 +434,6 @@ static void io_prep_async_work(struct io_kiocb *req) req->work.list.next = NULL; req->work.flags = 0; - req->work.cancel_seq = atomic_read(&ctx->cancel_seq); if (req->flags & REQ_F_FORCE_ASYNC) req->work.flags |= IO_WQ_WORK_CONCURRENT; @@ -473,17 +472,46 @@ static void io_prep_async_link(struct io_kiocb *req) } } +/* Called for every request that is issued: keep it inline */ +static __always_inline bool io_assign_file(struct io_kiocb *req, + const struct io_issue_def *def, + unsigned int issue_flags) +{ + if (req->file || !def->needs_file) + return true; + + if (req->flags & REQ_F_FIXED_FILE) + req->file = io_file_get_fixed(req, req->cqe.fd, issue_flags); + else + req->file = io_file_get_normal(req, req->cqe.fd); + + return !!req->file; +} + static void io_queue_iowq(struct io_kiocb *req) { struct io_uring_task *tctx = req->task->io_uring; BUG_ON(!tctx); + lockdep_assert_held(&req->ctx->uring_lock); if ((current->flags & PF_KTHREAD) || !tctx->io_wq) { io_req_task_queue_fail(req, -ECANCELED); return; } + /* + * The file is looked up as late as possible, as it may be the result + * of an earlier request of the link. This one is up now, and what kind + * of work it is depends on the file. If that fails, leave the error + * handling to the worker. + * + * Not for a splice: whatever it writes to, it may wait for its input + * for ever. It has to remain unbound work that isn't hashed. + */ + if (req->opcode != IORING_OP_SPLICE) + io_assign_file(req, &io_issue_defs[req->opcode], 0); + /* init ->work of the whole link before punting */ io_prep_async_link(req); @@ -1274,6 +1302,31 @@ void tctx_task_work(struct callback_head *cb) trace_io_uring_task_work_run(tctx, count, 1); } +static void io_poll_wq_wake_rcu(struct rcu_head *rcu) +{ + struct io_ring_ctx *ctx = container_of(rcu, struct io_ring_ctx, + poll_wq_rcu); + + atomic_set(&ctx->poll_wq_rcu_pending, 0); + io_poll_wq_wake(ctx); + percpu_ref_put(&ctx->refs); +} + +/* + * io_uring_poll() reports pending local work as EPOLLIN, as it takes a trip to + * the kernel to turn it into CQEs. Let those know who are asleep already. + * This may be called from a wake up handler with some wait queue lock held, + * and this wake up could lead back to it. Do it from a clean context. + */ +static void io_local_work_wake_poll(struct io_ring_ctx *ctx) +{ + if (!wq_has_sleeper(&ctx->poll_wq) || + atomic_xchg(&ctx->poll_wq_rcu_pending, 1)) + return; + percpu_ref_get(&ctx->refs); + call_rcu_hurry(&ctx->poll_wq_rcu, io_poll_wq_wake_rcu); +} + static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) { struct io_ring_ctx *ctx = req->ctx; @@ -1311,6 +1364,9 @@ static inline void io_req_local_work_add(struct io_kiocb *req, unsigned flags) atomic_or(IORING_SQ_TASKRUN, &ctx->rings->sq_flags); if (ctx->has_evfd) io_eventfd_signal(ctx, flags & IOU_F_TWQ_IN_WAKE); + /* whoever sets the flag wakes ->poll_wq afterwards */ + if (ctx->poll_activated) + io_local_work_wake_poll(ctx); } nr_wait = atomic_read(&ctx->cq_wait_nr); @@ -1830,20 +1886,6 @@ static __cold void io_drain_req(struct io_kiocb *req) spin_unlock(&ctx->completion_lock); } -static bool io_assign_file(struct io_kiocb *req, const struct io_issue_def *def, - unsigned int issue_flags) -{ - if (req->file || !def->needs_file) - return true; - - if (req->flags & REQ_F_FIXED_FILE) - req->file = io_file_get_fixed(req, req->cqe.fd, issue_flags); - else - req->file = io_file_get_normal(req, req->cqe.fd); - - return !!req->file; -} - #define REQ_ISSUE_SLOW_FLAGS (REQ_F_CREDS | REQ_F_ARM_LTIMEOUT) static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) @@ -1879,6 +1921,9 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) } if (ret == IOU_OK) { + /* not all handlers take care of the internal restart codes */ + if (unlikely(req->cqe.res <= -ERESTARTSYS)) + req->cqe.res = io_fixup_restart_res(req->cqe.res); if (issue_flags & IO_URING_F_COMPLETE_DEFER) io_req_complete_defer(req); else @@ -1887,8 +1932,11 @@ static int io_issue_sqe(struct io_kiocb *req, unsigned int issue_flags) return 0; } - if (ret != IOU_ISSUE_SKIP_COMPLETE) + if (ret != IOU_ISSUE_SKIP_COMPLETE) { + if (unlikely(ret <= -ERESTARTSYS)) + ret = io_fixup_restart_res(ret); return ret; + } /* If the op doesn't have a file, we're not polling for it */ if ((req->ctx->flags & IORING_SETUP_IOPOLL) && def->iopoll_queue) @@ -1904,17 +1952,29 @@ int io_poll_issue(struct io_kiocb *req, struct io_tw_state *ts) IO_URING_F_COMPLETE_DEFER); } -struct io_wq_work *io_wq_free_work(struct io_wq_work *work) +/* + * io-wq is done with @work. Returns true if it held the last reference, and + * hands out the next request of the link then. The caller has to make sure + * that cancellations can find the latter before it calls io_wq_free_work(). + */ +bool io_wq_put_work(struct io_wq_work *work, struct io_wq_work **linked) { struct io_kiocb *req = container_of(work, struct io_kiocb, work); struct io_kiocb *nxt = NULL; - if (req_ref_put_and_test_atomic(req)) { - if (req->flags & IO_REQ_LINK_FLAGS) - nxt = io_req_find_next(req); - io_free_req(req); - } - return nxt ? &nxt->work : NULL; + *linked = NULL; + if (!req_ref_put_and_test_atomic(req)) + return false; + if (req->flags & IO_REQ_LINK_FLAGS) + nxt = io_req_find_next(req); + if (nxt) + *linked = &nxt->work; + return true; +} + +void io_wq_free_work(struct io_wq_work *work) +{ + io_free_req(container_of(work, struct io_kiocb, work)); } void io_wq_submit_work(struct io_wq_work *work) @@ -1925,7 +1985,7 @@ void io_wq_submit_work(struct io_wq_work *work) bool needs_poll = false; int ret = 0, err = -ECANCELED; - /* one will be dropped by ->io_wq_free_work() after returning to io-wq */ + /* one will be dropped by io_wq_put_work() after returning to io-wq */ if (!(req->flags & REQ_F_REFCOUNT)) __io_req_set_refcount(req, 2); else @@ -2152,6 +2212,7 @@ static int io_init_req(struct io_ring_ctx *ctx, struct io_kiocb *req, req->file = NULL; req->rsrc_node = NULL; req->task = current; + req->cancel_seq_set = false; if (unlikely(opcode >= IORING_OP_LAST)) { req->opcode = 0; @@ -2491,7 +2552,8 @@ static int io_wake_function(struct wait_queue_entry *curr, unsigned int mode, */ if (io_should_wake(iowq) || io_has_work(iowq->ctx)) return autoremove_wake_function(curr, mode, wake_flags, key); - return -1; + /* keep going, a waiter further down may be content with less */ + return 0; } int io_run_task_work_sig(struct io_ring_ctx *ctx) @@ -2641,9 +2703,18 @@ static int io_cqring_wait(struct io_ring_ctx *ctx, int min_events, check_cq = READ_ONCE(ctx->check_cq); if (unlikely(check_cq)) { - /* let the caller flush overflows, retry */ - if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) + if (check_cq & BIT(IO_CHECK_CQ_OVERFLOW_BIT)) { io_cqring_do_overflow_flush(ctx); + /* + * No room for what has overflown. It takes the + * application to change that, so return to it. + */ + if (test_bit(IO_CHECK_CQ_OVERFLOW_BIT, + &ctx->check_cq)) { + ret = 0; + break; + } + } if (check_cq & BIT(IO_CHECK_CQ_DROPPED_BIT)) { ret = -EBADR; break; @@ -3148,6 +3219,7 @@ struct io_tctx_exit { struct callback_head task_work; struct completion completion; struct io_ring_ctx *ctx; + bool in_cancel; }; static __cold void io_tctx_exit_cb(struct callback_head *cb) @@ -3162,7 +3234,8 @@ static __cold void io_tctx_exit_cb(struct callback_head *cb) * tctx can be NULL if the queueing of this task_work raced with * work cancelation off the exec path. */ - if (tctx && !atomic_read(&tctx->in_cancel)) + work->in_cancel = tctx && atomic_read(&tctx->in_cancel); + if (tctx && !work->in_cancel) io_uring_del_tctx_node((unsigned long)work->ctx); complete(&work->completion); } @@ -3256,6 +3329,14 @@ static __cold void io_ring_exit_work(struct work_struct *work) * wait is marked as interruptible. */ wait_for_completion_interruptible(&exit.completion); + /* + * The task removes the node when it is done with its + * cancellation. Give it the time for that: it doesn't get back + * to it as long as it finds new task_work each time it has run + * the old one, and we may be one of many who send it. + */ + if (exit.in_cancel) + schedule_timeout_idle(HZ / 20); mutex_lock(&ctx->uring_lock); } mutex_unlock(&ctx->uring_lock); @@ -3491,12 +3572,11 @@ __cold void io_uring_cancel_generic(bool cancel_all, struct io_sq_data *sqd) prepare_to_wait(&tctx->wait, &wait, TASK_INTERRUPTIBLE); io_run_task_work(); io_uring_drop_tctx_refs(current); + /* only skip the sleep for local work this task may run */ xa_for_each(&tctx->xa, index, node) { - if (!llist_empty(&node->ctx->work_llist)) { - WARN_ON_ONCE(node->ctx->submitter_task && - node->ctx->submitter_task != current); + if (!llist_empty(&node->ctx->work_llist) && + io_allowed_defer_tw_run(node->ctx)) goto end_wait; - } } /* * If we've seen completions, retry without waiting. This diff --git a/io_uring/io_uring.h b/io_uring/io_uring.h index f845bf3eef0b..a3c97ffb4d14 100644 --- a/io_uring/io_uring.h +++ b/io_uring/io_uring.h @@ -91,7 +91,8 @@ int io_do_iopoll(struct io_ring_ctx *ctx, bool force_nonspin); void __io_submit_flush_completions(struct io_ring_ctx *ctx); int io_req_prep_async(struct io_kiocb *req); -struct io_wq_work *io_wq_free_work(struct io_wq_work *work); +bool io_wq_put_work(struct io_wq_work *work, struct io_wq_work **linked); +void io_wq_free_work(struct io_wq_work *work); void io_wq_submit_work(struct io_wq_work *work); void io_free_req(struct io_kiocb *req); @@ -192,6 +193,24 @@ static __always_inline bool io_fill_cqe_req(struct io_ring_ctx *ctx, return true; } +static inline ssize_t io_fixup_restart_res(ssize_t ret) +{ + switch (ret) { + case -ERESTARTSYS: + case -ERESTARTNOINTR: + case -ERESTARTNOHAND: + case -ERESTART_RESTARTBLOCK: + /* + * We can't just restart the syscall, since previously + * submitted sqes may already be in progress. Just fail + * this IO with EINTR. + */ + return -EINTR; + default: + return ret; + } +} + static inline void req_set_fail(struct io_kiocb *req) { req->flags |= REQ_F_FAIL; @@ -240,6 +259,29 @@ static inline void io_ring_submit_lock(struct io_ring_ctx *ctx, lockdep_assert_held(&ctx->uring_lock); } +/* + * For a handler that is run by io-wq and parks the request on a list where + * cancellations find it, rather than completing it. Whoever tried to cancel + * the request before that found it running. It has flagged the work item, + * has searched the list in vain, and won't come back. To be called with + * the lock the search takes held: if the search came first, the flag is + * visible. For most lists that is ->uring_lock, timeouts use ->timeout_lock. + */ +static inline bool __io_wq_cancel_seen(struct io_kiocb *req, + unsigned int issue_flags) +{ + return (issue_flags & IO_URING_F_IOWQ) && + (req->work.flags & IO_WQ_WORK_CANCEL); +} + +static inline bool io_wq_cancel_seen(struct io_kiocb *req, + unsigned int issue_flags) +{ + lockdep_assert_held(&req->ctx->uring_lock); + + return __io_wq_cancel_seen(req, issue_flags); +} + static inline void io_commit_cqring(struct io_ring_ctx *ctx) { /* order cqe stores with ring update */ diff --git a/io_uring/poll.c b/io_uring/poll.c index c4cfc7a5236d..8050261ffba0 100644 --- a/io_uring/poll.c +++ b/io_uring/poll.c @@ -258,12 +258,14 @@ static inline void io_poll_execute(struct io_kiocb *req, int res, */ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) { + bool again; int v; if (unlikely(io_should_terminate_tw(req->ctx))) return -ECANCELED; do { + again = false; v = atomic_read(&req->poll_refs); if (unlikely(v != 1)) { @@ -325,10 +327,17 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) } else { int ret; - /* multiple refs and HUP, ensure we loop once more */ + /* + * Multiple refs and HUP, ensure we loop once more. Drop + * down to one ref now, so the next pass ends unless a + * wake up came in. + */ if ((req->cqe.res & (POLLHUP | POLLRDHUP)) && - (v & IO_POLL_REF_MASK) != 1) - v--; + (v & IO_POLL_REF_MASK) != 1) { + atomic_sub((v & IO_POLL_REF_MASK) - 1, + &req->poll_refs); + again = true; + } ret = io_poll_issue(req, ts); if (ret == IOU_STOP_MULTISHOT) return IOU_POLL_REMOVE_POLL_USE_RES; @@ -345,8 +354,8 @@ static int io_poll_check_events(struct io_kiocb *req, struct io_tw_state *ts) * Release all references, retry if someone tried to restart * task_work while we were executing it. */ - } while (atomic_sub_return(v & IO_POLL_REF_MASK, &req->poll_refs) & - IO_POLL_REF_MASK); + v &= IO_POLL_REF_MASK; + } while (again || atomic_cmpxchg(&req->poll_refs, v, 0) != v); return IOU_POLL_NO_ACTION; } @@ -585,10 +594,7 @@ static int __io_arm_poll_handler(struct io_kiocb *req, struct io_poll_table *ipt, __poll_t mask, unsigned issue_flags) { - struct io_ring_ctx *ctx = req->ctx; - INIT_HLIST_NODE(&req->hash_node); - req->work.cancel_seq = atomic_read(&ctx->cancel_seq); io_init_poll_iocb(poll, mask); poll->file = req->file; req->apoll_events = poll->events; @@ -644,6 +650,17 @@ static int __io_arm_poll_handler(struct io_kiocb *req, io_poll_add_hash(req); + /* + * Whoever tried to cancel the request while io-wq was running it has + * flagged the work item. If it searched the hash table before the + * request got there it found nothing, and it won't come back. Both + * the search and the insertion are done with the lock of the hash + * table held, and the flag is set before the search. If the search + * came first, the flag is visible by now. + */ + if (ipt->owning && (req->work.flags & IO_WQ_WORK_CANCEL)) + io_poll_mark_cancelled(req); + if (mask && (poll->events & EPOLLET) && io_poll_can_finish_inline(req, ipt)) { __io_poll_execute(req, mask, 0); @@ -815,9 +832,8 @@ static struct io_kiocb *io_poll_find(struct io_ring_ctx *ctx, bool poll_only, if (poll_only && req->opcode != IORING_OP_POLL_ADD) continue; if (cd->flags & IORING_ASYNC_CANCEL_ALL) { - if (cd->seq == req->work.cancel_seq) + if (io_cancel_match_sequence(req, cd->seq)) continue; - req->work.cancel_seq = cd->seq; } *out_bucket = hb; return req; @@ -976,6 +992,18 @@ int io_poll_add(struct io_kiocb *req, unsigned int issue_flags) if (req->ctx->flags & (IORING_SETUP_SQPOLL|IORING_SETUP_SINGLE_ISSUER)) req->flags |= REQ_F_HASH_LOCKED; + /* + * A poll update or a spurious wake up arms the request again. Forget + * the entries of the previous round like io_arm_apoll() does, a stale + * second entry would be taken for a third wait queue. + */ + if (req_has_async_data(req)) { + kfree(req->async_data); + req->async_data = NULL; + req->flags &= ~REQ_F_ASYNC_DATA; + } + req->flags &= ~(REQ_F_SINGLE_POLL | REQ_F_DOUBLE_POLL); + ret = __io_arm_poll_handler(req, poll, &ipt, poll->events, issue_flags); if (ret > 0) { io_req_set_res(req, ipt.result_mask, 0); diff --git a/io_uring/rw.c b/io_uring/rw.c index 9c3932302cab..102f92dce672 100644 --- a/io_uring/rw.c +++ b/io_uring/rw.c @@ -140,24 +140,6 @@ void io_readv_writev_cleanup(struct io_kiocb *req) kfree(io->free_iovec); } -static inline ssize_t io_fixup_restart_res(ssize_t ret) -{ - switch (ret) { - case -ERESTARTSYS: - case -ERESTARTNOINTR: - case -ERESTARTNOHAND: - case -ERESTART_RESTARTBLOCK: - /* - * We can't just restart the syscall, since previously - * submitted sqes may already be in progress. Just fail - * this IO with EINTR. - */ - return -EINTR; - default: - return ret; - } -} - static inline void io_rw_done(struct kiocb *kiocb, ssize_t ret) { /* IO was queued async, completion will happen later */ diff --git a/io_uring/sqpoll.c b/io_uring/sqpoll.c index 489e66647e07..76643b4a08e9 100644 --- a/io_uring/sqpoll.c +++ b/io_uring/sqpoll.c @@ -300,6 +300,13 @@ static int io_sq_thread(void *data) */ smp_mb__after_atomic(); + /* + * Nothing gets submitted before the ring is + * enabled, which comes with a wake up. + */ + if (READ_ONCE(ctx->flags) & IORING_SETUP_R_DISABLED) + continue; + if (io_sqring_entries(ctx)) { needs_sched = false; break; diff --git a/io_uring/tctx.c b/io_uring/tctx.c index 84f6a8385720..8597bcb903b2 100644 --- a/io_uring/tctx.c +++ b/io_uring/tctx.c @@ -35,6 +35,7 @@ static struct io_wq *io_init_wq_offload(struct io_ring_ctx *ctx, data.hash = hash; data.task = task; + data.put_work = io_wq_put_work; data.free_work = io_wq_free_work; data.do_work = io_wq_submit_work; diff --git a/io_uring/timeout.c b/io_uring/timeout.c index edf4b2455c48..71f28e92affd 100644 --- a/io_uring/timeout.c +++ b/io_uring/timeout.c @@ -116,6 +116,14 @@ __cold void io_flush_timeouts(struct io_ring_ctx *ctx) spin_lock_irq(&ctx->timeout_lock); seq = ctx->cached_cq_tail - atomic_read(&ctx->cq_timeouts); + /* + * A fired timeout bumps ->cq_timeouts before its CQE is posted, which + * leaves the sequence behind the last flush until it is. No event has + * happened then, see io_timeout(). + */ + if ((s32)(seq - ctx->cq_last_tm_flush) < 0) + seq = ctx->cq_last_tm_flush; + list_for_each_entry_safe(timeout, tmp, &ctx->timeout_list, list) { struct io_kiocb *req = cmd_to_io_kiocb(timeout); u32 events_needed, events_got; @@ -257,22 +265,29 @@ static enum hrtimer_restart io_timeout_fn(struct hrtimer *timer) return HRTIMER_NORESTART; } -static struct io_kiocb *io_timeout_extract(struct io_ring_ctx *ctx, - struct io_cancel_data *cd) +static struct io_kiocb *io_timeout_find(struct io_ring_ctx *ctx, + struct io_cancel_data *cd) __must_hold(&ctx->timeout_lock) { struct io_timeout *timeout; - struct io_timeout_data *io; - struct io_kiocb *req = NULL; list_for_each_entry(timeout, &ctx->timeout_list, list) { - struct io_kiocb *tmp = cmd_to_io_kiocb(timeout); + struct io_kiocb *req = cmd_to_io_kiocb(timeout); - if (io_cancel_req_match(tmp, cd)) { - req = tmp; - break; - } + if (io_cancel_req_match(req, cd)) + return req; } + return NULL; +} + +static struct io_kiocb *io_timeout_extract(struct io_ring_ctx *ctx, + struct io_cancel_data *cd) + __must_hold(&ctx->timeout_lock) +{ + struct io_kiocb *req = io_timeout_find(ctx, cd); + struct io_timeout *timeout; + struct io_timeout_data *io; + if (!req) return ERR_PTR(-ENOENT); @@ -404,15 +419,24 @@ static int io_timeout_update(struct io_ring_ctx *ctx, __u64 user_data, __must_hold(&ctx->timeout_lock) { struct io_cancel_data cd = { .ctx = ctx, .data = user_data, }; - struct io_kiocb *req = io_timeout_extract(ctx, &cd); + struct io_kiocb *req = io_timeout_find(ctx, &cd); struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout); struct io_timeout_data *data; - if (IS_ERR(req)) - return PTR_ERR(req); + if (!req) + return -ENOENT; - timeout->off = 0; /* noseq */ data = req->async_data; + /* multishot requests only make sense with rel values */ + if ((data->flags & IORING_TIMEOUT_MULTISHOT) && mode == HRTIMER_MODE_ABS) + return -EINVAL; + + req = io_timeout_extract(ctx, &cd); + if (IS_ERR(req)) + return PTR_ERR(req); + /* for a multishot request this isn't a sequence, keep the repeats */ + if (!(data->flags & IORING_TIMEOUT_MULTISHOT)) + timeout->off = 0; /* noseq */ data->ts = *ts; list_add_tail(&timeout->list, &ctx->timeout_list); @@ -579,11 +603,19 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags) struct io_timeout *timeout = io_kiocb_to_cmd(req, struct io_timeout); struct io_ring_ctx *ctx = req->ctx; struct io_timeout_data *data = req->async_data; + struct io_timeout *first; struct list_head *entry; - u32 tail, off = timeout->off; + u32 tail, base, off = timeout->off; spin_lock_irq(&ctx->timeout_lock); + /* the list is searched under ->timeout_lock, see io_wq_cancel_seen() */ + if (__io_wq_cancel_seen(req, issue_flags)) { + spin_unlock_irq(&ctx->timeout_lock); + io_req_task_queue_fail(req, -ECANCELED); + return IOU_ISSUE_SKIP_COMPLETE; + } + /* * sqe->off holds how many events that need to occur for this * timeout event to be satisfied. If it isn't set, then this is @@ -595,13 +627,27 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags) } tail = data_race(ctx->cached_cq_tail) - atomic_read(&ctx->cq_timeouts); - timeout->target_seq = tail + off; - /* Update the last seq here in case io_flush_timeouts() hasn't. - * This is safe because ->completion_lock is held, and submissions - * and completions are never mixed in the same ->completion_lock section. + /* + * Update the last seq here in case io_flush_timeouts() hasn't, it only + * gets called once sequences are in use. Leave it alone if there are + * timeouts waiting for one. CQEs may have been posted since the last + * flush, and a target that has been reached must not fall behind it. */ - ctx->cq_last_tm_flush = tail; + first = list_first_entry_or_null(&ctx->timeout_list, struct io_timeout, + list); + if (!first || io_is_timeout_noseq(cmd_to_io_kiocb(first))) + ctx->cq_last_tm_flush = tail; + base = ctx->cq_last_tm_flush; + + /* + * A fired timeout bumps ->cq_timeouts before its CQE is posted, which + * leaves the sequence behind the last flush until it is. Nothing else + * separates the two then, so count from the flush. + */ + if ((s32)(tail - base) < 0) + tail = base; + timeout->target_seq = tail + off; /* * Insertion sort, ensuring the first entry in the list is always @@ -613,8 +659,8 @@ int io_timeout(struct io_kiocb *req, unsigned int issue_flags) if (io_is_timeout_noseq(nxt)) continue; - /* nxt.seq is behind @tail, otherwise would've been completed */ - if (off >= nextt->target_seq - tail) + /* no target is behind the last flush, see io_flush_timeouts() */ + if (timeout->target_seq - base >= nextt->target_seq - base) break; } add: diff --git a/kernel/cgroup/cgroup.c b/kernel/cgroup/cgroup.c index 97812c0b04dd..51e5303e50ae 100644 --- a/kernel/cgroup/cgroup.c +++ b/kernel/cgroup/cgroup.c @@ -6449,10 +6449,7 @@ static int cgroup_css_set_fork(struct kernel_clone_args *kargs) spin_lock_irq(&css_set_lock); cset = task_css_set(current); get_css_set(cset); - if (kargs->cgrp) - kargs->kill_seq = kargs->cgrp->kill_seq; - else - kargs->kill_seq = cset->dfl_cgrp->kill_seq; + kargs->kill_seq = cset->dfl_cgrp->kill_seq; spin_unlock_irq(&css_set_lock); if (!(kargs->flags & CLONE_INTO_CGROUP)) { @@ -6517,6 +6514,7 @@ static int cgroup_css_set_fork(struct kernel_clone_args *kargs) put_css_set(cset); fput(f); kargs->cgrp = dst_cgrp; + kargs->kill_seq = dst_cgrp->kill_seq; return ret; err: diff --git a/kernel/events/core.c b/kernel/events/core.c index a226f908490f..70b8d75d7c78 100644 --- a/kernel/events/core.c +++ b/kernel/events/core.c @@ -4363,6 +4363,7 @@ static bool perf_rotate_context(struct perf_cpu_pmu_context *cpc) if (cpu_event) { update_context_time(&cpuctx->ctx); + update_cgrp_time_from_cpuctx(cpuctx, false); __pmu_ctx_sched_out(cpu_epc, EVENT_FLEXIBLE); rotate_ctx(&cpuctx->ctx, cpu_event); __pmu_ctx_sched_in(&cpuctx->ctx, pmu); @@ -6407,7 +6408,7 @@ static void perf_mmap_close(struct vm_area_struct *vma) perf_pmu_output_stop(event); /* now it's safe to free the pages */ - atomic_long_sub(rb->aux_nr_pages - rb->aux_mmap_locked, &mmap_user->locked_vm); + atomic_long_sub(rb->aux_nr_pages - rb->aux_mmap_locked, &mmap_user->perf_mlock); atomic64_sub(rb->aux_mmap_locked, &vma->vm_mm->pinned_vm); /* this has to be the last one */ @@ -6482,7 +6483,7 @@ static void perf_mmap_close(struct vm_area_struct *vma) */ atomic_long_sub((size >> PAGE_SHIFT) + 1 - mmap_locked, - &mmap_user->locked_vm); + &mmap_user->perf_mlock); atomic64_sub(mmap_locked, &vma->vm_mm->pinned_vm); free_uid(mmap_user); @@ -6642,11 +6643,11 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) */ user_lock_limit *= num_online_cpus(); - user_locked = atomic_long_read(&user->locked_vm); + user_locked = atomic_long_read(&user->perf_mlock); /* * sysctl_perf_event_mlock may have changed, so that - * user->locked_vm > user_lock_limit + * user->perf_mlock > user_lock_limit */ if (user_locked > user_lock_limit) user_locked = user_lock_limit; @@ -6654,7 +6655,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) if (user_locked > user_lock_limit) { /* - * charge locked_vm until it hits user_lock_limit; + * charge perf_mlock until it hits user_lock_limit; * charge the rest from pinned_vm */ extra = user_locked - user_lock_limit; @@ -6706,7 +6707,7 @@ static int perf_mmap(struct file *file, struct vm_area_struct *vma) unlock: if (!ret) { - atomic_long_add(user_extra, &user->locked_vm); + atomic_long_add(user_extra, &user->perf_mlock); atomic64_add(extra, &vma->vm_mm->pinned_vm); atomic_inc(&event->mmap_count); @@ -13099,23 +13100,18 @@ perf_event_create_kernel_counter(struct perf_event_attr *attr, int cpu, } EXPORT_SYMBOL_GPL(perf_event_create_kernel_counter); -static void __perf_pmu_remove(struct perf_event_context *ctx, - int cpu, struct pmu *pmu, - struct perf_event_groups *groups, +static void __perf_pmu_remove(struct perf_event_context *ctx, struct pmu *pmu, struct list_head *events) { - struct perf_event *event, *sibling; + struct perf_event *event, *tmp; + + list_for_each_entry_safe(event, tmp, &ctx->event_list, event_entry) { + if (event->pmu_ctx->pmu != pmu) + continue; - perf_event_groups_for_cpu_pmu(event, groups, cpu, pmu) { perf_remove_from_context(event, 0); put_pmu_ctx(event->pmu_ctx); list_add(&event->migrate_entry, events); - - for_each_sibling_event(sibling, event) { - perf_remove_from_context(sibling, 0); - put_pmu_ctx(sibling->pmu_ctx); - list_add(&sibling->migrate_entry, events); - } } } @@ -13191,8 +13187,7 @@ void perf_pmu_migrate_context(struct pmu *pmu, int src_cpu, int dst_cpu) */ mutex_lock_double(&src_ctx->mutex, &dst_ctx->mutex); - __perf_pmu_remove(src_ctx, src_cpu, pmu, &src_ctx->pinned_groups, &events); - __perf_pmu_remove(src_ctx, src_cpu, pmu, &src_ctx->flexible_groups, &events); + __perf_pmu_remove(src_ctx, pmu, &events); if (!list_empty(&events)) { /* diff --git a/kernel/exit.c b/kernel/exit.c index 55a8aa868298..e2dae48578d9 100644 --- a/kernel/exit.c +++ b/kernel/exit.c @@ -1590,16 +1590,10 @@ static int do_wait_pid(struct wait_opts *wo) return 0; } -static long do_wait(struct wait_opts *wo) +static long __do_wait(struct wait_opts *wo) { - int retval; - - trace_sched_process_wait(wo->wo_pid); + long retval; - init_waitqueue_func_entry(&wo->child_wait, child_wait_callback); - wo->child_wait.private = current; - add_wait_queue(¤t->signal->wait_chldexit, &wo->child_wait); -repeat: /* * If there is nothing that can match our criteria, just get out. * We will clear ->notask_error to zero if we see any child that @@ -1611,24 +1605,23 @@ static long do_wait(struct wait_opts *wo) (!wo->wo_pid || !pid_has_task(wo->wo_pid, wo->wo_type))) goto notask; - set_current_state(TASK_INTERRUPTIBLE); read_lock(&tasklist_lock); if (wo->wo_type == PIDTYPE_PID) { retval = do_wait_pid(wo); if (retval) - goto end; + return retval; } else { struct task_struct *tsk = current; do { retval = do_wait_thread(wo, tsk); if (retval) - goto end; + return retval; retval = ptrace_do_wait(wo, tsk); if (retval) - goto end; + return retval; if (wo->wo_flags & __WNOTHREAD) break; @@ -1638,14 +1631,32 @@ static long do_wait(struct wait_opts *wo) notask: retval = wo->notask_error; - if (!retval && !(wo->wo_flags & WNOHANG)) { - retval = -ERESTARTSYS; - if (!signal_pending(current)) { - schedule(); - goto repeat; - } - } -end: + if (!retval && !(wo->wo_flags & WNOHANG)) + return -ERESTARTSYS; + + return retval; +} + +static long do_wait(struct wait_opts *wo) +{ + int retval; + + trace_sched_process_wait(wo->wo_pid); + + init_waitqueue_func_entry(&wo->child_wait, child_wait_callback); + wo->child_wait.private = current; + add_wait_queue(¤t->signal->wait_chldexit, &wo->child_wait); + + do { + set_current_state(TASK_INTERRUPTIBLE | TASK_FREEZABLE); + retval = __do_wait(wo); + if (retval != -ERESTARTSYS) + break; + if (signal_pending(current)) + break; + schedule(); + } while (1); + __set_current_state(TASK_RUNNING); remove_wait_queue(¤t->signal->wait_chldexit, &wo->child_wait); return retval; diff --git a/kernel/fork.c b/kernel/fork.c index 73e8a6b430a0..396a1d4b05e5 100644 --- a/kernel/fork.c +++ b/kernel/fork.c @@ -1299,6 +1299,9 @@ static struct mm_struct *mm_init(struct mm_struct *mm, struct task_struct *p, #endif mm_init_uprobes_state(mm); hugetlb_count_init(mm); +#ifdef CONFIG_FUTEX + atomic64_set(&mm->futex_unique_id, 0); +#endif if (current->mm) { mm->flags = mmf_init_flags(current->mm->flags); diff --git a/kernel/futex/core.c b/kernel/futex/core.c index 660d7061740b..8fb54800ce0a 100644 --- a/kernel/futex/core.c +++ b/kernel/futex/core.c @@ -151,39 +151,27 @@ futex_setup_timer(ktime_t *time, struct hrtimer_sleeper *timeout, } /* - * Generate a machine wide unique identifier for this inode. + * Generate a machine wide unique identifier for this object (inode or mm). * * This relies on u64 not wrapping in the life-time of the machine; which with * 1ns resolution means almost 585 years. - * - * This further relies on the fact that a well formed program will not unmap - * the file while it has a (shared) futex waiting on it. This mapping will have - * a file reference which pins the mount and inode. - * - * If for some reason an inode gets evicted and read back in again, it will get - * a new sequence number and will _NOT_ match, even though it is the exact same - * file. - * - * It is important that futex_match() will never have a false-positive, esp. - * for PI futexes that can mess up the state. The above argues that false-negatives - * are only possible for malformed programs. */ -static u64 get_inode_sequence_number(struct inode *inode) +static u64 get_object_id(atomic64_t *object_seq) { - static atomic64_t i_seq; + static atomic64_t last_assigned_seq; u64 old; - /* Does the inode already have a sequence number? */ - old = atomic64_read(&inode->i_sequence); + /* Does the object already have a sequence number? */ + old = atomic64_read(object_seq); if (likely(old)) return old; for (;;) { - u64 new = atomic64_add_return(1, &i_seq); + u64 new = atomic64_add_return(1, &last_assigned_seq); if (WARN_ON_ONCE(!new)) continue; - old = atomic64_cmpxchg_relaxed(&inode->i_sequence, 0, new); + old = atomic64_cmpxchg_relaxed(object_seq, 0, new); if (old) return old; return new; @@ -206,7 +194,7 @@ static u64 get_inode_sequence_number(struct inode *inode) * * ( inode->i_sequence, page->index, offset_within_page ) * - * [ also see get_inode_sequence_number() ] + * [ also see get_object_id() ] * * For private mappings (or when !@fshared), the key is: * @@ -358,8 +346,8 @@ int get_futex_key(u32 __user *uaddr, bool fshared, union futex_key *key, goto out; } - key->both.offset |= FUT_OFF_MMSHARED; /* ref taken on mm */ - key->private.mm = mm; + key->both.offset |= FUT_OFF_MMSHARED; + key->private.mm_seq = get_object_id(&mm->futex_unique_id); key->private.address = address; } else { @@ -394,7 +382,22 @@ int get_futex_key(u32 __user *uaddr, bool fshared, union futex_key *key, } key->both.offset |= FUT_OFF_INODE; /* inode-based key */ - key->shared.i_seq = get_inode_sequence_number(inode); + /* + * This relies on the fact that a well formed program will not + * unmap the file while it has a (shared) futex waiting on it. + * This mapping will have a file reference which pins the mount + * and inode. + * + * If for some reason an inode gets evicted and read back in + * again, it will get a new sequence number and will _NOT_ match, + * even though it is the exact same file. + * + * It is important that futex_match() will never have a + * false-positive, esp. for PI futexes that can mess up the state. + * The above argues that false-negatives are only possible for + * malformed programs. + */ + key->shared.i_seq = get_object_id(&inode->i_sequence); key->shared.pgoff = page_to_pgoff(tail); rcu_read_unlock(); } diff --git a/kernel/pid_namespace.c b/kernel/pid_namespace.c index e9b2bb260ee6..7ba5468aed82 100644 --- a/kernel/pid_namespace.c +++ b/kernel/pid_namespace.c @@ -215,9 +215,10 @@ void zap_pid_ns_processes(struct pid_namespace *pid_ns) * kernel_wait4() will also block until our children traced from the * parent namespace are detached and become EXIT_DEAD. */ + /* Task work must not busy-loop the reaper, see signal_pending(). */ + guard(no_notify_signal)(); do { clear_thread_flag(TIF_SIGPENDING); - clear_thread_flag(TIF_NOTIFY_SIGNAL); rc = kernel_wait4(-1, NULL, __WALL, NULL); } while (rc != -ECHILD); @@ -245,7 +246,7 @@ void zap_pid_ns_processes(struct pid_namespace *pid_ns) * free_pid() will awaken this task. */ for (;;) { - set_current_state(TASK_INTERRUPTIBLE); + set_current_state(TASK_IDLE | TASK_FREEZABLE); if (pid_ns->pid_allocated == init_pids) break; /* diff --git a/kernel/ptrace.c b/kernel/ptrace.c index 6196a495d9c0..3d3ad296cd79 100644 --- a/kernel/ptrace.c +++ b/kernel/ptrace.c @@ -1100,6 +1100,12 @@ int ptrace_request(struct task_struct *child, long request, case PTRACE_SETSIGMASK: { sigset_t new_set; + /* A user worker only ever takes SIGKILL and SIGSTOP. */ + if (child->flags & PF_USER_WORKER) { + ret = -EPERM; + break; + } + if (addr != sizeof(sigset_t)) { ret = -EINVAL; break; diff --git a/kernel/rcu/srcutiny.c b/kernel/rcu/srcutiny.c index 24aa302b3269..c38e5933a5d6 100644 --- a/kernel/rcu/srcutiny.c +++ b/kernel/rcu/srcutiny.c @@ -9,7 +9,6 @@ */ #include -#include #include #include #include @@ -38,7 +37,6 @@ static int init_srcu_struct_fields(struct srcu_struct *ssp) ssp->srcu_idx_max = 0; INIT_WORK(&ssp->srcu_work, srcu_drive_gp); INIT_LIST_HEAD(&ssp->srcu_work.entry); - init_irq_work(&ssp->srcu_irq_work, srcu_tiny_irq_work); return 0; } @@ -82,7 +80,6 @@ EXPORT_SYMBOL_GPL(init_srcu_struct); void cleanup_srcu_struct(struct srcu_struct *ssp) { WARN_ON(ssp->srcu_lock_nesting[0] || ssp->srcu_lock_nesting[1]); - irq_work_sync(&ssp->srcu_irq_work); flush_work(&ssp->srcu_work); WARN_ON(ssp->srcu_gp_running); WARN_ON(ssp->srcu_gp_waiting); @@ -159,20 +156,6 @@ void srcu_drive_gp(struct work_struct *wp) } EXPORT_SYMBOL_GPL(srcu_drive_gp); -/* - * Use an irq_work to defer schedule_work() to avoid acquiring the workqueue - * pool->lock while the caller might hold scheduler locks, causing lockdep - * splats due to workqueue_init() doing a wakeup. - */ -void srcu_tiny_irq_work(struct irq_work *irq_work) -{ - struct srcu_struct *ssp; - - ssp = container_of(irq_work, struct srcu_struct, srcu_irq_work); - schedule_work(&ssp->srcu_work); -} -EXPORT_SYMBOL_GPL(srcu_tiny_irq_work); - static void srcu_gp_start_if_needed(struct srcu_struct *ssp) { unsigned long cookie; @@ -183,7 +166,7 @@ static void srcu_gp_start_if_needed(struct srcu_struct *ssp) WRITE_ONCE(ssp->srcu_idx_max, cookie); if (!READ_ONCE(ssp->srcu_gp_running)) { if (likely(srcu_init_done)) - irq_work_queue(&ssp->srcu_irq_work); + schedule_work(&ssp->srcu_work); else if (list_empty(&ssp->srcu_work.entry)) list_add(&ssp->srcu_work.entry, &srcu_boot_list); } diff --git a/kernel/signal.c b/kernel/signal.c index cfc419613190..396ec7d89f8a 100644 --- a/kernel/signal.c +++ b/kernel/signal.c @@ -2985,6 +2985,10 @@ static void retarget_shared_pending(struct task_struct *tsk, sigset_t *which) sigset_t retarget; struct task_struct *t; + /* Nobody dequeues them in a dying group, see get_signal(). */ + if (tsk->signal->flags & SIGNAL_GROUP_EXIT) + return; + sigandsets(&retarget, &tsk->signal->shared_pending.signal, which); if (sigisemptyset(&retarget)) return; diff --git a/kernel/trace/ftrace.c b/kernel/trace/ftrace.c index ad61bd1cba55..8e19da45cf5a 100644 --- a/kernel/trace/ftrace.c +++ b/kernel/trace/ftrace.c @@ -3807,9 +3807,9 @@ subsys_initcall(ftrace_check_for_weak_functions); static int print_rec(struct seq_file *m, unsigned long ip) { - unsigned long offset; + unsigned long offset = 0; char str[KSYM_SYMBOL_LEN]; - char *modname; + char *modname = NULL; const char *ret; ret = kallsyms_lookup(ip, NULL, &offset, &modname, str); diff --git a/kernel/trace/trace.c b/kernel/trace/trace.c index 9ab20e1a936f..e79d2e7f0fe1 100644 --- a/kernel/trace/trace.c +++ b/kernel/trace/trace.c @@ -10040,6 +10040,12 @@ static struct notifier_block trace_module_nb = { static __init void tracer_init_tracefs_work_func(struct work_struct *work) { + /* + * Do not attempt to initialize tracefs if tracing was disabled or + * failed to allocate + */ + if (tracing_disabled) + return; event_trace_init(); diff --git a/kernel/trace/trace_events_filter.c b/kernel/trace/trace_events_filter.c index b6fc33ddb228..ad80649b8a65 100644 --- a/kernel/trace/trace_events_filter.c +++ b/kernel/trace/trace_events_filter.c @@ -482,18 +482,17 @@ predicate_parse(const char *str, int nr_parens, int nr_preds, int len; int N = 0; int i; - - nr_preds += 2; /* For TRUE and FALSE */ + int nr_preds_alloc = nr_preds + 2; /* For TRUE and FALSE */ op_stack = kmalloc_array(nr_parens, sizeof(*op_stack), GFP_KERNEL); if (!op_stack) return ERR_PTR(-ENOMEM); - prog_stack = kcalloc(nr_preds, sizeof(*prog_stack), GFP_KERNEL); + prog_stack = kcalloc(nr_preds_alloc, sizeof(*prog_stack), GFP_KERNEL); if (!prog_stack) { parse_error(pe, -ENOMEM, 0); goto out_free; } - inverts = kmalloc_array(nr_preds, sizeof(*inverts), GFP_KERNEL); + inverts = kmalloc_array(nr_preds_alloc, sizeof(*inverts), GFP_KERNEL); if (!inverts) { parse_error(pe, -ENOMEM, 0); goto out_free; @@ -512,7 +511,7 @@ predicate_parse(const char *str, int nr_parens, int nr_preds, switch (*next) { case '(': /* #2 */ - if (top - op_stack > nr_parens) { + if (top - op_stack >= nr_parens - 1) { ret = -EINVAL; goto out_free; } @@ -596,6 +595,12 @@ predicate_parse(const char *str, int nr_parens, int nr_preds, top--; /* #12 */ } } + /* Only the inner loop can complete a non-empty expression. */ + if (N) { + ret = -EINVAL; + parse_error(pe, FILT_ERR_INVALID_FILTER, ptr - str); + goto out_free; + } out: if (top != op_stack) { /* Too many '(' */ diff --git a/kernel/trace/trace_events_hist.c b/kernel/trace/trace_events_hist.c index 60ec8c7c6b67..7ae8160bb8f9 100644 --- a/kernel/trace/trace_events_hist.c +++ b/kernel/trace/trace_events_hist.c @@ -283,7 +283,7 @@ static u64 hist_field_log2(struct hist_field *hist_field, u64 val = hist_fn_call(operand, elt, buffer, rbe, event); - return (u64) ilog2(roundup_pow_of_two(val)); + return order_base_2(val); } static u64 hist_field_bucket(struct hist_field *hist_field, @@ -460,7 +460,7 @@ static u64 hist_field_unary_minus(struct hist_field *hist_field, { \ type *addr = (type *)(event + hist_field->field->offset); \ \ - return (u64)(unsigned long)*addr; \ + return (u64)*addr; \ } DEFINE_HIST_FIELD_FN(s64); @@ -542,7 +542,7 @@ struct field_var { }; struct field_var_hist { - struct hist_trigger_data *hist_data; + struct trace_event_file *file; char *cmd; }; @@ -866,7 +866,7 @@ static u64 hist_field_timestamp(struct hist_field *hist_field, struct hist_trigger_data *hist_data = hist_field->hist_data; struct trace_array *tr = hist_data->event_file->tr; - u64 ts = ring_buffer_event_time_stamp(buffer, rbe); + u64 ts = tracing_event_time_stamp(buffer, rbe); if (hist_data->attrs->ts_in_usecs && trace_clock_in_ns(tr)) ts = ns2usecs(ts); @@ -2098,7 +2098,8 @@ static int init_var_ref(struct hist_field *ref_field, ref_field->size = var_field->size; ref_field->is_signed = var_field->is_signed; ref_field->flags |= var_field->flags & - (HIST_FIELD_FL_TIMESTAMP | HIST_FIELD_FL_TIMESTAMP_USECS); + (HIST_FIELD_FL_TIMESTAMP | HIST_FIELD_FL_TIMESTAMP_USECS | + HIST_FIELD_FL_STACKTRACE); if (system) { ref_field->system = kstrdup(system, GFP_KERNEL); @@ -3076,8 +3077,8 @@ create_field_var_hist(struct hist_trigger_data *target_hist_data, return ERR_PTR(-ENOMEM); } - /* Save the compatible histogram information */ - var_hist->hist_data = hist_data; + /* Needed to remove the histogram when the target goes away */ + var_hist->file = file; /* Create the new histogram with our variable */ ret = event_hist_trigger_parse(&trigger_hist_cmd, file, @@ -3092,20 +3093,18 @@ create_field_var_hist(struct hist_trigger_data *target_hist_data, kfree(cmd); + n = target_hist_data->n_field_var_hists; + target_hist_data->field_var_hists[n] = var_hist; + target_hist_data->n_field_var_hists++; + /* If we can't find the variable, something went wrong */ event_var = find_synthetic_field_var(target_hist_data, subsys_name, event_name, field_name); if (IS_ERR_OR_NULL(event_var)) { - kfree(var_hist->cmd); - kfree(var_hist); hist_err(tr, HIST_ERR_SYNTH_VAR_NOT_FOUND, errpos(field_name)); return ERR_PTR(-EINVAL); } - n = target_hist_data->n_field_var_hists; - target_hist_data->field_var_hists[n] = var_hist; - target_hist_data->n_field_var_hists++; - return event_var; } @@ -3286,6 +3285,7 @@ static struct field_var *create_field_var(struct hist_trigger_data *hist_data, ret = -ENOMEM; goto err; } + var->flags |= val->flags & HIST_FIELD_FL_STACKTRACE; field_var->var = var; field_var->val = val; @@ -3904,7 +3904,10 @@ static int check_synth_field(struct synth_event *event, && field->is_dynamic) return 0; - if (strstr(hist_field->type, "long[") && field->is_stack) + if (!!(hist_field->flags & HIST_FIELD_FL_STACKTRACE) != field->is_stack) + return -EINVAL; + + if (field->is_stack) return 0; if (strcmp(field->type, hist_field->type) != 0) { @@ -4443,7 +4446,8 @@ static int create_var_field(struct hist_trigger_data *hist_data, if (!ret && hist_data->fields[val_idx]->flags & HIST_FIELD_FL_EXECNAME) update_var_execname(hist_data->fields[val_idx]); - if (!ret && hist_data->fields[val_idx]->flags & + if (!ret && !(hist_data->fields[val_idx]->flags & HIST_FIELD_FL_ALIAS) && + hist_data->fields[val_idx]->flags & (HIST_FIELD_FL_STRING | HIST_FIELD_FL_STACKTRACE)) hist_data->fields[val_idx]->var_str_idx = hist_data->n_var_str++; @@ -5185,7 +5189,8 @@ static void hist_trigger_elt_update(struct hist_trigger_data *hist_data, if (hist_field->flags & HIST_FIELD_FL_VAR) { var_idx = hist_field->var.idx; - if (hist_field->flags & + if (!(hist_field->flags & HIST_FIELD_FL_ALIAS) && + hist_field->flags & (HIST_FIELD_FL_STRING | HIST_FIELD_FL_STACKTRACE)) { unsigned int str_start, var_str_idx, idx; char *str, *val_str; @@ -6293,7 +6298,7 @@ static void unregister_field_var_hists(struct hist_trigger_data *hist_data) int ret; for (i = 0; i < hist_data->n_field_var_hists; i++) { - file = hist_data->field_var_hists[i]->hist_data->event_file; + file = hist_data->field_var_hists[i]->file; cmd = hist_data->field_var_hists[i]->cmd; ret = event_hist_trigger_parse(&trigger_hist_cmd, file, "!hist", "hist", cmd); @@ -6724,7 +6729,6 @@ static void hist_unreg_all(struct trace_event_file *file) struct event_trigger_data *test, *n; struct hist_trigger_data *hist_data; struct synth_event *se; - const char *se_name; lockdep_assert_held(&event_mutex); @@ -6737,8 +6741,7 @@ static void hist_unreg_all(struct trace_event_file *file) list_del_rcu(&test->list); trace_event_trigger_enable_disable(file, 0); - se_name = trace_event_name(file->event_call); - se = find_synth_event(se_name); + se = find_synth_event_call(file->event_call); if (se) se->ref--; @@ -6762,7 +6765,6 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops, struct hist_trigger_data *hist_data; char *param, *filter, *p, *start; struct synth_event *se; - const char *se_name; bool remove; int ret = 0; @@ -6856,8 +6858,7 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops, } event_trigger_unregister(cmd_ops, file, glob+1, trigger_data); - se_name = trace_event_name(file->event_call); - se = find_synth_event(se_name); + se = find_synth_event_call(file->event_call); if (se) se->ref--; ret = 0; @@ -6892,8 +6893,7 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops, if (ret) goto out_unreg; - se_name = trace_event_name(file->event_call); - se = find_synth_event(se_name); + se = find_synth_event_call(file->event_call); if (se) se->ref++; out: @@ -6912,6 +6912,8 @@ static int event_hist_trigger_parse(struct event_command *cmd_ops, trigger_data_free(trigger_data); + unregister_field_var_hists(hist_data); + destroy_hist_data(hist_data); goto out; } diff --git a/kernel/trace/trace_events_synth.c b/kernel/trace/trace_events_synth.c index 02e2440da829..106faa41e258 100644 --- a/kernel/trace/trace_events_synth.c +++ b/kernel/trace/trace_events_synth.c @@ -857,6 +857,16 @@ struct synth_event *find_synth_event(const char *name) return NULL; } +struct synth_event *find_synth_event_call(struct trace_event_call *call) +{ + struct synth_event *se; + const char *name; + + name = trace_event_name(call); + se = find_synth_event(name); + return se && &se->call == call ? se : NULL; +} + static struct trace_event_fields synth_event_fields_array[] = { { .type = TRACE_FUNCTION_TYPE, .define_fields = synth_event_define_fields }, diff --git a/kernel/trace/trace_probe.c b/kernel/trace/trace_probe.c index a43d974fbccc..d6bde7a03b57 100644 --- a/kernel/trace/trace_probe.c +++ b/kernel/trace/trace_probe.c @@ -1663,7 +1663,6 @@ const char **traceprobe_expand_meta_args(int argc, const char *argv[], int *new_argc, char *buf, int bufsize, struct traceprobe_parse_context *ctx) { - const struct btf_param *params = NULL; int i, j, n, used, ret, args_idx = -1; const char **new_argv = NULL; @@ -1681,7 +1680,7 @@ const char **traceprobe_expand_meta_args(int argc, const char *argv[], if (args_idx != -1) { /* $arg* requires BTF info */ trace_probe_log_err(0, NOSUP_BTFARG); - return (const char **)params; + return ERR_PTR(-EOPNOTSUPP); } *new_argc = argc; return NULL; diff --git a/kernel/trace/trace_synth.h b/kernel/trace/trace_synth.h index 43f6fb6078db..d4ebf01b7049 100644 --- a/kernel/trace/trace_synth.h +++ b/kernel/trace/trace_synth.h @@ -37,5 +37,6 @@ struct synth_event { }; extern struct synth_event *find_synth_event(const char *name); +extern struct synth_event *find_synth_event_call(struct trace_event_call *call); #endif /* __TRACE_SYNTH_H */ diff --git a/kernel/watch_queue.c b/kernel/watch_queue.c index 17254597accd..d5a76a5efd20 100644 --- a/kernel/watch_queue.c +++ b/kernel/watch_queue.c @@ -101,19 +101,20 @@ static bool post_one_notification(struct watch_queue *wqueue, struct pipe_inode_info *pipe = wqueue->pipe; struct pipe_buffer *buf; struct page *page; - unsigned int head, tail, mask, note, offset, len; + unsigned int head, tail, note, offset, len, nr_notes; bool done = false; spin_lock_irq(&pipe->rd_wait.lock); - mask = pipe->ring_size - 1; head = pipe->head; tail = pipe->tail; if (pipe_full(head, tail, pipe->ring_size)) goto lost; - note = find_first_bit(wqueue->notes_bitmap, wqueue->nr_notes); - if (note >= wqueue->nr_notes) + /* Pairs with the smp_store_release() in watch_queue_set_size(). */ + nr_notes = smp_load_acquire(&wqueue->nr_notes); + note = find_first_bit(wqueue->notes_bitmap, nr_notes); + if (note >= nr_notes) goto lost; page = wqueue->notes[note / WATCH_QUEUE_NOTES_PER_PAGE]; @@ -124,7 +125,7 @@ static bool post_one_notification(struct watch_queue *wqueue, memcpy(p + offset, n, len); kunmap_atomic(p); - buf = &pipe->bufs[head & mask]; + buf = pipe_buf(pipe, head); buf->page = page; buf->private = (unsigned long)wqueue; buf->ops = &watch_queue_pipe_buf_ops; @@ -147,7 +148,7 @@ static bool post_one_notification(struct watch_queue *wqueue, return done; lost: - buf = &pipe->bufs[(head - 1) & mask]; + buf = pipe_buf(pipe, head - 1); buf->flags |= PIPE_BUF_FLAG_LOSS; goto out; } @@ -298,7 +299,8 @@ long watch_queue_set_size(struct pipe_inode_info *pipe, unsigned int nr_notes) wqueue->notes = pages; wqueue->notes_bitmap = bitmap; wqueue->nr_pages = nr_pages; - wqueue->nr_notes = nr_notes; + /* Pairs with the smp_load_acquire() in post_one_notification(). */ + smp_store_release(&wqueue->nr_notes, nr_notes); return 0; error_p: diff --git a/mm/huge_memory.c b/mm/huge_memory.c index 28502c539cab..4aa032e96fa6 100644 --- a/mm/huge_memory.c +++ b/mm/huge_memory.c @@ -2746,6 +2746,8 @@ int split_huge_page_to_list(struct page *page, struct list_head *list) goto out; } + mapping_set_update(&xas, mapping); + xas_split_alloc(&xas, folio, folio_order(folio), gfp); if (xas_error(&xas)) { ret = xas_error(&xas); diff --git a/mm/khugepaged.c b/mm/khugepaged.c index ead6b0001524..42330aeb7638 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -19,6 +19,7 @@ #include #include #include +#include #include #include @@ -1800,6 +1801,8 @@ static int collapse_file(struct mm_struct *mm, unsigned long addr, if (result != SCAN_SUCCEED) goto out; + mapping_set_update(&xas, mapping); + __folio_set_locked(new_folio); if (is_shmem) __folio_set_swapbacked(new_folio); diff --git a/net/bluetooth/l2cap_core.c b/net/bluetooth/l2cap_core.c index c2bd2b25a809..94fddc456ac5 100644 --- a/net/bluetooth/l2cap_core.c +++ b/net/bluetooth/l2cap_core.c @@ -625,6 +625,10 @@ void __l2cap_chan_add(struct l2cap_conn *conn, struct l2cap_chan *chan) BT_DBG("conn %p, psm 0x%2.2x, dcid 0x%4.4x", conn, __le16_to_cpu(chan->psm), chan->dcid); + /* Caller must ensure l2cap_chan is linked to l2cap_conn only once */ + if (WARN_ON_ONCE(chan->conn || test_bit(FLAG_DEL, &chan->flags))) + return; + conn->disc_reason = HCI_ERROR_REMOTE_USER_TERM; chan->conn = l2cap_conn_get(conn); @@ -7332,7 +7336,8 @@ int l2cap_chan_connect(struct l2cap_chan *chan, __le16 psm, u16 cid, } } - if (cid && __l2cap_get_chan_by_dcid(conn, cid)) { + if ((cid && __l2cap_get_chan_by_dcid(conn, cid)) || chan->conn || + test_bit(FLAG_DEL, &chan->flags)) { hci_conn_drop(hcon); err = -EBUSY; goto chan_unlock; diff --git a/net/bluetooth/l2cap_sock.c b/net/bluetooth/l2cap_sock.c index c0da2b1168a1..7d2e59e97c2c 100644 --- a/net/bluetooth/l2cap_sock.c +++ b/net/bluetooth/l2cap_sock.c @@ -111,6 +111,7 @@ static int l2cap_sock_bind(struct socket *sock, struct sockaddr *addr, int alen) return -EINVAL; } + l2cap_chan_lock(chan); lock_sock(sk); if (sk->sk_state != BT_OPEN) { @@ -176,6 +177,7 @@ static int l2cap_sock_bind(struct socket *sock, struct sockaddr *addr, int alen) done: release_sock(sk); + l2cap_chan_unlock(chan); return err; } diff --git a/net/bridge/br_multicast.c b/net/bridge/br_multicast.c index 821112260a91..f7c8602bec33 100644 --- a/net/bridge/br_multicast.c +++ b/net/bridge/br_multicast.c @@ -812,7 +812,11 @@ static void __br_multicast_del_pg(struct net_bridge_mdb_entry *mp, struct hlist_node *tmp; rcu_assign_pointer(*pp, pg->next); - hlist_del_init(&pg->mglist); + /* Keep ->next (held under multicast_lock, freed later by the GC work): + * a port->mglist teardown walk may have latched this node as its next, + * and deleting other groups of the same port must not truncate it. + */ + hlist_del_init_rcu(&pg->mglist); br_multicast_eht_clean_sets(pg); hlist_for_each_entry_safe(ent, tmp, &pg->src_list, node) br_multicast_del_group_src(ent, false); @@ -847,6 +851,13 @@ static void br_multicast_find_del_pg(struct net_bridge *br, struct net_bridge_mdb_entry *mp; struct net_bridge_port_group *p; + /* A teardown walk over port->mglist can reach a group that an earlier + * iteration already deleted as a side effect. It is off mp->ports by + * now, so skip it instead of falling through to the WARN_ON() below. + */ + if (hlist_unhashed(&pg->mglist)) + return; + mp = br_mdb_ip_get(br, &pg->key.addr); if (WARN_ON(!mp)) return; diff --git a/net/bridge/br_netfilter_ipv6.c b/net/bridge/br_netfilter_ipv6.c index ad268bd19d5b..5f21fab5c05a 100644 --- a/net/bridge/br_netfilter_ipv6.c +++ b/net/bridge/br_netfilter_ipv6.c @@ -58,7 +58,7 @@ int br_validate_ipv6(struct net *net, struct sk_buff *skb) if (hdr->version != 6) goto inhdr_error; - pkt_len = ntohs(hdr->payload_len); + pkt_len = ipv6_payload_len(skb, hdr); if (hdr->nexthdr == NEXTHDR_HOP && nf_ip6_check_hbh_len(skb, &pkt_len)) goto drop; diff --git a/net/bridge/netfilter/nf_conntrack_bridge.c b/net/bridge/netfilter/nf_conntrack_bridge.c index 3b75d5b21617..381638ee9ac5 100644 --- a/net/bridge/netfilter/nf_conntrack_bridge.c +++ b/net/bridge/netfilter/nf_conntrack_bridge.c @@ -234,7 +234,7 @@ static int nf_ct_br_ipv6_check(const struct sk_buff *skb) if (hdr->version != 6) return -1; - len = ntohs(hdr->payload_len) + sizeof(struct ipv6hdr) + nhoff; + len = ipv6_payload_len(skb, hdr) + sizeof(struct ipv6hdr) + nhoff; if (skb->len < len) return -1; @@ -274,7 +274,7 @@ static unsigned int nf_ct_bridge_pre(void *priv, struct sk_buff *skb, if (!pskb_may_pull(skb, sizeof(struct ipv6hdr))) return NF_ACCEPT; - len = sizeof(struct ipv6hdr) + ntohs(ipv6_hdr(skb)->payload_len); + len = sizeof(struct ipv6hdr) + skb_ipv6_payload_len(skb); if (pskb_trim_rcsum(skb, len)) return NF_ACCEPT; diff --git a/net/core/page_pool.c b/net/core/page_pool.c index c8e92a12af35..2049428b2e70 100644 --- a/net/core/page_pool.c +++ b/net/core/page_pool.c @@ -751,7 +751,8 @@ struct page *page_pool_alloc_frag(struct page_pool *pool, size > max_size)) return NULL; - size = ALIGN(size, dma_get_cache_alignment()); + size = ALIGN(size, max_t(unsigned int, dma_get_cache_alignment(), + __alignof__(struct skb_shared_info))); *offset = pool->frag_offset; if (page && *offset + size > max_size) { diff --git a/net/ipv4/fib_semantics.c b/net/ipv4/fib_semantics.c index 454e2544820a..60202e3de3ca 100644 --- a/net/ipv4/fib_semantics.c +++ b/net/ipv4/fib_semantics.c @@ -857,7 +857,7 @@ static void fib_rebalance(struct fib_info *fi) change_nexthops(fi) { int upper_bound; - if (nexthop_nh->fib_nh_flags & RTNH_F_DEAD) { + if (!total || nexthop_nh->fib_nh_flags & RTNH_F_DEAD) { upper_bound = -1; } else if (ip_ignore_linkdown(nexthop_nh->fib_nh_dev) && nexthop_nh->fib_nh_flags & RTNH_F_LINKDOWN) { diff --git a/net/ipv4/tcp.c b/net/ipv4/tcp.c index f05d1699683f..a5dbbc65a61c 100644 --- a/net/ipv4/tcp.c +++ b/net/ipv4/tcp.c @@ -4089,9 +4089,11 @@ int do_tcp_getsockopt(struct sock *sk, int level, if (copy_from_sockptr(&len, optlen, sizeof(int))) return -EFAULT; - ca_ops = icsk->icsk_ca_ops; + rcu_read_lock(); + ca_ops = READ_ONCE(icsk->icsk_ca_ops); if (ca_ops && ca_ops->get_info) sz = ca_ops->get_info(sk, ~0U, &attr, &info); + rcu_read_unlock(); len = min_t(unsigned int, len, sz); if (copy_to_sockptr(optlen, &len, sizeof(int))) @@ -4104,16 +4106,24 @@ int do_tcp_getsockopt(struct sock *sk, int level, val = !inet_csk_in_pingpong_mode(sk); break; - case TCP_CONGESTION: + case TCP_CONGESTION: { + char ca_name[TCP_CA_NAME_MAX] = {}; + if (copy_from_sockptr(&len, optlen, sizeof(int))) return -EFAULT; len = min_t(unsigned int, len, TCP_CA_NAME_MAX); if (copy_to_sockptr(optlen, &len, sizeof(int))) return -EFAULT; - if (copy_to_sockptr(optval, icsk->icsk_ca_ops->name, len)) + + rcu_read_lock(); + memcpy(ca_name, READ_ONCE(icsk->icsk_ca_ops)->name, + sizeof(ca_name)); + rcu_read_unlock(); + + if (copy_to_sockptr(optval, ca_name, len)) return -EFAULT; return 0; - + } case TCP_ULP: if (copy_from_sockptr(&len, optlen, sizeof(int))) return -EFAULT; diff --git a/net/ipv4/tcp_cong.c b/net/ipv4/tcp_cong.c index 95dbb2799be4..2959243e98f0 100644 --- a/net/ipv4/tcp_cong.c +++ b/net/ipv4/tcp_cong.c @@ -225,7 +225,7 @@ void tcp_assign_congestion_control(struct sock *sk) ca = rcu_dereference(net->ipv4.tcp_congestion_control); if (unlikely(!bpf_try_module_get(ca, ca->owner))) ca = &tcp_reno; - icsk->icsk_ca_ops = ca; + WRITE_ONCE(icsk->icsk_ca_ops, ca); rcu_read_unlock(); memset(icsk->icsk_ca_priv, 0, sizeof(icsk->icsk_ca_priv)); @@ -255,7 +255,7 @@ static void tcp_reinit_congestion_control(struct sock *sk, struct inet_connection_sock *icsk = inet_csk(sk); tcp_cleanup_congestion_control(sk); - icsk->icsk_ca_ops = ca; + WRITE_ONCE(icsk->icsk_ca_ops, ca); icsk->icsk_ca_setsockopt = 1; memset(icsk->icsk_ca_priv, 0, sizeof(icsk->icsk_ca_priv)); diff --git a/net/ipv4/tcp_dctcp.c b/net/ipv4/tcp_dctcp.c index 8ad62713b0ba..54736d63e1d2 100644 --- a/net/ipv4/tcp_dctcp.c +++ b/net/ipv4/tcp_dctcp.c @@ -111,7 +111,7 @@ __bpf_kfunc static void dctcp_init(struct sock *sk) /* No ECN support? Fall back to Reno. Also need to clear * ECT from sk since it is set during 3WHS for DCTCP. */ - inet_csk(sk)->icsk_ca_ops = &dctcp_reno; + WRITE_ONCE(inet_csk(sk)->icsk_ca_ops, &dctcp_reno); INET_ECN_dontxmit(sk); } @@ -224,7 +224,7 @@ static size_t dctcp_get_info(struct sock *sk, u32 ext, int *attr, if (ext & (1 << (INET_DIAG_DCTCPINFO - 1)) || ext & (1 << (INET_DIAG_VEGASINFO - 1))) { memset(&info->dctcp, 0, sizeof(info->dctcp)); - if (inet_csk(sk)->icsk_ca_ops != &dctcp_reno) { + if (READ_ONCE(inet_csk(sk)->icsk_ca_ops) != &dctcp_reno) { info->dctcp.dctcp_enabled = 1; info->dctcp.dctcp_ce_state = (u16) ca->ce_state; info->dctcp.dctcp_alpha = ca->dctcp_alpha; diff --git a/net/ipv4/tcp_minisocks.c b/net/ipv4/tcp_minisocks.c index b89fd89c3c95..9c7103b740f5 100644 --- a/net/ipv4/tcp_minisocks.c +++ b/net/ipv4/tcp_minisocks.c @@ -450,7 +450,7 @@ void tcp_ca_openreq_child(struct sock *sk, const struct dst_entry *dst) ca = tcp_ca_find_key(ca_key); if (likely(ca && bpf_try_module_get(ca, ca->owner))) { icsk->icsk_ca_dst_locked = tcp_ca_dst_locked(dst); - icsk->icsk_ca_ops = ca; + WRITE_ONCE(icsk->icsk_ca_ops, ca); ca_got_dst = true; } rcu_read_unlock(); diff --git a/net/ipv4/tcp_output.c b/net/ipv4/tcp_output.c index 088f28fa3b75..327e38c81fef 100644 --- a/net/ipv4/tcp_output.c +++ b/net/ipv4/tcp_output.c @@ -3782,7 +3782,7 @@ static void tcp_ca_dst_init(struct sock *sk, const struct dst_entry *dst) if (likely(ca && bpf_try_module_get(ca, ca->owner))) { bpf_module_put(icsk->icsk_ca_ops, icsk->icsk_ca_ops->owner); icsk->icsk_ca_dst_locked = tcp_ca_dst_locked(dst); - icsk->icsk_ca_ops = ca; + WRITE_ONCE(icsk->icsk_ca_ops, ca); } rcu_read_unlock(); } diff --git a/net/ipv4/udp.c b/net/ipv4/udp.c index 86a1b2c9e71f..8ceaeef5c184 100644 --- a/net/ipv4/udp.c +++ b/net/ipv4/udp.c @@ -870,7 +870,16 @@ int __udp4_lib_err(struct sk_buff *skb, u32 info, struct udp_table *udptable) int udp_err(struct sk_buff *skb, u32 info) { - return __udp4_lib_err(skb, info, dev_net(skb->dev)->ipv4.udp_table); + const int type = icmp_hdr(skb)->type; + const int code = icmp_hdr(skb)->code; + struct net *net = dev_net(skb->dev); + + if (type == ICMP_DEST_UNREACH && code == ICMP_FRAG_NEEDED) + ipv4_update_pmtu(skb, net, info, 0, IPPROTO_UDP); + else if (type == ICMP_REDIRECT) + ipv4_redirect(skb, net, 0, IPPROTO_UDP); + + return __udp4_lib_err(skb, info, net->ipv4.udp_table); } /* diff --git a/net/ipv6/calipso.c b/net/ipv6/calipso.c index 72079ef2959b..f23bccb0ee5a 100644 --- a/net/ipv6/calipso.c +++ b/net/ipv6/calipso.c @@ -1322,7 +1322,7 @@ static int calipso_skbuff_setattr(struct sk_buff *skb, struct ipv6hdr *ip6_hdr; struct ipv6_opt_hdr *hop; unsigned char buf[CALIPSO_MAX_BUFFER]; - int len_delta, new_end, pad, payload; + int len_delta, new_end, pad; unsigned int start, end; ip6_hdr = ipv6_hdr(skb); @@ -1361,8 +1361,8 @@ static int calipso_skbuff_setattr(struct sk_buff *skb, sizeof(*ip6_hdr) + start); skb_reset_network_header(skb); ip6_hdr = ipv6_hdr(skb); - payload = ntohs(ip6_hdr->payload_len); - ip6_hdr->payload_len = htons(payload + len_delta); + /* 0 above IPV6_MAXPLEN (BIG TCP), the real length otherwise */ + ipv6_set_payload_len(ip6_hdr, skb->len - sizeof(*ip6_hdr)); } hop = (struct ipv6_opt_hdr *)(ip6_hdr + 1); diff --git a/net/ipv6/ip6_input.c b/net/ipv6/ip6_input.c index 9f8f17b3934f..c5ae346e29cf 100644 --- a/net/ipv6/ip6_input.c +++ b/net/ipv6/ip6_input.c @@ -261,7 +261,7 @@ static struct sk_buff *ip6_rcv_core(struct sk_buff *skb, struct net_device *dev, skb->transport_header = skb->network_header + sizeof(*hdr); IP6CB(skb)->nhoff = offsetof(struct ipv6hdr, nexthdr); - pkt_len = ntohs(hdr->payload_len); + pkt_len = ipv6_payload_len(skb, hdr); /* pkt_len may be zero if Jumbo payload option is present */ if (pkt_len || hdr->nexthdr != NEXTHDR_HOP) { diff --git a/net/ipv6/ip6_offload.c b/net/ipv6/ip6_offload.c index 84b17eeaa57c..a4e09b1305b2 100644 --- a/net/ipv6/ip6_offload.c +++ b/net/ipv6/ip6_offload.c @@ -387,12 +387,11 @@ INDIRECT_CALLABLE_SCOPE int ipv6_gro_complete(struct sk_buff *skb, int nhoff) hop_jumbo->jumbo_payload_len = htonl(payload_len + hoplen); iph->nexthdr = NEXTHDR_HOP; - iph->payload_len = 0; - } else { - iph = (struct ipv6hdr *)(skb->data + nhoff); - iph->payload_len = htons(payload_len); } + iph = (struct ipv6hdr *)(skb->data + nhoff); + ipv6_set_payload_len(iph, payload_len); + nhoff += sizeof(*iph) + ipv6_exthdrs_len(iph, &ops); if (WARN_ON(!ops || !ops->callbacks.gro_complete)) goto out; diff --git a/net/ipv6/output_core.c b/net/ipv6/output_core.c index b5205311f372..49c48be2713f 100644 --- a/net/ipv6/output_core.c +++ b/net/ipv6/output_core.c @@ -123,12 +123,7 @@ EXPORT_SYMBOL(ip6_dst_hoplimit); int __ip6_local_out(struct net *net, struct sock *sk, struct sk_buff *skb) { - int len; - - len = skb->len - sizeof(struct ipv6hdr); - if (len > IPV6_MAXPLEN) - len = 0; - ipv6_hdr(skb)->payload_len = htons(len); + ipv6_set_payload_len(ipv6_hdr(skb), skb->len - sizeof(struct ipv6hdr)); IP6CB(skb)->nhoff = offsetof(struct ipv6hdr, nexthdr); /* if egress device is enslaved to an L3 master device pass the diff --git a/net/iucv/af_iucv.c b/net/iucv/af_iucv.c index b9ce76ccd054..c56bf6d96ac0 100644 --- a/net/iucv/af_iucv.c +++ b/net/iucv/af_iucv.c @@ -207,12 +207,6 @@ static int afiucv_hs_send(struct iucv_message *imsg, struct sock *sock, phs_hdr->flags = flags; if (flags == AF_IUCV_FLAG_SYN) phs_hdr->window = iucv->msglimit; - else if ((flags == AF_IUCV_FLAG_WIN) || !flags) { - confirm_recv = atomic_read(&iucv->msg_recv); - phs_hdr->window = confirm_recv; - if (confirm_recv) - phs_hdr->flags = phs_hdr->flags | AF_IUCV_FLAG_WIN; - } memcpy(phs_hdr->destUserID, iucv->dst_user_id, 8); memcpy(phs_hdr->destAppName, iucv->dst_name, 8); memcpy(phs_hdr->srcUserID, iucv->src_user_id, 8); @@ -247,13 +241,22 @@ static int afiucv_hs_send(struct iucv_message *imsg, struct sock *sock, } skb->protocol = cpu_to_be16(ETH_P_AF_IUCV); + /* Claim the receive credit here, not while building the header: every + * way this frame can be dropped has now been ruled out, so the window + * is zeroed only for as long as the transmit itself takes. + */ + if (flags == AF_IUCV_FLAG_WIN || !flags) { + confirm_recv = atomic_xchg(&iucv->msg_recv, 0); + phs_hdr->window = confirm_recv; + if (confirm_recv) + phs_hdr->flags = phs_hdr->flags | AF_IUCV_FLAG_WIN; + } + atomic_inc(&iucv->skbs_in_xmit); err = dev_queue_xmit(skb); if (net_xmit_eval(err)) { atomic_dec(&iucv->skbs_in_xmit); - } else { - atomic_sub(confirm_recv, &iucv->msg_recv); - WARN_ON(atomic_read(&iucv->msg_recv) < 0); + atomic_add(confirm_recv, &iucv->msg_recv); } return net_xmit_eval(err); @@ -1239,6 +1242,7 @@ static int iucv_sock_recvmsg(struct socket *sock, struct msghdr *msg, struct iucv_sock *iucv = iucv_sk(sk); unsigned int copied, rlen; struct sk_buff *skb, *rskb, *cskb; + bool send_win = false; int err = 0; u32 offset; @@ -1329,16 +1333,20 @@ static int iucv_sock_recvmsg(struct socket *sock, struct msghdr *msg, if (skb_queue_empty(&iucv->backlog_skb_q)) { if (!list_empty(&iucv->message_q.list)) iucv_process_message_q(sk); - if (atomic_read(&iucv->msg_recv) >= - iucv->msglimit / 2) { - err = iucv_send_ctrl(sk, AF_IUCV_FLAG_WIN); - if (err) { - sk->sk_state = IUCV_DISCONN; - sk->sk_state_change(sk); - } - } + if (iucv->transport == AF_IUCV_TRANS_HIPER && + atomic_read(&iucv->msg_recv) >= + iucv->msglimit / 2) + send_win = true; } spin_unlock_bh(&iucv->message_q.lock); + + if (send_win) { + err = iucv_send_ctrl(sk, AF_IUCV_FLAG_WIN); + if (err) { + sk->sk_state = IUCV_DISCONN; + sk->sk_state_change(sk); + } + } } done: diff --git a/net/mptcp/protocol.c b/net/mptcp/protocol.c index 3e8e5aae4dc4..7b08b5899ef3 100644 --- a/net/mptcp/protocol.c +++ b/net/mptcp/protocol.c @@ -923,13 +923,6 @@ static bool __mptcp_finish_join(struct mptcp_sock *msk, struct sock *ssk) if (sk->sk_state != TCP_ESTABLISHED) return false; - /* The caller possibly is not holding the msk socket lock, but - * in the fallback case only the current subflow is touching - * the OoO queue. - */ - if (!RB_EMPTY_ROOT(&msk->out_of_order_queue)) - return false; - spin_lock_bh(&msk->fallback_lock); if (!msk->allow_subflows) { spin_unlock_bh(&msk->fallback_lock); diff --git a/net/mptcp/protocol.h b/net/mptcp/protocol.h index 64c6a6cb3d99..1cf67a557ff0 100644 --- a/net/mptcp/protocol.h +++ b/net/mptcp/protocol.h @@ -1127,6 +1127,14 @@ static inline bool __mptcp_try_fallback(struct mptcp_sock *msk) pr_debug("TCP fallback already done (msk=%p)\n", msk); return true; } + + /* The caller possibly is not holding the msk socket lock, but + * in the fallback case only the current subflow is touching + * the OoO queue. + */ + if (!RB_EMPTY_ROOT(&msk->out_of_order_queue)) + return false; + spin_lock_bh(&msk->fallback_lock); if (!msk->allow_infinite_fallback) { spin_unlock_bh(&msk->fallback_lock); diff --git a/net/netfilter/ipvs/ip_vs_xmit.c b/net/netfilter/ipvs/ip_vs_xmit.c index 863d48624c6a..cdb6fb98e267 100644 --- a/net/netfilter/ipvs/ip_vs_xmit.c +++ b/net/netfilter/ipvs/ip_vs_xmit.c @@ -985,7 +985,7 @@ ip_vs_prepare_tunneled_skb(struct sk_buff *skb, int skb_af, *next_protocol = IPPROTO_IPV6; if (payload_len) *payload_len = - ntohs(old_ipv6h->payload_len) + + ipv6_payload_len(skb, old_ipv6h) + sizeof(*old_ipv6h); old_dsfield = ipv6_get_dsfield(old_ipv6h); *ttl = old_ipv6h->hop_limit; diff --git a/net/netfilter/nf_conntrack_ovs.c b/net/netfilter/nf_conntrack_ovs.c index 068e9489e1c2..a6988eeb1579 100644 --- a/net/netfilter/nf_conntrack_ovs.c +++ b/net/netfilter/nf_conntrack_ovs.c @@ -121,7 +121,7 @@ int nf_ct_skb_network_trim(struct sk_buff *skb, int family) len = skb_ip_totlen(skb); break; case NFPROTO_IPV6: - len = ntohs(ipv6_hdr(skb)->payload_len); + len = skb_ipv6_payload_len(skb); if (ipv6_hdr(skb)->nexthdr == NEXTHDR_HOP) { int err = nf_ip6_check_hbh_len(skb, &len); diff --git a/net/netfilter/nf_log_syslog.c b/net/netfilter/nf_log_syslog.c index de51b15cf53d..e2c109d5f069 100644 --- a/net/netfilter/nf_log_syslog.c +++ b/net/netfilter/nf_log_syslog.c @@ -562,7 +562,7 @@ dump_ipv6_packet(struct net *net, struct nf_log_buf *m, /* Max length: 44 "LEN=65535 TC=255 HOPLIMIT=255 FLOWLBL=FFFFF " */ nf_log_buf_add(m, "LEN=%zu TC=%u HOPLIMIT=%u FLOWLBL=%u ", - ntohs(ih->payload_len) + sizeof(struct ipv6hdr), + ipv6_payload_len(skb, ih) + sizeof(struct ipv6hdr), (ntohl(*(__be32 *)ih) & 0x0ff00000) >> 20, ih->hop_limit, (ntohl(*(__be32 *)ih) & 0x000fffff)); diff --git a/net/netlink/af_netlink.c b/net/netlink/af_netlink.c index f3a8c8338d20..a7d3673ca72f 100644 --- a/net/netlink/af_netlink.c +++ b/net/netlink/af_netlink.c @@ -463,7 +463,7 @@ netlink_unlock_table(void) struct netlink_compare_arg { - possible_net_t pnet; + u64 net_cookie; u32 portid; }; @@ -478,14 +478,14 @@ static inline int netlink_compare(struct rhashtable_compare_arg *arg, const struct netlink_sock *nlk = ptr; return nlk->portid != x->portid || - !net_eq(sock_net(&nlk->sk), read_pnet(&x->pnet)); + sock_net(&nlk->sk)->net_cookie != x->net_cookie; } static void netlink_compare_arg_init(struct netlink_compare_arg *arg, struct net *net, u32 portid) { memset(arg, 0, sizeof(*arg)); - write_pnet(&arg->pnet, net); + arg->net_cookie = net->net_cookie; arg->portid = portid; } @@ -1345,6 +1345,8 @@ int netlink_unicast(struct sock *ssk, struct sk_buff *skb, if (netlink_is_kernel(sk)) return netlink_unicast_kernel(sk, skb, ssk); + skb_reset_network_header(skb); + skb_reset_mac_header(skb); if (sk_filter(sk, skb)) { err = skb->len; kfree_skb(skb); @@ -1479,6 +1481,8 @@ static void do_one_broadcast(struct sock *sk, goto out; } + skb_reset_network_header(p->skb2); + skb_reset_mac_header(p->skb2); if (sk_filter(sk, p->skb2)) { kfree_skb(p->skb2); p->skb2 = NULL; diff --git a/net/rds/rdma_transport.c b/net/rds/rdma_transport.c index b15cf316b23a..7be38be1228e 100644 --- a/net/rds/rdma_transport.c +++ b/net/rds/rdma_transport.c @@ -52,7 +52,7 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, { /* this can be null in the listening path */ struct rds_connection *conn = cm_id->context; - struct rds_transport *trans; + struct rds_transport *trans = &rds_ib_transport; int ret = 0; int *err; u8 len; @@ -60,9 +60,6 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, rdsdebug("conn %p id %p handling event %u (%s)\n", conn, cm_id, event->event, rdma_event_msg(event->event)); - if (cm_id->device->node_type == RDMA_NODE_IB_CA) - trans = &rds_ib_transport; - /* Prevent shutdown from tearing down the connection * while we're executing. */ if (conn) { @@ -82,16 +79,47 @@ static int rds_rdma_cm_event_handler_cmn(struct rdma_cm_id *cm_id, switch (event->event) { case RDMA_CM_EVENT_CONNECT_REQUEST: + /* Only the IB transport is supported, but RDS listens on + * every RDMA device: reject a request that arrived on any + * other kind. A non-zero return has rdma_cm destroy the + * request's id, which is what we want here and only here. + */ + if (cm_id->device->node_type != RDMA_NODE_IB_CA) { + ret = 1; + break; + } ret = trans->cm_handle_connect(cm_id, event, isv6); break; case RDMA_CM_EVENT_ADDR_RESOLVED: if (conn) { + /* The address resolved to a device RDS has no + * transport for. Do not go on to resolve a route: + * an iWARP route leaves cm_id->route.path_rec + * unset, which the ROUTE_RESOLVED case below + * dereferences. Drop the connection instead; the + * id is still ic->i_cm_id, so return 0 and let the + * shutdown destroy it. + */ + if (cm_id->device->node_type != RDMA_NODE_IB_CA) { + rds_conn_drop(conn); + break; + } rdma_set_service_type(cm_id, conn->c_tos); rdma_set_min_rnr_timer(cm_id, IB_RNR_TIMER_000_32); - /* XXX do we need to clean up if this fails? */ ret = rdma_resolve_route(cm_id, RDS_RDMA_RESOLVE_TIMEOUT_MS); + if (ret) { + /* A non-zero return has the rdma_cm destroy + * the id, but it is still ic->i_cm_id, which + * the connection's shutdown would then + * disconnect and destroy again. Drop the + * connection and keep the id for that + * shutdown. + */ + rds_conn_drop(conn); + ret = 0; + } } break; diff --git a/net/sched/act_api.c b/net/sched/act_api.c index 2954324f1230..9d3aeaf5ff9a 100644 --- a/net/sched/act_api.c +++ b/net/sched/act_api.c @@ -372,9 +372,10 @@ static int __tcf_action_put(struct tc_action *p, bool bind) { struct tcf_idrinfo *idrinfo = p->idrinfo; - if (refcount_dec_and_mutex_lock(&p->tcfa_refcnt, &idrinfo->lock)) { - if (bind) - atomic_dec(&p->tcfa_bindcnt); + mutex_lock(&idrinfo->lock); + if (bind) + atomic_dec(&p->tcfa_bindcnt); + if (refcount_dec_and_test(&p->tcfa_refcnt)) { idr_remove(&idrinfo->action_idr, p->tcfa_index); mutex_unlock(&idrinfo->lock); @@ -382,9 +383,7 @@ static int __tcf_action_put(struct tc_action *p, bool bind) return 1; } - if (bind) - atomic_dec(&p->tcfa_bindcnt); - + mutex_unlock(&idrinfo->lock); return 0; } diff --git a/net/sched/cls_flower.c b/net/sched/cls_flower.c index 8b4dc3930906..545213362e11 100644 --- a/net/sched/cls_flower.c +++ b/net/sched/cls_flower.c @@ -1589,6 +1589,11 @@ static int fl_set_enc_opt(struct nlattr **tb, struct fl_flow_key *key, return -EINVAL; } nla_opt_msk = nla_next(nla_opt_msk, &msk_depth); + + if (msk_depth && !nla_ok(nla_opt_msk, msk_depth)) { + NL_SET_ERR_MSG(extack, "A mask attribute is invalid"); + return -EINVAL; + } } return 0; diff --git a/net/sched/sch_cake.c b/net/sched/sch_cake.c index a25b45ab6fb0..3837ceb445fc 100644 --- a/net/sched/sch_cake.c +++ b/net/sched/sch_cake.c @@ -1280,7 +1280,7 @@ static struct sk_buff *cake_ack_filter(struct cake_sched_data *q, ipv6_addr_cmp(&ipv6h_check->daddr, &ipv6h->daddr)) continue; - seglen = ntohs(ipv6h_check->payload_len); + seglen = ipv6_payload_len(skb, ipv6h_check); } else { continue; } diff --git a/net/sctp/inqueue.c b/net/sctp/inqueue.c index 5f988b3a8814..d666cec6b194 100644 --- a/net/sctp/inqueue.c +++ b/net/sctp/inqueue.c @@ -212,8 +212,10 @@ struct sctp_chunk *sctp_inq_pop(struct sctp_inq *queue) chunk->chunk_end = ((__u8 *)ch) + SCTP_PAD4(ntohs(ch->length)); skb_pull(chunk->skb, sizeof(*ch)); chunk->subh.v = NULL; /* Subheader is no longer valid. */ - - if (chunk->chunk_end + sizeof(*ch) <= skb_tail_pointer(chunk->skb)) { + if (unlikely(ntohs(ch->length) < sizeof(*ch))) { + chunk->pdiscard = 1; + } else if (chunk->chunk_end + sizeof(*ch) <= + skb_tail_pointer(chunk->skb)) { /* This is not a singleton */ chunk->singleton = 0; } else if (chunk->chunk_end > skb_tail_pointer(chunk->skb)) { diff --git a/net/sctp/socket.c b/net/sctp/socket.c index 113e02e89403..df40d955b284 100644 --- a/net/sctp/socket.c +++ b/net/sctp/socket.c @@ -1785,6 +1785,18 @@ static int sctp_sendmsg_check_sflags(struct sctp_association *asoc, return 1; } +static int sctp_sendmsg_check_stream(struct sctp_association *asoc, + struct sctp_sndrcvinfo *sinfo) +{ + if (unlikely(sinfo->sinfo_stream >= asoc->stream.outcnt)) + return -EINVAL; + + if (unlikely(!SCTP_SO(&asoc->stream, sinfo->sinfo_stream)->ext)) + return sctp_stream_init_ext(&asoc->stream, sinfo->sinfo_stream); + + return 0; +} + static int sctp_sendmsg_to_asoc(struct sctp_association *asoc, struct msghdr *msg, size_t msg_len, struct sctp_transport *transport, @@ -1799,16 +1811,9 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc, long timeo; int err; - if (sinfo->sinfo_stream >= asoc->stream.outcnt) { - err = -EINVAL; + err = sctp_sendmsg_check_stream(asoc, sinfo); + if (err) goto err; - } - - if (unlikely(!SCTP_SO(&asoc->stream, sinfo->sinfo_stream)->ext)) { - err = sctp_stream_init_ext(&asoc->stream, sinfo->sinfo_stream); - if (err) - goto err; - } if (sp->disable_fragments && msg_len > asoc->frag_point) { err = -EMSGSIZE; @@ -1829,10 +1834,9 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc, err = sctp_wait_for_sndbuf(asoc, transport, &timeo, msg_len); if (err) goto err; - if (unlikely(sinfo->sinfo_stream >= asoc->stream.outcnt)) { - err = -EINVAL; + err = sctp_sendmsg_check_stream(asoc, sinfo); + if (err) goto err; - } } if (sctp_state(asoc, CLOSED)) { @@ -1847,6 +1851,11 @@ static int sctp_sendmsg_to_asoc(struct sctp_association *asoc, err = -ESRCH; goto err; } + err = sctp_sendmsg_check_stream(asoc, sinfo); + if (err) { + err = -ESRCH; + goto err; + } } else { wait_connect = true; } diff --git a/net/tipc/node.c b/net/tipc/node.c index 85f431ce9ea5..f2d83169fe22 100644 --- a/net/tipc/node.c +++ b/net/tipc/node.c @@ -1335,7 +1335,9 @@ static void tipc_node_reset_links(struct tipc_node *n) pr_warn("Resetting all links to %x\n", n->addr); + tipc_node_write_lock(n); trace_tipc_node_reset_links(n, true, " "); + tipc_node_write_unlock_fast(n); for (i = 0; i < MAX_BEARERS; i++) { tipc_node_link_down(n, i, false); } diff --git a/net/vmw_vsock/af_vsock.c b/net/vmw_vsock/af_vsock.c index 6b7bdd6166c0..9d6dce531199 100644 --- a/net/vmw_vsock/af_vsock.c +++ b/net/vmw_vsock/af_vsock.c @@ -334,6 +334,38 @@ struct sock *vsock_find_connected_socket(struct sockaddr_vm *src, } EXPORT_SYMBOL_GPL(vsock_find_connected_socket); +/** + * vsock_check_source - validate a packet source against a socket peer + * @vsk: socket receiving the packet + * @transport: transport receiving the packet + * @src: source address from the packet + * + * Return: true if the packet arrived on the socket's assigned transport and + * its source matches the stored peer. Loopback packets are generated + * internally and always use the local CID as their source, including + * connections using a valid CID alias. + * + * The caller must hold the socket lock and must not call this for listening + * sockets, which accept packets from any source and have no assigned + * transport. + */ +bool vsock_check_source(const struct vsock_sock *vsk, + const struct vsock_transport *transport, + const struct sockaddr_vm *src) +{ + if (vsk->transport != transport) + return false; + + if (src->svm_port != vsk->remote_addr.svm_port) + return false; + + if (src->svm_cid == vsk->remote_addr.svm_cid) + return true; + + return transport->get_local_cid() == VMADDR_CID_LOCAL; +} +EXPORT_SYMBOL_GPL(vsock_check_source); + void vsock_remove_sock(struct vsock_sock *vsk) { /* Transport reassignment must not remove the binding. */ @@ -1565,23 +1597,22 @@ static int vsock_connect(struct socket *sock, struct sockaddr *addr, timeout = schedule_timeout(timeout); lock_sock(sk); - /* Connection established. Whatever happens to socket once we - * release it, that's not connect()'s concern. No need to go + /* Connection was established. Whatever happens to socket once + * we release it, that's not connect()'s concern. No need to go * into signal and timeout handling. Call it a day. * * Note that allowing to "reset" an already established socket * here is racy and insecure. */ - if (sk->sk_state == TCP_ESTABLISHED) - break; + if (sk->sk_state == TCP_ESTABLISHED || + sk->sk_state == TCP_CLOSING) { + err = 0; + goto out_wait; + } /* If connection was _not_ established and a signal/timeout came * to be, we want the socket's state reset. User space may want - * to retry. - * - * sk_state != TCP_ESTABLISHED implies that socket is not on - * vsock_connected_table. We keep the binding and the transport - * assigned. + * to retry, so we keep the binding and the transport assigned. */ if (signal_pending(current) || timeout == 0) { err = timeout == 0 ? -ETIMEDOUT : sock_intr_errno(timeout); diff --git a/net/vmw_vsock/virtio_transport_common.c b/net/vmw_vsock/virtio_transport_common.c index 16431ef52799..e6e04cbb7a4b 100644 --- a/net/vmw_vsock/virtio_transport_common.c +++ b/net/vmw_vsock/virtio_transport_common.c @@ -1470,7 +1470,8 @@ void virtio_transport_recv_pkt(struct virtio_transport *t, * lock_sock (note: listener sockets are not assigned to any transport) */ if (sock_flag(sk, SOCK_DONE) || - (sk->sk_state != TCP_LISTEN && vsk->transport != &t->transport)) { + (sk->sk_state != TCP_LISTEN && + !vsock_check_source(vsk, &t->transport, &src))) { (void)virtio_transport_reset_no_sock(t, skb); release_sock(sk); sock_put(sk); diff --git a/samples/pidfd/Makefile b/samples/pidfd/Makefile index 9754e2d81f70..d82409f7061f 100644 --- a/samples/pidfd/Makefile +++ b/samples/pidfd/Makefile @@ -1,4 +1,4 @@ # SPDX-License-Identifier: GPL-2.0 -usertprogs-always-y += pidfd-metadata +userprogs-always-y += pidfd-metadata userccflags += -I usr/include diff --git a/security/keys/key.c b/security/keys/key.c index 4e49d2e3b858..8623a0e3b225 100644 --- a/security/keys/key.c +++ b/security/keys/key.c @@ -586,11 +586,18 @@ int key_reject_and_link(struct key *key, ret = -EBUSY; if (keyring) { - if (keyring->restrict_link) - return -EPERM; - link_ret = __key_link_lock(keyring, &key->index_key); if (link_ret == 0) { + /* + * Check the restriction under the keyring semaphore. + * keyring_restrict() installs it while holding the + * semaphore, so testing it beforehand races with a + * concurrent restriction install. + */ + if (keyring->restrict_link) { + __key_link_end(keyring, &key->index_key, edit); + return -EPERM; + } link_ret = __key_link_begin(keyring, &key->index_key, &edit); if (link_ret < 0) __key_link_end(keyring, &key->index_key, edit); diff --git a/security/keys/keyctl.c b/security/keys/keyctl.c index 127c17191785..e9fa1b94d0d0 100644 --- a/security/keys/keyctl.c +++ b/security/keys/keyctl.c @@ -677,12 +677,14 @@ long keyctl_describe_key(key_serial_t keyid, /* calculate how much information we're going to return */ ret = -ENOMEM; + down_read(&key->sem); infobuf = kasprintf(GFP_KERNEL, "%s;%d;%d;%08x;", key->type->name, from_kuid_munged(current_user_ns(), key->uid), from_kgid_munged(current_user_ns(), key->gid), key->perm); + up_read(&key->sem); if (!infobuf) goto error2; infolen = strlen(infobuf); @@ -806,8 +808,12 @@ static long __keyctl_read_key(struct key *key, char *buffer, size_t buflen) down_read(&key->sem); ret = key_validate(key); - if (ret == 0) - ret = key->type->read(key, buffer, buflen); + if (ret == 0) { + if (key->type->read) + ret = key->type->read(key, buffer, buflen); + else + ret = -EOPNOTSUPP; + } up_read(&key->sem); return ret; } @@ -861,11 +867,6 @@ long keyctl_read_key(key_serial_t keyid, char __user *buffer, size_t buflen) /* the key is probably readable - now try to read it */ can_read_key: - if (!key->type->read) { - ret = -EOPNOTSUPP; - goto key_put_out; - } - if (!buffer || !buflen) { /* Get the key length from the read method */ ret = __keyctl_read_key(key, NULL, 0); diff --git a/security/keys/keyring.c b/security/keys/keyring.c index fd95a0eb7a46..74ed1372d22c 100644 --- a/security/keys/keyring.c +++ b/security/keys/keyring.c @@ -14,6 +14,9 @@ #include #include #include +#include +#include +#include #include #include #include @@ -147,49 +150,46 @@ static int keyring_instantiate(struct key *keyring, } /* - * Multiply 64-bits by 32-bits to 96-bits and fold back to 64-bit. Ideally we'd - * fold the carry back too, but that requires inline asm. + * Give key types and domains address-independent identities. Zero means that + * an ID has not yet been allocated. As with inode sequence numbers used by + * futexes, this relies on the 64-bit counter not wrapping in a machine's + * lifetime. */ -static u64 mult_64x32_and_fold(u64 x, u32 y) +static u64 key_index_id(atomic64_t *id) { - u64 hi = (u64)(u32)(x >> 32) * y; - u64 lo = (u64)(u32)(x) * y; - return lo + ((u64)(u32)hi << 32) + (u32)(hi >> 32); + static atomic64_t next_id; + u64 old, new; + + old = atomic64_read(id); + if (likely(old)) + return old; + + do { + new = atomic64_inc_return(&next_id); + } while (WARN_ON_ONCE(!new)); + + old = 0; + if (!atomic64_try_cmpxchg_relaxed(id, &old, new)) + return old; + return new; } /* - * Hash a key type and description. + * Hash a key type, domain and description. KEYCTL_READ exposes the tree's + * traversal order, so no part of the index may contain kernel addresses. */ static void hash_key_type_and_desc(struct keyring_index_key *index_key) { const unsigned level_shift = ASSOC_ARRAY_LEVEL_STEP; const unsigned long fan_mask = ASSOC_ARRAY_FAN_MASK; - const char *description = index_key->description; - unsigned long hash, type; - u32 piece; + static siphash_key_t hash_key __read_mostly; + unsigned long hash; u64 acc; - int n, desc_len = index_key->desc_len; - - type = (unsigned long)index_key->type; - acc = mult_64x32_and_fold(type, desc_len + 13); - acc = mult_64x32_and_fold(acc, 9207); - piece = (unsigned long)index_key->domain_tag; - acc = mult_64x32_and_fold(acc, piece); - acc = mult_64x32_and_fold(acc, 9207); - - for (;;) { - n = desc_len; - if (n <= 0) - break; - if (n > 4) - n = 4; - piece = 0; - memcpy(&piece, description, n); - description += n; - desc_len -= n; - acc = mult_64x32_and_fold(acc, piece); - acc = mult_64x32_and_fold(acc, 9207); - } + + get_random_once(&hash_key, sizeof(hash_key)); + acc = siphash(index_key->description, index_key->desc_len, &hash_key); + acc = siphash_3u64(index_key->type_id, index_key->domain_id, acc, + &hash_key); /* Fold the hash down to 32 bits if need be. */ hash = acc; @@ -209,7 +209,7 @@ static void hash_key_type_and_desc(struct keyring_index_key *index_key) /* * Finalise an index key to include a part of the description actually in the - * index key, to set the domain tag and to calculate the hash. + * index key, to set the domain tag and index IDs, and to calculate the hash. */ void key_set_index_key(struct keyring_index_key *index_key) { @@ -225,6 +225,14 @@ void key_set_index_key(struct keyring_index_key *index_key) index_key->domain_tag = &default_domain_tag; } + /* Preserve the IDs when copying an existing key's index. In + * particular, its type may already have been unregistered. + */ + if (!index_key->type_id) + index_key->type_id = key_index_id(&index_key->type->index_id); + if (!index_key->domain_id) + index_key->domain_id = key_index_id(&index_key->domain_tag->index_id); + hash_key_type_and_desc(index_key); } @@ -263,15 +271,19 @@ void key_remove_domain(struct key_tag *domain_tag) /* * Build the next index key chunk. * + * The hash and inline description are followed by the 64-bit type and domain + * IDs, low word first, and then by the rest of the description. * We return it one word-sized chunk at a time. */ static unsigned long keyring_get_key_chunk(const void *data, int level) { + const unsigned int id_chunks = sizeof(u64) / sizeof(unsigned long); const struct keyring_index_key *index_key = data; + int desc_len = index_key->desc_len; unsigned long chunk = 0; - const u8 *d; - int desc_len = index_key->desc_len, n = sizeof(chunk); + int n = sizeof(chunk); unsigned int offset; + const u8 *d; level /= ASSOC_ARRAY_KEY_CHUNK_SIZE; switch (level) { @@ -279,27 +291,30 @@ static unsigned long keyring_get_key_chunk(const void *data, int level) return index_key->hash; case 1: return index_key->x; - case 2: - return (unsigned long)index_key->type; - case 3: - return (unsigned long)index_key->domain_tag; - default: - level -= 4; - offset = sizeof(index_key->desc) + level * sizeof(long); - if (desc_len <= offset) - return 0; - - d = index_key->description + offset; - desc_len -= offset; - if (desc_len > n) - desc_len = n; - d += desc_len; - do { - chunk <<= 8; - chunk |= *--d; - } while (--desc_len > 0); - return chunk; } + + level -= 2; + if (level < id_chunks) + return index_key->type_id >> (level * ASSOC_ARRAY_KEY_CHUNK_SIZE); + level -= id_chunks; + if (level < id_chunks) + return index_key->domain_id >> (level * ASSOC_ARRAY_KEY_CHUNK_SIZE); + level -= id_chunks; + + offset = sizeof(index_key->desc) + level * sizeof(long); + if (desc_len <= offset) + return 0; + + d = index_key->description + offset; + desc_len -= offset; + if (desc_len > n) + desc_len = n; + d += desc_len; + do { + chunk <<= 8; + chunk |= *--d; + } while (--desc_len > 0); + return chunk; } static unsigned long keyring_get_object_key_chunk(const void *object, int level) @@ -331,6 +346,7 @@ static int keyring_diff_objects(const void *object, const void *data) const struct keyring_index_key *b = data; unsigned long seg_a, seg_b; int level, i; + u64 diff; level = 0; seg_a = a->hash; @@ -339,28 +355,22 @@ static int keyring_diff_objects(const void *object, const void *data) goto differ; level += ASSOC_ARRAY_KEY_CHUNK_SIZE / 8; - /* The number of bits contributed by the hash is controlled by a - * constant in the assoc_array headers. Everything else thereafter we - * can deal with as being machine word-size dependent. - */ + /* The inline description contributes one machine-word-sized chunk. */ seg_a = a->x; seg_b = b->x; if ((seg_a ^ seg_b) != 0) goto differ; level += sizeof(unsigned long); - /* The next bit may not work on big endian */ - seg_a = (unsigned long)a->type; - seg_b = (unsigned long)b->type; - if ((seg_a ^ seg_b) != 0) - goto differ; - level += sizeof(unsigned long); + diff = a->type_id ^ b->type_id; + if (diff) + return level * 8 + __ffs64(diff); + level += sizeof(u64); - seg_a = (unsigned long)a->domain_tag; - seg_b = (unsigned long)b->domain_tag; - if ((seg_a ^ seg_b) != 0) - goto differ; - level += sizeof(unsigned long); + diff = a->domain_id ^ b->domain_id; + if (diff) + return level * 8 + __ffs64(diff); + level += sizeof(u64); i = sizeof(a->desc); if (a->desc_len <= i) @@ -1158,7 +1168,7 @@ struct key *find_keyring_by_name(const char *name, bool uid_keyring) * grants Search permission and that hasn't been revoked */ list_for_each_entry(keyring, &ns->keyring_name_list, name_link) { - if (!kuid_has_mapping(ns, keyring->user->uid)) + if (!kuid_has_mapping(ns, keyring->uid)) continue; if (test_bit(KEY_FLAG_REVOKED, &keyring->flags)) diff --git a/security/keys/process_keys.c b/security/keys/process_keys.c index b5d5333ab330..4b2b8a75eb1f 100644 --- a/security/keys/process_keys.c +++ b/security/keys/process_keys.c @@ -556,9 +556,8 @@ key_ref_t search_process_keyrings_rcu(struct keyring_search_context *ctx) ) { const struct cred *cred = ctx->cred; - if (key_validate(cred->request_key_auth) == 0) { - rka = ctx->cred->request_key_auth->payload.data[0]; - + rka = dereference_key_rcu(cred->request_key_auth); + if (rka && key_validate(cred->request_key_auth) == 0) { //// was search_process_keyrings() [ie. recursive] ctx->cred = rka->cred; key_ref = search_cred_keyrings_rcu(ctx); diff --git a/security/keys/trusted-keys/trusted_tpm1.c b/security/keys/trusted-keys/trusted_tpm1.c index 3b1ee879419c..78868b54c539 100644 --- a/security/keys/trusted-keys/trusted_tpm1.c +++ b/security/keys/trusted-keys/trusted_tpm1.c @@ -959,8 +959,10 @@ static int trusted_tpm_unseal(struct trusted_key_payload *p, char *datablob) ret = tpm2_unseal_trusted(chip, p, options); else ret = key_unseal(p, options); - if (ret < 0) + if (ret < 0) { pr_info("key_unseal failed (%d)\n", ret); + goto out; + } if (options->pcrlock) { ret = pcrlock(options->pcrlock); diff --git a/security/keys/trusted-keys/trusted_tpm2.c b/security/keys/trusted-keys/trusted_tpm2.c index bbd6ffb67cd6..ebd83cba18a7 100644 --- a/security/keys/trusted-keys/trusted_tpm2.c +++ b/security/keys/trusted-keys/trusted_tpm2.c @@ -124,7 +124,7 @@ static int tpm2_key_decode(struct trusted_key_payload *payload, if (ctx.priv_len + ctx.pub_len > MAX_BLOB_SIZE) return -EINVAL; - blob = kmalloc(ctx.priv_len + ctx.pub_len + 4, GFP_KERNEL); + blob = kmalloc(ctx.priv_len + ctx.pub_len, GFP_KERNEL); if (!blob) return -ENOMEM; diff --git a/sound/soc/codecs/cs42l52.c b/sound/soc/codecs/cs42l52.c index 4fc8a6ae8d92..2c7698618970 100644 --- a/sound/soc/codecs/cs42l52.c +++ b/sound/soc/codecs/cs42l52.c @@ -273,7 +273,7 @@ static const char * const cs42l52_ng_type_text[] = { }; static SOC_ENUM_SINGLE_DECL(ng_type_enum, - CS42L52_NOISE_GATE_CTL, 6, + CS42L52_NOISE_GATE_CTL, 7, cs42l52_ng_type_text); static const char * const left_swap_text[] = { diff --git a/sound/soc/codecs/cs42l56.c b/sound/soc/codecs/cs42l56.c index 1714857594fb..e04e174f449b 100644 --- a/sound/soc/codecs/cs42l56.c +++ b/sound/soc/codecs/cs42l56.c @@ -449,7 +449,7 @@ static const struct snd_kcontrol_new cs42l56_snd_controls[] = { 0, 0x3f, 0, 0), SOC_SINGLE_TLV("Limiter MAX", CS42L56_LIM_THRESHOLD_CTL, 5, 0x07, 1, alc_tlv), - SOC_SINGLE_TLV("Limiter Cushion", CS42L56_ALC_THRESHOLD, + SOC_SINGLE_TLV("Limiter Cushion", CS42L56_LIM_THRESHOLD_CTL, 2, 0x07, 1, alc_tlv), SOC_SINGLE("NG Switch", CS42L56_NOISE_GATE_CTL, 6, 1, 1), diff --git a/sound/soc/codecs/max98371.c b/sound/soc/codecs/max98371.c index f0e49179c38f..d0e7efce8096 100644 --- a/sound/soc/codecs/max98371.c +++ b/sound/soc/codecs/max98371.c @@ -166,7 +166,7 @@ static const struct snd_kcontrol_new max98371_snd_controls[] = { 0, (1<host_stream; - if (runtime->state == SNDRV_PCM_STATE_XRUN) + if (runtime->state != SNDRV_PCM_STATE_PREPARED) hdac_stream(host_stream)->prepared = false; if (hdac_stream(host_stream)->prepared) return 0; diff --git a/sound/soc/sof/ipc4-topology.c b/sound/soc/sof/ipc4-topology.c index 8911c075251c..8e16ec803838 100644 --- a/sound/soc/sof/ipc4-topology.c +++ b/sound/soc/sof/ipc4-topology.c @@ -1690,8 +1690,20 @@ sof_ipc4_prepare_copier_module(struct snd_sof_widget *swidget, ch_map >>= 4; } - step = ch_count / blob->alh_cfg.device_count; - mask = GENMASK(step - 1, 0); + if (ch_count < blob->alh_cfg.device_count) { + /* + * When there are fewer channels than aggregated devices + * (e.g. a mono capture stream split across multiple links), + * set step = 0 to ensure all the ch_mask is applied to all + * alh mappings. + */ + mask = ch_mask; + step = 0; + } else { + step = ch_count / blob->alh_cfg.device_count; + mask = GENMASK(step - 1, 0); + } + /* * Set each gtw_cfg.node_id to blob->alh_cfg.mapping[] * for all widgets with the same stream name diff --git a/sound/soc/stm/stm32_spdifrx.c b/sound/soc/stm/stm32_spdifrx.c index 4e17909d2126..4c2ff0997163 100644 --- a/sound/soc/stm/stm32_spdifrx.c +++ b/sound/soc/stm/stm32_spdifrx.c @@ -728,7 +728,7 @@ static irqreturn_t stm32_spdifrx_isr(int irq, void *devid) if (err) { regmap_read(spdifrx->regmap, STM32_SPDIFRX_CR, &cr); - sync_state = FIELD_GET(SPDIFRX_CR_SPDIFEN_MASK, cr) && + sync_state = FIELD_GET(SPDIFRX_CR_SPDIFEN_MASK, cr) == SPDIFRX_SPDIFEN_SYNC; /* SPDIFRX is in STATE_STOP. Disable SPDIFRX to clear errors */ diff --git a/tools/gpio/Makefile b/tools/gpio/Makefile index 342e056c8c66..1cd7636b9cac 100644 --- a/tools/gpio/Makefile +++ b/tools/gpio/Makefile @@ -7,7 +7,13 @@ bindir ?= /usr/bin # isn't set and when invoked from selftests build, where srctree # is set to ".". building_out_of_srctree is undefined for in srctree # builds +ifeq ($(srctree),) +update_srctree := 1 +endif ifndef building_out_of_srctree +update_srctree := 1 +endif +ifeq ($(update_srctree),1) srctree := $(patsubst %/,%,$(dir $(CURDIR))) srctree := $(patsubst %/,%,$(dir $(srctree))) endif diff --git a/tools/testing/selftests/ftrace/test.d/filter/event-filter-incomplete.tc b/tools/testing/selftests/ftrace/test.d/filter/event-filter-incomplete.tc new file mode 100644 index 000000000000..cc6df762845f --- /dev/null +++ b/tools/testing/selftests/ftrace/test.d/filter/event-filter-incomplete.tc @@ -0,0 +1,24 @@ +#!/bin/sh +# SPDX-License-Identifier: GPL-2.0 +# description: event filter - reject incomplete logical expressions +# requires: events/sched/sched_switch/filter +# flags: instance + +filter=events/sched/sched_switch/filter + +for base in \ + 'common_pid == 1' \ + 'common_pid == 1 || common_pid == 2' \ + 'common_pid == 1 && common_pid == 2' \ + '!(common_pid == 1 || common_pid == 2)' +do + printf '%s' "$base" > "$filter" + for tail in '&&' '||' '&& !' '|| !' '&& !! ' '|| !! '; do + if printf '%s' "$base $tail" > "$filter" 2>/dev/null; then + echo "Accepted incomplete filter: $base $tail" + exit_fail + fi + done +done + +echo 0 > "$filter"