From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f45.google.com (mail-pj1-f45.google.com [209.85.216.45]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B6ACF3019DA for ; Tue, 2 Dec 2025 07:41:38 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.216.45 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764661300; cv=none; b=pP2SXCXB+FjNbp3iB8A+RPRaWPXe9CEiaN+2Ry5ZnWcsF5L1tEdayUoLO0Vb3AFrzBAb82eveP+AGtuehxWXpvis3Uo6FA/HrYgfxv7kHh+JHZLNDIU1Tib9pO8ENbq7+2EBYY+yHi72kcOnXPKPhBFtJqHLntaY4Odx1t95gYk= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1764661300; c=relaxed/simple; bh=xu7Cgfrq/o8NggS6y3CuogqEmKosoKLCCe5luGQ7/so=; h=From:To:Cc:Subject:Date:Message-Id:In-Reply-To:References: MIME-Version; b=I6hy4CeRZETe7ZA88T4udWFThtv3craS43CvVAMCTauUVb6gvEhdll2qHLEVe1cPL2sT0IrRhJz2x9pjL0UkT9mDWaLTXWsbzDpxSjeok6PjH6lTXp7LUOgewM9XsjGgZeWLdWLbKZ9BopseH72crV0HY3VkZT6I2cZCX0WMGPw= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com; spf=pass smtp.mailfrom=gmail.com; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b=lBjRN/qa; arc=none smtp.client-ip=209.85.216.45 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=none dis=none) header.from=gmail.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=gmail.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="lBjRN/qa" Received: by mail-pj1-f45.google.com with SMTP id 98e67ed59e1d1-3436cbb723fso4323651a91.2 for ; Mon, 01 Dec 2025 23:41:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1764661298; x=1765266098; darn=vger.kernel.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=pdY669oPFp64Z/oUf+jdUTfbYCRwgKwWZWXra6Qr1vI=; b=lBjRN/qaLxRr4MjtE61P105KW0qo1N+DLaSqXXwNqZ61EeXeUCWxEEfydLRzOlQIz3 JtrbiF+GWAfi3kmJ23HLEjm52O/hb3I9hfeJdn5qBrqox5AhxgRCmGY5Dn+cpDwCg/SO 8ixR3VlhkApOzqiyCqdyqn6CgcPha56aFoOsvhNfCn5lT036gdLE1COqNgNU4VywD9ex 4l3ESTB+KpbQtghK3HAUPGfMigf/l8+XSnvredJeSi4qfQDWL6Aik8sUjG8+1/66SINE Uy7hkMMd4CNlJqXmi1UgGgnxgU7DFwufP6c2CH2JhDpPSm2zLUDp1LNDeEFmPd+xW8FJ Apeg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1764661298; x=1765266098; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-gg:x-gm-message-state:from :to:cc:subject:date:message-id:reply-to; bh=pdY669oPFp64Z/oUf+jdUTfbYCRwgKwWZWXra6Qr1vI=; b=pRHtth0QqlyvQKjoc9oE0pHJpNTA1Q9Lw61MfxfJRvfkv9RWnGVST3CSX9yD4AgIdT kK3PZDgU/EUfyZ0H7IvzGogypSokexkh65TsGkUgs0ZZndP35+RY/VZTz+wQI9gcscTa UAKBgasNifSynFcwiEsvQtHfQezQBk1q57Dzm2nLi6fDbljvrwjps0RdFLN3OXK5ov2V OEoJFFAZ7hXAAUjXZw0akHHPCBqYIwn7BRGvzOefLJS7+NlasUXdodG7ZjpZ+VH4gHWF JXwJTF22eean/dF+1KouurkPdesBO8rX+nfjIUzH2cZwhDngjuIcA54AcJdmHiS48yOS 4MQg== X-Forwarded-Encrypted: i=1; AJvYcCUJ/ardchlffMXvQv+O/4c8c1wa0hSD/u2MG48uYe8Mz+U6xMryyOTI69xyH+dujInuDYcn6zBWfuTDaB0=@vger.kernel.org X-Gm-Message-State: AOJu0Yyzli3k/hKa+M4m1KVuXDmH/nnf3ofM3F7L2ziSao+rbuUnT05C q55fIx++vIU0fJ+UnY1ms2J9gnKOIOFz/mIEEO4XCSu1U0Mtok6Ky8wX X-Gm-Gg: ASbGncvrw5aF1pJWviesB6rjXVJkPEbh7H/3fyXsRFn8qlNtzI5CggMbsbfnqKoJ3Fo dBGLdasv3mPAqU1e+dNbc5oirHcUOR6jLsXEYtW3g3xBTRRiVVk+Q5qK4LqYpO6QXQflEbFl4Eh d9p0JUu8gE15G/CiH55ePjbXkvvOo6y59ZkVlkAkvewaN8qCnnkENhiBg9Gc+z3uxl7AXx9oBai WHb8rxZd9uOjpgWpvyovI6RXV9wLPcbxh3TZNHB0EcnyI+vqsLmGb/ieEDN5D4q0pR+kXX5j3qz r91krGM+J1qTniDNOPyo9/qUbK2B15WIbvw+WHNAziPinzNegEnL+vf0uBXE2kOSNTYco0oMk4B hERKDfvsz8hsHFGUPd9g2iqMcIgP5pReO5Ya1XlRHHXr4c9Sp5Ck9a2OoNqg0Rj8BcEp90M5zYC TZ8DknvPLeIqVbsqkYE3yqA3XdTkUmbEpgtQ== X-Google-Smtp-Source: AGHT+IEi6+/L2LO/9oDve9kx9ZLziN85TJNVlxJPaA1DiO6RqWCP8Y1ivDHqQ4sUsQ/zqpXNrnohTg== X-Received: by 2002:a17:903:1a0f:b0:27c:56af:88ea with SMTP id d9443c01a7336-29bab3166acmr289375615ad.60.1764661297941; Mon, 01 Dec 2025 23:41:37 -0800 (PST) Received: from mao-OptiPlex-3050.hz.ali.com ([47.246.101.226]) by smtp.gmail.com with ESMTPSA id d9443c01a7336-29bceb7f69asm142957605ad.102.2025.12.01.23.41.35 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Dec 2025 23:41:37 -0800 (PST) From: maohan4761@gmail.com To: pjw@kernel.org, palmer@dabbelt.com Cc: guoren@kernel.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, Mao Han Subject: [PATCH 1/1] riscv: Optimize user copy with efficient unaligned access support Date: Tue, 2 Dec 2025 15:41:21 +0800 Message-Id: <20251202074121.81364-2-maohan4761@gmail.com> X-Mailer: git-send-email 2.34.1 In-Reply-To: <20251202074121.81364-1-maohan4761@gmail.com> References: <20251202074121.81364-1-maohan4761@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit From: Mao Han Introduce an optimized path in fallback_scalar_usercopy_sum_enabled for systems that support efficient unaligned memory accesses (i.e., when CONFIG_RISCV_EFFICIENT_UNALIGNED_ACCESS is enabled). This eliminates the overhead of bit-shifting and OR-ing partial words to reconstruct misaligned values, which was previously required for handling protential alignments. Medium-sized buffers between 8 and 9*SZREG also see noticeable improvement, as the original path would fall back to byte-by-byte copying. Signed-off-by: Mao Han --- arch/riscv/lib/uaccess.S | 113 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 113 insertions(+) diff --git a/arch/riscv/lib/uaccess.S b/arch/riscv/lib/uaccess.S index 4efea1b..bf124a1 100644 --- a/arch/riscv/lib/uaccess.S +++ b/arch/riscv/lib/uaccess.S @@ -54,6 +54,118 @@ EXPORT_SYMBOL(__asm_copy_from_user_sum_enabled) EXPORT_SYMBOL(__asm_copy_to_user_sum_enabled) SYM_FUNC_START(fallback_scalar_usercopy_sum_enabled) + /* + * Save the terminal address which will be used to compute the number + * of bytes copied in case of a fixup exception. + */ + add t5, a0, a2 + + /* + * Register allocation for code below: + * a0 - start of uncopied dst + * a1 - start of uncopied src + * a2 - size + * t0 - end of uncopied dst + */ + add t0, a0, a2 +#ifdef CONFIG_RISCV_EFFICIENT_UNALIGNED_ACCESS + /* If length < 8, go to byte copy */ + li a3, 8 + bltu a2, a3, .Lbyte_copy_tail + + /* check length >= 128 */ + li t1, 128 + bltu a2, t1, .L_len_less_16x_szreg + +.L_loop_16x_reg: + fixup REG_L a4, 0(a1), 10f + fixup REG_L a5, SZREG(a1), 10f + fixup REG_L a6, 2*SZREG(a1), 10f + fixup REG_L a7, 3*SZREG(a1), 10f + fixup REG_S a4, 0(a0), 10f + fixup REG_S a5, SZREG(a0), 10f + fixup REG_S a6, 2*SZREG(a0), 10f + fixup REG_S a7, 3*SZREG(a0), 10f + + fixup REG_L t1, 4*SZREG(a1), 10f + fixup REG_L t2, 5*SZREG(a1), 10f + fixup REG_L t3, 6*SZREG(a1), 10f + fixup REG_L t4, 7*SZREG(a1), 10f + fixup REG_S t1, 4*SZREG(a0), 10f + fixup REG_S t2, 5*SZREG(a0), 10f + fixup REG_S t3, 6*SZREG(a0), 10f + fixup REG_S t4, 7*SZREG(a0), 10f + + fixup REG_L a4, 8*SZREG(a1), 10f + fixup REG_L a5, 9*SZREG(a1), 10f + fixup REG_L a6, 10*SZREG(a1), 10f + fixup REG_L a7, 11*SZREG(a1), 10f + fixup REG_S a4, 8*SZREG(a0), 10f + fixup REG_S a5, 9*SZREG(a0), 10f + fixup REG_S a6, 10*SZREG(a0), 10f + fixup REG_S a7, 11*SZREG(a0), 10f + + fixup REG_L t1, 12*SZREG(a1), 10f + fixup REG_L t2, 13*SZREG(a1), 10f + fixup REG_L t3, 14*SZREG(a1), 10f + fixup REG_L t4, 15*SZREG(a1), 10f + fixup REG_S t1, 12*SZREG(a0), 10f + fixup REG_S t2, 13*SZREG(a0), 10f + fixup REG_S t3, 14*SZREG(a0), 10f + fixup REG_S t4, 15*SZREG(a0), 10f + + addi a1, a1, 16*SZREG + addi a0, a0, 16*SZREG + + addi t1, a0, 16*SZREG + bleu t1, t0, .L_loop_16x_reg + +.L_len_less_16x_szreg: + # Pre-check: ensure at least one register copy is possible + addi t1, a0, 4*SZREG + bgtu t1, t0, .L_len_less_4x_szreg + +.L_loop_4x_reg: + fixup REG_L a4, 0(a1), 10f + fixup REG_L a5, SZREG(a1), 10f + fixup REG_L a6, 2*SZREG(a1), 10f + fixup REG_L a7, 3*SZREG(a1), 10f + fixup REG_S a4, 0(a0), 10f + fixup REG_S a5, SZREG(a0), 10f + fixup REG_S a6, 2*SZREG(a0), 10f + fixup REG_S a7, 3*SZREG(a0), 10f + addi a1, a1, 4*SZREG + addi a0, a0, 4*SZREG + + # Check if another register copy is safe + addi t1, a0, 4*SZREG + bleu t1, t0, .L_loop_4x_reg + +.L_len_less_4x_szreg: + # Pre-check: ensure at least one register copy is possible + add t1, a0, SZREG + bgtu t1, t0, .Lbyte_copy_word + +.L_loop_reg: + fixup REG_L a4, 0(a1), 10f + addi a1, a1, SZREG + fixup REG_S a4, 0(a0), 10f + addi a0, a0, SZREG + + # Check if another register copy is safe + addi t1, a0, SZREG + bleu t1, t0, .L_loop_reg +.Lbyte_copy_word: +#if __riscv_xlen == 64 + add t1, a0, 4 + bgtu t1, t0, .Lbyte_copy_tail + + fixup lw a4, 0(a1), 10f + addi a1, a1, 4 + fixup sw a4, 0(a0), 10f + addi a0, a0, 4 +#endif +#else /* * Save the terminal address which will be used to compute the number * of bytes copied in case of a fixup exception. @@ -190,6 +302,7 @@ SYM_FUNC_START(fallback_scalar_usercopy_sum_enabled) /* Revert src to original unaligned value */ add a1, a1, a3 +#endif /* CONFIG_RISCV_EFFICIENT_UNALIGNED_ACCESS */ .Lbyte_copy_tail: /* * Byte copy anything left. -- 2.25.1