From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-3.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3F05C3F68F for ; Fri, 31 Jan 2020 07:41:44 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id B9D0320705 for ; Fri, 31 Jan 2020 07:41:44 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="VsXqKMr7" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1728164AbgAaHlo (ORCPT ); Fri, 31 Jan 2020 02:41:44 -0500 Received: from smtp-fw-4101.amazon.com ([72.21.198.25]:15475 "EHLO smtp-fw-4101.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1728089AbgAaHln (ORCPT ); Fri, 31 Jan 2020 02:41:43 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1580456503; x=1611992503; h=from:to:cc:subject:date:message-id:in-reply-to: mime-version; bh=hXqcxoeUTXv6Sl/D/o2xVhucn86dLhacZQT1P/cIuzM=; b=VsXqKMr7mO312v5ohuH8XjkUnzIt36KoDYQhKwgEHa6yIdpnjWHqB85X TNTn0Kk7o/x7ABgvPCoRDa9K3C+gJGtGJRRpyFACsWdoiH4PwE0GSwKgC KDaglHlrsojI/XxhHFBFRSLIKT8Xl/aj7S1yX+sUtX9QXKVFiso9fbq9/ 0=; IronPort-SDR: TDVLF5psLVoz4B8kmnPSKaCBxvX8j+VejGNDcH7bgeJuy0/p1gPPyG1+1SwQ0HFCFa+vcpb7p8 3JUDRyKLDbEg== X-IronPort-AV: E=Sophos;i="5.70,385,1574121600"; d="scan'208";a="14975291" Received: from iad12-co-svc-p1-lb1-vlan3.amazon.com (HELO email-inbound-relay-1d-74cf8b49.us-east-1.amazon.com) ([10.43.8.6]) by smtp-border-fw-out-4101.iad4.amazon.com with ESMTP; 31 Jan 2020 07:41:41 +0000 Received: from EX13MTAUEA002.ant.amazon.com (iad55-ws-svc-p15-lb9-vlan2.iad.amazon.com [10.40.159.162]) by email-inbound-relay-1d-74cf8b49.us-east-1.amazon.com (Postfix) with ESMTPS id CC8DEC08EB; Fri, 31 Jan 2020 07:41:37 +0000 (UTC) Received: from EX13D31EUA001.ant.amazon.com (10.43.165.15) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Fri, 31 Jan 2020 07:41:37 +0000 Received: from u886c93fd17d25d.ant.amazon.com (10.43.160.29) by EX13D31EUA001.ant.amazon.com (10.43.165.15) with Microsoft SMTP Server (TLS) id 15.0.1367.3; Fri, 31 Jan 2020 07:41:30 +0000 From: To: Eric Dumazet CC: , David Miller , "Alexei Starovoitov" , Daniel Borkmann , "Martin KaFai Lau" , Song Liu , Yonghong Song , , netdev , LKML , bpf , , Benjamin Herrenschmidt , Subject: Re: Re: Re: Latency spikes occurs from frequent socket connections Date: Fri, 31 Jan 2020 08:41:16 +0100 Message-ID: <20200131074116.8684-1-sjpark@amazon.com> X-Mailer: git-send-email 2.17.1 In-Reply-To: (raw) MIME-Version: 1.0 Content-Type: text/plain X-Originating-IP: [10.43.160.29] X-ClientProxiedBy: EX13D13UWA002.ant.amazon.com (10.43.160.172) To EX13D31EUA001.ant.amazon.com (10.43.165.15) Sender: linux-kernel-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thu, 30 Jan 2020 09:02:08 -0800 Eric Dumazet wrote: > On Thu, Jan 30, 2020 at 4:41 AM wrote: > > > > On Wed, 29 Jan 2020 09:52:43 -0800 Eric Dumazet wrote: > > > > > On Wed, Jan 29, 2020 at 9:14 AM wrote: > > > > > > > > Hello, > > > > > > > > > > > > We found races in the kernel code that incur latency spikes. We thus would > > > > like to share our investigations and hear your opinions. > > > > [...] > > > > > > I would rather try to fix the issue more generically, without adding > > > extra lookups as you did, since they might appear > > > to reduce the race, but not completely fix it. > > > > > > For example, the fact that the client side ignores the RST and > > > retransmits a SYN after one second might be something that should be > > > fixed. > > > > I also agree with this direction. It seems detecting this situation and > > adjusting the return value of tcp_timeout_init() to a value much lower than the > > one second would be a straightforward solution. For a test, I modified the > > function to return 1 (4ms for CONFIG_HZ=250) and confirmed the reproducer be > > silent. My following question is, how we can detect this situation in kernel? > > However, I'm unsure how we can distinguish this specific case from other cases, > > as everything is working as normal according to the TCP protocol. > > > > Also, it seems the value is made to be adjustable from the user space using the > > bpf callback, BPF_SOCK_OPS_TIMEOUT_INIT: > > > > BPF_SOCK_OPS_TIMEOUT_INIT, /* Should return SYN-RTO value to use or > > * -1 if default value should be used > > */ > > > > Thus, it sounds like you are suggesting to do the detection and adjustment from > > user space. Am I understanding your point? If not, please let me know. > > > > No, I was suggesting to implement a mitigation in the kernel : > > When in SYN_SENT state, receiving an suspicious ACK should not > simply trigger a RST. > > There are multiple ways maybe to address the issue. > > 1) Abort the SYN_SENT state and let user space receive an error to its > connect() immediately. > > 2) Instead of a RST, allow the first SYN retransmit to happen immediately > (This is kind of a challenge SYN. Kernel already implements challenge acks) > > 3) After RST is sent (to hopefully clear the state of the remote), > schedule a SYN rtx in a few ms, > instead of ~ one second. Thank you for this kind comment, Eric! I would prefer the second and third idea rather than first one. Anyway, I will send a patch soon. Will add a kselftest for this case, too. Thanks, SeongJae Park [...]