From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-ej1-f44.google.com (mail-ej1-f44.google.com [209.85.218.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 90E128C11 for ; Mon, 27 Jan 2025 13:46:13 +0000 (UTC) Authentication-Results: smtp.subspace.kernel.org; arc=none smtp.client-ip=209.85.218.44 ARC-Seal:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737985575; cv=none; b=sMxUOMtSAGQA+xLq/dNzys0s3Hf2m6fdmYJ/s58tUiBZ6rwVDB2gXHO65idus3SJSKlV5C5e+nkjPNypFxid30CB8MjyUJdmtwRxbkdln1id6Q3hmoqnonqkctZ6dhX5BtSkRLZK5lIxHv9yzAWJFotXigCtTEJ/aaPodTTy5B4= ARC-Message-Signature:i=1; a=rsa-sha256; d=subspace.kernel.org; s=arc-20240116; t=1737985575; c=relaxed/simple; bh=7RsaTiBPJFGfRDsPei14scgSRch6QhNcy9+wrnM/bJo=; h=Date:From:To:Cc:Subject:Message-ID:References:MIME-Version: Content-Type:Content-Disposition:In-Reply-To; b=Sw1H8UsmXZX2l2lBH1bsaFsGBNSSZTSaWo4bJy3m1ENJ2SYDs0y1AZO3Z7aZUopRQPxsd2L/IUr7qZ6QTgQQ4zB87xW/iehoZC3NyUc1F9ykLWlqO+lO3DRRLaY9KwhSq++AUP83xUl6hFjxr58K73VxvTD4InvWq1NL4wv+IRY= ARC-Authentication-Results:i=1; smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com; spf=pass smtp.mailfrom=suse.com; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b=OVLLclLL; arc=none smtp.client-ip=209.85.218.44 Authentication-Results: smtp.subspace.kernel.org; dmarc=pass (p=quarantine dis=none) header.from=suse.com Authentication-Results: smtp.subspace.kernel.org; spf=pass smtp.mailfrom=suse.com Authentication-Results: smtp.subspace.kernel.org; dkim=pass (2048-bit key) header.d=suse.com header.i=@suse.com header.b="OVLLclLL" Received: by mail-ej1-f44.google.com with SMTP id a640c23a62f3a-aaf900cc7fbso710298666b.3 for ; Mon, 27 Jan 2025 05:46:13 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=google; t=1737985572; x=1738590372; darn=vger.kernel.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=QgA+jhPoaB6UopiA2dnwfV9s+vayrM8epUivPe+KeK8=; b=OVLLclLLSAWn03VTBCRKgMzBqtTMgq6VaxpofQUhzNMmFql4cV9VftVffMUhxffGDz aD/5NjQhpp/wd09//BxOCwvl7GFQSbsgRuWDNMRaGBEzh6tTNIVHXU9dCq4FBv+izwB8 QeU5uWIiAtKm4mvjG9Wx0cNbLHvK/XGOg5IF4VBCegFahqA1v8n9tIxS2P6gY/XMgC82 gPZZ0yxuRVeBkWvnerDF9IY5ix9aKWHYdz3IftiumxkGrLTzK0tRSVyGQyB1syYC1MWc aoOd9Omves0KUdBaXbndK0tpuUUxtbpEAbM+7pzb22iPk4FgliB9vYJcUL9sQm4lsM3H NbgQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1737985572; x=1738590372; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=QgA+jhPoaB6UopiA2dnwfV9s+vayrM8epUivPe+KeK8=; b=Fuopdv4Lrf+RhI88DUABxE6/H40Lbpq+CfutE6EPgfoXoFj+q0j+jtpvjhXLHwwT9N GUvT7KvlSE2+wf9YHfs+xtUsWcjSUYd55tquqqcw6V7OUp/ggpBPrGCCZYxFMhIWH/GB hKuHb/hDN1S+s/ItN10HxmeqrU5AFkLdVIKIyMFSeLGeDOEFaqrx396L3znKhYOH5fe1 yDoLkWOVdiXeDDpldEiMW5LJC3spODIVcZ89DQXVbR3zBB6damAGAeP0nwv0c59u4qzW SCZHk8YzCd0SoTehMtxkbegObNpIEMMQp+OzCcoZyGpOHa5cXmxeVN8tMDm5AqAV2BMN P3nQ== X-Forwarded-Encrypted: i=1; AJvYcCWltbaIltV4QA6nxjif/I+ahckfzI0VFIirPb6Gvo2F1inOyHLN7sUAePu6QJJgDasLl6jV/yvtQuqs9UA=@vger.kernel.org X-Gm-Message-State: AOJu0Yx3bBqIyypdS0z5UBPyQKnC3kx7f47ixozmJOBBlLzBcETcagre rTLbZ69EOIaOXoGPAWlcTMaf7q8lwQeCxljXnpf00gq55ATSL8AmlKLubBW0ZmI= X-Gm-Gg: ASbGncvRAMr1IshiwGJGLvrnnOwCMb3YA/IU7XrNWhOnbMiqj8+1rfGY1a1jLvwyM5a N2UN8/mrw3PBAjOVPUVhyZifEqk/nj9Ff4wySNn3wQybhtSCUUwzMVPrRqicqRiiW1GYZelh7CV azAusuTG18maxcg+iN9ymjEmdOV+bPO4pf0o5+m5tM+D/Kst1zlIE/cBu/wLQSMmpiBLYSGwpSR 2du66CTZVsz/KsJP5K2k4LntD5m7GdQwQaunm04NENLhyaZS5JWoTO2kEH7Lpe0ME9GPtKsh+/Q HhycU9E= X-Google-Smtp-Source: AGHT+IFpP+WprrGt8pOi9On434eDQJtnLjlCQXP3IvnT73SfyycRykwTaB4RzGPd3QDd8SmzpifmzA== X-Received: by 2002:a17:907:930b:b0:aa6:a9fe:46e5 with SMTP id a640c23a62f3a-ab38b3dae38mr4141599166b.53.1737985571826; Mon, 27 Jan 2025 05:46:11 -0800 (PST) Received: from pathway.suse.cz ([176.114.240.130]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-ab6956a0700sm323965666b.175.2025.01.27.05.46.11 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 27 Jan 2025 05:46:11 -0800 (PST) Date: Mon, 27 Jan 2025 14:46:09 +0100 From: Petr Mladek To: Yafang Shao Cc: jpoimboe@kernel.org, jikos@kernel.org, mbenes@suse.cz, joe.lawrence@redhat.com, live-patching@vger.kernel.org, linux-kernel@vger.kernel.org Subject: Re: [RFC PATCH 0/2] livepatch: Add support for hybrid mode Message-ID: References: <20250127063526.76687-1-laoar.shao@gmail.com> Precedence: bulk X-Mailing-List: linux-kernel@vger.kernel.org List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250127063526.76687-1-laoar.shao@gmail.com> On Mon 2025-01-27 14:35:24, Yafang Shao wrote: > The atomic replace livepatch mechanism was introduced to handle scenarios > where we want to unload a specific livepatch without unloading others. > However, its current implementation has significant shortcomings, making > it less than ideal in practice. Below are the key downsides: > > - It is expensive > > During testing with frequent replacements of an old livepatch, random RCU > warnings were observed: > > [19578271.779605] rcu_tasks_wait_gp: rcu_tasks grace period 642409 is 10024 jiffies old. > [19578390.073790] rcu_tasks_wait_gp: rcu_tasks grace period 642417 is 10185 jiffies old. > [19578423.034065] rcu_tasks_wait_gp: rcu_tasks grace period 642421 is 10150 jiffies old. > [19578564.144591] rcu_tasks_wait_gp: rcu_tasks grace period 642449 is 10174 jiffies old. > [19578601.064614] rcu_tasks_wait_gp: rcu_tasks grace period 642453 is 10168 jiffies old. > [19578663.920123] rcu_tasks_wait_gp: rcu_tasks grace period 642469 is 10167 jiffies old. > [19578872.990496] rcu_tasks_wait_gp: rcu_tasks grace period 642529 is 10215 jiffies old. > [19578903.190292] rcu_tasks_wait_gp: rcu_tasks grace period 642529 is 40415 jiffies old. > [19579017.965500] rcu_tasks_wait_gp: rcu_tasks grace period 642577 is 10174 jiffies old. > [19579033.981425] rcu_tasks_wait_gp: rcu_tasks grace period 642581 is 10143 jiffies old. > [19579153.092599] rcu_tasks_wait_gp: rcu_tasks grace period 642625 is 10188 jiffies old. > > This indicates that atomic replacement can cause performance issues, > particularly with RCU synchronization under frequent use. Please, provide more details about the test: + List of patched functions. + What exactly is meant by frequent replacements (busy loop?, once a minute?) + Size of the systems (number of CPUs, number of running processes) + Were there any extra changes in the livepatch code code, e.g. debugging messages? > - Potential Risks During Replacement > > One known issue involves replacing livepatched versions of critical > functions such as do_exit(). During the replacement process, a panic > might occur, as highlighted in [0]. I would rather prefer to make the atomic replace safe. I mean to block the transition until all exiting processes are not gone. Josh made a good point. We should look how this is handled by RCU, tracing, or another subsystems which might have similar problems. > Other potential risks may also arise > due to inconsistencies or race conditions during transitions. What inconsistencies and race conditions you have in mind, please? > - Temporary Loss of Patching > > During the replacement process, the old patch is set to a NOP (no-operation) > before the new patch is fully applied. This creates a window where the > function temporarily reverts to its original, unpatched state. If the old > patch fixed a critical issue (e.g., one that prevented a system panic), the > system could become vulnerable to that issue during the transition. This is not true! Please, look where klp_patch_object() and klp_unpatch_objects() is called. Also look at how ops->func_stack is handled in klp_ftrace_handler(). Also you might want to read Documentation/livepatch/livepatch.rst > The current atomic replacement approach replaces all old livepatches, > even when such a sweeping change is unnecessary. This can be improved > by introducing a hybrid mode, which allows the coexistence of both > atomic replace and non atomic replace livepatches. > > In the hybrid mode: > > - Specific livepatches can be marked as "non-replaceable" to ensure they > remain active and unaffected during replacements. > > - Other livepatches can be marked as "replaceable", allowing targeted > replacements of only those patches. Honestly, I consider this as a workaround for a problem which should be fixed a proper way. The main advantage of the atomic replace is simplify the maintenance and debugging. It reduces the amount of possible combinations. The hybrid mode brings back the jungle. Best Regards, Petr