From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1758271AbYENObV (ORCPT ); Wed, 14 May 2008 10:31:21 -0400 Received: (majordomo@vger.kernel.org) by vger.kernel.org id S1754481AbYENObN (ORCPT ); Wed, 14 May 2008 10:31:13 -0400 Received: from mail2.shareable.org ([80.68.89.115]:52948 "EHLO mail2.shareable.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1754034AbYENObL (ORCPT ); Wed, 14 May 2008 10:31:11 -0400 Date: Wed, 14 May 2008 15:31:05 +0100 From: Jamie Lokier To: Evgeniy Polyakov Cc: Sage Weil , Jeff Garzik , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org Subject: Re: POHMELFS high performance network filesystem. Transactions, failover, performance. Message-ID: <20080514143105.GB14987@shareable.org> Mail-Followup-To: Evgeniy Polyakov , Sage Weil , Jeff Garzik , linux-kernel@vger.kernel.org, netdev@vger.kernel.org, linux-fsdevel@vger.kernel.org References: <20080513174523.GA1677@2ka.mipt.ru> <4829E752.8030104@garzik.org> <20080513205114.GA16489@2ka.mipt.ru> <20080514135156.GA23131@2ka.mipt.ru> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20080514135156.GA23131@2ka.mipt.ru> User-Agent: Mutt/1.5.13 (2006-08-11) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Evgeniy Polyakov wrote: > > For writes, Paxos is actually more or less optimal (in the non-failure > > cases, at least). Reads are trickier, but there are ways to keep that > > fast as well. FWIW, Ceph extends basic Paxos with a leasing mechanism to > > keep reads fast, consistent, and distributed. It's only used for cluster > > state, though, not file data. > > Well, it depends... If we are talking about single node perfromance, > then any protocol, which requries to wait for authorization (or any > approach, which waits for acknowledge just after data was sent) is slow. > > If we are talking about agregate parallel perfromance, then its basic > protocol with 2 messages is (probably) optimal, but still I'm not > convinced, that 2 messages case is a good choise, I want one :) Look up "one-phase commit" or even "zero-phase commit". (The terminology is cheating a bit.) As I've understood it, all commit protocols have a step where each node guarantees it can commit if asked and node failure at that point does not invalidate the guarantee if the node recovers (if it can't maintain the guarantee, the node doesn't recover in a technical sense and a higher level protocol may reintegrate the node). One/zero-phase commit extends that to guaranteeing a certain amounts and types of data can be written before it knows what the data is, so write messages within that window are sufficient for global commits. Guarantees can be acquired asynchronously in advance of need, and can have time and other limits. These guarantees are no different in principle from the 1-bit guarantee offered by the "can you commit" phase of other commit protocols, so they aren't as weak as they seem. Now combine it with a quorum protocol like Paxos, you can commit with async guarantees from a subset of nodes. Guarantees can be piggybacked on earlier requests. There, single node write performance with quorum robustness. -- Jamie