Extended Volume Manager API

Extended Volume Manager API

[Circuitsoft Development]

Question:

How hard would it be to implement a volume manager with an extended
API? I want to write a tightly-integrated volume manager/distributed
lock manager to arbitrate lock-free, atmoic read/compare/write
operations with other computers on the network. Also, I have a need to
have this API available to user-space applications. How would I do
this?

Details follow, if you're curious.

The idea is to make a distributed filesystem that won't hiccup when a
client node crashes. I've decided to implement all the gritty work in
the volume manager, and just design the filesystem to take advantage
of it.

Having done my own tests, I've determined that a 5msec timeout to
determine if a host has crashed would be appropriate. (My test was
basically looking at ping time over a saturated network - averaged
about 600 microseconds, topped at 3msec over 10 minutes) I figure that
5msec timeout won't add any noticeable lag to the volume manager, as
most disk seek times are in that range.

Anyway, we basically have the volume manager asking around for
everyone else on the network, to see if they have a particular area
locked. If so, wait for it to be unlocked. If not, record and
broadcast the lock, then do the atomic read/compare/write operation
while locked, and unlock before we return to the client (filesystem).

I need the API to this code to be available to userspace due to an
application that I cannot mention because of an informal NDA.

Thank you,
- Alex Austin, Circuitsoft Computer Services
"From Windows to MacOS, and the Linux in between"

Re: Extended Volume Manager API

[Valdis.Kletnieks]

[-- Attachment #1: Type: text/plain, Size: 987 bytes --]

On Mon, 01 May 2006 00:26:05 CDT, Circuitsoft Development said:

> Having done my own tests, I've determined that a 5msec timeout to
> determine if a host has crashed would be appropriate. (My test was
> basically looking at ping time over a saturated network - averaged
> about 600 microseconds, topped at 3msec over 10 minutes) I figure that
> 5msec timeout won't add any noticeable lag to the volume manager, as
> most disk seek times are in that range.W

Note that if you're setting 5ms as your timeout for detecting a *crash*,
and your *ping* takes 3ms, that leaves you a whole whopping 2ms.  If you
have 1ms scheduler latency at *each* end (remember - you're in userspace
at both ends, right?) you have approximately 0ms left for the remote end to
actually *do* anything, and for the local end to process the reply.

And if the remote end has to issue a syscall during processing the request,
you're basically screwed.

You need to be adding at least 1 zero to that timeout value.

[-- Attachment #2: Type: application/pgp-signature, Size: 226 bytes --]

Fwd: Extended Volume Manager API

[Circuitsoft Development]

On 5/1/06, Valdis.Kletnieks@vt.edu <Valdis.Kletnieks@vt.edu> wrote:
> On Mon, 01 May 2006 00:26:05 CDT, Circuitsoft Development said:
> > about 600 microseconds, topped at 3msec over 10 minutes) I figure that
> > 5msec timeout won't add any noticeable lag to the volume manager, as
> > most disk seek times are in that range.
>
> Note that if you're setting 5ms as your timeout for detecting a *crash*,
> and your *ping* takes 3ms, that leaves you a whole whopping 2ms.  If you
> have 1ms scheduler latency at *each* end (remember - you're in userspace

Volume/Lock manager in Kernelspace - Don't feel like dealing with
user-mode block devices

I was actually planning on a 5msec timeout to ignore that computer,
for now, then if I don't get a response within 100msec,  ping them,
and permenantly remove them from the list of peers and broadcast a
"this peer is dead" message to the network if the ping times out at
500msec.

> at both ends, right?) you have approximately 0ms left for the remote end to
> actually *do* anything, and for the local end to process the reply.
>
> And if the remote end has to issue a syscall during processing the request,
> you're basically screwed.

The code on the remote end is checking a list of locks to see if a
block is in it.

Re: Fwd: Extended Volume Manager API

[Rik van Riel]

On Mon, 1 May 2006, Circuitsoft Development wrote:

> I was actually planning on a 5msec timeout to ignore that computer,
> for now, then if I don't get a response within 100msec,  ping them,
> and permenantly remove them from the list of peers and broadcast a
> "this peer is dead" message to the network if the ping times out at
> 500msec.

How are you going to prevent your "dead" peer from writing
to the disk anyway ?

-- 
All Rights Reversed

Fwd: Fwd: Extended Volume Manager API

[Circuitsoft Development]

On 5/2/06, Rik van Riel <riel@redhat.com> wrote:
> On Mon, 1 May 2006, Circuitsoft Development wrote:
>
> > I was actually planning on a 5msec timeout to ignore that computer,
> > for now, then if I don't get a response within 100msec,  ping them,
> > and permenantly remove them from the list of peers and broadcast a
> > "this peer is dead" message to the network if the ping times out at
> > 500msec.
>
> How are you going to prevent your "dead" peer from writing
> to the disk anyway ?
>
> --
> All Rights Reversed
>

I'm not. They also need to get permission from the network before they
write to the disk, and they're not going to get permission without
hearing back from everybody. Besides, since the same network is used
to connect to the disks as is used to connect the computers to each
other, how would it be able to access the disks without being able to
access other computers which also connect to the disks?

(Sorry for the repeat, Rik)

Re: Fwd: Fwd: Extended Volume Manager API

[Lars Marowsky-Bree]

On 2006-05-08T16:17:07, Circuitsoft Development <circuitsoft.devel@gmail.com> wrote:

> I'm not. They also need to get permission from the network before they
> write to the disk, and they're not going to get permission without
> hearing back from everybody. Besides, since the same network is used
> to connect to the disks as is used to connect the computers to each
> other, how would it be able to access the disks without being able to
> access other computers which also connect to the disks?

You really should read up about split-brain scenarios, quorum, IO
fencing, cluster membership algorithms and the amazing variety of
different types of crashes.


Sincerely,
    Lars Marowsky-Brée

-- 
High Availability & Clustering
SUSE Labs, Research and Development
SUSE LINUX Products GmbH - A Novell Business	 -- Charles Darwin
"Ignorance more frequently begets confidence than does knowledge"

Re: Fwd: Fwd: Extended Volume Manager API

[Circuitsoft Development]

On 5/8/06, Lars Marowsky-Bree <lmb@suse.de> wrote:
> On 2006-05-08T16:17:07, Circuitsoft Development <circuitsoft.devel@gmail.com> wrote:
>
> > I'm not. They also need to get permission from the network before they
> > write to the disk, and they're not going to get permission without
> > hearing back from everybody. Besides, since the same network is used
> > to connect to the disks as is used to connect the computers to each
> > other, how would it be able to access the disks without being able to
> > access other computers which also connect to the disks?
>
> You really should read up about split-brain scenarios

I don't see how they'll happen if the heartbeat/management runs over
the same network that is used to connect to the disk.

> quorum

I'm aware of the idea. I think that a static quorum would be best, and
that it should be configured by the cluster administrator. See
http://lists.osdl.org/pipermail/osdlcluster/2004-January/000071.html
for a description of the problem with dynamic quorum.

> IO fencing

The primary target storage protocol is ATA-over-Ethernet, second is
iSCSI. As far as I know, it should be relatively simple, in both
circumstances, to tell the storage blade to cut off a computer until
it correctly re-registers itself with the cluster. Otherwise, a CISCO
managed switch should also be able to cut off a computer if it stops
responding to the cluster.

> cluster membership algorithms

Having trouble finding too many details on these. I'll keep looking,
but some pointers could be helpful.

> and the amazing variety of different types of crashes.

I figured that IO Fencing combined with STONITH (Shoot The Other Node
In The Head) could solve the problems caused by most crashes.

>
> Sincerely,
>     Lars Marowsky-Brée
>
> --
> High Availability & Clustering
> SUSE Labs, Research and Development
> SUSE LINUX Products GmbH - A Novell Business     -- Charles Darwin
> "Ignorance more frequently begets confidence than does knowledge"
>

These thoughts are based on my best understanding of how-stuff-works
so far. Any further comments would be greatly appreciated.


- Alex