Yet another design for /proc. Or actually /kernel.

Yet another design for /proc. Or actually /kernel.

[Erik Hensema]

Here's my go at a new design for /proc. I designed it from a userland
point of view and tried not to drown myself into details.

Design goals
============

- Backward compatibility
	The current /proc interface is coded into many applications.
	A new design must leave room for backward compatibility.
- Forward compatibility
	The new interface must be clearly defined and must not change
	in the future, unless that change is clearly indicated and it
	would cause horrible pain maintaining the current interface.
	(Note: I mean the interface to userland, kernel interfaces may
	change as much as they like)
- Parsable by applications
	Information in /proc isn't really meant for humans to read.
	Maybe programmers, but they aren't human anyway. The new /proc
	interface must be parsable by applications as easily as
	possible.
- Readable by humans
	When there's no compelling need to obscure information, don't.
- As simple as possible
	Don't bloat the interface with all kinds of fancy decorations.
	Let's leave that to userspace.


>From these goals I come to the following design:

Design
======

- Multiple values per file when needed
	A file is a two dimensional array: it has lines and every line
	can consist of multiple fields.
	A good example of this is the current /proc/mounts.
	This can be parsed very easily in all languages.
	No need for single-value files, that's oversimplification.
- No headers, no pretty formatting
	Headers are nice to humans, but applications tend to dislike
	them. Same goes for formatting.
	An example of this is in /proc/partitions or -- horrors --
	/proc/meminfo.
- ASCII values
	ASCII is the only format which can be read by humans, shell
	scripts, perl, C and the rest of the world.
- Maybe we should move out of /proc
	Since this design must leave room for backwards compatibility
	we must assume the current /proc stays around for a while. I
	propose we create a new kernelfs which is to be mounted at
	/kernel.
- Mandatory documentation
	Each and every value must be documented. This documentation
	may be very, very basic, but it must clearly define the entry
	and its intention. This is forward compatibility.


As an example, this is how I would organize the info currently in
/proc/meminfo (any boy, that file is UGLY!):

Every value is in bytes. Rounding to kbytes or mbytes should be done
in userspace.

New		Current entry in /proc/meminfo
--------------- -----------------------------------
mem/total	MemTotal
mem/free	MemFree
mem/shared	MemShared
mem/buffers	Buffers
mem/cached	Cached
mem/active	Active
mem/inactive	Inactive

mem/high/total	HighTotal
mem/high/free	HighFree

mem/low/total	LowTotal
mem/low/free	LowFree

mem/swap/total	SwapTotal
mem/swap/free	SwapFree
mem/swap/cached	SwapCached
mem/swap/files	(current /proc/swaps)
-- 
Erik Hensema (erik@hensema.net)
I'm on the list, no need to Cc: me, though I appreciate one if your
mailer doesn't support the References header.

Re: Yet another design for /proc. Or actually /kernel.

[Alan Cox]

> Here's my go at a new design for /proc. I designed it from a userland
> point of view and tried not to drown myself into details.

Did you have to change the subject line. It makes it harder to kill file 
when people keep doing that

/proc discussions

[Tim Jansen]

On Wednesday 07 November 2001 20:27, Alan Cox wrote:
> > Here's my go at a new design for /proc. I designed it from a userland
> > point of view and tried not to drown myself into details.
> Did you have to change the subject line. It makes it harder to kill file
> when people keep doing that

There is an easy way for you, or even better, Linus to stop these discussions:

Just say, in unambigous words, what kind of patch you would accept, if any. 
The open questions are: in which format should data (from driver and kernel) 
be exported? How can the kernel, drivers and devices be configured from user 
space?
Even better, you could also say a few words about the possible disadvantages 
that the decision has and why we shouldn't care. Something like "performance 
is irrelevant", "it doesn't matter if the format cannot be changed without 
breaking user applications", "for every format there must be a strict 
definition of how to parse it so that the new fields can be added later", "we 
don't need to define the format as long at it is readable for human readers", 
"if you need a pretty format write some user space tool that formats it", "it 
doesnt matter if each file uses a completely different format", "it is not 
acceptable to break any user space applications before 2.x", "strict typing 
is irrelevant"...


I REALLY do have problems to define any proc interface without having at 
least one of the disadvantages above. I have spent some time to find LT 
statements about proc/sysctl in order to find out which way may have a chance 
to get into the kernel (this time is possibly better spent than writing 
another proc replacement :). Maybe I have missed some important ones, in 
essence they are:

4 Nov 2001:
"In short: /proc is ASCII, and will so remain while I maintain a kernel.
Anything else is stupid.
Handling spaces and newlines is easy enough - see the patches from Al
Viro, for example."

4 Nov 2000:
"[context added]
>        create_proc_entries(NULL,
>                "test:{bar:{x:%d,y:%d,z:%d},foo:%f}",
>                &x, &y, &z, foo_fun);
>
> creates a "/proc/test" directory, which further contains a
> subdirectory "bar" and a file "foo". The "bar" subdirectory contains
> three files "x", "y" and "z".
> 
> The formatting argument "%d" takes a pointer to an integer. When
> reading such a file (in this case "x", "y", or "z"), the value is
> shown as ascii. Writing to the file (again in ascii) updates the
> value. The "%f" formatting argument allows you to pass an arbitrary
> user function for generating output. Clearly, there are potentially
> quite a number of standard/useful formatting arguments.
> 
> I've done a quick, dirty, unfinished implementation of this idea, so
> people can get the picture. Attached.
> 
> Many people will hate this because (1) it's doing parsing within the
> kernel, (2) it tends to favour ascii I/O, (3) it tends to favour deep
> directory hierarchies,(4) it uses recursion :-)

I like the idea a lot: it's simple, it's clean, and it does make it truly
trivial to build up a simple /proc architecture.
"

7 Jan 2000:
"In contrast, sysctl isn't all that performance-sensitive, AND they are
extremely hierarchical, AND they depend on configuration and timing. 

In short, sysctl NEEDS:
 - "naming": you cannot name the sysctl space with a number: it is much
   too dynamic for that. How do you enumerate drivers? Give them random
   numbers?
 - "listing": showing which sysctl's are there, in a hierarchical manner.
   Again, a listing is useless with a number.
 - "hierarchy". You have different devices, but they have the same
   controls. Do they get the same name? Yes. But in different places in
   the hierarchy.

In short, you NEED a filesystem. You need to be able to "ls" the thing.
You need to be able to search the thing. You need to be doing all the
things you can do with a real filesystem."

6 Jan 2000:
"The thing to do is to create a

        /proc/drivers/<drivername>/ 

directory. The /proc/drivers/ directory is already there, so you'd
basically do something like

        create_proc_info_entry("driver/mydriver/status", 0, NULL, 
mydriver_status_read);
"

1 Jan 1998:
"- I think it should be a separate filesystem type. It was a mistake in
 the first place to overload /proc as much as we did, we shouldn't
 continue that (I should really have split out the "per-process" things
 and the "global" things to two separate filesystems). "


Is the 4-Nov-2000 mail combined with 1-Jan-1998 the answer?

bye...

Re: /proc discussions

[Alexander Viro]

On Thu, 8 Nov 2001, Tim Jansen wrote:

> There is an easy way for you, or even better, Linus to stop these discussions:
> 
> Just say, in unambigous words, what kind of patch you would accept, if any. 

.procmailrc one would do nicely.

Re: Yet another design for /proc. Or actually /kernel.

[Daniel R. Warner]

Erik Hensema wrote:

> Here's my go at a new design for /proc. I designed it from a userland
> point of view and tried not to drown myself into details.

<snip>

> - Multiple values per file when needed
> 	A file is a two dimensional array: it has lines and every line
> 	can consist of multiple fields.
> 	A good example of this is the current /proc/mounts.
> 	This can be parsed very easily in all languages.


> 	No need for single-value files, that's oversimplification.
<snip>


This is fine for reading, but it makes it harder for humans to change 
values in /proc - eg, echo 0 > /proc/sys/net/ipv4/tcp_ecn

-D

Re: Yet another design for /proc. Or actually /kernel.

[Allen Campbell]

On Wed, Nov 07, 2001 at 02:42:48PM -0500, Daniel R. Warner wrote:
> Erik Hensema wrote:
> 
> > - Multiple values per file when needed
> > 	A file is a two dimensional array: it has lines and every line
> > 	can consist of multiple fields.
> > 	A good example of this is the current /proc/mounts.
> > 	This can be parsed very easily in all languages.
> 
> 
> > 	No need for single-value files, that's oversimplification.
> <snip>
> 
> 
> This is fine for reading, but it makes it harder for humans to change 
> values in /proc - eg, echo 0 > /proc/sys/net/ipv4/tcp_ecn
> 

'Multiple value' files can be made easy to 'write'.  The only
requirement is each 'field' in the file have a unique label.  Then
it's a common associative array, requiring some generic filesystem
write magic to handle the input:

echo "label:1" > /proc/...

The 'generic write magic' would require (at least, without even
more magic) that all /proc files conform to the schema.  This is
probably a _good_ thing.

-- 
  Allen Campbell       |  Lurking at the bottom of the
  allenc@verinet.com   |   gravity well, getting old.

Re: Yet another design for /proc. Or actually /kernel.

[H. Peter Anvin]

Followup to:  <slrn9uj1nf.5lj.spamtrap@dexter.hensema.xs4all.nl>
By author:    spamtrap@use.reply-to (Erik Hensema)
In newsgroup: linux.dev.kernel
> 
> - Multiple values per file when needed
> 	A file is a two dimensional array: it has lines and every line
> 	can consist of multiple fields.
> 	A good example of this is the current /proc/mounts.
> 	This can be parsed very easily in all languages.
> 	No need for single-value files, that's oversimplification.
> 

Actually, /proc/mounts is currently broken, and is an excellent
example of why the above statement simply isn't true unless you apply
another level of indirection: try mounting something on a directory
the name of which contains whitespace in any form (remember, depending
on your setup this may be doable by an unprivileged user...)

	-hpa
-- 
<hpa@transmeta.com> at work, <hpa@zytor.com> in private!
"Unix gives you enough rope to shoot yourself in the foot."
http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>

Re: Yet another design for /proc. Or actually /kernel.

[Justin A]

Ahh yes I've seen that happen.  It's not too common in standard unix
practices, but comes up a lot when mounting windows shares.  Mounting
"//bobs computer/my folder with stuff in it" does not have good results.
Only way I found to fix that was to umount -a -t smbfs.

-Justin

On Wed, Nov 07, 2001 at 12:58:24PM -0800, H. Peter Anvin wrote:
> Followup to:  <slrn9uj1nf.5lj.spamtrap@dexter.hensema.xs4all.nl>
> By author:    spamtrap@use.reply-to (Erik Hensema)
> In newsgroup: linux.dev.kernel
> > 
> > - Multiple values per file when needed
> > 	A file is a two dimensional array: it has lines and every line
> > 	can consist of multiple fields.
> > 	A good example of this is the current /proc/mounts.
> > 	This can be parsed very easily in all languages.
> > 	No need for single-value files, that's oversimplification.
> > 
> 
> Actually, /proc/mounts is currently broken, and is an excellent
> example of why the above statement simply isn't true unless you apply
> another level of indirection: try mounting something on a directory
> the name of which contains whitespace in any form (remember, depending
> on your setup this may be doable by an unprivileged user...)
> 
> 	-hpa
> -- 
> <hpa@transmeta.com> at work, <hpa@zytor.com> in private!
> "Unix gives you enough rope to shoot yourself in the foot."
> http://www.zytor.com/~hpa/puzzle.txt	<amsp@zytor.com>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at  http://www.tux.org/lkml/

Re: Yet another design for /proc. Or actually /kernel.

[Rusty Russell]

On 7 Nov 2001 19:09:35 GMT
spamtrap@use.reply-to (Erik Hensema) wrote:

> 
> Here's my go at a new design for /proc. I designed it from a userland
> point of view and tried not to drown myself into details.

> - Multiple values per file when needed
> 	A file is a two dimensional array: it has lines and every line
> 	can consist of multiple fields.
> 	A good example of this is the current /proc/mounts.
> 	This can be parsed very easily in all languages.
> 	No need for single-value files, that's oversimplification.

No, it deals nicely with any possible values in the file.  And without headers,
how do I know what's what?  And how do I update one value.

Meanwhile, there's far too much talk, far too little code.  Will post new patch
next week.

Rusty.

Re: Yet another design for /proc. Or actually /kernel.

[Stephen Satchell]

At 12:58 PM 11/7/01 -0800, H. Peter Anvin wrote:
>Actually, /proc/mounts is currently broken, and is an excellent
>example of why the above statement simply isn't true unless you apply
>another level of indirection: try mounting something on a directory
>the name of which contains whitespace in any form (remember, depending
>on your setup this may be doable by an unprivileged user...)

Good man.

Now you know why I insisted that ALL STRINGS have a standard escape system 
in my less-than-strawman proposal.

You'll find other places in /proc where white space is not necessarily a 
delimiter.  (Too lazy to go find them right now, however.)

Satch