Re: IRQ affinities (was: boot cgroup questions)

[Paul Jackson <pj@sgi.com>]

pj, talking to himself:
> That is, for example, one might want all the CPUs in cpusets
> foo, bar and baz, except the CPUs in cpuset blip, to handle
> IRQs so and so.

Ahh!  Perhaps that example has the keys to this kingdom.

How about this.  We add two files to each cpuset:

    irq_affinity_include	# IRQs to direct to CPUs in this cpuset
    irq_affinity_exclude	# IRQs -not- to direct to these CPUs

where irq_affinity_exclude overrides irq_affinity_include.

So, to determine to which CPUs a given interrupt (IRQ) can be directed:
 1) Combine (union) the 'cpus' of all the cpusets for which
    that IRQ is in that cpusets irq_affinity_include, then
 2) Remove (set substraction) the 'cpus' of any cpuset for which
    that IRQ is in that cpusets irq_affinity_exclude.

In the simplest case of just wanting to isolate some CPUs with their
own special list of interrupts, one would:
 1) include all interrupts in the top cpusets irq_affinity_include, and
 2) include the interrupts you don't want in the isolated cpusets
    irq_affinity_exclude.

Observe that there is no dependency on the cpuset hierarchy in the above.

The contents of the files irq_affinity_include and irq_affinity_exclude
would be inherited by child cpusets on creation from their parents.

The one detail that puzzles me at the moment is what ownership and
permissions these two irq_affinity_* files would have.  I am concerned
that the usual permissions, which allow a job to write its own cpuset
files would allow a job to affect the overall system to a greater
degree than is desired.  Perhaps an additional inheritance rule would
be useful and appropriate, such as a rule that a given cpusets
irq_affinity_include must be a subset of its parents or a rule that
a given cpusets irq_affinity_exclude must be a -superset- of its
parents; I'm unsure here.

-- 
                  I won't rest till it's the best ...
                  Programmer, Linux Scalability
                  Paul Jackson <pj@sgi.com> 1.940.382.4214