Re: [PATCH 5/9] x86/intel_rdt: Add new cgroup and Class of service management

[Marcelo Tosatti <mtosatti@redhat.com>]

On Sun, Aug 02, 2015 at 12:23:25PM -0400, Tejun Heo wrote:
> Hello,
> 
> On Fri, Jul 31, 2015 at 12:12:18PM -0300, Marcelo Tosatti wrote:
> > > I don't really think it makes sense to implement a fully hierarchical
> > > cgroup solution when there isn't the basic affinity-adjusting
> > > interface 
> > 
> > What is an "affinity adjusting interface" ? Can you give an example
> > please?
> 
> Something similar to sched_setaffinity().  Just a syscall / prctl or
> whatever programmable interface which sets per-task attribute.

You really want to specify the cache configuration "at once": 
having process-A exclusive access to 2MB of cache at all times,
and process-B 4MB exclusive, means you can't have process-C use 4MB of 
cache exclusively (consider 8MB cache machine).

But the syscall allows processes to set and retrieve

> > > and it isn't clear whether fully hierarchical resource
> > > distribution would be necessary especially given that the granularity
> > > of the target resource is very coarse.
> > 
> > As i see it, the benefit of the hierarchical structure to the CAT
> > configuration is simply to organize sharing of cache ways in subtrees
> > - two cgroups can share a given cache way only if they have a common
> > parent. 
> > 
> > That is the only benefit. Vikas, please correct me if i'm wrong.
> 
> cgroups is not a superset of a programmable interface.  It has
> distinctive disadvantages and not a substitute with hirearchy support
> for regular systemcall-like interface.  I don't think it makes sense
> to go full-on hierarchical cgroups when we don't have basic interface
> which is likely to cover many use cases better.  A syscall-like
> interface combined with a tool similar to taskset would cover a lot in
> a more accessible way.

How are you going to specify sharing of portions of cache by two sets
of tasks with a syscall interface?

> > > I can see that how cpuset would seem to invite this sort of usage but
> > > cpuset itself is more of an arbitrary outgrowth (regardless of
> > > history) in terms of resource control and most things controlled by
> > > cpuset already have countepart interface which is readily accessible
> > > to the normal applications.
> > 
> > I can't parse that phrase (due to ignorance). Please educate.
> 
> Hmmm... consider CPU affinity.  cpuset definitely is useful for some
> use cases as a management tool especially if the workloads are not
> cooperative or delegated; however, it's no substitute for a proper
> syscall interface and it'd be silly to try to replace that with
> cpuset.
> 
> > > Given that what the feature allows is restricting usage rather than
> > > granting anything exclusively, a programmable interface wouldn't need
> > > to worry about complications around priviledges
> > 
> > What complications about priviledges you refer to?
> 
> It's not granting exclusive access, so individual user applications
> can be allowed to do whatever it wanna do as long as the issuer has
> enough priv over the target task.

Priviledge management with cgroup system: to change cache allocation
requires priviledge over cgroups.

Priviledge management with system call interface: applications 
could be allowed to reserve up to a certain percentage of the cache.

> > > while being able to reap most of the benefits in an a lot easier way.
> > > Am I missing something?
> > 
> > The interface does allow for exclusive cache usage by an application.
> > Please read the Intel manual, section 17, it is very instructive.
> 
> For that, it'd have to require some CAP but I think just having
> restrictive interface in the style of CPU or NUMA affinity would go a
> long way.
> 
> > The use cases we have now are the following:
> > 
> > Scenario 1: Consider a system with 4 high performance applications
> > running, one of which is a streaming application that manages a very
> > large address space from which it reads and writes as it does its processing.
> > As such the application will use all the cache it can get but does
> > not need much if any cache. So, it spoils the cache for everyone for no
> > gain on its own. In this case we'd like to constrain it to the
> > smallest possible amount of cache while at the same time constraining
> > the other 3 applications to stay out of this thrashed area of the
> > cache.
> 
> A tool in the style of taskset should be enough for the above
> scenario.
> 
> > Scenario 2: We have a numeric application that has been highly optimized
> > to fit in the L2 cache (2M for example). We want to ensure that its
> > cached data does not get flushed from the cache hierarchy while it is
> > scheduled out. In this case we exclusively allocate enough L3 cache to
> > hold all of the L2 cache.
> >
> > Scenario 3: Latency sensitive application executing in a shared
> > environment, where memory to handle an event must be in L3 cache
> > for latency requirements to be met.
> 
> Either isolate CPUs or run other stuff with affinity restricted.
> 
> cpuset-style allocation can be easier for things like this but that
> should be an addition on top not the one and only interface.  How is
> it gonna handle if multiple threads of a process want to restrict
> cache usages to avoid stepping on each other's toes?  Delegate the
> subdirectory and let the process itself open it and write to files to
> configure when there isn't even a way to atomically access the
> process's own directory or a way to synchronize against migration?

One would preconfigure that in advance - but you are right, a 
syscall interface is more flexible in that respect.

> cgroups may be an okay management interface but a horrible
> programmable interface.
> 
> Sure, if this turns out to be as important as cpu or numa affinity and
> gets widely used creating management burden in many use cases, we sure
> can add cgroups controller for it but that's a remote possibility at
> this point and the current attempt is over-engineering solution for
> problems which haven't been shown to exist.  Let's please first
> implement something simple and easy to use.
> 
> Thanks.
> 
> -- 
> tejun