Re: [PATCH V15 00/11] x86: Intel Cache Allocation Technology Support

[Marcelo Tosatti <mtosatti@redhat.com>]

On Thu, Oct 15, 2015 at 01:37:02PM +0200, Peter Zijlstra wrote:
> On Tue, Oct 13, 2015 at 07:40:58PM -0300, Marcelo Tosatti wrote:
> > How can you fix the issue of sockets with different reserved cache
> > regions with hw in the cgroup interface?
> 
> No idea what you're referring to. But IOCTLs blow.

Tejun brought up syscalls. Syscalls seem too generic.
So ioctls were chosen instead.

It is necessary to perform the following operations:

1) create cache reservation (params = size, type).
2) delete cache reservation.
3) attach cache reservation (params = cache reservation id, pid).
4) detach cache reservation (params = cache reservation id, pid).

Can it done via cgroups? If so, works for me.

A list of problems with the cgroup interface has been written,
in the thread... and we found another problem.


List of problems with cgroup interface:

1) Global IPI on CBM <---> task change does not scale.

 * cbm_update_all() - Update the cache bit mask for all packages.
 */
static inline void cbm_update_all(u32 closid)
{
       on_each_cpu_mask(&rdt_cpumask, cbm_cpu_update, (void *)closid,
1);
}

Consider a machine with 32 sockets.

2) Syscall interface specification is in kbytes, not
cache ways (which is what must be recorded by the OS
to allow migration of the OS between different
hardware systems).

3) Compilers are able to configure cache optimally for
given ranges of code inside applications, easily,
if desired.

4) Problem-2: The decision to allocate cache is tied to application
initialization / destruction, and application initialization is
essentially random from the POV of the system (the events which trigger
the execution of the application are not visible from the system).

Think of a server running two different servers: one database
with requests that are received with poisson distribution, average 30
requests per hour, and every request takes 1 minute.

One httpd server with nearly constant load.

Without cache reservations, database requests takes 2 minutes.
That is not acceptable for the database clients.
But with cache reservation, database requests takes 1 minute.

You want to maximize performance of httpd and database requests
What you do? You allow the database server to perform cache
reservation once a request comes in, and to undo the reservation
once the request is finished.

Its impossible to perform this with a centralized interface.

5) Modify scenario 2 above as follows: each database request
is handled by two newly created threads, and they share a certain
percentage
of data cache, and a certain percentage of code cache.

So the dispatcher thread, on arrival of request, has to:

        - create data cache reservation = tcrid-A.
        - create code cache reservation = tcrid-B.
        - create thread-1.
        - assign tcird-A and B to thread-1.
        - create thread-2.
        - assign tcird-A and B to thread-2.

6) Create reservations in such a way that the sum is larger than
total amount of cache, and CPU pinning (example from Karen Noel):

VM-1 on socket-1 with 80% of reservation.
VM-2 on socket-2 with 80% of reservation.
VM-1 pinned to socket-1.
VM-2 pinned to socket-2.

Cgroups interface attempts to set a cache mask globally. This is the
problem the "expand" proposal solves:
https://lkml.org/lkml/2015/7/29/682

7) Consider two sockets with different region of L3 cache
shared with HW:

— CPUID.(EAX=10H, ECX=1):EBX[31:0] reports a bit mask. Each set bit
within the length of the CBM
indicates the corresponding unit of the L3 allocation may be used by
other entities in the platform (e.g. an
integrated graphics engine or hardware units outside the processor core
and have direct access to L3).
Each cleared bit within the length of the CBM indicates the
corresponding allocation unit can be configured
to implement a priority-based allocation scheme chosen by an OS/VMM
without interference with other
hardware agents in the system. Bits outside the length of the CBM are
reserved.

You want the kernel to maintain different bitmasks in the CBM:

        socket1 [range-A]
        socket2 [range-B]

And the kernel will automatically switch from range A to range B
when the thread switches sockets.

---------------------

Problems 6, 7 and 2 are fatal for us. If you can fix them in the cgroup
interface, we can use it (please understand these problems, you seem to 
ignore them for some reason).

Problems 1 4 and 5 seem to come from Tejun.

Problem 3 could be a possibility.