Re: [PATCH] x86/sgx: Reduce delay and interference of enclave release

[Reinette Chatre <reinette.chatre@intel.com>]

Hi Jarkko,

On 10/23/2022 1:06 PM, Jarkko Sakkinen wrote:
> On Tue, Oct 18, 2022 at 03:42:47PM -0700, Reinette Chatre wrote:

...

>> diff --git a/arch/x86/kernel/cpu/sgx/encl.c b/arch/x86/kernel/cpu/sgx/encl.c
>> index 1ec20807de1e..f7365c278525 100644
>> --- a/arch/x86/kernel/cpu/sgx/encl.c
>> +++ b/arch/x86/kernel/cpu/sgx/encl.c
>> @@ -682,9 +682,12 @@ void sgx_encl_release(struct kref *ref)
>>  	struct sgx_encl *encl = container_of(ref, struct sgx_encl, refcount);
>>  	struct sgx_va_page *va_page;
>>  	struct sgx_encl_page *entry;
>> -	unsigned long index;
>> +	unsigned long count = 0;
>> +
>> +	XA_STATE(xas, &encl->page_array, PFN_DOWN(encl->base));
>>  
>> -	xa_for_each(&encl->page_array, index, entry) {
>> +	xas_lock(&xas);
>> +	xas_for_each(&xas, entry, PFN_DOWN(encl->base + encl->size  - 1)) {
> 
> I would add to declarations:
> 
> unsigned long nr_pages = PFN_DOWN(encl->base + encl->size  - 1);
> 
> Makes this more readable.

Will do, but I prefer to name it "max_page_index" or something related instead.
"nr_pages" implies "number of pages" to me, which is not what
PFN_DOWN(encl->base + encl->size - 1) represents. What is represented is the
highest possible index of a page in page_array, where an index is the
pfn of a page.

> 
>>  		if (entry->epc_page) {
>>  			/*
>>  			 * The page and its radix tree entry cannot be freed
>> @@ -699,9 +702,20 @@ void sgx_encl_release(struct kref *ref)
>>  		}
>>  
>>  		kfree(entry);
>> -		/* Invoke scheduler to prevent soft lockups. */
>> -		cond_resched();
>> +		/*
>> +		 * Invoke scheduler on every XA_CHECK_SCHED iteration
>> +		 * to prevent soft lockups.
>> +		 */
>> +		if (!(++count % XA_CHECK_SCHED)) {
>> +			xas_pause(&xas);
>> +			xas_unlock(&xas);
>> +
>> +			cond_resched();
>> +
>> +			xas_lock(&xas);
>> +		}
>>  	}
> 
>         WARN_ON(count != nr_pages);
> 

nr_pages as assigned in your example does not represent a count of the
enclave pages but instead a pfn index into the page_array. Comparing it
to count, the number of removed enclave pages that are not being held
by reclaimer, is not appropriate.

This check would be problematic even if we create a "nr_pages" from
the range of possible indices. This is because of how enclave sizes are
required to be power-of-two that makes it likely for there to be indices
without pages associated with it.

>> +	xas_unlock(&xas);
>>  
>>  	xa_destroy(&encl->page_array);
>>  
>> -- 
>> 2.34.1
>>

Reinette