From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753880AbcG2VUG (ORCPT ); Fri, 29 Jul 2016 17:20:06 -0400 Received: from mail-db5eur01on0071.outbound.protection.outlook.com ([104.47.2.71]:57268 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1753208AbcG2VT5 (ORCPT ); Fri, 29 Jul 2016 17:19:57 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=cmetcalf@mellanox.com; Subject: Re: clocksource_watchdog causing scheduling of timers every second (was [v13] support "task_isolation" mode) To: Francis Giraldeau References: <1468529299-27929-1-git-send-email-cmetcalf@mellanox.com> <7a3f66f7-5011-7d59-2e0e-f57e4e42e6b6@mellanox.com> <2fefa17d-37c6-9669-724e-9ee0d841e7b2@mellanox.com> CC: Christoph Lameter , Gilad Ben Yossef , Steven Rostedt , Ingo Molnar , "Peter Zijlstra" , Andrew Morton , Rik van Riel , Tejun Heo , "Frederic Weisbecker" , Thomas Gleixner , "Paul E. McKenney" , Viresh Kumar , Catalin Marinas , "Will Deacon" , Andy Lutomirski , "Daniel Lezcano" , , , "linux-kernel@vger.kernel.org" From: Chris Metcalf Message-ID: <6f4107fb-9d42-d4a3-8681-317cf7175bf9@mellanox.com> Date: Fri, 29 Jul 2016 17:04:05 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [12.216.194.146] X-ClientProxiedBy: DM2PR0501CA0022.namprd05.prod.outlook.com (10.162.29.160) To VI1PR0501MB2768.eurprd05.prod.outlook.com (10.172.11.18) X-MS-Office365-Filtering-Correlation-Id: 92f75e3c-a3c0-4f23-67db-08d3b7f3e8c5 X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2768;2:Y/IoN+WttDhu+MggoZI0mvZmRZKSCcg7CVwk+S0LZmyfhawHaVqw66ZdNi3r1YrcZFQ/XSemBnn/+G2dnZr0S/NaNQpXxvCDLpJM18OMbgCsLoC7/KrITF+LL7Rvmy8YqQzRo3ofKwglK72xUScRsUx9pgt4awpfbLgan2KrW/o/KhRyGjJj9yYu2UatlsUn;3:EpVwF/PSXdYGM3dvljuOBYkFKu/wrbuLpCH3YZ1tj8vQpfW1+VAbYg3KBW2blzDuhnTyFgBP+OKWHt3QLyoHeVHZvnGzwqEeODY+G7JLYylJu5k3x1YLjSMXWxEC8Ehy;25:Crq+ABCAEoPo1Zi28nX2r6v7Qc5rMD5wlwYzHA3rCJS5JhvrH9lYuHnv5NoH6OT8XmrAUiNnDTO+uZTVZ9fqqlrp95qXHcWCD8UAuNqaCv5IIz54rAcLV5SAGZbk7uKSV1v9oKWOCAThndt/Rq8A5pNMY56RBhkuF+jEA36bsikaVtz/CGnqQdVPoa3WuFwfvLqYFCgkp1g9zPhfVuLyyJS7wxBmyF4lVJkhhq30Pg0YwqKVRiun1sJD4sHNCZ0jrHycEZGd3Gjo267uIBR0nTWewIar7ulnHxAtUm7AwbKJP3hiusdwclklQPCTBF5xbc4rvAs35oVG6tlpV0x+Zb77JlqxqNVitJhY6eQbJAvUjesLG/Rd+xHFEGjYf17O4AMpvLU2+zNS973HA/qz4Q37it8Oc/boI+SzJFuR9fk= X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:VI1PR0501MB2768; X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2768;31:gT9XYkRdtftE2IK+s6XtWYfe2uFVbwKdinNoc1g3oLMBIry4PZTJM7OoaTcO/wpe3b6sAqoZ+X3pASI+L+Y7y6vpi4zDzhA1TLLzEBfg7sSxdpuIMlANo7rWCQBsKJOKhpTSeRDT5D7V8zF6SmP9+73IUspJq3+IKGE6fT7LY3asbVVBnTmL4nLUHHeK6JZnZmSP30nLMGz5DHQtZb9Hmw==;20:DAMaHHfWNoVdnN81UV5hqsUaKzLEww7Zsc3siRPxL0d7QtWjmg4tn4m7oza9+rnCdJhneX4fn4jLUn+z45rCIhYmfsG7OhJXMh8lD7+QGu3f4/0UHIu7RUmGs2j855BfJVmEYEJkgq/WMicguq7X/8K8cY7raqPLuANGJwzm9pDQaJZRaSMAXxaGrTaxqx1A4KoFa9Hwxbu5R0anOY5/mljQpVNREiWyzlv9FcfobemZYSyt5AVpBto1cAkpA//jNQoBiehOCRnA/kUx9c3j5PvE8n08VZltiIrRbJ8cvzEtInkX4qCut6sMjn04pHlczDJANat/lSb8TfkaBDFl8eJ1KwBkyZpp72ONvBxsTy/UML8tUSJtL0UzRipD9yL965ahghpgvprH6Rwc8m3PYOAnWviI8VovlBA97E0L69+EtSzPE2XVmxyesRFYA2cRDta3Vg3LBe643dW7gXq16mVfZfn0xST8EcmX5lhhMiJ/n5cJY/3aSOcU1FMFcgNz X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(166708455590820)(171992500451332); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(8121501046)(3002001)(10201501046)(6055026);SRVR:VI1PR0501MB2768;BCL:0;PCL:0;RULEID:;SRVR:VI1PR0501MB2768; X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2768;4:5PaG01s0YNV8azrvkQzmAdk7Tq5njj49FArN7YFs7f7eNVGQXr83GGXJFRexmdl59AXqRC9R3ITSjSMLH147DvjQUVLwgrgld0ZEM/GtqSOlHddctiIlzv1t8O//ZJL9cUiaOOpdREekcWpNY67fJFt2WlvEiEK7iPrOychUroj0PFtvtVjZpa4ugAD1hxZrXipNVS0IwJo2WABlvw9PmeCV2uRXwciTAgDw/tyPOVxDPivRctZREK/0dBc7C3XEKIXCY13ktMboafeTqTra0QoSQm9Sm8wgeg3jNcFrUO0aOh6vYG3L3TYYg/YWcni6gZgrsjhk6Vz6K7zZN3fVsA8uuU0BvjBOo9Z3vnd+2GsO6UL+WG4JmpiB2r9kIhhwDPjabTVA28PaQkdnWt5dJjtRDiwfHL0Q6hQ04bgZS33BqdB9ea2Agu6TOxxEdtnDw0Xq3dJTk+cl890RIekSNg== X-Forefront-PRVS: 0018A2705B X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6049001)(6009001)(7916002)(24454002)(189002)(199003)(377454003)(4326007)(23676002)(81156014)(81166006)(105586002)(33646002)(4001350100001)(305945005)(92566002)(47776003)(101416001)(77096005)(68736007)(97736004)(7416002)(86362001)(19580395003)(31696002)(8676002)(2950100001)(50466002)(230700001)(36756003)(2906002)(65956001)(7736002)(7846002)(189998001)(15975445007)(83506001)(586003)(64126003)(50986999)(66066001)(110136002)(54356999)(31686004)(65806001)(3846002)(42186005)(106356001)(93886004)(6116002)(76176999)(65826006)(18886065003);DIR:OUT;SFP:1101;SCL:1;SRVR:VI1PR0501MB2768;H:[10.15.7.181];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?utf-8?B?MTtWSTFQUjA1MDFNQjI3Njg7MjM6bHhkOElPeWc1eFc1bUdBMXpobjR5L1BM?= =?utf-8?B?TnJoQytoaytZWXZDMkVsb3ZNMDRBazBGZTVyK253allRaHpGaXZxQUxJK0pE?= =?utf-8?B?S1BzTkFFb1VrQ1hsK2UzU1cxL1dXUzJjZ0NuaG1mbzRnN1E3bFZ2V2tHQ2tn?= =?utf-8?B?RnllUnlhYWcvcHh0MEZ1d2NkWXJkTWlzMFNqYXhSeDhrUTlDZEJxU2kremRJ?= =?utf-8?B?SGRLd2tiUGdOM3V1R2tXTm0rd3BoMWhieWVYeDVvWUlERzBjRlI5RlJ4Tmpj?= =?utf-8?B?WVI3dVhVQ1QveWxhMXRKa09qMFoxM1BMcUl5Wmpsa3NQSXVOK1dZa0VKWjlI?= =?utf-8?B?OE5xTURMaFRrdS9DYkVhRjVVbHdPMTN1aS9yazU5OW04bENOQWpKR2xiNUdW?= =?utf-8?B?TFZYT0NSd0dUTXZ3LzBJWWJqKzhBcXduNENZVXVTUGxZcXFQcDY4eW9Ub21v?= =?utf-8?B?ZzhOS1cyM1V6MXA1TWlORTNlNGcxeDJoU3JrRGg4dTV6WEpFSzI5djlFNnow?= =?utf-8?B?ejAzSkF0bVBEb3VJMnN2amplek9iNVlkaE91SUJCZ2Q3SUpUUitJdlZxcC9W?= =?utf-8?B?czIvY2ttNkw1Ylg5WHcyMDhNVlVSRkxJWjdjN0lXbTBTZ21RMGI4QWlKUnhZ?= =?utf-8?B?N3hTOFdSckE5TDZaYU8za0tWeEIyRWczTHc1NlpsWTl5TXhXeFgvT242WjVH?= =?utf-8?B?Uk51ODNBK1RkbGtGLzcvWWw5ZEFCOEovZW5CVDVsZklGQU5yNXkrZFYxTHYz?= =?utf-8?B?SGpOWlZoVzl2UUwxemdTbFJCTnlWd3N5K08xcjdZUEhLd1hOeVpONEVpZGJR?= =?utf-8?B?QlJvSlQzV2l1UWN4bzZJeGplTUlpZ1FzRSs4SnpBdHA3NUI0Z1VMRWlidzhH?= =?utf-8?B?MzBoUm5sUnAvUVdnUXFMVWs2ZFdLWHVucnl3bWI4SDlHdTJIY3RjRG95UlIx?= =?utf-8?B?anZKb3BqRmo5YkxybFRqOHgxNUw3VXpCeXRZcXJqNnpjTmVkZVBmWjBEZnZn?= =?utf-8?B?a3ozT0pwQy9ZV2RORVdDVnJjN3piZ2VpeGQ3NmoxQ0M0UldBdGhBaWFoNUFv?= =?utf-8?B?VjVuUlRoNk56ZGdYNUZrRzdBdGd0djlHbG56UmIrcnBoYTVnOERieEp0dXB1?= =?utf-8?B?YXlibWV4K2MzY0tYMkN0MEkrSnRXODBmRjhNUmV4bmQ5MjZoL1ZjOThyOG1E?= =?utf-8?B?NjZUeEw0MzltUlBEWjBpa0NGK21FN3dWclZNdGF4YzhzMlpZejd6WmNKOWNC?= =?utf-8?B?YUZqb1VFdUxISWw0NUF3NkowUHEyUkVpWEpUd3FkNmVuQStYeG5nQmFUN1lU?= =?utf-8?B?YldFMnVVcVpjOHVmeXllb0d5NHlIWFpRbTU0S1VHbGhNSklkQXNOUnVGQW1k?= =?utf-8?B?U085TnBJWXFLb3FDdTFuS3JkU1dscEJKOUZvUUNodytkZTJsS0JSVUN1d0U3?= =?utf-8?B?TGpjS2VrYnRicVMySDE4THFvMlY0cmZxZ2xyU3B3QURseHcrOENLUkdYU1ZK?= =?utf-8?B?VEM3bUMranM3UVlEbUZRRFB5SEtYenIvcGdqZnJ5dkhyL2ptaFZoU0ZzWEta?= =?utf-8?B?eEJXRnkyZlFoZ3RMakt5REFoU0lYWUpLVWpzM1NTcUVrUlRkancycFZOTnRZ?= =?utf-8?B?SEtycStQczE4VnVtVGhReXB0RHNEdzJad2xVUTRvRVFTb1owcXVwTzRrS1NK?= =?utf-8?B?RXNrbjBIY3lvM2tqdVdtMkROaWlZVG1MaTl5V1kvdjFTZFQ3ei9MR2hML09i?= =?utf-8?B?UWZIUnYvOXBuTFB4R29OajRxZjJybkhidysxZ0ZBd2o2OTFZM2pGeUpJV3hl?= =?utf-8?Q?w6Yd5vYVhN0Segg?= X-Microsoft-Exchange-Diagnostics: 1;VI1PR0501MB2768;6:Wp5vUmFLbrYXvBbTNAhjguI6PsQ/rHCE/k5RvHoSK/3G89Suvzgv+uwLGxzky43ZcQ1f2lSgiV6eIEaKqsWqIbvMOuu5HuqUbS/W1gaej+5OJIZG2ZoZzfkbpes3YhXlWjnyt3CtC2jhLmKkk+ifZS1w7mPCmK5hJ/bI6Xi8813vgxg4bFKoj8DaaORpJBdoL0QCVu9F1tK3uScwpNBSn7aG8tOTmPL4g6Nreci/2gIuPUHaw6p54tHwkkvk5mwiJSgci5TbdjzdyYe9W37siORaHWgeQ+88YNNkZbOD9/Jm86YG4PQY10eYoRTlbgnoGgWVpSnZcEjZBN3eE+iW+Q==;5:cCGT+9YGW8733D4LDcNPJO+Eb2gEWsOcgNNLZ4SMkoxXpv4hwswr9t1UCalmYQy5sBmKi1OXpB2eo1gL9hE2mq0mHWUrge24DcliNXWKqWfmHoiqjJof/Rpi9/ZdYRk0bkbNoHh0vN2tqjhI66LAeQ==;24:W8gyWUiSRJvrmNmRMpsg6W7j53EG5bR1xwkq/r8RJ/c3AnOvkQU5Dngj4m5OepCGrcMNICvI+vkOgRQYwNFO6Q4xvBjf+e28VYa0ZWTmEUk=;7:jQ/LFNAlLhAOT4UnUkXcKflJej5GRBUXxpo5vdv4REAty3CCsnO3DAK3pslTLUQiKDLIvzzwSNkvn9uZ4J6WVfzvIAJnoRWkxvZL1I5txowA7cvECx4ZLbcmG91dzQmHgPSLEWjoZ3NctCE5Yz7N4wubCVzsi3AdFJQWABYWCazQBai4GhHetSruD+Yx58F4qpH1Q907OKYcghoLoiIQk7y7NT8drgpAeAMAOUYpUEUng2eRbgWZkF96nkZNOA02 SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 29 Jul 2016 21:04:15.8824 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: VI1PR0501MB2768 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 7/29/2016 2:31 PM, Francis Giraldeau wrote: > I tested this patch on 4.7 and confirm that irq_work does not occurs anymore on > the isolated cpu. Thanks! Great! Let me know if you'd like me to add your Tested-by in the patch series. > I don't know of any utility to test the task isolation feature, so I started > one: > > https://github.com/giraldeau/taskisol > > The script exp.sh runs the taskisol to test five different conditions, but some > behavior is not the one I would expect. > > At startup, it does: > - register a custom signal handler for SIGUSR1 > - sched_setaffinity() on CPU 1, which is isolated > - mlockall(MCL_CURRENT) to prevent undesired page faults > > The default strict mode is set with: > > prctl(PR_SET_TASK_ISOLATION, PR_TASK_ISOLATION_ENABLE) > > And then, the syscall write() is called. From previous discussion, the SIGKILL > should be sent, but it does not occur. When instead of calling write() we force > a page fault, then the SIGKILL is correctly sent. This looks like it may be a bug in the x86-specific part of the kernel support. On tilegx and arm64, running your test does the right thing: # ./taskisol default syscall taskisol run taskisol/1855: task_isolation mode lost due to syscall 64 Killed I think the x86 support doesn't properly return right away from a bad syscall. The patch below should fix that; can you try it? However, it's not clear to me why the signal isn't getting delivered. Perhaps you can try adding some tracing to the syscall_trace_enter() path and see if we're actually running this code as expected? Thank you! :-) --- a/arch/x86/entry/common.c +++ b/arch/x86/entry/common.c @@ -90,8 +90,10 @@ unsigned long syscall_trace_enter_phase1(struct pt_regs *regs, u32 arch) /* In isolation mode, we may prevent the syscall from running. */ if (work & _TIF_TASK_ISOLATION) { - if (task_isolation_syscall(regs->orig_ax) == -1) - return -1; + if (task_isolation_syscall(regs->orig_ax) == -1) { + regs->orig_ax = -1; + return 0; + } work &= ~_TIF_TASK_ISOLATION; } I updated my dataplane branch on kernel.org with this fix. > When instead a custom signal handler SIGUSR1: > > prctl(PR_SET_TASK_ISOLATION, PR_TASK_ISOLATION_USERSIG | > PR_TASK_ISOLATION_SET_SIG(SIGUSR1) > > The signal is never delivered, either when the syscall is issued nor when the > page fault occurs. This is a bug in your test program. Try again with this fix: --- a/taskisol.c +++ b/taskisol.c @@ -79,8 +79,9 @@ int main(int argc, char *argv[]) * The program completes when using USERSIG, * but actually no signal is delivered */ - if (strcmp(argv[1], "signal") == 0) { - if (prctl(PR_SET_TASK_ISOLATION, PR_TASK_ISOLATION_USERSIG | + else if (strcmp(argv[1], "signal") == 0) { + if (prctl(PR_SET_TASK_ISOLATION, PR_TASK_ISOLATION_ENABLE | + PR_TASK_ISOLATION_USERSIG | PR_TASK_ISOLATION_SET_SIG(SIGUSR1)) < 0) { perror("prctl sigusr"); return -1; The prctl() API is intended to be one-shot, i.e. you set all the state you want with a single prctl(). The next call to prctl() will reset the state to whatever you specify (including if you don't specify "enable"). (Also, as a side note, I'd expect your Makefile to invoke $(CC) for taskisol, not $(CXX) - there doesn't seem to be any actual C++ in the program.) > I can confirm that, if two taskisol are created on the same CPU, the second one > fails with Resource temporarily unavailable, so that's fine. > > I can add more test cases depending on your comments, such as the TLB events > triggered by another thread on a non-isolated core. But maybe there is already > a test suite? The appended code is what I've been using as a test harness. It passes on tilegx and arm64. No guarantees as to production-level code quality :-) #define _GNU_SOURCE #include #include #include #include #include #include #include #include #include #include #include #include #include #ifndef PR_SET_TASK_ISOLATION // Not in system headers yet? # define PR_SET_TASK_ISOLATION 48 # define PR_GET_TASK_ISOLATION 49 # define PR_TASK_ISOLATION_ENABLE (1 << 0) # define PR_TASK_ISOLATION_USERSIG (1 << 1) # define PR_TASK_ISOLATION_SET_SIG(sig) (((sig) & 0x7f) << 8) # define PR_TASK_ISOLATION_GET_SIG(bits) (((bits) >> 8) & 0x7f) # define PR_TASK_ISOLATION_NOSIG \ (PR_TASK_ISOLATION_USERSIG | PR_TASK_ISOLATION_SET_SIG(0)) #endif // The cpu we are using for isolation tests. static int task_isolation_cpu; // Overall status, maintained as tests run. static int exit_status = EXIT_SUCCESS; // Set affinity to a single cpu. int set_my_cpu(int cpu) { cpu_set_t set; CPU_ZERO(&set); CPU_SET(cpu, &set); return sched_setaffinity(0, sizeof(cpu_set_t), &set); } // Run a child process in task isolation mode and report its status. // The child does mlockall() and moves itself to the task isolation cpu. // It then runs SETUP_FUNC (if specified), calls prctl(PR_SET_TASK_ISOLATION, ) // with FLAGS (if non-zero), and then invokes TEST_FUNC and exits // with its status. static int run_test(void (*setup_func)(), int (*test_func)(), int flags) { fflush(stdout); int pid = fork(); assert(pid >= 0); if (pid != 0) { // In parent; wait for child and return its status. int status; waitpid(pid, &status, 0); return status; } // In child. int rc = mlockall(MCL_CURRENT); assert(rc == 0); rc = set_my_cpu(task_isolation_cpu); assert(rc == 0); if (setup_func) setup_func(); if (flags) { int rc; do rc = prctl(PR_SET_TASK_ISOLATION, flags); while (rc != 0 && errno == EAGAIN); if (rc != 0) { printf("couldn't enable isolation (%d): FAIL\n", errno); exit(EXIT_FAILURE); } } rc = test_func(); exit(rc); } // Run a test and ensure it is killed with SIGKILL by default, // for whatever misdemeanor is committed in TEST_FUNC. // Also test it with SIGUSR1 as well to make sure that works. static void test_killed(const char *testname, void (*setup_func)(), int (*test_func)()) { int status = run_test(setup_func, test_func, PR_TASK_ISOLATION_ENABLE); if (WIFSIGNALED(status) && WTERMSIG(status) == SIGKILL) { printf("%s: OK\n", testname); } else { printf("%s: FAIL (%#x)\n", testname, status); exit_status = EXIT_FAILURE; } status = run_test(setup_func, test_func, PR_TASK_ISOLATION_ENABLE | PR_TASK_ISOLATION_USERSIG | PR_TASK_ISOLATION_SET_SIG(SIGUSR1)); if (WIFSIGNALED(status) && WTERMSIG(status) == SIGUSR1) { printf("%s (SIGUSR1): OK\n", testname); } else { printf("%s (SIGUSR1): FAIL (%#x)\n", testname, status); exit_status = EXIT_FAILURE; } } // Run a test and make sure it exits with success. static void test_ok(const char *testname, void (*setup_func)(), int (*test_func)()) { int status = run_test(setup_func, test_func, PR_TASK_ISOLATION_ENABLE); if (status == EXIT_SUCCESS) { printf("%s: OK\n", testname); } else { printf("%s: FAIL (%#x)\n", testname, status); exit_status = EXIT_FAILURE; } } // Run a test with no signals and make sure it exits with success. static void test_nosig(const char *testname, void (*setup_func)(), int (*test_func)()) { int status = run_test(setup_func, test_func, PR_TASK_ISOLATION_ENABLE | PR_TASK_ISOLATION_NOSIG); if (status == EXIT_SUCCESS) { printf("%s: OK\n", testname); } else { printf("%s: FAIL (%#x)\n", testname, status); exit_status = EXIT_FAILURE; } } // Mapping address passed from setup function to test function. static char *fault_file_mapping; // mmap() a file in so we can test touching an unmapped page. static void setup_fault(void) { char fault_file[] = "/tmp/isolation_XXXXXX"; int fd = mkstemp(fault_file); assert(fd >= 0); int rc = ftruncate(fd, getpagesize()); assert(rc == 0); fault_file_mapping = mmap(NULL, getpagesize(), PROT_READ | PROT_WRITE, MAP_SHARED, fd, 0); assert(fault_file_mapping != MAP_FAILED); close(fd); unlink(fault_file); } // Now touch the unmapped page (and be killed). static int do_fault(void) { *fault_file_mapping = 1; return EXIT_FAILURE; } // Make a syscall (and be killed). static int do_syscall(void) { write(STDOUT_FILENO, "goodbye, world\n", 13); return EXIT_FAILURE; } // Turn isolation back off and don't be killed. static int do_syscall_off(void) { prctl(PR_SET_TASK_ISOLATION, 0); write(STDOUT_FILENO, "==> hello, world\n", 17); return EXIT_SUCCESS; } // If we're not getting a signal, make sure we can do multiple system calls. static int do_syscall_multi(void) { write(STDOUT_FILENO, "==> hello, world 1\n", 19); write(STDOUT_FILENO, "==> hello, world 2\n", 19); return EXIT_SUCCESS; } #ifdef __aarch64__ /* ARM64 uses tlbi instructions so doesn't need to interrupt the remote core. */ static void test_munmap(void) {} #else // Fork a thread that will munmap() after a short while. // It will deliver a TLB flush to the task isolation core. static void *start_munmap(void *p) { usleep(500000); // 0.5s munmap(p, getpagesize()); return 0; } static void setup_munmap(void) { // First, go back to cpu 0 and allocate some memory. set_my_cpu(0); void *p = mmap(0, getpagesize(), PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_POPULATE|MAP_PRIVATE, 0, 0); assert(p != MAP_FAILED); // Now fire up a thread that will wait half a second on cpu 0 // and then munmap the mapping. pthread_t thr; int rc = pthread_create(&thr, NULL, start_munmap, p); assert(rc == 0); // Back to the task-isolation cpu. set_my_cpu(task_isolation_cpu); } // Global variable to avoid the compiler outsmarting us. volatile int munmap_spin; static int do_munmap(void) { while (munmap_spin < 1000000000) ++munmap_spin; return EXIT_FAILURE; } static void test_munmap(void) { test_killed("test_munmap", setup_munmap, do_munmap); } #endif #ifdef __tilegx__ // Make an unaligned access (and be killed). // Only for tilegx, since other platforms don't do in-kernel fixups. static int do_unaligned(void) { static int buf[2]; volatile int* addr = (volatile int *)((char *)buf + 1); *addr; asm("nop"); return EXIT_FAILURE; } static void test_unaligned(void) { test_killed("test_unaligned", NULL, do_unaligned); } #else static void test_unaligned(void) {} #endif // Fork a process that will spin annoyingly on the same core // for a second. Since prctl() won't work if this task is actively // running, we following this handshake sequence: // // 1. Child (in setup_quiesce, here) starts up, sets state 1 to let the // parent know it's running, and starts doing short sleeps waiting on a // state change. // 2. Parent (in do_quiesce, below) starts up, spins waiting for state 1, // then spins waiting on prctl() to succeed. At that point it is in // isolation mode and the child is completing its most recent sleep. // Now, as soon as the parent is scheduled out, it won't schedule back // in until the child stops spinning. // 3. Child sees the state change to 2, sets it to 3, and starts spinning // waiting for a second to elapse, at which point it exits. // 4. Parent spins waiting for the state to get to 3, then makes one // syscall. This should take about a second even though the child // was spinning for a whole second after changing the state to 3. volatile int *statep, *childstate; struct timeval quiesce_start, quiesce_end; int child_pid; static void setup_quiesce(void) { // First, go back to cpu 0 and allocate some shared memory. set_my_cpu(0); statep = mmap(0, getpagesize(), PROT_READ|PROT_WRITE, MAP_ANONYMOUS|MAP_SHARED, 0, 0); assert(statep != MAP_FAILED); childstate = statep + 1; gettimeofday(&quiesce_start, NULL); // Fork and fault in all memory in both. child_pid = fork(); assert(child_pid >= 0); if (child_pid == 0) *childstate = 1; int rc = mlockall(MCL_CURRENT); assert(rc == 0); if (child_pid != 0) { set_my_cpu(task_isolation_cpu); return; } // In child. Wait until parent notifies us that it has completed // its prctl, then jump to its cpu and let it know. *childstate = 2; while (*statep == 0) ; *childstate = 3; // printf("child: jumping to cpu %d\n", task_isolation_cpu); set_my_cpu(task_isolation_cpu); // printf("child: jumped to cpu %d\n", task_isolation_cpu); *statep = 2; *childstate = 4; // Now we are competing for the runqueue on task_isolation_cpu. // Spin for one second to ensure the parent gets caught in kernel space. struct timeval start, tv; gettimeofday(&start, NULL); while (1) { gettimeofday(&tv, NULL); double time = (tv.tv_sec - start.tv_sec) + (tv.tv_usec - start.tv_usec) / 1000000.0; if (time >= 0.5) exit(0); } } static int do_quiesce(void) { double time; int rc; rc = prctl(PR_SET_TASK_ISOLATION, PR_TASK_ISOLATION_ENABLE | PR_TASK_ISOLATION_NOSIG); if (rc != 0) { prctl(PR_SET_TASK_ISOLATION, 0); printf("prctl failed: rc %d", rc); goto fail; } *statep = 1; // Wait for child to come disturb us. while (*statep == 1) { gettimeofday(&quiesce_end, NULL); time = (quiesce_end.tv_sec - quiesce_start.tv_sec) + (quiesce_end.tv_usec - quiesce_start.tv_usec)/1000000.0; if (time > 0.1 && *statep == 1) { prctl(PR_SET_TASK_ISOLATION, 0); printf("timed out at %gs in child migrate loop (%d)\n", time, *childstate); char buf[100]; sprintf(buf, "cat /proc/%d/stack", child_pid); system(buf); goto fail; } } assert(*statep == 2); // At this point the child is spinning, so any interrupt will keep us // in kernel space. Make a syscall to make sure it happens at least // once during the second that the child is spinning. kill(0, 0); gettimeofday(&quiesce_end, NULL); prctl(PR_SET_TASK_ISOLATION, 0); time = (quiesce_end.tv_sec - quiesce_start.tv_sec) + (quiesce_end.tv_usec - quiesce_start.tv_usec) / 1000000.0; if (time < 0.4 || time > 0.6) { printf("expected 1s wait after quiesce: was %g\n", time); goto fail; } kill(child_pid, SIGKILL); return EXIT_SUCCESS; fail: kill(child_pid, SIGKILL); return EXIT_FAILURE; } int main(int argc, char **argv) { /* How many seconds to wait after running the other tests? */ double waittime; if (argc == 1) waittime = 10; else if (argc == 2) waittime = strtof(argv[1], NULL); else { printf("syntax: isolation [seconds]\n"); exit(EXIT_FAILURE); } /* Test that the /sys device is present and pick a cpu. */ FILE *f = fopen("/sys/devices/system/cpu/task_isolation", "r"); if (f == NULL) { printf("/sys device: FAIL\n"); exit(EXIT_FAILURE); } char buf[100]; char *result = fgets(buf, sizeof(buf), f); assert(result == buf); fclose(f); char *end; task_isolation_cpu = strtol(buf, &end, 10); assert(end != buf); assert(*end == ',' || *end == '-' || *end == '\n'); assert(task_isolation_cpu >= 0); printf("/sys device : OK\n"); // Test to see if with no mask set, we fail. if (prctl(PR_SET_TASK_ISOLATION, PR_TASK_ISOLATION_ENABLE) == 0 || errno != EINVAL) { printf("prctl unaffinitized: FAIL\n"); exit_status = EXIT_FAILURE; } else { printf("prctl unaffinitized: OK\n"); } // Or if affinitized to the wrong cpu. set_my_cpu(0); if (prctl(PR_SET_TASK_ISOLATION, PR_TASK_ISOLATION_ENABLE) == 0 || errno != EINVAL) { printf("prctl on cpu 0: FAIL\n"); exit_status = EXIT_FAILURE; } else { printf("prctl on cpu 0: OK\n"); } // Run the tests. test_killed("test_fault", setup_fault, do_fault); test_killed("test_syscall", NULL, do_syscall); test_munmap(); test_unaligned(); test_ok("test_off", NULL, do_syscall_off); test_nosig("test_multi", NULL, do_syscall_multi); test_nosig("test_quiesce", setup_quiesce, do_quiesce); // Exit failure if any test failed. if (exit_status != EXIT_SUCCESS) return exit_status; // Wait for however long was requested on the command line. // Note that this requires a vDSO implementation of gettimeofday(); // if it's not available, we could just spin a fixed number of // iterations instead. struct timeval start, tv; gettimeofday(&start, NULL); while (1) { gettimeofday(&tv, NULL); double time = (tv.tv_sec - start.tv_sec) + (tv.tv_usec - start.tv_usec) / 1000000.0; if (time >= waittime) break; } return EXIT_SUCCESS; } -- Chris Metcalf, Mellanox Technologies http://www.mellanox.com