From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S932289AbcHISUA (ORCPT ); Tue, 9 Aug 2016 14:20:00 -0400 Received: from mail-db5eur01on0080.outbound.protection.outlook.com ([104.47.2.80]:63424 "EHLO EUR01-DB5-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751919AbcHIST4 (ORCPT ); Tue, 9 Aug 2016 14:19:56 -0400 Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=cmetcalf@mellanox.com; Subject: Re: [PATCH v7 4/4] nmi_backtrace: generate one-line reports for idle cpus To: Petr Mladek References: <1470672218-16059-1-git-send-email-cmetcalf@mellanox.com> <1470672218-16059-5-git-send-email-cmetcalf@mellanox.com> <20160809124325.GG13300@pathway.suse.cz> CC: Peter Zijlstra , "Rafael J. Wysocki" , Russell King , Thomas Gleixner , Aaron Tomlin , Ingo Molnar , Andrew Morton , Daniel Thompson , , , , From: Chris Metcalf Message-ID: Date: Tue, 9 Aug 2016 12:43:58 -0400 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.2.0 MIME-Version: 1.0 In-Reply-To: <20160809124325.GG13300@pathway.suse.cz> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [12.216.194.146] X-ClientProxiedBy: BN6PR07CA0037.namprd07.prod.outlook.com (10.172.104.23) To AM4PR0501MB2754.eurprd05.prod.outlook.com (10.172.216.10) X-MS-Office365-Filtering-Correlation-Id: 18e1e4dd-efea-47d7-dba1-08d3c074616a X-Microsoft-Exchange-Diagnostics: 1;AM4PR0501MB2754;2:Ew7Qy9e0j3s5pI0mQTW1sPc/tDsN5QYA50cfkr9Izn11W6hjG4mMt8KyNuLmHpFBAqbrkNiXaf3OL8h3In5smog2Nn+zf1O4Bco3LZxxisNpMgcYfAeKERsRI+1LAHwUcC4q6gW9k1qe+e+ef3mI1hgh2sWhG2Qewugt6OvOQmdJduMKOdFisidZNJzWBMtn;3:pCR/DK/Iq44djdUYRRWAhzY+GKYptOdJKpt+HxuIgXAuSN5KejXNa/BlxxgRuydU99fleR3GJtyw3aZ9AK7pffLuXkMJP0u8B0vi5+9X735ERcWZO/OFT1Ct87mXTHDo X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:AM4PR0501MB2754; X-Microsoft-Exchange-Diagnostics: 1;AM4PR0501MB2754;25:NhYH2Z7sO94tzEX9bOwjvXr6zh4qNF7R3cN9Hg7eRfZTWSQWa60Fm1ZyoBQnPVnj6WH/j/aybD3JF1bZgZYXfvNcx7nODTc1ogb9CorTm6mjvAyHBtijdOFERyKyG67iCfLdNti2raDNEBnwi82PFxONDKz2YOdFfwtB0eJCkoo6AlGz5h1k4GUthUSXQU0CxPZL3ohC7qm0cyU1O5lxyHwCgyVQcjn/c6b0txZFB0nAdbcFGPlyKkwpEY/ylocdphuQGnMauLubf1O6Ya9kHxbF+buJd0v0k8fKwWgDIWuaoiLFgmgffTpiBbKjvYagE2H3+qlOy63A2wv7vROk7XHETF2/RIRi8gMeCM9fy2NhB0NJOJ1/MQg9DBEGBSHmdLTsrPPw1uF5YNYk1WIqdyLX8gpTVRbksiuMYiFW1+O4QDf+uPWhNAxkXkgoFyBgElNCzEuOc657fY9AnVY0oVgeIHz/Ua80KQYSa43CwvriGBQtvX+zQfOtltCBpYJ3IWf2NENYe1x8Y80vgqSYF8k0FxbOWLcvYXJeITP7J5EQtO9miwZ0Yv3+mKos4K1JvsH0Nm6+0ZMJLy8CyKakf1N8PWekGexxxgFLUkrjk83aITmNGQzpD7c2+3Zj206xhh54yCHvpCXhqeU7RVvt0Cxy+SGAt9b2bbzk5socHtMh3WdxJaCUku+QcrSwvWxOycQls/Kl4NJOlBDo1VhmqyJwYYK60lD6F9CiO9jE9JP4LSn1bGUSS8IGAng1i2KISKUVhP7eO/extttc2xdjPv+dVJiQq6dpUzVLHCcYyMg= X-Microsoft-Exchange-Diagnostics: 1;AM4PR0501MB2754;31:/Wjok0Zpwhq7/H7N37Ux9lUDSSjDBpUkGbf8D57uvmsNLYDUBLL6Rsc7C+M+ABi8DseBiPsYFLSKmiLmtKEsBPToLOranT+ZrqkEW0RFK8K8zegeZT0rNNn/CS7+Wuvavc+a/v24kpmXn58LWVd8dAQT0915dP5DQ0u6pHErOHcN5p/14M5IYlZ37bhEE8aVWgc5h65oocWJdindPPE96dS8ynsubfVbOyyOzGL0cvo=;20:AD15VIfo2nWFnFnlojd7QSKhXtQFjS0C57aafprxZf9GeW9MJ9WhTOLrPsSGcyRbwJRjAb8xkD7GQrTOgU2uvlAkkjuQbFeuYTzN7KJQTYlvtAifispepottP3i3DO3WuPmxK0mKvk8ox1PAlz/HaUZGp5R5K8vQxKz0BF27ztW5QmG56KusGpvYAnbfHvxzZ76xuKWEKKcCIBt2h3cvxEaw7p6d/nUrFkp3+RGDzI6YJjKVFc3mBA+wfIOSYFvnzgd6AQX+LmwYo7Qwp0vtdRaaflxhGpFs0bJrYgUzs/Cpueef0TxmvAfTeKrF/2YqrzOWgY//zkuGLWZJmVqHb2ja6ERe2fFRqJP/VXnSZap7fy+ZKA3mLFvtZBulO/UJhHec17AMUzrI73wLAfs/ZJgMIqziCavkJY0FxO6dliiwgtrhHkep3UBkSI+XGNRu/c8RJQis4BpSd73zeDPeNi7mVapNy6EtH8daslW24rXejVU39zF0VX+TRZyS1mSb X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(20558992708506)(171992500451332); X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(601004)(2401047)(8121501046)(5005006)(10201501046)(3002001)(6055026);SRVR:AM4PR0501MB2754;BCL:0;PCL:0;RULEID:;SRVR:AM4PR0501MB2754; X-Microsoft-Exchange-Diagnostics: 1;AM4PR0501MB2754;4:ddQ7Au/3wQn5t1upNZI1l4oUarDkS6kw9WSyTILssZ+YSgbpl/RY4Oshn54eDacXp74UG1SrdlK/IZAH4pQFrV5vnLoY0EQljvr1NZgAXk4YLviHzfj9/rDE5T8K0bJeUGCz5OsaEs+GWOUO5afgS4PQTMqPzgTTWgMkPYMS6zz2zvbfC73osvQNwmoGTA75m0krDEeKxfIgcUUXlQLDyRssFqTY2TrsDL/5vR5zP2Wd91U1sQKryZ6G+p6Z4Za+hbHTj5vL1pKkIBmqNwT5KCop1/D/upEtNrT3wfxkZsDd/gGRap5aq22b0UfbIrqysmziNFiQcKW3zE3zlrOG8C//xinHNvCmxx7eXnRpKmgFhKIHBSRj/eFJ3jfiJF8zRUSwlbDSjQUWhztbqDZWxYIKmJRNtokCR64/6KFQc1kP54Q5iypdiBjoYzktuIpyVIee+k0wDsvs1qyPEPBWjw== X-Forefront-PRVS: 0029F17A3F X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10009020)(4630300001)(6049001)(6009001)(7916002)(377424004)(24454002)(377454003)(199003)(189002)(15975445007)(92566002)(65806001)(65956001)(4001350100001)(97736004)(105586002)(77096005)(50466002)(110136002)(19580395003)(23746002)(64126003)(33646002)(42186005)(2950100001)(3846002)(31686004)(66066001)(106356001)(189998001)(86362001)(6116002)(2906002)(68736007)(4326007)(47776003)(7416002)(31696002)(7846002)(7736002)(81156014)(81166006)(8676002)(586003)(305945005)(230700001)(50986999)(101416001)(76176999)(54356999)(83506001)(36756003)(65826006)(18886065003);DIR:OUT;SFP:1101;SCL:1;SRVR:AM4PR0501MB2754;H:[10.15.7.181];FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;AM4PR0501MB2754;23:7HEIwgfmt4+95Xq0jmldUhKHWKpLkgNLFGO?= =?Windows-1252?Q?opEcaoKd/gW3lRx4j2QIJWw1V7stU+RoG4mlrquOgA61DwtvFtxCfmS/?= =?Windows-1252?Q?TfJVJAg3gs20VqkT3QL4uKkRXV19Yu1wEqob3m2i3b7reRxDyZgBiamn?= =?Windows-1252?Q?czZCeKW8HtHS3QXmSgkZEq35YADL6W91URMzrIUCqvh1SxU9o3Uj1er3?= =?Windows-1252?Q?E1kUzSY/yRSTuOKm/sFzM37ZEj2vzIvlhJq7DPA06pF7jDhddjZwrKjX?= =?Windows-1252?Q?ERenuNpTDNBY8SF61pfXDzw+z2nwQI6LH1YvY79JrKTi6uETacm6aoE3?= =?Windows-1252?Q?yLAe/Y4xilRK/EYakFzOm4UBBfIjTiYkzzmQPLpuw7krCmPS7tv0j5w8?= =?Windows-1252?Q?vQWpaSBBXKVJF5l7ZP0QqYk1vHM2uSxUmmLcZpr5Q1tz0PzwbalHlE4O?= =?Windows-1252?Q?NlIjjtOzXK7HQoJXy9dUZZKgW25gnamTKEL1TzCh6VYfc4n55/ZF7vKO?= =?Windows-1252?Q?cviaAdJeQBOWBwRxAwPPRWH143I9PsQaRQ+i0CON4er2/0MxUqbHpWI+?= =?Windows-1252?Q?JjBLYg4CikKqTh/vUZxZJfcT3LNph4HG4Z1SJt6qWoi7lhLmo8T8XMgW?= =?Windows-1252?Q?eWi1BwtUDsqJ/AlSngcLKCLfO7d4ZNJLseyT/wjdqWmJiNoshL/ht0rN?= =?Windows-1252?Q?q5NmqwxKF8UBwLFtY8PiMF6of/cHotsz06T8wtPOySgs/IMx4ouAul7t?= =?Windows-1252?Q?s4FA0YU2SmNRIjX9j/3yYUVav2i3sujpU+l0YKh5qcafY/rk6Hm7l/dX?= =?Windows-1252?Q?csOMxAj23w7jTmPN3Ajb3HaB32c8rljRp6Vci3IGDAylhqC9GMSdGJ4s?= =?Windows-1252?Q?TwcfwM8lRMFu0QTb00Eii1+RljsIgAhnh4eRlHRWbtlGxGy1nqAoW46K?= =?Windows-1252?Q?rXFrCVIPBb2m6i0IrRCJK0BqcUmxuf0kzscIeDA8lCZ7n2+MJFiMNv0B?= =?Windows-1252?Q?farYb37mcuXKrAvC9ebWrN3IJ3aSfnHIhyABFKujIs2I7fKAFXvwbUsY?= =?Windows-1252?Q?+qt4h9FFK8ldNSXGCPoPaJzyUd0PvckauLLWPTzxGLmcx5KTXszLl7WB?= =?Windows-1252?Q?6uv5AZL99lW4xEJ28zsYabimgDKGPnume/Ah/5sbmXzBc7wjdMqNKqX1?= =?Windows-1252?Q?+9BqdDa1qFbVq/ynyeTS946T8J9ktiXtatPxNia0hx3M+6PoG9gX7t3P?= =?Windows-1252?Q?8zRvMjAwxX6YkLAjS6N6aFUtcBY4ZxJu0RHmZDyXRkn7ruBbE26D2gfM?= =?Windows-1252?Q?o33iGt5s7CchmLM44EWcM3X9opoNaYlk9PyMumZeI+Q9e7kYNK0UCaxq?= =?Windows-1252?Q?V087V0oDfbCv0N/O3sQHl2/PthirT7zs4gk7qJipKQJX0eHHEWJ8Z9Sv?= =?Windows-1252?Q?kqIW2/LLNXQ1fV+1tjsI/?= X-Microsoft-Exchange-Diagnostics: 1;AM4PR0501MB2754;6:Kre0SnlmqbMj1wrffu3RRVkaAVD7LMJS0sPv693BHJ5OkqJ2oi3N9krVhX0kapfdHOgSh/7USMUNWxgboPl9DsNs746qNtE8intm+CzQ50nU2TsHB4CwirJrmxX/lhrbHTz8J6QsW3TvtXiTb/WotMwR3wum20xDiSJichfiTKZBkD2oEiSZnlF/yoGx4/gf86Y1Z/ZBvD0VX7daBnuASJlGagMElIAW7W2cbbQcLIP97IIAVsNLbRdIFtP030Khb2s1wHmvMPTBNBR9nbLLFfXKI64eXgGB6L6JQ5MDlps7LsBRWYFef9jqolQOb7FepJ9Ep6+H9mx8ZbODfcJ56w==;5:qJjGbYysDAq/8w5H2ftpWtcoOTdYqZOn8Swa/rpgmXnUr9MeMCBwNr0MsF7mnrQXdeZIKZp4sPplJUz3M3rOcFQRAMJH6GIYCTHnYWzI8mC5KpdkODDnTNmDA3xROFBLqH3NnnTSY8hNIU1yBABRCA==;24:yshHA0xnBiJzPOSvoW9cQ0pfCM4JC+Jn+G3H4VZEmDCBAwwuCos/VBE9Enb3Z4Nyt2Stw+F5DcSVIQiKBda8Pax1Y34O+wk8320/BRMFaLM=;7:Gr3OGADEaHTqv9nN7UjcvZQnJUKsWSCp0BfyhvdKIBUOVNFZ5+vkp4k/fjBPVzX57ITZzH7a53BQGivBXvzZAC49b/0BwIp31LGNJ8d3dhPJdoC8YCd1zfgwHUALMDY3VdUrTjwIJP+x8GzXK6ZglVm8Iih2w6ywO4kUvrcIUjJCCgKSPLQxwCxmUTvf2HG1kIYM4tRVhZ/F0N0rA9ntRV/aIpibqznH/9pMT7cTLwlSaloMAPwaYLw6ubszBSWT SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 09 Aug 2016 16:44:05.3868 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR0501MB2754 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 8/9/2016 8:43 AM, Petr Mladek wrote: > On Mon 2016-08-08 12:03:38, Chris Metcalf wrote: >> When doing an nmi backtrace of many cores, most of which are idle, >> the output is a little overwhelming and very uninformative. Suppress >> messages for cpus that are idling when they are interrupted and just >> emit one line, "NMI backtrace for N skipped: idling at pc 0xNNN". > Hmm, I do not see this message even though the CPU is in the idle state: > > [ 7918.884535] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.8.0-rc1-4-default+ #3088 > [ 7918.884538] Hardware name: Bochs Bochs, BIOS Bochs 01/01/2011 > [ 7918.884539] task: ffff88013a594380 task.stack: ffff88013a598000 > [ 7918.884541] RIP: 0010:[] [] native_safe_halt+0x6/0x10 > [ 7918.884543] RSP: 0018:ffff88013a59bea8 EFLAGS: 00000206 > [ 7918.884544] RAX: ffff88013a594380 RBX: 0000000000000003 RCX: 0000000000000000 > [ 7918.884546] RDX: ffff88013a594380 RSI: 0000000000000001 RDI: ffff88013a594380 > [ 7918.884548] RBP: ffff88013a59bea8 R08: 0000000000000000 R09: 0000000000000000 > [ 7918.884550] R10: 0000000000000001 R11: 0000000000000001 R12: 0000000000000003 > [ 7918.884551] R13: 0000000000000000 R14: ffff88013a598000 R15: ffff88013a598000 > [ 7918.884553] FS: 0000000000000000(0000) GS:ffff88013fd80000(0000) knlGS:0000000000000000 > [ 7918.884554] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 7918.884556] CR2: 00007f8afc65e000 CR3: 00000001383b8000 CR4: 00000000000006e0 > [ 7918.884557] Stack: > [ 7918.884559] ffff88013a59bec8 ffffffff819573d3 0000000000000003 0000000000000000 > [ 7918.884561] ffff88013a59bed8 ffffffff8102628f ffff88013a59bee8 ffffffff819579ea > [ 7918.884562] ffff88013a59bf30 ffffffff810bfe1a ffff88013a598000 ffff88013a598000 > [ 7918.884563] Call Trace: > [ 7918.884565] [] default_idle+0x23/0x170 > [ 7918.884566] [] arch_cpu_idle+0xf/0x20 > [ 7918.884568] [] default_idle_call+0x2a/0x50 > [ 7918.884570] [] cpu_startup_entry+0x16a/0x260 > [ 7918.884571] [] start_secondary+0xf6/0x100 > [ 7918.884573] Code: 00 00 00 00 00 55 48 89 e5 fa 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb 5d c3 66 0f 1f 84 00 00 00 00 00 55 48 89 e5 fb f4 <5d> c3 0f 1f 84 00 00 00 00 00 55 > 48 89 e5 f4 5d c3 66 0f 1f 84 > > Note that I test it in a virtual machine using qemu. > > The strange thing is that I do not see .cpuidle.text section in > the vmlinux binary. But it is possible that I have misunderstood > the concept. The .cpuidle.text sections are merged into the final kernel's overall text segment. What you should see is something like this in the "nm -n" output from the built vmlinux: [...] ffffffff81922aa8 T __cpuidle_text_start ffffffff81922ab0 T default_idle ffffffff81922b90 t mwait_idle ffffffff81922d20 T acpi_processor_ffh_cstate_enter ffffffff81922df0 T default_idle_call ffffffff81922e30 t cpu_idle_poll ffffffff81922f50 t intel_idle ffffffff81923085 t acpi_idle_do_entry ffffffff819230d0 t poll_idle ffffffff81923143 T __cpuidle_text_end [...] In other words, all the cpuidle functions grouped together and bracketed by the __cpuidle_text_{start,end} symbols. Perhaps you were running with a kernel that didn't have the actual patch 4/4 applied, but just the earlier patches? Or perhaps your host Linux has been patched, but not the guest Linux running in qemu? Or perhaps you are ending up doing an NMI backtrace on the host kernel, not the guest? Thanks for your other reviews as well. I have incorporated all of your suggestions into a v8 patch series and pushed it up to kernel.org. I will just wait to repost it until we sort out this issue that you've reported here. -- Chris Metcalf, Mellanox Technologies http://www.mellanox.com