From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756009AbdKGVog (ORCPT ); Tue, 7 Nov 2017 16:44:36 -0500 Received: from mx0b-00082601.pphosted.com ([67.231.153.30]:45178 "EHLO mx0a-00082601.pphosted.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754966AbdKGVoe (ORCPT ); Tue, 7 Nov 2017 16:44:34 -0500 Subject: Re: [PATCH 0/2][v5] Add the ability to do BPF directed error injection To: Josef Bacik , , , , , , , , , Peter Zijlstra References: <1510086523-8859-1-git-send-email-josef@toxicpanda.com> From: Alexei Starovoitov Message-ID: <04fd674a-6921-d2ea-8e89-856397ab297d@fb.com> Date: Wed, 8 Nov 2017 06:43:25 +0900 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:45.0) Gecko/20100101 Thunderbird/45.8.0 MIME-Version: 1.0 In-Reply-To: <1510086523-8859-1-git-send-email-josef@toxicpanda.com> Content-Type: text/plain; charset="windows-1252"; format=flowed Content-Transfer-Encoding: 7bit X-Originating-IP: [2620:10d:c094:180::1:23e4] X-ClientProxiedBy: PS1PR0601CA0103.apcprd06.prod.outlook.com (2603:1096:803:1::29) To BL2PR15MB0964.namprd15.prod.outlook.com (2603:10b6:201:15::26) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2073b9ad-f869-4375-6c96-08d526289f74 X-Microsoft-Antispam: UriScan:;BCL:0;PCL:0;RULEID:(22001)(4534020)(4602075)(4627115)(201703031133081)(201702281549075)(2017052603199);SRVR:BL2PR15MB0964; X-Microsoft-Exchange-Diagnostics: 1;BL2PR15MB0964;3:D3KWumJfjjH5YUeAgW8+5zSB87NOdMYgh+GzSers/v+p099GWZZFd7QqVLb6pJPnCmyr5EBc2AyX0Xzo6NDvyUHcreSzfCyOS8NSeDEOwi+yxrQZaz6MIJbTXBSRrJLhfF3XaXw8gTiq2FRaBzIqT5TMA5aR8W/Bd8F248FP7ES9Ss8sBBni7o3k7CnsXL2072qgjlrNJAw0vQuIRL3lPNEayeDb8bdgMm288Y6V5LMUEqeGyakK90e6gPGxnQbQ;25:7fvsRjskXHFyM1MdVKDxeHTztrhoUFKgb79OaCbP80MhfkbL8JjdgC2ZMM/QfthlnMQUwg6/n2kf2xYVtKe8bzzuzpAYlXSi8vZ8GfAnHU7O0xZbjU1XZM4RSjS7LLAHx45jP2gPhacHLVV8p/RRtp6LN2GaNVQv8piZLtNSe+uVkqce3fhKYZdt1XYXHAn+9mN7UInmWmeIwA12MDy9o835+SMyaxdFsurD0MPiKzcVRciCbOYnEJHug6EHPT3+SBZbDLZo0mxoojikn9PjsMWDmDlBmOCVR0HWX5KHLH+oyC8Zh1b4lmtiIES7o57Bl9kU3e13kITEfpOAd79lqw==;31:q9Yj517vtKE5h50KkX2khs7iDrQeI+W07UZEvmsE8VYp1y7BJNx6RcedqVlsPhMLgZMXqYBtz3SnfPnTMe6zyo2ogXxKCyAiJxC6KXn8jJFJ3SFTMHrPrfuzwxqIStq+N/+AHoiZlqUvJoUr6iDwQuzSeu+UbFy7YhYObD+x+HXE6CkylZLxdlC4mQ4eOjgc1bsVmkZ7QQwLlzBTxvp4mTUMXnwt5hDzkB8ySCmUDqI= X-MS-TrafficTypeDiagnostic: BL2PR15MB0964: X-Microsoft-Exchange-Diagnostics: 1;BL2PR15MB0964;20:Zd5A8v5vxm71pnTqOhEEoPB17k7/ZtiuW9e/4knLnlBPHlzCpNlKbvIALc+6801Va+QnpFOcZHOa4k0705RTZOD9TsP0ZF6TcdiD0oKPJFP4jJ3/aJT3pS7eBRUieuwVkvRt0Hz2L+y6ZDjfmmYOJ8grhoLMzpDiJPVUtzT040KliHMt3jHEJY4RCQT35eHKtdvcVMcd1utOuB3mH6FtuqQkj3sBnUobgFPAOfnCsreSB4Xtu09khOQAHFBhjVg6QymWUzRExpDvJfVyYfgrZEPxRJyJL+wTmJcv30ODp0golViItRa6Q9EQxVkveskLi0bWsc7HiTyxM28Vf5LF7AkcUI176nFE8jwyCoUSqEdkiVhfj0a73Iqi6901k4zaJlUCnSVF5+TPSDr927htq035Huhz1HFh02NhmTHvbL4iIDKDG5zyQd6LsfqY00WhHUOV+o6SDMEdhcXpawu+IPKhEY3v+a81sC4Qsi+WSmrRidqfgJg025oEvXdms1Lc;4:y4zWyG7icmOnpOrJnMjOTIP8ydFoXTXmcSmnG5915EQnX2Ko9DL6kRIitKeExndsMSkRxIbG4mSoAsukaudwuInea80REdbkOSc/pf5qAltuZh9c1Oj4kc73sPnXaLafGsehmuazwKnmrzdHkWEdG264eypaBZaAEXAgWvxjfo7wKcTYXkSYOvriEh5OzbCSZ9nvtStqeLD0CXqYssvQyyhIO3Uw73jkryIV476rBZuWqW5rckU6UOQIzA99RvbRMhB6eJUcmP5OTJFYoG/gpQ== X-Exchange-Antispam-Report-Test: UriScan:; X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(11241501159)(6040450)(2401047)(8121501046)(5005006)(100000703101)(100105400095)(3231021)(3002001)(93006095)(93001095)(10201501046)(6041248)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123558100)(20161123560025)(20161123564025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:BL2PR15MB0964;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:BL2PR15MB0964; X-Forefront-PRVS: 0484063412 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(6009001)(346002)(376002)(24454002)(189002)(199003)(101416001)(31686004)(50466002)(36756003)(105586002)(64126003)(33646002)(76176999)(54356999)(50986999)(106356001)(31696002)(478600001)(86362001)(229853002)(67846002)(25786009)(68736007)(83506002)(53546010)(47776003)(2906002)(6666003)(6246003)(65826007)(1706002)(7736002)(5890100001)(189998001)(53936002)(305945005)(6116002)(97736004)(316002)(5660300001)(230700001)(58126008)(65806001)(65956001)(110136005)(81166006)(23746002)(2950100002)(81156014)(8676002)(8936002)(6486002)(921003)(42262002)(1121003);DIR:OUT;SFP:1102;SCL:1;SRVR:BL2PR15MB0964;H:[IPv6:2620:10d:c0e1:1110::1004];FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; X-Microsoft-Exchange-Diagnostics: =?Windows-1252?Q?1;BL2PR15MB0964;23:5y43qlCiz+vOiwgPFVPt+UtLHiMNE9FTS9AF1?= =?Windows-1252?Q?azjWF+EwVJ1ufTXnaMRD5/hGo6mRv9B0B5kNpSs2BgA1ItejDbPvUXrq?= =?Windows-1252?Q?L8i05DeHgJs0z3U+PRoc9HALJIHi5soc0DlQR4XRIsInSayg6jgR3YLk?= =?Windows-1252?Q?Bip/GJ8Rd39oEjt8zLtUY0wZAiEb2esgRkFgPbnpD+qOwtWU7xNV0Q8R?= =?Windows-1252?Q?Tn8u09O7l64JEMSPx6fasCLiLa4mPtf6bJT6rM5s3f95lOXTQOYBx8xI?= =?Windows-1252?Q?W8DQ373M1YWzTSdAohBqkEOuM6S/qETldu4ULwFf6Xk3iLzN0T9Fe/Rt?= =?Windows-1252?Q?tc+OdBahZoyxJEgmUAdUC233Huu1v3O4iXzme12vPQti9eAALVFYx6z3?= =?Windows-1252?Q?ppoa+kKSdAzefnFIXHg1yL/c9jVIaP5jf6nQN4ZR9qnJFxLf/5XHrC0Z?= =?Windows-1252?Q?OluRmQgeD8ObdFbYcBFWijD8sewg9piapZbBluuZv2WeqTxYc3SNnDOL?= =?Windows-1252?Q?kIFxs/QuKOxtMWD8LnB+FZGhwqHFflgC53KtQpKCawem/yEeEhHbosaU?= =?Windows-1252?Q?90yO6L3K70DImS92iniLCCTaCaYG2BKqA5PFqcDoL7Drimj1re/+rsaB?= =?Windows-1252?Q?04Nfl8/OIxy1/mWAuSD5aTCN4MM+117L0iiwTWxcbQ7Co6XTPXQARJcC?= =?Windows-1252?Q?BTlqMB8HrmpcxB6yw2qbW6937kgIfiagPGq8dT84z9X4+EdKMNIZ674F?= =?Windows-1252?Q?bNZwxhnWVtVc58pSVH/IV9/biSjX7kxebO3U1aLRx6qeuVgjwX5cJsCQ?= =?Windows-1252?Q?VIWDbBct/NXg4dEWB0VfgvVRCNKoTFFSq0DTPAU66IpdXTWBiISelhol?= =?Windows-1252?Q?9ZAi1m+3q3dCDMXtwqT8S0lgvh6vopWaY2myA/JsFF45MXQNk8LDZbNJ?= =?Windows-1252?Q?frvjqdNoinpf+i7tkx+8XuHy/TXqSPcyMCjmXP20jfyQ/xqZMGuSyRkH?= =?Windows-1252?Q?kQIdaAL4CUSt1Glvsaz59dXbxcCEIL/idAWvLHZe+S3WGic+EIgO0LtL?= =?Windows-1252?Q?UThdWUX7rJWJbjWStGk94xkoSCqDT9qXmRS81ejbpy2wuatiEcr9OmBN?= =?Windows-1252?Q?zJ17u1lJHEInn+9VRjwCPoQOlMUH1S45QNvICoAMSj2icganoJxbs9eO?= =?Windows-1252?Q?dli2aI6iRrCN/0QP0IzghvAvD4oF3CgVy+OsW48Yog2azTh2AxqI9m1L?= =?Windows-1252?Q?yNLFG3laYAedjz1BsswFfRK0A92VUAOTwi9o2JOhL6SvWc2Xx0vZkbho?= =?Windows-1252?Q?Oe0dYLbAP3Lc+4ox5HCkV30V/mLvzxtcVMhHpQTtJ3ulCorjk9tdAtAu?= =?Windows-1252?Q?p6wMwVytmK1ArytxYPtiD/XMLTTkEdRBA=3D=3D?= X-Microsoft-Exchange-Diagnostics: 1;BL2PR15MB0964;6:fb6f14KxkizUY5Wq2Mhx8jynXMhtdDPYyWJC4Cgd1ObSpCufs+nXOK6mIEmzFj2UpSzIUoeoot4YxXFUTjnbqSamCVlNOP3y6nBRTSxUIxfHmaYD+FRESyHtTqbPWe61ZNG9qIlx933sL7/BSmN8Y/Xg/nUWYEqpKMZJFkpyPPeHCYLGu83NEocSUidd2+nRw3EYgC9nmQ0EgS2eBAOt9UyzCRQ6OF61tKEzGx+8JQ0M2SFLhNGyEy75e1vbvwUcalXKTAnKNu6ZJc9WQ9l9f75Ny+BFY6ygm51gTgR5hVEBc1pLI3U1/fbbazZglHR930io5ViZ2tH9diNBHS5jR40dhmOtFVNfsEqg2+eMqss=;5:sD2VYdIJzhoiXLILrCBpaKLWDZ1GKlJx2BBTv9T16dDfbGMY96Y5Bx272nSI6rdGU5wn999GHkbgY1FkJR9JoqY3OudasJXYk5REo8Cek7xO8w58ALrJ76o8Jm2h6tZHHvck86Apfb8bWZTsYA2C6DsDx2bFDrWyJnqTo1V3xcg=;24:2EMqAx3CrOPBRWRz1zgOmb1Pb3Rp5FHOtSRUtYHoLB2HhQTVvvS+Jsnv3WkzllUv4Q1W10J7q58M6ILtG5LiPhSg9teSGW8sOLb3TD6fKkc=;7:Kc2yETQBUcHQJ/0g94G1IJOYlGfXbGfV2O58p1cHjL41P6eGkyqwSDRXLq1iikdBysPPNQ0IkPDNOUVLVcfctIG3O2sI9pq2GKTrla/1QgG4wkb5oZTsqfZWkjG3NHSMSknXsLuNBHBknkyN6m2OxLWo5Wa+dVmYoMVKLfPfmedtHkNUnxQY25JBPlzO9l0aguHzEQ1J5izL4msGiifiW1CqR3Q0il7SV6RhWyht3m8PE3NuqglNQg6iMpQ8xouU SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1;BL2PR15MB0964;20:W3U8g+P2AndeiG+NYUbMdPlKYWu94Z9Idj1SmuUibhVvLBAO5RZZyzUjlJE12CfsG8tf0YcSvm5//rd/hlhs2IahJYWjX01TCs3y3NUtWVMdtYFAhqkCJFS4I3d1dVQtCfNPDnGNChrRR6jKa7m07JA13zh0eP9wqzQyVgg2dw0= X-MS-Exchange-CrossTenant-OriginalArrivalTime: 07 Nov 2017 21:43:41.4951 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2073b9ad-f869-4375-6c96-08d526289f74 X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-Transport-CrossTenantHeadersStamped: BL2PR15MB0964 X-OriginatorOrg: fb.com X-Proofpoint-Spam-Reason: safe X-FB-Internal: Safe X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2017-11-07_07:,, signatures=0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 11/8/17 5:28 AM, Josef Bacik wrote: > I'm sending this through Dave since it'll conflict with other BPF changes in his > tree, but since it touches tracing as well Dave would like a review from > somebody on the tracing side. > > v4->v5: > - disallow kprobe_override programs from being put in the prog map array so we > don't tail call into something we didn't check. This allows us to make the > normal path still fast without a bunch of percpu operations. looks great to me. Peter, could you please review x86 bits? > v3->v4: > - fix a build error found by kbuild test bot (I didn't wait long enough > apparently.) > - Added a warning message as per Daniels suggestion. > > v2->v3: > - added a ->kprobe_override flag to bpf_prog. > - added some sanity checks to disallow attaching bpf progs that have > ->kprobe_override set that aren't for ftrace kprobes. > - added the trace_kprobe_ftrace helper to check if the trace_event_call is a > ftrace kprobe. > - renamed bpf_kprobe_state to bpf_kprobe_override, fixed it so we only read this > value in the kprobe path, and thus only write to it if we're overriding or > clearing the override. > > v1->v2: > - moved things around to make sure that bpf_override_return could really only be > used for an ftrace kprobe. > - killed the special return values from trace_call_bpf. > - renamed pc_modified to bpf_kprobe_state so bpf_override_return could tell if > it was being called from an ftrace kprobe context. > - reworked the logic in kprobe_perf_func to take advantage of bpf_kprobe_state. > - updated the test as per Alexei's review. > > - Original message - > > A lot of our error paths are not well tested because we have no good way of > injecting errors generically. Some subystems (block, memory) have ways to > inject errors, but they are random so it's hard to get reproduceable results. > > With BPF we can add determinism to our error injection. We can use kprobes and > other things to verify we are injecting errors at the exact case we are trying > to test. This patch gives us the tool to actual do the error injection part. > It is very simple, we just set the return value of the pt_regs we're given to > whatever we provide, and then override the PC with a dummy function that simply > returns. > > Right now this only works on x86, but it would be simple enough to expand to > other architectures. Thanks, > > Josef >