From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753619Ab3KSXdS (ORCPT ); Tue, 19 Nov 2013 18:33:18 -0500 Received: from aserp1040.oracle.com ([141.146.126.69]:30057 "EHLO aserp1040.oracle.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753529Ab3KSXdO convert rfc822-to-8bit (ORCPT ); Tue, 19 Nov 2013 18:33:14 -0500 MIME-Version: 1.0 Message-ID: <8744a6a4-d7a8-4e6f-8934-48a4fd4da0ce@default> Date: Tue, 19 Nov 2013 15:33:06 -0800 (PST) From: Venkat Venkatsubra To: Honggang LI , Josh Hunt Cc: David Miller , jjolly@suse.com, LKML , netdev@vger.kernel.org Subject: RE: [PATCH] rds: Error on offset mismatch if not loopback References: <20120921213239.GJ14393@linux-tkdk.sfcn.org> <20120922.152524.1294103117346567757.davem@davemloft.net> <23964ca1-e7cb-41c3-9da2-5bc1b2b0c014@default> <52841F95.7040204@redhat.com> <41aa904c-6707-4c74-ae72-96e401c68e13@default> <528587D0.5060105@redhat.com> In-Reply-To: <528587D0.5060105@redhat.com> X-Priority: 3 X-Mailer: Oracle Beehive Extensions for Outlook 2.0.1.8 (707110) [OL 12.0.6680.5000 (x86)] Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 8BIT X-Source-IP: acsinet21.oracle.com [141.146.126.237] Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org We now have lot more information than we did before. When sending a "congestion update" in rds_ib_xmit() we are now returning an incorrect number as bytes sent: BUG_ON(off % RDS_FRAG_SIZE); BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header)); /* Do not send cong updates to IB loopback */ if (conn->c_loopback && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) { rds_cong_map_updated(conn->c_fcong, ~(u64) 0); scat = &rm->data.op_sg[sg]; ret = sizeof(struct rds_header) + RDS_CONG_MAP_BYTES; ret = min_t(int, ret, scat->length - conn->c_xmit_data_off); return ret; } It returns min(8240, 4096-0) i.e. 4096 bytes. The caller rds_send_xmit() is made to think a partial message (4096 out of 8240) was sent. It calls rds_ib_xmit() again with a data offset "off" of 4096-48 (rds header) (=4048 bytes). And we hit the BUG_ON. The reason I didn't hit the panic on my test on Oracle UEK2 which is based on 2.6.39 kernel is it had it like this: BUG_ON(off % RDS_FRAG_SIZE); BUG_ON(hdr_off != 0 && hdr_off != sizeof(struct rds_header)); /* Do not send cong updates to IB loopback */ if (conn->c_loopback && rm->m_inc.i_hdr.h_flags & RDS_FLAG_CONG_BITMAP) { rds_cong_map_updated(conn->c_fcong, ~(u64) 0); return sizeof(struct rds_header) + RDS_CONG_MAP_BYTES; } (So it wasn't 100% 2.6.39 ;-). ) It returned 8240 bytes. The caller rds_send_xmit decides the full message was sent (48 byte header + 4096 data + 4096 data). And it worked. Then I found this info on the change that was done upstream which now causes the panic: http://marc.info/?l=linux-netdev&m=129908332903057 http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/commit/?id=6094628bfd94323fc1cea05ec2c6affd98c18f7f Will investigate more into which problem the above change addressed. Venkat