Re: 3.7-rc2 regression : file copied to CIFS-mounted directory corrupted

Re: 3.7-rc2 regression : file copied to CIFS-mounted directory corrupted

[Jongman Heo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 4782 bytes --]


Hmm,

I've just met the issue, with the commit 5640f768 reverted.
It seems that the issue does not always happen. So, my bisection may not be correct.

At this moment, I don't have enough time to do bisection again..

Regards.

[sorry for top posting, our corporate MTA is not good at e-mail writing for mailing list... ]

------- Original Message -------
Sender : Jongman Heo<jongman.heo@samsung.com>
Date : 2012-10-23 12:30 (GMT+09:00)
Title : 3.7-rc2 regression : file copied to CIFS-mounted directory corrupted


Hi, all,

With 3.7-rc2, I noticed that file copied to CIFS-mounted directory is corrupted. 
Integrity is checked by md5sum.

  1. get md5sum of file located on local (ext4) disk
  2. copy the file to CIFS-mounted directory
  3. check md5sum of the copied file

Host   : Windows 7
Guest : Fedora 16 on VMWare
Kernel : 3.7-rc2

Shared directory exists on Windows 7 Host, and Linux Guest on VMware mounts it using CIFS.

I thought it was related to CIFS or VFS changes, but it wasn't.

Below is a git bisect log, and first bad commit is 5640f768, "net: use a per task frag allocator".

Reverting the commit 5640f768 fixes the corruption issue on CIFS directory.
During revert, I got conflict in net/ipv4/raw.c. I took "parent of 5640f76" hunk to resolve it.

# git bisect log
git bisect start
# bad: [6f0c0580b70c89094b3422ba81118c7b959c7556] Linux 3.7-rc2
git bisect bad 6f0c0580b70c89094b3422ba81118c7b959c7556
# good: [a0d271cbfed1dd50278c6b06bead3d00ba0a88f9] Linux 3.6
git bisect good a0d271cbfed1dd50278c6b06bead3d00ba0a88f9
# good: [fec344e3f31aa911297cd3a4639432d983b1f324] cifs: change cifs_call_async to use smb_rqst structs
git bisect good fec344e3f31aa911297cd3a4639432d983b1f324
# good: [3d6d854a13844223b603fd7a16a4a4a4afd62c72] cifs: add FL_CLOSE to fl_flags mask in cifs_read_flock
git bisect good 3d6d854a13844223b603fd7a16a4a4a4afd62c72
# good: [f065fd099fc475333fc7a55677a7f64764445d55] CIFS: Fix possible freed pointer dereference in CIFS_SessSetup
git bisect good f065fd099fc475333fc7a55677a7f64764445d55
# bad: [b7a10626c8bc88fd097a8bb4486c89558f89320c] [CIFS] WARN_ON_ONCE if kernel_sendmsg() returns -ENOSPC
git bisect bad b7a10626c8bc88fd097a8bb4486c89558f89320c
# good: [1d4ab9077681b7cce60ff46e3a42fe2dafa0b83d] [CIFS] Fix indentation of fs/cifs/Kconfig entries
git bisect good 1d4ab9077681b7cce60ff46e3a42fe2dafa0b83d
# bad: [aab174f0df5d72d31caccf281af5f614fa254578] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect bad aab174f0df5d72d31caccf281af5f614fa254578
# good: [d9a807461fc8cc0d6ba589ea0730d139122af012] Merge tag 'usb-3.6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb
git bisect good d9a807461fc8cc0d6ba589ea0730d139122af012
# good: [a20acf99f75e49271381d65db097c9763060a1e8] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc-next
git bisect good a20acf99f75e49271381d65db097c9763060a1e8
# bad: [d379142bc4d9b78cdd9fc5aa696ca1ea083fb7d4] be2net: fixup log messages
git bisect bad d379142bc4d9b78cdd9fc5aa696ca1ea083fb7d4
# good: [2a35cfa591ac63f17815c2d9432b799e37527980] r8169: add D-Link DGE-560T identifiers.
git bisect good 2a35cfa591ac63f17815c2d9432b799e37527980
# good: [9316f0e3c6ba524d8fa14bfded048b252891931a] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next into for-davem
git bisect good 9316f0e3c6ba524d8fa14bfded048b252891931a
# good: [623df484a777f3c00c1ea3d6a7565b8d8ac688a1] tcp: extract code to compute SYNACK RTT
git bisect good 623df484a777f3c00c1ea3d6a7565b8d8ac688a1
# bad: [c523530ce17defe6b28ccfe622c506488f430866] can: c_can: fix segfault during rmmod
git bisect bad c523530ce17defe6b28ccfe622c506488f430866
# good: [6ee584be3ee30f72dec8a8ca87bc10824e27a631] netfilter: nfnetlink_queue: add NFQA_CAP_LEN attribute
git bisect good 6ee584be3ee30f72dec8a8ca87bc10824e27a631
# good: [725b9c0425f54450d1f376befb85dae706ea0c7a] qeth: cleanup channel path descriptor function
git bisect good 725b9c0425f54450d1f376befb85dae706ea0c7a
# good: [0cf833aefaa85bbfce3ff70485e5534e09254773] net: loopback: set default mtu to 64K
git bisect good 0cf833aefaa85bbfce3ff70485e5534e09254773
# bad: [9e49e88958feb41ec701fa34b44723dabadbc28c] filter: add XOR instruction for use with X/K
git bisect bad 9e49e88958feb41ec701fa34b44723dabadbc28c
# bad: [5640f7685831e088fe6c2e1f863a6805962f8e81] net: use a per task frag allocator
git bisect bad 5640f7685831e088fe6c2e1f863a6805962f8e81
# good: [b98b8babd6e3370fadb7c6eaacb00eb2f6344a6c] gianfar: Change default HW Tx queue scheduling mode
git bisect good b98b8babd6e3370fadb7c6eaacb00eb2f6344a6c

ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥

Re: 3.7-rc2 regression : file copied to CIFS-mounted directory corrupted

[Eric Dumazet]

On Tue, 2012-10-23 at 05:38 +0000, Jongman Heo wrote:
> Hmm,
> 
> I've just met the issue, with the commit 5640f768 reverted.
> It seems that the issue does not always happen. So, my bisection may not be correct.
> 
> At this moment, I don't have enough time to do bisection again..
> 
> Regards.

What happens, if instead of reverting you try the following ?

If this solves the problem, then we shall find the driver that assumes
frags are order-0 pages only.

diff --git a/net/core/sock.c b/net/core/sock.c
index 8a146cf..a743e7c 100644
--- a/net/core/sock.c
+++ b/net/core/sock.c
@@ -1726,7 +1726,7 @@ struct sk_buff *sock_alloc_send_skb(struct sock *sk, unsigned long size,
 EXPORT_SYMBOL(sock_alloc_send_skb);
 
 /* On 32bit arches, an skb frag is limited to 2^15 */
-#define SKB_FRAG_PAGE_ORDER	get_order(32768)
+#define SKB_FRAG_PAGE_ORDER	0
 
 bool sk_page_frag_refill(struct sock *sk, struct page_frag *pfrag)
 {

Re: 3.7-rc2 regression : file copied to CIFS-mounted directory corrupted

[Shreyas Bhatewara]

Eric, thanks for the note. I will submit a patch to do it.

Shreyas

----- Original Message -----
> On Tue, 2012-10-23 at 08:17 +0000, Jongman Heo wrote:
> 
> > 
> > FYI, vmxnet3 driver is used for ethernet.
> 
> Yes, this driver needs some changes
> 
> #define VMXNET3_MAX_TX_BUF_SIZE  (1 << 14)
> 
> Thats 16KB
> 
> As we can now provide up to 32KB fragments we broke something.
> 
> vmxnet3_tq_xmit() needs to split large frags into 2 parts.
> (And without going to skb_linearize() of course !)
> 
> Any volunteer ?
> 
> Thanks !
> 
> 
> 

3.7-rc2 regression : file copied to CIFS-mounted directory corrupted

[Jongman Heo]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=utf-8, Size: 4264 bytes --]


Hi, all,

With 3.7-rc2, I noticed that file copied to CIFS-mounted directory is corrupted. 
Integrity is checked by md5sum.

  1. get md5sum of file located on local (ext4) disk
  2. copy the file to CIFS-mounted directory
  3. check md5sum of the copied file

Host   : Windows 7
Guest : Fedora 16 on VMWare
Kernel : 3.7-rc2

Shared directory exists on Windows 7 Host, and Linux Guest on VMware mounts it using CIFS.

I thought it was related to CIFS or VFS changes, but it wasn't.

Below is a git bisect log, and first bad commit is 5640f768, "net: use a per task frag allocator".

Reverting the commit 5640f768 fixes the corruption issue on CIFS directory.
During revert, I got conflict in net/ipv4/raw.c. I took "parent of 5640f76" hunk to resolve it.

# git bisect log
git bisect start
# bad: [6f0c0580b70c89094b3422ba81118c7b959c7556] Linux 3.7-rc2
git bisect bad 6f0c0580b70c89094b3422ba81118c7b959c7556
# good: [a0d271cbfed1dd50278c6b06bead3d00ba0a88f9] Linux 3.6
git bisect good a0d271cbfed1dd50278c6b06bead3d00ba0a88f9
# good: [fec344e3f31aa911297cd3a4639432d983b1f324] cifs: change cifs_call_async to use smb_rqst structs
git bisect good fec344e3f31aa911297cd3a4639432d983b1f324
# good: [3d6d854a13844223b603fd7a16a4a4a4afd62c72] cifs: add FL_CLOSE to fl_flags mask in cifs_read_flock
git bisect good 3d6d854a13844223b603fd7a16a4a4a4afd62c72
# good: [f065fd099fc475333fc7a55677a7f64764445d55] CIFS: Fix possible freed pointer dereference in CIFS_SessSetup
git bisect good f065fd099fc475333fc7a55677a7f64764445d55
# bad: [b7a10626c8bc88fd097a8bb4486c89558f89320c] [CIFS] WARN_ON_ONCE if kernel_sendmsg() returns -ENOSPC
git bisect bad b7a10626c8bc88fd097a8bb4486c89558f89320c
# good: [1d4ab9077681b7cce60ff46e3a42fe2dafa0b83d] [CIFS] Fix indentation of fs/cifs/Kconfig entries
git bisect good 1d4ab9077681b7cce60ff46e3a42fe2dafa0b83d
# bad: [aab174f0df5d72d31caccf281af5f614fa254578] Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs
git bisect bad aab174f0df5d72d31caccf281af5f614fa254578
# good: [d9a807461fc8cc0d6ba589ea0730d139122af012] Merge tag 'usb-3.6' of git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/usb
git bisect good d9a807461fc8cc0d6ba589ea0730d139122af012
# good: [a20acf99f75e49271381d65db097c9763060a1e8] Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/sparc-next
git bisect good a20acf99f75e49271381d65db097c9763060a1e8
# bad: [d379142bc4d9b78cdd9fc5aa696ca1ea083fb7d4] be2net: fixup log messages
git bisect bad d379142bc4d9b78cdd9fc5aa696ca1ea083fb7d4
# good: [2a35cfa591ac63f17815c2d9432b799e37527980] r8169: add D-Link DGE-560T identifiers.
git bisect good 2a35cfa591ac63f17815c2d9432b799e37527980
# good: [9316f0e3c6ba524d8fa14bfded048b252891931a] Merge branch 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next into for-davem
git bisect good 9316f0e3c6ba524d8fa14bfded048b252891931a
# good: [623df484a777f3c00c1ea3d6a7565b8d8ac688a1] tcp: extract code to compute SYNACK RTT
git bisect good 623df484a777f3c00c1ea3d6a7565b8d8ac688a1
# bad: [c523530ce17defe6b28ccfe622c506488f430866] can: c_can: fix segfault during rmmod
git bisect bad c523530ce17defe6b28ccfe622c506488f430866
# good: [6ee584be3ee30f72dec8a8ca87bc10824e27a631] netfilter: nfnetlink_queue: add NFQA_CAP_LEN attribute
git bisect good 6ee584be3ee30f72dec8a8ca87bc10824e27a631
# good: [725b9c0425f54450d1f376befb85dae706ea0c7a] qeth: cleanup channel path descriptor function
git bisect good 725b9c0425f54450d1f376befb85dae706ea0c7a
# good: [0cf833aefaa85bbfce3ff70485e5534e09254773] net: loopback: set default mtu to 64K
git bisect good 0cf833aefaa85bbfce3ff70485e5534e09254773
# bad: [9e49e88958feb41ec701fa34b44723dabadbc28c] filter: add XOR instruction for use with X/K
git bisect bad 9e49e88958feb41ec701fa34b44723dabadbc28c
# bad: [5640f7685831e088fe6c2e1f863a6805962f8e81] net: use a per task frag allocator
git bisect bad 5640f7685831e088fe6c2e1f863a6805962f8e81
# good: [b98b8babd6e3370fadb7c6eaacb00eb2f6344a6c] gianfar: Change default HW Tx queue scheduling mode
git bisect good b98b8babd6e3370fadb7c6eaacb00eb2f6344a6c
ÿôèº{.nÇ+‰·Ÿ®‰­†+%ŠËÿ±éݶ\x17¥Šwÿº{.nÇ+‰·¥Š{±þG«éÿŠ{ayº\x1dʇڙë,j\a­¢f£¢·hšïêÿ‘êçz_è®\x03(­éšŽŠÝ¢j"ú\x1a¶^[m§ÿÿ¾\a«þG«éÿ¢¸?™¨è­Ú&£ø§~á¶iO•æ¬z·švØ^\x14\x04\x1a¶^[m§ÿÿÃ\fÿ¶ìÿ¢¸?–I¥