From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754056AbbGXO7G (ORCPT ); Fri, 24 Jul 2015 10:59:06 -0400 Received: from smtpbg299.qq.com ([184.105.67.99]:40821 "EHLO smtpbg299.qq.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751878AbbGXO7E (ORCPT ); Fri, 24 Jul 2015 10:59:04 -0400 X-QQ-mid: bizesmtp5t1437749938t726t157 X-QQ-SSF: 0140000000200010F322B00A0000000 X-QQ-FEAT: lKg1JVH8Fzgm/p+Jyyl0YSPCk/s3E0EZ42xZSaFP+Ku0Bv1eQt6O44DCMsV75 2acpM2EVxNTqdZB/M4T91TPXuKTmTAA145Sfwb1xi2iWUZquSV5aAogzEI2pIX1DLvUUt/b sTkAOD/gYvzJ0D5Lsy1bPh/Yn2TteJFzf4wQ3XFkQ43OXgcU3vHaG+yVzpGafwaWKqjPL4D 34PQX2DNZJBtjuCU8Q2xq3QDD8JdWqhGn31cPYQE80yGYKriBiXx0 X-QQ-GoodBg: 2 Message-ID: <55B252B0.7010301@unitedstack.com> Date: Fri, 24 Jul 2015 22:58:56 +0800 From: juncheng bai User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:31.0) Gecko/20100101 Thunderbird/31.7.0 MIME-Version: 1.0 CC: linux-fsdevel@vger.kernel.org, "linux-kernel@vger.kernel.org" Subject: Performance degrad on big cluster References: In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit X-QQ-SENDSIZE: 520 X-QQ-FName: 0F3B690E592345008A7A9DD879D56BC2 X-QQ-LocalIP: 127.0.0.1 To: unlisted-recipients:; (no To-header on input) Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi all, We meet a performance degrad in one of our cluster. Our randwrite latency degraded from 1ms to 5ms(fio -ioengine=rbd iodepth=1) The cluster has about 200 osds runing on Intel 3500 SSD, we run both qemu and ceph-osd on the hosts. The network for ceph is 10GbE. While the cluster is smaller and not so many qemu processes, the IO latency is about 1ms, but now the latency is 5ms. I use strace to get the time for syscall, all syscall (writev, io_sub mit, recvfrom,sendmsg,lseek,fgetxattr etc.) use 300us to 600us. The syscall time on a small and idle cluster is near to 0us. After checked serval clusters, I come to a conclusion: num_of_osds num_of_threads_on_host time_of_syscall(us) 200 10000 300-600 100 5000 200-500 70 2500 100-300 9 750 20-60 The threads on one of 200 osds cluster's host is like this: name num_of_processes num_of_threads num_of_threads_per_process qemu-kvm 49 9748 198 ceph-osd 6 5707 951 I think too many threads on the host lead to high latency of ceph-osd process, and cause to high I/O latency from the client-side. Anyone's help is welcome. Thanks!