* perf install-python_ext installing .o files in python site-packages @ 2016-02-26 20:38 Josh Boyer 2016-02-26 21:14 ` Jiri Olsa 0 siblings, 1 reply; 4+ messages in thread From: Josh Boyer @ 2016-02-26 20:38 UTC (permalink / raw) To: Arnaldo Carvalho de Melo, Jiri Olsa; +Cc: Linux-Kernel@Vger. Kernel. Org Hi Arnaldo, We've had a report [1] that the python-perf package in Fedora installs a handful of .o files in /usr/lib64/python2.7/site-packages/. This is true on Linus' latest tree, but it goes back to arriving with the 4.2 kernel. Digging around a bit, it seems the files installed are those that are found as relative paths in the utils/python-ext-sources directory: /usr/lib64/python2.7/site-packages/bitmap.o /usr/lib64/python2.7/site-packages/find_bit.o /usr/lib64/python2.7/site-packages/hweight.o /usr/lib64/python2.7/site-packages/rbtree.o None of those files should really be installed at all, as they're temporary object files used to build perf.so. As far as I can tell, this is a side effect of using the python distutils module with it's setup.py script. The sources are listed there so they get included, but then the build_ext functionality interprets it as e.g.: -c ../lib/hweight.c -o python_ext_build/tmp/../lib/hweight.o That leaves it in the lib/ directory of the build output alongside the final perf.so instead of python_ext_build/tmp/util/ like the rest of the object files. It happens to be an unfortunate coincidence that install-lib looks in said .../lib/ directory and we just happen to conflict with that naming. So when install-python_ext is called, it simply copies everything in that directory, including the .o files. I'm not enough of a python wizard to figure out how to fix this quickly. We could remove the .o files in the .spec file, but that seems hacky and this should probably be fixed in the perf build system. Ideas? josh [1] https://bugzilla.redhat.com/show_bug.cgi?id=1312102 ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: perf install-python_ext installing .o files in python site-packages 2016-02-26 20:38 perf install-python_ext installing .o files in python site-packages Josh Boyer @ 2016-02-26 21:14 ` Jiri Olsa 2016-02-26 22:49 ` Josh Boyer 0 siblings, 1 reply; 4+ messages in thread From: Jiri Olsa @ 2016-02-26 21:14 UTC (permalink / raw) To: Josh Boyer; +Cc: Arnaldo Carvalho de Melo, Linux-Kernel@Vger. Kernel. Org On Fri, Feb 26, 2016 at 03:38:09PM -0500, Josh Boyer wrote: > Hi Arnaldo, > > We've had a report [1] that the python-perf package in Fedora installs > a handful of .o files in /usr/lib64/python2.7/site-packages/. This is > true on Linus' latest tree, but it goes back to arriving with the 4.2 > kernel. > > Digging around a bit, it seems the files installed are those that are > found as relative paths in the utils/python-ext-sources directory: > > /usr/lib64/python2.7/site-packages/bitmap.o > /usr/lib64/python2.7/site-packages/find_bit.o > /usr/lib64/python2.7/site-packages/hweight.o > /usr/lib64/python2.7/site-packages/rbtree.o > > None of those files should really be installed at all, as they're > temporary object files used to build perf.so. > > As far as I can tell, this is a side effect of using the python > distutils module with it's setup.py script. The sources are listed > there so they get included, but then the build_ext functionality > interprets it as e.g.: > > -c ../lib/hweight.c -o python_ext_build/tmp/../lib/hweight.o > > That leaves it in the lib/ directory of the build output alongside the > final perf.so instead of python_ext_build/tmp/util/ like the rest of > the object files. It happens to be an unfortunate coincidence that > install-lib looks in said .../lib/ directory and we just happen to > conflict with that naming. So when install-python_ext is called, it > simply copies everything in that directory, including the .o files. > > I'm not enough of a python wizard to figure out how to fix this > quickly. We could remove the .o files in the .spec file, but that > seems hacky and this should probably be fixed in the perf build > system. > > Ideas? looks like the python extension build system mixes up the '../lib' source files with final lib, which by 'accident' is located in '../lib' as well ;-) using full paths for sources sorted this for me (patch attached) but that will end up with following build tree: ./python_ext_build ./python_ext_build/lib ./python_ext_build/lib/perf.so ./python_ext_build/tmp ./python_ext_build/tmp/home ./python_ext_build/tmp/home/jolsa ./python_ext_build/tmp/home/jolsa/kernel ./python_ext_build/tmp/home/jolsa/kernel/linux-perf ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/counts.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/util.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/thread_map.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/cpumap.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/strlist.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/evlist.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/evsel.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/trace-event.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/xyarray.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/python.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/ctype.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/cgroup.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/rblist.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/string.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/hweight.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/rbtree.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/bitmap.o ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/find_bit.o not sure we want to come up with some 'nicer' solution jirka --- diff --git a/tools/perf/util/setup.py b/tools/perf/util/setup.py index 1833103768cb..6c4bb2ea3594 100644 --- a/tools/perf/util/setup.py +++ b/tools/perf/util/setup.py @@ -22,6 +22,7 @@ cflags = getenv('CFLAGS', '').split() # switch off several checks (need to be at the end of cflags list) cflags += ['-fno-strict-aliasing', '-Wno-write-strings', '-Wno-unused-parameter' ] +src_perf = getenv('srctree') + '/tools/perf' build_lib = getenv('PYTHON_EXTBUILD_LIB') build_tmp = getenv('PYTHON_EXTBUILD_TMP') libtraceevent = getenv('LIBTRACEEVENT') @@ -30,6 +31,8 @@ libapikfs = getenv('LIBAPI') ext_sources = [f.strip() for f in file('util/python-ext-sources') if len(f.strip()) > 0 and f[0] != '#'] +ext_sources = map(lambda x: '%s/%s' % (src_perf, x) , ext_sources) + perf = Extension('perf', sources = ext_sources, include_dirs = ['util/include'], ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: perf install-python_ext installing .o files in python site-packages 2016-02-26 21:14 ` Jiri Olsa @ 2016-02-26 22:49 ` Josh Boyer 2016-02-26 22:59 ` Jiri Olsa 0 siblings, 1 reply; 4+ messages in thread From: Josh Boyer @ 2016-02-26 22:49 UTC (permalink / raw) To: Jiri Olsa; +Cc: Arnaldo Carvalho de Melo, Linux-Kernel@Vger. Kernel. Org On Fri, Feb 26, 2016 at 4:14 PM, Jiri Olsa <jolsa@redhat.com> wrote: > On Fri, Feb 26, 2016 at 03:38:09PM -0500, Josh Boyer wrote: >> Hi Arnaldo, >> >> We've had a report [1] that the python-perf package in Fedora installs >> a handful of .o files in /usr/lib64/python2.7/site-packages/. This is >> true on Linus' latest tree, but it goes back to arriving with the 4.2 >> kernel. >> >> Digging around a bit, it seems the files installed are those that are >> found as relative paths in the utils/python-ext-sources directory: >> >> /usr/lib64/python2.7/site-packages/bitmap.o >> /usr/lib64/python2.7/site-packages/find_bit.o >> /usr/lib64/python2.7/site-packages/hweight.o >> /usr/lib64/python2.7/site-packages/rbtree.o >> >> None of those files should really be installed at all, as they're >> temporary object files used to build perf.so. >> >> As far as I can tell, this is a side effect of using the python >> distutils module with it's setup.py script. The sources are listed >> there so they get included, but then the build_ext functionality >> interprets it as e.g.: >> >> -c ../lib/hweight.c -o python_ext_build/tmp/../lib/hweight.o >> >> That leaves it in the lib/ directory of the build output alongside the >> final perf.so instead of python_ext_build/tmp/util/ like the rest of >> the object files. It happens to be an unfortunate coincidence that >> install-lib looks in said .../lib/ directory and we just happen to >> conflict with that naming. So when install-python_ext is called, it >> simply copies everything in that directory, including the .o files. >> >> I'm not enough of a python wizard to figure out how to fix this >> quickly. We could remove the .o files in the .spec file, but that >> seems hacky and this should probably be fixed in the perf build >> system. >> >> Ideas? > > looks like the python extension build system mixes > up the '../lib' source files with final lib, which > by 'accident' is located in '../lib' as well ;-) Right, that's what I was trying to say above. > using full paths for sources sorted this for me (patch attached) Ah, yeah I had the idea to use full paths, but couldn't quickly figure out how to do that. > but that will end up with following build tree: > > ./python_ext_build > ./python_ext_build/lib > ./python_ext_build/lib/perf.so > ./python_ext_build/tmp > ./python_ext_build/tmp/home > ./python_ext_build/tmp/home/jolsa > ./python_ext_build/tmp/home/jolsa/kernel > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/counts.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/util.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/thread_map.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/cpumap.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/strlist.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/evlist.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/evsel.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/trace-event.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/xyarray.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/python.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/ctype.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/cgroup.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/rblist.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/string.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/hweight.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/rbtree.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/bitmap.o > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/find_bit.o > > not sure we want to come up with some 'nicer' solution I don't think anything under python_ext_build/tmp/ actually matters in any significant way. As long as this doesn't negatively impact something via side-effect, it's probably good enough. Will you write up a full patch? josh > --- > diff --git a/tools/perf/util/setup.py b/tools/perf/util/setup.py > index 1833103768cb..6c4bb2ea3594 100644 > --- a/tools/perf/util/setup.py > +++ b/tools/perf/util/setup.py > @@ -22,6 +22,7 @@ cflags = getenv('CFLAGS', '').split() > # switch off several checks (need to be at the end of cflags list) > cflags += ['-fno-strict-aliasing', '-Wno-write-strings', '-Wno-unused-parameter' ] > > +src_perf = getenv('srctree') + '/tools/perf' > build_lib = getenv('PYTHON_EXTBUILD_LIB') > build_tmp = getenv('PYTHON_EXTBUILD_TMP') > libtraceevent = getenv('LIBTRACEEVENT') > @@ -30,6 +31,8 @@ libapikfs = getenv('LIBAPI') > ext_sources = [f.strip() for f in file('util/python-ext-sources') > if len(f.strip()) > 0 and f[0] != '#'] > > +ext_sources = map(lambda x: '%s/%s' % (src_perf, x) , ext_sources) > + > perf = Extension('perf', > sources = ext_sources, > include_dirs = ['util/include'], ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: perf install-python_ext installing .o files in python site-packages 2016-02-26 22:49 ` Josh Boyer @ 2016-02-26 22:59 ` Jiri Olsa 0 siblings, 0 replies; 4+ messages in thread From: Jiri Olsa @ 2016-02-26 22:59 UTC (permalink / raw) To: Josh Boyer; +Cc: Arnaldo Carvalho de Melo, Linux-Kernel@Vger. Kernel. Org On Fri, Feb 26, 2016 at 05:49:02PM -0500, Josh Boyer wrote: SNIP > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/evsel.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/trace-event.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/xyarray.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/python.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/ctype.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/cgroup.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/rblist.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/perf/util/string.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/hweight.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/rbtree.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/bitmap.o > > ./python_ext_build/tmp/home/jolsa/kernel/linux-perf/tools/lib/find_bit.o > > > > not sure we want to come up with some 'nicer' solution > > I don't think anything under python_ext_build/tmp/ actually matters in > any significant way. As long as this doesn't negatively impact > something via side-effect, it's probably good enough. right, and it's probably generic sollution too > > Will you write up a full patch? yep, will post it tomorrow jirka ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2016-02-26 22:59 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-02-26 20:38 perf install-python_ext installing .o files in python site-packages Josh Boyer 2016-02-26 21:14 ` Jiri Olsa 2016-02-26 22:49 ` Josh Boyer 2016-02-26 22:59 ` Jiri Olsa
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox
all inboxes | Powered by JetHome®