[PATCH] objtool: drop redundant flags generation

[PATCH] objtool: drop redundant flags generation

[Nicholas Mc Guire]

The generator was emitting quite a few duplicate flags which was making
doublebitand.cocci nervous. This awk hack resolves the duplicate issue.

Signed-off-by: Nicholas Mc Guire <der.herr@hofr.at>
---

The coccinelle complaints emitted was about 230 findings total:
./arch/x86/lib/inat-tables.c:214:10-20: duplicated argument to & or |
./arch/x86/lib/inat-tables.c:214:23-33: duplicated argument to & or |
./arch/x86/lib/inat-tables.c:218:10-20: duplicated argument to & or |
./arch/x86/lib/inat-tables.c:218:23-33: duplicated argument to & or |
....
./tools/objtool/arch/x86/insn/inat-tables.c:214:10-20: duplicated argument to & or |
./tools/objtool/arch/x86/insn/inat-tables.c:214:23-33: duplicated argument to & or |
./tools/objtool/arch/x86/insn/inat-tables.c:218:10-20: duplicated argument to & or |
./tools/objtool/arch/x86/insn/inat-tables.c:218:23-33: duplicated argument to & or |
...
spatch --sp-file scripts/coccinelle/tests/doublebitand.cocci inat-tables.c -D report
will give you the full list - all are caused by duplicates in the generated
output by the add_flags function in the two instances of gen-insn-attr-x86.awk.

Q: The two copies of gen-insn-attr-x86.awk are identical and its not actually clear
   why this duplication is needed ? Further the maintainers list emitted for the
   two files differ.

Patch was checked by manual review of the diff between the initial file and the
regenerated file after the below patch was applied.
Second verification was by make tools/objtool and comparing the generated binaries
in tools/objtool/arch/x86/decode.o with diff.

Patch is against 4.11-rc1 (localversion-next is next-20170306)

 arch/x86/tools/gen-insn-attr-x86.awk              | 12 ++++++++++--
 tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk | 12 ++++++++++--
 2 files changed, 20 insertions(+), 4 deletions(-)

diff --git a/arch/x86/tools/gen-insn-attr-x86.awk b/arch/x86/tools/gen-insn-attr-x86.awk
index a3d2c62..9cdeefe 100644
--- a/arch/x86/tools/gen-insn-attr-x86.awk
+++ b/arch/x86/tools/gen-insn-attr-x86.awk
@@ -226,8 +226,16 @@ function print_table(tbl,name,fmt,n)
 }
 
 function add_flags(old,new) {
-	if (old && new)
-		return old " | " new
+	if (old == new)
+		return old
+	if (old && new) {
+		if(match(old,new))
+			return old
+		else if(match(new,old))
+			return new
+		else
+			return old " | " new
+        }
 	else if (old)
 		return old
 	else
diff --git a/tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk b/tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk
index a3d2c62..9cdeefe 100644
--- a/tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk
+++ b/tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk
@@ -226,8 +226,16 @@ function print_table(tbl,name,fmt,n)
 }
 
 function add_flags(old,new) {
-	if (old && new)
-		return old " | " new
+	if (old == new)
+		return old
+	if (old && new) {
+		if(match(old,new))
+			return old
+		else if(match(new,old))
+			return new
+		else
+			return old " | " new
+        }
 	else if (old)
 		return old
 	else
-- 
2.1.4

Re: [PATCH] objtool: drop redundant flags generation

[Josh Poimboeuf]

On Mon, Mar 06, 2017 at 06:00:25PM +0100, Nicholas Mc Guire wrote:
> The generator was emitting quite a few duplicate flags which was making
> doublebitand.cocci nervous. This awk hack resolves the duplicate issue.
> 
> Signed-off-by: Nicholas Mc Guire <der.herr@hofr.at>
> ---
> 
> The coccinelle complaints emitted was about 230 findings total:
> ./arch/x86/lib/inat-tables.c:214:10-20: duplicated argument to & or |
> ./arch/x86/lib/inat-tables.c:214:23-33: duplicated argument to & or |
> ./arch/x86/lib/inat-tables.c:218:10-20: duplicated argument to & or |
> ./arch/x86/lib/inat-tables.c:218:23-33: duplicated argument to & or |
> ....
> ./tools/objtool/arch/x86/insn/inat-tables.c:214:10-20: duplicated argument to & or |
> ./tools/objtool/arch/x86/insn/inat-tables.c:214:23-33: duplicated argument to & or |
> ./tools/objtool/arch/x86/insn/inat-tables.c:218:10-20: duplicated argument to & or |
> ./tools/objtool/arch/x86/insn/inat-tables.c:218:23-33: duplicated argument to & or |
> ...
> spatch --sp-file scripts/coccinelle/tests/doublebitand.cocci inat-tables.c -D report
> will give you the full list - all are caused by duplicates in the generated
> output by the add_flags function in the two instances of gen-insn-attr-x86.awk.
> 
> Q: The two copies of gen-insn-attr-x86.awk are identical and its not actually clear
>    why this duplication is needed ? Further the maintainers list emitted for the
>    two files differ.
> 
> Patch was checked by manual review of the diff between the initial file and the
> regenerated file after the below patch was applied.
> Second verification was by make tools/objtool and comparing the generated binaries
> in tools/objtool/arch/x86/decode.o with diff.
> 
> Patch is against 4.11-rc1 (localversion-next is next-20170306)
> 
>  arch/x86/tools/gen-insn-attr-x86.awk              | 12 ++++++++++--
>  tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk | 12 ++++++++++--

There's actually a third copy of the decoder in:

  tools/perf/util/intel-pt-decoder/

Yes, the duplication is a pain, but it's part of an effort to keep
'tools/*' source independent from kernel code.

Maybe we can at least combine the objtool and perf versions someday.

-- 
Josh

Re: [PATCH] objtool: drop redundant flags generation

[Nicholas Mc Guire]

On Mon, Mar 06, 2017 at 11:25:37AM -0600, Josh Poimboeuf wrote:
> On Mon, Mar 06, 2017 at 06:00:25PM +0100, Nicholas Mc Guire wrote:
> > The generator was emitting quite a few duplicate flags which was making
> > doublebitand.cocci nervous. This awk hack resolves the duplicate issue.
> > 
> > Signed-off-by: Nicholas Mc Guire <der.herr@hofr.at>
> > ---
> > 
> > The coccinelle complaints emitted was about 230 findings total:
> > ./arch/x86/lib/inat-tables.c:214:10-20: duplicated argument to & or |
> > ./arch/x86/lib/inat-tables.c:214:23-33: duplicated argument to & or |
> > ./arch/x86/lib/inat-tables.c:218:10-20: duplicated argument to & or |
> > ./arch/x86/lib/inat-tables.c:218:23-33: duplicated argument to & or |
> > ....
> > ./tools/objtool/arch/x86/insn/inat-tables.c:214:10-20: duplicated argument to & or |
> > ./tools/objtool/arch/x86/insn/inat-tables.c:214:23-33: duplicated argument to & or |
> > ./tools/objtool/arch/x86/insn/inat-tables.c:218:10-20: duplicated argument to & or |
> > ./tools/objtool/arch/x86/insn/inat-tables.c:218:23-33: duplicated argument to & or |
> > ...
> > spatch --sp-file scripts/coccinelle/tests/doublebitand.cocci inat-tables.c -D report
> > will give you the full list - all are caused by duplicates in the generated
> > output by the add_flags function in the two instances of gen-insn-attr-x86.awk.
> > 
> > Q: The two copies of gen-insn-attr-x86.awk are identical and its not actually clear
> >    why this duplication is needed ? Further the maintainers list emitted for the
> >    two files differ.
> > 
> > Patch was checked by manual review of the diff between the initial file and the
> > regenerated file after the below patch was applied.
> > Second verification was by make tools/objtool and comparing the generated binaries
> > in tools/objtool/arch/x86/decode.o with diff.
> > 
> > Patch is against 4.11-rc1 (localversion-next is next-20170306)
> > 
> >  arch/x86/tools/gen-insn-attr-x86.awk              | 12 ++++++++++--
> >  tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk | 12 ++++++++++--
> 
> There's actually a third copy of the decoder in:
> 
>   tools/perf/util/intel-pt-decoder/
> 
> Yes, the duplication is a pain, but it's part of an effort to keep
> 'tools/*' source independent from kernel code.
> 
> Maybe we can at least combine the objtool and perf versions someday.
>
Bad - missed that one - did not build perf - the generator seems to
be the same though only differing by a single blank line - so pulling 
those together should be a non-issue atleast with respect to the 
generator as the x86-opcode-map.txt are all the same ? ...or what 
fun am I missing ?

thx!
hofrat

Re: [PATCH] objtool: drop redundant flags generation

[Josh Poimboeuf]

On Mon, Mar 06, 2017 at 05:54:01PM +0000, Nicholas Mc Guire wrote:
> On Mon, Mar 06, 2017 at 11:25:37AM -0600, Josh Poimboeuf wrote:
> > >  arch/x86/tools/gen-insn-attr-x86.awk              | 12 ++++++++++--
> > >  tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk | 12 ++++++++++--
> > 
> > There's actually a third copy of the decoder in:
> > 
> >   tools/perf/util/intel-pt-decoder/
> > 
> > Yes, the duplication is a pain, but it's part of an effort to keep
> > 'tools/*' source independent from kernel code.
> > 
> > Maybe we can at least combine the objtool and perf versions someday.
> >
> Bad - missed that one - did not build perf - the generator seems to
> be the same though only differing by a single blank line - so pulling 
> those together should be a non-issue atleast with respect to the 
> generator as the x86-opcode-map.txt are all the same ? ...or what 
> fun am I missing ?

In theory, all three copies of the decoder should be identical.  That
includes all the files: insn.[ch], inat.[ch], inat_types.h,
gen-insn-attr-x86.awk, x86-opcode-map.txt.

-- 
Josh

Re: [PATCH] objtool: drop redundant flags generation

[Nicholas Mc Guire]

On Mon, Mar 06, 2017 at 03:40:54PM -0600, Josh Poimboeuf wrote:
> On Mon, Mar 06, 2017 at 05:54:01PM +0000, Nicholas Mc Guire wrote:
> > On Mon, Mar 06, 2017 at 11:25:37AM -0600, Josh Poimboeuf wrote:
> > > >  arch/x86/tools/gen-insn-attr-x86.awk              | 12 ++++++++++--
> > > >  tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk | 12 ++++++++++--
> > > 
> > > There's actually a third copy of the decoder in:
> > > 
> > >   tools/perf/util/intel-pt-decoder/
> > > 
> > > Yes, the duplication is a pain, but it's part of an effort to keep
> > > 'tools/*' source independent from kernel code.
> > > 
> > > Maybe we can at least combine the objtool and perf versions someday.
> > >
> > Bad - missed that one - did not build perf - the generator seems to
> > be the same though only differing by a single blank line - so pulling 
> > those together should be a non-issue atleast with respect to the 
> > generator as the x86-opcode-map.txt are all the same ? ...or what 
> > fun am I missing ?
> 
> In theory, all three copies of the decoder should be identical.  That
> includes all the files: insn.[ch], inat.[ch], inat_types.h,
> gen-insn-attr-x86.awk, x86-opcode-map.txt.
>
Understood - but this is a different problem that is being
addressed with this cleanup - the duplicates make no sense in
any case as far as I can see - with or without consolidation of
the other files (and x86-opcode-map.txt does seem to be the
same in all 3 cases) - the point was that it is causing a quite
large number of coccicheck warnings which are iritating and
in this case easy to remove.

So the question is does it make sense to fix up this one aspect
or is this "fixing" an intermediate mess only that will go away,
once consolidation happens, anyway.

thx!
hofrat

Re: [PATCH] objtool: drop redundant flags generation

[Masami Hiramatsu]

On Tue, 7 Mar 2017 08:13:19 +0000
Nicholas Mc Guire <der.herr@hofr.at> wrote:

> On Mon, Mar 06, 2017 at 03:40:54PM -0600, Josh Poimboeuf wrote:
> > On Mon, Mar 06, 2017 at 05:54:01PM +0000, Nicholas Mc Guire wrote:
> > > On Mon, Mar 06, 2017 at 11:25:37AM -0600, Josh Poimboeuf wrote:
> > > > >  arch/x86/tools/gen-insn-attr-x86.awk              | 12 ++++++++++--
> > > > >  tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk | 12 ++++++++++--
> > > > 
> > > > There's actually a third copy of the decoder in:
> > > > 
> > > >   tools/perf/util/intel-pt-decoder/
> > > > 
> > > > Yes, the duplication is a pain, but it's part of an effort to keep
> > > > 'tools/*' source independent from kernel code.
> > > > 
> > > > Maybe we can at least combine the objtool and perf versions someday.
> > > >
> > > Bad - missed that one - did not build perf - the generator seems to
> > > be the same though only differing by a single blank line - so pulling 
> > > those together should be a non-issue atleast with respect to the 
> > > generator as the x86-opcode-map.txt are all the same ? ...or what 
> > > fun am I missing ?
> > 
> > In theory, all three copies of the decoder should be identical.  That
> > includes all the files: insn.[ch], inat.[ch], inat_types.h,
> > gen-insn-attr-x86.awk, x86-opcode-map.txt.
> >
> Understood - but this is a different problem that is being
> addressed with this cleanup - the duplicates make no sense in
> any case as far as I can see - with or without consolidation of
> the other files (and x86-opcode-map.txt does seem to be the
> same in all 3 cases) - the point was that it is causing a quite
> large number of coccicheck warnings which are iritating and
> in this case easy to remove.

Why would you apply coccinelle to auto-generated code?
I recommend you to change coccicheck to avoid checking
auto-generated code first.

Thank you,


-- 
Masami Hiramatsu <mhiramat@kernel.org>

Re: [PATCH] objtool: drop redundant flags generation

[Nicholas Mc Guire]

On Tue, Mar 07, 2017 at 02:55:21PM +0100, Masami Hiramatsu wrote:
> On Tue, 7 Mar 2017 08:13:19 +0000
> Nicholas Mc Guire <der.herr@hofr.at> wrote:
> 
> > On Mon, Mar 06, 2017 at 03:40:54PM -0600, Josh Poimboeuf wrote:
> > > On Mon, Mar 06, 2017 at 05:54:01PM +0000, Nicholas Mc Guire wrote:
> > > > On Mon, Mar 06, 2017 at 11:25:37AM -0600, Josh Poimboeuf wrote:
> > > > > >  arch/x86/tools/gen-insn-attr-x86.awk              | 12 ++++++++++--
> > > > > >  tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk | 12 ++++++++++--
> > > > > 
> > > > > There's actually a third copy of the decoder in:
> > > > > 
> > > > >   tools/perf/util/intel-pt-decoder/
> > > > > 
> > > > > Yes, the duplication is a pain, but it's part of an effort to keep
> > > > > 'tools/*' source independent from kernel code.
> > > > > 
> > > > > Maybe we can at least combine the objtool and perf versions someday.
> > > > >
> > > > Bad - missed that one - did not build perf - the generator seems to
> > > > be the same though only differing by a single blank line - so pulling 
> > > > those together should be a non-issue atleast with respect to the 
> > > > generator as the x86-opcode-map.txt are all the same ? ...or what 
> > > > fun am I missing ?
> > > 
> > > In theory, all three copies of the decoder should be identical.  That
> > > includes all the files: insn.[ch], inat.[ch], inat_types.h,
> > > gen-insn-attr-x86.awk, x86-opcode-map.txt.
> > >
> > Understood - but this is a different problem that is being
> > addressed with this cleanup - the duplicates make no sense in
> > any case as far as I can see - with or without consolidation of
> > the other files (and x86-opcode-map.txt does seem to be the
> > same in all 3 cases) - the point was that it is causing a quite
> > large number of coccicheck warnings which are iritating and
> > in this case easy to remove.
> 
> Why would you apply coccinelle to auto-generated code?

Why should autogenerated code have the priviledge of non-conformance
and limited understandability ?
Coccicheck is applied to the entire kernel and it is not imediately 
visible/clear what is autogenerated and what not and I see no reason
why autogenerated code should not conform to coding standards or 
to readability rules. Now in this particular case it might well be
thta nobody ever reads that generated file anyway - I cant say - never
the less it will show up in test-harneses as noise.

> I recommend you to change coccicheck to avoid checking
> auto-generated code first.

Frist there is no canonic way to even detect if code is generated or not 
and further if you give me a good reason why generated code does not need 
to follow coding standards and does not need to be readable and/or 
understandable ?

We have autogenerated code in the kernel that is so unbelievably
braindead it is scary that it ever made it into mainline !
Here a few highlights that coccinelle found with some scripts
that are not in mainline:

drivers/media/dvb-frontends/dib7000m.c:926 
  /* P_dintl_native, P_dintlv_inv, 
     P_hrch, P_code_rate, P_select_hp */
  value = 0; 
  if (1 != 0) 
          value |= (1 << 6); 
  if (ch->hierarchy == 1) 
          value |= (1 << 4);
  if (1 == 1)
          value |= 1;
  switch ((ch->hierarchy == 0 || 1 == 1) ?
          ch->code_rate_HP : ch->code_rate_LP) {

aside from braindead control flow - the comment has nothing to do with the
code - explaination by author: generated code !

drivers/staging/rtl8723au/hal/rtl8723a\_bt-coexist.c:7264 else duplicates if
   ...
   } else if (maxInterval == 2) {
           btdm_2AntPsTdma(padapter, true, 15);
           pBtdm8723->psTdmaDuAdjType = 15;
   } else if (maxInterval == 3) {
           btdm_2AntPsTdma(padapter, true, 15);
           pBtdm8723->psTdmaDuAdjType = 15;
   } else {
           btdm_2AntPsTdma(padapter, true, 15);
           pBtdm8723->psTdmaDuAdjType = 15;
   }

also an example of generated code - CamelCase naming +
totally usless control flow (actually its 56 lines that
collaps into a single if/else) - if you ever hit a problem
in such code it probably qualifies as hedonistic outbreak.

Now the generated code in the case of gen-insn-attr-x86.awk 
is admitedly not in the same "class" as the above examples but
never the less there is no reason for it, and generated code
in general, to reduce readable, understandable and maintainable 
notably long term (someone may need to look at autogenerated 
code in 15 years with the specific tools no longer avaialble).

In this specific case - given the overall complexity/size of
gen-insn-attr-x86.awk I do not quite see the big issue with adding
those 10 lines to remove almost 400 coccinelle warnings - but
I´m also not into the specifics of that code so if you feel that
it would cause more problems than benefit no issue with not taking
the proposed patch (and if someone has a nicer awk solution that
might be even better !)

thx!
hofrat

Re: [PATCH] objtool: drop redundant flags generation

[Masami Hiramatsu]

On Mon,  6 Mar 2017 18:00:25 +0100
Nicholas Mc Guire <der.herr@hofr.at> wrote:

> The generator was emitting quite a few duplicate flags which was making
> doublebitand.cocci nervous. This awk hack resolves the duplicate issue.

Yes, I know that.
I don't think that the duplicating those flags in "automatic generated"
source code is not so harmful. I personally prefer to keep awk code
simpler...

Thanks,

> 
> Signed-off-by: Nicholas Mc Guire <der.herr@hofr.at>
> ---
> 
> The coccinelle complaints emitted was about 230 findings total:
> ./arch/x86/lib/inat-tables.c:214:10-20: duplicated argument to & or |
> ./arch/x86/lib/inat-tables.c:214:23-33: duplicated argument to & or |
> ./arch/x86/lib/inat-tables.c:218:10-20: duplicated argument to & or |
> ./arch/x86/lib/inat-tables.c:218:23-33: duplicated argument to & or |
> ....
> ./tools/objtool/arch/x86/insn/inat-tables.c:214:10-20: duplicated argument to & or |
> ./tools/objtool/arch/x86/insn/inat-tables.c:214:23-33: duplicated argument to & or |
> ./tools/objtool/arch/x86/insn/inat-tables.c:218:10-20: duplicated argument to & or |
> ./tools/objtool/arch/x86/insn/inat-tables.c:218:23-33: duplicated argument to & or |
> ...
> spatch --sp-file scripts/coccinelle/tests/doublebitand.cocci inat-tables.c -D report
> will give you the full list - all are caused by duplicates in the generated
> output by the add_flags function in the two instances of gen-insn-attr-x86.awk.
> 
> Q: The two copies of gen-insn-attr-x86.awk are identical and its not actually clear
>    why this duplication is needed ? Further the maintainers list emitted for the
>    two files differ.
> 
> Patch was checked by manual review of the diff between the initial file and the
> regenerated file after the below patch was applied.
> Second verification was by make tools/objtool and comparing the generated binaries
> in tools/objtool/arch/x86/decode.o with diff.
> 
> Patch is against 4.11-rc1 (localversion-next is next-20170306)
> 
>  arch/x86/tools/gen-insn-attr-x86.awk              | 12 ++++++++++--
>  tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk | 12 ++++++++++--
>  2 files changed, 20 insertions(+), 4 deletions(-)
> 
> diff --git a/arch/x86/tools/gen-insn-attr-x86.awk b/arch/x86/tools/gen-insn-attr-x86.awk
> index a3d2c62..9cdeefe 100644
> --- a/arch/x86/tools/gen-insn-attr-x86.awk
> +++ b/arch/x86/tools/gen-insn-attr-x86.awk
> @@ -226,8 +226,16 @@ function print_table(tbl,name,fmt,n)
>  }
>  
>  function add_flags(old,new) {
> -	if (old && new)
> -		return old " | " new
> +	if (old == new)
> +		return old
> +	if (old && new) {
> +		if(match(old,new))
> +			return old
> +		else if(match(new,old))
> +			return new
> +		else
> +			return old " | " new
> +        }
>  	else if (old)
>  		return old
>  	else
> diff --git a/tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk b/tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk
> index a3d2c62..9cdeefe 100644
> --- a/tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk
> +++ b/tools/objtool/arch/x86/insn/gen-insn-attr-x86.awk
> @@ -226,8 +226,16 @@ function print_table(tbl,name,fmt,n)
>  }
>  
>  function add_flags(old,new) {
> -	if (old && new)
> -		return old " | " new
> +	if (old == new)
> +		return old
> +	if (old && new) {
> +		if(match(old,new))
> +			return old
> +		else if(match(new,old))
> +			return new
> +		else
> +			return old " | " new
> +        }
>  	else if (old)
>  		return old
>  	else
> -- 
> 2.1.4
> 


-- 
Masami Hiramatsu <mhiramat@kernel.org>