Number of bugs fixed in a release introduced in the previous one

Number of bugs fixed in a release introduced in the previous one

[Alejandro Colomar]

[-- Attachment #1: Type: text/plain, Size: 3344 bytes --]

Hi Greg, Jon,

I've compiled the data from LWN as Jon suggested me earlier today using
<https://lwn.net/ksdb/releases/v4.0/fixes> (with the appropriate
version), since there don't exist any graphs for this data.

Why I'm using immediate fixes and not all fixes is because at the moment
we only know how many bugs were introduced in 7.1 that were fixed in
7.2, but we don't know how many will appear.  To compare apples to
apples, we should compare to the same information we had back then when
for example 6.2 was released, when we only know the bugs introduced in
6.1 and fixed in 6.2, but not those fixed in later versions.

Here's the data since v4.0, which is old enough:

	Fixes in a version for bugs in the previous verion

	7.2	279	*
	7.1	253	*
	7.0	276	*

	6.19	236
	6.18	229
	6.17	245
	6.16	241
	6.15	169	-

	6.14	206
	6.13	214
	6.12	244
	6.11	230
	6.10	208

	6.9	278	*
	6.8	241
	6.7	193	-
	6.6	245
	6.5	210

	6.4	216
	6.3	221
	6.2	240
	6.1	243
	6.0	265	*

	5.19	236
	5.18	228
	5.17	237
	5.16	256	*
	5.15	229

	5.14	258	*
	5.13	227
	5.12	218
	5.11	259	*
	5.10	254	*

	5.9	243
	5.8	267	*
	5.7	241
	5.6	242
	5.5	275	*

	5.4	215
	5.3	217
	5.2	198	-
	5.1	219
	5.0	232

	4.20	200
	4.19	171	-
	4.18	173	-
	4.17	166	-
	4.16	203
	4.15	186	-

	4.14	197	-
	4.13	164	-
	4.12	135	--
	4.11	167	-
	4.10	167	-

	4.9	153	-
	4.8	115	--
	4.7	126	--
	4.6	91	---
	4.5	93	---

	4.4	85	---
	4.3	103	--
	4.2	66	---
	4.1	60	---
	4.0	53	---

I've added a '-' next to those numbers below 200 ('--' below 150;
'---' below 100).  I've added a '*' next to those numbers above 250.

There seems to be an overall increase of bugs immediately (next release)
fixed since the 4.x releases until the last 5.x releases, and it could
be due to any or many of:

-  More consistently using 'Fixes:'.
-  Having simpler code, which allows finding bugs earlier.
-  Better testing.
-  But it could also be due to actually introducing more bugs.

Interestingly, the number of immediate fixes decreased from 5.x to 6.x
(but not below 4.x levels).  I suspect programmers didn't stop using
'Fixes:' tags, and the code and testing was likely not worse than
before, so this seems a legitimate decrease in introduction of bugs in
6.x compared to 5.x.

However, if we compare the 7.x releases (those with a notable increase
in LLM usage) to the 6.x releases, we see the immediate fixes increased
again, and further than in 5.x, as we're now consistently above 250
immediate fixes in all releases.  It could be that LLMs use 'Fixes:'
tags more than humans, but it could also be a trend change towards
introducing more bugs.

It would be interesting to follow this trend in future releases, to see
if it continues to be sustained, and if these releases are indeed more
buggy or not.

It's still relatively fine, though, since it didn't skyrocket to
something like 300.  The increase is only slight (at least for now).

Another thing that keeps me worried is that the bugs introduced by LLMs
tend to be more obscure (since they are meant to look like legitimate
patches), and so their discovery might take longer, in which case, the
situtation might be worse than what we see here.


Have a lovely evening!
Alex

-- 
<https://www.alejandro-colomar.es>

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]