[PROPOSAL] Replace gcov and kcov with llvm-cov

[PROPOSAL] Replace gcov and kcov with llvm-cov

[Chuck Wolber]

Stemming from some hallway track conversations with maintainers at LPC
2026, I propose replacing gcov and kcov in the kernel with llvm-cov.

llvm-cov instruments at the AST which enables precise mapping back to
source code regardless of optimization level.

We have a set of patches [1] that implements kernel based llvm-cov. They
need to be updated and some additional patches are required to address
some stubs that are missing noinstr annotation. But otherwise I should
be able to send a full set of patches (including taking on the
maintainer role) to remove gcov and kcov and replace it with llvm-cov in
fairly short order.

If there is a desire to take this in pieces or implement other
intermediate steps, let me know. Otherwise I can generate a monolithic
set all at once.

[1] https://lore.kernel.org/lkml/20240905043245.1389509-1-wentaoz5@illinois.edu/

Thank you,

..Ch:W..

Re: [PROPOSAL] Replace gcov and kcov with llvm-cov

[Marco Elver]

On Tue, 6 Oct 2026 at 17:14, Chuck Wolber <chuck@wolber.net> wrote:
>
> Stemming from some hallway track conversations with maintainers at LPC
> 2026, I propose replacing gcov and kcov in the kernel with llvm-cov.

I know the origin of the discussion, and wanted to chat, but I guess
that didn't happen and instead we get this proposal. It would have
been good to understand the requirements of users of gcov and kcov
(more below).

> llvm-cov instruments at the AST which enables precise mapping back to
> source code regardless of optimization level.
>
> We have a set of patches [1] that implements kernel based llvm-cov. They
> need to be updated and some additional patches are required to address
> some stubs that are missing noinstr annotation. But otherwise I should
> be able to send a full set of patches (including taking on the
> maintainer role) to remove gcov and kcov and replace it with llvm-cov in
> fairly short order.

The hallway chat was supposed to figure out if there's a way to bridge
some of the gaps, but jumping the gun like this without having
addressed the requirements of kcov users does not make any sense at
all.  As for gcov, llvm-cov was designed as a better replacement for
gcov, so that makes more sense.

Ok, well you made me untangle the most important differences between
kcov vs. gcov/llvm-cov below so we have it for the record.

> If there is a desire to take this in pieces or implement other
> intermediate steps, let me know. Otherwise I can generate a monolithic
> set all at once.
>
> [1] https://lore.kernel.org/lkml/20240905043245.1389509-1-wentaoz5@illinois.edu/

llvm-cov is generally useful to have; llvm-cov is closest to gcov, so
that might make sense to replace.

KCOV, however, cannot be replaced by llvm-cov. KCOV isn't a coverage
reporting tool for users, it's a low-overhead interface for fuzzers
(syzkaller/syzbot and many others). llvm-cov has no equivalent of:

- per-task and remote (kthread/softirq/USB/vhost) coverage collection
(to avoid polluting coverage with concurrent tasks);
llvm-cov counters are global and polluted by everything else running;

- ordered PC traces (edge signal) and comparison operands (TRACE_CMP);

- performance: a per-task mmap'd buffer reset with one store, vs.
reading/resetting
megabytes of global counters after every program;

- GCC support (we may want to keep at least one GCC-supported coverage tool).

It's also UAPI (include/uapi/linux/kcov.h), with existing users.
syzkaller heavily relies on KCOV's features above to achieve good
performance and coverage; there's no reasonable way today to migrate
to anything else.

As-is, NAK on removing KCOV. Consider replacing gcov with it.

Thanks,
-- Marco

Re: [PROPOSAL] Replace gcov and kcov with llvm-cov

[Chuck Wolber]

On Tue, Oct 6, 2026, at 4:10 PM, Marco Elver wrote:
> On Tue, 6 Oct 2026 at 17:14, Chuck Wolber <chuck@wolber.net> wrote:
>>
>> Stemming from some hallway track conversations with maintainers at LPC
>> 2026, I propose replacing gcov and kcov in the kernel with llvm-cov.
>
> I know the origin of the discussion, and wanted to chat, but I guess
> that didn't happen and instead we get this proposal. It would have
> been good to understand the requirements of users of gcov and kcov
> (more below).

No problem, I still plan to seek you out to discuss this further. I just
wanted to get the discussion going in case there was a broader set of
opinions on this matter.


>> We have a set of patches [1] that implements kernel based llvm-cov.
>> They need to be updated and some additional patches are required to
>> address some stubs that are missing noinstr annotation. But otherwise
>> I should be able to send a full set of patches (including taking on
>> the maintainer role) to remove gcov and kcov and replace it with
>> llvm-cov in fairly short order.
>
> The hallway chat was supposed to figure out if there's a way to bridge
> some of the gaps, but jumping the gun like this without having
> addressed the requirements of kcov users does not make any sense at
> all.

I totally agree, and I am still up for that discussion.


>> If there is a desire to take this in pieces or implement other
>> intermediate steps, let me know. Otherwise I can generate a
>> monolithic set all at once.
>>
>> [1] https://lore.kernel.org/lkml/20240905043245.1389509-1-wentaoz5@illinois.edu/
>
> llvm-cov is generally useful to have; llvm-cov is closest to gcov, so
> that might make sense to replace.

Swapping llvm-cov for gcov to start with is pretty straightforward, so I
can aim my initial patch set there.


> KCOV, however, cannot be replaced by llvm-cov. KCOV isn't a coverage
> reporting tool for users, it's a low-overhead interface for fuzzers
> (syzkaller/syzbot and many others). llvm-cov has no equivalent of:
>
> - per-task and remote (kthread/softirq/USB/vhost) coverage collection
> (to avoid polluting coverage with concurrent tasks);
> llvm-cov counters are global and polluted by everything else running;

Correct, and I have pondered some ideas for doing exactly that, but they
would be quite experimental at this time. The patches come with the
ability to clear the counters so that specific tests can be run, but
that is still monolithic. I would love to discuss the potential for
other methods of masking coverage so we can get granular results.


> - ordered PC traces (edge signal) and comparison operands (TRACE_CMP);
>
> - performance: a per-task mmap'd buffer reset with one store, vs.
> reading/resetting
> megabytes of global counters after every program;
>
> - GCC support (we may want to keep at least one GCC-supported coverage tool).
>
> It's also UAPI (include/uapi/linux/kcov.h), with existing users.
> syzkaller heavily relies on KCOV's features above to achieve good
> performance and coverage; there's no reasonable way today to migrate
> to anything else.
>
> As-is, NAK on removing KCOV. Consider replacing gcov with it.

Understood. I will focus on replacing gcov for now, and I am interested
in exploring what else is possible.

Thank you,

..Ch:W..

Re: [PROPOSAL] Replace gcov and kcov with llvm-cov

[Masami Hiramatsu]

On Tue, 06 Oct 2026 16:27:11 +0000, Chuck Wolber <chuck@wolber.net> wrote:
> On Tue, Oct 6, 2026, at 4:10 PM, Marco Elver wrote:
> > On Tue, 6 Oct 2026 at 17:14, Chuck Wolber <chuck@wolber.net> wrote:

> >> If there is a desire to take this in pieces or implement other
> >> intermediate steps, let me know. Otherwise I can generate a
> >> monolithic set all at once.
> >>
> >> [1] https://lore.kernel.org/lkml/20240905043245.1389509-1-wentaoz5@illinois.edu/
> >
> > llvm-cov is generally useful to have; llvm-cov is closest to gcov, so
> > that might make sense to replace.
>
> Swapping llvm-cov for gcov to start with is pretty straightforward, so I
> can aim my initial patch set there.

Hmm, is it necessary to remove gcov? Would it be difficult to simply add
support for llvm-cov and enable (make kconfig selectable) one or the other
depending on the compiler?

Thanks,

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>

Re: [PROPOSAL] Replace gcov and kcov with llvm-cov

[Chuck Wolber]

On Wed, Oct 7, 2026, at 2:30 AM, Masami Hiramatsu wrote:
> On Tue, 06 Oct 2026 16:27:11 +0000, Chuck Wolber <chuck@wolber.net> wrote:
>> On Tue, Oct 6, 2026, at 4:10 PM, Marco Elver wrote:
>> > On Tue, 6 Oct 2026 at 17:14, Chuck Wolber <chuck@wolber.net> wrote:
>
>> >> If there is a desire to take this in pieces or implement other
>> >> intermediate steps, let me know. Otherwise I can generate a
>> >> monolithic set all at once.
>> >>
>> >> [1] https://lore.kernel.org/lkml/20240905043245.1389509-1-wentaoz5@illinois.edu/
>> >>
>> > >> > llvm-cov is generally useful to have; llvm-cov is closest to
>> > >> > gcov, so that might make sense to replace.
>>
>> Swapping llvm-cov for gcov to start with is pretty straightforward,
>> so I can aim my initial patch set there.
>
> Hmm, is it necessary to remove gcov? Would it be difficult to simply
> add support for llvm-cov and enable (make kconfig selectable) one or
> the other depending on the compiler?

Not strictly necessary, no. And what you are describing is what our
original patches do.

The concern was why we would want to have three separate code coverage
tools in the kernel. Each has their own way of doing things, and they
can all live together in the kernel harmoniously.

But the concern was raised so I am trying to find a way forward.

The llvm-cov approach gives results that are reliably tied to actual
lines of source, so that is the one I reflexively reach for any time I
need code coverage. I am at a loss (but definitely willing to be
educated) as to what value gcov style coverage provides in comparison.

I also found some interesting possibilities using intrinsics to enable
boot time tracing with llvm-cov. That is, of course, experimental and
not something I am considering for the intial patch-set. But it got me
thinking about broader configurability that supports more granular forms
of coverage that _may_ address kcov needs in the long term.

I have been working on this off-and on for a few years and still really
want to see this idea succeed. Any guidance on a workable path forward
would be greatly appreciated.

Another option, proposed by Sasha Levin[1], was to use a single
/sys/kernel/debug/coverage interface, quoting him here:

"To clarify, are you suggesting that we'll have something like a single
 /sys/kernel/debug/coverage interface that is producing the same structured
 output whether we use gcov or llvm?"

I am not sure it is feasible to use the same structured ouput, but it
would isolate things down to a single interface with KConfig knobs being
used to select which coverage data one can expect to find there.


[1] https://lore.kernel.org/lkml/aba_HgSbzLGm6VBQ@laps/


Thank you,

..Ch:W..

Re: [PROPOSAL] Replace gcov and kcov with llvm-cov

[Masami Hiramatsu]

On Wed, 07 Oct 2026 04:17:42 +0000, Chuck Wolber <chuck@wolber.net> wrote:
> On Wed, Oct 7, 2026, at 2:30 AM, Masami Hiramatsu wrote:
> > On Tue, 06 Oct 2026 16:27:11 +0000, Chuck Wolber <chuck@wolber.net> wrote:
> >> On Tue, Oct 6, 2026, at 4:10 PM, Marco Elver wrote:
> >> > On Tue, 6 Oct 2026 at 17:14, Chuck Wolber <chuck@wolber.net> wrote:
> >
> >> >> If there is a desire to take this in pieces or implement other
> >> >> intermediate steps, let me know. Otherwise I can generate a
> >> >> monolithic set all at once.
> >> >>
> >> >> [1] https://lore.kernel.org/lkml/20240905043245.1389509-1-wentaoz5@illinois.edu/
> >> >>
> >> > >> > llvm-cov is generally useful to have; llvm-cov is closest to
> >> > >> > gcov, so that might make sense to replace.
> >>
> >> Swapping llvm-cov for gcov to start with is pretty straightforward,
> >> so I can aim my initial patch set there.
> >
> > Hmm, is it necessary to remove gcov? Would it be difficult to simply
> > add support for llvm-cov and enable (make kconfig selectable) one or
> > the other depending on the compiler?
>
> Not strictly necessary, no. And what you are describing is what our
> original patches do.

OK, let me check it.

>
> The concern was why we would want to have three separate code coverage
> tools in the kernel. Each has their own way of doing things, and they
> can all live together in the kernel harmoniously.
>
> But the concern was raised so I am trying to find a way forward.
>
> The llvm-cov approach gives results that are reliably tied to actual
> lines of source, so that is the one I reflexively reach for any time I
> need code coverage. I am at a loss (but definitely willing to be
> educated) as to what value gcov style coverage provides in comparison.
>

I agree with llvm-cov is better than GCOV. But since gcc users can
NOT use this feature, for checking code coverage in the kernel,
we still need to keep it for gcc users.

> I also found some interesting possibilities using intrinsics to enable
> boot time tracing with llvm-cov. That is, of course, experimental and
> not something I am considering for the intial patch-set. But it got me
> thinking about broader configurability that supports more granular forms
> of coverage that _may_ address kcov needs in the long term.

Ah, that's an interesting idea :)

>
> I have been working on this off-and on for a few years and still really
> want to see this idea succeed. Any guidance on a workable path forward
> would be greatly appreciated.
>
> Another option, proposed by Sasha Levin[1], was to use a single
> /sys/kernel/debug/coverage interface, quoting him here:
>
> "To clarify, are you suggesting that we'll have something like a single
>  /sys/kernel/debug/coverage interface that is producing the same structured
>  output whether we use gcov or llvm?"
>
> I am not sure it is feasible to use the same structured ouput, but it
> would isolate things down to a single interface with KConfig knobs being
> used to select which coverage data one can expect to find there.

Yeah, that soulds reasonable for me.

Thank you,

>
>
> [1] https://lore.kernel.org/lkml/aba_HgSbzLGm6VBQ@laps/
>
>
> Thank you,
>
> ..Ch:W..

-- 
Masami Hiramatsu (Google) <mhiramat@kernel.org>