Moving ndctl development into the kernel tree?

Moving ndctl development into the kernel tree?

[Dan Williams]

Hi Linus,

Would you be open to the ndctl [1] project moving its development into
the kernel tree? The main reasons why I ask are:

* Unit test development can touch both the kernel-side emulated nvdimm
infrastructure in tools/testing/nvdimm/ and the corresponding tests in
tools/ndctl/test/ in the same commit or patch series.

* Like perf, ndctl borrows the sub-command architecture and option
parsing from git. So, this code could be refactored into something
shared / generic, i.e. the bits in tools/perf/util/.

We continue to see updates in the ACPI and UEFI specification for
nvdimm details and one of the capabilities added in ACPI 6.2 that
needs new test development is error injection. I'm also expecting to
merge patches from Oliver this cycle expanding nvdimm support to Open
Firmware / powerpc platforms.

The ndctl project includes GPLv2 utilities (ndctl and daxctl) as well
as LGPLv2.1 libraries (libndctl and libdaxctl).

The coupling of the tests to new libnvdimm sub-system capabilities,
and the architecture specific nature of some nvdimm enabling leads me
to believe ndctl would enjoy some synergies living in the same
repository as the kernel.

[1]: https://github.com/pmem/ndctl

Re: Moving ndctl development into the kernel tree?

[Ingo Molnar]

* Dan Williams <dan.j.williams@intel.com> wrote:

> [...]
> 
> * Like perf, ndctl borrows the sub-command architecture and option
> parsing from git. So, this code could be refactored into something
> shared / generic, i.e. the bits in tools/perf/util/.

Just as a side note, stacktool (tools/stacktool/) is using the Git sub-command and 
options parsing code as well, and it's already sharing it with perf, via the 
tools/lib/subcmd/ library.

ndctl could use that as well.

Thanks,

	Ingo

Re: Moving ndctl development into the kernel tree?

[Dan Williams]

On Fri, Jul 21, 2017 at 3:58 PM, Ingo Molnar <mingo@kernel.org> wrote:
>
> * Dan Williams <dan.j.williams@intel.com> wrote:
>
>> [...]
>>
>> * Like perf, ndctl borrows the sub-command architecture and option
>> parsing from git. So, this code could be refactored into something
>> shared / generic, i.e. the bits in tools/perf/util/.
>
> Just as a side note, stacktool (tools/stacktool/) is using the Git sub-command and
> options parsing code as well, and it's already sharing it with perf, via the
> tools/lib/subcmd/ library.
>
> ndctl could use that as well.

Ah, nice, that refactoring happened about a year after ndctl was born.
Which brings up the next question about what to do with the git
history, but I'd want to know if ndctl is even welcome upstream before
digging any deeper.

Re: Moving ndctl development into the kernel tree?

[Dan Williams]

[ adding Chris ]

On Fri, Jul 21, 2017 at 4:44 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> On Fri, Jul 21, 2017 at 3:58 PM, Ingo Molnar <mingo@kernel.org> wrote:
>>
>> * Dan Williams <dan.j.williams@intel.com> wrote:
>>
>>> [...]
>>>
>>> * Like perf, ndctl borrows the sub-command architecture and option
>>> parsing from git. So, this code could be refactored into something
>>> shared / generic, i.e. the bits in tools/perf/util/.
>>
>> Just as a side note, stacktool (tools/stacktool/) is using the Git sub-command and
>> options parsing code as well, and it's already sharing it with perf, via the
>> tools/lib/subcmd/ library.
>>
>> ndctl could use that as well.
>
> Ah, nice, that refactoring happened about a year after ndctl was born.
> Which brings up the next question about what to do with the git
> history, but I'd want to know if ndctl is even welcome upstream before
> digging any deeper.

I suspect this would be similar to what Chris did to merge btrfs while
retaining the standalone history. Chris, any pointers on what worked
well and what if anything you would do differently? I.e. I'm looking
to use git filter-branch to rewrite ndctl history as if if had always
been in tools/ndctl in the kernel tree. I found this old thread
https://lkml.org/lkml/2008/10/30/523 and it seems to also recommend
using an older kernel as the branch base.

Re: Moving ndctl development into the kernel tree?

[Dan Williams]

On Fri, Jul 21, 2017 at 7:52 PM, Dan Williams <dan.j.williams@intel.com> wrote:
> [ adding Chris ]
>
> On Fri, Jul 21, 2017 at 4:44 PM, Dan Williams <dan.j.williams@intel.com> wrote:
>> On Fri, Jul 21, 2017 at 3:58 PM, Ingo Molnar <mingo@kernel.org> wrote:
>>>
>>> * Dan Williams <dan.j.williams@intel.com> wrote:
>>>
>>>> [...]
>>>>
>>>> * Like perf, ndctl borrows the sub-command architecture and option
>>>> parsing from git. So, this code could be refactored into something
>>>> shared / generic, i.e. the bits in tools/perf/util/.
>>>
>>> Just as a side note, stacktool (tools/stacktool/) is using the Git sub-command and
>>> options parsing code as well, and it's already sharing it with perf, via the
>>> tools/lib/subcmd/ library.
>>>
>>> ndctl could use that as well.
>>
>> Ah, nice, that refactoring happened about a year after ndctl was born.
>> Which brings up the next question about what to do with the git
>> history, but I'd want to know if ndctl is even welcome upstream before
>> digging any deeper.
>
> I suspect this would be similar to what Chris did to merge btrfs while
> retaining the standalone history. Chris, any pointers on what worked
> well and what if anything you would do differently? I.e. I'm looking
> to use git filter-branch to rewrite ndctl history as if if had always
> been in tools/ndctl in the kernel tree. I found this old thread
> https://lkml.org/lkml/2008/10/30/523 and it seems to also recommend
> using an older kernel as the branch base.

So it wasn't as painful as I thought it would be, I just used the
script Linus recommended in that thread. Here is what I came up with
merging the last ndctl release on top of v4.9, and then applying the
pending development patches re-filtered to tools/ndctl:

    https://git.kernel.org/pub/scm/linux/kernel/git/djbw/nvdimm.git/log/?h=for-4.14/ndctl

...the next thing would be to rework the versioning to use the kernel
version and switch to using tools/lib/subcmd/.

Re: Moving ndctl development into the kernel tree?

[Chris Mason]

On 07/22/2017 02:49 PM, Dan Williams wrote:
> On Fri, Jul 21, 2017 at 7:52 PM, Dan Williams <dan.j.williams@intel.com> wrote:
>> [ adding Chris ]
>>
>> On Fri, Jul 21, 2017 at 4:44 PM, Dan Williams <dan.j.williams@intel.com> wrote:
>>> On Fri, Jul 21, 2017 at 3:58 PM, Ingo Molnar <mingo@kernel.org> wrote:
>>>>
>>>> * Dan Williams <dan.j.williams@intel.com> wrote:
>>>>
>>>>> [...]
>>>>>
>>>>> * Like perf, ndctl borrows the sub-command architecture and option
>>>>> parsing from git. So, this code could be refactored into something
>>>>> shared / generic, i.e. the bits in tools/perf/util/.
>>>>
>>>> Just as a side note, stacktool (tools/stacktool/) is using the Git sub-command and
>>>> options parsing code as well, and it's already sharing it with perf, via the
>>>> tools/lib/subcmd/ library.
>>>>
>>>> ndctl could use that as well.
>>>
>>> Ah, nice, that refactoring happened about a year after ndctl was born.
>>> Which brings up the next question about what to do with the git
>>> history, but I'd want to know if ndctl is even welcome upstream before
>>> digging any deeper.
>>
>> I suspect this would be similar to what Chris did to merge btrfs while
>> retaining the standalone history. Chris, any pointers on what worked
>> well and what if anything you would do differently? I.e. I'm looking
>> to use git filter-branch to rewrite ndctl history as if if had always
>> been in tools/ndctl in the kernel tree. I found this old thread
>> https://lkml.org/lkml/2008/10/30/523 and it seems to also recommend
>> using an older kernel as the branch base.
> 
> So it wasn't as painful as I thought it would be, I just used the
> script Linus recommended in that thread. Here is what I came up with
> merging the last ndctl release on top of v4.9, and then applying the
> pending development patches re-filtered to tools/ndctl:
> 
>      https://git.kernel.org/pub/scm/linux/kernel/git/djbw/nvdimm.git/log/?h=for-4.14/ndctl
> 
> ...the next thing would be to rework the versioning to use the kernel
> version and switch to using tools/lib/subcmd/.
> 

I'd like to say I figured it all out back then, but the truth is that 
Linus held my hand the whole way.  My memory of it is that his script 
worked really well, I just ran that and verified the results.

-chris