Re: [PATCH v1] Documentation/process: add soc maintainer handbook

Re: [PATCH v1] Documentation/process: add soc maintainer handbook

[Krzysztof Kozlowski]

On 15/05/2023 21:20, Conor Dooley wrote:
> From: Conor Dooley <conor.dooley@microchip.com>
> 
> Arnd suggested that adding maintainer handbook for the SoC "subsystem"
> would be helpful in trying to bring on board maintainers for the various
> new platforms cropping up in RISC-V land.
> 
> Add a document briefly describing the role of the SoC subsystem and some
> basic advice for (new) platform maintainers.
> 
> Suggested-by: Arnd Bergmann <arnd@arndb.de>
> Signed-off-by: Conor Dooley <conor.dooley@microchip.com>



> +
> +What the SoC tree is not, however, is a location for architecture specific code
> +changes.  Each architecture has it's own maintainers that are responsible for
> +architectural details, cpu errata and the like.
> +
> +Information for (new) Sub-maintainers
> +-------------------------------------
> +
> +As new platforms spring up, they often bring with them new submaintainers,
> +many of whom work for the silicon vendor, and may not be familiar with the
> +process.
> +
> +devicetree ABI stability
> +~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Perhaps one of the most important things to highlight is that dt-bindings
> +document the ABI between the devicetree and the kernel.  Once dt-bindings have
> +been merged (and appear in a release of the kernel) they are set in stone, and
> +any changes made must be compatible with existing devicetrees.  This means that,
> +when changing properties, a "new" kernel must still be able to handle an old
> +devicetree.  For many systems the devicetree is provided by firmware, and
> +upgrading to a newer kernel cannot cause regressions.  Ideally, the inverse is
> +also true, and a new devicetree will also be compatible with an old kernel,
> +although this is often not possible.

I would prefer to skip it and instead: enhance
Documentation/devicetree/bindings/ABI.rst and then reference it here.

> +
> +If changes are being made to a devicetree that are incompatible with old
> +kernels, the devicetree patch should not be applied until the driver is, or an
> +appropriate time later.  Most importantly, any incompatible changes should be
> +clearly pointed out in the patch description and pull request, along with the
> +expected impact on existing users.


> +
> +Driver branch dependencies
> +~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +A common problem is synchronizing changes between device drivers and devicetree
> +files, even if a change is compatible in both directions, this may require
> +coordinating how the changes get merged through different maintainer trees.
> +
> +Usually the branch that includes a driver change will also include the
> +corresponding change to the devicetree binding description, to ensure they are
> +in fact compatible.  This means that the devicetree branch can end up causing
> +warnings in the "make dtbs_check" step.  If a devicetree change depends on
> +missing additions to a header file in include/dt-bindings/, it will fail the
> +"make dtbs" step and not get merged.
> +
> +There are multiple ways to deal with this:
> +
> + - Avoid defining custom macros in include/dt-bindings/ for hardware constants
> +   that can be derived from a datasheet -- binding macros in header file should
> +   only be used as a last resort if there is no natural way to define a binding
> +
> + - Use literal values in the devicetree file in place of macros even when a
> +   header is required, and change them to the named representation in a
> +   following release

I actually prefer such solution:

 - Duplicate defines in the devicetree file hidden by #ifndef section
and remove them later in a following release

We can keep both, but mine above leads to cleaner changes in DTS file.

> +
> + - Defer the devicetree changes to a release after the binding and driver have
> +   already been merged
> +
> + - Change the bindings in a shared immutable branch that is used as the base for
> +   both the driver change and the devicetree changes

The policy told to me some time ago was that no merges from driver
branch or tree are allowed towards DTS branch, even if they come only
with binding header change. There are exceptions for this, e.g. [1], but
that would mean we need to express here rules for cross-tree merges.

[1]
https://lore.kernel.org/linux-samsung-soc/cced7901-a855-c733-e716-f4a7f822b213@canonical.com/T/#m6430408367e942ac28d01e447be84fcf34407743

> +
> +devicetree naming convention
> +~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +The general naming scheme for devicetree files are as follows.  The aspects of a
> +platform that are set at the SoC level, like cpu cores, are contained in a file
> +named $soc.dtsi, for example, jh7100.dtsi.  Integration details, that will vary
> +from board to board, are described in $soc-$board.dtsi.  An example of this is
> +jh7100-beaglev-starlight.dts.  Often many boards are variations on a theme, and
> +frequently there are intermediate files, such as jh7100-common.dtsi, which sit
> +between the $soc.dtsi and $soc-$board.dts files, containing the descriptions of
> +common hardware.
> +
> +Some platforms also have System on Modules, containing an SoC, which are then
> +integrated into several different boards. For these platforms, $soc-$som.dtsi
> +and $soc-$som-$board.dts are typical.
> +
> +Directories are usually named after the vendor of the SoC at the time of it's
> +inclusion, leading to some historical directory names in the tree.
> +
> +dtbs_check
> +~~~~~~~~~~
> +
> +``make dtbs_check`` can be used to validate that devicetree files are compliant
> +with the dt-bindings that describe the ABI.  Please see :ref:`running-checks`
> +for more information on the validation of devicetrees.
> +
> +For new platforms, or additions to existing ones, ``make dtbs_check`` should not
> +add any new warnings.  For RISC-V, as it has the advantage of being a newer
> +architecture, ``make dtbs_check W=1`` is required to not add any new warnings.
> +If in any doubt about a devicetree change, reach out to the devicetree
> +maintainers.
> +
> +
> +Branches and Pull Requests
> +~~~~~~~~~~~~~~~~~~~~~~~~~~
> +
> +Just as the main SoC tree has several branches, it is expected that
> +submaintainers will do the same. Driver, defconfig and devicetree changes should
> +all be split into separate branches and appear in separate pull requests to the
> +SoC maintainers.  Each branch should be usable by itself and avoid
> +regressions that originate from dependencies on other branches.
> +
> +Small sets of patches can also be sent as separate emails to soc@kernel.org,
> +grouped into the same categories.
> +
> +If changes do not fit into the normal patterns, there can be additional
> +top-level branches, e.g. for a treewide rework, or the addition of new SoC
> +platforms including dts files and drivers.
> +
> +Branches with a lot of changes can benefit from getting split up into separate
> +topics branches, even if they end up getting merged into the same branch of the
> +SoC tree.  An example here would be one branch for devicetree warning fixes, one
> +for a rework and one for newly added boards.
> +
> +Another common way to split up changes is to send an early pull request with the
> +majority of the changes at some point between rc1 and rc4, following up with one
> +or more smaller pull requests towards the end of the cycle that can add late
> +changes or address problems idenfied while testing the first set.
> +
> +While there is no cut-off time for late pull requests, it helps to only send
> +small branches as time gets closer to the merge window.
> +
> +Pull requests for bugfixes for the current release can be sent at any time, but
> +again having multiple smaller branches is better than trying to combine too many
> +patches into one pull request.
> +
> +The subject line of a pull request should begin with "[GIT PULL]" and made using
> +a tag, rather than a branch.  This tag should contain a short description

a signed tag

>  

Best regards,
Krzysztof

Re: [PATCH v1] Documentation/process: add soc maintainer handbook

[Conor Dooley]

[-- Attachment #1: Type: text/plain, Size: 4476 bytes --]

On Tue, May 16, 2023 at 10:31:19AM +0200, Krzysztof Kozlowski wrote:
> On 15/05/2023 21:20, Conor Dooley wrote:
> > From: Conor Dooley <conor.dooley@microchip.com>
> > 
> > Arnd suggested that adding maintainer handbook for the SoC "subsystem"
> > would be helpful in trying to bring on board maintainers for the various
> > new platforms cropping up in RISC-V land.
> > 
> > Add a document briefly describing the role of the SoC subsystem and some
> > basic advice for (new) platform maintainers.
> > 
> > Suggested-by: Arnd Bergmann <arnd@arndb.de>
> > Signed-off-by: Conor Dooley <conor.dooley@microchip.com>

> > +devicetree ABI stability
> > +~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +Perhaps one of the most important things to highlight is that dt-bindings
> > +document the ABI between the devicetree and the kernel.  Once dt-bindings have
> > +been merged (and appear in a release of the kernel) they are set in stone, and
> > +any changes made must be compatible with existing devicetrees.  This means that,
> > +when changing properties, a "new" kernel must still be able to handle an old
> > +devicetree.  For many systems the devicetree is provided by firmware, and
> > +upgrading to a newer kernel cannot cause regressions.  Ideally, the inverse is
> > +also true, and a new devicetree will also be compatible with an old kernel,
> > +although this is often not possible.
> 
> I would prefer to skip it and instead: enhance
> Documentation/devicetree/bindings/ABI.rst and then reference it here.

Sure.

> > +Driver branch dependencies
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~
> > +
> > +A common problem is synchronizing changes between device drivers and devicetree
> > +files, even if a change is compatible in both directions, this may require
> > +coordinating how the changes get merged through different maintainer trees.
> > +
> > +Usually the branch that includes a driver change will also include the
> > +corresponding change to the devicetree binding description, to ensure they are
> > +in fact compatible.  This means that the devicetree branch can end up causing
> > +warnings in the "make dtbs_check" step.  If a devicetree change depends on
> > +missing additions to a header file in include/dt-bindings/, it will fail the
> > +"make dtbs" step and not get merged.
> > +
> > +There are multiple ways to deal with this:
> > +
> > + - Avoid defining custom macros in include/dt-bindings/ for hardware constants
> > +   that can be derived from a datasheet -- binding macros in header file should
> > +   only be used as a last resort if there is no natural way to define a binding
> > +
> > + - Use literal values in the devicetree file in place of macros even when a
> > +   header is required, and change them to the named representation in a
> > +   following release
> 
> I actually prefer such solution:
> 
>  - Duplicate defines in the devicetree file hidden by #ifndef section
> and remove them later in a following release
> 
> We can keep both, but mine above leads to cleaner changes in DTS file.

I think all of the options involved are either a bit ugly, or a bit of a
pain in the hole.

> > + - Defer the devicetree changes to a release after the binding and driver have
> > +   already been merged
> > +
> > + - Change the bindings in a shared immutable branch that is used as the base for
> > +   both the driver change and the devicetree changes
> 
> The policy told to me some time ago was that no merges from driver
> branch or tree are allowed towards DTS branch, even if they come only
> with binding header change. There are exceptions for this, e.g. [1], but
> that would mean we need to express here rules for cross-tree merges.

I've got away with having an immutable branch for dt-binding headers!
That said, Arnd did actually have a look at this (and suggested some
changes) before I sent it & did not cry fowl about this section. IIRC,
this is actually his wording, not mine.

> > +Branches and Pull Requests
> > +~~~~~~~~~~~~~~~~~~~~~~~~~~

> > +The subject line of a pull request should begin with "[GIT PULL]" and made using
> > +a tag, rather than a branch.  This tag should contain a short description
> 
> a signed tag

I initially had that explicit wording, but I dropped it when I added the
ref to the pull requests doc since that talks about signing. It's
probably better to be explicit & re-adding it is trivial.

Thanks,
Conor.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v1] Documentation/process: add soc maintainer handbook

[Krzysztof Kozlowski]

On 16/05/2023 10:57, Conor Dooley wrote:
> On Tue, May 16, 2023 at 10:31:19AM +0200, Krzysztof Kozlowski wrote:
>> On 15/05/2023 21:20, Conor Dooley wrote:
>>> From: Conor Dooley <conor.dooley@microchip.com>
>>>
>>> Arnd suggested that adding maintainer handbook for the SoC "subsystem"
>>> would be helpful in trying to bring on board maintainers for the various
>>> new platforms cropping up in RISC-V land.
>>>
>>> Add a document briefly describing the role of the SoC subsystem and some
>>> basic advice for (new) platform maintainers.
>>>
>>> Suggested-by: Arnd Bergmann <arnd@arndb.de>
>>> Signed-off-by: Conor Dooley <conor.dooley@microchip.com>
> 
>>> +devicetree ABI stability
>>> +~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +Perhaps one of the most important things to highlight is that dt-bindings
>>> +document the ABI between the devicetree and the kernel.  Once dt-bindings have
>>> +been merged (and appear in a release of the kernel) they are set in stone, and
>>> +any changes made must be compatible with existing devicetrees.  This means that,
>>> +when changing properties, a "new" kernel must still be able to handle an old
>>> +devicetree.  For many systems the devicetree is provided by firmware, and
>>> +upgrading to a newer kernel cannot cause regressions.  Ideally, the inverse is
>>> +also true, and a new devicetree will also be compatible with an old kernel,
>>> +although this is often not possible.
>>
>> I would prefer to skip it and instead: enhance
>> Documentation/devicetree/bindings/ABI.rst and then reference it here.
> 
> Sure.
> 
>>> +Driver branch dependencies
>>> +~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +A common problem is synchronizing changes between device drivers and devicetree
>>> +files, even if a change is compatible in both directions, this may require
>>> +coordinating how the changes get merged through different maintainer trees.
>>> +
>>> +Usually the branch that includes a driver change will also include the
>>> +corresponding change to the devicetree binding description, to ensure they are
>>> +in fact compatible.  This means that the devicetree branch can end up causing
>>> +warnings in the "make dtbs_check" step.  If a devicetree change depends on
>>> +missing additions to a header file in include/dt-bindings/, it will fail the
>>> +"make dtbs" step and not get merged.
>>> +
>>> +There are multiple ways to deal with this:
>>> +
>>> + - Avoid defining custom macros in include/dt-bindings/ for hardware constants
>>> +   that can be derived from a datasheet -- binding macros in header file should
>>> +   only be used as a last resort if there is no natural way to define a binding
>>> +
>>> + - Use literal values in the devicetree file in place of macros even when a
>>> +   header is required, and change them to the named representation in a
>>> +   following release
>>
>> I actually prefer such solution:
>>
>>  - Duplicate defines in the devicetree file hidden by #ifndef section
>> and remove them later in a following release
>>
>> We can keep both, but mine above leads to cleaner changes in DTS file.
> 
> I think all of the options involved are either a bit ugly, or a bit of a
> pain in the hole.
> 
>>> + - Defer the devicetree changes to a release after the binding and driver have
>>> +   already been merged
>>> +
>>> + - Change the bindings in a shared immutable branch that is used as the base for
>>> +   both the driver change and the devicetree changes
>>
>> The policy told to me some time ago was that no merges from driver
>> branch or tree are allowed towards DTS branch, even if they come only
>> with binding header change. There are exceptions for this, e.g. [1], but
>> that would mean we need to express here rules for cross-tree merges.
> 
> I've got away with having an immutable branch for dt-binding headers!

Of course, all is in an immutable branch, but in which tree?

I talk about a case when driver tree, e.g. different clock maintainer,
takes the binding.

> That said, Arnd did actually have a look at this (and suggested some
> changes) before I sent it & did not cry fowl about this section. IIRC,
> this is actually his wording, not mine.


Best regards,
Krzysztof

Re: [PATCH v1] Documentation/process: add soc maintainer handbook

[Conor Dooley]

[-- Attachment #1: Type: text/plain, Size: 1837 bytes --]

On Tue, May 16, 2023 at 11:06:41AM +0200, Krzysztof Kozlowski wrote:
> On 16/05/2023 10:57, Conor Dooley wrote:
> > On Tue, May 16, 2023 at 10:31:19AM +0200, Krzysztof Kozlowski wrote:
> >> On 15/05/2023 21:20, Conor Dooley wrote:

> >>> + - Defer the devicetree changes to a release after the binding and driver have
> >>> +   already been merged
> >>> +
> >>> + - Change the bindings in a shared immutable branch that is used as the base for
> >>> +   both the driver change and the devicetree changes
> >>
> >> The policy told to me some time ago was that no merges from driver
> >> branch or tree are allowed towards DTS branch, even if they come only
> >> with binding header change. There are exceptions for this, e.g. [1], but
> >> that would mean we need to express here rules for cross-tree merges.
> > 
> > I've got away with having an immutable branch for dt-binding headers!
> 
> Of course, all is in an immutable branch, but in which tree?

For example:
- dt-bindings & header with the clock defines in the base/immutable branch
  on top of -rc1
- driver patches on top of the immutable branch, in a PR to Stephen
- dts patches on top of the immutable branch, PR to Arnd

So, clock tree doesn't get the dts, soc tree doesn't get the driver.
Hopefully that clarifies what I meant.

> I talk about a case when driver tree, e.g. different clock maintainer,
> takes the binding.

If the other tree just "takes the binding", without some coordination,
then you're SOOL and have to wait a release.

> > That said, Arnd did actually have a look at this (and suggested some
> > changes) before I sent it & did not cry fowl about this section. IIRC,
> > this is actually his wording, not mine.

Probably worth Arnd chiming in & just telling us what he is okay with
taking.

Cheers,
Conor.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 228 bytes --]

Re: [PATCH v1] Documentation/process: add soc maintainer handbook

[Arnd Bergmann]

On Tue, May 16, 2023, at 10:57, Conor Dooley wrote:
> On Tue, May 16, 2023 at 10:31:19AM +0200, Krzysztof Kozlowski wrote:
>> On 15/05/2023 21:20, Conor Dooley wrote:
>
>> > + - Defer the devicetree changes to a release after the binding and driver have
>> > +   already been merged
>> > +
>> > + - Change the bindings in a shared immutable branch that is used as the base for
>> > +   both the driver change and the devicetree changes
>> 
>> The policy told to me some time ago was that no merges from driver
>> branch or tree are allowed towards DTS branch, even if they come only
>> with binding header change. There are exceptions for this, e.g. [1], but
>> that would mean we need to express here rules for cross-tree merges.
>
> I've got away with having an immutable branch for dt-binding headers!
> That said, Arnd did actually have a look at this (and suggested some
> changes) before I sent it & did not cry fowl about this section. IIRC,
> this is actually his wording, not mine.

Yes, I merge a lot of shared branches with dt-binding changes into the
soc/dt branch, and I wasn't aware of a policy against that, certainly did
not enforce it.

I generally object to changes touching drivers/* or anything else besides
arch/*/boot/dts, Documentation/devicetree/bindings/ and include/dt-bindings
in the dt branches, but I have made expections in the past when there
was a particular important reason.

    Arnd

Re: [PATCH v1] Documentation/process: add soc maintainer handbook

[Krzysztof Kozlowski]

On 16/05/2023 11:16, Arnd Bergmann wrote:
> On Tue, May 16, 2023, at 10:57, Conor Dooley wrote:
>> On Tue, May 16, 2023 at 10:31:19AM +0200, Krzysztof Kozlowski wrote:
>>> On 15/05/2023 21:20, Conor Dooley wrote:
>>
>>>> + - Defer the devicetree changes to a release after the binding and driver have
>>>> +   already been merged
>>>> +
>>>> + - Change the bindings in a shared immutable branch that is used as the base for
>>>> +   both the driver change and the devicetree changes
>>>
>>> The policy told to me some time ago was that no merges from driver
>>> branch or tree are allowed towards DTS branch, even if they come only
>>> with binding header change. There are exceptions for this, e.g. [1], but
>>> that would mean we need to express here rules for cross-tree merges.
>>
>> I've got away with having an immutable branch for dt-binding headers!
>> That said, Arnd did actually have a look at this (and suggested some
>> changes) before I sent it & did not cry fowl about this section. IIRC,
>> this is actually his wording, not mine.
> 
> Yes, I merge a lot of shared branches with dt-binding changes into the
> soc/dt branch, and I wasn't aware of a policy against that, certainly did
> not enforce it.

Not as enforcement, but as your (or Olof's) preferred approach. See for
example:

https://lore.kernel.org/all/CAOesGMhfrWSvLtDtGRWBTJiAoeSwzGgsdUTm26j1mpoVu0ghDg@mail.gmail.com/

https://lore.kernel.org/all/CAOesGMi98hJnUYVLkgcbBpXsi-Xe6QPh-gtLaPWPO-EW+KcGuQ@mail.gmail.com/

I understand the "preferred approach" as part of discussions on the
patches, when the entire patchset can still be re-arranged or changed.
Not as strict policy applied on the actual git pull request from
sub-arch maintainer.

If we want to keep such recommendation, let's embed it into maintainer
handbook, so we will not have to dig the emails.

> 
> I generally object to changes touching drivers/* or anything else besides
> arch/*/boot/dts, Documentation/devicetree/bindings/ and include/dt-bindings
> in the dt branches, but I have made expections in the past when there
> was a particular important reason.

Best regards,
Krzysztof

Re: [PATCH v1] Documentation/process: add soc maintainer handbook

[Jessica Clarke]

On 22 May 2023, at 22:34, Conor Dooley <conor@kernel.org> wrote:
> 
> On Tue, May 16, 2023 at 10:31:19AM +0200, Krzysztof Kozlowski wrote:
>> On 15/05/2023 21:20, Conor Dooley wrote:
> 
>>> +devicetree ABI stability
>>> +~~~~~~~~~~~~~~~~~~~~~~~~
>>> +
>>> +Perhaps one of the most important things to highlight is that dt-bindings
>>> +document the ABI between the devicetree and the kernel.  Once dt-bindings have
>>> +been merged (and appear in a release of the kernel) they are set in stone, and
>>> +any changes made must be compatible with existing devicetrees.  This means that,
>>> +when changing properties, a "new" kernel must still be able to handle an old
>>> +devicetree.  For many systems the devicetree is provided by firmware, and
>>> +upgrading to a newer kernel cannot cause regressions.  Ideally, the inverse is
>>> +also true, and a new devicetree will also be compatible with an old kernel,
>>> +although this is often not possible.
>> 
>> I would prefer to skip it and instead: enhance
>> Documentation/devicetree/bindings/ABI.rst and then reference it here.
>> 
>>> +
>>> +If changes are being made to a devicetree that are incompatible with old
>>> +kernels, the devicetree patch should not be applied until the driver is, or an
>>> +appropriate time later.  Most importantly, any incompatible changes should be
>>> +clearly pointed out in the patch description and pull request, along with the
>>> +expected impact on existing users.
> 
> I'm not really sure that I like this truncated section so much, but here
> it is... I kept the last paragraph intact as it does not talk about the
> ABI, but rather exceptions of submaintainers.
> 
> devicetree ABI stability
> ~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Perhaps one of the most important things to highlight is that dt-bindings
> document the ABI between the devicetree and the kernel. Please see
> :ref:`devicetree-abi` for devicetree ABI rules.
> 
> If changes are being made to a devicetree that are incompatible with old
> kernels, the devicetree patch should not be applied until the driver is, or an
> appropriate time later.  Most importantly, any incompatible changes should be
> clearly pointed out in the patch description and pull request, along with the
> expected impact on existing users.

Do you have an opinion on acknowledging the existence of other OSes here?

Jess