Possible regression in driver meson-gx-mmc in kernel 5.0.0

Possible regression in driver meson-gx-mmc in kernel 5.0.0

[André Balsa]

Hello, my first post to the list, nice to meet everybody.

I cannot boot kernel 5.0.0 on my S905 TV box (using
meson-gxbb-p201.dtb) from an SD card, see the errors below (captured
from serial output).

If I revert only meson-gx-mmc.c to the version in kernel 4.20.13,
kernel 5.0.0 boots without any problems.

It seems the problems occur because of patches introduced by Jerome
Brunet in December, see
https://lkml.org/lkml/2018/12/6/745

My guess is the removal of the spinlock allows for race conditions
that cause the mmc driver to fail when writing to an SD card.

I am available to test the driver further with any proposed solution.

Regards,
Andrew

...
[    5.004812] EXT4-fs (mmcblk1p2): re-mounted. Opts:
commit=600,errors=remount-ro
[    5.146421] systemd-journald[1490]: Received request to flush
runtime journal from P
ID 1
[    5.384651] meson-gx-mmc d0070000.mmc: Linked as a consumer to regulator.2
[    5.384691] meson-gx-mmc d0070000.mmc: Linked as a consumer to regulator.1
[    5.384774] meson-gx-mmc d0070000.mmc: Dropping the link to regulator.1
[    5.384810] meson-gx-mmc d0070000.mmc: Dropping the link to regulator.2
[    5.400751] meson-gx-mmc d0070000.mmc: Linked as a consumer to regulator.2
[    5.400792] meson-gx-mmc d0070000.mmc: Linked as a consumer to regulator.1
[    5.400838] meson-gx-mmc d0070000.mmc: allocated mmc-pwrseq
[    5.423057] Registered IR keymap rc-empty
[    5.423126] rc rc0: meson-ir as
/devices/platform/soc/c8100000.bus/c8100580.ir/rc/rc
0
[    5.423365] input: meson-ir as
/devices/platform/soc/c8100000.bus/c8100580.ir/rc/rc0
/input0
[    5.423654] rc rc0: lirc_dev: driver meson-ir registered at minor =
0, raw IR receiv
er, no transmitter
[    5.424036] meson-ir c8100580.ir: receiver initialized
[    5.451279] mmc2: new high speed SDIO card at address 0001
[    6.123900] Generic PHY stmmac-0:00: attached PHY driver [Generic
PHY] (mii_bus:phy_
addr=stmmac-0:00, irq=POLL)
[    6.133968] meson8b-dwmac c9410000.ethernet eth0: No Safety
Features support found
[    6.133983] meson8b-dwmac c9410000.ethernet eth0: PTP not supported by HW
[    7.718995] print_req_error: I/O error, dev mmcblk1, sector
27401016 flags 801
[    7.719007] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:323:
I/O error 10 writi
ng to inode 805638 (offset 0 size 0 starting block 3425128)
[    7.719012] Buffer I/O error on device mmcblk1p2, logical block 3293031
[    8.162241] meson8b-dwmac c9410000.ethernet eth0: Link is Up -
100Mbps/Full - flow c
ontrol rx/tx
[    8.177336] print_req_error: I/O error, dev mmcblk1, sector
28635136 flags 801
[    8.177350] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:323:
I/O error 10 writi
ng to inode 805606 (offset 0 size 4096 starting block 3579393)
[    8.177358] Buffer I/O error on device mmcblk1p2, logical block 3447296
[    8.178186] print_req_error: I/O error, dev mmcblk1, sector
28570600 flags 801
[    8.178194] EXT4-fs warning (device mmcblk1p2): ext4_end_bio:323:
I/O error 10 writi
ng to inode 805589 (offset 0 size 0 starting block 3571326)
[    8.178198] Buffer I/O error on device mmcblk1p2, logical block 3439229
[    8.422031] print_req_error: I/O error, dev mmcblk1, sector
13734800 flags 4801
[    8.422045] Buffer I/O error on dev mmcblk1p2, logical block
1584754, lost sync page
...

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: Possible regression in driver meson-gx-mmc in kernel 5.0.0

[Jerome Brunet]

On Mon, 2019-03-04 at 14:46 +0100, André Balsa wrote:
> Hello, my first post to the list, nice to meet everybody.
> 
> I cannot boot kernel 5.0.0 on my S905 TV box (using
> meson-gxbb-p201.dtb) from an SD card, see the errors below (captured
> from serial output).

As I already pointed out to somebody else, it is difficult to give any form of
support for boards we know nothing about, especially the one that are
'conveniently' using the DT of the ref design

> 
> If I revert only meson-gx-mmc.c to the version in kernel 4.20.13,
> kernel 5.0.0 boots without any problems.

You reverted more than one patch here.

> 
> It seems the problems occur because of patches introduced by Jerome
> Brunet in December, see
> https://lkml.org/lkml/2018/12/6/745
> 
> My guess is the removal of the spinlock allows for race conditions
> that cause the mmc driver to fail when writing to an SD card.

1) As pointed out in the related commit message this spinlock was useless.
Anyone looking at the code should be able to see that. What you are seeing is
the result of CRC errors.

2) The last 2 patches of related series had been sent as RFT, but noboby
replied. 

These 2 changes helped several designs work and I could not see any regression
on the designs I had at the time, including the p200

Now we got several reports of regression. 

It is very difficult to find settings that will work for the all the designs.
The documentation is not verbose regarding this. You can't even take the
vendor kernel as reference, the code is extremely 'fancy' and phase settings
are perpetually cycling within the irq handler, which is unacceptable.

We are actively working to solve this and hopefully we will be able to share
something soon

> 
> I am available to test the driver further with any proposed solution.
> 
> Regards,
> Andrew



_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: Possible regression in driver meson-gx-mmc in kernel 5.0.0

[André Balsa]

Hi Jerome,
Thanks for the answer.

> You reverted more than one patch here."

Yes, I didn't know which patch exactly was causing the problem.
If you want me to test the driver with the spinlock removal patch(es)
applied but without the delay / phase setting patch(es), just say.

Regards,
Andrew

On Mon, Mar 4, 2019 at 4:34 PM Jerome Brunet <jbrunet@baylibre.com> wrote:
>
> On Mon, 2019-03-04 at 14:46 +0100, André Balsa wrote:
> > Hello, my first post to the list, nice to meet everybody.
> >
> > I cannot boot kernel 5.0.0 on my S905 TV box (using
> > meson-gxbb-p201.dtb) from an SD card, see the errors below (captured
> > from serial output).
>
> As I already pointed out to somebody else, it is difficult to give any form of
> support for boards we know nothing about, especially the one that are
> 'conveniently' using the DT of the ref design
>
> >
> > If I revert only meson-gx-mmc.c to the version in kernel 4.20.13,
> > kernel 5.0.0 boots without any problems.
>
> You reverted more than one patch here.
>
> >
> > It seems the problems occur because of patches introduced by Jerome
> > Brunet in December, see
> > https://lkml.org/lkml/2018/12/6/745
> >
> > My guess is the removal of the spinlock allows for race conditions
> > that cause the mmc driver to fail when writing to an SD card.
>
> 1) As pointed out in the related commit message this spinlock was useless.
> Anyone looking at the code should be able to see that. What you are seeing is
> the result of CRC errors.
>
> 2) The last 2 patches of related series had been sent as RFT, but noboby
> replied.
>
> These 2 changes helped several designs work and I could not see any regression
> on the designs I had at the time, including the p200
>
> Now we got several reports of regression.
>
> It is very difficult to find settings that will work for the all the designs.
> The documentation is not verbose regarding this. You can't even take the
> vendor kernel as reference, the code is extremely 'fancy' and phase settings
> are perpetually cycling within the irq handler, which is unacceptable.
>
> We are actively working to solve this and hopefully we will be able to share
> something soon
>
> >
> > I am available to test the driver further with any proposed solution.
> >
> > Regards,
> > Andrew
>
>

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic

Re: Possible regression in driver meson-gx-mmc in kernel 5.0.0

[André Balsa]

Hello, Jerome, list,

After more testing, I managed to trace the exact patch that was
causing the SD card write errors that I saw with kernel 5.0.0 and was
not seeing with kernel 4.20.13 and earlier:
it is very exactly [RFT PATCH 3/4] mmc: meson-gx: align default phase
on soc vendor tree
 from https://lkml.org/lkml/2018/12/6/746

The patch actually just changes a single value in a single line of code:

- clk_set_phase(host->tx_clk, 270);
+ clk_set_phase(host->tx_clk, 0);

The patch specifically states that the value of zero does not match
the recommended value in the (vendor) documentation, but sets the same
value as in the vendor tree. So it seems there is a mismatch between
the vendor tree and the vendor documentation (I am guessing that is
not an uncommon situation).

If you could kindly point me to the documentation and the vendor tree
repository, I would be able to perhaps bring some more clarifying
information for this issue?

On Mon, Mar 4, 2019 at 5:55 PM André Balsa <andrebalsa@gmail.com> wrote:
>
> Hi Jerome,
> Thanks for the answer.
>
> > You reverted more than one patch here."
>
> Yes, I didn't know which patch exactly was causing the problem.
> If you want me to test the driver with the spinlock removal patch(es)
> applied but without the delay / phase setting patch(es), just say.
>
> Regards,
> Andrew
>
> On Mon, Mar 4, 2019 at 4:34 PM Jerome Brunet <jbrunet@baylibre.com> wrote:
> >
> > On Mon, 2019-03-04 at 14:46 +0100, André Balsa wrote:
> > > Hello, my first post to the list, nice to meet everybody.
> > >
> > > I cannot boot kernel 5.0.0 on my S905 TV box (using
> > > meson-gxbb-p201.dtb) from an SD card, see the errors below (captured
> > > from serial output).
> >
> > As I already pointed out to somebody else, it is difficult to give any form of
> > support for boards we know nothing about, especially the one that are
> > 'conveniently' using the DT of the ref design
> >
> > >
> > > If I revert only meson-gx-mmc.c to the version in kernel 4.20.13,
> > > kernel 5.0.0 boots without any problems.
> >
> > You reverted more than one patch here.
> >
> > >
> > > It seems the problems occur because of patches introduced by Jerome
> > > Brunet in December, see
> > > https://lkml.org/lkml/2018/12/6/745
> > >
> > > My guess is the removal of the spinlock allows for race conditions
> > > that cause the mmc driver to fail when writing to an SD card.
> >
> > 1) As pointed out in the related commit message this spinlock was useless.
> > Anyone looking at the code should be able to see that. What you are seeing is
> > the result of CRC errors.
> >
> > 2) The last 2 patches of related series had been sent as RFT, but noboby
> > replied.
> >
> > These 2 changes helped several designs work and I could not see any regression
> > on the designs I had at the time, including the p200
> >
> > Now we got several reports of regression.
> >
> > It is very difficult to find settings that will work for the all the designs.
> > The documentation is not verbose regarding this. You can't even take the
> > vendor kernel as reference, the code is extremely 'fancy' and phase settings
> > are perpetually cycling within the irq handler, which is unacceptable.
> >
> > We are actively working to solve this and hopefully we will be able to share
> > something soon
> >
> > >
> > > I am available to test the driver further with any proposed solution.
> > >
> > > Regards,
> > > Andrew
> >
> >

_______________________________________________
linux-amlogic mailing list
linux-amlogic@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-amlogic