MMC/SD Root filesystem suspend/resume problems

MMC/SD Root filesystem suspend/resume problems

[Richard Purdie]

Hi Pierre,

Lots of Linux handhelds use MMC/SD devices as the root file system. This
has worked quite reliably for many kernel versions. In 2.6.22, it seems
that if you suspend such a system then resume it, the device locks up.
Trying to execute anything on the filesystem results in a "Permission
Denied" message. I did see a message from the MMC subsystem saying it
had redetected the card. There are also messages on the console like
"MMC: killing requests for dead queue" each time you suspend/resume.

I'm away from my serial cables at the moment but I may be able to
provide more debug when I have them over the weekend. Have you any ideas
on why this is breaking?

For reference, I've reproduced the problem with both the PXA host driver
and different driver not merged into mainline (ASIC3).

Cheers,

Richard

Re: MMC/SD Root filesystem suspend/resume problems

[Richard Purdie]

On Thu, 2007-07-19 at 16:57 +0100, Richard Purdie wrote:
> Lots of Linux handhelds use MMC/SD devices as the root file system. This
> has worked quite reliably for many kernel versions. In 2.6.22, it seems
> that if you suspend such a system then resume it, the device locks up.
> Trying to execute anything on the filesystem results in a "Permission
> Denied" message. I did see a message from the MMC subsystem saying it
> had redetected the card. There are also messages on the console like
> "MMC: killing requests for dead queue" each time you suspend/resume.
> 
> I'm away from my serial cables at the moment but I may be able to
> provide more debug when I have them over the weekend. Have you any ideas
> on why this is breaking?
> 
> For reference, I've reproduced the problem with both the PXA host driver
> and different driver not merged into mainline (ASIC3).

Just to follow up, if I boot with a rootfs from elsewhere and mount the
mmc card then suspend/resume, it corrupts the data on the card too so it
looks like some general suspend/resume problem on mounted filesystems.

Richard

Re: MMC/SD Root filesystem suspend/resume problems

[Pierre Ossman]

On Thu, 19 Jul 2007 16:53:39 +0100
Richard Purdie <rpurdie@rpsys.net> wrote:

> Hi Pierre,
> 
> Lots of Linux handhelds use MMC/SD devices as the root file system.
> This has worked quite reliably for many kernel versions. In 2.6.22,
> it seems that if you suspend such a system then resume it, the device
> locks up. Trying to execute anything on the filesystem results in a
> "Permission Denied" message. I did see a message from the MMC
> subsystem saying it had redetected the card. There are also messages
> on the console like "MMC: killing requests for dead queue" each time
> you suspend/resume.
> 

The card is removed when you suspend and readded when you resume.
That's the only safe thing we can do until we get suspend support in
the filesystems.

If you really want to shoot yourself in the foot, there is a Kconfig
option that keeps the card around across the suspend.

Rgds
-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org

Re: MMC/SD Root filesystem suspend/resume problems

[Richard Purdie]

On Thu, 2007-07-19 at 19:03 +0200, Pierre Ossman wrote:
> On Thu, 19 Jul 2007 16:53:39 +0100
> Richard Purdie <rpurdie@rpsys.net> wrote:
> > Lots of Linux handhelds use MMC/SD devices as the root file system.
> > This has worked quite reliably for many kernel versions. In 2.6.22,
> > it seems that if you suspend such a system then resume it, the device
> > locks up. Trying to execute anything on the filesystem results in a
> > "Permission Denied" message. I did see a message from the MMC
> > subsystem saying it had redetected the card. There are also messages
> > on the console like "MMC: killing requests for dead queue" each time
> > you suspend/resume.
> 
> The card is removed when you suspend and readded when you resume.
> That's the only safe thing we can do until we get suspend support in
> the filesystems.
> 
> If you really want to shoot yourself in the foot, there is a Kconfig
> option that keeps the card around across the suspend.

I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing was
"fixed". I think having this option is a bad idea (in its current form)
as it doesn't actually stop filesystem corruption.

With the option disabled, if a filesystem is mounted when you suspend my
tests show the filesystem is corrupted. At least if the option is
enabled, the filesystem is only corrupted if you remove the card whilst
suspended which is more preferable.

I guess the solution would be to abort the suspend if mounted systems
were detected and the option was disabled? Alternatively the option
could be "auto" enabled only for mounted systems maybe with a printk
warning?

Of course the best solution would be to have filesystems support
suspend/resume requests since other subsystems like pcmcia also suffer
this problem and would benefit from this but I accept that teaching
filesystems this is more difficult.

Regards,

Richard

Re: MMC/SD Root filesystem suspend/resume problems

[Pierre Ossman]

On Sun, 22 Jul 2007 14:18:33 +0100
Richard Purdie <rpurdie@rpsys.net> wrote:

> 
> I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing
> was "fixed". I think having this option is a bad idea (in its current
> form) as it doesn't actually stop filesystem corruption.
> 
> With the option disabled, if a filesystem is mounted when you suspend
> my tests show the filesystem is corrupted. At least if the option is
> enabled, the filesystem is only corrupted if you remove the card
> whilst suspended which is more preferable.
> 

I disagree. With this option you get silent corruption, without you get
noisy corruption. And I would always prefer the latter, even if it
increases the risk of it happening.

> I guess the solution would be to abort the suspend if mounted systems
> were detected and the option was disabled? Alternatively the option
> could be "auto" enabled only for mounted systems maybe with a printk
> warning?
> 

This is a general problem for all removable/hotpluggable storage. So
sticking it in the MMC block device would be the wrong layer IMO.

Until the filesystems can be made to store something sane on disk
before the suspend, I'd say this is best handled in user space. Let the
user space tools refuse to initiate the suspend as long as any
removable devices are mounted.

> Of course the best solution would be to have filesystems support
> suspend/resume requests since other subsystems like pcmcia also suffer
> this problem and would benefit from this but I accept that teaching
> filesystems this is more difficult.
> 

Doesn't mean we shouldn't do it. And if we keep papering over the
problems, you reduce the motivation of fixing this properly.

Rgds
-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org

Re: MMC/SD Root filesystem suspend/resume problems

[Richard Purdie]

On Sun, 2007-07-22 at 16:05 +0200, Pierre Ossman wrote:
> On Sun, 22 Jul 2007 14:18:33 +0100
> Richard Purdie <rpurdie@rpsys.net> wrote:
> > I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing
> > was "fixed". I think having this option is a bad idea (in its current
> > form) as it doesn't actually stop filesystem corruption.
> > 
> > With the option disabled, if a filesystem is mounted when you suspend
> > my tests show the filesystem is corrupted. At least if the option is
> > enabled, the filesystem is only corrupted if you remove the card
> > whilst suspended which is more preferable.
> 
> I disagree. With this option you get silent corruption, without you get
> noisy corruption. And I would always prefer the latter, even if it
> increases the risk of it happening.

Corruption is corruption and it shouldn't happen if we can avoid it. It
happens with complete certainty in one case and only happens in the
other if the user does something which is a fairly obvious bad idea
(which is documented as such).

> > I guess the solution would be to abort the suspend if mounted systems
> > were detected and the option was disabled? Alternatively the option
> > could be "auto" enabled only for mounted systems maybe with a printk
> > warning?
> 
> This is a general problem for all removable/hotpluggable storage. So
> sticking it in the MMC block device would be the wrong layer IMO.

It is however if the MMC layer is going to add Kconfig options which
corrupt things, it can add things to start fixing things too. If those
things can be adapted into more generic code paths, so much the better.

> Until the filesystems can be made to store something sane on disk
> before the suspend, I'd say this is best handled in user space. Let the
> user space tools refuse to initiate the suspend as long as any
> removable devices are mounted.

Given I can suspend the device with "echo mem > /sys/power/state", that
implies we need to fix echo? ;-)

> > Of course the best solution would be to have filesystems support
> > suspend/resume requests since other subsystems like pcmcia also suffer
> > this problem and would benefit from this but I accept that teaching
> > filesystems this is more difficult.
> > 
> 
> Doesn't mean we shouldn't do it. 

Agreed.

> And if we keep papering over the problems, you reduce the motivation
> of fixing this properly.

Maybe although I don't like existing functionality being broken even if
its less than ideal.

Regards.

Richard

Re: MMC/SD Root filesystem suspend/resume problems

[Pierre Ossman]

On Sun, 22 Jul 2007 15:28:00 +0100
Richard Purdie <rpurdie@rpsys.net> wrote:

> 
> Corruption is corruption and it shouldn't happen if we can avoid it.
> It happens with complete certainty in one case and only happens in the
> other if the user does something which is a fairly obvious bad idea
> (which is documented as such).
> 

The corruption will only occur if the filesystem is dirty. Granted, the
mount will be dead and useless, but I wouldn't call that corruption.

Anyway, this behaviour was selected after seeing the long discussion
about how USB should handle the same problem. It was decided that it
was best to play it safe and remove any devices that couldn't be
determined to have remained in the slot. We also have the USB_PERSIST
option these days, which does the same thing as MMC_UNSAFE_RESUME.

> 
> Given I can suspend the device with "echo mem > /sys/power/state",
> that implies we need to fix echo? ;-)
> 

Or that direct usage of /sys/power/state is only for those who know
what they are doing (and have umounted their filesystems beforehand).

> 
> > And if we keep papering over the problems, you reduce the motivation
> > of fixing this properly.
> 
> Maybe although I don't like existing functionality being broken even
> if its less than ideal.
> 

I am of the opinion that it was more broken before I touched it. Silent
corruption is never acceptable in my book. But if it is in yours, just
enable MMC_UNSAFE_RESUME and you'll have the old behaviour.

Rgds
-- 
     -- Pierre Ossman

  Linux kernel, MMC maintainer        http://www.kernel.org
  PulseAudio, core developer          http://pulseaudio.org
  rdesktop, core developer          http://www.rdesktop.org

Re: MMC/SD Root filesystem suspend/resume problems

[Pavel Machek]

Hi!

> > > Lots of Linux handhelds use MMC/SD devices as the root file system.
> > > This has worked quite reliably for many kernel versions. In 2.6.22,
> > > it seems that if you suspend such a system then resume it, the device
> > > locks up. Trying to execute anything on the filesystem results in a
> > > "Permission Denied" message. I did see a message from the MMC
> > > subsystem saying it had redetected the card. There are also messages
> > > on the console like "MMC: killing requests for dead queue" each time
> > > you suspend/resume.
> > 
> > The card is removed when you suspend and readded when you resume.
> > That's the only safe thing we can do until we get suspend support in
> > the filesystems.
> > 
> > If you really want to shoot yourself in the foot, there is a Kconfig
> > option that keeps the card around across the suspend.
> 
> I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing was
> "fixed". I think having this option is a bad idea (in its current form)
> as it doesn't actually stop filesystem corruption.
> 
> With the option disabled, if a filesystem is mounted when you suspend my
> tests show the filesystem is corrupted. At least if the option is
> enabled, the filesystem is only corrupted if you remove the card whilst
> suspended which is more preferable.

Are we talking _corruption_ here, or are we talking 'the kind of
corruption recoverable by fsck that happens on powerfail'?

							Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

Re: MMC/SD Root filesystem suspend/resume problems

[Richard Purdie]

On Wed, 2007-07-25 at 19:01 +0000, Pavel Machek wrote:
> > I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing was
> > "fixed". I think having this option is a bad idea (in its current form)
> > as it doesn't actually stop filesystem corruption.
> > 
> > With the option disabled, if a filesystem is mounted when you suspend my
> > tests show the filesystem is corrupted. At least if the option is
> > enabled, the filesystem is only corrupted if you remove the card whilst
> > suspended which is more preferable.
> 
> Are we talking _corruption_ here, or are we talking 'the kind of
> corruption recoverable by fsck that happens on powerfail'?

There was more damage to the system than just a dirty bit set. Yes, fsck
could fix it but I don't think it should happen in the first place...

Richard

Re: MMC/SD Root filesystem suspend/resume problems

[Pavel Machek]

On Wed 2007-07-25 20:20:42, Richard Purdie wrote:
> On Wed, 2007-07-25 at 19:01 +0000, Pavel Machek wrote:
> > > I enabled the MMC_UNSAFE_RESUME option and the problems I was seeing was
> > > "fixed". I think having this option is a bad idea (in its current form)
> > > as it doesn't actually stop filesystem corruption.
> > > 
> > > With the option disabled, if a filesystem is mounted when you suspend my
> > > tests show the filesystem is corrupted. At least if the option is
> > > enabled, the filesystem is only corrupted if you remove the card whilst
> > > suspended which is more preferable.
> > 
> > Are we talking _corruption_ here, or are we talking 'the kind of
> > corruption recoverable by fsck that happens on powerfail'?
> 
> There was more damage to the system than just a dirty bit set. Yes, fsck
> could fix it but I don't think it should happen in the first place...

Well, that's "ok", that happens on sudden powerdowns, too.

(Well, but we do sync() during suspend, so it is a bit strange). Do
you have fsck logs perhaps?
									Pavel

-- 
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html