Nasty git corruption problem

Nasty git corruption problem

[Alan Cox]

Just hit a real nasty, although I suspect its not a common case but it
does seem to show a problem git has other version control systems don't
(so far anyway)

During a git rebase my machine crashed. Git claims that the rebase is
complete but contains none of the outstanding 30 odd patches. There is
no .dotest directory and git-fsck-objects produces some warnings about a
few dangling objects, but these objects aren't the relevant ones (at
least directly)

CVS and SVN in crashes don't lose old stuff, though they are pretty good
at losing the last commit or two. Git rebase appears to be able to lose
two weeks of old changes even though they were stable on disk, which is
not good at all.

Doing

for i in *; do (cd $i; for j in *; do git-unpack-file $i$j; done; );
done

shows that lots of the changes are still somewhere in the object tree
but there seems to be no tool for fixing rather than moaning about
objects dangling, and also no obvious way to fix it. Also curiously many
of the objects appear linked somewhere but don't show up in the git-log
for the afflicted branch at all.

Re: Nasty git corruption problem

[Linus Torvalds]

On Wed, 26 Jul 2006, Alan Cox wrote:
> 
> During a git rebase my machine crashed. Git claims that the rebase is 
> complete but contains none of the outstanding 30 odd patches. There is 
> no .dotest directory and git-fsck-objects produces some warnings about a 
> few dangling objects, but these objects aren't the relevant ones (at 
> least directly)

They definitely should be, unless you actually did a "git prune" (or a 
"git repack -d").

> CVS and SVN in crashes don't lose old stuff, though they are pretty good
> at losing the last commit or two.

And git should be even harder to get to lose old stuff, because it won't 
even touch it. 

> Doing
> 
> for i in *; do (cd $i; for j in *; do git-unpack-file $i$j; done; );
> done
> 
> shows that lots of the changes are still somewhere in the object tree
> but there seems to be no tool for fixing rather than moaning about
> objects dangling, and also no obvious way to fix it.

Well, the "dangling objects" really should be the fix. We could make it 
even more obvious by creating links to the dangling objects in a 
"lost+found" directory, but I usually just do it by hand.

So the thing to do to recover any old stuff is

 - do "git-fsck-objects --full".

   The "--full" is going to make it much slower, but it means that it will 
   look _inside_ old packs too, and if you repacked the old stuff, that's 
   what you want. Besides, if you actually had a real crash in the middle 
   of a git op, you probably do want this anyway, although quite frankly, 
   a truly corrupted pack is pretty damn unlikely (the only case I 
   remember ever seing was due to actual hardware problems)

   I _suspect_ you didn't do the "--full". By default, git-fsck-objects 
   will only look at the unpacked objects, exactly because pack-files are 
   so stable. So the only reason to use "--full" is if you're really anal 
   and suspect hw issues, _or_ if you are looking for dangling work that 
   may be older than your last repack.

 - take all the dangling objects, and either list them explicitly to 
   "gitk" (or other visualizer), or you could save them off as

	.git/refs/lost+found/<some-random-names-here>

   and then do "gitk --all"

In fact, if you want to play around with a git patch, this trivial one 
should make git-fsck-objects create those lost-and-found entries 
automatically if you give it the "--lost-n-found" flag.

Anyway, I'm pretty sure that git is a hell of a lot _safer_ than either 
CVS or SVN have ever been, or will ever be.

		Linus

---
diff --git a/fsck-objects.c b/fsck-objects.c
index e167f41..fa50190 100644
--- a/fsck-objects.c
+++ b/fsck-objects.c
@@ -14,6 +14,7 @@ #include "tree-walk.h"
 #define REACHABLE 0x0001
 #define SEEN      0x0002
 
+static int lost_and_found = 0;
 static int show_root = 0;
 static int show_tags = 0;
 static int show_unreachable = 0;
@@ -102,8 +103,23 @@ static void check_connectivity(void)
 		}
 
 		if (!obj->used) {
-			printf("dangling %s %s\n", typename(obj->type),
-			       sha1_to_hex(obj->sha1));
+			char *hex = sha1_to_hex(obj->sha1);
+			printf("dangling %s %s\n", typename(obj->type), hex);
+			if (lost_and_found) {
+				int fd, ret;
+				mkdir(git_path("refs/lost+found"), 0777);
+				fd = open(git_path("refs/lost+found/%s", hex), O_CREAT | O_TRUNC | O_WRONLY, 0666);
+				if (fd < 0) {
+					perror("lost+found");
+					continue;
+				}
+				hex[40] = '\n';
+				ret = xwrite(fd, hex, 41);
+				close(fd);
+				if (ret != 41)
+					error("unable to write to lost+found");
+				continue;
+			}
 		}
 	}
 }
@@ -514,6 +530,10 @@ int main(int argc, char **argv)
 			check_strict = 1;
 			continue;
 		}
+		if (!strcmp(arg, "--lost-n-found")) {
+			lost_and_found = 1;
+			continue;
+		}
 		if (*arg == '-')
 			usage("git-fsck-objects [--tags] [--root] [[--unreachable] [--cache] [--full] [--strict] <head-sha1>*]");
 	}

Re: Nasty git corruption problem

[Johannes Schindelin]

Hi,

On Wed, 26 Jul 2006, Linus Torvalds wrote:

> Well, the "dangling objects" really should be the fix. We could make it 
> even more obvious by creating links to the dangling objects in a 
> "lost+found" directory, but I usually just do it by hand.

We _do_ have git-lost-found.sh.

Hth,
Dscho

Re: Nasty git corruption problem

[Linus Torvalds]

On Wed, 26 Jul 2006, Johannes Schindelin wrote:
>
> We _do_ have git-lost-found.sh.

Yeah, sure, I knew that, I was just checking who was awake..

I think the "git-fsck-objects --lost-n-found" patch may be nicer, though. 
Of course, only if somebody also adds some documentation to it, so that 
people can actually see that it is there.

		Linus

Re: Nasty git corruption problem

[Linus Torvalds]

On Wed, 26 Jul 2006, Linus Torvalds wrote:
> On Wed, 26 Jul 2006, Johannes Schindelin wrote:
> >
> > We _do_ have git-lost-found.sh.
> 
> Yeah, sure, I knew that, I was just checking who was awake..

Actually, now that I actually look at it, I notice that it doesn't pass in 
"--full" to git-fsck-objects, so it doesn't actually work that well in the 
presense of dangling objects that are inside pack-files.

(And if it wasn't already obvious, with my patch you still need to do 
"git-fsck-objects --full --lost-n-found" if you want to look inside those 
pack-files, but at least it's an option you can enable).

			Linus

Re: Nasty git corruption problem

[Alan Cox]

Ar Mer, 2006-07-26 am 10:43 -0700, ysgrifennodd Linus Torvalds:
> (And if it wasn't already obvious, with my patch you still need to do 
> "git-fsck-objects --full --lost-n-found" if you want to look inside those 
> pack-files, but at least it's an option you can enable).

git-lost-found turns up some of the missing stuff that was applied
earliest in the rebase but the other stuff is apparently neither visible
anywhere in the tree or missing (the tree I was rebasing "^^^..." never
shows it nor does the log). The changes are in the objects if you dump
every object and investigate them by hand.

Beats me but a mix of a restore and reapplying some stuff from archived
email along with rescued objects seems to have recovered all lost
changesets.

Alan

Re: Nasty git corruption problem

[Linus Torvalds]

On Thu, 27 Jul 2006, Alan Cox wrote:
> 
> git-lost-found turns up some of the missing stuff that was applied
> earliest in the rebase but the other stuff is apparently neither visible
> anywhere in the tree or missing (the tree I was rebasing "^^^..." never
> shows it nor does the log).

Did you try "git-fsck-objects --full"?

The git-lost-found script is apparently broken, exactly because it doesn't 
do a "full".

		Linus

Re: Nasty git corruption problem

[Johannes Schindelin]

Hi,

On Thu, 27 Jul 2006, Linus Torvalds wrote:

> On Thu, 27 Jul 2006, Alan Cox wrote:
> > 
> > git-lost-found turns up some of the missing stuff that was applied
> > earliest in the rebase but the other stuff is apparently neither visible
> > anywhere in the tree or missing (the tree I was rebasing "^^^..." never
> > shows it nor does the log).
> 
> Did you try "git-fsck-objects --full"?
> 
> The git-lost-found script is apparently broken, exactly because it doesn't 
> do a "full".

Of course, I was assuming that nothing like repacking or pruning took 
place after the crash...

Ciao,
Dscho

Re: Nasty git corruption problem

[Linus Torvalds]

On Thu, 27 Jul 2006, Johannes Schindelin wrote:
> 
> On Thu, 27 Jul 2006, Linus Torvalds wrote:
> 
> > On Thu, 27 Jul 2006, Alan Cox wrote:
> > > 
> > > git-lost-found turns up some of the missing stuff that was applied
> > > earliest in the rebase but the other stuff is apparently neither visible
> > > anywhere in the tree or missing (the tree I was rebasing "^^^..." never
> > > shows it nor does the log).
> > 
> > Did you try "git-fsck-objects --full"?
> > 
> > The git-lost-found script is apparently broken, exactly because it doesn't 
> > do a "full".
> 
> Of course, I was assuming that nothing like repacking or pruning took 
> place after the crash...

That's not the point.

If somebody does a "git rebase", he might be changing the heads that have 
already been packed, and replacing them them with heads that have _not_ 
yet been packed. So the _dangling_ links are the old ones (in the 
pack-file), and "git-fsck-objects --full" is needed to see them.

That said, I still don't think Alan sees what he says he sees. Even if 
something crashes in the middle of a "git rebase", I think the old head 
should have been saved in .git/ORIG_HEAD, for example. 

That said, some of the more invasive operations (and "git rebase" 
certainly counts) should probably have a few "sync" operations to make 
sure that things like ORIG_HEAD really are on disk, so that we would be 
able to recreate the tree even _without_ anything like "git-fsck-objects".

		Linus

Re: Nasty git corruption problem

[Johannes Schindelin]

Hi,

On Thu, 27 Jul 2006, Linus Torvalds wrote:

> On Thu, 27 Jul 2006, Johannes Schindelin wrote:
> > 
> > On Thu, 27 Jul 2006, Linus Torvalds wrote:
> > 
> > > On Thu, 27 Jul 2006, Alan Cox wrote:
> > > > 
> > > > git-lost-found turns up some of the missing stuff that was applied
> > > > earliest in the rebase but the other stuff is apparently neither visible
> > > > anywhere in the tree or missing (the tree I was rebasing "^^^..." never
> > > > shows it nor does the log).
> > > 
> > > Did you try "git-fsck-objects --full"?
> > > 
> > > The git-lost-found script is apparently broken, exactly because it doesn't 
> > > do a "full".
> > 
> > Of course, I was assuming that nothing like repacking or pruning took 
> > place after the crash...
> 
> If somebody does a "git rebase", he might be changing the heads that have 
> already been packed, and replacing them them with heads that have _not_ 
> yet been packed.

Okay, I see your point.

> That said, I still don't think Alan sees what he says he sees. Even if 
> something crashes in the middle of a "git rebase", I think the old head 
> should have been saved in .git/ORIG_HEAD, for example. 
> 
> That said, some of the more invasive operations (and "git rebase" 
> certainly counts) should probably have a few "sync" operations to make 
> sure that things like ORIG_HEAD really are on disk, so that we would be 
> able to recreate the tree even _without_ anything like "git-fsck-objects".

Why not just the good ole' locking mechanism? Update not the HEAD 
directly, but a HEAD.lock, and if all went well, rename it into HEAD (of 
course, by HEAD I mean refs/heads/<whatever-head-you-mean>).

Ciao,
Dscho

Re: Nasty git corruption problem

[Carl Worth]

[-- Attachment #1: Type: text/plain, Size: 429 bytes --]

On Wed, 26 Jul 2006 10:07:07 -0700 (PDT), Linus Torvalds wrote:
>> 
>  - take all the dangling objects, and either list them explicitly to 
>    "gitk" (or other visualizer), or you could save them off as
> 
> 	.git/refs/lost+found/<some-random-names-here>
> 
>    and then do "gitk --all"

One thing I've thought would be handy in a situation like this is a
new option for gitk along the lines of:

	gitk --show-dangling

-Carl

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: Nasty git corruption problem

[Alex Riesen]

Linus Torvalds, Wed, Jul 26, 2006 19:07:07 +0200:
> In fact, if you want to play around with a git patch, this trivial one 
> should make git-fsck-objects create those lost-and-found entries 
> automatically if you give it the "--lost-n-found" flag.

there is git-lost-found already. Still, it somewhat expected of fsck
to do such things.