Andrew, today's -git started crashing in the VM, during bootup in rc.sysinit in our tests: --------------------> EXT3-fs: recovery complete. EXT3-fs: mounted filesystem with ordered data mode. VFS: Mounted root (ext3 filesystem) readonly. debug: unmapping init memory c11c9000..c1398000 gzip used greatest stack depth: 5916 bytes left BUG: unable to handle kernel NULL pointer dereference at 00000000 IP: [] page_cgroup_zoneinfo+0xf/0x1b *pde = 00000000 Oops: 0000 [#1] SMP DEBUG_PAGEALLOC last sysfs file: Dumping ftrace buffer: (ftrace buffer empty) Pid: 3595, comm: rc.sysinit Not tainted (2.6.27 #44341) System Product Name EIP: 0060:[] EFLAGS: 00010246 CPU: 1 EIP is at page_cgroup_zoneinfo+0xf/0x1b EAX: 00000000 EBX: 00000003 ECX: 00000002 EDX: c199ae00 ESI: c2c7d898 EDI: c199ae0c EBP: f5e2dcf8 ESP: f5e2dcd4 DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Process rc.sysinit (pid: 3595, ti=f5e2c000 task=f5ddd240 task.ti=f5e2c000) Stack: c019027a 00000010 00120050 c199ae0c 00000005 f5e2dd34 c1babb90 f5dd0240 c1babb90 f5e2dd18 c0190993 00000000 00000000 00120050 c1babb90 f5ce4800 f6c24868 f5e2dd34 c0170cbd 000f0019 00120050 c1babb90 f5ce4800 00000000 Call Trace: [] ? mem_cgroup_charge_common+0x149/0x1c4 [] ? mem_cgroup_cache_charge+0x81/0x8f [] ? add_to_page_cache_locked+0x34/0xc9 [] ? add_to_page_cache_lru+0x28/0x63 [] ? find_or_create_page+0x44/0x67 [] ? __getblk+0xff/0x1d1 [] ? sb_getblk+0x16/0x18 [] ? __ext3_get_inode_loc+0xc5/0x27c [] ? _raw_spin_unlock+0x74/0x78 [] ? iget_locked+0xb1/0xdb [] ? ext3_iget+0x5b/0x310 [] ? _spin_unlock+0x22/0x25 [] ? ext3_lookup+0x73/0x93 [] ? do_lookup+0xbb/0x129 [] ? __link_path_walk+0x47c/0x5a9 [] ? path_walk+0x3e/0x77 [] ? do_path_lookup+0xec/0x125 [] ? getname+0x8f/0xab [] ? user_path_at+0x35/0x5b [] ? native_sched_clock+0x97/0xb5 [] ? trace_hardirqs_off+0xb/0xd [] ? time_hardirqs_off+0x1b/0x1f [] ? trace_hardirqs_off_caller+0x15/0x7b [] ? vfs_stat_fd+0x1e/0x45 [] ? vfs_stat+0x16/0x18 [] ? sys_stat64+0x19/0x2d [] ? __movsl_is_ok+0x9/0x25 [] ? trace_hardirqs_off_thunk+0xc/0x10 [] ? sysenter_exit+0xf/0x1a [] ? trace_hardirqs_on_thunk+0xc/0x10 [] ? trace_hardirqs_on_thunk+0xc/0x10 [] ? trace_hardirqs_on_caller+0x9c/0xce [] ? trace_hardirqs_on_thunk+0xc/0x10 [] ? sysenter_do_call+0x12/0x3f Code: 00 eb 13 c1 e3 07 83 84 1e 98 00 00 00 01 83 94 1e 9c 00 00 00 00 58 5b 5e 5f 5d c3 55 89 e5 e8 53 4e f7 ff 8b 50 04 8b 40 08 5d <8b> 00 c1 e8 1e 6b c0 50 03 42 4c c3 55 89 e5 57 56 53 e8 35 4e EIP: [] page_cgroup_zoneinfo+0xf/0x1b SS:ESP 0068:f5e2dcd4 ---[ end trace 525093d040e4a2a1 ]--- rc.sysinit used greatest stack depth: 5840 bytes left BUG: unable to handle kernel NULL pointer dereference at 00000000 IP: [] page_cgroup_zoneinfo+0xf/0x1b *pde = 00000000 Oops: 0000 [#2] SMP DEBUG_PAGEALLOC last sysfs file: Pid: 1, comm: init Tainted: G D (2.6.27 #44341) System Product Name EIP: 0060:[] EFLAGS: 00010246 CPU: 1 EIP is at page_cgroup_zoneinfo+0xf/0x1b EAX: 00000000 EBX: 00000003 ECX: 00000002 EDX: c199ae00 ESI: c2c7d8ac EDI: c199ae0c EBP: f704dbb0 ESP: f704db8c DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Process init (pid: 1, ti=f704c000 task=f7050000 task.ti=f704c000) Stack: c019027a 00000010 00120050 c199ae0c 00000005 f704dbec c1babbc8 f5d50000 c1babbc8 f704dbd0 c0190993 00000000 00000000 00120050 c1babbc8 f704dd68 f6c24868 f704dbec c0170cbd 0041a803 00120050 c1babbc8 f704dd68 00000000 Call Trace: [] ? mem_cgroup_charge_common+0x149/0x1c4 [] ? mem_cgroup_cache_charge+0x81/0x8f [] ? add_to_page_cache_locked+0x34/0xc9 [] ? add_to_page_cache_lru+0x28/0x63 [] ? find_or_create_page+0x44/0x67 [] ? __getblk+0xff/0x1d1 [] ? sb_getblk+0x16/0x18 [] ? ext3_getblk+0x65/0x111 [] ? constant_test_bit+0x9/0x20 [] ? dx_root_limit+0x8/0x1b [] ? ext3_bread+0x19/0x69 [] ? ext3_find_entry+0x121/0x52a [] ? kmem_cache_alloc+0x95/0xd4 [] ? kmem_cache_alloc+0x95/0xd4 [] ? trace_hardirqs_on+0xb/0xd [] ? ext3_lookup+0x2a/0x93 [] ? do_lookup+0xbb/0x129 [] ? __link_path_walk+0x47c/0x5a9 [] ? path_walk+0x3e/0x77 [] ? do_path_lookup+0xec/0x125 [] ? path_lookup+0x17/0x19 [] ? unix_find_other+0x30/0x15d [] ? kmem_cache_alloc+0x95/0xd4 [] ? time_hardirqs_on+0x1b/0x1f [] ? strlen+0x9/0x1a [] ? unix_dgram_connect+0x85/0x150 [] ? sys_connect+0x65/0x7f [] ? __movsl_is_ok+0x9/0x25 [] ? __copy_from_user_ll+0x16/0xd4 [] ? sys_socketcall+0x91/0x196 [] ? sysenter_do_call+0x12/0x3f Code: 00 eb 13 c1 e3 07 83 84 1e 98 00 00 00 01 83 94 1e 9c 00 00 00 00 58 5b 5e 5f 5d c3 55 89 e5 e8 53 4e f7 ff 8b 50 04 8b 40 08 5d <8b> 00 c1 e8 1e 6b c0 50 03 42 4c c3 55 89 e5 57 56 53 e8 35 4e EIP: [] page_cgroup_zoneinfo+0xf/0x1b SS:ESP 0068:f704db8c ---[ end trace 525093d040e4a2a1 ]--- init used greatest stack depth: 5036 bytes left Kernel panic - not syncing: Attempted to kill init! <-------------------- potential candidates would be: 52d4b9a: memcg: allocate all page_cgroup at boot c05555b: memcg: atomic ops for page_cgroup->flags addb9ef: memcg: optimize per-cpu statistics b7abea9: memcg: make page->mapping NULL before uncharge 7b85412: Unevictable LRU Page Statistics 894bc31: Unevictable LRU Infrastructure 4f98a2f: vmscan: split LRU lists into anon & file sets b69408e: vmscan: Use an indexed array for LRU variables is this known, or should i try to bisect? It's reproducible. Config attached. bad: a9b6148: USB: Fix unused label warnings in drivers/usb/host/ehci-hcd.c good: c813b4e: Merge git://git.kernel.org/pub/scm/linux/kernel/git/gregkh/ Ingo