The only thing I have that gives any hint as to what is using memory is
/proc/meminfo and free .  

        total:    used:    free:  shared: buffers:  cached:
Mem:  660455424 653500416  6955008        0  5558272 56893440
Swap: 1091526656 48332800 1043193856
MemTotal:       644976 kB
MemFree:          6792 kB
MemShared:           0 kB
Buffers:          5428 kB
Cached:          40688 kB
SwapCached:      14872 kB
Active:          71176 kB
Inact_dirty:     26196 kB
Inact_clean:     14768 kB
Inact_target:    22428 kB
HighTotal:           0 kB
HighFree:            0 kB
LowTotal:       644976 kB
LowFree:          6792 kB
SwapTotal:     1065944 kB
SwapFree:      1018744 kB
Committed_AS:   229088 kB

            total       used       free     shared    buffers     cached
Mem:        644976     638200      6776          0       7168      42560
-/+ buffers/cache:     588472      56504
Swap:      1065944      56624    1009320


now, usually it's people complaining about all this cache being used in
ram and their system swapping despite this cache.  But here it's
obviously a problem with buffers.  I get a gradual increase in buffers
as time goes by and after 2 days it is at what it is at now, after X
crashed.   I've restarted my normal array of programs I have running all
the time and ps and top both show very little memory being used by any
single program, nothing that could account for 560MB of buffers.  

So where do these buffers come from and how can I find out?  I'm
assuming it's a userlevel program causing all this buffer to be used but
then if that's the case, why does it not show up as being used by it in
ps ? 

closing X doesn't seem to release all the buffer or even much, so I
assume it's something running as a daemon (mysql or apache or whatever)
but then again, they are using almost no memory at all according to ps