Re: [OT] Confirmation Spam Blocking was: List 'linux-dvb' closed to public posts

["Kevin O'Connor" <kevin@koconnor.net>]

On Sat, Jan 24, 2004 at 01:12:34PM -0800, Linus Torvalds wrote:
> 
> On Sat, 24 Jan 2004, Kevin O'Connor wrote:
> > 
> > A good Bayesian spam filter isn't nearly as susceptible to random words as
> > some people think.  Words that are likely to be spam (along with words that
> > are frequently "ham") are given _exponentially_ more weight than other
> > words.
> 
> Especially if the "random words" in the spam end up being weighted by real
> frequency, you just _cannot_ use single-word bayes filters on it. Or if 
> you do, you'll eventually have those words either being neutral, or (worst 
> of all cases) you'll have real mail be marked as spam after having 
> aggressively trained the filter for the spams.

A "random" word will not occur frequently enough in spam messages (when
measured over a large sample of spam) to become a "spam" token or to
adversely effect it becoming a "ham" token.  (If it did, then it would be a
good indicator of "spam", and spammers wouldn't be using it in their
"random" word blocks.)  Also, the algorithms in the filter make sure that
words don't become spam tokens just because one receives more spam than ham
(ie. you can't train the filter to over aggressively catch spam).

Don't get me wrong - I agree that multi-word bayes filters can be more
precise.  But, I don't see them as being significantly different from
single word filters -- they're susceptible to the same "attack" you outline
above (with blocks of random sentences) -- and they will have larger
dictionaries and require more training time.

> It might not be that big of a deal especially if you have a fairly narrow 
> scope of emails in your ham-list, but people who get mail from varied 
> sources _will_ get screwed by this, one way or the other.

A lot of testing has been done on these filters (see for example
http://spambayes.sourceforge.net/background.html) with a very large email
corpus.  If you haven't looked at bayes filters recently, or have only been
looking at the simplistic ones, then I think you might have better luck
trying again.

-Kevin

-- 
 ---------------------------------------------------------------------
 | Kevin O'Connor                  "BTW, IMHO we need a FAQ for      |
 | kevin@koconnor.net               'IMHO', 'FAQ', 'BTW', etc. !"    |
 ---------------------------------------------------------------------