
 matchreply				- Tejun Heo <htejun@gmail.com>

====
 1. Introduction

matchreply checks whether a mail message is a reply to any message in
the specified Maildir folders.  To be exact, it checks whether any of
message IDs in the In-reply-to: and References: headers of the message
read from stdin matches the message ID in the Message-id: header of
any message in the specified Maildir folders.  If there is a match,
matchreply exits with 0; otherwise, 1.

Well, the above paragraph wasn't very intuitive, was it?

I'm subscribed to several mailing lists and one of them happens to be
LKML which pushes hundreds of mails per day.  My procmail nicely sorts
mails from lists into separate folders.  Due to its high traffic, I
tend to only skim through most messages in the LKML folder once a day
or two.

Problem arises when I find an interesting thread.  After reading upto
what's posted, I want to keep an eye on the follow-up messages but
it's not too easy as the folder soon gets flooded with other messages
and after a while I forget what I was interested in.  Labeling didn't
help much and also was cumbersome because I use multiple mail clients.

What I wanted was moving the thread to my INBOX folder and the
follow-up messages being delievered there.  So, that's what matchreply
does.  It helps follow-up messages to be delivered to the folders
where their associated threads are.

====
 2. Some details

* matchreply only supports Maildir format.  If you're subscribed to
  high-traffic mailing lists and still using mbox format, consider
  swithing to Maildir.  It's much better.

* Also, it uses inotify system calls to monitor changes in mail
  folders. Unfortunately, inotify system calls are avaliable only on
  Linux >= 2.6.13.

* When trying to process the first query, matchreply creates a server
  which maintains ID hash table and actually processes queries.  When
  a server starts, it scans all messages in the specified folders to
  build ID index.

  Indexing about 23000 messages takes slightly over 9 seconds on
  Celeron 2.7Ghz machine with 80GB IDE harddisk.  CPU usage is about
  1.2 seconds.  The rest is spent waiting for the disk.  Once started,
  index updates and queries are really cheap.

* On x86_64, RSS of matchreply server with ~23k messages indexed is
  about 4.5MB.  On i386, it's about 3MB.

====
 3. COMPILE & INSTALL

There's just one file.  matchreply.c.  Just do

$ gcc -o matchreply matchreply.c

Copy the resulting matchreply to any directory you want.  Simple.

Well, that was the ideal case.  Compilation will fail if your libc
doesn't support inotify syscalls yet.  As of this writing
(2005.10.12), most distributions isn't shipping inotify-ready libc
yet.

When compilation fails because of missing inotify support, do the
followings.

* If include/linux/inotify.h doesn't exist, copy it from linux source
  tree.

* If inotify syscalls (inotify_init, inotify_add_watch...) are not
  defined and you're on either i386 or x86_64...

  On i386,

  $ gcc -DI386_DEFINE_INOTIFY_SYSCALLS -o matchreply matchreply.c

  On x86_64,

  $ gcc -DX86_64_DEFINE_INOTIFY_SYSCALLS -o matchreply matchreply.c

* If inotify syscalls are not defined and you're not on either i386 or
  x86_64, add syscall definitions similary to i386 and x86_64.
  __NR_inotify_* constants are usually defined in
  include/asm-ARCH/unistd.h in the linux source tree and adding
  syscalls shouldn't be difficult once you have those syscall numbers.

====
 4. Setting up .procmailrc

It's very simple.  Let's say you want to sort any message which
belongs to a thread in your INBOX (root Maildir) to the INBOX.  Adding
the following above any other rules in .procmailrc will achieve that.

####

MAILDIR=$HOME/Maildir
DEFAULT=$HOME/Maildir/

:0 Wc
| matchreply -l $MAILDIR/mr.log -i inbox $MAILDIR/cur $MAILDIR/Maildir/new

:0 a
$DEFAULT

####

The first rule sends a copy of an incoming message to matchreply and
the second rule delivers the message to the root folder if the first
rule completes successfully - IOW, matchreply reports success.

-l option specifies log file.  matchreply usually generates one line
of log per message.  If you don't specify any, it will print logs to
stderr.  If you are *absolutely* sure that you don't want any log,
specify /dev/null.

-i option is mandatory.  This instance ID associates clients with
appropriate servers.  Use unique, short and descriptive IDs.  These
also appear in log.

The remaining arguments are Maildir directories to index and watch.
Note that you should always list both cur and new directories.

And that's about it.  If folders are added or removed or any option is
changed, the server will automatically restart.  So, you don't have to
worry about server management.  Writing matching rules is enough.

One thing to be careful about when writing rules is that you must not
use the same instance ID for different rules.  As said above, servers
will restart if configuration changes and if you use the same instance
ID for different rules, the server serving the instance ID will
restart everytime messages hit different rules.

====
 5. My .procmailrc

INBOX		: default folder
Linux-git	: git mailing list
Linux-ide	: linux-ide mailing list
Linux-scsi	: linux-scsi mailing list
LKML		: linux-kernel mailing list
News		: cron reports, LWN news letter
Spam		: finance / health / pe**s enlargement related

And here are what I wanna do.

a. all messages with matching threads in INBOX should go to INBOX

b. of messages which have X-Mailing-List header to one of the lists,

	a. messages which are addressed to me, to INBOX

	b. messages with matching threads in Linux-ide, to Linux-ide

	c. messages with matching threads in Linux-scsi, to Linux-scsi

	d. all other messages are delivered according to where it's
           addressed in the following order.

		Linux-ide
		Linux-scsi
		LKML
		Linux-git

	e. if all above rules don't match, to INBOX (shouldn't happen)

c. if LWN news letter or cron report, to News

d. run through spamassasin and if spam, to Spam

e. to INBOX

####

PATH=/usr/local/bin:/usr/bin/:/bin
MAILDIR=$HOME/Maildir
DEFAULT=$HOME/Maildir/
LOGFILE=$MAILDIR/log
VERBOSE=yes
MREPLY="/home/tj/bin/matchreply -l $HOME/Maildir/matchreply.log -v"

:0 Wc
| $MREPLY -i inbox $MAILDIR/cur $MAILDIR/new

:0 a
$DEFAULT

:0
* ^X-Mailing-List:.*(linux-.*|git)@vger\.kernel\.org
{
	#
	# Deliver mails addressed to me into INBOX
	#

	:0
	* ^(To|Cc):.*htejun@gmail\.com
	$DEFAULT

	#
	# Don't let cross-posted messages of a thread end up in
	# different folders.
	#

	:0 Wc
	| $MREPLY -i lide $MAILDIR/.Linux-ide/cur $MAILDIR/.Linux-ide/new

	:0 a
	.Linux-ide/

	:0 Wc
	| $MREPLY -i lscsi $MAILDIR/.Linux-scsi/cur $MAILDIR/.Linux-scsi/new

	:0 a
	.Linux-scsi/

	#
	# Okay, deliver according to address
	#

	:0
	* ^(X-Mailing-List|To|Cc):.*linux-ide@vger\.kernel\.org
	.Linux-ide/

	:0
	* ^(X-Mailing-List|To|Cc):.*linux-scsi@vger\.kernel\.org
	.Linux-scsi/

	:0
	* ^(X-Mailing-List|To|Cc):.*linux-kernel@vger\.kernel\.org
	.LKML/

	:0
	* ^(X-Mailing-List|To|Cc):.*git@vger\.kernel\.org
	.Linux-git/

	:0
	$DEFAULT
}

:0
* ^From: (LWN.net Daily Summary <lwn@lwn.net>|root@htj.dyndns.org \(Cron Daemon\))
News

:0fw: spamassassin.lock
* < 256000
| spamassassin

:0:
* ^X-Spam-Status: Yes
.Spam/
