Subject: IMAP extensions needed for SPAM/HAM and WHITE/BLACK listing

From: George Michaelson <[email-masked]>
Date: Fri, 3 Jul 2009 09:05:36 +1000

I would like to suggest that IMAP extensions could usefully be  
specified to permit MUA to signal to the MS over IMAP, that the  
message in question is, or is not spam. I would also like to suggest  
that an IMAP extension pair to signal that the specified address is to  
be white, or black listed, would be equally useful.

This is because a large number of people now depend on IMAP backed MS  
in places like google, or corporate maildrops, but also have in-client  
spam tagging. Where the spam filtering is not sufficiently complete to  
prevent significant leakage, and mis-attribution across the MUA/MS  
dialogue, the lack of an in-protocol method to pass the information  
means we are all using what I would call ad-hoc methods to fix this  
situation up.

If it was possible to communicate this 'first class' in IMAP, I think  
that it would significantly enhance the user experience.

As an example, at the moment if I wish to inform google that I have  
known spam in local folders, I have to go to the web interface and  
manually tag. If there was an IMAP extension, I could review my local  
baysian junk folder, remove all non-spam (and flag the senders as  
white-listed if need be), and request the rest to be flagged as spam  
back on the IMAP backed MS.

Likewise for false positives and blacklisted senders.

I am told that the IMAPEXT WG is closed. I mail this, in the hope that  
somebody can help decide if this is a viable topic for consideration.

Many thanks for guidance and advice thus far from Lisa Dusseault.

cheers

-George

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Sun, 5 Jul 2009 21:03:30 +0200

On 3 jul 2009, at 1:05, George Michaelson wrote:

> manually tag. If there was an IMAP extension, I could review my  
> local baysian junk folder, remove all non-spam (and flag the senders  
> as white-listed if need be), and request the rest to be flagged as  
> spam back on the IMAP backed MS.

Doesn't moving the spam messages to the spam folder accomplish this  
already? That's what my client does.

But if you want this, I'd say that it needs to be a fractional thing,  
not a binary spam/no spam indication. For instance, the server could  
give something a spam score of 2 and the client also 2 and together  
that would be 4 so the message is presumed to be spam (assuming the  
spam threshold is 3), but in a binary system no spam OR no spam = no  
spam.

---

From: Dave Cridland <[email-masked]>
Date: Mon, 06 Jul 2009 09:24:42 +0100

On Sun Jul  5 20:03:30 2009, Iljitsch van Beijnum wrote:
>> As an example, at the moment if I wish to inform google that I  
>> have  known spam in local folders, I have to go to the web  
>> interface and  manually tag. If there was an IMAP extension, I  
>> could review my  local baysian junk folder, remove all non-spam  
>> (and flag the senders  as white-listed if need be), and request  
>> the rest to be flagged as  spam back on the IMAP backed MS.
> 
There's also a proposal (expired?) to use keywords to signal  
spamminess.

> But if you want this, I'd say that it needs to be a fractional  
> thing,  not a binary spam/no spam indication. For instance, the  
> server could  give something a spam score of 2 and the client also  
> 2 and together  that would be 4 so the message is presumed to be  
> spam (assuming the  spam threshold is 3), but in a binary system no  
> spam OR no spam = no  spam.

Of course, if both client and server use precisely the same criteria,  
you've simply halved your threshold.

There's two cases:

1) The server has some feedback-based spam detection mechanism, like  
a bayesian filter. You want to teach the server's filter about your  
explicit decisions.

2) Your client has a spam filter (or some sort), you want the server  
to tell your client about it's preliminary findings.

The notion of using two spam filters in concert to attempt to make an  
overall decision is basically flawed, because it fits into one or  
other case above - if you do go to the effort of having a range based  
spamminess from one, you'd have to use it as mere input into the  
other's decision process, since combining them naïvely would produce  
poorly weighted results.

A simple keyword approach isn't quite as good as ranges, but it does  
have the useful property that nearly every service already offers  
arbitrary keywords, and so is quite likely to already offer it.

Ranges, on the other hand, require the use of annotations, which is a  
rather more complex area.

Dave.
-- 
Dave Cridland - mailto:[email-masked] - xmpp:[email-masked]
  - acap://acap.dave.cridland.net/byowner/user/dwd/bookmarks/
  - http://dave.cridland.net/
Infotrope Polymer - ACAP, IMAP, ESMTP, and Lemonade

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 6 Jul 2009 10:37:06 +0200

> The notion of using two spam filters in concert to attempt to make  
> an overall decision is basically flawed, because it fits into one or  
> other case above - if you do go to the effort of having a range  
> based spamminess from one, you'd have to use it as mere input into  
> the other's decision process, since combining them naïvely would  
> produce poorly weighted results.

Right. The filtering would have to be complimentary. So perhaps there  
should be an indication of the type of filtering done, too. The reason  
to combine server and client side approaches is that servers can  
easily do blacklists, and running any type of filtering on the server  
gives it the opportunity to reject the message during the SMTP  
session, creating a good error message for real senders but no useless  
bounces to innocent third parties in the case of spoofed senders. On  
the client is useful because the client can typically better afford to  
run CPU intensive stuff like baysian filters and the client probably  
has a better list of previous correspondents that it can use to  
whitelist.

---

From: Dave Cridland <[email-masked]>
Date: Mon, 06 Jul 2009 10:05:41 +0100

On Mon Jul  6 09:37:06 2009, Iljitsch van Beijnum wrote:
>>> server could  give something a spam score of 2 and the client  
>>> also  2 and together  that would be 4 so the message is presumed  
>>> to be  spam (assuming the  spam threshold is 3), but in a binary  
>>> system no  spam OR no spam = no  spam.
> 
>>  an overall decision is basically flawed, because it fits into one  
>> or  other case above - if you do go to the effort of having a  
>> range  based spamminess from one, you'd have to use it as mere  
>> input into  the other's decision process, since combining them  
>> naïvely would  produce poorly weighted results.
> 
> Right. The filtering would have to be complimentary. So perhaps  
> there  should be an indication of the type of filtering done, too.  
> The reason  to combine server and client side approaches is that  
> servers can  easily do blacklists, and running any type of  
> filtering on the server  gives it the opportunity to reject the  
> message during the SMTP  session, creating a good error message for  
> real senders but no useless  bounces to innocent third parties in  
> the case of spoofed senders. On  the client is useful because the  
> client can typically better afford to  run CPU intensive stuff like  
> baysian filters and the client probably  has a better list of  
> previous correspondents that it can use to  whitelist.

Whitelists should, in principle, be easily comminicable to the server  
(and easily processed).

Bayesian filters on the client are only really useful if the same  
client is always used - this is often not the case (and, in my  
experience, is becoming rarer). Otherwise the quality of your spam  
filtering radically changes depending on the client you happen to be  
using at the time.

And basically, any filtering at all that can be done on the server  
saves a vast amount of bandwidth, and makes use of email on smaller  
devices (like mobile phones) instantly more useful.

The one massive advantage that a client has is direct contact with  
the user, and the eyeball remains the best spam filtering technology  
we have.

I'm not suggesting that clients must not do spam filtering -  
obviously they do, and it's often useful - but I think we should aim  
to make server-side filtering the best we can. A very simple uniform  
feedback mechanism of whether the user thinks a message is spam or  
not makes this very much easier.

Dave.

---

From: Sam Johnston <[email-masked]>
Date: Mon, 6 Jul 2009 11:27:00 +0200

On Mon, Jul 6, 2009 at 11:05 AM, Dave Cridland <[email-masked]> wrote:

> I'm not suggesting that clients must not do spam filtering - obviously they
> do, and it's often useful - but I think we should aim to make server-side
> filtering the best we can. A very simple uniform feedback mechanism of
> whether the user thinks a message is spam or not makes this very much
> easier.

Agreed - I use the Gmail web interface exclusively now (after OS X Mail lost
its mind a few years ago and forgot what was read, unread, labeled, etc.)
but if the "Report Spam" button were available in mail clients then I might
consider moving back. Being able to report that messages are not spam is
useful too, as is being able to black/whitelist certain senders (albeit less
so).

I'm sure I'm not alone in using a laptop and a mobile client (many have
desktop and/or home clients in the mix too) so whatever can possibly be
centralised should be.

Another thing to consider is that it's not always black and white - certain
types of spam also fall into the phishing category and should be
specifically investigated (Gmail has a "Report Phishing" menu item for this
which allows them to label phishing messages accordingly) and Postini
categorises spam so you can lower or raise the threshold depending on
whether its type... that way accountants, lawers, etc. can avoid losing good
mail (and others can avoid receiving it).

Sam

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 06 Jul 2009 11:27:45 +0100

Dave Cridland wrote:

>>> (and flag the senders  as white-listed if need be), and request  the 
>>> rest to be flagged as  spam back on the IMAP backed MS.
>>
> There's also a proposal (expired?) to use keywords to signal  spamminess.

To expand on this: I believe the proposal is implemented by several 
major email clients (Thunderbird is one of them).

[For people not familiar with IMAP: IMAP keywords is a basic IMAP 
feature implemented by majority of IMAP servers I've seen.]

---

From: George Michaelson <[email-masked]>
Date: Mon, 6 Jul 2009 08:08:28 +1000

On 06/07/2009, at 5:03 AM, Iljitsch van Beijnum wrote:

>> local baysian junk folder, remove all non-spam (and flag the  
>> senders as white-listed if need be), and request the rest to be  
>> flagged as spam back on the IMAP backed MS.
>

I think thats a side effect. you depend on the server having a rule  
which says that the folder of a specific name defines spam, and that  
moving mails to and from it makes things happen.

I'm not driven to side ffects. I like first class behaviour. I think  
that is fine, in its own way, but I'd like it to have a way to say it  
without doing that.

for instance. what if I run TWO email accounts, off different imap  
servers. I'd like the JOIN of their state to be used to mark senders  
white or black listed sometimes.

> server could give something a spam score of 2 and the client also 2  
> and together that would be 4 so the message is presumed to be spam  
> (assuming the spam threshold is 3), but in a binary system no spam  
> OR no spam = no spam.

I want to start simple. Binary state spam/ham and white/black flagging.

if I say its spam, then it should be spam. In what sense does the MS  
know better than me?

If I say its whitelisted, in what sense does the MS know better than me?

-G

---

From: Ned Freed <[email-masked]>
Date: Sun, 05 Jul 2009 17:40:31 -0700 (PDT)

> already? That's what my client does.

Please explain how moving a message to a random, arbtirarily named folder tells
the server anything useful. People use all sorts of different naming schemes
for folders, and it's common for clients to provide lots of flexibility in this
area.

The obvious way to make this work in IMAP is with a standardized folder
annotation saying "messages put in this folder were flagged as spam". Then
moves of messages to the folder with this annotation provide the server with
the necessary information that this message is considered to be spam by the
user. And by the same token, moving a message from this folder can be construed
as the message not being considered spam after all.

It isn't perfect, but as long as this is taken as an indicator, not as an
absolute statement, it should get the job done.

> spam threshold is 3), but in a binary system no spam OR no spam = no

There's a place for fractional scoring in spam filtering, but this isn't
it. A user doesn't think "this message is 90% likely to be spam", they
either think it's spam or it isn't.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Mon, 06 Jul 2009 10:39:40 -0700 (PDT)

> feature implemented by majority of IMAP servers I've seen.]

The problem with using keywords for this is that you are now dependence on your
client supporting that particular set of keywords and their associated
semantics. Had we standardize such a keyword set 10 or so years ago this
approach miight have had a chance, but the client base is now too large and
too difficult to update.

This is why I think folder annotations are a better bet. You mark the spam
folder and use that marking to attach meaning to the movement of messages into
and out of that folder. It's not perfect, but it has the key advantage that it
works with any client (OK, any client that can use folders) once the annotation
is in place.

				Ned

---

From: Michael Kirkham <[email-masked]>
Date: Mon, 6 Jul 2009 10:26:10 -0700 (PDT)

On Sun, 5 Jul 2009, Ned Freed wrote:

> Please explain how moving a message to a random, arbtirarily named 
> folder tells the server anything useful.

Seems to me this is up to the server configuration and software running on 
the server.  I don't see why you need a specific contract between IMAP 
client and server for flagging spam, and it's of limited usefulness.  For 
one thing, it was pretty painless for me to set up cron jobs to suck in 
whatever messages I save to certain folders (that could have been named 
anything) for training my spam filter.  I also have other cron jobs set up 
to do other things with email completely unrelated to spam that also 
simply involve saving to a particular folder (e.g. a tickler file for 
email--I move messages to folders named for a month or day, and cron jobs 
automatiacally reinsert them back in my inbox later).  That kind of thing 
isn't addressed by having something spam-specific, but as far as my email 
client is concerned, they're flagged the same way.

Any additional detail or instructions could likely be provided by adding 
headers to the message being saved, if the client allows it.

Just my two (not seeing the point) cents.

--
Michael Kirkham
President & CEO
Muonics, Inc.
http://www.muonics.com/

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 6 Jul 2009 12:44:14 -0500

On Sun, Jul 05, 2009 at 05:40:31PM -0700, Ned Freed wrote:
> folder tells the server anything useful. People use all sorts of
> different naming schemes for folders, and it's common for clients to
> provide lots of flexibility in this area.

In particular the word 'spam' is really an English-language word.  L10N
would demand that the name of this folder not be 'spam' in all locales,
or at least that clients know how to map that folder name to the user's
language(s).  But even if we go for the latter, having special folder
names means denying or complicating the use of those names in other
languages for other purposes; it's bad enough we have one well-known
folder name (INBOX), let's have no more.

> The obvious way to make this work in IMAP is with a standardized
> folder annotation saying "messages put in this folder were flagged as
> spam". Then moves of messages to the folder with this annotation
> provide the server with the necessary information that this message is
> considered to be spam by the user. And by the same token, moving a
> message from this folder can be construed as the message not being
> considered spam after all.

+1

Or even just a message flag/annotation.  I understand the "move junk
mail to the junk mail folder" metaphor for flagging junk mail as such,
but one might want to flag junk mail and still move it to an "archived
mail" folder.  I don't find my argument for a per-message flag terribly
convincing, so I won't push it.

> It isn't perfect, but as long as this is taken as an indicator, not as
> an absolute statement, it should get the job done.
> 

+1

Fractional scoring by the server, or by software running on the client
can be useful, but the users themselves will generally make binary spam/
not-spam decisions.

Nico
--

---

From: Ned Freed <[email-masked]>
Date: Mon, 06 Jul 2009 11:00:24 -0700 (PDT)

> On Sun, 2009-07-05 at 17:40 -0700, Ned Freed wrote:
> > folder

> This is just trying to turn a folder into a keyword, but without full
> keyword semantics.

Actually, it's a hack t work around the lack of client support, which is why
I didn't propose a keyword-based mechanism.

> The most obvious flaw is that it constrains the
> client to storing all the spam in one bucket.

If there's a reason multiple folders can't be marked this way I sure don't
know what it is.

> It doesn't allow for (say)
> sorting junk by type into different folders (perhaps on different
> servers). A message's 'state of spamminess' is an attribute of the
> message, properly denoted by keywords that stick to (and with) the
> message.

I see no reason why this couldn't be done, if you really want to.

> If you want this functionality (and I think it's a useful service for
> servers to provide) you need to implement two keywords: $spam and
> $nospam (these are just example names). Setting $spam on a message would
> instruct the server to pass the message off to the filtering component
> as spam; setting $nospam would instruct the server to pass the message
> off to the filtering component as valid (non-spam). The two keywords are
> exclusive, and it would be up to the server to decide what action to
> take if the client tries to light up both at once (e.g. unset the other
> flag and take the corresponding action, reject the command, or silently
> ignore the request).

Again, the problem with using keywords is now you have to get a significant
fraction of clients on board before it's useful. I'd be willing to go along
with a keyword-based approach if I could convinced that's going to happen, but
everything recentlts has supported exactly the opposite conclusion.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 06 Jul 2009 19:14:38 +0100

Arnt Gulbrandsen wrote:

> Ned Freed writes:
>> spam".
> I think what Thunderbird and Apple Mail do is to set specific message 
> flags on messages in whatever folders they happen to be in: $Junk if 
> the user said a particular message is spam, $autojunk if tb 
> autonomously decided that message is spam, $nonjunk and $autononjunk 
> for the opposite. (I may be wrong about the exact flag names.)

Something like that, yes.

> This was specified in some expired draft,

Which was mine.

> which IMO now has been frozen by deployment. Apple Mail and 
> Thunderbird add up to >10% for me. Time for an informational RFC, 
> perhaps. "Widely-used IMAP flag names"?

Yes. I think even more important to create the IANA registry for keywords.
Do you want to help finish my draft?

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 06 Jul 2009 19:29:20 +0100

Lyndon Nerenberg wrote:

>But if your concern is adoption rates, a keyword-only solution will get
>wider take-up than something that requires support for annotations. (And
>from a purely engineering standpoint, the simplest solution wins.)
>  
>
+1.

---

From: Ned Freed <[email-masked]>
Date: Mon, 06 Jul 2009 11:07:18 -0700 (PDT)

> client and server for flagging spam,

The primary reason is so that the server can use the information to
automaticallly refine it's filtering policies on a per-user basis.

Other uses include, but are not limited to, setting expiration and arhival
policies.

> and it's of limited usefulness.

On the contrary, such a mechanism would be HUGELY useful. We have any number of
customers begging for it, as a matter of fact.

> For
> anything) for training my spam filter.

Are you serious? Cron jobs? Do you have any idea how minute a fraction of the
users of email even know what a cron job is, let alone know how to set one up?

The minute you start talking about setting up your own cron jobs, it  pretty
much goes without saying that you're operating entirely outside the space where
99+% of the use of IMAP takes place.

> I also have other cron jobs set up

And no doubt you also tweak SpamAssasin rules, set up milters, etc. etc. This
is far beyond the capability of most users.

				Ned

---

From: Michael Kirkham <[email-masked]>
Date: Mon, 6 Jul 2009 11:56:35 -0700 (PDT)

> automaticallly refine it's filtering policies on a per-user basis.

Which is what my setup does (assuming I use the folders I set aside for 
flagging spam/ham).

> Other uses include, but are not limited to, setting expiration and arhival

But I don't see what these things have to do with IMAP specifically. 
Consider Pine, for example, which saves a special message (which Pine 
hides from the user) to folders it interacts with, having some 
Pine-specific details about the folder/messages in it.  If what you want 
is to be able to set folder-specific policies, perhaps what you should be 
arguing for is standardizing an extension to the mbox (or other mail 
storage) format that does something similar.  A standard set of header 
fields in this hidden message, plus whatever client-specific stuff some 
other client might want to add, etc.

> On the contrary, such a mechanism would be HUGELY useful. We have any 
> number of customers begging for it, as a matter of fact.

I think you misunderstood what I was pointing at not being useful.  If 
these customers don't understand what a cron job is, they probably haven't 
much clue about IMAP either, other than plugging in the settings their IT 
person tells them to so their email just works.

I've no doubt that your customers want to be able to, from an arbitrary 
client and server, flag a message as spam (or not) to be handled as such 
by the server.  What I believe is less useful is addressing that 
requirement through something IMAP and SPAM specific.

My point wasn't that users should just set up cron jobs and custom scripts 
to do it as I've done.  Rather, my point was that it doesn't take an IMAP 
extension to accomplish these things, since it can just be done with 
folders, and that the solution would be far more useful if the domain of 
what you're addressing isn't focussed purely on spam/ham stuff.

I was also trying to address the argument that using a "special" folder 
doesn't work because of needing to reserve a folder name with I18N issues 
and such.  I'm not arguing that saving to specific folders is the ideal 
solution, but whatever them method you've got to have a server and client 
(or user) both set up with software and a config to use it, and it could 
(not necessarily should) be just as simple as a client config option for 
"this folder is spam" the IT person provides (with other settings for 
email to "just work") after setting up the server.

You bring up SpamAsassin.. Well, as far as I know SpamAssassin doesn't 
know anything about IMAP.  How does extending IMAP help an IMAP client 
feed a message back to SA as spam/ham?  It would seem to me (and, 
admittedly, I'm no IMAP expert) that you've still got to flag the message 
itself somehow for later processing.  It sounds like you want IMAP to do 
that.  But what would IMAP do, beyond tagging the message with an extra 
header, or tagging the folder itself somehow (say, folder name--in which 
case that's where we are now--or through headers in a hidden message, a la 
Pine)?

> it pretty much goes without saying that you're operating entirely 
> outside the space where 99+% of the use of IMAP takes place.

I believe that's my point. :)

--
Michael Kirkham
President & CEO
Muonics, Inc.
http://www.muonics.com/

---

From: Dave Cridland <[email-masked]>
Date: Mon, 06 Jul 2009 21:16:30 +0100

On Mon Jul  6 18:53:41 2009, Lyndon Nerenberg wrote:
> > annotation saying "messages put in this folder were flagged as  
> spam".
> > Then
> > the necessary information that this message is considered to be  
> spam
> > by the
> > user. And by the same token, moving a message from this folder  
> can be
> > construed

+1.

It's fundamentally no different from clients trying to replace the  
\Deleted keyword with a Trash mailbox, and has all the related  
issues, which we presumably all know. (Or else can find the attempt  
at documenting them I once tried).

This is compounded by the state being a tristate rather than binary -  
messages are "Ham", or "Spam", or else "The user has not expressed  
any decision". Should I move all my messages into another folder to  
mark them as good ham?

Dave.

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Mon, 6 Jul 2009 21:55:39 +0200

On 6 jul 2009, at 20:07, Ned Freed wrote:

>> it was pretty painless for me to set up cron jobs to suck in

> users of email even know what a cron job is, let alone know how to  

Hey, everyone needs a hobby.

I have my procmail set up to check whether I have an IMAP session with  
the server. If so, I let the client do the mail filtering, if not, the  
server does it. That way, the unread counters for folders get updated  
when the client is connected, but the client doesn't have to move  
large volume mailing list traffic to the correct folder, costing me  
expensive wireless data traffic and/or slowing things down.

About folders for spam: Apple's Mail lets the user select "use this  
folder for... {sent, drafts, junk, trash}, which is nice if you used a  
different client before that uses a different folder naming scheme for  
these things. Works well enough although a little more automation  
wouldn't hurt.

The point that working with folders can be done with existing software  
(I later train spamassassin with what Mail put in the "junk" folder,  
even though SA doesn't know about IMAP) is an important one. Requiring  
keywords that are hardcoded in clients and servers means deployment  
will be extremely slow, if it ever happens.

---

From: Tony Finch <[email-masked]>
Date: Mon, 6 Jul 2009 21:25:32 +0100

On Mon, 6 Jul 2009, Michael Kirkham wrote:
> > Please explain how moving a message to a random, arbtirarily named folder
> > tells the server anything useful.
>
> Seems to me this is up to the server configuration and software running on the
> server.  I don't see why you need a specific contract between IMAP client and
> server for flagging spam

So that clients can have user interfaces that expose the functionality
in a more friendly manner.

Tony.
-- 
f.anthony.n.finch  <[email-masked]>  http://dotat.at/
GERMAN BIGHT HUMBER: SOUTHWEST 5 TO 7. MODERATE OR ROUGH. SQUALLY SHOWERS.
MODERATE OR GOOD.

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 06 Jul 2009 22:06:43 +0100

Iljitsch van Beijnum wrote:
 [...]

> folder for... {sent, drafts, junk, trash}, which is nice if you used 
> a  different client before that uses a different folder naming scheme 
> for  these things. Works well enough although a little more 
> automation  wouldn't hurt.
>
> The point that working with folders can be done with existing 
> software  (I later train spamassassin with what Mail put in the "junk" 
> folder,  even though SA doesn't know about IMAP) is an important one. 
> Requiring  keywords that are hardcoded in clients and servers means 
> deployment  will be extremely slow, if it ever happens.

I don't think your argument in favour of folders (mailboxes) is valid. 
An IMAP server don't need to treat $Junk/$NotJunk any specially, it just 
needs to be able to store arbitrary IMAP keywords. Most servers already do.
A recent Thunderbird can also be configured to set arbitrary IMAP 
keywords. (It also recognized $Junk/$NonJunk natively, but this is 
besides my point).

---

From: Nicolas Williams <[email-masked]>
Date: Mon, 6 Jul 2009 15:31:43 -0500

> in a more friendly manner.

More specifically: so that users and their clients don't need to know
specific details of the server's configuration.

---

From: Lyndon Nerenberg <[email-masked]>
Date: Mon, 06 Jul 2009 11:53:41 -0600

This is just trying to turn a folder into a keyword, but without full
keyword semantics. The most obvious flaw is that it constrains the
client to storing all the spam in one bucket. It doesn't allow for (say)
sorting junk by type into different folders (perhaps on different
servers). A message's 'state of spamminess' is an attribute of the
message, properly denoted by keywords that stick to (and with) the
message.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 6 Jul 2009 19:50:08 +0200

This was specified in some expired draft, which IMO now has been frozen 
by deployment. Apple Mail and Thunderbird add up to >10% for me. Time 
for an informational RFC, perhaps. "Widely-used IMAP flag names"?

Arnt

---

From: Lyndon Nerenberg <[email-masked]>
Date: Mon, 06 Jul 2009 12:26:01 -0600

> If there's a reason multiple folders can't be marked this way I sure

You're right -- I didn't parse what you were saying correctly the first
time around.

--lyndon

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Mon, 6 Jul 2009 22:56:19 +0200

Lyndon Nerenberg writes:
> But if your concern is adoption rates, a keyword-only solution will 
> get wider take-up than something that requires support for 
> annotations.

Given that two significant clients (say two of the top five) use the 
keyword approach today, I think it's safe to say «has received wider 
take-up», not «will get wider».

Arnt

---

From: George Michaelson <[email-masked]>
Date: Tue, 7 Jul 2009 08:51:10 +1000

> works with any client (OK, any client that can use folders) once the  

I have two IMAP servers, one google, one not.

how do I synchronize state between them. After all, If I (and I do)  
sometimes have to send from one, to reply to mails from the other  
(mail clients are bad at selecting sender account) it would be useful  
to have at least white and blacklist sender synchronization between  
them. Or, I could be sharing explicit spam tagging information to  
improve BOTH of their tracking. Do I really have to pass the entire  
messages between imap servers?

An explicit extension/flag/keyword would be unquestionably more  
efficient use of the network.

And, removing the mails from the spam folder in your methodology  
implies saying 'its not spam' -so the server I move them FROM thinks  
they've been un-tagged.

Again, I say what I said to Ijllitch: I don't like depending on side- 
effects. I think explicit messaging is better.

-George

---

From: Ned Freed <[email-masked]>
Date: Mon, 06 Jul 2009 17:42:58 -0700 (PDT)

> I have two IMAP servers, one google, one not.

That's a problem in and of itself, because last time I looked gmail is rather
limited in its support of IMAP.

> how do I synchronize state between them.

By "state" you appear to be referring to spam filtering state. If so, the short
answer is you can't possibly do it.

> After all, If I (and I do)
> them.

Sorry, that's simply not realistic. The number of different spam filtering
mechanisms in use is much too large, and the lack of semantic equivalents
across them all makes this an intractable problem.

> Or, I could be sharing explicit spam tagging information to

If you want to train the server that doesn't have the message that you do,
or don't regard, it as spam, moving the entire message is the only option.

I will note that moving messages from one IMAP server to another without
having to download and upload them would be possible by relaxing the
relative URL restriction in the CATENATE extension defined in RFC 4469.

But sending misclassified messages to multiple servers is not going to get them
synchronized. Again, there are a myriad of approaches to spam filtering with
hugely different semantics. In fact it is entirely possible that you'll cause
their behavior to diverge instead of converging.

> efficient use of the network.

This makes no sense at all. Setting a flag associated with a message stored
on one server is not going to be visible to the other in any way. So the
flagging approach offers no solution to the multiple server problem at all.

The folder annotation approach doesn't work any better, of course.

> implies saying 'its not spam' -so the server I move them FROM thinks

No, not really. Since there's no facility in IMAP to move a message between
servers, the way you have to do it is to FETCH the message from one server and
APPEND it to the other. In constrast moving a message from one folder to
another on a single server is done with a COPY/SETFLAG sequence. Entirely
different.

> effects. I think explicit messaging is better.

I don't especially like depending on side effects either, but when the choice
is between a mechanism that works with the vast array of currently deployed
clients and one that requires deployment of new client software everywhere to
stand even a chance of wortking, I'm strongly inclined to pursue the former.

Again, had we standardized flags for marking messages as spam 10 years ago, I'd
be fully in favor of using them for this additional purpose because there would
be plenty of deployed client support by now. But we didn't, there isn't, and I
don't hold out a lot of hope for _anything_ deploying with reasonable speed in
the IMAP client space right now.

				Ned

---

From: George Michaelson <[email-masked]>
Date: Tue, 7 Jul 2009 11:28:53 +1000

hang on.

The MS has the message. It presumably has some held state of it,  
specific to its implementation. I don't want deep knowledge of that  
model, I really just want to be able to override it with my explicit  
knowledge.

I simply want to pass the toggle:
	
	If you class this as spam, please class as ham
or
	if you class this as ham, please class as spam

what it means, semantically, is of course MS specific. I don't care  
how its held, or what the effect on the weighting is, in the re-class  
as spam. In the class as ham, I care quite a lot that its sender is  
removed from hard blocks of course.

Yes, I see that moving this state between servers requires the entire  
message flow. But, if we forget that for now, and just consider that I  
have some mail in my local client cache, which I feel is now worthy of  
a state change, why do I have to pass full-state baysian model  
information? I just want to tell an MS "change your own flags to match  
my flags" in the hard sense: yes-or-no, not some qualitative weighting  
model. How it acts, is up to it. I don't see why I need a complex flag  
model to say what I think, as the consumer of the mail.

If I moved off SPAM/HAM to white/black on sender, would that be easier  
for you to consider?

-G

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Tue, 7 Jul 2009 09:21:11 +0200

On 6 jul 2009, at 19:53, Lyndon Nerenberg wrote:

> A message's 'state of spamminess' is an attribute of the

That's nice in the abstract, but why do we have servers classify  
messages as spam to begin with? In order to not have to look at them,  
or to be able to look at them when it suits us to clean out the spam  
rather than in between mail that we actually wanted to receive.

So moving spam to a separate folder is not something that happens  
because of lack of a better mechanism, but the thing that we want to  
happen 98% of the time anyway.

> $spam and $nospam

There are more states than that:

1. automatically flagged as spam
2. automatically flagged as non-spam
3. manually flagged as spam
4. manually flagged as non-spam

Where 1 and 2 / 3 and 4 are mutually exclusive but 1 + 4 and 2 + 3  
should trigger retraining of baysian filters etc.

---

From: Ned Freed <[email-masked]>
Date: Tue, 07 Jul 2009 06:54:49 -0700 (PDT)

> rather than in between mail that we actually wanted to receive.

Actually, the primary reason servers classify messages as spam is so they
can avoid having to deal with them. If the overhead could be ignored no
server operator would bother - spam filteringh is expensive and difficult.
The reason it's done is the alternate of receiving and dealing with the
those spam messages, with all the ramificaations, is even more expensive
and difficult.

This more than anything is why a score and not a yes/no verdict is important. A
score lets you do stuff like reject messages clearly found to be spam at the
SMTP level, while letting likely but less certain to be spam messages through
to specially designated spam folders or whatever.

> happen 98% of the time anyway.

Sometimes yes, sometimes no.

> should trigger retraining of baysian filters etc.

It's not the flag state of the message that triggers retraining,  it's
the change of flag state (or folder locaation). Your proposal is unnecessatily
complicated.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Tue, 07 Jul 2009 07:22:42 -0700 (PDT)

> from a purely engineering standpoint, the simplest solution wins.)

My concern isn't adoption rate. Rather, my concern is, as someone who provides
server software, how I can offer a better user experience in regards to spam.

Suppose I implement spam filter retraining based on setting of flags. how well
this works, or whether it works at all, depends entirely on the client people
use. And past experience on client uptake of new features like this is best
described as "lousy".

Now suppose I implement this with folder annotations. I'm in business
immediately. Things just work no matter what client people use.

There is, of course, the issue of setting the annotation on the proper folder
and  having a client to do that. But this isn't as much of a problem as it
sounds. Since the annotation only has to be set once, it can be done through a
simple web page, or better still, make it an option in the webmail interface.
(Everyone has a webmail interface these days.) And in many cases even this is
unnecessary, because many service providers predesignate a folder as the spam
folder.

The only way I could get behind a flag-based solution is if there really was an
common set of ad-hoc flag choices already in use by all the major clients - not
a couple, all. But I don't believe that's the case.

We've had to choose between client and server side solutions many times in the
past, and in almost every case the conclusion has been server side solutions
are a deployment win. I really wish this weren't true - I wish clients would
pick up new capabilities in a more timely fashion - but that's just not how
things work.

				Ned

---

From: Ned Freed <[email-masked]>
Date: Tue, 07 Jul 2009 07:42:42 -0700 (PDT)

> specific to its implementation.

Actually, that's highly unlikely. There is some amounts of state IMAP requires
that's associated with the message - flags/keywords, annotations, some remnants
of the envelope. But beyond that there's little point in keeping a bunch of
implementation-specific state. Why? Because there's no way to expose such state
through the IMAP protocol. So when messages are FETCHed the state stays behind.
And more to the point, when a client moves a message, implementation-specific
state tends to get lost, because the client doesn't know how to access it in
order to move it.

The result is that in most csaes IMAP servers don't store much beyond the
things the standard requires them to store. The differences between servers lie
in how the standard stuff is stored and to what extent metainformation, like
body structured, is cached instead of being recomputed.

> I don't want deep knowledge of that

> 	if you class this as ham, please class as spam

This is exactly what we're trying to accomplish. The question is what mechanism
to use to do it.

> removed from hard blocks of course.

I'm afraid you cannot count on that happening either. The problem is that
such blocks are often outside the control of the service provider to change.
In many cases the provider has subscribed to some sort of spam filtering
service.

Whitelisting may be an alternative in such cases, but you have to be careful
lest the amount of whitelisting information grow to the point where it is
unmanageable. (I've seen this happen.)

> information?

You appear to be arguing against a straw man of your own construction here,
because AFAIK nobody has suggested such a thing.

> I just want to tell an MS "change your own flags to match
> model.

The only "weight" people have mentioned in any of this is storing a spam score
rather than a yes/no indicator. I've already responded that while there is a
place for such information in spam handling, it makes no sense in this
particular context.

> How it acts, is up to it. I don't see why I need a complex flag

Again, you appear to be arguing against a proposal nobody has made. (Perhaps
some proposal was made in an earlier thread not on the imapext list.) I'll also
point out that flags are binary by definition in IMAP - you cannot implement
even a single-valued spam score using them. If you want to store a reviseable
per-message score, you have no choice but to use message-level annotations.

> If I moved off SPAM/HAM to white/black on sender, would that be easier

Whitelisting and blacklisting is actually  a fairly different problem.

				Ned

---

From: Cyrus Daboo <[email-masked]>
Date: Tue, 07 Jul 2009 11:15:04 -0400

Hi Ned,

> immediately. Things just work no matter what client people use.

Actually you need to consider what the user experience is likely to be on 
the client side. There are several things to consider: for spam messages 
users will typically want to delete them from their regular mailbox. With 
ham messages they will typically want to leave the message in place (or 
file them away after processing). In each case I certainly would not want 
the spam messages or an extra copy of the ham messages to reside on my 
server consuming quota or other resources, or being indexed as part of a 
server-wide search etc.

With that in mind lets look at how each solution might work client-side:

1) Keywords: for a spam message the user sets the spam keyword, then sets 
the deleted flag (and then may explicitly expunge if the client has that as 
an option as opposed to hide-deleted-messages with lazy expunge). For a ham 
message the user just sets the ham keyword. i.e. two user operations for 
spam, one for ham.

2) Mailbox (I am going to assume that the client has both a move and copy 
command available - the former implicitly setting the delete flag after the 
IMAP COPY command): for a spam message the user moves the message to the 
spam mailbox. For a ham message the user copies the message to the ham 
mailbox. NB both of these assume that the server does not actually store 
the spam/ham messages in the spam/ham mailboxes (or perhaps it garbage 
collects them).

Things to note: typically clients don't offer shortcuts "move to mailbox X" 
or "copy to mailbox X" so the mailbox operations will involve some drag and 
drop or menu list selection operation. The same may be true for setting 
arbitrary keywords (though Mail.app and I suspect Thunderbird do have 
buttons for this). Frankly the best user experience would be for the client 
to have spam/ham buttons and commands that would allow the user to do the 
appropriate actions in one click or keyboard shortcut operation - that is 
certainly important if the level of spam is moderate to high.

So at the end of the day, for the best (and frankly simplest) user 
experience clients do have to change for both the keyword and move to 
mailbox approaches. Now for keywords that change does have to be immediate 
if arbitrary keyword support is not already present in the client, whereas 
for move to mailbox, or clients already supporting keywords, that change 
can be done over time - though as Ned points out client changes tend to 
happen on geological time-scales!

So other approaches to consider:

1) An extension with commands SPAM and HAM to explicitly tag one or more 
messages.
2) A "forward to spam/ham" option that specifies special email addresses 
for the user where spam/ham can be forwarded to trigger training.
3) An IMAP URL handed off to the SPAM system with URLAUTH to allow that to 
pull details of the message for training. (This could be tied to (2) in 
that forward-without-download could be used in (2) with URLAUTH etc).

These of course all require new client/server behaviors and thus are 
probably non-starters, but they should not be dismissed entirely.

-- 
Cyrus Daboo

---

From: Ned Freed <[email-masked]>
Date: Tue, 07 Jul 2009 08:43:46 -0700 (PDT)

> certainly important if the level of spam is moderate to high.

The presence of spam/not-spam buttons in existing clients is an important
factor I hadn't really considered, but I think it has implications that go a
lot further than you let on here.

Simply put, if enough popular clients already have such buttons, and those
buttons do different things in different clients, I think we're pretty much
screwed. Given the presence of such buttons you're not going to be able to get
users to do something different to deal with spam, whether it's adding a
keyword or annotation or performing a folder move.

Our only recourse with such clients in the mix is to write a specification for
what such buttons should do in terms of signalling the server and then hope
that clients actually adopt whatever that specification says. And frankly, if
there's some degree of commonality in what those buttons do in existing clients
and it can be made to work, we probably should adopt that approach no matter
what it happens to be.

> happen on geological time-scales!

I agree completely that a keyword-based approach has the potential to offer
a better user experience. The question is whether the possibility of offering
a better user experience at some distant point down the road is worth giving
up on a more immediately deployable solution.

But as I indicated above, now that you've brought up the important point of
existing clients having spam classification buttons, that's the real gorilla in
the room we have to deal with.

> for the user where spam/ham can be forwarded to trigger training.

I believe this is what the buttons I've seen on at least a couple of
clients already do. If that's indeed the case, it's worth considering.

> probably non-starters, but they should not be dismissed entirely.

I'm actually not sure (2) does with all clients, but I agree regarding (1) and
(3).

				Ned

---

From: Lyndon Nerenberg <[email-masked]>
Date: Tue, 07 Jul 2009 11:31:13 -0600

> should trigger retraining of baysian filters etc.

The keywords I'm describing are intended to tell the server to process
the flagged message through its filters (i.e. train on the message). It
isn't relevant to the proposal whether a human sets them, just what the
server's behaviour is in their presence.

--lyndon

---

From: George Michaelson <[email-masked]>
Date: Wed, 8 Jul 2009 09:42:58 +1000

I have no desire to argue against my own proposal! sorry if I  
misunderstood the tenor of some comments to this suggestion.

The original posting asked about two things

1) spam/ham

2) white/black

so far, people seem to be focussing on 1). I acknowledge problems with  
1), I appreciate that people are trying to address 1) as a possibility.

I asked about 2). I asked you if, ignoring 1), it was possible or  
plausible to send white/black information about sender, if 1) was hard  
or intractable.

Does that make more sense? I do realize white/black listing is a  
different PROBLEM to spam/ham marking, but neither of them currently  
have support directly in imap, and I imagine both of them would use  
IMAP extensions to pass information client-server.

Sorry if I'm being unclear. I'm not subscribed to the IMAPEXT, or  
other lists, I have very little context for this beyond wanting a  
facility to cover a problem I see as a mail consumer. I'm not in the  
business of mail system development.

I appreciate people spending time discussing this. I still think there  
is a tractable problem here, some class of IMAP extension will permit  
better information flow from the client side.

cheers

-G

---

From: Ned Freed <[email-masked]>
Date: Tue, 07 Jul 2009 20:02:51 -0700 (PDT)

> plausible to send white/black information about sender, if 1) was hard

Of course it's possible, but this has no business being in IMAP. IMAP is a
message access protocol, not an address handling protocol. It is also already
quite complex, and extending its function to include what amountss to a
specialized address book capability is a really bad idea IMO.

In fact a pretty good case can be made that it's inappropriate to use 
IMAP for spam/ham classification, but at the spam/ham classification is
intrinsicly tied to messages in the store. The same cannot be said for
address white and blacklisting.

> IMAP extensions to pass information client-server.

The place where I'd like to see white and black listing done is in CARDDAV, not
IMAP.

				Ned

---

From: Iljitsch van Beijnum <[email-masked]>
Date: Wed, 8 Jul 2009 08:08:05 +0200

> The place where I'd like to see white and black listing done is in  

I never felt comfortable uploading all my contact info to Google. (May  
be somewhat strange since they do have all the mail sent to one of my  
email addresses.)

So I think white/blacklists shouldn't be complete contact info, but  
rather, just lists of hashes of email addresses.

---

From: Ned Freed <[email-masked]>
Date: Tue, 07 Jul 2009 23:19:32 -0700 (PDT)

> be somewhat strange since they do have all the mail sent to one of my

That's fine, but this doesn't translate into a need to put an addressbook
capability in IMAP.

> rather, just lists of hashes of email addresses.

So use CARDDAV to only upload the stuff that's needed to Google. Problem
solved.

I'll also note that addressbook programs have had facilities for exporting
subsets of the information they contain pretty much from the start. So doing
this  is hardly entering new territory.

And most users want an authoritative address book with all their contact
information in one place. I know I do - it's much too confusing otherwise.

				Ned

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 08 Jul 2009 11:08:28 +0100

+1

> The place where I'd like to see white and black listing done is in 

Or LDAP :-).

There is also a Sieve extension for making use of whitelists/blacklists 
during mail delivery. See draft-melnikov-sieve-external-lists-02.txt.
It can work with CARDDAV, LDAP and other things.

---

From: Anil SRIVASTAVA <[email-masked]>
Date: Tue, 07 Jul 2009 20:50:40 -0700 (PDT)

On 2009-07-07/20:02 [-0700], [email-masked] [Ned Freed] wrote:

> Of course it's possible, but this has no business being in IMAP. IMAP is 
> a message access protocol, not an address handling protocol. It is also 
> already quite complex, and extending its function to include what 
> amountss to a specialized address book capability is a really bad idea 
> IMO.
> 
> In fact a pretty good case can be made that it's inappropriate to 
> useIMAP for spam/ham classification, but at the spam/ham classification 
> is intrinsicly tied to messages in the store. The same cannot be said 
> for address white and blacklisting.
> 
> The place where I'd like to see white and black listing done is in 
> CARDDAV, not IMAP.
>

+1 

Anil
 
> 				Ned

_______________
Anil SRIVASTAVA  |  [email-masked]  | Sun Microsystems, Inc

---

From: Ned Freed <[email-masked]>
Date: Wed, 08 Jul 2009 11:54:05 -0700 (PDT)

> > CARDDAV, not

> Or LDAP :-).

That's actually how we do it now. We're hoping to move to CARDDAV in the
future.

> It can work with CARDDAV, LDAP and other things.

Interestingly, this extension in its current form is not compatible with
at least one of the proposals that was made here. The issue is one I've
raised before - the fact that :list is separate argument and not a match type.

Where there is clearly some utility in being able to say stuff along the
lines of:

    header :contains :list "subject" "tag:dirty-word-list"

The problem is that in order for this to work the list has to be enumerable.
Not all lists are enumerable, and even some that are are so large even though 
oyu can enumerate them in theory you can't afford to do so in practice.

One use case where this matters is when the list is a set of hashed values. The
way you find if something is "on the list" is to hash it and see if that value
appears. And even if the input string is short enough that you can enumerate
all the unique substrings, how about :matches and :regex? Good luck with
those.

Hashed lists have in fact been proposed in the present discussion as a means of
avoiding giving your address whitelist to the mail server. I happen not to
think this is a useful thing to do for a variety of reasons, mostly having to
do with address canonicalization (or lack thereof), but there are other
use cases where hashed lists make more sense.

So, although it reduces functionality, I believe :list should be a match type
and the underlying comparison type that's done should be a property of the list
itself.

				Ned

---

From: George Michaelson <[email-masked]>
Date: Thu, 9 Jul 2009 08:27:39 +1000

> The place where I'd like to see white and black listing done is in  

so there's only one real question: if it was standardized, would the  
likes of google implement it? How about the client side?

because, unless its in gmail AND in the Apple Mail client,  
Thunderbird, Outlook, its not useful.

-G

---

From: Ned Freed <[email-masked]>
Date: Wed, 08 Jul 2009 17:30:06 -0700 (PDT)

> likes of google implement it? How about the client side?

This is actually one of the reasons why CARDDAV is a better choice to piggyback
on.

RIght now proprietary environments like Exchange/Outlook and Lotus Notes
have a significant advantage over standards-based solutions: They provide
a fully integrated mail/calendar/addressbook client experience.

One reason standards-based solutions have a problem with this has been the lack
of open protocols for accessing and manipulating calendar and addressbook
information.

But this is changing. CALDAV addresses the calendar access issue, CARDDAV does
the same for address books. Once all the pieces are in place, and assuming we
get the details right, this will allow standards-based clients and servers all
to interoperate.

Uptake isn't going to be fast because these are complex protocols and
getting the integration right isn't easy. But for once the incentives
appear to be woring for us, not against us.

> Thunderbird, Outlook, its not useful.

That may be true now, but at some point given sufficient adoption things flip
around to  where not having support for this stuff makes your client or server
not useful.

I note, however, that now that CALDAV is complete we see Apple, Thunderbird,
and gmail all either supporting it or working on supporting it. It's a little
much to expect Outlook to shift from it's currrent proprietary way of doing
all this, but even Microsoft may eventually have no choice but to support
the emerging standards in this area.

				Ned

---

From: Cyrus Daboo <[email-masked]>
Date: Wed, 08 Jul 2009 22:50:22 -0400

Hi Ned,

--On July 8, 2009 5:30:06 PM -0700 Ned Freed <[email-masked]> wrote:

> I note, however, that now that CALDAV is complete we see Apple,
> Thunderbird, and gmail all either supporting it or working on supporting
> it.

For a more comprehensive list of implementations see: 
<http://caldav.calconnect.org/implementations.html>.

> It's a little much to expect Outlook to shift from it's currrent
> proprietary way of doing all this, but even Microsoft may eventually have
> no choice but to support the emerging standards in this area.

In the meantime the folks at <http://zideone.com> have a plugin for Outlook 
that does CalDAV and CardDAV.

-- 
Cyrus Daboo