Subject: [apps-discuss] Review of draft-ietf-appsawg-sieve-duplicate

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 14 Jan 2014 10:06:23 +0100

Hi,

I read this draft earlier and decided to ignore it. There's nothing
terribly bad about it, I wouldn't want to block it, but IMO also
nothing terribly good.

I have one real problem and a word nit.

Consider the main use case, from the document's introduction:

   Duplicate deliveries are a common side-effect of being subscribed
   to a mailing list.

I would expect that the draft makes handling that use case simple, and
a well-rounded set of other use cases possible. IMO it does poorly at
that use case.

Here's the first example for the Pigeonhole sieve server, modified to
suppress duplicates rather like how the first example in
sieve-duplicate does:

   require ["fileinto", "envelope", "duplicate"];
   if address :is "to" "[email-masked]" {
     fileinto "Dovecot-list";
   } elsif envelope :is "from" "[email-masked]" {
     fileinto "lists.cipe";
   } elsif anyof (header :contains "X-listname" "[email-masked]",
                  header :contains "List-Id" "Linux User Group Offenburg") {
     fileinto "ml.lugog";
   } elsif duplicate {
     discard;
   } else {
     # The rest goes into INBOX
     # default is "implicit keep", we do it explicitly here
     keep;
   }

If a message is sent to you personally with a cc to cipe-l or lugog,
this sieve script will file it into either lists.cipe/ml.lugg or
inbox, randomly, depending on which copy arrives first. This strikes
me as rather poor handling of the use case.

1. It's not what I want. What I personally want is to have a single
instance of the message filed in all the mailboxes fileinto/keep
specifies for any instance of the message, and the other instances
suppressed. But feel free to disregard this point, after all it's just
my personal desire.

2. It's not deterministic. And that's worse. This draft makes the
lists.cipe/ml.lugog mailbox into a randomly incomplete set of messages
to the relevant list, which isn't much of an improvement over the
status quo.

My nit concerns the word multiple. Four is a multiple of two, but
"several" and "many" are perfectly fine words to mean "integers larger
than one".

I once had a student who met his minimum pagecount by using long words
and hacky tex wordwrapping. He would write "multiple independent
purposes" instead of "several purposes" or "many purposes". Unlike
him, internet-drafts authors have no need to leverage multiple
unnecesssary polysyllabic words in order to meet university-imposed
arbitrary minimum length requirements.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 14 Jan 2014 10:25:47 +0100

Btw. I apologize for either the lateness of my review or its nature.
Pick whichever you prefer ;)

My tonsils and I are going to hospital in two days, to part our ways
forever, so replies may be late for a while.

Arnt

---

From: Stephan Bosch <[email-masked]>
Date: Tue, 14 Jan 2014 11:19:38 +0100

> to the relevant list, which isn't much of an improvement over the

There are two aspects to this problem.

First, this example is flawed. The duplicate test has a side effect: 
apart from testing for a duplicate it also records the identifier (in 
the bare case the Message-ID) for comparison in subsequent deliveries. 
So, if the test is not evaluated, the identifier is never recorded.

This example should address that problem:

if duplicate {
   discard;
} else {

The second aspect is a bigger issue. It is indeed uncertain which copy arrives first: the Cc or the message through the mailing list. The latter will have the List-Id header, the former will not. The above example would therefore still be problematic for the ml.lugog mailing list. I am not certain whether there could be any solution for that, other than not relying on the List-ID header and accepting that the destination folder will contain a mixture of the Cc and the actual mailing list messages.

> "several" and "many" are perfectly fine words to mean "integers larger

So, you want me to vary  a bit? :)

> arbitrary minimum length requirements.

I used the word independent to stress the fact that these applications 
would not influence each other when applied within the same Sieve 
script. I can word that differently, but I doubt it will become much 
shorter. :)

Regards,

Stephan.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 14 Jan 2014 12:24:08 +0100

On Tue, Jan 14, 2014 at 11:19:38AM +0100, Stephan Bosch wrote:
> apart from testing for a duplicate it also records the identifier
> (in the bare case the Message-ID) for comparison in subsequent
> deliveries. So, if the test is not evaluated, the identifier is
> never recorded.

Good point. So you might want to point out that this test has to go
first, then. Unless you want to assume that most reader are smarter
than I am, which I concede is a possibility.

> The second aspect is a bigger issue. It is indeed uncertain which
> copy arrives first: ...

The only way I can think of is to use an action instead of a test.

  merge-duplicates;

If merge-duplicates detects that the current message is a duplicate of
an already stored message, then

 - fileinto and keep act on the old copy, and copy/link that into the
   mailbox specified by fileinto/keep
 - discard discards this new copy and leaves the old copy as is
 - other actions use the new copy
 - all tests act on the new copy

> So, you want me to vary  a bit? :)

Say several if you mean several, many if you mean many.

Multiple is a fashionable word at the moment. Like ask as a noun.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Tue, 14 Jan 2014 12:31:02 +0100

> The only way I can think of is to use an action instead of a test.

No, there are many ways. Variations. But if you want predictable
results, you have to tie it to fileinto/keep somehow. For instance:

  fileinto :deduplicated "ml.lugog";

to either stores the new or widens the mailbox set for the old copy.
Many ways. None of them are particularly appealing, if you ask me.

Arnt

---

From: t.petch <[email-masked]>
Date: Tue, 21 Jan 2014 17:18:09 +0000

Stephan

I previously raised the issue of i18n, or Normalisation, which I still
think needs
more attention in this I-D.

This I-D allows comparisons of the message body while RFC5228, as far as
I can see, only covers the treatment of strings in the headers (s.2.7.2)

So if the body of a message is in base64, then is the comparison based
on

SGVsbG8gTWFuYXYsCgpZb3Ugd3JvdGU6Cgo+IEkgaGF2ZSBiZWVu

or

Hello Manav ...

And where the implementation cannot convert a header to UTF8, then
RFC5228 treats
anything over 127 as voiding the comparison.   So when the subject line
is

=?utf-8?b?5Zue5aSN77yaIHByb3Bvc2VkIGRyYWZ0cyBmb3IgYWxpZ25p?=
 =?utf-8?q?ng_MPLS-TP_PSC_linear_protection_protocol_to_transport_r?=
 =?utf-8?q?equirements?=

does that count as being below 127? is that used or

proposed drafts for aligning MPLS-TP PSC linear protection protocol to
transport requirements

I haven't got an example of messages arriving from the IETF via two
different ISP with different encodings, but suspect that that is only a
matter of time.

Like I said, I think that this topic is much more important now than it
was when sieve first appeared, and that the use of non-ASCII in e-mail
has grown enormously, so something needs to be said, even if it is just
that the results are unpredictable when .... is used, for some values of
....

Tom Petch

----- Original Message -----
From: "Stephan Bosch" <[email-masked]>
To: "Arnt Gulbrandsen" <[email-masked]>;
<[email-masked]>
Sent: Tuesday, January 14, 2014 10:19 AM

> The second aspect is a bigger issue. It is indeed uncertain which copy
arrives first: the Cc or the message through the mailing list. The
latter will have the List-Id header, the former will not. The above
example would therefore still be problematic for the ml.lugog mailing
list. I am not certain whether there could be any solution for that,
other than not relying on the List-ID header and accepting that the
destination folder will contain a mixture of the Cc and the actual
mailing list messages.
> _______________________________________________
> apps-discuss mailing list
> [email-masked]
> https://www.ietf.org/mailman/listinfo/apps-discuss
>

---

From: Stephan Bosch <[email-masked]>
Date: Tue, 21 Jan 2014 19:10:47 +0100

> I can see, only covers the treatment of strings in the headers (s.2.7.2)

The transcoding of the header content to UTF-8 is actually something I
should mention more explicitly by referring to that section.

> SGVsbG8gTWFuYXYsCgpZb3Ugd3JvdGU6Cgo+IEkgaGF2ZSBiZWVu

This extension does not access the message body directly. The only way
to achieve testing against the body is putting (parts of) the message
body in a variable. This can currently only be done using the
extracttext extension (http://tools.ietf.org/html/rfc5703#section-7).
The specification of extracttext states:

"Servers MUST support transcoding of any textual body part into UTF-8 for use with this action. This requires decoding any transfer encoding as well as transcoding from the indicated character set into UTF-8."

The issue you raise is therefore no problem at all.

The use of the extracttext extension is already mentioned in the document.

> anything over 127 as voiding the comparison.

That is a bit of a corner case. I think we can satisfactory solve this
by stating that in the case the implementation is not able to convert to
UTF-8, it should perform an octet comparison (voiding the 127 rule).
Keep in mind that we're not comparing against a user-provided string;
we're basically comparing a message against a copy of itself. I think
intermediaries that change the encoding are pretty rare and are not
likely in the future. And, in any case, implementations can solve this
by implementing proper transcoding to UTF-8 for headers.

> that the results are unpredictable when .... is used, for some values of

I'll make a new version soon, based on the comments we've seen so far
since the latest version. I'll add a little bit of text on the UTF-8
transcoding of headers. I don't think additional text is needed on body
comparisons.

Regards,

Stephan.

---

From: Stephan Bosch <[email-masked]>
Date: Tue, 21 Jan 2014 19:22:09 +0100

Hi Arnt,

> than I am, which I concede is a possibility.

Ok.

>  - all tests act on the new copy

I discussed this briefly with Ned.  The proposed extension is a simple
mechanism for straightforward duplicate elimination in the usual Sieve
context, which is at or around the time of final delivery. We think
eliminating duplicates at the store level, possibly by deleting
previously delivered messages and replacing them with a new version is
out of scope.

So, for your scenario a different new extension would have to be devised.

> Multiple is a fashionable word at the moment. Like ask as a noun.

Ok.

Regards,

Stephan.

---

From: Ned Freed <[email-masked]>
Date: Thu, 23 Jan 2014 14:18:04 -0800 (PST)

> think needs more attention in this I-D.

I strongly disagree. The fact is Sieve has a well defined i18n model and it is
neither necessary nor appropriate to reiterate it in this document.

> I can see, only covers the treatment of strings in the headers (s.2.7.2)

You're looking at the wrong document. The Sieve body test and associated
extensions are specified in RFC 5173. And given that people are now complaining
about redundant text in this document, I don't think reitering how the body
extension, variables, or anything else in Sieve handles i18n matters in this
document is a good idea.

> Hello Manav ...

See Stephan's response.

> proposed drafts for aligning MPLS-TP PSC linear protection protocol to

In the context of this draft, who cares? This draft is about detecting
duplicates. A duplicate is, you know, a duplicate. That means it has the same
content, and unless something very wierd is going on, it's going to use the
same charsets, normalizations, and so on.

Of course there can be cases where message handling differs for the two copies,
resulting in differences cropping up. Sieve is designed to handle most of
these, like body encodings, automatically.

But more exotic those are, and the tricker the duplicate mechanism you've
elected to use is, the more likely it is that you'll get a false negative.

And the reality is there are also cases where message-ids are replaced
wholesale. That breaks the most common form of duplicate testing completely.
And lots of other stuff having nothing to do with i18n will break other kinds
of checks. Not only is duplicate detection always going to be a best effort
sort of thing, the very concept of "duplicate message" has considerable slop to
it.

> different ISP with different encodings, but suspect that that is only a

If you're talking about content-transfer-encodings, that happens all the time.
And the Sieve body extension is designed to deal with it. (Or not, if you want
test such things.)

> that the results are unpredictable when .... is used, for some values of

You're assuming Sieve was developed without fully considering these issues. I
can assure you that is not the case - in fact lots of time was spent working
out how Sieve handles i18n matters. And Sieve was built on top of the
comparators schem first defined for ACAP - a lot of effort when into those,
too.

Now, this is not to say there aren't some open issues. There always are
with i18n stuff. In the case of Sieve they generally have to do with
the semantics of comparators, which don't provide a convenient way
to tease out the normalization part.

The main place where this matters is regular expression tests, but there are
also cases where a normalizing set action would be handy. (We basically 
ended up punting on this; see section 4.1 of RFC 5529 for details of what
we ended up doing.) 

But as I noted above, given the intended application of the duplicate etension,
all of this stuff is far, far, far afield. If you really care about i18n issues
in Sieve, I suggest you start a discussion of those issues on the Sieve list.
Additional  extensions to deal with the remaining issues are always possible.

				Ned

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 24 Jan 2014 10:56:27 +0100

> same charsets, normalizations, and so on.

The very weird thing is list managers decoding c-t-e and sometimes charset, 
adding a list-specific footer, then reencoding.

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 24 Jan 2014 11:15:26 +0100

I wrote:
> The very weird thing is list managers decoding c-t-e and 
> sometimes charset, adding a list-specific footer, then 
> reencoding.

Since this list has a footer, I looked at my own message's header to see 
what changed. The message I bcc'd myself has this content-type:

    Content-Type: text/plain; charset=utf-8; format=flowed

The one I got back from the list has this:

    Content-Type: text/plain; format=flowed

Arnt

---

From: Ned Freed <[email-masked]>
Date: Fri, 24 Jan 2014 05:58:37 -0800 (PST)

> adding a list-specific footer, then reencoding.

My response was in the context of header, not body, tests. Even so, your point
is well taken - duplicate applied to captured body text is not likely to be
your friend.

And these sorts of transformations are especially fun when the body is
text/html, not text/plain.

				Ned

---

From: t.petch <[email-masked]>
Date: Fri, 24 Jan 2014 17:52:09 +0000

Stephan

Thanks for that - you have addressed the points I had in mind.

A separate question, hence the top post.  If this function is used by
sieve script X which is included in both sieve scripts A and B, is there
an internal duplicate tracking list for X; or two, one for A.X and
one for B.X; or ... ?

Tom Petch

----- Original Message -----
From: "Stephan Bosch" <[email-masked]>
To: "t.petch" <[email-masked]>; "apps-discuss"
<[email-masked]>
Sent: Tuesday, January 21, 2014 6:10 PM
> "Servers MUST support transcoding of any textual body part into UTF-8
for use with this action. This requires decoding any transfer encoding
as well as transcoding from the indicated character set into UTF-8."

---

From: Stephan Bosch <[email-masked]>
Date: Fri, 24 Jan 2014 21:44:08 +0100

Hi Tom,

> an internal duplicate tracking list for X; or two, one for A.X and

First of all, this situation is not really relevant to Sieve in its
normal context: each user can have only one main active script, so
either A or B and not both. The new ImapSieve context would be an
exception, but this is out of the scope of this draft and the use of the
duplicate extension is explicitly marked as NOT RECOMMENDED for that
context.

Also, the document talks about Sieve executions and not about the
duplicate test being scoped/limited on a strictly per-script basis. A
Sieve execution is the whole act of running the main active script at
final delivery with all include actions it encounters. So, there will be
only one duplicate tracking list per user. I can add a sentence to
clarify this a little more, but I am not convinced that that is necessary.

Regards,

Stephan.

---

From: Ned Freed <[email-masked]>
Date: Fri, 24 Jan 2014 11:35:34 -0800 (PST)

> an internal duplicate tracking list for X; or two, one for A.X and

In the case of the include extension, there's only one. This is covered
explicitly in the draft:

   The "duplicate" test MUST only check for duplicates amongst message
   ID values encountered in previous executions of the Sieve script; it
   MUST NOT consider ID values encountered earlier in the current Sieve
   script execution as potential duplicates.  This means that all
   "duplicate" tests in a Sieve script execution, including those
   located in scripts included using the "include" [INCLUDE] extension,
   MUST always yield the same result if the arguments are identical.

Note the last sentence.

If you want your included scripts to operate in a different namespace, that's
what :handle is for.

More generally, this all gets very complex in a multiple sieve-per-message
environment. There's a correct answer, one that has been arrived at
independently by multiple implementors, but since such environments are outside
our current standards, I see no reason to worry about them here.

				Ned