Subject: [domainrep] Comments on drafts

From: "David F. Skoll" <[email-masked]>
Date: Tue, 25 Oct 2011 12:04:47 -0400

Hi, all,

I sent these comments directly to Murray; he asked me to forward to the
list.

Comments on draft-kucherawy-reputation-vocab-identity-01:

1) I would add the following assertion:

SENDS-TO-NONEXISTENT-RECIPIENT: The subject identity is associated with
mail attemmpted to be sent to nonexistent recipients

2) All of the assertions are for "bad" behavior.  Should there not be
SENDS-HAM and SENDS-TO-VALID-RECIPIENT assertions?  Or is that implicit
in the numerical value and sample size?

Comments on draft-kucherawy-reputation-model-00:

It's not clear to me how the "overall rating" and "number of data points"
are or should be related.  Here's an example:

Suppose [email-masked] has sent 1000 spams and 200 non-spams.  Suppose
we've seen 1200 attempts to valid recipients and 2000 attempts to nonexistent
recipients.

What is the rating and data-point number for SENDS-SPAM?
I would say the rating should be 0.833 and the data points should be 1200.
But it needs to be made clear that the number of data points considered should
only include those data points specifically indicating an assertion or the
negative of that assertion.  Saying "the number of data points underlying
that score" isn't sufficiently clear, IMO.

Comments on draft-kucherawy-reputation-query-http:

The URI template is interesting, but I can see naive clients doing two
GETs for every lookup.  Also, the full URI Template draft is pretty complex
and possibly overkill.  There should be some wording in there to the effect
that a server SHOULD specify and Expires: header when a client retrieves
a template and that a client SHOULD cache the header until it expires.

Also, it occurs to me that a URI template is likely to be short enough
and unchanging enough to be put into a DNS TXT record. :)  Would we want
to specify that?  It would certainly take care of caching.

If I have any more thoughts, I will send them.

Regards,

David.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 26 Oct 2011 01:33:02 -0700

Splitting up David Skoll's feedback into the documents they reference:

> -----Original Message-----
> From: [email-masked] [mailto:[email-masked]] On Behalf Of David F. Skoll
> Sent: Tuesday, October 25, 2011 9:05 AM
> To: [email-masked]
> Subject: [domainrep] Comments on drafts
> 

I just asked this question on the list where the templates document is discussed, and got back something pretty much along these lines.  I'll add that in the next version.

> to specify that?  It would certainly take care of caching.

Going the route of the well-known URI was the advice I got from one of the template draft's authors, representing the common way these things are done in web applications.  I'm fine with that because I'm an email guy treading in web space which isn't my area of expertise, but doing it in a TXT record would be fine with me if there's consensus that that's better.

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 26 Oct 2011 01:41:03 -0700

> mail attemmpted to be sent to nonexistent recipients

That seems potentially useful.  I can add it to the registration.

> in the numerical value and sample size?

That's something we've discussed, especially in the BoF I think, but isn't reflected in the draft yet.  The range of assertions is currently defined as being in the range [0, 1], meaning you disagree (0) or agree (1) with the assertion to varying degrees.  But we also considered [-1, 1], which as I recall means disagree (0), agree (1), or assert the opposite of (-1) what's being stated in the reply.

Maybe it's time to have that conversation now so that it's on the record.

What say you all?

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Wed, 26 Oct 2011 01:46:57 -0700

> that score" isn't sufficiently clear, IMO.

There are two approaches possible here:

1) This is something that a document creating a reputation application and/or assertions need to define.  This constrains service providers in how they produce their values, but makes transition between two reputation providers in the same space more painless.

2) This is something that a reputation service provider making those assertions defines to consumers in some out-of-band way.  This provides more implementation latitude but requires more caution during such transitions and possibly more client complexity.

It sounds like the general trend is that people would like to see (1), but would that limit adoption by reputation service providers?

---

From: "John Levine" <[email-masked]>
Date: 27 Oct 2011 12:03:10 -0000

>It sounds like the general trend is that people would like to see
>(1), but would that limit adoption by reputation service providers?

Seems to me the main value here would be the ability to switch data
sources without reprogramming, which tells me we need (1).

You can add all the local extensions you want, but for this to work there
needs to be a core vocabulary which always means the same thing no matter
who's saying it.

R's,
John

---

From: Nathaniel Borenstein <[email-masked]>
Date: Sun, 30 Oct 2011 14:59:05 -0500

> that score" isn't sufficiently clear, IMO.

This is a good point.  Would it be sufficient to change that to "the number of data points used to compute that score?" -- nsb

---

From: Nathaniel Borenstein <[email-masked]>
Date: Sun, 30 Oct 2011 14:59:11 -0500

> Maybe it's time to have that conversation now so that it's on the record.

My thinking on this one has been changing a bit.  I'm afraid that if we aren't careful, we'll make it too easy to create nonsense answers, e.g. giving me a high score as both a good guy and a bad guy.  If you make it too easy to do something stupid, people will.  So it would be nice to define the vocabulary in such a way that there were no nonsense-enabling redundancies; a -1 to 1 spectrum might make that clearer.  -- nsb

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Sun, 30 Oct 2011 15:03:45 -0700

I think that's how I've been using that value.  I can tweak the definition accordingly in the next version.

From: [email-masked] [mailto:[email-masked]] On Behalf Of Nathaniel Borenstein
Sent: Sunday, October 30, 2011 12:59 PM
To: David F. Skoll
Cc: [email-masked]
Subject: Re: [domainrep] Comments on drafts

---

From: "John Levine" <[email-masked]>
Date: 31 Oct 2011 01:33:49 -0000

> My thinking on this one has been changing a bit.  I'm afraid that if
> we aren't careful, we'll make it too easy to create nonsense
> answers, e.g. giving me a high score as both a good guy and a bad
> guy.

I'm pretty sure that any language that's sufficiently expressive to be
interesting is sufficiently expressive to have contradictions, and no
quantity of assertions can prevent that, e.g.:

  Is-Male: 1
  Number-Of-Pregnancies: 3

Is that contradictory?  Are you sure?

R's,
John

---

From: Nathaniel Borenstein <[email-masked]>
Date: Mon, 31 Oct 2011 10:08:19 -0500

Nothing's ever going to get rid of all possible ambiguities; transsexuals could be the obvious counterexample to your question.  All I'm suggesting is that we should try to avoid encouraging gratutiously obvious self-contradiction, such as separate scores for "is good" and "is bad".  But even that may be hopeless, I admit.  I can certainly be good sometimes and bad others...

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 31 Oct 2011 13:11:13 -0700

> From: Nathaniel Borenstein [mailto:[email-masked]]
> To: Murray S. Kucherawy
> Subject: Re: [domainrep] Comments on draft-kucherawy-reputation-vocab-identity
> 
> > That's something we've discussed, especially in the BoF I think, but
> > isn't reflected in the draft yet.  The range of assertions is currently
> > defined as being in the range [0, 1], meaning you disagree (0) or agree
> > (1) with the assertion to varying degrees.  But we also considered [-1,
> > 1], which as I recall means disagree (0), agree (1), or assert the
> > opposite of (-1) what's being stated in the reply.
> >
> My thinking on this one has been changing a bit.  I'm afraid that if we
> aren't careful, we'll make it too easy to create nonsense answers, e.g.
> giving me a high score as both a good guy and a bad guy.  If you make
> it too easy to do something stupid, people will.  So it would be nice
> to define the vocabulary in such a way that there were no nonsense-
> enabling redundancies; a -1 to 1 spectrum might make that clearer.  --
> nsb

I'm happy to go with consensus on this point, but I've been unable to formulate text to support it though I've been trying for days.

If we say that, for any given assertion, "1" means full agreement and "-1" actually asserts the opposite, then what does "0" mean?  For example, let's say "1" is SENDS-SPAM, and "-1" is thus SENDS-HAM.  What does "0" mean here?

If we say it means "sends some spam and some ham" then I'd say [-1, 1] is the same as [0, 1] only at double the scale, and furthermore the meaning of "0" is not intuitive (because it could also mean "I have no idea").  If we say it means "sends neither" then I suggest that's non-linear; if "1" means "sends mostly spam" and "-1" means "sends mostly ham" then the scale isn't linear anymore.

I couldn't come up with an example assertion that has an opposite for which [-1, 1] made sense when also considering the meaning of 0.  Does someone else have one?

-MSK

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 31 Oct 2011 13:13:59 -0700

> From: John Levine [mailto:[email-masked]]
> Sent: Thursday, October 27, 2011 5:03 AM
> To: [email-masked]
> Cc: Murray S. Kucherawy

So this means the registration for assertions within an application needs to define what 0 and 1 both mean, and also define the relationships among values in between.  For example, SENDS-SPAM might be defined as "0" disagrees completely with the assertion, "1" is full agreement, and values in between are linear (e.g., "X" means "sends about twice as much spam as someone with X/2").  Does that sound reasonable?  If so, I'll write that up.

-MSK

---

From: "David F. Skoll" <[email-masked]>
Date: Mon, 31 Oct 2011 16:19:46 -0400

On Mon, 31 Oct 2011 13:11:13 -0700
"Murray S. Kucherawy" <[email-masked]> wrote:

> If we say that, for any given assertion, "1" means full agreement and
> "-1" actually asserts the opposite, then what does "0" mean?  For
> example, let's say "1" is SENDS-SPAM, and "-1" is thus SENDS-HAM.
> What does "0" mean here?

It means "Sends the same amount of spam as ham".  And the real meaning of
"0" depends on the number of data points.

So with 100 datapoints:

SENDS-SPAM=1 means we've seen 100 spam, 0 ham.
SENDS-SPAM=0 means we've seen 50 spam, 50 ham.
SENDS-SPAM=-1 means we've seen 0 spam, 100 ham.

If we define [-1, 1] to map linearly to [0, NUM-DATA-POINTS] for the
events in question, then the meaning is clear.  Furthermore, if
NUM-DATA-POINTS is 0, then the rating MUST be 0 also.

> If we say it means "sends some spam and some ham" then I'd say [-1,
> 1] is the same as [0, 1] only at double the scale, and furthermore
> the meaning of "0" is not intuitive (because it could also mean "I
> have no idea").

"0" means "I have no idea" if NUM-DATA-POINTS is 0 or low.  It's only
by combining the score with the number of underlying data points that
we have a good idea of how well-known the rating is.  (And this is why
it's important that NUM-DATA-POINTS only count events that actually
support the given assertion or its negation.)

Regards,

David.

---

From: "David F. Skoll" <[email-masked]>
Date: Mon, 31 Oct 2011 16:22:40 -0400

On Mon, 31 Oct 2011 13:13:59 -0700
"Murray S. Kucherawy" <[email-masked]> wrote:

> So this means the registration for assertions within an application
> needs to define what 0 and 1 both mean, and also define the
> relationships among values in between.  For example, SENDS-SPAM might
> be defined as "0" disagrees completely with the assertion, "1" is
> full agreement, and values in between are linear (e.g., "X" means
> "sends about twice as much spam as someone with X/2").

Hmm... that seems unworkable.  I think my suggestion to map
either [-1, 1] or [0, 1] linearly to [0, NUM-DATA-POINTS] is easier to
implement and has a very clear meaning.

If you *really* want to compare how much spam example.org and
example.com send, you have both the ratings and the NUM-DATA-POINTS
for each, so you can compare apples to apples (sort-of, assuming the
reputation collector collects data points in proportion to the volume
of email sent.)

Regards,

David.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 31 Oct 2011 13:31:20 -0700

> Sent: Monday, October 31, 2011 1:20 PM
> To: [email-masked]
> Subject: Re: [domainrep] Comments on draft-kucherawy-reputation-vocab- identity
> 

What about the case where the assertion has no opposite?  For example, IS-PURPLE.  What would "-1" mean?

Maybe the registration must also define the range.  So:

SENDS-SPAM defines the range as [-1, 1], its use is linear (see my other message), and -1 means SENDS-HAM.  Meanwhile, some other thing uses [0, 1] because it has no opposite, and its use is (some expression of how values compare).

---

From: "David F. Skoll" <[email-masked]>
Date: Mon, 31 Oct 2011 16:37:52 -0400

On Mon, 31 Oct 2011 13:31:20 -0700
"Murray S. Kucherawy" <[email-masked]> wrote:

> What about the case where the assertion has no opposite?  For
> example, IS-PURPLE.  What would "-1" mean?

Point taken.  So I guess the registration MUST define the range
and SHOULD define the algorithm for converting data points to value.
There should be two predefined algorithms:

LINEAR [-1, 1] -> [0, NUM-DATA-POINTS]
LINEAR [ 0, 1] -> [0, NUM-DATA-POINTS]

Hmm... the more I think about it, the more I think the range
should be [0, 1] and we should explicitly have assertions that
are opposites like SENDS-HAM and SENDS-SPAM.  And we shouldn't worry
about contradictions.

Regards,

David.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 31 Oct 2011 14:32:29 -0700

> Sent: Monday, October 31, 2011 1:38 PM
> To: [email-masked]

That's mostly where I'm at.  The remaining point for me has to do with the data points themselves.  You appear to be using data points (in the spam case) as individual messages, but for me they're summarized days of data.  That makes the mapping less meaningful.

My vision for use of the data count is just to let the client say "I won't trust your reputation score until you tell me it's based on at least N data points," regardless of what the units are.

---

From: "Rolf E. Sonneveld" <[email-masked]>
Date: Mon, 31 Oct 2011 23:28:40 +0100

> My vision for use of the data count is just to let the client say "I won't trust your reputation score until you tell me it's based on at least N data points," regardless of what the units are.

Two comments on that last sentence:

 1. the word 'trust' should be used with great care here: I assume you
    mean: "I won't consider your reputation score of any value until I
    learn it's based on at least N data points"? Thats not the same as
    "I won't trust your reputation score until..." in my view.
 2. I'm not sure I understand what you mean with 'regardless of what the
    units are.'? This only works if server and client have the same
    understanding of what a 'unit' is (but I think that's what you want
    to say here, isn't it?)

This brings us to the question: shouldn't the query protocol include 
some sort of filtering 'language'? Like: give me the 'SENDS-SPAM' 
results for (domain) X where number of data points > Y and 'last updated 
date' > 1-nov-2011 etc.? This won't work out for DNS of course, but for 
the HTTP mechanism it might be a valuable extension?

/rolf

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Mon, 31 Oct 2011 15:31:14 -0700

Hi Rolf,

You're right on the "trust" point, and yes, your assumption of my meaning is correct.

The reason I'm resisting the idea of defining the units for the SAMPLE-SIZE is that doing so means making additional requirements on the reputation service provider in terms of implementation.  We're already saying the RATING has to be normalized, but I'm not sure I see the benefit to doing so for the sample size as well because it forces a specific quantization of the underlying data.

Put another way, David's idea appears to be to represent a ratio of spam-to-ham or something like that, where I want to give you an estimate of the likelihood that the next message(s) you get from that source will be spam.  His units are messages, while my units are days.  We go about it quite differently.  Forcing the sample count to be a specific unit means a lot less of that sort of variation in method is possible.

-MSK

From: Rolf E. Sonneveld [mailto:[email-masked]]
Sent: Monday, October 31, 2011 3:29 PM
To: Murray S. Kucherawy
Cc: [email-masked]
Subject: Re: [domainrep] Comments on draft-kucherawy-reputation-vocab-identity

-----Original Message-----

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of David F. Skoll

To: [email-masked]<mailto:[email-masked]>

about contradictions.

 1.  the word 'trust' should be used with great care here: I assume you mean: "I won't consider your reputation score of any value until I learn it's based on at least N data points"? Thats not the same as "I won't trust your reputation score until..." in my view.
 2.  I'm not sure I understand what you mean with 'regardless of what the units are.'? This only works if server and client have the same understanding of what a 'unit' is (but I think that's what you want to say here, isn't it?)

This brings us to the question: shouldn't the query protocol include some sort of filtering 'language'? Like: give me the 'SENDS-SPAM' results for (domain) X where number of data points > Y and 'last updated date' > 1-nov-2011 etc.? This won't work out for DNS of course, but for the HTTP mechanism it might be a valuable extension?

/rolf

---

From: "David F. Skoll" <[email-masked]>
Date: Mon, 31 Oct 2011 19:55:32 -0400

On Mon, 31 Oct 2011 14:32:29 -0700
"Murray S. Kucherawy" <[email-masked]> wrote:

> My vision for use of the data count is just to let the client say "I
> won't trust your reputation score until you tell me it's based on at
> least N data points," regardless of what the units are.

But if you don't know what a data point is, how can you make that decision?
In our system, one data point is one "event" as reported by an event
collector.  If one of your data points is a day's worth of events, then
one of your data points will correspond to several million of ours and
it's impossible to compare the systems.

I think we need a good definition of what a data point is, or failing that,
a good way for reputation systems to specify what they consider a data point
to be.

Regards,

David.

---

From: "John Levine" <[email-masked]>
Date: 1 Nov 2011 00:51:47 -0000

>a good way for reputation systems to specify what they consider a data point

I would suggest putting that on the list of things to resolve
eventually rather than trying to resolve it now.  If I were telling
you spam ratios per IP, I'd denominate it in perhaps tens or hundreds
of messages.  If it were somone like Return Path who aggregates data
from many large ISPs, it'd probably be hundreds of thousands.

I also suspect there will be providers with useful data who will consider
the counts proprietary.

R's,
John

---

From: "J.D. Falk" <[email-masked]>
Date: Mon, 31 Oct 2011 18:35:50 -0700

> from many large ISPs, it'd probably be hundreds of thousands.

Even better: not all data sources or types are weighted equally, so a raw count of email messages would be misleading.

> I also suspect there will be providers with useful data who will consider

Yep, and in those cases a self-identified "confidence" score makes more sense.

I think we'll need to leave room for both.

--
J.D. Falk
Return Path