Subject: Re: [domainrep] Comments on draft-kucherawy-reputation-vocab-identity

From: "Murray S. Kucherawy" <[email-masked]>
Date: Tue, 1 Nov 2011 03:52:15 -0700

My initial feeling is that exposing what amounts to SQL capabilities to clients is probably more access than servers are comfortable giving.  We need something a lot simpler.

Perhaps it's sufficient to indicate as part of the reply what the units of SAMPLE-SIZE are?

From: Rolf E. Sonneveld [mailto:[email-masked]]
Sent: Monday, October 31, 2011 3:29 PM
To: Murray S. Kucherawy
Cc: [email-masked]
Subject: Re: [domainrep] Comments on draft-kucherawy-reputation-vocab-identity

On 10/31/11 10:32 PM, Murray S. Kucherawy wrote:

-----Original Message-----

From: [email-masked]<mailto:[email-masked]> [mailto:[email-masked]] On Behalf Of David F. Skoll

Sent: Monday, October 31, 2011 1:38 PM

To: [email-masked]<mailto:[email-masked]>

Subject: Re: [domainrep] Comments on draft-kucherawy-reputation-vocab- identity

What about the case where the assertion has no opposite?  For

example, IS-PURPLE.  What would "-1" mean?

Point taken.  So I guess the registration MUST define the range

and SHOULD define the algorithm for converting data points to value.

There should be two predefined algorithms:

LINEAR [-1, 1] -> [0, NUM-DATA-POINTS]

LINEAR [ 0, 1] -> [0, NUM-DATA-POINTS]

Hmm... the more I think about it, the more I think the range

should be [0, 1] and we should explicitly have assertions that

are opposites like SENDS-HAM and SENDS-SPAM.  And we shouldn't worry

about contradictions.

That's mostly where I'm at.  The remaining point for me has to do with the data points themselves.  You appear to be using data points (in the spam case) as individual messages, but for me they're summarized days of data.  That makes the mapping less meaningful.

My vision for use of the data count is just to let the client say "I won't trust your reputation score until you tell me it's based on at least N data points," regardless of what the units are.

Two comments on that last sentence:

 1.  the word 'trust' should be used with great care here: I assume you mean: "I won't consider your reputation score of any value until I learn it's based on at least N data points"? Thats not the same as "I won't trust your reputation score until..." in my view.
 2.  I'm not sure I understand what you mean with 'regardless of what the units are.'? This only works if server and client have the same understanding of what a 'unit' is (but I think that's what you want to say here, isn't it?)

This brings us to the question: shouldn't the query protocol include some sort of filtering 'language'? Like: give me the 'SENDS-SPAM' results for (domain) X where number of data points > Y and 'last updated date' > 1-nov-2011 etc.? This won't work out for DNS of course, but for the HTTP mechanism it might be a valuable extension?

/rolf

---

From: Nathaniel Borenstein <[email-masked]>
Date: Wed, 9 Nov 2011 12:21:52 -0500

> If we say it means "sends some spam and some ham" then I'd say [-1, 1] is the same as [0, 1] only at double the scale, and furthermore the meaning of "0" is not intuitive (because it could also mean "I have no idea").  If we say it means "sends neither" then I suggest that's non-linear; if "1" means "sends mostly spam" and "-1" means "sends mostly ham" then the scale isn't linear anymore.

Or perhaps the scale is from certainty to certainty, with total uncertainty at zero.

	-1		0		1
     Certainly	     Color Me       Certainly
       False         Clueless	       True

The difference between this approach and David's ham/spam ratio is roughly the difference between "IS-A-SPAMMER" and "SENDS-SPAM."

I agree with David and Murray that we needn't require a single range or interpretation, but should let them be defined per-assertion.  -- Nathaniel

---

From: "David F. Skoll" <[email-masked]>
Date: Thu, 10 Nov 2011 09:43:14 -0500

On Wed, 9 Nov 2011 12:21:52 -0500

> Or perhaps the scale is from certainty to certainty, with total

I would rather have a separate confidence-level indicator.  (Not the
confidence in the identity being geniuine, but the confidence in the
score.)  So two numbers:

1) Reputation item:  0 (= not at all) --> 1 (= very much)
2) Confidence level: 0 (= no idea)    --> 1 (= I'm absolutely certain)

The confidence level is probably related to the number of data points,
but may be based on other information as well.

Regards,

David.

---

From: Peter Murray <[email-masked]>
Date: Thu, 10 Nov 2011 10:01:31 -0500

> but may be based on other information as well.

This approach allows for the consuming agent to determine how much confidence is acceptable for its purpose, our perhaps show confidence level to the user through a UI widget.  What I don't think we want to do is define what "confidence" means, which has the side effect of not allowing confidence levels from different sources to be compared but that is an acceptable tradeoff.

Peter
-- 
Peter Murray
Assistant Director, Technology Services Development
LYRASIS
[email-masked]
+1 678-235-2955
 
1438 West Peachtree Street NW
Suite 200
Atlanta, GA 30309
Toll Free: 800.999.8558
Fax: 404.892.7879 
www.lyrasis.org
 
LYRASIS: Great Libraries. Strong Communities. Innovative Answers.

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 10 Nov 2011 11:26:52 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of David F. Skoll
> Sent: Thursday, November 10, 2011 6:43 AM
> To: [email-masked]
> Subject: Re: [domainrep] [-1, 1] vs [0, 1] (was Re: Comments on draft-kucherawy-reputation-vocab-identity)
> 

I like this idea, especially if we keep in the idea of indicating the size of the sample set that yielded the result.  The three of those together would tell me quite a bit.

The problem (as someone else mentioned) is that how one service computes the confidence level may vary from that of another, which makes the client less portable.  But in this case, that bothers me less than it does with the reputation score itself.

John Levine suggested making the SAMPLE-SIZE parameter optional.  I think that's probably reasonable, especially if for some reason the service doesn't want to tell you that.  But I'm inclined to say this confidence parameter should always be there.  Others?

-MSK

---

From: Tony Hansen <[email-masked]>
Date: Thu, 10 Nov 2011 14:50:09 -0500

> John Levine suggested making the SAMPLE-SIZE parameter optional.  I think that's probably reasonable, especially if for some reason the service doesn't want to tell you that.  But I'm inclined to s…

+1 to always having the confidence parameter
+1 to making the sample size optional

     Tony Hansen
     [email-masked]

---

From: "Murray S. Kucherawy" <[email-masked]>
Date: Thu, 10 Nov 2011 11:55:37 -0800

> From: [email-masked] [mailto:[email-masked]] On Behalf Of Tony Hansen
> Sent: Thursday, November 10, 2011 11:50 AM
> To: [email-masked]
> > John Levine suggested making the SAMPLE-SIZE parameter optional.  I
> > think that's probably reasonable, especially if for some reason the
> > service doesn't want to tell you that.  But I'm inclined to say this
> > confidence parameter should always be there.  Others?
> 

Also, we already have a reporting parameter called "rater-authenticity", which is an expression of whether or not the reputation provider has confidence that the subject is genuine.  Could this be defined more clearly?  Or do we still need it at all?

The current text says:

        <t hangText="RATER-AUTHENTICITY:"> The level of confidence in that
                identity being genuine, expressed as a floating-point
                number between 0 and 1 inclusive. </t>

Perhaps "confidence" replaces this?  Or are they orthogonal?

-MSK

---

From: "David F. Skoll" <[email-masked]>
Date: Thu, 10 Nov 2011 15:06:21 -0500

On Thu, 10 Nov 2011 11:55:37 -0800

> Perhaps "confidence" replaces this?  Or are they orthogonal?

Orthogonal, I think.

For example, let's say I've had 75 reports of spam from "example.com"
and 25 reports of ham.  And let's say that each and every report was
DKIM-verified and reported to me by very trusted customers.  I might use:

RATER-AUTHENTICITY:  0.99  (DKIM-verified... pretty confident)
SENDS-SPAM:          0.75  (75% of all mail seen has been spam)
CONFIDENCE-LEVEL:    0.25  (But 100 data points is not very much...)

Obviously, computing the CONFIDENCE-LEVEL score requires judgement on
the part of the reputation provider and might not have a published
algorithm.

Regards,

David.

---

From: Nathaniel Borenstein <[email-masked]>
Date: Fri, 11 Nov 2011 16:45:40 -0500

> +1 to making the sample size optional

+1 to both of those
-1 to my previous idea