Subject: [lemonade] Re: Another WGLC on draft-ietf-lemonade-convert-08.txt

From: Alexey Melnikov <[email-masked]>
Date: Thu, 07 Jun 2007 10:29:36 +0100

Alexey Melnikov wrote:

> Alexey Melnikov wrote:
>
>> [email-masked] wrote:
>>
>>> A URL for this Internet-Draft is:
>>> http://www.ietf.org/internet-drafts/draft-ietf-lemonade-convert-08.txt
>>
> Due to the number of changes to the document, I would like to start 
> another 2 weeks Working Group Last Call.
> Please send your comments to me and/or to the mailing list before June 
> 6th.

The WGLC has ended yesterday. As there were no comments received I will 
assume that everybody is happy with the document and will request its 
publication.

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade
Supplemental Web Site:
http://www.standardstrack.com/ietf/lemonade

---

From: Eric Burger <[email-masked]>
Date: Mon, 11 Jun 2007 09:41:32 -0400

Hello, anybody home?

It is OK to send a, "I read the draft, and it looks good" message.  For that
matter, it is OK to send a, "I looked at the draft, I don't really care, but
the draft will not harm the Internet" message.

Capiche?

> assume that everybody is happy with the document and will request its

Notice:  This email message, together with any attachments, may contain information  of  BEA Systems,  Inc.,  its subsidiaries  and  affiliated entities,  that may be confidential,  proprietary,  copyrighted  and/or legally privileged, and is intended solely for the use of the individual or entity named in this message. If you are not the intended recipient, and have received this message in error, please immediately return this by email and then delete it.

---

From: "Vaudreuil, Greg M \(Greg\)" <[email-masked]>
Date: Mon, 11 Jun 2007 09:53:38 -0500

I read, but did not study the document.  Only two items caught my
attention.

- How does a client know if the server will convert non-leaf body parts?
It looks like a client gets to play "go fish".

- Convert.size is expensive, but I'm not sure a caution to clients
addresses the issue of over-use.  If one uses it to improve the UI
experience, it seems it would be over-use on the server.

I'd prefer if we could allow convert.size to be a statistical estimate
such that a server can provide low-cost estimates without actually
converting the material.  I can imagine a client wishing to offer
"small, medium, large" buttons (with an estimated download time) next to
each attachment and letting the end-user check.  Rather than say this us
too expensive, can we support it with estimates?

Greg V.

-----Original Message-----
From: Eric Burger [mailto:[email-masked]]
Sent: Monday, June 11, 2007 9:42 AM
To: [email-masked]
Subject: [lemonade] Just say "Yes"

Hello, anybody home?

It is OK to send a, "I read the draft, and it looks good" message.  For
that
matter, it is OK to send a, "I looked at the draft, I don't really care,
but
the draft will not harm the Internet" message.

Capiche?

> assume that everybody is happy with the document and will request its

Notice:  This email message, together with any attachments, may contain
information  of  BEA Systems,  Inc.,  its subsidiaries  and  affiliated
entities,  that may be confidential,  proprietary,  copyrighted  and/or
legally privileged, and is intended solely for the use of the individual
or entity named in this message. If you are not the intended recipient,
and have received this message in error, please immediately return this
by email and then delete it.

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 11 Jun 2007 17:15:59 +0100

Hi Greg,
Thank you for the feedback.

>It looks like a client gets to play "go fish".
The client can check if conversion from multipart/* is advertised by the 
server.

>too expensive, can we support it with estimates?
I think so. The estimated size should be no smaller than the actual 
converted size.
Can you suggest some specific text that you would like to be changed or 
added?

Thanks,
Alexey

---

From: "Vaudreuil, Greg M \(Greg\)" <[email-masked]>
Date: Mon, 11 Jun 2007 13:06:39 -0500

I don't think we can specify that the error must be only in the positive
direction without reducing the utility.   Errors in the overstating or
understating are both ungood, but I can think of use cases where either
would be worse than the other.  I suggest servers simply do their best
with what they know and let client vendors know not to rely on the
result for correct protocol operation.

Try the following text.

Normally, BODYSTRUCTURE provides an accurate size of a message body part
to the client. However, when converting into an alternate format, the
true size of the body part may not be know without performing the
conversion and checking the result.  For that reason, CONVERT.SIZE
provides only an estimate of the body part size, with no assurance that
the resulting conversion will be either larger or smaller by an
unspecified margin.  However, for ordinary contents, the range of likely
values for a given variable-size encoding is sufficiently small as to
have substantial utility with minimum server computation.

Greg V.

-----Original Message-----
From: Alexey Melnikov [mailto:[email-masked]]
Sent: Monday, June 11, 2007 12:16 PM
To: Vaudreuil, Greg M (Greg)
Cc: [email-masked]
Subject: Re: [lemonade] Just say "Yes"

Hi Greg,

server.

Thanks,
Alexey

---

From: Mark Crispin <[email-masked]>
Date: Mon, 11 Jun 2007 11:26:51 -0700 (Pacific Daylight Time)

In my opinion, if CONVERT.SIZE does not provide the exact value (the way 
that BODYSTRUCTURE and BINARY.SIZE does), then it is useless and (more 
importantly) evil.

I am amazed that Greg and Alexey would even suggest something like this 
that spits in the face of IMAP's architecture.

Note as well that the absence of a guarantee of exact sizes forever dooms 
the CONVERT functionality to be unusable with partial fetching.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Philip Guenther <[email-masked]>
Date: Mon, 11 Jun 2007 12:45:14 -0600

> result for correct protocol operation.

That sounds reasonable to me, but I think then the fetch item should then 
be renamed to CONVERT.ESIZE or CONVERT.SWAG or something instead of 
CONVERT.SIZE to make it clear that it's different from RFC822.SIZE and 
BINARY.SIZE.  If it's later determined that exact size info *is* worth the 
cost, then CONVERT.SIZE can be used for that.

Document nit: this snippet in section 5:
    BINARY.SIZE[section-part.CONVERT ("media type/subtype"
    (parameters))]<partial>

should read:

    BINARY.SIZE[section-part.CONVERT ("media type/subtype"
    (parameters))]

as BINARY.SIZE does not permit a partial specifier

Speaking of partials, I suggesting adding text similar to that in BINARY 
that notes that partials operate on the decoded data and not the source.

Hmm, conversion data is currently in the syntax 'under' msg-att-static, 
which implies that the result of a particular conversion for a particular 
section of a given message is immutable.  That's good and bad.  It's good 
because it means a conversion can be fetched in chunks via partials. 
It's bad because it means a server can't upgrade converters without 
unreasonable bookkeeping.

I have no opinion whether conversion data should be considered immutable, 
but the fact should be called out in prose and not just an implication of 
the grammar.

Philip Guenther

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 20 Jun 2007 14:02:31 +0100

Mark Crispin wrote:

> In my opinion, if CONVERT.SIZE does not provide the exact value (the 
> way that BODYSTRUCTURE and BINARY.SIZE does), then it is useless and 
> (more importantly) evil.
>
> I am amazed that Greg and Alexey would even suggest something like 
> this that spits in the face of IMAP's architecture.

Mark, I think CONVERT is slightly different.
We don't want to mandate that any conversion should be immutable forever 
(even though I think it should be immutable for at least the duration of 
the session). For example if the transcoding server is upgraded or 
replaced, the exact size after a particular conversion might change.

Should I add some text stating this?
Or do you still prefer that converted bodyparts be immutable forever?

> Note as well that the absence of a guarantee of exact sizes forever 
> dooms the CONVERT functionality to be unusable with partial fetching.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 20 Jun 2007 14:09:37 +0100

Vaudreuil, Greg M (Greg) wrote:

>unspecified margin.
Returning size which is bigger than the exact size is always better, as 
it would work with partial FETCH. A Partial range beyond the end of a 
converted bodypart returns no data. If the server returns size which is 
smaller, then the client may end up getting truncated data back.

>  However, for ordinary contents, the range of likely

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 20 Jun 2007 14:17:34 +0100

Philip Guenther wrote:

> That sounds reasonable to me, but I think then the fetch item should 
> then be renamed to CONVERT.ESIZE or CONVERT.SWAG or something instead 
> of CONVERT.SIZE to make it clear that it's different from RFC822.SIZE 
> and BINARY.SIZE.  If it's later determined that exact size info *is* 
> worth the cost, then CONVERT.SIZE can be used for that.

I think we need to decide if CONVERT.SIZE can be unexact or not.
I don't think we should have 2 versions (an exact and a "guess"), as 
clients are unlikely to use the "guess" version.

> as BINARY.SIZE does not permit a partial specifier

Good catch. Fixed.

> Speaking of partials, I suggesting adding text similar to that in 
> BINARY that notes that partials operate on the decoded data and not 
> the source.

Done.

> Hmm, conversion data is currently in the syntax 'under' 
> msg-att-static, which implies that the result of a particular 
> conversion for a particular section of a given message is immutable.  
> That's good and bad.  It's good because it means a conversion can be 
> fetched in chunks via partials. It's bad because it means a server 
> can't upgrade converters without unreasonable bookkeeping.

I agree that this needs to be clarified.
See also my reply to Mark.

> I have no opinion whether conversion data should be considered 
> immutable, but the fact should be called out in prose and not just an 
> implication of the grammar.

---

From: "Vaudreuil, Greg M \(Greg\)" <[email-masked]>
Date: Wed, 20 Jun 2007 08:49:13 -0500

I assert that an estimate is the only affordable value available to a
server:

Further, I believe this estimate can't be relied upon to be larger than
the true size of the converted content. The only way to return a value
that is larger than the exact size is to either:

  1) Compute the exact size (why then bother with the estimate?) 
  2) Always give a best-case (smallest) estimate, which could be very
far off since most compressions *can* result in tiny files (imagine a
minute of mostly silence compressed, or a picture that is mostly black
background)

So, I think we need to figure out how to make this protocol work with an
estimate, modifying what fetch partial returns when it tries to read
past the end of data.

Greg V.

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 20 Jun 2007 15:26:14 +0100

>estimate, modifying what fetch partial returns when it tries to read
As I said below, reading past the end of converted data is not a problem 
in IMAP.
Data truncation is. Data truncation can occur if the returned converted 
size is smaller than the actual converted size. If the client uses 
<partial> FETCH, it will not bother downloading the part of the 
converted bodypart which is between the returned and the actual size.

One possible way of allowing CONVERT.SIZE to be inexact while allowing 
for <partial> FETCH is to force return of the correct converted size 
upon a CONVERT request. But this looks suboptimal to me and might lead 
to interoperability problems. (Think about clients cashing the wrong 
value or being tested against the server that always returns the correct 
size)

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Jun 2007 08:24:33 -0700 (PDT)

On Wed, 20 Jun 2007, Alexey Melnikov wrote:
> (even though I think it should be immutable for at least the duration of the 
> session). For example if the transcoding server is upgraded or replaced, the 
> exact size after a particular conversion might change.

Immutable for the duration of the session (but not forever) is probably 
alright, but the authors of disconnected clients may disagree.

> Or do you still prefer that converted bodyparts be immutable forever?

I think that immutable forever, and control over the conversion algorithm 
is better.

-- Mark --

http://panda.com/mrc
Democracy is two wolves and a sheep deciding what to eat for lunch.
Liberty is a well-armed sheep contesting the vote.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Jun 2007 08:26:32 -0700 (PDT)

On Wed, 20 Jun 2007, Alexey Melnikov wrote:
> Returning size which is bigger than the exact size is always better, as it 
> would work with partial FETCH. A Partial range beyond the end of a converted 
> bodypart returns no data. If the server returns size which is smaller, then 
> the client may end up getting truncated data back.

This isn't how clients that do partial fetching work; and if you look at 
what the protocol says on this you'll see that the situation that you 
describe is ambiguous.  Some clients will end up looping.

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Jun 2007 08:28:23 -0700 (PDT)

> estimate, modifying what fetch partial returns when it tries to read

It is better to return no size at all than a lie.

NNTP is a great example of what happens when the protocol lies.  IMAP does 
not have this bug.

If you want to produce a "display" size, that may be OK, but then do it is 
something that is obviously not related to reality, such as a value in 
"K".

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Jun 2007 08:29:35 -0700 (PDT)

On Wed, 20 Jun 2007, Alexey Melnikov wrote:
> As I said below, reading past the end of converted data is not a problem in 
> IMAP.

As the author of client code that does partial fetching, I disagree.

-- Mark --

http://panda.com/mrc

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 20 Jun 2007 16:38:19 +0100

Mark Crispin wrote:

>> As I said below, reading past the end of converted data is not a 
>> problem in IMAP.
>

Ok, I can restate that: my server handles this gracefully ;-).

---

From: "Vaudreuil, Greg M \(Greg\)" <[email-masked]>
Date: Wed, 20 Jun 2007 10:52:31 -0500

I'm happy with redefining this as a display size, and even making the
granularity "k" to help client writers get the hint.  

What needs to be clear is that clients should only request, or better,
can only request, an accurate size as part of the converted content
fetch, where the conversion is performed anyway.

Greg V.

-----Original Message-----
From: [email-masked] [mailto:[email-masked]]On Behalf Of
Mark Crispin
Sent: Wednesday, June 20, 2007 11:28 AM
To: Vaudreuil, Greg M (Greg)
Cc: Alexey Melnikov; [email-masked]
Subject: RE: [lemonade] Just say "Yes"

NNTP is a great example of what happens when the protocol lies.  IMAP
does 
not have this bug.

If you want to produce a "display" size, that may be OK, but then do it
is 
something that is obviously not related to reality, such as a value in 
"K".

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Jun 2007 09:20:45 -0700 (PDT)

In that case, the specification MUST prohibit the use of partial fetching, 
stipulate that both in the text and in the formal syntax, AND explain why 
partial fetching is prohibited.

Otherwise, this particular conversation will never end; it will come up 
over, and over, again.  I can guarantee you that some client author is 
going to want to try partial fetching and will run up against the problem. 
If he is particularly unlucky, he will test against a server that returns 
exact values (although using "K" or even "M" will help avoid this) and not 
find out about servers which lie until later.

And, at some point, someone is going to ask why this wart is in IMAP when 
all modern servers have no trouble with computing exact sizes and why 
should we care about dinosaur software/hardware.

Such nightmares are the reality of NNTP, and the cretins in that group 
refused to fix it when they had a chance.  Fortunately, netnews is dying 
as an Internet service (all of about 60 people at UW still use it) so it 
doesn't really matter any more.

Presumably IMAP still has a little bit more life left in it.  We ought to 
try to design IMAP as if it is still alive, and not just dump in any old 
trash just because it is convenient.

I'd like to mention, as an aside, that the requirement in IMAP that 
message sizes be exact was a TERRIBLE blow to a certain well-known vendor 
who used a database as a message store and didn't have the message stored 
in RFC822 format at all.  They thought those sizes were estimates, and 
found out otherwise the hard way.  They ended up having to redo their 
message store in order to implement IMAP with any type of efficiency.

So it isn't as if this is a new problem.  It isn't.

-- Mark --

http://panda.com/mrc

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Jun 2007 09:29:20 -0700 (PDT)

I think that it is a serious design flaw that CONVERT is a FETCH data item 
at all.  It violates the design principle that all FETCH message data in 
IMAP is immutable (metadata being a different matter).

Instead, I recommend that CONVERT be a new command in IMAP.  What is being 
done is an additional server operation, not merely a FETCH.  Then, if you 
want to have partial fetching of converted data, you can create a new 
framework for doing so that doesn't cause the problems from IMAP's 
framework.

Arguably, IMAP BINARY should have been a different command, as well as any 
IMAP Unicode fetch, since those are also "conversions".  But those both 
return immutable data with sizes that are equally immutable, so that is a 
razor that can be justified.

CONVERT, as I understand it, is a server-based possibly lossy conversion 
that is server implementation-dependent (and thus different compliant 
servers can convert differently).  This is NOT something that should be 
part of the IMAP FETCH framework.

Most of my objections would become moot if you make this a new command.

I'm willing to help out on reforming the design to be a new command, at 
least to sketch out syntax.

-- Mark --

http://panda.com/mrc

---

From: Lyndon Nerenberg <[email-masked]>
Date: Wed, 20 Jun 2007 09:59:21 -0700 (PDT)

> I don't think we should have 2 versions (an exact and a "guess"), as clients 
> are unlikely to use the "guess" version.

Either it's exact, or it needs to come out.  Inaccurate data is worse than 
useless in this instance.

--lyndon

   Good judgement comes from experience. Unfortunately, experience
   comes from bad judgement.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 20 Jun 2007 20:04:04 +0200

> granularity "k" to help client writers get the hint.

+1

Arnt

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 20 Jun 2007 20:16:49 +0200

Lyndon Nerenberg writes:
> Either it's exact, or it needs to come out.  Inaccurate data is worse 
> than useless in this instance.

Why? Could you elaborate your viewpoint?

I see no need for exactness - returning an integer number of tenths 
would satisfy me. "About one tenth of the original size", "about three 
tenths ..." etc. This is because I only want to answer the question 
"how much speed will I earn by getting this approximating conversion?" 
and I'd rather get the options quickly than know the exact traffic 
savings. Obviously your thinking is very different, and I'm curious as 
to how.

Btw, I don't agree that clients are unlikely to use the guess version. 
If there's language saying "the server SHOULD be quick, even at the 
expense of accuracy..." and/or "clients wanting an answer quickly MUST 
use the inexact...", I think that provides real incentive to use an 
inexact version. For example, when you're displaying a dialog, it's a 
definite advantage to have it done within about 0.2-0.7 seconds 
depending on type of dialog.

Arnt

---

From: Curtis King <[email-masked]>
Date: Wed, 20 Jun 2007 12:24:26 -0600

On Jun 20, 2007, at 10:29 AM, Mark Crispin wrote:

> CONVERT, as I understand it, is a server-based possibly lossy  
> conversion that is server implementation-dependent (and thus  
> different compliant servers can convert differently).  This is NOT  
> something that should be part of the IMAP FETCH framework.
>

+1 on making CONVERT a new command.

ck

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Jun 2007 12:33:31 -0700 (Pacific Daylight Time)

> Why? Could you elaborate your viewpoint?

What purpose does this value purport to serve?

In every other place in IMAP, sizes are used as exact values to allocate 
space and for partial fetch markers.  For some reason, you guys seem to be 
advocating a different type of size, with no apparent purpose, yet 
omitting the size that software expects (and NEEDS), all with the weak 
excuse that "it'd be too inefficient for some servers."

> I see no need for exactness - returning an integer number of tenths would 
> satisfy me. "About one tenth of the original size", "about three tenths ..." 
> etc. This is because I only want to answer the question "how much speed will 
> I earn by getting this approximating conversion?" and I'd rather get the 
> options quickly than know the exact traffic savings. Obviously your thinking 
> is very different, and I'm curious as to how.

If this is the functionality that you seek to accomplish, then you should 
do it by some other mechanism than delivering a lie that will mislead 
clients into incorrect behavior.

> Btw, I don't agree that clients are unlikely to use the guess version. If 
> there's language saying "the server SHOULD be quick, even at the expense of 
> accuracy..." and/or "clients wanting an answer quickly MUST use the 
> inexact...", I think that provides real incentive to use an inexact version.

NNTP has this same type of thinking.  NNTP sucks.

-- Mark --

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 20 Jun 2007 21:42:45 +0200

Mark Crispin writes:
>>> Either it's exact, or it needs to come out.  Inaccurate data is 
>>> worse than useless in this instance.

Selecting which approximating conversion to fetch, if any?

Arnt

---

From: Lyndon Nerenberg <[email-masked]>
Date: Wed, 20 Jun 2007 12:44:57 -0700 (PDT)

> Why? Could you elaborate your viewpoint?

If the size is inaccurate it's because the server is guessing (to some 
unknown amount of accuracy) about the outcome of the transformation.  In 
the simple case, it's going to take the size of the original and run it 
through a simple algorithm to derive it's guess.  If a client considers an 
available transformation viable, it can just as easily compute this guess 
from the size information already at hand (and potentially saving an RTT). 
This makes CONVERT.SIZE superfluous.

The server might use it's knowledge of the content and structure of the 
part to perform a partial evaluation of the part's contents to calculate a 
more accurate guess.  It's my contention that once you start doing a 
partial evaluation, you're most of the way to just doing it right and 
calculating the exact value.

But why are inexact values inherently evil?  Because clients -- despite 
all the protests everyone cares to make to the contrary -- will treat them 
as exact values, and then fail when their assumptions about the data turn 
out to be wrong.  If you haven't learned by now that client writers will 
take the most stupid path imaginable, multiply the brain damage by 10, and 
then incorrectly implement their incorrect assumptions ... well, you 
shouldn't be in the IMAP business.

If the intent of CONVERT.SIZE is to provide a display value, any client 
that can make use of the converted type can calculate the approximate size 
of the converted object to a precision far exceeding what is necessary for 
display purposes (about three significant digits for any device I'm 
familiar with).

Therefore, not only is CONVERT.SIZE unnecessary for protocol 
functionality, it's introduction will actively harm interoperability.

Note that from a server implementation standpoint you can do lazy 
evaluation of CONTENT.SIZE -- just as with BINARY.SIZE -- and cache the 
result; providing accurate CONTENT.SIZE does not have to imply a general 
performance hit.

  --lyndon

---

From: Lyndon Nerenberg <[email-masked]>
Date: Wed, 20 Jun 2007 12:50:04 -0700 (PDT)

> Selecting which approximating conversion to fetch, if any?

Oh good lord, can we just drop the 'Engineering' from IETF?  This is just 
absurd.  If this is truly the reason then drop CONVERT.SIZE and have the 
client randomly pick one of the conversions.

--lyndon

   I have challenged the entire quality assurance team to a Bat-Leth
   contest.  They will not concern us again.

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 20 Jun 2007 22:10:20 +0200

Lyndon Nerenberg writes:
> Oh good lord, can we just drop the 'Engineering' from IETF?  This is 
> just absurd.  If this is truly the reason then drop CONVERT.SIZE and 
> have the client randomly pick one of the conversions.

Picking randomly is an advance over presenting a dialog like this?

         Choose format for downloading:
         [ ] Exact file (est. download time 10m)
         [ ] Blah conversion blah a (est. download time 3m)
         [ ] Blah conversion blah b (est. download time 30s)

>   contest.  They will not concern us again.

Strangely appropriate tag.

Arnt

---

From: Lyndon Nerenberg <[email-masked]>
Date: Wed, 20 Jun 2007 13:22:04 -0700 (PDT)

>        [ ] Blah conversion blah b (est. download time 30s)

The dialog is not the issue.  The issue is that if you're going to guess, 
the client can guess just as well as the server.

Therefore CONVERT.SIZE provides no benefit.

--lyndon

   NT as a file server is faster than a dead bat carrying Post-It notes
   underwater. But not by much.

---

From: Mark Crispin <[email-masked]>
Date: Wed, 20 Jun 2007 13:35:30 -0700 (Pacific Daylight Time)

On Wed, 20 Jun 2007, Lyndon Nerenberg wrote:
> If you haven't learned by now that client writers will take the 
> most stupid path imaginable, multiply the brain damage by 10, and then 
> incorrectly implement their incorrect assumptions ... well, you shouldn't be 
> in the IMAP business.

+1

And not just client writers.  Server writers do some amazingly bad things 
too.

-- Mark --

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 21 Jun 2007 19:02:02 +0100

Lyndon Nerenberg wrote:

> The dialog is not the issue.  The issue is that if you're going to 
> guess, the client can guess just as well as the server.

Not that I am in favor of inexact size, but I doubt that this claim is true.

---

From: Lyndon Nerenberg <[email-masked]>
Date: Thu, 21 Jun 2007 11:10:25 -0700 (PDT)

> Not that I am in favor of inexact size, but I doubt that this claim is true.

Then please explain why   x' = F(x)  where x is the original part size and 
F is the function that calculates the estimated size of the converted part 
would work on the server but not on the client.

And I want to see a logically verifiable proof, not a bunch of hand waving.

--lyndon

   This machine is a piece of GAGH!  I need dual Pentium processors if I am
   to do battle with this code!

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 21 Jun 2007 19:12:12 +0100

Lyndon Nerenberg wrote:

> Then please explain why   x' = F(x)  where x is the original part size 
> and F is the function that calculates the estimated size of the 
> converted part would work on the server but not on the client.

Function F might depend on the bodypart. Client doesn't have access to 
it. If it has, then it probably doesn't need CONVERT.

---

From: Alexey Melnikov <[email-masked]>
Date: Thu, 21 Jun 2007 19:14:37 +0100

Mark Crispin wrote:

> conversion that is server implementation-dependent (and thus different 
> compliant servers can convert differently).

Correct.

> This is NOT something that should be part of the IMAP FETCH framework.

(A side note: we really need to document IMAP extensibility framework, 
include the FETCH command. Any volunteers?)

> Most of my objections would become moot if you make this a new command.

A disadvantage of this change is that we suddenly need new IMAP URL 
syntax for converted parts, while before it came for free because IMAP 
URL was using  BODY[section] ABNF.

> I'm willing to help out on reforming the design to be a new command, 
> at least to sketch out syntax.

Please.

---

From: Mark Crispin <[email-masked]>
Date: Thu, 21 Jun 2007 11:26:34 -0700 (PDT)

On Thu, 21 Jun 2007, Alexey Melnikov wrote:
> Function F might depend on the bodypart. Client doesn't have access to it. If 
> it has, then it probably doesn't need CONVERT.

The client has access to the BODYSTRUCTURE data for the body part, 
including the type and its size; otherwise it would not know about the 
bodypart at all.

For Alexey's argument to hold, the actual body part data is needed to 
produce the estimate.  I am unconvinced by a claim that, using the body 
part data, it is somehow possible to come up with a better estimate than 
can be done with just the BODYSTRUCTURE data; yet it is somehow extremely 
difficult to come up with a precise value.

That is the claim that needs to be proven.

-- Mark --

http://panda.com/mrc

---

From: Lyndon Nerenberg <[email-masked]>
Date: Thu, 21 Jun 2007 11:30:03 -0700 (PDT)

> it has, then it probably doesn't need CONVERT.

Where in that formula does it say anything about the content.  The single 
parameter is size.

As I mentioned in my response to Arnt, if the server is going to start 
parsing the content to create a more accurate estimate, it should just 
parse the content and generate the exact result size.

I fail to see the rational for having support in the server to parse the 
content -- and this is a pre-requisite to offering the conversion, so that 
support must be there -- and not using it.  The only viable argument I see 
is that for certain transformations it might be expensive?  All I can say 
is "so what?"  If you want to use an expensive service, you pay the 
expensive price.  As I explained earlier, you can defer the work until 
it's actually needed, and then cache the result.

--lyndon

   Our users will know fear and cower before our software!  Ship it!  Ship it
   and let them flee like the dogs they are!

---

From: "Vaudreuil, Greg M \(Greg\)" <[email-masked]>
Date: Thu, 21 Jun 2007 14:13:52 -0500

I think I've convinced myself that a client-produced estimate may be
adequate.

Client receives a high-rez picture in image/fancy.  It can see this is a
2megabyte file.  The user is connected on GPRS, so can plan on using 2
hours of connect time to downaload and 1/2 of the monthly bucket of
bandwidth.  Bad.  Need conversion.

The client can accept JPEG's of 160x240, but has no idea how image/fancy
converts to JPEG.  It cannot estimate the percentage of reduction, but
it well could have an internal table with the dozen or so
format*resolution combinations it does know and that it can offer that
data to the user.

There is variation in how large a JPEG of 160x240.  To do better, the
server will have to examine at least the initial bytes or header of the
requested content.  For example, the content meta data may show that the
image is compressed 20% better than the average for a image/fancy of
that resolution.  This rough estimate is much less expensive than
converting the attachment to find the converted size.

However, I'm willing to say that the added accuracy is likely to be low
relative to the costs in protocol complexity.  Server-provided estimates
can also be added in later if we find we need it.

If this is acceptable to others, I'd like to see a note added to the
document explaining it is a really good idea not to trash the server by
asking "what if resolution X, and what if resolution Y, and what if
resolution Z", but to make client-side estimates to be presented to the
user and only fetch the convert.size when there is an intent to
download.

Greg V.

-----Original Message-----
From: Lyndon Nerenberg [mailto:[email-masked]]
Sent: Thursday, June 21, 2007 2:30 PM
To: Alexey Melnikov
Cc: [email-masked]
Subject: Re: [lemonade] Just say "Yes"

> Function F might depend on the bodypart. Client doesn't have access to
it. If 
> it has, then it probably doesn't need CONVERT.

Where in that formula does it say anything about the content.  The
single 
parameter is size.

content -- and this is a pre-requisite to offering the conversion, so
that 
support must be there -- and not using it.  The only viable argument I
see 
is that for certain transformations it might be expensive?  All I can
say 
is "so what?"  If you want to use an expensive service, you pay the 
expensive price.  As I explained earlier, you can defer the work until 
it's actually needed, and then cache the result.

--lyndon

   Our users will know fear and cower before our software!  Ship it!
Ship it
   and let them flee like the dogs they are!

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Thu, 21 Jun 2007 21:56:14 +0200

> And I want to see a logically verifiable proof, not a bunch of hand waving.

1. F is inherently related to the function which actually does the 
conversion, which is available on the server but not on the client. 
Conversion from another image format to PNG is a good example: If a 
server uses the PNG code used in xv, its PNG output will be a lot 
larger than if the server uses the same library as Adobe Photoshop does 
(I've seen differences larger than 50%).

2. F need not be F(x), but can be F(actual data) or F(part of the data). 
For example, one might implement an image conversion F by performing 
the conversion on one or a few randomly chosen part(s) of the image and 
multiplying.

Arnt

---

From: Lyndon Nerenberg <[email-masked]>
Date: Thu, 21 Jun 2007 13:24:45 -0700 (PDT)

> Conversion from another image format to PNG is a good example: If a server 
> uses the PNG code used in xv, its PNG output will be a lot larger than if the 
> server uses the same library as Adobe Photoshop does (I've seen differences 
> larger than 50%).

Not only that, but even within a known implementation there will always be 
cases where the estimate is completely out to lunch.  To avoid that, you 
have to examine the data to at least some degree.  Which, once again, 
brings up back to the issue of partial evaluation of the input vs. 
complete evaluation of the input.  Nobody has yet demonstrated that the 
incremental cost of full vs. partial evaluation is sufficient to 
necessitate partial evaluation to begin with.

> 2. F need not be F(x), but can be F(actual data) or F(part of the data). For 
> example, one might implement an image conversion F by performing the 
> conversion on one or a few randomly chosen part(s) of the image and 
> multiplying.

F is a function that can run equally well on the client or server, and 
therefore is restricted to having access to the same parameters; 
practically, this means the contents of ENVELOPE, of which the only useful 
element is SIZE.

But we now seem to be restricting the implementation to those which have 
access to the source data to be converted.  Fine.  So will someone please 
show me a practical example of where a partial-evaluation based estimate 
will provide any practical savings in the context of an IMAP protocol 
exchange in the target environment.  Remember we are talking a mobile 
environment, with RTTs in the 10s of milliseconds, and 100s of 
milliseconds of command/response turnaround when you factor in the transit 
delay of the actual protocol PDUs.  In order for an estimated CONVERT.SIZE 
to provide a visible benefit over an accurate CONVERT.SIZE, the savings of 
partial vs. complete evaluation of the conversion needs to save on the order 
of 100ms of wall clock time.  If you're going to justify the need for an 
estimated CONVERT.SIZE you must demonstrate that this 100ms scenario is 
going to happen in the real world on an at least semi-regular basis.

--lyndon

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 22 Jun 2007 10:27:12 +0200

This makes sense to me, a lot of sense. Your message seems to lead to 
one of these:

1. The size of a converted file can be estimated from the domain size of 
the data and the format, e.g. image resolution and image format. "An 
160*128 image/jpg is usually x-y bytes long".

2. The size of a converted file can be estimated from the same data as 
in 1, and the same data for input.
"An 160*128 image/jpg is usually x-y bytes big. Since the input file is 
20% bigger than the average for its resolution and format, we can 
assume that this one will be x*1.2-y*1.2 bytes".

Both seem plausible to me, although I've no idea which is closer to the 
truth. I'm converting the few hundred images from the London interop to 
200*200 pixels width and three formats and will post a followup when 
the job's done and I've looked at the sizes. (My desktop is fanless and 
very slow, so things take time.)

I'm not sure where this leads. If the client can get the domain size of 
the input data, it can do the estimate. But it can also get the size 
from the server, although that may be slow work for the server. In 
either case the client needs to get something from the server - domain 
size or estinated target octet size(s).

Arnt

---

From: "Ben Last" <[email-masked]>
Date: Fri, 22 Jun 2007 10:14:15 +0100

>From: Arnt Gulbrandsen [mailto:[email-masked]] 
>Sent: 22 June 2007 09:27
>This makes sense to me, a lot of sense. Your message seems to lead to one of these:
>1. The size of a converted file can be estimated from the domain size of the data and the format, e.g.
>image resolution and image format. "An 160*128 image/jpg is usually x-y bytes long".

True, one could generate a client-side list of formats and typical sizes and use that, and it would be an effective solution.  However, our client doesn't have a list of accepted formats.  The platform (in our case, S60/Symbian OS) has a list of supported formats, which list will almost certainly evolve over versions of the platform (it already has).  Thus the process of determining which format is best is based on matching up two dynamically* generated lists: what the server can provide and what the platform can support.  Our client is merely a broker between the two, and thus not the place to store a static list of rules of thumb.

Image conversion might be a misleading use case here, since images come in a reasonably well defined set of standard formats, for which such rules are easily generated from reference data.  How about conversion from Microsoft's Office 2007 XML format to Word 2003 format (as supported by many mobile office document viewing clients)?  The size of the converted document is going to be highly content-dependent.

Yours pedantically
b

* By "dynamically", I mean "not reliably predictable at the time of building the client software".

Ben Last
R&D Manager
EMCC Software Ltd

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 22 Jun 2007 11:52:30 +0200

Ben Last writes:
> True, one could generate a client-side list of formats and typical 
> sizes and use that, and it would be an effective solution.

It wasn't even very effective. The x-y range I mentioned would have to 
be very large.

> However, our client doesn't have a list of accepted formats. The 
> platform (in our case, S60/Symbian OS) has a list of supported 
> formats, which list will almost certainly evolve over versions of the 
> platform (it already has). Thus the process of determining which 
> format is best is based on matching up two dynamically* generated 
> lists: what the server can provide and what the platform can support. 
> Our client is merely a broker between the two, and thus not the place 
> to store a static list of rules of thumb.

Let me guess: You'd rather use CONVERT.SIZE, no matter how expensive 
that is for the server? Understandable in the circumstances.

Arnt

---

From: "Ben Last" <[email-masked]>
Date: Fri, 22 Jun 2007 10:57:43 +0100

>Let me guess: You'd rather use CONVERT.SIZE, no matter how expensive that is for the server? Understandable in the circumstances.

Actually, we have no plans as yet to include document conversion in the client.  It involves a complex UI that has to guide the user through a number of questions that may make no sense to them.  Instead, if the device can support a format, we show the appropriate icon in the attachment list.  If it can't, we show a "?" icon.  The user guide explains the difference.  No user has seen this as a problem; initial feedback is that it's very, very rare that users look at attachments on their mobile devices (though that behaviour will no doubt evolve over time with the device capabilities).

Thus I'm following this debate from a somewhat more neutral standpoint than usual, though with a client-side-development attitude :)

cheers
Ben

---

From: "Vaudreuil, Greg M \(Greg\)" <[email-masked]>
Date: Fri, 22 Jun 2007 08:28:58 -0500

A colleague of mine pointed out that my earlier post accepting client-side estimates missed an important use-case around time-based media.

If a client knows a file is 2meg in MP3 format, and it requests the audio in AMR format, it has no ability to guess the converted object size.  MP3 can be encoded in a wide variety of bit-rates.

The client can figure the size if it knows the duration of the media in the object and the average bit-rate of an AMR codec, but duration is an optional MIME parameter that in my experience is rarely populated outside of voicemail.

So, the client needs richer information, not just the output format, but also a way of interpreting the input (original) size relative to the encoding.

This throws me back over the fence... a lightweight client can't be expected to know all the likely input formats.  It needs server support.

That support may be different than the current proposal, that is, maybe we can introduce a command to fetch a dynamic list of conversion ratio's as part of learning the servers capabilities.  

Greg V. 

-----Original Message-----
From: Ben Last [mailto:[email-masked]]
Sent: Friday, June 22, 2007 5:14 AM
To: [email-masked]
Subject: RE: [lemonade] Just say "Yes"

Yours pedantically
b

Ben Last
R&D Manager
EMCC Software Ltd

---

From: Mark Crispin <[email-masked]>
Date: Fri, 22 Jun 2007 08:55:21 -0700 (PDT)

> This throws me back over the fence... a lightweight client can't be expected to know all the likely input formats.  It needs server support.

In which case, require that the server calculate and report the exact 
size.  It is a server implementation problem to render the cost 
reasonable.

A lie is worse than no information at all.  NNTP lies, and those lies 
present a major hassles for me that are ameliorating only because netnews 
is dying.

-- Mark --

http://panda.com/mrc