Subject: [lemonade] WGLC's

From: Eric Burger <[email-masked]>
Date: Fri, 19 Nov 2004 19:15:07 -0500

I screwed up -- we were going to WGLC URLAUTH, BURL, and CATENATE in one
fell swoop.

The good news is there is no penalty for getting URLAUTH out early.

The bad news is this puts pressure on the BURL and CATENATE authors to catch
up :)

_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Alexey Melnikov <[email-masked]>
Date: Sat, 20 Nov 2004 15:25:21 +0000

>The bad news is this puts pressure on the BURL and CATENATE authors to catch
CATENATE will require at least one more revision, for several reasons:
1). I am writing examples to be added as discussed during the last IETF 
meeting. I will be sending them soon.
2). I've just realized that the CATENATE extension interacts with 
UIDPLUS and the CATENATE command must return APPENDUID if both are 
implemented. I suggest addition of the following text:

CATENATE Interaction with UIDPLUS Extension

   Servers which support both CATENATE and [UIDPLUS] MUST return the 
APPENDUID response code
   in the tagged OK response of a successful CATENATE command. The 
APPENDUID response code contains as
   arguments the UIDVALIDITY of the destination mailbox and the UID 
assigned to the catenated message.
   The syntax of the APPENDUID response code is described in [UIDPLUS].

Add to Normative References:

 [UIDPLUS] Myers, J., "IMAP4 UIDPLUS extension", RFC 2359, June 1988.

3). IANA considerations section is missing, something like:

IANA Considerations

   IMAP4 capabilities are registered by publishing a standards track or
   IESG approved experimental RFC.  The registry is currently located
   at:

      http://www.iana.org/assignments/imap4-capabilities

   This document defines the CATENATE IMAP capability.  IANA is requested
   to add this capability to the registry.

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 21 Nov 2004 22:28:16 +0000

>I screwed up -- we were going to WGLC URLAUTH, BURL, and CATENATE in one
One more thing. I've noticed that in the URLAUTH documents all URLs are 
transferred as quoted strings,
while in CATENATE they transferred as is (MAILBOX-REFERRAL also 
transfers them with no surrounding quotes).

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 21 Nov 2004 23:07:12 +0000

Lines not starting with "C: " or "S: " are continuations of the previous
lines.

Example 1: The following example demonstrates how a CATENATE client
      can replace an attachment in a draft message, without the need
      to download it to the client and upload it back.
      The original message (UID = 20) has the following structure:

       multipart/mixed MIME message with two body parts:
       1  - text/plain
       2  - application/x-zip-compressed

      C: A003 CATENATE Drafts (\Seen \Draft $MDNSent) {<N1>}
      S: + Ready for literal data
      C: Return-Path: <[email-masked]>
      C: Received: from [127.0.0.2] 
      C:           by rufus.example.org via TCP (internal) with ESMTPA;
      C:           Thu, 11 Nov 2004 16:57:07 +0000
      C: Message-ID: <[email-masked]>
      C: Date: Thu, 11 Nov 2004 16:57:05 +0000
      C: From: Bob Ar <[email-masked]>
      C: X-Accept-Language: en-us, en
      C: MIME-Version: 1.0
      C: To: [email-masked]
      C: Subject: A message with attachment
      C: Content-Type: multipart/mixed;
      C:               boundary="------------030308070208000400050907"
      C: 
      C: --------------030308070208000400050907
      C:  imap://imap.example.org/Drafts;UIDVALIDITY=385759045/;UID=20;section
         =1.1 {<N2>}
      S: + Ready for literal data
      C: --------------030308070208000400050907
      C:  imap://imap.example.org/Drafts;UIDVALIDITY=385759045/;UID=30 {<N3>}
      C: --------------030308070208000400050907--
      C:
      S: A003 OK CATENATE Completed

     Here <N1>, <N2> and <N3> are lengths of the corresponding literals.

<<Question: is there a way to reference HEADERs of a particular body part in
IMAP URLs?>>

Example 2: The following example demonstrates how CATENATE can be used
      to replace edited text in a draft message. The previous version
      of the draft is marked as \Deleted. Note, that the server also
      supports the UIDPLUS extension, so the APPENDUID response code
      is returned in the successful OK response to the CATENATE command.

      C: A003 CATENATE Drafts (\Seen \Draft $MDNSent) {<N4>}
      S: + Ready for literal data
      C: Return-Path: <[email-masked]>
      C: Received: from [127.0.0.2] 
      C:           by rufus.example.org via TCP (internal) with ESMTPA;
      C:           Thu, 11 Nov 2004 16:57:07 +0000
      C: Message-ID: <[email-masked]>
      C: Date: Thu, 12 Nov 2004 16:57:05 +0000
      C: From: Bob Ar <[email-masked]>
      C: X-Accept-Language: en-us, en
      C: MIME-Version: 1.0
      C: To: [email-masked]
      C: Subject: About our holiday trip
      C: Content-Type: multipart/mixed;
      C:               boundary="------------030308070208000400050907"
      C: 
      C: --------------030308070208000400050907
      C: Content-Type: text/plain; charset=us-ascii; format=flowed
      C: Content-Transfer-Encoding: 7bit
      C:
      C: Our travel agent has sent the updated schedule.
      C:
      C: Cheers,
      C: Bob
      C: --------------030308070208000400050907
      C:  imap://imap.example.org/Drafts;UIDVALIDITY=385759045/;UID=20;
         Section=1.2 {<N5>}
      C: --------------030308070208000400050907--
      C:
      S: A003 OK [APPENDUID 385759045 45] CATENATE Completed
      C: A004 UID STORE 20 +FLAGS.SILENT (\Deleted)
      S: A004 OK STORE completed

     Here <N4> and <N5> are lengths of the corresponding literals.

Example 3: The following example demonstrates how CATENATE can be used
      to strip attachments. Below a PowerPoint attachment was replaced
      by a small text part explaining that the attachment was stripped.

      C: A003 CATENATE Drafts (\Seen \Draft $MDNSent) {<N6>}
      S: + Ready for literal data
      C: Return-Path: <[email-masked]>
      C: Received: from [127.0.0.2] 
      C:           by rufus.example.org via TCP (internal) with ESMTPA;
      C:           Thu, 11 Nov 2004 16:57:07 +0000
      C: Message-ID: <[email-masked]>
      C: Date: Thu, 11 Nov 2004 16:57:05 +0000
      C: From: Bob Ar <[email-masked]>
      C: X-Accept-Language: en-us, en
      C: MIME-Version: 1.0
      C: To: [email-masked]
      C: Subject: My presentation
      C: Content-Type: multipart/mixed;
      C:               boundary="------------030308070208000400050903"
      C: 
      C: --------------030308070208000400050903
      C:  imap://imap.example.org/Drafts;UIDVALIDITY=385759045/;UID=21;
         section=1.1 {<N7>}
      S: + Ready for literal data
      C: --------------030308070208000400050903
      C: Content-type: text/plain; charset="us-ascii"
      C: Content-transfer-encoding: 7bit
      C: 
      C: This bodypart contained a Power Point presentation, that was
      C: deleted upon your request.
      C: --------------030308070208000400050903--
      C:
      S: A003 OK CATENATE Completed

     Here <N6> and <N7> are lengths of the corresponding literals.

Example 4: The following example demonstrates a failed CATENATE command.
     The server returns the BADURL response code to indicate that one
     of the provided URLs is invalid.
     This example also demonstrates how CATENATE can be used to construct
     a digest of several messages.

      C: A003 CATENATE Sent (\Seen $MDNSent) {<N8>}
      S: + Ready for literal data
      C: Return-Path: <[email-masked]>
      C: Received: from [127.0.0.2] 
      C:           by rufus.example.org via TCP (internal) with ESMTPA;
      C:           Thu, 11 Nov 2004 16:57:07 +0000
      C: Message-ID: <[email-masked]>
      C: Date: Thu, 21 Nov 2004 16:57:05 +0000
      C: From: Farren Oo <[email-masked]>
      C: X-Accept-Language: en-us, en
      C: MIME-Version: 1.0
      C: To: [email-masked]
      C: Subject: Digest of the mailing list for today
      C: Content-Type: multipart/digest;
      C:               boundary="------------030308070208000400050904"
      C: 
      C: --------------030308070208000400050904
      C:  imap://imap.example.org/INBOX;UIDVALIDITY=785799047/;UID=11467
          {<N9>}
      S: + Ready for literal data
      C: --------------030308070208000400050904
      C:  imap://imap.example.org/INBOX;UIDVALIDITY=785799047/;UID=113330;
          section=1.5.9 {<N10>}
      S: + Ready for literal data
      C: --------------030308070208000400050904
      C:  imap://imap.example.org/INBOX;UIDVALIDITY=785799047/;UID=11916
          {<N11>}
      S: + Ready for literal data
      C: --------------030308070208000400050904--
      C:
      S: A003 NO [BADURL imap://imap.example.org/INBOX;UIDVALIDITY=785799047/
         ;UID=113330;section=1.5.9] CATENATE has failed, one message expunged

     Here <N8>, <N9>, <N10> and <N11> are lengths of the corresponding literals.
_______________________________________________
lemonade mailing list
[email-masked]
https://www1.ietf.org/mailman/listinfo/lemonade

---

From: Mark Crispin <[email-masked]>
Date: Sun, 21 Nov 2004 15:56:10 -0800 (PST)

On Sun, 21 Nov 2004, Alexey Melnikov wrote:
> while in CATENATE they transferred as is (MAILBOX-REFERRAL also transfers 
> them with no surrounding quotes).

The usage in CATENATE and MAILBOX-REFERRAL only works as long as IMAP URLs 
can not contain "]" or spaces.  That currently seems to be true; but I 
don't consider it to be a prudent assumption.

By using astring, you don't need to have a separate rule that knows about 
the syntax of imapurl just to parse the command or its response.

I think that CATENATE should use astring as well.

MAILBOX-REFERRAL is rarely used, and it's possible with that to use a rule 
that says "everything up to the ] is the URL.

-- Mark --

http://staff.washington.edu/mrc
Science does not emerge from voting, party politics, or public debate.
Si vis pacem, para bellum.

---

From: Alexey Melnikov <[email-masked]>
Date: Sun, 21 Nov 2004 23:25:16 +0000

Cyrus Daboo wrote:

> Hi Alexey,
>
> --On November 20, 2004 15:25:21 +0000 Alexey Melnikov 
> <[email-masked]> wrote:
>
>> UIDPLUS
>> and the CATENATE command must return APPENDUID if both are 
>> implemented. I
> UIDPLUS is already mentioned in the -02 draft in the paragraph right 
> before section 4.

Oops, I've missed that.
One of my examples has the APPENDUID response code.

> However, the language there is not as strong as you suggest above, and 
> perhaps it should be.

I suggest at least changing "it will also return an APPENDUID response 
code in the tagged OK response"
to "it MUST return an APPENDUID ..."

> It might also be worthwhile explicitly making the point that CATENATE 
> does not support the MULTIAPPEND extension behaviour, given that so 
> many other aspects of CATENATE appear to be similar to APPEND.

Yes, I was thinking about this too. However I've decided that there is 
no such thing as MULTICATENATE :-), so nothing has to be said.

Alexey

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 22 Nov 2004 11:41:33 +0000

Philip has noticed that there are two minor bugs (missing "S: + Ready 
for literal data" in a few places.)
He also pointed out that there is a way to reference top level headers 
in an IMAP URL. I've updated the examples.

      C: A003 CATENATE Drafts (\Seen \Draft $MDNSent) imap://imap.example.
         org/Drafts;UIDVALIDITY=385759045/;UID=20;section=HEADER {<N1>}
      S: + Ready for literal data
      C: --------------030308070208000400050907
      C:  imap://imap.example.org/Drafts;UIDVALIDITY=385759045/;UID=20;section
         =1.1 {<N2>}
      S: + Ready for literal data
      C: --------------030308070208000400050907
      C:  imap://imap.example.org/Drafts;UIDVALIDITY=385759045/;UID=30 {<N3>}
      S: + Ready for literal data
      C: --------------030308070208000400050907--
      C:
      S: A003 OK CATENATE Completed

Example 2: The following example demonstrates how CATENATE can be used
      to replace edited text in a draft message, as well as header fields
      for the top level message part (e.g. Subject has changed). The
      previous version of the draft is marked as \Deleted. Note, that
      the server also supports the UIDPLUS extension, so the APPENDUID
      response code is returned in the successful OK response to the
      CATENATE command.

      C: A003 CATENATE Drafts (\Seen \Draft $MDNSent) imap://imap.example
         .org/Drafts;UIDVALIDITY=385759045/;UID=21;section=HEADER {<N6>}

---

From: Mark Crispin <[email-masked]>
Date: Mon, 22 Nov 2004 08:15:11 -0800 (PST)

Alexey -

I assume that you are going to replace those "{<N1>}" tokens with actual 
values in the final version, and the <N1> is just for convenience until 
the final form is determined.

Otherwise, I object to that.  Some fool is certain to send "{<N1>}" if he 
sees it in an example.

-- Mark --

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 22 Nov 2004 16:29:41 +0000

Mark Crispin wrote:

> Alexey -
>
> I assume that you are going to replace those "{<N1>}" tokens with 
> actual values in the final version, and the <N1> is just for 
> convenience until the final form is determined.

There is a note after each example:
     Here <N1>, <N2> and <N3> are lengths of the corresponding literals.

>
> Otherwise, I object to that.  Some fool is certain to send "{<N1>}" if 
> he sees it in an example.

And it will fail. I would hope that people at least test what they write.

Do you really want me to calculate proper lengths? I was trying to avoid 
this.

---

From: Alexey Melnikov <[email-masked]>
Date: Mon, 22 Nov 2004 17:48:12 +0000

By popular demand: examples with the proper literal sizes.

      C: A003 CATENATE Drafts (\Seen \Draft $MDNSent) imap://imap.example.
         org/Drafts;UIDVALIDITY=385759045/;UID=20;section=HEADER {40}
      S: + Ready for literal data
      C: --------------030308070208000400050907
      C:  imap://imap.example.org/Drafts;UIDVALIDITY=385759045/;UID=20;section
         =1.1 {40}
      S: + Ready for literal data
      C: --------------030308070208000400050907
      C:  imap://imap.example.org/Drafts;UIDVALIDITY=385759045/;UID=30 {44}
      S: + Ready for literal data
      C: --------------030308070208000400050907--
      C: 
      S: A003 OK CATENATE Completed

      C: A003 CATENATE Drafts (\Seen \Draft $MDNSent) {738}
         Section=1.2 {44}

         .org/Drafts;UIDVALIDITY=385759045/;UID=21;section=HEADER {40}
         section=1.1 {255}

      C: A003 CATENATE Sent (\Seen $MDNSent) {541}
          {40}
          section=1.5.9 {40}
          {44}

---

From: Mark Crispin <[email-masked]>
Date: Mon, 22 Nov 2004 10:03:09 -0800 (PST)

> And it will fail. I would hope that people at least test what they write.

My friend, I'm afraid that you are too optimistic... :-(

> Do you really want me to calculate proper lengths? I was trying to avoid 

Yes, I do.  Thanks.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Mon, 22 Nov 2004 10:36:03 -0800 (PST)

> By popular demand: examples with the proper literal sizes.

TNX 1.0E6 (thanks a million)!!!

By the way, those are good examples.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Mon, 22 Nov 2004 20:51:34 -0600

On 11/21/04 at 3:56 PM -0800, Mark Crispin wrote:

>>One more thing. I've noticed that in the URLAUTH documents all URLs 
>>are transferred as quoted strings, while in CATENATE they 
>>transferred as is (MAILBOX-REFERRAL also transfers them with no 
>>surrounding quotes).
>
>The usage in CATENATE and MAILBOX-REFERRAL only works as long as 
>IMAP URLs can not contain "]" or spaces.  That currently seems to be 
>true; but I don't consider it to be a prudent assumption.
>
>By using astring, you don't need to have a separate rule that knows 
>about the syntax of imapurl just to parse the command or its 
>response.
>

With responses, I agree that astring is definitely a better way to 
go. There's no reason to start parsing URLs in reponses. But in the 
command itself, I'm a bit nervous. Folks who make e-mail clients (let 
alone web browsers) screw up URL quoting conventions and literals all 
over the place. (How many times have you seen &amp; or %hh where they 
didn't belong?) In the command, you're going to have to parse the URL 
anyway, and I'd worry that backslashes are going to make their way 
into the URL parser if we use astring. Also, if we put astring in the 
CATENATE syntax, it's only going to be a textual reference to IMAPURL.

I'm not saying we absolutely shouldn't use astring, but it does make 
me worry a bit. I'd like some input from other folks before I change 
the syntax.
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Pete Resnick <[email-masked]>
Date: Mon, 22 Nov 2004 20:53:36 -0600

>       S: A003 OK CATENATE Completed

Is the space before each URL required?

pr

---

From: Mark Crispin <[email-masked]>
Date: Mon, 22 Nov 2004 19:15:50 -0800 (Pacific Standard Time)

On Mon, 22 Nov 2004, Pete Resnick wrote:
> With responses, I agree that astring is definitely a better way to go. 
> There's no reason to start parsing URLs in reponses. But in the command 
> itself, I'm a bit nervous. Folks who make e-mail clients (let alone web 
> browsers) screw up URL quoting conventions and literals all over the place.

Understood.  But this is IMAP quoting, not URL quoting.

> In the 
> command, you're going to have to parse the URL anyway,

Aye, there's the rub.  *Something* has to parse the URL.  It may not 
necessarily be the IMAP server; it may be a separate engine that the IMAP 
server asks to resolve the URL.

I very much do not want to add a URL parser to my IMAP server engine.

> and I'd worry that 
> backslashes are going to make their way into the URL parser if we use 
> astring.

I think that this is unlikely, considering the RFC 3501 rules that say 
that the only time you can have a backslash is <\"> for <"> and <\\> for 
<\>.  Unlike RFC [2]822, backslash is not a general quoting mechanism in 
IMAP, so "foo\bar" is a syntax error.

> Also, if we put astring in the CATENATE syntax, it's only going to 
> be a textual reference to IMAPURL.

No, you'd use the syntax rule in IMAP.  Or do I misunderstand what you're 
saying here?

> I'm not saying we absolutely shouldn't use astring, but it does make me worry 
> a bit. I'd like some input from other folks before I change the syntax.

I'm worried about a future in which CATENATE allows other URIs besides 
IMAP URLs, and those URIs have syntax which can be confused with other 
IMAP syntax.

An alternate solution is to define the URL argument to CATENATE as an 
atom, e.g.

    catenate = "CATENATE" SP mailbox [SP flag-list] [SP date-time]
                        1*(SP (literal / url))

    url      = atom
                ; contains imapurl as defined in [IMAPURL] and extended
                ; by [URLAUTH]

This only works if we can guarantee that url will never have any of these 
characters:
 	( ) { * % ] " \ SP CTL

That's right -- } and [ are valid atom characters in IMAP!  Anyway, I 
think that % is going to be a problem here.

My proposal is similar:

    catenate = "CATENATE" SP mailbox [SP flag-list] [SP date-time]
                        1*(SP (literal / url))

    url      = astring
                ; contains imapurl as defined in [IMAPURL] and extended
                ; by [URLAUTH]

The difference is that the server must handle a quoted string and a 
literal, and of course a client can send such if it needs to send a 
non-atom character.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Mon, 22 Nov 2004 21:27:25 -0600

Uh, nevermind. Brain blip. Of course the space is required.

On 11/22/04 at 8:53 PM -0600, Pete Resnick wrote:

>>imap://imap.example.org/Drafts;UIDVALIDITY=385759045/;UID=20;section
>>          =1.1 {40}

---

From: Mark Crispin <[email-masked]>
Date: Mon, 22 Nov 2004 19:28:39 -0800 (Pacific Standard Time)

> Is the space before each URL required?

Yes.  Arguments are space-delimited, not only in IMAP but also in 
CATENATE.  If you want to remove the space, you would have to do two 
things:

First, you need to change:
    catenate = "CATENATE" SP mailbox [SP flag-list] [SP date-time]
                        1*(SP (literal / imapurl))
to something like:
    catenate = "CATENATE" SP mailbox [SP flag-list] [SP date-time] SP
                        *argument (imapurl / literal)

    argument = (imapurl SP) / literal

Second, you'd have to pry consent from my cold dead hands..... :-)

-- Mark --

---

From: Philip Guenther <[email-masked]>
Date: Mon, 22 Nov 2004 19:47:11 -0800

Mark Crispin <[email-masked]> writes:
...
>No, you'd use the syntax rule in IMAP.  Or do I misunderstand what you're 

He's saying that the CATENATE syntax would no longer make direct
reference to a production from the IMAP-URL syntax.

>> me worry a bit. I'd like some input from other folks before I change

Can't use plain 'astring', as that would make the grammar ambiguous: is
a literal an imapurl or just text to include?

>An alternate solution is to define the URL argument to CATENATE as an 
...
>think that % is going to be a problem here.

Very much so: it would exclude all i18d mailbox names.

>                ; contains imapurl as defined in [IMAPURL] and extended

This is ambiguous.

I suppose url could be defined as 'quoted / atom', so that literals
would be text to include while quoted or atoms would be URLs, but that's
kinda gross.  Perhaps URL arguments to CATENATE should be preceeded by
a known atom, say:

   catenate = "CATENATE" SP mailbox [SP flag-list] [SP date-time]
                       1*(SP (literal / "URL" url))

   url      = astring

<joke>
Then it could be later expanded to support URNs and URIs too!
</joke>

Philip Guenther

---

From: Mark Crispin <[email-masked]>
Date: Mon, 22 Nov 2004 19:54:13 -0800 (Pacific Standard Time)

>               ; contains imapurl as defined in [IMAPURL] and extended

I like this.  Unambiguous syntax, nicely extensible, and less reliance 
upon magic.  I am not proud of the syntax of APPEND.

-- Mark --

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 23 Nov 2004 14:07:48 +0000

> upon magic.  I am not proud of the syntax of APPEND.

Do you really mean something like:

URL"imap://imap.example.org/Drafts;UIDVALIDITY=385759045/;UID=20"

or is there a space missing?

---

From: Mark Crispin <[email-masked]>
Date: Tue, 23 Nov 2004 09:01:28 -0800 (PST)

> or is there a space missing?

Yeah, I agree that a space should be added.

In the name of magic elimination, how about:

   catenate = "CATENATE" SP mailbox [SP flag-list] [SP date-time]
                       1*(SP (("TEXT" SP literal) / ("URL" SP url)))

or even:

   catenate = "CATENATE" SP mailbox SP parameters
                       1*(SP (("TEXT" SP literal) / ("URL" SP url)))

   parameters = ("(" parameter [1*(SP parameter)] ")") / "NIL"

   paramter = ("FLAGS" SP flag-list]) / ("DATE" date-time)

-- Mark --

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 23 Nov 2004 17:21:07 +0000

>   paramter = ("FLAGS" SP flag-list]) / ("DATE" date-time)

Let's break everything :-).

Seriously, I think the latter will eliminate convoluted logic for 
checking for optional flags and date. It also allows for easy 
extensibility in the future.

---

From: Mark Crispin <[email-masked]>
Date: Tue, 23 Nov 2004 11:00:07 -0800 (Pacific Standard Time)

> Let's break everything :-).

Indeed.

> Seriously, I think the latter will eliminate convoluted logic for checking 
> for optional flags and date. It also allows for easy extensibility in the 
> future.

That's what I thought.  Copying APPEND was a good idea initially, but I 
believe that we have stretched it beyond its reasonable limits.

I am not proud of the syntax of APPEND in IMAP.  It was the ugly result of 
evolution, not planned creation; and it shows.  It we were going to 
implement catenate as an extension syntax to APPEND, we'd have to go this 
route; but as MULTIAPPEND has precluded that (and is also very useful) we 
have to have a separate CATENATE command.

There is, however, another possibility.  We *could* defend a new APPEND, 
identified with an APPENDPLUS extension, with the following syntax that 
permits both CATENATE and MULTIAPPEND functionality:

append    =/  "APPEND" SP parameters SP mailbox 1*(SP appendtext)

parameters = "(" *parameter ")"

parameter = ("FLAGS" SP flag-list]) / ("DATE" date-time)

appendtext = "( ("TEXT" SP astring) / ("URL" SP astring) ")"

There are a few warts with this.  The parameters have to be before the 
mailbox to distinguish from the IMAP4 APPEND command; and empty parameters 
would be by the empty list () instead of the more IMAP traditional NIL.

But these are small warts compared to the benefits of the cleanup.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Tue, 23 Nov 2004 13:26:04 -0600

So I want to go back to the beginning because things seem to be 
getting worse as we go along:

>about the syntax of imapurl just to parse the command or its 

In the command itself, we don't have to worry about "]". All we have 
to worry about is space. And space is thoroughly illegal in *any* 
URIs, not just IMAP ones. So there is no requirement to know the 
syntax of imapurl just to parse the CATENATE command: You look for 
the space. If the character after it is a "{", then you've got a 
literal. Else, it's a URI. Read until next space or EOL. Anything 
that turns out later to not be a legit URI (even if it's due to a bad 
client inserting a space into a URL where it doesn't belong) will 
simply generate a NO.

By putting it in an astring, you're requiring implementations to go 
through and esacpe backslashes and double-quotes. We all know that 
implementations will screw this up. Some will do stupid things like 
"%" escaping them instead of "\" escaping. Some will fail to do it at 
all. We'll end up with some server implementations trying to be 
"helpful". I think the chances for foul-up are much greater by 
requiring them to be astrings than if we put them in in the raw.

Now, the response code is a different story. We might have to worry 
about "]". But in the case of the response code, we're pretty much 
talking about trace information, so if a server screws up the 
escaping there, I'm not nearly so concerned.

With only my "member of the WG" hat on: My feeling is that we should 
stick to the current syntax on the command but make the response 
contain an astring.

pr

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 23 Nov 2004 19:38:52 +0000

Pete Resnick wrote:

> talking about trace information, so if a server screws up the escaping 
> there, I'm not nearly so concerned.
>

You can't return <"> or "{" in a response code, so we can't use 
"astring"/"quoted"/"literal". We can only use something like "atom".

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 23 Nov 2004 19:42:49 +0000

Pete Resnick wrote:

> implementations will screw this up.

As I understand it, just doing <"> url <"> should suffice.

Anyway, escape rules for quoted strings are well understood. Borrowing 
existing function for sending a quoted string from c-client or Cyrus 
should be no brainer.

> Some will do stupid things like "%" escaping them instead of "\" 
> escaping. Some will fail to do it at all. We'll end up with some 
> server implementations trying to be "helpful". I think the chances for 
> foul-up are much greater by requiring them to be astrings than if we 
> put them in in the raw.

Have you noticed how many times recently we try to work around dumb 
implementors?

---

From: Pete Resnick <[email-masked]>
Date: Tue, 23 Nov 2004 15:56:03 -0600

>As I understand it, just doing <"> url <"> should suffice.

"Should" is the key. The concern is that you're going to get a "\" or 
a <"> in a URL, at which point you're going to have to escape them 
with "\". So the algorithm will inevitably be to do the escaping.

>Anyway, escape rules for quoted strings are well understood. 
>Borrowing existing function for sending a quoted string from 
>c-client or Cyrus should be no brainer.

>Have you noticed how many times recently we try to work around dumb 

Implementors have no shortage in their ability to produce dumb 
implementations. In this case, I think there is less chance of us 
having to work around a dumb implementor by saying "insert space 
followed by literal or URL" instead of "insert space followed by 
literal or appropriately encoded URL."

pr

---

From: Philip Guenther <[email-masked]>
Date: Mon, 22 Nov 2004 19:34:43 -0800

>Is the space before each URL required?

Yes.  The grammar certainly requires a space before each imapurl.
Leaving them out only when the previous argument was a literal would
make the grammar more complicated for no gain and in fact make it
different from how other IMAP commands are parsed when literals are
used.

Philip Guenther

---

From: Alexey Melnikov <[email-masked]>
Date: Wed, 24 Nov 2004 12:01:36 +0000

From RFC 1738:

 >   Unsafe:
 >
 >   Characters can be unsafe for a number of reasons.  The space
 >   character is unsafe because significant spaces may disappear and
 >   insignificant spaces may be introduced when URLs are transcribed or
 >   typeset or subjected to the treatment of word-processing programs.
 >   The characters "<" and ">" are unsafe because they are used as the
 >   delimiters around URLs in free text; the quote mark (""") is used to
 >   delimit URLs in some systems.  The character "#" is unsafe and should
 >   always be encoded because it is used in World Wide Web and in other
 >   systems to delimit a URL from a fragment/anchor identifier that might
 >   follow it.  The character "%" is unsafe because it is used for
 >   encodings of other characters.  Other characters are unsafe because
 >   gateways and other transport agents are known to sometimes modify
 >   such characters. These characters are "{", "}", "|", "\", "^", "~",
 >   "[", "]", and "`".
 >
 >   All unsafe characters must always be encoded within a URL.

My reading of this is that unencoded "\" and <"> are not allowed in 
URLs, so you must ignore "should" in my previous statement ;-).

The server has to check for URL validity anyway. The main question is - 
should this cause IMAP protocol error, i.e. the server tries to decode 
an invalid URL sent as a quoted string and fails (<"> <url> <">), 
because some characters are not escaped, or should IMAP parsing succeed 
and the URL be rejected during URL validity check.

>> Borrowing existing function for sending a quoted string from c-client 
>> or Cyrus should be no brainer.
>

---

From: Pete Resnick <[email-masked]>
Date: Wed, 24 Nov 2004 12:41:06 -0600

>URLs, so you must ignore "should" in my previous statement ;-).

Well, right, but the same is true of "]" and " ", and if everyone 
followed the rules, we wouldn't even need to have this conversation: 
IMAP servers implementations wouldn't need the astring because they 
could simply look for the next space and be done with it. And if we 
really trust that client implementors will send only valid URLs, we 
should just leave things as they are.

>The server has to check for URL validity anyway. The main question 
>is - should this cause IMAP protocol error, i.e. the server tries to 
>decode an invalid URL sent as a quoted string and fails (<"> <url> 
><">), because some characters are not escaped, or should IMAP 
>parsing succeed and the URL be rejected during URL validity check.

So, as I was saying before, the only chance of an IMAP protocol error 
in the command itself is if a client sends a URL with a space in it. 
I think the odds of that happening are lower than an client screwing 
up astring escaping of a URL, so my opinion is to leave the command 
syntax alone. I am less sanguine about the response code (since a 
client could screw that up by including a "]"), but I'm willing to 
buy that the odds are low enough there too. Do we really think that 
complicating the syntax is worth it for clients who are able to get 
astring escaping correct but can't get URL escaping right?
--
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Philip Guenther <[email-masked]>
Date: Wed, 24 Nov 2004 11:06:09 -0800

>should just leave things as they are.

The problems wasn't so much with "]" or " ", but rather with "(", ")",
"*", and "%", all of which are legal in URLs (and mailbox names) but not
in IMAP atoms.

Philip Guenther

---

From: Mark Crispin <[email-masked]>
Date: Wed, 24 Nov 2004 11:25:24 -0800 (PST)

On Wed, 24 Nov 2004, Pete Resnick wrote:
> Do we really think that complicating the syntax is worth it for clients 
> who are able to get astring escaping correct but can't get URL escaping 
> right?

My opinion: it complicates the syntax worse to add a new type of token 
that is "everything up to the next space" (but not really, since it 
actually is the entire set of rules for URL), than it does to use an 
existing syntax rule -- astring.

As a server implementor, I very much want to parse an astring rather than 
to write a completely new syntax rule.

If the consensus is to do the "everything up to the next space", then I 
insist that a completely new syntax rule be written for IMAP that defines 
this rule.  Something like:

   urlatom = 1*urlatomchar

   urlatomchar = ATOM-CHAR / "(" / ")" / "{" / CTL / list-wildcards /
                 quoted-specials / resp-specials / urlhighchar

   urlhighchar = %x80-ff

Not nice?  Didn't think so.  But that is, literally, the "everything up to 
the next space rule".  If we dispense with any reference to other IMAP 
rules, then we may have something like:

   urlatom = 1*urlatomchar

   urlatomchar = %x01-09 / %x0b / %0x0d-1f / %0x21-ff

Not so nice either.

Or, if we exclude those characters that can't be in URLs (which is quite a 
bit different than "everything up to the next space"), we have an entirely 
different set of rules.  But then we can't use "everything up to the next 
space".  And, of course, it has to track any changes made to URLs.

Now, doesn't astring seem to be sooooooo much less complicated now?  :-)

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Wed, 24 Nov 2004 11:27:31 -0800 (PST)

On Wed, 24 Nov 2004, Philip Guenther wrote:
> The IMAP base spec doesn't (that I see) require the server to, for
> example, throw away everything up to the next newline when it receives a
> CREATE command with a misquoted mailbox name

Once upon a time, it did.

This was widely considered to have been a mistake, and it was amended. 
There have been no "rest of line" or "everything up to the next space" 
rules in IMAP for over a decade.

Hence my attempt to resist adding such now.

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Wed, 24 Nov 2004 13:44:31 -0600

(* Pete has the "aha!" moment *)

On 11/24/04 at 11:27 AM -0800, Mark Crispin wrote:

>This was widely considered to have been a mistake, and it was 
>amended. There have been no "rest of line" or "everything up to the 
>next space" rules in IMAP for over a decade.

OK, so this is about current parsing rules being relatively well 
defined and introducing the new one *does* complicate things. This 
makes a little more sense now.

But let me get back to Philip's earlier comment, which I don't get:

On 11/22/04 at 7:47 PM -0800, Philip Guenther wrote:

>Can't use plain 'astring', as that would make the grammar ambiguous: 
>is a literal an imapurl or just text to include?

I don't see how this is ambiguous. A literal is *always* text to 
include. Literals begin with "{". If it doesn't begin with "{", it's 
an astring which must be a URL. Care to explain why astring is not 
sufficient?

pr

---

From: Pete Resnick <[email-masked]>
Date: Wed, 24 Nov 2004 13:55:06 -0600

[And everybody pause while Pete answer's his own question again....]

>an astring which must be a URL. Care to explain why astring is not 

Because astring contains string which contains literal.

Now editing in Philip's syntax.
--
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Philip Guenther <[email-masked]>
Date: Wed, 24 Nov 2004 11:59:40 -0800

>an astring which must be a URL. Care to explain why astring is not 

reduce/reduce conflict:

astring         = 1*ASTRING-CHAR / string
string          = quoted / literal

So a literal is an astring too.  If you're thinking that a literal
would be 'reduced' directly as text to include instead of as a form
of astring holding a URL, then that should be directly expressed
by the grammar, ala:

   url      = atom / quoted

I find that unpleasant, as it creates a situation where a quoted
string has different semantics than a literal with the same value.
Thus my suggestion to 'tag' URLs and use astring for them.

Philip Guenther

---

From: Pete Resnick <[email-masked]>
Date: Wed, 24 Nov 2004 14:11:24 -0600

>Now editing in Philip's syntax.

OK, taking Mark's, Alexey's, and Philip's recommendations and a 
little ABNF cleanup, does this make everyone queasy or happy?

catenate = "CATENATE" SP mailbox SP parameters 1*(SP (text-literal / url))

parameters = ("(" parameter *(SP parameter) ")") / "NIL"

parameter = ("FLAGS" SP flag-list) / ("DATE" SP date-time)

text-literal = "TEXT" SP literal

url = "URL" SP astring
       ; astring contains an imapurl as defined by <xref target="RFC2192"/>

badurl_response_code = "BADURL" SP astring
       ; astring contains an imapurl as defined by <xref target="RFC2192"/>
-- 
Pete Resnick <http://www.qualcomm.com/~presnick/>
QUALCOMM Incorporated - Direct phone: (858)651-4478, Fax: (858)651-1102

---

From: Philip Guenther <[email-masked]>
Date: Wed, 24 Nov 2004 12:58:07 -0800

One issue: response codes can't contain literals.  When this was
recently discussed on the IMAP list for the BADCHARSET case, it resulted
in an erratum that adds a
  charset = atom / quoted
production, then used that in both the BADCHARSET response code and for
the CHARSET argument to SEARCH.

That precedent would suggest having both badurl_response_code and url
refer to an "atom / quoted" production.

<pause to listen to Pete tear his hair out>

Tweaking a couple names to be less generic, how about:

command-auth     =/ catenate

catenate         = "CATENATE" SP mailbox SP catenate-params
                   1*(SP (text-literal / catenate-url))

catenate-params  = ("(" catenate-param *(SP catenate-param) ")") / "NIL"

catenate-param   = ("FLAGS" SP flag-list) / ("DATE" SP date-time)

catenate-url     = "URL" SP url

url              = atom / quoted
                 ; contains an imapurl as defined by
                 ; <xref target="RFC2192"/>

resp-text-code   =/ badurl-resp-code

badurl-resp-code = "BADURL" SP url

Philip Guenther

(Hmm, when the I18N stuff is finished and the SORT/THREAD rfc is
unblocked, the latter should be edited to reference the 'charset'
non-terminal too.)

---

From: Pete Resnick <[email-masked]>
Date: Wed, 24 Nov 2004 15:01:17 -0600

On 11/24/04 at 12:58 PM -0800, Philip Guenther wrote:

>That precedent would suggest having both badurl_response_code and 
>url refer to an "atom / quoted" production.

If we're going to go down that road, is there a reason to introduce 
the "URL" and "TEXT" (or for that matter "FLAGS" and "DATE") tags, 
since there won't be any ambiguity between literal and url?

pr

---

From: Mark Crispin <[email-masked]>
Date: Wed, 24 Nov 2004 13:16:23 -0800 (Pacific Standard Time)

On Wed, 24 Nov 2004, Pete Resnick wrote:
>> That precedent would suggest having both badurl_response_code and url refer 
>> to an "atom / quoted" production.
> If we're going to go down that road, is there a reason to introduce the "URL" 
> and "TEXT" (or for that matter "FLAGS" and "DATE") tags, since there won't be 
> any ambiguity between literal and url?

Unlike commands, there is precedent for response codes to use "rest of 
code".  So, yes, URLs would use a different production for response code 
and command.

-- Mark --

---

From: Mark Crispin <[email-masked]>
Date: Wed, 24 Nov 2004 13:19:37 -0800 (Pacific Standard Time)

On Wed, 24 Nov 2004, Pete Resnick wrote:
> OK, taking Mark's, Alexey's, and Philip's recommendations and a little ABNF 
> cleanup, does this make everyone queasy or happy?

Very close.  Just one change to badurl-response-code:

badurl-response-code = "BADURL" SP 1*<any TEXT-CHAR except "]">
      ; value contains an imapurl as defined by <xref target="RFC2192"/>

-- Mark --

---

From: Philip Guenther <[email-masked]>
Date: Wed, 24 Nov 2004 13:24:45 -0800

>since there won't be any ambiguity between literal and url?

...thus my hair tearing comment.

IMHO, there's a difference between simply restricting how certain values
are expressed and changing the semantics of a value depending on how it
is expressed.  I believe APPEND is the only command where the semantics
of a given argument depend upon whether it is a quoted string or a
literal.  We should not replicate that situation.

Hmm, that actually suggests an ambiguity of sorts in the tagless syntax:
if the first thing after the flag list is a quoted string, you have to
examine the value of the string to decide whether it's a date-time or a
URL.

Please, let's keep the tagged syntax.

Philip Guenther

---

From: Mark Crispin <[email-masked]>
Date: Wed, 24 Nov 2004 13:33:17 -0800 (Pacific Standard Time)

On Wed, 24 Nov 2004, Philip Guenther wrote:
> I believe APPEND is the only command where the semantics

Strong agreement.

APPEND is a wart on the beautiful face of IMAP.  It is too late to remote 
that wart, but we can avoid adding another one. :-)

> Please, let's keep the tagged syntax.

Hear, hear!

-- Mark --

---

From: Pete Resnick <[email-masked]>
Date: Wed, 24 Nov 2004 15:47:01 -0600

>badurl-response-code = "BADURL" SP 1*<any TEXT-CHAR except "]">

OK, except that I did the explicit ABNF for "any TEXT-CHAR except 
']'" in a separate url-resp-text:

badurl_response_code = "BADURL" SP url-text

url-resp-text= 1*(%x01-09 / %x0B-0C / %x0E-5B / %x5D-FE)
           ; Any TEXT-CHAR except "]"

Now, what should the description of the command be? That is:

    Arguments:     mailbox name
                   OPTIONAL flag parenthesized list
                   OPTIONAL date/time string
                   one or more message parts to catenate, specified as:
                                  message literal
                                                 or
                                  message (or message part) URL

I can come up with the descriptive text, but I'd appreciate help on 
what would make the above clearest.

pr

---

From: Mark Crispin <[email-masked]>
Date: Wed, 24 Nov 2004 14:24:18 -0800 (Pacific Standard Time)

On Wed, 24 Nov 2004, Pete Resnick wrote:
> OK, except that I did the explicit ABNF for "any TEXT-CHAR except ']'" in a 
> separate url-resp-text:

Fine with me!

>                                 message (or message part) URL

I suggest leaving it at "one or more message parts to catenate", and then 
let the textual description below describe it further.  That is:

   Arguments:     mailbox name
                  OPTIONAL flag parenthesized list
                  OPTIONAL date/time string
                  one or more message parts to catenate
 	[. . .]

A message part consists of a message part name followed by the message 
part value.  The current part names are URL and TEXT.

The URL part blah blah blah

The TEXT part blah blah blah

My last suggestion is to consider a word other than part, to avoid 
confusion with MIME parts (since RFC 3501 uses "part" in that context).
Perhaps "message text component"?

-- Mark --

---

From: Philip Guenther <[email-masked]>
Date: Wed, 24 Nov 2004 15:01:12 -0800

...

Those should be replaced with
		   message parameter list or NIL

The currently defined message parameters are FLAGS and DATE.  etc.

Hmm, the new syntax would make extension to multi-message injection
unambiguous: an open paren or NIL tag would start the description of the
next message to catenate.  Did we decide "not going there"?

Philip Guenther