Subject: [lemonade] CAPABILITY IMAP command parsing fails....

From: Samsung SVL <[email-masked]>
Date: Tue, 5 Nov 2013 08:25:30 -0800 (PST)

Hi All,

Could anyone please guide me regarding the below issue ? 

Gmail and iCloud/me.com server sends IMAP CAPABILITY response in different format. iCloud.com server sends (LF instead of CR). Because of this client parsing fails. 
Most of the IMAP server sends in the same format where Gmail follows. Please guide me which one is correct as per the IMAP standard. 

Please find the log snippet:

Gmail:
-----
1-05 11:15:50.182 31946 32196 D Email   : open :: socket openjava.io.BufferedInputStream@42b943d8 | java.io.BufferedOutputStream@42b94898
11-05 11:15:50.202 31946 32196 D Email   : <<< #null# ["OK", "Gimap ready for requests from 71.172.250.42 p43if11821125yhp.24"]
11-05 11:15:50.202 31946 32196 D Email   : >>> 1 CAPABILITY
11-05 11:15:50.312 31946 32196 D Email   : <<< #null# ["CAPABILITY", "IMAP4rev1", "UNSELECT", "IDLE", "NAMESPACE", "QUOTA", "ID", "XLIST", "CHILDREN", "X-GM-EXT-1", "XYZZY", "SASL-IR", "AUTH=XOAUTH", "AUTH=XOAUTH2", "AUTH=PLAIN", "AUTH=PLAIN-CLIENTTOKEN"]
11-05 11:15:50.332 31946 32196 D Email   : <<< #1# ["OK", "Thats all she wrote! p43if11821125yhp.24"]
11-05 11:15:50.352 31946 32196 D Email   : >>> 2 ID ("name" "com.android.email" "os" "android" "os-version" "4.3; JSS15J" "vendor" "samsung" "x-android-device-model" "SM-N900V" "AGUID" "f5XkjdDr1zV/k3pCU/ErrWLM/qo=")
11-05 11:15:50.482 31946 32196 D Email   : <<< #null# ["ID", ["name", "GImap", "vendor", "Google, Inc.", "support-url", "http://support.google.com/mail", "remote-host", "71.172.250.42", "connection-token", "p43if11821125yhp.24"]]
11-05 11:15:50.492 31946 32196 D Email   : <<< #2# ["OK", "Success p43if11821125yhp.24"]
11-05 11:15:50.502 31946 32196 D Email   : >>> [IMAP command redacted]
11-05 11:15:50.672 31946 32196 D Email   : <<< #null# ["CAPABILITY", "IMAP4rev1", "UNSELECT", "IDLE", "NAMESPACE", "QUOTA", "ID", "XLIST", "CHILDREN", "X-GM-EXT-1", "UIDPLUS", "COMPRESS=DEFLATE", "ENABLE", "MOVE", "CONDSTORE", "ESEARCH"]
11-05 11:15:50.682 31946 32196 D Email   : <<< #3# ["OK", "[email-masked] jaytestimap authenticated (Success)"]
11-05 11:15:50.682 31946 32196 D Email   : >>> 4 CAPABILITY
11-05 11:15:50.822 31946 32196 D Email   : <<< #null# ["CAPABILITY", "IMAP4rev1", "UNSELECT", "IDLE", "NAMESPACE", "QUOTA", "ID", "XLIST", "CHILDREN", "X-GM-EXT-1", "UIDPLUS", "COMPRESS=DEFLATE", "ENABLE", "MOVE", "CONDSTORE", "ESEARCH"]
11-05 11:15:50.822 31946 32196 D Email   : <<< #4# ["OK", "Success"]
11-05 11:15:50.832 31946 32196 D Email   : Capability : [CAPABILITY,IMAP4rev1,UNSELECT,IDLE,NAMESPACE,QUOTA,ID,XLIST,CHILDREN,X-GM-EXT-1,UIDPLUS,COMPRESS=DEFLATE,ENABLE,MOVE,CONDSTORE,ESEARCH]

iCloud:
-------
11-05 10:50:52.462 29603 29988 D Email   : open :: socket openjava.io.BufferedInputStream@43a726a8 | java.io.BufferedOutputStream@43a72b68
11-05 10:50:52.502 29603 29988 D Email   : <<< #null# ["OK", ["CAPABILITY", "st11p00mm-iscream023", "1S", "XAPPLEPUSHSERVICE", "IMAP4", "IMAP4rev1", "SASL-IR", "AUTH=ATOKEN", "AUTH=PLAIN"], "iSCREAM ready to rumble (1S:1092) st11p00mm-iscream023 [42:4469:15:50:53:39]"]
11-05 10:50:52.502 29603 29988 D Email   : >>> 1 CAPABILITY
11-05 10:50:52.552 29603 29988 D Email   : <<< #null# ["CAPABILITY", "st11p00mm-iscream023", "1S", "XAPPLEPUSHSERVICE", "IMAP4", "IMAP4rev1", "SASL-IR", "AUTH=ATOKEN", "AUTH=PLAIN"]
11-05 10:50:52.562 29603 29988 D Email   : <<< #1# ["OK", "!!"]
11-05 10:50:52.582 29603 29988 D Email   : >>> [IMAP command redacted]
11-05 10:50:52.682 29603 29988 D Email   : <<< #2# ["OK", ["CAPABILITY", "XAPPLEPUSHSERVICE", "IMAP4", "IMAP4rev1", "ACL", "QUOTA", "LITERAL+", "NAMESPACE", "UIDPLUS", "CHILDREN", "BINARY", "UNSELECT", "SORT", "CATENATE", "URLAUTH", "LANGUAGE", "ESEARCH", "ESORT", "THREAD=ORDEREDSUBJECT", "THREAD=REFERENCES", "CONDSTORE", "ENABLE", "CONTEXT=SEARCH", "CONTEXT=SORT", "WITHIN", "SASL-IR", "SEARCHRES", "XSENDER", "X-NETSCAPE", "XSERVERINFO", "X-SUN-SORT", "ANNOTATE-EXPERIMENT-1", "X-UNAUTHENTICATE", "X-SUN-IMAP", "X-ANNOTATEMORE", "XUM1", "ID", "IDLE"], "User test.samus logged in"]
11-05 10:50:52.682 29603 29988 D Email   : >>> 3 CAPABILITY
11-05 10:50:52.742 29603 29988 W Email   : Exception detected: Expected 000a (
)1-05 10:50:52.742 29603 29988 W Email   : ) but got 000d (
11-05 10:50:52.752 29603 29988 E DiscourseLogger: Last activities: [Current Position - 8/64]
11-05 10:50:52.752 29603 29988 W DiscourseLogger: * OK [CAPABILITY st11p00mm-iscream023 1S XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 SASL-IR AUTH=ATOKEN AUTH=PLAIN] iSCREAM ready to rumble (1S:1092) st11p00mm-iscream023 [42:4469:15:50:53:39]
11-05 10:50:52.762 29603 29988 W DiscourseLogger: 1 CAPABILITY
11-05 10:50:52.762 29603 29988 W DiscourseLogger: * CAPABILITY st11p00mm-iscream023 1S XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 SASL-IR AUTH=ATOKEN AUTH=PLAIN 
11-05 10:50:52.772 29603 29988 W DiscourseLogger: 1 OK !!
11-05 10:50:52.772 29603 29988 W DiscourseLogger: [IMAP command redacted]
11-05 10:50:52.772 29603 29988 W DiscourseLogger: 2 OK [CAPABILITY XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES CONDSTORE ENABLE CONTEXT=SEARCH CONTEXT=SORT WITHIN SASL-IR SEARCHRES XSENDER X-NETSCAPE XSERVERINFO X-SUN-SORT ANNOTATE-EXPERIMENT-1 X-UNAUTHENTICATE X-SUN-IMAP X-ANNOTATEMORE XUM1 ID IDLE] User test.samus logged in
11-05 10:50:52.772 29603 29988 W DiscourseLogger: 3 CAPABILITY
11-05 10:50:52.782 29603 29988 W DiscourseLogger: * CAPABILITY XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS CHILDREN BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT THREAD=ORDEREDSUBJECT THREAD=REFERENCES CONDSTORE ENABLE CONTEXT=SEARCH CONTEXT=SORT WITHIN SASL-IR SEARCHRES XSENDER X-NETSCAPE XSERVERINFO X-SUN-SORT ANNOTATE-EXPERIMENT-1 X-UNAUTHENTICATE X-SUN-IMAP X-ANNOTATEMORE XUM1 ID IDLE
11-05 10:50:52.782 29603 29988 W System.err: java.io.IOException: Expected 000a (
)1-05 10:50:52.792 29603 29988 W System.err: ) but got 000d (
11-05 10:50:52.802 29603 29988 W System.err: at com.android.email.mail.store.imap.ImapResponseParser.expect(ImapResponseParser.java:237)
11-05 10:50:52.802 29603 29988 W System.err: at com.android.email.mail.store.imap.ImapResponseParser.parseElement(ImapResponseParser.java:389)
11-05 10:50:52.802 29603 29988 W System.err: at com.android.email.mail.store.imap.ImapResponseParser.parseElements(ImapResponseParser.java:472)
11-05 10:50:52.802 29603 29988 W System.err: at com.android.email.mail.store.imap.ImapResponseParser.parseResponse(ImapResponseParser.java:350)
11-05 10:50:52.802 29603 29988 W System.err: at com.android.email.mail.store.imap.ImapResponseParser.readResponse(ImapResponseParser.java:174)
11-05 10:50:52.802 29603 29988 W System.err: at com.android.email.mail.store.imap.ImapResponseParser.readResponse(ImapResponseParser.java:203)
11-05 10:50:52.812 29603 29988 W System.err: at com.android.email.mail.store.ImapStore$ImapConnection.executeSimpleCommand(ImapStore.java:2636)
11-05 10:50:52.812 29603 29988 W System.err: at com.android.email.mail.store.ImapStore$ImapConnection.executeSimpleCommand(ImapStore.java:2626)
11-05 10:50:52.812 29603 29988 W System.err: at com.android.email.mail.store.ImapStore$ImapConnection.doLogin(ImapStore.java:2806)
11-05 10:50:52.812 29603 29988 W System.err: at com.android.email.mail.store.ImapStore$ImapConnection.open(ImapStore.java:2453)
11-05 10:50:52.812 29603 29988 W System.err: at com.android.email.mail.store.ImapStore.checkSettings(ImapStore.java:650)
11-05 10:50:52.822 29603 29988 W System.err: at com.android.email.activity.setup.AccountCheckSettingsFragment$AccountCheckTask.doInBackground(AccountCheckSettingsFragment.java:759)
11-05 10:50:52.822 29603 29988 W System.err: at com.android.email.activity.setup.AccountCheckSettingsFragment$AccountCheckTask.doInBackground(AccountCheckSettingsFragment.java:1)
11-05 10:50:52.822 29603 29988 W System.err: at android.os.AsyncTask$2.call(AsyncTask.java:287)
11-05 10:50:52.822 29603 29988 W System.err: at java.util.concurrent.FutureTask.run(FutureTask.java:234)
11-05 10:50:52.822 29603 29988 W System.err: at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
11-05 10:50:52.822 29603 29988 W System.err: at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)
11-05 10:50:52.822 29603 29988 W System.err: at java.lang.Thread.run(Thread.java:841)

Regards,
Jayantheesh

---

From: Alexey Melnikov <[email-masked]>
Date: Tue, 05 Nov 2013 10:10:39 -0800

On 05/11/2013 08:25, Samsung SVL wrote:
> Gmail and iCloud/me.com server sends IMAP CAPABILITY response in 
> different format. iCloud.com server sends (LF instead of CR). Because 
> of this client parsing fails.
> Most of the IMAP server sends in the same format where Gmail follows. 
> Please guide me which one is correct as per the IMAP standard.
This looks like a bug in iCloud, as RFC 3501 requires use of CRLF at the 
end of each response line and not just LF.
> 1-05 11:15:50.182 31946 32196 D Email   : open :: socket 
> openjava.io.BufferedInputStream@42b943d8 | 
> java.io.BufferedOutputStream@42b94898
> 11-05 11:15:50.202 31946 32196 D Email   : <<< #null# ["OK", "Gimap 
> ready for requests from 71.172.250.42 p43if11821125yhp.24"]
> 11-05 11:15:50.312 31946 32196 D Email   : <<< #null# ["CAPABILITY", 
> "IMAP4rev1", "UNSELECT", "IDLE", "NAMESPACE", "QUOTA", "ID", "XLIST", 
> "CHILDREN", "X-GM-EXT-1", "XYZZY", "SASL-IR", "AUTH=XOAUTH", 
> "AUTH=XOAUTH2", "AUTH=PLAIN", "AUTH=PLAIN-CLIENTTOKEN"]
> 11-05 11:15:50.352 31946 32196 D Email   : >>> 2 ID ("name" 
> "com.android.email" "os" "android" "os-version" "4.3; JSS15J" "vendor" 
> "samsung" "x-android-device-model" "SM-N900V" "AGUID" 
> "f5XkjdDr1zV/k3pCU/ErrWLM/qo=")
> 11-05 11:15:50.482 31946 32196 D Email   : <<< #null# ["ID", ["name", 
> "GImap", "vendor", "Google, Inc.", "support-url", 
> "http://support.google.com/mail", "remote-host", "71.172.250.42", 
> "connection-token", "p43if11821125yhp.24"]]
> 11-05 11:15:50.672 31946 32196 D Email   : <<< #null# ["CAPABILITY", 
> "IMAP4rev1", "UNSELECT", "IDLE", "NAMESPACE", "QUOTA", "ID", "XLIST", 
> "CHILDREN", "X-GM-EXT-1", "UIDPLUS", "COMPRESS=DEFLATE", "ENABLE", 
> "MOVE", "CONDSTORE", "ESEARCH"]
> 11-05 11:15:50.682 31946 32196 D Email   : <<< #3# ["OK", 
> "[email-masked] jaytestimap authenticated (Success)"]
> 11-05 11:15:50.822 31946 32196 D Email   : <<< #null# ["CAPABILITY", 
> "IMAP4rev1", "UNSELECT", "IDLE", "NAMESPACE", "QUOTA", "ID", "XLIST", 
> "CHILDREN", "X-GM-EXT-1", "UIDPLUS", "COMPRESS=DEFLATE", "ENABLE", 
> "MOVE", "CONDSTORE", "ESEARCH"]
> 11-05 11:15:50.832 31946 32196 D Email   : Capability : 
> [CAPABILITY,IMAP4rev1,UNSELECT,IDLE,NAMESPACE,QUOTA,ID,XLIST,CHILDREN,X-GM-EXT-1,UIDPLUS,COMPRESS=DEFLATE,ENABLE,MOVE,CONDSTORE,ESEARCH]
>
>
> iCloud:
> -------
> 11-05 10:50:52.462 29603 29988 D Email   : open :: socket 
> openjava.io.BufferedInputStream@43a726a8 | 
> java.io.BufferedOutputStream@43a72b68
> 11-05 10:50:52.502 29603 29988 D Email   : <<< #null# ["OK", 
> ["CAPABILITY", "st11p00mm-iscream023", "1S", "XAPPLEPUSHSERVICE", 
> "IMAP4", "IMAP4rev1", "SASL-IR", "AUTH=ATOKEN", "AUTH=PLAIN"], 
> "iSCREAM ready to rumble (1S:1092) st11p00mm-iscream023 
> [42:4469:15:50:53:39]"]
> 11-05 10:50:52.552 29603 29988 D Email   : <<< #null# ["CAPABILITY", 
> "st11p00mm-iscream023", "1S", "XAPPLEPUSHSERVICE", "IMAP4", 
> "IMAP4rev1", "SASL-IR", "AUTH=ATOKEN", "AUTH=PLAIN"]
> 11-05 10:50:52.682 29603 29988 D Email   : <<< #2# ["OK", 
> ["CAPABILITY", "XAPPLEPUSHSERVICE", "IMAP4", "IMAP4rev1", "ACL", 
> "QUOTA", "LITERAL+", "NAMESPACE", "UIDPLUS", "CHILDREN", "BINARY", 
> "UNSELECT", "SORT", "CATENATE", "URLAUTH", "LANGUAGE", "ESEARCH", 
> "ESORT", "THREAD=ORDEREDSUBJECT", "THREAD=REFERENCES", "CONDSTORE", 
> "ENABLE", "CONTEXT=SEARCH", "CONTEXT=SORT", "WITHIN", "SASL-IR", 
> "SEARCHRES", "XSENDER", "X-NETSCAPE", "XSERVERINFO", "X-SUN-SORT", 
> "ANNOTATE-EXPERIMENT-1", "X-UNAUTHENTICATE", "X-SUN-IMAP", 
> "X-ANNOTATEMORE", "XUM1", "ID", "IDLE"], "User test.samus logged in"]
> 11-05 10:50:52.752 29603 29988 W DiscourseLogger: * OK [CAPABILITY 
> st11p00mm-iscream023 1S XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 SASL-IR 
> AUTH=ATOKEN AUTH=PLAIN] iSCREAM ready to rumble (1S:1092) 
> st11p00mm-iscream023 [42:4469:15:50:53:39]
> 11-05 10:50:52.762 29603 29988 W DiscourseLogger: * CAPABILITY 
> st11p00mm-iscream023 1S XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 SASL-IR 
> AUTH=ATOKEN AUTH=PLAIN
> 11-05 10:50:52.772 29603 29988 W DiscourseLogger: 2 OK [CAPABILITY 
> XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS 
> CHILDREN BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT 
> THREAD=ORDEREDSUBJECT THREAD=REFERENCES CONDSTORE ENABLE 
> CONTEXT=SEARCH CONTEXT=SORT WITHIN SASL-IR SEARCHRES XSENDER 
> X-NETSCAPE XSERVERINFO X-SUN-SORT ANNOTATE-EXPERIMENT-1 
> X-UNAUTHENTICATE X-SUN-IMAP X-ANNOTATEMORE XUM1 ID IDLE] User 
> test.samus logged in
> 11-05 10:50:52.782 29603 29988 W DiscourseLogger: * CAPABILITY 
> XAPPLEPUSHSERVICE IMAP4 IMAP4rev1 ACL QUOTA LITERAL+ NAMESPACE UIDPLUS 
> CHILDREN BINARY UNSELECT SORT CATENATE URLAUTH LANGUAGE ESEARCH ESORT 
> THREAD=ORDEREDSUBJECT THREAD=REFERENCES CONDSTORE ENABLE 
> CONTEXT=SEARCH CONTEXT=SORT WITHIN SASL-IR SEARCHRES XSENDER 
> X-NETSCAPE XSERVERINFO X-SUN-SORT ANNOTATE-EXPERIMENT-1 
> X-UNAUTHENTICATE X-SUN-IMAP X-ANNOTATEMORE XUM1 ID IDLE
> 11-05 10:50:52.802 29603 29988 W System.err: at 
> com.android.email.mail.store.imap.ImapResponseParser.expect(ImapResponseParser.java:237)
> 11-05 10:50:52.802 29603 29988 W System.err: at 
> com.android.email.mail.store.imap.ImapResponseParser.parseElement(ImapResponseParser.java:389)
> 11-05 10:50:52.802 29603 29988 W System.err: at 
> com.android.email.mail.store.imap.ImapResponseParser.parseElements(ImapResponseParser.java:472)
> 11-05 10:50:52.802 29603 29988 W System.err: at 
> com.android.email.mail.store.imap.ImapResponseParser.parseResponse(ImapResponseParser.java:350)
> 11-05 10:50:52.802 29603 29988 W System.err: at 
> com.android.email.mail.store.imap.ImapResponseParser.readResponse(ImapResponseParser.java:174)
> 11-05 10:50:52.802 29603 29988 W System.err: at 
> com.android.email.mail.store.imap.ImapResponseParser.readResponse(ImapResponseParser.java:203)
> 11-05 10:50:52.812 29603 29988 W System.err: at 
> com.android.email.mail.store.ImapStore$ImapConnection.executeSimpleCommand(ImapStore.java:2636)
> 11-05 10:50:52.812 29603 29988 W System.err: at 
> com.android.email.mail.store.ImapStore$ImapConnection.executeSimpleCommand(ImapStore.java:2626)
> 11-05 10:50:52.812 29603 29988 W System.err: at 
> com.android.email.mail.store.ImapStore$ImapConnection.doLogin(ImapStore.java:2806)
> 11-05 10:50:52.812 29603 29988 W System.err: at 
> com.android.email.mail.store.ImapStore$ImapConnection.open(ImapStore.java:2453)
> 11-05 10:50:52.812 29603 29988 W System.err: at 
> com.android.email.mail.store.ImapStore.checkSettings(ImapStore.java:650)
> 11-05 10:50:52.822 29603 29988 W System.err: at 
> com.android.email.activity.setup.AccountCheckSettingsFragment$AccountCheckTask.doInBackground(AccountCheckSettingsFragment.java:759)
> 11-05 10:50:52.822 29603 29988 W System.err: at 
> com.android.email.activity.setup.AccountCheckSettingsFragment$AccountCheckTask.doInBackground(AccountCheckSettingsFragment.java:1)
> 11-05 10:50:52.822 29603 29988 W System.err: at 
> android.os.AsyncTask$2.call(AsyncTask.java:287)
> 11-05 10:50:52.822 29603 29988 W System.err: at 
> java.util.concurrent.FutureTask.run(FutureTask.java:234)
> 11-05 10:50:52.822 29603 29988 W System.err: at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1080)
> 11-05 10:50:52.822 29603 29988 W System.err: at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:573)

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Wed, 6 Nov 2013 12:27:13 +0100

Icloud has two bugs: It sends LF, as Alexey points out. And it also
violates one of IMAP's most stunningly ridiculous requirements by
sending IMAP4 as the second item in the list, when it should have been
first. (See the first commend on page 84 for the rule.)

Arnt

---

From: Dave Cridland <[email-masked]>
Date: Wed, 6 Nov 2013 11:37:31 +0000

> first. (See the first commend on page 84 for the rule.)

I think that technically means it's simply not offering RFC 1730
compatibility, rather than violating anything. :-)

That requirement comes from RFC 1730 §7.2.1, incidentally. I'd find it
amazing if iCloud really did offer RFC 1730, actually, and given there are
no clients I can think of which would use it, I suspect any server offering
support has some severe bitrot.

Dave.