Subject: [Jmap] recommendations for gigantic mailboxes

From: Eric Wong <[email-masked]>
Date: Thu, 25 Jun 2020 09:03:36 +0000

Hi all, I'm considering implementing JMAP in public-inbox[1].

TL;DR: would it be reasonable to expose 10 million messages in a
single JMAP mailbox?  It can already be done for an NNTP newsgroup :>

public-inbox started with NNTP and HTTP support, but I recently
started implementing an IMAP server and it seems fine.  It's
mainly tested with mutt, isync|mbsync, offlineimap, and Gnus.

Whereas newsreaders typically have a rolling window where
messages expire; MUAs and IMAP <=> Maildir syncing tools tend to
hold messages forever.  That's slow when mailboxes get into the
order of 100K...

MUAs and Maildirs on traditional filesystems fall down with 400K
messages in the [email-masked] mailing list, so right now
it's broken into 9 segments capped at 50K messages, each:

  imaps://public-inbox.org/INBOX.comp.version-control.git.0
  ..
  imaps://public-inbox.org/INBOX.comp.version-control.git.8

400K is nothing, even.  LKML has 3.6 million and grows around
30K a month[2]:

  https://lore.kernel.org/lkml/
  nntp://nntp.lore.kernel.org/org.kernel.vger.linux-kernel

So, JMAP is fresh ground; so I wonder if having a single mailbox
on the order of 10 million messages is reasonable for JMAP clients.
NNTP clients have no problem handling a newsgroup that large.

Or if I should go route I did with IMAP and partition mailboxes
virtually into 50K or so.  It doesn't affect the underlying
storage at all.  I wasn't really happy to do that for IMAP, but
the prevalance and limitations of existing MUAs had to be taken
into account (and I still work on a Centrino laptop from 2005,
sometimes).

Thanks for reading!

[1] https://public-inbox.org/README.html
    Archives mail in git; indexes using Xapian + SQLite.

[2] LKML required designing the v2 public-inbox format
    back in 2018 for scalability:
    https://public-inbox.org/public-inbox-v2-format.html
    I've tested it up to 10M or so before running out of space;
    but I expect it to do more in ~10 years (if civilization
    survives that long!)

---

From: "Neil Jenkins" <[email-masked]>
Date: Fri, 26 Jun 2020 15:53:53 +1000

On Thu, 25 Jun 2020, at 19:03, Eric Wong wrote:
> single JMAP mailbox?

Yes, I don't see why not.

> on the order of 10 million messages is reasonable for JMAP clients.

I mean, this is really down to implementation. There's nothing in the protocol that makes this particularly problematic, but if it's trying to save every message locally then it could run into trouble depending on its choice of on-disk format. But JMAP makes it easy to page in just the portion of the mailbox you are currently interested in viewing, so as long as the client is working in this mode it should be fine.

As JMAP is very new, there are not many clients to test against yet to see how they work with such a large volume. You can point the demo webmail <https://jmap.io/jmap-demo-webmail/> client at a server to test with; I've tried it with the order of ~1 million messages but not with ~10 million. This is not caching anything locally other than temporarily for performance reasons, which is a good model for accessing this kind of data.

Neil.

---

From: "Bron Gondwana" <[email-masked]>
Date: Fri, 26 Jun 2020 20:54:35 +1000

> I mean, this is really down to implementation. There's nothing in the protocol that makes this particularly problematic, but if it's trying to save every message locally then it could run into trou…

Yeah, as Neil says - the protocol itself is going to be totally fine. The big issue for you is going to be your data model, particularly if you're trying to do Email/query and Email/queryChanges efficiently.

Overall, it's going to be nicer than IMAP because you don't have to do as much unsolicited work - if the client wants something it will ask for it, and if it doesn't ask, you don't need to create it.

Cheers,

Bron.

--
 Bron Gondwana, CEO, Fastmail Pty Ltd
 [email-masked]

---

From: Arnt Gulbrandsen <[email-masked]>
Date: Fri, 26 Jun 2020 13:57:50 +0200

On Friday 26 June 2020 12:54:35 CEST, Bron Gondwana wrote:
> Overall, it's going to be nicer than IMAP because you don't 
> have to do as much unsolicited work - if the client wants 
> something it will ask for it, and if it doesn't ask, you don't 
> need to create it.

Having a publicly available four-million-message mailbox will be nice for 
JMAP, too. So many IMAP clients fail miserably when they come up against 
million-message gmail accounts. Neil, Bron, I do hope you'll add a 
suggestion to test new clients against public-inbox if/when.

Arnt