Subject: [Endymail] Restart Endymail to discuss the Mathematical Mesh?

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 7 Apr 2019 13:23:07 -0400

OK so I have a very comprehensive end-to-end secure messaging protocol that
is capable of supporting end-to-end secure mailing lists as well as many
other advanced crypto features. Micali fair contracts anyone?

I don't propose this as a replacement for SMTP today or tomorrow. That
would be silly. But there are many applications that SMTP is horrible for:

* Large messages (video files as attachments)
* Secure by default
* Spam

80% of the emails I sent and received at Comodo and Verisign were internal
and we were all required to use Outlook An email system that is only for
internal use and required a plug in would be entirely acceptable provided
that it was zero-effort secure.

If the protocol was adopted by a major email client provider and was in the
base version of the client, we might get to a point where the enterprise
and its lawyers and accountants all used the Mesh Messaging protocols. So
now we can extend the security envelope to 90% of the emails we are
concerned with.

OK, so I want to transfer 100GB, 2TB files with an email protocol. To do
that I need to switch to a Pull protocol for obvious reasons. Which means
that my control messages can be very small because all they need to contain
is the stuff normally seen in SMTP headers and a link to where the larger
messages are.

So I am currently working with a hard limit of 64 Kilobytes on messages. If
you want to send a longer message, send it as a detachment. Keep the
control channel clear, lean and mean.

One consequence of this is that I can impose a hard limit on Mesh Service
transaction size and time. If I insist that a Mesh Service be on a 10MB/s
link or better, any transaction that takes more than 5 seconds to complete
can probably be dropped without impact on the user. Either the source or
the destination is overloaded or the message itself is some sort of DoS
attack.

Thoughts?

---

From: Jeremy Harris <[email-masked]>
Date: Sun, 7 Apr 2019 18:40:06 +0100

On 07/04/2019 18:23, Phillip Hallam-Baker wrote:
> If I insist that a Mesh Service be on a 10MB/s

I think that disenfranchises anyone on a basic DSL, plus three-quarters
of the African continent's rural population?
-- 
Cheers,
  Jeremy

---

From: Tom Mitchell <[email-masked]>
Date: Sun, 7 Apr 2019 12:49:04 -0700

On Sun, Apr 7, 2019 at 10:23 AM Phillip Hallam-Baker <[email-masked]>
wrote:

> OK so I have a very comprehensive end-to-end secure messaging protocol
> that is capable of supporting end-to-end secure mailing lists as well as
> many other advanced crypto features. Micali fair contracts anyone?
>
> I don't propose this as a replacement for SMTP today or tomorrow.
>
.....

> that it was zero-effort secure.
.... "Thoughts?"..

I see a like option set already in Gmail.

I can attach an object in line or add a link to it in their cloud.
I can forward with or without attachments.. (Note images are special cases,
marketing?).
Gmail has total message size limits by default, pick one.  References can
almost any size.
URIs outside of Google are possible.
I get warnings when I reply to someone not in my organization or address
book.
The attachment link can have partial or full credentials to open up an
identity exchange
even an access authorization token for the object.  Encryption and
cryptographic
class hashes are possible.  See Google Drive features.

It solves data duplication.
It leverages a central corporate repository strategy that can block
external transfers and audit access.
A central resource is not prohibited from watermarking fetched documents
much the way color printers
add yellow dots, more inventive tricks are possible.
It permits all messages and parts to be a document (attachment) and
separates the to/from/date/host(s) meta data
from the data. So all attachments even the text itself can require a fetch
from the central server a bit
like tiny, small, medium and large beacon (pay per eyeball)  images in
todays email.  Each object can have its own
fetch, see, save, download policy.

Gmail does have trouble with composing "PURE" text as kernel.org rules
require (sort of sux).
http://vger.kernel.org/majordomo-info.html
 must be TEXT/PLAIN, there can be no multipart messages, no VCARDs, nothing
``fancy''.

SMTP is store and forward but inside the system direct connection to a
central server
resource seems possible (eggs in one basket disclaimer). Store and forward
has MX record hooks
today that are possibly necessary to traverse firewalls including internal
ones.

Data retention policy on a central server could be enforced for good and
bad.

Data access can have a timer features enforced at the server.

Encryption of data at rest is possible is encryption of each component.

Address books can contain public keys so content by reference can be
encrypted with the public key of each recipient when fetched.  Server
latency
and availability may be an issue. Lost keys are a monster issue but the
server
has the master data and need not retain my encrypted copy.

So sure.
It is important to have a policy for "external" communications so the only
mail tool
anyone needs is the enhanced one.  My #1 feature compete requirement is
$EDITOR.
vi, vim, ed, edit, emacs, MStools.   Character sets, and language tags...

-- 
   T o m    M i t c h e l l

---

From: "John R. Levine" <[email-masked]>
Date: 7 Apr 2019 16:26:05 -0400

> I can attach an object in line or add a link to it in their cloud.

We've had message/external-body for over 20 years.  See RFC 2017.

I realize it's not implemented all that widely but it's there if anyone 
wants it.

Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 7 Apr 2019 18:22:32 -0400

> I realize it's not implemented all that widely but it's there if anyone

The aim of the Mesh was to start off with a completely green field approach
ignoring all existing infrastructure to see what the system would then look
like.

Having done that, it is of course entirely reasonable to look at how we
might retrofit the feature set to the existing mail infrastructure. And in
fact that is one of the things that the Mesh helps with. I can use the Mesh
to manage my S/MIME and OpenPGP keys across my devices and achieve
zero-effort security.

But this is also like building a high speed rail system and only using it
to move the spare parts needed to keep Amtrack running.

The other point here is that there is a big difference between use of
detachments as an optional feature you can choose to implement and use and
use of detachments as something you are absolutely forced to use because
Mesh Messages are limited to 65535 bytes and the service won't accept them
if they are 65536.

Limiting the messages in the control channel means that a service should
never end up being blocked because 16 different users are all receiving
10MB files at the same time. It also means that we can support really
enormous files seamlessly - provided that the recipient actually wants the
data.

---

From: "Steven M. Bellovin" <[email-masked]>
Date: Mon, 08 Apr 2019 14:31:49 -0400

Slightly off-topic, but folks on this list may be interested in
https://www.cs.columbia.edu/~smb/papers/eurosys-2019-submission408-e3-final-1.pdf.
We plan to release the code as open source.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 8 Apr 2019 15:24:11 -0400

> We plan to release the code as open source.

Looks interesting. I can see some clear convergence in mechanism. Though it
is not clear to me how you are dealing with the per device keys.

My approach is to split the decryption key. Which means I can add and
remove devices etc. in no time at all. When I provision a new device, the
administration device knows the private key for each of the applications I
might provision to it. To add S/MIME capabilities, it splits the private
key in two, sends one to the service and encrypts the other under the
public key of the device. So the service only has a random number and the
device can't decrypt without the device helping.

The crypto is a scheme Matt Blaze and Torben Pederssen discovered.
Depending on how you look at it, you can consider this to be Proxy
Re-encryption or Distributed Key Generation. It is actually a simple case
that sits at the intersection of both.

---

From: Mark Rousell <[email-masked]>
Date: Wed, 10 Apr 2019 18:04:03 +0100

On 07/04/2019 18:23, Phillip Hallam-Baker wrote:
> If the protocol was adopted by a major email client provider and was
> in the base version of the client, we might get to a point where the
> enterprise and its lawyers and accountants all used the Mesh Messaging
> protocols. So now we can extend the security envelope to 90% of the
> emails we are concerned with.

How about seeing if it could be integrated into a popular thick client
like Thunderbird? Conceivably this sort of protocol support might be
doable as an addon.

Initially, perhaps, this protocol could be proven and/or extended to
legacy mail thick clients using a local proxy.

> So I am currently working with a hard limit of 64 Kilobytes on
> messages. If you want to send a longer message, send it as a
> detachment. Keep the control channel clear, lean and mean.

If this were to be exposed to end users then it would be over-complex
and would be a disincentive to uptake. So if you are going to have a
hard limit of 64KB per message (which I have to say seems small to me)
then it needs to be utterly transparent to both senders and receivers.
No extra actions of any sort can be required for users on either end; it
has to look and act like any other email. And this aspect must not be
implementation-specific, i.e. the standard must require the UX to be
fully transparent, otherwise the social aspects of UX (which impact take
up and popularity) will be negatively impacted.

> If I insist that a Mesh Service be on a 10MB/s link or better, any
> transaction that takes more than 5 seconds to complete can probably be
> dropped without impact on the user. Either the source or the
> destination is overloaded or the message itself is some sort of DoS
> attack.

No, I can't see that scaling well or flexibly. Again, a hard limit like
this would be a serious disincentive to ubiquitous take up. Vast numbers
of users have slower connections than this, even in the first world.

I think that one can reasonably suggest recommended minimum bandwidth in
a non-normative/recommendations section of a standard but I believe it
is counter-productive to make specific bandwidths a requirement.

-- 
Mark Rousell

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 10 Apr 2019 13:53:53 -0400

On Wed, Apr 10, 2019 at 1:04 PM Mark Rousell <[email-masked]>
wrote:

> If the protocol was adopted by a major email client provider and was in
> the base version of the client, we might get to a point where the
> protocols. So now we can extend the security envelope to 90% of the emails
> we are concerned with.
>
>
> like Thunderbird? Conceivably this sort of protocol support might be doable
> as an addon.
>

> legacy mail thick clients using a local proxy.

That is exactly my strategy in the short term. I have an SMTP proxy but it
applies a much older approach.

After that, Outlook and TBird.

It is quite possible that Microsoft Windows Mail would be the easiest
though.

> So I am currently working with a hard limit of 64 Kilobytes on messages.
> If you want to send a longer message, send it as a detachment. Keep the
> If this were to be exposed to end users then it would be over-complex and
> would be a disincentive to uptake. So if you are going to have a hard limit
> of 64KB per message (which I have to say seems small to me) then it needs
> to be utterly transparent to both senders and receivers.
>

I totally agree. For some of the messages sent, 64KB is enough. But for an
SMTP replacement, the scheme has to be send everything as a control message
and detachments unless the message is short enough to fit in the control
message

The reason to allow content in the message at all is that the overwhelming
number of messages are really small, a few lines at most. So unless there
is a really long stream of included messages in the thread, we can avoid
the need for a detachment.

The reason for picking 64KB is that the Ethernet MTU will get up to the IP
max packet size at some point but it is really unlikely we will increase IP
packet size. Processing a 64KB message as 8 9001 byte Jumbo frames is
entirely practical.

I suspect that while we would set 64KB as the MUST ACCEPT size, most UDP
clients would likely send a message as a detachment if it was going to hit
the 9001 MTU. It doesn't make much difference on a TCP transport of course.

> implementation-specific, i.e. the standard must require the UX to be fully
> transparent, otherwise the social aspects of UX (which impact take up and
> popularity) will be negatively impacted.
>

Total agreement modulo a few small details when we get to managing huge
detachments.

If I wanted to send a video to this list, I couldn't. Too big. But I want
to be able to send files of 100GB without having to switch applications
like I do today. But when we deal with really large chunks of data, well,
the recipient might not want them and certainly not on their Apple Watch,
Android cufflinks etc.

So If I send an email that is 300KB to you and I am in your contacts
directory, your Mesh Service would perform access control on the inbound
message and decide something like

* Its from PHB, its authentic
* PHB in contacts accept the message
* Detachment is less than 1MB, PHB is a friend, account is not short of
space, prefetch it and stage.
If the detachment was larger, handling would likely be different. Instead
of staging it, the clients might pull it directly. For 1TB files we would
probably want the transfer to be direct.

> dropped without impact on the user. Either the source or the destination is
> overloaded or the message itself is some sort of DoS attack.
>
>
> this would be a serious disincentive to ubiquitous take up. Vast numbers of
> users have slower connections than this, even in the first world.
>
> I think that one can reasonably suggest recommended minimum bandwidth in a
> non-normative/recommendations section of a standard but I believe it is
> counter-productive to make specific bandwidths a requirement.
>

The key thing here is establishing some sort of expectation that you
deliver the data in a certain length of time most of the time.

The situation we have today with SMTP is that most email senders have to
hire an email deliverability specialist or use one of less than a hundred
MSPs or their email won't get delivered because it is considered to be
spam. I would rather have a published expectation than have it left to
random policies at random sites..

---

From: Mark Rousell <[email-masked]>
Date: Thu, 11 Apr 2019 06:20:16 +0100

On 10/04/2019 18:53, Phillip Hallam-Baker wrote:
>
> That is exactly my strategy in the short term. I have an SMTP proxy
> but it applies a much older approach.
>

Seems like a good plan.

Surprises me about Windows Mail, though. I've not looked at Windows Mail
but I would never have thought it was designed to be open to extension.

>  I totally agree. For some of the messages sent, 64KB is enough. But
> for an SMTP replacement, the scheme has to be send everything as a
> control message and detachments unless the message is short enough to
> fit in the control message
>
> The reason to allow content in the message at all is that the
> overwhelming number of messages are really small, a few lines at most.
> So unless there is a really long stream of included messages in the
> thread, we can avoid the need for a detachment.. 
>
> The reason for picking 64KB is that the Ethernet MTU will get up to
> the IP max packet size at some point but it is really unlikely we will
> increase IP packet size. Processing a 64KB message as 8 9001 byte
> Jumbo frames is entirely practical.
>
> I suspect that while we would set 64KB as the MUST ACCEPT size, most
> UDP clients would likely send a message as a detachment if it was
> going to hit the 9001 MTU. It doesn't make much difference on a TCP
> transport of course..

Ah I see, that makes sense, thanks for this further explanation.

>
> Total agreement modulo a few small details when we get to managing
> huge detachments.
>
> If I wanted to send a video to this list, I couldn't. Too big. But I
> want to be able to send files of 100GB without having to switch
> applications like I do today. But when we deal with really large
> chunks of data, well, the recipient might not want them and certainly
> not on their Apple Watch, Android cufflinks etc.
>
> directory, your Mesh Service would perform access control on the
> inbound message and decide something like
>
> * Detachment is less than 1MB, PHB is a friend, account is not short
> of space, prefetch it and stage.
> If the detachment was larger, handling would likely be different.
> Instead of staging it, the clients might pull it directly. For 1TB
> files we would probably want the transfer to be direct. 
>

That all makes sense.

> The situation we have today with SMTP is that most email senders have
> to hire an email deliverability specialist or use one of less than a
> hundred MSPs or their email won't get delivered because it is
> considered to be spam. I would rather have a published expectation
> than have it left to random policies at random sites..

I do note that you started this thread by suggesting (if I understood
correctly) that the protocol would initially be promoted for corporate
internal email-style use and so my concerns about potentially limited
bandwidth in the world at large would likely not apply in that scenario.
Nevertheless, it seems to me that the standard needs to be got right
from the beginning for both internal and broad external usage scenarios,
otherwise it risks not being able to catch on and grow outside of a niche.

(As an aside: It seems to me that for corporate scenarios your new
protocol will effectively compete against the combination of
Exchange/O365/Outlook/Sharepoint. In particular in my opinion, O365 is a
specially powerful tool for ecosystem lock in for Microsoft).

In general I can agree with documenting
reasonable/recommended/non-normative values as a kind of statement of
ideals but if you specify too much bandwidth as being a hard requirement
then I feel certain that it will kill broad uptake of a very innovative
new protocol like this one.

In a protocol that is (eventually, as I understand it) intended to
connect together heterogeneous nodes (heterogeneous in terms of
ownership, bandwidth and quality of connectivity) then I think it is
virtually impossible for the designer of the protocol to realistically
specify delivery time expectations. Surely the protocol (in order to (a)
work in the real world and (b) to be attractive to a wide range of
users) must work as well as possible over a wide range of connection
type and speeds (which means, in very common scenarios, considerably
less than 10Mb/s).

Despite my concerns, it seems to me that your concept has great
potential. We're in a time of considerable innovation amongst messaging
protocols but almost all of it is (a) within walled, non-standard
gardens, and (b) designed with a focus on instant messaging-style
applications, and so it would be very good indeed to see a new
standard-driven protocol that can provide the traditional benefits of
email in a more modern way (something which walled garden instant
messaging-style protocols and apps fail to do).

I had previously noticed discussion of your Mathematical Mesh drafts and
I've just gone to look at them, as well as mathmesh.com. I had not
realised until just now how expansive a concept you have. I realise now
that it's about far more than just a modernised form of email. It seems
to be very exciting indeed. It's so broad that focussing on a
potentially achievable goal such as email-style usage seems like it
might be a good way to proceed.

-- 
Mark Rousell