Subject: [Tools-discuss] matrix tests

From: Stephen Farrell <[email-masked]>
Date: Fri, 11 Dec 2020 20:52:41 +0000

Hiya,

I've used jabber at IETFs for ages. I don't find that at
all hard. (But my jabber accounts while old, seem to be
working fine.)

I've run my own matrix server for the last few years for
non-IETF purposes.

I appreciate the ability to use matrix as mobile clients
for matrix are more widely available and more up to date
than for jabber. That (better clients) might be more
noticeably beneficial when we get back to f2f meetings,
as in WFM-mode, I have plenty of screen real-estate.

For IEF109 I connected to the IETF rooms using my normal
matrix account from my own server. It's very good that
can be done.

I had expected there to be a matrix room corresponding
to each jabber room but I didn't investigate much due
to the TZ issues and the fact that jabber was fine.

It didn't occur to me to create any matrix rooms. If
it had, I'm not sure if that'd have been useful but
I can imagine it being so in future, e.g. for a bar
bof. Before that'd be a standard procedure I'd like
to understand the duration for which any such rooms
might last, how closed/open they may be etc.

I think the main feature in matrix clients that I like
that beats jabber is the ability to edit messages after
they've been sent. I'm not sure how that'd pan out with
the jabber logs, but am happy to do a test if that's
useful. I'm also happy to connect and play about in
general if that's helpful, just ping me.

Whatever experiments are done, I think an equivalent
to the jabber logs will be needed if matrix or another
scheme is to be used for more than experiments.

Cheers,
S.

---

From: Michael Richardson <[email-masked]>
Date: Sun, 13 Dec 2020 15:57:12 -0500

Stephen Farrell <[email-masked]> wrote:
    > It didn't occur to me to create any matrix rooms. If it had, I'm not
    > sure if that'd have been useful but I can imagine it being so in
    > future, e.g. for a bar bof. Before that'd be a standard procedure I'd
    > like to understand the duration for which any such rooms might last,
    > how closed/open they may be etc.

It would be interesting if we could get some kind of convergence between
gather.town and such rooms.

    > I think the main feature in matrix clients that I like that beats
    > jabber is the ability to edit messages after they've been sent. I'm not
    > sure how that'd pan out with the jabber logs, but am happy to do a test
    > if that's useful. I'm also happy to connect and play about in general
    > if that's helpful, just ping me.

I feel the urge to edit your text above and see if you notice :-)

    > Whatever experiments are done, I think an equivalent to the jabber logs
    > will be needed if matrix or another scheme is to be used for more than
    > experiments.

Of all the available technologies, I feel that a seemless transition towards matrix
pleases me the most.

--
]               Never tell me the odds!                 | ipv6 mesh networks [
]   Michael Richardson, Sandelman Software Works        |    IoT architect   [
]     [email-masked]  http://www.sandelman.ca/        |   ruby on rails    [

---

From: Matthew Wild <[email-masked]>
Date: Sun, 13 Dec 2020 23:01:26 +0000

Hi Stephen,

> as in WFM-mode, I have plenty of screen real-estate.

I'm curious which clients you tried. On Android there's (at least)
Conversations and yaxim, iOS has (at least) Siskin and Monal. All are
actively developed with releases in the past months, and all of them (to
answer a point you made later in your email) support editing messages.

> to the TZ issues and the fact that jabber was fine.

There is a Matrix bridge that should allow anyone to join the XMPP chats
from any Matrix client, as I understand it.

> general if that's helpful, just ping me.

As mentioned above, all modern clients support editing messages.

This is a common pattern. Clients such as Pidgin and Adium are popular
among people who are used to Jabber/XMPP from years ago, but both those
projects are seriously lacking behind today's standards. Unfortunately I
think many people form opinions of XMPP based on their experiences with
older "previous generation" software, but modern XMPP has long moved on. I
know the Pidgin folks are keen to catch up, but they have other
project-wide priorities they are focusing on.

On the logs note, I believe with the current setup editing a message will
probably display the updated message as a new message. As with a number of
things in the current IETF XMPP deployment, this is something that could be
improved upon. If the decision is made to stick with XMPP then I think a
bit of modernization of the deployment would be a great thing, and
something I'd be happy to help with.

Regards,
Matthew

---

From: Glen <[email-masked]>
Date: Sun, 13 Dec 2020 15:02:14 -0800

> how closed/open they may be etc.

I can actually report some technical data on that.

As I was performing server maintenance over the past few days, I 
discovered that the Matrix database was taking up a significant amount 
of space (91 GB for 37 local users.)  I put in an inquiry but didn't 
find anyone online (likely people are offline for the weekend) so I did 
some digging myself.

I discovered that our Matrix server was actually mirroring, and storing 
a local copy of, every single Matrix room that every local user had 
joined.  Here in no particular order are ten of the largest and most 
amusing rooms out of the 166 rooms that were found being served by our 
server:

#GodotLatino:matrix.org
#hedgedoc:matrix.org
#projectalice-shitpost:matrix.org
#marcopolo:matrix.org
#matrix:matrix.org
#monero:matrix.org
#lounge:privacytools.io
#rules:hive-mind.network
#synapse:matrix.org
#litecoin:matrix.org

Unlike Jabber, where a room is hosted on a single server; in Matrix, it 
turns out that every room that anyone belongs to is co-hosted on every 
server used by any of that room's users. [1] Even if all (local) users 
leave the room, the room remains on the server, along with all of the 
shared data.

Moreover, if I understand what the developers explained to me, it is not 
possible to delete a room "everywhere".  I can delete a room that exists 
on the local server, but that does not delete the room from other 
servers to which the room was shared.  And if our server somehow became 
disconnected from the rest, all of the rooms that had been shared out to 
other servers would continue to exist on those other servers forever as 
orphaned clones (for, as we like to say, some value of "forever"), 
outside of our ability to control (or perhaps even enumerate.)

So unlike operating a Jabber server, where the conference server is the 
"authoritative" location of the room (and logs are captured there), and, 
it turns out, unlike operating a Zulip server (which also has an 
authoritative location).... operating a Matrix server isn't really 
"operating" a server so much as "joining a huge network of peer servers."

As the IETF's operator, I do not set policy, so this is just my 
opinion... but I *think* that having our Matrix server mirroring some of 
the rooms in my list above seems... problematic.  If it turns out that 
this is the behavior desired by the IETF, then no changes are needed. 
If, however, the IETF wants to operate the Matrix server in a 
more-or-less authoritative, "Note Well"-compliant kind of way, then I 
*think* the Matrix server would need to be deployed with federation 
disabled (meaning, local user signups only, and no room sharing across 
the world), and public room sharing disabled as well.  We can certainly 
do this, but we have not tested that kind of setup yet.

But unless they make that choice, then the answer to Stephen's questions 
seem to be:

 > the duration for which any such rooms might last,

"forever", with no option to delete/close unless federation is disabled, and

 > how closed/open they may be etc.

"wide open to every Matrix server of every participating user."

Again speaking only for myself, my feeling is that all of the focus thus 
far on the technologies in this trial has been on how well we can bridge 
said technologies over to Jabber.  I suggest that if the IETF reaches a 
"trial2" stage, it would be more useful to focus on the technology 
itself (as if, as Michael suggests, a transition is the goal), rather 
than on how well things bridge to Jabber.  Bridging is fun... but I'm 
not sure it should really be the point... or, for that matter, even a 
factor.  I personally would be much more interested in how well each 
technology works *on its own*, rather than "as a copy of Jabber."

Hope this helps!

Glen

[1] ( https://matrix.org/faq/ , section: "Where do my conversations get 
stored?" which says, "Each replicated across all of the homeservers 
whose users are participating in a given room", and, farther down, under 
their definition of Federation, "data about rooms and message history is 
shared between servers of participating users.")

---

From: Stephen Farrell <[email-masked]>
Date: Sun, 13 Dec 2020 23:30:12 +0000

Hiya,

On 13/12/2020 23:01, Matthew Wild wrote:
> I'm curious which clients you tried. 

For jabber, pidgin on Ubuntu. For matrix, element/riot
on Ubuntu and IOS. (Not sure it's relevant but "tried"
isn't quite right - I've used pidgin for loadsa years
and the riot/element matrix client for maybe 3 years
of day-to-day usage, so for me those clients are well
known and not on trial.)

And I also had meetecho in a browser too of course, but
rarely use that for the chat room.

> On Android there's (at least)

I hadn't appreciated that. I think some form of link in the
agenda would be good if/when that's do-able.

> project-wide priorities they are focusing on.

Fair enough. I'd be happy to try other xmpp clients on Ubuntu
if someone had recommendations.

> bit of modernization of the deployment would be a great thing, 

Agreed.

Cheers,
S.

> and

---

From: Michael Richardson <[email-masked]>
Date: Sun, 13 Dec 2020 22:14:43 -0500

Glen <[email-masked]> wrote:
    > opinion... but I *think* that having our Matrix server mirroring some
    > of the rooms in my list above seems... problematic.

We can't be the only one with this problem.

My understanding is that it is used within the French government and
bureaucracy, and this must be a problem there too.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 14 Dec 2020 06:17:24 +0100

On 14. Dec 2020, at 00:01, Matthew Wild <[email-masked]> wrote:
> 
> iOS has (at least) Siskin 

How do you use this?
I immediately got hit with dozens of presence subscription requests from the usual spammers.
Each of these needs to be laboriously clicked away, which of course triggers new ones.

Grüße, Carsten

---

From: Matthew Hodgson <[email-masked]>
Date: Thu, 17 Dec 2020 23:13:02 +0000

Better late than never I can try to provide more clarity on Matrix's 
data processing/retention semantics - please see comments below...

On 13/12/2020 23:02, Glen wrote:

> of space (91 GB for 37 local users.).

There are several reasons for this:

  * The server has been provisioned using sqlite rather than postgres as 
the DB.  Sqlite is meant to be just for evaluation, not for production, 
and I dread to think how fragmented or unoptimised a 91GB sqlite DB has 
got :)
  * Synapse's schema is currently quite naive on how it snapshots the 
state of its chatrooms.  It currently takes a snapshot every 100 state 
changes (where state changes are things like room membership changes; 
not normal messages).  Thus a room with 10,000 members causes another 
10,000 db rows to be inserted every 100 joins or parts.  This is 
obviously terrible; there's a workaround to apply a better compression 
algorithm via https://github.com/matrix-org/rust-synapse-compress-state, 
which needs to be merged into Synapse proper.
  * As you've pointed out, Matrix works by replicating conversations 
between servers.  So even if you have 1 user, if they go nuts and try to 
join 1,000 busy rooms with huge numbers of participants, they can suck 
up a lot of disk space.  Conversely, if you have 1,000 users and they 
are all in tiny rooms, the opposite could be true.  Matrix is more like 
hosting an IMAP server than (say) an SMTP or XMPP server in this respective.
  * You can configure the server to have limited the data it stores for 
the rooms its users are in, and I assume this hasn't been configured: 
https://github.com/matrix-org/synapse/blob/44b7d4c6d6d5e8d78bd0154b407defea4a35aebd/docs/sample_config.yaml#L410-L421.

> I put in an inquiry but didn't find anyone online (likely people are 
> offline for the weekend) so I did some digging myself.

Sorry - I think our messages crossed while you were writing the original 
mail :)

> Unlike Jabber, where a room is hosted on a single server; in Matrix, 
> it turns out that every room that anyone belongs to is co-hosted on 
> every server used by any of that room's users. [1] Even if all (local) 
> users leave the room, the room remains on the server, along with all 
> of the shared data.

This isn't strictly true.  If all local users leave the room, and 
"forget" it (equivalent to expunging an IMAP folder - it's an option 
after you leave a room in the 'Historical' section in Element), then the 
data is free to be garbage collected.

> Moreover, if I understand what the developers explained to me, it is 
> not possible to delete a room "everywhere".  I can delete a room that 
> exists on the local server, but that does not delete the room from 
> other servers to which the room was shared.

In theory you could request other servers delete the room and hope they 
honour the request.  However, i'm not sure that I'd like some random 
admin/moderator/user on a remote server to decide to reach in and 
destroy my copy of a room on my server - so we haven't implemented it yet.

> And if our server somehow became disconnected from the rest, all of 
> the rooms that had been shared out to other servers would continue to 
> exist on those other servers forever as orphaned clones (for, as we 
> like to say, some value of "forever"), outside of our ability to 
> control (or perhaps even enumerate.)

They wouldn't really be "orphaned clones", any more than a local git 
repository clone is an "orphaned clone" of whatever you cloned it from.  
To be orphaned implies you once had parents.  In Matrix, the rooms are 
shared over their participating servers, and all servers are equal in 
that respect.

> So unlike operating a Jabber server, where the conference server is 
> the "authoritative" location of the room (and logs are captured 
> there), and, it turns out, unlike operating a Zulip server (which also 
> has an authoritative location).... operating a Matrix server isn't 
> really "operating" a server so much as "joining a huge network of peer 
> servers."

Yup, precisely. It's the same difference between using SVN and using 
Git.  Most people consider the lack of authoritive server to be an 
extremely desirable feature - it means that if I join IETF rooms from my 
personal server, I have full sovereignty over my copy of the 
conversation, even if my 'net connection goes down, or even if something 
terrible happened to the IETF server, etc.  I have network partition 
tolerance.

> of the rooms in my list above seems... problematic. 

I would expect this to be no worse than (say) running a caching HTTP 
proxy for users.  If your users choose to go view weird stuff out there 
on Matrix, then your server ends up with a local cache of that data 
until it expires.

However, if you wanted to limit the servers your server can federate 
with, that's possible too (but not very idiomatic - just as it wouldn't 
be idiomatic to specify a whitelist or blocklist of email servers to 
talk SMTP with): 
https://github.com/matrix-org/synapse/blob/44b7d4c6d6d5e8d78bd0154b407defea4a35aebd/docs/sample_config.yaml#L673-L695

> If it turns out that this is the behavior desired by the IETF, then no 
> changes are needed. If, however, the IETF wants to operate the Matrix 
> server in a more-or-less authoritative, "Note Well"-compliant kind of 
> way, then I *think* the Matrix server would need to be deployed with 
> federation disabled (meaning, local user signups only, and no room 
> sharing across the world), and public room sharing disabled as well.  
> We can certainly do this, but we have not tested that kind of setup yet.

You could do this, but it would very much miss the fundamental nature of 
Matrix - the equivalent of realising that because an internet-connected 
LAN could be used by its users to access unexpected content on the 
internet, one should isolate it from the internet.  I think this might 
be counter to the IETF's aethos :)

In terms of being authoritative in a Note Well kind-of way, it's worth 
noting that the replicated copies of rooms which other servers maintain 
are cryptographically signed to match each other.  So from a logical 
perspective, they do not diverge (other than during a netsplit).

> But unless they make that choice, then the answer to Stephen's 
> questions seem to be:
>

This is not correct. The server admin can boot rooms off the server, or 
specify a retention policy, or the room admin can specify a retention 
policy.  If all users have left and forgotten the room, the server could 
garbage collect it (if desired, although this hasn't landed in Synapse 
yet). Federation has no impact on the ability to delete/close a room.

> "wide open to every Matrix server of every participating user."

This is not necessarily true: if data privacy is a concern, you can 
enable end-to-end encryption on the room, and then the data is not 
visible to *any* server - just the authorised participating users' devices.

Aside from end-to-end encryption, Matrix rooms also impose decentralised 
ACLs to limit which users and servers can see the history based on the 
room admin's requirements; it turns out this is a novel area of computer 
science: 
https://matrix.org/blog/2020/06/16/matrix-decomposition-an-independent-academic-analysis-of-matrix-state-resolution.

So while it's true that the new messages in an unencrypted room is wide 
open and visible to all servers which have users currently participating 
in that room, it is not true to say that past history is visible (unless 
explicitly shared), and future messages are not replicated after a 
server's users leave.

Hope this provides some clarity.  Apologies that the mental process of 
familiarisation with Matrix is a paradigm shift similar to going from a 
VCS to a DVCS, but when it clicks, much like a DVCS, its characteristics 
hopefully feel desirable rather than unexpected and weird.

thanks,

Matthew

-- 
Matthew Hodgson
Matrix.org