Subject: [Tools-discuss] Chat services

From: "Salz, Rich" <[email-masked]>
Date: Tue, 30 Mar 2021 16:35:20 +0000

I am also a Zulip fan.  I liked Jay’s note [1] and will emphasize Jay’s mention of Tim’s suggestions [2] as they really show how to make good use of the tool.  I wonder if we have any feedback from our IT folks about the operational aspects of the various tools?  CC’ing Glen.

Zulip is great; it seems to be the only tool that got feedback post-110.   It’s open source, which some will find interesting and/or compelling.

[1] https://mailarchive.ietf.org/arch/msg/tools-discuss/OktGsA3dzqH4Oqj3XmNlvzKE0zg/
[2] https://mailarchive.ietf.org/arch/msg/tools-discuss/WAvOx6dkPKu-UanUYtzeNhGjydg/

---

From: Glen <[email-masked]>
Date: Tue, 30 Mar 2021 15:28:27 -0700

On 2021-03-30 09:35, Salz, Rich wrote:
> I am also a Zulip fan.  I liked Jay’s note [1] and will emphasize
> Jay’s mention of Tim’s suggestions [2] as they really show how to
> make good use of the tool.  I wonder if we have any feedback from our
> IT folks about the operational aspects of the various tools?  CC’ing
> Glen.
> Zulip is great; it seems to be the only tool that got feedback
> post-110.   It’s open source, which some will find interesting
> and/or compelling.

Hi Rich, Everyone -

Certainly.

So we had three different things we were trying out, and I'll first 
summarize my experience with each, from an IT perspective.

PROSODY + CONVERSE

Prosody is simply a Jabber server.  In this case, we were using Prosody 
as a platform on which we were issuing individual Jabber accounts to 
people.  This was in response to a commonly-heard concern that some 
people found it difficult to find Jabber servers of their own to use.  
In the past I have, every few meetings, gone out and searched for live 
Jabber servers that worked, and provided a "top 5" list to participants. 
  In this case, since we were doing trials, we just offered to let people 
sign up for IETF-provided (but temporary) accounts.

Converse is a Javascript-based Jabber client.  In this case, we used it 
to allow people with local Prosody accounts to use a web-based client 
for Jabber, and to connect to chatrooms on the IETF Jabber server.

This was the most simple of the systems overall.  It just provided a 
basic Jabber client, and Jabber account server.  As expected, it was 
simple to operate.  Deployment was via prepackaged RPM, with some hack 
work to integrate Converse.  It worked well in our private-cloud-based 
shared hosting setup, and was low maintenance.  Administrative things 
like adding and removing accounts and changing passwords was 
command-line easy, and utilization was low.  There is no database per 
se, just a directory tree of per-user .dat files, so it's pretty simple 
on the backend as well.

 From a user perspective, it's... Jabber.  It solves the connectivity 
problems we've heard people report, and it's fine in that role.

MATRIX

Matrix is an alternative instant messaging platform.  In this case, we 
were using Matrix directly, but as a kind of bridge-to-Jabber client, 
where individuals could connect through a bridge to IETF Jabber rooms, 
thereby using Matrix as kind of a glorified Jabber client.

This system was easy to install.  Deployment was via pip/PyPI, with a 
Python-standard virtual environment.  Configuration was straightforward.

Operationally, however, there were a number of challenges, most of which 
pertain to the Matrix design paradigm.  The first challenge I discovered 
as a result of a warning from my backup system that a server backup was 
taking longer than expected.  I discovered after a hunt that we had a 
database file that had grown (quickly!) to 90GB.  By default, Matrix 
uses Sqlite3, but I was later informed by the Matrix team that I should 
have chosen Postgres.  Nonetheless, the database size was problematic.

Searching online, I discovered that in Matrix, individual servers are 
all linked together (hence, I suspect, the name), and content - both 
user accounts and conversation history - exist on many servers at the 
same time.  Our Matrix node was not an "IETF Matrix Server" so much as 
an "IETF-hosted mirror of parts of the global Matrix network."  As I 
tried to work with the database by hand, I discovered that a number of 
(hidden and unwanted) chat rooms had been mirrored to our server, and a 
number of hidden chatrooms had been created on our server - which 
appeared to account for a lot of the space utilization I observed.  In 
Matrix, administration is based on room ownership, not server ownership; 
as a result, it was almost impossible for me to "break in" to a hidden 
room, or perform any type of real moderation functions from a server 
perspective.  Server operations have to be done via a CURL-driven API, 
and are not as robust as I'd need them to be.  That means that there is 
no way for us to impose structure on a Matrix deployment, or to enforce 
terms of Note Well participation in a Matrix environment.  I would also 
not have been able to handle account issues or password resets or 
requests of that type.  For our purposes, it's more or less an external 
public chat.

The final operational challenge I faced was bridging.  On Matrix, the 
bridging to Jabber was custom work done by the Matrix team, off-site 
(i.e. not a part of our server.)  To this date, I have no way of setting 
up a bridge between Matrix and Jabber.  As a result, I was unable to add 
access to some additional Jabber rooms for 110, or to interact with the 
server directly for that purpose.

I understand that Matrix is under active development and, in the future, 
I am certain that these issues will become more clearly defined and 
implemented.  For now, however, operating Matrix is challenging on these 
levels.  Because of the database loading and content, I would not 
operate this on a shared server, but I would set up separate hosts just 
for Matrix.

As others have mentioned, from a user perspective, it's essentially 
enhanced Jabber.  It's much more a global community/chat environment 
that isn't really "owned" by any participant.  In my mind it's kind of 
"what (fill in older messaging system of your choice) should have been."

ZULIP:

Zulip is not so much an instant messaging platform as a structured 
collaboration platform.  It does provide instant messaging, and it does 
work well, but it is much more specific in its design paradigm and does 
not attempt to be a global messaging system (like Jabber or Matrix.)  
Instead, it is focused on being operated and used by a specific 
organization (such as the IETF.) and on organizing conversations and 
content in an intuitive, readable way.

Unlike the other software, which could be deployed as individual 
packages in a shared server environment, Zulip requires a dedicated 
virtual server/host on which to run.  I never run anything without a 
backup, so in operation I'd want to set up at least two (or more) 
separate virtual hosts in order to provide the redundancy we give the 
IETF on its other platforms.  This is not a problem - on the AMS private 
cloud, it was easy to set up a new host, but the OS requirement was for 
a Linux flavor other than the one I normally use, so my initial path was 
a bit different.

However, once I provisioned the server, everything else was pretty much 
automatic.  Deployment was via a tarball download with a self-enclosed 
shell script, which ran a form of puppet to perform serverwide 
configuration of database, email, webserver... basically everything.  
After launching the script, it was very hands-off.   Pre-launch 
configuration involved just a few changes to some config files, followed 
by a re-run of the deployment to update everything to match.

Operationally, Zulip was very easy to administer and manage.  Since this 
was a separate server I had to make some minor changes to hook in email, 
and I used the "matterbridge" program to provide bridging between Jabber 
and Zulip.  In-system administration is really nice, with clear controls 
over streams, and lots of flexibility in how the system is configured.  
I was able to deal with typical administrative issues, and access and 
understand the database layout, without any trouble at all.

Zulip has a very interesting hybrid API/Bot system that let us do things 
like link in list archives into the Zulip system.  Of course the IETF 
has a powerful and custom Mail archive tool, but it was very interesting 
to see how easy it was to feed mailing list content into Zulip, in a way 
that was useful and intuitive.  There are other opportunities there as 
well:  At one point, for example, I found myself playing with server 
monitoring systems, and feeding server performance and/or uptime 
notifications into Zulip as kind of a "status page" test, and 
considering other ways of connecting Zulip with (for example) things 
like Gitlab and the like.  There is a lot of interesting potential 
there.

 From a user perspective, Zulip is not Jabber.  It has a Jabber-like 
ability, but it's really more like a combination of Jabber plus Slack 
plus an Email archive plus an application bridge.  Indeed, I could see 
Zulip being used by the IETF as a Slack replacement quite effectively:  
It brings a number of improvements over Slack that I believe are 
compelling, and the integration of Jabber conversations with Email lists 
plus Zulip's own stream/threading model could potentially bring the best 
of all worlds to a single site.

FINAL COMMENTS

In the deployment of all three systems, I was very grateful to have the 
support and guidance of experts in each of the particular systems we 
tried.  While each system has benefits and challenges, the people were 
universally patient and excellent.  I wish to thank all of the teams for 
answering my questions and being patient with my inquiries as I 
undertook to operate all three systems for the IETF.

Jabber is Jabber (by which I mean to imply: "We know it, it's been here 
forever, and we're used to it") and I think that it's useful, and I 
think it works reasonably well.  We do have the occasional number of 
connectivity complaints from time to time, and it's "server-to-server" 
mentality makes diagnosing connectivity issues challenging.  So I was 
kind of sad to see so much attention in these trials focused on 
"bridging to Jabber."  Speaking only for myself, I really wanted people 
to look at Prosody/Converse from the point of view of "did this solve my 
connectivity problem", and at Matrix/Zulip from the point of view of 
"what uniqueness/benefit might these platforms bring *beyond* Jabber."

In my role as the IETF's operations vendor, I don't set any policies or 
make any decisions, and that's as it should be:  It is the users who 
should be providing feedback to the leadership, and the leadership who 
should be making the decisions about the best ways forward.  I have 
tried to encourage this on all three platforms with messages posted on 
them from time to time during the trial period.  I myself gave all three 
platforms equal time and participation, both in terms of setup, 
administration/operations, and user testing.

That said, speaking only as an individual, at the end of the day, I 
found myself personally gravitating towards Zulip, not only because it 
was the most pleasant to administer and operate, but also because I felt 
it was the most unique environment, and brings with it the greatest 
potential for benefit to the IETF and the community.

I think we're going to need to continue to use Jabber for a long time, 
and I think that running Prosody/Converse to provide that might be a 
good thing to consider doing long-term.  But looking at each package 
individually, I think that Zulip brings such functional improvements 
that it is worth operating on its own, whether the IETF continues with 
Jabber, or not.

I hope this information was helpful, and would be happy to answer 
specific questions if that would be helpful.

Glen
--
Glen Barney
IT Director
AMS (IETF Secretariat)