Libervia

Libervia
Libervia progress note 2026-W16
23/04/2026, 15:40
jabber-xmpp-en
Libervia
project
libre
Libervia progress
progress
XMPP
Long time no see!
Wow, I can't believe that the last progress note was 3 years ago, that's crazy! As you can imagine, a lot of things have changed with Libervia since then.
The
Audio/Video implementation work
has been completed, with direct calls, group calls for small numbers of participants with MUJI (a MESH call, i.e. all participants call each other in P2P), group calls/conferences with SFU (using a component to distribute the audio/video streams selectively, usable for larger numbers of participants) based on the excellent
Galène
, and even remote desktop control! The last 2 are really experimental at this stage.
The
Email Gateway
has also been completed, so you can have your email directly in your XMPP client, and mailing lists are converted to XMPP forums. This was also the occasion to work on several nice features such as thread support in Libervia, and "Data Policy", which is a way to indicate how your data are handled by a service.
I've also worked on several specifications, and I'm a member of XMPP Council for the second term.
I'll try to blog in more detail about all that, but for this note, let's focus on more recent work.
There are 3 main things I'm currently focusing on:
Simplifying Libervia installation and settings
Focusing on Web frontend to make it fully usable
working on
Serverless and Metadata Reduction for XMPP
grant
Simplifying Libervia installation and settings
I believe that one of, if not the main, pain point with Libervia is the difficulty of installing and configuring it. It's a big project, and it has some technical debt due to its age (Libervia, formerly "Salut à Toi", was started before
asyncio
, the Python standard async platform, existed, before type hints in Python were a thing, and before many other things).
I've made several efforts in the past to improve that, and fortunately there have been nice people to make and maintain packages for
Debian
(thanks Debacle and other people before him),
Arch Linux
(thanks to jnanar and others), and
NixOS
(thanks To NLnet/NGI packaging team).
However those packages have to be maintained, and it should be possible to install the upstream version easily.
In recent years, a new tool,
uv
, caused quite a stir in the Python community. Packaging has for long been a notorious issue with Python, and even if some great improvements happened over time (such as
Poetry
or
Hatch
),
uv
was a major leap: it's an external tool written in Rust with good engineering, making it really fast; being an external tool, it can install a Python version by itself if it's missing; it follows standards that have been improved a lot in the last few years regarding Python packaging; it's relatively complete, replacing several tools; and it has a handy workflow and virtual environment management.
uv
can install and manage python executables (with
uv tool
commands), or run them for a one-off try with
uvx
(a command similar to
pipx
). If you like and want to install it, everything will be in cache, so
uv tool install
will be instantaneous.
So with that in mind, the goal was to make Libervia installable with a
uv
one-liner (which should work too with other tools thanks to Python standards).
From Mercurial to Git
The major problem was that
uv
doesn't support
Mercurial
as opposed to
pip
. When Libervia was started, the Git vs Mercurial debate was all the rage, and Libervia ("SàT" at the time) being in Python, Mercurial was chosen. It's a good DVCS, very comparable to Git, but the issue is that more and more tool only support Git nowadays (tools such as
ripgrep
handle
.gitignore
but not
.hgignore
), and people are used to it, even if Mercurial is simpler. I was thinking about moving for quite some time,
uv
was the last nail in the coffin.
So I had to replace the legacy
Mercurial repository
, but I want to keep my repositories on my own server, and I don't want to manage yet another external tool. Well, that's a thing that was on the TODO list of Libervia for a very long time: there is already an XMPP decentralised bug tracker, a merge request mechanism; to have a complete forge, we were mostly missing a repository browsing implementation.
So I've implemented it, and now a new "Forge" feature is available in Libervia; you can see my repositories at
. The existing ticket management has been integrated there, and you can browse code and individual commits. The merge request system has not been integrated yet as I want to refactor it to a better system. Everything is decentralised, and tickets are XMPP based (the code is decentralised by Git itself, the forge is mostly about rendering it, and serving for
git clone
).
That's one thing less to maintain (well, for now I keep the legacy Mercurial repository for archive), and if I miss a feature, I can easily add it myself.
Settings
When Libervia started to have settings, at the time, the obvious thing to do was to use the standard lib tools, so
ConfigParser
, an implementation similar to INI files by Microsoft, was used. While the format is simple, it doesn't support typing, and it's difficult to write complex data such as mappings or multi-line lists. The data were parsed and validated by Libervia, and suffixes such as
_path
_port
_dict
or
_list
were used. Also, there is nothing to facilitate documentation.
I've decided to move to
TOML
as it is the configuration format chosen by Python itself, so there is support in standard lib, and I can be sure that it will be maintained.
My goal was to have settings easy to use, easy to work with as a developer, with validation, type hints, etc.
I've quickly come to
Pydantic Settings
. Pydantic is a library to handle models with efficient and automatic validation. It's extremely popular in the Python world. The settings package integrates, as its name implies, configuration with Pydantic, and it handles TOML. Furthermore, with
Autodoc Pydantic
, we have auto documentation from sources.
Thanks to that, Libervia has moved to
libervia.toml
configuration files, with automatic validation, easy syntax to write complex structures, good typing, support for environment variables and other goodies. It was the occasion to also refactor plugin, component and frontend specific documentation.
This took me a lot of time, as configuration is used in many places, and I had to reorganise things well to have a future-proof design, but I'm very happy with the result.
Dependencies
Another problem, for making installation easier and also compatible with the latest version of Python (3.14), was with some dependencies.
Wokkel ➡ Tx-XMPP
Libervia was heavily based on
Wokkel
, a library bringing many improvements on top of
Twisted
's XMPP implementation. It's a great library and a ton of useful work has been done there. Unfortunately, it has been unmaintained for a very long time (last commit in December 2018), and some pull requests we needed were not merged. We were dealing with that with a "temporary" package which was doing some monkey patching in the hope of seeing Wokkel come back to life, but at this point it was clear that we had to move.
So I've made a (friendly) fork of Wokkel called
Tx-XMPP
with the needed patches and all the changes we did in
sat_tmp
(the temporary package) integrated. I've dropped Python 2 support which was making the package incompatible with Python 3.14 and latest Twisted, and I'm progressively adding modern type hints and fixing things or adding some features here and there.
That was another weight off my mind, as I don't have to maintain
sat_tmp
anymore nor do fragile monkey patching.
python-gnupg
Some other dependencies are causing trouble. The official
GPGME
Python binding, used for "ox" (
OpenPGP for XMPP
), needs to use the same version as GPG, so they are installed system-wide, which means that we can't totally isolate Libervia in a virtual environment, which is a show-stopper for the
uv
one-liner that I was targeting (and can bring other conflict troubles). So I've implemented an alternative backend to use
python-gnupg
instead, which doesn't have the same problem, and I plan to move to
PySequoia
at some point.
GLib and GStreamer
Another big issue with dependencies were GLib-based ones. Those need to bind to system libraries, and may need to be built. Two major components were using them in Libervia: the D-Bus implementation, and GStreamer.
For D-Bus, Libervia was using
dbus-python
for legacy reasons. This package doesn't have great design, according to its authors themselves. In the backend,
TxDBus
, a D-Bus implementation on top of Twisted, was used for quite some time. But for the frontend, it was more difficult because frontends can use various async backends with their own loops (Twisted, Asyncio, GLib, Qt), and dbus-python was used in "blocking" mode in a few places (long story short: that was technical debt, dbus-python could be used to block until a response is received, which is wrong in async code, but was handy before modern coroutines were available).
Tx-DBus is now used in backend and frontend, and
dbus-python
dependency has been removed.
For GStreamer, it's used for WebRTC implementation (necessary for audio/video calls, and also used for direct streams, notably with file transfer). This one is part of an
extra
(optional feature), so it's not installed by default (it's needed only for local frontends, the web one uses Web APIs instead). I have learned that they are working on a
bundle
with a complete GStreamer "distribution". That's great news that will solve the problem, but for now it's not available for GNU/Linux. I'm looking forward to it.
Term-Image
The last package to cause trouble was
term-image
, which, due to dependencies bounding and not being maintained for a year, is not compatible with Python 3.14. It's used to
display audio/video directly in the terminal
I've disabled it for now, and will probably move to
textual-image
which is integrated with
Rich
that I'm using in the CLI.
Bridge
One other big issue with installation was the bridge: by default Libervia uses
D-Bus
, which is installed and automatically launched in desktop environments. The problem is that it's not the case for most servers, or if you want to run Libervia as a SystemD service. Here, we had to run the D-Bus daemon manually, and run a script to configure it.
I've changed that to add an
auto
mode which first tries D-Bus, and falls back to
Perspective Broker
, a Twisted native IPC which can run on Unix sockets.
So in short, no more configuration or issue with Bridge, it's all automatic now!
Components
Another improvement is component configuration: before, you had to use the CLI frontend to set a profile and mark it as a component, and add a flag to make it start automatically with the backend. Now, thanks to the work done on settings mentioned above, it's all automatic and you just have to set
jid
and
secret
options in your component settings.
You can
check the documentation
for details.
Result
With all that in place, installation is now a matter of a one-liner:
uv tool install 'libervia-backend[main] @ git+https://www.goffi.org/forge/goffi/libervia-backend/'
And that's it! You then you can run
libervia-backend
and use
li
the short name of
libervia-cli
I'm finishing redesign/refactoring work, and it will soon be possible to run the web frontend in a similar way.
Focusing on Web frontend to make it fully usable
I'll stay short for following section, as the previous one was really long.
So, with the lack of resources (Libervia is a solo-dev project at this point), I've decided to focus on the web frontend for now until it's fully usable, and then take care of the other ones.
I started an architectural redesign of the web frontend several years ago, which is now based on
Brython
and
Jinja
templates. After making everything work, I'm on the final step of working on the UI and UX to make it really usable. Most pages are now based on a 3-panel view, with the left one to discover/change chat/blog/etc., the middle one being the main feature view, and the right one usually used for extra information. The goal is to make a feature immediately usable when you click on it.
It's mostly done for chat, blog, and forge. Calls are partially done. Other features should be easier, and will be finished hopefully soon.
The goal is to have a usable web frontend in a matter of weeks, with an alpha release, then beta once all features have been redesigned.
working on Serverless and Metadata Reduction for XMPP
In parallel of the work above, I'm working on my current
NLnet
NGI0
grant. It is about reducing metadata in XMPP, and working on a serverless implementation in Libervia, and authoring new specifications when necessary. You can follow the progress
on the forge
Tor
The first step was about integrating
Tor
to be able to access
.onion
server, or normal server while hiding IP. This was made possible thanks to
txtorcon
, a package integrating Tor with Twisted projects. The Tor daemon must be running and
configured appropriately
Tor is automatically activated (if possible, i.e. dependency and daemon available) when a
.onion
domain is used. Otherwise, it can be activated from settings, as explained in doc.
With the CLI, you can see the circuit used by Tor:
Pubsub Contacts
One of the obvious ways a server can have metadata on somebody is by analyzing their contacts. Currently XMPP uses a "Roster", which is stored in the clear by the server, because it needs it to know who has presence access to whom, and this may be used for things like filtering incoming messages.
Those contacts can have a name, and one or more "groups" (basically a tag to indicate that the contact is "work", "family", "friend", etc.). While handy, those data have obvious privacy implications.
As a step to avoid that, I've authored "End-to-End Encrypted Contacts Metadata" (which will be renamed soon to "Pubsub Contact" following feedback and discussions) which has the official number
XEP-0510
. With this, a user can (optionally) encrypt their contacts end-to-end, with the trade-off that the server can't access them anymore and then loses presence-related features.
It will also be useful to handle contacts for serverless use cases.
The CLI can retrieve and manipulate those contacts:
Pubsub (re)implementation
Libervia is heavily using Pubsub (Publish Subscribe, a way to store data with permissions management, and subscriptions with notifications). Usually, the Pubsub service is on the XMPP server, so in serverless use-case, it's necessary to have an alternative. That's the first reason to implement a pubsub component in Libervia.
The other one is that there is already a "Libervia Pubsub": many years ago the pubsub implementations in most XMPP servers were really basic. Most XMPP clients are focusing essentially on chat use-cases, so pubsub was not the first place where server teams were putting resources. However, it's necessary to have a good implementation for a client like Libervia, and in addition many features have been added along the way to add features to pubsub which are not yet available in server implementations.
So basically you had either to be blocked by server implementation, maybe choose one and contribute to it (but what about the others?), or to work around missing features by using inefficient or convoluted designs.
So, many years ago, I decided to make my own full-featured pubsub implementation to have everything that Libervia needed. Fortunately there was
Idavoll
, another project by Ralph Meijer, the author of
Wokkel
mentioned above (and one of the authors of
XEP-0060
, the core pubsub specification; many thanks to him and other people who worked on those specs and code).
One of the problems is that there is a special implementation of pubsub,
Personal Eventing Protocol
, which is a sort of "Pubsub per account", and could only be done by the server itself. So I've authored
XEP-0355 (Namespace Delegation)
and
XEP-0356 (Privileged Entity)
to make it possible. Those specifications tell a server "if you see this feature (namespace), just let me handle it", and give permission to do things normally only allowed to the server (like retrieving roster, necessary for presence permissions). I've also done the implementations for Prosody:
mod_delegation
and
mod_privilege
. EJabberd
did its own implementations
, I'm not sure about other servers.
OK now, Libervia Pubsub is working, and is the reference Pubsub/PEP implementation for Libervia. Why reimplementing it?
Well, first, it's a separate project to maintain. And second, it has a lot of technical debt, starting with the database handling which is done directly with PostgreSQL queries, and is difficult to update in case of schema changes.
With Libervia Backend, there was already everything needed to implement a pubsub component, as pubsub is already used by the pubsub cache, and by several components (the ActivityPub gateway and the Email gateway). It uses
SQLAlchemy
with
Alembic
, meaning that several database engines can be used (for now SQLite only, but PostgreSQL will be added at some point), and schema updates are easy to do.
Even if large parts were already implemented, it was a massive piece of work, but now there is a full-featured pubsub implementation in Libervia Backend, easy to maintain and update, and it's easy to update the schema.
It's, to my knowledge, the first pubsub service to implement
XEP-0496 (Pubsub Node Relationships)
and
XEP-0499 (Pubsub Extended Discovery)
. In short, those features let you use tree-like pubsub nodes, which is better for organisation and really useful for
forums
To give you a concrete use case: for the blogging feature (
XEP-0472
and
XEP-0277
), you have one node with the main blog posts, and one extra node per blog post for the comments. If you delete the main blog posts node, you'll have all the comment nodes staying with no purpose on your pubsub service.
With Pubsub Node Relationships, the main blog posts are in parent node, and all comment nodes are children. When you delete the parent, all children are automatically deleted too. It's also easier to explore the service, as you don't have a myriad of flat nodes.
The new pubsub component does not yet have feature parity with the old implementation, but the most difficult part is done, and other features should now be easy to add.
I'm really happy to have it finally integrated to the core Libervia architecture.
Once again, the CLI has a tool to explore a pubsub with node relationships, here shown with the email gateway:
Conclusion
Well, after all this time, this was quite a long blog post, and I've only mentioned some of the recent work.
As you can see, I was really busy, and that explains why I couldn't find the time to blog (it's time consuming: this post took me more than half a day to write). Hopefully, future notes should be shorter.
There is a lot coming, I'm finally seeing the light at the end of the tunnel. Stay tuned, and follow me on ActivityPub at
@Goffi@mastodon.social
Upcoming Event: Libervia at Open Source Experience in Paris
goffi
02/12/2024, 11:35
Libervia
libre
OSXP
Paris
event
📅 Join us at
Open Source Experience
on December 4-5 in Paris!
Create Your Personalized Badge
Create your personalized badge for the event using this link:
Create a personalized badge
. This link associates your registration with Libervia.
Booth Sharing
We'll be a the booth next to
@LaMouetteODF
, at the Associative Village.
See You There!
Looking forward to seeing you at the event! If you have any questions, feel free to reach out.
The European Union must keep funding free software
goffi
18/07/2024, 07:47
Union européenne
NGI
libre
NLnet
Open Letter to the European Commission.
Since 2020, Next Generation Internet (
NGI
) programmes, part of European Commission's Horizon programme, fund free software in Europe using a cascade funding mechanism (see for example NLnet's
calls
). This year, according to the Horizon Europe working draft detailing funding programmes for 2025, we notice that Next Generation Internet is not mentioned any more as part of Cluster 4.
NGI programmes have shown their strength and importance to supporting the European software infrastructure, as a generic funding instrument to fund digital commons and ensure their long-term sustainability. We find this transformation incomprehensible, moreover when NGI has proven efficient and economical to support free software as a whole, from the smallest to the most established initiatives. This ecosystem diversity backs the strength of European technological innovation, and maintaining the NGI initiative to provide structural support to software projects at the heart of worldwide innovation is key to enforce the sovereignty of a European infrastructure.
Contrary to common perception, technical innovations often originate from European rather than North American programming communities, and are mostly initiated by small-scaled organizations.
Previous Cluster 4 allocated 27 million euros to:
"Human centric Internet aligned with values and principles commonly shared in Europe" ;
"A flourishing internet, based on common building blocks created within NGI, that enables better control of our digital life" ;
"A structured ecosystem of talented contributors driving the creation of new internet commons and the evolution of existing internet commons".
In the name of these challenges, more than 500 projects received NGI funding in the first 5 years, backed by 18 organisations managing these European funding consortia.
NGI contributes to a vast ecosystem, as most of its budget is allocated to fund third parties by the means of open calls, to structure commons that cover the whole Internet scope - from hardware to application, operating systems, digital identities or data traffic supervision. This third-party funding is not renewed in the current program, leaving many projects short on resources for research and innovation in Europe.
Moreover, NGI allows exchanges and collaborations across all the Euro zone countries as well as "widening countries" [^1], currently both a success and an ongoing progress, likewise the Erasmus programme before us. NGI also contributes to opening and supporting longer relationships than strict project funding does. It encourages implementing projects funded as pilots, backing collaboration, identification and reuse of common elements across projects, interoperability in identification systems and beyond, and setting up development models that mix diverse scales and types of European funding schemes.
While the USA, China or Russia deploy huge public and private resources to develop software and infrastructure that massively capture private consumer data, the EU can't afford this renunciation.
Free and open source software, as supported by NGI since 2020, is by design the opposite of potential vectors for foreign interference. It lets us keep our data local and favors a community-wide economy and know-how, while allowing an international collaboration.
This is all the more essential in the current geopolitical context: the challenge of technological sovereignty is central, and free software allows addressing it while acting for peace and sovereignty in the digital world as a whole.
L'Union Européenne doit poursuivre le financement des logiciels libres
goffi
18/07/2024, 07:45
Union européenne
NGI
libre
NLnet
Lettre ouverte à la Commission Européenne
Depuis 2020, les programmes Next Generation Internet (
NGI
), sous-branche du programme Horizon Europe de la Commission Européenne financent en cascade (notamment, via les
appels
de NLnet) le logiciel libre en Europe. Cette année, à la lecture du brouillon du Programme de Travail de Horizon Europe détaillant les programmes de financement de la commission européenne pour 2025, nous nous apercevons que les programmes Next Generation Internet ne sont plus mentionnés dans le Cluster 4.
Les programmes NGI ont démontré leur force et leur importance dans le soutien à l'infrastructure logicielle européenne, formant un instrument générique de financement des communs numériques qui doivent être rendus accessibles dans la durée. Nous sommes dans l'incompréhension face à cette transformation, d'autant plus que le fonctionnement de NGI est efficace et économique puisqu'il soutient l'ensemble des projets de logiciel libre des plus petites initiatives aux mieux assises. La diversité de cet écosystème fait la grande force de l'innovation technologique européenne et le maintien de l'initiative NGI pour former un soutien structurel à ces projets logiciels, qui sont au cœur de l'innovation mondiale, permet de garantir la souveraineté d'une infrastructure européenne. Contrairement à la perception courante, les innovations techniques sont issues des communautés de programmeurs européens plutôt que nord-américains, et le plus souvent issues de structures de taille réduite.
Le Cluster 4 allouait 27.00 millions d'euros au service de :
- "Human centric Internet aligned with values and principles commonly shared in Europe" ;
- "A flourishing internet, based on common building blocks created within NGI, that enables better control of our digital life" ;
- "A structured eco-system of talented contributors driving the creation of new internet commons and the evolution of existing internet commons".
Au nom de ces enjeux, ce sont plus de 500 projets qui ont reçu un financement NGI0 dans les 5 premières années d'exercice, ainsi que plus de 18 organisations collaborant à faire vivre ces consortia européens.
NGI contribue à un vaste écosystème puisque la plupart du budget est dévolu au financement de tierces parties par le biais des appels ouverts (
open calls
). Ils structurent des communs qui recouvrent l'ensemble de l'Internet, du matériel aux applications d'intégration verticale en passant par la virtualisation, les protocoles, les systèmes d'exploitation, les identités électroniques ou la supervision du trafic de données. Ce financement des tierces parties n'est pas renouvelé dans le programme actuel, ce qui laissera de nombreux projets sans ressources adéquates pour la recherche et l'innovation en Europe.
Par ailleurs, NGI permet des échanges et des collaborations à travers tous les pays de la zone euro et aussi avec les
widening countries
[^1], ce qui est actuellement une réussite tout autant qu’un progrès en cours, comme le fut le programme Erasmus avant nous. NGI est aussi une initiative qui participe à l’ouverture et à l’entretien de relation sur un temps plus long que les financements de projets. NGI encourage également à l'implémentation des projets financés par le biais de pilotes, et soutient la collaboration au sein des initiatives, ainsi que l'identification et la réutilisation d'éléments communs au travers des projets, l'interopérabilité notamment des systèmes d'identification, et la mise en place de modèles de développement intégrant les autres sources de financements aux différentes échelles en Europe.
Alors que les États-Unis d’Amérique, la Chine ou la Russie déploient des moyens publics et privés colossaux pour développer des logiciels et infrastructures captant massivement les données des consommateurs, l’Union Européenne ne peut pas se permettre ce renoncement. Les logiciels libres et
open source
tels que soutenus par les projets NGI depuis 2020 sont, par construction, à l’opposée des potentiels vecteurs d’ingérence étrangère. Ils permettent de conserver localement les données et de favoriser une économie et des savoirs-faire à l’échelle communautaire, tout en permettant à la fois une collaboration internationale. Ceci est d’autant plus indispensable dans le contexte géopolitique que nous connaissons actuellement. L’enjeu de la souveraineté technologique y est prépondérant et le logiciel libre permet d’y répondre sans renier la nécessité d’œuvrer pour la paix et la citoyenneté dans l’ensemble du monde numérique.
Dans ces perspectives, nous vous demandons urgemment de réclamer la préservation du programme NGI dans le programme de financement 2025.
[^1]: Tels que définis par Horizon Europe, les États Membres élargis sont la Bulgarie, la Croatie, Chypre, la République Tchèque, l’Estonie, la Grèce, la Hongrie, la Lettonie, la Lituanie, Malte, la Pologne, le Portugal, la Roumanie, la Slovaquie et la Slovénie. Les pays associés élargies (sous conditions d’un accord d’association) l’Albanie, l’Arménie, la Bosnie-Herzégovine, les Îles Feroé, la Géorgie, le Kosovo, la Moldavie, le Monténégro, le Maroc, la Macédoine du Nord, la Serbie, la Tunisie, la Turquie et l’Ukraine. Les régions élargies d’outre-mer sont: la Guadeloupe, la Guyane Française, la Martinique, La Réunion, Mayotte, Saint-Martin, Les Açores, Madère, les Îles Canaries.
As defined by Horizon Europe, widening Member States are Bulgaria, Croatia, Cyprus, the Czech Republic, Estonia, Greece, Hungary, Latvia, Lituania, Malta, Poland, Portugal, Romania, Slovakia and Slovenia. Widening associated countries (under condition of an association agreement) include Albania, Armenia, Bosnia, Feroe Islands, Georgia, Kosovo, Moldavia, Montenegro, Morocco, North Macedonia, Serbia, Tunisia, Turkey and Ukraine. Widening overseas regions are : Guadeloupe, French Guyana, Martinique, Reunion Island, Mayotte, Saint-Martin, The Azores, Madeira, the Canary Islands.
Libervia progress note 2023-W22
goffi
31/05/2023, 20:39
jabber-xmpp-en
SàT
Libervia
project
libre
Libervia progress
SàT progress
XMPP
Hello,
It's been a while since my last progress note. I've been immersed in work on Libervia, with many tasks to accomplish. I'll keep this update brief.
A/V Implementation in Libervia
I'm thrilled to announce a new development that I haven't yet officially shared on this blog: Libervia has once again received a grant from
NLnet
, this time via the NGI Assure Fund. This funding will facilitate the implementation of A/V calls with Jingle across several frontends: Web, Desktop, and CLI. In addition to one-on-one calls, multi-party calls are in the plans. The first approach will be using
XEP-0272: Multiparty Jingle (Muji)
, which supports Peer-to-Peer (P2P) connections between participants without the need for a specific service—ideal for a small number of participants. For larger groups, a Selective Forwarding Unit (SFU) will be used, mirroring the technology used in platforms such as
Jitsi Meet
Alongside calls, we plan to offer Desktop Sharing, and even a specification and implementation for Remote Desktop Control atop XMPP!
Progress is well underway; one-on-one calls are already functional within the web frontend. However, there is still substantial work ahead.
To gain insight into the related tasks and what we've accomplished so far, you can check
the associated tickets
ActivityPub Gateway
I realize the high level of anticipation surrounding this gateway. While the gateway is operational (this blog utilizes it), it is yet to stabilize. I had hoped early adopters would install it and provide bug reports or other feedback. Unfortunately, this was not the case, with only one person (to my knowledge) having used it and provided prior feedback. This is understandable considering the requirement of an existing XMPP server, installation of a dev version of Libervia, and setting everything up. If you wish to try it out, you're welcome to join our official chat at
libervia@chat.jabberfr.org
http link
).
Nonetheless, I've decided to adjust my strategy. At some point, I plan to open a test instance. Due to limited resources for moderation, it will likely be invitation-only initially. The ultimate aim is to gather sufficient feedback to ensure stability.
Official Website and Test Instance
There have been several changes to the website:
The Flatpak and Docker images appeared to be malfunctioning. I've temporarily removed them and plan to rectify this issue later. If you wish to test Libervia in the meantime, the sources remain available.
I've added a roadmap at
. Many had requested this, so here you go.
I've discontinued the libervia.org test instance, after years of service, as I've relocated the official website there to align with the project's new name (Libervia versus formerly Salut à Toi). As mentioned above, I have plans for a new instance, but it will likely be invite-only initially. I'm also considering a local-only demo for a quick overview of the web frontend—a non-federated feature that self-resets every few hours. We'll see if I can make the time to set it up.
Talks
I'll be delivering two talks next month in Paris:
A 20-minute presentation in English at
OW2
: Catch me on stage
at 15:00 on Thursday, June 15
A 60-minute talk in French at
Pas Sage En Seine
: I'll be
at "Salle cinéma" at 17:30 on Friday, June 16
I would be delighted to engage in a conversation if you happen to be there.
Other
In addition to these projects, I've been focusing on parallel tasks like code refactoring, work on calendar events, and the web frontend. However, I'll delve into these topics in more detail at a later date.
That's all for this note.
Libervia progress note 2022-W45
goffi
24/11/2022, 11:58
jabber-xmpp-en
SàT
Libervia
project
libre
Libervia progress
SàT progress
XMPP
Hello, it's time for a long overdue progress note.
I'll talk here about the work made on ActivityPub (AP) gateway and on end-to-end encryption around pubsub.
Oh, and if everything goes well, this blog post should be accessible from XMPP
and
ActivityPub (and HTTP and ATOM feed), using the same identifier
goffi@goffi.org
Forewords
The work made on the AP gateway has been possible thanks to a
NLnet/NGI0 grant
(with financial support from the European Commission's Next Generation Internet programme).
I especially appreciated that the team was really there to help bring the ideas to life, and not once did they get in the way: little paperwork, no unnecessary pressure, caring, contacts when help was needed, etc.
I wish there were more organizations like this one that really help develop libre projects for the common good.
So once again I want to thank them for all that.
XMPP ⬌ ActivityPub Gateway
There is probably no need to explain here what is
ActivityPub
, we can simply write that it is an open protocol that allows to do things that XMPP also allows doing, and that until now these 2 protocols could not communicate together. The work on the ActivityPub gateway aims to allow software implementing one of these 2 protocols to communicate as easily as possible. I firmly believe that all open protocols should be able to communicate which each other, to avoid creating more silos, proprietary software is already good enough at that.
To be useful, a gateway must use the full potential of both protocols. A simple bot transcribing messages as we see too often, using unsuitable features (such as instant messaging for blog posts), or using a very limited set of features to ensure compatibility are flaws that I have tried to avoid. Building a good gateway is a difficult and time-consuming task. If done right, the gateway should be as invisible as possible to the end user.
XMPP is featuring blogging since long before AP, however the set of features is not exactly the same. Current use of AP is clearly inspired from commercial "social" networks, and metadata such as subscribers/subscribed nodes (or followers/following in AP terms) are highlighted, feature such as like/favourite were missing in XMPP, and some implementation such as Pleroma do implement reactions. To integrate that in the gateway, I've been working on new specifications:
Pubsub Public Subscriptions
: a way to publicly announce subscriptions, in an opt-in way. With this it's possible to implement followers/following features in a way respectful of privacy.
Pubsub Attachments
: a generic way to attach any kind of data to a pubsub item. It's notably used to implements noticed/favourite button (see
here
and reactions.
Calendar Events
: handling of events and all the
RSVP
mechanism. Libervia was handling events for years, but it was an experimental implementation, this specification is a next step in the effort to make it a standard.
Note that this XEP and the others linked below have been accepted but are not yet visible in official list.
You may wonder why there is a specification for Calendar Events… It's because the AP gateway also handles them, making it compatible with
Mobilizon
. The gateway may evolve in the future to support other non (micro)blogging use cases.
The gateway is now finished in terms of functionalities, however the code is clearly of an alpha quality for the moment. Now the goal in the coming months will be to stabilize and possibly implement other features if there is a demand for it.
Early adopters are encouraged to try and test it as long as they keep in mind that it's not stable. So if you do try it, I recommend keeping a separate ActivityPub account in whatever stable implementation you use at the moment, this way you can check if messages or media are missing, if there is any inconsistency or other bugs, and report them to me. If you test it, please join the XMPP room
libervia@chat.jabberfr.org
click here to connect from your browser
) for help and feedback. Stabilization will probably take weeks, but I hope to have it done by early 2023.
Installation instructions and details on how the conversion between protocols is done is available in
the documentation
and notably
here
A question I've been asked a lot: yes, you can use the same identifier for XMPP (JID) and AP (
WebFinger
actor handle) as long as you use "simple" characters (i.e. alphanumeric ASCII chars,
and
). If you use something more complicated, you'll have to use the escaping mechanism explained in the doc (this is due to constraints with some AP implementations).
As for blogs on pubsub nodes (what Movim calls "communities"), I made it simple: you can use directly the name of the node that holds the blog in the local part (i.e. before the "@") of your actor handle: a blog named
community_bog
at the XMPP pubsub service
pubsub.example.org
can thus be addressed with the AP actor
community_blog@pubsub.example.org
. This way you can use a rather user-friendly identifier to share your blog with people who are only on ActivityPub.
This gateway should work with any XMPP server, and any client that implement blogging features (Only
Libervia
itself and
Movim
implement it for now, but I have heard that other clients are planning support for it). To enjoy the whole feature set of the gateway, the new specifications need to be implemented by the clients, so you can start to fill feature requests…
With this gateway, the door is open to have a client able to talk to the ActivityPub network, while having the feature of XMPP, including e2e encrypted private messages (e2e encrypted only if you communicate with an XMPP account, not with an AP one).
Oh, and please update your graphics, drawing and other texts to include XMPP in the fediverse ;)
End-to-End Encryption
Much effort has also gone into end-to-end encryption.
OMEMO implementation has been updated (OMEMO:2 is now used), including
Stanza Content Encryption
which allows encrypting arbitrary elements instead of only the \ of the message, I believe that Libervia is the first XMPP client to implement it.
OpenPGP
for XMPP (or "OX") has also been implemented, all that thanks to the work of Tim Henkes "Syndace", the author of
python-omemo
Beside instant messaging, end-to-end encryption has also been introduced to pubsub. I've made specifications for two methods:
An
OpenPGP profile for pubsub
which is thought to encrypt a whole node, with a system of secret sharing/rotation/revocation. With it, it is easy to give access to new entities after publication, and to retrieve old items for newcomers. This specification can be used to encrypt
any
pubsub based features: (micro)blogging, calendar events, lists, etc.
Pubsub Targeted Encryption
which is a way to apply the same cryptographic system used in instant messaging to pubsub. This way, OMEMO can be used with its forward secrecy property. It is not a good option to use this specification to encrypt a whole node, as archive is then not accessible to newcomers, and to add access to a new entity you have to re-encrypt all items, but it's an interesting option to encrypt an element occasionally, for instance to restrict access of a specific post in an otherwise public blog.
Specifications have also been written to sign a pubsub item in a backward compatible way (client which don't implement those specifications can still work normally):
Pubsub Signing
Pubsub Signing: OpenPGP Profile
All those specifications are already implemented in Libervia, but they are only usable from CLI frontend at the moment. All you have to do is to use the
--encrypt
and/or
--sign
options from
pubsub
or
blog
commands (check documentation for details).
Uploaded files were already encrypted with
OMEMO Media Sharing
which is what is commonly used these days, but this method has not been accepted as a standard as it was a workaround for limitation of legacy OMEMO implementation. The proper way is now specified with
Stateless File Sharing
and is encrypted with
Encryption For Stateless File Sharing
. Those methods are currently only usable when OMEMO:2 is implemented in the peer client, and with them metadata on the shared file can be attached, including thumbnails.
Encryption has also been implemented for Jingle (
XEP-0391
and
XEP-0396
), which is notably used for Jingle File Transfer (specially useful for large files transfers).
So to summarize, nearly everything (instant messaging, files uploaded, large file transfers, all pubsub related features) can now be e2e encrypted with Libervia.
Possible Future
With the AP gateway permitting to reach the whole AP network, all the new features implemented, and the work done on e2e encryption, Libervia has everything to be a solid option for communication. After the recent events regarding a famous commercial network, we see a breakthrough of ActivityPub that will hopefully last over time. We can now access AP from XMPP, while having the possibility to have e2e encrypted private conversations or even blogs or calendar events.
As far as I know this is, so far, something unique for a Libre decentralized software. However, there is still work to do on stabilization on UI/UX update before this is really usable.
Those feature were planned for very long (years), but the lack of resources made them slow to come. The grant has made it possible to greatly accelerate the pace of development, and I doubt that it would have been possible to have all that without it.
Regarding how large the project is, and my family life, it's not possible any more to develop seriously this project on my free time alone (and I would like to do other things, sometimes, of my free time).
In other words, I need to find a way to sustain the development of Libervia for the years to come, so I can work full-time on it, and with some luck, build a team. I'm thinking very seriously about it these days, I'll probably write on this topic in a little while. If you are willing to help in any way, please contact me (on the Libervia room linked above for instance).
That's all for this progress note. I'm now working on stabilization and UI/UX update on the web frontend.
Libervia v0.8 « La Cecília »
goffi
30/11/2021, 23:02
planet-libre
seenthis
SàT
Libervia
project
libre
release
Je suis fier d'annoncer la sortie de Libervia 0.8 « La Cecília » (anciennement nommé « Salut à Toi »), après plus de 2 ans de développement.
Cette version est une avancée majeure pour préparer le futur du projet. Voyons les principales nouveautés.
Changement de nom
Par soucis de simplicité, le projet a été renommé en «
Libervia
» (qui était auparavant le nom du frontal web), et tous les frontaux officiels ont maintenant un nom évident comme
Libervia Web
Libervia Desktop
Mobile
(c'est le même frontal dans les 2 cas),
Libervia CLI
pour
Command-Line Interface
(interface en ligne de commande), et
Libervia TUI
pour
Terminal User Interface
(interface utilisateur pour terminaux). Le backend (service d'arrière-plan) devient, comme vous pouvez le deviner,
Libervia Backend
. Les anciens noms sont toujours utilisés comme alias.
En plus de la volonté de simplification, le changement de nom a été fait pour s'adapter à un public international : certaines personnes pensaient que « Salut à Toi » était réservé à des personnes francophones uniquement. Espérons que ce sera plus simple pour tout le monde, et que les gens ne seront plus perdus avec tous les noms qui étaient utilisés auparavant.
Notez que le changement de nom influence aussi votre fichier de configuration qui s'appelle désormais
libervia.conf
sat.conf
fonctionne toujours pour le moment). Les sections y ont été mises à jour avec de nouveau noms (pour configurer l'interface en ligne de commande, vous devez maintenant utiliser la section
[cli]
au lieu de
[jp]
, pour l'interface web, c'est à présent
[web]
au lieu de
[libervia]
). Veuillez vérifier la
documentation
si nécessaire.
Changements techniques
Libervia a été porté en Python 3, ce qui a ouvert la porte à d'autres changements.
Brython
a été intégré à Libervia Web, pour remplacer "Pyjamas" (un port en Python de
GWT
), qui n'était plus maintenu.
Nunjucks
est aussi désormais intégré à Libervia Web, ce qui permet l'utilisation commune de certains modèles avec
Jinja
. C'est particulièrement utile pour faire des pages qui fonctionnent avec ou sans JavaScript.
L'implémentation d'OMEMO a été complétée avec le chiffrement des salons de discussion (MUC) et des fichiers (via
XEP-0454: OMEMO Media Sharing
).
De nombreuses autres modifications ne sont pas expliquées dans cette note pour éviter de la rendre indigeste. Vous pouvez vous référer à
la liste des changements
(en anglais) pour plus d'informations.
Nouveau thème par défaut pour Libervia Web
Un nouveau thème basé sur le très bon cadriciel CSS
Bulma
est devenu le thème par défaut. Grâce à lui, l'interface est plus claire et agréable à utiliser.
Invitations
Un système d'invitation simple a été intégré au frontal web, et il peut être utilisé soit pour donner accès à quelque chose à quelqu'un ayant déjà un compte XMPP, soit pour inviter quelqu'un par courriel. Le but est d'être capable de partager des choses (comme un album photo ou un événement) avec famille et amis sans s'attendre à ce qu'ils ou elles installent un logiciel ou créent un compte quelconque.
Listes
Un gestionnaire de tickets décentralisé est implémenté depuis la verison 0.7, et il est notamment utilisé pour gérer les tickets de Libervia lui-même. Jusqu'ici, il fonctionnait grâce à une fonctionnalité non standard disponible uniquement sur Libervia Pubsub (anciennement « SàT Pubsub », un composant Pubsub/PEP indépendant du serveur développé en parallèle de Libervia).
Cette fonctionnalité a été renommée avec le terme plus générique « listes » et utilise désormais
XEP-0346: Form Discovery and Publishing
, ce qui la rend utilisable avec un service pubsub générique.
Toutes sortes de listes peuvent être créées, que ce soit des tickets pour suivre les rapports de bogues ou demandes de fonctionnalités pour un projet, des listes de tâches à faire, de course, etc. Grâce à l'utilisation de XMPP pubsub, ces listes peuvent être fédérées, et les permissions peuvent être gérées facilement (pour, par exemple, permettre aux membres de la famille de modifier une liste de courses).
Actuellement, 3 types de listes sont disponibles (tickets génériques, tâches, courses), mais on peut s'attendre à en voir plus dans les futures versions.
Albums photo
Des nombreuses améliorations ont été faites pour les albums photo du frontal web. Ils peuvent désormais être créés ou supprimés depuis Libervia Web, des photos ou des vidéos peuvent être téléversées, un diaporama utilisable avec un écran tactile et/ou sur petits écrans est disponible,
ogv.js
a été intégré pour rendre possible le visionnage de vidéos en Ogg Vorbis/Opus/Theora ans WebM VP8/VP9/AV1 sur les plateformes qui ne le supportent pas nativement et le système d'invitations mentionné plus haut a été intégré.
Bureau
L'interface utilisateur de Libervia Desktop a également été mise à jour : la barre de menu supérieure a été supprimée, le glisser/déposer de fichiers est maintenant possible suivant la plateforme, la messagerie instantanée a un défilement infini, un nouvel écran de « sélection de conversation » rend le choix d'une entité avec qui discuter ou d'un salon à joindre plus faciles, les pièces jointes des messages sont affichées de manière plus agréable, entre autres améliorations.
Du travail a aussi été effectué sur Libervia Mobile (qui ne fonctionne que sur Android pour le moment), mais ce frontend n'est pas encore assez stable pour les utilisateurs finaux.
Ligne de commande
L'interface en ligne de commande est à présent
entièrement documentée
(en anglais) et suite au changement de nom peut être utilisé soit avec la commande
libervia-cli
, soit la version courte
li
(l'ancien
jp
fonctionne encore pour le moment). Parmi les nouvelles commandes on peut mettre en avant
li file get
qui peut télécharger un fichier avec la gestion du schéma
aesgcm
(c.-à-d. « OMEMO Media Sharing », le partage de média via OMEMO), qui en fait une sorte de wget avec OMEMO.
li file upload
gère aussi le chiffrement de bout en bout, il est ainsi facile de partager un fichier chiffré depuis la ligne de commande ou un script.
La couleur d'arrière-plan est maintenant automatiquement détectée sur les émulateurs de terminaux compatibles et le thème est choisi en conséquence.
Mais aussi…
Composant de partage de fichiers
Libervia peut s'utiliser comme un composant (ce qui peut être vu comme un plugin générique pour serveur XMPP), et il inclut un composant de partage de fichiers.
Ce composant stocke les fichiers qui peuvent être retrouvés soit selon les permissions accordées, soit publiquement.
Les fichiers peuvent être téléversés ou téléchargés via
XEP-0234: Jingle File Transfer
et
XEP-0363: HTTP File Upload
est maintenant aussi implémenté, ce qui rend possible de partager des fichiers avec des liens HTTPS.
Ce composant peut maintenant être utilisé pour remplacer les implémentations internes aux serveurs XMPP de « HTTP File Upload ». En plus de la gestion des permissions fines, il n'y a pas de taille limite pour les fichiers et des quotas par utilisateurs peuvent être mis en place. Veuillez vous référer à
la documentation
pour voir comment faire. Les fichiers téléversés peuvent être retrouvés en utilisant
XEP-0329: File Information Sharing
et supprimés grâce aux commandes Ad-Hoc (
XEP-0050: Ad-Hoc Commands
).
Ce composant est nécessaire pour utiliser la fonctionnalité d'album photos.
Libervia Pubsub
Un composant Pubsub/PEP (anciennement appelé « SàT Pubsub ») est développé en parallèle de Libervia. Il vise à fournir une implémentation complète et indépendante du serveur.
Libervia Pubsub est publié en même temps que le client XMPP Libervia, et a, lui aussi, été porté sur Python 3.
Parmi les nouveautés, la recherche en texte plein a été implémentée (
XEP-0431: Full Text Search in MAM
), ainsi que la
XEP-0346: Form Discovery and Publishing
qui remplace les schémas de nœud (qui étaient non standard), et PEP fonctionne à présent pour le serveur lui-même, ce qui le rend utilisable pour la
XEP-0455: Service Outage Status
Images Docker
Les images Docker on été mises à jour et déplacées directement dans le dépôt
libervia-backend
(dans le sous-dossier
docker
).
Site officiel
Le
site officiel
a été mis à jour avec un nouveau thème (basé sur le nouveau thème de Libervia Web).
Installation
Libervia est disponible sur plusieurs distributions NU/Linux (au moins Debian et dérivées et Arch Linux). Malheureusement, la version actuellement dans Debian n'est pas à jour (à cause d'une publication à des dates différentes de Debian et Libervia), espérons que la nouvelle version sera disponible dans un dépôt « backport » rapidement.
Vous pouvez facilement installer Libervia sur n'importe quelle distribution en utilisant
pipx
$ pipx install libervia-backend
$ pipx install libervia-desktop
Ensuite lancez
libervia-backend
et un frontal (comme
libervia-cli
ou
libervia-desktop
). Reportez-vous à
la documentation
pour plus de détails.
Des images Docker sont disponibles, et en particulier un fichier
web-demo.yml
peut être utilisé avec
docker-compose
pour rapidement mettre en place une démo locale :
$ hg clone https://repos.goffi.org/libervia-backend
cd
libervia-backend/docker
$ docker-compose -f web-demo.yml up
Ensuite ouvrez votre butineur sur
et utilisez l'identifiant
demo
avec le mot de passe
demo
Et après ?
Un projet basé sur Libervia a été sélectionné par NLnet pour une subvention. Ce projet est en 2 parties : travailler sur une passerelle XMPP ⟺ ActivityPub, puis sur le chiffrement de bout en bout pour pubsub et les fichiers. Vous trouverez plus d'informations (en anglais) sur
ce billet de blog
et
sur la page du projet NLnet
. Ce projet est déjà bien entamé, et vous
pouvez suivre la progression sur mon blog
(qui est propulsé par Libervia/XMPP) ou sur le
système de gestion de tickets
(lui aussi propulsé par Libervia/XMPP). Un énorme merci à NLnet/NGI0 Discovery Fund !
D'autre part, il est prévu de travailler pour améliorer l'expérience utilisateur et la fonctionnalité de messagerie instantanée (en particulier sur le frontal web). Libervia vise à être particulièrement adapté pour les réseaux privés destinés à la famille et aux amis.
Enfin, j'ai eu la bonne surprise de voir que Libervia Web est utilisé pour le
blog de jmp.chat
JMP
est une entreprise qui vous fourni un numéro de téléphone qui peut être utilisé avec XMPP et SIP (vous pouvez appeler ce numéro depuis un téléphone traditionnel et recevoir l'appel vocal depuis un client XMPP)
Voilà qui conclu cette note de version. À bientôt !
Libervia v0.8 « La Cecília »
goffi
30/11/2021, 22:51
jabber-xmpp-en
SàT
Libervia
project
libre
release
I'm proud to announce the release of Libervia 0.8 « La Cecília » (formerly known as « Salut à Toi »), after more than 2 years of development.
This version is a big milestone preparing the future of the project. Let's have an overview of some major changes.
Project Renaming
In the interest of simplicity, the project has been renamed to "
Libervia
" (with was formerly the name of the web frontend), and all official frontends have now a straightforward name such as
Libervia Web
Libervia Desktop
Mobile
(same frontend for both),
Libervia CLI
for
Command-Line Interface
, and
Libervia TUI
for
Terminal User Interface
. The backend is, as you can guess,
Libervia Backend
. The former names are for now still used as aliases.
Beside simplicity, the name change was also due to concerns with international audience: some people were thinking that "Salut à Toi" was dedicated to French-speaking people only. Hopefully, it will be easier for everybody, and people won't get confused any more by all the names which were previously used.
Note that the renaming has implications on your configuration file which is now named
libervia.conf
sat.conf
is still working for now). The sections have been updated with new names (to configure the CLI frontend you now use
[cli]
section instead of
[jp]
, for the web frontend it's now
[web]
instead of
[libervia]
). Please check
documentation
in case of doubt.
Technical Changes
Libervia has been ported to Python 3, which has opened the door to other changes.
Brython
has been integrated to Libervia Web, to replace the unmaintained "Pyjamas" (which was a Python port of
GWT
).
Nunjucks
is also now integrated in Libervia Web, making it possible to share some templates with
Jinja
. This is notably useful to make some pages working with or without JavaScript.
OMEMO implementation has been completed with support for MUCs (group chats) and files (via
XEP-0454: OMEMO Media Sharing
).
Many other changes are not explained in this note to avoid it being indigestible, you can check the
CHANGELOG
for more information.
Libervia Web New Default Theme
A new theme for Libervia Web based on the nice
Bulma
CSS framework has been made and is now the default one. Thanks to it the interface is clearer and more pleasant to use.
Invitations
An easy to use invitation system has been implemented in Web frontend, and can be used either to give access to something to somebody with an existing XMPP account, or to invite somebody by email. The goal is to be able to share things (e.g. photo album, event) with family or friends without having to expect them to install a software or create an account.
Lists
A decentralised issue tracking system was implemented since version 0.7, which was notably used to manage Libervia's own tickets. It was using a non-standard feature available only in Libervia Pubsub (formerly "SàT Pubsub", server independent Pubsub/PEP component, a side project).
This feature has been renamed to "Lists" and now uses
XEP-0346: Form Discovery and Publishing
which makes it usable with a generic Pubsub service.
Any kind of list can be created, from project tickets to keep track of bug report of feature requests, to To-do list, grocery list, etc. Being based on XMPP pubsub, lists can be federated, and permissions can be managed easily (for instance to allow various family members to modify a shopping list).
For the moment 3 kinds of lists are available (generic tickets, To-Do, grocery), but more are expected to come in future versions.
Photo albums
Lots of improvements have been made on the photo albums in the web frontend. They can now be created or deleted from Libervia Web, photos or videos can be uploaded, a touch/mobile-friendly slideshow is available,
ogv.js
has been integrated to make possible the viewing of videos in Ogg Vorbis/Opus/Theora ans WebM VP8/VP9/AV1 on platforms not supporting them natively, and the invitation system mentioned above has been integrated.
Desktop
Libervia Desktop UI has also been updated, the top menu has been removed, file dropping is now possible on suitable platforms, chat has infinite scrolling, a new "chat selector" screen makes it easier to select entity to chat with or room to join, message attachments are show in a more user-friendly way, and several other improvements has been done.
Work has also been done on Libervia Mobile (which is Android only for now), but this frontend is not user-friendly enough yet for end-user.
CLI
The CLI frontend is now
fully documented
and following the renaming can now be accessed either by
libervia-cli
or the shorter
li
(legacy
jp
is still working for now). Among new commands we can highlight
li file get
which retrieve a file with support of
aesgcm
scheme (i.e. OMEMO Media Sharing), which makes it a kind of OMEMO enabled wget like.
li file upload
also handle end-to-end encryption, it's thus easy to share an encrypted file from command-line or a script.
Background colour is now automatically detected on compatible terminal emulator, and theme is adapted consequently.
But Also…
File Sharing Component
Libervia can act as a component (which can be seen as generic XMPP server plugins), and it includes a File Sharing Component.
This component store files which can be retrieved either according to given permissions or publicly.
Files can be uploaded or downloaded via
XEP-0234: Jingle File Transfer
and
XEP-0363: HTTP File Upload
is now also implemented, making it possible to share files via HTTPS link.
This component can now be used to replace internal XMPP servers HTTP File Upload implementations. In addition to the fine permission management, it does not have a size limit and user quotas can be set, check
the documentation
to see how to set them. Files uploaded can be retrieved using
XEP-0329: File Information Sharing
and deleted with
XEP-0050: Ad-Hoc Commands
This component is necessary to use the Photo Album feature.
Libervia Pubsub
A Pubsub/PEP component (formerly named "SàT Pubsub") is developed next to Libervia. It aims to provide a server independent feature-full implementation.
Libervia Pubsub is released at the same time as the Libervia XMPP client, and has also been ported to Python 3.
Among novelties, Full-Text Search has been implemented (
XEP-0431: Full Text Search in MAM
), as well as
XEP-0346: Form Discovery and Publishing
which replaces the former non-standard node schema, and PEP is now working for the server itself, making it usable for
XEP-0455: Service Outage Status
Docker Images
Docker images have been updated and moved directly to
libervia-backend
repository (in
docker
subdirectory).
Official Website
The
Official Website
has been updated with a new theme (based on Libervia Web new theme).
Installation
Libervia is available on several GNU/Linux distributions (at least Debian and derivative and Arch Linux). Unfortunately, the current Debian version is outdated (due to incompatible Debian and Libervia release dates), hopefully the new version will be available as a backport soon.
You can easily install Libervia on any distribution by using
pipx
$ pipx install libervia-backend
$ pipx install libervia-desktop
Then launch
libervia-backend
, and a frontend (e.g.
libervia-cli
or
libervia-desktop
). Check the
documentation
for details.
Docker images are available, and notably a
web-demo.yml
file can be used with
docker-compose
to quickly try a local demo:
$ hg clone https://repos.goffi.org/libervia-backend
cd
libervia-backend/docker
$ docker-compose -f web-demo.yml up
Then open your browser on
and use the login
demo
with password
demo
What's Next
A Libervia based project has been selected by NLnet for a grant. This project is in 2 parts: working on an XMPP ⟺ ActivityPub gateway, then on pubsub and files end-to-end encryption. You'll find more information on
this blog post
and on
NLnet project page
. The project has already well started, and you can
follow the progress on my blog
(which is Libervia/XMPP powered) or on the
ticket tracker
(which is also Libervia/XMPP powered). A huge thanks to NLnet/NGI0 Discovery Fund!
Besides, work is planned to improve user experience and instant messaging feature (notably on Web frontend). Libervia aims to be a good fit for private networks for family and friends.
Last but not least, I've been pleased to see that Libervia Web is used to power
jmp.chat blog
JMP
is a company which give you a real phone number which can be used with XMPP and SIP (you can call this number from a traditional phone and get the voice call from a XMPP client).
This concludes this release post. Stay tuned!
Libervia progress note 2021-W38
goffi
27/09/2021, 06:54
jabber-xmpp-en
SàT
Libervia
project
libre
Libervia progress
SàT progress
Hello,
it's time for a new progress note. The work is currently focused on ActivityPub Gateway, and progress has been done on pubsub cache search and the base component.
Pubsub Cache Full-Text Search
Next to the pubsub cache implementation, it was necessary to have a good way to search among items.
So far, Libervia was doing pubsub search using pubsub service's capabilities, and notably the
XEP-0431
(Full Text Search in MAM) implementation. This is working well (it's what is currently used on this very blog when you do use the search box), but has some pitfalls: the pubsub service must implement this XEP (and as far as I know, Libervia Pubsub is the only one which does it at the moment), the search can be done in a single node at a time only, each search request imply a new XMPP request to the pubsub service, and pubsub items must be in plain text (which is currently always the case, but pubsub end-to-end encryption is planned as second part of the granted NLNet project on which I'm working).
In regard to that, a local search is necessary. SQLAlchemy doesn't really have Full-Text Search (or FTS) support for SQLite out of the box, but it allows to use any SQL directly, thus I could use the really nice FTS engine available within it (
FTS5
). This is an extension, but in practice it is already installed most of the time (it is part of the
SQLite amalgamation
).
Thanks to the
JSON
support in SQLite, it is also possible to filter search requests on parsed data. That's really useful for features like blogs where you often want to do that (e.g. filtering on tags).
The cache search can be operated on all data in cache, that means that you can do search on items coming from multiple nodes and even multiple services. That opens the door to features like hashtags or blog suggestions.
Last but not least, search requests can be ordered by any parsed field. In other terms it will be possible to order a blog by declared publication date — which may be important if you want to import a blog —, or events by location.
To have an idea of the possibilities, you can check
the documentation of the CLI search command
Base ActivityPub Component
Once the preparatory steps have been done, the ActivityPub component itself could be started. In short, for people not used to XMPP, a "component" is a kind of generic plugin to server. You declare it in your server configuration, choose a JID and a "shared secret" (a password), run it with those parameters, and voilà.
For the AP gateway, Libervia runs the component. There is
documentation
to explain how to launch it, don't worry it's simple.
As I've got questions about this, here is a small schema giving an overview on how the whole thing is working:
I hope that it makes the whole thing more clear, otherwise don't hesitate to ask me for clarification.
As you can see, the gateway includes an HTTP server to communicate with AP software, but in many cases there will already be an HTTP server (website, XMPP web client, etc.). In this case, you'll have to redirect
/.well-known/webfinger
and
/_ap
requests to the gateway server.
For the development, I'm using Prosody as reference XMPP server implementation, and Mastodon as reference ActivityPub server implementation. I've set a local Mastodon installation, and I've chosen to use Docker for that, as it makes things easy to have a reproducible environment and to save and restore a specific state. It was not as trivial as I would expect to find the right configuration to use, I've found outdated tutorials, but I could manage to run the thing relatively easily.
Because we work with HTTPS, I've made a custom docker image with locale certification authority, so Mastodon could validate my gateway HTTP server certificate. I'm already doing that for docker image used for end-to-end tests of Libervia, nothing difficult. Surprisingly though, Mastodon could not resolve my instance, when
HTTPie
running from the same container could do it flawlessly. I've quickly realised that Mastodon was not respecting hosts declared in
/etc/hosts
(and added via
extra_hosts
in Compose file) and found
a relevant bug report
on Mastodon tracker. That was annoying, and I had to find a way to work around that. I've done it by running a local DNS Server, and
Twisted
offers a nice built-in one. Twisted DNS can easily use
/etc/hosts
to direct my local domains to my local IP, it's just a one liner such as
twistd3 -n dns --hosts-file=/etc/hosts -r
After that the domain was resolving, but to my surprise, Mastodon was still not able to communicate with my gateway, and even more bizarre my server was receiving no request at all. After a quick round of
tcpdump
wireshark
, I saw that indeed nothing was sent to my server.
Thanks to the Libre nature of Mastodon, I could resolve this by reading the source code, the
Mastodon::HostValidationError
led me to a section that made the whole picture clear: my server is on a local IP and Mastodon by default refuses to reach it (to avoid the
confused deputy attack
). With the
ALLOWED_PRIVATE_ADDRESSES
setting I could finally make Mastodon communicate with my server.
The
How to implement a basic ActivityPub server
tutorial made by
Eugen Rochko
(Mastodon original developer) is a nice article to start an ActivityPub implementation, it has been useful to build the base component (despite being a bit outdated, notably regarding signature).
I have to rant a bit, though, as the ActivityPub specification are not available in EPUB or PDF, making it difficult to read on an e-book reader. I could overcome that thanks to
pandoc
git clone https://github.com/w3c/activitypub.git
then
pandoc index.html --pdf-engine=xelatex -o activitypub.pdf
), it's really more comfortable to keep the reference like this.
So the base component is now available but only usable by developers (and only capable of sending message to ActivityPub for now). Things will be really exiting with the next 2 steps, as bidirectional communications will be available, and the gateway will be usable for early adopters. I don't expect those steps to be really long.
Oh, and to answer another question that I've had, yes you can use the same ActivityPub actor identifier as your XMPP JID. I'll explain next time how everything is accessed.
That's all for today.
Libervia progress note 2021-W31
goffi
03/08/2021, 11:03
jabber-xmpp-en
SàT
Libervia
project
libre
Libervia progress
SàT progress
Hello,
last weeks have been exhausting with lot of things at the same time. I've been preparing the release of the 0.8 version, and I wanted to have a couple of thing ready for that, notably a proper way to do translation.
Preparation of 0.8
As you may know,
I've implemented a docker integration into Libervia
to be able to run relatively easily third party software. This is working, but when testing in the production website I had to put the finishing touches to make it work (notably I've improved HTTP proxy and HTTPS management). I have then created projects and updated a couple of translations files.
As you can now see on
, there is a
translate
menu. Unfortunately I've closed the account creation for the moment, as I have to deal with licensing first. Indeed, nearly all Libervia ecosystem is for now in
AGPL v3+
, as there are only a few contributors (2 mains one, then only a small patches). The intent was and still is to be sure that the ecosystem stays in an Libre license, but this license may cause trouble in some edge cases, notably if we want to make an iOS frontend (the fruit store is notoriously causing trouble with AGPL licences).
Thus, I'll bring the subject at next general assemble of the "Salut à Toi" association, and see what we should do. One option could be to use
FSF's Fiducial Licence Agreement
to let the association the possibility to modify the licence as long as it stays a libre one. It would then be possible to add an exception for an iOS frontend. An other would be to avoid totally iOS. Anyway, this need some time and discussions, and if I open translations and get several contributions under AGPL v3+, it may be harder to set this up.
An other time consuming task was to continue with renaming and adapt package names (notably in
Pypi
). I've used a little trick to redirect legacy packages to the new ones: a new version of each legacy package is a simple
setup.py
depending on the new package (you can see it e.g. for
sat package
). I've also put in place a redirection on the Mercurial repositories, using the old repos will redirect to the new ones.
Finally, I've published the
0.8.0 beta 1
. You can install it easily with
pipx
First install
pipx
as explained in its documentation
Then install the
backend
with
pipx install libervia-backend
. You can follow the
documentation
to see how to configure it and launch it. This will include the
CLI
and
TUI
frontends.
If you want to test graphical frontends, you'll have to install
Libervia Media
with
hg clone https://repos.goffi.org/libervia-media
(assuming that you have Mercurial already installed), then
add it into your
libervia.conf
To install the
Desktop
frontend, use
pipx install libervia-desktop
To install the
Web
frontend, use
pipx install libervia-web
Note that the Desktop frontend is still for early adopters, I need to refactor message handling and do some optimisation and stabilisation to make it pleasant to use.
Please send feedbacks either as bug reports/feature requests on
the official bug tracker
, or on the XMPP chat room at
sat@chat.jabberfr.org
. I plan to only fix major issues though, as I'm now fully working on 0.9 and I'm focusing mainly on the ActivityPub gateway. However, bug reports/feature requests will be taken into account for 0.9 if not fixed directly in 0.8.
ActivityPub Gateway
After the hard work to move 0.8 close to the finish line has been done, I've started to work on 0.9 and thus the ActivityPub gateway. The first major thing to do was a refactoring of offline storage. Indeed Libervia (or SàT at the time) was started a long time ago with an Async framework (
Twisted
) long before
asyncio
even existed.
SQLite
has been chosen as backend to store data, and a hand made module based on
Twisted's adbapi
has been created. Despite the rough edges is has been working quite well all this time, and there was even a semi automatic way to update schemas between version. But the whole thing was starting to be difficult to maintain, and the schema update instructions were all kept in the same file.
Fortunately,
SQLAlchemy
, the most famous SQL databases abstraction with an optional Object Relational Mapper has recently added
support for asyncio
SQLAlchemy is a very powerful and widely used tool, so it has been a quite obvious decision to use it to replace the old system. But to use it, Twisted needs to use an asyncio loop, and Libervia was using GLib loop (or
reactor
in Twisted terms), due to the dependency to
dbus-python
Dbus-Python is, by its own authors words, not be the best D-Bus binding to use due to unfortunate design decision, so it was the occasion to replace it, and I've moved the backend to
TxDBus
, a Twisted native D-Bus implementation, which can run with any Twisted reactor. For technical reason, dbus-python is still used for frontends at the moment, but I plan to completely replace it before the end of 0.9 development.
This has required some work, but it was worth it, and after that I could switch to asyncio reactor and implement SQLAlchemy. I've decided to go with the ORM and not the core only as it is opening neat possibilities. I've first made a mapping corresponding to the last version of the database used for Libervia 0.8.
Once SQLAlchemy has been implemented and working, the next obvious decision was to use
Alembic
, the recommended SQLAlchemy based database migration tools (by the same authors). Thanks to this, migration files are now in separated files, and are really easy to create (Alembic can autogenerate a good part of a script when a migration is needed).
Thanks to all this, I can now easily make changes in database (while in old system I was hesitating due to the work implied). SQLAlchemy also paves the way to support other databases than SQLite. Even if I'm currently sticking with SQLite only, to keep focus, I'll probably add support for PostgreSQL and MariaDB/MySQL at some point.
Once all this work on storage backend has been finalised, the pubsub cache has been implemented.
Pubsub cache is operated transparently for end-user, and stores locally pubsub items (according to internal criteria). This is useful for many reasons: performances of course, but also it allows to do data analyse/indexing, for instance to retrieve all items with some terms (e.g.: to search by categories or hashtags). Pubsub cache is also useful to store data in a component (what is of interest for the ActivityPub gateway), or to store decrypted data (which will be of interest when we'll work on the e2e encryption with pubsub).
I'll pass the implementation details, you'll find the code on the
0.9 bookmark
, notably in the
pubsub cache plugin
, and I've written
documentation for developers
for some explanations.
New commands has been added to libervia-cli to manage the cache, in particular there is a
purge
command to delete items according to given criteria, which will save resources and disk space. With it, it's possible to delete only certain types of items (e.g. only blog posts), for all or only some profiles (for instance, only for the AP gateway). You can say a limit (e.g. delete all items which have not been updated for 6 months). Here again,
documentation has been written
to explain the commands.
While doing all that, I've noticed problem to cache correctly items (because of the flexibility of XMPP Pubsub, it's hard to impossible to say if we can share cache between users), thus I've written a protoXEP (i.e. a proposition for an XMPP Extension Protocol, or
XEP
) to fix the problem:
I've also submitted a
pull request to fix a problem in XEP-0060 (Publish-Subscribe)
While I was a working with standards, I've updated a XEP I've authored a couple of years ago to specify order of items:
XEP-0413: Order-By
Last but not least, while doing the
tests for the pubsub cache
I've created some helping methods (or
fixtures
in
pytest
terms) to help doing unit test.
This concludes the
first step
of the
XMPP-ActivityPub gateway
which was, as anticipated, a big one. The following steps should be done more quickly, and work on 0.8 should not be on the way anymore (I plan to publish 0.8 in early September).
That's all for this note, see you next time.