I’ve loved Wasmer, I still love Wasmer

This article could also have been titled How I failed to change Wasmer.

Today is my last day at Wasmer. For those who don’t know this name, it has a twofold meaning: it’s a very popular WebAssembly runtime, as well as a startup. I want to write about what I’ve been able to accomplish during my time at Wasmer (a high overview, not a technical view), and what forces me to leave the company despite being one of its co-founder. I reckon my testimony can help other people to avoid digging into the hell I (and my colleagues) had to endure. I’m available for work, you can contact me at ivan@mnt.io, @mnt_io, ivan-enderlin (LinkedIn).

From nothing to pure awesomeness

I’ve joined the Wasmer company at its early beginning, in March 2019. The company was 3 months old. My initial role was to write and to improve the runtime itself, and to create many embeddings, i.e. ways to integrate the Wasmer runtime inside various technologies, so that WebAssembly can run everywhere.

I can say with confidence that my work is a success. I’ve learned a lot, and I’ve worked on so many different projects, technologies, hacked so many things, collaborated with so many people, every action was led by the passion.

At the time of writing, Wasmer has an incredible growth. In 2.5 years only, the runtime has more than 10’500 stars on Github, and is one of the most popular WebAssembly runtime in the world! It’s used by many various companies, such as Confio, Fluence Labs, HOT-G, Brave, Google, Apple, SpaceMesh, Linkerd, SingleStore, CleverCloud or Kong to name a few (for the ones I can name though, however other companies are also using Wasmer in very critical environments).

Most of my engineering job happened on the Wasmer runtime itself. At the time of writing, I’m the #2 contributor on the project. I was working on every parts of the runtime: the API, the C API, the compilers, the ABI (mostly WASI), the engines, the middlewares, and the VM itself which is the most low-level foundamental layer of the runtime.

The runtime provides so many features. It is an impressively powerful runtime for WebAssembly, and I’m saying that with a neutral and respectful mindset. Not everything is perfect obviously but I did my best to set up a truly user-friendly learning environment, with an important documentation and a collection of examples that illustrate many features. I strongly believe it contributed to Wasmer’s popularity to great extent.

I would like to highlight the most notable embedding projects I’ve created:

  • wasmer-c-api is the C embedding for Wasmer. It’s part of the Wasmer runtime itself, and is fully written in Rust. The documentation, the C examples, everything is super polished to offer the best experience possible. Mark McCaskey and I are the authors of this project.
  • wasmer-python is the Python embedding for Wasmer. At the time of writing, it’s been installed more than 5 millions times (I’m counting the compiler packages too, like wasmer-compiler-cranelift and so on), and 1300 stars on Github. There is about 300’000 downloads per months, and it continues to grow! The code is written in Rust, and it relies on the awesome pyo3 project.
  • wasmer-go is the Go embedding for Wasmer. It’s hard to know how much total downloads we have because of how the Go ecosystem is designed, but we have about 60’000 downloads per months from Github (I’m excluding the forks of the project), and 1600 stars on Github. The code is written in Go and uses cgo to bind against the C API. Almost all blockchain projects that use WebAssembly are using wasmer-go, which is a popularity I wasn’t expecting.
  • wasmer-ruby is the Ruby embedding for Wasmer. It’s not as popular as the others, but it’s also very polished and it’s finding its place in the Ruby ecosystem. The code is written in Rust, and it relies on the awesome rutie project.
  • I won’t detail all the projects, but there is also wasmer-php, wasmer-java, wasmer-postgres… Because of the Wasmer runtime API and C API we have designed, many developers around the globe have been able to create a lot more embeddings, such as in C#, D, Elixir, R, Swift, Zig, Dart, Lisp and so on.

Other fun notable projects are:

As you might think, I’ve learned so much. The impostor syndrom was very present because I was constantly trying to do something I didn’t know. It’s part of the routine at Wasmer: Trying something for the first time. But it’s also what kept me motivated, and it was the energy for my passion.

This list above shows released projects, but I’ve also experimented (and sometimes at two hairs of a release) with:

  • Unikernels; this one was really fun, given a WebAssembly module and a filesystem, we were able to generate a unikernel that was executing the given program,
  • Parser; to write the fastest WebAssembly parser possible, it was working, but never released,
  • HAL (Hardware Abstraction Layer) ABI for WebAssembly, so that we can run WebAssembly on small chips super easily (think of IoT),
  • Networking; an extension to WASI to support networking (TCP and UDP sockets), with an implementation in Rust, libc, and even Zig! We were able to compile C programs to WebAssembly like cURL, or TCP servers written with kqueue or epoll etc, and to execute them on any platforms.

All those things were working.

It’s absolutely crazy what WebAssembly can do today, and I still truly and deeply believe in this technology. I’m not the only one: YCombinator and SpeedInvest are also founders that believe in Wasmer.

So. What a dream, huh?

The toxic working environment

WebAssembly is nothing without its community. I won’t name people to avoid missing important persons, but all the contributors are doing amazing work, to create something new, something special, something right.

Wasmer is a success. The Wasmer runtime is nothing without the incredible, marvelous, exceptional team of engineers behind it. In no particular order: Lachlan Sneff, Mark McCaskey, Julien Bianchi, Nick Lewycky, Heyang Zhou, Mackenzie Clark, Brandon Fish. All of them, with no exception, have put a lot of passion in this project. It is what it is today because of them and also because of the contributors we have been honored to welcome. The open source side of Wasmer was intense but also an important source of joy. It is a respectful place to work.

However, the inside reality was very different. All the employees hereinabove have left the company. Almost all of them due to a burn-out or conflicts or strong disagreements with the company leadership. I am leaving due to a severe burn-out. I would like to briefly share my journey to my first burn-out in few points:

  • I’ve started as an engineer. I love coding. I love hacking. In Wasmer, I’ve found a place to learn a lot and to express my passion. We had a lot of pressure mostly because our friendly “competitors” had more people dedicated to work on their runtimes, more money, more power, better marketing and so on. That’s the inevitable burden of any startup. When you’re competing against giants, that’s what happens. And that’s OK. It’s part of the game.
  • During that time, we were delivering more and more projects, more and more features, at an incredible pace. New hats: Release manager, project manager, more product ownership, more customers to be connected with, more contributors to help, more issues to triage, blog writer etc. The pace was accelerating too fast, something we did notice on multiple occasions.
  • The CEO, Syrus Akbary, had evidently a lot of pressure on its shoulders. It sadly resulted in the worst possible way: micro-management, stress, pressure, bad leadership, lack of vision, lack of trust in the employees, changing the roadmap constantly, lies, secrets etc.
  • As one of the older in the company, with a family of two kids, I probably got more “wisdom”. I’ve decided to create a safe place for employees to express their frustrations, their needs, to find solutions together. De facto, I became the “person of trust” in the company. I got new hats, new pressures.
  • SARS-CoV-2 hit. School at home. Lock-down. More micro-management, more stress. Wasmer was running out of money. I brought a new investor that saved the company. New hat.
  • After too many departures (85% of the engineering team!), I tried to take more space and to take more responsabilities in the company. That was at the beginning of 2021. It was my last attempt to save the company from a disaster before leaving. I couldn’t imagine leaving such brilliant and successful projects without having tried everything I could.
  • Then I became a late co-founder of Wasmer. Too many new hats: Doing hiring interviews, accountabilities, helping to define the roadmap (with another awesome person, friend, and employee), handling legal aspects to hire people in multiple countries with non-precarious contracts etc.
  • Obviously, I was also doing the job of all the engineers that have left. They were not replaced for unknown reasons. It was absolutely madness. The pace was unsustainable.
  • Finally, the crack. The CEO continued to change the roadmap, to take bad decisions, to not recognize all the efforts we were doing to save/grow the company. It was my turn to be declared in a severe burn-out by my physician. The last engineer to fall.

Another point: Syrus Akbary also has made many public errors that have created hostility against the company. Hopefully people were smart and kind enough to make the difference between the employees and the CEO (I won’t name the people but they will recognize themselves: Thank you). I tried to fix that situation. Discussing with dozens of person to restore empathy and forgiveness, to create better collaboration, to cure and move forward. It was exhausting. I know people have appreciated what I did, but my mental health was ruined.

Considering all the time I’ve devoted to the company, the very few consideration I got in return, the countless work hours (4 days per week, but frequently closing the computer at 1am due to very late meetings, I was working like hell), the precarious contract I had (did you ever see a co-founder with a freelancer contract?), the toxic working environment, the constant pressure etc., my passion was intact but my motivation was seriously damaged. Doing overtime was never recorded and was happening more than frequently, but taking half a day to take care of a sick child was immediately counted as holidays; the balance was broken. Criticisms. Micro-management. Disapprovals. Rewriting the facts and the reality to criticize what you’re doing, flipping things against you, avoiding discussion when things get stormy. We even had a meeting titled “Why you’re not productive enough” whilst everyone was working as hell, right after the rewrite of the entire runtime to release Wasmer 1.0, a period we all affectionately called “The Rewrite of Hell”. The team deserved vacations, congratulations, attentions, gratitude, … not such a shitty meeting. Well, you get the idea.

When I’ve been declared in a severe burn-out, I had to take a break. The reaction from the CEO was… unexpected: Zero empathy, asking to never ever being sick again (otherwise I will be fired), dividing my equities, asking me to work more, saying I’ve never been involved in the company etc. That was the final straw to me. That’s the wrong way to treat an employee, a collaborator, a contributor, the co-founder.

What’s next?

I need to recover. As you can imagine, working 2.5 years at this pace leaves sequelae. Hopefully a couple of months should suffice.

I’m still in love with Wasmer, the runtime, the open source projects we have created. It has a bright future. More companies are contributing to it, more individual contributors are bringing their stones to the monument. The project is owned by the public, by the users, by the contributors, they are doing most of the work today. It’s well tested, well documented, it’s easy to contribute to it. It’s a fabulous open source piece of technology.

I strongly hope Wasmer, the company, will change. The products that are going to be released are absolutely fantastic. It’s a technology shift. It will enable the Decentralized Web, how we compile and distribute programs, how we will even consume programs. Wasmer has a solid bright future. I really hope things will change, and I wish the best to and am passing on the torch to the adventurers that will continue to move the company forward. I’m just too skeptical that things can improve or even slightly change. We have built something great. Please take a great care of it.

As I said, I’m available for a new adventure, you can contact me at ivan@mnt.io, @mnt_io, ivan-enderlin (LinkedIn).

Discussions on Twitter and on HackerNews.

Bye bye WhatsApp, hello ?

Ce billet est en français, essentiellement à destination de mes ami(e)s et familles, il sert à vulgariser très rapidement les enjeux autour de WhatsApp, Signal, Telegram et Matrix (spoiler, c’est le gagnant). Tout le monde me pose la même question, alors voici une réponse rapidement en brouillon qui va m’éviter du copier/coller !

Je ne rentre volontairement pas dans les détails. Il faut que ce document reste à la portée de tous, sans aucune connaissance en réseau, chiffrement, sécurité etc. Ceux qui ont ces connaissances savent déjà que Matrix est le réseau vers lequel aller ;-).

Les bases

Quand on parle de messageries, il y a 2 choses primordiales plus 1 bonus :

  • le chiffrement ;
  • la topologie du réseau (c’est facile, n’ayez pas peur) ;
  • l’accès libre et gratuit sans restriction au code source (open source).

Nous pouvons aussi parler du modèle économique du réseau rapidement, voir le tableau comparatif.

Le chiffrement

Pour respecter la vie privée et éviter l’espionnage et le vol des données, il faut que le chiffrement se fasse de bout en bout (end to end encryption). Ça veut dire que vous seul avez la clé pour chiffrer et/ou déchiffrer vos messages, et personne d’autre. Par message, j’entends message texte, audio, image, vidéo, appels audios-vidéos, tout. Vos données sont à vous, et uniquement vous, et personne ne peut les utiliser, à part la personne avec qui vous les partagez (qui elle, normalement à une clé de déchiffrement par exemple). Les clés servent aussi à identifier la personne avec qui vous parlez, ça permet d’éviter le vol d’identité.

La topologie

La plupart des réseaux sont centralisés : ça veut dire qu’on a un gros silot, un énorme ordinateur/serveur, et que tout le monde est dessus. Ça pose plein de problèmes :

  • impossible d’avoir le contrôle dessus ;
  • impossible de faire confiance ;
  • faille unique.

Je prends l’exemple de WhatsApp pour illustrer tout ça parce que ça parle à tout le monde : Facebook décide unilatéralement de déchiffrer le réseau, personne n’a le contrôle dessus, c’est une violation grave de la vie privée de milliards de personnes et on ne peut rien faire (à part quitter le réseau). Avions-nous confiance dans ce que faisait Facebook avec nos données WhatsApp avant ? Non, aucunement. Ils disaient que c’était chiffré, l’était-ce vraiment ? J’accorde plus de confiance dans ceux qui ont créé et chiffré le réseau avant qu’il ne soit racheté par Facebook, donc j’ai envie d’y croire, mais… je ne peux pas le vérifier ! Pourquoi ? Parce que personne (en dehors de quelques employés chez Facebook) n’a accès au code source, aux programmes, qui font tourner WhatsApp. Et pour le côté faille unique, si Facebook est attaqué, c’est l’entièreté du réseau qui s’effondre, c’est une faille unique, un single point of failure comme on dit dans le métier. Pareil si le réseau est hacké, c’est un accès illimité à tout le réseau.

Aucune transparence = aucune confiance.

Mais il existe une alternative majeure bien sûr ! Les réseaux décentralisés. Au lieu d’avoir un serveur, il y a en des centaines, des milliers. Il n’y a plus de contrôle possible. Il n’y a plus de single point of failure. Un hacker ne peut accéder au pire qu’aux données d’un seul serveur, pas de tous les serveurs (il existe pleins d’exceptions mais je vulgarise, hein). Nous pouvons créer autant de serveurs que nous le souhaitons. Souvent, ce sont des réseaux open source, donc nous pouvons lire le code des programmes, vérifier qu’ils font bien ce qu’ils proclament faire.

Tableau comparatif

Comparons les services populaires avec ces critères de bases.

ServiceChiffrementTopologieOpen source
WhatsAppbout en bout (pour le moment)centralisé (US)non
Telegrambout en bout (pour le moment)centralisé (Dubaï, US)non
Signalbout en boutcentralisé (US)oui mais…
Matrixbout en boutdécentraliséoui
Comparons la base !

Signal est open source, mais nous ne pouvons pas vérifier ce qui est installé sur les serveurs, parce que le serveur est privé. De plus, le serveur open source n’a pas été mis à jour depuis avril 2020, en année Informatique, c’est très long. Ça cache quelque chose ? Aucune idée, je ne peux pas le savoir, car je n’ai pas d’éléments pour prendre une décision. Est-ce que je veux déposer mes données privées sur un service dans lequel je n’ai pas confiance ?

En plus, Signal comme WhatsApp sont hébergés/situés aux US, avec les lois liberticides qu’on leur connaît bien (comme le Cloud Act). Signal limite la casse grâce au chiffrement de bout en bout, mais peut être qu’une backdoor est présente et qu’on ne le saura jamais.

Aucune transparence = aucune confiance

Les réseaux décentralisés sont supérieurs à tous les niveaux (pas de contrôle, pas de hack massif etc.). Les réseaux open source sont ceux en qui nous pouvons avoir confiance. Donc le choix est vite fait, le gagnant ici est Matrix.

Comparons maintenant comment les services sont financés, parce que c’est important. Si un service n’est pas rentable, il pourrait avoir de l’appétit pour les données de ses utilisateurs, et là c’est dangereux (c’est exactement ce qu’il se passe avec Facebook et WhatsApp).

WhatsAppFacebook veut utiliser les données privées pour vendre de la publicité ciblée.
TelegramLes fondateurs sont millionnaires et injectent de l’argent.
Dans peu de temps, financement via pubs et comptes premiums.
SignalOrganisation à but non-lucratif qui opère via des dons.
MatrixMatrix développe, offre ou vend des services autour du réseau, mais pas autour des données !
Comment sont financés les services ?

Les gagnants ici sont Signal et Matrix.

Conclusion : Matrix gagnant

Dans le cas des réseaux centralisés, Signal est une meilleure alternative à WhatsApp et Telegram de part son mode de financement (donc son appétit pour les données des utilisateurs), mais ils sont tous sujets aux même problèmes : aucune confiance car pas de transparence, hébergés aux US etc.

Mais les réseaux décentralisés sont supérieurs car ils résolvent tous ces problèmes ! Matrix est décentralisé, est financé par des services autour du réseau mais pas par les données du réseau (qui sont inaccessibles de toute façon, elles n’existent que sur vos téléphones et ordinateurs, nul part ailleurs).

J’utilise Matrix. Je vous conseille d’utiliser Matrix. Partir sur Signal, c’est sortir de la gueule d’un loup pour aller dans celle d’un autre. Je suis admiratif du travail des développeurs de chez Signal, ils sont vraiment bons, leur protocole de chiffrement est magnifique, mais je n’ai pas confiance dans leur service parce que je ne peux pas. Et personne ne le peut.

J’utilise aussi WhatsApp et Signal pour rester en contact avec mes amis et ma famille, et leur dire d’utiliser Matrix, mais je n’y publierai jamais de données personnelles, photos ou quoi que ce soit, je n’ai aucune confiance. Libre à vous aussi d’utiliser plusieurs réseaux, après tout nous jonglons déjà avec plusieurs réseaux (mail, SMS, WhatsApp, Matrix, Twitter, Mastodon etc.), ça n’est pas un problème !

Premier pas avec Matrix

C’est parti, petit tuto Matrix. Le réseau est exceptionnel, mais le client officiel (Element) est encore un peu « brut » à utiliser comparé à Signal ou WhatsApp. Notez que ça évolue très très vite (je compte 616 contributeurs qui travaillent dessus bénévolement, encore une grande force de l’open source !).

Ce qui va vous titiller le plus c’est : vous ne pouvez pas toujours identifier vos contacts par numéro de téléphone (seulement s’ils sont enregistrés sur un serveur d’identité). Pourquoi ? Parce que votre compte à un identifiant, comme une adresse email. Le mien est @mnt_io:matrix.org (le format est @identifiant:serveur). C’est bien meilleur pour la vie privée. Et pis, ça n’est pas différent de MSN ou de tout autre réseau de l’époque, c’est vraiment WhatsApp qui a imposé la « découvertabilité » par le numéro de téléphone. Bien que très pratique, c’est dangereux pour la vie privée.

Donc, go, on installe le client :

Puis on crée un compte, et ajoutez moi. C’est parti !

Matrix/Element est basé sur les groupes. Les chats « directs » (1:1) sont des groupes aussi. Vous pouvez même rejoindre des rooms (gros groupes, des communautés) avec des centaines voire des milliers de personnes dedans. C’est très flexible.

Parce que c’est open source, n’importe qui peut écrire son propre client (programme qui se connecte au réseau). Il existe des clients alternatives, comme Nio ou FluffyChat, ou même Ditto Chat. Tous ces clients sont encore en beta, mais ça montre un futur très excitant pour Matrix avec des clients de plus en plus aboutis !

Matrix, Element, Vector, hein ?

  • Element c’est le nom de l’entreprise qui travaille/développe le réseau, les serveurs et le client ;
  • Matrix c’est le nom du réseau ;
  • Vector, c’est l’ancien nom d’Element.

On parle souvent de façon indifférentiée de Matrix ou Element, c’est un abus de langage.

Before: Mark as read.

Now: Mark has read.

Et par pitié, quittez Facebook…

Announcing the first Java library to run WebAssembly: Wasmer JNI

This is a copy of an article I wrote for Wasmer.

Image for post

WebAssembly is a portable binary format. That means the same file can run anywhere.

To uphold this bold statement, each language, platform and system must be able to run WebAssembly — as fast and safely as possible.

People who are familiar with Wasmer are used to this kind of announcement! Wasmer is written in Rust, and comes with an additional native C API. But you can use it in a lot of other languages. After having announced libraries to use Wasmer, and thus WebAssembly, in:

…we are jazzed to announce that Wasmer has now landed in Java!

Let’s discover the Wasmer JNI library together.


The Wasmer JNI (Java Native Interface) library is based on the Wasmer runtime, which is written in Rust, and is compiled to a shared library. For your convenience, we produce one JAR (Java Archive) per architecture and platform. By now, the following are supported, consistently tested, and pre-packaged (available in Bintray and Github Releases):

  • amd64-darwin for macOS, x86 64bits,
  • amd64-linux for Linux, x86 64 bits,
  • amd64-windows for Windows, x86 64 bits.

More architectures and more platforms will be added in the near future. If you need a specific one, feel free to ask! However, it is possible to produce your own JAR for your own platform and architecture.

The JAR files are named as follows: wasmer-jni-$(architecture)-$(os)-$(version).jar. Thus, to include Wasmer JNI as a dependency of your project (assuming you use Gradle), write for instance:

dependencies {
implementation "org.wasmer:wasmer-jni-amd64-linux:0.2.0"

JAR are hosted on the Bintray/JCenter repository under the wasmer-jni project. They are also attached to our Github releases as assets.

Calling a WebAssembly function from Java

As usual, let’s start with a simple Rust program that we will compile to WebAssembly, and then execute from Java.

pub extern fn sum(x: i32, y: i32) -> i32 {
    x + y

After compilation to WebAssembly, we get a file like this one, named simple.wasm.

The following Java program executes the sum exported function by passing 5 and 37 as arguments:

import org.wasmer.Instance;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

class SimpleExample {
    public static void main(String[] args) throws IOException {
        // Read the WebAssembly bytes.
        byte[] bytes = Files.readAllBytes(Paths.get("simple.wasm"));

        // Instantiate the WebAssembly module.
        Instance instance = new Instance(bytes);

        // Get the `sum` exported function, call it by passing 5 and 37, and get the result.
        Integer result = (Integer) instance.exports.getFunction("sum").apply(5, 37)[0];

        assert result == 42;


Great! We have successfully executed a Rust program, compiled to WebAssembly, in Java. As you can see, it is pretty straightforward. The API is very similar to the standard JavaScript API, or the other API we have designed for PHP, Python, Go, Ruby etc.

The assiduous reader might have noticed the [0] in .apply(5, 37)[0] pattern. A WebAssembly function can return zero to many values, and in this case, we are reading the first one.

Note: Java values passed to WebAssembly exported functions are automatically downcasted to WebAssembly values. Types are inferred at runtime, and casting is done automatically. Thus, a WebAssembly function acts as any regular Java function.

Technically, an exported function is a functional interface as defined by the Java Language Specification (i.e. it is a FunctionalInterface). Thus, it is possible to write the following code where sum is an actual function (of kind org.wasmer.exports.Function):

import org.wasmer.Instance;
import org.wasmer.exports.Function;

import java.io.IOException;
import java.nio.file.Files;
import java.nio.file.Paths;

class SimpleExample {
    public static void main(String[] args) throws IOException {
        // Read the WebAssembly bytes.
        byte[] bytes = Files.readAllBytes(Paths.get("simple.wasm"));

        // Instantiate the WebAssembly module.
        Instance instance = new Instance(bytes);

        // Declare the `sum` function, as a regular Java function.
        Function sum = instance.exports.getFunction("sum");
        // Call `sum`.
        Integer result = (Integer) sum.apply(1, 2)[0];

        assert result == 3;


But a WebAssembly module not only exports functions, it also exports memory.

Reading the memory

A WebAssembly instance has one or more linear memories, a contiguous and byte-addressable range of memory spanning from offset 0 and extending up to a varying memory size, represented by the org.wasmer.Memory class. Let’s see how to use it. Consider the following Rust program:

pub extern fn return_hello() -> *const u8 {
    b"Hello, World!\0".as_ptr()

The return_hello function returns a pointer to the statically allocated string. The string exists in the linear memory of the WebAssembly module. It is then possible to read it in Java:

import org.wasmer.Instance;
import org.wasmer.Memory;

import java.io.IOException;
import java.nio.ByteBuffer;
import java.nio.file.Files;
import java.nio.file.Paths;

class MemoryExample {
    public static void main(String[] args) throws IOException {
        // Read the WebAssembly bytes.
        byte[] bytes = Files.readAllBytes(Paths.get("memory.wasm"));

        // Instantiate the WebAssembly module.
        Instance instance = new Instance(bytes);

        // Get a pointer to the statically allocated string returned by `return_hello`.
        Integer pointer = (Integer) instance.exports.getFunction("return_hello").apply()[0];

        // Get the exported memory named `memory`.
        Memory memory = instance.exports.getMemory("memory");

        // Get a direct byte buffer view of the WebAssembly memory.
        ByteBuffer memoryBuffer = memory.buffer();

        // Prepare the byte array that will hold the data.
        byte[] data = new byte[13];

        // Let's position the cursor, and…
        // … read!

        // Let's encode back to a Java string.
        String result = new String(data);

        // Hello!
        assert result.equals("Hello, World!");


As we can see, the Memory API provides a buffer method. It returns a direct byte buffer (of kind java.nio.ByteBuffer) view of the memory. It’s a standard API for any Java developer. We think it’s best to not reinvent the wheel and use standard API as much as possible.

The WebAssembly memory is dissociated from the JVM memory, and thus from the garbage collector.

You can read the Greet Example to see a more in-depth usage of the Memory API.

More documentation

The project comes with a Makefile. The make javadoc command will generate a traditional local Javadoc for you, in the build/docs/javadoc/index.html file.

In addition, the project’s README.md file has an API of the wasmer library Section.

Finally, the project comes with a set of examples. Use the make run-example EXAMPLE=Simple to run the SimpleExample.java example for instance.


WebAssembly aims at being safe, but also fast. Since Wasmer JNI is the first Java library to execute WebAssembly, we can’t compare to prior works in the Java ecosystem. However, you might know that Wasmer comes with 3 backends: Singlepass, Cranelift and LLVM. We’ve even written an article about it: A WebAssembly Compiler tale. The Wasmer JNI library uses the Cranelift backend for the moment, which offers the best compromise between compilation-time and execution-time.


Asami (d0iasm on Twitter) has improved this project during its internship at Wasmer under my guidance. She finished the internship before the release of the Wasmer JNI project, but she deserves credits for pushing the project forward! Good work Asami!

This is an opportunity to remind everyone that we hire anywhere in the world. Asami was working from Japan while I am working from Switzerland, and the rest of the team is from US, Spain, China etc. Feel free to contact me (@mnt_io or @syrusakbary on Twitter) if you want to join us on this big adventure!


Wasmer JNI is a library to execute WebAssembly directly in Java. It embeds the WebAssembly runtime Wasmer. The first releases provide the core API with Module, Instance, and Memory. It comes pre-packaged as a JAR, one per architecture and per platform.

The source code is open and hosted on Github at https://github.com/wasmerio/java-ext-wasm. We are constantly improving the project, so if you have feedback, issues, or feature requests please open an issue in the repository, or reach us on Twitter at @wasmerio or @mnt_io.

We look forward to see what you build with this!

Announcing the first Postgres extension to run WebAssembly

This is a copy of an article I wrote for Wasmer.

Image for post
Elephant. Source of the photo.

WebAssembly is a portable binary format. That means the same program can run anywhere.

To uphold this bold statement, each language, platform and system must be able to run WebAssembly — as fast and safely as possible.

Let’s say it again. Wasmer is a WebAssembly runtime. We have successfully embedded the runtime in other languages:

The community has also embedded Wasmer in awesome projects:

It is now time to continue the story and to hang around… Postgres!

We are so happy to announce a newcrazy idea: WebAssembly on Postgres. Yes, you read that correctly. On Postgres.

Calling a WebAssembly function from Postgres

As usual, we have to go through the installation process. There is no package manager for Postgres, so it’s a manual step. The Installation Section of the documentation explains the details; here is a summary:

$ # Build the shared library.
$ just build

$ # Install the extension in the Postgres tree.
$ just install

$ # Activate the extension.
$ echo 'CREATE EXTENSION wasm;' | \
      psql -h $host -d $database

$ # Initialize the extension.
$ echo "SELECT wasm_init('$(pwd)/target/release/libpg_ext_wasm.dylib');" | \
      psql -h $host -d $database

Once the extension is installed, activated and initialized, we can start having fun!

The current API is rather small, however basic features are available. The goal is to gather a community and to design a pragmatic API together, discover the expectations, how developers would use this new technology inside a database engine.

Let’s see how it works. To instantiate a WebAssembly module, we use the wasm_new_instance function. It takes 2 arguments: The absolute path to the WebAssembly module, and a prefix for the module exported functions. Indeed, if a module exports a function named sum, then a Postgres function named prefix_sum calling the sum function will be created dynamically.

Let’s see it in action. Let’s start by editing a Rust program that compiles to WebAssembly:

pub extern fn sum(x: i32, y: i32) -> i32 {
    x + y

Once this file compiled to simple.wasm, we can instantiate the module, and call the exported sum function:

-- New instance of the `simple.wasm` WebAssembly module.
SELECT wasm_new_instance('/absolute/path/to/simple.wasm', 'ns');

-- Call a WebAssembly exported function!
SELECT ns_sum(1, 2);

--  ns_sum
-- --------
--       3
-- (1 row)

Et voilà ! The ns_sum function calls the Rust sum function through WebAssembly! How fun is that 😄?

Inspect a WebAssembly instance

This section shows how to inspect a WebAssembly instance. At the same time, it quickly explains how the extension works under the hood.

The extension provides two foreign data wrappers, gathered together in the wasm foreign schema:

  • wasm.instances is a table with the id and wasm_file columns, respectively for the unique instance ID, and the path of the WebAssembly module,
  • wasm.exported_functions is a table with the instance_id,name, inputs, and outputs columns, respectively for the instance ID of the exported function, its name, its input types (already formatted for Postgres), and its output types (already formatted for Postgres).

Let’s see:

-- Select all WebAssembly instances.
SELECT * FROM wasm.instances;

--                   id                  |          wasm_file
-- --------------------------------------+-------------------------------
--  426e17af-c32f-5027-ad73-239e5450dd91 | /absolute/path/to/simple.wasm
-- (1 row)

-- Select all exported functions for a specific instance.
    instance_id = '426e17af-c32f-5027-ad73-239e5450dd91';

--   name  |     inputs      | outputs
-- --------+-----------------+---------
--  ns_sum | integer,integer | integer
-- (1 row)

Based on these information, the wasm Postgres extension is able to generate the SQL function to call the WebAssembly exported functions.

It sounds simplistic, and… to be honest, it is! The trick is to use foreign data wrappers, which is an awesome feature of Postgres.

How fast is it, or: Is it an interesting alternative to PL/pgSQL?

As we said, the extension API is rather small for now. The idea is to explore, to experiment, to have fun with WebAssembly inside a database. It is particularly interesting in two cases:

  1. To write extensions or procedures with any languages that compile to WebAssembly in place of PL/pgSQL,
  2. To remove a potential performance bottleneck where speed is involved.

Thus we run a basic benchmark. Like most of the benchmarks out there, it must be taken with a grain of salt.

The goal is to compare the execution time between WebAssembly and PL/pgSQL, and see how both approaches scale.

The Postgres WebAssembly extension uses Wasmer as the runtime, compiled with the Cranelift backend (learn more about the different backends). We run the benchmark with Postgres 10, on a MacBook Pro 15″ from 2016, 2.9Ghz Core i7 with 16Gb of memory.

The methodology is the following:

  • Load both the plpgsql_fibonacci and the wasm_fibonacci functions,
  • Run them with a query like SELECT *_fibonacci(n) FROM generate_series(1, 1000) where n has the following values: 50, 500, and 5000, so that we can observe how both approaches scale,
  • Write the timings down,
  • Run this methodology multiple times, and compute the median of the results.

Here come the results. The lower, the better.

Comparing WebAssembly vs. PL/pgSQL when computing the Fibonacci sequence with n=50, 500 and 5000.

We notice that the Postgres WebAssembly extension is faster to run numeric computations. The WebAssembly approach scales pretty well compared to the PL/pgSQL approach, in this situation.

When to use the WebAssembly extension?

So far, the extension only supports integers (on 32- and 64-bits). The extension doesn’t support strings yet. It also doesn’t support records, views or other Postgres types. Keep in mind this is the very first step.

Hence, it is too soon to tell whether WebAssembly can be an alternative to PL/pgSQL. But regarding the benchmark results above, we are sure they can live side-by-side, WebAssembly has clearly a place in the ecosystem! And we want to continue to pursue this exploration.


We are already talking with people that are interested in using WebAssembly inside databases. If you have any particular use cases, please reach us at wasmer.io, or on Twitter at @wasmerio directly or me @mnt_io.

Everything is open source, as usual! Happy hacking.

Announcing the fastest WebAssembly runtime for Go: wasmer

This is a copy of an article I wrote for Wasmer.

Image for post
Go loves WebAssembly — image attributions to the original Gopher drawing.

WebAssembly is a portable binary format. That means the same file can run anywhere.

To uphold this bold statement, each language, platform and system must be able to run WebAssembly — as fast and safely as possible.

Wasmer is a WebAssembly runtime written in Rust. It goes without saying that the runtime can be used in any Rust application. We have also successfully embedded the runtime in other languages:

We are super happy to announce github.com/wasmerio/go-ext-wasm/wasmer, a Go library to run WebAssembly binaries, fast.

Calling a WebAssembly function from Go

First, let’s install wasmer in your go environment (with cgo support).

export CGO_ENABLED=1; export CC=gcc; go install github.com/wasmerio/go-ext-wasm/wasmer

Let’s jump immediately into some examples.github.com/wasmerio/go-ext-wasm/wasmer is a regular Go library. The installation is automated with import "github.com/wasmerio/go-ext-wasm/wasmer".

Let’s get our hands dirty. We will write a program that compiles to WebAssembly easily, using Rust for instance:

pub extern fn sum(x: i32, y: i32) -> i32 {
    x + y

After compilation to WebAssembly, we get a file like this one, named simple.wasm.
The following Go program executes the sum function by passing 5 and 37 as arguments:

package main

import (
	wasm "github.com/wasmerio/go-ext-wasm/wasmer"

func main() {
	// Reads the WebAssembly module as bytes.
	bytes, _ := wasm.ReadBytes("simple.wasm")
	// Instantiates the WebAssembly module.
	instance, _ := wasm.NewInstance(bytes)
	defer instance.Close()

	// Gets the `sum` exported function from the WebAssembly instance.
	sum := instance.Exports["sum"]

	// Calls that exported function with Go standard values. The WebAssembly
	// types are inferred and values are casted automatically.
	result, _ := sum(5, 37)

	fmt.Println(result) // 42!

Great! We have successfully executed a WebAssembly file inside Go.

Note: Go values passed to the WebAssembly exported function are automatically cast to WebAssembly values. Types are inferred and casting is done automatically. Thus, a WebAssembly function acts as any regular Go function.

WebAssembly calling Go funtions

A WebAssembly module exports some functions, so that they can be called from the outside world. This is the entry point to execute WebAssembly.

Nonetheless, a WebAssembly module can also have imported functions. Let’s consider the following Rust program:

extern {
    fn sum(x: i32, y: i32) -> i32;

pub extern fn add1(x: i32, y: i32) -> i32 {
    unsafe { sum(x, y) } + 1

The exported function add1 calls the sum function. Its implementation is absent, only its signature is defined. This is an “extern function”, and for WebAssembly, this is an imported function, because its implementation must be imported.

Let’s implement the sum function in Go! To do so, need to use cgo:

  1. The sum function signature is defined in C (see the comment above import "C"),
  2. The sum implementation is defined in Go. Notice the //export which is the way cgo uses to map Go code to C code,
  3. NewImports is an API used to create WebAssembly imports. In this code "sum" is the WebAssembly imported function name, sum is the Go function pointer, and C.sum is the cgo function pointer,
  4. Finally, NewInstanceWithImports is the constructor to use to instantiate the WebAssembly module with imports. That’s it.

Let’s see the complete program:

package main

// // 1️⃣ Declare the `sum` function signature (see cgo).
// #include <stdlib.h>
// extern int32_t sum(void *context, int32_t x, int32_t y);
import "C"

import (
	wasm "github.com/wasmerio/go-ext-wasm/wasmer"

// 2️⃣ Write the implementation of the `sum` function, and export it (for cgo).
//export sum
func sum(context unsafe.Pointer, x int32, y int32) int32 {
	return x + y

func main() {
	// Reads the WebAssembly module as bytes.
	bytes, _ := wasm.ReadBytes("import.wasm")

	// 3️⃣ Declares the imported functions for WebAssembly.
	imports, _ := wasm.NewImports().Append("sum", sum, C.sum)

	// 4️⃣ Instantiates the WebAssembly module with imports.
	instance, _ := wasm.NewInstanceWithImports(bytes, imports)

	// Close the WebAssembly instance later.
	defer instance.Close()

	// Gets the `add1` exported function from the WebAssembly instance.
	add1 := instance.Exports["add1"]

	// Calls that exported function.
	result, _ := add1(1, 2)

	//   add1(1, 2)
	// = sum(1 + 2) + 1
	// = 1 + 2 + 1
	// = 4
	// QED

Reading the memory

A WebAssembly instance has a linear memory. Let’s see how to read it. Consider the following Rust program:

pub extern fn return_hello() -> *const u8 {
    b"Hello, World!\0".as_ptr()

The return_hello function returns a pointer to a string. The string terminates by a null byte, à la C. Let’s jump on the Go side:

bytes, _ := wasm.ReadBytes("memory.wasm")
instance, _ := wasm.NewInstance(bytes)
defer instance.Close()

// Calls the `return_hello` exported function.
// This function returns a pointer to a string.
result, _ := instance.Exports["return_hello"]()

// Gets the pointer value as an integer.
pointer := result.ToI32()

// Reads the memory.
memory := instance.Memory.Data()

fmt.Println(string(memory[pointer : pointer+13])) // Hello, World!

The return_hello function returns a pointer as an i32 value. We get its value by calling ToI32. Then, we fetch the memory data with instance.Memory.Data().

This function returns a slice over the WebAssembly instance memory. It can be used as any regular Go slice.

Fortunately for us, we already know the length of the string we want to read, so memory[pointer : pointer+13] is enough to read the bytes, that are then cast to a string. Et voilà !

You can read the Greet Example to see a more advanced usage of the memory API.


So far, github.com/wasmerio/go-ext-wasm/wasmer has a nice API, but …is it fast?

Contrary to PHP or Ruby, there are already existing runtimes in the Go world to execute WebAssembly. The main candidates are:

  • Life, from Perlin Network, a WebAssembly interpreter
  • Wagon, from Go Interpreter, a WebAssembly interpreter and toolkit.

In our blog post about the PHP extension, we have used the n-body algorithm to benchmark the performance. Life provides more benchmarks: the Fibonacci algorithm (the recursive version), the Pollard’s rho algorithm, and the Snappy Compress operation. The latter works successfully with github.com/wasmerio/go-ext-wasm/wasmer but not with Life or Wagon. We have removed it from the benchmark suites. Benchmark sources are online.

We use Life 20190521143330–57f3819c2df0, and Wagon 0.4.0, i.e. the latest versions to date.

The benchmark numbers represent the average result for 10 runs each. The computer that ran these benchmarks is a MacBook Pro 15″ from 2016, 2.9Ghz Core i7 with 16Gb of memory.

Results are grouped by benchmark algorithm on the X axis. The Y axis represents the time used to run the algorithm, expressed in milliseconds. The lower, the better.

Speed comparison between Wasmer, Wagon and Life. Benchmark suites are the n-body, Fibonacci, and Pollard’s rho algorithms. Speed is expressed in ms. Lower is better.

While both Life and Wagon provide on average the same speed, Wasmer (github.com/wasmerio/go-ext/wasmer) is on average 72 times faster 🎉.

It is important to know that Wasmer comes with 3 backends: Singlepass, Cranelift, and LLVM. The default backend that is used by the Go library is Cranelift (learn more about Cranelift). Using LLVM will provide performance close to native, but we decided to start with Cranelift as it offers the best tradeoff between compilation-time and execution-time (learn more about the different backends, when to use them, pros and cons etc.).


github.com/wasmerio/go-ext-wasm/wasmer is a new Go library to execute WebAssembly binaries. It embeds the Wasmer runtime. The first version supports all the required API for the most common usages.

The current benchmarks (a mix from our benchmark suites and from Life suites) show that Wasmer is — on average — 72 times faster than Life and Wagon, the two major existing WebAssembly runtimes in the Go world.

If you want to follow the development, take a look at @wasmerio and @mnt_io on Twitter, or @wasmer@webassembly.social on Mastodon. And of course, everything is open source at https://github.com/wasmerio/go-ext-wasm.wasmerio/go-ext-wasm🐹🕸️ Go library to run WebAssembly binaries. Contribute to wasmerio/go-ext-wasm development by creating an account on…github.com

Thank you for your time, we can’t wait to see what you build with us!

🐘+🦀+🕸 php-ext-wasm: Migrating from wasmi to Wasmer

This is a copy of an article I wrote for Wasmer.

Image for post
Elephant in the forest. Source of the photo.

First as a joke, now as a real product, I started to develop php-ext-wasm: a PHP extension allowing to execute WebAssembly binaries.

The PHP virtual machine (VM) is Zend Engine. To write an extension, one needs to develop in C or C++. The extension was simple C bindings to a Rust library I also wrote. At that time, this Rust library was using wasmi for the WebAssembly VM. I knew that wasmi wasn’t the fastest WebAssembly VM in the game, but the API is solid, well-tested, it compiles quickly, and is easy to hack. All the requirements to start a project!

After 6 hours of development, I got something working. I was able to run the following PHP program:


$instance = new Wasm\Instance('simple.wasm');
$result = $instance->sum(1, 2);

var_dump($result); // int(3)

The API is straightforward: create an instance (here of simple.wasm), then call functions on it (here sum with 1 and 2 as arguments). PHP values are transformed into WebAssembly values automatically. For the record, here is the simple.rs Rust program that is compiled to a WebAssembly binary:

pub extern fn sum(x: i32, y: i32) -> i32 {
    x + y

It was great! 6 hours is a relatively small number of hours to go that far according to me.

However, I quickly noticed that wasmi is… slow. One of the promise of WebAssembly is:

WebAssembly aims to execute at native speed by taking advantage of common hardware capabilities available on a wide range of platforms.

And clearly, my extension wasn’t fulfilling this promise. Let’s see a basic comparison with a benchmark.

I chose the n-body algorithm from the Computer Language Benchmarks Game from Debian, mostly because it’s relatively CPU intensive. Also, the algorithm has a simple interface: based on an integer, it returns a floating-point number; this API doesn’t involve any advanced instance memory API, which is perfect to test a proof-of-concept.

As a baseline, I’ve run the n-body algorithm written in Rust, let’s call it rust-baseline. The same algorithm has been written in PHP, let’s call it php. Finally, the algorithm has been compiled from Rust to WebAssembly, and executed with the php-ext-wasm extension, let’s call that case php+wasmi. All results are for nbody(5000000):

  • rust-baseline: 287ms,
  • php: 19,761ms,
  • php+wasmi: 67,622ms.

OK, so… php-ext-wasm with wasmi is 3.4 times slower than PHP itself, it is pointless to use WebAssembly in such conditions!

It confirms my first intuition though: In our case, wasmi is really great to mock something up, but it’s not fast enough for our expectations.

Faster, faster, faster…

I wanted to use Cranelift since the beginning. It’s a code generator, à la LLVM (excuse the brutal shortcut, the goal isn’t to explain what Cranelift is in details, but that’s a really awesome project!). To quote the project itself:

Cranelift is a low-level retargetable code generator. It translates a target-independent intermediate representation into executable machine code.

It basically means that the Cranelift API can be used to generate executable code.

It’s perfect! I can replace wasmi by Cranelift, and boom, profit. But… there is other ways to get even faster code execution — at the cost of a longer code compilation though.

For instance, LLVM can provide a very fast code execution, almost at native speed. Or we can generate assembly code dynamically. Well, there is multiple ways to achieve that. What if a project could provide a WebAssembly virtual machine with multiple backends?

Enter Wasmer

And it was at that specific time that I’ve been hired by Wasmer. To be totally honest, I was looking at Wasmer a few weeks before. It was a surprise and a great opportunity for me. Well, the universe really wants this rewrite from wasmi to Wasmer, right 😅?

Wasmer is organized as a set of Rust libraries (called crates). There is even a wasmer-runtime-c-api crate which is a C and a C++ API on top of the wasmer-runtime crate and the wasmer-runtime-core crate, i.e. it allows running the WebAssembly virtual machine as you want, with the backend of your choice: Cranelift, LLVM, or Dynasm (at the time of writing). That’s perfect, it removes my Rust library between the PHP extension and wasmi. Then php-ext-wasm is reduced to a PHP extension without any Rust code, everything goes to wasmer-runtime-c-api. That’s sad to remove Rust from this project, but it relies on more Rust code!

Counting the time to make some patches on wasmer-runtime-c-api, I’ve been able to migrate php-ext-wasm to Wasmer in 5 days.

By default, php-ext-wasm uses Wasmer with the Cranelift backend, it does a great balance between compilation and execution time. It is really good. Let’s run the benchmark, with the addition of php+wasmer(cranelift):

  • rust-baseline: 287ms,
  • php: 19,761ms,
  • php+wasmi: 67,622ms,
  • php+wasmer(cranelift): 2,365ms 🎉.

Finally, the PHP extension provides a faster execution than PHP itself! php+wasmer(cranelift) is 8.6 times faster than php to be exact. And it is 28.6 times faster than php+wasmi. Can we reach the native speed (represented by rust-baseline here)? It’s very likely with LLVM. That’s for another article. I’m super happy with Cranelift for the moment. (See our previous blog post to learn how we benchmark different backends in Wasmer, and other WebAssembly runtimes).

More Optimizations

Wasmer provides more features, like module caching. Those features are now included in the PHP extension. When booting the nbody.wasm file (19kb), it took 4.2ms. By booting, I mean: reading the WebAssembly binary from a file, parsing it, validating it, compiling it to executable code and a WebAssembly module structure.

PHP execution model is: starts, runs, dies. Memory is freed for each request. If one wants to use php-ext-wasm, you don’t really want to pay that “booting cost” every time.

Hopefully, wasmer-runtime-c-api now provides a module serialization API, which is integrated into the PHP extension itself. It saves the “booting cost”, but it adds a “deserialization cost”. That second cost is smaller, but still, we need to know it exists.

Hopefully again, Zend Engine has an API to get persistent in-memory data between PHP executions. php-ext-wasm supports that API to get persistent modules, et voilà.

Now it takes 4.2ms for the first boot of nbody.wasm and 0.005ms for all the next boots. It’s 840 times faster!


Wasmer is a young — but mature — framework to build WebAssembly runtimes on top of. The default backend is Cranelift, and it shows its promises: It brings a correct balance between compilation time and execution time.

wasmi has been a good companion to develop a Proof-Of-Concept. This library has its place in other usages though, like very short-living WebAssembly binaries (I’m thinking of Ethereum contracts that compile to WebAssembly for instance, which is one of the actual use cases). It’s important to understand that no runtime is better than another, it depends on the use case.

The next step is to stabilize php-ext-wasm to release a 1.0.0 version.

See you there!

If you want to follow the development, take a look at @wasmerio and @mnt_io on Twitter.

Bye bye Automattic, hello Wasmer

Today is my first day at Wasmer.

Wasmer’s logo. Build Once, Run Anywhere.
Universal Binaries powered by WebAssembly.

It’s with a lot of regrets that I leave Automattic. To be clear, I’m not leaving because something negative happened, I’m leaving because I’ve received the same job offer, 3 times in 10 days, from Wasmer, Google and Mozilla. Namely to work with Rust or C++ to build a WebAssembly runtime. This is an offer I can barely decline. It’s an opportunity and a dream for me. And I was lucky enough to get a choice between 3 excellent companies!

I can only encourage you to work with Automattic. It’s definitely the best company I’ve ever work with; stealing the 1st place to Mozilla. Automattic is not only about WordPress.com and other services: It’s a way of living. The culture, the spirit, the interactions between people, the mission, everything is exceptional. It has been a super great experience.

I could write 100 pages about my team. They have all been remarkable in many ways. I’m closer to them although they live at 10’000km, rather than colleagues I met everyday in person in the past. Congrats to Matt for this incredible project.

Now it’s time to work on Wasmer. It’s a WebAssembly runtime written in Rust: My two current passions. It’s powerful, modular, well-designed, and it comes with great ambitions. I’m really exciting. I work with an extraordinary team: Syrus Akbary (the author of Graphene, a GraphQL framework in Python), Lachlan Sneff (the author of Nebulet, a microkernel that implements a WebAssembly “usermode” that runs in Ring 0), Brandon Fish (a great contributor of Truffleruby, a high performance implementation of Ruby with GraalVM), Mackenzie Clark, and soon more.

My job will consist to work on the runtime of course, and also to integrate/embed the runtime into different languages, such as PHP —like I did with php-ext-wasm, more to come on this blog—. More secret projects coming. Let’s turn them into realities 🎉!

From Rust to beyond: The PHP galaxy

This blog post is part of a series explaining how to send Rust beyond earth, into many different galaxies. Rust has visited:
The galaxy we will explore today is the PHP galaxy. This post will explain what PHP is, how to compile any Rust program to C and then to a PHP native extension.

What is PHP, and why?

PHP is a:
popular general-purpose scripting language that is especially suited to Web development. Fast, flexible, and pragmatic, PHP powers everything from your blog to the most popular websites in the world.
PHP has sadly acquired a bad reputation along the years, but recent releases (since PHP 7.0 mostly) have introduced neat language features, and many cleanups, which are excessively ignored by haters. PHP is also a fast scripting language, and is very flexible. PHP now has declared types, traits, variadic arguments, closures (with explicit scopes!), generators, and a huge backward compatibility. The development of PHP is led by RFCs, which is an open and democratic process. The Gutenberg project is a new editor for WordPress. The latter is written in PHP. This is naturally that we want a native extension for PHP to parse the Gutenberg post format. PHP is a language with a specification. The most popular virtual machine is Zend Engine. Other virtual machines exist, like HHVM (but the PHP support has been dropped recently in favor of their own PHP fork, called Hack), Peachpie, or Tagua VM (under development). In this post, we will create an extension for Zend Engine. This virtual machine is written in C. Great, we have visited the C galaxy in the previous episode!

Rust 🚀 C 🚀 PHP

Rust to PHP To port our Rust parser into PHP, we first need to port it to C. It’s been done in the previous episode. Two files result from this port to C: libgutenberg_post_parser.a and gutenberg_post_parser.h, respectively a static library, and the header file.

Bootstrap with a skeleton

PHP comes with a script to create an extension skeleton/template, called ext_skel.php. This script is accessible from the source of the Zend Engine virtual machine (which we will refer to as php-src). One can invoke the script like this:
$ cd php-src/ext/
$ ./ext_skel.php \
      --ext gutenberg_post_parser \
      --author 'Ivan Enderlin' \
      --dir /path/to/extension \
$ cd /path/to/extension
$ ls gutenberg_post_parser
The ext_skel.php script recommends to go through the following steps:
  • Rebuild the configuration of the PHP source (run ./buildconf at the root of the php-src directory),
  • Reconfigure the build system to enable the extension, like ./configure --enable-gutenberg_post_parser,
  • Build with make,
  • Done.
But our extension is very likely to live outside the php-src tree. So we will use phpize instead. phpize is an executable that comes with php, php-cgi, phpdbg, php-config etc. It allows to compile extensions against an already compiled php binary, which is perfect in our case! We will use it like this :
$ cd /path/to/extension/gutenberg_post_parser

$ # Get the bin directory for PHP utilities.
$ PHP_PREFIX_BIN=$(php-config --prefix)/bin

$ # Clean (except if it is the first run).
$ $PHP_PREFIX_BIN/phpize --clean

$ # “phpize” the extension.
$ $PHP_PREFIX_BIN/phpize

$ # Configure the extension for a particular PHP version.
$ ./configure --with-php-config=$PHP_PREFIX_BIN/php-config

$ # Compile.
$ make install
In this post, we will not show all the edits we have done, but we will rather focus on the extension binding. All the sources can be found here. Shortly, here is the config.m4 file:
PHP_ARG_ENABLE(gutenberg_post_parser, whether to enable gutenberg_post_parser support,
[  --with-gutenberg_post_parser          Include gutenberg_post_parser support], no)

if  test "$PHP_GUTENBERG_POST_PARSER" != "no"; then


  PHP_NEW_EXTENSION(gutenberg_post_parser, gutenberg_post_parser.c, $ext_shared)
What it does is basically the following:
  • Register the --with-gutenberg_post_parser option in the build system, and
  • Declare the static library to compile with, and the source of the extension itself.
We must add the libgutenberg_post_parser.a and gutenberg_post_parser.h files in the same directory (a symlink is perfect), to get a structure such as:
$ ls gutenberg_post_parser
tests/                       # from ext_skel
.gitignore                   # from ext_skel
CREDITS                      # from ext_skel
config.m4                    # from ext_skel (edited)
gutenberg_post_parser.c      # from ext_skel (will be edited)
gutenberg_post_parser.h      # from Rust
libgutenberg_post_parser.a   # from Rust
php_gutenberg_post_parser.h  # from ext_skel
The core of the extension is the gutenberg_post_parser.c file. This file is responsible to create the module, and to bind our Rust code to PHP.

The module, aka the extension

As said, we will work in the gutenberg_post_parser.c file. First, let’s include everything we need:
#include "php.h"
#include "ext/standard/info.h"
#include "php_gutenberg_post_parser.h"
#include "gutenberg_post_parser.h"
The last line includes the gutenberg_post_parser.h file generated by Rust (more precisely, by cbindgen, if you don’t remember, take a look at the previous episode). Then, we have to decide what API we want to expose into PHP? As a reminder, the Rust parser produces an AST defined as:
pub enum Node<'a> {
    Block {
        name: (Input<'a>, Input<'a>),
        attributes: Option<Input<'a>>,
        children: Vec<Node<'a>>
The C variant of the AST is very similar (with more structures, but the idea is almost identical). So in PHP, the following structure has been selected:
class Gutenberg_Parser_Block {
    public string $namespace;
    public string $name;
    public string $attributes;
    public array $children;

class Gutenberg_Parser_Phrase {
    public string $content;

function gutenberg_post_parse(string $gutenberg_post): array;
The gutenberg_post_parse function will output an array of objects of kind Gutenberg_Parser_Block or Gutenberg_Parser_Phrase, i.e. our AST. So, let’s declare those classes!

Declare the classes

Note: The next 4 code blocks are not the core of the post, it is just code that needs to be written, you can skip it if you are not about to write a PHP extension.
zend_class_entry *gutenberg_parser_block_class_entry;
zend_class_entry *gutenberg_parser_phrase_class_entry;
zend_object_handlers gutenberg_parser_node_class_entry_handlers;

typedef struct _gutenberg_parser_node {
    zend_object zobj;
} gutenberg_parser_node;
A class entry represents a specific class type. A handler is associated to a class entry. The logic is somewhat complicated. If you need more details, I recommend to read the PHP Internals Book. Then, let’s create a function to instanciate those objects:
static zend_object *create_parser_node_object(zend_class_entry *class_entry)
    gutenberg_parser_node *gutenberg_parser_node_object;

    gutenberg_parser_node_object = ecalloc(1, sizeof(*gutenberg_parser_node_object) + zend_object_properties_size(class_entry));

    zend_object_std_init(&gutenberg_parser_node_object->zobj, class_entry);
    object_properties_init(&gutenberg_parser_node_object->zobj, class_entry);

    gutenberg_parser_node_object->zobj.handlers = &gutenberg_parser_node_class_entry_handlers;

    return &gutenberg_parser_node_object->zobj;
Then, let’s create a function to free those objects. It works in two steps: Destruct the object by calling its destructor (in the user-land), then free it for real (in the VM-land):
static void destroy_parser_node_object(zend_object *gutenberg_parser_node_object)

static void free_parser_node_object(zend_object *gutenberg_parser_node_object)
Then, let’s initialize the “module”, i.e. the extension. During the initialisation, we will create the classes in the user-land, declare their attributes etc.
    zend_class_entry class_entry;

    // Declare Gutenberg_Parser_Block.
    INIT_CLASS_ENTRY(class_entry, "Gutenberg_Parser_Block", NULL);
    gutenberg_parser_block_class_entry = zend_register_internal_class(&class_entry TSRMLS_CC);

    // Declare the create handler.
    gutenberg_parser_block_class_entry->create_object = create_parser_node_object;

    // The class is final.
    gutenberg_parser_block_class_entry->ce_flags |= ZEND_ACC_FINAL;

    // Declare the `namespace` public attribute,
    // with an empty string for the default value.
    zend_declare_property_string(gutenberg_parser_block_class_entry, "namespace", sizeof("namespace") - 1, "", ZEND_ACC_PUBLIC);

    // Declare the `name` public attribute,
    // with an empty string for the default value.
    zend_declare_property_string(gutenberg_parser_block_class_entry, "name", sizeof("name") - 1, "", ZEND_ACC_PUBLIC);

    // Declare the `attributes` public attribute,
    // with `NULL` for the default value.
    zend_declare_property_null(gutenberg_parser_block_class_entry, "attributes", sizeof("attributes") - 1, ZEND_ACC_PUBLIC);

    // Declare the `children` public attribute,
    // with `NULL` for the default value.
    zend_declare_property_null(gutenberg_parser_block_class_entry, "children", sizeof("children") - 1, ZEND_ACC_PUBLIC);

    // Declare the Gutenberg_Parser_Block.

    … skip …

    // Declare Gutenberg parser node object handlers.

    memcpy(&gutenberg_parser_node_class_entry_handlers, zend_get_std_object_handlers(), sizeof(gutenberg_parser_node_class_entry_handlers));

    gutenberg_parser_node_class_entry_handlers.offset = XtOffsetOf(gutenberg_parser_node, zobj);
    gutenberg_parser_node_class_entry_handlers.dtor_obj = destroy_parser_node_object;
    gutenberg_parser_node_class_entry_handlers.free_obj = free_parser_node_object;

    return SUCCESS;
If you are still reading, first: Thank you, and second: Congrats! Then, there is a PHP_RINIT_FUNCTION and a PHP_MINFO_FUNCTION functions that are already generated by the ext_skel.php script. Same for the module entry definition and other module configuration details.

The gutenberg_post_parse function

We will now focus on the gutenberg_post_parse PHP function. This function takes a string as a single argument  and returns either false if the parsing failed, or an array of objects of kind Gutenberg_Parser_Block or Gutenberg_Parser_Phrase otherwise. Let’s write it! Notice that it is declared with the PHP_FUNCTION macro.
    char *input;
    size_t input_len;

    // Read the input as a string.
    if (zend_parse_parameters(ZEND_NUM_ARGS() TSRMLS_CC, "s", &input, &input_len) == FAILURE) {
At this step, the argument has been declared and typed as a string ("s"). The string value is in input and the string length is in input_len. The next step is to parse the input. (The length of the string is not needed). This is where we are going to call our Rust code! Let’s do that:
    // Parse the input.
    Result parser_result = parse(input);

    // If parsing failed, then return false.
    if (parser_result.tag == Err) {

    // Else map the Rust AST into a PHP array.
    const Vector_Node nodes = parse_result.ok._0;
The Result type and the parse function come from Rust. If you don’t remember those types, please read the previous episode about the C galaxy. Zend Engine has a macro called RETURN_FALSE to return… false! Handy isn’t it? Finally, if everything went well, we get back a collection of node as a Vector_Node type. The next step is to map those Rust/C types into PHP types, i.e. an array of the Gutenberg classes. Let’s go:
    // Note: return_value is a “magic” variable that holds the value to be returned.
    // Allocate an array.
    array_init_size(return_value, nodes.length);

    // Map the Rust AST.
    into_php_objects(return_value, &nodes);
Done 😁! Oh wait… the into_php_objects function need to be written!

The into_php_objects function

This function is not terribly complex: It’s just full of Zend Engine specific API as expected. We are going to explain how to map a Block into a Gutenberg_Parser_Block object, and to let the Phrase mapping to Gutenberg_Parser_Phrase for the assiduous readers. And there we go:
void into_php_objects(zval *php_array, const Vector_Node *nodes)
    const uintptr_t number_of_nodes = nodes->length;

    if (number_of_nodes == 0) {

    // Iterate over all nodes.
    for (uintptr_t nth = 0; nth < number_of_nodes; ++nth) {
        const Node node = nodes->buffer[nth];

        if (node.tag == Block) {
            // Map Block into Gutenberg_Parser_Block.
        } else if (node.tag == Phrase) {
            // Map Phrase into Gutenberg_Parser_Phrase.
Now let’s map a block. The process is the following:
  1. Allocate PHP strings for the block namespace, and for the block name,
  2. Allocate an object,
  3. Set the block namespace and the block name to their respective object properties,
  4. Allocate a PHP string for the block attributes if any,
  5. Set the block attributes to its respective object property,
  6. If any children, initialise a new array, and call into_php_objects with the child nodes and the new array,
  7. Set the children to its respective object property,
  8. Finally, add the block object inside the array to be returned.
const Block_Body block = node.block;
zval php_block, php_block_namespace, php_block_name;

// 1. Prepare the PHP strings.
ZVAL_STRINGL(&php_block_namespace, block.namespace.pointer, block.namespace.length);
ZVAL_STRINGL(&php_block_name, block.name.pointer, block.name.length);
Do you remember that namespace, name and other similar data are of type Slice_c_char? It’s just a structure with a pointer and a length. The pointer points to the original input string, so that there is no copy (and this is the definition of a slice actually). Well, Zend Engine has a ZVAL_STRINGL macro that allows to create a string from a pointer and a length, great! Unfortunately for us, Zend Engine does a copy behind the scene… There is no way to keep the pointer and the length only, but it keeps the number of copies small. I think it is to take the full ownership of the data, which is required for the garbage collector.
// 2. Create the Gutenberg_Parser_Block object.
object_init_ex(&php_block, gutenberg_parser_block_class_entry);
The object has been instanciated with a class represented by the gutenberg_parser_block_class_entry.
// 3. Set the namespace and the name.
add_property_zval(&php_block, "namespace", &php_block_namespace);
add_property_zval(&php_block, "name", &php_block_name);

The zval_ptr_dtor adds 1 to the reference counter. This is required for the garbage collector.
// 4. Deal with block attributes if some.
if (block.attributes.tag == Some) {
    Slice_c_char attributes = block.attributes.some._0;
    zval php_block_attributes;

    ZVAL_STRINGL(&php_block_attributes, attributes.pointer, attributes.length);

    // 5. Set the attributes.
    add_property_zval(&php_block, "attributes", &php_block_attributes);

It is similar to what has been done for namespace and name. Now let’s continue with children.
// 6. Handle children.
const Vector_Node *children = (const Vector_Node*) (block.children);

if (children->length > 0) {
    zval php_children_array;

    array_init_size(&php_children_array, children->length);

    // Recursion.
    into_php_objects(&php_children_array, children);

    // 7. Set the children.
    add_property_zval(&php_block, "children", &php_children_array);


free((void*) children);
Finally, add the block instance into the array to be returned:
// 8. Insert the object in the collection.
add_next_index_zval(php_array, &php_block);
The entire code lands here.

PHP extension 🚀 PHP userland

Now the extension is written, we have to compile it. That’s the repetitive set of commands we have shown above with phpize. Once the extension is compiled, the generated gutenberg_post_parser.so file must be located in the extension directory. This directory can be found with the following command:
$ php-config --extension-dir
For instance, in my computer, the extension directory is /usr/local/Cellar/php/7.2.11/pecl/20170718. Then, to enable the extension for a given execution, you must write:
$ php -d extension=gutenberg_post_parser -m | \
      grep gutenberg_post_parser
Or, to enable the extension for all executions, locate the php.ini file with php --ini and edit it to add:
Done! Now, let’s use some reflection to check the extension is correctly loaded and handled by PHP:
$ php --re gutenberg_post_parser
Extension [ <persistent> extension #64 gutenberg_post_parser version 0.1.0 ] {

  - Functions {
    Function [ <internal:gutenberg_post_parser> function gutenberg_post_parse ] {

      - Parameters [1] {
        Parameter #0 [ <required> $gutenberg_post_as_string ]

  - Classes [2] {
    Class [ <internal:gutenberg_post_parser> final class Gutenberg_Parser_Block ] {

      - Constants [0] {

      - Static properties [0] {

      - Static methods [0] {

      - Properties [4] {
        Property [ <default> public $namespace ]
        Property [ <default> public $name ]
        Property [ <default> public $attributes ]
        Property [ <default> public $children ]

      - Methods [0] {

    Class [ <internal:gutenberg_post_parser> final class Gutenberg_Parser_Phrase ] {

      - Constants [0] {

      - Static properties [0] {

      - Static methods [0] {

      - Properties [1] {
        Property [ <default> public $content ]

      - Methods [0] {
Everything looks good: There is one function and two classes that are defined as expected. Now, let’s write some PHP code for the first time in this blog post!

        '<!-- wp:foo /-->bar<!-- wp:baz -->qux<!-- /wp:baz -->'

 * Will output:
 *     array(3) {
 *       [0]=>
 *       object(Gutenberg_Parser_Block)#1 (4) {
 *         ["namespace"]=>
 *         string(4) "core"
 *         ["name"]=>
 *         string(3) "foo"
 *         ["attributes"]=>
 *         NULL
 *         ["children"]=>
 *         NULL
 *       }
 *       [1]=>
 *       object(Gutenberg_Parser_Phrase)#2 (1) {
 *         ["content"]=>
 *         string(3) "bar"
 *       }
 *       [2]=>
 *       object(Gutenberg_Parser_Block)#3 (4) {
 *         ["namespace"]=>
 *         string(4) "core"
 *         ["name"]=>
 *         string(3) "baz"
 *         ["attributes"]=>
 *         NULL
 *         ["children"]=>
 *         array(1) {
 *           [0]=>
 *           object(Gutenberg_Parser_Phrase)#4 (1) {
 *             ["content"]=>
 *             string(3) "qux"
 *           }
 *         }
 *       }
 *     }
It works very well!


The journey is:
  • A string written in PHP,
  • Allocated by the Zend Engine from the Gutenberg extension,
  • Passed to Rust through FFI (static library + header),
  • Back to Zend Engine in the Gutenberg extension,
  • To generate PHP objects,
  • That are read by PHP.
Rust fits really everywhere! We have seen in details how to write a real world parser in Rust, how to bind it to C and compile it to a static library in addition to C headers, how to create a PHP extension exposing one function and two objects, how to integrate the C binding into PHP, and how to use this extension in PHP. As a reminder, the C binding is about 150 lines of code. The PHP extension is about 300 lines of code, but substracting “decorations” (the boilerplate to declare and manage the extension) that are automatically generated, the PHP extension reduces to about 200 lines of code. Once again, I find this is a small surface of code to review considering the fact that the parser is still written in Rust, and modifying the parser will not impact the bindings (except if the AST is updated obviously)! PHP is a language with a garbage collector. It explains why all strings are copied, so that they are owned by PHP itself. However, the fact that Rust does not copy any data saves memory allocations and deallocations, which is the biggest cost most of the time. Rust also provides safety. This property can be questionned considering the number of binding we are going through: Rust to C to PHP: Does it still hold? From the Rust perspective, yes, but everything that happens inside C or PHP must be considered unsafe. A special care must be put in the C binding to handle all situations. Is it still fast? Well, let’s benchmark. I would like to remind that the first goal of this experiment was to tackle the bad performance of the original PEG.js parser. On the JavaScript ground, WASM and ASM.js have shown to be very much faster (see the WebAssembly galaxy, and the ASM.js galaxy). For PHP, phpegjs is used: It reads the grammar written for PEG.js and compiles it to PHP. Let’s see how they compare:
file PEG PHP parser (ms) Rust parser as a PHP extension (ms) speedup
demo-post.html 30.409 0.0012 × 25341
shortcode-shortcomings.html 76.39 0.096 × 796
redesigning-chrome-desktop.html 225.824 0.399 × 566
web-at-maximum-fps.html 173.495 0.275 × 631
early-adopting-the-future.html 280.433 0.298 × 941
pygmalian-raw-html.html 377.392 0.052 × 7258
moby-dick-parsed.html 5,437.630 5.037 × 1080
The PHP extension of the Rust parser is in average 5230 times faster than the actual PEG PHP implementation. The median of the speedup is 941. Another huge issue was that the PEG parser was not able to handle many Gutenberg documents because of a memory limit. Of course, it is possible to grow the size of the memory, but it is not ideal. With the Rust parser as a PHP extension, memory stays constant and close to the size of the parsed document. I reckon we can optimise the extension further to generate an iterator instead of an array. This is something I want to explore and analyse the impact on the performance. The PHP Internals Book has a chapter about Iterators. We will see in the next episodes of this series that Rust can reach a lot of galaxies, and the more it travels, the more it gets interesting. Thanks for reading!