Maintaining rust-libp2p & beyond

The beginnings

As part of my previous job, I was fortunate enough to get involved with the Rust community, in particular peer-to-peer networking. We were heavy users of rust-libp2p, a library which was originally developed by Parity to power Polkadot. On some day, more than three years ago now, Max - who had just moved from Parity to Protocol Labs to maintain rust-libp2p - reached out to me to discuss and learn how we were using rust-libp2p in our projects. The discussion was very fruitful and inspired me to learn more about the inner workings of rust-libp2p. Until then, most of my contributions were centered around surface-level improvements. As I started to understand things better, I noticed that some of the ideas I wanted to implement required deeper changes. Eventually, this led to me stepping up as a maintainer in late 2021.

Life as a maintainer

At the time, we already had several, independent projects using rust-libp2p and I saw an opportunity to encourage the number and size of contributions. I am a big proponent of focusing on improving development velocity:

• Optimizing for speed of change allows you to iterate faster.
• Faster iteration times lead to a build-up of momentum.
• Seeing progress motivates people which leads to more involvement.

Walking this chain backwards, the things I focused on initially were:

• Faster and more exhaustive CI (#3090 with many follow-ups over the months)
• Reducing compile times (for example, by redesigning the cargo features: #2173)
• Removing footguns (e.g. #3068)

In addition to this, there is another category of "infrastructure" work that I wanted to tackle: Removing accidental complexity. Every project or task has essential complexity. That is the one you can't remove without altering the scope. Accidental complexity on the other hand can be removed. It is the kind of code you look at and conclude "we don't actually need to do this". Keeping your code lean and simple is an investment in the future. It - again - ends up improving development velocity. Why? Because once you need to change a certain piece of code, knowing that what you are looking at are the essential requirements helps tremendously in making a safe change. Safe meaning without introducing (new) bugs and breaking changes. Especially the latter is something you want to be thinking about when maintaining a library but really extends to any kind of product.

Getting sponsored

About a year into my role as (part-time) maintainer, an opportunity presented itself for Protocol Labs (the company behind IPFS from which libp2p initially emerged) to sponsor my maintenance work. This was great! It meant that I could dedicate myself full-time to the project without having to either burn through savings or work another job next to it.

Empowering contributions

Let's face it. There is only so much you can get done by yourself. To really scale up the amount of work that happens, you need to empower others. But how?

In many ways, rust-libp2p is quite a complex project. A notable contributor to this complexity is the by-design modularity. Users should be able to combine an arbitrary number of transports (e.g. TCP, QUIC, WebRTC) with however many plugins¹ (e.g. connection management, Gossip protocol, DHT). The transports themselves are composable: On top of TCP, you should be able to layer your own multiplexer and encryption protocol. At the same time, we wanted to follow the Rust philosophy of "only pay for what you use". The combination of these requirements means you end up with many, heavily feature-flagged crates. At the time of writing, the entire workspace has 64! Navigating such a big workspace can be very daunting for users and contributors. To still facilitate new contributions, we ended up writing many issues of what would be simple tasks for us but are a good starting point for people that want to learn and contribute. It also meant that Max and myself could focus on more difficult problems that required more knowledge of the internals. Eventually, you get the best of both worlds: New people can get involved by tackling simpler issues whereas you as a maintainer can focus your time on more difficult problems. The only downside is that there is no timeline on when those simpler tasks will be resolved. One of the very first ones that I opened is still not entirely resolved :)

I don't have much data to point at but my experience has been that this worked very well. In fact, it worked so well that more than 50% of my time spent on rust-libp2p is just on code reviews, maintenance tasks and user support like responding to issues or discussing features. On one hand, this was great: continuous work meant a reasonably stable income as I was billing by the hour. On the other hand, continuous work also means it is more difficult to fully take time off. Whilst the work itself is very flexible, it does end up being on your mind a lot.

The future

People's priorities shift as time goes on and as such, I want to announce that my current sponsorship for maintaining rust-libp2p with Protocol Labs is set to conclude at the end of January 2024 as part of the changes in governance². For me, this is a good opportunity to stop and move on to other things. Most of my motivation for working on rust-libp2p came from my early days as a user. I wanted to make it easier and safer to use and I think we have succeeded on that front! At the same time, I've not been an active user for over 2 years. There are still many things I'd want to do or change in rust-libp2p but without being a user myself, it is somewhat difficult to judge which ones are the most important. I see the changes in governance as an opportunity for other community members to step up and drive the project based on their needs. At the same time, I think that there is a lot of value in having a sponsored, project-independent maintainer that coordinates feature requests and drives overall improvement efforts. This is essentially the hat I am passing on so if this sounds interesting to you, let us know!

I really enjoyed working on rust-libp2p and the libp2p project in general. Together with Max and countless other contributors, we solved many problems, big and small. I learned a lot over the last years, ranging from the details of network protocols, to the effective use of Rust in large workspaces and lots of open-source community work. A big thank you to everyone that was involved!