TL;DR

Intermodal is a new command-line BitTorrent metainfo¹ utility for Linux, Windows, and macOS. The binary is called imdl.

It can create, display, and verify .torrent files, as well as generate magnet links.

It has lots of features and niceties, is easy to install and run, and is hopefully just the beginning of an ambitious project to make decentralized content sharing better.

Features include:

• Attractive progress bars that display hashing throughput.
• Automatic piece length picker that uses the size of the torrent to pick a good piece length.
• Ignores junk files, like .DS_Store, by default.
• Detailed error messages.
• Warnings for common issues, like non power-of-two piece lengths.
• Support for all commonly used metadata fields, like info.source and info.private.
• File inclusion and exclusion with --glob PATTERN and --glob !PATTERN.
• Torrent verification with imdl torrent verify.
• Torrent display with imdl torrent show.

You can install the latest version of imdl to ~/bin with:

curl --proto '=https' --tlsv1.2 -sSf https://imdl.io/install.sh | bash

Development is hosted on GitHub, where you can find the code, the issue tracker, and more installation options.

Give it a try and let me know what you think!

I'm eager to hear what works, what doesn't, and what features you'd like to see added. I'll be working on novel functionality—more on that below—and I'd love to hear your critical feedback and ideas.

You can get in touch by open an issue, joining the discord server, or sending me an email.

Happy sharing!

¹ A BitTorrent .torrent file, also known as a metainfo file, contains information that allows peers to locate and retrieve the contents of a torrent. Creating and publishing this metainfo is required to make the contents of a torrent available to peers.

Mentioning BitTorrent always carries the risk of prompting discussions of the unauthorized sharing of copyrighted or censored content. Intermodal is not intended to facilitate such unauthorized sharing. Discussion of unauthorized sharing is not permitted in any project space, including GitHub and Discord.

What happened to your personal library?

Rows of LPs, shelves of VHS tapes, binders of CDs and DVDs, or maybe hard drives stuffed to overflowing.

It used to be common to maintain a personal content library, but in the last decade, most of us stopped. We ditched the tedium of tagging and sorting for Netflix, Spotify, and YouTube.

Whenever I think about this I feel a wistful longing for the past. And I don't think it's just nostalgia.

Modern content channels are usually designed around a feed and automatic recommendations, and don't make a strong distinction between things that are in your library and everything else. This makes it too easy to consume content compulsively and replaces active search and curation with guzzling down what's on tap, biasing you away from what you like and towards what everyone else likes.

Beyond that, once-accessible items often disappear due to obscure legal and contractual machinations out of one's control.

Worse still, content that is below an invisible popularity threshold is hard to find, not available, or doesn't appear in recommendation streams. One of my favorite tracks from my old music library was a song from a collection of music by people on Usenet called "It's Six AM and Gary's Refrigerator Plays a Raga". Definitely not available on Spotify.

And, if you make something yourself, you can't just put it into your own library, or send it to a friend so they can throw it in theirs.

This is profoundly saddening.

Why did we stop curating our own libraries?

Centralized apps have replaced decentralized content networks due to ease of use, user experience, and incentives.

These factors have pushed people away from BitTorrent, store-bought CDs mixtapes, and personal libraries, and towards apps like Spotify and Netflix.

Ease of Use

For all their flaws, centralized apps are incredibly easy to use. Just open a browser tab and press play.

To get content into your own library and play it, you need to use a number of applications in concert: a web browser to search, another app to download, a file browser to organize, and finally a player to listen or watch. A chore, even for technically adept users, and impossible for the non-technical majority.

Features

Centralized apps maintain large, proprietary databases of metadata, so features like search, recommendation, and preview work flawlessly.

Most desirable features are only as good as the metadata they're built on, so if you're curating a personal library, you have to curate the metadata along with it.

This is the primary reason I gave up on my personal music library: I spent endless amounts of time making sure that metadata was uniform and correct, and although I found some great tools to help, it still felt like a part-time job.

Incentives

Centralized apps usually have built-in monetization mechanisms. If you like content you find through decentralized channels and want to support the creator or publisher, there is no easy means to do so.

Because of this, first-party releases on decentralized content networks are rare, and resources are instead lavished on centralized apps.

Where do we go from here?

The feature-gap between centralized apps and decentralized content networks is vast.

Fortunately, there is a simple way to close that gap:

Developing standards for structured, machine-readable metadata in a simple, universal format.

Existing content does contain metadata. However, this metadata is limited to specific content types, for example MP3 tags; or modes of transport, for example BitTorrent.

By developing a standard for metadata manifests that can accompany any kind of content, across all modes of transportation, many desirable features become more reliable and dramatically easier to implement:

• Newly downloaded releases can be integrated into a library automatically, applying the user's preferred scheme for naming, sorting, and tagging.

• Content from a user's library can easily be exported for sharing in a format that's appropriate for transport, for example as a torrent, Usenet NZB file, or .zip archive.

• By identifying the location and format of content, releases can be transcoded for mobile devices and web browsers, either in advance or on the fly, and made available to all a user's devices without manual intervention.

• Rich search indices can be built from collections of content, without needing to resort to complex and error-prone heuristics.

• Immutable identifiers, for example ISBN numbers, allow metadata to be automatically updated from external sources.

• Creators and publishers can include a Bitcoin tipping address in releases, so end users can reward them directly.

• Digital signatures over and alongside metadata manifests can allow authenticity of releases to be verified, allow publishers to sign new updates to old releases, and form the basis of a decentralized, privacy-preserving reputation system.

These features have the potential to not just bring the decentralized content ecosystem to parity with centralized services, but to surpass them.

Intermodal

The project for developing these metadata standards, as well as tools and apps to make these standards useful, is called "Intermodal".

Seamless intermodal transportation, enabled by containerization, has led to enormous efficiency gains in the transport of physical goods.

Before the invention of the humble 40' shipping container, and the intermodal transportation network that it enabled, the majority of the world's goods were transported as so-called bulk break cargo.

Bulk break cargo is packed in bags, barrels, boxes, crates, and drums of varying sizes. Loading such cargo onto a truck, cargo ship, or train took labor and time, and was a major source of friction and cost in the shipping of goods from place to place.

The invention and standardization of intermodal containers, the 20' and 40' shipping containers of today, changed all that. Cargo could now be packed into uniformly sized containers of known strength and weight, of a size suitable for transportation by train, truck, or ship. This standardization made the once back-breaking work of moving cargo through multiple transportation modes easy and fast. What once required teams of stout men could now be done with cranes and other equipment.

In many ways, when it comes to decentralized digital content, we are very much living in an era of bulk break cargo. Painstaking effort is required to prepare content for transportation across different modes, e.g. BitTorrent, the web, or Usenet; and it is either impossible or takes complex heuristics to answer simple questions, like what is a piece of content, anyways?

By standardizing metadata, we can make more efficient the conveyance of digital content across different modes of transportation, and build rich services on top of that content.

Thus, "Intermodal".

imdl

I've been thinking about how to move the decentralized content ecosystem forward for a long time, and until recently I've been stuck on the question of what, exactly, to build first.

I've decided to start with a BitTorrent metainfo (that is to say, .torrent file) creator. The first version is full of features and niceties, and is ready for users today.

The binary is called imdl, and development is hosted on GitHub.

BitTorrent is widely used, and a torrent creator is much simpler than a torrent client, tracker, or index, making it a good place to start.

Although imdl does not today have any groundbreaking new features, and no functionality for creating metadata manifests, it is a natural place to start adding such features.

imdl is written in Rust. Rust is fast, correct, and makes it easy to distribute self-contained binaries to users. Additionally, Rust can be compiled to WebAssembly, so bits of imdl might eventually be adapted to run in the browser.

Over time, imdl will be extended with all manner of useful features, so torrent-creation functionality lives under the torrent subcommand: imdl torrent create, imdl torrent verify, imdl torrent show, and so on.

I owe a huge debt of gratitude to the many existing open-source torrent creators, which have been a font of inspiration and good ideas. As part of the process of developing this first release of imdl, I combed through them for useful and requested features, and implemented all of them.

What's next?

First and foremost, I want imdl to be a useful torrent creator today. If you find any bugs, or have feature requests, don't hesitate to open an issue or hop on the discord!

Contributions of code, documentation, clean up, tests, ideas, and discussion are all welcome!

Many features, such as integrity checking, signing, timestamping, and release metadata are gated on the basic design and implementation of the manifest, so that will be the next major area of work.

Your feedback on the design and implementation of the manifest would be very valuable.

Additionally, there are a bunch of more concrete good first issues on GitHub, some large, some small:

• Setting up code coverage on CI.
• Adding web seeds to torrents..
• Adding another kind of web seeds to torrents.
• Supporting the addition of arbitrary keys to created torrents.
• Adding a config file containing profiles to use for torrent file creation.
• Adding support for generating file padding.
• Adding a whole new subcommand to edit existing .torrent files.
• Fixing a no-doubt silly bug causing tests to leave behind .torrent files in /tmp.
• Adding file selections to magnet links.
• Creating .torrent files from magnet links.
• Verifying multiple .torrent files at a time.
• Showing nonstandard fields in imdl torrent show.
• Adding a --quiet flag to imdl torrent create.
• Showing corrupted piece information during verification.
• Supporting BitTorrent V2 torrents.

And finally…

Thank you so much for reading this rather long-winded blog post!

imdl, modest though it may be, is a love letter to the Internet, to sharing, and to BitTorrent. I hope you find it useful, and it becomes even more useful in the future.