TL;DR
I am pleased to announce the release of filepack 0.0.12.
Filepack is a content-addressed package format, metadata standard, command-line utility, and package server. It is motivated by a desire to improve the user experience of decentralized filesharing by providing a data model that can serve as a lingua franca for sharing and media applications.
The filepack binary is written in Rust and can be used to create, sign, and
verify packages, and includes a package server with an attractive front end, as
well as commands to upload and download packages.
A demo server with a nice collection of packages is available at filepack.com.
It is already featureful and useful, but the package and metadata formats are not stable, and there is as yet no guarantee of backwards compatibility.
Lofty Ambitions
The goal of filepack is to make decentralized filesharing as appealing and easy to use as centralized services. When you share audio, images, and video they should look good and be easy to search and play.
Existing filesharing technologies are hard to use because they transfer unstructured data: individual files or directories of files. With no structure or metadata, many desirable features are hard or impossible to build. For example, structured search, graphical user interfaces with package artwork, and automatically locating, playing, and transcoding media.
Filepack packages include structured, machine-readable metadata describing the package and its contents. This allows them to be presented in an attractive way, supports structured search, and for any media in the package to be automatically previewed and played.
All metadata displayed by the filepack.com demo server is served directly from the metadata included in each package. This includes package titles, artwork, image thumbnails, and video placeholder images. Filepack metadata also identifies the media type of the package and each individual media item, including its file format, and details like resolution, codec, and duration.
Inscrutable Details
Filepack packages are content-addressed Merkle trees, identified by a root hash called the package fingerprint. The inner hashes of the Merkle tree are BLAKE3 hashes, itself a tree hash, allowing for incremental transfer and verification. Packages may be signed using Ed25519.
Packages include both content and metadata, so a package can be downloaded with
filepack download --package FINGERPRINT and uploaded to a new server with
filepack upload --server URL.
Because filepack packages are normal directories of files, they can be shared with other filesharing technologies, like BitTorrent, Usenet, and ordinary web servers.
Metadata is optional, so filepack can also be used as a general purpose directory hashing, signing, and verification tool.
Filepack includes a web package type, which can served by the filepack server and have access to the server API. This allows creating and distributing web apps that can access and display the packages on a server.
Exhortations
I've been slaving away for months on this thing, so I would absolutely love it, and you, if you gave filepack a whirl.
You can download the latest release from
GitHub or build it from source
with cargo install filepack.
Try creating a package and running the server, for yourself or others.
Filepack only supports a limited number of media types and file formats, but I'm very happy to add more if there's demand.
Please feel free to report issues or request new features on GitHub, or just email me.
Future Directions
In its current state, quite a bit of imagination is required to see how Filepack could improve decentralized filesharing. My hope is to start reducing the amount of imagination required by adding features that start to bring some of the potential benefits of a standardized package and metadata format to life. This includes work in a few broad areas.
I plan on reaching out to archival, datahoarding, and filesharing communities, figuring out how they are underserved, and adding featueres to Filepack that make it useful to them.
I want to add support for new media types, like PDFs, ebooks, and ROMs. I think that support for each new type of content helps elucidate the future in which your entire media library is composed of Filepack packages of different kinds, with apps that take advantage of the standardized package and metadata format providing a seamless experience on top.
In addition to support for new media types, I want to add subtypes, which optimize the experience for specific categories of media within an existing type. A good example is webtoons, which fit into the existing image media type, but which should be displayed in a vertical scrolling format. Subtypes demonstrate that multiple disparate forms of media can be accomodated by the same software, without having to guess how something should be displayed, or shoehorn different kinds of content into a one-size-fits-all viewer.
One of the most potentially powerful ideas in Filepack is the web package. It enables web content to be packaged and shared, and when mounted, web packages have access to the Filepack server's API. I want to demonstrate this with a single-page media player app which exposes the content of a Filepack server in a interface that is competative with existing streaming services and media players.
There are also a large number of sharing features that I'd like to explore. Filepack packaging and metadata places a burdon on the packager, and that burdon only really starts to pay off when the benefits of packaging can be shared. I plan on adding link strings, which point to a package on a particular Filepack server, to allow a package to be downloaded by copying and pasting a single string.
And eventually, I'd like to create a filepack-native peer-to-peer filesharing network, with a rich client that's much more like a media player than a filesharing application. But that's a long way off.
Closing Remarks
Filepack is something that I've been thinking about and working on, in one form or another, for quite a while. I'm very happy to finally be able to release something that at least starts to articulate the vision of the project. There's still a long way to go, but hopefully it is useful in its current form and someone out there appreciates the ideas that it embodies.
