One of the things that I personally struggled with when learning Rust was how to organize large programs with multiple modules.

In this post, I'll explain how I organize the codebase of just, a command runner that I wrote.

just was the first large program I wrote in Rust, and its organization has gone through many iterations, as I discovered what worked for me and what didn't.

There are some things that could use improvement, and many of the choices I made are somewhat strange, so definitely don't consider the whole project a normative example of how to write rust.

Overview

Users mostly interact with just by running the main binary from the command line.

However, the crate actually consists of an executable target, in src/main.rs, and a library in src/lib.rs. The main function in main.rs is a thin wrapper that calls the run function in src/run.rs.

The reason that just is split into a executable target and a library target is because there is a fuzz tester in fuzz, and a regression testing framework at janus, and both of these use testing functions exposed by the library.

Submodule Organization

I prefer to keep my module tree flat, so you'll notice that all my source files are directly under src. I find that this makes it easy to remember what's where, since I don't need to remember what subdirectory each source file is in.

I use a fuzzy file searcher, fzf, to switch between files, so having a ton of files in my src directory doesn't bother me. If I used a tree-based file viewer in my editor, I might prefer to group modules into directories by topic.

Common Use Statements

I prefer to group all my use statements together in a single file called src/common.rs.

Then, at the top of every other file, I include them all with use crate::common::*;.

I think this is somewhat uncommon, and most people prefer to put use statements at the top of every file, with just those things used in that particular file.

Both approaches are totally reasonable. I find that grouping use statements into a single file saves a lot of duplication at the top of every file, and makes it easy to start a new file, since you can just write use crate::common::*; and have everything you need in scope.

This does require that I pick unique names for everything that I want to put in common.rs, but I haven't found that to be particularly burdensome.

Submodule Names and Contents

Most modules contain a single definition, either a function, trait, struct, or enum, that is used in the rest of the codebase. These modules are all named after the public definition, and that definition is exported in common.rs, so use crate::common::*; will bring that definition into scope, without needing to qualify it with the module name.

As an example, just's lexer is called Lexer, and is in src/lexer.rs.

In common.rs, it is exported with pub(crate) use lexer::Lexer;.

Since modules are always named after their sole export, it's pretty easy to figure out where something comes just from the name. Since Lexer is a type from just, and not a dependency, that means that it's probably defined in lexer.rs.

A few modules, like src/keyword.rs, contain more than one thing. For modules like that, common.rs just exports the module itself, with pub(crate) use crate::keyword;, and the module name is used when referring to the definitions inside keyword, like keyword::EXPORT.

If a name is from a dependency, then you can see where it comes from in common.rs.

Error Handling

For testing purposes, I like to use error enums, instead of Box<dyn Error, or equivalent. I find that this makes it easy to write tests that look for specific errors. Also, just has detailed error messages, and this lets me separate error message formatting from error value generation.

There are two main kinds of errors, CompilationError, in src/compilation_error.rs, and RuntimeError, in runtime_error.rs.

As you can guess, CompilationError is for problems related to compilation, e.g. lexing, parsing, and analyzing a justfile, and RuntimeError is for problems that occur when running a justfile, e.g. I/O errors and command execution errors.

I currently don't use any of the many error handling helper crates, but in other projects I use snafu, and if I were to rewrite just, I would definitely use snafu.

Clippy

I use clippy, the animate paperclip/rust linter, to automatically check the codebase for issues.

Clippy has many lints that are either pedantic, or which restrict things that are totally reasonable in many contexts. I like a lot of these, but I didn't want to go through all of them and decide which lints to enable, so I turned on all the lints, even the annoying ones, and then just disable lints that I don't like as I encounter them.

You can see this at the top of src/lib.rs.

How does it work?

just does a lot of stuff! This makes it hard to give a concise overview of how everything works, but I'll do my best!

Run

The run function is pretty short, so definitely check it out. It's in src/run.rs. It does some setup, like initializing Windows terminal color support and logging, then parses the command line arguments.

Configuration

Just calls parsed command line arguments a Config. The command line arguments are parsed with the venerable clap, and then stored in a Config struct, which is passed around the rest of the program.

Everything related to setting up the clap parser, and parsing the command line arguments is in src/config.rs.

Subcommand Running

just has a few distinct modes it can run in, e.g. actually running a recipe in a justfile, listing the recipes in a justfile, or evaluating the variables in a justfile. These are called subcommands, and you can see the different subcommands in the Subcommand enum in src/subcommand.rs.

Once a config is parsed, the function run_subcommand in config.rs handles executing the correct subcommand.

For the rest of this post, I'll cover Subcommand::Run, which is the subcommand which is responsible for actually running a justfile.

Compilation

The justfile source is read and the compiler is invoked in the run_subcommand function in config.rs. The Compiler is defined in src/compiler.rs, and has a single short method that calls the lexer, the parser, and the analyzer.

There's no particular reason for having a Compiler struct, since it doesn't have any fields, so it's really just for organization. I would be totally fine with having a module src/compile.rs, and just exporting a single compile function from that module.

Lexing

The first step of compilation is to split the source text into tokens, which is done by the Lexer in src/lexer.rs. The lexer looks a lot like a recursive descent parser. It has a bunch of different methods, and those methods call each other to produce the different tokens.

The entry-point to the lexer is Lexer::lex.

Lexer is relatively well-commented, so please take a look if you're interested!

Tokens

The lexer produces a Vec of Tokens. The Token type is in src/token.rs. Each Token contains a TokenKind, defined in src/token_kind.rs. A Token contains a reference to the soruce program, as well as information about the offset, length, line, and column of the token. A TokenKind tells you what kind of token it actually is.

Parsing

The Tokens produced by the Lexer are passed to a Parser, defined in src/parser.rs, and the main entry point is Parser::parse.

The parser is a recursive descent parser that walks over the tokens, figuring out what kind of construct it's parsing as it goes.

Modules

The output of the parser is a Module, defined in src/module.rs. A Module represents a successful parse, but has not been fully validated. Just does a lot of static analysis, like resolving names, inter-recipe dependencies, and inter-variable dependencies, so not every Module is valid.

You can think of a Module as being like an AST, that still hasn't been statically analyzed for correctness. Inside a Module are Items (src/item.rs), which contain the different source constructs, like Alias, Assignment, UnresolvedRecipe, and Set.

Analysis

The next phase of compilation is analysis, performed by the Analyzer, defined in src/analyzer.rs.

The Analyzer makes sure that all references to recipes and variables can be resolved, and that there are no circular dependencies.

Justfile

The output of the Analyzer is a Justfile, defined in src/justfile.rs.

A Justfile represents a parsed and analyzed justfile. It contains all the recipes, variables, and expressions, all resolved and ready to run. It is the totus porcus, as it were.

Running

A justfile is run with Justfile::run, which takes a Config, a Search with information about where the justfile is and where the working directory is, variable overrides passed on the command line, and a list of arguments.

The arguments are parsed into recipes and arguments to those recipes, and finally those recipes are run with Justfile::run_recipe, which actually executes each recipe, and all dependencies.

Testing

just takes files, parses commands out of those files, and then runs them. I am acutely aware of how this might go wrong, and would feel real bad if just somehow got confused and ran a command that nuked someone's hard drive.

Because of this, I go pretty crazy with testing. There are four kinds of tests: unit tests, integration tests, fuzz testing, and ecosystem-wide regression testing.

Unit Testing

Unit tests are spread around the codebase, in submodules named tests. Each tests submodule contains tests for whatever's in the containing module.

I'm not strict about covering everything with unit tests, but I do cover everything with integration tests. If something doesn't seem to be tested in a unit test, it's probably tested in an integration test.

Integration Testing

Integration tests are in the tests subdirectory, roughly organized by topic. The vast majority are in tests/integration.rs, which test a full run of the just binary, supplying standard input, args, and a justfile, and checking that standard output, standard error, and the exit code are correct.

Fuzz Testing

Fuzz testing was contributed to just by @RadicalZephyr, and is located in the fuzz directory. It generates random strings and feeds them to the parser. (NOT THE RUNNER, DEFINITELY NOT THE RUNNER.) If the parser succeeds or returns an error, that's a successful run. If the fuzzer is able to trigger a panic, then it's found a bug that needs to be fixed.

Regression Testing

Since a lot of people have written a lot of justfiles, I want to make sure I don't break them when I update just.

To do this, I wrote a tool called janus. Janus is inspired by Rust's crater.

Janus downloads all the justfiles that it can find on GitHub, and then compares how two versions of just compiles those justfiles. The two versions of just are usually the latest release, and a new version with a big, scary change.

Janus compiles all justfiles with both versions, and then compares the result. Ideally, every valid justfiles parses into the same Justfile with both versions, and every justfile with an error produces the same error.

Wrapping Up

That's everything I can think of! Ultimately, much of how you organize your Rust programs comes down to personal preference, so just start mashing the keyboard, see what works, and iterate on whatever doesn't.

glhf!