Just, Nix Shell and Podman are a Killer Combo

Let’s say, for some unclear reasons, you need to compile the “Hello World” C program using a variety of C compilers.

#include <stdio.h>

int main() {
    printf("Hello World");
    return 0;
}
hello.c

Let’s also say that your dev machine is a MacBook, and some of these C compilers run only on Linux.

Let’s start with Clang. Clang is the default C compiler on macOS, and you probably have it already installed. Let that be the case, and you compile hello.c by running:

clang -o hello-clang hello.c

Well and good. But you don’t want to be typing all that every time you need to compile (your unusual circumstances compell you to compile the file again and again). So you put it in a justfile:

build-clang:
    clang -o hello-clang hello.c
justfile

Great! Now you use Just, a modern command runner, to run Clang, like so:

just build-clang

Just when you think you troubles are over, you remember that you need to compile the file using GCC as well. macOS does not come with GCC installed, and now you need to figure out how to install it and its dependencies. You can Homebrew that stuff, but you know better.

Enter Nix. Nix is a lot of things, but for the purpose of this post, it is a way to easily create reproducible development environments. So after installing Nix, you quickly put together the shell.nix file that gathers your dependencies, and makes them present your shell’s $PATH:

with (import <nixpkgs> { });
mkShell {
  buildInputs = [ just gcc ];
}
shell.nix

Next, you expand the justfile so that it compiles hello.c with GCC:

in_nix_shell := env_var_or_default("IN_NIX_SHELL", "false")
root_dir := justfile_directory()

_run-in-nix-shell cmd *args:
    #!/usr/bin/env -S sh -eu
    if [ "{{ in_nix_shell }}" = "false" ]; then
        nix-shell "shell.nix" --run "just \"{{ root_dir }}/{{ cmd }}\" {{ args }}"
    else
        just "{{ root_dir }}/{{ cmd }}" {{ args }}
    fi

_build-gcc:
    gcc -o hello-gcc hello.c

build-gcc: (_run-in-nix-shell "_build-gcc")
justfile

With this setup, you compile hello.c with GCC by running:

just build-gcc

The _run-in-nix-shell Just command takes care of automatically starting the nix-shell if required. nix-shell downloads GCC and its dependencies for you, and sets them up correctly, so that you don’t have to care about a thing in the world.

Except one thing: now you also need to compile hello.c with TinyCC, and for some bizzare reasons, it so happens that TinyCC runs only on Linux, and not on macOS. You can spin up a Docker container, but again, you know better.

You decide to use Podman.

First you alter shell.nix to set up Podman et al., and TinyCC:

with (import <nixpkgs> { });
mkShell {
  buildInputs =
    # packages available on both linux and macos
    [ just gcc ]
    # packages available only on linux
    ++ (lib.optionals stdenv.isLinux [ tinycc ])
    # macos tooling to run linux packages
    ++ (lib.optionals stdenv.isDarwin [ podman qemu ]);
}
shell.nix

Then you write the Just commands to create and operate a Podman container:

container_name := "demo"

_create-vm:
    podman machine init --cpus 12 --memory 8192 --disk-size 50 \
      --volume $HOME:$HOME || true

_start-vm: _create-vm
    podman machine start || true

_stop-vm:
    podman machine stop

_create-container: _start-vm
    podman container ls -a | grep {{ container_name }} > /dev/null || \
        podman create -t --name {{ container_name }} -w /workdir \
            -v {{ root_dir }}:/workdir nixos/nix

_start: _create-container
    podman start {{ container_name }}

_stop: && _stop-vm
    podman stop {{ container_name }} || true
justfile

And the helper commands to run Just commands in the Podman container:

_podman-exec cmd *args: _start && _stop
    podman exec -it {{ container_name }} nix-shell \
      --command "just {{ cmd }} {{ args }}"

_run-in-podman cmd *args:
    #!/usr/bin/env -S sh -eu
    if [ "{{ os() }}" = "macos" ]; then
        just _podman-exec "{{ cmd }}" {{ args }}
    else
        just "{{ cmd }}" {{ args }}
    fi
justfile

And finally, the commands to run TinyCC on hello.c:

__build-tcc:
    tcc -o hello-tcc hello.c

_build-tcc: (_run-in-podman "__build-tcc")

build-tcc: (_run-in-nix-shell "_build-tcc")
justfile

Finally, you compile hello.c with TinyCC by running1:

just build-tcc

You watch in amazement as Nix downloads Podman and QEMU, Just sets up and runs the container, Nix downloads TinyCC within the container, and TinyCC finally compiles the file. Everything cleans up afterwards, and you are left with a hello-tcc binary file in your directory, which you cannot run because it was compiled on Linux, and you are on macOS2. But whatever. You job was to compile, not to run. You pack up your laptop, move to the living room, and open it again to browse Reddit. A day well spent3.

  1. If you have Just installed at the OS level, you can run Just commands from other directories as well, like this:

    just ~/Projects/just-nix-podman-demo/build-tcc
    ↩︎

  2. This justfile runs fine on a Linux machine as well, except the build-clang command.↩︎

  3. The use-case described in this post is a rather trivial and contrived example, but this pattern has served me well in real-world use-cases.↩︎

Like, repost, or comment

Posted by

Like this post? Subscribe to get future posts by email.

Got suggestions, corrections, or thoughts? Post a comment!

Cancel Reply
Markdown is allowed
Email is used just to show an avatar image and is not displayed.
Comments are moderated. They will appear below after they are approved.

35 comments

How does just compare to invoke?

https://www.pyinvoke.org/

peterbourgon

this behavior is desirable when using make as a build system, but not when using it as a command runner

make is by definition a build system and not a command runner, right?

Personally I prefer Apptainer (Formerly Singularity) instead of Podman. The latest release of Apptainer (1.1.0) is fully rootless with no special suid binary. A bonus is that it also doesn’t require the setup of subuid/subgid configurations. You just need unprivileged user namespaces enabled in your kernel or sysctl settings.

So what’s the benefit over Podman? I have Podman running fully rootless with no effort on NixOS.

Podman falls on its face in HPC environments or anywhere you use LDAP/AD (SSSD). Podman also doesn’t work over NFS, GPFS, or Lustre remote filesystems. Apptainer can run containers on these filesystems but does still require a local disk for builds.

Lastly the really nice thing with Apptainer is that containers are a single shareable file that is very easy to move around. No need for a container registry once your container is built. All around it’s currently a better tool and works better out of the box.

FWIW you can also use shell for this instead of Just, since you’re writing shell anyway: 

_run-in-nix-shell cmd *args:
    #!/usr/bin/env -S sh -eu
    if [ "{{ in_nix_shell }}" = "false" ]; then
        nix-shell "shell.nix" --run "just \"{{ root_dir }}/{{ cmd }}\" {{ args }}"
    else
        just "{{ root_dir }}/{{ cmd }}" {{ args }}
    fi

I would write this as: 

_run-in-nix-shell() {
  if [[ $in_nix_shell = false ]]; then
    # original and new one have some quoting issues here
    nix-shell shell.nix --run "just \"$root_dir/$cmd\" $@"
  else
     # no quoting issues
     just "$root_dir/$cmd" "$@"
  fi
}

"$@"  # run function $1 with args $2 $3 $4 ...

The {{ }} syntax in Just seems unnecessary, in adition to the *args. Shell already has stuff like that!

This introduced more quoting problems than shell already has!

Probably should write a blog post about this … Related comment:

https://lobste.rs/s/sq9h3p/unreasonable_effectiveness_makefiles#c_bwfha2

Oh my god, yes, taskfiles all day long! You have to know shell anyway, why not use it for what it’s good at?

Yup, sometimes I feel like we’re stuck in a loop where we keep inventing the same mistakes – e.g. make doing its own interpolation with $ on top of shell, various YAML-based languages and {{ }} embedding shell, and now Just and {{ }}.

It’s 70’s string soup, but invented in 2020. (I am glad that memory safety has become a meme. But it’s weird that string hygiene isn’t a meme, despite those problems being arguably more common!)

In meme format:

Programmers; don’t use shell, it’s too complex and unsafe!

Also programmers: check out this simple YAML-based config format! You can interpolate variables!

But I also recognize that people want convenience, and defaults set up for them, and shell has stagnated, and has distribution problems.

I mentioned in that comment that the Taskfile pattern in shell (ironically) doesn’t have autocompletion, and you want to generate help, etc. My former coworker made an attempt in pure shell, but it didn’t catch on: https://github.com/mbland/go-script-bash

So there is no single way to do it in shell. Someone also mentioned “scripts to rule them all”, which is fundamentally the same, but somehow didn’t catch on, despite being used by Github (at least for a time).

interesting, can you explain the _create-container: line in more detail? How does podman work with nix?

I also don’t quite get how tcc lands in the podman container. I’m guessing through the shell.nix with lib.optional for Linux but where exactly is that called?

_podman-exec starts a nix-shell, inside which a Just command is run. Starting the nix-shell fetches tcc due to the mention in shell.nix.

_create-container checks if the container already exists by grepping the output of podman container ls command. If the container does not exist, it is created by the podman create command. The root directory of the project is mounted at /workdir inside the container, which is also set at the default working directory of the container by the -w /workdir option.

When a Just command is executed inside the container using the _podman-exec command, it starts a nix-shell first, inside which the Just command is run.

Is there some nice way to add “task running” functionality directly to shell.nix? For example, I have the following shell.nix for my blog: 

λ bat shell.nix
# rm ./vendor -rf && nix-shell --run 'bundle install'
# nix-shell --run 'bundle exec jekyll serve --livereload'
with import <nixpkgs> {}; mkShell {
  packages = [ ruby ];
}

The “task running” functionality are the two comments above, which I just copy-paste in the shell.

Is there some short, non-horrible way to allow me to type nix shell –run serve and have that to what I want? Preferably with something like nix shell –run list-available-tasks.

Why not just put those tasks as separate scripts in a subdirectory?

Like: https://github.blog/2015-06-30-scripts-to-rule-them-all/

That can work, but I’d love too keep this scoped to a single file, to reduce fs clutter, and scoped to a single CLI entry point (nix shell) to reduce mental clutter (cc @grahamc, this actually sounds like a user pain potentially relevant to determinate systems).

I believe you could do something like this in a flake file: 

 apps."<system>".task-name = { type = "app"; program = lib.writeScript... ; };

And then run it with “nix run task-name”. I can’t remember the actual definition, so may need some fixing.

Similar https://www.ertt.ca/nix/shell-scripts/

Indeed!

Now the shell.nix has 

with import <nixpkgs> {}; mkShell {
  packages = [ 
    (writeShellScriptBin "serve" "bundle exec jekyll serve --livereload")
    (writeShellScriptBin "install" "rm ./vendor -rf && bundle install")
    ruby libffi pkg-config
  ];
}

and nix-shell –run serve does what I need, many thank!

Sure thing, you can just put these little scripts into derivations and make them available in the shell environment: 

let pkgs = import <nixpkgs> {};
    installScript = pkgs.writeScriptBin "install" "rm ./vendor -rf && bundle install";
    serveScript = pkgs.writeScriptBin "serve" "bundle exec jekyll serve --livereload"
in with pkgs; mkShell {
  packages = [ installScript serveScript ruby ];
}

With this, you can use nix-shell –run install and nix-shell –run serve.

If you put a little more sophistication into it, I’m sure you can define the scripts in a way that you can also generate a list-available-tasks script. Something like: 

let pkgs = import <nixpkgs> {};
    makeScripts = scriptDefinitions:
        let scripts = map ({name, source}: pkgs.writeScriptBin name source) scriptDefinitions;
            listTasksScript =
                pkgs.writeScriptBin "list-available-tasks"
                    (concatStringSep "" (map ({name}: "echo ${name}\n") scriptDefinitions));
        in scripts ++ [ listTasksScript ];
    scripts = makeScripts [
        {name = "install", source = "rm ./vendor -rf && bundle install"}
        {name = "serve", source = "bundle exec jekyll serve --livereload"}
    ];
in with pkgs; mkShell {
  packages = scripts ++ [ ruby ];
}

I didn’t test any of this, so there is probably a little bit of debugging required to make it work, but I’m confident it can work like this.

“Just” seems like a great tool, but it’s gonna be a pain to Google any issues.

It has a pretty detailed manual. I never faced any issue that I could not find a fix for in the manual.

Throwing in some direnv makes this even better, too. I use use flake absolutely everywhere.

devshell is pretty useful too :)

Apprehensive_Sir_243

What does devshell do better than direnv?

I never said anything about “better than”, I would say they complement eachother. Devshell gives a nice interface to define the development environment, I guess like mkShell on steroids. They have an authorship overlap too.

I do use direnv all the time. I wanted to keep the post focused so I didn’t mention it.

Cool, and thank you for the post! Always happy to see nix-related content out there.

Wouldn’t it be easier to just make each justfile target call nix-shell –run “some command”?

Apprehensive_Sir_243

What is the benefit of Just over a scripting language like Bash?

You cal list the possible commands to run with just -l.

I think Just has easier to understand syntax and fewer gotchas. Its much less capable than a scripting language, so fewer chances of messing things up.

It’s halfway between just having a bash script and having a makefile

for the purpose of this post, it is a way to easily create reproducible development environments

I don’t understand the advantage here versus all Nix and building a container, if necessary, through Nix. Using the Nix shell is quick/dirty, but it turns a stateless build tool into one that has the state of the shell. Why just have a reproducible development environment when the whole derivation top to bottom can be reproducible? Between packages and apps you can do just about everything and without the overhead of containers.

I understand Nix is a learning cliff to swallow, but it will, in the long run, simplify this entire setup to just one tool instead of multiple and this one-tool simplicity makes it a lot easier for teams to understand.

You are right. But it took me literally months to learn enough Nix to be able do everything in Nix only. This setup is easier, it works fine and I figured it out in less than a day. So I stuck to it.

One of the reasons why I see not going full build in nix is when one needs to collaborate with others that do not use nix. For example, I have to occasionally contribute to internal python project, where each has a different setup (no instructions, use virtualenv with requirements.txt, use conda, use an internal tool). When I’m not a regular contributor, getting into nuances of each setup is a big overhead. That’s why I set up a shell.nix file that bring me almost everything I need to just start coding. Sometimes that works, sometimes it doesn’t, especially when external, non-python dependencies are needed (ironic, right?). That way, others can continue doing work using tools they are comfortable with, while I get my safety net.

Sure, I bring a Flake to JS projects like this and don’t commit it, but I if it’s my own repo, I’m not going to try suggest folks use 3 tools (Nix, Just, Podman) when I could use 1 (Nix). Depending on the audience you can suggest a devShell or explain how to use the ‘normal’ community tooling, but even then, often the *2nix is overall a better experience and unlocks build sharing like Cachix.