Reproducible Build Artifacts

28 Nov 2025

Taproot Assets endeavours to provide completely reproducible and verifiable build artifacts for all releases. These include binaries for a number of target architectures/platforms, along with source code and vendored dependencies, all packaged up in gzipped tarballs, or in zip(1) files. Each release comes with a manifest of these artifacts along with accompanying SHA256 digests.

To attest the artifacts for a release, trusted contributors verify that they can produce the same manifest, and then they cryptographically sign it:

$ gpg --verify manifest-jtobin-v0.7.0.sig manifest-v0.7.0.txt
gpg: Signature made Thu 20 Nov 14:07:09 2025 +04
gpg:                using EDDSA key 7363F82F43B1324E1B0761310E4647D58F8A69E4
gpg: Good signature from "Jared Tobin (https://ppad.tech) <jared@ppad.tech>" [ultimate]
gpg:                 aka "Jared Tobin (jtobin) <jared@jtobin.io>" [ultimate]

If one is operating at an appropriate paranoia level, he can thus personally verify that, first, a quorum of trusted contributors has attested to a certain set of artifacts, and, second, that he can produce, from source and his own tool set, those very same artifacts, before running the software.

Interestingly, the binaries themselves are the least tricky part to verify, at least in principle. Modern Go compilers are able to produce completely reproducible binaries for any target, independent of the build host architecture. That is: given appropriate environment variables and linker flags (namely CGO_ENABLED="0" and -trimpath, probably -s and -w too), a modern Go compiler will produce, bit-for-bit, the same output, given consistent source, other flags, and so on.

You have to do a bit more legwork for the tarballs, etc. Putting together reproducible tarballs involves a salad of flags, e.g.:

  export TZ=UTC

  find "${dir}" -print0 | LC_ALL=C sort -r -z | tar \
    "--mtime=${BUILD_DATE}" --no-recursion --null --mode=u+rw,go+r-w,a+X \
    --owner=0 --group=0 --numeric-owner -c -T - | gzip -9n > "${dir}.tar.gz"

You have to be careful to lock down a bunch of stuff, like timestamps and user/group permissions. What’s more, you have to ensure you’re using the right variant of tar (BSD vs GNU), and ditto for at least gzip and zip.

The problem of producing a hermetic build environment is solved by Nix, of course (or Docker, more on which another time). But some interesting problems can actually pop up when using Nix to build the kinds of reproducible Go binaries I mentioned above.

Let’s start with the tarballs and zip files. Timestamps and permissions are normalized within the Nix store, so, easy: we merely need to put the following in the right place in a Nix flake:

  { nativeBuildInputs = with pkgs; [ coreutils gnutar gzip zip ]; }

Add a minimal set of reproducibility flags to the tar call, and then anyone building the output with the same Nix flake metadata should be able to produce consistent gzipped tarballs containing consistent compiled binaries.

Or so it would seem. When I first tried this, I was able to produce reproducible tarballs for both the project source and its vendored dependencies. The source is declared as follows:

  package = "taproot-assets";
  tag = "v0.7.0";

  # source info
  org = "lightninglabs";
  repo = package;
  tap_src = pkgs.fetchFromGitHub {
    owner = org;
    repo = repo;
    rev = tag;
    hash = "sha256-z4lNeVmy0AEDM8ivdfXfXqi/V1LIDHV2KtS5/19+Jlk=";
  };

and given a ‘tap’ derivation built by Nixpkgs’s buildGoModule, the following, in the buildPhase, can later get us the tarballs of interest:

  tar_gz $out/vendor.tar.gz ${tap.goModules} .
  tar_gz $out/taproot-assets-source-${tag}.tar.gz ${tap_src} .

But strangely, to me, I couldn’t generate reproducible binaries. That is: even though I pinned to a specific Go toolchain, I would produce binaries for different targets that differed depending on the architecture of the host. This was strange, since I know that the Go compiler shouldn’t have any issue doing this, and I was clearly using the right settings:

  # tapd builder
  build_tap = {
        goos ? null
      , goarch ? null
      , goarm ? null
      , tags ? tags_base
      , ldflags ? ldflags_base
      , gcflags ? [ ]
      }: pkgs.buildGoModule {
    pname = package;
    version = tag;
    src = tap_src;
    vendorHash = "sha256-p75eZoM6tSayrxcKTCoXR8Jlc3y2UyzfPCtRJmDy9ew=";
    subPackages = [ "cmd/tapd" "cmd/tapcli" ];

    inherit tags ldflags gcflags;

    preBuild = with pkgs; ''
      export CGO_ENABLED="0"
      export GOAMD64="v1"
      ${lib.optionalString (goos != null) "export GOOS=${goos}"}
      ${lib.optionalString (goarch != null) "export GOARCH=${goarch}"}
      ${lib.optionalString (goarm != null) "export GOARM=${goarm}"}
    '';

    doCheck = false;
  };

(‘-trimpath’, another requirement for reproducible builds, is added by default when using Nixpkgs’s buildGoModule, as is ‘-buildid=’, ensuring no rogue metadata creeps in.)

I dug into this by inspecting a couple of the produced binaries for a random architecture, one built on an amd64 Linux machine and the other on aarch64 Darwin. I produced a couple of dumps of ELF section headers via readelf -S $MY_BINARY and diffed them:

$ diff sections-linux.txt sections-darwin.txt
1c1
< There are 14 section headers, starting at offset 0xd4:
---
> There are 14 section headers, starting at offset 0x386ed30:
18c18
<   [13] .shstrtab         STRTAB          00000000 3870000 000094 00      0   0  1
---
>   [13] .shstrtab         STRTAB          00000000 386ecb0 00007d 00      0   0  1

The .shstrtab (“section name string table”) sections, which are pure ELF metadata, differed, so I produced hexdumps of those, this time via readelf -x .shstrtab $MY_BINARY, and diffed them:

$ diff shstrtab-linux.txt shstrtab-darwin.txt
3,12c3,10
<   0x00000000 002e7465 7874002e 6e6f7074 72646174 ..text..noptrdat
<   0x00000010 61002e64 61746100 2e627373 002e6e6f a..data..bss..no
<   0x00000020 70747262 7373002e 676f2e66 757a7a63 ptrbss..go.fuzzc
<   0x00000030 6e747273 002e676f 2e627569 6c64696e ntrs..go.buildin
<   0x00000040 666f002e 676f2e66 69707369 6e666f00 fo..go.fipsinfo.
<   0x00000050 2e656c66 64617461 002e726f 64617461 .elfdata..rodata
<   0x00000060 002e7479 70656c69 6e6b002e 69746162 ..typelink..itab
<   0x00000070 6c696e6b 002e676f 73796d74 6162002e link..gosymtab..
<   0x00000080 676f7063 6c6e7461 62002e73 68737472 gopclntab..shstr
<   0x00000090 74616200                            tab.
---
>   0x00000000 002e7465 7874002e 6e6f7074 72627373 ..text..noptrbss
>   0x00000010 002e6273 73002e67 6f2e6669 7073696e ..bss..go.fipsin
>   0x00000020 666f002e 676f2e62 75696c64 696e666f fo..go.buildinfo
>   0x00000030 002e7479 70656c69 6e6b002e 69746162 ..typelink..itab
>   0x00000040 6c696e6b 002e7368 73747274 6162002e link..shstrtab..
>   0x00000050 676f7063 6c6e7461 62002e67 6f73796d gopclntab..gosym
>   0x00000060 74616200 2e6e6f70 74726461 7461002e tab..noptrdata..
>   0x00000070 726f6461 7461002e 64617461 00       rodata..data.

Not much to go on there, but I eventually traced this to the fact that mkDerivation in Nixpkgs’s stdenv runs a “fixupPhase” by default, which performs its own stripping of ELF metadata, separate from what the Go compiler does when passed the ‘-s’ and ‘-w’ linker flags (for stripping symbol and debug information, plus DWARF debugging info, respectively). The stripping done by the fixupPhase seems to vary depending on the package set used, e.g. between aarch64-darwin and x86_64-linux.

I disabled the fixupPhase’s stripping by adding

  dontStrip = true;

to my builder function, and tried again. This time the resulting artifacts’ file sizes normalized, but they still produced different SHA256 digests. I produced another hexdump of each, this time using good old xxd(1), and diffed them again:

2028124,2028126c2028124,2028126
< 01ef25b0: 7265 2f32 7334 6870 7137 3368 6e34 396a  re/2s4hpq73hn49j
< 01ef25c0: 6438 3478 3937 366d 3361 6370 3372 6433  d84x976m3acp3rd3
< 01ef25d0: 6b31 782d 747a 6461 7461 2d32 3032 3562  k1x-tzdata-2025b
---
> 01ef25b0: 7265 2f78 686e 6231 6434 7767 6a33 3878  re/xhnb1d4wgj38x
> 01ef25c0: 6833 3833 7973 3162 7969 676e 3079 6861  h383ys1byign0yha
> 01ef25d0: 6371 682d 747a 6461 7461 2d32 3032 3562  cqh-tzdata-2025b
2028308,2028310c2028308,2028310
< 01ef3130: 2069 740a 2f6e 6978 2f73 746f 7265 2f34   it./nix/store/4
< 01ef3140: 6b77 3938 7138 3470 6276 326e 7771 3361  kw98q84pbv2nwq3a
< 01ef3150: 7931 3261 6176 6661 397a 6934 6439 7a2d  y12aavfa9zi4d9z-
---
> 01ef3130: 2069 740a 2f6e 6978 2f73 746f 7265 2f7a   it./nix/store/z
> 01ef3140: 6a62 3467 7763 7971 7670 3862 3236 716b  jb4gwcyqvp8b26qk
> 01ef3150: 7832 7163 3033 3776 6a6a 3631 7838 302d  x2qc037vjj61x80-
2028473,2028475c2028473,2028475
< 01ef3b80: 7374 6f72 652f 7279 6d64 7764 386b 3461  store/rymdwd8k4a
< 01ef3b90: 6268 3430 6233 6d77 3470 6636 3967 3739  bh40b3mw4pf69g79
< 01ef3ba0: 336d 6471 6269 2d69 616e 612d 6574 632d  3mdqbi-iana-etc-
---
> 01ef3b80: 7374 6f72 652f 7038 6369 3239 7637 6b37  store/p8ci29v7k7
> 01ef3b90: 6777 7773 786a 3138 3170 3035 6a63 6362  gwwsxj181p05jccb
> 01ef3ba0: 6e71 6236 3439 2d69 616e 612d 6574 632d  nqb649-iana-etc-
2028626,2028628c2028626,2028628
< 01ef4510: 6f72 652f 7279 6d64 7764 386b 3461 6268  ore/rymdwd8k4abh
< 01ef4520: 3430 6233 6d77 3470 6636 3967 3739 336d  40b3mw4pf69g793m
< 01ef4530: 6471 6269 2d69 616e 612d 6574 632d 3230  dqbi-iana-etc-20
---
> 01ef4510: 6f72 652f 7038 6369 3239 7637 6b37 6777  ore/p8ci29v7k7gw
> 01ef4520: 7773 786a 3138 3170 3035 6a63 6362 6e71  wsxj181p05jccbnq
> 01ef4530: 6236 3439 2d69 616e 612d 6574 632d 3230  b649-iana-etc-20

The difference here was immediately discernable: different binaries contained different hard-coded paths to the Nix store. This was at first immensely confusing to me, but it turns out that the default Go toolchains in Nixpkgs are not “official” Go toolchains, but patched ones. You can see this in the Go derivation for e.g. the nixos-25.05 package set, which contains the following ‘patches’ attribute:

  patches = [
    (replaceVars ./iana-etc-1.25.patch {
      iana = iana-etc;
    })
    # Patch the mimetype database location which is missing on NixOS.
    # but also allow static binaries built with NixOS to run outside nix
    (replaceVars ./mailcap-1.17.patch {
      inherit mailcap;
    })
    # prepend the nix path to the zoneinfo files but also leave the
    # original value for static binaries that run outside a nix server
    (replaceVars ./tzdata-1.19.patch {
      inherit tzdata;
    })
    ./remove-tools-1.11.patch
    ./go_no_vendor_checks-1.23.patch
  ];

The idea here is that the Go compiler bakes into the binaries it produces a number of fallback runtime paths for stuff like ‘tzdata’, e.g. ‘/usr/share/zoneinfo’. On NixOS this, and other paths, do not exist, so the ‘tzdata-1.19.patch’, for example, hard codes a path to the tzdata package in the Nix store that depends on the package set used. Since the package sets differ between platforms, these paths will also differ, and so will any binaries produced by the Go compilers they contain.

The solution to this is easy in Nix, provided you know you actually have to do it. Just build with an unpatched Go toolchain:

  # nixpkgs applies various patches to go toolchains; we need to remove
  # them to produce host-architecture-independent binaries.
  go-unpatched = pkgs.go.overrideAttrs (_: {
    patches = [ ];
  });

  buildGoModule = pkgs.buildGoModule.override {
    go = go-unpatched;
  };

  # tapd builder
  build_tap = {
        goos ? null
      , goarch ? null
      , ..
      , gcflags ? [ ]
      }: buildGoModule {
    ..
  };

This works as expected. The produced manifests look like this:

$ head -3 manifest-v0.7.0-darwin.txt | cut -f1 -d' '
ebebc38f71c7a902341bf4247d8ee2752d0d233deec2034953dab4b44ecb622a
737eb475c5435beedcdb49ae7adf0716c9980523e89682fea4622b59c5d132df
8570b3c659fc2df9cff365fc06bd95459d903b6a3e7c8fd2e5e4efc1a77f7adf

and diffing the manifests produced on both macOS and Linux hosts illustrates that the artifacts match, bit-for-bit, precisely:

$ diff manifest-v0.7.0-darwin.txt manifest-v0.7.0-linux.txt
<empty>

Here’s a gist to a flake for producing various tapd builds, as well as release artifacts, if you’re curious.