x86_64 linux builder on aarch64-darwin

[eureka-cpu]

I installed rosetta 2 on my M1 macbook pro, and tried to run some checks that only work on x86_64-linux and sadly it shows that even though I have the x86_64 darwin options set in my nix-darwin configuration, only the aarch64-linux builder appears in the list of available machines. Is it possible to add an x86_64 builder as well?

[gustafj]

By setting package = pkgs.darwin.linux-builder-x86_64; in the linux-builder config you are able to build for x86_64-linux.

However there are, to my knowledge, no way yet of concurrently running two linux-builders one for aarch64-linux and one for x86_64-linux (I'd like this though).

[nicknovitski]

I'm sure it's possible. Look at the config section of the module; setting nix.linux-builder.enable just sets other options, like launchd.daemons and nix.buildMachines. What if you set those options directly in your own modules, with the appropriate parts duplicated for the two builders? I'm going to give that a try.

If it is possible, then it should also be possible for someone to extend the nix-darwin modules to support a pair or attribute set of builders, someday.

(I wish it was as easy as adding config.virtualisation.rosetta.enable = true; to a single builder, but that doesn't work for me.)

[nicknovitski]

Yep, I did it 😄 :

% cat /etc/nix/machines                                 
ssh-ng://builder@linux-builder x86_64-linux /etc/nix/builder_ed25519 1 1 kvm,benchmark,big-parallel - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUpCV2N4Yi9CbGFxdDFhdU90RStGOFFVV3JVb3RpQzVxQkorVXVFV2RWQ2Igcm9vdEBuaXhvcwo=
ssh-ng://builder@linux-builder-aarch64 aarch64-linux /etc/nix/builder_ed25519 1 1 kvm,benchmark,big-parallel - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUpCV2N4Yi9CbGFxdDFhdU90RStGOFFVV3JVb3RpQzVxQkorVXVFV2RWQ2Igcm9vdEBuaXhvcwo=

[Samasaur1]

Can you share the relevant config? It might be possible to add a nix-darwin option to configure this behavior

[eureka-cpu]

I just added this one liner and I have the same output, except they are listed differently. Also might mention I haven't actually tried to use it yet.

{
  linux-builder = {
    enable = true;
    systems = with flake-utils.lib.system; [ aarch64-darwin x86_64-darwin ];
  };
}

$ cat /etc/nix/machines                                                                                                                                 
ssh-ng://builder@linux-builder aarch64-darwin,x86_64-darwin /etc/nix/builder_ed25519 1 1 kvm,benchmark,big-parallel - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUpCV2N4Yi9CbGFxdDFhdU90RStGOFFVV3JVb3RpQzVxQkorVXVFV2RWQ2Igcm9vdEBuaXhvcwo=

[nicknovitski]

I'm actually no longer sure it's possible to implement this in a convenient way in nix-darwin, and I'll explain why, in case someone can correct me.

The problem is ports: the default configuration of each builder gives them the same hostPort (31022), and you can't build a builder with a different configuration unless you already have a builder: only the default ones are in the binary cache. Like the nix-darwin docs say:

You should first use the Linux builder without changing the builder configuration otherwise you may not be able to build the Linux builder.

So to go from no builders running to two, I had to switch to configurations like this in order:

• one default builder
• a custom builder with the same architecture as the existing default builder but a non-conflicting port, and the default builder of the other architecture
• the two custom builders of non-conflicting ports

There's a lot that can go wrong. I don't see a way to avoid it without changing the hostport configuration of one of the cached builders in nixpkgs, which would technically be a breaking change, and the documentation would have to change. https://github.com/NixOS/nixpkgs/blob/a01ee2f7f3e0f33f6738b6f49d24d3653a415682/nixos/modules/profiles/nix-builder-vm.nix#L86

Oh, also, I had to run a lot of it with --option sandbox false, because of this issue: NixOS/nix#4119.

[eureka-cpu]

I think even just having docs on how to do it in general would be awesome, I don't mind having to go through some steps to get it working (:

[cpick]

I recently put together a rosetta-builder nix-darwin module that can replace (or run alongside) nix-darwin's nix.linux-builder. It uses Rosetta 2 (within the VM) and I don't believe pkgs.darwin.linux-builder-x86_64 does so it should be much faster and it supports aarch64-linux and x86_64-linux builds simultaneously.

[krad246]

I just added this one liner and I have the same output, except they are listed differently. Also might mention I haven't actually tried to use it yet.

{
  linux-builder = {
    enable = true;
    systems = with flake-utils.lib.system; [ aarch64-darwin x86_64-darwin ];
  };
}

$ cat /etc/nix/machines                                                                                                                                 
ssh-ng://builder@linux-builder aarch64-darwin,x86_64-darwin /etc/nix/builder_ed25519 1 1 kvm,benchmark,big-parallel - c3NoLWVkMjU1MTkgQUFBQUMzTnphQzFsWkRJMU5URTVBQUFBSUpCV2N4Yi9CbGFxdDFhdU90RStGOFFVV3JVb3RpQzVxQkorVXVFV2RWQ2Igcm9vdEBuaXhvcwo=

Yeah, I thought of the same thing.

Can't you just set boot.binfmt.emulatedSystems in the builder to include x86_64-linux?

Now, this isn't perfect because it's Qemu sharing the same underlying machine but that's what I do myself. I suspect one of the big disadvantages is it's possible for some shared state based corruptions within the internal VM. But I enabled sandboxing which ideally then makes that much harder to actually do. So really what I have seems to be closer to having multiple containers running, not multiple VMs running per se.

I'm assuming this is not actually what you want here though, and you specifically want to try out the Rosetta codepath, which definitely has advantages.

[krad246]

I recently put together a rosetta-builder nix-darwin module that can replace (or run alongside) nix-darwin's nix.linux-builder. It uses Rosetta 2 (within the VM) and I don't believe pkgs.darwin.linux-builder-x86_64 does so it should be much faster and it supports aarch64-linux and x86_64-linux builds simultaneously.

From what I've seen the linux-builder package is Qemu for the Host CPU. So it's basically like running UTM. Not too bad, but if you want to try something new Rosetta is a sidegrade.

[eureka-cpu]

Forgot to reply back here, after giving it a try with the above code snippet, it does not work unfortunately.

[krad246]

I think even just having docs on how to do it in general would be awesome, I don't mind having to go through some steps to get it working (:

I have some really crappy documentation in my flake but my flake has covered a lot of problems like this over time if you want to look. Been meaning to get mkdocs integration here

[krad246]

Forgot to reply back here, after giving it a try with the above code snippet, it does not work unfortunately.

For me I specifically did this:

  {
  self,
  config,
  lib,
  ...
}: let
  cfg = config.krad246.darwin.linux-builder;
  inherit (lib) options types;
in {
  options = {
    krad246.darwin.linux-builder = {
      ...

      systems = options.mkOption {
        type = types.listOf types.str;
        default = ["i386-linux" "i686-linux" "x86_64-linux" "aarch64-linux"];

        defaultText = ''
          The `nixpkgs.hostPlatform.system` of the build machine's final NixOS configuration.
        '';
        example = options.literalExpression ''
          [
            "x86_64-linux"
            "aarch64-linux"
          ]
        '';
        description = ''
          This option specifies system types the build machine can execute derivations on.

          This sets the corresponding `nix.buildMachines.*.systems` option.
        '';
      };
    };
  };

  config = {
    nix = {
      linux-builder = {
        ...
        inherit (cfg) systems;
      };
    };
  };
}

See this file implementation for the rest.

https://github.com/krad246/nix-systems/blob/2e84ce65a125636c278a950ad67283cbfd2f441b/modules/darwin/linux-builder.nix