Add Ninja build system to Mac, Linux and Windows

[Toxe]

Tool information

• Tool name: Ninja
• Add or update? Add
• Desired version: latest
• Approximate size: 100-300 KB (roughly)
• If this is an add request:
  • Brief description of tool: Ninja is a very popular build system, especially in the C++ world, and is replacing GNU Make in a lot of places.
  • URL for tool's homepage: https://ninja-build.org

Area for Triage: Packages

Question, Bug, or Feature?: Feature

Virtual environments affected

• macOS 10.15
• Ubuntu 16.04 LTS
• Ubuntu 18.04 LTS
• Windows Server 2016 R2
• Windows Server 2019

Can this tool be installed during the build?
Yes. See: https://github.com/ninja-build/ninja/wiki/Pre-built-Ninja-packages

Here is an example how I am using it in my own project:
https://github.com/Toxe/example-cpp-ci-project/blob/master/.github/workflows/ci.yml

• For macOS: brew install ninja
• For Linux: sudo apt-get install ninja-build
• For Windows: Sorry, I don't know and have not tried it.

Are you willing to submit a PR?
I might, but I am no expert about Actions/Environments and the manual installation is pretty simple anyways, so I suppose it would be pretty fast and easy to do for someone who actually know what they are doing.

[Toxe]

Added to Issue: Area for Triage: Packages

[maxim-lobanov]

@alepauly , Tool can be installed in runtime using the following commands:

# macOS
brew install ninja
# Ubuntu
sudo apt-get install ninja-build

Installation takes about 10 seconds

[alepauly]

Hi @Toxe, thanks for your suggestion but considering that it' easy to add Ninja to workflows on demand, we won't add it to the image just yet.

[Toxe]

That's a bummer. And I honestly don't really get it, because out of all CI solutions I have tried over the last weeks the Github Actions VMs seem to be the best equipped and up-to-date installations with (nearly) everything one would need. And if there are the newest compiler versions (among several other versions) and tools like GNU Make and CMake installed out-of-the-box I really don't get why a tiny tool like Ninja (that, like CMake, is becoming the defacto standard for a lot of people) does not get included.

In most of my CI scripts Ninja is pretty much the only thing I need to install by hand and then I am good to go. If it were available by default this would shave of a couple of seconds of runtime and would run Github Actions scripts even faster than they already do.

Yeah, sorry, as I said I don't really get it. But okay, it's your system and not the end of the world, obviously I can cope with it without a problem. But, would have been nice.

[alepauly]

@Toxe thanks for your thoughts. We are trying to improve on a few things about the images for virtual environments including the reliability of building the images themselves, the effort to keep them up to date, the disk foot print, and others. This means that we are being fairly picky right now about what to install by default and since there's an easy workaround for Ninja, it seems like we can avoid installing it. One alternative I can suggest is to have a setup Actions that does that install in one step?

We are happy to revisit this in the future as we improve on the things I mentioned, feel free to ping me whenever to remind. Thanks again for speaking up.

[Toxe]

Thank you for explaining your thoughts on this.

[marc-hb]

We are happy to revisit this in the future as we improve on the things I mentioned, feel free to ping me whenever to remind.

4 years later, please re-open and re-consider this.

This means that we are being fairly picky right now about what to install by default

Understandable, but:

For cross-platform C/C++ there are only 3 build systems in widespread use:

• GNU Make +/- autoconf. Not really cross-platform. Awkward, ancient and dying.
• CMake. Most popular. Its only cross-platform backend is: ninja.
• meson. Defaults to ninja.

So not having ninja installed by default means you don't care much about C/C++?

Besides running doxygen, there is basically nothing that can be done with C/C++ unless you build it. Even static analyzers usually work by intercepting the ninja build as their first step.

In fact even doxygen is often implemented as a CMake target for convenience.

And if there are the newest compiler versions (among several other versions) and tools like GNU Make and CMake installed out-of-the-box I really don't get why a tiny tool like Ninja (that, like CMake, is becoming the defacto standard for a lot of people) does not get included.

It is indeed super weird to have CMake installed by default without its most popular and only cross-platform backend. Very inconsistent. Moreover, CMake is quite complex and adding new features regularly which means the CMake version available by default is not always the version you want. Rarely ever any such compatibility issue with the so much simpler ninja.

We are trying to improve on a few things about the images for virtual environments including the reliability of building the images themselves,

ninja is by explicit design an extremely simple tool, delegating all the complexity to the front end: https://ninja-build.org/manual.html#_design_goals No "reliability" issue here.

the effort to keep them up to date,

As ninja is every simple, it extremely rarely needs to be updated (Unlike for instance: CMake)

the disk foot print,

ninja is very small.

ub22:~$ dpkg -s ninja-build
Package: ninja-build
Status: install ok installed
Installed-Size: 350 kilobytes!!

and others.

Other important considerations: CPU and bandwidth...

In most of my CI scripts Ninja is pretty much the only thing I need to install by hand and then I am good to go. If it were available by default this would shave of a couple of seconds of runtime

Pre-installing ninja would save more than just a couple seconds because by design you must run apt-get update before apt-get install anything:

• Ubuntu: Document requirement to always run "apt-get update" before install #2924

@maxim-lobanov, @alepauly please take a quick look at which package is most frequently installed at run-time across all C/C++ projects. I bet you have those stats internally because they translate to CPU cycles and bandwidth = money.

Installation takes about 10 seconds

19 seconds on Windows: https://github.com/thesofproject/sof/actions/runs/7689520973/job/20952092062

Worse: on Windows ninja ... comes and goes! #741

~19 seconds on Ubuntu:
https://github.com/thesofproject/sof/actions/runs/7689520973/job/20952090150

I have never tried brew install ninja in Github but I use brew often on a mac and anything with brew takes minutes because it always runs brew update first behind the scenes and that takes ages.

To build CMake projects on Ubuntu, a workaround is to use CMake+GNU Make. This can save the time to apt-get update && apt-get install ninja-build but this makes the build a bit slower and again this costs Github CPU cycles and money because Make is much less efficient.

[marc-hb]

We are happy to revisit this in the future as we improve on the things I mentioned, feel free to ping me whenever to remind.

Yet another year later...

Tool can be installed in runtime using the following commands:

Still no recommended way to install ninja on Windows...

Could you at least maintain an official GitHub Action hosted at https://github.com/actions that installs ninja? This would avoid guarding "install" code with OS-specific conditionals or auditing the security of 3rd party actions that already do this (like #741 (comment)).

including the reliability of building the images themselves, the effort to keep them up to date, the disk foot print, and others.

An official, install-ninja action would:

• not affect the build reliability and disk foot print of images themselves
• isolate the effort of keeping stuff up to date.

[Toxe]

Right, with each passing year I understand this less and less. Ninja is one of the most popular build tools for good reasons and also tiny. It improves build times by a lot and not having to install it manually would also reduce VM runtime.

But clearly resource costs and therefore saving money does not seem to matter here. 🤔

[marc-hb]

thanks for your suggestion but considering that it' easy to add Ninja to workflows on demand,...

I just asked runtime installation to be officially documented, please upvote:

• No official way to install ninja is documented #11235

Ever growing list of duplicates:

• Update/Add Ninja #741
• add ninja build to Linux images #742
• Update/Add Ninja #1531
• Update/Add [tool name] #4972
• windows-2022 image 20230918.1.0 removed the "ninja" tool #8343
• Add ninja #8348

[kdschlosser]

The biggest issue is with Windows. Unless Ninja is installed using the Visual Studio Installer there are build packages that are written for languages like Python that will not be able to properly detect it.

Here is an MSVC build system that integrates into setuptools and distutils for Python. It allows for compiling not only Python extension modules but also compiling executables as well at static and dynamic libraries. One of the things is does is it uses COM objects that are apart of the Visual Studio installer to locate various bits and pieces that are installed. The COM objects when I wrote the code were undocumented. My point is if it is not installed using the Visual Studio installer then anything that uses some kind of a build system may not be able to locate it. There are a lot of build systems that use vswhere and that uses the same COM objects as the system I wrote.

It doesn't make any sense that ninja is not installed by default. Just like it doesn't make sense that arm compilers are not installed either.

[Toxe]

It doesn't make any sense that ninja is not installed by default.

Yes, especially, and I cannot stress this enough, that we are talking about a tool that is tiny (0.5 MB on Windows).

[kdschlosser]

When you start adding up all of the people that are having to install it manually and the resources that end up being wasted from having to do so, tied to the cost of bandwidth.. from a monetary standpoint it really doesn't make any sense as to why it's not already installed. It looks like at one point in time it was at least with Windows.

What really throws me for a loop with the Windows images is Visual Studio is installed.. Why? Not like anyone is using the IDE portions of it in the CI... only the build tools need to be installed and Microsoft does have a package that has only the build tools and not the entire IDE. Talk about a waste there, Visual studio is a pretty large piece of software.

[marc-hb]

and Microsoft does have a package that has only the build tools and not the entire IDE.

Interesting, can you share some references? (I'm not familiar with Windows)

Just trying to play devil's advocate: maybe this lighter package is rarely used, poorly tested and has some subtle differences with the "real" thing that everyone uses?

That would not surprise me considering ninja already "came and went" even when sticking to the same Microsoft package! #741 (comment)
So imagine using a different flavor...

All toolchains are notoriously brittle, not just on Windows far from it.

[kdschlosser]

It installs all of the exact same pieces that get installed when you install the IDE except it doesn't install the IDE.

https://visualstudio.microsoft.com/visual-cpp-build-tools/

The installer that is used for the build tools is the same installer as Visual Studio, it just has the IDE bits left out.

[kdschlosser]

Toolchains are brittle because they change things from version to version. Important things like location, environment variable names. Things like that. With Windows it is actually worse because people that develop applications don't clean up the registry entries that are made when something gets updated or uninstalled. This leaves a bunch of invalid entries in the registry. Microsoft is the most guilty of doing this. The undocumented COM objects were added in Visual Studio 2017 and that makes it easier to know what is actually installed and what is not. It doesn't give all of the information needed to set up a proper build environment though. You still need to go into the registry to collect location paths and things of that nature.

The MSVC build script I wrote in Python supports locating installations of Visual Studio, Visual C++ and the build tools. all the way back to versions released in 2008. It detects Windows SDK installations back to version 6.0A (Windows 7) along with ALL .NET versions. it detects HTML Help, F#, and a plethora of other compilers that get installed with VS, VC++ or the build tools including Clang and Ninja. It also detects Nmake and CMake as well. It sets up a factory MSVC build environment that works 100% of the time. It took me years to iron all the bugs out of it due to Microsoft having zero consistency up until VS 2017 with where things are stored in both the registry and also the file system.

[kdschlosser]

All toolchains are notoriously brittle, not just on Windows far from it.

That is what a CI is supposed to solve by having commonly used tools pre installed.

[marc-hb]

https://visualstudio.microsoft.com/visual-cpp-build-tools/

Gotta love the "construction site" look of the guy in the picture at the top :-D I bet it's not a coincidence. Maybe it's AI-generated!