Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

dockerutil.BuildEmptyImage() fails with bad API version #646

Open
jimcottrell opened this issue Feb 26, 2024 · 23 comments
Open

dockerutil.BuildEmptyImage() fails with bad API version #646

jimcottrell opened this issue Feb 26, 2024 · 23 comments

Comments

@jimcottrell
Copy link

I've got a CI/CD pipeline that's run occasionally when we rebuild a image and create an optimized version with Slim. Without anything having changed that I'm aware of (same Slim version, same pipeline runner), this process has started failing with:

dockerutil.BuildEmptyImage: dockerapi.BuildImage() error = API error (400): client version 1.12 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version

I thought this might be related to #637, but it seems to be a separate issue, and the same thing happens whether or not I try setting --crt-api-version. Having updated everything to the latest versions, the problem continues. The issue seems to be specific to this environment, as a local build works.


Steps to Reproduce the Problem

  1. slim build --http-probe-off --continue-after=enter alpine

(trivial reproduction; of course not our actual image)

cmd=build state=started
cmd=build info=params target.image='alpine' continue.mode='enter' rt.as.user='true' keep.perms='true' tags='' image-build-engine='internal' target.type='image' 
cmd=build state=image.inspection.start
cmd=build info=image id='sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd' size.bytes='0' size.human='0 B' 
cmd=build info=image.stack index='0' name='alpine:latest' id='sha256:05455a08881ea9cf0e752bc48e61bbd71a34c029bb13df01e40e3e70e0d007bd' 
cmd=build state=image.inspection.done
cmd=build state=container.inspection.start
time="2024-02-26T18:12:39Z" level=error msg="dockerutil.BuildEmptyImage: dockerapi.BuildImage() error = API error (400): client version 1.12 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version\n / output: "
time="2024-02-26T18:12:39Z" level=fatal msg="slim: failure" error="API error (400): client version 1.12 is too old. Minimum supported API version is 1.24, please upgrade your client to a newer version\n" stack="goroutine 1 [running]:\nruntime/debug.Stack()\n\truntime/debug/stack.go:24 +0x5e\ngithub.com/slimtoolkit/slim/pkg/util/errutil.FailOn({0x231e3e0, 0xc0007e62a0})\n\tgithub.com/slimtoolkit/slim/pkg/util/errutil/errutil.go:32 +0x4b\ngithub.com/slimtoolkit/slim/pkg/app/master/inspectors/container.(*Inspector).RunContainer(0xc000464b00)\n\tgithub.com/slimtoolkit/slim/pkg/app/master/inspectors/container/container_inspector.go:430 +0x77a\ngithub.com/slimtoolkit/slim/pkg/app/master/command/build.OnCommand(_, _, {_, _}, _, {_, _}, {_, _}, {0x0, ...}, ...)\n\tgithub.com/slimtoolkit/slim/pkg/app/master/command/build/handler.go:1132 +0x4632\[ngithub.com/slimtoolkit/slim/pkg/app/master/command/build.glob..func1(0xc0003d35c0)\n\tgithub.com/slimtoolkit/slim/pkg/app/master/command/build/cli.go:760](http://ngithub.com/slimtoolkit/slim/pkg/app/master/command/build.glob..func1(0xc0003d35c0)/n/tgithub.com/slimtoolkit/slim/pkg/app/master/command/build/cli.go:760) +0x5d51\[ngithub.com/urfave/cli/v2.(*Command](http://ngithub.com/urfave/cli/v2.(*Command)).Run(0x3396180, 0xc0000af480)\n\[tgithub.com/urfave/cli/[email protected]/command.go:163](http://tgithub.com/urfave/cli/[email protected]/command.go:163) +0x583\[ngithub.com/urfave/cli/v2.(*App](http://ngithub.com/urfave/cli/v2.(*App)).RunContext(0xc000266d00, {0x233bb88?, 0x340d500}, {0xc00012e000, 0x7, 0x7})\n\[tgithub.com/urfave/cli/[email protected]/app.go:313](http://tgithub.com/urfave/cli/[email protected]/app.go:313) +0xaa5\[ngithub.com/urfave/cli/v2.(*App](http://ngithub.com/urfave/cli/v2.(*App)).Run(...)\n\[tgithub.com/urfave/cli/[email protected]/app.go:224\ngithub.com/slimtoolkit/slim/pkg/app/master.Run()\n\tgithub.com/slimtoolkit/slim/pkg/app/master/app.go:15](http://tgithub.com/urfave/cli/[email protected]/app.go:224/ngithub.com/slimtoolkit/slim/pkg/app/master.Run()/n/tgithub.com/slimtoolkit/slim/pkg/app/master/app.go:15) +0x45\nmain.main()\n\[tgithub.com/slimtoolkit/slim/cmd/slim/main.go:15](http://tgithub.com/slimtoolkit/slim/cmd/slim/main.go:15) +0x187\n" version="linux/amd64|Transformer|1.40.11|1b271555882eacdfb4e6598d6d0552e9b9b1449b|2024-02-02_01:36:22PM"

Specifications

  • Version: 1.40.11
  • Platform: linux/amd64 (self-hosted Bitbucket pipeline runner, which is configured for privileged Docker access so Slim can run)
  • Output of slim version:
cmd=version info=app container='true' dsimage='false' location='/opt/atlassian/pipelines/agent/build/dist_linux' current='1.40.11' verdict='you have the latest version' cmd='version' version='linux/amd64|Transformer|1.40.11|1b271555882eacdfb4e6598d6d0552e9b9b1449b|2024-02-02_01:36:22PM' 
cmd=version info=host cmd='version' osname='Ubuntu 22.04.3 LTS' osbuild='' version='#18~22.04.1-Ubuntu SMP Wed Jan 10 22:54:16 UTC 2024' release='6.2.0-1018-aws' sysname='Linux' 
cmd=version info=docker server.version='25.0.3' architecture='x86_64' cmd='version' name='a640edfcc04d' kernel.version='6.2.0-1018-aws' operating.system='Alpine Linux v3.19 (containerized)' ostype='linux' 
cmd=version info=dclient build.time='2024-02-06T21:13:08.000000000+00:00' git.commit='f417435' cmd='version' api.version='1.44' min.api.version='1.24' 
app='slim' message='GitHub Discussions' info='https://github.com/slimtoolkit/slim/discussions'
app='slim' message='Join the CNCF Slack channel to ask questions or to share your feedback' info='https://cloud-native.slack.com/archives/C059QP1RH1S'
app='slim' message='Join the Discord server to ask questions or to share your feedback' info='https://discord.gg/9tDyxYS'
app='slim' message='Join the Gitter channel to ask questions or to share your feedback' info='https://gitter.im/docker-slim/community'
@kcq
Copy link
Member

kcq commented Feb 28, 2024

Interesting... need to investigate/repro. Either way, the next release should address this. Check back a bit later or follow me on Github or Twitter to be notified when the new release is out.

@jimcottrell
Copy link
Author

Sounds good, thanks. I'll see how the next version works out. We're still having some build issues, but it looks like if we set a DOCKER_API_VERSION environment variable, we can at least get past this error.

@jimcottrell
Copy link
Author

For reference, other versions may work (1.44 did not), but we seem to be back in business with

export DOCKER_API_VERSION='1.25'

in our pipeline

@kcq
Copy link
Member

kcq commented Apr 6, 2024

@jimcottrell this might work as-is https://github.com/mintoolkit/mint/releases/tag/1.41.1 let me know if you still have problems there

@jimcottrell
Copy link
Author

I've tried with v1.41.1 and no explicit DOCKER_API_VERSION and I get that same error: client version 1.12 is too old

@kcq
Copy link
Member

kcq commented Apr 26, 2024

@jimcottrell which version of Docker do you have installed? Are you on Mac or Linux? If you are on Linux are you using Docker Desktop or only Docker Engine? If you are on a Mac is it an AMD or an M1/M2? Trying to repro... more info on your environment will help

@sammcj
Copy link

sammcj commented May 5, 2024

Getting this on Fedora 39 with the latest stable docker-ce and slim installed.

slim build --target nvapi --tag slim --http-probe=false --dockerfile Dockerfile --dockerfile-context .
...
cmd=build info=build.error status='standard.image.build.error' value='API /build requires version 1.25, requested version 1.24 is insufficient'
cmd=build state=exited code=33554435 version=linux/amd64|Transformer|1.40.11|1b271555882eacdfb4e6598d6d0552e9b9b1449b|2024-02-02_01:36:22PM location=/usr/local/bin
dnf info docker-ce-26.1.1-1.fc39.x86_64
Last metadata expiration check: 4:53:24 ago on Sun 05 May 2024 09:04:18.
Installed Packages
Name         : docker-ce
Epoch        : 3
Version      : 26.1.1
Release      : 1.fc39
Architecture : x86_64
Size         : 104 M
Source       : docker-ce-26.1.1-1.fc39.src.rpm
Repository   : @System
From repo    : docker-ce-stable
Summary      : The open-source application container engine
URL          : https://www.docker.com

Setting DOCKER_API_VERSION='1.25' makes no difference.

Seems to be a common problem looking at #637 and caused by #641

@kcq
Copy link
Member

kcq commented May 5, 2024

@sammcj do you get the same result with this version?

@kcq
Copy link
Member

kcq commented May 5, 2024

@sammcj Here's the Vagrant-based repro with Fedora 39 and Docker 26.1.1. You should be good with the release version in the repro config. The release version you shared in this issue is out of date.

@jimcottrell
Copy link
Author

@kcq Sorry for the delay in responding. As noted before, this is happening only on a self-hosted Bitbucket pipeline runner (linux/amd64). When I run locally, I have no issue. To make Slim work in the pipeline environment, it has to run via the docker:dind image.

Host Docker

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          25.0.2
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       fce6e0c
  Built:            Thu Feb  1 00:23:45 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker:dind

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:58 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.15
  GitCommit:        926c9586fe4a6236699318391cd44976a98e31f1
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Looking at this does seem to explain why I couldn't set it to use v1.44.

@kcq
Copy link
Member

kcq commented Jun 5, 2024

@kcq Sorry for the delay in responding. As noted before, this is happening only on a self-hosted Bitbucket pipeline runner (linux/amd64). When I run locally, I have no issue. To make Slim work in the pipeline environment, it has to run via the docker:dind image.

Host Docker

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          25.0.2
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       fce6e0c
  Built:            Thu Feb  1 00:23:45 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker:dind

+ docker version
Client:
 Version:           20.10.24
 API version:       1.41
 Go version:        go1.19.7
 Git commit:        297e128
 Built:             Tue Apr  4 18:17:06 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true
Server: Docker Engine - Community
 Engine:
  Version:          26.1.3
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       8e96db1
  Built:            Thu May 16 08:33:58 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.15
  GitCommit:        926c9586fe4a6236699318391cd44976a98e31f1
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Looking at this does seem to explain why I couldn't set it to use v1.44.

@jimcottrell Running with dind in Bitbucket runners makes it more complicated because each CI/CD vendor does the dind setup in slightly different ways, which requires different ways to connect to the (correct) Docker runtime. Some environments don't expose the Docker unix socket. Other environments need certs to connect to the Docker runtime. Some have more advanced network configurations that prevents direct container communication, so the --sensor-ipc-mode and --sensor-ipc-endpoint flags need to be used. The RUNNING CONTAINERIZED README section covers a few examples

To troubleshoot your setup we need two things to get started:

  1. How you invoke the slim command in your dind setup (if it's slim build --http-probe-off --continue-after=enter alpine then it needs to be updated to point to the dind Docker runtime. optionally you can try using the containerized version of the app, but that's not required).
  2. Your Bitbucket runner config that defines how the dind magic is configured.

@jimcottrell
Copy link
Author

Here's a simplified version of our bitbucket-pipelines.yml that fails with client version 1.12 is too old:

image:
  name: atlassian/default-image:4
definitions:
  services:
    privileged-docker:
      type: docker
      image:
        name: docker:dind
pipelines:
  custom:
    test:
      - step:
          runs-on:
            - linux
            - self.hosted
          name: Build Optimized Image
          services:
            - privileged-docker
          script:
            - curl https://downloads.dockerslim.com/releases/1.40.11/dist_linux.tar.gz -o slim.tar.gz
            - tar xvf slim.tar.gz
            #- export DOCKER_API_VERSION='1.25' # Temporary, see https://github.com/slimtoolkit/slim/issues/646
            - ./dist_linux/slim build --continue-after=enter alpine

I tried it with --in-container=true which didn't make any difference. What needs to be updated to point to the dind Docker runtime? Apart from this one case, I'm not very familiar with dind. The DOCKER_HOST environment variable exists, but I don't believe the Docker UNIX socket is present.

@schowave
Copy link

schowave commented Aug 1, 2024

Are there any updates here?

I am having the same problem with a gitlab pipeline and dind

@kcq
Copy link
Member

kcq commented Aug 1, 2024

The latest release, 1.41.5, shouldn't have the docker API problem.

To connect to the Docker daemon it might be necessary to extract the daemon IP address from the /etc/hosts file if the environment doesn't configure DOCKER_HOST on its own properly. Here's an example for Gitlab (need to double check if it still works): docker run -e DOCKER_HOST=tcp://$(grep docker /etc/hosts | cut -f1):2375 mintoolkit/mint slim your-docker-image-name

Might need to use the --sensor-ipc-mode and --sensor-ipc-endpoint flags too for dind depending on the CI/CD vendor.

I'll repro it with Gitlab first to confirm that the setup is still ok. Then I'll try to repro with a Bitbucket setup.

@schowave
Copy link

schowave commented Aug 2, 2024

Okay, than the installation routine should be adapted:
I installed it via:

curl -sL https://raw.githubusercontent.com/slimtoolkit/slim/master/scripts/install-slim.sh | bash -
and in the script, the latest version is:

curl -sL https://api.github.com/repos/slimtoolkit/slim/releases | grep tag_name | head -n1 | cut -d'"' -f4
1.40.11

@kcq
Copy link
Member

kcq commented Aug 2, 2024

Okay, than the installation routine should be adapted: I installed it via:

curl -sL https://raw.githubusercontent.com/slimtoolkit/slim/master/scripts/install-slim.sh | bash - and in the script, the latest version is:

curl -sL https://api.github.com/repos/slimtoolkit/slim/releases | grep tag_name | head -n1 | cut -d'"' -f4 1.40.11

should be this install script

@marcoaversanoigt
Copy link

marcoaversanoigt commented Nov 29, 2024

Hello @kcq !
I had same problem with latest mint 1.41.7 (installed with this install script), Docker latest (27.3.1), Docker API v1.47, but the only solution was to set explicitely => DOCKER_API_VERSION=1.47 (on the Github workflow running on a self-hosted runner)

@kcq
Copy link
Member

kcq commented Nov 29, 2024

Hello @kcq !
I had same problem with latest mint 1.41.7, Docker latest (27.3.1), Docker API v1.47, and the only solution was to set explicitely => DOCKER_API_VERSION=1.47

What image did you try to slim or was it a different command? Do you have a repro?

@marcoaversanoigt
Copy link

marcoaversanoigt commented Nov 29, 2024

Hello @kcq !
I had same problem with latest mint 1.41.7, Docker latest (27.3.1), Docker API v1.47, and the only solution was to set explicitely => DOCKER_API_VERSION=1.47

What image did you try to slim or was it a different command? Do you have a repro?

It is a custom internal image, already created.

Found just now that i'm forced to use the 'export' =>

export DOCKER_API_VERSION=1.47
mint slim [...]

or
to put the variable definition inline =>
DOCKER_API_VERSION=1.47 mint slim [...]

While if i do:

DOCKER_API_VERSION=1.47
mint slim [...]

It says again client API 1.12 is too old.

@kcq
Copy link
Member

kcq commented Dec 3, 2024

Thank you for the additional context @marcoaversanoigt ! This will help with my repro. We have the environment and the command context that should be enough. Working on the repro (hopefully can repro it without the Github workflow part :-))

@KarimABOUSSELHAM
Copy link

Hello guys.
I was running through the same issue discussed here a few days ago. I started the resolution it by reinstalling dockerd then I ensured that docker slim installed version matches with my OS/WSL distro then the famous command slim build <image-name> hopefully worked.

@Seluj78
Copy link

Seluj78 commented Dec 11, 2024

For anyone confused like me regarding the updates, version problem or whatever, mint is deprecated, you now need to install mint: https://github.com/mintoolkit/mint

@kcq
Copy link
Member

kcq commented Dec 16, 2024

For anyone confused like me regarding the updates, version problem or whatever, mint is deprecated, you now need to install mint: https://github.com/mintoolkit/mint

Yes, the latest enhancements are there. There's a new release coming soon too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants