[skip-ci] Packit/TMT: Run gating tests #1960

lsm5 · 2023-04-03T14:45:54Z

This commit adds enables unit, validate and integration testing via TMT.

Tests have only been added for the x86_64 arch on Fedora and CentOS Stream as the tests depend on quay.io/libpod/skopeo_cidev which isn't multi-arch currently.

Items for followup:

enable system tests
enable aarch64 tests
sync test files from upstream to downstream fedora

plans/main.fmf

github-actions · 2023-05-04T00:00:44Z

A friendly reminder that this PR had no activity for 30 days.

github-actions · 2023-06-04T00:00:49Z

A friendly reminder that this PR had no activity for 30 days.

github-actions · 2023-07-06T00:01:06Z

A friendly reminder that this PR had no activity for 30 days.

mtrmac · 2023-07-07T02:24:28Z

@lsm5 are you still working on this?

(If so, this is not urgent from my POV, no need to push it to the front of your queue — I’d just like to close the PR if there is absolutely no chance this work will continue.)

lsm5 · 2023-07-07T13:16:47Z

@mtrmac I do plan to work on this. But may take a while. I have bookmarked it for myself so I don't lose track. So please feel free to close it.

mtrmac · 2023-07-07T15:02:52Z

Thanks, if it is still relevant, let’s keep it open. It might help, or inspire, other contributors.

github-actions · 2023-08-07T00:00:55Z

A friendly reminder that this PR had no activity for 30 days.

github-actions · 2023-09-07T00:00:47Z

A friendly reminder that this PR had no activity for 30 days.

lsm5 · 2023-10-11T14:33:17Z

@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running test-system-local ? Asking mainly RE: packit + tmt tests.

mtrmac · 2023-10-11T14:43:31Z

@edsantiago is authoritative for system tests (question above).

edsantiago · 2023-10-11T14:50:08Z

Forgive me please, I don't understand the question. It's been a while since I've looked at skopeo gating tests, but my recollection is that they run on a plain system, via dnf install skopeo-tests. What's the context for "in a container"?

mtrmac · 2023-10-11T14:51:46Z

@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running test-system-local ?

IIRC the system tests run a registry server (container). So… running -local would be much simpler in that there would be no need for nested Podman; OTOH the tests would be much less isolated from the surrounding environment, which might be a difficulty.

I’m not immediately sure how much custom tooling exists in the CI container, and would need to be reproduced; https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh is relevant to that, but IIRC almost all of that exists for integration, not system, tests.

thrix · 2023-10-11T15:36:12Z

/packit test

thrix · 2023-10-11T15:57:16Z

So, unfortunately, the issue we are hitting is a bug in Packit:

packit/packit-service#2028

It was not considered a bug before, as people were not hitting it and we did not know it had this consequence.
They are working on a fix.

lsm5 · 2023-10-11T17:52:38Z

So, unfortunately, the issue we are hitting is a bug in Packit:

packit/packit-service#2028

It was not considered a bug before, as people were not hitting it and we did not know it had this consequence. They are working on a fix.

Thanks @thrix . I'll disable the rhel tests for now.

lsm5 · 2023-10-11T17:53:53Z

@mtrmac @vrothberg @cevich RE: system tests in CI, is it critical to run them in a container or would we be ok with running test-system-local ?

IIRC the system tests run a registry server (container). So… running -local would be much simpler in that there would be no need for nested Podman; OTOH the tests would be much less isolated from the surrounding environment, which might be a difficulty.

I’m not immediately sure how much custom tooling exists in the CI container, and would need to be reproduced; https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh is relevant to that, but IIRC almost all of that exists for integration, not system, tests.

ack, the test-system target seems to spawn some container too. Anyway, I'll try to mimic the cirrus setup in tmt. Thanks

lsm5 · 2024-09-25T12:59:57Z

@mtrmac PTAL. Only added x86_64 integration, validate and unit tests for now.

lsm5 · 2024-09-25T14:01:52Z

@mtrmac PTAL. Only added x86_64 integration, validate and unit tests for now.

If -local targets in CI tests are acceptable, that'd save us the pain of quay.io/libpod/skopeo_cidev image maintenance. Else, I'll look at making that image multiarch.

lsm5 · 2024-09-25T14:18:19Z

Hmm, and I guess running the tests in the skopeo_cidev container renders the multiple testing-farm- distro tests pointless.

mtrmac

I really don’t know what I’m doing here and how the things fit together — so a bunch of what I’m sure are stupid questions.

What is the longer-term plan? Is this going intended to be in addition to .cirrus.yml, or eventually replace it?

mtrmac · 2024-09-25T17:45:36Z

.packit.yaml

@@ -43,7 +43,7 @@ jobs:
    enable_net: true

  - job: copr_build
-    trigger: pull_request
+    trigger: ignore


Is this an intentional part of the PR? It’s not obvious to me that changes to tests should affect whether we test building.

mtrmac · 2024-09-25T17:51:33Z

hack/test-integration.sh


-make PREFIX=/usr install
+if [[ ! -f /usr/bin/skopeo ]]; then


AFAICS this introduces a risk that we are going to test some pre-existing system binary, not the one we just built (or, see elsewhere about SKOPEO_BINARY), and that it is not caller-determined for certain, but it depends on what happens to be in the environment. That ambiguity seems undesirable to me.

mtrmac · 2024-09-25T17:53:27Z

integration/tmt/main.fmf

+    - golang
+    - make
+    - podman
+    - skopeo


Is this requiring some other Skopeo binary than the one we are testing? Or does that guarantee to use the one from copr_build?

… and if this is the one we should be testing, how does that /usr/bin/skopeo get injected into the test-integration container? That doesn’t happen AFAICS.

mtrmac · 2024-09-25T17:54:07Z

rpm/skopeo.spec

@@ -71,7 +71,7 @@ BuildRequires: ostree-devel
 BuildRequires: glib2-devel
 BuildRequires: make
 BuildRequires: shadow-utils-subid-devel
-Requires: containers-common >= 4:1-21
+Requires: containers-common


How is this related to the rest of the PR?

mtrmac · 2024-09-25T17:55:03Z

.packit.yaml

@@ -31,7 +31,7 @@ jobs:
    notifications: &copr_build_failure_notification
      failure_comment:
        message: "Ephemeral COPR build failed. @containers/packit-build please check."
-    targets:
+    targets: &fedora_copr_targets


This, and centos_copr_targets, seem to be unused. Is that intentional?

mtrmac · 2024-09-25T17:57:09Z

.packit.yaml

+      failure_comment:
+        message: "Tests failed. @containers/packit-build please check."
+    tmt_plan: /plans/upstream
+    targets:


Note to self: Per https://packit.dev/docs/configuration/upstream/tests#required-parameters , targets which don’t specify an architecture imply x86_64 when used in tests.

Maybe worth a comment in the file? Especially when it differs from the build job semantics.

mtrmac · 2024-09-25T18:07:20Z

Makefile

+ifdef SKOPEO_BINARY
+$(info Skipping build as SKOPEO_BINARY is specified)
+test-integration-local:
+else
 test-integration-local: bin/skopeo
+endif


This suggests that SKOPEO_BINARY might be set. (I can’t see anything setting it. for integration tests.)

But the Go test code invoked by go test just assumes Skopeo is found in PATH, and noting sets up SKOPEO_BINARY to be handled.

How is this intended to work?

mtrmac · 2024-09-25T18:21:18Z

If -local targets in CI tests are acceptable, that'd save us the pain of quay.io/libpod/skopeo_cidev image maintenance. Else, I'll look at making that image multiarch.

From an extremely brief look at the Packit / TMT documentation, the tests seem to run in an isolated VM (chroot? container?) and install RPM test subjects into that. Running an extra level of isolation in a container seems unnecessary in that case. (Also, to my knowledge no-one has been using the $CONTAINER_RUN targets for some time, they might have been broken for months without anyone noticing.) So, using the -local targets does seem better to me.

Also, testing the binary, if it comes from a RPM, in its expected distribution, seems clearly more valuable than running it in some other frozen / differently-built container.

OTOH, at least one reason why the container exists, and is regularly rebuilt, is so that individual Skopeo CI runs don’t need to install and compile all of those tools, which rarely change, on every single CI RUN. Hence, in Cirrus, runner.sh uses the -local targets but fetches pre-built binaries from that container.

If the Packit tests are ever intended to be merge-blocking, something like that to shorten the CI time would be very valuable.

And, also, I’d prefer not to maintain the test tool setup code ( https://github.com/containers/automation_images/blob/main/skopeo_cidev/setup.sh and make tools ) in two duplicated copies, if that could be helped.

lsm5 · 2024-09-26T10:23:23Z

@mtrmac thanks a lot for all the comments. Setting back to draft while I work through them.

packit-as-a-service · 2024-12-20T13:55:23Z