OCI layout extensions #2633

vrothberg · 2024-11-14T12:07:37Z

Taken over from #2567 but:

Added tests for the three commits
Use reflinks when adding a file

vrothberg · 2024-11-14T12:07:46Z

In draft for now.

oci/layout/oci_dest.go

mtrmac

Just a quick skim; I’m afraid I didn’t read the added tests yet.

oci/layout/oci_dest.go

vrothberg · 2024-11-19T09:43:39Z

I'd love to have some explicit test condition on Linux to make sure that the reflinking works. We're using the function from c/storage which is e2e tested. The unit tests run on tmp, so we have to create the test dirs in $HOME ... not sure we should though.

mtrmac

Actual detailed review still pending, I’m afraid.

oci/layout/oci_dest.go

mtrmac

A full review now.

docs/containers-transports.5.md

oci/layout/oci_dest_test.go

oci/layout/oci_transport.go

mtrmac · 2024-12-03T16:34:28Z

oci/layout/oci_transport_test.go

@@ -148,11 +148,17 @@ func testParseReference(t *testing.T, fn func(string) (types.ImageReference, err
 		"relativepath",


Holding the line at “all of *_transport.go should have test coverage”, please port the two test cases added to TestGetManifestDescriptor from #1677 (and then see if more need to be added).

Isn't that already being tested now with fixtures/two_images_manifest ?

Tooling says that case ref.image != "" && ref.sourceIndex != -1: and case ref.sourceIndex != -1: is not covered by tests.

oci/layout/oci_transport_test.go

oci/layout/reader.go

oci/layout/reader_test.go

mtrmac · 2024-12-03T17:12:48Z

oci/layout/reader_test.go

+			require.Equal(t, test.names[i], ociRef.image)
+			if test.names[i] != "" {
+				require.True(t, strings.HasSuffix(res.Reference.StringWithinTransport(), test.names[i]))
+			}


else test that res.Reference is a sourceIndex reference with the right index?

Or, alternatively, use the returned reference to read a manifest and verify that it matches res.ManifestDescriptor.Digest.

I think we are doing that above, aren't we?

The test above verifies that res.ManifestDescriptor is valid, not that res.Reference is usable.

In particular, changing the index = manifestIndex line in reader.go to index = manifestIndex * 0, or … + 1, currently doesn’t fail tests.

oci/layout/oci_dest.go

mtrmac · 2024-12-03T17:22:48Z

I'd love to have some explicit test condition on Linux to make sure that the reflinking works.

…

The unit tests run on tmp, so we have to create the test dirs in $HOME ... not sure we should though.

Tests writing to non-test directories , and possibly leaving them around on aborts/failures, seems rather unexpected to me, some might consider it (or, at least, be worried that it is) hostile. So I’d prefer to leave this to e2e tests, maybe the future tests of the Podman artifact CLI.

(Just because I was looking it up, Google says reflinks don’t work on ext4. That’s not really a blocker, we could gate this behind a filesystem type check.)

Vendor containers/image/pull/2633 and update the code so we can start testing it. Signed-off-by: Valentin Rothberg <[email protected]>

Images in the index can now be referenced via the @sourceIndex syntax. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

The new API allows for listing all manifests in an OCI layout's index. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

Try to reflink the file and restort to copying it in case of failure. Also add an Options struct to be future proof. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

mtrmac · 2024-12-23T13:22:55Z

internal/reflink/reflink_linux.go

+#include <linux/fs.h>
+
+#ifndef FICLONE
+#define FICLONE		_IOW(0x94, 9, int)


x/sys/unix.FICLONE exists; if that works, we can avoid the CGo dependency.

oci/layout/oci_dest.go

mtrmac

Thanks! ACK overall.

Ideally, I’d prefer sharing the reflink call with c/storage, but the logic (usage of c/storage/pkg/pools requires c/storage to explicitly special-case CopyFileRange, which is otherwise automatic) is just different enough to make that awkward, and anyway it’s ~10 lines, so not worth worrying about.

mtrmac · 2024-12-23T13:31:35Z

internal/reflink/reflink_linux.go

+	"io"
+	"os"
+
+	"golang.org/x/sys/unix"


Please run go mod tidy to mark the package as a direct dependency.

mtrmac · 2024-12-23T13:36:08Z

oci/layout/oci_dest.go

 	blobFile.Close()
-	explicitClosed = true
+
+	*closed = true


(Nit: Move this one line above, please, to keep it immediately next to the actual .Close().)

mtrmac · 2024-12-23T13:37:26Z

oci/layout/oci_dest.go

@@ -300,6 +314,63 @@ func (d *ociImageDestination) CommitWithOptions(ctx context.Context, options pri
 	return os.WriteFile(d.ref.indexPath(), indexJSON, 0644)
 }

+// PutBlobFromLocalFileOptions is unused but may receive functionality in the future.
+type PutBlobFromLocalFileOptions struct{}


Suggested change

type PutBlobFromLocalFileOptions struct{}

type PutBlobFromLocalFileOption struct{}

please, multiple options can be independent items at the call site.

mtrmac · 2024-12-23T13:42:10Z

oci/layout/oci_dest.go

+	defer srcFile.Close()
+
+	// reflink.Copy will io.Copy() in case reflinking fails
+	err = reflink.Copy(srcFile, blobFile)


(Absolutely non-blocking, this is internal-only anyway, and it works just fine as is: Maybe the function could be named explicitly enough to make the comment unnecessary, something like ReflinkOrCopy. OTOH some linters gripe about reflink.Reflink….)

mtrmac · 2024-12-23T13:44:03Z

oci/layout/oci_dest.go

+		return "", -1, err
+	}
+
+	_, err = blobFile.Seek(0, 0)


Suggested change

_, err = blobFile.Seek(0, 0)

_, err = blobFile.Seek(0, io.SeekStart)

please

mtrmac · 2024-12-23T13:44:55Z

oci/layout/oci_dest.go

+	defer func() {
+		if !blobFileClosed {
+			blobFile.Close()
+		}


blobFile remains on the filesystem on failure; I think this needs the succeeded logic from PutBlobWithOptions.

mtrmac · 2024-12-23T13:53:30Z

oci/layout/oci_dest.go

+	return private.UploadedBlob{Digest: blobDigest, Size: size}, nil
+}
+
+func (d *ociImageDestination) blobFileSyncAndRename(blobFile *os.File, blobDigest digest.Digest, closed *bool) error {


Documenting the interaction of blobFile and *closed would be nice.

vrothberg mentioned this pull request Nov 14, 2024

WIP: oci/layout API extensions #2567

Closed

vrothberg commented Nov 14, 2024

View reviewed changes

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

mtrmac reviewed Nov 14, 2024

View reviewed changes

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

vrothberg force-pushed the layout-list branch 3 times, most recently from ae7457e to 234b45f Compare November 19, 2024 09:41

vrothberg marked this pull request as ready for review November 19, 2024 12:11

vrothberg force-pushed the layout-list branch from 234b45f to 6481647 Compare November 19, 2024 12:12

mtrmac reviewed Dec 2, 2024

View reviewed changes

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

oci/layout/oci_dest.go Outdated Show resolved Hide resolved

mtrmac reviewed Dec 3, 2024

View reviewed changes

vrothberg mentioned this pull request Dec 10, 2024

vendor c/image PR and update code baude/podman#40

Open

vrothberg added a commit to vrothberg/libpod that referenced this pull request Dec 10, 2024

vendor c/image PR and update code

ef6acd3

Vendor containers/image/pull/2633 and update the code so we can start testing it. Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg force-pushed the layout-list branch from 6481647 to ff76927 Compare December 18, 2024 14:17

mtrmac added 2 commits December 18, 2024 15:17

Add @sourceIndex syntax to oci/layout

b495713

Images in the index can now be referenced via the @sourceIndex syntax. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

Add oci/layout.List

786c896

The new API allows for listing all manifests in an OCI layout's index. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg force-pushed the layout-list branch from ff76927 to 30a132d Compare December 18, 2024 14:17

Add oci/layout.PutBlobFromLocalFile

8565739

Try to reflink the file and restort to copying it in case of failure. Also add an Options struct to be future proof. Signed-off-by: Miloslav Trmač <[email protected]> Signed-off-by: Valentin Rothberg <[email protected]>

vrothberg force-pushed the layout-list branch from 30a132d to 8565739 Compare December 18, 2024 14:22

mtrmac reviewed Dec 23, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

OCI layout extensions #2633

OCI layout extensions #2633

vrothberg commented Nov 14, 2024 •

edited

Loading

vrothberg commented Nov 14, 2024

mtrmac left a comment

vrothberg commented Nov 19, 2024

mtrmac left a comment

mtrmac left a comment

mtrmac Dec 3, 2024

vrothberg Dec 16, 2024

mtrmac Dec 23, 2024

mtrmac Dec 3, 2024

vrothberg Dec 17, 2024

mtrmac Dec 23, 2024 •

edited

Loading

mtrmac commented Dec 3, 2024

mtrmac Dec 23, 2024

mtrmac left a comment

mtrmac Dec 23, 2024

mtrmac Dec 23, 2024

mtrmac Dec 23, 2024

mtrmac Dec 23, 2024

mtrmac Dec 23, 2024

mtrmac Dec 23, 2024

mtrmac Dec 23, 2024

		@@ -148,11 +148,17 @@ func testParseReference(t *testing.T, fn func(string) (types.ImageReference, err
		"relativepath",

	type PutBlobFromLocalFileOptions struct{}
	type PutBlobFromLocalFileOption struct{}

	_, err = blobFile.Seek(0, 0)
	_, err = blobFile.Seek(0, io.SeekStart)

OCI layout extensions #2633

Are you sure you want to change the base?

OCI layout extensions #2633

Conversation

vrothberg commented Nov 14, 2024 • edited Loading

vrothberg commented Nov 14, 2024

mtrmac left a comment

Choose a reason for hiding this comment

vrothberg commented Nov 19, 2024

mtrmac left a comment

Choose a reason for hiding this comment

mtrmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mtrmac Dec 23, 2024 • edited Loading

Choose a reason for hiding this comment

mtrmac commented Dec 3, 2024

Choose a reason for hiding this comment

mtrmac left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vrothberg commented Nov 14, 2024 •

edited

Loading

mtrmac Dec 23, 2024 •

edited

Loading