WIP: OCI support for export and commit #619

alexlarsson · 2016-12-07T18:36:09Z

This series adds support to libostree to handle OCI repositories, and then adds support for the --oci argument the export and commit commands.

This allows you to export an ostree commit to an OCI image, and import single-layer OCI images into an ostree commit. Additionally, if the imported OCI image was exported from ostree the commit info like summary, body, timestamp and metadata are re-imported.

I'm putting this here not really as a request to pull it as-is, but rather as a starting point for a discussion on this.

This is a baseclass for json objects which can be used to marshal and demarshall the OCI json formats. OCI use all kinds of formats for the property names (mediaType, MemorySwap, diff_ids, os.version, etc) which makes the normal json-glib gproperty-based marshalling not work (due to property name canonicalization), and it has some other weird things, so we need a custom version.

This adds structs for all json objects and OstreeJson types for all toplevel file types, with all the properties in the official specs. However, not everything currently has accessors, only things that we will need right now.

This supports reading and writing to a local directory, and reading from a HTTP base uri.

This is useful if we already have an fd open to the archive (which may be an unlinked temp file which has no path).

This commits from a single-layer OCI image into an ostree branch.

cgwalters · 2016-12-07T19:07:13Z

Need something like this initially:

diff --git a/.redhat-ci.yml b/.redhat-ci.yml
index 6c170be..b2a2c10 100644
--- a/.redhat-ci.yml
+++ b/.redhat-ci.yml
@@ -6,6 +6,9 @@ branches:
 container:
     image: projectatomic/ostree-tester
 
+packages:
+  - pkgconfig(json-glib-1.0)
+
 env:
     CFLAGS: '-fsanitize=undefined'

cgwalters · 2016-12-07T19:28:54Z

Hum...do we need to expose json-glib as public API? Previously when dealing with JSON I've found it way easier to convert it to gvariant first thing, then define accessors for the variant.

alexlarsson · 2016-12-07T19:30:37Z

@cgwalters I don't see where it is exposed anywhere?

cgwalters · 2016-12-07T19:34:21Z

src/libostree/ostree-oci-registry.h

+
+
+_OSTREE_PUBLIC
+struct archive *ostree_oci_layer_writer_get_archive (OstreeOciLayerWriter  *self);


This will need to be #ifndef __GI_SCANNER__ or so.

alexlarsson · 2016-12-07T19:34:49Z

(I mean in a non -private header)

cgwalters · 2016-12-07T19:39:02Z

Ok, right, I see json-glib is only included in the -private.h header.

The scope of this is a lot larger than I thought - I didn't realize we were going to do downloading via HTTP. That said...it's clearly something we'll need to make this really usable.

Ok, so...how about making this a build-time option? That'd make it a bit easier to land in master while still allowing people who aren't using the functionality now to disable it, as well as avoiding committing to full API stability right away. Although, I guess flatpak will be doing what's in ot-builtin-commit.c internally, so it'd need this API? Hmm.

cgwalters · 2016-12-07T19:40:31Z

src/ostree/ot-builtin-commit.c

+
+  ostree_oci_registry_download_blob (registry, digest, cancellable,
+                                     download_layer_cb, &data);
+  g_main_loop_run (data.loop);


See #499 (comment)

alexlarsson · 2016-12-08T08:38:47Z

Initially I was hoping to be able to hide much more of this API in the private parts of ostree, which would allow it to change easier and would make it easier to directly expose things such as the json structs. However, the way ostree and libostree interact make this hard. Basically you would require a very highlevel "do everything oci" call in libostree, which doesn't fit very well with how things work in e.g. ot-builtins-commit.c.

Also, the OCI spec is still not 100% frozen which makes it tricky to freeze the API which refers to it.

So, I'm reconsidering where to put this code, and it seems like a better idea to put most of it in flatpak, maybe as a git submodule if something else wants to use it. Then we can put the minimal supporting code in libostree (for example ostree_repo_write_archive_fd_to_mtree, and maybe the mtree layer merging code).

giuseppe

a few more minor things:

src/libostree/ostree-repo-libarchive.c:895: Warning: OSTree: ostree_repo_write_archive_fd_to_mtree: unknown parameter 'archive' in documentation comment, should be 'fd'
src/libostree/ostree-json.h:43: Warning: OSTree: ostree_json_from_bytes: return value: Missing (transfer) annotation
src/libostree/ostree-json-oci.h:49: Warning: OSTree: ostree_oci_ref_get_urls: return value: Missing (transfer) annotation
src/libostree/ostree-json-oci.h:59: Warning: OSTree: ostree_oci_versioned_from_json: return value: Missing (transfer) annotation
src/libostree/ostree-json-oci.h:86: Warning: OSTree: ostree_oci_manifest_get_annotations: return value: Missing (transfer) annotation
src/libostree/ostree-oci-registry.h:59: Warning: OSTree: ostree_oci_registry_load_ref: return value: Missing (transfer) annotation
src/libostree/ostree-oci-registry.h:90: Warning: OSTree: ostree_oci_registry_store_json: return value: Missing (transfer) annotation
src/libostree/ostree-oci-registry.h:95: Warning: OSTree: ostree_oci_registry_load_versioned: return value: Missing (transfer) annotation
src/libostree/ostree-oci-registry.h:100: Warning: OSTree: ostree_oci_registry_write_layer: return value: Missing (transfer) annotation

Would it make sense to expose the remote registry part from the command line (even just for testing)?
Let's say I would like to retrieve busybox or alpine which are single layered, would that be possible?

giuseppe · 2016-12-09T13:44:26Z

src/libostree/ostree-oci-registry.c

+      self->tmp_fd = -1;
+    }
+
+  g_clear_object (&self->compressor);


g_clear_object (&self->compressor) should be moved after the if block with archive_write_free, as ostree_oci_layer_writer_compress can be still be used and self->compressor would be NULL

giuseppe · 2016-12-09T13:50:25Z

src/libostree/ostree-repo-libarchive.c


+/**
+ * ostree_repo_write_archive_fd_to_mtree:
+ * @self: An #OstreeRepo


s/archive/fd/

rh-atomic-bot · 2016-12-09T18:16:54Z

☔ The latest upstream changes (presumably 5aefe17) made this pull request unmergeable. Please resolve the merge conflicts.

cgwalters · 2016-12-10T15:46:30Z

In the short term, that makes sense to me. That sounds a bit like merging this approach with #443 plus the layer merging?

alexlarsson · 2016-12-12T10:16:08Z

@cgwalters Yeah, sounds like a plan.

openshift-ci-robot · 2021-04-12T21:19:18Z

@alexlarsson: PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci · 2023-06-29T14:44:03Z

@alexlarsson: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/sanity	`ca6d256`	link	true	`/test sanity`
ci/prow/fcos-e2e	`ca6d256`	link	true	`/test fcos-e2e`
ci/prow/images	`ca6d256`	link	true	`/test images`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

alexlarsson added 8 commits December 7, 2016 16:09

ostree-fetcher: Add g_autoptr support

c66bdc2

oci: Add OstreeJson objects for all OCI json types

f9cd0de

This adds structs for all json objects and OstreeJson types for all toplevel file types, with all the properties in the official specs. However, not everything currently has accessors, only things that we will need right now.

Add initial version of OstreeOciRegistry

eb37ed9

This supports reading and writing to a local directory, and reading from a HTTP base uri.

Add support for ostree export --oci -o OCIDIR REF

3f3ef42

Add ostree_repo_write_archive_fd_to_mtree

eca026f

This is useful if we already have an fd open to the archive (which may be an unlinked temp file which has no path).

Support ostree commit --oci

60a9892

This commits from a single-layer OCI image into an ostree branch.

Add tests for OCI export + commit

eb5776e

CI: require json-glib

bb9b508

fixup! Add tests for OCI export + commit

ca6d256

cgwalters reviewed Dec 7, 2016

View reviewed changes

giuseppe reviewed Dec 9, 2016

View reviewed changes

cgwalters mentioned this pull request Dec 16, 2016

ostree-union-fuse: new fuse filesystem to access multiple commits #491

Closed

cgwalters added the WIP label Jan 4, 2017

cgwalters mentioned this pull request Jan 17, 2017

pulling with --storage ostree doesn't enforce signatures projectatomic/atomic#829

Open

openshift-ci-robot added the needs-rebase label Apr 12, 2021



		_OSTREE_PUBLIC
		struct archive ostree_oci_layer_writer_get_archive (OstreeOciLayerWriter self);

WIP: OCI support for export and commit #619

Are you sure you want to change the base?

WIP: OCI support for export and commit #619

Uh oh!

Conversation

alexlarsson commented Dec 7, 2016

Uh oh!

cgwalters commented Dec 7, 2016

Uh oh!

cgwalters commented Dec 7, 2016

Uh oh!

alexlarsson commented Dec 7, 2016

Uh oh!

cgwalters Dec 7, 2016

Choose a reason for hiding this comment

Uh oh!

alexlarsson commented Dec 7, 2016

Uh oh!

cgwalters commented Dec 7, 2016

Uh oh!

cgwalters Dec 7, 2016

Choose a reason for hiding this comment

Uh oh!

alexlarsson commented Dec 8, 2016

Uh oh!

giuseppe left a comment

Choose a reason for hiding this comment

Uh oh!

giuseppe Dec 9, 2016

Choose a reason for hiding this comment

Uh oh!

giuseppe Dec 9, 2016

Choose a reason for hiding this comment

Uh oh!

rh-atomic-bot commented Dec 9, 2016

Uh oh!

cgwalters commented Dec 10, 2016

Uh oh!

alexlarsson commented Dec 12, 2016

Uh oh!

openshift-ci-robot commented Apr 12, 2021

Uh oh!

openshift-ci bot commented Jun 29, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants