Skip to content

support or disable direct unpack when running in userns #3762

Closed
#3763
Closed
support or disable direct unpack when running in userns#3762
#3763
Labels
kind/bug

Description

@AkihiroSuda
AkihiroSuda
opened on Oct 19, 2019

Description

Direct unpack added in #3528 (v1.3.0) doesn't work well when running in user namespace:

E1019 19:24:27.024977   21918 remote_runtime.go:200] CreateContainer in sandbox "941524a9dca443253b954cc64183131f5cc89054cd0dd70f4ab342671cd1bf1f" from runtime service failed: rpc error: code = Unknown desc = failed to create containerd container: error unpacking image: failed to extract layer sha256:d635f458a6f8a4f3dd57a597591ab8977588a5a477e0a68027d18612a248906f: failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not permitted: unknown
E1019 19:24:27.025143   21918 kuberuntime_manager.go:783] container start failed: CreateContainerError: failed to create containerd container: error unpacking image: failed to extract layer sha256:d635f458a6f8a4f3dd57a597591ab8977588a5a477e0a68027d18612a248906f: failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not permitted: unknown

The error happens because whiteout cannot be created with mknod in user namespace.

Moby implementation supports creating whiteout in a hacky alternative way:

https://github.com/moby/moby/blob/a8b04b17fd37ed797e34bea6534d307929c6337b/pkg/archive/archive_linux.go#L122-L175

// mknodChar0Overlay creates 0,0 char device by mounting overlayfs and unlinking.
// This function can be used for creating 0,0 char device in userns on Ubuntu.
//
// Steps:
// * Mkdir lower,upper,merged,work
// * Create lower/dummy
// * Mount overlayfs
// * Unlink merged/dummy
// * Unmount overlayfs
// * Make sure a 0,0 char device is created as upper/dummy
// * Rename upper/dummy to cleansedOriginalPath

Steps to reproduce the issue:

  • Run containerd/cri in rootless mode

Describe the results you received:

failed to convert whiteout file "etc/ca-certificates/.wh..wh..opq": operation not permitted: unknown

Describe the results you expected:

It should either support or disable direct unpack

Output of containerd --version:
1.3.0

Activity

  1. AkihiroSuda
    added
    kind/bug
    on Oct 19, 2019

  2. AkihiroSuda commented on Oct 19, 2019

    @AkihiroSuda
    AkihiroSuda
    on Oct 19, 2019
    MemberAuthor

    PR to disable direct unpack: #3763

  3. estesp
    closed this as completedin #3763on Oct 24, 2019
  4. AkihiroSuda
    added a commit that references this issue on Oct 25, 2019

    apply: use naive applier when running in UserNS

    f3c48da
  5. dmcgowan
    added a commit that references this issue on Nov 19, 2019

    Prepare v1.3.1 release

    d512382

  6. errordeveloper commented on Apr 3, 2020

    @errordeveloper
    errordeveloper
    on Apr 3, 2020
    Contributor

    In case anyone else reads this, I faced same issue with containerd 1.3.3 running in kata pod, turned out it was to do with /var/lib being on a 9p filesystem, which doesn't support mknod. For my use-case It was sufficient for to set root = /run/containerd/root and state = /run/containerd/state in the containerd config file, as /run is a tmpfs and I also got some performance improvements on image pulls that way (I do have plenty of memory to spare, at least at the moment).

  7. tussennet
    added a commit that references this issue on Sep 11, 2020

    apply: use naive applier when running in UserNS

    55f8fe7

  8. liiri commented on Sep 19, 2021

    @liiri
    liiri
    on Sep 19, 2021

    In case anyone else reads this, I faced same issue with containerd 1.3.3 running in kata pod, turned out it was to do with /var/lib being on a 9p filesystem, which doesn't support mknod. For my use-case It was sufficient for to set root = /run/containerd/root and state = /run/containerd/state in the containerd config file, as /run is a tmpfs and I also got some performance improvements on image pulls that way (I do have plenty of memory to spare, at least at the moment).

    I don't think you should use root on a tmpfs -- your state won't get saved on machine restart and will cause unexpected behavior when trying to start containerd service

  9. mathis-m commented on Nov 23, 2022

    @mathis-m
    mathis-m
    on Nov 23, 2022
    Contributor

    it fails in user namespace with containerd github.com/containerd/containerd v1.6.6 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1

    error

    FATA[0006] pulling image: rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/jupyterhub/configurable-http-proxy:4.5.3": failed to extract layer sha256:1be7bfe3945a9f69b537b3d4112245b7290837feadb68fca2214cc5f33a6386d: mount callback failed on /var/lib/containerd/tmpmounts/containerd-mount2456203030: failed to convert whiteout file "usr/local/lib/node_modules/.wh.npm": unlinkat /var/lib/containerd/tmpmounts/containerd-mount2456203030/usr/local/lib/node_modules/npm/node_modules/yallist: input/output error: unknown

    @AkihiroSuda how can I get this to work within the user namespace?

  10. AkihiroSuda commented on Nov 23, 2022

    @AkihiroSuda
    AkihiroSuda
    on Nov 23, 2022
    MemberAuthor

    @mathis-m Could you open a new issue with details (minimum repro steps, host kernel version, filesystem, etc.)

  11. errordeveloper
    added a commit that references this issue on May 21, 2025

    Persist containerd data and use default path

    089ad6f
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    kind/bug

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions