Proxy cache with a non-registry upstream is accepted as healthy and returns HTML as blob content with HTTP 200 #23882

Expected behavior and actual behavior:

When a proxy-cache project's upstream is not actually an OCI registry, Harbor accepts the endpoint as healthy and then serves the upstream's HTML response to clients as blob content with HTTP 200, labelled with the requested digest.

Concretely, https://gallery.ecr.aws is the ECR Public web gallery, not the registry API (https://public.ecr.aws). It returns 200 with a 1,477-byte HTML page for any /v2/ path:

$ curl -sS -o /dev/null -w 'status=%{http_code} bytes=%{size_download} type=%{content_type}\n' https://gallery.ecr.aws/v2/
status=200 bytes=1477 type=text/html

There appear to be two distinct issues.

1. The endpoint is accepted as healthy. With Provider: Docker Registry, the health check pings /v2/ and accepts any 2xx without checking that the response is a registry API, so a non-registry URL saves successfully and is reported healthy. (With Provider: Aws ECR the ECR-specific ping correctly fails — only the generic provider path is affected.)

Expected: the /v2/ probe should require a registry API response — e.g. the Docker-Distribution-Api-Version: registry/2.0 header that the distribution spec mandates — and reject text/html.

2. A failed upstream fetch is returned as a 200 blob response and cached. On pull, harbor-core logs that the proxy fetch failed to unmarshal and falls back to local, yet the client still receives 200 with the 1,477 bytes of HTML as the blob body, carrying Docker-Content-Digest: <requested digest>. Harbor's own registry rejects the identical content moments later with 400 DIGEST_INVALID:

[WARNING] [/server/middleware/repoproxy/proxy.go:247]: Artifact: <project>/docker/library/nginx:latest, digest: is not found in proxy cache, fetch it from remote repo
[WARNING] [/server/middleware/repoproxy/proxy.go:254]: Proxy to remote failed, fallback to local repo, error: error unmarshalling content: invalid character '<' looking for beginning of value
[ERROR]   [/controller/proxy/controller.go:281]: error while putting blob to local repo, http status code: 400, body: {"errors":[{"code":"DIGEST_INVALID","message":"provided digest did not match uploaded content","detail":{"Digest":"sha256:3368aaa907ddb03b0a0215c2285e4906a5f2ed4c3c26512dffac53b5c76cd6fb","Reason":{}}}]}

The access log for the same request shows it served as a cacheable success:

"GET /v2/<project>/docker/library/nginx/blobs/sha256:3368aaa9... HTTP/1.1" 200 1477 ... cache_status=MISS bytes=1477 0.162 0.161

Expected: an unverified or unparseable upstream fetch should not be returned as a 200 blob response, and Harbor should not assert Docker-Content-Digest for content it has not validated.

Not a security issue as far as we can tell: clients reject the content (containerd validates size and digest), so no invalid bytes reach a running container, and configuring the upstream requires admin access. Filing publicly on that basis — happy to move it to private reporting if you disagree.

Steps to reproduce the problem:

Reproduced on demo.goharbor.io:

1. Administration → Registries → New Endpoint
   • Provider: Docker Registry (not Aws ECR)
   • Endpoint URL: https://gallery.ecr.aws
   • No credentials
   • Test Connection succeeds and the endpoint saves as healthy — issue 1
2. Create a proxy-cache project using that endpoint (e.g. test-gallery-aws)
3. Pull anything through it:

$ docker pull demo.goharbor.io/test-gallery-aws/docker/library/nginx:latest
5cfaf92d7823: Downloading [==================================================>]  1.477kB
failed commit on ref "unknown-sha256:5cfaf92d78230e264e178a1d0009c9dc5b89fdbff1f1c950271e3f6e2f2c82d5": commit failed: "unknown-sha256:5cfaf92d78230e264e178a1d0009c9dc5b89fdbff1f1c950271e3f6e2f2c82d5" failed size validation: 1477 != 105: failed precondition

Versions:

• harbor version: reproduced on demo.goharbor.io (public demo, 2026-09-09); also observed on 2.13.0. Code references below are against main @ aea94352b.
• docker engine version: 29.1.3
• docker-compose version: n/a (demo instance)

Additional context:

Code paths (line numbers at main @ aea94352b):

Issue 1 — the ping accepts any 2xx:

• src/pkg/registry/client.go:144 — Ping() issues GET <endpoint>/v2/ (buildPingURL, :724) and returns nil on success without inspecting the body, Content-Type, or Docker-Distribution-Api-Version.
• src/pkg/registry/client.go:652 — do() only rejects non-2xx (if resp.StatusCode < 200 || resp.StatusCode > 299), so 200 text/html passes.

Issue 2 — unverified upstream bytes served under the requested digest:

• src/controller/proxy/remote.go:92 — BlobReader() passes PullBlob through with no validation of content type or digest.
• src/controller/proxy/controller.go:285 — ProxyBlob() returns that reader straight to the caller. The only digest validation (putBlobToLocal, which produces the 400 DIGEST_INVALID) runs in a detached goroutine whose error is only logged:

size, bReader, err := rHelper.BlobReader(remoteRepo, art.Digest)
desc := distribution.Descriptor{Size: size, Digest: digest.Digest(art.Digest)}
go func() {
    err := c.putBlobToLocal(remoteRepo, art.Repository, desc, rHelper)
    if err != nil {
        log.Errorf("error while putting blob to local repo, %v", err)
    }
}()
return size, bReader, nil

This explains the ordering above: the client is served 200 roughly 259 ms before Harbor logs that the same bytes failed digest validation.

• src/server/middleware/repoproxy/proxy.go:158 — handleBlob() calls serveBlob(w, reader, size, art.Digest), and setHeaders (:418) sets Content-Length: size plus Docker-Content-Digest/ETag to the requested digest — asserting a digest for content never checked against it. A well-formed 200 with a matching Content-Length is precisely what a downstream cache will store.

The existing comment on serveBlob shows this hazard was already considered for truncated reads ("truncated upstream reads would be framed as complete chunked 200 responses that caches may store"). This is the same hazard by a different route: the response is complete, but is not the requested content.

Why the client error is confusing — the digest in it is the sha256 of Harbor's own 404 body. The manifest request returns:

HTTP/1.1 404 Not Found
Content-Length: 105
Content-Type: application/json; charset=utf-8

{"errors":[{"code":"NOT_FOUND","message":"repository test-gallery-aws/docker/library/nginx not found"}]}

containerd ingests that body as content under unknown-sha256:<hash of the body>, then receives the 1,477-byte HTML on the fetch — hence 1477 != 105. Because the repo name is embedded in the error text, the body length and therefore the digest differ per path, making it look as though several distinct blobs are involved. Verified across four cases:

(Body is {"errors":[{"code":"NOT_FOUND","message":"repository <repo> not found"}]} plus a trailing newline.) No real image blob exists at any point.

Note the upstream response here is complete and fast (0.162 s) — this is not a truncated or interrupted transfer, simply the wrong content returned with a success status.

Suggested fixes:

1. Validate the /v2/ ping response, not just its status — require Docker-Distribution-Api-Version: registry/2.0, or at least reject text/html, so a non-registry endpoint cannot save as healthy.
2. On the blob path, verify the stream against art.Digest rather than relying on a fire-and-forget goroutine — e.g. wrap the reader in a digest.Verifier — and do not assert Docker-Content-Digest for unverified content. Rejecting an upstream blob response whose Content-Type is not a blob media type would catch this case cheaply.
3. Treat an unmarshal failure on the upstream manifest response as a hard fetch failure, or otherwise ensure the local fallback cannot result in upstream bytes being served as blob content.