grabber

A Go library for downloading files and directories from various sources using a URL string as input.

Grabber is an alternative to go-getter with a cleaner API and active development.

• Download files and directories from Git, Mercurial, S3, GCS, OCI, HTTP, and local filesystems
• Minimal downloads — one commit, one branch, no tags, and only the objects backing the subdirectory you asked for
• Programmatic credential injection — pass SSH keys, AWS credentials, GCP service account keys, OCI registry credentials, and HTTPS credentials via the Go API
• HTTPS credential matching — configure HTTPS credentials with git-style host/path matching, used automatically for Git and HTTP protocols
• SSH-to-HTTPS auto-transform — automatically convert SSH/SCP Git URLs to HTTPS (useful in CI environments without SSH key access)
• Custom TLS and proxying — trust extra CAs, present mutual-TLS client certificates (per host), and route through HTTP proxies (global or per host) for the HTTP, OCI, and Git protocols — all via functional options, no environment variables
• SSRF protection — outbound fetches are guarded against reaching loopback, link-local (cloud metadata), and private addresses by default; configurable and opt-out via WithSSRFProtection
• Pure Go — no system git or other CLI tools required (except hg for Mercurial; system git is used for credential helper support if available)
• Checksum verification — verify downloaded file integrity via URL query param (?checksum=sha256:abc...) or the explicit GrabWithSHA256Checksum() API
• Automatic archive extraction — downloaded archives are detected and extracted by extension
• Subdirectory support — use // in URLs to extract a subdirectory (e.g. github.com/user/repo//sub/dir)
• Protocol auto-detection — URLs are automatically routed to the right protocol based on hostname, scheme, and path
• Extensible — bring your own protocol implementations via WithProtocols()

Protocols are auto-detected from the URL.

Supported URL formats:

Auto-detected when:

• URL has .git suffix
• URL uses ssh:// or git:// scheme
• URL is SCP-style (git@host:user/repo)
• Host is a known Git provider: github.com, gitlab.com, bitbucket.org, codeberg.org, dev.azure.com, sr.ht

Query parameters:

• ref - branch, tag, or commit SHA to check out
• depth - shallow clone depth (e.g. ?depth=1)
• subdir - subdirectory to fetch, keeping the repository layout (see below)

Subdirectory support:

Two interchangeable spellings, both meaning "fetch only this directory":

Either way the repository layout is preserved, so the contents land at <dest>/modules/vpc. Keeping the layout is what lets several subdirectories of one repository be merged into a single destination tree. If both forms are given, // wins.

Minimal downloads:

grabber returns files, not a usable repository — it strips .git and never runs another git operation. So it always fetches the least it can, with no option to turn this off:

• A single commit (depth 1), only the branch needed, and no tags. On a mid-sized repo this alone is ~5x less data than a default clone.
• When a subdirectory is requested, only the objects backing that directory. Fetching internal/service/s3 out of terraform-provider-aws transfers ~4 MiB instead of ~236 MiB.

The exception is a commit hash ref, which keeps full history, all branches and all tags: the commit may not be reachable from the default branch tip, and resolving a short hash walks every ref.

Subdirectory selection follows git's cone-mode rules, matching what git sparse-checkout set <subdir> leaves in a working tree:

• The requested directory in full, recursively.
• Plus the files sitting directly in each directory along the path to it, starting at the repository root. So modules/vpc also brings any files directly in modules/ and at the root.
• Directories off that path are excluded entirely.
• With no subdirectory there is nothing to narrow to, so the whole repository is cloned. Every blob is needed anyway, and filtering them only to request them all back would cost an extra round trip for the same bytes.

A directory selected this way may contain references to paths outside it (e.g. a Terraform module with source = "../shared"). Those are not downloaded — request them as a second fetch into the same destination, which the preserved layout makes safe.

Narrowing to a subdirectory needs a remote that supports partial clone (--filter=blob:none) and can serve objects by hash — the same requirement the git binary's lazy fetch has. GitHub and GitLab both qualify. When a remote cannot (including local file:// remotes), grabber transparently falls back to a full clone, so the result is always correct and only the transfer size changes.

It is also skipped when WithGitRecurseSubmodules(true) is set, since submodules need a real working tree.

Scheme fallback: A clone is first attempted with the URL as given. On failure grabber falls back to the other scheme: an SSH/SCP URL falls back to its HTTPS equivalent, and an HTTPS/HTTP URL falls back to SSH when an SSH key is configured for the host (Azure DevOps SSH URLs are handled specially). WithGitSSHToHTTPS() forces the HTTPS form up front instead. Setting WithConnectProbeTimeout(d) makes an unreachable primary fail fast so the fallback is tried promptly rather than after a clone timeout.

Orphaned commit fallback: When ref is a commit SHA that the git protocol can't reach (e.g. an orphaned commit that is no longer reachable from any branch or tag), grabber falls back to downloading a tarball of that commit from the hosting platform's HTTP API. GitHub, GitLab, and Bitbucket are supported. Credentials are resolved from the same sources as clones (URL userinfo, configured HTTPS credentials, the git credential helper) and, failing those, from well-known API token environment variables: GH_TOKEN/GITHUB_TOKEN, GITLAB_TOKEN/GL_TOKEN, and BITBUCKET_TOKEN (unless WithNoSystemFallback() is set, which disables the environment-variable lookup). The result is a plain source snapshot with no .git directory. SSH keys cannot be used for this HTTP fallback, so SSH-only setups need HTTP credentials configured for private repositories.

Note: Mercurial support requires the hg CLI to be installed on the system.

Supported URL formats:

Auto-detected when:

• Host is a known Mercurial provider: bitbucket.org

Since Bitbucket also hosts Git repos (and Git has higher priority), use the hg:: prefix to force Mercurial: hg::https://bitbucket.org/user/repo

Query parameters:

• rev — revision, tag, or branch to check out (e.g. ?rev=v1.0.0)

Subdirectory support: Use // to specify a subdirectory: hg::bitbucket.org/user/repo//lib/core?rev=stable

Supported URL formats:

Auto-detected when (no s3:: prefix needed):

• URL uses s3:// scheme
• Hostname contains s3 and amazonaws.com

Keys ending in / are treated as directory prefixes - all objects under that prefix are downloaded.

Supported URL formats:

Auto-detected when:

• Hostname is storage.googleapis.com or storage.cloud.google.com
• Hostname ends with .storage.googleapis.com

Keys ending in / are treated as directory prefixes - all objects under that prefix are downloaded.

Supported URL formats:

Auto-detected when:

• URL uses oci:// scheme

Supported URL formats:

Auto-detected when:

• URL uses http:// or https:// scheme
• URL has no scheme (defaults to HTTPS)

HTTP is the lowest-priority protocol, so it acts as a fallback when no other protocol matches.

Supported URL formats:

Auto-detected when:

• URL uses file:// scheme
• URL is an absolute filesystem path
• URL starts with ./ or ../

If the source is a directory, all contents are copied recursively. If it's a file, it's copied as a single file (and may be auto-extracted if it's an archive).

Options are passed to grabber.New():

g := grabber.New(
    grabber.WithGitSSHKey(privateKey),
    grabber.WithAWSCredentials(keyID, secret, token, region),
)

When AWS/GCP credentials are not provided, the respective SDK default credential chains are used (env vars, shared config, IAM roles, etc.).

Git clones default to depth=1 (shallow) for performance, since go-git is slower than system git for full clones and full history is rarely needed. Commit hash refs (?ref=abc1234) automatically use a full clone so the commit is reachable. URL query parameters (?depth=1) override all defaults.

When grabber fetches a URL that an untrusted party controls (for example a Terraform module source in a CI run), the SSRF guard prevents it from reaching addresses that are only meant to be reachable internally. It is on by default at the Internal level — blocking loopback, RFC1918 private ranges, IPv6 ULA, link-local (including the 169.254.169.254 cloud metadata endpoint), and multicast.

Levels (WithSSRFProtection(level)):

Specific hosts can be allowlisted with WithSSRFAllowHosts(...), which accepts hostnames (matched case-insensitively), IP literals, and CIDR ranges; matching targets bypass the guard entirely. Use WithCustomSSRFProtection(func(net.IP) bool) for a bespoke policy. The guard works at two layers: a dialer check on the HTTP/OCI transports (which catches DNS rebinding and redirect-to-internal, since each dial is re-checked on the resolved IP), and a pre-fetch host check for Git and Mercurial (which use their own transports). s3/gcs only ever connect to fixed cloud endpoints, so they are not guarded. A configured proxy is exempt (the connection goes to the trusted proxy, not the target).

Note: the default blocks 127.0.0.1/localhost, so tests or tools that fetch from a local server must pass WithSSRFProtection(ssrf.None).

HTTPS credentials are matched using git-style semantics: host must match (case-insensitive), and if a path is specified it must be a prefix of the URL path. The most specific match (longest path prefix) wins, and among equally specific matches the first one configured wins.

Credentials for HTTP, Git-over-HTTPS, and OCI are resolved in this order: credentials embedded in the URL → a matching static credential (WithHTTPSCredential/WithOCICredentials) → the dynamic function from WithHTTPCredentialRequestFunction → the system git credential helper (unless disabled via WithNoSystemFallback). A source is only consulted once everything ahead of it has come up empty. The dynamic function receives the protocol, host, and path and returns a username/password (either may be nil) plus a boolean; returning false defers to the next source.

Git clones walk that whole order rather than stopping at the first match: a credential the remote rejects with a 401 is followed by the next matching one, then the dynamic function, then the system helper. One host often has several credentials configured with only some of them still valid, and this keeps a stale one from shadowing a working one. Any other failure (TLS, DNS, a missing ref) is returned immediately rather than retried against every credential. A complete user:password in the URL still wins outright, with no fallback.

g := grabber.New(
    // Matches any URL on github.com
    grabber.WithHTTPSCredential("github.com", "user", "token"),

    // Matches only URLs under github.com/my-org/... (takes priority over the above)
    grabber.WithHTTPSCredentialForPath("github.com", "/my-org", "org-user", "org-token"),
)

Credentials are applied automatically to both Git (HTTPS clones) and HTTP downloads. For Git, HTTPS credentials are checked after embedded URL credentials but before system git credential fill.

When WithGitSSHToHTTPS() is enabled, SSH and SCP-style Git URLs are automatically converted to HTTPS before cloning:

• git@github.com:user/repo.git → https://github.com/user/repo.git
• ssh://git@github.com/user/repo.git → https://github.com/user/repo.git

This is useful in CI environments where SSH keys are not available but HTTPS tokens are configured via WithHTTPSCredential().

When WithAutoExtract() is enabled (the default), downloaded files are automatically detected and extracted by extension:

Downloaded files can be verified against an expected checksum. This works for single-file downloads only (not directories).

Via URL query parameter:

// With explicit algorithm
err := g.Grab(ctx, "https://example.com/file.tar.gz?checksum=sha256:e3b0c44...", "./output")

// Without algorithm prefix — defaults to SHA-256
err := g.Grab(ctx, "https://example.com/file.tar.gz?checksum=e3b0c44...", "./output")

Via explicit API (recommended):

err := g.GrabWithSHA256Checksum(ctx, "https://example.com/file.tar.gz", "./output", "e3b0c44...")

When both a URL parameter and an explicit checksum are provided, the explicit one takes precedence.

The URL parameter supports other algorithms via the algo:hex format (e.g. ?checksum=sha512:cf83e13...). Supported algorithms: md5, sha1, sha256, sha512.

Checksum verification runs on the raw downloaded file, before archive extraction.

package main

import (
    "context"
    "log"

    "github.com/liamg/grabber"
)

func main() {
    g := grabber.New(
        grabber.WithGitSSHKey(privateKeyBytes),
    )

    // Clone a Git repo subdirectory
    err := g.Grab(context.Background(), "github.com/user/repo//modules/vpc?ref=v1.0.0", "./vpc")
    if err != nil {
        log.Fatal(err)
    }

    // Download from S3
    err = g.Grab(context.Background(), "s3.amazonaws.com/my-bucket/config.tar.gz", "./config")
    if err != nil {
        log.Fatal(err)
    }
}

A CLI tool is included for testing and quick downloads:

go install github.com/liamg/grabber/cmd/grabber@latest

# Download a file
grabber https://example.com/file.tar.gz ./output

# Clone a Git repo subdirectory
grabber github.com/user/repo//modules/vpc ./vpc

# Download with checksum verification
grabber -c e3b0c44298fc1c14... https://example.com/file.tar.gz ./output

# Copy a local file
grabber ./path/to/source ./destination

# Print version information
grabber --version

Run grabber --help for all available flags.

Release binaries are built with version metadata embedded via -ldflags, so grabber --version on a released binary reports the tag, commit, and build date. go install builds report dev.

Library:

go get github.com/liamg/grabber

CLI:

go install github.com/liamg/grabber/cmd/grabber@latest

Early development. API is not yet stable.

• Git credential helpers require system git — go-git doesn't support credential.helper from ~/.gitconfig, so grabber shells out to git credential fill when git is on PATH. Without system git, credential helpers won't work — use WithGitSSHKey(), WithHTTPSCredential(), or embed credentials in the URL instead. This shell-out (like the SSH agent and known_hosts defaults) is a system fallback and can be turned off entirely with WithNoSystemFallback().

Unlicense.