simplecloud

simplecloud

A tiny Go package for reading and writing objects across different storage backends with a unified interface.

Installation

go get github.com/mtgban/simplecloud

Supported Backends

Usage

All backends implement the same interface:

type Reader interface {
    NewReader(context.Context, string) (io.ReadCloser, error)
}

type Writer interface {
    NewWriter(context.Context, string) (io.WriteCloser, error)
}

Reading from GCS

bucket, err := simplecloud.NewGCSClient(ctx, "service-account.json", "my-bucket")
if err != nil {
    log.Fatal(err)
}

reader, err := bucket.NewReader(ctx, "path/to/file.txt")
if err != nil {
    log.Fatal(err)
}
defer reader.Close()

data, err := io.ReadAll(reader)

Writing to B2

bucket, err := simplecloud.NewB2Client(ctx, accessKey, secretKey, "my-bucket")
if err != nil {
    log.Fatal(err)
}

writer, err := bucket.NewWriter(ctx, "path/to/file.txt")
if err != nil {
    log.Fatal(err)
}

_, err = writer.Write([]byte("hello world"))
if err != nil {
    writer.Close()
    log.Fatal(err)
}

if err := writer.Close(); err != nil {
    log.Fatal(err)  // important: Close() flushes to cloud storage
}

HTTP base paths

The base URL passed to NewHTTPBucket may include a path prefix, which is preserved on every request; the per-call path is joined onto it. For example, a base of https://host/v1 reading /data.json requests https://host/v1/data.json. Any credentials in the base URL are reused and redacted from error messages.

Transparent Compression

Use InitReader and InitWriter to automatically handle compressed files based on extension:

// Automatically decompresses .gz file
reader, err := simplecloud.InitReader(ctx, bucket, "data.json.gz")
if err != nil {
    log.Fatal(err)
}
defer reader.Close()
// reader yields decompressed data

// Automatically compresses to .xz
writer, err := simplecloud.InitWriter(ctx, bucket, "output.json.xz")
if err != nil {
    log.Fatal(err)
}
// writes are compressed before storage

Copying Between Backends

Copy files between any backends, with automatic compression/decompression:

src, _ := simplecloud.NewGCSClient(ctx, "sa.json", "source-bucket")
dst, _ := simplecloud.NewB2Client(ctx, key, secret, "dest-bucket")

// Copy and transcode: decompress gzip, recompress as xz
n, err := simplecloud.Copy(ctx, src, dst, "input.json.gz", "output.json.xz")
if err != nil {
    log.Fatal(err)
}
fmt.Printf("copied %d bytes\n", n)

If the transfer fails partway, Copy aborts the destination instead of closing it. This matters because Close() is what commits an object on every cloud backend — closing a failed transfer would publish a truncated object that later readers cannot distinguish from a good one. The FileBucket, B2Bucket, GCSBucket, and S3Bucket writers all support this via the Aborter interface; a custom destination that doesn't implement it is closed instead, and retains whatever was written.

Opening by URL

Open picks a backend from the path's scheme so callers don't have to construct one, applying transparent decompression via InitReader:

Backend-specific configuration — credentials, the HTTP client, endpoints, concurrency — is passed with functional options, so the package never sources credentials on its own:

r, err := simplecloud.Open(ctx, "b2://my-bucket/data/report.json.xz",
    simplecloud.WithB2Credentials(keyID, appKey),
    simplecloud.WithConcurrentDownloads(8),
)
if err != nil {
    log.Fatal(err)
}
defer r.Close()
// r yields decompressed data

// S3-compatible (e.g. Cloudflare R2):
r, err = simplecloud.Open(ctx, "s3://my-bucket/data/report.json.gz",
    simplecloud.WithS3Credentials(accessKey, secretKey),
    simplecloud.WithS3Endpoint("https://<account>.r2.cloudflarestorage.com"),
)

// A local path needs no options and no scheme:
r, err = simplecloud.Open(ctx, "/data/report.json.gz")

For any other scheme — or to control client lifecycle instead of the per-call client the s3 and gs schemes create — supply a resolver. It is consulted before the built-in schemes (so it can also override them); returning (nil, nil) falls through:

gcs, _ := simplecloud.NewGCSClient(ctx, "sa.json", "my-bucket") // closed by you
r, err := simplecloud.Open(ctx, "gs://my-bucket/data/report.json.gz",
    simplecloud.WithResolver(func(_ context.Context, scheme, host string) (simplecloud.Reader, error) {
        if scheme == "gs" {
            return gcs, nil
        }
        return nil, nil
    }),
)

Limitations

This is a lightweight helper, and some operations are not covered:

• No Delete API
• No retry logic or exponential backoff
• No ACL or permission management
• No multipart upload configuration
• Context cancellation doesn't interrupt local file operations
• Cloud clients aren't exposed for cleanup (create short-lived or manage externally)

For advanced use cases, use the underlying SDKs directly:

• cloud.google.com/go/storage
• github.com/Backblaze/blazer/b2
• github.com/aws/aws-sdk-go-v2

Listing

The three cloud backends implement the optional Lister interface. Listing is flat (no delimiter) and pages internally, so breaking out of the loop stops the requests:

bucket, err := simplecloud.NewB2Client(ctx, keyID, appKey, "my-bucket")
if err != nil {
    log.Fatal(err)
}

for obj, err := range bucket.List(ctx, "magic/") {
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println(obj.Key, obj.Size, obj.LastModified)
}

FileBucket and HTTPBucket do not implement it — HTTP has no listing operation. Size is the stored (compressed) size, and LastModified means slightly different things per backend; see SPECIFICATIONS.md §10.

Further documentation

• SPECIFICATIONS.md — the behaviour contract: path and key handling, compression, abort semantics, error wrapping, and what is explicitly not guaranteed.
• AGENTS.md — for anyone changing this repo: the required checks, and the invariants that must not be broken (several exist because breaking them corrupted production data).
• todo/ — known improvements that are deliberately not done yet, each with its reasoning and blockers.

License

MIT