storage

package
v2.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 31, 2026 License: MIT Imports: 23 Imported by: 0

Documentation

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsNotFound 

func IsNotFound(err error) bool

IsNotFound returns true if err represents a "not found / 404" error from any backend.

func WithEndpoint 

func WithEndpoint(ctx context.Context, endpoint string) context.Context

WithEndpoint stores a custom endpoint in the context for S3-compatible backends.

Types

type AzureClient 

type AzureClient struct {
	// contains filtered or unexported fields
}

AzureClient implements Backend using Azure Blob Storage.

func (*AzureClient) CopyObject 

func (c *AzureClient) CopyObject(ctx context.Context, bucket, srcKey, destKey string) error

func (*AzureClient) DeleteObjects 

func (c *AzureClient) DeleteObjects(ctx context.Context, bucket string, keys []string) error

func (*AzureClient) DownloadDirectory 

func (c *AzureClient) DownloadDirectory(ctx context.Context, bucket, prefix, destDir string) error

func (*AzureClient) DownloadObjectToFile 

func (c *AzureClient) DownloadObjectToFile(ctx context.Context, bucket, key, localPath string) error

func (*AzureClient) GetObject 

func (c *AzureClient) GetObject(ctx context.Context, bucket, key string) ([]byte, error)

func (*AzureClient) HeadObjectExists 

func (c *AzureClient) HeadObjectExists(ctx context.Context, bucket, key string) (bool, error)

func (*AzureClient) ListKeys 

func (c *AzureClient) ListKeys(ctx context.Context, bucket, prefix string) ([]string, error)

func (*AzureClient) ListObjectsWithMeta 

func (c *AzureClient) ListObjectsWithMeta(ctx context.Context, bucket, prefix string) ([]ObjectMeta, error)

func (*AzureClient) PutObject 

func (c *AzureClient) PutObject(ctx context.Context, bucket, key string, data []byte) error

func (*AzureClient) TouchObject 

func (c *AzureClient) TouchObject(ctx context.Context, bucket, key string) error

TouchObject refreshes the blob's Last-Modified time via a metadata set.

func (*AzureClient) UploadFile 

func (c *AzureClient) UploadFile(ctx context.Context, localPath, bucket, key string, sc StorageClass) error

type Backend 

type Backend interface {
	GetObject(ctx context.Context, bucket, key string) ([]byte, error)
	PutObject(ctx context.Context, bucket, key string, data []byte) error
	HeadObjectExists(ctx context.Context, bucket, key string) (bool, error)
	ListKeys(ctx context.Context, bucket, prefix string) ([]string, error)
	ListObjectsWithMeta(ctx context.Context, bucket, prefix string) ([]ObjectMeta, error)
	DeleteObjects(ctx context.Context, bucket string, keys []string) error
	UploadFile(ctx context.Context, localPath, bucket, key string, sc StorageClass) error
	DownloadObjectToFile(ctx context.Context, bucket, key, localPath string) error
	DownloadDirectory(ctx context.Context, bucket, prefix, destDir string) error
	CopyObject(ctx context.Context, bucket, srcKey, destKey string) error
	// TouchObject refreshes an object's last-modified time (used by push to keep
	// dedup-skipped blobs inside the GC grace window).
	TouchObject(ctx context.Context, bucket, key string) error
}

Backend is the storage interface for all s3lo operations. *Client (S3/S3-compatible), *LocalClient, *GCSClient, *AzureClient all implement it.

func NewBackendFromRef 

func NewBackendFromRef(ctx context.Context, ref string) (Backend, error)

NewBackendFromRef creates the appropriate backend based on the URL scheme.

  • local:// → LocalClient (filesystem)
  • gs:// → GCSClient (Google Cloud Storage)
  • az:// → AzureClient (Azure Blob Storage)
  • s3:// (default) → S3 Client; uses custom endpoint from context for S3-compatible backends

type Client 

type Client struct {
	// contains filtered or unexported fields
}

func NewS3Client 

func NewS3Client(ctx context.Context) (*Client, error)

NewS3Client creates an S3 client using the default AWS credentials chain.

func (*Client) ClientForBucket 

func (c *Client) ClientForBucket(ctx context.Context, bucket string) (*s3.Client, error)

ClientForBucket returns an S3 client configured for the bucket's region (public). Clients are cached per-region to avoid re-loading AWS config on every call. For S3-compatible backends (endpoint set), region detection is skipped and the pre-configured endpoint client is returned directly.

func (*Client) CopyObject 

func (c *Client) CopyObject(ctx context.Context, bucket, srcKey, destKey string) error

CopyObject copies an S3 object within the same bucket.

func (*Client) DeleteObjects 

func (c *Client) DeleteObjects(ctx context.Context, bucket string, keys []string) error

DeleteObjects deletes multiple S3 objects in batches of up to 1000.

func (*Client) DownloadDirectory 

func (c *Client) DownloadDirectory(ctx context.Context, bucket, prefix, destDir string) error

DownloadDirectory downloads all objects under prefix into destDir.

func (*Client) DownloadObjectToFile 

func (c *Client) DownloadObjectToFile(ctx context.Context, bucket, key, localPath string) error

DownloadObjectToFile downloads a single S3 object to a local file path.

func (*Client) GetObject 

func (c *Client) GetObject(ctx context.Context, bucket, key string) ([]byte, error)

GetObject downloads a single S3 object and returns its contents.

func (*Client) HeadObjectExists 

func (c *Client) HeadObjectExists(ctx context.Context, bucket, key string) (bool, error)

HeadObjectExists returns true if an S3 object exists, false if it doesn't (404). Returns an error for non-404 failures (permissions, throttling, network).

func (*Client) ListKeys 

func (c *Client) ListKeys(ctx context.Context, bucket, prefix string) ([]string, error)

ListKeys returns all S3 object keys under prefix.

func (*Client) ListObjectsWithMeta 

func (c *Client) ListObjectsWithMeta(ctx context.Context, bucket, prefix string) ([]ObjectMeta, error)

ListObjectsWithMeta returns all S3 objects under prefix with size and last-modified metadata.

func (*Client) PresignGetObject 

func (c *Client) PresignGetObject(ctx context.Context, bucket, key string, ttl time.Duration) (string, error)

PresignGetObject returns a presigned GET URL for the given object valid for ttl. This satisfies serve.Presigner without importing pkg/serve.

func (*Client) PutObject 

func (c *Client) PutObject(ctx context.Context, bucket, key string, data []byte) error

PutObject uploads raw bytes to an S3 key.

func (*Client) TouchObject 

func (c *Client) TouchObject(ctx context.Context, bucket, key string) error

TouchObject refreshes an object's LastModified timestamp by copying it onto itself with metadata replacement. Push calls this when it dedups against an already-present blob so the blob re-enters the GC grace window; otherwise an old blob newly referenced by an in-flight push could be reaped by a concurrent GC before the push writes its manifest.

func (*Client) UploadDirectory 

func (c *Client) UploadDirectory(ctx context.Context, localDir, bucket, prefix string) error

UploadDirectory uploads all files in localDir to bucket at prefix/ with Standard storage class.

func (*Client) UploadFile 

func (c *Client) UploadFile(ctx context.Context, localPath, bucket, key string, sc StorageClass) error

UploadFile uploads a single local file to a specific S3 key.

type GCSClient 

type GCSClient struct {
	// contains filtered or unexported fields
}

GCSClient implements Backend using Google Cloud Storage.

func (*GCSClient) CopyObject 

func (c *GCSClient) CopyObject(ctx context.Context, bucket, srcKey, destKey string) error

func (*GCSClient) DeleteObjects 

func (c *GCSClient) DeleteObjects(ctx context.Context, bucket string, keys []string) error

func (*GCSClient) DownloadDirectory 

func (c *GCSClient) DownloadDirectory(ctx context.Context, bucket, prefix, destDir string) error

func (*GCSClient) DownloadObjectToFile 

func (c *GCSClient) DownloadObjectToFile(ctx context.Context, bucket, key, localPath string) error

func (*GCSClient) GetObject 

func (c *GCSClient) GetObject(ctx context.Context, bucket, key string) ([]byte, error)

func (*GCSClient) HeadObjectExists 

func (c *GCSClient) HeadObjectExists(ctx context.Context, bucket, key string) (bool, error)

func (*GCSClient) ListKeys 

func (c *GCSClient) ListKeys(ctx context.Context, bucket, prefix string) ([]string, error)

func (*GCSClient) ListObjectsWithMeta 

func (c *GCSClient) ListObjectsWithMeta(ctx context.Context, bucket, prefix string) ([]ObjectMeta, error)

func (*GCSClient) PutObject 

func (c *GCSClient) PutObject(ctx context.Context, bucket, key string, data []byte) error

func (*GCSClient) TouchObject 

func (c *GCSClient) TouchObject(ctx context.Context, bucket, key string) error

TouchObject refreshes the object's update time by rewriting it onto itself.

func (*GCSClient) UploadFile 

func (c *GCSClient) UploadFile(ctx context.Context, localPath, bucket, key string, sc StorageClass) error

type LocalClient 

type LocalClient struct{}

LocalClient implements Backend using the local filesystem. The "bucket" parameter in every method is the storage root directory.

func NewLocalClient 

func NewLocalClient() *LocalClient

NewLocalClient returns a LocalClient. No configuration needed — the bucket path serves as the storage root on every method call.

func (*LocalClient) CopyObject 

func (c *LocalClient) CopyObject(_ context.Context, bucket, srcKey, destKey string) error

func (*LocalClient) DeleteObjects 

func (c *LocalClient) DeleteObjects(ctx context.Context, bucket string, keys []string) error

func (*LocalClient) DownloadDirectory 

func (c *LocalClient) DownloadDirectory(_ context.Context, bucket, prefix, destDir string) error

func (*LocalClient) DownloadObjectToFile 

func (c *LocalClient) DownloadObjectToFile(_ context.Context, bucket, key, localPath string) error

func (*LocalClient) GetObject 

func (c *LocalClient) GetObject(ctx context.Context, bucket, key string) ([]byte, error)

func (*LocalClient) HeadObjectExists 

func (c *LocalClient) HeadObjectExists(ctx context.Context, bucket, key string) (bool, error)

func (*LocalClient) ListKeys 

func (c *LocalClient) ListKeys(ctx context.Context, bucket, prefix string) ([]string, error)

func (*LocalClient) ListObjectsWithMeta 

func (c *LocalClient) ListObjectsWithMeta(ctx context.Context, bucket, prefix string) ([]ObjectMeta, error)

func (*LocalClient) PutObject 

func (c *LocalClient) PutObject(ctx context.Context, bucket, key string, data []byte) error

func (*LocalClient) TouchObject 

func (c *LocalClient) TouchObject(_ context.Context, bucket, key string) error

TouchObject updates the blob file's modification time to now.

func (*LocalClient) UploadFile 

func (c *LocalClient) UploadFile(_ context.Context, localPath, bucket, key string, _ StorageClass) error

type ObjectMeta 

type ObjectMeta struct {
	Key          string
	Size         int64
	LastModified time.Time
	StorageClass string
}

ObjectMeta holds key and metadata for an S3 object.

type StorageClass 

type StorageClass string

StorageClass controls blob upload tiering. Backends map this to their own concepts. 

const (
	StorageClassStandard           StorageClass = "STANDARD"
	StorageClassIntelligentTiering StorageClass = "INTELLIGENT_TIERING"
)

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.