ccflow-s3

ccflow models for S3

ccflow-s3 provides public, domain-neutral S3 and S3-compatible storage callable models for ccflow workflows. It provides client and credential configuration, object reads/writes, existence checks, listing, manifests, atomic publish behavior, an S3-backed cache adapter, a generic artifact store adapter, and Hydra config groups exposed through the lerna plugin entry point.

It is storage-focused. Dataset schemas, domain-specific semantics, private bucket conventions, and application storage conventions are not part of this package.

• Implemented: S3Config, S3Credentials, S3ClientCredentials, S3Provider, flexible S3Session with direct AWS fields or ccflow-etl key/secret credentials, S3Client, client configs for AWS, Backblaze B2, Hetzner Object Storage, Cloudflare R2, and custom S3-compatible stores, shared /credentials/s3/... configs, anonymous unsigned access, S3 operation contexts/results, object reads, binary/text/json/CSV/gzip decoding through ccflow-etl PayloadCodec, pyarrow-backed parquet hooks, object existence checks through S3ExistsContext, metadata reads through S3HeadContext, prefix listing through S3ListContext, paged prefix walks through S3PrefixWalkContext, object copies through S3CopyContext, explicit deletes through S3DeleteContext, additive S3WriteDataContext writes for byte/string/dict/list payloads, codec-backed CSV/gzip/parquet write encoding, read-write orchestration through S3ReadWriteContext, temp-key copy finalization for atomic-style publishes, JSON manifest writes through S3ObjectManifest, S3ArtifactStore for ccflow-etl artifact write/publish contracts, S3 cache adapter, cache=s3, and output=/outputs/s3 config groups, and local fake-S3 tests with botocore error semantics.
• Partial: write mode skips existing objects unless overwrite=True, atomic publishes leave the temporary object in place rather than deleting it by default, and parquet support depends on pyarrow through ccflow-etl.
• Missing: manifest readers, temporary-object cleanup policy, live provider smoke tests, and broader moto/botocore-stub integration coverage.

Select a client group alongside output=/outputs/s3 or cache=s3:

cc-etl +client=/clients/s3/cloudflare +output=/outputs/s3 output.bucket=my-bucket output.prefix=raw
cc-etl +client=/clients/s3/backblaze +cache=s3 cache.store.bucket=my-bucket cache.store.prefix=cache

Available client groups:

Available standalone credentials groups: