Amazon S3 storage backend for @backloghq/termlog. Store termlog segment data in S3-compatible object stores (AWS S3, MinIO, Cloudflare R2, LocalStack).

npm install @backloghq/termlog @backloghq/termlog-s3

import { TermLog } from "@backloghq/termlog";
import { S3Backend } from "@backloghq/termlog-s3";
import { S3Client } from "@aws-sdk/client-s3";

const backend = new S3Backend({
  client: new S3Client({ region: "us-east-1" }),
  bucket: "my-bucket",
  prefix: "my-index/",
});

const index = await TermLog.open({ dir: "my-index", backend });
await index.add("doc-1", "hello world");
await index.add("doc-2", "hello termlog");

const results = await index.search("hello");
await index.close();

const backend = new S3Backend({
  client: new S3Client({
    region: "us-east-1",
    endpoint: "http://localhost:9000",
    forcePathStyle: true,
  }),
  bucket: "my-bucket",
  prefix: "my-index/",
});

new S3Backend({
  client: myS3Client,   // S3Client instance (required)
  bucket: "my-bucket",  // S3 bucket name (required)
  prefix: "my-index/",  // Key prefix — scope all objects under this prefix (optional)
});

S3 provides no distributed lock. You must ensure at most one writer per (bucket, prefix) combination at any given time. Multiple concurrent readers are safe.

createWriteStream uses S3 multipart upload (required for streaming segment writes). Parts are buffered at 5 MiB (S3 minimum part size). If the stream completes with zero bytes, the adapter falls back to a PutObject call (S3 rejects CompleteMultipartUpload with empty Parts).

Required IAM permissions for the multipart path: s3:CreateMultipartUpload, s3:UploadPart, s3:CompleteMultipartUpload, s3:AbortMultipartUpload.

Minimum required permissions:

{
  "Effect": "Allow",
  "Action": [
    "s3:GetObject",
    "s3:PutObject",
    "s3:DeleteObject",
    "s3:ListBucket",
    "s3:CreateMultipartUpload",
    "s3:UploadPart",
    "s3:CompleteMultipartUpload",
    "s3:AbortMultipartUpload"
  ],
  "Resource": [
    "arn:aws:s3:::my-bucket",
    "arn:aws:s3:::my-bucket/my-index/*"
  ]
}

Add a lifecycle rule to abort incomplete multipart uploads after 1 day to avoid storage charges from crashed writers:

{
  "Rules": [{
    "ID": "abort-incomplete-multipart",
    "Status": "Enabled",
    "Filter": { "Prefix": "my-index/" },
    "AbortIncompleteMultipartUpload": { "DaysAfterInitiation": 1 }
  }]
}

npm run build          # Compile TypeScript
npm run lint           # ESLint
npm test               # Run tests (in-memory mock S3, no AWS needed)
npm run test:coverage  # Tests with coverage
npm run test:integration  # Real S3 tests (requires S3_INTEGRATION=1 + credentials)

The CI integration job runs the full integration suite (including the 1500-object pagination stress test) against a MinIO container on every push — no AWS credentials, no cost. It runs after the unit-test matrix passes (needs: test).

To run against MinIO locally:

docker run -d --name minio -p 9000:9000 \
  -e MINIO_ROOT_USER=minioadmin -e MINIO_ROOT_PASSWORD=minioadmin \
  minio/minio server /data

AWS_ACCESS_KEY_ID=minioadmin AWS_SECRET_ACCESS_KEY=minioadmin \
  aws --endpoint-url http://localhost:9000 s3 mb s3://termlog-test

S3_INTEGRATION=1 S3_INTEGRATION_SLOW=1 \
  S3_TEST_BUCKET=termlog-test S3_TEST_ENDPOINT=http://localhost:9000 \
  npm run test:integration

To run against real AWS S3 (opt-in, costs money):

S3_INTEGRATION=1 S3_TEST_BUCKET=my-bucket S3_TEST_REGION=us-east-1 \
  AWS_PROFILE=my-profile npm run test:integration

MIT