Handle 403 on missing-key HEAD checks

The new objectExists helper only treats 404/NotFound as “missing”, but HeadObject commonly returns 403 for a non-existent key when the caller lacks s3:ListBucket on the bucket. In that IAM setup, every first-time export key is treated as an error, so the cron logs a per-metric failure and never reaches the ClickHouse INSERT for that metric. This is a regression introduced by the preflight HEAD path and will block exports unless the runtime credentials are broadened.

Useful? React with 👍 / 👎.

but HeadObject commonly returns 403 for a non-existent key when the caller lacks s3:ListBucket on the bucket

This one will make sure that the cron has access, so we get a 404 if the key/object does not exist

Avoid defaulting S3 region to a fixed value

BILLING_EVENTS_S3_REGION now defaults to us-west-2, and exec() always builds new S3Client({ region }) before the preflight HeadObject check. In any deployment where the export bucket is in a different region and this env var is not explicitly set, the HEAD call can fail with PermanentRedirect/region mismatch, causing each metric export to error out before INSERT INTO FUNCTION s3 runs. This is a regression from the previous flow (which did not require an SDK-side S3 region), so the region should be derived reliably (or redirect-following enabled) instead of hard-coding a default.

Useful? React with 👍 / 👎.

Remove hard-coded fallback for billing export S3 region

Defaulting BILLING_EVENTS_S3_REGION to us-west-2 makes the new HeadObject preflight run against a fixed region whenever the env var is unset, and S3Client is not configured with followRegionRedirects. In deployments where the billing export bucket is in another region, the existence check can fail (e.g., redirect/region mismatch), so each metric is marked failed before the ClickHouse export runs. This is a regression introduced by adding the SDK-side preflight check.

Useful? React with 👍 / 👎.

P1: Handle S3's 403-on-missing-object case, or ensure the writer role also has s3:ListBucket. Right now a missing key can fail the export instead of being treated as absent.

Check if this issue is valid — if so, understand the root cause and fix it. At packages/metering/lib/crons/billingEventsS3Export.ts, line 226:

<comment>Handle S3's 403-on-missing-object case, or ensure the writer role also has `s3:ListBucket`. Right now a missing key can fail the export instead of being treated as absent.</comment>

<file context>
@@ -191,11 +211,25 @@ export async function exec(): Promise<void> {
+        return true;
+    } catch (err) {
+        const status = (err as { $metadata?: { httpStatusCode?: number }; name?: string }).$metadata?.httpStatusCode;
+        if (status === 404 || (err as Error).name === 'NotFound') {
+            return false;
+        }
</file context>

Permissions have been added here

Thanks for the update!

This should be an environment variable with a default?

I think it would be better to construct an s3 client outside of the function scope and skip the destroy.

See previous comments. I would only call destroy if we were shutting down.

Use writer role for preflight object checks

The new existence gate can block exports even when the configured writer role is valid, because HeadObject here runs with the metering service’s own AWS credentials, not the BILLING_EVENTS_S3_WRITER_ROLE_ARN used by ClickHouse for the actual INSERT INTO FUNCTION s3(...) write path. In environments where the service identity lacks s3:GetObject/s3:ListBucket on this bucket (while the writer role has access), every metric will fail at step=s3_check and never execute the export query. Fresh evidence in this commit is that the preflight check is done via s3.send(...) in app code, while the role ARN is only injected later in SQL for ClickHouse.

Useful? React with 👍 / 👎.

P2: Validate the cron expression before scheduling; an invalid env value can now break cron registration at startup.

Check if this issue is valid — if so, understand the root cause and fix it. At packages/metering/lib/crons/billingEventsS3Export.ts, line 160:

<comment>Validate the cron expression before scheduling; an invalid env value can now break cron registration at startup.</comment>

<file context>
@@ -157,7 +157,7 @@ export function billingEventsS3ExportCron(): void {
     }
 
-    cron.schedule(CRON_SCHEDULE, () => {
+    cron.schedule(cronSchedule, () => {
         exec().catch((err: unknown) => {
             logger.error('Cron tick failed unexpectedly', err);
</file context>

I think we should keep this the same as the others - define minutes instead of cron pattern - in fact there is already one there CRON_BILLING_EVENTS_S3_EXPORT_MINUTES? Why can't that be used?

We also should not have cron value by default - they are usually set to zero so they don't run in self-hosted environments - and then we provide a value in the environment.

Delay skip-if-exists until late-ingest window has passed

With the new deterministic key, objectExists short-circuits all subsequent exports for that (day, metric) after the first successful write, so any ClickHouse rows that arrive later are permanently dropped for billing. The previous implementation could absorb lag because it re-exported hourly to new keys, but this branch now makes correctness depend on the first run time while the scheduler still uses */${cronMinutes} (top-of-hour when set to 60), so a run near UTC midnight can lock in an incomplete snapshot.

Useful? React with 👍 / 👎.

P1: Pin this cron to UTC; otherwise the offset is evaluated in the host timezone and the export window can drift away from the intended UTC day boundary.

Check if this issue is valid — if so, understand the root cause and fix it. At packages/metering/lib/crons/billingEventsS3Export.ts, line 163:

<comment>Pin this cron to UTC; otherwise the offset is evaluated in the host timezone and the export window can drift away from the intended UTC day boundary.</comment>

<file context>
@@ -156,7 +157,10 @@ export function billingEventsS3ExportCron(): void {
+    // Hourly cron with a configurable minute-of-the-hour offset
+    // (CRON_BILLING_EVENTS_S3_EXPORT_OFFSET_MINUTES — default 0). The MINUTES var
+    // stays the kill switch (0 = disabled, anything > 0 enables).
+    cron.schedule(`${cronOffsetMinutes} * * * *`, () => {
         exec().catch((err: unknown) => {
             logger.error('Cron tick failed unexpectedly', err);
</file context>

Preserve the existing cron enablement env var

In deployments that currently enable this export with CRON_BILLING_EVENTS_S3_EXPORT_MINUTES=60, replacing that schema entry with the new CRON_BILLING_EVENTS_S3_HOURLY_EXPORT_MINUTE means the old variable is ignored by parseEnvs, the new value defaults to -1, and billingEventsS3ExportCron() returns before scheduling anything. Unless every environment is updated in lockstep with this app deploy, billing event S3 exports will silently stop; keep the old kill-switch/interval env or derive the new minute from it during the transition.

Useful? React with 👍 / 👎.

P1: Defaulting this cron to -1 makes the billing export stop running whenever the env var is unset. If the new schedule should be on by default, use 15 here and keep -1 only as an explicit opt-out.

Check if this issue is valid — if so, understand the root cause and fix it. At packages/utils/lib/environment/parse.ts, line 88:

<comment>Defaulting this cron to `-1` makes the billing export stop running whenever the env var is unset. If the new schedule should be on by default, use `15` here and keep `-1` only as an explicit opt-out.</comment>

<file context>
@@ -81,11 +81,11 @@ export const ENVS = z.object({
+    // cron fires on (0–59). -1 (default) disables the cron entirely. 15 gives
+    // ClickHouse a ~15min buffer to ingest the previous UTC day's tail before we
+    // snapshot it.
+    CRON_BILLING_EVENTS_S3_HOURLY_EXPORT_MINUTE: z.coerce.number().min(-1).max(59).optional().default(-1),
 
     // Persist
</file context>