This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

When you're stuck — either unable to fix a bug after 2-3 attempts, or tempted to work around the actual problem by redefining the objective — stop and ask the user for directions. Explain clearly what the specific problem is (e.g., "pac_clip_sum(UBIGINT, DOUBLE) has no matching overload — should I add a DOUBLE overload or cast?"). The user knows this codebase deeply and can often point you to the right solution in one sentence. Do not silently change the goal, declare something impossible, or add bloated workarounds without consulting first. We work as a team.

Always test your changes with real queries (e.g., TPC-H on sf1) before declaring success, not just unit tests. Unit tests with wide boolean thresholds can pass even when the code is fundamentally broken.

Never execute git commands that could lose code. Always ask the user for permission on those.

• New features must have tests. Ask the user whether to create a new test file or extend an existing one in test/sql/.
• Never remove a failing test to "fix" a failure. If a test fails, fix the underlying bug. Tests exist for a reason.
• Before implementing anything, search the existing codebase for similar patterns or solutions. Check if a helper function, utility, or prior approach already addresses the problem. Reuse before reinventing.
• Use helper functions. Factor shared logic into helpers rather than duplicating code. Check src/include/utils/ and existing helpers in the file you're editing.
• Never edit the duckdb/ submodule. The DuckDB source is read-only. All PAC logic lives in src/ and test/. If you need DuckDB internals, use the public API or ask the user.
• Keep the paper in mind. The PAC mechanism is described in SIMD-PAC-DB: Pretty Performant PAC Privacy. Refer to it for the theoretical foundations (noise calibration, mutual information bounds, counter semantics) before making changes to core aggregate logic.
• Add PRIVACY_DEBUG_PRINT statements at major code flow points (entry/exit of compilation phases, aggregate rewrites, clipping decisions). Use the existing PRIVACY_DEBUG_PRINT macro from src/include/privacy_debug.hpp — it's compiled out when PRIVACY_DEBUG is 0.

privacy is a DuckDB extension that automatically privatizes SQL aggregate queries. It supports four privacy mechanisms, selected by the single SET privacy_mode setting (the dp_* settings tune the chosen mechanism, they do not select it):

• pac (default): PAC Privacy — empirical MIA resistance. Maintains 64 parallel counters per aggregate (one per "world" bit), adds noise calibrated to the query variance across sub-samples. Provides theoretical mutual-information bounds. Not differential privacy.
• dp_standard: Global-sensitivity DP — pure ε-DP for ungrouped queries, (ε,δ)-DP for grouped ones (see partition selection below). Bounds each privacy unit's total contribution to a fixed constant (per-PU contribution clipping), so global sensitivity equals that constant — data-independent. On a single PU table this is per-row clipping (COUNT sensitivity = 1, SUM = dp_sum_bound); on a join it pre-aggregates per PU (grouping by the PU-adjacent FK), clips each PU's partial to dp_sum_bound/dp_count_bound, then sums — so a join with COUNT requires dp_count_bound. Laplace noise calibrated to bound/epsilon. Grouped queries apply private partition selection by default (noised COUNT(DISTINCT pu) ≥ τ, the same Wilson/Google mechanism as dp_sass): the released key set is data-dependent, so this reserves ε_η = ε/(c+1) and needs δ_η, making grouped dp_standard (ε,δ)-DP and requiring dp_delta; ungrouped dp_standard stays pure ε-DP.
• dp_elastic: Elastic sensitivity DP — formal (ε,δ)-DP at the row level (FLEX/Chorus neighbor = change one tuple). Same Laplace pipeline as dp_standard but uses per-row clipping and the smoothed join-frequency envelope (2·SES_β, β derived from ε,δ) for sensitivity. Requires dp_delta. Not user-level: ∏mf bounds a single-tuple change, not a whole PU's contribution — a user-level extension would need per-user statistics and a new sensitivity proof (future work). It follows FLEX's public/enumerated-partition model (no DP partition selection), unlike the user-level dp_standard/dp_sass. Its privacy_min_group_count gate is a raw (un-noised) admin support filter, not DP partition selection — under FLEX's public-partition assumption it's a utility convenience; making it row-level-DP-safe would need a noised τ with C_u = 1 (the row-level branch), deferred.
• dp_sass: Sample-and-aggregate. Computes 64 sample (lane) aggregates per privacy unit and releases either the median (dp_sass_release='median', default; smooth-sensitivity Laplace, (ε,δ)-DP, requires dp_delta) or the mean (dp_sass_release='average'; GUPT-style, pure ε-DP, no dp_delta). Per-PU contribution is clipped (dp_count_bound/dp_sum_bound); the cross-group bound dp_max_groups_contributed (C_u) enters each aggregate's budget. Grouped queries apply private partition selection automatically (per the paper's Algorithm 2): a noised COUNT(DISTINCT pu) is released only when ≥ τ = 1 − C_u·log(2−2(1−δ_η)^{1/C_u})/ε_η, reserving ε_η = ε/(c+1) and δ_η = δ/(c+1) so each of the c aggregates gets ε/((c+1)·C_u). Grouped dp_sass therefore requires dp_delta (the pure-ε average release is only valid ungrouped). The released value is clamped to a public output domain (dp_sass_count_output_bound, dp_sass_sum_output_bound, dp_sass_avg_lower/upper_bound, dp_sass_minmax_lower/upper_bound); with dp_sass_private_range=true (average release only) a fraction dp_sass_range_budget_fraction of ε privately estimates that range (pure-ε exp-mechanism quantiles). Supports COUNT, COUNT(DISTINCT), SUM, AVG, MIN, MAX.

dp_standard and dp_elastic share one code path (CompileDPLaplaceQuery in privacy_mechanisms.cpp); they differ only in how contributions are clipped and how sensitivity is derived (ClipAndComputeSensitivities). All four modes share the same DDL (PRIVACY_KEY, PRIVACY_LINK, PROTECTED) and rewrite aggregate plans transparently — users write normal SQL.

GEN=ninja make                # build (release)
make test                     # run all tests (~25 tests, ~2200 assertions)

# single test
build/release/test/unittest "test/sql/pac_sum.test"

# C++ unit tests (parser, traversal, compiler)
build/release/extension/privacy/pac_test_runner

Build outputs go to build/release/. DuckDB is a git submodule in duckdb/.

The optimizer hook runs in pre_optimize_function — BEFORE DuckDB's built-in optimizers (join order, filter pushdown, column lifetime). This means:

• plan->ResolveOperatorTypes() must be called before accessing LogicalProjection::types (they're empty in raw plans)
• WHERE filters are still separate FILTER nodes
• LIMIT is a separate root node
• DuckDB's optimizers run automatically on the transformed plan

1. Compatibility check (privacy_compatibility_check.cpp) — decides if query needs PAC rewrite; rejects unsupported patterns (window functions, protected columns in GROUP BY)
2. FK join injection (pac_bitslice_add_fkjoins.cpp) — adds missing joins to reach PU tables
3. Aggregate transformation (pac_bitslice_compiler.cpp → pac_expression_builder.cpp) — replaces SUM/COUNT/etc. with pac_noised_sum/pac_noised_count/etc., inserts pac_hash projections above scans
4. Categorical rewrite (pac_categorical_rewriter.cpp) — when PAC aggregates appear in filters/comparisons, converts to counter lists (LIST<FLOAT>) with pac_filter/pac_select terminals
5. AVG decomposition (pac_avg_rewriter.cpp) — rewrites pac_noised_avg into pac_noised_div(pac_sum, pac_count)
6. Clip rewrite (pac_expression_builder.cpp:RewriteClipAggregates) — when pac_clip_support is set, inserts lower aggregate for per-PU pre-aggregation with clipping

dp_standard and dp_elastic run the same phases below via CompileDPLaplaceQuery, differing only in ClipAndComputeSensitivities: dp_standard does per-PU contribution clipping (a per-PU pre-aggregation on joins) → sensitivity = the bound (pure ε-DP, no δ); dp_elastic does per-row clipping + smooth elastic sensitivity 2·SES_β ((ε,δ)-DP, needs δ).

1. Compatibility check (privacy_compatibility_check.cpp) — shared structural checks; PU columns can only appear inside aggregates
2. FK chain extraction (dp_elastic_compiler.cpp:ExtractFKChain) — validates a linear FK path to the PU; chain may be explicit (joins in the SQL) or implicit (derived from PRIVACY_LINK metadata when the PU table is absent)
3. AVG rewrite (dp_elastic_compiler.cpp:RewriteAvgAggregates) — each AVG(x) is replaced in-place by SUM(x) and a COUNT(*) is appended; the original FILTER (WHERE …) predicate is propagated to both
4. Clipping (dp_elastic_compiler.cpp:ClipSumInputs) — SUM (and AVG-derived SUM) arguments wrapped in greatest(least(v, C), -C) where C = dp_sum_bound
5. Sensitivity computation (dp_elastic_compiler.cpp:ComputeMfK + ComputeElasticSensitivity) — per-table max frequency; global ES = ∏ mf; smooth ES with β = ε/(2·ln(2/δ)) when δ > 0
6. Budget split (CompileDPElasticQuery) — with k user-visible aggregates each gets ε/k (scale × k); AVG components further split to ε/(2k) (scale × 2k)
7. Noise injection (dp_elastic_compiler.cpp:WrapAggregateWithLaplace) — projection above the aggregate adds dp_laplace_noise(agg, scale) per column, cast back to the original type
8. τ-thresholding (dp_elastic_compiler.cpp:ApplyGroupSuppression) — if privacy_min_group_count set, a filter above the noise projection drops groups where |noised| < threshold · √2 · scale (skipped for AVG positions, whose ratio is not a simple Laplace output)
9. AVG ratio projection (dp_elastic_compiler.cpp:WrapAvgRatioProjection) — inserted above the suppression filter (or directly above the noise projection if no suppression). Computes cast(noised_sum, DOUBLE) / greatest(cast(noised_count, DOUBLE), 1.0) for each AVG; non-AVG outputs pass through. ColumnBindingReplacer remaps upstream references (including any HAVING filter) so they read the ratio rather than the raw SUM

pac_noised_* is the fused version of pac_noised(pac_*()). The unfused form is used when expressions operate on counters (list_transform/lambdas in categorical queries).

• pac_hash is always computed in a Projection above the scan, never inside an aggregate
• Pre-computed bindings become stale after RewriteBottomUp — always re-compute at point of use
• Use binder.GenerateTableIndex() for new table indices, never manual tracking
• Always call ResolveOperatorTypes() after creating or modifying a LogicalAggregate
• pac_noised_sum on DECIMAL uses BindDecimalPacSum to dispatch by physical type and set return_type to DECIMAL(38, scale) — any new sum variant needs the same pattern

• src/core/privacy_optimizer.cpp — optimizer hook entry point; dispatches to PAC or DP-elastic based on privacy_mode
• src/compiler/pac_bitslice_compiler.cpp — PAC compilation orchestrator (CompilePacBitsliceQuery)
• src/compiler/dp_elastic_compiler.cpp — DP-elastic compilation (CompileDPElasticQuery, ExtractFKChain, ComputeMfK)
• src/query_processing/pac_expression_builder.cpp — aggregate modification, clip rewrite, expression binding
• src/query_processing/pac_plan_traversal.cpp — plan traversal utilities (FindAllAggregates, AggregateGroupsByPUKey, etc.)
• src/include/aggregates/pac_aggregate.hpp — PacBindData, noise calibration, p-tracking
• src/categorical/pac_categorical_rewriter.cpp — categorical query transformation (~1770 lines)
• src/metadata/privacy_compatibility_check.cpp — shared compatibility check for both modes

Set #define PRIVACY_DEBUG 1 in src/include/privacy_debug.hpp for stderr trace output. Use EXPLAIN to see the transformed plan.

-- Declare privacy structure (shared by both modes)
ALTER TABLE customer ADD PRIVACY_KEY (c_custkey);
ALTER TABLE customer SET PU;
ALTER TABLE orders ADD PRIVACY_LINK (o_custkey) REFERENCES customer (c_custkey);

-- PAC mode (default)
SET privacy_mode = 'pac';
SET pac_mi = 0;                      -- disable noise for testing
SET privacy_seed = 42;               -- reproducible results
SET pac_clip_support = 40;           -- enable clip rewrite with support threshold
SET privacy_min_group_count = 4;     -- suppress low-SNR cells

-- DP-standard mode (formal pure ε-DP, per-PU contribution clipping)
SET privacy_mode = 'dp_standard';
SET dp_epsilon = 1.0;
SET dp_sum_bound = 1000;             -- required for SUM: max total contribution per PU
SET dp_count_bound = 50;             -- required for COUNT over a join: max rows per PU
-- (no dp_delta: pure ε-DP)

-- DP-elastic mode (formal (ε,δ)-DP, smoothed sensitivity)
SET privacy_mode = 'dp_elastic';
SET dp_epsilon = 1.0;
SET dp_delta = 1e-6;
SET dp_sum_bound = 1000;             -- required for SUM: max abs value per PU
SET privacy_min_group_count = 4;     -- τ-thresholding for GROUP BY

-- DP-sass mode (sample-and-aggregate)
SET privacy_mode = 'dp_sass';
SET dp_epsilon = 1.0;
SET dp_count_bound = 50;             -- per-PU contribution clip (COUNT/AVG)
SET dp_max_groups_contributed = 10;  -- C_u: groups one PU may affect (grouped queries)
SET dp_sass_count_output_bound = 1000000;  -- public output domain (per agg kind)
SET dp_sass_release = 'median';      -- 'median' (needs dp_delta) or 'average' (GUPT, pure-ε)
SET dp_delta = 1e-6;                 -- required only for 'median'
-- GUPT mean release + private range estimation (pure-ε):
-- SET dp_sass_release = 'average'; SET dp_delta = NULL;
-- SET dp_sass_private_range = true; SET dp_sass_range_budget_fraction = 0.25;

The project uses clang-tidy with DuckDB's configuration (.clang-tidy). Key naming rules:

• Classes/Enums: CamelCase (e.g., PacClipSumIntState)
• Functions: CamelCase (e.g., GetLevel, AllocateLevel)
• Variables/parameters/members: lower_case (e.g., max_level_used, key_hash)
• Constants/static/constexpr: UPPER_CASE (e.g., PAC2_NUM_LEVELS, PAC2_LEVEL_SHIFT)
• Macros: UPPER_CASE (e.g., PRIVACY_DEBUG_PRINT)
• Typedefs: lower_case_t suffix (e.g., aggregate_update_t)

Other style rules (from .clang-format, based on LLVM):

• Tabs for indentation, width 4
• Column limit: 120
• Braces: same line as statement (K&R / Allman-attached)
• Pointers: right-aligned (int *ptr, not int* ptr)
• No short functions on single line
• Templates: always break after template<...>
• Long arguments: align after open bracket

Run make format-fix to auto-format. Formatting runs automatically via hook after edits.

Attack scripts live in attacks/. Results are documented in attacks/clip_attack_results.md.

bash attacks/clip_attack_test.sh 2>/dev/null     # main attack suite
bash attacks/clip_multirow_test.sh 2>/dev/null   # 20K small items test
bash attacks/clip_hardzero_stress.sh 2>/dev/null  # stress tests