Skip to content

Dedup identical subtrees in Dcp2Cone canonicalization - #3355

Merged
PTNobel merged 16 commits into
masterfrom
ptn/dnlp2smooth-cse
Jun 18, 2026
Merged

PTNobel merged 16 commits into
masterfrom
ptn/dnlp2smooth-cse

Conversation

@PTNobel

@PTNobel PTNobel commented May 27, 2026

Copy link
Copy Markdown
Collaborator

Description

When the same Expression subtree appears in two places in a DCP problem, for example cp.norm1(x) in both the objective and a constraint, Dcp2Cone previously emitted a fresh set of auxiliary variables and epigraph constraints per occurrence. This PR adds a per-apply() structural-key cache so Dcp2Cone canonicalizes each structurally identical subtree once and reuses the canonical expression on later occurrences.

Worked example for the reported case:

cp.Problem(cp.Minimize(cp.norm1(x)), [cp.norm1(x) <= 1])

The canonicalized problem now has one epigraph variable t shared across the objective and the constraint, with one pair of t >= x, t >= -x inequalities.

The DNLP changes from earlier revisions were removed from this PR. The original DNLP work is preserved on branch ptn/dnlp2smooth-cse-dnlp-followup for follow-up issue discussion.

Issue link (if applicable):

Type of change

  • New feature (backwards compatible)
  • New feature (breaking API changes)
  • Bug fix
  • Other (Documentation, CI, ...)

Contribution checklist

  • Add our license to new files.
  • Check that your code adheres to our coding style.
  • Write unittests.
  • Run the unittests and check that they’re passing.
  • Run the benchmarks to make sure your change doesn’t introduce a regression.

@github-actions

github-actions Bot commented May 27, 2026

Copy link
Copy Markdown
Contributor

Benchmarks that have stayed the same:

   before           after         ratio
 [fb7ef0a2]       [148c648f]
    2.92±0.1s        3.17±0.1s     1.08  gini_portfolio.Cajas.time_compile_problem
   4.60±0.02s       4.75±0.02s     1.03  svm_l1_regularization.SVMWithL1Regularization.time_compile_problem
   4.25±0.07s        4.34±0.1s     1.02  simple_QP_benchmarks.SimpleQPBenchmark.time_compile_problem
   2.56±0.05s       2.61±0.03s     1.02  slow_pruning_1668_benchmark.SlowPruningBenchmark.time_compile_problem
   1.33±0.01s       1.34±0.01s     1.01  matrix_stuffing.ParamConeMatrixStuffing.time_compile_problem
   4.58±0.01s       4.60±0.01s     1.01  optimal_advertising.OptimalAdvertising.time_compile_problem
   1.76±0.03s       1.76±0.02s     1.00  semidefinite_programming.SemidefiniteProgramming.time_compile_problem
   2.43±0.02s       2.43±0.01s     1.00  matrix_stuffing.SmallMatrixStuffing.time_compile_problem
   1.87±0.02s       1.87±0.04s     1.00  simple_LP_benchmarks.SimpleScalarParametrizedLPBenchmark.time_compile_problem
   3.77±0.04s       3.76±0.01s     1.00  simple_QP_benchmarks.UnconstrainedQP.time_compile_problem
   21.4±0.02s       21.3±0.01s     1.00  sdp_segfault_1132_benchmark.SDPSegfault1132Benchmark.time_compile_problem
   3.55±0.04s       3.53±0.06s     0.99  matrix_stuffing.ParamSmallMatrixStuffing.time_compile_problem
   2.12±0.02s       2.11±0.02s     0.99  simple_QP_benchmarks.LeastSquares.time_compile_problem
   2.88±0.05s       2.86±0.02s     0.99  simple_QP_benchmarks.ParametrizedQPBenchmark.time_compile_problem
   1.50±0.02s       1.49±0.01s     0.99  tv_inpainting.TvInpainting.time_compile_problem
   3.54±0.06s       3.49±0.01s     0.99  matrix_stuffing.ConeMatrixStuffingBench.time_compile_problem
    1.94±0.1s       1.91±0.03s     0.98  finance.FactorCovarianceModel.time_compile_problem
      168±7ms          165±2ms     0.98  high_dim_convex_plasticity.ConvexPlasticity.time_compile_problem
   10.8±0.02s       10.5±0.04s     0.98  simple_LP_benchmarks.SimpleLPBenchmark.time_compile_problem
   4.27±0.09s       4.16±0.01s     0.97  huber_regression.HuberRegression.time_compile_problem
     746±20ms         724±10ms     0.97  simple_LP_benchmarks.SimpleFullyParametrizedLPBenchmark.time_compile_problem
   1.44±0.03s       1.38±0.04s     0.96  gini_portfolio.Yitzhaki.time_compile_problem
    13.6±0.2s        13.1±0.2s     0.96  finance.CVaRBenchmark.time_compile_problem
   3.12±0.08s       2.99±0.02s     0.96  quantum_hilbert_matrix.QuantumHilbertMatrix.time_compile_problem
   1.38±0.05s       1.24±0.03s     0.90  gini_portfolio.Murray.time_compile_problem

Comment thread cvxpy/reductions/subexpr_cache.py Outdated
@PTNobel
PTNobel changed the base branch from ptn/dcp2cone-cse to master May 27, 2026 22:48
@PTNobel PTNobel changed the title Dedup identical subtrees in Dnlp2Smooth canonicalization Dedup identical subtrees in Dcp2Cone and Dnlp2Smooth canonicalization May 27, 2026
PTNobel added a commit that referenced this pull request May 27, 2026
Two small fixes for PR #3355:

1. PR review nit: expr_key's docstring still claimed "Constants key by
   object identity," but small Constants (<= 64 elements) now key by
   value; updated to reflect both branches.

2. CI: test_copt_mi_socp_1 fails by ~7e-5 on the CSE-deduplicated
   formulation. The continuous SOCP relaxation (verified at high
   precision via CLARABEL) sits at x[0] = -0.78510, while COPT lands
   at -0.78503 with CSE -- both within typical MI-SOCP precision, but
   the test's hardcoded -0.78510265 expects 4 decimals. MOSEK, CPLEX,
   and SCIP already use places=3 on this same test for the same
   tolerance reason; align COPT.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
@SteveDiamond

Copy link
Copy Markdown
Collaborator

This generally looks really good. @Transurgeon can you review the changes to DNLP?

@SteveDiamond SteveDiamond left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed with differential testing of shared-vs-unshared canonicalization across many atoms/solvers — the CSE correctness looks solid (the DAG-mutation risk is correctly defended by the fresh-placeholder-id fix, and the _affine_above_relevant key logic mirrors canonicalize_tree). A few inline notes below; the main one is the O(N²) canonicalization regression on deep expression trees.

Comment thread cvxpy/reductions/subexpr_cache.py Outdated
Comment thread cvxpy/reductions/dcp2cone/dcp2cone.py Outdated
Comment thread cvxpy/reductions/dnlp2smooth/dnlp2smooth.py Outdated
Comment thread cvxpy/reductions/subexpr_cache.py
Comment thread cvxpy/reductions/dnlp2smooth/dnlp2smooth.py Outdated
Comment thread cvxpy/utilities/replace_quad_forms.py
@PTNobel

PTNobel commented Jun 5, 2026

Copy link
Copy Markdown
Collaborator Author

All my experiments show linear memory etc now!

PTNobel and others added 14 commits June 8, 2026 12:51
Add a per-apply common-subexpression cache to Dcp2Cone so that
structurally identical Expression subtrees share one canonicalized
expression and one set of auxiliary constraints within a reduction
pass. For cp.Problem(cp.Minimize(cp.norm1(x)), [cp.norm1(x) <= 1])
this collapses two epigraph variables and two pairs of abs-epigraph
inequalities down to one of each.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the cache embedded `Constant.value.tobytes()` in keys, which
made the cache footprint scale with the problem's constant data (~8 MB
for an LP 1000x500). Switch Constant keying to object identity, which
still deduplicates the common case of a shared Constant reference and
brings cache size below ~16 KB across representative problems.

Also skip caching any subtree whose canonicalization went through the
quad branch. Those canonicalizers emit SymbolicQuadForm markers that
downstream code (replace_quad_forms, coeff_extractor) identifies by
Python id and assumes are distinct per occurrence; sharing one across
sites silently halves quadratic coefficients. Track this with a counter
incremented on quad branches; don't cache if it advances under a node.

Release the cache at the end of apply() so it does not outlive the
reduction.

Add a regression test using the QuadForm 0.5*qf + 0.5*qf pattern from
test_qp_solvers.py::rep_quad_form.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
replace_quad_form previously gave the placeholder Variable the same id
as the quad form it replaced. When the same SymbolicQuadForm appeared
at multiple positions in the objective expression (e.g. via CSE, or via
0.5*qf + 0.5*qf with a shared qf), the placeholders collapsed onto a
single row in get_var_offsets and the quadratic coefficient was halved.

Mint a fresh placeholder id per replacement. quad_forms is keyed by
placeholder id, so all downstream lookups continue to work; only the
incidental identity quad_form.id == placeholder.id is dropped.

With this in place, the CSE cache in Dcp2Cone can also dedup subtrees
that go through the quad branch, so the _quad_canon_count guard is
removed.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
By the time coeff_extractor invokes this, Dcp2Cone has already rewritten
every QuadForm into either a SymbolicQuadForm or sum_squares, so the
isinstance check on QuadForm is defensive rather than load-bearing.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Cache hits discard the stored constraints anyway, so store just the
  canonical expression. Keeps the per-apply working set smaller.
- Add two tests that exercise Dcp2Cone(quad_obj=True): one for shared
  quad_over_lin subtrees within the objective (dedup to a single
  SymbolicQuadForm), one for the same subtree appearing in objective and
  constraint (different canonicalizations kept distinct by the
  affine_above component of the cache key).

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves the structural-key helpers from dcp2cone.py into a shared private
module cvxpy/reductions/_cse.py so Dnlp2Smooth can reuse them, and wires
an analogous per-apply cache into Dnlp2Smooth.canonicalize_tree. The NLP
chain (Dnlp2Smooth -> NLP solver, via nlp_solving_chain.py) is not
downstream of Dcp2Cone, so duplicate subtrees in DNLP problems were
producing duplicate aux Variables and constraints; this PR fixes that
the same way #3353 did for Dcp2Cone.

Also tightens _constant_key: small Constants (<= 64 elements) are now
keyed by value rather than id. This catches the case where two
structurally identical user expressions embed distinct Constant objects
for default scalar parameters (e.g. each cp.huber(x) call mints a fresh
Constant(0.5) for the default M), which would otherwise defeat the
merge. Large arrays stay id-keyed to avoid copying problem data into
cache keys.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cvxpy uses bare acronyms as module names (psd.py, soc_canon.py,
scs_conif.py, dcp2cone/, dgp2dcp/, dnlp2smooth/, etc.) and does not
prefix internal modules with an underscore. cse (Common Subexpression
Elimination) is descriptive enough on its own.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
'CSE' is compiler-optimization jargon and not widely understood outside
that niche. 'subexpression cache' describes what the module supports in
terms that don't require CS expertise.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two small fixes for PR #3355:

1. PR review nit: expr_key's docstring still claimed "Constants key by
   object identity," but small Constants (<= 64 elements) now key by
   value; updated to reflect both branches.

2. CI: test_copt_mi_socp_1 fails by ~7e-5 on the CSE-deduplicated
   formulation. The continuous SOCP relaxation (verified at high
   precision via CLARABEL) sits at x[0] = -0.78510, while COPT lands
   at -0.78503 with CSE -- both within typical MI-SOCP precision, but
   the test's hardcoded -0.78510265 expects 4 decimals. MOSEK, CPLEX,
   and SCIP already use places=3 on this same test for the same
   tolerance reason; align COPT.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matches the existing MOSEK/CPLEX/SCIP places=3 lines, which carry no
comment.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Constant.__init__ stores float64 ndarrays by reference (no copy in
ndarray_interface.const_to_matrix), so two cp.Constant(arr) wrappers
around the same source ndarray share _value. Keying on id(expr.value)
in that case catches the dedup without copying bytes into the cache
key.

Restricted to float64 ndarrays because other dtypes go through
astype(float64) (which copies) or scipy sparse's csc_array constructor
(which builds a fresh wrapper), so the id-of-underlying branch only
fires when sharing is real.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
subexpr_cache: give every helper an explicit return type and parameter
type, add Signature/ExprKey aliases, and parameterize the bare tuple/dict
annotations. expr_key takes `object` (it dispatches via isinstance and
raises UncacheableError for non-Expression inputs reached through a
partial_problem's args), so the defensive branch stays reachable.

dcp2cone/dnlp2smooth: parameterize the bare dict/list annotations on the
CSE-related signatures: cse_cache (keyed on ConeCacheKey vs ExprKey
respectively), affine_above_relevant_cache as dict[int, bool], and the
constraint lists as list[Constraint].

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Parameterize the Dcp2Cone and Dnlp2Smooth canonicalize_expr overrides:
expr: Expression, args: list[Expression], and the constraint list as
list[Constraint] (matching canonicalize_tree and the base class's
expr: Expression convention). Wrapped to stay under the line limit.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@PTNobel
PTNobel force-pushed the ptn/dnlp2smooth-cse branch from 42f010c to c93a129 Compare June 8, 2026 19:52
Mirror the base Canonicalization.canonicalize_tree @overload typing onto
the Dcp2Cone and Dnlp2Smooth overrides: expr is now typed Canonical and
the overloads preserve Expression->Expression, Constraint->Constraint,
Objective->Objective so callers (e.g. apply()) recover the precise
return type instead of a bare Expression. Also annotate the CSE cache
locals and the _affine_above_relevant helper.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Transurgeon

Copy link
Copy Markdown
Collaborator

nice progress on this PR and really nice job getting it to work!
I am a bit busy with other things now but this is definitely on my mind to review soon :).

@Transurgeon Transurgeon left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@PTNobel I got a chance to look at the PR now.
I think we should remove the changes for DNLP since there is not much benefit if we can't reuse the derivative computation. I will open an issue for that.

Also one clarifying question:

  • this PR allows us to not create new auxiliary variables, but we still can't reuse affine subexpressions right?

As an example, I was thinking of this control of car example. Where we could reuse x_curr across all constraints, instead of having to reform the indexing expression every time.
Im not saying we should do this (if its not done), but I am just trying to understand the scope of the PR.
After you answer this question and make the DNLP change, I think this is looking good to merge!

@PTNobel

PTNobel commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator Author

this PR allows us to not create new auxiliary variables, but we still can't reuse affine subexpressions right?

I am not sure that I know what it would mean to reuse affine subexpressions.

This ensures that identical (in formation) affine subexpressions refer to the same object when passed to the next step, which means that canonical_form returns the same LinOp object.

At this point, we do not cache canonicalizing LinOp objects. Therefore, this doesn't do anything. However, this PR makes it way easier to make that change (it becomes local to the backend rather than needing to be done across the codebase).

@PTNobel

PTNobel commented Jun 18, 2026

Copy link
Copy Markdown
Collaborator Author

#3423 goes ahead and adds the LinOp cache to SCIPY and COO.

@PTNobel
PTNobel requested a review from Transurgeon June 18, 2026 19:14
@rileyjmurray
rileyjmurray dismissed Transurgeon’s stale review June 18, 2026 19:17

Parth made the required change (dropping Dnlp2Smooth integration) on this branch directly.

Non-required desired changes have been made on a new branch/PR that will target master after this branch is merged.

@PTNobel PTNobel changed the title Dedup identical subtrees in Dcp2Cone and Dnlp2Smooth canonicalization Dedup identical subtrees in Dcp2Cone canonicalization Jun 18, 2026
@PTNobel
PTNobel merged commit 567258d into master Jun 18, 2026
54 of 55 checks passed
@PTNobel
PTNobel deleted the ptn/dnlp2smooth-cse branch June 18, 2026 19:18
@PTNobel PTNobel mentioned this pull request Jun 19, 2026
9 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants