Dedup identical subtrees in Dcp2Cone canonicalization - #3355
Conversation
|
Benchmarks that have stayed the same: |
Two small fixes for PR #3355: 1. PR review nit: expr_key's docstring still claimed "Constants key by object identity," but small Constants (<= 64 elements) now key by value; updated to reflect both branches. 2. CI: test_copt_mi_socp_1 fails by ~7e-5 on the CSE-deduplicated formulation. The continuous SOCP relaxation (verified at high precision via CLARABEL) sits at x[0] = -0.78510, while COPT lands at -0.78503 with CSE -- both within typical MI-SOCP precision, but the test's hardcoded -0.78510265 expects 4 decimals. MOSEK, CPLEX, and SCIP already use places=3 on this same test for the same tolerance reason; align COPT. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
This generally looks really good. @Transurgeon can you review the changes to DNLP? |
SteveDiamond
left a comment
There was a problem hiding this comment.
Reviewed with differential testing of shared-vs-unshared canonicalization across many atoms/solvers — the CSE correctness looks solid (the DAG-mutation risk is correctly defended by the fresh-placeholder-id fix, and the _affine_above_relevant key logic mirrors canonicalize_tree). A few inline notes below; the main one is the O(N²) canonicalization regression on deep expression trees.
|
All my experiments show linear memory etc now! |
Add a per-apply common-subexpression cache to Dcp2Cone so that structurally identical Expression subtrees share one canonicalized expression and one set of auxiliary constraints within a reduction pass. For cp.Problem(cp.Minimize(cp.norm1(x)), [cp.norm1(x) <= 1]) this collapses two epigraph variables and two pairs of abs-epigraph inequalities down to one of each. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Previously the cache embedded `Constant.value.tobytes()` in keys, which made the cache footprint scale with the problem's constant data (~8 MB for an LP 1000x500). Switch Constant keying to object identity, which still deduplicates the common case of a shared Constant reference and brings cache size below ~16 KB across representative problems. Also skip caching any subtree whose canonicalization went through the quad branch. Those canonicalizers emit SymbolicQuadForm markers that downstream code (replace_quad_forms, coeff_extractor) identifies by Python id and assumes are distinct per occurrence; sharing one across sites silently halves quadratic coefficients. Track this with a counter incremented on quad branches; don't cache if it advances under a node. Release the cache at the end of apply() so it does not outlive the reduction. Add a regression test using the QuadForm 0.5*qf + 0.5*qf pattern from test_qp_solvers.py::rep_quad_form. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
replace_quad_form previously gave the placeholder Variable the same id as the quad form it replaced. When the same SymbolicQuadForm appeared at multiple positions in the objective expression (e.g. via CSE, or via 0.5*qf + 0.5*qf with a shared qf), the placeholders collapsed onto a single row in get_var_offsets and the quadratic coefficient was halved. Mint a fresh placeholder id per replacement. quad_forms is keyed by placeholder id, so all downstream lookups continue to work; only the incidental identity quad_form.id == placeholder.id is dropped. With this in place, the CSE cache in Dcp2Cone can also dedup subtrees that go through the quad branch, so the _quad_canon_count guard is removed. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
By the time coeff_extractor invokes this, Dcp2Cone has already rewritten every QuadForm into either a SymbolicQuadForm or sum_squares, so the isinstance check on QuadForm is defensive rather than load-bearing. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
- Cache hits discard the stored constraints anyway, so store just the canonical expression. Keeps the per-apply working set smaller. - Add two tests that exercise Dcp2Cone(quad_obj=True): one for shared quad_over_lin subtrees within the objective (dedup to a single SymbolicQuadForm), one for the same subtree appearing in objective and constraint (different canonicalizations kept distinct by the affine_above component of the cache key). Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Moves the structural-key helpers from dcp2cone.py into a shared private module cvxpy/reductions/_cse.py so Dnlp2Smooth can reuse them, and wires an analogous per-apply cache into Dnlp2Smooth.canonicalize_tree. The NLP chain (Dnlp2Smooth -> NLP solver, via nlp_solving_chain.py) is not downstream of Dcp2Cone, so duplicate subtrees in DNLP problems were producing duplicate aux Variables and constraints; this PR fixes that the same way #3353 did for Dcp2Cone. Also tightens _constant_key: small Constants (<= 64 elements) are now keyed by value rather than id. This catches the case where two structurally identical user expressions embed distinct Constant objects for default scalar parameters (e.g. each cp.huber(x) call mints a fresh Constant(0.5) for the default M), which would otherwise defeat the merge. Large arrays stay id-keyed to avoid copying problem data into cache keys. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
cvxpy uses bare acronyms as module names (psd.py, soc_canon.py, scs_conif.py, dcp2cone/, dgp2dcp/, dnlp2smooth/, etc.) and does not prefix internal modules with an underscore. cse (Common Subexpression Elimination) is descriptive enough on its own. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
'CSE' is compiler-optimization jargon and not widely understood outside that niche. 'subexpression cache' describes what the module supports in terms that don't require CS expertise. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Two small fixes for PR #3355: 1. PR review nit: expr_key's docstring still claimed "Constants key by object identity," but small Constants (<= 64 elements) now key by value; updated to reflect both branches. 2. CI: test_copt_mi_socp_1 fails by ~7e-5 on the CSE-deduplicated formulation. The continuous SOCP relaxation (verified at high precision via CLARABEL) sits at x[0] = -0.78510, while COPT lands at -0.78503 with CSE -- both within typical MI-SOCP precision, but the test's hardcoded -0.78510265 expects 4 decimals. MOSEK, CPLEX, and SCIP already use places=3 on this same test for the same tolerance reason; align COPT. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Matches the existing MOSEK/CPLEX/SCIP places=3 lines, which carry no comment. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Constant.__init__ stores float64 ndarrays by reference (no copy in ndarray_interface.const_to_matrix), so two cp.Constant(arr) wrappers around the same source ndarray share _value. Keying on id(expr.value) in that case catches the dedup without copying bytes into the cache key. Restricted to float64 ndarrays because other dtypes go through astype(float64) (which copies) or scipy sparse's csc_array constructor (which builds a fresh wrapper), so the id-of-underlying branch only fires when sharing is real. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
subexpr_cache: give every helper an explicit return type and parameter type, add Signature/ExprKey aliases, and parameterize the bare tuple/dict annotations. expr_key takes `object` (it dispatches via isinstance and raises UncacheableError for non-Expression inputs reached through a partial_problem's args), so the defensive branch stays reachable. dcp2cone/dnlp2smooth: parameterize the bare dict/list annotations on the CSE-related signatures: cse_cache (keyed on ConeCacheKey vs ExprKey respectively), affine_above_relevant_cache as dict[int, bool], and the constraint lists as list[Constraint]. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Parameterize the Dcp2Cone and Dnlp2Smooth canonicalize_expr overrides: expr: Expression, args: list[Expression], and the constraint list as list[Constraint] (matching canonicalize_tree and the base class's expr: Expression convention). Wrapped to stay under the line limit. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
42f010c to
c93a129
Compare
Mirror the base Canonicalization.canonicalize_tree @overload typing onto the Dcp2Cone and Dnlp2Smooth overrides: expr is now typed Canonical and the overloads preserve Expression->Expression, Constraint->Constraint, Objective->Objective so callers (e.g. apply()) recover the precise return type instead of a bare Expression. Also annotate the CSE cache locals and the _affine_above_relevant helper. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
nice progress on this PR and really nice job getting it to work! |
Transurgeon
left a comment
There was a problem hiding this comment.
@PTNobel I got a chance to look at the PR now.
I think we should remove the changes for DNLP since there is not much benefit if we can't reuse the derivative computation. I will open an issue for that.
Also one clarifying question:
- this PR allows us to not create new auxiliary variables, but we still can't reuse affine subexpressions right?
As an example, I was thinking of this control of car example. Where we could reuse x_curr across all constraints, instead of having to reform the indexing expression every time.
Im not saying we should do this (if its not done), but I am just trying to understand the scope of the PR.
After you answer this question and make the DNLP change, I think this is looking good to merge!
I am not sure that I know what it would mean to reuse affine subexpressions. This ensures that identical (in formation) affine subexpressions refer to the same object when passed to the next step, which means that canonical_form returns the same LinOp object. At this point, we do not cache canonicalizing LinOp objects. Therefore, this doesn't do anything. However, this PR makes it way easier to make that change (it becomes local to the backend rather than needing to be done across the codebase). |
|
#3423 goes ahead and adds the LinOp cache to SCIPY and COO. |
Parth made the required change (dropping Dnlp2Smooth integration) on this branch directly.
Non-required desired changes have been made on a new branch/PR that will target master after this branch is merged.
Description
When the same Expression subtree appears in two places in a DCP problem, for example
cp.norm1(x)in both the objective and a constraint,Dcp2Conepreviously emitted a fresh set of auxiliary variables and epigraph constraints per occurrence. This PR adds a per-apply()structural-key cache soDcp2Conecanonicalizes each structurally identical subtree once and reuses the canonical expression on later occurrences.Worked example for the reported case:
The canonicalized problem now has one epigraph variable
tshared across the objective and the constraint, with one pair oft >= x,t >= -xinequalities.The DNLP changes from earlier revisions were removed from this PR. The original DNLP work is preserved on branch
ptn/dnlp2smooth-cse-dnlp-followupfor follow-up issue discussion.Issue link (if applicable):
Type of change
Contribution checklist