Skip to content

DRAFT: diffengine backend for ignore_dpp#3348

Draft
Transurgeon wants to merge 18 commits into
cvxpy:masterfrom
Transurgeon:diffengine-backend-ignoredpp
Draft

DRAFT: diffengine backend for ignore_dpp#3348
Transurgeon wants to merge 18 commits into
cvxpy:masterfrom
Transurgeon:diffengine-backend-ignoredpp

Conversation

@Transurgeon

Copy link
Copy Markdown
Collaborator

Description

Please include a short summary of the change.
Issue link (if applicable):

Type of change

  • New feature (backwards compatible)
  • New feature (breaking API changes)
  • Bug fix
  • Other (Documentation, CI, ...)

Contribution checklist

  • Add our license to new files.
  • Check that your code adheres to our coding style.
  • Write unittests.
  • Run the unittests and check that they’re passing.
  • Run the benchmarks to make sure your change doesn’t introduce a regression.

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.


Transurgeon seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
You have signed the CLA already but the status is still pending? Let us recheck it.

@github-actions

github-actions Bot commented May 27, 2026

Copy link
Copy Markdown

Benchmarks that have stayed the same:

   before           after         ratio
 [8fd791fb]       [910fdab8]
      1.18±0s          1.21±0s     1.02  gini_portfolio.Murray.time_compile_problem
      3.58±0s          3.67±0s     1.02  matrix_stuffing.ParamSmallMatrixStuffing.time_compile_problem
      1.86±0s          1.88±0s     1.01  simple_LP_benchmarks.SimpleScalarParametrizedLPBenchmark.time_compile_problem
      1.37±0s          1.39±0s     1.01  matrix_stuffing.ParamConeMatrixStuffing.time_compile_problem
      2.38±0s          2.40±0s     1.01  slow_pruning_1668_benchmark.SlowPruningBenchmark.time_compile_problem
      9.94±0s          10.0±0s     1.01  simple_LP_benchmarks.SimpleLPBenchmark.time_compile_problem
      4.38±0s          4.40±0s     1.01  svm_l1_regularization.SVMWithL1Regularization.time_compile_problem
      160±0ms          161±0ms     1.00  high_dim_convex_plasticity.ConvexPlasticity.time_compile_problem
      3.42±0s          3.44±0s     1.00  matrix_stuffing.ConeMatrixStuffingBench.time_compile_problem
      1.93±0s          1.94±0s     1.00  semidefinite_programming.SemidefiniteProgramming.time_compile_problem
      2.76±0s          2.78±0s     1.00  simple_QP_benchmarks.ParametrizedQPBenchmark.time_compile_problem
      4.05±0s          4.06±0s     1.00  simple_QP_benchmarks.SimpleQPBenchmark.time_compile_problem
      20.2±0s          20.2±0s     1.00  sdp_segfault_1132_benchmark.SDPSegfault1132Benchmark.time_compile_problem
      12.0±0s          12.0±0s     1.00  finance.CVaRBenchmark.time_compile_problem
      2.01±0s          2.01±0s     1.00  simple_QP_benchmarks.LeastSquares.time_compile_problem
      3.87±0s          3.87±0s     1.00  huber_regression.HuberRegression.time_compile_problem
      3.66±0s          3.67±0s     1.00  simple_QP_benchmarks.UnconstrainedQP.time_compile_problem
      5.16±0s          5.17±0s     1.00  optimal_advertising.OptimalAdvertising.time_compile_problem
      2.39±0s          2.39±0s     1.00  matrix_stuffing.SmallMatrixStuffing.time_compile_problem
      1.37±0s          1.36±0s     1.00  gini_portfolio.Yitzhaki.time_compile_problem
      1.87±0s          1.86±0s     1.00  finance.FactorCovarianceModel.time_compile_problem
      2.73±0s          2.72±0s     1.00  quantum_hilbert_matrix.QuantumHilbertMatrix.time_compile_problem
      1.56±0s          1.55±0s     1.00  tv_inpainting.TvInpainting.time_compile_problem
      713±0ms          710±0ms     1.00  simple_LP_benchmarks.SimpleFullyParametrizedLPBenchmark.time_compile_problem
      6.62±0s          6.58±0s     0.99  gini_portfolio.Cajas.time_compile_problem

Comment thread cvxpy/reductions/solvers/solving_chain.py Outdated
Comment thread cvxpy/reductions/solvers/solving_chain.py Outdated
@SteveDiamond

Copy link
Copy Markdown
Collaborator

This looks promising, but could you add benchmark timings? There were some benchmarks used for the COO backend PRs, or you could make more.

Comment thread cvxpy/reductions/dcp2cone/cone_matrix_stuffing.py Outdated
@Transurgeon Transurgeon force-pushed the diffengine-backend-ignoredpp branch from dcecba7 to 098b456 Compare June 3, 2026 11:04
…ackend

Reverts changes that are not required to introduce the DIFFENGINE backend:

- cone_matrix_stuffing.py: remove the top-level lower_and_order_constraints()
  helper. ConeMatrixStuffing.apply() now lowers + reorders the constraints once
  inline (as on master), then selects the backend afterward: the DIFFENGINE
  path is a small early-return branch, the standard path is otherwise unchanged.
  No duplicated lowering loop.
- Drop the use_diffengine property; inline the
  `canon_backend == DIFFENGINE_CANON_BACKEND` check (also addresses the review
  question of why it differs from canon_backend).
- solving_chain.py: move the use_quad/quad_obj block back to its original
  location; keep only the ignore_dpp -> DIFFENGINE wiring.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Transurgeon Transurgeon force-pushed the diffengine-backend-ignoredpp branch from 098b456 to f215ff9 Compare June 3, 2026 11:16
Transurgeon and others added 3 commits June 3, 2026 07:18
These will be rewritten by hand later.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- DiffengineConeProgram: extract the shared x=0 matrix evaluation (q, d, A, b, P)
  into _extract_matrices, used by both from_problem and apply_parameters instead of
  duplicating it. Behavior unchanged.
- convert_symbolic_quad_form / _lower_symbolic_power: condense the verbose
  docstrings/comments; logic unchanged (all sparse/dense/parametric/Power branches
  are exercised).
- Drop the verbose DIFFENGINE explanatory comments in cone_matrix_stuffing.py and
  solving_chain.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Both ParamProb backends (ParamConeProg and DiffengineConeProgram) already return
concrete (q, d, A, b) from apply_parameters(), and the perspective aux problem is
parameter-free. Use that single call instead of branching on the program type and
hand-decoding ParamConeProg's embedded sparse tensors. Removes the isinstance
branch, the DiffengineConeProgram import, and the brittle reshape(order="F") decode.

A is kept sparse (sparse @ x_canon works and avoids embedding a dense constant in
the enclosing problem).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@Transurgeon Transurgeon force-pushed the diffengine-backend-ignoredpp branch from a6c6f51 to f5924ce Compare June 3, 2026 12:04
Transurgeon and others added 8 commits June 3, 2026 08:23
Introduce DiffEngineExtractor (the non-DPP analog of CoeffExtractor) in the
diff_engine package: it owns the C_problem and recovers the concrete cone
matrices (q, d, A, b, P) by evaluating the autodiff graph at x=0, re-evaluating
on parameter changes. It absorbs the _build_jacobian_csc / _build_hessian_csc /
_extract_matrices helpers.

DiffengineConeProgram becomes a thin ParamProb that holds a DiffEngineExtractor
and delegates: apply_parameters calls extractor.update_parameters + extract,
from_problem builds the extractor, and apply_restruct_mat shares it with the
restructured instance. This mirrors the standard path (CoeffExtractor ->
ParamConeProg) with (DiffEngineExtractor -> DiffengineConeProgram).

Behavior-preserving; the ParamProb interface and the standard cone path are
unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The diffengine cone backend's quad_form handling requires the dense-quadform
bindings shipped in the 0.5.x line.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The diffengine cone backend is tightly coupled to the sparsediffpy binding ABI
(dense make_quad_form), so pin the exact version rather than a range.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Remove the lb_tensor/ub_tensor placeholders (parametric bounds are not supported
on the diffengine path and nothing reads them), the unused param_id_to_size, and
the self-referential quad_obj attribute (the solver already keys quadratic-ness
off `problem.P is None`). No external readers of any of these.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Nothing reads it (apply_parameters reads p.value directly rather than going
through a param_id->col map). The other accessors stay: variables (read by the
HIGHS/Gurobi/Xpress interfaces), var_id_to_col (perspective_canon + split_solution),
and id_to_var (split_solution).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Drop the intra-method grouping blank lines in __init__, apply_parameters,
apply_restruct_mat, and from_problem. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Mirror the Jacobian pattern for the Hessian: capture the (fixed) lower-triangular
Hessian sparsity once in build() via init_hessian_coo_lower_tri +
get_problem_hessian_sparsity_coo, then on each re-solve fetch only the lower-tri
values (eval_hessian_vals_coo_lower_tri) and mirror off-diagonals to the full
symmetric P the cone solvers expect. Reuses existing bindings (no engine change).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The lazy-operand variant was confirmed unrelated to the matmul slowdown/segfault
(reverting left both unchanged: Murray ~15.5s cold compile, OptimalAdvertising
still segfaults). The slowdown/crash live in make_dense/sparse_left_matmul + the C
engine, invoked identically by both forms. Keep the simpler eager version;
convert_expr converts all args up front and convert_matmul consumes children.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants