placebo_treatment_refuter crashes with LinAlgError: Singular matrix when using backdoor.econml.dml.DML estimator #1830

Environment

• DoWhy: 0.14
• EconML:0.17.0
• NumPy: latest
• Python: 3.11
• OS: Windows 10

Summary

When using DoWhy with a non-linear estimator (backdoor.econml.dml.DML with GradientBoostingRegressor for model_y/model_t), the estimation itself works correctly, but all three refuters (placebo_treatment_refuter, random_common_cause, data_subset_refuter) crash with numpy.linalg.LinAlgError: Singular matrix.

With the same data and OLS (backdoor.linear_regression), all three refuters run normally.

Root Cause (from traceback)

The crash occurs in EconML's sensitivity analysis module, not in DoWhy's refuter logic itself:

econml/dml/dml.py:183 → dml_sensitivity_values(T_res, Y_res)
econml/validate/sensitivity_analysis.py:151 → np.linalg.inv(G)

When the placebo refuter replaces the treatment with random noise, the DML cross-fitting residuals (T_res, Y_res) can become degenerate, producing a singular G matrix that np.linalg.inv() cannot invert.

Minimal Reproduction

import numpy as np
import pandas as pd
from dowhy import CausalModel
from sklearn.ensemble import GradientBoostingRegressor
from sklearn.linear_model import LinearRegression

np.random.seed(42)
n = 500
Z1 = np.random.randn(n)
Z2 = np.random.randn(n)
T = 0.6 * Z1 + np.random.randn(n) * 0.4
Y = 0.5 * Z1 + 0.3 * Z2 + np.random.randn(n) * 0.5
df = pd.DataFrame({'T': T, 'Y': Y, 'Z1': Z1, 'Z2': Z2})

model = CausalModel(data=df, treatment='T', outcome='Y', common_causes=['Z1', 'Z2'])
estimand = model.identify_effect(proceed_when_unidentifiable=True)

# Step 1: Estimate with DML+GBM (WORKS)
estimate = model.estimate_effect(
    estimand,
    method_name='backdoor.econml.dml.DML',
    method_params={'init_params': {
        'model_y': GradientBoostingRegressor(max_depth=3, n_estimators=30),
        'model_t': GradientBoostingRegressor(max_depth=3, n_estimators=30),
        'model_final': LinearRegression(),
        'discrete_treatment': False
    }}
)
print(f"Estimate: {estimate.value:.6f}")  # Works fine

# Step 2: Refute with placebo (CRASHES)
ref = model.refute_estimate(estimand, estimate, method_name='placebo_treatment_refuter')
# → numpy.linalg.LinAlgError: Singular matrix

Full Traceback (key frames)

File "dowhy/causal_refuters/placebo_treatment_refuter.py", line 155, in _refute_once
    new_estimator.fit(...)
File "dowhy/causal_estimators/econml.py", line 190, in fit
    self.estimator.fit(**estimator_data_args, **kwargs)
File "econml/dml/dml.py", line 616, in fit
    return super().fit(...)
File "econml/_ortho_learner.py", line 984, in _fit_final
    self._ortho_learner_model_final.fit(...)
File "econml/dml/dml.py", line 183, in fit
    self.sensitivity_params = dml_sensitivity_values(T_res, Y_res)
File "econml/validate/sensitivity_analysis.py", line 151
    G_inv = np.linalg.inv(G)
numpy.linalg.LinAlgError: Singular matrix

Systematic Test Results (36-cell matrix)

We ran a systematic matrix: 3 estimators (OLS, DML-GBM, DML-RF) × 4 scenarios (true causal, confounded, spurious, nonlinear DGP) × 3 refuters (placebo, random-CC, subset) = 36 cells:

• Trigger rate: 0/36 (0%) — no refuter triggered in any configuration (including spurious edges designed to fail)
• Crash rate: 8/36 (22%) — all crashes from DML backends with placebo (LinAlgError consistent across GBM/RF)
• OLS: all 12 cells run normally but zero triggering
• DML + placebo: 4/4 crash
• DML + random-CC/subset: run normally but zero triggering

Expected Behavior

The refuter should either:

1. Catch the LinAlgError and report "refuter could not run" (graceful degradation)
2. Skip sensitivity parameter computation when matrix is singular
3. Use np.linalg.pinv (pseudo-inverse) instead of np.linalg.inv

Workaround

Use OLS (backdoor.linear_regression) — refuters run normally. Or wrap refuter calls in try/except.

Impact

This makes DoWhy's refutation battery unusable with non-linear estimators, which are increasingly the norm for heterogeneous treatment effect estimation.