Skip to content

Validate perturbation feature masks before attribution (#1922) - #1922

Open
craymichael wants to merge 3 commits into
meta-pytorch:masterfrom
craymichael:export-D117601317
Open

Validate perturbation feature masks before attribution (#1922)#1922
craymichael wants to merge 3 commits into
meta-pytorch:masterfrom
craymichael:export-D117601317

Conversation

@craymichael

@craymichael craymichael commented Aug 28, 2026 •
edited by meta-codesync Bot
Loading

Copy link
Copy Markdown
Contributor

Summary:

Summary

Perturbation methods accepted feature-mask tuples with the wrong arity. Feature Ablation could then return zero attribution for an omitted input, while Shapley failed later or enumerated an invalid feature space. Shapley also accepted fractional, negative, non-finite, and complex group IDs even though its permutation loop requires non-negative integer IDs.

Counterexample

With two model inputs and a one-element feature-mask tuple, Feature Ablation perturbed only the first input and silently left the second input’s attribution at zero. With Shapley mask [0, 0.5], the integer feature loop omitted group 0.5.

Fix

  • Require explicit feature-mask tuples to have one tensor per input in the shared formatter.
  • Validate Shapley group IDs before feature enumeration.
  • Accept integral-valued float masks and bool masks for compatibility.
  • Keep value-domain validation out of Feature Ablation/Permutation so Greedy Feature Selection’s internal -1 sentinel and existing float-typed masks continue to work.

Test Plan

Before: four focused regression methods failed (Pass 132, Fail 4), including silent acceptance of short and long mask tuples and invalid Shapley IDs.

After:

  • Broad perturbation and wrapper suite — Pass 422, Fail 0, covering Feature Ablation, Feature Permutation, Shapley, DataLoaderAttribution, internal wrappers, WithinGroupSVS, SVS-P, AddOneBack, MarginalWithinGroups, Greedy Feature Selection, and shared mask utilities.
  • arc lint -a on changed files — no issues.
  • arc lint -a --engine extra --take CITRINEAGENT on implementation files — no issues.
  • arc pyre check-owning-targets on changed files — no type errors.

Differential Revision: D117601317

@meta-cla meta-cla Bot added the cla signed label Aug 28, 2026
@meta-codesync

meta-codesync Bot commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

@craymichael has exported this pull request. If you are a Meta employee, you can view the originating Diff in D117601317.

craymichael added a commit to craymichael/captum that referenced this pull request Aug 28, 2026
@craymichael @meta-codesync
Validate perturbation feature masks before attribution (meta-pytorch#…
560f9ed 
…1922)

Summary:
Pull Request resolved: meta-pytorch#1922

Summary

Perturbation methods accepted feature-mask tuples with the wrong arity. Feature Ablation could then return zero attribution for an omitted input, while Shapley failed later or enumerated an invalid feature space. Shapley also accepted fractional, negative, non-finite, and complex group IDs even though its permutation loop requires non-negative integer IDs.

Counterexample

With two model inputs and a one-element feature-mask tuple, Feature Ablation perturbed only the first input and silently left the second input’s attribution at zero. With Shapley mask `[0, 0.5]`, the integer feature loop omitted group `0.5`.

Fix

* Require explicit feature-mask tuples to have one tensor per input in the shared formatter.
* Validate Shapley group IDs before feature enumeration.
* Accept integral-valued float masks and bool masks for compatibility.
* Keep value-domain validation out of Feature Ablation/Permutation so Greedy Feature Selection’s internal `-1` sentinel and existing float-typed masks continue to work.

Test Plan

Before: four focused regression methods failed (`Pass 132, Fail 4`), including silent acceptance of short and long mask tuples and invalid Shapley IDs.

After:
* Broad perturbation and wrapper suite — Pass 422, Fail 0, covering Feature Ablation, Feature Permutation, Shapley, DataLoaderAttribution, internal wrappers, WithinGroupSVS, SVS-P, AddOneBack, MarginalWithinGroups, Greedy Feature Selection, and shared mask utilities.
* `arc lint -a` on changed files — no issues.
* `arc lint -a --engine extra --take CITRINEAGENT` on implementation files — no issues.
* `arc pyre check-owning-targets` on changed files — no type errors.

Differential Revision: D117601317
@meta-codesync meta-codesync Bot changed the title Validate perturbation feature masks before attribution Validate perturbation feature masks before attribution (#1922) Aug 28, 2026
@craymichael @facebook-github-bot
Validate perturbation baselines before attribution (meta-pytorch#1920)
dc1817b 
Summary:

Summary

Feature Ablation and Shapley formatted baselines but did not validate tuple arity or tensor shape before running the model. `_tensorize_baseline` also paired inputs and baselines with `zip`, silently discarding extra baselines.

Problem

For two inputs, a one-element baseline tuple reached the model with a missing argument, while a three-element tuple silently ignored its final baseline. A baseline pool with shape `[2, 1]` for a three-row input was neither a per-example baseline nor a singleton baseline and failed later through opaque broadcasting errors.

Fix

* Validate baselines in both synchronous and future Feature Ablation and Shapley entry points.
* Defensively reject arity mismatches in `_tensorize_baseline`.
* Require Shapley tensor baselines to match the input or use a singleton leading dimension with matching trailing dimensions.
* Preserve Feature Ablation’s documented support for any tensor shape that broadcasts exactly to the input, including `[F]` and 0-D tensors.

Test Plan

Before: the new regressions failed through late `IndexError`, missing-forward-argument, and broadcast errors; the extra-baseline case did not raise at all.

After:
* `buck test fbcode//pytorch/captum/tests/attr:test_common fbcode//pytorch/captum/tests/attr:test_feature_ablation fbcode//pytorch/captum/tests/attr:test_shapley` — Pass 131, Fail 0.
* `arc lint -a` on changed Python files — no source lint issues; focused autodeps updates applied.
* `arc lint -a --engine extra --take CITRINEAGENT` on changed Captum implementation files — no issues.
* `arc pyre check-owning-targets` on changed files — no type errors.

Differential Revision: D117601314
@craymichael @facebook-github-bot
Select perturbed values without arithmetic masking (meta-pytorch#1921)
6d210d4 
Summary:

Summary

Perturbation paths combined original and replacement values as `old * (1 - mask) + new * mask`. IEEE arithmetic makes `NaN * 0` and `Inf * 0` non-finite, so values outside the selected feature could corrupt model inputs and attributions.

Counterexample

For input `[1, 2]`, baseline `[0, NaN]`, and mask `[0, 1]`, ablating feature 0 should evaluate `[0, 2]`. Arithmetic masking instead produced `[0, NaN]`, so feature 0 incorrectly received a `NaN` attribution.

Fix

* Use `torch.where` for Feature Ablation, Feature Permutation, Shapley feature updates, and attribution-mask accumulation.
* Apply the same selection semantics to within-group baseline construction, add-back, and permutation helpers.
* Mirror the fix in the legacy UFO implementations so alternate call paths cannot retain the corruption.
* Keep masks and donor indices on the destination tensor device.

Test Plan

Before: six focused regressions failed across core and within-group paths (`Pass 159, Fail 6`).

After:
* `buck test fbcode//pytorch/captum/tests/attr:test_feature_ablation fbcode//pytorch/captum/tests/attr:test_feature_permutation fbcode//pytorch/captum/tests/attr:test_shapley fbcode//pytorch/captum/tests/attr/fb:test_within_groups_utils fbcode//pytorch/captum/tests/attr/fb:test_shapley_value_permutation` — Pass 178, Fail 0.
* Regressions cover `NaN`, `+Inf`, and `-Inf` in selected and inactive positions.
* `arc lint -a` on all changed Python files — no new lint issues; existing legacy UFO line-length advice remains unchanged.
* `arc lint -a --engine extra --take CITRINEAGENT` on implementation files — no issues.
* `arc pyre check-owning-targets` on changed files — no type errors.

Differential Revision: D117601315
@craymichael @facebook-github-bot
Validate perturbation feature masks before attribution (meta-pytorch#…
1b9eef6 
…1922)

Summary:

Summary

Perturbation methods accepted feature-mask tuples with the wrong arity. Feature Ablation could then return zero attribution for an omitted input, while Shapley failed later or enumerated an invalid feature space. Shapley also accepted fractional, negative, non-finite, and complex group IDs even though its permutation loop requires non-negative integer IDs.

Counterexample

With two model inputs and a one-element feature-mask tuple, Feature Ablation perturbed only the first input and silently left the second input’s attribution at zero. With Shapley mask `[0, 0.5]`, the integer feature loop omitted group `0.5`.

Fix

* Require explicit feature-mask tuples to have one tensor per input in the shared formatter.
* Validate Shapley group IDs before feature enumeration.
* Accept integral-valued float masks and bool masks for compatibility.
* Keep value-domain validation out of Feature Ablation/Permutation so Greedy Feature Selection’s internal `-1` sentinel and existing float-typed masks continue to work.

Test Plan

Before: four focused regression methods failed (`Pass 132, Fail 4`), including silent acceptance of short and long mask tuples and invalid Shapley IDs.

After:
* Broad perturbation and wrapper suite — Pass 422, Fail 0, covering Feature Ablation, Feature Permutation, Shapley, DataLoaderAttribution, internal wrappers, WithinGroupSVS, SVS-P, AddOneBack, MarginalWithinGroups, Greedy Feature Selection, and shared mask utilities.
* `arc lint -a` on changed files — no issues.
* `arc lint -a --engine extra --take CITRINEAGENT` on implementation files — no issues.
* `arc pyre check-owning-targets` on changed files — no type errors.

Differential Revision: D117601317
craymichael added a commit to craymichael/captum that referenced this pull request Aug 28, 2026
@craymichael @facebook-github-bot
Validate perturbation feature masks before attribution (meta-pytorch#…
010ddd9 
…1922)

Summary:

Summary

Perturbation methods accepted feature-mask tuples with the wrong arity. Feature Ablation could then return zero attribution for an omitted input, while Shapley failed later or enumerated an invalid feature space. Shapley also accepted fractional, negative, non-finite, and complex group IDs even though its permutation loop requires non-negative integer IDs.

Counterexample

With two model inputs and a one-element feature-mask tuple, Feature Ablation perturbed only the first input and silently left the second input’s attribution at zero. With Shapley mask `[0, 0.5]`, the integer feature loop omitted group `0.5`.

Fix

* Require explicit feature-mask tuples to have one tensor per input in the shared formatter.
* Validate Shapley group IDs before feature enumeration.
* Accept integral-valued float masks and bool masks for compatibility.
* Keep value-domain validation out of Feature Ablation/Permutation so Greedy Feature Selection’s internal `-1` sentinel and existing float-typed masks continue to work.

Test Plan

Before: four focused regression methods failed (`Pass 132, Fail 4`), including silent acceptance of short and long mask tuples and invalid Shapley IDs.

After:
* Broad perturbation and wrapper suite — Pass 422, Fail 0, covering Feature Ablation, Feature Permutation, Shapley, DataLoaderAttribution, internal wrappers, WithinGroupSVS, SVS-P, AddOneBack, MarginalWithinGroups, Greedy Feature Selection, and shared mask utilities.
* `arc lint -a` on changed files — no issues.
* `arc lint -a --engine extra --take CITRINEAGENT` on implementation files — no issues.
* `arc pyre check-owning-targets` on changed files — no type errors.

Differential Revision: D117601317
craymichael added a commit to craymichael/captum that referenced this pull request Aug 28, 2026
@craymichael @facebook-github-bot
Validate perturbation feature masks before attribution (meta-pytorch#…
80f0c97 
…1922)

Summary:

Summary

Perturbation methods accepted feature-mask tuples with the wrong arity. Feature Ablation could then return zero attribution for an omitted input, while Shapley failed later or enumerated an invalid feature space. Shapley also accepted fractional, negative, non-finite, and complex group IDs even though its permutation loop requires non-negative integer IDs.

Counterexample

With two model inputs and a one-element feature-mask tuple, Feature Ablation perturbed only the first input and silently left the second input’s attribution at zero. With Shapley mask `[0, 0.5]`, the integer feature loop omitted group `0.5`.

Fix

* Require explicit feature-mask tuples to have one tensor per input in the shared formatter.
* Validate Shapley group IDs before feature enumeration.
* Accept integral-valued float masks and bool masks for compatibility.
* Keep value-domain validation out of Feature Ablation/Permutation so Greedy Feature Selection’s internal `-1` sentinel and existing float-typed masks continue to work.

Test Plan

Before: four focused regression methods failed (`Pass 132, Fail 4`), including silent acceptance of short and long mask tuples and invalid Shapley IDs.

After:
* Broad perturbation and wrapper suite — Pass 422, Fail 0, covering Feature Ablation, Feature Permutation, Shapley, DataLoaderAttribution, internal wrappers, WithinGroupSVS, SVS-P, AddOneBack, MarginalWithinGroups, Greedy Feature Selection, and shared mask utilities.
* `arc lint -a` on changed files — no issues.
* `arc lint -a --engine extra --take CITRINEAGENT` on implementation files — no issues.
* `arc pyre check-owning-targets` on changed files — no type errors.

Differential Revision: D117601317
craymichael added a commit to craymichael/captum that referenced this pull request Aug 28, 2026
@craymichael @facebook-github-bot
Validate perturbation feature masks before attribution (meta-pytorch#…
78cb292 
…1922)

Summary:

Summary

Perturbation methods accepted feature-mask tuples with the wrong arity. Feature Ablation could then return zero attribution for an omitted input, while Shapley failed later or enumerated an invalid feature space. Shapley also accepted fractional, negative, non-finite, and complex group IDs even though its permutation loop requires non-negative integer IDs.

Counterexample

With two model inputs and a one-element feature-mask tuple, Feature Ablation perturbed only the first input and silently left the second input’s attribution at zero. With Shapley mask `[0, 0.5]`, the integer feature loop omitted group `0.5`.

Fix

* Require explicit feature-mask tuples to have one tensor per input in the shared formatter.
* Validate Shapley group IDs before feature enumeration.
* Accept integral-valued float masks and bool masks for compatibility.
* Keep value-domain validation out of Feature Ablation/Permutation so Greedy Feature Selection’s internal `-1` sentinel and existing float-typed masks continue to work.

Test Plan

Before: four focused regression methods failed (`Pass 132, Fail 4`), including silent acceptance of short and long mask tuples and invalid Shapley IDs.

After:
* Broad perturbation and wrapper suite — Pass 422, Fail 0, covering Feature Ablation, Feature Permutation, Shapley, DataLoaderAttribution, internal wrappers, WithinGroupSVS, SVS-P, AddOneBack, MarginalWithinGroups, Greedy Feature Selection, and shared mask utilities.
* `arc lint -a` on changed files — no issues.
* `arc lint -a --engine extra --take CITRINEAGENT` on implementation files — no issues.
* `arc pyre check-owning-targets` on changed files — no type errors.

Differential Revision: D117601317
craymichael added a commit to craymichael/captum that referenced this pull request Aug 28, 2026
@craymichael @facebook-github-bot
Validate perturbation feature masks before attribution (meta-pytorch#…
1da1fd2 
…1922)

Summary:

Summary

Perturbation methods accepted feature-mask tuples with the wrong arity. Feature Ablation could then return zero attribution for an omitted input, while Shapley failed later or enumerated an invalid feature space. Shapley also accepted fractional, negative, non-finite, and complex group IDs even though its permutation loop requires non-negative integer IDs.

Counterexample

With two model inputs and a one-element feature-mask tuple, Feature Ablation perturbed only the first input and silently left the second input’s attribution at zero. With Shapley mask `[0, 0.5]`, the integer feature loop omitted group `0.5`.

Fix

* Require explicit feature-mask tuples to have one tensor per input in the shared formatter.
* Validate Shapley group IDs before feature enumeration.
* Accept integral-valued float masks and bool masks for compatibility.
* Keep value-domain validation out of Feature Ablation/Permutation so Greedy Feature Selection’s internal `-1` sentinel and existing float-typed masks continue to work.

Test Plan

Before: four focused regression methods failed (`Pass 132, Fail 4`), including silent acceptance of short and long mask tuples and invalid Shapley IDs.

After:
* Broad perturbation and wrapper suite — Pass 422, Fail 0, covering Feature Ablation, Feature Permutation, Shapley, DataLoaderAttribution, internal wrappers, WithinGroupSVS, SVS-P, AddOneBack, MarginalWithinGroups, Greedy Feature Selection, and shared mask utilities.
* `arc lint -a` on changed files — no issues.
* `arc lint -a --engine extra --take CITRINEAGENT` on implementation files — no issues.
* `arc pyre check-owning-targets` on changed files — no type errors.

Differential Revision: D117601317
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

cla signed meta-exported

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@craymichael