Skip to content

Fix 'get_attr' call in dynamo 'run_node' - #127696

Closed
BowenBao wants to merge 4 commits into
gh/BowenBao/318/basefrom
gh/BowenBao/318/head
Closed

BowenBao wants to merge 4 commits into
gh/BowenBao/318/basefrom
gh/BowenBao/318/head

Conversation

[ghstack-poisoned]
@pytorch-bot

pytorch-bot Bot commented Jun 1, 2024 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/127696

Note: Links to docs will display an error until the docs builds have been completed.

✅ You can merge normally! (3 Unrelated Failures)

As of commit e0665e2 with merge base 9554300 (image):

FLAKY - The following jobs failed but were likely due to flakiness present on trunk:

UNSTABLE - The following job failed but was likely due to flakiness present on trunk and has been marked as unstable:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

BowenBao added 2 commits June 2, 2024 15:54
[ghstack-poisoned]
[ghstack-poisoned]
@BowenBao
BowenBao marked this pull request as ready for review June 2, 2024 22:56
@BowenBao BowenBao added the topic: bug fixes topic category label Jun 3, 2024
[ghstack-poisoned]
pytorchmergebot pushed a commit that referenced this pull request Jun 7, 2024
@clee2000

Copy link
Copy Markdown
Contributor

@pytorchbot revert -m "broke (executorch?) internal tests D58295865" -c ghfirst

@pytorchmergebot

Copy link
Copy Markdown
Collaborator

@pytorchbot successfully started a revert job. Check the current status here.
Questions? Feedback? Please reach out to the PyTorch DevX Team

pytorchmergebot added a commit that referenced this pull request Jun 10, 2024
This reverts commit 19b31d8.

Reverted #127698 on behalf of https://github.com/clee2000 due to broke (executorch?) internal tests D58295865 ([comment](#127696 (comment)))
pytorchmergebot added a commit that referenced this pull request Jun 10, 2024
This reverts commit b741819.

Reverted #127696 on behalf of https://github.com/clee2000 due to broke (executorch?) internal tests D58295865 ([comment](#127696 (comment)))
@pytorchmergebot

Copy link
Copy Markdown
Collaborator

@BowenBao your PR has been successfully reverted.

@clee2000

clee2000 commented Jun 10, 2024 •

Copy link
Copy Markdown
Contributor

stacktrace

Details
======================================================================
ERROR: test_backend_with_compiler_delegate_and_operator_with_two_modules (executorch.exir.backend.test.test_backends.TestBackends) (extract_delegate_segments=False)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "<truncated path>/executorch/exir/tracer.py", line 665, in dynamo_trace
    return torchdynamo.export(
  File "<truncated path>/torch/_dynamo/eval_frame.py", line 1378, in inner
    result_traced = opt_f(*args, **kwargs)
  File "<truncated path>/torch/nn/modules/module.py", line 1552, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "<truncated path>/torch/nn/modules/module.py", line 1561, in _call_impl
    return forward_call(*args, **kwargs)
  File "<truncated path>/torch/_dynamo/eval_frame.py", line 432, in _fn
    return fn(*args, **kwargs)
  File "<truncated path>/torch/nn/modules/module.py", line 1552, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "<truncated path>/torch/nn/modules/module.py", line 1561, in _call_impl
    return forward_call(*args, **kwargs)
  File "<truncated path>/torch/_dynamo/convert_frame.py", line 1115, in __call__
    return self._torchdynamo_orig_c
...
...
...
<truncated path>/torch/_dynamo/symbolic_convert.py", line 742, in call_function
    self.push(fn.call_function(self, args, kwargs))
  File "<truncated path>/torch/_dynamo/variables/higher_order_ops.py", line 1151, in call_function
    real_sub_args = pytree.tree_map_only(
  File "<truncated path>/torch/utils/_pytree.py", line 1120, in tree_map_only
    return tree_map(map_only(__type_or_types_or_pred)(func), tree, is_leaf=is_leaf)
  File "<truncated path>/torch/utils/_pytree.py", line 948, in tree_map
    return treespec.unflatten(map(func, *flat_args))
  File "<truncated path>/torch/utils/_pytree.py", line 787, in unflatten
    leaves = list(leaves)
  File "<truncated path>/torch/utils/_pytree.py", line 1066, in wrapped
    return func(x)
  File "<truncated path>/torch/_dynamo/variables/higher_order_ops.py", line 1152, in <lambda>
    torch.fx.Proxy, lambda a: get_real_value(a.node, tx.output), p_args
  File "<truncated path>/torch/_dynamo/utils.py", line 1941, in get_real_value
    args, kwargs = torch.fx.node.map_arg(
  File "<truncated path>/torch/fx/node.py", line 744, in map_arg
    return map_aggregate(a, lambda x: fn(x) if isinstance(x, Node) else x)
  File "<truncated path>/torch/fx/node.py", line 752, in map_aggregate
    t = tuple(map_aggregate(elem, fn) for elem in a)
  File "<truncated path>/torch/fx/node.py", line 752, in <genexpr>
    t = tuple(map_aggregate(elem, fn) for elem in a)
  File "<truncated path>/torch/fx/node.py", line 752, in map_aggregate
    t = tuple(map_aggregate(elem, fn) for elem in a)
  File "<truncated path>/torch/fx/node.py", line 752, in <genexpr>
    t = tuple(map_aggregate(elem, fn) for elem in a)
  File "<truncated path>/torch/fx/node.py", line 762, in map_aggregate
    return fn(a)
  File "<truncated path>/torch/fx/node.py", line 744, in <lambda>
    return map_aggregate(a, lambda x: fn(x) if isinstance(x, Node) else x)
  File "<truncated path>/torch/_dynamo/utils.py", line 1943, in <lambda>
    lambda n: get_real_value(n, tracer),
  File "<truncated path>/torch/_dynamo/utils.py", line 1961, in get_real_value
    raise TorchRuntimeError(str(e)).with_traceback(e.__traceback__) from None
  File "<truncated path>/torch/_dynamo/utils.py", line 1958, in get_real_value
    real_value = run_node(tracer, node, args, kwargs, nn_module)
  File "<truncated path>/torch/_dynamo/utils.py", line 1922, in run_node
    raise RuntimeError(make_error_message(e)).with_traceback(
  File "<truncated path>/torch/_dynamo/utils.py", line 1911, in run_node
    return tracer.output_graph.get_submodule(node.target)
torch._dynamo.exc.TorchRuntimeError: Failed running get_attr L__self___lowerable(*(), **{}):
'OutputGraph' object has no attribute 'output_graph'

from user code:
   File "<truncated path>/executorch/exir/backend/test/test_backends.py", line 533, in forward
    b = self.lowerable(a)
  File "<truncated path>/torch/nn/modules/module.py", line 1561, in _call_impl
    return forward_call(*args, **kwargs)
  File "<truncated path>/executorch/exir/lowered_backend_module.py", line 336, in forward
    return executorch_call_delegate(self, *args)


The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<truncated path>/executorch/exir/backend/test/test_backends.py", line 92, in wrapper
    test_method(self, extract_delegate_segments=extract_delegate_segments)
  File "<truncated path>/executorch/exir/backend/test/test_backends.py", line 546, in test_backend_with_compiler_delegate_and_operator_with_two_modules
    exir.capture(composite_model, model_inputs, exir.CaptureConfig())
  File "<truncated path>/executorch/exir/capture/_capture.py", line 232, in capture
    graph_module, _ = dynamo_trace(
  File "<truncated path>/executorch/exir/tracer.py", line 686, in dynamo_trace
    raise InternalError(
executorch.exir.error.InternalError: torchdynamo internal error occured. Please see above stacktrace

cc @jansel as reviewer and @digantdesai as edge oncall

@BowenBao

Copy link
Copy Markdown
Collaborator Author

Hi @clee2000, @digantdesai from stacktrace it appears an OutputGraph instance is passed as tracer into run_node. Is this expected or should it be fixed upstream?

  File "<truncated path>/torch/_dynamo/utils.py", line 1911, in run_node
    return tracer.output_graph.get_submodule(node.target)
torch._dynamo.exc.TorchRuntimeError: Failed running get_attr L__self___lowerable(*(), **{}):
'OutputGraph' object has no attribute 'output_graph'

pytorch-bot Bot pushed a commit that referenced this pull request Jun 10, 2024
Test Plan:
`buck2 test @//mode/dev-nosan //executorch/exir/backend/...`
https://www.internalfb.com/intern/testinfra/testrun/12666373989243932

Differential Revision: D58384518
@angelayi

Copy link
Copy Markdown
Contributor

@BowenBao #128358 should fix the issue your PR is running into

angelayi added a commit to angelayi/pytorch that referenced this pull request Jun 10, 2024
Summary: Pull Request resolved: pytorch#128358

Test Plan:
`buck2 test @//mode/dev-nosan //executorch/exir/backend/...`
https://www.internalfb.com/intern/testinfra/testrun/12666373989243932

Differential Revision: D58384518
angelayi added a commit to angelayi/pytorch that referenced this pull request Jun 11, 2024
Summary: Pull Request resolved: pytorch#128358

Test Plan:
`buck2 test @//mode/dev-nosan //executorch/exir/backend/...`
https://www.internalfb.com/intern/testinfra/testrun/12666373989243932

Differential Revision: D58384518
pytorchmergebot pushed a commit that referenced this pull request Jun 11, 2024
Test Plan:
`buck2 test @//mode/dev-nosan //executorch/exir/backend/...`
https://www.internalfb.com/intern/testinfra/testrun/12666373989243932

Differential Revision: D58384518

Pull Request resolved: #128358
Approved by: https://github.com/ydwu4
pytorchmergebot pushed a commit that referenced this pull request Jun 11, 2024
facebook-github-bot pushed a commit to pytorch/benchmark that referenced this pull request Jun 12, 2024
Summary:
Fixes #124858

X-link: pytorch/pytorch#127696
Approved by: https://github.com/jansel
ghstack dependencies: #127695

Reviewed By: clee2000

Differential Revision: D58443876

fbshipit-source-id: 87cccfb8ed74a6cfb3d3376b49fc84831c940c1e
pytorchbot pushed a commit that referenced this pull request Jun 12, 2024
Test Plan:
`buck2 test @//mode/dev-nosan //executorch/exir/backend/...`
https://www.internalfb.com/intern/testinfra/testrun/12666373989243932

Differential Revision: D58384518

Pull Request resolved: #128358
Approved by: https://github.com/ydwu4

(cherry picked from commit 4345d98)
TharinduRusira pushed a commit to TharinduRusira/pytorch that referenced this pull request Jun 14, 2024
TharinduRusira pushed a commit to TharinduRusira/pytorch that referenced this pull request Jun 14, 2024
TharinduRusira pushed a commit to TharinduRusira/pytorch that referenced this pull request Jun 14, 2024
This reverts commit 19b31d8.

Reverted pytorch#127698 on behalf of https://github.com/clee2000 due to broke (executorch?) internal tests D58295865 ([comment](pytorch#127696 (comment)))
TharinduRusira pushed a commit to TharinduRusira/pytorch that referenced this pull request Jun 14, 2024
This reverts commit b741819.

Reverted pytorch#127696 on behalf of https://github.com/clee2000 due to broke (executorch?) internal tests D58295865 ([comment](pytorch#127696 (comment)))
TharinduRusira pushed a commit to TharinduRusira/pytorch that referenced this pull request Jun 14, 2024
Test Plan:
`buck2 test @//mode/dev-nosan //executorch/exir/backend/...`
https://www.internalfb.com/intern/testinfra/testrun/12666373989243932

Differential Revision: D58384518

Pull Request resolved: pytorch#128358
Approved by: https://github.com/ydwu4
TharinduRusira pushed a commit to TharinduRusira/pytorch that referenced this pull request Jun 14, 2024
TharinduRusira pushed a commit to TharinduRusira/pytorch that referenced this pull request Jun 14, 2024
ignaciobartol pushed a commit to ignaciobartol/pytorch that referenced this pull request Jun 14, 2024
Test Plan:
`buck2 test @//mode/dev-nosan //executorch/exir/backend/...`
https://www.internalfb.com/intern/testinfra/testrun/12666373989243932

Differential Revision: D58384518

Pull Request resolved: pytorch#128358
Approved by: https://github.com/ydwu4
ignaciobartol pushed a commit to ignaciobartol/pytorch that referenced this pull request Jun 14, 2024
ignaciobartol pushed a commit to ignaciobartol/pytorch that referenced this pull request Jun 14, 2024
atalman pushed a commit that referenced this pull request Jun 18, 2024
[dynamo] Fix for #127696 (#128358)

Test Plan:
`buck2 test @//mode/dev-nosan //executorch/exir/backend/...`
https://www.internalfb.com/intern/testinfra/testrun/12666373989243932

Differential Revision: D58384518

Pull Request resolved: #128358
Approved by: https://github.com/ydwu4

(cherry picked from commit 4345d98)

Co-authored-by: Angela Yi <angelayi@meta.com>
@github-actions
github-actions Bot deleted the gh/BowenBao/318/head branch July 12, 2024 01:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants