Replace broken mouse click with ClickCoordinateEvent #3663
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- Add ClickCoordinateEvent to handle coordinate-based clicking via CDP - Replace broken page.mouse.click() with CDP Input.dispatchMouseEvent - Add safety checks for file inputs, select elements, and print dialogs - Add get_dom_element_at_coordinates() to BrowserSession for element detection - Add force parameter to ClickElementAction to optionally skip safety checks - Update _click_by_coordinate to dispatch ClickCoordinateEvent instead of using mouse This fixes coordinate clicking functionality by using CDP directly instead of the broken Actor mouse object, matching the implementation from mert/everything_coordinates.
Agent Task Evaluation Results: 2/2 (100%)View detailed results
Check the evaluate-tasks job for detailed task execution logs. |
Collaborator
Evaluation ResultsCommit: 📊 View DetailsSummary
Automated evaluation by browser-use |
There was a problem hiding this comment.
This PR is being reviewed by Cursor Bugbot
Details
You are on the Bugbot Free tier. On this plan, Bugbot will review limited PRs each billing cycle.
To receive Bugbot reviews on all of your PRs, visit the Cursor dashboard to activate Pro and start your 14-day free trial.
Collaborator
Evaluation ResultsCommit: 📊 View DetailsSummary
Automated evaluation by browser-use |
Contributor
There was a problem hiding this comment.
1 issue found across 5 files
Prompt for AI agents (all 1 issues)
Understand the root cause of the following 1 issues and fix them.
<file name="browser_use/browser/watchdogs/default_action_watchdog.py">
<violation number="1" location="browser_use/browser/watchdogs/default_action_watchdog.py:232">
`get_dom_element_at_coordinates` returns nodes without tag names or attributes, so the new safety checks in `on_ClickCoordinateEvent` never trigger and risky elements (file inputs, selects, print buttons) can still be clicked.</violation>
</file>
Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR
Contributor
There was a problem hiding this comment.
1 issue found across 5 files
Prompt for AI agents (all 1 issues)
Understand the root cause of the following 1 issues and fix them.
<file name="browser_use/browser/session.py">
<violation number="1" location="browser_use/browser/session.py:1968">
Constructed nodes for coordinate clicks always have an empty node_name, so tag‑based safety checks (file inputs, <select>, print buttons) never execute. Populate node_name from DOM.getNodeForLocation before returning the node.</violation>
</file>
Reply to cubic to teach it or ask questions. Re-run a review with @cubic-dev-ai review this PR
Collaborator
Evaluation ResultsCommit: 📊 View DetailsSummary
Automated evaluation by browser-use |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This fixes coordinate clicking functionality by using CDP directly instead of the broken Actor mouse object, matching the implementation from mert/everything_coordinates.
Note
Introduce ClickCoordinateEvent and replace mouse-based coordinate clicks with CDP Input.dispatchMouseEvent, adding safety checks and an optional force mode.
ClickCoordinateEvent(x/y/button,forceflag) with timeout.get_dom_element_at_coordinates(x, y)usingDOM.getNodeForLocationreturning minimalEnhancedDOMTreeNode.ClickCoordinateEventwith safety checks (file input,<select>, print -> PDF) and optionalforce._click_on_coordinateusing CDPInput.dispatchMouseEvent(move/press/release) with timeouts and logging.service.py,views.py)ClickElementActionwithforce: bool.page.mouse.clickwith dispatch ofClickCoordinateEvent; convert LLM-scaled coords; return metadata/validation errors; handleBrowserError.Written by Cursor Bugbot for commit 36b2b93. Configure here.
Summary by cubic
Fixes broken coordinate-based clicking by switching to CDP Input.dispatchMouseEvent. Adds ClickCoordinateEvent with safety checks and an optional force mode.
New Features
Bug Fixes
Written for commit b2c56f6. Summary will update automatically on new commits.