Add ArbigentHint feature for app-provided domain context #361
Conversation
📝 WalkthroughWalkthroughAdds an "appHints" lifecycle: collect ArbigentHint strings from view tree attributes, include them in UI tree data, thread them through context/prompt formatting, and inject them into the user prompt passed to the AI client. Changes
Sequence Diagram(s)sequenceDiagram
participant View as View Hierarchy
participant Device as ArbigentDevice
participant Holder as ArbigentContextHolder
participant Template as UserPromptTemplate
participant OpenAI as OpenAIAi
participant AI as OpenAI API
View->>Device: viewTreeString()
Device->>Device: findAllArbigentHints() (scan attributes for "ArbigentHint:")
Device->>Holder: ArbigentUiTreeStrings(allTree, optimizedTree, appHints)
Holder->>Template: format(goal, steps, appHints)
Template->>Template: build/apply APP_HINTS block when non-empty
Template->>Holder: formatted prompt
Holder->>OpenAI: buildPrompt(formattedPrompt, appHints)
OpenAI->>AI: send prompt (includes hint context)
AI-->>OpenAI: response
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Areas requiring attention:
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
✨ Finishing touches🧪 Generate unit tests (beta)
📜 Recent review detailsConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro 📒 Files selected for processing (1)
🧰 Additional context used🧠 Learnings (1)📓 Common learnings🪛 markdownlint-cli2 (0.18.1)README.md90-90: Unordered list indentation (MD007, ul-indent) 91-91: Unordered list indentation (MD007, ul-indent) 95-95: Unordered list indentation (MD007, ul-indent) ⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (12)
🔇 Additional comments (1)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (2)
arbigent-core/src/test/kotlin/io/github/takahirom/arbigent/UserPromptTemplateTest.kt (1)
116-131: Consider verifying the exact hint block format.The test uses
contains()assertions which confirm the presence of elements but don't validate the precise structure of the hint block. Consider adding an assertion that checks the exact rendered format to catch potential formatting issues.Example enhancement:
+ // Verify exact format of hint block + assertTrue(result.contains(""" +<HINT_FROM_APP> +- First hint +- Second hint +</HINT_FROM_APP> + """.trim()))arbigent-core/src/test/kotlin/io/github/takahirom/arbigent/TreeNodeExtensionsTest.kt (1)
8-112: Reduce boilerplate with a helper function.All tests create TreeNode instances with repetitive boilerplate parameters. This makes the tests verbose and harder to maintain. Consider introducing a helper function to reduce duplication and improve readability.
Add a helper function at the class level:
class TreeNodeExtensionsTest { private fun createTestNode( accessibilityText: String? = null, children: List<TreeNode> = emptyList() ) = TreeNode( attributes = accessibilityText?.let { mutableMapOf("accessibilityText" to it) } ?: mutableMapOf(), children = children, clickable = false, enabled = true, focused = false, checked = false, selected = false ) // Then simplify tests like: @Test fun `findAllArbigentHints should collect hint from single node`() { val node = createTestNode(accessibilityText = "ArbigentHint:Test hint") assertEquals(listOf("Test hint"), node.findAllArbigentHints()) } }
📜 Review details
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
arbigent-ai-openai/src/main/java/io/github/takahirom/arbigent/OpenAIAi.kt(2 hunks)arbigent-core-model/src/commonMain/kotlin/ArbigentResult.kt(1 hunks)arbigent-core/src/main/java/io/github/takahirom/arbigent/ArbigentContextHolder.kt(2 hunks)arbigent-core/src/main/java/io/github/takahirom/arbigent/ArbigentDevice.kt(2 hunks)arbigent-core/src/main/java/io/github/takahirom/arbigent/UserPromptTemplate.kt(3 hunks)arbigent-core/src/test/kotlin/io/github/takahirom/arbigent/TreeNodeExtensionsTest.kt(1 hunks)arbigent-core/src/test/kotlin/io/github/takahirom/arbigent/UserPromptTemplateTest.kt(2 hunks)
🧰 Additional context used
🧠 Learnings (1)
📓 Common learnings
Learnt from: takahirom
Repo: takahirom/arbigent PR: 329
File: arbigent-core/src/main/java/io/github/takahirom/arbigent/ArbigentAgent.kt:0-0
Timestamp: 2025-07-08T15:34:42.649Z
Learning: takahirom prefers simplified implementations over complex caching libraries, favoring manual cache management through methods like clearResolvedGoalCache() rather than using bounded cache implementations with automatic eviction policies.
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (12)
- GitHub Check: store-screenshot-test
- GitHub Check: compare-screenshot-test
- GitHub Check: cli-e2e-ios
- GitHub Check: cli-e2e-web
- GitHub Check: cli-e2e-android (2, 2)
- GitHub Check: cli-e2e-android (1, 2)
- GitHub Check: build
- GitHub Check: build (macos-latest)
- GitHub Check: Analyze (java-kotlin)
- GitHub Check: build (ubuntu-latest)
- GitHub Check: test
- GitHub Check: build (windows-latest)
🔇 Additional comments (12)
arbigent-core/src/main/java/io/github/takahirom/arbigent/UserPromptTemplate.kt (2)
14-18: LGTM: APP_HINTS placeholder integration.The placement of
APP_HINTSafterGOALin the template is appropriate—it provides domain context early in the prompt before step and UI state details.
88-100: LGTM: Clean conditional hint rendering.The implementation correctly handles both empty and non-empty hint lists. The
HINT_FROM_APPXML-style tags provide clear delineation for the AI to parse.arbigent-core-model/src/commonMain/kotlin/ArbigentResult.kt (1)
99-104: LGTM: Backward-compatible model extension.The default
emptyList()ensures existing serialized data deserializes correctly without the new field.arbigent-core/src/main/java/io/github/takahirom/arbigent/ArbigentContextHolder.kt (1)
96-110: LGTM: Clean pass-through of appHints.The parameter addition maintains backward compatibility via the default value and correctly forwards hints to the template formatter.
arbigent-ai-openai/src/main/java/io/github/takahirom/arbigent/OpenAIAi.kt (2)
214-224: LGTM: Proper integration of appHints into prompt building.The hints are correctly extracted from
uiTreeStrings.appHintsand passed through the prompt construction pipeline.
339-357: LGTM: buildPrompt correctly forwards appHints.The private helper properly accepts and passes the hints to the context holder's prompt method.
arbigent-core/src/main/java/io/github/takahirom/arbigent/ArbigentDevice.kt (2)
265-286: LGTM: viewTreeString() now includes collected hints.The integration correctly passes collected hints to
ArbigentUiTreeStrings. Note that the fallback return on line 282-285 doesn't includeappHints, which defaults toemptyList()via the data class default—this is acceptable behavior for the error case.
705-726: LGTM: Well-documented hint collection.The implementation is clean and the KDoc clearly explains the usage pattern for app developers. The recursive traversal correctly collects all hints from the view tree.
arbigent-core/src/test/kotlin/io/github/takahirom/arbigent/UserPromptTemplateTest.kt (3)
6-7: LGTM!The new assertion imports are appropriate for the test methods added.
134-147: LGTM!Good test coverage for the empty hints scenario. The assertions correctly verify that no hint block is rendered when the list is empty.
150-167: LGTM!Excellent test for validating the positioning of hints within the template. The custom assertion messages make failures easy to diagnose.
arbigent-core/src/test/kotlin/io/github/takahirom/arbigent/TreeNodeExtensionsTest.kt (1)
8-112: Test suite comprehensively validatesfindAllArbigentHints()extension function.The five tests correctly verify the implementation across all critical scenarios:
- Single and nested hint extraction with correct prefix handling
- Depth-first traversal order (parent hints collected before child hints)
- Graceful handling of non-hint text and missing attributes
- Deep tree structures with mixed hint/non-hint nodes
All test assertions align with the actual implementation in
ArbigentDevice.kt, which extracts hints by checking for the "ArbigentHint:" prefix, removing it, and recursively collecting hints from all children.
What
Add ArbigentHint feature that allows apps to provide domain-specific context to Arbigent AI via contentDescription.
Apps can set
view.contentDescription = "ArbigentHint:some context"and Arbigent will collect all hints from the view tree and include them in the AI prompt.Why
Visual UI testing cannot distinguish between screens that look identical but have different domain meanings (e.g., a video player showing realtime content vs vod content). This feature enables apps to provide semantic context that helps AI make better decisions.