Skip to content

feat(extension): Chrome MV3 side-panel extension — full implementation - #510

Merged
rschardosin merged 38 commits into
mainfrom
feature/extension-ui-fixes
Sep 10, 2026
Merged

rschardosin merged 38 commits into
mainfrom
feature/extension-ui-fixes

Conversation

@rschardosin

@rschardosin rschardosin commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Chrome MV3 Side-Panel Extension

This PR implements the complete Astonish Chrome extension, enabling users to chat with Astonish from any browser tab via a side panel. The extension connects to a running Astonish Studio backend and provides page-aware AI assistance.

Core Features

  • Side panel chat UI — Full chat interface with session management (create, switch, delete), markdown rendering (including tables), streaming responses, and stop button
  • Page tools — The AI can read page content, click elements, and fill forms on the current tab using CDP-based automation via chrome.debugger
  • Cross-origin iframe support — Page tools pierce cross-origin iframes using allFrames content script injection and a frame coordinator
  • Per-tab session isolation — Each browser tab maintains its own active session independently using chrome.storage.session keyed by tab ID
  • Session categorization — Extension sessions use a dedicated appName ("astonish-extension") so the dropdown shows all extension sessions across tabs, separate from Studio chat sessions

Architecture

  • Service worker (background/service-worker.ts) — Handles side panel lifecycle, page tool dispatch, CDP operations, and tab cleanup
  • Side panel (sidepanel/main.ts) — Chat UI, session picker, SSE streaming, markdown rendering
  • Content scripts (content/) — DOM capture, page click/fill, apply operations, cross-origin frame coordination
  • Shared libraries (lib/) — Astonish API client, auth/SSO, extension session storage, page tool orchestration

Backend Changes

  • pkg/api/chat_handlers.go — Added optional appName field to StudioChatRequest; sessions are created under the caller-specified app name when provided
  • pkg/api/session_handlers.go — Added ?app= query parameter to GET /api/studio/sessions for filtered session listing by app name
  • pkg/api/session_handlers_test.go — Tests for appName deserialization

Extension UI & UX

  • Classic color theme with Astonish icon
  • Session picker dropdown with delete confirmation
  • Streaming indicator with stop button, input disabled during streaming
  • Tool call/result notices grouped and collapsible
  • Markdown tables rendered correctly

Key Technical Decisions

  • Tab ID via URL param — The service worker embeds ?tabId=X in the side panel URL at open time; the panel reads it from location.search so it never changes even when the user switches tabs
  • Page tool targeting — Side panel sends its tabId in every message; service worker uses chrome.tabs.get(tabId) instead of activeTab() to ensure tools target the correct tab
  • CDP for interactionspage_click and page_fill use chrome.debugger (CDP Input.dispatchMouseEvent / DOM.focus + Input.insertText) for reliable cross-origin element interaction
  • Server-side session filtering — Extension passes ?app=astonish-extension to the sessions endpoint instead of client-side filtering, so sessions created in any tab are visible in all tabs

Testing

  • 96 extension tests pass (Vitest)
  • Extension builds clean (Vite production bundle)
  • Go build and API tests pass
  • golangci-lint clean

Files Changed

Extension (new):

  • extension/ — Complete Chrome MV3 extension: manifest, service worker, side panel, content scripts, shared libraries, tests, Vite build config

Backend (modified):

  • pkg/api/chat_handlers.go — Optional appName in chat request
  • pkg/api/session_handlers.go — ?app= query param for session listing
  • pkg/api/session_handlers_test.go — New test file
  • .gitignore — Added extension/dist/

- Remove redundant 'New' button from session bar (duplicate of 'New chat' option in dropdown)
- Clean up dead JavaScript references (newSessionButton variable, event listeners, disabled toggles)
- Remove dead CSS rule for #new-session
- Add deleteExtensionSession() function to extension-sessions.ts for per-session deletion
- Add trash button (🗑️) to session bar that:
  - Prevents deletion of 'New chat' (empty session)
  - Shows confirmation dialog
  - Deletes session and switches to next available or 'New chat'
  - Properly disables during streaming
- All 84 tests pass, build clean
…oordinator

- Add chrome.webNavigation permission to enumerate frames
- Add allFrames: true to content script injection in service worker
- Add MSG_FRAME_TOOL message type for frame relay
- Add getAccessibleDocuments() in dom-tools to traverse same-origin iframes
- Update collectCandidates, snapshot to query all accessible documents
- Add frame detection and iframe source tagging in snapshot output
- Add frame fan-out in service worker: getAllFrameIds, sendToFrame (with retry), mergeFrameResults
- Add per-frame DOM tool execution: runPageToolInFrame, collectCandidatesFrameLocal, snapshotFrameInteractive, queryFrame
- Add iframeSource() helper to tag refs from iframe elements
- Add INTERACTIVE selector support for [tabindex="0"] (calendar day cells, SAPUI5 components)
- Update architecture docs to document iframe support (same-origin traversal, cross-origin skip)
- All 90 tests pass; both Vite bundles build successfully

The extension now discovers and interacts with elements inside same-origin iframes.
For SAP SAPUI5 calendars with day cells in cross-origin iframe, refs now appear tagged
with (iframe: ...) and are clickable/fillable via page_click, page_fill, etc.
- Add extension/src/background/cdp-input.ts with cdpClick() and cdpFill()
  using chrome.debugger.attach/sendCommand/detach for real browser input events
- Add 'debugger' permission to extension/manifest.json
- Add getElementTabRect() helper in dom-tools.ts that walks iframe chain
  to produce tab-absolute coordinates for CDP dispatch
- Update click() to return bounding rect instead of dispatching synthetic
  DOM events; DOM .click() kept only for zero-rect (hidden/jsdom) elements
- Update fill() to return FillOutcome with rect; DOM .value= kept as
  belt-and-suspenders (so React/Angular work); CDP fills SAPUI5/custom inputs
- Add rect field to ClickOutcome, FillOutcome, PageToolRun, FrameToolResult,
  PageToolResult types; FrameToolResult.ok made optional (unused by mergeFrameResults)
- Update service-worker.ts to import cdpClick/cdpFill, pass rect through
  mergeFrameResults, and dispatch CDP input after runPageToolAllFrames
- Update tests: add rect assertions to page_click/page_fill tests, add new
  zero-rect fallback test (25 tests pass)
- Update docs/architecture/chrome-extension.md with CDP input dispatch section
  and corrected permission list

All 91 tests pass; both Vite bundles build cleanly.
Track winning frameId through mergeFrameResults; in runPageToolInTab,
when frameId != 0, query the top frame via chrome.scripting.executeScript
to find the iframe element's bounding rect and add it as an offset to the
element's iframe-local coordinates before CDP dispatch.

The content script's getElementTabRect() cannot traverse frameElement for
cross-origin iframes (SecurityError), so it returns only iframe-local coords.
This fix closes the gap so Input.dispatchMouseEvent lands at the correct
tab-absolute position for elements inside cross-origin iframes (e.g. SAP
Fiori CAT2 calendar embedded in an iframe).

URL matching uses origin+pathname comparison (not exact URL equality) to
handle cases where the frame URL has query params the iframe src does not.
Multi-step tasks like SAP CAT2 booking (snapshot → click day → snapshot →
click task → snapshot → fill → save) can easily exceed 8 rounds. 15 gives
enough room for complex workflows while still providing a safety ceiling.
- Add a red stop button (filled square icon) inside the composer form,
  matching the Studio chat's variant=destructive stop button styling
- Show the stop button while streaming; hide it when idle
- Disable the textarea and show 'Agent is responding…' placeholder while
  streaming, matching the Studio chat behavior
- On stop: abort the SSE connection, call POST /sessions/{id}/stop to
  kill the backend runner, and show a 'Stopped by user.' notice
- Add stopChat() to astonish-client.ts for the stop API call
- Add .stop-btn CSS class and .composer textarea:disabled opacity
Update CSS variables to use Astonish Classic theme (indigo/blue) instead
of Nova (pink/purple). This is the default theme and makes the extension
consistent with the rest of the Astonish UI.

Changes:
- --brand: #ff6b9d → #6366f1 (Nova pink → Classic indigo)
- --background: #160b1f → #0f172a (Nova dark → Classic dark slate)
- --card: #1d0f28 → #1e293b
- --foreground: #fceef7 → #f1f5f9
- --text-secondary/muted: Nova mauve → Classic slate
- --danger: #f05252 → #ef4444
- --info: #b478ff → #a78bfa (purple → violet)
The EXTENSION_PAGE_TOOLS_INSTRUCTIONS were being appended to the context
on every page-tool round, causing the system instructions to be echoed back
to the user in the visible chat. The instructions are already part of the
initial buildSystemContext() call, so they should NOT be re-appended after
each tool execution.

After page tools run, context should contain only formatPageToolResults(),
not the full instructions again.

Fixes the verbose 'user[timestamp] The Chrome extension ran...' leak.
Add Astonish logo as the extension icon in three sizes:
- icon-16.png (16x16 for Chrome toolbar)
- icon-48.png (48x48 for Chrome menu)
- icon-128.png (128x128 for Chrome Web Store)

Icons are exported from the Astonish logo SVG and configured in:
- manifest.json: action.default_icons and icons sections
- vite.config.ts: updated icon path handling for build

The extension toolbar button now displays the Astonish branding instead
of a generic placeholder icon.
Two UI defects in the extension side panel:

1. Markdown tables showed raw pipe syntax instead of rendered HTML tables.
   The renderMarkdown() parser now handles table rows (| col | col |),
   separator rows (| --- | --- |), and emits proper <table>/<thead>/<tbody>
   elements. Table CSS was already in panel.css; added overflow-x: auto
   for wide tables in the narrow side panel.

2. Tool call/result notices appeared as separate bubbles cluttering the
   transcript. Now they are grouped into a single collapsible 'Used N tools'
   block (matching Studio chat's tool fold pattern). Click to expand shows
   individual tool names. Both streaming (live) and history rendering use
   the same grouping.

Added 2 new tests for table rendering (93 total tests passing).
The previous approach of using chrome.tabs.query() from the side panel context
was unreliable and would often return tabId=0, causing all tabs to share the same
session storage slot. This fix uses chrome.runtime.sendMessage() to have the
service worker directly send the tab ID to the side panel based on the sender
context, which is reliable and works correctly across all tabs.
…tab ID resolution

- vite.config.ts: manualChunks routes extension-sessions and messages into the
  sidepanel entry chunk, eliminating the 'cross-world extension resource mismatch'
  modulepreload warning Chrome was showing in the extension errors panel

- service-worker.ts: MSG_GET_TAB_ID handler now queries active tab via
  chrome.tabs.query (lastFocusedWindow) instead of reading sender.tab which is
  undefined for side panel extension pages — fixes the tabId always being 0 and
  causing all tabs to share the same session storage slot
Chrome MV3 flags modulepreload for extension resources as a 'cross-world
extension resource mismatch' error. The chunks still load correctly via dynamic
imports, but the preload tags cause console warnings. Strip them out in the
build plugin's HTML post-processing step.
Instead of manually keying sessions by tabId in chrome.storage.local, use
chrome.storage.session which is automatically isolated per browser tab by Chrome.
This is simpler, more reliable, and handles tab cleanup automatically (session
storage is cleared when a tab closes).

Benefits:
- No manual tab ID resolution needed
- No message passing between service worker and side panel
- No service worker message listener required
- Each tab's side panel gets its own storage context by default
- Stale entries auto-cleaned when tab closes

Changed extension-sessions.ts to use session storage with a single state key
instead of a per-tab map in local storage. Updated tests accordingly.
… isolation

All previous approaches failed because there is no API to get a tab ID from
inside a side panel page context. The correct solution (per Chrome docs and
Chromium extensions mailing list) is to embed the tabId in the side panel URL
as a query parameter when the service worker opens the panel.

Changes:
- service-worker.ts: Disable openPanelOnActionClick (which suppresses
  action.onClicked). Use chrome.action.onClicked to explicitly open the panel
  with path 'sidepanel.html?tabId=X', so each tab gets its own URL.

- main.ts: Read tabId synchronously from location.search — zero async, zero
  message passing, 100% reliable.

- extension-sessions.ts: Restored per-tab map keyed by tabId in
  chrome.storage.session (shared in-memory store, keyed by tabId). This gives
  real isolation: tab 42 reads map[42], tab 99 reads map[99].

- Tests: Restored all per-tab isolation tests including removeTabState and
  different-tabs tests.
Remove default_path from manifest.json so the panel is not globally enabled.
Set chrome.sidePanel.setOptions({ enabled: false }) globally. Only enable the
panel per-tab when the user clicks the extension icon via action.onClicked.

This makes the side panel behave like Chrome DevTools:
- Click the icon on a tab → panel opens for that tab only
- Switch to another tab → panel is not visible
- Each tab that has the panel open has its own independent session

Also cleaned up the now-unused MSG_GET_TAB_ID message handler since tab IDs
are embedded in the panel URL.
The side panel now includes its tabId in every sendMessage call
(MSG_GET_CONTEXT, MSG_PAGE_TOOL, MSG_APPLY). The service worker reads
this tabId and uses chrome.tabs.get(tabId) instead of querying for
the active tab. This prevents page tools from drifting to whichever
tab the user is currently viewing when they switch away from the tab
where the agent is working.
@rschardosin
rschardosin force-pushed the feature/extension-ui-fixes branch from 071b287 to b560a91 Compare September 10, 2026 04:49
- Add optional AppName field to StudioChatRequest; when set, sessions are
  created under that app name instead of the default 'astonish'.
- Add optional ?app= query parameter to GET /api/studio/sessions; defaults
  to studioChatAppName for backward compatibility.
- Define EXTENSION_APP_NAME = 'astonish-extension' constant in the extension.
- connectChat now sends appName in the request body so new sessions are tagged.
- fetchSessions now appends ?app=astonish-extension so the dropdown shows all
  extension sessions regardless of which tab created them, not just sessions
  whose IDs are in the tab's local storage.
- Remove client-side filterExtensionSessions call from refreshExtensionSessions;
  server-side filtering via ?app= replaces it. Sessions created in another tab
  are now surfaced by adding backend-returned sessions missing from local state.
- Add pkg/api/session_handlers_test.go testing StudioChatRequest appName field.
Chrome MV3 requires the default_path field in the side_panel section.
Without it, the extension fails to load with:
  'Error at key side_panel.default_path. Manifest key is required.'
@rschardosin rschardosin changed the title extension: remove duplicate New button, add delete session feature feat(extension): Chrome MV3 side-panel extension — full implementation Sep 10, 2026
This fixes critical app-name namespace defects that prevented the extension
from reading, persisting, and deleting sessions correctly:

Backend fixes:
- session_handlers.go: StudioSessionHandler and StudioDeleteSessionHandler
  now read the optional ?app= query parameter (defaults to studioChatAppName
  for backward compatibility), so extension sessions (appName=astonish-extension)
  can be fetched and deleted using the correct namespace
- chat_handlers.go: /new and /distill commands now use effectiveApp
  instead of hardcoded studioChatAppName
- chat_utils.go, chat_runner.go: persistRunError and persistSessionMessage
  now accept an optional appName parameter (empty string uses context fallback),
  so slash commands persist events under the correct app namespace
- chat_handlers.go: Added context helpers (getAppNameFromContext) and
  pass effectiveApp to handleSlashCommand so all session operations use
  the caller-provided app name

Tests:
- Updated handleSlashCommand test call to pass effectiveApp parameter

Impact:
- Extension sessions created under appName=astonish-extension are now
  accessible to all handlers that read/modify/delete them
- Session history reload (GET detail) works correctly for extension sessions
- Slash command persistence (/new, /distill) works with the correct app name
- All existing clients (Studio chat) remain unaffected (backward-compatible)
…ory calls

Critical #1+#2 (backend):
- newChatRunner now accepts an appName parameter and injects it into cr.ctx
  via context.WithValue(ctx, appNameContextKey, appName). This makes the
  context fallback in persistRunError/persistSessionMessage live (previously
  appNameContextKey was defined and read but never set).
- runner.New() now uses getAppNameFromContext(cr.ctx) instead of hardcoded
  studioChatAppName, so ADK transcript events are persisted under the same
  app namespace as the session record.
- StudioChatHandler passes effectiveApp to newChatRunner.
- All test call sites updated to pass studioChatAppName as the appName argument.

Medium #3 (extension delete):
- Added deleteSession() to astonish-client.ts — sends DELETE
  /api/studio/sessions/{id}?app=astonish-extension.
- main.ts delete button handler now calls deleteSession() before
  deleteExtensionSession() (best-effort: local cleanup proceeds even on
  server error so the UI stays consistent).

Medium #4 (extension history):
- fetchSessionHistory() now appends ?app=astonish-extension to the GET URL
  so the backend reads from the correct session namespace.

Tests:
- fetchSessionHistory test updated to assert ?app= is present in the URL.
- Two new tests for deleteSession: success (204) and error (404 → throws).
…trust docs

- Fix CI Build check: use 'npm test' instead of 'npx vitest run' so the
  lockfile-pinned vitest@5.0.0 is used; fixes pre-existing TemplateDetail
  frontend test failure unrelated to PR #510
- Add build-extension job to build.yml: npm ci, tsc --noEmit, npm test,
  npm run build in extension/ on every push/PR
- Wire build-extension into Makefile build-all so 'make build-all' includes
  the extension bundle
- Expand session_handlers_test.go with TestStudioSessionsAppNameQueryParam
  (tests ?app= defaulting logic) and TestStudioChatRequestEffectiveApp
  (tests caller-appName vs studioChatAppName precedence)
- Add sender.id === chrome.runtime.id guard to service-worker onMessage
  as defence-in-depth against future onMessageExternal exposure
- Add 'Security and trust model' section to chrome-extension.md covering
  model-driven input + 15-round limit, https://* justification,
  debugger permission rationale, token storage trade-offs, and sender
  validation reasoning
- Add DOM.Iterable to tsconfig.json lib array — fixes all 10 tsc errors
  for NodeListOf/HTMLCollection iteration (Critical #1)
- Add 'typecheck' npm script so local devs can run 'npm run typecheck'
  matching the CI gate; update build.yml to use it
- Fix skipPageTools let-before-declaration in main.ts — move the
  declaration above its first use (found by tsc after DOM.Iterable fix)
- Add validAppName regex (^[a-z][a-z0-9_-]{0,63}$) and enforce it in
  StudioSessionsHandler, StudioSessionHandler, StudioDeleteSessionHandler,
  and StudioChatHandler — blocks path traversal, namespace squatting, and
  unbounded key lengths (Medium #2)
- Replace tautological session handler tests with tests that exercise the
  real validAppName regex, validate the resolution+validation code path
  end-to-end, and assert rejection of traversal/injection attempts (Medium #4)
…studioChatAppName

- Defense-in-depth: after GetSessionMeta (which looks up by globally-unique
  session ID without appName filtering), verify meta.AppName == appName and
  return 404 on mismatch. Applied to StudioSessionHandler (platform + file
  store paths) and StudioDeleteSessionHandler (platform path). This makes
  ?app= a real isolation boundary rather than advisory. (Medium #1)

- Replace hardcoded studioChatAppName with resolved appName in the
  personal-mode file-store paths of StudioSessionHandler: fleet transcript
  filepath.Join and the non-fleet fs.Get call. (Minor #1)

- Use npm ci instead of npm install in Makefile build-extension target so
  the lockfile is never mutated by a local build. (Medium #3)
…cleanup

- Move fleet Stop()/Cleanup()/Unregister() to AFTER the app-namespace
  verification in both platform-mode and personal-mode delete paths.
  Previously the fleet registry teardown (which destroys sandbox
  containers) ran unconditionally before any namespace check. Now the
  handler verifies meta.AppName == appName first, and only then proceeds
  to stop the fleet session. (Medium #1)

- Fail closed on GetSessionMeta errors in the platform-mode delete path:
  if the meta lookup errors or returns nil, return 404 instead of
  skipping the namespace check and falling through to Delete. (Medium #1)

- Add namespace verification to the personal-mode delete path: check
  file store meta.AppName before fleet cleanup and workspace deletion.

- Fix hardcoded studioChatAppName in personal-mode Delete call to use
  the resolved appName variable. (leftover from prior round)
- Refactor streamOnce() to track per-segment text: each text bubble
  only shows text from its current segment (since the last tool group),
  not the full accumulated assistant text. This fixes the regression
  where each new bubble showed all prior text concatenated.

- Add hadToolsSinceText flag: when a tool_call fires, sets the flag;
  when the next text chunk arrives, isNewSegment is raised and
  segmentStart advances to the current position in the full text.

- Update outer onText callback to create a new assistantBody when
  isNewSegment is true, so text arriving after tool groups gets its
  own bubble rather than appending to the previous one.

- Replace per-tool appendNotice() calls with appendOrUpdateToolGroup():
  consecutive tool calls share one collapsible <details> element
  (showing name + truncated args + completion checkmark). A new tool
  group is started whenever non-tool content appears between tools.

- Extend renderHistory() to use the same appendOrUpdateToolGroup()
  for tool_call messages, so historical sessions render with the same
  collapsible groups as live streaming.

- Add toolArgs field to HistoryMessage type for future args display
  in history (currently sent by backend but not yet used).

- Add CSS for .msg-tool-group, .tool-group-details, .tool-group-summary,
  .tool-group-item, .tool-call-name, .tool-args-preview, .tool-status.
Restore the correct indigo color palette that was accidentally corrupted
in the working tree (changed from #6366f1 indigo to #ff6b9d pink/magenta).
Also restore extension icons to manifest (icon-16, icon-48, icon-128) and
the correct side_panel.default_path (src/sidepanel/index.html instead of
sidepanel.html).
Update vite.config.ts to correctly copy icons from public/icons source
to dist/public/icons output, matching the manifest.json icon references.
Also update file name patterns from icon16.png to icon-16.png (with hyphens)
to match the actual icon filenames.
…ol groups

Restore the tab-scoped session architecture (currentTabId / resolveTabId)
that was lost when main.ts was edited from a corrupted working tree state.
Also re-apply the tool-group interleaving improvements on top of the correct
foundation.

Key fixes:
- Restore currentTabId + resolveTabId() from URL ?tabId= param (tab isolation)
- Restore MAX_PAGE_TOOL_ROUNDS = 15 (was reverted to 8)
- Restore all loadExtensionChat(currentTabId), replaceExtensionSessions(currentTabId, ...),
  startNewExtensionSession(currentTabId), rememberExtensionSession(currentTabId, ...)
  call sites with the tab-scoped API
- Re-apply appendOrUpdateToolGroup() for <details>-based collapsible groups
  with tool name + args preview + completion checkmarks
- Re-apply segment-text fix: each assistant bubble shows only text from its
  segment (since the last tool group), not the full accumulated text
- Re-apply isNewSegment signal threading through streamOnce → outer onText
  so text arriving after tool groups gets a fresh bubble
- Update renderHistory() to use appendOrUpdateToolGroup() for history replay
… delete

- handleSlashCommand: replace ctx := r.Context() with
  context.WithValue(r.Context(), appNameContextKey, effectiveApp) so
  all persistSessionMessage and persistDistillPreview calls inside
  resolve the correct app namespace instead of always defaulting to
  'astonish'. Fixes silent message drop for extension sessions.

- StudioDeleteSessionHandler (personal/file-store path): align with the
  Ent branch by failing closed when GetSessionMeta errors instead of
  falling through to Delete. Prevents orphaning fleet sandboxes and
  workspaces on transient meta-lookup errors.

Addresses medium and minor issues from PR #510 code review.
Add a build-extension job to both release.yml and beta-release.yml that
builds the Chrome extension, packages extension/dist into
astonish-extension.zip, and uploads it as a workflow artifact. The
existing release job already downloads all artifacts and publishes
./release/astonish-* via softprops/action-gh-release, so the zip is
automatically included alongside the platform binaries.
Add docs/website/docs/studio/chrome-extension.md covering:
- Requirements (Chrome 114+, running Astonish instance)
- Installation from release zip (unpacked extension)
- Sign-in flow (bearer token, Studio URL)
- Side panel UI (session picker, page context, transcript)
- Page tools reference table (page_snapshot/click/fill/navigate/etc)
- Apply-to-page workflow and two-step handshake for read-only fields
- Session isolation (astonish-extension appName)
- Troubleshooting

Wire into sidebar nav under Studio > Chrome Extension and add link
to Studio overview's Related Pages section.
@rschardosin
rschardosin merged commit e14ecf7 into main Sep 10, 2026
7 checks passed
@rschardosin
rschardosin deleted the feature/extension-ui-fixes branch September 10, 2026 19:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant