Fiji Large Language Model (LLM) Integration

This project brings extensible, reproducible AI assistance into Fiji, helping scientists discover tools, build image analysis workflows, and connect with both local and external language models.

Fiji-LLM was developed to help users access Fiji's capabilities through natural-language interaction, while allowing them to choose the AI model that best fits their needs.

• Make Fiji More Accessible: Help scientists discover relevant tools, learn unfamiliar workflows, and create reusable scripts through guided natural-language interactions.

• Enable Reproducible Agentic Workflows: Give AI agents structured access to Fiji’s application context and core capabilities, with an emphasis on generating familiar scripts and macros rather than de novo workflow formats.

• Provide an Extensible AI Foundation: Establish shared extension points for agentic tools and model connectivity so that developers can add new capabilities without creating isolated or incompatible integrations.

• Support Private and Equitable Model Access: Make local, open-weight models a first-class option while retaining flexible connectivity to external model providers and AI clients.

• In-App Chat Interface: Provides scientists with guided AI assistance directly inside Fiji.

• Model Context Protocol (MCP) Server: Exposes Fiji’s agentic tools to compatible external assistants, development environments, and other MCP clients.

• Annotated Tool Registry: Uses the SciJava plugin framework to dynamically discover and register capabilities that developers expose to AI agents.

• Extensible Model Engine: Separates model connectivity from agentic functionality, allowing new local or remote model providers to be added through plugins.

• Context-Aware Analysis Tools: Give agents structured access to Fiji’s environment, including installed commands, open images, analysis metadata, the Script Editor, and macro recorder.

• ImageJ Data Inspection: Provide read-only access to common ImageJ data objects, including Results Tables and the ROI Manager.

• Fiji Chat Quick Start
• MCP Server
  • VS Code
• User Guide
  • Basic Concepts
  • Supported AI Providers
  • General Work Flow
  • Using Tools
  • Tips for Better Results
  • Getting Help
• Developers: Adding Functionality
  • LLMProvider
  • ContextItemSupplier
  • AiToolPlugin
    • Tool Best Practices
  • Integration Testing
  • ChatbotService
• FAQ

• Technical Summary

1. Install Fiji: Download the Latest version from imagej.net/software/fiji

2. Add the Fiji-chat Update Site:

   • See instructions on adding unlisted update sites.
   • Add the (currently unlisted) Fiji-chat site: https://sites.imagej.net/Fiji-chat/
   • Restart Fiji afterwards

3. Open the chat: Run Help > Assistants > Fiji Chat... (shortcut: ctrl + 0)

   • The first time you run Fiji Chat... you will see a landing page where you select an AI Service and model.
   • Future Fiji Chat... runs will go right to chatting with your last selected service and model.

4. Select a model:

   Option A: Local Models (Recommended for beginners)

   • These are open models which run on your machine.
   • Download and install Ollama now (no Fiji restart needed)
   • We recommend starting with the curated Gemma4 - small (Ollama) service, which auto-selects a model aimed at maximizing hardware compatibility.
   • The general Ollama service allows for full model exploration.
   • Click OK to start chatting. You will need to wait for the selected model to download (only on first chat with a model)

   Option B: Cloud Models

   • If you have an account with an AI service (Gemini, Claude, ChatGPT, etc.) you can use it with Fiji chat. Most services require paid subscriptions or credits for this function.
   • Select your provider, then choose an available model. Different models have different pricing schemes.
   • Click OK to start chatting. You will be prompted for an API key and provided with a link to your provider's key page (only on first chat with a remote provider)

   Want to switch later? Use the ⚙️ button in chat to change models and/or providers.

All LLM tools in Fiji are accessed via an MCP Server. While we provide a basic, integrated chat interface, this also allows external applications ("harnesses") to interact with Fiji through this local server.

Currently, the MCP server is tied to a running Fiji application - which is where any tools will execute. When Fiji and the MCP server are running, it can be accessed at http://localhost:9090/mcp (note the default port 9090)

The image namespace includes fiji_image_list, fiji_image_details, fiji_image_view, and fiji_image_view_annotated. The view tools accept an image_id and return the rendered image as an MCP image content block with PNG data, allowing compatible external clients to inspect the image directly. fiji_image_view_annotated also includes visible ROIs and image overlays when available, plus a text content block describing the rendered display state. The Fiji chat attachment menu provides matching plain-image and image- with-overlays choices.

Available Configuration

• Set Port: Use Help > Assistants > Manage MCP Server... or preferences key sc.fiji.mcp.port
• Start Manually: Click "Start Server" in the Manage MCP Server dialog
• Auto-Launch: Enable Launch MCP on Startup in the Manage MCP Server dialog, or set preferences key sc.fiji.mcp.launchOnStartup to true

You can connect your VS Code LLMs to the Fiji MCP server! This allows your agents to run tasks in a local Fiji. Edit your mcp.json and add the following entry:

		"fiji-mcp": {
			"type": "http",
			"url": "http://localhost:9090/mcp",
		},

In your chat Configure Tools dialog, you should see a new fiji-mcp-server option that you can toggle on or off.

You may also want to update your settings.json and add:

	"chat.mcp.autostart": "newAndOutdated"

Because the Fiji MCP server is dynamic, being tied to a running Fiji instance, your agent will receive errors trying to use Fiji tools when Fiji is closed. But this setting should allow it to reconnect when Fiji is running again. It also will restart its MCP connection after the tool definition cache is cleared. If you prefer to manually manage this connection, set autostart to never.

You should run MCP: Reset Tool Caches any time deployed tools are revised.

You can manually check and manage MCP server status with MCP: List Servers, as well.

NB: Update the port in mcp.json as necessary NB: Your local Fiji application must be running first for the MCP server to be findable by VS Code. For best results, (re)start the server from mcp.json after launching Fiji.

This repository includes a VS Code agent and Fiji workflow skills in .github, where VS Code can discover project-scoped customizations when this repository is opened:

• .github/agents/fiji-mcp.agent.md
• .github/skills/fiji-script-workflow/
• .github/skills/fiji-macro-workflow/

Discovery makes the agent and skill available to this workspace; it does not automatically select the agent or load the skill for every conversation.

For personal use across workspaces, prefer symlinking these project files into the VS Code user-level discovery paths:

• .github/agents/fiji-mcp.agent.md -> <VS Code user profile>/prompts/fiji-mcp.agent.md
• .github/skills/fiji-script-workflow/ -> ~/.copilot/skills/fiji-script-workflow/
• .github/skills/fiji-macro-workflow/ -> ~/.copilot/skills/fiji-macro-workflow/

Prefer symlinking these destinations to the files in this checkout when the platform supports it, so updates are picked up immediately. Copy the files when symlinking is unavailable. The skills guide Fiji script and macro authoring, execution, diagnosis, repair, and verification workflows.

AI Service Providers - Companies that provide cloud-based access to trained language models.

Models - Specific language models offered by a provider (e.g., GPT-4o, Claude 3.5 Sonnet). Different models have different capabilities and costs.

Tokens - The unit of operation within an LLM: messages to and from the chatbot are encoded as a series of "tokens". Longer messages require more tokens. Importantly, any actions taken by the LLM in response to your message will use tokens. (such as editing a script or running a command)

API Keys - Credentials that authenticate you with an AI service provider. Often require per-token pay-as-you-go or a subscription plan.

Conversations - Your chat history with an AI assistant, independent of model. Conversations are saved locally and loaded when Fiji starts, so you can continue working where you left off. In long conversations, the model may not "see" the whole chat history.

Context Items - Information you can attach to chat message that helps the assistant understand your Fiji environment. For example, you could attach an open image or script.

Image rendering is kept separate from persisted context metadata. The reusable ImageRenderingService captures the current displayed plane and display state, creates a bounded PNG ImageContent payload, and returns its image metadata separately. Optional ROI drawing uses the active legacy ImageJ ROI when that bridge is available.

• Note: Gemini is currently the only supported provider that provides API Keys at no charge.
• Using a "free" API Key is subject to Google's rate limits and availability. It is suitable for testing and assessment, but not regular use.
• Getting an API Key:
  1. Visit aistudio.google.com/app/apikey
  2. Click Create API key and copy it

• Note: Ollama is a general gateway to pretrained models. Using local models bypasses the need for API keys or token considerations. However, running a local LLM can require significant resources (RAM, GPU, hard drive, power).
• Models typically come in varieants (7b, 20b, etc...), indicating the number of model parameters (in billions). More parameters means a better ability to conceptualize solutions, but also more resource use.-
• Fiji-chat is intended for use with models that support Tool Use.
• Vision support is reported for the selected model when Ollama provides it; use a model whose capabilities include vision to attach images.
• Installation:
  1. Download and install Ollama
  2. (Optionally) Use the ollama UI or command line tool to download a model of interest.
  3. When you can start a new chat you can choose from compatible models, which will be downloaded as needed.
• Recommended model(s):
  • Gemma4 - Depending on your available video memory, select small (8GB), medium (16GB), or large (24GB or more).

• Getting an API Key:
  1. Create an account at console.anthropic.com
  2. Go to Account settings > API keys (or click here)
  3. Click Create Key and copy it

• Getting an API Key:
  1. Create an account at platform.openai.com
  2. Go to Account settings > API keys (or click here)
  3. Click Create new secret key and copy it

1. Launch the Chat: Run Help > Assistants > Fiji Chat...
2. Select a Provider: Choose your preferred AI service
3. Select a Model: Pick a specific model. If an API Key is required and not found, you will be prompted automatically.
4. Start Chatting: Type your question or request in the input box
5. Attach Context (optional): Use the context buttons to provide relevant information from your Fiji environment

You can use Help > Assistants > Manage API Keys... to manage your key(s) at any time.

The benefit of having an assistant integrated into Fiji is that it can perform actions, beyond just conversation:

Script Writing - Ask the assistant to write scripts in Python, Groovy, JavaScript, or other SciJava-compatible languages. Describe the context of your analysis task and the assistant can generate executable scripts.

Script Editing - Attach scripts as context and ask the assistant to improve, debug, or adapt them for your specific needs.

Macro Recording - Ask the assistant for help creating ImageJ macros for guidance to relevant commands and plugins. The assistant can inspect the current recorder state and buffer with fiji_macro_recorder_state.

Script and Macro Execution - Run non-.ijm scripts with fiji_script_run, and poll a dialog-paused run with fiji_script_run_status. Run an active .ijm script with fiji_macro_run, which executes it through the visible Script Editor and returns output, errors, ImageJ and SciJava logs, environment impact, and a status. Timeout results remain timed_out at the top level, but they now include timeout_requested, execution_terminated, termination_status, and termination_failure so a caller can distinguish a requested timeout from an actual termination failure. If a script or macro pauses for a visible dialog, inspect the returned run_id with the corresponding status tool, then respond only through fiji_ui_dialog_respond using the exact dialog title and button text.

Command Execution - Use fiji_command_search to find a menu path, then call fiji_command_run. The result includes the command status and a lightweight before/after environment report covering open images, active-image changes, bounded SHA-256 pixel fingerprints, Results table metadata, visible dialogs, and ImageJ and SciJava log deltas. Live environment snapshots defer pixel hashing; large samples and lazy images report inconclusive pixel comparisons explicitly.

ImageJ Data Inspection - Use fiji_results_read to inspect Results Table headings and numeric rows, and fiji_rois_read to inspect ROI Manager availability, ROI summaries, and bounding boxes. Use fiji_rois_read_details with an ROI index when exact shape and polygon coordinates are needed. These tools are read-only.

UI Inspection and Vision - Use fiji_ui_windows_read to list visible AWT and Swing windows, then fiji_ui_controls_read with an exact window title to inspect supported controls and their state. Use fiji_ui_screenshot for a PNG screenshot that can be provided to vision-capable models. Screenshots capture current screen pixels, so overlapping or occluding windows may appear; activate_and_restore requests best-effort activation and focus restoration, but does not guarantee an unobstructed capture or successful restoration. Inspect the returned metadata to see whether focus restoration succeeded.

System Information - Use fiji_system_read to inspect ImageJ 1.x and application versions, Java and operating system details, JVM memory information, and active update sites. Use fiji_system_list_update_sites to list all available update sites and their active status.

General Information - Describe your image analysis goals and discuss options available in your Fiji environment.

• Be Specific: Describe your task in detail. Include what you're trying to analyze, what tools you've already tried, and what's not working.
• Provide Context: Use the context buttons to share relevant images, open scripts, or previous conversation history.
• Iterate: LLM responses aren't always perfect on the first try. Review the output, provide feedback, and ask follow-up questions.

A help button ( ? ) in the chat window provides in-app explanations of the UI and how to use each feature.

For questions, bug reports, and feature requests, visit the Image.sc Forum. The Fiji community is active there and happy to help!

This project provides langchain4j integration to the SciJava plugin framework. There are several key points of extension:

Determine which AI Services are available in chat.

Provide a mapping from the Fiji application environment to ContextItems, facilitating deeper understanding by the LLM. Context items can override getTooltipText() to explain what the attachment menu's active-item action will attach.

These plugins contain methods annotated with langchain4j's @Tool annotation. New tools enable code to be run by the AI assistants.

When adding an AiToolPlugin:

• Use scoped tool names. Give every @Tool an explicit lower-case snake_case name following the fiji_<scope>_<operation> pattern, such as fiji_script_read_content. The scope prevents collisions between plugins. Preserve an existing name when modifying a tool and use the exact name consistently in descriptions, errors, and recommendations.

• Choose useful parameter names. Use @P to provide descriptive, stable names to the LLM, usually in snake_case, even when the Java parameter uses camelCase: @P("image_id"), @P("menu_path"), or @P("start_line"). Include units, indexing conventions, and other constraints in the tool description when they matter.

• Keep descriptions at the right level. Keep getUsage() as a short overview of the whole fiji_<scope>_* namespace and its basic workflow. Put each tool's action, preconditions, safety restrictions, related tools, and return value in its @Tool(value = { ... }) description.

• Return structured results. Text tools should return valid JSON strings for every success and error result, using the shared helpers in AbstractAiToolPlugin. A multimodal tool may instead return LangChain4j Content, such as ImageContent; the MCP bridge preserves supported text and image content blocks for external clients.

Live script and macro integration tests should be run by the developer as-needed, e.g. after major tool changes.

For a live run, create a task-specific agent, i.e. via the VS Code MCP agent definition, then give it the checklist in doc/INTEGRATION_TESTS.md. The agent should execute the workflow against a running Fiji instance through the Fiji MCP tools and report the observed results, failures, and any cases that require manual interaction.

Keep this dedicated live-test session separate from the general coding agent so integration tests are run deliberately.

Keep the integration test document, technical summary, README, and relevant VS Code agent files in sync when public tools or APIs are added, removed, renamed, or behavior changes.

For developing chatbots in particular UI environments.

An MCP (Model Context Protocol) server exposes all registered AiToolPlugin implementations via local HTTP, making them accessible to external clients.

This repository is developed with assistance from AI coding tools; all changes are reviewed by human contributors.

A: Each individual LLM has a baseline "knowledge" that is frozen in time based on when it was trained. To obtain current information, LLMs can use tools (such as web searches). We provide a core set of tools to help connected LLMs gather information related to Fiji use.