GooDAnDReaDY/dsh-vision-bridge

---

⚡ Overview

dsh-vision-bridge is a comprehensive multimodal processing engine for DeepSeek Harness.

It solves two critical challenges:

1. Universal Multimodal Bridging: When users send images, diagrams, or PDFs to text-only LLMs, the plugin automatically intercepts the attachments, extracts rich structured visual descriptions via configured vision channels, and injects the evidence seamlessly into the prompt — completely preventing model does not support image input crashes.
2. Dedicated Vision Model Separation: Allows assigning a dedicated, specialized Vision model for all visual processing tasks, leaving the main chat model free to focus purely on fast reasoning and text generation.
3. 27 Agent Vision Tools: Equips agents with a full suite of computer vision tools (OCR, VQA, spatial grounding, UI wireframe parsing, PDF page extraction, image diffing, and batch processing).

graph LR
    subgraph UserTurn [User Input]
        Attach[🖼️ Image / Screenshot / PDF] --> Check{Active Chat Model Has Native Vision?}
    end

    subgraph DedicatedVision [Dedicated Vision Processing Layer]
        Check -->|Yes: Native VLM| Pass[Direct Model Pass-Through]
        Check -->|No: Text-Only LLM| Interceptor[Vision Bridge Interceptor]
        Interceptor --> Pick{Dedicated Vision Model Selection}
        Pick -->|Auto-Detect| AutoVLM[First Vision Model in Catalog]
        Pick -->|Explicit Override| UserVLM[Configured Dedicated Vision Model]
        Pick -->|Channel Fallback| Channels[Multi-Channel Router: Ollama / Webhook]
        AutoVLM --> Structured[Structured Visual Evidence]
        UserVLM --> Structured
        Channels --> Structured
        Structured --> Fuse[Prompt Injection & Context Fusion]
    end

    subgraph ChatLLM [Main Chat Model]
        Pass --> LLM[Fast Chat Model Reasoning]
        Fuse --> LLM
    end

    style UserTurn fill:#1e1e2e,stroke:#89b4fa,stroke-width:2px,color:#cdd6f4
    style DedicatedVision fill:#181825,stroke:#cba6f7,stroke-width:2px,color:#cdd6f4
    style ChatLLM fill:#11111b,stroke:#a6e3a1,stroke-width:2px,color:#cdd6f4

---

🎯 Dedicated Vision Model Selection & Bridge Operating Modes

You can completely decouple your chat reasoning model from your vision processing model:

1. Dedicated Vision Model Selection (visionProvider & visionModel)

• Auto-Discovery (Default): If left empty (""), the plugin automatically scans your active LLM catalog and picks the first model flagged with acceptsImages.
• Explicit Dedicated Override: Specify a dedicated visionProvider and visionModel in Settings → Vision Bridge. All background image descriptions and tool invocations will route exclusively to this specialized model without altering your main conversation model.

2. Bridge Operating Modes (mode)

• hybrid (Default): Automatic background visual description injection for text-only models PLUS all 27 vision tools available to the agent.
• llm: Background automatic description injection only (no manual agent tool exposure).
• tools: No automatic background rewriting; the chat model receives image references and explicitly calls vision tools (describe_image, vision_ocr, etc.) when needed.

---

🌐 Dynamic Vision Channel Architecture

dsh-vision-bridge dynamically routes image queries across 5 flexible channel types:

Channel Type (type)	Description	Example Configuration
dsh-catalog	Any vision-capable model already configured in your DSH providers	{ type: 'dsh-catalog', provider: 'provider-id', model: 'model-id' }
openai-compatible	Direct OpenAI-format vision endpoint (e.g. local engine, gateway)	{ type: 'openai-compatible', baseURL: 'http://localhost:8000/v1', model: '...' }
ollama	Local Ollama instance with automatic model discovery (autoLocalOllama)	{ type: 'ollama', baseURL: 'http://localhost:11434/v1', model: '...' }
custom	Custom HTTP payload via user-defined requestTemplate and responsePath	{ type: 'custom', baseURL: '...', requestTemplate: {...} }
webhook	Direct HTTP POST webhook	{ type: 'webhook', baseURL: 'https://...' }

[!TIP] Zero-Config Local Vision: If no channels are configured and Ollama is running locally (localhost:11434), dsh-vision-bridge automatically probes and binds to your local vision model on boot!

---

🛠️ Complete 27 Agent Vision Tools Matrix

dsh-vision-bridge registers 27 specialized tools in ctx.tools:

Tool Name	Purpose	Key Parameters
describe_image	Full semantic captioning and general description	image_path, detail_level
read_image	Direct text reading and reading order extraction	image_path, language
vision_ocr	High-accuracy optical character recognition	image_path, detect_orientation
vision_ocr_local	100% offline local OCR engine	image_path
vision_long_ocr	Multi-column, complex document OCR	image_path, preserve_layout
vision_pdf_pages	PDF multi-page extraction, rendering & OCR	pdf_path, pages, dpi
vision_describe_structured	Structured JSON evidence (objects, text, colors, hierarchy)	image_path, schema
vision_vqa	Visual Question Answering on specific regions	image_path, question, bbox
vision_ground	Spatial coordinate & bounding box grounding	image_path, query
vision_crop	Dynamic image cropping around points or bounding boxes	image_path, bbox, target_path
vision_detect	Object detection, counting, and classification	image_path, categories
vision_compare	Multi-image side-by-side semantic comparison	image_paths, aspects
vision_pixel_diff	Visual regression pixel-level comparison	image_a, image_b, threshold
vision_ui_layout	UI wireframe parsing, element hierarchy & buttons	image_path, framework
vision_translate_image	In-image text translation with replacement	image_path, target_lang
vision_colors	Color palette extraction & dominant contrast analysis	image_path, palette_size
vision_extract_foreground	Subject isolation and background removal	image_path, output_path
vision_trace	Vector diagram, blueprint, and line tracking	image_path, smoothness
vision_html_screenshot	Render HTML/CSS directly to screenshot	html_content, viewport
vision_video_describe	Video keyframe extraction & timeline summary	video_path, fps
vision_batch	Parallel batch analysis across multiple images	image_paths, task
vision_browser_snapshot	Headless browser page snapshot	url, wait_selector
vision_browser_click	Visual coordinate-based browser click	coordinate, element_text
vision_browser_navigate	Visual web page navigation	url
vision_present	Visual presentation & highlight formatting	image_path, highlights
vision_materialize	Visual asset materialization & caching	asset_ref
vision_page_persist	Multi-turn visual context persistence	session_id, state

---

📊 Performance, Caching & Cost Monitoring

• Disk LRU Evidence Cache: Repeated images or identical screenshots are hashed and served from cache with 0 latency and $0 token cost.
• Real-time Cost & Token Tracker: Exposes usage statistics via GET /dsh-vision-bridge/costs and GET /dsh-vision-bridge/stats.
• Built-in Benchmark Suite: Run latency diagnostics across all configured vision channels via GET /dsh-vision-bridge/bench.

---

📦 Quick Installation

dsh plugin --profile web add @goodandready/dsh-vision-bridge

[!IMPORTANT] Restart DSH Web UI after installation (systemctl --user restart dsh-web) and reload the browser tab.

---

⚙️ Configuration Example (settings.yaml)

dsh-vision-bridge:
  enabled: true
  visionProvider: my-provider
  visionModel: my-vision-model
  mode: hybrid
  sanitizeImages: true
  autoLocalOllama: true
  channels:
    - type: dsh-catalog
      provider: my-provider
      model: my-vision-model
    - type: openai-compatible
      baseURL: http://127.0.0.1:8000/v1
      model: my-model
      keyEnv: API_KEY_ENV
    - type: ollama
      baseURL: http://127.0.0.1:11434/v1
  channelFallback: sequential
  channelTimeoutMs: 30000
  cacheEnabled: true
  maxCacheMb: 250
  pdfDpi: 200

---

🔌 HTTP API Routes

Route	Method	Description
/dsh-vision-bridge/models	GET	Discovers available vision models in DSH catalog (acceptsImages)
/dsh-vision-bridge/channels	GET, POST	Inspects or updates active vision routing channels
/dsh-vision-bridge/costs	GET	Returns aggregated token consumption and estimated cost per channel
/dsh-vision-bridge/cache	GET, DELETE	Inspects or clears the disk visual evidence cache
/dsh-vision-bridge/stats	GET	Channel invocation statistics, average latency, and error counters
/dsh-vision-bridge/bench	GET	Latency and throughput benchmark across configured channels

---

📄 License

MIT © GooDAnDReaDY