Claude finished @modelorona's task —— View job

Implementation Plan for Dictation Feature in Houdini

I've analyzed the codebase and researched privacy-first speech-to-text solutions. Here are several concrete implementation approaches for adding voice dictation to Houdini's text input field.

Context Analysis

Current Implementation:

Houdini is WhoDB's AI chat interface located at frontend/src/pages/chat/chat.tsx
Text input uses the <Input> component from @clidey/ux (line 480-487)
Frontend stack: React/TypeScript, GraphQL-first architecture
No existing audio/speech dependencies
WhoDB is self-hosted and privacy-focused with Ollama integration

Key Requirements:

Privacy-first approach (consistent with WhoDB's local/self-hosted nature)
Voice recording button next to text input
Speech-to-text conversion output to input field
Maintain existing GraphQL-first architecture
Support both CE and EE editions

Implementation Approaches

Approach 1: Browser Web Speech API (Recommended)

Description:
Use the browser's native Web Speech API (SpeechRecognition), which processes speech entirely on the user's device without sending data to external servers.

Technical Details:

API: webkitSpeechRecognition / SpeechRecognition
Browser Support: Chrome, Edge, Safari (85%+ coverage)
Processing: Client-side only
Dependencies: None (native browser API)
Privacy: Excellent - no data leaves the browser

Implementation Steps:

Create a new React hook useVoiceInput in frontend/src/hooks/useVoiceInput.ts
Wrap browser's SpeechRecognition API with error handling and lifecycle management
Add a microphone button component next to the Input field in chat.tsx (line ~480)
Handle voice state (listening, processing, error) with visual feedback
Stream recognized text into the input field as the user speaks
Support both interim (real-time) and final results

Code Structure:

// frontend/src/hooks/useVoiceInput.ts
export const useVoiceInput = () => {
  const [isListening, setIsListening] = useState(false);
  const [transcript, setTranscript] = useState("");
  const recognitionRef = useRef<SpeechRecognition | null>(null);
  
  // Initialize, start, stop, error handling
  return { isListening, transcript, startListening, stopListening };
}

// frontend/src/pages/chat/chat.tsx
const { isListening, transcript, startListening, stopListening } = useVoiceInput();

// Add button with microphone icon next to Input

Pros:

Zero external dependencies
Complete privacy - no data sent to servers
Free and fast (real-time)
Simple implementation (~200 lines of code)
Works offline
Natural fit with WhoDB's self-hosted philosophy
Consistent with Ollama integration approach

Cons:

Browser support not universal (Firefox on desktop lacks support)
Accuracy depends on browser implementation
Limited language model customization
Requires HTTPS (or localhost for development)

Estimated Effort: 4-6 hours

Approach 2: Ollama with Whisper Model

Description:
Leverage the existing Ollama integration to run OpenAI's Whisper model locally for speech-to-text processing.

Technical Details:

Model: Whisper (small, medium, or large variants)
Backend: Ollama (already integrated)
Processing: Server-side (user's Ollama instance)
Privacy: Excellent - stays within user's infrastructure
Dependencies: Requires Ollama with Whisper model installed

Implementation Steps:

Record audio in browser using MediaRecorder API
Capture audio as WebM/WAV blob
Add GraphQL mutation transcribeAudio(audioFile: Upload!): String
Backend receives audio file and sends to Ollama Whisper endpoint
Return transcribed text to frontend
Update input field with transcription

Backend Changes:

// core/graph/schema.graphqls
type Mutation {
  transcribeAudio(audioFile: Upload!): String!
}

// core/graph/schema.resolvers.go
func (r *mutationResolver) TranscribeAudio(ctx context.Context, audioFile graphql.Upload) (string, error) {
  // Send to Ollama Whisper API
  // Return transcription
}

Frontend Changes:

Record button component with MediaRecorder
File upload via GraphQL multipart request
Handle audio recording state and errors

Pros:

Consistent with existing Ollama integration
High accuracy (Whisper is state-of-the-art)
Works across all browsers
Complete privacy control
Offline capable
Multiple language support
User controls which Whisper model (speed vs accuracy tradeoff)

Cons:

Requires Ollama with Whisper installed
Additional backend complexity (~500 lines across frontend/backend)
File upload overhead
Processing latency (not real-time for larger files)
Higher resource usage on Ollama server
Whisper models are large (75MB-1.5GB)

Estimated Effort: 12-16 hours

Approach 3: Hybrid - Web Speech API with Whisper Fallback

Description:
Combine Approach 1 and 2 - use browser Web Speech API when available, fall back to Ollama Whisper when not supported.

Technical Details:

Primary: Browser SpeechRecognition (instant, no server load)
Fallback: Ollama Whisper (when browser doesn't support)
Feature detection determines which path to use

Implementation Steps:

Implement Web Speech API hook (Approach 1)
Implement Ollama Whisper backend (Approach 2)
Add feature detection: 'SpeechRecognition' in window
Route to appropriate implementation based on browser capability
Unified UI that works with both backends

Code Structure:

const useVoiceInput = () => {
  const hasSpeechAPI = 'SpeechRecognition' in window || 'webkitSpeechRecognition' in window;
  
  if (hasSpeechAPI) {
    return useBrowserSpeechRecognition();
  } else {
    return useOllamaWhisper();
  }
}

Pros:

Best of both worlds
Maximum browser coverage
Optimal performance (uses browser when available)
Privacy maintained
User choice (can prefer Whisper even if browser supports it)

Cons:

Most complex implementation
Need to maintain two code paths
Testing complexity
Larger codebase (~700 lines total)

Estimated Effort: 16-20 hours

Approach 4: Client-Side WASM Whisper

Description:
Run Whisper model directly in the browser using WebAssembly (e.g., whisper.cpp compiled to WASM).

Technical Details:

Library: whisper-turbo or whisper.cpp WASM build
Processing: 100% client-side
Model: Tiny/Base Whisper models (~40-75MB)
Privacy: Excellent - everything in browser

Implementation Steps:

Add WASM Whisper library dependency
Download and cache Whisper model on first use
Record audio with MediaRecorder
Process audio through WASM Whisper in Web Worker
Update input with transcription
Cache model in browser storage

Pros:

No backend changes needed
Works in all modern browsers
Complete privacy
Offline after initial model download
High accuracy
No Ollama dependency

Cons:

Large initial download (40-150MB for model)
High CPU usage in browser
Memory intensive
Slower on low-end devices
Complex dependency management
Less mature ecosystem
Model loading time on first use

Estimated Effort: 20-24 hours

Recommendation

I recommend Approach 1 (Web Speech API) as the starting point, with the option to add Approach 2 (Ollama Whisper) later if needed.

Rationale:

Fastest to implement - Can have working prototype in a few hours
Privacy-first - Aligns perfectly with WhoDB's self-hosted philosophy
Zero dependencies - No new packages or backend changes
Excellent UX - Real-time transcription as user speaks
Low maintenance - Uses stable browser APIs
Good enough coverage - 85%+ browser support covers most users
Extensible - Easy to add Whisper fallback later if needed

Suggested Roadmap:

Phase 1 (v1): Implement Web Speech API (Approach 1)
Phase 2 (v2): Add Ollama Whisper fallback for Firefox users (Approach 2)
Phase 3 (future): Consider WASM option if offline support becomes critical

Technical Considerations

GraphQL Architecture:

Approach 1 requires no GraphQL changes (client-only)
Approach 2 requires new mutation for audio upload
Both follow GraphQL-first principles

CE vs EE:

Voice input should be CE feature (core functionality)
Keep all code in frontend/src/ not ee/frontend/

Browser Permissions:

Requires microphone permission from user
Need clear UI for permission requests
Handle permission denied gracefully

Accessibility:

Add keyboard shortcuts for starting/stopping recording
Provide visual feedback for recording state
Support for users who cannot use voice input

Error Handling:

Network errors (if using Ollama approach)
Microphone access denied
Unsupported browser
Noisy environment (poor recognition)
Empty/silent audio

Questions for Clarification

Browser Support Priority: Is Firefox desktop support critical? (Affects approach choice)
Ollama Requirement: Should this work without Ollama, or is Ollama a prerequisite for Houdini?
CE vs EE: Should this be CE or EE feature?
Language Support: English only, or multiple languages needed?
Real-time vs Batch: Prefer real-time transcription (Web Speech) or batch processing (Whisper)?

Let me know which approach you'd like to pursue, and I can begin implementation.

Job Run | Branch

[FR] - Dictation #583

Description

Activity

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions