Skip to content

mieweb/duckdbgo

Repository files navigation

DuckDBGo

Like Excel, but for structured row-based data. A proof-of-concept React component that enables visual data exploration, editing, and sharing of DuckDB files directly in the browser.

πŸŽ‰ Now with Real DuckDB WASM Integration! No more mock data - execute actual SQL queries against real DuckDB databases running entirely in your browser.

DuckDBGo is a modern web component built with Next.js that allows users to:

  • πŸ“ Load DuckDB files from local storage or URLs
  • πŸ‘οΈ Visually explore data with an intuitive spreadsheet-like interface
  • ✏️ Edit data in-place with immediate visual feedback
  • πŸ”„ Experiment with transformations through an integrated SQL shell
  • πŸ“€ Share DuckDB files with other users seamlessly

οΏ½ Vision

DuckDBGo aims to democratize data analysis by bringing the power of DuckDB's analytical engine to the browser. Unlike traditional database tools that require complex setup, DuckDBGo provides an Excel-like experience for structured data while leveraging the full analytical capabilities of DuckDB.

Key Differentiators

  • Zero Setup: No database installation or server configuration required
  • Visual Interface: Spreadsheet-like editing with SQL power underneath
  • File-Based: Work with DuckDB files that can be shared like any other file
  • Browser-Native: Runs entirely in the browser using WebAssembly
  • Collaborative: Easy sharing and collaboration through file exchange

Features

οΏ½ Visual Data Exploration

  • Spreadsheet-like Interface: Familiar grid view for browsing and editing data
  • Real-time Updates: See changes instantly as you edit cells
  • Column Sorting & Filtering: Interactive data exploration without writing SQL
  • Data Type Awareness: Automatic detection and handling of different data types
  • Pagination: Handle large datasets efficiently

✏️ In-Place Data Editing

  • Cell Editing: Click any cell to edit values directly
  • Bulk Operations: Select multiple cells for batch editing
  • Data Validation: Built-in validation for data integrity
  • Undo/Redo: Full history of changes with easy rollback
  • Auto-save: Changes persist automatically

πŸ”§ SQL-Powered Analysis

  • Integrated SQL Shell: Power users can write custom queries
  • Query Results in Grid: See SQL results in the same visual interface
  • Command History: Navigate through previous queries
  • Syntax Highlighting: Enhanced SQL editing experience
  • Query Templates: Common queries available as starting points

πŸ“ DuckDB File Management

  • Load from URLs: Import DuckDB files from web URLs
  • Local File Support: Open files from your device (future enhancement)
  • Export Functionality: Save modified data as new DuckDB files
  • File Sharing: Share DuckDB files with other users
  • Version Control: Track changes to datasets over time

🌐 Browser-First Design

  • No Installation: Works directly in any modern web browser
  • Cross-Platform: Windows, macOS, Linux, and mobile devices
  • Offline Capable: Work with data without internet connection
  • Progressive Web App: Install as an app for native-like experience
  • Secure: All data processing happens client-side

Technology Stack

  • Framework: Next.js 15 with App Router (React-based component)
  • Language: TypeScript for type safety and developer experience
  • Data Grid: AG Grid Community Edition for high-performance editing
  • Database Engine: DuckDB WASM for client-side analytical processing
  • Styling: Tailwind CSS with custom design system
  • UI Components: Radix UI primitives for accessibility
  • Icons: Lucide React for consistent iconography
  • State Management: React hooks with optimized re-rendering
  • File Handling: Web APIs for file upload/download operations

πŸš€ Deployment & Accessibility

Easy Deployment

Deploy anywhere with static hosting:

  • Vercel: One-click deployment with Next.js
  • Netlify: Static site hosting with instant deploys
  • GitHub Pages: Free hosting for public repositories
  • Any CDN: Serve static files from any content delivery network

Browser Compatibility

  • Modern Browsers: Chrome 90+, Firefox 88+, Safari 14+, Edge 90+
  • Mobile Support: Responsive design works on tablets and phones
  • Progressive Web App: Can be installed as a PWA for offline use
  • WebAssembly Support: Required for DuckDB WASM integration

πŸ§ͺ Testing the Integration

Quick Test

  1. Open http://localhost:3001 in your browser
  2. Wait for the welcome message confirming DuckDB WASM is loaded
  3. Try these commands:
    .tables          -- List available tables
    .schema          -- View table structure
    SELECT * FROM orders LIMIT 5  -- Query sample data
    SELECT COUNT(*) FROM orders   -- Get row count

Run Tests

npm run test:duckdb  # Test DuckDB WASM integration

🎯 What We've Accomplished

DuckDBGo now features real DuckDB WASM integration with:

βœ… Core Features Implemented

  • Real DuckDB WASM: No more mock data - actual DuckDB running in the browser
  • Data Loading: Automatic loading of sample TPC-H orders data
  • SQL Execution: Full SQL query execution with real results
  • Dynamic Tables: Tables are created and managed by DuckDB
  • Live Schema: Real table schemas from DuckDB metadata
  • Error Handling: Proper error handling for failed queries

πŸ”§ Technical Implementation

  • WebAssembly Integration: DuckDB compiled to WebAssembly for browser execution
  • Async Operations: All database operations are properly async
  • State Management: Centralized data management with caching
  • Type Safety: Full TypeScript support with proper error handling
  • Performance: Optimized for large datasets with pagination

🎨 User Experience

  • Visual Feedback: Real-time loading states and progress indicators
  • Interactive Grid: AG Grid with real data editing capabilities
  • Command History: Full SQL command history with navigation
  • Help System: Context-aware help and command suggestions
  • Responsive Design: Works on all device sizes

Getting Started

Prerequisites

  • Node.js 18+
  • npm, yarn, or pnpm

Quick Start

  1. Clone the repository:
git clone <repository-url>
cd duckdbgo
  1. Install dependencies:
npm install
  1. Start the development server:
npm run dev
  1. Open in browser:

As a Component

DuckDBGo is designed to be used as a reusable React component:

import DuckDBGo from '@/components/DuckDBGo'

function MyApp() {
  return (
    <DuckDBGo
      initialData="https://example.com/data.duckdb"
      onDataChange={(data) => console.log('Data changed:', data)}
    />
  )
}

Usage

Loading Data

From URLs:

# Load DuckDB file from a URL
LOAD 'https://example.com/sales_data.duckdb'

From Local Files (Future):

  • Drag and drop DuckDB files into the interface
  • Use file picker to select local .duckdb files
  • Import from CSV, JSON, or Parquet files

Visual Data Exploration

  1. Browse Data: Use the grid interface to scroll through your data
  2. Sort Columns: Click column headers to sort data
  3. Filter Data: Use built-in filters to find specific records
  4. Search: Global search across all visible data

Editing Data

  1. Edit Cells: Click any cell to edit values directly
  2. Bulk Edit: Select multiple cells for batch operations
  3. Add Rows: Insert new records with the + button
  4. Delete Rows: Remove records with selection and delete key
  5. Undo Changes: Use Ctrl+Z to rollback modifications

Advanced Analysis

SQL Queries:

-- Custom analysis with SQL
SELECT product_name, SUM(sales) as total_sales
FROM sales_data
WHERE date >= '2024-01-01'
GROUP BY product_name
ORDER BY total_sales DESC

Export Results:

  • Save query results as new DuckDB files
  • Export to CSV, JSON, or Parquet formats
  • Share datasets with other users

Keyboard Shortcuts

  • Ctrl+Z / Cmd+Z - Undo last change
  • Ctrl+Y / Cmd+Y - Redo change
  • Ctrl+F / Cmd+F - Search data
  • Delete - Remove selected rows
  • Enter - Edit selected cell
  • Escape - Cancel editing

πŸ“ Project Structure

duckdbgo/
β”œβ”€β”€ app/
β”‚   β”œβ”€β”€ layout.tsx          # Root layout with theme provider
β”‚   β”œβ”€β”€ page.tsx            # Demo page showcasing DuckDBGo component
β”‚   β”œβ”€β”€ globals.css         # Global styles and AG Grid themes
β”‚   └── loading.tsx         # Loading states and spinners
β”œβ”€β”€ components/
β”‚   β”œβ”€β”€ DuckDBGo.tsx        # Main component entry point
β”‚   β”œβ”€β”€ DataGrid.tsx        # Editable data grid with AG Grid
β”‚   β”œβ”€β”€ QueryShell.tsx      # SQL command interface
β”‚   β”œβ”€β”€ FileManager.tsx     # File import/export functionality
β”‚   └── ui/                 # Reusable UI components
β”‚       β”œβ”€β”€ button.tsx      # Custom button component
β”‚       β”œβ”€β”€ input.tsx       # Input field component
β”‚       └── ...
β”œβ”€β”€ lib/
β”‚   β”œβ”€β”€ duckdb.ts           # DuckDB WASM integration
β”‚   β”œβ”€β”€ data-manager.ts     # Data operations and state
β”‚   └── utils.ts            # Utility functions
β”œβ”€β”€ types/
β”‚   └── index.ts            # TypeScript type definitions
└── public/                 # Static assets and demo files

🀝 Use Cases

Data Analysis Workflows

  • Business Intelligence: Quick data exploration without complex tools
  • Data Cleaning: Visual editing and transformation of datasets
  • Research: Collaborative analysis of research data
  • Education: Teaching data concepts with interactive examples

Collaborative Scenarios

  • Team Data Review: Share datasets with stakeholders for feedback
  • Data Journalism: Collaborative fact-checking and data validation
  • Scientific Research: Multi-researcher data analysis and annotation
  • Product Analytics: Cross-team data exploration and insights

Individual Use Cases

  • Personal Data Management: Organize and analyze personal datasets
  • Learning SQL: Visual feedback while learning database concepts
  • Data Migration: Transform and validate data during migration projects
  • Prototype Development: Quick data modeling and testing

Development

Available Scripts

  • npm run dev - Start development server
  • npm run build - Build for production
  • npm run start - Start production server
  • npm run lint - Run ESLint

Key Components

DuckDBShell Component

The main component (app/page.tsx) handles:

  • SQL query execution (currently simulated)
  • Shell command processing
  • Result management and tab switching
  • UI state management

DataGrid Component

Displays query results using AG Grid with:

  • Column sorting and filtering
  • Row pagination
  • Cell editing capabilities
  • Responsive column sizing

πŸ—οΈ Architecture

Component-Based Design

DuckDBGo is built as a modular React component with clear separation of concerns:

DuckDBGo/
β”œβ”€β”€ DataManager/          # Handles DuckDB file operations
β”œβ”€β”€ GridEditor/          # Visual data editing interface
β”œβ”€β”€ QueryInterface/      # SQL shell for advanced users
β”œβ”€β”€ FileHandler/         # Import/export functionality
β”œβ”€β”€ StateManager/        # Centralized state management
└── UI Components/       # Reusable interface elements

Data Flow

  1. Data Loading: DuckDB files loaded via WebAssembly
  2. In-Memory Processing: All operations performed client-side
  3. Visual Rendering: Data displayed in editable grid interface
  4. Change Tracking: Modifications tracked for undo/redo functionality
  5. File Export: Modified data saved as new DuckDB files

Browser Integration

  • WebAssembly: DuckDB compiled for browser execution
  • File System Access: Direct file operations (future enhancement)
  • IndexedDB: Persistent storage for user preferences
  • Service Workers: Offline capability and caching
  • Web Workers: Background processing for heavy operations

πŸš€ Roadmap

Phase 1: Core Data Editing (Current)

  • Visual data grid with AG Grid
  • Basic cell editing capabilities
  • SQL shell integration
  • Mock data for demonstration
  • Real DuckDB WASM integration

Phase 2: Enhanced Editing

  • Advanced cell editors (date picker, dropdowns, etc.)
  • Data validation and constraints
  • Bulk editing operations
  • Find and replace functionality
  • Column resizing and reordering
  • Data type conversion tools

Phase 3: File Management

  • Local DuckDB file loading
  • Drag-and-drop file import
  • Multiple file support
  • File versioning and history
  • Auto-save functionality
  • File compression and optimization

Phase 4: Collaboration Features

  • Real-time collaborative editing
  • File sharing via URLs
  • Change tracking and comments
  • User permissions and access control
  • Conflict resolution for concurrent edits
  • Activity feeds and notifications

Phase 5: Advanced Analytics

  • Built-in data visualization charts
  • Pivot tables and cross-tabulation
  • Statistical analysis functions
  • Data cleaning and transformation tools
  • Machine learning model integration
  • Custom function support

Phase 6: Enterprise Features

  • Multi-user database connections
  • Data lineage tracking
  • Audit logs and compliance
  • API integration for external systems
  • Custom plugins and extensions
  • Performance monitoring and optimization

Contributing

  1. Fork the repository
  2. Create a feature branch
  3. Make your changes
  4. Add tests if applicable
  5. Submit a pull request

License

This project is licensed under the MIT License.

Acknowledgments

πŸ“„ License

This project is licensed under the MIT License - see the LICENSE file for details.

🀝 Contributing

We welcome contributions! DuckDBGo is an open-source project that benefits from community involvement.

Ways to Contribute

Development Setup

  1. Fork the repository
  2. Create a feature branch: git checkout -b feature/your-feature
  3. Make your changes and add tests
  4. Run the test suite: npm test
  5. Submit a pull request

πŸ“ž Support


DuckDBGo - Bringing the power of DuckDB to everyone, everywhere, through the magic of modern web technology. πŸš€

About

A single SPA DuckDB Shell explorer with AG Grid browser

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published