An streamlit app built for data professionals, powered by vanna.ai.

Data-Copilot is an AI Assistant, speaks both natural language (like English) and machine languages (such as SQL and Python), acts like an interpreter between Analysts and Data, Analysts can now talk to data and derive insight faster than ever. With gained productivity, they can spend more time on deep analysis and strategic decision-making.

• Movie Explorer
• Music Store

• ChatGPT: ask general question on LLM models of choice
• RAG: ask dataset-specific question via Retrieval Augmented Generation
  • Semantic Search: discover data schema
  • Text-to-SQL: generate SQL from plain text
  • Data-to-Plot: generate Python code to visualize data
• Data Privacy (optional) : leverage Ollama and open-source LLM models locally

Choose data source, vector store, LLM model

Review database schema, data

Define knowledge base with table schema and documentation

Ask question on data (built on RAG), get answer in SQL, dataframe, python, plotly chart

Question & answer pairs are saved to database

Various LLM models for compared for chinook dataset, accuracy above 90% has been achieved for top proprietary models and open-source models.

Import data from CSV or connect to database

## (1) create virtual env
## ============================
conda create -n data_copilot [python=3.11]
conda activate data_copilot

## (2) create your local working folder
mkdir rag  

## (3) Install forked Vanna library from source
## =============================
cd rag
git clone git@github.com:digital-duck/ssadata.git

cd ssadata
pip3 install -e .

## (4) Install Data-Copilot from source 
## ============================
cd rag  # back to root folder
git clone git@github.com:digital-duck/data-copilot.git

cd data-copilot
pip3 install -r requirements.txt 

## (5) Enable LLM model access
## =============================
## available top LLM models are listed here: 
##  https://lmarena.ai/?leaderboard

## API_KEY is required to access proprietary models (such as Google/Anthropic/OpenAI)
cp .env.example .env   # add API_KEY in .env

## open-source LLM models are available locally via Ollama here: 
## https://ollama.com/
## no API_KEY is required

## (6) launch streamlit app
## =============================
cd src
streamlit run Data-Copilot.py  # make sure "streamlit" command added to your PATH env var

open browser at URL: http://localhost:8501

Add business terms and descriptions into knowledgebase to enhance context search - prepare a csv file with the following columns - business_term - business_description - related_tables - related_columns - import it into table bus_term - see bus_term.csv for example

Visit ollama.ai to install open-source LLM models and run them locally.