Library to build personalized AI powered by what you've seen, said, or heard. Works with Ollama. Alternative to Rewind.ai. Open. Secure. You own your data. Rust.
We are shipping daily, make suggestions, post bugs, give feedback.

Building a reliable stream of audio and screenshot data, where a user simply clicks a button and the script runs in the background 24/7, collecting and extracting data from screen and audio input/output, can be frustrating.

There are numerous use cases that can be built on top of this layer. To simplify life for other developers, we decided to solve this non-trivial problem. It's still in its early stages, but it works end-to-end. We're working on this full-time and would love to hear your feedback and suggestions.

Struggle to get it running? I'll install it with you in a 15 min call.

brew install pkg-config
brew install ffmpeg
brew install jq

brew tap louis030195/screen-pipe https://github.com/louis030195/screen-pipe.git
brew install screenpipe

screenpipe 

screenpipe --disable-audio

screenpipe --save-text-files

screenpipe --debug

brew install pkg-config
brew install ffmpeg
brew install jq

git clone https://github.com/louis030195/screen-pipe

cd screen-pipe # enter cloned repo

cargo build --release --features metal

codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime ./target/release/screenpipe

./target/release/screenpipe # add "--disable-audio" if you don't want audio to be recorded
# "--save-text-files" if you want to save OCR data to text file in text_json folder in the root of your project (good for testing)
# "--debug" if you want to run screenpipe in debug mode to show more logs in terminal

# Install ffmpeg (you may need to download and add it to your PATH manually)
# Visit https://www.ffmpeg.org/download.html for installation instructions

# Install Chocolatey from https://chocolatey.org/install
# Then install pkg config
choco install pkgconfiglite

git clone https://github.com/louis030195/screen-pipe
cd screen-pipe

# This runs a local SQLite DB + an API + screenshot, ocr, mic, stt, mp4 encoding
cargo build --release --features cuda # remove "--features cuda" if you do not have a NVIDIA GPU

# then run it
./target/release/screenpipe

sudo apt-get update
sudo apt-get install -y libavformat-dev libavfilter-dev libavdevice-dev ffmpeg libasound2-dev tesseract-ocr libtesseract-dev

git clone https://github.com/louis030195/screen-pipe
cd screen-pipe

cargo build --release --features cuda # remove "--features cuda" if you do not have a NVIDIA GPU

# then run it
./target/release/screenpipe

git clone https://github.com/louis030195/screen-pipe

cd screen-pipe/examples/ts/vercel-ai-chatbot 

echo "OPENAI_API_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" > .env

npm install 

npm run dev

You can use terminal commands to query and view your data as shown below. Also, we recommend Tableplus.com to view the database, it has a free tier.

sqlite3 ~/.screenpipe/db.sqlite ".tables" 

sqlite3 ~/.screenpipe/db.sqlite ".mode json" ".once /dev/stdout" "SELECT * FROM audio_transcriptions ORDER BY id DESC LIMIT 1;" | jq .

sqlite3 ~/.screenpipe/db.sqlite ".mode json" ".once /dev/stdout" "SELECT * FROM ocr_text ORDER BY frame_id DESC LIMIT 1;" | jq -r '.[0].text'

ffplay "data/2024-07-12_01-14-14.mp4"

ffplay "data/Display 1 (output)_2024-07-12_01-14-11.mp4"

curl "http://localhost:3030/search?q=Neuralink&limit=5&offset=0&content_type=ocr" | jq

# 2. Search with content type filter (OCR)
curl "http://localhost:3030/search?q=QUERY_HERE&limit=5&offset=0&content_type=ocr"

# 3. Search with content type filter (Audio)
curl "http://localhost:3030/search?q=QUERY_HERE&limit=5&offset=0&content_type=audio"

# 4. Search with pagination
curl "http://localhost:3030/search?q=QUERY_HERE&limit=10&offset=20"

# 6. Search with no query (should return all results)
curl "http://localhost:3030/search?limit=5&offset=0"

• Search
  • Semantic and keyword search. Find information you've forgotten or misplaced
  • Playback history of your desktop when searching for a specific info
• Automation:
  • Automatically generate documentation
  • Populate CRM systems with relevant data
  • Synchronize company knowledge across platforms
  • Automate repetitive tasks based on screen content
• Analytics:
  • Track personal productivity metrics
  • Organize and analyze educational materials
  • Gain insights into areas for personal improvement
  • Analyze work patterns and optimize workflows
• Personal assistant:
  • Summarize lengthy documents or videos
  • Provide context-aware reminders and suggestions
  • Assist with research by aggregating relevant information
  • Live captions, translation support
• Collaboration:
  • Share and annotate screen captures with team members
  • Create searchable archives of meetings and presentations
• Compliance and security:
  • Track what your employees are really up to
  • Monitor and log system activities for audit purposes
  • Detect potential security threats based on screen content

Check this example of screenpipe which is a chatbot that make requests to your data to answer your questions

Alpha: runs on my computer (Macbook pro m3 32 GB ram) 24/7.

• screenshots + OCR
• audio + STT (works with multi input & output devices, like your iPhone + mac mic)
• optimised screen & audio recording (mp4 encoding, estimating 30 gb/m with default settings)
• sqlite local db
• local api
• MacOS CLI (silicon, intel)
• Windows CLI (nvidia, etc.)
• Linux CLI (x11, nvidia, etc.)
• desktop app for MacOS (launching within days)
• desktop app for Windows
• desktop app for Linux
• TS SDK
• multimodal embeddings
• cloud storage options (s3, pgsql, etc.)
• cloud computing options
• bug-free & stable
• custom storage settings: customizable capture settings (fps, resolution)
• data encryption options & higher security
• fast, optimised, energy-efficient modes
• webhooks (for automations)

• Data post-processing:
• • Wikipedia style data structuring
• • Knowledge graph
• Android integration
• Integration with wearables (e.g. Friend)
• Webcamera integration?
• Keylogger?
• UI low-code builder

Example with vercel/ai-chatbot project here inside the repo here:

Recent breakthroughs in AI have shown that context is the final frontier. AI will soon be able to incorporate the context of an entire human life into its 'prompt', and the technologies that enable this kind of personalisation should be available to all developers to accelerate access to the next stage of our evolution.

Contributions are welcome! If you'd like to contribute, please read CONTRIBUTING.md.

- RAG & question answering
- Automation (write code somewhere else while watching you coding, write docs, fill your CRM, sync company's knowledge, etc.)
- Analytics (track human performance, education, become aware of how you can improve, etc.)
- etc.
- We're constantly exploring new use cases and welcome community input!