Extract anything. From anywhere. Without limits.

Three focused tools. One philosophy. Zero setup.

Every tool in this series ships as Docker images. No source code. No build step. Pull and run.

The NScrape Series is a collection of purpose-built data extraction tools — each one solving a specific scraping problem, each deployable with a single docker compose up.

They share one design philosophy:

Three independently built and published data extraction tools, each solving a different scraping problem and shipped as production Docker images — pull, run, done. Every tool exposes a REST API (Swagger docs included), a web UI, and configurable export formats (PDF, DOCX, ZIP, JSON).

• LetsNScrape — general-purpose scraper with a dual-engine pipeline (HTTP + Playwright), auto-switching for JS-rendered sites.
• SnapNScrape — scraping with an OCR failsafe (Playwright screenshot + Tesseract) for bot-blocked or JS-hostile pages.
• ytNscrape — a 9-service pipeline (yt-dlp + ffmpeg + a MiniLM embedding model) that turns YouTube lectures into deduplicated, captioned study PDFs.

Core skills demonstrated: Docker · Docker Compose · FastAPI · Playwright · OCR / Tesseract · Redis · REST API design · Swagger/OpenAPI · yt-dlp · Microservice architecture · CI/CD image publishing

This repo covers the tools. To see them running in production on AWS + self-managed Kubernetes, with Ansible, Helm, and Prometheus/Grafana monitoring — check out NSeries Platform Engineering. Full breakdown in execution-results/assets/demo-link.md.

├── execution-results/
│   └── assets/
│       ├── .gitkeep
│       ├── demo-link.md
│       └── nscrape-banner.svg
├── .gitignore
├── .vbcignore
├── letsNscrape_documentation.md
├── README.md
├── snapNscrape_documentation.md
└── ytNscrape_documentation.md

This README covers each tool at a high level only. For setup, configuration, API reference, and troubleshooting, always refer to the dedicated documentation file for that tool:

• letsNscrape_documentation.md
• ytNscrape_documentation.md
• snapNscrape_documentation.md

The web is your database. LetsNScrape is your query.

A general-purpose web scraper that extracts exactly the data you need from any webpage. Text, images, links, metadata, headings, tables — individually or all at once. Handles both static HTML sites and JavaScript-rendered SPAs.

Most scrapers fail on modern sites. LetsNScrape runs a dual-engine pipeline — fast HTTP scrape first, then automatic Playwright fallback for JS-heavy sites. It auto-detects which engine is needed; you don't have to choose.

╔══════════════════════════════════════════════════════════════════╗
║  letsnscape-frontend  :3071   React UI + Nginx reverse proxy     ║
║  letsnscape-backend   :3070   FastAPI scraping API               ║
║  letsnscape-redis     :6380   Async job queue + result store     ║
╚══════════════════════════════════════════════════════════════════╝
                          |
              ┌───────────┴───────────┐
              ▼                       ▼
         Scrapy / HTTP           Playwright
         (static sites,         (React, Vue, Angular,
          fast path)             Next.js — auto-switched)

📖 Full setup guide, port/config reference, API docs, and troubleshooting: letsNscrape_documentation.md

Paste a YouTube lecture URL. Get a note-ready PDF and a ZIP of clean slide frames — automatically.

A YouTube lecture scraper that downloads a video, extracts and deduplicates slide frames by content type, syncs captions, and renders everything into a study-ready PDF and ZIP — fully automated.

Watching a lecture to take notes is slow. Pausing to screenshot slides is tedious. ytNscrape processes the entire video in the background and hands you a structured notes PDF with ruled lines for handwriting, plus a ZIP of every unique slide frame labelled by type (code, software UI, or slide).

Frontend  :3000  (nginx + vanilla JS)
  └── Gateway  :3082  (FastAPI + SSE live progress)
        └── Redis job queue  :3061
              ├── svc-ingest       yt-dlp + ffmpeg
              ├── svc-dedup        pHash dedup
              ├── svc-priority     OCR + MiniLM
              ├── svc-postprocess  Pillow crop/sharpen
              ├── svc-captions     VTT sync
              ├── svc-render-zip   ZIP output
              └── svc-render-pdf   PDF output
                        └── MinIO  :9000 / :9001

📖 Full setup guide, port/config reference, API docs, and troubleshooting: ytNscrape_documentation.md

Extract any web content. Without limits.

A containerised web scraping platform with a three-stage intelligent pipeline. Built for sites that actively block bots, rely on JavaScript rendering, or hide content behind dynamic layers. When HTTP fails and Playwright fails, it screenshots the page and reads the text with Tesseract OCR.

Gateway  :${PORT}  (Web UI · REST API · DOCX export · Redis cache)
  └── Scraper  :8001  (internal only — never exposed)
        ├── Stage 1 — HTTP scrape  (httpx + BeautifulSoup4)  fast path
        └── Stage 2 — OCR failsafe (Playwright Chromium + Tesseract)
Redis  :6379  (result cache — internal only)

📖 Full setup guide, port/config reference, API docs, and troubleshooting: snapNscrape_documentation.md