The Scrappey blog
Practical guides to web scraping, proxies, browser automation and web data pipelines. For short answers to specific questions, see the Q&A.
How to Rotate IP Address Safely in Web Scraping
Learn how to rotate IP address per request, per session or via a dynamic pool, plus retry logic, rate limits and compliance checks for web scrapers.
More articles
API integration patterns
API Integration Patterns: How to Choose the Right One
Learn the main API integration patterns, from request/response and webhooks to batch and streaming, with trade-offs, retry rules and scraping examples.
15 min readforbidden access denied
Forbidden Access Denied 403: Causes and Fixes for Scrapers
Forbidden Access Denied 403 errors blocking your scraper? Diagnose the cause, then fix headers, proxies, cookies and bot checks with curl examples.
10 min readsentiment analysis data
How to Source and Prepare Sentiment Analysis Data
Build sentiment analysis data end to end: pick sources, collect dynamic pages, label reliably, clean, split and check privacy with practical steps.
16 min readrental market analysis
Rental Market Analysis: A Step-by-Step Guide for 2026
Learn how to run a rental market analysis step by step: define KPIs, collect listing data, compute rent medians and vacancy, and build reports.
13 min readanti bot protection
Anti Bot Protection Explained: Defenses and Trade-offs
Learn how anti bot protection works: CAPTCHAs, rate limits, fingerprinting and WAF rules, plus fair, responsible scraping practices for site owners.
12 min readdatabase schema design
Database Schema Design: A Practical Guide for Scraped Data
Learn database schema design step by step: requirements, normalization, data types, indexes, and schemas for scraped product, price and lead data.
12 min readipv6 proxy service
IPv6 Proxy Service Explained: Setup, Limits and Providers
An IPv6 proxy service explained: how it differs from IPv4, where it works, how to integrate it in code and how to choose a provider.
12 min readcustomer feedback collection
Customer Feedback Collection Without the Bias
Customer feedback collection that counts the silent majority: channels, bias, consent, and a weekly public-review pipeline.
16 min readperformance benchmarking
How to Run Performance Benchmarking for Web Scrapers
Learn performance benchmarking for web scrapers: key metrics, repeatable test design, sample scripts and how to read p95 latency and success rates.
13 min readweb scraping
Web Scraping for Market Research: A Practical Guide
Learn when web scraping for market research beats surveys, which data to collect, how to build a reliable pipeline, and how to stay compliant.
14 min readweb scraping research
Web Scraping for Research: A Practical End-to-End Guide
Learn how to plan, legally check, build, clean and document web scraping for research, with a workflow table, Python code and scaling tips.
13 min readretail competitor pricing
Retail Competitor Pricing: How to Monitor and Use It
Learn how retail competitor pricing works, which pricing postures to use, how to collect and normalize rival prices fairly, and how to stay compliant.
14 min readencrypted dns traffic
Encrypted DNS Traffic: What Still Leaks in 2026
Encrypted DNS traffic hides names, not timing or the resolver. Compare DoH, DoT, DoQ and DNSCrypt, then fix monitoring and scraper failures.
13 min readcurl json post
Curl JSON POST: How to Send JSON Data with Curl
Curl JSON POST guide with working examples: headers, inline and file data, authentication, common errors and debugging fixes for 2026.
12 min readc# web scraping
A Practical Guide to C# Web Scraping in 2026
Learn modern C# web scraping with actionable advice. This guide covers libraries, dynamic sites, proxies, and scaling with real-world examples.
19 min readbackconnect proxies
Backconnect Residential Proxies Explained: How They Work
Learn how backconnect residential proxies route traffic, where they fall short, and how to check consent, rotation and latency before you rely on them.
11 min readeBay product research
eBay Product Research: A Seven-Step Playbook
Learn eBay product research step by step: pick the right tools, read sold data, validate demand and margins, and test listings before you buy stock.
11 min readbusiness intelligence tools
Best Tools for Business Intelligence: 10 Options Compared
Compare 10 of the best tools for business intelligence in 2026, from Power BI and Tableau to Metabase and Superset, by use case, scale, pricing, and fit.
14 min readreal-time data pipeline
Real Time Data Pipeline: How It Works and What to Build
Learn how a real time data pipeline works, compare it with batch, choose Kafka, Flink or Kinesis, and set freshness targets that match real business needs.
16 min readdata normalization techniques
8 Data Normalization Techniques Compared for 2026
Compare 8 data normalization techniques for ML, databases, text, and ETL, with use cases, Python and SQL examples, and fixes for common errors.
13 min readdata validation methods
Data Validation Methods for Scraped Data: A Guide
Learn the data validation methods that catch broken, stale and fabricated scraped data, with a table of checks, examples and when to use each.
13 min readweb scraping R
Web Scraping Using R: Packages, Steps and Fixes for 2026
Web scraping using R, step by step: pick the right package, parse HTML and JSON, handle JavaScript pages, blocks and rate limits.
12 min readprice monitoring api
Price Monitoring API: How to Build a Scalable System
Build a scalable price monitoring API pipeline step by step, from proxies and change detection to rate limits, alerts, storage and compliance.
12 min read