Google Scholar Scraper

A Windows desktop app for extracting, ranking, reviewing, and exporting Google Scholar search results.

Google Scholar Scraper V2 is a local research utility for collecting Scholar result pages, removing duplicates, applying optional lexical relevance ranking, and exporting reviewed results to Excel or CSV. It is source-available for noncommercial use under the PolyForm Noncommercial License 1.0.0, with separate commercial licensing available.

Google Scholar Scraper is not affiliated with Google. It does not solve CAPTCHA, bypass access controls, rotate proxies, use account credentials, or use stealth browser automation.

The original goal is simple: make small-scale academic result collection easier without turning the project into an unsafe crawler or a heavyweight data platform. V2 focuses on a dependable desktop workflow, explicit failure states, local-only ranking, deterministic exports, and Windows packaging.

What matters in practice:

• No API key is required.
• No LLM or external AI service is used.
• Smart Relevance Ranking runs locally.
• Rate limits, blocked pages, parser errors, and cancellations are explicit.
• Partial results are preserved when extraction stops early.
• Reviewed results export to Excel and CSV.
• Windows distribution supports both portable and installer-based workflows.

• Search Google Scholar by query and page count.
• Review collected articles in a responsive Tkinter/ttk desktop table.
• Open result links directly from the table.
• Enable or disable local relevance ranking.
• Preserve usable partial results when later pages fail or the user cancels.
• Export reviewed results to Excel or CSV.
• Run from source, portable Windows ZIP, or Windows installer artifacts.

Smart Relevance Ranking is optional and runs locally. It compares the query with article title and author/source metadata using deterministic token weighting and TF-IDF-style cosine scoring.

The score is shown as Relevance in the UI and Relevance Score in exports. It is a review aid, not a semantic understanding score, confidence value, citation-quality metric, or scientific assessment of a paper.

Ranking does not use:

• LLMs.
• External AI APIs.
• Embedding services.
• Model downloads.
• GPU acceleration.

The scraper uses a reusable HTTP session, explicit timeouts, conservative request headers, bounded retries, and request pacing. Google Scholar can still rate-limit, block, challenge, or change markup without notice.

V2 reports these states instead of treating every response as a successful empty result:

1. Enter a search query.
2. Choose the number of Google Scholar result pages to scan.
3. Choose an export folder.
4. Enable or disable Rank by relevance.
5. Click Search Scholar.
6. Review results in the table.
7. Open individual links as needed.
8. Export to Excel or CSV.

The UI remains responsive while extraction runs.

• Windows Installer
• Portable Windows ZIP
• SHA-256 checksums
• Full release notes

V2.0.1 is available as a Windows Installer and a Portable Windows ZIP. Checksums are published with the release:

• Google-Scholar-Scraper-v2.0.1-Portable-Windows-x64.zip
• Google-Scholar-Scraper-v2.0.1-Setup-Windows-x64.exe
• SHA256SUMS.txt

Release page: Google Scholar Scraper V2.0.1

1. Download the portable ZIP from the V2.0.1 release.
2. Verify the ZIP against SHA256SUMS.txt.
3. Extract the ZIP.
4. Run GoogleScholarScraper.exe from the extracted folder.

The portable build does not require a source checkout.

1. Download the installer from the V2.0.1 release.
2. Verify the installer against SHA256SUMS.txt.
3. Run the installer.
4. Launch it from the Start Menu or optional desktop shortcut.

The installer is built with Inno Setup and installs per user.

Requirements:

• Python 3.10 or newer.
• Tkinter available in the Python installation.

python -m pip install --upgrade pip
python -m pip install .
python -m google_scholar_scraper

Exported columns:

• Title
• Authors
• Link
• Relevance Score

When ranking is disabled, Relevance Score is left blank instead of inventing a score.

Google Scholar is an external service and may change behavior without notice. Automated access can be rate-limited, blocked, challenged, or affected by markup changes.

Use conservative page counts and respect applicable terms, policies, and laws. This project does not guarantee uninterrupted access or complete coverage of Google Scholar results.

• Ranking is performed locally.
• Exports are written to the selected local folder.
• No telemetry or analytics collection is implemented.
• No external AI service receives query or result data.
• Normal operation makes HTTP requests to Google Scholar for the query entered by the user.

python -m venv .venv
.\.venv\Scripts\Activate.ps1
python -m pip install --upgrade pip
python -m pip install -e .

Optional build dependency:

python -m pip install -c constraints\release-2.0.1.txt pyinstaller

Run the deterministic test suite:

python -m unittest discover -s tests

The tests use saved fixtures and mocked responses. They do not require live Google Scholar access.

Local portable build:

powershell -NoProfile -ExecutionPolicy Bypass -File scripts\build_windows.ps1

The script removes only known generated outputs (build/ and dist/) before building, then creates:

• PyInstaller onedir application under dist/Google-Scholar-Scraper-v2.0.1/
• Portable ZIP under dist/release/
• SHA256SUMS.txt

If Inno Setup is installed locally and -SkipInstaller is not used, the same script also compiles the installer. The Windows GitHub Actions build workflow is configured to install or locate Inno Setup and fail if the installer cannot be built.

Portable and installer distributions include LICENSE, NOTICE, COMMERCIAL_LICENSE.md, THIRD_PARTY_NOTICES.txt, and a generated third_party_licenses/ bundle collected from the exact constrained release-build environment.

src/google_scholar_scraper/
  app.py                  CLI/application entrypoint
  models.py               Article and extraction result models
  dedupe.py               Normalization, validation, duplicate removal
  ranking.py              Local lexical relevance ranking
  exporters.py            CSV and Excel export
  scraper/client.py       Request flow, retries, progress, cancellation
  scraper/parser.py       Google Scholar HTML parsing and page classification
  ui/tkinter_app.py       Desktop UI

tests/
  fixtures/               Saved deterministic HTML fixtures
  test_*.py               Parser, client, ranking, export, UI, packaging tests

packaging/pyinstaller/    PyInstaller configuration
installer/                Inno Setup definition
scripts/                  Windows build and release-compliance helpers
constraints/              Exact release-build dependency constraints
docs/                     Release and audit documentation

• V2.0.1 release notes
• V2.0.0 release notes
• V2.0.0 release verification record
• Changelog
• Third-party notices
• Third-party license bundle workflow
• PyInstaller spec
• Inno Setup definition
• Windows build script

Google Scholar Scraper is source-available for noncommercial use under the PolyForm Noncommercial License 1.0.0.

This is not an OSI-approved open-source license. See NOTICE for attribution and distribution notice information. See COMMERCIAL_LICENSE.md for the commercial licensing summary.

The public repository is available under the PolyForm Noncommercial License 1.0.0. Commercial use requires a separate written commercial license from Mahdi Navaei.

Commercial licensing inquiries are welcome. Pricing is not published publicly, and each commercial arrangement may be evaluated separately.

For commercial licensing inquiries, contact Mahdi Navaei.

Project sponsorship, relevant commercial collaborations, carefully scoped integrations, and ecosystem partnership discussions are welcome when they fit the project direction.

For sponsorship or integration inquiries, contact Mahdi Navaei.

Bug reports and focused issue reports are welcome. Larger code contribution policy is intentionally conservative until a licensing and assignment process is defined.

See NOTICE for attribution and distribution notice information.