A crawler for the web version of PTT, the largest online community in Taiwan.

Yet another PTT crawler but written in Rust. Can be used as binary directly or as crate.

• ptt-crawler (ptc)
• Table of Contents
  • Features
  • Getting started
    • Installation
      • From crates.io
      • From the sources
    • How to use
  • Used as crate
  • Run unit tests
  • Contributing
  • Links
  • Versioning
  • License

Created by gh-md-toc

• Single executable without any dependence
• Cross platforms supported
• Crawl single article or multiple articles in one board
• Anti-Anti-Crawler with random user agent and proxy server

The binary name for ptt-crawler is ptc . Currently, no precompiled binary is available. You need Rust 1.40 or higher and use cargo to build ptt-crawler from the sources.

> cargo install ptt-crawler

> git clone https://github.com/cwouyang/ptt-crawler.git
> cd ptt-crawler
> cargo build --release

• Crawls specific article

> ptc url https://www.ptt.cc/bbs/Gossiping/M.1597463395.A.478.html

Specify flags user agent -u and proxy -p used during crawling

> ptc -u "user/agent/string" -p "https://some.proxy" url https://www.ptt.cc/bbs/Gossiping/M.1597463395.A.478.html

# pass "random" to use randomly generated user agent
> ptc -u "random" https://www.ptt.cc/bbs/Gossiping/M.1597463395.A.478.html

• Crawls articles of board within page range

# From page 100 (https://www.ptt.cc/bbs/Gossiping/index100.html) to 200 (https://www.ptt.cc/bbs/Gossiping/index200.html)
> ptc board Gossiping -r 100 200

# From page 1 to latest page
> ptc board Gossiping

Use -l flag to list supported boards

> ptc board Gossiping --list

Add ptt-crawler as dependence in Cargo.toml file

[dependencies]
ptt-crawler = "0.1"

See document for usages.

> cargo test --all

If you'd like to contribute, please fork the repository and use a feature branch. Pull requests are warmly welcome.

Before submit pull request, make sure

• clippy was applied
• Commit messages with Conventional Commits (See here for detailed format)

• Project homepage: https://github.com/cwouyang/ptt-crawler/
• Issue tracker: https://github.com/cwouyang/ptt-crawler/issues
• Related projects
  • ptt-web-crawler: PTT crawler in Python version

We use SemVer for versioning. For the versions available, see the tags on this repository.

Copyright (c) 2020 cwouyang.

This project is licensed under the terms of MIT License. See the LICENSE file for details.