Crawly is an application framework for crawling web sites and extracting structured data which can be used for a wide range of useful applications, like data mining, information processing or historical archival.
- Elixir "~> 1.7"
- Works on Linux, Windows, OS X and BSD
- Generate an new Elixir project:
mix new <project_name> --sup - Add Crawly to you mix.exs file
def deps do [{:crawly, "~> 0.5.0"}] end
- Fetch crawly:
mix deps.get
In this section we will show how to bootstrap a small project and to setup Crawly for proper data extraction.
- Create a new Elixir project:
mix new crawly_example --sup - Add Crawly to the dependencies (mix.exs file):
defp deps do
[
{:crawly, "~> 0.5.0"}
]
end- Fetch dependencies:
mix deps.get - Define Crawling rules (Spider)
cat > lib/crawly_example/esl_spider.ex << EOF
defmodule EslSpider do
@behaviour Crawly.Spider
alias Crawly.Utils
@impl Crawly.Spider
def base_url(), do: "https://www.erlang-solutions.com"
@impl Crawly.Spider
def init(), do: [start_urls: ["https://www.erlang-solutions.com/blog.html"]]
@impl Crawly.Spider
def parse_item(response) do
hrefs = response.body |> Floki.find("a.more") |> Floki.attribute("href")
requests =
Utils.build_absolute_urls(hrefs, base_url())
|> Utils.requests_from_urls()
title = response.body |> Floki.find("article.blog_post h1") |> Floki.text()
%{
:requests => requests,
:items => [%{title: title, url: response.request_url}]
}
end
end
EOF- Configure Crawly: By default Crawly does not require any configuration. But obviously you will need a configuration for fine tuning the Crawls:
config :crawly,
closespider_timeout: 10,
concurrent_requests_per_domain: 8,
follow_redirects: true,
closespider_itemcount: 1000,
output_format: "csv",
item: [:title, :url],
item_id: :title,
middlewares: [
Crawly.Middlewares.DomainFilter,
Crawly.Middlewares.UniqueRequest,
Crawly.Middlewares.UserAgent
],
pipelines: [
Crawly.Pipelines.Validate,
Crawly.Pipelines.DuplicatesFilter,
Crawly.Pipelines.CSVEncoder
]- Start the Crawl:
iex -S mixCrawly.Engine.start_spider(EslSpider)
- Results can be seen in:
cat /tmp/EslSpider.jl
Documentation is available online at https://oltarasenko.github.io/crawly/#/ and in the docs directory.
The crawly tutorial: https://oltarasenko.github.io/crawly/#/?id=crawly-tutorial
- Cookies support
- XPath support
- Pluggable HTTP client
- Project generators (spiders)
- Retries support
- UI for jobs management
We would gladly accept your contributions!
- Blog post on Erlang Solutions website: https://www.erlang-solutions.com/blog/web-scraping-with-elixir.html
- Blog crawler: https://github.com/oltarasenko/crawly-spider-example
- E-commerce websites: https://github.com/oltarasenko/products-advisor
- Car shops: https://github.com/oltarasenko/crawly-cars