A Scrapy project for scraping workabroad.ph job posts. This was my project as a research assistant for Dr. Emily Beam for her research on employment trends.
pip install scrapy
To scrape data from workabroad.ph
scrapy crawl post -t json -o outputfile.json
To change scraping settings, open workabroad/settings.py.
# Scrape job posts from the top n most recent pages
DEPTH_LIMIT = n
# Number of seconds to wait before fetching page
DOWNLOAD_DELAY = 10
# Wait a random time between 0.5 and 1.5 * DOWNLOAD_DELAY
RANDOMIZE_DOWNLOAD_DELAY = True
./sanitize.py csv inputfile.json outputfile.csv
./sanitize.py json inputfile.json outputfile.json