Getting started with Scrapy
Get started with scraping your first website
Learn the various ways to selecting data on a web page
Learn to identify links and navigate from one page to another to continue scraping. Also learn how to selectively scrape patterns of urls to avoid scraping unnecessary pages.
Export data to formats that can be parsed easily by other programs. Learn about the JSON Lines format
Managing scraped data
Organize scraped data with objects and learn to post-process them with Item Pipelines.
Configuring a spider
Learn to do more with a spider with some handy tricks.
Get up to speed with scrapy concepts that will come in handy.
This section is a reminder that there are real people running websites and the things you can do to be polite when scraping.
Learn how to get past limitations like forms, login or scrape websites with dynamic content
And then there are times that you need to fly below the radar
Scraping other content
Learn to scrape different kinds of content using scrapy
Deploying scrapy projects
Sure you can run your scraping projects on your own computer. But for large scraping projects, running them on servers allow you to scale beyond your bedroom
Some tricks to speed up your implementation when scraping websites
From me to you
Everything in this course is from my learnings in the last few years building and maintaining large scraping projects, and helping other folks do the same.
This course is all my notes and learnings, so that you move onto your projects without struggle."
— Akash Manohar (@HashNuke)