Learn web scraping and crawling techniques to access unlimited data from any web source in any format. With this practical guide, you’ll learn how to use Python scripts and web APIs to gather and process data from thousands—or even millions—of web pages at once. Ideal for programmers, security professionals, and web administrators familiar with Python, this book not only teaches basic web scraping mechanics, but also delves into more advanced topics, such as analyzing raw data or using scrapers for frontend website testing. Code samples are available to help you understand the concepts in practice. Learn how to parse complicated HTML pages Traverse multiple pages and sites Get a general overview of APIs and how they work Learn several methods for storing the data you scrape Download, read, and extract data from documents Use tools and techniques to clean badly formatted data Read and write natural languages Crawl through forms and logins Understand how to scrape JavaScript Learn image processing and text recognition Table of Contents Part I. Building Scrapers Chapter 1. Your First Web Scraper Chapter 2. Advanced HTML Parsing Chapter 3. Starting to Crawl Chapter 4. Using APIs Chapter 5. Storing Data Chapter 6. Reading Documents Part II. Advanced Scraping Chapter 7. Cleaning Your Dirty Data Chapter 8. Reading and Writing Natural Languages Chapter 9. Crawling Through Forms and Logins Chapter 10. Scraping JavaScript Chapter 11. Image Processing and Text Recognition Chapter 12. Avoiding Scraping Traps Chapter 13. Testing Your Website with Scrapers Chapter 14. Scraping Remotely Appendix A. Python at a Glance Appendix B. The Internet at a Glance Appendix C. The Legalities and Ethics of Web Scraping