【python爬虫】-pythonwebscrapingtutorial资源-CSDN文库

需积分: 5 31 浏览量 2024-07-10 18:40:19 上传评论收藏 849KB PDF 举报

资源推荐

资源详情

资源评论

Python Web Scraping

Table of Contents

About the Tutorial .................................................................................................................................... i

Audience .................................................................................................................................................. i

Prerequisites ............................................................................................................................................ i

Copyright & Disclaimer ............................................................................................................................. i

Table of Contents .................................................................................................................................... ii

1. PYTHON WEB SCRAPING – INTRODUCTION ......................................................................... 1

What is Web Scraping?............................................................................................................................ 1

Origin of Web Scraping............................................................................................................................ 1

Web Crawling v/s Web Scraping ............................................................................................................. 1

Uses of Web Scraping .............................................................................................................................. 2

Components of a Web Scraper ................................................................................................................ 3

Working of a Web Scraper....................................................................................................................... 3

2. PYTHON WEB SCRAPING – GETTING STARTED WITH PYTHON ............................................. 5

Why Python for Web Scraping? ............................................................................................................... 5

Installation of Python .............................................................................................................................. 5

Setting Up the PATH ................................................................................................................................ 7

Running Python ....................................................................................................................................... 7

3. PYTHON WEB SCRAPING – PYTHON MODULES FOR WEB SCRAPING ................................... 9

Python Development Environments using virtualenv ......................................................................... 9

Python Modules for Web Scraping ........................................................................................................ 11

Requests ............................................................................................................................................... 11

Urllib3 ................................................................................................................................................... 12

Selenium ............................................................................................................................................... 13

Scrapy ................................................................................................................................................... 14

4. PYTHON WEB SCRAPING — LEGALITY OF WEB SCRAPING ................................................. 15

Python Web Scraping

iii

Introduction .......................................................................................................................................... 15

Research Required Prior to Scraping ..................................................................................................... 15

5. PYTHON WEB SCRAPING – DATA EXTRACTION .................................................................. 21

Web page Analysis ................................................................................................................................ 21

Different Ways to Extract Data from Web Page .................................................................................... 21

Beautiful Soup ....................................................................................................................................... 23

Lxml ...................................................................................................................................................... 24

6. PYTHON WEB SCRAPING – DATA PROCESSING .................................................................. 26

Introduction .......................................................................................................................................... 26

CSV and JSON Data Processing .............................................................................................................. 26

Data Processing using AWS S3 ............................................................................................................... 27

Data processing using MySQL ................................................................................................................ 28

Data processing using PostgreSQL ......................................................................................................... 30

7. PYTHON WEB SCRAPING – PROCESSING IMAGES AND VIDEOS ......................................... 31

Introduction .......................................................................................................................................... 31

Getting Media Content from Web Page ................................................................................................ 31

Extracting Filename from URL ............................................................................................................... 31

Information about Type of Content from URL ....................................................................................... 32

Generating Thumbnail for Images ......................................................................................................... 34

Screenshot from Website ...................................................................................................................... 34

Thumbnail Generation for Video ........................................................................................................... 35

Ripping an MP4 video to an MP3 .......................................................................................................... 36

8. PYTHON WEB SCRAPING – DEALING WITH TEXT ............................................................... 37

Introduction .......................................................................................................................................... 37

Getting started with NLTK ..................................................................................................................... 37

Installing Other Necessary packages ..................................................................................................... 38

Python Web Scraping

Tokenization ......................................................................................................................................... 38

Stemming .............................................................................................................................................. 39

Lemmatization ...................................................................................................................................... 39

Chunking ............................................................................................................................................... 40

Bag of Word (BoW) Model Extracting and converting the Text into Numeric Form ............................... 41

Building a Bag of Words Model in NLTK ................................................................................................ 42

Topic Modeling: Identifying Patterns in Text Data ................................................................................. 42

Topic Modeling Algorithms ................................................................................................................... 43

9. PYTHON WEB SCRAPING – SCRAPING DYNAMIC WEBSITES ............................................... 44

Introduction .......................................................................................................................................... 44

Dynamic Website Example .................................................................................................................... 44

Approaches for Scraping data from Dynamic Websites ......................................................................... 44

Reverse Engineering JavaScript ............................................................................................................. 45

Rendering JavaScript ............................................................................................................................. 46

10. PYTHON WEB SCRAPING — SCRAPING FORM BASED WEBSITES ....................................... 48

Introduction .......................................................................................................................................... 48

Interacting with Login forms ................................................................................................................. 48

Loading Cookies from the Web Server .................................................................................................. 49

Automating forms with Python ............................................................................................................. 50

11. PYTHON WEB SCRAPING — PROCESSING CAPTCHA .......................................................... 52

What is CAPTCHA? ................................................................................................................................ 52

Loading CAPTCHA with Python .............................................................................................................. 52

Pillow Python Package .......................................................................................................................... 53

OCR: Extracting Text from Image using Python ..................................................................................... 54

12. PYTHON WEB SCRAPING — TESTING WITH SCRAPERS ...................................................... 55

Introduction .......................................................................................................................................... 55

剩余20页未读，继续阅读

评论收藏

内容反馈