没有合适的资源?快使用搜索试试~ 我知道了~
Web Scraping with Python_Collecting Data from the Modern Web
需积分: 9 13 下载量 60 浏览量
2016-06-30
15:15:47
上传
评论
收藏 7MB PDF 举报
温馨提示
试读
252页
Web Scraping with Python_Collecting Data from the Modern Web,英文原版pdf
资源推荐
资源详情
资源评论
Ryan Mitchell
Boston
Web Scraping with Python
Collecting Data from the Modern Web
978-1-491-91027-6
[LSI]
Web Scraping with Python
by Ryan Mitchell
Copyright © 2015 Ryan Mitchell. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472.
O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are
also available for most titles (http://safaribooksonline.com). For more information, contact our corporate/
institutional sales department: 800-998-9938 or corporate@oreilly.com.
Editors: Simon St. Laurent and Allyson MacDonald
Production Editor: Shiny Kalapurakkel
Copyeditor: Jasmine Kwityn
Proofreader: Carla Thornton
Indexer: Lucie Haskins
Interior Designer: David Futato
Cover Designer: Karen Montgomery
Illustrator: Rebecca Demarest
June 2015: First Edition
Revision History for the First Edition
2015-06-10: First Release
See http://oreilly.com/catalog/errata.csp?isbn=9781491910276 for release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc. Web Scraping with Python, the cover
image, and related trade dress are trademarks of O’Reilly Media, Inc.
While the publisher and the author have used good faith efforts to ensure that the information and
instructions contained in this work are accurate, the publisher and the author disclaim all responsibility
for errors or omissions, including without limitation responsibility for damages resulting from the use of
or reliance on this work. Use of the information and instructions contained in this work is at your own
risk. If any code samples or other technology this work contains or describes is subject to open source
licenses or the intellectual property rights of others, it is your responsibility to ensure that your use
thereof complies with such licenses and/or rights.
Table of Contents
Preface. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii
Part I. Building Scrapers
1. Your First Web Scraper. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
Connecting 3
An Introduction to BeautifulSoup 6
Installing BeautifulSoup 6
Running BeautifulSoup 8
Connecting Reliably 9
2. Advanced HTML Parsing. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
You Don’t Always Need a Hammer 13
Another Serving of BeautifulSoup 14
find() and findAll() with BeautifulSoup 16
Other BeautifulSoup Objects 18
Navigating Trees 18
Regular Expressions 22
Regular Expressions and BeautifulSoup 27
Accessing Attributes 28
Lambda Expressions 28
Beyond BeautifulSoup 29
3. Starting to Crawl. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Traversing a Single Domain 31
Crawling an Entire Site 35
Collecting Data Across an Entire Site 38
Crawling Across the Internet 40
Crawling with Scrapy 45
4.
Using APIs. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
How APIs Work 50
iii
Common Conventions 50
Methods 51
Authentication 52
Responses 52
API Calls 53
Echo Nest 54
A Few Examples 54
Twitter 55
Getting Started 56
A Few Examples 57
Google APIs 60
Getting Started 60
A Few Examples 61
Parsing JSON 63
Bringing It All Back Home 64
More About APIs 68
5. Storing Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
Media Files 71
Storing Data to CSV 74
MySQL 76
Installing MySQL 77
Some Basic Commands 79
Integrating with Python 82
Database Techniques and Good Practice 85
“Six Degrees” in MySQL 87
Email 90
6. Reading Documents. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
Document Encoding 93
Text 94
Text Encoding and the Global Internet 94
CSV 98
Reading CSV Files 98
PDF 100
Microsoft Word and .docx 102
Part II. Advanced Scraping
7. Cleaning Your Dirty Data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Cleaning in Code 109
iv | Table of Contents
剩余251页未读,继续阅读
资源评论
AllenMood
- 粉丝: 4
- 资源: 77
上传资源 快速赚钱
- 我的内容管理 展开
- 我的资源 快来上传第一个资源
- 我的收益 登录查看自己的收益
- 我的积分 登录查看自己的积分
- 我的C币 登录后查看C币余额
- 我的收藏
- 我的下载
- 下载帮助
安全验证
文档复制为VIP权益,开通VIP直接复制
信息提交成功