Node.js基于express框架的小爬虫_爬虫框架对比资源-CSDN文库

共2000个文件

js：6183个

md：1251个

json：1242个

需积分: 12 34 浏览量 2018-06-23 12:30:41 上传评论收藏 12.75MB ZIP 举报

**Node.js基于Express框架的小爬虫** Node.js是一款强大的服务器端JavaScript运行环境，它使得开发者能够在服务器端使用JavaScript编写代码，极大地提高了开发效率。而Express是Node.js领域中最流行的Web应用框架，它简化了构建Web服务的过程。在这个“Node.js基于express框架的小爬虫”项目中，我们将学习如何结合这两者构建一个简单的网络爬虫。让我们深入理解Node.js爬虫的基本原理。爬虫是一种自动化程序，用于从互联网上抓取信息。它通过模拟浏览器发送HTTP请求到目标网站，接收返回的HTML或其他格式的数据，然后解析这些数据以提取所需信息。在这个项目中，我们使用`crawler.js`文件作为主要的爬虫实现。在`crawler.js`中，我们可能用到了以下核心模块： 1. **`http`或`https`模块**：用于发送HTTP/HTTPS请求。 2. **`cheerio`库**：这是一个快速、灵活且符合jQuery语法的HTML解析器，使我们可以像操作DOM一样处理HTML字符串。 3. **`request`或`axios`库**：提供了一种简单的方式来发送HTTP请求，处理响应。为了实现爬虫，我们需要进行以下步骤： 1. **设置请求**：使用`request`或`axios`库向目标URL发起GET请求，获取网页的HTML内容。 2. **解析HTML**：利用`cheerio`库加载HTML字符串，通过CSS选择器找到我们感兴趣的数据。如果你提到需要用户有jQuery基础，那么这部分操作会很熟悉，如`$('selector').text()`或`$('selector').attr('attribute')`等。 3. **处理数据**：一旦找到所需信息，可以对其进行清洗、转换、存储或进一步处理。这可能涉及字符串处理、正则表达式、JSON操作等。 4. **错误处理**：确保处理可能出现的网络错误、解析错误等，并采取适当的恢复策略。在Express框架中，我们可能将这个爬虫集成到一个简单的Web服务中，用户可以通过发送特定的请求参数来触发爬虫执行。例如，创建一个路由`/scrape`，接受URL作为参数，然后调用`crawler.js`中的函数进行爬取。 ```javascript const express = require('express'); const app = express(); const router = express.Router(); // 引入crawler.js const crawler = require('./Router/crawler'); router.get('/scrape', (req, res) => { const url = req.query.url; // 获取请求的URL参数 crawler.scrape(url, (err, data) => { // 调用爬虫函数 if (err) return res.status(500).send(err); res.json(data); // 返回爬取的数据 }); }); app.use('/', router); app.listen(3000, () => console.log('Server running on port 3000')); ``` 在这个简单的服务中，用户可以向`http://localhost:3000/scrape?url=<target_url>`发送GET请求，爬虫将运行并返回爬取的结果。需要注意的是，爬虫必须遵循`robots.txt`协议，尊重网站的爬取限制，避免对目标网站造成过大负担。此外，爬虫还应处理可能出现的验证码、登录验证等问题，以及处理动态加载的内容，这可能需要借助于如`puppeteer`这样的浏览器自动化工具。这个"Node.js基于Express框架的小爬虫"项目提供了一个基础的爬虫实现，使用者可以根据自身需求定制爬取规则，通过Express框架轻松地构建一个自定义的Web服务接口。了解和掌握这些技术将有助于你进行更复杂的数据抓取和分析任务。

资源推荐

资源详情

资源评论

收起资源包目录

Node.js基于express框架的小爬虫（2000个子文件）

style.css 111B

index.html 792B

error.html 83B

lodash.js 527KB

core.js 112KB

index.js 86KB

jquery-3.2.1.min.js 85KB

named_entity_data.js 72KB

lodash.min.js 71KB

index.js 63KB

ejs.js 42KB

sbcs-data-generated.js 31KB

_stream_readable.js 31KB

response.js 26KB

ejs.js 25KB

Tokenizer.js 25KB

index.js 23KB

parse.js 22KB

dbcs-codec.js 21KB

ejs.min.js 20KB

stringify.js 20KB

_stream_writable.js 20KB

共 2000 条

<h1 align="center">cheerio</h1> <h5 align="center">Fast, flexible & lean implementation of core jQuery designed specifically for the server.</h5> <div align="center"> <a href="http://travis-ci.org/cheeriojs/cheerio"> <img src="https://secure.travis-ci.org/cheeriojs/cheerio.svg?branch=master" alt="Travis CI" /> </a> <a href="https://coveralls.io/r/cheeriojs/cheerio"> <img src="http://img.shields.io/coveralls/cheeriojs/cheerio.svg?branch=master&style=flat" alt="Coverage" /> </a> <a href="https://gitter.im/cheeriojs/cheerio?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge"> <img src="https://badges.gitter.im/Join%20Chat.svg" alt="Join the chat at https://gitter.im/cheeriojs/cheerio" /> </a> <a href="#backers"> <img src="https://opencollective.com/cheerio/backers/badge.svg" alt="OpenCollective backers"/> </a> <a href="#sponsors"> <img src="https://opencollective.com/cheerio/sponsors/badge.svg" alt="OpenCollective sponsors"/> </a> </div> <br /> ```js const cheerio = require('cheerio') const $ = cheerio.load('<h2 class="title">Hello world</h2>') $('h2.title').text('Hello there!') $('h2').addClass('welcome') $.html() //=> <html><head></head><body><h2 class="title welcome">Hello there!</h2></body></html> ``` ## Installation `npm install cheerio` ## Features __❤ Familiar syntax:__ Cheerio implements a subset of core jQuery. Cheerio removes all the DOM inconsistencies and browser cruft from the jQuery library, revealing its truly gorgeous API. __ϟ Blazingly fast:__ Cheerio works with a very simple, consistent DOM model. As a result parsing, manipulating, and rendering are incredibly efficient. Preliminary end-to-end benchmarks suggest that cheerio is about __8x__ faster than JSDOM. __❁ Incredibly flexible:__ Cheerio wraps around [parse5](https://github.com/inikulin/parse5) parser and can optionally use @FB55's forgiving [htmlparser2](https://github.com/fb55/htmlparser2/). Cheerio can parse nearly any HTML or XML document. ## Cheerio is not a web browser Cheerio parses markup and provides an API for traversing/manipulating the resulting data structure. It does not interpret the result as a web browser does. Specifically, it does *not* produce a visual rendering, apply CSS, load external resources, or execute JavaScript. If your use case requires any of this functionality, you should consider projects like [PhantomJS](http://phantomjs.org/) or [JSDom](https://github.com/tmpvar/jsdom). ## Job Board Looking for a career upgrade? Check out the available Node.js & Javascript positions at these innovative companies: <a href="https://astro.netlify.com/automattic"><img src="https://astro.netlify.com/static/automattic.png"></a> <a href="https://astro.netlify.com/segment"><img src="https://astro.netlify.com/static/segment.png"></a> <a href="https://astro.netlify.com/auth0"><img src="https://astro.netlify.com/static/auth0.png"/></a> ## API ### Markup example we'll be using: ```html <ul id="fruits"> <li class="apple">Apple</li> <li class="orange">Orange</li> <li class="pear">Pear</li> </ul> ``` This is the HTML markup we will be using in all of the API examples. ### Loading First you need to load in the HTML. This step in jQuery is implicit, since jQuery operates on the one, baked-in DOM. With Cheerio, we need to pass in the HTML document. This is the _preferred_ method: ```js const cheerio = require('cheerio'); const $ = cheerio.load('<ul id="fruits">...</ul>'); ``` Optionally, you can also load in the HTML by passing the string as the context: ```js const $ = require('cheerio'); $('ul', '<ul id="fruits">...</ul>'); ``` Or as the root: ```js const $ = require('cheerio'); $('li', 'ul', '<ul id="fruits">...</ul>'); ``` If you need to modify parsing options for XML input, you may pass an extra object to `.load()`: ```js const $ = cheerio.load('<ul id="fruits">...</ul>', { xml: { normalizeWhitespace: true, } }); ``` The options in the `xml` object are taken directly from [htmlparser2](https://github.com/fb55/htmlparser2/wiki/Parser-options), therefore any options that can be used in `htmlparser2` are valid in cheerio as well. The default options are: ```js { withDomLvl1: true, normalizeWhitespace: false, xmlMode: true, decodeEntities: true } ``` For a full list of options and their effects, see [this](https://github.com/fb55/DomHandler) and [htmlparser2's options](https://github.com/fb55/htmlparser2/wiki/Parser-options). Some users may wish to parse markup with the `htmlparser2` library, and traverse/manipulate the resulting structure with Cheerio. This may be the case for those upgrading from pre-1.0 releases of Cheerio (which relied on `htmlparser2`), for those dealing with invalid markup (because `htmlparser2` is more forgiving), or for those operating in performance-critical situations (because `htmlparser2` may be faster in some cases). Note that "more forgiving" means `htmlparser2` has error-correcting mechanisms that aren't always a match for the standards observed by web browsers. This behavior may be useful when parsing non-HTML content. To support these cases, `load` also accepts a `htmlparser2`-compatible data structure as its first argument. Users may install `htmlparser2`, use it to parse input, and pass the result to `load`: ```js // Usage as of htmlparser2 version 3: const htmlparser2 = require('htmlparser2'); const dom = htmlparser2.parseDOM(document, options); const $ = cheerio.load(dom); ``` ### Selectors Cheerio's selector implementation is nearly identical to jQuery's, so the API is very similar. #### $( selector, [context], [root] ) `selector` searches within the `context` scope which searches within the `root` scope. `selector` and `context` can be a string expression, DOM Element, array of DOM elements, or cheerio object. `root` is typically the HTML document string. This selector method is the starting point for traversing and manipulating the document. Like jQuery, it's the primary method for selecting elements in the document, but unlike jQuery it's built on top of the CSSSelect library, which implements most of the Sizzle selectors. ```js $('.apple', '#fruits').text() //=> Apple $('ul .pear').attr('class') //=> pear $('li[class=orange]').html() //=> Orange ``` ### Attributes Methods for getting and modifying attributes. #### .attr( name, value ) Method for getting and setting attributes. Gets the attribute value for only the first element in the matched set. If you set an attribute's value to `null`, you remove that attribute. You may also pass a `map` and `function` like jQuery. ```js $('ul').attr('id') //=> fruits $('.apple').attr('id', 'favorite').html() //=> <li class="apple" id="favorite">Apple</li> ``` > See http://api.jquery.com/attr/ for more information #### .prop( name, value ) Method for getting and setting properties. Gets the property value for only the first element in the matched set. ```js $('input[type="checkbox"]').prop('checked') //=> false $('input[type="checkbox"]').prop('checked', true).val() //=> ok ``` > See http://api.jquery.com/prop/ for more information #### .data( name, value ) Method for getting and setting data attributes. Gets or sets the data attribute value for only the first element in the matched set. ```js $('<div data-apple-color="red"></div>').data() //=> { appleColor: 'red' } $('<div data-apple-color="red"></div>').data('apple-color') //=> 'red' const apple = $('.apple').data('kind', 'mac') apple.data('kind') //=> 'mac' ``` > See http://api.jquery.com/data/ for more information #### .val( [value] ) Method for getting and setting the value of input, select, and textarea. Note: Support for `map`, and `function` has not been added yet. ```js $('input[type="text"]').val() //=> input_text $('input[type="text"]').val('test').html() //=> <input type="text" value="test"/> ``` #### .removeAttr( name ) Method for removing attributes by `name`.

评论收藏

内容反馈