# Crawlab
<p>
<a href="https://hub.docker.com/r/tikazyq/crawlab/builds" target="_blank">
<img src="https://github.com/crawlab-team/crawlab/workflows/Docker/badge.svg">
</a>
<a href="https://hub.docker.com/r/tikazyq/crawlab" target="_blank">
<img src="https://img.shields.io/docker/pulls/tikazyq/crawlab?label=pulls&logo=docker">
</a>
<a href="https://github.com/crawlab-team/crawlab/releases" target="_blank">
<img src="https://img.shields.io/github/release/crawlab-team/crawlab.svg?logo=github">
</a>
<a href="https://github.com/crawlab-team/crawlab/commits/master" target="_blank">
<img src="https://img.shields.io/github/last-commit/crawlab-team/crawlab.svg">
</a>
<a href="https://github.com/crawlab-team/crawlab/issues?q=is%3Aissue+is%3Aopen+label%3Abug" target="_blank">
<img src="https://img.shields.io/github/issues/crawlab-team/crawlab/bug.svg?label=bugs&color=red">
</a>
<a href="https://github.com/crawlab-team/crawlab/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement" target="_blank">
<img src="https://img.shields.io/github/issues/crawlab-team/crawlab/enhancement.svg?label=enhancements&color=cyan">
</a>
<a href="https://github.com/crawlab-team/crawlab/blob/master/LICENSE" target="_blank">
<img src="https://img.shields.io/github/license/crawlab-team/crawlab.svg">
</a>
</p>
[中文](https://github.com/crawlab-team/crawlab/blob/master/README-zh.md) | English
[Installation](#installation) | [Run](#run) | [Screenshot](#screenshot) | [Architecture](#architecture) | [Integration](#integration-with-other-frameworks) | [Compare](#comparison-with-other-frameworks) | [Community & Sponsorship](#community--sponsorship) | [CHANGELOG](https://github.com/crawlab-team/crawlab/blob/master/CHANGELOG.md) | [Disclaimer](https://github.com/crawlab-team/crawlab/blob/master/DISCLAIMER.md)
Golang-based distributed web crawler management platform, supporting various languages including Python, NodeJS, Go, Java, PHP and various web crawler frameworks including Scrapy, Puppeteer, Selenium.
[Demo](http://crawlab.cn/demo) | [Documentation](http://docs.crawlab.cn)
## Installation
Three methods:
1. [Docker](http://docs.crawlab.cn/Installation/Docker.html) (Recommended)
2. [Direct Deploy](http://docs.crawlab.cn/Installation/Direct.html) (Check Internal Kernel)
3. [Kubernetes](http://docs.crawlab.cn/Installation/Kubernetes.html) (Multi-Node Deployment)
### Pre-requisite (Docker)
- Docker 18.03+
- Redis 5.x+
- MongoDB 3.6+
- Docker Compose 1.24+ (optional but recommended)
### Pre-requisite (Direct Deploy)
- Go 1.12+
- Node 8.12+
- Redis 5.x+
- MongoDB 3.6+
## Quick Start
Please open the command line prompt and execute the command below. Make sure you have installed `docker-compose` in advance.
```bash
git clone https://github.com/crawlab-team/crawlab
cd crawlab
docker-compose up -d
```
Next, you can look into the `docker-compose.yml` (with detailed config params) and the [Documentation (Chinese)](http://docs.crawlab.cn) for further information.
## Run
### Docker
Please use `docker-compose` to one-click to start up. By doing so, you don't even have to configure MongoDB and Redis databases. Create a file named `docker-compose.yml` and input the code below.
```yaml
version: '3.3'
services:
master:
image: tikazyq/crawlab:latest
container_name: master
environment:
CRAWLAB_SERVER_MASTER: "Y"
CRAWLAB_MONGO_HOST: "mongo"
CRAWLAB_REDIS_ADDRESS: "redis"
ports:
- "8080:8080"
depends_on:
- mongo
- redis
mongo:
image: mongo:latest
restart: always
ports:
- "27017:27017"
redis:
image: redis:latest
restart: always
ports:
- "6379:6379"
```
Then execute the command below, and Crawlab Master Node + MongoDB + Redis will start up. Open the browser and enter `http://localhost:8080` to see the UI interface.
```bash
docker-compose up
```
For Docker Deployment details, please refer to [relevant documentation](https://tikazyq.github.io/crawlab-docs/Installation/Docker.html).
## Screenshot
#### Login
#### Home Page
#### Node List
#### Node Network
#### Spider List
#### Spider Overview
#### Spider Analytics
#### Spider File Edit
#### Task Log
#### Task Results
#### Cron Job
#### Language Installation
#### Dependency Installation
#### Notifications
<img src="http://static-docs.crawlab.cn/notification-mobile.jpeg" height="480px">
## Architecture
The architecture of Crawlab is consisted of the Master Node and multiple Worker Nodes, and Redis and MongoDB databases which are mainly for nodes communication and data storage.
The frontend app makes requests to the Master Node, which assigns tasks and deploys spiders through MongoDB and Redis. When a Worker Node receives a task, it begins to execute the crawling task, and stores the results to MongoDB. The architecture is much more concise compared with versions before `v0.3.0`. It has removed unnecessary Flower module which offers node monitoring services. They are now done by Redis.
### Master Node
The Master Node is the core of the Crawlab architecture. It is the center control system of Crawlab.
The Master Node offers below services:
1. Crawling Task Coordination;
2. Worker Node Management and Communication;
3. Spider Deployment;
4. Frontend and API Services;
5. Task Execution (one can regard the Master Node as a Worker Node)
The Master Node communicates with the frontend app, and send crawling tasks to Worker Nodes. In the mean time, the Master Node synchronizes (deploys) spiders to Worker Nodes, via Redis and MongoDB GridFS.
### Worker Node
The main functionality of the Worker Nodes is to execute crawling tasks and store results and logs, and communicate with the Master Node through Redis `PubSub`. By increasing the number of Worker Nodes, Crawlab can scale horizontally, and different crawling tasks can be assigned to different nodes to execute.
### MongoDB
MongoDB is the operational database of Crawlab. It stores data of nodes, spiders, tasks, schedules, etc. The MongoDB GridFS file system is the medium for the Master Node to store spider files and synchronize to the Worker Nodes.
### Redis
Redis is a very popular Key-Value database. It offers node communication services in Crawlab. For example, nodes will execute `HSET` to set their info into a hash list named `nodes` in Redis, and the Master Node will identify online nodes according to the hash list.
### Frontend
Frontend is a SPA based on
[Vue-Element-Admin](https://github.com/PanJiaChen/vue-element-admin). It has re-used many Element-UI components to support corresponding display.
## Integration with Other Frameworks
[Crawlab SDK](https://github.com/crawlab-t
基于Golang的分布式爬虫管理平台 （533个子文件） 
Spiderfile.163_news 554B Spiderfile.baidu 598B scrapy.cfg 280B scrapy.cfg 274B scrapy.cfg 272B scrapy.cfg 266B scrapy.cfg 266B scrapy.cfg 258B .air.conf 1KB crawlab.conf 490B config 300B font-awesome.min.css 30KB description 73B .env.development 153B Dockerfile 2KB Dockerfile 928B Dockerfile 464B Dockerfile 336B Dockerfile 164B .dockerignore 110B .dockerignore 18B .dockerignore 15B .editorconfig 228B .eslintignore 6B exclude 240B .gitattributes 251B .gitignore 1KB .gitignore 274B .gitignore 235B docs.go 158KB spider.go 49KB task.go 26KB main.go 18KB cron_test.go 17KB task.go 16KB spider.go 15KB git.go 14KB task.go 12KB spider.go 12KB parser.go 11KB spec_test.go 11KB config_spider.go 11KB parser_test.go 10KB system.go 9KB cron.go 9KB system.go 9KB user.go 8KB file.go 8KB schedule.go 8KB doc.go 8KB config_spider.go 7KB scrapy.go 7KB scrapy.go 7KB redis.go 6KB node_test.go 6KB node.go 6KB spider.go 6KB schedule.go 6KB project.go 6KB chain_test.go 6KB node.go 5KB task.go 5KB node.go 5KB spec.go 5KB node.go 5KB challenge.go 5KB schedule.go 4KB system.go 4KB log.go 4KB user.go 4KB base.go 4KB spider.go 4KB log.go 4KB spider_test.go 4KB schedule_test.go 4KB task_test.go 4KB project.go 3KB base.go 3KB mail.go 3KB action.go 3KB system_tasks.go 3KB user.go 3KB get_installed_deps.go 3KB variable.go 3KB file_test.go 3KB schedule.go 3KB mongo.go 3KB logger.go 3KB repo.go 3KB action.go 2KB clean.go 2KB stats.go 2KB install_dep.go 2KB uninstall_dep.go 2KB chain.go 2KB pubsub.go 2KB context.go 2KB token.go 2KB repos.go 2KB git.go 2KB共 533 条
- 1
- 2
- 3
- 4
- 5
- 6
资源评论
