Abstract
With the rapid development of the Internet, the web has become the main
platform for people to search and publish. How to quickly and accurately find the
information that users need in the massive data has become the demand of people,
and the web crawler is to satisfy The field of research that arises from this need.
In reality, our resources are limited. How to distinguish the different needs of each
of us in limited resources, some want to listen to songs, some want to watch
movies, and some are confidential documents needed for work, do not want others
to see, only I can watch a series of such problems by myself. He can protect
everyone's privacy and meet the needs of different people. Therefore, it is
necessary to study the content of different users.
Whether it is in-site information retrieval or specific web information
collection, it is inseparable from the core module of the full-text search engine
system - web crawler. This paper describes in detail a design and implementation
scheme, including page collector and page indexer. Basic workflow, data storage
structure, core algorithms and major technical difficulties. The system was
actually operated and the results were good. Finally, the areas for further
improvement were given.
This website is based on the B/S mode to design the crawler website, and
requires simple crawling operation and clear multi-user data. Develop a crawler
website to implement multi-user management, hierarchical data management, and
store data to a specified database. in. Identify duplicate web pages and
deduplication problems; determine topic relevance; capture data faster; store data;
and visualize data.
