원문정보
초록
영어
With the rapid development of network and information technology, there is a wealth of huge amounts of data on the internet. But it’s a major problem faced by the majority of researchers how to effectively filter out a particular subject or field of information from these data. In this paper, we try to builder a focused crawler based on vector space model and TF-IDF text correlation analysis. We take the seed URL as a collection entrance and fetch web pages from internet. Then analysis page information though technological like web content extraction, page link analysis technology and get the main content of one page. By the correlation analysis method based on VSM and TF-IDF text, we calculation the correlation between pages and the topics what have been defined, so we can achieve the purpose of the focus areas of the web.
목차
1. Introduction
2. The Process of Focused Crawler
3. Key Technologies
3.1. Web Page Pretreatment
3.2. Web Page Analysis
3.3. The TF - IDF Text Relevancy Analysis based on the VSM
4. Experiment
4.1. Evaluation Index
4.2. Experimental Process
4.3. Results Analysis
5. Conclusion
References