SharpSpider: A Continuous, Parallel and Distributed Spider
Palomino, Marco Search engines have become so indispensable that they rank second only to e-mail as the most popular online activity. To respond to queries in a timely fashion, search engines make use of large indices of word occurrences on Web pages to cross-reference websites to keywords. Such indices are maintained by spiders, a special kind of computer program that browses the Web autonomously. However, due to a variety of technological limitations, a sin...