web crawler design
Spider - PHP Website Crawler
This PHP script allows you to search for files in ANY website. All you have to do is specify what website you want to search, the kind of files you are looking for then click Crawl to watch spider handle the rest. It comes packed with more than 150 file types pre-installed and multiple ways to view results, Spider is the tool at heart for people building crawlers and web scrappers. Features Over 150 file tpyes included Accepts custom file t...
PHP Search Engine & Crawler
Advanced PHP Search engine, with Full-Text search queries in BOOLEAN mode, and cURL page crawler. Update 1.5 [1/2/2013] Fixed a bug where in some cases the “Add URL” function from the Admin Panel wouldn’t work. MySQL Full-Text search engine PHP Crawler (only for domains and subdomains) Various filters (including “exact match”) Option to order the results by relevancy Adsense ready, 3 ad units Top 15 most searched keywords Cross Browser...