PHP Crawler project

home : quick installation : screenshots : download

Last version

Requirements

* MySQL 3.23.56 or higher
* PHP 4.0.6 or higher
* a webserver

This package

search/ - website frontend.
search.php - website search frontend.
index.html - simple redirect to search.php

search/crawler/ - crawler & libraries
_config.php - main CONFIG file. Please don't forget to edit it.
_crawler.php - crawler library
_db.php - wrapper for MySQL functions
_search.php - Fulltext search helper library
crawler.php - PHP Crawler
crawler.sql - crawler table structure (MySQL 3.23.56+ compatble)

search/contrib/ - search page template (pure html)

Installation

1. Remember. This is NOT production release. Use it on your own risk.
2. untar/gunzip package content to the website directory.
   Directory must be accessible through the web.
3. Open the file search/crawler/_config.php in an editor and edit the
   MySQL config lines and $CRAWL_ENTRY_POINT_URL to match your settings.
   DO NOT FORGET to put http:// prefix!!!
4. Create the necessary database tables. Make sure you have created
   the database ahead of time. You should know your mysql username and
   password as well.
   Get into the search/crawler directory and run the following command:
   

       mysql -u<username> -p<password> <databasename> < crawler.sql

   this will create phpcrawler_links table.
   
   You could also cut and paste the sql statements from crawler.sql into a
   package like phpMyAdmin or into your terminal window.
5. Run crawler via the web search/crawler/crawler.php or using command line

    
   	php -q crawler.php

   or

    
   	nohup php -q crawler.php > log 2>&1 &

6. Enjoy.

Contacts & support

http://astellar.com/support.html