PHP Crawler project

home : quick installation : screenshots : download

Last version


* MySQL 3.23.56 or higher
* PHP 4.0.6 or higher
* a webserver

This package

search/ - website frontend.
search.php - website search frontend.
index.html - simple redirect to search.php

search/crawler/ - crawler & libraries
_config.php - main CONFIG file. Please don't forget to edit it.
_crawler.php - crawler library
_db.php - wrapper for MySQL functions
_search.php - Fulltext search helper library
crawler.php - PHP Crawler
crawler.sql - crawler table structure (MySQL 3.23.56+ compatble)

search/contrib/ - search page template (pure html)


  1. Remember. This is NOT production release. Use it on your own risk.
  2. untar/gunzip package content to the website directory.
    Directory must be accessible through the web.
  3. Open the file search/crawler/_config.php in an editor and edit the
    MySQL config lines and $CRAWL_ENTRY_POINT_URL to match your settings.
    DO NOT FORGET to put http:// prefix!!!
  4. Create the necessary database tables. Make sure you have created
    the database ahead of time. You should know your mysql username and
    password as well.
    Get into the search/crawler directory and run the following command:
        mysql -u<username> -p<password> <databasename> < crawler.sql
    this will create phpcrawler_links table.

    You could also cut and paste the sql statements from crawler.sql into a
    package like phpMyAdmin or into your terminal window.
  5. Run crawler via the web search/crawler/crawler.php or using command line
    	php -q crawler.php
    	nohup php -q crawler.php > log 2>&1 &
  6. Enjoy.

Contacts & support