======================= fantomas spyFetcher(TM) ======================= ver. 2.01.01 Date of release: 2006-01-31 ************************************ * ONLINE HELP IS AVAILABLE AFTER * * SUCCESSFULL INSTALLATION BY * * CLICKING ON THE * * FANTOMAS LOGO! * ************************************ SYSTEM REQUIREMENTS INSTALLATION - UNIX UNINSTALLING THE PROGRAM WORKING WITH fantomas spyFetcher(TM) CONFIGURATION OF CRON JOBS ERROR HANDLING KNOWN ISSUES UPDATES + PROGRAM CHRONOLOGY CONTACT + SUPPORT ====================================================================== SYSTEM REQUIREMENTS ------------------- Language ------- Perl 5 Module ------ Perl module LWP::UserAgent UNIX ---- The Unix system requires an installed web server. Execution of CGI scripts must be enabled. A directory for execution of CGI scripts must be existent. Usually, this will be directoy /cgi-bin/. Tested under: SuSE LINUX with Apache Red Hat Linux with Apache BSDI Unix with Apache Browser ------- Script is called and executed via web browser. You will currently achieve best results under MS Internet Explorer 5+. Netscape 4.7 may require adjustment of font size. Tested under: IE6+, Netscape 4.7, Netscape 7.0, Opera 6.0 ====================================================================== INSTALLATION ------------ The following files are included: spyfetcher.cgi --- (program script) locs-sfe_en.txt --- (language file) fantomas.gif --- (logo/graphics file) sfehelp_en.txt --- (documentation in TXT format) <--- THIS FILE YOU ARE READING! fa_license-e.txt --- (License Agreement And Terms Of Usage - PLEASE READ!) ----------------------------------- ADJUSTMENTS IN FILE "spyfetcher.cgi" (please edit in ASCII or plain text editor like Notepad etc.) ----------------------------------- UNIX ---- * Please check path to location of Perl. The default path in the script is "/usr/bin/perl". If you don't know this path, you can check it out under telnet by entering Unix command "whereis perl". You may have to adjust the first line in the script "spyfetcher- e.cgi" accordingly. * The variables in the script "spyfetcher.cgi": "$cgi_dir", "$stats_dir", "$robot_file", "$log_file", "$sendmail", "$from_mail", "$to_mail", "$subject", $cloak_for_google, "$user and "$pw" may optionally be adjusted to your requirements. A comprehensive description of these variables can be found below in chapter "WORKING WITH fantomas spyFetcher(TM)". * The script "spyfetcher.cgi" and the file locs-sfe_en.txt must be copied into the Unix server's CGI directory. * The CGI directory must be endowed with the following permissions: "chmod 755" [drwxr-xr-x] * Next, create the directory defined as variable "$stats_dir" with the following permissions: "chmod 777" [drwxrwxrwx](Default name is "stats".) * Finally, create the following directories with the permissions "chmod 755" [drwxr-xr-x] BELOW your MAIN directory: helpfiles/ graphics/ Uploading the Online Help File ------------------------------- * Copy the following file into the directory "helpfiles/": sfehelp_en.txt Uploading the Logo/Graphics File ------------------------------- * Finally, copy the following file into the directory "graphics/": - fantomas.gif * When uploading via FTP, make sure to transfer ALL files in ASCII mode. EXCEPTION: the graphics file "fantomas.gif" which must be transferred in BINARY or AUTOMATIC mode. * Required file permissions: spyfetcher.cgi: "chmod 755" [-rwxr-xr-x] locs-sfe_en.txt: "chmod 444" [-r--r--r--] fantomas.gif: "chmod 444" [-r--r--r--] sfehelp_en.txt: "chmod 444" [-r--r--r--] ====================================================================== UNINSTALLING THE PROGRAM ------------------------ For complete uninstall, delete the following: spyfetcher.cgi locs-sfe_en.txt fantomas.gif sfehelp_en.txt Also, delete the following directories including all content: stats (or whatever directory you defined under "$stats_dir") helpfiles graphics ====================================================================== WORKING WITH fantomas spyFetcher(TM) ------------------------------------ Program Description ------------------- The fantomas spyFetcher(TM) is a script which allows you to get the latest fantomas spiderSpy(TM) botBase as a packed archive in .ZIP format. The botBase will be unpacked and saved on your server in the directory defined under "$stats_dir" with the file name defined under "$robot_file". --------------------------------- Customization of script variables --------------------------------- The following variables may optionally be customized in script "spyfetcher.cgi": * $stats_dir This variable defines the directory where the spider robots list file shall reside as absolute path in this format: Example: "/usr/www/htdocs/yourdomain/cgi-bin/stats" * $robot_file This variable defines the file name of the spider robots list file. Default file name is "spiderspy.txt". * $log_file This variable defines the file name of the transfer log file. Default file name is "transfer.log". Email Error Message ------------------- If the script is executed in batch mode via cron job, an email error message will be generated if the transfer of the fantomas spiderSpy(TM) botBase fails. For this email functionality you will need to specify the following variables: * $sendmail This variable defines the command call for the mail program. Default configuration is "/usr/lib/sendmail -t -n -oi". If you don't know this path, you can check it out under telnet by entering Unix command "whereis sendmail". Else, please inquire with your system administrator. * $from_mail This variable defines the email error message sender's address. * $to_mail This variable defines where you want the email error message to be sent. * $subject This variable defines the email error message's subject line. * $cloak_for_google If you want to cloak for Google, please set "$cloak_for_google = 1" and the Google spider entries in the fantomas spiderSpy(TM) botBase will be activated. User Authentication ------------------- After the sign up for the spiderSpy service, you received your user id and password for downloading the fantomas spiderSpy(TM) botBase. * $user This variable defines your user id (case sensitive). * $pw This variable defines your password (case sensitive). ******************** VERY IMPORTANT! ********************* If the variables "$user" and "$pw" are not correct, the download will fail because access is forbidden. SO PLEASE MAKE SURE TO SPECIFY YOUR ID AND PW EXACTLY AS ISSUED DURING SIGNUP! ******************** VERY IMPORTANT! ********************* ONLINE MODE ----------- * Script is activated by entering the appropriate URL into web browser's location/address field, e.g. "http://www.yourdomain.com/cgi-bin/spyfetcher.cgi". To start the download of the current version of the fantomas spiderSpy(TM) botBase, click button "Submit!". If the botBase is saved on your server, the next HTML template will display the message: "Transfer of fantomas spiderSpy(TM) botBase successful!" BATCH MODE ---------- You can manage the transfers of fantomas spiderSpy(TM) botBase automatically by defining a cron job. ====================================================================== CONFIGURATION OF CRON JOBS -------------------------- Cron is a mechanism for planning and scheduling batch jobs. The daemon "crond" is started automatically on system boot up. It runs one check per minute to see if there are any jobs to execute. The list of jobs to execute is created by the program "crontab". The following commands work from the assumption that you are either logged in by Telnet or locally on your Unix system. Entering the command "crontab -l" will display a list of current entries. By default, only entries owned by the logged in User will be displayed. Existing lists can be removed/deleted with command "crontab -r". To create a new list, it is recommended to read the entries from a file using command "crontab filename". The following examples will show you the format of this file. The file itself is created with an ASCII text editor. Example: 0 12 * * * /usr/www/htdocs/yourdomain/cgi-bin/spyfetcher.cgi start This entry consists of six parameters. The first five parameters define the time schedule, whereas the sixth parameter contains the command for executing the job. In our example above, this command consists of: - the full path and file name of the script - an argument Parameters defining the time schedule are: minute(0-59) hour(0-23) day of month(1-31) month(1-12) day of week(0-6) 0 = Sun Hence, the above sample entries: 0 12 * * * can be translated as: If Minute = 0 and Hour = 12, the script will be executed. Because the last three scheduling parms are defined by wildcard character "*", the job will be executed every day. Scheduling Week Days -------------------- If you wish to run the script on Mondays only, the following entry will do the trick: 0 12 * * 1 /usr/www/htdocs/yourdomain/cgi-bin/spyfetcher.cgi start Scheduling Turn of Month ------------------------ You can schedule the turn of the month in this manner: 0 0 1 * * /usr/www/htdocs/yourdomain/cgi-bin/spyfetcher.cgi start To Summarize ------------ Create a text file (e.g. "crontab.txt") and write the appropriate command on one single line. We recommend downloading the fantomas spiderSpy(TM) botBase once per day. The following syntax will generate (as explained above) a cron job which will run once a day: 0 12 * * * /usr/www/htdocs/yourdomain/cgi-bin/spyfetcher.cgi start IMPORTANT ========= Please modify the TIME OF DAY argument specified for your cron job to prevent all downloads happening at the same time - with hundreds of subscribers, this could incur a server overload on our system. Prevention of abuse: Per day, a maximum of six downloads of the botBase are permitted, beyond that the downloading IP will be blocked by our system. Enter the absolute path for the script as valid for *YOUR* specific system configuration. The argument to use is "start", as shown in our example above. Next, the command "crontab crontab.txt" will transmit this file to crontab. IMPORTANT ========= If crontab has been configured for prior jobs already, you must include them in the new file "crontab.txt" (example), as the command "crontab crontab.txt" will override all previous cron jobs owned by the specific user calling crontab! For further online explanations under Unix, you can choose one of the following commands: man crontab man 5 crontab man cron ====================================================================== ERROR HANDLING -------------- This section covers individual error messages. Stats directory --------------- "Stats directory ... does not exist!" Please create stats directory or adjust the directory name under variable "$stats_dir". Download error -------------- "Download of fantomas spiderSpy(TM) botBase failed!" Solution: Please check your user ID and Password. * In directory stats (defined under $stats_dir) new files could not created. Solution: Please check permissions of directory: "chmod 777" i.e. [drwxrwxrwx] Unzip error ----------- "Unzip of fantomas spiderSpy(TM) botBase failed!" Possible issues: * Call of gunzip is not functional. Solution: Please check your system's gunzip functionality. Change mode error ----------------- "Change mode of fantomas spiderSpy(TM) botBase failed!" Possible issues: * Call of chmod is not functional. Solution: Please check your system's chmod functionality. ====================================================================== UPDATES + PROGRAM CHRONOLOGY ============================ 2006-01-31: Version 02.01.01 CHANGES Transfer of botBase via spyfetcher.cgi to use LWP instead of wget for enhanced compliance with current provider practice. 2002-12-20: Version 01.02.01 NEW FUNCTIONS Localization supported (language files). 2002-03-30: Version 01.01.02-e NEW FUNCTIONS Cloaking for Google supported Google spider entries in the fantomas spiderSpy(TM)botBase will be optionally activated during download. 2001-09-06: official release of version 1.01.01-e ====================================================================== CONTACT + SUPPORT ================= Please send email to: techsupport@fantomaster.com Corporate contact info: For an overview of our business hours in relation to your specific time zone, please see: < http://fantomaster.com/index2.html#hours > ###################################################################### # (c) Copyright 2006 by fantomaster.com # # All rights reserved. # # Copying, modification or distribution requires permission # # in writing by copyright holder. # # fantomas spyFetcher(TM) is the protected trade mark of # # fantomaster.com GmbH. # # URL: < http://fantomaster.com > # # ------------------------------------------------------------------ # # OEM Licensing: # ######################################################################