Sphider
a PHP spider and search engine
Each variation of Sphider has a variation in changes made. Updates to the multi-byte edition of Sphider appears by default. You may also view the Lite edition.
Sphider-MB
Sphider 5.5.0, release date 20 December 2023
Fix deprecation errors (RAP)
Fix "index decimals" causes pages not to index (RAP)
Fix search for decimals - previously rejected (RAP)
Add user choice of decimal separator character (RAP)
Improve indexing numbers by stripping thousands separator (RAP)
Added ability to read xml.gz sitemap files (RAP)
Change column name in settings to avoid reserved MySQL word (RAP)
Changed files:
admin/configset.php
admin/db_backup.php
admin/install.php
admin/spiderfuncs.php
admin/version_update.php
include/commonfuncs.php
include/detectmobilebrowser.php
include/searchfuncs.php
settings/conf.php
sql/tables.sql
Sphider 5.4.1, release date 09 November 2023
Fix fatal error when indexing new site from Index tab (RAP)
Code deprecation fix (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/spider.php
admin/version_update.php
include/searchfuncs.php
sql/tables.sql
Sphider 5.4.0, release date 15 October 2023
Added indexing support for docx and odt files (RAP)
Robots.txt strings now are case sensitive (RAP)
Allow rule now considered when processing robots.txt (RAP)
Integrate all common text files into Sphider (RAP)
Allow page language to override default common text language (RAP)
Added Japanese as a search language (RAP)
Fixed failure of image indexing to follow must-not-include rules (RAP)
Corrected commandline spider help instructions (RAP)
Fix 'index to' level being empty (RAP)
Introduce pause during indexing (RAP)
Fix Sphider not reading width, height, and alt image attributes (RAP)
Added '.jpeg', '.webp', and '.svg' to images indexed (RAP)
Removed '.tif' from images indexed (RAP)
Changed files:
admin/admin.php
admin/configset.php
admin/db_backup.php
admin/install.php
admin/messages.php
admin/spider.php
admin/spiderfuncs.php
admin/version_update.php
include/commonfuncs.php
settings/conf.php
sql/tables.sql
New directory:
include/common
New files:
include/common/am_common.txt
include/common/ar_common.txt
include/common/bg_common.txt
include/common/bn_common.txt
include/common/ca_common.txt
include/common/cs_common.txt
include/common/da_common.txt
include/common/de_common.txt
include/common/el_common.txt
include/common/en_common.txt
include/common/es_common.txt
include/common/et_common.txt
include/common/fa_common.txt
include/common/fi_common.txt
include/common/fr_common.txt
include/common/hi_common.txt
include/common/hr_common.txt
include/common/hu_common.txt
include/common/it_common.txt
include/common/ja_common.txt
include/common/lv_common.txt
include/common/nl_common.txt
include/common/no_common.txt
include/common/pl_common.txt
include/common/pt_common.txt
include/common/ro_common.txt
include/common/ru_common.txt
include/common/sk_common.txt
include/common/sl_common.txt
include/common/sq_common.txt
include/common/sr_common.txt
include/common/sv_common.txt
include/common/sw_common.txt
include/common/tr_common.txt
include/common/zh-cn_common.txt
include/common/zh-tw_common.txt
languages/ja-language.php
Deleted file:
include/common.txt
Sphider 5.3.0, release date 4 September 2023
Drop mb_string emulations, mb_string now required (RAP)
Permanent fix for mt_rand() deprecation without user intervention (RAP)
Updated jquery (RAP)
Code cleanup to PSR-2 standards (RAP)
Cleaned up standard search form when no categories present (RAP)
Changed files:
admin/admin.php
admin/auth.php
admin/configset.php
admin/db_backup.php
admin/install.php
admin/version_update.php
admin/rss_spider.php
admin/spider.php
admin/spiderfuncs.php
common_template/header.php
common_template/search_form.php
include/categoryfuncs.php
include/commonfuncs.php
include/searchfuncs.php
js_suggest/suggest.php
sql/tables.sql
Sphider 5.2.1, release date 26 August 2023
Fix the deprecation fix!!! (RAP)
Changed files:
admin/auth.php
admin/db_backup.php
admin/install.php
admin/rss_spider.php
admin/spider.php
admin/version_update.php
sql/tables.sql
Sphider 5.2.0, release date 22 August 2023
PHP 8.3 deprecation fix (rand() deprecated) (RAP)
Fixed broken "Did you mean" (RAP)
Changed files:
admin/admin.php
admin/auth.php
admin/db_backup.php
admin/install.php
admin/rss_spider.php
admin/spider.php
admin/version_update.php
include/commonfuncs.php
include/searchfuncs.php
sql/tables.sql
Sphider 5.1.0, release date 22 July 2023
Fix wrong argument count in indexAll function (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/spider.php
admin/version_update.php
sql/tables.sql
Sphider 5.0.0, release date 17 July 2022
New feature, option to create a links report (RAP)
Changed default User Agent string (RAP)
Changed naming convention for created sitemaps (RAP)
Changed files:
admin/admin.php
admin/db_backup.php
admin/install.php
admin/messages.php
admin/spider.php
admin/version_update.php
sql/tables.sql
New folder:
admin/reports
Sphider 4.2.1, release date Unreleased
Refined indexing filters (RAP)
More changes due to code deprecation (RAP)
Improved character set determination (RAP)
Add file size to list of spidering logs (RAP)
Add date, filesize to sitemap list (RAP)
Improve image identification (RAP)
Changed files:
admin/admin.php
admin/db_backup.php
admin/install.php
admin/spider.php
admin/spiderfuncs.php
admin/version_update.php
include/commonfuncs.php
include/stem_class.php
sql/tables.sql
Sphider 4.2.0, release date 1 July 2022
[ Still multibyte. MB extension removed for simplicity. ]
Deprecated code updates (RAP)
Fix fatal mysqli sql exception (RAP)
Fix potential fatal mysqli error (RAP)
Added code for a special hack, if needed (RAP)
Fix issue with some unicode characters not indexing (RAP)
Fix issue with phrase searching (RAP)
Fix issue with description length (RAP)
Force pdf-to-text output to be UTF-8 (RAP)
Protect database by truncating insanely long urls (RAP)
Changed files:
search.php
admin/admin.php
admin/db_backup.php
admin/db_main.php
admin/install.php
admin/spider.php
admin/spiderfuncs.php
admin/version_update.php
calendar/calendar_form.php
calendar/classes/tc_calendar.php
common_template/img_search_form.php
common_template/img_search_results.php
common_template/rss_search_form.php
common_template/rss_search_results.php
common_template/search_form.php
common_template/search_results.php
include/categoryfuncs.php
include/commonfuncs.php
include/searchfuncs.php
include/simple_html_dom.php
js_suggest/suggest.php
sql/tables.sql
Sphider 4.1.0-MB, release date 18 April 2022
Removed interrupted re-index completion code (RAP)
Improved sitemap indexing capabilities (RAP)
Removed section of dead code (RAP)
Changed files:
admin/admin.php
admin/db_backup.php
admin/install.php
admin/spider.php
admin/spiderfuncs.php
admin/version_update.php
sql/tables.sql
Sphider 4.0.2-MB., release date 27 February 2022
Updated jquery (RAP)
Fixed bug in search form due to PHP 8 (RAP)
Fixed RSS date picker due to PHP 8 (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/version_update.php
calendar/calendar.css
calendar/calendar.js
calendar/calendar_form.js
calendar/calendar_form.php
calendar/calendar_functions.php
calendar/classes/tc_calendar.php
calendar/classes/tc_date.php
common_template/header.php
common_template/search_form.php
sql/tables.sql
New files:
calendar/classes/tc_date_main.php
Sphider 4.0.1-MB, release date 28 July 2021
Minor spelling fix (RAP)
Cosmetic changes (RAP)
Path correction (RAP)
Fix RSS search calendar (RAP)
Update several functions (RAP)
Added Russian stemmer (RAP)
Corrected error in messages (RAP)
Changed files:
admin/admin.php
admin/configset.php
admin/db_backup.php
admin/install.php
admin/messages.php
admin/spider.php
admin/version_update.php
calender/calendar_form.php
calendar/calendar_functions.php
calendar/classes/tc_calendar.php
calendar/classes/tc_date.php
calendar/classes/tc_date_main.php
common_template/rss_search_form.php
include/detectmobilebrowser.php
include/simple_html_dom.php
include/stem_class.php
sql/tables.sql
Deleted files/folders:
calendar/classes/tc_date_main.php
calendar/css/ (and all contents)
Sphider 4.0.0-MB, release date 18 January 2021
Added ability for interrupted re-index completion (RAP)
Prevent warning on connection reset (RAP)
Improved backup & restore (RAP)
Remove illegal characters from crawled text (RAP)
Improved determination of character sets (RAP)
Improved filtering for indexing keywords (RAP)
Updated/removed obsolete code (RAP)
Corrected and updated language files (RAP)
Code cleanup (RAP)
Changed files:
search.php
admin/admin.php
admin/db_backup.php
admin/db_main.php
admin/install.php
admin/rss_spider.php
admin/spider.php
admin/spiderfuncs.php
admin/version_update.php
common_template/header.php
common_template/img_search_results.php
common_template/rss_search_results.php
common_template/search_results.php
include/commonfuncs.php
include/searchfuncs.php
languages/am-language.php
languages/ar-language.php
languages/bg-language.php
languages/cs-language.php
languages/fa-language.php
languages/hr-language.php
languages/lv-language.php
languages/pl-language.php
languages/ru-language.php
languages/sl-language.php
languages/sr-language.php
languages/zh_cn-language.php
languages/zh_tw-language.php
sql/tables.sql
New files:
settings/my.cnf
Sphider 3.6.1-MB, (Unreleased, merged into 4.0.0)
Sphider 3.6.0-MB, release date 26 December 2020
Found and fixed more PHP warnings (RAP)
Fixed Sphider not acknowleging/reporting html codes (RAP)
Fixed errors when robots.txt not found (RAP)
Fixed error when feed not valid (RAP)
Fixed bug leading to missing titles (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/messages.php
admin/rss_spider.php
admin/spider.php
admin/sphiderfuncs.php
admin/version_update.php
include/commonfuncs.php
sql/tables.sql
Sphider 3.5.3-MB, release date 19 December 2020
Fixed minor issue reading title tags (RAP)
Found and eliminated many PHP warnings (RAP)
Removed deprecated code (RAP)
Changed files:
search.php
admin/admin.php
admin/configset.php
admin/db_backup.php
admin/install.php
admin/spiderfuncs.php
admin/version_update.php
sql/tables.sql
Sphider 3.5.2-MB, release date 9 December 2020
Fixed, search with no results returned previous search (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/version_update.php
include/searchfuncs.php
sql/tables.sql
Sphider 3.5.1-MB, release date 8 December 2020
Found and corrected problem in unicode string function (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/version_update.php
include/commonfuncs.php
sql/tables.sql
Sphider 3.5.0-MB, release date 6 December 2020
Made a change missed when updating code changes in PHP 7.2+ (RAP)
Split the text search function to improve efficiency (RAP)
Changed files:
search.php
admin/db_backup.php
admin/install.php
admin/version_update.php
common_template/search_results.php
include/searchfuncs.php
sql/tables.sql
Sphider 3.4.5-MB, release date 30 November 2020
Removed deprecated code to avoid error in PHP 8 (RAP)
Removed truncate tables function (RAP)
Removed obsolete Optimize tables function (RAP)
Fixed parsing issue with robots.txt files (RAP)
Changed files:
search.php
admin/db_backup.php
admin/db_main.php
admin/install.php
admin/spiderfuncs.php
admin/version_update.php
js_suggest/suggest.php
sql/tables.sql
Sphider 3.4.4-MB, release date 7 November 2020
Fix critical error in the truncate tables routine (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/version_update.php
sql/tables.sql
Sphider 3.4.3-MB, release date 30 August 2020
Thank you to doelf_07 who pointed out the need for many of
these changes.
Correct code to adjust for PHP changes affecting count() (RAP)
Convert special character '&' to HTML entity "&' (RAP)
Fix missing tags in search results (RAP)
Add missing variable definitions (RAP)
Fix issue preventing image indexing in PHP versions > 7.2 (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/version_update.php
calendar/classes/tc_calendar.php
common_template/img_search_results.php
common_template/rss_search_results.php
common_tempplate/search_results.php
include/commonfuncs.php
include/searchfuncs.php
include/simple_html_dom.php
sql/tables.sql
Sphider 3.4.2-MB, release date 15 August 2020
Updated PHP 7.4 deprecated code in stem_class.php (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/version_update.php
include/stem_class.php
sql/tables.sql
Sphider 3.4.1-MB, release date 18 July 2020
jquery updated to 3.5.1 (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/version_update.php
common_template/header.php
sql/tables.sql
Sphider 3.4.0-MB, release date 7 December 2019
Fixed issue with Sphider mangling relative path beginning with "./" (RAP)
If the stemmer determines a word is not UTF-8, it converts it instead of
throwing an exception (RAP)
jquery updated to 3.4.1 (RAP))
Added the ability to create a sitemap from the Options screen (RAP)
Changed files:
README_FIRST
admin/admin.php
admin/db_backup.php
admin/install.php
admin/spiderfuncs.php
common_template/header.php
include/stem_class.php
sql/tables.sql
New directory/files
/admin/sitemaps
/admin/version_update.php
Sphider 3.3.0-MB, release date 7 September 2019
Added foreign key constraints to tables (RAP)
Removed redundent code from admin due to database changes (RAP)
Changed files:
admin/admin.php
admin/db_backup.php
admin/install.php
admin/spiderfuncs.php
sql/tables.sql
New files:
README_FIRST
admin/convert_db.php
admin/update_to_320.php
Removed files:
admin/version_update.php
Sphider 3.2.1-MB, release date 3 August 2019
Fixed issue of some compound characters causing index ommissions (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/spiderfuncs.php
admin/version_update.php
sql/tables.sql
Sphider 3.2.0-MB, release date 16 June 2019
Added ability to show stars in place of percentage for query scores (RAP)
Fixed broken pure wildcard search for RSS and image searches (RAP)
Added iframe support (RAP)
Fixed indexing issue when url appears in an iframe as a parameter (RAP)
Added check for non-standard use of wildcard (*) in robots.txt (RAP)
Extended list of common (Eng) words (still shy of internet standard) (RAP)
Added ability to set minimum score for search results (RAP)
Fixed bug causing some keyword-link relationships to be omitted (RAP)
Changed files:
admin/configset.php
admin/db_backup.php
admin/install.php
admin/spiderfuncs.php
admin/version_update.php
common_template/search_results.php
include/common.txt
include/searchfuncs.php
languages/fr-language.php
settings/conf.php
sql/tables.sql
templates/newspaper/m_search.css
templates/newspaper/search.css
search.php
New directory/files:
img/star0.png
img/star1.png
img/star2.png
img/star3.png
img/star4.png
img/star5.png
img/star6.png
img/star7.png
img/star8.png
img/star9.png
img/star10.png
Sphider 3.1.1-MB, release date 16 May 2019
Improved unicode performance by eliminating the need for the removeAccents
function, which proved to be inefficient and problematic (RAP)
Changed files:
admin/db_backup.php
admin/install.php
admin/spiderfuncs.php
admin/update_version_number.php
include/commonfuncs.php
include/searchfuncs.php
sql/tables.sql/tables
Sphider 3.1.0-MB, release date 13 May 2019
Merge the capabilities of Sphider 2.4.x with Sphider 3.0.0-MB,
eliminating the mbstring extension requirement (RAP)
Eliminated using multibyte functions places they were not necessary (RAP)
Eliminated issue with too-short words were still being indexed (RAP)
Eliminated storing of duplicate words because of leading or trailing
apostrophes or hyphens(RAP)
Added removal of left and right double quotes from word indexing (RAP)
Corrected a problem of inconsistent Sphider module encoding (RAP)
Eliminated issue of utf-8 text possibly being misidentified as Windows-1252
or Latin-1 (RAP)
Improved unicode performance of the removeAccents() function (RAP)
Eliminated use of http-equiv to define charset in admin and search (RAP)
Improved indexing of meta descriptions (RAP)
Fixed problem with search page causing an error (RAP)
Improved highlighting of search term in the url (RAP)
Clarified setting explanation on the settings tab (RAP)
Changed files (in relation to 3.0.0-MB):
admin/admin.php
admin/db_backup.php
admin/configset.php
admin/install.php
admin/rss_spider.php
admin/spider.php
admin/spiderfuncs.php
common_template/header.php
common_template/search_results.php
include/commonfuncs.php
include/searchfuncs.php
include/stem_class.php
js_suggest/suggest.php
language/am-language.php
language/cs-language.php
language/da-language.php
language/de-language.php
language/es-language.php
language/et-language.php
language/fi-language.php
language/fr-language.php
language/hu-language.php
language/nl-language.php
language/no-language.php
language/pt-language.php
language/sk-language.php
language/sq-language.php
language/sv-language.php
language/tr-language.php
sql/tables.sql/tables
search.php
New file:
admin/version_update.php
Deleted file:
admin/update_version_number.php
Sphider 3.0.0-MB, release date 29 April 2019
Sphider 3-MB is a multi-byte string enabled fork of Sphider 2.4.x
What has NOT changed (directories and files)
/calendar/*
/languages/*
/templates/*
/admin/
admin.css
dbmain.js
ext.txt
index.html
common_template/
footer.php
include/
common.txt
detectmobilebrowser.php
simple_html_dom.php
stem_class.php
js_suggest/;
autocomplete.css
autocomplete.js
settings/
database.php
install.txt
sphider-logo.png
The database structure is unchanged from Sphider 2.4.x,
although you may want to update the version number. A
utility is provided. (update_version_number.php)