Search Engine Spider Detection With PHP

29th September 2008

Part of any search engine optimisation strategy should always be that the user and the search engine see the same thing. If you start delivering different content you will either end up not performing or just getting outright banned. However, there are certain circumstances where you will want to detect the presence of a search engine spider. For example, lets say that you had a link to a section of your site, and you wanted to add a counter to it that registered an action every time a user clicked on the link. One way of doing this would be to add a parameter to the URL of the link, if the parameter is present then it is a user going through the site and so the action will be registered. You don't want to register every time a search engine bot spiders the site so using the following function will allow you to turn off this parameter for these spiders.

  1. function spiderDetect() {
  2. $agentArray = array("ArchitextSpider", "Googlebot", "TeomaAgent",
  3. "Zyborg", "Gulliver", "Architext spider", "FAST-WebCrawler",
  4. "Slurp", "Ask Jeeves", "ia_archiver", "Scooter", "Mercator",
  5. "[email protected]", "Crawler", "InfoSeek Sidewinder",
  6. "almaden.ibm.com", "appie 1.1", "augurfind", "baiduspider",
  7. "bannana_bot", "bdcindexer", "docomo", "frooglebot", "geobot",
  8. "henrythemiragorobot", "sidewinder", "lachesis", "moget/1.0",
  9. "nationaldirectory-webspider", "naverrobot", "ncsa beta",
  10. "netresearchserver", "ng/1.0", "osis-project", "polybot",
  11. "pompos", "seventwentyfour", "steeler/1.3", "szukacz",
  12. "teoma", "turnitinbot", "vagabondo", "zao/0", "zyborg/1.0",
  13. "Lycos_Spider_(T-Rex)", "Lycos_Spider_Beta2(T-Rex)",
  14. "Fluffy the Spider", "Ultraseek", "MantraAgent","Moget",
  15. "T-H-U-N-D-E-R-S-T-O-N-E", "MuscatFerret", "VoilaBot",
  16. "Sleek Spider", "KIT_Fireball", "WISEnut", "WebCrawler",
  17. "asterias2.0", "suchtop-bot", "YahooSeeker", "ai_archiver",
  18. "Jetbot"
  19. );
  20.  
  21. $theAgent = $_SERVER["HTTP_USER_AGENT"];
  22.  
  23. foreach ($agentArray as $anAgent) {
  24. if (stripos($theAgent, $anAgent) !== false) {
  25. return true;
  26. };
  27. };
  28. return false;
  29. }

The function works by finding the current user agent string of the visitor and the comparing it to the list of user agents in an array. If the user agent is found then true is returned, otherwise the return value is false.

With this function present you can now include an if statement to see if the user agent is a search engine spider or not.

  1. if ( spiderDetect() ) {
  2. // do something for spiders
  3. }else{
  4. // do something for users
  5. }

Please be careful with this function. If you server different content to users and search engine spiders you will more than likely get banned for your efforts.

Also, this might be an incomplete list of search engine spider user agents, if you know any more then please write a comment and I will add them onto this list.

Update: Reworked part of the finding code to make it more efficient.

Comments

Permalink
Hi, in the loop you use: $agentArray[$f] but I guess you meant: $agentArray[$i] Great function, thanks for publishing!

Mark (Fri, 10/24/2008 - 23:02)

Permalink
Nice spot! I've corrected that now :)

philipnorton42 (Sun, 10/26/2008 - 21:37)

Permalink

Thanks for your marvelous posting! I truly enjoyed reading it, you could be a great author. I will remember to bookmark your blog and will often come back at some point. I want to encourage one to continue your great job, have a nice afternoon!

joshedmonds852344 (Wed, 05/30/2018 - 13:30)

Permalink

Thanks for your marvelous posting! I truly enjoyed reading it, you could be a great author. I will remember to bookmark your blog and will often come back at some point. I want to encourage one to continue your great job, have a nice afternoon!

joshedmonds852344 (Wed, 05/30/2018 - 13:30)

Add new comment

The content of this field is kept private and will not be shown publicly.