Tag: search engines

Isolating search engine crawlers for special limited site access and addressing surgical SEO concerns

08/02/2015 Aaron Belchamber No Comments

Here’s how you can allow different search engine bot crawlers if you prefer to address them individually.  There might be some crazy reasons why which I try to explain below.  For some companies, it seems web developers often don’t dev to create new web assets, they dev to squeeze any remaining SEO juice from their old assets.  Always diminishing returns when you measure the opportunity cost of not spending time and resources moving forward, but instead looking back.

SetEnvIfNoCase User-Agent .*bing.* search_robot
SetEnvIfNoCase User-Agent .*google.* search_robot
SetEnvIfNoCase User-Agent .*yahoo.* search_robot
SetEnvIfNoCase User-Agent .*bot.* search_robot
SetEnvIfNoCase User-Agent .*ask.* search_robot

Order Deny,Allow
Deny from All
Allow from env=search_robot

Here are some more .htaccess SetEnvIf & SetEnvIfNoCase Examples from Apache’s website.

PHP Logic for detecting different search engine crawlers

You may want to redirect your content to be customized for different search engine bots to repair certain SEO issues that you may encounter.

Here is the Search Engine Directory of Spider Names

if(stristr($_SERVER['HTTP_USER_AGENT'], "googlebot")){

    // what to do -- change "googlebot" to other spiders in list
}

For certain instances where some how a server that was supposed to be locked down was inadvertently crawled by a search engine, you don’t want to open your entire site for all crawlers, here’s a way you can open your site to confirm your site ownership file by the crawler and perhaps disavow content if you don’t have anything better to do than massage your site’s SEO and squeeze every drop of juice from it.

<html>
<head>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
<title>Web Development is a finite resource</title></head>
<body>
Does quality content matter any more?
</body>
</html>