Tag: seo

Isolating search engine crawlers for special limited site access and addressing surgical SEO concerns

08/02/2015 Aaron Belchamber No Comments

Here’s how you can allow different search engine bot crawlers if you prefer to address them individually.  There might be some crazy reasons why which I try to explain below.  For some companies, it seems web developers often don’t dev to create new web assets, they dev to squeeze any remaining SEO juice from their old assets.  Always diminishing returns when you measure the opportunity cost of not spending time and resources moving forward, but instead looking back.

SetEnvIfNoCase User-Agent .*bing.* search_robot
SetEnvIfNoCase User-Agent .*google.* search_robot
SetEnvIfNoCase User-Agent .*yahoo.* search_robot
SetEnvIfNoCase User-Agent .*bot.* search_robot
SetEnvIfNoCase User-Agent .*ask.* search_robot

Order Deny,Allow
Deny from All
Allow from env=search_robot

Here are some more .htaccess SetEnvIf & SetEnvIfNoCase Examples from Apache’s website.

PHP Logic for detecting different search engine crawlers

You may want to redirect your content to be customized for different search engine bots to repair certain SEO issues that you may encounter.

Here is the Search Engine Directory of Spider Names

if(stristr($_SERVER['HTTP_USER_AGENT'], "googlebot")){

    // what to do -- change "googlebot" to other spiders in list
}

For certain instances where some how a server that was supposed to be locked down was inadvertently crawled by a search engine, you don’t want to open your entire site for all crawlers, here’s a way you can open your site to confirm your site ownership file by the crawler and perhaps disavow content if you don’t have anything better to do than massage your site’s SEO and squeeze every drop of juice from it.

<html>
<head>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
<title>Web Development is a finite resource</title></head>
<body>
Does quality content matter any more?
</body>
</html>

 


Advanced searches in search engines will save you time

04/26/2015 Aaron Belchamber No Comments

For those of us old fogies who’ve been doing web marketing for decades, it comes second nature to us how to use search engines and do advanced searches that can help us filter out noise and really get to the results we need while getting better research for businesses.  It saves us time and helps us become more effective at what we’re looking for.

Be more effective at finding what you need

Google and the other search engines have a hidden pseudo language available at the search prompt.  Well, at least you can do some advanced searches at the search bar, to filter your search terms for your needs.

“site:Belchamber.us” filters only results on a certain site (replace “Belchamber.us” with your own domain, of course.)

Another handy search is:  “your search -word” where “-word” means find “your search” but exclude any content with “word”.  You get the point.  I guess one of the examples would be homonyms that could confuse search results.

Another useful one is “safesearch:breast cancer” so you can avoid any adult-related results showing up that may show another type of result where a page mentions something about “breast” that’s not so…. clean.

Click here to view the best Google cheat sheet I’ve found with examples.