Stop or Block Referrer Spam

Facebooktwittergoogle_plusredditpinterestlinkedintumblr

Stop or Block Referrer Spam

Lately, many website owners and online space operators have been facing an onslaught of referrer spam. For the newly initiated, referrer spam is pseudo traffic from bots, or automatic scripts, that generate a fake or inauthentic referral link. If the referrer spam is “successful”, the pseudo traffic will make the spam domain appear in your website’s analytics, hopefully encouraging you to visit.

Referral Spam – A Big Problem

Many website operators and online space owners dislike having their site analytics contaminated with fake, false or corrupt data. Besides affecting the purity of site analytics and working to better understand the user experience, dealing with referral spam can take a lot of time. Referrer spam has the potential to cause lasting harm to your Google SERP rankings, potentially meaning a damaging loss of revenue.

Normally, an HTTP or standard referrer is a header field that is sent as discrete data when a user’s browser navigates from one page to another page. When the user clicks on a link, the browser forwards the desired address to that page’s server, requesting that it hold and produce the desired web address. Part of that request includes the address referrer, which under normal circumstances identifies the last page the user visited before clicking on the relevant link.

When all of this works properly, website owners and online space operators can see the origin of their visiting traffic, helping them to gain a deeper understanding and appreciation of which external websites are passing along user traffic. Spam referrers cheat this transaction by providing a fake address to list as the previous originating location of the user’s browser so that it now promotes an intended, usually commercial, address.

Furthermore, while a visit to a spam referral link mandates a minimum amount of lost productivity, it may open a user’s computer to much more pernicious damage, including the unheralded installation of malicious software or increased vulnerability to digital viruses.

Semalt, King of Referral Spam

While tens of thousands of automated scripts and bots are now crawling the web and depositing spam referral links, one company in particular is a notable offender. Known as Semalt, the company bills its products as a suite of advanced analytical tools. Experts, however, agree that their browser engine is designed to deliberately mimic human, as opposed to automated script, traffic and thus hoodwink Google and other site ranking search engines into wrongly categorizing it.

The company’s traffic is of such a high volume that millions of website users and online space operators have had to implement special code specifically to deal with Semalt’s intrusions and corruption of their site analytics data. Some security experts even believe that Semalt is now illicitly using a network of hundreds of thousands of maliciously infected computers to drown websites in a tide of spam referrer links.

Why Do People Use Referrer Spam?

Originally, it was much easier to deceive certain automated ranking algorithms such as the one used by Google to return results for search queries. When a referrer spams a fake referral address, this can theoretically help boost a web page’s rankings because it falsely attributes a higher than accurate amount of visitor traffic to the intended online space. Modern algorithms are deployed to detect precisely these kinds of operations, but many scripts and spam purveyors still refuse to abate their noxious practices.

Otherwise, the purpose of spam referral is primarily to intrigue and hopefully lure inquisitive site owners and online space managers into visiting the supposed referral link in the pursuit of knowledge about how and why that exterior site is purporting to be sending significant amounts of traffic to the target website.

Filters versus a Block

Some site owners and web space operators know that filters can be used in Google Analytics or other site analysis software. Offending domains and spam referrals can be added, one at a time, to filters. But this process can be tedious and lengthy, and does nothing to stop referral spam from constantly evolving to impersonate other domains.

Furthermore, by filtering out offending referrals, a site owner or space operator may encounter data sampling problems when performing analytics. If the size of a data sample is altered, a website or commercial online space’s conversion rate could be affected, meaning that the revenue report could change as well. Google Analytics is a powerful tool when used correctly, and for best results it is recommended that all data be used for analysis precisely in order to omit sampling errors.

Stop Referral Spam at the Source

Instead of applying complicated filters and risking the contamination of data samples used in site analytics, a savvy site owner will work to stop spam before it even has a chance to register on your site as a valid referrer. The simplest and easiest way to do this is add code to your .htaccess file.

## SITE REFERRER BANNING
RewriteCond %{HTTP_REFERER} spamreferrer.com [NC,OR]
RewriteRule .* – [F]

Deflecting Spam Referral

Another popular technique for blocking spam referral bots before they can be registered as referrals is to use a Deflector, which returns the traffic back whence it came. Some web site owners and online space operators prefer this method and find it more powerful and flexible than other options.

First, create a text file named ‘deflector.map’ or something similar:

#
## deflector.map
##
##referer –> redirect target
http://spamreferral.com http://spamreferral.com

Then the .htaccess file is modified accordingly:

RewriteMap deflector txt:/path/to/deflector.map
RewriteCond %{HTTP_REFERER} !=””
RewriteCond ${deflector:%{HTTP_REFERER}} =-
RewriteRule ^ %{HTTP_REFERER} [R,L]

Blacklists

Less effective in some conditions but far easier to implement and very rapid to deploy, blacklists can significantly reduce referral spam. By using a list of banned referrers in combination with keywords commonly used by spam referral sites, blacklists can be assembled or purchased from online vendors.

WordPress plug-in

For site owners and online space operators who may feel overwhelmed by modifying and tinkering with important system files like .htaccess can now use a plug-in for their WordPress site.

Currently, the plug-in for WordPress only blocks a single spam referrer – Semalt – but there are plans in the works to allow users to add more sites to block in the future.

.htaccess Best Practices

While updating and modifying an .htaccess file may be an efficient and effective solution for small sites, some online space owners and operators find that maintaining and updating multiple .htaccess files to be too overwhelming. Luckily, it is possible to re-organize your websites so that all of your pages use an umbrella or master .htaccess file. Once it is updated, it will begin providing immediate coverage for all of the beneficiary pages.

6 Responses to Stop or Block Referrer Spam

  1. Thank you for informing about that spam Donna.

  2. Ruben says:

    Great information!
    Keep up the good work!

  3. Interesting post Donna.
    Another reason for cloaking a referrer is to disguise arbitrage traffic. Many of the tier one advertising providers are very strict about arbitrage and this is an attempt to mimic that the traffic is coming from Google/Yahoo/Facebook etc when in fact it’s largely mechanical.

    Really happy that you’ve raised this issue though!

  4. […] have been really annoyed by referrer spam messing up my stats and did a post  explaining how to keep it off of your sites but parking stats […]

  5. […] Next post will show ways to stop referrer spam […]

  6. Carter says:

    I am curious if this will block real traffic coming from disqus links and fb and twitter?

Leave a Reply