The below URLS are bots.

The below URLS are bots. We tried about three bot blocking services, and none of them worked. We are not being bothered by any other bots.

Does anybody have a solution? We are looking to hire a freelancer to stop this annoying problem. / referral / referral / referral / referral / referral / referral

#botblocking   #hirebotblocker  

6 replies on “The below URLS are bots.”

  1. Karl Gerber​​​​​ I have a dozen sites that I have blocked those (exact same) bots w/.htaccess > 2 different ways and it seems to work!

    Having just-today discovered about the gazillion (new-ish) ghost (spam) referrals, I filtered-out the ghosts from GA and I’ll see, over time, how that goes! (supposed to work well)

    But, .htaccess manipulation proves to block the actual bots, themselves.

    As you may know, you cannot block the ghost referrals, only hide them, but they do not hurt SEO, regardless. So, “filter hostname” should make the ghosts disappear. (I’ve been studying this for months)

  2. What you’re seeing is “referral spam”. They are “ghost hits” sent to GA (or whatever analysis you use) via scrupulous scripts. The majority of those bots are not even visiting your site. The scripts are just/only ‘spoofing’ the analytics to get the victim to visit the sites they ‘say’ they’re coming from. So, do not visit them.

    There is NO cure for this (yet). Google is aware, but haven’t developed a fix. The only way to get rid of the spoofs is to set a filter in your analytics.

    To see what’s what, follow these instructions for Google Analytics…

    In left (narrow) column: Acquisition > All Traffic > Referrals

    Right larger column: Secondary Dimension > type “hostname” (and go there)

    If the host name does not match yours, THAT is a ‘ghost’ and never went to your site ever. It was sent to GA by a malicious script.

    The ‘source’ on left side is what you want to block in .htaccess of those that match your hostname on the Right.

    Just copy & paste this into your .htaccess (and add to it at will)


    # Block fake traffic

    # Blocks all http and https referrals and all subdomains from “

    RewriteEngine on

    Options +FollowSymlinks

    RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*semalt\.com [NC,OR]

    RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*free-share-buttons\.com\ [NC,OR]

    RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*best-seo-solution\.com\ [NC,OR]

    # 5/22/2015 new bleeping bots that just came in (eg. of how I keep track)

    RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*buttons-for-your-website\.com\ [NC,OR]

    RewriteCond %{HTTP_REFERER} ^https?://([^.]+\.)*buttons-for-website\.com\ [NC]

    RewriteRule ^(.*)$ [L]


    (just, make sure everything before the last one has [NC,OR] at the end)

    You’ll see other ways of typing this on the net, but THIS is the most ‘correct way’, I have found.


    Typing up the ‘filter tutorial’ now (brb)

    (visitors, please click +1 if you find this helpful)

  3. To filter out the ‘ghost spam’ bots…

    • Admin > View > Filters > +NEW FILTER (button) >

    • Filter Name = Hostname Filter

    • Filter type = Custom

    • Tick “Include” circle

    • Filter Field = Hostname

    • Filter Pattern = yoursite\.com|www\.yoursite\.com|translate\.googleusercontent\.com|webcache\.googleusercontent\.com

    • Click Filter Verification to see how the results will be (bad on left, clear on right)

    • Save

    This method will only record & (from now-on) and the spoof (ghost) bots will not show, anymore.

    All the corrupt data from before you do this is permanent and nothing you can (really) do.

    I just did this yesterday. So, I await the final results and hopefully, there will be no more corrupted data. [edit: There hasn’t been (yAy)]

    However… It is important to block the ‘actual’ badbots in .htaccess (you can find them by following my prior post)

    The 2nd method to block them is as follows… (for .htaccess) This can be included with 1st method as well.

    copy exactly this… (and add-to-it at-will)

    # having the pound-sign before this text is like a ‘comment’ and will not execute (good for keeping notes)

    # I put the date (usually) before all my code for reference, later

    # 5/21/2015 from that Thrasher guy

    SetEnvIfNoCase Referer spammer=yes

    SetEnvIfNoCase Referer spammer=yes

    SetEnvIfNoCase Referer spammer=yes

    SetEnvIfNoCase Referer spammer=yes

    # 5/22/2015 new effing bots that just came in (eg. of how I keep track)

    SetEnvIfNoCase Referer spammer=yes

    SetEnvIfNoCase Referer spammer=yes

    SetEnvIfNoCase Referer spammer=yes

    SetEnvIfNoCase Referer spammer=yes

    SetEnvIfNoCase Referer spammer=yes

    SetEnvIfNoCase Referer spammer=yes

    Order allow,deny

    Allow from all

    Deny from env=spammer

    Just, be sure to not block ‘good’ bots, unless you don’t care about traffic.

    [Edit: If a bot is questionable and only leaves 1 hit with 100% bounce, block the bad bully!]

    Another thing I found (haven’t tried yet) is to change the UA-XXXXXXXX-1 to something like UA-XXXXXXXX-32 in all your html pages but…

    I haven’t really researched this yet. So, don’t do it unless you’re sure!

    See… The evil-people have taken your unique code and send fake hits to your analysis. The analysis people have to make a way to ‘validate’ the incoming data first and haven’t! (dern google) This world is becoming very bad for the Internet! ugh!

    Please let me know how it goes. I’d like to know!

    And, if anyone else stumbles into this thread, please add your personal experience(s) so we can all get a better understanding of this epidemic. Thanks!

    Visitors, please +1 each post you find ‘worthy’ (thx)

  4. Steve, you should do some kind of a formal post. You seem to be THE MAN.  All of these bot blocking companies don’t seem to have a clue. I am gonna try all of this now!!!

  5. Thank you very much! 🙂

    The spoof-spam-ghost thing just-started within the last two months. Some Russian mad-scientist wrote a script that lampoons any analytics-survey. It’s sssO new, it’s creepy, but thanks!

Comments are closed.