TomasWhen talking with clients about how WP Engine counts visitors, we often get asked questions like this: “who are these ‘bots’ you speak of?” and “why do their visits not show up in other analytics programs, like Google Analytics?”

The most common reason this subject comes up is that a client’s Google Analytics will show less traffic than their user dashboard does. This happens because Google Analytics does not count non-human traffic (bots) as unique visits to a website, but WP Engine shows all traffic human and non.

It’s good to know what these bots, or “crawlers,” are, since they can represent 50% or more of the actual traffic to your website—traffic normally completely hidden by Google Analytics and other JavaScript-based web analytics. To understand why, we will go over a few things: what bots / crawlers are, what they do, and why we want them. Lastly, we will explain why they don’t show up on your (other) analytics, and why WP Engine chooses choose to show them.

Growing traffic is hard.

I know. Over the course of my time helping companies in the online marketing space, there has always been one constant goal—to achieve more traffic. Have any of you site owners, marketers, or technical folks ever come across anyone who says, “I’d like less traffic please”? My guess is no.

One of my friends checks his traffic like a compulsion. I’ll be sitting with him at lunch as he pulls out his phone, just to tell me about his 3% increase this week. So I started playing the table phone stack game, which he of course loses constantly. Hey, nothing beats a free lunch!

But where does all this traffic come from?

Did you know that you have a private army that helps you grow your traffic every day? They go by many names online: Robots, Crawlers, Spiders… (Wait, is this starting to sound like a horror movie?) But these non-human helpers are not scary. In fact, they are what drive the matrix of the web.

Crawlers can come in all different forms and styles. But lets start with the most basic and well known crawler, Googlebot. Googlebot is the crawler that Google uses to create and index for their search engine.

From Wikipedia:

“Googlebot discovers pages by harvesting all of the links on every page it finds. It then follows these links to other web pages. New web pages must be linked to from other known pages on the web in order to be crawled and indexed or manually submitted by the webmaster.

A problem which webmasters have often noted with the Googlebot is that it takes up an enormous amount of bandwidth. This can cause websites to exceed their bandwidth limit and be taken down temporarily. This is especially troublesome for mirror sites which host many gigabytes of data. Google provides “Webmaster Tools” that allow website owners to throttle the crawl rate.”

Now the first thing to note here is that Googlebot is very smart and can even run some of the code of your website. The other takeaway is that it can be a hungry, data-eating beast when it comes to a site’s bandwidth. You can, however, change the settings on your webmasters tool to lower the amount of bandwidth Google eats, but that’s just a side note.

The real question is this: would you ever tell the Googlebot to go away (on purpose) just because it eats bandwidth? No, because of all the benefits a Googlebot brings to your site. Most site owners would say, “No, Googlebot, please stay. In fact let me buy you a steak dinner if you’ll increase my ranking! Here, why not have some champagne while I convince you how awesome my content is…” Also, if you turn off Googlebot, your site would be much less likely to pop up in search, and why would anyone want that?

Now that’s just one type of crawler we’re talking about. And like I said, you have an army of them working for you. One of the robot databases shows 302 active well-known crawlers searching the web right now! Imagine how much data all those crawlers eat on a daily basis. They’re hungry little monsters that feed off the data and turn it into traffic. Some of them can even pull gigabytes a month if your site is big enough. Imagine the traffic that’s generating!

This army of bots, or crawlers, searching through your site does three very important things. It indexes your site, sees your site’s progression over time, and, last but not least, helps the world find your site.

So why don’t they show up in Google Analytics?

Google Analytics is a well written and smart tool that knows the difference between a human and a crawler (most of the time). So when you’re obsessively checking how many people went to your site every day, it does not list the hits from crawlers in its numbers.

You open Google Analytics and see that 4000 unique people viewed your website today. You’re excited because yesterday only 3500 people viewed it. You can rest assured that 500 more people in fact visited your site because as a marketing tool, Google Analytics is only supposed to show you real human beings that are capable of reading or buying things. This is helpful information to have, but it serves a different purpose than the information in your WP Engine analytics, which includes your visits from traffic-boosting crawlers.

At WP Engine, our analytics are a more accurate portrayal of who (or what) is visiting your site. Just because crawlers aren’t capable of buying anything, doesn’t mean they aren’t beneficial. They take up space and do what they do, just like people, when visiting a site. It only makes sense that they pay an entrance fee just like everyone else.

Now the good news is that crawlers boost traffic. Just because they are not laughing at your jokes or ordering your merchandise does not mean they aren’t hard at work. They are working relentlessly, indexing and chomping up data…all to be turned into increased traffic, which is what every site owner wants, right?

What does this all really mean?

Crawlers eat data, and in turn, supply your site with traffic. They don’t show up on Google Analytics because they aren’t real humans. But the point is, real humans won’t be able to find your website without the crawlers visiting and indexing you. It is a symbiotic relationship in which humans and machines feed off each other in a wonderful information cycle.

So welcome these crawlers into to your site with open arms. Feed them well, feed them quickly, and then watch your traffic and business grow.