{"id":4452,"date":"2014-01-06T13:50:37","date_gmt":"2014-01-06T19:50:37","guid":{"rendered":"https:\/\/wpengine.com\/?p=4452"},"modified":"2021-11-21T14:45:15","modified_gmt":"2021-11-21T20:45:15","slug":"staying-one-step-ahead","status":"publish","type":"post","link":"https:\/\/wpengine.com\/resources\/staying-one-step-ahead\/","title":{"rendered":"Staying One Step Ahead"},"content":{"rendered":"

\"pie<\/a>Almost 2 out of every 3 visitors to your website is not human.<\/p>\n

That’s according to a recent report by Incapsula<\/a>, which found that up to 61.5% of all website traffic is bot (non-human) traffic. If Incapsula\u2019s figures are correct, bot traffic has grown by <\/a>20.6%<\/a> since 2012.<\/p>\n

As shown in the pie chart, one third of these bots are search engines or other “good” bots.<\/p>\n

Of the “bad” bots, 5% are scrapers, 4.5% are hacking tools, 0.5% are spammers, and 20.5% are other impersonators. This is an 8% increase in impersonator bots since 2012.<\/p>\n

Good Bot, Bad Bot<\/h3>\n

Some bots\u2014or software applications that run automated tasks over the internet\u2014are malicious.<\/p>\n

Malicious bots do things like leave spam comments and links, or attempt to exploit sites in order to install malware. Obviously you don\u2019t want these sorts of bots attacking your website. There are a number of things that we here at WP Engine do to stay ahead of these types of bots (more on this below).<\/p>\n

Other bots have good intentions. The “good” bots crawl the web to perform automated tasks that are helpful, such as indexing websites, collecting analytical data, or archiving internet content. You can read more about the benefits of good bots in this previous WP Engine blog post<\/a>.<\/p>\n

Although good bots are necessary for site growth, the increase in overall bot traffic can put more pressure on your website\u2019s servers. Luckily, so long as you are hosting with WP Engine<\/a> you don\u2019t need to worry. This is because our caching infrastructure is robust enough to handle a flood of traffic while still serving up fresh content.<\/p>\n

While the general message of the Incapsula report is accurate\u2014there is certainly a trend of more non-human traffic\u2014the numbers and breakdown of different types of bots may not be representative of the wider web.<\/p>\n

How Representative is the Data?<\/h3>\n

Incapsula obtained their data by observing 1.45 billion bot visits to the 20,000 sites operated by their clients over a 90 day period. As noted by Dr Ian Brown, quoted in this story by the BBC<\/a>, the figures are useful as an indication of the growth in non-human traffic, but may or may not be representative of the wider web.<\/p>\n

Jason Cosper, a WordPress and security expert here at WP Engine, agreed that while there certainly has been an increase in bot traffic, the Incapsula data does not necessarily reflect everyone’s experience. He explained:<\/p>\n

If you look at WordPress sites in general, you\u2019d see a lot more spam trying to hit them. People and bots trying to throw links into comments and things like that.<\/p>\n

There was also a concerted attack during this past spring<\/a> where a massive botnet was trying to guess weak administrator account passwords. That attack was handled admirably by our network configuration, but it was constantly hitting some sites well into the summer. That amounted to much more than 0.5% of our traffic [the number quoted in the Incapsula report].<\/p>\n

Their [Incapsula’s] numbers are what they\u2019re seeing based on their usage. However, it\u2019s not the whole picture.<\/p><\/blockquote>\n

So while Incapsula has seen a 75% drop in spam bots since their last report, that hasn\u2019t necessarily been experienced by WP Engine or the wider WordPress community.<\/p>\n

Being Proactive Against Attacks<\/h3>\n

WP Engine has number of measures in place for dealing with impersonators, spammers, and other types of malicious bots. We are constantly trying to stay one step ahead of hackers, as Jason notes:<\/p>\n

While I\u2019m the sort of person who finds this sort of thing fun, this is a constant uphill battle. I feel like Sisyphus sometimes, pushing the boulder up a hill.<\/p><\/blockquote>\n

One of the measures WP Engine has in place is to automatically filter a number of user agents that are known to be malicious. This basically blocks those attacks, preventing them from even hitting the server in the first place.<\/p>\n

Not only do we block known attacks, we also stay on top of incoming attacks from unknown user agents. So as an attack is coming, we add on-the-fly rules to handle the attack before it even gets to most customers WordPress installs.<\/p>\n

In addition, we constantly monitor the server, so if an attack like that does happen, we can find ways to clean it up very easily. This is done through our partnership with Sucuri<\/a>.<\/p>\n

Finally, WP Engine and Sucuri team members follow hacker oriented blogs and other nefarious fringe sites. That way, we can keep an eye on the attackers, following all of the latest tricks as they\u2019re made available. By staying on top of the most recent developments in malicious software, we can block, prevent, or scan for new attacks.<\/p>\n

Want to Know More?<\/h3>\n

While WP Engine does our best to ensure the safety of your site, if you\u2019d like to know about additional security measures you can take, check out Jason\u2019s post on Advanced Anti-Spam Techniques for Torque<\/a>.<\/p>\n

You can also check out these resources on anti-scraper techniques:<\/p>\n