Get Started 
Key Takeaways
Our team started receiving reports of unexplained 500 errors, and we realized they were being caused by bot traffic. Unlike humans who land on popular cached pages, crawlers bulk-read deep archives, paginated lists, and uncached URLs that consume massive PHP worker capacity.
WordPress builds pages on demand through PHP workers that execute one request at a time with effective queueing. An influx of concurrent automated traffic requests that exceeds available workers forces excess traffic to wait in line.
Because automated traffic often targets obscure and dynamic URLs that skip static delivery, servers are forced to spin up PHP workers for nearly every bot request, straining server resources that may be shared by multiple environments.
WP Engine implemented intelligent traffic management strategies and strict resource caps to protect human experiences. Capping automated traffic at a maximum of 80 percent of available PHP workers ensures that at least 20 percent are always available for human visitors.
A few weeks ago, customers on our shared infrastructure started reporting a wave of 500 errors we couldn’t make sense of. None of the installs had reported any real traffic surges; there were no product launches or viral posts. On paper, none of the sites had any reason to be struggling.
After some digging, we found a bottleneck most site owners aren’t thinking about, and it was caused by the web’s new majority constituent: bots. While AI has been transformational for businesses, creating new avenues to connect with customers, increased automated traffic can also clog up resources those very customers need to visit your website.
Here’s what we found and what we changed across our platform to course-correct for our customers.
First, what is a PHP worker?
WordPress®1 builds pages on demand in PHP. When someone requests an uncached page, the server hands that request to a PHP worker. As xCloud puts it, a PHP worker is “one PHP-FPM child process that can execute one PHP request at a time.” That PHP worker is what loads all the information required to assemble and serve the page a visitor has requested.
The key words there are “one request at a time.” If you have 10 workers and 15 uncached requests arrive at once, five of them have to wait in line. If they wait too long, they’ll eventually time out and get a 500 error instead of the page they wanted.
Caching keeps most of this in check by allowing a request to skip PHP entirely, which is why the WordPress Advanced Administration Handbook treats caching as a core performance strategy. But plenty of requests are not or cannot be cached (think deep cuts to your old content that doesn’t get regular traffic or dynamic cart pages on eCommerce sites). Too many concurrent requests for uncached pages strain your overall performance because of the volume of PHP workers required to fulfill them.
The problem: bots don’t browse like people
As we’ve all become acutely aware over the past few months, bot traffic has overtaken human traffic. This trend has been reported by multiple sources already, and there’s no sign of decline as far as I can tell.
More automated traffic on the web in and of itself isn’t a huge issue. WP Engine already filters out malicious automated traffic before it ever reaches the PHP layer, so the bots that remain are usually harmless. Problems are really showing up because of how bots move through the sites they visit, because they can consume resources human visitors need.
Humans tend to start their visit to a site on popular pages that are already cached: home pages, popular product pages, and the like. Crawlers, on the other hand, bulk-read everything: deep archives, paginated lists, filtered product views, and obscure URLs that nobody has requested in months. Those pages are far less likely to be cached, so every single request requires a PHP worker.
Turns out, that’s exactly what was causing all those 500 errors we were seeing.
Following 500 errors back to the source
On shared infrastructure, installs from multiple customers draw from the same pool of resources. This structure is what makes it more affordable, but it also sometimes means sites feel the impact of heavy traffic on a neighboring install. Dedicated infrastructure faces a version of this same issue. Even though one customer owns all the server space, they often have multiple installs, and a traffic spike on one site can pull resources away from the customer’s other environments.
When we investigated the customers who reported unexplained 500 errors, the cause was almost always another install on the same infrastructure. The customers who were impacted weren’t necessarily causing the problem, nor were the neighboring sites experiencing high traffic. And frankly, choosing shared infrastructure over dedicated doesn’t inherently cause the problem either. Automated traffic was simply eating up all the available PHP worker capacity across the affected servers, leaving none for real people.
How we solved it
We knew we had to take action, so we started by changing how requests wait in line.
Every request that comes into our platform is now classified as either human traffic or automated traffic. Humans go into a queue specific to the install they want to visit. Automated requests, no matter which install they’re aimed at, go into a single shared bot queue.
Imagine three installs on the same server. A, B, and C each have their own queues for human traffic, plus one shared bot queue. Our PHP workers might serve a request from A, then B, then C, then one from the bot queue, then A, B, C again.
That fixed most cases, but not all. Sometimes we saw that the bot load was so heavy, and human traffic so light, that workers were pulling only from the bot queue, because the bots had arrived first. Then, when a real visitor did arrive, no workers were free to fulfill their requests.
So we added a hard limit. Automated traffic can never consume more than 80% of the available PHP workers on WP Engine servers. If there are 10 workers, bots can occupy eight at most. At least two are always held in reserve for humans, no matter how hard a site is hit by bots.
We still left bots with plenty of resources, because the ones that reach the PHP layer are often bots you want on your site. These are the bots that help businesses get indexed, cited, and discovered. Bots still get served, just not at the expense of your actual customers, who will notice every second of delay if a site is slow to load.
We first rolled this out on shared plans because they were the canaries in the coal mine that alerted us to the issue in the first place. Soon after, we decided to preemptively solve the problem for premium environments as well. That way, if one of the installs on a dedicated server gets hammered by crawlers, it won’t drag down every other site the customer runs on it.
What we’ve seen so far
Since the change, our customer experience team has confirmed that ticket volume related to noisy-neighbor problems on shared infrastructure has decreased substantially.
While this problem may never disappear entirely, bot traffic can no longer cause it for WP Engine customers.
We’ll keep adapting on your behalf
Some automated traffic is valuable. It’s all but required for widespread discoverability, especially as more customers turn to AI for tasks like comparison shopping and research. Blocking all bots isn’t the answer, but neither is letting them crowd out the real people you want to serve most.
Our job is to make sure that when a person wants to buy, sign up, or read about you, there’s capacity available to take care of them. Capping automated traffic’s share of PHP workers is one concrete way we’re doing that, and we’ll continue to adapt our platform as the web changes to make sure our customers can effectively prioritize their customers.
- WP Engine is a proud member and supporter of the community of WordPress® users. The WordPress® trademarks are the intellectual property of the WordPress Foundation, and the Woo® and WooCommerce® trademarks are the intellectual property of WooCommerce, Inc. Uses of the WordPress®, Woo®, and WooCommerce® names in this website are for identification purposes only and do not imply an endorsement by WordPress Foundation or WooCommerce, Inc. WP Engine is not endorsed or owned by, or affiliated with, the WordPress Foundation or WooCommerce, Inc. ↩︎
