Bot Management in the AI Era

AI is a hot topic, with tech companies and their customers alike focusing on how they can use bots and automated agents to improve workflows and business outcomes.

As with everything in life, site owners must learn how to take the good with the bad, as both wanted and unwanted bots can affect how your site performs.

In the following sections, we’ll cover what bots are, the current state of bot traffic, the business impacts of poor bot management, and how WP Engine helps you manage the rising tide of bot traffic web-wide. Let’s dive in!

What are bots?

At their core, bots are automated programs designed to carry out specific tasks online. They’ve been part of the internet since its early days, quietly powering everything from search engines to site monitoring. While some bots are incredibly useful, others can cause serious headaches for site owners, depending on how they’re used and what they’re trying to do.

Some of the very first bots that influenced the evolution of the web were search engine crawlers, which began appearing in the early 1990s. These bots helped index the internet by scanning websites and cataloging their content so it could be surfaced in search results.

Since then, bots have evolved dramatically. Today, they power everything from customer service chat agents to price comparison tools, data scrapers, social media schedulers, and malicious actors attempting to exploit vulnerabilities or skew analytics. 

Understanding which bots interact with your site and (how) is a critical part of managing web performance in the AI era.

Tip

The Intelligent Web is the next evolution of how users are finding and interacting with information online. This advanced, AI-driven web uses machine learning and natural language processing to enhance user experiences on the internet and requires site owners to implement strategies that make their content accessible to both humans and AI systems.

The current state of bot traffic

In 2024, bots overtook human traffic for the first time, accounting for 51% of total traffic online according to a report by Imperva. The report attributes this tidal wave of traffic to the rise and rapid adoption of AI tools and Large Language Models (LLMs) like ChatGPT by OpenAI or Gemini by Google.

This rapid influx of bot traffic across the web can lead to poor performance and higher site operation costs for businesses of all sizes. 

The legal framework surrounding bot traffic, specifically regarding data scraping, is also in flux. Court rulings have already set precedents for how bots conduct data collection and other tasks. In the UK, the Information Commissioner’s Office (ICO) has updated its recommendations for what constitutes lawful bot scraping. In the U.S., the BOTS Act was created back in 2016 to stop individuals and organizations from attempting to automate ticket purchases en masse using bots. Now, ten years later, Congressional leaders are seeking bipartisan support for legislation to regulate AI use for children, among other issues.

As technology continues to evolve, so too will the legal, regulatory, and ethical landscapes around bots and bot traffic. The goal for businesses in the current landscape is to identify which bots are helpful for your business and which are not. 

Types of bots

Not all bots are bad. Some of them, like search engine crawlers and chatbots, can help sites rank and perform better, while others, like spambots or scraper bots, can flood your sites with malicious content or steal your company’s intellectual property.

The in-between bots, sometimes called grey bots, can be good or bad, depending on your business needs and goals. For example, the SEMrushBot is a crawler that collects site data for SEO purposes. While some customers specifically want this bot for site indexing, others see it as spam traffic and prefer to prevent it from crawling their content.

To understand if a bot is good, bad, or somewhere in between, we can categorize them into three buckets:

• Good bots serve helpful and often essential purposes for site owners. These include search engine crawlers like Googlebot that index your content for visibility, uptime monitors that check site availability, accessibility tools that support users with disabilities, and partner integrations that enable external services like Slack link previews or analytics tracking. Their intent is clearly beneficial, and they’re typically allowed.
• Bad bots act with malicious intent and pose a direct threat to security, performance, and revenue. These include bots used for credential stuffing, carding, scraping proprietary content for profit or duplication, and launching distributed denial-of-service (DDoS) attacks. Their goal is to exploit or disrupt systems, often at scale.
• Unverified bots fall into a middle ground where intent is ambiguous or context-dependent, and they often don’t self-identify (as opposed to something like a search engine crawler bot, which will let you know exactly what organization it comes from). These bots can either provide some value or create risks depending on who’s operating them and why. While not overtly malicious, they often consume resources, impact competitive advantage, or raise ethical and legal concerns.

All of these bots—good, bad, or unverified—can affect site performance or the satisfaction of your human visitors, so gaining visibility into what types of bots are accessing your site is vital for success on the modern web.

How bots and AI agents are being used 

Single-purpose bots—like the search engine crawlers and product scrapers—are still common, but the landscape is rapidly evolving.

AI agents represent the next generation of automation. Instead of completing a single task, agents can plan, gather information from multiple sources, and act across systems on behalf of users. This shift is already impacting how websites are built, nudging site owners to consider not just their human visitors, but also the experiences of AI agents on their web properties. This is ushering in an era of dual optimization, where content is shaped differently depending on whether it’s meant for a person or a machine.

“I would imagine we’ll see some of that in the beginning and then the systems will actually just catch up and the industry will settle on a norm,” said WP Engine VP of Product Christine McKee. “What that norm is, I don’t know yet.”

One sign of this change is the emergence of llms.txt, modeled after robots.txt. These files give website owners a way to signal how they want LLMs to interact with their content, offering both guidance and restrictions to AI crawlers consuming content on a site.

As the New York Times puts it: “People use software by touching buttons. A.I. agents use apps and websites by accessing their APIs.” AI Agents can already generate code to interact with any public-facing API, making them powerful and potentially risky tools.

Although this emerging technology is still in its early stages, researchers are quickly advancing the reasoning and learning abilities of AI agents and examining their broader implications through economic and legal lenses. However, key concerns around security, control, and trust remain unresolved.

Identifying bot traffic vs human traffic

The question for site owners is quickly shifting from “how do I block bots from accessing my site?” to “how do I identify bot traffic versus human traffic so I can prioritize great experiences for real people?”

Good news: while some bots are getting better at disguising themselves, there are some tell-tale signs to know when you’re dealing with automated versus human traffic.

By analyzing traffic to your website, you might notice record-high traffic on unusual pages, unusually short or exceptionally long web sessions, excessive login attempts or password reset requests, surges in failed transaction attempts, or other strange behavior.

The right hosting provider can be instrumental in identifying malicious traffic and enabling an intelligent traffic management strategy that allows or disallows specific bots. For example, WP Engine has introduced new features for customers with Global Edge Security (GES) to improve our customers’ visibility into their traffic patterns and their ability to act on it.

When traffic hits the site of a WP Engine customer on our GES network, it’s automatically sorted into one of four types:

• Human: This number encompasses visitors who identify and behave as human users.
• Verified bot: This number encompasses visitors that identify and behave as a good bot, like a search engine crawler or site monitor.
• Automated: This number represents traffic that identifies as non-human, but its intentions are unknown. These are some of the unverified bots we talked about above.
• Likely automated: This number accounts for visitors that claim to be human, but behave in a way that suggests they are bots. These are also more than likely unverified bots.

You’ll notice there’s no “malicious” category. Any bot verified as malicious has already been filtered out by WP Engine at the network’s edge. 

You can use the trends chart to track how these four traffic types are moving over time to identify spikes, drill into verified bot types to see what types of bots are on your site, see requests by location, add traffic mitigation rules, and even flip a switch to enter “Under Attack Mode.” This simple toggle puts your site into a site-wide managed challenge mode that only allows human users in until you decide to toggle it off again, giving you control to take fast action when traffic seems off.

As the future of bot traffic on the web is unknowable, it’s important to choose a hosting partner that’s already creating solutions to prioritize the human experience on a website while also creating a space where good, wanted bots can thrive.

Business impacts of bot management

Poor bot management across your site can lead to a number of negative impacts across your business and budget, while monitoring and optimizing for acceptable bots can lead to greater visibility in emerging LLM tools used by your audience.

Let’s take a quick look at some of the negative and positive impacts of a business’s bot management strategy.

Negative impacts of poor bot management

Poor bot management can significantly degrade site performance, especially during traffic spikes. Bots can overwhelm servers by generating a high volume of requests that compete with legitimate users for resources, leading to slower page load times, timeouts, or even outages, particularly on dynamic or resource-intensive pages. The result is a diminished user experience that may drive away visitors and hurt customer satisfaction. 

Inaccurate analytics are another major consequence of unmanaged bot traffic. Bots can inflate metrics such as page views, bounce rates, and conversion funnels, making it difficult for teams to understand real user behavior or measure the effectiveness of marketing and product decisions. This extra noise in the data can lead to misguided strategies and wasted investment in campaigns or features that seem to be performing well, but are actually being misrepresented by non-human traffic.

Security and cost implications are also serious concerns. Malicious bots are commonly used for credential stuffing, content scraping, or probing for vulnerabilities, putting customer data and intellectual property at risk. 

Even when infrastructure costs are not tied to bandwidth overages, bot traffic can increase CPU usage, database load, and edge service consumption, which may lead to the need for more powerful infrastructure or add-on services. Without proactive bot management, businesses may unknowingly burn resources protecting against or reacting to threats that could have been mitigated at the edge.

Positive impacts of proper bot management

Effective bot management ensures that server resources are prioritized for real users while still allowing beneficial bots to access your site. By filtering out harmful or unnecessary traffic, businesses can maintain faster load times and more reliable performance, especially during periods of high demand. This leads to a stronger user experience, increased engagement, and better results across marketing and sales channels.

Another key advantage is more accurate data. With bot traffic properly managed, analytics reflect true human behavior, enabling teams to make more informed decisions about content, conversion, and performance. WP Engine helps customers achieve this by deprioritizing self-identified bots and fine-tuning platform performance to surface meaningful user insights. Cleaner data means smarter strategies and greater impact.

Proper bot management also opens the door to emerging opportunities like Generative Engine Optimization (GEO). As LLMs become a primary way users discover content, it’s increasingly important for businesses to allow trusted LLM crawlers to index their sites. This positions their content to appear in generative search results and AI-powered discovery tools.

There is no one-size-fits-all approach to LLM bot access. Each brand must consider which bots align with their audience and goals. By enabling the right bots and blocking the harmful ones, businesses can stay competitive and future-proof their digital presence.

Be the boss of your bots with WP Engine

WP Engine consistently reviews how we refine our bot management strategy from the hosting side and works to equip our customers with the tools they can use to develop intelligent traffic management strategies on their own. Together, we can create intelligent web experiences that ensure the rise of bots doesn’t affect your human visitors or your bottom line.

Want to learn more about the current state of bots on the web? Check out the 2026 DE{CODE} agency track for a session that explores how bots and AI agents will affect digital engagement and how businesses can prepare for this AI-driven future, or talk to a representative to learn how tools like Global Edge Security can help you manage your bot traffic!