Redirect Bots Setting

The Redirect Bots setting 301 redirects well-known crawler user agents (bots) on the site to the parent page when they request a page ending in a number or in a query argument.


About Redirect Bots

Bots will see a URL that ends in a number or query like 1, or even a year like 2009, and will increment it (2010, 2011, etc.), continually and high as they want. Each new page they hit isn’t going to be cached, and so the server can start to experience lag and high load.

EX: site.com/page/2009, site.com/page/2010, site.com/page/2011

With the redirect bots setting turned on (default), our server would automatically redirect all of these pages to simply: site.com/page/

NOTE: We highly advise keeping a clean sitemap and robots.txt available for bots to help curb this behavior as well.


Disable Redirect Bots

If a service you’re using to scan your site is having issues or receiving a 301 redirect, there’s a chance this is due to the redirect bots setting.

For example, using Facebook’s URL debugger tool attempts to scrape a specific page of the site that ends in a number, using one of the well known user agents that is redirected by default. This causes the tool to show an error. With this setting turned off, it allows Facebook to properly scrape and analyze the data given.

  1. Log in to the User Portal
  2. Click on the environment name you wish to disable this on
  3. Click Utilities
  4. Locate Redirect Bots
  5. Check Disable
  6. Save

NEXT STEP: Learn how to read and use the robots.txt file

Still need help? Contact support!

We offer support 24 hours a day, 7 days a week, 365 days a year. Log in to your account to get expert one-on-one help.

The best in WordPress hosting.

See why more customers prefer WP Engine over the competition.