{"id":5728,"date":"2014-05-28T13:08:24","date_gmt":"2014-05-28T18:08:24","guid":{"rendered":"https:\/\/wpengine.com\/?p=5728"},"modified":"2024-06-06T08:49:39","modified_gmt":"2024-06-06T13:49:39","slug":"wp-engine-dns-gets-update","status":"publish","type":"post","link":"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/","title":{"rendered":"WP Engine DNS Gets an Update"},"content":{"rendered":"

Getting Rid of DNS Timeouts Causing Speed Issues with Outbound Connections<\/strong><\/em><\/p>\n

In our continuing efforts to drive faster site performance, we’ve uncovered and already fixed an intermittent issue affecting code that creates outbound connections.<\/p>\n

Summary:<\/strong><\/h3>\n
    \n
  1. Some customers might have found our caching DNS resolution was timing out<\/strong>, which caused outgoing API requests to 3rd-party services either take a long time to complete or fail outright.<\/li>\n
  2. Slow DNS resolution caused slow page loads<\/strong>, when the total time of the request exceeded 60 seconds. As a result, 500 errors occured when that connection was forcibly canceled by the platform\u2019s self-healing mechanism.<\/li>\n
  3. The fix has already been rolled out on all servers<\/strong>.<\/li>\n
  4. Not all servers were affected<\/strong>. However it\u2019s difficult to quantify how many were affected, because the problem was intermittent<\/strong> (i.e. appears and disappears over time and depends on frequency of outbound requests).<\/li>\n<\/ol>\n

    See below for an example graph of page-load time for one of our customers before and after this fix. The green area is \u201ctime taken for outbound API requests\u201d\u2014that\u2019s the metric this fix addressed. It\u2019s really obvious that the fix was applied at 01:38!<\/p>\n

    \"Web<\/a><\/p>\n

    For the Curious: Here are the Technical Details<\/strong><\/h3>\n

    In our investigation into platform performance with a few dozen customer sites, we kept finding slow external API calls to places like api.wordpress.org (where WordPress goes to check whether there are upgrades for core, plugins, and themes) and many other 3rd-party services corresponding to popular plugins and themes.<\/p>\n

    But the thing is, we know that api.wordpress.org is not slow! It doesn\u2019t make sense that it would be slow to access it from our platform. Furthermore, we would occasionally see something like this:<\/p>\n

    \"Trace<\/a>Do all of these services, two of which are represented by different vendors, take exactly 5 seconds each to complete? No way. Something else was going on.<\/p>\n

    The problem turned out to be a timeout in our external DNS provider, which was Google DNS.<\/p>\n

    The reason for the timeout is that Google has policies designed to protect them from DNS-based abuse\u2014DRDoS<\/a> on port 53 in particular\u2014which has been responsible for several of the largest DDoS attacks that have ever been performed on the internet. Google states that they have policies design to prevent abuse, but isn\u2019t specific about what those policies are or whether a particular user of Google DNS has tripped the \u201cabuse\u201d system.<\/p>\n

    Regardless, we were tripping the system. Some servers were tripping it almost all the time due to a large volume of requests (we assume), while others tripped it occasionally, depending on volume and on what Google\u2019s abuse algorithms contain.<\/p>\n

    It is completely reasonable that Google has such counter-measures in their system. Thus, we needed to stop using Google DNS as our primary outbound nameserver system.<\/p>\n

    The solution is to use data-center-specific internal DNS systems which have no such limits.<\/p>\n

    Why didn\u2019t we use that system to begin with?<\/h3>\n

    What we found was when the primary failed, sometimes the secondary would fail too, leaving us with no out-going DNS and thus causing failures for our customers.<\/p>\n

    So we switched to Google DNS, and haven\u2019t had problems in terms of connectivity.<\/p>\n

    Now we\u2019re putting monitoring in place for various things surrounding DNS, in particular checking once per minute that we can resolve external domains in <100ms rather than getting >1s time and sometimes >5s time as we saw with this problem.<\/p>\n

    Because of our previous experience with internal DNS, the fix we\u2019re rolling out is a combination platter. We use internal DNS as the primary (and tertiary) services, so that in the 99.9% case DNS is unrestricted and as fast as possible. But then we use Google DNS for the secondary and quaternary servers, so that we fail-over automatically to a service with even higher availability. Because we\u2019re not hitting it all the time, the higher service will not trigger a timeout as often, even in that special case.<\/p>\n

    Furthermore, we\u2019ve reduced the DNS timeout from 5 seconds (the Linux default) to 1 second. This means we can try all four of the configured DNS servers in less time than our current system was timing-out a single server.<\/p>\n

    Our new configuration\u2014coupled with monitoring that proactively alerts us whenever a DNS failure or time-out is occurring within 60s of the problem arising\u2014adds even more performance to the DNS component of our platform.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Getting Rid of DNS Timeouts Causing Speed Issues with Outbound Connections In our continuing efforts to drive faster site performance, we’ve uncovered and already fixed an intermittent issue affecting code that creates outbound connections. Summary: Some customers might have found our caching DNS resolution was timing out, which caused outgoing API requests to 3rd-party services…<\/span><\/span><\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[7],"tags":[296,297,281],"class_list":["post-5728","post","type-post","status-publish","format-standard","hentry","category-company","tag-dns","tag-performance","tag-updates"],"yoast_head":"\nWP Engine DNS Gets an Update | WP Engine<\/title>\n<meta name=\"description\" content=\"Stay updated on the latest improvements to WP Engine's DNS services. Learn how these important updates enhance website performance, security, and reliability.\" \/>\n<meta name=\"robots\" content=\"noindex, follow\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"WP Engine DNS Gets an Update | WP Engine\" \/>\n<meta property=\"og:description\" content=\"Stay updated on the latest improvements to WP Engine's DNS services. Learn how these important updates enhance website performance, security, and reliability.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/\" \/>\n<meta property=\"og:site_name\" content=\"WP Engine\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/wpengine\" \/>\n<meta property=\"article:published_time\" content=\"2014-05-28T18:08:24+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-06-06T13:49:39+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/wpengine.com\/resources\/wp-content\/uploads\/2014\/05\/Web-Transactions-Response-Time.png\" \/>\n<meta name=\"author\" content=\"Jason Cohen\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@asmartbear\" \/>\n<meta name=\"twitter:site\" content=\"@wpengine\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Jason Cohen\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/\",\"url\":\"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/\",\"name\":\"WP Engine DNS Gets an Update | WP Engine\",\"isPartOf\":{\"@id\":\"https:\/\/wpengine.com\/resources\/#website\"},\"datePublished\":\"2014-05-28T18:08:24+00:00\",\"dateModified\":\"2024-06-06T13:49:39+00:00\",\"author\":{\"@id\":\"https:\/\/wpengine.com\/resources\/#\/schema\/person\/7c15d2de57675ce86be8a2d03f9c0eea\"},\"description\":\"Stay updated on the latest improvements to WP Engine's DNS services. Learn how these important updates enhance website performance, security, and reliability.\",\"breadcrumb\":{\"@id\":\"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/wpengine.com\/resources\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"WP Engine DNS Gets an Update\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/wpengine.com\/resources\/#website\",\"url\":\"https:\/\/wpengine.com\/resources\/\",\"name\":\"WP Engine\",\"description\":\"Managed Hosting for WordPress\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/wpengine.com\/resources\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/wpengine.com\/resources\/#\/schema\/person\/7c15d2de57675ce86be8a2d03f9c0eea\",\"name\":\"Jason Cohen\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/wpengine.com\/resources\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/5816acd522677b42018f9d1dbd9492ba4f2def939b7d38034613e09b2564bc65?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/5816acd522677b42018f9d1dbd9492ba4f2def939b7d38034613e09b2564bc65?s=96&d=mm&r=g\",\"caption\":\"Jason Cohen\"},\"description\":\"Founder, CTO of WP Engine\",\"sameAs\":[\"https:\/\/blog.asmartbear.com\",\"https:\/\/twitter.com\/asmartbear\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"WP Engine DNS Gets an Update | WP Engine","description":"Stay updated on the latest improvements to WP Engine's DNS services. Learn how these important updates enhance website performance, security, and reliability.","robots":{"index":"noindex","follow":"follow"},"og_locale":"en_US","og_type":"article","og_title":"WP Engine DNS Gets an Update | WP Engine","og_description":"Stay updated on the latest improvements to WP Engine's DNS services. Learn how these important updates enhance website performance, security, and reliability.","og_url":"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/","og_site_name":"WP Engine","article_publisher":"https:\/\/www.facebook.com\/wpengine","article_published_time":"2014-05-28T18:08:24+00:00","article_modified_time":"2024-06-06T13:49:39+00:00","og_image":[{"url":"https:\/\/wpengine.com\/resources\/wp-content\/uploads\/2014\/05\/Web-Transactions-Response-Time.png"}],"author":"Jason Cohen","twitter_card":"summary_large_image","twitter_creator":"@asmartbear","twitter_site":"@wpengine","twitter_misc":{"Written by":"Jason Cohen","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/","url":"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/","name":"WP Engine DNS Gets an Update | WP Engine","isPartOf":{"@id":"https:\/\/wpengine.com\/resources\/#website"},"datePublished":"2014-05-28T18:08:24+00:00","dateModified":"2024-06-06T13:49:39+00:00","author":{"@id":"https:\/\/wpengine.com\/resources\/#\/schema\/person\/7c15d2de57675ce86be8a2d03f9c0eea"},"description":"Stay updated on the latest improvements to WP Engine's DNS services. Learn how these important updates enhance website performance, security, and reliability.","breadcrumb":{"@id":"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/wpengine.com\/resources\/wp-engine-dns-gets-update\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/wpengine.com\/resources\/"},{"@type":"ListItem","position":2,"name":"WP Engine DNS Gets an Update"}]},{"@type":"WebSite","@id":"https:\/\/wpengine.com\/resources\/#website","url":"https:\/\/wpengine.com\/resources\/","name":"WP Engine","description":"Managed Hosting for WordPress","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/wpengine.com\/resources\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/wpengine.com\/resources\/#\/schema\/person\/7c15d2de57675ce86be8a2d03f9c0eea","name":"Jason Cohen","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/wpengine.com\/resources\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/5816acd522677b42018f9d1dbd9492ba4f2def939b7d38034613e09b2564bc65?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/5816acd522677b42018f9d1dbd9492ba4f2def939b7d38034613e09b2564bc65?s=96&d=mm&r=g","caption":"Jason Cohen"},"description":"Founder, CTO of WP Engine","sameAs":["https:\/\/blog.asmartbear.com","https:\/\/twitter.com\/asmartbear"]}]}},"acf":[],"_links":{"self":[{"href":"https:\/\/wpengine.com\/resources\/wp-json\/wp\/v2\/posts\/5728","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/wpengine.com\/resources\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/wpengine.com\/resources\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/wpengine.com\/resources\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/wpengine.com\/resources\/wp-json\/wp\/v2\/comments?post=5728"}],"version-history":[{"count":0,"href":"https:\/\/wpengine.com\/resources\/wp-json\/wp\/v2\/posts\/5728\/revisions"}],"wp:attachment":[{"href":"https:\/\/wpengine.com\/resources\/wp-json\/wp\/v2\/media?parent=5728"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/wpengine.com\/resources\/wp-json\/wp\/v2\/categories?post=5728"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/wpengine.com\/resources\/wp-json\/wp\/v2\/tags?post=5728"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}