Growth is Hard
When WP Engine was young, life was simple. We didn’t have meetings, because everyone knew everything. We didn’t have training classes, because we hired about one person every three months, and they could just “ride shotgun” and learn the ropes. We were working out of an amazing accelerator called the Capital Factory and space was constantly getting tighter, which is a good thing, because energy increases in tight quarters. (Music and stand-up comedy are best in tight quarters too). It was small, intimate, and probably one of the most energizing spaces I have ever been in. I knew everyone’s name, the names of their dogs and kids, and what they liked on their Jimmy Johns subs.
Today I walked into our weekly All-Hands meeting. We have a lovely office to call all our own in downtown Austin, TX and it breathes with dedication and creativity. But I did notice something. There were people I didn’t instantly know—and when I saw a picture of their dog I didn’t know its name. (We’re big on pets here at WP Engine!)
This has shown itself in other more material ways, that are both good and challenging. We have more customers than ever before, and that means more folks working here, and more servers. WordPress and life on the Big Bad Internet are ever-changing, and this means there are more edge cases, more high traffic cases, more servers, and a lot more people we touch every day.
Growth is Hard
The hardest thing about growth is that rare things become common.
A Linux kernel bug that causes a hard-crash once every 4 years happens every single day when you have 2000 servers (as we do). A code-push we did yesterday impacted 4 customer installations out of over 150,000—something so rare that even in retrospect it would have been impossible to catch the problem beforehand, and in fact we didn’t catch it even with multiple code insepctions, QA testing, canary-server testing, and so on.
This same principle is equally biting in customer service. Sometimes, we will inevitably generate a bad customer experience. Sometimes, we’re going to tell a customer that the problem is X when in fact it is Y. Sometimes, we’re going to misread a response from a customer, or misread one of our own responses. Or a ticket will be handed off from one person to another, and everything gets lost in translation. Sometimes we’re not going to have disseminated information about a change, or a policy, or how our code works, or whether we’ve made a change to our platform that impacts customers, and thus some support folks won’t be in the loop and then turn around and tell our customers something incorrect.
This is Not An Excuse For Having Those Sorts of Problems!
Rather, these are the types of things we have to focus on, every day. We need to ask ourselves: How could that have been prevented? If not prevented, how could we have caught the problem before the customer did? If not caught, how could we have messaged the customer better? How can we communicate better internally and externally? How should we reorganize teams? How should our training programs change? How should on-boarding for new folks change? The list goes on.
All hosting companies will sometimes generate terrible experiences and sometimes generate great ones. So, to me, the two primary questions are:
- What does the company do with those bad experiences?
- What does the company do with statistics about the average, or a 95th percentile case?
I’d like to tell you our answers to those questions.
After every ticket we ask the customer whether they had a good experience—yes/no—with an optional comment box. A surprisingly large number of people give us that feedback—which we’re grateful for. The ones marked “no” we call an “UNSAT” (unsatisfactory).
Our CEO, Heather Brunner, along with everyone in the support management organization, reads every, single UNSAT. In fact, I myself get emails everyday from Heather, forwarding something with a note like “Could this be related to [an existing initiative]?” or “Isn’t this a pattern?” or “Is this something we need to create a special project around?” or “Do you think [person Y] could help with this?”
In other words, we treat UNSATs like the FAA treats airplane crashes—we look at every one and ask ourselves: Could this have been prevented? Do we need to give this feedback to some folks internally? Does this need to be institutionalized in training material? Could we write code to detect/fix/monitor/notify/analyze this? Only the most idiosyncratic UNSATs end up passing all the questions without action.
That’s the answer to question 1. As for question 2, let’s look at some data.
In mid-2013, we had a 96% satisfaction rating, meaning only 1 out of 25 rated tickets were UNSAT. That’s very good—in fact I haven’t heard of another hosting company with a higher statistical rating. We were happy with that. But we watched it like a hawk. A hawk who’s really into statistics.
By January of 2014, that rating had slipped to 93%. That number is still world-class compared to our peers, but that is not our yardstick. This was an all-hands-on-deck critical catastrophe as far as we were concerned.
To see why, consider that this meant twice as many people had UNSAT than before. When you think of it that way, you stop patting yourself on the back for a 93% satisfaction score.
So we took action. Lots of action.
Today, I’m happy to report that the last-30-day-rolling score is back up to 97% satisfaction. That doesn’t mean “we’re done.” What it means it “OK, that stuff worked, now don’t stop innovating in service or else that will happen again.”
So, what did we do exactly in the first half of 2014 to address this?
- Hiring. We closed our Series C financing in January and immediately put it to work in hiring in the Support Team. We’ve increase the team by 50% since then. It’s very hard to hire quickly and yet maintain our standards of both attitude (culture) and aptitude (ability). We’ve even hired additional internal recruiters to help us accelerate this process.
- Training. We already had a great training team, but they needed more help—more hands, more material, and more feedback about what would be useful, both for existing employees and new employees.
- Specialization. We learned that, especially during the first six months that someone is at WP Engine and (of course) is still learning the ropes, it’s better to specialize in a few subjects and become deep in those rather than getting the complete lay of the land. We also need to balance that against variety and ensuring that everyone has the chance to learn many things and become well-rounded.
- Re-Grouping. With more folks, we were able to reorganize the support groups around specific things. For example, now we have a special team who handles all tickets from brand new customers. New customers have special needs, they don’t know our platform yet, they have questions around things like site migrations, DNS, email, etc..
- Direct-to-Engineer. Some of our customers are highly technical, so whenever they contact us, it’s with difficult, interesting problems—not ones that can be solved with a knowledge base article or a simple, obvious response. Therefore, we started creating pathways for those customers to get to engineers faster—people who can work on the mind-bending stuff. Of course we don’t have that 24/7 yet, like we do with regular support. Fortunately, those problems are usually OK to be solved during normal business hours, so overall this approach has been effective.
- New Customer tools in User Portal. Sometimes, the best service is when our customers can take an action or solve a problem themselves, without having to contact us. We’ve launched a bunch of things in the User Portal to this effect. For example, you can now purchase, install, and configure SSL certificates, whereas before that took a few support tickets to get working. Not only is this faster for customers, it means our support folks can focus on the issues which do require human interaction.
- Unique Innovation in our Platform. The best way to have fewer “my site is slow” tickets is for the site to not be slow! To that end we’ve built and deployed improvements, much of which are novel innovation. We’ll talk about these things in greater detail in future blog posts, but one example is what we call “WPEngineX” which is our own customer module to nginx (our front-end web server technology) in which we do WordPress-specific traffic shaping. What this means, is when a human being is waiting for a page to render, and so is a robot (like a web crawler), the web crawler is forced to wait until the human being gets her page, so the human gets a better experience. There’s also traffic which is almost always nefarious but not always, so we make it even lower priority. There’s much more to it, but even just this high-level behavior is unique in the hosting industry. We have a lot more coming that we’ll be announcing over the next few months. (If you want early access to some of this, contact us!)
To anyone that has had a support issue in the past, or is having one now, I would like to say I’m sorry for your experience. But as you can see, we’ve been busy. We will continue to improve, and are seeing our current efforts working on many fronts. We do this because we love every customer here at WP Engine and your success is incredibly important to us.
The last thing we can be right now is complacent. We are guided by staying humble, staying honest, and working hard. If things are pointing in the right direction, that’s great. The moment we stop being hungry, stop reading those UNSAT’s, stop striving to become better, that will be the beginning of the end.
But in fact, this is just the beginning!
Our VP of Customer Experience, Tina Dobie, loves hearing directly from our customers. Her philosophy is that it’s only by listening to you that we can find out exactly what we need to do to continuously improve—and give you the best possible experience. If you have specific feedback, please let us know. Either open a Support Ticket or email Tina directly—[email protected].
Yes growth is hard, but that is no excuse. In the 80’s I worked in the main office of Trader Joes which at the time had only 32 stores in Southern CA. Now look at them – 400 stores all across the country. From what I can tell from my existing contacts, they managed their growth well!
As we say, this is not an excuse for having those sorts of problems. The best thing human beings or companies can do in these situations is be introspective, rededicate ourselves to our promise to our customers, and be transparent on actions and steps we are taking to help.
Like we said, we really do welcome your feedback or advice. Please don’t hesitate to write into Tina with any specific concerns.
Thanks for your comment Greg.
– Kirby
Funny you bring up Trader Joe’s, since I stopped going to my local store due to consistent subpar produce.
No company is perfect, and the fact that these people are even willing to admit they struggled a bit is admirable.
Thanks for sharing this. I’m in the majority category that has never had an UNSAT with you guys. But I definitely appreciate your openness and frankness in discussing what problems you’ve been having and what you are doing about them. Growth is hard. You guys have been managing it very well (much better than your competitors, if not always up to your own standards)
Hi Evan,
That’s great to hear. Thanks for your understanding and support as we double down on improving and innovating.
– Kirby
I’ve been really considering a move away from WPE, so I really do hope things start to change soon as the $99/mo isn’t worth it currently. Frustrating!
Hmmm, The response time seems to have slowed on responses to comments as well, especially to @scott, who said he is considering a move. I for one was seriously considering getting hosting with WPE but after reading a lot of reviews and the review from Matthew Woodward I think I will stay with my current host. Sorry guys, growth is not an excuse, it should be an opportunity to go farther, faster and do it even better.
You’re right that growth should be an excuse for accelerating leadership, not service issues. Of course, right in the post in bold letters I agreed with you that growth is not an excuse!
Rather, this is one of our defining moment as a company, a Defcon 1, where we either meet the challenges that growth creates or we don’t.
It’s fine that in posts like this we try to explain what’s going on and what sorts of things we’re doing about it, but of course all that matters in the end is the actual service level, both in customer support and in the technical platform.
We will have to earn our reputation either way, and the truth will win. Right now the truth is we still have a way to go, but also that we know that and are acting accordingly.
I hear you Scott. We all do, from the CEO to the Engineering folks to the Support folks.
We realize we literally live or die by our service — meaning both the technical platform and customer service, and that right now we’re in a crisis that will define whether folks like you will stay or go, and that it’s up to us to actually implement the changes needed to re-earn your trust.
I’ve experienced a slow WPEngine site on occasion and in the past there were some caching issues with some plugins but you’ve done an awesome job addressing those caching issues and I’m sure you’re working as hard as you can to speed everything up.
I understand growing pains are hard – and there’s only so much WordPress talent around. Thanks for being open here. I’m still a happy customer. 🙂
Thanks man! Encouragement is much appreciated while we continue to be at Defcon 1.
If you’re dealing with all these issues (never mind the thread running in the AWP group on FB), would you not serve existing and new customers well to not take any new customers and simply focus on fixing and getting things where they need to be for existing clients and then re-focus on growth once some level of stability is reached for the platform.
Hi David,
Thanks for the constructive criticism and idea.
In terms of what our Tech Ops and Engineering teams are working on, we have already done what you suggest — no new features, but rather focus on understanding what’s happening on the platform itself (monitoring, automation, fixes, deep-dives into particular customer sites, etc), as well as additional tools for Support so they can have more information and therefore be more accurate and effective.
In terms of turning off signups, in fact the majority of our support tickets come from existing customers, not new customers, therefore turning off signups wouldn’t drastically affect the totality of Support.
You guys have a 93% satisfaction rating? I’m assuming that’s after the individuals that were unsatisfied moved away? I came and went a while back and don’t remember ever getting a satisfaction survey.
Quite frankly, your customer service and tech support is the pits – and the worst part is, it’s trending worse, not better. I understand within growth stages there are going to be times when you have to ramp up, etc, but the changes within WPE reflect a paradigm shift and change in approach that should be worrisome for anyone on your platform.
It’s obvious WPE’s business and architecture decisions are being driven by marginal cost and focused on quantity, not quality. E.g. hoping Varnish hides the deplorable app server performance. (How many hours do devs have to waste waiting for your app servers to grind away in the backend?) Sure, for end users they get served cached copies, but the backend performance is laughable. The WPE app servers have become a punchline in and of itself.
You charge a premium for your service offering, but it isn’t reflected in your support. For example, there is no reasonable excuse for a ticket that has been received and touched during business hours to wait until the next day for resolution. None. Period. And the internal policies that allow for that kind of a situation belie the commitment that a vendor or service provider of your caliber should operate under – especially when you claim to provide a premium level of service.
This entire post is specious marketing-tripe. It even follows the standard cliche PR formula. Acknowledge the failings, apologize, point to progress being made, etc…except for the fact that there is little sign that WPE is going in the right direction. WPE’s reputation, at least among the developer groups I participate in, is getting progressively worse. Taking it further, the “marketing partners” that sing your praises and claim that customer service is unsurpassed are (rightfully) taking a reputation hit for attesting to something so patently untrue.
If you want to be taken seriously, make sure your actions and policies are commensurate with your claims. Spurious assertions can’t be cashed in, no matter how many times you repeat them while clicking your heels.
“Our support does not suck. Our support does not suck. Our support does not suck.”
Nope. Still sucks.
We’re not in Kansas anymore, and the WPE of yore no longer exists. You know that. We know that. You wish it weren’t true. We wish it weren’t true.
If you really believe, as this post states, that the issues with WPE are isolated and uncommon, then there really is no hope – because until you see it as an existential crisis, you won’t give the issue (and solution) the credence it requires.
I absolutely do not think bad support experiences and indeed platform issues as well at WP Engine is isolated and uncommon.
In fact, just the opposite. It is absolutely an existential crisis, as you correctly say. That’s why we’re mobilizing the entire company in service of this issue.
There’s engineering work on the platform (for betterment and monitoring rather than “new features”) — more on that in future blog posts. There’s the litany of things listed above.
But please don’t for a minute think that we’re slapping our hands and saying “problem solved!” Just the opposite.
Rather, this post was (intended!) to show that we do know the situation and a variety of things that are in progress to address it, not some sort of Bushian “Mission Accomplished” declaration!
I do, however, understand that it can come off that way. Some people have said this is “PR tripe,” but actually it’s a heartfelt diatribe from me, the founder, and not from a PR agency. Who knows, maybe an agency would have pointed out what you correctly point out, which is that it’s not clear enough in this post that we consider this Defcon 1 right now.
In any case, thank you for the direct and honest feedback, which I agree with, except for the parts where you say this was written by and for some disconnected PR concept.
I’m curious what your benchmark is. While WP Engine isn’t perfect, have you self-hosted, or been on unmanaged hosting or hosts that don’t specialize in WP? (I’m guessing you probably have, hence my push-back.) I can’t say I’ve tried out other direct competitors to WP Engine, but in comparison to everything I have tried, they are far superior. (And, yes, that doesn’t mean I’m not hoping for improvement in the few places I’ve experienced slight issues. I did complain once about backend slowness, but that was in comparison to previous performance… at it’s slowest, it was faster than anywhere else I hosted. I was bummed I couldn’t use Pretty Links Pro, and wish their redirection engine were a bit easier to use, etc. All pretty minor in the big picture though, so far.)
Bottom line… my business philosophy is centered on adding value to my customer’s communication via the Web. Any time spent mucking with technical problems is time better spent developing a solution that is going to benefit my clients. That’s worth a lot to me. Is it worth enough to pay the premium for WP Engine? So far, it certainly is. And if my business model keeps working, and WP Engine doesn’t fail in what this article stated, I don’t see that changing.
Growth is real hard. Yes, it happens but if you are running a business, you just got to grow.
For WPEngine, I think you guys are doing good (real good to be honest). Yes, from time to time there’s some bad comments but it looks like the ‘boss’ is really working on it.
Keep it up, team. Keep it up!
Thanks! We definitely have work to do but it’s good to hear encouragement as well.
While I’m probably the type of customer the WPE may consider to be a PITA, I’ve found their service and support to be well worth the money for the past several years.
If only I can learn to be as gracious under such growth and demand.
It’s because of their service that my business has experienced the same.
Jason and team, we’ve had our ups and downs with WPEngine. There have literally been several times I’ve written in support requests, “We want to love you guys but you make it so difficult.”
As a Premium Account holder, I am thrilled to hear about the direction you’re taking to steer the ship away from the rocks.
Please keep being open and honest with us. I’d love it if you literally made a one-page site detailing some of these new measures you’re taking, so we could share that with our clients, whom also have become concerned over the past few months.
Let’s do this FTW!
You bet, we will never stop being honest about what’s going on. It’s always been a core value, not just on our website but literally on the wall as you walk into the office (that all employees signed), and a key thing every day.
Of course being honest doesn’t fix it — that takes concerted, continuous work — but it’s part of the equation.
Jason, it would be nice to hear an honest response to what happened in Matthew Woodward’s case. While that hasn’t at all been my experience, my sites are tiny in comparison, so probably not pushing the bounds as it seems Matthew’s were. But, unless he’s lying about how things went, it seems like he’s one of the first cases I’ve seen where you guys were truly and clearly in the wrong, and where customer service totally went off the rails over and over again.
I’ve been highly critical of a number of other such complaints, as the people often did fairly dumb things, or seemed incredibly impatient with WP Engine. But 8 months of that kind of treatment is far more patient than I’d be, and the solutions (both technical and customer service level) seemed well within WP Engine’s capability to address.
What up? And if you did truly fail on that one, what did you do to try to make it right?
You are definitely correct that Woodward didn’t get the service that is up to our own standards. No doubt about that. No argument. And, neither have other customers. Mea culpa.
There is more to that story than what he presented — there’s always two sides. We are not willing to engage online with someone who clearly is interested in smearing instead of creating a real discussion. Even though it would help defend our honor, we’re not willing to insult him online.