Empowering Engineers Through Product-first Platforms

At WP Engine, our Product & Engineering organization applies a continuous improvement process to maintain an elite level of software development and delivery performance. 

This includes a laser focus on agility, represented by DevOps Research and Assessment (DORA) metrics for speed, stability, and quality. 

To achieve their goals, our engineering colleagues count on WP Engine’s internal Catalyst Software Delivery Platform (SDP)—and on us, the dedicated Catalyst team, which develops and maintains that platform—to provide a unified foundation for product development, release, and management. 

The aim of the Catalyst SDP is to reduce operational burden and cognitive load on our internal teams while significantly improving service observability, security, and speed of delivery. As the platform’s value to users has deepened over the past two years, the variety of workloads has expanded to include more than 30 applications and services. New teams and products continue to onboard every few weeks. 

However, in the early days of the SDP, engineers who successfully stood up production services on the platform also struggled with application configuration management. While the Catalyst team had settled on a workflow that used declarative configuration tools, we had not achieved the right balance between flexibility and ease-of-use. 

We needed to evaluate SDP users’ challenges more closely and take action. When we engaged with engineers, we learned that app development was indeed slowed by configuration tooling complexity; specifically, users found it difficult to craft Kubernetes configurations that met WP Engine’s security requirements. 

We understood their burden of using multiple tools, custom scripts, and complex configuration patches for customizing resources, as required by our configuration management solution. We also recognized duplicative effort across teams independently resolving common pain points. 

To restore product teams’ focus on best-in-class software development and delivery, we had a job to do. We also had a reputation to uphold as winners of Google’s 2021 DevOps Award for “Unleashing the full power of the cloud”!

Let’s explore the changes we introduced to the configuration management workflow on SDP, our rollout approach, and the impact these changes had on the engineering experience.

Reduced Complexity

To remove the need for overlay patterns, complex patching, or custom scripting, we consolidated the required tooling for operating on configurations. Engineers can now address basic use cases with a single tool, kpt, which simplifies functional configuration changes through its Configuration as Data paradigm.

Shared Functions

Our most impactful deliverable was a library of reusable Kubernetes resource definitions as kpt functions. Engineers can now compose applications with interoperable functional units of configuration, and can leverage packaged, time-tested k8s configurations, reducing code duplication and increasing maintainability.

Documented Examples

To further streamline implementation of common functional use cases, like configuring and authenticating to a database, we provided documented examples for invoking our kpt functions. With the help of a consistent and reliable starting point, engineers can confidently and quickly proceed to their value-added work.

Security by Default

For standard use cases like exposing an application to the internet or routing traffic between services within the mesh, we packaged baseline corporate security compliance into the kpt functions themselves. When applying our functions to their use cases, product teams benefit from the “default-closed” and “least privilege” security posture that we enforce, achieving improved application security without engineering overhead. 

Versioning and Pinning

We applied semantic versioning to our Catalyst cloud builders and kpt functions. Now engineers can pin their image versions and automatically pick up our latest non-breaking changes without needing to comprehend the details of each modification. 

Change Awareness

As part of the rollout of our new configuration management solution, we provided a utility for describing the logical differences between the user’s prior configuration and the new configuration introduced by the kpt functions. Using the utility, engineers can readily understand the functional changes to their applications. 

User Engagement and Results

We partnered with application teams to roll out these changes to 30 workloads on SDP. After making initial updates directly to user pipelines in a dev environment, we invited application owners to approve our pull requests and then test their applications with the updates. 

Direct feedback about these changes indicates that we’ve successfully addressed significant problems in configuration management, and that the transition to improved security and reliability required minimal effort for engineers. In turn, our engineering colleagues have been appreciative of our efforts to simplify their foundational work and restore their focus on creating valuable products. 

 “Using the new configuration management tooling with kpt has been a massive improvement, it’s way easier,’ said WP Engine Sr. Staff Software Engineer Adam Lassek.  

“We got a whole environment up and running inside a sprint, which I never would have expected was possible before.” 

Software Engineer Piotr Purwin also praised the changes, noting that the platform enabled his team to implement kubernetes best practices with fewer lines of code. 

“Sometimes we feel like cooking something fancy, which of course requires basic kitchen utensils, but also more complex and sophisticated tools that may not be available to everyone at home,” Purwin said. 

“Working within the SDP is like entering a big, fully-equipped kitchen and just enjoying the process of preparing the dish.”

What’s Ahead?

There’s more we can do to optimize application configuration management for our users. We will continue building out kpt functions for common configuration use cases, like adding a caching layer to an application. We also want to improve user experience in other areas, including observability. 

While software delivery innovations roll out to our internal teams via the SDP, the benefit cascades to WP Engine customers through faster delivery of new products.

Our vision for SDP is broad and deep, and the leaders of our engineering organization have high expectations for its future use. With more reusable software, out-of-the-box capabilities for applications, and platform enhancements, the Catalyst SDP will continue increasing engineering team velocity and helping WP Engine customers win online every day.

WP Engine Sr. Staff Software Engineer Rahul Dhir contributed to this article.