The Rise of the Platform Engineer: How to Deal With the Increasing Complexity of Software

Editor's Note: The following is an article written for and published in DZone's 2024 Trend Report, The Modern DevOps Lifecycle: Shifting CI/CD and Application Architectures.

DevOps — ✓
DevSecOps — ✓
Platform engineering — ?

Is platform engineering just another term used for a specialization of DevOps, or is it something different? The truth is probably somewhere in the middle. DevOps and its associated DevXOps flavors have a strong cultural spice that puts the individual teams at the center. Unfortunately, in many places, DevOps has led to new problems like tool proliferation and a lack of harmonization across the enterprise. One could say that in response to the very strict silos and strong centralization of the past, DevOps has pushed the pendulum too far toward federation — and, hence, a suboptimization at the team level — to the detriment of the organization. This has been felt most by the larger, more complex enterprises that have to deal with different technology stacks and differing levels of maturity across the organization.

Platform engineering has evolved as a response to this enterprise-wide challenge. Platform engineering is not a replacement for DevOps. Instead, platform engineering complements DevOps to address enterprise-wide challenges and provide a tooling platform that makes it easier for individual teams to do the right thing rather than break things while trying to maintain consistency across the organization.

IT delivery has increased in complexity over the last few years, given that more applications are moving at a faster pace. This means organizations cannot rely on individuals to control the complexity; they require systemic answers supported by the proper tooling. This is the problem statement that platform engineering has the ambition to address. With this, platform engineers have become crucial for organizations, as their role holds the keys to enabling security and engineering standards.

What Is a Platform Engineer?

The role of the platform engineer has three different parts.

Figure 1. Role of the platform engineer

The most obvious one is the role of a technical architect as they have to build an engineering platform that connects all tools and enables processes. The second aspect is a community enabler, which is similar to developer relations roles at technical tooling companies. The third part is a product manager; the competing interests and demands from the developer community need to be prioritized against the technical needs of the platform (consider things like security hardening and patching of outdated components).

Platform Engineer as Technical Architect

In organizations with moderate or high complexity within their technology stack, the number of tools required to build, release, and maintain software is at least a dozen, sometimes more. Integrating these tools and enabling the measurement of meaningful metrics is about as tricky as integrating business applications. After all, the challenges are very similar: Different processes need to be aligned, data models need to be transformed to make them usable, and integration points need to be connected to enable the end-to-end process.

The systems that run the software side of the business have become similarly challenging. The role of the platform engineer here is to look after the architecture of the tools that run the software side — the goal being to make the tools "disappear" and make the build and release of software appear easy.

Platform Engineer as Community Enabler

Software engineers tend to think their solutions are better than those from someone else. As such, the adoption of engineering platforms is a challenge to overcome. Telling engineers to use a specific tool has often been met with resistance. The platform engineer must be a community enabler who works with the engineers to promote the platform and convince them of the benefits. Communication goes both ways in this part of the role as the platform engineer also must listen to the problems and challenges of the platform and identify new features that are high in demand. This leads to the third part of the role.

Platform Engineer as Product Manager

Competing demands on the platform come from the engineers of an organization and other stakeholders like security and, of course, the platform engineers. Prioritizing these demands in a meaningful way is a difficult task as you have to find a balance between all the competing interests, especially as funding for the platform is often a challenge in itself, so speed to value is critical for the ongoing support of the platform. The platform engineer requires good negotiation skills to navigate these challenges.

Overview of Platform Engineering Architecture

We spoke about the role of the platform engineer, but what is in that platform that the platform engineer is building and maintaining? It is easiest to think about three layers and one target environment:

1. The top layer is the developer experience. These are the tools the developer directly engages with — tools that drive the overall workflow, like an Agile lifecycle management tool, a service management tool, and the developer IDE, fit into this.
2. The bottom layer comprises the infrastructure components that must be combined to build application environments. This can be from the public or private cloud and includes traditional data center technologies.
3. In the middle is where most of the complexity sits — the software engineering platform. Here, all the processes that are required to create and deliver software are being orchestrated: CI/CD, security scanning, environment provisioning, and release management.

Figure 2. Platform structure

Making the Switch: How to Adopt Platform Engineering Across DevOps Teams

So where should you start? One successful adoption pattern focuses on identifying developer journeys to define a minimum viable platform. Which capabilities are required to enable a developer journey to achieve an outcome? Think of a task like provisioning an environment, deploying a new API to production, or running a performance test suite. Each is a valid developer journey with multiple touchpoints that potentially require numerous tools. Once you have created the minimum viable platform for the first set of applications or technologies, adoption follows three dimensions: More applications (once the required capabilities are available), more capabilities, and more maturity, thus increasing the levels of automation and/or performance.

Besides worrying about building out the platform with a reasonable approach, three other aspects should be addressed early on: 

1. Community engagement
2. Funding
3. Measuring outcomes from the platform 

Defining a community engagement strategy can be very helpful. This strategy should contain how the information will be shared with the developer community, how feature requests can be made, and how the platform's benefits will be communicated. Defining the forums, the communications, and their respective frequency is also helpful. 

Funding can quickly become a bottleneck, so a funding strategy should be agreed upon early in the platform engineer adoption. This can be one of several strategies, such as dedicated funding, funding for the services provided, or a service tax on all software development. Each has its own benefits and challenges, a discussion of which is beyond the scope of this article. What is essential is to have a sustainable long-term funding strategy that does not depend on stakeholders' goodwill.

Last but not least, the platform engineer needs to be able to show results, which means we need to measure meaningful metrics that showcase why the company is better off with the platform in place. This is often forgotten or an afterthought. Understanding the organization's priorities and aligning the measurement framework to it can help achieve ongoing support. Unfortunately, this usually requires data alignment across multiple tools and is easiest to accomplish when thought about upfront — it becomes increasingly difficult the longer the data models of individual tools remain isolated.

Conclusion

Platform engineering is still pretty new, yet there is already a lot of content on it, which shows how quickly it has gained interest from organizations. There is even a dedicated conference for it, which began in 2022 and has thousands of participants. It's the early days, but current indications show that platform engineering has quickly found market adoption and a passionate community. And while this is happening, the role of the platform engineer will steadily increase in importance, which is already showing up in salaries too.

Hopefully, platform engineering will continue to help organizations reduce complexity for their developers while delivering on the DevOps promise: to provide better solutions faster and more securely.

This is an excerpt from DZone's 2024 Trend Report, The Modern DevOps Lifecycle: Shifting CI/CD and Application Architectures.

For more:

Read the Report