Process automation is a vital part of digital transformation for organizations across industries. Automation enables them to respond to customer and market demand faster, while also helping to improve customer service, lower costs, and drive business growth. However, many organizations find that dependencies between teams, software, and systems make it hard to efficiently automate processes, slowing down time-to-market for new products, features, and functionality.

You can address this challenge using a microservice architectural style and a workflow engine for microservice orchestration. A microservice architecture splits a system into individual services, each focused on a single business capability. A workflow engine manages business processes that require state handling, monitoring, reporting, and other functionality.

In this Refcard, we’ll take a look at key techniques for well-designed microservices, important things to consider when implementing a microservice architecture, and what role a workflow engine plays within a microservice architecture.

When designing a microservice architecture, you define logical boundaries around meaningful business or domain capabilities. Within a service boundary, the development team has the autonomy to do whatever it takes to implement business and technical requirements. In this way, microservices provide a high level of flexibility and agility; each team can deliver changes at their own pace without having to rebuild and redeploy a monolithic business application every time.

It’s important to keep in mind that an end-to-end, automated business process typically stretches across multiple individual microservices that must communicate and collaborate to achieve a business outcome. That’s where a workflow engine comes in. A workflow engine manages the flow of a business process across microservices by visualizing, operating, and reporting on the process. Note that a workflow engine doesn’t necessarily have to be implemented as a single, central component. You can opt for multiple, decentralized workflow engines to further increase team independence.

Advantages of a Workflow Engine for Microservices

Process automation that is powered by a workflow engine offers many advantages, such as:

• State handling – Persists the state of each instance of a business process (e.g., each order placed on an e-commerce website)
• Explicit processes – Makes business processes explicit instead of burying them in code (i.e., they’re easier for teams to understand and modify)
• Message correlation and coordination – Merges messages belonging to a single process instance and decide next steps — process modeling languages (e.g., Business Process Model and Notation) automatically implement message patterns such as sequences, synchronization, mutual exclusion, and timeouts
• Compensation for problems – Compensates if a business process encounters a problem that requires previously completed steps to be undone
• Timeout handling – Tracks the passage of time and, if a message does not arrive on time, automatically takes action or switches to another path in the process flow
• Error handling – Allows you to specify the behavior that should happen when an error occurs (e.g., retrying an action, taking another path)
• Transparency of status – Enables operations teams to monitor the status of process instances in real time
• Collaboration – Provides graphical models of business processes that facilitate discussion between business stakeholders, developers, and operations teams

Design Independent Microservices

Independent microservices allow you to scale your development workforce while maintaining agility and a rapid development pace. Well-designed microservices have the following attributes:

• Independent lifecycles – Teams must be able to develop, deploy, and make technology choices for their microservices without involving other services or teams. Starting, stopping, or changing a microservice should not interrupt other services.
• Stable interfaces – Microservice interfaces must not break during updates. If incompatible changes to an interface are required, they should be implemented by versioning the interface.
• Asynchronous communication – A microservice needs to communicate with other services, but the peer might not be available immediately. Asynchronous communication via messaging can remove dependencies on the availability of other services.
• Resilience and fault tolerance – A microservice should still run if other services in the system cause problems.
• Graceful degradation – If one microservice fails, the rest of the system should still function as best as possible (e.g., falling back to a default flow).
• Independent scalability – Each microservice should have access to the resources needed to respond to changes in system load without impacting other services.
• Local data storage – Each microservice should store a local copy of the data needed to fulfill its task.

Align Microservices to the Business Domain

Organizations often struggle to implement microservices that support an end-to-end business process while maintaining independence and preserving development team autonomy. For example, imagine that you have a fulfillment process for online orders of digital goods. It requires microservices such as payment, inventory, shipping, and so on. Although each service operates independently, the order fulfillment process requires a logical chain of events like the one shown in Figure 1.

Figure 1: Example of an order fulfillment process

In this scenario, an appropriate solution is to align a single order fulfillment microservice with the business need to fulfill digital orders. This microservice would be independent because it would coordinate all tasks required for an order: retrieving customer payments, fetching digital goods, and so on. This approach also preserves team autonomy because one team can develop, deploy, and redeploy the order fulfillment microservice on their own.

Balance Choreography and Orchestration

Microservices are often associated with the concept of event-driven communication. An event is something significant that happens in a system (e.g., a customer places an order). When communication is event-driven, services emit events without knowing which other services will pick them up. Services also react to events without knowing where they came from. Each service simply subscribes to topics that are relevant to its purpose and receives a notification when an event related to that topic is emitted. Event-driven communication between microservices is known as “choreography.”

Event-Driven vs. Command-Driven Communication

Event-driven communication allows a high degree of independence for microservices and autonomy for development teams. However, it poses a challenge because, as described earlier, many business activities require a logical chain of events. The event chain manifests as a series of event subscriptions that implement a logical flow; however, it is not explicitly defined anywhere, making it hard for people to understand, monitor, and troubleshoot the large-scale flow of activities.

Command-driven communication is a different approach through which a service can instruct another service to take an action. The recipient can accept or reject the command, but it cannot ignore it. In contrast to event-driven communication, the sender knows which service will receive its command. Command-driven communication between microservices is known as “orchestration.” This approach lets the microservice that implements the flow of business activity control the actions and timing of other services, thus maintaining an overview of the process. A workflow engine is extremely helpful for orchestration because it allows you to model the flow explicitly and monitor it as it runs in real time.

Choreography and orchestration are both valuable communication styles, and different scenarios will call for one or the other. Effective architectures incorporate both styles and apply them where they best suit the communication need. As you design and implement microservices, carefully consider whether events or commands are the right approach for every single communication link between services.

Manage State for Long-Running Processes

In addition to communication management, state management is important for microservices. A business process that can’t be completed right away requires the state of process instances to be persisted until the process is finished.

In the order fulfillment example, receiving payment for an order is potentially a long-running process. A payment that is processed through PayPal might take a few seconds, while payments via bank transfers could take days or even weeks. The order fulfillment process has to wait for the payment to complete, which means that the state of every order in the system must be persisted.

The need to persist state data usually leads to additional business requirements. For example, if a payment fails, you might give the customer seven days to try a different payment method, which requires the system to track time and timeouts.

Figure 2: Timeout for payment in an order fulfillment process

A workflow engine is an ideal tool for managing the state of long-running processes. Most workflow engines execute process models that are expressed in a process modeling language. A prominent example is the ISO standard Business Process Model and Notation (BPMN), which allows you to design processes that are both graphical and executable. The workflow engine persists all data for instances of processes, even over long periods of time. It also facilitates monitoring, alerting, and reporting on a process level, often leveraging the visual as shown in Figure 3 (which indicates the number of active process instances for each task).

Figure 3: Order fulfillment process expressed in BPMN

Whether you’re working on a greenfield technology project or migrating away from legacy business applications, microservice-based process automation can help you scale your software development efforts and drive business growth. Here’s how to get started.

Identify Your Biggest Pain Points

Look for processes that cause the most problems for the business:

• Which are the slowest?
• Which require the most manual effort?
• If some processes are already automated, which of those are the most brittle or the hardest to change?

These pain points indicate areas where replacing manual processes or legacy automation with a microservice will add the most immediate value for both technical and business stakeholders.

Align Boundaries to Business Processes

As you search for pain points, keep in mind that each microservice should be well aligned to your business requirements and goals. Don’t define boundaries between services based on organizational structure, technology choices, or the presence of technical debt. Taking the time to model your business processes will help you understand where service boundaries should lie.

Consider the Data Model

Instead of relying on a shared database as a single source of truth, allow state data to be distributed across microservices. This approach ensures that each microservice has the data it needs and will allow the system to scale in response to load.

Gradually Introduce Microservices

If you have monolithic business applications that must be replaced, a “big bang” approach is time-consuming and risky. Instead, focus on gradually replacing individual capabilities in each legacy application with microservices that fulfill the business need. Remember that as you build each microservice, it must respect the boundaries that you discovered when you modeled your business processes.

Manage Processes With a Workflow Engine

A workflow engine perfectly complements a microservice architecture because it can persist the state of process instances, allowing long-running processes. It can also manage timeouts and alerts, compensate when business transactions fail, and provide rich reporting on the efficiency of business processes. In addition, the right workflow engine can orchestrate processes that span microservices and legacy business applications, supporting the gradual transition from old to new.

Prepare for Communication Failures

A microservice environment is a highly distributed system, so communication between services will fail occasionally. When failures happen, it’s best to make each microservice responsible for resolving the failure itself. A workflow engine helps by persisting the state of process instances and handling related functionality such as monitoring for timeouts and executing fallback options. The engine can also report on metrics such as the number of workflows that couldn’t be processed in time.

Compensate for Failed Business Transactions

When something goes wrong in a business process, all process steps that were already executed must be undone in the correct order so that the overall system returns to a consistent state. In a traditional application architecture with a user interface, a business logic layer, and a relational database, this work is usually handled by database transactions. However, these types of transactions aren’t suited to a distributed system. Instead, a workflow engine can handle the necessary compensating activities, automatically and reliably, across multiple microservices.

Process automation is a critical concern for organizations seeking to deliver value to their customers faster, and one way to automate processes is to use a microservice architecture. In a microservice architecture, end-to-end business processes are distributed across services that operate independently and communicate with one another to accomplish business tasks. Microservices facilitate business agility by reducing dependencies between teams and systems so that new products, features, and functionality can be developed and rolled out as quickly and flexibly as possible.

Microservices come with some considerations — they must be:

• Able to communicate with one another without being tightly coupled.
• Aligned to the business domain in a way that preserves independence and team autonomy.
• Designed to handle long-running business processes that require the state of process instances to be persisted over time.

A workflow engine addresses these considerations effectively because it can manage complex communication between services, persist the state of instances through extended processes, track time and handle timeouts, and compensate for failed or cancelled business transactions. A workflow engine also provides value-adding features such as graphical visualizations, status monitoring, and reporting.