DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

Enterprise AI Trend Report: Gain insights on ethical AI, MLOps, generative AI, large language models, and much more.

2024 Cloud survey: Share your insights on microservices, containers, K8s, CI/CD, and DevOps (+ enter a $750 raffle!) for our Trend Reports.

PostgreSQL: Learn about the open-source RDBMS' advanced capabilities, core components, common commands and functions, and general DBA tasks.

AI Automation Essentials. Check out the latest Refcard on all things AI automation, including model training, data security, and more.

Related

  • Reporting in Microservices: How To Optimize Performance
  • How To Get Closer to Consistency in Microservice Architecture
  • Modeling Saga as a State Machine
  • Scaling Salesforce Apps Using Heroku Microservices - Part 2

Trending

  • The Future of Kubernetes: Potential Improvements Through Generative AI
  • Deploying Heroku Apps To Staging and Production Environments With GitLab CI/CD
  • The Data Streaming Landscape 2024
  • 10 Tips To Improve Python Coding Skills in 2024
  1. DZone
  2. Data Engineering
  3. Data
  4. Reliable Microservices Data Exchange With Streaming Database

Reliable Microservices Data Exchange With Streaming Database

You will learn how to implement the outbox pattern with a streaming database which can provide a reliable solution for microservices or multiple services data exchange.

By 
Bobur Umurzokov user avatar
Bobur Umurzokov
·
Jul. 26, 23 · Tutorial
Like (8)
Save
Tweet
Share
9.4K Views

Join the DZone community and get the full member experience.

Join For Free

Nowadays, we usually build multiple services for a single product to work, and client apps need to consume functionality from more than one service. Microservices architecture has become a popular approach for building scalable and resilient applications. In a microservices-based system, multiple loosely coupled services work together to deliver the desired functionality. One of the key challenges in such systems is exchanging data between microservices reliably and efficiently. One pattern that can help address this challenge is the Outbox pattern.

In this article, we will explore how to implement the outbox pattern with a streaming database which can provide a reliable solution for microservices or multiple services data exchange.

The Need for Reliable Microservices Data Exchange

In a microservices architecture, each microservice has business logic, is responsible for its own data, has its own local data store, and performs its own operations on that data (Data per service pattern). However, there are scenarios where microservices need to share data with each other or notify other services about any specific data change in real-time to maintain consistency and provide a cohesive experience to end users. For example, consider in a ride-hailing service, there may be multiple microservices responsible for different functionalities, such as user management, ride booking, driver management, and payment processing. When a user requests a ride, it triggers a series of events that need to be propagated to various microservices for processing and updating their data.

Ride-hailing services are where a customer orders the ride from a ride-hailing platform. The best-known such services are Uber, Lyft, and Bolt.

Traditional synchronous communication between microservices can lead to tight coupling and potential performance and reliability issues. A sending service needs to know the location, interface, and contract of other microservices, which can result in a complex web of dependencies. This can make it challenging to evolve, test, and deploy microservices independently, as any change in one microservice may require changes in multiple dependent microservices. Sometimes these target services might temporarily not be available and can introduce performance overhead due to the need for waiting and blocking until a response is received.

Asynchronous and Decoupled Data Exchange

On the other hand, the Outbox pattern promotes asynchronous and decoupled data exchange between microservices. When an event or change occurs in one microservice, it writes the event or changes to its outbox, which acts as a buffer. The outbox can be implemented as a separate database table that the service owns.

The microservice's outbox is then processed by an Outbox Processor (by a separate component or service such as a streaming database RisingWave which is explained in the below section), which reads the events or changes from the outbox and sends them to other microservices or data stores asynchronously. This allows microservices to continue processing requests without waiting for the data exchange to complete, resulting in improved performance and scalability.

Microservice B, which needs to be updated with the changes from Microservice A, receives the events or changes from the outbox processor and applies them to its own state or data store. This ensures that Microservice B's data remains consistent with the changes made in Microservice A.

Now let's see how the Outbox pattern can be implemented using the streaming database with an example of RisingWave. There are other streaming database options in the market; this post helps you understand what streaming database is, when, and why to use it, and discusses some key factors you should consider when choosing the right streaming database for your business.

RisingWave is a streaming database that helps in building real-time event-driven services. It can read directly database change events from traditional databases binlogs or Kafka topics and build a materialized view by joining multiple events together. RisingWave will keep the view up to date as new events come and allow you to query using SQL to have access to the latest changes made to the data.

Outbox pattern with a streaming database

The streaming database can act as a real-time streaming platform (outbox processor). It listens to any write/update operations in the specified database table using its built-in Change Data Capture (CDC) connector, captures changes, and propagates those changes to other microservices in real-time or near real-time with the help of Kafka topics (See how to sink data from RisingWave to a Kafka broker).

In this article, the author Gunnar Morling explains another implementation of the outbox pattern based on CDC and the Debezium connector.

One key advantage of using the streaming database is that you do not need to use both Debezium and Kafka Connect to achieve the same. Also, it has its own storage, you never lose streaming data, and you can create materialized views optimized for querying microservices which is explained in my other blog post.

Plus, it gives us the ability to analyze the data by delivering them to BI and data analytics platforms for making better business decisions based on your application usage. For example, a trip history view is an important feature in a ride-hailing service to provide passengers and drivers with access to their trip history. Instead of querying the individual trip records and calculating various statistics such as total trips, earnings, ratings, etc., a materialized view can be used to store the precomputed trip history information for each user in the streaming database.

Reliable Microservices Data Exchange

With the Outbox pattern and streaming database, here's how the data exchange could work for our example ride-hailing service:

  1. Ride Service (Microservice A): When a user requests a ride, the Ride Service creates the ride details and writes an event, such as "RideRequested," to its outbox table in its own database, let's say MySQL.
  2. By default, the streaming database captures the "RideRequested" event from the Ride Service's outbox table using its connector for MySQL CDC. It processes the event and sends it to other microservices that need to be updated, such as Driver Service (Microservice B), Payment Service (Microservice C), and Notification Service (Microservice D), using a message broker like Apache Kafka.
  3. Driver Service (Microservice B): The Driver Service receives the "RideRequested" event from the Outbox Processor and finds an available driver for the ride based on the ride details. It then writes an event, such as "DriverAssigned," to its own outbox.
  4. Payment Service (Microservice C): The Payment Service also receives the "RideRequested" event from another Kafka topic and calculates the fare for the ride based on the ride details. It then writes an event, such as "FareCalculated," to its own outbox.

We can create multiple Kafka topics to give options to consumer services to subscribe to only specific event types.

Conclusion

As we understood, synchronous communication between microservices can introduce tight coupling, performance overhead, lack of fault tolerance, limited scalability, versioning challenges, and reduced flexibility and agility. By leveraging real-time streaming platforms such as RisingWave and decoupling the data exchange process from the main transactional flow by applying an outbox pattern, you can achieve high performance, reliability, and consistency in your microservices architecture.

Data exchange Database Data (computing) Driver (software) kafka microservice

Published at DZone with permission of Bobur Umurzokov. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Reporting in Microservices: How To Optimize Performance
  • How To Get Closer to Consistency in Microservice Architecture
  • Modeling Saga as a State Machine
  • Scaling Salesforce Apps Using Heroku Microservices - Part 2

Partner Resources


Comments

ABOUT US

  • About DZone
  • Send feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends: