What is data mesh and why is it important?


7 Feb 2022

Image: © fantasyform/Stock.adobe.com

Forrester’s Michele Goetz explains the concept of data mesh and how businesses can execute it as part of their data strategies.

Click here to view the full Data Science Week series.

The new buzz in the data world is data mesh. With it comes a lot of confusion. The domain-centric and data governance-by-design principles feel simple and intuitive. Yet, everyone is asking, “How do you execute on data mesh?”

The secret is that data mesh principles are not new. Organisations already model data, stand up data warehouses, master their data and ensure data quality.

Data governance artefacts to define and establish policies, check. Utilisation of modern flexible data warehouses and knowledge graphs to navigate data and data relationships for insight, check. Ontologies, taxonomies and data catalogues created and populated by data subject matter experts, check.

The difference today is that data mesh shifts the data strategy from predominantly analytic visualisation to artificial intelligence and real-time solutions. Data development and application development collide as data is set in motion for real-time, distributed and IoT applications.

Operational systems from a customer relationship management (CRM) application to a smart city network capture much more data to see a complete picture of the ecosystem. Data mesh models data as a twin of the business in the language of the business.

Consider the logistics sector. Trucking, rail and shipping are deeply connected to supply chain operations and customers. The data ecosystem relies on several domains internally and under control, as well as external data with sharing and consent policies.

Data mesh addresses the foundation of interoperability by applying standards, definitions and protocols specific to the hand-off points for each decision and step in the process.

When there is a 20-minute back-up on a highway, data from the highway infrastructure can be picked up in real time and used to optimise a truck route to keep deliveries on time. Infrastructure and truck communicate in a common language for the right outcome.

There are five factors that shape the application of data mesh to evolve from watching the world to influencing the world through data-driven value.

Semantics

Expand logical domain definitions and models to represent semantic views and understand. By applying the business language in the form of relationships, classifications, labels and tags, working with data becomes declarative.

In the no-code/low-code application development environments, semantics improves and speeds up the mapping between the right data and what is needed in a business process. This means better interoperability between data and application.

Data products

Applications rely on services and APIs to access data sources and pipelines. These elements or components are data products.

Data products output a data source, event, query, schema, control or insight. They are designed to match the data requirement of the application and take on the heavy lifting of handling complex data logic to simplify application process routing.

Alternatively, they deliver services to balance and optimise the cost to performance for production payloads.

Portfolio management

As data products are defined at a more granular level, portfolio management is crucial to maintaining order and ensuring alignment, speed and reuse of capabilities.

Power levelling of data product portfolio management comes with harmonising data development with the broader solution and business digital portfolio. Thus, data comes into alignment by capability, priority and defined value and outcomes.

DataOps

Rather than executing data development and engineering for monolithic deployments, DataOps takes on the agile and continuous integration and delivery of data products.

Architects at the enterprise and line of business level provide patterns and blueprints as starting points that offset potential technical debt.

Data engineers own the products they develop, meaning DataOps takes on responsibility of the quality, speed and outcomes for data provisioning and through ongoing optimisation and life cycle management. Thus, data is governed by design and not an afterthought.

Federation

Circling back to the semantics of the data, connections to the subject matter experts must be strong and innate.

Large enterprises and global organisations are building organisation and operation models to cover centralised data and governance foundations and shared artefact, while also pushing data development up into solutions teams in lines of business.

In lines of business, data engineers are elevated to members of the overall application development team. They then take responsibility to provide their products and domain centric knowledge back to the centralised data services environment.

Up your business outcomes with data mesh-driven data decisioning. Assess your competency in the five data mesh factors for success across data management, engineering, governance and consumption practice.

Ensure these competency centres are coordinated and intertwined. Remember, data mesh is not just about the data, it is about making data work for a resilient, competitive business.

By Michele Goetz

Michele Goetz is vice-president and principal analyst for business insights at Forrester.

Don’t miss out on the knowledge you need to succeed. Sign up for the Daily Brief, Silicon Republic’s digest of need-to-know sci-tech news.