The blog at Martin Fowler’s site is a great place to catch up on latest consensus on data architecting. Last year on that blog, contributor Zhamak Dehghani introduced the notion of the Data Mesh, an shift from the “centralised data lake” paradigm, to attain a more distributed vision of data. The latest paper Data Mesh Principles and Logical Architecture goes into more depth and recommends 4x principles for implementation:
- Domain-oriented decentralized data ownership and architecture: the ecosystem creating and consuming data can scale out as the number of sources of data, number of use cases, and diversity of access models to the data increases; simply increase the autonomous nodes on the mesh.
- Data as a product: data users can easily discover, understand and securely use high quality data with a delightful experience; data that is distributed across many domains.
- Self-serve data infrastructure as a platform: the domain teams can create and consume data products autonomously using the platform abstractions, hiding the complexity of building, executing and maintaining secure and interoperable data products.
- Federated computational governance: data users can get value from aggregation and correlation of independent data products — the mesh is behaving as an ecosystem following global interoperability standards; standards that are baked computationally into the platform.