Designing Data-Intensive Applications
Data is at the center of many challenges in system design today. Difficult issues need to be figured out, such as scalability, consistency, reliability, efficiency, and maintainability. In addition, we have an overwhelming variety of tools, including relational databases, NoSQL datastores, stream or batch processors, and message brokers. What are the right choices for your application? How do you make sense of all these buzzwords? In this practical and comprehensive guide, author Martin Kleppmann helps you navigate this diverse landscape by examining the pros and cons of various technologies for processing and storing data. Software keeps changing, but the fundamental principles remain the same. With this book, software engineers and architects will learn how to apply those ideas in practice, and how to make full use of data in modern applications. Peer under the hood of the systems you already use, and learn how to use and operate them more effectively Make informed decisions by identifying the strengths and weaknesses of different tools Navigate the trade-offs around consistency, scalability, fault tolerance, and complexity Understand the distributed systems research upon which modern databases are built Peek behind the scenes of major online services, and learn from their architectures
Foundations of Data Reliability
Building robust data systems requires a balance between reliability, scalability, and maintainability. Reliability ensures the system functions correctly even during hardware or software failures. Scalability addresses the system's ability to handle growing load by adjusting resources, often measured through throughput and response times. Maintainability focuses on the ease with which developers can understand and evolve the codebase. Kleppmann argues that these pillars are the foundation of any data-intensive application. By understanding how different architectural choices affect these properties, engineers can design systems that remain resilient and performant as user demands and data volumes inevitably increase over time.
The Battle of Data Models
Data models define how we think about the world and structure our information. This section explores the historical dominance of the relational model and the rise of NoSQL alternatives like document and graph databases. Document databases provide flexibility for nested data, while graph databases excel at modeling complex relationships. Choosing the right model depends on the application's data structure and access patterns. The chapter also discusses query languages, comparing imperative and declarative approaches. Ultimately, the choice of a data model has a profound impact on how easily an application can be built, understood, and scaled to meet changing business requirements.
Beneath the Hood of Storage Engines
At the core of every database is a storage engine responsible for writing data to disk and retrieving it efficiently. Kleppmann contrasts log-structured storage, such as Log-Structured Merge (LSM) trees used in NoSQL databases, with page-oriented storage like B-trees found in traditional relational systems. LSM-trees optimize for high-throughput writes by appending data to logs, while B-trees offer balanced performance for read-heavy workloads. Understanding indexing techniques, including hash indexes and specialized secondary indexes, is crucial for optimizing query performance. This deep dive into storage internals reveals how low-level hardware constraints influence high-level architectural decisions in data-intensive systems.
Evolution through Encoding
As applications evolve, their data schemas must also change without causing downtime. This section examines data encoding formats like JSON, XML, and binary protocols such as Protocol Buffers, Thrift, and Avro. Effective encoding allows for forward and backward compatibility, enabling different versions of a service to communicate seamlessly. Kleppmann explores how schema evolution affects data stored in databases and messages sent over the network. By prioritizing compatibility, developers can deploy updates independently and avoid the pitfalls of "big bang" migrations. Mastering data evolution is essential for maintaining a long-lived system that can adapt to new features and changing requirements.
Scaling Out via Replication
Replication involves keeping copies of the same data on multiple machines to ensure high availability and low latency. This section covers various replication strategies, including single-leader, multi-leader, and leaderless architectures. Each approach offers different trade-offs regarding consistency, performance, and complexity. Kleppmann discusses the challenges of replication lag, where followers fall behind the leader, leading to "read-your-writes" and monotonic read issues. By understanding how to handle node failures and network partitions, engineers can build distributed systems that remain operational even when parts of the infrastructure fail. Replication is the cornerstone of modern, globally distributed data platforms.
The Art of Partitioning
For massive datasets that exceed the capacity of a single machine, partitioning or sharding becomes necessary. This section explores how to break large datasets into smaller chunks distributed across a cluster. Key challenges include selecting a partition key to avoid "hot spots" where a single node is overwhelmed by traffic. Kleppmann explains methods like range partitioning and hash partitioning, along with the complexities of rebalancing data as the cluster grows or shrinks. Effective partitioning ensures that both data and query load are distributed evenly, allowing the system to scale horizontally and maintain high performance as the total volume of data expands.
Mastering Transaction Isolation
Transactions provide a safety net by grouping multiple operations into an atomic unit, ensuring data integrity despite errors or concurrent access. This section dives into ACID properties and the reality of isolation levels like read committed, snapshot isolation, and serializability. Kleppmann highlights the dangers of race conditions, such as "dirty reads" and "write skew," which can lead to subtle but catastrophic bugs. By exploring how databases implement concurrency control through locking or multi-versioning, developers can choose the appropriate isolation level for their specific use case, balancing the need for strict consistency with the requirement for high-concurrency performance.
The Chaos of Distributed Realities
Moving from a single machine to a distributed environment introduces a host of unreliable factors. This section confronts the harsh realities of partial failures, network delays, and unreliable clocks. Kleppmann explains why we cannot assume a global notion of time or order in a distributed system, which complicates synchronization and event sequencing. The "Split-Brain" problem and the difficulty of detecting node failures are central themes. By acknowledging these inherent limitations, engineers can design systems that use heartbeats and timeouts to maintain a shared state of knowledge, preparing for the inevitable glitches that occur in large-scale cloud environments.
Achieving Consensus and Consistency
Achieving consensus is one of the most difficult challenges in distributed computing, requiring multiple nodes to agree on a value or an order of events. This section explores the concepts of linearizability, total order broadcast, and distributed transactions. Kleppmann explains the mechanics of consensus algorithms like Paxos and Raft, which provide the foundation for fault-tolerant coordination. The chapter also discusses the Two-Phase Commit (2PC) protocol and its limitations in high-latency environments. Mastering these concepts allows developers to build systems that maintain a consistent state across a cluster, ensuring that every user sees a unified and coherent view of the data.
Processing Data in Flux
The final chapters shift from request-response models to data-flow systems, covering batch and stream processing. Kleppmann contrasts the "bounded" data of MapReduce and Spark with the "unbounded" streams of Kafka and Flink. He explores the concept of "unbundling the database," where different tools are used for storage, indexing, and processing, all synchronized through a common log. This vision for the future emphasizes the integration of data from multiple sources to create rich, real-time applications. By embracing asynchronous event-driven architectures, developers can build flexible systems that derive maximum value from data as it flows through the organization.