Database Sharding & Replication

As applications grow, a single database can become a bottleneck. High traffic, complex queries, or large datasets can slow things down.

To handle increasing load and ensure responsiveness, databases need to scale. This lesson explores two key techniques: Replication and Sharding.

Database replication is the process of creating and maintaining multiple copies of a database.

These copies, often on different servers, serve two main purposes:

• Improved Read Performance: Distribute read requests across multiple copies.
• High Availability: If one server fails, another copy can take over.

The most common replication model is Master-Slave. Here's how it works:

• One database server is designated as the master. It handles all write operations (inserts, updates, deletes).
• Other servers are slaves. They receive a copy of the master's data and handle read operations.
• Changes from the master are continuously synchronized to the slaves.

While Master-Slave is robust, Multi-Master replication allows multiple database servers to accept write operations.

This can further improve write scalability and availability, but it introduces significant complexity in managing data consistency and resolving potential conflicts when the same data is updated on different masters simultaneously.

Benefits of Replication:

• Read Scalability: Distributes read load, improving response times.
• High Availability: Provides fault tolerance; a slave can become master if needed.
• Disaster Recovery: Data copies are available in case of data loss on one server.

Drawbacks:

• Write Latency: All writes still go through the master.
• Data Staleness: Slaves might lag the master, leading to 'eventual consistency'.
• Complexity: Setup and management require careful planning.

Database sharding is a technique that breaks a large database into smaller, more manageable pieces called shards. Each shard is a complete, independent database.

Instead of one massive database, you have multiple smaller databases, each storing a subset of your total data. This is also known as horizontal partitioning.

The core of sharding is the shard key (or partition key). This is a column or set of columns in your data that determines which shard a particular row belongs to.

Choosing an effective shard key is crucial for even data distribution and efficient querying. Common choices include user_id, tenant_id, or a geographic region.

How do we decide which data goes where?

• Range-Based Sharding: Data is partitioned based on a range of values in the shard key (e.g., users A-M on Shard 1, N-Z on Shard 2).
• Hash-Based Sharding: A hash function is applied to the shard key, and the result determines the shard. This often leads to more even distribution.
• Directory-Based Sharding: A lookup table (directory) maps the shard key to the appropriate shard.

While sharding offers immense scalability, it comes with significant challenges:

• Increased Complexity: More databases to manage, distribute, and back up.
• Cross-Shard Queries: Queries requiring data from multiple shards are complex and often less efficient.
• Data Rebalancing: If one shard becomes too large or hot, redistributing data across shards (rebalancing) is a difficult operation.
• Shard Key Choice: A poor shard key can lead to uneven distribution ('hot spots').

Test your knowledge on database scaling techniques.

We've explored two powerful database scaling techniques:

• Replication: Copies data for read scalability and high availability, often using a Master-Slave model.
• Sharding: Horizontally partitions data into independent shards, using a shard key to distribute data and scale write operations.

Both techniques significantly improve performance and resilience but introduce operational complexity. Choosing the right strategy depends on your application's specific needs and traffic patterns.