Erlang OTP: Distributed & Fault-Tolerant Systems Programming · Lesson

Handling Network Partitions

Explore strategies for gracefully handling network splits and merges in a distributed Erlang cluster to maintain system integrity.

Handling Network Partitions is a free Erlang OTP: Distributed & Fault-Tolerant Systems Programming lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Erlang OTP: Distributed & Fault-Tolerant Systems Programming learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

Understanding Network Partitions

In distributed systems, a network partition happens when parts of the system can no longer communicate with each other due to network failures. Think of it like a bridge collapsing, splitting a city into disconnected districts.

This can lead to a "split-brain" scenario, where different parts of your Erlang cluster believe they are the only active ones. This often results in data inconsistency and service disruption.

Erlang Node Connectivity

Erlang nodes communicate by forming a distributed system. They connect to each other using a process called net_kernel. When a node starts, it tries to find and connect to other known nodes.

Use -sname for short names (local network).
Use -name for full names (across networks).
All nodes must share the same magic cookie for security.

Here's a simple module. Compile it and run MyNode.get_name(). in the Erlang shell after starting with erl -sname mynode:

-module(my_node).
-export([get_name/0]).

get_name() ->
    node().

Monitoring Node Status

Erlang provides built-in mechanisms to detect when a node disconnects. The monitor_node/2 function allows a process to receive messages when the status of another node changes (e.g., up or down).

This is crucial for reacting to unexpected node failures or network issues. Let's see how a process can monitor another node:

-module(node_monitor).
-export([start/1]).

start(OtherNode) ->
    Pid = spawn(fun() -> init(OtherNode) end),
    {ok, Pid}.

init(OtherNode) ->
    io:format("~p monitoring ~p~n", [self(), OtherNode]),
    erlang:monitor_node(OtherNode, true),
    receive
        {nodeup, Node} ->
            io:format("Node ~p is UP~n", [Node]);
        {nodedown, Node} ->
            io:format("Node ~p is DOWN!~n", [Node])
    end,
    io:format("Monitor process ~p exiting.~n", [self()]).

Beyond Simple Disconnection

While monitor_node is powerful, it primarily tells you if a TCP connection to a node has dropped. This might not always mean a full "partition".

Short network blips or a slow network can cause temporary disconnections, leading to false positives. A true partition implies a sustained inability to communicate between groups of nodes.

Network lag can delay detection.
Brief outages might not warrant full system reaction.
Application-level health checks are often needed.

Quorum and Majority Wins

To avoid "split-brain" in a network partition, distributed systems often use quorum. A quorum is the minimum number of nodes that must agree on an operation (or simply be reachable) for it to be considered valid.

The "majority wins" strategy is a common quorum approach:

Only the partition containing more than half of the total nodes is allowed to continue operations.
Other partitions (minority) should halt or become read-only.

This prevents conflicting updates and ensures data consistency.

Tracking Active Membership

To implement "majority wins," each node needs to know the total cluster size and which nodes are currently reachable. This creates a "membership oracle".

While a full implementation is complex, we can simulate a basic reachability check by having each node periodically "ping" its known peers. If a node can reach a majority of its peers, it considers itself "active".

Here's a conceptual module for a node to ping others:

-module(ping_checker).
-export([start/2, ping_peers/1]).

start(KnownPeers, Interval) ->
    Pid = spawn(fun() -> init(KnownPeers, Interval) end),
    {ok, Pid}.

init(KnownPeers, Interval) ->
    ping_peers(KnownPeers),
    timer:sleep(Interval),
    init(KnownPeers, Interval).

ping_peers(Peers) ->
    io:format("~p: Pinging peers: ~p~n", [node(), Peers]),
    ActivePeers = lists:filter(fun(Peer) ->
        case net_adm:ping(Peer) of
            pong -> true;
            pang -> false
        end
    end, Peers),
    io:format("~p: Reachable peers: ~p~n", [node(), ActivePeers]),
    TotalNodes = length(Peers) + 1, % Include self
    ReachableCount = length(ActivePeers) + 1,
    if
        ReachableCount > TotalNodes / 2 ->
            io:format("~p: I am in the MAJORITY partition!~n", [node()]);
        true ->
            io:format("~p: I am in the MINORITY partition or isolated.~n", [node()])
    end.

Fencing for Safety

When a network partition occurs and a minority partition is identified, it's crucial to prevent it from causing harm (e.g., writing conflicting data). This process is called fencing.

Fencing ensures that only the "winning" (majority) partition can continue to operate and modify shared state. Common fencing actions include:

Shutting down services in the minority partition.
Disabling write operations.
Isolating resources (e.g., database access).

The goal is to prevent "split-brain" from corrupting data.

Reconciling Divergent States

After a network partition heals and nodes reconnect, their states might have diverged. This is because the active partition continued operations while the isolated ones were inactive or performing different actions.

Data reconciliation is the process of resolving these conflicts and bringing all nodes back to a consistent state. Common strategies include:

Last Write Wins (LWW): The most recent update (based on timestamp) is chosen.
Conflict Resolution Functions: Application-specific logic to merge data.

Designing for eventual consistency is key.

Partition Strategy Check

Consider a 5-node Erlang cluster. A network partition occurs, splitting it into two groups: Node A, B (Group 1) and Node C, D, E (Group 2). Which of the following statements about handling this partition are generally TRUE to maintain data integrity and availability?

Recap: Resilient Partitions

We've explored how to handle network partitions, a critical aspect of building resilient distributed Erlang applications. Key takeaways include:

Detection: Beyond simple disconnections, using application-level health checks.
Quorum: Employing strategies like "majority wins" to ensure only one active partition.
Fencing: Preventing minority partitions from causing data inconsistencies.
Reconciliation: Strategies for merging divergent states when partitions heal.

These principles help your Erlang systems remain available and consistent even in the face of network instability.

Frequently asked questions

Is the “Handling Network Partitions” lesson free?

Yes — the full text of “Handling Network Partitions” is free to read here on the web, and the Erlang OTP: Distributed & Fault-Tolerant Systems Programming course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Erlang OTP: Distributed & Fault-Tolerant Systems Programming course, upgrade to CoddyKit PRO.

What will I learn in “Handling Network Partitions”?

Explore strategies for gracefully handling network splits and merges in a distributed Erlang cluster to maintain system integrity. You practise Erlang OTP: Distributed & Fault-Tolerant Systems Programming with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Erlang OTP: Distributed & Fault-Tolerant Systems Programming?

No prior experience is required. Erlang OTP: Distributed & Fault-Tolerant Systems Programming on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Handling Network Partitions” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Erlang OTP: Distributed & Fault-Tolerant Systems Programming lesson?

Yes. Every Erlang OTP: Distributed & Fault-Tolerant Systems Programming lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Erlang OTP: Distributed & Fault-Tolerant Systems Programming