Erlang OTP: Distributed & Fault-Tolerant Systems Programming · Lesson

Designing for High Availability

Apply advanced OTP principles to design and implement highly available services that can withstand failures and remain operational.

Designing for High Availability is a free Erlang OTP: Distributed & Fault-Tolerant Systems Programming lesson on CoddyKit — lesson 1 of 4. You can read the complete lesson below for free — then practise it hands-on in the browser with a built-in code editor and a 24/7 AI tutor. It is part of the Erlang OTP: Distributed & Fault-Tolerant Systems Programming learning path, one of 4 lessons in the course, and your progress syncs across the web and the CoddyKit app.

High Availability: Always On

What is High Availability (HA)? It's about designing systems that keep running even when parts fail. Erlang and OTP are built from the ground up to achieve this.

Imagine a critical service like an online store. If it goes down, sales are lost! HA aims to minimize downtime, ensuring your application remains operational and accessible to users.

Core HA Design Pillars

Achieving High Availability relies on several key design principles:

Redundancy: Having multiple components capable of performing the same task.
Fault Tolerance: The ability to continue operating despite failures.
Automatic Recovery: Systems that detect failures and recover or switch automatically.
No Single Point of Failure (SPOF): Eliminating any component whose failure would bring down the entire system.

Active-Passive Redundancy

The Active-Passive pattern, also known as Hot Standby, involves one primary (active) component and one or more secondary (passive) components.

The active component handles all requests. If it fails, a passive component takes over, becoming the new active. This provides redundancy and minimizes downtime, but the passive component is idle until needed.

Simulating Active-Passive in Erlang

We can simulate an active-passive setup using Erlang processes and monitors. Here, a 'standby' process monitors a 'primary'. If the primary dies, the standby takes over. This is a simplified example of role switching.

Try running this code:

-module(ha_example).
-export([start/0, init/0, primary_loop/0, standby_loop/0]).

start() ->
    Pid = spawn(?MODULE, init, []),
    io:format("Started HA example with ~p~n", [Pid]),
    Pid.

init() ->
    % Simulate starting a primary and a standby
    PrimaryPid = spawn(?MODULE, primary_loop, []),
    StandbyPid = spawn(?MODULE, standby_loop, []),
    io:format("Primary started: ~p~n", [PrimaryPid]),
    io:format("Standby started: ~p~n", [StandbyPid]),

    % Standby monitors Primary to detect its failure
    monitor(process, PrimaryPid),

    % Keep the init process alive to show output
    receive
        _ -> ok
    end.

primary_loop() ->
    io:format("Primary is active and processing requests...~n"),
    timer:sleep(5000), % Simulate work
    io:format("Primary is going down!~n"),
    exit(primary_failure). % Primary fails

standby_loop() ->
    receive
        {'DOWN', _MonitorRef, process, _Pid, _Reason} ->
            io:format("Standby detected Primary failure! Taking over...~n"),
            % In a real system, the standby would now become active
            % and potentially start its own workers or re-register globally.
            become_active()
    end.

become_active() ->
    io:format("Standby is now the new Active!~n"),
    % A real active process would now enter its main loop to handle requests
    timer:sleep(infinity).

Active-Active for Scalability

In an Active-Active pattern, multiple components are simultaneously active, sharing the workload. This offers both redundancy and improved scalability by distributing tasks.

If one active component fails, the others continue processing requests, often with a slight performance degradation. This setup requires careful state management and load balancing to ensure requests are distributed efficiently.

State Replication in HA Systems

A major challenge in HA is maintaining consistent state across redundant components. If an active component fails, its replacement needs access to the most up-to-date information.

Strategies include:

Replication: Copying state changes to standby or other active components (e.g., using Mnesia or custom replication logic).
Shared Storage: Storing state in a highly available external database accessible by all nodes.
Stateless Design: Making components stateless, so any instance can handle any request without needing prior state.

Eliminating Single Points of Failure

A Single Point of Failure (SPOF) is any part of a system whose failure would stop the entire system from working. Identifying and eliminating SPOFs is crucial for HA.

Common SPOFs include:

A single database server.
A single network switch.
A central coordinator process without a backup.

Design your system with redundancy at every critical layer, from hardware to software components.

Liveness: Heartbeats & Health Checks

To enable automatic recovery and failover, components need a way to detect if others are still alive and healthy. This is done through heartbeating and health checks.

Processes can send periodic "I'm alive" messages.
Monitors can detect process crashes immediately (as seen in our example).
Nodes can monitor other nodes using net_kernel:monitor_nodes/1 for cluster-wide health.

Electing a Leader in a Cluster

Sometimes, even in an active-active system, a single coordinator or "leader" is needed to manage a shared resource or ensure global consistency. If this leader fails, a new one must be chosen.

Leader Election is the process of dynamically selecting a new leader from a set of potential candidates in a distributed system. Erlang's global module can help with simple global registration, but for robust election algorithms, custom solutions or libraries are often used.

HA Design Principles Check

Consider a critical Erlang service designed for high availability.

HA Design: Key Takeaways

We've explored how to design highly available Erlang OTP systems:

Understood the pillars: redundancy, fault tolerance, automatic recovery, and no SPOF.
Examined Active-Passive and Active-Active patterns.
Discussed state replication and consistency.
Learned about heartbeating and leader election concepts.

By applying these advanced OTP principles, you can build robust, resilient applications that remain operational even in the face of failures.

Frequently asked questions

Is the “Designing for High Availability” lesson free?

Yes — the full text of “Designing for High Availability” is free to read here on the web, and the Erlang OTP: Distributed & Fault-Tolerant Systems Programming course includes 4 lessons in total. To practise it interactively (a built-in code editor and a 24/7 AI tutor) and unlock the rest of the Erlang OTP: Distributed & Fault-Tolerant Systems Programming course, upgrade to CoddyKit PRO.

What will I learn in “Designing for High Availability”?

Apply advanced OTP principles to design and implement highly available services that can withstand failures and remain operational. You practise Erlang OTP: Distributed & Fault-Tolerant Systems Programming with hands-on code you run directly in the browser, and a 24/7 AI tutor answers your questions as you work through the lesson.

Do I need any experience to start Erlang OTP: Distributed & Fault-Tolerant Systems Programming?

No prior experience is required. Erlang OTP: Distributed & Fault-Tolerant Systems Programming on CoddyKit is structured for beginners through advanced learners; this is — lesson 1 of 4, so you can start here or from the beginning and move at your own pace.

How long does the “Designing for High Availability” lesson take?

Most CoddyKit lessons take about 5–10 minutes. Each one is bite-sized and interactive, so you make steady progress and pick up exactly where you left off across the web and the app.

Can I write and run code in this Erlang OTP: Distributed & Fault-Tolerant Systems Programming lesson?

Yes. Every Erlang OTP: Distributed & Fault-Tolerant Systems Programming lesson includes a built-in code editor, so you write and run real code right in your browser and get instant AI feedback — no local setup required.

All lessons in this course

← Back to Erlang OTP: Distributed & Fault-Tolerant Systems Programming